On Clustering Histograms with k-Means by Using Mixed α-Divergences

Abstract

Clustering sets of histograms has become popular thanks to the success of the generic method of bag-of-X used in text categorization and in visual categorization applications. In this paper, we investigate the use of a parametric family of distortion measures, called the α-divergences, for clustering histograms. Since it usually makes sense to deal with symmetric divergences in information retrieval systems, we symmetrize the α-divergences using the concept of mixed divergences. First, we present a novel extension of k-means clustering to mixed divergences. Second, we extend the k-means++ seeding to mixed α-divergences and report a guaranteed probabilistic bound. Finally, we describe a soft clustering technique for mixed α-divergences.

1. Introduction: Motivation and Background

1.1. Clustering Histograms in the Bag-of-Word Modeling Paradigm

A common task of information retrieval (IR) systems is to classify documents into categories. Given a training set of documents labeled with categories, one asks to classify new incoming documents. Text categorisation [1,2] proceeds by first defining a dictionary of words from a corpus. It then models each document by a word count yielding a word distribution histogram per document (see the University of California, Irvine, UCI, machine learning repository for such data-sets [3]). The importance of the words in the dictionary can be weighted by the term frequency-inverse document frequency [2] (tf-idf) that takes into account both the frequency of the words in a given document, but also of the frequency of the words in all documents: Namely, the tf-idf weight for a given word in a given document is the product of the frequency of that word in the document times the logarithm of the ratio of the number of documents divided by the document frequency of the word [2]. Defining a proper distance between histograms allows one to:

• Classify a new on-line document: We first calculate its word distribution histogram signature and seek for the labeled document, which has the most similar histogram to deduce its category tag.

• Find the initial set of categories: we cluster all document histograms and assign a category per cluster.

This text classification method based on the representation of the bag-of -words (BoWs) has also been instrumental in computer vision for efficient object categorization [4] and recognition in natural images [5]. This paradigm is called bag-of-features [6] (BoFs) in the general case. It first requires one to create a dictionary of “visual words” by quantizing keypoints (e.g., affine invariant descriptors of image patches) of the training database. Quantization is performed using the k-means [7–9] algorithm that partitions n data $\mathcal{X}=\left\{x_{\text{1}},\dots ,x_n\right\}$ into k pairwise disjoint clusters C₁,…, C_k, where each data element belongs to the closest cluster center (i.e., the cluster prototype). From a given initialization, batched k-means first assigns data points to their closest centers and then updates the cluster centers and reiterates this process until convergence is met to a local minimum (not necessarily the global minimum) after a provably finite number of steps. Csurka et al. [4] used the squared Euclidean distance for building the visual vocabulary. Depending on the chosen features, other distances have proven useful. For example, the symmetrized Kullback–Leibler (KL) divergence was shown to perform experimentally better than the Euclidean or squared Euclidean distances for a compressed histogram of gradient descriptors [10] (CHoGs), even if it is not a metric distance, since its fails to satisfy the triangular inequality. To summarize, k-means histogram clustering with respect to the symmetrized KL (called Jeffreys divergence J) can be used to quantize both visual words and document categories. Nowadays, the seminal bag-of-word method has been generalized fruitfully to various settings using the generic bag-of-X paradigm, like the bag-of-textons [6], the bag-of-readers [11], etc. Bag-of-X represents each data (e.g., document, image, etc.) as an histogram of codeword count indices. Furthermore, the semantic space [12] paradigm has been recently explored to overcome two drawbacks of the bag-of-X paradigms: the high-dimensionality of the histograms (number of bins) and difficult human interpretation of the codewords due to the lack of semantic information. In semantic space, modeling relies on semantic multinomials that are discrete frequency histograms; see [12].

In summary, clustering histograms with respect to symmetric distances (like the symmetrized KL divergence) is playing an increasing role. It turns out that the symmetrized KL divergence belongs to a 1-parameter family of divergences, called symmetrized α-divergences, or Jeffreys α-divergence [13].

1.2. Contributions

Since divergences D(p : q) are usually asymmetric distortion measures between two objects p and q, one has to often consider two kinds of centroids obtained by carrying the minimization process either on the left argument or on the right argument of the divergences; see [14]. In theory, it is enough to consider only one type of centroid, say the right centroid, since the left centroid with respect to a divergence D(p : q) is equivalent to the right centroid with respect to the mirror divergence D′(p : q) = D(q : p). In this paper, we consider mixed divergences [15] that allow one to handle in a unified way the arithmetic symmetrization $S(p,q)=\frac{1}{2}(D(p:q)+D(q:p))$ of a given divergence D(p : q) with both the sided divergences: D(p : q) and its mirror divergence D′(p : q). The mixed α-divergence is the mixed divergence obtained for the α-divergence. We term α-clustering the clustering with respect to α-divergences and mixed α-clustering the clustering w.r.t. mixed α-divergences [16]. Our main contributions are to extend the celebrated batched k-means [7–9] algorithm to mixed divergences by associating two dual centroids per cluster and to generalize the probabilistically guaranteed good seeding of k-means++ [17] to mixed α-divergences. The mixed α-seedings provide guaranteed probabilistic clustering bounds by picking up seeds from the data and do not require explicitly computing of centroids. Therefore, it follows a fast clustering technique in practice, even when cluster centers are not available in closed form. We also consider clustering histograms by explicitly building the symmetrized α-centroids and end up with a variational k-means when the centroids are not available in closed-form, Finally, we investigate soft mixed α-clustering and discuss topics related to α-clustering. Note that clustering with respect to non-symmetrized α-divergences has been recently investigated independently in [18] and proven useful in several applications.

1.3. Outline of the Paper

The paper is organized as follows: Section 2 introduces the notion of mixed divergences, presents an extension of k-means to mixed divergences and recalls some properties of α-divergences. Section 3 describes the α-seeding techniques and reports a probabilistically-guaranteed bound on the clustering quality. Section 4 investigates the various sided/symmetrized/mixed calculations of the α-centroids. Section 5 presents the soft α-clustering with respect to α-mixed divergences. Finally, Section 6 summarises the contributions, discusses related topics and hints at further perspectives. The paper is followed by two appendices. Appendix B studies several properties of α-divergences that are used to derive the guaranteed probabilistic performance of the α-seeding. Appendix C proves that α-sided centroids are quasi-arithmetic means for the power generator functions.

2. Mixed Centroid-Based k-Means Clustering

2.1. Divergences, Centroids and k-Means

Consider a set $\mathrm{\mathscr{H}}$ of n histograms h₁,…, h_n, each with d bins, with all positive real-valued bins: $h_j^i>0$, ∀1 ≤ i ≤ d, 1 ≤ j ≤ n. A histogram h is called a frequency histogram when its bins sums up to one: $w(h)=w_h={\displaystyle {\sum }_i{h}^i=1}$. Otherwise, it is called a positive histogram that can eventually be normalized to a frequency histogram:

$$\tilde{h}\doteq \frac{h}{w(h)}.$$

(1)

The frequency histograms belong to the (d-1)-dimensional open probability simplex Δ_d:

$${\mathrm{\Delta }}_d\doteq \left\{(x^1,\dots ,x^d)\in {ℝ}^d|\forall i,x^i>0,\text{and}{\displaystyle \sum _{i=1}^d{x}^i=1}\right\}.$$

(2)

That is, although frequency histograms have d bins, the constraint that those bin values should sum up to one yields d-1 degrees of freedom. In probability theory, the frequency or counting of histograms either model discrete multinomial probabilities or discrete positive measures (also called positive arrays [19]).

The celebrated k-means clustering [8,9] is one of the most famous clustering techniques that has been generalized in many ways [20,21]. In information geometry [22], a divergence D(p : q) is a smooth C³ differentiable dissimilarity measure that is not necessarily symmetric (D(p : q) ≠ D(q : p), hence the notation “:” instead of the classical “,” reserved for metric distances), but is non-negative and satisfies the separability property: D(p : q) = 0 iff p = q. More precisely, let ${\partial }_{i}D(x:y)=\frac{\partial }{\partial x^i}D(x:y)$, ${\partial }_{,i}D(x:y)=\frac{\partial }{\partial y^i}D(x:y)$. Then, we require ∂_iD(x : x) = ∂_,iD(x : x) = 0 and −∂_i∂_,jD(x : y) positive definite for defining a divergence. For a distance function D(· : ·), we denote by $D\left(x:\mathrm{\mathscr{H}}\right)$ the weighted average distance of x to a set a weighted histograms:

$$D(x:\mathrm{\mathscr{H}})\doteq {\displaystyle \sum _{j=1}^n{w}_{i}D(x:h_j)}.$$

(3)

An important class of divergences on frequency histograms is the f-divergences [23–25] defined for a convex generator f (with f′(1) = f′(1) = 0 and f″(1) = 1):

$$I_f(p:q)\doteq {\displaystyle \sum _{i=1}^d{q}^{i}f\left(\frac{p^i}{q^i}\right).}$$

Those divergences preserve information monotonicity [19] under any arbitrary transition probability (Markov morphisms). f-divergences can be extended to positive arrays [19].

The k-means algorithm on a set of weighted histograms can be tailored to any divergence as follows: First, we initialize the k cluster centers C = {c₁,…, c_k} (say, by picking up randomly arbitrary distinct seeds). Then, we iteratively repeat until convergence the following two steps:

• Assignment: Assign each histogram h_j to its closest cluster center:

  $$l(h_j)\doteq \arg \underset{l=1}{\overset{k}{min}}D(h_j:c_l).$$
  This yields a partition of the histogram set $\mathrm{\mathscr{H}}={\cup }_{l=1}^k{\mathcal{A}}_l$, where A_l denotes the set of histograms of the l-th cluster: ${\mathcal{A}}_l=\left\{h_j\right|l(h_j)=l\}$.

• Center relocation: Update the cluster centers by taking their centroids:

  $$c_l\doteq \arg \underset{x}{min}{\displaystyle \sum _{h_j\in {\mathcal{A}}_l}{w}_{j}D(h_j:x).}$$
  Throughout this paper, centroid shall be understood in the broader sense of a barycenter when weights are non-uniform.

2.2. Mixed Divergences and Mixed k-Means Clustering

Since divergences are potentially asymmetric, we can define two-sided k-means or always consider a right-sided k-means, but then define another sided divergence D′(p : q) = D(q : p). We can also consider the symmetrized k-means with respect to the symmetrized divergence: S(p, q) = D(p : q) + D(q : p). Eventually, we may skew the symmetrization with a parameter λ ∈ [0, 1]: S_λ(p, q) = λD(p : q) + (1 − λ)D(q : p) (and consider other averaging schemes instead of the arithmetic mean).

In order to handle those sided and symmetrized k-means under the same framework, let us introduce the notion of mixed divergences [15] as follows:

Definition 1 (Mixed divergence).

$$M_{\mathrm{\lambda }}(p:q:r)\doteq \mathrm{\lambda }D(p:q)+(1-\mathrm{\lambda })D(q:r),$$

(4)

for λ ∈ [0, 1].

A mixed divergence includes the sided divergences for λ ∈ {0, 1} and the symmetrized (arithmetic mean) divergence for $\mathrm{\lambda }=\frac{1}{2}$.

We generalize k-means clustering to mixed k-means clustering [15] by considering two centers per cluster (for the special cases of λ = 0, 1, it is enough to consider only one).

Algorithm 1:. Mixed divergence-based k-means clustering.

sketches the generic mixed k-means algorithm. Note that a simple initialization consists of choosing randomly the k distinct seeds from the dataset with l_i = r_i.

Notice that the mixed k-means clustering is different from the k-means clustering with respect to the symmetrized divergences S_λ that considers only one centroid per cluster.

2.3. Sided, Symmetrized and Mixed α-Divergences

For α ≠ ±1, we define the family of α-divergences [26] on positive arrays [27] as:

$$\begin{array}{l}{D}_{\alpha }(p:q)\doteq {\displaystyle \sum _{i=1}^d\frac{4}{1-{\alpha }^2}\left(\frac{1-\alpha }{2}{p}^i+\frac{1+\alpha }{2}{q}^i-{(p^i)}^{\frac{1-\alpha }{2}}{(q^i)}^{\frac{1+\alpha }{2}}\right)},\\ =D_{-\alpha }(q:p),\alpha \in ℝ\backslash \{0,1\},\end{array}$$

(5)

with the limit cases D₋₁(p : q) = KL(p : q) and D₁(p : q) = KL(q : p), where KL is the extended Kullback–Leibler divergence:

$$\text{KL}(p:q)\doteq {\displaystyle \sum _{i=1}^d{p}^i\log \frac{p^i}{q^i}+q^i-p^i}.$$

(6)

Divergence D₀ is the squared Hellinger symmetric distance (scaled by a multiplicative factor of four) extended to positive arrays:

$$D_0(p:q)=2{\displaystyle \int {\left(\sqrt{p(x)}-\sqrt{q(x)}\right)}^2}\mathrm{d}x=4H^2(p,q),$$

(7)

with the Hellinger distance:

$$H(p,q)=\sqrt{\frac{1}{2}{\displaystyle \int {\left(\sqrt{p(x)}-\sqrt{q(x)}\right)}^2\mathrm{d}x}}.$$

(8)

Note that α-divergences are defined for the full range of α values: α ∈ ℝ. Observe that α-divergences of Equation (5) are homogeneous of degree one: D_α(λp : λq) = λD_α(p : q) for λ > 0.

When histograms p and q are both frequency histograms, we have:

$$\begin{array}{l}{D}_{\alpha }(\tilde{p}:\tilde{q})=\frac{4}{1-{\alpha }^2}\left(1-{{\displaystyle \sum _{i=1}^d({\tilde{p}}^i)}}^{\frac{1-\alpha }{2}}{({\tilde{q}}^i)}^{\frac{1+\alpha }{2}}\right),\\ =D_{-\alpha }(\tilde{q}:\tilde{p}),\alpha \in ℝ\backslash \{0,1\},\end{array}$$

(9)

and the extended Kullback–Leibler divergence reduces to the traditional Kullback–Leibler divergence: $\text{KL}(\tilde{p}:\tilde{q})={\displaystyle {\sum }_{i=1}^d{\tilde{p}}^i\log \frac{{\tilde{p}}^i}{{\tilde{q}}^i}}$.

The Kullback–Leibler divergence between frequency histograms $\tilde{p}$ and $\tilde{q}$ (α = ±1) is interpreted as the cross-entropy minus the Shannon entropy:

$$\text{KL}(\tilde{p}:\tilde{q})\doteq H^{\times }(\tilde{p}:\tilde{q})-H(\tilde{p}).$$

Often, $\tilde{p}$ denotes the true model (hidden by nature), and $\tilde{q}$ is the estimated model from observations. However, in information retrieval, both $\tilde{p}$ and $\tilde{q}$ play the same symmetrical role, and we prefer to deal with a symmetric divergence.

The Pearson and Neyman χ² distances are obtained for α = −3 and α = 3, respectively:

$$D_3(\tilde{p}:\tilde{q})=\frac{1}{2}{\displaystyle \sum _i\frac{{({\tilde{q}}^i-{\tilde{p}}^i)}^2}{{\tilde{p}}^i},}$$

(10)

$$D_{-3}(\tilde{p}:\tilde{q})=\frac{1}{2}{\displaystyle \sum _i\frac{{({\tilde{q}}^i-{\tilde{p}}^i)}^2}{{\tilde{q}}^i}.}$$

(11)

The α-divergences belong to the class of Csiszár f-divergences with the following generator:

$$f(t)=\{\begin{array}{ll}\frac{4}{1-{\alpha }^2}(1-t^{(1+\alpha )/2}),& \text{if}\alpha \ne \pm 1,\\ t\ln t,& \text{if}\alpha =1,\\ -\ln t,& \text{if}\alpha =-1\end{array}$$

(12)

Remark 1. Historically, the α-divergences have been introduced by Chernoff [28,29] in the context of hypothesis testing. In Bayesian binary hypothesis testing, we are asked to decide whether an observation belongs to one class or the other class, based on prior w₁ and w₂ and class-conditional probabilities p₁ and p₂. The average expected error of the best decision maximum a posteriori (MAP) rule is called the probability of error, denoted by P_e. When prior probabilities are identical$(w_1=w_2=\frac{1}{2})$, we have$P_e(p_1,p_2)=\frac{1}{2}{\displaystyle \int min(p_1(x),p_2(x))\mathrm{d}x}$. Let S(p, q) = ∫ min(p(x), q(x))dx denote the intersection similarity measure, with 0 < S ≤ 1 (generalizing the histogram intersection distance often used in computer vision [30]). S is bounded by the α-Chernoff affinity coefficient:

$$S(p,q)\le C_{\beta }(p,q)={\displaystyle \int p^{\beta }(x)q^{1-\beta }(x)\mathrm{d}x,}$$

for all β ∈ [0, 1]. We can convert the affinity coefficient 0 < C_β ≤ 1 into a divergence D_β by simply taking D_β = 1 − C_β. Since the absolute value of divergences does not matter, we can rescale appropriately the divergence. One nice rescaling is by multiplying by$\frac{1}{\beta (1-\beta )}:D_{\beta }=\frac{1}{\beta (1-\beta )}(1-C_{\beta })$. This lets coincide the parameterized divergence with the fundamental Kullback–Leibler divergence for the limit values β ∈ {0, 1}. Last, by choosing$\beta =\frac{1-\alpha }{2}$, it yields the well-known expression of the α-divergences.

Interestingly, the α-divergences can be interpreted as a generalized α-Kullback–Leibler divergence [26] with deformed logarithms.

Next, we introduce the mixed α-divergence of a histogram x to two histograms p and q as follows:

Definition 2 (Mixed α-divergence). The mixed α-divergence of a histogram x to two histograms p and q is defined by:

$$\begin{array}{l}{M}_{\mathrm{\lambda },\alpha }(p:x:q)=\mathrm{\lambda }{D}_{\alpha }(p:x)+(1-\mathrm{\lambda })D_{\alpha }(x:q),\\ =\mathrm{\lambda }{D}_{-\alpha }(x:p)+(1-\mathrm{\lambda })D_{-\alpha }(q:x),\\ =M_{1-\mathrm{\lambda },-\alpha }(q:x:p),\end{array}$$

(13)

The α-Jeffreys symmetrized divergence is obtained for $\mathrm{\lambda }=\frac{1}{2}$:

$$S_{\alpha }(p,q)=M_{\frac{1}{2},\alpha }(q:p:q)=M_{\frac{1}{2},\alpha }(p:q:p).$$

The skew symmetrized α-divergence is defined by:

$$S_{\mathrm{\lambda },\alpha }(p:q)=\mathrm{\lambda }{D}_{\alpha }(p:q)+(1-\mathrm{\lambda })D_{\alpha }(q:p).$$

2.4. Notations and Hard/Soft Clusterings

Throughout the paper, superscript index i denotes the histogram bin numbers and subscript index j the histogram numbers. Index l is used to iterate on the clusters. The left-sided, right-sided and symmetrized histogram positive and frequency α-centroids are denoted by l_α, r_α, s_α and ${\tilde{l}}_{\alpha },{\tilde{r}}_{\alpha },{\tilde{s}}_{\alpha }$, respectively.

In this paper, we investigate the following kinds of clusterings for sets of histograms:

Hard clustering. Each histogram belongs to exactly one cluster:

• k-means with respect to mixed divergences M_λ,α.

• k-means with respect to symmetrized divergences S_λ,α.

• Randomized seeding for mixed/symmetrized k-means by extending k-means++ with guaranteed probabilistic bounds for α-divergences.

Soft clustering. Each histogram belongs to all clusters according to some weight distribution: the soft mixed α-clustering.

3. Coupled k-Means++ α-Seeding

It is well-known that the Lloyd k-means clustering algorithm monotonically decreases the loss function and stops after a finite number of iterations into a local optimal. Optimizing globally the k-means loss is NP-hard [17] when d > 1 and k > 1. In practice, the performance of the k-means algorithm heavily relies on the initialization. A breakthrough was obtained by the k-means++ seeding [17], which guarantees in expectation a good starting partition. We extend this scheme to the coupled α-clustering. However, we point out that although k-means++ prove popular and are often used in practice with very good results; it has been recently pointed out that “worst case” configurations exist and even in small dimensions, on which the algorithm cannot beat significantly its expected approximability with a high probability [31]. Still, the expected approximability ratio, roughly in O(log(k)), is very good, as long as the number of clusters is not too large.

Algorithm 2:. Mixed α-seeding; $\text{MAS}\left(\mathrm{\mathscr{H}},k,\mathrm{\lambda },\alpha \right)$

provides our adaptation of k-means++ seeding [15,17]. It works for all three of our sided/symmetrized and mixed clustering settings:

• Pick λ = 1 for the left-sided centroid initialization,

• Pick λ = 0 for the right-sided centroid initialization (a left-sided initialization for −α),

• with arbitrary λ, for the λ-J_α (skew Jeffreys) centroids or mixed λ centroids. Indeed, the initialization is the same (see the MAS procedure in Algorithm 2).

Our proof follows and generalizes the proof described for the case of mixed Bregman seeding [15] (Lemma 2). In fact, our proof is more precise, as it quantifies the expected potential with respect to the optimum only, whereas in [15], the optimal potential is averaged with a dual optimal potential, which depends on the optimal centers, but may be larger than the optimum sought.

Theorem 1. Let C_λ,α denote for short the cost function related to the clustering type chosen (left-, right-, skew Jeffreys or mixed) in MASand$C_{\mathrm{\lambda },\alpha }^{opt}$ denote the optimal related clustering in k clusters, for λ ∈ [0, 1] and α ∈ (−1, 1). Then, on average, with respect to distribution (14), the initial clustering of MAS satisfies:

(15)

Here $f(\mathrm{\lambda })=max\left\{\frac{1-\mathrm{\lambda }}{\mathrm{\lambda }},\frac{\mathrm{\lambda }}{1-\mathrm{\lambda }}\right\}$, g(k) = 2(2 + log k), $z(\alpha )={\left(\frac{1+\left|\alpha \right|}{1-\left|\alpha \right|}\right)}^{\frac{8|\alpha {|}^2}{{(1-|\alpha |)}^2}}$, $h(\alpha )={max}_i{p}_i^{\left|\alpha \right|}/{min}_i{p}_i^{\left|\alpha \right|}$; the min is defined on strictly positive coordinates, and π denotes distribution of Algorithm 2.

Remark 2. The bound is particularly good when λ is close to 1/2, and in particular for the α-Jeffreys clustering, as in these cases, the only additional penalty compared to the Euclidean case [17] is h²(α), a penalty that relies on an optimal triangle inequality for α-divergences that we provide in Lemma 8 below.

Remark 3. This guaranteed initialization is particularly useful for α-Jeffreys clustering, as there is no closed form solution for the centroids (except when α = ±1, see [32]).

Algorithm 3:. Mixed α-hard clustering: $\text{MAhC}\left(\mathrm{\mathscr{H}},k,\mathrm{\lambda },\alpha \right)$

presents the general hard mixed k-means clustering, which can be adapted also to left- (λ = 1) and right- (λ = 0) α-clustering.

For skew Jeffreys centers, since the centroids are not available in closed form [32], we adopt a variational approach of k-means by updating iteratively the centroid in each cluster (thus improving the overall loss function without computing the optimal centroids that would eventually require infinitely many iterations).

4. Sided, Symmetrized and Mixed α-Centroids

The k-means clustering requires assigning data elements to their closest cluster center and then updating those cluster centers by taking their centroids. This section investigates the centroid computations for the sided, symmetrized and mixed α-divergences.

Note that the mixed α-seeding presented in Section 3 does not require computing centroids and, yet, guarantees probabilistically a good clustering partition.

Since mixed α-divergences are f-divergences, we start with the generic f-centroids.

4.1. Csiszár f-Centroids

The centroids induced by f-divergences of a set of positive measures (that relaxes the normalisation constraint) have been studied by Ben-Tal et al. [33]. Those entropic centroids are shown to be unique, since f-divergences are convex statistical distances in both arguments. Let E_f denote the energy to minimize when considering f-divergences:

$$E_f\doteq \underset{x\in \mathcal{X}}{min}{I}_f(\mathrm{\mathscr{H}}:x)={\displaystyle \sum _{j=1}^n{w}_j{I}_f(h_j:x)},$$

(16)

$$=\underset{x\in \mathcal{X}}{min}{\displaystyle \sum _{j=1}^n{w}_j}{\displaystyle \sum _{i=1}^d{p}_j^{i}f}\left(\frac{c^i}{h_j^i}\right).$$

(17)

When the domain is the open probability simplex $\mathcal{X}={\mathrm{\Delta }}_d$, we get a constrained optimisation problem to solve. We transform this constrained minimisation problem (i.e., x ∈ Δ_d) into an equivalent unconstrained minimisation problem by using the Lagrange multiplier, γ:

$$\underset{x\in {ℝ}^d}{min}{\displaystyle \sum _{j=1}^n{w}_j{I}_f(h_j:c)+\gamma \left({\displaystyle \sum _{i=1}^d{x}^i-1}\right)}.$$

(18)

Taking the derivatives according to xⁱ, we get:

$$\forall i\in \{1,\dots ,d\},{\displaystyle \sum _{j=1}^n{w}_j{f}^{\prime }\left(\frac{x^i}{h_j^i}\right)-\gamma =0.}$$

(19)

We now consider this equation for α-divergences and symmetrized α-divergences, both f-divergences.

4.2. Sided Positive and Frequency α-Centroids

The positive sided α-centroids for a set of weighted histograms were reported in [34] using the representation Bregman divergence. We summarise the results in the following theorem:

Theorem 2 (Sided positive α-centroids [34]). The left-sided l_α and right-sided r_α positive weighted α-centroid coordinates of a set of n positive histograms h₁,…, h_n are weighted α-means:

$$r_{\alpha }^i=f_{\alpha }^{-1}\left({\displaystyle \sum _{j=1}^n{w}_j{f}_{\alpha }(h_j^i)}\right),l_{\alpha }^i=r_{-\alpha }^i$$

with $f_{\alpha }(x)=\{\begin{array}{ll}{x}^{\frac{1-\alpha }{2}}& \alpha \ne \pm 1,\\ \log x& \alpha =1.\end{array}$

Furthermore, the frequency-sided α-centroids are simply the normalized-sided α-centroids.

Theorem 3 (Sided frequency α-centroids [16]). The coordinates of the sided frequency α-centroids of a set of n weighted frequency histograms are the normalised weighted α-means.

Table 1. Positive and frequency α-centroids: the frequency α-centroids are normalized positive α-centroids, where w(h) denotes the cumulative sum of the histogram bins. The arithmetic mean is obtained for r₋₁ = l₁ and the geometric mean for r₁ = l₋₁.

	Positive centroid	Frequency centroid

summarizes the results concerning the sided positive and frequency α-centroids.

4.3. Mixed α-Centroids

The mixed α-centroids for a set of n weighted histograms is defined as the minimizer of:

$${\displaystyle \sum _j{w}_j{M}_{\mathrm{\lambda },\alpha }(l:h_j:r)}.$$

(20)

We state the theorem generalizing [15]:

Theorem 4. The two mixed α-centroids are the left-sided and right-sided α-centroids.

Figure 1 depicts some clustering result with our α-clustering software. We remark that the clusters found are all approximately subclusters of the “distinct” clusters that appear on the figure. When those distinct clusters are actually the optimal clusters—which is likely to be the case when they are separated by large minimal distance to other clusters—this is clearly a desirable qualitative property as long as the number of experimental clusters is not too large compared to the number of optimal clusters. We remark also that in the experiment displayed, there is no closed form solution for the cluster centers.

4.4. Symmetrized Jeffreys-Type α-Centroids

The Kullback–Leibler divergence can be symmetrized in various ways: Jeffreys divergence, Jensen–Shannon divergence and Chernoff information, just to mention a few. Here, we consider the following symmetrization of α-divergences extending Jeffreys J-divergence:

$$\begin{array}{l}{S}_{\alpha }(p,q)=\frac{1}{2}(D_{\alpha }(p:q)+D_{\alpha }(q:p))=S_{-\alpha }(p,q),\\ =M_{\frac{1}{2}}(p:q:p),\end{array}$$

(21)

For α = ±1, we get half of Jeffreys divergence:

$$S_{\pm 1}(p,q)=\frac{1}{2}{\displaystyle \sum _{i=1}^d(p^i-q^i)\log \frac{p^i}{q^i}}$$

In particular, when p and q are frequency histograms, we have for α ≠ ±1:

$$J_{\alpha }(\tilde{p}:\tilde{q})=\frac{8}{1-{\alpha }^2}\left(1+{\displaystyle \sum _{i=14}^d{H}_{\frac{1-\alpha }{2}}}({\tilde{p}}^i:{\tilde{q}}^i)\right),$$

(22)

where $H_{\frac{1-\alpha }{2}}(a,b)$ a symmetric Heinz mean [35,36]:

$$H_{\beta }(a,b)=\frac{a^{\beta }{b}^{1-\beta }+a^{1-\beta }{b}^{\beta }}{2}.$$

Heinz means interpolate the arithmetic and geometric means and satisfies the inequality:

$$\sqrt{ab}=H_{\frac{1}{2}}(a,b)\le H_{\alpha }(a,b)\le H_0(a,b)=\frac{a+b}{2}.$$

(Another interesting property of Heinz means is the integral representation of the logarithmic mean: $L(x,y)=\frac{x-y}{\log x-\log y}={\displaystyle {\int }_0^1{H}_{\beta }(x,y)\mathrm{d}\beta }$. This allows one to prove easily that $\sqrt{xy}\le L(x,y)\le \frac{x+y}{2}$.)

The J_α-divergence is a Csiszár f-divergence [24,25].

Observe that it is enough to consider α ∈ [0, ∞) and that the symmetrized α-divergence for positive and frequency histograms coincide only for α = ±1.

For α = ±1, S_α(p, q) tends to the Jeffreys divergence:

$$J(p,q)=\text{KL}(p,q)+\text{KL}(q,p)={\displaystyle \sum _{i=1}^d(p^i-q^i)(\log p^i-\log q^i).}$$

(23)

The Jeffreys divergence writes mathematically the same for frequency histograms:

$$J(\tilde{p},\tilde{q})=\text{KL}(\tilde{p},\tilde{q})+\text{KL}(\tilde{p},\tilde{q})={\displaystyle \sum _{i=1}^d({\tilde{p}}^i-{\tilde{q}}^i)(\log {\tilde{p}}^i-\log {\tilde{q}}^i)}.$$

(24)

We state the results reported in [32]:

Theorem 5 (Jeffreys positive centroid [32]). The Jeffreys positive centroid c = (c¹,…, c^d) of a set {h₁,…, h_n} of n weighted positive histograms with d bins can be calculated component-wise exactly using the Lambert W analytic function:

$$c^i=\frac{a^i}{W(\frac{a^i}{g^i}e)},$$

where$a^i={\displaystyle {\sum }_{j=1}^n{\pi }_j{h}_j^i}$ denotes the coordinate-wise arithmetic weighted means and$g^i={\displaystyle {\prod }_{j=1}^n{(h_j^i)}^{{\pi }_j}}$ the coordinate-wise geometric weighted means.

The Lambert analytic function W [37] (positive branch) is defined by W (x)e^W(x) = x for x ≥ 0.

Theorem 6 (Jeffreys frequency centroid [32]). Let$\tilde{c}$ denote the Jeffreys frequency centroid and${\tilde{c}}^{\prime }=\frac{c}{w_c}$ the normalised Jeffreys positive centroid. Then, the approximation factor${\alpha }_{{\tilde{c}}^{\prime }}=\frac{S_1({\tilde{c}}^{\prime },\widetilde{\mathrm{\mathscr{H}}})}{S_1(\tilde{c},\widetilde{\mathrm{\mathscr{H}}})}$ is such that$1\le {\alpha }_{{\tilde{c}}^{\prime }}\le \frac{1}{w_c}$ (with w_c ≤ 1)

Therefore, we shall consider α ≠ ±1 in the remainder.

We state the following lemma generalizing the former results in [38] that were tailored to the symmetrized Kullback–Leibler divergence or the symmetrized Bregman divergence [14]:

Lemma 1 (Reduction property). The symmetrized J_α-centroid of a set of n weighted histograms amount to computing the symmetrized α-centroid for the weighted α-mean and −α-mean:

$$min{J}_{\alpha }(x,\mathrm{\mathscr{H}})=\underset{x}{min}(D_{\alpha }(x:r_{\alpha })+D_{\alpha }(l_{\alpha }:x)).$$

Proof. It follows that the minimization problem ${min}_x{S}_{\alpha }(x,\mathrm{\mathscr{H}})={\displaystyle {\sum }_{j=1}^n{w}_j{S}_{\alpha }(x,h_j)}$ reduces to the following minimization:

$$min{\displaystyle \sum _{i=1}^d{x}^i-{(x^i)}^{\frac{1+\alpha }{2}}{\overline{h}}_{\alpha }^i}-{(x^i)}^{\frac{1-\alpha }{2}}{\overline{h}}_{-\alpha }^i.$$

(25)

This is equivalent to minimizing:

Note that α = ±1, the lemma states that the minimization problem is equivalent to minimizing KL(a : x) + KL(x : g) with respect to x, where a = l₁ and g = r₁ denote the arithmetic and geometric means, respectively. □

The lemma states that the optimization problem with n weighted histograms is equivalent to the optimization with only two equally weighted histograms.

The positive symmetrized α-centroid is equivalent to computing a representation symmetrized Bregman centroid [14,34].

The frequency symmetrized α-centroid asks to minimize the following problem:

$$\underset{\tilde{x}\in \mathrm{\Delta }d}{min}{\displaystyle \sum _j{w}_j{S}_{\alpha }(\tilde{x},{\tilde{h}}_i).}$$

Instead of seeking for $\tilde{x}$ in the probability simplex, we can optimize on the unconstrained domain ℝ^d−1 by using a reparameterization. Indeed, frequency histograms belong to the exponential families [39] (multinomials).

Exponential families also include many other continuous distributions, like the Gaussian, Beta or Dirichlet distributions. It turns out the α-divergences can be computed in closed-form for members of the same exponential family:

Lemma 2. The α-divergence for distributions belonging to the same exponential families amounts to computing a divergence on the corresponding natural parameters:

$$A_{\alpha }(p:q)=\frac{4}{1-{\alpha }^2}\left(1-e^{-J_F^{\left(\frac{1-\alpha }{2}\right)}}{}^{({\theta }_p:{\theta }_q)}\right),$$

where$J_F^{\beta }({\theta }_1:{\theta }_2)=\beta F({\theta }_1)+(1-\beta )F({\theta }_2)-F(\beta {\theta }_1+(1-\beta ){\theta }_2)$ is a skewed Jensen divergence defined for the log-normaliser F of the family.

The proof follows from the fact that ${\displaystyle \int p^{\alpha }(x)q^{1-\alpha }(x)\mathrm{d}x=e-J_F^{(\alpha )({\theta }_p:{\theta }_q)}}$; see [40].

First, we convert a frequency histogram $\tilde{h}$ to its natural parameter θ with ${\theta }^i=\log \frac{{\tilde{h}}^i}{{\tilde{h}}^d}$; see [39].

The log-normaliser is a non-separable convex function $F(\theta )=\log (1+{\displaystyle {\sum }_i{e}^{{\theta }_i}})$. To convert back a multinomial to a frequency histogram with d bins, we first set ${\tilde{h}}^d=\frac{1}{1+{\displaystyle {\sum }_{l=1}^{d-1}{e}^{{\theta }^l}}}$ and then retrieve the other bin values as ${\tilde{h}}^i={\tilde{h}}^d{e}^{{\theta }^i}$.

The centroids with respect to skewed Jensen divergences has been investigated in [13,40].

Remark 4. Note that for the special case of α = 0 (squared Hellinger centroid), the sided and symmetrized centroids coincide. In that case, the coordinates$s_0^i$ of the squared Hellinger centroid are:

$$s_0^i={\left({\displaystyle \sum _{j=1}^n{w}_j}\sqrt{h_j^i}\right)}^2,1\le i\le d.$$

Remark 5. The symmetrized positive α-centroids can be solved in special cases (α = ±3, α = ±1 corresponding to the symmetrized χ² and Jeffreys positive centroids). For frequency centroids, when dealing with binary histograms (d = 2), we have only one degree of freedom and can solve the binary frequency centroids. Binary histograms (and mixtures thereof) are used in computer vision and pattern recognition [41].

Remark 6. Since α-divergences are Csiszár f-divergences and f-divergences can always be symmetrized by taking generator$s(t)=f(t)+tf\left(\frac{1}{t}\right)$, we deduce that symmetrized α-divergences S_α are f-divergences for the generator:

$$f(t)=-\log ((1-\alpha )+\alpha t)-t\log \left((1-\alpha )+\frac{\alpha }{t}\right).$$

(26)

Hence, Sα divergences are convex in both arguments, and the sα centroids are unique.

5. Soft Mixed α-Clustering

Algorithm 4:. Mixed α-soft clustering; $\text{MAsC}\left(\mathrm{\mathscr{H}},k,\mathrm{\lambda },\alpha \right)$

reports the general clustering with soft membership, which can be adapted to left (λ_init = 1), right (λ_init = 0) or mixed clustering. We have not considered a weighted histogram set in order not load the notations and because the extension is straightforward.

Again, for skew Jeffreys centers, we shall adopt a variational approach. Notice that the soft clustering approach learns all parameters, including λ (if not constrained to zero or one) and α ∈ ℝ. This is not the case for Matsuyama’s α-expectation maximization (EM) algorithm [42] in which α is fixed beforehand (and, thus, not learned).

Assuming we model the prior for histograms by:

$$\begin{array}{l}{p}_{\mathrm{\lambda },\alpha ,j}(h_i)\propto \\ \mathrm{\lambda }\exp -D_{\alpha }(l_j:h_i)+(1-\mathrm{\lambda })\exp -D_{\alpha }(h_i:r_i),\end{array}$$

(27)

the negative log-likelihood involves the α-depending quantity:

(28)

because of the concavity of the logarithm function. Therefore, the maximization step for α involves finding:

$$\arg \underset{\alpha }{max}{\displaystyle \sum _{j=1}^k{\displaystyle \sum _{i=1}^m{M}_{\mathrm{\lambda },\alpha }(l_j:h_i:r_j)p(j|h_i)}.}$$

(29)

No closed-form solution are known, so we compute the gradient update in Algorithm 4 with:

$$\begin{array}{l}\frac{\partial M_{\mathrm{\lambda },\alpha }(l_j:h_i:r_j)}{\partial \alpha }=\\ \mathrm{\lambda }\frac{\partial D_{\alpha }(l_j:h_i)}{\partial \alpha }+(1-\mathrm{\lambda })\frac{\partial D_{\alpha }(h_i:r_i)}{\partial \alpha },\end{array}$$

(30)

(31)

The update in λ is easier as:

$$\frac{\partial M_{\mathrm{\lambda },\alpha }(l_j:h_i:r_j)}{\partial \mathrm{\lambda }}=D_{\alpha }(l_j:h_i)-D_{\alpha }(h_i:r_j).$$

(32)

Maximizing the likelihood in λ would imply choosing λ ∈ {0, 1} (a hard choice for left/right centers), yet we prefer the soft update for the parameter, like for α.

6. Conclusions

The family of α-divergences plays a fundamental role in information geometry: These statistical distortion measures are the canonical divergences of dual spaces on probability distribution manifolds with constant curvature $\kappa =\frac{1-{\alpha }^2}{4}$ and the canonical divergences of dually flat manifolds for positive distribution manifolds [19].

In this work, we have presented three techniques for clustering (positive or frequency) histograms using k-means:

(1)

Sided left or right α-centroid k-means,

(2)

Symmetrized Jeffreys-type α-centroid (variational) k-means, and

(3)

Coupled k-means with respect to mixed α-divergences relying on dual α-centroids.

Sided and mixed dual centroids are always available in closed-forms and are therefore highly attractive from the standpoint of implementation. Symmetrized Jeffreys centroids are in general not available in closed-form and require one to implement a variational k-means by updating incrementally the cluster centroids in order to monotonically decrease the k-means loss function. From the clustering standpoint, this appears not to be a problem when guaranteed expected approximations to the optimal clustering are enough.

Indeed, we also presented and analyzed an extension of k-means++ [17] for seeding those k-means algorithms. The mixed α-seeding initializations do not require one to calculate centroids and behaves like a discrete k-means by picking up the seeds among the data. We reported guaranteed probabilistic clustering bounds. Thus, it yields a fast hard/soft data partitioning technique with respect to mixed or symmetrized α-divergences. Recently, the advantage of clustering using α-divergences by tuning α in applications has been demonstrated in [18]. We thus expect the computationally fast mixed α-seeding with guaranteed performance to be useful in a growing number of applications.