Information Geometry of Positive Measures and Positive-Definite Matrices: Decomposable Dually Flat Structure

Abstract

Information geometry studies the dually flat structure of a manifold, highlighted by the generalized Pythagorean theorem. The present paper studies a class of Bregman divergences called the (ρ, τ )-divergence. A (ρ, τ )-divergence generates a dually flat structure in the manifold of positive measures, as well as in the manifold of positive-definite matrices. The class is composed of decomposable divergences, which are written as a sum of componentwise divergences. Conversely, a decomposable dually flat divergence is shown to be a (ρ, τ )-divergence. A (ρ, τ )-divergence is determined from two monotone scalar functions, ρ and τ. The class includes the KL-divergence, α-, β- and (α, β)-divergences as special cases. The transformation between an affine parameter and its dual is easily calculated in the case of a decomposable divergence. Therefore, such a divergence is useful for obtaining the center for a cluster of points, which will be applied to classification and information retrieval in vision. For the manifold of positive-definite matrices, in addition to the dually flatness and decomposability, we require the invariance under linear transformations, in particular under orthogonal transformations. This opens a way to define a new class of divergences, called the (ρ, τ )-structure in the manifold of positive-definite matrices.

1. Introduction

Information geometry, originated from the invariant structure of a manifold of probability distributions, consists of a Riemannian metric and dually coupled affine connections with respect to the metric [1]. A manifold having a dually flat structure is particularly interesting and important. In such a manifold, there are two dually coupled affine coordinate systems and a canonical divergence, which is a Bregman divergence. The highlight is given by the generalized Pythagorean theorem and projection theorem. Information geometry is useful not only for statistical inference, but also for machine learning, pattern recognition, optimization and even for neural networks. It is also related to the statistical physics of Tsallis q-entropy [2–4].

The present paper studies a general and unique class of decomposable divergence functions in ${\mathit{R}}_{+}^n$, the manifold of n-dimensional positive measures. This is the (ρ, τ )-divergences, introduced by Zhang [5,6], from the point of view of “representation duality”. They are Bregman divergences generating a dually flat structure. The class includes the well-known Kullback-Leibler divergence, α-divergence, β-divergence and (α, β)-divergence [1,7–9] as special cases. The merit of a decomposable Bregman divergence is that the θ-η Legendre transformation is computationally tractable, where θ and η are two affine coordinates systems coupled by the Legendre transformation. When one uses a dually flat divergence to define the center of a cluster of elements, the center is easily given by the arithmetic mean of the dual coordinates of the elements [10,11]. However, we need to calculate its primal coordinates. This is the θ-η transformation. Hence, our new type of divergences has an advantage of calculating θ-coordinates for clustering and related pattern matching problems. The most general class of dually flat divergences, not necessarily decomposable, is further given in ${\mathit{R}}_{+}^n$. They are the (ρ, τ ) divergence.

Positive-definite (PD) matrices appear in many engineering problems, such as convex programming, diffusion tensor analysis and multivariate statistical analysis [12–20]. The manifold, PD_n, of n × n PD matrices form a cone, and its geometry is by itself an important subject of research. If we consider the submanifold consisting of only diagonal matrices, it is equivalent to the manifold of positive measures. Hence, PD matrices can be regarded as a generalization of positive measures. There are many studies on geometry and divergences of the manifold of positive-definite matrices. We introduce a general class of dually flat divergences, the (ρ, τ )-divergence. We analyze the cases when a (ρ, τ )-divergence is invariant under the general linear transformations, Gl(n), and invariant under the orthogonal transformations, O(n). They not only include many well-known divergences of PD matrices, but also give new important divergences.

The present paper is organized as follows. Section 2 is preliminary, giving a short introduction to a dually flat manifold and the Bregman divergence. It also defines the cluster center due to a divergence. Section 3 defines the (ρ, τ )-structure in ${\mathit{R}}_{+}^n$. It gives dually flat decomposable affine coordinates and a related canonical divergence (Bregman divergence). Section 4 is devoted to the (ρ, τ )-structure of the manifold, PD_n, of PD matrices. We first study the class of divergences that are invariant under O(n). We further study a decomposable divergence that is invariant under Gl(n). It coincides with the invariant divergence derived from zero-mean Gaussian distributions with PD covariance matrices. They not only include various known divergences, but new remarkable ones. Section 5 discusses a general class of non-decomposable flat divergences and miscellaneous topics. Section 6 is the conclusions.

2. Preliminaries to Information Geometry of Divergence

2.1. Dually Flat Manifold

A manifold is said to have the dually flat Riemannian structure, when it has two affine coordinate systems θ = (θ¹, · · · , θⁿ) and η = (η₁, · · · , η_n) (with respect to two flat affine connections) together with two convex functions, ψ(θ) and ϕ(η), such that the two coordinates are connected by the Legendre transformations:

$$\mathit{\eta }=\nabla \psi (\mathit{\theta }),\mathrm{\hspace{0.17em} }\mathrm{\hspace{0.17em} }\mathrm{\hspace{0.17em} }\mathit{\theta }=\nabla \varphi (\mathit{\eta }),$$

(1)

where ∇ is the gradient operator. The Riemannian metric is given by:

$$(g_{ij}(\mathit{\theta }))=\nabla \nabla \psi (\mathit{\theta }),\mathrm{\hspace{0.17em} }\mathrm{\hspace{0.17em} }\mathrm{\hspace{0.17em} }\left(g^{ij}(\mathit{\eta })\right)=\nabla \nabla \varphi (\mathit{\eta })$$

(2)

in the respective coordinate systems. A curve that is linear in the θ-coordinates is called a θ-geodesic, and a curve linear in the η-coordinates is called an η-geodesic.

A dually flat manifold has a unique canonical divergence, which is the Bregman divergence defined by the convex functions,

$$D[P:Q]=\psi ({\mathit{\theta }}_P)+\varphi ({\mathit{\eta }}_Q)-{\mathit{\theta }}_P·{\mathit{\eta }}_Q,$$

(3)

where θ_P is the θ-coordinates of P, η_Q is the η-coordinates of Q and ${\mathit{\theta }}_P·{\eta }_Q={\sum }_i({\theta }_P^i)\hspace{0.17em}({\eta }_{Qi})$, where ${\theta }_P^i$ and η_Qi are components of θ_p and η_Q, respectively. The Pythagorean and projection theorems hold in a dually flat manifold:

Pythagorean Theorem

Given three points, P,Q,R, when the η-geodesic connecting P and Q is orthogonal to the θ-geodesic connecting Q and R with respect to the Riemannian metric,

$$D[P:Q]+D[Q:R]=D[P:R].$$

(4)

Projection Theorem

Given a smooth submanifold, S, let P_S be the minimizer of divergence from

$$P_S=\underset{Q\in S}{\text{min}}D[P:Q]$$

(5)

Then, P_S is the η-geodesic projection of P to S, that is the η-geodesic connecting P and P_S is orthogonal to S.

We have the dual of the above theorems where θ- and η-geodesics are exchanged and D [P : Q] is replaced by its dual D [Q : P].

2.2. Decomposable Divergence

A divergence, D [P : Q], is said to be decomposable, when it is written as a sum of component-wise divergences,

$$D[P:Q]=\sum _{i=1}^{n}d\left({\theta }_P^i,{\theta }_Q^i\right),$$

(6)

where ${\theta }_P^i$ and ${\theta }_Q^i$ are the components of θ_P and θ_Q and $d\hspace{0.17em}\left({\theta }_P^i,{\theta }_Q^i\right)$ is a scalar divergence function.

An f-divergence:

$$D_f[P:Q]=\sum p_{i}f\hspace{0.17em}\left(\frac{q_i}{p_i}\right)$$

(7)

is a typical example of decomposable divergence in the manifold of probability distributions, where P = (p) and Q = (q) are two probability vectors with ∑p_i = ∑q_i = 1. A convex function, ψ(θ), is said to be decomposable, when it is written as:

$$\psi (\mathit{\theta })=\sum _{i=1}^n\tilde{\psi }\hspace{0.17em}({\theta }^i)$$

(8)

by using a scalar convex function, ψ̃(θ). The Bregman divergence derived from a decomposable convex function is decomposable.

When ψ(θ) is a decomposable convex function, its Legendre dual is also decomposable. The Legendre transformation is given componentwise as:

$${\eta }_i={\tilde{\psi }}^{\prime }\hspace{0.17em}({\theta }_i),$$

(9)

where ′ is the differentiation of a function, so that it is computationally tractable. Its inverse transformation is also componentwise,

$${\theta }_i={\tilde{\varphi }}^{\prime }\hspace{0.17em}({\eta }_i),$$

(10)

where ϕ̃ is the Legendre dual of ψ̃.

2.3. Cluster Center

Consider a cluster of points P₁, · · · , P_m of which θ-coordinates are θ₁, · · · , θ_m and η-coordinates are η₁, · · · , η_m. The center, R, of the cluster with respect to the divergence, D [P : Q], is defined by:

$$R=\underset{Q}{\text{arg\hspace{0.17em}min}}\sum _{i=1}^{m}D[Q:P_i].$$

(11)

By differentiating ∑D [Q : P_i] by θ (the θ-coordinates of Q), we have:

$$\nabla \psi ({\mathit{\theta }}_R)=\frac{1}{m}\sum _{i=1}^m{\mathit{\eta }}_i.$$

(12)

Hence, the cluster-center theorem due to Banerjee et al. [10] follows; see also [11]:

Cluster-Center Theorem

The η-coordinates η_R of the cluster center are given by the arithmetic average of the η-coordinates of the points in the cluster:

$${\mathit{\eta }}_R=\frac{1}{m}\sum _{i=1}^m{\mathit{\eta }}_i.$$

(13)

When we need to obtain the θ-coordinates of the cluster center, it is given by the θ-η transformation from η_R,

$${\mathit{\theta }}_R=\nabla \varphi \hspace{0.17em}({\mathit{\eta }}_R).$$

(14)

However, in many cases, the transformation is computationally heavy and intractable when the dimensions of a manifold is large. The transformation is easy in the case of a decomposable divergence. This is motivation for considering a general class of decomposable Bregman divergences.

3. (ρ, τ ) Dually Flat Structure in ${\mathit{R}}_{+}^n$

3.1. (ρ, τ )-Coordinates of ${\mathit{R}}_{+}^n$

Let ${\mathit{R}}_{+}^n$ be the manifold of positive measures over n elements x₁, · · · , x_n. A measure (or a weight) of x_i is given by:

$${\xi }_i=m\hspace{0.17em}(x_i)>0$$

(15)

and ξ = (ξ₁, · · · , ξ_n) is a distribution of measures. When ∑ξ_i = 1 is satisfied, it is a probability measure. We write:

$${\mathit{R}}_n^{+}=\{\mathit{\xi }\mid {\xi }_i>0\}$$

(16)

and ξ forms a coordinate system of ${\mathit{R}}_{+}^n$.

Let ρ(ξ) and τ (ξ) be two monotonically increasing differentiable functions. We call:

$$\theta =\rho (\xi ),\mathrm{\hspace{0.17em} }\mathrm{\hspace{0.17em} }\mathrm{\hspace{0.17em} }\eta =\tau (\xi )$$

(17)

the ρ- and τ -representations of positive measure ξ. This is a generalization of the ±α representations [1] and was introduced in [5] for a manifold of probability distributions. See also [6].

By using these functions, we construct new coordinate systems θ and η of ${\mathit{R}}_{+}^n$. They are given, for θ = (θⁱ) and η = (η_i), by componentwise relations,

$${\theta }^i=\rho \hspace{0.17em}({\xi }_i),\mathrm{\hspace{0.17em} }\mathrm{\hspace{0.17em} }\mathrm{\hspace{0.17em} }{\eta }_i=\tau \hspace{0.17em}({\xi }_i).$$

(18)

They are called the ρ- and τ -representations of ξ ∈ $\mathit{\xi }\in {\mathit{R}}_{+}^n$, respectively. We search for convex functions, ψ_ρ,τ (θ) and ϕ_ρ,τ (η), which are Legendre duals to each other, such that θ and η are two dually coupled affine coordinate systems.

3.2. Convex Functions

We introduce two scalar functions of θ and η by:

$${\tilde{\psi }}_{\rho ,\tau }(\theta )={\int }_0^{{\rho }^{-1}(\theta )}\tau (\xi ){\rho }^{\prime }(\xi )d\xi ,$$

(19)

$${\tilde{\varphi }}_{\rho ,\tau }(\eta )={\int }_0^{{\tau }^{-1}(\eta )}\rho (\xi ){\tau }^{\prime }(\xi )d\xi .$$

(20)

Then, the first and second derivatives of ψ̃_ρ,τ are:

$${\tilde{\psi }}_{\rho ,\tau }^{\prime }(\theta )=\tau (\xi ),$$

(21)

$${\tilde{\psi }}_{\rho ,\tau }^{\prime \prime }(\theta )=\frac{{\tau }^{\prime }(\xi )}{{\rho }^{\prime }(\xi )}.$$

(22)

Since ρ′(ξ) > 0, τ′ (ξ) > 0, we see that ψ̃_ρ,τ (θ) is a convex function. So is ϕ̃_ρ,τ (η). Moreover, they are Legendre duals, because:

$${\tilde{\psi }}_{\rho ,\tau }(\theta )+{\tilde{\varphi }}_{\rho ,\tau }(\eta )-\theta \eta ={\int }_0^{\xi }\tau (\xi ){\rho }^{\prime }(\xi )d\xi +{\int }_0^{\xi }\rho (\xi ){\tau }^{\prime }(\xi )d\xi -\rho (\xi )\tau (\xi )$$

(23)

$$=0.$$

(24)

We then define two decomposable convex functions of θ and η by:

$${\psi }_{\rho ,\tau }(\mathit{\theta })=\sum {\tilde{\psi }}_{\rho ,\tau }\hspace{0.17em}\left({\theta }^i\right),$$

(25)

$${\varphi }_{\rho ,\tau }(\mathit{\eta })=\sum {\tilde{\varphi }}_{\rho ,\tau }\hspace{0.17em}\left({\eta }_i\right).$$

(26)

They are Legendre duals to each other.

3.3. (ρ, τ )-Divergence

The (ρ, τ )-divergence between two points, ξ, ${\mathit{\xi }}^{\prime }\in {\mathit{R}}_n^{+}$, is defined by:

$$D_{\rho ,\tau }[\mathit{\xi }:{\mathit{\xi }}^{\prime }]={\psi }_{\rho ,\tau }(\mathit{\theta })+{\varphi }_{\rho ,\tau }({\mathit{\eta }}^{\prime })-\mathit{\theta }·{\mathit{\eta }}^{\prime }$$

(27)

$$=\sum _{i=1}^n\left[{\int }_0^{{\xi }_i}\tau (\xi ){\rho }^{\prime }(\xi )d\xi +{\int }_0^{{\xi }_i}\rho (\xi ){\tau }^{\prime }(\xi )d\xi -\rho ({\xi }_i)\tau ({\xi }_i^{\prime })\right],$$

(28)

where θ and η′ are ρ- and τ -representations of ξ and ξ′, respectively.

The (ρ, τ )-divergence gives a dually flat structure having θ and η as affine and dual affine coordinate systems. This is originally due to Zhang [5] and a generalization of our previous results concerning the q and deformed exponential families [4]. The transformation between θ and η is simple in the (ρ, τ )-structure, because it can be done componentwise,

$${\theta }^i=\rho \{{\tau }^{-1}({\eta }_i)\},$$

(29)

$${\eta }_i=\tau \{{\rho }^{-1}({\theta }^i)\}.$$

(30)

The Riemannian metric is:

$$g_{ij}(\mathit{\xi })=\frac{{\tau }^{\prime }({\xi }_i)}{{\rho }^{\prime }({\xi }_i)}{\delta }_{ij},$$

(31)

and hence Euclidean, because the Riemann-Christoffel curvature due to the Levi-Civita connection vanishes, too.

The following theorem is new, characterizing the (ρ, τ )-divergence.

Theorem 1

The (ρ, τ )-divergences form a unique class of divergences in ${\mathit{R}}_{+}^n$ that are dually flat and decomposable.

3.4. Biduality: α-(ρ, τ ) Divergence

We have dually flat connections, (∇_ρ,τ, ${\nabla }_{\rho ,\tau }^{*}$), represented in terms of covariant derivatives, which are derived from D_ρ,τ. This is called the representation duality by Zhang [5]. We further have the α-(ρ, τ ) connections defined by:

$${\nabla }_{\rho ,\tau }^{(\alpha )}=\frac{1+\alpha }{2}{\nabla }_{\rho ,\tau }+\frac{1-\alpha }{2}{\nabla }_{\rho ,\tau }^{*}.$$

(32)

The α-(−α) duality is called the reference duality [5]. Therefore, ${\nabla }_{\rho ,\tau }^{(\alpha )}$ possesses the biduality, one concerning α and (−α), and the other with respect to ρ and τ.

The Riemann-Christoffel curvature of ${\nabla }_{\rho ,\tau }^{(\alpha )}$ is:

$$R_{\rho ,\tau }^{(\alpha )}=\frac{1-{\alpha }^2}{4}{R}_{\rho ,\tau }^{(0)}=0$$

(33)

for any α. Hence, there exists unique canonical divergence $D_{\rho ,\tau }^{(\alpha )}$ and α-(ρ, τ ) affine coordinate systems. It is an interesting future problem to obtain their explicit forms.

3.5. Various Examples

As a special case of the (ρ, τ )-divergence, we have the (α, β)-divergence obtained from the following power functions,

$$\rho (\xi )=\frac{1}{\alpha }{\xi }^{\alpha },\hspace{0.17em}\tau (\xi )=\frac{1}{\beta }{\xi }^{\beta }.$$

(34)

This was introduced by Cichocki, Cruse and Amari in [7,8].

The affine and dual affine coordinates are:

$${\theta }^i=\frac{1}{\alpha }{({\xi }_i)}^{\alpha },\mathrm{\hspace{0.17em} }\mathrm{\hspace{0.17em} }\mathrm{\hspace{0.17em} }{\eta }_i=\frac{1}{\beta }{({\xi }_i)}^{\beta }$$

(35)

and the convex functions are:

$$\psi (\mathit{\theta })=c_{\alpha ,\beta }\sum {\theta }_i^{\frac{\alpha +\beta }{\alpha }},\mathrm{\hspace{0.17em} }\mathrm{\hspace{0.17em} }\mathrm{\hspace{0.17em} }\varphi (\mathit{\eta })=c_{\beta ,\alpha }\sum {\eta }_i^{\frac{\alpha +\beta }{\beta }},$$

(36)

where:

$$c_{\alpha ,\beta }=\frac{1}{\beta (\alpha +\beta )}{\alpha }^{\frac{\alpha +\beta }{\alpha }}.$$

(37)

The induced (α, β)-divergence has a simple form,

$$D_{\alpha ,\beta }[\mathit{\xi }:{\mathit{\xi }}^{\prime }]=\frac{1}{\alpha \beta (\alpha +\beta )}\sum \{\alpha {\xi }_i^{\alpha +\beta }\beta {\xi }_i^{\prime \alpha +\beta }-(\alpha +\beta ){\xi }_i^{\alpha }{\xi }_i^{\prime \beta }\},$$

(38)

for ξ, ${\mathit{\xi }}^{\prime }\in {\mathit{R}}_n^{+}$. It is defined similarly in the manifold, S_n, of probability distributions, but it is not a Bregman divergence in S_n. This is because the total mass constraint ∑ξ_i = 1 is not linear in θ- or η-coordinates in general.

The α-divergence is a special case of the (α, β)-divergence, so that it is a (ρ, τ )-divergence. By putting:

$$\rho (\xi )=\frac{2}{1-\alpha }{\xi }^{\frac{1-\alpha }{2}},\mathrm{\hspace{0.17em} }\mathrm{\hspace{0.17em} }\mathrm{\hspace{0.17em} }\tau (\xi )=\frac{2}{1+\alpha }{\xi }^{\frac{1+\alpha }{2}},$$

(39)

we have:

$$D_{\alpha }[\mathit{\xi }:{\mathit{\xi }}^{\prime }]=\frac{4}{1-{\alpha }^2}\sum \left\{\frac{1-\alpha }{2}{\xi }_i+\frac{1+\alpha }{2}{\xi }_i^{\frac{1-\alpha }{2}}-{\xi }_i^{\alpha }{({\xi }_i^{\prime })}^{\frac{1+\alpha }{2}}\right\}.$$

(40)

The β-divergence [19] is obtained from:

$$\rho (\xi )=\xi ,\mathrm{\hspace{0.17em} }\mathrm{\hspace{0.17em} }\mathrm{\hspace{0.17em} }\tau (\xi )=\frac{1}{\beta }{\xi }^{1+\beta }.$$

(41)

It is written as:

$$D_{\beta }[\mathit{\xi }:{\mathit{\xi }}^{\prime }]=\frac{1}{\beta (\beta +1)}\sum _i\left[{\xi }_i^{\beta +1}+(\beta +1){\xi }_i^{\prime }-{({\xi }_i^{\prime })}^{\beta +1}-(\beta +1){\xi }_i{({\xi }_i^{\prime })}^{\beta }\right].$$

(42)

The β-divergence is special in the sense that it gives a dually flat structure, even in S_n. This is because u(ξ) is linear in ξ.

The classes of α-divergences and β-divergences intersect at the KL-divergence, and their duals are different in general. They are the only intersecting points of the two classes.

When ρ(ξ) = ξ and τ (ξ) = U′ (ξ) where U is a convex function, (ρ, τ )-divergence is Eguchi’s U-divergence [21].

Zhang already introduced the (α, β)-divergence in [5], which is not a (ρ, τ )-divergence in ${\mathit{R}}_{+}^n$ and different from ours. We regret for our confusing the naming of the (α, β)-divergence.

4. Invariant, Flat Decomposable Divergences in the Manifold of Positive-Definite Matrices

4.1. Invariant and Decomposable Convex Function

Let P be a positive-definite matrix and ψ(P) be a convex function. Then, a Bregman divergence is defined between two positive definite matrices, P and Q, by:

$$D[\mathbf{P}:\mathbf{Q}]=\psi (\mathbf{P})-\psi (\mathbf{Q})-\nabla \psi (\mathbf{P})·(\mathbf{P}-\mathbf{Q})$$

(43)

where ∇ is the gradient operator with respect to matrix P = (P_ij), so that ∇ψ(P) is a matrix and the inner product of two matrices is defined by:

$$\nabla \psi (\mathbf{Q})·\mathbf{P}=\text{tr}\{\nabla \psi (\mathbf{Q})\mathbf{P}\},$$

(44)

where tr is the trace of a matrix.

It induces a dually flat structure to the manifold of positive-definite matrices, where the affine coordinate system (θ-coordinates) is Θ = P and the dual affine coordinate system (η-coordinates) is:

$$\mathbf{H}=\nabla \psi (\mathbf{P}).$$

(45)

A convex function, ψ(P), is said to be invariant under the orthogonal group O(n), when:

$$\psi (\mathbf{P})=\psi \hspace{0.17em}({\mathbf{O}}^T\mathbf{PO})$$

(46)

holds for any orthogonal transformation O, where O^T is the transpose of O. An invariant function is written as a symmetric function of n eigenvalues λ₁, · · · , λ_n of P. See Dhillon and Tropp [12]. When an invariant convex function of P is written, by using a convex function, f, of one variable, in the additive form:

$$\psi (\mathbf{P})=\sum f\hspace{0.17em}({\lambda }_i),$$

(47)

it is said to be decomposable. We have:

$$\psi (\mathbf{P})=\text{tr}\hspace{0.17em}f(\mathbf{P}).$$

(48)

4.2. Invariant, Flat and Decomposable Divergence

A divergence D [P : Q] is said to be invariant under O(n), when it satisfies:

$$D[\mathbf{P}:\mathbf{Q}]=D\hspace{0.17em}[{\mathbf{O}}^T\mathbf{PO}:{\mathbf{O}}^T\mathbf{QO}].$$

(49)

When it is derived from a decomposable convex function, ψ(P), it is invariant, flat and decomposable.

We give well-known examples of decomposable convex functions and the divergences derived from them:

(1)

For f(λ) = (1/2)λ², we have:

$$\psi (\mathbf{P})=\frac{1}{2}\sum {\lambda }_i^2,$$

(50)

$$D[\mathbf{P}:\mathbf{Q}]=\frac{1}{2}\mid \mid \mathbf{P}-\mathbf{Q}\mid {\mid }^2,$$

(51)

where ||P||² is the Frobenius norm:

$$\mid \mid \mathbf{P}\mid {\mid }^2=\sum P_{ij}^2.$$

(52)

(2)

For f(λ) = −log λ

$$\psi (\mathbf{P})=-\text{log}\hspace{0.17em}(\text{det}\mid \mathbf{P}\mid ),$$

(53)

$$D[\mathbf{P}:\mathbf{Q}]=\text{tr}\hspace{0.17em}\left(\mathbf{P}{\mathbf{Q}}^{-1}\right)-\text{log}\hspace{0.17em}\left(\text{det}\mid \mathbf{P}{\mathbf{Q}}^{-1}\mid \right)-n.$$

(54)

The affine coordinate system is P, and the dual coordinate system is P⁻¹. The derived geometry is the same as that of multivariate Gaussian probability distributions with mean zero and covariance matrix P.

(3)

For f(λ) = λ log λ − λ,

$$\psi (\mathbf{P})=\text{tr}\hspace{0.17em}(\mathbf{P}\hspace{0.17em}\text{log}\hspace{0.17em}\mathbf{P}-\mathbf{P}),$$

(55)

$$D[\mathbf{P}:\mathbf{Q}]=\text{tr}\hspace{0.17em}(\mathbf{P}\hspace{0.17em}\text{log}\hspace{0.17em}\mathbf{P}-\mathbf{P}\hspace{0.17em}\text{log}\hspace{0.17em}\mathbf{Q}-\mathbf{P}+\mathbf{Q}).$$

(56)

This divergence is used in quantum information theory. The affine coordinate system is P, and the dual affine coordinate system is log P; and, ψ(P) is called the negative von Neuman entropy.

4.3. (ρ, τ )-Structure in Positive Definite Matrices

We extend the (ρ, τ )-structure in the previous section to the matrix case and introduce a general dually flat invariant decomposable divergence in the manifold of positive-definite matrices. Let:

$$\mathbf{\Theta }=\rho (\mathbf{P}),\mathrm{\hspace{0.17em} }\mathrm{\hspace{0.17em} }\mathrm{\hspace{0.17em} }\mathbf{H}=\tau (\mathbf{P})$$

(57)

be ρ- and τ -representations of matrices. We use two functions, ψ̃_ρ,τ (θ) and ϕ̃_ρ,τ (η), defined in Equations (19) and (20), for defining a pair of dually coupled invariant and decomposable convex functions,

$$\psi (\mathbf{\Theta })=\text{tr}\hspace{0.17em}{\tilde{\psi }}_{\rho ,\tau }\{\mathbf{\Theta }\},$$

(58)

$$\varphi (\mathbf{H})=\text{tr}\hspace{0.17em}{\tilde{\varphi }}_{\rho ,\tau }\{\mathbf{H}\}.$$

(59)

They are not convex with respect to P, but are convex with respect to Θ and H, respectively. The derived Bregman divergence is:

$$D[\mathbf{P}:\mathbf{Q}]=\psi \{\mathbf{\Theta }(\mathbf{P})\}+\varphi \{\mathbf{H}(\mathbf{Q})\}-\mathbf{\Theta }(\mathbf{P})·\mathbf{H}(\mathbf{Q}).$$

(60)

Theorem 2

The (ρ, τ )-divergences form a unique class of invariant, decomposable and dually flat divergences in the manifold of positive matrices.

The Euclidean, Gaussian and von Neuman divergences given in Equations (51), (54) and (56) are special examples of (ρ, τ )-divergences. They are given, respectively, by:

(1)

$$\rho (\xi )=\tau (\xi )=\xi ,$$

(61)

(2)

$$\rho (\xi )=\xi ,\mathrm{\hspace{0.17em} }\mathrm{\hspace{0.17em} }\mathrm{\hspace{0.17em} }\tau (\xi )=-\frac{1}{\xi },$$

(62)

(3)

$$\rho (\xi )=\xi ,\mathrm{\hspace{0.17em} }\mathrm{\hspace{0.17em} }\mathrm{\hspace{0.17em} }\tau (\xi )=\text{log}\hspace{0.17em}\xi .$$

(63)

When ρ and τ are power functions, we have the (α, β)-structure in the manifold of positive-definite matrices.

(4)

(α-β)-divergence.

By using the (α, β) power functions given by Equation (34), we have:

$$\psi (\mathbf{\Theta })=\frac{\alpha }{\alpha +\beta }\text{tr}\hspace{0.17em}{\mathrm{\Theta }}^{\frac{\alpha +\beta }{\alpha }}=\frac{\alpha }{\alpha +\beta }\text{tr}\hspace{0.17em}{\mathbf{P}}^{\alpha +\beta },$$

(64)

$$\varphi (\mathbf{H})=\frac{\beta }{\alpha +\beta }\text{tr}\hspace{0.17em}{\mathbf{H}}^{\frac{\alpha +\beta }{\beta }}=\frac{\beta }{\alpha +\beta }\text{tr}\hspace{0.17em}{\mathbf{P}}^{\alpha +\beta }$$

(65)

so that the (α, β)-divergence of matrices is:

$$D[\mathbf{P}:\mathbf{Q}]=\text{tr}\left\{\frac{\alpha }{\alpha +\beta }{\mathbf{P}}^{\alpha +\beta }+\frac{\beta }{\alpha +\beta }{\mathbf{Q}}^{\alpha +\beta }-{\mathbf{P}}^{\alpha }{\mathbf{Q}}^{\beta }\right\}.$$

(66)

This is a Bregman divergence, where the affine coordinate system is Θ = P^α and its dual is H = P^β.

(5)

The α-divergence is derived as:

$$\mathbf{\Theta }(\mathbf{P})=\frac{2}{1-\alpha }{\mathbf{P}}^{\frac{1-\alpha }{2}},$$

(67)

$$\psi (\mathbf{\Theta })=\frac{2}{1+\alpha }\mathbf{P},$$

(68)

$$D_{\alpha }[\mathbf{P}:\mathbf{Q}]=\frac{4}{1-{\alpha }^2}\text{tr}\hspace{0.17em}\left(-{\mathbf{P}}^{\frac{1-\alpha }{2}}{\mathbf{Q}}^{\frac{1+\alpha }{2}}+\frac{1-\alpha }{2}\mathbf{P}+\frac{1+\alpha }{2}\mathbf{Q}\right).$$

(69)

The affine coordinate system is $\frac{2}{1-\alpha }{\mathbf{P}}^{\frac{1-\alpha }{2}}$, and its dual is $\frac{2}{1+\alpha }{\mathbf{P}}^{\frac{1+\alpha }{2}}$.

(6)

The β-divergence is derived from Equation (41) as:

$$D_{\beta }[\mathbf{P}:\mathbf{Q}]=\frac{1}{\beta (\beta +1)}\text{tr}\hspace{0.17em}[{\mathbf{P}}^{\beta +1}+(\beta +1)\mathbf{Q}-{\mathbf{Q}}^{\beta +1}-(\beta +1)\mathbf{P}{\mathbf{Q}}^{\beta }].$$

(70)

4.4. Invariance Under Gl(n)

We extend the concept of invariance under the orthogonal group to that under the general linear group, Gl(n), that is the set of invertible matrices, L, det |L| ≠ 0. This is a stronger condition. A divergence is said to be invariant under Gl(n), when:

$$D[\mathbf{P}:\mathbf{Q}]=D\hspace{0.17em}[{\mathbf{L}}^T\mathbf{PL}:{\mathbf{L}}^T\mathbf{QL}]$$

(71)

holds for any L ∈ Gl(n).

We identify matrix P with the zero-mean Gaussian distribution:

$$p(\mathit{x},\mathbf{P})=\text{exp}\hspace{0.17em}\left\{-\frac{1}{2}{\mathit{x}}^T{\mathbf{P}}^{-1}\mathit{x}-\frac{1}{2}\text{log\hspace{0.17em}det}\mid \mathbf{P}\mid -c\right\},$$

(72)

where c is a constant. We know that an invariant divergence belongs to the class of f-divergences in the case of a manifold of probability distributions, where the invariance means the geometry does not change under a one-to-one mapping of x to y. Moreover, the only invariant flat divergence is the KL-divergence [22]. These facts suggest the following conjecture.

Proposition

The invariant, flat and decomposable divergence under Gl(n) is the KL-divergence given by:

$$D_{KL}[\mathit{P}:\mathit{Q}]=\text{tr}\hspace{0.17em}({\mathit{PQ}}^{-1})-\text{log}\hspace{0.17em}\left(\text{det}\mid {\mathit{PQ}}^{-1}\mid \right)-n.$$

(73)

5. Non-Decomposable Divergence

We have focused on flat and decomposable divergences. There are many interesting non-decomposable divergences. We first discuss a general class of flat divergences in ${\mathit{R}}_{+}^n$ and then touch upon interesting flat and non-flat divergences in the manifold of positive-definite matrices.

5.1. General Class of Flat Divergences in ${\mathit{R}}_{+}^n$

We can describe a general class of flat divergence in ${\mathit{R}}_{+}^n$, which are not necessarily decomposable. This is introduced in [23], which studies the conformal structure of general total Bregman divergences ([11,13]). When ${\mathit{R}}_{+}^n$ is endowed with a dually flat structure, it has a θ-coordinate system given by:

$$\mathit{\theta }=\mathit{\rho }(\mathit{\xi })$$

(74)

which is not necessarily a componentwise function. Any pair of invertible θ = ρ(ξ) and convex function ψ(θ) defines a dually flat structure and, hence, a Bregman divergence in ${\mathit{R}}_{+}^n$.

The dual coordinates η = τ (ξ) are given by:

$$\mathit{\eta }=\nabla \psi (\mathit{\theta })$$

(75)

so that we have:

$$\mathit{\eta }=\mathit{\tau }(\mathit{\xi })=\nabla \psi \{\mathit{\rho }(\mathit{\xi })\}.$$

(76)

This implies that a pair (ρ, τ ) of coordinate systems can define dually coupled affine coordinates and, hence, a dually flat structure, when and only when η = τ {ρ⁻¹(θ)} is a gradient of a convex function.

This is different from the case of decomposable divergence, where any monotone pair of ρ(ξ) and τ (ξ) gives a dually flat structure.

5.2. Non-Decomposable Flat Divergence in PD_n

Ohara and Eguchi [15,16] introduced the following function:

$${\psi }_V(\mathbf{P})=V\hspace{0.17em}(\text{det}\hspace{0.17em}\mid \mathbf{P}\mid ),$$

(77)

where V (ξ) is a monotonically decreasing scalar function. ψ_V is convex when and only when:

$$1+\frac{V^{″}(\xi ){\xi }^2}{V^{\prime }(\xi )}<\frac{1}{n}.$$

(78)

In such a case, we can introduce dually flat structure to PD_n, where P is an affine coordinate system with convex ψ_V (P), and the dual affine coordinate system is:

$$\mathbf{H}=V^{\prime }(\text{det}\mid \mid P\mid \mid ){\mathbf{P}}^{-1}.$$

(79)

The derived divergence is:

$$D_V[\mathbf{P}:\mathbf{Q}]=V(\text{det}\mid \mathbf{P})-V(\text{det}\mid \mathbf{Q})\mid$$

(80)

$$+V^{\prime }(\text{det}\hspace{0.17em}\mid \mathbf{Q}\mid )\text{tr}\hspace{0.17em}\{{\mathbf{Q}}^{-1}(\mathbf{Q}-\mathbf{P})\}.$$

(81)

When V (ξ) = −log ξ, it reduces to the case of Equation (54), which is invariant under Gl(n) and decomposable. However, the divergence D_V [P : Q] is not decomposable. It is invariant under O(n) and more strongly so under SGl(n) ⊂ Gl(n), defined by det |L| = ±1.

5.3. Flat Structure Derived from q-Escort Distribution

A dually flat structure is introduced in the manifold of probability distributions [4] as:

$${\tilde{D}}_{\alpha }[\mathit{p}:\mathit{q}]=\frac{1}{1-q}\frac{1}{H_q(\mathit{p})}\hspace{0.17em}\left(1-\sum p_i^{1-q}{q}_i^q\right),$$

(82)

where:

$$H_q(\mathit{p})=\sum p_i^q,$$

(83)

$$q=\frac{1+\alpha }{2}.$$

(84)

The dual affine coordinates are the q-escort distribution: [4]

$${\eta }_i=\frac{1}{H_q(\mathit{p})}{p}_i^q.$$

(85)

The divergence, D̃_q, is flat, but not decomposable.

We can generalize it to the case of PD_n,

$${\tilde{D}}_q[\mathbf{P}:\mathbf{Q}]=\frac{1}{1-q}\frac{1}{\text{tr}\hspace{0.17em}{\mathbf{P}}^q}\{(1-q)\hspace{0.17em}\text{tr}\hspace{0.17em}(\mathbf{P})+q\hspace{0.17em}\text{tr}\hspace{0.17em}(\mathbf{Q})-\text{tr}\hspace{0.17em}({\mathbf{P}}^{1-\text{q}}{\mathbf{Q}}^{\text{q}})\}.$$

(86)

This is flat, but not decomposable.

5.4. γ-Divergence in PD_n

The γ-divergence is introduced by Fujisawa and Eguchi [24]. It gives a super-robust estimator. It is interesting to generalize it to PD_n,

$$D_{\gamma }[\mathbf{P}:\mathbf{Q}]=\frac{1}{\gamma (\gamma -1)}\{\text{log\hspace{0.17em}tr}\hspace{0.17em}{\mathbf{P}}^{\gamma }-(\gamma -1)\hspace{0.17em}\text{log\hspace{0.17em}tr}\hspace{0.17em}{\mathbf{Q}}^{\gamma -1}-\gamma \hspace{0.17em}\text{log\hspace{0.17em}tr}\hspace{0.17em}\mathbf{P}{\mathbf{Q}}^{\gamma -1}\}.$$

(87)

This is not flat nor decomposable. This is a projective divergence in the sense that, for any c, c′ > 0,

$$D_{\gamma }[c\mathbf{P}:c^{\prime }\mathbf{Q}]=D_{\gamma }[\mathbf{P}:\mathbf{Q}].$$

(88)

Therefore, it can be defined in the submanifold of tr P = 1.

6. Concluding Remarks

We have shown that the (ρ, τ )-divergence introduced by Zhang [5] is a general dually flat decomposable structure of the manifold of positive measures. We then extended it to the manifold of positive-definite matrices, where the criterion of invariance under linear transformations (in particular, under orthogonal transformations) were added. The decomposability is useful from the computational point of view, because the θ-η transformation is tractable. This is the motivation for studying decomposable flat divergences.

When we treat the manifold of probability distributions, it is a submanifold of the manifold of positive measures, where the total sum of measures are restricted to one. This is a nonlinear constraint in the θ or η coordinates, so that the manifold is not flat, but curved in general. Hence, our arguments hold in this case only when at least one of the ρ and τ functions are linear. The U-divergence [21] and β-divergence [19] are such cases. However, for clustering, we can take the average of the η-coordinates of member probability distributions in the larger manifold of positive measures and then project it to the manifold of probability distributions. This is called the exterior average, and the projection is simply a normalization of the result. Therefore, the (ρ, τ )-structure is useful in the case of probability distributions. The same situation holds in the case of positive-definite matrices.

Quantum information theory deals with positive-definite Hermitian matrices of trace one [25,26]. We need to extend our discussions to the case of complex matrices. The trace one constraint is not linear with respect to θ- or η-coordinates, as is the same in the case of probability distributions. Many interesting divergence functions have been introduced in the manifold of positive-definite Hermitian matrices. It is an interesting future problem to apply our theory to quantum information theory.