The Fisher–Rao Distance between Multivariate Normal Distributions: Special Cases, Bounds and Applications

Abstract

The Fisher–Rao distance is a measure of dissimilarity between probability distributions, which, under certain regularity conditions of the statistical model, is up to a scaling factor the unique Riemannian metric invariant under Markov morphisms. It is related to the Shannon entropy and has been used to enlarge the perspective of analysis in a wide variety of domains such as image processing, radar systems, and morphological classification. Here, we approach this metric considered in the statistical model of normal multivariate probability distributions, for which there is not an explicit expression in general, by gathering known results (closed forms for submanifolds and bounds) and derive expressions for the distance between distributions with the same covariance matrix and between distributions with mirrored covariance matrices. An application of the Fisher–Rao distance to the simplification of Gaussian mixtures using the hierarchical clustering algorithm is also presented.

1. Introduction

A proper measure to determine the dissimilarity between probability distributions has been approached in many problems and applications. The Fisher–Rao distance is a very special metric for statistical models of probability distributions. This distance is invariant by reparametrization of the sample space and covariant by reparameterization of the parameter space [1]. Moreover, the Fisher–Rao metric is preserved under Markov morphisms and under centain conditions it is, up to a scaling factor, the unique Riemannian metric satisfying this condition [2,3]. Markov morphisms are associated with the notion of statistical sufficiency which express the criterion of passing from one statistical model to another with no loss of information [4,5,6]. Therefore it is natural to require the invariance of the geometric structures of statistical models under Markov morphisms. Between finite sample size simplex model $S_{k-1}=\{p\in {\mathbb{R}}^k;p_i\ge 0\mathrm{and}{\sum }_{i=1}^k{p}_i=1\}$, a Markov morphism is a linear map $T_Q\left(x\right)=xQ$, where $Q\in {\mathbb{R}}^{n\times l}$, with $n\le l$, is a matrix with non-negative entries such that every row sums to 1 and every column has precisely one non-zero element. The mapping $T_Q$ corresponds to probabilistic refining of the event space $\{1,\dots ,n\}\to \{1,\dots ,l\}$ where the refinement $i\to j$ occurs with probability $Q_{ij}$ [7]. Chentsov [8,9] has proved the Fisher–Rao uniqueness invariance property under Markov morphisms for the finite sample spaces. The extension of this result to more general statistical models requires careful formulations of statistical sufficiency and Markov morphisms and has been evolved since then [3,10]. More recently in [5,6] it is shown this uniqueness of the Fisher–Rao metric under an assumption of strong continuity of the information metric.

After previous papers [11,12,13] connecting geometry and statistics, C. R.Rao in an independent landmark paper [14] considered statistical models with the metric induced by the information matrix defined by R. Fisher in 1921 [15]. This work encouraged several authors to calculate the Fisher–Rao metric distance between other probability distributions [16,17,18] as well as stimulated approaches to other dissimilarity measures such as Kullback-Leibler divergence [19], total variation and Wasserstein distances [20]. Amari [3,4,21] unified the information geometry theory by organizing and introducing other concepts regarding statistical models [2].

An explicit form for the Fisher–Rao distance in the univariate normal distribution space is known via an association with the classical model of the hyperbolic plane [14,16,18,22]. It was applied to quantization of hyperspectral images [23] and to the space of projected lines in the paracatadioptric images [24]. This Fisher–Rao model was used to simplify Gaussian mixtures through the k-means method [25] and a hierarchical clustering technique [26].

An expression for the geodesic curve (initial value problem) in the multivariate normal distributions space was derived in [27] and in [28]. However, the calculus of the Fisher–Rao distance requires solving non-trivial differential equations under boundary conditions to find the geodesic connecting two distributions and then to calculate the integral along the geodesic. A closed form for this distance in the general case is still an open problem. Expressions for the distance are known only in special cases [16,17,18].

The Fisher–Rao distance between multivariate normal distributions in specific cases, such as distributions with a common mean, was considered in diffusion tensor image analysis [29,30,31], in color texture discrimination in several classification experiments [32], in the problem of distributed estimation fusion with unknown correlations [33], and in the machine learning technique [34]. In [35,36], the authors described shapes representing landmarks by a Gaussian model with diagonal covariance matrices and used the Fisher–Rao distance to quantify the difference between two shapes. In [17], this model was applied to statistical inference. Bounds for the Fisher–Rao distance were used to track quality monitoring [37].

This paper is organized as follows. In Section 2, we gather known results (closed forms for special cases and bounds) for the Fisher–Rao distance between multivariate normal distributions. In Section 3, we describe a closed form for the Fisher–Rao distance between distributions with the same covariance matrix and a non-linear system to find the distance between distributions with mirrored covariance matrices. An application of the Fisher–Rao distance to the simplification of Gaussian mixtures using the hierarchical clustering algorithm is presented in Section 4. Some conclusions and perspectives are drawn in Section 5.

2. The Fisher–Rao Distance in the Multivariate Normal Distribution Space: Special Submanifolds and Bounds

In this section, as in [38], we summarize previous results regarding the Fisher–Rao distance in the space of multivariate normal distributions including closed forms for this distance restricted to submanifolds and general bounds.

Given a statistical model $S=\{p_{\mathit{\theta }}=p(\mathit{x};\mathit{\theta });\mathit{\theta }=({\theta }_1,{\theta }_2,\dots ,{\theta }_k)\in \mathsf{\Theta }\subset {\mathbb{R}}^k\}$, a natural Riemannian structure [21] can be provided by the Fisher information matrix $G\left(\mathit{\theta }\right)=\left[g_{ij}\left(\mathit{\theta }\right)\right]$:

$$\begin{array}{cc}\hfill g_{ij}\left(\mathit{\theta }\right)=& E_{\mathit{\theta }}\left(\frac{\partial }{\partial {\theta }_i}logp(\mathit{x};\mathit{\theta })\frac{\partial }{\partial {\theta }_j}logp(\mathit{x};\mathit{\theta })\right)\hfill \end{array}$$

(1)

$$\begin{array}{cc}\hfill =& \int \frac{\partial }{\partial {\theta }_i}logp(\mathit{x};\mathit{\theta })\frac{\partial }{\partial {\theta }_j}logp(\mathit{x};\mathit{\theta })p(\mathit{x};\mathit{\theta })dx,\hfill \end{array}$$

(2)

where $E_{\mathit{\theta }}$ is the expected value with respect to the distribution $p_{\mathit{\theta }}$. This matrix can also be viewed as the Hessian matrix of the Shannon entropy (concave function) [39],

$$H(p)=-\int p(\mathit{x};\mathit{\theta })logp(\mathit{x};\mathit{\theta })dx,$$

(3)

and is used to establish connections between inequalities in information theory and geometrical inequalities.

The Fisher–Rao distance, $d_F(·,·)$, between two distributions $p_{{\mathit{\theta }}_1}$ and $p_{{\mathit{\theta }}_2}$ in $\mathcal{S}$, identified with their parameters ${\mathit{\theta }}_1$ and ${\mathit{\theta }}_2$, is given by the shortest length of a curve $\gamma \left(t\right)$ in the parameter space $\mathsf{\Theta }$ connecting these distributions, ${\displaystyle d_F(p_{{\mathit{\theta }}_1},p_{{\mathit{\theta }}_2})\equiv d_F({\mathit{\theta }}_1,{\mathit{\theta }}_2)=\underset{\gamma }{min}\int {\left|{\gamma }^{\prime }\left(t\right)\right|}_{G}dt}$, where $|{\gamma }^{\prime }{\left(t\right)|}_G=\sqrt{{\gamma }^{\prime }{\left(t\right)}^{t}G\left(\mathit{\theta }\right)\gamma \left(t\right)}$. Note that this is in fact a metric, since for any ${\mathit{\theta }}_1$, ${\mathit{\theta }}_2$, and ${\mathit{\theta }}_3$ in $\mathsf{\Theta }$, we have: (i) $d_F({\mathit{\theta }}_1,{\mathit{\theta }}_2)\ge 0$ and $d_F({\mathit{\theta }}_1,{\mathit{\theta }}_2)=0$ if only if ${\mathit{\theta }}_1={\mathit{\theta }}_2$; (ii) $d_F({\mathit{\theta }}_1,{\mathit{\theta }}_2)=d_F({\mathit{\theta }}_2,{\mathit{\theta }}_1)$; (iii) $d_F({\mathit{\theta }}_1,{\mathit{\theta }}_2)\le d_F({\mathit{\theta }}_1,{\mathit{\theta }}_3)+d_F({\mathit{\theta }}_3,{\mathit{\theta }}_2)$. A curve that provides the shortest length is called a geodesic and is given by the solutions of the differential equations

$$\frac{d^2{\theta }_m}{d{t}^2}+\sum _{i,j}{\mathsf{\Gamma }}_{ij}^m\frac{d{\theta }_i}{dt}\frac{d{\theta }_j}{dt}=0,m=1,\cdots ,k,$$

(4)

where ${\mathsf{\Gamma }}_{ij}^m$ are the Christoffel symbols,

$${\mathsf{\Gamma }}_{ij}^m=\frac{1}{2}\sum _l\left(\frac{\partial g_{jl}}{\partial {\theta }_i}+\frac{\partial g_{li}}{\partial {\theta }_j}-\frac{\partial g_{ij}}{\partial {\theta }_l}\right)g^{lm}$$

(5)

and $\left[g^{ij}\right]$ is the inverse matrix of the Fisher information matrix.

We consider here the space of the multivariate normal distributions given by:

$${\displaystyle p(\mathit{x};\mathit{\mu },\mathsf{\Sigma })=\frac{{\left(2\pi \right)}^{-\left(\frac{n}{2}\right)}}{\sqrt{Det\left(\mathsf{\Sigma }\right)}}exp\left(-\frac{{(\mathit{x}-\mathit{\mu })}^t{\mathsf{\Sigma }}^{-1}(\mathit{x}-\mathit{\mu })}{2}\right),}$$

(6)

where ${\mathit{x}}^t=(x_1,\dots ,x_n)\in {\mathbb{R}}^n$ is the variable vector, ${\mathit{\mu }}^t=({\mu }_1,\dots ,{\mu }_n)\in {\mathbb{R}}^n$ is the mean vector, and $\mathsf{\Sigma }$ is the covariance matrix in $P_n\left(\mathbb{R}\right)$, the space of order n positive definite symmetric matrices.

In this case, the model $\mathcal{S}=\mathcal{M}=\{p_{\mathit{\theta }};\mathit{\theta }=(\mathit{\mu },\mathsf{\Sigma })\in {\mathbb{R}}^n\times P_n\left(\mathbb{R}\right)\}$ is a statistical $\left(n+\frac{n(n+1)}{2}\right)$-dimensional manifold.

In this case, the model $\mathcal{S}=\mathcal{M}=\{p_{\mathit{\theta }};\mathit{\theta }=(\mathit{\mu },\mathsf{\Sigma })\in {\mathbb{R}}^n\times P_n\left(\mathbb{R}\right)\}$ is a statistical manifold of dimension $k=\left(n+\frac{n(n+1)}{2}\right)$. Considering a parametrization $(\mathit{\mu },\mathsf{\Sigma })=\varphi ({\theta }_1,\dots ,{\theta }_k)$ of the model $\mathcal{M}$, the Fisher information matrix is given by [40]

$$g_{ij}\left(\mathit{\theta }\right)=\frac{\partial {\mathit{\mu }}^t}{\partial {\theta }_i}{\mathsf{\Sigma }}^{-1}\frac{\partial \mathit{\mu }}{\partial {\theta }_j}+\frac{1}{2}tr\left({\mathsf{\Sigma }}^{-1}\frac{\partial \mathsf{\Sigma }}{\partial {\theta }_i}{\mathsf{\Sigma }}^{-1}\frac{\partial \mathsf{\Sigma }}{\partial {\theta }_i}\right).$$

(7)

The metric provided by this matrix is invariant with respect to affine transformations. In other words, for any $(\mathit{c},Q)\in {\mathbb{R}}^n\times G{L}_n\left(\mathbb{R}\right)$, where $G{L}_n\left(\mathbb{R}\right)$ is the group of non-singular n-square matrices, the mapping:

$$\begin{array}{cccc}{\psi }_{(\mathit{c},Q)}:& \mathcal{M}& \to & \mathcal{M}\\ & (\mathit{\mu },\mathsf{\Sigma })& \mapsto & (Q\mathit{\mu }+\mathit{c},Q\mathsf{\Sigma }{Q}^t),\end{array}$$

(8)

is an isometry in $\mathcal{M}$ [16]. Consequently, the Fisher–Rao distance between ${\mathit{\theta }}_1=({\mathit{\mu }}_1,{\mathsf{\Sigma }}_1)$ and ${\mathit{\theta }}_2=({\mathit{\mu }}_2,{\mathsf{\Sigma }}_2)$ in $\mathcal{M}$ satisfies:

$$d_F({\mathit{\theta }}_1,{\mathit{\theta }}_2)=d_F((Q{\mathit{\mu }}_1+\mathit{c},Q{\mathsf{\Sigma }}_1{Q}^t),(Q{\mathit{\mu }}_2+\mathit{c},Q{\mathsf{\Sigma }}_2{Q}^t))$$

(9)

for any $(\mathit{c},Q)\in {\mathbb{R}}^n\times G{L}_n\left(\mathbb{R}\right)$. In particular, for $Q={\mathsf{\Sigma }}_1^{-(1/2)}$ and $\mathit{c}=-{\mathsf{\Sigma }}_1^{(-1/2)}{\mathit{\mu }}_1$, ${\mathit{\theta }}_3=({\mathit{\mu }}_3,{\mathsf{\Sigma }}_3)=({\mathsf{\Sigma }}_1^{-(1/2)}({\mathit{\mu }}_2-{\mathit{\mu }}_1),{\mathsf{\Sigma }}_1^{-(1/2)}{\mathsf{\Sigma }}_2{\mathsf{\Sigma }}_1^{-(1/2)})$, the Fisher–Rao distance admits the form:

$$d_F({\mathit{\theta }}_1,{\mathit{\theta }}_2)=d_F({\mathit{\theta }}_0,{\mathit{\theta }}_3),$$

(10)

where ${\mathit{\theta }}_0=(\mathbf{0},I_n)$, $I_n$ is the n-order identity matrix, and $\mathbf{0}\in {\mathbb{R}}^n$ is the null vector.

The geodesic equations in $\mathcal{M}$ can be expressed as [17]:

$$\left\{\begin{array}{c}\frac{d^2\mathit{\mu }}{d{t}^2}-\left(\frac{d\mathsf{\Sigma }}{dt}\right){\mathsf{\Sigma }}^{-1}\left(\frac{d\mathit{\mu }}{dt}\right)=0\hfill \\ \frac{d^2\mathsf{\Sigma }}{d{t}^2}+\left(\frac{d\mathit{\mu }}{dt}\right){\left(\frac{d\mathit{\mu }}{dt}\right)}^t-\left(\frac{d\mathsf{\Sigma }}{dt}\right){\mathsf{\Sigma }}^{-1}\left(\frac{d\mathsf{\Sigma }}{dt}\right)=0.\hfill \end{array}\right.$$

(11)

and could be partially integrated [27]:

$$\left\{\begin{array}{c}\frac{d\mathit{\mu }}{dt}=\mathsf{\Sigma }\mathit{x}\hfill \\ \frac{d\mathsf{\Sigma }}{dt}=\mathsf{\Sigma }(B-{\mathit{x}}^t\mu ),\hfill \end{array}\right.$$

(12)

$$\left\{\begin{array}{c}\frac{d\Delta }{dt}=-B\Delta +\mathit{x}{\mathit{\delta }}^t\hfill \\ \frac{d\mathit{\delta }}{dt}=-B\mathit{\delta }+(1+\mathit{\delta }{\Delta }^{-1}\mathit{\delta })\mathit{x}\hfill \end{array}\right..$$

(13)

where $(\mathit{\delta }\left(t\right),\Delta \left(t\right))=({\mathsf{\Sigma }}^{-1}\left(t\right)\mathit{\mu }\left(t\right),{\mathsf{\Sigma }}^{-1}\left(t\right))$, $\mathit{x}\in {\mathbb{R}}^n$, and B is a symmetric matrix. The initial conditions for this problem can be taken as:

$$\left\{\begin{array}{c}(\mathit{\delta }\left(0\right),\Delta \left(0\right))=(\mathbf{0},I_n)\hfill \\ \left(\frac{d\mathit{\delta }}{dt}\left(0\right),\frac{d\Delta }{dt}\left(0\right)\right)=(\mathit{x},-B).\hfill \end{array}\right.$$

(14)

Eriksen [27] and Calvo and Oller [28], in independent works, solved this initial value problem. An explicit solution to the geodesic curve in $\mathcal{M}$ [28] is:

$$\left\{\begin{array}{cc}\mathit{\delta }\left(t\right)=\hfill & -B(cosh\left(tG\right)-I_n){\left(G^{-}\right)}^2\mathit{x}+sinh\left(tG\right)G^{-}\mathit{x}\hfill \\ \Delta \left(t\right)=\hfill & I_n+\frac{1}{2}(cosh\left(tG\right)-I_n)+\frac{1}{2}B(cosh\left(tG\right)-I_n){\left(G^{-}\right)}^{2}B\hfill \\ \hfill & -\frac{1}{2}sinh\left(tG\right)G^{-}B-\frac{1}{2}Bsinh\left(tG\right)G^{-}\hfill \end{array}\right.,$$

(15)

where $I_n$ is an n-order identity matrix, $G^2=B^2+2\mathit{x}{\mathit{x}}^t$, and $G^{-}$ is the generalized inverse square matrix of G, that is $G{G}^{-}G=G$.

Due the fact that the geodesic curve has constant velocity at any point, given $(\mathit{x},B)$ in the tangent space of $\mathcal{M}$, the Fisher–Rao distance between $(\mathbf{0},I_n)$ and $\left(\mathit{\delta }\right(1),\Delta (1\left)\right)$ is:

$${\int }_0^1\sqrt{\frac{d\mathit{\mu }}{dt}\left(0\right){\mathsf{\Sigma }}^{-1}\left(0\right)\frac{d\mathit{\mu }}{dt}\left(0\right)+\frac{1}{2}tr\left[{\left({\mathsf{\Sigma }}^{-1}\left(0\right)\frac{\mathsf{\Sigma }}{dt}\left(0\right)\right)}^2\right]}dt=\sqrt{\frac{1}{2}tr\left(B^2\right)+{\left|\mathit{x}\right|}^2},$$

(16)

where $|·|$ is the standard Euclidean norm. Note that the above expression provides the Fisher–Rao distance between two distributions only if we can determine the initial value problem from the boundary conditions, which usually is very difficult.

Han and Park in [31] presented a numerical shooting method for computing the minimum geodesic distance between two normal distributions, through parallel transport of a vector field defined along the geodesic curve given in Equation (15).

A closed form for the Fisher–Rao distance between two normal distributions in $\mathcal{M}$ is still an important open question. Next, we present closed forms for this distance in some submanifolds of $\mathcal{M}$.

2.1. Closed Forms for the Fisher–Rao Distance in Submanifolds of $\mathcal{M}$

In this subsection, we consider submanifolds ${\mathcal{M}}_{\ast }\subset \mathcal{M}$ with the distance induced by the Fisher–Rao metric in $\mathcal{M}$. It is important to remark that, in general, given two distributions ${\mathit{\theta }}_1$ and ${\mathit{\theta }}_2$ in ${\mathcal{M}}_{\ast }$, the distance between ${\mathit{\theta }}_1$ and ${\mathit{\theta }}_2$ when restricted to a submanifold ${\mathcal{M}}_{\ast }$ is bigger than the distance between ${\mathit{\theta }}_1$ and ${\mathit{\theta }}_2$ in $\mathcal{M}$, that is $d_{{\mathcal{M}}_{\ast }}({\mathit{\theta }}_1,{\mathit{\theta }}_2)\ge d_{\mathcal{M}}({\mathit{\theta }}_1,{\mathit{\theta }}_2)$. This is due to the fact that to get $d_{{\mathcal{M}}_{\ast }}$, we consider the minimum length of restricted curves, which are the ones contained in the submanifold ${\mathcal{M}}_{\ast }$. We say that ${\mathcal{M}}_{\ast }$ is totally geodesic if only if $d_{{\mathcal{M}}_{\ast }}({\mathit{\theta }}_1,{\mathit{\theta }}_2)=d_{\mathcal{M}}({\mathit{\theta }}_1,{\mathit{\theta }}_2)$, for any ${\mathit{\theta }}_1,{\mathit{\theta }}_2\in {\mathcal{M}}_{\ast }$, which means that the geodesic in $\mathcal{M}$ connecting ${\mathit{\theta }}_1$ and ${\mathit{\theta }}_2$ is contained in ${\mathcal{M}}_{\ast }$.

2.1.1. The Submanifold ${\mathcal{M}}_{\mathsf{\Sigma }}$ Where $\mathsf{\Sigma }$ Is Constant

In the n-dimensional manifold composed by multivariate normal distributions with common covariance matrix $\mathsf{\Sigma }$, ${\mathcal{M}}_{\mathsf{\Sigma }}=\{p_{\mathit{\theta }};\mathit{\theta }=(\mathit{\mu },\mathsf{\Sigma }),\mathsf{\Sigma }={\mathsf{\Sigma }}_0\in P_n\left(\mathbb{R}\right)\mathrm{constant}\}$, the Fisher–Rao distance between two distributions ${\mathit{\theta }}_1=({\mathit{\mu }}_1,{\mathsf{\Sigma }}_0)$ and ${\mathit{\theta }}_2=({\mathit{\mu }}_2,{\mathsf{\Sigma }}_0)$ is [18]:

$$d_{\mathsf{\Sigma }}({\mathit{\theta }}_1,{\mathit{\theta }}_2)=\sqrt{{({\mathit{\mu }}_1-{\mathit{\mu }}_2)}^t{\mathsf{\Sigma }}_0^{-1}({\mathit{\mu }}_1-{\mathit{\mu }}_2)}.$$

(17)

This distance is equal to the Mahalanobis distance [11], which is equal to the Euclidean distance between the image of ${\mathit{\mu }}_1$ and ${\mathit{\mu }}_2$ under the transformation $\mathit{\mu }\mapsto P^{-1}\mathit{\mu }$, where ${\mathsf{\Sigma }}_0=P{P}^t$ is the Cholesky decomposition [18]. This distance was one of the first dissimilarity measures between datasets with some correlation. Note that this submanifold is not totally geodesic, as it can be seen even in the space of univariate normal distributions [22] and in Example 1 in the next section.

A geodesic curve ${\gamma }_{\mathsf{\Sigma }}\left(t\right)$ in ${\mathcal{M}}_{\mathsf{\Sigma }}$ connecting ${\mathit{\theta }}_1$ and ${\mathit{\theta }}_2$ can be provided by:

$${\gamma }_{\mathsf{\Sigma }}\left(t\right)=((1-t){\mathit{\mu }}_1-t{\mathit{\mu }}_2,{\mathsf{\Sigma }}_0).$$

(18)

2.1.2. The Submanifold ${\mathcal{M}}_{\mu }$ Where $\mu$ Is Constant

A totally geodesic submanifold of $\mathcal{M}$ is given by ${\mathcal{M}}_{\mathit{\mu }}=\{p_{\mathit{\theta }};\mathit{\theta }=(\mathit{\mu },\mathsf{\Sigma }),\mathit{\mu }={\mathit{\mu }}_0\in {\mathbb{R}}^n\mathrm{constant}\}$ of dimension $\frac{n(n+1)}{2}$ composed by distributions that have the same mean vector ${\mathit{\mu }}_0$. The Fisher–Rao distance in ${\mathcal{M}}_{\mathit{\mu }}$ was studied by several authors in different contexts [16,18,30,41] and for ${\mathit{\theta }}_1=({\mathit{\mu }}_0,{\mathsf{\Sigma }}_1)$ and ${\mathit{\theta }}_2=({\mathit{\mu }}_0,{\mathsf{\Sigma }}_2)$ is given by:

$$d_F({\mathit{\theta }}_1,{\mathit{\theta }}_2)=\sqrt{\frac{1}{2}{\displaystyle \sum _{i=1}^n}{[log\left({\lambda }_i\right)]}^2},$$

(19)

where $0<{\lambda }_1\le {\lambda }_2\le \cdots \le {\lambda }_n$ are the eigenvalues of ${\mathsf{\Sigma }}_1^{-1/2}{\mathsf{\Sigma }}_2{\mathsf{\Sigma }}_1^{-1/2}$.

An expression for the geodesic curve connecting these two distributions is [30]:

$${\gamma }_{\mathit{\mu }}\left(t\right)=({\mathit{\mu }}_0,{\mathsf{\Sigma }}_1^{1/2}exp(tlog\left({\mathsf{\Sigma }}_1^{-1/2}{\mathsf{\Sigma }}_2{\mathsf{\Sigma }}_1^{-1/2}\right)){\mathsf{\Sigma }}_1^{1/2}).$$

(20)

2.1.3. The Submanifold ${\mathcal{M}}_D$ Where $\mathsf{\Sigma }$ Is Diagonal

Let ${\mathcal{M}}_D=\{p_{\mathit{\theta }};\mathit{\theta }=(\mathit{\mu },\mathsf{\Sigma }),\mathsf{\Sigma }=diag({\sigma }_1^2,{\sigma }_2^2,\dots ,{\sigma }_n^2),{\sigma }_i>0,i=1,\dots ,n\}$, the submanifold of $\mathcal{M}$ composed by distributions with a diagonal covariance matrix. If we consider the parameter $\mathit{\theta }=({\mu }_1,{\sigma }_1,{\mu }_2,{\sigma }_2,\dots ,{\mu }_n,{\sigma }_n)$, it can be shown [22] that the metric in the parametric space of ${\mathcal{M}}_D$ is equal to the product metric:

$$d_D({\mathit{\theta }}_1,{\mathit{\theta }}_2)=\sqrt{{\displaystyle \sum _{i=1}^n}{d}_{F\ast }^2(({\mu }_{1i},{\sigma }_{1i}),({\mu }_{2i},{\sigma }_{2i}))},$$

(21)

where $d_{F\ast }$ is the Fisher–Rao distance in the univariate case given by [22]:

$$d_{F\ast }(({\mu }_1,{\sigma }_1),({\mu }_2,{\sigma }_2))=\sqrt{2}log\frac{\left|\left(\frac{{\mu }_1}{\sqrt{2}},{\sigma }_1\right)-\left(\frac{{\mu }_2}{\sqrt{2}},-{\sigma }_2\right)\right|+\left|\left(\frac{{\mu }_1}{\sqrt{2}},{\sigma }_1\right)-\left(\frac{{\mu }_2}{\sqrt{2}},{\sigma }_2\right)\right|}{\left|\left(\frac{{\mu }_1}{\sqrt{2}},{\sigma }_1\right)-\left(\frac{{\mu }_2}{\sqrt{2}},-{\sigma }_2\right)\right|-\left|\left(\frac{{\mu }_1}{\sqrt{2}},{\sigma }_1\right)-\left(\frac{{\mu }_2}{\sqrt{2}},{\sigma }_2\right)\right|}.$$

(22)

In this space, a curve ${\gamma }_D\left(t\right)=({\gamma }_1\left(t\right),\dots ,{\gamma }_n\left(t\right))$ is a geodesic if, and only if, ${\gamma }_i\left(t\right)$ is a geodesic curve in the univariate case, for all $i=1,\dots ,n$. The geodesic curves in the univariate normal distributions space (upper half plane $\mathbb{R}\times {\mathbb{R}}^{+}$) are half-vertical lines and half-ellipses centered at $\sigma =0$, with eccentricity $\frac{1}{\sqrt{2}}$ [22].

It is important to note that ${\mathcal{M}}_D\subset \mathcal{M}$ is not totally geodesic. The submanifold of ${\mathcal{M}}_D$ composed only by normal distributions with covariance matrices which are multiples of the identity (round normals) is totally geodesic [22]. In fact, this submanifold of round normals is also contained in the totally geodesic submanifold described next.

2.1.4. The Submanifold ${\mathcal{M}}_{D\mu }$ Where $\mathsf{\Sigma }$ Is Diagonal and $\mu$ Is an Eigenvector of $\mathsf{\Sigma }$

Let ${\mathcal{M}}_{D\mathit{\mu }}$ be the $n+1$-dimensional submanifold composed by distributions with the mean vector $\mathit{\mu }={\mu }_1{\mathit{e}}_i$ for some ${\mathit{e}}_i\in \{{\mathit{e}}_1,\dots ,{\mathit{e}}_n\}$ (the canonical basis of ${\mathbb{R}}^n$) and diagonal covariance matrix $\mathsf{\Sigma }$, and without loss of generality, we shall assume that ${\mathit{e}}_i={\mathit{e}}_1$. An analytic expression for the distance in ${\mathcal{M}}_{D\mathit{\mu }}$ is:

$$d_{D\mathit{\mu }}^2({\mathit{\theta }}_1,{\mathit{\theta }}_2)=d_{F\ast }^2\left(\left({\mu }_{11},{\sigma }_{11}\right),\left({\mu }_{21},{\sigma }_{21}\right)\right)+\sum _{i=2}^n{d}_{F\ast }^2\left(\left(0,{\sigma }_{1i}\right),\left(0,{\sigma }_{2i}\right)\right).$$

(23)

We proved in [42] that this submanifold is totally geodesic.

2.2. Bounds for the Fisher–Rao in $\mathcal{M}$

As mentioned, a closed form for the Fisher–Rao distance between two general normal distributions is not known. In this subsection, we present some bounds for this distance.

2.2.1. A Lower Bound

Calvo and Oller [43] derived a lower bound for the Fisher–Rao distance through an isometric embedding of the parametric space $\mathcal{M}$ into the manifold of the positive definite matrices.

Proposition 1.

[43] Given ${\mathit{\theta }}_1=({\mathit{\mu }}_1,{\mathsf{\Sigma }}_1)$ and ${\mathit{\theta }}_2=({\mathit{\mu }}_2,{\mathsf{\Sigma }}_2)$, let:

$$S_i=\left(\begin{array}{cc}{\mathsf{\Sigma }}_i+{\mathit{\mu }}_i{\mathit{\mu }}_i^t& {\mathit{\mu }}_i^t\\ {\mathit{\mu }}_i& 1\end{array}\right),$$

(24)

$i=1,2$. A lower bound for the distance between ${\mathit{\theta }}_1$ and ${\mathit{\theta }}_2$ is:

$$LB({\mathit{\theta }}_1,{\mathit{\theta }}_2)=\sqrt{\frac{1}{2}{\displaystyle \sum _{i=1}^{n+1}}{[log\left({\lambda }_i\right)]}^2},$$

(25)

where ${\lambda }_i$, $1\le i\le n+1$, are the eigenvalues of $S_1^{-1/2}{S}_2{S}_1^{-1/2}$.

We note that this bound satisfies the distance proprieties in $\mathcal{M}$. In [44], through a similar approach, a lower bound for the Fisher–Rao distance was obtained in the more general space of elliptical distributions, restricted to normal distributions, is the above bound.

2.2.2. The Upper Bound $U{B}_1$

In [45], we proposed an upper bound based on an isometry (8) in the manifold $\mathcal{M}$ and on the distance in the non-totally geodesic submanifold ${\mathcal{M}}_D$ (21), as follows:

Proposition 2.

[45] The Fisher–Rao distance between two multivariate normal distributions ${\mathit{\theta }}_1=({\mathit{\mu }}_1,{\mathsf{\Sigma }}_1)$ and ${\mathit{\theta }}_2=({\mathit{\mu }}_2,{\mathsf{\Sigma }}_2)$ is upper bounded by,

$$U{B}_1({\mathit{\theta }}_1,{\mathit{\theta }}_2)=\sqrt{{\displaystyle \sum _{i=1}^n}{d}_{F\ast }^2((0,1),({\mu }_i,{\lambda }_i))},$$

(26)

where ${\lambda }_i$ are the diagonal terms of the matrix Λ given by the eigenvalues of $A={\mathsf{\Sigma }}_1^{-(1/2)}{\mathsf{\Sigma }}_2{\mathsf{\Sigma }}_1^{-(1/2)}=Q\mathsf{\Lambda }{Q}^t$, ${\mu }_i$ are the coordinates of $\mathit{\mu }=Q^t{\mathsf{\Sigma }}_1^{-(1/2)}({\mathit{\mu }}_2-{\mathit{\mu }}_1)$, Q is the orthogonal matrix whose columns are the eigenvectors of A and $d_{F\ast }$ is the Fisher–Rao distance between univariate normal distributions given in Equation (22).

2.2.3. The Upper Bounds $U{B}_2$ and $U{B}_3$

Considering the Fisher–Rao distance in the totally geodesic submanifold ${\mathcal{M}}_{D\mathit{\mu }}$ and the triangular inequality, we propose another upper bound [42].

Given ${\mathit{\theta }}_1=({\mathit{\mu }}_1,{\mathsf{\Sigma }}_1)$ and ${\mathit{\theta }}_2=({\mathit{\mu }}_2,{\mathsf{\Sigma }}_2)$, we consider the Fisher–Rao distance between ${\mathit{\theta }}_0=(\mathbf{0},I_n)$ and ${\mathit{\theta }}_3=({\mathit{\mu }}_3,{\mathsf{\Sigma }}_3)$ as in Equation (10). Let $\overline{\mathit{\theta }}=(\overline{\mathit{\mu }},\overline{\mathsf{\Sigma }})$; by the triangular inequality, it follows that:

$$d_F({\mathit{\theta }}_0,{\mathit{\theta }}_3)\le d_F({\mathit{\theta }}_0,\overline{\mathit{\theta }})+d_F(\overline{\mathit{\theta }},{\mathit{\theta }}_3).$$

(27)

To calculate this bound, we choose $\overline{\mathit{\theta }}$ appropriately. For $\overline{\mathit{\mu }}={\mathit{\mu }}_3$, note that $d_F(\overline{\mathit{\theta }},{\mathit{\theta }}_3)=d_{\mathit{\mu }}(\overline{\mathit{\theta }},{\mathit{\theta }}_3)$. Let P be an orthogonal matrix such that $P\mathit{\mu }=\left(\right|{\mathit{\mu }}_3|,0,\dots ,0)$ and $D=diag(d_1^2,d_2^2,\dots ,d_n^2)$ a diagonal matrix. We will consider $\overline{\mathsf{\Sigma }}=P^{-1}D{P}^{-t}$ and ${\mathit{\theta }}_P=(P\mathit{\mu },D)$. By the isometry ${\psi }_{(\mathit{c},Q)}$, given in Equation (9), for $Q=P^{-1}$ and $c=\mathbf{0}$, it follows:

$$d_F({\mathit{\theta }}_0,\overline{\mathit{\theta }})=d_{D\mathit{\mu }}({\mathit{\theta }}_0,{\mathit{\theta }}_P).$$

(28)

Then, combining Inequality (27) and Equation (28), the left side of the equation below is an upper bound for the Fisher–Rao distance between ${\mathit{\theta }}_1$ and ${\mathit{\theta }}_2$,

$$d_F({\mathit{\theta }}_0,{\mathit{\theta }}_3)\le d_{D\mathit{\mu }}({\mathit{\theta }}_0,{\mathit{\theta }}_P)+d_{\mathit{\mu }}(\overline{\mathit{\theta }},{\mathit{\theta }}_3).$$

(29)

In [42], we derived the upper bound:

$$U{B}_2=d_{D\mathit{\mu }}({\mathit{\theta }}_0,{\mathit{\theta }}_P)+d_{\mathit{\mu }}(\overline{\mathit{\theta }},{\mathit{\theta }}_3).$$

(30)

through a numerical minimization process by considering the diagonal elements of D as a vector that minimizes $d_{D\mathit{\mu }}({\mathit{\theta }}_0,{\mathit{\theta }}_P)+d_{\mathit{\mu }}(\overline{\mathit{\theta }},{\mathit{\theta }}_3)$,

$$({\overline{d}}_1,{\overline{d}}_2,\dots ,{\overline{d}}_n)=\underset{(d_1,d_2,\dots ,d_n)}{min}\{d_{D\mathit{\mu }}({\mathit{\theta }}_0,{\mathit{\theta }}_P)+d_{\mathit{\mu }}(\overline{\mathit{\theta }},{\mathit{\theta }}_3)\}.$$

(31)

We also derive an analytic upper bound $U{B}_3$ by minimizing of the distance $d_{D\mathit{\mu }}({\mathit{\theta }}_0,{\mathit{\theta }}_P)$. By expressing this distance in terms of the parameters $({\mathit{\theta }}_0,{\mathit{\theta }}_P)=\left((\mathbf{0},I_n),(P\mathit{\mu },D)\right)$, we can show that it reaches the minimum at:

$$D=diag\left(\sqrt{\frac{|{\mathit{\mu }}_3|+2}{2}},1,\cdots ,1\right).$$

(32)

The lower bound of Section 2.2.1 and the upper bounds of Section 2.2.2 and Section 2.2.3 are summarized in

Table 1. The lower bound $LB({\mathit{\theta }}_1,{\mathit{\theta }}_2)$ and the upper bounds $U{B}_1({\mathit{\theta }}_1,{\mathit{\theta }}_2)$, $U{B}_2({\mathit{\theta }}_1,{\mathit{\theta }}_2)$ and $U{B}_3({\mathit{\theta }}_1,{\mathit{\theta }}_2)$ for the Fisher–Rao distance, $d_F({\mathit{\theta }}_1,{\mathit{\theta }}_2)$, between distributions ${\mathit{\theta }}_1=({\mathit{\mu }}_1,{\mathsf{\Sigma }}_1)$ and ${\mathit{\theta }}_2=({\mathit{\mu }}_2,{\mathsf{\Sigma }}_2)$ in $\mathcal{M}$. $d_{F\ast }$ is the distance between univariate normal distributions given in Equation (22).

${\displaystyle LB({\mathit{\theta }}_1,{\mathit{\theta }}_2)=\sqrt{\frac{1}{2}{\displaystyle \sum _{i=1}^{n+1}}{[log\left({\lambda }_i\right)]}^2}}$	$\begin{array}{ll}–& S_i=\left(\begin{array}{cc}{\mathsf{\Sigma }}_i+{\mathit{\mu }}_i^t{\mathit{\mu }}_i& {\mathit{\mu }}_i^t\\ {\mathit{\mu }}_i& 1\end{array}\right);\\ –& {\lambda }_i\mathrm{are}\mathrm{the}\mathrm{eigenvalues}\mathrm{of}{S}_1^{-1/2}{S}_2{S}_1^{-1/2}.\end{array}$
$U{B}_1({\mathit{\theta }}_1,{\mathit{\theta }}_2)=\sqrt{{\displaystyle \sum _{i=1}^n}{d}_{F\ast }^2((0,1),({\mu }_i,{\lambda }_i))}$	$\begin{array}{ll}–& {\mathsf{\Sigma }}_1^{-(1/2)}{\mathsf{\Sigma }}_2{\mathsf{\Sigma }}_1^{-(1/2)}=Q\mathsf{\Lambda }{Q}^t;\\ –& {\lambda }_i\mathrm{are}\mathrm{the}\mathrm{diagonal}\mathrm{terms}\mathrm{of}\mathrm{the}\mathrm{matrix}\mathsf{\Lambda };\\ –& {\mu }_i\mathrm{are}\mathrm{the}\mathrm{coordinates}\mathrm{of}\mathit{\mu }=Q^t{\mathsf{\Sigma }}_1^{-(1/2)}({\mathit{\mu }}_2-{\mathit{\mu }}_1).\end{array}$
${\displaystyle U{B}_2({\mathit{\theta }}_1,{\mathit{\theta }}_2)=\sqrt{\frac{1}{2}{\displaystyle \sum _{i=1}^n}{[log\left({\lambda }_i\right)]}^2}+}$ ${\displaystyle \sqrt{d_{F\ast }^2((0,1),(|{\mathit{\mu }}_3|,{\overline{d}}_1\left)\right)+{\displaystyle \sum _{i=2}^m}{d}_{F\ast }^2((0,1),(0,{\overline{d}}_i))}}$	$\begin{array}{ll}–& {\mathit{\mu }}_3={\mathsf{\Sigma }}_1^{-(1/2)}({\mathit{\mu }}_2-{\mathit{\mu }}_1);\\ –& {\overline{d}}_i\mathrm{is}\mathrm{given}\mathrm{by}\left(31\right);\\ –& {\mathsf{\Sigma }}_3={\mathsf{\Sigma }}_1^{-(1/2)}{\mathsf{\Sigma }}_2{\mathsf{\Sigma }}_1^{-(1/2)};\\ –& P\mathrm{is}\mathrm{an}\mathrm{orthogonal}\mathrm{matrix}\mathrm{such}\mathrm{that}\\ & P\mathit{\mu }=\left(\right|{\mathit{\mu }}_3|,0,\dots ,0);\\ –& \overline{\mathsf{\Sigma }}=P^{-1}{\mathsf{\Sigma }}_3{P}^{-t};\\ –& {\lambda }_i\mathrm{are}\mathrm{the}\mathrm{eigenvalues}\mathrm{of}{\overline{\mathsf{\Sigma }}}^{-(1/2)}{\mathsf{\Sigma }}_3{\overline{\mathsf{\Sigma }}}^{-(1/2)}.\end{array}$
${\displaystyle U{B}_3({\mathit{\theta }}_1,{\mathit{\theta }}_2)=\sqrt{\frac{1}{2}{\displaystyle \sum _{i=1}^n}{[log\left({\lambda }_i\right)]}^2}+}$ ${\displaystyle d_{F\ast }\left((0,1),\left(|{\mathit{\mu }}_3|,\sqrt{\frac{|{\mathit{\mu }}_3|+2}{2}}\right)\right)}$	$\begin{array}{ll}–& {\mathit{\mu }}_3={\mathsf{\Sigma }}_1^{-(1/2)}({\mathit{\mu }}_2-{\mathit{\mu }}_1);\\ –& {\mathsf{\Sigma }}_3={\mathsf{\Sigma }}_1^{-(1/2)}{\mathsf{\Sigma }}_2{\mathsf{\Sigma }}_1^{-(1/2)};\\ –& P\mathrm{is}\mathrm{an}\mathrm{orthogonal}\mathrm{matrix}\mathrm{such}\mathrm{that}\\ & P\mathit{\mu }=\left(\right|{\mathit{\mu }}_3|,0,\dots ,0);\\ –& \overline{\mathsf{\Sigma }}=P^{-1}{\mathsf{\Sigma }}_3{P}^{-t};\\ –& {\lambda }_i\mathrm{are}\mathrm{the}\mathrm{eigenvalues}\mathrm{of}{\overline{\mathsf{\Sigma }}}^{-(1/2)}{\mathsf{\Sigma }}_3{\overline{\mathsf{\Sigma }}}^{-(1/2)}.\end{array}$

.

Upper and lower bounds have been used to estimate the Fisher–Rao distance in applications such as [37].

2.2.4. Comparisons of the Bounds

In this section, as in [42], we illustrate comparisons between the bounds presented previously.

We consider the bivariate normal distributions model ($n=2$) and distributions ${\mathit{\theta }}_0$ and $\widehat{\mathit{\theta }}=(\widehat{\mathit{\mu }},\widehat{\mathsf{\Sigma }})$, where:

$${\displaystyle \widehat{\mathit{\theta }}=(\widehat{\mathit{\mu }},\widehat{\mathsf{\Sigma }})=\left(\left(\begin{array}{c}\mu \\ 0\end{array}\right),\left(\begin{array}{cc}cos\left(\alpha \right)& sin\left(\alpha \right)\\ -sin\left(\alpha \right)& cos\left(\alpha \right)\end{array}\right)\left(\begin{array}{cc}{\lambda }_1& 0\\ 0& {\lambda }_2\end{array}\right)\left(\begin{array}{cc}cos\left(\alpha \right)& -sin\left(\alpha \right)\\ sin\left(\alpha \right)& cos\left(\alpha \right)\end{array}\right)\right).}$$

(33)

From (10), we can see that there always exists an isometry that converts any two pairs of bivariate distributions into a pair of distributions as above.

We present next a comparison between the lower bound “LB” (25), the upper bounds $U{B}_1$ (26), $U{B}_2$ (31), and $U{B}_3$ (32), and the numerical solution given by the geodesic shooting algorithm (GS) [31] in specific situations.

In Figure 1, we consider the eigenvalues ${\lambda }_1=2$, ${\lambda }_2=0.5$, and $\mu =1$ to be fixed and $\alpha$ varying from zero to ${\displaystyle \frac{\pi }{2}}$. We note that the upper bound $U{B}_1$ is very near the lower bound $LB$ and to the numerical solution $GS$. The other upper bounds are bigger than the bound $U{B}_1$. In Figure 2, it is considered $\mu =10$ and the previous eigenvalues. Now, the best performance is of bounds $U{B}_2$ and $U{B}_3$, which are similar. In Figure 3a, we again keep the eigenvalues; the rotation angle is fixed $\alpha =\frac{\pi }{4}$; and $\mu$ varies from zero to 10. We can see similar performances of $U{B}_2$ and $U{B}_3$, which are better than $U{B}_1$ for larger values of $\mu$.

We may also consider the upper bound:

$$U{B}_{123}({\mathit{\theta }}_1,{\mathit{\theta }}_2)=min\{U{B}_1({\mathit{\theta }}_1,{\mathit{\theta }}_2),U{B}_2({\mathit{\theta }}_1,{\mathit{\theta }}_2),U{B}_3({\mathit{\theta }}_1,{\mathit{\theta }}_2)\}.$$

(34)

Figure 3b displays the comparison between $LB$, $U{B}_{123}$, and $GS$ for the same data of the comparison in Figure 3a.

3. Fisher–Rao Distance Between Special Distributions

In this section, we describe the Fisher–Rao distance in the full space $\mathcal{M}$ between special kinds of distributions.

3.1. The Fisher–Rao Distance Between Distributions with Common Covariance Matrices

The Fisher–Rao distance between distributions with common covariance matrices given in Section 2.1.1 was restricted to non-totally geodesic submanifold ${\mathcal{M}}_{\mathsf{\Sigma }}$. We show next that using the isometry given in (8) and the distance in the submanifold ${\mathcal{M}}_{D\mathit{\mu }}$, it is possible to find a closed form for the distance between two distributions with the same covariance matrix, in the full manifold $\mathcal{M}$.

Proposition 3.

Given two distributions ${\mathit{\theta }}_1=({\mathit{\mu }}_1,\mathsf{\Sigma })$ and ${\mathit{\theta }}_2=({\mathit{\mu }}_2,\mathsf{\Sigma })$ in $\mathcal{M}$, let P be an orthogonal matrix such that $P({\mathit{\mu }}_2-{\mathit{\mu }}_1)=|{\mathit{\mu }}_2-{\mathit{\mu }}_1|{\mathit{e}}_1$, and consider the decomposition $UD{U}^t$ of the matrix $P\mathsf{\Sigma }{P}^t$,

$$P\mathsf{\Sigma }{P}^t=UD{U}^t,$$

(35)

where U is an upper triangular matrix with all diagonal entries equal to one and D is a diagonal matrix. The Fisher–Rao distance between ${\mathit{\theta }}_1$ and ${\mathit{\theta }}_2$ is given by:

$$d_F({\mathit{\theta }}_1,{\mathit{\theta }}_2)=d_{D\mathit{\mu }}((\mathbf{0},D),\left(\right|{\mathit{\mu }}_2-{\mathit{\mu }}_1|{\mathit{e}}_1,D\left)\right).$$

(36)

Proof. 

By considering the isometries $\psi ={\psi }_{(-P{\mathit{\mu }}_1,P)}$ and $\widehat{\psi }={\psi }_{(\mathbf{0},U^{-1})}$ and the decomposition given by Equation (35), it follows from Equation (9) that:

$$\begin{array}{cc}\hfill d_F({\mathit{\theta }}_1,{\mathit{\theta }}_2)& =d_F({\psi }_{(-P{\mathit{\mu }}_1,P)}\left({\mathit{\theta }}_1\right),{\psi }_{(-P{\mathit{\mu }}_1,P)}\left({\mathit{\theta }}_2\right))\hfill \\ \hfill & =d_F((P{\mathit{\mu }}_1-P{\mathit{\mu }}_1,P\mathsf{\Sigma }{P}^t),(P{\mathit{\mu }}_2-P{\mathit{\mu }}_1,P\mathsf{\Sigma }{P}^t))\hfill \\ \hfill & =d_F((\mathbf{0},P\mathsf{\Sigma }{P}^t),\left(\right|{\mathit{\mu }}_2-{\mathit{\mu }}_1|{\mathit{e}}_1,P\mathsf{\Sigma }{P}^t\left)\right).\hfill \\ \hfill & =d_F(\widehat{\psi }(\mathbf{0},P\mathsf{\Sigma }{P}^t),\widehat{\psi }\left(\right|{\mathit{\mu }}_2-{\mathit{\mu }}_1|{\mathit{e}}_1,P\mathsf{\Sigma }{P}^t\left)\right)\hfill \\ \hfill & =d_F((U^{-1}\mathbf{0},U^{-1}P\mathsf{\Sigma }{P}^t{U}^{-t}),\left(\right|{\mathit{\mu }}_2-{\mathit{\mu }}_1|U^{-1}{\mathit{e}}_1,U^{-1}P\mathsf{\Sigma }{P}^t{U}^{-t}\left)\right)\hfill \\ \hfill & =d_F((\mathbf{0},D),(|{\mathit{\mu }}_2-{\mathit{\mu }}_1|{\mathit{e}}_1,D\left)\right).\hfill \end{array}$$

(37)

Since the distributions $(\mathbf{0},D)$ and $\left(\right|{\mathit{\mu }}_2-{\mathit{\mu }}_1|{\mathit{e}}_1,D)$ belong to the submanifold ${\mathcal{M}}_{D\mathit{\mu }}$, we conclude that:

$$d_F({\mathit{\theta }}_1,{\mathit{\theta }}_2)=d_{D\mathit{\mu }}((\mathbf{0},D),\left(\right|{\mathit{\mu }}_2-{\mathit{\mu }}_1|{\mathit{e}}_1,D\left)\right).$$

(38)

□

Example 1.

Consider two bivariate normal distributions ${\mathit{\theta }}_1=({(-1,0)}^t,\mathsf{\Sigma })$ and ${\mathit{\theta }}_2=({(6,3)}^t,\mathsf{\Sigma })$ with the same covariance matrix:

$$\mathsf{\Sigma }=\left(\begin{array}{cc}1.1& 0.9\\ 0.9& 1.1\end{array}\right).$$

Figure 4a illustrates the normal distributions in the geodesic curve connecting connecting ${\mathit{\theta }}_1$ and ${\mathit{\theta }}_2$ in $\mathcal{M}$, and Figure 4b illustrates the geodesic in the submanifold ${\mathcal{M}}_{\mathsf{\Sigma }}$. We observe that in $\mathcal{M}$, the shape of the ellipses (contour curves) changes along the path. Furthermore, the Fisher–Rao distance between ${\mathit{\theta }}_1$ and ${\mathit{\theta }}_2$ is $d_F({\mathit{\theta }}_1,{\mathit{\theta }}_2)=5.00648$, which is less than the Mahalanobis distance given in Equation (17), $d_{\mathsf{\Sigma }}({\mathit{\theta }}_1,{\mathit{\theta }}_2)=8.06226$, as expected, since the submanifold ${\mathcal{M}}_{\mathsf{\Sigma }}$ is not totally geodesic.

3.2. The Fisher–Rao Distance Between Mirrored Distributions

We consider here two mirrored normal distributions; that is, without loss of generality, if we consider up rotation, the line connecting ${\mathit{\mu }}_1$ and ${\mathit{\mu }}_2$ as parallel to the ${\mathit{e}}_1$-axis, and the covariance matrices ${\mathsf{\Sigma }}_1$ and ${\mathsf{\Sigma }}_2$ satisfying:

$${\mathsf{\Sigma }}_2=M_1{\mathsf{\Sigma }}_1{M}_1,\mathrm{where}{M}_1=\left(\begin{array}{cc}-1& 0\\ 0& I_{n-1}\end{array}\right).$$

(39)

This condition implies also the same eigenvalues for both matrices.

For bivariate normal distributions, we then should have:

$${\mathit{\theta }}_1=\left(\left(\begin{array}{c}{\mu }_1\\ {\mu }_0\end{array}\right),\left(\begin{array}{cc}{\sigma }_{11}& {\sigma }_{12}\\ {\sigma }_{12}& {\sigma }_{22}\end{array}\right)\right)\mathrm{e}{\mathit{\theta }}_2=\left(\left(\begin{array}{c}{\mu }_2\\ {\mu }_0\end{array}\right)\left(\begin{array}{cc}{\sigma }_{11}& -{\sigma }_{12}\\ -{\sigma }_{12}& {\sigma }_{22}\end{array}\right)\right);$$

(40)

see Figure 5.

After several experiments using the algorithm geodesic shooting for the ${\mathit{\theta }}_1$ and ${\mathit{\theta }}_2$, we have observed that for $t=0$ the geodesic curve connecting these distributions ($\gamma \left(t\right)=\left(\mathit{\mu }\right(t),\mathsf{\Sigma }(t\left)\right)$, with $\gamma (-1)={\mathit{\theta }}_1$ and $\gamma \left(1\right)={\mathit{\theta }}_2$), satisfies

$$\gamma \left(0\right)\approx {\mathit{\theta }}_{1/2}=({\mathit{\mu }}_{1/2},{\mathsf{\Sigma }}_{1/2})=\left(\left(\begin{array}{c}\frac{{\mu }_1+{\mu }_2}{2}\\ \eta \end{array}\right),\left(\begin{array}{cc}{d}_{11}^2& 0\\ 0& d_{22}^2\end{array}\right)\right),$$

(41)

$${\gamma }^{\prime }\left(0\right)\approx {\widehat{\mathit{\theta }}}_{1/2}=({\widehat{\mathit{\mu }}}_{1/2},{\widehat{\mathsf{\Sigma }}}_{1/2})=\left(\left(\begin{array}{c}{\widehat{\mu }}_1\\ 0\end{array}\right),\left(\begin{array}{cc}0& {\widehat{\sigma }}_{12}\\ {\widehat{\sigma }}_{12}& 0\end{array}\right)\right).$$

(42)

where $\eta$, $d_{11}$, and $d_{22}$ are real values; see Figure 6.

The focus here is the “shape” of these distributions. Note that at $t=0$, the distribution $\gamma \left(0\right)$ appears as ${\mathit{\theta }}_{1/2}$, which has a diagonal covariance matrix, and the tangent vector ${\gamma }^{\prime }\left(0\right)$ appears as ${\widehat{\mathit{\theta }}}_{1/2}$, which is composed by a mean vector with the second entry equal to zero and by a symmetric covariance matrix with a null diagonal.

This observation inspired us to get an explicit expression for the geodesic connecting two mirrored distributions. Starting with the bi-dimensional case again, we will prove that in fact we have equality in Expressions (41) and (42).

Let $\gamma \left(t\right)=\left(\mathit{\mu }\right(t),\mathsf{\Sigma }(t\left)\right)$, $-1\le t\le 1$, and the geodesic curve in $\mathcal{M}$ connecting ${\mathit{\theta }}_1$ and ${\mathit{\theta }}_2$, and consider that $\gamma \left(0\right)={\mathit{\theta }}_{1/2}$ and ${\gamma }^{\prime }\left(0\right)={\widehat{\mathit{\theta }}}_{1/2}$. Given the isometry $\psi ={\psi }_{\left(-{\mathsf{\Sigma }}_{1/2}^{-1/2}{\mathit{\mu }}_{1/2},{\mathsf{\Sigma }}_{1/2}^{-1/2}\right)}$, we define:

$$\overline{\gamma }\left(t\right)=(\overline{\mathit{\mu }}\left(t\right),\overline{\mathsf{\Sigma }}\left(t\right)):=\psi \left(\gamma \left(t\right)\right)=\left({\mathsf{\Sigma }}_{1/2}^{-1/2}(\mathit{\mu }\left(t\right)-{\mathit{\mu }}_{1/2}),{\mathsf{\Sigma }}_{1/2}^{-1/2}\mathsf{\Sigma }\left(t\right){\mathsf{\Sigma }}_{1/2}^{-1/2}\right).$$

(43)

Then:

$${\overline{\gamma }}^{\prime }\left(t\right)=\left(\frac{d\overline{\mathit{\mu }}\left(t\right)}{dt},\frac{\overline{\mathsf{\Sigma }}\left(t\right)}{dt}\right)=\left({\mathsf{\Sigma }}_{1/2}^{-1/2}\left(\frac{d\mathit{\mu }\left(t\right)}{dt}\right),{\mathsf{\Sigma }}_{1/2}^{-1/2}\left(\frac{\mathsf{\Sigma }\left(t\right)}{dt}\right){\mathsf{\Sigma }}_{1/2}^{-1/2}\right),$$

(44)

$$\begin{array}{cc}\hfill \overline{\gamma }\left(0\right)=& \left({\mathsf{\Sigma }}_{1/2}^{-1/2}({\mathit{\mu }}_{1/2}-{\mathit{\mu }}_{1/2}),{\mathsf{\Sigma }}_{1/2}^{-1/2}{\mathsf{\Sigma }}_{1/2}{\mathsf{\Sigma }}_{1/2}^{-1/2}\right)\hfill \\ \hfill =& (\mathbf{0},I_2)=:{\mathit{\theta }}_0\hfill \end{array}$$

(45)

and:

$$\begin{array}{cc}\hfill {\overline{\gamma }}^{\prime }\left(0\right)& =\left({\mathsf{\Sigma }}_{1/2}^{-1/2}{\mathit{\mu }}_{1/2}^{\prime },{\mathsf{\Sigma }}_{1/2}^{-1/2}{\mathsf{\Sigma }}_{1/2}^{\prime }{\mathsf{\Sigma }}_{1/2}^{-1/2}\right)\hfill \\ \hfill & =\left(\left(\begin{array}{c}\frac{{\mu }^{\prime }\left(0\right)}{d_{11}}\\ 0\end{array}\right),\left(\begin{array}{cc}0& \frac{{\sigma }_{12}^{\prime }\left(0\right)}{d_{11}{d}_{22}}\\ \frac{{\sigma }_{12}^{\prime }\left(0\right)}{d_{11}{d}_{22}}& 0\end{array}\right)\right).\hfill \end{array}$$

(46)

Applying the natural changing of parameters:

$$(\mathit{\delta }\left(t\right),\Delta \left(t\right))=\phi (\overline{\mathit{\mu }}\left(t\right),\overline{\mathsf{\Sigma }}\left(t\right))=(\overline{\mathsf{\Sigma }}{\left(t\right)}^{-1}\overline{\mathit{\mu }}\left(t\right),\overline{\mathsf{\Sigma }}{\left(t\right)}^{-1}),$$

(47)

it follows that:

$$\left\{\begin{array}{cc}\hfill \frac{d\Delta }{dt}\left(t\right)=& -\Delta \left(t\right)\left(\frac{d\overline{\mathsf{\Sigma }}}{dt}\left(t\right)\right)\Delta \left(t\right)\hfill \\ \hfill \frac{d\mathit{\delta }}{dt}\left(t\right)=& \left(\frac{d\Delta }{dt}\left(t\right)\right)\overline{\mathit{\mu }}\left(t\right)+\Delta \left(t\right)\left(\frac{d\overline{\mathit{\mu }}}{dt}\left(t\right)\right)\hfill \end{array}\right..$$

(48)

Then, given that $(\mathit{\delta }\left(0\right),\Delta \left(0\right))=(\overline{\mathit{\mu }}\left(0\right),\overline{\mathsf{\Sigma }}\left(0\right))=(\mathbf{0},I_2)$,

$$\left\{\begin{array}{cc}\hfill \frac{d\Delta }{dt}\left(0\right)=& -\Delta \left(0\right)\left(\frac{d\overline{\mathsf{\Sigma }}}{dt}\left(0\right)\right)\Delta \left(0\right)=-\frac{d\overline{\mathsf{\Sigma }}}{dt}\left(0\right)\hfill \\ \hfill \frac{d\mathit{\delta }}{dt}\left(0\right)=& \left(\frac{d\Delta }{dt}\left(0\right)\right)\overline{\mathit{\mu }}\left(0\right)+\Delta \left(0\right)\left(\frac{d\overline{\mathit{\mu }}}{dt}\left(0\right)\right)=\frac{d\overline{\mathit{\mu }}}{dt}\left(0\right)\hfill \end{array}\right..$$

(49)

That is, at $t=0$, the tangent vector $\left(\frac{d\mathit{\delta }}{dt}\left(0\right),\frac{d\Delta }{dt}\left(0\right)\right)$ is equal to the tangent vector in (46). Furthermore, the distributions ${\mathbf{\vartheta }}_1=\phi \left({\overline{\mathit{\theta }}}_1\right)=({\overline{\mathsf{\Sigma }}}_1^{-1}{\overline{\mathit{\mu }}}_1,{\overline{\mathsf{\Sigma }}}_1^{-1})$ and ${\mathbf{\vartheta }}_2=\phi \left({\overline{\mathit{\theta }}}_2\right)=({\overline{\mathsf{\Sigma }}}_2^{-1}{\overline{\mathit{\mu }}}_2,{\overline{\mathsf{\Sigma }}}_2^{-1})$ are also mirrored ${\overline{\mathsf{\Sigma }}}_2^{-1}=M_1{\overline{\mathsf{\Sigma }}}_1^{-1}{M}_1$. In fact,

$$\begin{array}{cc}\hfill {\mathbf{\vartheta }}_1=& ({\overline{\mathsf{\Sigma }}}_1^{-1}{\overline{\mathit{\mu }}}_1,{\overline{\mathsf{\Sigma }}}_1^{-1})\hfill \\ \hfill =& ({\left({\mathsf{\Sigma }}_{1/2}^{-1/2}{\mathsf{\Sigma }}_1{\mathsf{\Sigma }}_{1/2}^{-1/2}\right)}^{-1}{\mathsf{\Sigma }}_{1/2}^{-1/2}({\mathit{\mu }}_1-{\mathit{\mu }}_{1/2}),{\left({\mathsf{\Sigma }}_{1/2}^{-1/2}{\mathsf{\Sigma }}_1{\mathsf{\Sigma }}_{1/2}^{-1/2}\right)}^{-1})\hfill \\ \hfill =& \left(\frac{1}{det\left({\mathsf{\Sigma }}_1\right)}\left(\begin{array}{c}{\sigma }_{22}{d}_{11}\frac{{\mu }_1-{\mu }_2}{2}-{\sigma }_{12}{d}_{11}({\mu }_0-\eta )\\ {\sigma }_{11}{d}_{22}({\mu }_0-\eta )-{\sigma }_{12}{d}_{22}\frac{{\mu }_1-{\mu }_2}{2}\end{array}\right),\frac{1}{det\left({\mathsf{\Sigma }}_1\right)}\left(\begin{array}{cc}{\sigma }_{22}{d}_{11}^2& -{\sigma }_{12}{d}_{11}{d}_{22}\\ -{\sigma }_{12}{d}_{11}{d}_{22}& {\sigma }_{11}{d}_{22}^2\end{array}\right)\right),\hfill \end{array}$$

(50)

and by similar arguments, we obtain:

$$\begin{array}{c}\hfill {\mathbf{\vartheta }}_2=\left(\frac{1}{det\left({\mathsf{\Sigma }}_1\right)}\left(\begin{array}{c}{\sigma }_{22}{d}_{11}\frac{{\mu }_2-{\mu }_1}{2}+{\sigma }_{12}{d}_{11}({\mu }_0-\eta )\\ {\sigma }_{11}{d}_{22}({\mu }_0-\eta )-{\sigma }_{12}{d}_{22}\frac{{\mu }_1-{\mu }_2}{2}\end{array}\right),\frac{1}{det\left({\mathsf{\Sigma }}_1\right)}\left(\begin{array}{cc}{\sigma }_{22}{d}_{11}^2& {\sigma }_{12}{d}_{11}{d}_{22}\\ {\sigma }_{12}{d}_{11}{d}_{22}& {\sigma }_{11}{d}_{22}^2\end{array}\right)\right).\end{array}$$

(51)

Figure 7 illustrates the distributions ${\mathit{\theta }}_0$, ${\mathbf{\vartheta }}_1$, and ${\mathbf{\vartheta }}_2$.

Conversely, by considering:

$$(\mathit{x},B)=\left(\left(\begin{array}{c}x\\ 0\end{array}\right),\left(\begin{array}{cc}0& b\\ b& 0\end{array}\right)\right)$$

(52)

in the initial value problem given in Equations (13) and (14), it follows that the matrix $G^2=B^2+\mathit{x}{\mathit{x}}^t$ is diagonal. Therefore, the geodesic curve $\left(\mathit{\delta }\right(t),\Delta (t\left)\right)$ with initial value $(\mathit{\delta }\left(0\right),\Delta \left(0\right))={\mathit{\theta }}_0$ and tangent vector $(\mathit{x},B)$ given in Equation (15) can be simplified as follows:

$$\left\{\begin{array}{cc}\hfill \mathit{\delta }\left(t\right)=& \left(\begin{array}{c}\frac{xsinh\left(t\sqrt{b^2+2x^2}\right)}{\sqrt{b^2+2x^2}}\\ -\frac{bx(cosh\left(t\sqrt{b^2+2x^2}\right)-1)}{b^2+2x^2}\end{array}\right)\hfill \\ \hfill \Delta \left(t\right)=& \left(\begin{array}{cc}\frac{1}{2}(cosh\left(bt\right)+cosh\left(t\sqrt{b^2+2x^2}\right))& -\frac{1}{2}\left(sinh\left(bt\right)+\frac{bsinh\left(t\sqrt{b^2+2x^2}\right)}{\sqrt{b^2+2x^2}}\right)\\ -\frac{1}{2}\left(sinh\left(bt\right)+\frac{bsinh\left(t\sqrt{b^2+2x^2}\right)}{\sqrt{b^2+2x^2}}\right)& \frac{1}{2}\left(cosh\left(bt\right)+\frac{2x^2+b^{2}cosh\left(t\sqrt{b^2+2x^2}\right)}{b^2+2x^2}\right)\end{array}\right).\hfill \end{array}\right.$$

(53)

From the parity of the functions $sinh\left(t\right)$ and $cosh\left(t\right)$, it is possible to show that, given $t_0\in \mathbb{R}$, the distributions:

$$(\mathit{\delta }(-t_0),\Delta (-t_0))\mathrm{and}(\mathit{\delta }\left(t_0\right),\Delta \left(t_0\right))$$

are also mirrored.

By the above discussion, we conclude that it is possible to calculate the geodesic curve connecting ${\mathit{\theta }}_1$ and ${\mathit{\theta }}_2$ making ${\psi }^{-1}\left({\phi }^{-1}(\mathit{\delta }(-1),\Delta (-1))\right)={\mathit{\theta }}_1$ and $\psi -1\left({\phi }^{-1}(\mathit{\delta }\left(1\right),\Delta \left(1\right))\right)={\mathit{\theta }}_2$. That is, we need to find the values of $\eta$, $d_{11}$, and $d_{22}$ of the isometry $\psi$ and the values of $(\mathit{x},B)$ such that:

$$\left\{\begin{array}{cc}\hfill \phi \left(\psi \left({\mathit{\theta }}_1\right)\right)=& \left(\mathit{\delta }\right(-1),\Delta (-1\left)\right)\hfill \\ \hfill \phi \left(\psi \left({\mathit{\theta }}_2\right)\right)=& \left(\mathit{\delta }\right(1),\Delta (1\left)\right)\hfill \end{array}\right..$$

(54)

Since the two equations above are equivalent, it is enough to solve the equation:

$$(\mathit{\delta }\left(1\right),\Delta \left(1\right))=\phi \left(\psi ({\mathit{\mu }}_2,{\mathsf{\Sigma }}_2)\right).$$

(55)

This is equivalent to solving the system:

$$\left\{\begin{array}{cc}\hfill & \left(\begin{array}{cc}\frac{1}{d_{11}}& 0\\ 0& \frac{1}{d_{22}}\end{array}\right)\Delta \left(1\right)\left(\begin{array}{cc}\frac{1}{d_{11}}& 0\\ 0& \frac{1}{d_{22}}\end{array}\right)={\Delta }_2\hfill \\ \hfill & \left(\begin{array}{cc}\frac{1}{d_{11}}& 0\\ 0& \frac{1}{d_{22}}\end{array}\right)\mathit{\delta }\left(1\right)+{\Delta }_2\left(\begin{array}{c}\frac{{\mu }_1+{\mu }_2}{2}\\ \eta \end{array}\right)={\mathit{\delta }}_2\hfill \end{array}\right.,$$

(56)

where $({\mathit{\delta }}_2,{\Delta }_2)=\phi ({\mathit{\mu }}_2,{\mathsf{\Sigma }}_2)$.

The above non-linear system has five equations and five variables $(d_{11},d_{22},\eta ,x,b)$ and can be solved by an iterative method. With the solution of this system, we can determine the geodesic curve connecting the distributions ${\mathit{\theta }}_1$ and ${\mathit{\theta }}_2$. Moreover, by Equation (16), the Fisher–Rao distance is:

$$d_F({\mathit{\theta }}_1,{\mathit{\theta }}_2)=2d_F({\mathit{\theta }}_0,{\mathit{\theta }}_2)=2d_F((\mathbf{0},I_n),(\mathit{\delta }\left(1\right),\Delta \left(1\right)))=2\sqrt{\frac{1}{2}tr\left(B^2\right)+{\mathit{x}}^t\mathit{x}}=2\sqrt{b^2+x^2}.$$

(57)

We also remark that the curve of the means $\mathit{\delta }\left(t\right)$ (and therefore, $\mathit{\mu }\left(t\right)$) satisfies the equation of a hyperbola; in fact:

$$\frac{{(-\frac{bx(cosh\left(t\sqrt{b^2+2x^2}\right)-1)}{b^2+2x^2}-\frac{bx}{b^2+2x^2})}^2}{{\left(\frac{bx}{b^2+2x^2}\right)}^2}-\frac{{\left(\frac{xsinh\left(t\sqrt{b^2+2x^2}\right)}{\sqrt{b^2+2x^2}}\right)}^2}{{\left(\frac{x}{\sqrt{b^2+2x^2}}\right)}^2}=1.$$

(58)

Summarizing the above discussion, we have:

Proposition 4.

(i) 

Expression (57) provides a closed form for the Fisher–Rao distance between two mirrored bivariate normal distributions, based on the solutions of the non-linear system (56).

(ii) 

The plane curve given by the coordinates of the mean vector in the geodesic connecting two of these distributions is a hyperbola.

Table 2. A time comparison between the numerical method proposed here and the geodesic shooting to calculate the distance between two mirrored distributions.

$\mathit{\mu }$	${\mathit{d}}_{\mathit{F}}({\mathit{\theta }}_1,{\mathit{\theta }}_2)$	Time Systems (s)	Time G.Shooting (s)
1	2.77395	0.046875	4.70313
2	3.67027	0.046875	5.60938
3	4.52933	0.0625	7.10938
4	5.26093	0.078125	9.17188
5	5.87480	0.046875	12.5313
6	6.39439	0.0625	18.4219
7	6.84043	0.078125	492.563
8	7.22903	0.0625	574.422
9	7.57221	0.046875	917.859
10	7.87896	0.046875	1007.13

shows a time comparison between the numerical method proposed here and the geodesic shooting to obtain the Fisher–Rao distance. The distributions used in this experiment were:

$${\mathit{\theta }}_1=\left(\left(\begin{array}{c}-\mu \\ 0\end{array}\right),\left(\begin{array}{cc}0.55& -0.45\\ -0.45& 0.55\end{array}\right)\right)\mathrm{and}{\mathit{\theta }}_2=\left(\left(\begin{array}{c}\mu \\ 0\end{array}\right)\left(\begin{array}{cc}0.55& 0.45\\ 0.45& 0.55\end{array}\right)\right).$$

(59)

for different values of $\mu$.

The method proposed here uses a non-linear system for the calculus of the Fisher–Rao distance, so it is faster the geodesic shooting algorithm. Furthermore, we remark that for $\mu \ge 7$, the geodesic shooting requires additional adaptation to convergence.

Next, we generalize the results of Proposition 4 to pairs of general multivariate normal mirrored distributions. Without loss of generality, we may assume:

$${\mathit{\theta }}_1=({\mu }_1{\mathit{e}}_1,{\mathsf{\Sigma }}_1)\mathrm{and}{\mathit{\theta }}_2=({\mu }_2{\mathit{e}}_1,{\mathsf{\Sigma }}_2),$$

(60)

with ${\mathsf{\Sigma }}_2=M_1{\mathsf{\Sigma }}_1{M}_1$ as in (39), that is:

$${\mathsf{\Sigma }}_2=\left\{\begin{array}{cc}\hfill {\widehat{\sigma }}_{1j}=& {\widehat{\sigma }}_{j1}=-{\sigma }_{1j},j=2,\dots ,n\hfill \\ \hfill {\widehat{\sigma }}_{ij}=& {\sigma }_{ij},\hfill & \hfill \mathrm{otherwise}.\end{array}\right..$$

Proposition 5.

The Fisher–Rao distance between a pair of multivariate mirrored normal distributions ${\mathit{\theta }}_1$ and ${\mathit{\theta }}_2$ (65) is:

$$d_F({\mathit{\theta }}_1,{\mathit{\theta }}_2)=2\sqrt{{\displaystyle \sum _{l=1}^{n-1}}{b}_l^2+x^2},$$

(61)

where:

$$(\mathit{x},B)=\left(\left(\begin{array}{c}x\\ 0\\ ⋮\\ 0\end{array}\right),\left(\begin{array}{cccc}0& b_1& \cdots & b_{n-1}\\ b_1& 0& \cdots & 0\\ ⋮& ⋮& \ddots & ⋮\\ b_{n-1}& 0& \cdots & 0\end{array}\right)\right).$$

(62)

The values x and $b_l$, the non-zero entries of $(\mathit{x},B)$, are obtained by the solution of the $\left(n+\frac{n(n+1)}{2}\right)$ order non-linear system:

$$\left\{\begin{array}{c}{L}^{-1}\Delta \left(1\right)L^{-1}={\Delta }_2\hfill \\ L^{-1}\mathit{\delta }\left(1\right)+{\Delta }_2{\mathit{\mu }}_{1/2}={\mathit{\delta }}_2\hfill \end{array}\right.,$$

(63)

where ${\mathit{\mu }}_{1/2}={\left(\frac{{\mu }_1+{\mu }_2}{2},{\eta }_1,\dots ,{\eta }_{n-1}\right)}^t$, L is the Cholesky factor of the matrix ${\mathsf{\Sigma }}_{1/2}=\left(\begin{array}{cc}{d}_{11}& {\mathbf{0}}^t\\ \mathbf{0}& D\end{array}\right)$, with D a symmetric $(n-1)$ order matrix, $({\mathit{\delta }}_2,{\Delta }_2)=\phi ({\mathit{\mu }}_2,{\mathsf{\Sigma }}_2)$, and $\left(\mathit{\delta }\right(t),\Delta (t\left)\right)$ is the geodesic curve with initial value $(\mathit{\delta }\left(0\right),\Delta \left(0\right))={\mathit{\theta }}_0$ and tangent vector $(\mathit{x},B)$ given in Equation (15).

Let $\gamma \left(t\right)=\left(\mathit{\mu }\right(t),\mathsf{\Sigma }(t\left)\right)$, $-1\le t\le 1$, be the geodesic curve in $\mathcal{M}$ connecting ${\mathit{\theta }}_1$ and ${\mathit{\theta }}_2$. The proof is similar to the bivariate case, by considering $\gamma \left(0\right)=({\mathit{\mu }}_{1/2},{\mathsf{\Sigma }}_{1/2})$,

$${\gamma }^{\prime }\left(0\right)={\widehat{\mathit{\theta }}}_{1/2}=({\widehat{\mathit{\mu }}}_{1/2},{\widehat{\mathsf{\Sigma }}}_{1/2}))=\left(\left(\begin{array}{c}\widehat{\mu }\\ 0\\ ⋮\\ 0\end{array}\right),\left(\begin{array}{cccc}0& {\widehat{\sigma }}_{12}& \cdots & {\widehat{\sigma }}_{1n}\\ {\widehat{\sigma }}_{12}& 0& \cdots & 0\\ ⋮& ⋮& \ddots & ⋮\\ {\widehat{\sigma }}_{1n}& 0& \cdots & 0\end{array}\right)\right)$$

(64)

and $\overline{\gamma }\left(t\right)=\psi \left(\gamma \left(t\right)\right)$ where $\psi ={\psi }_{\left(-L^{-1}{\mathit{\mu }}_{1/2}{L}^{-1}\right)}$.

$${\mathit{\theta }}_1=({\mu }_1{\mathit{e}}_1,{\mathsf{\Sigma }}_1)\mathrm{and}{\mathit{\theta }}_2=({\mu }_2{\mathit{e}}_1,{\mathsf{\Sigma }}_2),$$

(65)

with ${\mathsf{\Sigma }}_2=M_1{\mathsf{\Sigma }}_1{M}_1$.

Table 3. Closed forms for the Fisher–Rao distance in submanifolds of $\mathcal{M}$ and the distance in $\mathcal{M}$ between pairs of special distributions.

Distance in Non-totally Geodesic Submanifolds
Submanifold	Distance
${\displaystyle \begin{array}{ccc}\hfill {\mathcal{M}}_{\mathsf{\Sigma }}& =& \left\{\begin{array}{c}{p}_{\mathit{\theta }};\mathit{\theta }=(\mathit{\mu },\mathsf{\Sigma }),\\ \mathsf{\Sigma }={\mathsf{\Sigma }}_0\in P_n\left(\mathbb{R}\right)\mathrm{constant}\end{array}\right\},\hfill \\ \hfill {\mathit{\theta }}_i& =& ({\mathit{\mu }}_i,{\mathsf{\Sigma }}_0)\hfill \end{array}}$	${\displaystyle d_{\mathsf{\Sigma }}({\mathit{\theta }}_1,{\mathit{\theta }}_2)=\sqrt{{({\mathit{\mu }}_1-{\mathit{\mu }}_2)}^t{\mathsf{\Sigma }}_0^{-1}({\mathit{\mu }}_1-{\mathit{\mu }}_2)}}$
${\displaystyle \begin{array}{ccc}\hfill {\mathcal{M}}_D& =& \left\{\begin{array}{c}{p}_{\mathit{\theta }};\mathit{\theta }=(\mathit{\mu },\mathsf{\Sigma }),\\ \mathsf{\Sigma }=diag({\sigma }_1^2,{\sigma }_2^2,\dots ,{\sigma }_n^2)\end{array}\right\},\hfill \\ \hfill {\mathit{\theta }}_i& =& ({\mu }_{1i},{\sigma }_{1i},{\mu }_{2i},{\sigma }_{2i},\dots ,{\mu }_{ni},{\sigma }_{ni})\hfill \end{array}}$	${\displaystyle d_D({\mathit{\theta }}_1,{\mathit{\theta }}_2)=\sqrt{{\displaystyle \sum _{i=1}^n}{d}_F^2(({\mu }_{1i},{\sigma }_{1i}),({\mu }_{2i},{\sigma }_{2i}))}}$
Distance in Totally Geodesic Submanifolds
${\displaystyle \begin{array}{ccc}\hfill {\mathcal{M}}_{\mathit{\mu }}& =& \left\{\begin{array}{c}{p}_{\theta };\mathit{\theta }=(\mathit{\mu },\mathsf{\Sigma }),\\ \mathit{\mu }={\mathit{\mu }}_0\in {\mathbb{R}}^n\mathrm{contant}\end{array}\right\},\hfill \\ \hfill {\mathit{\theta }}_i& =& ({\mathit{\mu }}_0,{\mathsf{\Sigma }}_i)\hfill \end{array}}$	${\displaystyle d_F({\mathit{\theta }}_1,{\mathit{\theta }}_2)=\sqrt{\frac{1}{2}{\displaystyle \sum _{i=1}^n}{[log\left({\lambda }_i\right)]}^2}}$, where ${\lambda }_i$ are the eigenvalues of ${\displaystyle {\mathsf{\Sigma }}_1^{-1/2}{\mathsf{\Sigma }}_2{\mathsf{\Sigma }}_1^{-1/2}}$
${\displaystyle \begin{array}{ccc}\hfill {\mathcal{M}}_{D\mathit{\mu }}& =& \left\{\begin{array}{c}\{p_{\mathit{\theta }};\mathit{\theta }=(\mathit{\mu },\mathsf{\Sigma }),\\ \mathit{\mu }\mathrm{is}\mathrm{an}\mathrm{eigenvector}\mathrm{of}\\ \mathsf{\Sigma }=diag({\sigma }_1^2,{\sigma }_2^2,\cdots ,{\sigma }_n^2)\}\end{array}\right\},\hfill \\ \hfill {\mathit{\theta }}_i& =& ({\mu }_{1i},{\sigma }_{1i},{\sigma }_{2i},\dots ,{\sigma }_{ni})\hfill \end{array}}$	${\displaystyle d_{D\mathit{\mu }}({\mathit{\theta }}_1,{\mathit{\theta }}_2)=(d_F^2\left(\left({\mu }_{11},{\sigma }_{11}\right),\left({\mu }_{21},{\sigma }_{21}\right)\right)+}$ ${\displaystyle {\displaystyle \sum _{i=2}^n}{d}_F^2\left(\left(0,{\sigma }_{1i}\right),\left(0,{\sigma }_{2i}\right)\right){)}^{1/2}}$
Distance Between Special Distributions in $\mathcal{M}$
Distributions with Common Covariance Matrices, ${\mathit{\theta }}_i=({\mathit{\mu }}_i,{\mathsf{\Sigma }}_0)$	${\displaystyle d_F({\mathit{\theta }}_1,{\mathit{\theta }}_2)=d_{D\mathit{\mu }}\left((\mathbf{0},D),\left(\right|{\mathit{\mu }}_2-{\mathit{\mu }}_1|{\mathit{e}}_1,D)\right)}$, where P is an orthogonal matrix such that $P({\mathit{\mu }}_2-{\mathit{\mu }}_1)=|{\mathit{\mu }}_2-{\mathit{\mu }}_1|{\mathit{e}}_1$ and $P\mathsf{\Sigma }{P}^t=UD{U}^t$
Mirrored Distributions, ${\mathit{\theta }}_1=({\mu }_1{\mathit{e}}_1,{\mathsf{\Sigma }}_1)$ and ${\mathit{\theta }}_2=({\mu }_2{\mathit{e}}_1,{\mathsf{\Sigma }}_2)$, with ${\mathsf{\Sigma }}_2=M_1{\mathsf{\Sigma }}_1{M}_1$	${\displaystyle d_F({\mathit{\theta }}_1,{\mathit{\theta }}_2)=2\sqrt{{\displaystyle \sum _{l=1}^{n-1}}{b}_l^2+x^2}}$, where x and $b_i$ are obtained by the solution of Equation (63)

collects the results in Section 2.1 and the new results of this section.

4. Hierarchical Clustering for Diagonal Gaussian Mixture Simplification

A parameterized Gaussian mixture model f is a weighted sum of m multivariate normal distributions, that is,

$$f\left(\mathit{x}\right)={\displaystyle \sum _{i=1}^m}{w}_i{p}_i(\mathit{x};{\mathit{\mu }}_i,{\mathsf{\Sigma }}_i),$$

where $\mathit{x}\in {\mathbb{R}}^n$, $p_i(\mathit{x};{\mathit{\mu }}_i,{\mathsf{\Sigma }}_i)$, $i=1,\cdots ,m$, are normal distributions and $w_i$, $i=1,\cdots ,m$, are mixture, ${\sum }_{i=1}^m{w}_i=1$. In this paper, we call the diagonal Gaussian mixture model (DGMM) the mixture composed only by distributions with diagonal covariance matrices.

Gaussian mixture models (GMM) are used in modeling datasets: image processing, signal processing, and density estimation problems [46,47,48]. In many applications involving mixture models, the computational requirements are of a very high level due to the large number of mixture components. This can be handled if we reduce the number of components of the mixture: given a mixture f of m components, we want to find a mixture g of l components, $1\le l<m$, such that g is a good approximation of f with respect to a similarity measure [49]. Gaussian mixture simplification was considered in statistical inference in [50] and to decode low-density lattice codes [51].

In [49] was proposed a hierarchical clustering algorithm to simplify an exponential family mixture model based on Bregman divergences. This section describes an agglomerative hierarchical clustering method based on the Fisher–Rao distance in the submanifold ${\mathcal{M}}_D$ (21) to simplify DGMM, and we present an application to image segmentation, complementing what was developed in [52]. We start by introducing the concept of the centroid for a set of distributions in ${\mathcal{M}}_D$.

4.1. Centroids in the Submanifold ${\mathcal{M}}_D$

In [53], Galperin described centroids in the two-dimensional Minkowski model, which can be translated also to the Klein disk and Poincare half-plane models. Given a set of points ${\mathit{q}}_i=(x_{{\mathit{q}}_i},y_{{\mathit{q}}_i},z_{{\mathit{q}}_i})$ in the Minkowski model, with associated weights $u_i$, the centroid is computed and normalized as:

$${\mathit{c}}^{\prime }=\sum _i{u}_i{\mathit{q}}_i\mathrm{and}\mathit{c}=\frac{{\mathit{c}}^{\prime }}{-x_{{\mathit{c}}^{\prime }}^2-y_{{\mathit{c}}^{\prime }}^2+z_{{\mathit{c}}^{\prime }}^2}.$$

(66)

To calculate the centroid $\mathit{c}$ of a subset of points $\mathcal{C}=\left\{(w_i,{\mathit{\theta }}_i)\right\}$, ${\mathit{\theta }}_i=({\mu }_i,{\sigma }_i)$, the isometries presented in [25] and the relation between the media × standard deviation plane of parameters of univariate normal distributions and the Poincare half-plane given in [22] are used.

Given a dataset $\mathcal{C}=\left\{(w_i,{\mathit{\theta }}_i)\right\}$, where ${\mathit{\theta }}_i=({\mu }_{1i},{\sigma }_{1i},\cdots ,{\mu }_{ni},{\sigma }_{ni})$ are distributions in ${\mathcal{M}}_D$, the centroid of $\mathcal{C}$ is:

$$\mathit{c}:=({\mathit{c}}_1,\cdots ,{\mathit{c}}_n),$$

(67)

where ${\mathit{c}}_j$, $j=1,\cdots ,n$, is the centroid of ${\mathcal{C}}_j=\left\{(w_j,({\mu }_{ji},{\sigma }_{ji}))\right\}$ given in Equation (66).

4.2. Hierarchical Clustering Algorithm

Let a DGMM f with parameters $\mathcal{C}=\{(w_1,{\mathit{\theta }}_1),\cdots ,(w_m,{\mathit{\theta }}_m)\}$.

In order to apply the hierarchical clustering algorithm, we need to consider the distance between two subsets A and B. The three most common distances are called linkage criteria and are given by [54]:

• Single linkage:

  $$D(A,B)=min\{d_D(a,b);a\in A,b\in B\};$$

  (68)
• Complete linkage:

  $$D(A,B)=max\{d_D(a,b);a\in A,b\in B\};$$

  (69)
• Group average linkage:

  $$D(A,B)=\frac{1}{\left|A\right|\left|B\right|}\sum _{a\in A}\sum _{b\in B}{d}_D(a,b),$$

  (70)

  where $d_D$ is the distance in the submanifold ${\mathcal{M}}_D$ and $\left|X\right|$ is the number of elements of a set X.

A summary of the hierarchical clustering algorithm (Algorithm 1) [49] using one of these distances is given next.   

Algorithm 1: Hierarchical Clustering Algorithm

1: Form m clusters ${\mathcal{C}}_j=\left\{(w_j,{\mathit{\theta }}_j)\right\}$ with one element. 2: Find the two closest clusters, ${\mathcal{C}}_i$ and ${\mathcal{C}}_j$, with respect to a distance D, and merge them into asingle cluster ${\mathcal{C}}_i\cup {\mathcal{C}}_j$. 3: Compute distances between the new cluster and each of the old clusters. 4: Repeat Steps 2 and 3 until all items are clustered into a single cluster of size n.

The simplified DGMM:

$$g=\sum _{j=1}^l{\beta }_j{g}_j$$

of l components is built from the l subsets ${\mathcal{C}}_1$, …, ${\mathcal{C}}_l$ remaining after the iteration $n-l$ of the hierarchical clustering algorithm. In this work, we choose the parameters of $g_j$ in two ways: as the centroid in the submanifold ${\mathcal{M}}_D$ (Fisher–Rao hierarchical clustering) and as the Bregman left-sided centroid [49] (Bregman–Fisher–Rao hierarchical clustering) of the subset ${\mathcal{C}}_j$ with weights ${\displaystyle {\beta }_j=\sum _{(w_i,{\mathit{\theta }}_i)\in {\mathcal{C}}_j}{w}_i}$.

As remarked in [49], the hierarchical clustering algorithm allows introducing a method to learn the optimal number of components in the simplified mixture g. Thus, g must be as compact as possible and reach a minimum prescribed quality $d_{KL}\left(f\right|\left|g\right)\le \tau$, where $d_{KL}\left(f\right|\left|g\right)$ is the Kullback–Leibler divergence.

4.3. Experiments in Image Segmentation

We can apply the Fisher–Rao and the Bregman–Fisher–Rao hierarchical clusterings to simplify a mixture of exponential families in the context of clustering-based image segmentation as was done in [49] for the Bregman hierarchical clustering. Given an input color image I, we adapt the Bregman soft clustering algorithm to generate a DGMM f of 32 components, which models the image pixels. We point out that the restriction considered in this paper (only DGMM) is also used in many applications due its much lower computational cost. We consider here a pixel $\mathit{\rho }=({\rho }_R,{\rho }_G,{\rho }_B)$ as a point in ${\mathbb{R}}^3$, where ${\rho }_R$, ${\rho }_G$, and ${\rho }_B$ are the RGB color information. For image segmentation, we can say that the image pixel $\rho$ belongs to the class ${\mathcal{C}}_j$ when:

$$p_j(\mathit{\rho };{\mu }_j,{\mathsf{\Sigma }}_j)>p_i(\mathit{\rho };{\mu }_i,{\mathsf{\Sigma }}_i),\forall i\in \{1,\cdots ,m\}\setminus \left\{j\right\}.$$

Thus, the segmented image is illustrated by replacing the color value of the pixel $\mathit{\rho }$ by the mean ${\mu }_j$ of the Gaussian $p_j$.

Using the the Fisher–Rao and the Bregman–Fisher–Rao hierarchical clusterings, we simplify the mixture f into mixtures g of l components with $l=\{2,4,8,16\}$. Each mixture gives one image segmentation. The linkage criterion used here was the complete linkage (68), which has presented better results in our simulations. Figure 8 shows the segmentation of the Baboon, Lena, and Clown input images given by the Bregman–Fisher–Rao hierarchical clustering. The number of colors in each image is equal to the number of components in the simplified mixture g.

The quality of the segmentation was analyzed as a function of l through the Kullback–Leibler divergence estimated by the Monte Carlo method, since there was no closed form for this measure (five thousand points were randomly drawn to estimate $d_{KL}\left(f\right|\left|g\right)$). Figure 9, Figure 10 and Figure 11 show the evolution of the simplification quality as a function of the number of components l for the Baboon, Lena, and Clown images, using the Bregman, the Fisher–Rao, and the Bregman–Fisher–Rao hierarchical clustering algorithms. We observed that the image quality increased ($d_{KL}\left(f\right|\left|g\right)$ decreased) with l, as expected, and the behavior was similar in all clustering algorithms. In general, the Bregman–Fisher–Rao hierarchical clustering algorithm presented better results. Considering the constraint $\tau =0.2$, the learning process provided, for the Bregman–Fisher–Rao hierarchical clustering, mixtures of 19, 21, and 21 as optimal simplifications for the images of the Baboon, Lena, and Clown, respectively.

5. Concluding Remarks

The Fisher–Rao distance was approached here in the space of multivariate normal distributions. Initially, as in [38], we summarized some known closed forms for this distance in submanifolds of this model and some bounds for the general case. A closed form for the Fisher–Rao distance between distributions with the same covariance matrix was obtained in Proposition 3, and we also have derived a non-linear system characterizing the distance between two distributions with mirrored covariance matrices in Proposition 5. Some perspectives for future research related to this topic include deriving new bounds for the Fisher–Rao distance in the general case, by using these special distributions, to characterize as non-linear systems the distances between other types of distributions and to extend the closed forms and bounds presented here to the space of elliptical distributions. Finally, we have extended the analysis of the Bregman–Fisher–Rao hierarchical clustering algorithm to simplify Gaussian mixtures in the context of clustering-based image segmentation given in [52] with comparative results that encourage the use of the Fisher–Rao distance in other clustering or classification algorithms.