A Unified Approach to Aitchison’s, Dually Affine, and Transport Geometries of the Probability Simplex

Abstract

A critical processing step for AI algorithms is mapping the raw data to a landscape where the similarity of two data points is conveniently defined. Frequently, when the data points are compositions of probability functions, the similarity is reduced to affine geometric concepts; the basic notion is that of the straight line connecting two data points, defined as a zero-acceleration line segment. This paper provides an axiomatic presentation of the probability simplex’s most commonly used affine geometries. One result is a coherent presentation of gradient flow in Aichinson’s compositional data, Amari’s information geometry, the Kantorivich distance, and the Lagrangian optimization of the probability simplex.

1. Introduction

This paper discusses the geometry of the set of all probability measures whose support is a given finite set. The sample space $\Omega$ is the set of possible outcomes of the experiment of interest. While the use of a geometric language is well established in statistics, the formalization of a specific affine structure is due to the work of Amari, particularly in [1,2]. Amari defines a dually affine atlas of charts in the precise sense to be described below. This structure is fascinating because it gives an intrinsically geometric meaning to classical Fischerian objects such as log-likelihood, Fisher’s score, and Fisher’s information. This paper is based on a non-parametric representation of Amari’s seminal work, which was introduced in a series of papers, starting with [3].

1.1. Probability Simplex

On the finite sample space $\Omega$, a probability is characterized by its probability function $q\in {\mathbb{R}}^{\Omega }$, that is, the mapping $\Omega \to {\mathbb{R}}_{\ge 0}$ (the set of non-negative real numbers) in which the sum of the probabilities over all outcomes equals 1. Hence, the set of all probability functions, the probability simplex, is

$$\mathcal{P}\left(\Omega \right)=\left\{q:\Omega \to {\mathbb{R}}_{\ge 0}|\sum _{x\in \Omega }q\left(x\right)=1\right\}.$$

(1)

The probability simplex is a convex set, which means it contains all mixtures. Given two probability functions, $p,q\in \mathcal{P}\left(\Omega \right)$, the line segment joining p and q,

$$[0,1]\ni t\mapsto p\left(t\right)=(1-t)p+tq,$$

(2)

lies entirely within $\mathcal{P}\left(\Omega \right)$.

1.2. Compositions

According to [4,5], a composition, f, in the sample space $\Omega$ is a positive function of $\Omega$, $f:\Omega \to {\mathbb{R}}_{\ge 0}$. Two compositions, f and g, are equivalent if there exists a constant, $k\in {\mathbb{R}}_{>0}$, such that $g=kf$. Each composition, f, is normalized to a probability function via

$${\mathbb{R}}_{\ge 0}^{\Omega }\ni f\mapsto {\displaystyle \frac{f}{{\sum }_{x\in \Omega }f\left(x\right)}}\in \mathcal{P}\left(\Omega \right).$$

(3)

The above equation holds under the condition that $f\ne 0$ on $\Omega$, as normalization requires a non-zero sum of $f^{\prime }s$ values. If $f=0$ on $\Omega$, normalization would not be possible. Two compositions are equivalent if and only if the normalized compositions are equal so that the set of equivalent compositions is the same as the probability simplex. However, the notion of the composition suggests a geometric structure other than the mixture of Equation (2). Given compositions f and g, the product $h=f\circ g$ is a composition, and the proportionality relation holds. If $f_1=k_{1}f$ and $g_1=k_{2}g$, then $f_1{g}_1=\left(k_1{k}_2\right)fg$. Similarly, for the exponentiation $(\alpha ,f)\mapsto f^{\alpha }$, $\alpha \in \mathbb{R}$.

These two operations provide a way of connecting two probability functions with the so-called exponential arc,

$$[0,1]\ni t\mapsto {\displaystyle \frac{p^{1-t}{q}^t}{{\sum }_{x\in \Omega }p\left(x\right)q\left(x\right)}}\in \mathcal{P}\left(\Omega \right).$$

(4)

The set of all compositions with the operations we have just described is sometimes called Bayes space [6] because the Bayes formula is possibly the most important example of a normalized product of compositions.

The mixture (2) and the exponential arc (4) will be central objects in the geometric presentation of the probability simplex.

1.3. Polytopes and Simplexes

We need to review some basic facts of convex analysis; see, for example, [7,8]. A set, S, in a Euclidean space ${\mathbb{R}}^n$ is convex if, for any two points, $p,q\in S$, the line segment joining p and q of Equation (2) lies entirely within S. The convex hull of a set, $P\in {\mathbb{R}}^n$, is the set of all convex combinations of points in P,

$$conv\left(P\right)=\left\{\sum _{i=1}^k{\lambda }_i{p}_i|k\ge 1,p_i\in P,{\lambda }_i\ge 0,\sum _{i=1}^k{\lambda }_i=1\right\},$$

where the non-negative real numbers ${\lambda }_i$ are the weights or coefficients of the convex combination, and $p_i$ are points in P. In particular, a polytope is the convex hull of a finite set of points. A minimal set of such points is the set of vertices. The probability simplex on $\Omega$ (also called the standard simplex in other fields) is the polytope whose vertices are the delta probability functions ${\delta }_a\left(x\right)=(x=a)$ for all $a\in \Omega$; that is,

$${\delta }_a\left(x\right)=\left\{\begin{array}{cc}1\hfill & \mathrm{if}x=a,\hfill \\ 0\hfill & \mathrm{otherwise}.\hfill \end{array}\right.$$

(5)

Each ${\delta }_a$ represents a probability for which the entire probability mass is concentrated at a.

An affine combination of the vectors $p_1,\dots ,p_k\in {\mathbb{R}}^{\Omega }$ is a vector of the form ${\sum }_{j=1}^k{t}_j{p}_j$, with ${\sum }_{j=1}^k{t}_j=1$. The given vectors are affinely independent if no affine combination is zero. A simplex is a polytope generated via affinely independent points $a_1,\dots ,a_n$. All simplexes with the same number of vertices can be transformed into one another through a suitable affine function. Thus, any n-vertex simplex in any n-dimensional space is essentially the same shape, differing only by scaling, rotation, and translation.

The probability simplex is a closed set in its ambient Euclidean space. The topological border is a union of simplexes of smaller dimensions, each one corresponding to a reduced support of the probability functions. In statistical modeling, one usually assumes that the support of each probability function in the model of interest is the same. For this reason, we will mainly consider the open probability simplex ${\mathcal{P}}_{>}\left(\Omega \right)$, which is the set of strictly positive probability functions.

1.4. Overview

This piece of research presents, in a coordinated and principled way, some basic topics in the geometrical calculus of the probability simplex. The specific topics are Aitchison’s theory of compositional data and Bayes spaces, Amari’s dually affine information geometry, and transport theory in product sample spaces. We define a differential geometric structure with an explicitly defined expression of the tangent bundle and discuss the relevant calculus operations, especially the notions of a natural gradient and gradient flow.

Section 2 is devoted to the definition of affine space as a set, M, endowed with a displacement function that maps every couple of points to a vector in such a way that the rule of addition of vectors holds. An affine space is a differentiable manifold for which the displacement function with the first point fixed provides a chart from $\mathcal{M}$ to a vector space. The Aitchison geometry of compositional data is an instance of such a geometry.

Section 3 presents a generalization of the notion of affine space to a more complex object, which allows us to set information geometry within this geometric framework. In affine spaces, the tangent space is the same at each point. However, the classical Fisher’s methodology, together with Amari’s methodology, suggests taking the Fisher’s score as a definition of the tangent vector. As a consequence, it is convenient to assume that the primary expression of the tangent bundle is the set of weighted contrasts. This natural setting requires an extra structure, that is, an identification between tangent spaces, called a parallel transport. This section contains the main computational results, particularly the notions of natural gradient, duality of transports, and gradient flow.

Section 4 presents the analysis of the affine statistical bundle when the sample space is of product type. In this case, we introduce the notions of ANOVA decomposition and an optimal transport plan.

Section 5 deals with the second-order geometric analysis of the probability simplex, that is, the first-order geometric analysis of the statistical bundle. It is well known from differential geometry that second-order objects, such as accelerations, are not uniquely defined from the given atlas of charts on the base space. In our case, we have charts defined on a vector bundle expressing the probability functions and Fisher’s scores, which allows for statistically natural definitions of accelerations. The availability of second-order objects prompts the development of results inspired by notions of mechanics, such as the calculus of Lagrangian functions. It should be noted that the use of a language inspired by physics here does not refer to the push-up of the probability simplex of a dynamic on the sample space; rather, it refers to an irreducible dynamic of the probability simplex.

2. Affine Geometry of the Probability Simplex

The geometries we are going to discuss are supported on the probability simplex $\mathcal{P}\left(\Omega \right)$ or on the open probability simplex ${\mathcal{P}}_{>}\left(\Omega \right)$ of a finite sample space $\Omega$. The latter open convex set is the interior of the former full probability simplex (1). The term “geometry” is ambiguous and should be qualified by stating which type of geometry we are talking about. Actually, the field of statistics uses many types of geometries, such as linear, affine, metric, geodesic, Riemannian, and differential geometries. We discuss here affine geometry, to be precisely defined below. Our general references are the monographs [2,9,10] (Ch. I–II). The tutorials [11,12] present similar material as the present one but in a more restricted scope.

Affine geometry is a special instance of differential geometry. The former inherits the language of the latter: chart, atlas, expression, tangent space, and bundle…We refer to the non-parametric version of differential geometry as presented in [13,14]. Below, we list the basic concepts.

Given a set, M, and a real vector space, V, a chart is a one-to-one mapping, $s:U\to V$, where $U\subseteq M$, and $s\left(U\right)$ is open in V. Two charts, $s_1:U_1\to V_1$ and $s_2:U_2\to V_2$, are $C^k$-compatible if the change in chart mapping

$$s_2\circ s_1^{-1}:s_1(U_1\cap U_2)\to s_2(U_1\cap U_2)$$

is either empty or a $C^k$-diffeomorphism of open sets. (As usual, ∘ stands for the composition of functions.) A family of two by two $C^k$-compatible charts is a $C^k$-atlas. Two atlases are compatible if their union is an atlas. A maximal $C^k$-atlas is a $C^k$-manifold.

A curve is a map from a real interval to the manifold. We assume, unless otherwise stated, that the interval is an open neighborhood of 0. Given a curve, $\gamma :t\mapsto \gamma \left(t\right)\in M$, and a chart, s, in the atlas, the curve $s\circ \gamma :t\mapsto s\left(\gamma \right(t\left)\right)\in V$ is the expression of the curve in the chart s. If one expression (hence, all) is differentiable, then $t\mapsto {\displaystyle \frac{d}{dt}}s\circ \gamma \left(t\right)\in V$ is the expression of the velocity at t of the curve $\gamma$ in the chart s. The vector space of all possible expressions of velocities for curves passing through a given $p\in M$ is the expression of the tangent space at p.

2.1. Affine Space

The open probability simplex ${\mathcal{P}}_{>}\left(\Omega \right)$ and the probability simplex are, respectively, an open convex subset and a closed convex subset of the generated affine plane,

$$A(\Omega )=\left\{v\in {\mathbb{R}}^{\Omega }|\sum _{x\in \Omega }v\left(x\right)=1\right\},$$

(6)

where $A(\Omega )$ consists of all vectors in the vector space ${\mathbb{R}}^{\Omega }$ whose components sum to 1. Probability functions are non-negative and sum to 1. As $A(\Omega )$ is an Euclidean space, the open probability simplex inherits Euclidean geometry. In some contexts, it is useful to relax the non-negativity constraint and allow the “probabilities” to take on negative values while maintaining the sum-to-one condition. By allowing $v\left(x\right)$ to be negative, we extend our consideration from the standard probability simplex to a larger space.

Example 1.

Consider a simple three-dimensional space with outcomes $\Omega =\{1,2,3\}$. A probability vector $v\in {\mathbb{R}}^3$ in this space might look like $v=(v_1,v_2,v_3)$, where $v_1+v_2+v_3=1$. For standard probabilities, we require $v_1,v_2,v_3\ge 0$. However, if we relax this constraint, $v_1,v_2,v_3$ can assume negative values as long as they sum to 1. In the plot of Figure 1, the blue triangle represents the standard probability simplex where $v_1,v_2,v_3\ge 0$. The red points represent example vectors in the generalized affine space, including one where $v_3$ is negative and another where $v_1$ exceeds 1.

Example 2.

A score function, S, can be defined on the space of these generalized probability vectors. For a given vector, v, and an outcome, x, the score $S(v,x)$ quantifies how well the vector v predicts the outcome x. A common example of a score function is the Brier score [15], defined as follows:

$$S(v,x)=\sum _{y\in \Omega }{(v\left(y\right)-{\delta }_x\left(y\right))}^2$$

where ${\delta }_x$ is the indicator function of x, that is, 1 if $y=x$ and 0 otherwise.

The following definition, due to [16], extends the notion of geometry on a hyperplane to a more general situation.

Definition 1

(Weyl’s affine space). Given a set, M, and a real finite-dimensional vector space, V, an affine space, $(M,V,\overrightarrow{··})$, consists of displacement mapping,

$$M\times M\ni (p,q)\mapsto \overrightarrow{pq}\in V,$$

such that the following two assumptions hold.

1. 

For each point $p\in M$, the partial mapping $s_p:M\ni q\mapsto s_p\left(q\right)=\overrightarrow{pq}$ is injective (one-to-one) with the open image $s_p\left(M\right)$.

2. 

Given points p, q, and r, the displacement vectors satisfy the parallelogram law,

$$\overrightarrow{pq}+\overrightarrow{qr}=\overrightarrow{pr}.$$

Property 1 ensures that, for a fixed point, p, the mapping from points in M to the vector space, V, via the displacement vector $\overrightarrow{pq}$ does not map different points to the same vector. In the language of differential geometry, the mapping $s_p$ is a chart, and $v=s_p\left(q\right)$ is the coordinate of q in the p-chart. As $s_p\left(p\right)=0$, we will say that p is the origin of the chart. The inverse chart, also called a patch, is $s_p^{-1}:s_p\left(M\right)\to M$.

Property 2 means that the vector displacement from p to r is the same as the sum of the vector displacement from p to q and from q to r; see Figure 2. This property ensures that adding displacements is consistent with the geometric intuition of moving along a path from one point to another via an intermediate point. In terms of the charts,

$$s_p\left(q\right)+s_q\left(r\right)=s_p\left(r\right).$$

(7)

In particular, we have $s_p\left(p\right)=\overrightarrow{pp}=0$; hence $s_p$ is the chart centered at p. Also, $s_p\left(q\right)+s_q\left(p\right)=0$. The change-of-chart transformation from origin p to origin q follows from Equation (7).

$$s_q\circ s_p^{-1}\left(v\right)=s_q\left(s_p^{-1}\left(v\right)\right)=s_q\left(p\right)+s_p\left(s_p^{-1}\left(v\right)\right)=s_q\left(p\right)+v.$$

(8)

The change-of-chart from origin p to origin q is the translation via $s_q\left(p\right)=-s_p\left(q\right)$.

2.2. Affine Manifold

In the affine space $(M,V,\overrightarrow{··})$, the displacement $(p,q)\mapsto \overrightarrow{pq}$ defines a system of charts, $s_p:M\ni q\mapsto \overrightarrow{pq}$, as p varies in $\mathcal{M}$. Each $s_p$ is one-to-one, and the change-of-charts (8) are translations; hence, they are highly smooth. The affine manifold is defined by the atlas of charts $s_p:M\to V$, $p\in M$. The affine manifold differs from the $C^k$-manifold that it generates because the former is not maximal, while the latter is. Any vector basis of V provides a version of the affine space, where the vector space is a real vector space of parameters, producing a faithful parametrization of M.

Consider a curve, $t\mapsto \rho \left(t\right)\in M$, in the affine manifold. For any reference point $p\in M$, the curve $t\mapsto s_p\left(\rho \left(t\right)\right)$ is the $s_p$-expression of the curve. The derivative ${\displaystyle \frac{d}{dt}}{s}_p\left(\rho \left(t\right)\right)$ does not depend on the choice of the point p:

$${\displaystyle \frac{d}{dt}}{s}_p\left(\rho \left(t\right)\right)={\displaystyle \frac{d}{dt}}\left(s_p\left(q\right)+s_q\left(\rho \left(t\right)\right)\right)={\displaystyle \frac{d}{dt}}{s}_q\left(\rho \left(t\right)\right)$$

Thus, the derivative ${\displaystyle \frac{d}{dt}{s}_p\left(\rho \left(t\right)\right)}$ is independent of the choice of p and is called the affine velocity of the curve. It is denoted as

$${\displaystyle \frac{d}{dt}}\rho \left(t\right)\mathrm{or}\stackrel{★}{q}\left(t\right).$$

The vector space of all velocities evaluated at $t=0$ of all curves $t\mapsto \rho \left(t\right)\in M$, such that $\rho \left(0\right)=p$, is the tangent space at p of the affine manifold, namely

$$T_p=\left\{\stackrel{★}{q}\left(0\right)|t\mapsto q\left(t\right)\in M,q\left(0\right)=p\right\}$$

(9)

Conversely, given any $v\in V$, the curve $t\mapsto s_p^{-1}\left(tv\right)$ has affine velocity v, so that $T_{p}M=V$.

Example 3.

Let us discuss a basic elementary example that is relevant in the statistical literature on the design of experiments and regression modeling. If we take as base set M the hyperplane $A(\Omega )$ of Equation (6) and as vector space the vector space parallel to the probability simplex, that is, the space of contrasts,

$$CO\left(\Omega \right)=\left\{v\in {\mathbb{R}}^{\Omega }|\sum _{x\in \Omega }v\left(x\right)=0\right\},$$

(10)

the displacement is $\overrightarrow{pq}=q-p\in V$. In fact, ${\sum }_x(q\left(x\right)-p\left(x\right))=1-1=0$. And the mapping $s_p\left(q\right)=q-p$ is one-to-one. The parallelogram law is $(q-p)+(r-q)=(r-p)$. The tangent space is defined in Equation (10). In the case with $\Omega =\left\{1,2,3\right\}$, a vector basis of $CO\left(\Omega \right)$ is

$$v_1=(1,-1,0),v_2=(1,0,-1),$$

so that

$$(p,q)\mapsto s_p\left(q\right)=-(q\left(2\right)-p\left(2\right))\left(\begin{array}{c}1\\ -1\\ 0\end{array}\right)-(q\left(3\right)-p\left(3\right))\left(\begin{array}{c}1\\ 0\\ -1\end{array}\right).$$

The chart at p in parametric coordinates is

$$q\mapsto \left(\begin{array}{c}-\left(q\right(2)-p(2\left)\right)\\ -\left(q\right(3)-p(3\left)\right)\end{array}\right)\in {\mathbb{R}}^2.$$

Often, the interpretability of regression models depends on the suitable choice of vector space bases of $CO\left(\Omega \right)$ and charts.

Interestingly, many types of affine manifolds on the open probability simplex are of statistical interest. They all use the space of contrasts (10) as vector space. Various vector bases of $CO\left(\Omega \right)$ are commonly used in applications. One option is to choose a component $a\in \Omega$ and use the reduced basis ${\left({\delta }_x\right)}_{x\ne a}$. With this basis, the components of the projection of $v\in CO\left(\Omega \right)$ are ${(v\left(x\right)-v\left(a\right))}_{x\ne a}$. A second option is the preferred point basis. Fix $a\in \Omega$. If v is a contrast, it holds that $v\left(a\right)=-{\sum }_{x\ne a}v\left(x\right)$; hence $v={\sum }_{x\ne a}v\left(x\right)({\delta }_x-{\delta }_a)$ and ${({\delta }_x-{\delta }_a)}_{x\ne a}$ is a basis.

The space of contrasts is a Euclidean space, for example, with the restriction of the inner product in ${\mathbb{R}}^{\Omega }$. Other inner products can be used. While not part of the affine geometry, these metrics, applied to the vector space of contrasts, provide various useful instances of metric on the affine space; see Example 2. It is important to realize that the choice of an affine displacement and the choice of an inner product are independent modeling choices.

Example 4

(Open probability simplex). The construction in Example 3 is not feasible for the probability simplex because the image of the restricted mapping

$$\mathcal{P}\left(\Omega \right):q\mapsto q-p\in CO\left(\Omega \right)$$

is closed. We can use the same construction as before if we restrict to the open probability simplex,

$${\mathcal{P}}_{>}\left(\Omega \right):q\mapsto q-p\in CO\left(\Omega \right).$$

(11)

Aitchison’s Centered Log Ratio [5]

In this section, a key notion for statistical usage of Aitchison’s geometry, namely the centered log ratio (CLR), is interpreted as a displacement. This allows for the interpretation of the moments of a random variable of the cumulant function in this framework.

If f and $f_1=k_{1}f$ are equivalent positive compositions, then $log{f}_1=logf+log{k}_1$. The logarithms of equivalent compositions differ by an additive constant. This remark suggests using a logarithmic scale by considering the set of positive probability functions ${\mathcal{P}}_{>}\left(\Omega \right)$, the vector space of contrasts $CO\left(\Omega \right)$, $n=\#\Omega$, and the centered log ratio displacement

$$CLR:{\mathcal{P}}_{>}\left(\Omega \right)\times {\mathcal{P}}_{>}\left(\Omega \right)\ni (p,q)\mapsto log{\displaystyle \frac{q}{p}}-{\displaystyle \frac{1}{n}}\sum _{x\in \Omega }log{\displaystyle \frac{q\left(x\right)}{p\left(x\right)}}.$$

(12)

If f is a composition and $\varphi$ is a random variable of $\Omega$, then, if we define the $\mathbb{E}$-notation to include normalization,

$${\mathbb{E}}_f\left[\varphi \right]=\sum _{x\in \Omega }{\displaystyle \frac{f\left(x\right)}{{\sum }_{x}f\left(x\right)}}\varphi \left(x\right).$$

(13)

Let us check that all conditions for an affine manifold hold. The partial mapping at p is a chart with

$$\begin{array}{c}{s}_p:{\mathcal{P}}_{>}\left(\Omega \right)\ni q\mapsto {CLR}_p\left(q\right)=log{\displaystyle \frac{q}{p}}-{\mathbb{E}}_1\left[log{\displaystyle \frac{q}{p}}\right]\in CO\left(\Omega \right),\end{array}$$

(14)

$$\begin{array}{c}{s}_p^{-1}:CO\left(\Omega \right)\ni v\mapsto {\mathrm{e}}^{v-{\Psi }_p\left(v\right)}·p,{\Psi }_p\left(v\right)={\mathbb{E}}_1\left[log{\displaystyle \frac{p}{q}}\right]=log{\mathbb{E}}_p\left[{\mathrm{e}}^v\right].\end{array}$$

(15)

The axioms for affine geometry are satisfied, as the equality $s_p\left(q\right)+s_q\left(r\right)=s_p\left(r\right)$ is easily checked,

$$\begin{array}{c}\hfill {CLR}_p\left(q\right)+{CLR}_q\left(r\right)=log{\displaystyle \frac{q}{p}}-{\mathbb{E}}_1\left[log{\displaystyle \frac{q}{p}}\right]+log{\displaystyle \frac{r}{q}}-{\mathbb{E}}_1\left[log{\displaystyle \frac{r}{q}}\right]\\ \hfill =log{\displaystyle \frac{r}{p}}-{\mathbb{E}}_1\left[log{\displaystyle \frac{r}{p}}\right]={CLR}_p\left(r\right).\end{array}$$

The patch mapping is a non-parametric exponential model [17,18] (§ 5.5) in which the sufficient-statistics v is a contrast. The convex mapping ${\Psi }_p$ is sometimes called a cumulant function. The first and second derivatives of ${\Psi }_p$ in the direction h and the directions h and k, respectively, are (as computed in the references above),

$$d{\Psi }_p\left(v\right)\left[h\right]={\mathbb{E}}_q\left[h\right],d^2{\Psi }_p\left(v\right)[h,k]={Cov}_q\left(h,k\right).$$

(16)

The velocity of the curve $t\mapsto q\left(t\right)$ is

$${\displaystyle \frac{d}{dt}}{CLR}_p\left(q\left(t\right)\right)={\displaystyle \frac{d}{dt}}logq\left(t\right)-{\mathbb{E}}_1\left[{\displaystyle \frac{d}{dt}}logq\left(t\right)\right]={\displaystyle \frac{\dot{q}\left(t\right)}{q\left(t\right)}}-{\mathbb{E}}_1\left[{\displaystyle \frac{\dot{q}\left(t\right)}{q\left(t\right)}}\right].$$

where $\dot{q}\left(t\right)$ is the derivative computed in $A(\Omega )$. The random variable $\dot{q}\left(t\right)/q\left(t\right)$ is called Fisher’s score [18], and it has the remarkable property

$${\mathbb{E}}_{q\left(t\right)}\left[{\displaystyle \frac{\dot{q}\left(t\right)}{q\left(t\right)}}\right]=\sum _{x\in \Omega }\dot{q}(x;t)=0.$$

In the reduced basis, with $\Omega =\left\{1,\cdots ,n\right\}$, each $j\ne n$ component of the CLR displacement becomes

$$\begin{array}{c}\hfill {CLR}_p\left(q\right)\left(j\right)=\left(log{\displaystyle \frac{q\left(j\right)}{p\left(j\right)}}-{\mathbb{E}}_1\left[log{\displaystyle \frac{q}{p}}\right]\right)-\left(log{\displaystyle \frac{q\left(j\right)}{p\left(j\right)}}-{\mathbb{E}}_1\left[log{\displaystyle \frac{q}{p}}\right]\right)\\ \hfill =log{\displaystyle \frac{q\left(j\right)}{p\left(j\right)}}-log{\displaystyle \frac{q\left(n\right)}{p\left(n\right)}}=log{\displaystyle \frac{p\left(n\right)q\left(j\right)}{p\left(j\right)q\left(n\right)}}.\end{array}$$

See [4] for a detailed treatment.

Within Aitchison’s paradigm, we presented the idea of the centered log-ratio (CLR) transformation (12). Determining displacement mappings between probability distributions on the simplex depends critically on this transformation. By mapping compositions into a Euclidean space, the CLR makes the comparison and analysis of probability distributions easier in a way that makes sense with respect to the inherent geometry of the simplex (referring to the unique geometric properties of the probability simplex—a space that consists of probability distributions where all components are positive and sum to one). Aitchison geometry’s mathematical foundations are explored in more detail in Section 3.6 with special attention to how it relates to information geometry (IG). We discuss how Aitchison geometry may be considered a specific example of IG, simplifying the displacement mappings with a fixed reference chart and deriving important conclusions, including the parallelogram law that guarantees these mappings to be consistent over a wide range of distributions. A further explanation and detailed connections with the centered log-ratio transformation are provided in Section 3.6 below.

2.3. Tangent Bundle

The motivation for defining a manifold structure, $\mathcal{M}$, through an atlas, $\mathcal{A}$, is the availability of a coherent setup for performing calculus operations. A typical problem in our interest is minimizing a smooth scalar field $G:\mathcal{M}\to \mathbb{R}$ via a gradient-flow algorithm. To this end, we must consider points on the manifold and derivatives, which exist in a vector space attached to each point.

The tangent bundle $T\mathcal{M}$ is defined as a set of pairs,

$$T\mathcal{M}=\left\{(q,v)|q\in \mathcal{M},v\in T_q\mathcal{M}\right\},$$

where $T_q\mathcal{M}$ is the tangent space at q, that is, the vector space of all possible velocities of curves through q. The situation is straightforward in all the examples of affine manifolds we have seen up to now, as the tangent space identifies with the space of contrasts $CO\left(\Omega \right)$.

Example 5

(Exponential arc). Let $t\mapsto p\left(t\right)$ be the exponential arc in Equation (4) connecting the points q and $r\in {\mathcal{P}}_{>}\left(\Omega \right)$. In the CLR affine manifold, we represent the two equivalent probability densities with the CLR patch centered at p, as in Equation (15),

$$q={\mathrm{e}}^{v-{\Psi }_p\left(v\right)}·p,r={\mathrm{e}}^{w-{\Psi }_p\left(w\right)}·p,withv,w\in CO\left(\Omega \right),$$

and hence, the arc $t\mapsto p\left(t\right)$ is expressed in terms of a mixture of two random variables, v and w, and the convex mapping ${\Psi }_p$, that is, two canonical statistics and the cumulant function,

$$p\left(t\right)={\mathrm{e}}^{(1-t)v+tw-{\Psi }_p((1-t)v+tw)}·p\mathit{and}{CLR}_p\left(p\left(t\right)\right)=(1-t)v+tw.$$

The affine velocity is constant,

$$\stackrel{★}{q}\left(t\right)={\displaystyle \frac{d}{dt}}{CLR}_p\left(p\left(t\right)\right)={\displaystyle \frac{d}{dt}}((1-t)v-tw)=w-v.$$

The exponential arc is the line segment with constant velocity connecting q and r in the CLR affine geometry. The set of all possible velocities at a given point is an expression of the tangent space.

Example 6

(Gradient of the KL-divergence). For any $p\in {\mathcal{P}}_{>}\left(\Omega \right)$, the Kullback–Leibler divergence, in short KL-divergence (see [19] or § 3.3–4 in [9]), of q and p is the scalar field

$${\mathcal{P}}_{>}\left(\Omega \right)\ni q\mapsto D\left(p\parallel q\right)={\mathbb{E}}_p\left[log{\displaystyle \frac{p}{q}}\right]\in \mathbb{R}.$$

The patch at p for the CLR displacement is Equation (15); hence, the expression at p of the KL-divergence is

$$G:v\mapsto D\left(p\parallel {\mathrm{e}}^{v-{\Psi }_p\left(v\right)}·p\right)={\mathbb{E}}_p\left[-v+{\Psi }_p\left(v\right)\right]={\Psi }_p\left(v\right)-{\mathbb{E}}_p\left[v\right].$$

The expression is defined on the vector space $CO\left(\Omega \right)$, and hence, we can compute the ordinary derivative in the direction $h\in CO\left(\Omega \right)$ using Equation (16) and $q={\mathrm{e}}^{v-{\Psi }_p\left(v\right)}·p$,

$$\begin{array}{c}dG\left(v\right)\left[h\right]={\left.{\displaystyle \frac{d}{dt}}G(v+th)\right|}_{t=0}={\left.{\displaystyle \frac{d}{dt}}{\Psi }_p(v+th)\right|}_{t=0}-{\mathbb{E}}_p\left[h\right]\hfill \\ =d{\Psi }_p\left(v\right)\left[h\right]-{\mathbb{E}}_p\left[h\right]={\mathbb{E}}_q\left[h\right]-{\mathbb{E}}_p\left[h\right]={\mathbb{E}}_p\left[({\mathrm{e}}^{v-{\Psi }_p\left(v\right)}-1))h\right]\hfill \\ ={\mathbb{E}}_1\left[np({\mathrm{e}}^{v-{\Psi }_p\left(v\right)}-1)h\right].\hfill \end{array}$$

As $(np({\mathrm{e}}^{v-{\Psi }_1\left(v\right)}-1)\in CO\left(\Omega \right)$, if we define the inner product of $CO\left(\Omega \right)$ to be $(v,w)\mapsto 〈v,w〉={\mathbb{E}}_1\left[vw\right]$, then $np({\mathrm{e}}^{v-{\Psi }_1\left(v\right)}-1)$ is the gradient of G,

$$dG\left(v\right)\left[h\right]=〈pn({\mathrm{e}}^{v-{\Psi }_p\left(v\right)},h〉.$$

The gradient is also defined on the affine manifold as follows. If $t\mapsto q\left(t\right)$ is a smooth curve with expression in the CLR, the expression at p is $t\mapsto {CLR}_p\left(q\left(t\right)\right)$, and the affine velocity is $\stackrel{★}{q}\left(t\right)={\displaystyle \frac{d}{dt}}{CLR}_p\left(q\left(t\right)\right)={\displaystyle \frac{\dot{q}\left(t\right)}{q\left(t\right)}}-{\mathbb{E}}_1\left[{\displaystyle \frac{\dot{q}\left(t\right)}{q\left(t\right)}}\right]$. Hence,

$${\displaystyle \frac{d}{dt}}D\left(p\parallel q\left(t\right)\right)={\displaystyle \frac{d}{dt}}G({CLR}_p\left(q\left(t\right)\right)=〈n(q\left(t\right)-p),\stackrel{★}{q}\left(t\right)〉.$$

Example 7

(Entropy). The expression of the entropy [19]

$$\mathcal{H}\left(q\right):{\mathcal{P}}_{>}\left(\Omega \right)\ni q\mapsto -{\mathbb{E}}_q\left[logq\right]=-{\mathbb{E}}_p\left[{\displaystyle \frac{q}{p}}logq\right]$$

in the chart at p becomes, using the first Equation (16),

$$\begin{array}{c}{H}_p:CO\left(\Omega \right)\ni v\mapsto H_p\left(v\right)=-{\mathbb{E}}_p\left[{\mathrm{e}}^{v-{\Psi }_p\left(v\right)}(v-{\Psi }_p\left(v\right)+logp)\right]\hfill \\ =-d{\Psi }_p\left(v\right)\left[v\right]+{\Psi }_p\left(v\right)+\mathcal{H}\left(p\right),\hfill \end{array}$$

and hence, using the second Equation (16),

$$d{H}_p\left(v\right)\left[h\right]=-d^2{\Psi }_p\left(v\right)[v,h]-d{\Psi }_p\left(v\right)\left[h\right]+d{\Psi }_p\left(v\right)\left[h\right]={Cov}_{s_p^{-1}\left(v\right)}\left(v,h\right).$$

If $t\mapsto q\left(t\right)$ is a curve with expression $v\left(t\right)=log{\displaystyle \frac{q\left(t\right)}{p}}-{\mathbb{E}}_1\left[log{\displaystyle \frac{q\left(t\right)}{p}}\right]$ and velocity $\stackrel{★}{q}\left(t\right)={\displaystyle \frac{\dot{q}\left(t\right)}{q\left(t\right)}}-{\mathbb{E}}_1\left[{\displaystyle \frac{\dot{q}\left(t\right)}{q\left(t\right)}}\right]$,

$${\displaystyle \frac{d}{dt}}\mathcal{H}\left(q\left(t\right)\right)={\displaystyle \frac{d}{dt}}{H}_p\left({CLR}_p\left(q\left(t\right)\right)\right)={Cov}_{q\left(t\right)}\left(v\left(t\right),\stackrel{★}{q}\left(t\right)\right)={Cov}_{q\left(t\right)}\left(log{\displaystyle \frac{q\left(t\right)}{p}},{\displaystyle \frac{\dot{q}\left(t\right)}{q\left(t\right)}}\right).$$

3. Affine Statistical Bundle

We define other affine geometries on the open probability simplex, using a richer atlas of charts and introducing an essential notion of duality. This approach highlights the relationship between our topic and Fisher’s statistics and Boltzmann and Gibbs’s statistical physics [20].

Our geometry definitions are kinematic, based on calculating the velocity and acceleration of curves. A one-dimensional statistical model is a curve, that is, a mapping, $I\ni t\mapsto q\left(t\right)\in \mathcal{P}\left(\Omega \right)$, where $\mathcal{P}\left(\Omega \right)$ is the probability simplex on the finite sample space $\Omega$, and I is an open interval of $\mathbb{R}$. Higher dimensional statistical models are treated according to the usual conventions of multivariate analysis.

3.1. Fisher’s Score

As $\mathcal{P}\left(\Omega \right)\subset {\mathbb{R}}^{\Omega }$, the affine velocity $t\mapsto \dot{q}\left(t\right)$ is well defined and takes values in the space of contrasts $CO\left(\Omega \right)$, as shown in the previous section.

Smooth curves in ${\mathbb{R}}^{\Omega }$, whose values stay in the probability simplex, have a notable property related to their support. If the curve stays in the open simplex, then the support is $\Omega$; that is, $q(x;t)\ne 0$ for all $x\in \Omega$ and all $t\in I$ [21].

Otherwise, the curve hits the boundary of the probability simplex; that is, there exists at least a pair, $(a,\overline{t}))\in \Omega \times I$, such that $q(a;\overline{t})=0$. This implies $q(a;t)\ge q(a,\overline{t})$; hence, $\dot{q}(a;\overline{t})=0$ [3]. In conclusion,

$$q(x;t)=0\Rightarrow \dot{q}(x;t)=0.$$

In the language of measure theory, $\dot{q}\left(t\right)$ is absolutely continuous with respect to $q\left(t\right)$, $\dot{q}\left(t\right)\ll q\left(t\right)$ [22].

Hence, given any ${\mathbb{R}}^{\Omega }$-smooth curve in the probability complex, there exists a curve, $t\mapsto \stackrel{★}{q}\left(t\right)\in {\mathbb{R}}^{\Omega }$, called Fisher’s score, such that the following applies:

$$\dot{q}\left(t\right)=\stackrel{★}{q}\left(t\right)q\left(t\right)$$

Fisher’s score has two remarkable properties [23]. First, if $q(x;t)>0$, $t\in I$, then

$$\stackrel{★}{q}(x;t)={\displaystyle \frac{\dot{q}(x;t)}{q(x;t)}}={\displaystyle \frac{d}{dt}}logq(x;t)$$

and second,

$${\mathbb{E}}_{q\left(t\right)}\left[\stackrel{★}{q}\left(t\right)\right]=\sum _x\stackrel{★}{q}(x;t)q(x;t)=\sum _x\dot{q}(x;t)={\displaystyle \frac{d}{dt}}1=0.$$

(17)

Specifically, if $t\mapsto q\left(t\right)\in {\mathcal{P}}_{>}\left(\Omega \right)$ (the open probability simplex), then Fisher’s score is given by

$$\stackrel{★}{q}\left(t\right)={\displaystyle \frac{d}{dt}}logq\left(t\right).$$

There are many statistical reasons to take the Fisher score as a proper notion of the parameter rate of change in a statistical model. See the textbook [18] (Ch. 4) and the monograph [22].

3.2. Statistical Bundle and Parallel Transports

For the general notion of a bundle, see, for example, [14]. The previous discussion leads us to consider, for each positive probability function $q\in {\mathcal{P}}_{>}\left(\Omega \right)$, the vector space of q-contrasts, that is, the random variables that are centered for q,

$$B_q=\left\{u\in {\mathbb{R}}^{\Omega }|\sum _{x\in \Omega }u\left(x\right)q\left(x\right)=0\right\}.$$

(18)

The statistical bundle $S{\mathcal{P}}_{>}\left(\Omega \right)$ is the bundle $S{\mathcal{P}}_{>}\left(\Omega \right)\to {\mathcal{P}}_{>}\left(\Omega \right)$ of q-contrasts,

$$S{\mathcal{P}}_{>}\left(\Omega \right)=\left\{(q,u)|q\in {\mathcal{P}}_{>}\left(\Omega \right),u\in {\mathbb{R}}^{\Omega },{\mathbb{E}}_q\left[u\right]=0\right\}.$$

(19)

Each fiber $S_q{\mathcal{P}}_{>}\left(\Omega \right)=B_q$ will represent all Fisher’s scores at the point q. In turn, the Fisher’s score will be interpreted as velocities.

Example 8.

The plot in Figure 3 illustrates the probability simplex for two outcomes, the point $q=\left\{q\right(1),q(2\left)\right\}$ within the simplex, and the vectors $u_1$ and $u_2$ representing q-contrasts.

Each fiber $B_q$ is equipped with a natural inner product, the covariance at q,

$${〈u,v〉}_q={Cov}_q\left(u,v\right)={\mathbb{E}}_q\left[uv\right].$$

This inner product measures the angles and lengths of vectors within each fiber, contributing to the metric structure of the statistical bundle. The mapping $(q,u,v)\mapsto {〈u,v〉}_q$ is the metric of the statistical bundle.

Example 9

(Fisher’s metric). Given a smooth curve, $t\mapsto q\left(t\right)\in {\mathcal{P}}_{>}\left(\Omega \right)$, the Fisher’s score $\stackrel{★}{q}\left(t\right)={\displaystyle \frac{d}{dt}}logq\left(t\right)=\dot{q}\left(t\right)/q\left(t\right)$ provides a curve in the statistical bundle [24],

$$t\mapsto (q\left(t\right),\stackrel{★}{q}\left(t\right))\in S{\mathcal{P}}_{>}\left(\Omega \right)$$

The variance in the Fisher’s score is called Fisher’s information [25,26],

$$t\mapsto {〈\stackrel{★}{q}\left(t\right),\stackrel{★}{q}\left(t\right)〉}_{q\left(t\right)}={\mathbb{E}}_{q\left(t\right)}\left[{\left({\displaystyle \frac{\dot{q}\left(t\right)}{q\left(t\right)}}\right)}^2\right]=\sum _x{\displaystyle \frac{{\left(\dot{q}(x;t)\right)}^2}{q(x;t)}}.$$

Each fiber is a different vector space in the statistical bundle. Because of that, the mapping $t\mapsto (q\left(t\right),\stackrel{★}{q}\left(t\right))\in S{\mathcal{P}}_{>}\left(\Omega \right)$ requires a special handling of differentiation with a construction called connection. We define this notion as follows.

Definition 2

(Parallel transport). A parallel transport of the statistical bundle $S{\mathcal{P}}_{>}\left(\Omega \right)$ is a collection of bijective linear mappings between the fibers

$${\mathbb{U}}_p^q:S_p{\mathcal{P}}_{>}\left(\Omega \right)\to S_q{\mathcal{P}}_{>}\left(\Omega \right)$$

(20)

such that, for all $p,q,r\in {\mathcal{P}}_{>}\left(\Omega \right)$, the cocycle relation holds,

$${\mathbb{U}}_q^r{\mathbb{U}}_p^q={\mathbb{U}}_p^r.$$

(21)

Given a parallel transport, if $v\in S_q{\mathcal{P}}_{>}\left(\Omega \right)$ is proportional to ${\mathbb{U}}_p^{q}u$, $u\in S_p{\mathcal{P}}_{>}\left(\Omega \right)$, we say that u and v are parallel.

3.3. Affine Bundle

Here, we generalize Weyl’s definition of affine space to adapt it to the statistical bundle. The novelty is that there is a different vector space at each point of the space.

Definition 3.

Consider the statistical bundle $S{\mathcal{P}}_{>}\left(\Omega \right)$ and a cocycle of parallel transports, ${\mathbb{U}}_p^q$, $p,q\in {\mathcal{P}}_{>}\left(\Omega \right)$. An affine displacement is a mapping,

$$s:{\mathcal{P}}_{>}\left(\Omega \right)\times {\mathcal{P}}_{>}\left(\Omega \right)\ni (p,q)\mapsto s_p\left(q\right)\in S_p{\mathcal{P}}_{>}\left(\Omega \right),$$

such that the following applies:

1. 

For each fixed p, the mapping $s_p:q\mapsto s_p\left(q\right)$ is 1-to-1, and it has an open image;

2. 

The parallelogram law holds,

$$s_p\left(q\right)+{\mathbb{U}}_q^p{s}_q\left(r\right)=s_p\left(r\right).$$

The affine displacement on the statistical bundle provides an atlas of charts, $s_p$, $p\in {\mathcal{P}}_{>}\left(\Omega \right)$. Each chart is a local coordinate system around the origin point p. The change-of-chart map is $s_p\circ s_q^{-1}$. At $\rho =s_q^{-1}\left(w\right)$, $w\in S_q{\mathcal{P}}_{>}\left(\Omega \right)$, it holds that

$$s_p\circ s_q^{-1}\left(w\right)=s_p\left(\rho \right)=s_p\left(q\right)+{\mathbb{U}}_q^p{s}_q\left(\rho \right)=s_p\left(q\right)+{\mathbb{U}}_q^{p}w.$$

(22)

The change-of-origin map restricts an affine map to ${\mathbb{R}}^{\Omega }$, whose linear part is the parallel transport (Figure 4).

Example 10

(Barycenter and convex combination). Given points $q_1,\dots ,q_n\in {\mathcal{P}}_{>}\left(\Omega \right)$, coefficients ${\lambda }_1,\dots ,{\lambda }_n$ with ${\sum }_i{\lambda }_i=1$, and a reference, p, the convex combination of p-coordinates, ${\sum }_i{\lambda }_i{s}_p\left(q_i\right)$, is the p-coordinate of a point, called the barycenter,

$$\overline{q}=s_p^{-1}\left(\sum _i{\lambda }_i{s}_p\left(q_i\right)\right),$$

(23)

which does not depend on p. In fact, for $q\ne p$, we can use the change-of-chart of Equation (22) to obtain

$$\begin{array}{c}{s}_q^{-1}\left(\sum _i{\lambda }_i{s}_q\left(q_i\right)\right)=s_q^{-1}\left(\sum _i{\lambda }_i\left(s_q\left(p\right)+{\mathbb{U}}_p^q{s}_p\left(q_i\right)\right)\right)=\hfill \\ s_q^{-1}\left(s_q\left(p\right)+{\mathbb{U}}_p^q\sum _i{\lambda }_i{s}_p\left(q_i\right)\right)=s_q^{-1}\left(s_q\left(p\right)+{\mathbb{U}}_p^q{s}_p\left(\overline{q}\right)\right)=\overline{q}.\hfill \end{array}$$

Example 11

(Single fiber). If s is an affine displacement with parallel transports ${\mathbb{U}}_p^q$, and $\left(1\right)$ is the uniform probability function, then

$$\overline{s}(p,q)={\mathbb{U}}_p^{\left(1\right)}s(p,q)$$

defines an affine displacement with vector space $CO\left(\Omega \right)$. In fact,

$$\begin{array}{c}\overline{s}(p,q)+\overline{s}(q,r)={\mathbb{U}}_p^{\left(1\right)}s(p,q)+{\mathbb{U}}_q^{\left(1\right)}s(q,r)=\hfill \\ {\mathbb{U}}_p^{\left(1\right)}\left(s(p,q)+{\mathbb{U}}_q^{p}s(q,r)\right)={\mathbb{U}}_p^{\left(1\right)}s(p,r)=\overline{s}(p,r).\hfill \end{array}$$

3.4. Velocity and Natural Gradient

If $t\mapsto q\left(t\right)$ is a curve in the open probability simplex, and $s_p$ is a chart according to Definition 3, the expression of the velocity in the chart is, then,

$$t\mapsto {\displaystyle \frac{d}{dt}}{s}_p\left(q\left(t\right)\right)\in S_p{\mathcal{P}}_{>}\left(\Omega \right)$$

If we change the chart and use the second equations in Definition 3, the change in the expression of the velocity follows the parallel transport

$${\mathbb{U}}_q^p{\displaystyle \frac{d}{dt}}{s}_q\left(q\left(t\right)\right)={\displaystyle \frac{d}{dt}}{s}_p\left(q\left(t\right)\right).$$

Let us define an intrinsic notion of velocity using the so-called moving frame.

Definition 4

(Velocity). The velocity of the curve $t\mapsto q\left(t\right)$ is

$$\stackrel{★}{q}\left(t\right):t\mapsto {\mathbb{U}}_p^{q\left(t\right)}{\displaystyle \frac{d}{dt}}{s}_p\left(q\left(t\right)\right),$$

which does not depend on p. The velocity provides a lifting of the curve to the statistical bundle,

$$t\mapsto (q\left(t\right),\stackrel{★}{q}\left(t\right))\in S{\mathcal{P}}_{>}\left(\Omega \right).$$

The function $F:{\mathcal{P}}_{>}\left(\Omega \right)\to \mathbb{R}$ is smooth if its expression in the chart at p, $F_p=F\circ s_p^{-1}:S_p{\mathcal{P}}_{>}\left(\Omega \right)\to \mathbb{R}$ is differentiable. Let us recall that the gradient $\nabla f$ of a smooth function, $f:{\mathbb{R}}^n\to \mathbb{R}$, is defined by the equation ${\displaystyle \frac{d}{dt}}f\left(x\left(t\right)\right)=\nabla f\left(x\left(t\right)\right)·\dot{x}\left(t\right)$ for all smooth curves in ${\mathbb{R}}^n$. The same definition applies to any other inner product. After Amari, we call the natural gradient the gradient associated with the inner product on the statistical bundle.

Definition 5

(Natural gradient). If $t\mapsto q\left(t\right)\in {\mathcal{P}}_{>}\left(\Omega \right)$, $q=q\left(0\right)$ and $\stackrel{★}{q}=\stackrel{★}{q}\left(0\right)$, and $F{\mathcal{P}}_{>}\left(\Omega \right)\to \mathbb{R}$ is smooth, then

$$\begin{array}{c}\hfill {\left.{\displaystyle \frac{d}{dt}}F\circ q\left(t\right)\right|}_{t=0}={\left.{\displaystyle \frac{d}{dt}}{F}_q\circ s_q\left(q\left(t\right)\right)\right|}_{t=0}={\left.d{F}_q\left(q\left(t\right)\right)[\stackrel{★}{q}\left(t\right)]\right|}_{t=0}=\\ \hfill d{F}_q\left(0\right)[\stackrel{★}{q}]={〈gradF\left(q\right),\stackrel{★}{q}〉}_q,\end{array}$$

(24)

where $q\mapsto gradF\left(q\right)$ is the natural gradient of F; that is, $gradF\left(q\right)$ is the element of $S_q{\mathcal{P}}_{>}\left(\Omega \right)$ identified with the linear operator $dF\left(0\right)$ in the covariance inner product.

Definition 6

(Exponential of a parallel transport). Assume that, for each given $(q,v)\in S{\mathcal{P}}_{>}\left(\Omega \right)$, the differential equation

$$\stackrel{★}{q}\left(t\right)={\mathbb{U}}_q^{q\left(t\right)}v,q\left(0\right)=q$$

has a unique global solution. Such a curve is auto-parallel, $\stackrel{★}{q}\left(t\right)=\alpha \left(t\right){\mathbb{U}}_{q\left(s\right)}^{q\left(t\right)}\stackrel{★}{q}\left(s\right)$, with relative velocity $\alpha \left(t\right)=1$. The mapping

$$Exp:S{\mathcal{P}}_{>}\left(\Omega \right)\ni (q,v)\mapsto q\left(1\right)={Exp}_q\left(v\right)\in {\mathcal{P}}_{>}\left(\Omega \right).$$

is called the exponential of the affine bundle.

The previous construction, leading from the affine bundle to the moving frame velocity and the natural gradient, is essential to the non-parametric version of Amari’s information geometry. This specific construction is qualified below as the dually affine geometry of the probability simplex. It admits a number of immediate generalizations. In particular, it can be used to define a second-order calculus, as sketched in the example below.

Example 12

(Second-order affine geometry). As a basic set, let us assume the statistical bundle with the cocycle of parallel transports and, as a vector space, the product of the fibers. The following equation defines a displacement:

$$((p,u),(q,v))\mapsto (s_p\left(q\right),{\mathbb{U}}_q^{p}v-u)\in S_p{\mathcal{P}}_{>}\left(\Omega \right)\times S_p{\mathcal{P}}_{>}\left(\Omega \right).$$

If $t\mapsto (q\left(t\right),v\left(t\right))\in S{\mathcal{P}}_{>}\left(\Omega \right)$ is a curve in the statistical bundle, then the derivative at $(p,u)$ is

$$t\mapsto \left({\displaystyle \frac{d}{dt}}\left(q\left(t\right)\right),{\displaystyle \frac{d}{dt}}{\mathbb{U}}_{q\left(t\right)}^{p}v\left(t\right)\right)$$

In particular, if $v\left(t\right)=\stackrel{★}{q}\left(t\right)$, the second-order derivative is defined in the chart at p by

$$t\mapsto \left({\displaystyle \frac{d}{dt}}{s}_p\left(q\left(t\right)\right),{\displaystyle \frac{d}{dt}}{\mathbb{U}}_{q\left(t\right)}^p\stackrel{★}{q}\left(t\right)\right).$$

In the moving frame, the acceleration is defined by

$${\left.\stackrel{**}{q}\left(t\right)={\displaystyle \frac{d}{dt}}{\mathbb{U}}_{q\left(t\right)}^p\stackrel{★}{q}\left(t\right)\right|}_{p=q\left(t\right)}$$

(25)

3.5. Dually Affine Atlases on the Open Probability Simplex

According to the works of Amari [1] and Amari and Nagaoka [2], there are two main types of affine geometry on the probability simplex, the exponential one and the mixture one. We axiomatically describe such geometries via the notion of the statistical bundle endowed with parallel transport. This approach was initially presented in [27]. Each fiber of the statistical bundle is endowed with the covariance inner product to identify each fiber with its dual. In this identification, the two parallel transports are dual to each other.

Theorem 1

(Exponential transport and mixture transport).

1. 

The equation

$${{}^e\mathbb{U}}_p^q:S_p{\mathcal{P}}_{>}\left(\Omega \right)\ni u\mapsto u-{\mathbb{E}}_q\left[u\right]\in S_q{\mathcal{P}}_{>}\left(\Omega \right)$$

defines a cocycle of parallel transports. It is the exponential transport.

2. 

The equation

$${{}^m\mathbb{U}}_p^q:S_p{\mathcal{P}}_{>}\left(\Omega \right)\ni u\mapsto {\displaystyle \frac{p}{q}}u\in S_q{\mathcal{P}}_{>}\left(\Omega \right)$$

defines a cocycle of parallel transports. It is the mixture transport.

3. 

The exponential transport and the mixture transports are dual to each other in the covariance pairing. Namely, if $u\in S_p{\mathcal{P}}_{>}\left(\Omega \right)$ and $v\in S_q{\mathcal{P}}_{>}\left(\Omega \right)$,

$${〈v,{{}^e\mathbb{U}}_p^{q}u〉}_q={〈{{}^m\mathbb{U}}_q^{p}v,u〉}_p.$$

Proof. 

All checks for Definition 2 are immediate. The duality follows from

$${〈v,{{}^{\mathrm{e}}\mathbb{U}}_p^{q}u〉}_q={〈v,u-{\mathbb{E}}_q\left[u\right]〉}_q={〈v,u〉}_q={\mathbb{E}}_q\left[vu\right]={\mathbb{E}}_p\left[{\displaystyle \frac{q}{p}}vu\right]={〈{{}^{\mathrm{m}}\mathbb{U}}_q^{p}v,u〉}_p.$$

□

For each parallel transport, we define an affine displacement. Each affine displacement maps the open probability simplex onto a vector space of contrasts. Conversely, the inverse maps are statistical models parametrized by a vector space of contrasts. The normalizing constants are interpreted in terms of statistical information.

Theorem 2

(Exponential chart and mixture chart).

1. 

The mapping

$${\mathcal{P}}_{>}\left(\Omega \right)\times {\mathcal{P}}_{>}\left(\Omega \right)\ni (p,q)\mapsto s_p\left(q\right)=log{\displaystyle \frac{q}{p}}-{\mathbb{E}}_p\left[log{\displaystyle \frac{q}{p}}\right]\in S_p{\mathcal{P}}_{>}\left(\Omega \right),$$

is an affine displacement for the exponential parallel transport Theorem 1 item 1. The inverse of the exponential chart $s_p$ is the exponential model

$$s_p^{-1}:S_p{\mathcal{P}}_{>}\left(\Omega \right)\ni v\mapsto {\mathrm{e}}^{v-K_p\left(v\right)}·p,K_p\left(v\right)=log{\mathbb{E}}_p\left[{\mathrm{e}}^v\right].$$

2. 

The mapping

$${\mathcal{P}}_{>}\left(\Omega \right)\times {\mathcal{P}}_{>}\left(\Omega \right)\ni (p,q)\mapsto {\eta }_p\left(q\right)={\displaystyle \frac{q}{p}}-1\in S_p{\mathcal{P}}_{>}\left(\Omega \right),$$

is an affine displacement for the mixture displacement Theorem 1 item 2. The inverse of the mixture chart ${\eta }_p$ is the mixture model

$${\eta }_p^{-1}:S_p{\mathcal{P}}_{>}\left(\Omega \right)\supset \left\{v>-1\right\}\ni v\mapsto (1+v)·p.$$

Proof. 

• The generalized parallelogram law for the exponential displacement is

  $$\begin{array}{c}\left(log{\displaystyle \frac{q}{p}}-{\mathbb{E}}_p\left[log{\displaystyle \frac{q}{p}}\right]\right)+{{}^{\mathrm{e}}\mathbb{U}}_q^p\left(log{\displaystyle \frac{r}{q}}-{\mathbb{E}}_q\left[log{\displaystyle \frac{r}{q}}\right]\right)\hfill \\ =\left(log{\displaystyle \frac{q}{p}}-{\mathbb{E}}_p\left[log{\displaystyle \frac{q}{p}}\right]\right)+\left(log{\displaystyle \frac{r}{q}}-{\mathbb{E}}_p\left[log{\displaystyle \frac{r}{q}}\right]\right)\hfill \\ =log{\displaystyle \frac{r}{p}}-{\mathbb{E}}_p\left[log{\displaystyle \frac{r}{p}}\right].\hfill \end{array}$$
• The generalized parallelogram law for mixture displacement is

  $$\left({\displaystyle \frac{q}{p}}-1\right)+{{}^{\mathrm{m}}\mathbb{U}}_q^p\left({\displaystyle \frac{r}{q}}-1\right)=\left({\displaystyle \frac{q}{p}}-1\right)+{\displaystyle \frac{q}{p}}\left({\displaystyle \frac{r}{q}}-1\right)={\displaystyle \frac{r}{p}}-1.$$

    □

The inverse exponential chart is defined on all p-contrasts by

$$s_p^{-1}\left(v\right)=exp(v-K_p\left(v\right))·p=e_p\left(v\right)$$

where

$$K_p\left(v\right)=log{\mathbb{E}}_p\left[{\mathrm{e}}^v\right]$$

K is usually called the cumulant generating functional of v with respect to p. The inverse chart $s_p^{-1}\left(v\right)$ maps a vector p-contrast v back to a probability function. The expression $exp(v-K_p\left(v\right))·p$ involves an exponential map adjusted by a normalization factor $K_p\left(v\right)$. The quantity $exp(v-K_p\left(v\right))$ ensures that the resulting vector after exponentiation is properly normalized to be a valid probability distribution. The term $e_p\left(v\right)=s_p^{-1}$ denotes this normalized exponential family.

Kullback–Leibler Divergence

The Kullback–Leibler divergence between two positive probability functions, p and q, is $D\left(p\parallel q\right)={\mathbb{E}}_p\left[log{\displaystyle \frac{p}{q}}\right]$. Both partial functions

$$q\mapsto D\left(p\parallel q\right)\mathrm{and}p\mapsto D\left(p\parallel q\right)$$

are commonly used in statistical optimization procedures. The primary reference is [28].

Example 13

(Expression of KL divergence). Let us find an expression for the Kullback–Leibler divergence by using a mixture chart on the first variable and the exponential chart on the second variable. Namely, if $p=(1+u)·m$ and $q={\mathrm{e}}^{v-K_m\left(v\right)}·m$, the expression in the charts at m is

$$\begin{array}{c}{\kappa }_m:\left\{u\in S_m{\mathcal{P}}_{>}\left(\Omega \right)|1+u>0\right\}\times S_m{\mathcal{P}}_{>}\left(\Omega \right)\ni (u,v)\mapsto \\ {\mathbb{E}}_m\left[(1+u)log{\displaystyle \frac{1+u}{{\mathrm{e}}^{v-K_m\left(v\right)}}}\right]=\\ {\mathbb{E}}_m\left[(1+u)log(1+u)\right]-{\mathbb{E}}_m\left[(1+u)(v-K_m\left(v\right))\right]=\\ {\mathbb{E}}_m\left[(1+u)log(1+u)\right]-{\mathbb{E}}_m\left[(1+u)v\right]-{\mathbb{E}}_m\left[(v-K_m\left(v\right))\right]=\\ H_m\left(u\right)+K_m\left(v\right)-{\mathbb{E}}_m\left[uv\right],\end{array}$$

(26)

where $H_m\left(u\right)={\mathbb{E}}_m\left[(1+u)log(1+u)\right]$.

If we let $p=q$ in Equation (26), that is, $1+u={\mathrm{e}}^{(v-K-m)\left(v\right)}$ and ${\mathbb{E}}_m\left[u\right]={\mathbb{E}}_m\left[v\right]=0$, then $H_m\left(u\right)+K_m\left(v\right)={〈u,v〉}_m$.

Example 14

(Divergence and displacement). In Equation (26), we can turn back to the probability densities to obtain the generalized parallelogram theorem [29]

$$D\left(p\parallel q\right)=D\left(p\parallel m\right)+D\left(m\parallel q\right)-{〈u,v〉}_m=D\left(p\parallel m\right)+D\left(m\parallel q\right)-{〈{\eta }_m\left(p\right),s_m\left(q\right)〉}_m.$$

In particular, the generalized Pythagorean theorem holds,

$${〈{\eta }_m\left(p\right),s_m\left(q\right)〉}_m=0\Rightarrow D\left(p\parallel q\right)=D\left(p\parallel m\right)+D\left(m\parallel q\right).$$

(27)

The cumulant function $K_p\left(u\right)$ is central to the theory. It is the exponential normalizing constant of the exponential model $e_p\left(u\right)={\mathrm{e}}^{u-K_p\left(u\right)}·p$, and it equals $D\left(p\parallel e_p\left(u\right)\right)$. Moreover, it has remarkable computational properties, summarized in the following theorem. Notice that it is a formalism well known in statistical physics, where it is used to discuss the Gibbs–Boltzmann distribution. See, for example, [30].

Theorem 3

(Calculus of the cumulant functional).

1. 

The derivative of $K_p$ at v in the direction h is

$$d{K}_p\left(v\right)\left[h\right]={\mathbb{E}}_{e_p\left(v\right)}\left[h\right]={〈{\displaystyle \frac{e_p\left(v\right)}{p}}-1,h〉}_p.$$

(28)

2. 

The second derivative of $K_p$ at v in the directions h and k is

$$\begin{array}{c}{d}^2{K}_p\left(v\right)[h,k]={Cov}_{e_p\left(v\right)}\left(h,=\right){〈{{}^e\mathbb{U}}_p^{e_p\left(v\right)}h,{{}^e\mathbb{U}}_p^{e_p\left(v\right)}k〉}_{e_p\left(v\right)}=\hfill \\ {〈{{}^m\mathbb{U}}_{e_p\left(v\right)}^p{{}^e\mathbb{U}}_p^{e_p\left(v\right)}h,k〉}_p.\hfill \end{array}$$

(29)

3. 

The third derivative of $K_p$ at v in the directions $h_1,h_2,$ and $h_3$ is

$$d^3{K}_p\left(v\right)[h_1,h_2,h_3]={\mathbb{E}}_{e_p\left(v\right)}\left[\left({{}^e\mathbb{U}}_p^{e_p\left(v\right)}{h}_1\right)\left({{}^e\mathbb{U}}_p^{e_p\left(v\right)}{h}_2\right)\left({{}^e\mathbb{U}}_p^{e_p\left(v\right)}{h}_3\right)\right].$$

4. 

The cumulant functional $K_p$ is convex, and its gradient is the covariance at p given by $\nabla K_p\left(v\right)={\displaystyle \frac{e_p\left(v\right)}{p}}-1$. The inverse gradient mapping is

$${(\nabla K_p)}^{-1}:S_p{\mathcal{P}}_{>}\left(\Omega \right)\ni w\mapsto log(1+w)-{\mathbb{E}}_p\left[log(1+w)\right],$$

and the convex conjugate is

$$\begin{array}{c}\hfill K_p^{*}\left(w\right)={〈w,{(\nabla K)}^{-1}w〉}_p-K_p\circ \left({\nabla }^{-1}{K}_p\right)\left(w\right)={\mathbb{E}}_p\left[(1+w)log(1+w)\right].\end{array}$$

(30)

Proof. 

The proof of the first three items follows through direct computation.

• $$\begin{array}{c}d{K}_p\left(v\right)\left[h\right]={\left.{\displaystyle \frac{d}{dt}}{K}_p(v+th)\right|}_{t=0}={\left.{\displaystyle \frac{d}{dt}}log{\mathbb{E}}_p\left[e^{v+th}\right]\right|}_{t=0}\hfill \\ ={\left.{\displaystyle \frac{d}{dt}}{\left({\mathbb{E}}_p\left[e^{v+th}\right]\right)}^{-1}{\mathbb{E}}_p\left[h{e}^{v+th}\right]\right|}_{t=0}={\left({\mathbb{E}}_p\left[e^v\right]\right)}^{-1}{\mathbb{E}}_p\left[h{e}^v\right]\hfill \\ ={\mathbb{E}}_{e_p\left(v\right)}\left[h\right]={\mathbb{E}}_p\left[{\displaystyle \frac{e_p\left(v\right)}{p}}h\right]={〈{\displaystyle \frac{e_p\left(v\right)}{p}}-1,h〉}_p\hfill \end{array}$$
• $$\begin{array}{c}{d}^2{K}_p\left(v\right)[h,k]={\left.{\displaystyle \frac{d}{dt}}d{K}_p(v+tk)\left[h\right]\right|}_{t=0}={\left.{\displaystyle \frac{d}{dt}}{〈{\displaystyle \frac{e_p(v+tk)}{p}}-1,h〉}_p\right|}_{t=0}\hfill \\ ={〈{\left.{\displaystyle \frac{d}{dt}}{e}^{v+tk-K_p(v+tk)}\right|}_{t=0},h〉}_p={〈{\displaystyle \frac{e_p\left(v\right)}{p}}\left(k-{\left.{\displaystyle \frac{d}{dt}}{K}_p(v+tk)\right|}_{t=0}\right),h〉}_p\hfill \\ ={〈{\displaystyle \frac{e_p\left(v\right)}{p}}\left(k-{\mathbb{E}}_{e_p\left(v\right)}\left[k\right]\right),h〉}_p={\mathrm{Cov}}_{e_p\left(v\right)}(h,k)\hfill \end{array}$$
• $$\begin{array}{c}{d}^3{K}_p\left(v\right)[h_1,h_2,h_3]={\left.{\displaystyle \frac{d}{dt}}{d}^2{K}_p(v+t{h}_3)[h_1,h_2]\right|}_{t=0}\hfill \\ ={{\displaystyle \frac{d}{dt}}{\left.〈h_1,{\displaystyle \frac{d}{dt}}\left({\displaystyle \frac{e_p(v+t{h}_3)}{p}}\right)\left(h_2-{\mathbb{E}}_{e_p(v+t{h}_3)}\left[h_2\right]\right)\right|}_{t=0}〉}_p\\ ={\left.〈h_1,{\displaystyle \frac{d}{dt}}\left({\displaystyle \frac{e_p(v+t{h}_3)}{p}}\right)\left(h_2-{\mathbb{E}}_{e_p(v+t{h}_3)}\left[h_2\right]\right)\right|}_{t=0}{\rangle }_p={〈h_1,{\displaystyle \frac{e_p\left(v\right)}{p}}\left(k-{\mathbb{E}}_{e_p\left(v\right)}\left[k\right]\right),h〉}_p\\ ={〈{\displaystyle \frac{e_p\left(v\right)}{p}}{h}_1,{\displaystyle \frac{e_p\left(v\right)}{p}}\left(h_2-{\mathbb{E}}_{e_p\left(v\right)}\left[h_2\right]\right),{\displaystyle \frac{e_p\left(v\right)}{p}}{h}_3〉}_p\\ \hfill ={\mathbb{E}}_{e_p\left(v\right)}\left[\left(h_1-{\mathbb{E}}_{e_p\left(v\right)}\left[h_1\right]\right)\left(h_2-{\mathbb{E}}_{e_p\left(v\right)}\left[h_2\right]\right)\left(h_3-{\mathbb{E}}_{e_p\left(v\right)}\left[h_3\right]\right)\right]\end{array}$$
• The convexity follows from items 1 and 2. The last statement is, in essence, a summary of the theorem itself.

□

3.6. Aitchison Geometry and Information Geometry

In Section Aitchison’s Centered Log Ratio [5], we have shown that the Aitchison geometry of the probability simplex is an instance of affine geometry. The present section shows how to interpret the same formalism as using a particular atlas on the statistical bundle. For clarity, we admit some overlapping between the two sections.

Let ${\mathcal{P}}_{>}\left(\Omega \right)$ be the positive probability simplex over a finite set $\Omega$, and let $T{\mathcal{P}}_{>}\left(\Omega \right)$ be its tangent space. Define the displacement mapping, $s_p:{\mathcal{P}}_{>}\left(\Omega \right)\to T{\mathcal{P}}_{>}\left(\Omega \right)$, for a reference point, $p\in {\mathcal{P}}_{>}\left(\Omega \right)$, and any $q\in {\mathcal{P}}_{>}\left(\Omega \right)$ as follows:

$$s_p\left(q\right)=log{\displaystyle \frac{q}{p}}-\sum _{y\in \Omega }{\displaystyle \frac{1}{\#\Omega }}log{\displaystyle \frac{q\left(y\right)}{p\left(y\right)}},$$

where $log{\displaystyle \frac{q}{p}}$ denotes the element-wise logarithm of the ratio of the probability distributions q and p, and ${\sum }_{y\in \Omega }{\displaystyle \frac{1}{\#\Omega }}log{\displaystyle \frac{q\left(y\right)}{p\left(y\right)}}$ is the average of the log ratios over all elements $y\in \Omega$. This can also be written as follows:

$$s_p\left(q\right)=log{\displaystyle \frac{q}{p}}-\mathbb{E}\left(log{\displaystyle \frac{q}{p}}\right),$$

(31)

where $\mathbb{E}\left(log{\displaystyle \frac{q}{p}}\right)$ is the uniform expectation of the log ratio.

The critical difference with exponential charts is the centering term, which is an expectation with respect to the uniform measure instead of the initial point, p.

The displacement mapping $s_p$ of Equation (31) is surjective. For every $u\in T{\mathcal{P}}_{>}\left(\Omega \right)$, there exists a $q\in {\mathcal{P}}_{>}\left(\Omega \right)$, such that $s_p\left(q\right)=u$. The inverse mapping $s_p^{-1}$ is given as follows:

$$s_p^{-1}\left(u\right)=exp(u-K\left(u\right))·p,$$

where the normalizing function $K\left(u\right)$ is defined as follows:

$$K\left(u\right)=log\sum _{y\in \Omega }{e}^{u\left(y\right)}p\left(y\right)=log{\mathbb{E}}_p\left[e^u\right].$$

Notice the relationship between the two forms of the normalizing constant:

$$K\left(u\right)=\mathbb{E}\left(log{\displaystyle \frac{q}{p}}\right).$$

This means that the normalizing constant $K\left(u\right)$ can be expressed as the p-expectation of the log ratio of the probability distributions q and p.

The parallelogram law holds true:

$$\left(log{\displaystyle \frac{q}{p}}-\mathbb{E}\left(log{\displaystyle \frac{q}{p}}\right)\right)+\left(log{\displaystyle \frac{r}{q}}-\mathbb{E}\left(log{\displaystyle \frac{r}{q}}\right)\right)=log{\displaystyle \frac{r}{p}}-\mathbb{E}\left(log{\displaystyle \frac{r}{p}}\right).$$

This law states that the sum of the centered log ratios between distributions q and p, and r and q, is equal to the centered log ratio between r and p. It ensures consistency in the displacement mappings across different distributions.

Example 15

(Uniform Distribution Case). If p is uniform, that is, $p\left(x\right)=D^{-1}$, where $D=\#\Omega$ (the number of elements in the set Ω), then the displacement mapping is simplified as follows:

$$s_{D^{-1}}\left(q\right)\left(x\right)=logq\left(x\right)-\sum _{y\in \Omega }{\displaystyle \frac{1}{D}}logq\left(y\right)=log{\displaystyle \frac{q\left(x\right)}{{\left({\prod }_{y\in \Omega }q\left(y\right)\right)}^{1/D}}}.$$

The displacement mapping $s_{D^{-1}}\left(q\right)\left(x\right)$ involves taking the log of $q\left(x\right)$ and subtracting the average log of all $q\left(y\right)$. This is simplified to the log of $q\left(x\right)$, divided by the geometric mean of $q\left(y\right)$ over all $y\in \Omega$.

A few comments are relevant at this point. The particular chart, derived from the displacement mapping when p is uniform, matches a chart in the exponential atlas used in information geometry. Exponential affine geometry, which deals with exponential families of probability distributions, and Aitchison geometry, which deals with compositional data, are fundamentally the same, but they are represented differently. The tangent space is viewed as a statistical bundle in the exponential affine geometry. In Aitchison geometry, it is viewed as the tangent space of the simplex. This observation highlights the equivalence of these geometric frameworks when analyzing probability distributions or compositional data.

3.6.1. Mapping of n-Sequences

In Aitchison’s geometry, the coordinates given in the chart above are called centered log-ratio CLR coefficients. In the present section, the CLR map transforms a set of (data) points in the (interior of the) simplex into vectors with $\#\Omega -1$ real-valued components. Statistical models for compositional data are often built and interpreted onto this set of transformed vectors. See, for example, [4,5].

Any n-sequence (n-sample) $q_1,\dots ,q_n\in {\mathcal{P}}_{>}\left(\Omega \right)$ is mapped to an n-sequence in the space of coordinates (contrasts) via the CLR operation as follows:

$$u_1=\mathrm{CLR}\left(q_1\right),\dots ,u_n=\mathrm{CLR}\left(q_n\right)\in T{\mathcal{P}}_{>}\left(\Omega \right).$$

The mapping translates the original probability distributions in the simplex ${\mathcal{P}}_{>}\left(\Omega \right)$ into coordinates in the tangent space $T{\mathcal{P}}_{>}\left(\Omega \right)$.

The space of contrasts $T{\mathcal{P}}_{>}\left(\Omega \right)$ is a vector space with an inner product defined as follows:

$$\langle u,v\rangle =\mathbb{E}\left(uv\right)=\mathrm{Cov}(u,v),$$

In the space of contrasts, $\langle u,v\rangle$ denotes the inner product of vectors u and v, $\mathbb{E}\left(uv\right)$ represents the expected value of their product, and $Cov\left(u,v\right)$ represents their covariance. This inner product allows the computations required by us to perform all computations that require a vector-space structure: (1) vector addition and scalar multiplication are well defined in $T{\mathcal{P}}_{>}\left(\Omega \right)$; (2) the inner product induces a norm, making $T{\mathcal{P}}_{>}\left(\Omega \right)$ a normed vector space; and (3) it can compute angles between vectors and project one vector onto another using this inner product.

In the compositional data literature, the formalism described above is called a Bayes space. See, for example, [6].

3.6.2. Mean, Variance, Velocity, and Velocity Curve

When data are analyzed, providing statistics (data summaries) such as the mean and variances is essential. In Aitchison geometry, these are computed into the tangent space. To fully appreciate their meaning, we highlight that the mean in $T{\mathcal{P}}_{>}\left(\Omega \right)$ is a function of geometric means in the original simplex, ${\mathcal{P}}_{>}\left(\Omega \right)$.

The sample mean of n sequences in centered log-ratio (CLR) coordinates is given as follows:

$${\displaystyle \frac{1}{n}}\sum _{j=1}^n{u}_j={\displaystyle \frac{1}{n}}\sum _{j=1}^n\mathrm{CLR}\left(q_j\right)={\displaystyle \frac{1}{n}}\sum _{j=1}^n\left(log{q}_j-\sum _{y\in \Omega }{\displaystyle \frac{1}{\#\Omega }}log{q}_j\left(y\right)\right).$$

This is simplified to the following:

$${\displaystyle \frac{1}{n}}\sum _{j=1}^n\left(log{q}_j-\sum _{y\in \Omega }{\displaystyle \frac{1}{\#\Omega }}log{q}_j\left(y\right)\right)=log{\left(\prod _{j=1}^n{q}_j\right)}^{1/n}-\sum _{y\in \Omega }{\displaystyle \frac{1}{\#\Omega }}log{\left(\prod _{j=1}^n{q}_j\left(y\right)\right)}^{1/n}.$$

It can also be written as follows:

$$\mathrm{CLR}\left({\left(\prod _{j=1}^n{q}_j\right)}^{1/n}\right)=\mathrm{CLR}\left({\displaystyle \frac{{\left({\prod }_{j=1}^n{q}_j\right)}^{1/n}}{\mathbb{E}\left({\left({\prod }_{j=1}^n{q}_j\right)}^{1/n}\right)}}\right).$$

The sample variance is based on the squared norm:

$$\langle \mathrm{CLR}\left(q\right),\mathrm{CLR}\left(p\right)\rangle =\mathbb{E}\left(\mathrm{CLR}\right(q\left)\mathrm{CLR}\right(r\left)\right)=\mathbb{E}(logqlogr)-\mathbb{E}(logq)\mathbb{E}(logr)=\mathrm{Cov}(logq,logr).$$

For CLR-transformed distributions, $\langle \mathrm{clr}\left(q\right),\mathrm{clr}\left(p\right)\rangle$ denotes their inner product, $\mathbb{E}\left(\mathrm{clr}\right(q\left)\mathrm{clr}\right(r\left)\right)$ is the expectation of their product, $\mathbb{E}(logqlogr)$ is the expectation of the product of log-transformed distributions, $\mathbb{E}(logq)\mathbb{E}(logr)$ is the product of their expectations, and $\mathrm{Cov}(logq,logr)$ represents their covariance.

A statistical model is a curve into ${\mathcal{P}}_{>}\left(\Omega \right)$, often inverse-mapped from a statistical bundle in $T{\mathcal{P}}_{>}\left(\Omega \right)$. In turn, the statistical bundle is estimated from some CLR-transformed data points.

Most differential algorithms for model estimation and optimization are based on the computation of the parametric rate of change of statistics, which, in turn, are based on the computation of velocities, accelerations, and gradients.

The velocity is as follows:

$${\displaystyle \frac{D}{dt}}q\left(t\right)={\displaystyle \frac{d}{dt}}\mathrm{CLR}\left(q\left(t\right)\right)={\displaystyle \frac{\dot{q}\left(t\right)}{q\left(t\right)}}-\mathbb{E}\left({\displaystyle \frac{\dot{q}\left(t\right)}{q\left(t\right)}}\right),$$

where ${\displaystyle \frac{D}{dt}}q\left(t\right)$ represents the velocity of the distribution $q\left(t\right)$, ${\displaystyle \frac{d}{dt}}\mathrm{clr}\left(q\left(t\right)\right)$ is the time derivative of the clr-transformed distribution $q\left(t\right)$, ${\displaystyle \frac{\dot{q}\left(t\right)}{q\left(t\right)}}$ denotes the element-wise derivative of $q\left(t\right)$ divided by $q\left(t\right)$, and $\mathbb{E}\left({\displaystyle \frac{\dot{q}\left(t\right)}{q\left(t\right)}}\right)$ is the expectation of ${\displaystyle \frac{\dot{q}\left(t\right)}{q\left(t\right)}}$, ensuring that the result has null expectation.

The acceleration is defined in the usual way. The velocity curve $t\mapsto {\displaystyle \frac{D}{dt}}q\left(t\right)$ is a curve in a vector space, and its derivative is the acceleration,

$${\displaystyle \frac{d}{dt}}{\displaystyle \frac{D}{dt}}q\left(t\right)={\displaystyle \frac{\ddot{q}\left(t\right)}{q\left(t\right)}}-{\left({\displaystyle \frac{\dot{q}\left(t\right)}{q\left(t\right)}}\right)}^2-\mathbb{E}\left({\displaystyle \frac{\ddot{q}\left(t\right)}{q\left(t\right)}}-{\left({\displaystyle \frac{\dot{q}\left(t\right)}{q\left(t\right)}}\right)}^2\right).$$

In this context, ${\displaystyle \frac{d}{dt}}{\displaystyle \frac{D}{dt}}q\left(t\right)$ denotes the acceleration of the distribution $q\left(t\right)$, with ${\displaystyle \frac{\ddot{q}\left(t\right)}{q\left(t\right)}}$ representing the element-wise second derivative of $q\left(t\right)$ normalized by $q\left(t\right)$, ${\left({\displaystyle \frac{\dot{q}\left(t\right)}{q\left(t\right)}}\right)}^2$ as the square of the element-wise first derivative term and $\mathbb{E}\left({\displaystyle \frac{\ddot{q}\left(t\right)}{q\left(t\right)}}-{\left({\displaystyle \frac{\dot{q}\left(t\right)}{q\left(t\right)}}\right)}^2\right)$ as the expectation of the acceleration term, ensuring that the result is centered.

Notice that the velocity is a deviation from a mean value in the uniform probability. The corresponding intensity is the variance of ${\displaystyle \frac{d}{dt}}logq\left(t\right)$ in the uniform probability. This is a legitimate and consistent estimation of the rate of variation, which is different from the more standard estimation using Fisher’s information, which is a variance computed in the current probability, not the uniform probability.

The curves of zero acceleration are the curves of constant velocity. Such curves are commonly called geodesics. A geodesic from $p=\mathrm{CLR}\left(u\right)$ and initial velocity v has the following form:

$$q\left(t\right)={CLR}^{-1}(tv+u)=exp(tu-K_p\left(u\right))·p,$$

where ${CLR}^{-1}$ denotes the inverse centered log-ratio invertible chart, u represents the initial point in CLR coordinates, v indicates the initial velocity in CLR coordinates, $K_p\left(u\right)$ serves as the normalizing constant, and $exp(tu-K_p\left(u\right))·p$ defines the geodesic point at time t.

Most optimization algorithms look for a critical point of a scalar field, that is, a point where the gradient is null. Given a definition of the inner product in the vector space of an affine space, the gradient of a real function $\Phi$ on the base set (scalar field) is a mapping $\mathrm{grad}\Phi$ from the base set to the vector space or the tangent space, that is, a vector field, such that, for all smooth curves $q\left(t\right)$, the following holds:

$${\displaystyle \frac{d}{dt}}\Phi \left(q\left(t\right)\right)=〈grad\Phi \left(q\left(t\right)\right),{\displaystyle \frac{D}{dt}}q\left(t\right)〉,$$

where $grad\Phi$ denotes the gradient of $\Phi$, mapping from the base set to the tangent space to create a vector field. For a smooth curve, $q\left(t\right)$, ${\displaystyle \frac{d}{dt}}\Phi \left(q\left(t\right)\right)$ represents the time derivative of $\Phi$ along $q\left(t\right)$, while the inner product $〈\mathrm{grad}\Phi \left(q\left(t\right)\right),{\displaystyle \frac{D}{dt}}q\left(t\right)〉$ quantifies the rate of change of $\Phi$ in the direction of the velocity ${\displaystyle \frac{D}{dt}}q\left(t\right)$.

In Aitchison geometry, if $CLR\left(q\right(t\left)\right)=u\left(t\right)$ and $q\left(t\right)={CLR}^{-1}\left(u\left(t\right)\right)$, then the following applies:

$${\displaystyle \frac{d}{dt}}\Phi \circ {\mathrm{CLR}}^{-1}\left(u\left(t\right)\right)=\mathbb{E}\left(\mathrm{grad}\Phi \circ {\mathrm{CLR}}^{-1}\left(u\left(t\right)\right)\dot{u}\left(t\right)\right)={\displaystyle \frac{1}{\#\Omega }}\sum _{x\in \Omega }\mathrm{grad}\Phi \circ {\mathrm{CLR}}^{-1}\left(u(x;t)\right)\dot{u}(x;t).$$

The composition $\Phi \circ {\mathrm{CLR}}^{-1}$ represents the function $\Phi$ applied to the inverse CLR transformation, where $\mathbb{E}\left(\mathrm{grad}\Phi \circ {\mathrm{CLR}}^{-1}\left(u\left(t\right)\right)\dot{u}\left(t\right)\right)$ denotes the expected value of the product of the gradient of $\Phi$ composed with ${\mathrm{CLR}}^{-1}$, and the derivative of $u\left(t\right)$; ${\displaystyle \frac{1}{\#\Omega }}{\sum }_{x\in \Omega }$ represents the average over all elements x in $\Omega$, with $\mathrm{grad}\Phi \circ {\mathrm{CLR}}^{-1}\left(u(x;t)\right)$ being the gradient of $\Phi$ composed with ${\mathrm{CLR}}^{-1}$ at the point $u(x;t)$, and $\dot{u}(x;t)$ denoting the time derivative of $u(x;t)$.

It follows that

$${\displaystyle \frac{\partial }{\partial u\left(x\right)}}\Phi \circ {\mathrm{CLR}}^{-1}\left(u\right)={\displaystyle \frac{1}{\#\Omega }}\mathrm{grad}\Phi \circ {\mathrm{CLR}}^{-1}\left(u\right)\left(x\right).$$

or, equivalently,

$${\displaystyle \frac{1}{\#\Omega }}\mathrm{grad}\Phi \left(q\right)=\nabla (\Phi \circ {\mathrm{CLR}}^{-1})\circ \mathrm{CLR},$$

where ${\displaystyle \frac{\partial }{\partial u\left(x\right)}}\Phi \circ {\mathrm{CLR}}^{-1}\left(u\right)$ represents the partial derivative of $\Phi \circ {\mathrm{CLR}}^{-1}$ with respect to $u\left(x\right)$, ${\displaystyle \frac{1}{\#\Omega }}\mathrm{grad}\Phi \circ {\mathrm{CLR}}^{-1}\left(u\right)\left(x\right)$ is the averaged gradient of $\Phi$ composed with ${\mathrm{CLR}}^{-1}$ at $u\left(x\right)$, $\nabla (\Phi \circ {\mathrm{CLR}}^{-1})$ denotes the gradient of the composed function $\Phi \circ {\mathrm{CLR}}^{-1}$, and $\nabla (\Phi \circ {\mathrm{CLR}}^{-1})\circ \mathrm{CLR}$ is the gradient applied to CLR-transformed variables.

4. Product-Sample Spaces

The traditional presentation of the probability simplex’s convex geometry uses the language of linear programming (LP); see, for example, [7] (Ch. IV). The following section is an aside about LP from [7] (§ IV.10) to be skipped if the reader knows this topic.

4.1. Short Recap of Linear Programming

Linear programming deals with function optimization under equality and inequality constraints, where both the objective function and the constraints are linear functions. Every LP problem admits two forms: primal and dual forms.

The primal problem in the canonical form is

$$\begin{array}{c}\mathrm{Find}c=inf\sum _{x}c\left(x\right)r\left(x\right)\\ \mathrm{Subject}\mathrm{to}\sum _{x\in X}A(y,x)r\left(x\right)=\beta \left(y\right),\mathrm{for}\mathrm{all}y\in Y\mathrm{and}r\left(x\right)\ge 0\mathrm{for}\mathrm{all}x\in X,\end{array}$$

where X and Y are finite sets, A, $\beta$, and c are given real functions that define the problem, and r is a vector is the primal plan, representing the decision variables in the optimization problem.

Y typically represents the index set for the constraints in the optimization problem. Specifically, Y is the set of indices that correspond to the linear constraints ${\sum }_{x\in X}A(y,x)r\left(x\right)=\beta \left(y\right)$ that must be satisfied for each $y\in Y$. Each y in Y corresponds to a different constraint, where $A(y,x)$ represents the coefficients in the constraint matrix, and $\beta \left(y\right)$ represents the right-hand side values for the constraints.

The objective is to find the value c, which is the lower bound of the total cost ${\sum }_{x}c\left(x\right)r\left(x\right)$, where $c\left(x\right)$ represents the cost associated with x, and $r\left(x\right)$ represents the amount or level of activity at x. A plan, r, is feasible if it satisfies both constraints. That is, the linear equations must hold for all $y\in Y$, and $r\left(x\right)$ must be non-negative for all $x\in X$.

The dual problem in standard form is as follows:

$$\begin{array}{c}\mathrm{Find}\beta =sup\sum _y\beta \left(y\right)\lambda \left(y\right)\\ \mathrm{Subject}\mathrm{to}\sum _{y\in Y}A(y,x)\lambda \left(y\right)\le c\left(x\right)\mathrm{for}\mathrm{all}x\in X,\end{array}$$

where $\lambda$ is the dual plan. The objective is to find the value $\beta$, which is the upper bound of the sum ${\sum }_y\beta \left(y\right)\lambda \left(y\right)$.

Here, $\beta \left(y\right)$ represents the benefit or value associated with y, and $\lambda \left(y\right)$ represents the level or intensity at y. The constraint ${\sum }_{y}A(y,x)\lambda \left(y\right)\le c\left(x\right)$ ensures that the linear combination of $\lambda \left(y\right)$ with coefficients $A(y,x)$ does not exceed $c\left(x\right)$ for each $x\in X$.

A general theorem relates the primal and dual problems. The strong duality theorem states that the values are equal, $c=\beta$, provided that a feasible primal plan exists. If, moreover, $c>-\infty$, then primal and dual optimal plans exist.

4.2. Product Sample Space

In this section, we assume a product of two finite sample spaces, $\Omega ={\Omega }_1\times {\Omega }_2$, and discuss the probability simplex on such a sample space, $\mathcal{P}\left({\Omega }_1\times {\Omega }_2\right)$.

The two marginalization mappings are,

$$\begin{array}{c}{\Pi }_1:\mathcal{P}({\Omega }_1\times {\Omega }_2)\ni q(·,·)\mapsto \sum _{{\omega }_2}q(·,{\omega }_2)\in \mathcal{P}\left({\Omega }_1\right),\\ {\Pi }_2:\mathcal{P}({\Omega }_1\times {\Omega }_2)\ni q(·,·)\mapsto \sum _{{\omega }_1}q({\omega }_1,·)\in \mathcal{P}\left({\Omega }_2\right),\end{array}$$

and the product marginalization mapping is

$$\Pi :\mathcal{P}\left(\Omega \right)\ni q\mapsto ({\Pi }_{1}q,{\Pi }_{2}q)\in \mathcal{P}\left({\Omega }_1\right)\times \mathcal{P}\left({\Omega }_2\right).$$

Given marginal distributions $q_1\in \mathcal{P}{\left(\Omega \right)}_1$ and $q_2\in \mathcal{P}{\left(\Omega \right)}_2$, their counter images for $\Pi$ are called transport plans, and

$$\Pi (q_1,q_2)=\{q\in \mathcal{P}({\Omega }_1\times {\Omega }_2)\mid {\Pi }_{1}q=q_1,{\Pi }_{2}q=q_2\}.$$

(32)

is the set of transport plans. It contains the probability functions of all joint distributions on ${\Omega }_1\times {\Omega }_2$ with the marginals specified as $q_1$ and $q_2$. For each $q\in \Pi (q_1,q_2)$ the conditional ${\omega }_1\mapsto q({\omega }_1,·)/q_1\left(\omega \right)$ is seen as a rule to move a given mass from ${\omega }_1$ to ${\Omega }_2$, which justifies the name of the transport plan.

The set of transport plans, $\Pi (q_1,q_2)$, is non-empty (because it contains the product of the two given marginals), and it is a convex, closed, and bounded set in the real space (because it is the intersection of the probability simplex with a hyperplane).

We can rewrite the marginalization conditions using the indicator function $(\xi =\omega )$, which is 1 if $\xi =\omega$ and 0 otherwise,

$$\begin{array}{c}\sum _{({\omega }_1,{\omega }_2)\in {\Omega }_1\times {\Omega }_2}({\xi }_1={\omega }_1)q({\omega }_1,{\omega }_2)=q_1\left({\xi }_1\right),\\ \sum _{({\omega }_1,{\omega }_2)\in {\Omega }_1\times {\Omega }_2}({\xi }_2={\omega }_2)q({\omega }_1,{\omega }_2)=q_2\left({\xi }_2\right).\end{array}$$

Define a function, $A:({\Omega }_1\cup {\Omega }_2)\times ({\Omega }_1\times {\Omega }_2)\to \mathbb{R}$, as

$$\sum _{({\omega }_1,{\omega }_2)\in {\Omega }_1\times {\Omega }_2}A(\xi ;{\omega }_1,{\omega }_2)q({\omega }_1,{\omega }_2)=(q_1\cup q_2)\left(\xi \right),$$

that is,

$$A(\xi ;{\omega }_1,{\omega }_2)=\left\{\begin{array}{cc}({\xi }_1={\omega }_1),\hfill & \mathrm{if}\xi \in {\Omega }_1,\hfill \\ ({\xi }_2={\omega }_2),\hfill & \mathrm{if}\xi \in {\Omega }_2,\hfill \end{array}\right.$$

so that we can write the marginalization equations into a single equation,

$$\sum _{({\omega }_1,{\omega }_2)\in {\Omega }_1\times {\Omega }_2}A(\xi ;{\omega }_1,{\omega }_2)q({\omega }_1,{\omega }_2)=(q_1\cup q_2)\left(\xi \right),$$

which provides a better form of LP formalism.

Given the transport cost $c:{\Omega }_1\times {\Omega }_2\to \mathbb{R}$, the optimal transport problem or Kantorovich problem looks for a transport plan with a minimal expected cost for a transport plan with given marginals, that is, with value the $inf{\mathbb{E}}_q\left[c\right]$, where q varies in $\Pi (q_1,q_2)$. It is a primal problem in canonical form,

$$\begin{array}{c}\mathrm{Find}c=inf\sum _{{\omega }_1,{\omega }_2}c({\omega }_1,{\omega }_2)q({\omega }_1,{\omega }_2),\end{array}$$

(33)

$$\begin{array}{c}\mathrm{Subject}\mathrm{to}\sum _{({\omega }_1,{\omega }_2)\in {\Omega }_1\times {\Omega }_2}A(\xi ;{\omega }_1,{\omega }_2)q({\omega }_1,{\omega }_2)=(q_1\cup q_2)\left(\xi \right),q({\omega }_1,{\omega }_2)\ge 0.\end{array}$$

(34)

The dual problem in standard form is

$$\begin{array}{c}\mathrm{Find}\beta =sup\left(\sum _{{\xi }_1\in {\Omega }_1}{q}_1\left({\xi }_1\right){\lambda }_1\left({\xi }_1\right)+\sum _{{\xi }_2\in {\Omega }_2}{q}_2\left({\xi }_2\right){\lambda }_2\left({\xi }_2\right)\right),\end{array}$$

(35)

$$\begin{array}{c}\mathrm{Subject}\mathrm{to}{\lambda }_1\left({\omega }_1\right)+{\lambda }_2\left({\omega }_2\right)\le c({\omega }_1,{\omega }_2).\end{array}$$

(36)

A feasible transport plan exists, and the set of transport plans is closed and bounded. Hence, according to the strong duality theorem, the optimal values of the primal and dual problems are equal, and both primal and dual optimal solutions exist.

We can derive a more explicit statement. The optimal value of the primal problem equals the optimal value of the dual problem, namely

$$\begin{array}{c}\sum _{{\omega }_1,{\omega }_2}c({\omega }_1,{\omega }_2)\overline{q}({\omega }_1,{\omega }_2)=\sum _{{\omega }_1\in {\Omega }_1}{q}_1\left({\omega }_1\right){\overline{\lambda }}_1\left({\omega }_1\right)+\sum _{{\omega }_2\in {\Omega }_2}{q}_2\left({\omega }_2\right){\overline{\lambda }}_2\left({\omega }_2\right)=\hfill \\ \sum _{{\omega }_1,{\omega }_2}({\overline{\lambda }}_1\left({\omega }_1\right)+{\overline{\lambda }}_2\left({\omega }_2\right))\overline{q}({\omega }_1,{\omega }_2).\hfill \end{array}$$

where $\overline{q}$ is the optimal transport plan, and ${\overline{\lambda }}_1\left({\omega }_1\right)$ are the dual variables.

The inequality

$$c\ge {\lambda }_1\oplus {\lambda }_2({\omega }_1,{\omega }_2)={\lambda }_1\left({\omega }_1\right)+{\lambda }_2\left({\omega }_2\right)$$

ensures that, for the optimal transport plan, the cost $c({\omega }_1,{\omega }_2)$ is equal to the sum of the dual variables ${\overline{\lambda }}_1\left({\omega }_1\right)$ and ${\overline{\lambda }}_2\left({\omega }_2\right)$ whenever $\overline{q}({\omega }_1,{\omega }_2)\ne 0$, namely

$$c({\omega }_1,{\omega }_2)={\overline{\lambda }}_1\left({\omega }_1\right)+{\overline{\lambda }}_2\left({\omega }_2\right)\mathrm{provided}\overline{q}({\omega }_1,{\omega }_2)\ne 0.$$

In the transport literature, the equation above is called Kantorovich’s theorem, which specifies that the cost equals the sum of two single variable functions in support of an optimal plan. See, for example, [31].

4.3. ANOVA Splitting of the Statistical Bundle

In this section, we use the language of effects’ decomposition to analyze categorical tabular data. A classical treatment of this topic is [32].

The space of real random variables on ${\Omega }_1\times {\Omega }_2$ is endowed with the inner product with weights provided via a strictly positive density, $q\in {\mathcal{P}}_{>}\left({\Omega }_1\times {\Omega }_2\right)$, and the resulting Euclidean space is denoted as $L^2\left(q\right)$. The linear sub-spaces of $L^2\left(q\right)$, which respectively express the q-grand mean, the two q-simple effects, and the q-interaction, are

$$\begin{array}{cc}\hfill B_0\left(q\right)& \sim \mathbb{R},\hfill \\ \hfill B_1\left(q\right)& =\{f_1\circ {\Pi }_1\mid f_1\in L_0^2\left(q_1\right)\},\hfill \\ \hfill B_2\left(q\right)& =\{f_2\circ {\Pi }_2\mid f_2\in L_0^2\left(q_2\right)\},\hfill \\ \hfill B_{12}\left(q\right)& ={(B_0\left(q\right)+B_1\left(q\right)+B_2\left(q\right))}^{\perp },\hfill \end{array}$$

where ∘ stands for the operation of functions composition, $f\circ g\left(x\right)=f\left(g\right(x\left)\right)$, and the orthogonality is computed in the q-probability function, that is, in the inner product of $L^2\left(q\right)$, for which ${\langle u,v\rangle }_q={\mathbb{E}}_q\left[uv\right]$. Each element of the space $B_0\left(q\right)+B_1\left(q\right)+B_2\left(q\right)$ has the form $u=u_0+f_1\circ {\Pi }_1+f_2\circ {\Pi }_2$, where $u_0={\mathbb{E}}_q\left[u\right]$ and $f_1,f_2$ are uniquely defined.

For each $q\in {\mathcal{P}}_{>}\left(\Omega \right)$, there exists a unique orthogonal splitting:

$$L^2\left(q\right)=\mathbb{R}\oplus (B_1\left(q\right)+B_2\left(q\right))\oplus B_{12}\left(q\right).$$

Namely, each $u\in L^2\left(q\right)$ can be written uniquely as follows:

$$u=u_0+(u_1+u_2)+u_{12},$$

where $u_0=E_q\left[u\right]$, and $(u_1+u_2)$ is the q-orthogonal projection of $u-u_0$ onto $(B_1\left(q\right)+B_2\left(q\right))$. That is,

$${\mathbb{E}}_q\left(u_{12}|{\Pi }_1\right)=0\mathrm{and}{\mathbb{E}}_q\left(u_{12}|{\Pi }_2\right)=0.$$

In conclusion, the ANOVA splitting of the statistical bundle is

$$S_q{\mathcal{P}}_{>}\left(\Omega \right)=(B_1\left(q\right)+B_2\left(q\right))\oplus B_{12}\left(q\right),$$

(37)

and we can interpret each factor in the decomposition as a vector space of velocities.

Example 16.

Given $q_1\in {\mathcal{P}}_{>}\left({\Omega }_1\right)$ and $q_2\in {\mathcal{P}}_{>}\left({\Omega }_2\right)$, consider ${\Pi }^{\circ }(q_1,q_2)$ to be the open set consisting of all (strictly) positive probability densities in the set of transport plans $\Pi (q_1,q_2)$.

Consider a smooth curve with values in the interior of the open set of transport plans $t\mapsto q\left(t\right)\in {\Pi }^{\circ }(q_1,q_2)$ with $q\left(0\right)=q$. The velocity at q is given by $\stackrel{★}{q}\left(0\right)$. Such a velocity is an interaction, $\stackrel{★}{q}\left(0\right)\in B_{12}\left(q\left(0\right)\right)$. In fact, ${\mathbb{E}}_{q_0}\left[\stackrel{★}{q}\left(0\right)\right]$, and for all $f_i=g_i\circ {\Pi }_i\in B_i$, $i=1,2$,

$${〈\stackrel{★}{q}\left(0\right),f_i〉}_{q\left(0\right)}={\displaystyle \frac{d}{dt}}{\mathbb{E}}_{q\left(t\right)}\left[f_i\right]={\displaystyle \frac{d}{dt}}{\mathbb{E}}_{q_i}\left[g_i\right]=0.$$

Conversely, given any interaction $v\in B_{12}\left(q\right)$, consider the curve $t\mapsto q\left(t\right)=(1+tv)q$. This curve stays in ${\Pi }^{\circ }(q_1,q_2)$ for t in a neighborhood of 0. Moreover, the interaction v is equal to the velocity at $t=0$; i.e., $v=\stackrel{★}{q}\left(0\right)$.

We define the statistical bundle of positive transport plans as

$$\begin{array}{c}S{\Pi }^{\circ }(q_1,q_2)=\left\{(q,v)|q\in {\Pi }^{\circ }(q_1,q_2),v\in B_{12}\left(q\right)\right\}=\hfill \\ \left\{(q,v)|q\in {\Pi }^{\circ }(q_1,q_2),{\mathbb{E}}_q\left(v|{\Pi }_1\right)={\mathbb{E}}_q\left(v|{\Pi }_2\right)=0\right\}.\hfill \end{array}$$

The mixture transport can be restricted to $S{\Pi }^{\circ }(q_1,q_2)$ to provide a parallel transport between the fibers. If $q,r\in {\Pi }^{\circ }(q_1,q_2)$ and $v\in B_{12}\left(q\right)$, let us compute $w={{}^{\mathrm{m}}\mathbb{U}}_q^{r}v={\displaystyle \frac{q}{r}}v$ and check that $w\in B_{12}\left(r\right)$. If f is a function of ${\Omega }_i$ only, $i=1,2$, then

$${〈w,f〉}_r={\mathbb{E}}_r\left[{\displaystyle \frac{q}{r}}vf\right]={\mathbb{E}}_q\left[vf\right]={〈v,f〉}_q=0.$$

In particular, it follows that the mixture affine displacement $(q,r)\mapsto {\eta }_q\left(r\right)={\displaystyle \frac{r}{q}}-1$ is well defined in the open transport plan; that is, ${\eta }_q\left(r\right)\in B_{12}\left(q\right)$. In fact, for all $f=g\circ {\Pi }_i\in B_i\left(q\right)$, $i=1,2$,

$${〈{\displaystyle \frac{r}{q}}-1,f〉}_q={\mathbb{E}}_q\left[\left({\displaystyle \frac{r}{q}}-1\right)f\right]={\mathbb{E}}_r\left[f\right]-{\mathbb{E}}_p\left[f\right]={\mathbb{E}}_{q_i}\left[g\right]-{\mathbb{E}}_{q_i}\left[g\right]=0.$$

The open transport statistical model is a sub-affine space of the mixture affine space. The literature frequently describes it as “flat” in the sense that each mixture’s auto-parallel curve between two points with the same margins has constant margins.

Example 17

(Mixed parametrization). The ANOVA splitting of the statistical bundle (37) suggests a mixed parameterization of ${\mathcal{P}}_{>}\left({\Omega }_1\times {\Omega }_2\right)$. Mixed parametrization combines elements of different parameterization schemes to describe the space. This approach is classical in the statistics of contingency tables, where mixed parametrizations are often used to model complex dependencies and interactions between variables.

For each reference $q\in {\Pi }^{\circ }(q_1,q_2)$, consider the probability densities of the form

$$r={\mathrm{e}}^{v_1+v_2-K_p(v_1+v_2)}·q,v_1+v_2\in B_1\left(q\right)+B_2\left(q\right).$$

The set of such densities is a sub-manifold of the exponential affine manifold usually termed an additive model (§ 17.5 in [18]). The set of velocities at q is q-orthogonal to the space of transport plans that contain q.

Example 18

(Optimal transport). Kantorovich’s problem can be studied as a gradient-flow problem. Let $c:{\Omega }_1\times {\Omega }_2\to \mathbb{R}$ be a transport cost with ANOVA decomposition at q:

$$c=c_{0,q}+c_{1,q}+c_{2,q}+c_{12,q}.$$

The function $q\mapsto C\left(q\right)={\mathbb{E}}_q\left[c\right]$ restricted to the open transport model ${\Pi }^{\circ }(q_1,q_2)$ has a (statistical) gradient in $S{\Pi }^{\circ }(q_1,q_2)$. For each generic smooth curve in ${\Pi }^{\circ }(q_1,q_2)$ with $q\left(0\right)=q$, namely

$${\displaystyle \frac{d}{dt}}C\left(q\left(t\right)\right)={\displaystyle \frac{d}{dt}}{\mathbb{E}}_{q\left(t\right)}\left[c\right]={〈\stackrel{★}{q}\left(t\right),c〉}_{q\left(t\right)}={〈\stackrel{★}{q}\left(t\right),c_{12,q\left(t\right)}〉}_{q\left(t\right)}.$$

The natural gradient in the bundle is

$$gradC:{\Pi }^{\circ }(q_1,q_2)\ni q\mapsto c_{12,q}=c-c_{0,q}-(c_{1,q}+c_{2,q}),$$

and the gradient flow equation is

$$\stackrel{★}{q}\left(t\right)=-c_{12,q\left(t\right)}.$$

The natural gradient mapping $q\mapsto c_{12,q}$ is an orthogonal projection on the complete set of transport plans $\Pi (q_1,q_2)$. It is remarkable to note that, in the more extensive set, the condition of zero gradients is compatible with the form of the optimal transport cost, according to Kantorivich.

5. Second-Order Computations and Mechanical Models

The notion of acceleration in the statistical bundle was introduced in Example 12 via the simple idea of computing the velocity’s velocity. This was feasible because we deal with affine spaces. This section analyzes the definition in detail and considers its possible application to models inspired by mechanics. The term “mechanics” refers to the applications discussed, for example, the notion of a curve with minimal Fisher information, where we interpret the Fisher information along a curve as a Lagrangian function. This section requires an introductory notion of the language of analytic mechanics. See, for example, the monograph [33] or the introductions of the papers [12,34].

Consider a statistical bundle, $S{\mathcal{P}}_{>}\left(\Omega \right)$, with a cocycle of parallel transports, ${\mathbb{U}}_p^q$, $p,q\in {\mathcal{P}}_{>}\left(\Omega \right)$, and a compatible affine atlas, $s_p$, $p\in {\mathcal{P}}_{>}\left(\Omega \right)$. The statistical bundle is an affine space for the charts

$$s_{p,u}:S{\mathcal{P}}_{>}\left(\Omega \right)\ni (q,v)\mapsto \left(s_p\left(q\right),{\mathbb{U}}_q^{p}u-v\right)\in S_p{\mathcal{P}}_{>}\left(\Omega \right)\times S_p{\mathcal{P}}_{>}\left(\Omega \right).$$

If $\mathsf{\Gamma }:t\mapsto \left(q\right(t),v(t\left)\right)$ is a curve in the statistical bundle, its expression at $(p,u)$ is

$${\mathsf{\Gamma }}_{p,u}:t\mapsto \left(s_p\left(q\left(t\right)\right),{{}^{\mathrm{e}}\mathbb{U}}_{q\left(t\right)}^{p}v\left(t\right)-u\right)$$

and the expression of the velocity at $(p,u)$ is

$${\displaystyle \frac{d}{dt}}{\mathsf{\Gamma }}_{p,u}\left(t\right)=\left({\displaystyle \frac{d}{dt}}{s}_p\left(q\left(t\right)\right),{\displaystyle \frac{d}{dt}}{\mathbb{U}}_{q\left(t\right)}^{p}v\left(t\right)\right).$$

The velocity in the moving frame centered at $(p,u)=\left(q\right(t),v(t\left)\right)$ is

$$\stackrel{★}{\mathsf{\Gamma }}\left(t\right)=\left(\stackrel{★}{q}\left(t\right),{\left.{\displaystyle \frac{d}{dt}}{\mathbb{U}}_{q\left(t\right)}^{p}v\left(t\right)\right|}_{p=q\left(t\right)}\right).$$

Especially if $\mathsf{\Gamma }\left(t\right)=(q\left(t\right),\stackrel{★}{q}\left(t\right))$, we have the definition of affine acceleration,

$$\stackrel{**}{q}\left(t\right)={\left.{\displaystyle \frac{d}{dt}}{\mathbb{U}}_{q\left(t\right)}^p\stackrel{★}{q}\left(t\right)\right|}_{p=q\left(t\right)}.$$

(38)

In conclusion, each system of parallel transports has its own acceleraation. Precisely, the following applies:

• The exponential acceleration is

  $$\begin{array}{c}{\left.{\displaystyle \frac{d}{dt}}{{}^e\mathbb{U}}_{q\left(t\right)}^p\stackrel{★}{q}\left(t\right)\right|}_{p=q\left(t\right)}={\left.{\displaystyle \frac{d}{dt}}\left({\displaystyle \frac{\dot{q}\left(t\right)}{q\left(t\right)}}-{\mathbb{E}}_p\left[{\displaystyle \frac{\dot{q}\left(t\right)}{q\left(t\right)}}\right]\right)\right|}_{p=q\left(t\right)}=\hfill \\ {\left.{\displaystyle \frac{\ddot{q}\left(t\right)q\left(t\right)-{\left(\dot{q}\left(t\right)\right)}^2}{q{\left(t\right)}^2}}-{\mathbb{E}}_p\left[{\displaystyle \frac{\ddot{q}\left(t\right)q\left(t\right)-{\left(\dot{q}\left(t\right)\right)}^2}{q{\left(t\right)}^2}}\right]\right|}_{p=q\left(t\right)}=\hfill \\ {\displaystyle \frac{\ddot{q}\left(t\right)}{q\left(t\right)}}-\left({\left(\stackrel{★}{q}\left(t\right)\right)}^2-{\mathbb{E}}_{q\left(t\right)}\left[{\left(\stackrel{★}{q}\left(t\right)\right)}^2\right]\right).\hfill \end{array}$$

  (39)
• The mixture acceleration is

  $${\left.{\displaystyle \frac{d}{dt}}{{}^{\mathrm{m}}\mathbb{U}}_{q\left(t\right)}^p\stackrel{★}{q}\left(t\right)\right|}_{p=q\left(t\right)}={\left.{\displaystyle \frac{d}{dt}}{\displaystyle \frac{q\left(t\right)}{p}}{\displaystyle \frac{\dot{q}\left(t\right)}{q\left(t\right)}}\right|}_{p=q\left(t\right)}={\left.{\displaystyle \frac{d}{dt}}{\displaystyle \frac{\dot{q}\left(t\right)}{p}}\right|}_{p=q\left(t\right)}={\displaystyle \frac{\ddot{q}\left(t\right)}{q\left(t\right)}}.$$

  (40)

Remark 1.

It should be noted that, because of the very definition, the acceleration is zero if and only if the curve has a constant velocity in the proper parallel transport; that is, $\stackrel{★}{q}\left(s\right)={\mathbb{U}}_{q\left(t\right)}^{q\left(s\right)}\stackrel{★}{q}\left(t\right)$. In particular, the exponential acceleration is null for 1-dimensional exponential models, and the mixture acceleration is null for 1-dimensional mixture models.

The acceleration is appropriately defined as a triple of curves: the model, the velocity, and the acceleration. It is convenient to define a vector bundle whose fibers are products of two fibers of the statistical bundle. Moreover, to identify each fiber and to prepare for the use of the covariance inner product as a duality pairing, the two fibers are denoted, respectively, as $S_p{\mathcal{P}}_{>}\left(\Omega \right)$ and ${{}^{*}S}_p{\mathcal{P}}_{>}\left(\Omega \right)$.

The full bundle is

$${{}^{1}S}^1{\mathcal{P}}_{>}\left(\Omega \right)=\left\{(q,\eta ,v)|q\in {\mathcal{P}}_{>}\left(\Omega \right),\eta \in {{}^{*}S}_q{\mathcal{P}}_{>}\left(\Omega \right),v\in S_q{\mathcal{P}}_{>}\left(\Omega \right)\right\}.$$

With such a notation, any twice-smooth curve, q, is lifted to a curve in the full bundle,

$$(q,\stackrel{**}{q},\stackrel{★}{q})\in {{}^{1}S}^1{\mathcal{P}}_{>}\left(\Omega \right).$$

Remark 2.

The derivation of curves in the statistical bundle is also called covariant derivation. The treatment of covariant derivatives in standard monographs of differential geometry, such as, for example, [14,35], is considerably more involved than ours because we are considering the particular case of affine manifold endowed with an inner product instead of a general differentiable manifold or a general Riemannian manifold. In some cases, we must distinguish between exponential and mixture covariant derivatives.

Below, we provide an example of a problem of interest that requires a second-order formalism. After that, we discuss how general arguments originating in mechanics can help solve such problems.

5.1. Curves with Minimal Fisher’s Information

Let us consider a smooth curve in the open probability simplex, $q:[0,1]\ni t\mapsto q\left(t\right)\in {\mathcal{P}}_{>}\left(\Omega \right)$, connecting $q_0=q\left(0\right)$ to $q_1=q\left(1\right)$. The velocity of the curve in the statistical bundle is $]0,1[\ni t\mapsto \stackrel{★}{q}\left(t\right)={\displaystyle \frac{d}{dt}}logq\left(t\right)\in S_{q\left(t\right)}{\mathcal{P}}_{>}\left(\Omega \right)$, the Fisher’s score. Each fiber $S_q{\mathcal{P}}_{>}\left(\Omega \right)$ of the statistical bundle $S{P}_{>}(\Omega )$ is endowed with the covariance inner product ${〈v,w〉}_q={Cov}_q\left(v,w\right)$, which defines the Riemannian metric of the statistical bundle.

With a mechanical analogy, the (kinetic) energy of the curve $q(·)$ is the squared norm of the velocity, while the integral of the energy along the curve is the action,

$$L\left(q(·)\right)={\int }_1^0{〈\stackrel{★}{q}\left(t\right),\stackrel{★}{q}\left(t\right)〉}_{q\left(t\right)}dt={\int }_0^1\sum _{\omega \in \Omega }{\displaystyle \frac{\dot{q}{(\omega ;t)}^2}{q(\omega ;t)}}dt.$$

(41)

We can find a necessary condition for a minimum of the action by performing the classical Euler computation in the current context. Consider a curve in the statistical bundle $t\mapsto (q\left(t\right),v\left(t\right))\in S{\mathcal{P}}_{>}\left(\Omega \right)$, such that $v\left(0\right)=v\left(1\right)=0$. For each real $\theta$, let us consider

$$t\mapsto q_{\theta }\left(t\right)=e^{\theta v\left(t\right)-K_{q\left(t\right)}\left(\theta v\left(t\right)\right)}·q\left(t\right).$$

Such a perturbed smooth curve has the same endpoints as $q(·)$, and the velocity is

$$\begin{array}{c}{\stackrel{★}{q}}_{\theta }\left(t\right)={\displaystyle \frac{d}{dt}}log{q}_{\theta }\left(t\right)=\theta {\displaystyle \frac{d}{dt}}v\left(t\right)-{\displaystyle \frac{d}{dt}}{K}_{q\left(t\right)}\left(\theta v\left(t\right)\right)+{\displaystyle \frac{d}{dt}}logq\left(t\right)=\hfill \\ \theta {\displaystyle \frac{d}{dt}}v\left(t\right)-{\displaystyle \frac{d}{dt}}{K}_{q\left(t\right)}\left(\theta v\left(t\right)\right)+\stackrel{★}{q}\left(t\right).\hfill \end{array}$$

At any external point of the action, we must have

$${\left.{\displaystyle \frac{d}{d\theta }}{\int }_0^1{\langle \stackrel{★}{q_{\theta }}\left(t\right),\stackrel{★}{q_{\theta }}\left(t\right)\rangle }_{q_{\theta }\left(t\right)}dt\right|}_{\theta =0}=0.$$

The derivative of the integrand is

$$\begin{array}{c}{\displaystyle \frac{d}{d\theta }}\sum {\left(\theta \dot{v}\left(t\right)-{\displaystyle \frac{d}{dt}}{K}_{q\left(t\right)}\left(\theta v\left(t\right)\right)+\stackrel{★}{q}\left(t\right)\right)}^2{q}_{\theta }\left(t\right)\hfill \\ =\sum 2\left(\theta \dot{v}\left(t\right)-{\displaystyle \frac{d}{dt}}{K}_{q\left(t\right)}\left(\theta v\left(t\right)\right)+\stackrel{★}{q}\left(t\right)\right)\left(\dot{v}\left(t\right)-{\displaystyle \frac{d}{dt}}{\displaystyle \frac{d}{d\theta }}{K}_{q\left(t\right)}\left(\theta v\left(t\right)\right)\right)q_{\theta }\left(t\right)\hfill \\ +\sum {\left(\theta \dot{v}\left(t\right)-{\displaystyle \frac{d}{dt}}{K}_{q\left(t\right)}\left(\theta v\left(t\right)\right)+\stackrel{★}{q}\left(t\right)\right)}^2\left(v\left(t\right)-{\displaystyle \frac{d}{d\theta }}{K}_{q\left(t\right)}\left(\theta v\left(t\right)\right)\right)q_{\theta }\left(t\right).\hfill \end{array}$$

The derivatives ${\displaystyle \frac{d}{dt}}$, ${\displaystyle \frac{d}{d\theta }}$, ${\displaystyle \frac{d^2}{dtd\theta }}$ of the cumulant function are, respectively,

$$\begin{array}{c}{\displaystyle \frac{d}{dt}}{K}_{q\left(t\right)}\left(\theta v\left(t\right)\right)={\displaystyle \frac{d}{dt}}log\sum {\mathrm{e}}^{\theta v\left(t\right)}q\left(t\right)=\hfill \\ {\left(\sum {\mathrm{e}}^{\theta v\left(t\right)}q\left(t\right)\right)}^{-1}\left(\sum \theta \dot{v}\left(t\right){\mathrm{e}}^{\theta v\left(t\right)}q\left(t\right)+\sum {\mathrm{e}}^{\theta v\left(t\right)}\stackrel{★}{q}\left(t\right)q\left(t\right)\right)=\hfill \\ {\mathbb{E}}_{q_{\theta }\left(t\right)}\left[\theta \dot{v}\left(t\right)+\stackrel{★}{q}\left(t\right)\right],\hfill \end{array}$$

$${\displaystyle \frac{d}{d\theta }}{K}_{q\left(t\right)}\left(\theta v\left(t\right)\right)=log\sum {\mathrm{e}}^{\theta v\left(t\right)}q\left(t\right)={\left(\sum {\mathrm{e}}^{\theta v\left(t\right)}q\left(t\right)\right)}^{-1}\sum v\left(t\right){\mathrm{e}}^{\theta v\left(t\right)}q\left(t\right)={\mathbb{E}}_{q_{\theta }\left(t\right)}\left[v\left(t\right)\right],$$

$$\begin{array}{c}{\displaystyle \frac{d^2}{dtd\theta }}{K}_{q\left(t\right)}\left(\theta v\left(t\right)\right)={\displaystyle \frac{d}{dt}}{\mathbb{E}}_{q_{\theta }\left(t\right)}\left[v\left(t\right)\right]={\displaystyle \frac{d}{dt}}\sum v\left(t\right)q_{\theta }\left(t\right)=\hfill \\ \sum \dot{v}\left(t\right)q_{\theta }\left(t\right)+\sum v\left(t\right){\dot{q}}_{\theta }\left(t\right)={\mathbb{E}}_{q_{\theta }\left(t\right)}\left[\dot{v}\left(t\right)+v\left(t\right){\stackrel{★}{q}}_{\theta }\left(t\right)\right].\hfill \end{array}$$

The values at $\theta =0$ are, respectively, 0, 0, and ${\mathbb{E}}_{q\left(t\right)}\left[\dot{v}\left(t\right)+v\left(t\right)\stackrel{★}{q}\left(t\right)\right]$.

The $\theta$-derivative of the energy at $\theta =0$ is

$$\begin{array}{c}{\left.{\displaystyle \frac{d}{d\theta }}\sum {\left(\theta \dot{v}\left(t\right)-{\displaystyle \frac{d}{dt}}{K}_{q\left(t\right)}\left(\theta v\left(t\right)\right)+\stackrel{★}{q}\left(t\right)\right)}^2{q}_{\theta }\left(t\right)\right|}_{\theta =0}\hfill \\ 2\sum \stackrel{★}{q}\left(t\right)\left(\dot{v}\left(t\right)-{\mathbb{E}}_{q\left(t\right)}\left[\dot{v}\left(t\right)+v\left(t\right)\stackrel{★}{q}\left(t\right)\right]\right)+\sum {\left(\stackrel{★}{q}\left(t\right)\right)}^{2}v\left(t\right)q\left(t\right)=\hfill \\ 2{〈\stackrel{★}{q}\left(t\right),\dot{v}\left(t\right)-{\mathbb{E}}_{q\left(t\right)}\left[\dot{v}\left(t\right)\right]〉}_{q\left(t\right)}-2{〈{\stackrel{★}{q}\left(t\right)}^2-{\mathbb{E}}_{q\left(t\right)}\left[{\stackrel{★}{q}\left(t\right)}^2\right],〉}_{q\left(t\right)}.\hfill \end{array}$$

The variation in the energy in the direction $v(·)$ is

$${\int }_0^{1}2{〈\stackrel{★}{q}\left(t\right),\dot{v}\left(t\right)-{\mathbb{E}}_{q\left(t\right)}\left[\dot{v}\left(t\right)\right]〉}_{q\left(t\right)}dt+{\int }_0^1{〈\stackrel{★}{q}{\left(t\right)}^2-{\mathbb{E}}_{q\left(t\right)}\left[\stackrel{★}{q}{\left(t\right)}^2\right],v\left(t\right)〉}_{q\left(t\right)}dt,$$

where the first integral vanishes,

$$\sum _{x\in \Omega }{\int }_0^{1}2\dot{q}(x;t)\dot{v}(x;t)dt=\sum _{x\in \Omega }-{\int }_0^{1}2\ddot{q}(x;t)v(x;t)dt={\int }_0^1{〈-2{\displaystyle \frac{\ddot{q}\left(t\right)}{q\left(t\right)}},v\left(t\right)〉}_{q\left(t\right)}=0.$$

In conclusion,

$${\int }_0^1{〈{\displaystyle \frac{\ddot{q}\left(t\right)}{q\left(t\right)}}-{\displaystyle \frac{1}{2}}\left(\stackrel{★}{q}{\left(t\right)}^2-{\mathbb{E}}_{q\left(t\right)}\left[\stackrel{★}{q}{\left(t\right)}^2\right]\right),v\left(t\right)〉}_{q\left(t\right)}dt=0$$

for all $v(·)$. One can easily check that the Euler–Lagrange equation holds,

$${\displaystyle \frac{\ddot{q}\left(t\right)}{q\left(t\right)}}-{\displaystyle \frac{1}{2}}\left(\stackrel{★}{q}{\left(t\right)}^2-{\mathbb{E}}_{q\left(t\right)}\left[\stackrel{★}{q}{\left(t\right)}^2\right]\right)=0.$$

Remark 3.

The LHS of the equation above is the semi-sum of the two accelerations of Equations (39) and (40). As S.-I Amari first discussed, this particular acceleration is the Levi–Civita acceleration of the open probability simplex. See the monograph [2]. The present paper does not discuss geometries other than the affine ones.

5.2. Lagrangian and Hamiltonian Formalism on the Probability Simplex

Consider a smooth curve, $t\mapsto \left(q\right(t),\eta (t),w(t\left)\right)$, in the full statistical bundle ${{}^{1}S}^1{\mathcal{P}}_{>}\left(\Omega \right)$. Here, $q\left(t\right)$ represents the probability distribution, $w\left(t\right)$ is an element of the fiber at $q\left(t\right)$, interpreted as a velocity, and $\eta \left(t\right)$ is an element of the fiber at $q\left(t\right)$, interpreted as an acceleration. This second instance of the fiber is interpreted as a cotangent space in duality with the tangent space via the covariance inner product.

The derivative of the inner product ${\langle \eta \left(t\right),w\left(t\right)\rangle }_{q\left(t\right)}$ is given by the sum of the mixture covariant derivative of $\eta \left(t\right)$ and the exponential covariant derivative of $w\left(t\right)$,

$$\begin{array}{c}{\displaystyle \frac{d}{dt}}{\langle \eta \left(t\right),w\left(t\right)\rangle }_{q\left(t\right)}={\displaystyle \frac{d}{dt}}{〈{{}^{\mathrm{m}}\mathbb{U}}_{q\left(t\right)}^p\eta \left(t\right),{{}^{\mathrm{e}}\mathbb{U}}_{q\left(t\right)}^{p}w\left(t\right)〉}_p=\hfill \\ {〈{\displaystyle \frac{d}{dt}}{{}^{\mathrm{m}}\mathbb{U}}_{q\left(t\right)}^p\eta \left(t\right),{{}^{\mathrm{e}}\mathbb{U}}_{q\left(t\right)}^{p}w\left(t\right)〉}_p+{〈{{}^{\mathrm{m}}\mathbb{U}}_{q\left(t\right)}^p\eta \left(t\right),{\displaystyle \frac{d}{dt}}{{}^{\mathrm{e}}\mathbb{U}}_{q\left(t\right)}^{p}w\left(t\right)〉}_p=\hfill \\ {〈{\displaystyle \frac{{}^{m}D}{dt}}\eta \left(t\right),w\left(t\right)〉}_{q\left(t\right)}+{〈\eta \left(t\right),{\displaystyle \frac{{}^{m}D}{dt}}w\left(t\right)〉}_{q\left(t\right)}.\hfill \end{array}$$

(42)

Recall that, for a given scalar field, $F:{\mathcal{P}}_{>}\left(\Omega \right)\to \mathbb{R}$, the natural gradient of F is a section of the statistical bundle, such that, for all smooth curves, $t\mapsto q\left(t\right)$,

$${\displaystyle \frac{d}{dt}}F\left(q\left(t\right)\right)={\langle gradF\left(q\left(t\right)\right),\stackrel{★}{q}\left(t\right)\rangle }_{q\left(t\right)}.$$

This formulation ensures that the gradient $gradF$ captures the rate of change of the scalar field F along the curve $q\left(t\right)$. We will need a more general gradient form that includes both the natural gradient and Equation (42).

Let there be given a real function, $F:{{}^{1}S}^1{\mathcal{P}}_{>}\left(\Omega \right)\times D\to \mathbb{R}$, where D is a domain of ${\mathbb{R}}^k$. For a generic smooth curve,

$$t\mapsto (q\left(t\right),\eta \left(t\right),w\left(t\right),c\left(t\right))\in {{}^{1}S}^1{\mathcal{P}}_{>}\left(\times \right)D,$$

we want to write

$$\begin{array}{c}{\displaystyle \frac{d}{dt}}F(q\left(t\right),\eta \left(t\right),w\left(t\right),c\left(t\right))\hfill \\ ={〈gradF(q\left(t\right),\eta \left(t\right),w\left(t\right),c\left(t\right)),\stackrel{★}{q}\left(t\right)〉}_{q\left(t\right)}+{〈{\displaystyle \frac{{}^{m}D}{d\eta }}\left(t\right),{\nabla }_{m}F(q\left(t\right),\eta \left(t\right),w\left(t\right),c\left(t\right))〉}_{q\left(t\right)}\hfill \\ \hfill +{〈{\nabla }_{e}F(q\left(t\right),\eta \left(t\right),w\left(t\right),c\left(t\right)),{\displaystyle \frac{{}^{m}D}{dt}}w\left(t\right)〉}_{q\left(t\right)}+\nabla F(q\left(t\right),\eta \left(t\right),w\left(t\right),c\left(t\right))·\dot{c}\left(t\right)\end{array}$$

These terms collectively describe the total derivative: how the function F changes along the curve with values in the full bundle and a parametric set, D. The four gradient components can be computed in the proper chart by varying only one component at a time.

The dually affine geometry of the statistical bundle is naturally well suited to describing the dynamics of probability densities in Lagrangian and Hamiltonian formalism, as presented, for example, in the textbook [33]. The Lagrangian formulation of mechanics derives the fundamental laws of force equilibrium from variational principles. In this analogy, the open probability simplex ${\mathcal{P}}_{>}\left(\Omega \right)$ corresponds to the configuration space, while the statistical bundle $S{\mathcal{P}}_{>}\left(\Omega \right)$, which includes velocities, is associated with the mechanical phase space. In this section, we describe the general methodology in an analogy with the standard treatment in mechanics, and we apply it to a specific but quite general example. For the full proof, see [12].

Given a Lagrangian function,

$$L:S{\mathcal{P}}_{>}\left(\Omega \right)\times [0,1]\to \mathbb{R},$$

for each smooth curve $q:[0,1]\ni t\mapsto q\left(t\right)\in {\mathcal{P}}_{>}\left(\Omega \right)$, we consider its lift $t\mapsto (q\left(t\right),\stackrel{★}{q}\left(t\right))\in S{\mathcal{P}}_{>}\left(\Omega \right)$. The action is the integral of the Lagrangian along the lifted curve over the fixed time interval $[0,1]$,

$$q(·)\mapsto A\left(q\right)={\int }_1^{0}L(q\left(t\right),\stackrel{★}{q}\left(t\right),t)dt.$$

In mechanical terms, the action encapsulates the system’s dynamics in the Lagrangian function and the path taken by the curve in the configuration space. The discussion of Section 5.1 is an instance of this approach applied to statistics. Two further examples follow.

Example 19

(Mixture curve). The mixture curve $q\left(t\right)=q_0+t(q_1-q_0)$ has the velocity

$$\stackrel{★}{q}\left(t\right)={\displaystyle \frac{d}{dt}}log(q_0+t(q_1-q_0))={\displaystyle \frac{q_1-q_0}{q_0+t(q_1-q_0)}}.$$

The action of this curve for the Lagrangian $L(q,\stackrel{★}{q})={〈\stackrel{★}{q},\stackrel{★}{q}〉}_q$ is

$$\begin{array}{c}L\left(q(·)\right)={\int }_0^1\sum {\displaystyle \frac{{(q_1-q_0)}^2}{q_0+t(q_1-q_0)}}dt.=\hfill \\ {\left.\sum (q_1-q_0)log(q_0+t(q_1-q_0))\right|}_{t=0}^{t=1}=\sum (q_1-q_0)log{\displaystyle \frac{q_1}{q_0}}=\hfill \\ {〈{\eta }_{q_0}\left(q_1\right),s_{q_0}\left(q_1\right)〉}_{q_0}.\hfill \end{array}$$

Example 20

(Exponential curve). The exponential curve $q\left(t\right)=e^{tu-K_{q_0}\left(tu\right)}·q_0$ connects $q_0$ to $q_1=q\left(1\right)$. The velocity is

$$u-d{K}_{q_0}\left(tu\right)\left[u\right]=u-{\mathbb{E}}_{q\left(t\right)}\left[u\right].$$

The action of this curve for the Lagrangian $L(q,\stackrel{★}{q})={〈\stackrel{★}{q},\stackrel{★}{q}〉}_q$ is

$$L\left(q(·)\right)={\int }_0^1{\mathbb{E}}_{q\left(t\right)}\left[{(u-{\mathbb{E}}_{q\left(t\right)}\left[u\right])}^2\right]dt.$$

We use the equality

$${\mathbb{E}}_{q\left(t\right)}\left[{(u-{\mathbb{E}}_{q\left(t\right)}\left[u\right])}^2\right]={\displaystyle \frac{d}{dt}}{\mathbb{E}}_{q\left(t\right)}\left[u\right]$$

and ${\mathbb{E}}_{q_0}\left[u\right]=0$ to get

$$\begin{array}{c}L\left(q(·)\right)={\int }_0^1{\displaystyle \frac{d}{dt}}{\mathbb{E}}_{q\left(t\right)}\left[u\right]dt={\mathbb{E}}_{q_1}\left[u\right]={\mathbb{E}}_{q_1}\left[log{\displaystyle \frac{q_1}{q_0}}-{\mathbb{E}}_{q_0}\left[log{\displaystyle \frac{q_1}{q_0}}\right]\right]=\hfill \\ {\mathbb{E}}_{q_1}\left[log{\displaystyle \frac{q_1}{q_0}}\right]-{\mathbb{E}}_{q_0}\left[log{\displaystyle \frac{q_1}{q_0}}\right]=D(q_1\Vert q_0)+D(q_0\Vert q_1).\hfill \end{array}$$

Hamilton’s principle states that the action $A\left(q\right)$ has a critical point at the solution q in the space of curves on ${\mathcal{P}}_{>}\left(\Omega \right)$ if the Euler–Lagrange equation holds,

$${\displaystyle \frac{{}^{m}D}{dt}}{\nabla }_{e}L(q\left(t\right),\stackrel{★}{q}\left(t\right),t)=gradL(q\left(t\right),\stackrel{★}{q}\left(t\right),t).$$

This follows from a general principle in mechanics. See [36] for a detailed proof in the statistical case.

Let us define the Hamiltonian corresponding to a Lagrangian with the usual duality argument. The gradient mapping and Legendre transform are applied at each fixed density $q\in {\mathcal{P}}_{>}\left(\Omega \right)$ and each time t. The partial mapping $S_q{\mathcal{P}}_{>}\left(\Omega \right)\ni w\mapsto L_{q,t}\left(w\right)=L(q,w,t)$ has a gradient mapping in the covariance duality of ${{}^{*}S}_q{\mathcal{P}}_{>}\left(\Omega \right)\times S_q{\mathcal{P}}_{>}\left(\Omega \right)$, namely $w\mapsto {\nabla }_{e}L(q,w,t)$.

The Hamiltonian $H(q,\eta ,t)$ is the Legendre transform of the Lagrangian,

$$H(q,\eta ,t)={〈\eta ,{\left({\nabla }_e{L}_{q,t}\right)}^{-1}\left(\eta \right)〉}_q-L(q,{\left({\nabla }_e{L}_{q,t}\right)}^{-1}\left(\eta \right)).$$

Here, ${\left({\nabla }_e{L}_{q,t}\right)}^{-1}\left(\eta \right)$ denotes the inverse of the exponential gradient mapping applied to $\eta$.

These components collectively describe the system’s dynamics in terms of Hamilton’s principle and Hamiltonian formalism.

If the curve $t\mapsto q\left(t\right)$ is a solution of the Euler–Lagrange equation, then the curve $t\mapsto \zeta \left(t\right)=\left(q\right(t),\eta (t\left)\right)$ lies in the cotangent bundle ${}^{*}SE\left(\mu \right)$. The momentum $\eta \left(t\right)$ is defined as:

$$\eta \left(t\right)={\nabla }_{e}L(q\left(t\right),\stackrel{★}{q}\left(t\right),t).$$

In this context, the mixture bundle ${}^{*}SE\left(\mu \right)$ acts as the cotangent bundle in classical mechanics.

The Hamilton equations describe the evolution of the system:

$$\left\{\begin{array}{c}{\displaystyle \frac{{}^{m}D}{dt}}\eta \left(t\right)=-\nabla H(q\left(t\right),\eta \left(t\right),t)\hfill \\ \stackrel{★}{q}\left(t\right)={\nabla }_{m}H(q\left(t\right),\eta \left(t\right),t).\hfill \end{array}\right.$$

These equations describe the system’s evolution in terms of the Hamiltonian function $H(q,\eta ,t)$. The time derivative of the Hamiltonian H along the trajectory is given by the following:

$${\displaystyle \frac{d}{dt}}H(q\left(t\right),\eta \left(t\right),t)={\displaystyle \frac{\partial }{\partial t}}H(q\left(t\right),\eta \left(t\right),t).$$

This equation indicates that the system’s total energy changes according to the Hamiltonian’s explicit time dependence.

Example 21

(Compare with Section 5.1). Given the Lagrangian $L(q,w)={\displaystyle \frac{1}{2}}{\langle w,w\rangle }_q$, the Hamiltonian obtained via the Legendre transform is $H(q,\eta )={\displaystyle \frac{1}{2}}{\langle \eta ,\eta \rangle }_q$. The Lagrangian represents the kinetic energy, while the Hamiltonian represents the system’s total energy.

The gradients of the Hamiltonian and Lagrangian are computed as follows:

$$\begin{array}{c}\nabla H(q,\eta )=-{\displaystyle \frac{1}{2}}({\eta }^2-E_q\left[{\eta }^2\right]),\\ {\nabla }_{m}H(q,\eta )=\eta ,\\ \nabla L(q,w)={\displaystyle \frac{1}{2}}(w^2-E_q\left[w^2\right]),\\ {\nabla }_{e}L(q,w)=w..\end{array}$$

The Euler–Lagrange equation for $\stackrel{★}{q}=w\in {}^{*}SE\left(\mu \right)$ is given by the following:

$${\displaystyle \frac{{}^{m}D}{dt}}\stackrel{★}{q}\left(t\right)={\displaystyle \frac{1}{2}}(\stackrel{★}{q}{\left(t\right)}^2-E_{q\left(t\right)}\left[\stackrel{★}{q}{\left(t\right)}^2\right]),$$

where

$${\displaystyle \frac{{}^{m}D}{dt}}\stackrel{★}{q}\left(t\right)={\displaystyle \frac{\ddot{q}\left(t\right)}{q\left(t\right)}}.$$

The Hamilton equations are

$$\left\{\begin{array}{c}{\displaystyle \frac{{}^{m}D}{dt}}\eta \left(t\right)={\displaystyle \frac{1}{2}}\left({\eta }^2-E_q\left[{\eta }^2\right]\right),\hfill \\ \stackrel{★}{q}\left(t\right)=\eta \left(t\right),\hfill \end{array}\right.$$

The conserved energy is $H(q\left(t\right),\eta \left(t\right))={\displaystyle \frac{1}{2}}{E}_q\left[{\displaystyle \frac{\dot{q}{\left(t\right)}^2}{q\left(t\right)}}\right]$; that is, the Fisher’s information is constant in the solution.

6. Conclusions

This research has provided a general and cohesive geometric framework for studying several geometries of the probability simplex, including Aitchison’s and Amari’s dual affine and transport geometries. By providing a cogent presentation of these various geometric approaches, the framework has the potential to significantly improve the comprehension and manipulation of probability distributions in statistical modeling and machine learning.

This unified approach is based on an axiomatic approach called the affine geometry of the statistical bundle. This approach is beneficial when the accurate handling of compositional data is crucial, such as when developing computational tools, statistical model optimization, and gradient flow optimization are involved. In particular, our formalism provides a principled treatment of second-order computations, such as accelerations.

As a notable contribution, we have shown that Aitchison CLR provides the charts of an affine space isomorphic to the information geometry space defined by the exponential presentation of probability functions. Moreover, both approaches allow for a statistically natural calculus concerning gradients of scalar fields and gradient flows.

The framework can increase performance in activities ranging from data processing to implementing sophisticated machine learning algorithms by facilitating more accurate and efficient modeling, providing a consistent geometric perspective. Moreover, we have shown how other geometries of great current interest, those associated with optimal transport, fit into the scheme of gradient flows in the statistical bundle.

The paper’s final part contributed to discussing simple but relevant examples of statistical models controlled via second-order differential equations. This opens the discussion of dynamic statistical models with unusual time behavior, such as periodicity, without any reference to any dynamical system in the sample space.