A Partial Information Decomposition for Multivariate Gaussian Systems Based on Information Geometry

Abstract

There is much interest in the topic of partial information decomposition, both in developing new algorithms and in developing applications. An algorithm, based on standard results from information geometry, was recently proposed by Niu and Quinn (2019). They considered the case of three scalar random variables from an exponential family, including both discrete distributions and a trivariate Gaussian distribution. The purpose of this article is to extend their work to the general case of multivariate Gaussian systems having vector inputs and a vector output. By making use of standard results from information geometry, explicit expressions are derived for the components of the partial information decomposition for this system. These expressions depend on a real-valued parameter which is determined by performing a simple constrained convex optimisation. Furthermore, it is proved that the theoretical properties of non-negativity, self-redundancy, symmetry and monotonicity, which were proposed by Williams and Beer (2010), are valid for the decomposition $I_{\mathrm{ig}}$ derived herein. Application of these results to real and simulated data show that the $I_{\mathrm{ig}}$ algorithm does produce the results expected when clear expectations are available, although in some scenarios, it can overestimate the level of the synergy and shared information components of the decomposition, and correspondingly underestimate the levels of unique information. Comparisons of the $I_{\mathrm{ig}}$ and $I_{\mathrm{dep}}$ (Kay and Ince, 2018) methods show that they can both produce very similar results, but interesting differences are provided. The same may be said about comparisons between the $I_{\mathrm{ig}}$ and $I_{\mathrm{mmi}}$ (Barrett, 2015) methods.

1. Introduction

Williams and Beer [1] introduced a new method for the decomposition of information in a probabilistic system termed partial information decomposition (PID). This allows the joint mutual information between a number of input sources and a target (output) to be decomposed into components which quantify different aspects of the transmitted information in the system. These are the unique information that each source conveys about the target; the shared information that all sources possess about the target; the synergistic information that the sources in combination possess regarding the target. An additional achievement was to prove that the interaction information [2] is actually the difference between the synergy and redundancy in a system. Thus, a positive value for interaction information signifies that there is more synergy than redundancy in the system, while a negative value indicates the opposite. The work by Willliams and Beer has led to many new methods for defining a PID, mainly for discrete probabilistic systems [3,4,5,6,7,8,9,10,11,12,13] spawning a variety of applications [14,15,16,17,18,19].

There has been considerable interest in PID methods for Gaussian systems. The case of static and dynamic Gaussian systems with two scalar sources and a scalar target was considered in [20], which applied the minimum mutual information PID, $I_{\mathrm{mmi}}$. Further insights were developed in [21] regarding synergy. A PID for Gaussian systems based on common surprisal was published in [7]. Barrett’s work [20] was extended to multivariate Gaussian systems with two vector sources and a vector target in [22] using the $I_{\mathrm{dep}}$ method which was introduced for discrete systems in [8]. Further work based on the concept of statistical deficiency is reported in [23]. Application of PID for Gaussian systems has been used in a range of applications [18,24,25,26,27,28,29,30]

We focus in particular here on the method proposed by Niu and Quinn [3]. They applied standard results from information geometry [31,32,33] in order to define a PID for three scalar random variables which follow an exponential family distribution, including a trivariate Gaussian distribution.

Here, we extend this work in two ways: (a) we provide general formulae for a PID involving multivariate Gaussian systems which have two vector sources and a vector target by making use of the same standard methods from information geometry as in [3] and (b) we prove that the Williams–Beer properties of non-negativity, self-redundancy, symmetry and monotonicity are valid for this PID. We also provide some illustrations of the resulting algorithm using real and simulated data. The PID developed herein is based on some of the probability models in the same partially ordered lattice on which the $I_{\mathrm{dep}}$ algorithm is based. Therefore, we also compare the results obtained with those obtained by using the $I_{\mathrm{dep}}$ method. The $I_{\mathrm{ig}}$ results are also compared with those obtained using the $I_{\mathrm{mmi}}$ algorithm.

2. Methods

2.1. Notation

A generic ‘p’ will be used to denote an absolutely continuous probability density function (pdf), with the arguments of the function signifying which distribution is intended. Bold capital letters are used to denote random vectors, with their realised values appearing in bold lowercase—so that $p({\mathbf{x}}_1,{\mathbf{x}}_2,{\mathbf{x}}_3)$ denotes the joint pdf of the random vectors, ${\mathbf{X}}_1,{\mathbf{X}}_2,{\mathbf{X}}_3$, while $p({\mathbf{x}}_1,{\mathbf{x}}_3|{\mathbf{x}}_2)$ is the conditional pdf of $({\mathbf{X}}_1,{\mathbf{X}}_3)$ given a value for ${\mathbf{X}}_2$.

We consider the case where random vectors ${\mathbf{X}}_1,{\mathbf{X}}_2,{\mathbf{X}}_3$, of dimensions $n_1,n_2,n_3$, respectively, have partitioned mean vectors equal to zero vectors of lengths $n_1,n_2,n_3$, respectively, and a conformably partitioned covariance matrix. We stack these random vectors into the random vector $\mathbf{Z}$, so that $\mathbf{Z}$ has dimension $m=n_1+n_2+n_3,$ and assume that $\mathbf{Z}$ has a positive definite multivariate Gaussian distribution with pdf $p({\mathbf{x}}_1,{\mathbf{x}}_2,{\mathbf{x}}_3)$, mean vector $\mathbf{0}$ and covariance matrix given by

$$\Sigma =\left[\begin{array}{ccc}{\Sigma }_{11}& {\Sigma }_{12}& {\Sigma }_{13}\\ {\Sigma }_{12}^T& {\Sigma }_{22}& {\Sigma }_{23}\\ {\Sigma }_{13}^T& {\Sigma }_{23}^T& {\Sigma }_{33}\end{array}\right],$$

(1)

where the covariance matrices ${\Sigma }_{11},{\Sigma }_{22},{\Sigma }_{33}$ of ${\mathbf{X}}_1,{\mathbf{X}}_2,{\mathbf{X}}_3$, respectively, are of sizes $n_1\times n_1,n_2\times n_2,n_3\times n_3$, and ${\Sigma }_{12},{\Sigma }_{13},{\Sigma }_{33}$ are the pairwise cross-covariance matrices between the three vectors ${\mathbf{X}}_1,{\mathbf{X}}_2,{\mathbf{X}}_3$. We also denote the conformably partitioned precision (or concentration) matrix K by

$$K=\left[\begin{array}{ccc}{K}_{11}& K_{12}& K_{13}\\ K_{12}^T& K_{22}& K_{23}\\ K_{13}^T& K_{23}^T& K_{33}\end{array}\right],$$

where $K={\Sigma }^{-1}$. The pdf of $\mathbf{Z}$ is

$$f\left(\mathbf{z}\right|K)=\frac{\sqrt{\left|K\right|}}{{\left(2\pi \right)}^{\frac{1}{2}m}}\exp (-\frac{1}{2}{\mathbf{z}}^{T}K\mathbf{z})\equiv \frac{\sqrt{\left|K\right|}}{{\left(2\pi \right)}^{\frac{1}{2}m}}\exp \left(-\frac{1}{2}\underset{i\le j}{\sum _{i=1}^3\sum _{j=1}^3}{\mathbf{x}}_i^T{K}_{ij}{\mathbf{x}}_j\right),\mathbf{z}\in {\mathbb{R}}^m.$$

(2)

2.2. Some Information Geometry

We now describe some standard results from information geometry [32,33] as applied to zero-mean, partitioned multivariate Gaussian probability distributions. The fact that there is no loss of generality in making this zero-mean assumption will be justified by Lemma 1 in Section 3. The multivariate Gaussian pdf defined in (2) may be written in the form

$$\begin{array}{cc}\hfill & \frac{\sqrt{\left|K\right|}}{{\left(2\pi \right)}^{\frac{1}{2}m}}\exp (-\frac{1}{2}({\mathbf{x}}_1^T{K}_{11}{\mathbf{x}}_1+{\mathbf{x}}_2^T{K}_{22}{\mathbf{x}}_2+{\mathbf{x}}_3^T{K}_{33}{\mathbf{x}}_3)\hfill \\ \hfill & -({\mathbf{x}}_2^T{K}_{12}^T{\mathbf{x}}_1+{\mathbf{x}}_3^T{K}_{13}^T{\mathbf{x}}_1+{\mathbf{x}}_3^T{K}_{23}^T{\mathbf{x}}_2)),\hfill \end{array}$$

which may be written in terms of the Frobenius inner product as

$$\begin{array}{cc}\hfill & \exp (〈-\frac{1}{2}{K}_{11},{\mathbf{x}}_1{\mathbf{x}}_1^T〉+〈-\frac{1}{2}{K}_{22},{\mathbf{x}}_2{\mathbf{x}}_2^T〉+〈-\frac{1}{2}{K}_{33},{\mathbf{x}}_3{\mathbf{x}}_3^T〉\hfill \\ \hfill & +〈-K_{12},{\mathbf{x}}_1{\mathbf{x}}_2^T〉+〈-K_{13},{\mathbf{x}}_1{\mathbf{x}}_3^T〉+〈-K_{23},{\mathbf{x}}_2{\mathbf{x}}_3^T〉-\psi \left(\mathit{\theta }\right)),\hfill \end{array}$$

where

$$\psi \left(\mathit{\theta }\right)=-\frac{1}{2}\log \frac{\left|K\right|}{{\left(2\pi \right)}^m}.$$

This is of exponential family form [33] (p. 34) and [34] with natural parameter

$$\mathit{\theta }=\{-\frac{1}{2}{K}_{11},-\frac{1}{2}{K}_{22},-\frac{1}{2}{K}_{33},-K_{12},-K_{13},-K_{23}\},$$

and expectation parameter $\mathit{\eta }=\{{\eta }_{ij},i=1,2,3;j=1,2,3;i\le j\}$, where ${\eta }_{ij}=\mathbb{E}\left({\mathbf{x}}_i{\mathbf{x}}_j^T\right)={\Sigma }_{ij}$. We note that there is something of a terminal ambiguity here, since a ‘parameter’ is usually a real number. It is convenient to use the more compact notation provided by matrices since this enables all of the elements of a matrix natural parameter to be set to zero simultaneously.

The exponential family distribution in (2) is a dually flat manifold [31], which we denote by M.

We define the following e-flat submanifolds of M:

$$\begin{array}{cc}\hfill S_7& =\{p\in M:K_{12}=0\}\hfill \\ \hfill S_6& =\{p\in M:K_{13}=0\}\hfill \\ \hfill S_5& =\{p\in M:K_{23}=0\}\hfill \\ \hfill S_4& =\{p\in M:K_{12}=0\mathrm{and}{K}_{13}=0\}\hfill \\ \hfill S_3& =\{p\in M:K_{12}=0\mathrm{and}{K}_{23}=0\}\hfill \\ \hfill S_2& =\{p\in M:K_{13}=0\mathrm{and}{K}_{23}=0\}\hfill \\ \hfill S_1& =\{p\in M:K_{12}=0,K_{13}=0\mathrm{and}{K}_{23}=0\}\hfill \end{array}$$

which may be conveniently pictured as the partially ordered lattice in Figure 1. The submanifolds $S_5$ and $S_6$ are necessary for the definition of the information-geometric PID [3] and the others will be considered in the sequel. Lattices similar to that in Figure 1 appear in [8,35,36] in relation to information decomposition, and in [37] who consider dually flat manifolds on posets. See also [38], and references therein, for the use of a variety of lattices of models in statistical work.

Hierarchical chains of submanifolds were considered in [31] but here the submanifolds are not all in a hierarchical chain due to the presence of two antichains: $\{S_2,S_3,S_4\}$ and $\{S_5,S_6,S_7\}.$ There are, however, several useful chains within the lattice. Of particular relevance here are the chains $\{S_2,S_5,M\}$, $\{S_2,S_6,M\}$ and $\{S_2,M\}$. Application of Amari’s mixed-cut coordinates [31] and calculation of divergences produces measures of mutual information that are of direct relevance in PID (as was noted by [3] for three scalar random variables) in that the equations

$$\begin{array}{cc}\hfill I[{\mathbf{X}}_1;{\mathbf{X}}_2;{\mathbf{X}}_3]& =I[{\mathbf{X}}_1;{\mathbf{X}}_3]+I[{\mathbf{X}}_2;{\mathbf{X}}_3|{\mathbf{X}}_1]\hfill \\ \hfill & =I[{\mathbf{X}}_2;{\mathbf{X}}_3]+I[{\mathbf{X}}_1;{\mathbf{X}}_3|{\mathbf{X}}_2]\hfill \end{array}$$

are obtained—and they are standard results in information theory based on the chain rule for mutual information [39]. These are nice illustrations of Amari’s method.

We now consider m-projections from the pdf $p\in M$ to each of the submanifolds, $S_1-S_7$ [31]. It is easy to find the pdf in each submanifold that is closest to the given pdf in M, p, in terms of Kullback–Leibler (KL) divergence [40], ([Ch. 4]). They are given in Figure 1. We know [34,40] that setting a block of the inverse covariance for a multivariate Gaussian distribution to zero expresses a conditional dependence between the variables involved. For example, consider $S_5$. On this submanifold $K_{23}=0$ and so ${\mathbf{X}}_2$ and ${\mathbf{X}}_3$ are conditionally independent given a value for ${\mathbf{X}}_1$. Therefore, this pdf, which we denote by $p_5$, has the form $p_5({\mathbf{x}}_1,{\mathbf{x}}_2,{\mathbf{x}}_3)=p\left({\mathbf{x}}_2\right|{\mathbf{x}}_1)p\left({\mathbf{x}}_3\right|{\mathbf{x}}_1)p\left({\mathbf{x}}_1\right).$ On submanifold $S_2$, there are two conditional independences and so ${\mathbf{X}}_3$ and the pair $({\mathbf{X}}_1,{\mathbf{X}}_2)$ are independent and the closest pdf in $S_2$ to the pdf p has the form $p_2({\mathbf{x}}_1,{\mathbf{x}}_2,{\mathbf{x}}_3)=p({\mathbf{x}}_1,{\mathbf{x}}_2)p\left({\mathbf{x}}_3\right).$

The probability distributions defined by these information projections could also have been obtained by the method of maximum entropy, subject to constraints on model interactions [31], and they were obtained in this manner in [22] by making use of Gaussian graphical models [34,40].

We now mention important results from information geometry which are crucial for defining a PID [3]. Consider the pdfs $p_5,p_6,p$ belonging to the submanifolds $S_5,S_6$, and to manifold M, and the e-geodesic passing through $p_5$ and $p_6$. Then, any pdf on this e-geodesic path is also a zero-mean multivariate Gaussian pdf [41], ([Ch. 1]). We denote such a pdf by $p_t$. It has covariance matrix ${\Sigma }_t$, defined by

$${\Sigma }_t^{-1}=(1-t){\Sigma }_5^{-1}+t{\Sigma }_6^{-1}$$

(3)

provided that ${\Sigma }_t^{-1}$ is positive definite. We consider also an m-geodesic from p to $p_t$. Then, by standard results [31,33], this m-geodesic meets the e-geodesic through $p_5$ and $p_6$ at a unique pdf $p_{t^{\ast }}$ such that generalized Pythagorean relationships hold in terms of the KL divergence:

$$\begin{array}{cc}\hfill D\left(p\right|\left|p_5\right)& =D\left(p\right||p_{t^{\ast }})+D(p_{t^{\ast }}\left|\right|p_5)\hfill \end{array}$$

(4)

$$\begin{array}{cc}\hfill D\left(p\right|\left|p_6\right)& =D\left(p\right||p_{t^{\ast }})+D(p_{t^{\ast }}\left|\right|p_6)\hfill \end{array}$$

(5)

The pdf $p\left(t^{\ast }\right)$ minimizes the KL divergence between the pdf p in M and the pdf $p_t$ which lies on the e-geodesic which passes through pdfs $p_5$ and $p_6$.

2.3. The Partial Information Decomposition

Williams and Beer [1] introduce a framework called the partial information decomposition (PID) which decomposes the joint mutual information between a target and a set of multiple predictor variables into a series of terms reflecting information which is shared, unique or synergistically available within and between subsets of predictors. The joint mutual information, conditional mutual information and bivariate mutual information are defined as follows.

$$\begin{array}{cc}\hfill I[{\mathbf{X}}_1,{\mathbf{X}}_2;{\mathbf{X}}_3]& =\int \int \int p({\mathbf{x}}_1,{\mathbf{x}}_2,{\mathbf{x}}_3)\log \frac{p({\mathbf{x}}_1,{\mathbf{x}}_2,{\mathbf{x}}_3)}{p({\mathbf{x}}_1,{\mathbf{x}}_2)p\left({\mathbf{x}}_3\right)}d{\mathbf{x}}_{1}d{\mathbf{x}}_{2}d{\mathbf{x}}_3,\hfill \\ \hfill I[{\mathbf{X}}_1;{\mathbf{X}}_3|{\mathbf{X}}_2]& =\int \int \int p({\mathbf{x}}_1,{\mathbf{x}}_2,{\mathbf{x}}_3)\log \frac{p({\mathbf{x}}_1,{\mathbf{x}}_2,{\mathbf{x}}_3)p\left({\mathbf{x}}_2\right)}{p({\mathbf{x}}_1,{\mathbf{x}}_2)p({\mathbf{x}}_2,{\mathbf{x}}_3)}d{\mathbf{x}}_{1}d{\mathbf{x}}_{2}d{\mathbf{x}}_3,\hfill \\ \hfill I[{\mathbf{X}}_1;{\mathbf{X}}_3]& =\int \int p({\mathbf{x}}_1,{\mathbf{x}}_3)\log \frac{p({\mathbf{x}}_1,{\mathbf{x}}_3)}{p\left({\mathbf{x}}_1\right)p\left({\mathbf{x}}_3\right)}d{\mathbf{x}}_{1}d{\mathbf{x}}_3,\hfill \end{array}$$

Here, we focus on the case of two vector sources, ${\mathbf{X}}_1,{\mathbf{X}}_2$, and a vector target ${\mathbf{X}}_3$. Adapting the notation of [42], we express the joint mutual information in four terms as follows:

$\mathrm{Unq}1\equiv I_{unq}[{\mathbf{X}}_1;{\mathbf{X}}_3|{\mathbf{X}}_2]$	denotes the unique information that ${\mathbf{X}}_1$ conveys about ${\mathbf{X}}_3$;
$\mathrm{Unq}2\equiv I_{unq}[{\mathbf{X}}_2;{\mathbf{X}}_3|{\mathbf{X}}_1]$	is the unique information that ${\mathbf{X}}_2$ conveys about ${\mathbf{X}}_3$;
$\mathrm{Shd}\equiv I_{shd}[{\mathbf{X}}_1,{\mathbf{X}}_2;{\mathbf{X}}_3]$	gives the common (or redundant or shared) information that both ${\mathbf{X}}_1$ and ${\mathbf{X}}_2$ have about ${\mathbf{X}}_3$;
$\mathrm{Syn}\equiv I_{syn}[{\mathbf{X}}_1,{\mathbf{X}}_2;{\mathbf{X}}_3]$	is the synergy or information that the joint vector $({\mathbf{X}}_1,{\mathbf{X}}_2)$ has about ${\mathbf{X}}_3$ that cannot be obtained by observing ${\mathbf{X}}_1$ and ${\mathbf{X}}_2$ separately.

It is possible to make deductions about a PID by using the following four equations which give a link between the components of a PID and certain classical Shannon measures of mutual information. The following are from ([42] Equations (4) and (5)), with amended notation; see also [1].

$$\begin{array}{cc}\hfill I[{\mathbf{X}}_1;{\mathbf{X}}_3]& =\mathrm{Unq}1+\mathrm{Shd},\hfill \end{array}$$

(6)

$$\begin{array}{cc}\hfill I[{\mathbf{X}}_2;{\mathbf{X}}_3]& =\mathrm{Unq}2+\mathrm{Shd},\hfill \end{array}$$

(7)

$$\begin{array}{cc}\hfill I[{\mathbf{X}}_1;{\mathbf{X}}_3|{\mathbf{X}}_2]& =\mathrm{Unq}1+\mathrm{Syn},\hfill \end{array}$$

(8)

$$\begin{array}{cc}\hfill I[{\mathbf{X}}_2;{\mathbf{X}}_3|{\mathbf{X}}_1]& =\mathrm{Unq}2+\mathrm{Syn}.\hfill \end{array}$$

(9)

Also, the joint mutual information may be written as

$$I[{\mathbf{X}}_1,{\mathbf{X}}_2;{\mathbf{X}}_3]=\mathrm{Syn}+\mathrm{Unq}1+\mathrm{Unq}2+\mathrm{Shd}.$$

(10)

The Equations (6)–(9) are of rank 3 and so it is necessary to provide a value for any one of the components, and then the remaining terms can be easily calculated. The initial formulation of [1] was based on quantifying the shared information and deriving the other quantities, but others have focussed on quantifying unique information or synergy directly [4,5,8]. Also, the following form [16] of the interaction information [2] will be useful. It was shown [1] to be equal to the difference in Syn—Shd.

$$II[{\mathbf{X}}_1;{\mathbf{X}}_2;{\mathbf{X}}_3]=I[{\mathbf{X}}_1;{\mathbf{X}}_2|{\mathbf{X}}_3]-I[{\mathbf{X}}_1;{\mathbf{X}}_2].$$

(11)

3. Results

3.1. A PID for Gaussian Vector Sources and a Gaussian Vector Target

We now apply the results from the previous two sections in order to derive a partial information decomposition by making use of the method defined in [3]. The following lemma will confirm that without any loss of generality, we may assume, for all of the multivariate normal distributions considered herein, that the mean vector can be taken to be $\mathbf{0}$ and the covariance matrix of $\mathbf{Z}$, defined on ${\mathbb{R}}^m$, where $m=n_1+n_2+n_3$, can have the form

$$\Sigma =\left[\begin{array}{ccc}{I}_{n_1}& P& Q\\ P^T& I_{n_2}& R\\ Q^T& R^T& I_{n_3}\end{array}\right],$$

(12)

where the matrices $P,Q,R$ are of size $n_1\times n_2,n_1\times n_3,n_2\times n_3$, respectively, and are the cross-covariance (correlation) matrices between the three pairings of the three random vectors ${\mathbf{X}}_1,{\mathbf{X}}_2,{\mathbf{X}}_3,$ and so

$$\mathbb{E}\left({\mathbf{X}}_1{\mathbf{X}}_2^T\right)=P,\mathbb{E}\left({\mathbf{X}}_1{\mathbf{X}}_3^T\right)=Q,\mathbb{E}\left({\mathbf{X}}_2{\mathbf{X}}_3^T\right)=R.$$

The calculation of the partial information coefficients will involve the computation of KL divergences [43] between two multivariate Gaussian distributions associated with two submanifolds in the lattice, defined in Figure 1; see Lemma 1, with proof in Appendix C. These probability distributions will have two features in common: they each have the same partitioned mean vector and also the same variance–covariance matrices for the random vectors ${\mathbf{X}}_1$, ${\mathbf{X}}_2$ and ${\mathbf{X}}_3$, but different cross covariance matrices for each pair of the random vectors ${\mathbf{X}}_1$, ${\mathbf{X}}_2$ and ${\mathbf{X}}_3$.

Lemma 1.

Consider two multivariate Gaussian pdfs, $f_1$ and $f_2$, which have the same partitioned mean vector, $\mathit{\mu }={[{\mathit{\mu }}_1^T,{\mathit{\mu }}_2^T,{\mathit{\mu }}_3^T]}^T$, and conformably partitioned $m\times m$ covariance matrices

$$\Phi =\left[\begin{array}{ccc}{\Sigma }_{11}& {\Phi }_{12}& {\Phi }_{13}\\ {\Sigma }_{12}^T& {\Sigma }_{22}& {\Phi }_{23}\\ {\Phi }_{13}^T& {\Phi }_{23}^T& {\Sigma }_{33}\end{array}\right]\mathit{and}\Lambda =\left[\begin{array}{ccc}{\Sigma }_{11}& {\Lambda }_{12}& {\Lambda }_{13}\\ {\Lambda }_{12}^T& {\Sigma }_{22}& {\Lambda }_{23}\\ {\Lambda }_{13}^T& {\Lambda }_{23}^T& {\Sigma }_{33}\end{array}\right]$$

(13)

respectively, where the diagonal blocks ${\Sigma }_{11},{\Sigma }_{22},and{\Sigma }_{33}$ are square.

Then, the Kullback–Liebler divergence $D\left(f_1\right|\left|f_2\right)$ does not depend on the mean vector μ, nor does it depend directly on the variance–covariance matrices ${\Sigma }_{11},{\Sigma }_{22},{\Sigma }_{33}$. The divergence is equal to

$$D(f_1\left|\right|f_2)=\frac{1}{2}\left(\log \frac{|{\Lambda }_1|}{|{\Phi }_1|}-m+\mathrm{Tr}\left({\Phi }_1^{-1}{\Lambda }_1\right)\right),$$

(14)

where

$${\Phi }_1=\left[\begin{array}{ccc}{I}_{n_1}& P_{12}& P_{13}\\ P_{12}^T& I_{n_2}& P_{23}\\ P_{13}^T& P_{23}^T& I_{n_3}\end{array}\right]\mathit{and}{\Lambda }_1=\left[\begin{array}{ccc}{I}_{n_1}& L_{12}& L_{13}\\ L_{12}^T& I_{n_2}& L_{23}\\ L_{13}^T& L_{23}^T& I_{n_3}\end{array}\right]$$

with

$$P_{ij}={\Sigma }_{ii}^{-\frac{1}{2}}{\Phi }_{ij}{\Sigma }_{jj}^{-\frac{1}{2}},L_{ij}={\Sigma }_{ii}^{-\frac{1}{2}}{\Lambda }_{ij}{\Sigma }_{jj}^{-\frac{1}{2}},(i,j=1,2,3;i\le j),$$

which are the respective cross-correlation matrices among ${\mathbf{X}}_1,{\mathbf{X}}_2,{\mathbf{X}}_3$. The KL divergence depends only on these cross-correlation matrices.

3.2. Covariance Matrices

Table 1. Submanifold probability distributions with corresponding covariance matrices (modified from [22]).

Submanifold	${\mathbf{\Sigma }}_{\mathit{i}}$	Submanifold	${\mathbf{\Sigma }}_{\mathit{i}}$
$S_1:p\left({\mathbf{x}}_1\right)p\left({\mathbf{x}}_2\right)p\left({\mathbf{x}}_3\right)$	$\left[\begin{array}{ccc}{I}_{n_1}& 0& 0\\ 0& I_{n_2}& 0\\ 0& 0& I_{n_3}\end{array}\right]$	$S_5:p\left({\mathbf{x}}_2\right|{\mathbf{x}}_1)p\left({\mathbf{x}}_3\right|{\mathbf{x}}_1)p\left({\mathbf{x}}_1\right)$	$\left[\begin{array}{ccc}{I}_{n_1}& P& Q\\ P^T& I_{n_2}& P^{T}Q\\ Q^T& Q^{T}P& I_{n_3}\end{array}\right]$
$S_2:p({\mathbf{x}}_1,{\mathbf{x}}_2)p\left({\mathbf{x}}_3\right)$	$\left[\begin{array}{ccc}{I}_{n_1}& P& 0\\ P^T& I_{n_2}& 0\\ 0& 0& I_{n_3}\end{array}\right]$	$S_6:p\left({\mathbf{x}}_1\right|{\mathbf{x}}_2)p\left({\mathbf{x}}_3\right|{\mathbf{x}}_2)p\left({\mathbf{x}}_2\right)$	$\left[\begin{array}{ccc}{I}_{n_1}& P& PR\\ P^T& I_{n_2}& R\\ R^T{P}^T& R^T& I_{n_3}\end{array}\right]$
$S_3:p({\mathbf{x}}_1,{\mathbf{x}}_3)p\left({\mathbf{x}}_2\right)$	$\left[\begin{array}{ccc}{I}_{n_1}& 0& Q\\ 0& I_{n_2}& 0\\ Q^T& 0& I_{n_3}\end{array}\right]$	$S_7:p\left({\mathbf{x}}_1\right|{\mathbf{x}}_3)p\left({\mathbf{x}}_2\right|{\mathbf{x}}_3)p\left({\mathbf{x}}_3\right)$	$\left[\begin{array}{ccc}{I}_{n_1}& Q{R}^T& Q\\ R{Q}^T& I_{n_2}& R\\ Q^T& R^T& I_{n_3}\end{array}\right]$
$S_4:$$p({\mathbf{x}}_2,{\mathbf{x}}_3)p\left({\mathbf{x}}_1\right)$	$\left[\begin{array}{ccc}{I}_{n_1}& 0& 0\\ 0& I_{n_2}& R\\ 0& R^T& I_{n_3}\end{array}\right]$	$M:p({\mathbf{x}}_1,{\mathbf{x}}_2,{\mathbf{x}}_3)$	$\left[\begin{array}{ccc}{I}_{n_1}& P& Q\\ P^T& I_{n_2}& R\\ Q^T& R^T& I_{n_3}\end{array}\right]$

gives the covariance matrices corresponding to each of the projected distributions $p_1-p_7$ on the submanifolds. It is known from Gaussian graphical models [34,40] that the probability distributions associated with submanifolds $S_5$ and $S_6$ are defined by setting $K_{23}=0$ and $K_{13}=0$, respectively, in the precision matrix K. These conditions were shown in [22] to be equivalent to the equations $R=P^{T}Q$ and $Q=PR$, respectively. From Table 1, we see that the covariance matrices for pdfs $p_5$ and $p_6$ have the following form.

$${\Sigma }_5=\left[\begin{array}{ccc}{I}_{n_1}& P& Q\\ P^T& I_{n_2}& P^{T}Q\\ Q^T& Q^P& I_{n_3}\end{array}\right],\mathrm{and}{\Sigma }_6=\left[\begin{array}{ccc}{I}_{n_1}& P& PR\\ P^T& I_{n_2}& R\\ R^T{P}^T& R^T& I_{n_3}\end{array}\right].$$

(15)

The following lemma, which is proved in Appendix D, gives some useful results on determinants that will be used in the sequel.

Lemma 2.

The determinants of the matrices ${\Sigma }_5,{\Sigma }_6$ are given by

$$|{\Sigma }_5|=|I_{n_2}-P^{T}P\left|\right|I_{n_3}-Q^{T}Q\left|\mathit{and}\right|{\Sigma }_6|=|I_{n_2}-P^{T}P\left|\right|I_{n_3}-R^{T}R|.$$

Also,

$${\Sigma }_5={\Sigma }_6\iff Q=0\mathit{and}R=0\iff I({\mathbf{X}}_1,{\mathbf{X}}_2;{\mathbf{X}}_3)=0.$$

3.3. Feasible Values for the Parameter t

From (3), the m-projection from manifold M to the e-geodesic passing through the pdfs $p_5$ and $p_6$ meets in general at pdf $p_t$ which has covariance matrix ${\Sigma }_t$ defined by

$${\Sigma }_t^{-1}=(1-t){\Sigma }_5^{-1}+t{\Sigma }_6^{-1},$$

and ${\Sigma }_t$ must be positive definite. Therefore, when finding the optimal pdf $p_{t^{\ast }}$, we require to constrain the values of the parameter t to be such that ${\Sigma }_t$ is positive definite. We define the set of feasible values for t as

$$F=\{t\in \mathbb{R}:{\Sigma }_t\mathrm{is}\mathrm{positive}\mathrm{definite}\}.$$

(16)

F is a closed interval in $\mathbb{R}$ of the form $[-a,1+b]$, where $a,b>0$. The interior of F—the open interval $[-a,1+b]$—is an open convex set. To enable the derivation of explicit results, it is useful to define the matrix ${\Sigma }_t^{\prime }$ by

$${\Sigma }_t^{\prime -1}=(1-t){\Sigma }_6+t{\Sigma }_5.$$

(17)

We also require a feasible value $t^{\ast }$ for t when working with the matrix ${\Sigma }_t^{\prime }$, and so we define the set G of feasible values as follows

$$G=\{t\in \mathbb{R}:{\Sigma }_t^{\prime }\mathrm{is}\mathrm{positive}\mathrm{definite}\}.$$

(18)

It turns out that the sets of feasible values $F,G$ are actually the same set, as stated in the following lemma, which is proved in Appendix E, and this fact allows us to infer that ${\Sigma }_t^{\prime }$ is positive definite when ${\Sigma }_t$ is.

Lemma 3.

If the parameter t belongs to the closed interval [0,1], then the matrices ${\Sigma }_t$ and ${\Sigma }_t^{\prime }$ are both positive definite. Also, the two feasible sets F and G defined above in (16) and (18) are equal.

3.4. A Convex Optimisation Problem

The optimal value $t^{\ast }$ of the parameter t is defined by

$$t^{\ast }=\underset{t\in F}{\mathrm{arg\; min}}D\left(p\right||p_t)$$

(19)

The following lemma, with proof in Appendix F, provides details of the optimisation required to find $t^{\ast }$.

Lemma 4.

For $t\in F^{\circ }$, we define the real valued function g by $g\left(t\right)=D\left(p\right|\left|p_t\right)$. Then,

$$\mathrm{Tr}({\Sigma }_t^{-1}\Sigma )=m,$$

(20)

and

$$g\left(t\right)=\frac{1}{2}\log \frac{|I_{n_2}-P^T{P|}^2|I_{n_3}-Q^{T}Q\left|\right|I_{n_3}-R^{T}R|}{|\Sigma |}-\frac{1}{2}\log |(1-t){\Sigma }_6+t{\Sigma }_5|$$

(21)

Provided that the joint mutual information is positive, the minimization of g subject to the constraint $t\in F^{\circ }$, an open convex set, is a strictly convex problem, and the optimal value $t^{\ast }$ is unique.

The minimum value of g is equal to

$$\frac{1}{2}\log \frac{\left|\begin{array}{c}{I}_{n_2}-P^{T}P\end{array}\right|\left|\begin{array}{c}{I}_{n_3}-Q^{T}Q\end{array}\right|\left|\begin{array}{c}{I}_{n_3}-R^{T}R\end{array}\right|}{|\Sigma |}+\frac{1}{2}\log \frac{1}{d\left(t^{\ast }\right)},$$

(22)

where the determinant $d\left(t\right)$ is defined by

$$d\left(t\right)=\left|\begin{array}{c}{I}_{n_3}-{(1-t)}^2{Q}^{T}Q-t(1-t)(R^T{P}^{T}Q+Q^{T}PR)-t^2{R}^{T}R\end{array}\right|,(t\in F)$$

(23)

Alternatively, the minimum could occur at either endpoint of F.

We now define the PID components.

3.5. Definition of the PID Components

Following the proposal in [3], we define the synergy of the system to be

$$\mathrm{Syn}=D\left(p\right|\left|p_t^{\ast }\right)$$

(24)

and by Lemma 1 and (20) the expression for the synergy is

$$\mathrm{Syn}=\frac{1}{2}\log \frac{|{\Sigma }_{t^{\ast }}|}{|\Sigma |}-\frac{1}{2}m+\frac{1}{2}\mathrm{Tr}({\Sigma }_{t^{\ast }}^{-1}\Sigma )=\frac{1}{2}\log \frac{|{\Sigma }_{t^{\ast }}|}{|\Sigma |}.$$

(25)

Before defining the other PID terms, we require the following lemma, with proof in Appendix G.

Lemma 5.

The trace terms required in the definitions of the unique information are both equal to m:

$$\mathrm{Tr}\left({\Sigma }_5^{-1}{\Sigma }_{t^{\ast }}\right)=m\mathit{and}\mathrm{Tr}\left({\Sigma }_6^{-1}{\Sigma }_{t^{\ast }}\right)=m.$$

(26)

From (4), we know that

$$D\left(p\right||p_5)=D(p\left|\right|p_t^{\ast })+D(p_t^{\ast }\left|\right|p_5)$$

(27)

and we define the unique information in the system that is due to source ${\mathbf{X}}_2$ to be

$$\mathrm{Unq}2=D\left(p_t^{\ast }\right|\left|p_5\right)$$

(28)

as in [3]. By (5), we also have that

$$D\left(p\right||p_6)=D(p\left|\right|p_t^{\ast })+D(p_t^{\ast }\left|\right|p_6)$$

(29)

and we define the unique information in the system that is due to source ${\mathbf{X}}_1$ to be

$$\mathrm{Unq}1=D\left(p_t^{\ast }\right|\left|p_6\right).$$

(30)

as in [3]. Finding the optimal point, $t^{\ast }$, of minimisation of the KL divergence $D\left(p\right|\left|p_t\right)$, and the orthogonality provided by the generalised Pythagorean theorems, define a clear connection between the geometry of the tangent space to manifold M and the definition of the information-geometric PID developed herein.

By using two of the defining equations of a PID (6) and (7), there are two possible expressions for the shared information, Shd, in the system:

$$I({\mathbf{X}}_1;{\mathbf{X}}_3)-\mathrm{Unq}1\mathrm{or}I({\mathbf{X}}_2;{\mathbf{X}}_3)-\mathrm{Unq}2.$$

(31)

Using the result in Lemma 1, we may write the unique information terms as follows. The unique information provided by ${\mathbf{X}}_2$ is defined to be

$$\mathrm{Unq}2=D(p_{t^{\ast }}\left|\right|p_5)=\frac{1}{2}\log \frac{|{\Sigma }_5|}{|{\Sigma }_{t^{\ast }}|}-\frac{1}{2}m+\frac{1}{2}\mathrm{Tr}\left({\Sigma }_5^{-1}{\Sigma }_{t^{\ast }}\right)=\frac{1}{2}\log \frac{|{\Sigma }_5|}{|{\Sigma }_{t^{\ast }}|}.$$

by Lemma 5.

The unique information provided by ${\mathbf{X}}_1$ is defined to be

$$\mathrm{Unq}1=D(p_{t^{\ast }}\left|\right|p_6)=\frac{1}{2}\log \frac{|{\Sigma }_6|}{|{\Sigma }_{t^{\ast }}|}-\frac{1}{2}m+\frac{1}{2}\mathrm{Tr}\left({\Sigma }_6^{-1}{\Sigma }_{t^{\ast }}\right)=\frac{1}{2}\log \frac{|{\Sigma }_6|}{|{\Sigma }_{t^{\ast }}|}.$$

by Lemma 5.

3.6. The $I_{\mathit{ig}}$ PID

Explicit expressions for the PID components are given in Proposition 1, with proof in Appendix H.

Proposition 1.

The partial information decomposition $I_{ig}$ for the zero-mean multivariate Gaussian system defined in (12) has the following components.

$$\begin{array}{cc}\hfill \mathit{Syn}& =\frac{1}{2}\log \frac{\left|\begin{array}{c}{I}_{n_2}-P^{T}P\end{array}\right|\left|\begin{array}{c}{I}_{n_3}-Q^{T}Q\end{array}\right|\left|\begin{array}{c}{I}_{n_3}-R^{T}R\end{array}\right|}{|\Sigma |}+\frac{1}{2}\log \frac{1}{d\left(t^{\ast }\right)}\hfill \\ \hfill \mathit{Unq}1& =\frac{1}{2}\log \frac{1}{\left|\begin{array}{c}{I}_{n_3}-Q^{T}Q\end{array}\right|}-\frac{1}{2}\log \frac{1}{d\left(t^{\ast }\right)}\hfill \\ \hfill \mathit{Unq}2& =\frac{1}{2}\log \frac{1}{\left|\begin{array}{c}{I}_{n_3}-R^{T}R\end{array}\right|}-\frac{1}{2}\log \frac{1}{d\left(t^{\ast }\right)}\hfill \\ \hfill \mathit{Shd}& =\frac{1}{2}\log \frac{1}{d\left(t^{\ast }\right)}.\hfill \end{array}$$

where the determinant $d\left(t\right)$ is defined by

$$d\left(t\right)=\left|\begin{array}{c}{I}_{n_3}-{(1-t)}^2{Q}^{T}Q-t(1-t)(R^T{P}^{T}Q+Q^{T}PR)-t^2{R}^{T}R\end{array}\right|,(t\in F)$$

(32)

and F is the interval of real values of t for which ${\Sigma }_t$ is positive definite.

• The two possible expressions for the shared information in (31) are equal.

Theoretical properties of the $I_{\mathrm{ig}}$ PID are presented in Proposition 2, with proof in Appendix I.2.

Proposition 2.

The PID defined in Proposition 1 possesses the Williams–Beer properties of non-negativity, self-redundancy, symmetry and monotonicity.

3.7. Some Examples and Illustrations

Example 1.

Prediction of calcium contents.

This dataset was considered in [22]. The $I_{\mathrm{ig}}$ PID developed here, along with the $I_{\mathrm{dep}}$ PID [22] and $I_{\mathrm{mmi}}$ PID [20], was applied using data on 73 women involving one set of predictors ${\mathbf{X}}_1$ (Age, Weight, Height), another set of two predictors ${\mathbf{X}}_2$ (diameter of os calcis, diameter of radius and ulna), and target ${\mathbf{X}}_3$ (calcium content of heel and forearm). The following results were obtained.

PID	${\mathit{t}}^{\ast }$	Unq1	Unq2	Shd	Syn
$I_{\mathrm{ig}}$	0.2408	0.3581	0.0304	0.0728	0.1904
$I_{\mathrm{dep}}$		0.4077	0.0800	0.0232	0.1408
$I_{\mathrm{mmi}}$		0.3277	0	0.1032	0.2209

A plot of the ‘synergy’ function $g\left(t\right)$ is shown in Figure 2a. All three PIDs indicate the presence of synergy and a large component of unique information due to the variables in ${\mathbf{X}}_1$. The $I_{\mathrm{ig}}$ PID suggests the transmission of more of the joint mutual information as shared and synergistic information and correspondingly less unique information due to either source vector than does the $I_{\mathrm{dep}}$ PID. This is true also for the results from the $I_{\mathrm{mmi}}$ PID, but it has higher values for synergistic and shared information and a lower value for Unq1 than those produced by the $I_{\mathrm{ig}}$ PID. It was shown in [22] that pdf p in manifold M provides a better fit to these data than any of the submanifold distributions. This pdf contains pairwise cross-correlation between the vectors ${\mathbf{X}}_1$ and ${\mathbf{X}}_3$, and between ${\mathbf{X}}_{\mathbf{2}}$ and ${\mathbf{X}}_3$. Hence, it is no surprise to find that a relatively large Unq1 component. One might also anticipate a large value for Unq2. That this is not the case is explained, at least partly, by the presence of unique information asymmetry, in that the mutual information between ${\mathbf{X}}_{\mathbf{1}}$ and ${\mathbf{X}}_3$ (0.4309) is much larger than that between ${\mathbf{X}}_2$ and ${\mathbf{X}}_3$ (0.1032) and also bearing in mind the constraints imposed by (6)–(10).

The PIDs were also computed with the same ${\mathbf{X}}_1$ and ${\mathbf{X}}_3$ but taking ${\mathbf{X}}_2$ to be another set of four predictors (surface area, strength of forearm, strength of leg, area of os calcis). The following results were obtained.

PID	${\mathit{t}}^{\ast }$	Unq1	Unq2	Shd	Syn
$I_{\mathrm{ig}}$	0.0027	0.3522	0.0000	0.0787	0.0186
$I_{\mathrm{dep}}$		0.3708	0.0186	0.0601	0
$I_{\mathrm{mmi}}$		0.3522	0	0.0787	0.0186

A plot of the ‘synergy’ function $g\left(t\right)$ is shown in Figure 2b. In this case, the PIDs obtained from all three methods are very similar, with the main component being unique information due to the variables in ${\mathbf{X}}_1$. The PIDs indicate almost zero synergy and almost zero unique information due to the variables in ${\mathbf{X}}_2$. In [22], it was shown that the best of the pdfs is $p_5$ associated with submanifold $S_5$. If this model were to hold exactly, then a PID must have Syn and Unq2 components that are equal to zero. Therefore, all three PIDs perform very well here, and the fact that the Unq1 component is much larger than the Shd component is due to unique information asymmetry, since the mutual information between ${\mathbf{X}}_2$ and ${\mathbf{X}}_3$ is only 0.0787. In this dataset, the $I_{\mathrm{ig}}$ PID suggests the transmission just a little more of the joint mutual information as shared and synergistic information and correspondingly less unique information due to either source vector than does the $I_{\mathrm{dep}}$ PID. The $I_{\mathrm{ig}}$ and $I_{\mathrm{mmi}}$ PIDs produce identical results (to 4 d.p.).

When working with real or simulated data, it is important to use the correct covariance matrix. In order to use the results given in Proposition 1, it is essential that the input covariance matrix has the structure of $\Sigma$, as given in (12). Further detail is provided in Appendix J.

Example 2.

PID expectations and exact results.

Since there is no way to know the true PID for any given dataset it is useful to consider situations under which some values of the PID components can be predicted, and this approach has been used in developments of the topic. Here, we consider such expectations provided by the pdfs associated with the submanifolds $S_3-S_7$, defined in Figure 1. In submanifold $S_3$, the source ${\mathbf{X}}_2$ is independent of both the other source ${\mathbf{X}}_1$ and the target ${\mathbf{X}}_3$. Hence, we expect only unique information due to source ${\mathbf{X}}_1$ to be transmitted. Submanifold $S_4$ is similar but we expect only unique information due to source ${\mathbf{X}}_2$ to be transmitted. In manifold $S_5$, ${\mathbf{X}}_2$ and ${\mathbf{X}}_3$ are conditionally independent given a value for ${\mathbf{X}}_1$. Hence, from (9), we expect the Unq2 and Syn components to be zero. Similarly, for $S_6$, we expect the Unq1 and Syn components to be equal to zero, by (8). On submanifold $S_7$, the sources ${\mathbf{X}}_1,{\mathbf{X}}_2$ are conditionally independent given a value for the target ${\mathbf{X}}_3$ (which does not mean that the sources are marginally independent). Since the target ${\mathbf{X}}_3$ interacts with both source vectors, one might expect some shared information as well as unique information from both sources, and also perhaps some synergy. Here, from (11), the interaction information must be negative or zero, and so we can expect to see transmission of more shared information than synergy.

We will examine these expectations by using the following multivariate Gaussian distribution (which was used in [22]). The matrices $P,Q,R$ are given an equi-cross-correlation structure in which all the entries are equal within each matrix:

$$P=p{\mathbf{1}}_{n_1}{\mathbf{1}}_{n_2}^T,Q=q{\mathbf{1}}_{n_1}{\mathbf{1}}_{n_3}^T,R=r{\mathbf{1}}_{n_2}{\mathbf{1}}_{n_3}^T,$$

(33)

where $p,q,r$ denote here the constant cross correlations within each matrix and ${\mathbf{1}}_n$ denotes an n-dimensional vector whose entries are each equal to unity.

The values of $(p,q,r)$ are taken to be $(-0.15,0.15,0.15)$, with $n_1=3$, $n_2=4,n_3=3$. Covariance matrices for pdfs $p_3-p_7$ were computed using the results in Table 1. Thus, we have the exact covariance matrices which can be fed into the $I_{\mathrm{ig}}$, $I_{\mathrm{dep}}$ and $I_{\mathrm{mmi}}$ algorithms. The PID results are displayed in

Table 2. PID results for exact pdfs, reported as a percentage of the joint mutual information.

pdf	PID	Unq1	Unq2	Shd	Syn
p	$I_{\mathrm{ig}}$	4.3	6.6	1.5	87.7
	$I_{\mathrm{dep}}$	4.3	6.6	1.4	87.6
	$I_{\mathrm{mmi}}$	0.0	2.3	5.8	91.9
$p_7$	$I_{\mathrm{ig}}$	15.0	33.2	31.7	20.1
	$I_{\mathrm{dep}}$	35.1	53.3	11.6	0.0
	$I_{\mathrm{mmi}}$	0.0	18.2	46.7	35.1
$p_6$	$I_{\mathrm{ig}}$	0.0	75.9	24.1	0.0
	$I_{\mathrm{dep}}$	0.0	75.9	24.1	0.0
	$I_{\mathrm{mmi}}$	0.0	75.9	24.1	0.0
$p_5$	$I_{\mathrm{ig}}$	75.2	0.0	24.8	0.0
	$I_{\mathrm{dep}}$	75.2	0.0	24.8	0.0
	$I_{\mathrm{mmi}}$	75.2	0.0	24.8	0.0
$p_4$	$I_{\mathrm{ig}}$	0.0	100.0	0.0	0.0
	$I_{\mathrm{dep}}$	0.0	100.0	0.0	0.0
	$I_{\mathrm{mmi}}$	0.0	100.0	0.0	0.0
$p_3$	$I_{\mathrm{ig}}$	100.0	0.0	0.0	0.0
	$I_{\mathrm{dep}}$	100.0	0.0	0.0	0.0
	$I_{\mathrm{mmi}}$	100.0	0.0	0.0	0.0

.

From Table 2, we see that all three PIDs meet the expectations exactly for pdfs $p_3-p_6$, with only unique information transmitted when the pdfs $p_3,p_4$, are true, respectively, and zero unique for the relevant component and zero synergy when the models $p_5,p_6$ are true, respectively. When model $p_8$ is the true model, we find that the $I_{\mathrm{ig}}$ and $I_{\mathrm{dep}}$ PIDs produce virtually identical results: the joint mutual information is transmitted almost entirely as synergistic information. The $I_{\mathrm{mmi}}$ PID is slightly different, with less unique information transmitted about the variables in ${\mathbf{X}}_2$, and more shared and synergistic information transmitted than with the other two PIDs. The PIDs produce very different results for pdf $p_7$, although, as expected, they do express more shared information than synergy. When this model is satisfied, $I_{\mathrm{dep}}$ sets the synergy to 0, even if there is no compelling reason to support this. This curiosity is mentioned and illustrated in [22]. On the other hand, the $I_{\mathrm{ig}}$ PID suggests that each of the four components contributes to the transmission of the joint mutual information, with unique information due to ${\mathbf{X}}_2$ and shared information making more of a contribution than the other two components. The $I_{\mathrm{mmi}}$ PID transmits a higher percentage of the joint information as shared and synergistic information, and a smaller percentage due to the variables in ${\mathbf{X}}_2$, than is found with $I_{\mathrm{ig}}$; these differences are much stronger when comparison is made with the corresponding $I_{\mathrm{dep}}$ components. As with model p, it appears that the setting of the Unq1 component in $I_{\mathrm{ig}}$ to zero has been translated into its percentage being subtracted from the Unq2 component and added to both the Shd and Syn components in $I_{\mathrm{ig}}$ to produce $I_{\mathrm{mmi}}$.

Example 3.

Some simulations.

Taking the same values of $p,q,r$ and $n_1,n_2,n_3$ as in the previous example, a small simulation study was conducted. From each of the pdfs, $p_3-p_7,p$, a simple random sample of size 1000 was generated from the 10-dimensional distribution, a covariance matrix estimated from the data and the $I_{\mathrm{ig}}$, $I_{\mathrm{dep}}$ and $I_{\mathrm{mmi}}$ algorithms were applied. This procedure was repeated 1000 times. In order to make the PID results from the sample of 1000 datasets comparable each PID was normalized by dividing each of its components by the joint mutual information; see (10). A summary of the results is provided in

Table 3. PID results for simulated datasets from the pdfs, $p_3-p_7,p$, reported as median (in bold) and range of the sample of percentages of the joint mutual information, apart from $t^{\ast }$ which gives the median and range of the actual values obtained using the $I_{\mathrm{ig}}$ algorithm.

pdf	PID	${\mathit{t}}^{\ast }$	Unq1	Unq2	Shd	Syn
p	$I_{\mathrm{ig}}$	0.55	4.4	6.7	1.6	87.3
		(0.44, 0.63)	(2.6, 6.6)	(3.7, 9.0)	(1.1, 2.1)	(85.4, 89.2)
	$I_{\mathrm{dep}}$		4.5	6.8	1.4	87.2
			(2.8, 6.8)	(3.9, 9.3)	(1.0, 1.9)	(85.3, 89.0)
	$I_{\mathrm{mmi}}$		0.0	2.3	6.0	91.7
			(0.0, 2.9)	(3.2, 6.2)	(3.9, 8.0)	(89.2, 93.6)
$p_7$	$I_{\mathrm{ig}}$	0.59	15.1	33.0	31.3	20.2
		(0.47, 0.71)	(7.6, 26.5)	(19.9, 48.0)	(23.4, 37.3)	(15.2, 24.9)
	$I_{\mathrm{dep}}$		34.3	52.3	11.8	0.2
			(23.6, 46.4)	(38.9, 64.5)	(7.8, 20.3)	(0.0, 8.6)
	$I_{\mathrm{mmi}}$		0.0	18.0	46.4	35.6
			(0.0, 11.4)	(0.0, 45.9)	(30.4, 57.7)	(21.3, 45.9)
$p_6$	$I_{\mathrm{ig}}$	0.97	0.1	72.5	25.0	2.4
		(0.82, 1.06)	(0.0, 2.4)	(55.6, 84.7)	(11.4, 38.4)	(0.2, 8.1)
	$I_{\mathrm{dep}}$		2.5	74.9	22.6	0.0
			(0.2, 10.4)	(60.4, 88.6)	(7.6, 35.3)	(0.0, 0.0)
	$I_{\mathrm{mmi}}$		0.0	72.3	25.1	2.6
			(0.0, 0.0)	(45.7, 83.6)	(13.5, 45.4)	(0.3, 8.9)
$p_5$	$I_{\mathrm{ig}}$	0.06	68.0	0.3	27.4	4.3
		(−0.06, 0.20)	(50.0, 81.7)	(0.0, 2.9)	(12.5, 43.3)	(0.9, 12.1)
	$I_{\mathrm{dep}}$		72.3	4.7	22.8	0.0
			(54.9, 87.3)	(0.9, 13.3)	(6.8, 38.4)	(0.0, 0.0)
	$I_{\mathrm{mmi}}$		67.5	0.0	27.9	4.7
			(45.7, 82.6)	(0.0, 0.0)	(13.9, 45.5)	(0.9, 14.2)
$p_4$	$I_{\mathrm{ig}}$	0.97	0.06	95.0	2.5	2.4
		(0.91, 1.0)	(0.0, 0.7)	(83.0, 99.5)	(1.9, 7.8)	(0.3, 8.3)
	$I_{\mathrm{dep}}$		2.2	97.2	0.3	0.2
			(0.2, 8.4)	(90.7, 99.7)	(0.0, 2.6)	(0.0, 2.7)
	$I_{\mathrm{mmi}}$		0.0	94.9	2.5	2.6
			(0.0, 0.0)	(83.1, 99.3)	(0.1, 9.4)	(0.4, 8.6)
$p_3$	$I_{\mathrm{ig}}$	0.05	90.9	0.2	4.5	4.5
		(0.01, 0.14)	(75.7, 98.4)	(0.0, 1.8)	(0.7, 11.5)	(0.9, 11.5)
	$I_{\mathrm{dep}}$		94.8	4.2	0.3	0.3
			(86.2, 99.1)	(0.7, 12.6)	(0.0, 3.1)	(0.0, 3.5)
	$I_{\mathrm{mmi}}$		90.6	0.0	4.6	4.8
			(76.3, 97.6)	(0.0, 0.0)	(1.0, 11.9)	(1.2, 13.0)

. We focus here on the comparison of $I_{\mathrm{ig}}$ and $I_{\mathrm{dep}}$, and also $I_{\mathrm{ig}}$ and $I_{\mathrm{mmi}}$, since $I_{\mathrm{dep}}$ has been compared with $I_{\mathrm{mmi}}$ for Gaussian systems [22].

• $I_{\mathrm{ig}}$ vs. $I_{\mathrm{dep}}$

For pdf p, the $I_{\mathrm{ig}}$ and $I_{\mathrm{dep}}$ PIDs produce very similar results in terms of both median and range, and the median results are very close indeed to the corresponding exact values in Table 2. For pdf $p_7$, the differences between the PID components found in Table 2 persist here although each PID, respectively, produces median values of their components that are close to the exact results in Table 2. For the other four pdfs, there are some small but interesting differences among the results produced by the two PID methods. The $I_{\mathrm{ig}}$ method has higher median values for synergy and shared information than for the unique information, when compared against the corresponding exact values in Table 2. In particular, the values of unique information given by $I_{\mathrm{ig}}$ are much lower than expected for pdfs $p_3,p_4,p_6$, and the levels of synergy are larger than expected particularly for pdfs $p_3$ and $p_5$. On the other hand, the $I_{\mathrm{dep}}$ PID tends to have larger values for the unique information, and lower values for synergy, especially for datasets generated from pdfs $p_3$, $p_4$ and $p_5$. For models $p_3-p_6$, $I_{\mathrm{dep}}$ has median values of synergy that are closer to the corresponding exact values than those produced by $I_{\mathrm{ig}}$. The suggestion that the $I_{\mathrm{ig}}$ method can produce more synergy and shared information than the $I_{\mathrm{dep}}$ method, given the same dataset, is supported by the fact that for all the pdfs and all 6000 datasets considered, the $I_{\mathrm{ig}}$ method produced greater levels of synergy and shared information and smaller values of the unique information in every dataset. This raises a question of whether such a finding is generally the case and whether there is this type of a systematic difference between the methods. In the case of scalar variables, it is easy to derive general analytic formulae for the $I_{\mathrm{ig}}$ PID components and such a systematic difference is present in this case.

• $I_{\mathrm{ig}}$ vs. $I_{\mathrm{mmi}}$

The $I_{\mathrm{ig}}$ and $I_{\mathrm{mmi}}$ PIDs produce similar results for the datasets generated from pdf p, although the $I_{\mathrm{mmi}}$ PID suggests the transmission of more shared and synergistic information and less unique information than does $I_{\mathrm{ig}}$. For pdf $p_7$, the differences between the PID results are much more dramatic, with the $I_{\mathrm{mmi}}$ PID allocating an additional 15% of the joint mutual information to be shared and the synergistic information, and correspondingly 15% less of the unique information. Both methods produce almost identical summary statistics on the datasets generated from pdfs $p_3-p_6$. Since the same patterns are present for all four distributions, we discuss the results for pdf $p_5$ as an exemplar and compare them with the corresponding exact values in Table 2. The results for component Unq1 show that both methods produce an underestimate of approximately 7%, on average, of the joint mutual information. The median values of Unq2 are close to those expected. The underestimates on the Unq1 component are coupled with overestimates, on average, for the shared and synergistic components; they are 2.6% and 4.3%, respectively, with the $I_{\mathrm{ig}}$ method, and 3.1% and 4.7%, respectively, with $I_{\mathrm{mmi}}$.

As to be expected with percentage data, the variation in results for each component tends to be larger for values that are not extreme and much smaller for the extreme values. Also, the optimal values of $t^{\ast }$ are shown in Table 3. They were all found to be in the range [0, 1], except for 202 of the datasets generated from pdf $p_5$ or $p_6$.

4. Discussion

For the case of multivariate Gaussian systems with two vector inputs and a vector output, results have been derived using standard theorems from information geometry in order to develop simple, almost exact formulae for the $I_{\mathrm{ig}}$ PID, thus extending the scope of the work of [3] on scalar inputs and output. The formulae require one parameter to be determined by a simple, constrained convex optimisation. In addition, it has been proved that this $I_{\mathrm{ig}}$ PID algorithm satisfies the desirable theoretical properties of non-negativity, self-redundancy, symmetry and monotonicity, first postulated by Williams and Beer [1]. These results strengthen the confidence that one might have in using the $I_{\mathrm{ig}}$ method to separate the joint mutual information in a multivariate Gaussian system into shared, unique and synergistic components. The examples demonstrate that the $I_{\mathrm{ig}}$ method is simple to use and a small simulation study reveals that it is fairly robust, although in some of the scenarios considered the $I_{\mathrm{ig}}$ method produced more synergy and shared information than expected, and correspondingly less unique information; in some other scenarios, it performed as expected. Comparison of the $I_{\mathrm{ig}}$ and $I_{\mathrm{dep}}$ algorithms reveal that they can produce exactly the same, or very similar, results in some scenarios, but in other situations, it is clear that the $I_{\mathrm{ig}}$ method tends to have larger levels of shared information and synergy, and correspondingly, lower levels of unique information when compared with the results from the $I_{\mathrm{dep}}$ method.

For datasets generated from pdfs $p_3-p_6$, the PIDs produced using the $I_{\mathrm{ig}}$ and $I_{\mathrm{mmi}}$ methods are, on average, very similar indeed, and both methods overestimate synergy and shared information and underestimate unique information. The extent of these biases, as a percentage of the joint mutual information, is fairly small, on average, when pdf $p_4$ or $p_6$ is the true pdf, but larger, on average, when $p_3$ or $p_5$ is the true pdf. When pdf $p_7$ or p is the true pdf, the $I_{\mathrm{mmi}}$ algorithm produces even more shared and synergistic information than obtained with the $I_{\mathrm{ig}}$ method. This effect is particularly dramatic in the case of $p_7$, where on average with $I_{\mathrm{mmi}}$ 82% of the joint mutual information is transmitted as shared or synergistic information, as compared with 51.5% for $I_{\mathrm{ig}}$. It appears that the fact that the $I_{\mathrm{mmi}}$ method forces one of the unique informations to be zero leads to an underestimation of the other unique information and an overestimate of both the shared and synergistic information, especially when $p_7$ or p is the true pdf and both unique information are expected to be non-zero.

While some numerical support is presented here for the hypothesis that there might be a systematic difference in this type between the $I_{\mathrm{ig}}$ and $I_{\mathrm{dep}}$ methods further research would be required to investigate this possibility. Also, the $I_{\mathrm{ig}}$ developed here is a bivariate PID and it would be of interest to explore whether the method could be extended to deal with more than two source vectors.