Orders between Channels and Implications for Partial Information Decomposition

Abstract

The partial information decomposition (PID) framework is concerned with decomposing the information that a set of random variables has with respect to a target variable into three types of components: redundant, synergistic, and unique. Classical information theory alone does not provide a unique way to decompose information in this manner, and additional assumptions have to be made. Recently, Kolchinsky proposed a new general axiomatic approach to obtain measures of redundant information based on choosing an order relation between information sources (equivalently, order between communication channels). In this paper, we exploit this approach to introduce three new measures of redundant information (and the resulting decompositions) based on well-known preorders between channels, contributing to the enrichment of the PID landscape. We relate the new decompositions to existing ones, study several of their properties, and provide examples illustrating their novelty. As a side result, we prove that any preorder that satisfies Kolchinsky’s axioms yields a decomposition that meets the axioms originally introduced by Williams and Beer when they first proposed PID.

1. Introduction

Williams and Beer [1] proposed the partial information decomposition (PID) framework as a way to characterize or analyze the information that a set of random variables (often called sources) has about another variable (referred to as the target). PID is a useful tool for gathering insights and analyzing the way information is stored, modified, and transmitted within complex systems [2,3]. It has found applications in areas such as cryptography [4] and neuroscience [5,6], with many other potential use cases, such as in understanding how information flows function in gene regulatory networks [7], neural coding [8], financial markets [9], and network design [10].

Consider the simplest case, that of a three-variable joint distribution $p(y_1,y_2,t)$ describing three random variables: two sources $Y_1$ and $Y_2$ and a target T. Notice that despite the names sources and target, there is no directionality assumption, either causal or otherwise. The goal of PID is to decompose the information that $Y=(Y_1,Y_2)$ has about T into the sum of four non-negative quantities: the information that is present in both $Y_1$ and $Y_2$, known as redundant information R; the information that only $Y_1$ (respectively, $Y_2$) has about T, known as unique information $U_1$ (respectively, $U_2$); and the synergistic information S that is present in the pair $(Y_1,Y_2)$, and is not present in either $Y_1$ or $Y_2$ alone. That is, in this case with two variables, the goal is to write

$$I(T;Y)=R+U_1+U_2+S,$$

(1)

where $I(T;Y)$ is the mutual information between T and Y [11]. In this paper, mutual information is always assumed to refer to Shannon’s mutual information, which for two discrete variables $X\in \mathcal{X}$ and $Z\in \mathcal{Z}$ is provided by

$$I(X;Z)=\sum _{x\in \mathcal{X}}\sum _{z\in \mathcal{Z}}p(x,z)log\frac{p(x,z)}{p\left(x\right)p\left(z\right)},$$

and satisfies the following well-known fundamental properties: $I(X;Z)\ge 0$ and $I(X;Z)=0\iff X\perp Z$ (X and Z are independent) [11].

Because unique information and redundancy satisfy the relationship $U_i=I(T;Y_i)-R$ (for $i\in \{1,2\}$), it turns out that defining how to compute one of these quantities (R, $U_i$, or S) is enough to fully determine the others [1]. As the number of variables grows, the number of terms appearing in the PID of $I(T;Y)$ grows super-exponentially [12]. Williams and Beer [1] suggested a set of axioms that a measure of redundancy should satisfy and proposed a measure of their own. These axioms have become known as the Williams–Beer axioms, although the measure they proposed has subsequently been criticized for not capturing informational content, only information size [13].

Spawned by their initial work, other measures and axioms for information decomposition have been introduced; see, for example, the work by Bertschinger et al. [14], Griffith and Koch [15], and James et al. [16]. There is no consensus about what axioms any measure should satisfy or whether a given measure is capturing the information that it should capture other than the Williams–Beer axioms. Today, debate continues about which axioms a measure of redundant information ought to satisfy, and there is no general agreement on what makes for an appropriate PID [16,17,18,19,20].

Recently, Kolchinsky [21] suggested a new general approach to defining measures of redundant information, known as intersection information (II), the designation that we adopt hereinafter. The core of this approach is the choice of an order relation between information sources (random variables), which allows two sources to be compared in terms of how informative they are with respect to the target variable.

In this work, we use previously studied preorders between communication channels, which correspond to preorders between the corresponding output variables in terms of information content with respect to the input. Following Kolchinsky’s approach, we show that these orders lead to the definition of new II measures. The rest of the paper is organized as follows. In Section 2 and Section 3, we review Kolchinsky’s definition of an II measure and the degradation order. In Section 4, we describe a number of preorders between channels then, based on the work by Korner and Marton [22] and Américo et al. [23], we derive the resulting II measures and study of their properties. Section 5 presents comments on the optimization problems involved in computation of the proposed measures. In Section 6, we explore the relationships between the new II measures and previous PID approaches, then apply the proposed II measures to several famous PID problems. Section 7 concludes the paper by pointing out suggestions for future work.

2. Kolchinsky’s Axioms and Intersection Information

Consider a set of n discrete random variables $Y_1\in {\mathcal{Y}}_1,\dots ,Y_n\in {\mathcal{Y}}_n$, called the source variables, and let $T\in \mathcal{T}$ be the target variable (also discrete), with a joint distribution (probability mass function) $p(y_1,\dots ,y_n,t)$. Let ⪯ denote some preorder between random variables that satisfies the following axioms, herein referred to as Kolchinsky’s axioms [21]:

(i) 

Monotonicity of mutual information w.r.t. T: $Y_i⪯Y_j\Rightarrow I(Y_i;T)\le I(Y_j;T)$.

(ii) 

Reflexivity: $Y_i⪯Y_i$ for all $Y_i$.

(iii) 

For any $Y_i$, $C⪯Y_i⪯(Y_1,\dots ,Y_n)$, where $C\in \mathcal{C}$ is any variable taking a constant value with probability one, i.e., with a distribution that is a delta function or such that $\mathcal{C}$ is a singleton.

Kolchinsky [21] showed that such an order can be used to define an II measure via

$$I_{\cap }(Y_1,\dots ,Y_n\to T):=\underset{Q:Q⪯Y_i,i\in \{1,..,n\}}{sup}I(Q;T),$$

(2)

and we now show that this implies that the II measure in (2) satisfies the Williams–Beer axioms [1,2], establishing a strong connection between these formulations. Before stating and proving this result, we first recall the Williams–Beer axioms [2], where the definition of a source $A_i$ is that of a set of random variables, e.g., $A_1=\{X_1,X_2\}$.

Definition 1.

Let $A_1,\dots ,A_r$ be an arbitrary number of $r\ge 2$ sources. An intersection information measure $I_{\cap }$ is said to satisfy the Williams-Beer axioms if it satisfies the following:

1. 

Symmetry: $I_{\cap }$ is symmetric in the $A_i$s.

2. 

Self-redundancy: $I_{\cap }\left(A_i\right)=I(A_i;T)$.

3. 

Monotonicity: $I_{\cap }(A_1,\dots ,A_{r-1},A_r)\le I_{\cap }(A_1,\dots ,A_{r-1})$.

4. 

Equality for Monotonicity: If $A_{r-1}\subseteq A_r$, then $I_{\cap }(A_1,\dots ,A_{r-1},A_r)=I_{\cap }(A_1,\dots ,A_{r-1})$.

Theorem 1.

Let ⪯ be some preorder that satisfies Kolchinsky’s axioms, and define its corresponding II measure as in (2). Then, the corresponding II measure satisfies the Williams–Beer axioms.

Proof. 

Symmetry and monotonicity follow trivially given the form of (2) (the definition of the supremum and restriction set). Self-redundancy follows from the reflexivity of the preorder and monotonicity of mutual information. Now, suppose $A_{r-1}\subseteq A_r$, and let Q be a solution of $I_{\cap }(A_1,\dots ,A_{r-1})$, implying that $Q⪯A_{r-1}$. Now, because $A_{r-1}\subseteq A_r$, the third Kolchinsky axiom and transitivity of the preorder ⪯ guarantee that $Q⪯A_{r-1}⪯A_r$, meaning that Q is an admissible point of $I_{\cap }(A_1,\dots ,A_r)$. Therefore, $I_{\cap }(A_1,\dots ,A_{r-1},A_r)\ge I_{\cap }(A_1,\dots ,A_{r-1})$ and monotonicity guarantees that $I_{\cap }(A_1,\dots ,A_{r-1},A_r)=I_{\cap }(A_1,\dots ,A_{r-1})$.    □

In conclusion, every preorder relation that satisfies the set of axioms introduced by Kolchinsky [21] yields a valid II measure, in the sense that the measure satisfies the Williams–Beer axioms. Having a more informative relation ⪯ allows us to draw conclusions about information flowing from different sources, and allows for the construction of PID measures that are well-defined for more than two sources. In the following, we omit “$\to T$” from the notation unless we need to explicitly refer to it, with the understanding that the target variable is always some arbitrary discrete random variable T.

3. Channels and the Degradation/Blackwell Order

In an information-theoretical perspective, given two discrete random variables $X\in \mathcal{X}$ and $Z\in \mathcal{Z}$, the corresponding conditional distribution $p\left(z\right|x)$ corresponds to a discrete memoryless channel with a channel matrix K such that $K[x,z]=p\left(z\right|x)$ [11]. This matrix is row-stochastic, i.e., $K[x,z]\ge 0$ for any $x\in \mathcal{X}$ and $z\in \mathcal{Z}$, and ${\sum }_{z\in \mathcal{Z}}K[x,z]=1$.

The comparison of different channels (equivalently, different stochastic matrices) is an object of study with many applications in different fields [24]. Such investigations address order relations between channels and their properties. One such order, named the degradation order (or Blackwell order) and defined next, was used by Kolchinsky to obtain a particular II measure [21].

Consider the distribution $p(y_1,\dots ,y_n,t)$ and the channels $K^{\left(i\right)}$ between T and each $Y_i$, that is, $K^{\left(i\right)}$ is a $\left|\mathcal{T}\right|\times |{\mathcal{Y}}_i|$ row-stochastic matrix with the conditional distribution $p\left(y_i\right|t)$.

Definition 2.

We say that channel $K^{\left(i\right)}$ is a degradation of channel $K^{\left(j\right)}$, and write $K^{\left(i\right)}{⪯}_d{K}^{\left(j\right)}$ or $Y_i{⪯}_d{Y}_j$, if there exists a channel $K^U$ from $Y_j$ to $Y_i$, i.e., a $|{\mathcal{Y}}_j|\times |{\mathcal{Y}}_i|$ row-stochastic matrix, such that $K^{\left(i\right)}=K^{\left(j\right)}{K}^U$.

Intuitively, consider two agents, one with access to $Y_i$ and the other with access to $Y_j$. The agent with access to $Y_j$ has at least as much information about T as the one with access to $Y_i$, as it has access to channel $K^U$, which permits sampling from $Y_i$ conditionally on $Y_j$ [19]. Blackwell [25] showed that this is equivalent to saying that, for whatever decision game where the goal is to predict T and for whatever utility function, the agent with access to $Y_i$ cannot do better on average than the agent with access to $Y_j$.

Based on the degradation/Blackwell order, Kolchinsky [21] introduced the degradation II measure by plugging the “${⪯}_d$” order into (2):

$$I_{\cap }^d(Y_1,\dots ,Y_n):=\underset{Q:Q{⪯}_d{Y}_i,i\in \{1,..,n\}}{sup}I(Q;T).$$

(3)

As noted by Kolchinsky [21], this II measure has the following operational interpretation. Supposing that $n=2$ and considering two agents, 1 and 2, with access to variables $Y_1$ and $Y_2$, respectively, $I_{\cap }^d(Y_1,Y_2)$ is the maximum information that agent 1 (respectively 2) can have with respect to T without being able to do better than agent 2 (respectively 1) on any decision problem that involves guessing T. That is, the degradation II measure quantifies the existence of a dominating strategy for any guessing game.

4. Other Orders and Corresponding II Measures

4.1. The “Less Noisy” Order

Korner and Marton [22] introduced and studied preorders between channels with the same input. We follow most of their definitions, and change others when appropriate. We interchangeably write $Y_1⪯Y_2$ to mean $K^{\left(1\right)}⪯K^{\left(2\right)}$, where $K^{\left(1\right)}$ and $K^{\left(2\right)}$ are the channel matrices defined above.

Before introducing the next channel order, we need to review the notion of Markov chains [11]. We can say that three random variables $X_1$, $X_2$, and $X_3$ form a Markov chain, for which we write $X_1\to X_2\to X_3$, if the following equality holds: $p(x_1,x_3|x_2)=p\left(x_1\right|x_2)p\left(x_3\right|x_2)$, i.e., if $X_1$ and $X_3$ are conditionally independent given $X_2$. Of course, $X_1\to X_2\to X_3$ if and only if $X_3\to X_2\to X_1$.

Definition 3.

We say that channel $K^{\left(2\right)}$ is less noisy than channel $K^{\left(1\right)}$, and write $K^{\left(1\right)}{⪯}_{ln}{K}^{\left(2\right)}$, if for any discrete random variable U with finite support (such that both $U\to T\to Y_1$ and $U\to T\to Y_2$ hold) we have $I(U;Y_1)\le I(U;Y_2)$.

The less noisy order has been primarily used in network information theory to study the problems of the capacity regions of broadcast channels [26] and the secrecy capacity of wiretap and eavesdrop channels [27]. The secrecy capacity ($C_S$) is the maximum rate at which information can be transmitted over a communication channel while keeping the communication secure from eavesdroppers, that is, having zero information leakage [28,29]. It has been shown that $C_S>0$ unless $K^{\left(2\right)}{⪯}_{ln}{K}^{\left(1\right)}$, where $C_S$ is the secrecy capacity of the Wyner wiretap channel, with $K^{\left(2\right)}$ as the main channel and $K^{\left(1\right)}$ as the eavesdropper channel ([27], Corollary 17.11).

Plugging the less noisy order ${⪯}_{ln}$ into (2) yields a new II measure

$$I_{\cap }^{ln}(Y_1,\dots ,Y_n):=\underset{Q:Q{⪯}_{ln}{Y}_i,i\in \{1,..,n\}}{sup}I(Q;T).$$

(4)

Intuitively, $I_{\cap }^{ln}(Y_1,\dots ,Y_n)$ is the most information that a channel $K^Q$ can have about T such that it is less noisy than any other channel $K^{\left(i\right)},i=1,\dots ,n$, that is, a channel that leads to a positive secrecy capacity, as compared to any other channel $K^{\left(i\right)}$.

4.2. The “More Capable” Order

The next order we consider, termed “more capable”, has been used in calculating the capacity region of broadcast channels [30] and to help determine whether one system is more secure than another [31]; see the book by Cohen et al. [24] for more applications of the degradation, less noisy, and more capable orders.

Definition 4.

We say that channel $K^{\left(2\right)}$ is more capable than $K^{\left(1\right)}$, and write $K^{\left(1\right)}{⪯}_{mc}{K}^{\left(2\right)}$, if for any distribution $p\left(t\right)$ we have $I(T;Y_1)\le I(T;Y_2)$.

Inserting the “more capable” order into (2) leads to

$$I_{\cap }^{mc}(Y_1,\dots ,Y_n):=\underset{Q:Q{⪯}_{mc}{Y}_i,i\in \{1,..,n\}}{sup}I(Q;T),$$

(5)

that is, $I_{\cap }^{mc}(Y_1,\dots ,Y_n)$ is the information that the ‘largest’ (in the more capable sense), though no larger than any $Y_i$, that the random variable Q has with respect to T. Whereas under the degradation order it is guaranteed that agent 2 will make better decisions if $Y_1{⪯}_d{Y}_2$ for whatever decision game, on average, under the “more capable“ order, such a guarantee is not available. However, we do have a guarantee that, if $Y_1{⪯}_{mc}{Y}_2$, then for a given distribution $p\left(t\right)$ we know that agent 2 always has more information about T than agent 1. This has an interventional approach meaning; if we intervene on variable T by changing its distribution $p\left(t\right)$ in whichever way we see fit, we have $I(Y_1;T)\le I(Y_2;T)$ (assuming that the distribution $p(Y_1,\dots ,Y_n,T)$ can be modeled as a set of channels from T to each $Y_i$); that is to say, $I_{\cap }^{mc}(Y_1,\dots ,Y_n)$ is the highest information that a channel $K^Q$ can have about T such that for any change in $p\left(t\right)$, $K^Q$ knows less about T than any $Y_i,i=1,\dots ,n$. Because PID is concerned with decomposing a distribution that has fixed $p\left(t\right)$, the “more capable” measure is concerned with the mechanism by which T generates $Y_1,\dots ,Y_n$ for any $p\left(t\right)$, and is not concerned with the specific distribution $p\left(t\right)$ yielded by $p(Y_1,\dots ,Y_n,T)$.

For the sake of completeness, we could additionally study the II measure that would result from the capacity order. Recall that the capacity of the channel from a variable X to another variable Z, which is only a function of the conditional distribution $p\left(z\right|x)$, is defined as [11]

$$C=\underset{p\left(x\right)}{max}I(X;Z).$$

(6)

Definition 5.

We can write $W{⪯}_{c}V$ if the capacity of V is at least as large as the capacity of W.

Even though it is clear that $W{⪯}_{mc}V\Rightarrow W{⪯}_{c}V$, the ${⪯}_c$ order does not comply with the first of Kolchinsky’s axioms, as the definition of capacity involves the choice of a particular marginal that achieves the maximum in (6), which may not coincide with the marginal corresponding to $p(y_1,\dots ,y_n,t)$. For this reason, we do not define an II measure based on it.

4.3. The “Degradation/Supermodularity” Order

In order to introduce the last II measure, we follow the work and notation of Américo et al. [23]. Given two real vectors r and s with dimension n, let $r\vee s:=(max(r_1,s_1),$$\dots ,max(r_n,s_n))$ and $r\wedge s:=(min(r_1,s_1),\dots ,min(r_n,s_n))$. Consider an arbitrary channel K, and let $K_i$ be its ith column. From K, we may define a new channel which we construct column by column using the JoinMeet operator ${\diamond }_{i,j}$. Column l of the new channel is defined for $i\ne j$ as

$${\left({\diamond }_{i,j}K\right)}_l=\left\{\begin{array}{cc}{K}_i\vee K_j,\hfill & \mathrm{i}\mathrm{f} l=i\hfill \\ K_i\wedge K_j,\hfill & \mathrm{i}\mathrm{f} l=j\hfill \\ K_l,\hfill & otherwise\hfill \end{array}\right..$$

Américo et al. [23] used this operator to define the two new orders described below. Intuitively, the operator ${\diamond }_{i,j}$ makes the rows of the channel matrix more similar to each other by putting all the maxima in column i and the minima in column j between every pair of elements in columns i and j of every row. In the following definitions, the s stands for supermodularity, a concept we need not introduce in this work.

Definition 6.

We can write $W{⪯}_{s}V$ if there exists a finite collection of tuples $(i_k,j_k)$ such that $W={\diamond }_{i_1,j_1}({\diamond }_{i_2,j_2}(\dots \left({\diamond }_{i_m,j_m}V\right))$.

Definition 7.

Ww write $W{⪯}_{ds}V$ if there are m channels $U^{\left(1\right)},\dots ,U^{\left(m\right)}$ such that $W{⪯}_0{U}^{\left(1\right)}{⪯}_1{U}^{\left(2\right)}{⪯}_2\dots {⪯}_{m-1}{U}^{\left(m\right)}{⪯}_{m}V$, where each ${⪯}_i$ stands for ${⪯}_d$ or ${⪯}_s$. We call this the degradation/supermodularity order.

Using the “degradation/supermodularity” (ds) order, we can define the ds II measure as follows:

$$I_{\cap }^{ds}(Y_1,\dots ,Y_n):=\underset{Q:Q{⪯}_{ds}{Y}_i,i\in \{1,..,n\}}{sup}I(Q;T).$$

(7)

The ds order was recently introduced in the context of core-concave entropies [23]. Given a core-concave entropy H, the leakage about T through $Y_1$ is defined as $I_H(T;Y_1)=H\left(T\right)-H\left(T\right|Y_1)$. In this work, we are mainly concerned with the Shannon entropy H; however, as we elaborate in the future work section at the end of this paper, PID may be applied to other core-concave entropies. Although the operational interpretation of the ds order is not yet clear, it has found applications in privacy/security contexts, as well as in finding the most secure deterministic channel (under certain constraints) [23].

4.4. Relations between Orders

Korner and Marton [22] proved that $W{⪯}_{d}V\Rightarrow W{⪯}_{ln}V\Rightarrow W{⪯}_{mc}V$, and provided examples to show that the reverse implications do not hold in general. As Américo et al. [23] note, the degradation (${⪯}_d$), supermodularity (${⪯}_s$), and degradation/ supermodularity (${⪯}_{ds}$) orders are structural orders, in the sense that they only depend on the conditional probabilities that are defined by each channel. On the other hand, the less noisy and more capable orders are concerned with information measures resulting from different distributions. It is trivial to see (directly from the definition) that the degradation order implies the degradation/supermodular order. In turn, Américo et al. [23] showed that the degradation/supermodular order implies the more capable order. This set of implications is schematically depicted in Figure 1.

For any set of variables $Y_1,\dots ,Y_n,T$, these relations between the orders imply, via the corresponding definitions, that

$$\begin{array}{c}\hfill I_{\cap }^d(Y_1,\dots ,Y_n)\le I_{\cap }^{ln}(Y_1,\dots ,Y_n)\le I_{\cap }^{mc}(Y_1,\dots ,Y_n)\end{array}$$

(8)

and

$$\begin{array}{c}\hfill I_{\cap }^d(Y_1,\dots ,Y_n)\le I_{\cap }^{ds}(Y_1,\dots ,Y_n)\le I_{\cap }^{mc}(Y_1,\dots ,Y_n).\end{array}$$

(9)

These, in turn, imply the following result.

Theorem 2.

The preorders ${⪯}_{ln}$, ${⪯}_{mc}$, and ${⪯}_{ds}$, satisfy Kolchinsky’s axioms.

Proof. 

Let $i\in \{1,\dots ,n\}$. Because any of the introduced orders implies the more capable order, it follows that they all satisfy the axiom of monotonicity of mutual information. Axiom 2 is trivially true, as reflexivity is guaranteed by the definition of preorder. For axiom 3, the rows of a channel corresponding to a variable C taking a constant value must all be the same (and yield zero mutual information with any target variable T), from which it is clear that any $Y_i$ satisfies $C⪯Y_i$ for any of the introduced orders per the definition of each order. To see that $Y_i⪯Y=(Y_1,\dots ,Y_n)$ for the less noisy and the more capable, recall that for any U such that $U\to T\to Y_i$ and $U\to T\to Y$ it is trivial that $I(U;Y_i)\le I(U;Y)$; hence, $Y_i{⪯}_{ln}Y$. A similar argument can be used to show that $Y_i{⪯}_{mc}Y$, as $I(T;Y_i)\le I(T;Y)$. Finally, to see that $Y_i{⪯}_{ds}(Y_1,\dots ,Y_n)$, note that $Y_i{⪯}_d(Y_1,\dots ,Y_n)$ [21]; hence, $Y_i{⪯}_{ds}(Y_1,\dots ,Y_n)$.    □

5. Optimization Problems

We now focus on certain observations around optimization problems involving the introduced II measures. All of these problems seek to maximize $I(Q;T)$ (under different constraints) as a function of the conditional distribution $p\left(q\right|t)$, and equivalently with respect to the channel from T to Q, which we denote as $K^Q:=K^{Q|T}$. For fixed $p\left(t\right)$, as is the case in PID, $I(Q;T)$ is a convex function of $K^Q$ ([11], Theorem 2.7.4). As we will see, the admissible region of all problems is a compact set, and because $I(Q;T)$ is a continuous function of the parameters of $K^Q$, the supremum is achieved; thus, we replace sup here with max.

As noted by Kolchinsky [21], the computation of (3) can be rewritten as an optimization problem using auxiliary variables such that it involves only linear constraints, and because the objective function is convex, its maximum is attained at one of the vertices of the admissible region. The computation of the other measures, however, is not as simple, as shown in the following subsections.

5.1. The “Less Noisy” Order

To solve (4), we can use one of the necessary and sufficient conditions presented by (Makur and Polyanskiy [26], Theorem 1). For instance, let V and W be two channels with input T, and let ${\mathrm{\Delta }}^{T-1}$ be the probability simplex of the target T. Then, $V{⪯}_{ln}W$, if and only if the inequality

$${\chi }^2\left(p\left(t\right)W\right|\left|q\left(t\right)W\right)\ge {\chi }^2\left(p\left(t\right)V\right|\left|q\left(t\right)V\right)$$

(10)

holds for any pair of distributions $p\left(t\right),q\left(t\right)\in {\mathrm{\Delta }}^{T-1}$, where ${\chi }^2$ in the above equation denotes the ${\chi }^2$-distance between two vectors. The ${\chi }^2$ distance between two vectors u and v of dimension n is given by ${\chi }^2\left(u\right|\left|v\right)={\sum }_{i=1}^n{(u_i-v_i)}^2/v_i$. Notice that $p\left(t\right)W$ is the distribution of the output of channel W for input distribution $p\left(t\right)$; thus, intuitively, the condition in (10) means that the two output distributions of the less noisy channel are more different from each other than those of the other channel. Hence, computing $I_{\cap }^{ln}(Y_1,\dots ,Y_n)$ can be formulated as solving the problem

$$\begin{array}{cc}\hfill \underset{K^Q}{max}& I(Q;T)\hfill \\ \hfill \mathrm{s}.\mathrm{t}.& K^Q\mathrm{i}\mathrm{s} \mathrm{a} \mathrm{s}\mathrm{t}\mathrm{o}\mathrm{c}\mathrm{h}\mathrm{a}\mathrm{s}\mathrm{t}\mathrm{i}\mathrm{c} \mathrm{m}\mathrm{a}\mathrm{t}\mathrm{r}\mathrm{i}\mathrm{x},\hfill \\ \hfill & \forall p\left(t\right),q\left(t\right)\in {\mathrm{\Delta }}^{T-1},\forall i\in \{1,\dots ,n\},{\chi }^2(p\left(t\right)K^{\left(i\right)}\left|\right|q\left(t\right)K^{\left(i\right)})\ge {\chi }^2(p\left(t\right)K^Q\left|\right|q\left(t\right)K^Q).\hfill \end{array}$$

Although the restriction set is convex, as the ${\chi }^2$-divergence is an f-divergence with convex f [27], the problem is intractable because we have an infinite (uncountable) number of restrictions. It is possible to construct a set $\mathcal{S}$ by taking an arbitrary number of samples S of $p\left(t\right)\in {\mathrm{\Delta }}^{T-1}$ to define the problem

$$\begin{array}{cc}\hfill \underset{K^Q}{max}& I(Q;T),\hfill \\ \hfill \mathrm{s}.\mathrm{t}.& K^Q\mathrm{i}\mathrm{s} \mathrm{a} \mathrm{s}\mathrm{t}\mathrm{o}\mathrm{c}\mathrm{h}\mathrm{a}\mathrm{s}\mathrm{t}\mathrm{i}\mathrm{c} \mathrm{m}\mathrm{a}\mathrm{t}\mathrm{r}\mathrm{i}\mathrm{x},\hfill \\ \hfill & \forall p\left(t\right),q\left(t\right)\in \mathcal{S},\forall i\in \{1,\dots ,n\},{\chi }^2(p\left(t\right)K^{\left(i\right)}\left|\right|q\left(t\right)K^{\left(i\right)})\ge {\chi }^2(p\left(t\right)K^Q\left|\right|q\left(t\right)K^Q).\hfill \end{array}$$

(11)

The above problem yields an upper bound on $I_{\cap }^{ln}(Y_1,\dots ,Y_n)$.

5.2. The “More Capable” Order

To compute $I_{\cap }^{mc}(Y_1,\dots ,Y_n)$, we can define the problem

$$\begin{array}{cc}\hfill \underset{K^Q}{max}& I(Q;T)\hfill \\ \hfill \mathrm{s}.\mathrm{t}.& K^Q\mathrm{i}\mathrm{s} \mathrm{a} \mathrm{s}\mathrm{t}\mathrm{o}\mathrm{c}\mathrm{h}\mathrm{a}\mathrm{s}\mathrm{t}\mathrm{i}\mathrm{c} \mathrm{m}\mathrm{a}\mathrm{t}\mathrm{r}\mathrm{i}\mathrm{x},\hfill \\ \hfill & \forall p\left(t\right)\in {\mathrm{\Delta }}^{T-1},\forall i\in \{1,\dots ,n\},I(Y_i;T)\ge I(Q;T),\hfill \end{array}$$

(12)

which again leads to a convex restriction set, as $I(Q;T)$ is a convex function of $K^Q$. We can discretize the problem in the same manner as above to obtain a tractable version

$$\begin{array}{cc}\hfill \underset{K^Q}{max}& I(Q;T)\hfill \\ \hfill \mathrm{s}.\mathrm{t}.& K^Q\mathrm{i}\mathrm{s} \mathrm{a} \mathrm{s}\mathrm{t}\mathrm{o}\mathrm{c}\mathrm{h}\mathrm{a}\mathrm{s}\mathrm{t}\mathrm{i}\mathrm{c} \mathrm{m}\mathrm{a}\mathrm{t}\mathrm{r}\mathrm{i}\mathrm{x},\hfill \\ \hfill & \forall p\left(t\right)\in \mathcal{S},\forall i\in \{1,\dots ,n\},I(Y_i;T)\ge I(Q;T),\hfill \end{array}$$

(13)

which again yields an upper bound on $I_{\cap }^{mc}(Y_1,\dots ,Y_n)$.

5.3. The “Degradation/Supermodularity” Order

The final introduced measure, $I_{\cap }^{ds}(Y_1,\dots ,Y_n)$, is provided by

$$\begin{array}{cc}\hfill \underset{K^Q}{max}& I(Q;T)\hfill \\ \hfill \mathrm{s}.\mathrm{t}.& K^Q\mathrm{i}\mathrm{s} \mathrm{a} \mathrm{s}\mathrm{t}\mathrm{o}\mathrm{c}\mathrm{h}\mathrm{a}\mathrm{s}\mathrm{t}\mathrm{i}\mathrm{c} \mathrm{m}\mathrm{a}\mathrm{t}\mathrm{r}\mathrm{i}\mathrm{x},\hfill \\ \hfill & \forall i,K^Q{⪯}_{ds}{K}^{\left(i\right)}.\hfill \end{array}$$

(14)

To the best of our knowledge, there is currently no known condition to check whether $K^Q{⪯}_{ds}{K}^{\left(i\right)}$.

6. Relation to Existing PID Measures

Griffith et al. [32] introduced a measure of II as

$$\begin{array}{c}\hfill I_{\cap }^{◃}(Y_1,\dots ,Y_n):=\underset{Q}{max}I(Q;T),\mathrm{such}\mathrm{that}\forall iQ◃Y_i,\end{array}$$

(15)

with the order relation ◃ defined by $A◃B$ if $A=f\left(B\right)$ for some deterministic function f, that is, $I_{\cap }^{◃}$ is used to quantify the redundancy as the presence of deterministic relations between the input and target. If Q is a solution of (15), then there exist functions $f_1,\dots ,f_n$ such that $Q=f_i\left(Y_i\right),i=1,\dots ,n$, which implies that for all i it is the case that $T\to Y_i\to Q$ is a Markov chain. Therefore, Q is an admissible point of the optimization problem that defines $I_{\cap }^d(Y_1,\dots ,Y_n)$, and we have $I_{\cap }^{◃}(Y_1,\dots ,Y_n)\le I_{\cap }^d(Y_1,\dots ,Y_n)$.

Barrett [33] introduced the so-called minimum mutual information (MMI) measure of bivariate redundancy as

$$I_{\cap }^{\mathrm{MMI}}(Y_1,Y_2):=min\{I(T;Y_1),I(T;Y_2)\}.$$

It turns out that if $(Y_1,Y_2)$ is jointly Gaussian and T is univariate, then most of the introduced PIDs in the literature are equivalent to this measure [33]. Furthermore, as noted by Kolchinsky [21], it may be generalized to more than two sources:

$$I_{\cap }^{\mathrm{MMI}}(Y_1,\dots ,Y_n):=\underset{Q}{sup}I(Q;T),\mathrm{such}\mathrm{that}\forall iI(Q;T)\le I(Y_i;T),$$

which allows us to trivially conclude that for any set of variables $Y_1,\dots ,Y_n,T$,

$$I_{\cap }^{◃}(Y_1,\dots ,Y_n)\le I_{\cap }^d(Y_1,\dots ,Y_n)\le I_{\cap }^{mc}(Y_1,\dots ,Y_n)\le I_{\cap }^{\mathrm{MMI}}(Y_1,\dots ,Y_n).$$

One of the appeals of measures of II as defined by Kolchinsky [21] is that the underlying preorder determines what is intersection (or redundant) information. For example, taking the degradation II measure in the $n=2$ case, its solution Q satisfies $T\perp Q|Y_1$ and $T\perp Q|Y_2$; that is, if either $Y_1$ or $Y_2$ are known, then Q has no additional information about T. The same is not necessarily the case for the less noisy or the more capable II measures, where the solution Q may have additional information about T even when a source is known. However, the three proposed measures satisfy the property that any solution Q of the optimization problem satisfies

$$\forall i\in \{1,\dots ,n\},\forall t\in S_T,I(Y_i;T=t)\ge I(Q;T=t),$$

where $S_T$ is the support of T and $I(T=t;Y_i)$ refers to the so-called specific information [1,34]. This means that, independent of the outcome of T, Q has less specific information about $T=t$ than any source variable $Y_i$. This can be seen by noting that any of the introduced orders imply the more capable order. This is not the case, for example, for $I_{\cap }^{\mathrm{MMI}}$, which is arguably one of the reasons why it has been criticized for depending only on the amount of information and not on its content [21]. As mentioned, there is not much consensus as to which properties a measure of II should satisfy. The three proposed measures for partial information decomposition do not satisfy the so-called Blackwell property [14,35]:

Definition 8.

An intersection information measure $I_{\cap }(Y_1,Y_2)$ is said to satisfy the Blackwell property if the equivalence $Y_1{⪯}_d{Y}_2\iff I_{\cap }(Y_1,Y_2)=I(T;Y_1)$ holds.

This definition is equivalent to demanding that $Y_1{⪯}_d{Y}_2$ if and only if $Y_1$ has no unique information about T. Although the $(\Rightarrow )$ implication holds for the three proposed measures, the reverse implication does not, as shown by specific examples presented by Korner and Marton [22], which we mention below. If we define the “more capable” property by replacing the degradation order with the more capable order in the original definition of the Blackwell property, then it is clear that measure k satisfies the k property, with k referring to any of the three introduced intersection information measures.

In PID, the identity property (IP) has been frequently studied [13]. For this property, let the target T be a copy of the source variables, that is, let $T=(Y_1,Y_2)$. An II measure $I_{\cap }$ is said to satisfy the IP if

$$I_{\cap }(Y_1,Y_2)=I(Y_1;Y_2).$$

Criticism has been levied against this proposal for being too restrictive [16,36]. A less strict property was introduced by [20] under the name independent identity property (IIP). If the target T is a copy of the input, an II measure is said to satisfy the IIP if

$$I(Y_1;Y_2)=0\Rightarrow I_{\cap }(Y_1,Y_2)=0.$$

Note that the IIP is implied by the IP, while the reverse does not hold. It turns out that all the introduced measures, as is the case for the degradation II measure, satisfy the IIP and not the IP, as we show later. This can be seen from (8) and (9), as well as from the fact that $I_{\cap }^{mc}(Y_1,Y_2\to (Y_1,Y_2))$ equals 0 if $I(Y_1;Y_2)=0$, as we argue now. Consider the distribution where T is a copy of $(Y_1,Y_2)$, as presented in

Table 1. Copy distribution.

T	${\mathit{Y}}_1$	${\mathit{Y}}_2$	$\mathit{p}(\mathit{t},{\mathit{y}}_1,{\mathit{y}}_2)$
(0, 0)	0	0	$p(T=(0,0\left)\right)$
(0, 1)	0	1	$p(T=(0,1\left)\right)$
(1, 0)	1	0	$p(T=(1,0\left)\right)$
(1, 1)	1	1	$p(T=(1,1\left)\right)$

.

We assume that each of the four events has non-zero probability. In this case, channels $K^{\left(1\right)}$ and $K^{\left(2\right)}$ are provided by

$$K^{\left(1\right)}=\left[\begin{array}{cc}1& 0\\ 1& 0\\ 0& 1\\ 0& 1\end{array}\right],K^{\left(2\right)}=\left[\begin{array}{cc}1& 0\\ 0& 1\\ 1& 0\\ 0& 1\end{array}\right].$$

Note that for any distribution $p\left(t\right)=\left[p(0,0),p(0,1),p(1,0),p(1,1)\right]$, if $p(1,0)=p(1,1)=0$, then $I(T;Y_1)=0$, which implies that for any such distributions the solution Q of (12) must satisfy $I(Q;T)=0$. Thus, the first and second rows of $K^Q$ must be the same. The same is the case for any distribution $p\left(t\right)$ with $p(0,0)=p(0,1)=0$; on the other hand, if $p(0,0)=p(1,0)=0$ or $p(1,1)=p(0,1)=0$, then $I(T;Y_2)=0$, implying that $I(Q;T)=0$ for such distributions. Hence, $K^Q$ must be an arbitrary channel, that is, a channel that satisfies $Q\perp T$, yielding $I_{\cap }^{mc}(Y_1,Y_2)=0$.

Now, recall the Gács–Korner common information [37], defined as

$$\begin{array}{ccc}\hfill C(Y_1\wedge Y_2):=& \underset{Q}{sup}\hfill & \hfill H\left(Q\right)\\ \hfill & \mathrm{s}.\mathrm{t}.\hfill & \hfill Q◃Y_1\\ \hfill & \hfill & \hfill Q◃Y_2\end{array}$$

(16)

We use a similar argument, while slightly changing the notation, to show the following result.

Theorem 3.

Let $T=(X,Y)$ be a copy of the source variables; then, $I_{\cap }^{ln}(X,Y)=I_{\cap }^{ds}(X,Y)=I_{\cap }^{mc}(X,Y)=C(X\wedge Y)$.

Proof. 

As shown by Kolchinsky [21], $I_{\cap }^d(X,Y)=C(X\wedge Y)$. Thus, (8) implies that $I_{\cap }^{mc}(X,Y)$$\ge C(X\wedge Y)$. The proof is completed by showing that $I_{\cap }^{mc}(X,Y)\le C(X\wedge Y)$. Construct the bipartite graph with vertex set $\mathcal{X}\cup \mathcal{Y}$ and edges $(x,y)$ if $p(x,y)>0$. Consider the set of maximally connected components $MCC=\{C{C}_1,\dots ,C{C}_l\}$ for some $l\ge 1$, where each $C{C}_i$ refers to a maximal set of connected edges. Let $C{C}_i,i\le l$ be an arbitrary set in $MCC$. Suppose that the edges $(x_1,y_1)$ and $(x_1,y_2)$ (with $y_1\ne y_2$) are in $C{C}_i$. This means that the channels $K^X:=K^{X|T}$ and $K^Y:=K^{Y|T}$ have rows corresponding to the outcomes $T=(x_1,y_1)$ and $T=(x_1,y_2)$ of the form

$$K^X=\left[\begin{array}{ccccccc}& & & ⋮& & & \\ 0& \cdots & 0& 1& 0& \cdots & 0\\ 0& \cdots & 0& 1& 0& \cdots & 0\\ & & & ⋮& & & \end{array}\right],K^Y=\left[\begin{array}{cccccccc}& & & ⋮& & & & \\ 0& \cdots & 0& 1& 0& 0& \cdots & 0\\ 0& \cdots & 0& 0& 1& 0& \cdots & 0\\ & & & ⋮& & & & \end{array}\right].$$

Choosing $p\left(t\right)=[0,\dots ,0,a,1-a,0,\dots ,0]$, that is, $p(T=(x_1,y_1))=a$ and $p(T=(x_1,y_2))=1-a$, we have $\forall a\in [0,1],I(X;T)=0$, which implies that the solution Q must be such that $\forall a\in [0,1],I(Q;T)=0$ (from the definition of the more capable order), which in turn implies that the rows of $K^Q$ corresponding to these outcomes must be the same to ensure that they yield $I(Q;T)=0$ under this set of distributions. We may choose the values of those rows to be the same as those rows from $K^X$, that is, a row that is composed of zeros except for one of the positions whenever $T=(x_1,y_1)$ or $T=(x_1,y_2)$. On the other hand, if the edges $(x_1,y_1)$ and $(x_2,y_1)$ (with $x_1\ne x_2$) are in $C{C}_i$, the same argument leads to the conclusion that the rows of $K^Q$ corresponding to the outcomes $T=(x_1,y_1)$, $T=(x_1,y_2)$ and $T=(x_2,y_1)$ must be the same. Applying this argument to every edge in $C{C}_i$, we can conclude that the rows of $K^Q$ corresponding to outcomes $(x,y)\in C{C}_i$ must all be the same. Using this argument for every set $C{C}_1,\dots ,C{C}_l$ implies that if two edges are in the same $CC$, the corresponding rows of $K^Q$ must be the same. These corresponding rows of $K^Q$ may vary between different CCs; however, for the same CC they must be the same.

We are left with the choice of appropriate rows of $K^Q$ for each corresponding $C{C}_i$. Because $I(Q;T)$ is maximized by a deterministic relation between Q and T and, as suggested before, we choose a row that is composed of zeros except for one of the positions for each $C{C}_i$ such that Q is a deterministic function of T, this admissible point Q implies that $Q=f_1\left(X\right)$ and $Q=f_2\left(Y\right)$, as X and Y are also functions of T under the channel perspective. For this choice of rows, we have

$$\begin{array}{ccccc}{I}_{\cap }^{mc}(X,Y)=\hfill & \underset{Q}{sup}I(Q;T)\hfill & \le \hfill & \underset{Q}{sup}H\left(Q\right)\hfill & =C(X\wedge Y)\hfill \\ & \mathrm{s}.\mathrm{t}.Q{⪯}_{mc}X\hfill & & \mathrm{s}.\mathrm{t}.Q=f_1\left(X\right)\hfill & \\ & Q{⪯}_{mc}Y\hfill & & Q=f_2\left(Y\right)\hfill & \end{array}$$

where we have used the fact that $I(Q;T)\le min\left\{H\right(Q),H(T\left)\right\}$ to conclude that $I_{\cap }^{mc}(X,Y)\le C(X\wedge Y)$. Hence $I_{\cap }^{ln}(X,Y)=I_{\cap }^{ds}(X,Y)=I_{\cap }^{mc}(X,Y)=C(X\wedge Y)$ if T is a copy of the input.    □

Bertschinger et al. [14] suggested what later became known as the (*) assumption, which states that in the bivariate source case any sensible measure of unique information should only depend on $K^{\left(1\right)},K^{\left(2\right)}$, and $p\left(t\right)$. It is not clear that this assumption should hold for every PID. It is trivial to see that all the introduced II measures satisfy the (*) assumption.

We conclude with several applications of the proposed measures to famous (bivariate) PID problems; the results are shown in

Table 2. Application of the proposed measures to famous PID problems.

Target	${\mathit{I}}_{\cap }^{◃}$	${\mathit{I}}_{\cap }^{\mathit{d}}$	${\mathit{I}}_{\cap }^{\mathit{ln}}$	${\mathit{I}}_{\cap }^{\mathit{ds}}$	${\mathit{I}}_{\cap }^{\mathit{mc}}$	${\mathit{I}}_{\cap }^{\mathit{MMI}}$
$T=Y_1\mathrm{AND}{Y}_2$	0	0.311	0.311	0.311	0.311	0.311
$T=Y_1+Y_2$	0	0.5	0.5	0.5	0.5	0.5
$T=Y_1$	0	0	0	0	0	0
$T=(Y_1,Y_2)$	0	0	0	0	0	1

. Due to the channel design in these problems, computation of the proposed measures is fairly trivial. We assume that the input variables are binary (taking values in $\{0,1\}$), independent, and equiprobable.

We note that in these fairly simple toy distributions all of the introduced measures yield the same value. This is not surprising when the distribution $p(t,y_1,y_2)$ yields $K^{\left(1\right)}=K^{\left(2\right)}$, which implies that $I(T;Y_1)=I(T;Y_2)=I_{\cap }^k(Y_1,Y_2)$, where k refers to any of the introduced preorders, as is the case in the $T=Y_1\mathrm{AND}{Y}_2$ and $T=Y_1+Y_2$ examples. Less trivial examples lead to different values over the introduced measures. We present distributions showing that our three introduced measures lead to novel information decompositions by comparing them to the following existing measures: $I_{\cap }^{◃}$ from Griffith et al. [32], $I_{\cap }^{\mathrm{MMI}}$ from Barrett [33], $I_{\cap }^{\mathrm{WB}}$ from Williams and Beer [1], $I_{\cap }^{\mathrm{GH}}$ from Griffith and Ho [38], $I_{\cap }^{\mathrm{Ince}}$ from Ince [20], $I_{\cap }^{\mathrm{FL}}$ from Finn and Lizier [39], $I_{\cap }^{\mathrm{BROJA}}$ from Bertschinger et al. [14], $I_{\cap }^{\mathrm{Harder}}$ from Harder et al. [13], and $I_{\cap }^{\mathrm{dep}}$ from [16]. We used the dit package [40] to compute them, along with the code provided in [21]. Consider counterexample 1 from [22] with $p=0.25,ϵ=0.2,\delta =0.1$, provided by

$$K^{\left(1\right)}=\left[\begin{array}{cc}0.25& 0.75\\ 0.35& 0.65\end{array}\right],K^{\left(2\right)}=\left[\begin{array}{cc}0.675& 0.325\\ 0.745& 0.255\end{array}\right].$$

These channels satisfy $K^{\left(2\right)}{⪯}_{ln}{K}^{\left(1\right)}$ and $K_d^{\left(2\right)}{K}^{\left(1\right)}$ from Korner and Marton [22]. This is an example that satisfies $I_{\cap }^{ln}(Y_1,Y_2)=I(T;Y_2)$ for a given distribution $p\left(t\right)$. It is noteworthy to see that even though there is no degradation order between the two channels, we nonetheless have $I_{\cap }^d(Y_1,Y_2)>0$, as there is some non-trivial channel $K^Q$ that satisfies $K^Q{⪯}_d{K}^{\left(1\right)}$ and $K^Q{⪯}_d{K}^{\left(2\right)}$. In

Table 3. Different decompositions of $p(t,y_1,y_2)$.

${\mathit{I}}_{\cap }^{◃}$	${\mathit{I}}_{\cap }^{\mathit{d}}$	${\mathit{I}}_{\cap }^{\mathit{ln}}$	${\mathit{I}}_{\cap }^{\mathit{ds}}$	${\mathit{I}}_{\cap }^{\mathit{mc}}$	${\mathit{I}}_{\cap }^{\mathbf{MMI}}$	${\mathit{I}}_{\cap }^{\mathbf{WB}}$	${\mathit{I}}_{\cap }^{\mathbf{GH}}$	${\mathit{I}}_{\cap }^{\mathbf{Ince}}$	${\mathit{I}}_{\cap }^{\mathbf{FL}}$	${\mathit{I}}_{\cap }^{\mathbf{BROJA}}$	${\mathit{I}}_{\cap }^{\mathbf{Harder}}$	${\mathit{I}}_{\cap }^{\mathbf{dep}}$
0	0.002	0.004	*	0.004	0.004	0.004	0.002	0.003	0.047	0.003	0.004	0

, we present various PIDs under different measures after choosing $p\left(t\right)=[0.4,0.6]$ (which yields $I(T;Y_2)\approx 0.004$) and assuming $p(t,y_1,y_2)=p\left(t\right)p\left(y_1\right|t)p\left(y_2\right|t)$.

We write $I_{\cap }^{ds}=$ * here, as we do not yet have a way to find the ‘largest’ Q such that $Q{⪯}_{ds}{K}^{\left(1\right)}$ and $Q{⪯}_{ds}{K}^{\left(2\right)}$ (see counterexample 2 from [22] for an example of channels $K^{\left(1\right)},K^{\left(2\right)}$ that satisfy $K^{\left(2\right)}{⪯}_{mc}{K}^{\left(1\right)}$ while $K^{\left(2\right)}{⋠}_{ln}{K}^{\left(1\right)}$, leading to different values of the proposed II measures). An example of $K^{\left(3\right)},K^{\left(4\right)}$ that satisfy $K^{\left(4\right)}{⪯}_{ds}{K}^{\left(3\right)}$ while $K^{\left(4\right)}{⋠}_d{K}^{\left(3\right)}$ is presented by (Américo et al. [23], page 10), provided by

$$K^{\left(3\right)}=\left[\begin{array}{cc}1& 0\\ 0& 1\\ 0.5& 0.5\end{array}\right],K^{\left(4\right)}=\left[\begin{array}{cc}1& 0\\ 1& 0\\ 0.5& 0.5\end{array}\right].$$

There is no stochastic matrix $K^U$ such that $K^{\left(4\right)}=K^{\left(3\right)}{K}^U$ while $K^{\left(4\right)}{⪯}_{ds}{K}^{\left(3\right)}$, as $K^{\left(4\right)}={\diamond }_{1,2}{K}^{\left(3\right)}$. Using (10), it is possible to check whether there is any less noisy relation between the two channels. (Compute (10) with $V=K^{\left(4\right)},W=K^{\left(3\right)}$, $p\left(t\right)=[0,0,1]$, and $q\left(t\right)=[0.1,0.1,0.8]$ to conclude that $K^{\left(4\right)}{⋠}_{ln}{K}^{\left(3\right)}$, then switch the roles of V and W and set $p\left(t\right)=[0,1,0]$ and $q\left(t\right)=[0.1,0,0.9]$ to conclude that $K^{\left(3\right)}{⋠}_{ln}{K}^{\left(4\right)}$). We present the decomposition of $p(t,y_3,y_4)=p\left(t\right)p\left(y_3\right|t)p\left(y_4\right|t)$ for the choice of $p\left(t\right)=[0.3,0.3,0.4]$ (which yields $I(T;Y_4)\approx 0.322$) in

Table 4. Different decompositions of $p(t,y_3,y_4)$.

${\mathit{I}}_{\cap }^{◃}$	${\mathit{I}}_{\cap }^{\mathit{d}}$	${\mathit{I}}_{\cap }^{\mathit{ln}}$	${\mathit{I}}_{\cap }^{\mathit{ds}}$	${\mathit{I}}_{\cap }^{\mathit{mc}}$	${\mathit{I}}_{\cap }^{\mathbf{MMI}}$	${\mathit{I}}_{\cap }^{\mathbf{WB}}$	${\mathit{I}}_{\cap }^{\mathbf{GH}}$	${\mathit{I}}_{\cap }^{\mathbf{Ince}}$	${\mathit{I}}_{\cap }^{\mathbf{FL}}$	${\mathit{I}}_{\cap }^{\mathbf{BROJA}}$	${\mathit{I}}_{\cap }^{\mathbf{Harder}}$	${\mathit{I}}_{\cap }^{\mathbf{dep}}$
0	0	$0^{*}$	0.322	0.322	0.322	0.193	0	0	0.058	0	0	0

.

We write $I_{\cap }^{ln}=0^{*}$ because we conjecture, after numerical experiments based on (10), that the ‘largest’ channel that is less noisy than both $K^{\left(3\right)}$ and $K^{\left(4\right)}$ is a channel that satisfies $I(Q;T)=0$. (We tested all $3\times 3$ row-stochastic matrices with entries that take values in $\{0,0.1,0.2,\dots ,0.9,1\}$ with all distributions $p\left(t\right)$ and $q\left(t\right)$ having entries that take values in the same set.)

7. Conclusions and Future Work

In this paper, we have introduced three new measures of intersection information for the partial information decomposition (PID) framework based on preorders between channels implied by the degradation/Blackwell order. The new measures were obtained from the orders by following the approach recently proposed by Kolchinsky [21]. The main contributions and conclusions of this paper can be summarized as follows:

• We show that a measure of intersection information that satisfies the axioms by Kolchinsky [21] and is based on a preorder, satisfies the Williams–Beer axioms as well [1].
• As a corollary of the previous result, the proposed measures satisfy the Williams–Beer axioms, and can be extended beyond two sources.
• We demonstrate that if there is a degradation ordering between the sources, then the measures coincide in their decomposition. Conversely, if there is no degradation ordering (i.e., only a weaker ordering) between the source variables, the proposed measures lead to novel finer information decompositions that capture different finer information.
• We show that while the proposed measures do not satisfy the identity property (IP) [13], they do satisfy the independent identity property (IIP) [20].
• We formulate the optimization problems that yield the proposed measures, and derive bounds by relating them to existing measures.

Finally, we believe that this paper opens several avenues for future research; thus, we point to several directions that could be pursued in upcoming work:

• Investigating conditions to verify whether two channels $K^{\left(1\right)}$ and $K^{\left(2\right)}$ satisfy $K^{\left(1\right)}{⪯}_{ds}{K}^{\left(2\right)}$.
• Kolchinsky [21] showed that when computing $I_{\cap }^d(Y_1,\dots ,Y_n)$, it is sufficient to consider variables Q with a support size of at most ${\sum }_i\left|S_{Y_i}\right|-n+1$, which is a consequence of the admissible region of $I_{\cap }^d(Y_1,\dots ,Y_n)$ being a polytope. The same is not the case with the less noisy or the more capable measures; hence, it is not clear whether it is sufficient to consider Q with the same support size, which could represent a direction for future research.
• Studying the conditions under which different intersection information measures are continuous.
• Implementing the introduced measures by addressing their corresponding optimization problems.
• Considering the usual PID framework, except that instead of decomposing $I(T;Y)=H\left(Y\right)-H\left(Y\right|T)$, where H denotes the Shannon entropy, other mutual informations induced by different entropy measures could be considered, such as the guessing entropy [41] or the Tsallis entropy [42] (see the work of Américo et al. [23] for other core-concave entropies that may be decomposed under the introduced preorders, as these entropies are consistent with the introduced orders).
• Another line for future work might be to define measures of union information using the introduced preorders, as suggested by Kolchinsky [21], and to study their properties.
• As a more long-term research direction, it would be interesting to study how the approach taken in this paper can be extended to quantum information; the fact that partial quantum information can be negative might open up new possibilities or create novel difficulties [43].