Pointwise Partial Information Decomposition Using the Specificity and Ambiguity Lattices

Abstract

What are the distinct ways in which a set of predictor variables can provide information about a target variable? When does a variable provide unique information, when do variables share redundant information, and when do variables combine synergistically to provide complementary information? The redundancy lattice from the partial information decomposition of Williams and Beer provided a promising glimpse at the answer to these questions. However, this structure was constructed using a much criticised measure of redundant information, and despite sustained research, no completely satisfactory replacement measure has been proposed. In this paper, we take a different approach, applying the axiomatic derivation of the redundancy lattice to a single realisation from a set of discrete variables. To overcome the difficulty associated with signed pointwise mutual information, we apply this decomposition separately to the unsigned entropic components of pointwise mutual information which we refer to as the specificity and ambiguity. This yields a separate redundancy lattice for each component. Then based upon an operational interpretation of redundancy, we define measures of redundant specificity and ambiguity enabling us to evaluate the partial information atoms in each lattice. These atoms can be recombined to yield the sought-after multivariate information decomposition. We apply this framework to canonical examples from the literature and discuss the results and the various properties of the decomposition. In particular, the pointwise decomposition using specificity and ambiguity satisfies a chain rule over target variables, which provides new insights into the so-called two-bit-copy example.

1. Introduction

The aim of information decomposition is to divide the total amount of information provided by a set of predictor variables, about a target variable, into atoms of partial information contributed either individually or jointly by the various subsets of the predictors. Suppose that we are trying to predict a target variable T, with discrete state space $\mathcal{T}$, from a pair of predictor variables $S_1$ and $S_2$, with discrete state spaces ${\mathcal{S}}_1$ and ${\mathcal{S}}_2$. The mutual information $I(S_1;T)$ quantifies the information $S_1$ individually provides about T. Similarly, the mutual information $I(S_2;T)$ quantifies the information $S_2$ individually provides about T. Now consider the joint variable $S_{1,2}$ with the state space ${\mathcal{S}}_1\times {\mathcal{S}}_2$. The (joint) mutual information $I(S_{1,2};T)$ quantifies the total information $S_1$ and $S_2$ together provide about T. Although Shannon’s information theory provides the prior three measures of information, there are four possible ways $S_1$ and $S_2$ could contribute information about T: the predictor $S_1$ could uniquely provide information about T; or the predictor $S_2$ could uniquely provide information about T; both $S_1$ and $S_2$ could both individually, yet redundantly, provide the same information about T; or the predictors $S_1$ and $S_2$ could synergistically provide information about T which is not available in either predictor individually. Thus we have the following underdetermined set of equations,

$$\begin{array}{ccc}\hfill I(S_{1,2};T)& \hfill =R(S_1,S_2\to T)& +U(S_1\backslash S_2\to T)+U(S_2\backslash S_1\to T)+C(S_1,S_2\to T),\hfill \\ \hfill I(S_1;T)& \hfill =R(S_1,S_2\to T)& +U(S_1\backslash S_2\to T),\hfill \\ \hfill I(S_2;T)& \hfill =R(S_1,S_2\to T)& +U(S_2\backslash S_1\to T),\hfill \end{array}$$

(1)

where $U(S_1\backslash S_2\to T)$ and $U(S_2\backslash S_1\to T)$ are the unique information provided by $S_1$ and $S_2$ respectively, $R(S_1,S_2\to T)$ is the redundant information, and $C(S_1,S_2\to T)$ is the synergistic or complementary information. (The directed notation is utilise here to emphasis the privileged role of the variable T.) Together, the equations in (1) form the bivariate information decomposition. The problem is to define one of the unique, redundant or complementary information—something not provided by Shannon’s information theory—in order to uniquely evaluate the decomposition.

Now suppose that we are trying to predict a target variable T from a set of n finite state predictor variables $\mathit{S}=\{S_1,\dots ,S_n\}$. In this general case, the aim of information decomposition is to divide the total amount of information $I(S_1,\dots ,S_n;T)$ into atoms of partial information contributed either individually or jointly by the various subsets of $\mathit{S}$. But what are the distinct ways in which these subsets of predictors might contribute information about the target? Multivariate information decomposition is more involved than the bivariate information decomposition because it is not immediately obvious how many atoms of information one needs to consider, nor is it clear how these atoms should relate to each other. Thus the general problem of information decomposition is to provide both a structure for multivariate information which is consistent with the bivariate decomposition, and a way to uniquely evaluate the atoms in this general structure.

In the remainder of Section 1, we will introduce an intriguing framework called partial information decomposition (PID), which aims to address the general problem of information decomposition, and highlight some of the criticisms and weaknesses of this framework. In Section 2, we will consider the underappreciated pointwise nature of information and discuss the relevance of this to the problem of information decomposition. We will then propose a modified pointwise partial information decomposition (PPID), but then quickly repudiate this approach due to complications associated with decomposing the signed pointwise mutual information. In Section 3, we will discuss circumventing this issue by examining information on a more fundamental level, in terms of the unsigned entropic components of pointwise mutual information which we refer to as the specificity and the ambiguity. Then in Section 4—the main section of this paper—we will introduce the PPID using the specificity and ambiguity lattices and the measures of redundancy in Definitions 1 and 2. In Section 5, we will apply this framework to a number of canonical examples from the PID literature, discuss some of the key properties of the decomposition, and compare these to existing approaches to information decomposition. Section 6 will conclude the main body of the paper. Appendix A contains discussions regarding the so-called two-bit-copy problem in terms of Kelly gambling, Appendix B contains many of the technical details and proofs, while Appendix B contains some more examples.

1.1. Notation

The following notational conventions are observed throughout this article:

T, $\mathcal{T}$, t, $t^{\mathsf{c}}$

denote the target variable, event space, event and complementary event respectively;

S, $\mathcal{S}$, s, $s^{\mathsf{c}}$

denote the predictor variable, event space, event and complementary event respectively;

$\mathit{S}$, $\mathit{s}$

represent the set of n predictor variables $\{S_1,\dots ,S_n\}$ and events $\{s_1,\dots ,s_n\}$ respectively;

${\mathcal{T}}^t$, ${\mathcal{S}}^s$

denote the two-event partition of the event space, i.e., ${\mathcal{T}}^t=\{t,t^{\mathsf{c}}\}$ and ${\mathcal{S}}^s=\{s,s^{\mathsf{c}}\}$;

$H(T)$, $I(S;T)$

uppercase function names be used for average information-theoretic measures;

$h(t)$, $i(s,t)$

lowercase function names be used for pointwise information-theoretic measures.

When required, the following index conventions are observed:

$s^1$, $s^2$, $t^1$, $t^2$

superscripts distinguish between different different events in a variable;

$S_1$, $S_2$, $T_1$, $T_2$

subscripts distinguish between different variables;

$S_{1,2}$, $s_{1,2}$

multiple superscripts represent joint variables and joint events.

Finally, to be discussed in more detail when appropriate, consider the following:

${\mathit{A}}_1,\dots ,{\mathit{A}}_k$

sources are sets of predictor variables, i.e., ${\mathit{A}}_i\in {\mathcal{P}}_1(\mathit{S})$ where ${\mathcal{P}}_1$ is the power set without ∅;

${\mathit{a}}_1,\dots ,{\mathit{a}}_k$

source events are sets of predictor events, i.e., ${\mathit{a}}_i\in {\mathcal{P}}_1(\mathit{s})$.

1.2. Partial Information Decomposition

The partial information decomposition (PID) of Williams and Beer [1,2] was introduced to address the problem of multivariate information decomposition. The approach taken is appealing as rather than speculating about the structure of multivariate information, Williams and Beer took a more principled, axiomatic approach. They start by considering potentially overlapping subsets of $\mathit{S}$ called sources, denoted ${\mathit{A}}_1,\dots ,{\mathit{A}}_k$. To examine the various ways these sources might contain the same information, they introduce three axioms which “any reasonable measure for redundant information [$I_{\cap }$] should fulfil” ([3], p. 3502). Note that the axioms appear explicitly in [2] but are discussed in [1] as mere properties; a published version of the axioms can be found in [4].

W&B Axiom 1 (Commutativity).

Redundant information is invariant under any permutation σ of sources,

$$I_{\cap }\left({\mathit{A}}_1,\dots ,{\mathit{A}}_k\to T\right)=I_{\cap }\left(\sigma ({\mathit{A}}_1),\cdots ,\sigma ({\mathit{A}}_k)\to T\right).$$

W&B Axiom 2 (Monotonicity).

Redundant information decreases monotonically as more sources are included,

$$I_{\cap }\left({\mathit{A}}_1,\dots ,{\mathit{A}}_{k-1}\to T\right)\le I_{\cap }\left({\mathit{A}}_1,\dots ,{\mathit{A}}_k\to T\right)$$

with equality if ${\mathit{A}}_k\supseteq {\mathit{A}}_i$ for any ${\mathit{A}}_i\in \{{\mathit{A}}_1,\dots ,{\mathit{A}}_{k-1}\}$.

W&B Axiom 3 (Self-redundancy).

Redundant information for a single source ${\mathit{A}}_i$ equals the mutual information,

$$I_{\cap }\left({\mathit{A}}_i\to T\right)=I\left({\mathit{A}}_i;T\right).$$

These axioms are based upon the intuition that redundancy should be analogous to the set- theoretic notion of intersection (which is commutative, monotonically decreasing and idempotent). Crucially, Axiom 3 ties this notion of redundancy to Shannon’s information theory. In addition to these three axioms, there is an (implicit) axiom assumed here known as local positivity [5], which is the requirement that all atoms be non-negative. Williams and Beer [1,2] then show how these axioms reduce the number of sources to the collection of sources such that no source is a superset of any other. These remaining sources are called partial information atoms (PI atoms). Each PI atom corresponds to a distinct way the set of predictors $\mathit{S}$ can contribute information about the target T. Furthermore, Williams and Beer show that these PI atoms are partially ordered and hence form a lattice which they call the redundancy lattice. For the bivariate case, the redundancy lattice recovers the decomposition (1), while in the multivariate case it provides a meaningful structure for decomposition of the total information provided by an arbitrary number of predictor variables.

While the redundancy lattice of PID provides a structure for multivariate information decomposition, it does not uniquely determine the value of the PI atoms in the lattice. To do so requires a definition of a measure of redundant information which satisfies the above axioms. Hence, in order to complete the PID framework, Williams and Beer simultaneously introduced a measure of redundant information called $I_{\min }$ which quantifies redundancy as the minimum information that any source provides about a target event t, averaged over all possible events from T. However, not long after its introduction $I_{\min }$ was heavily criticised. Firstly, $I_{\min }$ does not distinguish between “whether different random variables carry the same information or just the same amount of information” ([5], p. 269; see also [6,7]). Secondly, $I_{\min }$ does not possess the target chain rule introduced by Bertschinger et al. [5] (under the name left chain rule). This latter point is problematic as the target chain rule is a natural generalisation of the chain rule of mutual information—i.e., one of the fundamental, and indeed characterising, properties of information in Shannon’s theory [8,9].

These issues with $I_{\min }$ prompted much research attempting to find a suitable replacement measure compatible with the PID framework. Using the methods of information geometry, Harder et al. [6] focused on a definition of redundant information called $I_{\mathrm{red}}$ (see also [10]). Bertschinger et al. [11] defined a measure of unique information $\widetilde{UI}$ based upon the notion that if one variable contains unique information then there must be some way to exploit that information in a decision problem. Griffith and Koch [12] used an entirely different motivation to define a measure of synergistic information ${\mathcal{S}}_{\mathrm{VK}}$ whose decomposition transpired to be equivalent to that of $\widetilde{UI}$ [11]. Despite this effort, none of these proposed measures are entirely satisfactory. Firstly, just as for $I_{\min }$, none of these proposed measures possess the target chain rule. Secondly, these measures are not compatible with the PID framework in general, but rather are only compatible with PID for the special case of bivariate predictors, i.e., the decomposition (1). This is because they all simultaneously satisfy the Williams and Beers axioms, local positivity, and the identity property introduced by Harder et al. [6]. In particular, Rauh et al. [13] proved that no measure satisfying the identity property and the Williams and Beer Axioms 1–3 can yield a non-negative information decomposition beyond the bivariate case of two predictor variables. In addition to these proposed replacements for $I_{\min }$, there is also a substantial body of literature discussing either PID, similar attempts to decompose multivariate information, or the problem of information decomposition in general [3,4,5,7,10,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28]. Furthermore, the current proposals have been applied to various problems in neuroscience [29,30,31,32,33,34]. Nevertheless (to date), there is no generally accepted measure of redundant information that is entirely compatible with PID framework, nor has any other well-accepted multivariate information decomposition emerged.

To summarise the problem, we are seeking a meaningful decomposition of the information provided an arbitrarily large set of predictor variables about a target variable, into atoms of partial information contributed either individually or jointly by the various subsets of the predictors. Crucially, the redundant information must capture when two predictor variables are carrying the same information about the target, not merely the same amount of information. Finally, any proposed measure of redundant information should satisfy the target chain rule so that net redundant information can be consistently computed for consistently for multiple target events.

2. Pointwise Information Theory

Both the entropy and mutual information can be derived from first principles as fundamentally pointwise quantities which measure the information content of individual events rather than entire variables. The pointwise entropy $h(t)=-logp(t)$ quantifies the information content of a single event t, while the pointwise mutual information

$$i(s;t)=log\frac{p(t|s)}{p(t)}=log\frac{p(s,t)}{p(s)p(t)}=log\frac{p(s|t)}{p(s)},$$

(2)

quantifies the information provided by s about t, or vice versa. To our knowledge, these quantities were first considered by Woodward and Davies [35,36] who noted that the average form of Shannon’s entropy “tempts one to enquire into other simpler methods of derivation [of the per state entropy]” ([35], p. 51). Indeed, they went on to show that the pointwise entropy and pointwise mutual information can be derived from two axioms concerning the addition of the information provided by the occurrence of individual events [36]. Fano [9] further formalised this pointwise approach by deriving both quantites from four postulates which “should be satisfied by a useful measure of information” ([9], p. 31). Taking the expectation of these pointwise quantities over all events recovers the average entropy $H(T)=〈h(t)〉$ and average mutual information $I(S;T)=〈i(s;t)〉$ first derived by Shannon [8]. Although both approaches arrive at the same average quantities, Shannon’s treatment obfuscates the pointwise nature of the fundamental quantities. In contrast, the approach of Woodward, Davis and Fano makes this pointwise nature manifestly obvious.

It is important to note that, in contrast to the average mutual information, the pointwise mutual information is not non-negative. Positive pointwise information corresponds to the predictor event s raising the probability $p(t|s)$ relative to the prior probability $p(t)$. Hence when the event t occurs it can be said that the event s was informative about the event t. Conversely, negative pointwise information corresponds to the event s lowering the posterior probability $p(t|s)$ relative to the prior probability $p(t)$. Hence when the event t occurs we can say that the event s was misinformative about the event t. (Not to be confused with disinformation, i.e., intentionally misleading information.) Although a source event s may be misinformative about a particular target event t, a source event s is never misinformative about the target variable T since the pointwise mutual information averaged over all target realisations is non-negative [9]. The information provided by s is helpful for predicting T on average; however, in certain instances this (typically helpful) information is misleading in that it lowers $p(t|s)$ relative to $p(t)$—typically helpful information which subsequently turns out to be misleading is misinformation.

Finally, before continuing, there are two points to be made about the terminology used to describe pointwise information. Firstly, in certain literature (typically in the context of time-series analysis), the word local is used instead of pointwise, e.g., [4,18]. Secondly, in contemporary information theory, the word average is generally omitted while the pointwise quantities are explicitly prefixed; however, this was not always the accepted convention. Woodward [35] and Fano [9] both referred to pointwise mutual information as the mutual information and then explicitly prefixed the average mutual information. To avoid confusion, we will always prefix both pointwise and average quantities.

2.1. Pointwise Information Decomposition

Now that we are familiar with pointwise nature of information, suppose that we have a discrete realisation from the joint event space $\mathcal{T}\times {\mathcal{S}}_1\times {\mathcal{S}}_2$ consisting of the target event t and predictor events $s_1$ and $s_2$. The pointwise mutual information $i(s_1;t)$ quantifies the information provided individually by $s_1$ about t, while the pointwise mutual information $i(s_2;t)$ quantifies the information provided individually by $s_2$ about t. The pointwise joint mutual information $i(s_{1,2};t)$ quantifies the total information provided jointly by $s_1$ and $s_2$ about t. In correspondence with the (average) bivariate decomposition (1), consider the pointwise bivariate decomposition, first suggested by Lizier et al. [4],

$$\begin{array}{ccc}\hfill i(s_{1,2};t)& \hfill =r(s_1,s_2\to t)& +u(s_1\backslash s_2\to t)+u(s_2\backslash s_1\to t)+c(s_1,s_2\to t),\hfill \\ \hfill i(s_1;t)& \hfill =r(s_1,s_2\to t)& +u(s_1\backslash s_2\to t),\hfill \\ \hfill i(s_2;t)& \hfill =r(s_1,s_2\to t)& +u(s_2\backslash s_1\to t).\hfill \end{array}$$

(3)

Note that the lower case quantities denote the pointwise equivalent of the corresponding upper case quantities in (1). This decomposition could be considered for every discrete realisation on the support of the joint distribution $P(S_1,S_2,T)$. Hence, consider taking the expectation of these pointwise atoms over all discrete realisations,

$$\begin{array}{cccc}U(S_1\backslash S_2\to T)& =〈u(s_1\backslash s_2\to t)〉,& R(S_1,S_2\to T)& =〈r(s_1,s_2\to t)〉,\\ U(S_2\backslash S_1\to T)& =〈u(s_2\backslash s_1\to t)〉,& C(S_1,S_2\to T)& =〈c(s_1,s_2\to t)〉.\end{array}$$

(4)

Since the expectation is a linear operation, this will recover the (average) bivariate decomposition (1). Equation (3) for every discrete realisation, together with (1) and (4) form the bivariate pointwise information decomposition. Just as in (1), these equations are underdetermined requiring a separate definition of either the pointwise unique, redundant or complementary information for uniqueness. (Defining an average atom is sufficient for a unique bivariate decomposition (1), but still leaves the pointwise decomposition (3) within each realisation underdetermined).

2.2. Pointwise Unique

Now consider applying this pointwise information decomposition to the probability distribution Pointwise Unique (PwUnq) in

Table 1. Example PwUnq. For each realisation, the pointwise mutual information provided by each individual and joint predictor events, about the target event has been evaluated. Note that one predictor event always provides full information about the target while the other provides zero information. Based on the this, it is assumed that there must be zero redundant information. The pointwise partial information (PPI) atoms are then calculated via (3).

p	${\mathit{s}}_1$	${\mathit{s}}_2$	t	$\mathit{i}({\mathit{s}}_1;\mathit{t})$	$\mathit{i}({\mathit{s}}_2;\mathit{t})$	$\mathit{i}({\mathit{s}}_{1,2};\mathit{t})$	$\mathit{u}({\mathit{s}}_1\backslash {\mathit{s}}_2\to \mathit{t})$	$\mathit{u}({\mathit{s}}_2\backslash {\mathit{s}}_1\to \mathit{t})$	$\mathit{r}({\mathit{s}}_1,{\mathit{s}}_2\to \mathit{t})$	$\mathit{c}({\mathit{s}}_1,{\mathit{s}}_2\to \mathit{t})$
$\raisebox{1ex}{$1$}\!\left/ \!\raisebox{-1ex}{$4$}\right.$	0	1	1	0	1	1	0	1	0	0
$\raisebox{1ex}{$1$}\!\left/ \!\raisebox{-1ex}{$4$}\right.$	1	0	1	1	0	1	1	0	0	0
$\raisebox{1ex}{$1$}\!\left/ \!\raisebox{-1ex}{$4$}\right.$	0	2	2	0	1	1	0	1	0	0
$\raisebox{1ex}{$1$}\!\left/ \!\raisebox{-1ex}{$4$}\right.$	2	0	2	1	0	1	1	0	0	0
Expected values				$\raisebox{1ex}{$1$}\!\left/ \!\raisebox{-1ex}{$2$}\right.$	$\raisebox{1ex}{$1$}\!\left/ \!\raisebox{-1ex}{$2$}\right.$	1	$\raisebox{1ex}{$1$}\!\left/ \!\raisebox{-1ex}{$2$}\right.$	$\raisebox{1ex}{$1$}\!\left/ \!\raisebox{-1ex}{$2$}\right.$	0	0

. In PwUnq, observing 0 in either of $S_1$ or $S_2$ provides zero information about the target T, while complete information about the outcome of T is obtained by observing 1 or a 2 in either predictor. The probability distribution is structured such that in each of the four realisations, one predictor provides complete information while the other predictor provides zero information—the two predictors never provide the same information about the target which is justified by noting that one of the two predictors always provides zero pointwise information.

Given that redundancy is supposed to capture the same information, it seems reasonable to assume there must be zero pointwise redundant information for each realisation. This assumption is made without any measure of pointwise redundant information; however, no other possibility seems justifiable. This assertion is used to determine the pointwise redundant information terms in Table 1. Then using the pointwise information decomposition (3), we can then evaluate the other pointwise atoms of information in Table 1. Finally using (4), we get that there is zero (average) redundant information, and $\raisebox{1ex}{$1$}\!\left/ \!\raisebox{-1ex}{$2$}\right.$ bit of (average) unique information from each predictor. From the pointwise perspective, the only reasonable conclusion seems to be that the predictors in PwUnq must contain only unique information about the target.

However, in contrast to the above, $I_{\min }$, $I_{\mathrm{red}}$, $\widetilde{UI}$, and ${\mathcal{S}}_{\mathrm{VK}}$ all say that the predictors in PwUnq contain no unique information, rather only $\raisebox{1ex}{$1$}\!\left/ \!\raisebox{-1ex}{$2$}\right.$ bit of redundant information plus $\raisebox{1ex}{$1$}\!\left/ \!\raisebox{-1ex}{$2$}\right.$ bit of complementary information. This problem, which will be referred to as the pointwise unique problem, is a consequence of the fact that these measures all satisfy Assumption ($\ast$) of Bertschinger et al. [11], which (in effect) states that the unique and redundant information should only depend on the marginal distributions $P(S_1,T)$ and $P(S_2,T)$. In particular, any measure which satisfies Assumption ($\ast$) will yield zero unique information when $P(S_1,T)$ is isomorphic to $P(S_2,T)$, as is the case for PwUnq. (Here, isomorphic should be taken to mean isomorphic probability spaces, e.g., [37], p. 27 or [38], p. 4.) It arises because Assumption ($\ast$) (and indeed the operational interpretation the led to its introduction) does not respect the pointwise nature of information. This operational view does not take into account the fact that individual events $s_1$ and $s_2$ may provide different information about the event t, even if the probability distributions $P(S_1,T)$ and $P(S_2,T)$ are the same. Hence, we contend that for any measure to capture the same information (not merely the same amount), it must respect the pointwise nature of information.

2.3. Pointwise Partial Information Decomposition

With the pointwise unique problem in mind, consider constructing an information decomposition with the pointwise nature of information as an inherent property. Let ${\mathit{a}}_1,\dots ,{\mathit{a}}_k$ be potentially intersecting subsets of the predictor events $\mathit{s}=\{s_1,\dots ,s_n\}$, called source events. Now consider rewriting the Williams and Beer axioms in terms of a measure of pointwise redundant information $i_{\cap }$ where the aim is to deriving a pointwise partial information decomposition (PPID).

PPID Axiom 1 (Symmetry).

Pointwise redundant information is invariant under any permutation σ of source events,

$$i_{\cap }\left({\mathit{a}}_1,\dots ,{\mathit{a}}_k\to t\right)=i_{\cap }\left(\sigma ({\mathit{a}}_1),\cdots ,\sigma ({\mathit{a}}_k)\to T\right).$$

PPID Axiom 2 (Monotonicity).

Pointwise redundant information decreases monotonically as more source events are included,

$$i_{\cap }\left({\mathit{a}}_1,\dots ,{\mathit{a}}_{k-1}\to t\right)\le i_{\cap }\left({\mathit{a}}_1,\dots ,{\mathit{a}}_k\to t\right)$$

with equality if ${\mathit{a}}_k\supseteq {\mathit{a}}_i$ for any ${\mathit{a}}_i\in \{{\mathit{a}}_1,\dots ,{\mathit{a}}_{k-1}\}$.

PPID Axiom 3 (Self-redundancy).

Pointwise redundant information for a single source event ${\mathit{a}}_i$ equals the pointwise mutual information,

$$i_{\cap }\left({\mathit{a}}_i\to t\right)=i\left({\mathit{a}}_i;t\right).$$

It seems that the next step should be to define some measure of pointwise redundant information which is compatible with these PPID axioms; however, there is a problem—the pointwise mutual information is not non-negative. While this would not be an issue for the examples like PwUnq, where none of the source events provide negative pointwise information, it is an issue in general (e.g., see RdnErr in Section 5.4). The problem is that set-theoretic intuition behind Axiom 2 (monotonicity) makes little sense when considering signed measures like the pointwise mutual information.

Given the desire to address the pointwise unique problem, there is a need to overcome this issue. Ince [18] suggested that the set-theoretic intuition is only valid when all source events provide either positive or negative pointwise information. Ince contends that information and misinformation are “fundamentally different” ([18], p. 11) and that the set-theoretic intuition should be admitted in the difficult to interpret situations where both are present. We however, will take a different approach—one which aims to deal with these difficult to interpret situations whilst preserving the set-theoretic intuition that redundancy corresponds to overlapping information.

By way of a preview, we first consider precisely how an event $s_1$ provides information about an event t by the means of two distinct types of probability mass exclusion. We show how considering the process in this way naturally splits the pointwise mutual information into particular entropic components, and how one can consider redundancy on each of these components separately. Splitting the signed pointwise mutual information into these unsigned entropic components circumvents the above issue with Axiom 2 (monotonicity). Crucially, however, by deriving these entropic components from the probability mass exclusions, we retain the set-theoretic intuition of redundancy—redundant information will correspond to overlapping probability mass exclusions in the two-event partition ${\mathcal{T}}^t=\{t,t^{\mathsf{c}}\}$.

3. Probability Mass Exclusions and the Directed Components of Pointwise Mutual Information

By definition, the pointwise information provided by s about t is associated with a change from the prior $p(t)$ to the posterior $p(t|s)$. As we explored from first principles in Finn and Lizier [39], this change is a consequence of the exclusion of probability mass in the target distribution $P(T)$ induced by the occurrence of the event s and inferred via the joint distribution $P(S,T)$. To be specific, when the event s occurs, one knows that the complementary event $s^{\mathsf{c}}=\{\mathcal{S}\backslash s\}$ did not occur. Hence one can exclude the probability mass in the joint distribution $P(S,T)$ associated with the complementary event, i.e., exclude $P(s^{\mathsf{c}},T)$, leaving just the probability mass $P(s,T)$ remaining. The new target distribution $P(T|s)$ is evaluated by normalising this remaining probability mass. In [39] we introduced probability mass diagrams in order to visually explore the exclusion process. Figure 1 provides an example of such a diagram. Clearly, this process is merely a description of the definition of conditional probability. Nevertheless, we content that by viewing the change from the prior to the posterior in this way—by focusing explicitly on the exclusions rather than the resultant conditional probability—the vague intuition that redundancy corresponds to overlapping information becomes more apparent. This point will elaborated upon in Section 3.3. However, in order to do so, we need to first discuss the two distinct types of probability mass exclusion (which we do in Section 3.1) and then relate these to information-theoretic quantities (which we do in Section 3.2).

3.1. Two Distinct Types of Probability Mass Exclusions

In [39] we examined the two distinct types of probability mass exclusions. The difference between the two depends on where the exclusion occurs in the target distribution $P(T)$ and the particular target event t which occurred. Informative exclusions are those which are confined to the probability mass associated with the set of elementary events in the target distribution which did not occur, i.e., exclusions confined to the probability mass of the complementary event $p(t^{\mathsf{c}})$. They are called such because the pointwise mutual information $i(s;t)$ is a monotonically increasing function of the total size of these exclusions $p(t^{\mathsf{c}})$. By convention, informative exclusions are represented on the probability mass diagrams by horizontal or vertical lines. On the other hand, the misinformative exclusion is confined to the probability mass associated with the elementary event in the target distribution which did occur, i.e., an exclusion confined to $p(t)$. It is referred to as such because the pointwise mutual information $i(s;t)$ is a monotonically decreasing function of the size of this type of exclusion $p(t)$. By convention, misinformative exclusions are represented on the probability mass diagrams by diagonal lines.

Although an event s may exclusively induce either type of exclusion, in general both types of exclusion are present simultaneously. The distinction between the two types of exclusions leads naturally to the following question—can one decompose the pointwise mutual information $i(s;t)$ into a positive informational component associated with the informative exclusions, and a negative informational component associated with the misinformative exclusions? This question is considered in detail in Section 3.2. However, before moving on, there is a crucial observation to be made about the pointwise mutual information which will have important implications for the measure of redundant information to be introduced later.

Remark 1.

The pointwise mutual information $i(s;t)$ depends only on the size of informative and misinformative exclusions. In particular, it does not depend on the apportionment of the informative exclusions across the set of elementary events contained in the complementary event $t^{\mathsf{c}}$.

In other words, whether the event s turns out to be net informative or misinformative about the event t—whether $i(s;t)$ is positive or negative—depends on the size of the two types of exclusions; but, to be explicit, does not depend on the distribution of the informative exclusion across the set of target events which did not occur. This remark will be crucially important when it comes to providing the operational interpretation of redundant information in Section 3.3. (It is also further discussed in terms of Kelly gambling [40] in Appendix A).

3.2. The Directed Components of Pointwise Information: Specificity and Ambiguity

We return now to the idea that one might be able to decompose the pointwise mutual information into a positive and negative component associated with the informative amd misinformative exclusions respectively. In [39] we proposed four postulates for such a decomposition. Before stating the postulates, it is important to note that although there is a “surprising symmetry” ([41], p. 23) between the information provided by s about t and the information provided by t about s, there is nothing to suggest that the components of the decomposition should be symmetric—indeed the intuition behind the decomposition only makes sense when considering the information is considered in a directed sense. As such, directed notation will be used to explicitly denote the information provided by s about t.

Postulate 1 (Decomposition).

The pointwise information provided by s about t can be decomposed into two non-negative components, such that $i(s;t)=i_{+}(s\to t)-i_{-}(s\to t)$.

Postulate 2 (Monotonicity).

For all fixed $p(s,t)$ and $p(s^{\mathsf{c}},t)$, the function $i_{+}(s\to t)$ is a monotonically increasing, continuous function of $p(t^{\mathsf{c}},s^{\mathsf{c}})$. For all fixed $p(t^{\mathsf{c}},s)$ and $p(t^{\mathsf{c}},s^{\mathsf{c}})$, the function $i_{-}(s\to t)$ is a monotonically increasing continuous function of $p(s^{\mathsf{c}},t)$. For all fixed $p(s,t)$ and $p(t^{\mathsf{c}},s)$, the functions $i_{+}(s\to t)$ and $i_{-}(s\to t)$ are monotonically increasing and decreasing functions of $p(t^{\mathsf{c}},s^{\mathsf{c}})$, respectively.

Postulate 3 (Self-Information).

An event cannot misinform about itself, $i_{+}(s\to s)=i(s;s)=-logp(s)$.

Postulate 4 (Chain Rule).

The functions $i_{+}(s_{1,2}\to t)$ and $i_{-}(s_{1,2}\to t)$ satisfy a chain rule, i.e.,

$$\begin{array}{cc}\hfill i_{+}(s_{1,2}\to t)& =i_{+}(s_1\to t)+i_{+}(s_2\to t|s_1)\hfill \\ & =i_{+}(s_2\to t)+i_{+}(s_1\to t|s_2),\hfill \\ \hfill i_{-}(s_{1,2}\to t)& =i_{-}(s_1\to t)+i_{-}(s_2\to t|s_1)\hfill \\ & =i_{-}(s_2\to t)+i_{-}(s_1\to t|s_2)\hfill \end{array}$$

In Finn and Lizier [39], we proved that these postulates lead to the following forms which are unique up to the choice of the base of the logarithm in the mutual information in Postulates 1 and 3,

$$\begin{array}{ccc}{i}^{+}(s_1\to t)\hfill & =h(s_1)\hfill & =-logp(s_1),\hfill \end{array}$$

(5)

$$\begin{array}{ccc}{i}^{+}(s_1\to t|s_2)\hfill & =h(s_1|s_2)\hfill & =-logp(s_1|s_2),\hfill \end{array}$$

(6)

$$\begin{array}{ccc}{i}^{+}(s_{1,2}\to t)\hfill & =h(s_{1,2})\hfill & =-logp(s_{1,2}),\hfill \end{array}$$

(7)

$$\begin{array}{ccc}{i}^{-}(s_1\to t)\hfill & =h(s_1|t)\hfill & =-logp(s_1|t),\hfill \end{array}$$

(8)

$$\begin{array}{ccc}{i}^{-}(s_1\to t|s_2)\hfill & =h(s_1|t,s_2)\hfill & =-logp(s_1|t,s_2),\hfill \end{array}$$

(9)

$$\begin{array}{ccc}{i}^{-}(s_{1,2}\to t)\hfill & =h(s_{1,2}|t)\hfill & =-logp(s_{1,2}|t).\hfill \end{array}$$

(10)

That is, the Postulates 1–4 uniquely decompose the pointwise information provided by s about t into the following entropic components,

$$\begin{array}{cc}\hfill i(s;t)& =i^{+}(s\to t)-i^{-}(s\to t)\hfill \\ & =h(s)-h(s|t).\hfill \end{array}$$

(11)

Although the decomposition of mutual information into entropic components is well-known, it is non-trivial that Postulates 1 and 3, based on the size of the two distinct types of probability mass exclusions, lead to this particular form, but not $i(s;t)=h(t)-h(t|s)$ or $i(s;t)=h(s)+h(t)-h(s,t)$.

It is important to note that although the original motivation was to decompose the pointwise mutual information into separate components associated with informative and misinformative exclusion, the decomposition (11) does not quite possess this direct correspondence:

• The positive informational component $i^{+}(s\to t)$ does not depend on t but rather only on s. This can be interpreted as follows: the less likely s is to occur, the more specific it is when it does occur, the greater the total amount of probability mass excluded $p(s^{\mathsf{c}})$, and the greater the potential for s to inform about t (or indeed any other target realisation).
• The negative informational component $i^{-}(s\to t)$ depends on both s and t, and can be interpreted as follows: the less likely s is to coincide with the event t, the more uncertainty in s given t, the greater size of the misinformative probability mass exclusion $p(s^{\mathsf{c}},t)$, and therefore the greater the potential for s to misinform about t.

In other words, although the negative informational component $i^{-}(s\to t)$ does correspond directly to the size of the misinformative exclusion $p(s^{\mathsf{c}},t)$, the positive informational component $i^{+}(s\to t)$ does not correspond directly to the size of the informative exclusion $p(t^{\mathsf{c}},s^{\mathsf{c}})$. Rather, the positive informational component $i^{+}(s\to t)$ corresponds to the total size of the probability mass exclusions $p(s^{\mathsf{c}})$, which is the sum of the sum of the informative and misinformative exclusions. For the sake of brevity, the positive informational component $i^{+}(s\to t)$ will be referred to as the specificity, while the negative informational component $i^{-}(s\to t)$ will be referred to as the ambiguity. The term ambiguity is due to Shannon: “[equivocation] measures the average ambiguity of the received signal” ([42], p. 67). Specificity is an antonym of ambiguity and the usage here is inline with the definition since the more specific an event s, the more information it could provide about t after the ambiguity is taken into account.

3.3. Operational Interpretation of Redundant Information

Arguing about whether one piece of information differs from another piece of information is nonsensical without some kind of unambiguous definition of what it means for two pieces of information to be the same. As such, Bertschinger et al. [11] advocate the need to provide an operational interpretation of what it means for information to be unique or redundant. This section provides our operational definition of what it means for information to be the same. This definition provides a concrete interpretation of what it means for information to be redundant in terms of overlapping probability mass exclusions.

The operational interpretation of redundancy adopted here is based upon the following idea: since the pointwise information is ultimately derived from probability mass exclusions, the same information must induce the same exclusions. More formally, the information provided by a set of predictor events $s_1,\dots ,s_k$ about a target event t must be the same information if each source event induces the same exclusions with respect to the two-event partition ${\mathcal{T}}^t=\{t,t^{\mathsf{c}}\}$. While this statement makes the motivational intuition clear, it is not yet sufficient to serve as an operational interpretation of redundancy: there is no reference to the two distinct types of probability mass exclusions, the specific reference to the pointwise event space ${\mathcal{T}}^t$ has not been explained, and there is no reference to the fact the exclusions from each source may differ in size.

Informative exclusions are fundamentally different from misinformative exclusions and hence each type of exclusion should be compared separately: informative exclusions can overlap with informative exclusions, and misinformative exclusions can overlap with misinformative exclusions. In information-theoretic terms, this means comparing the specificity and the ambiguity of the sources separately—i.e., considering a measure of redundant specificity and a separate measure of redundant ambiguity. Crucially, these quantities (being pointwise entropies) are unsigned meaning that the difficulties associated with Axiom 2 (Monotonicity) and signed pointwise mutual information in Section 2.3 will not be an issue here.

The specific reference to the two-event partition ${\mathcal{T}}^t$ in the above statement is based upon Remark 1 and is crucially important. The pointwise mutual information does not depend on the apportionment of the informative exclusions across the set of events which did not occur, hence the pointwise redundant information should not depend on this apportionment either. In other words, it is immaterial if two predictor events $s_1$ and $s_2$ exclude different elementary events within the target complementary event $t^{\mathsf{c}}$ (assuming the probability mass excluded is equal) since with respect to the realised target event t the difference between the exclusions is only semantic. This has important implications for the comparison of exclusions from different predictor events. As the pointwise mutual information depends on, and only depends on, the size of the exclusions, then the only sensible comparison is a comparison of size. Hence, the common or overlapping exclusion must be the smallest exclusion. Thus, consider the following operational interpretation of redundancy:

Operational Interpretation (Redundant Specificity).

The redundant specificity between a set of predictor events $s_1,\dots ,s_n$ is the specificity associated with the source event which induces the smallest total exclusions.

Operational Interpretation (Redundant Ambiguity).

The redundant ambiguity between a set of predictor events $s_1,\dots ,s_n$ is the ambiguity associated with the source event which induces the smallest misinformative exclusion.

3.4. Motivational Example

To motivate the above operational interpretation, and in particular the need to treat the specificity separately to the ambiguity, consider Figure 2. In this pointwise example, two different predictor events provide the same amount of pointwise information since $P(T|s_1^1)=P(T|s_2^1)$, and yet the information provided by each event is in some way different since each excludes different sections of the target distribution $P(T)$. In particular, $s_1^1$ and $s_2^1$ both preclude the target event $t^2$, while $s_2^1$ additionally excludes probability mass associated with target events $t^1$ and $t^3$. From the perspective of the pointwise mutual information the events $s_1^1$ and $s_2^1$ seem to be providing the same information as

$$i(s_1^1\to t^1)=i(s_2^1\to t^1)=log\raisebox{1ex}{$4$}\!\left/ \!\raisebox{-1ex}{$3$}\right.\mathrm{bit}.$$

(12)

However, from the perspective of the specificity and the ambiguity it can be seen that information is being provided in different ways since

$$\begin{array}{cccc}{i}^{+}(s_1^1\to t^1)\hfill & =log\raisebox{1ex}{$4$}\!\left/ \!\raisebox{-1ex}{$3$}\right.\mathrm{bit},& i^{-}(s_1^1\to t^1)\hfill & =0\mathrm{bit},\\ i^{+}(s_2^1\to t^1)\hfill & =log\raisebox{1ex}{$8$}\!\left/ \!\raisebox{-1ex}{$3$}\right.\mathrm{bit},& i^{-}(s_2^1\to t^1)\hfill & =1\mathrm{bit}.\end{array}$$

(13)

Now consider the problem of decomposing information into its unique, redundant and complementary components. Figure 2 shows where exclusions induced by $s_1^1$ and $s_2^1$ overlap where they both exclude the target event $t^2$ which is an informative exclusion. This is the only exclusion induced by $s_1^1$ and hence all of the information associated with this exclusion must be redundantly provided by the event $s_2^1$. Without any formal framework, consider taking the redundant specificity and redundant ambiguity,

$$\begin{array}{cccc}{r}^{+}(s_1^1,s_2^1\to t^1)\hfill & =i^{+}(s_1^1\to t^1)=\hfill & \hfill log\raisebox{1ex}{$4$}\!\left/ \!\raisebox{-1ex}{$3$}\right.& \mathrm{bit},\hfill \end{array}$$

(14)

$$\begin{array}{cccc}{r}^{-}(s_1^1,s_2^1\to t_1)\hfill & =i^{-}(s_1^1\to t^1)=\hfill & \hfill 0& \mathrm{bit}.\hfill \end{array}$$

(15)

This would mean that the event $s_2^1$ provides the following unique specificity and unique ambiguity,

$$\begin{array}{ccc}{u}^{+}(s_1^1\backslash s_2^1\to t^1)\hfill & =i^{+}(s_1^1\to t^1)-r^{+}(s_1^1,s_2^1\to t^1)\hfill & =1\mathrm{bit},\hfill \end{array}$$

(16)

$$\begin{array}{ccc}{u}^{-}(s_1^1\backslash s_2^1\to t^1)\hfill & =i^{-}(s_1^1\to t^1)-r^{-}(s_1^1,s_2^1\to t^1)\hfill & =1\mathrm{bit}.\hfill \end{array}$$

(17)

The redundant specificity log $\raisebox{1ex}{$4$}\!\left/ \!\raisebox{-1ex}{$3$}\right.$ bit accounts for the overlapping informative exclusion of the event $t^2$. The unique specificity and unique ambiguity from $s_2^1$ are associated with its non-overlapping informative and misinformative exclusions; however, both of these 1 bit and hence, on net, $s_2^1$ is no more informative than $s_1^1$. Although obtained without a formal framework, this example highlights a need to consider the specificity and ambiguity rather than merely the pointwise mutual information.

4. Pointwise Partial Information Decomposition Using Specificity and Ambiguity

Based upon the argumentation of Section 3, consider the following axioms:

Axiom 1 (Symmetry).

Pointwise redundant specificity $i_{\cap }^{+}$ and pointwise redundant ambiguity $i_{\cap }^{-}$ are invariant under any permutation σ of source events,

$$\begin{array}{cc}\hfill i_{\cap }^{+}\left({\mathit{a}}_1,\cdots ,{\mathit{a}}_k\to t\right)& =i_{\cap }^{+}\left(\sigma ({\mathit{a}}_1),\cdots ,\sigma ({\mathit{a}}_k)\to t\right),\hfill \\ \hfill i_{\cap }^{-}\left({\mathit{a}}_1,\cdots ,{\mathit{a}}_k\to t\right)& =i_{\cap }^{-}\left(\sigma ({\mathit{a}}_1),\cdots ,\sigma ({\mathit{a}}_k)\to t\right).\hfill \end{array}$$

Axiom 2 (Monotonicity).

Pointwise redundant specificity $i_{\cap }^{+}$ and pointwise redundant ambiguity $i_{\cap }^{-}$ decreases monotonically as more source events are included,

$$\begin{array}{cc}\hfill i_{\cap }^{+}\left({\mathit{a}}_1,\cdots ,{\mathit{a}}_{k-1},{\mathit{a}}_k\to t\right)& \le i_{\cap }^{+}\left({\mathit{a}}_1,\cdots ,{\mathit{a}}_{k-1}\to t\right),\hfill \\ \hfill i_{\cap }^{-}\left({\mathit{a}}_1,\cdots ,{\mathit{a}}_{k-1},{\mathit{a}}_k\to t\right)& \le i_{\cap }^{-}\left({\mathit{a}}_1,\cdots ,{\mathit{a}}_{k-1}\to t\right).\hfill \end{array}$$

with equality if ${\mathit{a}}_k\supseteq {\mathit{a}}_i$ for any ${\mathit{a}}_i\in \{{\mathit{a}}_1,\dots ,{\mathit{a}}_{k-1}\}$.

Axiom 3 (Self-redundancy).

Pointwise redundant specificity $i_{\cap }^{+}$ and pointwise redundant ambiguity $i_{\cap }^{-}$ for a single source event ${\mathit{a}}_i$ equals the specificity and ambiguity respectively,

$$\begin{array}{ccc}{i}_{\cap }^{+}({\mathit{a}}_i\to t)\hfill & =i^{+}({\mathit{a}}_i\to t)\hfill & =h({\mathit{a}}_i),\hfill \\ i_{\cap }^{-}({\mathit{a}}_i\to t)\hfill & =i^{-}({\mathit{a}}_i\to t)\hfill & =h({\mathit{a}}_i|t).\hfill \end{array}$$

As shown in Appendix B.1, Axioms 1–3 induce two lattices—namely the specificity lattice and ambiguity lattice—which are depicted in Figure 3. Furthermore, each lattice is defined for every discrete realisation from $P(S_1,\dots ,S_n,T)$. The redundancy measures $i_{\cap }^{+}$ or $i_{\cap }^{-}$ can be thought of as a cumulative information functions which integrate the specificity or ambiguity uniquely contributed by each node as one moves up each lattice. Finally, just as in PID, performing a Möbius inversion over each lattice yielding the unique contributions of specificity and ambiguity from each sources event.

Similarly to PID, the specificity and ambiguity lattices provide a structure for information decomposition, but unique evaluation requires a separate definition of redundancy. However, unlike PID (or even PPID), this evaluation requires both a definition of pointwise redundant specificity and pointwise redundant ambiguity. Before providing these definitions, it is helpful to first see how the specificity and ambiguity lattices can be used to decompose multivariate information in the now familiar bivariate case.

4.1. Bivariate PPID Using the Specificity and Ambiguity

Consider again the bivariate case where the aim is to decompose the information provided by $s_1$ and $s_2$ about t. The specificity lattice can be used to decompose the pointwise specificity,

$$\begin{array}{cc}\hfill i^{+}(s_{1,2}\to t)& =r^{+}(s_1,s_2\to t)+u^{+}(s_1\backslash s_2\to t)+u^{+}(s_2\backslash s_1\to t)+c^{+}(s_1,s_2\to t),\hfill \\ \hfill i^{+}(s_1\to t)& =r^{+}(s_1,s_2\to t)+u^{+}(s_1\backslash s_2\to t),\hfill \\ \hfill i^{+}(s_2\to t)& =r^{+}(s_1,s_2\to t)+u^{+}(s_2\backslash s_1\to t);\hfill \end{array}$$

(18)

while the ambiguity lattice can be used to decompose the pointwise ambiguity,

$$\begin{array}{cc}\hfill i^{-}(s_{1,2}\to t)& =r^{-}(s_1,s_2\to t)+u^{-}(s_1\backslash s_2\to t)+u^{-}(s_2\backslash s_1\to t)+c^{-}(s_1,s_2\to t),\hfill \\ \hfill i^{-}(s_1\to t)& =r^{-}(s_1,s_2\to t)+u^{-}(s_1\backslash s_2\to t),\hfill \\ \hfill i^{-}(s_2\to t)& =r^{-}(s_1,s_2\to t)+u^{-}(s_2\backslash s_1\to t).\hfill \end{array}$$

(19)

These equations share the same structural form as (3) only now decompose the specificity and the ambiguity rather than the pointwise mutual information, e.g., $r^{+}(s_1,s_2\to t)$ denotes the redundant specificity while $u^{-}(s_1\backslash s_2\to t)$ denoted the unique ambiguity from $s_1$. Just as in for (3), this decomposition could be considered for every discrete realisation on the support of the joint distribution $P(S_1,S_2,T)$.

There are two ways one can be combine these values. Firstly, in a similar manner to (4), one could take the expectation of the atoms of specificity, or the atoms of ambiguity, over all discrete realisations yielding the average PI atoms of specificity and ambiguity,

$$\begin{array}{cccc}\hfill U^{+}(S_1\backslash S_2\to T)& =〈u^{+}(s_1\backslash s_2\to t)〉,\hfill & U^{-}(S_1\backslash S_2\to T)\hfill & \hfill =〈u^{-}(s_1\backslash s_2\to t)〉,\\ \hfill U^{+}(S_2\backslash S_1\to T)& =〈u^{+}(s_2\backslash s_1\to t)〉,\hfill & U^{-}(S_2\backslash S_1\to T)\hfill & \hfill =〈u^{-}(s_2\backslash s_1\to t)〉,\\ \hfill R^{+}(S_1,S_2\to T)& =〈r^{+}(s_1,s_2\to t)〉,\hfill & R^{-}(S_1,S_2\to T)\hfill & \hfill =〈r^{-}(s_1,s_2\to t)〉,\\ \hfill C^{+}(S_1,S_2\to T)& =〈c^{+}(s_1,s_2\to t)〉.\hfill & C^{-}(S_1,S_2\to T)\hfill & \hfill =〈c^{-}(s_1,s_2\to t)〉.\end{array}$$

(20)

Alternatively, one could subtract the pointwise unique, redundant and complementary ambiguity from the pointwise unique, redundant and complementary specificity yielding the pointwise unique, pointwise redundant and pointwise complementary information, i.e., recover the atoms from PPID,

$$\begin{array}{cc}\hfill r(s_1,s_2\to t)& =r^{+}(s_1,s_2\to t)-r^{-}(s_1,s_2\to t),\hfill \\ \hfill u(s_1\backslash s_2\to t)& =u^{+}(s_1\backslash s_2\to t)-u^{-}(s_1\backslash s_2\to t),\hfill \\ \hfill u(s_2\backslash s_1\to t)& =u^{+}(s_2\backslash s_1\to t)-u^{-}(s_2\backslash s_1\to t),\hfill \\ \hfill c(s_1,s_2\to t)& =c^{+}(s_1,s_2\to t)-c^{-}(s_1,s_2\to t).\hfill \end{array}$$

(21)

Both (20) and (21) are linear operations, hence one could perform both of these operations (in either order) to obtain the average unique, average redundant and average complementary information, i.e., recover the atoms from PID,

$$\begin{array}{cc}\hfill R(S_1,S_2\to T)& =R^{+}(S_1,S_2\to T)-R^{-}(S_1,S_2\to T),\hfill \\ \hfill U(S_1\backslash S_2\to T)& =U^{+}(S_1\backslash S_2\to T)-U^{-}(S_1\backslash S_2\to T),\hfill \\ \hfill U(S_2\backslash S_1\to T)& =U^{+}(S_2\backslash S_1\to T)-U^{-}(S_2\backslash S_1\to T),\hfill \\ \hfill C(S_1,S_2\to T)& =C^{+}(S_1,S_2\to T)-C^{-}(S_1,S_2\to T).\hfill \end{array}$$

(22)

4.2. Redundancy Measures on the Specificity and Ambiguity Lattices

Now that we have a structure for our information decomposition, there is a need to provide a definition of the pointwise redundant specificity and pointwise redundant ambiguity. However, before attempting to provide such a definition, there is a need to consider Remark 1 and the operational interpretation of in Section 3.3. In particular, the pointwise redundant specificity $i_{\cap }^{+}$ and pointwise redundant ambiguity $i_{\cap }^{-}$ should only depend on the size of informative and misinformative exclusions. They should not depend on the apportionment of the informative exclusions across the set of elementary events contained in the complementary event $t^{\mathsf{c}}$. Formally, this requirement will be enshrined via the following axiom.

Axiom 4 (Two-event Partition).

The pointwise redundant specificity $i_{\cap }^{+}$ and pointwise redundant ambiguity $i_{\cap }^{-}$ are functions of the probability measures on the two-event partitions ${\mathcal{A}}_1^{{\mathit{a}}_1}\times {\mathcal{T}}^t,\cdots ,{\mathcal{A}}_k^{{\mathit{a}}_k}\times {\mathcal{T}}^t$.

Since the pointwise redundant specificity $i_{\cap }^{+}$ is specificity associated with the source event which induces the smallest total exclusions, and pointwise redundant ambiguity $i_{\cap }^{-}$ is the ambiguity associated with the source event which induces the smallest misinformative exclusion, consider the following definitions.

Definition 1.

The pointwise redundant specificity is given by

$$r_{min}^{+}\left({\mathit{a}}_1,\cdots ,{\mathit{a}}_k\to t\right)=\underset{{\mathit{a}}_i}{min}{i}^{+}({\mathit{a}}_i\to t)=\underset{{\mathit{a}}_i}{min}h({\mathit{a}}_i).$$

(23)

Definition 2.

The pointwise redundant ambiguity is given by

$$r_{min}^{-}\left({\mathit{a}}_1,\cdots ,{\mathit{a}}_k\to t\right)=\underset{{\mathit{a}}_i}{min}{i}^{-}({\mathit{a}}_i\to t)=\underset{{\mathit{a}}_j}{min}h({\mathit{a}}_j|t).$$

(24)

Theorem 1.

The definitions of $r_{min}^{+}$ and $r_{min}^{-}$ satisfy Axioms 1–4.

Theorem 2.

The redundancy measures $r_{min}^{+}$ and $r_{min}^{-}$ increase monotonically on the $〈\mathcal{A}(\mathit{s}),⪯〉$.

Theorem 3.

The atoms of partial specificity ${\pi }^{+}$ and partial ambiguity ${\pi }^{-}$ evaluated using the measures $r_{min}^{+}$ and $r_{min}^{-}$ on the specificity and ambiguity lattices (respectively), are non-negative.

Appendix B.2 contains the proof of Theorems 1–3 and further relevant consideration of Defintions 1 and 2. As in (20), one can take the expectation of the either the pointwise redundant specificity $r_{\min }^{+}$ or the pointwise redundant ambiguity $r_{\min }^{-}$ to get the average redundant specificity $R_{\min }^{+}$ or the average redundant ambiguity $R_{\min }^{-}$. Alternatively, just as in (21), one can recombine the pointwise redundant specificity $r_{\min }^{+}$ and the pointwise redundant ambiguity $r_{\min }^{-}$ to get the pointwise redundant information $r_{\min }$. Finally, as per (22), one could perform both of these (linear) operations in either order to obtain the average redundant information $R_{\min }$. Note that while Theorem 3 proves that the atoms of partial specificity ${\pi }^{+}$ and partial ambiguity ${\pi }^{-}$ are non-negative, it is trivial to see that $r_{\min }$ could be negative since when source events can redundantly provide misinformation about a target event. As shown in the following theorem, $R_{\min }$ can also be negative.

Theorem 4.

The atoms of partial average information $\Pi$ evaluated by recombining and averaging ${\pi }^{\pm }$ are not non-negative.

This means that the measure $R_{\min }$ does not satisfy local positivity. Nonetheless the negativity of $R_{\min }$ is readily explainable in terms of the operational interpretation of Section 3.3, as will be discussed further in Section 5.4. However, failing to satisfy local positivity does mean that $r_{\min }$ and $R_{\min }$ do not satisfy the target monotonicity property first discussed in Bertschinger et al. [5]. Despite this, as the following theorem shows, the measures do satisfy the target chain rule.

Theorem 5 (Pointwise Target Chain Rule).

Given the joint target realisation $t_{1,2}$, the pointwise redundant information $r_{min}$ satisfies the following chain rule,

$$\begin{array}{cc}\hfill r_{min}\left({\mathit{a}}_1,\dots ,{\mathit{a}}_k\to t_{1,2}\right)& =r_{min}\left({\mathit{a}}_1,\dots ,{\mathit{a}}_k\to t_1\right)+r_{min}\left({\mathit{a}}_1,\dots ,{\mathit{a}}_k\to t_2|t_1\right),\hfill \\ & =r_{min}\left({\mathit{a}}_1,\dots ,{\mathit{a}}_k\to t_2\right)+r_{min}\left({\mathit{a}}_1,\dots ,{\mathit{a}}_k\to t_1|t_2\right).\hfill \end{array}$$

(25)

The proof of the last theorem is deferred to Appendix B.3. Note that since the expectation is a linear operation, Theorem 5 also holds for the average redundant information $R_{\min }$. Furthermore, as these results apply to any of the source events, the target chain rule will hold for any of the PPI atoms, e.g., (21), and any of the PI atoms, e.g., (22). However, no such rule holds for the pointwise redundant specificity or ambiguity. The specificity depends only on the predictor event, i.e., does not depend on the target events. As such, when an increasing number of target events are considered, the specificity remains unchanged. Hence, a target chain rule cannot hold for the specificity, or the ambiguity alone.

5. Discussion

PPID using the specificity and ambiguity takes the ideas underpinning PID and applies them on a pointwise scale while circumventing the monotonicity issue associated with the signed pointwise mutual information. This section will explore the various properties of the decomposition in an example driven manner and compare the results to the most widely-used measures from the existing PID literature. (Further examples can be found in Appendix C.) The following shorthand notation will be utilised in the figures throughout this section:

$$\begin{array}{cccccc}\hfill i_1^{+}& =i^{+}(s_1\to t),\hfill & \hfill i_2^{+}& =i^{+}(s_2\to t),\hfill & \hfill i_{1,2}^{+}& =i^{+}(s_{1,2}\to t),\hfill \\ \hfill i_1^{-}& =i^{-}(s_1\to t),\hfill & \hfill i_2^{-}& =i^{-}(s_2\to t),\hfill & \hfill i_{1,2}^{-}& =i^{-}(s_{1,2}\to t),\hfill \end{array}$$

$$\begin{array}{cccccccc}\hfill u_1^{+}& =u^{+}(s_1\backslash s_2\to t),\hfill & \hfill u_2^{+}& =u^{+}(s_2\backslash s_1\to t),\hfill & \hfill r^{+}& =r^{+}(s_1,s_2\to t),\hfill & \hfill c^{+}& =c^{+}(s_1,s_2\to t),\hfill \\ \hfill u_1^{-}& =u^{-}(s_1\backslash s_2\to t),\hfill & \hfill u_2^{-}& =u^{-}(s_2\backslash s_1\to t),\hfill & \hfill r^{-}& =r^{-}(s_1,s_2\to t),\hfill & \hfill c^{-}& =c^{-}(s_1,s_2\to t).\hfill \end{array}$$

5.1. Comparison to Existing Measures

A similar approach to the decomposition presented in this paper is due to Ince [18], who also sought to define a pointwise information decomposition. Despite the similarity in this regard, the redundancy measure $I_{\mathrm{ccs}}$ presented in [18] approaches the pointwise monotonicity problem of Section 2.3 in a different way to the decomposition presented in this paper. Specifically, $I_{\mathrm{ccs}}$ aims to utilise the pointwise co-information as a measure of pointwise redundant information since it “quantifies the set-theoretic overlap of the two univariate [pointwise] information values” ([18], p. 14). There are, however, difficulties with this approach. Firstly (unlike the average mutual information and the Shannon inequalities), there are no inequalities which support this interpretation of pointwise co-information as the set-theoretic overlap of the univariate pointwise information terms—indeed, both the univariate pointwise information and the pointwise co-information are signed measures. Secondly, the pointwise co-information conflates the pointwise redundant information with the pointwise complementary information, since by (3) we have that

$$co-i(s_1;s_2;t)=i(s_1;t)+i(s_2;t)-i(s_{1,2},t)=r(s_1,s_2\to t)-c(s_1,s_2\to t).$$

(26)

Aware of these difficulties, Ince defines $I_{\mathrm{ccs}}$ such that it only interprets the pointwise co-information as a measure of set-theoretic overlap in the case where all three pointwise information terms have the same sign, arguing that these are the only situations which admit a clear interpretation in terms of a common change in surprisal. In the other difficult to interpret situations, $I_{\mathrm{ccs}}$ defines the pointwise redundant information to be zero. This approach effectively assumes that $c(s_1,s_2\to t)=0$ in (26) when $i(s_1;t)$, $i(s_2;t)$ and $co-i(s_1;s_2;t)$ all have the same sign.

In a subsequent paper, Ince [19] also presented a partial entropy decomposition which aims to decompose multivariate entropy rather than multivariate information. As such, this decomposition is more similar to PPID using specificity and ambiguity than Ince’s aforementioned decomposition. Although similar in this regard, the measure of pointwise redundant entropy $H_{\mathrm{cs}}$ presented in [19] takes a different approach to the one presented in this paper. Specifically, $H_{\mathrm{cs}}$ also uses the pointwise co-information as a measure of set-theoretic overlap and hence as a measure of pointwise redundant entropy. As the pointwise entropy is unsigned, the difficulties are reduced but remain present due to the signed pointwise co-information. In a manner similar to $I_{\mathrm{ccs}}$, Ince defines $H_{\mathrm{cs}}$ such that it only interprets the pointwise co-information as a measure of set-theoretic overlap when it is positive. As per $I_{\mathrm{ccs}}$, this effectively assumes that $c(s_1,s_2\to t)=0$ in (26) when all information terms have the same sign. When the pointwise co-information is negative, $H_{\mathrm{cs}}$ simply ignores the co-information by defining the pointwise redundant information to be zero. In contrast to both of Ince’s approaches, PPID using specificity and ambiguity does not dispose of the set-theoretic intuition in these difficult to interpret situations. Rather, our approach considers the notion of redundancy in terms of overlapping exclusions—i.e., in terms of the underlying, unsigned measures which are amenable to a set-theoretic interpretation.

The measures of pointwise redundant specificity $r_{\min }^{+}$ and pointwise redundant ambiguity $r_{\min }^{-}$, from Definitions 1 and 2 are also similar to both the minimum mutual information $I_{\mathrm{mmi}}$ [17] and the original PID redundancy measure $I_{\min }$ [1]. Specifically, all three of these approaches consider the redundant information to be the minimum information provided about a target event t. The difference is that $I_{\min }$ applies this idea to the sources ${\mathit{A}}_1,\dots ,{\mathit{A}}_k$, i.e., to collections of entire predictor variables from $\mathit{S}$, while $r_{\min }^{\pm }$ apply this notion to the source events ${\mathit{a}}_1,\dots ,{\mathit{a}}_k$, i.e., to collections of predictor events from $\mathit{s}$. In other words, while the measure $I_{\min }$ can be regarded as being semi-pointwise (since it considers the information provided by the variables $S_1,\dots ,S_n$ about an event t), the measures $r_{\min }^{\pm }$ are fully pointwise (since they consider the information provided by the events $s_1,\dots ,s_n$ about an event t). This difference in approach is most apparent in the probability distribution PwUnq—unlike PID, PPID using the specificity and ambiguity respects the pointwise nature of information, as we will see in Section 5.3.

PPID using specificity and ambiguity also share certain similarities with the bivariate PID induced by the measure $\widetilde{UI}$ of Bertschinger et al. [11]. Firstly, Axiom 4 can be considered to be a pointwise adaptation of their Assumption ($\ast$), i.e., the measures $r_{\min }^{\pm }$ depend only on the marginal distributions $P(S_1,T)$ and $P(S_2,T)$ with respect to the two-event partitions ${\mathcal{S}}_1^{s_1}\times {\mathcal{T}}^t$ and ${\mathcal{S}}_2^{s_2}\times {\mathcal{T}}^t$. Secondly, in PPID using specificity and ambiguity, the only way one can only decide if there is complementary information $c(s_1,s_2\to t)$ is by knowing the joint distribution $P(S_1,S_2,T)$ with respect to the joint two-event partitions ${\mathcal{S}}_1^{s_1}\times {\mathcal{S}}_2^{s_2}\times {\mathcal{T}}^t$. This is (in effect) a pointwise form of their Assumption ($\ast \ast$). Thirdly, by definition $r_{\min }^{\pm }$ are given by the minimum value that any one source event provides. This is the largest possible value that one could take for these quantities whilst still requiring that the unique specificity and ambiguity be non-negative. Hence, within each discrete realisation, $r_{\min }^{\pm }$ minimise the unique specificity and ambiguity whilst maximising the redundant specificity and ambiguity. This is similar to $\widetilde{UI}$ which minimises the (average) unique information while still satisfying Assumption ($\ast$). Finally, note that since the measure ${\mathcal{S}}_{\mathrm{VK}}$ produces a bivariate decomposition which is equivalent to that of $\widetilde{UI}$ [11], the same similarities apply between PPID using specificity and ambiguity and the decomposition induced by ${\mathcal{S}}_{\mathrm{VK}}$ from Griffith and Koch [12].

5.2. Probability Distribution Xor

Figure 4 shows the canonical example of synergy, exclusive-or (Xor) which considers two independently distributed binary predictor variables $S_1$ and $S_2$ and a target variable $T=S_1\mathrm{XOR}{S}_2$. There are several important points to note about the decomposition of Xor. Firstly, despite providing zero pointwise information, an individual predictor event does indeed induce exclusions. However, the informative and misinformative exclusions are perfectly balanced such that the posterior (conditional) distribution is equal to the prior distribution, e.g., see the red coloured exclusions induced by $S_1=0$ in Figure 4. In information-theoretic terms, for each realisation, the pointwise specificity equals 1 bit since half of the total probability mass remains while the pointwise ambiguity also equals 1 bit since half of the probability mass associated with the event which subsequently occurs (i.e., $T=0$), remains. These are perfectly balanced such that when recombined, as per (11), the pointwise mutual information is equal to 0 bit, as one would expect.

Secondly, $S_1=0$ and $S_2=0$ both induce the same exclusions with respect to the target pointwise event space ${\mathcal{T}}^{T=0}$. Hence, as per the operational interpretation of redundancy adopted in Section 3.3, there is 1 bit of pointwise redundant specificity and 1 bit of pointwise redundant ambiguity in each realisation. The presence of (a form of) redundancy in Xor is novel amongst the existing measures in the PID literature. (Ince [19] also identifies a form of redundancy in Xor.) Thirdly, despite the presence of this redundancy, recombining the atoms of pointwise specificity and ambiguity for each realisation, as per (21), leaves only one non-zero PPI atom: namely the pointwise complementary information $c(s_1,s_2\to t)$ = 1 bit. Furthermore, this is true for every pointwise realisation and hence, by (22), the only non-zero PI atom is the average complementary information $C(S_1,S_2\to T)$ = 1 bit.

5.3. Probability Distribution PwUnq

Figure 5 shows the probability distribution PwUnq introduced in Section 2.2. Recombining the decomposition via (21) yields the pointwise information decomposition proposed in Table 1—unsurprisingly, the explicitly pointwise approach results in a decomposition which does not suffer from the pointwise unique problem of Section 2.2.

In each realisation, observing a 0 in either source provides the same balanced informative and misinformative exclusions as in Xor. Observing either a 1 or 2 provides the same misinformative exclusion as observing the 0, but provides a larger informative exclusion than 0. This leaves only the probability mass associated with the event which subsequently occurs remaining (hence why observing a 1 and 2 is fully informative about the target). Information theoretically, in each realisation the predictor events provide 1 bit of redundant pointwise specificity and 1 bit of redundant pointwise ambiguity while the fully informative event additionally provides 1 bit of unique specificity.

5.4. Probability Distribution RdnErr

Figure 6 shows the probability distribution redundant-error (RdnErr) which considers two predictors which are nominally redundant and fully informative about the target, but where one predictor occasionally makes an erroneous prediction. Specifically, Figure 6 shows the decomposition of RdnErr where $S_2$ makes an error with a probability $\epsilon =\raisebox{1ex}{$1$}\!\left/ \!\raisebox{-1ex}{$4$}\right.$. The important feature to note about this probability distribution is that upon recombining the specificity and ambiguity and taking the expectation over every realisation, the resultant average unique information from $S_2$ is $U(S_2\backslash S_1\to T)$ = −0.811 bit.

On first inspection, the result that the average unique information can be negative may seem problematic; however, it is readily explainable in terms of the operational interpretation of Section 3.3. In RdnErr, a source event always excludes exactly $\raisebox{1ex}{$1$}\!\left/ \!\raisebox{-1ex}{$2$}\right.$ of the total probability mass, thus every realisation contains 1 bit of redundant pointwise specificity. The events of the error-free $S_1$ induce only informative exclusions and as such provide 0 bit of pointwise ambiguity in each realisation. In contrast, the events in the error-prone $S_2$ always induce a misinformative exclusion, meaning that $S_2$ provides unique pointwise ambiguity in every realisation. Since $S_2$ never provides unique specificity, the average unique information is negative on average.

Despite the negativity of the average unique information, in is important to observe that $S_2$ provides 0.189 bit of information since $S_2$ also provides 1 bit of average redundant information. It is not that $S_2$ provides negative information on average (as this is not possible); rather it is that not all of the information provided by $S_2$ (i.e., the specificity) is “useful” ([42], p. 21). This is in contrast to $S_1$ which only provides useful specificity. To summarise, it is the unique ambiguity which distinguishes the information provided by variable $S_2$ from $S_1$, and hence why $S_2$ is deemed to provide negative average unique information. This form of uniqueness can only be distinguished by allowing the average unique information to be negative. This of course, requires abandoning the local positivity as a required property, as per Theorem 4. Few of the existing measures in the PID literature consider dropping this requirement as negative information quantities are typically regarded as being “unfortunate” ([43], p. 49). However, in the context of the pointwise mutual information, negative information values are readily interpretable as being misinformative values. Despite this, the average information from each predictor must be non-negative; however, it may be that what distinguishes one predictor from another are precisely the misinformative predictor events, meaning that the unique information is in actual fact, unique misinformation. Forgoing local positivity makes the PPID using specificity and ambiguity novel (the other exception in this regard is Ince [18] who was first to consider allowing negative average unique information.)

5.5. Probability Distribution Tbc

Figure 7 shows the probability distribution two-bit-copy (Tbc) which considers two independently distributed binary predictor variables $S_1$ and $S_2$, and a target variable T consisting of a separate elementary event for each joint event $S_{1,2}$. There are several important points to note about the decomposition of Tbc. Firstly, due to the symmetry in the probability distribution, each realisation will have the same pointwise decomposition. Secondly, due to the construction of the target, there is an isomorphism (Again, isomorphism should be taken to mean isomorphic probability spaces, e.g., [37], p. 27 or [38], p. 4) between $P(T)$ and $P(S_1,S_2)$, and hence the pointwise ambiguity provided by any (individual or joint) predictor event is 0 bit (since given t, one knows $s_1$ and $s_2$). Thirdly, the individual predictor events $s_1$ and $s_2$ each exclude $\raisebox{1ex}{$1$}\!\left/ \!\raisebox{-1ex}{$2$}\right.$ of the total probability mass in $P(T)$ and so each provide 1 bit of pointwise specificity; thus, by (23), there is 1 bit of redundant pointwise specificity in each realisation. Fourthly, the joint predictor event $s_{1,2}$ excludes $\raisebox{1ex}{$3$}\!\left/ \!\raisebox{-1ex}{$4$}\right.$ of the total probability mass, providing 2 bit of pointwise specificity; hence, by (18), each joint realisation provides 1 bit of pointwise complementary specificity in addition to the 1 bit of redundant pointwise specificity. Finally, putting this together via (22), Tbc consists of 1 bit of average redundant information and 1 bit of average complementary information.

Although “surprising” ([5], p. 268), according to the operational interpretation adopted in Section 3.3, two independently distributed predictor variables can share redundant information. That is, since the exclusions induced by $s_1$ and $s_2$ are the same with respect to the two-event partition ${\mathcal{T}}^t$, the information associated with these exclusions is regarded as being the same. Indeed, this probability distribution highlights the significance of specific reference to the two-event partition in Section 3.3 and Axiom 4. (This can be seen in the probability mass diagram in Figure 7, where the events $S_1=0$ and $S_2=0$ exclude different elementary target events within the complementary event $0^{\mathsf{c}}$ and yet are considered to be the same exclusion with respect to the two-event partition ${\mathcal{T}}^0$.) That these exclusions should be regarded as being the same is discussed further in Appendix A. Now however, there is a need to discuss Tbc in terms of Theorem 5 (Target Chain Rule).

Tbc was first considered as a “mechanism” ([6], p. 3) where “the wires don’t even touch” ([12], p. 167), which merely copies or concatenates $S_1$ and $S_2$ into a composite target variable $T_{1,2}=(T_1,T_2)$ where $T_1=S_1$ and $T_2=S_2$. However, using causal mechanisms as a guiding intuition is dubious since different mechanisms can yield isomorphic probability distributions ([44], and references therein). In particular, consider two mechanisms which generate the composite target variables $T_{1,3}=(T_1,T_3)$ and $T_{2,3}=(T_2,T_3)$ where $T_3=S_1\mathrm{XOR}{S}_2$. As can be seen in Figure 7, both of these mechanisms generate the same (isomorphic) probability distribution $P(S_1,S_2,T)$ as the mechanism generating $T_{1,2}$. If an information decomposition is to depend only on the probability distribution $P(S_1,S_2,T)$, and no other semantic details such as labelling, then all three mechanisms must yield the same information decomposition—this is not clear from the mechanistic intuition.

Although the decomposition of the various composite target variables must be the same, there is no requirement that the three systems must yield the same decomposition when analysed in terms of the individual components of the composite target variables. Nonetheless, there ought to be a consistency between the decomposition of the composite target variables and the decomposition of the component target variables—i.e., there should be a target chain rule. As shown in Theorem 5, the measures $r_{\min }$ and $R_{\min }$ satisfy the target chain rule, whereas $I_{\min }$, $\widetilde{UI}$, $I_{\mathrm{red}}$ and ${\mathcal{S}}_{\mathrm{VK}}$ do not [5,7]. Failing to satisfy the target chain rule can lead to inconsistencies between the composite and component decompositions, depending on the order in which one considers decomposing the information (this is discussed further in Appendix A.3). In particular,

Table 2. Shows the decomposition of the quantities in the first row induced by the measures in the first column. For consistency, the decomposition of $I(S_{1,2};T_{1,3})$ should equal both the sum of the decomposition of $I(S_{1,2};T_1)$ and $I(S_{1,2};T_3|T_1)$, and the sum of the decomposition of $I(S_{1,2};T_3)$ and $I(S_{1,2};T_1|3)$. Note that the decomposition induced by $\widetilde{UI}$, $I_{\mathrm{red}}$ and ${\mathcal{S}}_{\mathrm{VK}}$ are not consistent. In contrast, $R_{\min }$ is consistent due to Theorem 5.

	$\mathit{I}({\mathit{S}}_{1,2};{\mathit{T}}_{1,3})$	$\mathit{I}({\mathit{S}}_{1,2};{\mathit{T}}_1)$	$\mathit{I}({\mathit{S}}_{1,2};{\mathit{T}}_3|{\mathit{T}}_1)$	$\mathit{I}({\mathit{S}}_{1,2};{\mathit{T}}_3)$	$\mathit{I}({\mathit{S}}_{1,2};{\mathit{T}}_1|{\mathit{T}}_3)$
$\begin{array}{c}\widetilde{UI},I_{\mathrm{red}},\\ {\mathcal{S}}_{\mathrm{VK}}\end{array}$	$\begin{array}{cc}\hfill U(S_1\backslash S_2\to T_{1,3})& =1\hfill \\ \hfill U(S_2\backslash S_1\to T_{1,3})& =1\hfill \end{array}$	$U(S_1\backslash S_2\to T_1)=1$	$U(S_2\backslash S_1\to T_3|T_1)=1$	$C(S_1,S_2\to T_3)=1$	$R(S_1,S_2\to T_1|T_3)=1$
$R_{\min }$	$\begin{array}{cc}\hfill R(S_1,S_2\to T_{1,3})& =1\hfill \\ \hfill C(S_1,S_2\to T_{1,3})& =1\hfill \end{array}$	$\begin{array}{cc}\hfill U(S_2\backslash S_1\to T_1)& =-1\hfill \\ \hfill R(S_1,S_2\to T_1)& =1\hfill \\ \hfill C(S_1,S_2\to T_1)& =1\hfill \end{array}$	$U(S_2\backslash S_1\to T_3|T_1)=1$	$C(S_1,S_2\to T_3)=1$	$R(S_1,S_2\to T_1|T_3)=1$

shows how $\widetilde{UI}$, $I_{\mathrm{red}}$ and ${\mathcal{S}}_{\mathrm{VK}}$ all provide the same inconsistent decomposition for Tbc when considered in terms of the composite target variable $T_{1,3}$. In contrast, $R_{\min }$ produces a consistent decomposition of $T_{1,3}$. Finally, based on the above isomorphism, consider the following (the proof is deferred to Appendix B.3).

Theorem 6.

The target chain rule, identity property and local positivity, cannot be simultaneously satisfied.

5.6. Summary of Key Properties

The following are the key properties of the PPID using the specificity and ambiguity. Property 1 follows directly from the Definitions 1 and 2. Property 2 follows from Theorems 3 and 4. Property 3 follows from the probability distribution Tbc in Section 5.5. Property 4 was discussed in Section 4.2. Property 5 is proved in Theorem 5.

Property 1.

When considering the redundancy between the source events ${\mathit{a}}_1,\dots ,{\mathit{a}}_k$, at least one source event ${\mathit{a}}_i$ will provide zero unique specificity, and at least one source event ${\mathit{a}}_j$ will provide zero unique ambiguity. The events ${\mathit{a}}_i$ and ${\mathit{a}}_j$ are not necessarily the same source event.

Property 2.

The atoms of partial specificity and partial ambiguity satisfy local positivity, ${\pi }^{\pm }\ge 0$. However, upon recombination and averaging, the atoms of partial information do not satisfy local positivity, $\Pi \ge 0$.

Property 3.

The decomposition does not satisfy the identity property.

Property 4.

The decomposition does not satisfy the target monotonicity property.

Property 5.

The decomposition satisfies the target chain rule.

6. Conclusions

The partial information decomposition of Williams and Beer [1,2] provided an intriguing framework for the decomposition of multivariate information. However, it was not long before “serious flaws” ([11], p. 2163) were identified. Firstly, the measure of redundant information $I_{\min }$ failed to distinguish between whether predictor variables provide the same information or merely the same amount of information. Secondly, $I_{\min }$ fails to satisfy the target chain rule, despite this addativity being one of the defining characteristics of information. Notwithstanding the problems, the axiomatic derivation of the redundancy lattice was too elegant to be abandoned and hence several alternate measures were proposed, i.e., $I_{\mathrm{red}}$, $\widetilde{UI}$ and ${\mathcal{S}}_{\mathrm{VK}}$ [6,11,12]. Nevertheless, as these measures all satisfy the identity property, they cannot produce a non-negative decomposition for an arbitrary number of variables [13]. Furthermore, none of these measures satisfy the target chain rule meaning they produce inconsistent decompositions for multiple target variables. Finally, in spite of satisfying the identity property (which many consider to be desirable), these measures still fail to identify when variables provide the same information, as exemplified by the pointwise unique problem presented in Section 2.

This paper took the axiomatic derivation of the redundancy lattice from PID and applied it to the unsigned entropic components of the pointwise mutual information. This yielded two separate redundancy lattices—the specificity and the ambiguity lattices. Then based upon an operational interpretation of redundancy, measures of pointwise redundant specificity $r_{\min }^{+}$ and pointwise redundant ambiguity $r_{\min }^{-}$ were defined. Together with specificity and ambiguity lattices, these measures were used to decompose multivariate information for an arbitrary number of variables. Crucially, upon recombination, the measure $r_{\min }$ satisfies the target chain rule. Furthermore, when applied to PwUnq, these measures do not result in the pointwise unique problem. In our opinion, this demonstrates that the decomposition is indeed correctly identifying redundant information. However, others will likely disagree with this point given that the measure of redundancy does not satisfy the identity property. According to the identity property, independent variables can never provide the same information. In contrast, according to the operational interpretation adopted in this paper, independent variables can provide the same information if they happen to provide the same exclusions with respect to the two-event target distribution. In any case, the proof of Theorem 6 and the subsequent discussion in Appendix B.3, highlights the difficulties that the identity property introduces when considering the information provided about events in separate target variables. (See further discussion in Appendix A.3).

Our future work with this decomposition will be both theoretical and empirical. Regarding future theoretical work, given that the aim of information decomposition is to derive measures pertaining to sets of random variables, it would be worthwhile to derive the information decomposition from first principles in terms of measure theory. Indeed, such an approach would surely eliminate the semantic arguments (about what it means for information to unique, redundant or complementary), which currently plague the problem domain. Furthermore, this would certainly be a worthwhile exercise before attempting to generalise the information decomposition to continuous random variables. Regarding future empirical work, there are many rich data sets which could be decomposed using this decomposition including financial time-series and neural recordings, e.g., [28,33,34].