Synergy as the Failure of Distributivity

Abstract

The concept of emergence, or synergy in its simplest form, is widely used but lacks a rigorous definition. Our work connects information and set theory to uncover the mathematical nature of synergy as the failure of distributivity. For the trivial case of discrete random variables, we explore whether and how it is possible to get more information out of lesser parts. The approach is inspired by the role of set theory as the fundamental description of part–whole relations. If taken unaltered, synergistic behavior is forbidden by the set-theoretic axioms. However, random variables are not a perfect analogy of sets: we formalize the distinction, highlighting a single broken axiom—union/intersection distributivity. Nevertheless, it remains possible to describe information using Venn-type diagrams. The proposed multivariate theory resolves the persistent self-contradiction of partial information decomposition and reinstates it as a primary route toward a rigorous definition of emergence. Our results suggest that non-distributive variants of set theory may be used to describe emergent physical systems.

1. Introduction

Reductionism is a standard scientific approach in which a system is studied by breaking it into smaller parts. However, some of the most interesting phenomena in physics and biology appear to resist such disentanglement. In these cases, complexity emerges from intricate interactions between many predominantly simple components [1]. Such synergic systems are typically described as “a whole that is greater than the sum of its parts”. To pour quantitative meaning into this equation-like definition, it is natural to borrow tools from the mathematical theory that describes part–whole relationships, namely set theory. Unfortunately, for finite sets, a simple Venn diagram suffices to demonstrate that the size of the whole ($A\cup B$) can never exceed the sum of the sizes of its parts (A and B):

$$\begin{array}{c}|A\cup B|=\left|A\right|+\left|B\right|-|A\cap B|\le \left|A\right|+\left|B\right|\hfill \end{array}$$

(1)

In fact, the trivial interaction, $A\cap B$, between the two parts of the system decreases the size of the whole rather than increasing it.

To allow for more intricate interactions, one can turn to the realm of random variables. It is well known that measuring the outcome of two random variables can provide more information than the sum of what is obtained when measuring each separately. Moreover, the textbook description of the interactions between random variables often involves set-theoretical-like Venn diagrams [2]. These two facts lead to the intriguing possibility that random variables may lend themselves to a mathematical description of non-trivial whole–part relationships.

Take two discrete variables W and Z: the information W contains about Z is determined by the mutual information function $I(W;Z)$ [3]. Cases for which W can be presented as a joint random variable $W=(X,Y)$ allow us to compare the whole against its parts:

$$\begin{array}{c}I\left(\right(X,Y);Z)⪋I(X;Z)+I(Y;Z)\hfill \end{array}$$

(2)

In other words, looking at both system parts together can convey either more or less information than their added values. Therefore, and in contrast to Equation (1), this formalism can be used to describe synergy.

In their seminal paper [4], Williams and Beer proposed the framework of partial information decomposition as a way of assessing the underlying structure of a two discrete random variable system and quantifying the amount of synergy between its parts. They suggested that, much like a set of elements, each variable can be decomposed into separate information “subsets”. These information atoms are assumed to have non-negative size and represent the information that is shared between two variables (R), uniquely present in only one of them ($U_X,U_Y$):

$$\begin{array}{c}I(X;Z)=R+U_X,\hfill \\ I(Y;Z)=R+U_Y,\hfill \\ I((X,Y);Z)=R+U_X+U_Y+S,\hfill \\ R,U_X,U_Y,S\ge 0\hfill \end{array}$$

(3)

See Figure 1 for clarification. An additional synergy term (S) was artificially introduced to provide a simple mechanism that allows the whole to be greater than the sum of its parts:

$$\begin{array}{c}I\left(\right(X,Y);Z)-I(X;Z)-I(Y;Z)=S-R>0\mathrm{iff}S>R\hfill \end{array}$$

(4)

A series of papers [5,6,7] focused on calculating these atoms’ sizes by fixing the single remaining degree of freedom in Equation (3). No consensus has yet been reached regarding a single physical solution. Meanwhile, the field of applications is getting wider [8,9]. Recent works extend the theory to continuous variables [10,11], introduce causality [12,13], and consider quantum information [14].

Unfortunately, partial information decomposition has a significant drawback that puts the whole approach into question: no extension beyond two variables is possible without a fundamental self-contradiction [15]. Some authors attempted to resolve this by abandoning the basic properties required of information atoms, including their non-negativity [16,17].

In what follows, we reconsider the foundations of partial information decomposition and pinpoint the source of its long-standing self-contradictions. To do this, we follow H. K. Ting [18] to establish a rigorous relation between information and set theories and highlight a fundamental distinction between them: random variables, unlike sets, do not adhere to the union/intersection distributivity axiom [19]. This leads us to study a distributivity-free variant of set theory as a possible self-consistent theory of information atoms. Within this framework, we demonstrate that the presence of synergistic properties is a direct consequence of the broken axiom. In the case of $N=3$ random variables, we show that the amount of synergistic information precisely coincides with the extent to which distributivity is breached. The acquired understanding allows us to resolve the contradictions and suggest a coherent multivariate theory, which may provide the foundations for quantifying emergence in large systems.

2. Set-Theoretic Approach to Information

In this section, we formalize the distinction between finite sets and discrete random variables. Clearly, it is linked to the synergistic behavior of the latter. We will first focus on a special illustrative example: the XOR gate. This system contains neither redundant nor unique information, which will emphasize the peculiar properties of synergy. A more general discussion, including arbitrary random variables, will be presented in the next section.

2.1. Basic Random Variable Operations

Some set-theoretic operations have straightforward extensions to random variables [18,20,21]. The first of these relies on the similarity between Equations (1) and (2) and identifies taking the joint variable with the union operator (∪). One can now go on to define random variable inclusion as:

$$\begin{array}{c}X\subseteq Y\iff \exists Z:X\cup Z=Y\hfill \end{array}$$

(5)

which is, actually, equivalent to X being a deterministic function of Y.

The inclusion–exclusion formula ([22], Chapter 3.1) applied to two random variables reveals mutual information as the size of the intersection between two random variables:

$$\begin{array}{c}H(X\cup Y)=H\left(X\right)+H\left(Y\right)-I(X;Y),\hfill \end{array}$$

(6)

where Shannon entropy H is regarded as a measure on the random variable space. Indeed, it complies with many properties required of a mathematical measure ([23], Chapter 1.4): non-negativity, monotonicity, and subadditivity. Furthermore, entropy is zero only for deterministic variables, which play the role of an empty set (Appendix A, Lemma A1):

$$\begin{array}{c}H\left(X\right)\ge 0,\hfill \\ X\subseteq Y\Rightarrow H\left(X\right)\le H\left(Y\right),\hfill \\ H\left(\bigcup _{i=1}^N{X}_i\right)\le \sum _{i=1}^{N}H\left(X_i\right),\hfill \\ H\left(X\right)=0\iff X=\varnothing \hfill \end{array}$$

(7)

A rigorous definition of intersection (∩) needs to comply with the inclusion order (5) $X\cap Y\subseteq X,X\cap Y\subseteq Y$, in addition to the size constraint. Unfortunately, a random variable satisfying both conditions does not always exist [24]. Nonetheless, a physically sensible intersection may be inferred in several cases:

$$\begin{array}{c}H(X\cup Y)=H\left(X\right)+H\left(Y\right)\iff X\cap Y=\varnothing ,\hfill \\ X\subseteq Y\iff H\left(X\right)=I(X;Y)\iff X\cap Y=X\hfill \end{array}$$

(8)

These simple parallels between information theory and set theory are enough to study information decomposition in a random variable XOR gate.

2.2. The Simplest Synergic System: XOR Gate

Consider three pairwise independent fair coins $O_1,O_2,O_3$ with an additionally imposed higher order interaction–parity rule $O_3=O_1\oplus O_2$. It fixes the value of the third variable to be 0 whenever the values of $O_1$ and $O_2$ coincide, and 1 otherwise.

Probability	$O_1$	$O_2$	$O_3$
$1/4$	0	0	0
0	0	0	1
0	0	1	0
$1/4$	0	1	1
0	1	0	0
$1/4$	1	0	1
$1/4$	1	1	0
0	1	1	1

One can easily calculate the amount of information $O_1,O_2$ and $(O_1,O_2)$ convey about $O_3$. The comparison of these contributions shows that the system is indeed synergic:

$$\begin{array}{c}I(O_1;O_3)=0\mathrm{bit},\hfill \\ I(O_2;O_3)=0\mathrm{bit},\hfill \\ I((O_1,O_2);O_3)=1\mathrm{bit},\hfill \\ I((O_1,O_2);O_3)>I(O_1;O_3)+I(O_2;O_3)\hfill \end{array}$$

(9)

Moreover, by substituting the above result into the decomposition Equation (3), we find that the system contains only a single non-zero information atom $S=1\mathrm{bit}$. This allows us to study synergy separately from any other contributions on this example.

2.3. Subdistributivity

When taking a closer look at the XOR gate, our set-theoretic intuition for random variables breaks down even further. The pairwise independence dictates $O_2\cap O_3=O_1\cap O_3=\varnothing$, while the parity rule makes $O_3$ a deterministic function of the joint variable $(O_1,O_2)$:

$$\begin{array}{c}{O}_3\subseteq (O_1\cup O_2)\Rightarrow (O_1\cup O_2)\cap O_3=O_3\hfill \end{array}$$

(10)

A simple conclusion from these facts is that the XOR-gate variables do not comply with the set-theoretic axiom of distributivity:

$$\begin{array}{c}(O_1\cup O_2)\cap O_3=O_3\ne \varnothing =(O_1\cap O_3)\cup (O_2\cap O_3)\hfill \end{array}$$

(11)

Nevertheless, it can be shown that a weaker relation of subdistributivity holds for any three random variables (Appendix A, Lemma A2):

$$\begin{array}{c}(X\cup Y)\cap Z\supset (X\cap Z)\cup (Y\cap Z)\hfill \end{array}$$

(12)

Even though it is evident that random variables are quite different from sets, we argue that some of the logic behind partial information decomposition may be recovered by extending set-theoretic notions, such as the inclusion–exclusion principle and Venn diagrams, to non-distributive systems.

2.4. Inclusion–Exclusion Formulas

The inclusion–exclusion formula for the XOR gate can be obtained by repeatedly applying the two-variable Equation (6) and using that $I(X;Y)=H(X\cap Y)$ when the intersection exists:

$$\begin{array}{c}H(O_1\cup O_2\cup O_3)=\hfill \\ =H(O_1\cup O_2)+H\left(O_3\right)-H((O_1\cup O_2)\cap O_3)=\hfill \\ =H\left(O_1\right)+H\left(O_2\right)+H\left(O_3\right)-H((O_1\cup O_2)\cap O_3)\hfill \end{array}$$

(13)

It disagrees with the analogous set-theoretic formula (for non-intersecting sets) only in the last term, which is non-zero precisely due to the subdistributivity. Note that while the rest of the terms are symmetric with respect to the permutation of indices, expression $(O_1\cup O_2)\cap O_3$ is not as it explicitly depends on the order of derivation. This essentially leads to three different inclusion–exclusion formulas. Nonetheless, the size of the distributivity-breaking term remains invariant:

$$\begin{array}{c}H((O_1\cup O_2)\cap O_3)=H((O_1\cup O_3)\cap O_2)=H((O_2\cup O_3)\cap O_1)\hfill \end{array}$$

(14)

2.5. Construction of Venn-Type Diagram for XOR Gate

The non-uniqueness of inclusion–exclusion formulas complicates the construction of Venn diagrams. A way of tackling this as well as some further intuition can be traced via our XOR gate example.

In set theory, Venn diagrams act as graphical representations of the inclusion–exclusion principle ([22], Chapter 3.1). The inclusion–exclusion formula computes the size of union as a sum of all possible intersections between the participating sets. For correct bookkeeping, this is achieved with alternating signs that account for the covering number—the number of times each intersection is counted as a part of some set. In classical set theory, the covering number of an intersection is trivially the number of sets which are being intersected. However, (13) includes the distributivity-breaking term, which is absent from this classical theory and whose covering number is not evident. It appears with a negative sign which signifies an even-times covered region. In this three variable system, the only even alternative is a 2-covered region. From another perspective, in each of the three possible formulas $O_k$ is covered once by itself and one more time by the union $O_i\cup O_j$ (though not by $O_i$ or $O_j$ individually). As for the size of this region, independent of k, it measures at 1 bit of information. Denoting this area as ${\Pi }_s$, we have:

$$\begin{array}{c}{\Pi }_s\left[2\right]=H((O_i\cup O_j)\cap O_k)=H\left(O_k\right)=1\mathrm{bit},\hfill \end{array}$$

(15)

where the covering number is indicated in the brackets $\left[\right]$. To find the rest of our diagram’s regions, we borrow two properties of set-theoretic diagrams.

First of all, in a system of N arbitrary random variables $X_1,\cdots X_N$, the total entropy of the system is equal to the sum of all diagram regions ${\Pi }_i\left[c_i\right]$:

$$\begin{array}{c}H(X_1,\cdots X_N)=\sum _i{\Pi }_i\left[c_i\right]\hfill \end{array}$$

(16)

Second, the sum of individual variables’ entropies is equal to the sum of region sizes times their corresponding covering numbers $c_i$:

$$\begin{array}{c}H\left(X_1\right)+H\left(X_2\right)+\cdots +H\left(X_N\right)=\sum _i{c}_i{\Pi }_i\left[c_i\right]\hfill \end{array}$$

(17)

These properties may be viewed as the information conservation law: adding new sources should either introduce new information or increase the covering of existing regions.

Let us assume that in addition to ${\Pi }_s$ the diagram of the XOR gate contains several more regions ${\Pi }_{j\ne s}$. To calculate their sizes and coverings we apply (16) and (17):

$$\begin{array}{c}\sum _{j\ne s}(c_j-1){\Pi }_j\left[c_j\right]=0\mathrm{bit}\hfill \end{array}$$

(18)

We use the fact that information is non-negative and discard meaningless empty regions. The above equation then allows for a single 1-bit region, which is covered once:

$$\begin{array}{c}{\Pi }_g\left[1\right]=1\mathrm{bit}\hfill \end{array}$$

(19)

To respect the physical meaning behind the diagram regions as pieces of information, we demand the structure of the diagram to be well-defined. In other words, despite the existence of three different versions of inclusion–exclusion formula (13), they are all assumed to describe the same system. Indeed, our result remains invariant with respect to index permutations in terms of region sizes and covering numbers.

In regard to the shape of the Venn diagram, this assumption dictates along with (15) that region ${\Pi }_s$ corresponds to all variables at the same time:

$$\begin{array}{c}{\Pi }_s=H\left(O_1\right)=H\left(O_2\right)=H\left(O_3\right)\hfill \end{array}$$

(20)

One can think of ${\Pi }_s$ as a 2-covered triple intersection between $O_1,O_2$, and $O_3$. This is a drastic divergence from classical set theory, where an intersection between n sets is covered exactly n times. As we shall see, without distributivity, n variables can have multiple intersection regions with different covering numbers $1\le c\le n$.

Moving on to the second region in this system: ${\Pi }_g$ appears as a leftover when taking the difference between the whole system and ${\Pi }_s$ and by set-theoretic intuition, it does not intersect with $O_k$ for any k. As such, it is not a part of any single variable.

Finally, we combine all findings into a system of equations, which generates the Venn-type diagram of the information distribution inside the XOR gate (Figure 2):

Figure 2. A Venn-type diagram for the XOR gate. Each variable is represented by a primary color circle (red, yellow, blue) while the outer circle outlines the whole system. Of the total 2 bits of the XOR gate, one is covered two times and is represented by the inner disk. Since it is covered twice, this area is colored by pairwise color-blends (orange, purple, and green). Since it is covered by three variables, it includes patches of all three possible blends. A critical difference between this diagram and a set-theoretic one is that even though the three variables have no pairwise intersections, the inner disk representing the mutual content of all three variables is non-empty. The remaining 1 bit is covered once and resides only inside the joint variable. Since this area is covered once, it is colored by primary colors. Patches of all three colors are used since this area does not belong to any single variable.

Figure 2. A Venn-type diagram for the XOR gate. Each variable is represented by a primary color circle (red, yellow, blue) while the outer circle outlines the whole system. Of the total 2 bits of the XOR gate, one is covered two times and is represented by the inner disk. Since it is covered twice, this area is colored by pairwise color-blends (orange, purple, and green). Since it is covered by three variables, it includes patches of all three possible blends. A critical difference between this diagram and a set-theoretic one is that even though the three variables have no pairwise intersections, the inner disk representing the mutual content of all three variables is non-empty. The remaining 1 bit is covered once and resides only inside the joint variable. Since this area is covered once, it is colored by primary colors. Patches of all three colors are used since this area does not belong to any single variable.

$$\begin{array}{c}H\left(O_1\right)={\Pi }_s,\hfill \\ H\left(O_2\right)={\Pi }_s,\hfill \\ H\left(O_3\right)={\Pi }_s,\hfill \\ H(O_1\cup O_2)={\Pi }_s+{\Pi }_g,\hfill \\ H(O_2\cup O_3)={\Pi }_s+{\Pi }_g,\hfill \\ H(O_1\cup O_3)={\Pi }_s+{\Pi }_g,\hfill \\ H(O_1\cup O_2\cup O_3)={\Pi }_s+{\Pi }_g\hfill \end{array}$$

(21)

In usual Venn diagrams, intersections represent correlations between different parts. Similarly, in the XOR gate the higher-order parity interaction added on top of the non-correlated variables is responsible for the appearance of a 2-covered triple intersection.

2.6. Synergy as an Information Atom

We can compare our set-theory-inspired results against the expectations of the partial information decomposition. Namely, Equations (3) state that the information $O_1$ and $O_2$ carry about $O_3$ can be described by the atoms $R=U_1=U_2=0\mathrm{bit},S=1\mathrm{bit}$. The left side of each line in (3) may be rewritten by definition as an intersection of random variables:

$$\begin{array}{c}I(X;Z)=H(X\cap Z),\hfill \\ I(Y;Z)=H(Y\cap Z),\hfill \\ I\left(\right(X,Y);Z)=H\left(\right(X\cup Y)\cap Z)\hfill \end{array}$$

(22)

For the XOR gate, the former two are empty, while the last line links the original definition of synergistic information to the non-set-theoretic term of the inclusion–exclusion Formula (13) and the peculiar region of the corresponding diagram:

$$\begin{array}{c}S=I((O_1,O_2);O_3)=H((O_1\cup O_2)\cap O_3)={\Pi }_s\hfill \end{array}$$

(23)

Curiously, synergistic behavior of mutual information does not contradict the subadditivity of entropy. The synergistic information piece S is not new to the system and is always contained in the variables’ full entropy.

The nature of ghost atom $G={\Pi }_g$ is deeply connected to this outcome, even though it does not explicitly participate in the decomposition. Consider the individual contributions by each of the sources $O_1,O_2$:

$$\begin{array}{c}I(O_{i=1,2};O_3)=H\left(O_i\right)+H\left(O_3\right)-H(O_i\cup O_3)\hfill \end{array}$$

(24)

Using (21), we can rewrite this in terms of information atoms:

$$\begin{array}{c}I(O_i;O_3)={\Pi }_s+{\Pi }_s-({\Pi }_s+{\Pi }_g)=S-G=0\hfill \end{array}$$

(25)

The equality between the synergistic and ghost atoms ensures that the former is exactly canceled from the individual contribution by each source. Synergistic information is, of course, still present in the “whole” (23). This circumstance is responsible for creating the illusion of synergy appearing out of nowhere when sources are combined.

3. General Trivariate Decomposition

The XOR gate example studied above is a degenerate example with a sole synergistic information atom. We will now expand our description into a system with non-synergistic components with the aim to characterize any three variables using information atoms.

3.1. Extended Random Variable Space

The lack of a proper description for information intersections severely limits our ability to decompose the information content of more general random variable systems. Our solution for this issue is inspired by an elegant duality between set theory and information quantities found by H. K. Ting in [18] and further elaborated in [21]. It simply extends the space of random variables to include all elements produced by operations $\cup ,\cap ,\setminus$ (2), (8) and (28). Entropy is extended as a (non-negative) measure $\widehat{H}$ such that:

$$\begin{array}{c}\widehat{H}\left(X\right)=0\iff X=\varnothing ,\hfill \\ X\cap Y=\varnothing \iff \widehat{H}(X\cup Y)=\widehat{H}\left(X\right)+\widehat{H}\left(Y\right)\hfill \end{array}$$

(26)

To approach the problem of characterizing information atoms in the trivariate case, we derive the corresponding inclusion–exclusion formula. As stated previously, the bivariate version (6) holds without alterations (Appendix A, Lemma A3). Now, in contrast, we get a distributivity-breaking difference term, which, to make matters even worse, depends on the order of derivation (Appendix A, Theorem A1). One possible variant of this formula is portrayed in Figure 3:

$$\begin{array}{c}\widehat{H}(X_1\cup X_2\cup X_3)=\hfill \\ =\widehat{H}\left(X_1\right)+\widehat{H}\left(X_2\right)+\widehat{H}\left(X_3\right)-\hfill \\ -\widehat{H}(X_1\cap X_2)-\widehat{H}(X_1\cap X_3)-\widehat{H}(X_2\cap X_3)+\hfill \\ +\widehat{H}(X_1\cap X_2\cap X_3)-\Delta \widehat{H},\hfill \end{array}$$

(27)

where $\Delta \widehat{H}=\widehat{H}(((X_{\sigma \left(1\right)}\cup X_{\sigma \left(2\right)})\cap X_{\sigma \left(3\right)})\setminus ((X_{\sigma \left(1\right)}\cap X_{\sigma \left(3\right)})\cup (X_{\sigma \left(2\right)}\cap X_{\sigma \left(3\right)})))$ for any permutation of indices $\sigma$. The difference is defined as:

$$\begin{array}{c}D=X\setminus Y\iff D\cap Y=\varnothing ,D\cup (X\cap Y)=X\hfill \end{array}$$

(28)

In general, due to subdistributivity the difference may not be unique (Appendix A, (A20)). Its size, on the other hand, is fixed as $\widehat{H}(X\setminus Y)=\widehat{H}\left(X\right)-\widehat{H}(X\cap Y)$.

Figure 3. A single realization of the inclusion–exclusion principle for three variables. The new region, corresponding to the distributivity-breaking difference is represented via a checkered pattern. Covering numbers are written for each sector and highlighted by the colors. This is not a full Venn-type diagram that defines the information atoms, and hence, its structure is clearly not invariant with respect to variable permutations.

Figure 3. A single realization of the inclusion–exclusion principle for three variables. The new region, corresponding to the distributivity-breaking difference is represented via a checkered pattern. Covering numbers are written for each sector and highlighted by the colors. This is not a full Venn-type diagram that defines the information atoms, and hence, its structure is clearly not invariant with respect to variable permutations.

3.2. Set-Theoretic Solution

Before going to arbitrary variables, consider a system where distributivity axiom holds. Under such condition the setup becomes effectively equivalent to set theory. A trivariate system can, therefore, be illustrated by the same Venn diagram as that of three sets:

$$\begin{array}{c}H\left(X_1\right)={\Pi }_{\left\{1\right\}}+{\Pi }_{\left\{1\right\}\left\{2\right\}}+{\Pi }_{\left\{1\right\}\left\{3\right\}}+{\Pi }_{\left\{1\right\}\left\{2\right\}\left\{3\right\}},\hfill \\ H\left(X_2\right)={\Pi }_{\left\{2\right\}}+{\Pi }_{\left\{1\right\}\left\{2\right\}}+{\Pi }_{\left\{2\right\}\left\{3\right\}}+{\Pi }_{\left\{1\right\}\left\{2\right\}\left\{3\right\}},\hfill \\ H\left(X_3\right)={\Pi }_{\left\{3\right\}}+{\Pi }_{\left\{1\right\}\left\{3\right\}}+{\Pi }_{\left\{2\right\}\left\{3\right\}}+{\Pi }_{\left\{1\right\}\left\{2\right\}\left\{3\right\}},\hfill \\ H(X_1,X_2)={\Pi }_{\left\{1\right\}}+{\Pi }_{\left\{2\right\}}+{\Pi }_{\left\{1\right\}\left\{2\right\}}+{\Pi }_{\left\{1\right\}\left\{3\right\}}+{\Pi }_{\left\{2\right\}\left\{3\right\}}+{\Pi }_{\left\{1\right\}\left\{2\right\}\left\{3\right\}},\hfill \\ H(X_1,X_3)={\Pi }_{\left\{1\right\}}+{\Pi }_{\left\{3\right\}}+{\Pi }_{\left\{1\right\}\left\{2\right\}}+{\Pi }_{\left\{1\right\}\left\{3\right\}}+{\Pi }_{\left\{2\right\}\left\{3\right\}}+{\Pi }_{\left\{1\right\}\left\{2\right\}\left\{3\right\}},\hfill \\ H(X_2,X_3)={\Pi }_{\left\{2\right\}}+{\Pi }_{\left\{3\right\}}+{\Pi }_{\left\{1\right\}\left\{2\right\}}+{\Pi }_{\left\{1\right\}\left\{3\right\}}+{\Pi }_{\left\{2\right\}\left\{3\right\}}+{\Pi }_{\left\{1\right\}\left\{2\right\}\left\{3\right\}},\hfill \\ H(X_1,X_2,X_3)=\hfill \\ ={\Pi }_{\left\{1\right\}}+{\Pi }_{\left\{2\right\}}+{\Pi }_{\left\{3\right\}}+{\Pi }_{\left\{1\right\}\left\{2\right\}}+{\Pi }_{\left\{1\right\}\left\{3\right\}}+{\Pi }_{\left\{2\right\}\left\{3\right\}}+{\Pi }_{\left\{1\right\}\left\{2\right\}\left\{3\right\}}\hfill \end{array}$$

(29)

By calculating the sizes of atoms, we derive (Appendix B, (A29)) the criterion for their non-negativity: the whole must be less or equal to the sum of the parts:

$$\begin{array}{c}I(X_1,X_2;X_3)-I(X_1;X_3)-I(X_2;X_3)\le 0\hfill \end{array}$$

(30)

3.3. Main Result: Arbitrary Trivariate System

At this point, we have studied two opposite cases: a completely synergic system (XOR gate) and one without any synergy (set-theoretic solution). To describe three arbitrary variables, any general decomposition must be able to replicate both of them. It turns out that a combination of the already known atoms (Figure 4) suffices in providing a non-negative decomposition (presented in detail in Appendix B (A47); for the proof see Lemma A6):

$$\begin{array}{c}H\left(X_i\right)={\Pi }_s+\sum _{\mathrm{set}‐\mathrm{theor}.\mathrm{atoms}}\Pi ,\hfill \\ H(X_i,X_{j\ne i})={\Pi }_s+{\Pi }_g+\sum _{\mathrm{s}.\mathrm{t}.\mathrm{atoms}}\Pi ,\hfill \\ H(X_1,X_2,X_3)={\Pi }_s+{\Pi }_g+\sum _{\mathrm{s}.\mathrm{t}.\mathrm{atoms}}\Pi ,\hfill \\ {\Pi }_s={\Pi }_g\hfill \end{array}$$

(31)

This is the minimal solution to the problem as it contains the smallest set of necessary atoms. The whole and parts are now related by the difference of two terms:

$$\begin{array}{c}I((X_1,X_2);X_3)-I(X_1;X_3)-I(X_2;X_3)={\Pi }_s-{\Pi }_{\left\{1\right\}\left\{2\right\}\left\{3\right\}}⪋0\hfill \end{array}$$

(32)

We can gain major insight by substituting the left side using the inclusion–exclusion formulas (6) and (27):

$$\begin{array}{c}\Delta \widehat{H}-\widehat{H}(X_1\cap X_2\cap X_3)={\Pi }_s-{\Pi }_{\left\{1\right\}\left\{2\right\}\left\{3\right\}}\hfill \end{array}$$

(33)

Remember that the only 3-covered area in the system is $X_1\cap X_2\cap X_3$. Therefore, the size of ${\Pi }_s$ is determined by the distributivity-breaking difference:

$$\begin{array}{c}{\Pi }_{\left\{1\right\}\left\{2\right\}\left\{3\right\}}=\widehat{H}(X_1\cap X_2\cap X_3),\hfill \\ {\Pi }_s=\Delta \widehat{H}\hfill \end{array}$$

(34)

To find the physical meaning behind the recovered solution, we once again compare it to the partial information decomposition of the same system. Only four of the diagram regions (Figure 4) appear in the corresponding equations:

$$\begin{array}{c}I(X_1;X_3)={\Pi }_{\left\{1\right\}\left\{2\right\}\left\{3\right\}}+{\Pi }_{\left\{1\right\}\left\{3\right\}},\hfill \\ I(X_2;X_3)={\Pi }_{\left\{1\right\}\left\{2\right\}\left\{3\right\}}+{\Pi }_{\left\{2\right\}\left\{3\right\}},\hfill \\ I((X_1,X_2);X_3)={\Pi }_{\left\{1\right\}\left\{2\right\}\left\{3\right\}}+{\Pi }_s+{\Pi }_{\left\{1\right\}\left\{3\right\}}+{\Pi }_{\left\{2\right\}\left\{3\right\}}\hfill \end{array}$$

(35)

The result fully captures the structure behind Williams and Beer’s definitions [4]:

$$\begin{array}{c}{\Pi }_{\left\{1\right\}\left\{2\right\}\left\{3\right\}}\equiv \mathrm{Redundancy},\hfill \\ {\Pi }_{\left\{1\right\}\left\{3\right\}}\equiv \mathrm{Unique}\mathrm{information}\mathrm{in}{X}_1,\hfill \\ {\Pi }_{\left\{2\right\}\left\{3\right\}}\equiv \mathrm{Unique}\mathrm{information}\mathrm{in}{X}_2,\hfill \\ {\Pi }_s\equiv \mathrm{Synergy}\hfill \end{array}$$

(36)

We have, thus, shown how information synergy naturally follows from set-theoretic arguments. The synergistic contribution is contained in the entropy of the parts and is precisely equal to the distributivity-breaking difference $\Delta \widehat{H}$. The interaction responsible for the synergistic contribution is depicted on the Venn diagram as an intersection with unconventional covering number ${\Pi }_s$. Finally, the illusion of a whole being greater than the sum of its parts comes from the fact that the mutual information terms on the left-hand size of Equation (3) do not account for all regions of the Venn-diagram (Figure 4).

4. Towards a Multivariate Information Decomposition

In this section, we lay the foundation for a consistent theory of multivariate decomposition and resolve the contradictions between partial information decomposition axioms [15].

4.1. Information Atoms Based on Part–Whole Relations

To rigorously define the information atoms, we may think of them as basic pieces of information, which make up all more complex quantities. Previously, we have used the inclusion–exclusion principle to break down the entropy of the whole system into smaller parts step by step. Even without writing the formula for N variables, one can find the general form of the terms participating in this process:

$$\begin{array}{c}\Xi \left[C\right]=\bigcap \left(\bigcup X_i\right)\hfill \end{array}$$

(37)

The covering number C is defined trivially as the number of intersecting union-brackets in (37) and determines the sign of the associated term by the inclusion–exclusion principle. Similarly to the Möbius inversion used in set theory [25], the decomposition of non-distributive space will rely on the inclusion order lattice $(L^{\Xi },\subseteq )$ of terms $\Xi$. A general description of the decomposition through part–whole relations was proposed in [26] in the form of the parthood table. It is a matrix with entries 0 or 1, which define whether a given atom $\Pi$ is a part of a particular larger information piece, i.e., the inclusion–exclusion term (37):

$$\begin{array}{c}\widehat{H}\left({\Xi }_i\left[C_i\right]\right)=\sum _j{f}_{ij}{\Pi }_j\left[c_j\right],\hfill \\ f_{ij}=0,1\hfill \end{array}$$

(38)

The parthood table depends on the initial variables through the monotonicity axiom, or compliance with the inclusion lattice $(L^{\Xi },\subseteq )$:

$$\begin{array}{c}{\Xi }_i\subseteq {\Xi }_j\Rightarrow \forall k{f}_{ik}\le f_{jk}\hfill \end{array}$$

(39)

It relates the table’s entries within themselves by a simple rule: if one $\Xi$ term is included in the other, all the atoms from the decomposition of former should be present in the decomposition of the latter.

The summands $\Pi$ are non-negative functions and represent the sizes of atoms. The covering number $c_j$ of each atom is defined by the coverings of inclusion–exclusion terms $C_i$:

$$\begin{array}{c}{c}_j=\underset{i:f_{ij}=1}{max}{C}_i\hfill \end{array}$$

(40)

This rule remains unchanged from the classical set theory.

The information conservation law (17) is the final condition that preserves the physical meaning of the covering numbers—the number of times the same information appears in the system.

The existence of a general solution for N variables is not guaranteed. Besides, linear system (38) is undetermined for $N>2$. For a specific set of degenerate cases it is, however, still possible to calculate the sizes of all atoms. We will next list several such examples while specifying how information is distributed among their different parts.

• Set-Theoretic Solution for N Variables

In a distributive system, the solution is a particular case of Möbius inversion [25] (Appendix B, (A30)). Mutual information as a function of random variables becomes subadditive (Appendix B, Lemma A5) proving that the lack of distributivity is a necessary condition for emergence.

• XOR Gate

The solution found for the XOR gate is unique in the parthood table formalism (Appendix B, Theorem A2). This reinforces our proposal of synergistic and ghost atoms as physical entities.

• N-Parity

Generalizing the XOR gate to an arbitrary number of variables yields the N-parity setup. It allows a solution of the similar form (Appendix B, (A42)–(A46)):

$$\begin{array}{c}{\Pi }_s\left[2\right]=1\mathrm{bit},\hfill \\ {\Pi }_{g_{n=\overline{1,N-2}}}\left[1\right]=1\mathrm{bit},\hfill \\ \forall n,\sigma H(X_{\sigma \left(1\right)},X_{\sigma \left(2\right)},\cdots X_{\sigma \left(n\right)})={\Pi }_s+\sum _{i=2}^{n-1}{\Pi }_{g_i}\hfill \end{array}$$

(41)

4.2. Resolving the Partial Information Decomposition Self-Contradiction

The existence of any multivariate decomposition was previously believed to be disproved [15] by employing a simple example that could not be solved without discarding one of the partial information decomposition axioms. The information inside three XOR variables, $O_1,O_2,O_3$, about their joint variable $O_4=(O_1,O_2,O_3)$ was claimed to be grouped into three 1-bit synergistic atoms that, using our notation, corresponding to $O_1\cap (O_2\cup O_3)\cap O_4$, $O_2\cap (O_1\cup O_3)\cap O_4$, and $O_3\cap (O_1\cup O_2)\cap O_4$. These were summed up to give three bits of information—more than the total of two bits present in the entire system. The authors of [15] concluded that the non-negativity of information was not respected.

To resolve this discrepancy, first notice that partial information decomposition atoms are a subset of of the full set of atoms $\{\Pi \}$. In the system with N sources of information $X_1,\cdots X_N$ and target $X_{N+1}$ they lie inside the intersection $I(X_1,\cdots ,X_N;X_{N+1})=\widehat{H}((X_1\cup \cdots \cup X_N)\cap X_{N+1})$ and are defined by the submatrix of the full parthood table $f_{ij}:{\Xi }_i\subseteq (X_1\cup X_2\cup \cdots X_N)\cap X_{N+1}$. In particular, when the output is equal to the joint variable of all inputs, the entropy of inputs coincides with mutual information, and hence, all atoms appear in the partial information decomposition (Appendix C, Lemma A7). The set of atoms $\{\Pi \}$ itself is then identical to that of the system $X_1,\cdots X_N$ alone with the exception of all covering numbers being increased by one to comply with the additional cover of $X_{N+1}$ (Appendix C, Theorem A3). This is exactly the type of system that was used in [15]. Using the solution of the XOR gate, we find:

$$\begin{array}{c}{\Pi }_s\left[3\right]=1\mathrm{bit},\hfill \\ {\Pi }_g\left[2\right]=1\mathrm{bit},\hfill \\ I(O_1;O_4)={\Pi }_s,\hfill \\ I(O_2;O_4)={\Pi }_s,\hfill \\ I(O_3;O_4)={\Pi }_s,\hfill \\ I((O_1,O_2);O_4)={\Pi }_s+{\Pi }_g,\hfill \\ I((O_1,O_3);O_4)={\Pi }_s+{\Pi }_g,\hfill \\ I((O_2,O_3);O_4)={\Pi }_s+{\Pi }_g,\hfill \\ I((O_1,O_2,O_3);O_4)={\Pi }_s+{\Pi }_g\hfill \end{array}$$

(42)

In place of three, there is only one symmetric atom ${\Pi }_s\left[3\right]$. The confusion in [15] occurred since different forms of the inclusion–exclusion principle were considered separately and it was assumed that each version would create its own synergistic atom.

5. Discussion

Previous attempts for studying synergistic information using set-theoretic intuition have led to self-contradictions. In this work, we point out that the non-distributivity of random variables corresponds to a well-defined variant of set-theory. We employ our results to construct a Venn-like diagram for an arbitrary three-variable system and demonstrate how synergism to be a direct consequence of distributivity breaking.

Our results do not fully solve the problem at hand. First, precise calculation of atom sizes was left unanswered and might require a more explicit description of information intersections. Another caveat is that although we constructed the equations that describe a self-consistent multivariate information decomposition, the existence of a solution for N arbitrary random variables is yet to be proven.

Nevertheless, this work lays the basis for a self-consistent multivariate theory. Our analysis reestablishes the concept of information decompositions as a foundation for further enquiry in quantifying emergence. In this context, information theory serves as a mere illustration: the mechanism we describe offers an explanation of the nature of synergy which uses solely set-theoretic concepts and can be applied to any emergent physical system.

From the physical standpoint, synergistic properties of information are a consequence of entropy reordering inside the system of inputs and outputs. However, this is only possible because the mathematical entities under consideration (discrete random variables) possess the property of subdistributivity, whose origin and interpretation in terms of the underlying physical system is yet to be found. One could also take a different function to represent the size of random variables. This might lead to additional positive (synergic) or negative (redundant) contributions and requires further investigation. Examples of measures other than entropy that still obey set-theoretic logic are discussed in [21].