Multivariate Dependence beyond Shannon Information

Abstract

Accurately determining dependency structure is critical to understanding a complex system’s organization. We recently showed that the transfer entropy fails in a key aspect of this—measuring information flow—due to its conflation of dyadic and polyadic relationships. We extend this observation to demonstrate that Shannon information measures (entropy and mutual information, in their conditional and multivariate forms) can fail to accurately ascertain multivariate dependencies due to their conflation of qualitatively different relations among variables. This has broad implications, particularly when employing information to express the organization and mechanisms embedded in complex systems, including the burgeoning efforts to combine complex network theory with information theory. Here, we do not suggest that any aspect of information theory is wrong. Rather, the vast majority of its informational measures are simply inadequate for determining the meaningful relationships among variables within joint probability distributions. We close by demonstrating that such distributions exist across an arbitrary set of variables.

1. Introduction

Information theory is a general, broadly applicable framework for understanding a system’s statistical properties [1]. Due to its focus on probability distributions, it allows one to compare dissimilar systems (e.g., species abundance to ground state configurations of a spin system) and has found many successes in the physical, biological and social sciences [2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19] far outside its original domain of communication. Often, the issue on which it is brought to bear is discovering and quantifying dependencies [20,21,22,23,24,25]. Here, we define a dependency to be any deviation from statistical independence. It is possible for a single multivariate distribution to consist of many, potentially overlapping, dependencies. Consider the simple case of three variables X, Y, Z, where X and Y are coin flips and Z is their concatenation. We would say here that there are two dependencies: an $XZ$ dependency and a $YZ$ dependency. It is important to note that, though there are some similarities, this notion of a dependency is distinct from that used within the Bayesian network community.

The past two decades, however, produced a small, but important body of results detailing how standard Shannon information measures are unsatisfactory for determining some aspects of dependency and shared information. Within information-theoretic cryptography, the conditional mutual information has proven to be a poor bound on secret key agreement [26,27]. The conditional mutual information has also been shown to be unable to accurately measure information flow ([28] and references therein). Finally, the inability of standard methods of decomposing the joint entropy to provide any semantic understanding of how information is shared has motivated entirely new methods of decomposing information [29,30]. Common to all these is the fact that conditional mutual information conflates intrinsic dependence with conditional dependence. To be clear, the conditional mutual information between X and Y given Z cannot distinguish the case where X and Y are related ignoring Z (intrinsic dependence) from the case where X and Y are related due to the influence of Z (conditional dependence).

Here, we demonstrate a related, but deeper issue: Shannon information measures—entropy, mutual information and their conditional and multivariate versions—can fail to distinguish joint distributions with vastly differing internal dependencies.

Concretely, we start by constructing two joint distributions, one with dyadic sub-dependencies and the other with strictly triadic sub-dependencies. From there, we demonstrate that no standard Shannon-like information measure, and exceedingly few nonstandard methods, can distinguish the two. Stated plainly: when viewed through Shannon’s lens, these two distributions are erroneously equivalent. While distinguishing these two (and their internal dependencies) may not be relevant to a mathematical theory of communication, it is absolutely critical to a mathematical theory of information storage, transfer and modification [31,32,33,34]. We then demonstrate two ways in which these failures generalize to the multivariate case. The first generalizes our two distributions to the multivariate and polyadic case via “dyadic camouflage”. The second details a method of embedding an arbitrary distribution into a larger variable space using hierarchical dependencies, a technique we term “dependency diffusion”. In this way, one sees that the initial concerns about information measures can arise in virtually any statistical multivariate analysis. In this short development, we assume a working knowledge of information theory, such as found in standard textbooks [35,36,37,38].

2. Development

We begin by considering the two joint distributions shown in

Table 1. The (a) dyadic and (b)triadic probability distributions over the three random variables X, Y and Z that take values in the four-letter alphabet $\{0,1,2,3\}$. Though not directly apparent from their tables of joint probabilities, the dyadic distribution is built from dyadic (pairwise) sub-dependencies, while the triadic from triadic (three-way) sub-dependencies.

(a) Dyadic				(b) Triadic
X	Y	Z	Pr	X	Y	Z	Pr
0	0	0	1/8	0	0	0	1/8
0	2	1	1/8	1	1	1	1/8
1	0	2	1/8	0	2	2	1/8
1	2	3	1/8	1	3	3	1/8
2	1	0	1/8	2	0	2	1/8
2	3	1	1/8	3	1	3	1/8
3	1	2	1/8	2	2	0	1/8
3	3	3	1/8	3	3	1	1/8

. The first represents dyadic relationships among three random variables X, Y, and Z. Additionally, the second, the triadic relationships among them. (This distribution was first considered as RDNXOR in [39], though for other, but related reasons.) These appellations are used for reasons that will soon be apparent. How are these distributions structured? Are they structured identically or are they qualitatively distinct? It is clear from inspection that they are not identical, but a lack of isomorphism is less obvious.

We can develop a direct picture of underlying dependency structure by casting the random variables’ four-symbol alphabet used in Table 1 into composite binary random variables, as displayed in

Table 2. Expansion of the (a) dyadic and (b) triadic distributions. In both cases, the variables from Table 1 were interpreted as two binary random variables, translating, e.g., $X=3$ into $(X_0,X_1)=(1,1)$. In this light, it becomes apparent that the dyadic distribution consists of the sub-dependencies $X_0=Y_1$, $Y_0=Z_1$ and $Z_0=X_1$, while the triadic distribution consists of $X_0+Y_0+Z_0=0mod2$ and $X_1=Y_1=Z_1$. These relationships are pictorially represented in Figure 1.

(a) Dyadic											(b) Triadic
X			Y			Z					X			Y			Z
X0	X1		Y0	Y1		Z0	Z1		Pr		X0	X1		Y0	Y1		Z0	Z1		Pr
0	0		0	0		0	0		1/8		0	0		0	0		0	0		1/8
0	0		1	0		0	1		1/8		0	1		0	1		0	1		1/8
0	1		0	0		1	0		1/8		0	0		1	0		1	0		1/8
0	1		1	0		1	1		1/8		0	1		1	1		1	1		1/8
1	0		0	1		0	0		1/8		1	0		0	0		1	0		1/8
1	0		1	1		0	1		1/8		1	1		0	1		1	1		1/8
1	1		0	1		1	0		1/8		1	0		1	0		0	0		1/8
1	1		1	1		1	1		1/8		1	1		1	1		0	1		1/8

. It can be readily verified that the dyadic distribution follows three simple rules: $X_0=Y_1$, $Y_0=Z_1$ and $Z_0=X_1$; in particular, three dyadic rules. The triadic distribution similarly follows simple rules: $X_0+Y_0+Z_0=0mod2$ (the xor relation [40], or equivalently, any one of them is the xor of the other two), and $X_1=Y_1=Z_1$; two triadic rules.

While this expansion to binary sub-variables is not unique, it is representative of the distributions. One could expand the dyadic distribution, for example, in such a way that some of the sub-variables would be related by xor. However, those same sub-variables would necessarily be involved in other relationships, limiting their expression in a manner similar to that explored in [41]. This differs from our triadic distribution in that its two sub-dependencies are independent. That these binary expansions are, in fact, representative and that the triadic distribution cannot be written in a way that relies only on dyadic relationships can be seen in the connected information explored later in the section. For the dyadic distribution, there is no difference between the maximum entropy distribution constraining pairwise interactions from the distribution itself. However, the maximum entropy distribution obtained by constraining the pairwise interactions in the triadic distribution has a larger entropy than the triadic distribution itself, implying that there is structure that exists beyond the pairwise interactions.

These dependency structures are represented pictorially in Figure 1. Our development from this point on will not use any knowledge of these structures, but rather, it will attempt to distinguish the structures using only information measures.

What does an information-theoretic analysis say? Both the dyadic and triadic distributions describe events over three variables, each with an alphabet size of four. Each consists of eight joint events, each with a probability of 1/8. As such, each has a joint entropy of $\mathrm{H}[X,Y,Z]=3$ bit (The SI standard unit for time is the second, and its symbol is s; analogously, the standard (IEC 60027-2, ISO/IEC 80000-13) unit for information is the bit, and its symbol is bit. As such, it is inappropriate to write 3 bits, just as it would be inappropriate to write 3 ss). Our first observation is that any entropy—conditional or not—and any mutual information—conditional or not—will be identical for the two distributions. Specifically, the entropy of any variable conditioned on the other two vanishes: $\mathrm{H}[X\mid Y,Z]=\mathrm{H}[Y\mid X,Z]=\mathrm{H}[Z\mid X,Y]=0$ bit; the mutual information between any two variables conditioned on the third is unity: $\mathrm{I}[X:Y\mid Z]=\mathrm{I}[X:Z\mid Y]=\mathrm{I}[Y:Z\mid X]=1$ bit; and the three-way co-information also vanishes: $\mathrm{I}[X:Y:Z]=0$ bit. These conclusions are compactly summarized in the form of the information diagrams (I-diagrams) [42,43] shown in Figure 2. This diagrammatically represents all of the possible Shannon information measures (I-measures) [43] of the distribution: effectively, all the multivariate extensions of the standard Shannon measures, called atoms. It is important to note that the analogy between information theory and set theory should not be taken too far: while set cardinality is strictly nonnegative, information atoms need not be; see [38] for more details. The values of the information atoms are identical between the two distributions.

As a brief aside, it is of interest to note that it has been suggested (e.g., in [44,45], among others) that zero co-information implies that at least one variable is independent of the others; that is, in this case, a lack of three-way interactions. Krippendorff [46] early on demonstrated that this is not the case, though these examples more clearly exemplify this fact.

We now turn to the implications of the two information diagrams, Figure 2a,b, being identical. There are measures [20,22,44,47,48,49,50,51,52,53] and expansions [54,55,56] purporting to measure or otherwise extract the complexity, magnitude or structure of dependencies within a multivariate distribution. Many of these techniques, including those just cited, are sums and differences of atoms in these information diagrams. As such, they are unable to differentiate these distributions.

To drive home the point that the concerns raised here are very broad,

Table 3. Suite of information measures applied to the dyadic and triadic distributions, where: H [•] is the Shannon entropy [35], H₂ [•] is the order-2 Rényi entropy [61], S_q [•] is the Tsallis entropy [62], I [•] is the co-information [44], T [•] is the total correlation [47], B [•] is the dual total correlation [48,63], J [•] is the CAEKL mutual information [49], II [•] is the interaction information [64], K [•] is the Gács–Körner common information [57], C [•] is the Wyner common information [65,66], G [•] is the exact common information [67], F [•] is the functional common information,${}^{\mathrm{a}}$ M [•] is the MSS common information,${}^{\mathrm{b}}$ I [• ↓ •] is the intrinsic mutual information [26],${}^{\mathrm{c}}$ I [• ⇓ •] is the reduced intrinsic mutual information [27],${}^{\mathrm{c},\mathrm{d}}$ X [•] is the extropy [68], R [•] is the residual entropy or erasure entropy [60,63], P [•] is the perplexity [69], D [•] is the disequilibrium [51], C_LMRP [•] is the LMRP complexity [51] and TSE [•] is the TSE complexity [59]. Only the Gács–Körner common information and the intrinsic mutual information, highlighted, are able to distinguish the two distributions; the Gács–Körner common information via the construction of a sub-variable ($X_1=Y_1=Z_1$) common to X, Y and Z and the intrinsic mutual information via the relationship $X_0=Y_1$ being independent of Z.

Measures		Dyadic		Triadic
H [X,Y,Z]		3 bit		3 bit
H2 [X,Y,Z]		3 bit		3 bit
S2 [X,Y,Z]		0.875 bit		0.875 bit
I [X:Y:Z]		0 bit		0 bit
T [X:Y:Z]		3 bit		3 bit
B [X:Y:Z]		3 bit		3 bit
J [X:Y:Z]		1.5 bit		1.5 bit
II [X:Y:Z]		0 bit		0 bit
K [X:Y:Z]		0 bit		1 bit
C [X:Y:Z]		3 bit		3 bit
G [X:Y:Z]		3 bit		3 bit
F [X:Y:Z] ${}^{\mathrm{a}}$		3 bit		3 bit
M [X:Y:Z] ${}^{\mathrm{b}}$		3 bit		3 bit
I [X:Y↓Z] ${}^{\mathrm{c}}$		1 bit		0 bit
I [X:Y⇓Z] ${}^{\mathrm{c},\mathrm{d}}$		1 bit		0 bit
X [X,Y,Z]		1.349 bit		1.349 bit
R [X:Y:Z]		0 bit		0 bit
P [X,Y,Z]		8		8
D [X,Y,Z]		0.761 bit		0.761 bit
CLMRP [X,Y,Z]		0.381 bit		0.381 bit
TSE [X:Y:Z]		2 bit		2 bit

${}^{\mathrm{a}}{\displaystyle \mathrm{F}\left[\right\{X_i\left\}\right]=\underset{\begin{array}{c}ￒX_i|V\\ V=f\left(\left\{X_i\right\}\right)\end{array}}{min}\mathrm{H}[V]}$, where $ￒX_i|V$ means that the $X_i$ are conditionally independent given V. ${}^{\mathrm{b}}{\displaystyle \mathrm{M}\left[\right\{X_i\left\}\right]=\mathrm{H}[\vee (X_i\searrow X_{\overline{i}})}$, where $X\searrow Y$ is the minimal sufficient statistic [35] of X about Y, and ∨ denotes the informational union of variables. ${}^{\mathrm{c}}$ Though this measure is generically dependent on which variable(s) is chosen to be conditioned on, due to the symmetry of the dyadic and triadic distributions, the values reported here are insensitive to permutations of the variables. ${}^{\mathrm{d}}$ The original work [27] used the slightly more verbose notation I[• ↓↓ •].

enumerates the result of applying a great many information measures to this pair of distributions. It is organized from top to bottom into four sections: entropies, mutual information, common information and other measures.

None of the entropies, dependent only on the probability mass function of the distribution, can distinguish the two distributions. Nor can any of the mutual information, as they are functions of the information atoms in the I-diagrams of Figure 2.

The common information, defined via auxiliary variables satisfying particular properties, can potentially isolate differences in the dependencies. Though only one of them—the Gács–Körner common information K [•] [57,58], involving the construction of the largest “subrandom variable” common to the variables—discerns that the two distributions are not equivalent because the triadic distribution contains the subrandom variable $X_1=Y_1=Z_1$ common to all three variables.

Finally, only two of the other measures identify any difference between the two. Some fail because they are functions of the probability mass function. Others, like the TSE complexity [59] and erasure entropy [60], fail since they are functions of the I-diagram atoms. Only the intrinsic mutual information I [• ↓ •] [26] and the reduced mutual information I [• ⇓ •] [27] distinguish the two since the dyadic distribution contains three dyadic sub-variables each of which is independent of the third variable, whereas in the triadic distribution, the conditional dependence of the xor relation can be destroyed.

Figure 3 demonstrates three different information expansions—that, roughly speaking, group variables into subsets of difference sizes or “scales”—applied to our distributions of interest. The first is the complexity profile [55]. At scale k, the complexity profile is the sum of all I-diagram atoms consisting of at least k variables conditioned on the others. Here, since the I-diagrams are identical, so are the complexity profiles. The second profile is the marginal utility of information [56], which is a derivative of a linear programming problem whose constraints are given by the I-diagram, so here, again, they are identical. Finally, we have Schneidman et al.’s connected information [70], which comprise the differences in entropies of the maximum entropy distributions whose k- and $k-1$-way marginals are fixed to match those of the distribution of interest. Here, all dependencies are detected once pairwise marginals are fixed in the dyadic distribution, but it takes the full joint distribution to realize the xor sub-dependency in the triadic distribution.

While it is well known that causality cannot be determined from probability distributions alone [71], here we point out a related, though different issue. While causality is, in some sense, the determination of the precedence within a dependency, the results above demonstrate that many measures of Granger-like causality are insensitive to the order (dyadic, triadic, etc.) of a dependency (Note that here we do not demonstrate that the order of a dependency cannot be determined from the probability distribution, as Pearl has done for causality [71]. Rather, our demonstration is limited to Shannon-like information measures.). Neither the transfer entropy [20], the transinformation [53], the directed information [52], the causation entropy [22], nor any of their generalizations based on conditional mutual information differentiate between intrinsic relationships and those induced by the variables they condition on (As discussed there, the failure of these measures stems from the possibility of conditional dependence, whereas the aim for these directed measures is to quantify the information flow from one time series to another excluding the influences of the second. In this light, we propose $T_{X\to Y}^{\prime }=\mathrm{I}[X_0^t:Y_t\downarrow Y_0^t]$ [26] as an incremental improvement over the transfer entropy). This limitation underlies prior criticisms of these functions as measures of information flow [28]. Furthermore, separating out these contributions to the transfer entropy has been discussed in the context of the partial information decomposition in [72].

A promising approach to understanding informational dependencies is the partial information decomposition (PID) [29]. This framework seeks to decompose a mutual information of the form I[(I₀,I₁):O] into four nonnegative components: the information R that both inputs $I_0$ and $I_1$ redundantly provide the output O, the information $U_0$ that $I_0$ uniquely provides O, the information $U_1$ that $I_1$ uniquely provides O and, finally, the information S that both $I_0$ and $I_1$ synergistically or collectively provide O.

Under this decomposition, our two distributions take on very different characteristics (Here, we quantified the partial information lattice using the best-in-class technique of [73], though calculations using three other techniques [74,75,76] match. There is a recent debate suggesting that the measure of Bertschinger et al. is not in fact correct, but it is likely, due to the agreement among these measures, that any “true” measure of redundancy would result in the same decomposition. The original PID measure $I_{\min }$, however, assigns both distributions: 1 bit of redundant information and 1 bit of synergistic information.). For both, the decomposition is invariant as far as which variables are selected as $I_0$, $I_1$ and O. For the dyadic distribution, PID identifies both bits in I[(I₀,I₁):O] as unique, one from each input $I_i$, corresponding to the dyadic sub-dependency shared by $I_i$ and O. Orthogonally, for the triadic distribution PID identifies one of the bits as redundant, stemming from $X_1=Y_1=Z_1$, and the other as synergistic, resulting from the xor relation among $X_0$, $Y_0$ and $Z_0$. These decompositions are displayed pictorially in Figure 4.

Another somewhat similar approach is that of integrated information theory [77]. However, this approach requires a known dynamic over the variables and is, in addition, highly sensitive to the dynamic. Here, in contrast, we considered only simple probability distributions without any assumptions as to how they might arise from the dynamics of interacting agents. That said, one might associate an integrated information measure with a distribution via the maximal information integration over all possible dynamics that give rise to the distribution. We leave this task for a later study.

3. Discussion

The broad failure of Shannon information measures to differentiate the dyadic and polyadic distributions has far-reaching consequences. Consider, for example, an experiment where a practitioner places three probes into a cluster of neurons, each probe touching two neurons and reporting zero when they are both quiescent, one when the first is excited but the second quiescent, two when the second is excited, but the first quiescent, and three when both are excited. Shannon-like measures—including the transfer entropy and related measures—would be unable to differentiate between the dyadic situation consisting of three pairs of synchronized neurons, the triadic situation consisting of a trio of synchronized neurons and a trio exhibiting the xor relation, a relation requiring nontrivial sensory integration. Such a situation might arise when probing the circuitry of the Drosophila melanogaster connectome [78], for instance.

Furthermore, while partitioning each variable into sub-variables made the dependency structure clear, we do not believe that such a refinement should be a necessary step in discovering such a structure. Consider that we demonstrated that refinement is not strictly needed, since the partial information decomposition (as quantified using current techniques) was able to discover the distribution’s internal structure without it.

These results, observations and the broad survey clearly highlight the need to extend Shannon’s theory. In particular, the extension must introduce a fundamentally new measure, not merely sums and differences of the standard Shannon information measures. While the partial information decomposition was initially proposed to overcome the interpretational difficulty of the (potentially negative valued) co-information, we see here that it actually overcomes a vastly more fundamental weakness with Shannon information measures. While negative information atoms can subjectively be seen as a flaw, the inability to distinguish dyadic from polyadic relations is a much deeper and objective issue.

This may lead one to consider the partial information decomposition as the needed extension to Shannon theory. As it currently stands, we do not. The partial information decomposition depends on interpreting some random variables as “inputs” and others as “outputs”. While this may be perfectly natural in some contexts, it is not satisfactory in general. It is possible that, were an agreeable multivariate partial information measure to be developed, the decomposition of, e.g., $\mathrm{I}\left[\right(X_0,X_1,X_2):X_0{X}_1{X}_2]$ could lead to a satisfactory symmetric decomposition. In any case, there has been longstanding interest in creating a symmetric decomposition analogous to the partial information decomposition [46] with some recent progress [79,80,81].

4. Dyadic Camouflage and Dependency Diffusion

The dyadic and triadic distributions we analyzed thus far were deliberately chosen to have small dimensionality in an effort to make them and the failure of Shannon information measures as comprehensible and intuitive as possible. Since a given dataset may have exponentially many different three-variable subsets, even just the two distributions examined here represent hurdles that information-based methods of dependency assessment must overcome. However, this is simply a starting point. We will now demonstrate that there exist distributions of arbitrary size whose k-way dependencies are masked, meaning the k-way co-information ($k\ge 3$) are all zero, and so, from the perspective of Shannon information theory, are indistinguishable from a distribution of the same size containing only dyadic relationships. Furthermore, we show how any such distribution may be obfuscated over any larger set of variables. This likely mandates a search over all partitions of all subsets of a system, making the problem of finding such distributions in the EXPTIME computational complexity class [82], meaning such a procedure will take time exponential in the size of the distribution.

Specifically, consider the four-variable parity distribution consisting of four binary variables such that $X_0+X_1+X_2+X_3=0mod2$. This is a straightforward generalization of the xor distribution used in constructing the triadic distribution. We next need a generalization of the “giant bit” [63], which we call dyadic camouflage, to mix with the parity, informationally “canceling out” the higher-order mutual information even though dependencies of such orders exist in the distribution. An example dyadic camouflage distribution for four variables is given in Figure 5.

Generically, consider an n-variable parity distribution, that is a distribution where $\sum X_i=0mod2$. It has an associated n-variable dyadic camouflage distribution with an alphabet size for each random variable of $2^{n-2}$, and the entire joint distribution consists of $2^{\frac{(n-2)·(n-1)}{2}}$ equally likely outcomes, both numbers determined due to entropy considerations. Specifically, in a parity distribution, each variable has 1 bit of entropy, and when mixed with its camouflage, it should have $n-1$ bits. Therefore, each variable in the camouflage distribution needs $n-2$ bits of entropy and needs, with uniform probability over those outcomes, $2^{n-2}$ characters in the alphabet. Furthermore, since the parity distribution itself has $n-1$ bits total, while its camouflaged form will have n choose two (=$(n-2)·(n-1)/2$) bits and, again with uniform probability, there must be $2^{(n-2)·(n-1)/2}$ outcomes. The distribution is constrained such that any two variables are completely determined by the remaining $n-2$. Moreover, each m-variable ($m<n$) sub-distribution consists of m mutually independent random variables.

The goal, then, is to construct such a distribution. One method of doing so is to begin by writing down one variable in increasing lexicographic order such that it has the correct number of outcomes; e.g., column W in Figure 5a. Then, find $n-1$ permutations of this column such that any two columns are determined from the remaining $n-2$. While such a search may be difficult, a distribution with these properties provably exists [83].

Finally, one can obfuscate any distribution by embedding it in a larger collection of random variables. Given a distribution D over n variables, associate each random variable i of D with a k-variable subset of a distribution $D^{\prime }$ in such a way that there is a mapping from the k-outcomes in the subset of $D^{\prime }$ to the outcome of the variable i in D. For example, one can embed the xor distribution over $X,Y,Z$ into six variables $X_0,X_1,Y_0,Y_1,Z_0,Z_1$ via $X_0\oplus X_1=X$, $Y_0\oplus Y_1=Y$ and $Z_0\oplus Z_1=Z$. In other words, the parity of $(Z_0,Z_1)$ is equal to the xor of the parities of $(X_0,X_1)$ and $(Y_0,Y_1)$. In this way, one must potentially search over all partitions of all subsets of $D^{\prime }$ in order to discover the distribution D hiding within. We refer to this method of obfuscation as dependency diffusion.

The first conclusion is that the challenges of conditional dependence can be found in joint distributions over arbitrarily large sets of random variables. The second conclusion, one that heightens the challenge to discovery, is that even finding which variables are implicated in polyadic dependencies can be exponentially difficult. Together, the camouflage and diffusion constructions demonstrate how challenging it is to discover, let alone work with, multivariate dependencies. This difficulty strongly implies that the current state of information-theoretic tools is vastly underpowered for the types of analyses required of our modern, data-rich sciences.

It is unlikely that the parity plus dyadic camouflage distribution discussed here is the only example of Shannon measures conflating the arity of dependencies and thus producing an information diagram identical to that of a qualitatively distinct distribution. This suggests an important challenge: find additional, perhaps simpler, joint distributions exhibiting this phenomenon.

5. Conclusions

To conclude, we constructed two distributions that cannot be distinguished using conventional (and many non-conventional) Shannon-like information measures. In fact, of the more than two dozen measures we surveyed, only five were able to separate the distributions: the Gács–Körner common information, the intrinsic mutual information, the reduced intrinsic mutual information, the connected information and the partial information decomposition.

The failure of the Shannon-type measures is perhaps not surprising: nothing in the standard mathematical theories of information and communication suggests that such measures should be able to distinguish these distributions [84]. However, distinguishing dependency structures such as dyadic from triadic relationships is of the utmost importance to the sciences; consider for example determining multi-drug interactions in medical treatments. Critically, since interpreting dependencies in random distributions is traditionally the domain of information theory, we propose that new extensions to information theory are needed.

These results may seem like a deal-breaking criticism of employing information theory to determine dependencies. Indeed, these results seem to indicate that much existing empirical work and many interpretations have simply been wrong and, worse even, that the associated methods are misleading while appearing quantitatively consistent. We think not, though. With the constructive and detailed problem diagnosis given here, at least we can see this issue. It is now a necessary step to address it. This leads us to close with a cautionary quote:

“The tools we use have a profound (and devious!) influence on our thinking habits, and, therefore, on our thinking abilities”.

(Edsger W. Dijkstra [85])