Coarse-Graining and the Blackwell Order

Abstract

Suppose we have a pair of information channels, ${\kappa }_1,{\kappa }_2$, with a common input. The Blackwell order is a partial order over channels that compares ${\kappa }_1$ and ${\kappa }_2$ by the maximal expected utility an agent can obtain when decisions are based on the channel outputs. Equivalently, ${\kappa }_1$ is said to be Blackwell-inferior to ${\kappa }_2$ if and only if ${\kappa }_1$ can be constructed by garbling the output of ${\kappa }_2$. A related partial order stipulates that ${\kappa }_2$ is more capable than ${\kappa }_1$ if the mutual information between the input and output is larger for ${\kappa }_2$ than for ${\kappa }_1$ for any distribution over inputs. A Blackwell-inferior channel is necessarily less capable. However, examples are known where ${\kappa }_1$ is less capable than ${\kappa }_2$ but not Blackwell-inferior. We show that this may even happen when ${\kappa }_1$ is constructed by coarse-graining the inputs of ${\kappa }_2$. Such a coarse-graining is a special kind of “pre-garbling” of the channel inputs. This example directly establishes that the expected value of the shared utility function for the coarse-grained channel is larger than it is for the non-coarse-grained channel. This contradicts the intuition that coarse-graining can only destroy information and lead to inferior channels. We also discuss our results in the context of information decompositions.

1. Introduction

Suppose we are given the choice of two channels that both provide information about the same random variable, and that we want to make a decision based on the channel outputs. Suppose that our utility function depends on the joint value of the input to the channel and our resultant decision based on the channel outputs. Suppose as well that we know the precise conditional distributions defining the channels, and the distribution over channel inputs. Which channel should we choose? The answer to this question depends on the choice of our utility function as well as on the details of the channels and the input distribution. So, for example, without specifying how we will use the channels, in general we cannot just compare their information capacities to choose between them.

Nonetheless, for certain pairs of channels we can make our choice, even without knowing the utility functions or the distribution over inputs. Let us represent the two channels by two (column) stochastic matrices ${\kappa }_1$ and ${\kappa }_2$, respectively. Then, if there exists another stochastic matrix $\lambda$ such that ${\kappa }_1=\lambda ·{\kappa }_2$, there is never any reason to strictly prefer ${\kappa }_1$; if we choose ${\kappa }_2$, we can always make our decision by chaining the output of ${\kappa }_2$ through the channel $\lambda$ and then using the same decision function we would have used had we chosen ${\kappa }_1$. This simple argument shows that whatever the three stochastic matrices are and whatever the decision rule we would use if we chose channel ${\kappa }_1$, we can always get the same expected utility by instead choosing channel ${\kappa }_2$ with an appropriate decision rule. In this kind of situation, where ${\kappa }_1=\lambda ·{\kappa }_2$, we say that ${\kappa }_1$ is a garbling (or degradation) of ${\kappa }_2$. It is much more difficult to prove that the converse also holds true:

Theorem 1.

(Blackwell’s theorem [1]) Let ${\kappa }_1,{\kappa }_2$ be two stochastic matrices representing two channels with the same input alphabet. Then the following two conditions are equivalent:

1. 

When the agent chooses ${\kappa }_2$ (and uses the decision rule that is optimal for ${\kappa }_2$), her expected utility is always at least as big as the expected utility when she chooses ${\kappa }_1$ (and uses the optimal decision rule for ${\kappa }_1$), independent of the utility function and the distribution of the input S.

2. 

${\kappa }_1$ is a garbling of ${\kappa }_2$.

Blackwell formulated his result in terms of a statistical decision maker who reacts to the outcome of a statistical experiment. We prefer to speak of a decision problem instead of a statistical experiment. See [2,3] for an overview.

Blackwell’s theorem motivates looking at the following partial order over channels ${\kappa }_1,{\kappa }_2$ with a common input alphabet:

$${\kappa }_1\le {\kappa }_2:⟺\left\{\begin{array}{c}\mathrm{one}\mathrm{of}\mathrm{the}\mathrm{two}\mathrm{statements}\hfill \\ \mathrm{in}\mathrm{Blackwell}\text{’}\mathrm{s}\mathrm{theorem}\mathrm{holds}\mathrm{true}.\hfill \end{array}\right.$$

We call this partial order the Blackwell order (this partial order is called degradation order by other authors [4,5]). If ${\kappa }_1\le {\kappa }_2$, then ${\kappa }_1$ is said to be Blackwell-inferior to ${\kappa }_2$. Strictly speaking, the Blackwell order is only a preorder, since there are channels ${\kappa }_1\ne {\kappa }_2$ that satisfy ${\kappa }_1\le {\kappa }_2\le {\kappa }_1$ (when ${\kappa }_1$ arises from ${\kappa }_2$ by permuting the output alphabet). However, for our purposes, such channels can be considered as equivalent. We write ${\kappa }_1<{\kappa }_2$ if ${\kappa }_1\le {\kappa }_2$ and ${\kappa }_1\ngeqq {\kappa }_2$. By Blackwell’s theorem, this implies that ${\kappa }_2$ performs at least as well as ${\kappa }_1$ in any decision problem and that there exist decision problems in which ${\kappa }_2$ outperforms ${\kappa }_1$.

For a given distribution of S, we can also compare ${\kappa }_1$ and ${\kappa }_2$ by comparing the two mutual informations $I(S;X_1)$, $I(S;X_2)$ between the common input S and the channel outputs $X_1$ and $X_2$. The data processing inequality shows that ${\kappa }_2\ge {\kappa }_1$ implies $I(S;X_2)\ge I(S;X_1)$. However, the converse implication does not hold. The intuitive reason is that for the Blackwell order, not only the amount of information is important. Rather, the question is how much of the information that ${\kappa }_1$ or ${\kappa }_2$ preserve is relevant for a given fixed decision problem (that is, a given fixed utility function).

Given two channels ${\kappa }_1,{\kappa }_2$, suppose that $I(S;X_2)\ge I(S;X_1)$ for all distributions of S. In this case, we say that ${\kappa }_2$ is more capable than ${\kappa }_1$. Does this imply that ${\kappa }_1\le {\kappa }_2$? The answer is known to be negative in general [6]. In Proposition 2 we introduce a new surprising example of this phenomenon with a particular structure. In fact, in this example, ${\kappa }_1$ is a Markov approximation of ${\kappa }_2$ by a deterministic function, in the following sense: Consider another random variable $f\left(S\right)$ that arises from S by applying a (deterministic) function f. Given two random variables S, X, denote by $X\leftarrow S$ the channel defined by the conditional probabilities $P_{X|S}\left(x\right|s)$, and let ${\kappa }_2:=(X\leftarrow S)$ and ${\kappa }_1:=(X\leftarrow f\left(S\right))·(f\left(S\right)\leftarrow S)$. Thus, ${\kappa }_1$ can be interpreted as first replacing S by $f\left(S\right)$ and then sampling X according to the conditional distribution $P_{X|S}\left(x\right|f\left(s\right))$. Which channel is superior? Using the data processing inequality, it is easy to see that ${\kappa }_1$ is less capable than ${\kappa }_2$. However, as Proposition 2 shows, in general ${\kappa }_1\nleqq {\kappa }_2$.

We call ${\kappa }_1$ a Markov approximation, because the output of ${\kappa }_1$ is independent of the input S given $f\left(S\right)$. The channel ${\kappa }_1$ can also be obtained from ${\kappa }_2$ by “pre-garbling” (Lemma 3); that is, there is another stochastic matrix ${\lambda }^f$ that satisfies ${\kappa }_1={\kappa }_2·{\lambda }^f$. It is known that pre-garbling may improve the performance of a channel (but not its capacity) as we recall in Section 2. What may be surprising is that this can happen for pre-garblings of the form ${\lambda }^f$, which have the effect of coarse-graining according to f.

The fact that the more capable preorder does not imply the Blackwell order shows that “Shannon information,” as captured by the mutual information, is not the same as “Blackwell information,” as needed for the Blackwell decision problems. Indeed, our example explicitly shows that even though coarse-graining always reduces Shannon information, it need not reduce Blackwell information. Finally, let us mention that there are further ways of comparing channels (or stochastic matrices); see [5] for an overview.

Proposition 2 builds upon another effect that we find paradoxical: Namely, there exist random variables $S,X_1,X_2$ and there exists a function $f:\mathcal{S}\to {\mathcal{S}}^{\prime }$ from the support of S to a finite set $S^{\prime }$ such that the following holds:

• S and $X_1$ are independent given $f\left(S\right)$.
• $(X_1\leftarrow f\left(S\right))\le (X_2\leftarrow f\left(S\right))$.
• $(X_1\leftarrow S)\nleqq (X_2\leftarrow S)$.

Statement (1) says that everything $X_1$ knows about S, it knows through $f\left(S\right)$. Statement (2) says that $X_2$ knows more about $f\left(S\right)$ than $X_1$. Still, (3) says that we cannot conclude that $X_2$ knows more about S than $X_1$. The paradox illustrates that it is difficult to formalize what it means to “know more.”

Understanding the Blackwell order is an important aspect of understanding information decompositions; that is, the quest to find new information measures that separate different aspects of the mutual information $I(S;X_1,\cdots ,X_k)$ of k random variables $X_1,\cdots ,X_k$ and a target variable S (see the other contributions of this special issue and references therein). In particular, [7] argues that the Blackwell order provides a natural criterion when a variable $X_1$ has unique information about S with respect to $X_2$. We hope that the examples we present here are useful in developing intuition on how information can be shared among random variables and how it behaves when applying a deterministic function, such as a coarse-graining. Further implications of our examples on information decompositions are discussed in [8]. In the converse direction, information decomposition measures (such as measures of unique information) can be used to study the Blackwell order and deviations from the Blackwell order. We illustrate this idea in Example 4.

The remainder of this work is organized as follows: In Section 2, we recall how pre-garbling can be used to improve the performance of a channel. We also show that the pre-garbled channel will always be less capable and that simultaneous pre-garbling of both channels preserves the Blackwell order. In Section 3, we state a few properties of the Blackwell order, and we explain why we find these properties counter-intuitive and paradoxical. In particular, we show that coarse-graining the input can improve the performance of a channel. Section 4 contains a detailed discussion of an example that illustrates these properties. In Section 5 we use the unqiue information measure from [7], which has properties similar to the Le Cam’s deficiency, to illustrate deviations from the Blackwell relation.

2. Pre-Garbling

As discussed above (and as made formal in Blackwell’s theorem (Theorem 1)), garbling the output of a channel (“post-garbling”) never increases the quality of a channel. On the other hand, garbling the input of a channel (“pre-garbling”) may increase the performance of a channel, as the following example shows.

Example 1.

Suppose that an agent can choose an action from a finite set $\mathcal{A}$. She then receives a utility $u(a,s)$ that depends both on the chosen action $a\in \mathcal{A}$ and on the value s of a random variable S. Consider the channels

$${\kappa }_1=\left(\begin{array}{cc}0.9& 0\\ 0.1& 1\end{array}\right)\mathit{and}{\kappa }_2={\kappa }_1·\left(\begin{array}{cc}0& 1\\ 1& 0\end{array}\right)=\left(\begin{array}{cc}0& 0.9\\ 1& 0.1\end{array}\right),$$

and the utility function

$$\begin{array}{cccc}s& 00& 1& 1\\ a& 01& 0& 1\\ u(s,a)& 20& 0& 1\end{array}$$

For uniform input, the optimal decision rule for ${\kappa }_1$ is

$$a\left(0\right)=0,a\left(1\right)=1$$

and the opposite

$$a\left(0\right)=1,a\left(1\right)=0$$

for ${\kappa }_2$. The expected utility with ${\kappa }_1$ is 1.4, while using ${\kappa }_2$, it is slightly higher (1.45).

It is also not difficult to check that neither of the two channels is a garbling of the other (cf. Propsition 3.22 in [5]).

The intuitive reason for the difference in the expected utilities is that the channel ${\kappa }_2$ transmits one of the states without noise and the other state with noise. With a convenient pre-processing, it is possible to make sure that the relevant information for choosing an action and for optimizing expected utility is transmitted with less noise.

Note the symmetry of the example: each of the two channels arises from the other by a convenient pre-processing, since the pre-processing is invertible. Hence, the two channels are not comparable by the Blackwell order. In contrast, two channels that only differ by an invertible garbling of the output are equivalent with respect to the Blackwell order.

The pre-garbling in Example 1 is invertible, and so it is more aptly described as a pre-processing. In general, though, pure pre-garbling and pure pre-processing are not easily distinguishable, and it is easy to perturb Example 1 by adding noise without changing the conclusion. In Section 3, we will present an example in which the pre-garbling consists of coarse-graining. It is much more difficult to understand how coarse-graining can be used as sensible pre-processing.

Even though pre-garbling can make a channel better (or, more precisely, more suited for a particular decision problem at hand), pre-garbling cannot invert the Blackwell order:

Lemma 1.

If ${\kappa }_1<{\kappa }_2·\lambda$, then ${\kappa }_1\ngeqq {\kappa }_2$.

Proof. 

Suppose that ${\kappa }_1<{\kappa }_2·\lambda$. Then the capacity of ${\kappa }_1$ is less than the capacity of ${\kappa }_2·\lambda$, which is bounded by the capacity of ${\kappa }_2$. Therefore, the capacity of ${\kappa }_1$ is less than the capacity of ${\kappa }_2$. ☐

Additionally, it follows directly from Blackwell’s theorem that

$${\kappa }_1\le {\kappa }_2\mathrm{implies}{\kappa }_1·\lambda \le {\kappa }_2·\lambda$$

for any channel $\lambda$, where the input and output alphabets of $\lambda$ equal the input alphabet of ${\kappa }_1,{\kappa }_2$. Thus, pre-garbling preserves the Blackwell order when applied to both channels simultaneously.

Finally, let us remark that certain kinds of simultaneous pre-garbling can also be “hidden” in the utility function; namely, in Blackwell’s theorem, it is not necessary to vary the distribution of $\mathcal{S}$ as long as the support of the (fixed) input distribution has full support S (that is, every state of the input alphabet of ${\kappa }_1$ and ${\kappa }_2$ appears with positive probability). In this setting, it suffices to look only at different utility functions. When the input distribution is fixed, it is more convenient to think in terms of random variables instead of channels, which slightly changes the interpretation of the decision problem. Suppose we are given random variables $S,X_1,X_2$ and a utility function $u(a,s)$ depending on the value of S and an action $a\in \mathcal{A}$ as above. If we cannot look at both $X_1$ and $X_2$, should we choose to look at $X_1$ or at $X_2$ to make our decision?

Theorem 2.

(Blackwell’s theorem for random variables [7]) The following two conditions are equivalent:

1. 

Under the optimal decision rule, when the agent chooses $X_2$, her expected utility is always at least as large as the expected utility when she chooses $X_1$, independent of the utility function.

2. 

$(X_1\leftarrow S)\le (X_2\leftarrow S)$.

3. Pre-Garbling by Coarse-Graining

In this section we present a few counter-intuitive properties of the Blackwell order.

Proposition 1.

There exist random variables $S,X_1,X_2$ and a function $f:\mathcal{S}\to \mathcal{S}$ from the support of S to a finite set $S^{\prime }$ such that the following holds:

1. 

S and $X_1$ are independent given $f\left(S\right)$.

2. 

$(X_1\leftarrow f\left(S\right))<(X_2\leftarrow f\left(S\right))$.

3. 

$(X_1\leftarrow S)\nleqq (X_2\leftarrow S)$.

This result may at first seem paradoxical. After all, property (3) implies that there exists a decision problem involving S for which it is better to use $X_1$ than $X_2$. Property (1) implies that any information that $X_1$ has about S is contained in $X_1$’s information about $f\left(S\right)$. One would therefore expect that, from the viewpoint of $X_1$, any decision problem in which the task is to predict S and to react on S looks like a decision problem in which the task is to react to $f\left(S\right)$. But property (2) implies that for such a decision problem, it may in fact be better to look at $X_2$.

Proof of Proposition 1.

The proof is by Example 2, which will be given in Section 4. This example satisfies

• S and $X_1$ are independent given $f\left(S\right)$.
• $(X_1\leftarrow f\left(S\right))\le (X_2\leftarrow f\left(S\right))$.
• $(X_1\leftarrow S)\nleqq (X_2\leftarrow S)$.

It remains to show that it is also possible to achieve the strict relation $(X_1\leftarrow f\left(S\right))<(X_2\leftarrow f\left(S\right))$ in the second statement. This can easily be done by adding a small garbling to the channel $X_1\leftarrow f\left(S\right)$ (e.g., by adding a binary symmetric channel with sufficiently small noise parameter $ϵ$). This ensures $(X_1\leftarrow f\left(S\right))<(X_2\leftarrow f\left(S\right))$, and if the garbling is small enough, this does not destroy the property $(X_1\leftarrow S)\nleqq (X_2\leftarrow S)$. ☐

The example from Proposition 1 also leads to the following paradoxical property:

Proposition 2.

There exist random variables $S,X$ and there exists a function $f:\mathcal{S}\to {\mathcal{S}}^{\prime }$ from the support of S to a finite set $S^{\prime }$ such that the following holds:

$$(X\leftarrow f(S\left)\right)·\left(f\right(S)\leftarrow S)\nleqq X\leftarrow S.$$

Let us again give a heuristic argument for why we find this property paradoxical. Namely, the combined channel $(X\leftarrow f(S\left)\right)·\left(f\right(S)\leftarrow S)$ can be seen as a Markov chain approximation of the direct channel $X\leftarrow S$ that corresponds to replacing the conditional distribution

$$P_{X|S}\left(x\right|s)=\sum _{f\left(s\right)}{P}_{X\left|Sf\right(S)}\left(x\right|s,f\left(s\right))P_{f\left(S\right)|S}\left(f\left(s\right)\right|s).$$

by

$$\sum _{f\left(s\right)}{P}_{X\left|f\right(S)}\left(x\right|f\left(s\right))P_{f\left(S\right)|S}\left(f\left(s\right)\right|s).$$

Proposition 2 together with Blackwell’s theorem states that there exist situations where this approximation is better than the correct channel.

Proof of Proposition 2.

Let $S,X_1,X_2$ be as in Example 2 in Section 4 that also proves Proposition 1, and let $X=X_2$. In that example, the two channels $X_1\leftarrow f\left(S\right)$ and $X_2\leftarrow f\left(S\right)$ are equal. Moreover, $X_1$ and S are independent given $f\left(S\right)$. Thus, $(X\leftarrow f\left(S\right))·(f\left(S\right)\leftarrow S)=(X_1\leftarrow S)$. Therefore, the statement follows from $(X_1\leftarrow S)\nleqq (X_2\leftarrow S)$. ☐

On the other hand, the channel $(X\leftarrow f(S\left)\right)·\left(f\right(S)\leftarrow S)$ is always less capable than $X\leftarrow S$:

Lemma 2.

For any random variables S, X, and function $f:\mathcal{S}\to \mathcal{S}$, the channel $(X\leftarrow f(S\left)\right)·\left(f\right(S)\leftarrow S)$ is less capable than $X\leftarrow S$.

Proof. 

For any distribution of S, let $X^{\prime }$ be the output of the channel $(X\leftarrow f(S\left)\right)·\left(f\right(S)\leftarrow S)$. Then, $X^{\prime }$ is independent of S given $f\left(S\right)$. On the other hand, since f is a deterministic function, $X^{\prime }$ is independent of $f\left(S\right)$ given S. Together, this implies $I(S;X^{\prime })=I(f\left(S\right);X^{\prime })$. Using the fact that the joint distributions of $(X,f(S\left)\right)$ and $(X^{\prime },f\left(S\right))$ are identical and applying the data processing inequality gives

$$I(S;X^{\prime })=I(f\left(S\right);X^{\prime })=I(f\left(S\right);X)\le I(S;X).☐$$

The setting of Proposition 2 can also be understood as a specific kind of pre-garbling. Namely, consider the channel ${\lambda }^f$ defined by

$${\lambda }_{s^{\prime },s}^f:=P_{S\left|f\right(S)}\left(s^{\prime }\right|f\left(s\right)).$$

The effect of this channel can be characterized as a randomization of the input: the precise value of S is forgotten, and only the value of $f\left(S\right)$ is preserved. Then, a new value $s^{\prime }$ is sampled for S according to the conditional distribution of S given $f\left(S\right)$.

Lemma 3.

$(X\leftarrow f\left(S\right))·(f\left(S\right)\leftarrow S)=(X\leftarrow S)·{\lambda }^f$.

Proof. 

$\begin{array}{ll}{\displaystyle \sum _{s_1}{P}_{X|S}\left(x\right|s_1)P_{S\left|f\right(S)}\left(s_1\right|f\left(s\right))}& {\displaystyle =\sum _{s_1,t}{P}_{X|S}\left(x\right|s_1)P_{S\left|f\right(S)}\left(s_1\right|t)P_{f\left(S\right)|S}\left(t\right|s)}\\ & ={\displaystyle \sum _t}{P}_{X\left|f\right(S)}\left(x\right|t)P_{f\left(S\right)|S}\left(t\right|s),\end{array}$ where we have used that $X-S-f\left(S\right)$ forms a Markov chain. ☐

While it is easy to understand that pre-garbling can be advantageous in general (since it can work as preprocessing), we find it surprising that this can also happen in the case where the pre-garbling is done in terms of a function f; that is, in terms of a channel ${\lambda }^f$ that does coarse-graining.

4. Examples

Example 2.

Consider the joint distribution

$$\begin{array}{ccccc}f\left(s\right)& s& x_1& x_2& P_{f\left(S\right)S{X}_1{X}_2}\\ 0& 0& 0& 0& 1/4\\ 0& 1& 0& 1& 1/4\\ 0& 0& 1& 0& 1/8\\ 0& 1& 1& 0& 1/8\\ 1& 2& 1& 1& 1/4\end{array}$$

and the function f: {0, 1, 2} → {0, 1} with f(0) = f(1) = 0 and f(2) = 1. Then, $X_1$ and $X_2$ are independent uniform binary random variables, and f(S) = And(X₁, X₂). By symmetry, the joint distributions of the pairs $(f\left(S\right),X_1)$ and $(f\left(S\right),X_2)$ are identical, and so the two channels $X_1\leftarrow f\left(S\right)$ and $X_2\leftarrow f\left(S\right)$ are identical. In particular, $(X_1\leftarrow f\left(S\right))\le (X_2\leftarrow f\left(S\right))$.

On the other hand, consider the utility function

$$\begin{array}{ccc}s& a& u(s,a)\\ 0& 0& 0\\ 0& 1& 0\\ 1& 0& 1\\ 1& 1& 0\\ 2& 0& 0\\ 2& 1& 1\end{array}$$

To compute the optimal decision rule, let us look at the conditional distributions:

$$\begin{array}{cc}\begin{array}{ccc}s& x_1& P_{S|X_1}\left(s\right|x_1)\\ 0& 0& 1/2\\ 1& 0& 1/2\\ 0& 1& 1/4\\ 1& 1& 1/4\\ 2& 1& 1/2\end{array}& \begin{array}{ccc}s& x_2& P_{S|X_2}\left(s\right|x_2)\\ 0& 0& 3/4\\ 1& 0& 1/4\\ 0& 1& 0\\ 1& 1& 1/2\\ 2& 1& 1/2\end{array}\end{array}$$

The optimal decision rule for $X_1$ is a(0) = 0, a(1) = 1, with expected utility

$$u_{X_1}:=1/2·1/2+1/2·1/2=1/2.$$

The optimal decision rule for $X_2$ is a(0) = 0, a(1) ∈ {0, 1} (this is not unique in this case), with expected utility

$$u_{X_2}:=1/2·1/4+1/2·1/2=3/8<1/2.$$

How can we understand this example? Some observations:

• It is easy to see that $X_2$ has more irrelevant information than $X_1$: namely, $X_2$ can determine relatively precisely when $S=0$. However, since $S=0$ gives no utility independent of the action, this information is not relevant. It is more difficult to understand why $X_2$ has less relevant information than $X_1$. Surprisingly, $X_1$ can determine more precisely when $S=1$: if $S=1$, then $X_1$ “detects this” (in the sense that $X_1$ chooses action 0) with probability $2/3$. For $X_2$, the same probability is only $1/3$.
• The conditional entropies of S given $X_2$ are smaller than the conditional entropies of S given $X_1$:

  $$\begin{array}{cccc}\hfill H\left(S\right|X_1=0)& =log\left(2\right),\hfill & \hfill H\left(S\right|X_1=1)& =\frac{3}{2}log\left(2\right),\hfill \\ \hfill H\left(S\right|X_2=0)& =2log\left(2\right)-\frac{3}{2}log\left(3\right)\approx 0.4150375log\left(2\right),\hfill & \hfill H\left(S\right|X_2=1)& =log\left(2\right).\hfill \end{array}$$
• One can see in which sense $f\left(S\right)$ captures the relevant information for $X_1$, and indeed for the whole decision problem: knowing $f\left(S\right)$ is completely sufficient in order to receive the maximal utility for each state of S. However, when information is incomplete, it matters how the information about the different states of S is mixed, and two variables $X_1,X_2$ that have the same joint distribution with $f\left(S\right)$ may perform differently. It is somewhat surprising that it is the random variable that has less information about S and that is conditionally independent of S given $f\left(S\right)$ which actually performs better.

Example 2 is different from the pre-garbling Example 1 discussed in Section 2. In the latter, both channels had the same amount of information (mutual information) about S, but for the given decision problem the information provided by ${\kappa }_2$ was more relevant than the information provided by ${\kappa }_1$. The first difference in Example 2 is that $X_1$ has less mutual information about S than $X_2$ (Lemma 2). Moreover, both channels are identical with respect to $f\left(S\right)$; i.e., they provide the same information about $f\left(S\right)$, and for $X_1$ it is the only information it has about S. So, one could argue that $X_2$ has additional information that does not help, but decreases the expected utility instead.

We give another example which shows that $X_2$ can also be chosen as a deterministic function of S.

Example 3.

Consider the joint distribution

$$\begin{array}{ccccc}f\left(s\right)& s& x_1& x_2& P_{f\left(S\right)S{X}_1{X}_2}\\ 0& 0& 0& 0& 1/6\\ 0& 0& 1& 0& 1/6\\ 0& 1& 0& 1& 1/6\\ 0& 1& 1& 1& 1/6\\ 1& 2& 1& 1& 1/3\end{array}$$

The function f is as above, but now also $X_2$ is a function of S. Again, the two channels $X_1\leftarrow f\left(S\right)$ and $X_2\leftarrow f\left(S\right)$ are identical, and $X_1$ is independent of S given $f\left(S\right)$. Consider the utility function

$$\begin{array}{ccc}s& a& u(s,a)\\ 0& 1& 0\\ 1& 0& 0\\ 1& 0& 0\\ 1& 1& 1\\ 2& 0& 0\\ 2& 1& -1\end{array}$$

One can show that it is optimal for an agent who relies on $X_2$ to always choose action 0, which brings no reward (and no loss). However, when the agent knows that $X_1$ is zero, he may safely choose action 1 and has a positive probability of receiving a positive reward.

To add another interpretation to the last example, we visualize the situation in the following Bayesian network:

$$X\leftarrow S\to f\left(S\right)\to X^{\prime },$$

where, as in Proposition 2 and its proof, we let $X=X_2$, and we consider $X^{\prime }=X_1$ as an approximation of X. Then, S denotes the state of the system that we are interested in, and X denotes a given set of observables of interest. $f\left(S\right)$ can be considered as a “proxy” in situations where it is difficult to observe X directly. For example, in neuroimaging, instead of directly measuring the neural activity X, one might look at an MRI signal $f\left(S\right)$. In economic and social sciences, monetary measures like the GDP are used as a proxy for prosperity.

A decision problem can always be considered as a classification problem defined by the utility $u(s,a)$ by considering the optimal action as the class label of state S. Proposition 2 now says that there exist $S,X,f\left(S\right)$, and a classification problem $u(s,a)$, such that the approximated features $X^{\prime }$ (simulated from $f\left(S\right)$) allow for a better classification (higher utility) than the original features X.

In such a situation, looking at $f\left(S\right)$ will always be better than looking at either X or $X^{\prime }$. Thus, the paradox will only play a role in situations where it is not possible to base the decision on $f\left(S\right)$ directly. For example, $f\left(S\right)$ might still be too large, or X might have a more natural interpretation, making it easier to interpret for the decision taker. However, when it is better to base a decision on a proxy rather than directly on the observable of interest, this interpretation may be erroneous.

5. Information Decomposition and Le Cam Deficiency

Given two channels ${\kappa }_1,{\kappa }_2$, how can one decide whether or not ${\kappa }_1\le {\kappa }_2$? The easiest way is to check whether the equation ${\kappa }_1=\lambda ·{\kappa }_2$ has a solution $\lambda$ that is a stochastic matrix. In the finite alphabet case, this amounts to checking the feasibility of a linear program, which is considered computationally easy. However, when the feasibility check returns a negative result, this approach does not give any more information (e.g., how far ${\kappa }_1$ is away from being a garbling of ${\kappa }_2$). A function that quantifies how far ${\kappa }_1$ is from being a garbling of ${\kappa }_2$ is given by the (Le Cam) deficiency and its various generalizations [9]. Another such function is given by $UI$ defined in [7] that accounts for the fact that the channels we consider are of the form ${\kappa }_1=(X_1\leftarrow S)$ and ${\kappa }_2=(X_2\leftarrow S)$; that is, they are derived from conditional distributions of random variables. In contrast to the deficiencies, $UI$ depends on the input distribution to these channels.

Let $P_{S{X}_1{X}_2}$ be a joint distribution of S and the outputs $X_1$ and $X_2$. Let ${\Delta }_P$ be the set of all joint distributions of the random variables $S,X_1,X_2$ (with the same alphabets) that are compatible with the marginal distributions of $P_{S{X}_1{X}_2}$ for the pairs $(S,X_1)$ and $(S,X_2)$; i.e.,

$${\Delta }_P:=\left\{Q_{S{X}_1{X}_2}\in \Delta :Q_{S{X}_1}=P_{S{X}_1},Q_{S{X}_2}=P_{S{X}_2}\right\}.$$

In other words, ${\Delta }_P$ consists of all joint distributions that are compatible with ${\kappa }_1$ and ${\kappa }_2$ and that have the same distribution for S as $P_{S{X}_1{X}_2}$. Consider the function

$$\begin{array}{c}\hfill UI(S;X_1\backslash X_2):=\underset{Q\in {\Delta }_P}{min}{I}_Q(S;X_1|X_2),\end{array}$$

where $I_Q$ denotes the conditional mutual information evaluated with respect to the the joint distribution Q. This function has the following property: $UI(S;X_1\backslash X_2)=0$ if and only if ${\kappa }_1\le {\kappa }_2$ [7]. Computing $UI$ is a convex optimization problem. However, the condition number can be very bad, which makes the problem difficult in practice.

$UI$ is interpreted in [7] as a measure of the unique information that $X_1$ conveys about S (with respect to $X_2$). So, for instance, with this interpretation Example 2 can be summarized as follows: neither $X_1$ nor $X_2$ has unique information about $f\left(S\right)$. However, both variables have unique information about S, although $X_1$ is conditionally independent of S given $f\left(S\right)$ and thus—in contrast to $X_2$—contains no “additional” information about S. We now apply $UI$ to a parameterized version of the And gate in Example 2.

Example 4.

Figure 1a shows a heat map of $UI$ computed on the set of all distributions of the form

$$\begin{array}{ccccc}f\left(s\right)& s& x_1& x_2& P_{f\left(S\right)S{X}_1{X}_2}\\ 0& 0& 0& 0& 1/8+2b\\ 0& 1& 0& 0& 1/8-2b\\ 0& 0& 0& 1& 1/8+a\\ 0& 1& 0& 1& 1/8-a\\ 0& 0& 1& 0& 1/8+MMa/2+b\\ 0& 1& 1& 0& 1/8-MMa/2-b\\ 1& 2& 1& 1& 1/4\end{array}$$

where $-1/8\le a\le 1/8$ and $-1/16\le b\le 1/16$. This is the set of distributions of $S,X_1,X_2$ that satisfy the following constraints:

1. 

$X_1,X_2$ are independent;

2. 

f(S) = And(X₁, X₂), where f is as in Example 2; and

3. 

$X_1$ is independent of S given $f\left(S\right)$.

Along the secondary diagonal $b=a/2$, the marginal distributions of the pairs $(S,X_1)$ and $(S,X_2)$ are identical. In such a situation, the channels $(X_1\leftarrow S)$ and $(X_2\leftarrow S)$ are Blackwell-equivalent, and so $UI$ vanishes. Further away from the diagonal, the marginal distributions differ, and $UI$ grows. The maximum value is achieved at the corners for $(a,b)=(-\raisebox{1ex}{$1$}\!\left/ \!\raisebox{-1ex}{$8$}\right.,\raisebox{1ex}{$1$}\!\left/ \!\raisebox{-1ex}{$16$}\right.)$, $(\raisebox{1ex}{$1$}\!\left/ \!\raisebox{-1ex}{$8$}\right.,-\raisebox{1ex}{$1$}\!\left/ \!\raisebox{-1ex}{$16$}\right.)$. At the upper left corner $(a,b)=\pm (-\raisebox{1ex}{$1$}\!\left/ \!\raisebox{-1ex}{$8$}\right.,\raisebox{1ex}{$1$}\!\left/ \!\raisebox{-1ex}{$16$}\right.)$, we recover Example 2.

Example 5.

Figure 1b shows a heat map of $UI$ computed on the set of all distributions of the form

$$\begin{array}{ccccc}f\left(s\right)& s& x_1& x_2& P_{f\left(S\right)S{X}_1{X}_2}\\ 0& 0& 0& 0& a^2/(a+b)\\ 0& 0& 1& 0& ab/(a+b)\\ 0& 1& 0& 1& ab/(a+b)\\ 0& 1& 1& 1& b^2/(a+b)\\ 1& 2& 1& 1& 1-a-b\end{array}$$

where $a,b\ge 0$ and $a+b\le 1$. This extends Example 3, which is recovered for $a=b=\raisebox{1ex}{$1$}\!\left/ \!\raisebox{-1ex}{$3$}\right.$. This is the set of distributions of $S,X_1,X_2$ that satisfy the following constraints:

1. 

$X_2$ is a function of S, where the function is as in Example 3.

2. 

$X_1$ is independent of S given $f\left(S\right)$.

3. 

The channels $X_1\leftarrow f\left(S\right)$ and $X_2\leftarrow f\left(S\right)$ are identical.