The Information Loss of a Stochastic Map

Abstract

We provide a stochastic extension of the Baez–Fritz–Leinster characterization of the Shannon information loss associated with a measure-preserving function. This recovers the conditional entropy and a closely related information-theoretic measure that we call conditional information loss. Although not functorial, these information measures are semi-functorial, a concept we introduce that is definable in any Markov category. We also introduce the notion of an entropic Bayes’ rule for information measures, and we provide a characterization of conditional entropy in terms of this rule.

1. Introduction

The information loss $K\left(f\right)$ associated with a measure-preserving function $(X,p)\stackrel{f}{\to }(Y,q)$ between finite probability spaces is given by the Shannon entropy difference

$$K\left(f\right):=H\left(p\right)-H\left(q\right),$$

where $H\left(p\right):=-{\sum }_{x\in X}{p}_{x}log{p}_x$ is the Shannon entropy of p (and similarly for q). In [1], Baez, Fritz, and Leinster proved that the information loss satisfies, and is uniquely characterized up to a non-negative multiplicative factor by, the following conditions:

0.

Positivity: $K\left(f\right)\ge 0$ for all $(X,p)\stackrel{f}{\to }(Y,q)$. This says that the information loss associated with a deterministic process is always non-negative.

1.

Functoriality: $K(g\circ f)=K\left(g\right)+K\left(f\right)$ for every composable pair $(f,g)$ of measure-preserving maps. This says that the information loss of two successive processes is the sum of the information losses associated with each process.

2.

Convex Linearity: $K(\lambda f\oplus (1-\lambda \left)g\right)=\lambda K\left(f\right)+(1-\lambda )K\left(g\right)$ for all $\lambda \in (0,1)$. This says that the information loss associated with tossing a (possibly unfair) coin in deciding amongst two processes is the associated weighted sum of their information losses.

3.

Continuity: $K\left(f\right)$ is a continuous function of f. This says that the information loss does not change much under small perturbations (i.e., is robust with respect to errors).

As measure-preserving functions may be viewed as deterministic stochastic maps, it is natural to ask whether there exist extensions of the Baez–Fritz–Leinster (BFL) characterization of information loss to maps that are inherently random (i.e., stochastic) in nature. In particular, what information-theoretic quantity captures such an information loss in this larger category?

This question is answered in the present work. Namely, we extend the BFL characterization theorem, which is valid on deterministic maps, to the larger category of stochastic maps. In doing so, we also find a characterization of the conditional entropy. Although the resulting extension is not functorial on the larger category of stochastic maps, we formalize a weakening of functoriality that restricts to functoriality on deterministic maps. This weaker notion of functoriality is definable in any Markov category [2,3], and it provides a key axiom in our characterization.

To explain how we arrive at our characterization, let us first recall the definition of stochastic maps between finite probability spaces, for which the measure-preserving functions are a special case. A stochastic map $(X,p)\stackrel{f}{⇝}(Y,q)$ associates with every $x\in X$ a probability distribution $f_x$ on Y such that $q_y={\sum }_{x\in X}{f}_{yx}{p}_x$, where $f_{yx}$ is the distribution $f_x$ evaluated at $y\in Y$. In terms of information flow, the space $(X,p)$ may be thought of as a probability distribution on the set of inputs for a communication channel described by the stochastic matrix $f_{yx}$, while $(Y,q)$ is then thought of as the induced distribution on the set of outputs of the channel.

Extending the information loss functor by assigning $H\left(p\right)-H\left(q\right)$ to any stochastic map $(X,p)\stackrel{f}{⇝}(Y,q)$ would indeed result in an assignment that satisfies conditions 1–3 listed above. However, it would no longer be positive and the interpretation as an information loss would be gone. Furthermore, no additional information about the stochasticity of the map f would be used in determining this assignment. In order to guarantee positivity, an additional term, depending on the stochasticity of f, is needed. This term is provided by the conditional entropy of $(X,p)\stackrel{f}{⇝}(Y,q)$ and is given by the the non-negative real number

$$H\left(f\right|p):=\sum _{x\in X}{p}_{x}H\left(f_x\right),$$

where $H\left(f_x\right)$ is the Shannon entropy of the distribution $f_x$ on Y (in the case that $(X,p)$ and $(Y,q)$ are probability spaces associated with the alphabets of random variables $\mathcal{X}$ and $\mathcal{Y}$, then $H\left(f\right|p)$ coincides with conditional entropy $H\left(\mathcal{Y}\right|\mathcal{X})$ [4]). If $(X,p)\stackrel{f}{⇝}(Y,p)$ is in fact deterministic, i.e., if $f_x$ is a point-mass distribution for all $x\in X$, then $H\left(f_x\right)=H\left(f\right|p)=0$ for all $x\in X$. As such, $H\left(f\right|p)$ is a measure of the uncertainty (or randomness) of the outputs of f averaged over the prior distribution p on the set X of its inputs. Indeed, $H\left(f\right|p)$ is maximized precisely when $f_x$ is the uniform distribution on Y for all $x\in X$.

Therefore, given a stochastic map $(X,p)\stackrel{f}{⇝}(Y,q)$, we call

$$K\left(f\right):=H\left(p\right)-H\left(q\right)+H\left(f\right|p)$$

the conditional information loss of $(X,p)\stackrel{f}{⇝}(Y,q)$ (the same letter K is used here because it agrees with the Shannon entropy difference when f is deterministic). As $H\left(f\right|p)=0$ whenever f is deterministic, the conditional information loss restricts to the category of measure preserving functions as the information loss functor of Baez, Fritz, and Leinster, while also satisfying conditions 0, 2, and 3 (i.e., positivity, convex linearity, and continuity) on the larger category of stochastic maps. However, conditional information loss is not functorial in general, and while this may seem like a defect at first glance, we prove that there is no extension of the information loss functor that remains functorial on the larger category of stochastic maps if the positivity axiom is to be preserved, thus retaining an interpretation as information loss. In spite of this, conditional information loss does satisfy a weakened form of functoriality, which we briefly describe now.

A pair $(X,p)\stackrel{f}{⇝}(Y,q)\stackrel{g}{⇝}(Z,r)$ of composable stochastic maps is a.e. coalescable if and only if for every pair of elements $z\in Z$ and $x\in X$ for which $r_z>0$ and $p_x>0$, there exists a unique $y\in Y$ such that $f_{yx}>0$ and $g_{zy}>0$. Intuitively, this says that the information about the intermediate step can be recovered given knowledge about the input and output. In particular, if f is deterministic, then the pair $(f,g)$ is a.e. colescable (for obvious reasons, since knowing x alone is enough to determine the intermediate value). However, there are other many situations where a pair could be a.e. coalescable and the maps need not be deterministic. With this definition in place (which we also generalize to the setting of arbitrary Markov categories), we replace functoriality with the following weaker condition.

1^🟉.

Semi-functoriality: $K(g\circ f)=K\left(g\right)+K\left(f\right)$ for every a.e. coalescable pair $(X,p)\stackrel{f}{⇝}(Y,q)\stackrel{g}{⇝}(Z,r)$ of stochastic maps. This says that the conditional information loss of two successive processes is the sum of the conditional information losses associated with each process provided that the information in the intermediate step can always be recovered.

Replacing functoriality with semi-functoriality is not enough to characterize the conditional information loss. However, it comes quite close, as only one more axiom is needed. Assuming positivity, semi-functoriality, convex linearity, and continuity, there are several equivalent axioms that may be stipulated to characterize the conditional information loss. To explain the first option, we introduce a convenient factorization of every stochastic map $(X,p)\stackrel{f}{⇝}(Y,q)$. The bloom-shriek factorization of f is given by the decomposition $f={\pi }_Y\circ {¡}_f$, where ${¡}_f:X⇝X\times Y$ is the bloom of f whose value at $x^{\prime }$ is the probability measure on $X\times Y$ given by sending $(x,y)$ to ${\delta }_{x^{\prime }x}{f}_{y{x}^{\prime }}$, where ${\delta }_{x^{\prime }x}$ is the Kronecker delta. In other words, ${¡}_f$ records each of the probability measures $f_x$ on a copy of Y indexed by $x\in X$. A visualization of the bloom of f is given in Figure 1a. When one is given the additional data of probability measures p and q on X and Y, respectively, then Figure 1b illustrates the bloom-shriek factorization of f. From this point of view, ${¡}_f$ keeps track of the information encoded in both p and f, while the projection map ${\pi }_Y$ forgets, or loses, some of this information.

Figure 1. A visualization of bloom and the bloom-shriek factorization via water droplets as inspired by Gromov [5]. The bloom of f splits each water droplet of volume 1 (an element of X) into several water droplets whose total volume equates to 1. If X has a probability p on it, then the initial volume of that water droplet is scaled by this probability. The stochastic map therefore splits the water droplet using this scale.

With this in mind, our final axiom to characterize the conditional information loss is

4(a).

Reduction: $K\left(f\right)=K\left({\pi }_Y\right)$, where $f={\pi }_Y\circ {¡}_f$ is the bloom-shriek factorization of f. This says that the conditional information loss of f equals the information loss of the projection using the associated joint distribution on $X\times Y$.

Note that this axiom describes how K is determined by its action on an associated class of deterministic morphisms. These slightly modified axioms, namely, semi-functoriality, convex linearity, continuity, and reduction, characterize the conditional information loss and therefore extend Baez, Fritz, and Leinster’s characterization of information loss. A much simpler axiom that may be invoked in place of the reduction axiom which also characterizes conditional information loss is the following.

4(b).

Blooming: $K\left({¡}_p\right)=0$, where ${¡}_p$ is the unique map $(•,1)⇝(X,p)$ from a one point probability space to $(X,p)$. This says that if a process begins with no prior information, then there is no information to be lost in the process.

The conditional entropy itself can be extracted from the conditional information loss by a process known as Bayesian inversion, which we now briefly recall. Given a stochastic map $(X,p)\stackrel{f}{⇝}(Y,q)$, there exists a stochastic map $(Y,q)\stackrel{\overline{f}}{⇝}(Y,q)$ such that $f_{yx}{p}_x={\overline{f}}_{xy}{q}_y$ for all $x\in X$ and $y\in Y$ (the stochastic map $\overline{f}$ is the almost everywhere unique conditional probability so that Bayes’ rule holds). Such a map is called a Bayesian inverse of f. The Bayesian inverse can be visualized using the bloom-shriek factorization because it itself has a bloom-shriek factorization $\overline{f}={\pi }_X\circ {¡}_{\overline{f}}$. This is obtained by finding the stochastic maps in the opposite direction of the arrows so that they reproduce the appropriate volumes of the water droplets.

Given this perspective on Bayesian inversion, we prove that the conditional entropy of $(X,p)\stackrel{f}{⇝}(Y,q)$ equals the conditional information loss of its Bayesian inverse $(Y,q)\stackrel{\overline{f}}{⇝}(X,p)$. Moreover, since the conditional information loss of $\overline{f}$ is just the information loss of ${\pi }_X$, this indicates how the conditional entropy and conditional information losses are the ordinary information losses associated with the two projections ${\pi }_X$ and ${\pi }_Y$ in Figure 1b. This duality also provides an interesting perspective on conditional entropy and its characterization. Indeed, using Bayesian inversion, we also characterize the conditional entropy as the unique assignment F sending measure-preserving stochastic maps between finite probability spaces to real numbers satisfying conditions 0, $1^{🟉}$, 2, and 3 above, but with a new axiom that reads as follows.

4(c).

Entropic Bayes’ Rule: $F\left(f\right)+F\left({¡}_p\right)=F\left(\overline{f}\right)+F\left({¡}_q\right)$ for all $(X,p)\stackrel{f}{⇝}(Y,q)$. This is an information theoretic analogue of Bayes’ rule, which reads $f_{yx}{p}_x={\overline{f}}_{xy}{q}_y$ for all $x\in X$ and $y\in Y$, or in more traditional probabilistic notation $\mathbb{P}\left(y\right|x\left)\mathbb{P}\right(x)=\mathbb{P}(x\left|y\right)\mathbb{P}\left(y\right).$

In other words, we obtain a Bayesian characterization of the conditional entropy. This provides an entropic and information-theoretic description of Bayes’ rule from the Markov category perspective, in a way that we interpret as answering an open question of Fritz [6].

2. Categories of Stochastic Maps

In the first few sections, we define all the concepts involved in proving that the conditional information loss satisfies the properties that we will later prove characterize it. This section introduces the domain category and its convex structure.

Definition 1.

Let X and Y be finite sets. Astochastic map$f:X⇝Y$ associates a probability measure $f_x$ to every $x\in X$. If $f:X⇝Y$ is such that $f_x$ is a point-mass distribution for every $x\in X$, then f is said to bedeterministic.

Notation 1.

Given a stochastic map $f:X⇝Y$ (also written as $X\stackrel{f}{⇝}Y$), the value $f_x\left(y\right)\in [0,1]$ will be denoted by $f_{yx}$. As there exists a canonical bijection between deterministic maps of the form $X⇝Y$ and functions $X\to Y$, deterministic maps from X to Y will be denoted by the functional notation $X\to Y$.

Definition 2.

A stochastic map of the form $•⇝X$ from a single element set to a finite set X is a single probability measure on X. Its unique value at x will be denoted by $p_x$ for all $x\in X$. The set $N_p:=\{x\in X|p_x=0\}$ will be referred to as thenullspaceof p.

Definition 3.

LetFinStochbe the category of stochastic maps between finite sets. Given a finite set X, the identity map of X inFinStochcorresponds to the identity function ${\mathrm{id}}_X:X\to X$. Second, given stochastic maps $f:X⇝Y$ and $g:Y⇝Z$, the composite $g\circ f:X⇝Z$ is given by the Chapmann–Kolmogorov equation ${(g\circ f)}_{zx}:={\sum }_{y\in Y}{g}_{zy}{f}_{yx}.$

Definition 4.

Let X be a finite set. Thecopyof X is the diagonal embedding ${\mathsf{\Delta }}_X:X\to X\times X$, and thediscardof X is the unique map from X to the terminal object • inFinStoch, which will be denoted by ${!}_X:X\to •$. If Y is another finite set, theswap mapis the map $\gamma :X\times Y\to Y\times X$ given by $(x,y)\mapsto (y,x)$. Given morphisms $f:X⇝X^{\prime }$ and $g:Y⇝Y^{\prime }$ inFinStoch, theproductof f and g is the stochastic map $f\times g:X\times Y⇝X^{\prime }\times Y^{\prime }$ given by ${(f\times g)}_{(x^{\prime },y^{\prime })(x,y)}:=f_{x^{\prime }x}{g}_{y^{\prime }y}.$

The product of stochastic maps endows FinStoch with the structure of a monoidal category. Together with the copy, discard, and swap maps, FinStoch is a Markov category [2,3].

Definition 5.

Let $\mathbf{FinPS}$ (this stands for “finiteprobabilities andstochastic maps”) be the co-slice category $•\downarrow \mathbf{FinStoch}$, i.e., the category whose objects are pairs $(X,p)$ consisting of a finite set X equipped with a probability measure p, and a morphism from $(X,p)$ to $(Y,q)$ is a stochastic map $X\stackrel{f}{⇝}Y$ such that ${\displaystyle q_y=\sum _{x\in X}{f}_{yx}{p}_x}$ for all $y\in Y$. The subcategory of deterministic maps in $\mathbf{FinPS}$ will then be denoted by $\mathbf{FinPD}$ (which stands for “finiteprobabilities anddeterministic maps”). A pair $(f,g)$ of morphisms in $\mathbf{FinPS}$ is said to be acomposable pairiff $g\circ f$ exists.

Note that the category $\mathbf{FinPD}$ was called $\mathbf{FinProb}$ in [1].

Remark 1.

Though it is often the case that we will denote a morphism $(X,p)\stackrel{f}{⇝}(Y,q)$ in $\mathbf{FinPS}$ simply by f, such notation is potentially ambiguous, as the morphism $(X,p^{\prime })\stackrel{f}{⇝}(Y,q^{\prime })$ is distinct from the morphsim $(X,p)\stackrel{f}{⇝}(Y,q)$ whenever $p\ne p^{\prime }$. As such, we will only employ the shorthand of denoting a morphism in $\mathbf{FinPS}$ by its underlying stochastic map whenever the source and target of the morphism are clear from the context.

Lemma 1.

The object $(•,1)$ given by a single element set equipped with the unique probability measure is a zero object (i.e., terminal and initial) in $\mathbf{FinPS}$.

Definition 6.

Given an object $(X,p)$ in $\mathbf{FinPS}$, theshriekandbloomof p are the unique maps to and from $(•,1)$ respectively, which will be denoted ${!}_p:(X,p)\to (•,1)$ and ${¡}_p:(•,1)⇝(X,p)$ (the former is deterministic, while the latter is stochastic). The underlying stochastic maps associated with ${!}_p$ and ${¡}_p$ are ${!}_X:X\to •$ and $p:•⇝X$, respectively.

Example 1.

Since $(•,1)$ is a zero object, given any two objects $(X,p)$ and $(Y,q)$, there exists at least one morphism $(Y,q)⇝(X,p)$, namely the composite $(Y,q)\stackrel{{!}_q}{\to }(•,1)\stackrel{{¡}_p}{⇝}(X,p)$.

Definition 7.

Let $(X,p)\stackrel{f}{⇝}(Y,q)$ be a morphism in $\mathbf{FinPS}$. Thejoint distributionassociated with f is the probability measure $\vartheta \left(f\right):•⇝X\times Y$ given by $\vartheta {\left(f\right)}_{(x,y)}=f_{yx}{p}_x$.

It is possible to take convex combinations of both objects and morphisms in $\mathbf{FinPS}$, and such assignments will play a role in our characterization of conditional entropy.

Definition 8.

Let $p:•⇝X$ be a probability measure and let ${\left\{(Y_x,q^x)\right\}}_{x\in X}$ be a collection of objects in $\mathbf{FinPS}$ indexed by X. The p-weighted convex sum${⨁}_{x\in X}{p}_x(Y_x,q^x)$ is defined to be the set ${\coprod }_{x\in X}{Y}_x$ equipped with the probability measure ${⨁}_{x\in X}{p}_x{q}^x$ given by

$${\left(\underset{x\in X}{⨁}{p}_x{q}^x\right)}_z:=\left\{\begin{array}{c}{p}_x{q}_z^{x}ifz\in Y_x\hfill \\ 0\mathit{otherwise}.\hfill \end{array}\right.$$

In addition, if $(Y_x,q^x)\stackrel{Q^x}{⇝}(Y_x^{\prime },q^{\prime x})$ is a collection of morphisms in $\mathbf{FinPS}$ indexed by X, the p-weighted convex sum${⨁}_{x\in X}{p}_x{Q}^x:\left({\coprod }_{x\in X}{Y}_x,{⨁}_{x\in X}{p}_x{q}^x\right)⇝\left({\coprod }_{x\in X}{Y}_x^{\prime },{⨁}_{x\in X}{p}_x{q}^{\prime x}\right)$ is given by

$${\left(\underset{x\in X}{⨁}{p}_x{Q}^x\right)}_{z^{\prime }z}:=\left\{\begin{array}{c}{p}_x{\left(Q^x\right)}_{z^{\prime }z}\mathit{if}(z^{\prime },z)\in Y_x^{\prime }\times Y_x\hfill \\ 0\mathit{otherwise}.\hfill \end{array}\right.$$

3. The Baez–Fritz–Leinster Characterization of Information Loss

In [1], Baez, Fritz, and Leinster (BFL) characterized the Shannon entropy difference associated with measure-preserving functions between finite probability spaces as the only non-vanishing, continuous, convex linear functor from $\mathbf{FinPD}$ to the non-negative reals (up to a multiplicative constant). It is then natural to ask whether there exist either extensions or analogues of their result by including non-deterministic morphisms from the larger category $\mathbf{FinPS}$. Before delving deeper into such inquiry, we first recall in detail the characterization theorem of BFL.

Definition 9.

Let $\mathbb{B}\mathbb{R}$ be the convex category consisting of a single object and whose set of morphisms is $\mathbb{R}$. The composition in $\mathbb{B}\mathbb{R}$ is given by addition. Convex combinations of morphisms are given by ordinary convex combinations of numbers. The subcategory of non-negative reals will be denoted $\mathbb{B}{\mathbb{R}}_{\ge 0}$.

In the rest of the paper, we will not necessarily assume that assignments from one category to another are functors. Nevertheless, we do assume they form (class) functions (see ([7], Section I.7) for more details). Furthermore, we assume that they respect or reflect source and targets in the following sense. If $\mathcal{C}$ and $\mathcal{D}$ are two categories, all functions $F:\mathcal{C}\to \mathcal{D}$ are either covariant or contravariant in the sense that for any morphism $a\stackrel{\gamma }{\to }b$ in $\mathcal{C}$, $F\left(\gamma \right)$ is a morphism from $F\left(a\right)$ to $F\left(b\right)$ or from $F\left(b\right)$ to $F\left(a\right)$, respectively. These are the only types of functions between categories we will consider in this work. As such, we therefore abuse terminology and use the term functions for such assignments throughout. If M is a commutative monoid and $\mathbb{B}M$ denotes its one object category, then every covariant function $\mathcal{C}\to \mathbb{B}M$ is also contravariant and vice-versa.

We now define a notion of continuity for functions of the form $F:\mathbf{FinPS}\to \mathbb{B}\mathbb{R}$.

Definition 10.

A sequence of morphisms $(X_n,p_n)\stackrel{f_n}{⇝}(Y_n,q_n)$ in $\mathbf{FinPS}$convergesto a morphism $(X,p)\stackrel{f}{⇝}(Y,q)$ if and only if the following two conditions hold.

(a) 

There exists an $N\in \mathbb{N}$ for which $X_n=X$ and $Y_n=Y$ for all $n\ge N$.

(b) 

The following limits hold: ${\displaystyle \underset{n\to \infty }{lim}{p}_n=p}$ and ${\displaystyle \underset{n\to \infty }{lim}{f}_n=f}$ (note that these limits necessarily imply ${\displaystyle \underset{n\to \infty }{lim}{q}_n=q}$).

A function $F:\mathbf{FinPS}\to \mathbb{B}\mathbb{R}$ iscontinuousif and only if ${\displaystyle \underset{n\to \infty }{lim}F\left(f_n\right)=F\left(f\right)}$ whenever $\left\{f_n\right\}$ is a sequence in $\mathbf{FinPS}$ converging to f.

Remark 2.

In the subcategory $\mathbf{FinPD}$, since the topology of the collection of functions from a finite set X to another finite set Y is discrete, one can equivalently assume that a sequence $f_n$ as in Definition 10, but this time with all $f_n$ deterministic, converges to $(X,p)\stackrel{f}{\to }(Y,q)$ if and only if the following two conditions hold.

(a) 

There exists an $N\in \mathbb{N}$ for which $X_n=X,Y_n=Y,$$f_n=f$ for all $n\ge N$.

(b) 

For $n\ge N$, one has ${\displaystyle \underset{n\to \infty }{lim}{p}_n=p}$.

In this way, our definition of convergence agrees with the definition of convergence of BFL on the subcategory $\mathbf{FinPD}$ [1].

Definition 11.

A function $F:\mathbf{FinPS}\to \mathbb{B}\mathbb{R}$ is said to beconvex linearif and only if for all objects $(X,p)$ in $\mathbf{FinPS}$,

$$F\left(\underset{x\in X}{⨁}{p}_x{Q}^x\right)=\sum _{x\in X}{p}_{x}F\left(Q^x\right)$$

for all collections $\left\{(Y_x,q^x)\stackrel{Q^x}{⇝}(Y_x^{\prime },q^{\prime x})\right\}{}_{x\in X}$ in $\mathbf{FinPS}$.

Definition 12.

A function $F:\mathbf{FinPS}\to \mathbb{B}\mathbb{R}$ is said to befunctorialif and only if it is in fact a functor, i.e., if and only if $F(g\circ f)=F\left(f\right)+F\left(g\right)$ for every composable pair $(f,g)$ in $\mathbf{FinPS}$.

Definition 13.

Let $p:•⇝X$ be a probability measure. TheShannon entropyof p is given by

$$H\left(p\right):=-\sum _{x\in X}{p}_{x}log\left(p_x\right).$$

When considering any entropic quantity, we will always adhere to the convention that $0log\left(0\right)=0$.

Definition 14.

Given a map $(X,p)\stackrel{f}{\to }(Y,p)$ in $\mathbf{FinPD}$, the Shannon entropy difference $H\left(p\right)-H\left(q\right)$ will be referred to as theinformation lossof f. Information loss defines a functor $K:\mathbf{FinPD}\to \mathbb{B}\mathbb{R}$, henceforth referred to as theinformation loss functoron $\mathbf{FinPD}$.

Theorem 1

(Baez–Fritz–Leinster [1]). Suppose $F:\mathbf{FinPD}\to \mathbb{B}{\mathbb{R}}_{\ge 0}$ is a function which satisfies the following conditions.

1. 

F is functorial.

2. 

F is convex linear.

3. 

F is continuous.

Then F is a non-negative multiple of information loss. Conversely, the information loss functor is non-negative and satisfies conditions 1–3.

In light of Theorem 1, it is natural to question whether or not there exists a functor $K:\mathbf{FinPS}\to \mathbb{B}{\mathbb{R}}_{\ge 0}$ that restricts to $\mathbf{FinPD}$ as the information loss functor. It turns out that no such non-vanishing functor exists, as we prove in the following proposition.

Proposition 1.

If $F:\mathbf{FinPS}\to \mathbb{B}{\mathbb{R}}_{\ge 0}$ is a functor, then $F\left(f\right)=0$ for all morphisms f in $\mathbf{FinPS}$.

Proof. 

Let $(X,p)\stackrel{f}{⇝}(Y,q)$ be a morphism in $\mathbf{FinPS}$. Since F is a functor,

$$F\left({¡}_q\right)=F(f\circ {¡}_p)=F\left(f\right)+F\left({¡}_p\right)\Rightarrow 0\le F\left(f\right)=F\left({¡}_q\right)-F\left({¡}_p\right).$$

Let $(Y,q)\stackrel{g}{⇝}(X,p)$ be any morphism in $\mathbf{FinPS}$ (which necessarily exists by Example 1, for instance). Then a similar calculation yields

$$0\le F\left(g\right)=F\left({¡}_p\right)-F\left({¡}_q\right)=-F\left(f\right).$$

Hence, $F\left(f\right)=0$. □

4. Extending the Information Loss Functor

Proposition 1 shows it is not possible to extend the information loss functor to a functor on $\mathbf{FinPS}$. Nevertheless, in this section, we define a non-vanishing function$K:\mathbf{FinPS}\to \mathbb{B}{\mathbb{R}}_{\ge 0}$ that restricts to the information loss functor on $\mathbf{FinPD}$, which we refer to as conditional information loss. While K is not functorial, we show that it satisfies many important properties such as continuity, convex linearity, and invariance with respect to compositions with isomorphisms. Furthermore, in Section 5 we show K is functorial on a restricted class of composable pairs of morphisms (cf. Definition 18), which are definable in any Markov category. At the end of this section we characterize conditional information loss as the unique extension of the information loss functor satisfying the reduction axiom 4(a) as stated in the introduction. In Section 8, we prove an intrinsic characterization theorem for K without reference to the deterministic subcategory $\mathbf{FinPD}$ inside $\mathbf{FinPS}$. Appendix A provides an interpretation of the vanishing of conditional information loss in terms of correctable codes.

Definition 15.

Theconditional information lossof a morphism $(X,p)\stackrel{f}{⇝}(Y,q)$ in $\mathbf{FinPS}$ is the real number given by

$$K\left(f\right):=H\left(p\right)-H\left(q\right)+H\left(f\right|p),$$

where

$$H\left(f\right|p):=\sum _{x\in X}{p}_{x}H\left(f_x\right)$$

is theconditional entropyof $(X,p)\stackrel{f}{⇝}(Y,q)$.

Proposition 2.

The function $K:\mathbf{FinPS}\to \mathbb{B}\mathbb{R}$, uniquely determined on morphisms by sending $(X,p)\stackrel{f}{⇝}(Y,q)$ to $K\left(f\right)$, satisfies the following conditions.

(i) 

$K\left(f\right)\ge 0$.

(ii) 

K restricted to $\mathbf{FinPD}$ agrees with the information loss functor (cf. Definition 14).

(iii) 

K is convex linear.

(iv) 

K is continuous.

(v) 

Given $(X,p)\stackrel{f}{⇝}(Y,q)$, then $K\left(f\right)=K\left({\pi }_Y\right)$, where $(X\times Y,\vartheta \left(f\right))\stackrel{{\pi }_Y}{\to }(Y,q)$ is the projection and $\vartheta \left(f\right)$ is the joint distribution (cf. Definition 7).

Lemma 2.

Let $(X,p)\stackrel{f}{⇝}(Y,q)$ be a morphism in $\mathbf{FinPS}$. Then

$$K\left(f\right)=-\sum _{x\in X\backslash N_p}\sum _{y\in Y\backslash N_{f_x}}{f}_{yx}{p}_{x}log\left(\frac{f_{yx}{p}_x}{q_y}\right).$$

Proof of Lemma 2.

Applying K to f yields

$$\begin{array}{cc}\hfill K\left(f\right)& =-\sum _{x,y}{f}_{yx}{p}_{x}log\left(f_{yx}\right)-\sum _x{p}_{x}log\left(p_x\right)+\sum _y{q}_{y}log\left(q_y\right)\hfill \\ & =-\sum _{x,y}{f}_{yx}{p}_{x}log\left(f_{yx}\right)-\sum _x\left(\sum _y{f}_{yx}\right)p_{x}log\left(p_x\right)+\sum _y\left(\sum _x{f}_{yx}{p}_x\right)log\left(q_y\right)\hfill \\ & =-\sum _{x\in X\backslash N_p}\sum _{y\in Y\backslash N_{f_x}}{f}_{yx}{p}_{x}log\left(\frac{f_{yx}{p}_x}{q_y}\right). \square \hfill \end{array}$$

Proof of Proposition 2.

(i)

The non-negativity of K follows from Lemma 2 and the equality $q_y={\sum }_{x^{\prime }\in X}{f}_{y{x}^{\prime }}{p}_{x^{\prime }}\ge f_{yx}{p}_x$.

(ii)

This follows from the fact that $H\left(f\right|p)=0$ for all deterministic f.

(iii)

Let $p:•⇝X$ be a probability measure, and let $(Y_x,q^x)\stackrel{Q^x}{⇝}(Y_x^{\prime },q^{\prime x})$ be a collection of morphisms in $\mathbf{FinPS}$ indexed by X. Then the p-weighted convex sum ${⨁}_{x\in X}{p}_x{Q}^x$ is a morphism in $\mathbf{FinPS}$ of the form $(Z,r)\stackrel{h}{⇝}(Z^{\prime },r^{\prime })$, where $Z:={\coprod }_{x\in X}{Y}_x$, $Z^{\prime }:={\coprod }_{x\in X}{Y}_x^{\prime }$, $h:={⨁}_{x\in X}{p}_x{Q}^x$, $r:={⨁}_{x\in X}{p}_x{q}^x$, and $r^{\prime }:={⨁}_{x\in X}{p}_x{q}^{\prime x}$. Then

$$\begin{array}{ccc}\hfill K\left(h\right)& =& -\sum _{z\in Z}{r}_{z}log\left(r_z\right)+\sum _{z^{\prime }\in Z^{\prime }}{r}_{z^{\prime }}^{\prime }log\left(r_{z^{\prime }}^{\prime }\right)-\sum _{z\in Z}\sum _{z^{\prime }\in Z^{\prime }}{r}_z{h}_{z^{\prime }z}log\left(h_{z^{\prime }z}\right)\hfill \\ & =& -\sum _{x\in X}\sum _{y_x\in Y_x}\sum _{y_x^{\prime }\in Y_x^{\prime }}{p}_x{q}_{y_x}^x{Q}_{y_x{y}_x^{\prime }}^{x}log\left(Q_{y_x{y}_x^{\prime }}^x\right)\hfill \\ & =& \sum _{x\in X}{p}_{x}H\left(Q^x|q^x\right),\hfill \end{array}$$

which shows that K is convex linear.

(iv)

Let $\left(X^{\left(n\right)},p^{\left(n\right)}\right)\stackrel{f^{\left(n\right)}}{⇝}\left(Y^{\left(n\right)},q^{\left(n\right)}\right)$ be a sequence (indexed by $n\in \mathbb{N}$) of probability-preserving stochastic maps such that $X^{\left(n\right)}=X$ and $Y^{\left(n\right)}=Y$ for large enough n, and where ${\displaystyle \underset{n\to \infty }{lim}{f}^{\left(n\right)}=f,\underset{n\to \infty }{lim}{p}^{\left(n\right)}=p,}$ and ${\displaystyle \underset{n\to \infty }{lim}{q}^{\left(n\right)}=q}$. Then

$$\underset{n\to \infty }{lim}K\left(f^{\left(n\right)}\right)=-\underset{n\to \infty }{lim}\sum _{x,y}{f}_{yx}^{\left(n\right)}{p}_x^{\left(n\right)}log\left(\frac{f_{yx}^{\left(n\right)}{p}_x^{\left(n\right)}}{{\sum }_{x^{\prime }}{f}_{y{x}^{\prime }}^{\left(n\right)}{p}_{x^{\prime }}^{\left(n\right)}}\right)=K\left(f\right),$$

where the last equality follows from the fact that the limit and sum (which is finite) can be interchanged and all expressions are continuous on $[0,1]$.

(v)

This follows from

$$\begin{array}{cc}\hfill H\left(\vartheta \left(f\right)\right)& =-\sum _{\begin{array}{c}x\in X\\ y\in Y\end{array}}{f}_{yx}{p}_{x}log\left(f_{yx}{p}_x\right)\hfill \\ & =-\sum _{\begin{array}{c}x\in X\\ y\in Y\end{array}}{f}_{yx}{p}_{x}log\left(f_{yx}\right)-\underset{=1}{\underbrace{{\sum }_{y\in Y}{f}_{yx}}}\sum _{x\in X}{p}_{x}log\left(p_x\right)\hfill \\ & =H\left(f\right|p)+H(p)\hfill \end{array}$$

and the fact that $K\left({\pi }_Y\right)=H\left(\vartheta \left(f\right)\right)-H\left(q\right)$. □

Remark 3.

Since conditional entropy vanishes for deterministic morphisms, conditional information loss restricts to $\mathbf{FinPD}$ as the information loss functor. It is important to note that if the term $H\left(f\right|p)$ was not included in the expression for $K\left(f\right)$, then the inequality $K\left(f\right)\ge 0$ would fail in general. When f is deterministic, Baez, Fritz, and Leinster proved $H\left(p\right)-H\left(q\right)\ge 0$. However, when f is stochastic, the inequality $H\left(p\right)-H\left(q\right)\ge 0$ does not hold in general. This has to do with the fact that stochastic maps may increase entropy, whereas deterministic maps always decrease it(while this claim holds in the classical setting as stated, it no longer holds for quantum systems [8]). As such, the term $H\left(f\right|p)$ is needed to retain non-negativity as one attempts to extend BFL’s functor K on $\mathbf{FinPD}$ to a function on $\mathbf{FinPS}$.

Item (v) of Proposition 2 says that the conditional information loss of a map $(X,p)\stackrel{f}{⇝}(Y,q)$ in $\mathbf{FinPS}$ is the information loss of the deterministic map $(X\times Y,\vartheta \left(f\right))\stackrel{{\pi }_Y}{\to }(Y,q)$ in $\mathbf{FinPD}$, so that conditional information loss of a morphism in $\mathbf{FinPS}$ may always be reduced to the information loss of a deterministic map in $\mathbf{FinPD}$ naturally associated with it having the same target. This motivates the following definition.

Definition 16.

A function $F:\mathbf{FinPS}\to \mathbb{B}\mathbb{R}$ isreductiveif and only if $F\left(f\right)=F\left({\pi }_Y\right)$ for every morphism $(X,p)\stackrel{f}{⇝}(Y,q)$ in $\mathbf{FinPS}$ (cf. Proposition 2 item (v) for notation).

Proposition 3

(Reductive characterization of conditional information loss). Let $F:\mathbf{FinPS}\to \mathbb{B}{\mathbb{R}}_{\ge 0}$ be a function satisfying the following conditions.

(i) 

F restricted to $\mathbf{FinPD}$ is functorial, convex linear, and continuous.

(ii) 

F is reductive.

Then F is a non-negative multiple of conditional information loss. Conversely, conditional information loss satisfies conditions (i) and (ii).

Proof. 

This follows immediately from Theorem 1 and item (v) of Proposition 2. □

In what follows, we will characterize conditional information loss without any explicit reference to the subcatgeory $\mathbf{FinPD}$ or the information loss functor of Baez, Fritz, and Leinster. To do this, we first need to develop some machinery.

5. Coalescable Morphisms and Semi-Functoriality

While conditional information loss is not functorial on $\mathbf{FinPS}$, we know it acts functorially on deterministic maps. As such, it is natural to ask for which pairs of composable stochastic maps does the conditional information loss act functorially. In this section, we answer this question, and then we use our result to define a property of functions $\mathbf{FinPS}\to \mathbb{B}\mathbb{R}$ that is a weakening of functoriality, and which we refer to as semi-functoriality. Our definitions are valid in any Markov category (cf. Appendix B).

Definition 17.

A deterministic map $Z\times X\stackrel{h}{\to }Y$ is said to be amediatorfor the composable pair $(X,p)\stackrel{f}{⇝}(Y,q)\stackrel{g}{⇝}(Z,r)$ in $\mathbf{FinPS}$ if and only if

$${(g\circ f)}_{zx}=g_{zh(z,x)}{f}_{h(z,x)x}\mathrm{for}\mathrm{all}(z,x)\in Z\times (X\backslash N_p).$$

(1)

If in fact Equation (1) holds for all $(z,x)\in Z\times X$, then h is said to be astrong mediatorfor the composable pair $X\stackrel{f}{⇝}Y\stackrel{g}{⇝}Z$ inFinStoch.

Remark 4.

Mediators do not exist for general composable pairs, as one can see by considering any composable pair $(•,1)\stackrel{p}{⇝}(X,p)\stackrel{f}{⇝}(Y,q)$ such that $H\left(\vartheta \right(f\left)\right)\ne H\left(q\right)$ (cf. Definitions 7 and 13).

Proposition 4.

Let $(X,p)\stackrel{f}{⇝}(Y,q)\stackrel{g}{⇝}(Z,r)$ be a composable pair of morphisms in $\mathbf{FinPS}$. Then the following statements are equivalent.

(a) 

For every $x\in X\backslash N_p$ and $z\in Z$, there exists at most one $y\in Y$ such that $g_{zy}{f}_{yx}\ne 0$.

(b) 

The pair $(f,g)$ admits a mediator $Z\times X\stackrel{h}{\to }Y$.

(c) 

There exists a function $Z\times X\stackrel{h}{\to }Y$ such that

$$h_{y(z,x)}{(g\circ f)}_{zx}{p}_x=g_{yz}{f}_{yx}{p}_x\forall (z,y,x)\in Z\times Y\times X.$$

(2)

Proof. 

((a)⇒(b)) For every $(z,x)\in Z\times (X\backslash N_p)$ for which such a y exists, set $h(z,x):=y$. If no such y exists or if $x\in N_p$, set $h(z,x)$ to be anything. Then h is a mediator for $(f,g)$.

((b)⇒(c)) Let h be a mediator for $(f,g)$. Since (2) holds automatically for $x\in N_p$, suppose $x\in X\backslash N_p$, in which case (2) is equivalent to $h_{y(z,x)}{(g\circ f)}_{zx}=g_{yz}{f}_{yx}$ for all $(z,y)\in Z\times Y$. This follows from Equation (1) and the fact that h is a function.

((c)⇒(a)) Let $(z,x)\in Z\times (X\backslash N_p)$ and suppose ${(g\circ f)}_{zx}>0$. If h is the mediator, then ${\sum }_{y\in Y}{g}_{zy}{f}_{yx}={(g\circ f)}_{zx}=g_{zh(z,x)}{f}_{h(z,x)x}$. But since $g_{zy}{f}_{yx}=0$ for all $y\ne h(z,x)$, there is only one non-vanishing term in this sum, and it is precisely $g_{zh(z,x)}{f}_{h(z,x)x}$. □

Theorem 2

(Functoriality of Conditional Entropy). Let $(X,p)\stackrel{f}{⇝}(Y,q)\stackrel{g}{⇝}(Z,r)$ be a composable pair of morphisms in $\mathbf{FinPS}$. Then

$$H(g\circ f|p)=H(f\left|p\right)+H\left(g\right|q)$$

(3)

holds if and only if there exists a mediator $Z\times X\stackrel{h}{\to }Y$ for $(X,p)\stackrel{f}{⇝}(Y,q)\stackrel{g}{⇝}(Z,r)$.

We first prove two lemmas.

Lemma 3.

Let $(X,p)\stackrel{f}{⇝}(Y,q)\stackrel{g}{⇝}(Z,r)$ be a pair of composable morphisms. Then

$$H\left((g\times {\mathrm{id}}_Y)\circ {\mathsf{\Delta }}_Y\circ f|p\right)=H\left(g\right|q)+H\left(f\right|p).$$

In particular, $H(g\circ f|p)=H(g\left|q\right)+H\left(f\right|p)$ if and only if $H(g\circ f|p)=H\left((g\times {\mathrm{id}}_Y)\circ {\mathsf{\Delta }}_Y\circ f|p\right)$.

Proof of Lemma 3.

On components, ${\left((g\times {\mathrm{id}}_Y)\circ {\mathsf{\Delta }}_Y\circ f\right)}_{(z,y)x}=g_{zy}{f}_{yx}$. Hence,

$$\begin{array}{cc}\hfill H\left((g\times {\mathrm{id}}_Y)\circ {\mathsf{\Delta }}_Y\circ f|p\right)& =-\sum _x{p}_x\left(\sum _{y,z}{g}_{zy}{f}_{yx}log\left(g_{zy}{f}_{yx}\right)\right)\hfill \\ & =-\sum _{x,y,z}{p}_x{g}_{zy}{f}_{yx}log\left(g_{zy}\right)-\sum _{x,y,z}{p}_x{g}_{zy}{f}_{yx}log\left(f_{yx}\right)\hfill \\ & =-\sum _{y,z}{q}_y{g}_{zy}{f}_{yx}log\left(g_{zy}\right)-\sum _{x,y}{p}_x{f}_{yx}log\left(f_{yx}\right)\hfill \\ & =H\left(g\right|q)+H(f\left|p\right).\hfill \end{array}$$

Note that this equality still holds if $g_{zy}=0$ or $f_{yx}=0$ as each step in this calculation accounted for such possibilities. □

Lemma 4.

Let $(X,p)\stackrel{f}{⇝}(Y,q)\stackrel{g}{⇝}(Z,r)$ be a pair of composable morphisms in $\mathbf{FinPS}$. Then

$$\begin{array}{cc}\hfill 0& \le H\left((g\times {\mathrm{id}}_Y)\circ {\mathsf{\Delta }}_Y\circ f|p\right)-H(g\circ f|p)\hfill \\ \hfill & =-\sum _{x\in X\backslash N_p}\sum _{y\in Y\backslash N_{f_x}}\sum _{z\in Z\backslash N_{g_y}}{p}_x{g}_{zy}{f}_{yx}log\left(\frac{g_{zy}{f}_{yx}}{{\sum }_{y^{\prime }}{g}_{z{y}^{\prime }}{f}_{y^{\prime }x}}\right).\hfill \end{array}$$

(4)

Note that the order of the sums matters in this expression and also note that it is always well-defined since $g_{zy}{f}_{yx}\ne 0$ implies ${(g\circ f)}_{zx}\ne 0$.

Proof of Lemma 4.

For convenience, temporarily set $\aleph :=H\left((g\times {\mathrm{id}}_Y)\circ {\mathsf{\Delta }}_Y\circ f|p\right)-H(g\circ f|p)$. Then

$$\begin{array}{cc}\hfill \aleph & =-\sum _{x,y,z}{p}_x{g}_{zy}{f}_{yx}log\left(g_{zy}{f}_{yx}\right)+\sum _{x,y,z}{p}_x{g}_{zy}{f}_{yx}log\left({(g\circ f)}_{zx}\right)\hfill \\ & =-\sum _{x\in X\backslash N_p}\sum _{y\in Y\backslash N_{f_x}}\sum _{z\in Z\backslash N_{g_y}}{p}_x{g}_{zy}{f}_{yx}log\left(\frac{g_{zy}{f}_{yx}}{{(g\circ f)}_{zx}}\right),\hfill \end{array}$$

which proves the claim due to the definition of the composition of stochastic maps. □

Proof of Theorem 2.

Temporarily set $\aleph :=H\left((g\times {\mathrm{id}}_Y)\circ {\mathsf{\Delta }}_Y\circ f|p\right)-H(g\circ f|p)$. In addition, note that the set of all $x\in X\backslash N_p$ and $z\in Z\backslash N_{{(g\circ f)}_x}$ can be given a more explicit description in terms of the joint distribution $•\stackrel{s:=\gamma \circ \vartheta (g\circ f)}{⇝}Z\times X$ associated with the composite $g\circ f$ and prior p, namely $s_{(z,x)}:={(g\circ f)}_{zx}{p}_x$. Then,

$$\left\{(z,x):x\in X\backslash N_p,z\in Z\backslash N_{{(g\circ f)}_x}\right\}=(Z\times X)\backslash N_s.$$

(5)

(⇒) Suppose $\aleph =0$, which is equivalent to Equation (3) by Lemma 3. Then since each term in the sum from Lemma 4 is non-negative,

$$0=-g_{zy}{f}_{yx}{p}_{x}log\left(\frac{g_{zy}{f}_{yx}}{{\sum }_{y^{\prime }}{g}_{z{y}^{\prime }}{f}_{y^{\prime }x}}\right)\forall x\in X\backslash N_p,y\in Y\backslash N_{f_x},z\in Z\in N_{g_y}.$$

Hence, fix such an $x\in X\backslash N_p,y\in Y\backslash N_{f_x},z\in Z\in N_{g_y}$. The expression here vanishes if and only if

$$g_{zy}{f}_{yx}={(g\circ f)}_{zx},\mathrm{i}.\mathrm{e}.,g_{z{y}^{\prime }}{f}_{y^{\prime }x}=0\forall y^{\prime }\in Y\backslash \left\{y\right\}.$$

(6)

Hence, for every $x\in X\backslash N_p$ and $z\in Z\backslash N_{{(g\circ f)}_x}$, there exists a unique $y\in Y$ such that $g_{zy}{f}_{yx}\ne 0$. But by (5), this means that for every $(z,x)\in (Z\times X)\backslash N_s$, there exists a unique $y\in Y$ such that $g_{zy}{f}_{yx}\ne 0$. This defines a function $(Z\times X)\backslash N_s\to Y\backslash N_q$ which can be extended in an s-a.e. unique manner to a function $Z\times X\stackrel{h}{\to }Y.$

We now show the function h is in fact a mediator for the composable pair $(g,f)$. The equality clearly holds if $x\in N_p$ since both sides vanish. Hence, suppose that $x\in X\backslash N_p$. Given $y\in Y,z\in Z$, the left-hand-side of (2) is given by

$${\delta }_{yh(z,x)}{(g\circ f)}_{zx}{p}_x=\left\{\begin{array}{cc}{g}_{zy}{f}_{yx}{p}_x\hfill & \mathrm{if} z\in Z\backslash N_{{(g\circ f)}_x}\mathrm{and} y=h(z,x)\hfill \\ 0\hfill & \mathrm{otherwise}.\hfill \end{array}\right.$$

by Equation (6). Similarly, if $x\in X\backslash N_p$ and $z\in N_{{(g\circ f)}_x}$, then $g_{zy}{f}_{yx}=0$ for all $y\in Y$ because otherwise ${(g\circ f)}_{zx}{p}_x$ would be nonzero. If instead $z\in Z\backslash N_{{(g\circ f)}_x}$, then $g_{zh(z,x)}{f}_{h(z,x)x}\ne 0$ and $g_{zy}{f}_{yx}=0$ for all $y\in Y\backslash \left\{h\right(z,x\left)\right\}$ by (6). Therefore, (2) holds.

(⇐) Conversely, suppose a mediator h exists and let $X\stackrel{k}{⇝}Z\times Y$ be the stochastic map given on components by $k_{(z,y)x}:=h_{y(z,x)}{(g\circ f)}_{zx}.$ Then

$$\begin{array}{cc}\hfill H\left(g\right|q)+H(f\left|p\right)& =H\left((g\times {\mathrm{id}}_Y)\circ {\mathsf{\Delta }}_Y\circ f|p\right)\mathrm{by}\mathrm{Lemma}3\hfill \\ & =H\left(k\right|p)\mathrm{by}\mathrm{Proposition}4\mathrm{item}\left(\mathrm{c}\right)\hfill \\ & =-\sum _{x\in X\backslash N_p}\sum _{z\in Z\backslash N_{{(g\circ f)}_x}}\sum _{y\in Y}{h}_{y(z,x)}{(g\circ f)}_{zx}{p}_{x}log\left(h_{y(z,x)}{(g\circ f)}_{zx}\right)\hfill \\ & =-\sum _{(z,x)\in (Z\times X)\backslash N_s}\sum _{y\in Y}{\delta }_{yh(z,x)}{(g\circ f)}_{zx}{p}_{x}log\left({\delta }_{yh(z,x)}{(g\circ f)}_{zx}\right)\hfill \\ & =-\sum _{(z,x)\in (Z\times X)\backslash N_s}{(g\circ f)}_{zx}{p}_{x}log\left({(g\circ f)}_{zx}\right)\hfill \\ & =H\left((g\circ f)|p\right),\hfill \end{array}$$

as desired. □

Corollary 1

(Functoriality of Conditional Information Loss). Let $(X,p)\stackrel{f}{⇝}(Y,q)\stackrel{g}{⇝}(Z,r)$ be a composable pair of morphisms in $\mathbf{FinPS}$. Then $K(g\circ f)=K\left(f\right)+K\left(g\right)$ if and only if there exists a mediator $Z\times X\stackrel{h}{\to }Y$ for the pair $(X,p)\stackrel{f}{⇝}(Y,q)\stackrel{g}{⇝}(Z,r)$.

Proof. 

Since the Shannon entropy difference is always functorial, the conditional information loss is functorial on a pair of morphisms if and only if the conditional entropy is functorial on that pair. Theorem 2 then completes the proof. □

Example 2.

In the notation of Theorem 2, suppose that f isa.e. deterministic, which means $f_{yx}={\delta }_{yf\left(x\right)}$ for all $x\in X\backslash N_p$ for some function f (abusive notation is used). In this case, the deviation from functoriality, (4), simplifies to

$$H\left((g\times {\mathrm{id}}_Y)\circ {\mathsf{\Delta }}_Y\circ f|p\right)-H(g\circ f|p)=-\sum _{x\in X\backslash N_p}\sum _{z\in Z}{p}_x{g}_{zf\left(x\right)}log\left(\frac{g_{zf\left(x\right)}}{g_{zf\left(x\right)}}\right)=0.$$

Therefore, if f is p-a.e. deterministic, $H\left(g\right|q)+H(f\left|p\right)=H(g\circ f|p)$. In this case, the mediator $Z\times X\stackrel{h}{\to }Y$ is given by $h:={!}_Z\times f$.

Definition 18.

A pair $(X,p)\stackrel{f}{⇝}(Y,q)\stackrel{g}{⇝}(Z,r)$ of composable morphisms in $\mathbf{FinPS}$ is calleda.e. coalescableif and only if $(X,p)\stackrel{f}{⇝}(Y,q)\stackrel{g}{⇝}(Z,r)$ admits a mediator $Z\times X\stackrel{h}{\to }Y$. Similarly, a pair $X\stackrel{f}{⇝}Y\stackrel{g}{⇝}Z$ of composable morphisms inFinStochis calledcoalescableiff $X\stackrel{f}{⇝}Y\stackrel{g}{⇝}Z$ admits a strong mediator $Z\times X\stackrel{h}{\to }Y$.

Remark 5.

Example 2 showed that if $(X,p)\stackrel{f}{⇝}(Y,q)$ is p-a.e. deterministic, then the pair $(X,p)\stackrel{f}{⇝}(Y,q)\stackrel{g}{⇝}(Z,r)$ is a.e. coalescable for any g. In particular, every pair of composable morphisms in $\mathbf{FinPD}$ is coalescable.

In light of Theorem 2 and Corollary 1, we make the following definition, which will serve as one of the axioms in our later characterizions of both conditional information loss and conditional entropy.

Definition 19.

A function $F:\mathbf{FinPS}\to \mathbb{B}\mathbb{R}$ is said to besemi-functorialiff $F(g\circ f)=F\left(g\right)+F\left(f\right)$ for every a.e. coalescable pair $(X,p)\stackrel{f}{⇝}(Y,q)\stackrel{g}{⇝}(Z,r)$ in $\mathbf{FinPS}$.

Example 3.

By Theorem 2 and Corollary 1, conditional information loss and conditional entropy are both semi-functorial.

Proposition 5.

Suppose $F:\mathbf{FinPS}\to \mathbb{B}\mathbb{R}$ is semi-functorial. Then the restriction of F to $\mathbf{FinPD}$ is functorial. In particular, if F is, in addition, convex linear, continuous, and reductive, then F is a non-negative multiple of conditional information loss.

Proof. 

By Example 2, every pair of composable morphisms in $\mathbf{FinPD}$ is a.e. coalescable. Therefore, F is functorial on $\mathbf{FinPD}$. The second claim then follows from Proposition 3. □

The following lemma will be used in later sections and serves to illustrate some examples of a.e. coalescable pairs.

Lemma 5.

Let $(W,s)\stackrel{e}{\to }(X,p)\stackrel{f}{⇝}(Y,q)\stackrel{g}{\to }(Z,r)$ be a triple of composable morphisms with e deterministic and g invertible. Then each of the following pairs are a.e. coalescable:

(i) 

$(W,s)\stackrel{e}{\to }(X,p)\stackrel{f}{⇝}(Y,q)$

(ii) 

$(X,p)\stackrel{f}{⇝}(Y,q)\stackrel{g}{\to }(Z,r)$

(iii) 

$(W,s)\stackrel{e}{\to }(X,p)\stackrel{g\circ f}{⇝}(Z,r)$

(iv) 

$(W,s)\stackrel{f\circ e}{⇝}(Y,q)\stackrel{g}{\to }(Z,r)$

Proof. 

The proof that $(W,s)\stackrel{e}{\to }(X,p)\stackrel{f}{⇝}(Y,q)$ is coalescable was provided (in a stronger form) in Example 2. To see that $(X,p)\stackrel{f}{⇝}(Y,q)\stackrel{g}{\to }(Z,r)$ is coalescable, note that since g is an isomorphism we have ${(g\circ f)}_{zx}=g_{z{g}^{-1}\left(z\right)}{f}_{g^{-1}\left(z\right)x}$. Thus, $Z\times X\stackrel{g^{-1}\times {!}_X}{\to }Y\times •\cong Y$ is a mediator function for $g\circ f$, thus $g\circ f$ is coalescable. The last two claims follow from the proofs of the first two claims. □

6. Bayesian Inversion

In this section, we recall the concepts of a.e. equivalence and Bayesian inversion phrased in a categorical manner [2,3,9], as they will play a significant role moving forward.

Definition 20.

Let $(X,p)\stackrel{f}{⇝}(Y,q)$ and $(X,p)\stackrel{g}{⇝}(Y,q)$ be two morphisms in $\mathbf{FinPS}$ with the same source and target. Then f and g are said toalmost everywhere equivalent(or p-a.e.equivalent) if and only if $f_{yx}=g_{yx}$ for every $x\in X$ with $p_x\ne 0$. In such a case, the p-a.e. equivalence of f and g will be denoted $f\underset{p}{=}g$.

Theorem 3

(Bayesian Inversion [2,9,10]). Let $(X,p)\stackrel{f}{⇝}(Y,q)$ be a morphism in $\mathbf{FinPS}$. Then there exists a morphism $(Y,q)\stackrel{\overline{f}}{⇝}(X,p)$ such that ${\overline{f}}_{xy}{q}_y=f_{yx}{p}_x$ for all $x\in X$ and $y\in Y$. Furthermore, for any other morphism $(Y,q)\stackrel{{\overline{f}}^{\prime }}{⇝}(X,p)$ satisfying this condition, $\overline{f}\underset{q}{=}{\overline{f}}^{\prime }$.

Definition 21.

The morphism $(Y,q)\stackrel{\overline{f}}{⇝}(X,p)$ appearing in Theorem 3 will be referred to as aBayesian inverseof $(X,p)\stackrel{f}{⇝}(Y,q)$. It follows that ${\overline{f}}_{xy}=p_x{f}_{yx}/q_y$ for all $y\in Y$ with $q_y\ne 0$.

Proposition 6.

Bayesian inversion satisfies the following properties.

(i) 

Suppose $(X,p)\stackrel{f}{⇝}(Y,q)$ and $(X,p)\stackrel{g}{⇝}(Y,q)$ are p-a.e. equivalent, and let $\overline{f}$ and $\overline{g}$ be Bayesian inverses of f and g, respectively. Then $\overline{f}\underset{q}{=}\overline{g}$.

(ii) 

Given two morphisms $(X,p)\stackrel{f}{⇝}(Y,q)$ and $(Y,q)\stackrel{g}{⇝}(X,p)$ in $\mathbf{FinPS}$, then f is a Bayesian inverse of g if and only if g is a Bayesian inverse of f.

(iii) 

Let $(Y,q)\stackrel{\overline{f}}{⇝}(X,p)$ be a Bayesian inverse of $(X,p)\stackrel{f}{⇝}(Y,q)$, and let $\gamma :X\times Y\to Y\times X$ be the swap map (as in Definition 4). Then $\vartheta \left(f\right)=\gamma \circ \vartheta \left(\overline{f}\right)$

(iv) 

Let $(f,g)$ be a composable pair of morphisms in $\mathbf{FinPS}$, and suppose $\overline{f}$ and $\overline{g}$ are Bayesian inverses of f and g respectively. Then $(\overline{g},\overline{f})$ is a composable pair, and $\overline{f}\circ \overline{g}$ is a Bayesian inverse of $g\circ f$.

Proof. 

These are immediate consequences of the categorical definition of a Bayesian inverse (see [3,10,11] for proofs). □

Definition 22.

A contravariant function $\mathcal{B}:\mathbf{FinPS}\to \mathbf{FinPS}$ is said to be aBayesian inversion functorif and only if $\mathcal{B}$ acts as the identity on objects and $\mathcal{B}\left(f\right)$ is a Bayesian inverse of f for all morphisms f in $\mathbf{FinPS}$.

This is mildly abusive terminology since functoriality only holds in the a.e. sense, as explained in the following remark.

Remark 6.

A Bayesian inversion functor exists. Given any $(X,p)\stackrel{f}{⇝}(Y,q)$, set $Y\stackrel{\overline{f}}{⇝}X$ to be given by ${\overline{f}}_{xy}=p_x{f}_{yx}/q_y$ for all $y\in Y$ with $q_y\ne 0$ and ${\overline{f}}_{xy}=1/\left|X\right|$ for all $y\in Y$ with $q_y=0$. Note that this does not define a functor. Indeed, if $(X,p)$ is a probability space with $p_{x_0}=0$ for some $x_0\in X$, then ${\left(\overline{{\mathrm{id}}_X}\right)}_{x_0}$ is the uniform measure on X instead of the Dirac delta measure concentrated on $x_0$. In other words, $\overline{{\mathrm{id}}_X}\ne {\mathrm{id}}_X$. Similar issues of measure zero occur, indicating that $\overline{g\circ f}\ne \overline{f}\circ \overline{g}$ for a composable pair of morphisms $(X,p)\stackrel{f}{⇝}(Y,q)\stackrel{g}{⇝}(Z,r)$. Nevertheless, Bayesian inversion is a.e. functorial in the sense that $\overline{g\circ f}\underset{r}{=}\overline{f}\circ \overline{g}$ and $\overline{{\mathrm{id}}_{(X,p)}}\underset{p}{=}{\mathrm{id}}_{(X,p)}$.

Corollary 2.

${\mathcal{B}}^2\left(f\right)\underset{p}{=}f$ for any Bayesian inversion functor $\mathcal{B}$ and every $(X,p)\stackrel{f}{⇝}(Y,q)$ in $\mathbf{FinPS}$.

Proposition 7.

Let $\mathcal{B}$ be a Bayesian inversion functor on $\mathbf{FinPS}$ (as in Definition 22). Then $\mathcal{B}$ isa.e. convex linearin the sense that

$$\mathcal{B}\left(\underset{x\in X}{⨁}{p}_x{Q}^x\right)\underset{q^{\prime }}{=}\underset{x\in X}{⨁}{p}_x\mathcal{B}\left(Q^x\right),$$

where $q^{\prime }:={⨁}_{x\in X}{p}_x{q}^{\prime x}$ and the other notation is as in Definition 8.

Proof. 

First note that it is immediate that $\mathcal{B}$ is convex linear on objects since Bayesian inversion acts as the identity on objects. Let $p:•⇝X$ be a probability measure, $(Y_x,q^x)\stackrel{Q^x}{⇝}(Y_x^{\prime },q^{\prime x})$ be a collection of morphisms in $\mathbf{FinPS}$ indexed by X, and suppose $\mathcal{B}$ is a Bayesian inversion functor. Then for $(z,z^{\prime })\in Y_x\times Y_x^{\prime }$ with $p_x{q}_{z^{\prime }}^{\prime x}\ne 0$, we have

$$\begin{array}{ccc}\hfill \mathcal{B}{\left(\underset{x\in X}{⨁}{p}_x{Q}^x\right)}_{z{z}^{\prime }}& =& \frac{p_x{q}_z^x{\left({⨁}_{x\in X}{p}_x{Q}^x\right)}_{z^{\prime }z}}{p_x{q}_{z^{\prime }}^{\prime x}}=\frac{q_z^x\left(p_x{Q}_{z^{\prime }z}^x\right)}{q_{z^{\prime }}^{\prime x}}=p_x\left(\frac{q_z^x{Q}_{z^{\prime }z}^x}{q_{z^{\prime }}^{\prime x}}\right)\hfill \\ & =& p_x\mathcal{B}{\left(Q^x\right)}_{z{z}^{\prime }}={\left(\underset{x\in X}{⨁}{p}_x\mathcal{B}\left(Q^x\right)\right)}_{z{z}^{\prime }}.\hfill \end{array}$$

Thus, $\mathcal{B}$ is a.e. convex linear. □

Proposition 8.

Given $(X,p)\stackrel{f}{⇝}(Y,q)\stackrel{g}{⇝}(Z,r)$ in $\mathbf{FinPS}$, and let $\overline{f}$ and $\overline{g}$ be Bayesian inverses of f and g respectively. Then $(f,g)$ is a.e. coalescable if and only if $(\overline{g},\overline{f})$ is a.e. coalescable.

Proof. 

Since Bayesian inversion is a dagger functor on a.e. equivalence classes ([3], (Remark 13.10)), it suffices to prove one direction in this claim. Hence, suppose $(f,g)$ is a.e. coalescable and let h be a mediator function realizing this. Then $h\circ \gamma$ is a mediator for $(\overline{g},\overline{f})$ because

$$\begin{array}{cc}\hfill {\overline{f}}_{xy}{\overline{g}}_{yz}{r}_z& ={\overline{f}}_{xy}{g}_{zy}{q}_y=g_{zy}{f}_{yx}{p}_x=h_{y(z,x)}{(g\circ f)}_{zx}{p}_x\hfill \\ & =h_{y(z,x)}{\overline{(g\circ f)}}_{xz}{r}_z={(h\circ \gamma )}_{y(x,z)}{\overline{(g\circ f)}}_{xz}{r}_z.\hfill \end{array}$$

A completely string-diagrammatic proof is provided in Appendix B. □

The following proposition is a reformulation of the conditional entropy identity $H\left(\mathcal{Y}\right|\mathcal{X})+H(\mathcal{X})=H(\mathcal{X}\left|\mathcal{Y}\right)+H\left(\mathcal{Y}\right)$ in terms of Bayesian inversion.

Proposition 9.

Let $(X,p)\stackrel{f}{⇝}(Y,q)$ be a morphism in $\mathbf{FinPS}$, and suppose $\overline{f}$ is a Bayesian inverse of f. Then

$$H\left(f\right|p)+H\left(p\right)=H\left(\overline{f}\right|q)+H\left(q\right).$$

(7)

Proof. 

This follows from the fact that both sides of (7) are equal to $H\left(\vartheta \right(f\left)\right)$. □

Proposition 9 implies Bayesian inversion takes conditional entropy to conditional information loss and vice versa, which is formally stated as follows.

Corollary 3.

Let $K:\mathbf{FinPS}\to \mathbb{B}{\mathbb{R}}_{\ge 0}$ and $H:\mathbf{FinPS}\to \mathbb{B}{\mathbb{R}}_{\ge 0}$ be given by conditional information loss and conditional entropy, respectively, and let $\mathcal{B}:\mathbf{FinPS}\to \mathbf{FinPS}$ be a Bayesian inversion functor. Then, $H=K\circ \mathcal{B}$ and $K=H\circ \mathcal{B}$.

Remark 7.

If $(X,p)\stackrel{f}{\to }(Y,q)$ is a deterministic morphism in $\mathbf{FinPS}$, Baez, Fritz, and Leinster point out that the information loss of f is in fact the conditional entropy of x given y [1]. Here, we see this duality as a special case of Corollary 3 applied to deterministic morphisms.

7. Bloom-Shriek Factorization

We now introduce a simple, but surprisingly useful, factorization for every morphism in $\mathbf{FinPS}$, and we use it to prove some essential lemmas for our characterization theorems for conditional information loss and conditional entropy, which appear in the following sections.

Definition 23.

Given a stochastic map $X\stackrel{f}{⇝}Y$, thebloom of fis the stochastic map $X\stackrel{{¡}_f}{⇝}X\times Y$ given by the composite $X\stackrel{{\mathsf{\Delta }}_X}{⇝}X\times X\stackrel{{\mathrm{id}}_X\times f}{⇝}X\times Y$, and theshriek of fis the deterministic map $X\times Y\stackrel{{!}_f}{\to }X$ given by the projection ${\pi }_X$.

Proposition 10.

Let $(X,p)\stackrel{f}{⇝}(Y,q)$ be a morphism in $\mathbf{FinPS}$. Then the following statement hold.

(i) 

The composite $(X,p)\stackrel{{¡}_f}{⇝}(X\times Y,\vartheta \left(f\right))\stackrel{{!}_f}{\to }(X,p)$ is equal to the identity $i{d}_X$.

(ii) 

The morphism f equals the composite $(X,p)\stackrel{{¡}_f}{⇝}(X\times Y,\vartheta \left(f\right))\stackrel{{!}_{\overline{f}}\circ \gamma }{\to }(Y,q)$, where $\overline{f}$ denotes any Bayesian inverse of f and $\gamma :X\times Y\to Y\times X$ is the swap map.

(iii) 

The pair $(X,p)\stackrel{{¡}_f}{⇝}(X\times Y,\vartheta \left(f\right))\stackrel{{!}_{\overline{f}}\circ \gamma \equiv {\pi }_Y}{\to }(Y,q)$ is coalescable.

Definition 24.

The decomposition in item (ii) Proposition 10 will be referred to as thebloom-shriek factorizationof f.

Proof of Propostion 10.

Element-wise proofs are left as exercises. Appendix B contains an abstract proof using string diagrams in Markov categories. □

The bloom of f can be expressed as a convex combination of simpler morphisms up to isomorphism. To describe this and its behavior under convex linear semi-functors, we introduce the notion of an invariant and examine some of its properties.

Definition 25.

A function $F:\mathbf{FinPS}\to \mathbb{B}\mathbb{R}$ is said to be aninvariantif and only if for every triple of composable morphisms $(W,s)\stackrel{e}{\to }(X,p)\stackrel{f}{⇝}(Y,q)\stackrel{g}{\to }(Z,r)$ such that e and g are isomorphisms, then $F\left(f\right)=F(g\circ f\circ e)$.

Lemma 6.

If a function $F:\mathbf{FinPS}\to \mathbb{B}{\mathbb{R}}_{\ge 0}$ is semi-functorial, then F is an invariant.

Proof. 

Consider a composable triple $(W,s)\stackrel{e}{\to }(X,p)\stackrel{f}{⇝}(Y,q)\stackrel{g}{\to }(Z,r)$ such that e and g are isomorphisms. Then

$$F\left(g\circ f\circ e\right)=F\left(g\circ (f\circ e)\right)=F\left(g\right)+F(f\circ e)=F\left(g\right)+F\left(f\right)+F\left(e\right)$$

by Lemma 5. Secondly, since g and e are isomorphisms, and since the pairs $(g,g^{-1})$ and $(e,e^{-1})$ are coalescable, $F\left(\mathrm{id}\right)=F(g\circ g^{-1})=F\left(g\right)+F\left(g^{-1}\right)$. But since $F\left(\mathrm{id}\right)=0$ (by semi-functoriality), this requires that $F\left(g\right)=0$ for an isomorphism g since $F\left(g\right)\ge 0$ and $F\left(g^{-1}\right)\ge 0$. The same is true for $F\left(e\right)$. Hence, $F\left(g\circ f\circ e\right)=F\left(f\right)$. □

Lemma 7.

Let $(X,p)\stackrel{f}{⇝}(Y,q)$ be a morphism in $\mathbf{FinPS}$, and suppose $F:\mathbf{FinPS}\to \mathbb{B}{\mathbb{R}}_{\ge 0}$ is semi-functorial and convex linear. Then the following statements hold.

(i) 

$F\left(f\right)=F\left({!}_{\overline{f}}\right)+F\left({¡}_f\right)$

(ii) 

$F\left({¡}_f\right)={\sum }_{x\in X}{p}_{x}F\left({¡}_{f_x}\right)$

(iii) 

$F\left({!}_f\right)={\sum }_{x\in X}{p}_{x}F\left({!}_{f_x}\right)$

Proof. 

For item (i), we have

$$\begin{array}{cccc}\hfill F\left(f\right)& =F({!}_{\overline{f}}\circ \gamma \circ {¡}_f)\hfill & \hfill & \mathrm{by}\mathrm{item}\left(\mathrm{ii}\right)\mathrm{of}\mathrm{Proposition}10\hfill \\ \hfill & =F({!}_{\overline{f}}\circ \gamma )+F\left({¡}_f\right)\hfill & \hfill & \mathrm{by}\mathrm{item}\left(\mathrm{iii}\right)\mathrm{of}\mathrm{Proposition}10\hfill \\ \hfill & =F\left({!}_{\overline{f}}\right)+F\left({¡}_f\right)\hfill & \hfill & \mathrm{by}\mathrm{Lemma}6.\hfill \end{array}$$

For items (ii) and (iii), note that ${¡}_f$ and ${!}_f$ can be expressed as composites of isomorphisms and certain convex combinations, namely

(8)

Hence,

$$\begin{array}{cccccc}\hfill F\left({¡}_f\right)& =F(\phi \circ \ell \circ \psi )\hfill & \hfill F\left({!}_f\right)& =F({\phi }^{\prime }\circ {\ell }^{\prime }\circ {\psi }^{\prime })\hfill & \hfill & \mathrm{by}\left(8\right)\hfill \\ \hfill & =F(\ell )\hfill & \hfill & =F\left({\ell }^{\prime }\right)\hfill & \hfill & \mathrm{by}\mathrm{Lemma}6\hfill \\ \hfill & =\sum _{x\in X}{p}_{x}F\left({¡}_{f_x}\right)\hfill & \hfill & =\sum _{x\in X}{p}_{x}F\left({!}_{f_x}\right)\hfill & \hfill & \mathrm{since}F\mathrm{is}\mathrm{convex}\mathrm{linear}. \square \hfill \end{array}$$

Proposition 11.

Suppose $F:\mathbf{FinPS}\to \mathbb{B}{\mathbb{R}}_{\ge 0}$ is semi-functorial and convex linear. If $f,g:(X,p)⇝(Y,q)$ are two morphisms in $\mathbf{FinPS}$ such that $f\underset{p}{=}g$, then $F\left(f\right)=F\left(g\right)$.

Proof. 

Suppose $(X,p)\stackrel{f}{⇝}(Y,q)$ and $(X,p)\stackrel{g}{⇝}(Y,q)$ are such that $f\underset{p}{=}g$, and let $\overline{f}$ and $\overline{g}$ be Bayesian inverses for f and g. Then

$$\begin{array}{cccc}\hfill F\left(f\right)& =F\left({!}_{\overline{f}}\right)+F\left({¡}_f\right)\hfill & \hfill & \mathrm{by}\mathrm{item}\left(\mathrm{i}\right)\mathrm{of}\mathrm{Lemma}7\hfill \\ \hfill & =\sum _{y\in Y}{q}_{y}F\left({!}_{{\overline{f}}_y}\right)+\sum _{x\in X}{p}_{x}F\left({¡}_{f_x}\right)\hfill & \hfill & \mathrm{by}\mathrm{items}\left(\mathrm{ii}\right)\mathrm{and}\left(\mathrm{iii}\right)\mathrm{of}\mathrm{Lemma}7\hfill \\ \hfill & =\sum _{y\in Y}{q}_{y}F\left({!}_{{\overline{g}}_y}\right)+\sum _{x\in X}{p}_{x}F\left({¡}_{g_x}\right)\hfill & \hfill & \mathrm{since}f\underset{p}{=}g\mathrm{and}\overline{f}\underset{q}{=}\overline{g}\hfill \\ \hfill & =F\left({!}_{\overline{g}}\right)+F\left({¡}_g\right)\hfill & \hfill & \mathrm{by}\mathrm{items}\left(\mathrm{ii}\right)\mathrm{and}\left(\mathrm{iii}\right)\mathrm{of}\mathrm{Lemma}7\hfill \\ \hfill & =F\left(g\right)\hfill & \hfill & \mathrm{by}\mathrm{item}\left(\mathrm{i}\right)\mathrm{of}\mathrm{Lemma}7,\hfill \end{array}$$

as desired. □

8. An Intrinsic Characterization of Conditional Information Loss

Theorem 4.

Suppose $F:\mathbf{FinPS}\to \mathbb{B}{\mathbb{R}}_{\ge 0}$ is a function satisfying the following conditions.

1. 

F is semi-functorial.

2. 

F is convex linear.

3. 

F is continuous.

4. 

$F\left({¡}_p\right)=0$ for every probability distribution $•\stackrel{p}{⇝}X$.

Then F is a non-negative multiple of conditional information loss. Conversely, conditional information loss satisfies conditions 1–4.

Proof. 

Suppose F satisfies conditions 1–4, let $(X,p)\stackrel{f}{⇝}(Y,q)$ be an arbitrary morphism in $\mathbf{FinPS}$, and let $\gamma :Y\times X\to X\times Y$ be the swap map, so that ${!}_{\overline{f}}={\pi }_Y\circ \gamma$. Then

$$\begin{array}{cccc}\hfill F\left(f\right)& =F\left({!}_{\overline{f}}\right)+F\left({¡}_f\right)\hfill & \hfill & \mathrm{by}\mathrm{item}\left(\mathrm{i}\right)\mathrm{of}\mathrm{Lemma}7\hfill \\ \hfill & =F\left({!}_{\overline{f}}\right)+\sum _{x\in X}{p}_{x}F\left({¡}_{f_x}\right)\hfill & \hfill & \mathrm{by}\mathrm{item}\left(\mathrm{ii}\right)\mathrm{of}\mathrm{Lemma}7\hfill \\ \hfill & =F\left({!}_{\overline{f}}\right)\hfill & \hfill & \mathrm{by}\mathrm{condition}4\hfill \\ \hfill & =F\left({\pi }_Y\circ \gamma \right)\hfill \\ \hfill & =F\left({\pi }_Y\right)\hfill & \hfill & \mathrm{since}F\mathrm{is}\mathrm{an}\mathrm{invariant}\mathrm{by}\mathrm{Lemma}6.\hfill \end{array}$$

Thus, F is reductive (see Definition 16) and Proposition 5 applies. □

Remark 8.

Under the assumption that $F:\mathbf{FinPS}\to \mathbb{B}\mathbb{R}$ is semi-functorial and convex linear, one may show F satisfies condition 4 in Theorem 4 if and only if F is reductive (see Definition 16 and Proposition 5). While the reductive axiom specifies how the semi-functor acts on all morphisms in $\mathbf{FinPS}$, condition 4 in Theorem 4 only specifies how it acts on morphisms from the initial object. This gives not just a simple mathematical criterion, but one with a simple intuitive interpretation as well. Namely, condition 4 says that if a process begins with no prior information, then there is no information to be lost in the process.

We now use Theorem 4 and Bayesian inversion to prove a statement dual to Theorem 4.

Theorem 5.

Suppose $F:\mathbf{FinPS}\to \mathbb{B}{\mathbb{R}}_{\ge 0}$ is a function satisfying the following conditions.

1. 

F is semi-functorial.

2. 

F is convex linear.

3. 

F is continuous.

4. 

$F\left({!}_p\right)=0$ for every probability distribution $•\stackrel{p}{⇝}X$.

Then F is a non-negative multiple of conditional entropy. Conversely, conditional entropy satisfies conditions 1–4.

Before giving a proof, we introduce some terminology and prove a few lemmas. We also would like to point out that condition 4 may be given an operational interpretation as follows: if a communication channel has a constant output, then it has no conditional entropy.

Definition 26.

Let $F:\mathbf{FinPS}\to \mathbb{B}\mathbb{R}$ be a function and let $\mathcal{B}$ be a Bayesian inversion functor. Then $\overline{F}:=F\circ \mathcal{B}$ will be referred to as aBayesian reflectionof F.

Remark 9.

By Proposition 11, if $F:\mathbf{FinPS}\to \mathbb{B}\mathbb{R}$ is a convex linear semi-functor, then a Bayesian reflection is independent of the choice of a Bayesian inversion functor, and as such, is necessarily unique.

Lemma 8.

Let $(X,p)\stackrel{f}{⇝}(Y,q)$ be a morphism in $\mathbf{FinPS}$, suppose $F:\mathbf{FinPS}\to \mathbb{B}{\mathbb{R}}_{\ge 0}$ is a convex linear semi-functor, and let $\overline{f}$ be a Bayesian inverse of f. Then $\overline{F}\left(f\right)=F\left(\overline{f}\right)$.

Proof of Lemma 8.

Let $\mathcal{B}$ be a Bayesian inversion functor, so that $\mathcal{B}\left(f\right)\underset{q}{=}\overline{f}$. Then $\overline{F}\left(f\right)=F\left(\mathcal{B}\left(f\right)\right)=F\left(\overline{f}\right),$ where the last equality follows from Proposition 11. □

Lemma 9.

Let $\mathcal{B}:\mathbf{FinPS}\to \mathbf{FinPS}$ be a Bayesian inversion functor and let $(X_n,p_n)\stackrel{f_n}{⇝}(Y_n,q_n)$ be a sequence of morphisms in $\mathbf{FinPS}$ converging to $(X,p)\stackrel{f}{⇝}(Y,q)$. Then ${\displaystyle \underset{n\to \infty }{lim}\mathcal{B}\left(f_n\right)\underset{q}{=}\mathcal{B}\left(f\right)}$.

Proof of Lemma 9.

Set $f^{\left(n\right)}:=f_n$. For all $y\in Y$ with $q_y\ne 0$, we have

$$\underset{n\to \infty }{lim}\mathcal{B}{\left(f^{\left(n\right)}\right)}_{xy}=\underset{n\to \infty }{lim}\frac{p_x^{\left(n\right)}{f}_{yx}^{\left(n\right)}}{q_y^{\left(n\right)}}=\frac{p_x{f}_{yx}}{q_y}=\mathcal{B}{\left(f\right)}_{xy}. \square$$

Lemma 10.

Suppose $F:\mathbf{FinPS}\to \mathbb{B}{\mathbb{R}}_{\ge 0}$ is a function satisfying conditions 1–4 of Theorem 5. Then the Bayesian reflection $\overline{F}$ is a non-negative multiple of conditional information loss.

Proof of Lemma 10.

We show $\overline{F}$ satisfies conditions 1–4 of Theorem 4. Throughout the proof, let $\mathcal{B}$ denote a Bayesian inversion functor, so that $\overline{F}=F\circ \mathcal{B}$.

Semi-functoriality: Suppose $(X,p)\stackrel{f}{⇝}(Y,q)\stackrel{g}{⇝}(Z,r)$ is an a.e. coalescable pair of composable morphisms in $\mathbf{FinPS}$. Then

$$\begin{array}{cccc}\hfill \overline{F}(g\circ f)& =F\left(\overline{g\circ f}\right)\hfill & \hfill & \mathrm{by}\mathrm{Lemma}8\hfill \\ \hfill & =F\left(\overline{f}\circ \overline{g}\right)\hfill & \hfill & \mathrm{by}\mathrm{item}\left(\mathrm{iv}\right)\mathrm{of}\mathrm{Proposition}6\hfill \\ \hfill & =F\left(\overline{f}\right)+F\left(\overline{g}\right)\hfill & \hfill & \mathrm{by}\mathrm{Proposition}8\hfill \\ \hfill & =\overline{F}\left(f\right)+\overline{F}\left(g\right)\hfill & \hfill & \mathrm{by}\mathrm{Lemma}8.\hfill \end{array}$$

Thus, $\overline{F}$ is semi-functorial.

Convex Linearity: Given any probability space $(X,p)$ and a family of morphisms $(Y_x,q^x)\stackrel{Q^x}{⇝}(Y_x^{\prime },q^{\prime x})$ in $\mathbf{FinPS}$ indexed by X,

$$\begin{array}{cccc}\hfill \overline{F}\left(\underset{x\in X}{⨁}{p}_x{Q}^x\right)& =F\left(\underset{x\in X}{⨁}{p}_x\mathcal{B}\left(Q^x\right)\right)\hfill & \hfill & \mathrm{by}\mathrm{Propositions}7\mathrm{and}11\hfill \\ \hfill & =\sum _{x\in X}{p}_{x}F\left(\mathcal{B}\left(Q^x\right)\right)\hfill & \hfill & \sin \mathrm{ce}F\mathrm{is}\mathrm{convex}\mathrm{linear}\hfill \\ \hfill & =\sum _{x\in X}\overline{F}\left(Q^x\right)\hfill & \hfill & \mathrm{by}\mathrm{definition}\mathrm{of}\overline{F}.\hfill \end{array}$$

Thus, $\overline{F}$ is convex linear.

Continuity: This follows from Lemma 9 and Proposition 11.

$\overline{F}\left({¡}_p\right)=0$ for every probability distribution $•\stackrel{p}{⇝}X$: This follows from Lemma 8, since ${!}_p$ is the unique Bayesian inverse of ${¡}_p$. □

Proof of Theorem 5.

Suppose $F:\mathbf{FinPS}\to \mathbb{B}\mathbb{R}$ is a function satisfying conditions 1–4 of Theorem 5, and let $\mathcal{B}$ be a Bayesian inversion functor. Since F is semi-functorial and convex linear it follows from Proposition 11 that $F=\overline{F}\circ \mathcal{B}$, and by Lemma 10 it follows that $\overline{F}=cK$ for some non-negative constant $c\ge 0$. We then have $F=\overline{F}\circ \mathcal{B}=cK\circ \mathcal{B}=cH$. Thus, F is a non-negative multiple of conditional entropy. □

9. A Bayesian Characterization of Conditional Entropy

We now prove a reformulation of Theorem 5, where condition 4 is replaced by a condition that we view as an ‘entropic Bayes’ rule’.

Definition 27.

A function $F:\mathbf{FinPS}\to \mathbb{B}\mathbb{R}$ satisfies anentropic Bayes’ ruleif and only if

$$F\left(f\right)+F\left({¡}_p\right)=F\left(\overline{f}\right)+F\left({¡}_q\right)$$

for every morphism $(X,p)\stackrel{f}{⇝}(Y,q)$ in $\mathbf{FinPS}$ and any Bayesian inverse $\overline{f}$ of f.

Remark 10.

The entropic Bayes’ rule is an abstraction of the conditional entropy identity 7.

Theorem 6

(A Bayesian characterization of conditional entropy). Suppose $F:\mathbf{FinPS}\to \mathbb{B}{\mathbb{R}}_{\ge 0}$ is a function satisfying the following conditions.

1. 

F is semi-functorial.

2. 

F is convex linear.

3. 

F is continuous.

4. 

F satisfies an entropic Bayes’ rule.

Then F is a non-negative multiple of conditional entropy. Conversely, conditional entropy satisfies conditions 1–4.

Proof. 

By Theorem 5, it suffices to show $F\left({!}_p\right)=0$ for every object $(X,p)$ in $\mathbf{FinPS}$. For this, first note that ${¡}_1={¡}_1\circ {¡}_1$, where $(•,1)$ is the point-mass distribution on a single point. Since F is semi-functorial and ${¡}_1\circ {¡}_1$ is coalescable, we have $F\left({¡}_1\right)=F\left({¡}_1\right)+F\left({¡}_1\right)$, which implies $F\left({¡}_1\right)=0$. Applying the entropic Bayes’ rule from Definition 27 to the morphism ${!}_p:(X,p)⇝(•,1)$ yields

$$F\left({!}_p\right)+F\left({¡}_p\right)=F\left({¡}_p\right)+F\left({¡}_1\right)\Rightarrow F\left({!}_p\right)=F\left({¡}_1\right)=0,$$

as desired. □

Remark 11.

In ([6], slide 21), Fritz asked if there is a Markov category for information theory explaining the analogy between Bayes’ rule $\mathbb{P}\left(A\right|B\left)\mathbb{P}\right(B)=\mathbb{P}(B\left|A\right)\mathbb{P}\left(A\right)$ and the conditional entropy identity $H\left(A\right|B)+H(B)=H(B\left|A\right)+H\left(A\right)$. In light of our work, we feel we have an adequate categorical explanation for this analogy, which we now explain.

Let $(X,p)\stackrel{f}{⇝}(Y,q)$ be an arbitrary morphism in $\mathbf{FinPS}$, and suppose $F:\mathbf{FinPS}\to \mathbb{B}{\mathbb{R}}_{\ge 0}$ is semi-functorial. Then the commutative diagram (cf. Definition A4)

(9)

is a coalescable square (where γ is the swap map), i.e., ${¡}_f\circ {¡}_p$ and $(\gamma \circ {¡}_{\overline{f}})\circ {¡}_q$ are both coalescable. The semi-functoriality of F then implies the identity $F\left({¡}_f\right)+F\left({¡}_p\right)=F\left({¡}_{\overline{f}}\right)+F\left({¡}_q\right)$. Now suppose—as in the case of conditional entropy—that F satisfies the further condition that $F\left(f\right)=F\left({¡}_f\right)$. Then commutivity of (9) and this are equivalent to the following two respective equations:

$$\begin{array}{rc}\underset{\_}{\mathit{Bayes}’ \mathit{Rule}}:& f_{yx}{p}_x={\overline{f}}_{xy}{q}_y\\ \underset{\_}{\mathit{Entropic} \mathit{Bayes}’ \mathit{Rule}}:& F(f)+F\left({¡}_p\right)=F(\overline{f})+F\left({¡}_q\right).\end{array}$$

In the case that $F=H$, where H is the conditional entropy, we have $H\left({¡}_r\right|1)=H\left(r\right)$ for every object $(Z,r)$ in $\mathbf{FinPS}$ (where $H\left(r\right)$ is the Shannon entropy). Thus, the entropic Bayes’ rule becomes $H\left(f\right|p)+H\left(p\right)=H\left(\overline{f}\right|q)+H\left(q\right)$, which is the classical identity for conditional entropy.

10. Concluding Remarks

In this paper, we have provided novel characterizations of conditional entropy and the information loss of a probability-preserving stochastic map. The constructions we introduced to prove our main results are general enough to be applicable in the recent framework of synthetic probability [3]. By weakening functoriality and finding the appropriate substitute that we call semi-functoriality, we have shown how certain aspects of quantitative information theory can be done in this categorical framework. In particular, we have illustrated how Bayes’ rule can be formulated entropically and used as an axiom in characterizing conditional entropy.

Immediate questions from our work arise, such as the extendibility of conditional entropy to other Markov categories or even quantum Markov categories [10]. Work in this direction might offer a systematic approach towards conditional entropy in quantum mechanics. It would also be interesting to see what other quantitative features of information theory can be described from such a perspective, or if new ones will emerge.