Conditional Rényi Entropy and the Relationships between Rényi Capacities

Abstract

The analogues of Arimoto’s definition of conditional Rényi entropy and Rényi mutual information are explored for abstract alphabets. These quantities, although dependent on the reference measure, have some useful properties similar to those known in the discrete setting. In addition to laying out some such basic properties and the relations to Rényi divergences, the relationships between the families of mutual informations defined by Sibson, Augustin-Csiszár, and Lapidoth-Pfister, as well as the corresponding capacities, are explored.

1. Introduction

Shannon’s information measures (entropy, conditional entropy, the Kullback divergence or relative entropy, and mutual information) are ubiquitous both because they arise as operational fundamental limits of various communication or statistical inference problems, and because they are functionals that have become fundamental in the development and advancement of probability theory itself. Over a half century ago, Rényi [1] introduced a family of information measures extending those of Shannon [2], parametrized by an order parameter $\alpha \in [0,\infty ]$. Rényi’s information measures are also fundamental– indeed, they are (for $\alpha >1$) just monotone functions of $L^p$-norms, whose relevance or importance in any field that relies on analysis need not be justified. Furthermore, they show up in probability theory, PDE, functional analysis, additive combinatorics, and convex geometry (see, e.g., [3,4,5,6,7,8,9]), in ways where understanding them as information measures instead of simply as monotone functions of $L^p$-norms is fruitful. For example, there is an intricate story of parallels between entropy power inequalities (see, e.g., [10,11,12]), Brunn-Minkowski-type volume inequalities (see, e.g., [13,14,15]) and sumset cardinality (see, e.g., [16,17,18,19,20]), which is clarified by considering logarithms of volumes and Shannon entropies as members of the larger class of Rényi entropies. It is also recognized now that Rényi’s information measures show up as fundamental operational limits in a range of information-theoretic or statistical problems (see, e.g., [21,22,23,24,25,26,27]). Therefore, there has been considerable interest in developing the theory surrounding Rényi’s information measures (which is far less well developed than the Shannon case), and there has been a steady stream of recent papers [27,28,29,30,31,32,33,34,35,36] elucidating their properties beyond the early work of [37,38,39,40]. This paper, part of which was presented at ISIT 2019 [41], is a further contribution along these lines.

More specifically, three notions of Rényi mutual information have been considered in the literature (usually named after Sibson, Arimoto and Csiszár) for discrete alphabets. Sibson’s definition has also been considered for abstract alphabets, but Arimoto’s definition has not. Indeed Verdú [31] asserts: “One shortcoming of Arimoto’s proposal is that its generalization to non-discrete alphabets is not self-evident.” The reason it is not self-evident is because although there is an obvious generalized definition, the mutual information arising from this notion depends on the choice of reference measure on the abstract alphabet, which is not a desirable property. Nonetheless, the perspective taken in this note is that it is still interesting to develop the properties of the abstract Arimoto conditional Rényi entropy keeping in mind the dependence on reference measure. The Sibson definition is then just a special case of the Arimoto definition where we choose a particular, special reference measure.

Our main motivation comes from considering various notions of Rényi capacity. While certain equivalences have been shown between various such notions by Csiszár [21] for finite alphabets and Nakiboğlu [36,42] for abstract alphabets, the equivalences and relationships are further extended in this note.

This paper is organized in the following manner. In Section 2 below we begin by defining conditional Rényi entropy for random variables taking values in a Polish space. Section 3 presents a variational formula for the conditional Rényi entropy in terms of Rényi divergence, which will be a key ingredient in several results later. Basic properties that the abstract conditional Rényi entropy satisfies akin to its discrete version are proved in Section 4, including description for special orders $0,1$ and ∞, monotonicity in the order, reduction of entropy upon conditioning, and a version of the chain rule. Section 5 discusses and compares several notions of $\alpha$-mutual information. The various notions of channel capacity arising out of different notions of $\alpha$-mutual information are studied in Section 6, which are then compared using results from the preceding section.

2. Definition of Conditional Rényi Entropies

Let S be a Polish space and ${\mathcal{B}}_S$ its Borel $\sigma$-algebra. We fix a $\sigma$-finite reference measure $\gamma$ on $(S,{\mathcal{B}}_S)$. Our study of entropy and in particular all $L^p$ spaces we talk about will be with respect to this measure space, unless stated otherwise.

Definition 1.

Let X be an S-valued random variable with density f with respect to γ. We define the Rényi entropy of X of order$\alpha \in (0,1)\cup (1,\infty )$by

$$h_{\alpha }^{\gamma }\left(X\right)=\frac{\alpha }{1-\alpha }\log {\parallel f\parallel }_{\alpha }.$$

It will be convenient to write down Rényi entropy as

$$h_{\alpha }^{\gamma }\left(X\right)=-\log {\mathrm{Ren}}_{\alpha }^{\gamma }\left(X\right),$$

where ${\mathrm{Ren}}_{\alpha }^{\gamma }\left(X\right)={\parallel f\parallel }_{\alpha }^{\frac{\alpha }{\alpha -1}}$ will be called the Rényi probability of order $\alpha$ of X.

Let T be another Polish space with a fixed measure $\eta$ on its Borel $\sigma$-algebra ${\mathcal{B}}_T$. Now suppose $X,Y$ are, resepectively, $S,T$-valued random variables with a joint density $F:S\times T\to \mathbb{R}$ w.r.t. the reference $\gamma \otimes \eta$. We will denote the marginals of F on S and T by f and g respectively. This in particular means that X has density f w.r.t. $\gamma$ and Y has density g w.r.t. $\eta$. Just as Rényi probability of X, one can define the Rényi probability of the conditional X given $y=y$ by the expression ${\mathrm{Ren}}_{\alpha }^{\gamma }\left(X\right|Y=y)={\parallel \frac{F(·,y)}{g\left(y\right)}\parallel }_{\alpha }^{\frac{\alpha }{\alpha -1}}$. The following generalizes ([30], Definition 2).

Definition 2.

Let$\alpha \in (0,1)\cup (1,\infty )$. We define the conditional Rényi entropy $h_{\alpha }^{\gamma }\left(X\right|Y)$in terms of a weighted mean of conditional Rényi probabilities${\mathrm{Ren}}_{\alpha }^{\gamma }\left(X\right|Y=y)$,

$$h_{\alpha }^{\gamma }\left(X\right|Y)=-\log {\mathrm{Ren}}_{\alpha }^{\gamma }\left(X\right|Y),$$

where

$$\begin{array}{c}{\mathrm{Ren}}_{\alpha }^{\gamma }\left(X\right|Y)={\left({\int }_T{\mathrm{Ren}}_{\alpha }^{\gamma }{\left(X\right|Y=y)}^{\frac{\alpha -1}{\alpha }}\mathrm{d}{\mathbb{P}}_Y\right)}^{\frac{\alpha }{\alpha -1}}\hfill \end{array}$$

We can re-write ${\mathrm{Ren}}_{\alpha }^{\gamma }\left(X\right|Y)$ as

$${\mathrm{Ren}}_{\alpha }^{\gamma }\left(X\right|Y)={\left({\int }_{\mathrm{supp}\left({\mathbb{P}}_Y\right)}g\left(y\right){\left({\int }_S{\left(\frac{F(x,y)}{g\left(y\right)}\right)}^{\alpha }\mathrm{d}\gamma \left(x\right)\right)}^{\frac{1}{\alpha }}\mathrm{d}\eta \left(y\right)\right)}^{\frac{\alpha }{\alpha -1}},$$

which is the expected $L^{\alpha }(S,\gamma )$ norm of the conditional density under the measure ${\mathbb{P}}_Y$ raised to a power which is the Hölder conjugate of $\alpha$. Using Fubini’s theorem the formula for ${\mathrm{Ren}}_{\alpha }^{\gamma }\left(X\right|Y)$ can be further written down only in terms of the joint density,

$${\mathrm{Ren}}_{\alpha }^{\gamma }\left(X\right|Y)={\left({\int }_T{\left({\int }_{S}F{(x,y)}^{\alpha }\mathrm{d}\gamma \left(x\right)\right)}^{\frac{1}{\alpha }}\mathrm{d}\eta \left(y\right)\right)}^{\frac{\alpha }{\alpha -1}}.$$

Remark 1.

Suppose${\mathbb{P}}_{X|Y=y}$, for each $y\in \mathrm{supp}\left(g\right)$, denotes the conditional distribution of X given$Y=y$, i.e., the probability measure on S with density $F(x,y)/g\left(y\right)$with respect to γ, then the conditional Rényi entropy can be written as

$$h_{\alpha }^{\gamma }\left(X\right|Y)=\frac{\alpha }{1-\alpha }\log {\int }_T{e}^{-\frac{1-\alpha }{\alpha }{D}_{\alpha }({\mathbb{P}}_{X|Y=y}\parallel \gamma )}\mathrm{d}{\mathbb{P}}_Y\left(y\right),$$

where $D_{\alpha }(·\parallel ·)$ denotes Rényi divergence (see Definition 3).

When X and Y are independent random variables one can easily check that ${\mathrm{Ren}}_{\alpha }^{\gamma }\left(X\right|Y)={\mathrm{Ren}}_{\alpha }^{\gamma }\left(X\right)$, therefore $h_{\alpha }^{\gamma }\left(X\right|Y)=h_{\alpha }^{\gamma }\left(X\right)$ as expected. Since the independence of X and Y means that all the conditionals ${\mathbb{P}}_{X|Y=y}$ are equal to ${\mathbb{P}}_X$, the fact that $h_{\alpha }^{\gamma }\left(X\right|Y)=h_{\alpha }^{\gamma }\left(X\right)$ in this case can also be verified from the expression in Remark 1. The converse is also true, i.e., $h_{\alpha }^{\gamma }\left(X\right|Y)=h_{\alpha }^{\gamma }\left(X\right)$ implies the independence of X and Y, if $\alpha \ne 0,\infty$. This is noted later in Corollary 2.

Clearly, unlike conditional Shannon entropy, the conditional Rényi entropy is not the average Rényi entropy of the conditional distribution. The average Rényi entropy of the conditional distribution,

$${\tilde{h}}_{\alpha }^{\gamma }\left(X\right|Y):={\mathbb{E}}_Y{h}_{\alpha }^{\gamma }\left(X\right|Y=y),$$

has been proposed as a candidate for conditional Rényi entropy, however it does not satisfy some properties one would expect such a notion to satisfy, like monotonicity (see [30]). When $\alpha \ge 1$ it follows from Jensen’s inequality that $h_{\alpha }^{\gamma }\left(X\right|Y)\le {\tilde{h}}_{\alpha }^{\gamma }\left(X\right|Y)$, while the inequality is reversed when $0<\alpha <1$.

3. Relation to Rényi Divergence

We continue to consider an S-valued random variable X and a T-valued random variable Y with a given joint distribution ${\mathbb{P}}_{X,Y}$ with density F with respect to $\gamma \otimes \eta$. Densities, etc are with respect to the fixed reference measures on the state spaces, unless mentioned otherwise.

Let $\mu$ be a Borel probability measure with density p on a Polish space $\mathsf{\Omega }$ and let $\nu$ be a Borel measure with density q on the same space with respect to a common measure $\gamma$.

Definition 3

(Rényi divergence). Suppose $\alpha \in (0,1)\cup (1,\infty )$. Then, the Rényi divergence of order α between measures μ and ν is defined as

$$D_{\alpha }(\mu \parallel \nu )=\log {\left({\int }_{\mathsf{\Omega }}p{\left(x\right)}^{\alpha }q{\left(x\right)}^{1-\alpha }\mathrm{d}\gamma \left(x\right)\right)}^{\frac{1}{\alpha -1}}.$$

For order $0,1,\infty ,$ the Rényi divergence is defined by the respective limits.

Definition 4.

1.

$D_0(\mu \parallel \nu ):={lim}_{\alpha \to 0}{D}_{\alpha }(\mu \parallel \nu )=-\log \nu \left(\mathrm{supp}\left(p\right)\right)$;

2.

$D_1(\mu \parallel \nu ):=D(\mu \parallel \nu )=\int p\left(x\right)\log \frac{p\left(x\right)}{q\left(x\right)}\mathrm{d}\gamma \left(x\right)$; and

3.

$D_{\infty }(\mu \parallel \nu ):={lim}_{\alpha \to \infty }{D}_{\alpha }(\mu \parallel \nu )=\log \left(\mathrm{ess}{\sup }_{\mu }\frac{p\left(x\right)}{q\left(x\right)}\right).$

Remark 2.

These definitions are independent of the reference measure γ.

Remark 3.

${\lim }_{\alpha \to 1}{D}_{\alpha }(\mu \parallel \nu )=1$if some $D_{\alpha }(\mu \parallel \nu )<\infty$. See [29].

The conditional Rényi entropy can be written in terms of Rényi divergence from the joint distribution using a generalized Sibson’s identity we learnt from B. Nakiboğlu [43] (also see [36], and [38] where this identity for $\alpha \ne 1$ appears to originate from). The proof for abstract alphabets presented here is also due to B. Nakiboğlu [43], which simplifies our original proof [41] of the second formula below.

Theorem 1.

Let$X,Y$be random variables taking vaules in spaces$S,T$respectively. We assume they are jointly distributed with density F with respect to the product reference measure$\gamma \otimes \eta$. For $\alpha \in (0,\infty )$, and any probability measure λ absolutely continuous with respect to η, we have

$$D_{\alpha }({\mathbb{P}}_{X,Y}\parallel \gamma \otimes \lambda )=D_{\alpha }({\mathbb{P}}_{X,Y}\parallel \gamma \otimes q_{\star })+D_{\alpha }(q_{\star }\parallel \lambda )=-h_{\alpha }^{\gamma }\left(X\right|Y)+D_{\alpha }(q_{\star }\parallel \lambda ),$$

where$q_{\star }=\frac{{\mu }_{\star }}{\parallel {\mu }_{\star }\parallel }$, ${\mu }_{\star }$is the measure having density$\varphi \left(y\right)={\left({\int }_{S}F{(x,y)}^{\alpha }\mathrm{d}\gamma \left(x\right)\right)}^{\frac{1}{\alpha }}$with respect to η, and$\parallel {\mu }_{\star }\parallel ={\int }_T{\left({\int }_{S}F{(x,y)}^{\alpha }\mathrm{d}\gamma \left(x\right)\right)}^{\frac{1}{\alpha }}\mathrm{d}\eta \left(y\right)$is the normalization factor.

As a consequence, we have

$$h_{\alpha }^{\gamma }\left(X\right|Y)=-\underset{\lambda \in \mathcal{P}\left(T\right)}{min}{D}_{\alpha }({\mathbb{P}}_{X,Y}\parallel \gamma \otimes \lambda ).$$

Proof. 

Suppose $\lambda$ has density h with respect to $\eta$. Then $\gamma \otimes \lambda$ has density $h\left(y\right)$ with respect to $\mathrm{d}\gamma \left(x\right)\mathrm{d}\eta \left(y\right)$. Now, for $\alpha \ne 1$,

$$\begin{array}{ll}{D}_{\alpha }({\mathbb{P}}_{X,Y}\parallel \gamma \otimes \lambda )& =\frac{1}{\alpha -1}\log {\int }_T{\int }_{S}F{(x,y)}^{\alpha }h{\left(y\right)}^{1-\alpha }\mathrm{d}\gamma \left(x\right)\mathrm{d}\eta \left(y\right)\hfill \\ & =\frac{1}{\alpha -1}\log {\int }_{T}h{\left(y\right)}^{1-\alpha }{\int }_{S}F{(x,y)}^{\alpha }\mathrm{d}\gamma \left(x\right)\mathrm{d}\eta \left(y\right)\hfill \\ & =\frac{1}{\alpha -1}\log {\int }_{T}h{\left(y\right)}^{1-\alpha }\varphi {\left(y\right)}^{\alpha }\mathrm{d}\eta \left(y\right)\hfill \\ & =\frac{1}{\alpha -1}\log {\int }_T{\left(\frac{\mathrm{d}\lambda }{\mathrm{d}\eta }\right)}^{1-\alpha }{\left(\frac{\mathrm{d}{\mu }_{\star }}{\mathrm{d}\eta }\right)}^{\alpha }\mathrm{d}\eta \left(y\right)\hfill \\ & =\frac{1}{\alpha -1}\log {\int }_T{\left(\frac{\mathrm{d}\lambda }{\mathrm{d}\eta }\right)}^{1-\alpha }{\left(\frac{\mathrm{d}{q}_{\star }}{\mathrm{d}\eta }\right)}^{\alpha }{\parallel {\mu }_{\star }\parallel }^{\alpha }\mathrm{d}\eta \left(y\right)\hfill \\ & =\frac{\alpha }{\alpha -1}\log \parallel {\mu }_{\star }\parallel +\frac{1}{\alpha -1}\log {\int }_T{\left(\frac{\mathrm{d}\lambda }{\mathrm{d}\eta }\right)}^{1-\alpha }{\left(\frac{\mathrm{d}{q}_{\star }}{\mathrm{d}\eta }\right)}^{\alpha }\mathrm{d}\eta \left(y\right)\hfill \\ & =\frac{\alpha }{\alpha -1}\log \parallel {\mu }_{\star }\parallel +D_{\alpha }(q_{\star }\parallel \lambda )\hfill \\ & =-h_{\alpha }^{\gamma }\left(X\right|Y)+D_{\alpha }(q_{\star }\parallel \lambda ).\hfill \end{array}$$

The case $\alpha =1$ is straightforward and well-known, and the optimal $q_{\star }$ in this case is the distribution of Y.□

Remark 4.

The identities above and the measure$q_{\star }$are independent of the reference measure η. η is only used to write out the Rényi divergence concretely in terms of densities.

4. Basic Properties

4.1. Special Orders

We will now look at some basic properties of the conditional Rényi entropy we have defined above. First we see that the conditional Rényi entropy is consistent with the notion of conditional Shannon entropy of X given Y defined by

$$h^{\gamma }\left(X\right|Y)=-{\int }_T{\int }_{S}F(x,y)\log \frac{F(x,y)}{g\left(y\right)}\mathrm{d}\gamma \left(x\right)\mathrm{d}\eta \left(y\right).$$

Proposition 1.

$$\underset{\alpha \to 1^{+}}{\lim }{h}_{\alpha }^{\gamma }\left(X\right|Y)=h_1^{\gamma }\left(X\right|Y)=h^{\gamma }\left(X\right|Y),$$

if$h_{\alpha }^{\gamma }\left(X\right|Y)<\infty$for some$\alpha >1$.

Proof. 

We will use the formula in Theorem 1. By the monotonicity in the order $\alpha$ of $h_{\alpha }\left(X\right|Y)$, all limits ${lim}_{\alpha \to 1^{+}},{lim}_{\alpha \to 1^{-}},{lim}_{\alpha \to 0}={lim}_{\alpha \to 0^{+}}$ exist. Furthermore, for every $\eta$,

$$\underset{\lambda }{min}{D}_{\alpha }(P_{X,Y}\parallel \gamma \otimes \lambda )\le D_{\alpha }(P_{X,Y}\parallel \gamma \otimes \eta ),$$

so

$$\underset{\alpha \to 1^{+}}{lim}\underset{\lambda }{min}{D}_{\alpha }(P_{X,Y}\parallel \gamma \otimes \lambda )\le \underset{\alpha \to 1^{+}}{lim}{D}_{\alpha }(P_{X,Y}\parallel \gamma \otimes \eta ),$$

that is,

$$-\underset{\alpha \to 1^{+}}{lim}{h}_{\alpha }^{\gamma }\left(X\right|Y)\le D(P_{X,Y}\parallel \gamma \otimes \eta ).$$

Now by minimizing over $\eta$ and hitting both sides with a minus sign yields,

$$\underset{\alpha \to 1^{+}}{lim}{h}_{\alpha }^{\gamma }\left(X\right|Y)\ge h^{\gamma }\left(X\right|Y).$$

Suppose $\alpha \ge 1$, then by nondecreasing-ness of the Rényi divergence in order, for every $\lambda$ we have

$$D_{\alpha }(P_{X,Y}\parallel \gamma \otimes \lambda )\ge D(P_{X,Y}\parallel \gamma \otimes \lambda ),$$

and so by minimization over $\lambda$ and hitting with minus signs we obtain $h_{\alpha }^{\gamma }\left(X\right|Y)\le h^{\gamma }\left(X\right|Y)$. This shows that

$$\underset{\alpha \to 1^{+}}{lim}{h}_{\alpha }^{\gamma }\left(X\right|Y)=h^{\gamma }\left(X\right|Y).$$

□

We can extend our definition of Rényi probability of order $\alpha$ to $\alpha =0$ by taking limits, thereby obtaining

$${\mathrm{Ren}}_0^{\gamma }\left(X\right)=\frac{1}{\gamma \left(\mathrm{supp}\right(f\left)\right)}.$$

In the next proposition we define the conditional Rényi entropy of order 0 and record a consequence.

Proposition 2.

$$h_0^{\gamma }\left(X\right|Y):={\parallel h_0^{\gamma }\left(X\right|Y=y)\parallel }_{L^{\infty }\left({\mathbb{P}}_Y\right)}=\underset{\alpha \to 0}{\lim }{h}_{\alpha }^{\gamma }\left(X\right|Y)$$

Proof. 

We will again use the formula from 1 in this proof. Just as in the last proof, for every probability measure $\eta$,

$$\underset{\lambda }{\min }{D}_{\alpha }(P_{X,Y}\parallel \gamma \otimes \lambda )\le D_{\alpha }(P_{X,Y}\parallel \gamma \otimes \eta ),$$

so

$$\underset{\alpha \to 0}{lim}\underset{\lambda }{min}{D}_{\alpha }(P_{X,Y}\parallel \gamma \otimes \lambda )\le \underset{\alpha \to 0}{\lim }{D}_{\alpha }(P_{X,Y}\parallel \gamma \otimes \eta ),$$

that is,

$$-\underset{\alpha \to 0}{\lim }{h}_{\alpha }^{\gamma }\left(X\right|Y)\le D_0(P_{X,Y}\parallel \gamma \otimes \eta ).$$

Now by minimizing over $\eta$ and hitting both sides with a minus sign yields,

$$\underset{\alpha \to 0}{\lim }{h}_{\alpha }^{\gamma }\left(X\right|Y)\ge h_0^{\gamma }\left(X\right|Y).$$

Suppose $\alpha \ge 0$, then by nondecreasing-ness of the Rényi divergence in order, for every $\lambda$ we have

$$D_{\alpha }(P_{X,Y}\parallel \gamma \otimes \lambda )\ge D_0(P_{X,Y}\parallel \gamma \otimes \lambda ),$$

and so by minimization over $\lambda$ and hitting with minus signs we obtain $h_{\alpha }^{\gamma }\left(X\right|Y)\le h_0^{\gamma }\left(X\right|Y)$. This shows that

$$\underset{\alpha \to 0}{\lim }{h}_{\alpha }^{\gamma }\left(X\right|Y)=h_0^{\gamma }\left(X\right|Y).$$

□

4.2. Monotonicity in Order

The (unconditional) Rényi entropy decreases with order, and the same is true of the conditional version.

Proposition 3.

For random variables X and Y,

$$h_{\beta }^{\gamma }\left(X\right|Y)\le h_{\alpha }^{\gamma }\left(X\right|Y),$$

for all$0<\alpha \le \beta \le \infty$.

The proof is essentially the same as in the discrete setting, and follows from Jensen’s inequality.

Proof. 

We obtain separately the cases $1<\alpha \le \beta <\infty$, $0<\alpha \le \beta <1$, so that the entire Proposition follows from taking limits and the transitivity of inequality. Let $\alpha \le \beta$ be positive numbers, set $e=\frac{\beta -1}{\beta }\frac{\alpha }{\alpha -1}$. Consider the following argument.

$$\begin{array}{cc}\hfill {\mathrm{Ren}}_{\beta }\left(X\right|Y)& ={\left({\int }_{\mathrm{supp}\left({\mathbb{P}}_Y\right)}{\mathrm{Ren}}_{\beta }^{\gamma }{\left(X\right|Y=y)}^{\frac{\beta -1}{\beta }}\mathrm{d}{\mathbb{P}}_Y\right)}^{\frac{\beta }{\beta -1}}\hfill \\ & \ge {\left({\int }_{\mathrm{supp}\left({\mathbb{P}}_Y\right)}{\mathrm{Ren}}_{\alpha }^{\gamma }{\left(X\right|Y=y)}^{\frac{\beta -1}{\beta }}\mathrm{d}{\mathbb{P}}_Y\right)}^{\frac{\beta }{\beta -1}}\hfill \\ & ={\left({\int }_{\mathrm{supp}\left({\mathbb{P}}_Y\right)}{\mathrm{Ren}}_{\alpha }^{\gamma }{\left(X\right|Y=y)}^{\frac{\alpha -1}{\alpha }e}\mathrm{d}{\mathbb{P}}_Y\right)}^{\frac{\beta }{\beta -1}}\hfill \\ & \ge {\left({\int }_{\mathrm{supp}\left({\mathbb{P}}_Y\right)}{\mathrm{Ren}}_{\alpha }^{\gamma }{\left(X\right|Y=y)}^{\frac{\alpha -1}{\alpha }}\mathrm{d}{\mathbb{P}}_Y\right)}^{e\frac{\beta }{\beta -1}}\hfill \\ & ={\mathrm{Ren}}_{\alpha }^{\gamma }\left(X\right|Y).\hfill \end{array}$$

The first inequality above follows from the fact that the unconditional Rényi entropy (probability) decreases (increases) with order. Note that $e\ge 1$ when $1<\alpha \le \beta <\infty$ and hence the function $r\mapsto r^e$ is convex making the second inequality an application of Jensen’s inequality in this case. When $0<\alpha \le \beta <1$, the exponent satisfies $0<e\le 1$ so the function $r\mapsto r^e$ is concave but the outer exponent $\frac{\beta }{\beta -1}$ is negative which turns the (second) inequality in the desired direction. □

4.3. Conditioning Reduces Rényi Entropy

As is the case for Shannon entropy, we find that the conditional Rényi entropy obeys monotonicity too; the proof of the theorem below adapts the approach for the discrete case taken in [30] by using Minkowski’s integral inequality.

Theorem 2.

[Monotonicity] Let$\alpha \in [0,\infty ]$, and X be S-valued and$Y,Z$be T-valued random variables. Then,

$$h_{\alpha }^{\gamma }\left(X\right|YZ)\le h_{\alpha }^{\gamma }\left(X\right|Z).$$

Proof. 

We begin by proving the result for an empty Z.

First, we deal with the case $1<\alpha <\infty$. In terms of Rényi probabilities we must show that conditioning increases Rényi probability. Indeed,

$$\begin{array}{cc}\hfill {\mathrm{Ren}}_{\alpha }^{\gamma }\left(X\right|Y)& ={\left({\int }_T{\left[{\int }_{S}F{(x,y)}^{\alpha }\mathrm{d}\gamma \left(x\right)\right]}^{\frac{1}{\alpha }}\mathrm{d}\eta \left(y\right)\right)}^{\frac{\alpha }{\alpha -1}}\hfill \\ & \ge {\left[{\int }_S{\left({\int }_{T}F(x,y)\mathrm{d}\eta \left(y\right)\right)}^{\alpha }\mathrm{d}\gamma \left(x\right)\right]}^{\frac{1}{\alpha }\frac{\alpha }{\alpha -1}}\hfill \\ & ={\left({\int }_{S}f{\left(x\right)}^{\alpha }\mathrm{d}\gamma \left(x\right)\right)}^{\frac{1}{\alpha -1}}={\mathrm{Ren}}_{\alpha }^{\gamma }\left(X\right).\hfill \end{array}$$

(1)

The inequality above is a direct application of Minkowski’s integral inequality ([44], Theorem 2.4), which generalizes the summation in the standard triangle inequality to integrals against a measure.

For the case $0<\alpha <1$, we apply the triangle inequality for $1/\alpha$ then the fact that now $\frac{1}{\alpha -1}$ is negative flips the inequality in the desired direction:

$$\begin{array}{cc}\hfill {\mathrm{Ren}}_{\alpha }^{\gamma }\left(X\right|Y)& ={\left({\int }_T{\left[{\int }_{S}F{(x,y)}^{\alpha }\mathrm{d}\gamma \left(x\right)\right]}^{\frac{1}{\alpha }}\mathrm{d}\eta \left(y\right)\right)}^{\frac{\alpha }{\alpha -1}}\hfill \\ & \ge {\left[{\int }_S{\left({\int }_{T}F{(x,y)}^{\frac{\alpha }{\alpha }}\mathrm{d}\eta \left(y\right)\right)}^{\alpha }\mathrm{d}\gamma \left(x\right)\right]}^{\frac{1}{\alpha -1}}\hfill \\ & ={\left({\int }_{S}f{\left(x\right)}^{\alpha }\mathrm{d}\gamma \left(x\right)\right)}^{\frac{1}{\alpha -1}}={\mathrm{Ren}}_{\alpha }^{\gamma }\left(X\right).\hfill \end{array}$$

Now to extend this to non-empty Z we observe the following. Suppose V is an S-valued random variable, W is a T-valued random variable and $h\in \mathbb{R}$ is such that

$$h_{\alpha }^{\gamma }\left(V\right|W,Z=z)\ge h_{\alpha }^{\gamma }\left(X\right|Y,Z=z)+h,$$

for every z in the support of ${\mathbb{P}}_Z$. Then

$$h_{\alpha }^{\gamma }\left(V\right|WZ)\ge h_{\alpha }^{\gamma }\left(X\right|YZ)+h.$$

In terms of Renyi probabilities this means that if for every $z\in \mathrm{supp}\left({\mathbb{P}}_Z\right)$,

$${\mathrm{Ren}}_{\alpha }^{\gamma }\left(V\right|W,Z=z)\le \frac{{\mathrm{Ren}}_{\alpha }^{\gamma }\left(X\right|Y,Z=z)}{\log h},$$

then

$${\mathrm{Ren}}_{\alpha }^{\gamma }\left(V\right|WZ)\le \frac{{\mathrm{Ren}}_{\alpha }^{\gamma }\left(X\right|YZ)}{\log h}.$$

Indeed,

$$\begin{array}{cc}\hfill {\mathrm{Ren}}_{\alpha }^{\gamma }\left(V\right|WZ)& ={\left({\int }_{\mathrm{supp}\left(Z\right)}{\mathrm{Ren}}_{\alpha }^{\gamma }{\left(V\right|W,Z=z)}^{\frac{\alpha -1}{\alpha }}\mathrm{d}{\mathbb{P}}_Z\left(z\right)\right)}^{\frac{\alpha }{\alpha -1}}\hfill \\ & \le {\left({\int }_{\mathrm{supp}\left(Z\right)}{\left(\frac{{\mathrm{Ren}}_{\alpha }^{\gamma }\left(X\right|Y,Z=z)}{\log h}\right)}^{\frac{\alpha -1}{\alpha }}\mathrm{d}{\mathbb{P}}_Z\left(z\right)\right)}^{\frac{\alpha }{\alpha -1}}\hfill \\ & =\frac{{\mathrm{Ren}}_{\alpha }^{\gamma }\left(X\right|YZ)}{\log h},\hfill \end{array}$$

since the functions $r\mapsto r^{\frac{\alpha -1}{\alpha }}$ and $r\mapsto r^{\frac{\alpha }{\alpha -1}}$ are both strictly increasing or strictly decreasing, based on the value of $\alpha$. Finally, the claim for non-empty Z follows from this observation given we have already demonstrated $h_{\alpha }^{\gamma }\left(X\right|Y,Z=z)\le h_{\alpha }^{\gamma }\left(X\right|Z=z)$ throughout $\mathrm{supp}\left(\mathrm{Z}\right)$. The cases $\alpha =0,\infty$ are obtained by taking limits. For $\alpha =1$ this is well-known.□

Corollary 1.

If $X\to Y\to Z$ forms a Markov chain, then $h_{\alpha }^{\gamma }\left(X\right|Y)\le h_{\alpha }^{\gamma }\left(X\right|Z)$.

When we specialize to “empty Z” (i.e., the $\sigma$-field generated by Z being the Borel $\sigma$-field on T, or not conditioning on anything), we find that “conditioning reduces Rényi entropy”.

Corollary 2.

Let $\alpha \in (0,\infty )$. Then $h_{\alpha }^{\gamma }\left(X\right|Y)\le h_{\alpha }^{\gamma }\left(X\right)$, with equality iff X and Y are independent.

While the inequality in Corollary 2 follows immediately from Theorem 2, the conditions for equality follow from those for Minkowski’s inequality (which is the key inequality used in the proof of Theorem 2, see, e.g., ([44], Theorem 2.4)): given the finiteness of both sides in the display (1), equality holds if and only if $F(x,y)=\varphi \left(x\right)\psi \left(y\right)$$\gamma \otimes \eta$-a.e. for some functions $\varphi$ and $\psi$. In our case, this means that equality holds in $h_{\alpha }^{\gamma }\left(X\right|Y)\le h_{\alpha }^{\gamma }\left(X\right)$ if and only if X and Y are independent ($\alpha \in (0,1)\cup (1,\infty )$). The corresponding statement for $\alpha =1$ is well-known.

Since ${\tilde{h}}_{\alpha }^{\gamma }\left(X\right|Y)\le h_{\alpha }^{\gamma }\left(X\right|Y)$ when $0<\alpha <1$, as noted in Section 2, we have “conditioning reduces Rényi entropy” in this case as well.

Corollary 3.

${\tilde{h}}_{\alpha }^{\gamma }\left(X\right|Y)\le h_{\alpha }^{\gamma }\left(X\right)$, when$0<\alpha <1$.

Remark 5.

The above corollary is not true for large values of α. For a counter-example, see ([28], Theorem 7).

From the special case when Y is discrete random variable taking finitely many values $y_i$ with probability $p_i$, $1\le i\le n$, and the conditional density of X given $Y=y_i$ is $f_i\left(x\right)$, we obtain the concavity of Rényi entropy (for orders below 1) which is already known in the literature.

Corollary 4.

Let$0\le \alpha \le 1$. Suppose$f_i$are densities on S and$p_i$non-negative numbers,$1\le i\le n$, such that ${\sum }_i{p}_i=1$. Then,$h_{\alpha }^{\gamma }\left({\sum }_{i=1}^n{p}_i{f}_i\right)\ge {\sum }_{i=1}^n{p}_i{h}_{\alpha }^{\gamma }\left(f_i\right)$.

4.4. A chain Rule

In this subsection we deduce a version of the chain rule from Theorem 1. For the discrete case, this has been done by Fehr and Berens in ([30], Theorem 3). If $\eta$ is a probability measure, then we have $h_{\alpha }^{\gamma }\left(X\right|Y)\ge -D_{\alpha }({\mathbb{P}}_{X,Y}\parallel \gamma \otimes \eta )=h_{\alpha }^{\gamma \otimes \eta }(X,Y)$. If we relax the condition on $\eta$ to be a measure under which ${\mathbb{P}}_Y$ is absolutely continuous and supported on a set of finite measure, we obtain $h_{\alpha }^{\gamma }\left(X\right|Y)\ge h_{\alpha }^{\gamma \otimes \eta }(X,Y)-h_0^{\eta }\left(Y\right)$. Since this inequality trivially holds true when Y is supported on a set of infinite $\eta$-measure, we have proved the following inequality that (although weaker being an inequality rather an identity) is reminiscent of the chain rule for Shannon entropy.

Proposition 4.

For any$\alpha >0$,

$$h_{\alpha }^{\gamma \otimes \eta }(X,Y)\le h_{\alpha }^{\gamma }\left(X\right|Y)+h_0^{\eta }\left(Y\right).$$

Proof. 

Recall that

$${\mathrm{Ren}}_{\alpha }^{\gamma }\left(X\right|Y)={\left[{\int }_T{\left({\int }_{S}F{(x,y)}^{\alpha }\mathrm{d}\gamma \left(x\right)\right)}^{\frac{1}{\alpha }}\mathrm{d}\eta \left(y\right)\right]}^{\frac{\alpha }{\alpha -1}},$$

where the outer integral can be restricted to the support of ${\mathbb{P}}_Y$, which we will keep emphasizing in the first few steps.

$$\begin{array}{cc}\hfill {\mathrm{Ren}}_{\alpha }^{\gamma }\left(X\right|Y)& ={\left[{\int }_{\mathrm{supp}\left({\mathbb{P}}_Y\right)}{\left({\int }_{S}F{(x,y)}^{\alpha }\mathrm{d}\gamma \left(x\right)\right)}^{\frac{1}{\alpha }}\mathrm{d}\eta \left(y\right)\right]}^{\frac{\alpha }{\alpha -1}}\hfill \\ & ={\left[{\int }_{\mathrm{supp}\left({\mathbb{P}}_Y\right)}{\left(\eta \left(\mathrm{supp}\left({\mathbb{P}}_Y\right)\right){\int }_{S}F{(x,y)}^{\alpha }\mathrm{d}\gamma \left(x\right)\right)}^{\frac{1}{\alpha }}\mathrm{d}\frac{\eta \left(y\right)}{\eta \left(\mathrm{supp}\left({\mathbb{P}}_Y\right)\right)}\right]}^{\frac{\alpha }{\alpha -1}}\hfill \\ & ={\left[{\int }_{\mathrm{supp}\left({\mathbb{P}}_Y\right)}{\left({\int }_S{\left(\eta \left(\mathrm{supp}\left({\mathbb{P}}_Y\right)\right)F(x,y)\right)}^{\alpha }\mathrm{d}\gamma \left(x\right)\right)}^{\frac{1}{\alpha }}\mathrm{d}\frac{\eta \left(y\right)}{\eta \left(\mathrm{supp}\left({\mathbb{P}}_Y\right)\right)}\right]}^{\frac{\alpha }{\alpha -1}}.\hfill \end{array}$$

By Jensen’s inequality, when $\alpha >1$,

$$\begin{array}{cc}\hfill {\mathrm{Ren}}_{\alpha }^{\gamma }\left(X\right|Y)& \le {\left[{\int }_{\mathrm{supp}\left({\mathbb{P}}_Y\right)}\left({\int }_S{\left(\eta \left(\mathrm{supp}\left({\mathbb{P}}_Y\right)\right)F(x,y)\right)}^{\alpha }\mathrm{d}\gamma \left(x\right)\right)\mathrm{d}\frac{\eta \left(y\right)}{\eta \left(\mathrm{supp}\left({\mathbb{P}}_Y\right)\right)}\right]}^{\frac{1}{\alpha -1}}\hfill \\ & =\eta \left(\mathrm{supp}\left({\mathbb{P}}_Y\right)\right){\left[{\int }_{\mathrm{supp}\left({\mathbb{P}}_Y\right)}{\int }_{S}F{(x,y)}^{\alpha }\mathrm{d}\gamma \left(x\right)\mathrm{d}\eta \left(y\right)\right]}^{\frac{1}{\alpha -1}}\hfill \\ & =\eta \left(\mathrm{supp}\left({\mathbb{P}}_Y\right)\right){\mathrm{Ren}}_{\alpha }^{\gamma \otimes \eta }(X,Y).\hfill \end{array}$$

Note that the above calculation also holds when $\alpha \in (0,1)$ because even though Jensen’s inequality is flipped because $\frac{1}{\alpha }$ is now convex, the inequality flips again, now to the desired side, since the exponent $\frac{\alpha }{\alpha -1}$ is negative. Taking logarithms and hitting both sides with a minus sign concludes the proof.□

Remark 6.

These inequalities are tight. Equality is attained when X, Y are independent and Y is uniformly distributed on finite support.

4.5. Sensitivity to Reference Measure

Proposition 5.

Suppose$\frac{\mathrm{d}\gamma }{\mathrm{d}\mu }=\psi$which is bounded by a number M. Then$h_{\alpha }^{\mu }\left(X\right|Y)\ge h_{\alpha }^{\gamma }\left(X\right|Y)-\log M.$

Proof. 

Then the joint density of $(X,Y)$ under the measure $\mu \otimes \mu$ becomes $F(x,y)\psi \left(x\right)\psi \left(y\right)$ if the joint density was $F(x,y)$ under $\gamma$. Now suppose $\alpha \ge 1$ then,

$$\begin{array}{c}{h}_{\alpha }^{\mu }\left(X\right|Y)\hfill \\ =\frac{-\alpha }{\alpha -1}\log \left[{\int }_T{\left({\int }_{S}F{(x,y)}^{\alpha }\psi {\left(x\right)}^{\alpha }\psi {\left(y\right)}^{\alpha }\mathrm{d}\mu \left(x\right)\right)}^{\frac{1}{\alpha }}\mathrm{d}\mu \left(y\right)\right]\hfill \\ =\frac{-\alpha }{\alpha -1}\log \left[{\int }_T{\left({\int }_{S}F{(x,y)}^{\alpha }\psi {\left(x\right)}^{\alpha }\mathrm{d}\mu \left(x\right)\right)}^{\frac{1}{\alpha }}\psi \left(y\right)\mathrm{d}\mu \left(y\right)\right]\hfill \\ =\frac{-\alpha }{\alpha -1}\log \left[{\int }_T{\left({\int }_{S}F{(x,y)}^{\alpha }\psi {\left(x\right)}^{\alpha -1}\mathrm{d}\gamma \left(x\right)\right)}^{\frac{1}{\alpha }}\mathrm{d}\gamma \left(y\right)\right]\hfill \\ \ge \frac{-\alpha }{\alpha -1}\left(\log \left[{\int }_T{\left({\int }_{S}F{(x,y)}^{\alpha }\mathrm{d}\gamma \left(x\right)\right)}^{\frac{1}{\alpha }}\mathrm{d}\gamma \left(y\right)\right]+\log M^{\frac{\alpha -1}{\alpha }}\right)\hfill \\ =h_{\alpha }^{\gamma }\left(X\right|Y)-\log M.\hfill \end{array}$$

If $\alpha \in (0,1)$, the same inequality holds if $\mu$ is also absolutely continuous w.r.t. $\gamma$.□

5. Notions of $\alpha$-Mutual Information

Arimoto [39] used his conditional Rényi entropy to define a mutual information that we extend to the general setting as follows.

Definition 5.

Let X be an S-valued random variable and let Y be a T-valued random variable, with a given joint distribution. Then, we define

$$I_{\alpha }^{\left(\gamma \right)}(X⇝Y)=h_{\alpha }^{\gamma }\left(X\right)-h_{\alpha }^{\gamma }\left(X\right|Y).$$

We use the squiggly arrow to emphasize the lack of symmetry in X and Y, but nonetheless to distinguish from the notation for directed mutual information, which is usually written with a straight arrow. By Corollary 2, for $\alpha \in (0,\infty )$, $I_{\alpha }^{\left(\gamma \right)}(X⇝Y)=0$ if and only if X and Y are independent. Therefore $I_{\alpha }^{\left(\gamma \right)}(X⇝Y)$, for any choice of reference measure $\gamma$, can be seen as a measure of dependence between X and Y.

Let us discuss a little further the validity of $I_{\alpha }^{\left(\gamma \right)}(X⇝Y)$ as a dependence measure. If the conditional distributions are denoted by ${\mathbb{P}}_{X|Y=y}$ as in Remark 1, using the fact that $h_{\alpha }^{\nu }\left(Z\right)=-D_{\alpha }(Z\parallel \nu )$ for any random variable Z, we have for any $\alpha \in (0,1)\cup (1,\infty )$ that

$$\begin{array}{cc}\hfill I_{\alpha }^{\left(\gamma \right)}(X⇝Y)& =h_{\alpha }^{\gamma }\left(X\right)-h_{\alpha }^{\gamma }\left(X\right|Y)\hfill \\ & =\frac{\alpha }{1-\alpha }\log e^{-\frac{1-\alpha }{\alpha }{D}_{\alpha }({\mathbb{P}}_X\parallel \gamma )}-\frac{\alpha }{1-\alpha }\log {\int }_T{e}^{-\frac{1-\alpha }{\alpha }{D}_{\alpha }({\mathbb{P}}_{X|Y=y}\parallel \gamma )}\mathrm{d}{\mathbb{P}}_Y\left(y\right)\hfill \\ & =-\frac{\alpha }{1-\alpha }\log {\int }_T{e}^{-\frac{1-\alpha }{\alpha }[D_{\alpha }({\mathbb{P}}_{X|Y=y}\parallel \gamma )-D_{\alpha }({\mathbb{P}}_X\parallel \gamma )]}\mathrm{d}{\mathbb{P}}_Y\left(y\right).\hfill \end{array}$$

Furthermore, when $\alpha \in (0,1)$, by ([29], Proposition 2), we may also write

$$\begin{array}{cc}\hfill I_{\alpha }^{\left(\gamma \right)}(X⇝Y)& =-\frac{\alpha }{1-\alpha }\log {\int }_T{e}^{-[D_{1-\alpha }(\gamma \parallel {\mathbb{P}}_{X|Y=y})-D_{1-\alpha }(\gamma \parallel {\mathbb{P}}_X)]}\mathrm{d}{\mathbb{P}}_Y\left(y\right).\hfill \end{array}$$

Note that Rényi divergence is convex in the second argument (see [29], Theorem 12) when $\alpha \in (0,1)$, and the last equation suggests that Arimoto’s mutual information can be seen as a quantification of this convexity gap.

One can also see clearly from the above expressions why this quantity controls, at least for $\alpha \in (0,1)$, the dependence between X and Y: indeed, one has for any $\alpha \in (0,1)$ and any $t>0$ that,

$$\begin{array}{cc}\hfill {\mathbb{P}}_Y\{D_{\alpha }({\mathbb{P}}_{X|Y}\parallel \gamma )-D_{\alpha }({\mathbb{P}}_X\parallel \gamma )<t\}& ={\mathbb{P}}_Y\{e^{-\beta [D_{\alpha }({\mathbb{P}}_{X|Y}\parallel \gamma )-D_{\alpha }({\mathbb{P}}_X\parallel \gamma )]}>e^{-\beta t}\}\le e^{\beta t}{e}^{-\beta I_{\alpha }^{\left(\gamma \right)}(X⇝Y)},\hfill \end{array}$$

where the inequality comes from Markov’s inequality, and we use $\beta =\frac{1-\alpha }{\alpha }$. Thus, when $I_{\alpha }^{\left(\gamma \right)}(X⇝Y)$ is large, the probability that the conditional distributions of X given Y cluster at around the same “Rényi divergence” distance from the reference measure $\gamma$ as the unconditional distribution of X (which is of course a mixture of the conditional distributions) is small, suggesting a significant “spread” of the conditional distributions and therefore strong dependence. This is illustrated in Figure 1. Thus, despite the dependence of $I_{\alpha }^{\left(\gamma \right)}(X⇝Y)$ on the reference measure $\gamma$, it does guarantee strong dependence when it is large (at least for $\alpha <1$). When $\alpha \to 1^{-}$ we have $\beta \to 0$, and consequently the upper bound $e^{\beta t}{e}^{-\beta I_{\alpha }^{\left(\gamma \right)}(X⇝Y)}\to 1$ making the inequality trivial.

The “mutual information” quantity $I_{\alpha }^{\left(\gamma \right)}(X⇝Y)$ clearly depends on the choice of the reference measure $\gamma$. Nonetheless, there are 3 families of Rényi mutual informations that are independent of the choice of reference measure, which we now introduce.

Definition 6.

Fix$\alpha \ge 0$.

1.

The Lapidoth-Pfister α-mutual information is defined as

$$J_{\alpha }(X;Y):=\underset{\mu \in \mathcal{P}\left(S\right),\nu \in \mathcal{P}\left(T\right)}{\min }{D}_{\alpha }({\mathbb{P}}_{X,Y}\parallel \mu \otimes \nu ).$$

2.

The Augustin-Csiszár α-mutual information is defined as

$$K_{\alpha }^{X⇝Y}\left({\mathbb{P}}_{X,Y}\right)=\underset{\mu \in \mathcal{P}\left(T\right)}{min}{\mathbb{E}}_X{D}_{\alpha }({\mathbb{P}}_{Y|X}(·|X)\parallel \mu ).$$

3.

Sibson’s α-mutual information is defined as

$$I_{\alpha }^{X⇝Y}\left({\mathbb{P}}_{X,Y}\right)=\underset{\mu \in \mathcal{P}\left(T\right)}{min}{D}_{\alpha }({\mathbb{P}}_{X,Y}\parallel {\mathbb{P}}_X\otimes \mu ).$$

The quantity $J_{\alpha }$ was recently introduced by Lapidoth and Pfister as a measure of independence in [45] (cf., [25,27,32]). The Augustin-Csiszár mutual information was originally introduced in [40] by Udo Augustin with a slightly different parametrization, and gained much popularity following Csiszár’s work in [21]. For a discussion on early work on this quantity and applications also see [42] and references therein. Both [40] and [42] treat abstract alphabets however the former is limited to $\alpha \in (0,1)$ while the latter treats all $\alpha \in (0,\infty )$. Sibson’s definition originates in [38] where he introduces $I_{\alpha }$ in the form of information radius (see, e.g, [33]), which is often written in terms of Gallager’s function (from [46]). Since all the quantities in the above definition are stated in terms of Rényi divergences not involving the reference measure $\gamma$, they themselves are independent of the reference measure. Their relationship with the Rényi divergence also shows that all of them are non-negative. Moreover, putting $\mu ={\mathbb{P}}_X,\nu ={\mathbb{P}}_Y$ in the expression for $J_{\alpha }$ and $\mu ={\mathbb{P}}_Y$ in expressions for $K_{\alpha }$ and $I_{\alpha }$ when $X,Y$ are independent shows that they all vanish under independence.

While these notions of mutual information are certainly not equal to $I_{\alpha }^{\left(\gamma \right)}$ in general when $\alpha \ne 1$, they do have a direct relationship with conditional Rényi entropies, by varying the reference measure.

Since ${min}_{\mu }{min}_{\nu }{D}_{\alpha }({\mathbb{P}}_{X,Y}\parallel \mu \otimes \nu )={\min }_{\mu }-h_{\alpha }^{\mu }\left(X\right|Y)=-{\max }_{\mu }{h}_{\alpha }^{\mu }\left(X\right|Y)$, where all optimizations are done over probability measures, we can write Lapidoth and Pfister’s mutual information as

$$J_{\alpha }(X;Y)=-\underset{\mu }{\max }{h}_{\alpha }^{\mu }\left(X\right|Y)=\underset{\mu }{\min }-\frac{\alpha }{1-\alpha }\log {\int }_T{e}^{-\frac{1-\alpha }{\alpha }{D}_{\alpha }({\mathbb{P}}_{X|Y=y}\parallel \mu )}\mathrm{d}{\mathbb{P}}_Y\left(y\right).$$

Note that it is symmetric by definition: $J_{\alpha }(X;Y)=J_{\alpha }(Y;X)$, which is why we do not use squiggly arrows to denote it. By writing down Rényi divergence as Rényi entropy w.r.t. reference measure, Augustin-Csiszár’s $K_{\alpha }$ can be recast in a similar form, this time using the average Rényi entropy of the conditionals instead of Arimoto’s conditional Rényi entropy,

$$K_{\alpha }^{X⇝Y}=-\underset{\mu }{\max }{\mathbb{E}}_X{h}_{\alpha }^{\mu }\left(Y\right|X=x)=-\underset{\mu }{\max }{\tilde{h}}_{\alpha }^{\mu }\left(Y\right|X).$$

In light of Theorem 1, Sibson’s mutual information can clearly be written in terms of conditional Rényi entropy as

$$I_{\alpha }^{X⇝Y}=-h_{\alpha }^{{\mathbb{P}}_X}\left(X\right|Y).$$

This leads to the observation that Sibson’s mutual information can be seen as a special case of Arimoto’s mutual information, when the reference measure is taken to be the distribution of X:

$$\begin{array}{c}{I}_{\alpha }^{X⇝Y}=-h_{\alpha }^{{\mathbb{P}}_X}\left(X\right|Y)\hfill \\ =h_{\alpha }^{{\mathbb{P}}_X}\left(X\right)-h_{\alpha }^{{\mathbb{P}}_X}\left(X\right|Y)=I_{\alpha }^{\left({\mathbb{P}}_X\right)}(X⇝Y).\hfill \end{array}$$

For the sake of comparision with the corresponding expression for $I_{\alpha }^{\left(\gamma \right)}(X⇝Y)$, we also write $I_{\alpha }^{X⇝Y}$ as

$$I_{\alpha }^{X⇝Y}=-\frac{\alpha }{1-\alpha }\log {\int }_T{e}^{-\frac{1-\alpha }{\alpha }{D}_{\alpha }({\mathbb{P}}_{X|Y=y}\parallel {\mathbb{P}}_X)}\mathrm{d}{\mathbb{P}}_Y\left(y\right).$$

The following inequality, which relates the three families when $\alpha \ge 1$, turns out to be quite fruitful.

Theorem 3.

For$\alpha \ge 1$, we have

$$K_{\alpha }^{X⇝Y}\le J_{\alpha }(X;Y)\le I_{\alpha }^{X⇝Y}.$$

Proof. 

Suppose $\alpha \ge 1$. Then, $h_{\alpha }^{\mu }\left(Y\right|X)\le {\tilde{h}}_{\alpha }^{\mu }\left(Y\right|X)$ so that

$$\begin{array}{cc}\hfill K_{\alpha }^{X⇝Y}=-\underset{\mu }{\max }{\tilde{h}}_{\alpha }^{\mu }\left(Y\right|X)& \le -\underset{\mu }{\max }{h}_{\alpha }^{\mu }\left(Y\right|X)\hfill \\ & =J_{\alpha }(Y;X)=J_{\alpha }(X;Y).\hfill \end{array}$$

Moreover,

$$J_{\alpha }(X;Y)=-\underset{\mu }{\max }{h}_{\alpha }^{\mu }\left(X\right|Y)\le -h_{\alpha }^{{\mathbb{P}}_X}\left(X\right|Y)=I_{\alpha }^{X⇝Y},$$

which completes the proof. □

Remark 7.

When$\alpha \in (0,1)$, from the straightforward observation$J_{\alpha }(X;Y)\le I_{\alpha }^{X⇝Y}$and [21], we have$J_{\alpha }(X;Y)\le I_{\alpha }^{X⇝Y}\le K_{\alpha }^{X⇝Y}$.

We note that the relation between $K_{\alpha }^{X⇝Y}$ and $I_{\alpha }^{X⇝Y}$ (in the finite alphabet case) goes back to Csiszár [21]. In the next and final section, we explore the implications of Theorem 3 for various notions of capacity.

6. Channels and Capacities

We begin by defining channel and capacities. Throughout this section, assume $\alpha \ge 1$.

Definition 7.

Let$(A,\mathcal{A}),(B,\mathcal{B})$be measurable spaces. A function$W:A\times \mathcal{B}\to \mathbb{R}$is called a probability kernel or a channel from the input space$(A,\mathcal{A})$to the output space$(B,\mathcal{B})$if

1.

For all$a\in A$, the function$W(·|a):\mathcal{B}\to \mathbb{R}$is a probability measure on$(B,\mathcal{B})$, and

2.

For every$V\in \mathcal{B}$, the function$W\left(V\right|·):A\to \mathbb{R}$is a $\mathcal{A}$-measurable function.

In our setting, the conditional distributions X given $Y=y$ define a channel W from $\mathrm{supp}\left({\mathbb{P}}_Y\right)$ to S. In terms of this, one can write a density-free expression for the conditional Rényi entropy:

$$h_{\alpha }^{\gamma }\left(X\right|Y)=\frac{\alpha }{1-\alpha }\log {\int }_T{e}^{-\frac{1-\alpha }{\alpha }{D}_{\alpha }(W(·|y)\parallel \gamma )}\mathrm{d}{\mathbb{P}}_Y\left(y\right).$$

Definition 8.

Let$(B,\mathcal{B})$be a measurable space and$\mathcal{W}\subseteq \mathcal{P}\left(B\right)$a set of probability measures on B. Following [36], define the order-α Rényi radius of$\mathcal{W}$relative to$q\in \mathcal{P}\left(B\right)$by

$$S_{\alpha ,\mathcal{W}}\left(q\right)=\underset{w\in \mathcal{W}}{sup}{D}_{\alpha }(w\parallel q).$$

The order-α Rényi radius of$\mathcal{W}$is defined as

$$S_{\alpha }\left(\mathcal{W}\right)=\underset{q\in \mathcal{P}\left(B\right)}{\inf }{S}_{\alpha ,\mathcal{W}}\left(q\right).$$

Given a joint distribution ${\mathbb{P}}_{X,Y}$ of $(X,Y)$, one can consider the quantities $I_{\alpha }^{X⇝Y},I_{\alpha }^{Y⇝X},K_{\alpha }^{X⇝Y},K_{\alpha }^{Y⇝X}$ and $J_{\alpha }(X;Y)$, as functions of an “input” distribution ${\mathbb{P}}_X$ and the channel from X to Y (i.e., the probability kernel formed by the conditional distributions of Y given $X=x$). For example, one can consider $I_{\alpha }^{Y⇝X}={\inf }_{\nu \in \mathcal{P}\left(S\right)}{D}_{\alpha }({\mathbb{P}}_{X,Y}\parallel \nu \otimes {\mathbb{P}}_Y)$ as a function of ${\mathbb{P}}_X$ and the probability kernel formed by the conditional distributions of Y given $X=x$. Under this interpretation, we first define various capacities.

Definition 9.

Given a channel W from X to Y, we define capacities of order α by

1.

${\mathcal{C}}_{K,\alpha }^{X⇝Y}\left(W\right)={\sup }_{P\in \mathcal{P}\left(S\right)}{K}_{\alpha }^{X⇝Y}(P,W)$,

2.

${\mathcal{C}}_{J,\alpha }\left(W\right)={\sup }_{P\in \mathcal{P}\left(S\right)}{J}_{\alpha }(P,W)$, and

3.

${\mathcal{C}}_{I,\alpha }^{X⇝Y}\left(W\right)={\sup }_{P\in \mathcal{P}\left(S\right)}{I}_{\alpha }^{X⇝Y}(P,W)$.

Theorem 3 allows us to extend ([31], Theorem 5) to include the capacity based on the Lapidoth-Pfister mutual information.

Theorem 4.

Let$\alpha \ge 1$, and fix a channel W from X to Y. Then,

$$\begin{array}{cc}\hfill {\mathcal{C}}_{K,\alpha }^{Y⇝X}\left(W\right)\le {\mathcal{C}}_{I,\alpha }^{X⇝Y}\left(W\right)& ={\mathcal{C}}_{J,\alpha }\left(W\right)={\mathcal{C}}_{K,\alpha }^{X⇝Y}\left(W\right)=S_{\alpha }\left(W\right)\le {\mathcal{C}}_{I,\alpha }^{Y⇝X}\left(W\right).\hfill \end{array}$$

Proof. 

Theorem 3 implies that,

$$\underset{P\in \mathcal{P}\left(\mathcal{S}\right)}{\sup }{K}_{\alpha }^{X⇝Y}(P,W)\le \underset{P\in \mathcal{P}\left(\mathcal{S}\right)}{\sup }{J}_{\alpha }(P,W)\le \underset{P\in \mathcal{P}\left(\mathcal{S}\right)}{\sup }{I}_{\alpha }^{X⇝Y}(P,W),$$

that is,

$${\mathcal{C}}_{K,\alpha }^{X⇝Y}\left(W\right)\le {\mathcal{C}}_{J,\alpha }\left(W\right)\le {\mathcal{C}}_{I,\alpha }^{X⇝Y}\left(W\right).$$

It was shown by Csiszár [21] in the finite alphabet setting (in fact, he showed this for all $\alpha >0$) that ${\mathcal{C}}_{I,\alpha }^{X⇝Y}\left(W\right)={\mathcal{C}}_{K,\alpha }^{X⇝Y}\left(W\right)$. Nakiboğlu demonstrates ${\mathcal{C}}_{I,\alpha }^{X⇝Y}\left(W\right)=S_{\alpha }\left(W\right)$ in [36] and ${\mathcal{C}}_{K,\alpha }^{X⇝Y}\left(W\right)=S_{\alpha }\left(W\right)$ in [42] for abstract alphabets. Putting all this together, we have

$${\mathcal{C}}_{I,\alpha }^{X⇝Y}\left(W\right)={\mathcal{C}}_{J,\alpha }\left(W\right)={\mathcal{C}}_{K,\alpha }^{X⇝Y}\left(W\right)=S_{\alpha }\left(W\right).$$

Finally, using the symmetry of $J_{\alpha }$ and Theorem 3 in a similar fashion again, we get

$${\mathcal{C}}_{K,\alpha }^{Y⇝X}\left(W\right)\le {\mathcal{C}}_{J,\alpha }\left(W\right)\le {\mathcal{C}}_{I,\alpha }^{Y⇝X}\left(W\right).$$

This completes the proof.□

The two inequalities in the last theorem cannot be improved to be equalities, this follows from a counter-example communicated to the authors by C. Pfister. Note that this theorem corrects Theorem V.1 in [41].

Since $J_{\alpha }(X;Y)$ is no longer sandwiched between $K_{\alpha }^{X⇝Y}$ and $I_{\alpha }^{X⇝Y}$ when $\alpha \in (0,1)$ the same argument cannot be used to deduce the equality of various capacities in this case. However, when $\alpha \in [1/2,1)$, a direct demonstration proves that the Lapidoth-Pfister capacity of a channel equals Rényi radius when the state spaces are finite.

Theorem 5.

Let$\alpha \in [\frac{1}{2},1)$, and fix a channel W from X to Y where X and Y take values in finite sets S and T respectively. Then,

$$\underset{P\in \mathcal{P}\left(S\right)}{\sup }{J}_{\alpha }(P,W)=S_{\alpha }\left(\mathcal{W}\right).$$

Proof. 

We continue using integral notation instead of summation. Note that,

$$J_{\alpha }(X;Y)=-\underset{\mu \in \mathcal{P}\left(T\right)}{\max }-\beta \log {\int }_S{e}^{\frac{1}{\beta }{D}_{\alpha }(W\left(x\right)\parallel \mu )}\mathrm{d}P\left(x\right)=\underset{\mu \in \mathcal{P}\left(T\right)}{\min }\beta \log {\int }_S{e}^{\frac{1}{\beta }{D}_{\alpha }(W\left(x\right)\parallel \mu )}\mathrm{d}P\left(x\right),$$

where $\beta =\frac{\alpha }{\alpha -1}.$ We consider the function $f(P,\mu )=\beta \log {\int }_S{e}^{\frac{1}{\beta }{D}_{\alpha }(W\left(x\right)\parallel \mu )}\mathrm{d}P\left(x\right)$ defined on $\mathcal{P}\left(S\right)\times \mathcal{P}\left(T\right).$ Observe that the function $g(P,\mu )=-e^{\frac{1}{\beta }f(P,\mu )}=-{\int }_S{e}^{\frac{1}{\beta }{D}_{\alpha }(W\left(x\right)\parallel \mu )}\mathrm{d}P\left(x\right)$ has the same minimax properties as f. We make the following observations about this function.

• g is linear in P.
• g is convex in μ.
  Follows from the proof in ([27], Lemma 17).
• g is continuous in each of the variables P and μ. Continuity in $\mu$ follows from continuity of $D_{\alpha }$ in the second coordinate (see, for example, in [29]) whereas continuity in P is a consequence of linearity of the integral (summation).

The above observations ensure that we can apply von Neumann’s convex minimax theorem to g, and therefore to f to conclude that

$$\underset{P}{\sup }{J}_{\alpha }(P,W)=\underset{P}{\sup }\underset{\mu }{\min }f(P,\mu )=\underset{\mu }{\min }\underset{P}{\sup }f(P,\mu ).$$

For a fixed $\mu$ however, ${\sup }_{P}f(P,\mu )={\sup }_P\beta \log \int e^{\frac{1}{\beta }{D}_{\alpha }(W\left(x\right)\parallel \mu )}\mathrm{d}P\left(x\right)={\sup }_x{D}_{\alpha }(W\left(x\right)\parallel \mu )$ (the RHS is clearly bigger than the LHS, for the other direction use measures $P={\delta }_{x_n}$ where $x_n$ is a supremum achieving sequence for the RHS). This shows that when $\alpha \ge 1/2$ the capacity coming from the $J_{\alpha }(X;Y)$ equals the Rényi radius if the state spaces are finite.□

Though we do not treat capacities coming from Arimoto’s mutual information in this paper due to its dependence on a reference measure, a remark can be made in this regard following B. Nakiboğlu’s [43] observation that Arimoto’s mutual information w.r.t. $\gamma$ of a joint distribution $(X,Y)$ can be written as a Sibson mutual information of some input probability measure P and the channel W from X to Y corresponding to ${\mathbb{P}}_{X,Y}$. Let $X,Y$ denote the marginals of $(X,Y)$. As before there are reference measures $\gamma$ on the state space S of X. Let P denote the probability measure on S with density $\frac{\mathrm{d}P}{\mathrm{d}\gamma }=e^{(1-\alpha )D_{\alpha }({\mathbb{P}}_X\parallel \gamma )}{\left(\frac{\mathrm{d}{\mathbb{P}}_X}{\mathrm{d}\gamma }\right)}^{\alpha }$. Then a calculation shows that

$$I_{\alpha }^{\left(\gamma \right)}(X⇝Y)=I_{\alpha }^{X⇝Y}(P,W).$$

Therefore, it follows that if a reference measure $\gamma$ is fixed, then the capacity of order $\alpha$ of a channel W calculated from Arimoto’s mutual information will be less than the capacity based on Sibson mutual information (which equals the Rényi radius of W).