On Relations Between the Relative Entropy and χ²-Divergence, Generalizations and Applications

Abstract

This paper is focused on a study of integral relations between the relative entropy and the chi-squared divergence, which are two fundamental divergence measures in information theory and statistics, a study of the implications of these relations, their information-theoretic applications, and some generalizations pertaining to the rich class of f-divergences. Applications that are studied in this paper refer to lossless compression, the method of types and large deviations, strong data–processing inequalities, bounds on contraction coefficients and maximal correlation, and the convergence rate to stationarity of a type of discrete-time Markov chains.

1. Introduction

The relative entropy (also known as the Kullback–Leibler divergence [1]) and the chi-squared divergence [2] are divergence measures which play a key role in information theory, statistics, learning, signal processing, and other theoretical and applied branches of mathematics. These divergence measures are fundamental in problems pertaining to source and channel coding, combinatorics and large deviations theory, goodness-of-fit and independence tests in statistics, expectation–maximization iterative algorithms for estimating a distribution from an incomplete data, and other sorts of problems (the reader is referred to the tutorial paper by Csiszár and Shields [3]). They both belong to an important class of divergence measures, defined by means of convex functions f, and named f-divergences [4,5,6,7,8]. In addition to the relative entropy and the chi-squared divergence, this class unifies other useful divergence measures such as the total variation distance in functional analysis, and it is also closely related to the Rényi divergence which generalizes the relative entropy [9,10]. In general, f-divergences (defined in Section 2) are attractive since they satisfy pleasing features such as the data–processing inequality, convexity, (semi)continuity, and duality properties, and they therefore find nice applications in information theory and statistics (see, e.g., [6,8,11,12]).

In this work, we study integral relations between the relative entropy and the chi-squared divergence, implications of these relations, and some of their information-theoretic applications. Some generalizations which apply to the class of f-divergences are also explored in detail. In this context, it should be noted that integral representations of general f-divergences, expressed as a function of either the DeGroot statistical information [13], the $E_{\gamma }$-divergence (a parametric sub-class of f-divergences, which generalizes the total variation distance [14] [p. 2314]) and the relative information spectrum, have been derived in [12] [Section 5], [15] [Section 7.B], and [16] [Section 3], respectively.

Applications in this paper are related to lossless source compression, large deviations by the method of types, and strong data–processing inequalities. The relevant background for each of these applications is provided to make the presentation self contained.

We next outline the paper contributions and the structure of our manuscript.

1.1. Paper Contributions

This work starts by introducing integral relations between the relative entropy and the chi-squared divergence, and some inequalities which relate these two divergences (see Theorem 1, its corollaries, and Proposition 1). It continues with a study of the implications and generalizations of these relations, pertaining to the rich class of f-divergences. One implication leads to a tight lower bound on the relative entropy between a pair of probability measures, expressed as a function of the means and variances under these measures (see Theorem 2). A second implication of Theorem 1 leads to an upper bound on a skew divergence (see Theorem 3 and Corollary 3). Due to the concavity of the Shannon entropy, let the concavity deficit of the entropy function be defined as the non-negative difference between the entropy of a convex combination of distributions and the convex combination of the entropies of these distributions. Then, Corollary 4 provides an upper bound on this deficit, expressed as a function of the pairwise relative entropies between all pairs of distributions. Theorem 4 provides a generalization of Theorem 1 to the class of f-divergences. It recursively constructs non-increasing sequences of f-divergences and as a consequence of Theorem 4 followed by the usage of polylogairthms, Corollary 5 provides a generalization of the useful integral relation in Theorem 1 between the relative entropy and the chi-squared divergence. Theorem 5 relates probabilities of sets to f-divergences, generalizing a known and useful result by Csiszár for the relative entropy. With respect to Theorem 1, the integral relation between the relative entropy and the chi-squared divergence has been independently derived in [17], which also derived an alternative upper bound on the concavity deficit of the entropy as a function of total variational distances (differing from the bound in Corollary 4, which depends on pairwise relative entropies). The interested reader is referred to [17], with a preprint of the extended version in [18], and to [19] where the connections in Theorem 1 were originally discovered in the quantum setting.

The second part of this work studies information-theoretic applications of the above results. These are ordered by starting from the relatively simple applications, and ending at the more complicated ones. The first one includes a bound on the redundancy of the Shannon code for universal lossless compression with discrete memoryless sources, used in conjunction with Theorem 3 (see Section 4.1). An application of Theorem 2 in the context of the method of types and large deviations analysis is then studied in Section 4.2, providing non-asymptotic bounds which lead to a closed-form expression as a function of the Lambert W-function (see Proposition 2). Strong data–processing inequalities with bounds on contraction coefficients of skew divergences are provided in Theorem 6, Corollary 7 and Proposition 3. Consequently, non-asymptotic bounds on the convergence to stationarity of time-homogeneous, irreducible, and reversible discrete-time Markov chains with finite state spaces are obtained by relying on our bounds on the contraction coefficients of skew divergences (see Theorem 7). The exact asymptotic convergence rate is also obtained in Corollary 8. Finally, a property of maximal correlations is obtained in Proposition 4 as an application of our starting point on the integral relation between the relative entropy and the chi-squared divergence.

1.2. Paper Organization

This paper is structured as follows. Section 2 presents notation and preliminary material which is necessary for, or otherwise related to, the exposition of this work. Section 3 refers to the developed relations between divergences, and Section 4 studies information-theoretic applications. Proofs of the results in Section 3 and Section 4 (except for short proofs) are deferred to Section 5.

2. Preliminaries and Notation

This section provides definitions of divergence measures which are used in this paper, and it also provides relevant notation.

Definition 1.

[12] [p. 4398] Let P and Q be probability measures, let μ be a dominating measure of P and Q (i.e., $P,Q\ll \mu$), and let $p:=\frac{\mathrm{d}P}{\mathrm{d}\mu }$ and $q:=\frac{\mathrm{d}Q}{\mathrm{d}\mu }$ be the densities of P and Q with respect to μ. The f-divergence from P to Q is given by

$$\begin{array}{c}\hfill D_f(P\parallel Q):=\int qf\left(\frac{p}{q}\right)\mathrm{d}\mu ,\end{array}$$

(1)

where

$$\begin{array}{cc}\hfill & f\left(0\right):=\underset{t\to 0^{+}}{lim}f\left(t\right),0f\left(\frac{0}{0}\right):=0,\hfill \end{array}$$

(2)

$$\begin{array}{cc}\hfill & 0f\left(\frac{a}{0}\right):=\underset{t\to 0^{+}}{lim}tf\left(\frac{a}{t}\right)=a\underset{u\to \infty }{lim}\frac{f\left(u\right)}{u},a>0.\hfill \end{array}$$

(3)

It should be noted that the right side of (1) does not depend on the dominating measure μ.

Throughout the paper, we denote by $1\left\{\mathrm{relation}\right\}$ the indicator function; it is equal to 1 if the relation is true, and it is equal to 0 otherwise. Throughout the paper, unless indicated explicitly, logarithms have an arbitrary common base (that is larger than 1), and $exp(·)$ indicates the inverse function of the logarithm with that base.

Definition 2.

[1] The relative entropy is the f-divergence with $f\left(t\right):=tlogt$ for $t>0$,

$$\begin{array}{}\hfill \mathrm{(4)}& \hfill D(P\parallel Q)& :=D_f(P\parallel Q)\hfill \hfill \mathrm{(5)}& & =\int plog\frac{p}{q}\mathrm{d}\mu .\hfill \end{array}$$

Definition 3.

The total variation distance between probability measures P and Q is the f-divergence from P to Q with $f\left(t\right):=|t-1|$ for all $t\ge 0$. It is a symmetric f-divergence, denoted by $|P-Q|$, which is given by

$$\begin{array}{}\hfill \mathrm{(6)}& |P-Q|& :=D_f(P\parallel Q)\hfill \mathrm{(7)}& & =\int |p-q|\mathrm{d}\mu .\hfill \end{array}$$

Definition 4.

[2] The chi-squared divergence from P to Q is defined to be the f-divergence in (1) with $f\left(t\right):={(t-1)}^2$ or $f\left(t\right):=t^2-1$ for all $t>0$,

$$\begin{array}{}\hfill \mathrm{(8)}& {\chi }^2(P\parallel Q)\hfill & :=D_f(P\parallel Q)\hfill \hfill \mathrm{(9)}& & =\int \frac{{(p-q)}^2}{q}\mathrm{d}\mu =\int \frac{p^2}{q}\mathrm{d}\mu -1.\hfill \end{array}$$

The Rényi divergence, a generalization of the relative entropy, was introduced by Rényi [10] in the special case of finite alphabets. Its general definition is given as follows (see, e.g., [9]).

Definition 5.

[10] Let P and Q be probability measures on $\mathcal{X}$ dominated by μ, and let their densities be respectively denoted by $p=\frac{\mathrm{d}P}{\mathrm{d}\mu }$ and $q=\frac{\mathrm{d}Q}{\mathrm{d}\mu }$. The Rényi divergence of order $\alpha \in [0,\infty ]$ is defined as follows:

• If $\alpha \in (0,1)\cup (1,\infty )$, then

  $$\begin{array}{}\hfill \mathrm{(10)}& D_{\alpha }(P\parallel Q)& =\frac{1}{\alpha -1}log\mathbb{E}\left[p^{\alpha }\left(Z\right)q^{1-\alpha }\left(Z\right)\right]\hfill \hfill \mathrm{(11)}& & =\frac{1}{\alpha -1}log\sum _{x\in \mathcal{X}}{P}^{\alpha }\left(x\right)Q^{1-\alpha }\left(x\right),\hfill \end{array}$$

  where $Z\sim \mu$ in (10), and (11) holds if $\mathcal{X}$ is a discrete set.
• By the continuous extension of $D_{\alpha }(P\parallel Q)$,

  $$\begin{array}{}\hfill \mathrm{(12)}& D_0(P\parallel Q)=\underset{\mathcal{A}:P\left(\mathcal{A}\right)=1}{max}log\frac{1}{Q\left(\mathcal{A}\right)},\hfill \hfill \mathrm{(13)}& D_1(P\parallel Q)=D(P\parallel Q),\hfill \hfill \mathrm{(14)}& D_{\infty }(P\parallel Q)=logess\; sup\frac{p\left(Z\right)}{q\left(Z\right)}.\hfill \end{array}$$

The second-order Rényi divergence and the chi-squared divergence are related as follows:

$$\begin{array}{c}\hfill D_2(P\parallel Q)=log\left(1+{\chi }^2(P\parallel Q)\right),\end{array}$$

(15)

and the relative entropy and the chi-squared divergence satisfy (see, e.g., [20] [Theorem 5])

$$\begin{array}{c}\hfill D(P\parallel Q)\le log\left(1+{\chi }^2(P\parallel Q)\right).\end{array}$$

(16)

Inequality (16) readily follows from (13), (15), and since $D_{\alpha }(P\parallel Q)$ is monotonically increasing in $\alpha \in (0,\infty )$ (see [9] [Theorem 3]). A tightened version of (16), introducing an improved and locally-tight upper bound on $D(P\parallel Q)$ as a function of ${\chi }^2(P\parallel Q)$ and ${\chi }^2(Q\parallel P)$, is introduced in [15] [Theorem 20]. Another sharpened version of (16) is derived in [15] [Theorem 11] under the assumption of a bounded relative information. Furthermore, under the latter assumption, tight upper and lower bounds on the ratio $\frac{D(P\parallel Q)}{{\chi }^2(P\parallel Q)}$ are obtained in [15] [(169)].

Definition 6.

[21] The Györfi–Vajda divergence of order $s\in [0,1]$ is an f-divergence with

$$\begin{array}{c}\hfill f\left(t\right)={\varphi }_s\left(t\right):=\frac{{(t-1)}^2}{s+(1-s)t},t\ge 0.\end{array}$$

(17)

Vincze–Le Cam distance (also known as the triangular discrimination) ([22,23]) is a special case with $s=\frac{1}{2}$.

In view of (1), (9) and (17), it can be verified that the Györfi–Vajda divergence is related to the chi-squared divergence as follows:

$$D_{{\varphi }_s}(P\parallel Q)=\{\begin{array}{ll}\frac{1}{s^2}·{\chi }^2\left(P\parallel (1-s)P+sQ\right),& s\in (0,1],\\ {\chi }^2(Q\parallel P),& s=0.\end{array}$$

(18)

Hence,

$$\begin{array}{}\hfill \mathrm{(19)}& D_{{\varphi }_1}(P\parallel Q)={\chi }^2(P\parallel Q),\hfill \hfill \mathrm{(20)}& D_{{\varphi }_0}(P\parallel Q)={\chi }^2(Q\parallel P).\hfill \end{array}$$

3. Relations between Divergences

We introduce in this section results on the relations between the relative entropy and the chi-squared divergence, their implications, and generalizations. Information–theoretic applications are studied in the next section.

3.1. Relations between the Relative Entropy and the Chi-Squared Divergence

The following result relates the relative entropy and the chi-squared divergence, which are two fundamental divergence measures in information theory and statistics. This result was recently obtained in an equivalent form in [17] [(12)] (it is noted that this identity was also independently derived by the coauthors in two separate un-published works in [24] [(16)] and [25]). It should be noted that these connections between divergences in the quantum setting were originally discovered in [19] [Theorem 6]. Beyond serving as an interesting relation between these two fundamental divergence measures, it is introduced here for the following reasons:

(a)

New consequences and applications of it are obtained, including new shorter proofs of some known results;

(b)

An interesting extension provides new relations between f-divergences (see Section 3.3).

Theorem 1.

Let P and Q be probability measures defined on a measurable space $(\mathcal{X},\mathcal{F})$, and let

$$\begin{array}{c}\hfill R_{\lambda }:=(1-\lambda )P+\lambda Q,\lambda \in [0,1]\end{array}$$

(21)

be the convex combination of P and Q. Then, for all $\lambda \in [0,1]$,

$$\begin{array}{cc}\hfill \frac{1}{log\mathrm{e}}D(P\parallel R_{\lambda })& ={\int }_0^{\lambda }{\chi }^2(P\parallel R_s)\frac{\mathrm{d}s}{s},\hfill \end{array}$$

(22)

$$\begin{array}{cc}\hfill \frac{1}{2}{\lambda }^2{\chi }^2(R_{1-\lambda }\parallel Q)& ={\int }_0^{\lambda }{\chi }^2(R_{1-s}\parallel Q)\frac{\mathrm{d}s}{s}.\hfill \end{array}$$

(23)

Proof. 

See Section 5.1. □

A specialization of Theorem 1 by letting $\lambda =1$ gives the following identities.

Corollary 1.

$$\begin{array}{c}\hfill \frac{1}{log\mathrm{e}}D(P\parallel Q)={\int }_0^1{\chi }^2(P\parallel (1-s)P+sQ)\frac{\mathrm{d}s}{s},\end{array}$$

(24)

$$\begin{array}{c}\hfill \frac{1}{2}{\chi }^2(P\parallel Q)={\int }_0^1{\chi }^2(sP+(1-s)Q\parallel Q)\frac{\mathrm{d}s}{s}.\end{array}$$

(25)

Remark 1.

The substitution $s:=\frac{1}{1+t}$ transforms (24) to [26] [Equation (31)], i.e.,

$$\begin{array}{c}\hfill \frac{1}{log\mathrm{e}}D(P\parallel Q)={\int }_0^{\infty }{\chi }^2\left(P\parallel \frac{tP+Q}{1+t}\right)\frac{\mathrm{d}t}{1+t}.\end{array}$$

(26)

In view of (18) and (21), an equivalent form of (22) and (24) is given as follows:

Corollary 2.

For $s\in [0,1]$, let ${\varphi }_s:[0,\infty )\to \mathbb{R}$ be given in (17). Then,

$$\begin{array}{}\hfill \mathrm{(27)}& \hfill \frac{1}{log\mathrm{e}}D(P\parallel R_{\lambda })& ={\int }_0^{\lambda }s{D}_{{\varphi }_s}(P\parallel Q)\mathrm{d}s,\lambda \in [0,1],\hfill \hfill \mathrm{(28)}& \hfill \frac{1}{log\mathrm{e}}D(P\parallel Q)& ={\int }_0^{1}s{D}_{{\varphi }_s}(P\parallel Q)\mathrm{d}s.\hfill \end{array}$$

By Corollary 1, we obtain original and simple proofs of new and old f-divergence inequalities.

Proposition 1.

(f-divergence inequalities).

(a) 

Pinsker’s inequality:

$$\begin{array}{c}\hfill D(P\parallel Q)\ge \frac{1}{2}{|P-Q|}^{2}log\mathrm{e}.\end{array}$$

(29)

(b) 

$$\begin{array}{c}\hfill \frac{1}{log\mathrm{e}}D(P\parallel Q)\le \frac{1}{3}{\chi }^2(P\parallel Q)+\frac{1}{6}{\chi }^2(Q\parallel P).\end{array}$$

(30)

Furthermore, let $\left\{P_n\right\}$ be a sequence of probability measures that is defined on a measurable space $(\mathcal{X},\mathcal{F})$, and which converges to a probability measure P in the sense that

$$\begin{array}{c}\hfill \underset{n\to \infty }{lim}ess\; sup\frac{\mathrm{d}{P}_n}{\mathrm{d}P}\left(X\right)=1,\end{array}$$

(31)

with $X\sim P$. Then, (30) is locally tight in the sense that its both sides converge to 0, and

$$\begin{array}{c}\hfill \underset{n\to \infty }{lim}\frac{\frac{1}{3}{\chi }^2(P_n\parallel P)+\frac{1}{6}{\chi }^2(P\parallel P_n)}{\frac{1}{log\mathrm{e}}D(P_n\parallel P)}=1.\end{array}$$

(32)

(c) 

For all $\theta \in (0,1)$,

$$\begin{array}{c}\hfill D(P\parallel Q)\ge (1-\theta )log\left(\frac{1}{1-\theta }\right)D_{{\varphi }_{\theta }}(P\parallel Q).\end{array}$$

(33)

Moreover, under the assumption in (31), for all $\theta \in [0,1]$

$$\begin{array}{c}\hfill \underset{n\to \infty }{lim}\frac{D(P\parallel P_n)}{D_{{\varphi }_{\theta }}(P\parallel P_n)}=\frac{1}{2}log\mathrm{e}.\end{array}$$

(34)

(d) 

[15] [Theorem 2]:

$$\begin{array}{c}\hfill \frac{1}{log\mathrm{e}}D(P\parallel Q)\le \frac{1}{2}{\chi }^2(P\parallel Q)+\frac{1}{4}|P-Q|.\end{array}$$

(35)

Proof. 

See Section 5.2. □

Remark 2.

Inequality (30) is locally tight in the sense that (31) yields (32). This property, however, is not satisfied by (16) since the assumption in (31) implies that

$$\begin{array}{c}\hfill \underset{n\to \infty }{lim}\frac{log\left(1+{\chi }^2(P_n\parallel P)\right)}{D(P_n\parallel P)}=2.\end{array}$$

(36)

Remark 3.

Inequality (30) readily yields

$$\begin{array}{cc}\hfill & D(P\parallel Q)+D(Q\parallel P)\le \frac{1}{2}\left({\chi }^2(P\parallel Q)+{\chi }^2(Q\parallel P)\right)log\mathrm{e},\hfill \end{array}$$

(37)

which is proved by a different approach in [27] [Proposition 4]. It is further shown in [15] [Theorem 2 b)] that

$$\begin{array}{c}\hfill sup\frac{D(P\parallel Q)+D(Q\parallel P)}{{\chi }^2(P\parallel Q)+{\chi }^2(Q\parallel P)}=\frac{1}{2}log\mathrm{e},\end{array}$$

(38)

where the supremum is over $P\ll \gg Q$ and $P\ne Q$.

3.2. Implications of Theorem 1

We next provide two implications of Theorem 1. The first implication, which relies on the Hammersley–Chapman–Robbins (HCR) bound for the chi-squared divergence [28,29], gives the following tight lower bound on the relative entropy $D(P\parallel Q)$ as a function of the means and variances under P and Q.

Theorem 2.

Let P and Q be probability measures defined on the measurable space $(\mathbb{R},\mathcal{B})$, where $\mathbb{R}$ is the real line and $\mathcal{B}$ is the Borel σ–algebra of subsets of $\mathbb{R}$. Let $m_P$, $m_Q$, ${\sigma }_P^2$, and ${\sigma }_Q^2$ denote the expected values and variances of $X\sim P$ and $Y\sim Q$, i.e.,

$$\begin{array}{cc}\hfill & \mathbb{E}\left[X\right]=:m_P,\mathbb{E}\left[Y\right]=:m_Q,\mathrm{Var}\left(X\right)=:{\sigma }_P^2,\mathrm{Var}\left(Y\right)=:{\sigma }_Q^2.\hfill \end{array}$$

(39)

(a) 

If $m_P\ne m_Q$, then

$$\begin{array}{c}\hfill D(P\parallel Q)\ge d(r\parallel s),\end{array}$$

(40)

where $d(r\parallel s):=rlog\frac{r}{s}+(1-r)log\frac{1-r}{1-s}$, for $r,s\in [0,1]$, denotes the binary relative entropy (with the convention that $0log\frac{0}{0}=0$), and

$$\begin{array}{}\hfill \mathrm{(41)}& r:=\frac{1}{2}+\frac{b}{4av}\in [0,1],\hfill \mathrm{(42)}& s:=r-\frac{a}{2v}\in [0,1],\hfill \hfill \mathrm{(43)}& a:=m_P-m_Q,\hfill \hfill \mathrm{(44)}& b:=a^2+{\sigma }_Q^2-{\sigma }_P^2,\hfill \hfill \mathrm{(45)}& v:=\sqrt{{\sigma }_P^2+\frac{b^2}{4a^2}}.\hfill \end{array}$$

(b) 

The lower bound on the right side of (40) is attained for P and Q which are defined on the two-element set $\mathcal{U}:=\{u_1,u_2\}$, and

$$\begin{array}{c}\hfill P\left(u_1\right)=r,Q\left(u_1\right)=s,\end{array}$$

(46)

with r and s in (41) and (42), respectively, and for $m_P\ne m_Q$

$$\begin{array}{cc}\hfill & u_1:=m_P+\sqrt{\frac{(1-r){\sigma }_P^2}{r}},u_2:=m_P-\sqrt{\frac{r{\sigma }_P^2}{1-r}}.\hfill \end{array}$$

(47)

(c) 

If $m_P=m_Q$ and ${\sigma }_P$ and ${\sigma }_Q$ are selected arbitrarily, then

$$\begin{array}{c}\hfill \underset{P,Q}{inf}D(P\parallel Q)=0,\end{array}$$

(48)

where the infimum on the left side of (48) is taken over all P and Q which satisfy (39).

Proof. 

See Section 5.3. □

Remark 4.

Consider the case of the non-equal means in Items (a) and (b) of Theorem 2. If these means are fixed, then the infimum of $D(P\parallel Q)$ is zero by choosing arbitrarily large equal variances. Suppose now that the non-equal means $m_P$ and $m_Q$ are fixed, as well as one of the variances (either ${\sigma }_P^2$ or ${\sigma }_Q^2$). Numerical experimentation shows that, in this case, the achievable lower bound in (40) is monotonically decreasing as a function of the other variance, and it tends to zero as we let the free variance tend to infinity. This asymptotic convergence to zero can be justified by assuming, for example, that $m_P,m_Q$, and ${\sigma }_Q^2$ are fixed, and $m_P>m_Q$ (the other cases can be justified in a similar way). Then, it can be verified from (41)–(45) that

$$\begin{array}{c}\hfill r=\frac{{(m_P-m_Q)}^2}{{\sigma }_P^2}+O\left(\frac{1}{{\sigma }_P^4}\right),s=O\left(\frac{1}{{\sigma }_P^4}\right),\end{array}$$

(49)

which implies that $d(r\parallel s)\to 0$ as we let ${\sigma }_P\to \infty$. The infimum of the relative entropy $D(P\parallel Q)$ is therefore equal to zero since the probability measures P and Q in (46) and (47), which are defined on a two-element set and attain the lower bound on the relative entropy under the constraints in (39), have a vanishing relative entropy in this asymptotic case.

Remark 5.

The proof of Item (c) in Theorem 2 suggests explicit constructions of sequences of pairs probability measures $\left\{(P_n,Q_n)\right\}$ such that

(a) 

The means under $P_n$ and $Q_n$ are both equal to m (independently of n);

(b) 

The variance under $P_n$ is equal to ${\sigma }_P^2$, and the variance under $Q_n$ is equal to ${\sigma }_Q^2$ (independently of n);

(c) 

The relative entropy $D(P_n\parallel Q_n)$ vanishes as we let $n\to \infty$.

This yields in particular (48).

A second consequence of Theorem 1 gives the following result. Its first part holds due to the concavity of $exp\left(-D(P\parallel ·)\right)$ (see [30] [Problem 4.2]). The second part is new, and its proof relies on Theorem 1. As an educational note, we provide an alternative proof of the first part by relying on Theorem 1.

Theorem 3.

Let $P\ll Q$, and $F:[0,1]\to [0,\infty )$ be given by

$$\begin{array}{c}\hfill F\left(\lambda \right):=D\left(P\parallel (1-\lambda )P+\lambda Q\right),\forall \lambda \in [0,1].\end{array}$$

(50)

Then, for all $\lambda \in [0,1]$,

$$\begin{array}{c}\hfill F\left(\lambda \right)\le log\left(\frac{1}{1-\lambda +\lambda exp\left(-D(P\parallel Q)\right)}\right),\end{array}$$

(51)

with an equality if $\lambda =0$ or $\lambda =1$. Moreover, F is monotonically increasing, differentiable, and it satisfies

$$\begin{array}{cc}\hfill & F^{\prime }\left(\lambda \right)\ge \frac{1}{\lambda }\left[exp\left(F\left(\lambda \right)\right)-1\right]log\mathrm{e},\forall \lambda \in (0,1],\hfill \end{array}$$

(52)

$$\begin{array}{cc}\hfill & \underset{\lambda \to 0^{+}}{lim}\frac{F^{\prime }\left(\lambda \right)}{\lambda }={\chi }^2(Q\parallel P)log\mathrm{e},\hfill \end{array}$$

(53)

so the limit in (53) is twice as large as the value of the lower bound on this limit as it follows from the right side of (52).

Proof. 

See Section 5.4. □

Remark 6.

By the convexity of the relative entropy, it follows that $F\left(\lambda \right)\le \lambda D(P\parallel Q)$ for all $\lambda \in [0,1]$. It can be verified, however, that the inequality $1-\lambda +\lambda exp(-x)\ge exp(-\lambda x)$ holds for all $x\ge 0$ and $\lambda \in [0,1]$. Letting $x:=D(P\parallel Q)$ implies that the upper bound on $F\left(\lambda \right)$ on the right side of (51) is tighter than or equal to the upper bound $\lambda D(P\parallel Q)$ (with an equality if and only if either $\lambda \in \{0,1\}$ or $P\equiv Q$).

Corollary 3.

Let ${\left\{P_j\right\}}_{j=1}^m$, with $m\in \mathbb{N}$, be probability measures defined on a measurable space $(\mathcal{X},\mathcal{F})$, and let ${\left\{{\alpha }_j\right\}}_{j=1}^m$ be a sequence of non-negative numbers that sum to 1. Then, for all $i\in \{1,\dots ,m\}$,

$$\begin{array}{cc}\hfill D\left(P_i\parallel \sum _{j=1}^m{\alpha }_j{P}_j\right)& \le -log\left({\alpha }_i+(1-{\alpha }_i)exp\left(-\frac{1}{1-{\alpha }_i}\sum _{j\ne i}{\alpha }_{j}D(P_i\parallel P_j)\right)\right).\hfill \end{array}$$

(54)

Proof. 

For an arbitrary $i\in \{1,\dots ,m\}$, apply the upper bound on the right side of (51) with $\lambda :=1-{\alpha }_i$, $P:=P_i$ and $Q:=\frac{1}{1-{\alpha }_i}{\displaystyle \sum _{j\ne i}}{\alpha }_j{P}_j$. The right side of (54) is obtained from (51) by invoking the convexity of the relative entropy, which gives $D(P_i\parallel Q)\le \frac{1}{1-{\alpha }_i}{\displaystyle \sum _{j\ne i}}{\alpha }_{j}D(P_i\parallel P_j)$. □

The next result provides an upper bound on the non-negative difference between the entropy of a convex combination of distributions and the respective convex combination of the individual entropies (it is also termed as the concavity deficit of the entropy function in [17] [Section 3]).

Corollary 4.

Let ${\left\{P_j\right\}}_{j=1}^m$, with $m\in \mathbb{N}$, be probability measures defined on a measurable space $(\mathcal{X},\mathcal{F})$, and let ${\left\{{\alpha }_j\right\}}_{j=1}^m$ be a sequence of non-negative numbers that sum to 1. Then,

$$\begin{array}{c}\hfill 0\le H\left(\sum _{j=1}^m{\alpha }_j{P}_j\right)-\sum _{j=1}^m{\alpha }_{j}H\left(P_j\right)\le -\sum _{i=1}^m{\alpha }_{i}log\left({\alpha }_i+(1-{\alpha }_i)exp\left(-\frac{1}{1-{\alpha }_i}\sum _{j\ne i}{\alpha }_{j}D(P_i\parallel P_j)\right)\right).\end{array}$$

(55)

Proof. 

The lower bound holds due to the concavity of the entropy function. The upper bound readily follows from Corollary 3, and the identity

$$\begin{array}{c}\hfill H\left(\sum _{j=1}^m{\alpha }_j{P}_j\right)-\sum _{j=1}^m{\alpha }_{j}H\left(P_j\right)=\sum _{i=1}^m{\alpha }_{i}D\left(P_i\parallel \sum _{j=1}^m{\alpha }_j{P}_j\right).\end{array}$$

(56)

 □

Remark 7.

The upper bound in (55) refines the known bound (see, e.g., [31] [Lemma 2.2])

$$\begin{array}{c}\hfill H\left(\sum _{j=1}^m{\alpha }_j{P}_j\right)-\sum _{j=1}^m{\alpha }_{j}H\left(P_j\right)\le \sum _{j=1}^m{\alpha }_{j}log\frac{1}{{\alpha }_j}=H\left(\underline{\alpha }\right),\end{array}$$

(57)

by relying on all the $\frac{1}{2}m(m-1)$ pairwise relative entropies between the individual distributions ${\left\{P_j\right\}}_{j=1}^m$. Another refinement of (57), expressed in terms of total variation distances, has been recently provided in [17] [Theorem 3.1].

3.3. Monotonic Sequences of f-Divergences and an Extension of Theorem 1

The present subsection generalizes Theorem 1, and it also provides relations between f-divergences which are defined in a recursive way.

Theorem 4.

Let P and Q be probability measures defined on a measurable space $(\mathcal{X},\mathcal{F})$. Let $R_{\lambda }$, for $\lambda \in [0,1]$, be the convex combination of P and Q as in (21). Let $f_0:(0,\infty )\to \mathbb{R}$ be a convex function with $f_0\left(1\right)=0$, and let ${\left\{f_k(·)\right\}}_{k=0}^{\infty }$ be a sequence of functions that are defined on $(0,\infty )$ by the recursive equation

$$\begin{array}{c}\hfill f_{k+1}\left(x\right):={\int }_0^{1-x}{f}_k(1-s)\frac{\mathrm{d}s}{s},x>0,k\in \{0,1,\dots \}.\end{array}$$

(58)

Then,

(a) 

${\left\{D_{f_k}(P\parallel Q)\right\}}_{k=0}^{\infty }$ is a non-increasing (and non-negative) sequence of f-divergences.

(b) 

For all $\lambda \in [0,1]$ and $k\in \{0,1,\dots \}$,

$$\begin{array}{c}\hfill D_{f_{k+1}}(R_{\lambda }\parallel P)={\int }_0^{\lambda }{D}_{f_k}(R_s\parallel P)\frac{\mathrm{d}s}{s}.\end{array}$$

(59)

Proof. 

See Section 5.5. □

We next use the polylogarithm functions, which satisfy the recursive equation [32] [Equation (7.2)]:

$${\mathrm{Li}}_k\left(x\right):=\{\begin{array}{ll}\frac{x}{1-x},& ifk=0,\hfill \\ \hfill {\int }_0^x\frac{{\mathrm{Li}}_{k-1}\left(s\right)}{s}\mathrm{d}s,& ifk\ge 1.\end{array}$$

(60)

This gives ${\mathrm{Li}}_1\left(x\right)=-{log}_{\mathrm{e}}(1-x)$, ${\mathrm{Li}}_2\left(x\right)=-{\int }_0^x\frac{1}{s}{log}_{\mathrm{e}}(1-s)\mathrm{d}s$ and so on, which are real–valued and finite for $x<1$.

Corollary 5.

Let

$$\begin{array}{c}\hfill f_k\left(x\right):={\mathrm{Li}}_k(1-x),x>0,k\in \{0,1,\dots \}.\end{array}$$

(61)

Then, (59) holds for all $\lambda \in [0,1]$ and $k\in \{0,1,\dots \}$. Furthermore, setting $k=0$ in (59) yields (22) as a special case.

Proof. 

See Section 5.6. □

3.4. On Probabilities and f-Divergences

The following result relates probabilities of sets to f-divergences.

Theorem 5.

Let $(\mathcal{X},\mathcal{F},\mu )$ be a probability space, and let $\mathcal{C}\in \mathcal{F}$ be a measurable set with $\mu \left(\mathcal{C}\right)>0$. Define the conditional probability measure

$$\begin{array}{c}\hfill {\mu }_{\mathcal{C}}\left(\mathcal{E}\right):=\frac{\mu (\mathcal{C}\cap \mathcal{E})}{\mu \left(\mathcal{C}\right)},\forall \mathcal{E}\in \mathcal{F}.\end{array}$$

(62)

Let $f:(0,\infty )\to \mathbb{R}$ be an arbitrary convex function with $f\left(1\right)=0$, and assume (by continuous extension of f at zero) that $f\left(0\right):={\displaystyle \underset{t\to 0^{+}}{lim}}f\left(t\right)<\infty$. Furthermore, let $\tilde{f}:(0,\infty )\to \mathbb{R}$ be the convex function which is given by

$$\begin{array}{c}\hfill \tilde{f}\left(t\right):=tf\left(\frac{1}{t}\right),\forall t>0.\end{array}$$

(63)

Then,

$$\begin{array}{c}\hfill D_f({\mu }_{\mathcal{C}}\parallel \mu )=\tilde{f}\left(\mu \left(\mathcal{C}\right)\right)+\left(1-\mu \left(\mathcal{C}\right)\right)f\left(0\right).\end{array}$$

(64)

Proof. 

See Section 5.7. □

Connections of probabilities to the relative entropy, and to the chi-squared divergence, are next exemplified as special cases of Theorem 5.

Corollary 6.

In the setting of Theorem 5,

$$\begin{array}{cc}\hfill & D\left({\mu }_{\mathcal{C}}\parallel \mu \right)=log\frac{1}{\mu \left(\mathcal{C}\right)},\hfill \end{array}$$

(65)

$$\begin{array}{cc}\hfill & {\chi }^2\left({\mu }_{\mathcal{C}}\parallel \mu \right)=\frac{1}{\mu \left(\mathcal{C}\right)}-1,\hfill \end{array}$$

(66)

so (16) is satisfied in this case with equality. More generally, for all $\alpha \in (0,\infty )$,

$$\begin{array}{c}\hfill D_{\alpha }\left({\mu }_{\mathcal{C}}\parallel \mu \right)=log\frac{1}{\mu \left(\mathcal{C}\right)}.\end{array}$$

(67)

Proof. 

See Section 5.7. □

Remark 8.

In spite of its simplicity, (65) proved very useful in the seminal work by Marton on transportation–cost inequalities, proving concentration of measures by information-theoretic tools [33,34] (see also [35] [Chapter 8] and [36] [Chapter 3]). As a side note, the simple identity (65) was apparently first explicitly used by Csiszár (see [37] [Equation (4.13)]).

4. Applications

This section provides applications of our results in Section 3. These include universal lossless compression, method of types and large deviations, and strong data–processing inequalities (SDPIs).

4.1. Application of Corollary 3: Shannon Code for Universal Lossless Compression

Consider $m>1$ discrete, memoryless, and stationary sources with probability mass functions ${\left\{P_i\right\}}_{i=1}^m$, and assume that the symbols are emitted by one of these sources with an a priori probability ${\alpha }_i$ for source no. i, where ${\left\{{\alpha }_i\right\}}_{i=1}^m$ are positive and sum to 1.

For lossless data compression by a universal source code, suppose that a single source code is designed with respect to the average probability mass function $P:={\displaystyle \sum _{j=1}^m}{\alpha }_j{P}_j$.

Assume that the designer uses a Shannon code, where the code assignment for a symbol $x\in \mathcal{X}$ is of length $\ell \left(x\right)=\lceil log\frac{1}{P\left(x\right)}\rceil$ bits (logarithms are on base 2). Due to the mismatch in the source distribution, the average codeword length ${\ell }_{\mathrm{avg}}$ satisfies (see [38] [Proposition 3.B])

$$\begin{array}{c}\hfill \sum _{i=1}^m{\alpha }_{i}H\left(P_i\right)+\sum _{i=1}^m{\alpha }_{i}D(P_i\parallel P)\le {\ell }_{\mathrm{avg}}\le \sum _{i=1}^m{\alpha }_{i}H\left(P_i\right)+\sum _{i=1}^m{\alpha }_{i}D(P_i\parallel P)+1.\end{array}$$

(68)

The fractional penalty in the average codeword length, denoted by $\nu$, is defined to be equal to the ratio of the penalty in the average codeword length as a result of the source mismatch, and the average codeword length in case of a perfect matching. From (68), it follows that

$$\begin{array}{c}\hfill \frac{{\displaystyle \sum _{j=1}^m}{\alpha }_{i}D(P_i\parallel P)}{1+{\displaystyle \sum _{j=1}^m}{\alpha }_{i}H\left(P_i\right)}\le \nu \le \frac{1+{\displaystyle \sum _{j=1}^m}{\alpha }_{i}D(P_i\parallel P)}{{\displaystyle \sum _{j=1}^m}{\alpha }_{i}H\left(P_i\right)}.\end{array}$$

(69)

We next rely on Corollary 3 to obtain an upper bound on $\nu$ which is expressed as a function of the $m(m-1)$ relative entropies $D(P_i\parallel P_j)$ for all $i\ne j$ in $\{1,\dots ,m\}$. This is useful if, e.g., the m relative entropies on the left and right sides of (69) do not admit closed form expressions, in contrast to the $m(m-1)$ relative entropies $D(P_i\parallel P_j)$ for $i\ne j$. We next exemplify this case.

For $i\in \{1,\dots ,m\}$, let $P_i$ be a Poisson distribution with parameter ${\lambda }_i>0$. For all $i,j\in \{1,\dots ,m\}$, the relative entropy from $P_i$ to $P_j$ admits the closed-form expression

$$\begin{array}{cc}\hfill & D(P_i\parallel P_j)={\lambda }_{i}log\left(\frac{{\lambda }_i}{{\lambda }_j}\right)+({\lambda }_j-{\lambda }_i)log\mathrm{e}.\hfill \end{array}$$

(70)

From (54) and (70), it follows that

$$\begin{array}{cc}\hfill D(P_i\parallel P)& \le -log\left({\alpha }_i+(1-{\alpha }_i)exp\left(-\frac{f_i(\underline{\alpha },\underline{\lambda })}{1-{\alpha }_i}\right)\right),\hfill \end{array}$$

(71)

where

$$\begin{array}{}\hfill \mathrm{(72)}& \hfill f_i(\underline{\alpha },\underline{\lambda })& :=\sum _{j\ne i}{\alpha }_{j}D(P_i\parallel P_j)\hfill \hfill \mathrm{(73)}& & =\sum _{j\ne i}\left\{{\alpha }_j\left[{\lambda }_{i}log\left(\frac{{\lambda }_i}{{\lambda }_j}\right)+({\lambda }_j-{\lambda }_i)log\mathrm{e}\right]\right\}.\hfill \end{array}$$

The entropy of a Poisson distribution, with parameter ${\lambda }_i$, is given by the integral representation [39,40,41]

$$\begin{array}{c}\hfill H\left(P_i\right)={\lambda }_{i}log\left(\frac{\mathrm{e}}{{\lambda }_i}\right)+{\int }_0^{\infty }\left({\lambda }_i-\frac{1-{\mathrm{e}}^{-{\lambda }_i(1-{\mathrm{e}}^{-u})}}{1-{\mathrm{e}}^{-u}}\right)\frac{{\mathrm{e}}^{-u}}{u}\mathrm{d}ulog\mathrm{e}.\end{array}$$

(74)

Combining (69), (71) and (74) finally gives an upper bound on $\nu$ in the considered setup.

Example 1.

Consider five discrete memoryless sources where the probability mass function of source no. i is given by $P_i=\mathrm{Poisson}\left({\lambda }_i\right)$ with $\underline{\lambda }=[16,20,24,28,32]$. Suppose that the symbols are emitted from one of the sources with equal probability, so $\underline{\alpha }=\left[\frac{1}{5},\frac{1}{5},\frac{1}{5},\frac{1}{5},\frac{1}{5}\right]$. Let $P:=\frac{1}{5}(P_1+\dots +P_5)$ be the average probability mass function of the five sources. The term ${\sum }_i{\alpha }_{i}D(P_i\parallel P)$, which appears in the numerators of the upper and lower bounds on ν (see (69)), does not lend itself to a closed-form expression, and it is not even an easy task to calculate it numerically due to the need to compute an infinite series which involves factorials. We therefore apply the closed-form upper bound in (71) to get that ${\sum }_i{\alpha }_{i}D(P_i\parallel P)\le 1.46$ bits, whereas the upper bound which follows from the convexity of the relative entropy (i.e., ${\sum }_i{\alpha }_i{f}_i(\underline{\alpha },\underline{\lambda })$) is equal to 1.99 bits (both upper bounds are smaller than the trivial bound ${log}_{2}5\approx 2.32$ bits). From (69), (74), and the stronger upper bound on ${\sum }_i{\alpha }_{i}D(P_i\parallel P)$, the improved upper bound on ν is equal to $57.0\%$ (as compared to a looser upper bound of $69.3\%$, which follows from (69), (74), and the looser upper bound on ${\sum }_i{\alpha }_{i}D(P_i\parallel P)$ that is equal to 1.99 bits).

4.2. Application of Theorem 2 in the Context of the Method of Types and Large Deviations Theory

Let $X^n=(X_1,\dots ,X_n)$ be a sequence of i.i.d. random variables with $X_1\sim Q$, where Q is a probability measure defined on a finite set $\mathcal{X}$, and $Q\left(x\right)>0$ for all $x\in \mathcal{X}$. Let $\mathcal{P}$ be a set of probability measures on $\mathcal{X}$ such that $Q\notin \mathcal{P}$, and suppose that the closure of $\mathcal{P}$ coincides with the closure of its interior. Then, by Sanov’s theorem (see, e.g., [42] [Theorem 11.4.1] and [43] [Theorem 3.3]), the probability that the empirical distribution ${\widehat{P}}_{X^n}$ belongs to $\mathcal{P}$ vanishes exponentially at the rate

$$\begin{array}{c}\hfill \underset{n\to \infty }{lim}\frac{1}{n}log\frac{1}{\mathbb{P}[{\widehat{P}}_{X^n}\in \mathcal{P}]}=\underset{P\in \mathcal{P}}{inf}D(P\parallel Q).\end{array}$$

(75)

Furthermore, for finite n, the method of types yields the following upper bound on this rare event:

$$\begin{array}{}\hfill \mathrm{(76)}& \mathbb{P}[{\widehat{P}}_{X^n}\in \mathcal{P}]& \le \left(\genfrac{}{}{0pt}{}{n+\left|\mathcal{X}\right|-1}{\left|\mathcal{X}\right|-1}\right)exp\left(-n\underset{P\in \mathcal{P}}{inf}D(P\parallel Q)\right)\hfill \hfill \mathrm{(77)}& & \le {(n+1)}^{\left|\mathcal{X}\right|-1}exp\left(-n\underset{P\in \mathcal{P}}{inf}D(P\parallel Q)\right),\hfill \end{array}$$

whose exponential decay rate coincides with the exact asymptotic result in (75).

Suppose that Q is not fully known, but its mean $m_Q$ and variance ${\sigma }_Q^2$ are available. Let $m_1\in \mathbb{R}$ and ${\delta }_1,{\epsilon }_1,{\sigma }_1>0$ be fixed, and let $\mathcal{P}$ be the set of all probability measures P, defined on the finite set $\mathcal{X}$, with mean $m_P\in [m_1-{\delta }_1,m_1+{\delta }_1]$ and variance ${\sigma }_P^2\in [{\sigma }_1^2-{\epsilon }_1,{\sigma }_1^2+{\epsilon }_1]$, where $|m_1-m_Q|>{\delta }_1$. Hence, $\mathcal{P}$ coincides with the closure of its interior, and $Q\notin \mathcal{P}$.

The lower bound on the relative entropy in Theorem 2, used in conjunction with the upper bound in (77), can serve to obtain an upper bound on the probability of the event that the empirical distribution of $X^n$ belongs to the set $\mathcal{P}$, regardless of the uncertainty in Q. This gives

$$\begin{array}{cc}\hfill \mathbb{P}[{\widehat{P}}_{X^n}\in \mathcal{P}]& \le {(n+1)}^{\left|\mathcal{X}\right|-1}exp\left(-n{d}^{*}\right),\hfill \end{array}$$

(78)

where

$$\begin{array}{c}\hfill d^{\ast }:=\underset{m_P,{\sigma }_P^2}{inf}d(r\parallel s),\end{array}$$

(79)

and, for fixed $(m_P,m_Q,{\sigma }_P^2,{\sigma }_Q^2)$, the parameters r and s are given in (41) and (42), respectively.

Standard algebraic manipulations that rely on (78) lead to the following result, which is expressed as a function of the Lambert–W function [44]. This function, which finds applications in various engineering and scientific fields, is a standard built–in function in mathematical software tools such as Mathematica, Matlab, and Maple. Applications of the Lambert–W function in information theory and coding are briefly surveyed in [45].

Proposition 2.

For $\epsilon \in (0,1)$, let $n^{\ast }:=n^{\ast }\left(\epsilon \right)$ denote the minimal value of $n\in \mathbb{N}$ such that the upper bound on the right side of (78) does not exceed $\epsilon \in (0,1)$. Then, $n^{\ast }$ admits the following closed-form expression:

$$\begin{array}{c}\hfill n^{\ast }=max\left\{⌈-\frac{\left(\left|\mathcal{X}\right|-1\right)W_{-1}\left(\eta \right)log\mathrm{e}}{d^{\ast }}⌉-1,1\right\},\end{array}$$

(80)

with

$$\begin{array}{cc}\hfill & \eta :=-\frac{d^{\ast }{\left(\epsilon exp(-d^{\ast })\right)}^{1/\left(\right|\mathcal{X}|-1)}}{\left(\left|\mathcal{X}\right|-1\right)log\mathrm{e}}\in [-\frac{1}{\mathrm{e}},0),\hfill \end{array}$$

(81)

and $W_{-1}(·)$ on the right side of (80) denotes the secondary real–valued branch of the Lambert–W function (i.e., $x:=W_{-1}\left(y\right)$ where $W_{-1}:[-\frac{1}{\mathrm{e}},0)\to (-\infty ,-1]$ is the inverse function of $y:=x{\mathrm{e}}^x$).

Example 2.

Let Q be an arbitrary probability measure, defined on a finite set $\mathcal{X}$, with mean $m_Q=40$ and variance ${\sigma }_Q^2=20$. Let $\mathcal{P}$ be the set of all probability measures P, defined on $\mathcal{X}$, whose mean $m_P$ and variance ${\sigma }_P^2$ lie in the intervals $[43,47]$ and $[18,22]$, respectively. Suppose that it is required that, for all probability measures Q as above, the probability that the empirical distribution of the i.i.d. sequence $X^n\sim Q^n$ that is included in the set $\mathcal{P}$ is at most $\epsilon ={10}^{-10}$. We rely here on the upper bound in (78), and impose the stronger condition where it should not exceed ε. By this approach, it is obtained numerically from (79) that $d^{\ast }=0.203$ nats. We next examine two cases:

(i)

If $\left|\mathcal{X}\right|=2$, then it follows from (80) that $n^{\ast }=138$.

(ii)

Consider a richer alphabet size of the i.i.d. samples where, e.g., $\left|\mathcal{X}\right|=100$. By relying on the same universal lower bound $d^{\ast }$, which holds independently of the value of $\left|\mathcal{X}\right|$ ($\mathcal{X}$ can possibly be an infinite set), it follows from (80) that $n^{\ast }=4170$ is the minimal value such that the upper bound in (78) does not exceed ${10}^{-10}$.

We close this discussion by providing numerical experimentation of the lower bound on the relative entropy in Theorem 2, and comparing this attainable lower bound (see Item b) of Theorem 2) with the following closed-form expressions for relative entropies:

(a)

The relative entropy between real-valued Gaussian distributions is given by

$$\begin{array}{c}\hfill D\left(\mathcal{N}(m_P,{\sigma }_P^2)\parallel \mathcal{N}(m_Q,{\sigma }_Q^2)\right)=log\frac{{\sigma }_Q}{{\sigma }_P}+\frac{1}{2}\left[\frac{{(m_P-m_Q)}^2+{\sigma }_P^2}{{\sigma }_Q^2}-1\right]log\mathrm{e}.\end{array}$$

(82)

(b)

Let $E_{\mu }$ denote a random variable which is exponentially distributed with mean $\mu >0$; its probability density function is given by

$$\begin{array}{c}\hfill e_{\mu }\left(x\right)=\frac{1}{\mu }{\mathrm{e}}^{-x/\mu }1\{x\ge 0\}.\end{array}$$

(83)

Then, for $a_1,a_2>0$ and $d_1,d_2\in \mathbb{R}$,

$$D(E_{a_1}+d_1\parallel E_{a_2}+d_2)=\{\begin{array}{ll}log\frac{a_2}{a_1}+\frac{d_1+a_1-d_2-a_2}{a_2}log\mathrm{e},& d_1\ge d_2,\hfill \\ \infty ,& d_1<d_2.\end{array}$$

(84)

In this case, the means under P and Q are $m_P=d_1+a_1$ and $m_Q=d_2+a_2$, respectively, and the variances are ${\sigma }_P^2=a_1^2$ and ${\sigma }_Q^2=a_2^2$. Hence, for obtaining the required means and variances, set

$$\begin{array}{c}\hfill a_1={\sigma }_P,a_2={\sigma }_Q,d_1=m_P-{\sigma }_P,d_2=m_Q-{\sigma }_Q.\end{array}$$

(85)

Example 3.

We compare numerically the attainable lower bound on the relative entropy, as it given in (40), with the two relative entropies in (82) and (84):

(i)

If $(m_P,m_Q,{\sigma }_P^2,{\sigma }_Q^2)=(45,40,20,20)$, then the lower bound in (40) is equal to 0.521 nats, and the two relative entropies in (82) and (84) are equal to 0.625 and 1.118 nats, respectively.

(ii)

If $(m_P,m_Q,{\sigma }_P^2,{\sigma }_Q^2)=(50,35,10,20)$, then the lower bound in (40) is equal to 2.332 nats, and the two relative entropies in (82) and (84) are equal to 5.722 and 3.701 nats, respectively.

4.3. Strong Data–Processing Inequalities and Maximal Correlation

The information contraction is a fundamental concept in information theory. The contraction of f-divergences through channels is captured by data–processing inequalities, which can be further tightened by the derivation of SDPIs with channel-dependent or source-channel dependent contraction coefficients (see, e.g., [26,46,47,48,49,50,51,52]).

We next provide necessary definitions which are relevant for the presentation in this subsection.

Definition 7.

Let $Q_X$ be a probability distribution which is defined on a set $\mathcal{X}$, and that is not a point mass, and let $W_{Y|X}:\mathcal{X}\to \mathcal{Y}$ be a stochastic transformation. The contraction coefficient for f-divergences is defined as

$$\begin{array}{c}\hfill {\mu }_f(Q_X,W_{Y|X}):=\underset{P_X:D_f(P_X\parallel Q_X)\in (0,\infty )}{sup}\frac{D_f(P_Y\parallel Q_Y)}{D_f(P_X\parallel Q_X)},\end{array}$$

(86)

where, for all $y\in \mathcal{Y}$,

$$\begin{array}{cc}\hfill & P_Y\left(y\right)=\left(P_X{W}_{Y|X}\right)\left(y\right):={\int }_{\mathcal{X}}\mathrm{d}{P}_X\left(x\right)W_{Y|X}\left(y\right|x),\hfill \end{array}$$

(87)

$$\begin{array}{cc}\hfill & Q_Y\left(y\right)=\left(Q_X{W}_{Y|X}\right)\left(y\right):={\int }_{\mathcal{X}}\mathrm{d}{Q}_X\left(x\right)W_{Y|X}\left(y\right|x).\hfill \end{array}$$

(88)

The notation in (87) and (88) is consistent with the standard notation used in information theory (see, e.g., the first displayed equation after (3.2) in [53]).

The derivation of good upper bounds on contraction coefficients for f-divergences, which are strictly smaller than 1, lead to SDPIs. These inequalities find their applications, e.g., in studying the exponential convergence rate of an irreducible, time-homogeneous and reversible discrete-time Markov chain to its unique invariant distribution over its state space (see, e.g., [49] [Section 2.4.3] and [50] [Section 2]). It is in sharp contrast to DPIs which do not yield convergence to stationarity at any rate. We return to this point later in this subsection, and determine the exact convergence rate to stationarity under two parametric families of f-divergences.

We next rely on Theorem 1 to obtain upper bounds on the contraction coefficients for the following f-divergences.

Definition 8.

For $\alpha \in (0,1]$, the α-skew K-divergence is given by

$$\begin{array}{c}\hfill K_{\alpha }(P\parallel Q):=D\left(P\parallel (1-\alpha )P+\alpha Q\right),\end{array}$$

(89)

and, for $\alpha \in [0,1]$, let

$$\begin{array}{}\hfill \mathrm{(90)}& \hfill S_{\alpha }(P\parallel Q)& :=\alpha D\left(P\parallel (1-\alpha )P+\alpha Q\right)+(1-\alpha )D\left(Q\parallel (1-\alpha )P+\alpha Q\right)\hfill \hfill \mathrm{(91)}& & =\alpha K_{\alpha }(P\parallel Q)+(1-\alpha )K_{1-\alpha }(Q\parallel P),\hfill \end{array}$$

with the convention that $K_0(P\parallel Q)\equiv 0$ (by a continuous extension at $\alpha =0$ in (89)). These divergence measures are specialized to the relative entropies:

$$\begin{array}{cc}\hfill K_1(P\parallel Q)=D(P\parallel Q)=S_1(P\parallel Q),& S_0(P\parallel Q)=D(Q\parallel P),\hfill \end{array}$$

(92)

and $S_{\frac{1}{2}}(P\parallel Q)$ is the Jensen–Shannon divergence [54,55,56] (also known as the capacitory discrimination [57]):

$$\begin{array}{}\hfill \mathrm{(93)}& \hfill S_{\frac{1}{2}}(P\parallel Q)& =\frac{1}{2}D\left(P\parallel \frac{1}{2}(P+Q)\right)+\frac{1}{2}D\left(Q\parallel \frac{1}{2}(P+Q)\right)\hfill \hfill \mathrm{(94)}& & =H\left(\frac{1}{2}(P+Q)\right)-\frac{1}{2}H\left(P\right)-\frac{1}{2}H\left(Q\right):=\mathrm{JS}(P\parallel Q).\hfill \end{array}$$

It can be verified that the divergence measures in (89) and (90) are f-divergences:

$$\begin{array}{cc}\hfill & K_{\alpha }(P\parallel Q)=D_{k_{\alpha }}(P\parallel Q),\alpha \in (0,1],\hfill \end{array}$$

(95)

$$\begin{array}{cc}\hfill & S_{\alpha }(P\parallel Q)=D_{s_{\alpha }}(P\parallel Q),\alpha \in [0,1],\hfill \end{array}$$

(96)

with

$$\begin{array}{}\hfill \mathrm{(97)}& \hfill k_{\alpha }\left(t\right)& :=tlogt-tlog\left(\alpha +(1-\alpha )t\right),t>0,\alpha \in (0,1],\hfill \hfill \mathrm{(98)}& \hfill s_{\alpha }\left(t\right)& :=\alpha tlogt-\left(\alpha t+1-\alpha \right)log\left(\alpha +(1-\alpha )t\right)\hfill \hfill \mathrm{(99)}& & =\alpha k_{\alpha }\left(t\right)+(1-\alpha )t{k}_{1-\alpha }\left(\frac{1}{t}\right),t>0,\alpha \in [0,1],\hfill \end{array}$$

where $k_{\alpha }(·)$ and $s_{\alpha }(·)$ are strictly convex functions on $(0,\infty )$, and vanish at 1.

Remark 9.

The α-skew K-divergence in (89) is considered in [55] and [58] [(13)] (including pointers in the latter paper to its utility). The divergence in (90) is akin to Lin’s measure in [55] [(4.1)], the asymmetric α-skew Jensen–Shannon divergence in [58] [(11)–(12)], the symmetric α-skew Jensen–Shannon divergence in [58] [(16)], and divergence measures in [59] which involve arithmetic and geometric means of two probability distributions. Properties and applications of quantum skew divergences are studied in [19] and references therein.

Theorem 6.

The f-divergences in (89) and (90) satisfy the following integral identities, which are expressed in terms of the Györfi–Vajda divergence in (17):

$$\begin{array}{cc}\hfill & \frac{1}{log\mathrm{e}}{K}_{\alpha }(P\parallel Q)={\int }_0^{\alpha }s{D}_{{\varphi }_s}(P\parallel Q)\mathrm{d}s,\alpha \in (0,1],\hfill \end{array}$$

(100)

$$\begin{array}{cc}\hfill & \frac{1}{log\mathrm{e}}{S}_{\alpha }(P\parallel Q)={\int }_0^1{g}_{\alpha }\left(s\right)D_{{\varphi }_s}(P\parallel Q)\mathrm{d}s,\alpha \in [0,1],\hfill \end{array}$$

(101)

with

$$\begin{array}{c}\hfill g_{\alpha }\left(s\right):=\alpha s1\left\{s\in (0,\alpha ]\right\}+(1-\alpha )(1-s)1\left\{s\in [\alpha ,1)\right\},(\alpha ,s)\in [0,1]\times [0,1].\end{array}$$

(102)

Moreover, the contraction coefficients for these f-divergences are related as follows:

$$\begin{array}{cc}\hfill {\mu }_{{\chi }^2}(Q_X,W_{Y|X})& \le {\mu }_{k_{\alpha }}(Q_X,W_{Y|X})\le \underset{s\in (0,\alpha ]}{sup}{\mu }_{{\varphi }_s}(Q_X,W_{Y|X}),\alpha \in (0,1],\hfill \end{array}$$

(103)

$$\begin{array}{cc}\hfill {\mu }_{{\chi }^2}(Q_X,W_{Y|X})& \le {\mu }_{s_{\alpha }}(Q_X,W_{Y|X})\le \underset{s\in (0,1)}{sup}{\mu }_{{\varphi }_s}(Q_X,W_{Y|X}),\alpha \in [0,1],\hfill \end{array}$$

(104)

where ${\mu }_{{\chi }^2}(Q_X,W_{Y|X})$ denotes the contraction coefficient for the chi-squared divergence.

Proof. 

See Section 5.8. □

Remark 10.

The upper bounds on the contraction coefficients for the parametric f-divergences in (89) and (90) generalize the upper bound on the contraction coefficient for the relative entropy in [51] [Theorem III.6] (recall that $K_1(P\parallel Q)=D(P\parallel Q)=S_1(P\parallel Q)$), so the upper bounds in Theorem 6 are specialized to the latter bound at $\alpha =1$.

Corollary 7.

Let

$$\begin{array}{c}\hfill {\mu }_{{\chi }^2}\left(W_{Y|X}\right):=\underset{Q}{sup}{\mu }_{{\chi }^2}(Q_X,W_{Y|X}),\end{array}$$

(105)

where the supremum on the right side is over all probability measures $Q_X$ defined on $\mathcal{X}$. Then,

$$\begin{array}{cc}\hfill {\mu }_{{\chi }^2}(Q_X,W_{Y|X})& \le {\mu }_{k_{\alpha }}(Q_X,W_{Y|X})\le {\mu }_{{\chi }^2}\left(W_{Y|X}\right),\alpha \in (0,1],\hfill \end{array}$$

(106)

$$\begin{array}{cc}\hfill {\mu }_{{\chi }^2}(Q_X,W_{Y|X})& \le {\mu }_{s_{\alpha }}(Q_X,W_{Y|X})\le {\mu }_{{\chi }^2}\left(W_{Y|X}\right),\alpha \in [0,1].\hfill \end{array}$$

(107)

Proof. 

See Section 5.9. □

Example 4.

Let $Q_X=\mathrm{Bernoulli}\left(\frac{1}{2}\right)$, and let $W_{Y|X}$ correspond to a binary symmetric channel (BSC) with crossover probability ε. Then, ${\mu }_{{\chi }^2}(Q_X,W_{Y|X})={\mu }_{{\chi }^2}\left(W_{Y|X}\right)={(1-2\epsilon )}^2$. The upper and lower bounds on ${\mu }_{k_{\alpha }}(Q_X,W_{Y|X})$ and ${\mu }_{s_{\alpha }}(Q_X,W_{Y|X})$ in (106) and (107) match for all α, and they are all equal to ${(1-2\epsilon )}^2$.

The upper bound on the contraction coefficients in Corollary 7 is given by ${\mu }_{{\chi }^2}\left(W_{Y|X}\right)$, whereas the lower bound is given by ${\mu }_{{\chi }^2}(Q_X,W_{Y|X})$, which depends on the input distribution $Q_X$. We next provide alternative upper bounds on the contraction coefficients for the considered (parametric) f-divergences, which, similarly to the lower bound, scale like ${\mu }_{{\chi }^2}(Q_X,W_{Y|X})$. Although the upper bound in Corollary 7 may be tighter in some cases than the alternative upper bounds which are next presented in Proposition 3 (and in fact, the former upper bound may be even achieved with equality as in Example 4), the bounds in Proposition 3 are used shortly to determine the exponential rate of the convergence to stationarity of a type of Markov chains.

Proposition 3.

For all $\alpha \in (0,1]$,

$$\begin{array}{}\hfill \mathrm{(108)}& {\mu }_{{\chi }^2}(Q_X,W_{Y|X})\le {\mu }_{k_{\alpha }}(Q_X,W_{Y|X})\le \frac{1}{\alpha Q_{min}}·{\mu }_{{\chi }^2}(Q_X,W_{Y|X}),\hfill \hfill \mathrm{(109)}& {\mu }_{{\chi }^2}(Q_X,W_{Y|X})\le {\mu }_{s_{\alpha }}(Q_X,W_{Y|X})\le \frac{(1-\alpha ){log}_{\mathrm{e}}\left(\frac{1}{\alpha }\right)+2\alpha -1}{(1-3\alpha +3{\alpha }^2)Q_{min}}·{\mu }_{{\chi }^2}(Q_X,W_{Y|X}),\hfill \end{array}$$

where $Q_{min}$ denotes the minimal positive mass of the input distribution $Q_X$.

Proof. 

See Section 5.10. □

Remark 11.

In view of (92), at $\alpha =1$, (108) and (109) specialize to an upper bound on the contraction coefficient of the relative entropy (KL divergence) as a function of the contraction coefficient of the chi-squared divergence. In this special case, both (108) and (109) give

$$\begin{array}{c}\hfill {\mu }_{{\chi }^2}(Q_X,W_{Y|X})\le {\mu }_{\mathrm{KL}}(Q_X,W_{Y|X})\le \frac{1}{Q_{min}}·{\mu }_{{\chi }^2}(Q_X,W_{Y|X}),\end{array}$$

(110)

which then coincides with [48] [Theorem 10].

We next apply Proposition 3 to consider the convergence rate to stationarity of Markov chains by the introduced f-divergences in Definition 8. The next result follows [49] [Section 2.4.3], and it provides a generalization of the result there.

Theorem 7.

Consider a time-homogeneous, irreducible, and reversible discrete-time Markov chain with a finite state space $\mathcal{X}$, let W be its probability transition matrix, and $Q_X$ be its unique stationary distribution (reversibility means that $Q_X\left(x\right){\left[W\right]}_{x,y}=Q_X\left(y\right){\left[W\right]}_{y,x}$ for all $x,y\in \mathcal{X}$). Let $P_X$ be an initial probability distribution over $\mathcal{X}$. Then, for all $\alpha \in (0,1]$ and $n\in \mathbb{N}$,

$$\begin{array}{}\hfill \mathrm{(111)}& K_{\alpha }(P_X{W}^n\parallel Q_X)\le {\mu }_{k_{\alpha }}(Q_X,W^n)K_{\alpha }(P_X\parallel Q_X),\hfill \hfill \mathrm{(112)}& S_{\alpha }(P_X{W}^n\parallel Q_X)\le {\mu }_{s_{\alpha }}(Q_X,W^n)S_{\alpha }(P_X\parallel Q_X),\hfill \end{array}$$

and the contraction coefficients on the right sides of (111) and (112) scale like the n-th power of the contraction coefficient for the chi-squared divergence as follows:

$$\begin{array}{}\hfill \mathrm{(113)}& {\left({\mu }_{{\chi }^2}(Q_X,W)\right)}^n\le {\mu }_{k_{\alpha }}(Q_X,W^n)\le \frac{1}{\alpha Q_{min}}·{\left({\mu }_{{\chi }^2}(Q_X,W)\right)}^n,\hfill \hfill \mathrm{(114)}& {\left({\mu }_{{\chi }^2}(Q_X,W)\right)}^n\le {\mu }_{s_{\alpha }}(Q_X,W^n)\le \frac{(1-\alpha ){log}_{\mathrm{e}}\left(\frac{1}{\alpha }\right)+2\alpha -1}{(1-3\alpha +3{\alpha }^2)Q_{min}}·{\left({\mu }_{{\chi }^2}(Q_X,W)\right)}^n.\hfill \end{array}$$

Proof. 

Inequalities (111) and (112) hold since $Q_X{W}^n=Q_X$, for all $n\in \mathbb{N}$, and due to Definition 7 and (95) and (96). Inequalities (113) and (114) hold by Proposition 3, and due to the reversibility of the Markov chain which implies that (see [49] [Equation (2.92)])

$$\begin{array}{c}\hfill {\mu }_{{\chi }^2}(Q_X,W^n)={\left({\mu }_{{\chi }^2}(Q_X,W)\right)}^n,n\in \mathbb{N}.\end{array}$$

(115)

 □

In view of (113) and (114), Theorem 7 readily gives the following result on the exponential decay rate of the upper bounds on the divergences on the left sides of (111) and (112).

Corollary 8.

For all $\alpha \in (0,1]$,

$$\begin{array}{c}\hfill \underset{n\to \infty }{lim}{\left({\mu }_{k_{\alpha }}(Q_X,W^n)\right)}^{1/n}={\mu }_{{\chi }^2}(Q_X,W)=\underset{n\to \infty }{lim}{\left({\mu }_{s_{\alpha }}(Q_X,W^n)\right)}^{1/n}.\end{array}$$

(116)

Remark 12.

Theorem 7 and Corollary 8 generalize the results in [49] [Section 2.4.3], which follow as a special case at $\alpha =1$ (see (92)).

We end this subsection by considering maximal correlations, which are closely related to the contraction coefficient for the chi-squared divergence.

Definition 9.

The maximal correlation between two random variables X and Y is defined as

$$\begin{array}{c}\hfill {\rho }_{\mathrm{m}}(X;Y):=\underset{f,g}{sup}\mathbb{E}\left[f\left(X\right)g\left(Y\right)\right],\end{array}$$

(117)

where the supremum is taken over all real-valued functions f and g such that

$$\begin{array}{c}\hfill \mathbb{E}\left[f\left(X\right)\right]=\mathbb{E}\left[g\left(Y\right)\right]=0,\mathbb{E}\left[f^2\left(X\right)\right]\le 1,\mathbb{E}\left[g^2\left(Y\right)\right]\le 1.\end{array}$$

(118)

It is well-known [60] that, if $X\sim Q_X$ and $Y\sim Q_Y=Q_X{W}_{Y|X}$, then the contraction coefficient for the chi-squared divergence ${\mu }_{{\chi }^2}(Q_X,W_{Y|X})$ is equal to the square of the maximal correlation between the random variables X and Y, i.e.,

$$\begin{array}{c}\hfill {\rho }_{\mathrm{m}}(X;Y)=\sqrt{{\mu }_{{\chi }^2}(Q_X,W_{Y|X})}.\end{array}$$

(119)

A simple application of Corollary 1 and (119) gives the following result.

Proposition 4.

In the setting of Definition 7, for $s\in [0,1]$, let $X_s\sim (1-s)P_X+s{Q}_X$ and $Y_s\sim (1-s)P_Y+s{Q}_Y$ with $P_X\ne Q_X$ and $P_X\ll \gg Q_X$. Then, the following inequality holds:

$$\begin{array}{c}\hfill \underset{s\in [0,1]}{sup}{\rho }_{\mathrm{m}}(X_s;Y_s)\ge max\left\{\sqrt{\frac{D(P_Y\parallel Q_Y)}{D(P_X\parallel Q_X)}},\sqrt{\frac{D(Q_Y\parallel P_Y)}{D(Q_X\parallel P_X)}}\right\}.\end{array}$$

(120)

Proof. 

See Section 5.11. □

5. Proofs

This section provides proofs of the results in Section 3 and Section 4.

5.1. Proof of Theorem 1

Proof of (22): We rely on an integral representation of the logarithm function (on base $\mathrm{e}$):

$$\begin{array}{cc}\hfill {log}_{\mathrm{e}}x& ={\int }_0^1\frac{x-1}{x+(1-x)v}\mathrm{d}v,\forall x>0.\hfill \end{array}$$

(121)

Let $\mu$ be a dominating measure of P and Q (i.e., $P,Q\ll \mu$), and let $p:=\frac{\mathrm{d}P}{\mathrm{d}\mu }$, $q:=\frac{\mathrm{d}Q}{\mathrm{d}\mu }$, and

$$\begin{array}{c}\hfill r_{\lambda }:=\frac{\mathrm{d}{R}_{\lambda }}{\mathrm{d}\mu }=(1-\lambda )p+\lambda q,\forall \lambda \in [0,1],\end{array}$$

(122)

where the last equality is due to (21). For all $\lambda \in [0,1]$,

$$\begin{array}{}\hfill \mathrm{(123)}& \hfill \frac{1}{log\mathrm{e}}D(P\parallel R_{\lambda })& =\int p{log}_{\mathrm{e}}\left(\frac{p}{r_{\lambda }}\right)\mathrm{d}\mu \hfill \hfill \mathrm{(124)}& & ={\int }_0^1\int \frac{p(p-r_{\lambda })}{p+v(r_{\lambda }-p)}\mathrm{d}\mu \mathrm{d}v,\hfill \end{array}$$

where (124) holds due to (121) with $x:=\frac{p}{r_{\lambda }}$, and by swapping the order of integration. The inner integral on the right side of (124) satisfies, for all $v\in (0,1]$,

$$\begin{array}{}\hfill \mathrm{(125)}& \hfill \int \frac{p(p-r_{\lambda })}{p+v(r_{\lambda }-p)}\mathrm{d}\mu & =\int (p-r_{\lambda })\left(1+\frac{v(p-r_{\lambda })}{p+v(r_{\lambda }-p)}\right)\mathrm{d}\mu \hfill \hfill \mathrm{(126)}& & =\int (p-r_{\lambda })\mathrm{d}\mu +v\int \frac{{(p-r_{\lambda })}^2}{p+v(r_{\lambda }-p)}\mathrm{d}\mu \hfill \hfill \mathrm{(127)}& & =v\int \frac{{(p-r_{\lambda })}^2}{(1-v)p+v{r}_{\lambda }}\mathrm{d}\mu \hfill \hfill \mathrm{(128)}& & =\frac{1}{v}\int \frac{{\left(p-\left[(1-v)p+v{r}_{\lambda }\right]\right)}^2}{(1-v)p+v{r}_{\lambda }}\mathrm{d}\mu \hfill \hfill \mathrm{(129)}& & =\frac{1}{v}{\chi }^2\left(P\parallel (1-v)P+v{R}_{\lambda }\right),\hfill \end{array}$$

where (127) holds since $\int p\mathrm{d}\mu =1$, and $\int r_{\lambda }\mathrm{d}\mu =1$. From (21), for all $(\lambda ,v)\in [0,1]\times [0,1]$,

$$\begin{array}{c}\hfill (1-v)P+v{R}_{\lambda }=(1-\lambda v)P+\lambda vQ=R_{\lambda v}.\end{array}$$

(130)

The substitution of (130) into the right side of (129) gives that, for all $(\lambda ,v)\in [0,1]\times (0,1]$,

$$\begin{array}{c}\hfill \int \frac{p(p-r_{\lambda })}{p+v(r_{\lambda }-p)}\mathrm{d}\mu =\frac{1}{v}{\chi }^2(P\parallel R_{\lambda v}).\end{array}$$

(131)

Finally, substituting (131) into the right side of (124) gives that, for all $\lambda \in (0,1]$,

$$\begin{array}{}\hfill \mathrm{(132)}& \hfill \frac{1}{log\mathrm{e}}D(P\parallel R_{\lambda })& ={\int }_0^1\frac{1}{v}{\chi }^2(P\parallel R_{\lambda v})\mathrm{d}v\hfill \hfill \mathrm{(133)}& & ={\int }_0^{\lambda }\frac{1}{s}{\chi }^2(P\parallel R_s)\mathrm{d}s,\hfill \end{array}$$

where (133) holds by the transformation $s:=\lambda v$. Equality (133) also holds for $\lambda =0$ since we have $D(P\parallel R_0)=D(P\parallel P)=0$.

Proof of (23): For all $s\in (0,1]$,

$$\begin{array}{}\hfill & {\chi }^2(P\parallel Q)& =\int \frac{{(p-q)}^2}{q}\mathrm{d}\mu \hfill \hfill \mathrm{(134)}& & =\frac{1}{s^2}\int \frac{{\left[\left(sp+(1-s)q\right)-q\right]}^2}{q}\mathrm{d}\mu \hfill \hfill \mathrm{(135)}& & =\frac{1}{s^2}\int \frac{{\left(r_{1-s}-q\right)}^2}{q}\mathrm{d}\mu \hfill \hfill \mathrm{(136)}& & =\frac{1}{s^2}{\chi }^2\left(R_{1-s}\parallel Q\right),\hfill \end{array}$$

where (135) holds due to (122). From (136), it follows that for all $\lambda \in [0,1]$,

$$\begin{array}{c}\hfill {\int }_0^{\lambda }\frac{1}{s}{\chi }^2\left(R_{1-s}\parallel Q\right)\mathrm{d}s={\int }_0^{\lambda }s\mathrm{d}s{\chi }^2(P\parallel Q)=\frac{1}{2}{\lambda }^2{\chi }^2(P\parallel Q).\end{array}$$

(137)

5.2. Proof of Proposition 1

(a)

Simple Proof of Pinsker’s Inequality: By [61] or [62] [(58)],

$${\chi }^2(P\parallel Q)\ge \{\begin{array}{ll}{|P-Q|}^2,& \mathrm{if}|P-Q|\in [0,1],\\ \frac{|P-Q|}{2-|P-Q|},& \mathrm{if}|P-Q|\in (1,2].\end{array}$$

(138)

We need the weaker inequality ${\chi }^2(P\parallel Q)\ge {|P-Q|}^2$, proved by the Cauchy–Schwarz inequality:

$$\begin{array}{}\hfill \mathrm{(139)}& \hfill {\chi }^2(P\parallel Q)& =\int \frac{{(p-q)}^2}{q}\mathrm{d}\mu \int q\mathrm{d}\mu \hfill \hfill \mathrm{(140)}& & \ge {\left(\int \frac{|p-q|}{\sqrt{q}}·\sqrt{q}\mathrm{d}\mu \right)}^2\hfill \hfill \mathrm{(141)}& & ={|P-Q|}^2.\hfill \end{array}$$

By combining (24) and (139)–(141), it follows that

$$\begin{array}{}\hfill \mathrm{(142)}& \hfill \frac{1}{log\mathrm{e}}D(P\parallel Q)& ={\int }_0^1{\chi }^2(P\parallel (1-s)P+sQ)\frac{\mathrm{d}s}{s}\hfill \hfill \mathrm{(143)}& & \ge {\int }_0^1|P-\left((1-s)P+sQ\right){|}^2\frac{\mathrm{d}s}{s}\hfill \hfill \mathrm{(144)}& & ={\int }_0^{1}s{|P-Q|}^2\mathrm{d}s\hfill \hfill \mathrm{(145)}& & =\frac{1}{2}{|P-Q|}^2.\hfill \end{array}$$

(b)

Proof of (30) and its local tightness:

$$\begin{array}{}\hfill \mathrm{(146)}& \hfill \frac{1}{log\mathrm{e}}D(P\parallel Q)& ={\int }_0^1{\chi }^2(P\parallel (1-s)P+sQ)\frac{\mathrm{d}s}{s}\hfill \hfill \mathrm{(147)}& & ={\int }_0^1\left(\int \frac{{\left[p-\left(\right(1-s)p+sq)\right]}^2}{(1-s)p+sq}\mathrm{d}\mu \right)\frac{\mathrm{d}s}{s}\hfill \hfill \mathrm{(148)}& & ={\int }_0^1\int \frac{s{(p-q)}^2}{(1-s)p+sq}\mathrm{d}\mu \mathrm{d}s\hfill \hfill \mathrm{(149)}& & \le {\int }_0^1\int s{(p-q)}^2\left(\frac{1-s}{p}+\frac{s}{q}\right)\mathrm{d}\mu \mathrm{d}s\hfill \hfill \mathrm{(150)}& & ={\int }_0^1{s}^2\mathrm{d}s\int \frac{{(p-q)}^2}{q}\mathrm{d}\mu +{\int }_0^{1}s(1-s)\mathrm{d}s\int \frac{{(p-q)}^2}{p}\mathrm{d}\mu \hfill \hfill \mathrm{(151)}& & =\frac{1}{3}{\chi }^2(P\parallel Q)+\frac{1}{6}{\chi }^2(Q\parallel P),\hfill \end{array}$$

where (146) is (24), and (149) holds due to Jensen’s inequality and the convexity of the hyperbola.

We next show the local tightness of inequality (30) by proving that (31) yields (32). Let $\left\{P_n\right\}$ be a sequence of probability measures, defined on a measurable space $(\mathcal{X},\mathcal{F})$, and assume that $\left\{P_n\right\}$ converges to a probability measure P in the sense that (31) holds. In view of [16] [Theorem 7] (see also [15] [Section 4.F] and [63]), it follows that

$$\begin{array}{c}\hfill \underset{n\to \infty }{lim}D(P_n\parallel P)=\underset{n\to \infty }{lim}{\chi }^2(P_n\parallel P)=0,\end{array}$$

(152)

and

$$\begin{array}{}\hfill \mathrm{(153)}& \underset{n\to \infty }{lim}\frac{D(P_n\parallel P)}{{\chi }^2(P_n\parallel P)}=\frac{1}{2}log\mathrm{e},\hfill \hfill \mathrm{(154)}& \underset{n\to \infty }{lim}\frac{{\chi }^2(P_n\parallel P)}{{\chi }^2(P\parallel P_n)}=1,\hfill \end{array}$$

which therefore yields (32).

(c)

Proof of (33) and (34): The proof of (33) relies on (28) and the following lemma.

Lemma 1. 

For all $s,\theta \in (0,1)$,

$$\begin{array}{c}\hfill \frac{D_{{\varphi }_s}(P\parallel Q)}{D_{{\varphi }_{\theta }}(P\parallel Q)}\ge min\left\{\frac{1-\theta }{1-s},\frac{\theta }{s}\right\}.\end{array}$$

(155)

Proof. 

$$\begin{array}{}\hfill \mathrm{(156)}& \hfill D_{{\varphi }_s}(P\parallel Q)& =\int \frac{{(p-q)}^2}{(1-s)p+sq}\mathrm{d}\mu \hfill \hfill \mathrm{(157)}& & =\int \frac{{(p-q)}^2}{(1-\theta )p+\theta q}\frac{(1-\theta )p+\theta q}{(1-s)p+sq}\mathrm{d}\mu \hfill \hfill \mathrm{(158)}& & \ge min\left\{\frac{1-\theta }{1-s},\frac{\theta }{s}\right\}\int \frac{{(p-q)}^2}{(1-\theta )p+\theta q}\mathrm{d}\mu \hfill \hfill \mathrm{(159)}& & =min\left\{\frac{1-\theta }{1-s},\frac{\theta }{s}\right\}{D}_{{\varphi }_{\theta }}(P\parallel Q).\hfill \end{array}$$

 □

From (28) and (155), for all $\theta \in (0,1)$,

$$\begin{array}{}\hfill \mathrm{(160)}& \hfill \frac{1}{log\mathrm{e}}D(P\parallel Q)& ={\int }_0^{\theta }s{D}_{{\varphi }_s}(P\parallel Q)\mathrm{d}s+{\int }_{\theta }^{1}s{D}_{{\varphi }_s}(P\parallel Q)\mathrm{d}s\hfill \hfill \mathrm{(161)}& & \ge {\int }_0^{\theta }\frac{s(1-\theta )}{1-s}·D_{{\varphi }_{\theta }}(P\parallel Q)\mathrm{d}s+{\int }_{\theta }^1\theta D_{{\varphi }_{\theta }}(P\parallel Q)\mathrm{d}s\hfill \hfill \mathrm{(162)}& & =\left[-\theta +{log}_{\mathrm{e}}\left(\frac{1}{1-\theta }\right)\right](1-\theta )D_{{\varphi }_{\theta }}(P\parallel Q)+\theta (1-\theta )D_{{\varphi }_{\theta }}(P\parallel Q)\hfill \hfill \mathrm{(163)}& & =(1-\theta ){log}_{\mathrm{e}}\left(\frac{1}{1-\theta }\right)D_{{\varphi }_{\theta }}(P\parallel Q).\hfill \end{array}$$

This proves (33). Furthermore, under the assumption in (31), for all $\theta \in [0,1]$,

$$\begin{array}{}\hfill \mathrm{(164)}& \hfill \underset{n\to \infty }{lim}\frac{D(P\parallel P_n)}{D_{{\varphi }_{\theta }}(P\parallel P_n)}& =\underset{n\to \infty }{lim}\frac{D(P\parallel P_n)}{{\chi }^2(P\parallel P_n)}\underset{n\to \infty }{lim}\frac{{\chi }^2(P\parallel P_n)}{D_{{\varphi }_{\theta }}(P\parallel P_n)}\hfill \hfill \mathrm{(165)}& & =\frac{1}{2}log\mathrm{e}·\frac{2}{{\varphi }_{\theta }^{″}\left(1\right)}\hfill \hfill \mathrm{(166)}& & =\frac{1}{2}log\mathrm{e},\hfill \end{array}$$

where (165) holds due to (153) and the local behavior of f-divergences [63], and (166) holds due to (17) which implies that ${\varphi }_{\theta }^{″}\left(1\right)=2$ for all $\theta \in [0,1]$. This proves (34).

(d)

Proof of (35): From (24), we get

$$\begin{array}{}\hfill \mathrm{(167)}& \hfill \frac{1}{log\mathrm{e}}D(P\parallel Q)& ={\int }_0^1{\chi }^2(P\parallel (1-s)P+sQ)\frac{\mathrm{d}s}{s}\hfill \hfill \mathrm{(168)}& & ={\int }_0^1\left[{\chi }^2(P\parallel (1-s)P+sQ)-s^2{\chi }^2(P\parallel Q)\right]\frac{\mathrm{d}s}{s}+{\int }_0^{1}s\mathrm{d}s{\chi }^2(P\parallel Q)\hfill \hfill \mathrm{(169)}& & ={\int }_0^1\left[{\chi }^2(P\parallel (1-s)P+sQ)-s^2{\chi }^2(P\parallel Q)\right]\frac{\mathrm{d}s}{s}+\frac{1}{2}{\chi }^2(P\parallel Q).\hfill \end{array}$$

Referring to the integrand of the first term on the right side of (169), for all $s\in (0,1]$,

$$\begin{array}{}\hfill & \frac{1}{s}\left[{\chi }^2(P\parallel (1-s)P+sQ)-s^2{\chi }^2(P\parallel Q)\right]\hfill \mathrm{(170)}& =s\int {(p-q)}^2\left[\frac{1}{(1-s)p+sq}-\frac{1}{q}\right]\mathrm{d}\mu \hfill \hfill \mathrm{(171)}& =s(1-s)\int \frac{{(q-p)}^3}{q\left[(1-s)p+sq\right]}\mathrm{d}\mu \hfill \hfill \mathrm{(172)}& =s(1-s)\int |q-p|·\underset{\le \frac{1}{s}1\{q\ge p\}}{\underbrace{\frac{|q-p|}{q}·\frac{q-p}{p+s(q-p)}}}\mathrm{d}\mu \hfill \hfill \mathrm{(173)}& \le (1-s)\int (q-p)1\{q\ge p\}\mathrm{d}\mu \hfill \hfill \mathrm{(174)}& =\frac{1}{2}(1-s)|P-Q|,\hfill \end{array}$$

where the last equality holds since the equality $\int (q-p)\mathrm{d}\mu =0$ implies that

$$\begin{array}{}\hfill \mathrm{(175)}& \hfill \int (q-p)1\{q\ge p\}\mathrm{d}\mu & =\int (p-q)1\{p\ge q\}\mathrm{d}\mu \hfill \hfill \mathrm{(176)}& & =\frac{1}{2}\int |p-q|\mathrm{d}\mu =\frac{1}{2}|P-Q|.\hfill \end{array}$$

From (170)–(174), an upper bound on the right side of (169) results. This gives

$$\begin{array}{}\hfill \mathrm{(177)}& \hfill \frac{1}{log\mathrm{e}}D(P\parallel Q)& \le \frac{1}{2}{\int }_0^1(1-s)\mathrm{d}s|P-Q|+\frac{1}{2}{\chi }^2(P\parallel Q)\hfill \hfill \mathrm{(178)}& & =\frac{1}{4}|P-Q|+\frac{1}{2}{\chi }^2(P\parallel Q).\hfill \end{array}$$

It should be noted that [15] [Theorem 2(a)] shows that inequality (35) is tight. To that end, let $\epsilon \in (0,1)$, and define probability measures $P_{\epsilon }$ and $Q_{\epsilon }$ on the set $\mathcal{A}=\{0,1\}$ with $P_{\epsilon }\left(1\right)={\epsilon }^2$ and $Q_{\epsilon }\left(1\right)=\epsilon$. Then,

$$\begin{array}{cc}\hfill & \underset{\epsilon \downarrow 0}{lim}\frac{\frac{1}{log\mathrm{e}}D(P_{\epsilon }\parallel Q_{\epsilon })}{\frac{1}{4}|P_{\epsilon }-Q_{\epsilon }|+\frac{1}{2}{\chi }^2(P_{\epsilon }\parallel Q_{\epsilon })}=1.\hfill \end{array}$$

(179)

5.3. Proof of Theorem 2

We first prove Item (a) in Theorem 2. In view of the Hammersley–Chapman–Robbins lower bound on the ${\chi }^2$ divergence, for all $\lambda \in [0,1]$

$$\begin{array}{c}\hfill {\chi }^2\left(P\parallel (1-\lambda )P+\lambda Q\right)\ge \frac{{\left(\mathbb{E}\left[X\right]-\mathbb{E}\left[Z_{\lambda }\right]\right)}^2}{\mathrm{Var}\left(Z_{\lambda }\right)},\end{array}$$

(180)

where $X\sim P$, $Y\sim Q$ and $Z_{\lambda }\sim R_{\lambda }:=(1-\lambda )P+\lambda Q$ is defined by

$$Z_{\lambda }:=\{\begin{array}{ll}\hfill X,& \mathrm{with}\mathrm{probability}1-\lambda ,\hfill \\ \hfill Y,& \mathrm{with}\mathrm{probability}\lambda .\hfill \end{array}$$

(181)

For $\lambda \in [0,1]$,

$$\begin{array}{c}\hfill \mathbb{E}\left[Z_{\lambda }\right]=(1-\lambda )m_P+\lambda m_Q,\end{array}$$

(182)

and it can be verified that

$$\begin{array}{c}\hfill \mathrm{Var}\left(Z_{\lambda }\right)=(1-\lambda ){\sigma }_P^2+\lambda {\sigma }_Q^2+\lambda (1-\lambda ){(m_P-m_Q)}^2.\end{array}$$

(183)

We now rely on identity (24)

$$\begin{array}{cc}\hfill \frac{1}{log\mathrm{e}}D(P\parallel Q)& ={\int }_0^1{\chi }^2(P\parallel (1-\lambda )P+\lambda Q)\frac{\mathrm{d}\lambda }{\lambda }\hfill \end{array}$$

(184)

to get a lower bound on the relative entropy. Combining (180), (183) and (184) yields

$$\begin{array}{c}\hfill \frac{1}{log\mathrm{e}}D(P\parallel Q)\ge {(m_P-m_Q)}^2{\int }_0^1\frac{\lambda }{(1-\lambda ){\sigma }_P^2+\lambda {\sigma }_Q^2+\lambda (1-\lambda ){(m_P-m_Q)}^2}\mathrm{d}\lambda .\end{array}$$

(185)

From (43) and (44), we get

$$\begin{array}{c}\hfill {\int }_0^1\frac{\lambda }{(1-\lambda ){\sigma }_P^2+\lambda {\sigma }_Q^2+\lambda (1-\lambda ){(m_P-m_Q)}^2}\mathrm{d}\lambda ={\int }_0^1\frac{\lambda }{(\alpha -a\lambda )(\beta +a\lambda )}\mathrm{d}\lambda ,\end{array}$$

(186)

where

$$\begin{array}{cc}\hfill \alpha & :=\sqrt{{\sigma }_P^2+\frac{b^2}{4a^2}}+\frac{b}{2a},\hfill \end{array}$$

(187)

$$\begin{array}{cc}\hfill \beta & :=\sqrt{{\sigma }_P^2+\frac{b^2}{4a^2}}-\frac{b}{2a}.\hfill \end{array}$$

(188)

By using the partial fraction decomposition of the integrand on the right side of (186), we get (after multiplying both sides of (185) by $log\mathrm{e}$)

$$\begin{array}{}\hfill \mathrm{(189)}& \hfill D(P\parallel Q)& \ge \frac{{(m_P-m_Q)}^2}{a^2}\left[\frac{\alpha }{\alpha +\beta }log\left(\frac{\alpha }{\alpha -a}\right)+\frac{\beta }{\alpha +\beta }log\left(\frac{\beta }{\beta +a}\right)\right]\hfill \hfill \mathrm{(190)}& & =\frac{\alpha }{\alpha +\beta }log\left(\frac{\alpha }{\alpha -a}\right)+\frac{\beta }{\alpha +\beta }log\left(\frac{\beta }{\beta +a}\right)\hfill \hfill \mathrm{(191)}& & =d(\frac{\alpha }{\alpha +\beta }\left.∥\frac{\alpha -a}{\alpha +\beta }\right),\hfill \end{array}$$

where (189) holds by integration since $\alpha -a\lambda$ and $\beta +a\lambda$ are both non-negative for all $\lambda \in [0,1]$. To verify the latter claim, it should be noted that (43) and the assumption that $m_P\ne m_Q$ imply that $a\ne 0$. Since $\alpha ,\beta >0$, it follows that, for all $\lambda \in [0,1]$, either $\alpha -a\lambda >0$ or $\beta +a\lambda >0$ (if $a<0$, then the former is positive, and, if $a>0$, then the latter is positive). By comparing the denominators of both integrands on the left and right sides of (186), it follows that $(\alpha -a\lambda )(\beta +a\lambda )\ge 0$ for all $\lambda \in [0,1]$. Since the product of $\alpha -a\lambda$ and $\beta +a\lambda$ is non-negative and at least one of these terms is positive, it follows that $\alpha -a\lambda$ and $\beta +a\lambda$ are both non-negative for all $\lambda \in [0,1]$. Finally, (190) follows from (43).

If $m_P-m_Q\to 0$ and ${\sigma }_P\ne {\sigma }_Q$, then it follows from (43) and (44) that $a\to 0$ and $b\to {\sigma }_P^2-{\sigma }_Q^2\ne 0$. Hence, from (187) and (188), $\alpha \ge \left|\frac{b}{a}\right|\to \infty$ and $\beta \to 0$, which implies that the lower bound on $D(P\parallel Q)$ in (191) tends to zero.

Letting $r:=\frac{\alpha }{\alpha +\beta }$ and $s:=\frac{\alpha -a}{\alpha +\beta }$, we obtain that the lower bound on $D(P\parallel Q)$ in (40) holds. This bound is consistent with the expressions of r and s in (41) and (42) since, from (45), (187) and (188),

$$\begin{array}{cc}\hfill r& =\frac{\alpha }{\alpha +\beta }=\frac{v+\frac{b}{2a}}{2v}=\frac{1}{2}+\frac{b}{4av},\hfill \end{array}$$

(192)

$$\begin{array}{cc}\hfill s& =\frac{\alpha -a}{\alpha +\beta }=r-\frac{a}{\alpha +\beta }=r-\frac{a}{2v}.\hfill \end{array}$$

(193)

It should be noted that $r,s\in [0,1]$. First, from (187) and (188), $\alpha$ and $\beta$ are positive if ${\sigma }_P\ne 0$, which yields $r=\frac{\alpha }{\alpha +\beta }\in (0,1)$. We next show that $s\in [0,1]$. Recall that $\alpha -a\lambda$ and $\beta +a\lambda$ are both non-negative for all $\lambda \in [0,1]$. Setting $\lambda =1$ yields $\alpha \ge a$, which (from (193)) implies that $s\ge 0$. Furthermore, from (193) and the positivity of $\alpha +\beta$, it follows that $s\le 1$ if and only if $\beta \ge -a$. The latter holds since $\beta +a\lambda \ge 0$ for all $\lambda \in [0,1]$ (in particular, for $\lambda =1$). If ${\sigma }_P=0$, then it follows from (41)–(45) that $v=\frac{b}{2\left|a\right|}$, $b=a^2+{\sigma }_Q^2$, and (recall that $a\ne 0$)

(i)

If $a>0$, then $v=\frac{b}{2a}$ implies that $r=\frac{1}{2}+\frac{b}{4av}=1$, and $s=r-\frac{a}{2v}=1-\frac{a^2}{b}=\frac{{\sigma }_Q^2}{{\sigma }_Q^2+a^2}\in [0,1]$;

(ii)

If $a<0$, then $v=-\frac{b}{2a}$ implies that $r=0$, and $s=r-\frac{a}{2v}=\frac{a^2}{b}=\frac{a^2}{a^2+{\sigma }_Q^2}\in [0,1]$.

We next prove Item (b) in Theorem 2 (i.e., the achievability of the lower bound in (40)). To that end, we provide a technical lemma, which can be verified by the reader.

Lemma 2.

Let $r,s$ be given in (41)–(45), and let $u_{1,2}$ be given in (47). Then,

$$\begin{array}{cc}\hfill & (s-r)(u_1-u_2)=m_Q-m_P,\hfill \end{array}$$

(194)

$$\begin{array}{cc}\hfill & u_1+u_2=m_P+m_Q+\frac{{\sigma }_Q^2-{\sigma }_P^2}{m_Q-m_P}.\hfill \end{array}$$

(195)

Let $X\sim P$ and $Y\sim Q$ be defined on a set $\mathcal{U}=\{u_1,u_2\}$ (for the moment, the values of $u_1$ and $u_2$ are not yet specified) with $P[X=u_1]=r$, $P[X=u_2]=1-r$, $Q[Y=u_1]=s$, and $Q[Y=u_2]=1-s$. We now calculate $u_1$ and $u_2$ such that $\mathbb{E}\left[X\right]=m_P$ and $\mathrm{Var}\left(X\right)={\sigma }_P^2$. This is equivalent to

$$\begin{array}{}\hfill \mathrm{(196)}& r{u}_1+(1-r)u_2=m_P,\hfill \mathrm{(197)}& r{u}_1^2+(1-r)u_2^2=m_P^2+{\sigma }_P^2.\end{array}$$

Substituting (196) into the right side of (197) gives

$$\begin{array}{c}\hfill r{u}_1^2+(1-r)u_2^2={\left[r{u}_1+(1-r)u_2\right]}^2+{\sigma }_P^2,\end{array}$$

(198)

which, by rearranging terms, also gives

$$\begin{array}{c}\hfill u_1-u_2=\pm \sqrt{\frac{{\sigma }_P^2}{r(1-r)}}.\end{array}$$

(199)

Solving simultaneously (196) and (199) gives

$$\begin{array}{cc}\hfill & u_1=m_P\pm \sqrt{\frac{(1-r){\sigma }_P^2}{r}},\hfill \end{array}$$

(200)

$$\begin{array}{cc}\hfill & u_2=m_P\mp \sqrt{\frac{r{\sigma }_P^2}{1-r}}.\hfill \end{array}$$

(201)

We next verify that, by setting $u_{1,2}$ as in (47), one also gets (as desired) that $\mathbb{E}\left[Y\right]=m_Q$ and $\mathrm{Var}\left(Y\right)={\sigma }_Q^2$. From Lemma 2, and, from (196) and (197), we have

$$\begin{array}{}\hfill \mathrm{(202)}& \hfill \mathbb{E}\left[Y\right]& =s{u}_1+(1-s)u_2\hfill \hfill \mathrm{(203)}& & =\left(r{u}_1+(1-r)u_2\right)+(s-r)(u_1-u_2)\hfill \hfill \mathrm{(204)}& & =m_P+(s-r)(u_1-u_2)=m_Q,\hfill \hfill \mathrm{(205)}& \hfill \mathbb{E}\left[Y^2\right]& =s{u}_1^2+(1-s)u_2^2\hfill \hfill \mathrm{(206)}& & =r{u}_1^2+(1-r)u_2^2+(s-r)(u_1^2-u_2^2)\hfill \hfill \mathrm{(207)}& & =\mathbb{E}\left[X^2\right]+(s-r)(u_1-u_2)(u_1+u_2)\hfill \hfill \mathrm{(208)}& & =m_P^2+{\sigma }_P^2+(m_Q-m_P)\left(m_P+m_Q+\frac{{\sigma }_Q^2-{\sigma }_P^2}{m_Q-m_P}\right)\hfill \hfill \mathrm{(209)}& & =m_Q^2+{\sigma }_Q^2.\hfill \end{array}$$

By combining (204) and (209), we obtain $\mathrm{Var}\left(Y\right)={\sigma }_Q^2$. Hence, the probability mass functions P and Q defined on $\mathcal{U}=\{u_1,u_2\}$ (with $u_1$ and $u_2$ in (47)) such that

$$\begin{array}{c}\hfill P\left(u_1\right)=1-P\left(u_2\right)=r,Q\left(u_1\right)=1-Q\left(u_2\right)=s\end{array}$$

(210)

satisfy the equality constraints in (39), while also achieving the lower bound on $D(P\parallel Q)$ that is equal to $d(r\parallel s)$. It can be also verified that the second option where

$$\begin{array}{c}\hfill u_1=m_P-\sqrt{\frac{(1-r){\sigma }_P^2}{r}},u_2=m_P+\sqrt{\frac{r{\sigma }_P^2}{1-r}}\end{array}$$

(211)

does not yield the satisfiability of the conditions $\mathbb{E}\left[Y\right]=m_Q$ and $\mathrm{Var}\left(Y\right)={\sigma }_Q^2$, so there is only a unique pair of probability measures P and Q, defined on a two-element set that achieves the lower bound in (40) under the equality constraints in (39).

We finally prove Item (c) in Theorem 2. Let $m\in \mathbb{R},{\sigma }_P^2$, and ${\sigma }_Q^2$ be selected arbitrarily such that ${\sigma }_Q^2\ge {\sigma }_P^2$. We construct probability measures $P_{\epsilon }$ and $Q_{\epsilon }$, depending on a free parameter $\epsilon$, with means $m_P=m_Q:=m$ and variances ${\sigma }_P^2$ and ${\sigma }_Q^2$, respectively (means and variances are independent of $\epsilon$), and which are defined on a three-element set $\mathcal{U}:=\{u_1,u_2,u_3\}$ as follows:

$$\begin{array}{cc}\hfill & P_{\epsilon }\left(u_1\right)=r,P_{\epsilon }\left(u_2\right)=1-r,P_{\epsilon }\left(u_3\right)=0,\hfill \end{array}$$

(212)

$$\begin{array}{cc}\hfill & Q_{\epsilon }\left(u_1\right)=s,Q_{\epsilon }\left(u_2\right)=1-s-\epsilon ,Q_{\epsilon }\left(u_3\right)=\epsilon ,\hfill \end{array}$$

(213)

with $\epsilon >0$. We aim to set the parameters $r,s,u_1,u_2$ and $u_3$ (as a function of $m,{\sigma }_P,{\sigma }_Q$ and $\epsilon$) such that

$$\begin{array}{c}\hfill \underset{\epsilon \to 0^{+}}{lim}D(P_{\epsilon }\parallel Q_{\epsilon })=0.\end{array}$$

(214)

Proving (214) yields (48), while it also follows that the infimum on the left side of (48) can be restricted to probability measures which are defined on a three-element set.

In view of the constraints on the means and variances in (39), with equal means m, we get the following set of equations from (212) and (213):

$$\begin{array}{c}\hfill \left\{\begin{array}{cc}\hfill & r{u}_1+(1-r)u_2=m,\hfill \\ \hfill & s{u}_1+(1-s-\epsilon )u_2+\epsilon u_3=m,\hfill \\ \hfill & r{u}_1^2+(1-r)u_2^2=m^2+{\sigma }_P^2,\hfill \\ \hfill & s{u}_1^2+(1-s-\epsilon )u_2^2+\epsilon u_3^2=m^2+{\sigma }_Q^2.\hfill \end{array}\right.\end{array}$$

(215)

The first and second equations in (215) refer to the equal means under P and Q, and the third and fourth equations in (215) refer to the second moments in (39). Furthermore, in view of (212) and (213), the relative entropy is given by

$$\begin{array}{c}\hfill D(P_{\epsilon }\parallel Q_{\epsilon })=rlog\frac{r}{s}+(1-r)log\frac{1-r}{1-s-\epsilon }.\end{array}$$

(216)

Subtracting the square of the first equation in (215) from its third equation gives the equivalent set of equations

$$\begin{array}{c}\hfill \left\{\begin{array}{cc}\hfill & r{u}_1+(1-r)u_2=m,\hfill \\ \hfill & s{u}_1+(1-s-\epsilon )u_2+\epsilon u_3=m,\hfill \\ \hfill & r(1-r){(u_1-u_2)}^2={\sigma }_P^2,\hfill \\ \hfill & s{u}_1^2+(1-s-\epsilon )u_2^2+\epsilon u_3^2=m^2+{\sigma }_Q^2.\hfill \end{array}\right.\end{array}$$

(217)

We next select $u_1$ and $u_2$ such that $u_1-u_2:=2{\sigma }_P$. Then, the third equation in (217) gives $r(1-r)=\frac{1}{4}$, so $r=\frac{1}{2}$. Furthermore, the first equation in (217) gives

$$\begin{array}{cc}\hfill & u_1=m+{\sigma }_P,\hfill \end{array}$$

(218)

$$\begin{array}{cc}\hfill & u_2=m-{\sigma }_P.\hfill \end{array}$$

(219)

Since r, $u_1$, and $u_2$ are independent of $\epsilon$, so is the probability measure $P_{\epsilon }:=P$. Combining the second equation in (217) with (218) and (219) gives

$$\begin{array}{c}\hfill u_3=m-\left(1+\frac{2s-1}{\epsilon }\right){\sigma }_P.\end{array}$$

(220)

Substituting (218)–(220) into the fourth equation of (217) gives a quadratic equation for s, whose selected solution (such that s and $r=\frac{1}{2}$ be close for small $ϵ>0$) is equal to

$$\begin{array}{c}\hfill s=\frac{1}{2}\left[1-\epsilon +\sqrt{\left(\frac{{\sigma }_Q^2}{{\sigma }_P^2}-1+\epsilon \right)\epsilon }\right].\end{array}$$

(221)

Hence, $s=\frac{1}{2}+O\left(\sqrt{\epsilon }\right)$, which implies that $s\in (0,1-\epsilon )$ for sufficiently small $\epsilon >0$ (as it is required in (213)). In view of (216), it also follows that $D(P\parallel Q_{\epsilon })$ vanishes as we let $\epsilon$ tend to zero.

We finally outline an alternative proof, which refers to the case of equal means with arbitrarily selected ${\sigma }_P^2$ and ${\sigma }_Q^2$. Let $({\sigma }_P^2,{\sigma }_Q^2)\in {(0,\infty )}^2$. We next construct a sequence of pairs of probability measures $\left\{(P_n,Q_n)\right\}$ with zero mean and respective variances $({\sigma }_P^2,{\sigma }_Q^2)$ for which $D(P_n\parallel Q_n)\to 0$ as $n\to \infty$ (without any loss of generality, one can assume that the equal means are equal to zero). We start by assuming $({\sigma }_P^2,{\sigma }_Q^2)\in {(1,\infty )}^2$. Let

$$\begin{array}{c}\hfill {\mu }_n:=\sqrt{1+n\left({\sigma }_Q^2-1\right)},\end{array}$$

(222)

and define a sequence of quaternary real-valued random variables with probability mass functions

$$Q_n\left(a\right):=\{\begin{array}{ll}\frac{1}{2}-\frac{1}{2n}& a=\pm 1,\hfill \\ \frac{1}{2n}& a=\pm {\mu }_n.\end{array}$$

(223)

It can be verified that, for all $n\in \mathbb{N}$, $Q_n$ has zero mean and variance ${\sigma }_Q^2$. Furthermore, let

$$P_n\left(a\right):=\{\begin{array}{ll}\frac{1}{2}-\frac{\xi }{2n}& a=\pm 1,\\ \frac{\xi }{2n}& a=\pm {\mu }_n,\end{array}$$

(224)

with

$$\begin{array}{c}\hfill \xi :=\frac{{\sigma }_P^2-1}{{\sigma }_Q^2-1}.\end{array}$$

(225)

If $\xi >1$, for $n=1,\dots ,\lceil \xi \rceil$, we choose $P_n$ arbitrarily with mean 0 and variance ${\sigma }_P^2$. Then,

$$\begin{array}{cc}\hfill & \mathrm{Var}\left(P_n\right)=1-\frac{\xi }{n}+\frac{\xi }{n}{\mu }_n^2={\sigma }_P^2,\hfill \end{array}$$

(226)

$$\begin{array}{cc}\hfill & D(P_n\parallel Q_n)=d\left(\frac{\xi }{n}\parallel \frac{1}{n}\right)\to 0.\hfill \end{array}$$

(227)

Next, suppose $min\{{\sigma }_P^2,{\sigma }_Q^2\}:={\sigma }^2<1$, then construct $P_n^{\prime }$ and $Q_n^{\prime }$ as before with variances $\frac{2{\sigma }_P^2}{{\sigma }^2}>1$ and $\frac{2{\sigma }_Q^2}{{\sigma }^2}>1$, respectively. If $P_n$ and $Q_n$ denote the random variables $P_n^{\prime }$ and $Q_n^{\prime }$ scaled by a factor of $\frac{\sigma }{\sqrt{2}}$, then their variances are ${\sigma }_P^2$, ${\sigma }_Q^2$, respectively, and $D(P_n\parallel Q_n)=D(P_n^{\prime }\parallel Q_n^{\prime })\to 0$ as we let $n\to \infty$.

To conclude, it should be noted that the sequences of probability measures in the latter proof are defined on a four-element set. Recall that, in the earlier proof, specialized to the case of (equal means with) ${\sigma }_P^2\le {\sigma }_Q^2$, the introduced probability measures are defined on a three-element set, and the reference probability measure P is fixed while referring to an equiprobable binary random variable.

5.4. Proof of Theorem 3

We first prove (52). Differentiating both sides of (22) gives that, for all $\lambda \in (0,1]$,

$$\begin{array}{}\hfill \mathrm{(228)}& \hfill F^{\prime }\left(\lambda \right)& =\frac{1}{\lambda }{\chi }^2\left(P\parallel R_{\lambda }\right)log\mathrm{e}\hfill \hfill \mathrm{(229)}& & \ge \frac{1}{\lambda }\left[exp\left(D(P\parallel R_{\lambda })\right)-1\right]log\mathrm{e}\hfill \hfill \mathrm{(230)}& & =\frac{1}{\lambda }\left[exp\left(F\left(\lambda \right)\right)-1\right]log\mathrm{e},\hfill \end{array}$$

where (228) holds due to (21), (22) and (50); (229) holds by (16) and (230) is due to (21) and (50). This gives (52).

We next prove (53), and the conclusion which appears after it. In view of [16] [Theorem 8], applied to $f\left(t\right):=-logt$ for all $t>0$, we get (it should be noted that, by the definition of F in (50), the result in [16] [(195)–(196)] is used here by swapping P and Q)

$$\begin{array}{c}\hfill \underset{\lambda \to 0^{+}}{lim}\frac{F\left(\lambda \right)}{{\lambda }^2}=\frac{1}{2}{\chi }^2(Q\parallel P)log\mathrm{e}.\end{array}$$

(231)

Since $\underset{\lambda \to 0^{+}}{lim}F\left(\lambda \right)=0$, it follows by L’Hôpital’s rule that

$$\begin{array}{c}\hfill \underset{\lambda \to 0^{+}}{lim}\frac{F^{\prime }\left(\lambda \right)}{\lambda }=2\underset{\lambda \to 0^{+}}{lim}\frac{F\left(\lambda \right)}{{\lambda }^2}={\chi }^2(Q\parallel P)log\mathrm{e},\end{array}$$

(232)

which gives (53). A comparison of the limit in (53) with a lower bound which follows from (52) gives

$$\begin{array}{}\hfill \mathrm{(233)}& \hfill \underset{\lambda \to 0^{+}}{lim}\frac{F^{\prime }\left(\lambda \right)}{\lambda }& \ge \underset{\lambda \to 0^{+}}{lim}\frac{1}{{\lambda }^2}\left[exp\left(F\left(\lambda \right)\right)-1\right]log\mathrm{e}\hfill \hfill \mathrm{(234)}& & =\underset{\lambda \to 0^{+}}{lim}\frac{F\left(\lambda \right)}{{\lambda }^2}\underset{\lambda \to 0^{+}}{lim}\frac{exp\left(F\left(\lambda \right)\right)-1}{F\left(\lambda \right)}·log\mathrm{e}\hfill \hfill \mathrm{(235)}& & =\underset{\lambda \to 0^{+}}{lim}\frac{F\left(\lambda \right)}{{\lambda }^2}\underset{u\to 0}{lim}\frac{{\mathrm{e}}^u-1}{u}\hfill \hfill \mathrm{(236)}& & =\frac{1}{2}{\chi }^2(Q\parallel P)log\mathrm{e},\hfill \end{array}$$

where (236) relies on (231). Hence, the limit in (53) is twice as large as its lower bound on the right side of (236). This proves the conclusion which comes right after (53).

We finally prove the known result in (51), by showing an alternative proof which is based on (52). The function F is non-negative on $[0,1]$, and it is strictly positive on $(0,1]$ if $P\ne Q$. Let $P\ne Q$ (otherwise, (51) is trivial). Rearranging terms in (52) and integrating both sides over the interval $[\lambda ,1]$, for $\lambda \in (0,1]$, gives that

$$\begin{array}{}\hfill \mathrm{(237)}& \hfill {\int }_{\lambda }^1\frac{F^{\prime }\left(t\right)}{exp\left(F\left(t\right)\right)-1}\mathrm{d}t& \ge {\int }_{\lambda }^1\frac{\mathrm{d}t}{t}log\mathrm{e}\hfill \hfill \mathrm{(238)}& & =log\frac{1}{\lambda },\forall \lambda \in (0,1].\hfill \end{array}$$

The left side of (237) satisfies

$$\begin{array}{}\hfill \mathrm{(239)}& \hfill {\int }_{\lambda }^1\frac{F^{\prime }\left(t\right)}{exp\left(F\left(t\right)\right)-1}\mathrm{d}t& ={\int }_{\lambda }^1\frac{F^{\prime }\left(t\right)exp\left(-F\left(t\right)\right)}{1-exp\left(-F\left(t\right)\right)}\mathrm{d}t\hfill \hfill \mathrm{(240)}& & ={\int }_{\lambda }^1\frac{\mathrm{d}}{\mathrm{d}t}\left\{log\left(1-exp\left(-F\left(t\right)\right)\right)\right\}\mathrm{d}t\hfill \hfill \mathrm{(241)}& & =log\left(\frac{1-exp\left(-D(P\parallel Q)\right)}{1-exp\left(-F\left(\lambda \right)\right)}\right),\hfill \end{array}$$

where (241) holds since $F\left(1\right)=D(P\parallel Q)$ (see (50)). Combining (237)–(241) gives

$$\begin{array}{c}\hfill \frac{1-exp\left(-D(P\parallel Q)\right)}{1-exp\left(-F\left(\lambda \right)\right)}\ge \frac{1}{\lambda },\forall \lambda \in (0,1],\end{array}$$

(242)

which, due to the non-negativity of F, gives the right side inequality in (51) after rearrangement of terms in (242).

5.5. Proof of Theorem 4

Lemma 3.

Let $f_0:(0,\infty )\to \mathbb{R}$ be a convex function with $f_0\left(1\right)=0$, and let ${\left\{f_k(·)\right\}}_{k=0}^{\infty }$ be defined as in (58). Then, ${\left\{f_k(·)\right\}}_{k=0}^{\infty }$ is a sequence of convex functions on $(0,\infty )$, and

$$\begin{array}{c}\hfill f_k\left(x\right)\ge f_{k+1}\left(x\right),\forall x>0,k\in \{0,1,\dots \}.\end{array}$$

(243)

Proof. 

We prove the convexity of $\left\{f_k(·)\right\}$ on $(0,\infty )$ by induction. Suppose that $f_k(·)$ is a convex function with $f_k\left(1\right)=0$ for a fixed integer $k\ge 0$. The recursion in (58) yields $f_{k+1}\left(1\right)=0$ and, by the change of integration variable $s:=(1-x)s^{\prime }$,

$$\begin{array}{cc}\hfill f_{k+1}\left(x\right)& ={\int }_0^1{f}_k(s^{\prime }x-s^{\prime }+1)\frac{\mathrm{d}{s}^{\prime }}{s^{\prime }},x>0.\hfill \end{array}$$

(244)

Consequently, for $t\in (0,1)$ and $x\ne y$ with $x,y>0$, applying (244) gives

$$\begin{array}{}\hfill \mathrm{(245)}& \hfill f_{k+1}((1-t)x+ty)& ={\int }_0^1{f}_k\left(s^{\prime }[(1-t)x+ty]-s^{\prime }+1\right)\frac{\mathrm{d}{s}^{\prime }}{s^{\prime }}\hfill \hfill \mathrm{(246)}& & ={\int }_0^1{f}_k\left((1-t)(s^{\prime }x-s^{\prime }+1)+t(s^{\prime }y-s^{\prime }+1)\right)\frac{\mathrm{d}{s}^{\prime }}{s^{\prime }}\hfill \hfill \mathrm{(247)}& & \le (1-t){\int }_0^1{f}_k(s^{\prime }x-s^{\prime }+1)\frac{\mathrm{d}{s}^{\prime }}{s^{\prime }}+t{\int }_0^1{f}_k(s^{\prime }y-s^{\prime }+1)\frac{\mathrm{d}{s}^{\prime }}{s^{\prime }}\hfill \hfill \mathrm{(248)}& & =(1-t)f_{k+1}\left(x\right)+t{f}_{k+1}\left(y\right),\hfill \end{array}$$

where (247) holds since $f_k(·)$ is convex on $(0,\infty )$ (by assumption). Hence, from (245)–(248), $f_{k+1}(·)$ is also convex on $(0,\infty )$ with $f_{k+1}\left(1\right)=0$. By mathematical induction and our assumptions on $f_0$, it follows that ${\left\{f_k(·)\right\}}_{k=0}^{\infty }$ is a sequence of convex functions on $(0,\infty )$ which vanish at 1.

We next prove (243). For all $x,y>0$ and $k\in \{0,1,\dots \}$,

$$\begin{array}{}\hfill \mathrm{(249)}& \hfill f_{k+1}\left(y\right)& \ge f_{k+1}\left(x\right)+f_{k+1}^{\prime }\left(x\right)(y-x)\hfill \hfill \mathrm{(250)}& & =f_{k+1}\left(x\right)+\frac{f_k\left(x\right)}{x-1}(y-x),\hfill \end{array}$$

where (249) holds since $f_k(·)$ is convex on $(0,\infty )$, and (250) relies on the recursive equation in (58). Substituting $y=1$ into (249)–(250), and using the equality $f_{k+1}\left(1\right)=0$, gives (243). □

We next prove Theorem 4. From Lemma 3, it follows that $D_{f_k}(P\parallel Q)$ is an f-divergence for all integers $k\ge 0$, and the non-negative sequence $\{D_{f_k}{(P\parallel Q)\}}_{k=0}^{\infty }$ is monotonically non-increasing. From (21) and (58), it also follows that, for all $\lambda \in [0,1]$ and integer $k\in \{0,1,\dots \}$,

$$\begin{array}{}\hfill \mathrm{(251)}& \hfill D_{f_{k+1}}(R_{\lambda }\parallel P)& =\int p{f}_{k+1}\left(\frac{r_{\lambda }}{p}\right)\mathrm{d}\mu \hfill \hfill \mathrm{(252)}& & =\int p{\int }_0^{(p-q)\lambda /p}{f}_k(1-s)\frac{\mathrm{d}s}{s}\mathrm{d}\mu \hfill \hfill \mathrm{(253)}& & =\int p{\int }_0^{\lambda }{f}_k\left(1+\frac{(q-p)s^{\prime }}{p}\right)\frac{\mathrm{d}{s}^{\prime }}{s^{\prime }}\mathrm{d}\mu \hfill \hfill \mathrm{(254)}& & ={\int }_0^{\lambda }\int p{f}_k\left(\frac{r_{s^{\prime }}}{p}\right)\mathrm{d}\mu \frac{\mathrm{d}{s}^{\prime }}{s^{\prime }}\hfill \hfill \mathrm{(255)}& & ={\int }_0^{\lambda }{D}_{f_k}(R_{s^{\prime }}\parallel P)\frac{\mathrm{d}{s}^{\prime }}{s^{\prime }},\hfill \end{array}$$

where the substitution $s:=\frac{(p-q)s^{\prime }}{p}$ is invoked in (253), and then (254) holds since $\frac{r_{s^{\prime }}}{p}=1+\frac{(q-p)s^{\prime }}{p}$ for $s^{\prime }\in [0,1]$ (this follows from (21)) and by interchanging the order of the integrations.

5.6. Proof of Corollary 5

Combining (60) and (61) yields (58); furthermore, $f_0:(0,\infty )\to \mathbb{R}$, given by $f_0\left(x\right)=\frac{1}{x}-1$ for all $x>0$, is convex on $(0,\infty )$ with $f_0\left(1\right)=0$. Hence, Theorem 4 holds for the selected functions ${\left\{f_k(·)\right\}}_{k=0}^{\infty }$ in (61), which therefore are all convex on $(0,\infty )$ and vanish at 1. This proves that (59) holds for all $\lambda \in [0,1]$ and $k\in \{0,1,\dots \}$. Since $f_0\left(x\right)=\frac{1}{x}-1$ and $f_1\left(x\right)=-{log}_{\mathrm{e}}\left(x\right)$ for all $x>0$ (see (60) and (61)), then, for every pair of probability measures P and Q:

$$\begin{array}{c}\hfill D_{f_0}(P\parallel Q)={\chi }^2(Q\parallel P),D_{f_1}(P\parallel Q)=\frac{1}{log\mathrm{e}}D(Q\parallel P).\end{array}$$

(256)

Finally, combining (59), for $k=0$, together with (256), gives (22) as a special case.

5.7. Proof of Theorem 5 and Corollary 6

For an arbitrary measurable set $\mathcal{E}\subseteq \mathcal{X}$, we have from (62)

$$\begin{array}{c}{\mu }_{\mathcal{C}}\left(\mathcal{E}\right)={\int }_{\mathcal{E}}\frac{1_{\mathcal{C}}\left(x\right)}{\mu \left(\mathcal{C}\right)}\mathrm{d}\mu \left(x\right),\end{array}$$

(257)

where $1_{\mathcal{C}}:\mathcal{X}\to \{0,1\}$ is the indicator function of $\mathcal{C}\subseteq \mathcal{X}$, i.e., $1_{\mathcal{C}}\left(x\right):=1\{x\in \mathcal{C}\}$ for $x\in \mathcal{X}$. Hence,

$$\begin{array}{c}\frac{\mathrm{d}{\mu }_{\mathcal{C}}}{\mathrm{d}\mu }\left(x\right)=\frac{1_{\mathcal{C}}\left(x\right)}{\mu \left(\mathcal{C}\right)},\forall x\in \mathcal{X},\end{array}$$

(258)

and

$$\begin{array}{}\hfill \mathrm{(259)}& \hfill D({\mu }_{\mathcal{C}}\parallel \mu )& ={\int }_{\mathcal{X}}f\left(\frac{\mathrm{d}{\mu }_{\mathcal{C}}}{\mathrm{d}\mu }\right)\mathrm{d}\mu \hfill \hfill \mathrm{(260)}& & ={\int }_{\mathcal{C}}f\left(\frac{1}{\mu \left(\mathcal{C}\right)}\right)\mathrm{d}\mu \left(x\right)+{\int }_{\mathcal{X}\backslash \mathcal{C}}f\left(0\right)\mathrm{d}\mu \left(x\right)\hfill \hfill \mathrm{(261)}& & =\mu \left(\mathcal{C}\right)f\left(\frac{1}{\mu \left(\mathcal{C}\right)}\right)+\mu (\mathcal{X}\backslash \mathcal{C})f\left(0\right)\hfill \hfill \mathrm{(262)}& & =\tilde{f}\left(\mu \left(\mathcal{C}\right)\right)+(1-\mu \left(\mathcal{C}\right))f\left(0\right),\hfill \end{array}$$

where the last equality holds by the definition of $\tilde{f}$ in (63). This proves Theorem 5. Corollary 6 is next proved by first proving (67) for the Rényi divergence. For all $\alpha \in (0,1)\cup (1,\infty )$,

$$\begin{array}{}\hfill \mathrm{(263)}& \hfill D_{\alpha }\left({\mu }_{\mathcal{C}}\parallel \mu \right)& =\frac{1}{\alpha -1}log{\int }_{\mathcal{X}}{\left(\frac{\mathrm{d}{\mu }_{\mathcal{C}}}{\mathrm{d}\mu }\right)}^{\alpha }\mathrm{d}\mu \hfill \hfill \mathrm{(264)}& & =\frac{1}{\alpha -1}log{\int }_{\mathcal{C}}{\left(\frac{1}{\mu \left(\mathcal{C}\right)}\right)}^{\alpha }\mathrm{d}\mu \hfill \hfill \mathrm{(265)}& & =\frac{1}{\alpha -1}log\left({\left(\frac{1}{\mu \left(\mathcal{C}\right)}\right)}^{\alpha }\mu \left(\mathcal{C}\right)\right)\hfill \hfill \mathrm{(266)}& & =log\frac{1}{\mu \left(\mathcal{C}\right)}.\hfill \end{array}$$

The justification of (67) for $\alpha =1$ is due to the continuous extension of the order-$\alpha$ Rényi divergence at $\alpha =1$, which gives the relative entropy (see (13)). Equality (65) is obtained from (67) at $\alpha =1$. Finally, (66) is obtained by combining (15) and (67) with $\alpha =2$.

5.8. Proof of Theorem 6

Equation (100) is an equivalent form of (27). From (91) and (100), for all $\alpha \in [0,1]$,

$$\begin{array}{}\hfill \mathrm{(267)}& \hfill \frac{1}{log\mathrm{e}}{S}_{\alpha }(P\parallel Q)& =\alpha \frac{1}{log\mathrm{e}}{K}_{\alpha }(P\parallel Q)+(1-\alpha )\frac{1}{log\mathrm{e}}{K}_{1-\alpha }(Q\parallel P)\hfill \hfill \mathrm{(268)}& & =\alpha {\int }_0^{\alpha }s{D}_{{\varphi }_s}(P\parallel Q)\mathrm{d}s+(1-\alpha ){\int }_0^{1-\alpha }s{D}_{{\varphi }_s}(Q\parallel P)\mathrm{d}s\hfill \hfill \mathrm{(269)}& & =\alpha {\int }_0^{\alpha }s{D}_{{\varphi }_s}(P\parallel Q)\mathrm{d}s+(1-\alpha ){\int }_{\alpha }^1(1-s)D_{{\varphi }_{1-s}}(Q\parallel P)\mathrm{d}s.\hfill \end{array}$$

Regarding the integrand of the second term in (269), in view of (18), for all $s\in (0,1)$

$$\begin{array}{}\hfill \mathrm{(270)}& \hfill D_{{\varphi }_{1-s}}(Q\parallel P)& =\frac{1}{{(1-s)}^2}·{\chi }^2\left(Q\parallel (1-s)P+sQ\right)\hfill \hfill \mathrm{(271)}& & =\frac{1}{s^2}·{\chi }^2\left(P\parallel (1-s)P+sQ\right)\hfill \hfill \mathrm{(272)}& & =D_{{\varphi }_s}(P\parallel Q),\hfill \end{array}$$

where (271) readily follows from (9). Since we also have $D_{{\varphi }_1}(P\parallel Q)={\chi }^2(P\parallel Q)=D_{{\varphi }_0}(Q\parallel P)$ (see (18)), it follows that

$$\begin{array}{c}\hfill D_{{\varphi }_{1-s}}(Q\parallel P)=D_{{\varphi }_s}(P\parallel Q),s\in [0,1].\end{array}$$

(273)

By using this identity, we get from (269) that, for all $\alpha \in [0,1]$

$$\begin{array}{}\hfill \mathrm{(274)}& \hfill \frac{1}{log\mathrm{e}}{S}_{\alpha }(P\parallel Q)& =\alpha {\int }_0^{\alpha }s{D}_{{\varphi }_s}(P\parallel Q)\mathrm{d}s+(1-\alpha ){\int }_{\alpha }^1(1-s)D_{{\varphi }_s}(P\parallel Q)\mathrm{d}s\hfill \hfill \mathrm{(275)}& & ={\int }_0^1{g}_{\alpha }\left(s\right)D_{{\varphi }_s}(P\parallel Q)\mathrm{d}s,\hfill \end{array}$$

where the function $g_{\alpha }:[0,1]\to \mathbb{R}$ is defined in (102). This proves the integral identity (101).

The lower bounds in (103) and (104) hold since, if $f:(0,\infty )\to \mathbb{R}$ is convex, continuously twice differentiable and strictly convex at 1, then

$$\begin{array}{c}\hfill {\mu }_{{\chi }^2}(Q_X,W_{Y|X})\le {\mu }_f(Q_X,W_{Y|X}),\end{array}$$

(276)

(see, e.g., [46] [Proposition II.6.5] and [50] [Theorem 2]). Hence, this holds in particular for the f-divergences in (95) and (96) (since the required properties are satisfied by the parametric functions in (97) and (98), respectively). We next prove the upper bound on the contraction coefficients in (103) and (104) by relying on (100) and (101), respectively. In the setting of Definition 7, if $P_X\ne Q_X$, then it follows from (100) that for $\alpha \in (0,1]$,

$$\begin{array}{}\hfill \mathrm{(277)}& \hfill \frac{K_{\alpha }(P_Y\parallel Q_Y)}{K_{\alpha }(P_X\parallel Q_X)}& =\frac{{\int }_0^{\alpha }s{D}_{{\varphi }_s}(P_Y\parallel Q_Y)\mathrm{d}s}{{\int }_0^{\alpha }s{D}_{{\varphi }_s}(P_X\parallel Q_X)\mathrm{d}s}\hfill \hfill \mathrm{(278)}& & \le \frac{{\int }_0^{\alpha }s{\mu }_{{\varphi }_s}(Q_X,W_{Y|X})D_{{\varphi }_s}(P_X\parallel Q_X)\mathrm{d}s}{{\int }_0^{\alpha }s{D}_{{\varphi }_s}(P_X\parallel Q_X)\mathrm{d}s}\hfill \hfill \mathrm{(279)}& & \le \underset{s\in (0,\alpha ]}{sup}{\mu }_{{\varphi }_s}(Q_X,W_{Y|X}).\hfill \end{array}$$

Finally, taking the supremum of the left-hand side of (277) over all probability measures $P_X$ such that $0<K_{\alpha }(P_X\parallel Q_X)<\infty$ gives the upper bound on ${\mu }_{k_{\alpha }}(Q_X,W_{Y|X})$ in (103). The proof of the upper bound on ${\mu }_{s_{\alpha }}(Q_X,W_{Y|X})$, for all $\alpha \in [0,1]$, follows similarly from (101), since the function $g_{\alpha }(·)$ as defined in (102) is positive over the interval $(0,1)$.

5.9. Proof of Corollary 7

The upper bounds in (106) and (107) rely on those in (103) and (104), respectively, by showing that

$$\begin{array}{c}\hfill \underset{s\in (0,1]}{sup}{\mu }_{{\varphi }_s}(Q_X,W_{Y|X})\le {\mu }_{{\chi }^2}\left(W_{Y|X}\right).\end{array}$$

(280)

Inequality (280) is obtained as follows, similarly to the concept of the proof of [51] [Remark 3.8]. For all $s\in (0,1]$ and $P_X\ne Q_X$,

$$\begin{array}{}\hfill & \frac{D_{{\varphi }_s}(P_X{W}_{Y|X}\parallel Q_X{W}_{Y|X})}{D_{{\varphi }_s}(P_X\parallel Q_X)}\mathrm{(281)}& =\frac{{\chi }^2(P_X{W}_{Y|X}\parallel (1-s)P_X{W}_{Y|X}+s{Q}_X{W}_{Y|X})}{{\chi }^2(P_X\parallel (1-s)P_X+s{Q}_X)}\hfill \mathrm{(282)}& \le {\mu }_{{\chi }^2}((1-s)P_X+s{Q}_X,W_{Y|X})\hfill \hfill \mathrm{(283)}& \le {\mu }_{{\chi }^2}\left(W_{Y|X}\right),\hfill \end{array}$$

where (281) holds due to (18), and (283) is due to the definition in (105).

5.10. Proof of Proposition 3

The lower bound on the contraction coefficients in (108) and (109) is due to (276). The derivation of the upper bounds relies on [49] [Theorem 2.2], which states the following. Let $f:[0,\infty )\to \mathbb{R}$ be a three–times differentiable, convex function with $f\left(1\right)=0$, $f^{″}\left(1\right)>0$, and let the function $z:(0,\infty )\to \mathbb{R}$ defined as $z\left(t\right):=\frac{f\left(t\right)-f\left(0\right)}{t}$, for all $t>0$, be concave. Then,

$$\begin{array}{c}\hfill {\mu }_f(Q_X,W_{Y|X})\le \frac{f^{\prime }\left(1\right)+f\left(0\right)}{f^{″}\left(1\right)Q_{min}}·{\mu }_{{\chi }^2}(Q_X,W_{Y|X}).\end{array}$$

(284)

For $\alpha \in (0,1]$, let $z_{\alpha ,1}:(0,\infty )\to \mathbb{R}$ and $z_{\alpha ,2}:(0,\infty )\to \mathbb{R}$ be given by

$$\begin{array}{cc}\hfill z_{\alpha ,1}\left(t\right)& :=\frac{k_{\alpha }\left(t\right)-k_{\alpha }\left(0\right)}{t},t>0,\hfill \end{array}$$

(285)

$$\begin{array}{cc}\hfill z_{\alpha ,2}\left(t\right)& :=\frac{s_{\alpha }\left(t\right)-s_{\alpha }\left(0\right)}{t},t>0,\hfill \end{array}$$

(286)

with $k_{\alpha }$ and $s_{\alpha }$ in (97) and (98). Straightforward calculus shows that, for $\alpha \in (0,1]$ and $t>0$,

$$\begin{array}{}\mathrm{(287)}& \frac{1}{log\mathrm{e}}{z}_{\alpha ,1}^{″}\left(t\right)& =& -\frac{{\alpha }^2+2\alpha (1-\alpha )t}{t^2{\left[\alpha +(1-\alpha )t\right]}^2}<0,\hfill \hfill & \frac{1}{log\mathrm{e}}{z}_{\alpha ,2}^{″}\left(t\right)& =& -\frac{{\alpha }^2\left[\alpha +2(1-\alpha )t\right]}{t^2{\left[\alpha +(1-\alpha )t\right]}^2}\hfill \hfill \mathrm{(288)}& & & -\frac{2(1-\alpha )}{t^3}\left[{log}_{\mathrm{e}}\left(1+\frac{(1-\alpha )t}{\alpha }\right)-\frac{(1-\alpha )t}{\alpha +(1-\alpha )t}-\frac{{(1-\alpha )}^2{t}^2}{2{\left[\alpha +(1-\alpha )t\right]}^2}\right].\hfill \end{array}$$

The first term on the right side of (288) is negative. For showing that the second term is also negative, we rely on the power series expansion ${log}_{\mathrm{e}}(1+u)=u-\frac{1}{2}{u}^2+\frac{1}{3}{u}^3-\dots$ for $u\in (-1,1]$. Setting $u:=-\frac{x}{1+x}$, for $x>0$, and using Leibnitz theorem for alternating series yields

$$\begin{array}{c}\hfill {log}_{\mathrm{e}}(1+x)=-{log}_{\mathrm{e}}\left(1-\frac{x}{1+x}\right)>\frac{x}{1+x}+\frac{x^2}{2{(1+x)}^2},x>0.\end{array}$$

(289)

Consequently, setting $x:=\frac{(1-\alpha )t}{\alpha }\in [0,\infty )$ in (289), for $t>0$ and $\alpha \in (0,1]$, proves that the second term on the right side of (288) is negative. Hence, $z_{\alpha ,1}^{″}\left(t\right),z_{\alpha ,2}^{″}\left(t\right)<0$, so both $z_{\alpha ,1},z_{\alpha ,2}:(0,\infty )\to \mathbb{R}$ are concave functions.

In view of the satisfiability of the conditions of [49] [Theorem 2.2] for the f-divergences with $f=k_{\alpha }$ or $f=s_{\alpha }$, the upper bounds in (108) and (109) follow from (284), and also since

$$\begin{array}{}\hfill \mathrm{(290)}& k_{\alpha }\left(0\right)=0,& k_{\alpha }^{\prime }\left(1\right)=\alpha log\mathrm{e},& k_{\alpha }^{″}\left(1\right)={\alpha }^{2}log\mathrm{e},\hfill \mathrm{(291)}& s_{\alpha }\left(0\right)=-(1-\alpha )log\alpha ,& s_{\alpha }^{\prime }\left(1\right)=(2\alpha -1)log\mathrm{e},& s_{\alpha }^{″}\left(1\right)=(1-3\alpha +3{\alpha }^2)log\mathrm{e}.\end{array}$$

5.11. Proof of Proposition 4

In view of (24), we get

$$\begin{array}{}\hfill \mathrm{(292)}& \hfill \frac{D(P_Y\parallel Q_Y)}{D(P_X\parallel Q_X)}& =\frac{{\int }_0^1{\chi }^2(P_Y\parallel (1-s)P_Y+s{Q}_Y)\frac{\mathrm{d}s}{s}}{{\int }_0^1{\chi }^2(P_X\parallel (1-s)P_X+s{Q}_X)\frac{\mathrm{d}s}{s}}\hfill \hfill \mathrm{(293)}& & \le \frac{{\int }_0^1{\mu }_{{\chi }^2}((1-s)P_X+s{Q}_X,W_{Y|X}){\chi }^2(P_X\parallel (1-s)P_X+s{Q}_X)\frac{\mathrm{d}s}{s}}{{\int }_0^1{\chi }^2(P_X\parallel (1-s)P_X+s{Q}_X)\frac{\mathrm{d}s}{s}}\hfill \hfill \mathrm{(294)}& & \le \underset{s\in [0,1]}{sup}{\mu }_{{\chi }^2}((1-s)P_X+s{Q}_X,W_{Y|X}).\hfill \end{array}$$

In view of (119), the distributions of $X_s$ and $Y_s$, and since $\left((1-s)P_X+s{Q}_X\right)W_{Y|X}=(1-s)P_Y+s{Q}_Y$ holds for all $s\in [0,1]$, it follows that

$$\begin{array}{c}\hfill {\rho }_{\mathrm{m}}(X_s;Y_s)=\sqrt{{\mu }_{{\chi }^2}((1-s)P_X+s{Q}_X,W_{Y|X})},s\in [0,1],\end{array}$$

(295)

which, from (292)–(295), implies that

$$\begin{array}{c}\hfill \underset{s\in [0,1]}{sup}{\rho }_{\mathrm{m}}(X_s;Y_s)\ge \sqrt{\frac{D(P_Y\parallel Q_Y)}{D(P_X\parallel Q_X)}}.\end{array}$$

(296)

Switching $P_X$ and $Q_X$ in (292)–(294) and using the mapping $s\mapsto 1-s$ in (294) gives (due to the symmetry of the maximal correlation)

$$\begin{array}{c}\hfill \underset{s\in [0,1]}{sup}{\rho }_{\mathrm{m}}(X_s;Y_s)\ge \sqrt{\frac{D(Q_Y\parallel P_Y)}{D(Q_X\parallel P_X)}},\end{array}$$

(297)

and, finally, taking the maximal lower bound among those in (296) and (297) gives (120).