Reversing Jensen’s Inequality for Information-Theoretic Analyses

Abstract

In this work, we propose both an improvement and extensions of a reverse Jensen inequality due to Wunder et al. (2021). The new proposed inequalities are fairly tight and reasonably easy to use in a wide variety of situations, as demonstrated in several application examples that are relevant to information theory. Moreover, the main ideas behind the derivations turn out to be applicable to generate bounds to expectations of multivariate convex/concave functions, as well as functions that are not necessarily convex or concave.

1. Introduction

It is very well known that Jensen’s inequality is one of the most elementary mathematical tools, which is extremely useful in a variety of fields, including information theory. At the general level, it covers many other well-known inequalities, which are useful in their own right, as special cases. Examples in general applied mathematics include the Shwartz–Cauchy inequality (which in turn serves the uncertainty principle and the Cramér–Rao inequality), the Hölder inequality, the Lyapunov inequality, and the inequalities of arithmetic, geometric, and harmonic means, just to name a few. More specifically, in information theory, it is basis of the information inequality (i.e., the non-negativity of the Kullback–Leibler divergence), the data processing inequality (which, in turn, leads to the Fano inequality), and the property that conditioning reduces entropy. Moreover, it has a pivotal role in the generation of single-letter formulas in Shannon theory and in the theory of maximum entropy under moment constraints (see, for example, Chapter 12 of [1]).

As often as not, however, applied mathematicians, and in particular, information theorists (the author included), encounter the somewhat frustrating situation that Jensen’s inequality works in the opposite direction than the one they would hope for along their way to obtaining their desired results. It is conceivable to speculate that it is this fact that has triggered a considerable research effort in developing a variety of versions of the so-called reverse Jensen inequality (RJI) (see, e.g., Refs. [2,3,4,5,6,7,8,9,10,11] for a non-exhaustive sample of articles on this topic). In most of these (and other) works, the inequalities derived are demonstrated in a variety of applications, for example, useful relationships between arithmetic and geometric means, reverse bounds on the entropy, the Kullback–Leibler divergence (and, more generally, Csiszár’s f-divergence), reverse versions of the Hölder inequality, etc. In many of the abovementioned papers, the main results are given in the form of an upper bound on the difference, $\mathbb{E}\left\{f\right(X\left)\right\}-f\left(\mathbb{E}\right\{X\left\}\right)$, where f is the convex function, $\mathbb{E}\{·\}$ is the expectation operator, and X is the random variable. These upper bounds, however, depend mostly on global properties of the function f, such as its range and domain, but not quite on the underlying probability density function (PDF) of X (or its probability mass function, in the discrete case). For example, a desirable property of an RJI would be tightness when the PDF of X is well concentrated in the vicinity of its mean, just like the same well-known property of the ordinary Jensen inequality,

$$\mathbb{E}\left\{f\right(X\left)\right\}\ge f\left(\mathbb{E}\right\{X\left\}\right).$$

(1)

In this work, we build on [9] and provide several new variants of the RJI, which possess the abovementioned desired property of tightness under measure concentration. Our starting point is the same as in (the proof of) Lemma 1 of [9], but the continuation of our derivation is substantially different. As a result of this difference, the proposed approach yields considerably tighter bounds, which are reasonably convenient to analyze and calculate in many cases of interest, as we demonstrate throughout this work. We also extend the scope to functions of more than one variable, which are convex (or concave) in each variable alone, but not necessarily so jointly, in all variables. Finally, using similar ideas, we also derive upper and lower bounds on expectations of functions that are not necessarily convex or concave altogether.

To summarize, the main contributions of this work relative to [9] are as follows.

• Improvement of the tightness of the lower bound on the expectation of a concave function.
• Relaxing some of the assumptions on the concave function.
• Proposing a more convenient (and perhaps more natural) passage from lower bounds to expectations of concave functions to upper bounds on expectations of convex functions.
• Extension to bivariate (and multivariate) functions that are concave (or convex) in each variable.
• Providing examples of usefulness in information theory (other than the mutual information estimation of [9]).

The outline of this article is as follows. In Section 2, we present the basic inequality, rooted in Lemma 1 of [9], and discuss its weaknesses. In Section 3, we present two alternative approaches to improve on the basic bound of [9] and discuss the relative strengths and weaknesses of each one. In Section 4, we provide several examples for the usefulness of our proposed bounds. In Section 5, we mostly discuss variations and modifications of our main results, and finally, in Section 6, we extend the scope to bivariate (and multivariate) functions.

2. The Basic Reverse Inequality

Our starting point, which is very similar to Lemma 1 of [9], is the following Lemma.

Lemma 1.

Let $f:{\mathbb{R}}^{+}\to \mathbb{R}$ be a concave function with $f\left(x\right)\ge f\left(0\right)$ for every $x\ge 0$. Let X be a non-negative random variable with a finite mean, $\mathbb{E}\left\{X\right\}=\mu$. Then,

$$\mathbb{E}\left\{f\left(X\right)\right\}\ge \underset{a>0}{sup}\left[\frac{\mu }{a}·f\left(a\right)+\left(1-\frac{\mu }{a}\right)·f\left(0\right)-\frac{f\left(a\right)-f\left(0\right)}{a}·\mathbb{E}\left\{X·\mathcal{I}[X>a]\right\}\right],$$

(2)

where $\mathcal{I}[X>a]$ denotes the indicator function of event $\{X>a\}$.

The proof is almost the same as in the first few steps in the proof of ([9], Lemma 1), but nevertheless, we present it here for completeness.

Proof of Lemma 1.

For every $x\in [0,a]$,

$$\begin{array}{ccc}f\left(x\right)\hfill & =& \hfill f\left(\left[1-\frac{x}{a}\right]·0+\frac{x}{a}·a\right)\end{array}$$

(3)

$$\begin{array}{ccc}& \hspace{1em}\hspace{1em}\ge \hfill & \hfill \left(1-\frac{x}{a}\right)·f\left(0\right)+\frac{x}{a}·f\left(a\right)\end{array}$$

(4)

$$\begin{array}{ccc}& \hfill \hspace{1em}\hspace{1em}=& \hfill f\left(0\right)+x·\frac{f\left(a\right)-f\left(0\right)}{a},\end{array}$$

(5)

and so,

$$\frac{f\left(x\right)-f\left(0\right)}{x}\ge \frac{f\left(a\right)-f\left(0\right)}{a}\mathrm{whenever}0\le x\le a.$$

(6)

Thus,

$$\begin{array}{ccc}\hfill \mathbb{E}\left\{f\right(X\left)\right\}& =& \mathbb{E}\left\{\frac{f\left(X\right)-f\left(0\right)}{X}·X\right\}+f\left(0\right)\hfill \end{array}$$

(7)

$$\begin{array}{ccc}& \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\ge \hfill & \mathbb{E}\left\{\frac{f\left(X\right)-f\left(0\right)}{X}·X·\mathcal{I}[X\le a]\right\}+f\left(0\right)\hfill \end{array}$$

(8)

$$\begin{array}{ccc}& \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\ge \hfill & \mathbb{E}\left\{\frac{f\left(a\right)-f\left(0\right)}{a}·X·\mathcal{I}[X\le a]\right\}+f\left(0\right)\hfill \end{array}$$

(9)

$$\begin{array}{ccc}& \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}=\hfill & \frac{f\left(a\right)-f\left(0\right)}{a}·\mathbb{E}\left\{X·\mathcal{I}[X\le a]\right\}+f\left(0\right)\hfill \end{array}$$

(10)

$$\begin{array}{ccc}& \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}=\hfill & \frac{f\left(a\right)-f\left(0\right)}{a}·\mathbb{E}\left\{X·(1-\mathcal{I}[X>a\left]\right)\right\}+f\left(0\right)\hfill \end{array}$$

(11)

$$\begin{array}{ccc}& \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}=\hfill & \frac{f\left(a\right)-f\left(0\right)}{a}·\mu +f\left(0\right)-\frac{f\left(a\right)-f\left(0\right)}{a}·\mathbb{E}\left\{X·\mathcal{I}[X>a]\right\}\hfill \end{array}$$

(12)

$$\begin{array}{ccc}& \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}=\hfill & \frac{\mu }{a}·f\left(a\right)+\left(1-\frac{\mu }{a}\right)·f\left(0\right)-\frac{f\left(a\right)-f\left(0\right)}{a}·\mathbb{E}\left\{X·\mathcal{I}[X>a]\right\},\hfill \end{array}$$

(13)

where in the first inequality, we relied on the assumption that $f\left(x\right)\ge f\left(0\right)$ for every $x\ge 0$ and in the second inequality we have used (6). Since this chain of inequalities holds true for every $a>0$, we may take the supremum of the right-hand side (r.h.s.) over all $a>0$. This completes the proof of Lemma 1. □

Lemma 1 serves as the basis of our subsequent derivations throughout this paper. The main remaining issue now is how to assess the term

$$q\left(a\right)\stackrel{▵}{=}\mathbb{E}\left\{X·\mathcal{I}[X>a]\right\}.$$

(14)

In simple cases, $q\left(a\right)$ can be calculated exactly by a closed form expression, for example, when the PDF of X is uniform, or triangular, or exponential, etc. In most cases of interest, however, it is difficult, if not impossible, to derive an exact, closed-form expression for $q\left(a\right)$, and then one has to resort to upper bounds in order to further lower bound the r.h.s. of Equation (2).

In [9] (Lemma 1), it was proposed to upper bound $q\left(a\right)$ by applying the Hölder inequality and then the Markov inequality, to obtain

$$q\left(a\right)\le \underset{p>1}{inf}\left\{{\left(\mathbb{E}\left\{X^p\right\}\right)}^{1/p}·{\left(\frac{\mu }{a}\right)}^{1-1/p}\right\}.$$

(15)

This is a very interesting bound, but unfortunately, it suffers from two main weaknesses. The first is that in many cases, the calculation of high order moments of X may not be a trivial task, let alone the step of taking the infimum over p. More importantly, the second weakness is that, as mentioned in the Introduction, it does not yield a tight lower bound to $\mathbb{E}\left\{f\right(X\left)\right\}$ when the PDF of X concentrates strongly around $\mu$. To see the intuition, consider the following argument. When X fluctuates in the vicinity of $\mu$ with high probability, it is clear that the optimal choice of a, in the sense of maximizing the r.h.s. of (2), is slightly larger than $\mu$, because then $q\left(a\right)$ would be small and the main term of (2) would be close to $f\left(\mu \right)$, which, in turn, is also the Jensen upper bound to $\mathbb{E}\left\{f\right(X\left)\right\}$ and hence it is fairly tight. Now, by the Lypunov inequality, the factor $\mathbb{E}\left\{X^p\right\}{]}^{1/p}$ is increasing in p, and so, for every $p\ge 1$, ${\left[\mathbb{E}\left\{X^p\right\}\right]}^{1/p}\ge \mathbb{E}\left\{X\right\}=\mu$. At the same time, for every $a>\mu$, we have ${(\mu /a)}^{1-1/p}\ge \mu /a$. Consequently, if we take, for the sake of simplicity, $f\left(0\right)=0$, the lower bound of [9] (Lemma 1), cannot be larger than

$$\underset{a>\mu }{sup}\frac{f\left(a\right)}{a}·\left(\mu -\frac{{\mu }^2}{a}\right)=\underset{a>\mu }{sup}f\left(a\right)·\frac{\mu }{a}\left(1-\frac{\mu }{a}\right)=\underset{0\le \theta <1}{sup}f\left(\frac{\mu }{\theta }\right)\theta (1-\theta ).$$

(16)

In fact, this is a very generous assessment of the bound of ([9], Lemma 1), as the inequality ${\left[\mathbb{E}\left\{X^p\right\}\right]}^{1/p}\ge \mathbb{E}X=\mu$ is met with equality for $p=1$, whereas the inequality ${(\mu /a)}^{1-1/p}\ge \mu /a$ is met for $p\to \infty$.

For example, if $f\left(x\right)=\sqrt{x}$,9] (Lemma 1) yields a lower bound which is smaller than

$$\underset{0\le \theta <1}{sup}\sqrt{\frac{\mu }{\theta }}\theta (1-\theta )=\sqrt{\mu }·\underset{0\le \theta <1}{sup}\sqrt{\theta }(1-\theta )=\frac{2\sqrt{3}}{9}·\sqrt{\mu }=0.3849\sqrt{\mu }$$

(17)

even if $X=\mu$ with probability one.

These observations motivate the quest for improvements of the upper bound on $q\left(a\right)$, which is the subject of the next section.

3. Alternative Upper Bounds to $\mathit{q}\left(\mathit{a}\right)$

For cases where $q\left(a\right)$ does not lend itself to an exact closed-form expression, we propose two basic alternative approaches for upper bounding $q\left(a\right)$, which both have the property that when X concentrates around $\mu$, $q\left(a\right)$ is small even when a is only slightly larger than $\mu$, which in turn yields a tight bound very close to $f\left(\mu \right)$.

The choice between the two approaches depends on the problem at hand and the capability to obtain closed-form expressions for the moments involved, if they exist at all.

• The Chernoff approach. The first approach is to upper bound the indicator function, $\mathcal{I}\{x>a\}$, by the exponential function $e^{s(x-a)}$ ($s\ge 0$), exactly like in the Chernoff bound. This would yield

  $$q\left(a\right)\le \underset{s\ge 0}{inf}\mathbb{E}\left\{X{e}^{s(X-a)}\right\}=\underset{s\ge 0}{inf}\left[e^{-as}\mathbb{E}\left\{X{e}^{sX}\right\}\right]=\underset{s\ge 0}{inf}\left[e^{-as}{\Phi }^{\prime }\left(s\right)\right]\stackrel{▵}{=}{q}_{Chernoff}\left(a\right),$$

  (18)

  where ${\Phi }^{\prime }\left(s\right)$ is the derivative of the moment generating function (MGF), $\Phi \left(s\right)\stackrel{▵}{=}\mathbb{E}\left\{e^{sX}\right\}$. Thus, Equation (2) is further lower bounded as

  $$\mathbb{E}\left\{f\left(X\right)\right\}\ge \underset{a>0}{sup}\left[\frac{\mu }{a}·f\left(a\right)+\left(1-\frac{\mu }{a}\right)·f\left(0\right)-\frac{f\left(a\right)-f\left(0\right)}{a}·q_{Chernoff}\left(a\right)\right].$$

  (19)

This bound is useful when X has a finite MGF, $\Phi \left(s\right)$, at least in some range of $s>0$, and $\Phi \left(s\right)$ is differentiable in that range. Moreover, for the bound to be useful, $q_{Chernoff}\left(a\right)$ must lend itself to a reasonably simple closed-form expression.

As a slight variation of the Chernoff approach is to upper bound the function $x·\mathcal{I}[x>a]$ itself (and not just the indicator function factor) by an exponential function of the form $a·e^{s(x-a)}$, where s is such that the derivative w.r.t. x at $x=a$ is not less than 1, so that it is at least tangential to the function $x·\mathcal{I}[x>a]$ for $x\downarrow a$. This means $as\ge 1$, or $s\ge 1/a$. Thus,

$$q\left(a\right)\le a·\underset{s\ge 1/a}{inf}\{e^{-as}\Phi \left(s\right)\}\stackrel{▵}{=}{\tilde{q}}_{Chernoff}\left(a\right)$$

(20)

which, of course, may replace $q_{Chernoff}\left(a\right)$ in Equation (19). The usefulness of this version of the bound is essentially under the same circumstances as those of $q_{Chernoff}\left(a\right)$. It has the small advantage that there is no need to differentiate $\Phi \left(s\right)$, but the range of the optimization over s is somewhat smaller.

2.

The Chebychev–Cantelli approach. According to this approach, the function $x·\mathcal{I}[x>a]$ is upper bounded by a quadratic function, in the spirit of the Chebychev–Cantelli inequality, i.e.,

$$x·\mathcal{I}[x>a]\le \frac{a{(x+s)}^2}{{(a+s)}^2},$$

(21)

where the parameter $s\ge 0$ is optimized under the constraint that the derivative at $x=a$, which is $2a/(a+s)$, is at least 1 (again, to be at least tangential to the function itself at $x\downarrow a$), which is equivalent to the requirement, $s\le a$. In this case, denoting ${\sigma }^2=Var\left\{X\right\}$, we obtain

$$q\left(a\right)\le \frac{a\mathbb{E}\left\{{(X+s)}^2\right\}}{{(a+s)}^2}=\frac{a[{\sigma }^2+{(\mu +s)}^2]}{{(a+s)}^2},$$

(22)

which, when minimized over $s\in [0,a]$, yields

$$s^{*}=min\left\{a,\frac{{\sigma }^2}{a-\mu }-\mu \right\},$$

(23)

and then the best bound is given by

$$q\left(a\right)\le q_{Cheb-Cant}\left(a\right)\stackrel{▵}{=}\left\{\begin{array}{cc}\frac{{\sigma }^2+{(a+\mu )}^2}{4a}\hfill & a<a_c\hfill \\ \frac{a{\sigma }^2}{{\sigma }^2+{(a-\mu )}^2}\hfill & a\ge a_c\hfill \end{array}\right.$$

(24)

where $a_c\stackrel{▵}{=}\sqrt{{\sigma }^2+{\mu }^2}$.

The Chernoff approach yields better bounds than the Chebychev–Cantelli approach in many cases. Suppose, for example, that $X={\sum }_{i=1}^n{Y}_i$, where $Y_1,\dots ,Y_n$ are independently and identically distributed (i.i.d.) random variables, all having mean ${\mu }_Y$, variance ${\sigma }_Y^2$, and MGF ${\Phi }_Y\left(s\right)$. Then, of course, $\mu =n{\mu }_Y$, ${\sigma }^2=n{\sigma }_Y^2$, and $\Phi \left(s\right)={\left[{\Phi }_Y\left(s\right)\right]}^n$. For simplicity, suppose also that $f\left(0\right)=0$. In this case, the Chernoff approach yields

$$\begin{array}{ccc}\hfill \mathbb{E}\left\{f\left(\sum _{i=1}^n{Y}_i\right)\right\}& \ge & \frac{n{\mu }_Y}{a}·f\left(a\right)-\frac{f\left(a\right)}{a}\underset{s\ge 0}{inf}\left\{e^{-sa}\frac{d}{ds}{\left[{\Phi }_Y\left(s\right)\right]}^n\right\}\hfill \\ & =& \frac{n{\mu }_Y}{a}·f\left(a\right)-\frac{nf\left(a\right)}{a}\underset{s\ge 0}{inf}\left\{e^{-sa}{\left[{\Phi }_Y\left(s\right)\right]}^{n-1}{\Phi }_Y^{\prime }\left(s\right)\right\}\hfill \\ & =& \frac{nf\left(a\right)}{a}\left[{\mu }_Y-\underset{s\ge 0}{inf}\left\{e^{-sa}{\left[{\Phi }_Y\left(s\right)\right]}^n·\frac{dln{\Phi }_Y\left(s\right)}{ds}\right\}\right].\hfill \end{array}$$

(25)

Now, if $Y_1,Y_2,\dots$ obey a large deviations principle, the second term in the square brackets tends to zero exponentially for the choice $a=n({\mu }_Y+ϵ)$ with arbitrarily small $ϵ>0$. In this case, let $s^{*}>0$ be the maximizer of $[s(\mu +ϵ)-ln{\Phi }_Y\left(s\right)]$, yielding $I\left(ϵ\right)=s^{*}(\mu +ϵ)-ln{\Phi }_Y\left(s^{*}\right)$. Then,

$$\mathbb{E}\left\{f\left(\sum _{i=1}^n{Y}_i\right)\right\}\ge \frac{f\left[({\mu }_Y+ϵ)n\right]}{{\mu }_Y+ϵ}\left[{\mu }_Y-e^{-nI\left(ϵ\right)}\frac{dln{\Phi }_Y\left(s\right)}{ds}{|}_{s=s^{*}}\right].$$

(26)

For large enough n, the second term in the square brackets becomes negligible, and the lower bound becomes arbitrarily close to $f\left[({\mu }_Y+ϵ)n\right]·{\mu }_Y/({\mu }_Y+ϵ)$. On the other hand, Jensen’s upper bound is $f\left({\mu }_{Y}n\right)$. In some cases, the difference is not very large, at least for asymptotic evaluations. For example, if $f\left(x\right)=ln(1+x)$, which is a frequently encountered concave function in information theory, $ln[1+n({\mu }_Y+ϵ)]\ge lnn+ln({\mu }_Y+ϵ)$, whereas $ln(1+n{\mu }_Y)\le lnn+ln({\mu }_Y+1/n)$, which are very close for large n and small $ϵ>0$. In Section 4.4, we will get back to the example of the logarithm of the sum independent random variables.

In the Chebychev–Cantelli approach, on the other hand, we have $a_c=\sqrt{n^2{\mu }_Y^2+n{\sigma }_Y^2}\sim n{\mu }_Y$ for large n. Thus, if we take $a=n({\mu }_Y+ϵ)>a_c$, we have

$$q_{Cheb-Cant}\left[n({\mu }_Y+ϵ)\right]=\frac{n{\sigma }_Y^2}{n{\sigma }_Y^2+n^2{ϵ}^2}=\frac{{\sigma }_Y^2}{{\sigma }_Y^2+n{ϵ}^2},$$

(27)

which tends to zero, but only at the rate of $1/n$, as opposed to the exponential decay in the Chernoff approach. Still, for large n, the main term of the bound becomes asymptotically tight, as before.

In spite of the superiority of the Chernoff approach relative to the Chebychev–Cantelli approach, as we demonstrated, one should keep in mind that there are also situations where the random variable X does not have an MGF (i.e., when the PDF of X has a heavy tail), yet it does have a mean and a variance. In such cases, the Chebychev–Cantelli approach is applicable while the Chernoff approach is not. However, even when the MGF exists, in certain cases, the calculation of the first and the second moment is easier than the calculation of the exponential moment.

To conclude this section, we summarize its main findings in the form of a theorem.

Theorem 1.

Under the conditions of Lemma 1,

$$\mathbb{E}\left\{f\left(X\right)\right\}\ge \underset{a>0}{sup}\left[\frac{\mu }{a}·f\left(a\right)+\left(1-\frac{\mu }{a}\right)·f\left(0\right)-\frac{f\left(a\right)-f\left(0\right)}{a}·q_{min}\left(a\right)\right],$$

(28)

where

$$q_{min}\left(a\right)=min\{q_{Chernoff}\left(a\right),{\tilde{q}}_{Chernoff}\left(a\right),q_{Cheb-Cant}\left(a\right)\}.$$

(29)

4. Examples

In this section, we demonstrate the lower bound in several information-theoretic application examples.

4.1. Example 1—Capacity of the Gaussian Channel with Random SNR

We begin with a simple example of a zero-mean, circularly symmetric complex Gaussian channel whose signal-to-noise ratio (SNR), Z, is a random variable (e.g., due to fading), known to both the transmitter and the receiver. The capacity is given by $C=\mathbb{E}\{ln(1+gZ\left)\right\}$, where g is a certain deterministic gain factor and the expectation is with respect to (w.r.t.) the randomness of Z. For simplicity, let us assume that Z is distributed exponentially, i.e.,

$$p_Z\left(z\right)=\theta e^{-\theta z},z\ge 0,$$

(30)

where the parameter $\theta >0$ is given. In this case, $f\left(x\right)=ln(1+gx)$, $\mu =1/\theta$ and $q\left(a\right)$ can be easily derived in closed form, to obtain

$$q\left(a\right)=\theta ·{\int }_a^{\infty }z{e}^{-\theta z}dz=\left(a+\frac{1}{\theta }\right)·e^{-\theta a}.$$

(31)

Consequently,

$$\begin{array}{ccc}C\hfill & \ge \hfill & \underset{a\ge 1/\theta }{sup}\frac{ln(1+ga)}{a}\left[\frac{1}{\theta }-\left(a+\frac{1}{\theta }\right)·e^{-a\theta }\right]\hfill \\ \hfill & =\hfill & \underset{s\ge 1}{sup}\left[\frac{1-(s+1)e^{-s}}{s}\right]·ln\left(1+\frac{gs}{\theta }\right),\hfill \end{array}$$

(32)

whereas the Jensen upper bound is $C\le ln(1+g/\theta )$. Figure 1 displays both the upper bound and the lower bound to C as functions of $\theta$. As can be seen, the gap between the bounds depends strongly on $\theta$. For large $\theta$, which is the case where the $p_Z$ becomes narrower and more concentrated, the gap between the bounds decays rapidly in the sense that not only does the difference shrink (which by itself is not surprising, since both bounds tend to zero), but their ratio also becomes closer to unity.

4.2. Example 2—Moments of the Number of Guesses in Randomized Guessing

Consider the problem of guessing the realization of a random variable, which takes on values in a finite alphabet, using a sequence of yes/no questions of the form “Is $U=u_1$?”, “Is $U=u_2$?”, etc., until a positive response is provided by a party that observes the actual realization of U (see, e.g., Ref. [12] whose Introduction includes an overview on this problem). Given a distribution of U, a commonly used performance metric for this problem is the ${\rho }^{\mathrm{th}}$ moment of the number of guesses until U is guessed successfully (for a given $\rho >0$). For guessing random vectors of length n, minimizing the moments of the number of guesses by different (deterministic or randomized) guessing strategies has several applications and motivations in information theory, such as sequential decoding, guessing passwords, etc., and it is also strongly related to lossless source coding. In this vector case, the moments of the number of guesses behave asymptotically exponentially in n. Here, we refer to randomized guessing strategies, rather than deterministic strategies.

Let the random variable U take on values in a finite alphabet $\mathcal{U}$. Consider a random guessing strategy where the guesser sequentially submits a sequence of independently drawn random guesses according to a certain probability distribution, $P(·)$, defined on $\mathcal{U}$. Randomized guessing strategies have the advantage that they can be used by multiple asynchronous agents, which submit their guesses concurrently (again, see [12] and references therein). Let $\mathit{u}\in {\mathcal{U}}^n$ be any realization of U, and let the guessing distribution, P, be given. The random number, X, of independent guesses until success has a geometric distribution:

$$\begin{array}{c}\hfill \mathrm{Pr}\{X=k\}={[1-P\left(\mathit{u}\right)]}^{k-1}·P\left(\mathit{u}\right),k=1,2,\dots \end{array}$$

(33)

and we are interested in a lower bound to $\mathbb{E}\left\{X^{\rho }\right\}$ for $\rho \in (0,1)$, which is the case where $f\left(x\right)=x^{\rho }$ is concave. Obviously, the condition $f\left(x\right)=x^{\rho }\ge 0=f\left(0\right)$ is met as well in this case. We next use the shorthand notation p instead of $P\left(\mathit{u}\right)$, and later we will think of this quantity as an exponential function of n, denoted $e^{-nE}$, where $E>0$ is some constant that depends on the type of u. We will also denote $q=1-p$. Then,

$$\begin{array}{ccc}\mathbb{E}\left\{X^{\rho }\right\}\hfill & \ge \hfill & \frac{\mu }{a}·a^{\rho }-\frac{a^{\rho }}{a}\underset{s\ge 0}{inf}\left\{e^{-sa}{\Phi }^{\prime }\left(s\right)\right\}\hfill \\ \hfill & =\hfill & a^{\rho -1}\left[\mu -\underset{s\ge 0}{inf}\left\{e^{-sa}{\Phi }^{\prime }\left(s\right)\right\}\right].\hfill \end{array}$$

(34)

Now, X is a geometric random variable with $\mu =1/p$, and

$$\begin{array}{cc}\hfill \Phi \left(s\right)& =\sum _{k=1}^{\infty }{e}^{ks}\mathrm{Pr}\{X=k\}\hfill \\ \hfill & =\frac{p{e}^s}{1-q{e}^s},s<ln\frac{1}{q}.\hfill \end{array}$$

(35)

which yields

$${\Phi }^{\prime }\left(s\right)=\frac{p{e}^s}{{(1-q{e}^s)}^2}.$$

(36)

Choosing $s=s^{*}=ln[(a-1)/aq]$, with $a=e^{n(E+ϵ)}$ (for some arbitrarily small $ϵ>0$), gives

$$\begin{array}{ccc}\hfill e^{-as}{\Phi }^{\prime }\left(s\right)& =& \frac{p}{q}·\frac{a(a-1)}{{(1-1/a)}^a}·q^a\hfill \\ & \le & \frac{e^{-nE}}{1-e^{-nE}}·\frac{e^{2nE}}{{\left(1-e^{-n(E+ϵ)}\right)}^{e^{n(E+ϵ)}}}·{(1-e^{-nE})}^{e^{n(E+ϵ)}}\hfill \\ & =& \frac{e^{nE}}{(1-e^{-nE}){\left(1-e^{-n(E+ϵ)}\right)}^{e^{n(E+ϵ)}}}·{(1-e^{-nE})}^{e^{n(E+ϵ)}}.\hfill \end{array}$$

(37)

Now, as $n\to \infty$, the denominator tends to $1/e$ and the factor ${(1-e^{-nE})}^{e^{n(E+ϵ)}}$ tends to zero double-exponentially, and so the entire expression goes to zero double-exponentially. Thus, the main term of the lower bound is

$$\mu a^{\rho -1}=e^{nE\rho -(1-\rho )ϵ},$$

(38)

which is roughly $e^{nE\rho }$ on the exponential scale. The Jensen upper bound is ${\left(\mathbb{E}\left\{X\right\}\right)}^{\rho }=1/p^{\rho }=e^{nE\rho }$, which is of the same exponential order if $ϵ$ is neglected.

4.3. Example 3—Moments of the Error in Parameter Estimation

Consider the estimation of a parameter $\theta$ from given observations. Suppose that we have an expression (or a lower bound), ${ϵ}^2\left(\theta \right)$, on the mean square error (MSE) of a certain estimator, and the question is how to obtain a lower bound to other moments of the estimation error, such as $\mathbb{E}\left\{\right|\widehat{\theta }-\theta {|}^q\}$ where $q<2$. Using the proposed approach, we have, in this example, $X={(\widehat{\theta }-\theta )}^2$ and $f\left(x\right)=x^{q/2}$. Consequently,

$$\begin{array}{ccc}\hfill \mathbb{E}\left\{\right|\widehat{\theta }-\theta {|}^q\}& =& \mathbb{E}\left\{|{(\widehat{\theta }-\theta )}^2{|}^{q/2}\right\}\hfill \\ & \ge & \frac{a^{q/2}}{a}·\left[\mathbb{E}\{{(\widehat{\theta }-\theta )}^2\}-\mathbb{E}\{{(\widehat{\theta }-\theta )}^2·\mathcal{I}[{(\widehat{\theta }-\theta )}^2\ge a]\}\right]\hfill \\ & \ge & a^{q/2-1}·\left[{ϵ}^2\left(\theta \right)-\mathbb{E}\{{(\widehat{\theta }-\theta )}^2·\mathcal{I}[{(\widehat{\theta }-\theta )}^2\ge a]\}\right].\hfill \end{array}$$

(39)

Now, if the given estimator has the additional large-deviations property to make the second term exponentially small for large n, then we can lower bound $\mathbb{E}\left\{\right|\widehat{\theta }-\theta {|}^q\}$ by an expression whose main term is ${\left[{ϵ}^2\left(\theta \right)\right]}^{q/2}$. For example, if $\theta$ is the mean of a Bernoulli source and $\widehat{\theta }$ is the empirical relative frequency, or $\theta$ is the mean of Gaussian observations, we can calculate the bound relatively easily. We will not pursue this example any further here.

4.4. Logarithms of Sums of Independent Random Variables

Referring to the discussion near the end of Section 3, where X is given by the sum of n independent random variables, we now focus on the special case where $f\left(x\right)=ln(1+x)$, which was also mentioned therein (see the discussion around Equation (26)). Here, we list several examples where this example is encountered in information theory applications.

4.4.1. Example 4—Universal Source Coding

Consider the evaluation of the expected code length associated with the universal lossless source code due to Krichevsky and Trofimov [13]. In a nutshell, this is a universal code for memoryless sources. In the binary case, at each time instant t, it sequentially assigns probabilities to the next binary symbol according to (a biased version of) the empirical distribution pertaining to the source data observed so far, $s_1,\dots ,s_t$. Specifically, consider the ideal code-length function (in nats),

$$\begin{array}{c}\hfill L\left(s^n\right)=-\sum _{t=0}^{n-1}lnQ\left(s_{t+1}\right|s_1,\dots ,s_t),\end{array}$$

(40)

where

$$\begin{array}{c}\hfill Q(s_{t+1}=s|s_1,\dots ,s_t)=\frac{N_t\left(s\right)+1}{t+2},\end{array}$$

(41)

and $N_t\left(s\right)$, $s\in \{0,1\}$ is the number of occurrences of the symbol s in $(s_1,\dots ,s_t)$. Therefore,

$$\begin{array}{ccc}\mathbb{E}\left\{L\left(S^n\right)\right\}\hfill & =\hfill & \sum _{t=0}^{n-1}ln(t+2)-{\displaystyle \sum _{t=0}^{n-1}}\mathbb{E}\{ln[N_t\left(S_{t+1}\right)+1]\}\hfill \\ \hfill & =\hfill & ln[(n+1)!]-{\displaystyle \sum _{t=0}^{n-1}}\mathbb{E}\{ln(1+{\displaystyle \sum _{i=0}^t}\mathcal{I}[S_i=S_{t+1}]\left)\right\}\hfill \\ \hfill & =\hfill & ln[(n+1)!]-p·{\displaystyle \sum _{t=0}^{n-1}}\mathbb{E}\{ln(1+{\displaystyle \sum _{i=0}^t}\mathcal{I}[S_i=1]\left)\right\}-\hfill \\ \hfill & \hfill & (1-p)·{\displaystyle \sum _{t=0}^{n-1}}\mathbb{E}\{ln(1+{\displaystyle \sum _{i=0}^t}\mathcal{I}[S_i=0]\left)\right\},\hfill \end{array}$$

(42)

where $\mathcal{I}[·]$ are indicator functions of the corresponding events and where p and $1-p$ are the probabilities of “1” and “0”, respectively. Now, in order to obtain an upper bound to $\mathbb{E}\left\{L\left(S^n\right)\right\}$, one can lower bound each of the terms, $\mathbb{E}\left\{ln\left(1+{\sum }_{i=0}^t\mathcal{I}[S_i=1]\right)\right\}$ and $\mathbb{E}\left\{ln\left(1+{\sum }_{i=0}^t\mathcal{I}[S_i=0]\right)\right\}$. This falls in the framework of logarithms of sums of i.i.d. random variables, where $\left\{U_i\right\}$ are binary.

4.4.2. Example 5—Ergodic Capacity of the Rayleigh SIMO Channel

Consider the single-input, multiple-output (SIMO) channel with L receiving antennas and assume that the channel transfer coefficients, $h_1,\dots ,h_L$, are independent, zero-mean, circularly symmetric complex Gaussian random variables with variances ${\sigma }_1^2,\dots ,{\sigma }_L^2$. The ergodic capacity (in nats per channel use) of the SIMO channel is given by

$$\begin{array}{c}\hfill C=\mathbb{E}\left\{ln\left(1+A\sum _{\ell =1}^L{\left|h_{\ell }\right|}^2\right)\right\}=\mathbb{E}\left\{ln\left(1+A\sum _{\ell =1}^L\left(f_{\ell }^2+g_{\ell }^2\right)\right)\right\},\end{array}$$

(43)

where $f_{\ell }=\mathrm{Re}\left\{h_{\ell }\right\}$, $g_{\ell }=\mathrm{Im}\left\{h_{\ell }\right\}$, and A is the SNR (see, e.g., Refs [14,15] and many references therein).

Once again, here $f\left(x\right)=ln(1+x)$, and now, $X=A{\sum }_{\ell =1}^L{\left|h_{\ell }\right|}^2$. Here, the transfer coefficients are assumed independent, but not identically distributed. Therefore, the means should be summed, and so should the variances, whereas the MGFs should be multiplied. Clearly, the case where they all share the same PDF is a special case.

4.4.3. Example 6—Differential Entropy of the Generalized Multivariate Cauchy Distribution

Let $(Y_1,\dots ,Y_n)$ be a multivariate Cauchy random vector, whose PDF is given by

$$\begin{array}{c}\hfill p(y_1,\dots ,y_n)=\frac{C_n}{{\left[1+{\sum }_{i=1}^n{y}_i^2\right]}^{(n+1)/2}}.\end{array}$$

(44)

Using the Laplace transform relation,

$$\begin{array}{c}\hfill \frac{1}{s^{(n+1)/2}}=\frac{1}{\Gamma \left(\right(n+1)/2)}{\int }_0^{\infty }{t}^{(n-1)/2}{e}^{-st}\mathrm{d}t,\mathrm{Re}\left(s\right)>0,\end{array}$$

(45)

f can be represented as a mixture of product measures:

$$\begin{array}{cc}\hfill f(y_1,\dots ,y_n)& =\frac{C_n}{{\left[1+{\sum }_{i=1}^n{y}_i^2\right]}^{(n+1)/2}}\hfill \\ \hfill & =\frac{C_n}{\Gamma \left(\right(n+1)/2)}{\int }_0^{\infty }{t}^{(n-1)/2}{e}^{-t}exp\left\{-t\sum _{i=1}^n{y}_i^2\right\}\mathrm{d}t.\hfill \end{array}$$

(46)

Defining

$$\begin{array}{c}\hfill Z\left(t\right)\stackrel{▵}{=}{\int }_{-\infty }^{\infty }{e}^{-t{y}^2}\mathrm{d}y=\sqrt{\frac{\pi }{t}},t>0,\end{array}$$

(47)

we get from (46),

$$\begin{array}{cc}\hfill 1& =\frac{C_n}{\Gamma \left(\right(n+1)/2)}{\int }_0^{\infty }{t}^{(n-1)/2}{e}^{-t}{\int }_{{\mathbb{R}}^n}exp\left\{-t\sum _{i=1}^n{y}_i^2\right\}\mathrm{d}{y}_1\dots \mathrm{d}{y}_n\mathrm{d}t\hfill \\ \hfill & =\frac{C_n}{\Gamma \left(\right(n+1)/2)}{\int }_0^{\infty }{t}^{(n-1)/2}{e}^{-t}{\left({\int }_{-\infty }^{\infty }{e}^{-t{y}^2}\mathrm{d}y\right)}^n\mathrm{d}t\hfill \\ \hfill & =\frac{C_n}{\Gamma \left(\right(n+1)/2)}{\int }_0^{\infty }{t}^{(n-1)/2}{e}^{-t}{\left(\frac{\pi }{t}\right)}^{n/2}\mathrm{d}t,\hfill \end{array}$$

(48)

and so,

$$\begin{array}{c}\hfill C_n=\frac{\Gamma \left(\right(n+1)/2)}{{\displaystyle {\int }_0^{\infty }{t}^{(n-1)/2}{e}^{-t}{\left(\frac{\pi }{t}\right)}^{n/2}\mathrm{d}t}}.\end{array}$$

(49)

The calculation of the differential entropy is associated with the evaluation of the expectation $\mathbb{E}\left\{ln\left[1+{\sum }_{i=1}^n{Y}_i^2\right]\right\}$. This falls within the framework of this application example, where $X={\sum }_{i=1}^n{Y}_i^2$. Here, $Y_i^2$ are not i.i.d., but they are distributed under a mixture of Gaussian i.i.d. distributions, as can be seen in (46). In particular,

$$\begin{array}{cc}\hfill p(y_1,\dots ,y_n)& =\frac{C_n}{{\left[1+{\sum }_{i=1}^n{y}_i^2\right]}^{(n+1)/2}}\hfill \\ \hfill & =\frac{C_n}{\Gamma \left(\right(n+1)/2)}{\int }_0^{\infty }{t}^{(n-1)/2}{e}^{-t}{\left(\frac{\pi }{t}\right)}^{n/2}\frac{exp\left\{-t{\sum }_{i=1}^n{y}_i^2\right\}}{{(\pi /t)}^{n/2}}\mathrm{d}t\hfill \\ \hfill & =\frac{C_n{\pi }^{n/2}}{\Gamma \left(\right(n+1)/2)}{\int }_0^{\infty }{t}^{-1/2}{e}^{-t}\frac{exp\left\{-t{\sum }_{i=1}^n{y}_i^2\right\}}{{(\pi /t)}^{n/2}}\mathrm{d}t\hfill \end{array}$$

(50)

Let us denote

$$w\left(t\right)=\frac{C_n{\pi }^{n/2}{e}^{-t}}{\Gamma ((n+1)/2)\sqrt{t}},t>0.$$

(51)

Then, $(Y_1,\dots ,Y_n)$ is governed by a mixture of i.i.d., zero-mean Gaussians of variance $1/\left(2t\right)$ with weight function $w\left(t\right)$, $t>0$. Therefore,

$$\mathbb{E}\left\{ln\left(1+\sum _{i=1}^n{Y}_i^2\right)\right\}={\int }_0^{\infty }dtw\left(t\right)\mathbb{E}\left\{ln\left(1+\sum _{i=1}^n{Y}_i^2\right)|t\right\},$$

(52)

and we can now lower bound $\mathbb{E}\{ln\left(1+{\sum }_{i=1}^n{Y}_i^2\right)|t\}$ for each $t>0$ and, finally, integrate the result (choosing possibly different a and s for every t). Following the discussion near the end of Section 3, the result will be roughly

$${\int }_0^{\infty }dtw\left(t\right)ln\left(1+\frac{n}{2t}\right),$$

(53)

which is in agreement with the Jensen upper bound.

5. Discussion

In this section, we discuss several additional aspects and observations regarding our main result, as well as modifications and suggestions for relaxing certain assumptions. We summarize our comments below.

• The maximization over a is not necessary. Our first comment is quite trivial but, nevertheless, it is important to mention at least as a reminder. The explicit maximization over the parameter a may not be trivial to carry out in most examples, but for certain purposes, it may not be necessary. One can select an arbitrary value of a and obtain a legitimate lower bound. In some cases, however, it is not too difficult to guess what could be a good choice of this value, as we saw in some of the examples of Section 4.

2.

Softening the assumption $f\left(x\right)\ge f\left(0\right)$. One may partially relax the assumption $f\left(x\right)\ge f\left(0\right)$ for all $x\ge 0$, and replace it with the softer assumption that there exists $\Delta \ge 0$, such that $f\left(x\right)+\Delta ·x\ge f\left(0\right)$ for all $x\ge 0$ (or more precisely, within the support of the PDF of X). By applying Lemma 1 and Theorem 1 to $f\left(x\right)+\Delta ·x$ and compensating for the term $\mathbb{E}\{\Delta ·X\}=\Delta \mu$, we can easily use exactly the same technique and obtain the modified lower bound,

$$\mathbb{E}\left\{f\left(X\right)\right\}\ge \underset{a>0}{sup}\left[\frac{\mu }{a}·f\left(a\right)+\left(1-\frac{\mu }{a}\right)f\left(0\right)-\left[\frac{f\left(a\right)-f\left(0\right)}{a}+\Delta \right]·q\left(a\right)\right],$$

(54)

and so the cost of this relaxation is the extra $\Delta$ in the last term. This means that the best choice of $\Delta$ is the smallest one for which $f\left(x\right)+\Delta x\ge f\left(0\right)$ for all $x\ge 0$, namely,

$$\Delta ={\Delta }^{*}\stackrel{▵}{=}\underset{x>0}{sup}\frac{f\left(0\right)-f\left(x\right)}{x}.$$

(55)

Nevertheless, if we can make $q\left(a\right)$ small by selecting a to be just slightly above $\mu$, this cost can be kept small. The results presented in Section 3 then become the special case of $\Delta =0$. Moreover, the artificially added (and subtracted) term, $\Delta x$, can also be replaced by a non-linear term, say, $g\left(x\right)$, provided that $f\left(x\right)+g\left(x\right)$ remains concave and $f\left(x\right)+g\left(x\right)\ge f\left(0\right)+g\left(0\right)$. This is useful, of course, only if $\mathbb{E}\left\{g\right(X\left)\right\}$ lends itself to an explicit calculation, for example, $g\left(x\right)=x^n$, or $g\left(x\right)=e^{ax}$.

3.

Convex functions. So far we have dealt with RJIs for concave functions. RJIs associated with expectations of convex functions (upper bounds) can be obtained in exactly the same manner, except that the signs are flipped. Specifically, by replacing f with $-f$, we have similar statements for convex functions: Let $f:{\mathbb{R}}^{+}\to \mathbb{R}$ be a convex function with $f\left(x\right)\le f\left(0\right)$ for every $x\ge 0$. Then,

$$\mathbb{E}\left\{f\left(X\right)\right\}\le \underset{a>0}{inf}\left[\frac{\mu }{a}·f\left(a\right)+\left(1-\frac{\mu }{a}\right)·f\left(0\right)+\frac{f\left(0\right)-f\left(a\right)}{a}·q\left(a\right)\right],$$

(56)

and, again, the assumption $f\left(x\right)\le f\left(0\right)$ can be softened in the same manner as described in item 2 above to obtain

$$\mathbb{E}\left\{f\left(X\right)\right\}\le \underset{a>0}{inf}\left[\frac{\mu }{a}·f\left(a\right)+\left(1-\frac{\mu }{a}\right)·f\left(0\right)+\left[\frac{f\left(0\right)-f\left(a\right)}{a}+\Delta \right]·q\left(a\right)\right].$$

(57)

4.

Functions that are neither convex nor concave. Using the same line of thought as in item 2 above, one can obtain upper and lower bounds to expectations of general functions, that are not necessarily convex or concave. Indeed, let $f:[a,b]\to \mathbb{R}$ be a real, continuous function that satisfies the following condition: $y>x$ implies $f\left(y\right)\le f\left(x\right)+\Delta (y-x)$ for all $x,y$. Assume also that f is bounded from below by a constant, $f_{min}$. Then, $f\left(x\right)+\Delta ·(b-x)-f_{min}$ is monotonically non-increasing and positive. Thus, for every $u\in [a,b]$,

$$f\left(x\right)+\Delta ·(b-x)-f_{min}\ge [f\left(u\right)+\Delta (b-u)-f_{min}]·\mathcal{I}\{x\le u\},$$

(58)

and so,

$$\mathbb{E}\left\{f\left(X\right)\right\}+\Delta ·(b-\mu )-f_{min}\ge [f\left(u\right)+\Delta (b-u)-f_{min}]·Pr\{X\le u\},$$

(59)

or

$$\begin{array}{ccc}\hfill \mathbb{E}\left\{f\right(X\left)\right\}& \ge & f_{min}-\Delta ·(b-\mu )+f\left(u\right)+\Delta ·(b-u)-f_{min}-\hfill \end{array}$$

$$\begin{array}{ccc}& & [f\left(u\right)+\Delta (b-u)-f_{min}]·Pr\{X\ge u\}\hfill \end{array}$$

(60)

$$\begin{array}{ccc}& \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}=& f\left(u\right)-\Delta ·(u-\mu )-[f\left(u\right)+\Delta (b-u)-f_{min}]·Pr\{X\ge u\}.\hfill \end{array}$$

(61)

so, finally,

$$\mathbb{E}\left\{f\left(X\right)\right\}\ge \underset{u\in [a,b]}{sup}\left\{f\left(u\right)-\Delta ·(u-\mu )-[f\left(u\right)+\Delta (b-u)-f_{min}]·\mathrm{Pr}\{X\ge u\}\right\},$$

with the understanding that we can further lower bound $\mathbb{E}\left\{f\right(X\left)\right\}$ by using any of the available upper bounds on $Pr\{X\ge u\}$ (Markov, Chebychev, Chebychev–Cantelli, Chernoff, Hoeffding, etc.). The choice depends on considerations of tightness and the calculability of the bound, as described before.

Likewise, if $f:[a,b]\to \mathbb{R}$ is a real, continuous function that satisfies the following condition: $y>x$ implies $f\left(y\right)\ge f\left(x\right)-\Delta (y-x)$ for all $x,y$, and is upper bounded by $f_{max}<\infty$, we obtain, in the same manner,

$$\mathbb{E}\left\{f\left(X\right)\right\}\le \underset{u\in [a,b]}{inf}\left\{f\left(u\right)+\Delta ·(u-\mu )-[f\left(u\right)-\Delta (b-u)-f_{max}]·Pr\{X\ge u\}\right\}.$$

(62)

6. Extension to Bivariate (and Multivariate) Concave Functions

We conclude this article by outlining a possible extension of our main results to bivariate concave functions, with the understanding that the main ideas also extend to multivariate functions of more than two variables. It should also be understood that a parallel upper bound for convex functions can easily be obtained by flipping the signs as before. For simplicity, we assume parallel assumptions to those of Lemma 1 and Theorem 1. In this section, we confine attention to the Chernoff approach, but the other approaches are valid as well.

For a pair of random variables $(X,Y)$, governed by a given joint PDF, we define the following functions:

$$\begin{array}{ccc}\hfill \Phi (s,t)& \stackrel{▵}{=}& \mathbb{E}\left\{e^{sX+tY}\right\}\hfill \end{array}$$

(63)

$$\begin{array}{ccc}\hfill {\Phi }_s\left(s\right)& \stackrel{▵}{=}& \frac{\partial \Phi (s,0)}{\partial s}=\mathbb{E}\left\{X{e}^{sX}\right\}\hfill \end{array}$$

(64)

$$\begin{array}{ccc}\hfill {\Phi }_t\left(t\right)& \stackrel{▵}{=}& \frac{\partial \Phi (0,t)}{\partial t}=\mathbb{E}\left\{Y{e}^{tY}\right\}\hfill \end{array}$$

(65)

$$\begin{array}{ccc}\hfill {\Phi }_{st}(s,t)& \stackrel{▵}{=}& \frac{{\partial }^2\Phi (s,t)}{\partial s\partial t}=\mathbb{E}\left\{XY{e}^{sX+tY}\right\}\hfill \end{array}$$

(66)

$$\begin{array}{ccc}\hfill q_1\left(a\right)& =& \underset{s\ge 0}{inf}\left\{e^{-sa}{\Phi }_s\left(s\right)\right\}\hfill \end{array}$$

(67)

$$\begin{array}{ccc}\hfill q_2\left(b\right)& =& \underset{t\ge 0}{inf}\left\{e^{-tb}{\Phi }_t\left(t\right)\right\}\hfill \end{array}$$

(68)

$$\begin{array}{ccc}\hfill q_3\left(a\right)& =& \underset{s\ge 0}{inf}\left\{e^{-sa}{\Phi }_{st}(s,0)\right\}\hfill \end{array}$$

(69)

$$\begin{array}{ccc}\hfill q_4\left(b\right)& =& \underset{t\ge 0}{inf}\left\{e^{-tb}{\Phi }_{st}(0,t)\right\}.\hfill \end{array}$$

(70)

We then have the following theorem.

Theorem 2.

Let $f:{\mathbb{R}}_{+}^2\to \mathbb{R}$ have the property $f(x,y)\ge f(x,0)$ and $f(x,y)\ge f(0,y)$ for all positive x and y in their domains, and assume that f is concave in x for fixed y, and concave in y for fixed x (but not necessarily jointly concave in both variables). Then,

$$\begin{array}{ccc}\hfill \mathbb{E}\left\{f\right(X,Y\left)\right\}& \ge & \underset{a>0}{sup}\underset{b>0}{sup}[\frac{f(a,b)-f(a,0)}{ab}[\mathbb{E}\left\{XY\right\}-q_3\left(a\right)-q_4\left(b\right)]+\hfill \\ & & \frac{f(a,0)}{a}[\mathbb{E}\left\{X\right\}-q_1\left(a\right)]-\frac{1}{a}\mathbb{E}\left\{X·f(0,\mathbb{E}\{Y\left|X\right\})\right\}+\hfill \\ & & \frac{f(0,b)-f(0,0)}{b}·[\mathbb{E}\left\{Y\right\}-q_2\left(b\right)]+f(0,0)].\hfill \end{array}$$

(71)

Likewise,

$$\begin{array}{ccc}\hfill \mathbb{E}\left\{f\right(X,Y\left)\right\}& \ge & \underset{a>0}{sup}\underset{b>0}{sup}[\frac{f(a,b)-f(0,b)}{ab}[\mathbb{E}\left\{XY\right\}-q_3\left(a\right)-q_4\left(b\right)]+\hfill \\ & & \frac{f(0,b)}{b}[\mathbb{E}\left\{Y\right\}-q_2\left(b\right)]-\frac{1}{b}\mathbb{E}\left\{Y·f\left(\mathbb{E}\right\{X\left|Y\right\},0)\right\}+\hfill \\ & & \frac{f(a,0)-f(0,0)}{a}·[\mathbb{E}\left\{X\right\}-q_1\left(a\right)]+f(0,0)].\hfill \end{array}$$

(72)

Proof of Theorem 2.

We begin with the simple identity,

$$\mathbb{E}\left\{f\right(X,Y\left)\right\}=\mathbb{E}\left\{\mathbb{E}\right\{f(X,Y)\left|Y\right\}\}.$$

(73)

Now, applying Theorem 1 to $f(X,y)$ (for fixed y) w.r.t. the conditional PDF of X given $Y=y$, we obtain:

$$\mathbb{E}\left\{f(X,y)\right|Y=y\}\ge \frac{f(a,y)-f(0,y)}{a}·\mathbb{E}\{X·\mathcal{I}\{X\le a\}|Y=y\}+f(0,y).$$

(74)

Thus,

$$\begin{array}{ccc}\hfill \mathbb{E}\left\{f\right(X,Y\left)\right\}& \stackrel{\left(\mathrm{a}\right)}{\ge }& \frac{1}{a}\mathbb{E}\left\{\left[f\right(a,Y)-f(0,Y\left)\right]X·\mathcal{I}\{X\le a\}\right\}+\mathbb{E}\left\{f(0,Y)\right\}\hfill \\ & \stackrel{\left(\mathrm{b}\right)}{\ge }& \frac{1}{a}\mathbb{E}\left\{\left[f\right(a,Y)X·\mathcal{I}\{X\le a\}\right\}-\frac{1}{a}\mathbb{E}\left\{\left[f\right(0,Y)X·\mathcal{I}\{X\le a\}\right\}+\hfill \end{array}$$

(75)

$$\begin{array}{ccc}& & \frac{f(0,b)-f(0,0)}{b}·(\mathbb{E}\left\{Y\right\}-\mathbb{E}\{Y·\mathcal{I}\{Y\ge b\}\})+f(0,0)\hfill \\ & \stackrel{\left(\mathrm{c}\right)}{\ge }& \frac{1}{a}\mathbb{E}\left\{\left[\frac{f(a,b)-f(a,0)}{b}·Y·\mathcal{I}\{Y\le b\}+f(a,0)\right]X·\mathcal{I}\{X\le a\}\right\}-\hfill \end{array}$$

(76)

$$\begin{array}{ccc}& & \frac{1}{a}\mathbb{E}\left\{\left[f\right(0,Y)X\right\}+\frac{f(0,b)-f(0,0)}{b}·(\mathbb{E}\left\{Y\right\}-\mathbb{E}\{Y·\mathcal{I}\{Y\ge b\}\})+f(0,0)\hfill \\ & \stackrel{\left(\mathrm{d}\right)}{=}& \frac{f(a,b)-f(a,0)}{ab}\mathbb{E}\{XY·\mathcal{I}\{X\le a\}·\mathcal{I}\{Y\le b\}\}+\frac{f(a,0)}{a}\mathbb{E}\{X·\mathcal{I}\{X\le a\}\}-\hfill \\ & & \frac{1}{a}\mathbb{E}\left\{\mathbb{E}\left\{f(0,Y)X|X\right\}\right\}+\frac{f(0,b)-f(0,0)}{b}·(\mathbb{E}\left\{Y\right\}-\mathbb{E}\{Y·\mathcal{I}\{Y\ge b\}\})+f(0,0)\hfill \\ & \stackrel{\left(\mathrm{e}\right)}{\ge }& \frac{f(a,b)-f(a,0)}{ab}\left[\mathbb{E}\left\{XY\right\}-\mathbb{E}\{XY·\mathcal{I}\{X\ge a\left\}\right\}-\mathbb{E}\{XY·\mathcal{I}\{Y\ge b\}\right]+\hfill \\ & & \frac{f(a,0)}{a}[\mathbb{E}\left\{X\right\}-\mathbb{E}\{X·\mathcal{I}\{X\ge a\}]-\frac{1}{a}\mathbb{E}\left\{X·f(0,\mathbb{E}\{Y\left|X\right\})\right\}+\hfill \\ & & \frac{f(0,b)-f(0,0)}{b}·(\mathbb{E}\left\{Y\right\}-\mathbb{E}\{Y·\mathcal{I}\{Y\ge b\}\})+f(0,0)\hfill \\ & \stackrel{\left(\mathrm{f}\right)}{=}& \frac{f(a,b)-f(a,0)}{ab}[\mathbb{E}\left\{XY\right\}-q_3\left(a\right)-q_4\left(b\right)]+\hfill \\ & & \frac{f(a,0)}{a}[\mathbb{E}\left\{X\right\}-q_1\left(a\right)]-\frac{1}{a}\mathbb{E}\left\{X·f(0,\mathbb{E}\{Y\left|X\right\})\right\}+\hfill \end{array}$$

(77)

$$\begin{array}{ccc}& & \frac{f(0,b)-f(0,0)}{b}·[\mathbb{E}\left\{Y\right\}-q_2\left(b\right)]+f(0,0),\hfill \end{array}$$

(78)

where (a) is by taking the expectation over Y on both sides of (74), (b) is by applying Theorem 1 to $f(0,Y)$, (c) is by applying Theorem 1 to $f(a,Y)$ (first, w.r.t. to the conditional PDF of Y given X, and then by averaging w.r.t. X), (d) is by rearranging terms, (e) is by the inequality $\mathcal{I}\{X\le a\}·\mathcal{I}\{Y\le b\}\ge 1-\mathcal{I}\{X\ge a\}-\mathcal{I}\{Y\ge b\}$, and (f) is by the definitions of the functions $q_1(·),\dots ,q_4(·)$. The second inequality in Theorem 2 is proved in the same manner by switching the roles of: (i) X and Y, (ii) a and b, (iii) $q_3\left(a\right)$ and $q_4\left(b\right)$, and (iv) $f(a,0)$ and $f(0,b)$. In other words, we first condition on X rather than on Y. This completes the proof of Theorem 2. □

The usefulness of this bound depends on the ability to calculate, or at least to upper bound the term, $\frac{1}{a}\mathbb{E}\left\{X·f(0,\mathbb{E}\{Y\left|X\right\})\right\}$. If the function $g\left(x\right)=x·f(0,\mathbb{E}\{Y|X=x\})$ happens to be a concave function, we can use Jensen’s inequality. If it is convex, we can apply a version of reverse Jensen for convex functions. If it is neither convex nor concave, we can use the method presented in the previous section to upper bound it. When X and Y are independent, this term simplifies significantly, as it becomes ${\mu }_X·f(0,{\mu }_Y)$, where ${\mu }_X$ and ${\mu }_Y$ are the means of X and Y, respectively.

Another case where the lower bound simplifies considerably is the case where $f(a,0)=f(0,b)=0$ for all a and b, as it becomes

$$\mathbb{E}\left\{f(X,Y)\right\}\ge \frac{f(a,b)}{ab}[\mathbb{E}\left\{XY\right\}-q_3\left(a\right)-q_4\left(b\right)].$$

(79)

We next provide two application examples where this is the case.

• Example 7—minimum between two sums of independent random variables. Let $X={\sum }_{i=1}^n{A}_i$ and $Y={\sum }_{j=1}^m{B}_j$, where both $\left\{A_i\right\}$ and $\left\{B_j\right\}$ are all non-negative, independent random variables. Obviously, ${\mu }_X={\sum }_{i=1}^n{\mu }_{A_i}$ and ${\mu }_Y={\sum }_{j=1}^m{\mu }_{B_j}$. Further, let $f(x,y)=min\{x,y\}$, which is concave in x for fixed y and vice versa. Then,

  $$\mathbb{E}\left\{min\left\{\sum _{i=1}^n{A}_i,\sum _{i=1}^m{B}_i\right\}\right\}\ge \frac{min\{a,b\}}{ab}\left[\sum _{i=1}^n\sum _{j=1}^m{\mu }_{A_i}{\mu }_{B_j}-q_3\left(a\right)-q_4\left(b\right)\right],$$

  (80)

  which is essentially $min\{{\mu }_X,{\mu }_Y\}$ for large n and m, provided that X and Y concentrate around their means, as discussed above. This example has a few applications, all of them are relevant in situations where there is a certain additive cost associated with a given task, there are two possible routes (or strategies), and the one with the smaller cost is chosen. For example, suppose we are compressing a realization, $u_1,\dots ,u_n$, of a random source vector, $U_1,\dots ,U_n$, and we have two side informations (available at both ends), $V_1,\dots ,V_n$ and $W_1,\dots ,W_n$, which are both conditionally independent noisy versions of $u_1,\dots ,u_n$, but for practical reasons, we use only one of them—the one for which code-length is shorter for the given realization (also adding a flag bit). In this case, $n=m$, $A_i=-logP\left(u_i\right|V_i)$, and $B_i=-logP\left(u_i\right|W_i)$, which are independent. As a second step, we have, of course, to take the expectation over $(U_1,\dots ,U_n)$. Other examples of costs might be prices, distances, waiting times, bit errors, etc.

• Example 8—channel capacity revisited. Here, we combine Example 1 (capacity of the Gaussian channel with random SNR) and Example 5 (ergodic capacity of the SIMO channel). Consider the expression

  $$C=\mathbb{E}\left\{log\left(1+Z·\sum _{\ell =1}^L{\left|h_{\ell }\right|}^2\right)\right\},$$

  (81)

  where, as in Example 1, $\left\{h_{\ell }\right\}$ are zero-mean, circularly symmetric, complex Gaussian random variables with variances $\left\{{\sigma }_{\ell }^2\right\}$, and as in Example 5, Z is an exponentially distributed random variable with parameter $\theta$, and independent of $\left\{h_{\ell }\right\}$. In principle, we could have treated this problem in the framework of univariate functions, using the concavity of $f\left(X\right)=ln(1+X)$, where the random variable X is defined as $X=Z·{\sum }_{\ell =1}^L{h}_{\ell }^2$. However, the calculation of the characteristic function of X is not convenient to analyze since it is a product of two random variables. Instead, we treat it as a bivariate function, $f(X,Y)=ln(1+XY)$, where $X=Z$ and $Y={\sum }_{\ell =1}^L{h}_{\ell }^2$. Clearly, f is concave in each one of its arguments when the other one is kept fixed. We then have

  $$\begin{array}{ccc}\hfill C& \ge & \frac{ln(1+ab)}{ab}\left[\frac{1}{\theta }·\sum _{\ell =1}^L{\sigma }_{\ell }^2-q_3\left(a\right)-q_4\left(b\right)\right].\hfill \end{array}$$

  (82)

As in Example 5, instead of the Chernoff-like bound, $q_3\left(a\right)$, we have the exact expression

$$\begin{array}{ccc}\hfill \mathbb{E}\{XY·\mathcal{I}[X\ge a\left]\right\}& =& \mathbb{E}\left\{Y\right\}·\mathbb{E}\{X·\mathcal{I}[X\ge a\left]\right\}\hfill \end{array}$$

(83)

$$\begin{array}{ccc}& =& \left(\sum _{\ell =1}^L{\sigma }_{\ell }^2\right)·\left(a+\frac{1}{\theta }\right)·e^{-\theta a}.\hfill \end{array}$$

(84)

The other term can be readily bounded using the Chernoff approach.