Gaussian Optimality for Derivatives of Differential Entropy Using Linear Matrix Inequalities ^†

Abstract

Let Z be a standard Gaussian random variable, X be independent of Z, and t be a strictly positive scalar. For the derivatives in t of the differential entropy of $X+\sqrt{t}Z$, McKean noticed that Gaussian X achieves the extreme for the first and second derivatives, among distributions with a fixed variance, and he conjectured that this holds for general orders of derivatives. This conjecture implies that the signs of the derivatives alternate. Recently, Cheng and Geng proved that this alternation holds for the first four orders. In this work, we employ the technique of linear matrix inequalities to show that: firstly, Cheng and Geng’s method may not generalize to higher orders; secondly, when the probability density function of $X+\sqrt{t}Z$ is log-concave, McKean’s conjecture holds for orders up to at least five. As a corollary, we also recover Toscani’s result on the sign of the third derivative of the entropy power of $X+\sqrt{t}Z$, using a much simpler argument.

1. Introduction

For a general continuous random variable X with probability density function $f\left(x\right)$, its differential entropy is defined as

$$\begin{array}{c}\hfill h\left(X\right)=-{\int }_{-\infty }^{+\infty }f\left(x\right)lnf\left(x\right)dx,\end{array}$$

given that the above integral exists. In [1], Shannon considered the entropy power $N\left(X\right)=\frac{1}{2\pi e}{e}^{2h\left(X\right)}$, and introduced the celebrated Entropy Power Inequality (EPI):

$$\begin{array}{c}\hfill e^{2h\left(X+Y\right)}\ge e^{2h\left(X\right)}+e^{2h\left(Y\right)},\end{array}$$

where X and Y are independent, and equality holds if and only if X and Y are Gaussian. This inequality is nontrivial and was rigorously proved later by Stam [2].

Remark 1.

This is the full version of a conference paper submitted to ISIT 2018 [3].

There have been numerous generalizations of the EPI. In [4], Costa considered the case where X is perturbed by an independent standard Gaussian Z, and showed that $N(X+\sqrt{t}Z)$ is concave in t for $t>0$:

$$\begin{array}{c}\hfill \frac{d^2}{d{t}^2}N(X+\sqrt{t}Z)\le 0,t>0.\end{array}$$

Toscani [5] further showed that $\frac{d^3}{d{t}^3}N(X+\sqrt{t}Z)\ge 0$, under the condition that the probability density function of $X+\sqrt{t}Z$ is log-concave.

Later, Villani [6] simplified Costa’s proof by directly studying the second derivative in t of the differential entropy instead of the second derivative of the entropy power. In the proof, it was noticed [6,7,8] that the signs of the first two derivatives of $h(X+\sqrt{t}Z)$ alternate. Along this line, Cheng and Geng [9] showed that this alternation holds for the first four derivatives, and they made the following conjecture that the alternation is true for general orders of derivatives.

Conjecture 1

([9]). The derivatives of the differential entropy $h(X+\sqrt{t}Z)$ satisfy ${(-1)}^{n-1}\times \frac{d^n}{d{t}^n}h(X+\sqrt{t}Z)\ge 0$ for $t>0$ and $n\ge 1$.

According to Equation (3) in Lemma 2 and the comments, $2\times \frac{d}{dt}h(X+\sqrt{t}Z)$ is the Fisher information $J(X+\sqrt{t}Z)$. The above conjecture is equivalent to hypothesizing that the Fisher information of $X+\sqrt{t}Z$ is completely monotone, thus admitting a very simple characterization using the Laplace Transform [10]: there exists a finite Borel measure $\mu (·)$ such that

$$\begin{array}{c}\hfill J(X+\sqrt{t}Z)={\int }_0^{+\infty }{e}^{-\lambda t}\mu \left(d\lambda \right).\end{array}$$

(1)

Back in 1966, McKean [7] also studied the derivatives in t of $h(X+\sqrt{t}Z)$, and noticed that Gaussian X achieves the minimum of $\frac{d}{dt}h(X+\sqrt{t}Z)$ and $-\frac{d^2}{d{t}^2}h(X+\sqrt{t}Z)$, subject to $\mathrm{Var}\left(X\right)={\sigma }^2$. Then, McKean implicitly made the following conjecture that Gaussian optimality holds generally:

Conjecture 2

([7]). Subject to $\mathrm{Var}\left(X\right)={\sigma }^2$, Gaussian X with variance ${\sigma }^2$ achieves the minimum of ${(-1)}^{n-1}\times \frac{d^n}{d{t}^n}h(X+\sqrt{t}Z)$ for $t>0$ and $n\ge 1$.

Notice that, if $X_G$ is Gaussian with variance ${\sigma }^2$, by routine calculation,

$$\begin{array}{cc}\hfill 2h(X_G+\sqrt{t}Z)& =ln2\pi e({\sigma }^2+t),\hfill \\ \hfill {(-1)}^{n-1}\times 2\frac{d^n}{d{t}^n}h(X_G+\sqrt{t}Z)& =\left(n-1\right)!\times {({\sigma }^2+t)}^{-n}>0.\hfill \end{array}$$

(2)

Hence, McKean’s conjecture implies the one by Cheng and Geng.

Compared with the progress made by Cheng and Geng [9] on Conjecture 1, there has been little progress on Conjecture 2. Most of the existing results are on the second derivative of the differential entropy (or the mutual information), and on generalizing the EPI to other settings. For example: Guo et al. [11] represents the derivatives in the signal-to-noise ratio of the mutual information in terms of the minimum mean-square estimation error, building on de Bruijn’s identity [2]; Wibisono and Jog [12] study the mutual information along the density flow defined by the heat equation and show that it is a convex function of time if the initial distribution is log-concave; Wang and Madiman [13] recover the proof of the EPI via rearrangements; Courtade [14] generalizes Costa’s EPI to non-Gaussian additive perturbations; and König and Smith [15] propose a quantum version of the EPI.

In this paper, we work on Conjecture 2. The main results are to show that Conjecture 2 holds for higher orders up to at least five under the log-concavity condition, and the introduction of the technique of linear matrix inequalities.

The paper is organized as follows: in Section 2, we obtain the formulae for the derivatives of the differential entropy $h(X+\sqrt{t}Z)$ (Theorem 1) and show that McKean’s conjecture holds for higher orders up to at least five under the log-concavity condition (Corollary 1). As a corollary, we recover Toscani’s result [5] on the third derivative of the entropy power, using the Cauchy–Schwartz inequality, which is much simpler. In Section 3, we introduce the linear matrix inequality approach, and transform the above two conjectures to the feasibility check of semidefinite programming problems. With this approach, we can easily obtain the coefficients in Theorem 1. Then, we show that the direct generalization of the method by Cheng and Geng might not work for orders higher than four for proving Conjecture 1. In Section 4, we prove the main theorem of Section 2.

2. Main Results

We first introduce the notation that is used throughout this paper. When the functions are single-variate, we use $\frac{d·}{d·}$ for its derivative. For the multi-variate cases, we use $\frac{\partial ·}{\partial ·}$ for the partial derivative. To simplify the notation, for the derivatives of a general single-variate function $g\left(y\right)$, we also use $g^{\prime }\left(y\right)$, $g^{″}\left(y\right)$ and $g^{‴}\left(y\right)$ to represent the first, second and third derivatives, respectively; and $g^{\left(n\right)}\left(y\right)$ denotes the n-th derivative for $n\ge 1$.

In the rest of the paper, let Z be a standard Gaussian random variable, and X be independent of Z. Denote

$$Y_t:=X+\sqrt{t}Z,t>0.$$

According to [4,16], $Y_t$ has nice properties: The probability density function $f(y,t)$ of $Y_t$ exists, is strictly positive and infinitely differentiable; The differential entropy $h\left(Y_t\right)$ exists. Denote

$$\begin{array}{cc}\hfill f_n:=& \frac{{\partial }^n}{\partial y^n}f(y,t),\hfill \\ \hfill T_n:=& \frac{{\partial }^n}{\partial y^n}lnf(y,t),n=0,1,2,\dots ,\hfill \end{array}$$

where it is understood that $f_n$ and $T_n$ are functions of $(y,t)$. We also present some properties of $f(y,t)$ in the following lemma. The proof can be found in, say, [2,16] and Propositions 1 and 2 in [9].

Lemma 1.

For $t>0$, the probability density function $f(y,t)$ satisfies the following properties:

(1) 

The heat equation holds: $\frac{\partial }{\partial t}f=\frac{1}{2}\frac{{\partial }^2}{\partial y^2}f$.

(2) 

${lim}_{\left|y\right|\to \infty }{f}_n=0$, $\forall n\ge 0$, $\forall t>0$.

(3) 

The expectation of the product of the $T_i$, $\mathrm{E}\left[{\prod }_i{T}_i\right]$ exists, and ${lim}_{\left|y\right|\to \infty }f{\prod }_i{T}_i=0$, $\forall t>0$.

In Lemma 1, part (3), in writing $\mathrm{E}\left[{\prod }_i{T}_i\right],$ we think of each $T_i$ as a function of $(Y_t,t)$.

Notice that, given X and Z, the differential entropy $h(X+\sqrt{t}Z)$ is a function of t. The formulae for the first and second derivatives of $h(X+\sqrt{t}Z)$ are presented in the following lemma. According to Stam [2], the first equality is due to de Bruijn, and the right-hand side is actually the Fisher information (page 671 of [17]); the second one is due to McKean [7], Toscani [8] and Villani [6]; the Gaussian optimality is due to McKean [7].

Lemma 2.

For the first and second derivatives of the differential entropy $h(X+\sqrt{t}Z)$, the following expressions hold for $t>0$:

$$\begin{array}{cc}\hfill 2h^{\prime }(X+\sqrt{t}Z)& =\mathrm{E}\left[{\left(\frac{f_1}{f}\right)}^2\right],\hfill \end{array}$$

(3)

$$\begin{array}{cc}\hfill -2h^{″}(X+\sqrt{t}Z)& =\mathrm{E}\left[{\left(\frac{f_2}{f}-\frac{f_1^2}{f^2}\right)}^2\right].\hfill \end{array}$$

(4)

Subject to $\mathrm{Var}X={\sigma }^2$, Gaussian X with variance ${\sigma }^2$ minimizes $h^{\prime }(X+\sqrt{t}Z)$ and $-h^{″}(X+\sqrt{t}Z)$.

By standard manipulations, one has

$$\begin{array}{c}\hfill T_1=\frac{f_1}{f},T_2=\frac{f_2}{f}-\frac{f_1^2}{f^2}.\end{array}$$

(5)

Thus, it is straightforward to rewrite the derivatives as

$$\begin{array}{cc}\hfill 2h^{\prime }(X+\sqrt{t}Z)& =\mathrm{E}\left[T_1^2\right],\hfill \end{array}$$

(6)

$$\begin{array}{cc}\hfill -2h^{″}(X+\sqrt{t}Z)& =\mathrm{E}\left[T_2^2\right].\hfill \end{array}$$

(7)

For the third and fourth derivatives, one can refer to Theorems 1 and 2 in [9], where they were represented by the $f_i$. Notice that these representations are not unique, and the ones in [9] are sufficient for identifying the signs. Instead, in Theorem 1, we use the $T_i$, and this will facilitate our proof of the Gaussian optimality in Corollary 1.

Theorem 1.

For $t>0$, the derivatives of the differential entropy $h(X+\sqrt{t}Z)$ can be expressed as:

$$\begin{array}{cc}\hfill 2h^{\left(3\right)}(X+\sqrt{t}Z)& =\mathrm{E}\left[T_3^2-2T_2^3\right],\hfill \end{array}$$

(8)

$$\begin{array}{cc}\hfill -2h^{\left(4\right)}(X+\sqrt{t}Z)& =\mathrm{E}\left[T_4^2+6T_2^4-12T_3^2{T}_2\right],\hfill \end{array}$$

(9)

$$\begin{array}{cc}\hfill 2h^{\left(5\right)}(X+\sqrt{t}Z)& =\mathrm{E}\left[T_5^2-24T_2^5-8T_4^2{T}_2-6T_3^2{T}_2{T}_1^2+12T_5{T}_3{T}_2+114T_3^2{T}_2^2\right].\hfill \end{array}$$

(10)

The proof to this theorem is left to Section 4. The existence of such expressions and how to obtain the coefficients are left to Section 3, where the method of linear matrix inequalities is introduced.

Log-Concave Case

Lemma 2 already ensures the optimality of Gaussians, subject to $\mathrm{Var}\left(X\right)={\sigma }^2$, for the first and second derivatives. For higher ones, we do not know if we can show the optimality based on the expressions in Theorem 1. Here, we impose the constraint of log-concavity on $f(y,t)$ and summarize the results in Corollaries 1–3.

A nonnegative function $f(·)$ is logarithmically concave (or log-concave for short) if its domain is convex and it satisfies the inequality

$$\begin{array}{c}\hfill f\left(\theta x+(1-\theta )y\right)\ge f{\left(x\right)}^{\theta }f{\left(y\right)}^{1-\theta }\end{array}$$

for all $x,y$ in the domain and $0<\theta <1$. If f is strictly positive, this is equivalent to saying that the logarithm of the function is concave (Section 2.5 of [18]). In our case, assuming that $f(y,t)$ is log-concave in y is equivalent to $T_2\le 0$.

Examples of log-concave distributions include the Gaussian, exponential, Laplace, and the Gamma with parameter larger than one. Notice that, if the probability density function of X is log-concave, then so is that of $X+\sqrt{t}Z$ (Section 3.5.2 of [18]).

Corollary 1.

If the probability density function of $X+\sqrt{t}Z$ is log-concave, then, subject to $\mathrm{Var}\left(X\right)={\sigma }^2$, Gaussian X with variance ${\sigma }^2$ achieves the minimum of ${(-1)}^{n-1}{h}^{\left(n\right)}(X+\sqrt{t}Z)$ for $t>0$ and $3\le n\le 5$.

Proof. 

Let $X_G$ be Gaussian with mean $\mu$ and variance ${\sigma }^2$. The probability density function of $Y_G:=X_G+\sqrt{t}Z$ is

$$f\left(y_G,t\right)=\frac{1}{\sqrt{2\pi \left({\sigma }^2+t\right)}}\times \exp \{-\frac{{\left(y_G-\mu \right)}^2}{2\left({\sigma }^2+t\right)}\}.$$

The key observation is that the second derivative of the logarithm in the Gaussian case is

$$\begin{array}{c}\hfill T_{2,G}:=\frac{{\partial }^2}{\partial y_G^2}lnf(y_G,t)=-{({\sigma }^2+t)}^{-1}.\end{array}$$

Hence, from Equation (2), the derivatives of the differential entropy in the Gaussian case are

$${(-1)}^{n-1}\times 2h^{\left(n\right)}(X_G+\sqrt{t}Z)=(n-1)!\times {({\sigma }^2+t)}^{-n}=(n-1)!\times \mathrm{E}{\left[-T_{2,G}\right]}^n.$$

Now, if one can show the following chain of inequalities:

$$\begin{array}{c}\hfill {(-1)}^{n-1}\times 2h^{\left(n\right)}(X+\sqrt{t}Z)\stackrel{\left(a\right)}{\ge }(n-1)!\times \mathrm{E}\left[{\left(-T_2\right)}^n\right]\stackrel{\left(b\right)}{\ge }(n-1)!\times \mathrm{E}{\left[-T_2\right]}^n\stackrel{\left(c\right)}{\ge }(n-1)!\times \mathrm{E}{\left[-T_{2,G}\right]}^n,\end{array}$$

(11)

then one is done.

For inequality $\left(b\right)$, the log-concavity condition, namely $T_2\le 0$, suffices.

For inequality $\left(c\right)$, it suffices to show that $E[-T_2]\ge \mathrm{E}[-T_{2,G}]\ge 0$. This can be proved using Lemma 2: Notice that

$$\mathrm{E}\left[\frac{f_2}{f}\right]={\int }_{-\infty }^{+\infty }{f}_2(y,t)dy={\int }_{-\infty }^{+\infty }d{f}_1(y,t)=f_1(y,t)\left|{}_{-\infty }^{+\infty }\right.=0-0,$$

where the last equality is due to Lemma 1.

Now, from Equation (5),

$$\begin{array}{cc}\hfill \mathrm{E}\left[-T_2\right]& =\mathrm{E}\left[-\frac{f_2}{f}+\frac{f_1^2}{f^2}\right]=\mathrm{E}\left[\frac{f_1^2}{f^2}\right].\hfill \end{array}$$

(12)

Combining this with Lemma 2, one has

$$\mathrm{E}\left[-T_2\right]=2h^{\prime }(X+\sqrt{t}Z)\ge 2h^{\prime }(X_G+\sqrt{t}Z)=E\left[-T_{2,G}\right].$$

This part is finished by noticing that $\mathrm{E}\left[-T_{2,G}\right]>0$ from Equation (2).

For inequality (a), we show each case of n using Theorem 1 and the condition $T_2\le 0$. For $n=3$,

$$2h^{\left(3\right)}(X+\sqrt{t}Z)=\mathrm{E}\left[T_3^2-2T_2^3\right]\ge \mathrm{E}\left[-2T_2^3\right]=(n-1)!\times \mathrm{E}\left[{\left(-T_2\right)}^n\right],n=3.$$

For $n=4$,

$$-2h^{\left(4\right)}(X+\sqrt{t}Z)=\mathrm{E}\left[T_4^2+6T_2^4-12T_3^2{T}_2\right]\ge \mathrm{E}\left[6T_2^4\right]=(n-1)!\times \mathrm{E}\left[{(-T_2)}^n\right],n=4,$$

where the inequality is due to $T_2\le 0$, thus $\mathrm{E}[-12T_3^2{T}_2]\ge 0$. For $n=5$,

$$\begin{array}{cc}\hfill 2h^{\left(5\right)}(X+\sqrt{t}Z)& =\mathrm{E}\left[T_5^2-24T_2^5-8T_4^2{T}_2-6T_3^2{T}_2{T}_1^2+12T_5{T}_3{T}_2+114T_3^2{T}_2^2\right]\hfill \\ & =\mathrm{E}\left[{\left(T_5+6T_3{T}_2\right)}^2-24T_2^5-8T_4^2{T}_2-6T_3^2{T}_2{T}_1^2+78T_3^2{T}_2^2\right]\hfill \\ & \ge \mathrm{E}\left[-24T_2^5\right]\hfill \\ & =(n-1)!\times \mathrm{E}\left[{(-T_2)}^n\right],n=5.\hfill \end{array}$$

Now, the proof is finished. ☐

The following corollary deals with the fifth-order case in [9], under the log-concavity assumption. The proof follows directly from Corollary 1 and Equation (2).

Corollary 2.

If the probability density function of $X+\sqrt{t}Z$ is log-concave, then the fifth derivative of the differential entropy is strictly positive: $h^{\left(5\right)}(X+\sqrt{t}Z)>0$ for $t>0$.

Regarding the entropy power, it is already known that $N^{\prime }(X+\sqrt{t}Z)\ge 0$ from the connection with Fisher information, and $N^{″}(X+\sqrt{t}Z)\le 0$ according to [4]. For the third derivative, Toscani showed that $N^{\left(3\right)}(X+\sqrt{t}Z)\ge 0$, under the log-concavity assumption. Here, we simplify Toscani’s proof, using a Cauchy–Schwartz argument.

Corollary 3.

If the probability density function of $X+\sqrt{t}Z$ is log-concave, then the third derivative of the entropy power is nonnegative: $N^{\left(3\right)}(X+\sqrt{t}Z)\ge 0$ for $t>0$.

Proof. 

For brevity, let $h^{\prime }:=h^{\prime }(X+\sqrt{t}Z)$, and, similarly, we omit the arguments for higher orders. Routine manipulations yield that

$$\begin{array}{c}\hfill N^{\left(3\right)}(X+\sqrt{t}Z)=\frac{d^3}{d{t}^3}\frac{1}{2\pi e}{e}^{2h(X+\sqrt{t}Z)}=\frac{1}{2\pi e}{e}^{2h(X+\sqrt{t}Z)}\left[{\left(2h^{\prime }\right)}^3+3\times 2h^{\prime }\times 2h^{″}+2h^{‴}\right].\end{array}$$

Thus, it suffices to show ${\left(2h^{\prime }\right)}^3+3\times 2h^{\prime }\times 2h^{″}+2h^{‴}\ge 0$. First, we express $h^{\prime }$, $h^{″}$ and $h^{‴}$ in the form of the $T_i$: according to Lemma 2 and Equation (12), $2h^{\prime }=\mathrm{E}[-T_2]$; from Equation (7), $2h^{″}=-\mathrm{E}\left[T_2^2\right]$; copying from Equation (8), $2h^{‴}=\mathrm{E}[T_3^2-2T_2^3]$.

Also notice that, from Lemma 2, $2h^{\prime }(X+\sqrt{t}Z)\ge 2h^{\prime }(X_G+\sqrt{t}Z)={({\sigma }_X^2+t)}^{-1}>0$. Hence, $\mathrm{E}[-T_2]>0$ (it cannot be zero). Now, under the log-concavity condition, namely $T_2\le 0$, from the Cauchy–Schwartz inequality for random variables, we have:

$$\begin{array}{c}\hfill \mathrm{E}[-T_2]\mathrm{E}[-T_2^3]=\mathrm{E}\left[{\sqrt{-T_2}}^2\right]\mathrm{E}\left[{\sqrt{-T_2^3}}^2\right]\ge \mathrm{E}{\left[\sqrt{-T_2}\sqrt{-T_2^3}\right]}^2=\mathrm{E}{\left[T_2^2\right]}^2.\end{array}$$

Thus, we have

$$\begin{array}{cc}\hfill {\left(2h^{\prime }\right)}^3+3\times 2h^{\prime }\times 2h^{″}+2h^{‴}& =\mathrm{E}{[-T_2]}^3-3\times \mathrm{E}[-T_2]\times \mathrm{E}\left[T_2^2\right]+\mathrm{E}[T_3^2-2T_2^3]\hfill \\ & \ge \mathrm{E}{[-T_2]}^3-3\times \mathrm{E}[-T_2]\times \mathrm{E}\left[T_2^2\right]+\mathrm{E}[-2T_2^3]\hfill \\ & ={\left(E[-T_2]\right)}^{-1}\left(\mathrm{E}{[-T_2]}^4-3\times \mathrm{E}{[-T_2]}^2\times \mathrm{E}\left[T_2^2\right]+2\mathrm{E}[-T_2]\mathrm{E}[-T_2^3]\right)\hfill \\ & \ge {\left(E[-T_2]\right)}^{-1}\left(\mathrm{E}{[-T_2]}^4-3\times \mathrm{E}{[-T_2]}^2\times \mathrm{E}\left[T_2^2\right]+2\mathrm{E}{\left[T_2^2\right]}^2\right)\hfill \\ & ={\left(E[-T_2]\right)}^{-1}\left(\mathrm{E}\left[T_2^2\right]-\mathrm{E}{[-T_2]}^2\right)\left(2\mathrm{E}\left[T_2^2\right]-\mathrm{E}{[-T_2]}^2\right).\hfill \end{array}$$

The proof is finished by noticing that $\mathrm{E}\left[T_2^2\right]\ge \mathrm{E}{[-T_2]}^2\ge 0$, which implies that the right-hand side is nonnegative. ☐

3. Linear Matrix Inequalities

In this section, we introduce the method of linear matrix inequalities (LMI), and transform the proof of Conjectures 1 and 2 to the feasibility problem of LMI. This transformation also enables us to find the right coefficients in Theorem 1.

Recall that, in [9], the authors first obtained the fourth derivative as the following (Equation (27) in [9])

$$\begin{array}{c}\hfill 2h^{\left(4\right)}(X+\sqrt{t}Z)=\mathrm{E}\left[-\frac{f_4^2}{f^2}-\frac{4f_2{f}_3^2}{f^3}+\frac{4f_1^2{f}_3^2}{f^4}-\frac{3f_2^4}{f^4}+\frac{24f_1^2{f}_2^3}{f^5}-\frac{36f_1^4{f}_2^2}{f^6}+\frac{90f_1^8}{7f^8}\right].\end{array}$$

(13)

Then, with some equalities (from integration by parts), they showed this derivative can be expressed as the negative of a sum of squares (Theorem 2 in [9]):

$$\begin{array}{cc}\hfill 2h^{\left(4\right)}(X+\sqrt{t}Z)& =-\mathrm{E}\left[{\left(\frac{f_4}{f}-\frac{6}{5}\frac{f_1{f}_3}{f^2}-\frac{7}{10}\frac{f_2^2}{f^2}+\frac{8}{5}\frac{f_1^2{f}_2}{f^3}-\frac{1}{2}\frac{f_1^4}{f^4}\right)}^2+{\left(\frac{2}{5}\frac{f_1{f}_3}{f^2}-\frac{1}{3}\frac{f_1^2{f}_2}{f^3}+\frac{9}{100}\frac{f_1^4}{f^4}\right)}^2\right.\hfill \\ & \left.+{\left(-\frac{4}{100}\frac{f_1^2{f}_2}{f^3}+\frac{4}{100}\frac{f_1^4}{f^4}\right)}^2+\frac{1}{300}\frac{f_2^4}{f^4}+\frac{56}{\mathrm{90,000}}\frac{f_1^4{f}_2^2}{f^6}+\frac{13}{\mathrm{70,000}}\frac{f_1^8}{f^8}\right].\hfill \end{array}$$

(14)

Hence, the fourth derivative is nonpositive. The sum of squares has a natural connection with positive semidefinite matrices. The right-hand side of Equation (14) can be written as $-\mathrm{E}\left[u^{T}Fu\right]$, where u is the column vector with coordinates $u=(f_4/f,f_1{f}_3/f^2,f_2^2/f^2,f_1^2{f}_2/f^3,f_1^4/f^4)$ and F is a positive semidefinite matrix. Thus, the method in [9] is actually to verify the existence of a suitable positive semidefinite matrix F. This can be cast as the feasibility of a linear matrix inequality.

A linear matrix inequality (Chapter 2 of [18]) has the form

$$\begin{array}{c}\hfill F(x,y):=F_0+\sum _{i=1}^I{x}_i{F}_i+\sum _{j=1}^J{y}_j{G}_j⪰0,\end{array}$$

(15)

where the $m\times m$ symmetric matrices $F_0$, $F_i$, $G_j$, $i=1,\dots ,I$, $j=1,\dots ,J$ are given, variables $x_i$ are real and $y_j^{\prime }$ are nonnegative, and the notation $F(x,y)⪰0$ means $F(x,y)$ is positive semidefinite. The feasibility problem refers to identifying if there exists a set of $x_i$ and $y_j$ such that $F(x,y)$ is positive semidefinite.

To reformulate the method used by Cheng and Geng [9] as an LMI feasibility problem, using the fourth derivative as an illustrative example, the main idea is: first, transform the original expression of the derivative to the form

$$-2h^{\left(4\right)}(X+\sqrt{t}Z)=\mathrm{E}\left[u^T{F}_{0}u\right].$$

Then, transform the equalities resulting from integration by parts to the form

$$0=\mathrm{E}\left[u^T{F}_{i}u\right],i=1,2,\dots ,I.$$

Finally, try to find a set of variables $x_i$ such that $F_0+{\sum }_i{x}_i{F}_i⪰0$, which is sufficient to show that

$$-2h^{\left(4\right)}(X+\sqrt{t}Z)=\mathrm{E}\left[u^T{F}_{0}u\right]=\mathrm{E}[u^T(F_0+\sum _i{x}_i{F}_i)u]\ge 0.$$

One can notice that there is no matrix $G_j$ in the above statement. This is mainly because only equalities were available in [9]. When one imposes inequality constraints, for example $T_2\le 0$, as in this paper, then one will be able to construct matrices $G_j$.

Before we proceed to introduce the details on constructing those matrices, the following observations are clear regarding $u=(f_4/f,f_1{f}_3/f^2,f_2^2/f^2,f_1^2{f}_2/f^3,f_1^4/f^4)$ and the fourth derivative $2h^{\left(4\right)}(X+\sqrt{t}Z)$ (see Equation (13)): (a) the sum-order of derivatives for each entry of u is four, for example, the sum-order of $f_1^2{f}_2/f^3$ is $1\times 2+2=4$; (b) the highest order of a single term in the entries of u is four, namely $f_4/f$; (c) the sum-order of each entry in the fourth derivative is eight, which is twice that of u.

In the following, we take the fourth derivative as an example, and show how to construct these matrices $F_0$ (Section 3.3), $F_i$ (Section 3.1 and Section 3.2), and $G_j$ (Section 3.4). We decide to use the $T_k$ as the entries of u, instead of the $f_k$, the motivation for which is clear from the proof of Corollary 1 and the desire to exploit the assumption $T_2\le 0$. Based on the above observation and the expressions in Equation (5), our vector u is

$$\begin{array}{c}\hfill u=\left(\begin{array}{ccccc}{T}_4,& T_3{T}_1,& T_2^2,& T_2{T}_1^2,& T_1^4\end{array}\right).\end{array}$$

(16)

Thus, $F_0$, $F_i$, $G_j$ are $5\times 5$ symmetric matrices. Here, we mention that the expressions appearing as coordinates in u correspond to the integer partitions of four.

The organization of this section is as follows: Section 3.1, Section 3.2 and Section 3.3 deal with the sign of the fourth derivative with only equality constraints (see Conjecture 1); Section 3.4 further incorporates the inequality constraints, namely $T_2\le 0$; Section 3.5 shows the manipulation for the optimality of Gaussian inputs (see Conjecture 2). In Section 3.6, we consider the sign and the Gaussian optimality for the fifth derivative.

3.1. Matrices $F_i$ from Multiple Representations

The matrices $F_i$ are such that $\mathrm{E}\left[u^T{F}_{i}u\right]=0$. A trivial case is to notice that different products of the form $u\left(i\right)u\left(j\right)$ may map to the same term, for example

$$T_2^2{T}_1^4=\left(T_2^2\right)\left(T_1^4\right)=\left(T_2{T}_1^2\right)\left(T_2{T}_1^2\right),\Rightarrow u\left(3\right)u\left(5\right)=u\left(4\right)u\left(4\right).$$

That is, $T_2^2{T}_1^4$ admits multiple representations as $u\left(i\right)u\left(j\right)$. It is easy to construct the corresponding matrix $F_1$ such that $u^T{F}_{1}u=0$:

$$\begin{array}{c}\hfill F_1=\left[\begin{array}{ccccc}0& 0& 0& 0& 0\\ 0& 0& 0& 0& 0\\ 0& 0& 0& 0& -1\\ 0& 0& 0& 2& 0\\ 0& 0& -1& 0& 0\end{array}\right].\end{array}$$

For the fourth derivative, only one term has multiple representations. There is none for the third derivative, and three for the fifth ($F_1$, $F_2$ and $F_3$ in Section 3.6).

3.2. Matrices $F_i$ from Integration by Parts

The equalities of the type $\mathrm{E}\left[u^T{F}_{i}u\right]=0$ used in [9] are from integration by parts. Here, we list them one by one.

Notice that all the possible terms with sum-order eight and highest-order four are the following (the numbers in the left column are indices):

$$\begin{array}{c}\hfill \begin{array}{cccccc}1-5:& T_4^2,& T_4{T}_3{T}_1,& T_4{T}_2^2,& T_4{T}_2{T}_1^2,& T_4{T}_1^4,\\ 6-9:& T_3^2{T}_1^2,& T_3{T}_2^2{T}_1,& T_3{T}_2{T}_1^3,& T_3{T}_1^5,& \\ 10-13:& T_2^4,& T_2^3{T}_1^2,& T_2^2{T}_1^4,& T_2{T}_1^6,& \\ 14:& T_1^8,& & & & \\ 15:& T_3^2{T}_2.& & & & \end{array}\end{array}$$

Denote this vector as w.

These terms are arranged in the order such that the first (fourteen) terms can be expressed as $u\left(i\right)u\left(j\right)$ for some i and j, while the last term(s) cannot be. We call the first terms the quadratic part $w^{qua}$, and the last term(s) the non-quadratic part $w^{non}$. Thus, $w=(w^{qua},w^{non})$.

It is not difficult to conclude that, for non-repetition, one only needs to perform integration by parts on the entries whose highest-order term is of power one. All of these entries are (eight in total):

$$\begin{array}{c}\hfill \begin{array}{cccccccc}{T}_4{T}_3{T}_1,& T_4{T}_2^2,& T_4{T}_2{T}_1^2,& T_4{T}_1^4,& T_3{T}_2^2{T}_1,& T_3{T}_2{T}_1^3,& T_3{T}_1^5,& T_2{T}_1^6.\end{array}\end{array}$$

Taking $T_4{T}_3{T}_1$ as an example, one can show that (Equation (18), see the end of this subsection)

$$\mathrm{E}\left[2T_4{T}_3{T}_1+T_3^2{T}_1^2+T_3^2{T}_2\right]=0.$$

In addition, this can be written as $\mathrm{E}\left[c_1^{T}w\right]=0$, where

$$c_1\in {\mathbb{R}}^{15},c_1\left([2,6,15]\right)=[2,1,1].$$

There are eight equalities in total and hence there are vectors $c_1,\dots ,c_8$. We put each $c_i$ as the i-th row of $C\in {\mathbb{R}}^{8\times 15}$, and write those equalities as

$$\mathrm{E}\left[Cw\right]=\mathbf{0},$$

where

$$C=\begin{array}{cccc}& \begin{array}{ccccccccccccccc}1& 2& 3& 4& 5& 6& 7& 8& 9& 10& 11& 12& 13& 14& 15\end{array}& & \\ [& \begin{array}{ccccccccccccccc}& 2& & & & 1& & & & & & & & & 1\\ & & 1& & & & 1& & & & & & & & 2\\ & & & 1& & 1& 2& 1& & & & & & & \\ & & & & 1& & & 4& 1& & & & & & \\ & & & & & & 3& & & 1& 1& & & & \\ & & & & & & & 2& & & 3& 1& & & \\ & & & & & & & & 1& & & 5& 1& & \\ & & & & & & & & & & & & 7& 1& \end{array}& ]& \begin{array}{c}1\\ 2\\ 3\\ 4\\ 5\\ 6\\ 7\\ 8\end{array}\end{array}.$$

(17)

The entries can be found in Equations (18)–(25).

We need to extract matrices F from these eight equalities $\mathrm{E}\left[Cw\right]=\mathbf{0}$, such that $\mathrm{E}\left[u^{T}Fu\right]\equiv 0$. The main problem is that $c_k^{T}w$ may contain entries that are not expressible as $u\left(i\right)u\left(j\right)$. In particular, for the fourth derivative, this happens when $c_k\left(15\right)\ne 0$. One needs to do some work to cancel these entries. The general method, which can also be used in higher-order cases, is stated below:

• Firstly, since $w=(w^{qua},w^{non})$, we separate the blocks of C accordingly,

  $$\begin{array}{c}\hfill C=\left[\begin{array}{cc}{C}_{11}& C_{12}\\ C_{21}& \mathbf{0}\end{array}\right],\mathrm{E}[\left[\begin{array}{cc}{C}_{11}& C_{12}\\ C_{21}& \mathbf{0}\end{array}\right]\left[\begin{array}{c}{w}^{qua}\\ w^{non}\end{array}\right]]=\mathbf{0}.\end{array}$$
  In the above, $C_{11}\in {\mathbb{R}}^{2\times 14}$, $C_{12}\in {\mathbb{R}}^{2\times 1}$, $C_{21}\in {\mathbb{R}}^{6\times 14}$.
• Secondly, each row of $C_{21}$ corresponds to a symmetric matrix $F_i$ such that $\mathrm{E}\left[u^T{F}_{i}u\right]\equiv 0$. In particular, for the first row of $C_{21}$, the matrix is

  $$\begin{array}{c}\hfill F_2=\left[\begin{array}{ccccc}0& 0& 0& 1& 0\\ 0& 2& 2& 1& 0\\ 0& 2& 0& 0& 0\\ 1& 1& 0& 0& 0\\ 0& 0& 0& 0& 0\end{array}\right],\end{array}$$

  such that $\frac{1}{2}{u}^T{F}_{2}u=T_4{T}_2{T}_1^2+T_3^2{T}_1^2+2T_3{T}_2^2{T}_1+T_3{T}_2{T}_1^3$. Notice a scaling of a factor of two is added here just for conciseness, and this does not affect the feasibility of (15). Similarly, the other five matrices, corresponding to the remaining rows of $C_{21}$, are

  $$\begin{array}{cc}\hfill F_3& =\left[\begin{array}{ccccc}0& 0& 0& 0& 1\\ 0& 0& 0& 4& 1\\ 0& 0& 0& 0& 0\\ 0& 4& 0& 0& 0\\ 1& 1& 0& 0& 0\end{array}\right],F_4=\left[\begin{array}{ccccc}0& 0& 0& 0& 0\\ 0& 0& 3& 0& 0\\ 0& 3& 2& 1& 0\\ 0& 0& 1& 0& 0\\ 0& 0& 0& 0& 0\end{array}\right],F_5=\left[\begin{array}{ccccc}0& 0& 0& 0& 0\\ 0& 0& 0& 2& 0\\ 0& 0& 0& 3& 1\\ 0& 2& 3& 0& 0\\ 0& 0& 1& 0& 0\end{array}\right],\hfill \end{array}$$

  $$\begin{array}{cc}\hfill F_6& =\left[\begin{array}{ccccc}0& 0& 0& 0& 0\\ 0& 0& 0& 0& 1\\ 0& 0& 0& 0& 5\\ 0& 0& 0& 0& 1\\ 0& 1& 5& 1& 0\end{array}\right],F_7=\left[\begin{array}{ccccc}0& 0& 0& 0& 0\\ 0& 0& 0& 0& 0\\ 0& 0& 0& 0& 0\\ 0& 0& 0& 0& 7\\ 0& 0& 0& 7& 2\end{array}\right].\hfill \end{array}$$
• Thirdly, for $C_{11}$ and $C_{12}$, the equalities are $\mathrm{E}[C_{11}{w}^{qua}+C_{12}{w}^{non}]=\mathbf{0}$. Notice $w^{non}$ cannot be expressed in a quadratic form. Supposing that we can find a column vector z such that $z^T{C}_{12}=0$, then $\mathrm{E}\left[z^T{C}_{11}{w}^{qua}\right]=\mathrm{E}\left[z^T(C_{11}{w}^{qua}+C_{12}{w}^{non})\right]=0$. The vector z actually lies in the null space of $C_{12}^T$, and it suffices to find the basis. One way is to do the $QR$ decomposition:

  $$C_{12}=Q\left[\begin{array}{c}U\\ \mathbf{0}\end{array}\right],$$

  where U is upper-triangular. The null-space of $C_{12}^T$ has the same dimensions as the number of rows of $\mathbf{0}$ above, and a basis as the last several columns of Q—in particular, for the fourth derivative

  $$C_{12}=\left[\begin{array}{c}1\\ 2\end{array}\right]=QR=\left[\begin{array}{cc}-\frac{1}{\sqrt{5}}& -\frac{2}{\sqrt{5}}\\ -\frac{2}{\sqrt{5}}& \frac{1}{\sqrt{5}}\end{array}\right]\left[\begin{array}{c}-\sqrt{5}\\ 0\end{array}\right].$$
  Hence, one takes z as the second column of Q, which is (after scaling for conciseness) $z^T=\left[\begin{array}{cc}-2,& 1\end{array}\right]$. Then, one calculates $z^T{C}_{11}{w}^{qua}=-4T_4{T}_3{T}_1+T_4{T}_2^2-2T_3^2{T}_1^2+T_3{T}_2^2{T}_1$, and the corresponding matrix $F_8$ (scaled by a factor of two) is

  $$\begin{array}{c}\hfill F_8=\left[\begin{array}{ccccc}0& -4& 1& 0& 0\\ -4& -4& 1& 0& 0\\ 1& 1& 0& 0& 0\\ 0& 0& 0& 0& 0\\ 0& 0& 0& 0& 0\end{array}\right].\end{array}$$

The rest of this subsection is devoted to calculating the equalities obtained from integration by parts. This is similar to that in [9], except in the form of the $T_i$. To begin, we need the following lemma.

Lemma 3.

Let A be a linear combination of terms of products of the $T_i$, then, for $n\ge 2$,

$$\mathrm{E}\left[T_{n}A+T_{n-1}\frac{\partial }{\partial y}A+T_{n-1}{T}_{1}A\right]=0.$$

Proof. 

From calculus,

$$\begin{array}{cc}\hfill \mathrm{E}\left[T_{n}A\right]& =\int f{T}_{n}Ady=\int fAd{T}_{n-1}\hfill \\ & \stackrel{\left(a\right)}{=}0-\int T_{n-1}d\left(fA\right)\hfill \\ & =-\int T_{n-1}\left(f_{1}A+f\frac{\partial }{\partial y}A\right)dy\hfill \\ & \stackrel{\left(b\right)}{=}-\int T_{n-1}\left(f{T}_{1}A+f\frac{\partial }{\partial y}A\right)dy\hfill \\ & =-\mathrm{E}\left[T_{n-1}\frac{\partial }{\partial y}A+T_{n-1}{T}_{1}A\right],\hfill \end{array}$$

where $\left(a\right)$ is due to Lemma 1, and $\left(b\right)$ is due to Equation (5). ☐

Now, using Lemma 3, one obtains the following equalities:

$$\begin{array}{cc}\hfill T_n=T_4,A=T_3{T}_1,\Rightarrow & \mathrm{E}\left[2T_4{T}_3{T}_1+T_3^2{T}_2+T_3^2{T}_1^2\right]=0,\hfill \end{array}$$

(18)

$$\begin{array}{cc}\hfill T_n=T_4,A=T_2^2,\Rightarrow & \mathrm{E}\left[T_4{T}_2^2+2T_3^2{T}_2+T_3{T}_2^2{T}_1\right]=0,\hfill \end{array}$$

(19)

$$\begin{array}{cc}\hfill T_n=T_4,A=T_2{T}_1^2,\Rightarrow & \mathrm{E}\left[T_4{T}_2{T}_1^2+T_3^2{T}_1^2+2T_3{T}_2^2{T}_1+T_3{T}_2{T}_1^3\right]=0,\hfill \end{array}$$

(20)

$$\begin{array}{cc}\hfill T_n=T_4,A=T_1^4,\Rightarrow & \mathrm{E}\left[T_4{T}_1^4+4T_3{T}_2{T}_1^3+T_3{T}_1^5\right]=0,\hfill \end{array}$$

(21)

$$\begin{array}{cc}\hfill T_n=T_3,A=T_2^2{T}_1,\Rightarrow & \mathrm{E}\left[3T_3{T}_2^2{T}_1+T_2^4+T_2^3{T}_1^2\right]=0,\hfill \end{array}$$

(22)

$$\begin{array}{cc}\hfill T_n=T_3,A=T_2{T}_1^3,\Rightarrow & \mathrm{E}\left[2T_3{T}_2{T}_1^3+3T_2^3{T}_1^2+T_2^2{T}_1^4\right]=0,\hfill \end{array}$$

(23)

$$\begin{array}{cc}\hfill T_n=T_3,A=T_1^5,\Rightarrow & \mathrm{E}\left[T_3{T}_1^5+5T_2^2{T}_1^4+T_2{T}_1^6\right]=0,\hfill \end{array}$$

(24)

$$\begin{array}{cc}\hfill T_n=T_2,A=T_1^6,\Rightarrow & \mathrm{E}\left[7T_2{T}_1^6+T_1^8\right]=0.\hfill \end{array}$$

(25)

With these equalities, matrix (17) can be constructed.

3.3. Matrix $F_0$ from the Derivative

Suppose we have already obtained the fourth derivative in the form (see Equation (30) later)

$$-2h^{\left(4\right)}(X+\sqrt{t}Z)=\mathrm{E}\left[d^{T}w\right]=\mathrm{E}[d_1^T{w}^{qua}+d_2^T{w}^{non}],$$

where $d_1\in {\mathbb{R}}^{14}$, $d_2\in {\mathbb{R}}^1$. Then, similar to $F_8$, we can find the matrix $F_0$ such that $-2h^{\left(4\right)}(X+\sqrt{t}Z)=\mathrm{E}\left[u^T{F}_{0}u\right]$.

To cancel the non-quadratic term $d_2^T{w}^{non}$, we solve for $z_2^T{C}_{12}=d_2^T$ (the solution $z_2$ should exist, otherwise it is not possible to find a quadratic form and the LMI approach fails). Then, since $\mathrm{E}[C_{11}{w}^{qua}+C_{12}{w}^{non}]=\mathbf{0}$, we have

$$\begin{array}{cc}\hfill -2h^{\left(4\right)}(X+\sqrt{t}Z)& =E\left[d_1^T{w}^{qua}+d_2^T{w}^{non}\right]\hfill \\ & =E\left[d_1^T{w}^{qua}+d_2^T{w}^{non}-z_2^T\left(C_{11}{w}^{qua}+C_{12}{w}^{non}\right)\right]\hfill \\ & =E\left[\left(d_1^T-z_2^T{C}_{11}\right)w^{qua}\right].\hfill \end{array}$$

Now, $F_0$ can be constructed from $d_1^T-z^T{C}_{11}$.

The details are as follows. First, we need to express the derivative using the entries of w. This can be done recursively using the following lemma.

Lemma 4.

Let A be a linear combination of terms of products of the $T_i$. The following equalities hold:

$$\begin{array}{c}2\frac{\partial }{\partial t}{T}_n=\frac{{\partial }^n}{\partial y^n}(T_2+T_1^2)=T_{n+2}+\sum _{k=0}^n{C}_n^k{T}_{k+1}{T}_{n-k+1},n\ge 0,\hfill \end{array}$$

(26)

$$\begin{array}{c}\frac{d}{dt}\mathrm{E}\left[A\right]=\mathrm{E}\left[\frac{1}{2}(T_2+T_1^2)A+\frac{\partial }{\partial t}A\right],\hfill \end{array}$$

(27)

$$\begin{array}{c}\frac{d}{dt}\mathrm{E}\left[T_n^2\right]=\mathrm{E}\left[-T_{n+1}^2+T_n\sum _{k=1}^{n-1}{C}_n^k{T}_{k+1}{T}_{n-k+1}\right],n\ge 1.\hfill \end{array}$$

(28)

The proof is left to Appendix A. Now, with Equation (7):

$$-2h^{″}(X+\sqrt{t}Z)=\mathrm{E}\left[T_2^2\right],$$

and Equation (28), one can easily obtain that

$$\begin{array}{c}\hfill 2h^{\left(3\right)}(X+\sqrt{t}Z)=-\frac{d}{dt}\mathrm{E}\left[T_2^2\right]=\mathrm{E}\left[T_3^2-2T_2^3\right].\end{array}$$

(29)

For the fourth derivative,

$$\begin{array}{cc}\hfill -2h^{\left(4\right)}(X+\sqrt{t}Z)& =-\frac{d}{dt}\mathrm{E}\left[T_3^2-2T_2^3\right]\hfill \\ & \stackrel{\left(a\right)}{=}-\mathrm{E}\left[-T_4^2+T_3\left(3T_2{T}_3+3T_3{T}_2\right)\right]-\frac{d}{dt}\mathrm{E}\left[-2T_2^3\right]\hfill \\ & \stackrel{\left(b\right)}{=}-\mathrm{E}\left[-T_4^2+T_3\left(3T_2{T}_3+3T_3{T}_2\right)\right]-\mathrm{E}\left[-\left(T_2+T_1^2\right)T_2^3-3T_2^2\frac{\partial }{\partial t}2T_2\right]\hfill \\ & \stackrel{\left(c\right)}{=}-\mathrm{E}\left[-T_4^2+6T_3^2{T}_2-T_2^4-T_2^3{T}_1^2-3T_2^2(T_4+2T_3{T}_1+2T_2{T}_2)\right]\hfill \\ & =\mathrm{E}\left[T_4^2+3T_4{T}_2^2-6T_3^2{T}_2+6T_3{T}_2^2{T}_1+7T_2^4+T_2^3{T}_1^2\right],\hfill \end{array}$$

(30)

where $\left(a\right)$, $\left(b\right)$, $\left(c\right)$ are due to Equations (28), (27), (26), respectively.

Hence, we have the vector $d=(d_1,d_2)\in {\mathbb{R}}^{15}$ and its blocks $d_1\in {\mathbb{R}}^{14}$, $d_2\in {\mathbb{R}}^1$:

$$\begin{array}{c}d[1,3,7,10,11]=[1,3,6,7,1],\hfill \\ d\left[15\right]=-6.\hfill \end{array}$$

One solves for $z_2$ such that $z_2^T{C}_{12}=d_2^T$ and obtains

$$\begin{array}{c}\hfill z_2^T=\left[\begin{array}{cc}0,& -3\end{array}\right].\end{array}$$

Now, $d_1^T-z_2^T{C}_{11}$ has nonzero entries at locations $[1,3,7,10,11]$, with values $[1,6,9,7,1],$ respectively. Furthermore, $F_0$ (scaled by a factor of two) is found as

$$\begin{array}{c}\hfill F_0=\left[\begin{array}{ccccc}2& 0& 6& 0& 0\\ 0& 0& 9& 0& 0\\ 6& 9& 14& 1& 0\\ 0& 0& 1& 0& 0\\ 0& 0& 0& 0& 0\end{array}\right].\end{array}$$

By the end of this subsection, it is easy to see that Cheng and Geng’s method can be reformulated as identifying if there exist $x_1,\dots ,x_8\in \mathbb{R}$ such that

$$F_0+\sum _{i=1}^8{x}_i{F}_i⪰0.$$

We use the convex optimization package [19] to identify the feasibility of the above LMI problem, and it turns out to be feasible as it should be according to Equation (14).

3.4. Matrices $G_j$ from Log-Concavity

Recall that, in [9], there is no matrix $G_j$, since there is no inequality constraint. In this paper, we consider the log-concave case $T_2\le 0$, thus introducing inequality constraints.

For the fourth order, $T_2\le 0$ actually implies that the following entries in w are nonpositive:

$$T_2^3{T}_1^2,T_2{T}_1^6,T_2{T}_3^2.$$

It is clear that the powers of $T_2$ are odd, and the others are even.

To transform these nonpositive terms into matrices $G_j$, the first two terms, $T_2^3{T}_1^2$ and $T_2{T}_1^6$ are trivial, since they can be expressed by $u\left(i\right)u\left(j\right)$ directly:

$$\begin{array}{c}\hfill 0\ge 2\mathrm{E}\left[T_2^3{T}_1^2\right]=\mathrm{E}\left[u^T{G}_{1}u\right],\\ \hfill 0\ge 2\mathrm{E}\left[T_2{T}_1^6\right]=\mathrm{E}\left[u^T{G}_{2}u\right],\end{array}$$

where

$$\begin{array}{c}\hfill G_1=\left[\begin{array}{ccccc}0& 0& 0& 0& 0\\ 0& 0& 0& 0& 0\\ 0& 0& 0& 1& 0\\ 0& 0& 1& 0& 0\\ 0& 0& 0& 0& 0\end{array}\right],G_2=\left[\begin{array}{ccccc}0& 0& 0& 0& 0\\ 0& 0& 0& 0& 0\\ 0& 0& 0& 0& 0\\ 0& 0& 0& 0& 1\\ 0& 0& 0& 1& 0\end{array}\right].\end{array}$$

For the term $T_2{T}_3^2$, the idea is similar to the third part in Section 3.2. One first finds $z_3\in {\mathbb{R}}^2$ such that $z_3^T{C}_{12}{w}^{non}=T_2{T}_3^2$, namely $z_3^T{C}_{12}=1$. The solution is $z_3^T=\left[\begin{array}{cc}0,& 1/2\end{array}\right]$. Then,

$$\begin{array}{c}\hfill \mathrm{E}\left[T_2{T}_3^2\right]=\mathrm{E}[T_2{T}_3^2-z_3^T(C_{11}{w}^{qua}+C_{12}{w}^{non})]=\mathrm{E}[-z_3^T{C}_{11}{w}^{qua}]=\mathrm{E}[-\frac{1}{2}{T}_4{T}_2^2-\frac{1}{2}{T}_3{T}_2^2{T}_1].\end{array}$$

Now, it is routine to obtain

$$\begin{array}{c}\hfill 0\ge 4\mathrm{E}\left[T_2{T}_3^2\right]=\mathrm{E}\left[u^T{G}_{3}u\right],\end{array}$$

where

$$\begin{array}{c}\hfill G_3=\left[\begin{array}{ccccc}0& 0& -1& 0& 0\\ 0& 0& -1& 0& 0\\ -1& -1& 0& 0& 0\\ 0& 0& 0& 0& 0\\ 0& 0& 0& 0& 0\end{array}\right].\end{array}$$

At this point, we are done with the procedure for calculating all these matrices $F_0$, the $F_i$ and the $G_j$. To show the negativity of the fourth derivative, it suffices to find a set of variables $x_i\in \mathbb{R}$ and $y_j\ge 0$ such that

$$\begin{array}{c}\hfill F_0+\sum _{i=1}^8{x}_i{F}_i+\sum _{j=1}^3{y}_j{G}_j⪰0.\end{array}$$

Remark 2.

The matrix $G_2$ is actually redundant, since we know that $\mathrm{E}\left[T_2{T}_1^6\right]\equiv -\frac{1}{7}\mathrm{E}\left[T_1^8\right]\le 0$, which is already included in the matrices $F_i$ (in particular, matrix $F_7$ in Section 3.2). Including $G_2$ will not affect the feasibility check.

3.5. Matrix ${\tilde{F}}_0$ for Gaussian Optimality

However, to show the optimality of the Gaussian, the above formulation is not enough. According to inequality $\left(a\right)$ in Equation (11), it would suffice to show that

$${(-1)}^{n-1}\times 2h^{\left(n\right)}(X+\sqrt{t}Z)-(n-1)!\times \mathrm{E}\left[{(-T_2)}^n\right]\ge 0$$

instead of ${(-1)}^{n-1}\times 2h^{\left(n\right)}(X+\sqrt{t}Z)\ge 0$. Thus, one needs to calculate the matrix $F_0$ such that

$${(-1)}^{n-1}\times 2h^{\left(n\right)}(X+\sqrt{t}Z)-(n-1)!\times \mathrm{E}\left[{(-T_2)}^n\right]=\mathrm{E}\left[u^T{F}_{0}u\right].$$

The procedure is the same as that in Section 3.3.

In particular, for the fourth derivative, since $n=4$ is even, we directly have the quadratic form $\mathrm{E}\left[{(-T_2)}^n\right]=u\left(3\right)u\left(3\right)$. It is straightforward to construct the matrix ${\tilde{F}}_0$ (scaled by a factor of two) here

$$\begin{array}{c}\hfill {\tilde{F}}_0=F_0-\left[\begin{array}{ccccc}0& 0& 0& 0& 0\\ 0& 0& 0& 0& 0\\ 0& 0& 2\times 3!& 0& 0\\ 0& 0& 0& 0& 0\\ 0& 0& 0& 0& 0\end{array}\right]=\left[\begin{array}{ccccc}2& 0& 6& 0& 0\\ 0& 0& 9& 0& 0\\ 6& 9& 2& 1& 0\\ 0& 0& 1& 0& 0\\ 0& 0& 0& 0& 0\end{array}\right],\end{array}$$

such that $\mathrm{E}[u^T{\tilde{F}}_{0}u]=-4h^{\left(4\right)}(X+\sqrt{t}Z)-12\mathrm{E}\left[T_2^4\right]$.

Now, the LMI is updated as ${\tilde{F}}_0+{\sum }_{i=1}^8{x}_i{F}_i+{\sum }_{j=1}^3{y}_j{G}_j⪰0.$ Again, we use the convex optimization package [19] to check the feasibility. It turns out to be feasible and the solution helps us to identify the coefficients in Equation (9).

3.6. Fifth Derivative

For the fifth derivative, we omit the details of the manipulations since they are routine, and just provide the matrices here. For brevity, we only list out the nonzero entries of the upper-triangular part of a symmetric matrix. These matrices (with scaling) are

$$\begin{array}{cc}\hfill F_0:& F\left[\right(1,1),(1,3),(1,5),(2,3),(2,5),(3,3),(3,4),(3,5),(3,6),(5,5),(5,6\left)\right]\hfill \\ & =[2,20,29,\frac{214}{3},\frac{49}{2},\frac{178}{3},-\frac{37}{3},58,-6,45,-\frac{1}{2}],\hfill \\ \hfill F_1:& F\left[\right(3,6),(4,5\left)\right]=[-1,1],\hfill \\ \hfill F_2:& F\left[\right(3,7),(4,6\left)\right]=[-1,1],\hfill \\ \hfill F_3:& F\left[\right(5,7),(6,6\left)\right]=[-1,2],\hfill \\ \hfill F_4:& F\left[\right(1,4),(2,2),(2,3),(2,4\left)\right]=[1,2,2,1],\hfill \\ \hfill F_5:& F\left[\right(1,6),(2,4),(2,5),(2,6\left)\right]=[1,1,3,1],\hfill \\ \hfill F_6:& F\left[\right(1,7),(2,6),(2,7\left)\right]=[1,5,1],\hfill \\ \hfill F_7:& F\left[\right(2,4),(3,4),(4,4\left)\right]=[2,3,2],\hfill \\ \hfill F_8:& F\left[\right(2,5),(3,4),(3,5),(3,6\left)\right]=[1,2,2,1],\hfill \\ \hfill F_9:& F\left[\right(2,6),(3,6),(3,7),(4,4\left)\right]=[1,4,1,2],\hfill \\ \hfill F_{10}:& F\left[\right(2,7),(3,7),(4,7\left)\right]=[1,6,1],\hfill \\ \hfill F_{11}:& F\left[\right(3,6),(5,5),(5,6\left)\right]=[3,6,1],\hfill \\ \hfill F_{12}:& F\left[\right(3,7),(5,6),(5,7\left)\right]=[2,5,1],\hfill \\ \hfill F_{13}:& F\left[\right(4,7),(5,7),(6,7\left)\right]=[1,7,1],\hfill \\ \hfill F_{14}:& F\left[\right(6,7),(7,7\left)\right]=[9,2],\hfill \\ \hfill F_{15}:& F\left[\right(1,2),(1,3),(2,2),(2,3),(3,3),(3,4\left)\right]=[6,-3,6,-5,-2,-1],\hfill \\ \hfill F_{16}:& F\left[\right(1,5),(2,3),(2,5),(3,3),(3,5\left)\right]=[-1,-2,-1,6,1].\hfill \end{array}$$

For the sign of the fifth derivative, we used the convex optimization package [19] to solve the following LMI problem,

$$F_0+\sum _{i=1}^{16}{x}_i{F}_i⪰0,$$

but could not find a feasible solution $x_1,\dots ,x_{16}\in \mathbb{R}$. This suggests to us that a direct generalization of Cheng and Geng’s method may not work for the fifth derivative.

Instead, if we consider the log-concavity constraint $T_2\le 0$ and check the optimality of Gaussian inputs, then we have a new matrix ${\tilde{F}}_0$ (similar to Section 3.5) and several matrices $G_j$ as the following:

$$\begin{array}{cc}\hfill {\tilde{F}}_0:& F\left[\right(1,1),(1,3),(1,5),(2,3),(2,5),(3,3),(3,4),(3,5),(3,6),(5,5),(5,6\left)\right]\hfill \\ & =[2,20,29,\frac{214}{3},\frac{49}{2},\frac{178}{3},-\frac{37}{3},-38,-6,-3,-\frac{1}{2}],\hfill \\ \hfill G_1:& G\left[\right(3,4\left)\right]=\left[1\right],\hfill \\ \hfill G_2:& G\left[\right(5,6\left)\right]=\left[1\right],\hfill \\ \hfill G_3:& G\left[\right(6,7\left)\right]=\left[1\right],\hfill \\ \hfill G_4:& G\left[\right(1,3),(2,3),(3,3),(3,4\left)\right]=[-3,-5,-2,-1],\hfill \\ \hfill G_5:& G\left[\right(3,5),(5,5\left)\right]=[-2,-1].\hfill \end{array}$$

Now, one would like to find $x_1,\dots ,x_{16}\in \mathbb{R}$ and $y_1,\dots ,y_5\in {\mathbb{R}}^{+}$ such that

$${\tilde{F}}_0+\sum _{i=1}^{16}{x}_i{F}_i+\sum _{i=1}^5{y}_i{G}_i⪰0.$$

This can be solved by the convex optimization package [19]. Again, the solution helps us to arrive at Equation (10).

4. Proof of Theorem 1

Proof. 

For the third derivative, according to Equation (29), we have

$$\begin{array}{c}\hfill 2h^{\left(3\right)}(X+\sqrt{t}Z)=\mathrm{E}\left[T_3^2-2T_2^3\right].\end{array}$$

For the fourth derivative, according to Equation (30):

$$\begin{array}{c}\hfill -2h^{\left(4\right)}(X+\sqrt{t}Z)=\mathrm{E}\left[T_4^2+3T_4{T}_2^2-6T_3^2{T}_2+6T_3{T}_2^2{T}_1+7T_2^4+T_2^3{T}_1^2\right].\end{array}$$

Adding multiples of the left-hand sides of the equations: $-3\times \left(19\right)-\left(22\right)$, we obtain

$$\begin{array}{cc}\hfill -2h^{\left(4\right)}(X+\sqrt{t}Z)& =\mathrm{E}\left[T_4^2+3T_4{T}_2^2-6T_3^2{T}_2+6T_3{T}_2^2{T}_1+7T_2^4+T_2^3{T}_1^2\right]\hfill \\ & \stackrel{\left(a\right)}{=}\mathrm{E}\left[T_4^2+\left(-6T_3^2{T}_2-3T_3{T}_2^2{T}_1\right)-6T_3^2{T}_2+6T_3{T}_2^2{T}_1+7T_2^4+T_2^3{T}_1^2\right]\hfill \\ & =\mathrm{E}\left[T_4^2-12T_3^2{T}_2+3T_3{T}_2^2{T}_1+7T_2^4+T_2^3{T}_1^2\right]\hfill \\ & \stackrel{\left(b\right)}{=}\mathrm{E}\left[T_4^2-12T_3^2{T}_2+\left(-T_2^4-T_2^3{T}_1^2\right)+7T_2^4+T_2^3{T}_1^2\right]\hfill \\ & =\mathrm{E}\left[T_4^2-12T_3^2{T}_2+6T_2^4\right],\hfill \end{array}$$

where $\left(a\right)$ is due to Equation (19), and $\left(b\right)$ is due to Equation (22).

For the fifth derivative,

$$\begin{array}{c}\hfill 2h^{\left(5\right)}(X+\sqrt{t}Z)=\frac{d}{dt}\mathrm{E}\left[-T_4^2-6T_2^4+12T_3^2{T}_2\right]=\frac{d}{dt}\mathrm{E}\left[-T_4^2\right]+\frac{d}{dt}\mathrm{E}\left[-6T_2^4\right]+\frac{d}{dt}\mathrm{E}\left[12T_3^2{T}_2\right].\end{array}$$

For each term above on the right-hand side: According to Equation (28),

$$\begin{array}{c}\hfill \frac{d}{dt}\mathrm{E}\left[-T_4^2\right]=\mathrm{E}\left[T_5^2-8T_4^2{T}_2-6T_4{T}_3^2\right].\end{array}$$

(31)

For the second term,

$$\begin{array}{cc}\hfill \frac{d}{dt}\mathrm{E}\left[-6T_2^4\right]& \stackrel{\left(c\right)}{=}\mathrm{E}\left[-3(T_2+T_1^2)T_2^4-12T_2^3\times \frac{\partial }{\partial t}\left(2T_2\right)\right]\hfill \\ & \stackrel{\left(d\right)}{=}\mathrm{E}\left[-3T_2^5-3T_2^4{T}_1^2-12T_2^3\left(T_4+2T_3{T}_1+2T_2^2\right)\right]\hfill \\ & =\mathrm{E}\left[-3T_2^4{T}_1^2-12T_4{T}_2^3-24T_3{T}_2^3{T}_1-27T_2^5\right],\hfill \end{array}$$

(32)

where $\left(c\right)$ is due to Equation (27), and $\left(d\right)$ is due to Equation (26). For the third term, according to Equation (27),

$$\begin{array}{c}\frac{d}{dt}\mathrm{E}\left[12T_3^2{T}_2\right]=\mathrm{E}\left[6\left(T_2+T_1^2\right)T_3^2{T}_2+\frac{\partial }{\partial t}\left(12T_3^2{T}_2\right)\right],\hfill \end{array}$$

where the last term is

$$\begin{array}{cc}\hfill \frac{\partial }{\partial t}\left(12T_3^2{T}_2\right)& =12T_3^2\frac{\partial }{\partial t}{T}_2+24T_3{T}_2\times \frac{\partial }{\partial t}{T}_3\hfill \\ & \stackrel{\left(e\right)}{=}6T_3^2\left(T_4+2T_3{T}_1+2T_2^2\right)+12T_3{T}_2\left(T_5+2T_4{T}_1+6T_3{T}_2\right)\hfill \\ & =12T_5{T}_3{T}_2+6T_4{T}_3^2+24T_4{T}_3{T}_2{T}_1+12T_3^3{T}_1+84T_3^2{T}_2^2,\hfill \end{array}$$

where $\left(e\right)$ is due to Equation (26). Hence

$$\begin{array}{cc}\hfill \frac{d}{dt}\mathrm{E}\left[12T_3^2{T}_2\right]& =\mathrm{E}\left[12T_5{T}_3{T}_2+6T_4{T}_3^2+24T_4{T}_3{T}_2{T}_1+6T_3^2{T}_2{T}_1^2+12T_3^3{T}_1+90T_3^2{T}_2^2\right].\hfill \end{array}$$

(33)

Finally, combining Equations (31)–(33), we get

$$\begin{array}{cc}\hfill 2h^{\left(5\right)}(X+\sqrt{t}Z)& =\frac{d}{dt}E\left[-T_4^2\right]+\frac{d}{dt}E\left[-6T_2^4\right]+\frac{d}{dt}E\left[12T_3^2{T}_2\right]\hfill \\ & =E\left[T_5^2-8T_4^2{T}_2-3T_2^4{T}_1^2-12T_4{T}_2^3-24T_3{T}_2^3{T}_1\right.\hfill \\ & \left.-27T_2^5+12T_5{T}_3{T}_2+24T_4{T}_3{T}_2{T}_1+6T_3^2{T}_2{T}_1^2+12T_3^3{T}_1+90T_3^2{T}_2^2\right].\hfill \end{array}$$

(34)

To simplify Equation (34), using Lemma 3, one first obtains the following equalities:

$$\begin{array}{cc}\hfill T_n=T_4,A=T_3{T}_2{T}_1,\Rightarrow & E\left[2T_4{T}_3{T}_2{T}_1+T_3^3{T}_1+T_3^2{T}_2^2+T_3^2{T}_2{T}_1^2\right]=0,\hfill \end{array}$$

(35)

$$\begin{array}{cc}\hfill T_n=T_3,A=T_2^3{T}_1,\Rightarrow & E\left[4T_3{T}_2^3{T}_1+T_2^5+T_2^4{T}_1^2\right]=0,\hfill \end{array}$$

(36)

$$\begin{array}{cc}\hfill T_n=T_4,A=T_2^3,\Rightarrow & E\left[T_4{T}_2^3+3T_3^2{T}_2^2+T_3{T}_2^3{T}_1\right]=0.\hfill \end{array}$$

(37)

Then, adding multiples of the left-hand sides of Equations (35)–(37), we have

$$\begin{array}{cc}\hfill 2h^{\left(5\right)}(X+\sqrt{t}Z)& =2h^{\left(5\right)}(X+\sqrt{t}Z)-12\times \left(35\right)+3\times \left(36\right)+12\times \left(37\right)\hfill \\ & =E\left[T_5^2-24T_2^5-8T_4^2{T}_2-6T_3^2{T}_2{T}_1^2+12T_5{T}_3{T}_2+114T_3^2{T}_2^2\right].\hfill \end{array}$$

 ☐

5. Discussion

5.1. On the Derivatives

We are not able to say anything conclusive about the sign of the fifth derivative of the differential entropy $h(X+\sqrt{t}Z)$. If we impose the log-concavity condition, namely $T_2\le 0$, then the fifth derivative is at least $4!\times \mathrm{E}[{(-T_2)}^5]$. This motivates us to consider the following problem: Without additional constraints, what are the values $c_5>0$ such that

$$2h^{\left(5\right)}(X+\sqrt{t}Z)\ge c_5\times \mathrm{E}[{(-T_2)}^5].$$

If one finds such a value $c_5$, then so long as $\mathrm{E}[{(-T_2)}^5]\ge 0$, the sign of the fifth derivative is determined. This condition is much weaker than $T_2\le 0$.

For the computational part, one only needs to construct the matrix ${\tilde{F}}_0$ such that $2h^{\left(5\right)}(X+\sqrt{t}Z)-c_5\times \mathrm{E}[{(-T_2)}^5]=\mathrm{E}[u^T{\tilde{F}}_{0}u]$, and then solve the problem (see Section 3.6 for the matrices $F_i$)

$${\tilde{F}}_0+\sum _{i=1}^{16}{x}_i{F}_i⪰0.$$

It turns out that $c_5=0.13$ works, while $c_5=0.125$ fails. The authors guess that $c_5\in [0.13,24]$ works, but, at the moment, can just partly confirm this with limited simulation.

Notice that the third derivative of the entropy power $N(X+\sqrt{t}Z)$ was shown to be nonnegative under the log-concavity condition [5], and we recover this in Corollary 3. We also considered the fourth derivative, but failed to obtain the sign because we were unable to apply the Cauchy–Schwartz inequality as we did for the third derivative.

5.2. Possible Proofs

To prove Conjecture 1, besides the method proposed in [9], we are also considering the following ways: the first one is constructive and inspired by Equation (1). Given a random variable X, if we can construct a proper measure $\mu (·)$ such that Equation (1) holds, then one proves Conjecture 1. However, this is difficult even when X is binary symmetric, which is a very simple random variable.

The second one is recursive. Suppose one can find a formula for the n-th derivative such that

$$\begin{array}{cc}\hfill {(-1)}^{n-1}\times h^{\left(n\right)}(X+\sqrt{t}Z)& =\mathrm{E}[\sum _{i=1}^{k_n}{A}_i^2],\hfill \\ \hfill \frac{d}{dt}\mathrm{E}\left[A_i^2\right]& =\mathrm{E}[-B_i^2],\hfill \end{array}$$

then it is clear that

$$\begin{array}{cc}\hfill {(-1)}^n\times h^{(n+1)}(X+\sqrt{t}Z)& =\mathrm{E}[\sum _{i=1}^{k_n}{B}_i^2]\ge 0.\hfill \end{array}$$

However, this fails for $n=2$ (see Equation (7) and Theorem 1). Instead, one may expect that

$$\begin{array}{cc}\hfill \frac{d}{dt}\mathrm{E}\left[A_i^2\right]& =\mathrm{E}[-B_i^2-C_i^2+B_{i+1}^2],\hfill \end{array}$$

and then

$$\begin{array}{cc}\hfill {(-1)}^n\times h^{(n+1)}(X+\sqrt{t}Z)& =\mathrm{E}[-B_1^2+B_{k_n+1}^2-\sum _{i=1}^{k_n}{C}_i^2].\hfill \end{array}$$

If further one can show that $\mathrm{E}[-B_1^2+B_{k_n+1}^2]=\mathrm{E}[-C_{k_n+1}^2]$ for some $C_{k_n+1}$, then one finishes the proof. Notice here that a clever observation is needed for this way to work.

5.3. Applications

The topic of Gaussian optimality has wide applications, for example in [20,21]. In this work, besides the Gaussian optimality, we also have some new observations. In [11], the derivatives in the signal-noise ratio (snr) of $I(X;\sqrt{snr}X+Z)$ are studied. In particular, the first four derivatives are obtained in the language of the minimum mean-square error (Equations (69)–(72) in Corollary 1 of [11]). However, it is not clear whether some of these derivatives are signed or not.

With some standard manipulations, it is not difficult to show that

$$\begin{array}{c}\hfill I(X;\sqrt{snr}X+Z)=h(\sqrt{snr}X+Z)-h\left(Z\right)=h(X+\frac{1}{\sqrt{snr}}Z)+log\sqrt{snr}-\frac{1}{2}log2\pi e.\end{array}$$

By letting $t=1/\sqrt{snr}$, one can easily connect the minimum mean-square error formulae in [11] with the signs of the derivatives of $h(X+\sqrt{t}Z)$ in t. The verification of Conjectures 1 and 2 would imply the bounding and extremal properties of Equations (69)–(72) in [11], and thus deepen our understanding of the minimum mean-square error estimation under the additive-Gaussian setting.

In addition, notice that the probability density function $f(y,t)$ of $Y=X+\sqrt{t}Z$ is the solution of the heat equation $\frac{\partial }{\partial t}f(y,t)=\frac{1}{2}\frac{{\partial }^2}{\partial y^2}f(y,t)$ with the initial condition that $f(y,0)=f_X\left(y\right)$. Hence, Conjectures 1 and 2, if true, reveal the properties of the differential entropy of functions that satisfy the heat equation. For more results related to diffusion equations, one may refer to [22].

6. Conclusions

In this paper, we studied two conjectures on the derivatives of the differential entropy of a general random variable with added Gaussian noise. Regarding the conjecture on the signs of the derivatives made by Cheng and Geng, we introduced the linear matrix inequality approach to provide evidence that their original method might not generalize to orders higher than four. Instead, we considered imposing an additional constraint, namely the log-concavity assumption, and showed the optimality of Gaussian random variables for orders three, four and five. Thus, we made progress on McKean’s conjecture, under a mild condition.