Exploiting the Massey Gap

Abstract

In this paper, we find new refinements to the Massey inequality, which relates the Shannon and guessing entropies, introducing a new concept: the Massey gap. By shrinking the Massey gap, we improve all previous work without introducing any new parameters, providing closed-form strict refinements, as well as a numerical procedure improving them even further.

1. Introduction

The guessing entropy associated with a probabilistic vector $\mathbf{p}=\left(p_1,p_2,\cdots ,p_n\right)$ with $p_1\ge \cdots \ge p_n>0$ is the expected value of the random variable $G\left(\mathbf{p}\right)$ given by $\mathrm{Pr}\left[G\left(\mathbf{p}\right)=i\right]=p_i\forall i\in \overline{1,n}$, i.e., $\mathbb{E}\left[G\left(\mathbf{p}\right)\right]={\sum }_{i=1}^{n}i{p}_i$, and it corresponds to the optimal average number of binary questions required guessing the value of a random variable distributed according to $\mathbf{p}$. Meanwhile, the Shannon entropy of $\mathbf{p}$ is $H\left(\mathbf{p}\right)=-{\sum }_{i=1}^n{p}_{i}log{p}_i$, and the first relation between them was provided by J. Massey in [1], namely that if $H\left(\mathbf{p}\right)\ge 2$, then $\mathbb{E}\left[G\left(\mathbf{p}\right)\right]\ge 2^{H\left(\mathbf{p}\right)-2}+1$.

While the Massey inequality has been shown to have many applications such as password guessing [2] and encourages multiple developments such as moments inequalities (e.g., [3,4]), there are only two known direct refinements to the Massey inequality, both of which were published in 2019. In an ISITpaper, Popescu and Choudary [5] found that:

$$\mathbb{E}\left[G\left(\mathbf{p}\right)\right]\ge 2^{H\left(\mathbf{p}\right)+2p_n-2}+1-p_n$$

(1)

$$\ge 2^{H\left(\mathbf{p}\right)+p_n-2}+1-\frac{1}{2}{p}_n$$

(2)

$$\ge 2^{H\left(\mathbf{p}\right)-2}+1,$$

(3)

subject to the same conditions as the Massey inequality. Meanwhile, Rioul’s inequality [6], published in a CHESpaper [7], states that for all values of $H\left(\mathbf{p}\right)$, we have:

$$\mathbb{E}\left[G\left(\mathbf{p}\right)\right]>2^{H\left(\mathbf{p}\right)}/e$$

(4)

a bound that is similar to the Massey inequality and refines it when $H\left(\mathbf{p}\right)\ge log\frac{e}{1-e/4}$.

Remarkably, the refinement in [5] is actually strict since $\mathbf{p}$ has finite support, $\left|\mathbf{p}\right|=n\le 1/inf\mathbf{p}<\infty$. In fact, applying the method from [5], it can be easily observed that Equation (4) could be further improved. For example, applying just the first step of the method presented in [5], we can obtain that for $H\left(\mathbf{p}\right)\ge log\frac{e}{2ln2}$, we have a new further refinement of Equation (4),

$$\mathbb{E}\left[G\left(\mathbf{p}\right)\right]>2^{H\left(\mathbf{p}\right)+p_n}/e-\frac{1}{2}{p}_n>2^{H\left(\mathbf{p}\right)}/e.$$

Motivated by this insight, in this paper, we find and fully investigate a method to refine an inequality regarding finite support vectors using only itself. Here, we apply our analysis to the famous Massey inequality and explore multiple avenues of applying our method: shrinking the Massey gap.

2. Results

In this section, we present our results, as well as their proofs. In Section 2.1, we find a refinement over the initial result of [5], i.e., Equation (2), which we improve in Section 2.2 beyond even the strongest result of [5], i.e., Equation (1). Then, in Section 2.3, we introduce a new concept, the Massey gap, and use it to generalize the method in Section 2.2. Next, in Section 2.4, we show how using the Massey gap, we can almost always improve our refinement, by rebalancing the middle of the distribution, and finally, in Section 2.5, we find closed-form easy to compute refinements over the work of [5].

2.1. An Improved Refinement to the Massey Inequality

In this section, we find a new relation between the Shannon and guessing entropy, dependent on the minimal probability of a given distribution, refining the initial bound, Equation (2), from the work of [5].

Consider a decreasing probabilistic vector $\mathbf{p}=\left(p_1,p_2,\cdots ,p_n\right)$ with $H\left(\mathbf{p}\right)\ge 2$. Inspired by the trick in [5], we construct the new probabilistic vector $\mathbf{q}=\left(p_1,p_2,\cdots ,p_{n-1},\left(1-\alpha \right)p_n,\alpha p_n\right)$, which is decreasing and strictly positive if and only if $\alpha \in \left(0,1/2\right]$. From the grouping property of entropy,

$$H\left(\mathbf{q}\right)=H\left(\mathbf{p}\right)+p_{n}h\left(\alpha \right),$$

(5)

where $h\left(\alpha \right)=-\alpha log\alpha -\left(1-\alpha \right)log\left(1-\alpha \right)$ is the binary entropy function. Moreover, we calculate:

$$\mathbb{E}\left[G\left(\mathbf{q}\right)\right]=\mathbb{E}\left[G\left(\mathbf{p}\right)\right]+\alpha p_n.$$

(6)

Thus, because $H\left(\mathbf{q}\right)\ge 2$, we can apply the Massey inequality for $\mathbf{q}$ to obtain:

$$\begin{array}{c}\hfill \mathbb{E}\left[G\left(\mathbf{p}\right)\right]=\mathbb{E}\left[G\left(\mathbf{q}\right)\right]-\alpha p_n\ge 2^{H\left(\mathbf{q}\right)-2}+1-\alpha p_n=2^{H\left(\mathbf{p}\right)+p_{n}h\left(\alpha \right)-2}+1-\alpha p_n.\end{array}$$

Now, we note that $h\left(0\right)=0$ and $h\left(1/2\right)=1>\frac{1}{2}\frac{1}{ln2}\approx 0.7213$; thus, by the strict concavity of h, we have $h\left(\alpha \right)>\alpha \frac{1}{ln2}$. Using the inequality $2^x>1+xln2$ for $x=p_{n}h\left(\alpha \right)>0$, we find that:

$$\begin{array}{cc}\hfill 0& \le p_n\left(h\left(\alpha \right)ln2-\alpha \right)<2^{p_{n}h\left(\alpha \right)}-1-\alpha p_n\hfill \\ \hfill & \le 2^{H\left(\mathbf{p}\right)-2}\left(2^{p_{n}h\left(\alpha \right)}-1\right)-\alpha p_n=\left(2^{H\left(\mathbf{p}\right)+p_{n}h\left(\alpha \right)-2}+1-\alpha p_n\right)-\left(2^{H\left(\mathbf{p}\right)-2}+1\right),\hfill \end{array}$$

i.e., our lower bound is always strictly tighter than Massey’s,

$$\mathbb{E}\left[G\left(\mathbf{p}\right)\right]\ge 2^{H\left(\mathbf{p}\right)+p_{n}h\left(\alpha \right)-2}+1-\alpha p_n>2^{H\left(\mathbf{p}\right)-2}+1\forall \alpha \in \left(0,1/2\right].$$

(7)

For the particular value $\alpha =1/2$, the bound above produces the initial bound, Equation (2), of [5]. Thus, we conclude this subsection with the following refinement to Equation (2) summarizing the above:

Theorem 1.

For strictly positive descending probabilistic vectors $\mathbf{p}\in {\mathbb{R}}^n$ with $H\left(\mathbf{p}\right)\ge 2$, we have:

$$\begin{array}{cc}\hfill \mathbb{E}\left[G\left(\mathbf{p}\right)\right]& \ge \underset{\alpha \in \left[0,1/2\right]}{sup}{2}^{H\left(\mathbf{p}\right)+p_{n}h\left(\alpha \right)-2}+1-\alpha p_n\ge 2^{H\left(\mathbf{p}\right)+p_n-2}+1-\frac{1}{2}{p}_n\hfill \\ \hfill & >\underset{\alpha \in \left[0,1/2\right]}{inf}{2}^{H\left(\mathbf{p}\right)+p_{n}h\left(\alpha \right)-2}+1-\alpha p_n=2^{H\left(\mathbf{p}\right)-2}+1.\hfill \end{array}$$

Proof. 

The first inequality follows taking the supremum over $\alpha$ in Equation (7). The second and third inequality follow taking $\alpha =1/2$, which yields the right-hand side in Equation (2), where the strictness of the last inequality follows from the strictness of (7), the right-hand side of which is the bound obtained taking $\alpha =0$. □

Up to this point, we found a direct refinement over the initial result of [5]. In the next section, we will leverage Theorem 1 in order to refine the strongest result, the bound in Equation (1).

2.2. Refinement through Generalization

In this section, we extend the reasoning of [5] to improve our bound from Theorem 1 beyond (1). We leverage a technique previously used in [8], namely applying the Massey inequality to increasingly complicated perturbations of the input in order to obtain ever tighter bounds for the original input.

Given an initial decreasing probabilistic vector $\mathbf{p}$, we construct a list of probabilistic sequences $\left\{{\mathbf{Q}}_k\right\}$, which we recursively define according to the procedure in the previous subsection.

We begin by fixing an arbitrary parameter $\alpha \in \left[0,1/2\right]$ as in the section above. Now, we explicate our construction. Denoting by $Q_{k,i}$ the ith component of the sequence ${\mathbf{Q}}_k$, we define the terms of the list $\left\{{\mathbf{Q}}_k\right\}$ as follows. We let the support of the first term coincide with $\mathbf{p}$, i.e., ${\mathbf{Q}}_0=\left(p_0,p_1,\cdots ,p_n,0,0,\cdots ,0,\cdots \right)$, and we define the other terms by recurrence:

$${\mathbf{Q}}_{k+1}=\left(Q_{k,0},Q_{k,1},\cdots ,Q_{k,n+k-1},\left(1-\alpha \right)Q_{k,n+k},\alpha Q_{k,n+k},0,0,\cdots ,0,\cdots \right).$$

Now, we expand the recursion relation for the Shannon entropy, $H\left({\mathbf{Q}}_{k+1}\right)$, and guessing entropy $\mathbb{E}\left[G\left({\mathbf{Q}}_{k+1}\right)\right]$, which are analogous to Equations (5) and (6), respectively, to obtain:

$$\begin{array}{cc}\hfill H\left({\mathbf{Q}}_{k+1}\right)& =H\left({\mathbf{Q}}_k\right)+Q_{k,n+k}h\left(\alpha \right)=H\left({\mathbf{Q}}_k\right)+{\alpha }^k{p}_{n}h\left(\alpha \right)\hfill \\ \hfill & =H\left(\mathbf{p}\right)+p_{n}h\left(\alpha \right)\sum _{j=0}^k{\alpha }^j=H\left(\mathbf{p}\right)+p_{n}h\left(\alpha \right)\frac{1-{\alpha }^{k+1}}{1-\alpha },\hfill \\ \hfill \mathbb{E}\left[G\left({\mathbf{Q}}_{k+1}\right)\right]& =\mathbb{E}\left[G\left({\mathbf{Q}}_k\right)\right]+\alpha Q_{k,n+k}=\mathbb{E}\left[G\left({\mathbf{Q}}_k\right)\right]+{\alpha }^{k+1}{p}_n\hfill \\ \hfill & =\mathbb{E}\left[G\left(\mathbf{p}\right)\right]+p_n\sum _{j=0}^k{\alpha }^{j+1}=\mathbb{E}\left[G\left(\mathbf{p}\right)\right]+p_n\alpha \frac{1-{\alpha }^{k+1}}{1-\alpha },\hfill \end{array}$$

so, at each step k, we can use ${\mathbf{Q}}_{k+1}$ to obtain an approximation of $\mathbb{E}\left[G\left({\mathbf{Q}}_k\right)\right]$ that is tighter than Massey’s. This procedure allows us to obtain ever tighter approximations of the guessing entropy.

For example, fixing an arbitrary mixing rate $\alpha >0$ and step number $k>0$, we apply the Massey inequality for the distribution ${\mathbf{Q}}_{k+1}$ and find that as per Equation (7),

$$\begin{array}{c}\hfill \mathbb{E}\left[G\left({\mathbf{Q}}_k\right)\right]=\mathbb{E}\left[G\left({\mathbf{Q}}_{k+1}\right)\right]-\alpha Q_{k,n+k}\ge 2^{H\left({\mathbf{Q}}_{k+1}\right)-2}+1-\alpha Q_{k,n+k}>2^{H\left({\mathbf{Q}}_{\mathbf{k}}\right)-2}+1.\end{array}$$

Now, this refinement trickles down producing ever tighter bounds on the guessing entropy of lower index elements of the list up to the initial element $\mathbf{p}={\mathbf{Q}}_0$,

$$\begin{array}{cc}\hfill \mathbb{E}\left[G\left(\mathbf{p}\right)\right]& =\mathbb{E}\left[G\left({\mathbf{Q}}_k\right)\right]-p_n\alpha \frac{1-{\alpha }^k}{1-\alpha }=\mathbb{E}\left[G\left({\mathbf{Q}}_k\right)\right]+\sum _{j=0}^{k-1}\left(\mathbb{E}\left[G\left({\mathbf{Q}}_j\right)\right]-\mathbb{E}\left[G\left({\mathbf{Q}}_{j+1}\right)\right]\right)\hfill \\ \hfill & \ge 2^{H\left({\mathbf{Q}}_k\right)-2}+1+\sum _{j=0}^{k-1}\left(\mathbb{E}\left[G\left({\mathbf{Q}}_j\right)\right]-\mathbb{E}\left[G\left({\mathbf{Q}}_{j+1}\right)\right]\right)\hfill \\ \hfill & >2^{H\left({\mathbf{Q}}_{k-1}\right)-2}+1+\sum _{j=0}^{k-2}\left(\mathbb{E}\left[G\left({\mathbf{Q}}_j\right)\right]-\mathbb{E}\left[G\left({\mathbf{Q}}_{j+1}\right)\right]\right)\hfill \\ \hfill & >\cdots >2^{H\left({\mathbf{Q}}_0\right)-2}+1=2^{H\left(\mathbf{p}\right)-2}+1,\hfill \end{array}$$

where the tightest of the enumerated bounds is:

$$\begin{array}{cc}\hfill \mathbb{E}\left[G\left(\mathbf{p}\right)\right]& \ge 2^{H\left({\mathbf{Q}}_k\right)-2}+1+\sum _{j=0}^{k-1}\left(\mathbb{E}\left[G\left({\mathbf{Q}}_j\right)\right]-\mathbb{E}\left[G\left({\mathbf{Q}}_{j+1}\right)\right]\right)\hfill \\ \hfill & =2^{H\left(\mathbf{p}\right)+p_{n}h\left(\alpha \right)\frac{1-{\alpha }^k}{1-\alpha }-2}+1-p_n\alpha \frac{1-{\alpha }^k}{1-\alpha },\hfill \end{array}$$

which as we have shown increases with k up to the limit:

$$\begin{array}{c}\hfill \mathbb{E}\left[G\left(\mathbf{p}\right)\right]\ge 2^{H\left(\mathbf{p}\right)+p_n\frac{h\left(\alpha \right)}{1-\alpha }-2}+1-p_n\frac{\alpha }{1-\alpha }\forall \alpha \in \left[0,1/2\right],\end{array}$$

(8)

where the optimal value of $\alpha$ will be discussed in the following sections.

We now conclude this section with an even stronger refinement of the Massey inequality.

Theorem 2.

For strictly positive descending probabilistic vectors $\mathbf{p}\in {\mathbb{R}}^n$ with $H\left(\mathbf{p}\right)\ge 2$, we have:

$$\begin{array}{cc}\hfill \mathbb{E}\left[G\left(\mathbf{p}\right)\right]& \ge \underset{\alpha \in \left[0,1/2\right]}{sup}{2}^{H\left(\mathbf{p}\right)+\frac{h\left(\alpha \right)}{1-\alpha }{p}_n-2}+1-\frac{\alpha }{1-\alpha }{p}_n\hfill \\ \hfill & \ge 2^{H\left(\mathbf{p}\right)+2p_n-2}+1-p_n>2^{H\left(\mathbf{p}\right)-2}+1.\hfill \end{array}$$

Proof. 

The first inequality follows taking the supremum over $\alpha$ in Equation (8) and the rest taking $\alpha =1/2$. □

Furthermore, note that Theorem 2 is a refinement over the strongest result of [5], namely Equation (1).

2.3. The Massey Gap

In this section, we reevaluate the scaffolding we used to prove Theorem 2 and re-frame our previous technique as an optimization process applied to an objective function we name the Massey gap, the function $\mathcal{M}\left(\mathbf{x}\right)$ described below.

The first observation that we make is that Theorem 2 also applies to descending probabilistic infinite sequences $\mathbf{p}$, by taking n to be the size of its support, $\left|\mathbf{p}\right|$. Moreover, we notice that the core of the argument was not the precise construction of the list $\left\{{\mathbf{Q}}_k\right\}$, but rather the fact that:

$$2^{H\left({\mathbf{Q}}_k\right)-2}+1+\mathbb{E}\left[G\left({\mathbf{Q}}_{k-1}\right)\right]-\mathbb{E}\left[G\left({\mathbf{Q}}_k\right)\right]>2^{H\left({\mathbf{Q}}_{k-1}\right)-2}+1\forall k\ge 1,$$

i.e., the sequences $\left\{{\mathbf{Q}}_k\right\}$ incrementally optimize the objective function $\mathcal{M}\left(\mathbf{x}\right)=\mathbb{E}\left[G\left(\mathbf{x}\right)\right]-2^{H\left(\mathbf{x}\right)-2}-1$, with tightness being achieved if and only if ${lim}_{k\to \infty }\mathcal{M}\left({\mathbf{Q}}_k\right)=0$.

Motivated by this insight, we now take an alternative view and focus not on the precise sequence of distributions, but rather the objective function $\mathcal{M}\left(\mathbf{x}\right)$, which we name the Massey gap.

This view leads us to the following useful result:

Remark 1.

Let $\left\{{\mathbf{X}}_k\right\}$ be a list of probabilistic sequences with $H\left({\mathbf{X}}_k\right)\ge 2$ and $\mathcal{M}\left({\mathbf{X}}_k\right)\ge \mathcal{M}\left({\mathbf{X}}_{k+1}\right)\ge 0$ for all values of k. Then, the sequence $y_k=2^{H\left({\mathbf{X}}_k\right)-2}+1+\mathbb{E}\left[G\left({\mathbf{X}}_0\right)\right]-\mathbb{E}\left[G\left({\mathbf{X}}_k\right)\right]$ is increasing.

Notice that this remark provides a sequence $\left\{y_k\right\}$ of ever tighter lower bounds on $\mathbb{E}\left[G\left({\mathbf{X}}_0\right)\right]$, the weakest of which, $y_0$, reduces to the Massey inequality, while the strongest is given by ${lim}_{k\to \infty }{y}_k$. As an example, considering the list ${\mathbf{X}}_k:={\mathbf{Q}}_k$, this limit coincides with the result of Theorem 2.

2.4. Mixing Probabilities

In the previous section, we presented a refinement to (1) by generalizing the tail rebalancing method of [5]. In this section, we investigate whether we can obtain even better results by applying the same rebalancing technique in the middle of the distribution, rather than at the tail.

In the previous section, we showed how Theorem 2 follows from Remark 1 for some particular $\left\{{\mathbf{X}}_k\right\}$. However, this option is not necessarily unique. In this section, we consider alternative methods of optimizing the Massey gap, that is we fix an arbitrary intermediary configuration ${\mathbf{X}}_k$, and we find conditions on the existence of available ${\mathbf{X}}_{k+1}$ that decrease the Massey gap, i.e., $\mathcal{M}\left({\mathbf{X}}_k\right)\ge \mathcal{M}\left({\mathbf{X}}_{k+1}\right)$.

In order to simplify the notation, for the remainder of this section, we denote the available configuration ${\mathbf{X}}_k$ by $\mathbf{p}$ and the potential next configuration ${\mathbf{X}}_{k+1}$ by $\mathbf{q}$.

Now, we are ready for a trick similar to the one in Section 2.2: we fix an arbitrary integer $i\in \mathbb{N}$ such that $p_i\ne 0$ and an arbitrary mixing rate $\alpha \in \left(0,1/2\right]$ and consider $\mathbf{q}$ of the form:

$$\mathbf{q}=\left(p_0,p_1,\cdots ,p_{i-1},\left(1-\alpha \right)p_i+\alpha p_{i+1},\alpha p_i+\left(1-\alpha \right)p_{i+1},p_{i+2},p_{i+3},\cdots \right),$$

(9)

with the same support as $\mathbf{p}$.

For $\mathbf{q}$ as defined above in Equation (9), we find that $\mathbb{E}\left[G\left(\mathbf{q}\right)\right]=\mathbb{E}\left[G\left(\mathbf{p}\right)\right]+\alpha \left(p_i-p_{i+1}\right)$ while:

$$H\left(\mathbf{q}\right)=H\left(\mathbf{p}\right)+\left(p_i+p_{i+1}\right)\left(h\left(\frac{q_i}{q_i+q_{i+1}}\right)-h\left(\frac{p_i}{p_i+p_{i+1}}\right)\right);$$

thus, the condition $\mathcal{M}\left(\mathbf{p}\right)\ge \mathcal{M}\left(\mathbf{q}\right)$ is equivalent to:

$$h\left(\frac{q_i}{q_i+q_{i+1}}\right)\ge h\left(\frac{p_i}{p_i+p_{i+1}}\right)+\frac{1}{p_i+p_{i+1}}log\left(1+\frac{\alpha \left(p_i-p_{i+1}\right)}{2^{H\left(\mathbf{p}\right)-2}}\right),$$

(10)

which, as we have seen in the previous section, always holds at least for some i, namely $i=\left|\mathbf{p}\right|$.

Now, we check whether alternative values for i may be eligible, i.e., whether we may mix probabilities in the middle of the distribution. Using the inequality (Tighter bounds may also be used.) $1-h\left(\frac{1+y}{2}\right)\le \frac{y^2}{2ln2}$ with $y=\frac{q_i-q_{i+1}}{q_i+q_{i+1}}=\left(1-2\alpha \right)\frac{p_i-p_{i+1}}{p_i+p_{i+1}}$, we find that:

$$h\left(\frac{q_i}{q_i+q_{i+1}}\right)\ge 1-{\left(1-2\alpha \right)}^2{\left(\frac{p_i-p_{i+1}}{p_i+p_{i+1}}\right)}^2$$

Thus, to satisfy $\mathcal{M}\left(\mathbf{p}\right)\ge \mathcal{M}\left(\mathbf{q}\right)$, corresponding to a decrease in the Massey gap, it is sufficient that:

$$\begin{array}{cc}\hfill 1-{\left(1-2\alpha \right)}^2{\left(\frac{p_i-p_{i+1}}{p_i+p_{i+1}}\right)}^2\ge & h\left(\frac{p_i}{p_i+p_{i+1}}\right)+\frac{1}{p_i+p_{i+1}}log\left(1+\frac{\left(p_i-p_{i+1}\right)}{2^{H\left(\mathbf{p}\right)-2}}\right).\hfill \end{array}$$

i.e.,

$$\alpha \ge \frac{1}{2}-\frac{1}{2}\frac{p_i+p_{i+1}}{p_i-p_{i+1}}{\left[1-h\left(\frac{p_i}{p_i+p_{i+1}}\right)-\frac{1}{p_i+p_{i+1}}log\left(1+\frac{\left(p_i-p_{i+1}\right)}{2^{H\left(\mathbf{p}\right)-2}}\right)\right]}^{1/2}\ge 0,$$

(11)

so depending on the value of $H\left(\mathbf{p}\right)$, multiple indices i and mixing ratios $\alpha$ may be selected for rebalancing, as outlined in the following

Theorem 3.

Let $\mathbf{p}$ be a descending probabilistic sequence with $H\left(\mathbf{p}\right)\ge 2$ and $i\in \mathbb{N}$ be such that $p_i\ne 0$ and $\alpha \in \left(0,1/2\right]$ satisfy Equation (10) or, sufficiently, Equation (11). Then, defining $\mathbf{q}$ as in Equation (9), we have:

$$\mathbb{E}\left[G\left(\mathbf{p}\right)\right]\ge 2^{H\left(\mathbf{q}\right)-2}+1-\alpha \left(p_i-p_{i+1}\right)\ge 2^{H\left(\mathbf{p}\right)-2}+1.$$

Proof. 

We have shown that Equation (11) is a sufficient condition for us to have $\mathcal{M}\left(\mathbf{p}\right)\ge \mathcal{M}\left(\mathbf{q}\right)$; thus, applying Remark 1 to the sequence $\mathbf{X}=\left(\mathbf{p},\mathbf{q},\mathbf{q},\mathbf{q},\cdots \right)$, the result follows. □

We conclude this section by remarking that Theorem 3 has the potential to further refine bounds obtained using Theorem 1 depending on the form of the distribution $\mathbf{p}$.

2.5. Strict Refinement and Optimal Mixing Rate

In this section, we consider the problem of finding the optimal mixing rate $\alpha$.

We notice that $\alpha$ must be a critical point of the difference between the sides of (10), which is:

$$f\left(\alpha \right)=\frac{p_i-p_{i+1}}{p_i+p_{i+1}}\left[log\frac{q_i}{q_{i+1}}-\frac{1}{\alpha \left(p_i-p_{i+1}\right)+2^{H\left(\mathbf{p}\right)-2}}\right].$$

For example, we notice that if $H\left(\mathbf{p}\right)\ge 2$, then the right denominator is greater than one and:

$$f\left(1/2\right)=-\frac{p_i-p_{i+1}}{p_i+p_{i+1}}\frac{1}{\alpha \left(p_i-p_{i+1}\right)+2^{H\left(\mathbf{p}\right)-2}}<0$$

so the mixing rate $\alpha =1/2$ is never optimal; thus, Theorem 2 is a strict refinement over the work of [5].

Investigating whether $\mathbf{q}$ actually decreases the Massey gap, we now consider the optimality of $\alpha =0$. In this case, $q_i=p_i$ and $q_{i+1}=p_{i+1}$; thus, when $p_i\ge p_{i+1}{2}^{2^{2-H\left(\mathbf{p}\right)}}$, then $f\left(0\right)>0$, so $\alpha =0$ is certainly sub-optimal as $\mathcal{M}\left(\mathbf{q}\right)-\mathcal{M}\left(\mathbf{p}\right)$ is increasing around zero.

Moreover, we note that the simple upper bound:

$$f\left(\alpha \right)\le \frac{p_i-p_{i+1}}{p_i+p_{i+1}}\left[log\frac{q_i}{q_{i+1}}-\frac{1}{\left(p_i-p_{i+1}\right)+2^{H\left(\mathbf{p}\right)-2}}\right]$$

imposes that the optimal mixing ratio ${\alpha }^{*}$ satisfy:

$$\frac{p_{i+1}+p_i}{q_{i+1}}-1=\frac{q_i}{q_{i+1}}\ge 2^{{\left(\left(p_i-p_{i+1}\right)+2^{H\left(\mathbf{p}\right)-2}\right)}^{-1}}$$

i.e.,

$$q_{i+1}\le \frac{p_i+p_{i+1}}{1+2^{{\left(\left(p_i-p_{i+1}\right)+2^{H\left(\mathbf{p}\right)-2}\right)}^{-1}}}$$

and hence:

$$\begin{array}{cc}\hfill {\alpha }^{*}\le & \frac{1}{p_i-p_{i+1}}\left[-p_{i+1}+\frac{p_i+p_{i+1}}{1+2^{{\left(\left(p_i-p_{i+1}\right)+2^{H\left(\mathbf{p}\right)-2}\right)}^{-1}}}\right]\le \frac{1}{2},\hfill \end{array}$$

(12)

which is a non-trivial bound.

For example, to apply Equation (12) to the particular case of mixing only at the tail of the distribution, i.e., Theorem 2, we take $i=n$, hence $p_{i+1}=0$, and substituting in Equation (12), we obtain the requirement:

$${\alpha }^{*}\le 1/\left(1+2^{1/\left(p_n+2^{H\left(\mathbf{p}\right)-2}\right)}\right)<1/2$$

on the mixing rate realizing the supremum, so we have a closed-form strict refinement over Equation (2):

Theorem 4.

Let $\mathbf{p}$ be a descending probabilistic sequence with $H\left(\mathbf{p}\right)\ge 2$ and $i\in \mathbb{N}$ be such that $p_i>2^{2^{2-H\left(\mathbf{p}\right)}}{p}_{i+1}$, and let $\alpha =1/\left(1+2^{1/\left(p_n+2^{H\left(\mathbf{p}\right)-2}\right)}\right)$. Then, defining $\mathbf{q}$ as in Equation (9), we have:

$$\mathbb{E}\left[G\left(\mathbf{p}\right)\right]\ge 2^{H\left(\mathbf{q}\right)-2}+1-\alpha \left(p_i-p_{i+1}\right)>2^{H\left(\mathbf{p}\right)+p_{\left|\mathbf{p}\right|}-2}+1-\frac{1}{2}{p}_{\left|\mathbf{p}\right|}\ge 2^{H\left(\mathbf{p}\right)-2}+1.$$

We end this section by remarking that Theorem 4 can be used in conjunction with Theorem 2, complementing the construction of the list of sequences defined in the proof of Theorem 2.

3. Discussion

In this section, we discuss in a more detailed manner the context of our work, evaluate the saturation conditions of the discussed bounds, and present prospects for future work.

We begin our discussion by taking a closer look at the most prevalently used form of the Massey inequality, namely Equation (3). Notably, this inequality only holds for the case that $H\left(\mathbf{p}\right)\ge 2$, suggesting that its nice shape is a case of form vs. function, as illustrated below.

Massey [1] found that the set of probabilistic vectors of arbitrary length, which have a fixed guessing entropy A, are convex for any given A; thus, by strict convexity, the Shannon entropy attains its maximum over this set at an interior point. Using Lagrange multipliers, Massey found that the critical $\mathbf{p}$ must be a geometric distribution, and imposing the condition $\mathbb{E}\left[G\left(\mathbf{p}\right)\right]=A$, he found $p_i=\frac{1}{A-1}{\left(1-\frac{1}{A}\right)}^i$, hence:

$$H\left(\mathbf{p}\right)=log\left(A-1\right)+log\left[{\left(1-\frac{1}{A}\right)}^{-A}\right],$$

(13)

Because this distribution maximizes the Shannon entropy, for arbitrary distributions $\mathbf{p}$, we have:

$$H\left(\mathbf{p}\right)\le log\left(\mathbf{E}\left[G\left(\mathbf{p}\right)\right]-1\right)+log\left[{\left(1-\frac{1}{\mathbf{E}\left[G\left(\mathbf{p}\right)\right]}\right)}^{-\mathbf{E}\left[G\left(\mathbf{p}\right)\right]}\right],$$

(14)

an unwieldy form that is saturated only for geometric distributions. Thus, the bound in Equation (14) is only ever saturated when $\left|\mathbf{p}\right|=\infty$; therefore, the Massey inequality is never tight for distributions encountered in most applications such as cryptography.

To find a simpler form of Equation (14), Massey noticed that when $A\ge 2$, the second term of Equation (13) is at most two, with equality if and only if $A=2$. Thus, when $H\left(\mathbf{p}\right)\ge 2$, $\mathbf{E}\left[G\left(\mathbf{p}\right)\right]\ge 2^{H\left(\mathbf{p}\right)-2}+1$. However, the simplicity of this form impacts its tightness, as we can remark:

Remark 2.

While the well-known form of Massey’s lower bound Equation (3) is only saturated when $p_i=1/2^i$, his original inequality Equation (14) is saturated by all geometric distributions, i.e., for all values of $\mathbb{E}\left[G\left(\mathbf{p}\right)\right]$.

From this remark, we conclude that any inequality derived from an application of the Massey inequality is saturated if and only if the distribution for which the Massey inequality is ultimately applied is the geometric distribution of ratio $1/2$. It follows that for the initial result of [5], Equation (2), as well as for our own initial result from Theorem 1, the bounds are saturated only for the geometric distribution of the ratio $1/2$, even if they are tighter than the Massey inequality.

However, when those inequalities are applied after an unbounded number of steps, it is possible to obtain bounds saturated by more inputs. Indeed, for the stronger result of [5], as well as our own stronger result from Theorem 2, the bounds are saturated for any truncation of the geometric distribution of the ratio $1/2$, i.e., when the original distribution has the form $\mathbf{p}=\left(1/2,1/4,\cdots ,1/2^{n-2},1/2^{n-1},1/2^{n-1}\right)$ for some $n\ge 2$, i.e., for a countable range of $\mathbb{E}\left[G\left(\mathbf{p}\right)\right]$.

It is then natural to ask how the tightness improvement in Theorems 1 and 2 is reflected in saturation conditions. To this end, we notice that the limiting factor is the requirement that the limit of the list of probabilistic distributions is required to be the geometric distribution of the ratio $1/2$; hence, for saturation, we necessarily have $\alpha =1/2$, i.e., this is a direct consequence of using the nice form of the Massey inequality, Equation (3). As such, we propose the open question of finding a similar technique exploiting the Massey gap with respect to the original form of the Massey inequality, Equation (14).

We continue our discussion by comparing the many refinements to the Massey inequality that we have presented on a simple numerical example. Let $\mathbf{p}=\left(p_1,\cdots ,p_{10}\right)$ be the descending probabilistic vector containing the first nine terms of the probabilistic geometric sequence of the ratio $0.8$,

$$\mathbf{p}=\left(\begin{array}{cccccccccc}\frac{1}{5}& \frac{4}{25}& \frac{262144}{1953125}& \frac{16}{125}& \frac{64}{625}& \frac{256}{3125}& \frac{1024}{15625}& \frac{4096}{78125}& \frac{16384}{390625}& \frac{65536}{1935125}\end{array}\right),$$

with Shannon entropy $H\left(\mathbf{p}\right)\approx 3.32516>log\frac{e}{1-e/4}>2$ and guessing entropy $\mathbb{E}\left[G\left(\mathbf{p}\right)\right]\approx 4.02939$. The state-of-the-art provides various lower bounds on the guessing entropy: $3.18126$ from the Massey inequality, $3.21581$ from the weaker result Equation (2) of [5], $3.25157$ from their stronger result Equation (1), and $3.20977$ from Rioul’s inequality [6,7]. Our new lower bounds for this example are: $3.21767$ as the supremum in Theorem 1 taking $\alpha \approx 0.38977$ (improving Equation (2)), $3.25157$ as the supremum in Theorem 2 taking $\alpha =0.5$ (matching Equation (1)), and $3.21751$ as the explicit value in Theorem 4 taking $i=10$ (improving Equation (2)). In this particular case, Theorem 3 cannot be applied for $i\ne 10$ as $p_i/p_{i+1}\le 1.2$, while $2^{2^{2-H\left(\mathbf{p}\right)}}\approx 1.3189$. We notice that for this example, Theorem 2 matches the stronger result of [5].

The final topic of our discussion addresses a direction that was previously left unaddressed in the literature, rebalancing the initial distribution. We take an example and use it to showcase the effectiveness of the approach in Theorem 3. Considering the initial distribution $\mathbf{p}=\left(5/8,1/8,1/8,1/2^{16},1/2^{16},\cdots ,1/2^{16}\right)$, we notice that $H\left(\mathbf{p}\right)\ge 1/8log{2}^{16}=2$, and moreover, ${\mathbf{p}}_1/{\mathbf{p}}_2=5\ge 2^{2^{2-H\left(\mathbf{p}\right)}}$. Hence, according to Section 2.4, we may rebalance our distribution, even at its head. Explicitly taking $i=1$ and $\alpha =1/5<1/\left(1+2^{1/\left(2^{-16}+2^{H\left(\mathbf{p}\right)-2}\right)}\right)$, we construct $\mathbf{q}=\left(21/40,19/40,1/8,1/2^{16},1/2^{16},\cdots ,1/2^{16}\right)$ such that $\mathcal{M}\left(\mathbf{p}\right)\ge \mathcal{M}\left(\mathbf{q}\right)\ge 0$. From Remark 1, it follows that each such rebalancing can be prepended to the list construction in Section 2.2.

However, continuing our example, we note that ${\mathbf{q}}_1/{\mathbf{q}}_2=2^{13}$, so taking $i=3$ and $\alpha =1/5$, we may once again apply this procedure to obtain $\mathbf{r}=\left(21/40,19/40,(1+2^{-15})/10,1/40+1/2^{14},1/2^{16},\cdots ,1/2^{16}\right)$ with $\mathcal{M}\left(\mathbf{p}\right)\ge \mathcal{M}\left(\mathbf{q}\right)\ge \mathcal{M}\left(\mathbf{r}\right)\ge 0$. Noting that now ${\mathbf{r}}_2/{\mathbf{r}}_3\ge 19/5=4.25$ and ${\mathbf{r}}_4/{\mathbf{r}}_5\ge 2^{16}/40=1638.4$, the procedure can continue, gradually trickling down mass from the head up to the tail.

We emphasize that the indices eligible for rebalancing change at every step and the indices that were once ineligible (such as 2 and 4) can become eligible at future steps. In fact, if we rebalance at Index 4, we notice that the fourth position in the newly created vector would be smaller than $(1+2^{-8})/50$, so Index 3 would once again become eligible. It follows that some indices can become eligible for rebalancing more than once.

In conclusion, in this paper, we strictly refined the work of [5] presenting a generalized approach in the form of Theorem 2, as well as a data-dependent numerical procedure resulting in tighter refinements. The rebalancing technique in Section 2.4 and Theorem 3 provides a significant improvement to the procedure in Theorem 2; thus, for future work, we propose a precise study of the convergence of this method on real data, as well as a follow-up work showing how the same method can be applied to other inequalities, such as Rioul’s inequality, Equation (4).