An Intrinsic Characterization of Shannon’s and Rényi’s Entropy

Abstract

All characterizations of the Shannon entropy include the so-called chain rule, a formula on a hierarchically structured probability distribution, which is based on at least two elementary distributions. We show that the chain rule can be split into two natural components, the well-known additivity of the entropy in case of cross-products and a variant of the chain rule that involves only a single elementary distribution. The latter is given as a proportionality relation and, hence, allows a vague interpretation as self-similarity, hence intrinsic property of the Shannon entropy. Analogous characterizations are given for the Rényi entropy and its limits, the min-entropy and the Hartley entropy.

1. Introduction

The Shannon entropy H has been fully characterized by different sets of conditions [1,2,3,4]. The conditions themselves have been investigated deeply by [5,6], for instance. All characterizations use variants of the so-called chain rule, which can be summarized as follows. Let $p=\{p_1,\dots ,p_n\}$ and $q=\{q_1,\dots ,q_m\}$ be discrete probability distributions, and

$$\begin{array}{c}\hfill p_{k,q}=\{p_1,\dots ,p_{k-1},p_k{q}_1,\dots ,p_k{q}_m,p_{k+1},\dots ,p_n\}.\end{array}$$

(1)

The chain rule states that

$$\begin{array}{c}\hfill H\left(p_{k,q}\right)=H\left(p\right)+p_{k}H\left(q\right).\end{array}$$

(2)

The construction in Equation (1) can be iterated, i.e., p is extended at several positions $k_i$ with different distributions $q^{\left(i\right)}$. In case of two positions, this is

$$p_{k_1,q^{\left(1\right)};k_2,q^{\left(2\right)}}:={\left(p_{k_2,q^{\left(2\right)}}\right)}_{k_1,q^{\left(1\right)}},$$

where $k_1<k_2$. The most elementary version of the chain rule is due to Faddeev [2], where q in Equation (1) is restricted to a Bernoulli distribution. Carcassi et al. [7] considered the full version of Equation (1), where q is arbitrary, as preferable from a didactical point of view, since this allows an immediate interpretation in various areas of application. Often, the completely iterated version,

$$\begin{array}{c}\hfill H\left(p_{1,q^{\left(1\right)};\dots ;n,q^{\left(n\right)}}\right)=H\left(p\right)+\sum _{k=1}^n{p}_{k}H\left(q^{\left(k\right)}\right),\end{array}$$

(3)

is taken [4], since it allows the practical interpretation that first an alphabet ${\alpha }_k$ is chosen with probability $p_k$ and subsequently a letter within ${\alpha }_k$ according to $q^{\left(k\right)}$.

Baez et al. [8] give the first algebraic approach, namely, in terms of information loss. The latter is defined as $F_{p,q}=H\left(p\right)-H\left(q\right)$, where the probability distribution q is a function of p, hence $F_{p,q}$ quantifies the change in entropy of the transformation. The information loss $F_{p,q}$ is uniquely characterized by a number of properties, including $F_{p,q}$, being a convex linear map, i.e., $H(\lambda p\cup (1-\lambda \left)q\right)=\lambda H\left(p\right)+(1-\lambda )H\left(q\right)$, which might be considered as a restatement of Faddeev’s chain rule.

Common to all versions of the chain rule is that they describe situations where, canonically, at least two different distributions, p and q, are involved. Ebanks et al. ([6], Definition 3.1.1) introduced an alternative notation of Faddeev’s condition, which formally considers only one distribution but, nevertheless, relies on computing the Shannon entropy of two distributions, namely, p and a Bernoulli distribution, to state the chain rule.

The Rényi entropy generalizes the Shannon entropy and can be fully characterized via quasi-arithmetic means ([5], Section 5). Another approach to characterize the Rényi entropy relies on a transform. The order-$\alpha$ information energy S with Onicescu’s information energy as a special case [9] is, up to a multiplicative constant, the sum of the $\alpha$-powers of the probabilities. Pardo [10] gives a full characterization by means of a chain rule that is similar to Faddeev’s variant,

$$\begin{array}{c}\hfill S\left(p_{k,q}\right)=S\left(p\right)+p_k^{\alpha }(S\left(q\right)-\mathrm{const}),\end{array}$$

(4)

where q is a Bernoulli distribution and the constant depends on $\alpha$ and the normalization of S. Pardo [10] also generalizes this equation by introducing weighted sums. Leinster ([4], Theorem 4.5.1) characterizes the information energy by a formula that is analogous to the fully iterated version of the chain rule (3).

The limits of the Rényi entropy as its parameter goes to 0 and ∞ are the Hartley entropy [11] and the min-entropy [12], respectively. Mathematical characterizations of these two entropies are rare, e.g., conical combinations of the Hartley entropy and the Shannon entropy are characterized in [13] as the only reasonable simultaneous solution to the chain rule and the subadditivity property. Ref. [13] also characterized the Hartley entropy as being insensitive. An advanced investigation of the min-entropy exists in form of the Fourier min-entropy [14].

Schlather [15] deals with a rather general approach to the entropy, which is axiomatically based on the additivity assumption of the entropy for independent systems. Under certain conditions, called scale invariance there, the entropy is unique. This result, however, is not applicable here, since it is closely related to scalar real numbers and not to probability distributions. Yet, the characterizations given here are in this spirit.

To be specific, let $q=p$ in Equation (1). Then, we obtain

$$\begin{array}{c}\hfill p_{k,p}=\{p_1,\dots ,p_{k-1},p_k{p}_1,\dots ,p_k{p}_n,p_{k+1},\dots ,p_n\},\end{array}$$

(5)

and, hence, a proportionality relation for the Shannon entropy by Equation (2),

$$\begin{array}{c}\hfill H\left(p_{k,p}\right)=(1+p_k)H\left(p\right).\end{array}$$

Theorem 1 in Section 2 states that the chain rule in Khinchin’s (1953) characterization [1,2] can be split into two canonical components, the additivity property and the proportionality between $H\left(p_{k,p}\right)$ and $H\left(p\right)$. The characterization of the Rényi entropy is in analogy to the characterization of the Shannon entropy, except that the functional Equation (4), i.e.,

$$\begin{array}{c}\hfill S\left(p_{k,p}\right)=(1+p_k^{\alpha })S\left(p\right)-p_k^{\alpha },\end{array}$$

reflects the proportionality relation. Related characterizations also exist for the min-entropy and the Hartley entropy, cf. Theorems 3 and 4. All technical parts are postponed to Section 3. There, the geometric distribution plays an overwhelming role. Section 4 illuminates practical and theoretical aspects of this paper. Section 5 finalizes with a digest of our approach.

2. Results

We denote by $\{.\}$ an ordered set of non-negative numbers and let

$$\begin{array}{ccc}\hfill a\{p_1,p_2,\dots \}& =& \{a{p}_1,a{p}_2,\dots \},a\ge 0,\hfill \\ \hfill \{p_1,p_2,\dots \}\cup \{q_1,q_2,\dots \}& =& \{p_1,p_2,\dots ,q_1,q_2,\dots \}.\hfill \end{array}$$

(6)

As we will address only the very first elements of an ordered set, the intuitive, but sloppy, notation on the right-hand side of Equation (6) is sufficient.

Let us denote by P the set of all discrete probability distributions and by $P_n\subset P$ the set of distributions with at most n atomic events that have a probability greater than zero. We denote such a probability distribution by $\{p_1,\dots ,p_n\}\in P_n$. The uniform distribution is denoted by $U_n=\{1/n,\dots ,1/n\}$. Let $P_0={\bigcup }_n{P}_n$ be the set of all discrete probability distributions with finite support.

Reordering the elements in Equation (5) will turn out to be useful in the proofs. We define

$$\begin{array}{ccc}\hfill p_{\widehat{k}}& =& \{p_i:i\ne k\},\hfill \\ \hfill {\partial }_{k}p& =& p_{k}p\cup p_{\widehat{k}},\hfill \end{array}$$

(7)

where $p=\{p_1,p_2,\dots \}\in P$. For instance,

$$\begin{array}{ccc}\hfill {\partial }_3\{p_1,p_2,p_3\}& =& \{p_3{p}_1,p_3{p}_2,p_3{p}_3,p_1,p_2\}.\hfill \end{array}$$

We use the convention $0log0=0$.

Theorem 1.

Let $H:P\to [0,\infty ]$ be a function. Then, the following two assertions are equivalent:

1. 

H is the Shannon entropy,

$$H\left(p\right)=-\sum _{p_i\in p}{p}_{i}log{p}_i,$$

up to a positive multiplicative constant.

2. 

H has the following properties:

(a) 

H is a continuous function in the topology of convergence in distribution;

(b) 

$H\left(\{p_1,\dots ,p_n\}\right)=H\left(\{p_{\pi \left(1\right)},\dots ,p_{\pi \left(n\right)}\}\right)$ for $\{p_1,\dots ,p_n\}\in P_0$ and any permutation π;

(c) 

$H\left(\{p_1,\dots ,p_n\}\right)=H\left(\{0,p_1,\dots ,p_n\}\right)$ for all $\{p_1,\dots ,p_n\}\in P_0$;

(d) 

$H\left(p\right)\le H\left(U_n\right)<\infty$ for all $p\in P_n$, $n\in \mathbb{N}$;

(e) 

$H\left(p\right)\in (0,\infty )$ if p is a nondegenerate geometric distribution;

(f) 

$H(p\times q)=H\left(p\right)+H\left(q\right)$ for all $p,q\in P$;

(g) 

a function $f:[0,1]\to [0,\infty )$ exists such that

$$\begin{array}{c}\hfill H\left({\partial }_{1}p\right)=(1+f\left(p_1\right))H\left(p\right),p\in P.\end{array}$$

(8)

Conditions 2(a)–2(g) can be regarded as interpretable, hence intrinsic properties ([5], Section 1.2). For this reason, we prefer Condition 2(d) over a monotonicity assumption on H. Note that Conditions 2(a)–2(d) are Khinchin’s (1953) conditions as given in [2] except for the chain rule, which is replaced by Conditions 2(e)–2(g).

Remark 1.

Note that, here and in the subsequent theorems, the function f is not unique, since it is undetermined at 1. A convenient choice is to take $f\left(1\right)={lim}_{a\to 1}f\left(a\right)$.

Remark 2.

Our approach has a simple, but nice implication. Let $p,q\in P_0$. Then,

$$\begin{array}{ccc}\hfill \mathrm{KL}({\partial }_{1}p,{\partial }_{1}q)& =& (1+p_1)\mathrm{KL}(p,q),\hfill \end{array}$$

where $\mathrm{KL}$ denotes the Kullback–Leibler divergence.

Theorem 2.

Let $H:P\to [0,\infty ]$ be a function. Then, the following two assertions are equivalent:

1. 

H is the Rényi entropy,

$$H\left(p\right)={\displaystyle \frac{1}{1-\alpha }}log\sum _{p_i\in p}{p}_i^{\alpha },$$

for some $\alpha \in (0,\infty )\setminus \left\{1\right\}$, up to a positive multiplicative constant.

2. 

H satisfies Conditions 2(a)–2(f) of Theorem 1. A value $\gamma \in \mathbb{R}\setminus \left\{0\right\}$ and a function $f:[0,1]\to [0,\infty )$ exist such that for $S=e^{\gamma H}$ we have

$$\begin{array}{c}\hfill S\left({\partial }_{1}p\right)=(1+f\left(p_1\right))S\left(p\right)-f\left(p_1\right),p\in P.\end{array}$$

(9)

We denote the max-operator by ∨ and the min-operator by ∧.

Theorem 3.

Let $H:P\to [0,\infty ]$ be a function. Then, the following two assertions are equivalent:

1. 

H is the min-entropy,

$$H\left(p\right)=-log\underset{p_i\in p}{\bigvee }{p}_i,$$

up to a positive multiplicative constant.

2. 

H satisfies Conditions 2(a)–2(f) of Theorem 1. A function $f:[0,1]\to [0,\infty )$ exists, such that for $S=e^{-H}$, we have

$$\begin{array}{c}\hfill f\left(p_1\right)\vee S\left({\partial }_{1}p\right)=S\left(p\right),p\in P.\end{array}$$

(10)

As the Hartley entropy H is discontinuous on P and $H\left(g_a\right)=\infty$, Conditions 2(a) and 2(e) are replaced by Equations (11) and (12), respectively.

Theorem 4.

Let $H:P\to [0,\infty ]$ be a function. Then, the following two assertions are equivalent:

1. 

H is the Hartley entropy,

$$H\left(p\right)=log\sum _{p_i\in p}{\mathbf{1}}_{(0,1]}\left(p_i\right),$$

where ${\mathbf{1}}_{(0,1]}\left(p_i\right)=1$ if $p_i\in (0,1]$, and 0 otherwise.

2. 

H satisfies Conditions 2(b)–2(d) and 2(f) of Theorem 1 and $H\left(U_2\right)\in (0,\infty )$. Further, for $S=e^H$ holds

$$\begin{array}{c}\hfill \underset{n\to \infty }{lim}{\displaystyle \frac{S(\{p_1,\dots ,p_n\}\cup q{\sum }_{i=n+1}^{\infty }{p}_i)}{S(\{p_1,\dots ,p_n\}/{\sum }_{i=1}^n{p}_i)}}\in \{1,\infty \},\{p_1,p_2,\dots \},q\in P.\end{array}$$

(11)

Moreover, $S\left(p\right)<\infty ,p\in P$ with $p_1<1$ implies $S(p_{\widehat{1}}/(1-p_1))<\infty$. A function $f:[0,1]\to [0,\infty )$ exists such that Equation (9) holds true and

$$\begin{array}{c}\hfill S\left({\displaystyle \frac{{\left({\partial }_{1}p\right)}_{\widehat{1}}}{1-p_1^2}}\right)=(1+f\left(p_1\right))S\left({\displaystyle \frac{p_{\widehat{1}}}{1-p_1}}\right),p\in P,p_1<1.\end{array}$$

(12)

3. Proofs

We use the following abbreviations:

$$\begin{array}{ccc}\hfill F_n\left(a\right)& =& \prod _{k=0}^{n-1}\left(1+f\left(a^{2^k}\right)\right),a\in [0,1],\hfill \\ \hfill F\left(a\right)& =& \prod _{k=0}^{\infty }\left(1+f\left(a^{2^k}\right)\right),a\in (0,1).\hfill \end{array}$$

3.1. Functional Equations

Lemma 1.

Let f be a real-valued function on $(0,1)$. The following three assertions are equivalent:

1. 

f is continuous and, for all $a\in (0,1)$, we have

$$\begin{array}{ccc}\hfill f\left(a\right)+f(1-a)& =& 1\hfill \end{array}$$

(13)

$$\begin{array}{ccc}\hfill (1-f(a\left)\right)F\left(a\right)& =& 1;\hfill \end{array}$$

(14)

2. 

f is continuous and, for all $a\in (0,1)$, Equation (13) holds and $f\left(a^2\right)=f^2\left(a\right)$;

3. 

f is the identity.

Proof. 

Let f be the identity. Equation 0.266 in [16] shows that Equation (14) holds true. The other properties are obvious.

Now, assume that the first condition applies. Then, the second condition holds true, since

$$\begin{array}{ccc}\hfill 1& =& (1-f\left(a\right))(1+f\left(a\right))\prod _{k=1}^{\infty }(1+f\left(a^{2^k}\right))\hfill \\ & =& (1-f\left(a\right))(1+f\left(a\right))/(1-f\left(a^2\right)),\hfill \end{array}$$

from which the equation $f\left(a^2\right)=f^2\left(a\right)$, $a\in (0,1)$, follows.

We now show that the second condition implies that f is the identity. For an endomorphism h, we denote by $h^{\left(n\right)}$ the n-fold composition $h\circ \dots \circ h$. The assumptions imply that $f\left({(1-a^{2^k})}^{2^r}\right)={(1-f^{2^k}\left(a\right))}^{2^r}$ for all $a\in (0,1)$ and $k,r\in \mathbb{Z}$. Note that the set of maps $\{a\mapsto {(1-a^{2^k})}^{2^r}:k,r\in \mathbb{Z}\}$ is identical to the set of its inverses. Equation (13) yields $f(1/2)=1/2$. Let $T\left(A\right)=\{{(1-a^{2^k})}^{2^r}:a\in A,r,k\in \mathbb{Z}\}\cup A$ for $A\subset (0,1)$, and $D={\bigcup }_n{T}^{\left(n\right)}\left(\{1/2\}\right)$. It suffices to show that D is dense in $(0,1)$. Assume that it is not, i.e., an interval $(a,b)\subset (0,1)$ exists with $D\cap (a,b)=\varnothing$. Then, necessarily, also $D\cap {\bigcup }_n{T}^{\left(n\right)}\left((a,b)\right)=\varnothing$. Since $1/2\in D$, we may assume, without loss of generality, that $(a,b)\subset (1/2,1)$. Consider the map h defined on the open intervals of $(0,1)$,

$$h\left((a,b)\right)\mapsto \left\{\begin{array}{cc}(a^2,b^2),\hfill & \mathrm{if}a\ge 1/2,b^2>1/2\hfill \\ (1-b^2,1-a^2),\hfill & \mathrm{if}a\ge 1/2,b^2\le 1/2\hfill \\ (a,b),\hfill & \mathrm{if}a<1/2\hfill \end{array}\right..$$

Note that $(1-b^2,1-a^2)\subset T\left((a,b)\right)$ and $(a^2,b^2)\subset T^{\left(2\right)}\left((a,b)\right)$. Let $(c,d)=h^{(n-1)}\left((a,b)\right)$. Per assumption, h maps $(a,b)\subset (1/2,1)$ to either $(a^2,b^2)$ or $(1-b^2,1-a^2)$. Per construction, if $h^{\left(n\right)}\left((a,b)\right)=(1-c^2,1-d^2)$, then $h^{(n+1)}\left((a,b)\right)\ne (1-c^2,1-d^2)$. If, eventually, $h^{\left(n\right)}\left((a,b)\right)=h^{(n+1)}\left((a,b)\right)=(c^2,d^2)$, then it always holds that $1/2\in (c^2,d^2)$. Let $\left|\right(a,b\left)\right|$ be the length of an interval $(a,b)$. If $a\ge 1/2$, then

$$\left|h(a,b)\right|=b^2-a^2=(b+a)(b-a)>b-a.$$

Hence, $h^{\left(n\right)}\left((a,b)\right)$ finally contains $1/2$ or its length is strictly monotonously increasing. If the latter were true, the limit would exist as $\left|h\right|\le 1/2$. Close to the limit, we would have $|h^{(n+1)}\left((a,b)\right)-h^{\left(n\right)}\left((a,b)\right)|=|d^2-c^2-(d-c)|=|(d-c)(d+c-1)|<ϵ$ for some $ϵ>0$, i.e., both boundaries of $h^{\left(n\right)}\left((a,b)\right)$ must converge to $1/2$, contradicting the fact that an interval with upper bound close to $1/2$ is mapped to an interval with upper bound close to $3/4$. □

Lemma 2.

Let $\Phi :[0,1]\to [0,1]$ be an endomorphism and $c>0$. The following two assertions are equivalent:

1. 

Φ is continuous on $(0,1)$ with values in $(0,1)$, $\Phi (1/2)>0$ and $\Phi (1/n)\to 0=\Phi \left(0\right)$ as $n\to \infty$. For all $n\in \mathbb{N}$ and $0\le k<n$, we have

$$\begin{array}{ccc}\hfill \Phi \left(a^n\right)\vee \Phi (1-a)& =& {\Phi }^n\left(a\right)\vee \Phi (1-a),a\in (0,1),\hfill \\ \hfill {\displaystyle \frac{\Phi (1-k/n)}{{(n-k)}^c}}\vee \Phi (k/n)& =& \prod _{m=k+1}^n\Phi (1-1/m)\vee \underset{\ell =k+1}{\overset{n}{\bigvee }}\Phi (1/\ell )\prod _{m=\ell +1}^n\Phi (1-1/m);\hfill \end{array}$$

2. 

$\Phi \left(a\right)=a^c$.

Proof. 

We assume that the first condition holds true. To begin with, we show that $\Phi$ is strictly increasing. For $j\in \mathbb{N}$, we have

$${\displaystyle \frac{\Phi (1-k/n)}{j^c{(n-k)}^c}}\vee \Phi (k/n)=\prod _{m=jk+1}^{jn}\Phi (1-1/m)\vee \underset{\ell =jk+1}{\overset{jn}{\bigvee }}\Phi (1/\ell )\prod _{m=\ell +1}^{jn}\Phi (1-1/m).$$

Letting $j\to \infty$, we obtain

$$\Phi (k/n)=\underset{j\to \infty }{lim}\prod _{m=jk+1}^{jn}\Phi (1-1/m),$$

so that $\Phi (k/n)\le \Phi \left(\right(k+1)/n)$ for all $0<k<n-1$. Hence, $\Phi$ is increasing on $(0,1)\cap \mathbb{Q}$, thus increasing on $(0,1)$. In particular, $\Phi \ge \Phi (1/2)>0$ on $[1/2,1)$. Let $n/2\le k<n-1$ and $j\in \mathbb{N}$ such that $j^c{(n-k-1)}^c\Phi (k/n)>1$. Then,

$$\begin{array}{ccc}\hfill \Phi \left(\right(k+1)/n)& =& \Phi (1-(k+1)/n)j^{-c}{(n-k-1)}^{-c}\vee \Phi ((k+1)/n)\hfill \\ & =& \prod _{m=j(k+1)+1}^{jn}\Phi (1-1/m)\hfill \\ & >& \prod _{m=jk+1}^{jn}\Phi (1-1/m)=\Phi (k/n),\hfill \end{array}$$

so that $\Phi$ is strictly increasing on $[1/2,1)$. Therefore, $\Phi \left(a^n\right)={\Phi }^n\left(a\right)$ for all $a\in (0,1)$ and $n\in \mathbb{N}$ with $1-a<a^n$. Let $m,k\in \mathbb{N}$. Let ℓ be a multiple of k such that $(a\wedge a^{m/k})+a^{1/\ell }-1>0$. Then,

$$\begin{array}{c}\hfill \Phi \left(a^{m/k}\right)=\Phi \left({\left(a^{1/\ell }\right)}^{m\ell /k}\right)={\left(\Phi \left(a^{1/\ell }\right)\right)}^{m\ell /k}={\left({\left(\Phi \left(a^{1/\ell }\right)\right)}^{\ell }\right)}^{m/k}={(\Phi \left(a\right))}^{m/k}.\end{array}$$

The continuity of $\Phi$ implies that $\Phi \left(a^x\right)=\Phi {\left(a\right)}^x$ for $a\in (0,1)$ and $x>0$, so that $\Phi \left(a\right)=a^{\gamma }$ for a $\gamma \in \mathbb{R}$. Since $\Phi (1/n)\to 0$ and $\Phi (1/2)>0$, we have $\gamma >0$. Finally, for $k=0$, we obtain from the assumptions that $\Phi \left(1\right)/n^c=1/n^{\gamma }$, $n\in \mathbb{N}$, so that $\Phi \left(1\right)=1$ and $c=\gamma$. □

Finally, we restate two results from the literature for the reader’s convenience.

Lemma 3.

Let $H:P\to [0,\infty ]$ be a function fulfilling Conditions 2(d) and 2(f) of Theorem 1. Then, $H\left(U_n\right)=clogn$ for all $n\in \mathbb{N}$ and some $c\ge 0$.

Proof. 

The inclusion $P_n\subset P_{n+1}$ and Condition 2(d) immediately imply monotonicity, i.e., $H\left(U_n\right)\le H\left(U_{n+1}\right)$ for all $n\in \mathbb{N}$. By Condition 2(f) and Implication 1 of Theorem V in [17], the claim is followed. □

Lemma 4.

Let H be the Shannon entropy, the Rényi entropy, the min-entropy, or the Hartley entropy. Then, $H\left(U_n\right)$ is maximal on $P_n$.

See, for instance, Lemma 2.2.4 and Remark 4.4.4 in [4] for proofs.

3.2. Properties of the Operator ${\partial }_k$

We will use the operator ${\partial }_k$, defined in Equation (7), iteratively. For instance,

$$\begin{array}{ccc}\hfill {\partial }_2{\partial }_{k}p& =& \left(p_2{p}_k\right)(p_{k}p\cup p_{\widehat{k}})\cup {(p_{k}p\cup p_{\widehat{k}})}_{\widehat{2}},\hfill \\ \hfill {\partial }_2{\partial }_3\{p_1,p_2,p_3\}& =& {\partial }_2\{p_3{p}_1,p_3{p}_2,p_3{p}_3,p_1,p_2\}\hfill \\ & =& \begin{array}{ccccc}\hfill \{p_3{p}_2{p}_3{p}_1,& \hfill p_3{p}_2{p}_3{p}_2,& \hfill p_3{p}_2{p}_3{p}_3,& \hfill p_3{p}_2{p}_1,& \hfill p_3{p}_2{p}_2,\\ \hfill p_3{p}_1,& & \hfill p_3{p}_3,& \hfill p_1,& \hfill p_2\}\end{array}\hfill \\ & =& \{p_1{p}_2{p}_3^2,p_2^2{p}_3^2,p_2{p}_3^3,p_1{p}_2{p}_3,p_2^2{p}_3,p_1{p}_3,p_3^2,p_1,p_2\}.\hfill \end{array}$$

The proofs of the theorems in Section 2 rely on the properties of various probability distributions. We denote by $b_a$ the Bernoulli distribution, i.e.,

$$b_a=\{a,1-a\},a\in (0,1),$$

and by $g_a$ the geometric distribution, i.e.,

$$g_a=(1-a)\{1,a,a^2,\dots \},a\in [0,1).$$

For $n\in \mathbb{N}$, let

$$\begin{array}{ccc}\hfill g_{n,a}& =& {\displaystyle \frac{1-a}{1-a^n}}\{1,a,a^2,\dots ,a^{n-1}\},a\in [0,1),\hfill \end{array}$$

$$\begin{array}{ccc}\hfill G_{n,a}& =& \left\{a^n\right\}\cup (1-a)\{1,a,a^2,\dots ,a^{n-1}\},a\in [0,1],\hfill \end{array}$$

(15)

$$\begin{array}{ccc}\hfill U_{k,n}& =& \{1-k/n,\underset{k\mathrm{times}}{\underbrace{1/n,\dots ,1/n}}\},0\le k<n.\hfill \end{array}$$

(16)

Let $G_{0,a}=U_{0,n}=\left\{1\right\}$ and

$$\begin{array}{ccc}\hfill {\widehat{\partial }}_k^{n}p& =& {\displaystyle \frac{1}{1-p_k^{2^n}}}{\left({\partial }_k^{n}p\right)}_{\widehat{k}},p_k<1,n\in \mathbb{N},\hfill \\ \hfill {\widehat{\partial }}_k^{0}p& =& p_{\widehat{k}}/(1-p_k),p_k<1.\hfill \end{array}$$

We denote by $\dot{=}$ equality up to inserting zeros and reordering.

Lemma 5.

Let $p\in P$ and ${\partial }_k^{\infty }p:={lim}_{n\to \infty }{\partial }_k^{n}p$. Then,

$$\begin{array}{ccc}\hfill {\partial }_k^{\infty }p& \dot{=}& g_{p_k}\times {\widehat{\partial }}_k^{0}p,p_k<1,\hfill \end{array}$$

(17)

$$\begin{array}{ccc}\hfill \left\{1\right\}& \dot{=}& \left\{1\right\}\times \left\{1\right\},\hfill \end{array}$$

(18)

$$\begin{array}{ccc}\hfill {\widehat{\partial }}_k^{n}p& \dot{=}& g_{2^n,p_k}\times {\widehat{\partial }}_k^{0}p,p_k<1.\hfill \end{array}$$

(19)

In particular, for $a\in (0,1)$ and $n\in \mathbb{N}$ we have

$$\begin{array}{ccc}\hfill {\partial }_1^{\infty }{b}_a& \dot{=}& g_a,\hfill \end{array}$$

(20)

$$\begin{array}{ccc}\hfill {\partial }_1^{\infty }{g}_a& \dot{=}& g_{1-a}\times g_a,\hfill \end{array}$$

(21)

$$\begin{array}{ccc}\hfill {\partial }_1^{\infty }{G}_{n,a}& \dot{=}& g_{a^n}\times g_{n,a}\dot{=}{g}_a\dot{=}{G}_{\infty ,a},\hfill \end{array}$$

(22)

$$\begin{array}{ccc}\hfill {\partial }_2^{\infty }{G}_{n,a}& \dot{=}& g_{1-a}\times G_{n-1,a},\hfill \end{array}$$

(23)

$$\begin{array}{ccc}\hfill {\partial }_1^{\infty }{U}_{k,n}& \dot{=}& g_{1-k/n}\times U_k,1\le k<n,\hfill \end{array}$$

(24)

$$\begin{array}{ccc}\hfill {\partial }_2^{\infty }{U}_{k,n}& \dot{=}& g_{1/n}\times U_{k-1,n-1},1\le k<n,\hfill \end{array}$$

(25)

$$\begin{array}{ccc}\hfill {\partial }_3^{\infty }{\partial }_2{b}_a& \dot{=}& g_a\times b_a.\hfill \end{array}$$

(26)

Furthermore, for $a\in (0,1)$ and $n\in \mathbb{N}$,

$$\begin{array}{ccc}\hfill {\partial }_1{G}_{n,a}& \dot{=}& G_{2n,a},\hfill \end{array}$$

(27)

$$\begin{array}{ccc}\hfill {\left(G_{n,a}\right)}_{\widehat{2}}& =& a{G}_{n-1,a}.\hfill \end{array}$$

(28)

Proof. 

Without loss of generality, let $k=1$ in Equation (17). We have

$$\begin{array}{c}\hfill {\partial }_1^{n}p=\left\{p_1^{2^n}\right\}\cup \bigcup _{\ell \in \{2^n-1,\dots ,0\}}{p}_1^{\ell }{p}_{\widehat{1}},\end{array}$$

since this is true for $n=1$ and

$$\begin{array}{ccc}\hfill {\partial }_1^{n+1}p& =& {\partial }_1\left({\partial }_1^{n}p\right)={\partial }_1\left(\left\{p_1^{2^n}\right\}\cup \bigcup _{\ell \in \{2^n-1,\dots ,0\}}{p}_1^{\ell }{p}_{\widehat{1}}\right)\hfill \\ & =& p_1^{2^n}\left(\left\{p_1^{2^n}\right\}\cup \bigcup _{\ell \in \{2^n-1,\dots ,0\}}{p}_1^{\ell }{p}_{\widehat{1}}\right)\cup \bigcup _{\ell \in \{2^n-1,\dots ,0\}}{p}_1^{\ell }{p}_{\widehat{1}}\hfill \\ & =& \left\{p_1^{2^{n+1}}\right\}\cup \bigcup _{\ell \in \{2^{n+1}-1,\dots ,2^n\}}{p}_1^{\ell }{p}_{\widehat{1}}\cup \bigcup _{\ell \in \{2^n-1,\dots ,0\}}{p}_1^{\ell }{p}_{\widehat{1}}\hfill \end{array}$$

by induction. Hence,

$$\begin{array}{c}\hfill {\partial }_1^{\infty }p\dot{=}\bigcup _{\ell \in \{\dots ,1,0\}}{p}_1^{\ell }{p}_{\widehat{1}}.\end{array}$$

Now, $p_1^{\ell }{p}_{\widehat{1}}=(1-p_1)p_1^{\ell }·{\displaystyle \frac{p_{\widehat{1}}}{1-p_1}}$ for $p_1<1$, implying Equation (17). Further, we have ${\partial }_2{b}_a=(1-a)b_a\cup \left\{a\right\}$, so that Equation (26) holds by Equation (17). The other equalities of the lemma are immediate. □

Lemma 6.

Let $S:P\to [0,\infty ]$ and $f:[0,1]\to [0,\infty )$, such that Equation (9) holds true. Then,

$$\begin{array}{ccc}\hfill S\left({\partial }_1^{n}p\right)& =& (S\left(p\right)-1)F_n\left(p_1\right)+1,p\in P,n\in \mathbb{N},\hfill \end{array}$$

(29)

$$\begin{array}{ccc}\hfill S\left(G_{2n,a}\right)& =& (1+f\left(a^n\right))S\left(G_{n,a}\right)-f\left(a^n\right),n\in \mathbb{N}.\hfill \end{array}$$

(30)

Proof. 

It is easy to check by induction that Equation (9) implies

$$\begin{array}{c}\hfill S\left({\partial }_1^{n}p\right)=S\left(p\right)\prod _{k=0}^{n-1}\left(1+f\left(p_1^{2^k}\right)\right)-\sum _{\ell =0}^{n-1}f\left(p_1^{2^{\ell }}\right)\prod _{k=\ell +1}^{n-1}\left(1+f\left(p_1^{2^k}\right)\right).\end{array}$$

Now,

$$\begin{array}{c}\hfill 1+\sum _{i=1}^n{c}_i\prod _{j=i+1}^n(1+c_j)=\prod _{j=1}^n\left(1+c_j\right),c_j\in \mathbb{R},n\in \mathbb{N},\end{array}$$

by induction, so that the first assertion of the lemma holds. Equation (27) yields the second assertion. □

3.3. Proof of Theorem 1

Assume that H is the Shannon entropy. Then,

$$H\left(g_a\right)=-\sum _{k=0}^{\infty }(1-a)a^{k}log\left((1-a)a^k\right)=-(aloga+(1-a)log(1-a))/(1-a)\in (0,\infty )$$

for $a\in (0,1)$. All the other assertions in the second condition hold obviously, when f is chosen to be the identity. Now, assume that the second condition applies. By Condition 2(a), Condition 2(b) holds for all $p\in P$. Condition (f) and Equation (18) yield $H\left(\right\{1\left\}\right)=0$. It is easy to check that $H\left({\partial }_1^{n}p\right)=H\left(p\right)F_n\left(p_1\right),$ so that

$$\begin{array}{c}\hfill H\left({\partial }_1^{\infty }p\right)=H\left(\underset{n\to \infty }{lim}{\partial }_1^{n}p\right)=\underset{n\to \infty }{lim}H\left({\partial }_1^{n}p\right)=H\left(p\right)F\left(p_1\right).\end{array}$$

(31)

The function F is finite due to Conditions 2(e) and 2(f) and Equations (21) and (31), i.e.,

$$\begin{array}{c}\hfill H\left(g_a\right)F(1-a)=H\left(g_{1-a}\right)+H\left(g_a\right).\end{array}$$

It follows that

$$F(1-a)-1={\displaystyle \frac{H\left(g_{1-a}\right)}{H\left(g_a\right)}},a\in (0,1).$$

Replacing a by $1-a$ in the above equation yields

$$\begin{array}{c}\hfill \left(F\right(1-a)-1)\left(F\right(a)-1)=1,a\in (0,1).\end{array}$$

(32)

Now,

$$\begin{array}{c}\hfill H\left(b_a\right)F\left(a\right)=H\left(g_a\right)>0,\end{array}$$

(33)

by Equations (20) and (31) and Condition 2(e). Equations (26), (8), and (31) yield

$$\begin{array}{c}\hfill H\left(b_a\right)(1+f(1-a))F\left(a\right)=H\left(g_a\right)+H\left(b_a\right);\end{array}$$

thus, $f(1-a)F\left(a\right)=1$ for $a\in (0,1)$. Equation (32) now writes $f\left(a\right)+f(1-a)=1$. Plugging in $p=b_a$ into Equation (8) shows that f is continuous on $(0,1)$. Now, Lemma 1 yields $f\left(a\right)=a$ and $F\left(a\right)=1/(1-a)$ for $a\in (0,1)$. Equation (8) implies that $f\left(0\right)=0$, so that f is the identity on $[0,1)$. Lemma 3 delivers $H\left(U_n\right)=clogn$ for a $c\ge 0$. Condition 2(f), Equations (17) and (31) yield

$$\begin{array}{c}\hfill H\left(p\right)F\left(p_1\right)=H\left(g_{p_1}\right)+H\left({\widehat{\partial }}_1^{0}p\right),p_1<1.\end{array}$$

(34)

This implies that

$$\begin{array}{c}\hfill clogn·F(1/n)=H\left(g_{1/n}\right)+clog(n-1)\end{array}$$

for $n\ge 2$, so that $c>0$ and

$$\begin{array}{c}\hfill c^{-1}H\left(g_{1/n}\right)={\displaystyle \frac{1}{1-1/n}}logn-log(n-1)=-{\displaystyle \frac{1/n}{1-1/n}}log{\displaystyle \frac{1}{n}}-log\left(1-{\displaystyle \frac{1}{n}}\right).\end{array}$$

Henceforth, we may assume that $c=1$. Due to the recursion Formula (34), it suffices to show that the above formula extends to $H\left(g_a\right)=-a{(1-a)}^{-1}loga-log(1-a)$, $a\in (0,1)$, to finish the proof. By Equations (25) and (31), we obtain

$$H\left(U_{n-k,n}\right)F(1/n)=H\left(g_{1/n}\right)+H\left(U_{n-k-1,n-1}\right),k\ge 1,$$

i.e.,

$$H\left(U_{n-k,n}\right)=\sum _{i=0}^{n-k-1}{\displaystyle \frac{H\left(g_{1/(n-i)}\right)}{{\prod }_{j=n-i}^{n}F(1/j)}}={\displaystyle \frac{1}{n}}\sum _{i=0}^{n-k-1}(n-i-1)H\left(g_{1/(n-i)}\right)$$

by iteration. On the other hand, by Equations (24) and (31),

$$H\left(U_{n-k,n}\right)F(k/n)=H\left(g_{k/n}\right)+log(n-k),$$

so that

$$\begin{array}{ccc}\hfill H\left(g_{k/n}\right)+log(n-k)& =& {\displaystyle \frac{1}{n-k}}\sum _{i=0}^{n-k-1}\left((n-i)log(n-i)-\left(n-i-1\right)log\left(n-i-1\right)\right)\hfill \\ & =& {\displaystyle \frac{1}{n-k}}\left(nlogn-klogk\right)=-{\displaystyle \frac{k/n}{1-k/n}}log{\displaystyle \frac{k}{n}}+logn.\hfill \end{array}$$

Condition 2(a), i.e., the continuity of H, finalizes the proof.

3.4. Proof of Theorem 2

Assume that H is the Rényi entropy. Let $S\left(p\right)=exp\left((1-\alpha )H\left(p\right)\right)$, i.e., $S\left(p\right)={\parallel p\parallel }_{\alpha }^{\alpha }$. Then,

$$S\left({\partial }_{1}p\right)=p_1^{\alpha }\sum _i{p}_i^{\alpha }+\sum _i{p}_i^{\alpha }-p_1^{\alpha },$$

so that $f\left(r\right)=r^{\alpha }$. Reversely, assume that the second condition holds. Since $H\left(\right\{1\left\}\right)=0$ by Condition 2(f) and Equation (18), we have $S\left(\right\{1\left\}\right)=1$. Furthermore, S is continuous. If $S\left(p\right)=1$, then $1=S\left({\partial }_{1}p\right)=S\left({\partial }_{2}p\right)=\dots =S\left({\partial }_1^{\infty }p\right)$ by Equation (9). Hence, $S\left(b_a\right)\ne 1$ for $a\in (0,1)$, as otherwise we would obtain a contradiction to Condition 2(e) by Equation (20). It follows that $f\left(a\right)=(S\left({\partial }_1{b}_a\right)-S\left(b_a\right))/(S\left(b_a\right)-1)$ is continuous on $(0,1)$. Equation (9) implies that $f\left(0\right)\left(S\right(\left\{0\right\}\cup p)-1)=0$ for all $p\in P$, so that $f\left(0\right)=0$. Condition 2(f) reads

$$\begin{array}{c}\hfill S(p\times q)=S\left(p\right)S\left(q\right).\end{array}$$

(35)

By Condition 2(a), Condition 2(b) holds for all $p\in P$. Henceforth, we assume that $a\in (0,1)$. Condition 2(e), Equations (29) and (21) yield

$$\begin{array}{c}\hfill \infty >S\left(g_a\right)S\left(g_{1-a}\right)\leftarrow (S\left(g_{1-a}\right)-1)F_n\left(a\right)+1,\end{array}$$

as $n\to \infty$. Hence, $F\left(a\right)={lim}_{n\to \infty }F\left(a\right)$ is finite. Now, by Equations (17), (35) and (29),

$$\begin{array}{c}\hfill (S\left(p\right)-1)F\left(p_1\right)+1=S\left(g_{p_1}\right)S\left({\widehat{\partial }}_1^{0}p\right),p_1<1.\end{array}$$

(36)

In particular,

$$\begin{array}{c}\hfill (S\left(b_a\right)-1)F\left(a\right)+1=S\left(g_a\right).\end{array}$$

(37)

We obtain by means of Equation (26) that

$$\left[S\left(b_a\right)(1+f(1-a))-f(1-a)-1\right]F\left(a\right)+1=S\left(g_a\right)S\left(b_a\right),$$

so that, by Equation (37),

$$\left[S\left(b_a\right)(1+f(1-a))-f(1-a)\right]F\left(a\right)-S\left(b_a\right)F\left(a\right)+S\left(g_a\right)=S\left(g_a\right)S\left(b_a\right),$$

i.e., as $S\left(b_a\right)\ne 1$,

$$\begin{array}{c}\hfill f(1-a)F\left(a\right)=S\left(g_a\right).\end{array}$$

(38)

Since $S\left(g_a\right)\ne 0$, it follows that $f\left(a\right),F\left(a\right)\ne 0$. Equations (36) and (38) yield for $p=g_{1-a}$ that $1-F{\left(a\right)}^{-1}=(1-f(1-a))S\left(g_{1-a}\right)$, so that Equation (36) now reads

$$\begin{array}{c}\hfill S\left(p\right)-(1-f(1-p_1))S\left(g_{1-p_1}\right)=f(1-p_1)S\left({\widehat{\partial }}_1^{0}p\right),\end{array}$$

(39)

again by Equation (38). In line with the idea from Equation (23), Equations (39) and (28) yield

$$\begin{array}{ccc}\hfill S\left(G_{n,a}\right)& =& (1-f\left(a\right))S\left(g_a\right)+f\left(a\right)S\left(G_{n-1,a}\right)\hfill \\ & =& (1-f\left(a\right))S\left(g_a\right)\sum _{k=0}^{n-1}{f}^k\left(a\right)+f^n\left(a\right)\hfill \\ & =& f^n\left(a\right)+(1-f^n\left(a\right))S\left(g_a\right).\hfill \end{array}$$

(40)

Plugging in Equation (40) into Equation (30) yields

$$\begin{array}{c}\hfill f^{2n}\left(a\right)+(1-f^{2n}\left(a\right))S\left(g_a\right)=(1+f\left(a^n\right))(f^n\left(a\right)+(1-f^n\left(a\right))S\left(g_a\right))-f\left(a^n\right),\end{array}$$

i.e.,

$$\begin{array}{c}\hfill (f\left(a^n\right)-f^n\left(a\right))(1-f^n\left(a\right))(S\left(g_a\right)-1)=0.\end{array}$$

Assume that $f\left(a\right)=1$. Then, by Equation (40), $1=S\left(G_{n,a}\right)\to S\left(g_a\right)$ as $n\to \infty$. This contradicts $S\left(g_a\right)\ne 1$. Hence, we obtain for $n=2$ that

$$\begin{array}{c}\hfill f^2\left(a\right)=f\left(a^2\right),a\in (0,1).\end{array}$$

(41)

Lemma 3 implies that $H\left(U_n\right)=\tilde{c}logn$ for a $\tilde{c}\ge 0$; hence, $S\left(U_n\right)=n^c$ for $c=\gamma \tilde{c}\in \mathbb{R}$. By $S\left(U_2\right)=S\left(b_{1/2}\right)\ne 1$, it follows that $c\ne 0$. Equation (39) gives

$$\begin{array}{c}\hfill (1-f(1-1/n))S\left(g_{1-1/n}\right)=n^c-f(1-1/n){(n-1)}^c.\end{array}$$

(42)

Furthermore, taking the ideas from Equations (24) and (25), we obtain, by Equation (39),

$$S\left(U_{k,n}\right)=(1-f(k/n))S\left(g_{k/n}\right)+f(k/n)k^c$$

and

$$\begin{array}{ccc}\hfill S\left(U_{k,n}\right)& =& (1-f(1-1/n))S\left(g_{1-1/n}\right)+f(1-1/n)S\left(U_{k-1,n-1}\right)\hfill \\ & =& n^c+f(1-1/n)\left[-{(n-1)}^c+S\left(U_{k-1,n-1}\right)\right]\hfill \\ & =& n^c+(1-{(n-k)}^c)\prod _{j=0}^{k-1}f(1-1/(n-j)),\hfill \end{array}$$

by Equation (42) and iteration. Thus, for $0<k<n$, we have

$$\begin{array}{c}\hfill (1-f(k/n))S\left(g_{k/n}\right)=n^c+(1-{(n-k)}^c){\displaystyle \frac{M_n}{M_{n-k}}}-f(k/n)k^c,\end{array}$$

(43)

where

$$M_1=1,M_n=\prod _{j=2}^{n}f(1-1/j),n\ge 2.$$

Note that $M_n>0$ for all $n\in \mathbb{N}$ as a consequence of Equation (38). We obtain for $n=2k$ in Equation (43) that

$$\begin{array}{c}\hfill (1-f(1/2))S\left(g_{1/2}\right)={\left(2k\right)}^c+(1-k^c){\displaystyle \frac{M_{2k}}{M_k}}-f(1/2)k^c.\end{array}$$

(44)

For $k=1$, this is

$$\begin{array}{c}\hfill (1-f(1/2))S\left(g_{1/2}\right)=2^c-f(1/2).\end{array}$$

(45)

Plugging Equation (45) into Equation (44) we obtain $M_{2k}/M_k=2^c-f(1/2)$ for $k>1$. An immediate consequence of this constant ratio is that $M_{2k}/M_{2\ell }=M_k/M_{\ell }$ for $\ell ,k>1$. Equation (43), the symmetry of the Bernoulli distribution, and the fact that, by Equation (39), $S\left(b_a\right)=(1-f\left(a\right))S\left(g_a\right)+f\left(a\right)$, yield

$$\begin{array}{c}\hfill (1-{(n-k)}^c){\displaystyle \frac{M_n}{M_{n-k}}}-f(k/n)(k^c-1)=(1-k^c){\displaystyle \frac{M_n}{M_k}}-f(1-k/n)({(n-k)}^c-1),\end{array}$$

i.e., for all $\ell \ge 0$ and $1<k<n-1$,

$$\begin{array}{ccc}\hfill f(k/n)& =& {\displaystyle \frac{M_{2^{\ell }n}}{M_{2^{\ell }k}}}-N\left(2^{\ell }\right)\left[{\displaystyle \frac{M_{2^{\ell }n}}{M_{2^{\ell }(n-k)}}}-f(1-k/n)\right]\hfill \\ & =& {\displaystyle \frac{M_n}{M_k}}-N\left(2^{\ell }\right)\left[{\displaystyle \frac{M_n}{M_{n-k}}}-f(1-k/n)\right]\hfill \end{array}$$

(46)

with

$$N\left(j\right)={\displaystyle \frac{1-j^c{(n-k)}^c}{1-{\left(jk\right)}^c}}.$$

In order for the right-hand side of Equation (46) to be independent of ℓ, we necessarily have $f(1-k/n)=M_n/M_{n-k}$. Now, Equation (43) ensures that for all $\ell \ge 1$ and $1<k<n-1$, we have

$$\begin{array}{ccc}\hfill (1-f(k/n))S\left(g_{k/n}\right)& =& {\ell }^c{n}^c+(1-{\ell }^c{(n-k)}^c)f(1-k/n)-f(k/n){\ell }^c{k}^c\hfill \\ \hfill & =& f(1-k/n)+k^c{l}^c\left({(n/k)}^c-{(n/k-1)}^{c}f(1-k/n)-f(k/n)\right).\hfill \end{array}$$

In order for the right-hand side to be independent of ℓ, we necessarily have

$$\begin{array}{c}\hfill {(k/n)}^{c}f(k/n)+{(1-k/n)}^{c}f(1-k/n)=1.\end{array}$$

Since f is continuous, we even have

$$a^{c}f\left(a\right)+{(1-a)}^{c}f(1-a)=1,a\in (0,1).$$

Let $\tilde{f}\left(a\right)=a^{c}f\left(a\right)$; then $\tilde{f}\left(a^2\right)={\tilde{f}}^2\left(a\right)$ by Equation (41) and $\tilde{f}\left(a\right)+\tilde{f}(1-a)=1$. Lemma 1 yields that $\tilde{f}$ is the identity on $(0,1)$; hence, $f\left(a\right)=a^{\alpha }$ with $\alpha =1-c$.

3.5. Proof of Theorem 3

Assume that H is the min-entropy and let f be the identity. Then, Equation (10) obviously holds true. Reversely, assume that the second condition of the theorem applies. Condition 2(f) and Equation (18) imply that $H\left(\right\{1\left\}\right)=0$ and $S\left(\right\{1\left\}\right)=1$. Equations (10) and (17) yield

$$\begin{array}{c}\hfill S\left(p\right)=S\left({\partial }_1^{\infty }p\right)\vee \Phi \left(p_1\right)=S\left(g_{p_1}\right)S\left({\widehat{\partial }}_1^{0}p\right)\vee \Phi \left(p_1\right),p_1<1,\end{array}$$

(47)

where $\Phi \left(a\right)={sup}_{k=0}^{\infty }f\left(a^{2^k}\right)$. Equations (10), (17), and (21) deliver $S\left(g_a\right)=S\left(g_{1-a}\right)S\left(g_a\right)\vee \Phi (1-a)$, so that $\Phi \left(a\right)$ is finite for $a\in (0,1)$. Since $S\left(g_{1-a}\right)\in (0,1)$ by Condition 2(e), it follows that

$$\begin{array}{c}\hfill S\left(g_a\right)=\Phi (1-a)\in (0,1),a\in (0,1).\end{array}$$

(48)

Thus, the recursion formula (47) determines H uniquely as soon as $\Phi$ is determined. Since S is continuous, the function $\Phi$ is continuous on $(0,1)$. Lemma 3 yields

$$\begin{array}{c}\hfill n^{-c}=S\left(U_n\right)=\Phi (1-1/n){(n-1)}^{-c}\vee \Phi (1/n),n\in \mathbb{N},\end{array}$$

for some $c\ge 0$. In particular, $\Phi (1/n)\le n^{-c}$ and $\Phi (1/2)=2^{-c}$. Assuming $c=0$ leads to $1=\Phi (1-1/n)\vee \Phi (1/n)$, contradicting Equation (48). Equation (10) implies that $0\le \Phi \left(0\right)\le {inf}_{n\in \mathbb{N}}S\left(U_n\right)=0$. Equations (47), (22), and (48) yield

$$\begin{array}{c}\hfill S\left(G_{n,a}\right)=S\left(g_a\right)\vee \Phi \left(a^n\right)=\Phi (1-a)\vee \Phi \left(a^n\right).\end{array}$$

On the other hand, Equations (23) and (48) and iterating Equation (47) yield

$$\begin{array}{c}\hfill S\left(G_{n,a}\right)=\Phi \left(a\right)S\left(G_{n-1,a}\right)\vee \Phi (1-a)={\Phi }^n\left(a\right)\vee \Phi (1-a).\end{array}$$

We define $U_{n,n}=\left\{0\right\}\cup U_n$ and $\Phi \left(1\right)=1$. Then, Equations (24) and (25) yield

$$\begin{array}{c}\hfill S\left(U_{n-k,n}\right)=\Phi (1-k/n){(n-k)}^{-c}\vee \Phi (k/n)\end{array}$$

and

$$\begin{array}{ccc}\hfill S\left(U_{n-k,n}\right)& =& \Phi (1-1/n)S\left(U_{n-k-1,n-1}\right)\vee \Phi (1/n)\hfill \\ & =& \prod _{m=k+1}^n\Phi (1-1/m)\vee \underset{\ell =k+1}{\overset{n}{\bigvee }}\Phi (1/\ell )\prod _{m=\ell +1}^n\Phi (1-1/m)\hfill \end{array}$$

for $0\le k<n$. Lemma 2 finalizes the proof.

3.6. Proof of Theorem 4

Assume that H is the Hartley entropy and let $f\left(a\right)={\mathbf{1}}_{(0,1]}\left(a\right)$. Then, the second condition in the theorem holds true. Reversely, assume that the second condition applies. Equations (12) and (19) yield

$$S\left(g_{2^n,p_1}\right)S\left({\widehat{\partial }}_1^{0}p\right)=S\left({\widehat{\partial }}_1^{n}p\right)=S\left({\widehat{\partial }}_1^{0}p\right)F_n\left(p_1\right),p_1<1,p\in P,$$

so that

$$\begin{array}{c}\hfill S\left(g_{2^n,a}\right)=F_n\left(a\right),a\in (0,1),\end{array}$$

(49)

as $S\left(p\right)\in [1,\infty )$ for $p\in P_0$. Since $F_n\left(a\right)$ is monotone in n, the limit $F\left(a\right)={lim}_{n\to \infty }{F}_n\left(a\right)$ always exists, i.e., $F\left(a\right)\in [1,\infty ]$. Assume that $F\left(a_{*}\right)<\infty$ for some $a_{*}\in (0,1)$. Condition 2(f) and Equations (29), (17), (49), and (11) yield for $p\in P_0$ that

$$\begin{array}{ccc}\hfill \infty & >& \left(S(\left\{a_{*}\right\}\cup (1-a_{*})p)-1\right)F\left(a_{*}\right)+1\hfill \\ & =& \underset{n\to \infty }{lim}S\left({\partial }_1^n\left(\left\{a_{*}\right\}\cup (1-a_{*})p\right)\right)\hfill \\ & =& \underset{n\to \infty }{lim}{\displaystyle \frac{S(a_{*}^{2^n}\left\{1\right\}\cup (1-a_{*}^{2^n})g_{2^n,a_{*}}\times p)}{S(g_{2^n,a_{*}}\times p)}}\underset{n\to \infty }{lim}{F}_n\left(a_{*}\right)S\left(p\right)\hfill \\ & =& F\left(a_{*}\right)S\left(p\right).\hfill \end{array}$$

(50)

Assume that $F\left(a_{*}\right)=1$. Equations (50) and (11) deliver for $p\in P_0$

$$\begin{array}{ccc}\hfill \infty >S\left(p\right)& =& S(\left\{a_{*}\right\}\cup (1-a_{*})p)=\dots =S((1-{(1-a_{*})}^n)g_{n,1-a_{*}}\cup {(1-a_{*})}^{n}p)\hfill \\ & =& \underset{n\to \infty }{lim}S\left(g_{n,1-a_{*}}\right).\hfill \end{array}$$

(51)

Lemma 3 yields $S\left(U_n\right)=n^c$ for $c\ge 0$. Now, Equation (51) implies that $c=0$, while the assumption in the theorem, $H\left(U_2\right)>0$, implies that $c>0$. Hence, $F\left(a_{*}\right)$ cannot be equal to 1. Plugging $p=G_{k,1-a_{*}}$ into Equation (50) gives

$$S\left(G_{k-1,1-a_{*}}\right)F\left(a_{*}\right)=\underset{n\to \infty }{lim}S\left({\partial }_2^n{G}_{k,1-a_{*}}\right)=(S\left(G_{k,1-a_{*}}\right)-1)F\left(a_{*}\right)+1,$$

by Equation (28). As $F\ge 1$, iterating yields $S\left(G_{k,1-a_{*}}\right)=1+k(1-1/F\left(a_{*}\right))$, so that

$$\begin{array}{c}\hfill 1+2k(1-1/F\left(a_{*}\right))=(1+f\left(a_{*}^k\right))(1+k(1-1/F\left(a_{*}\right)))-f\left(a_{*}^k\right),k\in \mathbb{N},\end{array}$$

by Equation (30). Since $F\left(a_{*}\right)\ne 1$, we obtain $f\left(a_{*}^k\right)=1$, $k\in \mathbb{N}$, contradicting the finiteness of $F\left(a_{*}\right)$. Hence, $F\left(a\right)=\infty$ for all $a\in (0,1)$. Equations (29), (11), and (49) yield

$$S\left(p\right)-1={\displaystyle \frac{S\left({\partial }_1^{n}p\right)-1}{F_n\left(p_1\right)S\left({\widehat{\partial }}_1^{0}p\right)}}S\left({\widehat{\partial }}_1^{0}p\right)\to S\left({\widehat{\partial }}_1^{0}p\right),p_1\in (0,1),S\left(p\right)<\infty ,$$

i.e.,

$$\begin{array}{c}\hfill S\left(p\right)=S\left({\widehat{\partial }}_1^{0}p\right)+1,p_1\in (0,1),S\left(p\right)<\infty .\end{array}$$

(52)

The assertion of the theorem follows for $p\in P_0$ by Conditions 2(b)–2(d) and the fact that $S\left(\right\{1\left\}\right)=1$. In particular, $f\left(a\right)={\mathbf{1}}_{(0,1]}\left(a\right)$. Assume now that $S\left(p\right)<\infty$ for some $p\in P\setminus P_0$. Without loss of generality, $p_i>0$ for all $i\in \mathbb{N}$. Then, by induction, Equation (52) delivers $S\left({\left({\widehat{\partial }}_1^0\right)}^{n}p\right)<\infty$ and $S\left(p\right)=S\left({\left({\widehat{\partial }}_1^0\right)}^{n}p\right)+n$, $n\in \mathbb{N}$. This contradicts $S\ge 1$ and $S\left(p\right)<\infty$. Hence, $S\left(p\right)=\infty$ for $p\in P\setminus P_0$.

4. Discussion

Although all conditions in the characterizations of Shannon’s and Rényi’s entropy have intrinsic motivations and interpretations, their relevance might differ. Continuity and finiteness offer advantages also from a technical point of view and will usually be retained. Condition 2(b) is intrinsic for probability distributions. The additivity might be considered to be axiomatic. Therefore, we would like to discuss here the importance of the following three points:

• The discrete nature of the objects;
• The hierarchical, self-similar structure of the objects, cf. Condition 2(g);
• The parity among the objects at each hierarchical level, cf. Condition 2(b).

4.1. Practical Implications

The three conditions above are obviously tailormade for tree-like graphs and other hierarchical structures, without the necessity to appeal information-theorical arguments. Examples that may fit our approach are, for instance,

• Classification systems: forms of art (e.g., music genres), languages and their dialects.
• Biology: taxonomie, diversity, genome.
• Administration units: districts, corporate structure, military.
• Universe: clustering.
• Material sciences [18,19].
• Networks [20].
• Statistics: graphs and cluster analysis.

For all these examples, Shannon’s or Rényi’s entropy may at least be a default choice. The min-entropy may suit graphical models for extremes [21].

4.2. Modifications

The three properties above might be the most questionable from a nonideal, practical point of view. While we believe that nondiscrete objects lead to different theories, modifications and variants of the self-similarity property and the parity property might be accessible to some extent by theoretical advancements of this framework. Also modifications of the operator ∂ could be of practical interest. Since the tree built by ${\partial }^{\infty }$ can be seen as a deterministic two-type Kesten tree [22], randomizations of ${\partial }^{\infty }$ bridge to the Galton–Watson processes.

4.3. The Geometric Distribution

In comparison to Faddeev’s approach, our proofs emphasize the role of the geometric distribution $g_a$, which arises as the infinite iteration of the Bernoulli distribution $b_a$, $a\in (0,1)$, i.e., ${\partial }_1^{\infty }{b}_a=g_a$. The uniqueness proof for the Shannon entropy uses almost only the resulting proportionality relation of the geometric distribution, simplifying Faddeev’s approach. The important role of the geometric distribution is already known in information theory: it maximizes the Shannon entropy among discrete probability distributions given the mean ([23], Theorem 5.8) and minimizes the min-entropy under fixed variance, among all discrete log-concave random variables in $\mathbb{Z}$ ([12], Theorem 1). The geometrical distribution also appears in branching processes [24] and in physics to describe the canonical ensemble [25]. By our research, we add that the geometric distribution is central to describe the self-similarity law of the Shannon entropy. Obviously, the operator ∂ can be inverted, so that the distribution is not refined at a leaf of the graph, but the structure is extended at the root, i.e., ${\partial }^{-1}(q,a,p):=(1-a)p\cup aq$, so that ${\partial }^{-1}({\partial }_1^{\infty }p,p_1,{\widehat{\partial }}_1^{0}p)={\partial }_1^{\infty }p$. Hence, ${\partial }_1^{\infty }p$ can be interpreted as a kind of limit distribution that is invariant to ${\partial }^{-1}$. Whether this leads to any formulation of a limit law including a domain of attraction is an open question. Note that the operator ${\partial }^{-1}$ has already been addressed in [8].

4.4. On the Proportionality Relation Characterizing the Rényi Entropy

Although the characterization and the proofs of the Shannon and the Rényi entropy follow the same scheme, they show remarkable differences. For instance, the generalization $G_{n,a}$ of the geometric distribution (see Equation (15)) does not play a role in the case of the Shannon entropy. Also, the functional characterization in Equation (8) of the Shannon entropy is not the limit of the characterization of the Rényi entropy (see Equation (9)). However, Equation (8) might be understood as the derivative of Equation (9) in the following sense, cf. [26]. Assume that f in Equation (9) is differentiable. Then,

$${\displaystyle \frac{d}{d\alpha }}S\left({\partial }_{1}p\right)={\displaystyle \frac{d}{d\alpha }}\left((1+f\left(p_1\right))S\left(p\right)-f\left(p_1\right)\right).$$

For $\alpha =1$, we define $S\left(p\right)={lim}_{\alpha \to 1}{\parallel p\parallel }_{\alpha }^{\alpha }$. Then, we obtain

$${\displaystyle \frac{d}{d\alpha }}S\left({\partial }_{1}p\right)=(1+f\left(p_1\right)){\displaystyle \frac{d}{d\alpha }}S\left(p\right).$$

Another interpretation of our results is that the order-$\alpha$ information energy S is not a natural transformation of the Rényi entropy, in contrast to

$$T\left(p\right)=S\left(p\right)-S\left(\right\{1\left\}\right),$$

which equals the Harvda–Charvát entropy [27] or Tsallis entropy [28], up to a multiplicative constant. Therefore, T bridges to the generalized information function ([5], Section 6.2), but also to the definite functions ([29], Chapter 3, Corollary 3.3 and proof of Theorem 2.2, and Chapter 6, Example 5.16). Indeed, Equation (9) can be rewritten in full analogy to Equation (8), namely, as

$$\begin{array}{c}\hfill T\left({\partial }_{1}p\right)=(1+f\left(p_1\right))T\left(p\right),\end{array}$$

where $T\left({\partial }_{1}p\right)=S\left({\partial }_{1}p\right)-S\left(\left\{1\right\}\right)$. Note that also the interpretation $T\left({\partial }_{1}p\right)=S\left({\partial }_{1}p\right)-S\left({\partial }_1\left\{1\right\}\right)$ is possible.

4.5. Tsallis and Other Entropies

It is an open question whether further practically relevant chain-rule-like conditions such as Condition 2(g) can be found, or, equivalently, recursive formulae such as Equations (34), (36), and (47), and, if so, whether they will be based on the geometric distribution. Reversely, it is also unclear which other existing definitions of an entropy can be described by conditions similar to Conditions 2(a)–2(g). Reference [15] considers additivity, i.e., Condition 2(f), as the only axiomatic property that an entropy must have. The prevalent interpretation of the Tsallis entropy [28] is that it is not additive; hence, the Tsallis entropy and many others are, per se, excluded from this framework, or the framework would have to be largely extended. Reference [15] remarks that the additivity law has two ingredients—the independence of the components before being joined and the requirement that the components do not interact when being joined. In other words, one may add an additivity law to the Tsallis entropy so that the Tsallis entropy indeed fits this framework. The practical use of such an approach may show the future.

4.6. Shannon’s Notion of Entropy

Shannon [30] took a very general perspective on which properties an entropy should have, namely, that “any monotonic function of this number [the finite number of messages] can be regarded as a measure of the information produced when one message is chosen from the set, all choices being equally likely”. Further, Shannon justified his and Hartley’s choice by the statement that “the logarithmic measure is more convenient”. Shannon added three arguments, namely, “it is practically more useful”, “it is nearer to our intuitive feeling as to the proper measure”, and “it is mathematically more suitable”. Shannon is known for his repeated warnings against the blind use of his approach outside the framework of telecommunication [31]. The approach of [15] is, in this spirit, trying to make Shannon’s three arguments mathematically more concrete and generally applicable. To this end, the entropy is characterized in algebraic terms only, which reflects the desired properties. Ideally, these properties are rich enough such that the choice of entropy becomes unique. Preferably, the desired properties stem from common sense, invariance considerations, and limit laws. This paper and the presented characterizations of the entropies follow these requirements, giving a deep justification for the use of the Shannon entropy in a large spectrum of applications. But it also indicates its limits of application, unless further deep characterizations are found.

5. Conclusions

The characterizations given here are not only novel with respect to splitting the chain rule into its natural components, they are also novel in the sense that only a rather weak relation is required for the chain rule, i.e., the proportionality factor itself is not given, but arises naturally. Furthermore, the recursion formulae have the advantage that they allow an interpretation as an intrinsic property, since only a single distribution is involved. Further, the recursion formulae may allow additional practical interpretations such as self-similarity or scale invariance. Lastly, while Equation (2) is a rule, proportionality can be considered a law. This yields a deeper understanding of the Shannon entropy and may justify a broad spectrum of applications outside telecommunication.