Characterizing the Asymptotic Per-Symbol Redundancy of Memoryless Sources over Countable Alphabets in Terms of Single-Letter Marginals

Abstract

The minimum expected number of bits needed to describe a random variable is its entropy, assuming knowledge of the distribution of the random variable. On the other hand, universal compression describes data supposing that the underlying distribution is unknown, but that it belongs to a known set $𝒫$ of distributions. However, since universal descriptions are not matched exactly to the underlying distribution, the number of bits they use on average is higher, and the excess over the entropy used is the redundancy. In this paper, we study the redundancy incurred by the universal description of strings of positive integers (ℤ₊), the strings being generated independently and identically distributed (i.i.d.) according an unknown distribution over ℤ₊ in a known collection $𝒫$. We first show that if describing a single symbol incurs finite redundancy, then $𝒫$ is tight, but that the converse does not always hold. If a single symbol can be described with finite worst-case regret (a more stringent formulation than redundancy above), then it is known that describing length n i.i.d. strings only incurs vanishing (to zero) redundancy per symbol as n increases. On the contrary, we show it is possible that the description of a single symbol from an unknown distribution of $𝒫$ incurs finite redundancy, yet the description of length n i.i.d. strings incurs a constant (> 0) redundancy per symbol encoded. We then show a sufficient condition on single-letter marginals, such that length n i.i.d. samples will incur vanishing redundancy per symbol encoded.

1. Introduction

A number of statistical inference problems of significant contemporary interest, such as text classification, language modeling and DNA microarray analysis, are what are called large alphabet problems. They require inference on sequences of symbols where the symbols come from a set (alphabet) with a size comparable or even larger than the sequence length. For instance, language models for speech recognition estimate distributions over English words using text samples much smaller than the English vocabulary.

An abstraction behind several of these problems is universal compression over large alphabets. The general idea here is to model the problem at hand with a collection of models $𝒫$ instead of a single distribution. The model underlying the data is assumed or known to belong to the collection $𝒫$, but the exact identity of the model remains unknown. Instead, we aim to use a universal description of data.

The universal description uses more bits on average (averaged over the random sample) than if the underlying model were known, and the additional number of bits used by the universal description is called the redundancy against the true model. The average excess bits over the entropy of the true model will be referred to as the model redundancy for that model. Since one does not know the true model in general, a common approach is to consider collection redundancy or simply redundancy, which is the supremum of the model redundancy, the supremum being taken over all models of the collection.

Typically, we look at sequences of i.i.d. symbols, and therefore, we usually refer to the redundancy of distributions over length n sequences obtained by i.i.d. sampling from distributions from $𝒫$. The length n of sequences considered will typically be referred to as the sample size.

The nuances of prediction, the compression or estimation where the alphabet size and sample size are roughly equal are not well captured by studying a collection over a finite alphabet when the sample size is increased to infinity. Rather, they are better captured when we begin with a countably infinite support and let the sample size approach infinity or when we let the alphabet size scale as a function of the sample size. However, the collection of all i.i.d. distributions over countably infinite supports has infinite redundancy that renders most estimations or prediction problems impossible. Therefore, there are several alternative formulations to tackle language modeling, classification and estimation questions over large alphabets.

Patterns: One line of work is the patterns [1] approach that considers the compression of the pattern of a sequence rather than the sequence itself. Patterns abstract the identities of symbols and indicate only the relative order of appearance. For example, the pattern of TATTLE is 121134, while that of HONOLULU is 12324545. The point to note is that patterns of length n i.i.d. sequences can be compressed (no matter what the underlying countably infinite alphabet is) with redundancy that grows sublinearly in n [1]; therefore, the excess bits needed to describe patterns are asymptotically vanishing per symbol encoded. Indeed, insights learned in this line of work will be used to understand the compression of sequences, as well, in this paper.

Envelope on Model Classes: A second line of work considers restricted model classes for applications, particularly where the collection of models can be described in terms of an envelope [2]. This approach leads to an understanding of the worst-case formulations. In particular, we are interested in the result that if the worst-case regret (different from and a more stringent formulation than the redundancy described here) of describing a single sample is finite, then the per-symbol redundancy diminishes to zero. We will interpret this result towards the end of the Introduction. While envelope classes are usually chosen so that they are compressible in the worst case, a natural extension is the possibility of choosing classes that are only average-case, but not worst-case, compressible. For this, we need to understand how the single-letter average case redundancy of a class influences the redundancy of compressing strings sampled i.i.d. from distributions in the class—the focus of this paper.

Data-derived Consistency: A third line of work ignores the uniform convergence framework underlying redundancy or regret formulations. This is useful for large or infinite alphabet model collections that have poor or no redundancy guarantees, but ask a question that cannot be answered with the approaches above. In this line of work, one obtains results on the model redundancy described above instead of (the collection) redundancy. For example, a model collection is said to be weakly compressible if there is a universal measure that ensures that for all models, the model redundancy normalized by the sample size (per-symbol) diminishes to zero. The rate at which the per-symbol model redundancy diminishes to zero depends on the underlying model and for some models could be arbitrarily slower than others. Given a particular block length n, however large, there may be, hence, no non-trivial guarantee that holds over the entire model collection, unlike the redundancy formulation.

However, if we add on the additional constraint that we should estimate the rate of convergence from the data, we get the data-derived consistency formulations in [3]. Fundamental to further research in this direction is a better understanding of how single-letter redundancy (of $𝒫$) relates to the redundancy of length n strings (that of $𝒫$ⁿ). The primary theme of this paper is to collect such results on the redundancy of classes over countably infinite support.

In the fixed alphabet setting, this connection is well understood. If the alphabet has size k, the redundancy of $𝒫$ is easily seen to be always finite (in fact, ≤ log k) and that of $𝒫$ⁿ scales as $\frac{k-1}{2}\text{log}n$. However, when $𝒫$ does not have a finite support, the above bounds are meaningless.

Redundancy Capacity Theorem: On the other hand, the redundancy of a collection $𝒫$ over a countably infinite support may be infinite. In this paper we let ℤ₊ = {1, 2, 3, ...} be the set of positive integers and ℕ = {0, 1, 2, ...} be the set of non-negative integers. However, what about the case where the redundancy of a collection $𝒫$ over ℤ₊ is finite? Now, a well-known redundancy-capacity [4] argument can be used to interpret the redundancy, which equates the redundancy to the amount of information we can get about the source from the data. In this case, finite (infinite, respectively) redundancy of $𝒫$ implies that a single symbol contains a finite (infinite, respectively) amount of information about the model.

The natural question then is the following. If a collection $𝒫$ over ℤ₊ has finite redundancy, does it imply that the redundancy of length n i.i.d. strings from $𝒫$ grows sublinearly? Equivalently, do finite redundancy collections behave similar to their fixed alphabet counterparts? If true, roughly speaking, such a result would inform us that as the universal encoder sees more and more of the sequence, it learns less and less of the underlying model. This would be in line with our intuition, where seeing more data fixes the model. Therefore, the more data we have already seen, the less there is to learn. Yet, as we will show, that is not the case.

Results: To understand these nuances, we first show that if the redundancy of a collection $𝒫$ of distributions over ℤ₊ is finite, then $𝒫$ is tight. This turns out to be a useful tool to check if the redundancy is finite in [3], for example.

However, in a departure from other worst-case regret formulations, as in [2], we demonstrate that it is possible for a class $𝒫$ to have finite redundancy, yet the asymptotic per-symbol redundancy of strings sampled i.i.d. from $𝒫$ is bounded away from zero. Therefore, roughly speaking, no matter how much of the sequence the universal encoder has seen, it learns at least a constant number of bits about the underlying model each time it sees an additional symbol. No matter how much data we see, there is more to learn about the underlying model! We finally obtain a sufficient condition on a class $𝒫$, such that the asymptotic per-symbol redundancy of length n i.i.d. strings diminishes to zero.

2. Notation and Background

We introduce the notation used in the paper, as well as some prior results that will be used. Following information theoretic conventions, log indicates logarithms to base two and ln to base e. In this paper we let ℤ₊ = {1, 2, 3, ...} be the set of positive integers and ℕ = {0, 1, 2, ... } be the set of non-negative integers.

2.1. Redundancy

The notation used here is mostly standard, but we include it for completeness. Let $𝒫$ be a collection of distributions over ℤ₊. Let $𝒫$ⁿ be the set of distributions over length-n sequences obtained by i.i.d. sampling from distributions in $𝒫$.

$𝒫$^∞ is the collection of measures over infinite length sequences of ℤ₊ obtained by i.i.d. sampling as follows. Observe that ${\mathbb{Z}}_{+}^n$ is countable for every n. For simplicity of exposition, we will think of each length n string x as a subset of ${\mathbb{Z}}_{+}^{\infty }$—the set of all semi-infinite strings of positive integers that begin with x. Each subset of ${\mathbb{Z}}_{+}^n$ is therefore a subset of ${\mathbb{Z}}_{+}^{\infty }$. Now the collection $𝒥$ of all subsets of ${\mathbb{Z}}_{+}^n$ and all n ∈ ℤ₊, is a semi-algebra [5]. The probabilities i.i.d. sampling assigns to finite unions of disjoint sets in $𝒥$ is the sum of that assigned to the components of the union. Therefore, there is a sigma-algebra over the uncountable set ${\mathbb{Z}}_{+}^{\infty }$ that extends $𝒥$ and matches the probabilities assigned to sets in $𝒥$ by i.i.d. sampling. The reader can assume that $𝒫$^∞ is the measure on the minimal sigma-algebra that extends $𝒥$ and matches what the probabilities i.i.d. sampling gives to sets in $𝒥$. See, e.g., [5], for a development of elementary measure theory that lays out the above steps.

Let q be a measure over infinite sequences that we call:

$$R_n({𝒫}^{\infty })=\underset{q}{\text{inf}}\underset{p\in {𝒫}^{\infty }}{\text{sup}}{E}_p\text{log}\frac{p(X^n)}{q(X^n)}$$

(1)

the redundancy of length n sequences, or length n i.i.d. redundancy, or simply length n redundancy. The single-letter redundancy refers to the special case when n = 1. We often normalize R_n($𝒫$^∞) in (1) by the block length n. We will call R_n($𝒫$^∞)/n the per-symbol length n redundancy.

In particular, note the distinction between single letter and per-symbol length n redundancy. In the definition (1), we do not require q to be i.i.d.. The single-letter redundancy would correspond to obtaining the infimum in (1) only over the restricted class of i.i.d. measures, while the per-symbol length n redundancy allows for the infimum over all possible measures q. Thus, the per-symbol length n redundancy is upper bounded by the single letter redundancy. Any difference between the two can be thought of as the advantage accrued, because the universal measure learns the underlying measure p.

In this paper, our primary goal is to understand the connections between the single-letter redundancy, on the one hand, and the behavior of length n i.i.d. redundancy, on the other. As mentioned in the Introduction, length n redundancy is the capacity of a channel from $𝒫$ to ${\mathbb{Z}}_{+}^n$, where the conditional probability distribution over ${\mathbb{Z}}_{+}^n$ given p ∈ $𝒫$ is simply the distribution p over length n sequences. Roughly speaking, it quantifies how much information about the source we can extract from the sequence.

We will often speak of the per-symbol length n redundancy, which is simply length n redundancy normalized by n, i.e.,, R_n($𝒫$^∞)/n. Furthermore, the limit lim sup_n→∞ R_n($𝒫$^∞)/n is the asymptotic per-symbol redundancy. Whether the asymptotic per-symbol redundancy is zero (we will equivalently say that the asymptotic per-symbol redundancy diminishes to zero to keep in line with prior literature) is in many ways a litmus test for compression, estimation and other related problems. Loosely speaking, if R_n($𝒫$^∞)/n → 0, the redundancy-capacity interpretation [4] mentioned above implies that after a point, there is little further information to be learned when we see an additional symbol, no matter what the underlying source is. In this sense, this is the case where we can actually learn the underlying model at a uniform rate over the entire class.

We note that it is possible to define an even more stringent notion—a worst-case-regret. For length n sequences, this is:

$$\underset{p}{\text{inf}}\underset{p\in {𝒫}^{\infty }}{\text{sup}}\underset{X^n}{\text{sup}}\text{log}\frac{p(X^n)}{q(X^n)}.$$

Single-letter regret is the special case where n = 1, and asymptotic per-symbol regret is the limit as n → ∞ of the length n regret normalized by n. We will not concern ourselves with the worst case formulation in this paper, but mention it in passing for comparison. In the worst-case setting, finite single letter redundancy is necessary and sufficient [2] for the asymptotic per-symbol worst-case regret to diminish to zero.

Yet, we show in this paper that it is not necessarily the case for redundancy. It is quite possible that collections with finite single-letter redundancy have asymptotic per-symbol redundancy bounded away from zero.

2.2. Patterns

Recent work [1] has formalized a similar framework for countably infinite alphabets. This framework is based on the notion of patterns of sequences that abstract the identities of symbols and indicates only the relative order of appearance. For example, the pattern of PATTERN is 1233456. The k-th distinct symbol of a string is given an index k when it first appears, and that index is used every time the symbol appears henceforth. The crux of the patterns approach is to consider the set of measures induced over patterns of the sequences instead of considering the set of measures $𝒫$ over infinite sequences,

Denote the pattern of a string x by Ψ(x). There is only one possible pattern of strings of length one (no matter what the alphabet, the pattern of a length one string is one), two possible patterns of strings of length two (11 and 12), and so on. The number of possible patterns of length n is the n-th Bell number [1], and we denote the set of all possible length n patterns by Ψⁿ. The measures induced on patterns by a corresponding measure p on infinite sequences of positive integers assigns to any pattern ψ a probability:

$$p(\psi )=p\left(\{\mathbf{x}:\Psi (\mathbf{x})=\psi \}\right).$$

In [1], the length n pattern redundancy,

$$\underset{p}{\text{inf}}\underset{p\in {𝒫}^{\infty }}{\text{sup}}{E}_p\text{log}\frac{p(\Psi (X^n))}{q(\Psi (X^n))},$$

was shown to be upper bounded by $\pi (\text{log}e)\sqrt{\frac{2n}{3}}$. It was also shown in [6] that there is a measure q over infinite length sequences that satisfies for all n simultaneously:

$$\underset{p\in {𝒫}^{\infty }}{\text{sup}}\underset{X^n}{\text{sup}}\text{log}\frac{p(\Psi (X^n))}{q(\Psi (X^n))}\le \pi (\text{log}e)\sqrt{\frac{2n}{3}}+\text{log}(n(n+1)).$$

Let the measure induced on patterns by q be denoted as q_Ψ for convenience.

We can interpret the probability estimator q_Ψ as a sequential prediction procedure that estimates the probability that the symbol X_n+1 will be “new” (has not appeared in $X_1^n$) and the probability that X_n+1 takes a value that has been seen so far. This view of estimation also appears in the statistical literature on Bayesian nonparametrics that focuses on exchangeability. Kingman [7] advocated the use of exchangeable random partitions to accommodate the analysis of data from an alphabet that is not bounded or known in advance. A more detailed discussion of the history and philosophy of this problem can be found in the works of Zabell [8,9] collected in [10].

2.3. Cumulative Distributions and Tight Collections

For our purposes, the cumulative distribution function of any probability distribution p on ℤ₊ (ℕ, respectively) is a function F_p : ℝ ∪ {∞} → [0, 1] defined in the following (slightly unconventional) way. We let F_p(0) = 0 in case the support is ℤ₊ (F_p(−1) = 0 if the support is ℕ, respectively). We then define F_p on points in the support of p in the way cumulative distribution functions are normally defined. Specifically for all y in the support of p,

$$F_p(y)=\sum _{j\ge 0}^{y}p(j).$$

We let F_p(−∞) ≔ 0 and F_p(∞) ≔ 1. Finally, we extend the definition of F_p to all real numbers by linearly interpolating between the values defined already.

Let $F_p^{-1}:[0,1]\mapsto ℝ\cup \{\infty \}$ denote the inverse function of F_p defined as follows. To begin with,

$$F_p^{-1}(0)=\text{sup}\{y:F_p(y)=0\},$$

If p has infinite support, then $F_p^{-1}(1)=\infty$, else $F_p^{-1}(1)$ is the smallest positive integer y, such that F_p(y) = 1. It follows [11] then that:

$$p\left\{x\in {\mathbb{Z}}_{+}:x\ge F_q^{-1}(1-\gamma )\right\}>\gamma \hspace{0.17em}\text{and}\hspace{0.17em}\hspace{0.17em}p\left\{x\in {\mathbb{Z}}_{+}:x>2F_q^{-1}\left(1-\frac{\gamma }{2}\right)\right\}\le \gamma$$

A collection $𝒫$ of distributions on ℤ₊ is defined to be tight if for all γ > 0,

$$\underset{p\in 𝒫}{\text{sup}}{F}_p^{-1}(1-\gamma )<\infty .$$

3. Redundancy and Tightness

We focus on the single-letter redundancy in this section and explore the connections between the single-letter redundancy of a collection $𝒫$ and the tightness of $𝒫$.

Lemma 1. A collection $𝒫$ over ℕ with bounded length n redundancy is tight. Namely, if the single-letter redundancy of $𝒫$ is finite, then for any γ > 0:

$$\underset{p\in 𝒫}{\text{sup}}{F}_p^{-1}(1-\gamma )<\infty .$$

Proof Since $𝒫$ has bounded single-letter redundancy, fix a distribution q over ℕ, such that:

$$\underset{p\in 𝒫}{\text{sup}}D(p\Vert q)<\infty .$$

We define R ≝ sup_p∈$𝒫$ D(p‖q) where D(p‖q) is the Kullback–Leibler distance between p and q. We will first show that for all p ∈ $𝒫$ and any m > 0,

$$p\left(\left|\text{log}\frac{p(X)}{q(X)}\right|>m\right)\le (R+(2\text{log}e)/e)/m.$$

(2)

To see Equation (2), let S be the set of all x ∈ ℕ, such that p(x) < q(x). A well-known convexity argument shows that the partial contribution to KL divergence from S,

$$\sum _{x\in S}p(x)\text{log}\frac{p(x)}{q(x)}\ge p(S)\text{log}\frac{p(S)}{q(S)}\ge -\frac{\text{log}e}{e},$$

and hence:

$$\sum _{x\in {\mathbb{Z}}_{+}}p(x)\left|\text{log}\frac{p(x)}{q(x)}\right|=\sum _{x\in {\mathbb{Z}}_{+}}p(x)\text{log}\frac{p(x)}{q(x)}-2\sum _{x\in S}p(x)\text{log}\frac{p(x)}{q(x)}\le R+2\frac{\text{log}e}{e}.$$

Then, Equation (2) follows by a simple application of Markov’s inequality.

We will now use Equation (2) to complete the proof of the lemma. Specifically, we will show that for all γ > 0,

$$\underset{p\in 𝒫}{\text{sup}}{F}_p^{-1}(1-\gamma )<2F_q^{-1}(1-\gamma /2^{m^{*}+2})$$

where m^* is the smallest integer, such that (R + (2 log e)/e)/m^* < γ/2. Equivalently, for all γ > 0 and p ∈ $𝒫$, we show that:

$$p\left\{x\in \mathbb{N}:x>2F_q^{-1}\left(1-\gamma /2^{m^{*}+2}\right)\right\}\le \gamma .$$

We prove the above by partitioning q′s tail; numbers $x\ge 2F_q^{-1}(1-\gamma /2^{m^{*}+2})$ into two parts.

(i)

the set $W_1=\{x\in \mathbb{N}:x>2F_q^{-1}(1-\gamma /2^{m+2}$ and $\text{log}\frac{p(x)}{q(x)}>m^{*}\}$. Clearly:

$$W_1\subseteq \left\{y\in \mathbb{N}:\left|\text{log}\frac{p(y)}{q(y)}\right|>m^{*}\right\},$$

and thus:

$$p(W_1)\le p\left\{y\in \mathbb{N}:\left|\text{log}\frac{p(y)}{q(y)}\right|>m^{*}\right\}\le \frac{\gamma }{2}$$

where the right inequality follows from Equation (2).

(ii)

the set $W_2=\{x\in \mathbb{N}:x>2F_q^{-1}(1-\gamma /2^{m^{*}+2})$ and $\text{log}\frac{p(x)}{q(x)}\le m^{*}\}$. Clearly:

$$W_2\subseteq \left\{y\in \mathbb{N}:y>2F_q^{-1}\left(1-\gamma /2^{m^{*}+2}\right)\right\}$$

and therefore:

$$q\left(W_2\right)\le q\left\{y\in \mathbb{N}:y>2F_q^{-1}\left(1-\gamma /2^{m^{*+2}}\right)\right\}\le \frac{\gamma }{2^{m^{*}+1}}.$$

By definition, all x ∈ W₂ satisfy $\text{log}\frac{p(x)}{q(x)}\le m^{*}$ or that p(x) ≤ q(x)2^m*. Hence, we have:

$$p(W_2)\le q(W_2)2^{m^{*}}\le \frac{\gamma 2^{m^{*}}}{2^{m^{*}+1}}=\frac{\gamma }{2}.$$

The lemma follows.

The converse is not necessarily true. Tight collections need not have finite single-letter redundancy, as the following example demonstrates.

Construction: Consider the following collection $ℐ$ of distributions over ℤ₊. First, partition the set of positive integers into the sets T_i, i ∈ ℕ, where:

$$T_i=\{2^i,\dots ,2^{i+1}-1\}.$$

Note that |T_i| =2ⁱ. Now, $ℐ$ is the collection of all possible distributions that can be formed as follows: for all i ∈ ℤ₊, pick exactly one element of T_i and assign probability 1/((i +1)(i +2)) to the element of T_i chosen choosing the support as above implicitly assumes the axiom of choice. Note that the set $ℐ$ is uncountably infinite.

Corollary 2. The set $ℐ$ of distributions is tight.

Proof For all p ∈ $ℐ$,

$$\sum _{\begin{array}{l}x\ge 2^k\\ x\in {\mathbb{Z}}_{+}\end{array}}p(x)=\frac{1}{k+1},$$

namely, all tails are uniformly bounded over the collection $ℐ$. Put another way, for all δ > 0 and all distributions: p ∈ $ℐ$,

$$F_p^{-1}(1-\delta )\le 2^{\frac{1}{\delta }}.$$

On the other hand:

Proposition 1. The collection $ℐ$ does not have finite redundancy.

Proof Suppose q is any distribution over ℤ₊. We will show that ∃p ∈ $ℐ$, such that:

$$\sum _{x\in {\mathbb{Z}}_{+}}p(x)\text{log}\frac{p(x)}{q(x)}$$

is not finite. Since the entropy of every p ∈ $ℐ$ is finite, we just have to show that for any distribution q over ℤ₊, there exists p ∈ $ℐ$, such that:

$$\sum _{x\in {\mathbb{Z}}_{+}}p(x)\text{log}\frac{1}{q(x)}$$

is not finite.

Consider any distribution q over ℤ₊. Observe that for all i, |T_i| =2ⁱ. It follows that for all i, there is x_i ∈ T_i, such that:

$$q(x_i)\le \frac{1}{2^i}.$$

However, by construction, $ℐ$ contains a distribution p^* that has for its support {x_i : i ∈ ℤ₊} identified above. Furthermore p^* assigns:

$$p^{*}(x_i)=\frac{1}{(i+1)(i+2)}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\forall i\in {\mathbb{Z}}_{+}.$$

The KL divergence from p^* to q is not finite, and the Lemma follows, since q is arbitrary.

4. Length n Redundancy

We study how the single-letter properties of a collection $𝒫$ of distributions influence the compression of length n strings obtained by i.i.d. sampling from distributions in $𝒫$. Namely, we try to characterize when the length n redundancy of $𝒫$^∞ grows sublinearly in the block length n.

Lemma 3. Let $𝒫$ be a collection of distributions over a countable support $𝒳$. For some m ∈ ℤ₊, consider m pairwise disjoint subsets S_i ⊂ $𝒳$ (1 ≤ i ≤ m), and let δ > 1/2. If there exist p₁, ..., p_m ∈ $𝒫$, such that:

$$p_i(S_i)\ge \delta ,$$

then for all distributions q over $𝒳$,

$$\underset{p\in 𝒫}{\text{sup}}D(p\Vert q)\ge \delta \text{log}m.$$

In particular if there are an infinite number of sets S_i, i ∈ ℤ₊ and distributions p_i ∈ $𝒫$, such that p_i(S_i) ≥ δ, then the redundancy is infinite.

Proof This is a simplified formulation of the distinguishability concept in [4]. For a proof, see e.g., [12].

4.1. Counterexample

We now show that it is possible for the single-letter redundancy of a collection $ℬ$ of distributions to be finite, yet the asymptotic per-symbol redundancy (the length n redundancy of $ℬ$^∞ normalized by n) remains bounded away from zero; in the limit, the block length goes to infinity. To show this, we obtain such a collection $ℬ$.

Construction: As before, partition the set ℤ₊ into T_i = {2ⁱ, ..., 2ⁱ⁺¹ − 1}, i ∈ ℕ. Recall that T_i has 2ⁱ elements. For all 0 < ∊ ≤ 1, let $n_{∊}=\lfloor \frac{1}{∊}\rfloor$. Let 1 ≤ j ≤ 2^n_∊, and let p_∊,j be a distribution on ℤ₊ that assigns probability 1 − ∊ to the number one (or equivalently, to the set T₀) and ∊ to the j-th smallest element of T_n∊, namely the number 2^n_∊ + j − 1. $ℬ$ (mnemonic for binary, since every distribution has a support of size two) is the collection of distributions p_∊,j for all ∊ > 0 and 1 ≤ j ≤ 2^n_∊. $ℬ$^∞ is the set of measures over infinite sequences of numbers corresponding to i.i.d. sampling from $ℬ$.

We first verify that the single-letter redundancy of $ℬ$ is finite.

Proposition 2. Let q be a distribution that assigns $q(T_i)=\frac{1}{(i+1)(i+2)}$ and for all j ∈ T_i,

$$q(j|T_i)=\frac{1}{\left|T_i\right|}.$$

Then:

$$\underset{p\in ℬ}{\text{sup}}\sum _{x\in {\mathbb{Z}}_{+}}p(x)\text{log}\frac{p(x)}{q(x)}\le 2.$$

However, the redundancy of compressing length n sequences from $ℬ$^∞ scales linearly with n.

Proposition 3. For all n ∈ ℤ₊,

$$\underset{q}{\text{inf}}\underset{p\in B^{\infty }}{\text{sup}}{E}_p\text{log}\frac{p(X^n)}{q(X^n)}\ge n{\left(1-\frac{1}{n}\right)}^n.$$

Proof Let the set {1ⁿ} denote a set containing a length n sequence of only ones. For all n, define 2ⁿ pairwise disjoint sets S_i of ${\mathbb{Z}}_{+}^n$, 1 ≤ i ≤ 2ⁿ, where:

$$S_i={\left\{1,2^n+i-1\right\}}^n-\left\{1^n\right\}$$

is the set of all length n strings containing at most two numbers (one and 2ⁿ + i − 1) and at least one occurrence of 2ⁿ + i − 1. Clearly, for distinct i and j between one and 2ⁿ, S_i and S_j are disjoint. Furthermore, the measure $p_{\frac{1}{n},i}\in {ℬ}^{\infty }$ assigns S_i the probability:

$$p_{\frac{1}{n},i}(S_i)=1-{\left(1-\frac{1}{n}\right)}^n>1-\frac{1}{e}.$$

From Lemma 3, it follows that length n redundancy of $ℬ$^∞ is lower bounded by:

$$\left(1-\frac{1}{e}\right)\text{log}{2}^n=n\left(1-\frac{1}{e}\right).$$

In a preview of what is to come, we notice that though the single-letter redundancy of the class $ℬ$ over ℤ₊ is finite, the single-letter tail redundancy, as described in the equation below, does not diminish to zero; namely, for all M:

$$\underset{p\in ℬ}{\text{sup}}\sum _{x\ge M}p(x)\text{log}\frac{p(x)}{q(x)}\ge 1.$$

In fact, in the next section, we relate the single-letter tail redundancy above diminishing to zero to sublinear growth of the i.i.d. length n redundancy.

4.2. Sufficient Condition

In this section, we show a sufficient condition on single-letter marginals of $𝒫$ and its redundancy that allows for i.i.d. length n redundancy of $𝒫$^∞ to grow sublinearly with n. This condition is, however, not necessary; and the characterization of a condition that is both necessary and sufficient is as yet open.

For all ∊ > 0, let A_p,∊ be the set of all elements in the support of p with probability ≥∊, and let T_p,∊ = ℤ₊ − A_p,∊. Let G₀ = {ϕ}, where ϕ denotes the empty string. For all i, the sets:

$$G_i=\left\{x^i:A_{p,\frac{2\text{ln}(i+1)}{i}}\subseteq \{x_1,x_2,\dots ,x_i\}\right\}$$

where, in a minor abuse of notation, we use {x₁, ...,x_i} to denote the set of distinct symbols in the string $x_1^i$. Let B₀ = {}, and let $B_i={\mathbb{Z}}_{+}^i-G_i$. Observe from an argument similar to the coupon collector problem that:

Lemma 4. For all i ≥ 2,

$$p(B_i)\le \frac{1}{(i+1)\text{ln}(i+1)}.$$

Proof The proof follows from an elementary union bound:

$$\begin{array}{ll}p(B_i)\hfill & \le \left|A_{p,\frac{2\text{ln}(i+1)}{i}}\right|{\left(1-\frac{2\text{ln}(i+1)}{i}\right)}^i\hfill \\ \hfill & \le \frac{i}{2\text{ln}(i+1)}{\left(1-\frac{2\text{ln}(i+1)}{i}\right)}^i\hfill \\ \hfill & \le \frac{i}{2\text{ln}(i+1)}{e}^{\frac{-2i\text{ln}(i+1)}{i}}\hfill \\ \hfill & \le \frac{i}{2{(i+1)}^2\text{ln}(i+1)}\hfill \\ \hfill & \le \frac{1}{(i+1)\text{ln}(i+1)}.\hfill \end{array}$$

Theorem 5. Suppose $𝒫$ is a collection of distributions over ℤ₊. Let the entropy be uniformly bounded over the entire collection, and in addition, let the redundancy of the collection be finite. Namely,

$$\underset{p\in 𝒫}{\text{sup}}\sum _{x\in {\mathbb{Z}}_{+}}p(x)\text{log}\frac{1}{p(x)}\stackrel{\scriptscriptstyle\mathrm{def}}{=}H<\infty \hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\text{and}\hspace{0.17em}\hspace{0.17em}\exists q_1\hspace{0.17em}\hspace{0.17em}\text{over}\hspace{0.17em}\hspace{0.17em}{\mathbb{Z}}_{+}\hspace{0.17em}\hspace{0.17em}\text{s}.\text{t}.\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\underset{p\in 𝒫}{\text{sup}}\sum _{x\in {\mathbb{Z}}_{+}}p(x)\text{log}\frac{p(x)}{q_1(x)}<\infty .$$

We will denote:

$$R=\underset{p\in 𝒫}{\text{sup}}\sum _{x\in {\mathbb{Z}}_{+}}p(x)\text{log}\frac{p(x)}{q_1(x)}.$$

Recall that for any distribution p, the set T_p,δ denotes the support of p, all of whose probabilities are < δ. Let:

$$\underset{\delta \to 0}{\text{lim}}\underset{p\in 𝒫}{\text{sup}}\sum _{x\in T_{p,\delta }}p(x)\text{log}\frac{1}{p(x)}=0\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\text{and}\hspace{0.17em}\hspace{0.17em}\exists q_1\hspace{0.17em}\hspace{0.17em}\text{over}\hspace{0.17em}\hspace{0.17em}{\mathbb{Z}}_{+}\hspace{0.17em}\hspace{0.17em}\text{s}.\text{t}.\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\underset{\delta \to 0}{\text{lim}}\underset{p\in 𝒫}{\text{sup}}\sum _{x\in T_{p,\delta }}p(x)\text{log}\frac{p(x)}{q_1(x)}=0.$$

(3)

Then, the redundancy of length n distributions obtained by i.i.d. sampling from distributions in $𝒫$, denoted by R_n($𝒫$^∞), grows sublinearly:

$$\underset{n\to \infty }{\text{limsup}}\frac{1}{n}{R}_n({𝒫}^{\infty })=0.$$

Remark If the conditions of the theorem are met, we can always assume without loss of generality that there is a distribution q₁ that satisfies (3) and simultaneously has finite redundancy. To see this, suppose $q_1^{\prime }$ satisfies the finite-redundancy condition, namely:

$$\underset{p\in 𝒫}{\text{sup}}\sum _{x\in T_{p,\delta }}p(x)\text{log}\frac{p(x)}{q_1^{\prime }(x)}=0;$$

while a different distribution $q_1^{″}$ satisfies the second tail-redundancy condition,

$$\underset{\delta \to 0}{\text{lim}}\underset{p\in 𝒫}{\text{sup}}\sum _{x\in T_{p,\delta }}p(x)\text{log}\frac{p(x)}{q_1^{″}(x)}=0.$$

It is easy to verify that the distribution q that assigns to any x ∈ ℤ₊, $q_1(x)=\frac{q_1^{\prime }(x)+q_1^{″}(x)}{2}$ satisfies both conditions simultaneously.

Proof In what follows, xⁱ represents a string x₁, ..., x_i and x⁰ denotes the empty string. For all n, we denote Ψ(xⁿ) = ψ₁, ..., ψ_n and Ψ(Xⁿ)=Ψ₁, ..., Ψ_n.

We will construct q, such that $\text{lim}{\text{sup}}_{n\to \infty }\frac{1}{n}{E}_p\text{log}\frac{p(X^n)}{q(X^n)}=0$. Recall that q_Ψ is the optimal universal pattern encoder over patterns of i.i.d. sequences defined in Section 2.2. Furthermore, recall that the redundancy of $𝒫$ is finite and that q₁ is the universal distribution over ℤ₊ that attains redundancy R for $𝒫$.

The universal encoder q is now defined as follows:

$$\begin{array}{ll}q(x^n)\hfill & =q(x^n,\Psi (x^n))\hfill \\ \hfill & =q\left({\psi }_1,x_1,{\psi }_2,x_2,\dots ,{\psi }_n,x_n\right)\hfill \\ \hfill & =\prod _{i\in {\mathbb{Z}}_{+}}q\left({\psi }_i|{\psi }_1^{i-1},x_1^{i-1}\right)\prod _{j\in {\mathbb{Z}}_{+}}q\left(x_j|{\psi }_1^j,x_1^{j-1}\right)\hfill \\ \hfill & \stackrel{\scriptscriptstyle\mathrm{def}}{=}\prod _{i\in {\mathbb{Z}}_{+}}{q}_{\Psi }\left({\psi }_i|{\psi }_1^{i-1}\right)\prod _{j\in {\mathbb{Z}}_{+}}q\left(x_j|{\psi }_1^j,x_1^{j-1}\right).\hfill \end{array}$$

Furthermore, we define for all $x_1^{i-1}\in {\mathbb{Z}}_{+}^{i-1}$ and all ψⁱ ∈ Ψⁱ, such that ψⁱ⁻¹ =Ψ(xⁱ⁻¹),

$$q\left(x_i|{\psi }_1^i,x_1^{i-1}\right)\stackrel{\scriptscriptstyle\mathrm{def}}{=}\{\begin{array}{ll}1\hfill & \text{if}\hspace{0.17em}{x}_i\in \{x_1,\dots ,x_{i-1}\}\hspace{0.17em}\hspace{0.17em}\text{and}\hspace{0.17em}\hspace{0.17em}\Psi (x^i)={\psi }^i\hfill \\ q_1(x_i)\hfill & \text{if}\hspace{0.17em}{x}_i\notin \{x_1,\dots ,x_{i-1}\}\hspace{0.17em}\hspace{0.17em}\text{and}\hspace{0.17em}\hspace{0.17em}\Psi (x^i)={\psi }^i.\hfill \end{array}$$

Namely, we use an optimal universal pattern encoder over patterns of i.i.d. sequences and encode any new symbol using a universal distribution over $𝒫$. We now bound the redundancy of q as defined above. We have for all p ∈ $𝒫$^∞,

$$\begin{array}{ll}{E}_p\text{log}\frac{p(X^n)}{q(X^n)}\hfill & =\sum _{x^n}p(x^n)\text{log}\prod _{i\in {\mathbb{Z}}_{+}}\frac{p\left({\psi }_i|{\psi }_1^{i-1},x_1^{i-1}\right)}{q_{\Psi }\left({\psi }_i|{\psi }_1^{i-1}\right)}\prod _{j\in {\mathbb{Z}}_{+}}\frac{p\left(x_j|{\psi }_1^j,x_1^{j-1}\right)}{q\left(x_j|{\psi }_1^j,x_1^{j-1}\right)}\hfill \\ \hfill & =\sum _{x^n}p(x^n)\sum _{i=1}^n\text{log}\frac{p\left({\psi }_i|{\psi }_1^{i-1},x_1^{i-1}\right)}{q_{\Psi }\left({\psi }_i|{\psi }_1^{i-1}\right)}+\sum _{x^n}p(x^n)\sum _{j=1}^n\text{log}\frac{p\left(x_j|{\psi }_1^j,x_1^{j-1}\right)}{q\left(x_j|{\psi }_1^j,x_1^{j-1}\right)}.\hfill \end{array}$$

Since ψ₁ is always one, p(ψ₁) = q_Ψ(ψ₁) = 1. Therefore, we have:

$$\sum _{x^n}p(x^n)\sum _{i=1}^n\text{log}\frac{p\left({\psi }_i|{\psi }_1^{i-1},x_1^{i-1}\right)}{q_{\Psi }\left({\psi }_i|{\psi }_1^{i-1}\right)}=\sum _{x^n}p(x^n)\sum _{i=2}^n\text{log}\frac{p\left({\psi }_i|{\psi }_1^{i-1},x_1^{i-1}\right)}{q_{\Psi }\left({\psi }_i|{\psi }_1^{i-1}\right)}.$$

The first term, normalized by n, can be upper bounded as follows:

$$\begin{array}{l}\frac{1}{n}\sum _{x^n}p(x^n)\sum _{i=2}^n\text{log}\frac{p\left({\psi }_i|{\psi }_1^{i-1},x_1^{i-1}\right)}{q_{\Psi }\left({\psi }_i|{\psi }_1^{i-1}\right)}\\ \hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\le \frac{1}{n}\sum _{i=2}^n\sum _{x_1^i}p(x_1^i)\text{log}\frac{p\left({\psi }_i|{\psi }_1^{i-1},x_1^{i-1}\right)}{p\left({\psi }_i|{\psi }_1^{i-1}\right)}+\frac{1}{n}\left(\pi \text{log}e\sqrt{\frac{2n}{3}}+\text{log}n(n+1)\right)\\ \hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}=\frac{1}{n}\sum _{i=2}^n\left(H\left({\Psi }_i|{\Psi }_1^{i-1}\right)-H\left({\Psi }_i|X_1^{i-1}\right)\right)+\frac{1}{n}\left(\pi \text{log}e\sqrt{\frac{2n}{3}}+\text{log}n(n+1)\right)\\ \hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\le \frac{1}{n}(n{H}_p)-\frac{1}{n}\sum _{i=2}^{n}H\left({\Psi }_i|X_1^{i-1})\right)+\frac{1}{n}\left(\pi \text{log}e\sqrt{\frac{2n}{3}}+\text{log}n(n+1)\right)\end{array}$$

where we define H_p as:

$$H_p\stackrel{\scriptscriptstyle\mathrm{def}}{=}\sum _{x\in {\mathbb{Z}}_{+}}p(x)\text{log}\frac{1}{p(x)}$$

and the last inequality follows, since:

$$H\left({\Psi }^n\right)\le H(X^n)=n{H}_p.$$

Now for i ≥ 2,

$$\begin{array}{ll}H\left({\Psi }_i|X_1^{i-1}\right)\hfill & =\sum p\left({\psi }_i|x^{i-1}\right)p(x^{i-1})\text{log}\frac{1}{p\left({\psi }_i|x_1^{i-1}\right)}\hfill \\ \hfill & =\sum p(x^{i-1})\left(\sum _{x\in \{x_1,\dots ,x_{i-1}\}}p(x)\text{log}\frac{1}{p(x)}+\sum _{y\notin \{x_1,\dots ,x_{i-1}\}}p(y)\text{log}\frac{1}{{\sum }_{y\notin \{x_1,\dots ,x_{i-1}\}}p(y)}\right)\hfill \end{array}$$

Then,

$$\begin{array}{ll}{H}_p-H\left({\Psi }_i|X_1^{i-1}\right)\hfill & =\sum _{x^{i-1}}p(x^{i-1})H_p-H\left({\Psi }_i|X_1^{i-1}\right)\hfill \\ \hfill & =\sum _{x^{i-1}}p(x^{i-1})\sum _{x_i\notin \{x_1,\dots ,x_{i-1}\}}p(x_i)\text{log}\frac{1}{p(x_i)}\hfill \\ \hfill & \hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}-\sum _{x^{i-1}}p(x^{i-1})\sum _{x_i\notin \{x_1,\dots ,x_{i-1}\}}p(x_i)\text{log}\frac{1}{{\sum }_{x_i\notin \{x_1,\dots ,x_{i-1}\}}p(x_i)}\hfill \\ \hfill & \le \sum _{x^{i-1}}p\left(x_1^{i-1}\right)\sum _{x_i\notin \{x_1,\dots ,x_{i-1}\}}p(x_i)\text{log}\frac{1}{p(x_i)}\hfill \\ \hfill & \le p(G_{i-1})\sum _{x_i\notin T_{p,2\frac{\text{ln}i}{i-1}}}p(x_i)\text{log}\frac{1}{p(x_i)}+p(B_{i-1})H\hfill \\ \hfill & \le \sum _{x_i\in T_{p,2\frac{\text{ln}i}{i-1}}}p(x_i)\text{log}\frac{1}{p(x_i)}+\frac{H}{i\text{ln}i}.\hfill \end{array}$$

We have split the length i − 1 sequences into the sets G_i−1 and B_i−1 and use separate bounds on each set that hold uniformly over the entire model collection. The last inequality above follows from Lemma 4. From Condition (3) of the Theorem, we have that:

$$\underset{i\to \infty }{\text{lim}}\underset{p\in 𝒫}{\text{sup}}\sum _{x\in T_{p,2\frac{\text{ln}i}{i-1}}}p(x)\text{log}\frac{1}{p(x)}=0.$$

Therefore, we have:

$$\begin{array}{c}\underset{n\to \infty }{\text{lim}}\underset{p\in 𝒫}{\text{sup}}\frac{1}{n}\sum _{i=2}^n\left(\sum _{x\in T_{p,2\frac{\text{ln}i}{i-1}}}p(x)\text{log}\frac{1}{p(x)}+\frac{H}{i\text{ln}i}\right)\le \underset{n\to \infty }{\text{lim}}\frac{1}{n}\sum _{i=2}^n\left(\underset{p\in 𝒫}{\text{sup}}\sum _{x\in T_{p,2\frac{\text{ln}i}{i-1}}}p(x)\text{log}\frac{1}{p(x)}+\frac{H}{i\text{ln}i}\right)\\ \stackrel{(a)}{=}0.\end{array}$$

The first term on the left in the first equation above is non-negative, hence the limit above has to equal zero. The equality (a) follows from Cesaro’s lemma asserting that for any sequence {a_i, i ∈ ℤ₊} with a_i < ∞ for all i, if lim_i→∞ a_i exists, then:

$$\underset{i\to \infty }{\text{lim}}{a}_i=\underset{n\to \infty }{\text{lim}}\frac{1}{n}\sum _{j=1}^n{a}_j.$$

Therefore,

$$\underset{n\to \infty }{\text{lim}}\underset{p\in 𝒫}{\text{sup}}\frac{1}{n}\sum _{x^n}p(x^n)\sum _{i=2}^n\text{log}\frac{p\left({\psi }_i|{\psi }_1^{i-1},x_1^{i-1}\right)}{q_{\Psi }\left({\psi }_i|{\psi }_1^{i-1}\right)}=0.$$

For the second term, observe that:

$$\sum _{x^n}p(x^n)\sum _{j=1}^n\text{log}\frac{p\left(x_j|{\psi }_1^j,x_1^{j-1}\right)}{q\left(x_j|{\psi }_1^j,x_1^{j-1}\right)}\le R+\sum _{x^n}p(x^n)\sum _{j=2}^n\text{log}\frac{p\left(x_j|{\psi }_1^j,x_1^{j-1}\right)}{q\left(x_j|{\psi }_1^j,x_1^{j-1}\right)}.$$

Furthermore,

$$\begin{array}{ll}\sum _{x^n}p(x^n)\sum _{j=2}^n\text{log}\frac{p\left(x_j|{\psi }_1^j,x_1^{j-1}\right)}{q\left(x_j|{\psi }_1^j,x_1^{j-1}\right)}\hfill & =\sum _{j=2}^n\sum _{x^j}p(x^j)\text{log}\frac{p\left(x_j|{\psi }_1^j,x_1^{j-1}\right)}{q\left(x_j|{\psi }_1^j,x_1^{j-1}\right)}\hfill \\ \hfill & \le \sum _{j=2}^n\sum _{x^{j-1}}p(x^{j-1})\sum _{x_j\notin \{x_1,\dots ,x_{j-1}\}}p(x_j)\text{log}\frac{p(x_j)}{q_1(x_j)}\hfill \\ \hfill & \le \sum _{j=2}^n\left(p(G_{j-1})\sum _{x_j\in T_{p,\frac{2\text{ln}j}{j-1}}}p(x_j)\text{log}\frac{p(x_j)}{q_1(x_j)}+Rp(B_{j-1})\right)\hfill \\ \hfill & \le \sum _{j=2}^n\sum _{x_j\in T_{p,\frac{2\text{ln}j}{j-1}}}p(x_j)\text{log}\frac{p(x_j)}{q_1(x_j)}+\frac{R}{j\text{ln}j}.\hfill \end{array}$$

As before, the last inequality is from Lemma 4. Again, from Condition (3), we have:

$$\underset{j\to \infty }{\text{lim}}\left(\underset{p\in 𝒫}{\text{sup}}\sum _{x_j\in T_{p,\frac{2\text{ln}j}{j-1}}}p(x_j)\text{log}\frac{p(x_j)}{q_1(x_j)}+\frac{R}{j\text{ln}j}\right)=0.$$

Therefore, as before:

$$\underset{n\to \infty }{\text{lim}}\underset{p\in 𝒫}{\text{sup}}\frac{1}{n}\left(\sum _{j=1}^n\sum _{x_j\in T_{p,\frac{2\text{ln}j}{j-1}}}p(x_j)\text{log}\frac{p(x_j)}{q_1(x_j)}+\sum _{j=2}^n\frac{R}{j\text{ln}j}\right)=0$$

as well. The theorem follows.

A few comments about (3) in Theorem 5 are in order. Neither condition automatically implies the other. The set $ℬ$ of distributions in Section 4.1 is an example where every distribution has finite entropy, the redundancy of $ℬ$ is finite,

$$\underset{\delta \to 0}{\text{lim}}\underset{p\in ℬ}{\text{sup}}\sum _{x\in T_{p,\delta }}p(x)\text{log}\frac{1}{p(x)}=0\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\text{but}\hspace{0.17em}\hspace{0.17em}\forall q\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\text{over}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}{\mathbb{Z}}_{+}\hspace{0.17em}\hspace{0.17em}\text{s}.\text{t}.\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\underset{\delta \to 0}{\text{lim}}\underset{p\in ℬ}{\text{sup}}\sum _{x\in T_{p,\delta }}p(x)\text{log}\frac{p(x)}{q(x)}>0.$$

We will now construct another set $𝒰$ of distributions over ℤ₊, such that every distribution in $𝒰$ has finite entropy, the redundancy of $𝒰$ is finite,

$$\underset{\delta \to 0}{\text{lim}}\underset{p\in 𝒰}{\text{sup}}\sum _{x\in T_{p,\delta }}p(x)\text{log}\frac{1}{p(x)}>0\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\text{but}\hspace{0.17em}\hspace{0.17em}\exists q\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\text{over}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}{\mathbb{Z}}_{+}\hspace{0.17em}\hspace{0.17em}\text{s}.\text{t}.\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\underset{\delta \to 0}{\text{lim}}\underset{p\in 𝒰}{\text{sup}}\sum _{x\in T_{p,\delta }}p(x)\text{log}\frac{p(x)}{q(x)}=0.$$

(4)

At the same time, the length n redundancy of $𝒰$^∞ diminishes sublinearly. This is therefore also an example to show that the conditions in Theorem 5 are only sufficient, but, in fact, not necessary. It is yet open to find a condition on single-letter marginals that is both necessary and sufficient for the asymptotic per-symbol redundancy to diminish to zero.

Construction: $𝒰$ is a countable collection of distributions p_k, k ∈ ℤ₊, on ℕ where:

$$p_k(x)=\{\begin{array}{ll}1-\frac{1}{k^2}\hfill & x=0,\hfill \\ \frac{1}{k^2{2}^{k^2}}\hfill & 1\le x\le 2^{k^2},\hfill \\ 0\hfill & x>2^{k^2}.\hfill \end{array}$$

The entropy of p_k ∈ $𝒰$ is therefore $1+h\left(\frac{1}{k^2}\right)$. Note that the redundancy of $𝒰$ is finite, too. To see this, first note that:

$$\sum _{x\in {\mathbb{Z}}_{+}}\underset{k\in {\mathbb{Z}}_{+}}{\text{sup}}{p}_k(x)\le \sum _{x\in {\mathbb{Z}}_{+}}\sum _{p_k:k\in {\mathbb{Z}}_{+}}{p}_k(x)=\sum _{p_k:k\in {\mathbb{Z}}_{+}}\sum _{x\in {\mathbb{Z}}_{+}}{p}_k(x)=\sum _{p_k:k\in {\mathbb{Z}}_{+}}\frac{1}{k^2}=\frac{{\pi }^2}{6}.$$

(5)

Now, letting: R⁺ ≝ log(∑_x∈ℤ+ sup_k∈ℕ p_k(x)), observe that the distribution:

$$q(x)=\{\begin{array}{ll}1/2\hfill & x=0,\hfill \\ \frac{{\text{sup}}_{k\in {\mathbb{Z}}_{+}}{p}_k(x)}{2^{R^{+}+1}}\hfill & x\in {\mathbb{Z}}_{+}.\hfill \end{array}$$

satisfies for all p_k ∈ $𝒰$:

$$\sum _{x\in \mathbb{N}}{p}_k(x)\text{log}\frac{p_k(x)}{q(x)}\le 1+\frac{R^{+}+1}{k^2}\le R^{+}+2,$$

implying that the redundancy of $𝒰$ is ≤ R⁺ + 2. Furthermore, Equation (5) implies that worst-case regret is finite, and from [2] the length n redundancy of $𝒰$^∞ diminishes sublinearly. Now, pick an integer m ∈ ℤ₊. We have for all p ∈ $𝒰$,

$$\sum _{x\in T_{p,\frac{1}{m^2{2}^{m^2}}}}p(x)\text{log}\frac{p(x)}{q(x)}\le \frac{R^{+}+1}{m^2},$$

yet, for all k ≥ m, we have:

$$\sum _{x\in T_{p,\frac{1}{m^2{2}^{m^2}}}}{p}_k(x)\text{log}\frac{1}{p_k(x)}=1.$$

Thus, the length n redundancy of $𝒰$^∞ diminishes to zero, while not satisfying all of the requirements of Theorem 5. Therefore, the conditions of Theorem 5 are only sufficient, not necessary.

5. Open Problems

We have demonstrated that finite single-letter redundancy of a collection $𝒫$ of distributions over a countably infinite support does not imply that the asymptotic per-symbol redundancy of i.i.d. samples from $𝒫$ diminishes to zero. This is in contrast to the scenario for worst-case regret, where single-letter worst-case regret, being finite, is both necessary and sufficient for asymptotic per-symbol regret to diminish to zero. We have also demonstrated sufficient conditions on the collection $𝒫$, so that asymptotic per-symbol redundancy of i.i.d. samples diminish to zero in this paper. However, as we show, the sufficient conditions we provide are not necessary. It is yet open to find a condition on single-letter marginals that is both necessary and sufficient for the asymptotic per-symbol redundancy to diminish to zero.