Universality of Logarithmic Loss in Successive Refinement ^†

Abstract

We establish an universal property of logarithmic loss in the successive refinement problem. If the first decoder operates under logarithmic loss, we show that any discrete memoryless source is successively refinable under an arbitrary distortion criterion for the second decoder. Based on this result, we propose a low-complexity lossy compression algorithm for any discrete memoryless source.

1. Introduction

In the lossy compression problem, logarithmic loss is a criterion allowing a “soft” reconstruction of the source, a departure from the classical setting of a deterministic reconstruction. In this setting, the reconstruction alphabet is the set of probability distributions over the source alphabet. More precisely, let x be the source symbol from the source alphabet $\mathcal{X}$, and $q(·)$ be the reconstruction symbol which is the probability measure on $\mathcal{X}$. Then the logarithmic loss is given by

$$\begin{array}{c}\hfill \ell (x,q)=log\frac{1}{q\left(x\right)}.\end{array}$$

Clearly, if the reconstruction $q(·)$ has a small probability on the true source symbol x, the amount of loss will be large.

Although logarithmic loss plays a crucial role in the theory of learning and prediction, relatively little work has been done in the context of lossy compression, notwithstanding the two-encoder multi-terminal source coding problem under logarithmic loss [1,2], or recent work on the single-shot approach to lossy source coding under logarithmic loss [3]. Note that lossy compression under logarithmic loss is closely related to the information bottleneck method [4,5,6]. In this paper, we focus on universal properties of logarithmic loss in the context of successive refinement.

Successive refinement is a network lossy compression problem where one encoder wishes to describe the source to two decoders [7,8]. Instead of having two separate coding schemes, the successive refinement encoder designs a code for the decoder with a weaker link, and sends extra information to the second decoder on top of the message of the first decoder. In general, successive refinement coding cannot do as well as two separate encoding schemes optimized for the respective decoders. However, if we can achieve the point-to-point optimum rates using successive refinement coding, we say the source is successively refinable.

Although necessary and sufficient conditions of successive refinability is known [7,8], proving (or disproving) successive refinability of the source is not a simple task. Equitz and Cover [7] found a discrete source that is not successively refinable using Gerrish problem [9]. Chow and Berger found a continuous source that is not successively refinable using Gaussian mixture [10]. Lastras and Berger showed that all sources are nearly successively refinable [11]. However, still only a few sources are known to be successively refinable. In this paper, we show that any discrete memoryless source is successively refinable as long as the weaker link employs logarithmic loss, regardless of the distortion criterion used for the stronger link.

In the second part of the paper, we show that this result can be useful to design a lossy compression algorithm with low complexity. Recently, the idea of successive refinement is applied to reduce the complexity of point-to-point lossy compression algorithm. Venkataramanan et al. proposed a new lossy compression for Gaussian source where the codewords are linear combination of sub-codewords [12]. No and Weissman also proposed a low-complexity lossy compression algorithm for Gaussian source using extreme value theory [13]. Both algorithms are successively describing source and achieve low complexity. Roughly speaking, successive refinement algorithm provides a smaller size of codebook. For example, the naive random coding scheme has a codebook of size $e^{nR}$ when the blocklength is n and the rate is R. On the other hand, if we can design a successive refinement scheme with half rate in the weaker link, then the size of codebook is $e^{nR/2}$ each. Thus, the overall codebook size is $2e^{nR/2}$. The above idea can be generalized to successive refinement scheme with L decoders [12,14]

The universal property of logarithmic loss in successive refinement implies that, for any point-to-point lossy compression of discrete memoryless source, we can insert a virtual intermediate decoder (weaker link) under logarithmic loss without losing any rates at the actual decoder (stronger link). As we discussed, this property allows us to design a lossy compression algorithm with low-complexity for any discrete source and distortion pair. Note that previous works only focused on specific source and distortion pair such as binary source with Hamming distortion.

The remainder of the paper is organized as follows. In Section 2, we revisit some of the known results pertaining to logarithmic loss. Section 3 is dedicated to successive refinement under logarithmic loss in the weaker link. In Section 4, we propose a low complexity compression scheme that can be applied to any discrete lossy compression problem. Finally, we conclude in Section 5.

Notation: $X^n$ denotes an n-dimensional random vector $(X_1,X_2,\dots ,X_n)$ while $x^n$ denotes a specific possible realization of the random vector $X^n$. $\mathcal{X}$ denotes a support of random variable X. Also, Q denotes a random probability mass function while q denotes a specific probability mass function. We use natural logarithm and nats instead of bits.

2. Preliminaries

2.1. Successive Refinability

In this section, we review the successive refinement problem with two decoders. Let the source $X^n$ be i.i.d. random vector with distribution $p_X$. The encoder wants to describe $X^n$ to two decoders by sending a pair of messages $(m_1,m_2)$ where $1\le m_i\le M_i$ for $i\in \{1,2\}$. The first decoder reconstructs ${\widehat{X}}_1^n\left(m_1\right)\in {\widehat{\mathcal{X}}}_1^n$ based only on the first message $m_1$. The second decoder reconstructs ${\widehat{X}}_2^n(m_1,m_2)\in {\widehat{\mathcal{X}}}_2^n$ based on both $m_1$ and $m_2$. The setting is described in Figure 1.

Let $d_i(·,·):\mathcal{X}\times {\widehat{\mathcal{X}}}_i\to [0,\infty )$ be a distortion measure for i-th decoder. The rates of code $(R_1,R_2)$ are simply defined as

$$\begin{array}{cc}\hfill R_1=& \frac{1}{n}log{M}_1\hfill \\ \hfill R_2=& \frac{1}{n}log{M}_1{M}_2.\hfill \end{array}$$

An $(n,R_1,R_2,D_1,D_2,ϵ)$-successive refinement code is a coding scheme with block length n and excess distortion probability $ϵ$ where rates are $(R_1,R_2)$ and target distortions are $(D_1,D_2)$. Since we have two decoders, the excess distortion probability is defined by $\mathrm{Pr}\left[d_i(X^n,{\widehat{X}}_i^n)>D_i \mathrm{for} \mathrm{some} i\right]$.

Definition 1.

A rate-distortion tuple $(R_1,R_2,D_1,D_2)$ is said to be achievable if there is a family of $(n,R_1^{\left(n\right)},R_2^{\left(n\right)}$, $D_1,D_2,{ϵ}^{\left(n\right)})$-successive refinement code where

$$\begin{array}{c}\underset{n\to \infty }{lim}{R}_i^{\left(n\right)}=R_i for all i,\hfill \\ \underset{n\to \infty }{lim}{ϵ}^{\left(n\right)}=0.\hfill \end{array}$$

For some special cases, both decoders can achieve the point-to-point optimum rates simultaneously.

Definition 2.

Let $R_i\left(D_i\right)$ denote the rate-distortion function of the i-th decoder for $i\in \{1,2\}$. If the rate-distortion tuple $(R_1\left(D_1\right),R_2\left(D_2\right),D_1,D_2)$ is achievable, then we say the source is successively refinable at $(D_1,D_2)$. If the source is successively refinable at $(D_1,D_2)$ for all $D_1,D_2$, then we say the source is successively refinable.

The following theorem provides a necessary and sufficient condition of successive refinable sources.

Theorem 1 ([7,8]).

A source is successively refinable at $(D_1,D_2)$ if and only if there exists a conditional distribution $p_{{\widehat{X}}_1,{\widehat{X}}_2|X}$ such that $X-{\widehat{X}}_2-{\widehat{X}}_1$ forms a Markov chain and

$$\begin{array}{cc}\hfill R_i\left(D_i\right)& =I(X;{\widehat{X}}_i)\hfill \\ \hfill \mathbb{E}\left[d_i(X,{\widehat{X}}_i)\right]& \le D_i\hfill \end{array}$$

for $i\in \{1,2\}$.

Note that the above results of successive refinability can easily be generalized to the case of k decoders.

2.2. Logarithmic Loss

Let $\mathcal{X}$ be a set of discrete source symbols ($\left|\mathcal{X}\right|<\infty$), and $\mathcal{M}\left(\mathcal{X}\right)$ be the set of probability measures on $\mathcal{X}$. Logarithmic loss $\ell :\mathcal{X}\times \mathcal{M}\left(\mathcal{X}\right)\to [0,\infty ]$ is defined by

$$\begin{array}{c}\hfill \ell (x,q)=log\frac{1}{q\left(x\right)}\end{array}$$

for $x\in \mathcal{X}$ and $q\in \mathcal{M}\left(\mathcal{X}\right)$. Logarithmic loss between n-tuples is defined by

$$\begin{array}{c}\hfill {\ell }_n(x^n,q^n)=\frac{1}{n}\sum _{i=1}^{n}log\frac{1}{q_i\left(x_i\right)},\end{array}$$

i.e., the symbol-by-symbol extension of the single letter loss.

Let $X^n$ be the discrete memoryless source with distribution $p_X$. Consider the lossy compression problem under logarithmic loss where the reconstruction alphabet is $\mathcal{M}\left(\mathcal{X}\right)$. The rate-distortion function is given by

$$\begin{array}{cc}\hfill R\left(D\right)=& \underset{p_{Q|X}:\mathbb{E}\left[\ell (X,Q)\right]\le D}{inf}I(X;Q)\hfill \\ \hfill =& H\left(X\right)-D.\hfill \end{array}$$

The following lemma provides a property of the rate-distortion function achieving conditional distribution.

Lemma 1.

The rate-distortion function achieving conditional distribution $p_{Q^{\star }|X}$ satisfies

$$\begin{array}{cc}\hfill p_{X|Q^{\star }}(·|q)=& q\hfill \end{array}$$

(1)

$$\begin{array}{cc}\hfill H\left(X\right|Q^{\star })=& D\hfill \end{array}$$

(2)

for $p_{Q^{\star }}$ almost every $q\in \mathcal{M}\left(\mathcal{X}\right)$. Conversely, if $p_{Q|X}$ satisfies (1) and (2), then it is a rate-distortion function achieving conditional distribution, i.e.,

$$\begin{array}{cc}\hfill I(X;Q)=& R\left(D\right)=H\left(X\right)-D\hfill \\ \hfill \mathbb{E}\left[\ell (X,Q)\right]=& D.\hfill \end{array}$$

The key idea is that we can replace Q by $p_{X|Q}(·|Q)$, and have lower rate and distortion, i.e.,

$$\begin{array}{cc}\hfill I(X;Q)\ge & I(X;p_{X|Q}(·|Q))\hfill \\ \hfill \mathbb{E}\left[\ell (X,Q)\right]\ge & \mathbb{E}\left[\ell (X,p_{X|Q}(·|Q)\right],\hfill \end{array}$$

which directly implies (1).

Interestingly, since the rate-distortion function in this case is a straight line, a simple time sharing scheme achieves the optimal rate-distortion trade-off. More precisely, the encoder losslessly compresses only the first $\frac{H\left(X\right)-D}{H\left(X\right)}$ fraction of the source sequence components. Then, the decoder perfectly recovers those losslessly compressed components and uses $p_X$ as its reconstruction for the remaining part. The resulting scheme obviously achieves distortion D with rate $H\left(X\right)-D$.

Furthermore, this simple scheme directly implies successive refinability of the source. For $D_1>D_2$, suppose the encoder losslessly compresses the first $\frac{H\left(X\right)-D_2}{H\left(X\right)}$ fraction of the source. Then, the first decoder can perfectly reconstruct $\frac{H\left(X\right)-D_1}{H\left(X\right)}$ fraction of the source with the message of rate $H\left(X\right)-D_1$ and distortion $D_1$ while the second decoder can achieve distortion $D_2$ with rate $H\left(X\right)-D_2$. Since both decoders can achieve the best rate-distortion pair, it follows that any discrete memoryless source under logarithmic loss is successively refinable.

We can formally prove successive refinability of discrete memoryless source under logarithmic loss using Theorem 1. I.e., by finding random probability mass functions $Q_1,Q_2\in \mathcal{M}\left(\mathcal{X}\right)$ that satisfy

$$\begin{array}{cc}\hfill I(X;Q_1)=& H\left(X\right)-D_1,\mathbb{E}\left[\ell (X,Q_1)\right]=D_1,\hfill \end{array}$$

(3)

$$\begin{array}{cc}\hfill I(X;Q_2)=& H\left(X\right)-D_2,\mathbb{E}\left[\ell (X,Q_2)\right]=D_2,\hfill \end{array}$$

(4)

where $X-Q_2-Q_1$ forms a Markov chain.

Let $e_x$ be a deterministic probability mass function (pmf) in $\mathcal{M}\left(\mathcal{X}\right)$ that has a unit mass at x. In other words,

$$\begin{array}{c}\hfill e_x\left(\tilde{x}\right)=\left\{\begin{array}{cc}1\hfill & \mathrm{if} \tilde{x}=x\hfill \\ 0\hfill & \mathrm{otherwise}.\hfill \end{array}\right.\end{array}$$

Then, consider random pmfs $Q_1,Q_2\in \{e_x:x\in \mathcal{X}\}\cup \left\{p_X\right\}$. Since the support of $Q_1$ and $Q_2$ is finite, we can define the following conditional pmfs.

$$\begin{array}{cc}\hfill p_{Q_2|X}\left(q_2\right|x)=& \left\{\begin{array}{cc}\frac{H\left(X\right)-D_2}{H\left(X\right)}\hfill & \mathrm{if} q_2=e_x\hfill \\ \frac{D_2}{H\left(X\right)}\hfill & \mathrm{if} q_2=p_X\hfill \\ 0\hfill & \mathrm{otherwise}\hfill \end{array}\right.\hfill \\ \hfill p_{Q_1|Q_2}\left(q_1\right|q_2)=& \left\{\begin{array}{cc}\frac{H\left(X\right)-D_1}{H\left(X\right)-D_2}\hfill & \mathrm{if} q_1=q_2=e_x\mathrm{for} \mathrm{some} x\hfill \\ \frac{D_1-D_2}{H\left(X\right)-D_2}\hfill & \mathrm{if} q_1=p_X \mathrm{and} q_2=e_x\mathrm{for} \mathrm{some} x\hfill \\ 1\hfill & \mathrm{if} q_1=q_2=p_X\hfill \\ 0\hfill & \mathrm{otherwise}.\hfill \end{array}\right.\hfill \end{array}$$

It is not hard to show that the above conditional pmfs satisfies (3) and (4).

3. Successive Refinability

Main Results

Consider the successive refinement problem with a discrete memoryless source as described in Section 2.1. Specifically, we are interested in the case where the first decoder is under logarithmic loss and the second decoder is under some arbitrary distortion measure $d(·,·)$. We only have a following benign assumption that if $d(x,{\widehat{x}}_1)=d(x,{\widehat{x}}_2)$ for all x, then ${\widehat{x}}_1={\widehat{x}}_2$. This is not a hard restriction since if ${\widehat{x}}_1$ and ${\widehat{x}}_2$ have the same distortion values for all x, then there is no reason to have both reconstruction symbols.

The following theorem shows that any discrete memoryless source is successive refinable as long as the weaker link is under logarithmic loss. This implies an universal property of logarithmic loss in the context of successive refinement.

Theorem 2.

Let the source be arbitrary discrete memoryless. Suppose the distortion criterion of the first decoder is logarithmic loss while that of the second decoder is an arbitrary distortion criterion $d:\mathcal{X}\times \widehat{\mathcal{X}}\to [0,\infty ]$. Then the source is successively refinable.

Proof. 

The source is successively refinable at $(D_1,D_2)$ if and only if there exists a $X-\widehat{X}-Q$ such that

$$\begin{array}{cc}\hfill I(X;Q)=& R_1\left(D_1\right),\mathbb{E}\left[\ell (X,Q)\right]\le D_1\hfill \\ \hfill I(X;\widehat{X})=& R_2\left(D_2\right),\mathbb{E}\left[d(X,\widehat{X})\right]\le D_2.\hfill \end{array}$$

Let $p_{{\widehat{X}}^{\star }|X}$ be the conditional distribution for the second decoder that achieves the informational rate-distortion function $R_2\left(D_2\right)$. i.e.,

$$\begin{array}{c}\hfill I(X;{\widehat{X}}^{\star })=R_2\left(D_2\right),\mathbb{E}\left[d(X,{\widehat{X}}^{\star })\right]=D_2.\end{array}$$

Since the weaker link is under logarithmic loss, we have $R_1\left(D_1\right)\le R_2\left(D_2\right)$. This implies that $H\left(X\right)-D_1\le H\left(X\right)-H(X|{\widehat{X}}^{\star })$. Thus, we can assume $H(X|{\widehat{X}}^{\star })\le D_1$ throughout the proof. For simplicity, we further have a benign assumption that there is no $\widehat{x}$ such that $p_X\left(x\right)=p_{X|{\widehat{X}}^{\star }}\left(x\right|\widehat{x})$ for all x. (See Remark 1 for the case where such $\widehat{x}$ exists.)

Without loss of generality, suppose $\widehat{\mathcal{X}}=\{0,1,\dots ,s-1\}$. Consider a random variable $Y\in \mathcal{Y}=\{0,1,\dots ,s\}$ with the following pmf for some $0\le ϵ\le 1$:

$$\begin{array}{c}\hfill p_Y\left(y\right)=\left\{\begin{array}{cc}(1-ϵ)p_{{\widehat{X}}^{\star }}\left(y\right)\hfill & \mathrm{if} y\le s-1\hfill \\ ϵ\hfill & \mathrm{if} y=s.\hfill \end{array}\right.\end{array}$$

The conditional distribution is given by

$$\begin{array}{c}\hfill p_{{\widehat{X}}^{\star }|Y}\left(\widehat{x}\right|y)=\left\{\begin{array}{cc}1\hfill & \mathrm{if} \widehat{x}=y\le s-1\hfill \\ 0\hfill & \mathrm{if} \widehat{x}\ne y\le s-1\hfill \\ p_{{\widehat{X}}^{\star }}\left(\widehat{x}\right)\hfill & \mathrm{if} y=s.\hfill \end{array}\right.\end{array}$$

The joint distribution of $X,{\widehat{X}}^{\star },Y$ is given by

$$\begin{array}{c}\hfill p_{X,{\widehat{X}}^{\star },Y}(x,\widehat{x},y)=p_{X,{\widehat{X}}^{\star }}(x,\widehat{x})p_{Y|{\widehat{X}}^{\star }}\left(y\right|\widehat{x}).\end{array}$$

It is clear that $H\left(X\right|Y)=H(X|{\widehat{X}}^{\star })$ if $ϵ=0$ and $H\left(X\right|Y)=H(X)$ if $ϵ=1$. Since $H(X|{\widehat{X}}^{\star })\le D_1$, there exists an $0\le ϵ\le 1$ such that $H\left(X\right|Y)=D_1$.

We are now ready to define the Markov chain. Let $Q=p_{X|Y}(·|Y)$ and $q^{\left(y\right)}=p_{X|Y}(·|y)$ for all $y\in \mathcal{Y}$. The following lemma implies that there is a one-to-one mapping between q and y.

Lemma 2.

If $p_{X|Y}\left(x\right|y_1)=p_{X|Y}\left(x\right|y_2)$ for all $x\in \mathcal{X}$, then $y_1=y_2$.

The proof of lemma is given in Appendix A. Since $Q=p_{X|Y}(·|Y)$ is a one-to-one mapping, we have

$$\begin{array}{cc}\hfill I(X;Q)=& I(X;Y)=H\left(X\right)-D_1=R_1\left(D_1\right).\hfill \end{array}$$

Also, we have

$$\begin{array}{cc}\hfill \mathbb{E}\left[\ell (X,Q)\right]=& \mathbb{E}\left[log\frac{1}{p_{X|Y}\left(X\right|Y)}\right]=H\left(X\right|Y)=D_1.\hfill \end{array}$$

Furthermore, $X-{\widehat{X}}^{\star }-Q$ forms a Markov chain since $X-{\widehat{X}}^{\star }-Y$ forms a Markov chain. This concludes the proof. □

The key idea of the theorem is that (1) is the only loose required condition for the rate-distortion function achieving conditional distribution. Thus, for any distortion criterion in the second stage, we are able to choose an appropriate distribution $p_{X,\widehat{X},Q}$ that satisfies both (1) and the condition for successive refinability.

Remark 1.

The assumption $p_{X|{\widehat{X}}^{\star }}(·|\widehat{x})\ne p_X(·)$ for all $\widehat{x}$ is not necessary. Appendix B shows another joint distribution $p_{X,{\widehat{X}}^{\star },Y}$ that satisfies conditions for successive refinability when the above assumption does not hold.

The distribution in the above proof is one simple example that has a single parameter ϵ, but we can always find other distributions that satisfy the condition for successive refinability. In the next section, we propose totally different distribution that achieves a Markov chain $X-{\widehat{X}}^{\star }-Y$ with $H\left(X\right|Y)=D_1$. This implies that the above proof does not rely on the assumption.

Remark 2.

In the proof, we used random variable Y to define $Q=p_{X|Y}(·|Y)$. On the other hand, if the joint distribution $p_{X,{\widehat{X}}^{\star },Q}$ satisfies conditions of successive refinability, there exists a random variable Y where $X-{\widehat{X}}^{\star }-Y$ forms a Markov chain and $Q=p_{X|Y}(·|Y)$. This is simply because we can set $Y=Q$, which implies $p_{X|Y}(·|Y)=p_{X|Q}(·|Q)=Q$.

Theorem 2 can be generalized to successive refinement problem with K intermediate decoders. Consider random variables $Y_k\in \mathcal{Y}$ for $1\le k\le K$ such that $X-{\widehat{X}}^{\star }-Y_K-\cdots -Y_1$ forms a Markov chain and the joint distribution of $X,{\widehat{X}}^{\star },Y_1,\dots ,Y_K$ is given by

$$\begin{array}{c}{p}_{X,{\widehat{X}}^{\star },Y_1,\dots ,Y_K}(x,\widehat{x},y_1,\dots ,y_K)\hfill \\ =p_{X,{\widehat{X}}^{\star }}(x,\widehat{x})p_{Y_1|{\widehat{X}}^{\star }}\left(y_1\right|\widehat{x})\prod _{k=1}^{K-1}{p}_{Y_{k+1}|Y_k}\left(y_{k+1}\right|y_k)\hfill \end{array}$$

where $H\left(X\right|Y_k)=D_k$. Similar to the proof of Theorem 2, we can show that $Q_k=p_{X|Y_k}(·|Y_k)$ for all $1\le k\le K$ satisfy the condition for successive refinability (where posterior distributions $p_{X|Y_k}(·|y_k)$ should be distinct for all $y_k\in \mathcal{Y}$ to guarantee one-to-one correspondence). Thus, we can conclude that any discrete memoryless source with K intermediate decoders is successively refinable as long as all the intermediate decoders are under logarithmic loss.

4. Toward Lossy Compression with Low Complexity

As we mentioned in Remark 1, the choice of joint distribution $p_{X,{\widehat{X}}^{\star },Q}$ in the proof of Theorem 2 is not unique. In this section, we propose another joint distribution $p_{X,{\widehat{X}}^{\star },Q}$ that satisfies the conditions for successive refinability. It naturally suggests a new lossy compression algorithm which we will discuss in Section 4.3.

4.1. Rate-Distortion Achieving Joint Distribution: Small $D_1$

Recall that $H(X|{\widehat{X}}^{\star })\le D_1$. We first consider the case where $D_1$ is not too large so that $D_1$ is close to $H(X|{\widehat{X}}^{\star })$. We will clarify this later. For simplicity, we further assume that $p_{{\widehat{X}}^{\star }}\left(0\right)\ge \cdots \ge p_{{\widehat{x}}^{\star }}(s-1)$. Consider a random variable $Z_{ϵ}^{\left(s\right)}\in \widehat{\mathcal{X}}$ with the following pmf for some $0\le ϵ\le (s-1){min}_{\widehat{x}}{p}_{{\widehat{X}}^{\star }}\left(\widehat{x}\right)$

$$\begin{array}{c}\hfill p_{Z_{ϵ}^{\left(s\right)}}\left(z\right)=\left\{\begin{array}{cc}1-ϵ\hfill & \mathrm{if} z=0\hfill \\ \frac{ϵ}{s-1}\hfill & \mathrm{if} 1\le z\le s-1.\hfill \end{array}\right.\end{array}$$

If it is clear from context, we simply use $Z\equiv Z_{ϵ}^{\left(s\right)}$ for the sake of notation. We further define a random variable Y that is independent to Z such that ${\widehat{X}}^{\star }=Y{\oplus }_{s}Z$, where ${\oplus }_s$ denotes a sum modulo s. This can be achieved by following pmf and conditional pmf.

$$\begin{array}{cc}\hfill p_Y\left(y\right)=& \frac{p_{{\widehat{X}}^{\star }}\left(y\right)-\frac{ϵ}{s-1}}{1-\frac{s}{s-1}ϵ}\hfill \\ \hfill p_{{\widehat{X}}^{\star }|Y}\left(\widehat{x}\right|y)=& \left\{\begin{array}{cc}1-ϵ\hfill & \mathrm{if} \widehat{x}=y\hfill \\ \frac{ϵ}{s-1}\hfill & \mathrm{if} \widehat{x}\ne y.\hfill \end{array}\right.\hfill \end{array}$$

(5)

If $ϵ=0$, we have $H\left(X\right|Y)=H(X|{\widehat{X}}^{\star })$. Also, it is clear that $H\left(X\right|Y)$ increases as $ϵ$ increases. Since we assume that $D_1$ is not too large, there exists $0\le ϵ\le (s-1)min{p}_{{\widehat{X}}^{\star }}\left(\widehat{x}\right)$ such that $H\left(X\right|Y)=D_1$. We will discuss about the case of general $D_1$ in Section 4.2. The joint distribution of $X,{\widehat{X}}^{\star },Y$ is given by

$$\begin{array}{c}\hfill p_{X,{\widehat{X}}^{\star },Y}(x,\widehat{x},y)=p_{X,{\widehat{X}}^{\star }}(x,\widehat{x})p_{Y|{\widehat{X}}^{\star }}\left(y\right|\widehat{x}).\end{array}$$

We are now ready to define the Markov chain. Let $Q=p_{X|Y}(·|Y)$ and $q^{\left(y\right)}=p_{X|Y}(·|y)$ for all $y\in \mathcal{Y}$ where $\mathcal{Y}=\widehat{\mathcal{X}}=\{0,1,\dots s-1\}$. For simplicity, we assume that $p_{X|Y}(·|y_1)$ and $p_{X|Y}(·|y_2)$ are not the same for all $y_1\ne y_2$. Since $Q=p_{X|Y}(·|Y)$ is a one-to-one mapping, we have

$$\begin{array}{cc}\hfill I(X;Q)=& I(X;Y)=H\left(X\right)-D_1=R_1\left(D_1\right).\hfill \end{array}$$

Also, we have

$$\begin{array}{cc}\hfill \mathbb{E}\left[\ell (X,Q)\right]=& \mathbb{E}\left[log\frac{1}{p_{X|Y}\left(X\right|Y)}\right]=H\left(X\right|Y)=D_1.\hfill \end{array}$$

Furthermore, $X-{\widehat{X}}^{\star }-Q$ forms a Markov chain since $X-{\widehat{X}}^{\star }-Y$ forms a Markov chain. Thus, the above construction of joint distribution $p_{X,{\widehat{X}}^{\star },Q}$ satisfies the conditions for successive refinability.

4.2. Rate-Distortion Achieving Joint Distribution: General $D_1$

The joint distribution in the previous section only works for small $D_1$. It is because $ϵ$ has a natural upper-bound from (5) which is $ϵ\le (s-1)min{p}_{{\widehat{X}}^{\star }}\left(\widehat{x}\right)$. In this section, we generalize the proof in the previous section for general $D_1$. The key observation is that if we pick the maximum $ϵ=(s-1)min{p}_{{\widehat{X}}^{\star }}\left(\widehat{x}\right)$, then $p_Y(s-1)=0$. This implies that we can focus on the smaller set of reconstruction alphabet $\mathcal{Y}=\{0,1,\dots s-2\}$.

Let $U_s={\widehat{X}}^{\star }$, and define random variables $\{U_k:1\le k\le s-1\}$ recursively. More precisely, we define the random variable $U_{k-1}$ based on $U_k$ for $2\le k\le s$.

$$\begin{array}{cc}\hfill U_k=& U_{k-1}{\oplus }_k{Z}_{{ϵ}_k}^{\left(k\right)}\hfill \\ \hfill p_{Z_{{ϵ}_k}^{\left(k\right)}}\left(z\right)=& \left\{\begin{array}{cc}1-{ϵ}_k\hfill & \mathrm{if} z=0\hfill \\ \frac{{ϵ}_k}{k-1}\hfill & \mathrm{if} 1\le z\le k-1\hfill \end{array}\right.\hfill \end{array}$$

where

$$\begin{array}{c}\hfill {ϵ}_k=(k-1)\underset{u}{min}{p}_{U_k}\left(u\right).\end{array}$$

Similar to the definition of Y, we assume $U_{k-1}$ and $Z_{{ϵ}_k}^{\left(k\right)}$ are independent, and ${\oplus }_k$ denotes modulo k sum. Each time step, the alphabet size of $U_k$ decreases by one. Thus, we have $0\le U_k\le k-1$, and therefore $U_1=0$ with probability 1. Furthermore, we have

$$\begin{array}{c}\hfill H\left(X\right|U_s)\le H\left(X\right|U_{s-1})\le \cdots \le H\left(X\right|U_1)=H\left(X\right).\end{array}$$

For $H(X|{\widehat{X}}^{\star })\le D_1<H\left(x\right)$, there exists k such that $H\left(X\right|U_k)>D_1\ge H\left(X\right|U_{k-1})$. Thus, there exists Y that satisfies $H\left(X\right|Y)=D_1$ and $U_k=Y{\oplus }_k{Z}_{ϵ}^{\left(k\right)}$ for some $0\le ϵ\le {ϵ}_k$. This implies that

$$\begin{array}{c}\hfill {\widehat{X}}^{\star }=Z_{{ϵ}_s}^{\left(s\right)}{\oplus }_s\left[Z_{{ϵ}_{s-1}}^{(s-1)}{\oplus }_{s-1}\cdots \left(Z_{{ϵ}_{k+1}}^{(k+1)}{\oplus }_{k+1}\left(Z_{ϵ}^{\left(k\right)}{\oplus }_{k}Y\right)\right)\right].\end{array}$$

Similar to the previous section, we assume that $p_{X|Y}(·|y_1)\ne p_{X|Y}(·|y_2)$ if $y_1\ne y_2$. Then, we can set $Q=p_{X|Y}(·|Y)$ which satisfies the conditions for successive refinability.

4.3. Iterative Lossy Compression Algorithm

The joint distribution from the previous section naturally suggests a simple successive refinement scheme. Consider the lossy compression problem where the source is i.i.d. $\sim p_X$ and the distortion measure is $d:\mathcal{X}\times \widehat{\mathcal{X}}\to [0,\infty )$. Let D be the target distortion, and $R>R\left(D\right)$ be the rate of the scheme where $R\left(D\right)$ is the rate-distortion function. Let $p_{X,{\widehat{X}}^{\star }}$ be the rate-distortion achieving distribution.

For block length n, we propose a new lossy compression scheme that mimics successive refinement with $s-1$ decoders. Similar to the previous section, let ${ϵ}_k=(k-1){min}_u{p}_{U_k}\left(u\right)$ and

$$\begin{array}{cc}\hfill {\widehat{X}}^{\star }=U_s=& U_{s-1}{\oplus }_s{Z}_{{ϵ}_s}^{\left(s\right)}\hfill \\ \hfill U_{s-1}=& U_{s-2}{\oplus }_{s-1}{Z}_{{ϵ}_{s-1}}^{(s-1)}\hfill \\ \hfill ⋮\\ \hfill U_2=& U_1{\oplus }_2{Z}_{{ϵ}_2}^{\left(2\right)}.\hfill \end{array}$$

We further let $R_{k-1}>I(X;U_k)-I(X;U_{k-1})$ for $2\le k\le s$ that satisfy $R={\sum }_{k=2}^s{R}_{k-1}$. Now, we are ready to describe our coding scheme. Generate a sub-codebook ${\mathcal{C}}_1=\{z^n(1,m):1\le m\le e^{R_1}\}$ where each sequence is generated according to $Z^n\sim$ i.i.d. $p_{Z_{{ϵ}_2}^{\left(2\right)}}$ for all m. Similarly, generate sub-codebooks ${\mathcal{C}}_k=\{z^n(k,m):1\le m\le e^{n{R}_k}\}$ for $2\le k\le s-1$ where each sequence is generated according to $Z^n\sim$ i.i.d. $p_{Z_{{ϵ}_{k+1}}^{(k+1)}}$ for all m.

Upon observing $x^n\in {\mathcal{X}}^n$, the encoder finds $m_1\in {\mathcal{C}}_1$ that minimizes $d_1(x^n,z^n(1,m_1))$ where the distortion measure $d_1(·,·)$ is defined as follows.

$$\begin{array}{c}\hfill d_1(x^n,z^n)=\frac{1}{n}\sum _{i=1}^{n}log\frac{1}{p_{X|U_2}\left(x_i\right|z_i)}.\end{array}$$

Note that $d_1(x,z)$ is simply the logarithmic loss between x and $p_{X|U_2}(·|z)$.

Similarly, for $2\le k\le s-1$, the encoder iteratively finds $m_k\in {\mathcal{C}}_k$ that minimizes $d_k\left(x^n,\left[[z^n(1,m_1){\oplus }_3\cdots {\oplus }_k{z}^n(k-1,m_{k-1})]{\oplus }_{k+1}{z}^n(k,m_k)\right]\right)$ where

$$\begin{array}{c}\hfill d_k(x^n,z^n)=\frac{1}{n}\sum _{i=1}^{n}log\frac{1}{p_{X|U_{k+1}}\left(x_i\right|z_i)}.\end{array}$$

Upon receiving $m_1,m_2,\dots ,m_{s-1}$, the decoder reconstructs

$$\begin{array}{c}\hfill {\widehat{x}}^n=\left[\left[z^n(1,m_1){\oplus }_3{z}^n(2,m_2)\right]\oplus \cdots {\oplus }_s{z}^n(s-1,m_{s-1})\right].\end{array}$$

Suppose $R_1\approx R_2\approx \cdots \approx R_{s-1}\approx \frac{R}{s-1}$, and $L=s-1$. Similar to [12,14], this scheme has two main advantages compare to naive random coding scheme. First, the number of codewords in the proposed scheme is $L·e^{nR/L}$, while the naive scheme requires $e^{nR}$ codewords. Also, in each iteration, the encoder finds the best codeword among $e^{nR/L}$ sub-codewords. Thus, the overall complexity is $L·e^{nR/L}$ as well. On the other hand, the naive scheme requires $e^{nR}$ complexity.

Remark 3.

The proposed scheme constructs ${\widehat{X}}^n$ from binary sequences. The reconstruction after each stage can be viewed as

$$\begin{array}{c}\hfill u_k^n(m_1,\dots m_{k-1})=\left[[z^n(1,m_1){\oplus }_3\cdots ]{\oplus }_k{z}^n(k-1,m_{k-1})\right]\end{array}$$

where $0\le u_k\le k-1$. Thus, the decoder starts from binary sequence $u_2^n\left(m_1\right)$, and the alphabet size increases by 1 at each iteration. After $(s-1)$-th iteration, it reaches the final reconstruction ${\widehat{X}}^n$ where the size of alphabet is s.

5. Conclusions

To conclude our discussion, we summarize our main contributions. In the context of successive refinement problem, we showed another universal property of logarithmic loss that any discrete memoryless source is successively refinable as long as the intermediate decoders operate under logarithmic loss. We applied the result to the point-to-point lossy compression problem and proposed a lossy compression scheme with lower complexity.