Rate-Distortion Region of a Gray–Wyner Model with Side Information

Abstract

In this work, we establish a full single-letter characterization of the rate-distortion region of an instance of the Gray–Wyner model with side information at the decoders. Specifically, in this model, an encoder observes a pair of memoryless, arbitrarily correlated, sources $(S_1^n,S_2^n)$ and communicates with two receivers over an error-free rate-limited link of capacity $R_0$, as well as error-free rate-limited individual links of capacities $R_1$ to the first receiver and $R_2$ to the second receiver. Both receivers reproduce the source component $S_2^n$ losslessly; and Receiver 1 also reproduces the source component $S_1^n$ lossily, to within some prescribed fidelity level $D_1$. In addition, Receiver 1 and Receiver 2 are equipped, respectively, with memoryless side information sequences $Y_1^n$ and $Y_2^n$. Important in this setup, the side information sequences are arbitrarily correlated among them, and with the source pair $(S_1^n,S_2^n)$; and are not assumed to exhibit any particular ordering. Furthermore, by specializing the main result to two Heegard–Berger models with successive refinement and scalable coding, we shed light on the roles of the common and private descriptions that the encoder should produce and the role of each of the common and private links. We develop intuitions by analyzing the developed single-letter rate-distortion regions of these models, and discuss some insightful binary examples.

1. Introduction

The Gray–Wyner source coding problem was originally formulated, and solved, by Gray and Wyner in [1]. In their original setting, a pair of arbitrarily correlated memoryless sources $(S_1^n,S_2^n)$ is to be encoded and transmitted to two receivers that are connected to the encoder each through a common error-free rate-limited link as well as a private error-free rate-limited link. Because the channels are rate-limited, the encoder produces a compressed bit string $W_0$ of rate $R_0$ that it transmits over the common link, and two compressed bit strings, $W_1$ of rate $R_1$ and $W_2$ of rate $R_2$, that transmits over the private links each to their respective receiver. The first receiver uses the bit strings $W_0$ and $W_1$ to reproduce an estimate ${\widehat{S}}_1^n$ of the source component $S_1^n$ to within some prescribed distortion level $D_1$, for some distortion measure $d_1(·,·)$. Similarly, the second receiver uses the bit strings $W_0$ and $W_2$ to reproduce an estimate ${\widehat{S}}_2^n$ of the source component $S_2^n$ to within some prescribed distortion level $D_2$, for some different distortion measure $d_2(·,·)$. In [1], Gray and Wyner characterized the optimal achievable rate triples $(R_0,R_1,R_2)$ and distortion pairs $(D_1,D_2)$.

Figure 1 shows a generalization of the original Gray–Wyner model in which the receivers also observe correlated memoryless side information sequences, $Y_1^n$ at Receiver 1 and $Y_2^n$ at Receiver 2. Some special cases of the Gray–Wyner model with side information of Figure 1 have been solved (see the Section 1.2 below). However, in its most general form, i.e., when the side information sequences are arbitrarily correlated among them and with the sources, this problem has so-far eluded single-letter characterization of the optimal rate-distortion region. Indeed, the Gray–Wyner problem with side information subsumes the well known Heegard–Berger problem [2], obtained by setting $R_1=R_2=0$ in Figure 1, which remains, to date, an open problem.

In this paper, we study an instance of the Gray–Wyner model with side information of Figure 1 in which the reconstruction sets are degraded, meaning, both receivers reproduce the source component $S_2^n$ losslessly and Receiver 1 wants also to reproduce the source component $S_1^n$ lossily, to within some prescribed distortion level $D_1$. It is important to note that, while the reconstruction sets are nested, and so degraded, no specific ordering is imposed on the side information sequences, which then can be arbitrarily correlated among them and with the sources $(S_1^n,S_2^n)$.

As in the Gray–Wyner original coding scheme, the encoder produces a common description of the sources pair $(S_1^n,S_2^n)$ that is intended to be recovered by both receivers, as well as individual or private descriptions of $(S_1^n,S_2^n)$ that are destined to be recovered each by a distinct receiver. Because the side information sequences do not exhibit any specific ordering, the choice of the information that each description should carry, and, the links over which each is transmitted to its intended receiver, are challenging questions that we answer in this work.

To build the understanding of the role of each of the links and of the descriptions in the optimal coding scheme for the setting of Figure 2, we will investigate as well two important underlying problems, which are Heegard–Berger type models with refinement links as shown in Figure 3. In both models, only one of the two refinement individual links has non-zero rate.

In the model of Figure 3a, the receiver that accesses the additional rate-limited link (i.e., Receiver 1) is also required to reproduce a lossy estimate of the source component $S_1^n$, in addition to the source component $S_2^n$ which is to be reproduced losslessly by both receivers. We will refer to this model as a “Heegard–Berger problem with successive refinement”. Reminiscent of successive refinement source coding, this model may be appropriate to model applications in which descriptions of only some components (e.g., $S_2^n$) of the source suffices at the first use of the data; and descriptions of the remaining components (e.g., $S_1^n$) are needed only at a later stage.

The model of Figure 3b has the individual rate-limited link connected to the receiver that is required to reproduce only the source component $S_2^n$. We will refer to this model as a “Heegard–Berger problem with scalable coding”, reusing a term that was introduced in [3] for a similar scenario, and in reference to that user 1 may have such a “good quality” side information that only a minimal amount of information from the encoder suffices, thus, so as not to constrain the communication by user 2 with the lower quality side information, an additional rate limited link $R_2$ is added to balance the decoding capabilities of both users.

1.1. Main Contributions

The main result of this paper is a single-letter characterization of the optimal rate-distortion region of the Gray–Wyner model with side information and degraded reconstruction sets of Figure 2. To this end, we derive a converse proof that is tailored specifically for the model with degraded reconstruction sets that we study here. For the proof of the direct part, we develop a coding scheme that is very similar to one developed in the context of coding for broadcast channels with feedback in [4], but with an appropriate choice of the variables which we specify here. The specification of the main result to the Heegard–Berger models with successive refinement and scalable coding of Figure 3 sheds light on the roles of the common and private descriptions and what they should carry optimally. We develop intuitions by analyzing the established single-letter optimal rate-distortion regions of these two models, and illustrate our discussion through some binary examples.

1.2. Related Works

In [4], Shayevitz and Wigger study a two-receiver discrete memoryless broadcast channel with feedback. They develop an efficient coding scheme which treats the feedback signal as a source that has to be conveyed lossily to the receivers in order to refine their messages’ estimates, through a block Markov coding scheme. In doing so, the users’ channel outputs are regarded as side information sequences; thus, the scheme clearly connects with the Gray–Wyner model with side information of Figure 1—as is also clearly explicit in [4]. The Gray–Wyner model with side information for which Shayevitz and Wigger’s develop a (source) coding scheme, as part of their study of the broadcast channel with feedback, assumes general, possibly distinct, distortion measures at the receivers (i.e., not necessarily nested) and side information sequences that are arbitrarily correlated among them and with the source. In this paper, we show that, when specialized to the model with degraded reconstruction sets of Figure 2 that we study here, Shayevitz and Wigger’s coding scheme for the Gray–Wyner model with side information of [4] yields a rate-distortion region that meets the converse result that we here establish, thus is optimal.

The Gray–Wyner model with side information generalizes another long standing open source coding problem, the famous Heegard–Berger problem [2]. Full single-letter characterization of the optimal rate-distortion function of the Heegard–Berger problem is known only in few specific cases, the most important of which are the cases of : (i) stochastically degraded side information sequences [2] (see also [5]); (ii) Sgarro’s result [6] on the corresponding lossless problem; (iii) Gaussian sources with quadratic distortion measure [3,7]; (iv) some instances of conditionally less-noisy side information sequences [8]; and (v) the recently solved HB model with general side information sequences and degraded reconstruction sets [9], i.e., the model of Figure 2 with $R_1=R_2=0$— in the lossless case, a few other optimal results were shown, such as for the so-called complementary delivery [10]. A lower bound for general instances of the rate distortion problem with side information at multiple decoders, which is inspired by a linear-programming lower bound for index coding, has been developed recently by Unal and Wagner in [11].

Successive refinement of information was investigated by Equitz et al. in [12], wherein the description of the source is successively refined to a collection of receivers which are required to reconstruct the source with increasing quality levels. Extensions of successive refinement to cases in which the receivers observe some side information sequences was first investigated by Steinberg et al. in [13] who establish the optimal rate-distortion region under the assumption that the receiver that observes the refinement link, say receiver 1, observes also a better side information sequence than the opposite user, i.e., the Markov chain $S\sout{○}{Y}_1\sout{○}{Y}_2$ holds. Tian et al. give in [7] an equivalent formulation of the result of [13] and extend it to the N-stage successive refinement setting. In [3], Tian et al. investigate another setting, coined as “side information scalable coding”, in which it is rather the receiver that accesses the refinement link, say receiver 2, which observes the less good side information sequence, i.e., $S\sout{○}{Y}_1\sout{○}{Y}_2$. Balancing refinement quality and side information asymmetry for such a side-information scalable source coding problem allows authors in [3] to derive the rate-distortion region in the degraded side information case. The previous results on successive refinement in the presence of side information, which were generalized by Timo et al. in [14], all assume, however, a specific structure in the side information sequences.

1.3. Outline

An outline of the remainder of this paper is as follows. Section 2 describes formally the Gray–Wyner model with side information and degraded reconstruction sets of Figure 2 that we study in this paper. Section 3 contains the main result of this paper, a full single-letter characterization of the rate-distortion region of the model of Figure 2, together with some useful discussions and connections. A formal proof of the direct and converse parts of this result appear in Section 6. In Section 4 and Section 5, we specialize the result, respectively, to the Heegard–Berger model with successive refinement of Figure 3a and the Heegard–Berger model with scalable coding of Figure 3b. These sections also contain insightful discussions illustrated by some binary examples.

Notation

Throughout the paper, we use the following notations. The term pmf stands for probability mass function. Upper case letters are used to denote random variables, e.g., X; lower case letters are used to denote realizations of random variables, e.g., x; and calligraphic letters designate alphabets, i.e., $\mathcal{X}$. Vectors of length n are denoted by $X^n=(X_1,\cdots ,X_n)$, and $X_i^j$ is used to denote the sequence $(X_i,\cdots ,X_j)$, whereas $X_{<i>}\triangleq (X_1,\cdots ,X_{i-1},X_{i+1},\cdots ,X_n)$. The probability distribution of a random variable X is denoted by $P_X(x)\triangleq \mathbb{P}(X=x)$. Sometimes, for convenience, we write it as $P_X$. We use the notation $\mathbb{E}(X)$ to denote the expectation of a random variable X. A probability distribution of a random variable Y given X is denoted by $P_{Y|X}$. The set of probability distributions defined on an alphabet $\mathcal{X}$ is denoted by $\mathcal{P}(\mathcal{X})$. The cardinality of a set $\mathcal{X}$ is denoted by $\parallel \mathcal{X}\parallel$. For random variables X, Y and Z, the notation $X\sout{○}Y\sout{○}Z$ indicates that X, Y and Z, in this order, form a Markov Chain, i.e., $P_{XYZ}(x,y,z)=P_Y(y)P_{X|Y}(x|y)P_{Z|Y}(z|y)$. The set ${\mathcal{T}}_{[X]}^{(n)}$ denotes the set of sequences strongly typical with respect to the probability distribution $P_X$ and the set ${\mathcal{T}}_{[X|y^n]}^{(n)}$ denotes the set of sequences $x^n$ jointly typical with $y^n$ with respect to the joint pmf $P_{XY}$. Throughout this paper, we use $h_2(\alpha )$ to denote the entropy of a Bernoulli $(\alpha )$ random variable, i.e., $h_2(\alpha )=-\alpha log(\alpha )-(1-\alpha )log(1-\alpha )$. In addition, the indicator function is denoted by $\mathrm{𝟙}(·)$. For real-valued scalars a and b, with $a\le b$, the notation $[a,b]$ means the set of real numbers comprised between a and b. For integers $i\le j$, $[i:j]$ denotes the set of integers comprised between i and j, i.e., $[i:j]=\{i,i+1,\cdots ,j\}$. Finally, throughout the paper, logarithms are taken to base 2.

2. Problem Setup and Formal Definitions

Consider the Gray–Wyner source coding model with side information and degraded reconstruction sets shown in Figure 2. Let $({\mathcal{S}}_1\times {\mathcal{S}}_2\times {\mathcal{Y}}_1\times {\mathcal{Y}}_2,P_{S_1,S_2,Y_1,Y_2})$ be a discrete memoryless vector source with generic variables $S_1$, $S_2$, $Y_1$ and $Y_2$. In addition, let ${\widehat{\mathcal{S}}}_1$ be a reconstruction alphabet and, $d_1$ a distortion measure defined as:

$$\begin{array}{ccc}\hfill d_1:{\mathcal{S}}_1\times {\widehat{\mathcal{S}}}_1& \to & {\mathbb{R}}_{+}\hfill \\ \hfill (s_1,{\widehat{s}}_1)& \to & d_1(s_1,{\widehat{s}}_1).\hfill \end{array}$$

(1)

Definition 1.

An $(n,M_{0,n},M_{1,n},M_{2,n},D_1)$ code for the Gray–Wyner source coding model with side information and degraded reconstruction sets of Figure 2 consists of:

-

Three sets of messages ${\mathcal{W}}_0\triangleq [1:M_{0,n}]$, ${\mathcal{W}}_1\triangleq [1:M_{1,n}]$, and ${\mathcal{W}}_2\triangleq [1:M_{2,n}]$.

-

Three encoding functions, $f_0$, $f_1$ and $f_2$ defined, for $j\in \{0,1,2\}$ as

$$\begin{array}{ccc}\hfill f_j:{\mathcal{S}}_1^n\times {\mathcal{S}}_2^n& \mapsto & {\mathcal{W}}_j\hfill \\ \hfill (S_1^n,S_2^n)& \mapsto & W_j=f_j(S_1^n,S_2^n).\hfill \end{array}$$

(2)

-

Two decoding functions $g_1$ and $g_2$, one at each user:

$$\begin{array}{ccc}\hfill g_1:{\mathcal{W}}_0\times {\mathcal{W}}_1\times {\mathcal{Y}}_1^n& \mapsto & {\widehat{\mathcal{S}}}_2^n\times {\widehat{\mathcal{S}}}_1^n\hfill \\ \hfill (W_0,W_1,Y_1^n)& \mapsto & ({\widehat{S}}_{2,1}^n,{\widehat{S}}_1^n)=g_1(W_0,W_1,Y_1^n),\hfill \end{array}$$

(3)

and

$$\begin{array}{ccc}\hfill g_2:{\mathcal{W}}_0\times {\mathcal{W}}_2\times {\mathcal{Y}}_2^n& \mapsto & {\widehat{\mathcal{S}}}_2^n\hfill \\ \hfill (W_0,W_2,Y_2^n)& \mapsto & {\widehat{S}}_{2,2}^n=g_2(W_0,W_2,Y_2^n).\hfill \end{array}$$

(4)

The expected distortion of this code is given by

$$\mathbb{E}\left(d_1^{(n)}(S_1^n,{\widehat{S}}_1^n)\right)\triangleq \mathbb{E}{\displaystyle \frac{1}{n}}{\displaystyle \sum _{i=1}^n}{d}_1(S_{1,i},{\widehat{S}}_{1,i}).$$

(5)

The probability of error is defined as

$$P_e^{(n)}\triangleq \mathbb{P}\left({\widehat{S}}_{2,1}^n\ne S_2^n\mathit{or}{\widehat{S}}_{2,2}^n\ne S_2^n\right).$$

(6)

Definition 2.

A rate triple $(R_0,R_1,R_2)$ is said to be $D_1$-achievable for the Gray–Wyner source coding model with side information and degraded reconstruction sets of Figure 2 if there exists a sequence of $(n,M_{0,n},M_{1,n},M_{2,n},D_1)$ codes such that:

$$\begin{array}{ccc}\hfill \underset{n\to \infty }{lim\; sup}{P}_e^{(n)}& =& 0,\hfill \end{array}$$

(7)

$$\begin{array}{ccc}\hfill \underset{n\to \infty }{lim\; sup}\mathbb{E}\left(d_1^{(n)}(S_1^n,{\widehat{S}}_1^n)\right)& \le & D_1,\hfill \end{array}$$

(8)

$$\begin{array}{ccc}\hfill \underset{n\to \infty }{lim\; sup}{\displaystyle \frac{1}{n}}{log}_2(M_{j,n})& \le & R_j\mathit{for}j\in \{0,1,2\}.\hfill \end{array}$$

(9)

The rate-distortion region $\mathcal{RD}$ of this problem is defined as the union of all rate-distortion quadruples $(R_0,R_1,R_2,D_1)$ such that $(R_0,R_1,R_2)$ is $D_1$-achievable, i.e,

$$\mathcal{RD}\triangleq \cup \left\{(R_0,R_1,R_2,D_1):(R_0,R_1,R_2)\mathit{is}{D}_1-\mathit{achievable}\right\}.$$

(10)

As we already mentioned, we shall also study the special case Heegard–Berger type models shown in Figure 3. The formal definitions for these models are similar to the above, and we omit them here for brevity.

3. Gray–Wyner Model with Side Information and Degraded Reconstruction Sets

In the following, we establish the main result of this work, i.e., the single-letter characterization of the optimal rate-distortion region $\mathcal{RD}$ of the Gray–Wyner model with side information and degraded reconstructions sets shown in Figure 2. We then describe how the result subsumes and generalizes existing rate-distortion regions for this setting under different assumptions.

Theorem 1.

The rate-distortion region $\mathcal{RD}$ of the Gray–Wyner problem with side information and degraded reconstruction set of Figure 2 is given by the sets of all rate-distortion quadruples $(R_0,R_1,R_2,D_1)$ satisfying:

$$\begin{array}{ccc}\hfill R_0+R_1& \ge & H(S_2|Y_1)+I(U_0{U}_1;S_1|S_2{Y}_1)\hfill \end{array}$$

(11a)

$$\begin{array}{ccc}\hfill R_0+R_2& \ge & H(S_2|Y_2)+I(U_0;S_1|S_2{Y}_2)\hfill \end{array}$$

(11b)

$$\begin{array}{ccc}\hfill R_0+R_1+R_2& \ge & H(S_2|Y_2)+I(U_0;S_1|S_2{Y}_2)+I(U_1;S_1|U_0{S}_2{Y}_1)\hfill \end{array}$$

(11c)

for some product pmf $P_{U_0{U}_1{S}_1{S}_2{Y}_1{Y}_2}$, such that:

(1) 

The following Markov chain is valid:

$$(Y_1,Y_2)\sout{○}(S_1,S_2)\sout{○}(U_0,U_1)$$

(12)

(2) 

There exists a function $\varphi :{\mathcal{Y}}_1\times {\mathcal{U}}_0\times {\mathcal{U}}_1\times {\mathcal{S}}_2\to {\widehat{\mathcal{S}}}_1$ such that:

$$\mathbb{E}{d}_1(S_1,{\widehat{S}}_1)\le D_1.$$

(13)

Proof. 

The detailed proof of the direct part and the converse part of this theorem appear in Section 6.

The proof of converse, which is the most challenging part, uses appropriate combinations of bounding techniques for the transmitted rates based on the system model assumptions and Fano’s inequality, a series of analytic bounds based on the underlying Markov chains, and most importantly, a proper use of Csiszár–Körner sum identity in order to derive single letter bounds.

As for the proof of achievability, it combines the optimal coding scheme of the Heegard–Berger problem with degraded reconstruction sets [9] and the double-binning based scheme of Shayevitz and Wigger (Theorem 2, [4]) for the Gray–Wyner problem with side information, and is outlined in the following.

The encoder produces a common description of $(S_1^n,S_2^n)$ that is intended to be recovered by both receivers, and an individual description that is intended to be recovered only by Receiver 1. The common description is chosen as $V_0^n=(U_0^n,S_2^n)$ and is thus designed to describe all of $S_2^n$, which both receivers are required to reproduce lossessly, but also all or part of $S_1^n$, depending on the desired distortion level $D_1$. Since we make no assumptions on the side information sequences, this is meant to account for possibly unbalanced side information pairs $(Y_1^n,Y_2^n)$, in a manner that is similar to [9]. The message that carries the common description is obtained at the encoder through the technique of double-binning of Tian and Diggavi in [3], used also by Shayevitz and Wigger (Theorem 2, [4]) for a Gray–Wyner model with side information. In particular, similar to the coding scheme of (Theorem 2, [4]), the double-binning is performed in two ways, one that is tailored for Receiver 1 and one that is tailored for Receiver 2.

More specifically, the codebook of the common description is composed of codewords $v_0^n$ that are drawn randomly and independently according to the product law of $P_{V_0}$; and is partitioned uniformly into $2^{n{\tilde{R}}_{0,0}}$ superbins, indexed with ${\tilde{w}}_{0,0}\in [1:2^{n{\tilde{R}}_{0,0}}]$. The codewords of each superbin of this codebook are partitioned in two distinct ways. In the first partition, they are assigned randomly and independently to $2^{n{\tilde{R}}_{0,1}}$ subbins indexed with ${\tilde{w}}_{0,1}\in [1:2^{n{\tilde{R}}_{0,1}}]$, according to a uniform pmf over $[1:2^{n{\tilde{R}}_{0,1}}]$. Similarly, in the second partition, they are assigned randomly and independently to $2^{n{\tilde{R}}_{0,2}}$ subbins indexed with ${\tilde{w}}_{0,2}\in [1:2^{n{\tilde{R}}_{0,2}}]$, according to a uniform pmf over $[1:2^{n{\tilde{R}}_{0,2}}]$. The codebook of the private description is composed of codewords $u_1^n$ that are drawn randomly and independently according to the product law of $P_{U_1|V_0}$. This codebook is partitioned similarly uniformly into $2^{n{\tilde{R}}_{1,0}}$ superbins indexed with ${\tilde{w}}_{1,0}\in [1:2^{n{\tilde{R}}_{1,0}}]$, each containing $2^{n{\tilde{R}}_{1,1}}$ subbins indexed with ${\tilde{w}}_{1,1}\in [1:2^{n{\tilde{R}}_{1,1}}]$ codewords $u_1^n$.

Upon observing a typical pair $(S_1^n,S_2^n)=(s_1^n,s_2^n)$, the encoder finds a pair of codewords $(v_0^n,u_1^n)$ that is jointly typical with $(s_1^n,s_2^n)$. Let ${\tilde{w}}_{0,0}$, ${\tilde{w}}_{0,1}$ and ${\tilde{w}}_{0,2}$ denote respectively the indices of the superbin, subbin of the first partition and subbin of the second partition of the codebook of the common description, in which lies the found $v_0^n$. Similarly, let ${\tilde{w}}_{1,0}$ and ${\tilde{w}}_{1,1}$ denote respectively the indices of the superbin and subbin of the codebook of the individual description in which lies the found $u_1^n$. The encoder sets the common message $W_0$ as $W_0=({\tilde{w}}_{0,0},{\tilde{w}}_{1,0})$ and sends it over the error-free rate-limited common link of capacity $R_0$. In addition, it sets the individual message $W_1$ as $W_1=({\tilde{w}}_{0,1},{\tilde{w}}_{1,1})$ and sends it the error-free rate-limited link to Receiver 1 of capacity $R_1$; and the individual message $W_2$ as $W_2={\tilde{w}}_{0,2}$ and sends it the error-free rate-limited link to Receiver 2 of capacity $R_2$. For the decoding, Receiver 2 utilizes the second partition of the codebook of the common description; and looks in the subbin of index ${\tilde{w}}_{0,2}$ of the superbin of index ${\tilde{w}}_{0,0}$ for a unique $v_0^n$ that is jointly typical with its side information $y_2^n$. Receiver 1 decodes $v_0^n$ similarly, utilizing the first partition of the codebook of the common description and its side information $y_1^n$. It also utilizes the codebook of the individual description; and looks in the subbin of index ${\tilde{w}}_{1,1}$ of the superbin of index ${\tilde{w}}_{1,1}$ for a unique $u_1^n$ that is jointly typical with the pair $(y_1^n,v_0^n)$. In the formal proof in Section IV, we argue that with an appropropriate choice of the communication rates ${\tilde{R}}_{0,0}$, ${\tilde{R}}_{0,1}$, ${\tilde{R}}_{0,2}$, ${\tilde{R}}_{1,0}$ and ${\tilde{R}}_{1,1}$, as well as the sizes of the subbins, this scheme achieves the rate-distortion region of Theorem 1. ☐

A few remarks that connect Theorem 1 to known results on related models are in order.

Remark 1.

The setting of Figure 1 generalizes two important settings which are the Gray–Wyner problem, through the presence of side-information sequences $Y_1^n$ and $Y_2^n$, and the Heegard–Berger problem, through the presence of private links of rates $R_1$ and $R_2$. As such, the coding scheme for the setting of Figure 2 differs from that of the Gray–Wyner problem and that of the Heegard–Berger problem in many aspects as shown in Figure 4.

First, the presence of side information sequences imposes the use of “binning” for each of the produced descriptions $V_0^n,V_1^n$ and $V_2^n$ in the Gray–Wyner code construction. However, unlike the binning performed in the Heegard–Berger coding scheme, the binning of the common codeword $V_0^n$ needs to be performed with two different indices, each tailored to a side information sequence at the respective receivers, i.e., “double binning”. Another different aspect is the role of the private and common links. When in the original Gray–Wyner work, these links carried each a description, i.e., $V_0^n$ on the common link and $V_1^n$ with respect to $V_2^n$ on the private links of rates $R_1$ with respect to $R_2$, and when in the Heegard–Berger the three descriptions $V_0^n,V_1^n$ and $V_2^n$ are all carried through the common link only, in the optimal coding scheme of the setting of Figure 2, the private and common links play different roles. Indeed, the common description $V_0^n$ and the private description $V_j^n$ are transmitted on both the common link and the private link of rates $R_0$ and $R_j$, for $j\in \{1,2\}$, through rate-splitting. As such, these key differences imply an intricate interplay between the side information sequences and the role of the common and private links, which we will emphasize later on in Section 4 and Section 5.

Remark 2.

In the special case in which $R_1=R_2=0$, the Gray–Wyner model with side information and degraded reconstruction sets of Figure 2 reduces to a Heegard–Berger problem with arbitrary side information sequences and degraded reconstruction sets, a model that was studied, and solved, recently in the authors’ own recent work [9]. Theorem 1 can then be seen as a generalization of (Theorem 1, [9]) to the case in which the encoder is connected to the receivers also through error-free rate-limited private links of capacity $R_1$ and $R_2$ respectively. The most important insight in the Heegard–Berger problem with degraded reconstruction sets is the role that the common description $V_0$ should play in such a setting. Authors show in (Theorem 1, [9]) that the optimal choice of this description is to contain, intuitively, the common source $S_2$ intended to both users, and, maybe less intuitive, an additional description $U_0$, i.e., $V_0=(U_0,S_2)$, which is used to piggyback part of the source $S_1$ in the common codeword though not required by both receivers, in order to balance the asymmetry of the side information sequences. In Section 4 and Section 5 we show that the utility of this description will depend on both the side information sequences and the rates of the private links.

Remark 3.

In [15], Timo et al. study the Gray–Wyner source coding model with side information of Figure 1. They establish the rate-region of this model in the specific case in which the side information sequence $Y_2^n$ is a degraded version of $Y_1^n$, i.e., $(S_1,S_2)\sout{○}{Y}_1\sout{○}{Y}_2$ is a Markov chain, and both receivers reproduce the component $S_2^n$ and Receiver 1 also reproduces the component $S_1^n$, all in a lossless manner. The result of Theorem 1 generalizes that of (Theorem 5, [15]) to the case of side information sequences that are arbitrarily correlated among them and with the source pair $(S_1,S_2)$ and lossy reconstruction of $S_1$. In [15], Timo et al. also investigate, and solve, a few other special cases of the model, such as those of single source $S_1=S_2$ (Theorem 4, [15]) and complementary delivery $(Y_1,Y_2)=(S_2,S_1)$ (Theorem 6, [15]). The results of (Theorem 4, [15]) and (Theorem 6, [15]) can be recovered from Theorem 1 as special cases of it. Theorem 1 also generalizes (Theorem 6, [15]) to the case of lossy reproduction of the component $S_1^n$.

4. The Heegard–Berger Problem with Successive Refinement

An important special case of the Gray–Wyner source coding model with side information and degraded reconstruction sets of Figure 2 is the case in which $R_2=0$. The resulting model, a Heegard–Berger problem with successive refinement, is shown in Figure 3a.

In this section, we derive the optimal rate distortion region for this setting, and show how it compares to existing results in literature. Besides, we discuss the utility of the common description $U_0$ depending, not only on the side information sequences structures, but also on the refinement link rate $R_1$. We illustrate through a binary example that the utility of $U_0$, namely the optimality of the choice of a non-degenerate $U_0\ne ⌀$, is governed by the quality of the refinement link rate $R_1$ and the side information structure.

4.1. Rate-Distortion Region

The following theorem states the optimal rate-distortion region of the Heegard–Berger problem with successive refinement of Figure 3a.

Corollary 1.

The rate-distortion region of the Heegard–Berger problem with successive refinement of Figure 3a is given by the set of rate-distortion triples $(R_0,R_1,D_1)$ satisfying:

$$\begin{array}{ccc}\hfill R_0& \ge & H(S_2|Y_2)+I(U_0;S_1|S_2{Y}_2)\hfill \end{array}$$

(14a)

$$\begin{array}{ccc}\hfill R_0+R_1& \ge & H(S_2|Y_1)+I(U_0{U}_1;S_1|S_2{Y}_1)\hfill \end{array}$$

(14b)

$$\begin{array}{ccc}\hfill R_0+R_1& \ge & H(S_2|Y_2)+I(U_0;S_1|S_2{Y}_2)+I(U_1;S_1|U_0{S}_2{Y}_1)\hfill \end{array}$$

(14c)

for some product pmf $P_{U_0{U}_1{S}_1{S}_2{Y}_1{Y}_2}$, such that:

(1) 

The following Markov chain is valid:

$$(U_0,U_1)\sout{○}(S_1,S_2)\sout{○}(Y_1,Y_2)$$

(15)

(2) 

There exists a function $\varphi :{\mathcal{Y}}_1\times {\mathcal{U}}_0\times {\mathcal{U}}_1\times {\mathcal{S}}_2\to {\widehat{\mathcal{S}}}_1$ such that:

$$\mathbb{E}{d}_1(S_1,{\widehat{S}}_1)\le D_1.$$

(16)

Proof. 

The proof of Corollary 1 follows from that of Theorem 1 by setting $R_2=0$ therein. ☐

Remark 4.

Recall the coding scheme of Theorem 1. If $R_2=0$, the second partition of the codebook of the common description, which is relevant for Receiver 2, becomes degenerate since, in this case, all the codewords $v_0^n$ of a superbin ${\mathcal{B}}_{00}({\tilde{w}}_{0,0})$ are assigned to a single subbin. Correspondingly, the common message that the encoder sends over the common link carries only the index ${\tilde{w}}_{0,0}$ of the superbin ${\mathcal{B}}_{00}({\tilde{w}}_{0,0})$ of the codebook of the common description in which lies the typical pair $v_0^n=(s_2^n,u_0^n)$, in addition to the index ${\tilde{w}}_{1,0}$ of the subbin ${\mathcal{B}}_{10}({\tilde{w}}_{1,0})$ of the codebook of the individual description in which lies the recovered typical $u_1^n$. Constraint (14a) on the common rate $R_0$ is in accordance with that Receiver 2 utilizes only the index ${\tilde{w}}_{0,0}$ in the decoding. Furthermore, note that Constraints (14b) and (14c) on the sum-rate $(R_0+R_1)$ can be combined as

$$R_0+R_1\ge max\left\{I(U_0{S}_2;S_1{S}_2|Y_1),I(U_0{S}_2;S_1{S}_2|Y_2)\right\}+I(U_1;S_1|U_0{S}_2{Y}_1)$$

(17)

which resembles the Heegard–Berger result of (Theorem 2, p. 733, [2]).

Remark 5.

As we already mentioned, the result of Corollary 1 holds for side information sequences that are arbitrarily correlated among them and with the sources. In the specific case in which the user who gets the refinement rate-limited link also has the “better-quality” side information, in the sense that $(S_1,S_2)\sout{○}{Y}_1\sout{○}{Y}_2$ forms a Markov chain, the rate-distortion region of Corollary 1 reduces to the set of all rate-distortion triples $(R_0,R_1,D_1)$ that satisfy

$$\begin{array}{ccc}\hfill R_0& \ge & H(S_2|Y_2)+I(U_0;S_1|S_2{Y}_2)\hfill \end{array}$$

(18a)

$$\begin{array}{ccc}\hfill R_0+R_1& \ge & H(S_2|Y_2)+I(U_0;S_1|S_2{Y}_2)+I(U_1;S_1|U_0{S}_2{Y}_1).\hfill \end{array}$$

(18b)

for some joint pmf $P_{U_0{U}_1{S}_1{S}_2{Y}_1{Y}_2}$ for which (15) and (16) hold. This result can also be obtained from previous works on successive refinement for the Wyner–Ziv source coding problem by Steinberg and Merhav (Theorem 1, [13]) and Tian and Diggavi (Theorem 1, [7]). The results of (Theorem 1, [13]) and (Theorem 1, [7]) hold for possibly distinct, i.e., not necessarily nested, distortion measures at the receivers; but they require the aforementioned Markov chain condition which is pivotal for their proofs. Thus, for the considered degraded reconstruction sets setting, Corollary 1 can be seen as generalizing (Theorem 1, [13]) and (Theorem 1, [7]) to the case in which the side information sequences are arbitrarily correlated among them and with the sources $(S_1,S_2)$, i.e., do not exhibit any ordering.

Remark 6.

In the case in which it is the user who gets only the common rate-limited link that has the “better-quality” side information, in the sense that $(S_1,S_2)\sout{○}{Y}_2\sout{○}{Y}_1$ forms a Markov chain, the rate distortion region of Corollary 1 reduces to the set of all rate-distortion triples $(R_0,R_1,D_1)$ that satisfy

$$\begin{array}{ccc}\hfill R_0& \ge & H(S_2|Y_2)+I(U_0;S_1|S_2{Y}_2)\hfill \end{array}$$

(19a)

$$\begin{array}{ccc}\hfill R_0+R_1& \ge & H(S_2|Y_1)+I(U_0{U}_1;S_1|S_2{Y}_1)\hfill \end{array}$$

(19b)

for some joint pmf $P_{U_0{U}_1{S}_1{S}_2{Y}_1{Y}_2}$ for which (15) and (16) hold. This result can also be conveyed from [3]. Specifically, in [3] Tian and Diggavi study a therein referred to as “side-information scalable” source coding setup where the side informations are degraded, and the encoder produces two descriptions such that the receiver with the better-quality side information (Receiver 2 if $(S_1,S_2)\sout{○}{Y}_2\sout{○}{Y}_1$ is a Markov chain) uses only the first description to reconstruct its source while the receiver with the low-quality side information (Receiver 1 if $(S_1,S_2)\sout{○}{Y}_2\sout{○}{Y}_1$ is a Markov chain) uses the two descriptions in order to reconstruct its source. They establish inner and outer bounds on the rate-distortion region of the model, which coincide when either one of the decoders requires a lossless reconstruction or when the distortion measures are degraded and deterministic. Similar to the previous remark, Corollary 1 can be seen as generalizing the aforementioned results of [3] to the case in which the side information sequences are arbitrarily correlated among them and with the sources $(S_1,S_2)$.

Remark 7.

A crucial remark that is in order for the Heegard–Berger problem with successive refinement of Figure 3a, is that, depending on the rate of the refinement link $R_1$, resorting to a common auxiliary variable $U_0$ might be unnecessary. Indeed, in the case in which $S_1$ needs to be recovered losslessly at the first receiver, for instance, parts of the rate-region can be achieved without resorting to the common auxiliary variable $U_0$, setting $U_0=⌀$, while other parts of the rate region can only be achieved through a non-trivial choice of $U_0$.

As such, if $R_1\ge H(S_1|S_2{Y}_1)$, then letting $U_0=⌀$ yields the optimal rate region. To see this, note that the rate constraints under lossless construction of $S_1$ write as:

$$\begin{array}{ccc}\hfill R_0& \ge & H(S_1{S}_2|Y_2)-H(S_1|S_2{Y}_2{U}_0)\hfill \end{array}$$

(20a)

$$\begin{array}{ccc}\hfill R_0+R_1& \ge & H(S_1{S}_2|Y_1)\hfill \end{array}$$

(20b)

$$\begin{array}{ccc}\hfill R_0+R_1& \ge & H(S_1{S}_2|Y_2)-H(S_1|S_2{Y}_2{U}_0)+H(S_1|U_0{S}_2{Y}_1)\hfill \end{array}$$

(20c)

which, can be rewritten as follows

$$\begin{array}{ccc}\hfill *R_0& \ge & H(S_1{S}_2|Y_2)+\underset{P_{U_0|S_1{S}_2}}{min}\left[{(H(S_1|S_2{Y}_1{U}_0)-R_1)}^{+}-H(S_1|S_2{Y}_2{U}_0)\right]\hfill \end{array}$$

(21a)

$$\begin{array}{ccc}\hfill R_0+R_1& \ge & H(S_1{S}_2|Y_1)\hfill \end{array}$$

(21b)

where ${(x)}^{+}\triangleq max\{0,x\}$.

Under the constraint that $R_1\ge H(S_1|S_2{Y}_1)$, the constraints in (21a) reduce to the following

$$\begin{array}{ccc}\hfill R_0& \ge & H(S_1{S}_2|Y_2)-\underset{P_{U_0|S_1{S}_2}}{max}H(S_1|S_2{Y}_2{U}_0)\hfill \end{array}$$

(22a)

$$\begin{array}{ccc}\hfill R_0+R_1& \ge & H(S_1{S}_2|Y_1).\hfill \end{array}$$

(22b)

Next, by noting that ${max}_{P_{U_0|S_1{S}_2}}H(S_1|S_2{Y}_2{U}_0)=H(S_1|S_2{Y}_2)$ is achieved by $U_0=⌀$, the claim follows.

However, when $R_1<H(S_1|S_2{Y}_1)$, the choice of $U_0=⌀$ might be strictly sub-optimal (as shown in the following binary example).

4.2. Binary Example

Let $X_1$, $X_2$, $X_3$ and $X_4$ be four independent $\mathrm{Ber}(1/2)$ random variables. Let the sources be $S_1\triangleq (X_1,X_2,X_3)$ and $S_2\triangleq X_4$. Now, consider the Heegard–Berger model with successive refinement shown in Figure 5. The first user, which gets both the common and individual links, observes the side information $Y_1=(X_1,X_4)$ and wants to reproduce the pair $(S_1,S_2)$ losslessly. The second user gets only the common link, has side information $Y_2=(X_2,X_3)$ and wants to reproduce only the component $S_2$, losslessly.

The side information at the decoders do not exhibit any degradedness ordering, in the sense that none of the Markov chain conditions of Remarks 5 and 6 hold. The following claim provides the rate-region of this binary example.

Claim 1.

The rate region of the binary Heegard–Berger example with successive refinement of Figure 5 is given by the set of rate pairs $(R_0,R_1)$ that satisfy

$$\begin{array}{ccc}\hfill R_0& \ge & 1\hfill \end{array}$$

(23a)

$$\begin{array}{ccc}\hfill R_0+R_1& \ge & 2.\hfill \end{array}$$

(23b)

Proof. 

The proof of Claim 1 follows easily by computing the rate region

$$\begin{array}{ccc}\hfill R_0& \ge \hfill & H(S_1{S}_2|Y_2)-H(S_1|S_2{Y}_2{U}_0)\end{array}$$

(24a)

$$\begin{array}{ccc}\hfill R_0+R_1& \ge \hfill & H(S_1{S}_2|Y_1)\end{array}$$

(24b)

$$\begin{array}{ccc}\hfill R_0+R_1& \ge \hfill & H(S_1{S}_2|Y_2)-H(S_1|S_2{Y}_2{U}_0)+H(S_1|U_0{S}_2{Y}_1)\end{array}$$

(24c)

in the binary setting under study.

First, we note that

$$\begin{array}{ccc}\hfill H(S_1{S}_2|Y_2)& =\hfill & H(X_1{X}_4|X_2{X}_3)=2\end{array}$$

(25)

$$\begin{array}{ccc}\hfill H(S_1{S}_2|Y_1)& =\hfill & H(X_2{X}_3|X_1{X}_4)=2\end{array}$$

(26)

which allows then to rewrite the rate region as

$$\begin{array}{cc}\hfill R_0& \ge 2-H(X_1|X_4{U}_0)\ge 2-H(X_1|X_4)=1\hfill \end{array}$$

(27a)

$$\begin{array}{cc}\hfill R_0+R_1& \ge 2+max\{0,H(X_2{X}_3|X_1{X}_4{U}_0)-H(X_1|X_2{X}_3{X}_4{U}_0)\}\ge 2\hfill \end{array}$$

(27b)

The proof of the claim follows by noticing that the following inequalities hold with equality for the choices $U_0=(X_2,X_3)$ or $U_0=X_2$ or $U_0=X_3$. ☐

The rate region of Claim 1 is depicted in Figure 6. It is insightful to notice that although the second user is only interested in reproducing the component $S_2=X_4$, the optimal coding scheme that achieves this region sets the common description that is destined to be recovered by both users as one that is composed of not only $S_2$ but also some part $U_0=(X_2,X_3)$, or $U_0=X_2$ or $U_0=X_3$, of the source component $S_1$ (though the latter is not required by the second user). A possible intuition is that this choice of $U_0$ is useful for user 1, who wants to reproduce $S_1=(X_1,X_2,X_3)$, and its transmission to also the second user does not cost any rate loss since this user has available side information $Y_2=(X_2,X_3)$.

5. The Heegard–Berger Problem with Scalable Coding

In the following, we consider the model of Figure 3b. As we already mentioned, the reader may find it appropriate for the motivation to think about the side information $Y_2^n$ as being of lower quality than $Y_1^n$, in which case, the refinement link that is given to the second user is intended to improve its decoding capability. In this section, we describe the optimal coding scheme for this setting, and show that it can be recovered, independently, from the work of Timo et al. [14] through a careful choice of the coding sets. Next, we illustrate through a binary example the interplay between the utility of the common description $U_0$ and the side information sequences, and the refinement rate $R_2$.

5.1. Rate-Distortion Region

The following theorem states the rate-distortion region of the Heegard–Berger model with scalable coding of Figure 3b.

Corollary 2.

The rate-distortion region of the Heegard–Berger model with scalable coding of Figure 3b is given by the set of all rate-distortion triples $(R_0,R_2,D_1)$ that satisfy

$$\begin{array}{ccc}\hfill R_0& \ge & H(S_2|Y_1)+I(U_0{U}_1;S_1|S_2{Y}_1)\hfill \end{array}$$

(28a)

$$\begin{array}{ccc}\hfill R_0+R_2& \ge & H(S_2|Y_2)+I(U_0;S_1|S_2{Y}_2)+I(U_1;S_1|U_0{S}_2{Y}_1)\hfill \end{array}$$

(28b)

for some product pmf $P_{U_0{U}_1{S}_1{S}_2{Y}_1{Y}_2}$, such that:

(1) 

The following Markov chain is valid:

$$(U_0,U_1)\sout{○}(S_1,S_2)\sout{○}(Y_1,Y_2)$$

(29)

(2) 

There exists a function $\varphi :{\mathcal{Y}}_1\times {\mathcal{U}}_0\times {\mathcal{U}}_1\times {\mathcal{S}}_2\to {\widehat{\mathcal{S}}}_1$ such that:

$$\mathbb{E}{d}_1(S_1,{\widehat{S}}_1)\le D_1.$$

(30)

Proof. 

The proof of Corollary 2 follows from that of Theorem 1 by seeting $R_1=0$ therein. ☐

Remark 8.

In the specific case in which Receiver 2 has a better-quality side information in the sense that $(S_1,S_2)\sout{○}{Y}_2\sout{○}{Y}_1$ forms a Markov chain, the rate distortion region of Corollary 2 reduces to one that is described by a single rate-constraint, namely

$$\begin{array}{ccc}\hfill R_0& \ge \hfill & H(S_2|Y_1)+I(U;S_1|S_2{Y}_1)\end{array}$$

(31)

for some conditional $P_{U|S_1{S}_2}$ that satisfies $\mathbb{E}[d_1(S_1,{\widehat{S}}_1)]\le D_1$. This is in accordance with the observation that, in this case, the transmission to Receiver 1 becomes the bottleneck, as Receiver 2 can recover the source component $S_2$ losslessly as long as so does Receiver 1.

Remark 9.

Consider the case in which $S_1$ needs to be recovered losslessly as well at Receiver 1. Then, the rate region is can be expressed as follows

$$\begin{array}{cc}\hfill R_0& \ge H(S_1{S}_2|Y_1)\hfill \end{array}$$

(32a)

$$\begin{array}{cc}\hfill R_0+R_2& \ge H(S_1{S}_2|Y_2)+\underset{P_{U_0|S_1{S}_2}}{min}\left[H(S_1|U_0{S}_2{Y}_1)-H(S_1|U_0{S}_2{Y}_2)\right].\hfill \end{array}$$

(32b)

An important comment here is that the optimization problem in $P_{U_0|S_1{S}_2}$ does not depend on the refinement link $R_2$, and the optimal solution to it, i.e., the optimal choice of $U_0$, meets the solution to the Heegard–Berger problem without refinement link, $R_2=0$, rendering it optimal for all choices of $R_2$, which is a main difference with the Heegard–Berger problem with refinement link of Figure 3a in which the solution to the Heegard–Berger problem (with $R_1=0$) might not be optimal for all values of $R_1$.

Remark 10.

In (Theorem 1, [14]), Timo et al. present an achievable rate-region for the multistage successive-refinement problem with side information. Timo et al. consider distortion measures of the form ${\delta }_l:\mathcal{X}\times {\widehat{\mathcal{X}}}_l\to {\mathbb{R}}_{+}$, where $\mathcal{X}$ is the source alphabet and ${\widehat{\mathcal{X}}}_l$ is the reconstruction at decoder l, $l\in \{1,\cdots ,t\}$; and for this reason this result is not applicable as is to the setting of Figure 3b, in the case of two decoders. However, the result of (Theorem 1, [14]) can be extended to accommodate a distortion measure at the first decoder that is vector-valued; and the direct part of Corollary 2 can then be obtained by applying this extension. Specifically, in the case of two decoders, i.e., $t=2$, and with $X=(S_1,S_2)$, and two distortion measures ${\delta }_1:{\mathcal{S}}_1\times {\mathcal{S}}_2\times {\widehat{\mathcal{S}}}_{1,1}\times {\widehat{\mathcal{S}}}_{1,2}\to \{0,1\}\times {\mathbb{R}}_{+}$ and ${\delta }_2:{\mathcal{S}}_1\times {\mathcal{S}}_2\times {\widehat{\mathcal{S}}}_{1,2}\times {\widehat{\mathcal{S}}}_{2,2}\to \{0,1\}$ chosen such that

$${\delta }_1\left((s_1,s_2),({\widehat{s}}_{1,1},{\widehat{s}}_{2,1})\right)=\left(d_H(s_2,{\widehat{s}}_{2,1}),d_1(s_1,{\widehat{s}}_{1,1})\right)$$

(33)

and

$${\delta }_2\left((s_1,s_2),({\widehat{s}}_{1,2},{\widehat{s}}_{2,2})\right)=d_H(s_2,{\widehat{s}}_{2,2})$$

(34)

where $d_H(·,·)$ is the Hamming distance, letting $d_1=(0,D_1)$ and $d_2=0$, a straightforward extension of (Theorem 1, [14]) to this setting yields a rate-region that is described by the following rate constraints (using the notation of (Theorem 1, [14]))

$$\begin{array}{cc}\hfill R_0& \ge \mathsf{\Phi }({\mathcal{T}}_0,1)+\mathsf{\Phi }({\mathcal{T}}_1,1)\hfill \end{array}$$

(35a)

$$\begin{array}{cc}\hfill R_0+R_2& \ge \mathsf{\Phi }({\mathcal{T}}_0,2)+\mathsf{\Phi }({\mathcal{T}}_1,2)+\mathsf{\Phi }({\mathcal{T}}_2,2)\hfill \end{array}$$

(35b)

where ${\mathcal{T}}_0=\{1,2\}$, ${\mathcal{T}}_1=\{1\}$, $T_2=\{2\}$, and for $j=0,1,2$ and $l\in 1,2$ such that ${\mathcal{T}}_j\cap \{1,\cdots ,l\}\ne ⌀$, the function $\mathsf{\Phi }({\mathcal{T}}_j,l)$, $j=0,1,2$, is defined as

$${\displaystyle \mathsf{\Phi }({\mathcal{T}}_j,l)=I\left(S_1{S}_2{\mathcal{A}}_{{\mathcal{T}}_j}^{†};U_{{\mathcal{T}}_j}|{\mathcal{A}}_{{\mathcal{T}}_j}^{\supset }\right)-\underset{l^{\prime }\in {\mathcal{T}}_j\cap [1:l]}{min}I\left(U_{{\mathcal{T}}_j};{\mathcal{A}}_{{\mathcal{T}}_j,l^{\prime }}^{‡}{Y}_{l^{\prime }}|{\mathcal{A}}_{{\mathcal{T}}_j}^{\supset }\right)}$$

(36)

where $\mathcal{A}=\{U_{12},U_1,U_2\}$ and the sets ${\mathcal{A}}_{{\mathcal{T}}_j}^{-}$, ${\mathcal{A}}_{{\mathcal{T}}_j}^{\supset }$, ${\mathcal{A}}_{{\mathcal{T}}_j}^{+}$, ${\mathcal{A}}_{{\mathcal{T}}_j}^{†}$, ${\mathcal{A}}_{{\mathcal{T}}_j,1}^{‡}$, ${\mathcal{A}}_{{\mathcal{T}}_j,2}^{‡}$, evaluated in this case, are given in

Table 1. Auxiliary random variables associated with the subsets that appear in (36).

	${\mathcal{T}}_0$	${\mathcal{T}}_1$	${\mathcal{T}}_2$
${\mathcal{A}}_{{\mathcal{T}}_j}^{-}$	∅	∅	$U_1$
${\mathcal{A}}_{{\mathcal{T}}_j}^{\supset }$	∅	$U_{12}$	$U_{12}$
${\mathcal{A}}_{{\mathcal{T}}_j}^{+}$	$\{U_1,U_2\}$	∅	∅
${\mathcal{A}}_{{\mathcal{T}}_j}^{†}$	∅	∅	∅
${\mathcal{A}}_{{\mathcal{T}}_j,1}^{‡}$	∅	∅	∅
${\mathcal{A}}_{{\mathcal{T}}_j,2}^{‡}$	∅	∅	∅

. It is easy to see that the region described by (35) can be written more explicitly in this case as

$$\begin{array}{cc}\hfill R_0& \ge I(U_{12};S_1{S}_2|Y_1)\hfill \end{array}$$

(37a)

$$\begin{array}{cc}\hfill R_0+R_2& \ge max\{I(U_{12};S_1{S}_2|Y_1),I(U_{12};S_1{S}_2|Y_2)\}+I(U_1;S_1{S}_2|Y_1{U}_{12})+I(U_2;S_1{S}_2|Y_2{U}_{12}).\hfill \end{array}$$

(37b)

Also, setting $U_{12}=(U_0,S_2)$ and $U_2=S_2$ in (37) one recovers the rate-region of Corollary 2. (Such a connection can also be stated for the result of Corollary 1).

5.2. Binary Example

Consider the setting of Figure 7. Let $X_1$, $X_2$, $X_3$ and $X_4$ be four independent $\mathrm{Ber}(1/2)$ random variables. Let the sources be $S_1\triangleq (X_1,X_2,X_3)$ and $S_2\triangleq X_4$. Now, consider the Heegard–Berger model with scalable coding shown in Figure 7. The first user, which gets both only the common link, observes the side information $Y_1=(X_1,X_4)$ and wants to reproduce the pair $(S_1,S_2)$ losslessly. The second user gets both the common and private links, has side information $Y_2=(X_2,X_3)$ and wants to reproduce only the component $S_2$, losslessly.

Claim 2.

The rate region of the binary Heegard–Berger example with scalable coding of Figure 7 is given by the set of all rate pairs $(R_0,R_2)$ that satisfy $R_2\ge 0$ and $R_0\ge 2$.

Proof. 

The proof of Claim 2 follows easily by specializing, and computing, the result of Remark 9 for the example at hand. First note that

$$\begin{array}{}\mathrm{(38a)}& \hfill R_0+R_2& \ge H(S_2{S}_1|Y_2)+\underset{P_{U_0|S_1{S}_2}}{min}\left[H(S_1|U_0{S}_2{Y}_1)-H(S_1|U_0{S}_2{Y}_2)\right]\hfill \mathrm{(38b)}& & =2+\underset{P_{U_0|S_1{S}_2}}{min}[H(X_2{X}_3|X_1{X}_4{U}_0)-H(X_1|X_2{X}_3{X}_4{U}_0)]\hfill \mathrm{(38c)}& & \ge 2+\underset{P_{U_0|S_1{S}_2}}{min}[-H(X_1|X_2{U}_0)]\hfill \mathrm{(38d)}& & \ge 1\hfill \end{array}$$

where equality in all previous inequalities is satisfied with $U_0=(X_2,X_3)$ or with $U_0=X_2$ or $U_0=X_3$.

Note as well that the single rate constraint on $R_0$ writes as:

$$\begin{array}{}\mathrm{(39a)}& \hfill R_0& \ge H(S_1{S}_2|Y_1)\hfill \mathrm{(39b)}& & =2\hfill \end{array}$$

which renders the sum-rate constraint redundant and ends the proof of the claim. ☐

The optimal rate region of Claim 2 is depicted in Figure 8, as the region delimited by the lines $R_0=1$ and $R_2=0$. Note that for this example, the source component $X_2$, which is the only source component that is required by Receiver 2, needs to be transmitted entirely on the common link to also be recovered losslessly by Receiver 1. For this reason, the refinement link is not-constrained and appears to be useless for this example.

There is a sharp difference with the binary Heegard–Berger example with successive refinement of Figure 5 for which the refinement link may sometimes be instrumental to reducing the required rate on the common link. With scalable coding, the refinement link with rate $R_0$ does not improve the rate transmitted on the common link.

Also, it is insightful to notice that for this example, because of the side information configuration, the choice $U_0=⌀$ in Corollary 2 is strictly suboptimal and results in the smaller region that is described by

$$\begin{array}{c}\hfill R_0\ge 2\end{array}$$

(40a)

$$\begin{array}{c}\hfill R_0+R_2\ge 3.\end{array}$$

(40b)

6. Proof of Theorem 1

In the following, we give the proof of the converse part and the direct part of Theorem 1.

The converse part is strongly dependent on the system model we investigate and consists in a series of careful bounding steps resorting to Fano’s inequality, Markov chains and Csiszár–Körner sum-identity.

The proof of achievability is two-fold, and consists in proving a general result that holds for a Gray–Wyner setting with side information, and then deriving the optimal choice of the auxiliary codewords involved for the specific setting with degraded reconstruction sets.

6.1. Proof of Converse Part

Assume that a rate triple $(R_0,R_1,R_2)$ is $D_1$-achievable. Then, let $W_j=f_j(S_1^n,S_2^n)$, where $j\in \{0,1,2\}$, be the encoded indices and let ${\widehat{S}}_1^n=g_1(W_0,W_1,Y_1^n)$ be the reconstruction sequence at the first decoder such that $\mathbb{E}{d}_1^{(n)}(S_1^n,{\widehat{S}}_1^n)\le D_1$.

Using Fano’s inequality, the lossless reconstruction of the source $S_2^n$ at both decoders implies that there exists a sequence ${ϵ}_n\underset{n\to \infty }{\to }0$ such that:

$$\begin{array}{ccc}\hfill H(S_2^n|W_0{W}_1{Y}_1^n)& \le \hfill & n{ϵ}_n,\end{array}$$

(41)

$$\begin{array}{ccc}\hfill H(S_2^n|W_0{W}_2{Y}_2^n)& \le \hfill & n{ϵ}_n.\end{array}$$

(42)

We start by showing the following sum-rate constraint,

$$R_0+R_1+R_2\ge H(S_2|Y_2)+I(U_0;S_1|S_2{Y}_2)+I(U_1;S_1|U_0{S}_2{Y}_1).$$

(43)

We have that

$$\begin{array}{}& & n(R_0+R_1+R_2)\hfill \mathrm{(44a)}& \ge & H(W_0)+H(W_2)+H(W_1)\hfill \mathrm{(44b)}& \ge & H(W_0)+H(W_2|W_0)+H(W_1)\hfill \mathrm{(44c)}& =& H(W_0{W}_2)+H(W_1)\hfill \mathrm{(44d)}& \ge & H(W_0{W}_2|Y_2^n)+H(W_1|W_0{S}_2^n{Y}_1^n)\hfill \mathrm{(44e)}& \ge & I(W_0{W}_2;S_1^n{S}_2^n|Y_2^n)+I(W_1;S_1^n|W_0{S}_2^n{Y}_1^n)\hfill \mathrm{(44f)}& =& H(S_1^n{S}_2^n|Y_2^n)-H(S_1^n{S}_2^n|W_0{W}_2{Y}_2^n)+H(S_1^n|W_0{S}_2^n{Y}_1^n)-H(S_1^n|W_0{W}_1{S}_2^n{Y}_1^n)\hfill \mathrm{(44g)}& \stackrel{(a)}{\ge }& H(S_1^n{S}_2^n|Y_2^n)-H(S_1^n|W_0{W}_2{S}_2^n{Y}_2^n)+H(S_1^n|W_0{S}_2^n{Y}_1^n)-H(S_1^n|W_0{W}_1{S}_2^n{Y}_1^n)-n{ϵ}_n\hfill \mathrm{(44h)}& \ge & H(S_1^n{S}_2^n|Y_2^n)-H(S_1^n|W_0{S}_2^n{Y}_2^n)+H(S_1^n|W_0{S}_2^n{Y}_1^n)-H(S_1^n|W_0{W}_1{S}_2^n{Y}_1^n)-n{ϵ}_n\hfill \end{array}$$

where $(a)$ in (44) stems from Fano’s inequality (42), which results from the lossless reconstruction of $S_2^n$ at receiver 2.

Let us define then:

$$\begin{array}{ccc}\hfill A& \triangleq \hfill & H(S_1^n|W_0{S}_2^n{Y}_1^n)-H(S_1^n|W_0{S}_2^n{Y}_2^n),\end{array}$$

(45)

$$\begin{array}{ccc}\hfill B& \triangleq \hfill & H(S_1^n|W_0{W}_1{S}_2^n{Y}_1^n).\end{array}$$

(46)

In the following, we aim for single-letter bounds on the two quantities A and B.

Since the side information sequences $Y_1^n$ and $Y_2^n$ are not degraded and do not exhibit any structure, together with the sources $(S_1^n,S_2^n)$, single-letterizing the quantity A can be obtained through some judicious bounding steps that are reported below, in which some important Markov chain are shown to hold and quantities are manipulated appropriately, together with several invocations of Csiszár–Körner sum identity .

Let us start by writing that

$$\begin{array}{}\mathrm{(47a)}& \hfill A& \triangleq H(S_1^n|W_0{S}_2^n{Y}_1^n)-H(S_1^n|W_0{S}_2^n{Y}_2^n)\hfill \mathrm{(47b)}& & =I(S_1^n;Y_2^n|W_0{S}_2^n)-I(S_1^n;Y_1^n|W_0{S}_2^n)\hfill \mathrm{(47c)}& & ={\displaystyle \sum _{i=1}^n}\left[I(S_1^n;Y_{2,i}|W_0{Y}_2^{i-1}{S}_2^n)-I(S_1^n;Y_{1,i}|W_0{Y}_{1,i+1}^n{S}_2^n)\right]\hfill \mathrm{(47d)}& & \stackrel{(a)}{=}{\displaystyle \sum _{i=1}^n}\left[I(S_1^n{Y}_{1,i+1}^n;Y_{2,i}|W_0{Y}_2^{i-1}{S}_2^n)-I(S_1^n{Y}_2^{i-1};Y_{1,i}|W_0{Y}_{1,i+1}^n{S}_2^n)\right]\hfill \mathrm{(47e)}& & \stackrel{(b)}{=}{\displaystyle \sum _{i=1}^n}\left[I(S_1^n;Y_{2,i}|W_0{Y}_2^{i-1}{Y}_{1,i+1}^n{S}_2^n)-I(S_1^n;Y_{1,i}|W_0{Y}_2^{i-1}{Y}_{1,i+1}^n{S}_2^n)\right]\hfill \mathrm{(47f)}& & \stackrel{(c)}{=}{\displaystyle \sum _{i=1}^n}\left[I(S_{1,i};Y_{2,i}|W_0{Y}_2^{i-1}{Y}_{1,i+1}^n{S}_2^n)-I(S_{1,i};Y_{1,i}|W_0{Y}_2^{i-1}{Y}_{1,i+1}^n{S}_2^n)\right]\hfill \mathrm{(47g)}& & ={\displaystyle \sum _{i=1}^n}\left[H(S_{1,i}|Y_{1,i}{W}_0{Y}_2^{i-1}{Y}_{1,i+1}^n{S}_2^n)-H(S_{1,i}|Y_{2,i}{W}_0{Y}_2^{i-1}{Y}_{1,i+1}^n{S}_2^n)\right]\hfill \mathrm{(47h)}& & ={\displaystyle \sum _{i=1}^n}\left[H(S_{1,i}|Y_{1,i}{S}_{2,i}{U}_{0,i})-H(S_{1,i}|Y_{2,i}{S}_{2,i}{U}_{0,i})\right]\hfill \end{array}$$

where $U_{0,i}\triangleq (W_0,Y_2^{i-1},Y_{1,i+1}^n,S_{2,<i>})$ (note that the lossless reconstruction of $S_2^n$ at both receivers is instrumental to the definition of $U_0$ which plays the role of the common auxiliary variable in the proof of converse), and where $(a)$ in (47) follows using the following Csiszár–Körner sum-identity

$$\sum _{i=1}^{n}I(Y_2^{i-1};Y_{1,i}|S_1^n{W}_0{Y}_{1,i+1}^n{S}_2^n)=\sum _{i=1}^{n}I(Y_{1,i+1}^n;Y_{2,i}|S_1^n{W}_0{Y}_2^{i-1}{S}_2^n),$$

(48)

$(b)$ in (47) follows using the Csiszár–Körner sum-identity given by

$$\begin{array}{c}{\displaystyle \sum _{i=1}^n}I(Y_2^{i-1};Y_{1,i}|W_0{Y}_{1,i+1}^n{S}_2^n)={\displaystyle \sum _{i=1}^n}I(Y_{1,i+1}^n;Y_{2,i}|W_0{Y}_2^{i-1}{S}_2^n),\hfill \end{array}$$

(49)

while $(c)$ in (47) is the consequence of the following sequence of Markov chains

$$\begin{array}{c}\hfill (S_1^{i-1},S_{1,i+1}^n,S_2^{i-1},S_{2,i+1}^n,Y_{1,i+1}^n,Y_2^{i-1})\sout{○}(S_{1,i},S_{2,i})\sout{○}{Y}_{j,i}\end{array}$$

(50a)

$$\begin{array}{c}\hfill \stackrel{(a)}{\Rightarrow }(S_1^{i-1},S_{1,i+1}^n,S_2^{i-1},S_{2,i+1}^n,Y_{1,i+1}^n,Y_2^{i-1},W_0)\sout{○}(S_{1,i},S_{2,i})\sout{○}{Y}_{j,i}\end{array}$$

(50b)

$$\begin{array}{c}\hfill \Rightarrow (S_1^{i-1},S_{1,i+1}^n)\sout{○}(S_2^{i-1},S_{2,i+1}^n,Y_{1,i+1}^n,Y_2^{i-1},W_0,S_{1,i},S_{2,i})\sout{○}{Y}_{j,i}\end{array}$$

(50c)

where (50a) results from that the source sequences $(S_1^n,S_2^n,Y_1^n,Y_2^n)$ are memoryless, while $(a)$ in (50) is a consequence of that $W_0$ is a function of the pair of sequences $(S_1^n,S_2^n)$.

To upper-bound the term B, note the following

$$\begin{array}{}\mathrm{(51a)}& \hfill B& \triangleq H(S_1^n|W_0{W}_1{S}_2^n{Y}_1^n)\hfill \mathrm{(51b)}& & ={\displaystyle \sum _{i=1}^n}H(S_{1,i}|W_0{W}_1{S}_2^n{Y}_1^n{S}_1^{i-1})\hfill \mathrm{(51c)}& & ={\displaystyle \sum _{i=1}^n}H(S_{1,i}|S_{2,i}{Y}_{1,i}{W}_0{S}_{2,<i>}{Y}_{1,i+1}^n{S}_1^{i-1}{W}_1{Y}_1^{i-1})\hfill \mathrm{(51d)}& & \stackrel{(a)}{=}{\displaystyle \sum _{i=1}^n}H(S_{1,i}|S_{2,i}{Y}_{1,i}{W}_0{S}_{2,<i>}{Y}_{1,i+1}^n{S}_1^{i-1}{Y}_2^{i-1}{W}_1{Y}_1^{i-1})\hfill \mathrm{(51e)}& & \le {\displaystyle \sum _{i=1}^n}H(S_{1,i}|S_{2,i}{Y}_{1,i}{W}_0{S}_{2,<i>}{Y}_{1,i+1}^n{Y}_2^{i-1}{W}_1{Y}_1^{i-1})\hfill \end{array}$$

where $(a)$ in (51) is a consequence of the following sequence of Markov chains:

$$\begin{array}{}\mathrm{(52a)}& & Y_2^{i-1}\sout{○}(S_1^{i-1},S_2^{i-1},Y_1^{i-1})\sout{○}(S_{1,i},S_{1,i+1}^n,S_{2,i},S_{2,i+1}^n,Y_{1,i+1}^n)\hfill \mathrm{(52b)}& \stackrel{(a)}{\Rightarrow }& Y_2^{i-1}\sout{○}(S_1^{i-1},S_2^{i-1},Y_1^{i-1})\sout{○}(S_{1,i},S_{1,i+1}^n,S_{2,i},S_{2,i+1}^n,Y_{1,i+1}^n,W_0,W_1)\hfill \mathrm{(52c)}& \Rightarrow & Y_2^{i-1}\sout{○}(S_1^{i-1},S_2^{i-1},Y_1^{i-1},S_{2,i},S_2^{i-1},Y_{1,i+1}^n,W_0,W_1)\sout{○}{S}_{1,i}.\hfill \end{array}$$

where (52a) results from that the source sequences $(S_1^n,S_2^n,Y_1^n,Y_2^n)$ are memoryless, while $(a)$ in (52) is a consequence of that $W_0$ and $W_1$ are each function of the pair of sequences $(S_1^n,S_2^n)$.

Finally, letting $U_{1,i}\triangleq (W_1,Y_1^{i-1})$ so that the choice of $(U_{0,i},U_{1,i})$ satisfy the condition ${\widehat{S}}_{1,i}=g_i(Y_{1,i},U_{0,i},U_{1,i},S_{2,i})$, we write the resulting sum-rate constraint as

$$\begin{array}{ccc}\hfill n(R_0+R_1+R_2)& \ge \hfill & nH(S_1{S}_2|Y_2)+{\displaystyle \sum _{i=1}^n}\left[H(S_{1,i}|S_{2,i}{Y}_{1,i}{U}_{0,i})-H(S_{1,i}|S_{2,i}{Y}_{2,i}{U}_{0,i})\right]\\ & & -{\displaystyle \sum _{i=1}^n}H(S_{1,i}|S_{2,i}{Y}_{1,i}{U}_{0,i}{U}_{1,i})-n{ϵ}_n.\end{array}$$

(53)

Let us now prove that the following bound holds

$$R_0+R_1\ge H(S_2{S}_1|Y_1)-H(S_1|U_0{U}_1{Y}_1{S}_2).$$

(54)

We have

$$\begin{array}{}\mathrm{(55a)}& \hfill n(R_0+R_1)& \ge H(W_0)+H(W_1|W_0)\hfill \mathrm{(55b)}& & =H(W_0,W_1)\hfill \mathrm{(55c)}& & \ge H(W_0{W}_1|Y_1^n)\hfill \mathrm{(55d)}& & \ge I(W_0{W}_1;S_1^n{S}_2^n|Y_1^n)\hfill \mathrm{(55e)}& & =H(S_1^n{S}_2^n|Y_1^n)-H(S_1^n{S}_2^n|W_0{W}_1{Y}_1^n)\hfill \mathrm{(55f)}& & \stackrel{(a)}{\ge }H(S_1^n{S}_2^n|Y_1^n)-H(S_1^n|W_0{W}_1{S}_2^n{Y}_1^n)-n{ϵ}_n\hfill \mathrm{(55g)}& & =nH(S_1{S}_2|Y_1)-B-n{ϵ}_n\hfill \mathrm{(55h)}& & \stackrel{(b)}{\ge }nH(S_1{S}_2|Y_1)-{\displaystyle \sum _{i=1}^n}H(S_{1,i}|S_{2,i}{Y}_{1,i}{U}_{0,i}{U}_{1,i})-n{ϵ}_n,\hfill \end{array}$$

where $(a)$ in (55) is a consequence of Fano’s inequality in (41), which results from the lossless reconstruction of $S_2^n$ at receiver 1, and $(b)$ in (55) results from the upper bound on B in (51e). As for the third rate constraint

$$R_0+R_2\ge H(S_1{S}_2|Y_2)-H(S_1|U_0{Y}_2{S}_2),$$

(56)

we write

$$\begin{array}{}\mathrm{(57a)}& \hfill n(R_0+R_2)& \ge H(W_0{W}_2)\hfill \mathrm{(57b)}& & \ge H(W_0{W}_2|Y_2^n)\hfill \mathrm{(57c)}& & \ge I(W_0{W}_2;S_1^n{S}_2^n|Y_2^n)\hfill \mathrm{(57d)}& & =H(S_1^n{S}_2^n|Y_2^n)-H(S_1^n{S}_2^n|W_0{W}_2{Y}_2^n)\hfill \mathrm{(57e)}& & \stackrel{(a)}{\ge }H(S_1^n{S}_2^n|Y_2^n)-H(S_1^n|W_0{W}_2{S}_2^n{Y}_2^n)-n{ϵ}_n\hfill \mathrm{(57f)}& & \ge H(S_1^n{S}_2^n|Y_2^n)-H(S_1^n|W_0{S}_2^n{Y}_2^n)-n{ϵ}_n\hfill \mathrm{(57g)}& & =nH(S_1{S}_2|Y_2)-{\displaystyle \sum _{i=1}^n}H(S_{1,i}|S_{2,i}{Y}_{2,i}{W}_0{S}_{2,<i>}{Y}_{2,<i>}{S}_{1,i+1}^n)-n{ϵ}_n\hfill \mathrm{(57h)}& & \stackrel{(b)}{=}nH(S_1{S}_2|Y_2)-{\displaystyle \sum _{i=1}^n}H(S_{1,i}|S_{2,i}{Y}_{2,i}{W}_0{S}_{2,<i>}{Y}_{2,<i>}{S}_{1,i+1}^n{Y}_{1,i+1}^n)-n{ϵ}_n\hfill \mathrm{(57i)}& & \ge nH(S_1{S}_2|Y_2)-{\displaystyle \sum _{i=1}^n}H(S_{1,i}|S_{2,i}{Y}_{2,i}{W}_0{S}_{2,<i>}{Y}_2^{i-1}{Y}_{1,i+1}^n)-n{ϵ}_n\hfill \mathrm{(57j)}& & =nH(S_1{S}_2|Y_2)-{\displaystyle \sum _{i=1}^n}H(S_{1,i}|S_{2,i}{Y}_{2,i}{U}_{0,i})-n{ϵ}_n.\hfill \end{array}$$

where $(a)$ in (57) is a consequence of Fano’s inequality in (42) and $(b)$ in (57) stems for the following sequence of Markov Chains.

$$\begin{array}{}\mathrm{(58a)}& & Y_{1,i+1}^n\sout{○}(S_{2,i+1}^n,S_{1,i+1}^n,Y_{1,i+1}^n)\sout{○}(S_{1,i},S_1^{i-1},S_{2,i},S_2^{i-1},Y_1^{i-1})\hfill \mathrm{(58b)}& \stackrel{(a)}{\Rightarrow }& Y_{1,i+1}^n\sout{○}(S_{2,i+1}^n,S_{1,i+1}^n,Y_{1,i+1}^n)\sout{○}(S_{1,i},S_1^{i-1},S_{2,i},S_2^{i-1},Y_1^{i-1},W_0,W_1)\hfill \mathrm{(58c)}& \Rightarrow & Y_{1,i+1}^n\sout{○}(S_{2,i+1}^n,S_{1,i+1}^n,Y_{1,i+1}^n,S_{2,i},S_2^{i-1},Y_1^{i-1},W_0,W_1)\sout{○}{S}_{1,i}.\hfill \end{array}$$

where (58a) results from that the source sequences $(S_1^n,S_2^n,Y_1^n,Y_2^n)$ are memoryless, while $(a)$ in (58) is a consequence of that $W_0$ and $W_1$ are each function of the pair of sequences $(S_1^n,S_2^n)$.

Let Q be an integer-valued random variable, ranging from 1 to n, uniformly distributed over [1:n] and independent of all other variables $(S_1,S_2,U_0,U_1,Y_1,Y_2)$. We have

$$\begin{array}{}& \hfill R_0+R_1+R_2& \ge H(S_1{S}_2|Y_2)+{\displaystyle \frac{1}{n}}{\displaystyle \sum _{i=1}^n}\left[H(S_{1,i}|S_{2,i}{Y}_{1,i}{U}_{0,i})-H(S_{1,i}|S_{2,i}{Y}_{2,i}{U}_{0,i})\right]\hfill \mathrm{(59a)}& & -{\displaystyle \frac{1}{n}}{\displaystyle \sum _{i=1}^n}H(S_{1,i}|S_{2,i}{Y}_{1,i}{U}_{0,i}{U}_{1,i})-n{ϵ}_n\hfill & & =H(S_1{S}_2|Y_2)+{\displaystyle \sum _{i=1}^n}\mathbb{P}(Q=i)\left[H(S_{1,Q}|S_{2,Q}{Y}_{1,Q}{U}_{0,Q},Q=i)-H(S_{1,Q}|S_{2,Q}{Y}_{2,Q}{U}_{0,Q},Q=i)\right]\hfill \mathrm{(59b)}& & -{\displaystyle \sum _{i=1}^n}\mathbb{P}(Q=i)H(S_{1,Q}|S_{2,Q}{Y}_{1,Q}{U}_{0,Q}{U}_{1,Q},Q=i)-n{ϵ}_n\hfill & & =H(S_1{S}_2|Y_2)+H(S_{1,Q}|S_{2,Q}{Y}_{1,Q}{U}_{0,Q}Q)-H(S_{1,Q}|S_{2,Q}{Y}_{2,Q}{U}_{0,Q}Q)\hfill \mathrm{(59c)}& & -H(S_{1,Q}|S_{2,Q}{Y}_{1,Q}{U}_{0,Q}{U}_{1,Q}Q)-n{ϵ}_n\hfill & & \stackrel{(a)}{=}H(S_1{S}_2|Y_2)+H(S_1|S_2{Y}_1{U}_{0,Q}Q)-H(S_1|S_2{Y}_2{U}_{0,Q}Q)\hfill \mathrm{(59d)}& & -H(S_1|S_2{Y}_1{U}_{0,Q}{U}_{1,Q}Q)-n{ϵ}_n\hfill \end{array}$$

where $(a)$ in (59) is a consequence of that all sources $(S_1^n,S_2^n,Y_1^n,Y_2^n)$ are memoryless.

Let us now define $U_1\triangleq (Q,U_{1,Q})$ and $U_0\triangleq (Q,U_{0,Q})$, we obtain

$$R_0+R_1+R_2\ge H(S_1{S}_2|Y_2)+H(S_1|S_2{Y}_1{U}_0)-H(S_1|S_2{Y}_2{U}_0)-H(S_1|S_2{Y}_1{U}_0{U}_1).$$

(60)

The two other rate constraints can be written in a similar fashion,

$$\begin{array}{ccc}\hfill R_0+R_1& \ge \hfill & H(S_2{S}_1|Y_1)-H(S_1|U_0{U}_1{Y}_1{S}_2)\end{array}$$

(61a)

$$\begin{array}{ccc}\hfill R_0+R_2& \ge \hfill & H(S_1{S}_2|Y_2)-H(S_1|U_0{Y}_2{S}_2);\end{array}$$

(61b)

and this completes the proof of converse.  ☐

6.2. Proof of Direct Part

We first show that the rate-distortion region of the proposition that will follow is achievable. The achievability of the rate-distortion region of Theorem 1 follows by choosing then the random variable $V_0$ of the proposition as $V_0=(U_0,S_2)$.

Proposition 1.

An inner bound on the rate-distortion region of the Gray–Wyner model with side information and degraded reconstruction sets of Figure 2 is given by the set of all rate-distortion quadruples $(R_0,R_1,R_2,D_1)$ that satisfy

$$\begin{array}{cc}\hfill R_0+R_1& \ge I(V_0{U}_1;S_1{S}_2|Y_1)\hfill \end{array}$$

(62a)

$$\begin{array}{cc}\hfill R_0+R_2& \ge I(V_0;S_1{S}_2|Y_2)\hfill \end{array}$$

(62b)

$$\begin{array}{cc}\hfill R_0+R_1+R_2& \ge max\{I(V_0;S_1{S}_2|Y_1),I(V_0;S_1{S}_2|Y_2)\}+I(U_1;S_1{S}_2|V_0{Y}_1)\hfill \end{array}$$

(62c)

for some choice of the random variables $(V_0,U_1)$ such that $(V_0,U_1)\sout{○}(S_1,S_2)\sout{○}(Y_1,Y_2)$ and there exist functions $g_1$, $g_{2,1}$, and $g_{2,2}$ such that:

$$\begin{array}{cc}\hfill {\widehat{S}}_1& =g_1(V_0,U_1,Y_1)\hfill \end{array}$$

(63a)

$$\begin{array}{cc}\hfill S_2& =g_{2,1}(V_0,U_1,Y_1)\hfill \end{array}$$

(63b)

$$\begin{array}{cc}\hfill S_2& =g_{2,2}(V_0,Y_2),\hfill \end{array}$$

(63c)

and

$$\mathbb{E}{d}_1(S_1;{\widehat{S}}_1)\le D_1.$$

(64)

Proof of Proposition 1.

We now describe a coding scheme that achieves the rate-distortion region of Proposition 1. The scheme is very similar to one that is developed by Shayevitz and Wigger (Theorem 2, [4]) for a Gray–Wyner model with side information. In particular, similar to (Theorem 2, [4]), it uses a double-binning technique for the common codebook, one that is relevant for Receiver 1 and one that is relevant for Receiver 2. Note, however, that, formally, the result of Proposition 1 cannot be obtained by readily applying (Theorem 2, [4]) as is; and one needs to extend the result of (Theorem 2, [4]) in a manner that accounts for that the source component $S_2^n$ is to be recovered losslessly by both decoders. This can be obtained by extending the distortion measure of (Theorem 2, [4]) to one that is vector-valued, i.e., $d\left((s_1,s_2),({\widehat{s}}_1,{\widehat{s}}_2)\right)=\left(d_1(s_1,{\widehat{s}}_1),d_H(s_2,{\widehat{s}}_2)\right)$, where $d_H(·,·)$ denotes the Hamming distance. For reasons of completeness, we provide here a proof of Proposition 1.

Our scheme has the following parameters: a conditional joint pmf $P_{V_0{U}_1|S_1{S}_2}$ that satisfies (63) and (64), and non-negative communication rates $T_0$, $T_1$, $T_{0,0}$, $T_{0,p}$, $T_{1,0}$, $T_{1,1}$, ${\tilde{R}}_{0,0}$, ${\tilde{R}}_{0,1}$, ${\tilde{R}}_{0,2}$, ${\tilde{R}}_{1,0}$ and ${\tilde{R}}_{1,1}$ such that

$$\begin{array}{cc}\hfill T_0& =T_{0,0}+T_{0,p},0\le {\tilde{R}}_{0,0}\le T_{0,0},0\le {\tilde{R}}_{0,1}\le T_{0,p},0\le {\tilde{R}}_{0,2}\le T_{0,p}\hfill \end{array}$$

(65a)

$$\begin{array}{cc}\hfill T_1& =T_{1,0}+T_{1,1},0\le {\tilde{R}}_{1,0}\le T_{1,0},0\le {\tilde{R}}_{1,1}\le T_{1,1}.\hfill \end{array}$$

(65b)

6.2.1. Codebook Generation

(1)

Randomly and independently generate $2^{n{T}_0}$ length-n codewords $v_0^n(k_0)$ indexed with the pair of indices $k_0=(k_{0,0},k_{0,p})$, where $k_{0,0}\in [1:2^{n{T}_{0,0}}]$ and $k_{0,p}\in [1:2^{n{T}_{0,p}}]$. Each codeword $v_0^n(k_0)$ has i.i.d. entries drawn according to ${\displaystyle \prod _{i=1}^n{P}_{V_0}(v_{0,i}(k_0))}$. The codewords $\{v_0^n(k_0)\}$ are partitioned into superbins whose indices will be relevant for both receivers; and each superbin is partitioned int two different ways, each into subbins whose indices will be relevant for a distinct receiver (i.e., double-binning). This is obtained by partitioning the indices $\{(k_{0,0},k_{0,p})\}$ as follows. We partition the $2^{n{T}_{0,0}}$ indices $\{k_{0,0}\}$ into $2^{n{\tilde{R}}_{0,0}}$ bins by randomly and independently assigning each index $k_{0,0}$ to an index ${\tilde{w}}_{0,0}(k_{0,0})$ according to a uniform pmf over $[1:2^{n{\tilde{R}}_{0,0}}]$. We refer to each subset of indices $\{k_{0,0}\}$ with the same index ${\tilde{w}}_{0,0}$ as a bin ${\mathcal{B}}_{00}({\tilde{w}}_{0,0})$, ${\tilde{w}}_{0,0}\in [1:2^{n{\tilde{R}}_{0,0}}]$. In addition, we make two distinct partitions of the $2^{n{T}_{0,p}}$ indices $\{k_{0,p}\}$, each relevant for a distinct receiver. In the first partition, which is relevant for Receiver 1, the indices $\{k_{0,p}\}$ are assigned randomly and independently each to an index ${\tilde{w}}_{0,1}(k_{0,p})$ according to a uniform pmf over $[1:2^{n{\tilde{R}}_{0,1}}]$. We refer to each subset of indices $\{k_{0,p}\}$ with the same index ${\tilde{w}}_{0,1}$ as a bin ${\mathcal{B}}_{01}({\tilde{w}}_{0,1})$, ${\tilde{w}}_{0,1}\in [1:2^{n{\tilde{R}}_{0,1}}]$. Similarly, in the second partition, which is relevant for Receiver 2, the indices $\{k_{0,p}\}$ are assigned randomly and independently each to an index ${\tilde{w}}_{0,2}(k_{0,p})$ according to a uniform pmf over $[1:2^{n{\tilde{R}}_{0,2}}]$; and refer to each subset of indices $\{k_{0,p}\}$ with the same index ${\tilde{w}}_{0,2}$ as a bin ${\mathcal{B}}_{02}({\tilde{w}}_{0,2})$, ${\tilde{w}}_{0,2}\in [1:2^{n{\tilde{R}}_{0,2}}]$.

(2)

For each $k_0\in [1:2^{n{T}_0}]$, randomly and independently generate $2^{n{T}_1}$ length-n codewords $u_1^n(k_1,k_0)$ indexed with the pair of indices $k_1=(k_{1,0},k_{1,1})$, where $k_{1,0}\in [1:2^{n{T}_{1,0}}]$ and $k_{1,1}\in [1:2^{n{T}_{1,1}}]$. Each codeword $u_1^n(k_1,k_0)$ is with i.i.d. elements drawn according to ${\displaystyle \prod _{i=1}^n{P}_{U_1|V_0}(u_{1,i}(k_1,k_0)|v_{0,i}(k_0))}$. We partition the $2^{n{T}_{1,0}}$ indices $\{k_{1,0}\}$ into $2^{n{\tilde{R}}_{1,0}}$ bins by randomly and independently assigning each index $k_{1,0}$ to an index ${\tilde{w}}_{1,0}(k_{1,0})$ according to a uniform pmf over $[1:2^{n{\tilde{R}}_{1,0}}]$. We refer to each subset of indices $\{k_{1,0}\}$ with the same index ${\tilde{w}}_{1,0}$ as a bin ${\mathcal{B}}_{10}({\tilde{w}}_{1,0})$, ${\tilde{w}}_{1,0}\in [1:2^{n{\tilde{R}}_{1,0}}]$. Similarly, we partition the $2^{n{T}_{1,1}}$ indices $\{k_{1,1}\}$ into $2^{n{\tilde{R}}_{1,1}}$ bins by randomly and independently assigning each index $k_{1,1}$ to an index ${\tilde{w}}_{1,1}(k_{1,1})$ according to a uniform pmf over $[1:2^{n{\tilde{R}}_{1,1}}]$; and refer to each subset of indices $\{k_{1,1}\}$ with the same index ${\tilde{w}}_{1,1}$ as a bin ${\mathcal{B}}_{11}({\tilde{w}}_{1,1})$, ${\tilde{w}}_{1,1}\in [1:2^{n{\tilde{R}}_{1,1}}]$.

(3)

Reveal all codebooks and their partitions to the encoder, the codebook of $\{v_0^n(k_0)\}$ and its partitions to both receivers, and the codebook of $\{u_1^n(k_1,k_0)\}$ and its partitions to only Receiver 1.

6.2.2 Encoding

Upon observing the source pair $(S_1^n,S_2^n)=(s_1^n,s_2^n)$, the encoder finds an index $k_0=(k_{0,0},k_{0,p})$ such that the codeword $v_0^n(k_0)$ is jointly typical with $(s_1^n,s_2^n)$, i.e.,

$$\left(s_1^n,s_2^n,v_0^n(k_0)\right)\in {\mathcal{T}}_{[S_1{S}_2{V}_0]}^{(n)}.$$

(66)

By the covering lemma (Chapter 3, [16]), the encoding in this step is successful as long as n is large and

$$T_0\ge I(V_0;S_1{S}_2).$$

(67)

Next, it finds an index $k_1=(k_{1,0},k_{1,1})$ such that the codeword $u_1^n(k_1,k_0)$ is jointly typical with the triple $(s_1^n,s_2^n,v_0^n(k_0))$, i.e.,

$$\left(s_1^n,s_2^n,v_0^n(k_0),u_1^n(k_1,k_0)\right)\in {\mathcal{T}}_{[S_1{S}_2{V}_0{U}_1]}^{(n)}.$$

(68)

Again, by the covering lemma (Chapter 3, [16]), the encoding in this step is successful as long as n is large and

$$T_1\ge I(U_1;S_1{S}_2|V_0).$$

(69)

Let ${\tilde{w}}_{0,0}$, ${\tilde{w}}_{0,1}$ and ${\tilde{w}}_{0,2}$ be the bin indices such that $k_{0,0}\in {\mathcal{B}}_{00}({\tilde{w}}_{0,0})$, $k_{0,p}\in {\mathcal{B}}_{01}({\tilde{w}}_{0,1})$ and $k_{0,p}\in {\mathcal{B}}_{02}({\tilde{w}}_{0,2})$. In addition, let ${\tilde{w}}_{1,0}$ and ${\tilde{w}}_{1,1}$ be the bin indices such that $k_{1,0}\in {\mathcal{B}}_{10}({\tilde{w}}_{1,0})$ and $k_{1,1}\in {\mathcal{B}}_{11}({\tilde{w}}_{1,1})$. The encoder then sends the product message $W_0=({\tilde{w}}_{0,0},{\tilde{w}}_{1,0})$ over the error-free rate-limited common link of capacity $R_0$. In addition, it sends the product message $W_1=({\tilde{w}}_{0,1},{\tilde{w}}_{1,1})$ over the error-free rate-limited individual link to Receiver 1 of capacity $R_1$, and the message $W_2={\tilde{w}}_{0,2}$ over the error-free rate-limited individual link to Receiver 2 of capacity $R_2$.

6.2.3 Decoding

Receiver 1 gets the messages $(W_0,W_1)=({\tilde{w}}_{0,0},{\tilde{w}}_{1,0},{\tilde{w}}_{0,1},{\tilde{w}}_{1,1})$. It seeks a codeword $v_0^n(k_0)$ and a codeword $u_1^n(k_1,k_0)$, with the indices $k_0=(k_{0,0},k_{0,p})$ and $k_1=(k_{1,0},k_{1,1})$ satisfying $k_{0,0}\in {\mathcal{B}}_{00}({\tilde{w}}_{0,0})$, $k_{0,p}\in {\mathcal{B}}_{01}({\tilde{w}}_{0,1})$, $k_{1,0}\in {\mathcal{B}}_{10}({\tilde{w}}_{1,0})$ and $k_{1,1}\in {\mathcal{B}}_{11}({\tilde{w}}_{1,1})$, and such that

$$\left(v_0^n(k_0),u_1^n(k_1,k_0),y_1^n\right)\in {\mathcal{T}}_{[V_0{U}_1{Y}_1]}^{(n)}.$$

(70)

By the multivariate packing lemma (Chapter 12, [16]), the error in this decoding step at Receiver 1 vanishes exponentially as long as n is large and

$$\begin{array}{cc}\hfill T_{0,0}-{\tilde{R}}_{0,0}+T_{0,p}-{\tilde{R}}_{0,1}& \le I(V_0;Y_1)\hfill \end{array}$$

(71a)

$$\begin{array}{cc}\hfill T_{1,0}-{\tilde{R}}_{1,0}+T_{1,1}-{\tilde{R}}_{1,1}& \le I(U_1;Y_1|V_0).\hfill \end{array}$$

(71b)

Receiver 1 then sets its reproduced codewords ${\widehat{s}}_{2,1}^n$ and ${\widehat{s}}_1^n$ , respectively, as

$$\begin{array}{cc}\hfill {\widehat{s}}_{2,1}^n& =g_{2,1}\left(v_0^n(k_0),u_1^n(k_1,k_0),y_1^n\right)\hfill \end{array}$$

(72a)

$$\begin{array}{cc}\hfill {\widehat{s}}_1^n& =g_1\left(v_0^n(k_0),u_1^n(k_1,k_0),y_1^n\right).\hfill \end{array}$$

(72b)

Similary, Receiver 2 gets the message $(W_0,W_2)=({\tilde{w}}_{0,0},{\tilde{w}}_{1,0},{\tilde{w}}_{0,2})$. It seeks a codeword $v_0^n(k_0)$, with $k_0=(k_{0,0},k_{0,p})$ satisfying $k_{0,0}\in {\mathcal{B}}_{00}({\tilde{w}}_{0,0})$ and $k_{0,p}\in {\mathcal{B}}_{02}({\tilde{w}}_{0,2})$, and such that

$$\left(v_0^n(k_0),y_1^n\right)\in {\mathcal{T}}_{[V_0{Y}_2]}^{(n)}.$$

(73)

Again, using the multivariate packing lemma (Chapter 12, [16]), the error in this decoding step at Receiver 2 vanishes exponentially as long as n is large and

$$T_{0,0}-{\tilde{R}}_{0,0}+T_{0,p}-{\tilde{R}}_{0,2}\le I(V_0;Y_2).$$

(74)

Receiver 2 then sets its reconstructed codeword ${\widehat{s}}_{2,1}^n$ as

$${\widehat{s}}_{2,2}^n=g_{2,2}\left(v_0^n(k_0),y_2^n\right).$$

(75)

Summarizing, combining Equations (67), (69), (71) and (74), the communication rates $T_0$, $T_1$, $T_{0,0}$, $T_{0,p}$, $T_{1,0}$, $T_{1,1}$, ${\tilde{R}}_{0,0}$, ${\tilde{R}}_{0,1}$, ${\tilde{R}}_{0,2}$, ${\tilde{R}}_{1,0}$ and ${\tilde{R}}_{1,1}$ satisfy the following inequalities

$$\begin{array}{cc}\hfill T_0& \ge I(V_0;S_1{S}_2)\hfill \end{array}$$

(76a)

$$\begin{array}{cc}\hfill T_1& \ge I(U_1;S_1{S}_2|V_0)\hfill \end{array}$$

(76b)

$$\begin{array}{cc}\hfill T_{0,0}-{\tilde{R}}_{0,0}+T_{0,p}-{\tilde{R}}_{0,1}& \le I(V_0;Y_1)\hfill \end{array}$$

(76c)

$$\begin{array}{cc}\hfill T_{0,0}-{\tilde{R}}_{0,0}+T_{0,p}-{\tilde{R}}_{0,2}& \le I(V_0;Y_2)\hfill \end{array}$$

(76d)

$$\begin{array}{cc}\hfill T_{1,0}-{\tilde{R}}_{1,0}+T_{1,1}-{\tilde{R}}_{1,1}& \le I(U_1;Y_1|V_0).\hfill \end{array}$$

(76e)

Choosing ${\tilde{R}}_{0,0}$, ${\tilde{R}}_{1,1}$, ${\tilde{R}}_{0,2}$, ${\tilde{R}}_{1,0}$ and ${\tilde{R}}_{1,1}$ to also satisfy the rate relations

$$\begin{array}{cc}\hfill R_0& ={\tilde{R}}_{0,0}+{\tilde{R}}_{1,0}\hfill \end{array}$$

(77a)

$$\begin{array}{cc}\hfill R_1& ={\tilde{R}}_{0,1}+{\tilde{R}}_{1,1}\hfill \end{array}$$

(77b)

$$\begin{array}{cc}\hfill R_2& ={\tilde{R}}_{0,2}.\hfill \end{array}$$

(77c)

and, finally, using Fourier-Motzkin elimination (FME) to successively project out the nuisance variables $T_{0,0}$, $T_{0,p}$, $T_{1,0}$, $T_{1,1}$, $T_0$, $T_1$, and then ${\tilde{R}}_{0,0}$, ${\tilde{R}}_{0,1}$, ${\tilde{R}}_{0,2}$, ${\tilde{R}}_{1,0}$ and ${\tilde{R}}_{1,1}$ from the set of relations formed by (65), (76) and (77), we get the region of Proposition 1.

This completes the proof of the proposition; and so that of the direct part of Theorem 1. ☐