Rate-Distortion Analysis of Distributed Indirect Source Coding

Abstract

Motivated by task-oriented semantic communication and distributed learning systems, this paper studies a distributed indirect source coding problem where M correlated sources are independently encoded for a central decoder. The decoder has access to correlated side information in addition to the messages received from the encoders and aims to recover a latent random variable under a given distortion constraint rather than recovering the sources themselves. We characterize the exact rate-distortion function for the case where the sources are conditionally independent given the side information. Furthermore, we develop a distributed Blahut–Arimoto (BA) algorithm to numerically compute the rate-distortion function. Numerical examples are provided to demonstrate the effectiveness of the proposed approach in calculating the rate-distortion region.

1. Introduction

Consider the multiterminal source coding setup as shown in Figure 1. Let $(T,X_1,\dots ,X_M,Y)$$\sim p(t,x_1,\dots ,x_M,y)$ be a discrete memoryless source (DMS) taking values in the finite alphabets $\mathcal{T}\times {\mathcal{X}}_1\times \cdots \times {\mathcal{X}}_M\times \mathcal{Y}$ according to a fixed and known probability distribution $p(t,x_1,\dots ,x_M,y)$. In this setup, the encoder $m,m\in \mathcal{M}:=\{1,\dots ,M\}$ has local observations $X_m^n:=(X_m^1,\dots ,X_m^n)$. The agents independently encode their observations into binary sequences at rates $\{R_1,\dots ,R_M\}$ bits per input symbol, respectively. The decoder with side information $Y^n=(Y_1,\dots ,Y_n)$ aims to recover some task-oriented latent information $T^n:=(T_1,\dots ,T_n)$ which is correlated with $(X_1^n,\dots ,X_M^n)$, but it is not observed directly by any of the encoders. We are interested in the lossy reconstruction of $T^n$ with the average distortion measured by $\mathbb{E}\left[{\displaystyle \frac{1}{n}}{\sum }_{i=1}^{n}d(T_i,{\widehat{T}}_i)\right]$ for some prescribed single-letter distortion measure $d(·,·)$. A formal $(2^{n{R}_1},\dots ,2^{n{R}_M},n)$ rate-distortion code for this setup consists of the following:

Figure 1. Distributed remote compression of a latent variable with M correlated sources at distributed transmitters and side information at the receiver.

• M independent encoders, where encoder $m\in \mathcal{M}$ assigns an index $s_m\left(x_m^n\right)\in \left\{1,\dots ,2^{n{R}_m}\right\}$ to each sequence $x_m^n\in {\mathcal{X}}_m^n$;
• A decoder that produces the estimate ${\widehat{t}}^n(s_1,\dots ,s_M,y^n)\in {\mathcal{T}}^n$ to each index tuple $(s_1,\dots ,s_M)$ and side information $y^n\in {\mathcal{Y}}^n$.

A rate tuple $(R_1,\dots ,R_M)$ is said to be achievable with the distortion measure $d(·,·)$ and the distortion value D if there exists a sequence of $(2^{n{R}_1},\dots ,2^{n{R}_M},n)$ codes that satisfy

$$\begin{array}{c}\hfill \underset{n\to \infty }{limsup}\mathbb{E}\left[{\displaystyle \frac{1}{n}}\sum _{i=1}^{n}d(T_i,{\widehat{T}}_i)\right]\le D.\end{array}$$

(1)

The rate-distortion region $R_{X_1,\dots ,X_m|Y}^{*}\left(D\right)$ for this distributed source coding problem is the closure of the set of all achievable rate tuples $(R_1,\dots ,R_M)$ that permit the reconstruction of the latent variable $T^n$ within the average distortion constraint D.

The problem as illustrated in Figure 1 is motivated by semantic/task-oriented communication and distributed learning problems. In semantic/task-oriented communication [1], the decoder only needs to reconstruct some task-oriented information implied by the sources. For instance, it might extract hidden features from a scene captured by multiple cameras positioned at various angles. Here, $T_i$ may also be a deterministic function of the source samples $(X_{1,i},\dots ,X_{M,i})$, which then reduces to the problem of lossy distributed function computation [2,3,4]. A similar problem also arises in distributed training. Consider $Y^n$ as the global model available at the server at an iteration of a federated learning process and $(X_1^n,\dots ,X_M^n)$ as the independent correlated versions of this model after downlink transmission and local training. The server aims to recover the updated global model, $T^n$, based on the messages received from all M clients. It is often assumed that the global model is transmitted to the clients intact, but in practical scenarios where downlink communication is limited, the clients may receive noisy or compressed versions of the global model [5,6,7].

For the case of $M=1$, the problem reduces to the remote compression in a point-to-point scenario with side information available at the decoder. In [8,9], the authors studied this problem without the correlated side information at the receiver, which is motivated in the context of semantic communication. This problem is known in the literature as the remote rate-distortion problem [10,11], and the rate-distortion trade-off is fully characterized in the general case. The authors studied this trade-off in detail for specific source distributions in [8]. Similarly, the authors of [12] characterized the remote rate-distortion trade-off when correlated side information is available both at the encoder and decoder. Our problem for $M=1$ can be solved by combining the remote rate-distortion problem with the classical Wyner–Ziv rate-distortion function [13,14].

The rate-distortion region for the multi-terminal version of the remote rate-distortion problem considered here remains open. Wagner et al. developed an improved outer bound for a general multiterminal source model [15]. Sung et al. proposed an achievable rate region for the distributed lossy computation problem without giving an conclusive rate-distortion function [16]. Gwanmo et al. considered a special case in which the sources are independent and derived a single-letter expression for the rate-distortion region [17]. Gastpar [18] considered the lossy compression of the source sequences in the presence of side information at the receiver. He characterized the rate-distortion region for the special case, in which $X_i$ values are conditionally independent given the side information.

To provide a performance reference for practical coding schemes, it is necessary not only to characterize the exact theoretical expression for the rate-distortion region but also to calculate the rate-distortion region for a given distribution and a specific distortion metric. In the traditional single-source direct scenario, determining the rate-distortion function involves solving a convex optimization problem, which can be addressed using the globally convergent iterative Blahut–Arimoto algorithm, as discussed in [17]. In this paper, we are interested in computing the rate-distortion region $R_{X_1,\dots ,X_m|Y}^{*}\left(D\right)$ for the general distributed coding problem. We pay particular attention to the special case in which the sources are conditionally independent given the side information, which is motivated by the aforementioned examples. For the sake of brevity of the presentation, we set $M=3$ in this paper with the understanding that the results can be readily extended to an arbitrary number of sources. To numerically compute the rate-distortion region, we introduce a distributed iterative optimization framework that generalizes the classical Blahut–Arimoto (BA) algorithm. While the standard BA algorithm is designed for single-source point-to-point settings, our framework extends its alternating minimization structure to a distributed scenario with multiple encoders, indirect source reconstruction, and decoder-side side information. This extension enables the computation of rate-distortion regions in settings that are significantly more general than those considered in the existing literature.

In Section 2, we derive an achievable region $R_a\left(D\right)\subseteq R_{X_1,X_2,X_3|Y}^{*}\left(D\right)$. In Section 3, we determine a general outer bound $R_o\left(D\right)\supseteq R_{X_1,X_2,X_3|Y}^{*}\left(D\right)$. In Section 4, we show that the two regions coincide and the region is optimal when the sources $(X_1,X_2,X_3)$ are conditionally independent given the side information Y. In Section 5, we develop an alternating minimization framework to calculate the rate-distortion region by generalizing the Blahut–Arimoto (BA) algorithm. In Section 6, we demonstrate the effectiveness of the proposed framework through numerical examples.

2. An Achievable Rate Region

In this section, we introduce an achievable rate region $R_a\left(D\right)$, which is contained within the goal rate-distortion region $R_a\left(D\right)\subseteq R_{X_1,X_2,X_3|Y}^{*}\left(D\right)$.

Theorem 1.

$R_a\left(D\right)\subseteq R_{X_1,X_2,X_3|Y}^{*}\left(D\right)$, where $R_a\left(D\right)$ is the set of all rate tuples $(R_1,R_2,R_3)$ such that there exists a tuples $(W_1,W_2,W_3)$ of discrete random variables with $p\left(w_1,w_2,w_3,x_1,x_2,x_3,y\right)=$$p\left(w_1|x_1\right)p\left(w_2|x_2\right)p\left(w_3|x_3\right)p\left(x_1,x_2,x_3,y\right)$, for which the following conditions are satisfied

$$\begin{array}{cc}\hfill \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em} R_1& \hfill ⩾I\left(X_1;W_1\right)-I\left(W_1;W_2,W_3,Y\right)\end{array}$$

(2a)

$$\begin{array}{cc}\hfill \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em} R_2& \hfill ⩾I\left(X_2;W_2\right)-I\left(W_2;W_1,W_3,Y\right)\end{array}$$

(2b)

$$\begin{array}{cc}\hfill \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em} R_3& ⩾I\left(X_3;W_3\right)-I\left(W_3;W_1,W_2,Y\right)\hfill \end{array}$$

(2c)

$$\begin{array}{cc}\hfill \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em} R_1+R_2& \hfill ⩾I\left(X_1;W_1\right)+I\left(X_2;W_2\right)-I\left(W_1;W_2,W_3,Y\right)\\ & -I\left(W_2;W_1,W_3,Y\right)+I\left(W_1;W_2|W_3,Y\right)\hfill \end{array}$$

(2d)

$$\begin{array}{cc}\hfill \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em} R_1+R_3& \hfill ⩾I\left(X_1;W_1\right)+I\left(X_3;W_3\right)-I\left(W_1;W_2,W_3,Y\right)\\ & -I\left(W_3;W_1,W_2,Y\right)+I\left(W_1;W_3|W_2,Y\right)\hfill \end{array}$$

(2e)

$$\begin{array}{cc}\hfill \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em} R_2+R_3& \hfill ⩾I\left(X_2;W_2\right)+I\left(X_3;W_3\right)-I\left(W_2;W_1,W_3,Y\right)\\ & -I\left(W_3;W_1,W_2,Y\right)+I\left(W_2;W_3|W_1,Y\right),\hfill \end{array}$$

(2f)

$$\begin{array}{cc}\hspace{1em}\hspace{1em} R_1+R_2+R_3\hfill & ⩾I\left(X_1;W_1\right)+I\left(X_2;W_2\right)+I\left(X_3;W_3\right)\hfill \\ & -I\left(W_1;W_2,W_3,Y\right)-I\left(W_2;W_1,W_3,Y\right)\hfill \\ & -I\left(W_3;W_1,W_2,Y\right)+I\left(W_1;W_2|W_3,Y\right)\hfill \\ & +I\left(W_1,W_2;W_3|Y\right),\hfill \end{array}$$

(2g)

and there exist a decoder $g\left(·\right)$ such that

$$\begin{array}{c}\hfill Ed(T,g(W_1,W_2,W_3,Y))⩽D.\end{array}$$

(3)

The auxiliary random variables $W_1$, $W_2$ and $W_3$ serve as intermediate variables in the encoding process, which was introduced to optimize compression efficiency. Ideally, $W_1$, $W_2$ and $W_3$ should carry information about the source $X_1$, $X_2$ and $X_3$, respectively. But this information should be as independent as possible from the side information Y in order to avoid redundancy and fully exploit the decoder-side knowledge. This helps minimize the required transmission rate R. The rigorous proof of Theorem 1 is provided in Appendix A.

Corollary 1.

The conditions (25) of Theorem 1 can be expressed equivalently as

$$\begin{array}{cc}\hfill \hspace{1em}\hspace{1em}\hspace{1em} R_1& ⩾I\left(X_1,X_2,X_3;W_1|W_2,W_3,Y\right)\hfill \end{array}$$

(4a)

$$\begin{array}{cc}\hfill \hspace{1em}\hspace{1em}\hspace{1em} R_2& ⩾I\left(X_1,X_2,X_3;W_2|W_1,W_3,Y\right)\hfill \end{array}$$

(4b)

$$\begin{array}{cc}\hfill \hspace{1em}\hspace{1em}\hspace{1em} R_3& ⩾I\left(X_1,X_2,X_3;W_3|W_1,W_2,Y\right)\hfill \end{array}$$

(4c)

$$\begin{array}{cc}\hfill \hspace{1em}\hspace{1em}{R}_1+R_2& ⩾I\left(X_1,X_2,X_3;W_1,W_2|W_3,Y\right)\hfill \end{array}$$

(4d)

$$\begin{array}{cc}\hfill \hspace{1em}\hspace{1em}{R}_1+R_3& ⩾I\left(X_1,X_2,X_3;W_1,W_3|W_2,Y\right)\hfill \end{array}$$

(4e)

$$\begin{array}{cc}\hfill \hspace{1em}\hspace{1em}{R}_2+R_3& ⩾I\left(X_1,X_2,X_3;W_2,W_3|W_1,Y\right)\hfill \end{array}$$

(4f)

$$\begin{array}{cc}\hfill R_1+R_2+R_3& ⩾I(X_1,X_2,X_3;W_1,W_2,W_3|Y)\hfill \end{array}$$

(4g)

Proof. 

First, we prove $R_1⩾I\left(X_1;W_1\right)-I\left(W_1;W_2,W_3,Y\right)=I\left(X_1,X_2,X_3;W_1|W_2,W_3,Y\right)$. The bound of (4a) can be written as

$$\begin{array}{cc}\hfill & I\left(X_1,X_2,X_3;W_1|W_2,W_3,Y\right)\hfill \\ \hfill & =I\left(X_1;W_1|W_2,W_3,Y\right)+\underset{=0}{\underbrace{I\left(X_2,X_3;W_1|X_1,W_2,W_3,Y\right)}},\hfill \end{array}$$

(5)

where $I\left(X_2,X_3;W_1|X_1,W_2,W_3,Y\right)=0$ because $(X_2,X_3,Y)$ is conditionally independent of $W_1$ for given $X_1$. For the first term of the right in (5), we have

$$\begin{array}{cc}\hfill & I\left(X_1;W_1|W_2,W_3,Y\right)+I\left(W_1;W_2,W_3,Y\right)\hfill \\ \hfill & =I\left(W_1;X_1,W_2,W_3,Y\right)\hfill \\ \hfill & =I\left(W_1;X_1\right)+\underset{=0}{\underbrace{I\left(W_2,W_3,Y;W_1|X_1\right)}},\hfill \end{array}$$

(6)

where $I\left(W_2,W_3,Y;W_1|X_1\right)=0$ because $(W_2,W_3,Y)$ is conditionally independent of $W_1$ given $X_1$. Then, we have

$$\begin{array}{cc}\hfill & I\left(X_1,X_2,X_3;W_1|W_2,W_3,Y\right)\hfill \\ \hfill & =I\left(X_1;W_1|W_2,W_3,Y\right)\hfill \\ \hfill & =I\left(W_1;X_1\right)-I\left(W_1;W_2,W_3,Y\right)\hfill \\ \hfill & \le R_1.\hfill \end{array}$$

(7)

This completes the proof of (4a). (4b) and (4c) can be proved in the same way.

Then, we prove (4d), the bound of the sum rate $R_1+R_2$ can be written as

$$\begin{array}{cc}\hfill & I\left(X_1,X_2,X_3;W_1,W_2|W_3,Y\right)\hfill \\ \hfill & =I\left(X_1;W_1,W_2|W_3,Y\right)+I\left(X_2,X_3;W_1,W_2|W_3,Y,X_1\right)\hfill \end{array}$$

(8a)

$$\begin{array}{c}\hspace{1em}\hspace{1em}=I\left(X_1;W_2|W_3,Y\right)+\underset{=I\left(X_1;W_1\right)-I\left(W_1;W_2,W_3,Y\right)\le R_1}{\underbrace{I\left(X_1;W_1|W_2,W_3,Y\right)}}\hfill \\ \hspace{1em}\hspace{1em}\underset{=0}{\underbrace{I\left(X_2,X_3;W_1|W_3,Y,X_1\right)}}+I(X_2,X_3;W_2|W_1,W_3,Y,X_1),\end{array}$$

(8b)

where (8a) and (8b) are obtained by the chain rule of mutual information, $I(X_2,X_3;W_1|W_3,$$Y,X_1)=0$, because $(X_2,X_3)$ is conditionally independent of $W_1$ given $X_1$. For the last term in (8b), we have

$$\begin{array}{c}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}I\left(X_2,X_3;W_2|W_1,W_3,Y,X_1\right)+I\left(X_1;W_2|W_1,W_3,Y\right)\hfill \end{array}$$

(9a)

$$\begin{array}{c}=I\left(X_1,X_2,X_3;W_2|W_1,W_3,Y\right)\hfill \end{array}$$

(9b)

$$\begin{array}{c}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}=I\left(X_2,X_3;W_2|W_1,W_3,Y\right)+\underset{=0}{\underbrace{I\left(X_1;W_2|W_1,W_3,Y,X_2,X_3\right)}}\hfill \end{array}$$

(9c)

$$\begin{array}{c}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}=\underset{=I\left(X_2;W_2\right)-I\left(W_2;W_1,W_3,Y\right)\le R_2}{\underbrace{I\left(X_2;W_2|W_1,W_3,Y\right)}}+\underset{=0}{\underbrace{I\left(X_1,X_3;W_2|W_1,W_3,Y,X_2\right)}}\hfill \end{array}$$

(9d)

where $I\left(X_1;W_2|W_1,W_3,Y,X_2,X_3\right)=0$ because $X_1$ is conditionally independent of $W_2$ given $X_2$, and $I\left(X_1,X_3;W_2|W_1,W_3,Y,X_2\right)=0$ because $(X_1,X_3)$ is conditionally independent of $W_2$ given $X_2$. Thus, the last term in (8b) can be written as

$$\begin{array}{cc}\hfill & I\left(X_2,X_3;W_2|W_1,W_3,Y,X_1\right)\hfill \\ \hfill =& \underset{=I\left(X_2;W_2\right)-I\left(W_2;W_1,W_3,Y\right)\le R_2}{\underbrace{I\left(X_2;W_2|W_1,W_3,Y\right)}}-I\left(X_1;W_2|W_1,W_3,Y\right)\hfill \end{array}$$

(10)

For the last term in the right-hand side, we have

$$\begin{array}{cc}\hfill & I\left(X_1;W_2|W_1,W_3,Y\right)+I\left(W_1;W_2|W_3,Y\right)\hfill \\ \hfill & =I\left(X_1,W_1;W_2|W_3,Y\right)\hfill \\ \hfill & =I\left(X_1;W_2|W_3,Y\right)+\underset{=0}{\underbrace{I\left(W_1;W_2|X_1,W_1,W_3,Y\right)}},\hfill \end{array}$$

(11)

thus, the last term in (10) can be written as

$$\begin{array}{cc}\hfill & I\left(X_1;W_2|W_1,W_3,Y\right)\hfill \\ \hfill =& I\left(X_1;W_2|W_3,Y\right)-I\left(W_1;W_2|W_3,Y\right)\hfill \end{array}$$

(12)

By combining (8), (9), (10) and (12), we have

$$\begin{array}{cc}\hfill R_1+R_2& ⩾I\left(X_1,X_2,X_3;W_1,W_2|W_3,Y\right)\hfill \\ & =I\left(X_1;W_1\right)+I\left(X_2;W_2\right)\hfill \\ \hfill & -I\left(W_1;W_2,W_3,Y\right)-I\left(W_2;W_1,W_3,Y\right)\hfill \\ \hfill & +I\left(W_1;W_2|W_3,Y\right).\hfill \end{array}$$

(13)

The rest sum rate bounds in (4) can be proved in the same way, which is omitted here. □

3. A General Outer Bound

In this section, we derive a region $R_o\left(D\right)$ which contains the goal rate-distortion region $R_o\left(D\right)\supseteq R_{X_1,X_2,X_3|Y}^{*}\left(D\right)$.

Theorem 2.

$R_o\left(D\right)\supseteq R_{X_1,X_2,X_3|Y}^{*}\left(D\right)$, where $R_o\left(D\right)$ is the set of all rate triples $(R_1,R_2,R_3)$ such that there exists a triple $(W_1,W_2,W_3)$ of discrete random variables with $p\left(w_1|x_1,x_2,x_3,y\right)=p\left(w_1|x_1\right)$, $p\left(w_2|x_1,x_2,x_3,y\right)=p\left(w_2|x_2\right)$ and $p\left(w_3|x_1,x_2,x_3,y\right)=p\left(w_3|x_3\right)$, for which the following conditions are satisfied

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill R_1& ⩾I\left(X_1,X_2,X_3;W_1|W_2,W_3,Y\right)\hfill \\ \hfill R_2& ⩾I\left(X_1,X_2,X_3;W_2|W_1,W_3,Y\right)\hfill \\ \hfill R_3& ⩾I\left(X_1,X_2,X_3;W_3|W_1,W_2,Y\right)\hfill \\ \hfill R_1+R_2& ⩾I\left(X_1,X_2,X_3;W_1,W_2|W_3,Y\right)\hfill \\ \hfill R_1+R_3& ⩾I\left(X_1,X_2,X_3;W_1,W_3|W_2,Y\right)\hfill \\ \hfill R_2+R_3& ⩾I\left(X_1,X_2,X_3;W_2,W_3|W_1,Y\right)\hfill \\ \hfill R_1+R_2+R_3& ⩾I(X_1,X_2,X_3;W_1,W_2,W_3|Y)\hfill \end{array}\end{array}$$

(14)

and there exists a decoding function $g\left(·\right)$ such that

$$\begin{array}{c}\hfill Ed(T,g_1(W_1,W_2,W_3,Y))⩽D,\end{array}$$

(15)

The rigorous proof of Theorem 2 is provided in Appendix B.

While the expressions of the inner bound (4) and the outer bound (14) are the same, these two regions do not coincide because the marginal constrains $p\left(w_1,w_2,w_3,x_1,x_2,x_3,y\right)=$$p\left(w_1|x_1\right)p\left(w_2|x_2\right)p\left(w_3|x_3\right)p\left(x_1,x_2,x_3,y\right)$ in Theorem 1 limit the degree of freedom for choosing the auxiliary random variables $(W_1,W_2,W_3)$ compared with the marginal constraints in Theorem 2. In the next section, we will demonstrate that the additional degree of freedom in choosing the auxiliary random variables $(W_1,W_2,W_3)$ in Theorem 2 cannot lower the value of the rate-distortion functions.

4. Conclusive Rate-Distortion Results

Corollary 2.

If $X_1,X_2,X_3$ are conditionally independent given the side information Y, $R_a\left(D\right)\subseteq R_{X_1,X_2,|Y}^{*}\left(D\right)$ where $R_a\left(D\right)$ is the set of all rate triples $(R_1,R_2,R_3)$ such that there exists a triple $(W_1,W_2,W_3)$ of random variables with $p\left(w_1,w_2,w_3,x_1,x_2,x_3,y\right)=p\left(w_1|x_1\right)p\left(w_2|x_2\right)p\left(w_3|x_3\right)$$p\left(x_1|y\right)p\left(x_2|y\right)p\left(x_3|y\right)p\left(y\right)$, for which the following conditions are satisfied

$$\begin{array}{cc}\hfill R_1& ⩾I\left(X_1;W_1\right)-I\left(W_1;Y\right)\hfill \\ \hfill R_2& ⩾I\left(X_2;W_2\right)-I\left(W_2;Y\right)\hfill \\ \hfill R_3& ⩾I\left(X_3;W_3\right)-I\left(W_3;Y\right)\hfill \end{array}$$

(16)

and there exist decoding functions $g\left(·\right)$ such that

$$\begin{array}{c}\hfill Ed(T,g(W_1,W_2,W_3,Y))⩽D.\end{array}$$

(17)

Proof. 

Since the joint distribution can be written as

$$\begin{array}{cc}\hfill & p\left(w_1,w_2,w_3,x_1,x_2,x_3,y\right)\hfill \\ & =p\left(w_1,w_2,w_3|x_1,x_2,x_3,y\right)p\left(x_1,x_2,x_3|y\right)p\left(y\right)\hfill \\ & =p\left(w_1|x_1\right)p\left(w_2|x_2\right)p\left(w_3|x_3\right)p\left(x_1|y\right)\hfill \\ & \times p\left(x_2|y\right)p\left(x_3|y\right)p\left(y\right),\hfill \end{array}$$

(18)

the terms $I\left(W_1;W_2|W_3,Y\right)$ in the sum rate bound (2d) are 0 because $W_2$ is conditionally independent of $W_1$. Similarly, the terms $I\left(W_1;W_3|W_2,Y\right),I\left(W_2;W_3|W_1,Y\right)$ and $I\left(W_1;W_2|W_3,Y\right)+$$I\left(W_1,W_2;W_3|Y\right)$ in the sum rate bound (2e)–(2g) are all 0. Therefore, the sum rate bound can be expressed as the combination of the side bounds and hence can be omitted. Meanwhile, the term $I\left(W_1;W_2,W_3,Y\right)$ in the side bound in (2a) can be written as

$$\begin{array}{c}\hfill I\left(W_1;W_2,W_3,Y\right)=I\left(W_1;Y\right)+\underset{=0}{\underbrace{I\left(W_2,W_3;W_1|Y\right)}}.\end{array}$$

(19)

Similarly, we have

$$\begin{array}{c}\hfill I\left(W_2;W_1,W_3,Y\right)=I\left(W_2;Y\right)+\underset{=0}{\underbrace{I\left(W_1,W_3;W_2|Y\right)}}\\ \hfill I\left(W_3;W_1,W_2,Y\right)=I\left(W_3;Y\right)+\underset{=0}{\underbrace{I\left(W_1,W_2;W_3|Y\right)}}.\end{array}$$

(20)

This completes the proof Corollary 2. □

Corollary 3.

If $X_1,X_2,X_3$ are conditionally independent given the side information Y, ${R_o}^{\prime }\left(D\right)\supseteq R_o\left(D\right)$, and hence ${R_o}^{\prime }\left(D\right)\supseteq R_{X_1,X_2,W_3|Y}^{*}\left(D\right)$ where ${R_o}^{\prime }\left(D\right)$ is the set of all rate triples $(R_1,R_2,R_3)$ such that there exists a triple $(W_1,W_2,W_3)$ of discrete random variables with $p\left(w_1|x_1,x_2,w_3,y\right)=p\left(w_1|x_1\right)$, $p\left(w_2|x_1,x_2,w_3,y\right)=p\left(w_2|x_2\right)$ and $p\left(w_3|x_1,x_2,x_3,y\right)=p\left(w_3|x_3\right)$, for which the following conditions are satisfied

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill R_1& ⩾I\left(X_1;W_1\right)-I\left(W_1;Y\right)\hfill \\ \hfill R_2& ⩾I\left(X_2;W_2\right)-I\left(W_2;Y\right)\hfill \\ \hfill R_3& ⩾I\left(X_3;W_3\right)-I\left(W_3;Y\right),\hfill \end{array}\end{array}$$

(21)

and there exists decoding functions $g\left(·\right)$ such that

$$\begin{array}{c}\hfill Ed(T,g_1(W_1,W_2,W_3,Y))⩽D,\end{array}$$

(22)

Proof. 

First, we can enlarge the region $R_o\left(D\right)$ by omitting the sum rate bound in (14). Then, the side rate bounds in (14) can be relaxed as

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill R_1& ⩾I\left(X_1,X_2,X_3;W_1|W_2,W_3,Y\right)\hfill \\ & ⩾I\left(X_1;W_1|W_2,W_3,Y\right)+I\left(X_2,X_3;W_1|W_2,W_3,Y,X_1\right)\hfill \\ & ⩾I\left(X_1;W_1|W_2,W_3,Y\right)\hfill \\ & =I\left(X_1;W_1,W_2,W_3|Y\right)-\underset{=0}{\underbrace{I\left(X_1;W_2,W_3|Y\right)}}.\hfill \end{array}\end{array}$$

(23)

According to the conditional independence relations, we have $I\left(X_1;W_2,W_3|Y\right)=0$, and then we have

$$\begin{array}{cc}\hfill R_1& ⩾I\left(X_1;W_1,W_2,W_3|Y\right)\hfill \end{array}$$

$$\begin{array}{c}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}=I\left(X_1;W_1|Y\right)+\underset{=0}{\underbrace{I\left(X_1;W_2,W_3|Y,W_1\right)}}\hfill \end{array}$$

(24a)

$$\begin{array}{c}\hfill \hspace{1em}\hspace{1em}\hspace{1em}=I\left(X_1,Y;W_1\right)-I\left(W_1;Y\right)\end{array}$$

(24b)

$$\begin{array}{c}\hfill \hspace{1em}\hspace{1em}=I\left(X_1;W_1\right)-I\left(W_1;Y\right)\end{array}$$

(24c)

where (24a) is obtained by the condition that $X_1,X_2,X_3$ are conditionally independent given the side information Y, (24b) follows from the chain rule of mutual information and (24c) is derived by the Markov chain $Y-X_1-W_1$. The same derivation can be applied to $R_2$ and $R_3$; this proves Corollary 3. □

Theorem 3.

If $X_1,X_2,X_3$ are conditionally independent given the side information Y,

$$\begin{array}{c}\hfill R_a\left(D\right)=R_o\left(D\right)=R_{X_1,X_2,X_3|Y}^{*}\left(D\right).\end{array}$$

(25)

Proof. 

We note that the only difference between $R_a\left(D\right)$ and $R_o\left(D\right)$ is the degrees of freedom when choosing the auxiliary random variables $(W_1,W_2,W_3)$, and all of the mutual information functions in (16) and (21) only depend on the marginal distribution $(X_1,W_1,Y)$, $(X_2,W_2,Y)$ and $(X_3,W_3,Y)$. Randomly choose a certain rate triple $(R_1,R_2,R_3)$ with a auxiliary random variable triple $(W_1,W_2,W_3)$ meeting the conditions of Corollary 3, and the corresponding joint distribution is given in (18). Then, we construct the auxiliary random variables $(W_1^{\prime },W_2^{\prime },W_3^{\prime })$ such that

$$\begin{array}{c}\hfill \begin{array}{c}\hfill p_{W_1^{\prime }|X_1}\left(w_1|s_1\right)=\sum _{w_2,s_2,w_3,s_3}p\left(w_1,w_2,w_3|s_1,s_2,s_3\right)p\left(s_2,s_3|s_1\right)\\ \hfill p_{W_2^{\prime }|X_2}\left(w_2|s_2\right)=\sum _{w_1,s_1,w_3,s_3}p\left(w_1,w_2,w_3|s_1,s_2,s_3\right)p\left(s_1,s_3|s_2\right)\\ \hfill p_{W_3^{\prime }|X_3}\left(w_3|s_3\right)=\sum _{w_1,s_1,w_2,s_2}p\left(w_1,w_2,w_3|s_1,s_2,s_3\right)p\left(s_1,s_2|s_3\right).\end{array}\end{array}$$

(26)

The joint distribution

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill & p\left(w_1^{\prime },w_2^{\prime },w_3^{\prime },x_1,x_2,x_3,Y\right)\hfill \\ & =p\left(w_1^{\prime }|x_1\right)p\left(w_2^{\prime }|x_2\right)p\left(w_3^{\prime }|x_3\right)p\left(x_1|y\right)p\left(x_2|y\right)p\left(x_3|y\right)p\left(y\right)\hfill \end{array}\end{array}$$

(27)

has the same marginal distributions on $(X_1,W_1,Y)$, $(X_2,W_2,Y)$ and $(X_3,W_3,Y)$. Therefore, the additional degree of freedom for choosing the auxiliary random variables $(W_1,W_2,W_3)$ in Corollary 3 cannot lower the value of rate-distortion functions. This proves Theorem 3. The arguments leading to Theorem 3 indicate that the result extends to the M sources scenario

$$\begin{array}{c}\hfill R_i\ge I(X_i;W_i)-I(W_i;Y)\mathrm{for}\mathrm{all}l\in \{1,\dots ,M\}.\end{array}$$

(28)

□

5. Iterative Optimization Framework Based on BA Algorithm

In this section, we present the iterative optimization framework for calculating the rate distortion region. Starting with the standard Lagrange multiplier method, the problem of calculating the rate-distortion region $R_{X_1,\dots ,X_m|Y}^{*}\left(D\right)$ is equivalent to minimize

$$\sum _{i\in \mathcal{M}:=\{1,\dots ,M\}}I(W_i;X_i)-I(W_i;Y)+\lambda (E\left[d(T,\widehat{T})\right]-D)$$

(29)

By the definition of mutual information, we can rewrite (29) as

$$\begin{array}{cc}\hfill & {\mathcal{L}}_{\lambda }(\mathit{Q},\mathit{q},q^{\prime })\hfill \\ & =\sum _{y,\widehat{t},x_i,w_i,i\in \mathcal{M}}p(y,x_i)q_i(w_i\mid x_i)q^{\prime }(\widehat{t}\mid y,\mathit{w})log{\displaystyle \frac{q_i(w_i\mid x_i)}{Q_i(w_i\mid y)}}\hfill \\ & +\lambda \sum _{\mathit{w},\mathit{x},t,\widehat{t},y}d(t,\widehat{t})p(t,\mathit{x},y)q^{\prime }(\widehat{t}\mid y,\mathit{w})\prod _{i\in \mathcal{M}}{q}_i(w_i\mid x_i),\hfill \end{array}$$

(30)

where $\mathit{Q},\mathit{q},q^{\prime }$ represent the distributions that need to be iteratively updated, and the vectorized notation $\mathit{Q}$ represents the conditional distribution of the auxiliary variables given Y, i.e., $[Q_i(w_i\mid y)\mid w_i\in {\mathcal{W}}_i,y\in \mathcal{Y},i\in \mathcal{M}]$, $\mathit{q}$ represents the conditional distribution of the auxiliary variables given sources $X_i$, $[q_i(w_i\mid x_i)\mid w_i\in {\mathcal{W}}_i,x_i\in {\mathcal{X}}_i,i\in \mathcal{M}]$ and $q^{\prime }$ represents the conditional distribution of the indirect variable T given Y and auxiliary variables, $q^{\prime }(\widehat{t}\mid y,w_1,\dots ,w_M)$.

Lemma 1

(Optimization of $\mathit{Q}$). For a fixed ${\mathit{Q}}_{\setminus m},\mathit{q},q^{\prime }$, the Lagrangian ${\mathcal{L}}_{\lambda }(\mathit{Q},\mathit{q},q^{\prime })$ is minimized by

$$Q_m^{*}(w_m\mid y)\triangleq {\displaystyle \frac{{\sum }_{x_m,\widehat{t}}p(y,x_m)q(w_m\mid x_m)q^{\prime }(\widehat{t}\mid y,\mathit{w})}{{\sum }_{x_m,\widehat{t},w_m}p(y,x_m)q(w_m\mid x_m)q^{\prime }(\widehat{t}\mid y,\mathit{w}),}}$$

(31)

where ${\mathit{Q}}_{\setminus m}=\left[Q_i(w_i\mid y)\mid w_i\in {\mathcal{W}}_i,y\in \mathcal{Y},i\in \mathcal{M}\setminus m\right]$.

Proof. 

For any $Q_m$

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill & {\mathcal{L}}_{\lambda }(Q_m^{*},\mathit{q},q^{\prime })-{\mathcal{L}}_{\lambda }(Q_m,\mathit{q},q^{\prime })\hfill \\ & =\sum _{y,\widehat{t},x_m,w_m}p(y,x_m)q_i(w_m\mid x_m)q^{\prime }(\widehat{t}\mid y,\mathit{w})log{\displaystyle \frac{q_m(w_m\mid x_m)}{Q_m^{*}(w_m\mid y)}}\hfill \\ & -\sum _{y,\widehat{t},x_m,w_m}p(y,x_m)q_i(w_m\mid x_m)q^{\prime }(\widehat{t}\mid y,\mathit{w})log{\displaystyle \frac{q_m(w_m\mid x_m)}{Q_m(w_m\mid y)}}\hfill \\ & =\sum _{y,\widehat{t},x_m,w_m}p(y,x_m)q_i(w_m\mid x_m)q^{\prime }(\widehat{t}\mid y,\mathit{w})log{\displaystyle \frac{Q_m(w_m\mid y)}{Q_m^{*}(w_m\mid y)}}\hfill \\ & \stackrel{\left(a\right)}{\le }\sum _{y,\widehat{t},x_m,w_m}p(y,x_m)q_i(w_m\mid x_m)q^{\prime }(\widehat{t}\mid y,\mathit{w})({\displaystyle \frac{Q_m(w_m\mid y)}{Q_m^{*}(w_m\mid y)}}-1)\hfill \\ & =0,\hfill \end{array}\end{array}$$

(32)

where (a) follows from the fact that $log(1+x)\le x$, and the equality is achieved if $Q_m^{*}=Q_m$. This completes the proof of Lemma 1. □

Lemma 2

(Optimization of $\mathit{q}$). For a fixed $\mathit{Q},{\mathit{q}}_{\setminus m},q^{\prime }$, the Lagrangian ${\mathcal{L}}_{\lambda }(\mathit{Q},\mathit{q},q^{\prime })$ is minimized by

$$q^{*}(w_m\mid x_m)={\displaystyle \frac{exp\left[{\sum }_{y}p(y\mid x_m)logQ(w_m\mid y)-\lambda {\sum }_{{\mathit{w}}_{\setminus m},{\mathit{x}}_{\setminus m},t,\widehat{t},y}d(t,\widehat{t})p(t,\mathit{x},y)q^{\prime }(\widehat{t}\mid y,\mathit{w}){\mathit{q}}_{\setminus m}({\mathit{w}}_{\setminus m}\mid {\mathit{x}}_{\setminus m})\right]}{{\sum }_{w_m}exp\left[{\sum }_{y}p(y\mid x_m)logQ(w_m\mid y)-\lambda {\sum }_{{\mathit{w}}_{\setminus m},{\mathit{x}}_{\setminus m},t,\widehat{t},y}d(t,\widehat{t})p(t,\mathit{x},y)q^{\prime }(\widehat{t}\mid y,\mathit{w}){\mathit{q}}_{\setminus m}({\mathit{w}}_{\setminus m}\mid {\mathit{x}}_{\setminus m})\right],}}$$

(33)

and the minimum is given by

$$\begin{array}{cc}\hfill & {\mathcal{L}}_{\lambda }(\mathit{Q},{\mathit{q}}_{\setminus m}^{*},q^{\prime })\hfill \\ & =\sum _{x_i,i\in \mathcal{M}}p\left(x_i\right)\underset{w_i}{min}[\sum _{y,\widehat{t}}p(y\mid x_i)q^{\prime }(\widehat{t}\mid y,\mathit{w})log{\displaystyle \frac{q^{*}(w_i\mid x_i)}{Q(w_i\mid y)}}\hfill \\ & +\lambda \sum _{\mathit{w},\mathit{x},t,\widehat{t},y}d(t,\widehat{t})p(t,\mathit{x},y)q^{\prime }(\widehat{t}\mid y,\mathit{w})\prod _{i\in \mathcal{M}}{q}_i(w_i\mid x_i)].\hfill \end{array}$$

(34)

Proof. 

For a fixed $\mathit{Q},{\mathit{q}}_{\setminus m},q^{\prime }$, the Lagrangian ${\mathcal{L}}_{\lambda }(\mathit{Q},\mathit{q},q^{\prime })$ is minimized by $q^{*}(w_m\mid x_m)$ if and only if the following Kuhn–Tucker (KT) conditions are satisfied

$${\left.{\displaystyle \frac{\partial {\mathcal{L}}_{\lambda }}{\partial q_m}}\right|}_{q_m^{*}}=\gamma ,\mathrm{if}{q}^{*}(w_m\mid x_m)>0,$$

(35)

and

$${\left.{\displaystyle \frac{\partial {\mathcal{L}}_{\lambda }}{\partial q_m}}\right|}_{q_m^{*}}\le \gamma ,\mathrm{if}{q}_m^{*}(w_m\mid x_m)=0.$$

(36)

Since

$$\begin{array}{cc}\hfill & {\displaystyle \frac{\partial {\mathcal{L}}_{\lambda }}{\partial q_m}}\hfill \\ & =\sum _{x_m,y}p(x_m,y)\left(log{\displaystyle \frac{p(w_m\mid x_m)}{p(w_m\mid y)}}+1\right)\hfill \\ & +\lambda \sum _{\mathit{w},\mathit{x},t,\widehat{t},y}d(t,\widehat{t})p(t,\mathit{x},y)q^{\prime }(\widehat{t}\mid y,\mathit{w})q_{\setminus m}({\mathit{u}}_{\setminus m}\mid {\mathit{x}}_{\setminus m}),\hfill \end{array}$$

(37)

the first KT condition (35) becomes

$$\begin{array}{cc}\hfill & \tilde{\gamma }\hfill \\ & =\sum _{x_m,y}p(x_m,y)\left(logp(w_m\mid x_m)-logp(w_m\mid y)\right)\hfill \\ & +\lambda \sum _{{\mathit{w}}_{\setminus m},{\mathit{x}}_{\setminus m},t,\widehat{t},y}d(t,\widehat{t})p(t,\mathit{x},y)q^{\prime }(\widehat{t}\mid y,\mathit{w})q_{\setminus m}({\mathit{w}}_{\setminus m}\mid {\mathit{x}}_{\setminus m}),\hfill \end{array}$$

(38)

where

$$q_{\setminus m}({\mathit{w}}_{\setminus m}\mid {\mathit{x}}_{\setminus m})=\prod _{i\in \mathcal{M}\setminus m}{q}_i(w_i\mid x_i)$$

(39)

and we have

$$\begin{array}{cc}\hfill & q(w_m\mid x_m)\hfill \\ & =exp\left({\displaystyle \frac{\tilde{\gamma }}{p\left(x_m\right)}}\right)exp(\sum _{y}p(y\mid x_m)logQ(w_m\mid y)\hfill \\ & -\lambda \sum _{{\mathit{w}}_{\setminus m},{\mathit{x}}_{\setminus m},t,\widehat{t},y}d(t,\widehat{t})p(t,\mathit{x},y)q^{\prime }(\widehat{t}\mid y,\mathit{w})q_{\setminus m}({\mathit{w}}_{\setminus m}\mid {\mathit{x}}_{\setminus m})).\hfill \end{array}$$

(40)

Then, (33) can be obtained after normalizing $q(w_m\mid x_m)$. □

Lemma 3

(Optimization of $q^{\prime }$). For a fixed $\mathit{Q},\mathit{q}$, the Lagrangian ${\mathcal{L}}_{\lambda }(\mathit{Q},\mathit{q},q^{\prime })$ is minimized by the maximum Bayes detector.

$$q^{\prime *}(\widehat{t}\mid \mathit{w},y)=\left\{\begin{array}{cc}\alpha \left(\mathit{w}\right),\hfill & \widehat{t}\in arg\underset{\widehat{t}\in \widehat{\mathcal{T}}}{min}\mathbb{E}[d(t,\widehat{t})\mid \mathit{W}=\mathit{w}],\hfill \\ 0,\hfill & \mathrm{otherwise}\hfill \end{array}\right.$$

(41)

where $\alpha \left(\mathit{w}\right)$ is selected to guarantee

$$\sum _{\widehat{t}}{q}^{\prime \ast }(\widehat{t}\mid \mathit{w},y)=1$$

(42)

and $\mathbb{E}[d(t,\widehat{t})\mid \mathit{W}=\mathit{w}]$ denotes

$$\sum _{{\mathit{w}}_{\setminus m},{\mathit{x}}_{\setminus m},t,\widehat{t},y}d(t,\widehat{t})p(t,\mathit{x},y)\prod _{i\in \mathcal{M}}{q}_i(w_i\mid x_i)$$

(43)

Proof. 

We note that the distortion term in the Lagrangian ${\mathcal{L}}_{\lambda }$ that depends on $q^{\prime }(\widehat{t}\mid y,\mathit{w})$ is the mean distortion, which can be minimized by a Bayes detector.

Based on Lemmas 1–3, we have the iterative algorithm for computing the rate-distortion region for the distributed indirect source with decoder side information. For a given $i\in \mathcal{M}$, we iteratively calculate the following (44) and (45) to alternately update $Q_i$ and $q_i$

$$Q_i^{\ell ,k_i}={\displaystyle \frac{{\sum }_{x_i,\widehat{t}}p(y,x_i)q^{(\ell ,k_i)}(w_i\mid x_i)q^{\prime }(\widehat{t}\mid y,\mathit{w})}{{\sum }_{x_i,\widehat{t},w_i}p(y,x_i)q^{(\ell ,k_i)}(w_i\mid x_i)q^{\prime }(\widehat{t}\mid y,\mathit{w})}}$$

(44)

$$q_i^{(l,k_i+1)}={\displaystyle \frac{exp\left[{\sum }_{y}p(y\mid x_i)log{Q}^{(l,k_i)}(w_i\mid y)-\lambda {\sum }_{{\mathit{w}}_{\setminus i},{\mathit{x}}_{\setminus i},t,\widehat{t},y}d(t,\widehat{t})p(t,\mathit{x},y)q^{\prime (\ell ,i)}(\widehat{t}\mid y,\mathit{w}){\mathit{q}}_{\setminus i}^{(\ell ,i)}({\mathit{w}}_{\setminus i}\mid {\mathit{x}}_{\setminus i})\right]}{{\sum }_{w_i}exp\left[{\sum }_{y}p(y\mid x_i)log{Q}^{(\ell ,k_i)}(w_i\mid y)-\lambda {\sum }_{{\mathit{w}}_{\setminus i},{\mathit{x}}_{\setminus i},t,\widehat{t},y}d(t,\widehat{t})p(t,\mathit{x},y)q^{\prime (\ell ,i)}(\widehat{t}\mid y,\mathit{w}){\mathit{q}}_{\setminus i}^{(\ell ,i)}({\mathit{w}}_{\setminus i}\mid {\mathit{x}}_{\setminus i})\right]}}$$

(45)

until the Lagrangian ${\mathcal{L}}_{\lambda }$ converges, and the associated $q_i$ is

$$q_i^{(\ell ,*)}(w_i\mid x_i)=\underset{k_i\to \infty }{lim}{q}_i^{(\ell ,k_i)}(w_i\mid x_i)=:q_i^{(\ell +1,1)}.$$

(46)

Then, we update $q^{\prime }(\widehat{t}\mid \mathit{w},y)$ according to

$$q^{\prime {\ell }^{\prime },i^{\prime }}=\left\{\begin{array}{cc}{\alpha }^{{\ell }^{\prime },i^{\prime }}\left(\mathit{w}\right),\hfill & \widehat{t}\in arg\underset{\widehat{t}\in \widehat{\mathcal{T}}}{min}{\mathit{J}}^{(\ell ,i)}(\widehat{t},\mathit{w}),\hfill \\ 0,\hfill & \mathrm{otherwise}\hfill \end{array}\right.$$

(47)

with $i^{\prime }=i+1$ and ${\ell }^{\prime }=\ell$ if $i<M$ and $i^{\prime }=1$ and ${\ell }^{\prime }=\ell +1$ if $i=M$, and where ${\alpha }^{{\ell }^{\prime },i^{\prime }}\left(\mathit{w}\right)$ is selected to guarantee

$$\sum _{\widehat{t}}{q}^{\prime ({\ell }^{\prime },i^{\prime })}(\widehat{t}\mid \mathit{w},y)=1$$

(48)

and

$$\begin{array}{cc}\hfill {\mathit{J}}^{(\ell ,i)}(\widehat{t},\mathit{w})& =\sum _{\mathit{x},t,y}d(t,\widehat{t})p(t,\mathit{x},y)\hfill \\ & \times q_{\setminus i}^{(\ell ,i)}(w_{\setminus i}\mid x_{\setminus i})q_i^{(\ell ,*)}(w_i\mid x_i).\hfill \end{array}$$

(49)

□

Next, we repeat the process for the next user, i.e., $(i\leftarrow i+1\mathrm{if}i<M\mathrm{and}i\leftarrow 1,\ell \leftarrow \ell +1\mathrm{if}i=M)$.

Convergence analysis:

The algorithm employs an alternating minimization approach that produces Lagrangian values that are monotonically non-increasing and bounded below, thereby generating a convergent sequence of Lagrangians. Since the rate component of the Lagrangian is convex, and the sum of convex functions remains convex, the Lagrangian will also be convex if the expected distortion is a convex function. As a result, the proposed iterative optimization framework is capable of achieving the global minimum [19].

$$\sum _{\mathit{w},\mathit{x},t,\widehat{t},y}d(t,\widehat{t})p(t,\mathit{x},y)q^{\prime }(\widehat{t}\mid y,\mathit{w})\prod _{i\in \mathcal{M}}{q}_i(w_i\mid x_i)$$

(50)

We note that the expected distortion (50) includes a product of variables in the optimization process, it is not a linear function, and the Lagrangian may exhibit non-convex behavior. Even when the problem is non-convex, the authors in [20] demonstrate that the BA-based iterative algorithm initialized randomly and followed by selecting the minimum Lagrangian among all converged values can still provide highly effective information-theoretic inner bounds for the rate-distortion region, serving as a benchmark for practical quantization schemes.

6. Numerical Examples

In this section, we provide an example to illustrate the proposed iterative algorithms for computing the rate-distortion region of a distributed source coding problem with decoder-side information. As in the problem considered in this paper, distributed edge devices compress their observations $\{X_1,\dots ,X_M\}$ and transmit them to a central server (CEO). The central server then aims to recover the indirect information T from the received data, utilizing side information Y. For the convenience of demonstration, we consider a simple case where $M=2$ and the sources are binary, i.e., ${\mathcal{X}}_1={\mathcal{X}}_2=\mathcal{Y}=\{0,1\}$. The joint distributions, denoted by $Q(x_1,y)$ and $Q(x_2,y)$, are given by

$$\begin{array}{c}\hfill Q(x_1,y)={\displaystyle \frac{(1-p_1)}{2}}{\delta }_{x_1,y}+{\displaystyle \frac{p_1}{2}}(1-{\delta }_{x_1,y}),\\ \hfill Q(x_2,y)={\displaystyle \frac{(1-p_2)}{2}}{\delta }_{x_2,y}+{\displaystyle \frac{p_2}{2}}(1-{\delta }_{x_2,y}),\end{array}$$

(51)

where the Kronecker delta function ${\delta }_{x,y}$ equals 1 when x = y and 0 otherwise. We can consider Y as the input to two different binary symmetric channels (BSCs) with crossover probabilities $p_1,p_2$, respectively, where $0\le p_i\le {\displaystyle \frac{1}{2}},i\in \{1,2\}$. The corresponding outputs of these channels are $X_1$ and $X_2$. In this example, we set $p_1=p_2=0.3$.

We also assume that the information of interest is directly the combination of the two distributed sources, i.e., $T=\{X_1,X_2\}$. The distortion measure is given by $d(t,\widehat{t})=d(x_1,{\widehat{x}}_1)+d(x_2,{\widehat{x}}_2)$, where

$$d(x_i,{\widehat{x}}_i)=\left\{\begin{array}{cc}0,\hfill & x_i={\widehat{x}}_i,\hfill \\ 1,\hfill & x_i\ne {\widehat{x}}_i.\hfill \end{array}\right.$$

(52)

By applying the proposed iterative optimization framework, we can obtain the optimal transition probability distribution $Q_i^{\ast }(w_i\mid y),q_i^{\ast }(w_i\mid x_i)$ and $q^{\prime \ast }(\widehat{t}\mid \mathit{w},y)$ that meets a given distortion constraint D on $d(t,\widehat{t})$, and the corresponding minimum rate $R_i$ can be calculated by

$$\begin{array}{c}\hfill R_i=\sum _{y,x_i,w_i}p(y,x_i)q_i^{*}(w_i\mid x_i)log{\displaystyle \frac{q_i^{*}(w_i\mid x_i)}{Q_i^{*}(w_i\mid y)}}\end{array}$$

(53)

The contour plot of the rate-distortion region for this scenario is presented in Figure 2, while Figure 3 displays a surface plot depicting the rate-distortion region. We note that when $M=1$, the considered problem reduces to the traditional point-to-point Wyner–Ziv problem. In Figure 4, we compare the rate-distortion results computed using the proposed approach with the theoretical analysis by Wyner et al. [13]. We observe that the two rate-distortion function curves coincide, demonstrating the effectiveness of the proposed iterative approach for calculating rate distortion.

Figure 2. Contour plot of the rate distortion region with two distributed binary sources $\{X_1,X_2\}$, where the labels on the contours represent the distortion values D on $d(t,\widehat{t})$.

Figure 3. Surface plot of the rate-distortion region.

Figure 4. The rate-distortion function for the case when $M=1$, i.e., the Wyner–Ziv problem.

7. Conclusions

This paper explored a variant of the rate-distortion problem motivated by semantic communication and distributed learning systems, where correlated sources are independently encoded for a central decoder to reconstruct the indirect source of interest. In addition to receiving messages from the encoders, the decoder has access to correlated side information and aims to reconstruct the indirect source under a specified distortion constraint. We derived the exact rate-distortion function for the case where the sources are conditionally independent given the side information. Furthermore, we introduced a distributed iterative optimization framework based on the Blahut–Arimoto (BA) algorithm to numerically compute the rate-distortion function. A numerical example has been provided to demonstrate the effectiveness of the proposed approach.