Rate–Distortion–Perception Trade-Off in Information Theory, Generative Models, and Intelligent Communications

Abstract

Traditional rate–distortion (RD) theory examines the trade-off between the average length of the compressed representation of a source and the additive distortions of its reconstruction. The rate–distortion–perception (RDP) framework, which integrates the perceptual dimension into the RD paradigm, has garnered significant attention due to recent advancements in machine learning, where perceptual fidelity is assessed by the divergence between input and reconstruction distributions. In communication systems where downstream tasks involve generative modeling, high perceptual fidelity is essential, despite distortion constraints. However, while zero distortion implies perfect realism, the converse is not true, highlighting an imbalance in the significance of distortion and perceptual constraints. This article clarifies that incorporating perceptual constraints does not decrease the necessary rate; instead, under certain conditions, additional rate is required, even with the aid of common and private randomness, which are key elements in generative models. Consequently, we project an increase in expected traffic in intelligent communication networks with the consideration of perceptual quality. Nevertheless, a modest increase in rate can enable generative models to significantly enhance the perceptual quality of reconstructions. By exploring the synergies between generative modeling and communication through the lens of information-theoretic results, this article demonstrates the benefits of intelligent communication systems and advocates for the application of the RDP framework in advancing compression and semantic communication research.

1. Introduction

For decades, rate–distortion (RD) theory has served as the de facto standard for evaluating real-world applications involving compression and communication. With the advances in artificial intelligence technologies, the dimension of perceptual fidelity has been increasingly recognized as a crucial aspect. While traditional distortion measures like mean squared error (MSE) have served well in many applications, they fail to capture the perceptual quality that humans actually experience [1,2,3]. This disconnect has motivated the integration of perceptual measures into RD, leading to the emergence of the rate–distortion–perception (RDP) framework [4]. The information RDP function incorporates an additional perceptual constraint measured by the divergence between the input and reconstruction distributions within the context of lossy compression using machine learning. Unlike conventional distortion measures, perceptual fidelity captures the global properties of the entire coding block rather than focusing on individual sample positions, giving rise to new coding theorems. Consequently, RD plots, which have long been used to evaluate practical source coding algorithms, should be updated to incorporate this new dimension in alignment with the emerging coding theorems (see Figure 1b,c for a comparison of the RDP frontier with the RD curves). This development has also inspired a variety of innovative source coding methods that rely on generative models [5,6,7,8,9,10,11]. As emerging techniques are increasingly applied to lossy source coding and communication systems in general, the RDP trade-off offers a novel perspective for evaluating the performance of machine-learning-aided systems. Recent advances in AI-empowered communication have reimagined communication systems as a joint optimization of transmitter and receiver through end-to-end training for reconstruction tasks [12,13]. Therefore, in communication systems where downstream tasks demand high perceptual fidelity, such as interactive AI agents and the reconstruction of visually pleasing content, the RDP framework is particularly useful. This paper explores the critical role of perceptual constraints in information theory and their impact on practical generative modeling scenarios, which drives the development of a wide range of compression and communication algorithms that integrate various generative model architectures with end-to-end training objectives. We demonstrate how incorporating perceptual constraints into the traditional RD framework offers a comprehensive perspective for evaluating the performance of AI-empowered systems in communication and potentially increases network traffic.

The capacity to perceive, communicate, and learn from the environment, as well as to construct mental models that internally simulate potential outcomes, is fundamental to both human and artificial intelligence. In early neuroscience research, perception was conceptualized as a form of hypothesis testing [16], a notion that naturally aligns with the divergence measure for perceptual quality in information theory. More recently, the theory of predictive coding has posited that generative models of the world are derived from sensory input, perception being understood as the minimization of prediction errors [17,18]. Developing models capable of mimicking human-like perception and reasoning, including visual aesthetics and consistency; language perplexity and fluency, among others, is essential for the creation of useful generative models. Contemporary large language models (LLMs) and multi-modal models have demonstrated remarkable capabilities, such as memorization [19], generation of images and videos based on descriptive prompts [20], retrieval and reasoning [21]. During the training of these generative models, the loss function typically combines both distortion-oriented MSE and perception-oriented cross-entropy losses, weighted to balance distortion and perception constraints (see Equation (21)). For downstream tasks such as image synthesis, a key challenge is to balance the faithfulness to user inputs and the realism of the generated outputs [22]. Within the RDP framework, the perceptual quality of reconstruction is assessed by the similarity between the distributions of the input and the reconstruction, with the encoder and decoder potentially aided by generative models. This alignment naturally supports the objectives of generative modeling. Thus, in contemporary intelligent compression and communication systems, the quality of reconstruction should be evaluated based on both the perceptions of end users, whether human or machine, and the average distortions.

The perception axis plays a pivotal role in both source coding and communication systems in general. In information theory, the source coding problem aims to minimize the amount of information needed to transmit a message by eliminating redundancy, thereby achieving effective compression. To assess the performance, a certain level of reconstruction error is permissible. For instance, when compressing and subsequently recovering a bit string using a decoder, the Hamming distance can be used to quantify the number of errors, thereby providing a measure of distortion. Conventional RD theory, originally developed by Claude Shannon, has effectively characterized the minimal rate required to satisfy distortion constraints. For stationary sources, it is well established that deterministic encoders and decoders can achieve asymptotically optimal RD performance [23]. However, the introduction of perceptual constraints suggests that deterministic coding may no longer be optimal [24]. Randomization has been demonstrated to be essential for achieving realism constraints through stochastic coding [25,26,27]. The requirement for the output to conform to a specified distribution is known as the realism constraint [28,29,30,31]. The zero-distortion constraint corresponds to the well-known zero-error source coding problem [23], which also implies perfect realism. However, perfect realism, which requires the distributions of the source and reconstruction to be identical, does not necessarily imply zero distortion. Nevertheless, perceptual constraints restrict the reconstruction process, thereby increasing the required rate. For example, consider a pangram—a sentence using each letter of the English alphabet with balanced letter counts. If a decoder is required to reconstruct a sentence while adhering to a specific letter distribution, this scenario exemplifies the perfect realism condition. A “perfect pangram” is extremely rare due to the overuse of common letters and the scarcity of rare ones; for instance, the classic pangram “The quick brown fox jumps over a lazy dog” contains repeated Es and Os. This suggests that perceptual constraints are closely related to semantics and can be highly restrictive, thereby carrying significant amounts of information.

In practical communication systems, the idealized assumptions underlying the separation theorem, upon which modern communication systems are founded, are rarely met. The separation theorem posits that separate source and channel coding (SSCC) is theoretically optimal in terms of distortion, with source coding eliminating data redundancy and channel coding ensuring reliable transmission in noisy and interference-prone environments. Moreover, under certain conditions, incorporating perceptual constraints renders separation suboptimal in the absence of common randomness between the encoder and decoder [32]. Recent research has increasingly focused on AI-empowered communication systems, where communication nodes are enhanced through machine learning techniques, such as generative modeling, to improve the comprehension and generation of high-fidelity content for downstream tasks, including image and video transmission, augmented and virtual reality (AR/VR), the Internet of Things (IoT), and vehicular-to-everything (V2X) applications. These systems are trained end-to-end for downstream tasks using generative objectives that inherently incorporate perceptual measures. Inspired by these information-theoretic results, which suggest that common and private randomness at the encoder and decoder provide advantages in terms of the RDP trade-off, AI-empowered generative models can be deployed at the transmitter and receiver nodes, with these randomness sources serving as shared or private random seeds.

As the main purpose of this paper, we are devoted to highlighting efforts in the field of information theory aimed at elucidating the fundamental limits of source coding and communication with perceptual-oriented metrics. Inspired by these findings, we demonstrate the utility of common and private randomness in generative modeling architectures, such as variational auto-encoders (VAEs), generative adversarial networks (GANs), diffusion models, and transformers, through the lens of the RDP trade-off. These generative modeling architectures can be designed and deployed at communication nodes to support downstream tasks requiring high perceptual fidelity. We provide a comprehensive overview of common perceptual measures in Section 2. Subsequently, Section 3 discusses the information-theoretic results and their implications with respect to the additional rate required to meet the new perceptual constraint. Section 4 introduces several generative modeling architectures, including VAEs, GANs, diffusion models, and transformers, used in communication systems, emphasizing the importance of high perceptual fidelity in downstream machine learning tasks. Section 5 characterizes the role of generative models in AI-empowered semantic communication systems. Finally, Section 6 explores future directions in information theory and communication. Through the lens of the RDP trade-off, we demonstrate that the introduced perceptual constraint leads to an increase in expected traffic within communication networks. The key takeaways include the following:

• Perceptual constraints fundamentally alter the rate–distortion trade-off, requiring additional rate even with unlimited common randomness.
• Generative models naturally align with RDP objectives, making them powerful tools for developing perceptually optimized communication systems.
• The practical implementation of RDP-aware systems faces challenges related to computational complexity and the need for robustness across diverse channel conditions.

Notations

We use capital letters such as X to denote random variables, lowercase letters such as x to denote their corresponding instances, and calligraphic letters such as $\mathcal{X}$ to denote sets of alphabets. Let $\mathrm{Prob}\left(\mathcal{X}\right)$ denote the set of probability measures defined on $\mathcal{X}$, and let $P_X\left(x\right),P_{XY}(x,y)$, and $P_{Y|X}\left(y\right|x)$ denote the probability distribution of X, the joint distribution, and the conditional distribution of Y given $X=x$, respectively. We omit the arguments of the distributions when doing so does not cause ambiguity. The n-sequence $(X_1,X_2,\dots ,X_n)$ is denoted by $X^n$. Suppose that ${\left\{X_i\right\}}_{i=1}^{\infty }$ and ${\left\{Y_i\right\}}_{i=1}^{\infty }$ are memoryless sources; then, $P_{X^n{Y}^n}={\prod }_{i=1}^n{P}_{XY}={\otimes }_{i=1}^n{P}_{X_i{Y}_i}$. We use $\left[m\right]$ to denote the set $\{1,2,\dots ,\lfloor m\rfloor \}$ for $m>0$, and we use $\mathcal{C}$ to denote the codebook. For sequences $(x^n,y^n)\in {\mathcal{X}}^n\times {\mathcal{Y}}^n$, the empirical distribution is defined as

$${\widehat{P}}_{x^n,y^n}(x,y)={\displaystyle \frac{1}{n}}\sum _{i=1}^n\mathbf{1}\{(x_i,y_i)=(x,y)\}.$$

(1)

Extensions to multiple arguments can be defined similarly. Throughout the paper, we use $\mathrm{\Delta }(·,·):{\mathcal{X}}^n\times {\mathcal{X}}^n\mapsto \mathbb{R}$ to denote an additive distortion measure, and we use the calligraphic letter $\mathcal{D}(·,·):\mathrm{Prob}\left({\mathcal{X}}^n\right)\times \mathrm{Prob}\left({\mathcal{X}}^n\right)\mapsto \mathbb{R}$ to denote the measure of perceptual fidelity. We define ${\overline{\mathbb{R}}}_{\ge 0}:={\mathbb{R}}_{\ge 0}\cup \{+\infty \}$.

2. Distortion and Perceptual Measures

We set out to explore the interplay between conventional distortion measures and the recently introduced perceptual measures. Details regarding the emerging class of perceptual measures, including the total variation (TV) distance, the Kullback–Leibler (KL) divergence, and the Wasserstein distance, are provided. Additionally, we present a set of assumptions and a series of distance–divergence inequalities. These elements highlight the fundamental differences and connections between distortion and perceptual measures, thereby motivating the information-theoretic results. We demonstrate that neither the distortion measure nor any of the perceptual measures, including the TV distance, KL divergence, and Wasserstein distance, dominate each other. Instead, they synergistically contribute to the RDP trade-off.

2.1. Distortion Measures

In the conventional additive distortion setting, the distortion measure is defined as the average of a per-letter distance.

Definition 1. 

Given a per-letter distortion measure $\delta :\mathcal{X}\times \mathcal{X}\mapsto [0,d_{\max }]$ with $d_{\max }<\infty$, the (average) distortion between two sequences $x^n$ and $y^n$ is defined as

$$\mathrm{\Delta }(x^n,y^n):={\displaystyle \frac{1}{n}}\sum _{i=1}^n\delta (x_i,y_i).$$

Here, we assume that the distortion is upper-bounded by $d_{\max }$.

This bounded distortion enables the derivation of a bound relating the distortion measure to the perceptual measure, as shown in Equation (11), which will be introduced below.

2.2. Perceptual Measures in Information Theory

To quantify the proximity between the reconstruction distribution and another probability distribution, several perceptual measures are widely studied in information theory. These include the total variation (TV) distance, the Kullback–Leibler (KL) divergence, and the Wasserstein distance.

2.2.1. Total Variation Distance

The TV distance defined in the following is widely used in information theory. Given two distributions $P_X$ and $P_Y$ defined on the same $\sigma$-algebra $(\mathcal{X},\mathcal{F})$, the TV distance is defined as

$${\mathcal{D}}_{\mathrm{TV}}(P_X,P_Y)=\underset{S\subseteq \mathcal{X}}{sup}|P_X\left(S\right)-P_Y\left(S\right)|.$$

The TV distance is equivalent to the $L_1$ norm when the alphabets are finite, making it a natural choice in the Lebesgue space, i.e., when $P_X$ and $P_Y$ are probability mass functions,

$${\mathcal{D}}_{\mathrm{TV}}(P_X,P_Y)={\displaystyle \frac{1}{2}}\parallel P_X-P_Y{\parallel }_1={\displaystyle \frac{1}{2}}\sum _{x\in \mathcal{X}}|P_X\left(x\right)-P_Y\left(x\right)|.$$

(2)

From Equation (2) and the triangle inequality for real numbers, we have the triangle inequality

$${\mathcal{D}}_{\mathrm{TV}}(P_X,P_Y)\le {\mathcal{D}}_{\mathrm{TV}}(P_X,P_Z)+{\mathcal{D}}_{\mathrm{TV}}(P_Z,P_Y).$$

This implies that the TV distance does not tensorize. In fact, we have the following folklore knowledge:

$${\mathcal{D}}_{\mathrm{TV}}(P_{X_1{X}_2},P_{Y_1{Y}_2})\le {\mathcal{D}}_{\mathrm{TV}}(P_{X_1},P_{Y_1})+{\mathcal{D}}_{\mathrm{TV}}(P_{X_2},P_{Y_2})-{\mathcal{D}}_{\mathrm{TV}}(P_{X_1},P_{Y_1})·{\mathcal{D}}_{\mathrm{TV}}(P_{X_2},P_{Y_2}).$$

(3)

2.2.2. KL Divergence

Another widely used divergence measure in information theory is the KL divergence. Given two distributions $P_X$ and $P_Y$ defined on the same $\sigma$-algebra $(\mathcal{X},\mathcal{F})$, when $\mathcal{X}$ is finite,

$${\mathcal{D}}_{\mathrm{KL}}(P_X\parallel P_Y)=\sum _{x\in \mathcal{X}}{P}_X\left(x\right)log{\displaystyle \frac{P_X\left(x\right)}{P_Y\left(x\right)}}.$$

When $P_X$ and $P_Y$ are probability density functions, the KL divergence is defined as

$${\mathcal{D}}_{\mathrm{KL}}(P_X\parallel P_Y)={\int }_{x\in \mathcal{X}}{P}_X\left(x\right)log{\displaystyle \frac{P_X\left(x\right)}{P_Y\left(x\right)}}\mathrm{d}x,$$

and, by convention, ${\mathcal{D}}_{\mathrm{KL}}(P_X\parallel P_Y)=\infty$ when $P_X$ is not absolutely continuous with regard to $P_Y$. The KL divergence is not a distance metric because it is asymmetric between the two arguments and does not satisfy the triangle inequality.

2.2.3. Wasserstein Distance

The Wasserstein distance, rooted in the theory of optimal transport, is a powerful tool with broad applications in deep learning as a perceptual measure [33]. It has also garnered significant attention in information theory, particularly following the pioneering work of Marton on distance–divergence inequalities [34,35].

Definition 2. 

Given $p\ge 1$ and a Polish space $(\mathcal{X},c)$, the $L_p$-Wasserstein distance is defined as

$$W_p(P_X,P_Y)=inf{\left(\mathbb{E}\left[c^p(X,Y)\right]\right)}^{{\displaystyle \frac{1}{p}}},$$

where the infimum is over all jointly distributed random variables $(X,Y)\in \mathrm{Prob}(\mathcal{X}\times \mathcal{X})$.

For each $p\ge 1$, $W_p(, )$ defines a metric on $\mathrm{Prob}\left(\mathcal{X}\right)$. In practice, $W_1$ and $W_2$ are most widely used.

2.2.4. Common Assumptions

In the more general case, several assumptions are made regarding perceptual measures across different settings, which we briefly discuss here.

Assumption 1 

(Non-negativity). For arbitrary distributions $P_X$ and $P_Y$ defined on the σ-algebra $(\mathcal{X},\mathcal{F})$, the perceptual measure satisfies

$$\mathcal{D}(P_X,P_Y)\ge 0\mathit{and}\mathcal{D}(P_X,P_Y)=0\mathit{if}\mathit{and}\mathit{only}\mathit{if}{P}_X\stackrel{d}{=}{P}_Y.$$

Assumption 2 

(Convexity in the second argument). For ${\lambda }_1,{\lambda }_2,\dots ,{\lambda }_n\in {\mathbb{R}}_{\ge 0}$ such that ${\sum }_i{\lambda }_i=1$,

$$\mathcal{D}(P,\sum _{i=1}^n{\lambda }_i{Q}_i)\le \sum _{i=1}^n{\lambda }_i{\mathcal{D}}_{\mathrm{TV}}(P,Q_i).$$

(4)

This convexity is a property satisfied by the f-divergence family, including the TV distance and the KL divergence, as well as the 1-Wasserstein distance.

Some members of the f-divergence family, such as the Hellinger distance and the ${\chi }^2$-divergence, exhibit tensorization. This means that, for product distributions, these divergences can be easily expressed in terms of the marginal distributions. Tensorization, or additivity under products, is crucial for establishing converse proofs.

Assumption 3 

(Tensorization).

$$\mathcal{D}({\otimes }_{i=1}^n{P}_{X_i},P_{Y^n})\ge \sum _{i=1}^n\mathcal{D}(P_{X_i},P_{Y_i}).$$

(5)

Notably, the KL divergence and $W_2$ distance are tensorizable, in the sense that

$$\begin{array}{cc}\hfill {\mathcal{D}}_{\mathrm{KL}}({\otimes }_{i=1}^n{P}_{X^i}\parallel {\otimes }_{i=1}^n{P}_{Y^n})& =\sum _{i=1}^n{\mathcal{D}}_{\mathrm{KL}}(P_{X_i}\parallel P_{Y_i}),\hfill \end{array}$$

(6)

$$\begin{array}{cc}\hfill W_2({\otimes }_{i=1}^n{P}_{X^i}\parallel {\otimes }_{i=1}^n{P}_{Y^n})& =\sum _{i=1}^n{W}_2(P_{X_i}\parallel P_{Y_i}).\hfill \end{array}$$

(7)

The TV distance, however, is not tensorizable.

We may also consider a sequence of perceptual measures to analyze performance over coding blocks. The sub-decomposable assumption introduced in [32] ensures that coding over longer sequences is advantageous.

Assumption 4 

(Sub-decomposability). A sequence of perception measures ${\mathcal{D}}_n(·,·):\mathrm{Prob}\left({\mathcal{X}}^n\right)\times \mathrm{Prob}\left({\mathcal{X}}^n\right)\mapsto \mathbb{R}$ is called sub-decomposable if

$${\mathcal{D}}_n({\otimes }_{i=1}^k{P}_{X_{l_{i-1}+1}^{l_i}},{\otimes }_{i=1}^k{P}_{X_{l_{i-1}+1}^{l_i}})\le \sum _{i=1}^k{\mathcal{D}}_{n_i}(P_{X_{l_{i-1}+1}^{l_i}},P_{Y_{l_{i-1}+1}^{l_i}})$$

(8)

for any positive integers $n_1,n_2,\dots ,n_k$ such that ${\sum }_{i=1}^k{n}_i=n$, where $l_i:={\sum }_{j=1}^i{n}_i$ with $l_0:=0$.

By definition, any tensorizable measure, such as the KL divergence, is also sub-decomposable. The non-tensorizable TV distance is sub-decomposable as well, as shown in Equation (3).

2.3. Interplay Between Distortion and Perception Constraints

Distortion and perception measures, though distinct, are closely intertwined through a series of distance–divergence inequalities. For instance, in the product space ${\mathcal{X}}^n$, the transportation–cost inequalities pioneered by Marton [34] can be employed to derive concentration results.

A notable bound relates the expected values of bounded functions to the TV distance:

$$|{\mathbb{E}}_{P_X}\left[f\left(X\right)\right]-{\mathbb{E}}_{P_Y}\left[f\left(X\right)\right]|\le \underset{x}{sup}\left|f\left(x\right)\right|{\mathcal{D}}_{\mathrm{TV}}(P_X,P_Y),$$

(9)

which facilitates the analysis of expected values using the TV distance. This relationship is particularly useful in machine learning scenarios where the objective is to minimize loss functions that depend on expected values. In such cases, the optimization problem can be reformulated as minimizing the TV distance. Specifically, given that the distortion is bounded by $d_{\max }$, and assuming that $P_{X^n{Y}^n}$ and $Q_{X^n{Y}^n}$ are i.i.d. distributions, we have the following:

$$|{\mathbb{E}}_P\left[D(X^n,Y^n)\right]-{\mathbb{E}}_Q\left[D(X^n,Y^n)\right]\left]\right|\le d_{\max }{\mathcal{D}}_{\mathrm{TV}}(P,Q).$$

(10)

Similarly, the bounded distortion measure exhibits a property elucidated in Lemma 5 of [36] with respect to the TV distance: if $P_{XY},Q_{XY}$ are probability mass functions with ${\mathbb{E}}_{P_{XY}}\left[\delta (X,Y)\right]=A$ and ${\mathcal{D}}_{\mathrm{TV}}(P_{XY},Q_{XY})<ϵ$, then

$${\mathbb{E}}_{Q_{XY}}\left[\delta (X,Y)\right]\le A+ϵd_{\max }.$$

(11)

Perceptual measures can also be related through a set of distance–divergence inequalities. A fundamental relationship between the KL divergence and the TV distance, originally due to Pinsker [37], states the following:

$${\mathcal{D}}_{\mathrm{TV}}(P_X,P_Y)\le \sqrt{{\displaystyle \frac{1}{2}}{\mathcal{D}}_{\mathrm{KL}}(P_X,P_Y)}.$$

(12)

For a finite alphabet, the reverse Pinsker inequality is available (see Lemma 6.3 in [38]):

$${\mathcal{D}}_{\mathrm{KL}}(P_X,P_Y)\le {\displaystyle \frac{{\mathcal{D}}_{\mathrm{TV}}^2(P_X,P_Y)}{{min}_{y\in \mathcal{X}}{P}_Y\left(y\right)}}.$$

(13)

Moreover, for i.i.d. distributions $P_{Y^n}$ and $P_{X^n}\ll P_{Y^n}$, we have (see Equation (30) in [39])

$${\mathcal{D}}_{\mathrm{KL}}(P_{X^n},P_{Y^n})=O\left((n-log\left({\mathcal{D}}_{\mathrm{TV}}(P_{X^n},P_{Y^n})\right)){\mathcal{D}}_{\mathrm{TV}}(P_{X^n},P_{Y^n})\right).$$

(14)

Assuming that the diameter of $\mathcal{X}$ is bounded by B, we also have (see Theorem 6.15 in [40]) the following:

$$W_1(P_X,P_Y)\le B·{\mathcal{D}}_{\mathrm{TV}}(P_X,P_Y).$$

(15)

For coding blocks on $\mathrm{Prob}\left({\mathcal{X}}^n\right)$, Wasserstein distances in product spaces can be naturally related to the distortion measure:

$${\mathrm{\Delta }}_{\beta ,n}(P_{X^n},P_{Y^n}):=\underset{\pi \in \mathrm{\Pi }(P_{X^n},P_{Y^n})}{inf}{\left({\mathbb{E}}_{\pi }\left[\sum _{i=1}^n{\delta }^p(X_i,Y_i)\right]\right)}^{{\displaystyle \frac{1}{p}}},$$

and, according to [35], under certain conditions,

$$W_{1,n}(P_{X^n},P_{Y^n})\le \mathrm{Const}.\sqrt{{\displaystyle \frac{1}{2}}{\mathcal{D}}_{\mathrm{KL}}(P_{X^n},P_{Y^n})}.$$

(16)

These relationships connect common perceptual measures to the TV distance, which is frequently considered in information-theoretic settings. The perceptual measures discussed—TV distance, KL divergence, and Wasserstein distance—each offer unique advantages depending on the application context. TV distance provides a straightforward measure of distributional similarity, KL divergence captures directional divergence between distributions, and Wasserstein distance offers a geometrically intuitive measure of the transport cost between probability distributions.

3. Information-Theoretic Results

Having established the theoretical foundations of perceptual measures, we now turn to their implications in information theory. Understanding how these measures interact with rate and distortion constraints requires a rigorous information-theoretic analysis of the RDP trade-off. This section summarizes recent information-theoretic results for the RDP problem. Typically, a set of distortion constraints (D) and a set of perceptual constraints (P) are enforced to ensure that the distortion and perceptual fidelities are bounded by given constants D and P, respectively.

As in classical information theory, we consider a memoryless source ${\left\{X_i\right\}}_{i=1}^{\infty }$ drawn from finite alphabets $\mathcal{X}$. For continuous sources, Assumption 2 in [26] can be invoked to ensure that distortion and perceptual measures are not overly sensitive to discretizations.

3.1. Realism Constraints

We summarize a spectrum of perceptual constraints (P) in

Table 1. Taxonomy of realism constraints (P).

Realism Constraint	Definition	References
Weak realism	$\mathcal{D}({\widehat{P}}_{X^n},{\widehat{P}}_{Y^n})\le P$	[41,42,43]
Per-symbol realism	$\mathcal{D}(P_X,P_{Y_t})\le P,\forall t\in \left[n\right]$	[26]
Average marginal realism	${sup}_n{\displaystyle \frac{1}{n}}{\sum }_{t=1}^n\mathcal{D}(P_X,P_{Y_t})\le P$	[9]
Strong realism	$\mathcal{D}(P_{X^n},P_{Y^n})\le P$	[43,44,45]
Perfect strong realism	$Y^n\sim P_{X^n}$	[25,31,39]
Near-perfect per-symbol realism	${lim}_{n\to \infty }{max}_{1\le t\le n}\mathcal{D}(P_X,P_{Y_t})=0$	[26,27]
Near-perfect strong realism	${lim}_{n\to \infty }\mathcal{D}(P_{X^n},P_{Y^n})=0$	[25,27]

.

3.1.1. Weak and Strong Realism Constraints

The notion of weak realism is inspired by machine learning practices, such as GAN-based image reconstruction schemes [4]. In information theory, weak realism has been studied within the framework of empirical coordination [41,42] using the empirical distribution (type). In contrast, strong realism compares joint distributions over blocks of symbols, thereby imposing stricter constraints. Matsumoto [44,45] investigated the RDP trade-off for general information sources using fixed-length and variable-length coding, suggesting the need for channel resolvability codes. Single-letter expressions can be derived under the assumption of i.i.d. output [31,43].

3.1.2. Perfect and Imperfect Realism Constraints

The concept of perfect realism [25,29] is analogous to the lossless scenario in rate–distortion problems, requiring that the output distribution closely matches the source distribution in terms of a perceptual measure. This aligns closely with lossy coding under distribution constraints [31], where the output sequence must adhere to a specified i.i.d. distribution while also satisfying distortion constraints. The near-perfect realism constraint requires that the reconstruction distribution be asymptotically arbitrarily close to the source distribution. Wagner [25] demonstrated that achievability with perfect realism is equivalent to achievability with near-perfect realism when the distortion–perception pair is uniformly integrable. Specifically, deterministic coding is sufficient when $D>0$ and $P>0$ [26].

3.2. Information-Theoretic System Model

As an example, we consider the general system model introduced in [27], which incorporates both common randomness and private randomness (see Figure 2). The encoder and the decoder have access to a shared source of randomness J uniformly distributed over $\left[2^{n{R}_c}\right]$. The encoder and decoder also have private randomness sources $L_e\sim P_{L_e}$ and $L_d$ uniformly distributed over $\left[2^{n{R}_d}\right]$, respectively. The encoder observes the source $X^n,J,L_e$ and selects a message $M\in \left[2^{nR}\right]$. The decoder observes the message, has access to J and $L_d$, and attempts to reconstruct the input as $Y^n$. Hamdi et al. [27] showed that the private randomness of the encoder is not useful under the constraint of near-perfect realism. Other variants of the system model include models with (common) side information [43,46].

Definition 3. 

An $(n,2^{nR},2^{n{R}_c},2^{n{R}_d})$ fixed-length code consists of an encoding function

$$f_n:{\mathcal{X}}^n\times \left[2^{n{R}_c}\right]\times {\mathcal{L}}_d\mapsto \left[2^{nR}\right](\mathit{possibly}\mathit{stochastic})$$

and a decoding function

$$g_n:\left[2^{nR}\right]\times \left[2^{n{R}_c}\right]\times \left[2^{n{R}_d}\right]\mapsto {\mathcal{X}}^n(\mathit{possibly}\mathit{stochastic}).$$

The system presented here is designed for fixed-length codes. For variable-length codes, the uniform distributions of J and $L_d$ are replaced with the following constraints:

$$R\ge {\displaystyle \frac{1}{n}}H\left(M\right),R_c\ge {\displaystyle \frac{1}{n}}H\left(J\right),R_d\ge {\displaystyle \frac{1}{n}}H\left(L_d\right).$$

In stochastic coding, it is assumed that various degrees of common randomness between the encoder and decoder, as well as private randomness, are available [27]. When common or private randomness is not available, the corresponding random variables $J,L_e$, or $L_d$ are set to constant, and their respective rates become zero. Specifically, when J and $L_e$ are constant, the encoding function $f_n$ becomes deterministic with respect to $X^n$. Similarly, when J and $L_d$ are constant, the decoding function $g_n$ becomes deterministic with respect to $X^n$.

3.3. Coding Theorems

Within the RDP framework, two scenarios, the one-shot and the asymptotic, have been extensively studied by Theis and Wagner [47] and Chen et al. [26]. The proofs for the one-shot scenario mainly rely on the strong functional representation lemma [48], whereas the proofs for the asymptotic scenario predominantly utilize techniques in channel synthesis [39,41]. Coding theorems in information theory typically involve both achievability and converse results.

Definition 4 

(Achievablility). The tuple $(R,R_c,R_d,D,P)\in {\mathbb{R}}_{\ge 0}\times {\overline{\mathbb{R}}}_{\ge 0}^4$ is achievable if, for any $ϵ>0$, there exists a sequence of $(n,2^{n(R+ϵ)},2^{n(R_c+ϵ)},2^{n(R_d+ϵ)})$ codes $(f_n,g_n)$ such that the distortion constraint

$${\mathbb{E}}_P\left[\mathrm{\Delta }(X^n,Y^n)\right]\le D+ϵ,$$

(D)

and the perceptual constraint(s) (P) are both satisfied, where $Y^n=g_n(f_n(X^n,Z^n,K),Z^n,K)$.

Definition 5. 

The rate region ${\mathcal{R}}_{D,P}$ is the closure of the set of achievable rates, i.e.,

$${\mathcal{R}}_{D,P}=\mathrm{cl}\left(\{(R,R_c,R_d)\in {\mathbb{R}}_{\ge 0}\times {\overline{\mathbb{R}}}_{\ge 0}^2:(R,R_c,R_d,D,P)\mathit{is}\mathit{achievable}\}\right).$$

Given a rate region ${\mathcal{R}}_{D,P}$ induced by a coding scheme, the converse theorem asserts that any rate tuples $(R,R_c,R_d)$ outside ${\mathcal{R}}_{D,P}$ are unachievable. Proofs of the converse often leverage the time-sharing technique (e.g., see [25,31,43]), which relies on the assumption that the output sequence is i.i.d. However, the single-letter characterization can also be derived under tensorizable and decomposable conditions [26], which implicitly imply the optimality of an i.i.d. output sequence.

3.4. The Rate–Distortion–Perception Frontier

The rate–distortion function in the traditional (additive) distortion setting has been successful in characterizing the minimum rate required to describe a source within a specified distortion. For a (not necessarily finite) set of distortion measures ${\left\{{\mathrm{\Delta }}_{\lambda }(·,·)\right\}}_{\lambda \in \mathrm{\Lambda }}$, it is shown in [49] (Exercise 7.14, Page 117) that the rate–distortion function is

$$R\left(\left\{D_{\lambda }\right\}\right)=\underset{P_{\widehat{X}|X}}{inf}I(X;\widehat{X})\mathrm{s}.\mathrm{t}.\mathbb{E}\left[{\mathrm{\Delta }}_{\lambda }(X,\widehat{X})\right]\le D_{\lambda }\forall \lambda \in \mathrm{\Lambda },$$

(17)

assuming that ${inf}_{D_{\lambda }\ne 0}{D}_{\lambda }>0$.

The information rate–distortion–perception (I-RDP) function was first proposed in analogy to the traditional RD function within the machine learning community.

Definition 6 

(I-RDP). Given a random variable X that represents the source distribution, the information rate–distortion–perception function [4] is defined as

$$R^{\left(I\right)}(D,P):=\underset{P_{\widehat{X}|X}}{inf}I(X;\widehat{X})s.t.\mathbb{E}\left[\mathrm{\Delta }(X,\widehat{X})\right]\le D,\mathcal{D}(P_X,P_{\widehat{X}})\le P.$$

(18)

When the perceptual constraint (P) is relaxed, i.e., when $P=\infty$, the RDP function reduces to the classic RD function $R(D,\infty )=R\left(D\right)$. Efficient computation of the I-RDP function has been explored in [50,51,52].

Based on the I-RDP function, ref. [8] demonstrated that the cost of achieving perfect realism is a doubling of the lowest achievable MSE distortion, i.e.,

$$R^{\left(I\right)}(D,0)\le R^{\left(I\right)}({\displaystyle \frac{1}{2}}D,\infty ).$$

Specifically, [53] considered the MSE distortion measure and the 2-Wasserstein distance (see Figure 1b), showing that the distortion–perception function using these measures is quadratic. In [47], the authors first established that the I-RDP function provides a lower bound on the one-shot achievable rate using the Poisson functional representation lemma [48]. Let $R^{*}(D,P)$ denote the one-shot RDP function. The study in [9] subsequently established the relationship between the information function and the operational functions using the same strategy, resulting in the following upper and lower bounds:

$$R^{\left(I\right)}(D,P)\le R^{*}(D,P)\le R^{\left(I\right)}(D,P)+log(1+R^{\left(I\right)}(D,P))+5.$$

In the asymptotic setting, the rate region of the general system illustrated in Figure 2 is given by ${\mathcal{R}}_{D,0}$ as presented in [27]:

$$\begin{array}{cc}\hfill {\mathcal{R}}_{D,0}=\{(R,R_c,R_d,D):\exists P_{XVY}& \in \mathcal{A}\mathrm{s}.\mathrm{t}.\hfill \\ \hfill R& \ge I_P(X;V)\hfill \\ \hfill R+R_c& \ge I_P(Y;V)\hfill \\ \hfill R_d& \ge H_P\left(Y\right|V)\hfill \\ \hfill D& \ge {\mathbb{E}}_P\left[\delta (X,Y)\right]\},\hfill \\ \hfill \mathrm{where}\mathcal{A}:=\{P_{XVY}:X& \sim P_X,Y\sim P_X\hfill \\ \hfill X& \perp Y|V\hfill \\ \hfill \left|\mathcal{V}\right|& \le {\left|\mathcal{X}\right|}^2+1\}.\hfill \end{array}$$

The RDP function can be derived from the achievability and converse of the coding theorems.

Corollary 1. 

When common randomness and decoder randomness are unlimited, the RDP function induced by Equation (20) is given by the following:

$$R(D,P)=\underset{P_{Y|X}}{inf}I(X;Y)s.t.\mathbb{E}\left[\delta (X,Y)\right]\le D,Y\sim P_X.$$

(19)

This result is obtained by setting $R_c=R_d=\infty$. Consequently, the rate region becomes

$$\begin{array}{cc}\hfill {\mathcal{R}}_{D,0}^{\infty ,\infty }& =\{(R,\infty ,\infty ,D):\exists P_{XVY}\in \mathcal{A}\mathrm{s}.\mathrm{t}.R\ge I_P(X;V),D\ge {\mathbb{E}}_P\left[\delta (X,Y)\right]\}\hfill \\ \hfill & =\{(R,\infty ,\infty ,D):X\sim P_X,Y\sim P_X,R\ge I_P(X;Y),D\ge {\mathbb{E}}_P\left[\delta (X,Y)\right]\},\hfill \end{array}$$

(20)

where the equality in Equation (20) follows from the data processing inequality $I_P(X;V)\ge I_P(X;Y)$, given that $X\perp Y|V$ in $\mathcal{A}$. Therefore, the rate region ${\mathcal{R}}_{D,0}^{\infty ,\infty }$ is a subset of the RD region ${\mathcal{R}}_D$, i.e., ${\mathcal{R}}_{D,0}^{\infty ,\infty }\subseteq {\mathcal{R}}_D$.

In light of these results, we draw the following conclusions regarding the introduction of the realism constraint:

• Common randomness and private randomness are helpful in achieving (near) perfect realism.
• The introduction of the perceptual constraint results in a higher rate requirement, even when common randomness and decoder randomness are unlimited.

As illustrated in Figure 1a, where a Gaussian source is considered, the perceptual constraint modestly increases the bitrate required in theory, suggesting an increase in traffic in communication networks. The red arrows indicate the additional rates needed under conditions of no common randomness and unlimited common randomness, respectively. In practice, decoder randomness can be implemented using a random number generator, while common randomness can be achieved through a shared random seed in end-to-end training, where neural network models are deployed at both the encoder and decoder.

The information-theoretic results presented reveal that incorporating perceptual constraints necessitates a fundamental rethinking of RD analysis. Key findings include the establishment of the RDP function as a lower bound on achievable rates, the demonstration that common and private randomness can improve performance under perceptual constraints, and the characterization of how these constraints affect system design in practical communication scenarios.

4. Generative Modeling as a Distribution Approximation Process

The theoretical framework of RDP provides valuable insights into how generative models can be designed to balance fidelity and perceptual quality. In practice, these models must navigate complex trade-offs between maintaining statistical properties of the source data and ensuring efficient representation and transmission.

The growing demand for machine-generated content has spurred the development of generative models across various domains, including language, image, and video generation. Given that these models aim to produce content that appears realistic to humans, evaluation often relies on perceptual quality metrics, which have been integrated into several benchmarks. For instance, the PyIQA package [54] provides a convenient means of computing distortion and perceptual quality measures for image compression. This section underscores the significance of high perceptual fidelity in downstream machine learning tasks, particularly in generative modeling. By examining architectures such as VAEs, GANs, transformers, and diffusion models, along with their objectives, we demonstrate that generative modeling can be interpreted as a distribution approximation process, where the models are trained to match the distribution of the training data.

4.1. Objectives, Tasks, and Architectures

The objective of generative modeling is to train a neural network $P_{\theta }$ capable of generating new samples $y\in \mathcal{X}$ from random seeds (and conditional samples) that closely resemble those in the training data $\mathcal{S}$. From a frequentist viewpoint, the observations $x\in \mathcal{S}$ are drawn i.i.d. from an unknown true distribution: $\mathcal{S}={\left\{x_i\right\}}_{i=1}^N\sim P_X.$ The goal is to learn a generator $g_{\theta }$ that can produce samples approximating $P_X$ by passing a random variable U with a fixed distribution, typically $\mathcal{N}(0,1)$, through the network. Since these models are trained to minimize specific perceptual objectives, the random seed U can be regarded as private randomness that enhances the perceptual quality of the reconstruction, conditioned on given messages. The decoding process in generative models can be viewed as a sampling procedure. For example, in the context of LLMs, beam search [55] is a common maximum a posteriori (MAP) decoding algorithm. Other sampling algorithms, such as temperature sampling [56], top-k sampling [57], nucleus sampling [58], and typical sampling [59], have been developed to generate content faithfully according to the learned distribution.

A variety of generative model architectures have been proposed, with the most prominent ones including variational auto-encoders (VAEs) [60], generative adversarial networks (GANs) [61], diffusion models [62,63], and transformers [64]. These models have achieved notable success in generating images, videos, and language content. They are trained by minimizing specific loss functions that quantify the similarity between the model distribution $P_{\theta }$ and the data distribution $P_{\mathcal{D}}$. For instance, VAEs and diffusion models minimize the negative evidence lower bound (ELBO), which effectively reduces the KL divergence, while GANs optimize objectives ranging from the Jensen–Shannon divergence [61] to the Wasserstein distance [65]. At the core of generative modeling are the divergence measures $\mathcal{D}(P_{\theta },P_{\mathcal{D}})$ between the model and data distributions. Building on these foundations, neural networks for specific tasks, such as lossy compression, can be trained to minimize a combined distortion and perception loss with a weighting factor $\lambda$:

$$\mathcal{L}=\mathrm{\Delta }(X,\widehat{X})+\lambda \mathcal{D}(P_X,P_{\widehat{X}}).$$

(21)

Variants of this loss function have been applied in numerous settings, including those described in [5,6,7,8,9,10,11], to achieve high perceptual quality. These applications underscore the perceptual–distortion trade-off inherent in the training of generative models in practice. As the realism depends heavily on the statistical properties of the training datasets, different dataset statistics may affect the RDP trade-off of generative models.

4.1.1. Variational Auto-Encoder

VAEs can effectively learn the distribution of data $P\left(x\right)$ and generate samples $x\sim P\left(X\right)$ using latent variables Z from a prior distribution $P\left(Z\right)$. The marginal likelihood for inference requires the integration of the latent variable $P\left(x\right)=\int P(x,z)\mathrm{d}z$, which is often intractable. Therefore, an inference model $Q_{\varphi }\left(z\right|x)$ and generative decoder $P_{\theta }\left(x\right|z)$ are trained end-to-end by maximizing the evidence lower bound (ELBO):

$$\begin{array}{cc}\hfill logP\left(x\right)& =log\int Q\left(z\right|x){\displaystyle \frac{P(x,z)}{Q\left(z\right|x)}}\mathrm{d}z\ge \int Q\left(z\right|x)log\left({\displaystyle \frac{P\left(x\right|z\left)P\right(z)}{Q\left(z\right|x)}}\right)\hfill \\ \hfill & ={\mathbb{E}}_{Q_{\varphi }\left(z\right|x)}[log{P}_{\theta }\left(x\right|z)]-{\mathcal{D}}_{\mathrm{KL}}(Q_{\varphi }\left(z\right|x)\parallel P_{\theta }\left(z\right))=:{\mathcal{L}}_{\mathrm{ELBO}}(x;\theta ,\varphi )\hfill \end{array}$$

(22)

for continuous z via stochastic gradient descent. Semi-supervised VAEs [66] are capable of incorporating discrete “semantic labels” as additional variables.

4.1.2. Generative Adversarial Network

The objective of a GAN is to minimize the divergence between the real data distribution $P_X$ and the generative model distribution $P_G$. For example, the vanilla GAN [67] employs the Jensen–Shannon divergence

$${\mathcal{D}}_{\mathrm{JS}}(P_X,P_Y)={\displaystyle \frac{1}{2}}{\mathcal{D}}_{\mathrm{KL}}(P_X\parallel {\displaystyle \frac{P_X+P_Y}{2}})+{\displaystyle \frac{1}{2}}{\mathcal{D}}_{\mathrm{KL}}(P_Y\parallel {\displaystyle \frac{P_X+P_Y}{2}}).$$

The f-GAN [68] has been proposed to minimize a variational objective for different f-divergences:

$$d_f(P_X\parallel P_Y)=\sum _{T\in \tau }({\mathbb{E}}_X\left[T\left(X\right)\right]-{\mathbb{E}}_Y\left[f^{*}\left(T\left(Y\right)\right)\right])$$

where $f^{*}$ is the conjugate function of f. The Wasserstein GAN (WGAN) [65,69] minimizes the 1-Wasserstein distance. A combination of JS divergence and Wasserstein distance is proposed as the training objective in [70].

4.1.3. Transformer

Transformer networks, introduced by [64], consist of a stack of homogeneous layers, including multi-head attention and feed-forward layers. The pre-training process can be interpreted as learning the energy of the data distribution using associative memory [19]. The transformer architecture has been extensively applied to language models, where the alphabet $\mathcal{X}$ comprises tokens that can be understood as codewords. Additionally, a distinguished symbol $\mathrm{EOS}\notin \mathcal{X}$ is introduced to signify the end of the sequence. Let $\overline{\mathcal{X}}:=\mathcal{X}\cup \left\{\mathrm{EOS}\right\}$, a language model, be parametrized by $\theta \in \mathrm{\Theta }$, which is a (discrete) distribution $p_{\theta }$ over ${\mathcal{X}}^{*}$, such that

$$p_{\theta }\left(\mathbf{x}\right)=\widehat{p}\left(\mathrm{EOS}\right|\mathbf{x})\prod _{i=1}^T\widehat{p}\left(x_i\right|{\mathbf{x}}_{<i})$$

(23)

for $\mathbf{x}\in {\mathcal{X}}^{*}$ and $T=\left|\mathbf{x}\right|$, where the conditional distribution $\widehat{p}\left(x\right|\mathbf{x})$ is a sequence model for $x\in \mathcal{X}$ and $\mathbf{x}\in {\mathcal{X}}^{*}$. During decoding, the next token is sampled according to a sampler $q\left(p_{\theta }\right)$ to generate sequences at the decoder that approximate the distribution of the language model.

The autoregressive modeling of images using convolutional architectures has achieved success for low-resolution images. However, transformers consistently outperform their convolutional counterparts for high-resolution images [71]. Specifically, ref. [72] utilizes transformers to simplify neural video compression.

4.1.4. Diffusion Model

In diffusion models [62,63], particularly score-based diffusion models [73,74], refinement information is obtained by incrementally adding white noise to the signal, thereby transforming the source distribution into a Gaussian shape after a Markov chain of diffusion steps. Compared to other image generation methods, such as generative GANs, diffusion-based image generation exhibits superior image sample quality [75].

The forward diffusion process encodes the refinement information using the following Gaussian transition kernel:

$$\begin{array}{c}\hfill p_t\left(x_t\right|x_{t-1})=\mathcal{N}(x_t;\sqrt{1-{\beta }_t}{x}_{t-1},{\beta }_{t}I).\end{array}$$

(24)

A learned diffusion process starts at a trivial known distribution, typically $\mathcal{N}(0,I)$. Furthermore, $X_t$ can be sampled directly according to the cumulative kernel [63], such that

$$\begin{array}{c}\hfill X_t=\sqrt{{\overline{\alpha }}_t}{X}_0+\sqrt{1-{\overline{\alpha }}_t}ϵ,\end{array}$$

(25)

where ${\overline{\alpha }}_t={\prod }_{s=1}^t(1-{\beta }_s)$ and $ϵ\sim \mathcal{N}(0,I)$. Therefore, the generated samples $X_t$ approximately follow the same distribution as the data distribution.

While diffusion models can be interpreted as a special case of VAEs with a particular choice of inference model and generative decoder, the training objective is the ELBO loss with simple data augmentation [76].

4.2. Practical Distortion and Perceptual Quality Measures

In practical applications such as image quality assessment, two types of metrics focus on pixel quality and statistical properties, respectively. Distortion measures, such as mean squared error (MSE), structural similarity index (SSIM), its multiscale variant (MS-SSIM) [77], and peak signal-to-noise ratio (PSNR), require a ground-truth image for comparative analysis (see Figure 1c). Traditional video codecs are designed to efficiently remove spatial–temporal redundancies. Current coding standards, including H.264/AVC [78], H.265/HEVC [79], and H.266/VVC [80], achieve efficient compression through rate–distortion optimization, where the rate is represented by bits-per-pixel (BPP) and distortion is measured by PSNR.

In contrast, perceptual quality measures focus on the statistical properties of images. For example, the Fréchet Inception distance (FID) [81] uses the Inception Network to assess image quality without a reference image by comparing the distributions of generated and real images used to train the generator. The learned perceptual image patch similarity (LPIPS) [82] computes the distance between features in the latent space of a trained neural network and has been shown to effectively capture the underlying semantics of the data. In [83], an unsupervised perceptual information metric (PIM) is proposed using information-theoretic objectives. It is common practice, especially in the domain of LLMs, to employ a powerful model as a judge of semantic similarities. This judge can be conceptualized as a function, and the optimal rate of lossless functional compression can be determined using graph entropy [84,85,86].

Learning-based compression methods not only consider pixel-wise signal quality metrics such as PSNR but also emphasize perceptual quality. These methods are increasingly outperforming standard codecs in many cases. Inspired by the information-theoretic setting [31], ref. [3] employs a GAN-based model to optimize the RD trade-off under the constraint that reconstructed samples follow the training data distribution. Source coding based on diffusion models is also gaining prominence [87,88]. In [89], a posterior sampling-based compression (PSC) method using diffusion is proposed, with two configurations: PSC-Perception and PSC-Distortion, focusing on high perceptual quality and low-distortion reconstructions, respectively. With the increasing demand for processing longer contexts, significant research has also been devoted to the RD trade-off in text compression using large language models (LLMs) [90,91,92].

4.3. Experimental Validation in Image Compression

To demonstrate the practical implications of our theoretical findings, we conducted experiments on image compression using the RDP framework. We implemented several learning-based compression methods that incorporate perceptual measures and compared them against traditional compression codecs.

We evaluated the performance of five compression methods: BPG (baseline), VAE-based compression [93], attention-based compression [15], GAN-based compression [7], and diffusion-based compression [88] at a low bpp regime ($R\approx 0.2$ bpp). We used the CLIC2020 dataset [94], which contains 428 high-quality color images, for our experiments. The performance metrics include PSNR, MS-SSIM, and the perceptual-oriented LPIPS.

The results are summarized in

Table 2. Comparison of compression methods on the CLIC2020 dataset.

Method ($\mathit{R}\approx 0.2$ bpp)	PSNR (↑)	MS-SSIM (↑)	LPIPS (↓)
BPG	33	0.96	0.20
Attention-based [15]	34	0.97	0.14
VAE-based [93]	31	0.95	0.20
GAN-based [7]	32	0.97	0.06

. The GAN-based method shows the most significant improvement in perceptual quality metrics while maintaining competitive distortion metrics. The attention-based method provided better PSNR than traditional methods but showed less improvement in perceptual metrics.

5. AI-Empowered Communication with Perceptual Constraint

The advances in generative modeling discussed thus far have significant implications for communication systems, particularly in scenarios where high perceptual quality is essential for downstream tasks. These models can be deployed at various points in the communication pipeline to enhance both compression efficiency and reconstruction quality.

Recent years have witnessed substantial progress in applying machine learning algorithms, particularly generative modeling, to communication networks, driven by their capacity to enhance perceptual quality. Downstream tasks, ranging from image reconstruction [13] to steganography [51], are highly sensitive to distribution and can be executed at the wireless edge using neural networks. Learning-based methods can more easily account for and optimize perceptual qualities as the training of these neural network models inherently addresses the distribution of the output. Thus, perceptual requirements are both natural and essential for AI-empowered communication systems.

5.1. Source–Channel Coding for Perceptual Oriented Communications

Current digital communication systems are designed based on Shannon’s source–channel separation theorem [95], which posits that there is no loss of optimality in applying separate source coding followed by channel coding in the asymptotic infinite block length regime, provided that the source and channel statistics are ergodic and the distortion measure is additive. As shown in Figure 3a, the source-encoded data are transmitted through the wireless channel after channel coding and modulation. The receiver reverses these procedures by first demodulating and decoding the channel code, followed by decompression to reconstruct the original data.

While the optimality of separation holds for distortion–perception measures under assumptions of perceptual measures, including sub-decomposability, it has been demonstrated that separation is no longer optimal when common randomness between the encoder and decoder is unavailable [32]. Moreover, in practice, idealized assumptions, such as memorylessness, are rarely met [96], rendering separation-based digital communication systems suboptimal [97]. This motivates the adoption of joint source–channel coding (JSCC) to enhance end-to-end performance in practical communication systems (see Figure 3b). Recently, in the context of semantic communications [98], deep-learning-based JSCC methods, such as DeepJSCC [13], have achieved remarkable results by learning mappings directly from training data (both source and channel) [13,99,100,101,102,103,104]. Furthermore, a hybrid scheme that combines SSCC and JSCC has been proposed in [105] (see Figure 3c), demonstrating significant potential in mediating between the two systems.

Figure 3. (a) Separate source and channel coding (SSCC) in current communication system. (b) Joint source–channel coding (JSCC) scheme as in [13]. (c) The dashed lines link SSCC and JSCC to form a hybrid communication scheme, as in [105].

Figure 3. (a) Separate source and channel coding (SSCC) in current communication system. (b) Joint source–channel coding (JSCC) scheme as in [13]. (c) The dashed lines link SSCC and JSCC to form a hybrid communication scheme, as in [105].

5.2. AI-Empowered Semantic Communication

Driven by advances in machine learning, there has been a significant increase in interest in semantic communication systems [98,106,107], with intelligent network architectures being progressively integrated into existing communication standards [108,109,110,111,112,113]. Broadly, the objective of communication is to transmit meaningful information. In the context of generative models, the term “meaningful” is contingent upon the semantic meaning of the information, which represents a fundamentally different global property compared to traditional average distortion measures, such as Hamming distance and MSE. The notions of semantic and pragmatic communication problems are addressed by drawing parallels to linguistics [107]. By integrating machine learning algorithms, particularly generative modeling, into the widely deployed digital communication infrastructure, semantic communication aims to deliver content with higher perceptual quality more robustly over wireless channels, which are frequently influenced by environmental factors such as weather conditions, buildings, and device activities.

5.2.1. Theoretical Foundation and Practical Motivation

In the RDP framework, common and private randomness have been shown to play crucial roles in improving perceptual fidelity while maintaining efficient communication. Theoretically, common randomness (shared between the transmitter and receiver) and private randomness (available only to one node) provide additional degrees of freedom that can be leveraged to optimize the trade-off between rate, distortion, and perception. In practical systems, this can be implemented through several methods.

Shared Random Seeds in End-to-End Training

In learning-based communication systems, common randomness can be implemented by sharing random seeds between the transmitter and receiver during the training phase. For example, in systems using VAEs or GANs, the encoder and decoder can be trained with a shared random seed that initializes their respective random number generators. This ensures that both ends of the communication system can generate correlated random variations, which can be used to improve perceptual quality.

Pilot Signals and Synchronization

In wireless communication systems, common randomness can be established through pilot signals transmitted from the base station to user equipment. These pilot signals can be used to synchronize random number generators at both ends of the communication link. For instance, in massive MIMO systems, the base station can transmit reference signals that user equipment uses to align its random seed with the base station’s generator.

Implementation of Private Randomness

Both transmitters and receivers can incorporate local random number generators that do not require coordination with the other end of the communication system. These generators can be based on physical phenomena (such as thermal noise) or algorithmic approaches (such as pseudorandom number generators with locally maintained seeds).

In the following, we identify three use cases where generative models can be deployed on user equipment (UE), base stations (BSs), and the core network (CN), as illustrated in Figure 4.

5.2.2. Generative Modeling in User Equipment

Due to the increasing hardware computing capability, UE such as smartphones is increasingly capable of performing intelligent tasks involving generative models. In wireless communications, UE encompasses a wide range of devices, including smartphones, tablets, specialized Internet of Things (IoT) devices, and industrial sensors, all of which are capable of transmitting and receiving reliable and efficient communication services. UE features software-defined encoders and decoders for audio, video, and other data formats, as well as support for wireless communication protocols such as LTE and 5G NR. More recently, LLMs have been deployed on edge devices, such as smartphones. Applications like prompt compression [92] enable extended model inference (EMI) and native model inference (NMI), which apply prompt compression and inference on the same devices and facilitate interactions between edge devices and cloud servers, respectively.

Recent research indicates that the joint source–channel coding (JSCC) scheme, combined with a generative model for reconstruction, can achieve substantial reductions in bandwidth while significantly enhancing the perceptual quality of the reconstruction. Utilizing VAEs, ref. [12] reimagines communication systems as a joint optimization of the transmitter (Tx) and receiver (Rx) in an end-to-end manner for reconstruction tasks. The source node (Tx) encodes the data using neural networks that create compact representations while ensuring reliability. For example, a pre-trained generative model based on GANs is employed in [126], and the destination node (Rx) utilizes a generative model to reconstruct the data. In [131], a diffusion model is deployed at the receiver to achieve high-perceptual-quality wireless image transmission. In [134], common modulation schemes are incorporated into the encoded representation as well. In [100], a JSCC video transmission scheme is proposed using deep neural networks. Additionally, Rx may provide feedback to the source node, enabling dynamic adjustments to enhance communication quality [135].

5.2.3. Generative Modeling in Base Station

As part of the radio Access Network (RAN), the BS possesses powerful Tx and Rx capabilities to communicate simultaneously with multiple UE. Generative models have been applied to the BS for channel estimation, channel state information (CSI) acquisition, and resource allocation.

The BS features encoders and decoders to process both uplink (from UE to the BS) and downlink (from the BS to UE) data streams. Modern cellular networks typically operate in full-duplex mode [136], enabling simultaneous bidirectional communication between UE and the BS for concurrent transmission and reception. Antenna arrays in the BS are used to send and receive radio signals, employing sophisticated beamforming techniques. The BS can estimate the current state of the channel, thereby optimizing downlink data transmission. In frequency division duplexing systems, the CSI is fed back to the BS for beamforming. In [118], a GAN-based CSI feedback approach is proposed, where the encoder learns a compressed codeword of the CSI and passes it to the generator for reconstruction in the BS. In [119], a CSI-GPT framework is proposed to efficiently acquire downlink CSI information in massive MIMO systems. In [137], the generalization ability of multi-modal large models across diverse base station environments is explored for various CSI-based tasks. Intelligent algorithms [115,116,117] can be applied to resource allocation and scheduling to dynamically allocate frequency, time, and power resources, thereby meeting real-time user needs and maximizing overall system performance.

5.2.4. Generative Modeling in Core Network

The core network may include multiple types of encoders and decoders to handle data from different protocol stack layers, such as voice encoding and video encoding, to meet diverse service requirements and quality of service (QoS) standards. In [114], two models, 5GCSeq2Seq and 5GC-former, are proposed based on the vanilla Seq2Seq model and the transformer, respectively, to accurately replicate the principal functionalities of the 5G core network control plane, including registration, authentication, handover, and DPU sessions.

5.2.5. Basic Unit for AI-Empowered Communications

In contemporary communication networks, formatted blocks of data are transmitted across various layers of the protocol stack. For instance, in the physical layer of 5G networks, binary digits (bits) are modulated and transmitted over the air interface. In the transport layer, messages from applications are encapsulated into segments such as TCP segments, which include headers, source and destination port numbers, and checksums. This segmentation focuses on the accurate transmission of application messages, with the goal of minimizing the average distortion measure, which is additive in terms of segmental distortion.

In DeepJSCC-like systems, analog signals are employed for end-to-end uncoded transmission [13,135]. Ref. [134] further integrates uncoded signals into a digital modulation system, which is then transmitted and received by the decoder. In the context of low-latency transmission, where information freshness is crucial, short packets serve as the basic unit [138], and the age–distortion trade-off has been studied [139]. Ref. [105] combines analog signals for JSCC with conventional digital signals to form a hybrid communication scheme.

In many generative modeling architectures, tokens function as the fundamental unit. Ref. [140] introduces a token-domain multiple-access scheme, which allows multiple devices to transmit over the same multiple-access channel in a non-orthogonal manner. In the realm of source coding, ref. [141] utilizes 1D tokenization to project 2D images into variable-length, ordered 1D token sequences. This method describes images in a coarse-to-fine manner, thereby achieving a trade-off between the number of tokens (rate) and perceptual quality. The SoTA tokenizer FlowMo is a transformer-based diffusion autoencoder that combines mode matching pre-training and mode seeking post-training [142]. In [143], a video tokenizer is developed for both videos and images using a common token vocabulary. These systems are trained end-to-end using losses that incorporate perceptual quality. Furthermore, technologies such as chunking and the semantic compression of token chunks have been developed in [90], leveraging the parallel computational power of GPUs. In this approach, chunks of tokens serve as the fundamental unit. Given the diverse representation of information, the integration of signals such as tokens and architectures such as tokenizers into intelligent communication systems is necessitated through the lens of the RDP framework.

The integration of generative models into communication systems demonstrates substantial potential for improving perceptual quality while maintaining efficient transmission. Joint source–channel coding approaches like DeepJSCC have shown particular promise by directly optimizing for end-to-end reconstruction quality rather than treating source and channel coding as separate problems. However, practical implementation faces challenges related to computational complexity, training data requirements, and the need for robustness against varying channel conditions.

6. Conclusions and Future Direction

6.1. Conclusions

This paper has explored the critical role of perceptual constraints in information theory and their impact on practical generative modeling scenarios. We have demonstrated that incorporating perceptual constraints into the RD framework, resulting in the RDP framework, offers a novel and comprehensive perspective for evaluating the performance of AI-empowered systems in compression and communication. We have discussed the implications of the perceptual constraint on the design and deployment of generative models, which is particularly relevant in intelligent communication systems, where downstream tasks demand high perceptual fidelity, such as interactive AI agents and visually pleasing content. By highlighting the importance of perceptual constraints and demonstrating their impact on system design and performance, we project an increase in expected traffic in intelligent communication networks with consideration of perceptual quality. While our theoretical analysis suggests that incorporating perceptual constraints generally requires additional rate, we acknowledge that intelligent compression techniques can significantly mitigate this traffic growth.

6.2. Future Direction

6.2.1. Information-Theoretic Directions

In [31,43], it is assumed that the output follows the i.i.d. distribution for i.i.d. sources. When imposing the perfect realism constraint, the i.i.d. condition is naturally imposed on the output. However, it is unclear whether this assumption is valid or redundant. Therefore, we pose the following conjecture for further investigation.

Conjecture 1. 

Consider the system model in Figure 2 and the $(n,2^{nR},2^{n{R}_c},2^{n{R}_d})$ fixed-length code with encoder and decoder $(f_n,g_n)$. For i.i.d. source ${\left\{X_i\right\}}_{i=1}^{\infty }$, suppose that the distortion and perceptual constraints are

$$\begin{array}{cc}\hfill {\mathbb{E}}_P\left[\mathrm{\Delta }(X^n,Y^n)\right]& \le D,\hfill \end{array}$$

(D)

$$\begin{array}{cc}\hfill \mathcal{D}(P_{X^n},P_{Y^n})& \le P,\hfill \end{array}$$

(P)

where $Y^n=g_n(f_n(X^n,Z^n,K),Z^n,K)$. Then, the capacity achieving $Y^n$ is also i.i.d. when $n\to \infty$ for perception measures $\mathcal{D}(·,·)$ that belong to the f-divergence family.

Analogously to the RD function with the multiple-distortion constraint, we pose the following conjecture.

Conjecture 2. 

For a (not necessarily finite) set of distortion measures ${\left\{{\mathrm{\Delta }}_{\lambda }(·,·)\right\}}_{\lambda \in \mathrm{\Lambda }}$ and a (not necessarily finite) set of f-divergences ${\left\{D_{f_{\gamma }}(·,·)\right\}}_{\gamma \in \mathrm{\Gamma }}$, suppose that the source ${\left\{X_i\right\}}_{i=1}^{\infty }$ and reconstruction Y are from the same discrete alphabet $\mathcal{X}$. Consider the (possibly stochastic) code $(f_n,g_n)$; then, the RDP function with unlimited common randomness and unlimited private randomness is

$$\begin{array}{cc}\hfill R(\left\{D_{\lambda }\right\},\left\{P_{\gamma }\right\})& =\underset{P_{Y|X}}{inf}I(X;Y)\hfill \\ \hfill s.t.& \mathbb{E}\left[{\mathrm{\Delta }}_{\lambda }(X,Y)\right]\le D_{\lambda },\forall \lambda \in \mathrm{\Lambda }\hfill \\ \hfill & D_{f_{\gamma }}(P_{X^n},P_{Y^n})\le P_{\gamma },\forall \gamma \in \mathrm{\Gamma }\hfill \end{array}$$

(26)

where ${inf}_{D_{\lambda }\ne 0}{D}_{\lambda }>0$ and ${inf}_{P_{\gamma }\ne 0}{P}_{\gamma }>0$.

When the realism constraint is defined in terms of weak realism, the conjecture can be proven using the method of types [42]. However, it remains unclear whether the conjecture holds under other realism constraints, such as strong realism, or under a combination of different constraints.

6.2.2. Architectural Improvement for Intelligent Communication Systems

Future architectural designs for semantic communications have been discussed in [144,145,146,147]. The integration of perceptual fidelity into the design of intelligent communication architectures represents a paradigm shift from traditional communication systems optimized solely for distortion. The development of systems that better align with human-centric quality assessments and support emerging applications requiring high perceptual fidelity is a promising direction for future research and development, particularly as applications increasingly demand high perceptual quality in transmitted data. By bridging the gap between theoretical advances in information theory and practical deployment considerations, the RDP framework can guide intelligent communication architectures that better serve human-centric needs.

Interactive AI Agents

In interactive AI agent scenarios, communication systems must support real-time decision making while maintaining perceptual consistency with human expectations. RDP trade-off provides a theoretical foundation for balancing the rate requirements against both distortion and perceptual constraints in these dynamic environments. The deployment of generative models at both transmitter and receiver nodes allows for the incorporation of common and private randomness, which has been shown to improve perceptual fidelity in RDP frameworks. For interactive agents, this translates to more natural human–AI interactions where the AI can generate responses that are not only factually accurate but also perceptually coherent with the user’s expectations. Architecturally, this requires communication systems that can dynamically adjust their rate allocation based on the perceptual importance of different components of the transmitted information. For example, in multi-modal communication systems involving text, image, and video data, the RDP function can guide the allocation of bandwidth to prioritize perceptually critical features across different data types.

Perceptually Optimized Data Transmission

RDP frameworks enable the design of communication systems that explicitly optimize for perceptual quality rather than just fidelity to the original data. This is particularly important in applications where the end-user experience depends on how natural/realistic the reconstructed data appear rather than on strict pixel-wise accuracy. In image and video transmission, for example, RDP-informed architectures can prioritize the preservation of distributions that contribute significantly to human perception of quality, even if this requires slightly higher bitrates. Conversely, they can allocate fewer bits to components that have minimal impact on perceptual quality, achieving more efficient compression. For example, systems employing generative models like GANs can achieve high perceptual quality with manageable increases in bitrate by leveraging the models’ ability to capture the underlying data distribution.

Reinforcement learning has emerged as a potentially transformative approach for developing agent-based systems, including embodied AI, as downstream tasks for intelligent communications. Recently, the rate–reward trade-off has been introduced [148]. Moreover, the remote coordination problem has been introduced [149], driven by the necessity to develop coordinated policies among distributed agents [150]. In this context, exploring the fundamental limits of coordination under rate-constrained communication emerges as a significant area of investigation.

Human-Centric Quality Systems

Communication systems prioritizing human-centric quality should account for the subjective nature of human perception, which often differs from objective distortion metrics in current system design. This requires the development of communication architectures that can adapt to the specific perceptual requirements of different applications and user contexts. For example, in tele-medicine applications, the communication system might prioritize the accurate transmission of fine details in medical images that are critical for diagnosis, while, in video conferencing, the system might focus on maintaining smooth motion and natural facial expressions. From an implementation perspective, future design should consider adaptability through modular architectures where different perceptual quality modules can be activated based on the application requirements. These modules could employ specialized generative models trained on task-specific data to optimize the perceptual quality for particular use cases.

6.2.3. Implementation Challenges in Deploying Generative Models for Intelligent Communications

While generative models offer significant theoretical advantages for intelligent communication systems, practical implementation presents several challenges that must be addressed for successful deployment in real-world scenarios.

Computational Complexity

The computational requirements for training and deploying generative models represent a significant implementation challenge. VAEs, GANs, and diffusion models each have distinct computational profiles. VAEs typically require substantial computational resources for training, particularly when dealing with high-dimensional data such as images or video. The encoding and decoding processes involve complex neural network operations that can be computationally intensive. The adversarial training process for GANs between generator and discriminator networks can lead to increased computational complexity. The need for careful balancing between the two networks adds to the computational burden during training. Diffusion models generally require the most computational resources due to their iterative denoising process. The multiple forward and backward passes during both training and inference phases result in higher computational demands compared to VAEs and GANs. The following directions are important for addressing these challenges.

• Model architecture simplification through techniques like depthwise separable convolutions and reduced latent space dimensions.
• Knowledge distillation methods where smaller models are trained to mimic the behavior of larger, more complex generative models.
• The quantization of model weights and activations to reduce precision requirements and computational intensity.

Latency Considerations

Real-time communication systems often have strict latency requirements that must be met for effective operation. The latency introduced by generative models can impact system performance in end-to-end latency, model inference time, and transmission delay: the total delay from the input to output must be minimized, especially in interactive applications like video conferencing or augmented reality; the time required to generate or reconstruct data at the receiver must be compatible with the system’s real-time constraints; the additional bits required for perceptual quality may increase transmission time, creating a trade-off between quality and latency. Future directions include model architecture design specifically targeting low-latency inference, such as using lightweight models.

Hardware Constraints

The hardware limitations of communication devices significantly impact the deployment of generative models. Many communication systems rely on edge devices with limited computational capabilities, memory, and power. Deploying generative models on these devices requires careful optimization. While specialized hardware can mitigate some computational limitations, it introduces additional design considerations and potential costs. Moreover, in wireless communication networks, energy consumption is a critical concern. Generative models should be optimized to operate within the power constraints of battery-powered devices. Future research exploring various solutions to these hardware constraints include:

• On-device model compression techniques, including pruning and quantization.
• Federated learning approaches where model training is distributed across multiple devices.
• Hybrid architectures that perform partial processing at the edge and partial processing in the cloud.