Conditional Encoder-Based Adaptive Deep Image Compression with Classification-Driven Semantic Awareness

Abstract

This paper proposes a new algorithm for adaptive deep image compression (DIC) that can compress images for different purposes or contexts at different rates. The algorithm can compress images with semantic awareness, which means classification-related semantic features are better protected in lossy image compression. It builds on the existing conditional encoder-based DIC method and adds two features: a model-based rate-distortion-classification-perception (RDCP) framework to control the trade-off between rate and performance for different contexts, and a mechanism to generate coding conditions based on image complexity and semantic importance. The algorithm outperforms the QMAP2021 benchmark on the ImageNet dataset. Over the tested rate range, it improves the classification accuracy by $11\%$ and the perceptual quality by $12.4\%$, $32\%$, and $1.3\%$ on average for NIQE, LPIPS, and FSIM metrics, respectively.

1. Introduction

Image compression is widely required in vision-related processing, storage, and transmission over the Internet [1,2,3]. Deep image compression (DIC) is a learning-based approach that outperforms traditional image-coding algorithms, such as JPEG, JPEG2000, and BPG [2,4]. DIC uses deep neural networks to implement and optimize the image coding framework in an end-to-end manner. This approach has two main advantages. First, it can jointly optimize the performance of signal-processing modules that are traditionally separated. Second, DIC can customize the codec to optimize for specific applications, such as various downstream computer vision tasks [5].

The context of image coding refers to the external environment where image coding is used. Among the various factors that contribute to the context, the most important factors are resource constraint and intent, i.e., the compression rate requirement (bits used to encode an image) and the intended purpose of the image application. The intended purpose can be further divided into three categories: reconstruction, high-level tasks, and perception [6]. Reconstruction aims for pixel-level fidelity, and its performance is measured by pixel-level distortion. High-level tasks aim to support downstream tasks, such as image classification, object detection, and semantic segmentation. The focus is on encoding features that are most relevant to a task instead of treating every pixel equally. Perception aims to present an image that has close statistics (thus being perceived similarly) compared to a given image dataset (e.g., a natural image dataset).

In practice, image coding usually needs to serve multiple purposes. For example, in surveillance applications, an image should, on the one hand, support proper reconstruction for human-oriented perception and verification, and, on the other hand, support machine-oriented detection and classification algorithms. Coding for multiple purposes is also called hybrid-context image coding [7]. In this paper, we study a type of hybrid-context DIC focused on image reconstruction and classification. In the deep learning literature, it is well-established that classification-driven learning can effectively encode high-level image features that are useful and transferable to a wide range of other computer vision tasks [8]. The classification-relevant, high-level features are also called the “semantics” of an image [5]. It is important to note that different purposes have distinct rate requirements. Reconstruction typically demands a much higher bit rate than classification and perception [4,8,9]. More importantly, there are complex performance trade-offs among encoded rates and multiple purposes. This is formulated as the rate-distortion-perception-classification (RDPC) trade-off problem [7]. Our paper will explore the RDPC trade-off framework for image codec design.

In a communications scenario, hybrid-context DIC should be able to perform fast rate and context adaptation to accommodate variations in channel capacity and changes in user application. A common way to do this is to train different DIC models for different bitrates and contexts [10]. However, this method can only achieve discrete adaptation, and it needs significant storage and computation resources to cover different combinations of contexts and rates. The conditional encoder-based, rate-adaptive DIC uses a single deep model and implements variable-rate image coding by generating different latent vectors according to the input conditions [11,12]. However, these encoding mechanisms are mainly optimized for image reconstruction, and they can degrade the image quality at low bitrates, leading to negative performance impacts on downstream machine tasks. To solve this problem, researchers have proposed an algorithm in [13], that uses a quality map (Qmap) to control the coding quality of each pixel at different bitrates and provide context-adaptive coding. This approach prioritizes the high-quality reconstruction of semantic salient regions at low bitrates to preserve the computer vision task’s performance, while also ensuring the high-quality reconstruction of the whole image at high bitrates. This adaptive DIC mainly relies on a variable-rate regions of interest (ROI)-based coding scheme, which balances the classification and reconstruction contexts in the allocation of bits.

This paper proposes an efficient conditional encoder-based adaptive DIC method for hybrid contexts that can satisfy both human visual perception and downstream classification task requirements. The method aims to produce reconstructed images that have low distortion and high perceptual and classification quality. To achieve this, we define the optimization objective of hybrid context DIC as a joint optimization of distortion, classification, and perception. Three loss functions are applied to train the DIC model: the mean squared error (MSE), semantic feature matching (SFM), and generative adversarial network (GAN) loss [14,15]. We use the weighted sum of these three loss terms to control the trade-off between DCP at different bitrates. A three-stage DIC model training procedure is proposed to generate visually pleasing images while retaining relevant information for downstream tasks. During adaptive coding, we propose a Qmap generation mechanism based on image complexity and semantic importance to control the reconstruction quality of each pixel at different rates. At lower rates, the Qmap generation mechanism prioritizes the reconstruction quality of semantic important regions. As the rate increases, the mechanism also allocates some bits to improve the visual quality of large regions with low complexity while improving classification accuracy.

This paper makes the following main contributions:

• First, we extend the conditional encoder-based rate-adaptive DIC to hybrid contexts scenarios. We propose a rate-distortion-perception-classification (RDPC) joint optimization framework to train the neural network model for hybrid contexts.
• Second, we propose a Qmap generation mechanism based on image complexity and semantic importance, which allows variable-rate encoding within the rate range and controls the trade-off between classification and reconstruction contexts during bit allocation.
• Third, we evaluate the proposed DIC using performance metrics that correspond to DCP objectives. We show that the proposed DIC can generalize to different datasets and downstream classifiers and achieve superior RDCP trade-off performance.

2. Related Work

2.1. DIC for Single Context

DIC for contex of Image reconstruction. The goal of image reconstruction context DIC is to balance the compression rate and distortion of the reconstructed image with the original image [3]. Many studies have tried to improve this trade-off by optimizing the backbone network structure, quantization, and entropy model. The backbone network is usually based on the autoencoder architecture, and techniques such as generalized dividing normalization (GDN) [16], multi-stacked residual blocks [17], pyramid decomposition [18], attention [19], and vision transformer (ViT) [20] have been used to improve the efficiency of representation learning. For quantization, methods such as multi-level, soft, and non-uniform quantization have been explored to deal with the differentiability problem in DIC model training and to enhance encoding efficiency [21,22,23]. Entropy models are used to estimate the distribution of quantized vectors and reduce redundancy in the encoding process. The hyperprior model is currently the most popular entropy model in DIC, and researchers are still investigating correlations of the spatial and channel dimensions to develop more effective entropy models [24,25,26]. GAN is used to generate images that resemble a given distribution. Therefore, the reconstructed images obtained from GAN-based DIC have a similar distribution to the original image and show better perceptual quality [27,28,29].

DIC for context of high-level tasks. In high-level task context DIC, DNN models are optimized for a specific task, such as classification or semantic segmentation. In this approach, the decoder directly outputs the task result without generating a reconstructed image. This allows only task-relevant information to be encoded, resulting in high task accuracy and compression rates. For example, studies such as [30,31] have used classification and semantic segmentation loss functions to optimize the decoder for task accuracy that is similar to the original image.

2.2. DIC for Hybrid Contexts

The hybrid contexts DIC is designed to optimize both image reconstruction and specific high-level tasks, while catering to both human and machine perception. This DIC can be classified into two types: compression feature-based and reconstruction image-based hybrid contexts DIC, depending on the input at the receiver performing the computer vision task [4,32,33,34,35,36]. In this paper, we focus on the latter, as it only requires one image reconstruction network to achieve both high-quality reconstructed images and accurate machine task results, while the former needs multiple decoders for different machine tasks. This can lead to higher model complexity and limited scalability.

To improve the accuracy of specific machine tasks in reconstruction image-based hybrid contexts DIC, there are two common techniques. First, the loss term related to the machine task is added to the objective function of the DIC model training. Second, more bits are allocated to regions relevant to the machine task during bit allocation. For example, several papers, such as [36,37,38], have adopted the first technique. Moreover, ref. [36] uses class activation mapping (CAM) to obtain semantic information about the image and integrates it with the attention mechanism as prior knowledge to assign more bits to semantic salient regions.

2.3. Adaptive DIC

DIC that can support rate adaptation using a single DNN model can be divided into three types: conditional encoder-based, quantization-based, and latent vector-based rate-adaptive DIC. In conditional encoder-based rate-adaptive DIC, the encoder generates different latent vectors for different bitrates, usually using Lagrange multipliers or pre-mapped Lagrange multipliers [11,12]. Quantization techniques, which are widely used in traditional source coding, have been adapted and applied in DIC [19,39,40]. In these DIC, the elements of the latent vector are scaled using different coefficients and assigned to different quantization intervals. Latent vector-based rate-adaptive DIC, on the other hand, generates a single latent vector that can be encoded to produce a bitstream for various bitrates. For example, [21,41,42] transformed the original image into layered latent vectors with ordered dependencies using a residual network, while [43] used a spatial mask to modulate the channel and achieve the desired rate.

However, all these rate-adaptive DIC are focused on image reconstruction and cannot achieve context adaptation. To address this, a conditional encoder-based adaptive DIC is proposed in [13], which uses an encoding quality map as a conditional input to control the encoding quality of each pixel. This adaptive DIC can perform ROI encoding at different bitrates, prioritizing the reconstruction quality of relevant pixels important for machine tasks. However, it only uses MSE as the distortion term during model training. Another latent vector-based adaptive DIC proposed in [7] uses the weighted sum of MSE, SFM, and GAN losses as the distortion term during training. This DIC uses a mask for bit allocation, which causes an overhead that limits fine-grained bit allocation.

3. Conditional Encoder-Based Adaptive DIC in Hybrid Contexts

3.1. The RDCP Trade-Off

This study proposes an adaptive DIC algorithm for hybrid contexts, which aims to achieve high classification accuracy and good visual quality of reconstructed images at variable rates. Good visual quality includes low distortion and high perceptual quality, as defined in previous works [44,45]. To jointly optimize rate, distortion, classification, and perception, we use the definitions of the corresponding metrics in [6,46]. Specifically, the distortion metric measures the pixel-wise difference between the original and reconstructed image using various full-reference loss functions, while the perceptual metric measures the distributional divergence between the original and reconstructed image. The classification metric measures the classification error rate of the reconstructed image using a pre-trained classifier. The RDCP optimization problem in hybrid contexts is formulated as follows:

$$\begin{array}{c}\hfill R(D,C,P)=\underset{E,G}{min}I(\mathbf{X};\widehat{\mathbf{X}}),\mathrm{subject}\mathrm{to}\left\{\begin{array}{c}\mathbb{E}[\Delta (\mathbf{X},\widehat{\mathbf{X}})]\le D\hfill \\ d(q_{\mathbf{X}},q_{\widehat{\mathbf{X}}})\le P\hfill \\ ϵ\left(\widehat{\mathbf{X}}\right)\le C\hfill \end{array}\right.,\end{array}$$

(1)

where E and G are the encoder and decoder of the DIC, respectively. Specifically, the encoder maps an input image $\mathbf{X}$ to a latent vector $\mathbf{Y}$, i.e., $\mathbf{Y}=E\left(\mathbf{X}\right)$. Subsequently, the decoder reconstructs the image using the latent vector, i.e., $\widehat{\mathbf{X}}=G\left(\mathbf{Y}\right)$. The rate of encoding is determined by the mutual information $I(\mathbf{X};\widehat{\mathbf{X}})$ between the original and reconstructed images, which measures the amount of information needed to represent the image.

Designing optimal codecs for achieving optimal bounds for the RDCP trade-off in hybrid contexts DIC is a highly challenging task. To simplify the problem, we adopt the approach from our previous work [7] and reduce the problem to finding the optimal parameters in the proposed DIC model to achieve the RDCP trade-off. Moreover, we consider the trade-off between distortion and classification in the design of the bit allocation strategy.

3.2. Framework of Proposed DIC

Figure 1 shows the proposed DIC framework, which has two main modules: a coding quality assignment module and a GAN-based deep coding network. The coding quality assignment module generates a Qmap for an input image $\mathbf{x}\in {\mathbb{R}}^{H\times W\times 3}$ based on the target rate and the contextual importance of each pixel. The Qmap vector is $\mathbf{q}\in {\mathbb{R}}^{H\times W}$, $q_{i,j}\in [0,1]$. The coding quality assignment module analyzes the image by the semantic analysis unit and the image complexity analysis unit to obtain the semantic importance map (SIM) and the image complexity map (ICM), respectively. The SIM measures the importance of each pixel for classification, and the ICM measures the compression complexity of pixels [47,48]. The Qmap generation unit generates the Qmap based on the SIM, ICM, and target rate, where the target rate is determined by the underlying channel. Here, the channel is a conceptual model of rate-limited channels that can correspond to various real physical channels. The Qmap and the original image are then used as inputs to the encoder to generate the latent vector $\mathbf{y}$. The latent vector is quantized using a uniform scalar quantizer, and the entropy model extracts contextual information from the quantized vector $\widehat{\mathbf{y}}$ to generate parameters for estimating its distribution. The entropy encoder calculates its distribution based on the entropy model and performs entropy encoding to generate the code stream, which is sent to the receiver. The entropy decoder estimates its distribution based on the entropy model and decodes the received stream to obtain the quantized vector. Finally, the GAN-based decoder generates the reconstructed image $\widehat{\mathbf{x}}$ from the quantized vector.

3.3. Loss Function Design

This paper formulates designing adaptive DIC for hybrid contexts as an RDCP performance trade-off control problem, where the distortion, classification, and perception performances should be simultaneously optimized at different rates.

For the distortion performance, the most common distortion metric for reconstructed images is the pixel MSE, which is computed as

$$\begin{array}{c}\hfill {\mathcal{L}}_{\mathrm{MSE}}=\frac{1}{3HW}\sum _{i=1}^H\sum _{j=1}^W\sum _{k=1}^3{\Vert x_{i,j,k}-{\widehat{x}}_{i,j,k}\Vert }^2.\end{array}$$

(2)

For classification performance, we use the SFM loss, which measures the similarity between classification features extracted from the original image and the reconstructed image, respectively. The SFM loss is computed as

$$\begin{array}{c}\hfill {\mathcal{L}}_{\mathrm{SFM}}=\frac{1}{H_{\mathrm{F}}{W}_{\mathrm{F}}{C}_{\mathrm{F}}}\sum _{i=1}^{H_{\mathrm{F}}}\sum _{j=1}^{W_{\mathrm{F}}}\sum _{k=1}^{C_{\mathrm{F}}}{\Vert \mathrm{F}{\left(\mathbf{x}\right)}_{i,j,k}-\mathrm{F}{\left(\widehat{\mathbf{x}}\right)}_{i,j,k}\Vert }_1,\end{array}$$

(3)

where $H_{\mathrm{F}}\times W_{\mathrm{F}}\times C_{\mathrm{F}}$ is the size of the feature extracted from $\mathbf{x}$ and $\widehat{\mathbf{x}}$, and $\mathrm{F}(·)$ is a pre-trained feature extractor in the semantic analysis network.

For perceptual performance, it is defined as the measurement of the distributional divergence between the original and reconstructed image. Therefore, we adopt the GAN loss to bring the statistical distribution of the reconstructed image closer to that of the input image. Specifically, we use the conditional GAN [14,15] and choose the unsaturated GAN [14] as the optimal loss for GAN training. The loss functions of the generator and discriminator are as follows:

$$\begin{array}{c}\hfill {\mathcal{L}}_{\mathrm{GAN}}=-\log \left(D(G(\mathbf{y},\mathbf{q}),\mathbf{q})\right),\end{array}$$

(4)

$$\begin{array}{c}\hfill {\mathcal{L}}_{\mathrm{D}}=-\log (1-D(G(\mathbf{y},\mathbf{q}),\mathbf{q}))-\log \left(D(\mathbf{x},\mathbf{q})\right),\end{array}$$

(5)

where D is the discriminator and $\mathbf{q}$ is the conditional vector.

As illustrated in Figure 1, the input image is encoded into the latent vector with the Qmap’s control. The rate of the quantized latent vector is

$$\begin{array}{c}\hfill R=-{\log }_{2}P\left(\widehat{\mathbf{y}}\right|\mathbf{q}),\end{array}$$

(6)

where P is the entropy model conditioned on the vector $\mathbf{q}$.

In summary, the proposed DIC loss function is

$$\begin{array}{c}\hfill \mathcal{L}=R+{\lambda }_{\mathrm{MSE}}{\mathcal{L}}_{\mathrm{MSE}}+{\lambda }_{\mathrm{SFM}}{\mathcal{L}}_{\mathrm{SFM}}+{\lambda }_{\mathrm{GAN}}{\mathcal{L}}_{\mathrm{GAN}},\end{array}$$

(7)

where ${\lambda }_{\mathrm{MSE}}$, ${\lambda }_{\mathrm{SFM}}$ and ${\lambda }_{\mathrm{GAN}}$ are the weights of MSE, SFM, and GAN losses, respectively. These weights globally control the three objectives of hybrid contexts DIC at different rates and balance the RDCP trade-off.

3.4. Method for Training the Proposed DIC

In [13], a variable rate codec network controlled by Qmap is trained using pixel-by-pixel rate-distortion optimization. The optimization process aims to minimize the sum of the rate term R and a weighted MSE term $\Lambda \mathrm{J}$, where $\Lambda \in {\mathbb{R}}^{H\times W}$ is a vector composed of Lagrange multipliers corresponding to each pixel, and $\mathrm{J}\in {\mathbb{R}}^{H\times W}$ is a vector composed of each pixel’s MSE. The element ${\lambda }_{i,j}$ of $\Lambda$ is obtained by mapping the element $q_{i,j}$ of the Qmap $\mathbf{q}$ through a pre-defined monotonic increasing function $\mathrm{T}:[0,1]\to {\mathbb{R}}^{+}$, i.e., ${\lambda }_{i,j}=\mathrm{T}\left(q_{i,j}\right)$. This enables pixel-wise control of the encoding quality, with higher values of $q_{i,j}$ leading to higher values of ${\lambda }_{i,j}$, resulting in higher encoding rates and better reconstruction quality at that pixel.

The distortion loss is calculated pixel-by-pixel, while the classification and perception losses are computed after feature extraction for the entire image. To control the coding quality for each pixel, we propose a stepwise optimization training method. First, the rate-distortion optimization method used in [13] is employed to obtain the variable rate encoding for DIC in the reconstruction context. Then, the encoder and decoder are trained with distortion and classification losses by adding the ${\mathcal{L}}_{\mathrm{SFM}}$ term. Finally, the network parameters at the encoding end are frozen, and the decoder network based on the conditional GAN generator is alternately and iteratively trained using the loss functions (7) and (5) to optimize distortion, classification, and perception at the same time.

3.5. Qmap-Based Bits Assignment

The proposed conditional encoder-based rate adaptation mechanism can balance the trade-off between different objectives in hybrid contexts in the spatial domain. The Qmap can adjust the encoding rates and the encoding quality of different image regions according to their semantic importance and complexity.

Figure 2 shows how different Qmap generation strategies affect bit allocation and reconstructed images at the same rate, using a lighthouse photo from the Kodak dataset as an example. The bit allocation map shows the negative log-likelihood of each element of the quantized latent vector, where lower rates correspond to darker regions and vice versa. By comparing the bit allocation maps, we can see that complex regions need more bits to encode. However, giving more bits to complex regions may not improve the reconstruction quality much. A Qmap that considers image complexity gradually distributes limited bits to more regions to enhance the overall reconstruction quality. On the other hand, the strategy of allocating bits based on SIM achieves high reconstruction quality for semantic salient regions, but it may ignore the background regions. We combine the advantages of different allocation strategies and propose a coding quality generation mechanism that considers both image complexity and semantic importance to improve the overall reconstruction quality while prioritizing the semantic salient regions.

4. Implementation of the Proposed DIC

4.1. Network Structure

Figure 3 illustrates the network structure of this implementation, which consists of an encoder $f_e$, a decoder $f_d$, an entropy model side information encoder $h_e$, an entropy model parameter generator g, and a discriminator D. The variable rate codec backbone network is similar to [13], except that we use conditional GAN to train the decoder. During training, the coding quality vector $\mathbf{q}$ and the quantized latent vector $\widehat{\mathbf{y}}$ serve as the conditional input for the conditional GAN. The codec process of the proposed algorithm can be written as follows:

$$\begin{array}{c}\hfill \left\{\begin{array}{c}\hspace{1em}\mathbf{y}=f_e(\mathbf{x},\mathbf{q})\hfill \\ \hspace{1em}\mathbf{z}=h_e(\mathbf{y},\mathbf{q})\hfill \\ \hspace{1em}\widehat{\mathrm{y}}=\mathrm{Round}\left(\mathbf{y}\right)\hfill \\ \hspace{1em}\widehat{\mathbf{z}}=\mathrm{Round}\left(\mathbf{z}\right)\hfill \\ \hspace{1em}\mathbf{\mu },\mathbf{\sigma }=g\left(\widehat{\mathbf{z}}\right)\hfill \\ \hspace{1em}\widehat{\mathbf{x}}=f_d(\widehat{\mathbf{y}},\widehat{\mathbf{z}})\hfill \end{array}\right.,\end{array}$$

(8)

where $\mathrm{Round}(·)$ is the quantization operation during coding, while uniform noise $U(-1/2,1/2)$ is used to approximate this operation during training to overcome the non-derivability of quantization. The side information encoder takes the quantized latent vector and the Qmap as inputs to generate the hyper-prior vector $\mathbf{z}$, which is used to generate the mean and variance parameters to estimate the latent vector distribution. The quantized latent vector is further coded by an entropy codec.

We adopt the spatial feature transform (SFT) layer [49] to generate a pair of parameters $(\mathsf{\gamma },\mathsf{\beta })$ based on the input conditions $\mathbf{a}$, and use $(\mathsf{\gamma },\mathsf{\beta })$ to affine transform the input feature vectors $\mathbf{b}$ to obtain the features for a certain context and rate. The affine transform is defined as

$$\begin{array}{c}\hfill \mathrm{SFT}(\mathbf{a},\mathbf{b})=\mathsf{\gamma }\circ \mathbf{b}+\mathsf{\beta },\end{array}$$

(9)

where the symbol ∘ represents the Hadamard product, which is an element-wise multiplication between vector $\mathsf{\gamma }$ and vector $\mathbf{b}$. To learn better latent space representations with deeper networks, our model also uses residual blocks based on the SFT layer, as shown in Figure 4.

4.2. Deep Codec Model Training

To achieve adaptive DIC, a deep model training method with objective-by-objective incremental optimization is proposed. First, a pixel-by-pixel rate-distortion optimization is used to train whole network parameters to obtain a variable rate coding model controlled by Qmap. The training loss function is

$$\begin{array}{c}\hfill {\mathcal{L}}_1=R_1+\frac{1}{3HW}\sum _{i=1}^H\sum _{j=1}^W\sum _{k=1}^3{\lambda }_{\mathrm{MSE}}^{i,j}{\Vert x_{i,j,k}-{\widehat{x}}_{i,j,k}\Vert }^2,\end{array}$$

(10)

where $R_1$ is the rate estimate of the compressed stream, which includes the quantized latent vector and the side information vector. Following the variable rate mechanism in [13], we set ${\lambda }_{\mathrm{MSE}}^{i,j}=t_1\exp \left(t_2{q}_{i,j}\right)$, where $t_1$ and $t_2$ adjust the interval of the variable rate.

$$\begin{array}{c}\hfill R_1=-lo{g}_2{P}_1\left(\widehat{\mathbf{y}}\right|(\mathbf{q},\mathbf{z}))-lo{g}_2{P}_2\left(\widehat{\mathbf{z}}\right|\mathbf{q}),\end{array}$$

(11)

where $P_1$ is the conditional Gaussian model and $P_2$ is the factorized entropy model.

Second, after obtaining the variable-rate DIC model trained by the MSE loss, the SFM loss term is added to further fine-tune the model’s parameters. The optimization loss function of this step is

$$\begin{array}{c}\hfill {\mathcal{L}}_2={\mathcal{L}}_1+{\lambda }_{\mathrm{SFM}}{\mathcal{L}}_{\mathrm{SFM}}\end{array}$$

(12)

Finally, the parameters of $f_e$, $h_e$, and g are frozen and the parameters of $f_d$ are fine-tuned using a generative adversarial approach. This step alternately trains $f_d$ and the discriminator D, with the following loss functions:

$$\begin{array}{c}\hfill {\mathcal{L}}_3={\mathcal{L}}_2-{\lambda }_{\mathrm{GAN}}\log \left(D(f_d(\widehat{\mathbf{y}},\widehat{\mathbf{z}}),\widehat{\mathbf{y}},\mathbf{q})\right),\end{array}$$

(13)

$$\begin{array}{c}\hfill {\mathcal{L}}_{\mathrm{D}}=-\log (1-D(f_d(\widehat{\mathbf{y}},\widehat{\mathbf{z}}),\widehat{\mathbf{y}},\mathbf{q}))-\log \left(D(\mathbf{x},\widehat{\mathbf{y}},\mathbf{q})\right)\end{array}$$

(14)

4.3. Qmap Generation

SIM generation. The semantic analysis unit generates the SIM and consists of a classification network and a CAM analysis operation. First, the original image is fed into the classification network to obtain the category. A pre-trained VGG16 [50] model is used as the classification network, which was pre-trained on the ImageNet dataset with 1000 categories. Next, the CAM method is used to analyze the semantic importance of each pixel in the image. The obtained semantic importance mapping is further normalized to [0,1], yielding the SIM. The SIM is a two-dimensional vector denoted as $\mathbf{s}\in {\mathbb{R}}^{H\times W}$, where higher values indicate that the corresponding pixel is more important for the classification result. The gradient-based CAM method is used as a semantic importance analysis tool because it can be applied to any CNN-based classifier for semantic analysis [51].

ICM generation. We proposed a calculation method based on the image’s spatial information to obtain the ICM, which measures the image’s complexity, in [48]. The detailed ICM calculation process is as follows:

First, the Sobel operator [52] in the horizontal and vertical directions is used to calculate the spatial information matrix $\mathbf{SI}$. The calculation process is

$$\begin{array}{c}\hfill \mathbf{SI}=\sqrt{{{\mathbf{SV}}_{\mathrm{R}}}^2+{{\mathbf{SH}}_{\mathrm{R}}}^2}+\sqrt{{{\mathbf{SV}}_{\mathrm{G}}}^2+{{\mathbf{SH}}_{\mathrm{G}}}^2+{\sqrt{{\mathbf{SV}}_{\mathrm{B}}}}^2+{{\mathbf{SH}}_{\mathrm{B}}}^2},\end{array}$$

(15)

where ${\mathbf{SV}}_{\mathrm{R}}$, ${\mathbf{SH}}_{\mathrm{R}}$, ${\mathbf{SV}}_{\mathrm{G}}$, ${\mathbf{SH}}_{\mathrm{G}}$, ${\mathbf{SV}}_{\mathrm{B}}$, and ${\mathbf{SH}}_{\mathrm{B}}$ are matrices obtained by applying the Sobel operator to the red, green, and blue channels, respectively, in both horizontal and vertical directions.

Second, the obtained $\mathbf{SI}$ is partitioned into blocks of size $\frac{H}{K}\times \frac{W}{K}$, and the mean value of each block is calculated. The mean value of each block is taken as the complexity of that block, and these mean values form the matrix $\overline{\mathbf{SI}}$. Here, K is the number of times the encoder downsamples the original image, and H and W are the height and width of the original image, respectively.

Third, nearest neighbor adaptive interpolation is applied to $\overline{\mathbf{SI}}$ to obtain a matrix of the same size as the image, and the elements of this matrix are normalized to [0,1]; the resulting matrix is the ICM, which is noted as $\mathbf{c}\in {\mathbb{R}}^{H\times W}$.

Qmap generation. The Qmap generation unit generates a Qmap for each pixel in an image that assigns a specific encoding quality level based on the target rate, SIM, and ICM. A Qmap with all elements set to 0 generates a stream with the lowest possible rate, resulting in the lowest-quality reconstructed image. Conversely, a Qmap with all elements set to 1 generates a stream with the highest possible rate, resulting in the highest-quality reconstructed image. However, when the target rate is lower than the highest possible rate, we cannot achieve the best reconstruction quality for every pixel in the image. Therefore, the Qmap generation unit prioritizes certain pixel regions based on contextual targets and available rates in the Qmap generation process. According to Section 3.5, the priority order for bit allocation is: semantic salient regions, low complexity regions, and high complexity regions. The Qmap generation process is as follows:

The first step is generating the importance map of each pixel based on SIM and ICM according to the proposed priority order for bit allocation. To begin with, elements of the semantic importance vector $\mathbf{s}$ are binarized to either 0 or 1. This operation divides the image into semantic salient and non-salient regions. The OTSU algorithm [53] is adopted to determine the binarization threshold. For the semantic non-salient regions, the pixel importance is inversely proportional to the complexity, and the relative importance of this region can be expressed as $(1-\mathbf{c})$, where $\mathbf{c}$ is the ICM. The importance map of each pixel is represented as a vector $\mathbf{v}\in {\mathbb{R}}^{H\times W},(v_{i,j}\in [0,1])$, which is calculated as follows:

$$\begin{array}{c}\hfill \mathbf{v}=\mathbf{s}\circ \mathrm{Binary}\left(\mathbf{s}\right)+\mathrm{Norm}\left(\right(1-\mathbf{c})\circ (1-\mathrm{Binary}\left(\mathbf{s}\right)),\end{array}$$

(16)

where $\mathrm{Norm}(·)$ normalizes all elements in the input vector between 0 and the minimum of the non-zero elements of $\mathbf{s}\circ \mathrm{Binary}\left(\mathbf{s}\right)$. $\mathrm{Norm}\left(\right(1-\mathbf{c})\circ (1-\mathrm{Binary}\left(\mathbf{s}\right))$ ensures that the semantic non-salient regions are less important than the semantic salient regions.

The second step is mapping vector $\mathbf{v}$ to vector $\mathbf{q}$ to get the Qmap corresponding to the target rate using the mapping function $\Gamma :\mathbf{v}\to \mathbf{q}$, $\mathbf{v}\in [0,1]$, $\mathbf{q}\in [0,1]$. In this mapping, the elements of these two vectors are mapped one-by-one according to $q_{i,j}=\Gamma \left(v_{i,j}\right)$, where $\Gamma (·)$ is defined as

$$\begin{array}{c}\hfill \Gamma \left(v_{i,j}\right)=\left\{\begin{array}{cc}\tau v_{i,j},\hfill & \mathrm{Rate}(\mathbf{x},\mathbf{v})\ge R_T,\tau \in [0,1]\hfill \\ {(1-{(1-v_{i,j})}^{\rho })}^{1/\rho },\hfill & \mathrm{Rate}(\mathbf{x},\mathbf{v})<R_T,\rho \in (1,+\infty )\hfill \end{array}\right.,\end{array}$$

(17)

where $R_T$ is the target rate and $\mathrm{Rate}(\mathbf{x},\mathbf{v})$ is the rate of the quantized latent vector obtained by encoding the input image $\mathbf{x}$ with the condition vector $\mathbf{v}$.

The principle of bit assignment is to make $\mathrm{Rate}(\mathbf{x},\Gamma (\mathbf{v}\left)\right)$ as close to the $R_T$ as possible while maintaining pixel priority order. Since the vector $\mathbf{v}$ for an image is fixed, the coding quality assignment at the target rate becomes selecting the appropriate variable $\tau$ or $\rho$ of the function $\Gamma$ to obtain $\mathrm{Rate}(\mathbf{x},\Gamma \left(\mathbf{v}\right))\approx R_T$. However, since $\mathrm{Rate}(·,·)$ involves non-linear operation, it is impossible to directly solve for $\tau$ or $\rho$ from $R_T$. It is stated that $\Gamma \left(v_{i,j}\right)$ is a monotonically increasing function concerning the variables $\tau$ and $\rho$ in its domain, and that $\mathrm{Rate}(\mathbf{x},\Gamma (\mathbf{v}\left)\right)$ is also an increasing function concerning $\Gamma \left(v_{i,j}\right)$. Therefore, $\mathrm{Rate}(\mathbf{x},\Gamma (\mathbf{v}\left)\right)$ is an increasing function with respect to $\tau$ and $\rho$, and a search algorithm is used to determine $\tau$ or $\rho$. When $v_{i,j}=0$, $\Gamma \left(v_{i,j}\right)$ is always 0. To allocate more bits to each pixel when the bitrate is sufficient, the value of $v_{i,j}=0$ is set to the minimum non-zero element in $\mathbf{v}$.

Therefore, the process of generating Qmap according to the importance map of each pixel includes determining the mapping function based on $R_T$, using the golden section search algorithm to obtain $\tau$ or $\rho$ that results in a bitrate close to $R_T$, and finally mapping $\mathbf{v}$ to $\mathbf{q}$ according to the determined mapping function.

5. Experimental Results

5.1. Experimental Setup

The proposed DIC model is trained on the COCO [54] dataset with data augmentation by randomly cropping $256\times 256$ images without resizing. Random Qmaps are generated using methods described in [13]. The performance of the DIC model is evaluated on the Kodak [55] and ImageNet [56] datasets compared to other image compression algorithms. For the Kodak dataset, 23 original images are used for algorithm performance evaluation. For the ImageNet dataset, 100 categories are randomly selected from the ImageNet validation set, and then 5 images are randomly selected from each category separately; finally, the 500 selected images are used to evaluate the algorithm’s performance.

The DIC model is implemented based on Pytorch [57] and the CompreAI library [58], and then trained in three stages. In the first stage, the model is trained iteratively for 2 million steps with the loss function (10). In the second stage, 300,000 steps of training are performed with the loss function (12). In the third stage, the decoder and discriminator are trained alternately with 300,000 steps using the loss functions (13) and (14), respectively. The Adam optimizer [59] is used with a learning rate of $1\times {10}^{-4}$. initially and adjusted to $1\times {10}^{-5}$ after 1.4 million steps. The batch size is set to 8 and the gradient clipping is used for stable training. Default hyperparameters include ${\lambda }_{\mathrm{SFM}}=0.01$, ${\lambda }_{\mathrm{GAN}}=0.5$, $t_1=4\times {10}^{-4}$ and $t_2=2.8$. The coding rate range is controlled by $t_1$ and $t_2$, and the average rate range on the Kodak dataset varies from 0.05 bpp to 0.35 bpp using default settings. The VGG16 with pre-training parameters published on the official Tensorflow website is used as the classifier to compute the classification accuracy, and its feature extraction network is used to compute the SFM loss and SIM.

Several metrics are used to evaluate the proposed DIC model and other baseline models, such as the classification accuracy, peak signal-to-noise ratio (PSNR) [60], structural similarity index measure (SSIM) [61], feature similarity index measure (FSIM) [62], learned perceptual image patch similarity (LPIPS) [63], and natural image quality evaluator (NIQE) [64]. SSIM, FSIM, LPIPS, and NIQE are all perceptual metrics, but they have different characteristics and applications. SSIM, FSIM, and LPIPS are referential metrics that measure the perceptual similarity between two images (e.g., original and reconstructed images) using different image features. SSIM evaluates the distortion in luminance, contrast, and structure; FSIM considers perceptually sensitive local features, such as phase congruency and gradient magnitude; and LPIPS uses higher-level “deep features” that are also significant for semantic tasks (such as classification). In short, SSIM, FSIM, and LPIPS estimate the feature-level distortions in a particular domain. Conversely, NIQE is a non-referential metric that assesses how natural an image appears by comparing its statistics with those of natural images. This matches the theoretical formulation of the DCP trade-off, which defines perceptual quality as the divergence between image distributions. Minimizing such a divergence is also the motivation for using GAN in our proposed DIC method. It is important to note that smaller values of the LPIPS and NIQE metrics indicate better performance, while larger values of the other metrics indicate better performance.

5.2. Comparison of Loss Terms’ Weights Settings

We use the loss terms corresponding to distortion, classification, and perception to optimize the codec network parameters simultaneously, and their weights have clear impacts on the RDCP tradeoff. This section discusses the impact of different weight settings on the DCP metrics through experiments. The proposed DIC achieves variable rate compression based on pixel-by-pixel RD optimization, and, therefore, we fix the weights of the rate and distortion loss terms. We set the weight of the rate to the weights of the rate and the distortion loss terms are fixed. The weight of the rate is set to 1 and $t_1=4\times {10}^{-4}$ and $t_2=2.8$, which determine the variable rate range. The different settings of ${\lambda }_{\mathrm{SFM}}$ and ${\lambda }_{\mathrm{GAN}}$ are as follows:

(A) ${\lambda }_{\mathrm{SFM}}=0,{\lambda }_{\mathrm{GAN}}=0$;

(B) ${\lambda }_{\mathrm{SFM}}=0.005,{\lambda }_{\mathrm{GAN}}=0$;

(C) ${\lambda }_{\mathrm{SFM}}=0.01,{\lambda }_{\mathrm{GAN}}=0$;

(D) ${\lambda }_{\mathrm{SFM}}=0.05,{\lambda }_{\mathrm{GAN}}=0$;

(E) ${\lambda }_{\mathrm{SFM}}=0.01,{\lambda }_{\mathrm{GAN}}=0.5$;

(F) ${\lambda }_{\mathrm{SFM}}=0.01,{\lambda }_{\mathrm{GAN}}=10$.

We train different DIC models according to the above settings, and then use 500 images extracted from the ImageNet dataset to evaluate the performance of the six different DIC models.

Figure 5 shows a performance comparison of all six cases in terms of classification accuracy, PSNR, and NIQE. Comparing the performance metrics under cases (A), (B), (C) and (D), the following phenomena can be observed: First, as ${\lambda }_{\mathrm{SFM}}$ increases, both top-1 and top-5 classification accuracy improve, but the improvement shows a decreasing trend. Second, increasing the ${\lambda }_{\mathrm{SFM}}$ within a certain range does not affect the PSNR. Third, increasing ${\lambda }_{\mathrm{SFM}}$ also improves the NIQE, but the improvement is not monotonically increasing with ${\lambda }_{\mathrm{SFM}}$. The comparison of cases (C), (E), and (F) shows that raising the ${\lambda }_{\mathrm{GAN}}$ increases the NIQE metrics while lowering the PSNR metrics. When ${\lambda }_{\mathrm{GAN}}$ is modest, classification accuracy is hardly impacted. When ${\lambda }_{\mathrm{GAN}}$ is set too high, classification accuracy suffers dramatically.

In conclusion, perception-distortion is a strict trade-off, while classification-perception and classification-distortion are not strict trade-offs. The relationship between distortion, classification, and perception can be balanced by reasonable ${\lambda }_{\mathrm{SFM}}$ and ${\lambda }_{\mathrm{GAN}}$ settings, and the degradation in distortion performance can be reduced while improving classification and perception performance.

5.3. Comparison of Qmap Generation Policies

The Qmap generation strategies provide control mechanisms for the reconstruction and classification trade-off at different rates under the proposed coding framework. Five Qmap generation strategies are introduced for specific rate coding, as follows: (A) all elements in the Qmap have the same values; (B) elements corresponding to the more complex regions in the Qmap have larger values; (C) elements corresponding to the regions of lower complexity in the Qmap have larger values; (D) elements corresponding to the region of higher semantic importance in the Qmap are larger; (E) the higher the semantic importance in the semantic salient region of the Qmap, the higher the value, while the lower the image complexity in the semantic non-salient regions, the higher the values. Strategies (A), (B), and (C) are optimized for the reconstruction; strategy (D) is optimized for the classification; and strategy (E) prioritizes ensuring the reconstruction quality in semantic salient regions and allocates limited bits to more pixels in semantic non-significant regions, thus improving the overall reconstruction quality. To compare the impacts of the Qmap generation strategies on classification and the reconstruction metrics, we first train the proposed DIC model according to the default hyperparameter configuration. Then, we use the above five Qmap generation strategies to encode images at different rates on the ImageNet dataset. Finally, we compare the classification accuracy, PSNR, and SSIM performance metrics of the reconstructed images obtained under different Qmap generation strategies. PSNR and SSIM measure the pixel-wise and structural similarity of the image to the original image, respectively.

Figure 6 shows a comparison of the classification accuracy, PSNR, and SSIM metrics of the reconstructed images with varied rates under different Qmap generation strategies. We use Unimap, ICM, 1-ICM, SIM, and Blended to denote the results obtained using strategies (A), (B), (C), (D), and (E), respectively. According to the comparison results, we can observe the following phenomena: First, in strategies (D) and (E), the pixels with high semantic importance have higher priority and better reconstruction quality at different rates, thus obtaining better classification accuracy than those obtained from other strategies. Second, strategies (A), (B), and (C) are all reconstruction-based allocation strategies but obtain different results. Among them, ICM favors complex pixel regions, which are difficult to compress, and raising the same bits brings a lower reconstruction gain; therefore, it has the worst performance among them. Unimap is oriented toward global average reconstruction optimization, so it has the best PSNR metrics. The strategy 1-ICM reduces the coding quality of complex regions and allocates the saved bits to more pixels, so the 1-ICM achieves the best SSIM metric, and the PSNR metric is also close to that of Unimap. Third, the blended strategy, which combines SIM and 1-ICM, achieves classification performance that is similar to that of the SIM-based strategy. Additionally, the PSNR and SSIM metrics for this strategy are close to those of the Unimap-based and 1-ICM-based strategies. The three phenomena mentioned above demonstrate that different strategies for generating Qmaps represent a trade-off between distortion and classification. The proposed blended Qmap generation strategy strikes a better balance between distortion and classification and performs well in both classification and distortion metrics. Therefore, the blended Qmap generation strategy, which combines SIM and ICM, is adopted as the Qmap generation strategy for different rate coding in subsequent sections.

5.4. Performance Evaluation and Comparison

In this subsection, we use evaluation metrics, including PSNR, SSIM, FSIM, LPIPS, NIQE, and classification accuracy, to assess the performance of the encoder-based variable rate image coding algorithm implemented for hybrid contexts. To evaluate classification accuracy, the VGG16 classifier is used to classify the original images in the ImageNet test set, achieving a Top-1 accuracy of $72\%$ and a Top-5 accuracy of $90\%$. The DIC model is first trained with default settings, and then the Qmap generation strategy based on a mixture of SIM and ICM is used to generate the Qmap and reconstruct images at different rates. Finally, the algorithm’s performance is evaluated on the ImageNet and Kodak datasets.

Ablation analysis. The model implemented in [13] is denoted as QMAP2021, the model trained with $\mathcal{L}1$ and $\mathcal{L}2$ is denoted as $\mathrm{HEVRC}\mathrm{w}/\mathrm{o}\mathrm{GAN}$, the decoder of the $\mathrm{HEVRC}\mathrm{w}/\mathrm{o}\mathrm{GAN}$ model is fine-tuned with the $\mathcal{L}3$, and the final model is denoted as HEVRC. Our approach $\mathrm{HEVRC}\mathrm{w}/\mathrm{o}\mathrm{GAN}$ improves QMAP2021 in three ways: First, ${\mathcal{L}}_{\mathrm{SFM}}$ is added to optimize the classification objective. Second, a mechanism for generating Qmap based on pixel context importance is designed to achieve encoding at any bitrate within the rate range. Third, both SIM and ICM are considered during the Qmap generation process. HEVRC introduced a GAN loss term on the basis of $\mathrm{HEVRC}\mathrm{w}/\mathrm{o}\mathrm{GAN}$ to optimize the perceptual objective. In order to achieve image encoding at any bitrate within the bitrate range, QMAP2021 generated Qmap according to the mechanism proposed in this paper based on SIM.

As shown in Figure 7, $\mathrm{HEVRC}\mathrm{w}/\mathrm{o}\mathrm{GAN}$ and QMAP2021 have similar PSNR, but $\mathrm{HEVRC}\mathrm{w}/\mathrm{o}\mathrm{GAN}$ outperforms QMAP2021 for other metrics. The most significant improvement is in classification accuracy, with average increases of $15\%$ and $13\%$ in the Top-1 and Top-5 classification accuracy, respectively, within the rate range. For NIQE, LPIPS, FSIM, and SSIM, the relative performance average improvements are about $9.7\%$, $23.5\%$, $2.1\%$, and $1.7\%$, respectively. These results demonstrate that appropriate weighting of ${\mathcal{L}}_{\mathrm{SFM}}$ can simultaneously improve classification and perception performance. HEVRC has a comparable performance in classification accuracy to $\mathrm{HEVRC}\mathrm{w}/\mathrm{o}\mathrm{GAN}$, but the superiority of HEVRC is mainly in the perception performance, with relative performance improvement values of $13.1\%$ and $15.5\%$ for NIQE and LPIPS, respectively. In terms of PSNR and SSIM performance, HEVRC has a 0.95 dB lower PSNR and a relative decrease of $3.9\%$ in SSIM metrics compared to $\mathrm{HEVRC}\mathrm{w}/\mathrm{o}\mathrm{GAN}$. The comparison between HEVRC and $\mathrm{HEVRC}\mathrm{w}/\mathrm{o}\mathrm{GAN}$ further confirms the trade-off between perception and distortion.

RDCP trade-off performance analysis. The above ablation analysis shows that the proposed encoding framework and rate-adaptive mechanism can improve the performance of DIC for hybrid contexts of classification and reconstruction. To further evaluate its performance, this section selects three other DIC algorithms, and one traditional image compression algorithm, to compare with the two algorithms implemented in this paper, HEVRC and $\mathrm{HEVRC}\mathrm{w}/\mathrm{o}\mathrm{GAN}$. The algorithms for comparison include: BMSHJ2018 [24], MBT2018 [25], CHENG2020 [10], BM2022 [40] and BPG [65]. Among them, BMSHJ2018, MBT2018, and CHENG2020 train DIC models at different bit rates by changing Lagrange multipliers, and they are single-context DIC algorithms aimed for reconstruction. BM2022 is a state-of-the-art variable-rate DIC algorithm for image reconstruction context.

In terms of classification accuracy, the performance of HEVRC and $\mathrm{HEVRC}\mathrm{w}/\mathrm{o}\mathrm{GAN}$ is superior to other algorithms in the rate range. The average Top-1 and Top-5 classification accuracy of the two proposed algorithms is approximately $7\%$ higher than that of the BPG algorithm, and about $9\%$ higher than other DIC algorithms. The superior performance of these two algorithms in classification accuracy is mainly due to two factors. First, the ${\mathcal{L}}_{\mathrm{SFM}}$ loss term is added in model training. This loss term can better preserve semantic relevant information during the encoding and decoding. Second, a strategy of using a combination of SIM and ICM is adopted to generate Qmap, which better protects the pixel reconstruction of semantic salient regions at different bit rates.

In terms of PSNR performance, HEVRC and $\mathrm{HEVRC}\mathrm{w}/\mathrm{o}\mathrm{GAN}$ are approximately 0.6 dB and 0.3 dB worse than BM2022, respectively. The main reason is that BM2022 is a quantization-based variable-rate DIC and its backbone model uses the autoregressive entropy model to further eliminate redundancy between pixels. Moreover, during the training process of HEVRC, a GAN loss term is added, which encourages the joint distribution of decoded image pixels to resemble the training dataset. This is different from aiming for high fidelity of each pixel. Therefore, the PSNR performance of HEVRC is worse than BM2022. Due to the superior performance of the backbone network compared to MBT2018, BMSHJ2018, and BPG, the PSNR performance of the two proposed DIC models is better than all three algorithms.

In terms of perceptual metrics, both HEVRC and $\mathrm{HEVRC}\mathrm{w}/\mathrm{o}\mathrm{GAN}$ achieve better performance in FSIM, LPIPS, and NIQE, indicating that both ${\mathcal{L}}_{\mathrm{SFM}}$ and ${\mathcal{L}}_{\mathrm{GAN}}$ can improve these perceptual metrics. The results show that adding ${\mathcal{L}}_{\mathrm{GAN}}$ further improves all three metrics. The main purpose of using GAN is to make the reconstructed image more realistic, which is consistent with NIQE. Therefore, HEVRC with ${\mathcal{L}}_{\mathrm{GAN}}$ achieves the best NIQE performance.

Generalization to other datasets. All algorithms implemented in this section use the COCO dataset for training and are tested on the ImageNet dataset in comparison with other algorithms. The results show that the proposed algorithm still performs well when the training and test datasets are different. To further test the generalizability of the proposed DIC models to different datasets, we use the Kodak dataset to evaluate the performance of the proposed algorithms in terms of the PSNR, SSIM, FSIM, LPIPS, and NIQE metrics. Figure 8 shows the performance comparison. The results indicate that the performance of our proposed algorithms on the Kodak dataset is consistent with that on the ImageNet dataset.

Adaptation analysis of different classification models. In the experiments above, the same feature extractor network is used for both semantic analysis and SFM loss calculation in HEVRC. To validate the adaptability of our proposed method to different downstream classifiers, we further evaluated the classification performance of HEVRC using four other pre-trained classification models. Figure 9 shows the results of classifying the reconstructed images using Resnet18 [66], Inceptionv3 [67], Googlenet [68], and Alexnet [69] on the ImageNet dataset. The results show that HEVRC achieves the best classification performance in different classification models, and the classification accuracy is close to the original image’s accuracy at higher bitrates. These results demonstrate that the proposed SFM loss-based method for improving the classification accuracy of reconstructed images has a certain generality and can adapt to multiple different classification models. In addition, the Qmap generation mechanism based on the fusion of SIM and ICM also shows generality in protecting semantic information at different bitrates across different classification models.

Visual effects. The proposed DIC framework uses SFM and GAN loss terms to optimize the similarity of high-level semantic features and the distribution similarity between the reconstructed image and the original image, respectively. The high-level semantic similarity makes the reconstructed image easier to recognize, while the distribution similarity makes the reconstructed image look more realistic and perceptually better. In Qmap-based rate-adaptive coding, a Qmap generation mechanism combines SIM and ICM to ensure priority reconstruction of semantic salient regions while also considering the reconstruction of semantic non-salient regions. In the previous sections, we showed that the HEVRC algorithm achieved the best performance in both classification accuracy and perceptual quality in the quantitative performance comparison analysis. In this section, we select two typical images from the Kodak dataset and compare the visual effects of HEVRC with QMAP2021, BM2022 and BPG to demonstrate the superior visual performance of HEVRC.

Figure 10 shows the reconstructed images produced by four different algorithms at various bitrates. The following observations can be made: First, at lower bitrates, BPG’s reconstructed images show block artifacts, while QMAP2021, BM2022, and HEVRC do not. This is because BPG uses block coding, such that differences between blocks at low bitrates can cause block artifacts. In contrast, DIC algorithms encode the whole image. Second, the reconstructed images produced by BPG, BM2022, and QMAP2021 look blurry, especially at low bitrates, while those produced by HEVRC look sharper and more realistic. This is because BPG, BM2022, and QMAP2021 are optimized with MSE loss, while HEVRC adds the GAN loss, which is known to yield perceptually sharper and more realistic images. Third, in HEVRC and QMAP2021, the pixels in the regions most relevant to the classification category have better reconstruction quality (e.g., the parrot’s head and lighthouse top), even at lower bit rates. Moreover, when comparing the non-salient regions of the reconstructed images generated by HEVRC, BM2022, and QMAP2021, HEVRC has better visual quality for objects such as clouds, grass, and road signs in the background. This is mainly due to the GAN loss and SFM loss, which can better preserve the contour information in the entire image. Furthermore, during the Qmap generation process, HEVRC improves the overall coding quality by suppressing the coding quality of complex regions, providing higher reconstruction quality for a larger range of pixels at the same bitrate.

6. Conclusions

This paper proposes an encoder-based DIC with classification-driven semantic awareness. The proposed algorithm expands the conditional encoder-based rate-adaptive DIC to hybrid contexts in three ways. First, the loss function is designed for RDCP optimization, which is the weighted sum of rate, MSE loss, SFM loss, and GAN loss. Second, the objective-by-objective optimization training approach is introduced to train DIC models in hybrid contexts. Third, the Qmap generation mechanism, which takes into account both image complexity and semantic importance, is designed to control the distortion-classification trade-off during bits assignment. The experimental results obtained on multiple open datasets show that, compared with the benchmark algorithm QMAP2021, the proposed DIC can achieve an average improvement in classification accuracy of $11\%$, along with other improvements in the LPIPS, NIQE, and FSIM metrics, ranging from $1.3\%$ to $32\%$. However, the improvement comes with a reasonable cost in PSNR reduction around 0.5 dB, and degradation in SSIM performance around $2.3\%$. In summary, the proposed DIC can flexibly control the RDCP trade-off performance, such that the reconstructed images have good visual perception and superior accuracy in downstream classification tasks.

In this paper, adaptive coding in hybrid contexts is achieved by conditional encoders. However, the adaptive process involves repeated calls to the encoder, which increases computational complexity and processing delay. In the future, distillation learning methods can be explored to obtain encoders with lower complexity but similar performance. In addition, the encoding quality of each pixel is assigned based on the relative values of SIM and ICM in the proposed algorithm. We note that this assignment policy is essentially a heuristic policy instead of an optimal one. To further improve the trade-off performance of RDCP, one may formulate a multi-objective optimization problem to explore the Pareto-front of the performance trade-off. Some initial efforts towards a theoretical understanding of this trade-off can be found in [6].