Effects of Different Full-Reference Quality Assessment Metrics in End-to-End Deep Video Coding

Abstract

Visual quality assessment is often used as a key performance indicator (KPI) to evaluate the performance of electronic devices. There exists a significant association between visual quality assessment and electronic devices. In this paper, we bring attention to alternative choices of perceptual loss function for end-to-end deep video coding (E2E-DVC), which can be used to reduce the amount of data generated by electronic sensors and other sources. Thus, we analyze the effects of different full-reference quality assessment (FR-QA) metrics on E2E-DVC. First, we select five optimization-suitable FR-QA metrics as perceptual objectives, which are differentiable and thus support back propagation, and use them to optimize an E2E-DVC model. Second, we analyze the rate–distortion (R-D) behaviors of an E2E-DVC model under different loss function optimizations. Third, we carry out subjective human perceptual tests on the reconstructed videos to show the performance of different FR-QA optimizations on subjective visual quality. This study reveals the effects of the competing FR-QA metrics on E2E-DVC and provides a guide for further future study on E2E-DVC in terms of perceptual loss function design.

1. Introduction

Visual quality assessment pertains to the evaluation of the aesthetic presentation of images or videos. Electronic products serve as the means by which these visual media are displayed. Thus, there exists a significant association between visual quality assessment and electronic devices, given that the quality of the latter directly impacts the visual presentation of media content. Employing visual quality assessment methods can effectively gauge the optimal display performance of electronic products, thereby enabling the enhancement of user enjoyment and satisfaction, as well as facilitating quality control and improvement of electronic devices.

Visual quality evaluation focuses on users’ subjective perception of images or video quality, while security privacy evaluation focuses on the degree to which sensitive information is protected and the ability to mitigate potential threats. The combination of these evaluation methods can ensure that the system or product not only provides a high-quality user experience but also has the necessary security and privacy protection capabilities.

In the context of electronics devices, visual quality assessment can be used to evaluate the performance of systems that use visual sensors [1], such as cameras, to monitor physical processes. For example, in a manufacturing plant, cameras may be used to monitor the quality of products as they move through the production line. The visual data captured by these cameras can then be analyzed to detect defects or other issues that may affect product quality. In addition, electronic devices generate and store large amounts of data from sensors and other sources [2], which must be processed and transmitted efficiently to enable real-time control [3] and decision making. Therefore, compression technology [4,5] is important for electronic devices to reduce the amount of data generated by sensors and other sources [6,7], which can help to optimize system performance. Obviously, a good quality assessment (QA) metric is the key to guide and evaluate the compression coding.

Since the original distortion-free images/frames are available during the coding process, the QA metrics referred to in this paper are full-reference (FR) metrics, which derive the quality of a distorted image/frame by comparing it with the original one. Nowadays, some FR-QA metrics, such as mean squared error (MSE) and peak signal-to-noise ratio (PSNR), are already widely used in video compression, such as High Efficiency Video Coding (HEVC) [8] and Versatile Video Coding (VVC) [9]. However, many FR-QR metrics have poor correlation with the perception of the human visual system (HVS) and are not precise enough to reflect the perceptual quality of an image. To address this issue, a variety of perceptual FR-QA metrics have been developed in the past two decades, which can be divided in to early prior knowledge based models [10,11], traditional machine learning based algorithms [12,13], and the latest, emerging end-to-end deep-learning-based models [14,15].

In traditional hybrid video coding frameworks (shown in Figure 1), such as AVC, HEVC, and VVC, coding tools basically perform rate–distortion optimization on their own within a specific codec structure. Currently, modularized neural video coding (MOD-NVC) integrates a modular learning-based coding tool into the traditional video coding framework, either as a stand-alone module replacing the original module, as an enhancement to the original module, or as a part of the coding strategy. MOD-NVC can improve the performance of traditional video coding algorithms but is still constrained by the coupling between coding tools; so, MOD-NVC is a local optimization of the traditional hybrid video coding framework. In contrast, end-to-end deep video coding (E2E-DVC) can overcome these shortcomings and enable a simple replacement of the network model to an overall end-to-end optimization. As shown in Figure 1, an E2E-DVC model is essentially a deep learning model based on autoencoders (AE), and most existing E2E-DVC models still follow the basic idea of hybrid coding. Currently, a variety of different neural network structures are used in the design of AE, including the use of convolutional neural network (CNN), recurrent neural network (RNN), generative adversarial network (GAN), long short-term memory (LSTM), gated recurrent unit (GRU), and attention mechanism [16], which significantly improve the E2E-DVC models’ performance.

The choice of a loss function is an important factor when designing and configuring neural networks. By using a stochastic gradient descent (SGD) algorithm to decrease a differentiable loss function, parameters of a neural networks model can be optimized. There have been some studies on loss functions for image processing [17], such as image restoration [18]. However, the impact of the loss functions has not received much attention in the context of E2E-DVC. Most existing E2E-DVC models use MSE or SSIM [19] as a part of the loss function. Since an important application of the FR-QA metrics is evaluating and guiding image/video compression coding, there are many other FR-QA metrics to be chosen as a loss function, and it can be challenging to know what to choose or even what role it plays.

To address the above problem, we bring attention to alternative choices of perceptual loss function for E2E-DVC and analyze the effects of different FR-QA metrics on E2E-DVC. The main contributions of this study are as follows:

• We select five optimization-suitable FR-QA metrics as perceptual objectives, which are differentiable and thus support back propagation, and use them to optimize an E2E-DVC model.
• We analyze the rate–distortion (R-D) behaviors of an E2E-DVC model under under different loss function optimizations, which provides a basis for further future study of the rate control of E2E-DVC.
• We conduct subjective human perceptual tests on the reconstructed videos to show the effects of different FR-QA optimizations on subjective visual quality and reveal the relative performance of the competing FR-QA metrics.

The remainder of this paper is structured in the following way. The relevant research is presented in Section 2. Section 3 describes the study in detail. In Section 4, comprehensive experiments are conducted to discuss the effects of different FR-QA optimizations from various aspects. Finally, we draw a conclusion in Section 5.

2. Related Work

2.1. End-to-End Deep Video Coding

The classic E2E-DVC model was proposed by Lu et al. [20] in 2019, showing the possibilities of E2E video coding techniques. Most existing E2E-DVC models that emerged subsequently still follow the framework of traditional hybrid coding, using supervised learning methods with different algorithms to efficiently represent spatial texture, temporal motion, and prediction residuals. Chen et al. [21] extended their nonlocal attention optimization and improved context (NLAIC) method for intraframe and residual coding and used second-order stream-to-stream prediction for a more compact motion representation, showing consistent R-D gain across content and bit rates. Rippel et al. [22] used composite features for the joint encoding of motion streams and residuals, embedded in aggregated multiframe information, to efficiently generate the encoding of motion streams and residuals. For the effective representation of temporal information in video coding, Liu et al. [23] proposed a learned video compression method via joint spatial–temporal correlation exploration. In this method, temporal correlation was predicted by first-order optical flow and second-order optical flow, and a single-stage unsupervised learning method was used to encapsulate the optical flow into quantized successive frame features, followed by context-adaptive entropy coding to remove second-order correlation.

2.2. QA Metrics for Perceptual Optimization

The use of FR-QA metrics as targets for designing and optimizing new image processing algorithms is one of its less studied but very promising applications [24]. The tuning of model parameters is usually performed in image processing systems, with minimizing MSE as the objective function. SSIM [19] was used in early studies for perceptual optimization and applied in image recovery, wireless video streaming, and image synthesis. Zhao et al. [18] found that the optimization of image recovery tasks using mean absolute error (MAE, ${\ell }_1$-norm) was better than using MSE (${\ell }_2$-norm). A multiscale structural similarity metric (MS-SSIM) [25] is often used to guide neural-network-based image super-resolution and image compression. Laparra et al. [26] used normalized Laplace pyramid-based metrics [27] to optimize image rendering algorithms. Ding et al. [17] used different FR-QA metrics as loss functions to train deep neural networks for low-level image processing tasks, including denoising, deblurring, super-resolution, and image compression. Zhang et al. [28] designed a mixed loss function containing MSE and a mixed loss function for perceptual optimization of deep video compression.

3. FR-QA Model for Perceptual Optimization

3.1. Deep Video Compression

As shown in Figure 2, the encoding procedure of the deep video compression (DVC) [20] in the encoder side is as follows.

Step 1. Motion estimation:

$$v_t=OPFN(x_t,{\widehat{x}}_{t-1}),$$

(1)

where $x_t$ is the frame at time step t (current frame), and ${\widehat{x}}_t$ is the reconstructed frame. $OPFN$ is the optical flow net, and the output $v_t$ is the optical flow map.

Step 2. Motion information coding and quantification:

$$m_t=ME\left(v_t\right),$$

(2)

$${\widehat{m}}_t=Quantizer\left(m_t\right),$$

(3)

where $m_t$ is the result of the optical flow map encoded by an automatic encoder $ME$, and ${\widehat{m}}_t$ is the quantized version of $m_t$.

Step 3. Motion compensation:

$${\widehat{v}}_t=MD\left({\widehat{m}}_t\right),$$

(4)

$${\overline{x}}_t=MCNet({\widehat{x}}_{t-1},{\widehat{v}}_t),$$

(5)

where $MD$ is the motion information decoder, $MCNet$ is the motion compensation net, ${\widehat{v}}^t$ is the reconstructed optical flow map, and ${\overline{x}}_t$ is the predicted frame.

Step 4. Residual coding and quantification:

$$r_t=x_t-{\overline{x}}_t,$$

(6)

$$y_t=RE\left(r_t\right),$$

(7)

$${\widehat{y}}_t=Quantizer\left(y_t\right),$$

(8)

where $r_t$ is the residual between $x_t$ and ${\overline{x}}_t$, $y_t$ is the result of $r_t$ encoded by the residual encoder $RE$, and ${\widehat{y}}_t$ is the quantization result of $y_t$.

Step 5. Frame reconstruction:

$${\widehat{r}}_t=RD\left({\widehat{y}}_t\right),$$

(9)

$${\widehat{x}}_t={\overline{x}}_t+{\widehat{r}}_t,$$

(10)

where $RE$ is the residual decoder, and ${\widehat{r}}^t$ and ${\widehat{x}}_t$ represent the reconstructed residual and reconstructed frame, respectively.

The loss function of the model is rate–distortion loss, which is defined as follows:

$$L=\left(R\left({\widehat{m}}_t\right)+R\left({\widehat{y}}_t\right)\right)+\lambda ·d(x_t,{\widehat{x}}_t),$$

(11)

where $d(x_t,{\widehat{x}}_t)$ denotes the perceptual distortion between $x_t$ and ${\widehat{x}}_t$, and $R(·)$ represents the number of bits used for encoding the representations. $\lambda$ is the super-parameter that balances the rate–distortion trade-off, commonly referred to as the Lagrangian coefficient.

3.2. Perceptual Quality Optimization

Since the loss function should encourage E2E-DVC models to reconstruct natural and perceptually pleasing videos, we chose five commonly used or the state-of-the-art FR-QA metrics to guide the learning of DVC model as follows:

1.

Mean Absolute Error (MAE).

MAE, a popular error measurement method, is intuitive because, unlike RMSE, its variation is linear. MSE and RMSE have more of a “penalty” for larger errors, because the square of the error value increases the average error value. In MAE, the MAE value is measured as the average of the absolute error values, and the different errors are not weighted more or less but increase linearly as the error increases. When the MAE value is used as the quality loss in Equation (11), it is calculated as follows:

$$d_{MSE}(x_t,{\widehat{x}}_t)=\frac{{\sum }_{i=1}^W{\sum }_{j=1}^H\left|x_t(i,j)-{\widehat{x}}_t(i,j)\right|}{W\times H},$$

(12)

where $W\times H$ is the resolution of the original image $x_t$ and the reconstructed image ${\widehat{x}}_t$, and $(i,j)$ is the location coordinate of the pixel point. Although MAE focuses only on pixel-level changes in the image, it has shown better performance in image restoration tasks [18].

2.

Multiscale Structural Similarity (MS-SSIM)

Unlike MAE, which compares pixel-level differences across an image, MS-SSIM [25] works on blocks of sliding windows. This approach better simulates the HVS function, because HVS can easily perceive local information differences in specific regions of two images rather than individual differences in pixel values over the entire region. MS-SSIM, as an SSIM-based method, extends SSIM at a single scale by retaining the results after multiple downsamplings to measure changes in structural information at multiple scales. When the MS-SSIM value is used as the quality optimization objective in Equation (11), it is calculated as follows:

$$d_{MS-SSIM}(x_t,{\widehat{x}}_t)=1-\frac{{\left[l_M(i,j)\right]}^{{\alpha }_M}{\prod }_{m=1}^M{\left[c_m(i,j)\right]}^{{\beta }_m}{\left[s_j(i,j)\right]}^{{\gamma }_m}}{W\times H},$$

(13)

where $W\times H$ is the resolution of the original image $x_t$ and the reconstructed image ${\widehat{x}}_t$, $(i,j)$ is the location coordinate of the pixel point, and $l_m(i,j)$, $c_m(i,j)$, and $s_m(i,j)$ are the luminance contrast factor, contrast factor, and structure contrast factor of the original image at $(i,j)$ for the sliding window block at the h-th scale, respectively. For the original image, i.e., $m=1$, it is calculated as follows:

$$l_1(i,j)=\frac{2{\mu }_{x_t}(i,j)·{\mu }_{{\widehat{x}}_t}(i,j)+C_1}{{\mu }_{x_t}^2(i,j)+{\mu }_{{\widehat{x}}_t}^2(i,j)+C_1},$$

(14)

$$c_1(i,j)=\frac{2{\sigma }_{x_t}(i,j)·{\sigma }_{{\widehat{x}}_t}(i,j)+C_2}{{\sigma }_{x_t}^2(i,j)+{\sigma }_{{\widehat{x}}_t}^2(i,j)+C_2},$$

(15)

$$s_1(i,j)=\frac{{\sigma }_{x_t{\widehat{x}}_t(i,j)}+C_3}{{\sigma }_{x_t(i,j)}{\sigma }_{{\widehat{x}}_t(i,j)}+C_3},$$

(16)

where ${\mu }_{x_t}(i,j)$ and ${\mu }_{{\widehat{x}}_t}(i,j)$ are the means of $x_t$ and ${\widehat{x}}_t$ within the sliding window block at $(i,j)$, respectively. ${\sigma }_{x_t}^2(i,j)$ and ${\sigma }_{{\widehat{x}}_t}^2(i,j)$ are the variances of $x_t$ and ${\widehat{x}}_t$ within the sliding window block at $(i,j)$, respectively. ${\sigma }_{x_t{\widehat{x}}_t(i,j)}$ is the covariance between $x_t$ and ${\widehat{x}}_t$ within the sliding window block at $(i,j)$. The three fixed constants $C_1$, $C_2$, and $C_3$ in the formulas are set to $C_1={\left(K_{1}L\right)}^2$, $C_2={\left(K_{2}L\right)}^2$, and $C_3=C_2/2$, where L is the dynamic range of the pixel values (256 for 8-bit images), and the default settings for $K_1$ and $K_2$ are 0.01 and 0.03, respectively. For the m-th scale, $l_m(i,j)$, $c_m(i,j)$, and $s_m(i,j)$ are calculated on the image obtained after m low-pass filtering and downsampling with a factor of 2 (i.e., the image is reduced by $2^{m-1}$), and their calculation formulas are the same as $l_m(i,j)$, $c_m(i,j)$, and $s_m(i,j)$.

3.

Visual Information Fidelity (VIF)

VIF [10] is an FR-QA method based on natural scene statistics and HVS to extract image information concepts, which uses information-theoretic metrics to measure image fidelity. In the information-theoretic framework, the reference image is modeled as the output of a random “natural” source that passes through the HVS channel and is subsequently processed by the brain. The information content of the reference image is quantified as the mutual information between the input and output of the HVS channel, which is the information that the brain can ideally extract from the reference image. Similarly, in the presence of an image distortion channel, the information capacity of the distorted image is quantified as the mutual information between the input of the distortion channel and the output of the HVS channel of the test image, which is the information that the brain can ideally extract from the distorted image.

VIF uses the Gaussian scale mixture (GSM) method to statistically model the wavelet coefficients after image pyramidal decomposition, i.e., an HVS multichannel model for a given sub-band of multiscale multidirectional decomposition.

The original image information model is

$$\mathcal{C}=\mathcal{S}·\mathcal{U}=\{{\overrightarrow{C}}_i=S_i{\overrightarrow{U}}_i:i\in \mathcal{I}\},$$

(17)

where $\mathcal{S}=\{S_i:i\in \mathcal{I}\}$ is a scalar field taking positive values, $\mathcal{U}=\{{\overrightarrow{U}}_i:i\in \mathcal{I}\}$ is a Gaussian vector field with mean 0 and covariance ${\mathbf{C}}_U$, and $\mathcal{I}$ is a spatial index set of sub-bands.

The distorted image information model is

$$\mathcal{D}=\mathcal{G}·\mathcal{C}+\mathcal{V}=\{{\overrightarrow{D}}_i=g_i{\overrightarrow{C}}_i+{\overrightarrow{V}}_i:i\in \mathcal{I}\},$$

(18)

where $\mathcal{G}=\{g_i:i\in \mathcal{I}\}$ is a deterministic scalar gain field, $\mathcal{V}=\{{\overrightarrow{V}}_i:i\in \mathcal{I}\}$ is a Gaussian vector field with mean 0 and covariance ${\mathbf{C}}_V={\sigma }_v^2\mathbf{I}$, and $\mathbf{I}$ is the identity matrix.

The HVS output models for the original image and the distorted image, respectively, are

$$\mathcal{E}=\mathcal{C}+\mathcal{N}=\{{\overrightarrow{E}}_i={\overrightarrow{C}}_i+{\overrightarrow{N}}_i:i\in \mathcal{I}\},$$

(19)

$$\mathcal{F}=\mathcal{D}+{\mathcal{N}}^{\prime }=\{{\overrightarrow{F}}_i={\overrightarrow{D}}_i+{\overrightarrow{N}}_i^{\prime }:i\in \mathcal{I}\},$$

(20)

where both $\mathcal{N}$ and ${\mathcal{N}}^{\prime }$ are Gaussian vector fields with mean 0 and covariance ${\mathbf{C}}_N={\sigma }_n^2\mathbf{I}$.

Let the superscript indicate that there are $N_j$ blocks in the j-th sub-band, then when VIF is used as the quality optimization objective in Equation (11), it is calculated as follows:

$$d_{VIF}(x_t,{\widehat{x}}_t)=1-\frac{{\sum }_{j\in \mathrm{subbands}}I({\overrightarrow{C}}^{N_j};{\overrightarrow{F}}^{N_j}\mid s^{N_j})}{{\sum }_{j\in \mathrm{subbands}}I({\overrightarrow{C}}^{N_j};{\overrightarrow{E}}^{N_j}\mid s^{N_j})},$$

(21)

where $s^{N_j}$ is the maximum likelihood estimate of $S^{N_j}$. $I({\overrightarrow{C}}^{N_j};{\overrightarrow{E}}^{N_j}\mid s^{N_j})$ is the mutual information between $\mathcal{E}$ and $\mathcal{C}$, i.e., the information that the brain gets from the original image $x_t$; $I({\overrightarrow{C}}^{N_j};{\overrightarrow{F}}^{N_j}\mid s^{N_j})$ is the mutual information between $\mathcal{F}$ and $\mathcal{C}$, i.e., the information that the brain gets from the reconstructed image ${\widehat{x}}_t$, which can be calculated as follows:

$$I({\overrightarrow{C}}^{N_j};{\overrightarrow{E}}^{N_j}\mid s^{N_j})=\frac{1}{2}{\sum }_{i=1}^{N_j}{log}_2\left(\frac{\mid s_i^2{\mathbf{C}}_U+{\sigma }_n^2\mathbf{I}\mid }{\mid {\sigma }_n^2\mathbf{I}\mid }\right),$$

(22)

$$I({\overrightarrow{C}}^{N_j};{\overrightarrow{F}}^{N_j}\mid s^{N_j})=\frac{1}{2}{\sum }_{i=1}^N{log}_2\left(\frac{\mid g_i^2{s}_i^2{\mathbf{C}}_U+({\sigma }_v^2+{\sigma }_n^2)\mathbf{I}\mid }{\mid ({\sigma }_v^2+{\sigma }_n^2)\mathbf{I}\mid }\right).$$

(23)

4.

Learned Perceptual Image Patch Similarity (LPIPS)

LPIPS [29], as a deep-learning-based approach, uses a CNN feature extractor to generate distances between low-dimensional features for the reference and distorted images and then a shallow network to produce a better probability of one over the other. the LPIPS model also solves the problem caused by the extraction of different ranges of features by each layer and introduces learning weights to weight the importance of different features.

Let the feature map of the original image $x_t$ obtained after the l-th layer of convolution be $x_t^l$ with a size of $H_l\times W_l\times C_l$. Let the feature of $x_t^l$ on the k-th channel be $x_t^{l,k}$; the associated symbols of the reconstructed image ${\widehat{x}}_t$ are defined similarly. When the LPIPS value is used as the quality loss in Equation (11), it is calculated as follows:

$$d_{LPIPS}(x_i,{\widehat{x}}_i)={\sum }_l\frac{1}{W_i\times H_i}{\sum }_{i=1}^{W_j}{\sum }_{j=1}^{H_j}{\sum }_{k=1}^{C_l}{\left[w_{i,k}·\left(x_i^{l,k}(i,j)-{\widehat{x}}_t^{l,\widehat{k}}(i,j)\right)\right]}^2$$

(24)

where $w_{i,k}$ is the weight parameter obtained by LPIPS after training.

5.

Deep Image Structure and Texture Similarity (DISTS)

DISTS [30], as a deep-learning-based model, is based on a pretrained VGG network as a feature extractor of the image and converts the image nonlinearly into a multiscale overcomplete representation, whose quality is then evaluated by a distance metric. Similar to the PIPAL, DISTS allows for small inconsistencies between the distorted and reference images in the texture information. DISTS can be seen as a version of the SSIM method in the depth domain, but DISTS contains only the luminance factor and the structure factor.

Let the feature map of the original image $x_t$ obtained after the l-th layer of convolution be $x_t^l$, with a size of $H_l\times W_l\times C_l$. Let the feature of $x_t^l$ on the k-th channel be $x_t^{l,k}$; the associated symbols of the reconstructed image ${\widehat{x}}_t$ are defined similarly. When the LPIPS value is used as the quality loss in Equation (11), it is calculated as follows:

$$d_{DISTS}(x_t,{\widehat{x}}_t)=1-{\sum }_l{\sum }_{k=1}^{C_l}\left({\alpha }_{l,k}L(x_t^{l,k},{\widehat{x}}_t^{l,k})+{\beta }_{l,k}S(x_t^{l,k},{\widehat{x}}_t^{l,k})\right)$$

(25)

where ${\alpha }_{l,k}$ and ${\beta }_{l,k}$ are the weight parameters obtained from DISTS after training, and $L(x_t^{l,k},{\widehat{x}}_t^{l,k})$ and $S(x_t^{l,k},{\widehat{x}}_t^{l,k})$ are the luminance factor and structure factor, respectively, which are defined as follows:

$$L(x_t^{l,k},{\widehat{x}}_t^{l,k})=\frac{2{\mu }_{x_t}^{l,k}{\mu }_{{\widehat{x}}_t}^{l,k}+C_1}{{\left({\mu }_{x_t}^{l,k}\right)}^2+{\left({\mu }_{{\widehat{x}}_t}^{l,k}\right)}^2+C_1},$$

(26)

$$S(x_t^{l,k},{\widehat{x}}_t^{l,k})=\frac{2{\sigma }_{x_t,{\widehat{x}}_t}^{l,k}+C_2}{{\left({\sigma }_{x_t}^{l,k}\right)}^2+{\left({\sigma }_{{\widehat{x}}_t}^{l,k}\right)}^2+C_2},$$

(27)

where ${\mu }_{x_t}^{l,k}$ and ${\mu }_{{\widehat{x}}_t}^{l,k}$ are the means of $x_t^{l,k}$ and ${\widehat{x}}_t^{l,k}$, respectively, ${\sigma }_{x_t}^{l,k}$ and ${\sigma }_{{\widehat{x}}_t}^{l,k}$ are the standard deviations of $x_t^{l,k}$ and ${\widehat{x}}_t^{l,k}$, respectively, and ${\sigma }_{x_t,{\widehat{x}}_t}^{l,k}$ is the covariance between $x_t^{l,k}$ and ${\widehat{x}}_t^{l,k}$. $C_1$ and $C_2$ are the same as in Equations (14) and (15).

Since the scales of these metrics vary widely, we normalized them as follows for a fair comparison when training the DVC model. We normalize the difference mean opinion scores (DMOS) in the LIVE database [31] (the lower the value, the better the perceptual quality) into the range [0, 1] by min-max scaling. For each FR-QA metric Q, a nonlinear four-parameter logistic (4PL) regression is used to fit the relationship between normalized DMOS and Q as follows:

$$\tilde{Q}(x_t,{\widehat{x}}_t)=\frac{k_1-k_2}{1+exp\left(k_3(Q(x_t,{\widehat{x}}_t)-k_4)\right)}+k_2,$$

(28)

where $\tilde{Q}$ is the predicted quality score, and $k_i$$(i=1,2,3,4)$ are parameters to be estimated. Thus, the above five FR-QA metrics are mapped to around [0, 1] using the 4PL functions. This processing allows the DVC models trained with the same $\lambda$ but with different FR-QA optimizations to encode a video at the most similar bit rates as possible.

Since the five selected FR-QA metrics are differentiable almost everywhere, and the 4PL function is also differentiable, $\tilde{Q}$ supports backpropagation. In addition, as $\tilde{Q}(x_t,{\widehat{x}}_t)$ is an approximation to DMOS of ${\widehat{x}}_t$, a smaller $\tilde{Q}(x_t,{\widehat{x}}_t)$ indicates a higher perceptual quality of the reconstructed frame ${\widehat{x}}_t$. Therefore, in order to make the videos reconstructed by the DVC model have better perceptual quality, we optimize the DVC model by making $d(x_t,{\widehat{x}}_t)$ in Equation (11) equal to $\tilde{Q}(x_t,{\widehat{x}}_t)$ in Equation (28). In addition, the normalization method used in this section allows the E2E-DVC model to obtain encoded videos with as similar bit rates as possible when trained with the same and different FR-QA optimizations.

4. Experiments

4.1. Implementation Details

We train (PyTorch 1.7.1, Nvidia GeForce GTX 3090 GPU with 24 GB memory, and Intel${}^{\circledR }$ Core${}^{\mathrm{TM}}$ i9-12900K CPU with 64 GB memory) the DVC on the Vimeo-90k [32], which is built for evaluating different video processing tasks, such as video interpolation, video denoising/deblocking, and video super-resolution. The dataset contains 89,800 video clips covering a rich variety of video content. The model is trained using the Adam optimizer with parameters set to ${\beta }_1=0.9$ and ${\beta }_2=0.999$, the batch size is set to 4, with an initial learning rate of 0.0001, and the learning rate is divided by 10 when the loss becomes stable. Both the codes for DVC and FR-QA metrics are online available at [33]. The rate–distortion weight factor $\lambda$ is set to 256,512,1024,2048, and the resolution of the images for training is 256 × 256.

During the testing period, the HEVC standard test sequence was used to evaluate the performance of E2E-DVC, which is widely used to measure the performance of video compression algorithms because of the diversity of video content and resolutions in this dataset.

4.2. Objective Results

4.2.1. R-D Performance

As shown in Figure 3, we compare the coding perfomance of the DVC models optimized under different FR-QA loss functions on the HEVC Class B sequences. Since peak signal-to-noise ratio (PSNR) is the most commonly used metric for lossy compression codecs, we use it to describe R-D performance. The detailed PSNR values are shown in

Table 1. Objective or subjective comparisons.

	DISTS	LPIPS	VIF	MS-SSIM	MAE
MOS	4.66	4.20	1.66	2.86	2.68
PSNR	24.6	24.2	22.6	25.4	25.7

. We also found that the R-D data can be well fitted by the following function:

$$PSNR=\alpha ·log\left(R\right)+\beta ,$$

(29)

where $\alpha$ and $\beta$ are parameters to be estimated, which are determined by the codec models and video content.

4.2.2. Objective Model Ranking

For a fairer comparison, the five FR-IQAs self-evaluated and mutually evaluated the videos encoded and reconstructed by their respective optimized E2E-DVC models under the same settings. The average ranking on HEVC Class B sequences is shown in Figure 4, where 1 indicates the best and 5 indicates the worst.

From Figure 4, it can be seen that the traditional MAE and MS-SSIM methods consider the deep-learning-based methods to perform more poorly when evaluating the deep-learning-based LPIPS and DISTS methods; conversely, the deep-learning-based LPIPS and DISTS methods also consider the traditional MAE and MS-SSIM methods to perform more poorly. This indicates that traditional methods and deep-learning-based methods are very different in nature. The human subjective perception believes that the E2E-DVC model optimized by the deep-learning-based methods can encode videos with higher visual quality. Moreover, VIF received the worst ranking in both human eye and other FR-IQA evaluations, which implies that modeling the visual channel of the human eye is a complex process and that using simple source channel coding theory in information theory is not sufficient to portray the perceptual characteristics of HVS; so, VIF is not suitable for optimizing and guiding image/video processing algorithms.

4.3. Subjective Results

It should be noted that PSNR has shown poor performance when it comes to estimating the perceptual quality of images and videos as perceived by humans. Thus, it is necessary to compare perceptual quality by performing subjective tests. We use the double-stimulus impairment scale (DSIS) method according to the methodology for the subjective assessment of the quality of television pictures of Recommendation ITU-R BT.500-13.

The monitor used in the subjective experiment was a Dell-U2720Q, with a viewing distance of 2.5 times the height of the screen. A total of 25 experimental observers between the ages of 22 and 35 participated in the subjective test, 11 of whom were female and 14 were male. All observers did not work in the video imaging industry, meaning that they had no expertise in the field of image and video processing, they had normal or normal corrected vision using glasses, and there were no color-weak or color-blind persons. A five-point scale was used for the subjective evaluation, with five indicating that the video distortion was imperceptible, four indicating that the video was perceptible but not annoying, three indicating that the video distortion was slightly annoying, two indicating that the video distortion was annoying, and one indicating that the video distortion was very annoying. Finally, the abnormal testers and abnormal scores were removed using the test screening method, and the observer’s mean opinion score (MOS) were derived as the quality scores of the video samples. Finally, the human eye subjective quality rankings of the five FR-IQA-optimized E2E-DVC model-encoded reconstructed videos are listed in Figure 4, and their rankings are LPIPS, DISTS, MS-SSIM, MAE, and VIF, in order from good to bad. The detailed MOS values are shown in Table 1.

The frame of the “BasketballDrive” sequence in HEVC class B constructed by the DVC models optimized by the five FR-QA metrics are shown in Figure 5. The MAE-guided reconstructed frame is somewhat blurred. The VIF-guided reconstructed frame has overenhanced global contrast and a lot of noise. Both LPIPS and DISTS succeed in reconstructed details of the original frame.

5. Conclusions

Visual quality assessment is often used as a KPI to evaluate the performance of ICPS. In this paper, we bring attention to alternative choices of perceptual loss function for E2E-DVC, which can be used to reduce the amount of data generated by sensors and other sources in ICPS. We selected five optimization-suitable FR-QA metrics as perceptual objectives for analysis of effects of different FR-QA metrics on E2E-DVC. The R-D function of the DVC model in terms of PSNR is similar to traditional video coding. The subjective human perceptual tests on the reconstructed videos showed the perceptual performance of different FR-QA optimizations. In addition, further study of the R-D performance of E2E-DVC and perceptual loss function design is needed in the future.