IV-SSIM—The Structural Similarity Metric for Immersive Video

Abstract

In this paper, we present a new objective quality metric designed for immersive video applications—IV-SSIM. The proposed IV-SSIM metric is an evolution of our previous work—IV-PSNR (immersive video peak signal-to-noise ratio)—which became a commonly used metric in research and ISO/IEC MPEG standardization activities on immersive video. IV-SSIM combines the advantages of IV-PSNR and metrics based on the structural similarity of images, being able to properly mimic the subjective quality perception of immersive video with its characteristic distortions induced by the reprojection of pixels between multiple views. The effectiveness of IV-SSIM was compared with 16 state-of-the-art quality metrics (including other metrics designed for immersive video). Tested metrics were evaluated in an immersive video coding scenario and against a commonly used image quality database—TID2013—showing their performance in both immersive and typical, non-immersive use cases. As presented, the proposed IV-SSIM metric clearly outperforms other metrics in immersive video applications, while also being highly competitive for 2D image quality assessment. The authors of this paper have provided a publicly accessible, efficient implementation of the proposed IV-SSIM metric, which is used by ISO/IEC MPEG video coding experts in the development of the forthcoming second edition of the MPEG immersive video (MIV) coding standard.

1. Introduction

The core concept behind immersive video is to grant a user the freedom to virtually navigate within a scene [1], which was captured by a multicamera system consisting of several perspective or omnidirectional cameras (blue cameras in Figure 1). The virtual navigation is enabled by a rendering of virtual views [2], i.e., the reprojection of pixels from input cameras to the virtual camera (orange camera in Figure 1) using spatial representation of the scene, usually given as depth maps and camera parameters (DIBR, depth-image-based rendering) [3].

In order to provide a high level of user immersion, rendered views should be subjectively perceived as having good quality. Unfortunately, subjective quality evaluation is a tedious and time-consuming process, which is highly inconvenient and expensive in practical systems. Therefore, a natural alternative to performing exhaustive subjective tests is objectively assessing the quality.

Significant challenges in immersive video are the rendering-induced distortions that are unique to this medium. Traditional video quality metrics may not adequately capture these distortions, as they are tailored to conventional two-dimensional video content. This necessitates the development of a specialized objective quality metric that can address the specific issues associated with immersive video, such as the slight shift of reprojected pixels and variations in color characteristics between input views. Development of such a specialized metric is crucial for ensuring that the virtual scenes appear natural and are free from visual discomfort, thereby maintaining the overall immersive experience.

2. Related Work

There are numerous objective quality metrics described in the literature. Many of these methods can properly mimic the behavior of the human visual system in typical image and video processing applications, e.g., several PSNR-based metrics: IV-PSNR (immersive video peak signal-to-noise ratio) [4], CS-PSNR (color-sensitivity-based combined PSNR) [5], PSNR-HA (modified PSNR, which considers the human visual system properties and the change in contrast and mean value), or PSNR-HMA (PSNR-HA, which additionally considers the visual masking of discrete cosine transform coefficients) [6]; metrics based on trained models such as VMAF (video multi-method assessment fusion) [7], VIF (visual information fidelity) [8], LPIPS (learned perceptual image patch similarity) [9], or SFF (sparse feature fidelity) [10]; metrics based on an idea of measuring the structural similarity of images: SSIM (structural similarity index) [11], MS-SSIM (multi-scale SSIM) [12]; the ones based on image feature similarity, e.g., FSIM (feature similarity index) [13], and metrics utilizing temporal information of video sequences [14,15,16].

All the abovementioned methods were designed for assessing the quality of typical, two-dimensional video; therefore, they are not optimal for video systems that allow the user to look around (360 video, 3DoF—3 degrees of freedom—systems [1]) or navigate within an acquired scene (immersive video systems). Visual quality assessment of 360 videos is widely studied [17], and many efficient quality metrics were proposed. For instance, in [18], the authors have proposed the OV-PSNR metric, integrating both spatial and temporal aspects of distortions and focusing on temporal eye fixation analysis. In [19] and [20], the visual quality assessment is based on neural networks and is focused on the influence of projection of the 360 video into the view watched by a viewer [19] or the influence of viewing directions [20]. Some research focuses on adapting existing methods (e.g., VMAF) to 360 video [21].

However, 360 video systems do not provide a full immersion of the user into the scene, as they do not provide motion parallax, which is inconsistent with human visual perception and leads to visual discomfort [22]. Immersive video introduces the possibility of changing the position of the viewer, which is enabled by view rendering [3]. Most state-of-the-art metrics cannot accurately assess immersive video quality with its typical, rendering-induced distortions [4], such as a slight shift of reprojected pixels and different color characteristics of input views (cf. Section 3).

Most of the existing methods designed for DIBR are computationally complex (e.g., require SIFT—scale-invariant feature transform—for detection of disoccluded areas [23] or perform an exhaustive search for similar regions in compared images [24]), designed for scenarios much simpler than immersive video, e.g., stereoscopic perspective [25,26,27,28,29] or omnidirectional video [30,31,32,33]. The approaches to stereoscopic and omnidirectional video quality assessment vary significantly across the different studies. For instance, in [25], the authors introduce a novel blind stereoscopic video quality metric, which incorporates measurement of the disparity entropy for assessing the binocular vision quality. In [26], the authors focus on the efficiency of the proposed metric in terms of processing time, developing a real-time metric designed for stereoscopic video streaming scenarios. Other approaches can require the analysis of 3D saliency maps, both sparse [27] or dense [29], and the metrics designed for stereoscopic perspective video are often based on machine learning techniques [28,29]. The saliency map analysis, as well as neural networks, can also be used for omnidirectional video quality assessment [31]. Other state-of-the-art approaches are focused on the analysis of the behavior used in the omnidirectional scene [32,33].

The state-of-the-art quality metrics designed for stereoscopic perspective or omnidirectional video are often very efficient and competitive in their application. However, as stated above, such applications are much simpler than the immersive video, where a user can freely navigate within the 3D scene but changing his or her viewing angle, as well as viewing position.

Such a characteristic of the immersive video system introduces artifacts and distortions which do not occur in simpler applications, and which are induced by view rendering. On the other hand, the quality metrics designed for immersive video often require the use of depth maps [34], being less versatile (e.g., they cannot be used for the assessment of virtual views presented to the final user of the immersive video system). A more practical approach is based on adapting more straightforward metrics to characteristics of immersive video [4,35,36].

The objective quality metric, which provides the highest correlation with MOS (mean opinion score) in immersive video applications, is IV-PSNR, proposed by the authors of this paper in [4]. The IV-PSNR metric was designed to properly handle and assess distortions typical to immersive video. As written in [4], IV-PSNR provides reliable results despite not utilizing any temporal information and being based on PSNR, which is simple and widely used but certainly not the best objective quality metric.

3. Challenges in Immersive Video Quality Assessment

In this paper, we propose to use structural similarity [11] as a foundation and combine it with the ideas constituting the advantages of IV-PSNR [4]. The usage of a much better base metric (structural similarity instead of MSE—mean squared error [4]) allowed us to increase the correlation with MOS (when compared to IV-PSNR) and resulted in the formulation of IV-SSIM—the structural similarity metric designed for immersive video.

Similarly to IV-PSNR, the proposed IV-SSIM metric does not analyze the temporal consistency of the video, measuring each frame of video independently. However, previous research on the quality of synthesized (rendered) video [37] demonstrated that temporal distortions in immersive video are primarily introduced by changing the viewport rather than by temporal inconsistencies. Therefore, for immersive video applications, it is much more important to properly assess the rendering-induced distortions than the temporal consistency of the content itself.

The most significant source of distortion in immersive video is the process of virtual view rendering (synthesis). The rendering introduces two distinct types of error: corresponding pixel shift error and global color offset error.

3.1. Corresponding Pixel Shift Error

The corresponding pixel shift error (CPSE) is caused by the quantization of depth maps (usually represented with 8 or 10 bits per value), imperfect camera parameters, and inaccuracies in the virtual view rendering process. All mentioned causes are unavoidable during the rendering and induce minor changes (shifts) in the position of rendered objects. This type of error is unnoticeable for viewers; however, it downgrades the “quality” measured by most typical objective quality metrics and makes the results irrelevant.

An example of a rendering-introduced object shift is shown in Figure 2. The figures show an original image (Figure 2A) and a synthesized image (Figure 2B) with the dancer being slightly shifted (by less than 2 pixels). This change in object position is irrelevant to the experience of a viewer; however, it significantly degrades the SSIM metric value (0.9954→0.9893). The proposed IV-SSIM metric is immune to CPSE artifacts (the IV-SSIM metric value is the same for both examples), exposing a better correlation with subjective quality perception. The example shown in Figure 2C presents the object shift exceeding the range expected from reprojection error. In this case, both metrics indicate quality degradation.

3.2. Global Color Offset Error

In order to obtain a high-quality rendered view, in practical immersive video systems, it has to be rendered using at least one real view. Real views are often captured by physical cameras, which sometimes differ in capture parameters (exposure setting, color balance), especially when cameras force automatic mode. These sources lead to slight, global differences between the real and the rendered view, called global color offset error (GCOE). Similarly to CPSE, this type of distortion is imperceptible for the viewer while significantly changing metric value.

An example of the abovementioned errors is presented in Figure 3. The image presented in Figure 3B is a result of virtual view synthesis from two real cameras. Those cameras significantly differ in color characteristics (when compared to each other and the reference camera—Figure 3A). It leads to distractive artifacts clearly visible in the background of the exemplary scene. The image in Figure 3C contains a result of virtual view synthesis with color correction applied during the rendering process. The addition of the color correction eliminates background color inconsistency. However, SSIM metric values (0.9216 for B and 0.9107 for C) do not match the subjective perception of image C, which is less distorted than image B. When the IV-SSIM metric is considered, the image quality of example C is evaluated as higher than example B. Such a metric behavior is consistent with subjective perception. Please note that the definition of IV-SSIM (cf. Section 4) implies that its value is usually significantly higher than for SSIM, thus IV-SSIM higher than 0.99 does not mean that the image has near-to-perfect quality (as it would be the case for SSIM). However, for objective quality metrics, the crucial aspect is the correlation with subjective quality perception.

4. IV-SSIM

The proposed IV-SSIM metric is based on the structural similarity index (SSIM, [11]), which is one of the most widely used metrics in research on image and video processing. The SSIM metric measures the similarity of two images by using three properties: intensity L, contrast C, and structure S [11]. These properties are defined by analyzing local means (${\mu }_c^I$ and ${\mu }_c^J$), standard deviations (${\sigma }_c^I$ and ${\sigma }_c^J$), and covariance (${\sigma }_c^{I,J}$) within each color component $c$ of images $I$ and $J$. The IV-SSIM metric preserves a general idea of the SSIM metric; however, the calculation of several properties and statistics (intensity, local means, standard deviations, and covariance) was significantly changed. These changes adapt the IV-SSIM metric to distortion types occurring in immersive video (described in Section 3).

4.1. Local Image Statistics

In the proposed IV-SSIM metric, the local mean and standard deviation of image $I$ are calculated in the same way as for the classic SSIM

$${\mu }_c^I\left(x,y\right)={\displaystyle \sum }_{i=x-k}^{x+k}{\displaystyle \sum }_{j=y-k}^{y+k}\left[\omega \left(i-x,j-y\right)·I_c\left(i,j\right)\right],$$

(1)

$${\sigma }_c^I\left(x,y\right)=\sqrt{{\displaystyle \sum }_{i=x-k}^{x+k}{\displaystyle \sum }_{j=y-k}^{y+k}\left[\omega \left(i-x,j-y\right)·{\left(I_c\left(i,j\right)\right)}^2\right]-{\left({\mu }_c^I\left(x,y\right)\right)}^2},$$

(2)

where ${\mu }_c^I$ is the local mean and ${\sigma }_c^I$ is the standard deviation of the image $I$, and $\omega$ is the weighting mask of size $\left(2k+1\right)\times \left(2k+1\right)$. Both ${\mu }_c^I\left(x,y\right)$ and ${\sigma }_c^I\left(x,y\right)$ correspond to pixel coordinates $\left(x,y\right)$ of the component $c$ of the image $I$. $I_c\left(i,j\right)$ is the single picture element of component $c$ of the image $I$ at coordinates $\left(i,j\right)$. The typical weighting mask $\omega$ has the size $11\times 11$ ($k=5$) and is calculated using the circular-symmetric Gaussian weighting function [11].

The IV-SSIM metric includes modifications dedicated to the characteristics of immersive video, implying changes in the calculation of the local statistics of image $J$ and the covariance between images $I$ and $J$:

$${\mu }_c^J\left(x,y\right)={\displaystyle \sum }_{i=x-k}^{x+k}{\displaystyle \sum }_{j=y-k}^{y+k}\left[\omega \left(i-x,j-y\right)·J_c\left(i^{\prime },j^{\prime }\right)\right],$$

(3)

$${\sigma }_c^J\left(x,y\right)=\sqrt{{\displaystyle \sum }_{i=x-k}^{x+k}{\displaystyle \sum }_{j=y-k}^{y+k}\left[\omega \left(i-x,j-y\right)·{\left(J_c\left(i^{\prime },j^{\prime }\right)\right)}^2\right]-{\left({\mu }_c^J\left(x,y\right)\right)}^2},$$

(4)

$${\sigma }_c^{I\to J}\left(x,y\right)={\displaystyle \sum }_{i=x-k}^{x+k}{\displaystyle \sum }_{j=y-k}^{y+k}\left[\omega \left(i-x,j-y\right)·I_c\left(i,j\right)·J_c\left(i^{\prime },j^{\prime }\right)\right]-{\mu }_c^I\left(x,y\right)·{\mu }_c^J\left(x,y\right),$$

(5)

where ${\mu }_c^J$ is the local mean and ${\sigma }_c^J$ is the standard deviation of the image $J,$ and ${\sigma }_c^{I\to J}$ is the covariance between images $I$ and $J$.

The $\left(i^{\prime },j^{\prime }\right)$ is a position of the pixel within image $J$, defined as:

$$i^{\prime }=i+s_x\left(i,j\right), j^{\prime }=j+s_y\left(i,j\right),$$

(6)

where $s_x(·),s_y(·)$ are the corresponding pixel shift (in the horizontal and vertical direction, respectively)—cf. Section 3.1. The corresponding pixel shift is calculated independently for each pixel $\left(x,y\right)$ of image $I$.

The shift $s_x\left(i,j\right),s_y\left(i,j\right)$ is calculated by analyzing the $B\times B$ neighborhood of the pixel $\left(i,j\right)$ within image $J$ in search of the pixel, which is most similar to the pixel $\left(i,j\right)$ in image $I$:

$$s_x\left(x,y\right),s_y\left(x,y\right)\in \left[-B,B\right]\cap \mathbb{Z}\ni \left|I_C\left(x,y\right)-J_C\left(x+s_x\left(x,y\right),y+s_y\left(x,y\right)\right)\right|=\underset{\begin{array}{c}w\in \left[-B,B\right]\\ h\in \left[-B, B\right]\end{array}}{min}\left|I_C\left(x,y\right)-J_C\left(x+w,y+h\right)\right|,$$

(7)

where $B$ is the maximum considered shift of the corresponding pixel, by default set to 2 (resulting in a $5\times 5$ search range). The $5\times 5$ search range was chosen to preserve consistency with the IV-PSNR metric [4].

It should be emphasized that the modifications of the covariance calculation made it asymmetrical, thus:

$${\sigma }_c^{I\to J}\left(x,y\right)\ne {\sigma }_c^{J\to I}\left(x,y\right).$$

(8)

The problem of IV-SSIM asymmetricity, together with the proposed solution, is presented in Section 4.4.

4.2. Image Properties Similarity

In the second step of IV-SSIM calculation, all the abovementioned local statistics are used for the estimation of the similarity of three properties of images $I$ and $J$: intensity $L$, contrast $C$, and structure $S$.

$$L_c^{I\to J}\left(x,y\right)={\displaystyle \frac{2·{\mu }_c^I\left(x,y\right)·\left[{\mu }_c^J\left(x,y\right)+s_c^{I\to J}\right]+C_1}{{\mu }_c^I{\left(x,y\right)}^2+{\left[{\mu }_c^J\left(x,y\right)+s_c^{I\to J}\right]}^2+C_1}},$$

(9)

$$C_c^{I\to J}\left(x,y\right)={\displaystyle \frac{2·{\sigma }_c^I\left(x,y\right)·{\sigma }_c^J\left(x,y\right)+C_2}{{\sigma }_c^I{\left(x,y\right)}^2+{\sigma }_c^J{\left(x,y\right)}^2+C_2}},$$

(10)

$$S_c^{I\to J}\left(x,y\right)={\displaystyle \frac{{\sigma }_c^{I\to J}\left(x,y\right)+C_3}{{\sigma }_c^I\left(x,y\right)·{\sigma }_c^J\left(x,y\right)+C_3}},$$

(11)

where $s_c^{I\to J}$ is the global offset (cf. Section 3.2) of component $c$ between images $I$ and $J$, defined as:

$$s_c^{I\to J}={\displaystyle \frac{1}{W_c·H_c}}{\displaystyle \sum }_{y=0}^{H_c-1}{\displaystyle \sum }_{x=0}^{W_c-1}\left(I_c\left(x,y\right)-J_c\left(x,y\right)\right),$$

(12)

and $C_1$, $C_2$, and $C_3$ are constants providing numerical stability:

$$C_1={\left(K_1·(2^b-1)\right)}^2,C_2={\left(K_2·(2^b-1)\right)}^2,C_3={\displaystyle \frac{C_2}{2}},$$

(13)

where $b$ is the bit-depth of the image, and:

$$K_1=0.01,K_2=0.03.$$

(14)

$W_c$ and $H_c$ denote the width and height of the image (i.e., width and height of its $c$-th color plane). The comprehensive description regarding the calculation and usage of $C_1$, $C_2$, $C_3$, $K_1$, and $K_2$ coefficients can be found in [40].

In the third step, all property similarities are combined, resulting in local quality scores for component $c$:

$$Q_c^{I\to J}\left(x,y\right)={\left[L_c^{I\to J}\left(x,y\right)\right]}^{\alpha }·{\left[C_c^{I\to J}\left(x,y\right)\right]}^{\beta }·{\left[S_c^{I\to J}\left(x,y\right)\right]}^{\gamma },$$

(15)

where $\alpha$, $\beta$, and $\gamma$ are parameters describing the relative importance of $L_c^{I\to J}$, $C_c^{I\to J}$, and $S_c^{I\to J},$ respectively. As proposed in [11], the values $\alpha =\beta =\gamma =1$ are used.

4.3. Calculation of the IV-SSIM Value

Local quality scores $Q_c^{I\to J}\left(x,y\right)$ could be used for comparing the quality of different parts of the image, being potentially useful for visual localization of distortions. However, these scores have to be combined to assess the quality of the entire image. The IV-SSIM value for each color component $c$ is calculated by averaging these scores over the entire image (entire color plane of component $c$):

$$IV-SSI{M}_c^{I\to J}={\displaystyle \frac{1}{W_c·H_c}}{\displaystyle \sum }_{y=0}^{H_c-1}{\displaystyle \sum }_{x=0}^{W_c-1}{Q}_c^{I\to J}\left(x,y\right).$$

(16)

In order to produce a single IV-SSIM value, values calculated for single color components are combined using the weighted average:

$$IV-SSI{M}_{YUV}^{I\to J}={\displaystyle \frac{IV-SSI{M}_Y^{I\to J}·w_Y+IV-SSI{M}_U^{I\to J}·w_U+IV-SSI{M}_V^{I\to J}·w_v}{w_Y+w_U+w_V}},$$

(17)

where $w_Y$, $w_U$, and $w_V$ denote weights for luma and both chroma components. By default, $w_Y=4$, $w_U=1$, and $w_V=1$. Such values are in line with default weights for IV-PSNR [4] and ISO/IEC MPEG VC common test conditions for MPEG immersive video (MIV CTC) [41]; they provide a high correlation with subjective quality assessment and refer to the proportion of luma and chroma samples in the most commonly used chroma subsampling format—4:2:0.

4.4. Symmetricity of IV-SSIM

A robust and practical objective quality metric should be symmetrical; thus, it should produce the same output independently of the order of input images. In order to provide the symmetricity, the final value of the IV-SSIM between images $I$ and $J$ is calculated as:

$$IV-SSIM\left(I,J\right)=\min \left(IV-SSI{M}_{YUV}^{I\to J}, IV-SSI{M}_{YUV}^{J\to I}\right).$$

(18)

4.5. IV-SSIM for Video Sequences

The calculation of the IV-SSIM value (18) is performed independently for each frame of a video sequence. In order to assess the quality of the video sequence, IV-SSIM values calculated for all the frames are averaged.

4.6. Publicly Available IV-SSIM Implementation

The reference implementation of the IV-SSIM algorithm is available under the 3-clause BSD license within the Quality Metrics for Immersive Video (QMIV) software v1.0.

The QMIV software was implemented by the authors of this paper and is publicly available in the public git repository of the MPEG Video Coding experts group: https://gitlab.com/mpeg-i-visual/qmiv (accessed on 11 August 2024).

5. Overview of Experiments

The proposed IV-SSIM metric was compared with 16 state-of-the-art quality metrics with publicly available implementations: PSNR_Y (PSNR of luma component), PSNR_YUV (weighted average of PSNR for three components with luma weight six times higher than the weights for both chroma components, as described in [42]), IV-PSNR [4], CS-PSNR [5], PSNR-HA and PSNR-HMA [6], VMAF [7], a pixel-based VIF [8], LPIPS [9], SFF [10], SSIM_Y [11] and its multiscale version, MS-SSIM [12], FSIM [13], MW-PSNR (morphological wavelet PSNR) [35], and MP-PSNR (morphological pyramid PSNR) [36]. Moreover, we have added an SSIM_YUV metric, calculated for all three color components and weighted in the same way as PSNR_YUV.

The effectiveness of the proposed IV-SSIM was evaluated in two experiments. In the first one (Section 6), the influence of different techniques of immersive video coding was assessed. In the second experiment (Section 7), IV-SSIM was compared against the commonly used image quality assessment database—TID2013 [43]—in order to present its performance in non-immersive video applications.

In each experiment, the metrics were compared using two commonly used correlation coefficients: PLCC (Pearson’s linear correlation coefficient) and SROCC (Spearman’s rank-order correlation coefficient) [44].

The calculation of PSNR_Y, PSNR_YUV, and IV-PSNR metrics was performed using the IV-PSNR software v4.0 [45] developed by the authors of the IV-PSNR metric [4]. The VMAF metric was calculated using the implementation provided by its authors [46]. Among all tested metrics, only these two software packages include a parallel variant of metric calculation, therefore, in Section 8 we provided results for both single- and multi-threaded computational time evaluation. All metrics were calculated on the same computer equipped with an AMD Ryzen 9 3900XT processor (with 12 identical cores) in connection with 64 GiB of DDR4/3200MT memory. The processor was operating in the default dynamic frequency setting; therefore, the core frequency for single-threaded loads was significantly higher than for multi-threaded variants.

6. Effectiveness in Immersive Video Coding

In the first experiment, IV-SSIM was evaluated using the results of the “MPEG Call for Proposals on 3DoF+ Visual” [47], which contain videos compressed using seven different immersive video coding techniques, five test sequences (both natural and computer-generated), and four rate points (6.5, 10, 15, and 25 Mbit/s). The subjective quality evaluation was performed using the Double Stimulus Impairment Scale test method [48] on a group of 18 naïve viewers [49]. The participants assessed the quality of the “pose trace” videos, simulating the virtual navigation of the viewer within a scene [50]. The objective quality was evaluated by assessing the quality of synthesized input views (virtual views rendered in the position of real cameras that captured the scene). A detailed methodology for subjective and objective evaluation of the immersive video coding dataset can be found in the publicly available ISO/IEC MPEG document [47].

The dataset was used to evaluate a correlation between MOS (mean opinion score) and 17 objective quality metrics. The results are presented in Figure 4. As shown, IV-SSIM achieved the highest SROCC and PLCC among all 17 tested metrics. It outperformed the second-best metric—IV-PSNR—by 0.065 and 0.068 for SROCC and PLCC, respectively. When compared to other metrics, the difference is even higher, showing the superiority of IV-SSIM in the assessment of immersive video.

The difference between correlation coefficients measured for IV-SSIM and SSIM_YUV (three-component SSIM, not adapted to immersive video) is 0.205 for SROCC and 0.218 for PLCC (cf.

Table 1. Comparison between the proposed IV-SSIM metric, typical SSIM [11], IV-PSNR [4], and PSNR. Three-component versions of PSNR and SSIM (i.e., PSNR_YUV and SSIM_YUV) were used.

Scenario	Correlation Coefficient	PSNR	IV-PSNR [4]	SSIM [11]	IV-SSIM
immersive video coding (Section 6)	SROCC	0.527	0.728	0.588	0.793
	PLCC	0.520	0.731	0.581	0.799
general applications—TID2013 (Section 7)	SROCC	0.818	0.825	0.839	0.850
	PLCC	0.816	0.820	0.823	0.825

). A similar difference was measured for IV-PSNR and PSNR_YUV (0.201 and 0.211). Such a result shows that the handling of typical immersive video distortions increases the correlation between objective and subjective quality in a consistent way, independently of the objective metric, which was used as a basis (PSNR or SSIM).

Despite the fact that the proposed IV-SSIM metric does not assess the temporal consistency of video, it significantly outperforms other metrics, including the ones that measure the temporal similarity (e.g., VMAF). The reason for such a behavior is, as mentioned in Section 3, an atypical characteristic of immersive video, where temporal distortions are introduced not by the timeline, but by changing the viewpoint.

7. Effectiveness in General Applications (TID2013)

In the second experiment, all tested metrics were compared against the TID2013 database [43], which includes 24 miscellaneous types of distortions, such as several types of noise, blurring, compression, change in brightness and contrast, as well as compression and transmission errors. It is important to note that the TID2013 database does not contain images with reprojection-related types of distortion (described in Section 3).

The correlation (calculated as SROCC) between subjective and objective quality was estimated independently for each type of distortion. Figure 5 presents SROCC and PLCC values averaged over all 24 types of distortions.

As presented in Figure 5, the proposed IV-SSIM metric outperforms most tested metrics also for non-immersive video applications. For SROCC, IV-SSIM is the second-best metric, and its performance is nearly the same as that of VMAF. VMAF is a meta-metric containing several sub-metrics combined with weights trained for natural images; therefore, its high performance for natural images from the TID2013 database is not surprising. Considering PLCC, it is only worse than VMAF, PSNR-HA, and PSNR-HMA (however, the difference is negligible). Such a discrepancy shows that the correlation between IV-SSIM and MOS is less linear than for these three metrics. However, correlation linearity is less crucial than the ability to assess which video or image is better (what is measured by the rank-order SROCC metric).

The discrepancy between the two used correlation coefficients exists (and it is consistent for all SSIM metrics: IV-SSIM, SSIM_YUV, and SSIM_Y), but its impact can be considered negligible, as both coefficients indicate that the proposed IV-SSIM metric performs well even though it was designed for handling a different type of data. Therefore, it could be stated that IV-SSIM properly mimics the subjective perception of quality not only for immersive video but also for any video-related application.

8. Computational Time Estimation

A practical objective quality metric should not only correlate with subjective quality perception but should also assess the quality as fast as possible. Unfortunately, the SSIM metric, as defined in [11], introduces high computational and memory bandwidth complexity. Assuming the default size of the Gaussian weighting mask (ω) equal to 11 × 11, the calculation of SSIM requires 242 memory accesses and ~1000 multiplications for each component of a pixel. To mitigate the mentioned complexity, the proposed IV-SSIM metric was designed to be parallelizable by spreading the metric calculation across multiple CPU cores.

The computational performance was evaluated using our experimental, parallel implementation of SSIM_Y, SSIM_YUV, and IV-SSIM. This experimental implementation is not fully optimized and leaves possibilities for prospective performance improvements. We expect that fully optimized and vectorized implementation would be noticeably faster, but production-quality software implementation is out of the scope of this paper.

Figure 6 compares IV-SSIM with 16 evaluated state-of-the-art metrics in terms of the computational time required for assessing the quality of a single frame of a 4K × 4K video. These results are presented together with SROCC values estimated in the immersive video experiment (cf. Section 6).

As presented in Figure 6, quality assessment of a single frame of a high-resolution sequence requires only 1.3 s (in a multi-thread implementation), which is faster than most tested implementations of state-of-the-art metrics. Simultaneously, IV-SSIM provides the highest correlation with MOS (mean opinion score).

9. Conclusions

The proposed IV-SSIM metric is a successor and potential replacement of our previous work—IV-PSNR [4]—which is a PSNR-based metric adapted for the characteristics of immersive video. IV-SSIM is a full-reference objective quality metric based on a calculation of structural similarity (SSIM) between two images.

IV-SSIM combines the advantages of SSIM and IV-PSNR, properly mimicking a human visual system in assessing the quality of immersive video, with its typical distortions caused by reprojection of pixels between multiple views: slight shift of reprojected pixels and color artifacts originated from various color characteristics of different views.

The IV-SSIM metric was compared to 16 state-of-the-art metrics in two experiments. In the first one, its efficiency in an immersive video coding scenario was evaluated. In the second experiment, IV-SSIM was evaluated against the TID2013 database [43], showing its performance in a typical non-immersive video application. The experiments have shown that IV-SSIM clearly outperforms state-of-the-art metrics in assessing the quality of immersive video, also being highly competitive for the assessment of various kinds of distortions introduced to typical 2D images.

Moreover, the calculation of IV-SSIM is straightforward and parallelizable, making the proposed metric practical and convenient to use in real, practical immersive video systems.

Its efficiency was appreciated by the experts of the MPEG Video Coding group, resulting in including IV-SSIM in the common test conditions for MPEG immersive video—MIV CTC [51] and implicit neural visual representation—INVR CTC [52]. The implementation of the IV-SSIM metric was provided by the authors of this paper within the Quality Metrics for Immersive Video (QMIV) software v1.0 [53] and is publicly available on the MPEG Video Coding public git repository: https://gitlab.com/mpeg-i-visual/qmiv (accessed on 11 August 2024).

Currently, IV-SSIM is based on the default definition of the structural similarity index (SSIM) presented in [11]. However, such a definition (e.g., using the Gaussian weighting mask (ω) equal to 11 × 11) is not efficient in terms of computational complexity. On the other hand, several more efficient implementations of SSIM are known and commonly used, e.g., the rectangular mask used in the ffmpeg framework [54]. Therefore, in the future, different implementations of the structural similarity in the IV-SSIM metric should be analyzed.

Moreover, similarly to SSIM, the proposed IV-SSIM lacks temporal video stability analysis. In the vast majority of scenarios, rendering-induced distortions in immersive video dominate over temporal distortions. However, in some cases (e.g., significant scene geometry changes over a short period of time, resulting in dramatically unstable depth maps), the metric may be insensitive to video flickering, which decreases the subjective quality of video. Therefore, the second direction for future work is to increase the efficiency of IV-SSIM by implementing temporal consistency analysis.