IV-SSIM—The Structural Similarity Metric for Immersive Video
Abstract
:1. Introduction
2. Related Work
3. Challenges in Immersive Video Quality Assessment
3.1. Corresponding Pixel Shift Error
3.2. Global Color Offset Error
4. IV-SSIM
4.1. Local Image Statistics
4.2. Image Properties Similarity
4.3. Calculation of the IV-SSIM Value
4.4. Symmetricity of IV-SSIM
4.5. IV-SSIM for Video Sequences
4.6. Publicly Available IV-SSIM Implementation
5. Overview of Experiments
6. Effectiveness in Immersive Video Coding
7. Effectiveness in General Applications (TID2013)
8. Computational Time Estimation
9. Conclusions
Author Contributions
Funding
Institutional Review Board Statement
Informed Consent Statement
Data Availability Statement
Conflicts of Interest
References
- Wien, M.; Boyce, J.; Stockhammer, T.; Peng, W.H. Standardization status of immersive video coding. IEEE J. Emerg. Sel. Top. Circuits Syst. 2019, 9, 5–17. [Google Scholar] [CrossRef]
- Dziembowski, A.; Mieloch, D.; Stankiewicz, O.; Domański, M.; Lee, G.; Seo, J. Virtual view synthesis for 3DoF+ video. In Proceedings of the Picture Coding Symposium (PCS), Ningbo, China, 12–15 November 2019. [Google Scholar] [CrossRef]
- Müller, K.; Merkle, P.; Wiegand, T. 3-D video representation using depth maps. Proc. IEEE 2011, 99, 643–656. [Google Scholar] [CrossRef]
- Dziembowski, A.; Mieloch, D.; Stankowski, J.; Grzelka, A. IV-PSNR—The objective quality metric for immersive video applications. IEEE Trans. Circuits Syst. Video Technol. 2022, 32, 7575–7591. [Google Scholar] [CrossRef]
- Shang, X.; Liang, J.; Wang, G.; Zhao, H.; Wu, C.; Lin, C. Color-sensitivity-based combined PSNR for objective video quality assessment. IEEE Trans. Circuits Syst. Video Technol. 2019, 29, 1239–1250. [Google Scholar] [CrossRef]
- Ponomarenko, N.; Ieremeiev, O.; Lukin, V.; Egiazarian, K.; Carli, M. Modified image visual quality metrics for contrast change and mean shift accounting. In Proceedings of the 11th International Conference the Experience of Designing and Application of CAD Systems in Microelectronics (CADSM), Polyana, Ukraine, 23–25 February 2011. [Google Scholar]
- Li, Z.; Aaron, A.; Katsavounidis, I.; Moorthy, A.; Manohara, M. Toward a Practical Perceptual Video Quality Metric. In Netflix Technology Blog; Technical Report; 2016; Available online: https://netflixtechblog.com/toward-a-practical-perceptual-video-quality-metric-653f208b9652 (accessed on 11 August 2024).
- Sheikh, H.R.; Bovik, A.C. Image information and visual quality. IEEE Trans. Image Process. 2006, 15, 430–444. [Google Scholar] [CrossRef]
- Zhang, R.; Isola, P.; Efros, A.A.; Shechtman, E.; Wang, O. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Salt Lake City, UT, USA, 18–23 June 2018; pp. 586–595. [Google Scholar] [CrossRef]
- Chang, H.W.; Yang, H.; Gan, Y.; Wang, M.H. Sparse feature fidelity for perceptual image quality assessment. IEEE Trans. Image Process. 2013, 22, 4007–4018. [Google Scholar] [CrossRef] [PubMed]
- Wang, Z.; Bovik, A.C.; Sheikh, H.R.; Simoncelli, E.P. Image quality assessment: From error measurement to structural similarity. IEEE Trans. Image Process. 2004, 13, 600–612. [Google Scholar] [CrossRef] [PubMed]
- Wang, Z.; Simoncelli, E.P.; Bovik, A.C. Multiscale structural similarity for image quality assessment. In Proceedings of the 37th Asilomar Conference on Signals, Systems & Computers, Pacific Grove, CA, USA, 9–12 November 2003. [Google Scholar] [CrossRef]
- Zhang, L.; Zhang, L.; Mou, X.; Zhang, D. FSIM: A feature similarity index for image quality assessment. IEEE Trans. Image Process. 2011, 20, 2378–2386. [Google Scholar] [CrossRef]
- Zeng, K.; Wang, Z. 3D-SSIM for video quality assessment. In Proceedings of the 19th IEEE International Conference on Image Processing, Orlando, FL, USA, 30 September–3 October 2012. [Google Scholar] [CrossRef]
- Mantiuk, R.; Denes, G.; Chapiro, A.; Kaplanyan, A.; Rufo, G.; Bachy, R.; Lian, T.; Patney, A. FovVideoVDP: A visible difference predictor for wide field-of-view video. ACM Trans. Graph. 2021, 40, 49. [Google Scholar] [CrossRef]
- Wang, Y.; Jiang, T.; Ma, S.; Gao, W. Spatio-temporal SSIM index for video quality assessment. In Proceedings of the 2012 Visual Communications and Image Processing, San Diego, CA, USA, 27–30 November 2012. [Google Scholar] [CrossRef]
- Zakharchenko, V.; Choi, K.P.; Park, J.H. Quality metric for spherical panoramic video. Proc. SPIE 2016, 9970, 57–65. [Google Scholar] [CrossRef]
- Gao, P.; Zhang, P.; Smolic, A. Quality assessment for omnidirectional video: A spatio-temporal distortion modeling approach. IEEE Trans. Multimed. 2020, 24, 1–16. [Google Scholar] [CrossRef]
- Guo, J.; Huang, L.; Chien, W.C. Multi-viewport based 3D convolutional neural network for 360-degree video quality assessment. Multimed. Tools Appl. 2022, 81, 16813–16831. [Google Scholar] [CrossRef]
- Guo, J.; Luo, Y. No-reference omnidirectional video quality assessment based on generative adversarial networks. Multimed. Tools Appl. 2021, 80, 27531–27552. [Google Scholar] [CrossRef]
- Orduna, M.; Díaz, C.; Muñoz, L.; Pérez, P.; Benito, I.; García, N. Video Multimethod Assessment Fusion (VMAF) on 360VR contents. IEEE Trans. Consum. Electron. 2020, 66, 22–31. [Google Scholar] [CrossRef]
- Vadakital, V.K.M.; Dziembowski, A.; Lafruit, G.; Thudor, F.; Lee, G.; Alface, P.R. The MPEG immersive video standard—current status and future outlook. IEEE Multimed. 2022, 29, 101–111. [Google Scholar] [CrossRef]
- Li, L.; Zhou, Y.; Gu, K.; Lin, W.; Wang, S. Quality assessment of DIBR-synthesized images by measuring local geometric distortions and global sharpness. IEEE Trans. Multimed. 2018, 20, 914–926. [Google Scholar] [CrossRef]
- Battisti, F.; Bosc, E.; Carli, M.; Le Callet, P.; Perugia, S. Objective image quality assessment of 3D synthesized views. Signal Process. Image Commun. 2015, 30, 78–88. [Google Scholar] [CrossRef]
- Chen, Z.; Zhou, W.; Li, W. Blind stereoscopic video quality assessment: From depth perception to overall experience. IEEE Trans. Image Process. 2018, 27, 721–734. [Google Scholar] [CrossRef] [PubMed]
- Han, Y.; Yuan, Z.; Muntean, G.M. An Innovative No-Reference Metric for Real-Time 3D Stereoscopic Video Quality Assessment. IEEE Trans. Broadcast. 2016, 62, 654–663. [Google Scholar] [CrossRef]
- Yang, J.; Ji, C.; Jiang, B.; Lu, W.; Meng, Q. No Reference Quality Assessment of Stereo Video Based on Saliency and Sparsity. IEEE Trans. Broadcast. 2018, 64, 341–353. [Google Scholar] [CrossRef]
- Imani, H.; Islam, M.B.; Junayed, M.S.; Aydin, T.; Arica, N. Stereoscopic video quality measurement with fine-tuning 3D ResNets. Multimed. Tools Appl. 2022, 81, 42849–42869. [Google Scholar] [CrossRef]
- Li, C.; Yun, L.; Xu, S. Blind stereoscopic image quality assessment using 3D saliency selected binocular perception and 3D convolutional neural network. Multimed. Tools Appl. 2022, 81, 18437–18455. [Google Scholar] [CrossRef]
- Hu, Z.; Liu, L.; Sang, Q. Omnidirectional Video Quality Assessment with Causal Intervention. IEEE Trans. Broadcast. 2024, 70, 238–250. [Google Scholar] [CrossRef]
- Zhou, M.; Chen, L.; Wei, X.; Liao, X.; Mao, Q.; Wang, H.; Pu, H.; Luo, J.; Xiang, T.; Fang, B. Perception-Oriented U-Shaped Transformer Network for 360-Degree No-Reference Image Quality Assessment. IEEE Trans. Broadcast. 2023, 69, 396–405. [Google Scholar] [CrossRef]
- Jiang, H.; Jiang, G.; Yu, M.; Luo, T.; Xu, H. Multi-angle projection based blind omnidirectional image quality assessment. IEEE Trans. Circuits Syst. Video Technol. 2021, 32, 4211–4223. [Google Scholar] [CrossRef]
- Sui, X.; Ma, K.; Yao, Y.; Fang, Y. Perceptual quality assessment of omnidirectional images as moving camera videos. IEEE Trans. Vis. Comput. Graph. 2022, 28, 3022–3034. [Google Scholar] [CrossRef] [PubMed]
- Li, L.; Huang, Y.; Wu, J.; Gu, K.; Fang, Y. Predicting the quality of view synthesis with color-depth image fusion. IEEE Trans. Circuits Syst. Video Technol. 2021, 31, 2509–2521. [Google Scholar] [CrossRef]
- Sandić-Stanković, D.; Kukolj, D.; Le Callet, P. DIBR-synthesized image quality assessment based on morphological multi-scale approach. EURASIP J. Image Video Process. 2016, 2017, 4. [Google Scholar] [CrossRef]
- Sandić-Stanković, D.; Kukolj, D.; Le Callet, P. DIBR synthesized image quality assessment based on morphological pyramids. In Proceedings of the 2015 3DTV-Conference: The True Vision—Capture, Transmission and Display of 3D Video (3DTV-CON), Lisbon, Portugal, 26–29 May 2015. [Google Scholar] [CrossRef]
- Tian, S.; Zhang, L.; Zou, W.; Su, T.; Morin, L.; Deforges, O. Quality assessment of DIBR-synthesized views: An overview. Neurocomputing 2021, 423, 158–178. [Google Scholar] [CrossRef]
- Klóska, D.; Mieloch, D.; Dziembowski, A.; Szydełko, B.; Stankowski, J.; Lee, G. A new natural content proposal: Choreo. In Document ISO/IEC JTC1/SC29/WG04 MPEG VC, M66990; International Organization for Standardization: Rennes, France, 2024. [Google Scholar]
- Mieloch, D.; Dziembowski, A.; Domański, M. [MPEG-I Visual] Natural outdoor test sequences. In Document ISO/IEC JTC1/SC29/WG11 MPEG M51598; International Organization for Standardization: Brussels, Belgium, 2020. [Google Scholar]
- Venkataramanan, A.K.; Wu, C.; Bovik, A.; Katsavounidis, I.; Shahid, Z. A hitchhiker’s guide to structural similarity. IEEE Access 2021, 9, 28872–28896. [Google Scholar] [CrossRef]
- ISO/IEC. Common test conditions for MPEG immersive video. In Document ISO/IEC JTC1/SC29/WG04 MPEG VC N0406; International Organization for Standardization: Hannover, Germany, 2023. [Google Scholar]
- Huang, Y.; Qi, H.; Li, B.; Xu, J. Adaptive weighted distortion optimization for video coding in RGB color space. In Proceedings of the IEEE International Conference on Image Processing (ICIP), Paris, France, 27–30 October 2014. [Google Scholar] [CrossRef]
- Ponomarenko, N.; Jin, L.; Ieremeiev, O.; Lukin, V.; Egiazarian, K.; Astola, J.; Vozel, B.; Chehdi, K.; Carli, M.; Battisti, F.; et al. Image database TID2013: Peculiarities, results and perspectives. Signal Process. Image Commun. 2015, 30, 57–77. [Google Scholar] [CrossRef]
- Chikkerur, S.; Sundaram, V.; Reisslein, M.; Karam, L. Objective Video Quality Assessment Methods: A Classification, Review, and Performance Comparison. IEEE Trans. Broadcast. 2011, 57, 165–182. [Google Scholar] [CrossRef]
- Stankowski, J.; Dziembowski, A. IV-PSNR: Software for immersive video objective quality evaluation. SoftwareX 2023, 24, 101592. [Google Scholar] [CrossRef]
- Git Repository. Available online: https://github.com/Netflix/vmaf (accessed on 21 February 2024).
- ISO/IEC. Call for proposals on 3DoF+ visual. In Document ISO/IEC JTC1/SC29/WG11 MPEG N18145; International Organization for Standardization: Marrakech, Morocco, 2019. [Google Scholar]
- ITU-T. Subjective video quality assessment methods for multimedia applications. In Recommendation ITU-T P.910; International Telecommunication Union: Geneva, Switzerland, 2008. [Google Scholar]
- ISO/IEC. Evaluation results of the call for proposals on 3DoF+ visual. In Document ISO/IEC JTC1/SC29/WG11 MPEG N18353; International Organization for Standardization: Geneva, Switzerland, 2019. [Google Scholar]
- Boyce, J.; Doré, R.; Dziembowski, A.; Fleureau, J.; Jung, J.; Kroon, B.; Salahieh, B.; Vadakital, V.K.M.; Yu, L. MPEG immersive video coding standard. Proc. IEEE 2021, 109, 1521–1536. [Google Scholar] [CrossRef]
- ISO/IEC. Common test conditions for MPEG immersive video. In Document ISO/IEC JTC1/SC29/WG4 MPEG VC N0539; International Organization for Standardization: Sapporo, Japan, 2024. [Google Scholar]
- ISO/IEC. Common test conditions on radiance field representation and compression. In Document ISO/IEC JTC1/SC29/WG4 MPEG VC N0561; International Organization for Standardization: Sapporo, Japan, 2024. [Google Scholar]
- ISO/IEC. Software manual of QMIV. In Document ISO/IEC JTC1/SC29/WG4 MPEG VC N0535; International Organization for Standardization: Sapporo, Japan, 2024. [Google Scholar]
- Ffmpeg Framework. Available online: https://ffmpeg.org (accessed on 7 August 2024).
Scenario | Correlation Coefficient | PSNR | IV-PSNR [4] | SSIM [11] | IV-SSIM |
---|---|---|---|---|---|
immersive video coding (Section 6) | SROCC | 0.527 | 0.728 | 0.588 | 0.793 |
PLCC | 0.520 | 0.731 | 0.581 | 0.799 | |
general applications—TID2013 (Section 7) | SROCC | 0.818 | 0.825 | 0.839 | 0.850 |
PLCC | 0.816 | 0.820 | 0.823 | 0.825 |
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. |
© 2024 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Share and Cite
Dziembowski, A.; Nowak, W.; Stankowski, J. IV-SSIM—The Structural Similarity Metric for Immersive Video. Appl. Sci. 2024, 14, 7090. https://doi.org/10.3390/app14167090
Dziembowski A, Nowak W, Stankowski J. IV-SSIM—The Structural Similarity Metric for Immersive Video. Applied Sciences. 2024; 14(16):7090. https://doi.org/10.3390/app14167090
Chicago/Turabian StyleDziembowski, Adrian, Weronika Nowak, and Jakub Stankowski. 2024. "IV-SSIM—The Structural Similarity Metric for Immersive Video" Applied Sciences 14, no. 16: 7090. https://doi.org/10.3390/app14167090