No-Reference Image Quality Assessment with Multi-Scale Orderless Pooling of Deep Features
Abstract
:1. Introduction
1.1. Related Work
1.2. Contributions
1.3. Structure
2. Proposed Method
3. Experimental Results and Analysis
3.1. Datasets
3.2. Evaluation Criteria and Environment
3.3. Parameter Study
3.3.1. Effect of the Number of Patches
3.3.2. Effect of the Scales
3.4. Comparison to the State-of-the-Art
3.5. Cross Database Test
3.6. Computational Complexity of Feature Extraction
4. Conclusions
Funding
Institutional Review Board Statement
Informed Consent Statement
Acknowledgments
Conflicts of Interest
References
- Lin, H.; Hosu, V.; Saupe, D. KonIQ-10K: Towards an ecologically valid and large-scale IQA database. arXiv 2018, arXiv:1803.08489. [Google Scholar]
- Fang, Y.; Zhu, H.; Zeng, Y.; Ma, K.; Wang, Z. Perceptual quality assessment of smartphone photography. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA, 13–19 June 2020; pp. 3677–3686. [Google Scholar]
- Mittal, A.; Moorthy, A.K.; Bovik, A.C. No-reference image quality assessment in the spatial domain. IEEE Trans. Image Process. 2012, 21, 4695–4708. [Google Scholar] [CrossRef]
- Kang, L.; Ye, P.; Li, Y.; Doermann, D. Convolutional neural networks for no-reference image quality assessment. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Columbus, OH, USA, 24–27 June 2014; pp. 1733–1740. [Google Scholar]
- Bosse, S.; Maniry, D.; Müller, K.R.; Wiegand, T.; Samek, W. Deep neural networks for no-reference and full-reference image quality assessment. IEEE Trans. Image Process. 2017, 27, 206–219. [Google Scholar] [CrossRef] [Green Version]
- Ye, P.; Kumar, J.; Kang, L.; Doermann, D. Unsupervised feature learning framework for no-reference image quality assessment. In Proceedings of the 2012 IEEE Conference on Computer Vision and Pattern Recognition, Providence, RI, USA, 16–21 June 2012; IEEE: Piscataway, NJ, USA, 2012; pp. 1098–1105. [Google Scholar]
- Lv, Y.; Jiang, G.; Yu, M.; Xu, H.; Shao, F.; Liu, S. Difference of Gaussian statistical features based blind image quality assessment: A deep learning approach. In Proceedings of the 2015 IEEE International Conference on Image Processing (ICIP), Quebec City, QC, Canada, 27–30 September 2015; IEEE: Piscataway, NJ, USA, 2015; pp. 2344–2348. [Google Scholar]
- Li, Y.; Po, L.M.; Xu, X.; Feng, L.; Yuan, F.; Cheung, C.H.; Cheung, K.W. No-reference image quality assessment with shearlet transform and deep neural networks. Neurocomputing 2015, 154, 94–109. [Google Scholar] [CrossRef]
- Gao, X.; Gao, F.; Tao, D.; Li, X. Universal blind image quality assessment metrics via natural scene statistics and multiple kernel learning. IEEE Trans. Neural Netw. Learn. Syst. 2013, 24, 2013–2026. [Google Scholar] [CrossRef] [PubMed]
- Moorthy, A.K.; Bovik, A.C. A two-step framework for constructing blind image quality indices. IEEE Signal Process. Lett. 2010, 17, 513–516. [Google Scholar] [CrossRef]
- Saad, M.A.; Bovik, A.C.; Charrier, C. Blind image quality assessment: A natural scene statistics approach in the DCT domain. IEEE Trans. Image Process. 2012, 21, 3339–3352. [Google Scholar] [CrossRef] [PubMed]
- Saad, M.A.; Bovik, A.C.; Charrier, C. A DCT statistics-based blind image quality index. IEEE Signal Process. Lett. 2010, 17, 583–586. [Google Scholar] [CrossRef] [Green Version]
- Ye, P.; Doermann, D. No-reference image quality assessment using visual codebooks. IEEE Trans. Image Process. 2012, 21, 3129–3138. [Google Scholar] [PubMed]
- Varga, D. No-reference image quality assessment based on the fusion of statistical and perceptual features. J. Imaging 2020, 6, 75. [Google Scholar] [CrossRef]
- Varga, D. No-Reference Image Quality Assessment with Global Statistical Features. J. Imaging 2021, 7, 29. [Google Scholar] [CrossRef]
- Li, J.; Zou, L.; Yan, J.; Deng, D.; Qu, T.; Xie, G. No-reference image quality assessment using Prewitt magnitude based on convolutional neural networks. Signal Image Video Process. 2016, 10, 609–616. [Google Scholar] [CrossRef]
- Hou, W.; Gao, X.; Tao, D.; Li, X. Blind image quality assessment via deep learning. IEEE Trans. Neural Netw. Learn. Syst. 2014, 26, 1275–1286. [Google Scholar]
- Ravela, R.; Shirvaikar, M.; Grecos, C. No-reference image quality assessment based on deep convolutional neural networks. In Proceedings of the Real-Time Image Processing and Deep Learning 2019, International Society for Optics and Photonics, Baltimore, MD, USA, 15–16 April 2019; Volume 10996, p. 1099604. [Google Scholar]
- Fan, C.; Zhang, Y.; Feng, L.; Jiang, Q. No reference image quality assessment based on multi-expert convolutional neural networks. IEEE Access 2018, 6, 8934–8943. [Google Scholar] [CrossRef]
- Krizhevsky, A.; Sutskever, I.; Hinton, G.E. Imagenet classification with deep convolutional neural networks. Adv. Neural Inf. Process. Syst. 2012, 25, 1097–1105. [Google Scholar] [CrossRef]
- Simonyan, K.; Zisserman, A. Very deep convolutional networks for large-scale image recognition. arXiv 2014, arXiv:1409.1556. [Google Scholar]
- Bianco, S.; Celona, L.; Napoletano, P.; Schettini, R. On the use of deep learning for blind image quality assessment. Signal Image Video Process. 2018, 12, 355–362. [Google Scholar] [CrossRef]
- Gao, F.; Yu, J.; Zhu, S.; Huang, Q.; Tian, Q. Blind image quality prediction by exploiting multi-level deep representations. Pattern Recognit. 2018, 81, 432–442. [Google Scholar] [CrossRef]
- Varga, D. Multi-pooled inception features for no-reference image quality assessment. Appl. Sci. 2020, 10, 2186. [Google Scholar] [CrossRef] [Green Version]
- Lin, K.Y.; Wang, G. Hallucinated-IQA: No-reference image quality assessment via adversarial learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA, 18–22 June 2018; pp. 732–741. [Google Scholar]
- Goodfellow, I.J.; Pouget-Abadie, J.; Mirza, M.; Xu, B.; Warde-Farley, D.; Ozair, S.; Courville, A.; Bengio, Y. Generative adversarial networks. arXiv 2014, arXiv:1406.2661. [Google Scholar] [CrossRef]
- Ma, J.; Wu, J.; Li, L.; Dong, W.; Xie, X.; Shi, G.; Lin, W. Blind Image Quality Assessment with Active Inference. IEEE Trans. Image Process. 2021, 30, 3650–3663. [Google Scholar] [CrossRef] [PubMed]
- Jégou, H.; Douze, M.; Schmid, C.; Pérez, P. Aggregating local descriptors into a compact image representation. In Proceedings of the 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, San Francisco, CA, USA, 13–18 June 2010; IEEE: Piscataway, NJ, USA, 2010; pp. 3304–3311. [Google Scholar]
- Zhang, R.; Isola, P.; Efros, A.A.; Shechtman, E.; Wang, O. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA, 18–22 June 2018; pp. 586–595. [Google Scholar]
- Babenko, A.; Lempitsky, V. Aggregating local deep features for image retrieval. In Proceedings of the IEEE International Conference on Computer Vision, Santiago, Chile, 11–18 December 2015; pp. 1269–1277. [Google Scholar]
- Luo, Z.; Mishra, A.; Achkar, A.; Eichel, J.; Li, S.; Jodoin, P.M. Non-local deep features for salient object detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA, 21–26 July 2017; pp. 6609–6617. [Google Scholar]
- Ma, J.; Jiang, X.; Fan, A.; Jiang, J.; Yan, J. Image matching from handcrafted to deep features: A survey. Int. J. Comput. Vis. 2021, 129, 23–79. [Google Scholar] [CrossRef]
- Lazebnik, S.; Schmid, C.; Ponce, J. Beyond bags of features: Spatial pyramid matching for recognizing natural scene categories. In Proceedings of the 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’06), New York, NY, USA, 17–22 June 2006; IEEE: Piscataway, NJ, USA, 2006; Volume 2, pp. 2169–2178. [Google Scholar]
- Ghadiyaram, D.; Bovik, A.C. Massive online crowdsourced study of subjective and objective picture quality. IEEE Trans. Image Process. 2015, 25, 372–387. [Google Scholar] [CrossRef] [PubMed] [Green Version]
- Deng, J.; Dong, W.; Socher, R.; Li, L.J.; Li, K.; Fei-Fei, L. Imagenet: A large-scale hierarchical image database. In Proceedings of the 2009 IEEE Conference on Computer Vision and Pattern Recognition, Miami, FL, USA, 20–25 June 2009; IEEE: Piscataway, NJ, USA, 2009; pp. 248–255. [Google Scholar]
- Rasmussen, C.E.; Nickisch, H. Gaussian processes for machine learning (GPML) toolbox. J. Mach. Learn. Res. 2010, 11, 3011–3015. [Google Scholar]
- Zhang, N.; Xiong, J.; Zhong, J.; Leatham, K. Gaussian process regression method for classification for high-dimensional data with limited samples. In Proceedings of the 2018 Eighth International Conference on Information Science and Technology (ICIST), Cordoba, Spain, 30 June–6 July 2018; IEEE: Piscataway, NJ, USA, 2018; pp. 358–363. [Google Scholar]
- Thomee, B.; Shamma, D.A.; Friedland, G.; Elizalde, B.; Ni, K.; Poland, D.; Borth, D.; Li, L.J. YFCC100M: The new data in multimedia research. Commun. ACM 2016, 59, 64–73. [Google Scholar] [CrossRef]
- Ponomarenko, N.; Jin, L.; Ieremeiev, O.; Lukin, V.; Egiazarian, K.; Astola, J.; Vozel, B.; Chehdi, K.; Carli, M.; Battisti, F.; et al. Image database TID2013: Peculiarities, results and perspectives. Signal Process. Image Commun. 2015, 30, 57–77. [Google Scholar] [CrossRef] [Green Version]
- Varga, D.; Saupe, D.; Szirányi, T. DeepRN: A content preserving deep architecture for blind image quality assessment. In Proceedings of the 2018 IEEE International Conference on Multimedia and Expo (ICME), San Diego, CA, USA, 23–27 July 2018; IEEE: Piscataway, NJ, USA, 2018; pp. 1–6. [Google Scholar]
- Min, X.; Zhai, G.; Gu, K.; Liu, Y.; Yang, X. Blind image quality estimation via distortion aggravation. IEEE Trans. Broadcast. 2018, 64, 508–517. [Google Scholar] [CrossRef]
- Liu, L.; Dong, H.; Huang, H.; Bovik, A.C. No-reference image quality assessment in curvelet domain. Signal Process. Image Commun. 2014, 29, 494–505. [Google Scholar] [CrossRef]
- Moorthy, A.K.; Bovik, A.C. Blind image quality assessment: From natural scene statistics to perceptual quality. IEEE Trans. Image Process. 2011, 20, 3350–3364. [Google Scholar] [CrossRef]
- Chen, X.; Zhang, Q.; Lin, M.; Yang, G.; He, C. No-reference color image quality assessment: From entropy to perceptual quality. EURASIP J. Image Video Process. 2019, 2019, 77. [Google Scholar] [CrossRef] [Green Version]
- Xue, W.; Mou, X.; Zhang, L.; Bovik, A.C.; Feng, X. Blind image quality assessment using joint statistics of gradient magnitude and Laplacian features. IEEE Trans. Image Process. 2014, 23, 4850–4862. [Google Scholar] [CrossRef] [PubMed]
- Ou, F.Z.; Wang, Y.G.; Zhu, G. A novel blind image quality assessment method based on refined natural scene statistics. In Proceedings of the 2019 IEEE International Conference on Image Processing (ICIP), Taipei, Taiwan, 22–25 September 2019; IEEE: Piscataway, NJ, USA, 2019; pp. 1004–1008. [Google Scholar]
- Venkatanath, N.; Praneeth, D.; Bh, M.C.; Channappayya, S.S.; Medasani, S.S. Blind image quality evaluation using perception based features. In Proceedings of the 2015 Twenty First National Conference on Communications (NCC), Bombay, India, 27 February–1 March 2015; IEEE: Piscataway, NJ, USA, 2015; pp. 1–6. [Google Scholar]
- Liu, L.; Hua, Y.; Zhao, Q.; Huang, H.; Bovik, A.C. Blind image quality assessment by relative gradient statistics and adaboosting neural network. Signal Process. Image Commun. 2016, 40, 1–15. [Google Scholar] [CrossRef]
- Liu, L.; Liu, B.; Huang, H.; Bovik, A.C. No-reference image quality assessment based on spatial and spectral entropies. Signal Process. Image Commun. 2014, 29, 856–863. [Google Scholar] [CrossRef]
Database | Year | #Distorted Images | Resolution | Environment |
---|---|---|---|---|
TID2013 [39] | 2013 | 3000 | laboratory | |
CLIVE [34] | 2015 | 1162 | crowdsourcing | |
KonIQ-10k [1] | 2018 | 10,073 | crowdsourcing | |
SPAQ [2] | 2020 | 11,125 | ∼ | laboratory |
Computer model | STRIX Z270H Gaming |
CPU | Intel(R) Core(TM) i7-7700K CPU 4.20 GHz (8 cores) |
Memory | 15 GB |
GPU | Nvidia GeForce GTX 1080 |
Network | Depth | Size | Parameters (Millions) | Image Input Size |
---|---|---|---|---|
AlexNet [20] | 8 | 227 MB | 61.0 | |
VGG16 [21] | 16 | 515 MB | 138 | |
VGG19 [21] | 19 | 535 MB | 144 |
CLIVE [34] | ||||
---|---|---|---|---|
#Patches—Scale2 | #Patches—Scale3 | PLCC | SROCC | KROCC |
3 | 4 | 0.825 | 0.797 | 0.604 |
6 | 8 | 0.827 | 0.800 | 0.607 |
9 | 12 | 0.828 | 0.800 | 0.607 |
12 | 16 | 0.831 | 0.801 | 0.607 |
15 | 20 | 0.831 | 0.801 | 0.607 |
KonIQ-10k [1] | ||||
---|---|---|---|---|
#Patches—Scale2 | #Patches—Scale3 | PLCC | SROCC | KROCC |
3 | 4 | 0.888 | 0.872 | 0.690 |
6 | 8 | 0.895 | 0.878 | 0.696 |
9 | 12 | 0.898 | 0.882 | 0.701 |
12 | 16 | 0.899 | 0.884 | 0.703 |
15 | 20 | 0.901 | 0.885 | 0.703 |
CLIVE [34] | KonIQ-10k [1] | |||||
---|---|---|---|---|---|---|
PLCC | SROCC | KROCC | PLCC | SROCC | KROCC | |
Scale 1 | 0.810 | 0.778 | 0.586 | 0.888 | 0.873 | 0.687 |
Scale 2 | 0.817 | 0.787 | 0.595 | 0.893 | 0.874 | 0.690 |
Scale 3 | 0.830 | 0.800 | 0.600 | 0.900 | 0.883 | 0.700 |
All | 0.831 | 0.801 | 0.607 | 0.901 | 0.885 | 0.703 |
CLIVE [34] | KonIQ-10k [1] | |||||
---|---|---|---|---|---|---|
Method | PLCC | SROCC | KROCC | PLCC | SROCC | KROCC |
BLIINDER [23] | 0.782 | 0.763 | 0.576 | 0.876 | 0.864 | 0.668 |
DeepRN [40] | 0.784 | 0.753 | 0.579 | 0.866 | 0.880 | 0.666 |
BLIINDS-II [11] | 0.473 | 0.442 | 0.291 | 0.574 | 0.575 | 0.414 |
BMPRI [41] | 0.541 | 0.487 | 0.333 | 0.637 | 0.619 | 0.421 |
BRISQUE [3] | 0.524 | 0.497 | 0.345 | 0.707 | 0.677 | 0.494 |
CurveletQA [42] | 0.636 | 0.621 | 0.421 | 0.730 | 0.718 | 0.495 |
DIIVINE [43] | 0.617 | 0.580 | 0.405 | 0.709 | 0.693 | 0.471 |
ENIQA [44] | 0.596 | 0.564 | 0.376 | 0.761 | 0.745 | 0.544 |
GRAD-LOG-CP [45] | 0.607 | 0.604 | 0.383 | 0.705 | 0.696 | 0.501 |
NBIQA [46] | 0.629 | 0.604 | 0.427 | 0.771 | 0.749 | 0.515 |
PIQE [47] | 0.172 | 0.108 | 0.081 | 0.208 | 0.246 | 0.172 |
OG-IQA [48] | 0.545 | 0.505 | 0.364 | 0.652 | 0.635 | 0.447 |
SSEQ [49] | 0.487 | 0.436 | 0.309 | 0.589 | 0.572 | 0.423 |
MSDF-IQA | 0.831 | 0.801 | 0.607 | 0.901 | 0.885 | 0.703 |
SPAQ [2] | |||
---|---|---|---|
Method | PLCC | SROCC | KROCC |
BLIINDER [23] | 0.872 | 0.869 | 0.683 |
DeepRN [40] | 0.870 | 0.850 | 0.676 |
BLIINDS-II [11] | 0.676 | 0.675 | 0.486 |
BMPRI [41] | 0.739 | 0.734 | 0.506 |
BRISQUE [3] | 0.726 | 0.720 | 0.518 |
CurveletQA [42] | 0.793 | 0.774 | 0.503 |
DIIVINE [43] | 0.774 | 0.756 | 0.514 |
ENIQA [44] | 0.813 | 0.804 | 0.603 |
GRAD-LOG-CP [45] | 0.786 | 0.782 | 0.572 |
NBIQA [46] | 0.802 | 0.793 | 0.539 |
PIQE [47] | 0.211 | 0.156 | 0.091 |
OG-IQA [48] | 0.726 | 0.724 | 0.594 |
SSEQ [49] | 0.745 | 0.742 | 0.549 |
MSDF-IQA | 0.900 | 0.894 | 0.692 |
TID2013 [39] | |||
---|---|---|---|
Method | PLCC | SROCC | KROCC |
BLIINDER [23] | 0.834 | 0.816 | 0.720 |
DeepRN [40] | 0.745 | 0.636 | 0.560 |
BLIINDS-II [11] | 0.558 | 0.513 | 0.339 |
BMPRI [41] | 0.701 | 0.588 | 0.427 |
BRISQUE [3] | 0.478 | 0.427 | 0.278 |
CurveletQA [42] | 0.553 | 0.505 | 0.359 |
DIIVINE [43] | 0.692 | 0.599 | 0.431 |
ENIQA [44] | 0.604 | 0.555 | 0.397 |
GRAD-LOG-CP [45] | 0.671 | 0.627 | 0.470 |
NBIQA [46] | 0.723 | 0.628 | 0.427 |
PIQE [47] | 0.464 | 0.365 | 0.257 |
OG-IQA [48] | 0.564 | 0.452 | 0.321 |
SSEQ [49] | 0.618 | 0.520 | 0.375 |
MSDF-IQA | 0.727 | 0.448 | 0.311 |
CLIVE [34] | KonIQ-10k [1] | SPAQ [2] | TID2013 [39] | |
---|---|---|---|---|
BLIINDER [23] | 1 | 1 | 1 | 0 |
DeepRN [40] | 1 | 1 | 1 | 0 |
BLIINDS-II [11] | 1 | 1 | 1 | 1 |
BMPRI [41] | 1 | 1 | 1 | 1 |
BRISQUE [3] | 1 | 1 | 1 | 1 |
CurveletQA [42] | 1 | 1 | 1 | 1 |
DIIVINE [43] | 1 | 1 | 1 | 1 |
ENIQA [44] | 1 | 1 | 1 | 1 |
GRAD-LOG-CP [45] | 1 | 1 | 1 | 1 |
NBIQA [46] | 1 | 1 | 1 | 1 |
PIQE [47] | 1 | 1 | 1 | 1 |
OG-IQA [48] | 1 | 1 | 1 | 1 |
SSEQ [49] | 1 | 1 | 1 | 1 |
Method | PLCC | SROCC | KROCC |
---|---|---|---|
BLIINDER [23] | 0.748 | 0.730 | 0.503 |
DeepRN [40] | 0.746 | 0.725 | 0.481 |
BLIINDS-II [11] | 0.107 | 0.090 | 0.063 |
BMPRI [41] | 0.453 | 0.389 | 0.298 |
BRISQUE [3] | 0.509 | 0.460 | 0.310 |
CurveletQA [42] | 0.496 | 0.505 | 0.347 |
DIIVINE [43] | 0.479 | 0.434 | 0.299 |
ENIQA [44] | 0.428 | 0.386 | 0.272 |
GRAD-LOG-CP [45] | 0.427 | 0.384 | 0.261 |
NBIQA [46] | 0.503 | 0.509 | 0.284 |
OG-IQA [48] | 0.442 | 0.427 | 0.289 |
SSEQ [49] | 0.270 | 0.256 | 0.170 |
MSDF-IQA | 0.764 | 0.749 | 0.552 |
CLIVE [34] | KonIQ-10k [1] | SPAQ [2] | TID2013 [39] | |
---|---|---|---|---|
BLIINDER [23] | 1.85 | 4.67 | 16.74 | 1.58 |
DeepRN [40] | 1.31 | 1.74 | 5.67 | 1.30 |
BLIINDS-II [11] | 15.23 | 47.25 | 1365.82 | 11.96 |
BMPRI [41] | 0.29 | 0.78 | 21.54 | 0.24 |
BRISQUE [3] | 0.03 | 0.11 | 3.36 | 0.03 |
CurveletQA [42] | 0.65 | 1.75 | 26.65 | 0.49 |
DIIVINE [43] | 6.99 | 18.79 | 543.68 | 5.27 |
ENIQA [44] | 4.19 | 13.00 | 363.22 | 3.25 |
GRAD-LOG-CP [45] | 0.03 | 0.10 | 3.05 | 0.03 |
NBIQA [46] | 6.35 | 20.07 | 580.72 | 5.04 |
PIQE [47] | 0.06 | 0.17 | 4.58 | 0.05 |
OG-IQA [48] | 0.03 | 0.10 | 3.15 | 0.02 |
SSEQ [49] | 0.41 | 1.28 | 36.44 | 0.33 |
MSDF-IQA | 1.45 | 1.94 | 5.85 | 1.34 |
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations. |
© 2021 by the author. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Share and Cite
Varga, D. No-Reference Image Quality Assessment with Multi-Scale Orderless Pooling of Deep Features. J. Imaging 2021, 7, 112. https://doi.org/10.3390/jimaging7070112
Varga D. No-Reference Image Quality Assessment with Multi-Scale Orderless Pooling of Deep Features. Journal of Imaging. 2021; 7(7):112. https://doi.org/10.3390/jimaging7070112
Chicago/Turabian StyleVarga, Domonkos. 2021. "No-Reference Image Quality Assessment with Multi-Scale Orderless Pooling of Deep Features" Journal of Imaging 7, no. 7: 112. https://doi.org/10.3390/jimaging7070112