Enhancement of Underwater Images through Parallel Fusion of Transformer and CNN
Abstract
:1. Introduction
- (1)
- A novel transformer model that extracts local features has been proposed. It incorporates both PSNR attention and linear operations to effectively reduce computational load and alleviate color artifacts.
- (2)
- Furthermore, a novel global feature extraction network is devised, which leverages both temporal and frequency domain characters to enrich image features.
- (3)
- Additionally, a feature fusion method leveraging Fourier transform has been proposed to optimize global and local feature weights. High-frequency Fourier components enhance the global features, while low-frequency components refine local features. This approach effectively integrates various features without leaving noticeable fusion trace at the boundaries.
2. Materials and Methods
2.1. Two Branches’ Feature Extraction Network
2.2. Implementation of the CNN Branch
2.3. Implementation of the Transformer Branch
2.4. Fusion Attention Based on High-Pass and Low-Pass Filters
3. Experimental Validation
3.1. Dataset and Experimental Designation
3.2. Ablation Study
3.3. Feature Visualization Process
3.4. Comparison with Current Methods
3.4.1. Qualitative Analysis
3.4.2. Quantitative Analysis
3.5. Comparison on Detection Tasks
4. Conclusions
Author Contributions
Funding
Institutional Review Board Statement
Informed Consent Statement
Data Availability Statement
Conflicts of Interest
References
- Zhang, W.; Liu, W.; Li, L. Underwater Single-Image Restoration with Transmission Estimation Using Color Constancy. J. Mar. Sci. Eng. 2022, 10, 430. [Google Scholar] [CrossRef]
- Chiang, J.Y.; Chen, Y.-C. Underwater Image Enhancement by Wavelength Compensation and Dehazing. IEEE Trans. Image Process. 2011, 21, 1756–1769. [Google Scholar] [CrossRef] [PubMed]
- Yang, H.; Tian, F.; Qi, Q.; Wu, Q.M.J.; Li, K. Underwater image enhancement with latent consistency learning-based color transfer. IET Image Process. 2022, 16, 1594–1612. [Google Scholar] [CrossRef]
- Mustafa, W.A.; Kader, M.M.M.A. A Review of Histogram Equalization Techniques in Image Enhancement Application. J. Physics: Conf. Ser. 2018, 1019, 012026. [Google Scholar] [CrossRef]
- Zhou, J.; Wei, X.; Shi, J.; Chu, W.; Zhang, W. Underwater image enhancement method with light scattering characteristics. Comput. Electr. Eng. 2022, 100, 898–915. [Google Scholar] [CrossRef]
- Peng, Y.-T.; Zhao, X.; Cosman, P.C. Single underwater image enhancement using depth estimation based on blurriness. In Proceedings of the IEEE International Conference on Image Processing (ICIP), Quebec City, QC, Canada, 27–30 September 2015; pp. 4952–4956. [Google Scholar] [CrossRef]
- Song, W.; Wang, Y.; Huang, D. A rapid scene depth estimation model based on underwater light attenuation prior for under-water image restoration. In Proceedings of the 2018 Advances in Multimedia Information Processing, Hefei, China, 21–22 September 2018; pp. 678–688. [Google Scholar] [CrossRef]
- Cheng, C.; Zhang, H.; Li, G. Overview of Underwater Image Enhancement and Restoration Methods. In Proceedings of the International Conference on CYBER Technology in Automation, Control, and Intelligent Systems (CYBER), Baishan, China, 27–31 July 2022; pp. 520–525. [Google Scholar] [CrossRef]
- Drews, P.; Nascimento, E.R.; Botelho, S.S.C.; Campos, M.F.M. Underwater Depth Estimation and Image Restoration Based on Single Images. IEEE Comput. Graph. Appl. 2016, 36, 24–35. [Google Scholar] [CrossRef]
- Li, J.; Hou, G.; Wang, G. Underwater image restoration using oblique gradient operator and light attenuation prior. Multimedia Tools Appl. 2023, 82, 6625–6645. [Google Scholar] [CrossRef]
- Ma, Z.; Oh, C. A wavelet-based dual-stream network for underwater image enhancement. In Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing, Singapore, 23–27 May 2022; pp. 2769–2773. [Google Scholar] [CrossRef]
- Zhang, Y.; Jiang, Q.; Liu, P.; Gao, S.; Pan, X.; Zhang, C. Underwater Image Enhancement Using Deep Transfer Learning Based on a Color Restoration Model. IEEE J. Ocean. Eng. 2023, 48, 489–514. [Google Scholar] [CrossRef]
- Wang, K.; Hu, Y.; Chen, J.; Wu, X.; Zhao, X.; Li, Y. Underwater Image Restoration Based on a Parallel Convolutional Neural Network. Remote. Sens. 2019, 11, 1591. [Google Scholar] [CrossRef]
- Ueki, Y.; Ikehara, M. Underwater Image Enhancement with Multi-Scale Residual Attention Network. In Proceedings of the IEEE International Conference on Visual Communications and Image Processing (VCIP), Munich, Germany, 5–8 December 2021; pp. 1–5. [Google Scholar] [CrossRef]
- Xing, Z.; Cai, M.; Li, J. Improved Shallow-UWnet for Underwater Image Enhancement. In Proceedings of the International Conference on Unmanned Systems (ICUS), Guangzhou, China, 28–30 October 2022; pp. 1191–1196. [Google Scholar] [CrossRef]
- Li, C.; Anwar, S.; Porikli, F. Underwater scene prior inspired deep underwater image and video enhancement. Pattern Recognit. 2020, 98, 107038. [Google Scholar] [CrossRef]
- Chen, C.; Chen, Q.; Xu, J.; Koltun, V. Learning to See in the Dark. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA, 18–23 June 2018; pp. 3291–3300. [Google Scholar] [CrossRef]
- Wang, H.; Yang, M.; Yin, G.; Dong, J. Self-Adversarial Generative Adversarial Network for Underwater Image Enhancement. IEEE J. Ocean. Eng. 2024, 49, 237–248. [Google Scholar] [CrossRef]
- Wang, Y.; Er, M.J.; Chen, J.; Wu, J. A Novel Generative Adversarial Network for Underwater Image Enhancement. In Proceedings of the International Conference on Intelligent Autonomous Systems (ICoIAS), Dalian, China, 23–25 September 2022; pp. 84–89. [Google Scholar] [CrossRef]
- Fabbri, C.; Islam, M.J.; Sattar, J. Enhancing underwater imagery using generative adversarial networks. In Proceedings of the 2018 IEEE International Conference on Robotics and Automation (ICRA), Brisbane, Australia, 21–25 May 2018; pp. 7159–7165. [Google Scholar] [CrossRef]
- Balakrishnan, G.; Zhao, A.; Dalca, A.V.; Durand, F.; Guttag, J. Synthesizing images of humans in unseen poses. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA, 18–23 June 2018; pp. 8340–8348. [Google Scholar] [CrossRef]
- Hu, X.; Naiel, M.A.; Wong, A.; Lamm, M.; Fieguth, P. RUNet: A robust UNet architecture for image super-resolution. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), Long Beach, CA, USA, 16–17 June 2019; pp. 505–507. [Google Scholar] [CrossRef]
- Wu, J.; Liu, X.; Lu, Q.; Lin, Z.; Qin, N.; Shi, Q. FW-GAN: Underwater image enhancement using generative adversarial network with multi-scale fusion. Signal Process. Image Commun. 2022, 109, 116855. [Google Scholar] [CrossRef]
- Terayama, K.; Shin, K.; Mizuno, K.; Tsuda, K. Integration of sonar and optical camera images using deep neural network for fish monitoring. Aquac. Eng. 2019, 86, 102000. [Google Scholar] [CrossRef]
- Zhang, T.; Johnson-Roberson, M. Beyond NeRF Underwater: Learning Neural Reflectance Fields for True Color Correction of Marine Imagery. IEEE Robot. Autom. Lett. 2023, 8, 6467–6474. [Google Scholar] [CrossRef]
- Liu, Z.; Lin, Y.; Cao, Y.; Hu, H.; Wei, Y.; Zhang, Z.; Lin, S.; Guo, B. Swin Transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF International Con-ference on Computer Vision, Montreal, QC, Canada, 10–17 October 2021; pp. 9992–10002. [Google Scholar] [CrossRef]
- Kovács, L.; Csépányi-Fürjes, L.; Tewabe, W. Transformer Models in Natural Language Processing. In International Conference In-terdisciplinarity in Engineering; Lecture Notes in Networks and Systems; Springer Nature Switzerland: Cham, Switzerland, 2023; Volume 929–945. [Google Scholar] [CrossRef]
- Liu, C.; Wang, G.; Zhang, C.; Patimisco, P.; Cui, R.; Feng, C.; Sampaolo, A.; Spagnolo, V.; Dong, L.; Wu, H. End-to-end methane gas detection algorithm based on transformer and multi-layer perceptron. Opt. Express 2024, 32, 987–1002. [Google Scholar] [CrossRef]
- Zamir, S.; Arora, A.; Khan, S. Restormer: Efficient transformer for high-resolution image restoration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, LA, USA, 18–24 June 2022; IEEE Press: Piscataway, NJ, USA, 2022; pp. 5718–5729. [Google Scholar] [CrossRef]
- Song, Y.; He, Z.; Qian, H.; Du, X. Vision Transformers for Single Image Dehazing. IEEE Trans. Image Process. 2023, 32, 1927–1941. [Google Scholar] [CrossRef] [PubMed]
- Berman, D.; Levy, D.; Avidan, S.; Treibitz, T. Underwater single image color restoration using haze-lines and a new quantita-tive dataset. IEEE Trans. Pattern Anal. Mach. Intell. 2021, 43, 2822–2837. [Google Scholar] [CrossRef]
- Charu, C. Aggarwal. Neural Networks and Deep Learning; Springer: Berlin/Heidelberg, Germany, 2018. [Google Scholar]
- Lu, Z.; Jiang, X.; Kot, A. Deep Coupled ResNet for Low-Resolution Face Recognition. IEEE Signal Process. Lett. 2018, 25, 526–530. [Google Scholar] [CrossRef]
- Huang, J.; Liu, Y.; Zhao, F.; Yan, K.; Zhang, J.; Huang, Y.; Zhou, M.; Xiong, Z. Deep Fourier-Based Exposure Correction Network with Spatial-Frequency Interaction. In Proceedings of the European Conference Computer Vision, Tel Aviv, Israel, 23–27 October 2022; Springer: Cham, Switzerland, 2022; pp. 163–180. [Google Scholar] [CrossRef]
- Zhou, J.; Ni, J.; Rao, Y. Block-Based Convolutional Neural Network for Image Forgery Detection. In Proceedings of the Digital Forensics and Watermarking: 16th International Workshop IWDW, Magdeburg, Germany, 23–25 August 2017; Lecture Notes in Computer Science. Springer International Publishing: Cham, Switzerland, 2017; Volume 10431. [Google Scholar] [CrossRef]
- Zou, B.J.; Guo, Y.D.; He, Q.; Ouyang, P.B.; Liu, K.; Chen, Z.L. 3D Filtering by Block Matching and Convolutional Neural Network for Image Denoising. J. Comput. Sci. Technol. 2018, 33, 838–848. [Google Scholar] [CrossRef]
- Abbas Shahri, A.; Maghsoudi Moud, F. Landslide susceptibility mapping using hybridized block modular intelligence model. Bull. Eng. Geol. Environ. 2021, 80, 267–284. [Google Scholar] [CrossRef]
- Liu, Q.; Su, Y.; Xu, P. Implementation of Artificial Intelligence Anime Styl-ization System Based on PyTorch. In Proceedings of the Annual International Conference on Net-work and Information Systems for Computers (ICNISC), Wuhan, China, 27–29 October 2023; pp. 84–87. [Google Scholar] [CrossRef]
- Peng, L.; Zhu, C.; Bian, L. U-Shape Transformer for Underwater Image Enhancement. IEEE Trans. Image Process. 2023, 32, 3066–3079. [Google Scholar] [CrossRef]
- Li, C.; Guo, C.; Ren, W.; Cong, R.; Hou, J.; Kwong, S.; Tao, D. An Underwater Image Enhancement Benchmark Dataset and Beyond. IEEE Trans. Image Process. 2019, 29, 4376–4389. [Google Scholar] [CrossRef]
- Basha, C.; Pravallika, B.; Shankar, E. An Efficient Face Mask Detector with PyTorch and Deep Learning. EAI Endorsed Trans. Pervasive Health Technol. 2021, 7, 167843. [Google Scholar] [CrossRef]
- Li, W.; Li, S.; Liu, R. Channel Shuffle Reconstruction Network for Image Compressive Sensing. In Proceedings of the IEEE International Conference on Image Processing (ICIP), Abu Dhabi, United Arab Emirates, 25–28 October 2020; pp. 2880–2884. [Google Scholar] [CrossRef]
- Zhang, Y.; Liu, Y.; Li, X.; Zhang, C. Salt and pepper noise removal in surveillance video based on low-rank matrix recovery. Comput. Vis. Media 2015, 1, 59–68. [Google Scholar] [CrossRef]
- Yao, J.; Liu, G. Improved SSIM IQA of contrast distortion based on the contrast sensitivity characteristics of HVS. IET Image Process. 2018, 12, 872–879. [Google Scholar] [CrossRef]
- Liu, R.; Jiang, Z.; Yang, S.; Fan, X. Twin Adversarial Contrastive Learning for Underwater Image Enhancement and Beyond. IEEE Trans. Image Process. 2022, 31, 4922–4936. [Google Scholar] [CrossRef]
- Zamir, S.W.; Arora, A.; Khan, S.; Hayat, M.; Khan, F.S.; Yang, M.H.; Shao, L. Learning enriched features for real image restoration and en-hancement. IEEE Trans. Pattern Anal. Mach. Intell. 2020, 45, 1934–1948. [Google Scholar] [CrossRef]
- Liu, X.; Chen, G.; Sun, X.; Knoll, A. Ground Moving Vehicle Detection and Movement Tracking Based On the Neuromorphic Vision Sensor. IEEE Internet Things J. 2020, 7, 9026–9039. [Google Scholar] [CrossRef]
- Liu, X.; Yang, Z.; Hou, J.; Huang, W. Dynamic Scene’s Laser Localization by NeuroIV-based Moving Objects Detection and LIDAR Points Evaluation. IEEE Trans. Geosci. Remote Sens. 2022, 6, 5230414. [Google Scholar] [CrossRef]
- Tan, L.; Huangfu, T.; Wu, L.; Chen, W. Comparison of RetinaNet, SSD, and YOLO v3 for real-time pill identification. BMC Med. Informatics Decis. Mak. 2021, 21, 324–337. [Google Scholar] [CrossRef]
- Ronneberger, O.; Fischer, P.; Brox, T. U-net: Convolutional network for biomedical image segmentation. In Proceedings of the Medical Image Computing and Computer-Assisted Intervention (MICCAI), Munich, Germany, 5–9 October 2015; pp. 234–241. [Google Scholar] [CrossRef]
Parameters | Computation Load | Computation Load Summation |
---|---|---|
d*n | 3d*n | |
q | d*n | |
d*n |
Structures | Fusion | LSUI | UIEB | ||||||
---|---|---|---|---|---|---|---|---|---|
CNN | Fourier | Transformer | SNR Attention | Additive Fusion | Fourier Fusion | PSNR | SSIM | PSNR | SSIM |
✓ | 15.22 | 0.47 | 13.03 | 0.42 | |||||
✓ | ✓ | 18.82 | 0.64 | 16.77 | 0.60 | ||||
✓ | ✓ | ✓ | ✓ | 24.83 | 0.79 | 21.70 | 0.70 | ||
✓ | ✓ | ✓ | ✓ | ✓ | 24.42 | 0.75 | 21.56 | 0.69 | |
✓ | ✓ | ✓ | ✓ | ✓ | 26.53 | 0.83 | 23.85 | 0.78 |
Backbones | Attention | Latency (ms) |
---|---|---|
Transformer | Dot-product Transformer | 2.5 ms |
Element-wise Transformer | 2.1 ms | |
Transformer + CNN | Dot-product Transformer | 3.0 ms |
Element-wise Transformer | 2.6 ms |
Methods | LSUI | UIEB | ||
---|---|---|---|---|
PSNR | SSIM | PSNR | SSIM | |
CNN [16] | 15.28 | 0.50 | 13.68 | 0.48 |
MIR-Net [40] | 18.80 | 0.66 | 16.78 | 0.63 |
U-net [41] | 19.45 | 0.78 | 17.46 | 0.76 |
WaterNet [43] | 19.62 | 0.80 | 19.27 | 0.83 |
Ucolor [44] | 21.62 | 0.84 | 20.67 | 0.81 |
Transformer [26] | 22.83 | 0.79 | 21.70 | 0.70 |
Ours | 24.49 | 0.85 | 22.79 | 0.81 |
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. |
© 2024 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Share and Cite
Liu, X.; Chen, Z.; Xu, Z.; Zheng, Z.; Ma, F.; Wang, Y. Enhancement of Underwater Images through Parallel Fusion of Transformer and CNN. J. Mar. Sci. Eng. 2024, 12, 1467. https://doi.org/10.3390/jmse12091467
Liu X, Chen Z, Xu Z, Zheng Z, Ma F, Wang Y. Enhancement of Underwater Images through Parallel Fusion of Transformer and CNN. Journal of Marine Science and Engineering. 2024; 12(9):1467. https://doi.org/10.3390/jmse12091467
Chicago/Turabian StyleLiu, Xiangyong, Zhixin Chen, Zhiqiang Xu, Ziwei Zheng, Fengshuang Ma, and Yunjie Wang. 2024. "Enhancement of Underwater Images through Parallel Fusion of Transformer and CNN" Journal of Marine Science and Engineering 12, no. 9: 1467. https://doi.org/10.3390/jmse12091467