CT and MRI Image Fusion via Coupled Feature-Learning GAN
Abstract
:1. Introduction
- We propose an end-to-end deep learning-based fusion model termed Coupled Feature-Learning GAN (CFGAN) for preserving the locational information of dense structures, as well as soft tissue details in multi-source images.
- We introduce the discriminative feature extraction (DFE) block with various dilation rates to improve the robustness of generators at diverse scales.
- We design a cross-dimension interaction attention (CIA) block for the coupled generators, integrating the salient information of cross-dimensional features to refine the feature representations.
2. Related Work
2.1. Traditional-Based Methods
2.2. CNN-Based Methods
2.3. GAN-Based Methods
3. Proposed Method
3.1. Overview
Algorithm 1: Training algorithm for CFGAN. |
3.2. Generators Architecture
3.2.1. Network Design
- A Batch-Normalization (BN) layer follows each layer.
- A LReLU [37] activation function in the first four layers.
- A Tanh activation function in the fifth layer.
3.2.2. Discriminative Feature Extraction Block
3.2.3. Cross-Dimension Interaction Attention Block
3.3. Discriminator Architecture
3.4. Loss Function
3.4.1. Generator Loss Function
3.4.2. Discriminator Loss Function
4. Experimental Results and Analysis
4.1. Dataset and Training Details
4.2. Experiments and Analysis
4.2.1. Case Study
4.2.2. Qualitative Comparisons
4.3. Ablation Study
- “Baseline” refers to the vanilla generator model without any component.
- “Baseline + DFE” denotes the baseline model with a single DFE block.
- “Baseline + CIA” represents the baseline model with a single CIA block.
- “Baseline + CIA_DFE” refers to the baseline model with the CIA block and DFE block sequentially connected.
- “Baseline + DFE_CIA” refers to the baseline model with the DFE block and CIA block sequentially connected.
5. Conclusions
6. Future Work
Author Contributions
Funding
Data Availability Statement
Acknowledgments
Conflicts of Interest
References
- Tawfik, N.; Elnemr, H.A.; Fakhr, M.; Dessouky, M.I.; El-Samie, A.; Fathi, E. Survey study of multimodality medical image fusion methods. Multimed. Tools Appl. 2021, 80, 6369–6396. [Google Scholar] [CrossRef]
- Du, J.; Li, W.; Lu, K.; Xiao, B. An overview of multi-modal medical image fusion. Neurocomputing 2016, 215, 3–20. [Google Scholar] [CrossRef]
- Yin, M.; Liu, X.; Liu, Y.; Chen, X. Medical image fusion with parameter-adaptive pulse coupled neural network in nonsubsampled shearlet transform domain. IEEE Trans. Instrum. Meas. 2018, 68, 49–64. [Google Scholar] [CrossRef]
- Huang, B.; Yang, F.; Yin, M.; Mo, X.; Zhong, C. A review of multimodal medical image fusion techniques. Comput. Math. Methods Med. 2020, 2020, 8279342. [Google Scholar] [CrossRef]
- Zhou, T.; Li, Q.; Lu, H.; Cheng, Q.; Zhang, X. GAN review: Models and medical image fusion applications. Inf. Fusion 2023, 91, 134–148. [Google Scholar] [CrossRef]
- Mao, Q.; Yang, X.; Zhang, R.; Jeon, G.; Hussain, F.; Liu, K. Multi-focus images fusion via residual generative adversarial network. Multimed. Tools Appl. 2022, 81, 12305–12323. [Google Scholar] [CrossRef]
- Huang, Y.; Li, W.; Gao, M.; Liu, Z. Algebraic multi-grid based multi-focus image fusion using watershed algorithm. IEEE Access 2018, 6, 47082–47091. [Google Scholar] [CrossRef]
- Li, Q.; Lu, L.; Li, Z.; Wu, W.; Liu, Z.; Jeon, G.; Yang, X. Coupled GAN with relativistic discriminators for infrared and visible images fusion. IEEE Sens. J. 2019, 21, 7458–7467. [Google Scholar] [CrossRef]
- Zhai, W.; Song, W.; Chen, J.; Zhang, G.; Li, Q.; Gao, M. CT and MRI image fusion via dual-branch GAN. Int. J. Biomed. Eng. Technol. 2023, 42, 52–63. [Google Scholar] [CrossRef]
- Li, S.; Kang, X.; Hu, J. Image fusion with guided filtering. IEEE Trans. Image Process. 2013, 22, 2864–2875. [Google Scholar]
- Shreyamsha Kumar, B. Image fusion based on pixel significance using cross bilateral filter. Signal Image Video Process. 2015, 9, 1193–1204. [Google Scholar] [CrossRef]
- Li, W.; Xie, Y.; Zhou, H.; Han, Y.; Zhan, K. Structure-aware image fusion. Optik 2018, 172, 1–11. [Google Scholar] [CrossRef]
- Liu, Y.; Chen, X.; Peng, H.; Wang, Z. Multi-focus image fusion with a deep convolutional neural network. Inf. Fusion 2017, 36, 191–207. [Google Scholar] [CrossRef]
- Li, H.; Wu, X.J. DenseFuse: A fusion approach to infrared and visible images. IEEE Trans. Image Process. 2018, 28, 2614–2623. [Google Scholar] [CrossRef]
- Zhang, Y.; Liu, Y.; Sun, P.; Yan, H.; Zhao, X.; Zhang, L. IFCNN: A general image fusion framework based on convolutional neural network. Inf. Fusion 2020, 54, 99–118. [Google Scholar] [CrossRef]
- Xu, H.; Ma, J.; Le, Z.; Jiang, J.; Guo, X. Fusiondn: A unified densely connected network for image fusion. In Proceedings of the AAAI Conference on Artificial Intelligence, New York, NY, USA, 7–12 February 2020; Volume 34, pp. 12484–12491. [Google Scholar]
- Xu, H.; Ma, J.; Jiang, J.; Guo, X.; Ling, H. U2Fusion: A Unified Unsupervised Image Fusion Network. IEEE Trans. Pattern Anal. Mach. Intell. 2022, 44, 502–518. [Google Scholar] [CrossRef]
- Song, W.; Zeng, X.; Abdelmoniem, A.M.; Zhang, H.; Gao, M. Cross-Modality Interaction Network for Medical Image Fusion. IEEE Trans. Consum. Electron. 2024. [Google Scholar] [CrossRef]
- Song, W.; Zeng, X.; Li, Q.; Gao, M.; Zhou, H.; Shi, J. CT and MRI image fusion via multimodal feature interaction network. Netw. Model. Anal. Health Inform. Bioinform. 2024, 13, 13. [Google Scholar] [CrossRef]
- Goodfellow, I.; Pouget-Abadie, J.; Mirza, M.; Xu, B.; Warde-Farley, D.; Ozair, S.; Courville, A.; Bengio, Y. Generative adversarial nets. Adv. Neural Inf. Process. Syst. 2014, 27. [Google Scholar]
- Ma, J.; Xu, H.; Jiang, J.; Mei, X.; Zhang, X.P. DDcGAN: A dual-discriminator conditional generative adversarial network for multi-resolution image fusion. IEEE Trans. Image Process. 2020, 29, 4980–4995. [Google Scholar] [CrossRef]
- Fu, Y.; Wu, X.J.; Durrani, T. Image fusion based on generative adversarial network consistent with perception. Inf. Fusion 2021, 72, 110–125. [Google Scholar] [CrossRef]
- Yang, Z.; Chen, Y.; Le, Z.; Ma, Y. GANFuse: A novel multi-exposure image fusion method based on generative adversarial networks. Neural Comput. Appl. 2021, 33, 6133–6145. [Google Scholar] [CrossRef]
- RattÁ, G.; Vega, J.; Murari, A.; Contributors, J. Image fusion: Advances in the state of the art. Inf. Fusion 2007, 8, 114–118. [Google Scholar]
- Mitianoudis, N.; Stathaki, T. Pixel-based and region-based image fusion schemes using ICA bases. Inf. Fusion 2007, 8, 131–142. [Google Scholar] [CrossRef]
- Zhang, Q.; Guo, B.l. Multifocus image fusion using the nonsubsampled contourlet transform. Signal Process. 2009, 89, 1334–1346. [Google Scholar] [CrossRef]
- Chen, C.I. Fusion of PET and MR brain images based on IHS and log-Gabor transforms. IEEE Sens. J. 2017, 17, 6995–7010. [Google Scholar] [CrossRef]
- Jian, L.; Yang, X.; Liu, Z.; Jeon, G.; Gao, M.; Chisholm, D. SEDRFuse: A symmetric encoder-decoder with residual block network for infrared and visible image fusion. IEEE Trans. Instrum. Meas. 2020, 70, 1–15. [Google Scholar] [CrossRef]
- Wang, X.; Jiang, J.; Gao, M.; Liu, Z.; Zhao, C. Activation ensemble generative adversarial network transfer learning for image classification. J. Electron. Imaging 2021, 30, 013016. [Google Scholar] [CrossRef]
- Song, W.; Zhai, W.; Gao, M.; Li, Q.; Chehri, A.; Jeon, G. Multiscale aggregation and illumination-aware attention network for infrared and visible image fusion. Concurr. Comput. Pract. Exp. 2024, 36, e7712. [Google Scholar] [CrossRef]
- Song, W.; Gao, M.; Li, Q.; Guo, X.; Wang, Z.; Jeon, G. Optimizing Nighttime Infrared and Visible Image Fusion for Long-haul Tactile Internet. IEEE Trans. Consum. Electron. 2024, 70, 4277–4286. [Google Scholar] [CrossRef]
- Liu, J.; Lin, R.; Wu, G.; Liu, R.; Luo, Z.; Fan, X. Coconet: Coupled contrastive learning network with multi-level feature ensemble for multi-modality image fusion. Int. J. Comput. Vis. 2024, 132, 1748–1775. [Google Scholar] [CrossRef]
- Mu, P.; Wu, G.; Liu, J.; Zhang, Y.; Fan, X.; Liu, R. Learning to Search a Lightweight Generalized Network for Medical Image Fusion. IEEE Trans. Circuits Syst. Video Technol. 2023, 34, 5921–5934. [Google Scholar] [CrossRef]
- Li, J.; Liu, J.; Zhou, S.; Zhang, Q.; Kasabov, N.K. Gesenet: A general semantic-guided network with couple mask ensemble for medical image fusion. IEEE Trans. Neural Netw. Learn. Syst. 2023. [Google Scholar] [CrossRef] [PubMed]
- Ma, J.; Yu, W.; Liang, P.; Li, C.; Jiang, J. FusionGAN: A generative adversarial network for infrared and visible image fusion. Inf. Fusion 2019, 48, 11–26. [Google Scholar] [CrossRef]
- Xu, H.; Ma, J.; Zhang, X.P. MEF-GAN: Multi-exposure image fusion via generative adversarial networks. IEEE Trans. Image Process. 2020, 29, 7203–7216. [Google Scholar] [CrossRef]
- Maas, A.L.; Hannun, A.Y.; Ng, A.Y. Rectifier nonlinearities improve neural network acoustic models. In Proceedings of the International Conference on International Conference on Machine Learning (ICML), Citeseer, Atlanta, GA, USA, 16 June–21 June 2013; Volume 30, p. 3. [Google Scholar]
- Johnson, K.A.; Becker, J.A. The Whole Brain Atlas database of Harvard Medical School. Available online: http://www.med.harvard.edu/aanlib/home.html (accessed on 15 May 2023).
- Parekh, A.; Patil, N.; Biju, R.; Shah, A. Multimodal Medical Image Fusion to Detect Brain Tumors. 2020. Available online: https://github.com/ashna111/multimodal-image-fusion-to-detect-brain-tumors (accessed on 15 May 2023).
- Bavirisetti, D.P.; Kollu, V.; Gang, X.; Dhuli, R. Fusion of MRI and CT images using guided image filter and image statistics. Int. J. Imaging Syst. Technol. 2017, 27, 227–237. [Google Scholar]
- Da, K. A method for stochastic optimization. arXiv 2014, arXiv:1412.6980. [Google Scholar]
- Eskicioglu, A.M.; Fisher, P.S. Image quality measures and their performance. IEEE Trans. Commun. 1995, 43, 2959–2965. [Google Scholar] [CrossRef]
- Wang, Z.; Li, Q. Information content weighting for perceptual image quality assessment. IEEE Trans. Image Process. 2010, 20, 1185–1198. [Google Scholar] [CrossRef]
- Mukaka, M.M. A guide to appropriate use of correlation coefficient in medical research. Malawi Med J. 2012, 24, 69–71. [Google Scholar]
- Wang, Z.; Bovik, A.C.; Sheikh, H.R.; Simoncelli, E.P. Image quality assessment: From error visibility to structural similarity. IEEE Trans. Image Process. 2004, 13, 600–612. [Google Scholar] [CrossRef]
- Han, Y.; Cai, Y.; Cao, Y.; Xu, X. A new image fusion performance metric based on visual information fidelity. Inf. Fusion 2013, 14, 127–135. [Google Scholar] [CrossRef]
- Qu, G.; Zhang, D.; Yan, P. Information measure for performance of image fusion. Electron. Lett. 2002, 38, 313–315. [Google Scholar] [CrossRef]
Methods | SD | PSNR | CC | SSIM | VIF | MI |
---|---|---|---|---|---|---|
GFF [10] | 10.5126 | 15.7217 | 0.7924 | 0.6542 | 0.5450 | 3.0995 |
CBF [11] | 10.4616 | 15.2092 | 0.7695 | 0.6506 | 0.4582 | 3.2984 |
CNN [13] | 10.5146 | 15.3438 | 0.7643 | 0.6614 | 0.5721 | 3.3877 |
SAIF [12] | 10.6046 | 14.8446 | 0.7587 | 0.6575 | 0.5750 | 3.3796 |
FusionGAN [35] | 8.9678 | 12.4935 | 0.7900 | 0.2610 | 0.4436 | 3.2101 |
Densefuse [14] | 9.7168 | 14.1000 | 0.7258 | 0.1653 | 0.2010 | 2.5983 |
IFCNN [15] | 10.6539 | 16.0550 | 0.8132 | 0.6584 | 0.4731 | 3.1806 |
DDcGAN [21] | 10.5925 | 12.2912 | 0.7916 | 0.2341 | 0.3456 | 3.0899 |
FusionDN [16] | 10.5334 | 11.5345 | 0.7938 | 0.2742 | 0.4315 | 3.2145 |
MEF-GAN [36] | 10.5422 | 14.1312 | 0.7878 | 0.6377 | 0.4211 | 3.0332 |
PerceptualFusion [22] | 10.6509 | 12.5574 | 0.8209 | 0.2889 | 0.4393 | 3.2912 |
U2Fusion [17] | 10.4145 | 16.2216 | 0.8094 | 0.3732 | 0.3993 | 3.1125 |
Ours | 10.6910 | 16.5646 | 0.7953 | 0.6836 | 0.5759 | 3.4058 |
Methods | SD | PSNR | CC | SSIM | VIF | MI |
---|---|---|---|---|---|---|
GFF [10] | 9.4583 | 14.8625 | 0.8124 | 0.7089 | 0.6221 | 2.7335 |
CBF [11] | 9.2601 | 13.8956 | 0.7903 | 0.7014 | 0.4652 | 2.7443 |
CNN [13] | 9.2192 | 18.2652 | 0.7777 | 0.7408 | 0.5749 | 3.2288 |
SAIF [12] | 9.2985 | 13.6398 | 0.7526 | 0.7253 | 0.6994 | 2.9468 |
FusionGAN [35] | 8.0180 | 13.1487 | 0.8258 | 0.1782 | 0.4517 | 2.7466 |
Densefuse [14] | 9.3583 | 11.6880 | 0.6073 | 0.0601 | 0.0659 | 1.8011 |
IFCNN [15] | 9.4512 | 15.7070 | 0.8453 | 0.6850 | 0.5271 | 2.8248 |
DDcGAN [21] | 9.4113 | 10.4151 | 0.8007 | 0.1418 | 0.2751 | 2.5835 |
FusionDN [16] | 9.1808 | 10.0367 | 0.7838 | 0.2000 | 0.4092 | 2.6537 |
MEF-GAN [36] | 9.2921 | 14.8140 | 0.8339 | 0.6550 | 0.4572 | 2.8217 |
PerceptualFusion [22] | 9.5938 | 12.9201 | 0.8544 | 0.2183 | 0.4480 | 2.7561 |
U2Fusion [17] | 9.2378 | 15.1122 | 0.8453 | 0.2495 | 0.4489 | 2.7753 |
CFGAN (Ours) | 9.5857 | 15.9410 | 0.8350 | 0.7410 | 0.6541 | 3.0756 |
Methods | SD | PSNR | CC | SSIM | VIF | MI |
---|---|---|---|---|---|---|
GFF [10] | 10.3692 | 14.3832 | 0.7663 | 0.6704 | 0.5638 | 3.0355 |
CBF [11] | 10.3748 | 13.9841 | 0.7378 | 0.6702 | 0.4505 | 3.2002 |
CNN [13] | 9.8755 | 14.0121 | 0.7290 | 0.6909 | 0.4346 | 3.3365 |
SAIF [12] | 9.6132 | 13.1604 | 0.7198 | 0.6771 | 0.5445 | 3.2127 |
FusionGAN [35] | 8.0868 | 12.2890 | 0.7386 | 0.1762 | 0.4057 | 3.0279 |
Densefuse [14] | 9.6829 | 12.4549 | 0.6183 | 0.0708 | 0.1048 | 2.3603 |
IFCNN [15] | 10.4489 | 14.6458 | 0.7562 | 0.6482 | 0.4790 | 3.1004 |
DDcGAN [21] | 10.5841 | 11.4985 | 0.7390 | 0.1692 | 0.3079 | 2.8855 |
FusionDN [16] | 10.3115 | 11.0779 | 0.7782 | 0.2351 | 0.4106 | 3.0387 |
MEF-GAN [36] | 10.2738 | 12.8709 | 0.7750 | 0.5986 | 0.4383 | 3.1169 |
PerceptualFusion [22] | 10.6757 | 12.2461 | 0.7912 | 0.2437 | 0.4488 | 3.0562 |
U2Fusion [17] | 9.9803 | 14.9898 | 0.7804 | 0.2786 | 0.3976 | 3.0455 |
CFGAN (Ours) | 10.5600 | 15.2686 | 0.7650 | 0.6931 | 0.5838 | 3.3411 |
Methods | SD | PSNR | CC | SSIM | VIF | MI |
---|---|---|---|---|---|---|
GFF [10] | 10.2951 | 14.8623 | 0.8270 | 0.7053 | 0.4982 | 2.9309 |
CBF [11] | 9.9778 | 14.0533 | 0.7987 | 0.7050 | 0.4317 | 3.2012 |
CNN [13] | 10.1570 | 15.0837 | 0.7961 | 0.7254 | 0.5409 | 3.2684 |
SAIF [12] | 9.9229 | 13.5342 | 0.7783 | 0.7139 | 0.5776 | 3.2500 |
FusionGAN [35] | 9.2530 | 13.5302 | 0.8135 | 0.2090 | 0.4482 | 3.0701 |
Densefuse [14] | 9.4289 | 11.4488 | 0.6030 | 0.0730 | 0.0791 | 2.0421 |
IFCNN [15] | 10.3208 | 15.2448 | 0.8529 | 0.6836 | 0.4638 | 3.0585 |
DDcGAN [21] | 10.2889 | 11.7470 | 0.8019 | 0.1795 | 0.2906 | 2.8856 |
FusionDN [16] | 10.1830 | 11.8313 | 0.8305 | 0.2442 | 0.3776 | 2.9502 |
MEF-GAN [36] | 10.2693 | 13.4217 | 0.8490 | 0.6324 | 0.4237 | 3.0947 |
PerceptualFusion [22] | 10.3464 | 13.3984 | 0.8571 | 0.2551 | 0.4208 | 3.0005 |
U2Fusion [17] | 10.1849 | 15.2428 | 0.8521 | 0.2948 | 0.3827 | 2.9726 |
CFGAN (Ours) | 10.3870 | 15.7633 | 0.8353 | 0.7330 | 0.5850 | 3.3249 |
Methods | SD | PSNR | CC | SSIM | VIF | MI |
---|---|---|---|---|---|---|
Baseline | 9.7235 | 12.0252 | 0.7976 | 0.2417 | 0.5585 | 3.0487 |
Baseline + DFE | 9.7658 | 16.3007 | 0.8066 | 0.3019 | 0.5798 | 3.1881 |
Baseline + CIA | 9.6916 | 15.8922 | 0.8023 | 0.2862 | 0.5828 | 3.1567 |
Baseline + CIA_DFE | 9.8536 | 16.8894 | 0.8136 | 0.5758 | 0.6096 | 3.2192 |
Baseline + DFE_CIA | 9.8524 | 16.9842 | 0.8105 | 0.7281 | 0.6597 | 3.2662 |
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. |
© 2024 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Share and Cite
Mao, Q.; Zhai, W.; Lei, X.; Wang, Z.; Liang, Y. CT and MRI Image Fusion via Coupled Feature-Learning GAN. Electronics 2024, 13, 3491. https://doi.org/10.3390/electronics13173491
Mao Q, Zhai W, Lei X, Wang Z, Liang Y. CT and MRI Image Fusion via Coupled Feature-Learning GAN. Electronics. 2024; 13(17):3491. https://doi.org/10.3390/electronics13173491
Chicago/Turabian StyleMao, Qingyu, Wenzhe Zhai, Xiang Lei, Zenghui Wang, and Yongsheng Liang. 2024. "CT and MRI Image Fusion via Coupled Feature-Learning GAN" Electronics 13, no. 17: 3491. https://doi.org/10.3390/electronics13173491