ACSwinNet: A Deep Learning-Based Rigid Registration Method for Head-Neck CT-CBCT Images in Image-Guided Radiotherapy
Abstract
:1. Introduction
2. Materials and Methods
2.1. Swin Transformer-Based Registration Network
2.2. Perceptual Similarity Metric Network
2.3. Anatomical Constraint Encoder Network
3. Experiments
3.1. Datasets and Pre-Processing
3.2. Baseline Methods
3.3. Measurement
3.4. Implementation Details
4. Results
4.1. Registration Performance Comparison
4.2. Contribution of Anatomically Constrained Network
4.3. Contribution of Perceptual Similarity Metric Network
5. Discussion
6. Conclusions
Author Contributions
Funding
Institutional Review Board Statement
Informed Consent Statement
Data Availability Statement
Conflicts of Interest
References
- Bang, C.; Bernard, G.; Le, W.T.; Lalonde, A.; Kadoury, S.; Bahig, H. Artificial Intelligence to Predict Outcomes of Head and Neck Radiotherapy. Clin. Transl. Radiat. Oncol. 2023, 39, 100590. [Google Scholar] [CrossRef]
- Hwang, C. Impact of Dose Distribution on Rotational Setup Errors in Radiotherapy for Prostate Cancer. Med. Dosim. 2023, 48, 181–186. [Google Scholar] [CrossRef] [PubMed]
- de Crevoisier, R.; Lafond, C.; Mervoyer, A.; Hulot, C.; Jaksic, N.; Bessières, I.; Delpon, G. Image-Guided Radiotherapy. Cancer/Radiothérapie 2022, 26, 34–49. [Google Scholar] [CrossRef] [PubMed]
- Veiga, C.; McClelland, J.; Moinuddin, S.; Lourenço, A.; Ricketts, K.; Annkah, J.; Modat, M.; Ourselin, S.; D’Souza, D.; Royle, G. Toward Adaptive Radiotherapy for Head and Neck Patients: Feasibility Study on Using CT-to-CBCT Deformable Registration for “Dose of the Day” Calculations. Med. Phys. 2014, 41, 031703. [Google Scholar] [CrossRef] [PubMed]
- Johari, Z.I.; Azmi, N.A.; Ab Muin, N.F.; Latiff, R.A.; Fadzil, M.S.A.; Razali, R.A. Two-Dimensional and Three-Dimensional Image-Guided Evaluation of Patient Positioning for Intensity-Modulated Radiation Therapy of Head and Neck Cancer. Malays. J. Fundam. Appl. Sci. 2023, 19, 915–926. [Google Scholar] [CrossRef]
- Andronache, A.; von Siebenthal, M.; Székely, G.; Cattin, P. Non-Rigid Registration of Multi-Modal Images Using Both Mutual Information and Cross-Correlation. Med. Image Anal. 2008, 12, 3–15. [Google Scholar] [CrossRef]
- Viola, P.; Wells III, W.M. Alignment by Maximization of Mutual Information. Int. J. Comput. Vis. 1997, 24, 137–154. [Google Scholar] [CrossRef]
- Knops, Z.F.; Maintz, J.B.A.; Viergever, M.A.; Pluim, J.P.W. Normalized Mutual Information Based Registration Using K-Means Clustering and Shading Correction. Med. Image Anal. 2006, 10, 432–439. [Google Scholar] [CrossRef]
- Ilunga-Mbuyamba, E.; Avina-Cervantes, J.G.; Lindner, D.; Cruz-Aceves, I.; Arlt, F.; Chalopin, C. Vascular Structure Identification in Intraoperative 3D Contrast-Enhanced Ultrasound Data. Sensors 2016, 16, 497. [Google Scholar] [CrossRef]
- Duan, L.; Ni, X.; Liu, Q.; Gong, L.; Yuan, G.; Li, M.; Yang, X.; Fu, T.; Zheng, J. Unsupervised Learning for Deformable Registration of Thoracic CT and Cone-Beam CT Based on Multiscale Features Matching with Spatially Adaptive Weighting. Med. Phys. 2020, 47, 5632–5647. [Google Scholar] [CrossRef]
- Suwanraksa, C.; Bridhikitti, J.; Liamsuwan, T.; Chaichulee, S. CBCT-to-CT Translation Using Registration-Based Generative Adversarial Networks in Patients with Head and Neck Cancer. Cancers 2023, 15, 2017. [Google Scholar] [CrossRef]
- Teuwen, J.; Gouw, Z.A.R.; Sonke, J.-J. Artificial Intelligence for Image Registration in Radiation Oncology. Semin. Radiat. Oncol. 2022, 32, 330–342. [Google Scholar] [CrossRef] [PubMed]
- Zhu, X.; Ding, M.; Huang, T.; Jin, X.; Zhang, X. PCANet-Based Structural Representation for Nonrigid Multimodal Medical Image Registration. Sensors 2018, 18, 1477. [Google Scholar] [CrossRef]
- Chen, J.; Liu, Y.; Wei, S.; Bian, Z.; Subramanian, S.; Carass, A.; Prince, J.L.; Du, Y. A Survey on Deep Learning in Medical Image Registration: New Technologies, Uncertainty, Evaluation Metrics, and Beyond. arXiv 2023, arXiv:2307.15615. [Google Scholar]
- Fu, Y.; Lei, Y.; Wang, T.; Curran, W.J.; Liu, T.; Yang, X. Deep Learning in Medical Image Registration: A Review. Phys. Med. Biol. 2020, 65, 20TR01. [Google Scholar] [CrossRef]
- Miao, S.; Wang, Z.J.; Zheng, Y.; Liao, R. Real-Time 2D/3D Registration via CNN Regression. In Proceedings of the 2016 IEEE 13th International Symposium on Biomedical Imaging, Prague, Czech Republic, 13–16 April 2016. [Google Scholar]
- Guo, H.; Kruger, M.; Xu, S.; Wood, B.J.; Yan, P. Deep Adaptive Registration of Multi-Modal Prostate Images. Comput. Med. Imaging Graph. 2020, 84, 101769. [Google Scholar] [CrossRef]
- Salehi, S.S.M.; Khan, S.; Erdogmus, D.; Gholipour, A. Real-Time Deep Pose Estimation with Geodesic Loss for Image-to-Template Rigid Registration. IEEE Trans. Med. Imaging 2018, 38, 470–481. [Google Scholar] [CrossRef]
- Deng, X.; Liu, E.; Li, S.; Duan, Y.; Xu, M. Interpretable Multi-Modal Image Registration Network Based on Disentangled Convolutional Sparse Coding. IEEE Trans. Image Process. 2023, 32, 1078–1091. [Google Scholar] [CrossRef] [PubMed]
- Abbasi, S.; Tavakoli, M.; Boveiri, H.R.; Mosleh Shirazi, M.A.; Khayami, R.; Khorasani, H.; Javidan, R.; Mehdizadeh, A. Medical Image Registration Using Unsupervised Deep Neural Network: A Scoping Literature Review. Biomed. Signal Process. Control 2022, 73, 103444. [Google Scholar] [CrossRef]
- Chen, X.; Zhou, B.; Xie, H.; Guo, X.; Zhang, J.; Duncan, J.S.; Miller, E.J.; Sinusas, A.J.; Onofrey, J.A.; Liu, C. DuSFE: Dual-Channel Squeeze-Fusion-Excitation Co-Attention for Cross-Modality Registration of Cardiac SPECT and CT. Med. Image Anal. 2023, 88, 102840. [Google Scholar] [CrossRef]
- Song, X.; Chao, H.; Xu, X.; Guo, H.; Xu, S.; Turkbey, B.; Wood, B.J.; Sanford, T.; Wang, G.; Yan, P. Cross-Modal Attention for Multi-Modal Image Registration. Med. Image Anal. 2022, 82, 102612. [Google Scholar] [CrossRef]
- Che, T.; Wang, X.; Zhao, K.; Zhao, Y.; Zeng, D.; Li, Q.; Zheng, Y.; Yang, N.; Wang, J.; Li, S. AMNet: Adaptive Multi-Level Network for Deformable Registration of 3D Brain MR Images. Med. Image Anal. 2023, 85, 102740. [Google Scholar] [CrossRef]
- Li, M.; Hu, S.; Li, G.; Zhang, F.; Li, J.; Yang, Y.; Zhang, L.; Liu, M.; Xu, Y.; Fu, D.; et al. The Successive Next Network as Augmented Regularization for Deformable Brain MR Image Registration. Sensors 2023, 23, 3208. [Google Scholar] [CrossRef]
- Liu, R.; Lehman, J.; Molino, P.; Petroski Such, F.; Frank, E.; Sergeev, A.; Yosinski, J. An Intriguing Failing of Convolutional Neural Networks and the CoordConv Solution. In Proceedings of the Advances in Neural Information Processing Systems; Bengio, S., Wallach, H., Larochelle, H., Grauman, K., Cesa-Bianchi, N., Garnett, R., Eds.; Curran Associates, Inc.: Red Hook, NY, USA, 2018; Volume 31. [Google Scholar]
- Thisanke, H.; Deshan, C.; Chamith, K.; Seneviratne, S.; Vidanaarachchi, R.; Herath, D. Semantic Segmentation Using Vision Transformers: A Survey. Eng. Appl. Artif. Intell. 2023, 126, 106669. [Google Scholar] [CrossRef]
- Dosovitskiy, A.; Beyer, L.; Kolesnikov, A.; Weissenborn, D.; Zhai, X.; Unterthiner, T.; Dehghani, M.; Minderer, M.; Heigold, G.; Gelly, S.; et al. An Image Is Worth 16x16 Words: Transformers for Image Recognition at Scale. arXiv 2020, arXiv:2010.11929. [Google Scholar]
- Chen, J.; He, Y.; Frey, E.C.; Li, Y.; Du, Y. ViT-V-Net: Vision Transformer for Unsupervised Volumetric Medical Image Registration. arXiv 2021, arXiv:2104.06468. [Google Scholar]
- Mok, T.C.W.; Chung, A.C.S. Affine Medical Image Registration with Coarse-to-Fine Vision Transformer. arXiv 2022, arXiv:2203.15216. [Google Scholar]
- Chen, Z.; Zheng, Y.; Gee, J.C. TransMatch: A Transformer-Based Multilevel Dual-Stream Feature Matching Network for Unsupervised Deformable Image Registration. IEEE Trans. Med. Imaging 2024, 43, 15–27. [Google Scholar] [CrossRef]
- Drake, R.; Vogl, A.W.; Mitchell, A.W.M. Gray’s Anatomy for Students E-Book; Elsevier Health Sciences: Amsterdam, The Netherlands, 2009; ISBN 1437720552. [Google Scholar]
- Bentzen, S.M.; Constine, L.S.; Deasy, J.O.; Eisbruch, A.; Jackson, A.; Marks, L.B.; Ten Haken, R.K.; Yorke, E.D. Quantitative Analyses of Normal Tissue Effects in the Clinic (QUANTEC): An Introduction to the Scientific Issues. Int. J. Radiat. Oncol. 2010, 76, S3–S9. [Google Scholar] [CrossRef]
- Balakrishnan, G.; Zhao, A.; Sabuncu, M.; Guttag, J.; Dalca, A. V VoxelMorph: A Learning Framework for Deformable Medical Image Registration. IEEE TMI Trans. Med. Imaging 2019, 38, 1788–1800. [Google Scholar] [CrossRef] [PubMed]
- Mansilla, L.; Milone, D.H.; Ferrante, E. Learning Deformable Registration of Medical Images with Anatomical Constraints. Neural Netw. 2020, 124, 269–279. [Google Scholar] [CrossRef] [PubMed]
- Liu, Z.; Lin, Y.; Cao, Y.; Hu, H.; Wei, Y.; Zhang, Z.; Lin, S.; Guo, B. Swin Transformer: Hierarchical Vision Transformer Using Shifted Windows. In Proceedings of the 2021 IEEE/CVF International Conference on Computer Vision (ICCV), Montreal, BC, Canada, 11–17 October 2021. [Google Scholar]
- Vaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones, L.; Gomez, A.N.; Kaiser, Ł.; Polosukhin, I. Attention Is All You Need. In Advances in Neural Information Processing Systems; NIPS: Long Beach, CA, USA, 2017; pp. 5999–6009. [Google Scholar]
- Johnson, J.; Alahi, A.; Fei-Fei, L. Perceptual Losses for Real-Time Style Transfer and Super-Resolution. In Lecture Notes in Computer Science. (Including Subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Springer: Cham, Switzerland, 2016; pp. 694–711. [Google Scholar] [CrossRef]
- Hémon, C.; Texier, B.; Chourak, H.; Simon, A.; Bessières, I.; de Crevoisier, R.; Castelli, J.; Lafond, C.; Barateau, A.; Nunes, J.-C. Indirect Deformable Image Registration Using Synthetic Image Generated by Unsupervised Deep Learning. Image Vis. Comput. 2024, 148, 105143. [Google Scholar] [CrossRef]
- An, T.; Mao, B.; Xue, B.; Huo, C.; Xiang, S.; Pan, C. Patch Loss: A Generic Multi-Scale Perceptual Loss for Single Image Super-Resolution. Pattern Recognit. 2023, 139, 109510. [Google Scholar] [CrossRef]
- Xia, Y.; Xiong, Y.; Wang, K. A Transformer Model Blended with CNN and Denoising Autoencoder for Inter-Patient ECG Arrhythmia Classification. Biomed. Signal Process. Control 2023, 86, 105271. [Google Scholar] [CrossRef]
- Huazhong University of Science and Technology. Head Neck CBCT CT datasets [Data set]. Zenodo 2024. [Google Scholar] [CrossRef]
- Avants, B.; Tustison, N.; Johnson, H. Advanced Normalization Tools (ANTS). Insight J. 2009, 1–35. [Google Scholar]
- Zhao, S.; Lau, T.; Luo, J.; Chang, E.I.C.; Xu, Y. Unsupervised 3D End-to-End Medical Image Registration with Volume Tweening Network. IEEE J. Biomed. Health Inform. 2020, 24, 1394–1404. [Google Scholar] [CrossRef]
- de Vos, B.D.; Berendsen, F.F.; Viergever, M.A.; Sokooti, H.; Staring, M.; Išgum, I. A Deep Learning Framework for Unsupervised Affine and Deformable Image Registration. Med. Image Anal. 2019, 52, 128–143. [Google Scholar] [CrossRef]
Method | ROI | Modality | Architecture | Dimension | Transformation |
---|---|---|---|---|---|
C2FViT [29] | Brain | MR | Transformer | 3D | Affine |
ViT-V-Net [28] | Brain | MR | Transformer + CNN | 3D | Deformable |
VoxelMorph [33] | Brain | MR | CNN | 3D | Deformable |
AC-RegNet [34] | Lung | X-Ray | CNN | 2D | Deformable |
Attention-Reg [22] | Prostate | MR-TRUS | CNN | 3D | Rigid/Deformable |
InMIR-Net [19] | Brain/Nature | MR T1-T2 | CNN | 2D | Rigid/Deformable |
MSReg [17] | Prostate | MR-TRUS | CNN | 3D | Rigid |
3DPose-Net [18] | Brain | MR | CNN | 3D | Rigid |
ACSwinNet (Ours) | Head&Neck | CT-CBCT | Swin Transformer + CNN | 3D | Rigid |
DSC ↑ | SSIM ↑ | TRE_H (mm) ↓ | TRE_N (mm) ↓ | Time (s) ↓ | |
---|---|---|---|---|---|
Initial | 0.390 (0.084) | 0.651 (0.105) | 8.74 (3.49) | 10.28 (4.23) | / |
ANTs | 0.743 (0.051) | 0.854 (0.044) | 2.14 (0.45) | 2.33 (0.51) | 7.31 |
ANTs-Local | 0.728 (0.062) | 0.822 (0.058) | 1.76 (0.36) | 3.01 (0.83) | 6.47 |
AC-ConvNet | 0.734 (0.069) | 0.830 (0.062) | 2.07 (0.51) | 2.67 (0.74) | 0.12 |
AC-VTN | 0.737 (0.058) | 0.847 (0.057) | 2.04 (0.50) | 2.65 (0.68) | 0.12 |
Attention-Reg | 0.746 (0.055) | 0.859 (0.056) | 1.95 (0.44) | 2.58 (0.62) | 0.16 |
ACSwinNet | 0.755 (0.053) | 0.870 (0.043) | 1.82 (0.39) | 2.50 (0.60) | 0.27 |
DSC ↑ | SSIM ↑ | TRE_H (mm) ↓ | TRE_N (mm)↓ | Time (s) ↓ | |
---|---|---|---|---|---|
ANTs | 0.743 (0.051) | 0.854 (0.044) | 2.14 (0.45) | 2.33 (0.51) | 7.31 |
ANTs-Local | 0.728 (0.062) | 0.822 (0.058) | 1.76 (0.36) | 3.01 (0.83) | 6.47 |
ACSwinNet-1 | 0.732 (0.060) | 0.829 (0.065) | 2.26 (0.61) | 2.49 (0.72) | 0.27 |
ACSwinNet-2 | 0.759 (0.049) | 0.879 (0.041) | 2.08 (0.43) | 2.21 (0.52) | 0.27 |
ACSwinNet | 0.755 (0.053) | 0.870 (0.043) | 1.82 (0.39) | 2.50 (0.60) | 0.27 |
DSC ↑ | SSIM ↑ | TRE_H (mm) ↓ | TRE_N (mm) ↓ | Time (s) ↓ | |
ACSwinNet-1 | 0.732 (0.060) | 0.829 (0.065) | 2.26 (0.61) | 2.49 (0.72) | 0.27 |
ACSwinNet (Ldice) | 0.745 (0.059) | 0.853 (0.050) | 1.97 (0.51) | 2.53 (0.65) | 0.27 |
ACSwinNet (Lae) | 0.751 (0.054) | 0.861 (0.044) | 1.87 (0.45) | 2.54 (0.62) | 0.27 |
ACSwinNet (Ldice+ae) | 0.755 (0.053) | 0.870 (0.043) | 1.82 (0.39) | 2.50 (0.60) | 0.27 |
DSC ↑ | SSIM ↑ | TRE_H (mm) ↓ | TRE_N (mm) ↓ | Time (s) ↓ | |
---|---|---|---|---|---|
ACSwinNet-LSSIM | 0.739 (0.058) | 0.860 (0.040) | 2.05 (0.48) | 2.55 (0.67) | 0.27 |
ACSwinNet-LNCC | 0.736 (0.059) | 0.841 (0.056) | 2.09 (0.53) | 2.63 (0.72) | 0.27 |
ACSwinNet-LMI | 0.739 (0.055) | 0.849 (0.042) | 2.03 (0.50) | 2.60 (0.64) | 0.27 |
ACSwinNet-Lperceptual | 0.755 (0.053) | 0.870 (0.043) | 1.82 (0.39) | 2.50 (0.60) | 0.27 |
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. |
© 2024 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Share and Cite
Peng, K.; Zhou, D.; Sun, K.; Wang, J.; Deng, J.; Gong, S. ACSwinNet: A Deep Learning-Based Rigid Registration Method for Head-Neck CT-CBCT Images in Image-Guided Radiotherapy. Sensors 2024, 24, 5447. https://doi.org/10.3390/s24165447
Peng K, Zhou D, Sun K, Wang J, Deng J, Gong S. ACSwinNet: A Deep Learning-Based Rigid Registration Method for Head-Neck CT-CBCT Images in Image-Guided Radiotherapy. Sensors. 2024; 24(16):5447. https://doi.org/10.3390/s24165447
Chicago/Turabian StylePeng, Kuankuan, Danyu Zhou, Kaiwen Sun, Junfeng Wang, Jianchun Deng, and Shihua Gong. 2024. "ACSwinNet: A Deep Learning-Based Rigid Registration Method for Head-Neck CT-CBCT Images in Image-Guided Radiotherapy" Sensors 24, no. 16: 5447. https://doi.org/10.3390/s24165447