DKA-RG: Disease-Knowledge-Enhanced Fine-Grained Image–Text Alignment for Automatic Radiology Report Generation
Abstract
:1. Introduction
- We build DKA-RG, a novel framework for automatic radiology report generation. The framework utilizes multi-level contrastive learning, which more comprehensively facilitates cross-modal semantic alignment of radiology images and reports during the encoding process.
- We design a disease decoder network using the medical knowledge graph, which not only takes into account medical disease relationships but also extracts disease information from images and reports at a fine-grained level. We construct disease-level alignment to improve the feature extraction capability of the image encoder.
- We conduct comprehensive qualitative and quantitative experiments on two widely used datasets, MIMIC-CXR [19] and IU-Xray [20]. The experimental results demonstrate that our method achieves high performance in terms of both language generation metrics and medical clinical evaluation metrics, outperforming existing methods. We also construct qualitative experiments to analyze the accuracy of generated reports.
2. Related Work
2.1. Contrastive Learning
2.2. Medical Knowledge Graph
2.3. Radiology Report Generation
3. Method
3.1. Radiology Image and Text Feature Extraction
3.2. Disease Knowledge Graph Embedding
3.3. Image–Report Alignment
3.4. Image to Report Generation
4. Experiments
4.1. Datasets
4.2. Evaluation Metrics
4.3. Implementation Details
4.4. Main Results
4.5. Ablation Study
4.6. Qualitative Experiment
5. Discussion
5.1. Main Novelty
5.2. Limitations
5.3. Practical Application
6. Conclusions and Future Works
6.1. Conclusions
6.2. Future Works
Author Contributions
Funding
Data Availability Statement
Conflicts of Interest
References
- Bayareh-Mancilla, R.; Medina-Ramos, L.A.; Toriz-Vázquez, A.; Hernández-Rodríguez, Y.M.; Cigarroa-Mayorga, O.E. Automated computer-assisted medical decision-making system based on morphological shape and skin thickness analysis for asymmetry detection in mammographic images. Diagnostics 2023, 13, 3440. [Google Scholar] [CrossRef]
- Cui, H.; Hu, L.; Chi, L. Advances in computer-aided medical image processing. Appl. Sci. 2023, 13, 7079. [Google Scholar] [CrossRef]
- Bruno, M.A.; Walker, E.A.; Abujudeh, H.H. Understanding and confronting our mistakes: The epidemiology of error in radiology and strategies for error reduction. Radiographics 2015, 35, 1668–1676. [Google Scholar] [CrossRef]
- Jing, B.; Xie, P.; Xing, E. On the automatic generation of medical imaging reports. arXiv 2017, arXiv:1711.08195. [Google Scholar]
- Zhang, Y.; Wang, X.; Xu, Z.; Yu, Q.; Yuille, A.; Xu, D. When radiology report generation meets knowledge graph. In Proceedings of the AAAI Conference on Artificial Intelligence, New York, NY, USA, 7–12 February 2020; Volume 34, pp. 12910–12917. [Google Scholar]
- Akhter, Y.; Ranjan, R.; Singh, R.; Vatsa, M.; Chaudhury, S. On AI-assisted pneumoconiosis detection from chest x-rays. In Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence, Macao, China, 19–25 August 2023; pp. 6353–6361. [Google Scholar]
- Vinyals, O.; Toshev, A.; Bengio, S.; Erhan, D. Show and tell: A neural image caption generator. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA, 7–12 June 2015; pp. 3156–3164. [Google Scholar]
- Xu, K.; Ba, J.; Kiros, R.; Cho, K.; Courville, A.; Salakhudinov, R.; Zemel, R.; Bengio, Y. Show, attend and tell: Neural image caption generation with visual attention. In Proceedings of the International Conference on Machine Learning, Lille, France, 7–9 July 2015; pp. 2048–2057. [Google Scholar]
- Bai, S.; An, S. A survey on automatic image caption generation. Neurocomputing 2018, 311, 291–304. [Google Scholar] [CrossRef]
- Mokady, R.; Hertz, A.; Bermano, A.H. Clipcap: Clip prefix for image captioning. arXiv 2021, arXiv:2111.09734. [Google Scholar]
- Yan, A.; He, Z.; Lu, X.; Du, J.; Chang, E.; Gentili, A.; McAuley, J.; Hsu, C.N. Weakly supervised contrastive learning for chest X-ray report generation. arXiv 2021, arXiv:2109.12242. [Google Scholar]
- Li, M.; Lin, B.; Chen, Z.; Lin, H.; Liang, X.; Chang, X. Dynamic Graph Enhanced Contrastive Learning for Chest X-ray Report Generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Vancouver, BC, Canada, 17–24 June 2023; pp. 3334–3343. [Google Scholar]
- You, D.; Liu, F.; Ge, S.; Xie, X.; Zhang, J.; Wu, X. Aligntransformer: Hierarchical alignment of visual regions and disease tags for medical report generation. In Proceedings of the Medical Image Computing and Computer Assisted Intervention–MICCAI 2021: 24th International Conference, Strasbourg, France, 27 September–1 October 2021; Proceedings, Part III 24. Springer: Berlin/Heidelberg, Germany, 2021; pp. 72–82. [Google Scholar]
- Tanida, T.; Müller, P.; Kaissis, G.; Rueckert, D. Interactive and Explainable Region-guided Radiology Report Generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Vancouver, BC, Canada, 17–24 June 2023; pp. 7433–7442. [Google Scholar]
- Dosovitskiy, A.; Beyer, L.; Kolesnikov, A.; Weissenborn, D.; Zhai, X.; Unterthiner, T.; Dehghani, M.; Minderer, M.; Heigold, G.; Gelly, S.; et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv 2020, arXiv:2020.11929. [Google Scholar]
- Devlin, J.; Chang, M.W.; Lee, K.; Toutanova, K. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv 2018, arXiv:1810.04805. [Google Scholar]
- Veličković, P.; Cucurull, G.; Casanova, A.; Romero, A.; Lio, P.; Bengio, Y. Graph attention networks. arXiv 2017, arXiv:1710.10903. [Google Scholar]
- Vaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones, L.; Gomez, A.N.; Kaiser, Ł.; Polosukhin, I. Attention is all you need. In Advances in Neural Information Processing Systems, Proceedings of the 31st Conference on Neural Information Processing Systems (NIPS 2017), Long Beach, CA, USA, 4–9 December 2017; MIT Press: Cambridge, MA, USA, 2017; Volume 30. [Google Scholar]
- Johnson, A.E.; Pollard, T.J.; Greenbaum, N.R.; Lungren, M.P.; Deng, C.y.; Peng, Y.; Lu, Z.; Mark, R.G.; Berkowitz, S.J.; Horng, S. MIMIC-CXR-JPG, a large publicly available database of labeled chest radiographs. arXiv 2019, arXiv:1901.07042. [Google Scholar]
- Demner-Fushman, D.; Kohli, M.D.; Rosenman, M.B.; Shooshan, S.E.; Rodriguez, L.; Antani, S.; Thoma, G.R.; McDonald, C.J. Preparing a collection of radiology examinations for distribution and retrieval. J. Am. Med. Inform. Assoc. 2016, 23, 304–310. [Google Scholar] [CrossRef] [PubMed]
- Chen, T.; Kornblith, S.; Norouzi, M.; Hinton, G. A simple framework for contrastive learning of visual representations. In Proceedings of the International Conference on Machine Learning, Virtual, 13–18 July 2020; pp. 1597–1607. [Google Scholar]
- He, K.; Fan, H.; Wu, Y.; Xie, S.; Girshick, R. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA, 13–19 June 2020; pp. 9729–9738. [Google Scholar]
- Zhang, Y.; Jiang, H.; Miura, Y.; Manning, C.D.; Langlotz, C.P. Contrastive learning of medical visual representations from paired images and text. In Proceedings of the Machine Learning for Healthcare Conference, Durham, NC, USA, 5–6 August 2022; pp. 2–25. [Google Scholar]
- Yang, S.; Wu, X.; Ge, S.; Zhou, S.K.; Xiao, L. Knowledge matters: Chest radiology report generation with general and specific knowledge. Med. Image Anal. 2022, 80, 102510. [Google Scholar] [CrossRef]
- Yuan, J.; Liao, H.; Luo, R.; Luo, J. Automatic radiology report generation based on multi-view image fusion and medical concept enrichment. In Proceedings of the Medical Image Computing and Computer Assisted Intervention–MICCAI 2019: 22nd International Conference, Shenzhen, China, 13–17 October 2019; Proceedings, Part VI 22. Springer: Berlin/Heidelberg, Germany, 2019; pp. 721–729. [Google Scholar]
- Yi, X.; Fu, Y.; Liu, R.; Zhang, H.; Hua, R. TSGET: Two-Stage Global Enhanced Transformer for Automatic Radiology Report Generation. IEEE J. Biomed. Health Inform. 2024, 28, 2152–2162. [Google Scholar] [CrossRef]
- Gu, T.; Liu, D.; Li, Z.; Cai, W. Complex organ mask guided radiology report generation. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, Waikoloa, HI, USA, 1–6 January 2024; pp. 7995–8004. [Google Scholar]
- Li, Y.; Liang, X.; Hu, Z.; Xing, E.P. Hybrid retrieval-generation reinforced agent for medical image report generation. In Advances in Neural Information Processing Systems, Proceedings of the 32nd International Conference on Neural Information Processing Systems, Montréal, QC, Canada, 3–8 December 2018; Curran Associates Inc.: Red Hook, NY, USA, 2018; Volume 31. [Google Scholar]
- Nakaura, T.; Yoshida, N.; Kobayashi, N.; Shiraishi, K.; Nagayama, Y.; Uetani, H.; Kidoh, M.; Hokamura, M.; Funama, Y.; Hirai, T. Preliminary assessment of automated radiology report generation with generative pre-trained transformers: Comparing results to radiologist-generated reports. Jpn. J. Radiol. 2024, 42, 190–200. [Google Scholar] [CrossRef]
- Achiam, J.; Adler, S.; Agarwal, S.; Ahmad, L.; Akkaya, I.; Aleman, F.L.; Almeida, D.; Altenschmidt, J.; Altman, S.; Anadkat, S.; et al. Gpt-4 technical report. arXiv 2023, arXiv:2303.08774. [Google Scholar]
- Tang, Y.; Wang, D.; Zhang, L.; Yuan, Y. An efficient but effective writer: Diffusion-based semi-autoregressive transformer for automated radiology report generation. Biomed. Signal Process. Control 2024, 88, 105651. [Google Scholar] [CrossRef]
- Jorg, T.; Halfmann, M.C.; Stoehr, F.; Arnhold, G.; Theobald, A.; Mildenberger, P.; Müller, L. A novel reporting workflow for automated integration of artificial intelligence results into structured radiology reports. Insights Into Imaging 2024, 15, 80. [Google Scholar] [CrossRef] [PubMed]
- Wang, Z.; Liu, L.; Wang, L.; Zhou, L. R2gengpt: Radiology report generation with frozen llms. Meta-Radiology 2023, 1, 100033. [Google Scholar] [CrossRef]
- Ma, X.; Liu, F.; Yin, C.; Wu, X.; Ge, S.; Zou, Y.; Zhang, P.; Sun, X. Contrastive attention for automatic chest X-ray report generation. arXiv 2021, arXiv:2106.06965. [Google Scholar]
- Huang, Z.; Zhang, X.; Zhang, S. KiUT: Knowledge-injected U-Transformer for Radiology Report Generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Vancouver, BC, Canada, 17–24 June 2023; pp. 19809–19818. [Google Scholar]
- Radford, A.; Kim, J.W.; Hallacy, C.; Ramesh, A.; Goh, G.; Agarwal, S.; Sastry, G.; Askell, A.; Mishkin, P.; Clark, J.; et al. Learning transferable visual models from natural language supervision. In Proceedings of the International Conference on Machine Learning, Virtual, 18–24 July 2021; pp. 8748–8763. [Google Scholar]
- Li, J.; Selvaraju, R.; Gotmare, A.; Joty, S.; Xiong, C.; Hoi, S.C.H. Align before fuse: Vision and language representation learning with momentum distillation. In Advances in Neural Information Processing Systems, Proceedings of the 35th Conference on Neural Information Processing Systems (NeurIPS 2021), Online, 6–14 December 2021; Curran Associates, Inc.: Red Hook, NY, USA, 2021; Volume 34, pp. 9694–9705. [Google Scholar]
- Li, J.; Li, D.; Xiong, C.; Hoi, S. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In Proceedings of the International Conference on Machine Learning, Baltimore, MD, USA, 17–23 July 2022; pp. 12888–12900. [Google Scholar]
- Chen, Z.; Song, Y.; Chang, T.H.; Wan, X. Generating radiology reports via memory-driven transformer. arXiv 2020, arXiv:2010.16056. [Google Scholar]
- Papineni, K.; Roukos, S.; Ward, T.; Zhu, W.J. Bleu: A method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, Philadelphia, PA, USA, 6–12 July 2002; pp. 311–318. [Google Scholar]
- Chin-Yew, L. Rouge: A package for automatic evaluation of summaries. In Proceedings of the Workshop on Text Summarization Branches Out, Barcelona, Spain, 25–26 July 2004. [Google Scholar]
- Banerjee, S.; Lavie, A. METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. In Proceedings of the Acl Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, Ann Arbor, MI, USA, 29 June 2005; pp. 65–72. [Google Scholar]
- Vedantam, R.; Lawrence Zitnick, C.; Parikh, D. Cider: Consensus-based image description evaluation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA, 7–12 June 2015; pp. 4566–4575. [Google Scholar]
- Smit, A.; Jain, S.; Rajpurkar, P.; Pareek, A.; Ng, A.Y.; Lungren, M.P. CheXbert: Combining automatic labelers and expert annotations for accurate radiology report labeling using BERT. arXiv 2020, arXiv:2004.09167. [Google Scholar]
- Deng, J.; Dong, W.; Socher, R.; Li, L.J.; Li, K.; Fei-Fei, L. Imagenet: A large-scale hierarchical image database. In Proceedings of the 2009 IEEE Conference on Computer Vision and Pattern Recognition, Miami, FL, USA, 20–25 June 2009; IEEE: Piscataway, NJ, USA, 2009; pp. 248–255. [Google Scholar]
- Liu, L.; Jiang, H.; He, P.; Chen, W.; Liu, X.; Gao, J.; Han, J. On the variance of the adaptive learning rate and beyond. arXiv 2019, arXiv:1908.03265. [Google Scholar]
- Liu, F.; Wu, X.; Ge, S.; Fan, W.; Zou, Y. Exploring and distilling posterior and prior knowledge for radiology report generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Nashville, TN, USA, 20–25 June 2021; pp. 13753–13762. [Google Scholar]
- Chen, Z.; Shen, Y.; Song, Y.; Wan, X. Cross-modal memory networks for radiology report generation. arXiv 2022, arXiv:2204.13258. [Google Scholar]
- Wang, Z.; Tang, M.; Wang, L.; Li, X.; Zhou, L. A medical semantic-assisted transformer for radiographic report generation. In Proceedings of the International Conference on Medical Image Computing and Computer-Assisted Intervention, Vancouver, BC, Canada, 8–12 October 2022; Springer: Berlin/Heidelberg, Germany, 2022; pp. 655–664. [Google Scholar]
Method Type | Representative Methods | Advantage | Disadvantage |
---|---|---|---|
Non Alignment | Jing et al. [4] Wang et al. [33] | Simple end-to-end training | Weak ability to extract disease features |
Global Alignment | Ma et al. [34] Li et al. [12] | Unify multimodal features of image and text to enhance feature extraction capabilities | Lack of fine-grained image text feature alignment |
Knowledge enhancement | Huang et al. [35] Yang et al. [24] | Medical knowledge enhances performance | The way of introducing knowledge is too complicated |
The advantages of our method | Global and local alignment of diseases enhances the ability to extract disease features. The disease knowledge graph serves as a direct guide for contrastive learning, which is simple and efficient. |
Datasets | IU-Xray | MIMIC-CXR | ||||
---|---|---|---|---|---|---|
TRAIN | VALIDATION | TEST | TRAIN | VALIDATION | TEST | |
Num of Images | 5226 | 748 | 1496 | 368,960 | 2991 | 5159 |
Num of Reports | 2770 | 395 | 790 | 222,758 | 1808 | 3269 |
Num of Patients | 2770 | 395 | 790 | 64,586 | 500 | 293 |
AVG. LEN. | 37.56 | 36.78 | 33.62 | 53.00 | 53.05 | 66.40 |
Dataset | Methods | B-1 | B-2 | B-3 | B-4 | ROUGE-L | METEOR | CIDEr |
---|---|---|---|---|---|---|---|---|
IU X-ray | R2Gen [39] | 0.470 | 0.304 | 0.219 | 0.165 | 0.371 | 0.187 | - |
CA [34] | 0.492 | 0.314 | 0.222 | 0.169 | 0.381 | 0.193 | - | |
PPKED [47] | 0.483 | 0.315 | 0.224 | 0.168 | 0.376 | - | 0.351 | |
R2GenCMN [48] | 0.475 | 0.309 | 0.222 | 0.170 | 0.375 | 0.191 | - | |
KnowMat [24] | 0.496 | 0.327 | 0.238 | 0.178 | 0.381 | - | 0.382 | |
MSAT [49] | 0.481 | 0.316 | 0.226 | 0.171 | 0.372 | 0.190 | 0.394 | |
METransformer [33] | 0.483 | 0.322 | 0.228 | 0.172 | 0.380 | 0.192 | 0.435 | |
DCL [12] | - | - | - | 0.163 | 0.383 | 0.193 | 0.586 | |
DKA-RG (Ours) | 0.496 | 0.328 | 0.239 | 0.182 | 0.376 | 0.201 | 0.621 | |
MIMIC -CXR | R2Gen [39] | 0.353 | 0.218 | 0.145 | 0.103 | 0.277 | 0.142 | - |
CA [34] | 0.350 | 0.219 | 0.152 | 0.109 | 0.283 | 0.151 | - | |
PPKED [47] | 0.360 | 0.224 | 0.149 | 0.106 | 0.284 | 0.149 | - | |
R2GenCMN [48] | 0.353 | 0.218 | 0.148 | 0.106 | 0.278 | 0.142 | - | |
KnowMat [24] | 0.363 | 0.228 | 0.156 | 0.115 | 0.284 | - | 0.203 | |
MSAT [49] | 0.373 | 0.235 | 0.162 | 0.120 | 0.282 | 0.143 | 0.299 | |
KiUT [35] | 0.393 | 0.243 | 0.159 | 0.113 | 0.285 | 0.160 | - | |
DCL [12] | - | - | - | 0.109 | 0.284 | 0.150 | 0.281 | |
DKA-RG (Ours) | 0.395 | 0.254 | 0.165 | 0.122 | 0.282 | 0.154 | 0.303 |
Methods | P | R | F1 |
---|---|---|---|
R2Gen | 0.333 | 0.273 | 0.276 |
R2GenCMN | 0.334 | 0.275 | 0.278 |
KnowMat | 0.458 | 0.348 | 0.371 |
KiUT | 0.371 | 0.318 | 0.321 |
DCL | 0.471 | 0.352 | 0.373 |
DKA-RG (ours) | 0.489 | 0.384 | 0.430 |
Settings | Global CL | Disease CL | Knowledge Graph | BLEU-4 | ROUGE-L | METEOR | CIDEr | AVG. |
---|---|---|---|---|---|---|---|---|
Base | 0.166 | 0.365 | 0.194 | 0.454 | - | |||
(a) | ✔ | 0.178 | 0.373 | 0.201 | 0.547 | +10.2% | ||
(b) | ✔ | 0.175 | 0.368 | 0.197 | 0.541 | +8.7% | ||
(c) | ✔ | ✔ | 0.181 | 0.373 | 0.203 | 0.589 | +14.2% | |
(d) | ✔ | ✔ | ✔ | 0.182 | 0.376 | 0.201 | 0.621 | +17.0% |
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. |
© 2024 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Share and Cite
Yin, H.; Wu, W.; Hao, Y. DKA-RG: Disease-Knowledge-Enhanced Fine-Grained Image–Text Alignment for Automatic Radiology Report Generation. Electronics 2024, 13, 3306. https://doi.org/10.3390/electronics13163306
Yin H, Wu W, Hao Y. DKA-RG: Disease-Knowledge-Enhanced Fine-Grained Image–Text Alignment for Automatic Radiology Report Generation. Electronics. 2024; 13(16):3306. https://doi.org/10.3390/electronics13163306
Chicago/Turabian StyleYin, Heng, Wei Wu, and Yongtao Hao. 2024. "DKA-RG: Disease-Knowledge-Enhanced Fine-Grained Image–Text Alignment for Automatic Radiology Report Generation" Electronics 13, no. 16: 3306. https://doi.org/10.3390/electronics13163306