Development and Validation of a Deep Learning System for Diagnosing Glaucoma Using Optical Coherence Tomography
Abstract
:1. Introduction
2. Materials and Methods
2.1. Subjects—Training, Internal Validation, and External Validation Datasets
2.2. Glaucoma Diagnosis by Glaucoma Specialists
2.3. Spectral-Domain Optical Coherence Tomography Examination
2.4. Deep Learning Framework
2.5. Heatmap Analysis
2.6. Statistical Analyses
3. Results
3.1. Diagnostic Ability of Deep Learning Systems Using OCT Maps
3.2. Heatmap Analysis
4. Discussion
Supplementary Materials
Author Contributions
Funding
Conflicts of Interest
References
- McCarthy, J.; Minsky, M.L.; Rochester, N.; Shannon, C.E. A proposal for the dartmouth summer research project on artificial intelligence, 31 August, 1955. AI Mag. 2006, 27, 12–14. [Google Scholar] [CrossRef]
- Ting, D.S.W.; Peng, L.; Varadarajan, A.V.; Keane, P.A.; Burlina, P.M.; Chiang, M.F.; Schmetterer, L.; Pasquale, L.R.; Bressler, N.M.; Webster, D.R.; et al. Deep learning in ophthalmology: The technical and clinical considerations. Prog. Retin. Eye Res. 2019, 72, 100759. [Google Scholar] [CrossRef] [PubMed]
- LeCun, Y.; Bengio, Y.; Hinton, G. Deep learning. Nature 2015, 521, 436–444. [Google Scholar] [CrossRef] [PubMed]
- Zheng, C.; Johnson, T.V.; Garg, A.; Boland, M.V. Artificial intelligence in glaucoma. Curr. Opin. Ophthalmol. 2019, 30, 97–103. [Google Scholar] [CrossRef]
- Leung, C.K.; Cheung, C.Y.; Weinreb, R.N.; Qiu, Q.; Liu, S.; Li, H.; Xu, G.; Fan, N.; Huang, L.; Pang, C.P.; et al. Retinal nerve fiber layer imaging with spectral-domain optical coherence tomography: A variability and diagnostic performance study. Ophthalmology 2009, 116, 1257–1263. [Google Scholar] [CrossRef]
- Leung, C.K.; Lam, S.; Weinreb, R.N.; Liu, S.; Ye, C.; Liu, L.; He, J.; Lai, G.W.; Li, T.; Lam, D.S. Retinal nerve fiber layer imaging with spectral-domain optical coherence tomography: Analysis of the retinal nerve fiber layer map for glaucoma detection. Ophthalmology 2010, 117, 1684–1691. [Google Scholar] [CrossRef] [PubMed]
- Kim, K.E.; Park, K.H. Macular imaging by optical coherence tomography in the diagnosis and management of glaucoma. Br. J. Ophthalmol. 2018, 102, 718–724. [Google Scholar] [CrossRef]
- Hwang, Y.H.; Kim, Y.Y.; Kim, H.K.; Sohn, Y.H. Ability of cirrus high-definition spectral-domain optical coherence tomography clock-hour, deviation, and thickness maps in detecting photographic retinal nerve fiber layer abnormalities. Ophthalmology 2013, 120, 1380–1387. [Google Scholar] [CrossRef]
- Liu, Y.; Simavli, H.; Que, C.J.; Rizzo, J.L.; Tsikata, E.; Maurer, R.; Chen, T.C. Patient characteristics associated with artifacts in Spectralis optical coherence tomography imaging of the retinal nerve fiber layer in glaucoma. Am. J. Ophthalmol. 2015, 159, 565–576. [Google Scholar] [CrossRef] [Green Version]
- Mansberger, S.L.; Menda, S.A.; Fortune, B.A.; Gardiner, S.K.; Demirel, S. Automated segmentation errors when using optical coherence tomography to measure retinal nerve fiber layer thickness in glaucoma. Am. J. Ophthalmol. 2017, 174, 1–8. [Google Scholar] [CrossRef] [Green Version]
- Kim, K.E.; Jeoung, J.W.; Park, K.H.; Kim, D.M.; Kim, S.H. Diagnostic classification of macular ganglion cell and retinal nerve fiber layer analysis: Differentiation of false-positives from glaucoma. Ophthalmology 2015, 122, 502–510. [Google Scholar] [CrossRef] [PubMed]
- Litjens, G.; Kooi, T.; Bejnordi, B.E.; Setio, A.A.A.; Ciompi, F.; Ghafoorian, M.; van der Laak, J.; van Ginneken, B.; Sanchez, C.I. A survey on deep learning in medical image analysis. Med. Image Anal. 2017, 42, 60–88. [Google Scholar] [CrossRef] [PubMed] [Green Version]
- Liu, H.; Li, L.; Wormstone, I.M.; Qiao, C.; Zhang, C.; Liu, P.; Li, S.; Wang, H.; Mou, D.; Pang, R.; et al. Development and validation of a deep learning system to detect glaucomatous optic neuropathy using fundus photographs. JAMA Ophthalmol. 2019, 137, 1353–1360. [Google Scholar] [CrossRef] [PubMed]
- Li, Z.; He, Y.; Keel, S.; Meng, W.; Chang, R.T.; He, M. Efficacy of a deep learning system for detecting glaucomatous optic neuropathy based on color fundus photographs. Ophthalmology 2018, 125, 1199–1206. [Google Scholar] [CrossRef] [PubMed] [Green Version]
- Phan, S.; Satoh, S.; Yoda, Y.; Kashiwagi, K.; Oshika, T.; Japan Ocular Imaging Registry Research Group. Evaluation of deep convolutional neural networks for glaucoma detection. Jpn. J. Ophthalmol. 2019, 63, 276–283. [Google Scholar] [CrossRef] [PubMed]
- Anderson, D.R.; Patella, V.M. Automated Static Perimetry, 2nd ed.; Mosby: St. Louis, MO, USA, 1999. [Google Scholar]
- Hodapp, E.; Parrish, R.K.; Anderson, D.R. Clinical Decisions in Glaucoma; Mosby: St. Louis, MO, USA, 1993. [Google Scholar]
- Simonyan, K.; Zisserman, A. Very deep convolutional networks for lage-scale image recognition. In Proceedings of the ICLR, San Diego, CA, USA, 7–9 May 2015. [Google Scholar]
- Selvaraju, R.R.; Cogswell, M.; Das, A. Grad-CAM: Visual explanations from deep networks via gradient-based localization. In Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy, 22–29 October 2017; pp. 618–626. [Google Scholar]
- DeLong, E.R.; DeLong, D.M.; Clarke-Pearson, D.L. Comparing the areas under two or more correlated receiver operating characteristic curves: A nonparametric approach. Biometrics 1988, 44, 837–845. [Google Scholar] [CrossRef]
- Brusini, P. OCT Glaucoma Staging System: A new method for retinal nerve fiber layer damage classification using spectral-domain OCT. Eye 2018, 32, 113–119. [Google Scholar] [CrossRef] [Green Version]
- Christopher, M.; Belghith, A.; Bowd, C.; Proudfoot, J.A.; Goldbaum, M.H.; Weinreb, R.N.; Girkin, C.A.; Liebmann, J.M.; Zangwill, L.M. Performance of deep learning architectures and transfer learning for detecting glaucomatous optic neuropathy in fundus photographs. Sci. Rep. 2018, 8, 16685. [Google Scholar] [CrossRef] [Green Version]
- Muhammad, H.; Fuchs, T.J.; De Cuir, N.; De Moraes, C.G.; Blumberg, D.M.; Liebmann, J.M.; Ritch, R.; Hood, D.C. Hybrid deep learning on single wide-field optical coherence tomography scans accurately classifies glaucoma suspects. J. Glaucoma 2017, 26, 1086–1094. [Google Scholar] [CrossRef]
- Lee, J.; Kim, Y.K.; Park, K.H.; Jeoung, J.W. Diagnosing glaucoma with spectral-domain optical coherence tomography using deep learning classifier. J. Glaucoma 2020, 29, 287–294. [Google Scholar] [CrossRef]
- Maetschke, S.; Antony, B.; Ishikawa, H.; Wollstein, G.; Schuman, J.; Garnavi, R. A feature agnostic approach for glaucoma detection in OCT volumes. PLoS ONE 2019, 14, e0219126. [Google Scholar] [CrossRef] [PubMed]
- Ran, A.R.; Cheung, C.Y.; Wang, X.; Chen, H.; Luo, L.; Chan, P.; Wong, M.; Chang, R.; Mannil, S.; Young, A.; et al. Detection of glaucomatous optic neuropathy with spectral-domain optical coherence tomography: A retrospective training and validation deep-learning analysis. Lancet Digit. Health 2019, 1, e172–e182. [Google Scholar] [CrossRef] [Green Version]
- Thompson, A.C.; Jammal, A.A.; Berchuck, S.I.; Mariottoni, E.B.; Medeiros, F.A. Assessment of a segmentation-free deep learning algorithm for diagnosing glaucoma from optical coherence tomography scans. JAMA Ophthalmol. 2020, 138, 333–339. [Google Scholar] [CrossRef] [PubMed]
- Asaoka, R.; Murata, H.; Hirasawa, K.; Fujino, Y.; Matsuura, M.; Miki, A.; Kanamoto, T.; Ikeda, Y.; Mori, K.; Iwase, A.; et al. Using deep learning and transfer learning to accurately diagnose early-onset glaucoma from macular optical coherence tomography images. Am. J. Ophthalmol. 2019, 198, 136–145. [Google Scholar] [CrossRef] [PubMed]
- Oddone, F.; Lucenteforte, E.; Michelessi, M.; Rizzo, S.; Donati, S.; Parravano, M.; Virgili, G. Macular versus retinal nerve fiber layer parameters for diagnosing manifest glaucoma: A systematic review of diagnostic accuracy studies. Ophthalmology 2016, 123, 939–949. [Google Scholar] [CrossRef]
- Jeoung, J.W.; Choi, Y.J.; Park, K.H.; Kim, D.M. Macular ganglion cell imaging study: Glaucoma diagnostic accuracy of spectral-domain optical coherence tomography. Investig. Ophthalmol. Vis. Sci. 2013, 54, 4422–4429. [Google Scholar] [CrossRef] [Green Version]
- Nouri-Mahdavi, K.; Nowroozizadeh, S.; Nassiri, N.; Cirineo, N.; Knipping, S.; Giaconi, J.; Caprioli, J. Macular ganglion cell/inner plexiform layer measurements by spectral domain optical coherence tomography for detection of early glaucoma and comparison to retinal nerve fiber layer measurements. Am. J. Ophthalmol. 2013, 156, 1297–1307.e2. [Google Scholar] [CrossRef] [Green Version]
- Leung, C.K.; Mohamed, S.; Leung, K.S.; Cheung, C.Y.; Chan, S.L.; Cheng, D.K.; Lee, A.K.; Leung, G.Y.; Rao, S.K.; Lam, D.S. Retinal nerve fiber layer measurements in myopia: An optical coherence tomography study. Investig. Ophthalmol. Vis. Sci. 2006, 47, 5171–5176. [Google Scholar] [CrossRef] [Green Version]
- Kang, S.H.; Hong, S.W.; Im, S.K.; Lee, S.H.; Ahn, M.D. Effect of myopia on the thickness of the retinal nerve fiber layer measured by Cirrus HD optical coherence tomography. Investig. Ophthalmol. Vis. Sci. 2010, 51, 4075–4083. [Google Scholar] [CrossRef] [Green Version]
- Leite, M.T.; Zangwill, L.M.; Weinreb, R.N.; Rao, H.L.; Alencar, L.M.; Sample, P.A.; Medeiros, F.A. Effect of disease severity on the performance of Cirrus spectral-domain OCT for glaucoma diagnosis. Investig. Ophthalmol. Vis. Sci. 2010, 51, 4104–4109. [Google Scholar] [CrossRef]
- Hood, D.C.; Raza, A.S.; de Moraes, C.G.; Liebmann, J.M.; Ritch, R. Glaucomatous damage of the macula. Prog. Retin. Eye Res. 2013, 32, 1–21. [Google Scholar] [CrossRef] [PubMed] [Green Version]
Training Dataset | Internal Validation Dataset | External Validation Dataset | ||||
---|---|---|---|---|---|---|
Control (n = 332) | Glaucoma (n = 1490) | Control (n = 104) | Glaucoma (n = 321) | Control (n = 108) | Glaucoma (n = 247) | |
Age (years) | 54.0 ± 14.5 | 59.1 ± 13.8 | 55.9 ± 13.1 | 58.7 ± 13.7 | 53.4 ± 15.5 | 58.4 ± 14.5 |
Male (n) | 121 (49.4%) | 569 (58.9%) | 42 (42.9%) | 177 (59.2%) | 64 (59.3%) | 151 (61.1%) |
Axial length (mm) | 24.2 ± 1.1 | 25.1 ± 1.6 | 24.3 ± 1.5 | 25.0 ± 1.7 | 24.2 ± 1.3 | 24.7 ± 24.4 |
Average RNFL thickness (µm) | 94.7 ± 8.9 | 71.6 ± 11.9 | 91.1 ± 9.5 | 69.7 ± 11.2 | 93.7 ± 7.2 | 73.5 ± 52.3 |
Average GCIPL thickness (µm) | 82.1 ± 6.7 | 67.6 ± 10.5 | 80.3 ± 9.2 | 66.0 ± 9.2 | 82.3 ± 4.3 | 67.7 ± 9.1 |
HVF MD (dB) | −0.9 ± 2.6 | −7.2 ± 7.6 | −0.9 ± 2.4 | −7.9 ± 7.0 | −0.9 ± 1.2 | −7.8 ± 7.2 |
Early Glaucoma (MD > −6 dB) n = 162 (A) | Moderate Glaucoma (−6 dB ≥ MD > −12 dB) n = 79 (B) | Severe Glaucoma (MD ≥ −12 dB) n = 80 (C) | p | Post Hoc Analysis | |
---|---|---|---|---|---|
Age (years) | 57.1 ± 13.5 | 58.8 ± 14.9 | 61.7 ± 12.6 | 0.074 | |
Male (n) | 88 (56.4%) | 41 (59.4%) | 48 (64.9%) | 0.475 | |
Axial length (mm) | 25.2 ± 1.7 | 25.2 ± 1.6 | 25.1 ± 1.4 | 0.869 | |
Average RNFL thickness (µm) | 74.7 ± 10.4 | 67.9 ± 9.2 | 61.1 ± 8.3 | <0.001 | A > B > C |
Average GCIPL thickness (µm) | 69.3 ± 8.8 | 64.8 ± 7.9 | 60.1 ± 7.9 | <0.001 | A > B > C |
HVF MD (dB) | −2.5 ± 1.9 | −8.9 ± 1.9 | −17.9 ± 4.9 | <0.001 | A > B > C |
AUROC (95% Confidence Interval) | Sensitivity at 90% Specificity (%) | Sensitivity at 80% Specificity (%) | |
---|---|---|---|
RNFL analysis | |||
Thickness map | 0.987 (0.971–0.995) | 97.8 | 98.2 |
Deviation map | 0.974 (0.954–0.987) | 93.2 | 97.2 |
Thickness map and axial length | 0.975 (0.956–0.988) | 93.5 | 95.3 |
GCIPL analysis | |||
Thickness map | 0.966 (0.943–0.981) | 92.5 | 94.6 |
Deviation map | 0.903 (0.871–0.929) | 86.6 | 93.1 |
Thickness map and axial length | 0.950 (0.925–0.969) | 88.8 | 93.7 |
Combination set | |||
RNFL deviation and GCIPL deviation map | 0.979 (0.961–0.991) | 94.1 | 97.2 |
RNFL deviation and GCIPL thickness map | 0.963 (0.941–0.979) | 91.6 | 96.2 |
RNFL thickness and GCIPL deviation map | 0.952 (0.927–0.970 | 94.4 | 95.9 |
RNFL thickness and GCIPL thickness map | 0.964 (0.942–0.980) | 96.4 | 97.5 |
All 4 maps (RNFL/GCIPL thickness and deviation maps) | 0.977 (0.958–0.989) | 93.5 | 96.6 |
All 4 maps with axial length | 0.961 (0.938–0.977) | 92.8 | 94.0 |
AUROC (95% Confidence Interval) | |||
---|---|---|---|
Early Glaucoma (n = 162) | Moderate Glaucoma (n = 79) | Severe Glaucoma (n = 80) | |
RNFL analysis | |||
Thickness map | 0.974 (0.948–0.990) | 0.999 (0.979–1.000) | 0.999 (0.980–1.000) |
Deviation map | 0.956 (0.924–0.977) | 0.993 (0.967–1.000) | 0.993 (0.967–1.000) |
Thickness map and axial length | 0.951 (0.918–0.974) | 0.999 (0.980–1.000) | 0.999 (0.979–1.000) |
GCIPL analysis | |||
Thickness map | 0.940 (0.905–0.965) | 0.991 (0.964–0.999) | 0.992 (0.965–0.999) |
Deviation map | 0.879 (0.834–0.916) | 0.919 (0.869–0.954) | 0.935 (0.889–0.966) |
Thickness map and axial length | 0.916 (0.876–0.947) | 0.981 (0.950–0.996) | 0.988 (0.960–0.998) |
Combination set | |||
RNFL deviation and GCIPL deviation map | 0.965 (0.936–0.984) | 0.988 (0.959–0.998) | 0.999 (0.979–1.000) |
RNFL deviation and GCIPL thickness map | 0.947 (0.912–0.970) | 0.980 (0.947–0.995) | 0.981 (0.949–0.995) |
RNFL thickness and GCIPL deviation map | 0.939 (0.903–0.965) | 0.965 (0.927–0.987) | 0.965 (0.927–0.986) |
RNFL thickness and GCIPL thickness map | 0.952 (0.919–0.975) | 0.976 (0.942–0.993) | 0.976 (0.942–0.993) |
All 4 maps (RNFL/GCIPL thickness and deviation maps) | 0.955 (0.923–0.977) | 0.999 (0.980–1.000) | 0.999 (0.979–1.000) |
All 4 maps with axial length | 0.932 (0.895–0.959) | 0.990 (0.962–0.999) | 0.990 (0.963–0.999) |
© 2020 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).
Share and Cite
Kim, K.E.; Kim, J.M.; Song, J.E.; Kee, C.; Han, J.C.; Hyun, S.H. Development and Validation of a Deep Learning System for Diagnosing Glaucoma Using Optical Coherence Tomography. J. Clin. Med. 2020, 9, 2167. https://doi.org/10.3390/jcm9072167
Kim KE, Kim JM, Song JE, Kee C, Han JC, Hyun SH. Development and Validation of a Deep Learning System for Diagnosing Glaucoma Using Optical Coherence Tomography. Journal of Clinical Medicine. 2020; 9(7):2167. https://doi.org/10.3390/jcm9072167
Chicago/Turabian StyleKim, Ko Eun, Joon Mo Kim, Ji Eun Song, Changwon Kee, Jong Chul Han, and Seung Hyup Hyun. 2020. "Development and Validation of a Deep Learning System for Diagnosing Glaucoma Using Optical Coherence Tomography" Journal of Clinical Medicine 9, no. 7: 2167. https://doi.org/10.3390/jcm9072167