Next Article in Journal
Histologic Characterization of Tumor-Adjacent Mammary Adipose Tissue in Normal-Weight and Overweight/Obese Patients with Triple-Negative Breast Cancer
Previous Article in Journal
Evolving Practice and Outcomes in Grade 2 Glioma: Real-World Data from a Multi-Institutional Registry
Previous Article in Special Issue
SkinLesNet: Classification of Skin Lesions and Detection of Melanoma Cancer Using a Novel Multi-Layer Deep Convolutional Neural Network
 
 
Search for Articles:
Title / Keyword
Author / Affiliation / Email
Journal
Article Type
 
 
Advanced Search
 
Section
Special Issue
Volume
Issue
Number
Page
 
Journals
Cancers
Volume 16
Issue 20
10.3390/cancers16203516
cancers-logo
Submit to this Journal Review for this Journal Propose a Special Issue
► Article Menu

Article Menu

share Share announcement Help format_quote Cite question_answer Discuss in SciProfiles thumb_up
...
Endorse textsms
...
Comment
first_page
settings
Order Article Reprints
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

Artificial Intelligence and Machine Learning in Ocular Oncology, Retinoblastoma (ArMOR): Experience with a Multiracial Cohort

by
Vijitha S. Vempuluru
1,†,
Rajiv Viriyala
1,†,
Virinchi Ayyagari
1,
Komal Bakal
1,
Patanjali Bhamidipati
2,
Krishna Kishore Dhara
3,
Sandor R. Ferenczy
4,
Carol L. Shields
4 and
Swathi Kaliki
1,*
1
The Operation Eyesight Universal Institute for Eye Cancer, LV Prasad Eye Institute, Hyderabad 500034, India
2
The International Institute of Information Technology, Hyderabad 500032, India
3
The Center for Innovation, Bourntec, Madhapur, Hyderabad 500081, India
4
Ocular Oncology Service, Wills Eye Hospital, Thomas Jefferson University, 840 Walnut Street, 14th Floor, Philadelphia, PA 19107, USA
*
Author to whom correspondence should be addressed.
†
These authors contributed equally to this work.
Cancers 2024, 16(20), 3516; https://doi.org/10.3390/cancers16203516
Submission received: 31 July 2024 / Revised: 4 October 2024 / Accepted: 8 October 2024 / Published: 17 October 2024
(This article belongs to the Collection Artificial Intelligence and Machine Learning in Cancer Research)
Browse Figures
Review Reports Versions Notes

Abstract

:

Simple Summary

In the last decade, artificial intelligence and machine learning (AI/ML) have been increasingly explored in the field of intraocular tumors. Retinoblastoma (RB) is the most common eye cancer in childhood. Early detection is crucial for optimizing outcomes in RB. Hence, we aimed to employ AI/ML to develop a potential screening tool for RB and established the feasibility of training an AI model to detect and classify RB from fundus images in an Asian Indian cohort previously. Taking this work ahead, we explored the model’s ability to detect and classify RB in a multiracial cohort. Despite unequal frequency distribution between the races, we identified the scope for improvement and retrained the AI model to detect and classify RB. The AI model displayed an accuracy of 97% for detecting RB and 98%, 93%, >99%, 94%, and 93% for grouping tumors into the International Classification of Retinoblastoma groups A to E, respectively.

Abstract

Background: The color variation in fundus images from differences in melanin concentrations across races can affect the accuracy of artificial intelligence and machine learning (AI/ML) models. Hence, we studied the performance of our AI model (with proven efficacy in an Asian-Indian cohort) in a multiracial cohort for detecting and classifying intraocular RB (iRB). Methods: Retrospective observational study. Results: Of 210 eyes, 153 (73%) belonged to White, 37 (18%) to African American, 9 (4%) to Asian, 6 (3%) to Hispanic races, based on the U.S. Office of Management and Budget’s Statistical Policy Directive No.15 and 5 (2%) had no reported race. Of the 2473 images in 210 eyes, 427 had no tumor, and 2046 had iRB. After training the AI model based on race, the sensitivity and specificity for detection of RB in 2473 images were 93% and 96%, respectively. The sensitivity and specificity of the AI model were 74% and 100% for group A; 88% and 96% for group B; 88% and 100% for group C; 73% and 98% for group D, and 100% and 92% for group E, respectively. Conclusions: The AI models built on a single race do not work well for other races. When retrained for different races, our model exhibited high sensitivity and specificity in detecting RB and classifying RB.

1. Introduction

Retinoblastoma (RB) is a relatively new medical frontier that artificial intelligence (AI) has touched. The earliest literature evidence of artificial intelligence in the field of RB was in 2017 by Ciller C et al., who demonstrated the use of AI for the longitudinal tracking of RB tumors [1]. While this opened doors to several possibilities for AI’s application in RB, the primary focus has been detecting RB rather than follow-up [2,3,4,5,6,7,8,9,10,11,12]. This is probably the right direction since early detection of RB is a bigger problem worldwide than monitoring response to therapy, especially in middle and low-income countries [13,14]. Only a few groups, including ours, have trained and tested the AI and machine learning (ML) models for detecting RB from RetCamTM/fundus images of patients from their respective centers [6,10,11,12]. In contrast to other studies that focused on the detection of RB in any given fundus image [1,10], we have looked into the accuracy of detection of smaller tumors, i.e., tumors classified as group A and B based on the International Classification of Retinoblastoma (ICRB) [15]. We noted that the sensitivity for detecting group A tumors ranged from 56% to 100%, and group B tumors ranged from 82% to 95% in eyes with RB, identifying a definite scope for improvement [11]. The accuracy of the AI model can be improved by retraining and testing the AI model with larger volumes of high-quality data that exhibit adequate diversity and granularity [16]. With this goal, we undertook the present study in a large and diverse multiracial image database, and herein we report the results.

2. Methods

2.1. Data Collection

This was a retrospective study that adhered to the tenets of the Declaration of Helsinki and was compliant with the U.S. Health Insurance Portability and Accountability Act norms. The Institutional Review Boards approved the study at LV Prasad Eye Institute, Hyderabad, India, and the Wills Eye Hospital, Philadelphia, U.S.A. Fundus images at presentation were retrieved for patients with RB availing the Ocular Oncology Service, Wills Eye Hospital, Philadelphia, USA, between July 2022 and June 2023. The only data obtained besides the images was the patient’s race since we anticipated a change in the existing AI model’s ability to detect RB due to heterogeneity in the fundus pigmentation in different races [17,18,19,20,21]. The information about the patient’s race was retrieved from the medical records as reported by the parent or caregiver. Per institute protocols at the Ocular Oncology Service, Wills Eye Hospital, Philadelphia, U.S.A, race was recorded as White, Hispanic, African American, Asian Indian, Asian Oriental, or Others.
As of March 2024, the Statistical Policy Directive no. 15 on standards for maintaining, collecting, and presenting federal data on race and ethnicity combined race and ethnicity into one single question and listed the following categories: American Indian or Alaska Native, Asian, Black or African American, Hispanic or Latino, Middle Eastern or North African, Native Hawaiian or Pacific Islander, and White [22,23]. We largely adhered to this format. Although “Hispanic” is considered more appropriate as an ethnonym than a race, this cohort was deemed distinct to the study since their fundus pigmentation is known to vary from the White population [22,23,24,25]. Hispanics included individuals of Mexican, Puerto Rican, Salvadoran, Cuban, Dominican, Guatemalan, and other Central or South American or Spanish culture or origin [22]. We included consecutive patients of all races, and heterogeneity was expected in the number of patients of different races, given the predominance of White patients seeking services at Wills Eye Hospital, Philadelphia, U.S.A. No identifiable information was retrieved, and the Health Insurance Portability and Accountability Act was adhered to. Poor-quality images were excluded from the database. Individual images and the respective eyes were assigned an ICRB group label (A to E) [16], or as ‘normal’ by two independent ocular oncologists (VSV, KB). The degree of interobserver variability was assessed for images and eyes.

2.2. Feature Extraction

Our previous work elaborates on the details of the methods used for feature extraction [11,12]. Broadly, a combination of the MultiLabel Classification Model of Deep Learning, Mobile-Net v2 SSD-based deep learning model for transfer learning, Computer Vision techniques, and geometric calculations was used. Application of our pre-existing algorithm with multiple AI/ML and vision models that were trained on fundus images of Asian Indian eyes [12] performed poorly for a multiracial cohort.

2.3. Identification of Specific Problems and Development of Solutions

Figure 1 illustrates the methodology employed. The overview’s upper section (Figure 1, above the dotted line) describes steps in extracting various features of the images using deep learning (AI) and computer vision models. The overview’s lower section (Figure 1, below the dotted line) contains an AI/ML classifier model for the final diagnosis. As anticipated, we noted that the problems with the AI model’s interpretation were primarily due to differences in fundus color due to variable melanin based on race, contrasting blood vessels and hemorrhage with respect to the fundus, and subtle feature interactions for the classification of RB. Detailed analysis pointed towards the need for changes in the AI model to detect optic discs, tumors, blood vessels, and hemorrhage (Figure 1, marked in red).
With these changes, while the accuracy detection of RB was acceptable, we learned that further retraining was needed to improve the accuracy of the classification of RB. We further elaborate on the changes made to the previously validated model here:
  • Fundus color: This significantly impacted the AI/ML models for detecting optic discs and tumors across various races. While the larger tumors did not pose much of a problem, the smaller ones, especially groups A and B, often blended within the fundus color (Figure 2). This problem was mitigated by retraining, testing, and validating against the multiracial data, thus ensuring no data imbalance across different categories.
  • Fundus contrast: The fundus images for Asian-Indian and African American races had a clear contrast between blood vessels, hemorrhages, and the background fundus color. The fundus images did not have such a contrast for the White race. Our default Computer Vision feature extraction model from the pre-existing algorithm proved too broad, resulting in incorrect grouping to C, D, and E groups. Sometimes, the AI model assigned group D or E labels to group A tumors. To bring about a more versatile broader algorithm applicable to all races, we appended a pre-processing step to perform color segmentation and a set of enhancements. The color segmentation and contrast enhancement approaches were used only to extract features such as blood vessels and hemorrhages. We retrained the models for other features to capture other variabilities, as a standard practice described in the literature [26]. This pre-processing step transformed the cohort with multiracial fundus images into a common framework so that the blood vessel and hemorrhage detection computer models could be uniformly applied. Figure 3 illustrates the three-step process we employed: pre-processing, identification of blood vessels/hemorrhage, and computing features for classification, such as the area covered and adjacent retina visible or not visible.
  • Retraining of the AI model: Due to the differences in fundus color and contrast, the AI model showed a propensity towards assigning inaccurate C, D, and E group labels in the multiracial dataset. To delineate and enable the model to learn this subtle difference from the data set, we re-trained the AI model using a new ML model on the features extracted from various new/modified models, as described in Figure 1 and Figure 2. A total of 14 extracted and derived features were used to build an XGBoost ML model for grouping RB.
  • Assignment of a group label to an eye: Since different images of the same eye would show variation in the presence or absence of the tumor, its size, and associated features, variation was expected for group labels assigned to the images. Several factors were carefully considered when assigning a group label to the eye based on group labels of individual images. After grouping every image of an eye to arrive at a patient-level diagnosis, we followed a two-step process, which we termed aggregation. The first step in the aggregation was to remove outliers. We noted that within the large dataset, there were subtle inadequacies, such as a lack of focus only on a small set of images of an eye. Instead of invalidating such eyes and limiting the analysis, the outlier step in our aggregation algorithm removed these if (i) there were ≥5 such images in an eye and (ii) there was only one image with a group assigned ≥2 groups away from the rest of the images of that eye. The second step in assigning a group label to the eye was to look for the most frequent group label assigned within the images of that eye that is greater than a threshold of 65% or defaults to a conservative stance of the highest group among the images if there is no clear majority.

2.4. Performance Metrics and Statistical Analysis

The overall accuracy of the AI model, as well as misclassification, under-classification, and over-classification rates, were calculated. The sensitivity, specificity, positive, and negative predictive values were calculated for the entire cohort based on the ICRB group and race.

3. Results

3.1. Dataset, Groups, and Distribution

The study included 2473 images from 210 eyes. The eyes were classified as ICRB group A (n = 140; 6%), group B (n = 876; 35%), group C (n = 92; 4%), group D (n = 556; 22%), group E (n = 382; 15%), and normal (n = 427; 17%) by VSV and KB. The interobserver variability was <1% for images and <1% for eyes. The distribution of images and eyes of the dataset between different ICRB groups and races is listed in Table 1. A majority of the eyes belonged to group B and the White race. The distribution of images used for training, validation, and testing are tabulated in Table 2.

3.2. Performance Metrics

The sensitivity, specificity, positive predictive value, and negative predictive value for detection of RB in 2473 images were 93%, 96%, 99%, and 74%, respectively, and in 210 eyes were 98%, 96%, 99%, and 92%, respectively. The sensitivity, specificity, positive predictive value, and negative predictive value of the AI model were 74%, 100%, 100%, and 97% for group A; 88%, 96%, 91%, and 94% for group B; 88%, 100%, 100%, and >99% for group C; 73%, 98%, 90%, and 94% for group D, and 100%, 92%, 67%, and 100% for group E, respectively. The performance metrics for images are tabulated alongside the metrics for eyes in Table 3.
The accuracy for detecting RB was 94% for images. The under-classification rate (including 142 images that were misclassified as ‘normal’) for 2473 images was 11% (n = 219), and the over-classification rate was 5% (n = 96). Regarding the assignment of ICRB group labels, 84% were correctly assigned overall, 90% within one ICRB group difference, and 98% within two ICRB group differences. These accuracies improved at an eye level, which was expected. The accuracy for detecting RB was 97% for eyes. The under-classification rate (including 4 eyes that were misclassified as ‘normal’) for 210 eyes was 3% (n = 5), and the over-classification rate was 12% (n = 20). With respect to the ICRB group labels, 87% were assigned the correct group overall, 94% were within one ICRB group difference, and 98% were within two ICRB group differences. The comparison of the ICRB group assigned by the AI model versus that assigned by the ocular oncologists is depicted as matrices for the images and eyes in Figure 4.

3.3. Analysis of the Sensitivity for Detecting RB by the AI Model Based on the Race and ICRB Group

Table 4 lists the AI model’s sensitivities within each racial cohort, further subclassified by the ICRB group. Overall, the sensitivity was highest at 100% for the Asian Indian and Asian Oriental cohorts, 86% for the White, and 89% for the African American cohorts.
Overall, the AI model detected and classified RB in diverse races and all ICRB groups (Figure 5).

4. Discussion

Large-volume and high-quality data are essential for effectively training AI models [16]. This is also expected to improve over time as datasets enlarge and become more diverse. An excellent example is the improvement in sensitivity and specificity of the AI models used to detect ROP [27,28]. Early studies dating back to 2002 with fundus images, 11 ROP and 9 non-ROP participants showed 82% and 75% sensitivity and specificity, respectively [29]. However, with a dataset of 227,326 retinal images, in 2023, the sensitivity and specificity reached 91% and 91%, respectively [30]. This trend will likely continue for RB if the AI model is trained with expanded datasets.
The major leap in this work from our previous reports is the improvement in the detection of ICRB group A tumors from 56% to 74%. While other studies report an accuracy of 82% to 100% based on various models, the data on the AI model’s efficacy for tumors of different ICRB groups was limited. In a study by Zhang et al., with a reported overall accuracy of 99% for detecting RB, the sensitivity was 66% for eyes with group A tumors [10].
The second important aspect of this work is understanding race’s impact on the AI model’s efficacy in detecting RB. For an AI model to function as an efficient RB screening tool and be utilizable globally, it should provide reliable results irrespective of racial and ethnic differences from good-quality fundus images. The background color in fundus images can vary significantly based on the amount of melanin present in the retinal pigment epithelium and choroid [17,18,19,20,21]. Even the density of macular pigment and vessel architecture varies between cohorts of different ethnicities [18,19]. The color of a target object and its background can significantly affect the accuracy of AI models, and this has been shown with the detection of objects such as medicine pills [31] and in real-time fundus images [32]. Thus, the background fundus color is a crucial factor that needs to be considered while employing AI-based tools for early detection of RB in a community setting.
Coyner et al. described the ability of AI models to “learn” the ethnicity from fundus images, which leads to a bias in the interpretation of images of newborns screened for retinopathy of prematurity [32]. The findings in our study were in contrast to the observations by Coyner et al. [32], wherein the AI model trained in Asian Indian eyes did not perform as predicted with eyes of White origin before retraining. Nevertheless, in both scenarios, the AI model’s output was affected by race, and to improve accuracy, the AI models need to be trained from diverse cohorts. Some groups have attempted to use AI to predict the degree of fundus pigmentation and guide additional training on the AI model [17]. Since this was not the primary problem we aimed to solve with AI, we employed an alternate approach that seemed to overcome this drawback and improve the AI model’s accuracy.
Thus, in this study, we saw that the accuracy of the AI model varied between different races, which warranted retraining. Promising results were obtained after retraining the AI model for all races. Another option to improve the accuracy is to develop other models for various features and a separate AI model for each race. We deliberately avoided this for the following reasons.
  • We developed a generic model that can be easily deployed across a clinical practice with multiracial data, avoiding the need for multiple models. A single algorithm with one set of models will be versatile in deployment, retraining, and testing.
  • Further, prior definitive information on the race may only sometimes be available to apply an appropriate model based on that information. Instead, selecting a common model for multiracial data agnostic to the race is more versatile. We measured the performance against various races and verified that our proposed approach worked, proving that the learning of different AI/ML models is uniform.
This study’s diverse database enabled our AI model to improve its accuracy in detecting smaller tumors. It brought to light an important aspect that affects the accuracy of the AI model in detecting and grouping RB, which is the race of affected children with RB. Nevertheless, further training with eyes of different origins is needed before it can be translated into a universally deployable RB screening technology.
Being one of the initial studies in this arena, this study has limitations that could not be eliminated. The primary goal was to improve the AI model’s accuracy for small tumors by training it on a large and diverse cohort. Firstly, the distribution of images from different races was not uniform. Because the study was based on data from a single Ocular Oncology Center in the United States, most of the patients were White [23]. Unequal distribution in the different cohorts may have affected the performance of the AI model for each race. Thus, this AI model‘s generalizability beyond Asian Indians and Whites is still limited. On the same lines, whether a screening program based on this model needs to be race-specific or universal cannot be answered yet. Multicentric data-sharing studies are warranted to bridge these lacunae, especially for underrepresented groups and groups not represented in this study. However, challenges to overcome on this front would be acquiring uniformly good quality. The second limitation of this study was that despite acquiring a large cohort of images, the numbers in ICRB Groups A and B were smaller. To employ this technology as a screening tool needs improvement and further training on these sub-cohorts. Lastly, a few features may have been missed by the combination of approaches we used to train the model, affecting the model’s accuracy, and this can be improved through further studies over time.

5. Conclusions

This is a first-of-its-kind study to explore the ability of an AI model to detect and classify RB in a multiracial cohort. As we advance, multicentric global collaborations and data sharing can enhance the AI model’s accuracy for detecting RB in different global cohorts and smaller tumors.

Author Contributions

Conceptualization, V.S.V., S.K. and K.K.D.; Methodology, V.S.V., K.B., R.V., V.A., P.B. and S.R.F.; Software and validation, R.V., V.A., P.B. and K.K.D.; Resources, S.R.F. and C.L.S.; Data curation, S.K.; Writing—Original Draft Preparation, V.S.V. and R.V.; Writing—Review and editing, K.K.D., S.R.F., C.L.S. and S.K.; Visualization, V.S.V. and R.V.; Supervision and project administration, S.K. and K.K.D. All authors have read and agreed to the published version of the manuscript.

Funding

Support was provided by The Operation Eyesight Universal Institute for Eye Cancer (V.S.V., S.K.) and Hyderabad Eye Research Foundation (V.S.V., S.K.), Hyderabad, India. Support was provided in part by the Jerry A Shields MD Eye Cancer Fund, Philadelphia, PA (C.L.S.), and the Eye Tumor Research Foundation, Philadelphia, PA (C.L.S.). The funders had no role in the design and conduct of the study, in the collection, analysis, and interpretation of the data, or in the manuscript’s preparation, review, or approval.

Institutional Review Board Statement

The study was approved by the Institutional Review Boards at the Wills Eye Hospital, Philadelphia (IRB#2024-59), U.S.A., and the LV Prasad Eye Institute, Hyderabad, India (LEC-BHR-01-20-390).

Informed Consent Statement

Patient consent was waived due to use of deidentified data for the study.

Data Availability Statement

Data is available upon request.

Conflicts of Interest

Krishna Kishore Dhara was employed by Bourntec. The remaining authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

References

  1. Ciller, C.; Ivo, S.; De Zanet, S.; Apostolopoulos, S.; Munier, F.L.; Wolf, S.; Thiran, J.P.; Cuadra, M.B.; Sznitman, R. Automatic segmentation of retinoblastoma in fundus image photography using convolutional neural networks. Investig. Ophthalmol. Vis. Sci. 2017, 58, 3332. [Google Scholar]
  2. Andayani, U.; Siregar, B.; Sandri, W.E.; Muchtar, M.; Syahputra, M.; Fahmi, F.; Nasution, T. Identification of retinoblastoma using backpropagation neural network. J. Phys. Conf. Ser. 2019, 1235, 012093. [Google Scholar] [CrossRef]
  3. Jaya, I.; Andayani, U.; Siregar, B.; Febri, T.; Arisandi, D. Identification of retinoblastoma using the extreme learning machine. J. Phys. Conf. Ser. 2019, 1235, 12057. [Google Scholar] [CrossRef]
  4. Deva Durai, C.A.; Jebaseeli, T.J.; Alelyani, S.; Mubharakali, A. Early prediction and diagnosis of retinoblastoma using deep learning techniques. arXiv 2021, arXiv:2103.07622. [Google Scholar]
  5. Strijbis, V.I.J.; de Bloeme, C.M.; Jansen, R.W.; Kebiri, H.; Nguyen, H.G.; de Jong, M.C.; Moll, A.C.; Bach-Cuadra, M.; de Graaf, P.; Steenwijk, M.D. Multi-view convolutional neural networks for automated ocular structure and tumor segmentation in retinoblastoma. Sci. Rep. 2021, 11, 14590. [Google Scholar] [CrossRef]
  6. Rahdar, A.; Ahmadi, M.J.; Naseripour, M.; Akhtari, A.; Sedaghat, A.; Hosseinabadi, V.Z.; Yarmohamadi, P.; Hajihasani, S.; Mirshahi, R. Semi-supervised segmentation of retinoblastoma tumors in fundus images. Sci. Rep. 2013, 13, 13010. [Google Scholar] [CrossRef]
  7. Jebaseeli, T.; David, D. Diagnosis of ophthalmic retinoblastoma tumors using 2.75 D CNN segmentation technique. In Computational Methods and Deep Learning for Ophthalmology; Academic Press: Cambridge, MA, USA, 2023; pp. 107–119. [Google Scholar]
  8. Alruwais, N.; Obayya, M.; Al-Mutiri, F.; Assiri, M.; Alneil, A.A.; Mohamed, A. Advancing retinoblastoma detection based on binary arithmetic optimization and integrated features. PeerJ Comput. Sci. 2023, 22, e1681. [Google Scholar] [CrossRef]
  9. Aldughayfiq, B.; Ashfaq, F.; Jhanjhi, N.Z.; Humayun, M. Explainable AI for retinoblastoma diagnosis: Interpreting deep learning models with LIME and SHAP. Diagnostics 2023, 13, 1932. [Google Scholar] [CrossRef]
  10. Zhang, R.; Dong, L.; Li, R.; Zhang, K.; Li, Y.; Zhao, H.; Shi, J.; Ge, X.; Xu, X.; Jiang, L.; et al. Automatic retinoblastoma screening and surveillance using deep learning. Br. J. Cancer 2023, 129, 466–474. [Google Scholar] [CrossRef]
  11. Kaliki, S.; Vempuluru, V.S.; Ghose, N.; Patil, G.; Viriyala, R.; Dhara, K.K. Artificial intelligence and machine learning in ocular oncology: Retinoblastoma. Indian J. Ophthalmol. 2023, 71, 424–430. [Google Scholar] [CrossRef]
  12. Habeb, A.; Zhu, N.; Taresh, M.; Ali, A. Deep ocular tumor classification model using cuckoo search algorithm and Caputo fractional gradient descent. PeerJ Comput. Sci. 2024, 10, e1923. [Google Scholar] [CrossRef]
  13. Fabian, I.D.; Abdallah, E.; Abdullahi, S.U.; Abdulqader, R.A.; Adamou Boubacar, S.; Ademola-Popoola, D.S.; Adio, A.; Afshar, A.R.; Aggarwal, P.; Aghaji, A.E.; et al. Global retinoblastoma presentation and analysis by national income level. JAMA Oncol. 2020, 6, 685–695. [Google Scholar]
  14. Kaliki, S.; Ji, X.; Zou, Y.; Sultana, S.; Taju Sherief, S.; Cassoux, N.; Diaz Coronado, R.Y.; Luis Garcia Leon, J.; López, A.M.Z.; Polyakov, V.G.; et al. Lag time between onset of first symptom and treatment of retinoblastoma: An international collaborative study of 692 patients from 10 countries. Cancers 2021, 13, 1956. [Google Scholar] [CrossRef]
  15. Shields, C.L.; Mashayekhi, A.; Au, A.K.; Czyz, C.; Leahey, A.; Meadows, A.T.; Shields, J.A. The International Classification of Retinoblastoma predicts chemoreduction success. Ophthalmology 2006, 113, 2276–2280. [Google Scholar] [CrossRef]
  16. Luca, A.R.; Ursuleanu, T.F.; Gheorghe, L.; Grigorovici, R.; Iancu, S.; Hlusneac, M.; Grigorovici, A. Impact of quality, type, and volume of data used by deep learning models in the analysis of medical images. Inform. Med. Unlocked 2022, 29, 100911. [Google Scholar] [CrossRef]
  17. Rajesh, A.E.; Olvera-Barrios, A.; Warwick, A.N.; Wu, Y.; Stuart, K.V.; Biradar, M.; Ung, C.Y.; Khawaja, A.P.; Luben, R.; Foster, P.J.; et al. EPIC Norfolk, UK Biobank Eye and Vision Consortium. Ethnicity is not biology: Retinal pigment score to evaluate biological variability from ophthalmic imaging using machine learning. medRxiv 2023. [Google Scholar] [CrossRef]
  18. Wolf-Schnurrbusch, U.E.K.; Röösli, N.; Weyermann, E.; Heldner, M.R.; Höhne, K.; Wolf, S. Ethnic differences in macular pigment density and distribution. Investig. Ophthalmol. Vis. Sci. 2007, 48, 3783–3787. [Google Scholar] [CrossRef]
  19. Li, X.; Wong, W.L.; Cheung, C.Y.; Cheng, C.; Ikram, M.K.; Li, J.; Chia, K.S.; Wong, T.Y. Racial differences in retinal vessel geometric characteristics: A multiethnic study in healthy Asians. Investig. Ophthalmol. Vis. Sci. 2013, 5, 3650–3656. [Google Scholar] [CrossRef]
  20. Seely, K.R.; McCall, M.; Ying, G.S.; Prakalapakorn, S.G.; Freedman, S.F.; Toth, C.A.; BabySTEPS Group. Ocular pigmentation impact on retinal versus choroidal optical coherence tomography imaging in preterm infants. Transl. Vis. Sci. Technol. 2023, 12, 7. [Google Scholar] [CrossRef]
  21. Davey, P.G.; Lievens, C.; Ammono-Monney, S. Differences in macular pigment optical density across four ethnicities: A comparative study. Ther. Adv. Ophthalmol. 2020, 12, 2515841420924167. [Google Scholar] [CrossRef]
  22. U.S. Office of Management and Budget’s Statistical Policy Directive No. 15: Standards for Maintaining, Collecting, and Presenting Federal Data on Race and Ethnicity. Available online: https://spd15revision.gov/ (accessed on 24 September 2024).
  23. Vargas, E.A.; Scherer, L.A.; Fiske, S.T.; Barabino, G.A.; National Academies of Sciences, Engineering, and Medicine (Eds.) Population Data and Demographics in the United States. In Advancing Antiracism, Diversity, Equity, and Inclusion in STEMM Organizations: Beyond Broadening Participation; National Academies Press (US): Washington, DC, USA, 2023. Available online: https://www.ncbi.nlm.nih.gov/books/NBK593023/ (accessed on 24 September 2024).
  24. Lopez, M.H.; Krogstad, J.M.; Passel, J.S. “Who Is Hispanic?”. Pew Research Center. Available online: https://www.pewresearch.org/fact-tank/2020/09/15/who-is-hispanic/ (accessed on 24 September 2024).
  25. U.S. Department of Transportation. ‘Hispanic Americans’, Which Includes Persons of Mexican, Puerto Rican, Cuban, Dominican, Central or South American, or Other Spanish or Portuguese Culture or Origin, Regardless of Race. Available online: https://www.fhwa.dot.gov/hep/guidance/superseded/49cfr26.cfm (accessed on 24 September 2024).
  26. Gaudio, A.; Smailagic, A.; Campilho, A. Enhancement of retinal fundus images via pixel color amplification. In Image Analysis and Recognition; Karray, F., Campilho, A., Wang, Z., Eds.; ICIAR 2020. Lecture Notes in Computer Science; Springer: Cham, Switzerland, 2020; Volume 11663. [Google Scholar]
  27. Jafarizadeh, A.; Maleki, S.F.; Pouya, P.; Sobhi, N.; Abdollahi, M.; Pedrammehr, S.; Lim, C.P.; Asadi, H.; Alizadehsani, R.; Tan, R.S.; et al. Current and future roles of artificial intelligence in retinopathy of prematurity. arXiv 2024, arXiv:2402.09975. [Google Scholar] [CrossRef]
  28. Ramanathan, A.; Athikarisamy, S.E.; Lam, G.C. Artificial intelligence for the diagnosis of retinopathy of prematurity: A systematic review of current algorithms. Eye 2023, 37, 2518–2526. [Google Scholar] [CrossRef]
  29. Heneghan, C.; Flynn, J.; O’Keefe, M.; Cahill, M. Characterization of changes in blood vessel width and tortuosity in retinopathy of prematurity using image analysis. Med. Image Anal. 2002, 6, 407–429. [Google Scholar] [CrossRef]
  30. Rao, D.P.; Savoy, F.M.; Tan, J.Z.E.; Fung, B.P.; Bopitiya, C.M.; Sivaraman, A.; Vinekar, A. Development and validation of an artificial intelligence based screening tool for detection of retinopathy of prematurity in a South Indian population. Front. Pediatr. 2023, 11, 1197237. [Google Scholar] [CrossRef] [PubMed]
  31. Cha, K.; Woo, H.K.; Park, D.; Chang, D.K.; Kang, M. Effects of background colors, flashes, and exposure values on the accuracy of a smartphone-based pill recognition system using a deep convolutional neural network: Deep learning and experimental approach. JMIR Med. Inform. 2021, 9, e26000. [Google Scholar] [CrossRef]
  32. Coyner, A.S.; Singh, P.; Brown, J.M.; Ostmo, S.; Chan, R.V.P.; Chiang, M.F.; Kalpathy-Cramer, J.; Campbell, J.P. Imaging and informatics in retinopathy of prematurity consortium. Association of biomarker-based artificial intelligence with risk of racial bias in retinal Images. JAMA Ophthalmol. 2023, 141, 543–552. [Google Scholar] [CrossRef]
Cancers 16 03516 g001
Figure 1. Overview of the methodology employed.
Figure 1. Overview of the methodology employed.
Cancers 16 03516 g001
Cancers 16 03516 g002
Figure 2. Fundus photograph with subtle retinoblastoma (arrow).
Figure 2. Fundus photograph with subtle retinoblastoma (arrow).
Cancers 16 03516 g002
Cancers 16 03516 g003
Figure 3. The three-step process employed in image processing. Illustration of the original image (A) that was pre-processed (B) to ensure uniformity in the identification of blood vessels/hemorrhages across races (C). Further, features such as the area covered by the tumor (D) were computed for grouping.
Figure 3. The three-step process employed in image processing. Illustration of the original image (A) that was pre-processed (B) to ensure uniformity in the identification of blood vessels/hemorrhages across races (C). Further, features such as the area covered by the tumor (D) were computed for grouping.
Cancers 16 03516 g003
Cancers 16 03516 g004
Figure 4. Performance metrics of the AI model for detection and classification of RB depicted as matrices for 2473 images and 210 eyes.
Figure 4. Performance metrics of the AI model for detection and classification of RB depicted as matrices for 2473 images and 210 eyes.
Cancers 16 03516 g004
Cancers 16 03516 g005
Figure 5. The AI model. The AI model detects optic disc (green bounding box with % confidence) and retinoblastoma (blue bounding box with % confidence) in ICRB groups A (A–C), B (D–F), C (G–I), D (J–L), and E (M–O) in White, African American, Hispanic, and Asian-Indian races.
Figure 5. The AI model. The AI model detects optic disc (green bounding box with % confidence) and retinoblastoma (blue bounding box with % confidence) in ICRB groups A (A–C), B (D–F), C (G–I), D (J–L), and E (M–O) in White, African American, Hispanic, and Asian-Indian races.
Cancers 16 03516 g005
Table 1. Artificial intelligence and machine learning in retinoblastoma: Distribution of images and eyes used for training, validation, and test.
Table 1. Artificial intelligence and machine learning in retinoblastoma: Distribution of images and eyes used for training, validation, and test.
Learning ModelNumber of Images Labelled for Training + Validation + Test of the AI Model (n)Number of Images Used for Training (n)Number of Images Used for Cross-Validation (n)Number of Images Used for Test (n)
Deep learning (MultiLabel Classification: optic disc + tumor)784501126157
Deep Learning (seeds)132842127
Final AI model (XGBoost model)24731582396495
Table 2. Artificial intelligence and machine learning in retinoblastoma: Distribution of 2473 images of 210 eyes by ICRB group and race.
Table 2. Artificial intelligence and machine learning in retinoblastoma: Distribution of 2473 images of 210 eyes by ICRB group and race.
No. of ImagesNo. of Eyes
RaceRace
ICRB GroupWAAHAIAONAAll RacesWAAHAIAONAAll Races
Normal32872135634273110211146
A120171002140162000119
B705125221509876579120372
C70601600925102008
D394118281330556258211037
E2768413090382197101028
All groups1893422774918142473153376635210
W = White; AA = African American; H = Hispanic; AI = Asian Indian; AO = Asian Oriental; NA = not available.
Table 3. Artificial intelligence and machine learning in retinoblastoma: Overall performance metrics of the final AI model in 2473 images of 210 eyes.
Table 3. Artificial intelligence and machine learning in retinoblastoma: Overall performance metrics of the final AI model in 2473 images of 210 eyes.
Detection of RBNormalABCDE
N = 2473n = 210N = 427n = 46N = 140n = 19N = 876n = 72N = 92n = 8N = 556n = 37N = 382n = 28
Ac94%97%94%97%98%98%90%93%99%>99%92%94%95%93%
Sn93%98%96%96%71%74%84%88%70%88%80%73%85%100%
Sp96%96%93%98%>99%100%93%96%100%100%96%98%97%92%
PPV99%99%74%92%95%100%87%91%100%100%85%90%85%67%
NPV74%92%99%99%98%97%91%94%99%>99%94%94%97%100%
N = no. of images; n = no. of eyes; Ac = accuracy, Sn = sensitivity; Sp = specificity; PPV = positive predictive value; NPV = negative predictive value.
Table 4. Artificial intelligence and machine learning in retinoblastoma: Sensitivity of the AI model based on International Classification of Retinoblastoma (ICRB) group and race in 210 eyes.
Table 4. Artificial intelligence and machine learning in retinoblastoma: Sensitivity of the AI model based on International Classification of Retinoblastoma (ICRB) group and race in 210 eyes.
Race
Normal
(n)		A
(n)		B
(n)		C
(n)		D
(n)		E
(n)		Total
(n)
White
(n = 153)		94% (31)75% (16)88% (57)80% (5)68% (25)100% (19)86% (153)
African American
(n = 37)		100% (10)50% (2)89% (9)100% (1)75% (8)100% (7)89% (37)
Asian Indian
(n = 6)		100% (1)na100% (2)100% (2)100% (1)na100% (6)
Asian Oriental
(n = 3)		100% (1)nanana100% (1)100% (1)100% (3)
Hispanic (n = 6)100% (2)na100% (1)na100% (2)100% (1)100% (6)
Not available
(n = 5)		100% (1)100% (1)67% (3)naNana80% (5)
Total (n = 210)96% (46)74% (19)88% (72)88% (8)73% (37)100% (28)87% (210)
na = sensitivity could not be computed since there were no eyes in that sub-cohort.
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.

Share and Cite

MDPI and ACS Style

Vempuluru, V.S.; Viriyala, R.; Ayyagari, V.; Bakal, K.; Bhamidipati, P.; Dhara, K.K.; Ferenczy, S.R.; Shields, C.L.; Kaliki, S. Artificial Intelligence and Machine Learning in Ocular Oncology, Retinoblastoma (ArMOR): Experience with a Multiracial Cohort. Cancers 2024, 16, 3516. https://doi.org/10.3390/cancers16203516

AMA Style

Vempuluru VS, Viriyala R, Ayyagari V, Bakal K, Bhamidipati P, Dhara KK, Ferenczy SR, Shields CL, Kaliki S. Artificial Intelligence and Machine Learning in Ocular Oncology, Retinoblastoma (ArMOR): Experience with a Multiracial Cohort. Cancers. 2024; 16(20):3516. https://doi.org/10.3390/cancers16203516

Chicago/Turabian Style

Vempuluru, Vijitha S., Rajiv Viriyala, Virinchi Ayyagari, Komal Bakal, Patanjali Bhamidipati, Krishna Kishore Dhara, Sandor R. Ferenczy, Carol L. Shields, and Swathi Kaliki. 2024. "Artificial Intelligence and Machine Learning in Ocular Oncology, Retinoblastoma (ArMOR): Experience with a Multiracial Cohort" Cancers 16, no. 20: 3516. https://doi.org/10.3390/cancers16203516

APA Style

Vempuluru, V. S., Viriyala, R., Ayyagari, V., Bakal, K., Bhamidipati, P., Dhara, K. K., Ferenczy, S. R., Shields, C. L., & Kaliki, S. (2024). Artificial Intelligence and Machine Learning in Ocular Oncology, Retinoblastoma (ArMOR): Experience with a Multiracial Cohort. Cancers, 16(20), 3516. https://doi.org/10.3390/cancers16203516

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop