Application of Machine Learning Predictive Models for Early Detection of Glaucoma Using Real World Data

Round 1

Reviewer 1 Report

The article presents a proposal for building predictive models for early detection (1 year earlier) of glaucoma. Uses an extensive EHR database of over 650 US hospitals (CERNER Health Facts). The tested machine learning (ML) models were 4: XGBoost, MLP, RF, LR. The results obtained by the 3 best models point to 81% AUC.

 

The article is well organized and well written. The chosen approach uses features extracted from EHR as opposed to other approaches, for example, based on fundus images (computer vision and image processing approach).

 

In the introduction (section 1) they could have discussed a little more other approaches to diagnosing glaucoma. Articles such as "Bragança, C.P.; Torres, J.M.; Soares, C.P.d.A.; Macedo, L.O. Detection of Glaucoma on Fundus Images Using Deep Learning on a New Image Set Obtained with a Smartphone and Handheld Ophthalmoscope. Healthcare 2022, 10, 2345. doi.org/10.3390/healthcare10122345" and others discuss these approaches a little further.

 

The 32 attributes (line124) should be explicitly listed (in a table or annex)

 

Authors must explain if they used validation set. If not, why not? (feature selection (FS) and hyperparameter tuning (HT) usually require use of validation set)

 

Although the best models are XGBoost, MLP, and RF, the LR model has the lowest FN (Table 3). This should be discussed in the article and how it relates to FS and HT. In the models, in particular in the LR, did they use Zero-one loss functions? Should the FN cost be greater than the FP cost?

 

They could have tested an ensemble scheme (bagging, boosting, voting, ...) that included XGBoost and MLP, for example. Note that RF is already an ensemble.

Author Response

Please see the attachment

Author Response File: Author Response.pdf

Reviewer 2 Report

The authors performed a retrospective analysis of a very large number of electronic health records (HER) using data-mining techniques and machine learning algorithms. The topic is clinically relevant, the study design is sound, and the manuscript is concisely written and already addresses almost all relevant issues.

Clinical decision making in the management of glaucoma patients are impacted by a number of factors, including (1) the frequent dissociation between the main target of treatment (intraocular pressure) and disease progression, (2) the heterogeneity of “the glaucomas”, (3) the difficulty of early diagnosis (many patients with varying risk are labelled as “glaucoma suspects”), and (4) the long time spans involved. Often, a poor decision will manifest itself only many years later. Therefore, any scientific support in decision making is clinically useful.

Methods section

The dataset contains data over a span of 15 years. However, this does not mean that regular visits have taken place over this time span, does it? Is it certain that no visits have taken place elsewhere when they are not in the dataset? What are the average time spans between first visit and the relevant interval 2014/15? Could you discuss this in more detail?

Bias

As the authors have stated, there are a number of potential sources of bias when analyzing EHR datasets. For example, the ICD9 group 365 includes borderline glaucoma (365.0) as a very heterogeneous group. Could the relative frequency of this subgroup in the glaucoma group be specified from the EHR data?

The risk of labelling a patient with borderline glaucoma (measuring an borderline IOP or of some physician classifying a large optic nerve head as glaucoma) might also increase with the number of “eye related visits” – for example in patients with diabetes mellitus.

Clinical relevance

It should be stated clearly that a high risk with being labelled with the diagnosis glaucoma is not identical with a high risk of going blind (for example see diagnosis 365.04 - ocular hypertension - and the results of the Ocular Hypertension Treatment Trial, where there was a relatively low conversion rate to glaucoma even in the untreated group; or the diagnosis 365.01: open angle with borderline findings, low risk). This is clinically relevant, because over-treatment also poses a significant risk to glaucoma suspects.

Angle closure vs. open angle glaucomas

Furthermore, the angle-closure glaucomas are very different from primary open-angle glaucoma and have a very different set of risk factors. They are usually more frequent in Asians than in African Americans, and they would be expected to be more closely associated with cataract. This should also be discussed.

Lacrimal disorders

The large heterogeneity of the diagnosis groups might als affect the other, diagnosis groups, both eye-related and not eye-related. What diagnoses are included in lacrimal disorders? Does this include dry eye, where patients might receive steroids?

Conclusion

The predictive value of these data as demonstrated by the ROC analysis is surprisingly high, because no direct glaucoma-related parameters were involved in the prediction. Still, I’m not completely sure how these insights should be translated into clinical practice. Should they only prompt hypothesis-driven research by identifying new associations? Here, the difficulties with interpreting relationships between parameters might get in the way, but it would greatly improve clinicians' trus in these models. Or should hospitals screen their patient records for patients at risk and recommend screening? Would they have an incentive to do so? Do insurance companies in the U.S. have access to these data (I’m from Europe). I would also discuss the feasibility of performing similar studies with development of visual disability from glaucoma as an end point.

Author Response

Please see the attachment

Author Response File: Author Response.pdf

Reviewer 3 Report

Title: Application of Machine Learning Predictive Models for Early 2 Detection of Glaucoma using Real World Data

General comments: The subject and methodology of the investigation are empathetic topics, and the study itself is necessary. Nonetheless, additional information and clearer expression are required. In addition, the data analysis approach appears to be reasonable and well-designed overall, although it lacks tables and figures for specific outcomes and methods.

 

1.    In the abstract, there is insufficient description regarding the performance analysis outcomes; therefore, they must be rewritten in more depth.

2.    Introduction: It is crucial to specify whether the glaucoma label refers to open-angle glaucoma or all glaucoma. Since in table 4, angle closure glaucoma is more closed to cataract than OAG.

3.    Method: It is necessary to reinforce the implementation method in detail about the ML method so that general ophthalmologists can understand it.

4.    Result: It is required to present study results on web data, an FTP server, or github in order to prove or support them or summary statics data as supplementary for data availablity.

5.    Cataract, atherosclerosis, diabetes, obesity may be major factor for glaucoma, however, the relationship between lacrimal dysfunction and glaucoma is obscure.

6.    It will aid comprehension if you illustrate the clinical implications of section 4.1 (Interpretation and Clinical Relevance).

 

7.    Lipoid metabolism -> lipid metabolism looks better (line 259)

Author Response

Please see the attachment

Author Response File: Author Response.pdf

Round 2

Reviewer 3 Report

After revision, MS improved.

minor comments

1. Table 4 "Arial Font" changed into MDPI font Palatino Linotype

2. Fig 2. require enhancing of image quality and improve resolutions with fonts

 

Author Response

Please see the attachment.

Author Response File: Author Response.docx