Exploring the Prognostic Impact of Non-Obstructive Coronary Artery Lesions through Machine Learning

Round 1

Reviewer 1 Report

Comments and Suggestions for Authors

Although the topic is advanced and the subject is very interesting, unfortunately they are not able to submit this manuscript for evaluation because the reference list that was used was not made available to me. I request the authors to resubmit the manuscript with a completed references section. 

Author Response

Reviewer 1:

Although the topic is advanced and the subject is very interesting, unfortunately they are not able to submit this manuscript for evaluation because the reference list that was used was not made available to me. I request the authors to resubmit the manuscript with a completed references section. 

Response:

We apologize to Reviewer 1. There must have been an error, which we are unable to explain, that prevented you from accessing the bibliography. The other reviewers seem to have been able to review the complete manuscript, including that section. We are submitting this new version with the comments from the other reviewers and trust that this time there will be no issues preventing you from reviewing the complete content.

Reviewer 2 Report

Comments and Suggestions for Authors

The paper reports a study on the use of machine learning for prognostics of non-obstructive coronary arteries. The study is interesting, some suggestions are reported in the following to improve the manuscript:

- A figure/graphical abstract with the workflow of the study can be useful

- Details and figures about the data and the dataset need to be described more in details 

- Introduction, abstract, and conclusions can be improved highlighting the added value of the study

Comments on the Quality of English Language

The quality of English is ok

Author Response

Reviewer 2:

The paper reports a study on the use of machine learning for prognostics of non-obstructive coronary arteries. The study is interesting, some suggestions are reported in the following to improve the manuscript:

- A figure/graphical abstract with the workflow of the study can be useful

- Details and figures about the data and the dataset need to be described more in details 

- Introduction, abstract, and conclusions can be improved highlighting the added value of the study

Response:

The authors appreciate your review and comments. We now address the three key issues you raised:

1. We have added a figure that succinctly and systematically summarizes the inclusion of patients and the classification performed.
2. In the Results section, we have included various metrics that provide a more detailed understanding of the dataset's behavior (Tables 2 and 3). Additionally, we have reoriented the discussion and conclusions according to the new findings presented in these tables. Specifically, we have discussed the limitations of the machine learning models we developed, which demonstrate insufficient predictive capability for the event, despite their satisfactory performance in predicting the absence of the event.
3. The Introduction, Conclusions, and Abstract sections have been revised to emphasize the main findings of the study.

 

Reviewer 3 Report

Comments and Suggestions for Authors

The research utilizes diverse machine learning algorithms to evaluate the predictive importance of non-obstructive coronary artery disease (CAD). This novel method improves the precision of forecasting future cardiovascular events, as demonstrated by the support vector machines model achieving an accuracy rate of 87.352%. However, the authors need to address these concerns before the article can be published:

1.     The study's retrospective design may include biases associated with the collection of data and selection of patients. Retrospective studies depend on existing data, which may not necessarily encompass all pertinent variables or could have discrepancies. How did the authors mitigate biases? A comprehensive understanding of the inclusion and exclusion criteria is necessary.

2.     Was the validation of the machine learning models conducted using an external dataset? The uncertainty over the generalizability of the model's prediction performance to other populations persists in the absence of external validation.

3.     Although the average follow-up duration of 43 months is a reasonable timescale to evaluate long-term outcomes, it may not be enough to capture all cardiovascular events that occur later. Extended years of follow-up could yield more thorough insights into the long-term prognostic importance of non-obstructive coronary artery disease (CAD).

4.     The opening lacks sufficient motivation and fails to establish the necessity of earlier studies.

 

5.     The work employs diverse machine learning algorithms. However, without explicit information on how overfitting was mitigated, there exists a potential danger that the models might exhibit high performance on the training data but demonstrate poor performance on new, unseen data. This matter is especially pertinent in intricate models such as support vector machines.

Comments on the Quality of English Language

Moderate editing of English language required

Author Response

Reviewer 3:

The research utilizes diverse machine learning algorithms to evaluate the predictive importance of non-obstructive coronary artery disease (CAD). This novel method improves the precision of forecasting future cardiovascular events, as demonstrated by the support vector machines model achieving an accuracy rate of 87.352%. However, the authors need to address these concerns before the article can be published:

1. The study's retrospective design may include biases associated with the collection of data and selection of patients. Retrospective studies depend on existing data, which may not necessarily encompass all pertinent variables or could have discrepancies. How did the authors mitigate biases? A comprehensive understanding of the inclusion and exclusion criteria is necessary.
2. Was the validation of the machine learning models conducted using an external dataset? The uncertainty over the generalizability of the model's prediction performance to other populations persists in the absence of external validation.
3. Although the average follow-up duration of 43 months is a reasonable timescale to evaluate long-term outcomes, it may not be enough to capture all cardiovascular events that occur later. Extended years of follow-up could yield more thorough insights into the long-term prognostic importance of non-obstructive coronary artery disease (CAD).
4. The opening lacks sufficient motivation and fails to establish the necessity of earlier studies.
5. The work employs diverse machine learning algorithms. However, without explicit information on how overfitting was mitigated, there exists a potential danger that the models might exhibit high performance on the training data but demonstrate poor performance on new, unseen data. This matter is especially pertinent in intricate models such as support vector machines.

Response:

The authors of this work sincerely appreciate your review and comments, which help us improve the quality of our research. We will now address your questions point by point.

1.-      The authors acknowledge that the retrospective design of the study may be associated with biases due to the lack of collection of certain variables and the absence of randomization, primarily due to confounding factors in the latter case. In response to your comment, we have further emphasized this point in the second sentence of the limitations section. In estimating the risk of events associated with the presence of non-obstructive coronary artery disease, bias was addressed using statistical methods in our previous publication (Rodríguez-Capitán J, Sánchez-Pérez A, Ballesteros-Pradas S et al.. Prognostic Implication of Non-Obstructive Coronary Lesions: A New Classification in Different Settings. J Clin Med. 2021 Apr 25;10(9):1863. doi: 10.3390/jcm10091863), where models were developed adjusted for age, sex, and other factors (including smoking habit, dyslipidemia, diabetes, heart failure, kidney failure, indication for angiography [chest pain/chronic coronary syndrome as opposed to acute coronary syndrome], atrial fibrillation, and left ventricular ejection fraction). In the current analysis, various machine learning models were tested, along with a SHAP analysis, whose results are consistent with our previous analysis, highlighting the prognostic value of non-obstructive coronary artery disease.

 

As this work consists of a secondary analysis with a differentiated methodology, based on a previous study whose methodology and results have already been published (Rodríguez-Capitán J, Sánchez-Pérez A, Ballesteros-Pradas S et al.. Prognostic Implication of Non-Obstructive Coronary Lesions: A New Classification in Different Settings. J Clin Med. 2021 Apr 25;10(9):1863. doi: 10.3390/jcm10091863), the initial methodology with the sample inclusion and exclusion criteria was briefly described in the text of the current manuscript. In this revised version of the manuscript, we have added a figure (Figure 1) summarizing the inclusion criteria and the patient flow leading to the composition of the comparison groups.

 

2.- In the present analysis, no external validation of the results obtained by the machine learning models was performed, which represents a limitation that we have included in the corresponding section

3.- Without a doubt, a longer follow-up period would contribute to a better estimation of the prognostic value associated with non-obstructive coronary artery disease. In accordance with your suggestion, the authors have added this consideration to the limitations section.

4.- We have rewritten the introduction to clarify the rationale and necessity of the present study.

5.- The authors agree with these statements regarding the limitation of the results obtained. This risk of overfitting is linked to the absence of an external validation cohort, as we have already discussed in our response to point 2 and also in the limitations section of the manuscript."

Reviewer 4 Report

Comments and Suggestions for Authors

Need more evidence of data and results

Author Response

Need more evidence of data and results

The authors appreciate your comment. We have added two tables (Table 2 and Table 3) in which we have detailed the performance metrics of the various machine learning models developed. Unfortunately, we must acknowledge that despite satisfactory accuracy and good predictive capability for the absence of the event, all models exhibit insufficient predictive capability for the event itself, resulting in poor recall, precision, and F1 scores. These new limitations have been addressed in the Discussion.

Reviewer 5 Report

Comments and Suggestions for Authors

Accuracy, while a useful metric, only provides a general overview of the model's performance. It does not take into account the balance between sensitivity (true positive rate) and specificity (true negative rate), which is crucial in medical predictions where both false negatives and false positives can have significant consequences. Moreover, accuracy can be misleading in imbalanced datasets where one class significantly outnumbers the other. In such cases, a model might achieve high accuracy by simply predicting the majority class, while failing to correctly identify instances of the minority class, which could be the more critical one in a medical context.

You need to include a table where, for each ML tested, you also state other performance metrics such as precision, recall, F1 score, and most importantly, the Area Under the Receiver Operating Characteristic Curve (AUC-ROC). The AUC-ROC is a robust metric that considers both sensitivity and specificity and is less sensitive to imbalanced datasets. Only then you can decide which of the tested ML algorithms is the best-performing. 

You also need to include the confusion matrix, which allows for a more comprehensive evaluation of the model's performance. It provides insights into not just the overall accuracy of the model, but also its precision (how many of the positive identifications were actually correct), recall (how many of the actual positives were identified correctly), and F1 score (the harmonic mean of precision and recall). By providing the number of false positives and false negatives, a confusion matrix gives insight into the type of errors the model is making. This is particularly important in a medical context, where different types of errors can have different consequences. Confusion matrices can also be useful for comparing different models. While one model might have a higher overall accuracy, another might have a better balance of precision and recall, which might be more appropriate depending on the specific context.

Author Response

Accuracy, while a useful metric, only provides a general overview of the model's performance. It does not take into account the balance between sensitivity (true positive rate) and specificity (true negative rate), which is crucial in medical predictions where both false negatives and false positives can have significant consequences. Moreover, accuracy can be misleading in imbalanced datasets where one class significantly outnumbers the other. In such cases, a model might achieve high accuracy by simply predicting the majority class, while failing to correctly identify instances of the minority class, which could be the more critical one in a medical context.

You need to include a table where, for each ML tested, you also state other performance metrics such as precision, recall, F1 score, and most importantly, the Area Under the Receiver Operating Characteristic Curve (AUC-ROC). The AUC-ROC is a robust metric that considers both sensitivity and specificity and is less sensitive to imbalanced datasets. Only then you can decide which of the tested ML algorithms is the best-performing. 

You also need to include the confusion matrix, which allows for a more comprehensive evaluation of the model's performance. It provides insights into not just the overall accuracy of the model, but also its precision (how many of the positive identifications were actually correct), recall (how many of the actual positives were identified correctly), and F1 score (the harmonic mean of precision and recall). By providing the number of false positives and false negatives, a confusion matrix gives insight into the type of errors the model is making. This is particularly important in a medical context, where different types of errors can have different consequences. Confusion matrices can also be useful for comparing different models. While one model might have a higher overall accuracy, another might have a better balance of precision and recall, which might be more appropriate depending on the specific context.

The authors appreciate your thorough analysis of our work. In the current version of the manuscript, we have added two tables (Table 2 and Table 3) that include the new metrics calculated for each of the machine learning models. The data indicate that all models exhibit acceptable accuracy based on good discrimination of the absence of events; however, their prediction of events is insufficient, leading to poor recall, precision, and F1 scores. This is undoubtedly a significant limitation of our work, prompting us to reorient the entire Discussion, Conclusions, and Abstract. Given the current data, we believe that this study demonstrates that applying machine learning to estimate the prognostic value of non-obstructive coronary lesions is both novel and feasible. Our models have shown good predictive capability for the absence of events during follow-up and even suggest a potential relationship between these lesions and the event, as indicated by the SHAP analysis, although this is limited by the new results we have discussed.

Round 2

Reviewer 1 Report

Comments and Suggestions for Authors

The authors present an interesting multicentre paper looking at coronary lesions that have undergone prognostic evaluation using machine learning models. However, it presents little new information, and seems so far not very innovative, with the only conclusion of the paper being a single sentence in the conclusion: "The analyses suggest the prognostic value of these non-obstructive coronary lesions (20-50%)."

 

The introduction is well written, but brief. I would suggest pointing out possible weaknesses of current methods used to assess coronary lesions, in a few sentences.

 

The method is written in a sequence, so it should be divided in a rational way. E.g. into data on study and control groups, their classification methods, and recruitment (including inclusion and exclusion criteria). Provide a title for each subsection.

 

Do the same with the results as with M&M.

 

The discussion is finely written and exhaustive. 

 

 

 

Detail amendments:

 

Line 94 - these exclusionary guidelines would be worth putting on the diagram.

 

Line 106 - please rearrange the diagram and give the group sizes as they are in the results section. It seems illegible to combine patients in the normal coronary arteries group and patients in the non-obstructive group.

 

Line 147 - the methodology lacks a separate subsection on methods, with an explanation of the tests used, the current explanation is not sufficient. Was only one test used?

 

Figure 2. is of very poor quality, this makes it difficult to read. Please improve it and also present the data from Shap val in tables.

Author Response

The authors express their gratitude for your thorough review and insightful comments, which have contributed to the improvement of the manuscript. Please find now our response to your comments:

The authors present an interesting multicentre paper looking at coronary lesions that have undergone prognostic evaluation using machine learning models. However, it presents little new information, and seems so far not very innovative, with the only conclusion of the paper being a single sentence in the conclusion: "The analyses suggest the prognostic value of these non-obstructive coronary lesions (20-50%)."

 The introduction is well written, but brief. I would suggest pointing out possible weaknesses of current methods used to assess coronary lesions, in a few sentences.

We have added a few sentences addressing the interesting aspect you suggested (lines 67-77).

 The method is written in a sequence, so it should be divided in a rational way. E.g. into data on study and control groups, their classification methods, and recruitment (including inclusion and exclusion criteria). Provide a title for each subsection.

We have divided the Materials and Methods section into three subsections, each with its own heading.

 Do the same with the results as with M&M.

We have also divided the Results section into three subsections, each with its corresponding heading.

 The discussion is finely written and exhaustive. 

Detail amendments:

 Line 94 - these exclusionary guidelines would be worth putting on the diagram. Line 106 - please rearrange the diagram and give the group sizes as they are in the results section. It seems illegible to combine patients in the normal coronary arteries group and patients in the non-obstructive group.

We have redesigned Figure 1 according to your suggestions: the exclusion criteria are now shown after the inclusion criteria, and the only specific numerical data presented correspond to the final classification of the patients, on which all subsequent analyses were conducted, to avoid misinterpretation of the results.

 Line 147 - the methodology lacks a separate subsection on methods, with an explanation of the tests used, the current explanation is not sufficient. Was only one test used?

We have added a new subsection in the Materials and Methods to detail the methodology applied, including both machine learning techniques and classical statistical methods. The use of classical statistical tools was limited, as most of the analyses pertained to machine learning. Nevertheless, following your suggestions, we have added a sentence explicitly describing the statistical methodology used to create Table 1, which was mistakenly omitted in the previous version of the manuscript

 Figure 2. is of very poor quality, this makes it difficult to read. Please improve it and also present the data from Shap val in tables.

We have increased the resolution of Figure 2 and created a table summarizing the main SHAP values with numerical data. Since Figure 2 is more informative, including individual data for each patient and variable, we have kept it in the main manuscript. The table, which only summarizes the key numerical data, has been moved to the supplementary material (unless the reviewer suggests otherwise).

Reviewer 3 Report

Comments and Suggestions for Authors

The authors have included all the limitations of the study in their report. However, a longer follow-up is a must in this case.

Comments on the Quality of English Language

Minor editing of English language required.

Author Response

The authors express their gratitude for your meticulous review and insightful comments.

The authors have included all the limitations of the study in their report. However, a longer follow-up is a must in this case.

As per your suggestion, we have explicitly stated that longer follow-up is a "must" in this context.

Comments on the Quality of English Language: Minor editing of English language required.

We have carefully reviewed the manuscript and corrected all errors we were able to identify

Reviewer 5 Report

Comments and Suggestions for Authors

You've thoroughly addressed all of my comments to my satisfaction, and we've all gained valuable insights from this. Thank you!

Author Response

You've thoroughly addressed all of my comments to my satisfaction, and we've all gained valuable insights from this. Thank you!

Thank you once again, both for your thorough review and for your kind words.

Round 3

Reviewer 3 Report

Comments and Suggestions for Authors

The article can be accepted