Frequency of Missed Findings on Chest Radiographs (CXRs) in an International, Multicenter Study: Application of AI to Reduce Missed Findings

Round 1

Reviewer 1 Report

This paper needs major revision :

 

-why the  abstract included in the paper is not clear.

 

-very short introduction section

 

The proposed work seems to be good but still some changes if made will surely improve the quality of the paper.

-The related work can be improved by providing a comparative study stating what is observed from the literature.

-The related work can be rewritten.

- The metrics for evaluating the proposed work can be given.

- The example illustration is good, but the experimental results can be validated and tested.

- The clarity of the figures can be still improved.

 

- In general update that list by the following reference related to predictions:

 

-However, the approach has several flaws, the most important one is that the objective functions are not in conflict. Then the use of a multi-objective optimization approach is not justified .

 

-The novelty of the method is limited. The presentation (clarity/structure) is average and the description for the method is not very easy to understand.

 

- Check the mathematical notation especially for the proposed method.

 

 - Add a new figure to show the general procedures of the proposed method

 

 - I think the subject and object of this paper is very ambiguous in the introduction even though authors well described the previous work, I hope the author should consider why this paper is necessary to the read. 

 

Check the mathematical notations

 

Please, elaborate the discussion with the advantage of the proposed method

 

 

 

 

 

Author Response

Dear Editor and Reviewers,

 

Thank you very much for your letter and for the reviewers’ comments concerning our manuscript titled” Performance of a chest radiography AI algorithm for detection of missed or mislabeled findings: A multicenter study” (ID: 1876902). Those comments are all valuable and very helpful for revising and improving our paper, as well as the important guiding significance to our research. We have studied comments carefully and have made correction which we hope will meet with approval.

Reviewer 1

1-why the abstract included in the paper is not clear.

 Response: thank you for your comment. We made the changes

2-very short introduction section. The proposed work seems to be good but still some changes if made will surely improve the quality of the paper.

 

Response: Thanks for your comment. We added the following content:

“If successful, AI algorithm could help improve the quality of radiology reports, enhance patient care, and help avoid malpractice lawsuits from missed radiologic findings. Although there are multiple prior publications on AI performance, to our best knowledge, there are sparse data on the performance of AI algorithm on missed radiological findings.” (R1-2)

3-The related work can be improved by providing a comparative study stating what is observed from the literature.

Response: Thank you for your comment. We added the related work to introduction.

4-The related work can be rewritten.

Response: we made the changes

5-The metrics for evaluating the proposed work can be given.

Response: we are grateful to the reviewer for this suggestion. We have described this in great details in the method section. We did not mention it in the introduction to avoid duplicating our text.

6- The example illustration is good, but the experimental results can be validated and tested.

Thank you for your kind comment, we have added a citation on the use of the platform in a prior study.

7-The clarity of the figures can be still improved.

Reply: We replaced the figures with sharper images

 

8- In general update that list by the following reference related to predictions:

 We have added a few more reference to the manuscript.

9-However, the approach has several flaws, the most important one is that the objective functions are not in conflict. Then the use of a multi-objective optimization approach is not justified.

Our sincere apologies, but we are unable to understand your comment on multi-objective optimization approach. We did use a multi-finding AI algorithm that has been reported in several prior studies.

 

10-The novelty of the method is limited. The presentation (clarity/structure) is average and the description for the method is not very easy to understand.

 Response: We are sorry for the confusion. We made the changes.

11-Check the mathematical notation especially for the proposed method.

 Response: We made the changes

12-Add a new figure to show the general procedures of the proposed method

Response: We added a figure summarizing the proposed method

 

 13-I think the subject and object of this paper is very ambiguous in the introduction even though authors well described the previous work, I hope the author should consider why this paper is necessary to the read. 

  Response: We are sorry for the confusion. We made the changes.

14-Check the mathematical notations

Response: We made the changes

 

Reviewer 2 Report

Frequency of missed findings on chest radiographs (CXRs) in an international, multicenter study: Application of AI to reduce missed findings

1- The language is very challenging. The text should be reviewed in English
2- "Despite an overwhelming use, CXR interpretation is subjective and prone to wide interobserver inconsistencies based on readers' knowledge and experience [5]." Despite an overwhelming use, CXR interpretation is subjective and prone to wide interobserver inconsistencies based on readers' knowledge and experience. But reference 5 is too old (2007) to interpret today's conditions. 5- Mahesh PA, Vidyasagar B, Jayaraj BS. Principles and Interpretation of Chest X-rays. Orient Blackswan; 2007. Support this claim with timely literature.
3- Who are the XX, YY and ZZ mentioned between line 75 and 80. Are they people in the author list or unacademic employees?
4- Excel file does not have a special meaning. Instead, an expression such as "data retrieved in the tabular form" should be used.
5- Figure 4 is too small to see and understand. It should be improved.
6- Database distribution in terms of ground-truth findings needs to be presented in a table.
7- Is there any extrafindings in the CXRs? (Extra finding is the case when the doctors examining the patient said there was a finding in the CRX, but in reality (ground-truth) there was no finding.)  What kind of method was followed regarding the extra findings. Are these also considered in the status of missing findings?
8-  "Our study demonstrates that a substantial number of clinically important findings are missed on CXRs regardless of the practice type and location." In order to claim this according to your work, not the literature, you must submit the following information. Which percentage of total patients have misfinding or extrafinding (after this, i call both of them as wrong finding) on their CXRs? Based on the perceived clinical importance determined by two professional employee, what are the average perceived clinical importance values of each findings mentioned in the article.
9- And the narration of the article is quite challenging to understand. The article should be made more fluid and understandable.

In this study, it is claimed that artificial intelligence will reduce misdiagnoses. The most important point of your study is ground-truth determination of CRX images. Although some interesting information is presented in this study, the originality of the study is weak. Because in many recent studies, it is presented that using CRXs images, the diagnosis of different ailments (recently, for example, COVID-19, pneumonia, etc.) with artificial intelligence applications has been achieved with high accuracy. Even now, the effects of fine details on the subject such as which of the deep learning methods are more successful or the competition of innovative multilayer neural networks with deep learning are examined. Also your references are quite old.

Author Response

Dear Editor and Reviewers,

 

Thank you very much for your letter and for the reviewers’ comments concerning our manuscript titled” Performance of a chest radiography AI algorithm for detection of missed or mislabeled findings: A multicenter study” (ID: 1876902). Those comments are all valuable and very helpful for revising and improving our paper, as well as the important guiding significance to our research. We have studied comments carefully and have made correction which we hope will meet with approval.

 

 Reviewer 2

• The language is very challenging. The text should be reviewed in English

 

Response: we have reviewed and improved the language throughout the manuscript.

2- "Despite an overwhelming use, CXR interpretation is subjective and prone to wide interobserver inconsistencies based on readers' knowledge and experience [5]." Despite an overwhelming use, CXR interpretation is subjective and prone to wide interobserver inconsistencies based on readers' knowledge and experience. But reference 5 is too old (2007) to interpret today's conditions. 5- Mahesh PA, Vidyasagar B, Jayaraj BS. Principles and Interpretation of Chest X-rays. Orient Blackswan; 2007. Support this claim with timely literature.

 

Response: we added the following citations:

 

Bruno MA, Walker EA, Abujudeh HH. Understanding and confronting our mistakes: the epidemiology of error in radiology and strategies for error reduction. Radiographics. 2015 Oct;35(6):1668-76.

Dillon DG, Rodriguez RM. Screening performance of the chest X-ray in adult blunt trauma evaluation: Is it effective and what does it miss?. The American Journal of Emergency Medicine. 2021 Nov 1;49:310-4.

3- Who are the XX, YY and ZZ mentioned between line 75 and 80. Are they people in the author list or unacademic employees?

 

These are people in the author list.

4- Excel file does not have a special meaning. Instead, an expression such as "data retrieved in the tabular form" should be used.

 

Response: we made the changes

5- Figure 4 is too small to see and understand. It should be improved.

 

Response: we replaced the figure with the sharper figures.

6- Database distribution in terms of ground-truth findings needs to be presented in a table.

 

Response: Thank you for your comment. We have added this information to table 1 and 2.

7- Is there any extrafindings in the CXRs? (Extra finding is the case when the doctors examining the patient said there was a finding in the CXR, but in reality (ground-truth) there was no finding.)  What kind of method was followed regarding the extra findings. Are these also considered in the status of missing findings?

 

Response: We are sorry for the confusion but given the fact that we only included CXRs with completely normal report (for findings and impression sections), we did not assess false positive radiology reports.

8- "Our study demonstrates that a substantial number of clinically important findings are missed on CXRs regardless of the practice type and location." In order to claim this according to your work, not the literature, you must submit the following information. Which percentage of total patients have misfinding or extrafinding (after this, i call both of them as wrong finding) on their CXRs? Based on the perceived clinical importance determined by two professional employees, what are the average perceived clinical importance values of each finding mentioned in the article.

 

Response: we agree with your comment but as stated above we did not include any radiographs where radiology reports described a finding. However, we have added information on the percentage of radiographs with missed findings (17.1%) to the results section.

9- And the narration of the article is quite challenging to understand. The article should be made more fluid and understandable.

In this study, it is claimed that artificial intelligence will reduce misdiagnoses. The most important point of your study is ground-truth determination of CRX images. Although some interesting information is presented in this study, the originality of the study is weak. Because in many recent studies, it is presented that using CRXs images, the diagnosis of different ailments (recently, for example, COVID-19, pneumonia, etc.) with artificial intelligence applications has been achieved with high accuracy. Even now, the effects of fine details on the subject such as which of the deep learning methods are more successful or the competition of innovative multilayer neural networks with deep learning are examined. Also your references are quite old.

 

Response: we have added statements on importance of our study. We have made the narrative more fluid in the method and results. We have added new references as requested.

 

Author Response File: Author Response.docx

Reviewer 3 Report

Please provide a literature review section detailing the existing studies on the similar themes. Compare your proposed methods with the exisiting algorithms. Conclusion section is missing. Write the conclusion and highlight the limitations of your study.

Author Response

Dear Editor and Reviewers,

 

Thank you very much for your letter and for the reviewers’ comments concerning our manuscript titled” Performance of a chest radiography AI algorithm for detection of missed or mislabeled findings: A multicenter study” (ID: 1876902). Those comments are all valuable and very helpful for revising and improving our paper, as well as the important guiding significance to our research. We have studied comments carefully and have made correction which we hope will meet with approval.

 

Reviewer 3

1.Please provide a literature review section detailing the existing studies on the similar themes.

Response: thanks for your comment. We added the “related works” in introduction.

2.Compare your proposed methods with the existing algorithms.

Response: thank you for your comments. We have added more literature on existing algorithms in the introduction section.

3.Conclusion section is missing. Write the conclusion and highlight the limitations of your study.

Response: We made the changes

 

Round 2

Reviewer 1 Report

 Accept in present form

Reviewer 2 Report

It has been observed that the authors carefully examined the issues stated in the criticisms and made the necessary corrections. In this way, the work has become more interesting, fluid and understandable.