Diagnostic Efficacy of Chest Computed Tomography with a Dual-Reviewer Approach in Patients Diagnosed with Pneumonia Secondary to Severe Acute Respiratory Syndrome Coronavirus 2

Round 1

Reviewer 1 Report (Previous Reviewer 2)

This study evaluated the diagnostic accuracy using two reviewers on two groups of data (A: 482 positive, B: 408 negative). The individual diagnosis results (Reviewer 1: 108 N vs 374 P in A and 266 N vs 142 P in B; Reviewer 2: 150 N vs 332 P in A and 277 N vs 131 P in B) were compared to the dual-reviewer census (87 N vs 395 P in A and 274 N vs 134 P in B). This study has merit in testing the dual-reviewer method and quantifying the differences, which is useful in documenting the benefits of using two reviewers vs. one reviewer. However, there are several concerns that dampen my support for this paper.

Results: (1) The diagnosis of COVID using chest CT is subjective and experience-dependent. A team of two reviewers or more does not guarantee to make much improved predictions, but rather more influenced by the most senior. (2) The false negative and false positive are very high in this study, which makes the proposed method not appealing, even though it was better than a single reviewer evaluation. Why is the accuracy this low? (3) Numerous studies using deep learning classification have achieved 90%+ accuracy using lung CT scan and 95%+ using chest X-ray images, which is much higher than the results in this study.     

Presentation: (1) several paragraphs in Discussion should be moved to Introduction. (2) The text does not flow well. Some sentences are hard to understand. They could be more clear, concise, and straightforward.

Details:  

1.     Please give the full name of RT-PCR at its first appearance: reverse-transcription polymerase chain reaction (RT-PCR)

2.     Lines 264-274: not directly related to the results in this study. They should be in Introduction, not Discussion.  

3.     Unclear what the authors tried to express in lines 279-286: (1) It appears that the authors tried to justify why a dual-reviewer evaluation is used (i.e., not a standard method). Such justification is not needed at all. (2) “Hence, we emphasize that the result was superior to that published by other studies.” It is unclear what “the result” refers to? Is it the result in this study, or in Ref [7}. Similarly, it is unclear what “that published in other studies”? (3) Lines 283-286: I guess what the authors tried to express is: they found disagreements between the two reviewers, and improvements are needed in reviewers’ accuracy. Please rewrite these sentences to be more readable.   

4.     line 294: “Ai et al. [11] in the most extensive article on the subject published with 1014 patients,”. What do you mean “the most extensive article”?  “the most cited”? Please rephrase it.  

 

5.     Lines 300-308: should be in Introduction than Discussion. Only studies that were directly compared to the results of this study should be included in Discussion.  

Some sentences are hard to understand. They could be more clear, concise, and straightforward

Author Response

Guadalajara, Jalisco. Mexico. August 16, 2023.

 

Prof. Dr. Emilio Quaia.

Prof. Dr. Pascal N. Tyrrell

Editors-in-Chief.

 

I am writing to submit the corrections for the peer-review comments on our manuscript recently submitted to your journal.

We have made the necessary corrections to address the comments provided by the reviewers. Furthermore, we have included a detailed response for each point raised by the reviewers.

We believe that these changes have substantially improved the clarity and accuracy of our paper.

It also includes a certificate of revision of the English language by the company OnLine English (Superior English Editing Service).

Thank you for your time and effort in reviewing our manuscript. We are grateful for the opportunity to revise our paper and hope that our revised manuscript meets the standards of MDPI Tomography.

 

Sincerely,

 

_________________________________________

Alejandro González-Ojeda, M.D., Ph.D., F.A.C.S.

Corresponding author.

 

 

Reviewer 1,

We would like to thank you for your valuable feedback on our manuscript titled “Diagnostic Efficacy of Chest Computed Tomography with a Dual-Reviewer Approach in Patients Diagnosed with Pneumonia Secondary to Severe Acute Respiratory Syndrome Coronavirus 2.”

Your comments and suggestions have been helpful in improving the quality of our paper. We appreciate the time and effort you have taken to review our manuscript thoroughly. In response to the comments you provided, we have revised the manuscript, which we believe address each of the points raised. In this letter, we will discuss in detail the changes we made in response to your comments.

1. Please give the full name of RT-PCR at its first appearance: reverse-transcription polymerase chain reaction (RT-PCR)
   • Answer: Thanks for the recommendation, the change was made in line 45
2. Lines 264-274: not directly related to the results in this study. They should be in Introduction, not Discussion.  
   • Answer: line 264-274 was changed to the introduction, remaining on line 68
3. Unclear what the authors tried to express in lines 279-286: (1) It appears that the authors tried to justify why a dual-reviewer evaluation is used (i.e., not a standard method). Such justification is not needed at all. (2) “Hence, we emphasize that the result was superior to that published by other studies.” It is unclear what “the result” refers to? Is it the result in this study, or in Ref [7}. Similarly, it is unclear what “that published in other studies”? (3) Lines 283-286: I guess what the authors tried to express is: they found disagreements between the two reviewers, and improvements are needed in reviewers’ accuracy. Please rewrite these sentences to be more readable.   
   • Answer: We rewrite the sentences to make them clearer, staying at line 328
4. line 294: “Ai et al. [11] in the most extensive article on the subject published with 1014 patients,”. What do you mean “the most extensive article”?  “the most cited”? Please rephrase it.  
   • Answer: We rewrite the sentences to to make them clearer, remaining on line 335, change to “in the article with de most extensive number of patients published with 1014 patients, in which they establish that two radiologists evaluated chest CT findings, concluded a diagnostic yield with sensitivity close to 97%, and established that the improvement in CT is ahead of the negativization of RT–PCR”.
5. Lines 300-308: should be in Introduction than Discussion. Only studies that were directly compared to the results of this study should be included in Discussion.  
   • Answer: line 300-308 was changed to the introduction, remaining on line 80

 

6. Some sentences are hard to understand. They could be more clear, concise, and straightforward
   • Answer: We've altered some sentences to make them more comprehensible in response to your feedback.

Author Response File: Author Response.docx

Reviewer 2 Report (New Reviewer)

Major comments

This paper shows that the dual-reviewer approach is superior to the single-reviewer approach for coronavirus disease 2019 (COVID-19) pneumonia imaging diagnosis with computed tomography (CT). This excellent study provides a means by which image analysis by two radiologists can improve the results of diagnosis compared with image analysis performed by one radiologist.

 

However, to ensure publication, the following must be mentioned.

1.     Images of COVID-19 pneumonia were already analyzed by two radiologists in the study cited by the author in the Discussion. The author's results are equivalent to these results, and it seems to us that demonstrating that the results are superior to the analysis of images by only one radiologist is of limited significance. If you wish to emphasize the superiority of the dual-reviewer approach, you must demonstrate this as a warning about how the judgment of only one person reduces the grades. The important point of this paper is to demonstrate how even one additional person can improve the accuracy of image analysis.

2.     Describing in detail how the reading results of two radiologists will be combined into one result can enhance the paper.

3.     This study attempts to provide diagnoses by using the COVID-19 Reporting and Data System (CO-ARDS) with CT images. A diagnosis of COVID-19 pneumonia is acceptable, but some patients with COVID-19 do not have pneumonia. Nearly 20% of the 87 patients (consensus) that were included in CO-ARDS levels 1–3 & PCR-positive were not considered to have pneumonia and were found to be clinically negative. According to the paper cited by the author, polymerase chain reaction (PCR) has a diagnostic probability of 80%–90% at CO-ARDS levels 3–4 for diagnosing COVID-19 pneumonia. Therefore, is there any reason to rely on CT images for diagnosis?

4.     Severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) PCR is negative, but there are some patients considered CO-ARDS levels 3–4. What is the cause? It would be beneficial to indicate in the Discussion the diseases that were included.

 

Minor comments

1.     The figure number and text description are incorrect.

2.     The value for Area for Reviewer 2 is missing in Table 5.

3.     Are Figures 5 and 6 in the text the same?

4.     Figure legends are missing for Figures 5 and 6.

5.     What is the diagnosis in non-COVID-19 cases that are identified as CO-ARDS levels 1–3?

6.     Please consider whether CO-ARDS levels 1–3 in cases that were diagnosed with COVID-19 have disease other than those from SARS-CoV-2.

Author Response

Guadalajara, Jalisco. Mexico. August 16, 2023.

 

Prof. Dr. Emilio Quaia.

Prof. Dr. Pascal N. Tyrrell

Editors-in-Chief.

 

I am writing to submit the corrections for the peer-review comments on our manuscript recently submitted to your journal.

We have made the necessary corrections to address the comments provided by the reviewers. Furthermore, we have included a detailed response for each point raised by the reviewers.

We believe that these changes have substantially improved the clarity and accuracy of our paper.

It also includes a certificate of revision of the English language by the company OnLine English (Superior English Editing Service).

Thank you for your time and effort in reviewing our manuscript. We are grateful for the opportunity to revise our paper and hope that our revised manuscript meets the standards of MDPI Tomography.

 

Sincerely,

 

_________________________________________

Alejandro González-Ojeda, M.D., Ph.D., F.A.C.S.

Corresponding author.

We would like to thank you for your valuable feedback on our manuscript titled “Diagnostic Efficacy of Chest Computed Tomography with a Dual-Reviewer Approach in Patients Diagnosed with Pneumonia Secondary to Severe Acute Respiratory Syndrome-Related Coronavirus 2.”

Your comments and suggestions have been helpful in improving the quality of our paper. We appreciate the time and effort you have taken to review our manuscript thoroughly. In response to the comments you provided, we have made revisions to the manuscript, which we believe address each of the points raised. In this letter, we will discuss in detail the changes we made in response to your comments.

However, to ensure publication, the following must be mentioned.

1. Images of COVID-19 pneumonia were already analyzed by two radiologists in the study cited by the author in the Discussion. The author's results are equivalent to these results, and it seems to us that demonstrating that the results are superior to the analysis of images by only one radiologist is of limited significance. If you wish to emphasize the superiority of the dual-reviewer approach, you must demonstrate this as a warning about how the judgment of only one person reduces the grades. The important point of this paper is to demonstrate how even one additional person can improve the accuracy of image analysis. 
   • Answer: at line 333, we believe it is significant because the article with the largest number of patients published 1014 patients, in which they establish that two radiologists evaluated chest CT findings, concluded a diagnostic yield with sensitivity close to 97%, and established that the improvement in CT is prior to the negative RT–PCR result. From the previous example, it can be deduced that the judgment of a single person diminishes the value of an image study's qualification; having an additional person evaluate the images enhances the analysis and the precision with which the images are evaluated.
2. Describing in detail how the reading results of two radiologists will be combined into one result can enhance the paper.
   • Answer: at line 192, How the results of each reviewer and the consensus will be obtained will be as follows: each one of the reviewers will separately assess each of the cases and record their result with one of the CO-RADS categories, if applicable. If they coincide in the same category, it is taken as consensus; if they do not coincide, both analyze the study together and issue a single result in a CO-RADS category.
3. This study attempts to provide diagnoses by using the COVID-19 Reporting and Data System (CO-ARDS) with CT images. A diagnosis of COVID-19 pneumonia is acceptable, but some patients with COVID-19 do not have pneumonia. Nearly 20% of the 87 patients (consensus) that were included in CO-ARDS levels 1–3 & PCR-positive were not considered to have pneumonia and were found to be clinically negative. According to the paper cited by the author, polymerase chain reaction (PCR) has a diagnostic probability of 80%–90% at CO-ARDS levels 3–4 for diagnosing COVID-19 pneumonia. Therefore, is there any reason to rely on CT images for diagnosis?
   • Answer: Even though the RT-PCR test has a high predictive value, it is typically limited in the country where the study is conducted; thus, it would be extremely advantageous to have access to alternative diagnostic techniques. During the pandemic, there was a shortage of these, and it persists to this day. Utilizing the patient's clinic as a diagnostic tool, some hospitals in this country are equipped with CT scanners, making them useful.
4. Severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) PCR is negative, but there are some patients considered CO-ARDS levels 3–4. What is the cause? It would be beneficial to indicate in the Discussion the diseases that were included.
   • Answer: In this study, patients who did not have SARS-COV-2 were diagnosed with a viral respiratory syndrome that included pneumonia due to influenza. Due to a dearth of resources, no additional tests were conducted to search for other etiologies, as the study was conducted during the pandemic. Moreover, most resources were devoted to the diagnosis of COVID.
5. The figure number and text description are incorrect.
   • Answer: Thank you; all the figures have been reviewed and the necessary adjustments made.
6. The value for Area for Reviewer 2 is missing in Table 5
   • Answer: The value of the area under the curve was added to table 5, which was absent, to make up for the oversight.
7. Are Figures 5 and 6 in the text the same?
   • Answer: They differ because one compares the reviewers to the consensus while the other does not, but the information is repeated, so the comment is appreciated and figure 5 is removed, leaving figure 6 (in the latest version, figure 7)
8. Figure legends are missing for Figures 5 and 6.
   • Answer: The correction has been made, and the legend has been appended to the text, line 245
9. What is the diagnosis in non-COVID-19 cases that are identified as CO-ARDS levels 1–3
   • Answer: Because the study was conducted during the pandemic, no additional research was conducted to rule out other pathologies.
10. Please consider whether CO-ARDS levels 1–3 in cases that were diagnosed with COVID-19 have disease other than those from SARS-CoV-2
    • Answer: The patients had no additional comorbidities, and their status was confirmed by RT-PCR.

Author Response File: Author Response.docx

Round 2

Reviewer 1 Report (Previous Reviewer 2)

The authors have addressed my concerns and can be accepted for publication. 

This manuscript is a resubmission of an earlier submission. The following is a list of the peer review reports and author responses from that submission.

Round 1

Reviewer 1 Report

 

It was interesting for me to examine the work of Jaime E. Catellanos-Bermejo and co-authors.

 

Strengths:

Very interesting and important study aiming to quantify the diagnostic efficacy of chest CT with a dual-reviewer approach compared with a conventional single-reviewer approach in patients with SARS- 64
CoV-2-associated pneumonia by its sensitivity, specificity, positive predictive value 65(PPV), and negative predictive value (NPV). The study well done with important conclusions.

I therefore suggest the acceptance after minor corrections (vide infra).

Weaknesses/Improvements:

1. The CO-RADS system is not clearly described , whereas I think it should be noted quickly what levels means what in this system, because analysing and looking at Tables 1,2 become difficult to understand for people who don't know this system by heart.

2. I would suggest to improve the description of inclusion criteria of group A nad B to be better understandable for ex.:

" The inclusion criteria for group A were that they were RT‒PCR (+) for SARS-CoV-2 with a (POSITIVE?) chest CT radiological report and included data from 482 patients"

Some sentences also need to be corrected for English grammatical mistakes -lines 71-74, "case definitions " in 113 sounds really impersonal etc....

Thank you.

 

Reviewer 2 Report

This study compared the diagnostic results of COVID based on CT images between dual-reviewers and a single reviewer. The database had two groups: A 488 patients with positive RT-PCR and B 408 patients with negative RT-PCR. They reported that dual-reviewers performed better than single-reviewer.

In my opinion, the objective of this study is unclear. The conclusion that “dual-reviewers performed better than single-reviewer” is too apparent to be worthy of proving. If the authors felt that their diagnostic method has something unique to report, they should emphasize them and rephrase the title of this paper.  At the present form, I do not recommend this paper for publication.   

 

Line 74: group B has 408 patients. The number “311” is inconsistent with the patient numbers in the Abstract. 

The English language is fine.