Automation Radiomics in Predicting Radiation Pneumonitis (RP)

Round 1

Reviewer 1 Report

The authors have presented a very interesting research topic to explore gains in predictive model performance for Radiation Pneumonitis. Here are some of the major concerns which must be addressed before publication:

1.       In the abstract section, authors should mention the outcomes/results. Which classifier gives higher classification outcomes?

2.       The source of data is missing. Which hospital or laboratory has given access of data to  perform this study?

3.       Authors must present some description about SMOTE.

4.       Which features were acquired from the given data for the classification purpose?

5.       There is no information about GLCM, GLRLM, and Laws texture energy measure. Add some information and compare difference between these features extraction techniques.

6.       In Model Evaluation Section, the authors didn’t mention about data labelling. Similary, the label of Fig 2 needed to be improved. Authors must mention the labelling of two classes. Class 0 and class 1 represents which data. The confusion matrix can be presented more descriptively and drawn prettily. 

7.       Reference is required for the description of “Performance Metrics”. Here are some of the relevant references that can be studied and used:

    https://doi.org/10.1016/j.saa.2022.122206

8.       The authors should modify the heading by “Performance Metrics” instead of “ Our Performance Metrics”.

9.       The number of dataset and number of features are missing. How authors created a confusion matrix and the number of values in confusion matrix is very less. 

Minor revisions required. 

Author Response

Please see the attachment.

Author Response File: Author Response.pdf

Reviewer 2 Report

The paper „Automation Radiomics in Predicting Radiation Pneumonitis (RP)” describes a CT-based Radiomic study for predicting Pneuomotis after radiation therapy due to lung cancer.

The paper is well-written, with a solid analysis. The findings and conclusions and nicely within the results from the experiments, something not common for this kind of paper. Overall, the paper made a solid impression with robust research.

The major drawbacks are the rather limited novelty in this highly investigated research field and the organization of the paper.

To further improve the paper, I suggest to address the following points:

1.           The organization of the paper is rather odd and makes understanding the paper difficult. For example, SMOTE is introduced in section 2 “Data Collection”. Similarly, the feature extraction tool PyRadiomics was introduced in the result section 3.1. It would significantly improve the reading flow to have proper headlines and better organization.

2.           Maybe I missed the information, but it would be important to describe the used data in more detail. How many patients/images were finally used? What is the distribution of important cofounders (Age, Sex, Weight, Smoking)? What are the Imaging parameters (Resolution, Reconstruction kernel, Energy…)?  

3.           The DNN approach needs a better description. Given the flexibility of Deep Learning, the given details are not sufficient to reproduce the results. What layers had been used (Type, Size)? What is the training method (SGD or ADAM or… ) and which initialization was chosen?  Etc.. etc...

4.           SMOTE is usually applied to tabular data. Please clarify how it was used in combination with the DNN that relies on images as input.

5.           Please clarify if any augmentations were used during the training of the DNN? If yes, please state which. If not, please justify why not even simple augmentations like rotation or flipping were applied.

6.           Feature Importance is used as a feature selection method. It is well-researched that the best feature selection method also depends on the ML algorithm. Did you investigate alternative approaches, especially for the SVM?

7.           Please show the ROC-curves for the results, especially since AUC-ROC is reported.

8.           Figure 2 seems to be oddly formatted.

9.           Could you please rationalize the decision to use R for the statistical analysis? Feedback within the review process is enough, I am just curious.

From my point of view, the paper is solid research without major flaws. However, currently, there are clear points that need improvement. Consequently, I suggest a major revision to allow the authors to address these points. I am looking forward to reading the revised paper. 

Author Response

Please see the attachment.

Author Response File: Author Response.pdf

Round 2

Reviewer 1 Report

The authors have carefully revised the manuscript. 

However, in figure 2, there is a gap between figure and label. That should be removed before publication.

Author Response

The required changes have been implemented.

Reviewer 2 Report

With their review, the authors addressed some of my previous points. However, the real points were not addressed and there is still need for further improvement before the paper is ready for publication:

1.           Clarity and organization: This is still a real issue with the paper and I think that a major redesign is necessary. Information about the experimental design, including the data, the classification algorithms etc.. should be described in the section “methods” and not in the “results and discussion” section. Right now, the reader has to look at different sections in the paper to get all information.

2.           Related to 1: Please check the language of the paper carefully and adapt it to the scientific standard. For example, the sentence “The forthcoming iteration of the manuscript furnished a thorough exposition of the strata employed in our deep neural network.” Does not meet the standard of scientific English.

3.           I acknowledge that the authors added a paragraph about the data confounder. However, it is not sufficient to write that confounding parameters like sex, weight etc… have been gathered, but they must also be reported. I would advise to use a table for this. This is important since sometimes these confounding factors are actually better predictors than the novel designed biomarkers and reporting them allows the reader to judge that these wasn’t the case for the given study.

4.           Thanks for adding the description of the DNN architecture. It is sufficient. If wanted, it might be further improved by giving a figure showing the network graph/structure/architecture.

5.           Thanks for the answer regarding the combination of SMOTE and the DNN approach. I am aware that it is possible to use SMOTE on images, but there are different approaches for this and it would be important to describe the actual movement. However, do I understand it correct that this was not the case in this paper?

6.           Related to my previous point: Right now, the paper gives the impression that the DL-approach is based directly on the images. Especially fig. 1 leads to this impression. This should definitely be clarified and corrected in the paper.

7.           My previous point about the augmentations is based on a direct usage of images for the DL-approach. This is not the case, right? Please clarify.

8.           I agree that a full study on feature selection might be out of scope for this paper. However, not doing so leads to some significant limitations. For example, the found set of features might not be optimal. Also, the performance of individual approaches can be suboptimal. The limitations should be discussed in the paper.

Given these points, I suggest another major revision of the paper.   

The language needs to improve. Both in terms of structuring the text which does not follow good english writing as well as in terms of wording. 

The text is good in terms of grammar and spelling. 

Author Response

Please see the attachment

Author Response File: Author Response.pdf

Round 3

Reviewer 2 Report

No further comments

English needs significant improvements. Especially the usage of words, and the proper writing tone.

Author Response

Thank you for this suggestion. We revised the text according to your suggestions and Academic Editor Notes. In particular, we have added a clear definition and explanation of the term "Automation Radiomics" in the Introduction section to clarify its usage in our paper. We acknowledge that this term may not be widely established in the field and have provided a brief explanation of its intended meaning within the context of our study. Additionally, we have emphasized the automated aspects of our data analysis and prediction process in the Methods section. We highlight the use of advanced computational techniques, machine learning algorithms, and automated feature extraction to streamline the analysis workflow and minimize manual intervention. We believe that these revisions will provide a better understanding of the automated nature of our study and the term "Automation Radiomics." In addition, we have made the necessary revisions to clarify the nature and purpose of our paper. We acknowledge the confusion that may have arisen due to descriptions in different sections of the paper. To address this issue, we have updated the Introduction and Methodology sections to provide a clear and concise statement about the paper's intention and type. Specifically, we have specified that this paper is a research paper aimed at presenting original research findings in the field of Automation Radiomics for predicting Radiation Pneumonitis. We have provided a brief explanation of how Automation Radiomics refers to the automated analysis and prediction of radiomic features using advanced computational techniques. These revisions help to clarify the paper's purpose and nature, ensuring that readers understand it as a research paper presenting original research findings in the field of Automation Radiomics. In conclusion, based on the findings of this study, there is a promising opportunity to develop a viable application that automates the prediction of radiation pneumonitis. An automated prediction application would provide numerous benefits, including enhanced clinical decision-making efficiency and precision. By reducing the manual effort required for analyzing medical images and extracting radiomic features, healthcare professionals would be able to make real-time predictions, allowing for opportune interventions and individualized treatments. To validate the developed model and translate it into an application that can seamlessly integrate with existing clinical workflows, additional research is required.

Finally, we want to thank you again as all your comments have significantly improved our manuscript. See the attachment file.

Author Response File: Author Response.docx