Reproducibility in Radiomics: A Comparison of Feature Extraction Methods and Two Independent Datasets

Round 1

Reviewer 1 Report

Sometimes it is very difficult for the doctor to review the radiomics reports of the patients except the experienced doctor, there are still some faults because of the lack of experience or special cases of the patients. The treatments need the very accuracy judgements. The special meeting is often needed by organizing the experience doctors to analyze the special cases of the treatments for the special patients. To decrease the manmade fault, one needs the special machine to do the accuracy validation. Nowadays AI can help the mankind to do so. This manuscript has shown the possible solution. As the authors conclude in the final section ‘Radiomics features reported as stable were analyzed for reproducibility using RIDER lung dataset with. 29 of 43 features found to be reproducible to changes in the feature extraction toolkits when intensity threshold was applied maintaining stable rank ordering (rs > 0.8) and are recommended for use for biomarker analysis.’’ Useful radiomics features should be selected based on reproducibility. This study identified a set of features that meet this requirement and validated the methodology for evaluating reproducibility between datasets.’

I agree the authors’ conclusions. The methods used in this manuscript could be expanded to all area. The point is we need the machines to learn more cases and to get more information , create the different standards for the different cancer and different parameters. This paper could be published as an example for the medical development after the small art works.

1, we need more clear from Fig.1 to Fig.3, figure captions are ok

2, it is better for the authors to prepare the index at the end of document for the abbreviation words such as GUQ, IUQ, GLCM, GLDM, GLRLM，GLZSM，NGTDM,…, and so on, the reader might not understand them so well.

3, we need a criterion to validate the exact situation of image. Such a standard could be used for all the medical images got by different scanner and for different cancers. I would like to ask for the authors to give the general methods to describe the way to review the images. The reproducibility is just one of the parameters. Do we have any other parameters which can support the machine (AI) to do the right choice?

Author Response

Point 1:

We need more clear from Fig.1 to Fig.3, figure captions are ok

Response 1: Figure clarity has been improved, as suggested.

 

Point 2:

It is better for the authors to prepare the index at the end of document for the abbreviation words such as GUQ, IUQ, GLCM, GLDM, GLRLM，GLZSM，NGTDM,…, and so on, the reader might not understand them so well.

Response 2: Abbreviation list has been included at the end of the manuscript.

 

Point 3:

We need a criterion to validate the exact situation of image. Such a standard could be used for all the medical images got by different scanner and for different cancers. I would like to ask for the authors to give the general methods to describe the way to review the images. The reproducibility is just one of the parameters. Do we have any other parameters which can support the machine (AI) to do the right choice?

Response 3: We agree with the reviewer that reproducibility is only one of the parameters. We have expanded the discussion to included methods to review the images as a precursor to radiomics studies plus suggestions from the literature.

Author Response File: Author Response.docx

Reviewer 2 Report

Interesting and necessary topic. Good presentation.

Please improve the resolution of figures 1, 2, 3. 

In line 362, number 9 is, probably, the refference - must be written accordingly.

Author Response

Point 1:

Interesting and necessary topic. Good presentation.

Response 1: Thank you.

 

Point 2:

Please improve the resolution of figures 1, 2, 3.

Response 2: Figure clarity has been improved, as suggested for all figures.

 

Point 3:

In line 362, number 9 is, probably, the reference - must be written accordingly.

Response 3: The reference has been corrected.

Author Response File: Author Response.docx

Reviewer 3 Report

Comments:

1.     The resolution of the figures is too low, and many words cannot be seen clearly. As a result, I am unable to assess or justify their conclusions based on these blurry figures.

2.     Being a data analysis paper, it is essential for the authors to provide the code used in these analyses. This allows readers to replicate and verify the results using their own code. How, I did not find any mention of code information in the manuscript or in the "Code Availability Statement."

3.     Regarding Table 1, it is necessary for the authors to provide the exact values of Spearman's correlation coefficient for the features listed.

4.     In the Discussion section, it is important for the authors to allocate more words to thoroughly discuss the contributions and limitations of this study within the field.

5.     The two paragraphs from line 185 to 192 look weird. They may appear to be unrelated to this study and seem out of place.

6.     [Line 114] “RIDER dataset consists of 31 non-contrast enhanced PET/CT images” and [line 117] “Each patient had a repeat scan 15 minutes after the first scan using the same scanner and imaging protocol”   Since each patient has two scans, it seems peculiar that the overall image count would be an odd number.

7.     [Line 147-151] “Using Pyradiomics a total of 103 features were extracted …”   The summation of the eight types is “18+14+14+22+16+16+5=105”, not 103. Please double-check for any typos or clarify the 103 features more explicitly.

8.     Please provide the full name of all abbreviations in manuscript when they are first mentioned. Such as “SABR” in line 116.

9.     The manuscript should undergo editing by someone expertise in technical English, with a specific focus on improving grammar, sentence structure, and clarity. This will ensure that the goals and results of the study are effectively communicated to the reader.

Comments for author File: Comments.docx

1.     The manuscript should undergo editing by someone expertise in technical English, with a specific focus on improving grammar, sentence structure, and clarity. This will ensure that the goals and results of the study are effectively communicated to the reader.

Author Response

Point 1:

The resolution of the figures is too low, and many words cannot be seen clearly. As a result, I am unable to assess or justify their conclusions based on these blurry figures.

Response 1: Figure clarity has been improved for all figures.

Point 2:

Being a data analysis paper, it is essential for the authors to provide the code used in these analyses. This allows readers to replicate and verify the results using their own code. How, I did not find any mention of code information in the manuscript or in the "Code Availability Statement."

Response 2: The code for Matlab and Pyradiomics feature extraction have been included in appendices.

 

Point 3:

Regarding Table 1, it is necessary for the authors to provide the exact values of Spearman's correlation coefficient for the features listed.

Response 3: Spearman correlation coefficient values have been included for the individual tests in Table 1.

 

Point 4:

In the Discussion section, it is important for the authors to allocate more words to thoroughly discuss the contributions and limitations of this study within the field.

Response 4: The discussion has been expanded to further discuss the contributions and limitations of this study.

 

Point 5:

The two paragraphs from line 185 to 192 look weird. They may appear to be unrelated to this study and seem out of place.

Response 5: The unrelated information has been removed.

 

Point 6:

[Line 114] “RIDER dataset consists of 31 non-contrast enhanced PET/CT images” and [line 117] “Each patient had a repeat scan 15 minutes after the first scan using the same scanner and imaging protocol” Since each patient has two scans, it seems peculiar that the overall image count would be an odd number.

Response 6: This sentence has been rephrased for clarity.

 

Point 7:

[Line 147-151] “Using Pyradiomics a total of 103 features were extracted …” The summation of the eight types is “18+14+14+22+16+16+5=105”, not 103. Please double-check for any typos or clarify the 103 features more explicitly.

Response 7: Thanks for picking up the typo. It has been corrected. The numbers now add up.

 

Point 8:

Comment : Please provide the full name of all abbreviations in manuscript when they are first mentioned. Such as “SABR” in line 116.

Response 8: All abbreviations are expanded at first reference throughout the study. An abbreviation index has been included.

 

Point 9:

The manuscript should undergo editing by someone expertise in technical English, with a specific focus on improving grammar, sentence structure, and clarity. This will ensure that the goals and results of the study are effectively communicated to the reader.

Response 9: The manuscript has been read and edited by several native English speakers including those in the author list. This includes the senior co-authors (Prof Philip Evans and Prof Andrew Nisbet) to ensure clarity of the technical language.

 

Comments on the Quality of English Language

Point 10:

The manuscript should undergo editing by someone expertise in technical English, with a specific focus on improving grammar, sentence structure, and clarity. This will ensure that the goals and results of the study are effectively communicated to the reader.

Response 10: The manuscript has been read and edited by several native English speakers including those in the author list. This includes the senior co-authors (Prof Philip Evans and Prof Andrew Nisbet) to ensure clarity of the technical language.

Author Response File: Author Response.docx

Round 2

Reviewer 3 Report

The authors have addressed my concerns