Mammographic Classification of Breast Cancer Microcalcifications through Extreme Gradient Boosting

Round 1

Reviewer 1 Report

The manuscript entitled “Mammographic Classification of Breast Cancer Microcalcifications Through Extreme Gradient Boosting” has been investigated in detail. The topic addressed in the manuscript is potentially interesting and the manuscript contains some practical meanings, however, there are some issues which should be addressed by the authors:

1)      In the first place, I would encourage the authors to extend the abstract more with the key results. As it is, the abstract is a little thin and does not quite convey the interesting results that follow in the main paper. The "Abstract" section can be made much more impressive by highlighting your contributions. The contribution of the study should be explained simply and clearly.

2)      The readability and presentation of the study should be further improved. The paper suffers from language problems.

3)      The importance of the design carried out in this manuscript can be explained better than other important studies published in this field. I recommend the authors to review other recently developed works.

4)      What makes the proposed method suitable for this unique task? What new development to the proposed method have the authors added (compared to the existing approaches)? These points should be clarified.

5)      “Discussion” section should be edited in a more highlighting, argumentative way. The author should analysis the reason why the tested results is achieved.

6)      The authors should clearly emphasize the contribution of the study. Please note that the up-to-date of references will contribute to the up-to-date of your manuscript. The study named "Histogram based vehicle license plate recognition with KNN method" - can be used to explain the method in the study or to indicate the contribution in the “Introduction” section.

7)      The complexity of the proposed model and the model parameter uncertainty are not enough mentioned.

8)      It will be helpful to the readers if some discussions about insight of the main results are added as Remarks.

This study may be proposed for publication if it is addressed in the specified problems.

Author Response

1) In the first place, I would encourage the authors to extend the abstract more with the key results. As it is, the abstract is a little thin and does not quite convey the interesting results that follow in the main paper. The "Abstract" section can be made much more impressive by highlighting your contributions. The contribution of the study should be explained simply and clearly.

Response:

Thanks for your suggestion. We already revised our abstract and made it more clearly.

2) The readability and presentation of the study should be further improved. The paper suffers from language problems.

Response:

Thanks for your suggestion. We already revised our paper carefully and revised the language problems, making our paper more readable.

3) The importance of the design carried out in this manuscript can be explained better than other important studies published in this field. I recommend the authors to review other recently developed works.

Response:

Thanks for the comments. We have added the description of more related studies and the contribution of this paper in the section of related work.

4) What makes the proposed method suitable for this unique task? What new development to the proposed method have the authors added (compared to the existing approaches)? These points should be clarified.

Response:

The corresponding changes are presented now revised discussion section. Our experimental results show that our method obtains the best performance. This result demonstrates that it is essential for the classification of microcalcification to use the feature engineering method to choose the best composition of features.

5) “Discussion” section should be edited in a more highlighting, argumentative way. The author should analysis the reason why the tested results is achieved.

Response:

Thanks for the advice. We have analyzed the reason of results in the discussion section now.

6) The authors should clearly emphasize the contribution of the study. Please note that the up-to-date of references will contribute to the up-to-date of your manuscript. The study named "Histogram based vehicle license plate recognition with KNN method" - can be used to explain the method in the study or to indicate the contribution in the “Introduction” section.

Response:

Thanks for the advice. We have added the description of more related studies and the contribution of this paper in the section of related work and discussion.

7) The complexity of the proposed model and the model parameter uncertainty are not enough mentioned.

Response:

Thanks for your advice. The complexity of the proposed model are added to section 4.

8) It will be helpful to the readers if some discussions about insight of the main results are added as Remarks.

Response:

Thanks for your advice. Some discussions about insight of the main results are added to section 5.

Reviewer 2 Report

This is a very interesting paper with real application.

The authors evaluated several models for calcification on mammography with a large medical dataset. They have used various criteria to compare the proposed approach and they have compared the results. What is missing here, is a statistical test to compare the performance of all methods. I would suggest the authors compare the best method results with others using a well-known used statistical test, ‘’A Kolmogorov-Smirnov Based Test for Comparing the Predictive Accuracy of Two Sets of Forecasts’’. The test is very suitable for such analysis and with the R code available.

After this revision, the paper will be improved a lot. 

Author Response

The authors evaluated several models for calcification on mammography with a large medical dataset. They have used various criteria to compare the proposed approach and they have compared the results. What is missing here, is a statistical test to compare the performance of all methods. I would suggest the authors compare the best method results with others using a well-known used statistical test, ‘’A Kolmogorov-Smirnov Based Test for Comparing the Predictive Accuracy of Two Sets of Forecasts’’. The test is very suitable for such analysis and with the R code available.

Response:

Thanks for the helpful comments. We made the modification according to your suggestion, we have compared the paper you mentioned. As shown in Fig.4, we compared six popular techniques: kNN, decision tree, adaboostM1, RDF, GBDT, and XGBoost.  XGBoost model combining with clinical, mammographic, ultrasonographic, and histopathologic findings, assisted prognosis predication in patients with breast cancer, reaching an accuracy of 0.84 (95% confidence interval [CI], 0.76-0.91), an AUC of 0.93 (95% CI, 0.87-0.95). In comparison, with our composition of features, the accuracy, sensitivity and efficiency are improved significantly.

Reviewer 3 Report

The paper employed segmentation algorithms to extract calcification features from mammograms, including morphologic and textural features, and adopted extreme gradient boosting (XGBoost) to classify micro-calcifications in mammogram images, which indicate breast cancer. The results of the experimentations are presented.

Comments:

1.       The study used a well-known machine model (XGBoost)), which have been extensively used for various biomedical imaging tasks before, including the breast tumour recognition. What is the specific novelty and innovation of this study? Explicitly state your difference from previous studies.

2.       Improve introduction by discussing various modalities of images used in breast cancer research, including but not limited to thermal and ultrasound images.

3.       Did you consider the deep learning methods, which have been proven very successful for performing similar tasks? Add a discussion of available alternatives and their pros and cons.

4.       The related works section lacks of a systematic approach towards structuring and discussing the papers. You could start by discussing a plethora of available methods as identified by some systematic reviews on breast cancer recognition (such as, for example, Systematic Review of Computing Approaches for Breast Cancer Detection Based Computer Aided Diagnosis Using Mammogram Images) and discuss them as your related work. Then, clearly state your contribution to the body of knowledge with respect to the previous studies, such as „Breast cancer detection using mammogram images with improved multi-fractal dimension approach and feature fusion“. Finally, discuss the limitations of previous works as a motivation for your work.

5.       The mathematical description that starts from Line 69 does not belong to the related works section and should make a new section.

6.       The description of the methodology presented in Section 3 is very concise. You should extend it and discuss step-by-step in more detail. Illustrate by adding a workflow diagram of your approach.

7.       Present the confusion matrices. Discuss some examples of misclassification and the most likely reasons.

8.       How do you avoid/prevent the overfitting problem? Explain the cross-validation procedure in more detail.

9.       The difference between the performance of both models is very small. Is it statistically significant? Perform the statistical analysis of the results.

10.   Conclusions are rather weak. Provide more in-depth insights from the results of your review.

Author Response

1. The study used a well-known machine model (XGBoost)), which have been extensively used for various biomedical imaging tasks before, including the breast tumour recognition. What is the specific novelty and innovation of this study? Explicitly state your difference from previous studies.

Response:

Thanks for the helpful comments. One of the contributions of this study is to present the best composition of features for efficient classification of breast cancers. This paper finds out a way to select the best discriminative features as a collection to improve the accuracy. This study showed the highest accuracy (90.24%) for classifying microcalcifications with AUC=0.89.

So far, there are several researchers using XGBoost in breast cancer. Most of them are focused on electing the relevant biomarkers or histopathological images in predicting diagnosis and therapy by XGBoost[1-7]. Besides, XGBoost is also helpful in imaging in aiding the diagnosis of breast cancer. It has been reported that enhanced CT radiomics analysis by XGBoost predicated the efficacy of anti-HER2 therapy for patients with liver metastases from breast cancer[8]. The integration of Ensemble Learning methods within mpMRI radiomic analysis improves the performance of computer-assisted diagnosis of breast cancer lesions[9].Radiomics and machine learning based on PET/CT images help to predict HER2 status in breast cancer[10]. Similarly, Vu et al.[11] found that the XGBoost model combining with clinical, mammographic, ultrasonographic, and histopathologic findings, assisted prognosis predication in patients with breast cancer, reaching an accuracy of 0.84 (95% confidence interval [CI], 0.76-0.91), an AUC of 0.93 (95% CI, 0.87-0.95). In our study, we only employed XGBoost as a tool to discriminate among microcalcifications in mammograms automatically. We found that, with XGBoost, the main warning signs and even the only signs for breast cancer alone have low sensitivity and efficiency. In comparison, with our composition of features, the accuracy, sensitivity and efficiency are improved significantly.

2. Improve introduction by discussing various modalities of images used in breast cancer research, including but not limited to thermal and ultrasound images.

Response:

We thank you for your conscientious suggestions regarding supplying other modalities of images in introduction. Relevant description please see in revised introduction section.

3. Did you consider the deep learning methods, which have been proven very successful for performing similar tasks? Add a discussion of available alternatives and their pros and cons.

Response:

We respect reviewer’s concern here. We didn’t consider the Deep Learning methods because XGBoost has high accuracy and low false negatives, which is most essential in diagnosing cancer. Relevant description please see from paragraph 4 in revised discussion section. Due to the small dimension of input data, we compared six popular techniques: kNN, decision tree, adaboostM1, RDF, GBDT, and XGBoost.

4. The related works section lacks of a systematic approach towards structuring and discussing the papers. You could start by discussing a plethora of available methods as identified by some systematic reviews on breast cancer recognition (such as, for example, Systematic Review of Computing Approaches for Breast Cancer Detection Based Computer Aided Diagnosis Using Mammogram Images) and discuss them as your related work. Then, clearly state your contribution to the body of knowledge with respect to the previous studies, such as „Breast cancer detection using mammogram images with improved multi-fractal dimension approach and feature fusion“. Finally, discuss the limitations of previous works as a motivation for your work.

Response:

Thanks for the comments. We have added this part to the section of related work.

5. The mathematical description that starts from Line 69 does not belong to the related works section and should make a new section.

Response:

Thanks for the correction. We have revised this in the discussion section. The mathematical description that starts from Line 69 is about XGBoost and actually belongs to the related works.

6. The description of the methodology presented in Section 3 is very concise. You should extend it and discuss step-by-step in more detail. Illustrate by adding a workflow diagram of your approach.

Response:

Thanks for the comments. We have added the details in Section 3. In section 3 we mentioned many features, including autocorrelation, contrast, cluster prominence, cluster shade, dissimilarity, energy, entropy, homogeneity, maximum probability, the sum of squares, sum average, sum variance, sum entropy, difference variance, difference entropy, information measure of correlation, inverse difference normalized, and inverse difference moment normalized. These features can be easily obtained through APIs in MATLAB.

7. Present the confusion matrices. Discuss some examples of misclassification and the most likely reasons.

Response:

As shown in Fig.4, these ROC curves compare the discriminative performances of individual features versus combinations of features. The frequent occurred misclassification is in examples of images with blurred points that are often ignored by doctors. The most likely reason is that the images are not discriminative enough.

8. How do you avoid/prevent the overfitting problem? Explain the cross-validation procedure in more detail.

Response:

We avoid overfitting problems by using the optimal parameter in those methods like kNN, decision tree, adaboostM1, RDF, GBDT, and XGBoost. in the process of cross-validation, the original data sample is randomly divided into several subsets. The machine learning model trains on all subsets, except one. After training, the model is tested by making predictions on the remaining subset.

9. The difference between the performance of both models is very small. Is it statistically significant? Perform the statistical analysis of the results.

Response:

We compared six popular techniques: kNN, decision tree, adaboostM1, RDF, GBDT, and XGBoost. XGBoost model combining with clinical, mammographic, ultrasonographic, and histopathologic findings, assisted prognosis predication in patients with breast cancer, reaching an accuracy of 0.84 (95% confidence interval [CI], 0.76-0.91), an AUC of 0.93 (95% CI, 0.87-0.95).

10. Conclusions are rather weak. Provide more in-depth insights from the results of your review.

Response:

We have improved the conclusions now.

References

1. Ai H: GSEA-SDBE: A gene selection method for breast cancer classification based on GSEA and analyzing differences in performance metrics. PLoS One 2022, 17:e0263171.
2. Thalor A, Kumar Joon H, Singh G, Roy S, Gupta D: Machine learning assisted analysis of breast cancer gene expression profiles reveals novel potential prognostic biomarkers for triple-negative breast cancer. Comput Struct Biotechnol J 2022, 20:1618-1631.
3. Li Q, Yang H, Wang P, Liu X, Lv K, Ye M: XGBoost-based and tumor-immune characterized gene signature for the prediction of metastatic status in breast cancer. J Transl Med 2022, 20:177.
4. Jang JY, Ko EY, Jung JS, Kang KN, Kim YS, Kim CW: Evaluation of the Value of Multiplex MicroRNA Analysis as a Breast Cancer Screening in Korean Women under 50 Years of Age with a High Proportion of Dense Breasts. J Cancer Prev 2021, 26:258-265.
5. Jang BS, Kim IA: Machine-learning algorithms predict breast cancer patient survival from UK Biobank whole-exome sequencing data. Biomark Med 2021, 15:1529-1539.
6. Roy SD, Das S, Kar D, Schwenker F, Sarkar R: Computer Aided Breast Cancer Detection Using Ensembling of Texture and Statistical Image Features. Sensors (Basel) 2021, 21.
7. Chai H, Zhou X, Zhang Z, Rao J, Zhao H, Yang Y: Integrating multi-omics data through deep learning for accurate cancer prognosis prediction. Comput Biol Med 2021, 134:104481.
8. He M, Hu Y, Wang D, Sun M, Li H, Yan P, Meng Y, Zhang R, Li L, Yu D, Wang X: Value of CT-Based Radiomics in Predicating the Efficacy of Anti-HER2 Therapy for Patients With Liver Metastases From Breast Cancer. Front Oncol 2022, 12:852809.
9. Vamvakas A, Tsivaka D, Logothetis A, Vassiou K, Tsougos I: Breast Cancer Classification on Multiparametric MRI - Increased Performance of Boosting Ensemble Methods. Technol Cancer Res Treat 2022, 21:15330338221087828.
10. Chen Y, Wang Z, Yin G, Sui C, Liu Z, Li X, Chen W: Prediction of HER2 expression in breast cancer by combining PET/CT radiomic analysis and machine learning. Ann Nucl Med 2022, 36:172-182.
11. Vy VPT, Yao MM, Khanh Le NQ, Chan WP: Machine Learning Algorithm for Distinguishing Ductal Carcinoma In Situ from Invasive Breast Cancer. Cancers (Basel) 2022, 14.

Round 2

Reviewer 1 Report

All my comments have been thoroughly addressed. It is acceptable in the present form.

Author Response

Thanks a lot.

Reviewer 2 Report

The authors have answered other referees' comments and suggestions but not mine. 

I have asked for a statistical test clearly in my letter. The test will improve their paper and its results. 

in my view, the authors need to perform such a test to see if the results are statistically significant. 

Author Response

Response:

Thank you for the valuable feedback. We have made modifications following your suggestion. We have carried out the Kolmogorov-Smirnov Predictive Accuracy (KSPA) Test to compare the XGBoost model and other models. The experimental result (Table 5-7) indicated that there is indeed a significant statistical difference in the prediction errors between XGBoost and other models, and XGBoost has a lower random error rate. The detailed description is presented in the revised section of results and Table 5-7.

Reviewer 3 Report

The authors have addressed all my comments and revised the manuscript accordingly. The quality has improved.

I have only one minor comment:

- Some cells in table 3 are empty. All features should be explained.

Author Response

Response:

Thanks for your advice. All features remarks have been added to table 3, including morphologic and textural features.

Round 3

Reviewer 2 Report

I am ok with the revised version.