Machine Learning Approach for Chronic Kidney Disease Risk Prediction Combining Conventional Risk Factors and Novel Metabolic Indices
, Adama N. S. Bah 3, Karamo Bah 3, Chien-Yeh Hsu 4,5,* and Kuo-Chung Chu 4,6,*
Round 1
Reviewer 1 Report
This is an interesting work that combines conventional risk factors and novel metabolic indices as input for machine learning models for chronic kidney disease risk prediction.
- What is the meaning of IRB in section 2.1?
- Figure 2 shows that the imbalanced nature of data was considered, however, there is no reference in this section on how you have dealt with it. Only in section 3.3 is it said that SMOTE was used to deal with this issue, which I expected to see close to the explanation of Figure 2.
- Why have you balanced the testing data?
- Explain what you mean by offline in training and online in testing.
- Explain why AUC was used as the metric for evaluating the models instead of other popular metrics used for imbalanced data.
- Please detail the data preprocessing performed to avoid inaccurate decision-making by the models, as stated in section 3.2.
- Tables 1 and 2 have footers but no reference to them in the body of the tables.
- Section 3.3 Group predictors: Has the use of techniques, such as the Permutation Feature Importance or others, been considered to choose the best predictors?
- Tables 3, 4, and 5 need formatting according to journal standards.
- Figure 4: The text at the right of the confusion matrix is not necessary.
- It would interesting to see in the conclusion, the result of the evaluation achieved (the metric).
Author Response
Reviewer 1
This is an interesting work that combines conventional risk factors and novel metabolic indices as input for machine learning models for chronic kidney disease risk prediction.
- What is the meaning of IRB in section 2.1?
Respond 1: IRB is an abbreviation for institutional review board. We wrote the full name in the manuscript. Thank you
- Figure 2 shows that the imbalanced nature of data was considered, however, there is no reference in this section on how you have dealt with it. Only in section 3.3 is it said that SMOTE was used to deal with this issue, which I expected to see close to the explanation of Figure 2.
Respond 2: Thank you for that observation. The method applied to resolved the imbalanced data is explained in the method section (2.3).
- Why have you balanced the testing data?
Respond 3: Thank you for your question. We did not balance the testing data. Only the training data was balanced using SMOTE. What reflects in Figure 2 was an error. It is now resolved.
- Explain what you mean by offline in training and online in testing.
Respond 3: The offline in training and online in testing was adopted from previous literatures. However, those literature did not provide clear explanation of the meaning. The researchers only referred offline system as the training model and online system as testing model. We corrected it now. Thank you
- Explain why AUC was used as the metric for evaluating the models instead of other popular metrics used for imbalanced data.
Respond 4: Sorry, not only AUC was used to evaluation the models. We used several evaluation metrics (training & testing accuracy, sensitivity, specificity, and F-score values) to evaluate the performance of each model and was mentioned in section 3.3. line 362. We understood that it was not the clear in the method section but is now added. Thank you.
- Please detail the data preprocessing performed to avoid inaccurate decision-making by the models, as stated in section 3.2.
Respond 5: Thank you for your suggestion. We added more detail of the data preprocessing in section 2.3. section 3.2 only explained the relationship between the predictors and their target outcome (CKD) and comparison of
- Tables 1 and 2 have footers but no reference to them in the body of the tables.
Respond 6: Thank you for that observation. We corrected it.
- Section 3.3 Group predictors: Has the use of techniques, such as the Permutation Feature Importance or others, been considered to choose the best predictors?
Respond 6: Thank you for that question. We added more explanation how we choose the best predictors in Section 3.3.
The best predictors were identified by performing Cox regression and ROC curve analysis. The criteria applied was based on increased Hazard ratio, significance P-value <0.05 and review of previous literatures, which was further validated by the AUC (95%CI) as shown in Table 2.
- Tables 3, 4, and 5 need formatting according to journal standards.
Respond 7: The Tables (3, 4, and 5) are now represented according to the journal standards. Thank you
- Figure 4: The text at the right of the confusion matrix is not necessary.
Respond 8: The text is deleted. Thank you for that observation.
- It would interesting to see in the conclusion, the result of the evaluation achieved (the metric).
Respond 9: Thank you for that suggestion. We added the result of the evaluation achieved in the conclusion
Author Response File: 
Author Response.docx
Reviewer 2 Report
In this study, a machine learning-based CKD risk prediction model is developed. The prediction effects with different variables are analyzed. The work of this study is valuable for the early detection and timely treatment of chronic kidney disease. However, some issues need to be solved before publication.
1. Modeling variable selection is one of the keys to affecting the prediction effects of machine learning. As stated in the last paragraph of the Introduction, the goal of the study was to identify novel metabolic indices. And different subsets of metabolic indexes are used to establish the CKD risk prediction model, which aims to control the impute features and training time.
a) However, I do not think there is a clear difference between the four variables (all) and three variables (each subset) used for modeling. In other words, the creation of three subsets for prediction effects analysis doesn't make much sense.
b) What’s more, some algorithms have feature selection functions, such as RF. If more variables are used for modeling, will the model effect be better?
2. Why the model performance with creatinine is much better than that without creatinine? Maybe the variable creatinine is the most important influence factor for the risk of CKD prediction, while other variables are optional. Please explain it.
3. What are the structures of the established prediction models? As we all know, the structure of the model has a significant impact on prediction effects. For example, what are the numbers of layers and leaves in the RF algorithm?
Author Response
Reviewer 2
Comments and Suggestions for Authors
In this study, a machine learning-based CKD risk prediction model is developed. The prediction effects with different variables are analyzed. The work of this study is valuable for the early detection and timely treatment of chronic kidney disease. However, some issues need to be solved before publication.
- Modeling variable selection is one of the keys to affecting the prediction effects of machine learning. As stated in the last paragraph of the Introduction, the goal of the study was to identify novel metabolic indices. And different subsets of metabolic indexes are used to establish the CKD risk prediction model, which aims to control the impute features and training time.
- a) However, I do not think there is a clear difference between the four variables (all) and three variables (each subset) used for modeling. In other words, the creation of three subsets for prediction effects analysis doesn't make much sense.
Respond 1: Thank you for that observation. The ultimate goal of the study was to develop a simple, high-precision machine learning model to identify individuals at risk of developing CKD using the novel metabolic indices and some conventional risk factors with high predictive ability of CKD. The sentence ‘’ the goal of the study was to identify novel metabolic indices’’ has been modified. Moreover, with regards to the novel metabolic indices, the study wants to investigate which of the indices can better combine with the conventional CKD risk factors and provide better result. This what exactly control the input features.
- b) What’s more, some algorithms have feature selection functions, such as RF. If more variables are used for modeling, will the model effect be better?
Respond 2: Thank you for that question. Usually, the more variables you add to a ML model there is high possibility that the performance will be better. However, in some scenarios, this is not usually the case most of the time because after you input all the data in the model you will want to see which features contribute to the model effect. Those that do not literally contribute anything or correlated variables are usually removed in the preprocessing stage. In our case, when added all the variables the performance of the models was still better but the models were less stable and it also took longer time to train them.
In our study, we want to develop a simple and high precision machine learning model using fewer features and can equally predict CKD comparable to previously develop models which were trained using large of features. In reality, to develop model that depend on large scale of variables is sometimes not cost-effective and convenient for the patients. Sometimes, to obtain all data required for the model is not possible, especially when the medical exam is invasive, costly and not available. And, this will easily impose some shortcomings for the model to make accurate decision due the incomplete data.
- Why the model performance with creatinine is much better than that without creatinine? Maybe the variable creatinine is the most important influence factor for the risk of CKD prediction, while other variables are optional. Please explain it.
Respond 3: Thank you for your question. Generally, creatinine is one of the most important predictors of CKD and is use to calculate the estimated glomerular filtration rate (eGFR) for diagnosing CKD in the clinical settings. As a result, it has a great influence in the models providing the most excellent result. In Table 2, it can be evidenced that creatinine had the highest discriminatory ability with AUC of 0.909 (CI, 0.903-0.915).
- What are the structures of the established prediction models? As we all know, the structure of the model has a significant impact on prediction effects. For example, what are the numbers of layers and leaves in the RF algorithm?
Respond 4: Thank you for that question. In respond to the question, we have plotted the random forest tree for all the all the predictive subsets (A. B &C) to visualize the number layers and nodes. The RF Figures are in the supplementary file 1.
Author Response File: 
Author Response.pdf
Round 2
Reviewer 2 Report
I do not have more questions.
