Prediction of Confusion Attempting Algebra Homework in an Intelligent Tutoring System through Machine Learning Techniques for Educational Sustainable Development

Round 1

Reviewer 1 Report

This paper use machine learning to classify the confused and non-confused students of online tutoring system which is an practical issues. However there are some points need to describe clearly. 

The section of related works need to describe the previous researches of mining tools and online tutoring. The writhing also need to improved in this section. There are too many paragraphs are only 2 or 3 lines. The description is not clearly in telling readers the development of online tutoring, the finding of online tutoring via mining tools, and the important of using classification tools to recognize confused students.  

The three variables are treated as predicted variable in line 216 is not described clearly about which three variables and how to combine. 

The basic description of feature variables and predict variables are missing.

The figure 5 results is hard to recognized by black and white print.

The accuracy can extend to the three or four digit after the decimal point to get the best performance algorithm. 

Author Response

Dear Reviewer,

Please find attached "Response to reviewer-1" file herewith for your kind reference and perusal. We are very thankful for your valuable comments and suggestions concerning our manuscript (ID: sustainability-400277).

We found these comments and suggestions are very helpful for revising and improving our article, as well as important guiding significance to our research. We have studied comments and suggestions carefully and thoroughly and have made corrections accordingly which we hope to meet with approval.

Moreover, revised portions are marked in the article with Microsoft track changes and we have again checked our manuscript by Grammarly Premium version for English language and style and we hope these changes are up to the mark and meet your standard.

With kind regards,

Author Response File: Author Response.docx

Reviewer 2 Report

Presented manuscript depicts the machine learning algorithms used to classify ASSISTment data.

major objections:

the presented problem is typical in machine learning and it is solved using typical and well known methods, widely described in literature. The methods used are standard, as well as presented pipeline, ie data preparation, data integration, feature selection, model training on train dataset

and model validation on testing dataset. The description of this process is too wide (lines 162-400), and in my opinion the description of standard methods is uncecessary.

For example it is not necessary to depict what is Decision Tree (DT) and Random Forest (RF).

Moreover, the important parameters of algorithm used are not given for the reader, for example number of trees in random forests or maximal depth in decision tree algorithm. Please focus rather on parameters.

The definition of ROC curves, AUC, confusion matrix and other parameters like accuracy, precision is unnecessary. Such definitions are widely available. Moreover, the same information is repeated, like list of classifiers used Table 3 and lines 269--324.

The second comment: the algorithms used have random part (e.g. RF draws attributes for each tree), therefore typically the average values from few runs should be depicts.

The third comment is about input dataset, there is no information, how many records are available, as well as how many attributes are available, how many attribute values were missing, how many examples are for each class, which attributes are generated etc. The ROC (line 421-422) suggests that there are not too much examples (the ROC is not smooth).

The used attributes are not summarized, what is helpful. When attribute names are used, for example in Figure 4, they make a fuss, the reader do not know, what this names means.

Next, it is standard to separate training and testing set. In manuscript it is not depicted in what percentage the input dataset was divided.

I have objections for regression analysis, why the estimation are given with seven decimal places. Are you sure you have such precision?

I suggest to make the dataset used freely available in supplementary materials.

minor comments:

Fig1. missing letter in 'Backward Elimination'.

Author Response

Dear Reviewer,

Please find attached "Response to reviewer-1" file herewith for your kind reference and perusal. We are very thankful for your valuable comments and suggestions concerning our manuscript (ID: sustainability-400277).

We found these comments and suggestions are very helpful for revising and improving our article, as well as important guiding significance to our research. We have studied comments and suggestions carefully and thoroughly and have made corrections accordingly which we hope to meet with approval.

Moreover, revised portions are marked in the article with Microsoft track changes and we have again checked our manuscript by Grammarly Premium version for English language and style and we hope these changes are up to the mark and meet your standard.

With kind regards,

Raza Abidi

Author Response File: Author Response.docx

Reviewer 3 Report

This paper proposed to learn a binary classifier to identify whether a student is confused in solving mathematical homework. The experimental data were obtained from ASSISTments, SKill-builder data. Authors applied data preprocessing for feature extraction and feature selection. Then seven machine learning models were applied to learn classifiers for this problem. Detailed analytical results were provided in comparing the performance of different models. The results supported that the student's confusion is identifiable by learned models and suggested that GLM, DT and RF have the best performance.

Though authors have provided comprehensive experiment results, I think there are still a few things to improve in convincing the readers. 

(1) There could be a concern that the result is dependent on the current data split. The data split can be decomposed into a few folds. So that there would be some variety in the results in comparing different models, which will reveal more information about the model performance. 

(2) Authors could provide some more information on the training process, e.g. parameter selection and the convergence of the training. It is possible that some models, like DL, have not yet converged so the precision is much less than the top three ones.

(3) The data preprocessing and the size of the data could greatly simplify the difficulty of the learning problem. So that the strength of a deep structure could not contribute to the data approximation. This could be the gap in applying to real-world data. Applying models to raw data might show some extra information on this problem.

(4) A minor concern is that the selection of the models. It will be better to provide the motivation of choosing these models. I think DT, RF and Gradient Boost are very similar. Also the structural designs of the models can impact the performance. It would be nice to provide more details and explain why.

Author Response

Dear Reviewer,

Please find attached "Response to reviewer-1" file herewith for your kind reference and perusal. We are very thankful for your valuable comments and suggestions concerning our manuscript (ID: sustainability-400277).

We found these comments and suggestions are very helpful for revising and improving our article, as well as important guiding significance to our research. We have studied comments and suggestions carefully and thoroughly and have made corrections accordingly which we hope to meet with approval.

Moreover, revised portions are marked in the article with Microsoft track changes and we have again checked our manuscript by Grammarly Premium version for English language and style and we hope these changes are up to the mark and meet your standard.

With kind regards,

Raza Abidi

Author Response File: Author Response.docx

Reviewer 4 Report

This paper has focused ASSISTments, an ITS in this study and scrutinized the skill-builder data using machine learning techniques and methods. The authors used seven candidate models that include: Naïve Bayes (NB), Generalized Linear Model (GLM), Logistic Regression (LR), Deep Learning (DL), Decision Tree (DT), Random Forest (RF), and Gradient Boosted Trees (XGBoost). The authors trained, validated and tested learning algorithms, performed stratified cross-validation and measured the performance of the models through various performance metrics i.e., ROC (Receiver Operating Characteristic), Accuracy, Precision, Recall, F-Measure, Sensitivity & Specificity.

(1)    What is the research gap? Please define it clearly in the introduction.

(2)    Line 116, should be put in the previous line.

(3)    Section 2 is suggested to list a table, comparing the advantages and disadvantages of all methods.

(4)    Why discretization is necessary? Can you use categorical variable here?

(5)    What types of statistical analysis did you use? Students t-test or Wilcoxon test? How do you set the confidence level? How did avoid Type-I error?

(6)    Did you test on multi-class classification problem?

(7)    Some related papers could be discussed, see “Dual-Tree Complex Wavelet Transform and Twin Support Vector Machine for Pathological Brain Detection” and “Detection of Left-Sided and Right-Sided Hearing Loss via Fractional Fourier Transform”.

Author Response

Dear Reviewer,

Please find attached "Response to reviewer-1" file herewith for your kind reference and perusal. We are very thankful for your valuable comments and suggestions concerning our manuscript (ID: sustainability-400277).

We found these comments and suggestions are very helpful for revising and improving our article, as well as important guiding significance to our research. We have studied comments and suggestions carefully and thoroughly and have made corrections accordingly which we hope to meet with approval.

Moreover, revised portions are marked in the article with Microsoft track changes and we have again checked our manuscript by Grammarly Premium version for English language and style and we hope these changes are up to the mark and meet your standard.

With kind regards,

Raza Abidi

Author Response File: Author Response.docx

Round 2

Reviewer 1 Report

Please check all citation is properly ex line98. 

Author Response

Dear Reviewer,

Please find attached "Response to reviewer-1" file herewith for your kind reference and perusal. We are very thankful for your valuable comments and suggestions concerning our manuscript (ID: sustainability-400277).

We found these comments and suggestions are very helpful for revising and improving our article, as well as important guiding significance to our research. We have studied comments and suggestions carefully and thoroughly and have made corrections accordingly which we hope to meet with approval.

Moreover, revised portions are marked in the article with Microsoft track changes for your kind attention.

With kind regards,

Raza Abidi

Author Response File: Author Response.docx

Reviewer 2 Report

The authors responded on my previous comments, however there are still issues to be improved.

Major objections

-----------------------

1) Every percentage results should be given with one decimal place only. It is enough to understood results and compare methods.

2) Do not repeat yourself. Table 1 repeats Section 2 (Related Work). If you provide Table 1 please remove repeated information from lines 116-184.

3) Table 2 is bad; either remove it or give valid values.

4) Table 3 as well as definition in lines 309-325 are well known, you could remove it from the manuscript.

5) You completely removed the names of used classifiers, therefore Figure 6, 7, 8 are not clear. Please give some description with model names.

To do this for each model acronym give model name (eg NB is Naive Bayesian) and all model parameters (line 389 is good place to do this, you should give the names of seven models and the model all parameters). Please give the library used for calculation.

6) Figure 5 is not clean. Please give the only average value in the plot.

7) In figure 6, 7, 8 please use box-plots.

8) Figure 2, 3, 4 is not clear. Please give a table where attributes are described.

Minor comments

-----------------------

Table 1: In my opinion columns,  'Advantages' and 'Disadvantages' should be wider, and reference to the article should be given;

maybe just give the reference instead of authors and publication year, therefore the first column could have methods used and reference.

Author Response

Dear Reviewer,

Please find attached "Response to reviewer-2" file herewith for your kind reference and perusal. We are very thankful for your valuable comments and suggestions concerning our manuscript (ID: sustainability-400277).

We found these comments and suggestions are very helpful for revising and improving our article, as well as important guiding significance to our research. We have studied comments and suggestions carefully and thoroughly and have made corrections accordingly which we hope to meet with approval.

Moreover, revised portion are marked in the article with Microsoft track changes and we have carefully checked and revised the English language by more proofreading and using Grammarly premium version (http://www.grammarly.com/), and we hope these changes are up to the mark and meet your standard.

With kind regards,

Raza Abidi

Author Response File: Author Response.docx

Reviewer 4 Report

Accept

Author Response

Dear Reviewer,

Please find attached "Response to reviewer-4" file herewith for your kind reference and perusal. We are very thankful for your valuable comments and suggestions concerning our manuscript (ID: sustainability-400277).

We have carefully checked and revised the English language by more proofreading and using Grammarly premium version (https://www.grammarly.com/). Revised portions are marked in the article with Microsoft track changes and we hope these changes are up to the mark and meet your standard.

With kind regards,

Raza Abidi

Author Response File: Author Response.docx