Evaluation of Feature Selection Methods on Psychosocial Education Data Using Additive Ratio Assessment

Round 1

Reviewer 1 Report

The authors investigate the influence of feature selection on the performance of machine learning. This topic is interesting and the paper is well written. Some major concerns are listed as follows:

1. It is not clear why more dimensions may lead to worse results. Please give more explanations.
2. According to the results given in Tabel 3, the performance after feature selection is slightly improved compared to the baseline. So is there a necessity to perform the feature selection? Try to give more analysis in results.
3. The quality of figures should be improved.

 

Author Response

Please see the attachment

Author Response File: Author Response.pdf

Reviewer 2 Report

This manuscript presents an interesting feature selection method and comparatively evaluates the advantages and limitations of seven different feature selection methods. Overall, the subject is interesting, the writing is easy to follow, and the plots/tables are appropriate. However, before possible publication, the reviewer has the following comments.

 

1. Section 1, Line 37: the author states “… in Google Scholar showed 212.000 results of …”. Is this number 212 or 212 000? If it is the latter, please change the decimal point to a comma to accurately reflect the number.

 

1. Equation (3): please revise the “f-ratio” such that the ratio is the subscript. In this case, the reader will not interpret that as “f minus ratio”. The corresponding text should also be revised.

 

1. Equation (9): what does a line on top of “0, m” and “1, n” mean? Please revise these two symbols to avoid ambiguity. Similar comments for all other equations that use a similar symbol.

 

1. Equations (10) and (13): what does the star symbol mean in these two equations? Please clarify.

 

1. Equation (18): the varible Ki represents the ranking of the alternatives. Is this correct? Please make it clear in the text.

 

1. Equation (22): this equation contains a mistake. Please fix it.

 

1. Section 3: the author indicates that the decision tree is adopted for the classification purpose (see Figure 2). The reviewer is curious about how Lasso is combined with the decision tree for classification and feature selection purposes? Could the author explain a bit?

 

1. Section 4: Table 2 states that only 10 features are selected based on the Lasso. In the reviewer’s opinion, the number of selected features for Lasso is highly dependent on the regularization parameter (i.e., the lambda). Does the author use cross-validation to determine the optimal lambda value? Or just assumed a value? This would make a difference for the feature selection result.

 

1. Sections 3 and 4: the author adopted an exhaustive search feature method for the feature selection. This method tries every possible combination of features and selects the one that yields the best performance. According to this definition, this feature selection method should generate the highest/best performance among all feature selection methods. The only drawback associated with this approach is its computational demand. However, the author’s result does not concide with this statement. Could the author explain this?

Author Response

Please see the attachment

Author Response File: Author Response.pdf

Reviewer 3 Report

Dear authors, 

congruts for your good work. 

Still the paper can be improved. I suggest to incoud the following references. More specificanly in the  related work: Please refer to educational social communities that can be formed and detected via AI detection methods 

1. Souravlas, S, Anastasiadou, S. Katsavounis, S. 2021.  A Survey on the Recent Advances of Deep Community Detection. Applied Sciences, 2021, 11(16), 7179.

After the "On the other hand, the success of artificial intelligence and big data influences deci- 39
sion-making perspectives, particularly those based on predictive problems. Big data can 40
handle data that has more large-scale amounts, more complex varieties, and higher dimensions......[5][6] you should add the following works 

2. Souravlas, S., Anastasiadou, S. Katsavounis, S., 2021. More on Pipelined Dynamic Scheduling of Big Data Streams. Applied Sciences 2021, 11(1), 61;

3. Souravlas, S., Anastasiadou, S. 2020. Pipelined Dynamic Scheduling of Big Data Streams. Applied Sciences. 10, 4796.

Regarding educational quality in the  related work  you should ad the folloeing work

4. Anastasiadou S., Zirinoglou P. 2015. Teachers’ attitudes toward Quality Assurance Dimensions regarding EFQM Model in Primary Education in Greece, Procedia Economics and Finance, vol. 33, pp. 411-431.

6. Anastasiadou S., Zirinoglou P. 2015. EFQM dimensions in Greek Primary Education System. Procedia Economics and Finance, vol 33, pp. 411 – 431.

7. Anastasiadou, S. Zirinoglou, P. 2020. Total Quality Management: Statistical Process Control and Quality Control’ Tools regarding Measuring Organizations’ Quality. The case of Pareto Chart. International Journal of Entrepreneurship and Innovative Competitiveness – IJEIC, Vol 2. Ι 2. URI: http://hdl.handle.net/11728/11540.

8. Anastasiadou, S. Zirinoglou, P. 2020. Statistical Process Control and Quality Control’ Tools regarding Measuring Organizations’ Quality: The case of Ishikawa's approach in Greek tertiary education system evaluation in respect of quality assurance International Journal of Entrepreneurship and Innovative Competitiveness – IJEIC, Vol 2. Ι 2. URI: http://hdl.handle.net/11728/11539.

9. Anastasiadou, S. Zirinoglou, P. 2020. Quality assessment in Greek Tertiary Education using Gap Analysis International Journal of Entrepreneurship and Innovative Competitiveness – IJEIC. Vol 2, Issue 1, http://hephaestus.nup.ac.cy/handle/11728/11531.

10. Anastasiadou, S., Taraza, E. (2020). Six Sigma in Tertiary Education: A Win of Change regarding Quality Improvement in Education, Proceedings of of 14th annual International Technology, Education and Development Conference (INTED2020), Valencia, Spain, pp. 9595-9601.

Author Response

Please see the attachment

Author Response File: Author Response.pdf

Reviewer 4 Report

The manuscript proposes a technique to evaluate the effectiveness of seven feature selection methods for the particular field of psychosocial education. They obtain a variable, but relatively small, amount of features, and are subsequently compared via classification to a baseline which uses all features. The authors conclude that their technique is appropriate to evaluate feature selection methods.

I personally find serious methodological problems in this manuscript:

• The authors imply that their research is aimed at the field of psychological education ("but research to determine the best method of feature selection in psychological education has not been established" in lines 18-19; "The experimental results showed that the ARAS is effective and promising to be used to evaluate and recommend the best feature selection method on psychosocial education data" in lines 23-24). It is not possible to draw such large-scale conclusions from just one dataset (hence also the lack of comparative statistical assessment). More datasets would be needed in order to test how generalizable the conclusions of this manuscript are.
• The experimentation lacks detail. Most prominently, the authors do not discuss at any point training/validation/test dataset splits nor on which of them the accuracy (and similar measures) are obtained. Even if they did it right, mentioning this information is essential for the reader to know if the results are methodologically correct.
• The results on the dataset the manuscript uses hardly serve to highlight the strengths of the proposed technique: the final ranking is completely dictated by the training time because all the other performance metrics are so similar across all compared alternatives that they become uninformative. I could predict the first and last positions just by looking at Table 4 for a few seconds. This dataset is simply not able to show how the proposal behaves when metrics with more inter-method variability are aggregated.

In addition, the writing has just enough quality to be able to understand the most relevant points. Copy-editing by a proficient English user would greatly improve readability. I also suggest a bit more familiarization with machine learning terminology in order to be more consistent when naming concepts; this would make reading the manuscript easier.

Finally, some figures to visually present the results would help convey the information. This manuscript still needs a significant amount of work (mainly in the experimental methodology) before it is publishable.

Author Response

Please see the attachment

Author Response File: Author Response.pdf

Round 2

Reviewer 1 Report

The authors have successfully addressed all my questions.

Author Response

Please see the attachment

Author Response File: Author Response.pdf

Reviewer 4 Report

The authors have addressed one of my main concerns about the generalization of their conclusions to the field of psychosocial education: their financial and time constraints prevent them from creating another comparable dataset, and the literature in the field does not have one with their requirements either. While I find this fair, I suggest that they be consistently clear throughout the manuscript in telling the reader that these conclusions cannot be extrapolated to the entire field yet, and that experiments in future related datasets are necessary to draw more general conclusions.

Another important issue was the lack of information about how the dataset was used. Particularly, dataset splits and their role in each part of the experimentation, as making a mistake here can produce invalid results. In this regard, the authors clarify that they have used a standard 80/20 (training/testing) split. I assume that feature selection was done exclusively with the training split, but this detail should be mentioned for transparency reasons. In addition, I have one more question: was the standardization of the data performed before splitting into training/testing, or after? Standardization with the whole dataset would be incorrect (and therefore the results technically invalid) as testing information would be leaked into the training of the model. If it was performed only with the training dataset, it is correct. Could you clarify this as well?

The authors have also included two new pictures (Figure 3) to illustrate the results of the experimentation. I agree with the conceptual separation between the accuracy-related metrics and the time-related metrics. However, I would like to make a couple of suggestions: (i) the legend in the left one is incomplete (some method names and colors are not shown); (ii) it would be better to put each picture in its own separate Figure to allow them to be bigger and thus easier to interpret.

Regarding the writing, I appreciate their check for typos. Nonetheless, I believe that a thorough improvement is still necessary. This would not only make reading easier, but it is also relevant because the authors state that they are the first to do a comparison of feature selection methods in that field of application. The researchers who enter the field after them should be able to learn something valuable from this paper, and this is facilitated by proper English grammar use and coherent, to-the-point sentences. Consider the following excerpt from lines 271-276: "After the preprocessing, the next step is the machine learning phase. The first step in this phase is to perform a features selection. This stage determines the selection of features based on an algorithm feature selection. For most features selection methods, the percentile of the estimator is used to limit the number of feature selections. The next step is to perform classification using the features obtained from the previous stage". Apart from some grammar mistakes, the expression "feature selection" is not only repeated multiple times in a few lines, but it is also used with different meanings: "feature selection" referring to a method to discard useless features, but also "feature selection" talking about concrete subsets of all the original features. Arbitrarily using the same words to refer to different concepts only makes reading harder.

Author Response

Please see the attachment

Author Response File: Author Response.pdf

Round 3

Reviewer 4 Report

The paper meets the necessary quality to be published.