A Weighted Ensemble Learning Algorithm Based on Diversity Using a Novel Particle Swarm Optimization Approach

Round 1

Reviewer 1 Report

The paper is written well and the approaches outlined using the two-stage weighted ensemble learning method produce interesting and significant results. I would recommend reducing the verbosity in Section 2 and creating a table with necessary notations and variables put in them. Also cite the following related papers: 1. Large, J., Lines, J. & Bagnall, A. A probabilistic classifier ensemble weighting scheme based on cross-validated accuracy estimates. Data Min Knowl Disc 33, 1674–1709 (2019). https://doi.org/10.1007/s10618-019-00638-y 2. Sengupta, S.; Basak, S.; Peters, R.A., II. Particle Swarm Optimization: A Survey of Historical and Recent Developments with Hybridization Perspectives. Mach. Learn. Knowl. Extr. 2019, 1, 157-191. 3. Shahhosseini, M.; Hu, G.; Pham, H. Optimizing Ensemble Weights and Hyperparameters of Machine Learning Models for Regression Problems. arXiv 2019, arXiv:1908.05287. An additional optional recommendation would be to compare and contrast your technique with other evolutionary optimization algorithms (same schema with DE/ACO/GA etc) using further experimentation or existing work in literature. That would lead to a more meaningful interpretation of the efficacy of the WELAD framework. Please make these changes and resubmit.

Author Response

"Please see the attachment

Author Response File: Author Response.docx

Reviewer 2 Report

This paper presents a novel ensemble scheme for classification problems. The algorithm proposed here is based on the PSO metaheuristic. The presentation of the methods is explanatory, yet some important details seem to be missing (or they may not be very explicitly presented).
While I appreciate the presentation in the paper, I identified several issues, that you will find below:

1. line 78, the phrase "In this paper, a novel ..." should be separated into a new paraghaph (the old paragraph split in two)
2. in the formal description of the WELAD algorithm, there is no mention with respect to the algorithm used to infer the individual learners. This is very important to mention at this point.
3. tWith respect to the grid search method used to identify the appropriate parameter values (line 292) - please specify the bounds for the grid search.
4. Please specify if AdaBoost and the other ensembles are used with the same individual learners (the trees) as the WELAD.
5. Why are the "regression trees" mentioned here (line 291), if only classification problems are attempted?
6. What parameters are used for the classification trees algorithm? Also, which specific type of algorithm is used to induce the decision trees (there exist several...ID3, C4.5...)?
7. I suggest that other performance measures apart from accuracy to be emplyed, because accuracy alone cannot describe a classifier. Also, it depends on the dataset - there exist specific features of data that suggest the appropriate performace measure to be used.
8. the best performances reported in table 2 should be tested for statistical significance. This is to say that statistical significance test results should be reported on each dataset.

9. According to Google Scholar, the correct reference for [1] is :

   Rokach, L. Ensemble Learning: Pattern Classification Using Ensemble Methods. World Scientific Publishing Co Pte Ltd, 2019.

10. Reference [15] was not published in 2014!!
    This is an old book, first published in 1993!!
    The paper should cite the correct reference.
    
    I urge the authors to verify all the cited references (I think they were mislead by the Google scholar citation, which showed the year 2014, probably the year that the book was indexed...)

Author Response

Please see the attachment.

Author Response File: Author Response.docx

Reviewer 3 Report

1. In Introduction, I think that the Authors should provide a more detailed review of related work, highlighting in particular how the cited works differ with respect to their original contributions in the present paper. This is so as to further explain to users who are not very familiar with this field, the purpose of appreciating the precise contribution made by this paper.
2. In my opinion, the bibliography and readability could be strengthened by providing some more recent examples of ensemble learning and machine learning in multiobjective optimization. In particular, the Authors could include and discuss:

(1)Single-cell RNA-seq interpretations using evolutionary multiobjective ensemble pruning. Bioinformatics, 2019, 35(16): 2809-2817.

(2) Evolutionary multiobjective clustering and its applications to patient stratification. IEEE transactions on cybernetics, 2018, 49(5): 1680-1693.

(3) Nature-inspired multiobjective epistasis elucidation from genome-wide association studies. IEEE/ACM transactions on computational biology and bioinformatics, 2018.

(4) Multiobjective patient stratification using evolutionary multiobjective optimization. IEEE journal of biomedical and health informatics, 2017, 22(5): 1619-1629.

(5) Elucidating genome-wide protein-RNA interactions using differential evolution. IEEE/ACM transactions on computational biology and bioinformatics, 2017, 16(1): 272-282.

3. There should be no symbols (, and .) after the equation, the Authors should check those equations carefully.
4. In Section 4.1.2, how did the Authors choose the final parameters? Are there any experimental results?
5. In Section 2.3, I would suggest to strengthen the bibliography by including the following works that other good applications on particle swarm optimization, filling a gap in the review of related literature included in the paper.

(1) Evolving Multiobjective Cancer Subtype Diagnosis from Cancer Gene Expression Data. IEEE/ACM Transactions on Computational Biology and Bioinformatics, 2020.

(2) A particle swarm inspired cuckoo search algorithm for real parameter optimization. Soft Computing, 2016, 20(4): 1389-1413.

6. The computational complexities or running time can be discussed since the proposed approach can be broadly applicable across multiple datasets .
7. Some sentences are hardly comprehensible. Therefore, I strongly suggest a hard revise of the English.

Author Response

Please see the attachment.

Author Response File: Author Response.docx

Reviewer 4 Report

Regarding the numerical experiments

• In Line 295, the authors state that “the training dataset is split by the K-fold cross-validation method such that K=7.” while in Line 303 the authors state “each algorithm in the dataset is tested by 10 cross-validation”. I believe that the first one (7-Cross Validation) is used for the development of the ensemble components. However, it is not clear.
• The authors should use more performance metrics such as F1, AUC, Recall, Precision etc
  Accuracy is not safe to provide convincing arguments regarding the performance of an algorithm.

 

In the 1^st paragraph of Section 1, an interesting collection/book provided by Algorithms MDPI https://www.mdpi.com/1999-4893/13/6/140 should be adequately cited.

 

The recent work [on ensemble techniques of weight-constrained neural networks. Evolving Systems, 1-13] which presents in detail the advantages and properties of voting, bagging and boosting should be properly presented in the Introduction.

 

The authors should include some works regarding “weighted ensemble strategies” proposed in the literature.

 

Some minor typos. Spell check required

Author Response

Please see the attachment.

Author Response File: Author Response.docx

Round 2

Reviewer 2 Report

I am satisfied with the additions to the paper; they touched the previous issues outlined with in the earlier review phase.

 

Reviewer 4 Report

The authors address the previous comments.

I can recommend the publication of the manuscript.

This manuscript is a resubmission of an earlier submission. The following is a list of the peer review reports and author responses from that submission.