Optimizing the Parameters of Long Short-Term Memory Networks Using the Bees Algorithm
Round 1
Reviewer 1 Report
Overall the work is clear to read. English can be improved.
The proposed approach optimizes the learning rates of the LSTM modules via a nature-inspired algorithm. Thus, I would not write that the LSTM topology is optimized (see Abstract).
Experimental validation should be improved.
The experiments are conducted on very few numbers of image sequences. No cross-validation is conducted.
The results obtained by the proposed approach seem very similar to that of the baseline (CNN+LSTM). Sometimes, using a CNN alone (i.e., with no recurrent network) gives similar results, too. When is the performance gap (i.e., the difference) statistically significant?
Some related works should be cited and the difference w.r.t. the proposed approach discussed. Some examples are:
- Qureshi, Ayyaz-Ul-Haq, et al. "RNN-ABC: A new swarm optimization based technique for anomaly detection." Computers 8.3 (2019): 59.
- Zeybek, Sultan, et al. "An improved bees algorithm for training deep recurrent networks for sentiment classification." Symmetry 13.8 (2021): 1347.
- Kumar, Raghavendra, Pardeep Kumar, and Yugal Kumar. "Integrating big data driven sentiments polarity and ABC-optimized LSTM for time series forecasting." Multimedia Tools and Applications 81.24 (2022): 34595-34614.
- Jacob, Minu Susan, and P. Selvi Rajendran. "Fuzzy artificial bee colony‐based CNN‐LSTM and semantic feature for fake product review classification." Concurrency and Computation: Practice and Experience 34.1 (2022): e6539.
Author Response
Dear,
Thank You for your comments,
Please find the following responses:
- Reviewer Comment: The proposed approach optimizes the learning rates of the LSTM modules via a nature-inspired algorithm. Thus, I would not write that the LSTM topology is optimized (see Abstract).
Response: I changed it in the abstract to “Improving the performance of deep learning (DL) algorithm is a challenging problem, Long Short-Term Memory (LSTM) is one of the DL networks that deal with time series or sequence data. This paper attempts to overcome this problem by optimizing LSTM parameters using the Bees Algorithm (BA)”.
- Reviewer Comment: Experimental validation should be improved. The experiments are conducted on very few numbers of image sequences. No cross-validation is conducted.
Response: The datasets are shuffled at every epoch in order to minimize the data biases and improve experiments validation as mentioned in at the beginning of section 4 in page 13.
- Reviewer Comment: The results obtained by the proposed approach seem very similar to that of the baseline (CNN+LSTM). Sometimes, using a CNN alone (i.e., with no recurrent network) gives similar results, too. When is the performance gap (i.e., the difference) statistically significant?
Response: As can be seen from table 6 in page 16, adding the LSTM network to CNN improved the prediction accuracy in all training, validation and testing sets. The testing accuracy is increased by 8% in the testing set to be 93.17%. The CNN-LSTM algorithm is further developed by adding BA to optimize LSTM parameters which increased the prediction accuracy in the validation and testing sets from 95.67% and 93.17% to 96.33% and 95.5% respectively, so the improvement in the testing set is 2% from the CNN-LSTM algorithm and 10% from BA-RCNN algorithm.
- Reviewer Comment: Some related works should be cited and the difference w.r.t. the proposed approach discussed. Some examples are:
- Qureshi, Ayyaz-Ul-Haq, et al. "RNN-ABC: A new swarm optimization based technique for anomaly detection." Computers3 (2019): 59.
- Zeybek, Sultan, et al. "An improved bees algorithm for training deep recurrent networks for sentiment classification." Symmetry8 (2021): 1347.
- Kumar, Raghavendra, Pardeep Kumar, and Yugal Kumar. "Integrating big data driven sentiments polarity and ABC-optimized LSTM for time series forecasting." Multimedia Tools and Applications24 (2022): 34595-34614.
- Jacob, Minu Susan, and P. Selvi Rajendran. "Fuzzy artificial bee colony‐based CNN‐LSTM and semantic feature for fake product review classification." Concurrency and Computation: Practice and Experience1 (2022): e6539.
Response: Thank you for sending the studies, I added them to in addition to other studies to section 2.3 in pages 5 and 6 and discussed in the results section as well in section 4.1 in page 16.
I attached the revised manuscript with highlighted changes.
Thank You
Author Response File: 
Author Response.docx
Reviewer 2 Report
As I reviewed the manuscript entitled " Optimizing the Parameters of Long Short-Term Memory Network using Bees Algorithm" in detail. In this article, Bees algorithm is used to optimize the parameters of LSTM. I found a some flaws in the paper that should be fixed to meet the high standard of the Journal as well as the research community The presented work is good. However, some concerns need to be resolved, in the next revision, which is given as follows:
1. Motivation behind this work need to be included in introduction section.
2. Specify the contributions clearly.
3. The quality of figure 1 and 5 is not good. I suggest to redraw the figure 1 and 5.
4. In section 4, specify the system configurations used for implementing this work.
5. Some grammatical mistakes in the manuscript, it should be removed, and scientific language needs to be used to meet the higher standard of the journal.
Author Response
Dear,
Thank You for your comments,
Please find the following responses:
- Reviewer Comment: Motivation behind this work need to be included in introduction section.
Response: I added the motivation behind the work in the introduction in page 2.
- Reviewer Comment: Specify the contributions clearly.
Response: The contribution is shown in the introduction in page 2. The contribution of the paper is “Improving the performance of LSTM network in predicting sequence data using BA. As the input data are images, CNN is added to extract the image features yielding a hybrid algorithm (BA-CNN-LSTM) that provides a more accurate prediction of porosity percentage appearing in sequential layers of artificial porosity images that mimic CT scan images of parts manufactured by SLM process”.
- Reviewer Comment: The quality of figure 1 and 5 is not good. I suggest to redraw the figure 1 and 5?
Response: I drew figure 1 in page 4 and figure 5 in page 10 to improve the figures quality.
- Reviewer Comment: In section 4, specify the system configurations used for implementing this work.
Response: I added the system configuration at the beginning of section 4 in page 13.
- Reviewer Comment: Some grammatical mistakes in the manuscript, it should be removed, and scientific language needs to be used to meet the higher standard of the journal.
Response: I revised the whole manuscript to improve the language.
I attached the revised manuscript with highlighted changes.
Thank You
Author Response File: Author Response.docx
Reviewer 3 Report
- In the abstract, it is not clear what was the position of the proposed algorithm in the structure of the LSTM problem? The abstract is focused on the algorithm and not the challenging problem raised in the title!!!
- Different aspects of primary motivations, innovation, and contributions of the manuscript should be listed in the introduction.
- Sec. 1 should include all related research from 2020 to 2023. It should finally be reviewed, summarized, and presented in the form of a comprehensive table in terms of all specialized indicators (LSTM and optimization).
- Sec. 3 is very brief and incomplete! Mathematical formulation and mechanization of the problem should be fully presented and explained. In this section, valid references and problem's challenges should be well stated.
- Why is the proposed algorithm improved? Why is the ability of other powerful algorithms not used for the problem or for comparison? Such as gravitational search algorithm, inclined planes system optimization, etc
- The final comparative results should be reported numerically and statistically in the abstract.
Author Response
Dear,
Thank you for your comments,
Please find the following responses:
- Reviewer Comment: In the abstract, it is not clear what was the position of the proposed algorithm in the structure of the LSTM problem? The abstract is focused on the algorithm and not the challenging problem raised in the title!!!
Response: I added the position of the proposed algorithm in the structure of the LSTM problem in the abstract.
- Reviewer Comment: Different aspects of primary motivations, innovation, and contributions of the manuscript should be listed in the introduction.
Response: The contribution is shown in the introduction in page 2. The contribution of the paper is “Improving the performance of LSTM network in predicting sequence data using BA. As the input data are images, CNN is added to extract the image features yielding a hybrid algorithm (BA-CNN-LSTM) that provides a more accurate prediction of porosity percentage appearing in sequential layers of artificial porosity images that mimic CT scan images of parts manufactured by SLM process”. Also, I added the motivation and innovation in the introduction in page 2.
- Reviewer Comment: 1 should include all related research from 2020 to 2023. It should finally be reviewed, summarized, and presented in the form of a comprehensive table in terms of all specialized indicators (LSTM and optimization).
Response: I added additional state of the art studies in section 2.3 in page 5 and 6.
- Reviewer Comment: 3 is very brief and incomplete! Mathematical formulation and mechanization of the problem should be fully presented and explained. In this section, valid references and problem's challenges should be well stated.
Response: I added the mathematical equations, explanations, problem mechanism and problems challenges in section 3.2 in page 8.
- Reviewer Comment: Why is the proposed algorithm improved? Why is the ability of other powerful algorithms not used for the problem or for comparison? Such as gravitational search algorithm, inclined planes system optimization, etc.
Response: Bees algorithm is better in dealing with high dimensional objective function in regression problems. I compared the proposed algorithm in the paper with existing algorithm that uses Bayesian Optimization to optimize the parameters of LSTM network. In addition, I compared it with the traditional LSTM that uses stochastic gradient descent with momentum to train LSTM network as shown in section 4.
- Reviewer Comment: The final comparative results should be reported numerically and statistically in the abstract.
Response: I reported the comparative results in the abstract.
I attached the revised manuscript with highlighted changes.
Thank You
Author Response File: Author Response.docx
Round 2
Reviewer 1 Report
The authors answered most of my questions. However, two issues have not been addressed yet. This should be done before reconsidering this manuscript for publication.
1. Statistical significance of the results
Since the data sets have few samples, the statistical significance interval should be computed and reported in the manuscript.
2. Cross-validation
The authors shuffled the training samples during the training procedure. This is not equivalent to a cross-validation procedure in which the same method is trained several times after swapping samples between the training and the validation set. By relying on the average results, obtained with different training and validation sets, a much more reliable performance evaluation is obtained.
Author Response
Dear,
Thank you for your comments,
Please find the following responses:
- Reviewer Comment: Statistical significance of the results. Since the data sets have few samples, the statistical significance interval should be computed and reported in the manuscript.
Response: I added 2 sample t-test to test the significance of the results as highlighted in section 4.
- Reviewer Comment: Cross-validation. The authors shuffled the training samples during the training procedure. This is not equivalent to a cross-validation procedure in which the same method is trained several times after swapping samples between the training and the validation set. By relying on the average results, obtained with different training and validation sets, a much more reliable performance evaluation is obtained.
Response: I added 10-fold cross-validation to evaluate the performance of the model as highlighted in section 4.
I attached the revised manuscript with highlighted changes.
Thank You
Author Response File: Author Response.docx
Reviewer 3 Report
The authors have responded well to the questions, I agree that the article will be published
Author Response
Dear,
Thank you for accepting the paper.
