Real-Time sEMG Pattern Recognition of Multiple-Mode Movements for Artificial Limbs Based on CNN-RNN Algorithm

Round 1

Reviewer 1 Report

In this study, the investigators have used CNN and LSTM networks to identify hand gestures from 1d ECG signals. The topic is studied and published previously. Moreover, the small database size is a concerning factor for the reproducibility of the proposed outcomes. I have shared some comments for the reference of the researchers. 

 

1. To address the issue that the classification recognition rate and time delay cannot 12 be considered simultaneously, a 1D Convolutional Recurrent Neural Network classification model 13 for recognizing online finger and wrist movements in real-time was proposed, which could effec-14 tively combine the advantages of Convolutional Neural Network and Recurrent Neural Network. " is too long statement for the following. It should be split into two statements.

2. Please clarify the "methods" mentioned in line 44. Also, check for grammatical errors in the text. 

3. Please clarify "traditional classifiers" mentioned in line 51.

4. In line 80, you can use "linear SVM" instead of "linear kernel svm". Even you don't need to highlight the linearity of the SVM since it will be assumed linear as long as not described as non-linear.

5. Please clarify the statement starting at line 81, especially "combinations of SVM nuclei and electrodes"

6. I kindly suggest mentioning the reasoning behind linear SVM learning the behavior than non-linear. in line 86.

7. You should cite studies to support your claim "Traditional machine learning tech-niques cannot efficiently categorize and train on abstract, noisy, and high-dimensional 91 data, and it is a great challenge to achieve high classification accuracy for unprocessed raw sEMG signals. "

8. Please use alternative term instead of "and others".

9. Please revise the hypothesis statement starting at line 121.

10. Please remove the last paragraph of the introduction section which is not part of the introduction. You may remove it to merge it with the discussion section. 

11. Please use the standard citation format for the companies, e.g. (Company Name, City, State) or (Company Name, City, non-US Country)

12. Please check the format of the manuscript in terms of font size, etc.

13. Please regenerate the Figure 2 with high resolution.

14. Please use another verb instead of "make up" in line 201.

15.Which technique did you use for the standardization?

16. Please remove the description of CNN, RNN and LSTM networks, just diagrams are enough since you're not proposing a modified network. Also, all the formulas can be removed.

17. Please remove the formula for the evaluation metrics. They're all known and not needed to describe here.

18. Please correct the title for the discussion section.

19. For time comparison, did you use the same system?

20. Please revise the conclusion section. Some of the claims are too strong to release, e.g. The delay with the proposed approach is enough.

 

 

 

Author Response

Response to Reviewer 1 Comments

We are pleased by the positive evaluation that our manuscript received from you. We carefully read the comments and suggestions and have now completed a revision of the manuscript that addresses the concerns. According to your suggestions, we gave full point-to-point responses. We thank you for the constructive suggestions that have improved both the quality and the clarity of the manuscript. Please see the attachment for the marked version.

 

Point 1: To address the issue that the classification recognition rate and time delay cannot be considered simultaneously, a 1D Convolutional Recurrent Neural Network classification model for recognizing online finger and wrist movements in real-time was proposed, which could effectively combine the advantages of Convolutional Neural Network and Recurrent Neural Network. " is too long statement for the following. It should be split into two statements.

 

Response 1: Thanks a lot for the reviewer’s suggestion. This sentence was split into two statements as shown in the marked version.

"A 1D Convolutional Recurrent Neural Network classification model for recognizing online finger and wrist movements in real-time was proposed to address the issue that the classification recognition rate and time delay cannot be considered simultaneously. This model could effectively combine the advantages of Convolutional Neural Network and Recurrent Neural Network. "

 

Point 2: Please clarify the "methods" mentioned in line 44. Also, check for grammatical errors in the text.

 

Response 2: the "methods" is the "pattern recognition methods" which is mentioned in line 43. We revised the whole sentence so that the reader can understand better. Please see the marked version.

Moreover, we have extensively revised the entire manuscript, especially the grammatical issue. Please see the marked version.

 

Point 3: Please clarify "traditional classifiers" mentioned in line 51.

 

Response 3: Sorry for the ambiguous statement! In this article, We added the explaination of "traditional classifiers". Please see the marked version.

" ‘traditional classifiers’ means traditional classification method based on machine learning. This statement was mentioned to distinguish it from the classification method based on deep learning."

 

Point 4: In line 80, you can use "linear SVM" instead of "linear kernel svm". Even you don't need to highlight the linearity of the SVM since it will be assumed linear as long as not described as

 

Response 4: Thanks a lot for the reviewer’s helpful suggestion. We used "linear SVM" instead of "linear kernel svm".

 

Point 5: Please clarify the statement starting at line 81, especially "combinations of SVM nuclei and electrodes"

 

Response 5: Amirabdollahian F, et.al. studied application of support vector machines in detecting hand grasp gestures. In the study, different Kernel functions and electrode combinations were studied. They conducted experiments with different kernel functions and electrode numbers to investigate the effects of these factors on the performance of the classifier. As shown in experimental results, the linear kernel and eight channel electrode combination can achieve the best accuracy of 94.9%.

 

Point 6: I kindly suggest mentioning the reasoning behind linear SVM learning the behavior than non-linear. in line 86.

 

Response 6: Thanks a lot for the reviewer’s helpful suggestion. We didn’t mention non-linear SVM. This sentence mainly described the study of Amirabdollahian F, et. al. In their study, the linear kernel and eight channel electrode combination can achieve the best accuracy.

 

Point 7: You should cite studies to support your claim "Traditional machine learning techniques cannot efficiently categorize and train on abstract, noisy, and high-dimensional data, and it is a great challenge to achieve high classification accuracy for unprocessed raw sEMG signals. "

 

Response 7: Thanks a lot for the reviewer’s helpful suggestion. We reviewed an article to support our claim. This article’s name is “1D CNN based network intrusion detection with normalization on imbalanced data”. This article evaluated the performance of 1D-CNN. RF and SVM were exploited for comparison. The result is that 1D-CNN and its variant architectures have outperformed compared to the classical machine learning classifiers.

 

Point 8: Please use alternative term instead of "and others".

 

Response 8: Thanks a lot for the reviewer’s helpful suggestion. We used "etc." instead of "and others". Please see the marked version.

 

Point 9: Please revise the hypothesis statement starting at line 121.

 

Response 9: Thanks a lot for the reviewer’s suggestion. We removed the hypothesis statement. Please see the marked version.

 

Point 10: Please remove the last paragraph of the introduction section which is not part of the introduction. You may remove it to merge it with the discussion section. 

 

Response 10: Thanks a lot for the reviewer’s suggestion. In the marked version, we removed the last paragraph of the introduction section.

 

Point 11: Please use the standard citation format for the companies, e.g. (Company Name, City, State) or (Company Name, City, non-US Country)

 

Response 11: Thanks a lot for the reviewer’s suggestion. We revised the citation format for the company. Please see the marked version.

 

Point 12: Please check the format of the manuscript in terms of font size, etc.

 

Response 12: Thanks a lot for the reviewer’s suggestion. We checked the format of the manuscript according to template provied by electronics. The Microsoft Word template was downloaded form "https://www.mdpi.com/journal/electronics/instructions".

 

Point 13: Please regenerate the Figure 2 with high resolution.

 

Response 13: Thanks a lot for the reviewer’s suggestion. We regenerated the Figure 2 with high resolution. Please see the marked version.

 

Point 14: Please use another verb instead of "make up" in line 201.

 

Response 14: Thanks a lot for the reviewer’s suggestion. We revised the whole sentense to avoid using "make up".

"Data preprocessing mainly includes filtering and noise reduction, standardi-zation, and active segment extraction."

 

Point 15: Which technique did you use for the standardization?

 

Response 15: Sorry for the ambiguous statement! We added the description of standardized method which we used. Please see the marked version.

"The standardization method used in this article is Z-Score standardization. This is a frequently-used standardized technique. The transformation formula is (x-μ) /σ. In this formula, μ is Mean and σ is Standard Deviation. It converts data of different orders of magnitude into unitless values. "

 

Point 16: Please remove the description of CNN, RNN and LSTM networks, just diagrams are enough since you're not proposing a modified network. Also, all the formulas can be removed.

 

Response 16: Thanks a lot for the reviewer’s suggestion. We removed the description of CNN, RNN and LSTM networks, and the formulas.

 

Point 17: Please remove the formula for the evaluation metrics. They're all known and not needed to describe here.

 

Response 17: Thanks a lot for the reviewer’s suggestion. We removed the formula for the evaluation metrics.

 

Point 18: Please correct the title for the discussion section.

 

Response 18: Thanks a lot for the reviewer’s suggestion. We changed "Discussions" to "Discussion".

 

Point 19: For time comparison, did you use the same system?

 

Response 19: We didn’t use the same system. Different systems have great influence on model training time. However, the trained model requires less computational power. Compared with the effect of algorithm performance on delay, the effect of different system is negligible. Therefore, it is feasible to account for time comparison without using the same system.

 

Point 20: Please revise the conclusion section. Some of the claims are too strong to release, e.g. The delay with the proposed approach is enough.

 

Response 20: Thanks a lot for the reviewer’s suggestion. We have revised some claims. In addition, we added a reference article to support our conlusion. Please see the marked version.

Reviewer 2 Report

This paper develops the 1D-CNN-RNN model suitable for pattern recognition of sEMG signals. The topic is very actual. The one main question is following: Why did you select the ML architecture: 1D-CNN-RNN and not the models which employ: e.g. SVM  and moreover another hybrid models, e.g CNN-LSTM? And please add the limitations of the work.

Author Response

Response to Reviewer 2 Comments

We are pleased by the positive evaluation that our manuscript received from you. We carefully read the comments and suggestions and have now completed a revision of the manuscript that addresses the concerns. We thank you for the constructive suggestions that have improved both the quality and the clarity of the manuscript. Please see the attachment for the marked version.

 

Point 1: Why did you select the ML architecture: 1D-CNN-RNN and not the models which employ: e.g. SVM and moreover another hybrid models, e.g CNN-LSTM? And please add the limitations of the work.

 

Response 1: SVM is a classification model based on traditional machine learning. The application of SVM in gesture recognition was mentioned in Section Introduction. 1D-CNN-RNN is a hybrid neural network model based on deep learning. Some studies have shown that neural networks have a higher utilization rate of data. It performs better when analyzing large amounts of data. This was also mentioned in Section Introduction. Therefore, we expect that algorithms based on deep learning can achieve better classification results. The reason why we use 1D-CNN-RNN hybrid model is that they're both excellent at processing time series signals. The structure of 1D-CNN-RNN model was described in Section 2.2. As shown in the structure, the hybrid model contains 1D-CNN and LSTM. LSTM is an improved variation of RNN. In other words, LSTM is RNN essentially. So, we perfer to call the model 1D-CNN-RNN rather than CNN-LSTM.

In terms of limitations, we have added the limitations of the work in the last paragraph of Section Discussion. There are two limitations we mentioned. First, the classification outcomes of four motions were generally poor. Second, the sEMG signals were derived from the healthy, therefore, the classification performance in amputees was not yet known. Please see the marked version. If you have any comment on this, please do not hesitate to contact us.

 

Author Response File: Author Response.docx

Reviewer 3 Report

1. Introduction. The authors did not present the Introduction comprehensively. Some methods are missing, for example:

- Pattern recognition of EMG signal using Principal component analysis (PCA)

- Finger movement classification using Adaptive Neuro-fuzzy inference system (ANFIS)

2. Please present an example of the raw EMG signal for each wrist motion.

3. Please improve Figures 2. 6, and 7 as the font size is too small and the resolution needs to be enhanced.

4. What does the meaning of the online experiment in Section 2.1.1 real-time pattern recognition? I could not find any figure or discussion of the embedded deep learning model in microprocessors or hardware.

5. What does the adaptive double threshold refer to in lines 214 and Eqs. (1)-(5). Are they feature extraction?

6. Please also revise Figure 8 by adding the box line inside the confusion matrix. The font size inside the confusion matrix is also too small.

Author Response

Response to Reviewer 3 Comments

We are pleased by the positive evaluation that our manuscript received from you. We carefully read the comments and suggestions and have now completed a revision of the manuscript that addresses the concerns. According to your suggestions, we gave full point-to-point responses. We thank you for the constructive suggestions that have improved both the quality and the clarity of the manuscript. Please see the attachment for the marked version.

 

Point 1: Introduction. The authors did not present the Introduction comprehensively. Some methods are missing, for example:

- Pattern recognition of EMG signal using Principal component analysis (PCA)

- Finger movement classification using Adaptive Neuro-fuzzy inference system (ANFIS)

 

Response 1: : Thanks for your suggestions and sorry for the negligence! At present, many methods have been applied to gesture recognition. We just focused on the pattern recognition methods for the research contrast in Section Introduction. PCA as a EMG processing method is mainly used to reduce the dimension of data, and reduces the computational complexity. ANFIS is rarely used in action recognition. We reviewed an article about this methods. The article name is “EMG finger movement classification based on ANFIS”. We added theis method in Section Introduction. Please see the marked version.

 

Point 2: Please present an example of the raw EMG signal for each wrist motion.

 

Response 2: We have the complete raw EMG signal for each wrist motion. But we didn’t upload in the MDPI submission system. If you or editor required to present the raw EMG signal, , please don’t hesitate to contact us through this email: zhangyuezcy@163.com.

 

Point 3: Please improve Figures 2. 6, and 7 as the font size is too small and the resolution needs to be enhanced.

 

Response 3: Thanks a lot for the reviewer’s helpful suggestion. We regenerated the Figure 2, 6 and 7 with high resolution, see the marked version.

 

Point 4: What does the meaning of the online experiment in Section 2.1.1 real-time pattern recognition? I could not find any figure or discussion of the embedded deep learning model in microprocessors or hardware.

 

Response 4: The online experiment in Section 2.1.1 real-time pattern recognition means that the data is uploaded to the classification model in real time after the signal is collected, and the classification results are output in real-time. The classification model runs on PC currently, not on microprocessors or hardware.

 

Point 5: What does the adaptive double threshold refer to in lines 214 and Eqs. (1)-(5). Are they feature extraction?

 

Response 5: The adaptive double threshold approach refer to in lines 214 is an algorithm for active segment extraction. It is a part of data preprocessing. They are not feature extraction. Its purpose is to distinguish the signals in the active segment from the signals in the resting segment. It can avoid the signals from two classifications mixing with each other.

 

Point 6: Please also revise Figure 8 by adding the box line inside the confusion matrix. The font size inside the confusion matrix is also too small.

 

Response 6: Thanks a lot for the reviewer’s helpful suggestion. We have added the box line inside the confusion matrix. Please see the marked version.

 

Reviewer 4 Report

Real-time sEMG Pattern Recognition of Multiple-mode Movements for Artificial Limbs Based on CNN-RNN Algorithm

1. Very interesting research entitled “Real-time sEMG Pattern Recognition of Multiple-mode Movements for Artificial Limbs Based on CNN-RNN Algorithm”.  (See attached file).

2. Correct the structure of the article (it's a suggestion). (See attached file).

3. I suggest reorganizing the article according to the structure that handles electronics-MDPI.

4. Develop an algorithm of the 1D-CNN-RNN classification model. I suggest that the algorithms in this article use the following format: (See attached file).

5. I suggest to elaborate another algorithm on acquisition and evaluation of sEMG signals.

6. Show with an example the following: Signal acquisition, preprocessing, segmentation, feature extraction and classification. Explain what happens at each stage.

7. I suggest reviewing the following article:

 

 Zhou, X., Li, Y., & Liang, W. (2020). CNN-RNN based intelligent recommendation for online medical pre-diagnosis support. IEEE/ACM Transactions on Computational Biology and Bioinformatics, 18(3), 912-921. https://doi.org/10.1109/TCBB.2020.2994780

 

8. I suggest reviewing the following article:

 

Azizjon, M., Jumabek, A., & Kim, W. (2020, February). 1D CNN based network intrusion detection with normalization on imbalanced data. In 2020 International Conference on Artificial Intelligence in Information and Communication (ICAIIC) (pp. 218-224). IEEE.  https://doi.org/10.1109/ICAIIC48513.2020.9064976

 

9. I suggest including future works.

 

10. Very good bibliography.  

 

Authors are requested to make all indicated corrections.

 

I request to make all the corrections indicated and the corrections of the other reviewers.

                               Kind regards.

Comments for author File: Comments.pdf

Author Response

Response to Reviewer 4 Comments

We are pleased by the positive evaluation that our manuscript received from you. We carefully read the comments and suggestions and have now completed a revision of the manuscript that addresses the concerns. According to your suggestions, we gave full point-to-point responses. We thank you for the constructive suggestions that have improved both the quality and the clarity of the manuscript. Please see the attachment for the marked version.

 

Point 1: Very interesting research entitled “Real-time sEMG Pattern Recognition of Multiple-mode Movements for Artificial Limbs Based on CNN-RNN Algorithm”.  (See attached file).

 

Response 1: Thank you for your kindful words.

 

Point 2: Correct the structure of the article (it's a suggestion). (See attached file).

 

Response 2: Sorry for the negligence! The structure of the article has been corrected. Please see the marked version.

 

Point 3: I suggest reorganizing the article according to the structure that handles electronics-MDPI.

 

Response 3: Thanks a lot for the reviewer’s helpful suggestion. The structure of the article has been corrected. Please see the marked version.

 

Point 4: Develop an algorithm of the 1D-CNN-RNN classification model. I suggest that the algorithms in this article use the following format: (See attached file).

 

Response 4: Thanks a lot for the reviewer’s suggestion. We have reviewed many articles published in electronics, and most of them use thepresent format. So, we prefer to keep the format unchanged. Thanks again for your advices.

 

Point 5: I suggest to elaborate another algorithm on acquisition and evaluation of sEMG signals.

 

Response 5: Thanks a lot for the reviewer’s helpful suggestion. We have elabroated another two algorithm: CNN and LSTM. We designed classification models respectively based on CNN and LSTM to carry out off-line experiments. And we compared the results with 1D-CNN-RNN proposed in this article. It was mentioned in Section 3.1. However, since CNN and LSTM are widely used in gesture recognition, we have removed the introduction of them.

 

Point 6: Show with an example the following: Signal acquisition, preprocessing, segmentation, feature extraction and classification. Explain what happens at each stage.

 

Response 6: Thanks a lot for the reviewer’s helpful suggestion and Sorry for the negligence! We revised the structure and added subheads so that each stage can be described more clearly. Please see the marked version.

Feature extraction refers to calculating some characteristic values of data with formulas, such as mean absolute value (MAV) variance (VAR), and et. al. This method can transform a large amount of data into several features without losing the data differentiation. However, in this article, we didn’t need this stage because CNN can automatically extract features through different convolution kernels.

 

Point 7: I suggest reviewing the following article:

Zhou, X., Li, Y., & Liang, W. (2020). CNN-RNN based intelligent recommendation for online medical pre-diagnosis support. IEEE/ACM Transactions on Computational Biology and Bioinformatics, 18(3), 912-921. https://doi.org/10.1109/TCBB.2020.2994780

 

Response 7: Thanks a lot for the reviewer’s helpful suggestion. This article described a great application of CNN-RNN. It proved that CNN-RNN has significant advantages in processing large amounts of data. We reviewed this article to support our claim. Please see the marked version.

 

Point 8: I suggest reviewing the following article:

Azizjon, M., Jumabek, A., & Kim, W. (2020, February). 1D CNN based network intrusion detection with normalization on imbalanced data. In 2020 International Conference on Artificial Intelligence in Information and Communication (ICAIIC) (pp. 218-224). IEEE.  https://doi.org/10.1109/ICAIIC48513.2020.9064976

 

Response 8: Thanks a lot for the reviewer’s helpful suggestion. This article is very suitable for our research background. It demonstrates the performance difference between 1D-CNN and algorithms based on classical machine learning. We reviewed this article to support our claim. Please see the marked version.

 

Point 9: I suggest including future works.

 

Response 9: Thanks a lot for the reviewer’s helpful suggestion. We have added future works in the last paragraph of Section Discussion and Conclusion.

“In order to enhance the pattern recognition algorithm framework further and perform adaptive adjustments in accordance with the actual scenario of amputees, the sEMG sig-nal from amputees will be applied in the subsequent study.”

“Furthermore, the stability and accuracy of real-time recognition would be included in the future studies.”

Please see the marked version. If you have any comment on this, please do not hesitate to contact us.

 

Point 10: Very good bibliography. 

 

Response 10: Thank you for your kindful words.

 

Round 2

Reviewer 3 Report

Dear Authors,

Thank you for providing the revised version. I have read Author's notes as well as the revised paper. I still would like to suggest Authors plot the raw EMG signal in the paper for the scientific soundness of the paper. Without plotting the EMG signal, the reader has no idea what the EMG signal looks like.

Kind regards,

- Reviewer 3 -

Author Response

We thanks a lot for the helpful suggestion. There was not enough space to show the raw signal of all motions. We just added an example of raw sEMG signal in the ending of Section 2.1. We hope this can help readers understand the sEMG signal. Please see the marked version in the attachment.

Reviewer 4 Report

I thank the authors for the corrections to the article.

Author Response

I thank the reviewer for the kind comments. We improved the article according to the reviewer’s report.