*3.3. Analysis of the Feature Sets' Data Point Characterization Using F1-Score Metric*

In this section, the performance of the feature sets in terms of data point characterization were also examined by computing the F1-score values for both the amputees and able-bodied subjects across all the movement classes and the obtained results are presented in Tables 3 and 4 as follows using window length ranging from 150 to 300 ms with 100 ms increment. It could be seen from the results presented in Table 3 that the NTDFS, TD-PSD, and TD5AR6 features achieved relatively high F1-scores of approximately 0.99 ± 0.003%, 0.97 ± 0.005%, and 0.96 ± 0.005%, respectively, as against the PKF and MDF features that recorded the lowest accuracy of 0.58 ± 0.03%, and 0.72 ± 0.04%, across subjects. *Symmetry* **2020**, *12*, x FOR PEER REVIEW 12 of 20 *Symmetry* **2020**, *12*, x FOR PEER REVIEW 12 of 20


**Table 3.** Average motion classification accuracies based on F1\_score metric for able-bodied subject. **Table 3**. Average motion classification accuracies based on F1\_score metric for able-bodied subject. **Table 3**. Average motion classification accuracies based on F1\_score metric for able-bodied subject.

 TD5AR6 0.9390 0.9517 0.9586 0.9619 0.9528 ± 0.0101 TD-PSD 0.9559 0.9657 0.9714 0.9753 0.9671 ± 0.0084 TD-PSD 0.9559 0.9657 0.9714 0.9753 0.9671 ± 0.0084 **Table 4.** Average motion classification accuracies based on F1\_score metric for amputee subject. **Table 3**. Average motion classification accuracies based on F1\_score metric for able-bodied subject. **Table 3**. Average motion classification accuracies based on F1\_score metric for able-bodied subject.

TD5AR6 0.9390 0.9517 0.9586 0.9619 0.9528 ± 0.0101


 TD2 0.9123 0.9184 0.9221 0.9256 0.9196 ± 0.0057 TDAR6 0.9449 0.9517 0.9559 0.9604 0.9532 ± 0.0066 TD5AR6 0.9516 0.9583 0.9612 0.9643 0.9589 ± 0.0054 TD-PSD 0.9622 0.9656 0.9694 0.9730 0.9676 ± 0.0047 NTDFS 0.9830 0.9863 0.9877 0.9895 0.9866 ± 0.0028 TD2 0.9123 0.9184 0.9221 0.9256 0.9196 ± 0.0057 TDAR6 0.9449 0.9517 0.9559 0.9604 0.9532 ± 0.0066 TD5AR6 0.9516 0.9583 0.9612 0.9643 0.9589 ± 0.0054 TD-PSD 0.9622 0.9656 0.9694 0.9730 0.9676 ± 0.0047 NTDFS 0.9830 0.9863 0.9877 0.9895 0.9866 ± 0.0028 TD-PSD 0.9559 0.9657 0.9714 0.9753 0.9671 0.0084 NTDFS 0.9904 0.9934 0.9952 0.9964 0.9938 0.0026 **Table 4.** Average motion classification accuracies based on F1\_score metric for amputee subject. **S/No. Window Length**  TD-PSD 0.9559 0.9657 0.9714 0.9753 0.9671 0.0084 NTDFS 0.9904 0.9934 0.9952 0.9964 0.9938 0.0026 **Table 4.** Average motion classification accuracies based on F1\_score metric for amputee subject. **S/No. Window Length**  Also for the amputee subjects, similar trend was observed regarding the features performance with no significant differences across the varying window lengths. Taking a closer look at the F1-score results for both the able-bodied subjects and amputees, it could be seen that the abled-bodied subjects recorded relatively higher values, basically because the amputees had limited residual muscle and could not provide sufficient EMG information for accurate decoding of their targeted limb movement

PKF 0.5495 0.5743 0.5835 0.6141 0.5804 ± 0.0267

TD5AR6 0.9390 0.9517 0.9586 0.9619 0.9528 0.0101

PKF 0.5495 0.5743 0.5835 0.6141 0.5804 ± 0.0267

TD5AR6 0.9390 0.9517 0.9586 0.9619 0.9528 0.0101

 MNP 0.9366 0.9422 0.9461 0.9497 0.9436 ± 0.0056 PSR 0.8761 0.8870 0.8919 0.8755 0.8826 ± 0.0081 TTP 0.8958 0.9057 0.9097 0.9143 0.9064 ± 0.0079 VAR 0.8819 0.8919 0.8960 0.9013 0.8928 ± 0.0082 AR4 0.9262 0.9384 0.9458 0.9514 0.9405 ± 0.0109 AR6 0.9065 0.9198 0.9287 0.9346 0.9224 ± 0.0122 WL 0.9100 0.9100 0.9200 0.9200 0.9150 ± 0.0058 MDF 0.6703 0.7199 0.7415 0.7671 0.7247 ± 0.0411 PKF 0.5495 0.5743 0.5835 0.6141 0.5804 ± 0.0267 TD2 0.9123 0.9184 0.9221 0.9256 0.9196 ± 0.0057 TDAR6 0.9449 0.9517 0.9559 0.9604 0.9532 ± 0.0066 TD5AR6 0.9516 0.9583 0.9612 0.9643 0.9589 ± 0.0054 TD-PSD 0.9622 0.9656 0.9694 0.9730 0.9676 ± 0.0047 NTDFS 0.9830 0.9863 0.9877 0.9895 0.9866 ± 0.0028

Also for the amputee subjects, similar trend was observed regarding the features performance with no significant differences across the varying window lengths. Taking a closer look at the F1 score results for both the able-bodied subjects and amputees, it could be seen that the abled-bodied subjects recorded relatively higher values, basically because the amputees had limited residual muscle and could not provide sufficient EMG information for accurate decoding of their targeted limb movement intents. This phenomenon has also been verified by a number of previous studies [14,16]. Overall, the characteristics of the feature extraction methods based on the F1-score metric is also found to be consistent with the previous two metrics, which further confirms the validity of our

Also for the amputee subjects, similar trend was observed regarding the features performance with no significant differences across the varying window lengths. Taking a closer look at the F1 score results for both the able-bodied subjects and amputees, it could be seen that the abled-bodied subjects recorded relatively higher values, basically because the amputees had limited residual muscle and could not provide sufficient EMG information for accurate decoding of their targeted limb movement intents. This phenomenon has also been verified by a number of previous studies [14,16]. Overall, the characteristics of the feature extraction methods based on the F1-score metric is also found to be consistent with the previous two metrics, which further confirms the validity of our

**Feature Sets 150 200 250 300 Mean ± SD** 

score results for both the able-bodied subjects and amputees, it could be seen that the abled-bodied subjects recorded relatively higher values, basically because the amputees had limited residual muscle and could not provide sufficient EMG information for accurate decoding of their targeted limb movement intents. This phenomenon has also been verified by a number of previous studies [14,16]. Overall, the characteristics of the feature extraction methods based on the F1-score metric is also found to be consistent with the previous two metrics, which further confirms the validity of our

 MAV 0.8852 0.8931 0.8979 0.9034 0.8949 ± 0.0077 MNP 0.9366 0.9422 0.9461 0.9497 0.9436 ± 0.0056 PSR 0.8761 0.8870 0.8919 0.8755 0.8826 ± 0.0081 TTP 0.8958 0.9057 0.9097 0.9143 0.9064 ± 0.0079 VAR 0.8819 0.8919 0.8960 0.9013 0.8928 ± 0.0082 AR4 0.9262 0.9384 0.9458 0.9514 0.9405 ± 0.0109 AR6 0.9065 0.9198 0.9287 0.9346 0.9224 ± 0.0122 WL 0.9100 0.9100 0.9200 0.9200 0.9150 ± 0.0058 MDF 0.6703 0.7199 0.7415 0.7671 0.7247 ± 0.0411 PKF 0.5495 0.5743 0.5835 0.6141 0.5804 ± 0.0267 TD2 0.9123 0.9184 0.9221 0.9256 0.9196 ± 0.0057 TDAR6 0.9449 0.9517 0.9559 0.9604 0.9532 ± 0.0066 TD5AR6 0.9516 0.9583 0.9612 0.9643 0.9589 ± 0.0054 TD-PSD 0.9622 0.9656 0.9694 0.9730 0.9676 ± 0.0047 NTDFS 0.9830 0.9863 0.9877 0.9895 0.9866 ± 0.0028

with no significant differences across the varying window lengths. Taking a closer look at the F1 score results for both the able-bodied subjects and amputees, it could be seen that the abled-bodied subjects recorded relatively higher values, basically because the amputees had limited residual muscle and could not provide sufficient EMG information for accurate decoding of their targeted limb movement intents. This phenomenon has also been verified by a number of previous studies [14,16]. Overall, the characteristics of the feature extraction methods based on the F1-score metric is also found to be consistent with the previous two metrics, which further confirms the validity of our

intents. This phenomenon has also been verified by a number of previous studies [14,16]. Overall, the characteristics of the feature extraction methods based on the F1-score metric is also found to be consistent with the previous two metrics, which further confirms the validity of our findings, thus far. driven stability index (S\_Index) metric defined in Section 2.6 (Equation (5)) was utilized to evaluate the robustness of the feature sets considered and the obtained results for the able-bodied and amputee subjects were presented in Figures 6 and 7 respectively.

From the result illustrated in Figures 6 and 7, it could be seen that the CE of the feature sets

extraction methods to characterize the participants' limb movement intents. Thus, a statistically

*Symmetry* **2020**, *12*, x FOR PEER REVIEW 13 of 20

#### *3.4. E*ff*ect of Disturbance on the Feature Set Performance* decreased with increase in window length, which is consistent with previous results. Furthermore,

*3.4. Effect of Disturbance on the Feature Set Performance* 

The robustness of the individual feature extraction methods were examined by introducing specific amount of random noise into the sEMG signals and then using each of the selected feature extraction methods to characterize the participants' limb movement intents. Thus, a statistically driven stability index (S\_Index) metric defined in Section 2.6 (Equation (5)) was utilized to evaluate the robustness of the feature sets considered and the obtained results for the able-bodied and amputee subjects were presented in Figures 6 and 7 respectively. the results show that multi-features are more robust to disturbance compared to single features. As the noise was introduced, NTDFS feature recorded the least CE values ranging between 4.19%~5.94% for both able-bodied and amputee subjects, correspondingly across window lengths (150 ms~300 ms) compared to other feature sets. Meanwhile, the PKF and MDF are mostly affected by the noise, thus recording CE between 47.51%~54.14%, and 46.30%~37.41%, respectively, indicating high-level of instability in the presence of noise.

**Figure 6** Mean classification error of the features in terms of their robustness to external noise across varying window length at window increment of 100 ms for abled-bodied subjects. **Figure 6.** Mean classification error of the features in terms of their robustness to external noise across varying window length at window increment of 100 ms for abled-bodied subjects. *Symmetry* **2020**, *12*, x FOR PEER REVIEW 14 of 20

**Figure 7.** Mean classification error of the features in terms of their robustness to external NOISE across varying window length at window increment of 100 ms for amputee subjects. **Figure 7.** Mean classification error of the features in terms of their robustness to external NOISE across varying window length at window increment of 100 ms for amputee subjects.

all the feature sets using sEMG recordings from 2, 4, and 6 channels, and the obtained results are presented in Figure 8a–f for both the able-bodied and amputee subjects. Figure 8a,b represent the classification performances of all the features using two channels. Therefore, the CE of all the feature sets (except PKF) decreased with increasing window length and this trend was consistent with the other results obtained when sEMG recordings from 4 and 6 channels were utilized (Figure 8c–f) for both able-bodied and amputee subjects. In other word, the CE reduces with increasing window

In like manner, the classification performances of the feature extraction methods were again observed to be better for the able-bodied subjects compared to the amputee subjects (Figure 8a–f) when sEMG recordings from the same number of electrode channels were utilized. In other words, it could be seen that the PKF (able-bodied: 65.65 ± 1.58%, amputee: 72.90 ± 0.72%) and MDF (ablebodied: 52.65 ± 3.02%, amputee: 56.21 ± 3.21%) features recorded the highest CE values for 2-channel sEMG recordings regardless of the type of participants, while the NTDF (able-bodied: 4.05 ± 0.83%, amputee: 6.15 ± 1.54%) and TD-PSD (able-bodied: 16.54 ± 1.78%, amputee: 22.81 ± 2.55%) features

In similar trend, the same phenomenon was observed for 4-channels and 6-channels sEMG recording though with slight decrease in CE values. In general, utilizing sEMG recordings from 6 channels achieved the lowest CE for all the feature sets. Hence, such variability indicate that the number channels utilized may influence the classification performance of EMG-PR classifiers. It is important to note that the computation time of all the features across varying window length increases with increasing number of electrode channels, indicating a trade-off between number of

*3.5. The Effect of Number of Channels on the Feature Sets Across Windowing Parameters* 

length and number of channels regardless of the kind of feature set employed.

achieved the least CE values.

electrode channels and computation time.

From the result illustrated in Figures 6 and 7, it could be seen that the CE of the feature sets decreased with increase in window length, which is consistent with previous results. Furthermore, the results show that multi-features are more robust to disturbance compared to single features. As the noise was introduced, NTDFS feature recorded the least CE values ranging between 4.19%~5.94% for both able-bodied and amputee subjects, correspondingly across window lengths (150 ms~300 ms) compared to other feature sets. Meanwhile, the PKF and MDF are mostly affected by the noise, thus recording CE between 47.51%~54.14%, and 46.30%~37.41%, respectively, indicating high-level of instability in the presence of noise.

Additionally, the standard error bars in Figure 6 were observed to be relatively lower than those in Figure 7 across subjects and window lengths, thus, indicating that the able-bodied subjects' data resulted in a better S\_Index compared to the amputee subjects. In other words, the amputee subjects are more susceptible to the disturbance compared to the able-bodied subjects. Since the amputee subjects are the end-user of the myoelectric device, there is a need to employ a robust feature sets that could enhance movement intent decoding task needed for the EMG-PR control system regardless of the windowing parameters adopted.
