The training period of the system was 7 days, and each subject trained for 5 cycles per day, performing 6 MI tasks per cycle and 10 experiments per task. The 6 MI tasks are: left fist (T1), right fist (T2), left foot (T3), right foot (T4), left thumb (T5), and right thumb (T6).
The timing diagram of single MI training is shown in
Figure 8, and the training time was 8 s. At the 0th second, the system generated a prompt tone, followed by pictures of the left fist, right fist, left foot, right foot, left thumb, and right thumb on the screen, guiding the subject to imagine the corresponding body movements according to the pictures: “left fist clenched”, “right fist clenched”, “extend left foot”, “extend right foot”, “erect left thumb”, and “erect right thumb”; at the 5th second, the pictures disappeared and the subject relaxed; at the 8th second, this training ended and the subject was prompted to move on to the next MI task.
3.3. Classification Results
In this paper, 10 groups of single-subject experiments were conducted, and 900 samples were collected from each subject and divided into five equal parts. The training set (540 samples), validation set (180 samples), and test set (180 samples) were divided in the ratio of 3:1:1. All samples in the training and validation sets were sent to the CNN network through uniform normalization. Afterwards, the model was trained based on the training set and the optimal parameters were selected on the validation set to achieve stable convergence and obtain the optimal model. Then the test set was fed into the optimal model to obtain the test results, and the average of the five results was used as the classification result. This can reduce the randomness caused by data partitioning and help improve the stability of the model.
Figure 11 shows the loss curves of 10 individual subjects obtained by the CNN model during training. Their loss values decrease continuously with the increase of the number of iterations, and reach the minimum value and remain basically stable at about 800 iterations, so as to obtain the optimal training effect. It can be seen from the figure that the CNN model trained on the data of 10 individual subjects is convergent.
In this paper, the classification performance of the BCI system on the self-collected data set of individual subjects is measured by the common evaluation metrics of accuracy, precision, recall, and the F1-score, as shown in
Table 2. As can be seen from the table, subject S1 has the best test performance and S8 has the worst. The mean values of accuracy, precision, recall, and the F1-score for the 10 subjects are 81.08%, 82.77%, 92.07%, and 87.13%, respectively. The results verified that the BCI system achieved a good classification effect on the self-collected data sets of 10 subjects. Subjects with classification accuracy, from high to low, were S1, S7, S10, S3, S6, S4, S9, S2, S5, and S8, and the top six subjects all achieved more than 80% accuracy, and they were selected to participate in the next session of the testing experiment.
Figure 12 shows the accuracy histograms of six MI tasks for 10 subjects, with the highest accuracy of 90.75% (S1) and the lowest of 71.25% (S8) for T1; the highest accuracy of 93.23% (S1) and the lowest of 71.58% (S8) for T2; the highest accuracy of 87.60% (S7) and the lowest of 71.14% (S8) for T3; the accuracy ranged from a high of 92.91% (S1) to a low of 71.45% (S8) for T4; T5 had a high of 83.29% (S1) and a low of 66.85% (S8); and T6 had a high of 84.70% (S1) and a low of 66.92% (S8). The average accuracy of the six MI tasks was 84.41% (T1), 85.59% (T2), 82.67% (T3), 83.75% (T4), 74.01% (T5), and 76.05% (T6), ranked as T2 (right fist) > T1 (left fist) > T4 (right foot) > T3 (left foot) > T6 (right thumb) > T5 (left thumb). The best MI task for classification is the right fist, and the worst is the left thumb.
Figure 13 shows the ROC curve and AUC value of a single subject obtained when the CNN model was tested, from which it can be seen that the average AUC value for the 10 subjects was 0.945; S1 had the largest AUC value of 0.963 and S8 had the smallest AUC value of 0.920. Then, the CNN model achieved a good classification performance on the self-collected dataset of 10 subjects, S1 had the best classification effect after training, and S8 had the worst effect.
In summary, the six-classification CNN model constructed and trained by the method proposed in this paper achieved high classification accuracy on the self-collected datasets of 10 subjects, indicating that the method has a good generalization performance and can effectively complete the task of EEG intention classification.
3.4. MI-BCI System Experiment Result
After system debugging, the hardware and software of the MI-BCI are connected properly and the communication is in good condition. The control strategy module will continuously accumulate the classification results output by the CNN model, and convert the results into corresponding control instructions only when they reach the set threshold, which will be transmitted wirelessly to the smart car through the Bluetooth module.
Table 3 shows the corresponding control functions and instructions of the six MI tasks in the BCI system.
In this paper, the intelligent car developed by our research group is selected as the external controlled device, which is mainly composed of a Microcontroller Unit (MCU) module, a drive motor control module, a steering gear control module, a speed detection module, a path information acquisition module, a power module, and a wireless Bluetooth transmission module.
The online test experiment of the MI-BCI system is carried out based on the trained optimal CNN model. During the experiment, the smart car was placed on the designed track, which was a square of 1 m in length and width surrounded by black tape on the white background floor. The car was required to run the whole course at a uniform speed according to the subjects’ brain intention. The online test environment of the MI-BCI system was shown in
Figure 14.
Before the BCI system experiment, the steering gear angle and speed and threshold parameters of the intelligent car need to be set. The track is square with four 90-degree turns and the rest is straight. In order to turn the car smoothly and reduce the impact of the car turning left and right in a straight line by misoperation, the maximum angle of the steering gear is set at 23 degrees. Since it takes 5s for the BCI system to collect the MI-EEG signal, the car speed and threshold should not be set too high. Choose two values for the speed, 2 cm/s and 4 cm/s, and three values for the threshold, 1, 2, and 3. Six modes are then determined:
Mode 1: Speed is 2 cm/s and threshold is 1;
Mode 2: Speed is 2 cm/s and threshold is 2;
Mode 3: Speed is 2 cm/s and threshold is 3;
Mode 4: Speed is 4 cm/s and threshold is 1;
Mode 5: Speed is 4 cm/s and threshold is 2;
Mode 6: Speed is 4 cm/s and threshold is 3.
Subjects S1, S3, S4, S6, S7, and S10 were selected to participate in the BCI system experiment. Each subject carried out six modes of training experiments, respectively, and in each mode, the car completed five laps clockwise and five laps counterclockwise.
Table 4 shows the statistics of the number of laps completed by the car under the control of six subjects in different modes. The number of complete clockwise laps of the car was 22, 26, 21, 25, 10 and 3, totaling 107, while the corresponding counterclockwise laps were 18, 20, 18, 20, 8 and 1, totaling 85. In different modes, six subjects performed better in clockwise than counterclockwise direction, which also verified the experimental results of MI-EEG signal classification, where T2 (right fist) was classified more accurately than T1 (left fist), and so the right-turn command was more accurate than the left-turn command when translated into car commands. The total number of laps in each mode is as follows: 40 laps in mode 1, 46 laps in mode 2, 39 laps in mode 3, 45 laps in mode 4, 18 laps in mode 5, and 4 laps in mode 6. It can be seen that mode 2 and mode 4 were completed well, with the total number of laps being 46 and 45, respectively. These two parameter settings can be selected for the online testing of the BCI system.
After the training, the relevant parameters were selected, and the adaptive coordination between the subjects and the intelligent car was completed. The online test experiment of the BCI system can be carried out. The experiment consisted of six subjects running three sets of tests in two selected modes, each consisting of a clockwise and counterclockwise lap.
Table 5 shows the test results in mode 2. The mean time and the mean total number of instructions for subject S1 to control the car to complete a set of tests were 561.8 s and 58, respectively, 655.0 s and 64 for subject S3, 696.7 s and 69 for subject S4, 685.1 s and 66 for subject S6, 672.8 s and 65 for subject S7, and 708.8 s and 70 for subject S10. In this mode, it can be seen that the average of the total number of instructions is proportional to the average of the time spent, with subject S1 taking the shortest time, having the least total number of instructions, and the best test result.
Table 6 shows the test results in mode 4. The mean time and mean total number of instructions for subject S1 to control the car to complete a set of tests were 484.0 s and 68, respectively, 544.1 s and 80 for subject S3, 581.8 s and 82 for subject S4, 525.4 s and 78 for subject S6, 520.2 s and 75 for subject S7, and 569.0 s and 81 for subject S10. It can be seen that the test effect of subject S1 is also the best in mode 4.
The speed of
Table 5 is 2 cm/s and the speed of
Table 6 is 4 cm/s. To complete the same test, the time used in the former should be theoretically twice as long as the time used in the latter. However, from the data listed in the two tables, it can be seen that the time used in
Table 5 is obviously less than the two times of
Table 6. In addition, the number of executed instructions listed in
Table 5 is less than that in
Table 6, because the threshold value of
Table 6 is 1, and the car will execute the instruction immediately after receiving it once. While the threshold value of
Table 5 is 2, the car will execute the instruction only when it receives the same instruction twice in a row, so the response will be slow, but it can better eliminate some wrong operations.
Figure 15 shows the car motion track recorded by subject S1 when he controlled the car to run the whole course under two modes. The orange dots in the figure is the starting position of the car and the blue dots in the figure is the motion track of car. It can be seen that, in the same mode, the track curve of the clockwise direction has less fluctuation than that of the counterclockwise direction, and it fits the track of black line more closely. This is because T2 (right fist) has higher classification accuracy than T1 (left fist), so the corresponding right turn instruction is more accurate than the left command, and the motion track of the clockwise fist is better than the counterclockwise fist. In mode 2, although the setting speed of the car is relatively low and the time used is longer, the threshold is set to two, which filters out some misoperations. The motion track of the car in mode 2 is better and has less deviation than that in mode 4.
To sum up, the parameters have different effects on the system. When setting a high threshold value, the flexibility of the car will also be reduced although it can reduce some misoperation. When setting a high speed, although the overall running time will be reduced, the deviation caused by the wrong instruction is greater and the effect of the motion trajectory is not good. The test effect cannot be compared directly with the different parameters set and the parameters should be selected according to the actual situation. When the parameters were the same, T2 (right fist) had a higher classification accuracy than T1 (left fist), and the right turn instruction was more accurate than the left turn instruction. The time used, the total number of instructions, and the track curve all verified that the subject performed better in the clockwise direction than the counterclockwise direction. In general, the experiment time is proportional to the total number of corresponding instructions. The fewer instructions received by the subjects, the shorter the time spent, and the better the test effect.