Continuous Hybrid BCI Control for Robotic Arm Using Noninvasive Electroencephalogram, Computer Vision, and Eye Tracking

Abstract

The controlling of robotic arms based on brain–computer interface (BCI) can revolutionize the quality of life and living conditions for individuals with physical disabilities. Invasive electroencephalography (EEG)-based BCI has been able to control multiple degrees of freedom (DOFs) robotic arms in three dimensions. However, it is still hard to control a multi-DOF robotic arm to reach and grasp the desired target accurately in complex three-dimensional (3D) space by a noninvasive system mainly due to the limitation of EEG decoding performance. In this study, we propose a noninvasive EEG-based BCI for a robotic arm control system that enables users to complete multitarget reach and grasp tasks and avoid obstacles by hybrid control. The results obtained from seven subjects demonstrated that motor imagery (MI) training could modulate brain rhythms, and six of them completed the online tasks using the hybrid-control-based robotic arm system. The proposed system shows effective performance due to the combination of MI-based EEG, computer vision, gaze detection, and partially autonomous guidance, which drastically improve the accuracy of online tasks and reduce the brain burden caused by long-term mental activities.

1. Introduction

Millions of people in the world suffer from neurological disorders, such as muscular dystrophy, amyotrophic lateral sclerosis (ALS), and cerebral palsy, and lose the ability to control their limbs. These diseases disrupt the communication between the brain and body, which adversely affects a person’s general quality of life. Nevertheless, most of these persons’ brains still retain the function to produce motor-related neural activities like nondisabled [1]. Brain–computer interface (BCI) technology provides alternatives for individuals to interact with the surrounding environment and can offer daily assistance for neurological disorders patients [2,3,4]. At present, there is a growing variety of external devices that humans can control with BCI technology, especially in controlling virtual cursors [5,6,7], quadcopters [8], wheelchairs [9,10], and exoskeleton robots [11,12]. Furthermore, some researchers use BCI to help people with neurological disorders control robotic arms to grasp objects [13,14,15,16]. However, due to the high number of degrees of freedom (DOFs) of the robotic arm, grasping multiple objects in 3D space in a continuous and effective way using BCI is still a hard problem.

Researchers have tried various methods to record neurophysiological signals from the brain, and these methods can be broadly divided into invasive and noninvasive BCI [17,18]. Invasive BCI systems use an implanted microelectrode array in the cortex to collect the neuronal action potentials directly, so the neurophysiological signals have higher spatial resolution and signal-to-noise rate than noninvasive BCI. However, these implants increase clinical risks of infection and brain tissue damage, so the invasive BCI is mostly applied to animal research [19,20]. Compared with the invasive BCI, the noninvasive BCI is more convenient, economical, and safe. Therefore, it is widely used to record neurophysiological signals. Electroencephalography (EEG) is a very popular noninvasive BCI for recording brain activity by placing electrodes on the scalp. EEG has been widely used in the study of BCI since the University of California started to research it [21].

For decades, multiple patterns of EEG signals collected from different scalp areas have been a basis of BCI systems, including steady-state visual evoked potential (SSVEP) [17,22], sensorimotor rhythms [23,24,25,26], P300 potentials [27], and hybrid systems [12,13]. These kinds of evoked BCI systems like SSVEP and P300 potentials do not need to train for a long time, but subjects have to keep their eyes on the screen to receive external stimuli during the online control, which makes them unable to detect changes in the external environment effectively. Moreover, staring at the stimulating screen for a long time also leads to vision problems [28]. The sensorimotor potentials occur in sensorimotor areas, directly related to motion, and do not depend on visual or auditory stimulation. McFarland et al. indicated that motor imagery (MI) could be an effective way to control sensorimotor rhythm amplitude like movement [29]. Therefore, it is a good method to control the robotic arm by event-related desynchronization (ERD) and event-related synchronization (ERS) which are evoked by the motor imagery (MI) of the subjects [30]. In most previous MI-based robotic arm control systems [12,31,32], the modulated EEG was classified and mapped into multiple discrete control instructions to trigger predetermined actions in a workspace. Nevertheless, these systems typically lack continuous and efficient operations. Some studies have been able to control a robotic arm for continuous random target tracking [33], but it has not been applied to the multitarget grasping experiment with obstacles.

It is still a challenge to control a multi-DOF robotic arm continuously to achieve grab tasks with the noninvasive EEG, especially if there are obstacles in the workspace. Integrating EEG signals with additional biological signals containing the intentions of the users, such as electrooculography, electromyography, and gaze information, provides a new way to increase the degree of control flexibility. Compared with other signals, eye tracking can easily obtain the position of the user’s gaze in real time. Some studies obtained the position of the target that the users intended to obtain by detecting the gaze point of their eyes, and then the EEG signals were used to trigger the robot to automatically perform grasp tasks [14,34]. However, recent studies have shown that users prefer to be participants rather than bystanders when they use assistive devices to complete tasks [35]. It is obvious that fully automatic control systems reduce users’ participation and cannot fully explore their motor initiatives. Hybrid control systems can be a good approach to balance these two aspects. In such a system, human subjects and intelligent devices work together to achieve a common goal [32]. Subjects are responsible for high-level decisions and some simple tasks are completed by intelligent devices, reducing the fatigue caused by long-term mental activities and increasing subjects’ participation. Such hybrid control methods also performed well in EEG-based BCI systems [36,37,38].

In this paper, we propose a noninvasive EEG-based hybrid control system for reaching and grasping targets with a robotic arm in a 3D environment with obstacles. The entire grasp process consisted of three stages. Firstly, a two-dimensional (2D) continuous signal generated by MI controlled the end effector to approach the target. Secondly, the target was detected by computer vision technology and selected by the subject’s gaze point. Finally, the robotic arm reached and grasped the target automatically. We decoded the subjects’ MI-based EEG in real time to output 2D continuous control signals. Obstacles were added to the workspace to simulate a more realistic environment, and a red, green, blue, and depth (RGBD) camera was used to capture the depth information of the scene. The eye-tracker and USB camera were utilized to assist the subjects in selecting the intended target by capturing his/her intention of eyes. Subjects were invited to take part in a group of experiments with increasing levels of difficulty. In online experiments, all the subjects were able to complete the grabbing operation through the robotic arm hybrid control system in a 3D environment with obstacles. The experimental results showed a potential application of the hybrid control BCI system in complicated tasks. The control strategy indicated the efficiency and feasibility of the proposed system due to the combination of MI-based EEG, gaze detection, and computer vision technology.

2. Methods and Experiments

2.1. System Architecture

The architecture of the experimental setup is given in Figure 1, which demonstrates how the subject can control the robotic arm based on our proposed system. The whole system mainly consists of the following components: an EEG record device, an eye-tracker, a UR5 robotic arm, a USB camera, Kinect, and two computers. To settle the difficulty of multitarget grasping with obstacles that might be faced in daily life applications, we divided the reach and grasp process into three stages.

In the first stage, the subjects imagined different types of movement of their hands (right-hand movement, left-hand movement, both-hand movement, and relaxing), and the EEG signals of the subject were recorded by the BCI system. The signals were decoded to control the virtual ball, which was in front of the subject and moved continuously on a 2D plane. The position of the virtual ball would be transmitted in real time to the robotic arm controller installed on the robot operating system (ROS), which controlled the robotic arm to move. When the virtual ball stopped moving, so did the robotic arm. In the second stage, a scene camera mounted on the end effector of the UR5 robot would obtain videos of the workspace and identified the potential targets, which would be selected with a rectangle and displayed on the screen in front of the subject. An eye tracker fixed under the monitor detected the position of subjects’ gaze and controlled the movement of the mouse, assisting subjects in selecting the target. Once the target was chosen, the position would be sent to the robotic arm controller. In the third stage, the target would be grasped automatically after the robot controller obtained the detailed position. Moreover, a depth camera was used to locate the position of obstacles in the workspace, and the robotic arm would avoid the obstacle during motion planning.

2.1.1. BCI System

EEG signals were recorded by the Neuroscan SynAmps2 amplifier with a sampling frequency of 500 Hz. In this research, we selected 20 electrodes from an elastic ActiCap (Brain Products, Gilching, Germany) that attached standard Ag/AgCl 64 electrodes with the international 10–20 system [39], as shown in Figure 2. The left mastoid was adopted as the reference channel while the ground channel was located on the forehead. EEG data were acquired by keeping electrode impedances below 10 kΩ at the beginning of the experiment, and the electrode impedance would be checked again in the middle of each session to remain below 10 kΩ.

During EEG recording, a quiet environment was provided to keep the subjects’ attention focused. Participants were seated in a comfortable chair in front of an LCD screen at a distance of about 80 cm. They were asked to keep all muscles relaxed and avoid eye movements, blinks, swallowing, body adjustments, or other movements during the experiment. Following the signal acquisition, a notch filter at 50 Hz and common average reference (CAR) filter [40], as shown in Equation (1), were utilized to preprocess the raw EEG signals:

$$V_i^{CAR}(t)=V_i(t)-\frac{1}{n}{\displaystyle \sum _{j=1}^n{V}_j(t)},$$

(1)

where V_i(t) is the potential between the reference and the ith electrode at time t, and n is the number of electrodes used in this study.

To evaluate the amplitudes in the mu frequency band for each user, an autoregressive (AR) model was built in the experiment as follows:

$$x_t={\displaystyle \sum _{i=1}^p{a}_i{x}_{t-i}+\epsilon },$$

(2)

where x_t is the estimated signal at time t, p is the order of the AR model, ε is a zero-mean white noise, and a_i is the weight coefficient. The value of a_i is estimated by the least squares criterion. The least squares criterion is used to adjust the weights to minimize the difference between the actual signal and the estimated signal predicted by Equation (2).

The sensorimotor rhythm amplitude of each subject is used to move the virtual ball and robotic arm in the current study. The movement of the virtual ball in the vertical direction is determined by the parameter K_V, which is evaluated by [5]:

$$K_V=w_{RV}{R}_V+w_{LV}{L}_V+d_V,$$

(3)

where d_V is the offset. R_V and L_V are the amplitude on the right side and the left side, respectively. The initial values of w_RV and w_LV were both +1.0, and d_V is 0.0. When the value of K_V is negative or positive, the virtual ball is moved up or down, respectively.

The movement of the virtual ball in the horizontal direction is determined by the parameter K_H, as follows [5]:

$$K_H=w_{RH}{R}_H+w_{LH}{L}_H+d_H,$$

(4)

where d_H is the offset. R_H and L_H are the amplitude on the right side and the left side, respectively. The initial values of w_RH, w_LH, and d_H were +1.0, −1.0, and 0.0, respectively. When the value of K_H is negative or positive, the virtual ball is moved left or right, respectively.

Previous research on ERD/ERS has shown that when the subjects perform unilateral hand motor imagination, the EEG amplitude of the ipsilateral sensorimotor area will increase, and the contralateral side will be inhibited. The virtual ball movement in the vertical direction was initially controlled by the sum of the amplitudes on both sides. Therefore, if the participants imagined both-hand movement, both R_V and L_V would decrease, the virtual ball was moved upwards. In contrast, when the subjects imagined both hands relaxing, the amplitude on the left side (L_V) and right side (R_V) would increase, and the virtual ball was controlled to move downward. The virtual ball movement in the horizontal direction was initially determined based on the difference between R_H and L_H. The control principle in the horizontal direction was the same as that in the vertical direction.

The weights and offsets in Equations (3) and (4) were not fixed values, and they would automatically adjust according to the least-mean-square (LMS) algorithm according to the previous trials. The adjusted parameters would minimize the difference between the real target location and the predicted target location, thereby making the subsequent EEG-based virtual ball control more accurate [5,41].

2.1.2. Object Identification and Target Selection

In this study, computer vision was utilized for object identification, and gaze tracking was chosen for target selection in the online task. For the object identification, a USB camera (1280 × 720 pixels) mounted on the gripper was employed to obtain the information of the workspace. The captured video was transmitted to the computer by USB3.0 port and presented on the monitor in front of the subjects. For the target selection, a low-cost desktop eye-tracker, Tobii eye-tracker 4C (Tobii Tech, Stockholm, Sweden), was mounted on the screen to detect the subject’s gaze position and map it to the computer to control the mouse cursor on the monitor in front of the subject. The gaze data were sent to the computer through the USB3.0 port in real time. Moreover, it was necessary to calibrate the eye tracker for each subject before the experiment.

Four kinds of cube blocks (30 × 30 × 30 mm) with different colors were used as target blocks in this study. The target blocks in the workspace were detected by the background subtraction algorithm based on their color. The object identification sample was demonstrated in Figure 3. The raw image was in red, green, blue (RGB) color space. Firstly, in order to relieve the influence of illumination change on object identification, the raw image was converted to the hue, saturation, value (HSV) color space. Secondly, the target blocks in the image would be identified based on the difference between colors and the threshold (Figure 3B). At last, all the blocks would be successfully identified through background subtraction, and the blocks would be outlined on the screen as potential targets by a green rectangle (Figure 3C). Once a left-click was taken in a potential target’s green rectangle, the target would be selected, and visual feedback would be offered by changing the color of the rectangle surrounding the selected target to red. The mouse cursor was controlled to move on the monitor based on the subjects’ gazed position. When the subjects moved the gaze-controlled mouse cursor into the green rectangle surrounding the desired target and held the cursor in the rectangle for more than 2 s, the left-click would be triggered and the detailed position of the selected target would be sent to the ROS system.

2.1.3. Robotic Arm Control and Obstacle Avoidance

A robotic arm with six DOFs, UR5 (Universal Robots, Odense, Denmark), was used in this research. The maximum working radius of the robotic arm was 0.85 m, and the moving range was limited within the workspace to avoid collisions with objects and subjects outside the workspace. The gripper equipped at the end of the UR5 was nGripper90 (NONEAD Corporation, Suzhou, China) with two fingers. The robotic arm was driven by ROS running on the Ubuntu operating system. The inverse kinematics solver on ROS could convert the position and orientation of the end effector to the corresponding joint positions. Consequently, we could directly provide the robotic arm control system with the endpoint information, and the solver would calculate the inverse kinematic solution and plan the route to the target location. The solver used in this experiment was TRAC-IK [42]. Compared with the traditional solver, it is faster and more reliable.

To simulate a more realistic scene of disabled people’s daily life in this study, we added obstacles to the experimental workspace, which made the task more difficult to finish. If the robotic arm was directly controlled by the EEG to avoid the obstacle, the success rate of online tasks would be possibly low; also, the subject’s mental burden would be huge. In order to overcome the above difficulties, a depth camera, Kinect v2 (Microsoft, Redmond, WA, USA), was employed to collect the cloud point information of the workspace, so that the position of the obstacle would be detected and sent to the robotic arm controller.

The Kinect, mounted on the tripod, was placed on the left side of the workspace, 0.6 m high from the table. Point clouds collected by Kinect outside the workspace were deleted using a passthrough filter to reduce the interference. A voxel grid filter was then applied for downsampling in order to cut down the amount of calculation while not destroying the geometric structure of the point clouds. To remove some outliers, the RadiusOutlierRemoval filter was then utilized in this study. After the preprocessing, the filtered point clouds (Figure 4B) were converted into Octomap (Figure 4C) [43] in ROS and added to its workspace. Therefore, the obstacle could be avoided by the robotic arm during motion planning on ROS. For Octomap, the probability that for each leaf node m could be calculated based on measurements z_1:t:

$$P(m|z_{1:t})={[1+\frac{1-P(m|z_t)}{P(m|z_t)}\frac{1-P(m|z_{1:t-1})}{P(m|z_{1:t-1})}\frac{P(m)}{1-P(m)}]}^{-1},$$

(5)

where z_t is current measurement at time t, P(m) is prior probability, and P(m|z_1:t−1) denotes the previous estimation. P(m|z_t) represents the probability that voxel m is occupied at time t.

2.2. Experimental Paradigm

2.2.1. Subjects

Seven right-handed subjects (average age: 23.1 ± 0.6 years, four males) were recruited from the campus to participate in this study. All of them were naïve for BCI experiment and without any history of neurological disease [5,29,33]. Each subject was informed of the method and research activities of this study, and the informed consent was signed before they participated in the experiment. All procedures and protocols were in accordance with the Helsinki Declaration and were approved by the Ethics Committee of Southeast University.

2.2.2. MI without Feedback

The ERD/ERS phenomenon were identical for everyone. Therefore, it is necessary to obtain the EEG data during MI without feedback experiments before the training experiments for all the subjects. The EEG data in this stage were used to obtain the optimal frequencies, spatial channels, and the action of MI, maximizing the distinction between different MI tasks. These features would be used in the training experiments.

There were two sessions in this stage, and each session was divided into nine runs. In the MI without feedback experiments, the subjects were asked to imagine three different actions. From run 1 to run 3, the MI action was the flexion and extension of the forearm (Figure 5A, action 1). The opening and closing of the hand were imagined from run 4 to run 6 (Figure 5B, action 2). In the last three runs, the subjects were asked to imagine an action they are conversant with. Each run consisted of thirty consecutive trials separated from each other by 3 s. The timing scheme of an individual trial is given in Figure 5C. The screen was black at the beginning of a trial (t = 0 s), and the subject was instructed to stay relaxed and still. After three seconds (t = 3 s), a black arrow, which pointed to one of the four directions (left, right, up, or down, corresponding to movement imagination of left hand, right hand, both hands, or relaxing, respectively) randomly, was shown in the middle of the black screen, together with a 0.2 s auditory beep to prompt the subject to start MI. The cue was held for 4 s and the subjects were instructed to start MI according to the cue. After four seconds (t = 6 s), the screen would return to black and the subjects had three seconds to relax before the next trial.

2.2.3. Virtual Ball Movement Control Training

To improve the ability to control the robotic arm continuously based on MI, the subjects were asked to perform the virtual ball control training by MI. We designed the training experiments with progressively increasing difficulty, and each subject’s MI action, frequency, and main electrodes were all set according to the results of the MI without feedback experiments.

As shown in Figure 5D, the training experiments were divided into three stages. In the first stage, the subjects were instructed to control the virtual ball move upward or downward in one dimension (1D, UD) by MI. In the second stage, the virtual ball was moved left or right (1D, LR). In the third stage, the virtual ball was moved in two dimensions (2D). The subjects were instructed to imagine right-hand movement, left-hand movement, both-hand movement, or relaxing to control the virtual ball movement to the right, left, up, or down, respectively. Each session consisted of 6 runs, and there were 44 trials of virtual ball movement in each run. In the 1D virtual ball training stage, only when the average correct rate in three consecutive runs for one session reached 90% could subjects start the next training stage. The condition for completing the 2D virtual ball control training of each subject was that the average correct rate in three consecutive runs exceeds 80%. At the beginning of each trial (t = 0), there was a 2 s preparation period with a black screen for the subject. When t = 2 s, the pink target block randomly appeared at different fixed locations on the screen. After 2 s (t = 4 s), the pink virtual ball appeared on the center of the screen and began to move according to the subject’s MI-based EEG. Each subject was given up to 10 s in each trial to move the virtual ball to hit the target block. Once the virtual ball hit the target, the color of the virtual ball and target would change from pink to yellow. There would be a 3 s break before the next trial.

Due to the individual differences, each person’s training time was different, depending on their ability and performance, but there were no more than 7 sessions for each stage. Each subject was not trained on the same day, and the average interval between sessions was 5.16 ± 2.47 days. All seven subjects participated in the first and second stages of the training experiments, but subject 7 dropped out of the third stage of training and the rest of the experiments due to scheduling conflicts. Extra sessions were designed to estimate the chance performance for each training stage; to be more specific, there were 528 trials for each training stage. The experimental paradigm was the same with 1D and 2D training experiments. However, the virtual ball was controlled by random EEG signals. All the virtual ball control experiments in this study were presented by an official stable version of BCI2000 software [44].

2.2.4. Online Robotic Arm Control Experiments

The experimental environment of the online experiment is displayed in Figure 6A. The monitor was divided into the virtual ball movement area and the USB camera display area. The online experiments consisted of three tasks with increasing difficulty, and the experiment time of each task was less than two and a half hours, including the rest time. A square area (60 cm × 60 cm) was defined on the table as the workspace to place the targets and obstacles in this experiment, as illustrated in Figure 6B. For each trial, the initial position of the robotic arm end-effector was in the center of the workspace. When the experiment started, the subject had a waiting time of 3 s to calm down and prepare for MI. The virtual ball movement area on the monitor was blank. After 3 s, the virtual ball appeared at the center of the virtual ball movement area and began to move based on participants’ MI-based EEG.

Meanwhile, the position of the virtual ball was sent to ROS in real time to drive the robotic arm to move. When the virtual ball hit the area boundary or the movement reached 10 s, the virtual ball would stop moving. Consequently, the robotic arm would temporarily stop moving. Next, the subject needed to use the eye tracker to drive the mouse movement on the screen, and select the target on the screen. Once the target was chosen, its position would be transmitted to the robot arm control system immediately to drive the robotic arm to grasp the target automatically. During the entire process of the experiment, the robotic arm control system would detect the position of the obstacle through the depth camera and direct the robotic arm to bypass the obstacle.

In task A, the experiment was one target without an obstacle. Subjects were asked to complete only one target-grabbing task without any obstacles in the workspace. Eighty random locations were selected in the workspace to place the target block, and only one block was placed on the workspace in each trial. Therefore, there were 80 single trials for each subject to complete in this task. When the movement of the robotic arm was temporarily stopped due to the end of EEG control, it would be regarded as a failure trial if there was no target in the view of the camera. Apart from this, if the grasp failed due to the error of visual guidance or the robotic controller, the trial also would be defined as a failure.

In task B, there was one target with an obstacle. A cylindrical obstacle with a radius of 2.5 cm and a height of 25 cm was placed in the workspace. Only one target was placed in the workspace in each trial. Moreover, the cylindrical obstacle was placed at a position between the initial position of the robotic arm end-effector and the target block to ensure that the robotic arm needed to avoid the obstacle during movement. This task included the same target location and experiment sequence as task A. The definition of failure trial was the same as in the first task.

In task C, the experiment of four targets with an obstacle was performed. Compared with task B, three additional blocks were placed around the target, and the distance between two target blocks was less than 3 cm. In order to distinguish which one was the target, only one green block was placed as the target, and the other three blocks were other colors. In each trial, subjects were asked to grasp the green block. The obstacle was placed the same as that in task B.

3. Results

3.1. Performance of MI without Feedback

After the motor imagery without feedback experiments, the best placement of the electrodes, MI action, and frequency for each subject were selected and are reported in

Table 1. The best features of all the subjects in the experiment.

Subject ID	Main Electrode	Actions of Hands	Frequency (Hz)
A	C3, C4	Action 2	9
B	CP3, CP4	Action 2	9
C	C3, C4	Action 1	12
D	C3, C4	Action 1	10
E	C3, C4	Other action	9
F	CP3, CP4	Other action	12
G	C3, C4	Other action	12

, according to the highest R² value under different MI actions. It can be found that the best features of different subjects vary from person to person. The best feature was obtained from a frequency band which can be uniquely identified by a center frequency and a bandwidth of 3 Hz. The best electrode on the left is C3 or CP3, and the best electrode on the right is C4 or CP4. As is shown in Table 1, all of the frequency bands were chosen from the mu rhythm.

3.2. Performance of Virtual Ball Control Training

All seven subjects participated in the 1D virtual ball control training. In the 2D training stage, subject G withdrew from the experiment due to scheduling conflicts. The average success rates of each subject for virtual ball movement control before and after training were calculated. Suppose that T_s is the number of the success target hit trials and T_All is the total number of the trials. The average success rate was defined as follows:

$$R=\frac{T_S}{T_{All}},$$

(6)

which is different from another study [13]. They defined an indicator excluding those trials when neither the correct target nor an incorrect target was hit. Moreover, we defined the failed trials corresponding to those tasks when the virtual ball hit the wrong target or failed to hit any target within the limited time. Meanwhile, the completion time of all the trials was recorded, which was defined as the duration from when the virtual ball started to move to when it hit the target.

The average success rates of each subject in 1D UD, 1D LR, and 2D virtual ball movement control are displayed in Figure 7A–C. In the first session of 1D UD virtual ball control training, the average success rate for all the subjects was 41.7% ± 14.8% and rose to 89.1% ± 5.4% at the last training session. The average success rate increased by nearly 115% for 1D UD training. The average success rate increased from 45.9% ± 17.3% in the first session to 88.2% ± 5.3% in the last session and had a more than 90% increase in the 1D LR virtual ball control training. The 1D control training was shown to be helpful to subjects’ performance in 2D control training. The average success rate at the first session for 2D virtual ball movement control (49.9% ± 9.2%) was higher than that at 1D virtual ball movement control. In the last session of 2D virtual ball control training, the average success rate improved by over 60% compared to the first session.

Due to the differences between individuals, the number of training sessions needed for each participant to achieve the goal of each training stage was different. Among all the subjects, subject B participated in 17 sessions for virtual ball control training and had the maximum participation rate. On the contrary, subject E had the least total number of training sessions, with only 11 sessions. The chance level performance is 22.1% for 1D virtual ball control and 15.2% for 2D virtual ball control, which is indicated by red dotted lines in the figure. After training, the average success rate of the virtual ball control for all the subjects is much higher than the chance level. The three graphs in Figure 7D from top to bottom show the average completion time in success trails for virtual ball control training in 1D UD, 1D LR, and 2D, respectively. The average completion time for 1D UP virtual control training for all the subjects in the first session was 6.7 ± 0.4 s, which was reduced to 5.5 ± 0.2 s in the last session. Similarly, as the training continued, there was a gradual reduction in the average completion time for the 1D LR and 2D virtual ball control training, reaching 5.3 ± 0.2 s and 5.4 ± 0.1 s, respectively.

Figure 8A provides the changes in the maximum sensorimotor R² values before and after the virtual ball control training. The maximum sensorimotor R² values were evaluated as the regression output between the task conditions and the EEG power. Before the training, the average maximum R² value of the subjects between the MI of both hands and relaxation was 0.290. The R² value between the MI of the left hand and right hand was 0.242. After a series of training sessions, the average maximum R² values increased to 0.657 and 0.594, separately. The statistical results revealed that the virtual ball training could significantly improve the correlation between EEG and different MI labels. The R² topographies for subjects at 12 Hz before and after the virtual ball control training are shown in Figure 8B. The areas with larger R² values were concentrated near the electrodes of C3 and C4, which were the same as the sensorimotor area stimulated by MI. After the training, the R² values increased significantly, but the area where the maximum R² values were located did not change.

Figure 8C,D shows the group-level average voltage spectra across all the subjects and sessions at the C3 and C4 electrodes before and after virtual ball control training. It should be noted that the amplitude difference between different mental states (MI of both-hand movement versus relaxing and left-hand movement versus right-hand movement) is focused at mu rhythm. From Figure 8C, we can observe that the amplitudes at C3 and C4 electrodes are smaller during the subjects imagining both their hands moving than during the subjects imagining relaxing, which represented the ERD phenomenon. After the virtual ball control training, the difference in EEG amplitude between the imagination of both hands and relaxation increased significantly at the mu rhythm. As can be seen from Figure 8D, before training, it is difficult to distinguish the EEG amplitude at C3 and C4 electrodes between the imagination of the left hand and right hand. After training, on both sides of the sensorimotor cortex, the difference in EEG amplitude becomes higher than before training.

In this study, we further analyzed the changes of EEG topography on the subject’s sensorimotor cortex in the different training stages. Figure 9 demonstrates ERD/ERS topographical map in the mu frequencies of subject D during the three stages of virtual ball control training. From the last session, the contralateral ERD and ipsilateral ERS were clearly observed from the topographical map when the subject imagined unilateral hand movement, and the power of the contralateral motor cortex was suppressed compared with the relaxation status. When the subject imagined both-hand movement, the bilateral ERD also could be clearly observed in the last session. However, this ERD/ERS phenomenon was not obvious in the first session, and the situation improved in the middle session. As the training continued, the energy of the sensorimotor cortex on both sides of the brain in the mu rhythm increased when the subject was in the relaxed status, and the contralateral ERD and ipsilateral ERS became more obvious in the movement imagination. Combined with the changes of R² values and voltage spectra, these changes provided convincing evidence that the enhanced behavioral improvement shown in the virtual ball control training was accompanied by coincident neural changes in sensorimotor modulation.

3.3. Performance of Online Experiments

Figure 10 displays the performance of online experiments, where the subjects were instructed to grasp the target via the robotic arm. It should be noted that there were three steps for the subjects to complete in the online task using our hybrid control system. In the first step, the movement of the robotic arm was accompanied by the EEG-controlled virtual ball movement (the robotic arm was continuously controlled to move to the surrounding area above the target on a 2D plane). In a subsequent step, the target was detected by computer vision and selected by the subject’s gaze point. In the last step, the robotic arm reached and grasped the target automatically. The average success rates of the three tasks in the online experiment were 90.2%, 89.2%, and 82.3%, respectively, as shown in Figure 10A. All the unsuccessful trials in the online experiments were due to the error of EEG control or the wrong target being selected through the eye-tracker. There was only one target in task A and task B, and the only difference was that there was an obstacle in the workspace in task B. The average success rate and the average completion time of successful trials for all the subjects in task A and task B were close, and the main reason was that both tasks had only one target in each trial and the robotic arm could always avoid obstacles successfully. The success rate was reduced to 82.3% and the average completion time of successful trials increased to 23.5 ± 3.2 s in task C mainly because the subjects had to select the correct target from four crowded blocks on the screen. Subject C and subject E had a higher success rate than other subjects in the online experiment and spent less time than other subjects to complete the task.

The trajectory samples of the robotic arm for the online experiments of one target with an obstacle are displayed in Figure 10C for all six subjects. The red circle in the figure represents the position of the obstacle, and different colors are used to distinguish the trajectories of different targets. The red line indicates that the robotic arm grasped the target automatically. Some EEG-controlled trajectories closed to the target directly while others had redundancy. Once the target was selected via the user’s gaze point, there was no more redundancy in the remaining trajectory. The workspace was divided into 25 regions of equal size, and the success rate distribution for each region of the workspace across all subjects and all online experiment tasks is shown in Figure 10D. The middle of the workspace had a lower success rate of 69.4%. The four regions with the highest success rate were in the middle of the four edges of the workspace, mainly due to the fact that these regions were similar to the positions where the target appeared in the 2D virtual ball control training.

The EEG signals of all subjects and all tasks were recorded to demonstrate more details in the online experiments. We analyzed the EEG signals of the C3 electrode over the left hemisphere and the C4 electrode over the right sensorimotor cortex from these trials, as shown in the trajectory sample in Figure 10C. Figure 11 shows the time–frequency spectral changes for these trials from −2 to 6 s, and the movement of the robotic arm is guided by EEG during these periods. The vertical dotted black line at 0 s indicates the time when the virtual ball appears on the screen, and the virtual ball and robotic arm begin to move at 1 s. The baseline spectra were estimated from the EEG signal selected from −2 to 0 s during which the subject remained in a holding state. Note that the ERD (blue color) and ERS (red color) patterns can be observed from around 0.3 to 0.5 s after the virtual ball appears. Strong ERD/ERS can be seen in mu bands. Moreover, weak ERD in the beta band also existed.

4. Discussion

In the previous studies on the control of assistive manipulators based on BCI to assist patients suffering from motor impairment, the invasive BCI approaches have shown that control of the multi-DOF robotic devices to complete daily activities in a laboratory setting is possible [3,11,12]. Nevertheless, such BCI systems driven by invasive technology are limited by the medical risks associated with surgery and implantation of the microelectrode array in the cortex. Currently, the EEG signals recorded via noninvasive BCI technology have a low signal-to-noise rate. Therefore, the control of robotic arms with many degrees of freedom is still difficult, especially in a complex environment with obstacles, which restricts the practical application of noninvasive BCI in the control of robotic arms. Some studies have come up with another method based on hybrid BCI, which combines MI with evoked BCIs to control the robotic arm. However, the evoked BCIs will cause distraction and brain burden. Additionally, EEG was combined with electromyography (EMG) to control external devices in previous research [18,23].

In this study, a group of healthy subjects started from the simplest MI without feedback experiment and gradually mastered the control of a virtual ball’s movement in the 2D plane through MI after a series of virtual ball control training experiments with progressively increasing task difficulty. Due to the relatively lower control accuracy of the robotic arm using MI independently, the users only need to move the robotic arm to the approximate area above the target through BCI and then select the target through the gaze point to obtain the accurate position of the target. Finally, the robotic arm performs automatic grasping. The online control implemented in this study is quite similar to that required in daily activities, where there are obstacles in the workspace with the target surrounded by other objects. The experimental results have demonstrated that all the subjects can control a robotic arm to finish the online tasks using noninvasive BCI combined with computer vision technology in complex environments.

The results of the MI without feedback for all the subjects show that EEG is different when subjects imagined different actions, which is consistent with the previous research [45]. The MI without feedback experiments are conducted to select the best placement of the electrode, MI action, and frequency for each subject. In this way, the control performance of the robotic arm for the participants is greatly improved.

ERS and ERD have been considered to be the product of specific cognitive processes, which are associated with the motor nervous system [46]. The modulation of the mu rhythm is quantified by the R² values and the topographic map (Figure 8A and Figure 9). As the training continued, we found that users’ ability to control the virtual ball gradually improved. Moreover, the ERD/ERS was more obvious over time, indicating that the MI training with feedback could regulate the movement rhythm of subjects. This further validated that MI training could promote cognitive processes and might be essential for BCI control [47].

Throughout the virtual ball control training experiments, we observed that there were significant differences in the ability of different subjects to control the virtual ball. The success rate of subject G after five training sessions in 1D UD was 62.5 ± 13.2%, while the success rate of subject C training for the first time reached 62.5 ± 9.2%. We also found that the performance of the online experiment was directly affected by the virtual ball control training performance. The success rate of subject E in online experiment task A was up to 95.0%, and the average success rate for 2D virtual ball was 87.1%. The subject also reported that it was easier for her to control the virtual ball and the robotic arm. We noticed that the experiment performance was also related to the state of the subjects during the experiment. The success rate of the subjects in the bad state (an inattentive or distracted state) in the virtual ball control training and online experiment would decrease compared with the previous session.

Several former typical hybrid BCI studies focused on decoding EEG signals to discrete commands [14,32]. It is a challenge to generate a continuous control signal [48]. Our system was based on an online regression model of MI-EEG, providing natural, continuous, and efficient operations. The use of computer vision and eye-tracker can also reduce the brain burden caused by long-term mental activities. To sum up, the proposed continuous hybrid BCI system showed potential in complex reach and grasp tasks with obstacles.

5. Limitations and Future Work

The proposed system was designed to be user-friendly for potential users, including healthy people and the disabled. The aim of our work is to investigate the potential application of the proposed continuous hybrid BCI system. Due to the MI-based BCI systems’ certain requirement for users to master motor imagination, the MI without feedback stage (Section 2.2.2) and the virtual ball movement control training stage (Section 2.2.3) were designed to obtain the optimal frequency-channel parameters of the proposed system and make naïve subjects master the MI paradigm. This method was verified and performed well for both naïve healthy subjects and real patients to master 1D and 2D MI control in previous studies [5,49,50,51].

The proposed system still has several limitations to be settled for practical applications. On the one hand, real patients were not recruited for the experiment. For the present, the subjects in our experiments were all healthy people with functional limbs. Several former works have also designed different kinds of BCI control strategies and verified the performance of their methods on healthy subjects [13,15,33,37]. Even though the MI-based training method was verified and performed well for both naïve healthy subjects and real patients, it would be better to further verify whether the proposed hybrid BCI system is effective for real patients. On the other hand, the targets grasped in online experiments are simple at present, but the shapes of the targets that the users intend to grab are various in reality. Moreover, the robotic arm is fixed on the desktop to finish the tasks in a local field, but it cannot grab a target outside the workspace.

We are also trying to make some additional effort to enhance this system and expand the continuous hybrid control strategy to more complex scenarios in the future. Practical functional modules (e.g., a mobile platform or a wheelchair) could be integrated into the BCI system to extend the system to broader scenarios in daily lives. In addition to expanding the function of such BCI systems, it is necessary to recruit real patients to participate in the following research. After all, healthy and disabled subjects are both the end-users of the proposed system.

6. Conclusions

In this study, we have proposed a noninvasive EEG-based hybrid BCI control system for a robotic arm. Such a system enables subjects to complete reach and grasp tasks in a 3D environment with obstacles. After a period of MI training with feedback, all the subjects mastered the ability to control the virtual ball on a two-dimensional plane via BCI. With the assistance of gaze tracking and obstacle detection, the subject can control the robotic arm to pick up a target and avoid obstacles accurately. The subject only needs to control the robotic arm to continuously move to the approximate area above the target on a 2D plane through MI, which will reduce the user’s brain burden and improve the accuracy of the task. The experimental results demonstrate that all subjects can complete the online tasks with a high success rate through the proposed robotic arm control system. Moreover, we also find that the motor imagery can promote cognitive processes, which may be of great significance for patients’ neural rehabilitation training in the future. The proposed system combining BCI with computer vision, gaze detection, and semiautonomous guidance drastically improves the accuracy of reach and grasp tasks and reduces the brain burden caused by long-term mental activities.