Next Article in Journal
Propofol Exposure in Pregnant Rats Induces Neurotoxicity and Persistent Learning Deficit in the Offspring
Previous Article in Journal
The Neurodevelopmental Impact of Neonatal Morphine Administration
 
 
Correction published on 19 September 2014, see Brain Sci. 2014, 4(3), 488-508.
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

Subject Combination and Electrode Selection in Cooperative Brain-Computer Interface Based on Event Related Potentials

1
School of Computing and Intelligent Systems, University of Ulster, Derry BT48 7JL, Northern Ireland, UK
2
GIPSA-lab CNRS UMR 5216, Grenoble Universities, Saint Martin d'Hères 38400, France
*
Author to whom correspondence should be addressed.
Brain Sci. 2014, 4(2), 335-355; https://doi.org/10.3390/brainsci4020335
Submission received: 17 January 2014 / Revised: 18 March 2014 / Accepted: 20 March 2014 / Published: 30 April 2014

Abstract

:
New paradigms are required in Brain-Computer Interface (BCI) systems for the needs and expectations of healthy people. To solve this issue, we explore the emerging field of cooperative BCIs, which involves several users in a single BCI system. Contrary to classical BCIs that are dependent on the unique subject’s will, cooperative BCIs are used for problem solving tasks where several people shall be engaged by sharing a common goal. Similarly as combining trials over time improves performance, combining trials across subjects can significantly improve performance compared with when only a single user is involved. Yet, cooperative BCIs may only be used in particular settings, and new paradigms must be proposed to efficiently use this approach. The possible benefits of using several subjects are addressed, and compared with current single-subject BCI paradigms. To show the advantages of a cooperative BCI, we evaluate the performance of combining decisions across subjects with data from an event-related potentials (ERP) based experiment where each subject observed the same sequence of visual stimuli. Furthermore, we show that it is possible to achieve a mean AUC superior to 0.95 with 10 subjects and 3 electrodes on each subject, or with 4 subjects and 6 electrodes on each subject. Several emerging challenges and possible applications are proposed to highlight how cooperative BCIs could be efficiently used with current technologies and leverage BCI applications.

Graphical Abstract

1. Introduction

The field of non-invasive Brain-Computer Interface (BCI) has been particularly active during the last two decades thanks to pioneering works such as the P300 speller [1], which opened the path to a new research field combining neuroscience, signal processing, machine learning and human-computer interaction. Over these years, BCIs have mainly been used as a way to control a computer and to use different type of devices. In all these contexts, the interaction between the subject and the devices has always been in relation to the subject’s will. The paradigms have largely been related to communication purposes: to send commands to a computer, e.g., spelling words [2], and in assistive technologies [3]. Due to the low signal-to-noise ratio (SNR) of the electroencephalogram (EEG) signal, many strategies have been proposed to increase the SNR, e.g., the combination of several trials over time thanks to the repetition of different events [4]. In the rest of the paper, we distinguish trials from events. A trial corresponds to the EEG signal recorded during an event, which is time locked. Several trials can be recorded for the same event, i.e., from several subjects; several trials can be recorded for different events that have the same meaning, e.g., the presentation at different times of the same item in a P300 speller.
Whereas combining EEG signals over time has been used since the first P300 speller [1], where the stimuli appearing on the items of the speller layout are repeated several times, combining EEG signals across subjects in BCI has not been efficiently used until recently for decreasing reaction time [5,6], and for 2-D pointer control with two users [7]. In this type of BCI, several users are implicated for enabling a command. In [8], the activity of multiple brains was analyzed to reveal causal connections between regions of different brains (hyperconnectivity). In [9], it was suggested that multi-brain computing can be used as a technique to rapidly, in parallel, gather increased information about the environment, and to access collective perceptual/cognitive choices and mental states. In [10,11], the interest of combining behavioral responses and neural features was shown as a means to improve accuracy when compared with individual performance.
Although it seems that combining EEG signals across subjects is identical to the combination of EEG signals over time at the signal processing level, we show that cooperative BCIs need specific paradigms, and should be used only in particular conditions. By considering a state-of-the-art experiment with additional constraints due to the cooperative feature, we address in this study two key questions: (i) what is the best combination method for combining decisions across subjects; and (ii) to what extent is there a difference of performance between considering as inputs a large set of electrodes on a single subject and a small set of electrodes with several subjects.
The remainder of this paper is organized as follows. First, we review in Section 2 some current issues and bottlenecks in BCI that limit commercial applications. In Section 3, we define the notion of cooperative BCI, and its difference compared with classical BCI paradigms. Then, the performance obtained by combining signals across subjects is presented for the detection of event-related potentials (ERPs). This experiment is used to determine the best trade-off between the number of subjects and the number of electrodes common to each subject to optimize the overall performance, i.e., is it better to consider a single subject wearing an EEG cap with eight electrodes, or four subjects with each one having only two electrodes? Finally, the future challenges and possibilities of cooperative BCI are detailed in Section 6.

2. BCI Issues

In spite of all the recent advances in non-invasive BCI, there remain a long list of issues that prevent BCIs from being used efficiently by healthy people. In order to improve the reliability of BCI, several paths have been explored. First, new signal processing and machine learning techniques have been proposed to increase the performance of detecting brain responses. These methods currently address the challenges related to the non-stationary characteristics of the EEG signal and the high variability that can happen over time and across subjects [12,13,14]. For instance, the performance of the P300 speller can be increased by using intersubject information and online adaptation [15], by sensor selection [16,17], and by decreasing the number of repetitions [18]. Despite all such improvements, the accuracy of single-trial detection remains far from sufficient. In addition, improvements can come from an optimization of the BCI in relation to the application, e.g., word completion and prediction for virtual keyboards [19]. Second, to face the obstacles related to the poor performance, BCIs have been combined with other communication devices or multiple BCIs. This idea was introduced as the hybrid BCI concept [20,21]. In hybrid BCIs, it is important to distinguish pure hybrid BCIs where the control remains fully BCI dependent and other hybrid systems that embed other physiological signals or communication devices.
Moreover, commercial non-invasive BCI remains limited to a general audience due to the lack of efficient systems with dry electrodes [22,23]. It is, in fact, difficult to obtain a good signal on the hairy parts of the head. This is why new studies explore the effect of electrodes on non-hair-bearing areas [24]. Companies such as NeuroSky, Neuroelectrics, Quasar [25], G.tec with g.Sahara, and Mindo from the Brain Research Center (BRC) of the National Chiao Tung University, Taiwan [26], have proposed portable solutions with dry electrodes for recording EEG signals. These efforts show that issues related to the uncomfortable aspect of BCI tend to be solved in the near future and the remaining issue would become the BCI performance only (speed and accuracy).

3. Cooperative BCI

3.1. Extending the Notion of BCI

While hybrid BCI [21] has extended the concept of BCI, we propose here to classify a BCI according to the number of brain response types that are involved in the system and to the number of people that are involved in the task. Hence, four types of BCI can be distinguished:
  • The classical BCI: it involves a single person and a single brain modality, i.e., a single type of brain response (ERP or SSVEP, motor imagery, …) (see Figure 1a).
  • The hybrid BCI: it involves a single person, and the possibility to consider several brain modalities (see Figure 1b).
  • The cooperative BCI: it involves a group of people, and a single brain modality (see Figure 1c).
  • The cooperative-hybrid BCI: it involves a group of people, and the possibility to consider several brain modalities. The type of brain response can depend on the subject, i.e., the BCI can be hybrid only for some subjects (see Figure 1d).
Figure 1. Examples of different BCI types.
Figure 1. Examples of different BCI types.
Brainsci 04 00335 g001
In current BCIs, the subject typically interacts with the computer by sending commands through the detection of brain responses. Those commands correspond to the user’s will: the subject is the only person controlling the application. In a cooperative BCI, the decision is based on the combination of brain signals from multiple users [27,28]. Cooperative BCIs can be seen as particular types of pure hybrid BCIs. Instead of having a system accepting inputs from a single subject and several types of signals (e.g., P300 and SSVEP) (see Figure 1b), a cooperative system (hybrid or not) accepts inputs from several subjects and one or several types of signals (see Figure 1c,d). In this sense, the BCI accepts different inputs coming from different subjects. Contrary to pure hybrid BCIs, where the user may have to pay attention to several types of stimuli, i.e., a flickering light for SSVEP, and an oddball paradigm for ERPs, each subject can focus his attention on a single type of stimulus in a cooperative BCI. Since differences exist between cooperative BCIs and current BCIs, we would like to distinguish single-trial and single-event detection. In single-trial detection, the response is detected with only one trial, whereas in single-event detection the event can be represented by a series of trials that were recorded across several subjects.
It is worth noting that cooperative BCIs cannot be used for applications depending only on the subject’s will. For instance, for virtual keyboard applications, the selection of the letters that are written depends only on the person who is writing the text (it is possible to do copy spelling in cooperative mode but it is not the main goal of a BCI virtual keyboard). A typical scheme of a cooperative BCI based on the detection of event-related potential is presented in Figure 2. In this example, a pool of subjects shall pay attention to a stimulus and the detection of this stimulus enables a command, which provides a feedback to the pool of subjects. In order to have all the subjects paying simultaneous attention to the stimulus, an adapted task must be chosen. Yet, this situation can happen in the case of an emergency signal where an event triggers a particular response across all subjects.
Figure 2. Cooperative BCI based on the detection of ERPs.
Figure 2. Cooperative BCI based on the detection of ERPs.
Brainsci 04 00335 g002

3.2. Combination Strategies

The literature of multi-classifier systems offers a large choice of methods for combining the signals and/or the decisions from different subjects [29,30]. Increasing the number of subjects can have different impact on the performance. By adding more subjects, it is possible to add subjects who have had bad performances: although their individual performance does not allow them to use a BCI as a single user, their brain features can still somewhat contribute to the global performance. The choice of the combination strategy may depend on how possible it is to find subjects that can complement each other, particularly for adaptive combination methods. In this study, we limit the applications to strategies for binary classification.
Let Oi denote the output value of a binary classifier for subject i at a particular trial. Oi (0 ≤ Oi ≤ 1) represents the confidence value of assigning the input to one of the two classes. Six classical combination scores over Nsub subjects are considered:
  • The sum of the classifier outputs (Osum):
    O sum = i = 1 N sub O i
  • The weighted sum of the classifier outputs (Ow_sum):
    O w sum = w 0 + i = 1 N sub w i O i
  • The product of classifier outputs (Oprod):
    O proud = i = 1 N sub O i
  • The selection of the maximum score (Omax):
    O max = max 1 i N sub O i
  • The selection of the minimum score (Omin):
    O min = min 1 i sub O i
  • The voting (Ov(k)):
    O v ( k ) = { 1 , if i = 1 N sub D i ( α i ) k 0 , otherwise
    where Di(α) is the binary decision associated with the i-th subject defined by
    D i ( α ) = { 1 , if O i > α 0 , otherwise
    and α is a threshold for the decision to assign the input signal to the target or non-target class. Two particular cases of voting can be distinguished: the majority vote where k = Nsub/2, and the consensus vote where k = Nsub, which are denoted Omaj and Ocons, respectively.
It is worth mentioning that these scores are related to different intuitive strategies. Hence, the maximum score (Omax) and the minimum score (Omin) are related to the notions of OR and AND in logic, respectively. The maximum score makes its decision based on the most relevant subject (i.e., with the highest individual score Oi): a single relevant subject is sufficient to provide a high combined score. On the contrary, the minimum score makes its decision based on the least relevant subject (i.e., with the lowest individual score Oi): all the subjects have to be relevant to provide a high combined score. Moreover, the sum (Osum) and weighted sum (Owsum) scores can be seen as smooth or weighted OR, while the product score (Oprod) is a smooth or weighted AND.
The choice of the combined score depends not only on these considerations but also on the distribution of the estimation errors. Kittler et al. [31] showed that for Gaussian distributions it is better to use the sum (or mean) strategy than majority vote (i.e., Ov(k) with k = Nsub/2). Yet, for heavy tail distributions, the majority vote can give better results than the sum.

3.3. Performance Evaluation

To show the relevance and the type of performance that can be expected by combining the decision from several individuals, we consider data obtained from a P300 speller experiment where the goal is to detect ERPs. The performance will be reported with the Area Under the ROC curve (AUC) [32] and the information transfer rate (ITR) [33] in bits per minute (bpm) defined by:
ITR = 60 T ϑ
where ϑ is the capacity of the channel between the interface and the user’s brain:
ϑ = log 2 ( N out ) + P log 2 ( P ) + ( 1 P ) log 2 ( 1 P N out 1 )
with P being the probability of a good detection (i.e., the accuracy), Nout being the number of possible different outputs, and T being the time in seconds of recorded EEG signal that is required to make the decision among the Nout outputs.
In addition, the ITR of a cooperative BCI can be normalized to take into account the number of people involved in the decision, as the overall performance depends on the number of people involved in the task, the performance of the detection, and the time to perform the task. We define the normalized ITR by:
ITRP f = 60 T f ( N sub ) ϑ
where f(Nsub) is a cost function that takes as argument the number of subjects, and f(l) = 1. As amplifiers have typically a number of electrodes that is a power of 2, we consider f(Nsub) = 1 + log2(Nsub).
Since the performance of a BCI depends on the number of electrodes, and the number of electrodes largely determines the cost of a BCI system, the number of electrodes becomes a pertinent parameter for the performance estimation.

4. Single-Event ERP Detection

4.1. Participants and Study Design

To evaluate the effect of combining several subjects, we considered classifier outputs from ten healthy subjects (age = 25.5 ± 4.4 years old, three females) that were acquired during P300 speller experiments. Although the P300 speller is not the typical cooperative BCI application, the P300 experiment represents the most common paradigm in BCI based on ERP detection, and it has become a type of benchmark experiment for evaluating new BCI approaches. In addition, there may be applications that would require several people to spell the same word. Each subject had to spell a total of 40 characters, and the experiments were carried out sequentially. Because subjects could be at different location and the final results may be transferred to a different person, the sequential experiment was not an issue for simulating the performance of a cooperative BCI. The most important characteristic of the experiments was that the characters to spell and the order of the stimuli were identical across subjects. Hence, each subject observed the same sequence of visual stimuli. The matrix of the P300 speller was 6 × 6 and displayed on a 27 inch LCD screen with a brightness of 375 cd/m2. Subjects sat on a chair at about 60 cm from the screen in a non-shielded room. The stimulus onset asynchrony was set to 133 ms and the inter-stimulus interval was 66 ms, the signal was recorded on O1, O2, P3, p4, p7, p8, pz and FCz. F7 and F8 were dedicated to the ground and the reference, respectively. The amplifier was a FirstAmp (Brain Products GmbH) with a sampling frequency of 2 kHz.

4.2. Signal Processing and Classification

The signal was first bandpass filtered (Butterworth filter of order 4) with cutoff frequencies at 1 and 12.5 Hz, then downsampled to 25 Hz. The next processing step was spatial filtering with the xDAWN method, which maximizes the signal to signal-plus-noise ratio [34,35]. The technique has been successfully applied in P300-based BCI and in RSVP tasks [16,36]. Let us denote the spatial filters by U ∈ ℝNs×Nf, where Nf is the number of spatial filters. The signal after spatial filtering is defined by Xfilt = XU, where X ∈ ℝNt×Ns is the recorded signal, Nt is the number of sampling points, and Ns is the number of electrodes. An algebraic model of the enhanced signals XU is composed of three terms: the ERP responses on a target class (D1A1), a response common to all stimuli, i.e., all targets and non-targets confound (D2A2) and the residual noise (H, a real matrix of size Nt × Ns), which are filtered spatially with U.
X U = ( D 1 A 1 + D 2 A 2 + H ) U
where D1 and D2 are two real Toeplitz matrices of size Nt × N1 and Nt x N2, respectively. D1 has its first column elements set to zero except for those that correspond to a target onset, which are represented with a value equal to one. For D2, its first column elements are set to zero except for those that correspond to stimuli onset. N1 and N2 are the number of sampling points representing the target and superimposed evoked potentials, respectively. A1 ∈ ℝNNs and A2 ∈ ℝN2 × Ns represent the ERP response on a target class and the ERP response on every stimulus, respectively. The spatial filters U maximize the signal to signal-plus-noise ratio (SSNR):
SSNR ( U ) = arg max U T r ( U T A ^ 1 T D 1 T D 1 A ^ 1 U ) T r ( U T X T X U )
where Â1 corresponds to the least mean square estimation of A1.
A ^ = [ A ^ 1 A ^ 2 ]
A = ( [ D 1 ; D 2 ] T [ D 1 ; D 2 ] ) 1 [ D 1 ; D 2 ] T X
where [D1;D2] is a matrix of size Nt × (N1 + N2) obtained by concatenation of D1 and D2. Spatial filters are obtained through the Rayleigh quotient by maximizing the SSNR [34]. During the experiments, four spatial filters (Nf = 4) were used. The input vector for the classifier was obtained by the concatenation of the Nf time-course signals across spatial filiters for a single subject. The detection score, Oi, was obtained with the Bayesian linear discriminant analysis (BLDA) classifier [37]. The training and test database contained 4800 trials for each subject (800 for target, 4000 for non-target). We evaluated the impact of the number of subjects on single-event detection by considering the classifier outputs from the different subjects with Monte-Carlo simulations (combination of trials corresponding to same events, e.g., the presentation of a visual stimulus, and combination of trials from different subsets of subjects).

4.3. Results

For single-event detection, the mean AUC across subjects is 0.810 ± 0.067. The results for single-trial detection are depicted in Figure 3 and Figure 4; they represent the mean AUC, accuracy, ITR, and ITRPf across all the possible combinations of n subjects (Figure 3a,c and Figure 4a,c), and across all the possible combinations of n trials from the same subject (Figure 3b,d and Figure 4b).
The results presented in Figure 3a confirm that aggregating subjects can improve the classification accuracy estimated by AUC. For instance, with the sum combined score (Osum) the AUC increases from 0.810 ± 0.061 with a single subject to 0.995 ± 0.000 with ten subjects. However, as expected, the AUC achieved with the product (Oprod) and minimum (Omin) scores do not monotonically increase with the number of subjects: there is a maximum AUC at about five subjects. Consequently, as already noted, these combined scores require that all the individual scores be relatively high to achieve a good detection while increasing the number of subjects reduces this probability.
Figure 3. Detection results based on the AUC and the accuracy (ACC). (a,b) AUC for single-event detection as a function of the number of subjects, and AUC for a single subject as a function of the number of trials, respectively. (c,d) Accuracy for single-event detection as a function of the number of subjects, and accuracy for a single subject as a function of the number of trials, respectively.
Figure 3. Detection results based on the AUC and the accuracy (ACC). (a,b) AUC for single-event detection as a function of the number of subjects, and AUC for a single subject as a function of the number of trials, respectively. (c,d) Accuracy for single-event detection as a function of the number of subjects, and accuracy for a single subject as a function of the number of trials, respectively.
Brainsci 04 00335 g003
The evolution of AUC as a function of the number of subjects involved in the decision (Figure 3a) and as a function of the number of trials combined over time (Figure 3b) presents the same behavior depending on the number of combined scores. The accuracy is depicted in Figure 3c,d. In both cases, the accuracy starts at about 63.17% ± 16.29% with a single-trial, and increases to about 85% with only two trials, then it reaches a plateau at 90%. Finally, the accuracy becomes 93.72% ± 1.69% with Osum and 92.74% ± 3.20% for Owsum with ten subjects.
Figure 4a shows the theoretical ITR (8) as a function of the number of subjects that are involved in the decision. The ITR was estimated based on the choice among six commands (Nout = 6), because of the matrix size and a time of 0.133 • 6 s for the selection of a command (T = 0.8 s). The best performance is obtained with the sum method (Osum) with an ITR of 158.14 ± 7.83 bpm with 10 subjects. For Osum, the ITRPf decreases with the addition of subjects to reach 36.59 bpm with 10 subjects, as depicted in Figure 4c. As expected from the AUC plots, when the AUC increases with the number of involved subjects, the ITR also increases. Contrary to combination over trials (with a single subject), which increases the time to make a decision, which is defined by T = 0.8 * #trials, this time is exactly the same when combining over subjects with a single trial for each. As a consequence, the increase of AUC is directly reflected on the ITR (Figure 4a), while it is not the case when combining over trials (Figure 4b). One can compare the aggregation of trials with sequential processes while the aggregation of subjects is more related to parallel processes. The evolution of the performance indicates a continuous increase of performance with the addition of new subjects. With about four or five subjects, it is possible to achieve a performance that is reliable enough for single-event detection.
Figure 4. Detection results based on the ITR and ITRPf. (a,b) theoretical ITR as a function of the number of subjects and the number of trials, respectively. (c) theoretical ITRPf as a function of the number of subjects and the number of trials, respectively.
Figure 4. Detection results based on the ITR and ITRPf. (a,b) theoretical ITR as a function of the number of subjects and the number of trials, respectively. (c) theoretical ITRPf as a function of the number of subjects and the number of trials, respectively.
Brainsci 04 00335 g004

5. Electrode Selection in Cooperative BCIs

The rationale behind considering several subjects is the same as considering several electrodes: the goal is to increase the number of information channels that are used in the decision. Those information channels can come from different locations on the head and/or different subjects. In this section, we propose to compare the performance obtained with different number of electrodes and different number of subjects. We evaluate the best electrode subsets from size 1 to Ns that maximizes the mean SSNR across subjects. Here, SSNR is estimated after the xDAWN spatial filtering method. Hence, the score function is estimated globally as a function of the mean grouped SSNR from the virtual electrodes. We consider two approaches for electrode selection. In the first approach (SSindividual), we determine the electrode subsets for each individual, as described in [16], hence Nsub electrode subsets are created. In the second approach (SScommon), electrode selection optimizes the mean SSNR across subjects and not the SSNR of each subject, because EEG systems with a low number of electrodes are typically fixed, and it is not possible to adjust the location of the electrodes. With SScommon, a single electrode subset is created.
In both approaches, the electrode selection procedure is based on a recursive backward elimination. Starting with all electrodes, it consists in testing each electrode for its significance across subjects and in removing the least relevant one at each iteration step. An irrelevant electrode is an electrode whose removal barely impairs the performance or selection criterion. Elimination goes on until every electrode has been eliminated. The score corresponding to the subset selection for SScornrnon is given for each electrode between brackets in Figure 5. The best electrodes are O2 and P8. The electrode selected for each subject with SSindividual are depicted in Figure 6.
Figure 5. Electrode scores after electrode selection based on the maximization of the mean SSNR across subjects (SScommon).
Figure 5. Electrode scores after electrode selection based on the maximization of the mean SSNR across subjects (SScommon).
Brainsci 04 00335 g005
The electrode subsets are estimated with data from the first session, which contain 400 patterns corresponding to the presentation of a target and 2000 patterns for the presentation of a non-target. The remaining data are considered for evaluating classifier performance (AUC). We define the AUC performance auc(i, j) by the AUC obtained by considering i electrodes, 1 ≤ iNs and j subjects, 1 ≤ JNsub. auc(i, j) is computed by considering the mean AUC with the Osum rule of all the possible combinations of j subjects (Nsub!/((Nsubj)! · j!), where! is the factorial function).
Figure 6. Electrode scores after electrode selection based on the maximization of the SSNR for each subject (SSindividual). (A low gray level represents a high score for the electrode.)
Figure 6. Electrode scores after electrode selection based on the maximization of the SSNR for each subject (SSindividual). (A low gray level represents a high score for the electrode.)
Brainsci 04 00335 g006
Figure 7 represents the AUC as a function of the number of electrodes and number of subjects that are involved in the classification for SScommon. With only one electrode and one subject, the mean AUC is 0.649. The maximum AUC (0.989) is obtained with the maximum information, i.e., eight electrodes and ten subjects. For a better understanding, Figure 7c,d depicts in black when a configuration of electrodes and subjects allows an AUC superior to a defined threshold for SScommon. Particularly, it is possible to achieve a mean AUC superior to 0.95 with 10 subjects and 3 electrodes on each subject, or with 4 subjects and 6 electrodes on each subject. Similar results were observed with electrode selection obtained with SSindividual, as depicted in Figure 8.
For a constant number of electrodes involved in the decision, we have compared the AUC between the configurations where the number of electrodes in a single subject is greater than the number of subjects and vice versa. We define a configuration of subjects and electrodes by C(i, j), where i is the number of subjects, and j is the number of electrodes. For instance, we compared the performance of having one subject with eight electrodes against having eight subjects and one electrode, and having four subjects each with two electrodes. Hence, we obtained 56 configurations (C(1, 2)vs.C(2,1), C(1, 8)vs.C(2,4), C(1, 8) vs. C(4, 2), C(1, 8) vs. C(8,1),…). The analysis of all the possible combinations resulted into 44 pairwise comparisons with the same global number of electrodes.
With SScommon, a pairwise t-test comparison revealed that the configurations with more electrodes in a single subject, than subjects involved in the decision, provided better performance (t43 = 3.52, p = 0.001). These results suggest that it is better to consider smaller number of subjects with more electrodes than otherwise. The same pattern of performance was observed with SSindividual, (t43 = 4.10, p < 1.76e − 4). However, there exist 12 and 17 configurations (for SScommon and SSindividual, respectively), where the performance is higher while considering a higher number of subjects compared with the number of electrodes per subject.
Figure 7. AUC based on the number of electrodes and subjects with the SScommon method for electrode selection and Osum for fusing the classifier outputs across subjects. (a) 3D representation of the AUC; (b) 2D representation of the AUC; (c) AUC > 0.90; (d) AUC > 0.95. (In Figure 7b, a dark gray level represents a high AUC.)
Figure 7. AUC based on the number of electrodes and subjects with the SScommon method for electrode selection and Osum for fusing the classifier outputs across subjects. (a) 3D representation of the AUC; (b) 2D representation of the AUC; (c) AUC > 0.90; (d) AUC > 0.95. (In Figure 7b, a dark gray level represents a high AUC.)
Brainsci 04 00335 g007
Figure 8. AUC based on the number of electrodes and subjects with the SSindividual method for electrode selection and Osum for fusing the classifier outputs across subjects. (a) 3D representation of the AUC; (b) 2D representation of the AUC. (In Figure 8b, a dark gray level represents a high AUC.)
Figure 8. AUC based on the number of electrodes and subjects with the SSindividual method for electrode selection and Osum for fusing the classifier outputs across subjects. (a) 3D representation of the AUC; (b) 2D representation of the AUC. (In Figure 8b, a dark gray level represents a high AUC.)
Brainsci 04 00335 g008

6. Discussion

Cooperative BCIs involve the creation of new systems and new ways of thinking in terms of BCI systems. We have shown that it is possible to significantly increase the ITR with simple combination methods by considering the detection of event-related potentials occurring simultaneously across subjects. However, a key constraint is the number of subjects to involve in the decision. The trade-off between the time needed to perform a task and the number of subjects involved in the decision should be chosen in relation to the application. A key result is the difference in performance that can occur between a classical BCI and a cooperative BCI for the detection of two brain responses.
Furthermore, it is difficult to compare a single-user BCI and a cooperative BCI involving several users because the goal of both systems can be different, but it is still possible to evaluate the accuracy and the speed to produce commands. In particular, when it comes to applications, it is not relevant to compare the mean performance across subjects for a single BCI user with trials combined over time against the performance of a cooperative BCI with trials combined over subjects, because these applications have different purposes. Yet, there are exceptions such as when a group of subjects should write the same sentence with a BCI speller. Like other BCI experiments, some limitations of the study include the effects that are related to the motivation and the interest of the subjects in the experiment.
The results suggest that differences between two types of brain responses that could not be efficiently used for single-user BCI application may be advantageously useful in cooperative BCIs. Particularly, ERP components that require a large number of subjects to be detected could be used in a cooperative BCI system. In fact, ERP components such as the P300 and N200 are used in BCI because their amplitudes allow them to be relatively easy to detect at the single-trial detection level. Yet, particular ERP features (amplitude and latency) are used in psychology to model differences across experimental conditions [38]. These features could be considered in a cooperative BCI if a high number of trials can be obtained per event. The following sections indicate relevant issues and challenges that would leverage the use of BCIs for healthy people in various situations.

6.1. Social Psychology

As several subjects shall be present in a cooperative BCI, it would be important to analyze people’s feelings, thoughts, and behaviors during a cooperative BCI experiment, which could be performed with people in the same place or online. In [39,40], the neural activity of different brains during daily life interactions is measured simultaneously by using EEG in order to understand the neural processes generating and generated by social cooperation or competition. Their results suggest that EEG hyperscannings could be used to investigate experimental paradigms where the knowledge of the simultaneous interactions between the subjects has a value. In a cooperative BCI, it would be possible to create paradigms replicating social situations, and the BCI feedback could provide key insights on human behavior. Cooperative BCI could be a tool to bridge the gap between cognitive neuroscience and social psychology. Furthermore, as motivation and reward have an impact on ERP characteristics, the cooperative aspect can be combined with a competitive approach where a feedback could indicate who is the main contributor of the command. Moreover, it may be easier to accept an error in a cooperative BCI system because the cause of the error can be attributed to other subjects.

6.2. Entertainment and Education

As described in the first section, a large number of subjects would increase the performance of a cooperative BCI. Typical environments where people are involved as a group in a simultaneous shared task can be classrooms and theaters. In a classroom, a cooperative BCI setting could provide a precise feedback to the teacher about the current mental state of the whole classroom (fatigue, difficulty of the class, …). In a theater, a cooperative BCI could be used to interact in real-time with the performer by providing feedback via the analysis of α-activity or motor imagery responses.
While BCIs have been appealing for games [41,42], current BCI games stay at the prototype stage by considering well-established paradigms [43,44]: they do not offer the level of performance that is expected in terms of accuracy, and they propose no added values compared with other devices for healthy people. For non-video games, the concept of cooperative BCI has already been used in the Mattel Mindflex toy. The cooperative BCI approach is used as a part of the game, where the goal is to race through an obstacle course as fast as possible. To leverage the use of BCI in video games, cooperative BCI systems are particularly promising due to the possible performance but also the inner principle of cooperative gaming. Cooperative games have a long history in the video game industry, e.g., two players co-op mode with Double Dragon (1987, Taito corporation, Shibuya, Tokyo, Japan), and it has continued to gain popularity over the years, e.g., having up to four players with Diablo 3 (2012, Blizzard Entertainment, Irvine, CA, USA). The increase of games with a cooperative option is mainly due to the development of the internet and new controllers. Among the different features present in a cooperative game, the idea of resource sharing between subjects is brought into consideration.
In the framework of cooperative BCIs based on ERP detection, a resource shared between subjects can be the attention capability [45,46]. The strength of an ERP, and therefore its detection, depends on attentional capacity. Subjects should therefore master their attention to enable ERP detection. In video games, quick time event (QTE) is a method of context-sensitive gameplay where a player performs an action after the appearance of a visual or auditory stimulus. The acronym QTE is attributed to Yu Suzuki, director of the game Shenmue, Sega, 1999. QTEs are often used during cutscene sequences to keep the attention of the player and to propose specific contextual gameplay (e.g., Resident Evil 4, Capcom, 2005). QTEs are typically the kind of event that would evoke an ERP and would enable the knowledge of the stimulus onset for ERP detection (the gameplay is not self-paced). Therefore, QTEs are the best type of event that could be successfully used in a cooperative BCI based on ERP detection. In addition, an ERP such as the P300 can be detected in a low frequency band (under 10 Hz), avoiding jitter issues in online games due to the lag.
Cooperative BCIs are relevant in multiplayer online video games where players work as a team to achieve a particular goal. In such cases a group of players cooperate as a collective to complete a quest (a raid). For instance, raid groups can have from 5 up to 40 people doing the same global task in World of Warcraft (2004, Blizzard Entertainment). In this context, the activation of events in the game could be achieved only if a particular mental state can be found across all subjects. In this situation, it would be easier to have access to a large pool of users aiming at achieving the same task. A recent trend in video games involves motion gaming, e.g., Wii fit (Nintendo), Kinect (Microsoft), Playstation Move (Sony). Though non-invasive detection of particular human motion is unreliable in comparison with camera or sensors placed on the body, the combination of brain responses corresponding to particular movement could be efficiently used. For instance, in a video game based on dance, several players would perform synchronously a series of movements from which one can detect the combination of signals coming from each subject. The activation of the command would depend not only on the performance of the subject but also on the performance of the whole group.

6.3. Materials and Technical Issues

The cost of a BCI for a single person can still be expensive and could be an obstacle for the development of cooperative BCIs. A cooperative BCI system should ideally require at least five people to provide reliable results in the case of ERP-based BCI. For instance, we have shown that it is possible to achieve a mean AUC of 0.939 with four electrodes common to five subjects. Therefore, each subject should wear an EEG cap and have a personal amplifier (if the subjects are at the same location, and if the amplifier can cater for different ground and reference configurations, it would be possible to share the same amplifier). Further investigations should be carried out at the economic level to determine if it is more likely to find customers willing to buy an expensive BCI product (with many electrodes) or customers who wish to pay for a cheaper product (with less electrodes) that would require several people to perform efficiently. In addition to hardware issues for setting a cooperative BCI, BCI frameworks should embed features for transferring signals and decisions online. These features are already available in current BCI frameworks, e.g., BCI2000, OpenViBE, MATLAB+Simulink [47,48], facilitating the development of cooperative BCIs. With cooperative BCIs for healthy people, it would be more efficient to have several users with relatively cheap systems with few electrodes than to have a single user with a very expensive system, e.g., with 32 channels. By considering the EEG signals from different users, a remaining issue is the synchronization of the different EEG signals to the different visual stimuli to avoid jitter effects.

7. Conclusions

A classification of BCI according to the number of BCI paradigms and the number of subjects involved has been proposed: the classical BCI involving one subject with one paradigm is extended to several subjects with one paradigm (cooperative), to one subject with several paradigms (hybrid), or several subjects and several paradigms (coop-hybrid). Since the early days of ERP analysis, trials have been averaged both over time and across subjects to increase the signal-to-noise ratio and to determine key findings in cognitive neuroscience. We have shown that combining trials across subjects could be of great use in particular BCI applications, as it has been achieved over time in ERP-based BCI; an analogy is from sequential process to parallel process. The best combination method for combining decisions across subjects is the mean, and while it is better to consider a larger number of subjects than a lot of electrodes for single-event detection, several applications could benefit from a configuration with multiple subjects, and a moderate number of electrodes. Cooperative BCIs represent a fundamental new approach compared with current BCI, because the user does not work independently but interacts with other people having the same goal. Moreover, the performance does not depend on a single user but on the group of users, hence to some extent limiting the drawback related to BCI illiteracy: each subject has a contribution to the overall performance even if this contribution is slight.

Conflicts of Interest

The authors declare no conflicts of interest.

References

  1. Farwell, L.; Donchin, E. Talking off the top of your head: Toward a mental prosthesis utilizing event-related brain potentials. Electroencephalogr. Clin. Neurophysiol. 1988, 70, 510–523. [Google Scholar]
  2. Cecotti, H. Spelling with Non-Invasive Brain-Computer Interfaces—Current and future trends. J. Physiol. Paris. 2011, 105, 106–114. [Google Scholar]
  3. Millán, J.R.; Rupp, R.; Müller-Putz, G.R.; Murray-Smith, R.; Giugliemma, C.; Tangermann, M.; Vidaurre, C.; Cincotti, F.; Kübler, A.; Leeb, R. Combining brain-computer interfaces and assistive technologies: State-of-the-art and challenges. Front. Neurosci. 2010, 4, 1–15. [Google Scholar]
  4. Townsend, G.; LaPallo, B.K.; Boulay, C.B.; Krusienski, D.J.; Frye, G.E.; Hauser, C.K.; Schwartz, N.E.; Vaughan, T.M.; Wolpaw, J.R.; Sellers, E.W. A novel P300-based brain-computer interface stimulus presentation paradigm: Moving beyond rows and columns. Clin. Neurophysiol. 2010, 121, 1109–1120. [Google Scholar]
  5. Yuan, P.; Wang, Y.; Wu, W.; Xu, H.; Gao, X.; Gao, S. Study on an online collaborative BCI to accelerate response to visual targets. Proceedings of the 34th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), San Diego, CA, USA, 28 August–1 September 2012; pp. 1736–1739.
  6. Yuan, P.; Wang, Y.; Gao, X.; Jung, T.P.; Gao, S. A collaborative brain-computer interface for accelerating human decision making. Proceedings of the 7th International Conference on Universal Access in Human-Computer Interaction: Design Methods, Tools, and Interaction Techniques for eInclusion, UAHCI 2013, Las Vegas, NV, USA, 21–26 July 2013; pp. 672–681.
  7. Poli, R.; Cinel, C.; Matran-Fernandez, A.; Sepulveda, F.; Stoica, A. Some Steps towards Realtime Control of a Space-craft Simulator via a Brain-computer Interface; Technical Report CES-525; University of Essex: Colchester, UK, 2012; pp. 1–23. [Google Scholar]
  8. Babiloni, F.; Cincotti, F.; Mattia, D.; Mattiocco, M.; Fallani, D.V.F.; Tocci, A.; Bianchi, L.; Marciani, M.G.; Astolfi, L. Hypermethods for EEG hyperscanning. Proceedings of the 28th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), New York, NY, USA, 30 August–3 September 2006; Volume 1, pp. 3666–3669.
  9. Eckstein, M.; Das, K.; Pham, B.T.; Peterson, M.; Abbey, C.; Sy, J.; Giesbrecht, B. Neural decoding of collective wisdom with multi-brain computing. NeuroImage 2012, 59, 94–108. [Google Scholar]
  10. Poli, R.; Cinel, C.; Sepulveda, F.; Stoica, A. A Preliminary Study of a Collaborative Brain-computer Interface in a Visual Matching Task; Technical Report CES-524; University of Essex: Colchester, UK, 2012; pp. 1–22. [Google Scholar]
  11. Poli, R.; Cinel, C.; Sepulveda, F.; Stoica, A. Improving decision-making based on visual perception via a collaborative brain-computer interface. Proceedings of the IEEE International Multi-Disciplinary Conference on Cognitive Methods in Situation Awareness and Decision Support CogSIMA 2013, San Diego, CA, USA, 25–28 February 2013; pp. 1–8.
  12. Allison, B.Z.; Lüth, T.; Valbuena, D.; Teymourian, A.; Volosyak, I.; Gräser, A. BCI Demographics: How Many (and What Kinds of) People Can Use an SSVEP BCI? IEEE Trans. Neural Syst. Rehabil. Eng. 2010, 18, 107–116. [Google Scholar]
  13. Guger, C.; Daban, S.; Sellers, E.; Holznera, C.; Krausza, G.; Carabalonac, R.; Gramaticac, F.; Edlinger, G. How many people are able to control a P300-based brain.computer interface (BCI)? Neurosci. Lett. 2009, 462, 94–98. [Google Scholar]
  14. Samek, W.; Vidaurre, C.; Müller, K.R.; Kawanabe, M. Stationary common spatial patterns for brain-computer interfacing. J. Neural Eng. 2012, 9, 026013. [Google Scholar] [CrossRef]
  15. Lu, S.; Guan, C.; Zhang, H. Unsupervised Brain Computer Interface based on Inter-Subject Information and Online Adaptation. IEEE Trans. Neural Syst. Rehabil. Eng. 2009, 17, 1–11. [Google Scholar]
  16. Cecotti, H.; Rivet, B.; Congedo, M.; Jutten, C.; Bertrand, O.; Maby, E.; Mattout, J. A robust sensor selection method for P300 Brain-Computer Interfaces. J. Neural Eng. 2011, 8, 016001. [Google Scholar] [CrossRef]
  17. Rivet, B.; Cecotti, H.; Maby, E.; Mattout, J. Impact of Spatial Filters During Sensor Selection in a Visual P300 Brain-Computer Interface. Brain Topogr. 2012, 12, 55–63. [Google Scholar]
  18. Thulasidas, M.; Guan, C. Optimization of BCI Speller Based on P300 Potential. Proceedings of the 27th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), Shanghai, China, 1–4 September 2005; Volume 5, pp. 5396–5399.
  19. Ryan, D.B.; Frye, G.E.; Townsend, G.; Berry, D.R.; Mesa-G, S.; Gates, N.A.; Sellers, E.W. Predictive Spelling With a P300-Based Brain-Computer Interface: Increasing the Rate of Communication. Int. J. Hum. Comput. Interact. 2011, 27, 69–84. [Google Scholar]
  20. Allison, B.Z.; Leeb, R.; Brunner, C.; Müller-Putz, G.R.; Bauernfeind, G.; Kelly, J.W.; Neuper, C. Toward smarter BCIs: Extending BCIs through hybridization and intelligent control. J. Neural Eng. 2012, 9, 1–7. [Google Scholar]
  21. Pfurtscheller, G.; Allison, B.Z.; Bauernfeind, G.; Brunner, C.; Escalante, T.S.; Scherer, R.; Zander, T.O.; Mueller-Putz, G.; Neuper, C.; Birbaumer, N. The hybrid BCI. Front. Neurosci. 2010, 4, 1–11. [Google Scholar]
  22. Sullivan, T.J.; Deiss, S.R.; Jung, T.P.; Cauwenberghs, G. A Brain-Machine Interface using Dry-Contact, Low-Noise EEG Sensors. Proceedings of the IEEE International Symposium on Circuits and Systems (ISCAS), Seattle, WA, USA, 18–21 May 2008; pp. 1986–1989.
  23. Gargiulo, G.; Calvo, R.A.; Bifulco, P.; Cesarelli, M.; Jin, C.; Mohamed, A.; Schaik, A. A new EEG recording system for passive dry electrodes. Clin. Neurophysiol 2010, 121, 686–693. [Google Scholar]
  24. Wang, Y.T.; Wang, Y.; Jung, T.P. Measuring steady-state visual evoked potentials from non-hair-bearing areas. Proceedings of the 34th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), San Diego, CA, USA, 28 August–1 September 2012; pp. 1806–1809.
  25. Estepp, J.R.; Christensen, J.C.; Monnin, J.W.; Davis, I.M.; Wilson, G.F. Validation of a Dry Electrode System for EEG. Proceedings of the Human Factors and Ergonomics Society, 53rd Annual Meeting, San Antonio, TX, USA, 19–23 October 2009; pp. 1171–1175.
  26. Lin, C.T.; Ko, L.W.; Chang, C.J.; Wang, Y.T.; Chung, C.H.; Yang, F.S.; Duann, J.R.; Jung, T.P.; Chiou, J.C. Wearable and Wireless Brain-Computer Interface and Its Applications. Proceedings of the 5th International Conference on Foundations of Augmented Cognition, San Diego, CA, USA, 19–24 July 2009; pp. 741–748.
  27. Wang, Y.; Jung, T.P. A Collaborative Brain-Computer Interface for Improving Human Performance. PLoS One 2011, 6, e20422. [Google Scholar]
  28. Wang, Y.; Wang, Y.T.; Jung, T.P.; Gao, X.; Gao, S. A collaborative brain computer interface. Proceedings of the 4th International Conference on BioMedical Engineering and Informatics (BMEI), Shanghai, China, 15–17 October 2011; pp. 583–586.
  29. Kuncheva, L.I. Combining Pattern Classifiers: Methods and Algorithms; Wiley: Hoboken, NJ, USA, 2004. [Google Scholar]
  30. Fumera, G.; Roli, F. Performance Analysis and Comparison of Linear Combiners for Classifier Fusion. 6–9 August 2002; pp. 424–432.
  31. Kittler, J.; Alkoot, F.M. Sum Versus Vote Fusion in Multiple Classifier Systems. IEEE Trans. Pattern Anal. Mach. Intell 2003, 25, 110–115. [Google Scholar]
  32. Fawcett, T. An introduction to ROC analysis. Pattern Recognit. Lett. 2006, 27, 861–874. [Google Scholar]
  33. Shannon, C.E.; Weaver, W. The Mathematical Theory of Communication; University of Illinois Press: Urbana, IL, USA, 1964. [Google Scholar]
  34. Rivet, B.; Souloumiac, A.; Attina, V.; Gibert, G. xDAWN algorithm to enhance evoked potentials: Application to brain-computer interface. IEEE Trans. Biomed. Eng. 2009, 56, 2035–2043. [Google Scholar]
  35. Rivet, B.; Souloumiac, A. Optimal linear spatial filters for event-related potentials based on a spatio-temporal model: Asymptotical performance analysis. Signal Process 2013, 93, 387–398. [Google Scholar]
  36. Cecotti, H.; Kasper, R.; Elliott, J.C.; Eckstein, M.P.; Giesbrecht, B. Multimodal target detection using single trial evoked EEG responses in single and dual-tasks. 6311–6314.
  37. Hoffmann, U.; Vesin, J.; Diserens, K.; Ebrahimi, T. An efficient P300-based brain-computer interface for disabled subjects. J. Neurosci. Methods 2008, 167, 115–125. [Google Scholar]
  38. Luck, S.J. A Introduction to the Event-Related Potential Technique; The MIT Press: Cambridge, MA, USA, 2005. [Google Scholar]
  39. Astolfi, L.; Toppi, J.; Fallani, F.D.V.; Vecchiato, G.; Salinari, S.; Mattia, D.; Cincotti, F.; Babiloni, F. Neuroelectrical hyperscanning measures simultaneous brain activity in humans. Brain Topogr. 2010, 23, 243–256. [Google Scholar]
  40. Astolfi, L.; Toppi, J.; Fallani, F.D.V.; Vecchiato, G.; Cincotti, F.; Wilke, C.T.; Yuan, H.; Mattia, D.; He, B.; Babiloni, F. Imaging the Social Brain by Simultaneous Hyperscanning During Subject Interaction. IEEE Intell. Syst. 2011, 26, 38–45. [Google Scholar]
  41. Krepki, R.; Blankertz, B.; Curio, G.; Müller, K.R. The Berlin brain-computer interface (BBCI): towards a new communication channel for online control in gaming applications. J. Multimed. Tools Appl. 2007, 33, 73–90. [Google Scholar]
  42. Tangermann, M.; Krauledat, M.; Grzeska, K.; Sagebaum, M.; Blankertz, B.; Vidaurre, C.; Müller, K.R. Playing Pinball with non-invasive BCI. Proceedings of the Conference of Neural Information Processing Systems (NIPS’08), Vancouver, Canada, 8–10 December 2008; pp. 1641–1648.
  43. Finke, A.; Lenhards, A.; Ritter, H. The MindGame: A P300-based brain-computer interface game. Neural Netw. 2009, 22, 1329–1333. [Google Scholar]
  44. Mül, C.; Görkök, H.; Plass-Oude Bos, D.; Thurlings, M.E.; Scherffig, L.; Duvinage, M.; Elbakyan, A.A.; Kang, S.; Poel, M.; Heylen, D. Bacteria Hunt: A multimodal, multiparadigm BCI game. J. Multimodal User Interfaces 2010, 4, 11–25. [Google Scholar]
  45. Huang, L.; Pashler, H. Attention capacity and task difficulty in visual search. Cognition 2005, 94, 101–111. [Google Scholar]
  46. Pashler, H.; Johnston, J.C. Attentional limitations in dual-task performance. In Attention; Psychology Press: Hove, UK, 1998; pp. 155–189. [Google Scholar]
  47. Schalk, G.; McFarland, D.J.; Hinterberger, T.; Birbaumer, N.; Wolpaw, J. BCI2000: A general-purpose brain-computer interface (BCI) system. IEEE Trans. Biomed. Eng. 2004, 51, 1034–1043. [Google Scholar]
  48. Renard, Y.; Lotte, F.; Gibert, G.; Congedo, M.; Maby, E.; Delannoy, V.; Bertrand, O.; Lecuyer, A. OpenViBE: An Open-Source Software Platform to Design, Test, and Use Brain-Computer Interfaces in Real and Virtual Environments. Presence Teleoperators Virtual Environ. 2010, 19, 35–53. [Google Scholar]

Share and Cite

MDPI and ACS Style

Cecotti, H.; Rivet, B. Subject Combination and Electrode Selection in Cooperative Brain-Computer Interface Based on Event Related Potentials. Brain Sci. 2014, 4, 335-355. https://doi.org/10.3390/brainsci4020335

AMA Style

Cecotti H, Rivet B. Subject Combination and Electrode Selection in Cooperative Brain-Computer Interface Based on Event Related Potentials. Brain Sciences. 2014; 4(2):335-355. https://doi.org/10.3390/brainsci4020335

Chicago/Turabian Style

Cecotti, Hubert, and Bertrand Rivet. 2014. "Subject Combination and Electrode Selection in Cooperative Brain-Computer Interface Based on Event Related Potentials" Brain Sciences 4, no. 2: 335-355. https://doi.org/10.3390/brainsci4020335

Article Metrics

Back to TopTop