Improvement of the Classification Accuracy of Steady-State Visual Evoked Potential-Based Brain-Computer Interfaces by Combining L1-MCCA with SVM

Round 1

Reviewer 1 Report

Summary:

In their study, the authors systematically investigated different feature extraction methods, classification methods and time window lengths for decoding steady-state visually evoked potential (SSVEP) signals. They found that the best performance was achieved using L1-MCCA in combination with particle swarm optimized support vector machine classification and the longest available time window. They argue that the advantage of their method are subject-specific information which are used for feature extraction in comparison to the commonly used CCA method and the PSA-SVM approach. The BCI community is continuously seeking for algorithms which improve the decoding accuracy, in order to overcome the low SNR in EEG signals and make BCIs more reliable. With their study, the authors introduce a promising approach to achieve high decoding accuracies.

 

While the results are convincing that their approach significantly improves accuracy, it is not quite clear what their new contribution to the filed is, given that Yu Zhang et al. already has published several papers about SSVEP decoding using MCCA.

In the introduction, the authors state that PSD determines frequency components corresponding to the peak of the spectrum. To me it is not clear what they mean with peak of the spectrum. In SSVEP analysis commonly only frequency components related to the stimulus frequencies are considered. The peak of the entire spectrum often lies in other frequencies.

In line 58, “subject-specific experimental and test information is missing” – what exactly is meant? I assume the authors want to argue that individual brain responses require individual decoding models, however this is hard to understand from this sentence.

In line 62, the authors write “multiway EEG signals recorded by multiple experiments”. What is a multiway EEG and why can they use data from a single experiment when multiple Experiments are required for MCCA? Also, the meaning of “channel array of the EEG tensor and the test channel array” must be explained.

The authors argue that MCCA is an unsupervised method, which is the reason that they use a supervised classifier to improve results. I don’t agree with this argumentation. MCCA is a feature extraction method, not a classification method. Commonly, CCA features are classified trial-wise with the argmax function in an unsupervised manner, but argmax is the classifier, not MCCA. I think the advantage of SVM (and other linear classifiers) is its ability to learn multivariate relationships. Furthermore, CCA could also be applied in a supervised manner, e.g. when spatial filters are previously obtained from a set of training data. I suggest to skip the whole supervised/unsupervised argumentation since it provides no additional value to the study and confuses the reader.

Methods Section:

The subjects and study design section is not well structured. Ethics statement could be placed at the end of the first para. The description of the paradigm doesn't follow a logical order (first describes the experiment, then refers to pre-experimental issues). The experiment is not sufficiently described. From the description and the figure 1 the following questions remain:
- What was the cue? How did subjects know where to focus?

- Did all checkerboards flicker simultaneously? If not, the setup is not comparable to a BCI setting which should be discussed.

- Were the cycles randomized across conditions or was one block per condition recorded (as suggested by the clause “subjects were allowed a 2-minute break after each condition”) Again, the latter case would constitute an unrealistic BCI setup.

- “subjects were required to focus on the target stimulus” is in contrast to the sentence “subjects were required to look at the center of the screen”

The authors report the distance from the screen and its resolution but this information is useless when the size of the screen is unknown.

The EEG was recorded at 5000 Hz and then downsampled to 1000 Hz where the highest stimulation frequency was around 15Hz. It is assumed that frequencies exceeding the gamma range cannot be measured with EEG, anyway. The authors should state why they used such a high sampling frequency which enhances the computational costs.

The authors state that electrodes were arranged according to the 10-20 system but Oz, PO3, PO4 and POz are not part of the 10-20 system.

The authors applied ICA to remove eye blink artifacts. They should explain how this can be implemented in a real-time (closed-loop) BCI setup.

Also, they claim that baseline correction was required due to the DC shift, but the bandpass filter they applied should already have removed the DC shift.

Furthermore, the authors first filtered the data and then segmented the data. In a real BCI setup this would not be possible but only short intervals are available before filtering, which results in edge effects and could affect decoding accuracy. The authors should add a paragraph about the limitations of their offline approach to the discussion section.

The authors excluded three subjects. What is meant by poor data quality? Excluding a subject because the algorithm produces bad results is not a valid approach, since it falsifies the distribution of the algorithm’s performance across a realistic population. Again, a closed-loop BCI would have to deal with those subjects and results must be reported.

The sentence in line 142 is wrong. rho is the maximum correlation coefficient between Xw and Yv, not X and Y and also referred to as canonical correlation coefficient (because it’s the correlation between the canonical variates x and y, as stated in line 132)

In equation 3, isn’t here argmax required instead of max? How can the maximum of several correlations result in a frequency?

In the Figure 2 caption, what is the “experimental signal”? (line 184). In the figure it should be better emphasized what are training data, what are test data? Z1 to Zn correspond to the stimulation frequencies? In (a), why is here only one CCA result fed into the SVM?

What remains unclear is whether MCCA is performed separately per frequency or whether all frequencies are provided simultaneously to obtain the optimized reference signals. If MCCA requires information about the stimulus frequency in the different trials to optimize reference functions, it is a supervised method as well, and not unsupervised, as claimed in the paper.

The methods section neglects the fact that CCA results in multiple canonical correlation coefficients, depending on the number of sensors and number of reference functions. The authors should explain, how they deal with the several components.

The authors claim that SVM have the advantage of “perfect theory”. What does this mean? Also, they argue that SVMs can better solve multi-classification problems, but SVM naturally is a binary classifier and not suited for multi-class discrimination. Instead, they are suited for multidimensional (high dimensional) feature spaces.

The PSO algorithm is explained but it is not defined, e.g. what particle means in the current data set. PSO optimizes the kernel parameter g but which kernel was used?

In Line 222, “ the average accuracy is averaged 10 times” – this must be explained in more detail.

Results section:

In Figure 4 is a typo: TWs from 1 to 4; should be 1 to 6

In line 247, how is the classification effect predicted?

In line 250 the authors mention that they performed leave-one-run-out cross-validation. What is a run in this case, a cycle? Is CV also applied for classification? The validation approach should be already explained in the methods section.

Figure 5 essentially shows the feature space. What was the TW? Does the plot show averages? If not, how is a single correlation coefficient calculated from the many trials and CV-folds, respectively? I wonder whether it is necessary at all to show this figure, which is meant to demonstrate a trivial relationship.

p-values are reported but which test was applied?

Table 1 and Figure 6 provide redundant information. One should be considered to be moved to a supplemental material section. Table one should also report standard deviation.

Discussion section:

In line 304, the authors claim that their study introduces L1-MMCA but this method already has been introduced in 2013 (which they cite in the introduction). The authors should emphasize which aspect is new in their study. In line 305 they state that they used SVM as a supervision method. This is an unusual term to describe a classifier method.

The sentence starting at line 311 mixes previous research with own results which might confuse the reader.

In line 318, what is meant by “providing a guarantee”?

The explanation that differences in features have an impact on classification performance is trivial. I don’t get why the authors make so much effort in discussing this issue.

In the conclusions, L1 regularization is attributed to increase learning ability. The regularization is performed during feature extraction. How does this influence to the learning ability of the classifier?

The last para in the conclusions (limitations) could be moved to the discussion section.

 

General comment:

citations should be placed behind the authors

 

Examples for bad English style:

“under” TW (several times in manuscript)

“And” as beginning of sentence (line 22)

“technology … communicates with external devices” ( line 29)

“researches” ( line 32)

“used to its advantages” (line 40)

“et al.” without name (line 39) This sentence also seems to be incomplete

“noises” ( line 50)

PSD has “defects” ( line 50)

“Use L1 regularization” ( line 66)

line 77: “The PSO …” actually extends the previous sentence

“recorded on participants” ( line 78)

“ages” ( line 83)

“white-and-black” (line 87)

“Because …, therefore…” (line 175)

“as the maximize” (line 176)

“evaluated comparison … ways before” ( line 307)

“are by the previous research” (line 310)

“in compared” (line 317)

“et al.” at end of sentence (line 339)

“researches” (line 358)

“L1-MCCA_PSO_SVM” (linew 375)

Author Response

Please see the attachment.

Author Response File: Author Response.docx

Reviewer 2 Report

The paper entitled “Improvement of the classification accuracy of steady-state visual evoked potential-based brain-computer interfaces by L1-MCCA combined with SVM” introduces an improvement of the L1-MCCA classification algorithm, by adding a PSO-optimized SVM layer to the classification procedure. The paper is well presented and addresses an important issue of the BCI community, namely improving the classification accuracy of the SSVEP detection. However, I still have some major concerns regarding the paper that I try to list below:

• The contribution of this paper is to have combined the L1-MCCA algorithm with an optimized SVM. However, a significant part of the paper including Introduction, Results and Discussion are dedicated to evaluating L1-MCCA itself. As much as I value the fact of reproducing and confirming results of published studies, I think the focus of the paper should be more on the optimization of L1-MCCA rather than on L1-MCCA.
• The results of the study seem to confirm the effect of optimization of on the classification accuracy, but important aspects of the statistical analysis are missing. In particular, throughout the results section, there are several mentions of p-values, but no mention of the performed statistical tests nor the corresponding statistics.
• In order to assess the benefit of the presented algorithm, it is important to compare it with state of the art methods for e.g: TrCA, CCA Clustering, and MsetCCA.
• I don’t see in the introduction any mention of citation [37] related to L1-MCCA. Although it seems to be the basis of this work.
• What motivated the use of PSO optimizations? As I understand, PSO is a metaheuristic which, in this case, helps find the best subset of features to train the SVM preventing the curse of dimensionality and overfitting. However, in the case of L1-MCCA, the number of features is relatively small: An MCCA coefficient with each template of stimulation frequency (9 in this case). Could the authors comment on the particular intuition behind the optimisation of SVM in this particular case?
• In Section 2, it is stated that a break of 2 minutes. was allowed between each condition. Does this mean that the conditions were run sequentially, and not in a random order between the frequencies? If so, then the order may influence the results of some conditions potentially due to fatigue effect.
• In section 2.1, the authors mentioned that the participants were rewarded, but do not mention the details of this reward, which may be required for reproducibility.
• Authors should provide the number reference of the ethical committee’s approval.
• In section 2.3 (line 150), the authors state that “the classification accuracy of CCA may not be good enough”, is it possible to be more specific on “when” CCA is not good enough and what is the considered threshold?
• In section 2.3 (line 201) the PSO algorithm is presented in general. I would recommend the authors to specifically describe how it is applied in this case.
• Can the authors comment on the self-paced performance of their algorithm? As I understand, their classification results (which are pretty high) concern synchronous interaction, but do not include a “rest” or “no selection” class. Could the authors comment on how their algorithm could be adapted to work in more realistic “asynchronous” setups?
• The key of figure 6 is hard to read, I suggest the authors resize the picture.
• Could the authors comment on Figure 5, specifically on the reason why no peak is observed for 14.7Hz?

Author Response

Please see the attachment.

Author Response File: Author Response.docx

Round 2

Reviewer 1 Report

The authors revised their manuscript and responded to all author comments. Some of the responses do not entirely clarify the question or induce new questions. With the new paras new language style issues are added. It’s not my intention to provide another list of typos. I strongly recommend a proof reading by a native speaker or a professional service.

Here are some minor comments on the current version:

In line 12, “potential” is missing in “steady-state visual evoked based BCI”

In line 50, it is stated that PSD has the advantage of a low SNR. How can a decomposition method have an SNR?

Line 112: A hint for future experiments: An EOG electrode at the outer canthus is best suited to record horizontal eye movements. For blink detection, electrodes above and below the eye would have been much better.

Line 142: “The number of reference signals is equal to number of stimulation frequencies” My question was not about the number of reference signals but about the number of components that result from CCA. This number depends on the number of functions in the reference signal but also on the number of channels. I assume the SVM takes all available components but the argmax classification might consider only the first component.

Line 206: “maximum interval in the feature space” is misleading, since a distance is maximized, but “interval” commonly refers to a time period. “SVM has a relatively complete theoretical explanation” is still an odd explanation. I would skip that statement. I don’t agree that SVM has “simple calculation”. SVM solves an optimization problem which I would not call simple calculation. What the authors might want to say is that it is relatively easy to apply because powerful toolboxes exist.

I still don’t get the mystery about the subject-specific information. The authors argue several times that L1-MCCA optimizes reference signals with subject-specific information. It would be very helpful when an example of such a reference signal would be shown in a figure. What is the difference in individual subjects (spatial distribution, amplitudes, additional oscillations, varying shape of oscillations?)? I agree that the sinusoidal functions used in CCA do not provide a good fit for the rectangular signal of the stimulation. The sine/cosine pairs are used in CCA because the rules of trigonometric identity enables the CCA to find arbitrary phase of a sinusoidal signal by determining an appropriate linear combination if the sine and cosine function. How is the phase shift represented in the optimized reference functions?

The data in Figure 4 are also shown in Figure 6. I wonder whether it is necessary to show the data in two different figures.

In Figure 5, we see one correlation coefficient per frequency and condition. These are data of one subject but subjects performed several trials per condition. So, do we see one single trial per condition or an average across trials? If the latter is true, error bars are missing.

In Line 87, the authors write that subjects were rewarded. I think they mean that subjects received payment. In neuroscientific studies so-called reward experiments investigate the influence of a real reward during the experiment, i.e. the height of the subjects’ payment depends on their performance in the experiment. If this was not the case in this study, the payment shouldn’t be named reward because this is misleading.

L1-MCCA is performed in a Leave-one-out cross-validation framework. What about the other SVM based classifiers? They also require a training and testing set. Is the optimization result based on the left-out data? If yes this is not a valid method since in new data information about the condition would not be available. Here a nested cross-validation would be required. I think the validation approach of all classification approaches must be better explained.

line 387: What is meant by “pretreatment stage of the experiment”? Edge effects could be a problem in a BCI scenario where short intervals must be filtered. The authors’ explanation sounds like the other way around.

In their response the authors write that the stimuli were presented randomly but this answers not my question whether the stimulation was presented simultaneously. In the discussion they write “An additional limitation is that the experimental paradigm was not the same as the real BCI applications”. They should specify what exactly was not the same. Was it that only one stimulus flickered? If yes, this must be clearly written in the methods/discussion.

Author Response

Please see the attachment.

Author Response File: Author Response.docx

Reviewer 2 Report

Most of my comments have been addressed.

I still think that further comparisons with other approaches should be conducted to assess the benefit of PSO-optimized SVM compared to other approaches. In fact, using CCA (and its extensions) as a feature extraction method in conjunction with a classifier has already been proven beneficial. Hence, it would be interesting to compare SVM with these other methods. But I think the paper can still be published.

Author Response

Please see the attachment.

Author Response File: Author Response.docx