Brain–Computer-Interface-Based Smart-Home Interface by Leveraging Motor Imagery Signals

Abstract

In this work, we propose a brain–computer-interface (BCI)-based smart-home interface which leverages motor imagery (MI) signals to operate home devices in real-time. The idea behind MI-BCI is that different types of MI activities will activate various brain regions. Therefore, after recording the user’s electroencephalogram (EEG) data, two approaches, i.e., Regularized Common Spatial Pattern (RCSP) and Linear Discriminant Analysis (LDA), analyze these data to classify users’ imagined tasks. In such a way, the user can perform the intended action. In the proposed framework, EEG signals were recorded by using the EMOTIV helmet and OpenVibe, a free and open-source platform that has been utilized for EEG signal feature extraction and classification. After being classified, such signals are then converted into control commands, and the open communication protocol for building automation KNX (“Konnex”) is proposed for the tasks’ execution, i.e., the regulation of two switching devices. The experimental results from the training and testing stages provide evidence of the effectiveness of the users’ intentions classification, which has subsequently been used to operate the proposed home automation system, allowing users to operate two light bulbs.

1. Introduction

The human brain has been studied for decades due to its interest as a dynamic and complex structure, and brain–computer interface systems have evolved to explore new ways to harness their power to improve human lives. Particularly, the goal of brain–computer interface systems is to establish a direct communication pathway between the brain and an external device, bypassing the body’s more typical pathways of nerves and muscles [1]. A huge number of studies have explored the potential of BCI systems in various applications, including rehabilitation [2], navigation and robotic control [3], environmental control [4], and gaming and entertainment [5].

Depending on the selected experimental approach and expected neurophysiological activation pattern, many forms of task-related information can be retrieved from brain waves. Significant examples include evoked potentials (EPs), steady-state evoked potentials (SSEPs) [6], event-related potentials (ERP) [7], and sensorimotor rhythms as motor imagery [8].

Motor-imagery-based BCI, in which users imagine performing a specific movement without actual execution, has emerged as a promising approach for facilitating communication and control both for individuals with motor disabilities [9] and for general-purpose applications. In stroke rehabilitation applications, robotic arms controlled by MI have been used to direct patients’ arm motions in stroke recovery [10], while a virtual reality has been employed for upper limb rehabilitation [11]. A continuous game control via MI-BCI has been developed in [12], while an immersive virtual-reality-based embodiable feedback has been implemented in [13] to improve MI-BCI control. In [14], a motor-imagery-based, adaptive BCI speller has been designed.

Moreover, as the number of smart devices in our homes continues to grow, so does the need for efficient and convenient control systems. In this direction, several approaches have been presented in the literature in recent years. A prototype of SSVEP-based BCI for home appliances control is presented in [15]. Surface electromyography (sEMG) readings from the occipitalis region have been used to drive a home automation system [16]. The “Neurophone” [17] uses hidden Markov models trained to recognize mental instructions via the Gamma feature band. Powered by a P300 control interface, the “BackHome” system assembles a suite of services—including smart home control, cognitive stimulation, online browsing, remote telemonitoring, and home support tools—to promote autonomy at home for users and carers without specialized training [18]. Among the offered approaches, to the author’s knowledge, the potential of MI-BCIs to provide intuitive means of controlling home devices, particularly for individuals with physical disabilities, has not been entirely explored, and this paper intends to fill this space.

MI-BCI-based systems require that the user’s intended movements are interpreted. Such a task is accomplished by first classifying the EEG signals and then converting them into commands. In recent years, several classifiers have emerged as popular techniques for addressing this challenge. Among these techniques include a Bayesian approach [19], pattern matching [20], neural networks [21], support vector machines [22], whitening techniques based on Gram–Schmidt orthogonalization [23], and Linear Discriminant Analysis [24]. LDA is a supervised classification algorithm that has been widely and successfully applied to BCI problems for its simplicity and high accuracy in classifying into different categories.

Furthermore, if the feature space dimension is large, a spatial filter could be employed to reduce the number of features, thus keeping the classifier from overfitting. The most straightforward approach is to manually select features from an a priori data inspection. Automatizing, however, could be accomplished by the use of statistical methodologies. A well-known method for extracting brain activity that is used in MI-BCIs is represented by common spatial patterns (CSP) [25], which is a feature extraction technique that optimizes the separation of different signal classes. In order to improve the performance of the CSP algorithm in high-dimensional data settings, the addition of a regularized term to the CSP algorithm represents a valid solution to improve robustness against noisy or incomplete data [26]. The combination of these two well-known methods, i.e., LDA and regularized CSP, leads to a more accurate and efficient classification of EEG signals.

This is why, in this work, we propose a novel use of an MI-based BCI system that makes use of LDA and RCSP to drive a home automation system. The entire software architecture is based on Konnex, a standardized protocol for home automation that facilitates the communication between hardware devices to control various home appliances in real-time. In addition, the proposed framework enables the control of two different devices at once by providing real-time information on the devices’ state.

Our findings demonstrate that the proposed BCI system achieves good classification accuracy and good response times, indicating its potential for use in home automation systems.

The paper is organized as follows: Section 2 introduces the participants, the experiments, the data acquisition, the pre-processing, and the classification phases. The experimental sections comprising the software and hardware architectures as well as the experimental results are detailed in Section 3. Finally, Section refsec:conc presents conclusions and future works.

2. Materials and Methods

2.1. Participants

The study was conducted on four subjects. Participants were asked to remain focused and attentive throughout the MI training session to obtain accurate and reliable results. The participants’ composure and focus during the training session contribute to the positive outcomes of the experiment, as it helps to reduce anxiety and stress levels, which can affect the performance of motor skills.

2.2. Experiments

In the standard paradigm for the discrimination of two mental states, the experimental task is to imagine either a right-hand or a left-hand movement depending on a visually presented stimulus. During the resting phase, the participants were introduced to the BCI system and instructed on what to do during the training and tests session. Therefore, it was crucial to provide them with thorough clarifications and instructions.

Specifically, participants were asked to stare intently at a screen located around 150 $\left[\mathrm{cm}\right]$ in front of them (see Figure 1).

Figure 2 shows the sequence in time of each trial for both the training and the test phases, beginning at time $t_0=0\left[\mathrm{s}\right]$. The sequence begins with an initial resting phase ($t_r=40\left[\mathrm{s}\right]$) during which the participant relaxes and concentrates on the session to be performed. After the initial resting phase, as shown in Figure 2, the trials start with the fixation cross that appears on the screen (see Figure 3b). Afterward, it is overlaid with an arrow at the center of the monitor for $t_{arr}=$ 1.25 $\left[\mathrm{s}\right]$, pointing either to the left or to the right (as shown in Figure 3a,c). In this phase, depending on the direction of the arrow, the subject is instructed to imagine a left- or right-hand movement for $t_{MI}=$ 3.75 $\left[\mathrm{s}\right]$. Each trial lasts $t_{trial}$ = 8 $\left[\mathrm{s}\right]$ in total. One entire training run includes twenty trials per class (forty in total), while one entire test run includes ten trials. Overall, in order to evaluate the performance of the training and test phases, subjects were required to stay focused for $t_{train}=$ 7 $\left[\min \right]$ for the training phase and $t_{test}=$ 2 $\left[\min \right]$ for the testing phase.

2.3. Data Acquisition

In order to acquire EEG signals, an Emotiv EPOC X headset was used. Recordings were made using a total of 14 electrodes. The selected electrodes were AF3, F3, F7, FC5, T7, P7, and O1 (Left side); and AF4, F4, F8, FC6, T8, P8, and O2 (Right side). Figure 4 shows the EEG topography distribution for subject A in one MI task execution during the training phase. All electrodes are arranged according to the international 10–20 system seen in Figure 5.

The wireless device uses advanced sensor technology to capture brainwave signals with high accuracy and precision at a sampling rate of 128 $\left[\mathrm{Hz}\right]$.

2.4. Pre-Processing and Classification

With the final aim of developing a complete framework to command home appliances, we selected some of the most reliable approaches to pre-process and classify EEG data. In particular, filtering, the application of an RCSP filter, and a final classification performed by an LDA classifier are the three main stages that comprise the post-processing analysis and classification, summarized in the following text. A graphical block representation of these three main phases is shown in Figure 6.

In the filtering phase, each trial was band-pass filtered in the range of 8–30 [Hz], using a 5th-order Butterworth filter. The objective of this filtering procedure was to eliminate any noise or unwanted artifacts from the EEG data that could potentially interfere with the subsequent analysis. The 8–30 $\left[\mathrm{Hz}\right]$ frequency range used in this study was chosen to highlight the most pertinent frequencies to the analysis of sensorimotor rhythms, which are known to be associated with movement and motor imagery.

Subsequently, after the band-pass filtering was performed, a regularized common spatial filter (RCSP) [26] was applied to the data, aiming at improving the signal-to-noise ratio of EEG data with a particular focus on inter-subject variability and spatial resolution improvement. In general, supposing that there are two classes, the CSP’s goal is to train spatial filters ${\mathit{w}}_{\mathit{CSP}}$ that minimize the variance of filtered EEG signals for one class, while maximizing it for the other, according to

$${\mathit{J}}_{}\left({\mathit{w}}_{\mathit{CSP}}\right)=arg\; max\frac{{\mathit{w}}_{\mathit{CSP}}^{\top }{\mathit{X}}_1^{\top }{\mathit{X}}_1{\mathit{w}}_{\mathit{CSP}}}{{\mathit{w}}_{\mathit{CSP}}^{\top }{\mathit{X}}_2^{\top }{\mathit{X}}_2{\mathit{w}}_{\mathit{CSP}}}=arg\; max\frac{{\mathit{w}}_{\mathit{CSP}}^{\top }{\mathit{C}}_1{\mathit{w}}_{\mathit{CSP}}}{{\mathit{w}}_{\mathit{CSP}}^{\top }{\mathit{C}}_2{\mathit{w}}_{\mathit{CSP}}}$$

(1)

where the superscript ${}^{\top }$ denotes the transpose matrix, ${\mathit{X}}_i$ is the data matrix, and ${\mathit{C}}_i$ is the spatial covariance matrix for the class i. Regularization is obtained by adding a penalty function $P\left({\mathit{w}}_{\mathit{CSP}}\right)$ measuring how much the spatial filter ${\mathit{w}}_{\mathit{CSP}}$ satisfies a given prior according to

$${\mathit{J}}_P\left({\mathit{w}}_{\mathit{CSP}}\right)=arg\; max\frac{{\mathit{w}}_{\mathit{CSP}}^{\top }{\mathit{C}}_1{\mathit{w}}_{\mathit{CSP}}}{{\mathit{w}}_{\mathit{CSP}}^{\top }{\mathit{C}}_2{\mathit{w}}_{\mathit{CSP}}+\alpha P\left({\mathit{w}}_{\mathit{CSP}}\right)}$$

(2)

where $\alpha \in {\mathbb{R}}^{}\ge 0$ is an a priori defined regularization parameter. The more the spatial filters ${\mathit{w}}_{\mathit{CSP}}$ satisfy it, the lower $P\left({\mathit{w}}_{\mathit{CSP}}\right)$ is. Hence, to maximize ${\mathit{J}}_P\left({\mathit{w}}_{\mathit{CSP}}\right)$, $P\left({\mathit{w}}_{\mathit{CSP}}\right)$ should be minimized, thus ensuring that the spatial filters satisfy the prior. This approach has been selected since it collects the spatial filters with the highest discriminative power. The aim was to avoid possible CSP sensitivity to noise and overfitting. We refer the reader to Figure 7 for an illustration of the set of CSP scalp projections generated with the RCSP algorithm for one participant in the user study.

Spatial filtering was followed by a classification process using Linear Discriminant Analysis, a common approach for distinguishing between characteristics of different classes—more specifically, between two classes. The approach is fast and allows for improving the classification performance by finding a rotation that maximizes the (normalized) distance between the centers of the two sets of data. In particular, when two groups A and B are assumed to have independent Gaussian distributions, the methodology calculates a projection vector that minimizes the variances of the projected populations while maximizing the mean distance between them. Thus, defining the two classes as A and B, the mean difference and the pooled covariance matrices can be written, respectively, as

$$\mathbf{\Delta }\mathit{\mu }={\mathit{\mu }}_{\mathit{B}}\mathit{-}{\mathit{\mu }}_{\mathit{A}}$$

(3)

$${\mathbf{\Sigma }}_{}=\frac{1}{2}\left({\mathbf{\Sigma }}_B+{\mathbf{\Sigma }}_A\right).$$

(4)

The LDA projection vector can be defined as

$${\mathit{w}}_{LDA}=\mathbf{\Delta }\mathbf{\mu }{\Sigma }^{-1}$$

(5)

Therefore, given an arbitrary input $\mathit{x}$, the symmetric LDA score is given by

$$\mathit{p}={\mathit{w}}_{LDA}{\mathit{x}}^{\top }-d$$

(6)

where the offset term is

$$d=\frac{1}{2}{\mathit{w}}_{LDA}{\left({\mathit{\mu }}_{\mathit{B}}+{\mathit{\mu }}_{\mathit{A}}\right)}^{\top }$$

(7)

Using this method, the classification between two groups A and B can be performed.

3. Experimental Results

3.1. Experimental Setup

In this section, we present the experimental setup and results that show the effectiveness of the presented BCI control framework, which is schematically shown in Figure 8. The integration of several technologies, presented here, has allowed the challenging design of an MI-BCI-based system capable of driving home appliances devices to be performed.

First, EEG data were recorded through Emotiv EPOC X through the 14 sensors that subjects placed on their scalp. EMOTIVPRO allowed interactions with a fixed workstation. Then, OpenVIBE (http://openvibe.inria.fr/, accessed on 11 July 2023) was integrated to perform the signals’ post-processing, feature extraction, and classification for both the training and the testing phases. Lab Streaming Layer (LSL), an open-source software framework for real-time acquisition and synchronization of various types of data, facilitated the data exchange between EMOTIVPRO and OpenVIBE. Once the feature extraction and classification phases were complete, Node-RED, an immediate programming tool for interconnecting hardware devices, APIs, and online services over TCP/IP, used the gathered information for both commanding the different hardware devices according to the users’ intentions and received feedback about the status of the devices.

Ultimately, the KNX protocol enables the completion of the binding process that takes place between the Node-RED and the hardware interface. In particular, commands and devices’ statuses are sent and received via KNX by performing a gateway and port setting. In such a way, instructed actions such as switching a light bulb on or off or setting a dimmer can be performed. The advantage of the KNX standard is that it allows devices to be linked to the line in any topological installation configuration; in addition, it also allows the system to be controlled using a large range of interfaces, which are pivotal characteristics in our BCI implementation. To show the results, the implemented KNX architecture was composed of two switches that allow the control of two lighting systems that, from now on, will be referred to as LS-A and LS-B; these switches were connected through a twisted pair shared bus (see Figure 1). It should be mentioned that actuator relays and sensors, such as brightness or temperature sensors, could be integrated into the architecture.

3.2. Software Implementation

The designed software architecture makes use of Node-RED for the software association between the OpenVIBE outputs and the KNX real devices, fully using its advantage of being highly suitable for low-level programming of event-driven applications and for making stable connections between hardware components. A schematic representation is shown in Figure 9. The data transfer is performed over TCP via a custom Node-RED library which sends classified signals in string form. After an initial set of the port number within the Inject block, the TCP Client Node block can be set up in listening mode. This allows for listening of the data coming from OpenVIBE over TCP. Afterward, the users’ provided commands are decoded to drive the selected devices; the devices’ states are continuously fed back to the system and then sent to the real devices by using the KNXs blocks.

By using such a designed software architecture, two devices were handled by switching their states as needed by the imagined right and left motions. This indicates that the state of the light system LS-A changed when users considered a left-handed motion, and the status of the light system LS-B changed when users considered a right-handed motion. By using this strategy whose strength is in enabling users to handle not just one but two devices at once, MI can be used to manage more than one device at a time.

Results

This section is devoted to presenting the results of the efficacy of the proposed MI-BCI-based home automation system. In particular, trials included a training phase followed by a testing phase in which participants thought about left and right movements in order to simultaneously have control of the state of two devices, i.e., the KNX smart light LS-A and the KNX smart light LS-B. The interested reader is referred to the Supplementary Video to see the system in action (http://www.dees.unict.it/mbucolo/index.php/resources, accessed on 11 July 2023).

Four healthy subjects were involved in testing both the software and hardware architectures. The system’s effectiveness was first defined by determining how accurate the system was at recognizing the left and right commands in the recorded EEG signals. The results of this first phase are summarized in

Table 1. Training accuracy.

Subjects	Number of Trials	Training Phase Accuracy
1	40	73.32%
2	40	71.20%
3	40	70.01%
4	40	71.50%

, which displays the right and left training accuracies for each individual across 40 trials. During the training phase, each subject performed the task described in Section 2.2, completing 20 trials for each class, i.e., left or right. The whole training phase (40 trials) lasted 7 min. This amount of time was considered sufficient in terms of computational efficiency, adequacy of the data, and overfitting mitigation. As a matter of fact, 7 min allowed for obtaining a good balance between reliable results, efficient computational resource management, and the risk of the model being overly specialized. Moreover, from an experimental perspective, the longer the training phase, the greater the risk of subjects’ loss of attention, which could affect the experiments’ performance. By looking at the results, they show that all the target end users involved in the evaluation were able to achieve satisfactory accuracies > 70%. Moreover, a small difference is observable between the best-obtained performance (subject 1) and the worst one (subject 4), which provides evidence that the approach is reliable under various conditions.

A second analysis was performed to analyze how accurate the proposed MI-BCI-based home automation system was at executing the users’ intentions in test sessions. Each participant completed 10 tasks which, in turn, resulted in a command generation to drive the proposed home automation system by changing the light system states. The outcomes of such an analysis are shown in

Table 2. Test phase accuracy.

Subject	Number of Tasks	Testing Phase Accuracy
1	10	80%
2	10	70%
3	10	70%
4	10	70%

. The testing phase lasted 2 min and took place exactly as the training phase, except for the fact that feedback was provided in the form of a blue bar decoding the intensity of the response to the visualized stimulus. In this case, the results show that all the target end users involved in the evaluation were able to gain control over the MI-BCI system with an average score of ≈72%. Furthermore, the fact that the training phase results are consistent among participants, with an upper bound of 80% and a lower bound of 70%, is further proof of the validity of the proposed approach. Overall, the solution proposed in this work exhibits satisfactory performance and holds significant potential for further development and implementation.

4. Conclusions

In this work, we addressed the challenging problem of designing an MI-BCI-based smart home interface to drive smart-home appliances in real time. Our work shows that the problem can be solved using a software and hardware architecture that was designed, implemented, and thoroughly tested with different subjects to show the effectiveness of the proposed solution. In particular, the integration of accessible programming tools, such as Node-RED, and reliable communication protocols for hardware devices’ setup and communication, such as “KNX”, made the realization of a control framework to drive two switching devices with sufficient accuracy possible.

Further study should be carried out to cope with daily fluctuations of EEG signals, and more has to be done to ensure a more reliable performance. In particular, applying such a system to a wider number of users and focusing on long-term experiments can also be crucial next steps to be performed. In addition, in light of the versatility of the proposed communication protocol, tests on more devices could be performed to allow users to extend the set of possible devices that users can effectively control.

However, despite the effectiveness of the proposed approach and in light of the complexity of the domain, additional challenges are still required to be faced. Additional investigations should be carried out to reduce the training phase time. Conspicuous subjects’ efforts are required in the current implementation. In addition, from a control perspective, further improvements should be made in the direction of enhancing the system outputs. In fact, users should be given more control degrees of freedom to perform regulation of more than two devices at a time. MI multi-class classifications can significantly help in this direction. From a technological point of view, further experiments need to be performed to see the approach reliability out of research laboratories, in real-world conditions, and across a wide variety of subjects. These experiments could raise additional challenges not yet considered, such as the impact that real-world environmental disturbances can have on such systems.