Conceptual Design of BCI for Mobile Robot Control

Abstract

This paper presents an application of Hierarchical Systems (HS) technology in conceptual and detailed design of Brain Computer Interface (BCI) system to control a mobile robot. The BCI is a biomechatronic system that includes biological (brain), computer (control PC), electronic (sensors), visual informatics (LCD—liquid crystal display, GUI—graphic user interface) and executive electro-mechanical (MR—mobile robot) subsystems. Therefore, the conceptual model of the designed BCI system should present connected formal models of the above subsystems presented in the general systemic basis. The structure of the BCI system, its dynamic realization as a unit in its environment and MR environment are presented formally as well. In addition, the conceptual model should also take into account the BCI inter-level relations performed by MR coordinator implemented in the form of the design and control system. Therefore, HS model (and its standard block aed—ancient Greek word) is selected and described as the formal basis of the conceptual model of BCI system in the first part of the given paper. BCI system detailed design is under consideration in the second part of the paper. BCI control system and MR design results, as well as MR control process are also described in the final part of the paper.

1. Introduction

At the end of the 20th century, leading research centers around the world, i.e., Graz University of Technology, Austria; University of Rochester, USA; EPFL, Switzerland; etc., initiated research aimed at developing Brain Computer Interfaces—BCI, which provide communication between a computer and a human brain. In the presented work, we deal with the conceptual design of the laboratory-level BCI system created in Process Control Dept., AGH University of Science and Technology, Poland [1,2]. This BCI system consists of measuring and computer control subsystems, mobile robot (MR) and man-operator. All the subsystems are briefly described in this paper. Formal basis for the Conceptual Design (CD) of the BCI system and its brief conceptual model are presented in the first part (Section 2.1 and Section 2.2) of the paper. The results of the BCI system detailed design and MR control processes are described in the second part (Section 2.2 and Section 2.3). A review of the literature on BCI systems will not be given in the paper since the focus of this work is to apply the Hierarchical Systems technology [2] to the BCI system design, which is a novel concept to this area of research.

Concerning the first part, the conceptual model of BCI being created at the conceptual design phase [2] must be coordinated with numerical and geometrical systems, taking into account connected mechanical, electro-mechanical, electronic, computer, and biological subsystems of BCI biomechatronic system. Besides, a conceptual model, which will be implemented in systems coordination (design and control), should describe all the subsystems, the mechanism of their interaction, technology of subsystems integration in the structure of higher level, and the system environment in common formal basis. Therefore, creation of a BCI conceptual model that will meet all the above mentioned requirements is a task of a high importance. This task is crucial for mechatronic systems and computer aided design (CAD) systems in particular [3].

According to the above formulated requirements, the conceptual model implemented in BCI design must be a hierarchical construction that connects any level BCI subsystem with its lower and higher levels. Therefore, Hierarchical System (HS) [4], with its standard block aed [5,6,7,8,9], which meets all of the above requirements for the BCI conceptual model, was selected as the formal basis for BCI system conceptual design (Section 2) in this paper. An overview of the other conceptual design methods is presented in the monograph [9] available online. An issue of interest in this study was to verify whether HS technology and the conceptual model could be implemented in the conceptual design of the BCI system taking into account all the subsystems (mechanical, electromechanical, electronic, and biological) of a BCI-MR biomechatronic system. The results of this study are given below.

Concerning the second part, the BCI system detailed design for performing MR control processes is executed in a laboratory environment. The purpose of this work is to control MR utilizing bioelectric activity of the human brain. This study was based on the use of visually evoked potentials, which are the brain’s response to retinal light stimulation. The possibility to use the BCI system by people with disabilities is one of the tasks of this work. The desired characteristics of the BCI system are real-time efficient operation, safety and ease of use. Providing real-time control without response delay requires proper optimization of the implemented algorithm and reduction of hardware requirements where possible. Ease of use can be achieved with high-frequency visually evoked potentials. In such conditions, a person will not be exposed to excessive noticeable stimuli of flickering light. On the other hand, the detection of the higher frequencies of electroencephalogram (EEG) signals is complicated [10]. Another important characteristic is safety. To minimize the possibility of the MR getting into forbidden places, one should be focused on the mobile robot motion. The definition of the MR workspace and the appropriate MR control algorithms should ensure the safety of the patient and his environment.

2. Materials and Methods

2.1. Theoretical Basis for Conceptual Design

Aed model $S^{\ell }$ considered below unites the codes of the two level system [5] and general systems theory [11], numeric positional codes $L^s$, geometry, and cybernetic methods. Aed is a standard element of Hierarchical Systems [5,6,7,8,9], which implements the general laws of systems organization on each level and the inter-level connections (see Figure 1). Aed $S^{\ell }$ contains ${\omega }^{\ell }$ and ${\sigma }^{\ell }$ models connected by coordinator $S_0^{\ell }$:

$$S^{\ell }\leftrightarrow {\{ \omega ,S_0,\sigma \}}^{\ell }$$

(1)

where ${\omega }^{\ell }$ is the dynamic representation of the level $\ell$ system in its environment ${}_{\epsilon }S{}^{\ell }$; ${\sigma }^{\ell }$ is the system structure; $S_0^{\ell }$ is the coordinator; ℓ ∈ L^s, L^s is numerical positional system. Aed scheme is given in Figure 1. Connected dynamic representations ${\omega }^{\ell }$ of aed elements, i.e., object ${}_{o}S{}^{\ell }$, processes ${}_{o\pi }S{}^{\ell },$ ${}_{\epsilon \pi }S{}^{\ell }$ and environment ${}_{\epsilon }S{}^{\ell }$ are presented in ${(\overline{\rho },\overline{\varphi })}^{\ell }$ form:

$$\begin{gathered} {\overline{\rho }}^{\ell }={\left\{{\rho }_t:C_t\times X_t\to Y_t\&t\in T\right\}}^{\ell } \\ {\overline{\varphi }}^{\ell }={\left\{{\varphi }_{t{t}^{\prime }}:C_t\times X_{t{t}^{\prime }}\to C_{t^{\prime }}\&t,t^{\prime }\in T\&t^{\prime }>t\right\}}^{\ell } \end{gathered}$$

(2)

where ${\overline{\rho }}^{\ell }$ and ${\overline{\varphi }}^{\ell }$ are reactions and state transition functions respectively, $C^{\ell }$ is state, $X^{\ell }$ is input, $Y^{\ell }$ is output, and $T^{\ell }$ is time of level $\ell$. Object ${}_{o}S{}^{\ell }$, processes ${}_{\pi }S{}^{\ell }\leftrightarrow$ {${}_{o\pi }S{}^{\ell },$ ${}_{\epsilon \pi }S{}^{\ell }$} and environment ${}_{\epsilon }S{}^{\ell }$ are connected by their states, inputs and outputs [5,6,7,8,9].

The structure of system $S^{\ell }$ is defined in the following way:

$${\sigma }^{\ell }\leftrightarrow \text{{}{S}_0^{\ell },{\tilde{\sigma }}^{\ell }\text{}},$$

(3)

and ${\tilde{\sigma }}^{\ell }$ is as follows:

$${\tilde{\sigma }}^{\ell }\leftrightarrow {\left\{\right\{{\{{\omega }_i:i\in I\}}^{\ell \mp \tau }:\tau \in L^s\left\}{,}_{\sigma }\gamma \right\}}^{\ell }\leftrightarrow {\{\left\{{\sigma }^{\ell \mp \tau }:\tau \in L^s\right\},{\omega }^{\ell \mp 0}\}}^{\ell },$$

(4)

where $S_0^{\ell }$ is a coordinator, ${\overline{\omega }}^{\ell \pm 1}$ are aggregated dynamic models of subsystems ${\overline{S}}^{\ell \pm \tau }$, and ${}_{\sigma }\gamma {}^{\ell }$ are structural connections. ${\tilde{\sigma }}^{\ell }$ is the set of ${\overline{\omega }}^{\ell \pm 1}$ subsystems and their structural connections ${}_{\sigma }\gamma {}^{\ell }$ coordinated with external ones ${}_{\omega }\gamma {}^{\ell }={}_{\sigma }\gamma {}^{\ell \pm \tau }|S^{\ell }$.

Being the main element of hierarchical systems, the coordinator performs the design and control tasks on its strata [5,6,7,8,9]. The coordinator is presented according to aed model (1) in the form (5):

$$S_0^{\ell }\leftrightarrow \{{\omega }_0^{\ell },S_{00}^{\ell },{\sigma }_0^{\ell }\}$$

(5)

where ${\omega }_0^{\ell }$ is the dynamic realization of $S_0^{\ell }$, ${\sigma }_0^{\ell }$ is the structure of coordinator $S_0^{\ell }$, and $S_{00}^{\ell }$ is the coordinator control unit. Coordinator $S_0^{\ell }$ is constructed recursively, and performs the design and control tasks on its selection, learning and self-organization strata [5,6,7,8,9].

Geometrical signs and metric characteristics μ^ℓ of systems S^ℓ being designed and controlled are defined in the framework of aed formal model in codes of numeric positional system $L^S$ [5,6,7,8,9]. The main of them are constructive dimension δ^ℓ and connection defect ξ^ℓ. For instance, the constructive dimension δ^ℓ is presented in code $L^S$ as follows:

$${\tilde{\delta }}^{\ell }={\left(n_3\dots n_0\right)}_{\delta },{\tilde{\delta }}^{\ell }\in \{{\delta }_{\sigma }^{\ell },{\delta }_{\omega }^{\ell }\},\text{\&}{(n_i)}_{\delta }={(n_{3-i})}_{\xi },{(n_i)}_{\delta }\in N, i=0,1,2,3;$$

(6)

where ${\delta }_{\omega }^{\ell }$ and ${\delta }_{\sigma }^{\ell }$ are constructive dimensions of ${\sigma }^{\ell }$ and ${\omega }^{\ell }$ respectively.

This representation of geometrical information about systems S^ℓ being designed allows execution of all computer operations with geometric images as operations with numeric codes.

Aed formal model briefly described above presents a theoretical basis for the BCI biomechatronic system conceptual design and control.

2.2. Conceptual Model for BCI System Design

The conceptual model of the BCI system is briefly presented below, as an example of a biomechatronic system description in the theoretical basis of hierarchical systems (HS). The object under consideration in this work is the BCI system (Figure 2), with its measuring, computer, man-operator, and mobile robot subsystems, developed in AGH University of Science and Technology, Poland [1,2]. Systemic conceptual model of the BCI system is given in aed form (6) of HS:

$${ }_{\mathrm{BCI}}{S}^{\ell }\leftrightarrow { }_{\mathrm{BCI}}{ \left\{\omega ,S_0,\sigma \right\}}^{\ell }$$

(7)

where, ${\omega }^{\ell }$ is an aggregated dynamic representation of BCI, ${\sigma }^{\ell }$ is the system structure, $S_0^{\ell }$ is BCI system coordinator (design & control system), and $\ell$ is the index of level.

Structural scheme of the BCI system being designed is presented in Figure 2 in agreement with aed diagram (Figure 1). According to (3,4), BCI system structure ${\sigma }^{\ell }$ contains coordinator $S_0^{\ell }$, the set of sub-systems ${\overline{\omega }}^{\ell -1}$ and their structural connections ${}_{\sigma }\gamma {}^{\ell }$. The BCI subsystems ${\overline{S}}^{\ell \pm \tau }$ presented in ${\overline{\omega }}^{\ell -1}$ form (2) are:

${\omega }_1^{\ell -1}$: man-operator; ${\omega }_2^{\ell -1}$: measuring system;

${\omega }_3^{\ell -1}$: computer control system; ${\omega }_4^{\ell -1}$: mobile robot.

BCI system coordinator $S_0^{\ell }$ is presented by man-coordinator and distributed design and control units. Computer control BCI subsystem ${\omega }_3^{\ell -1}$ partially performs coordinator functions as well.

In their turn, each subsystem _iS^ℓ−1 of level ℓ−1 has its own structural elements—lower level ℓ−2 subsystems presented in aggregated dynamic _ijω^ℓ−2 form. For the Mindstorms NXT 2.0 mobile robot ₄ω^ℓ−1 they are executive subsystem, i.e., wheels ₄₁ω^ℓ−2 and servos ₄₂ω^ℓ−2, and robot’s own NXT 2.0 control unit ₄₃ω^ℓ−2. Subsystems of the mobile robot are connected by their structural connections _σγ^ℓ−2. For example, servos ₄₂ω^ℓ−2 and wheels ₄₁ω^ℓ−2 are connected by gear transmission, which is the structural connection of them, and at the same time is subsystem of level l−3, i.e., structural element of servo ₄₂ω^ℓ−2. Similarly, the higher level subsystems ${\overline{\omega }}^{\ell -1}$ are connected by their common parts, i.e., structural connections _σγ^ℓ−1 that are the elements of lower levels. Man-operator ₁ω^ℓ−1 and measuring subsystem ₂ω^l−1 are connected by their common elements—electrodes attached to human’s head.

Aggregated dynamic realizations ω^ℓ−1, i.e., dynamic models _i(ρ,φ)^ℓ−1 of BCI subsystems are formed after the definition of their inputs–outputs according to each concrete sub-process they execute. As for the mobile robot ₄S^ℓ−1 subsystem, concerning its motion process, the realization ₄ω^ℓ−1 used at the conceptual design phase in (ρ,φ) form (2) can be presented in form (8) at the detailed design phase as the following kinematics equations [12]:

$$\begin{gathered} x\left(t\right)=\frac{1}{2}{{\displaystyle \int }}_0^t[v_r\left(t\right)+v_l\left(t\right)]cos\phi \left(t\right)dt,y\left(t\right)=\frac{1}{2}{{\displaystyle \int }}_0^t[v_r\left(t\right)+v_r\left(t\right)]sin\phi \left(t\right)dt, \\ \phi \left(t\right)=\frac{1}{a}{\displaystyle \underset{0}{\overset{t}{{\displaystyle \int }}}}[v_r\left(t\right)-v_l\left(t\right)]dt \end{gathered}$$

(8)

which connects linear $\dot{x},\dot{y}$ and angular $\dot{\phi }$ velocities of robot center point P(x,y) (located in the middle of the axis connecting driven robot wheels) as the output ₄Y^ℓ−1 with the input ₄X^ℓ−1, i.e., linear velocities of the centers of the left v_l and right v_r wheels of robot. a is the distance between the centers of the left and right wheels, and φ is the angle between Xg and Xr axes of the global g and the local robot r coordinate systems respectively (Figure 3). The robot designed has the independent drives of both wheels and Instantaneous Center of Curvature (ICC), around which the rotary motion is performed by robot. ICC is placed in the line connecting the centers of the two wheels. For the case of the robot motion shown in Figure 3, ICC is separated from point P(x,y) by the distance R. The rotation around the ICC point takes place if speed values of each wheel are different.

The model of robot kinematics was implemented within the frameworks of Matlab/Simulink program environment for performing a kinematic task at the detailed design phase. The general view of the model is presented in Figure 4. The inputs of RobotKinematicModel block are linear velocities of the centers of the left v_l and right v_r wheels of the robot respectively. The outputs are point P coordinates (x,y), ICC coordinates xICC and yICC, distance R, and angle φ value phi (Figure 4a). The detailed structure of the RobotKinematicModel block is shown in Figure 4b, which illustrates the graphical representation of the robot kinematics equations describing the robot’s location.

At the detailed design phase, simulation studies of the mobile robot movement were carried out. For this purpose, a special environment was created to animate the movement of the robot. The robot’s movement was set by the linear velocity value of the left v_l and the right v_r of the wheel. Figure 5a presents the mentioned linear velocities of both wheels. Vertical bold lines indicate two time moments (t₁ = 2.49 s and t₂ = 9.19 s). Figure 5b,c show pictures from the simulation showing the current position of the robot along with the trajectory of the point P (black) and the point Pp (blue) for t₁ and t₂ time moments respectively. The green lines represent linear velocity vectors with the beginnings in the center of each wheel. The lengths of the vectors represent the values of the current linear velocity shown in the upper right corners of the simulation Figure 5b,c. Additionally, the current values of the following parameters are shown: simulation time (t), current position of the P point, i.e., x,y coordinates, and angle φ value. The current position of the ICC and radius R (red line) are also presented.

As for the robot dynamics, aggregated dynamic model $(\overline{\rho },\overline{\phi })$ of the MR can be presented at the phase of the detailed design like in [13] or in the following form:

$$\dot{\mathrm{x}}=\mathit{A}\mathrm{x}+\mathit{B}\mathrm{u},\mathrm{y}=\mathit{C}\mathrm{x}$$

(9)

State equation in (9) corresponds to the state transition function $\overline{\phi }$ in MR conceptual model (2) and the second output equation corresponds to the reaction $\overline{\rho }$. Vectors x, y, u and matrices A, B, C must be predefined. In the case of the mobile robot motion, the elements of states vector x = [x₁ x₂]^T are the displacement x₁ and velocity x₂. Mobile robot kinematics and dynamics tasks, briefly described above, belong to analysis design task of the BCI subsystem being designed, i.e., mobile robot in our case.

Concerning the man-operator subsystem, its model ${\omega }_1^{\ell -1}$ can be constructed on its inputs $X_1^{\ell -1}$, i.e., visual stimuli signals, perceived by human eyes; its outputs $Y_1^{\ell -1}$, i.e., EEG signals recorded by measuring system ${\omega }_2^{\ell -1}$, taking into consideration human’s physiological state $C_1^{\ell -1}$.

Measuring subsystem ${\omega }_2^{\ell -1}$ purpose is EEG signals registration, processing and data transmitting to BCI control system by TrueScan 24 subsystem. TruScan Acquisition program is used as software means to perform impedance measurements. The electroencephalographic (EEG) signal was recorded using the EEG cap of Deymed Diagnostic company (Figure 6a), containing 19 chlorine silver electrodes (Ag/AgCl) arranged according to the International System 10–20. In addition, the cap contained a reference electrode and a grounding electrode (Figure 6b). The cap is placed on the head of the subject, i.e., man-operator, being examined (Figure 6c). EEG gel application to the measuring electrodes allowed to maintain electrodes impedance at 2–5 kΩ level during the conducted tests.

The BCI environment has its own structure and contains the following sub-systems: panel ${\omega }_1^{\ell +1}$ which generates light stimuli perceived by man-operator’s eyes; floor area ${\omega }_2^{\ell +1}$ where the mobile robot moves; man-coordinator ${\omega }_3^{\ell +1}$, which places electrodes on man-operator’s ${\omega }_1^{\ell -1}$ head and attaches the measuring system; the higher level l+1 design and control system ${\omega }_4^{\ell +1}$.

The BCI system contains ${}_{\pi }S{}_i^{\ell }$ sub-processes, which are the parts of the higher-level process ${}_{\pi }S{}_{}^{\ell }$(see Figure 1) performed by the entire BCI system: (${}_{\pi }S{}_1^{\ell }$) ionic current flows generation within the neurons of the human brain, performed by the man-operator, (${}_{\pi }S{}_2^{\ell }$) EEG signal acquisition by the measuring system; (${}_{\pi }S{}_3^{\ell }$) data analysis and control signals generation for the mobile robot performed by computer subsystem; (${}_{\pi }S{}_4^{\ell }$) mobile robot motion. Each process contains its lower-level sub-processes, e.g., (${}_{\pi }S{}_3^{\ell }$) process of computer subsystem contains low-level ℓ−1 process of data reading ${}_{\pi }S{}_{31}^{\ell -1}$, digital filtering ${}_{\pi }S{}_{32}^{\ell -1}$, Fast Fourier Transformation ${}_{\pi }S{}_{33}^{\ell -1}$, signal spectral analysis ${}_{\pi }S{}_{34}^{\ell -1}$, frequency verification ${}_{\pi }S{}_{35}^{\ell -1}$, and generation of control signals for the mobile robot ${}_{\pi }S{}_{36}^{\ell -1}$. All these processes are described according to (2).

So, all the subsystems of the BCI biomechatronic system, i.e., man-operator $S_1^{\ell }$, measuring $S_2^{\ell }$, computer $S_3^{\ell }$, and mobile robot $S_4^{\ell }$ systems have their aggregated dynamic ${\omega }^{\ell }$ and structural ${\sigma }^{\ell }$ descriptions. All the connected descriptions of the subsystems ${\overline{S}}_i^{\ell }$ and their processes ${}_{\pi }\overline{S}{}_i^{\ell }$ are presented in the knowledge base of the coordinator that performs the design and control tasks, connecting in this way structure ${\sigma }^{\ell }$ and dynamic realization ${\omega }^{\ell }$ of the BCI system being coordinated.

BCI coordinator $S_0^{\ell }$ is described according to (5) and implemented by the human–computer design and control BCI subsystems including the MR controller. The coordinator maintains BCI functional modes by the man-operator and control system and performs the design process by the higher level CAD system. The BCI control system is described in the following Section 2.3.

2.3. Design of BCI Control System

The design process of BCI control system of the mobile robot (MR) also contains two stages, i.e., conceptual and detailed design phases. The conceptual model of the BCI control system is presented in form (5) of HS coordinator ($S_0^{\ell }$) briefly described above in Section 2.1 as well as in another work [2]. Aggregated dynamic model ${\omega }_0^{\ell }$ of the BCI control system ($S_0^{\ell }$) is built on the relation of the control system inputs $X_{0t}^{\ell }$, outputs $Y_{0t}^{\ell }$ and states $C_{0t}^{\ell }$, presented in the following ${(\overline{\rho },\overline{\phi })}_0^{\ell }$ form:

$$\begin{gathered} {\left(\overline{\rho },\overline{\phi }\right)}_0^{\ell }:{\overline{\rho }}_0^{\ell }={\left\{{\rho }_{0t}:C_{0t}\times X_{0t}\to Y_{0t}\&t\in T\right\}}^{\ell } \\ {\overline{\phi }}_0^{\ell }={\left\{{}_o\phi {}_{0t{t}^{\prime }}:C_{0t}\times X_{0t{t}^{\prime }}\to C_{0t^{\prime }}\&t,t^{\prime }\in T\&t^{\prime }>t\right\}}^{\ell } \end{gathered}$$

(10)

The inputs X₀ and outputs Y₀ of the coordinator of ℓ − 1 level are defined on the sets of its coordination signals G and feedbacks W as follows:

$$X_0^{\ell -1}=\text{{}{G}^{\ell },W^{\ell -1}\text{}},\hspace{1em}{Y}_0^{\ell -1}=\text{{}{G}^{\ell -1},W^{\ell }\text{}},$$

(11)

As for the control system of Mindstorms mobile robot (Figure 7), it is presented by the internal control unit NXT 2.0 (which consists of ARM7 and AVR microcontrollers) and BCI control PC in the form of the coordinator according to the scheme presented in Figure 1 (Section 2.1).

In Figure 7, G^ℓ−1 are coordination signals from the internal NXT 2.0 control unit of the MR to MR _oS₄ actuating systems, i.e., servos and wheels. The MR position measured by MR encoder with 1 degree precision is a feedback $W_{}^{\ell -1}$ from MR actuating systems _oS₄ to NXT 2.0 control unit ₄S₀; W^ℓ is a feedback from MR, i.e., NXT 2.0 control unit, to the BCI PC control system S₀ of higher level ℓ. G^ℓ are the control signals from control PC S₀ to mobile robot ₄S^ℓ−1. The NXT 2.0 controller ₄S₀ is directly connected by its input X^ℓ = (G^ℓ,W^ℓ−1) and output Y^ℓ = (G^ℓ−1,W^ℓ) with the MR actuating systems and with the higher level ℓ coordinator S₀, i.e., BCI PC design and control system, where (G^ℓ−1, W^ℓ−1) and (G^ℓ, W^ℓ) are lower level ℓ−1 and higher level ℓ coordination G and feedback W signals respectively.

Communication of the BCI PC control system S₀ with lower ℓ−1 level NXT 2.0 control unit ₄S₀ of Mindstorms mobile robot is realized on the program level with the help of RWTH—Mindstorms NXT Toolbox integrated in MATLAB environment of the control PC. Control G^ℓ and information W^ℓ signals are transmitted with 12 Mbit/s speed. The transmission is possible owing to the implementation of Bluethoot 2.0 technology in NXT 2.0 control unit.

Concrete values of the BCI PC control system $S_0^{\ell }$ inputs $X_0^{\ell }$, outputs $Y_0^{\ell }$ and states $C_0^{\ell }$ are defined and detailed in the design phase and described below. A concrete model of BCI control system $S_0^{\ell }$ was implemented in the form of finite state machine [14] at the detailed design phase according to its conceptual model. Finite state machine model of BCI control system (CS) was implemented within the frameworks of Matlab program system using Simulink Stateflow module. To implement control algorithms by generating control codes, Matlab, Simulink, and Embedded Coders were used. The proposed control system developed in Matlab is presented in Figure 8. The main element of the BCI control system is MainController unit, which realizes logic laws of control of TruStan 24 electroencephalograph (EEG) unit, the light stimulator and the mobile robot. According to the BCI CS conceptual model, inputs $X_0^{\ell }$ and outputs $Y_0^{\ell }$ of MainController.ock unit are given in

Table 1. Selected input and output variables of MainController.

X0/Y0	Variable Name	Type	Description
INPUTS (X0)	Start	boolean	assumes a value of 1 for the running BCI system
	eegStatus	boolean	takes the value 1 when there is a connection error with the TruScan 24 device
	isDone	boolean	takes the value 1 when the robot has finished the task
	in	double	table with registered EEG signal
	imp	double	table with registered impedance values
OUTPUTS (Y0)	idle	boolean	takes the value 1 when state Idle is active
	error1	boolean	takes the value 1 when there is a connection error with the TruScan 24 device
	error2	boolean	takes the value 1 when there is an error related to reaching a certain signal length
	error3	boolean	takes the value 1 when there is a connection error with the TruScan 24 device
	led	boolean	takes the value 1 when the light stimulator in on
	dataRec	boolean	takes the value 1 when the state DataRecording is active
	dataProc	boolean	takes the value 1 when the state DataProcessing is active
	dataShow	boolean	takes the value 1 when the state DataShowing is active
	dataPre	boolean	takes the value 1 when the state DataPreprocessing is active
	fEProc	boolean	takes the value 1 when the state FeatureExtraction is active
	classProc	boolean	takes the value 1 when the state Classification is active
	class	integer	takes the value from the set {−1, 1, 2, 3, 4} being the result of Classification state
	robProc	boolean	takes the value 1 when the state RobotWorking is active

.

Table 2. BCI PC control system $S_0^{\ell }$ states $C_0^{\ell }$.

State 1C0	State 2C0	State 3C0
Idle	Running	DataProcessing

presents the BCI control system states $C_0^{\ell }$_.

The first state ₁C₀ of the proposed BCI control system is the Idle state (Table 2). It is an idle state in which the connection to the TruScan 24 electroencephalograph is checked. The process of establishing the connection with the electroencephalograph is simulated by the variable eegStatus, which takes the value 1 for a correctly established connection. Idle state activity is represented by the idle variable. After receiving the connection information, the system remains in Idle status until the BCI system is started by the Start button being pressed, which is reflected in the start variable.

When there is no error related to the connection to the TruScan 24 device and the Start button is on, the system goes into ₂C₀ Running state, in which the light stimulator is activated first. The activation of the light stimulator is represented by the variable led = 1. In this state there are three sub-states: Waiting ₂₁C₀, DataRecStart ₂₂C₀, and DataRecStop ₂₃C₀. After starting the light stimulator, the system goes to the Waiting sub-state. From this sub-state, the system goes to DataRecStart after 1 s. This sub-state is responsible for the process of switching on the EEG signal recording by the TruScan 24 device, which has been set for 5 s. The DataRecStart subroutine activity is indicated by the variable dataRec. In the next step, after the 5 s have passed, the system goes to the DataRecStop sub-state where the system disables the EEG signal registration process.

After leaving the DataRecStop sub-station, the quality of recorded data is checked. For this purpose, errorLen and errorImp functions are used. The errorLen function examines the length of the recorded signal. The result of the function is error error2 = 1, when the registered signal does not contain an adequate number of samples. The errorImp function examines the impedance value of measuring electrodes. The result of this function is error error3 = 1, when the impedance value exceeds 5 kΩ. If error2 or error3 occurs, immediate return to Idle status takes place. State ₂C₀ Running is abandoned when the connection to the electroencephalograph is broken error1 = 1 or when the value of start = 0 (start button disabling). The exit from the state turns off the light stimulator (led = 0).

When errors error2 and error3 do not occur, the system goes to DataProcessing state ₃C₀. The activity of this state is represented by the variable dataProc. This ₃C₀ state consists of four substates, listed in the order of calling: ₃₁C₀ DataShowing, ₃₂C₀ DataPreprocessing, ₃₃C₀ FeatureExtraction and ₃₄C₀ Classification. In the DataShowing sub-state, the recorded results are displayed using the show function. The dataShow value represents the DataShowing sub-state activity. Then the system goes to DataPreprocessing sub-state ₃₂C₀, in which the pre-processing of the registered signal is performed, signaled by the variable dataPre = 1.

Pre-processing of the registered signal consists in extracting a fragment of data of a specific length (dataPreProc function) and filtration using the predefined filters (filtration function). The obtained results are further analyzed in the FeatureExtraction state ₃₃C₀. The purpose of this state is to isolate characteristic features from the signal.

In order to test the designed BCI state machine, the detection of characteristic parameters was based on the FFT analysis [15]. This method is based on reading the frequency at which the largest amplitude value was obtained. Determination of the Fourier transform in the FeatureExtraction state ₃₃C₀ is performed by the fftProc function, and the reading of the frequency with the highest amplitude value is performed by the fMaxFind function. The FeatureExtraction state activity represents the fEPoc variable. The last sub-state ₃₄C₀ of DataProcessing ₃C₀ state is the Classification, in which a classification of characteristics and sending a control command to the mobile robot is made. Since the task of the developed BCI system is the mobile robot control, one of five characteristics (modes) may be selected as a result of classification:

-

MoveForward;

-

MoveRight;

-

MoveBack;

-

MoveLeft;

-

Unclassified.

The command controlling the robot movement is called after that in the state machine using the coder. extrinsic function.

3. Results

Verification of BCI Control System

To verify the correct operation of the MainController-based BCI control system, the necessary tests were carried out. Testing was performed by “injecting” recorded signals, i.e., (EEG) electro-encephalographic signals from one measuring electrode and its corresponding impedance value. In addition, a set of simulation signals was introduced into the system: activation of the start button (variable start), connection to the TruScan 24 device (variable eegStatus) and execution of the task by the mobile robot (isDone variable).

There are two control scenarios realized in the experimental study:

(1)

properly operating machine;

(2)

occurrence of impedance error.

The first scenario which presents one cycle of the correct functioning of the BCI control system is shown in the work. Data “injected” into the BCI system is presented in Figure 9.

The operation of the BCI control system is shown in Figure 10. The initial state (initializing) is the Idle state. It is a waiting state from which the output takes place when a connection is established with the TruScan 24 device (variable error1 = 0) and the start button is on (variable start = 1). Calling the errorConnection function in each simulation step determines the value of the error1 variable.

Because the connection to the TruScan 24 device has been established correctly, after the start button is triggered in a 0.5 s simulation (Figure 10a) the system goes into the Running state, activating its default Waiting condition. Switching to the Running state means leaving the Idle state, which reflects the change in the idle value and the light stimulator activation (Figure 10c).

After 1 s, the Waiting sub-state switches to the DataRecording sub-state, in which the control system waits for 5 s for the recorded data from the TruScan 24 device. This step illustrates the variable datRec shown in Figure 10d. The quality of the received data is checked by the errorLen and errorImp functions, and their results are assigned to the error variables error2 and error3 respectively. Then the logical value of these variables is checked. The quality of the “injected” data met the above criteria, and the system went into the DataProcessing state. This moment (in 6.5 s) reflects the change in the value of the led variable and dataRec of Running status (Figure 10d) and the variable dataProc of the DataProcessing state (Figure 10e).

Transitions between consecutive substates: DataShowing, DataPreprocessing, FeatureExtraction and Classification are shown in the DataProcessing graph in Figure 10e. After classifying and sending a control command (6.9 s), the system moves from the DataProcessing state to the RobotWorking state. This is shown in Figure 10f by the variable robProc. In 8.8 s of the simulation, the system receives information about the robot completing the task, which is shown by the isDone variable on the RobotWorking chart (Figure 10a,f).

After leaving the RobotWorking state, the Running status is activated again. It was activated for 0.5 s because the start button was turned off, which caused the system transition to the Idle state. The system remained in this state till the end of the simulation.

4. Discussion

One of the research issues addressed in this work was the design of a BCI system and its implementation in mobile robot control. For this aim, the conceptual model of the BCI system was developed at the conceptual design phase first, and concrete construction of the BCI system including BCI control system were implemented at the detailed design phase after that. The BCI control system was designed to fulfil the control modes of mobile robot being controlled. Another research problem addressed in the work was the use of biological signal processing algorithms that would allow the extraction of characteristic parameters from the electroencephalographic signal that could be used to control the brain–computer interfaces. The main attention was paid to the analysis and detection of the characteristic parameters of Steady-State Visual Evoked Potentials (SSVEP) [10], generated in the brain by stimulation of the retina with a light stimulus at a frequency above 6 Hz. The development of the system using the characteristic parameters of the electroencephalographic signal to control the selected output device was preceded by experiments aimed at selecting the best light stimulus dedicated to BCI systems.

When designing a new BCI control system, it was important to use the Model-Based Design (MBD) methodology of HS, i.e., design using systems models. This methodology is based on the implementation of each sub-system that is a part of the new system of higher level, using models built in the MATLAB/Simulink software. By preparing the object model, one can simulate the entire control system and check how it affects the objects before the construction of the prototype. The main advantage of the MBD methodology of HS is that the hierarchical connections between particular components of the designed system are strictly defined at the modeling stage of the conceptual and detailed design phases. After verifying the correctness of the system operation, using the automatic code generation tools available in the MATLAB (MATLAB Coder, Simulink Coder, Embedded Coder) package, one can implement the control algorithm on the target device. Control of the executive element in the designed BCI system was carried out based on the state machine, which was implemented in the MATLAB 2014a/Simulink environment. As part of the aforementioned Simulink environment, Stateflow module was used for modeling and simulation of event-controlled systems with finite state machines. The main advantage of the Stateflow module is the availability of linguistic elements; hierarchy; and deterministic executive semantics that allow to describe complex logic in a natural, understandable and legible way. In addition, Stateflow module has got the following advantages: (a) application of flow diagram notations and state changes on one diagram, traditional C language syntax with input and output arguments; (b) description of the operation of the finite machine proposed by David Harel [15], enabling the representation of the hierarchy; (c) parallel operation and history within the states; (d) scheduling state changes using time operators; and (e) equipping with an integrated debugger that allows viewing data step-by-step through the subsequent stages of the model. BCI system implementation requires a control system development that will meet the relevant requirements. The most important element of the designed BCI control system is the MainController unit, which contains the entire logic necessary to control BCI sub-systems, i.e., TruStan 24 electroencephalograph, a light stimulator and a mobile robot.

The development of a BCI system using the characteristic parameters of the electroencephalographic signal to control the mobile robot allowed formulation of the following conclusions. The offline tests of the proposed algorithms for SSVEP characteristic parameters detection confirmed that the accuracy of the parameters classification is influenced by the length of the analyzed signal, the number of multiples taken for analysis, and the number of measuring electrodes. The offline tests allowed the selection of control algorithms based on statistical methods (LASSO) to be more efficient in detecting characteristic parameters of SSVEP [10] in comparison with methods using spectral analysis (PSDA). A virtual environment was created that fully imitated the brain–computer interface, implementing feedback based on the control system model and the mobile robot’s kinematic model. This environment allowed free offline testing by “injecting” previously registered data, and then used it to conduct final online tests. This approach significantly reduced the time needed to develop the resulting system. The conducted online tests confirmed the SSVE potentials fitness to control a mobile robot. The presented conclusions also confirm the effectiveness of the electroencephalographic signals application as control signals in the brain–computer interfaces.

5. Conclusions

Application of novel Hierarchical Systems technology in conceptual and detailed design of a BCI (Brain Computer Interface) system implemented to control a mobile robot is presented in this paper. Proposed conceptual model of the designed biomechatronic BCI system (Section 2.2) contains connected systemic models of the BCI subsystems of various natures, i.e., biological (brain), computer (control PC), visual informatics (LCD, GUI), electronic (sensors) and executive electro-mechanical (mobile robot), presented in the general formal basis of HS. Connections of the conceptual systemic models of the BCI subsystems with their models used at the detailed design phase are revealed. The BCI control system is presented at the conceptual design phase in the form of an HS coordinator. At the detailed design phase, the BCI control system was implemented by internal control unit NXT 2.0, which consists of ARM7 and AVR microcontrollers and BCI control PC. A concrete model of BCI CS was implemented in BCI control PC in the form of finite state machine within the frameworks of Matlab program system using Simulink Stateflow module. The main element of the BCI control system is the MainController unit, which realizes logic laws of control of TruStan 24 electroencephalograph (EEG) unit, light stimulator and mobile robot. The mobile robot being controlled is Mindstorms robot. Necessary tests were carried out to verify the MainController-based BCI control system designed. Testing was performed by “injecting” recorded electroencephalographic (EEG) signals and introducing a set of simulation signals into the system. The offline tests allowed the selection of control algorithms based on statistical methods more efficient in detecting characteristic parameters of SSVEP [1,10]. The conducted tests confirmed the effectiveness of SSVE potential application in a mobile robot control, as well as EEG signals application as control signals in a BCI system. Further research aimed at implementing learning strategies of HS coordinator in the BCI MR control system to realize high-level behaviors of mobile robot learning from its low-level motions while performing selected tasks is planned. It will make possible further improvement of the BCI MR control system and realization of learning and decision-making functions in robot control tasks.