CameraEEG: Synchronous Recording of Electroencephalogram and Video Data for Neuroergonomics Applications ^†

Abstract

Lab-confined electroencephalogram experiments generally impel the subject’s mobility. Hence, we provide a wearable solution enabling human brain activity while monitoring during everyday activities, especially for neuroergonomics. This paper introduces CameraEEG, a new Android application that allows for synchronized smartphone acquisition of electroencephalogram (EEG) and camera data. Using a button on the app, the subject can record the witnessed audio-visual events of interest. Android SDK version 28 and mBrainTrain’s Smarting Mobi SDK were used to develop the app. The app can be used across all Android smartphones that have Android OS–Lollipop at least. In this paper, we used the app to record synchronized video and EEG data from four subjects during two tasks (namely, closed and open eyes), each under sitting conditions. We used the POz electrode data for analysis. There was a visible difference between the power spectrums of both the tasks, with the eyes-closed task reflecting an alpha band peak. Also, the obtained video and EEG data showed accurate synchronization. A download weblink for the .apk file along with a detailed help document for the developed app is provided for further testing.

1. Introduction

With the advancements in cognitive psychology, the present requirement is to investigate the neural responses that are naturally triggered by a dynamic environment in synchronicity with the participant’s task [1]. Applications using restricted stimuli in natural settings have also been developed in order to examine the variations in brain responses [2]. Electroencephalogram (EEG) and smartphone integration have made brain activity measurement possible on the move [3]. Attention studies that gauge the neural activity corresponding to different contexts within a classroom also employ portable electroencephalogram systems [4]. Recently, a study on the neuronal dynamics of interactive synchrony among musicians reported using smartphone-based hyper-scanning [5]. The above investigations might have been effective in locating events of interest if the camera data were also added alongside the EEG.

Synchronized EEG and camera data streams seem to be of current interest for natural environment experiments. Hence, we set up to develop a smartphone Android app called “CameraEEG” that enables this. Since the app was built for natural environment cognitive experiments, the EEG and rear camera data stream are synchronized. We used a transparent mobile holder pouch for placing the smartphone to enable hands-free video mode recording. Eventually, we tested the app’s performance with four subjects performing eyes-closed and eyes-open tasks while sitting. Offline analysis was carried out for the synchronization between the camera and electroencephalogram data streams. We then studied each task’s power spectral density (PSD) from the EEG signals. We believe that the CameraEEG app is the first of its kind that enables synchronized video and brain signals to be recorded on an Android platform.

Wascher et al. reviewed mobileEEG in neuroergonomics to understand mental states in workplace environments [6]. Our CameraEEG app will play a vital role in such scenarios by providing the opportunity to obtain EEG as well as video data. A portable solution for examination of cognitive states in workplace and daily life situations is becoming increasingly important for the neuroergonomics community.

2. Materials and Methodology

2.1. Software Architecture of Android Application

Java Development Kit (JDK) was used for the app development on Android Studio (AS). The EEG device manufacturer’s software development kit (SDK), version 2019 for data acquisition was also used, thus restricting the app to a particular EEG device, namely, mBrainTrain. However, the performance was robust. Two modules related to camera and EEG made up the app. An application programming interface (API) named camera2 was used from AS to modify the camera module. Built-in camera unit facilitates and stores image frames through a buffer. Google’s “Camera2Basic” functions were used to create the camera preview of the app (https://github.com/googlearchive/android-Camera2Video, accessed on 1 December 2023). This function uses a background handler thread to parallelize camera preview as well as record the video on smartphone. The closing and opening overlay of the camera device object was hindered by a binary semaphore. Resolution of the recorded video through the back camera was kept at 480 p. The camera preview feed is automatically scaled to fit the surface by rescaling its dimensions through a modified SurfaceTexture class called “AutofitTexture”.

2.2. Operation of Android Application

The steps for utilizing the app to record data are shown in Figure 1. First, we must connect the app to mBraintrain’s smarting EEG device’s Bluetooth. The app then activates the recording button (Figure 2), which records the synchronized EEG data as well as the video feed onto the smartphone as .bdf and .mp4 files, respectively.

Stopping and re-recording can be carried out using the same button on the app (Figure 2d). An event marker button is also available for the subject to mark any unique events during the experiment (Figure 2c). The apps. apk file is available for download at the following link: https://github.com/NeuralLabIITGuwahati/CameraEEG, accessed on 1 December 2023.

2.3. Synchronized EEG and Video Data Acquisition

Consent was obtained from our institute’s ethics committee at IIT Guwahati for the collection of data from human participants. Data were recorded from four subjects, three male and one female with a mean age of 25 and standard deviation 2.16 for this study. We used the developed app with an Easycap 24-channel headcap of mBraintrain smarting device (https://mbraintrain.com/smarting-wireless-eeg/, accessed on 1 December 2023). However, it is also compatible with 20-channel concealed-EEG (cEEGrid) electrodes. Before starting the synchronized EEG and camera recording, impedance levels of each electrode were checked using the SMARTING app. The Android smartphone was placed in a clear mobile holder pouch as in Figure 3, which permitted the recording of video stream with the rear camera and enabling touch-screen operation. The cell phone was not held continuously in hand by the subject but worn as in Figure 3, with the rear camera pointed away from the subject. A wearable experimental setup for the CameraEEG app is shown in Figure 3 and can include 20 cEEGrid electrode or headcap EEG setups.

Synchronization of video and EEG streams as well as data fidelity were checked when the subjects performed two tasks, namely, with their eyes open and closed. Each task lasted for about 5 min. The app had an event marker button to record situations of interest, but this was not used in this study.

3. Results

The video and EEG from the synchronized recordings were analyzed for any information loss and mismatch. The EEG and video data streams matched with less than a ±5 ms inaccuracy for all four subjects when we evaluated the recorded sessions for synchronization.

The recorded EEG time series during the two tasks were partitioned into two segments using MATLAB (Version 2021a) and EEGLAB (Version 2021.1) for further analysis. Custom scripts were written to analyze the data from the two tasks offline. The POz channel from the 24-channel EEG electrode setup was employed in this study. Following this, the change in the alpha activity was obtained as reported in [7]. Power spectral analysis depicted a peak around ~10 Hz during the eyes-closed task (Figure 4) across all subjects, indicating, as expected, a higher-alpha power.

4. Discussion

The app’s performance was deciphered from the data fidelity using the eyes-closed and eyes-open sessions, as performed in earlier works [8]. We observed an alpha peak in the eyes-closed sessions across all subjects, showing that the built app correctly acquired EEG data. Our CameraEEG app offers a straightforward integrated application to record unrestricted, natural electroencephalogram and video data for neuroergonomics applications. This data can also be utilized as a biomarker in intelligent building applications [9]. It also finds applications in understanding the mental implications for people living in urban environments [10,11]. Garza et al. found that high-beta power PSDs from prefrontal and frontal electrodes appeared during a museum walk compared with the baseline condition (staring at the wall) [12]. They also observed high-alpha power PSDs in central, parietal and occipital electrodes for the baseline condition. Hence, through this work, we demonstrate that good-quality synchronized video–electroencephalogram data can be acquired using an Android smartphone. This cost-effective solution also has numerous biomedical applications like monitoring epileptic, Alzheimer’s and sleep disorder patients.

We show that a single Android app can record synchronized EEG and video streams, in contrast to other research works, which appear to use several apps [1,13]. This configuration makes it simple to re-engineer for different applications by adding extra modules, decreasing the smartphone’s load compared with using multiple apps at once. However, our app is only compatible with mBrainTrain-based EEG devices, because it uses libraries from that specific EEG device manufacturer.

Although the recording sessions provided primary validation of the application’s operation, longer-duration recordings are necessary to thoroughly assess the app’s robustness. Running the CameraEEG application on Android phone models from later than 2020 (checked on the Redmi Note 7 pro with 4 GB RAM and Realme 5 pro with 6 GB RAM) showed very little heating. However, an older Android smartphone may become heated during extended recording sessions. Furthermore, since only 480 p resolution camera recordings were used, higher resolutions can degrade the smartphone’s performance by causing stutters of the toolbar. However, an enhanced video quality may provide increased resolutions for observing specific natural environment stimulus. Going forward, future challenges for this work include exploring the possibilities of online EEG analysis on smartphones like artefact removal and classifying tasks.