Assessment of Construction Workers’ Spontaneous Mental Fatigue Based on Non-Invasive and Multimodal In-Ear EEG Sensors

Abstract

Construction activities are often conducted in outdoor and harsh environments and involve long working hours and physical and mental labor, which can lead to significant mental fatigue among workers. This study introduces a novel and non-invasive method for monitoring and assessing mental fatigue in construction workers. Based on cognitive neuroscience theory, we analyzed the neurophysiological mapping of spontaneous mental fatigue and developed multimodal in-ear sensors specifically designed for construction workers. These sensors enable real-time and continuous integration of neurophysiological signals. A cognitive experiment was conducted to validate the proposed mental fatigue assessment method. Results demonstrated that all selected supervised classification models can accurately identify mental fatigue by using the recorded neurophysiological data, with evaluation metrics exceeding 80%. The long short-term memory model achieved an average accuracy of 92.437%. This study offers a theoretical framework and a practical approach for assessing the mental fatigue of on-site workers and provides a basis for the proactive management of occupational health and safety on construction sites.

1. Introduction

According to the “2023 Construction Industry Development Statistical Analysis” (China Construction Industry Association, 2024), the construction industry’s total output value in 2023 reached CNY 31.6 trillion, indicating a year-on-year increase of 5.77%. The completed output value amounted to CNY 13.8 trillion, reflecting a year-on-year growth rate of 3.77%, while the total contract value signed reached CNY 72.5 trillion, demonstrating an annual increase of 2.78%. This statistic highlights the enduring importance of the construction industry as a fundamental pillar of the national economy. Furthermore, 45.82 million rural migrant workers were engaged in the construction industry in 2023, accounting for 15.4% of the total number of migrant workers nationwide (National Bureau of Statistics of China, 2024). The industry faces challenges, such as high workforce mobility and an aging workforce. In this regard, the importance of construction safety management in China should be emphasized to mitigate risks and enhance workers’ well-being.

On construction sites, the worker is the most active and a core factor in the complex system of “person–work–environment”. More than 80% of casualties in the construction industry were caused by unsafe human behavior [1]. Workers go through a complex cognitive information-processing process when responding to external stimuli. From the perspective of cognitive theories, cognitive failure is a major cause of unsafe behavior [2]. Therefore, construction workers should undergo a well-defined and comprehensive cognitive process during specific construction tasks to help them stay continuously alert and attentive and enable accurate risk perception and judgment in a dynamic construction environment. The occurrence of unsafe behavior can be prevented by identifying potential risks [3].

Mental fatigue is an important representation of an individual’s mental load and is defined as a complex and multi-dimensional mental state of tiredness caused by long-term mental activities. Reducing workers’ cognitive abilities can cause cognitive failures, ultimately leading to unsafe behavior [4]. The execution of construction activities involves numerous specialized and technical tasks, which are characterized by their labor-intensive, high-risk, and mentally demanding nature [5]. As such, construction workers are prone to mental fatigue, with prolonged reaction time, distraction, and decreased alertness. The risk perception and decision-making ability of construction workers in a dynamic construction environment can be reduced [6]. In this circumstance, construction workers may tend to choose unsafe behavior, resulting in risky safety accidents.

The issue of mental fatigue should be addressed when managing the unsafe behavior of construction workers to enhance occupational health and safety [3]. Further, it is an embodiment of the people-oriented concept in construction safety management. Efficiently monitoring and identifying the level of mental fatigue experienced by construction workers is essential for the effective implementation of targeted intervention and management.

However, the accurate and dynamic assessment of mental fatigue in construction workers is still a debated topic because of the complex construction environment and activities. Current assessment methods relying on subjective self-reports or intrusive equipment are impractical for continuous on-site monitoring. This debate highlights the need for innovative and non-invasive approaches that can provide reliable and continuous data.

The implementation of brain–computer interface (BCI) technology and biological metrics has shown promising implications in safety and security fields. In contrast to research that primarily focuses on general safety management, the present study specifically addresses the dynamic monitoring and management of mental fatigue among construction workers. A wearable in-ear device with multimodal sensors was developed considering the distinctive and dynamic nature of construction sites as well as the activities carried out therein to enable the real-time and continuous integration of neurophysiological signal mapping of the development of spontaneous mental fatigue. The proposed multimodal in-ear device represents a pioneering application within the construction industry and is the first of its kind in this context. Furthermore, a comprehensive model for assessing mental fatigue was developed by leveraging the advanced capabilities of machine learning and deep learning algorithms. This study contributes to the existing body of knowledge by providing a novel and portable neurophysiological-based mental fatigue monitoring method customized for construction workers. The research findings have the potential to enhance on-site occupational health and safety management systems and a valuable contribution to the construction industry.

This paper is organized as follows. Section 2 presents the research background, which covers the theoretical foundations and literature review. Section 3 provides details on the multimodal in-ear sensors and describes the model for assessing mental fatigue. Section 4 presents the evaluation and results of the proposed method. Section 5 discusses the effectiveness of the proposed fatigue assessment method, its research implications, and limitations, and potential future work. Finally, Section 6 concludes the study.

2. Research Background

2.1. Management of Construction Workers’ Mental Load

According to the accident causation sequence model and the accident causation model [7], individuals typically experience different stages of cognitive failure as internal manifestations before the occurrence of unsafe behavior. The process of construction workers engaging in unsafe behavior can be categorized into five specific cognitive stages considering the characteristics of the construction industry: information acquisition, information comprehension, perception response, response selection, and action implementation. Construction workers must undergo a clear and well-defined cognitive process during specific construction activities to ensure the effective completion of these cognitive stages [8]. Mental load, also known as cognitive load, refers to the degree of cognitive resources required when an individual performs tasks or handles situations that demand focused attention, analysis, decision-making, and problem-solving [2]. It describes the psychological state and stress level experienced by an individual and encompasses a range of factors that affect the mental and physiological state. In the field of construction engineering, it reflects workers’ enduring cognitive stress due to task complexity knowledge level and environmental pressure [4,5].

Cognitive neuroscience is an emerging interdisciplinary field that integrates cognitive psychology, computer science, and neuroscience. Its primary approach aims to elucidate brain mechanisms underlying individual cognitive activities from the perspective of “gene–brain–behavior–cognition” [9]. Based on cognitive neuroscience theory, advanced neuroscience measurement tools and physiological parameter measurement devices are used to record individual behavioral processes and outcomes, collect objective data on accompanying neural and physiological external manifestations, and explore the psychological and behavioral patterns reflected in the data. In this context, integrating individual neural activities, cognitive activities, physiological states, emotional states, and behavioral patterns into safety management goals will achieve scientific, rational, safe, and efficient management in construction activities [9].

According to the methodology of cognitive neuroscience, computation-based methods utilizing vital signs have demonstrated significant potential for continuous and objective monitoring and assessment of the mental load of construction workers [10]. These methods involve the independent or combined application of vital sign parameters, such as electroencephalography (EEG), skin conductance, body temperature, respiration, electrocardiogram (ECG), and facial features. By capturing these vital signals, researchers have developed specific assessment models and algorithms for psychological load states, such as emotional stress, mental fatigue, and mood changes. They use techniques, such as EEG power spectral analysis, functional imaging, event-related potentials, and photoplethysmography. The application of machine learning techniques (such as transfer learning, active learning, and deep learning) plays a crucial role in decoding and analyzing complex physiological data and significantly improves analysis efficiency and prediction accuracy [10,11]. Assessment models can reveal the effects of specific construction environments and activities on the mental load state of worker groups and support the design and implementation of targeted intervention measures (e.g., [12,13]).

Construction workers often experience high levels of tension and pressure, irregular work schedules, and significant responsibility for their performance and safety. Therefore, their negative mental state is universal and affects individual cognitive abilities. This phenomenon is one of the important reasons for the reduction in production efficiency and the occurrence of safety accidents on construction sites [2]. Researchers have increasingly focused on managing construction workers from the perspective of their individual mental states. For example, Chen et al. (2016) [2] developed a novel technique for monitoring the mental states of construction workers for hazard evaluation. Wang et al. (2017) [14] investigated a quantitative and automated approach to evaluation of the attention levels of construction workers. Xing et al. (2019) [15] introduced a targeted intervention method to address the mental states of high-altitude construction workers. Xing et al. (2020) [5] examined the interaction between physical and mental fatigue among construction workers and provided strategies for managing fatigue. Ke et al. (2021) [12] investigated the relationship between the intrinsic cognitive states of construction workers and their exposure to noise, providing empirical support for enhanced noise management and safety measures. Ke et al. (2021) [13] monitored the distraction and attention of construction workers for safety management. Jeon and Cai (2023) [16] investigated the identification method of construction hazards on sites by developing a multi-class EEG signal classifier. Nino et al. (2023) [17] concentrated on the physical risk management of workers based on its interaction with perceived mental workload.

In summary, recent studies on occupational health and safety management on construction sites are increasingly incorporating mental and cognitive status as new variables. In this process, the integrated application of cognitive neuroscience methodologies and biometric methods provides a scientific basis for understanding and assessing the mental load levels of construction workers in specific environments.

2.2. Mental Fatigue Identification and Assessment

Mental fatigue is a crucial and common phenomenon in all workplaces that deserves public attention. It can generally be interpreted as a complex, multidimensional feeling of tiredness resulting from intense physical or mental work [18]. In particular, the construction industry is distinct from other sectors and presents significant challenges to the occupational health and safety of workers. In addition to physically demanding tasks, construction work often entails a variety of mentally strenuous and high-risk activities [5]. Workers must maintain constant vigilance and awareness of their dynamic surroundings to identify potential hazards [5]. Consequently, construction sites impose particular demands on the physical and mental states of workers. Typically, construction workers experience high mental workloads and often perform tasks while mentally fatigued. Further, mental fatigue can impair cognitive functions, resulting in slower reaction times, diminished vigilance, reduced decision-making capabilities, increased distractions, and loss of situational awareness [19]. Such conditions can negatively impact work quality and productivity and increase the likelihood of unsafe behavior on site [20]. Therefore, implementing effective strategies to measure, alleviate, and manage mental fatigue is essential to enhance overall safety management on construction sites.

Assessment of mental fatigue states (e.g., identification and quantification of fatigue states) is one of the most important aspects to promote the development of mental fatigue management. The status of mental fatigue can be observed through subjective self-reports, behavioral metrics, and neurophysiological data [21]. Studies on mental fatigue identification and assessment were usually carried out based on the above manifestations. Questionnaires and interviews based on self-reports are the most commonly used methods in fatigue assessment, and they inevitably have subjective and memory biases [22]. These subjective evaluation methods are cumbersome to implement in the construction industry because the time and effort required to answer the questions are likely to impair workers’ task performance [23]. An increasing number of researchers have used biological data to evaluate mental states considering the occurrence of local and overall physiological changes (e.g., variations in local muscle tissue, metabolism, and body temperature) during physically intensive activities [24]. According to cognitive neuroscience and basic psychology, biological signals mainly applied in reflecting individual mental fatigue are summarized as follows:

• EEG quantitatively reflects the brain’s electrical activity and offers an objective alternative to traditional survey-based assessments of mental states [25,26]. This non-invasive method measures voltage fluctuations from cortical neurons and has been widely used in research on mental load detection [27].
• ECG reflects the activity of the heart. Under the regulation of autonomic nerves, heart rate variability (HRV) can reflect the status of sympathetic nerves [28]. Regarding physiological responses, the potential correlations of mental fatigue with HRV have been verified [29].
• Respiration signals are used to estimate an individual’s mental status (e.g., mental fatigue, vigilance, and drowsiness) for mobile healthcare [30]. Along with other neurophysiological responses (e.g., ECG, EEG, HR, and eye movement), the multiple linear regression model and the machine learning model can be developed to achieve a high correlation to workers’ mental statuses [31].
• Galvanic skin response (GSR) is a generic term that indicates electrical activities originating from sweat glands, epidermal tissues, and dermal tissues. As a non-invasive technical tool, GSR reflects the robustness and sensitivity of mental status [32].
• Blood lactate and sweat lactate have demonstrated strong correlations with the mental fatigue status of construction workers [33]. In particular, as a non-invasive indicator, a sweat lactate-based sensor holds the potential for further development in assessing fatigue on construction sites.
• Eye movement data contains abundant information that can reflect the development of human fatigue during extended cognitive tasks [34]. Eye-tracking data can be continuously and unobtrusively measured by interacting with the human interface. The captured features can be used for real-time human fatigue detection [35,36].
• Facial features (e.g., eye aspect ratio, eye distance, mouth aspect ratio, face area, and head motion) have been explored as indicators of the mental fatigue of construction workers [37]. Similar to the eye-moving tracking technology, facial features can be applied in other scenarios, such as watching videos, driving, and performing surgical operations.
• In addition to the above physiological measures, contactless vital monitoring methods using cameras, wireless radar frequency, near-infrared spectroscopy, and acoustic-based sensing techniques, have increasingly been utilized in research related to mental fatigue [29,38,39].

Various approaches have been employed to recognize and measure mental fatigue based on captured signals. For example, Ishii et al. (2014) [40] introduced a conceptual model that utilizes a dual regulation system to explore the neural mechanisms of mental fatigue during cognitive tasks. EEG was used objectively to measure brain electrical activity and avoid subjective biases commonly found in traditional survey-based methods [25]. Li et al. (2012) [41] developed an EEG processing technique for evaluating the effects of driver fatigue. Duc (2014) [42] combined functional magnetic resonance imaging and EEG to study the neural regulation mechanisms of mental fatigue in specific brain regions. Yin and Zhang (2018) [43] proposed a classification method for mental fatigue based on different distributions of EEG features during various cognitive tasks. Importantly, machine learning technologies have played a pivotal role in decoding and interpreting neurophysiological data, across different work environments [44]. Techniques such as transfer learning, active learning, and deep learning have been applied to handle the complexities of data analysis [10]. Hajinoroozi et al. (2017) [45] introduced deep covariance learning models for predicting the drowsy and alert states of drivers by using EEG signals; the models demonstrated superior performance to shallow learning methods, particularly when CNN models are applied to spatial EEG covariance matrices. Tang et al. (2021) [46] proposed a promising method for detecting fatigue driving by using EEG signals; the method includes a Euclidean space data alignment approach to reduce individual differences and an efficient long short-term memory (LSTM) network structure to consider spatial correlations. Wang et al. (2023) [11] proposed a continuous wavelet transform and convolutional neural network to identify the mental fatigue states of construction workers. Mehmood et al. (2023) [47] investigated the deep learning-based mental fatigue identification of construction equipment operators by using wearable EEG sensors.

Considering the unique and dynamic environment as well as workplace activities on construction sites, limited biological-based approaches have focused on the dynamic monitoring of the mental fatigue of construction workers [10]. The adopted approaches are usually invasive, susceptible to interferences, and not portable or limited in an application environment [48]. For example, scalp EEG usually requires numerous electrodes to be firmly in contact with the scalp of the workers. It is unsuitable for practical engineering scenarios with frequent head movements or scalp sweating [49]. Applicable and novel assessment approaches need to be developed to mitigate the shortcomings of existing methods.

2.3. In-Ear EEG Development and Applications

Looney et al. (2012) [50] introduced the concept of in-ear EEG to a device for the recording of EEG signals. This portable and wearable device meets the criteria for effective monitoring, resembling typical earphones, earbuds, and earplugs while being compact and fitting directly around the ear. In recent times, in-ear EEG recording has gained attention as a user-centered and wearable brain monitoring method, showing potential for various emerging interactive applications such as BCI and biometric authentication [50]. Recent studies have focused on the materials, design, practicality, and signal quality of in-ear EEG systems. For example, Goverdovsky et al. (2016) [51] developed an innovative in-ear sensor using a viscoelastic substrate and conductive cloth electrodes capable of capturing high-quality brain activity from the ear canal. Similarly, Kappel et al. (2019) [52] designed an in-ear EEG with a soft custom-molded earpiece that demonstrated excellent signal quality and suitability for extended EEG monitoring.

Table 1. Summary of applications in previous ear–EEG studies.

Ear-EEG Type	Application	Selected Feature	Reference
Around-ear EEG	Auditory attention	Event-related potential	[54]
In-ear EEG	Sleep monitoring	Multi-scale fuzzy entropy	[55]
Around-ear EEG	Cognitive tasks	Common spatial pattern	[56]
In-ear EEG	Sleep monitoring	Power spectral density	[57]
In-ear EEG	Attention classification	Power spectral density and temporal features	[58]
In-ear EEG	Sleep staging assessment	Power spectral density and temporal features	[59]
Around-ear EEG	Eye-state identification	Filtered time-series	[60]

illustrates some specific applications of the in-ear EEG in previous research, including selected features. It can be concluded from Table 1 that medical and healthcare applications are the main themes of in-ear EEG research. Notably, sleep monitoring garners special attention [53]. Inspired by the rapid development and successful applications of in-ear EEG in these fields, its application on mental fatigue monitoring and feedback in the construction industry was investigated in this research. Enhanced implementation of occupational health and safety measures can be achieved on construction sites based on this premise.

3. Materials and Methods

3.1. Overview

This study developed a multimodal in-ear device for detecting the mental fatigue state of construction workers. The workflow of the study is illustrated in Figure 1 and consists of the following steps:

Step 1: Feasibility validation of the developed multimodal in-ear device. The proposed multimodal in-ear device was validated by comparing the data collected from in-ear EEG sensors with that obtained from the scalp EEG device. This validation step ensures the accurate capture of brainwave signals from subjects by the in-ear device.

Step 2: Data collection. EEG and ECG data were collected from participants by using the developed multimodal in-ear device. These data reflect participants’ brainwaves and heart activities and serve as the foundation for subsequent mental fatigue detection.

Step 3: Data preprocessing. Several preprocessing steps were performed to ensure accurate analysis and processing of the collected data.

Step 4: Feature extraction and selection. Relevant features related to mental fatigue were extracted from the data, and correlation-based feature selection (CFS) method was employed to identify the most significant features, thereby reducing model complexity and enhancing classification performance.

Step 5: Model establishment and training. Based on machine learning and deep learning techniques, we developed mental fatigue monitoring models, and the models utilized the extracted features for training.

Step 6: Model evaluation. The well-trained models were evaluated using appropriate evaluation metrics to assess accuracy in predicting mental fatigue. The best-performing model can be identified by comparing the performance of different models, and the results provide a viable solution for mental fatigue monitoring in real-world applications.

3.2. Multimodal In-Ear Sensors for Psychophysiological Signal Integration

In the pursuit of achieving non-invasive detection of mental fatigue in construction workers, the present study employed psychophysiological measures that adhere to the standards of cognitive neuroscience, fundamental psychology, and prior research paradigms. Section 2.1 delves into the comprehensive analysis of neurophysiological signals indicative of individual mental fatigue, encompassing EEG, ECG, respiration, and GSR, etc. Of particular significance is the exploration of in-ear EEG, an unintrusive methodology capturing aggregated electrical brain activities through the utilization of custom-designed earpiece electrodes embedded within the ear canal. Considering the critical aspects of portability and feasibility in data collection and analysis, this study adopts a psychophysiological metric system that integrates in-ear EEG and ECG measurements.

Expanding on the aforementioned metric framework, this research utilized a multimodal in-ear monitoring device (Figure 2) to assess workers’ mental fatigue. This device was developed by our research team in collaboration with the Hong Kong startup MindAmp (https://www.mindampltd.com/) and designed as a neck-mounted headset. The main unit is worn around the neck to enhance its performance, while two ECG electrodes extend from the device and are attached to the skin on the left side of the chest for precise signal acquisition. The in-ear EEG device is configured with a sampling rate of 500 Hz. This high sampling rate is chosen to capture fine-grained EEG signal variations, ensuring the temporal resolution needed for accurate analysis. This configuration offers several advantages. First, it provides stability during movement and minimizes any potential interference or artifacts that may affect signal quality. Additionally, this design enhances user convenience by allowing easy adjustment of electrode positions without compromising accuracy. Moreover, the lightweight design enhances comfort during extended monitoring sessions and is particularly beneficial when conducting long-term studies or assessments where participant compliance plays a vital role in obtaining reliable results.

Figure 2 illustrates the proper positioning of EEG electrodes within the ears, considering the bio-signal capture of the outer ear and the positioning of ear pieces. The position ensures accurate and reliable measurements by placing the electrodes near the brain’s electrical activity. Specifically, the earpiece pad substrate material is composed of elastic rubber, which helps absorb artifacts caused by small and large mechanical deformations of ear canal walls. An EEG recording electrode, which is used to detect brain activity, is embedded in the left ear pad. On the right ear pad, a reference electrode and a ground electrode are installed to ensure stable signal reception. The flexible in-ear EEG device is designed with a strategic distribution of electrodes on the earphone, allowing it to accommodate various ear shapes and sizes. This ensures optimal signal detection for accurate and reliable brainwave monitoring.

In addition to its ergonomic design, the in-ear device operates wirelessly and eliminates the need for cumbersome wires or cables that can restrict movement or cause discomfort. With just a simple press of a button located on its body, users can easily power on/off the device and provide convenience and ease of use. To ensure seamless data transmission, this innovative device utilizes Bluetooth low-energy technology. This wireless communication protocol enables efficient and secure transfer of brainwave signals from the device to a connected smartphone or computer. Users can conveniently access their real-time physiological activity data without any interruption or interference.

Prior to its application in mental fatigue assessment, the integration of neurophysiological signals from the in-ear sensor has been validated through initial experiments with three participants. The scalp EEG cap (EMOTIV EPOC+ 14 Channel Mobile EEG) (Figure 3a) was used for comparison. The feedback received from the participants was crucial in evaluating the usability of the in-ear device. Prior to the preliminary experiment, participants wore two devices separately in the powered-off state while engaging in routine learning activities, such as walking, sitting, and working, for a duration exceeding 30 min. After completing the wearing experience, they reported that it fit comfortably and did not hinder their movements, indicating that it can be worn without causing any discomfort or inconvenience during daily activities. During the preliminary experiments, participants were equipped with two devices at the same time and instructed to alternate between keeping their eyes open and closed for 30 s each. The correlation coefficients between the in-ear EEG channel and the scalp EEG channels were then calculated. The measurement values of the scalp EEG recording electrodes are higher than those of the designed in-ear device because of the different reference electrode placements between the utilized scalp EEG device and our designed multimodal in-ear device. For the correlation experiment, the utilized scalp EEG device used electrodes behind the ears as reference electrodes, while the designed in-ear EEG device adopted the electrode placed in the right ear as the reference electrode.

Figure 3 illustrates the topological correlation coefficients, which reveal a positive correlation between the activity captured by the in-ear EEG channel (left ear electrode) and the corresponding scalp EEG channels located on the left hemisphere. In contrast, a negative correlation was observed with the channels on the right hemisphere. This finding suggests that the in-ear EEG activity can provide meaningful insights into brain activity, particularly with respect to temporal dynamics. Furthermore, the heatmap representing the correlation coefficients provides visual confirmation that in-ear EEG signals can effectively reflect brain activities. The notable correlations highlight the potential of in-ear devices for monitoring cognitive states, especially temporal activities related to mental fatigue. Given the increasing demand for wearable technology in neurophysiological assessments, the findings from the initial experiments support the feasibility of using in-ear EEG sensors as a practical tool for continuous monitoring of mental fatigue.

In summary, aiming at the developed in-ear sensors, the following objectives were achieved through the preliminary experiments: (1) the verification of effectiveness in the real-time and continuous integration of neurophysiological signals, (2) the verification of feasibility in reflecting EEG signals, and (3) the verification of its comfort as a wearable device. Availability and effectiveness of the multimodal in-ear sensors in mental fatigue assessment were then explored through the following formal experiment.

3.3. Mental Fatigue Assessment Model Based on In-Ear Sensors

3.3.1. Dataset Construction for Model Establishment

Experiment Protocol and Data Collection

In this research, a mental fatigue dataset needs to be developed for the assessment model establishment to verify the suitability of the proposed in-ear device. According to the research objective, a cognitive experiment was designed and conducted for multimodal data collection (Figure 4).

The experiment involved the recruitment of 16 healthy male individuals with a background in construction engineering and prior experience working on construction sites. These participants were selected from the pool of students at the Hong Kong Polytechnic University to serve as experimental subjects, and their demographic information is presented in

Table 2. Demographic information of the subjects included in the study.

No.	Age	Gender	Total Number	Features
1	28	Male	8	Graduate students who possessed practical experience in the fields of construction engineering and management
2	27
3	25
4	28
5	28
6	28
7	27
8	25
9	30	Male	8	Individuals involved in work related to engineering management.
10	33
11	29
12	32
13	30
14	29
15	31
16	35

. Referring to existing methodologies of cognitive psychological and physiological experiments [33,61,62], the choice of participants was based on their accessibility and alignment with specific demographic characteristics. This strategy allowed us to gather data from individuals who have the necessary technical knowledge and practical insights relevant to our research objectives. According to the protocol developed for the objective of this experiment, all the subjects were in optimal physical condition and exhibited no signs of psychological disorders. They were required to keep a regular daily routine the day before the experiment to ensure a good physical and mental state.

The experiments were conducted in a controlled indoor laboratory environment, where conditions such as temperature and lighting were kept constant to minimize external influences on signals. Prior to the experiment, the entire experimental procedures and the device used were introduced to all the subjects. All participants were given sufficient time to practice and be familiarized with the trial procedure. In this process, they were required to wear the in-ear device while performing routine activities such as walking, sitting, and standing. This step was conducted to evaluate the device’s stability under different movement conditions, preventing signal loss or electrode displacement.

During the experiment, all participants were required to wear the in-ear device and scalp EEG device simultaneously to ensure accurate and comprehensive neurophysiological data throughout the entire duration of the study. The cognitive task chosen for this experiment was the computer n-back task, which is a well-established test for assessing sustained attention and working memory abilities [30]. Mental fatigue is characterized by a diminished state of alertness and impaired cognitive performance resulting from prolonged periods of mentally demanding activities [63]. In the n-back task, participants are required to monitor a continuous sequence of stimuli and indicate when the current stimulus matches the one from n steps earlier in the sequence. The difficulty of the task increases with higher n values, which require participants to hold and manipulate more information in their working memory. In this study, subjects were asked to complete a computer 1-back task (Figure 4). More specifically, the experiment employed the triple stimuli, which encompassed the integration of position, audio, and color elements. Each stimulus was presented for 2 s, by which the participants responded to targeted elements the same as the last stimulus by pressing 0–3 different keyboard buttons.

Before the cognitive task, participants were asked to sit quietly for 5 min to record baseline signals, serving as a reference point in subsequent data analysis. Then, different levels of mental fatigue can be induced by subjecting participants to an 80 min long cognitive task. In this study, we made the following assumptions to simplify the research. Participants’ self-assessment of mental fatigue levels exhibits similarities and lacks significant differences. This finding implies that regardless of whether the assessment is conducted before or after task execution, individuals’ subjective perception of their own mental fatigue levels remains consistent and does not show significant changes. In addition, the fatigue level evaluated by a modified Rating of Fatigue (ROF) scale [64] and the neurophysiological signal reflected by the scalp EEG were used as the ground truth of the real-time mental state (Figure 4). First, to monitor the participants’ subjective perception of their own fatigue levels, we conducted self-assessments every 20 min during the task by using a standardized scale called ROF (Rating of Fatigue). This allowed us to gather valuable information about how fatigued each participant felt at different time points throughout the experiment. In addition to self-assessment measures, we relied on objective indicators such as scalp EEG recordings and performance decrements in the cognitive task itself. Scalp EEG provides insights into brain activity patterns associated with different states of consciousness, including fatigue. By analyzing these neurophysiological signals in real-time via Bluetooth transmission, we were able to observe any changes or fluctuations indicative of increasing or decreasing levels of fatigue. For each participant, the scalp EEG indicator was determined by calculating the grand average of (theta + alpha)/beta for each 60-s data segment. To ensure the reliability of fatigue assessment, we evaluated the progression of mental fatigue by examining the correlation between this indicator and the results obtained from the ROF scale. Please refer to existing studies for the specific methods of judging the development trend of mental fatigue by the scalp EEG [5,35]. Finally, three 60 s raw data segments were chosen for each subject, reflecting distinct levels of mental fatigue: low, medium, and high levels of mental fatigue.

Feature Selection and Dataset Construction

Signal preprocessing is a crucial and indispensable step in ensuring the accuracy and reliability of subsequent data analysis. Considering numerous frequency noises due to respiration, heartbeats, and other needless power frequencies, the raw EEG data were preprocessed mainly through the band-pass filter, independent component analysis (ICA), and invalid segment elimination, based on widely accepted standards in EEG signal processing [35]. Specifically, the band-pass filter was set with a cutoff frequency range of 0.5 Hz to 50 Hz, which is based on widely accepted standards in EEG signal processing. This range effectively retains the essential components of the EEG signal while filtering out most of the non-brain-related artifacts, such as low-frequency movement artifacts and high-frequency environmental noise. ICA is a well-established blind source separation technique that can effectively disentangle different signal components without losing important information from the original data. In this research, ICA was used to further separate and remove artifact signals, such as those caused by eye movements, muscle activity, or sweating. In practice, we manually inspected the independent components identified by ICA and used recognized artifact patterns described in the literature to filter out the artifact components. The raw ECG data underwent an initial processing step with the Butterworth filter [65] to eliminate high- and low-frequency noise, enhancing the signal quality. Baseline drift method [66] was employed to mitigate baseline fluctuations, simplifying signal analysis and interpretation. Finally, the processed ECG signals were passed through the Pan–Tompkins algorithm [67] for R-peak detection, which was then used to calculate important indicators, such as heart rate (HR) and heart rate variability (HRV).

A moving window technique with a duration of 10 s and a 9.75 s overlap was employed [68,69]. The window size ensures each ECG data segment has sufficient heart rate information. Moreover, mental fatigue assessment based on every 10 s data segment can meet the timely needs in future practical applications. By integrating the ECG preprocessing process, we eliminated some error samples in the data segments, resulting in a final dataset of 10,512 samples. Data features potentially mapping to the mental fatigue assessment were extracted. For the single channel of the in-ear EEG, the targeted frequency bands (delta (δ) ranging from 0.5 Hz to 4 Hz, theta (θ) ranging from 4 Hz to 8 Hz, alpha (α) ranging from 8 Hz to 13 Hz, beta (β) ranging from 13 Hz to 30 Hz, and gamma (γ) ranging from 30 Hz to 40 Hz) were utilized to compute temporal features, as illustrated in

Table 3. EEG-related features extracted from the developed multimodal in-ear device.

Signal Type	EEG Signals	Frequency	Extracted Features	Number
In-ear EEG	delta (δ)	0.5–4 Hz	Mean amplitude, standard deviation, peak-to-peak amplitude, skewness and kurtosis calculated from waves of δ, θ, α, β and γ.	25
	theta (θ)	4–8 Hz
	alpha (α)	8–13 Hz
	beta (β)	13–30 Hz
	gamma (γ)	30–40 Hz

. Additionally, for the ECG data, features related to the heart rate variability were collected and can be found in

Table 4. ECG-related features extracted from the developed multimodal in-ear device.

Signal Type	Extracted Features	Description	Number
ECG	mRR	Mean duration between two consecutive R waves (R-R intervals) in the QRS signal on ECG.	6
	SDRR	Standard deviation of all of the R-R intervals.
	RMSSD	Root mean of the squared differences between consecutive RR intervals.
	LF-HRV	Normalization of the low-frequency band in heart rate variability (HRV).
	HF-HRV	Normalization of the high-frequency band in heart rate variability (HRV).
	LF/HF	Normalized ratio between the LF-HRV and the HF-HRV.

.

Among the extracted multi-features, each feature usually has different orders of magnitude as a result of its different nature. For example, the value of LF/HF reaches thousands or even tens of thousands, while the value of HF-HRV is less than 1 ms². If the features, LF/HF and HF-HRV, are directly input into the fatigue classification model, the role of LF/HF in the trained model will be highlighted, resulting in the inaccurate classification performance of the trained model. Therefore, to improve the reliability of the classification model and optimize convergence speed effectively, a normalization process was performed on the input features prior to their utilization in the training and prediction stages of the assessment models.

In addition, the feature selection operation was necessary to extract the optimal feature subset from the original feature space for establishing the fatigue classification model. Theoretically, more input features can provide richer information for mental fatigue prediction model; however, too many features may cause feature redundancy and excessive calculation, especially noise can be introduced, which can have a detrimental effect on the accuracy of the model’s predictions. To address this, this study incorporated the correlation-based feature selection (CFS) method to select the optimal input features, which reduces the dimensionality of the features considering feature–feature correlations and the contribution of the features themselves to the model prediction performance [70]. CFS is a filtering feature selection method that aims to identify a subset of features with minimal inter-feature correlation and strong correlation with the target variable, thus effectively removing redundant and weakly correlated features to maintain or improve the predictive power of the model. This method evaluates the quality of each established feature subset according to the proposed Merit scoring function. A higher Merit score of the extracted feature subset indicates strong correlation with the classification results and independence among the included features. The presented Merit score function can be computed according to Equation (1).

$$Meri{t}_S={\displaystyle \frac{k\overline{r_{cf}}}{\sqrt{k+k(k-1)\overline{r_{ff}}}}}$$

(1)

where k denotes the number of features contained in the feature subset S, $\overline{r_{ff}}$ represents the average feature–feature correlation, and $\overline{r_{cf}}$ denotes the average feature–class correlation. Here, for the calculation of correlation, the Pearson correlation coefficient was used, which can be expressed by Equation (2): x represents a feature in S, \y refers to a feature or a classification result, and N is the number of samples.

$$r_{xy}={\displaystyle \frac{{\displaystyle \sum xy-\frac{{\displaystyle \sum x{\displaystyle \sum y}}}{N}}}{\sqrt{[{\displaystyle \sum x^2}-\frac{{({\displaystyle \sum x})}^2}{N}][{\displaystyle \sum y^2}-\frac{{({\displaystyle \sum y})}^2}{N}]}}}$$

(2)

CFS method is implemented using a best-first search strategy to identify the optimal feature subset, denoted as S. It begins by calculating the correlation matrices for feature–class and feature–feature relationships based on the training data. This method then selects the features with the highest Merit value and adds it to the initial empty feature subset, S. Subsequently, additional features are sequentially considered for inclusion in the feature subset. If the overall Merit value of the feature subset S increases with the addition of a feature, then it is incorporated into S; otherwise, this feature is discarded from the subset. The aforementioned steps are repeated iteratively until the feature subset with the highest Merit score is obtained, representing the optimal feature subset. The features included in this optimal subset are subsequently utilized as inputs for training the fatigue classification model.

In this research, the number of features and their corresponding Merit values are increasing when implementing the CFS method and the best first search to find the features that meet the requirements (

Table 5. The generated subsets of features based on the CFS method.

No.	Features Combinations	Merit Values
1	mRR	0.420
2	mRR, β-kurtosis	0.488
3	mRR, β-kurtosis, γ-kurtosis	0.515
4	mRR, β-kurtosis, γ-kurtosis, β-standard deviation	0.520
5	mRR, β-kurtosis, γ-kurtosis, β-standard deviation, SDRR	0.521

). The final feature combination selected for this study is {mRR, β-kurtosis, γ-kurtosis, β-standard deviation, SDRR}, with the Merit value reaching 0.521. Among them, β-kurtosis, γ-kurtosis, and β-standard deviation represent the kurtosis of the β wave, the kurtosis of the γ wave, and the standard deviation of the β wave, respectively.

The chosen combination of features is considered optimal, as it meets the criteria of having low inter-feature correlation and high correlation with the target class, which can be reflected from Figure 5 and Figure 6. Figure 5 showcases the correlation among the five selected features. The correlation between any two selected features is relatively low, with the maximum not exceeding 0.3 (β-kurtosis vs. β-standard deviation). Figure 6 provides the boxplots for the selected features. It is evident that the boxplots of these five features do not completely overlap (no boxes of the same size), which suggests that the selected features have individual contributions in distinguishing the mental fatigue levels.

According to above feature selection, for application to the later assessment model establishment, the dataset was denoted as ${D=\{D}_p\mathrm{丨}p\in \{1,\mathrm{2,3},\dots ,P\}\}$, where P signifies the count of participants; D_p represents the data collected from participant p. In this dataset, $X={\{x_1,x_2,\dots ,x_t,\dots ,x_n\}}^T\in {\mathbb{R}}^{\mathrm{n}\times \mathrm{j}}$, where n represents the sample size; j is the number of features of in-ear EEG and ECG. That is, $x_t=({IEEG}_t,{ECG}_t)\in {\mathbb{R}}^2$, where ${IEEG}_t$ represents values of in-ear EEG features, ${ECG}_t$ represents values of ECG features. According to the ROF scale, the mental fatigue level $Y_i\in \left\{\mathrm{0,1},2\right\},i=\mathrm{0,1},2\dots ,n$, was taken as the output of the model.

3.3.2. Model Establishment for Mental Fatigue Assessment

Following the preprocessing, feature selection, and dataset creation steps outlined in Section 3.3.1, a range of machine learning and deep learning classifiers were utilized to establish mental fatigue classification models, which were then used to investigate the feasibility of the proposed multimodal in-ear device for monitoring mental fatigue. Note that the classifiers selected in this study were chosen based on their established effectiveness in similar studies on fatigue classification, including six widely recognized models from the fields of machine learning and deep learning such as k-nearest neighbor (KNN) [71], support vector machine (SVM) [62], decision tree (DT) [61], random forest (RF) [71], convolutional neural network (CNN) [45] and LSTM network [46]. These models have demonstrated good classification performance in prior research [72,73]. The open-source Python-based machine learning library (scikit-learn), as well as the deep learning library (i.e., Keras), can easily implement efficient model training and assessment, enabling an evaluation of the practical feasibility of the developed multimodal in-ear device for monitoring workers’ mental fatigue in real-world applications. Note that for machine learning models, such as DT and RF, that contain many hyperparameters, we summarize the hyperparameters of the mentioned machine learning models in

Table 6. Hyperparameters of machine learning models.

Models	Hyperparameters
KNN	1. metric: “chebyshev”, “euclidean”, “manhattan” or “minkowski”. 2. n_neighbors: count of nearest neighbors to consider during the classification process. 3. weights: weighting method: “uniform” or “distance”.
SVM	1. C: regularization parameter. 2. kernel: kernel function: “linear”, “poly”, “sigmoid”, or “rbf”.
DT	1. max_depth: the maximum number of levels that the tree can grow. 2. criterion: metric employed to assess the quality of a split. 3. max_features: the maximum number of features to consider when looking for the best split. 4. min_samples_leaf: the minimum number of data samples needed in a leaf node.
RF	1. n_estimators: the count of decision trees to be constructed in the ensemble of random forests. 2. min_samples_split: the minimum number of data samples needed at an internal node in order to perform a split. 3. min_samples_leaf: the minimum number of data samples needed to form a leaf node. 4. max_features: the upper limit on the number of features considered during the splitting process of a node. 5. max_depth: the maximum number of levels or depth that a decision tree can reach within the ensemble. 6. bootstrap: the specific sampling method used during the construction of decision trees.

. Moreover, the hyperparameters of each machine learning model were optimized using a combination of grid search and K-fold cross-validation, as outlined in

Table 7. Steps for selecting the hyperparameters of machine learning models by using a combination of grid search and K-fold cross-validation.

Steps	Description
1	Randomly partition the entire dataset into a training set and a test set.
2	Define the grid search space and determine the hyperparameters for each model.
3	Divide the training set into k subsets, reserving one subset as the validation data for evaluating the prediction performance of each model with different hyperparameter combinations. The remaining k − 1 subsets are used for training the model.
4	Repeat the process in Step 3 for k times, ensuring that each subset is used as the validation data exactly once.
5	Iterate through all hyperparameter combinations of the models, repeating Steps 3 and 4 for each combination.
6	Identify the hyperparameter combination that achieves the highest score for each model from the k-time training and validation. Utilize this optimal hyperparameter combination to evaluate the mental fatigue classification performance on the test set.

. A detailed description of the machine learning models used in this study, including their theoretical foundations, algorithmic structures, and typical applications, can be found in [74]. This reference encompasses pivotal models utilized in this study such as KNN, SVM, DT, and RF. In addition, CNN and LSTM networks are the deep learning models utilized in this study, with their network architectures illustrated in Figure 7a. These networks employ different approaches in processing data. The CNN network processes data through convolutional layers, fully connected layers, and maxpooling layers, whereas the LSTM network processes data through stacked LSTM cells. As shown in Figure 7b, a typical LSTM cell consists of a forget gate, an input gate, an output gate, and the cell state. The input gate i_t controls the update of new information to the current state c_t, the forget gate f_t determines which information from the previous cell state c_t−1 should be discarded, and the output gate o_t regulates the output h_t of the LSTM cell. The update of the cell state in relation to the three gates can be described as follows:

$$\left\{\begin{array}{l}{f}_t=\sigma (W_f·[h_{t-1},x_t]+b_f)\\ i_t=\sigma (W_i·[h_{t-1},x_t]+b_i)\\ o_t=\sigma (W_o·[h_{t-1},x_t]+b_o)\\ {\tilde{c}}_t=\tanh (W_c·[h_{t-1},x_t]+b_c)\\ c_t=f_t\odot c_{t-1}+i_t\odot {\tilde{c}}_t\\ h_t=o_t\odot \tanh (c_t)\end{array}\right.$$

(3)

where t represents the time step, x_t is the input vector. $\sigma$ and tanh correspond to the sigmoid activation function and hyperbolic tangent activation function, respectively. C_t−1 represents the cell state from the previous time step, while c_t is the new candidate value vector used to update the current cell state. h_t−1 refers to the hidden state from the previous time step. $W_{\alpha }$ and $b_{\alpha }$ ($\alpha =\left\{f,i,o,c\right\}$) represent the weight matrices and bias terms for the different gates. The symbol $\odot$ indicates the Hadamard product.

For the deep learning models, the hyperparameters were selected based on the recommended settings from the relevant literature. A limited number of training sessions were conducted because of slow training speed and challenges in achieving convergence with the loss function. Finally, the hyperparameter values of the deep models can be obtained based on the preliminary training results.

A series of multi-categorical evaluation metrics, including macro average precision (i.e., macroP), macro average recall (i.e., macroR), macro average F₁ score (i.e., macroF₁) [36,75], and Average Accuracy [76], was used to assess the classification capability of each model and obtain a holistic assessment of the classification performance for workers’ mental fatigue. macroP and macroR calculate the precision and recall for each class in a one against all manner first and then derive the aggregate measures by averaging the precision and recall values across all classes, which can be written as Equations (4) and (5), respectively. macroF₁ is the harmonic mean of macroP and macroR, as shown in Equation (6). Additionally, Average Accuracy presents the percentage of correct classifications for each model, which can be expressed as Equation (7).

$$macroP={\displaystyle \frac{1}{c}}{\displaystyle {\sum }_{i=1}^c\frac{T{P}_i}{T{P}_i+F{P}_i}}$$

(4)

$$macroR={\displaystyle \frac{1}{c}}{\displaystyle {\sum }_{i=1}^c\frac{T{P}_i}{T{P}_i+F{N}_i}}$$

(5)

$$macro{F}_1=2\times {\displaystyle \frac{macroP\times macroR}{macroP+macroR}}$$

(6)

$$AverageAccuracy={\displaystyle \frac{1}{c}}{\displaystyle {\sum }_{i=1}^c\frac{T{P}_i+T{N}_i}{T{P}_i+F{N}_i+F{P}_i+T{N}_i}}$$

(7)

where c is the number of workers’ mental fatigue levels (Y_i). In this study, c is set to 3. TP_i denotes the count of samples from class Y_i that the model accurately predicts, while FP_i represents the count of samples from class Y_i that the model misclassifies. FN_i corresponds to the count of samples from class Y_i that are erroneously classified as other classes by the model, while TN_i represents the count of samples correctly classified by the model as categories other than class Y_i. In addition, the utilization of a confusion matrix allows for a visual depiction of the disparities between the classification outputs of each model and the actual ground-truth values. The confusion matrix is a square matrix of size $n\times n$, where each row represents the true mental fatigue category of each sample, while each column denotes the mental fatigue category predicted by the model. Therefore, the values along the diagonal line of the confusion matrix, spanning from the top left to the bottom right, indicate the overall count of samples that are accurately classified by the model, i.e., ${\displaystyle {\sum }_{i=1}^{n}T{P}_i}$.

4. Evaluation and Results

In this study, the dataset established in Section 3.3.1 was split into two sets: a training set and a test set, with a ratio of 9:1. Prior to establishing the dataset, all samples were randomly shuffled. The training set was subsequently divided into five equal segments to facilitate the implementation of fivefold cross-validation (K = 5) for grid searching optimal hyperparameter combinations across the machine learning-based models employed in this study. Moreover, the sensitivity of hyperparameters for each machine learning model was assessed, and minimal performance differences were observed across various hyperparameter settings. The second column of

Table 8. Mental fatigue classification performance of the six implemented models.

Models	Hyperparameter	Value	macroP	macroR	macroF1	Average Accuracy
KNN	metric	“manhattan”	85.594%	85.549%	85.572%	89.880%
	neighbors	6
	weights	“distance”
SVM	C	1000	81.935%	81.336%	81.634%	86.757%
	kernel	“rbf”
DT	max_depth	“None”	84.060%	84.029%	84.044%	88.826%
	criterion	“entropy”
	max_features	8
	min_samples_leaf	1
RF	n_estimators	2240	86.888%	86.871%	86.879%	90.647%
	min_samples_split	60
	min_samples_leaf	8
	max_features	“auto”
	max_depth	20
	bootstrap	“False”
CNN	1st convolutional layer: filters; kernel_size	64; 2	82.127%	82.118%	82.122%	87.332%
	2nd convolutional layer: filters; kernel_size	64; 3
LSTM	number of LSTM unit	256	89.067%	89.068%	89.068%	92.437%
	number of LSTM layers	2

lists the optimal hyperparameter combinations for each model.

In the case of the deep learning models, the CNN model employed a two-layer configuration of 1D CNN units, while the LSTM model utilized two layers of LSTM units to encode the input features. We adopted the deep learning framework of Keras and trained on a server equipped with four NVIDIA GP104GL (Tesla P4) graphics cards. The Python version employed was 3.8. In the CNN model, ReLU was employed as the activation function for the convolutional layers, while Softmax was utilized for the fully connected layers. Likewise, in the LSTM model, the initial learning rate was set to 0.001, tanh was designated as the activation function for the LSTM layers, and Softmax was chosen for the fully connected layers. Throughout the training process, both deep learning models were configured with a batch size of 32, utilized the Adam optimizer, and were trained for 100 epochs. Figure 8 illustrates variations in loss function and accuracy throughout the training process for the CNN and LSTM models. As shown in Figure 8a, the CNN and LSTM models exhibited a rapid reduction in loss during the initial stages of training. With increasing training epochs, the training loss gradually decreased and eventually stabilized. The CNN model achieved stability at approximately 40 epochs with a loss value of around 0.47, while the LSTM model stabilized at approximately 55 epochs with a loss value of about 0.25. Correspondingly, the CNN and LSTM models demonstrated an initial rapid increase in training accuracy, which improved gradually with additional training epochs until it reached a stable level. The CNN model achieved a training accuracy of 83.368%, while the LSTM model reached a higher training accuracy of 92.283%. Notably, the LSTM model took longer to reach a stable performance level, but it converged to a smaller training loss value, which suggests that the LSTM model effectively learned the provided feature data.

After obtaining the optimal hyperparameters, the performance of each model for mental fatigue classification was evaluated based on the test set by using four metrics, namely, macroP, macroR, macroF₁, and Average Accuracy. Table 8 also reports the calculated values of the evaluation metrics for the implemented six machine learning and deep learning models. In the task of mental fatigue classification, the six implemented models had performance differences. The LSTM model exhibits excellent performance, attaining the highest Average Accuracy (92.437%) and macroP (89.067%) and also achieving the best results in terms of macroR (89.068%) and macroF₁ (89.068%). These outcomes highlight the outstanding capabilities of the LSTM model in mental fatigue classification, making it highly suitable for research and practical applications. Additionally, the RF model also demonstrates high performance, with an Average Accuracy of 90.647%, macroP of 86.888%, macroR of 86.871%, and macroF₁ of 86.879%. These results underscore the accuracy and reliability of the RF model in the classification of mental fatigue. Comparatively, the KNN and DT models exhibit relatively balanced performance across all metrics, with the Average Accuracy of 89.880% and 88.826%, and the macroF₁ of 85.572% and 84.044%, respectively. However, the SVM and CNN models exhibit slightly lower performance. The SVM model achieves a macroR of 81.336%, while the CNN model achieves a macroR of only 82.118%. Despite these relatively lower scores, the SVM and CNN models exceed the threshold of 80% for all evaluation metrics. Hence, machine learning and deep learning models can accurately identify mental fatigue levels by using the data captured by the proposed multimodal in-ear device, which further validates the feasibility and effectiveness of the designed multimodal in-ear device for monitoring mental fatigue. In practical applications, the EEG and ECG signals obtained from the multimodal in-ear device undergo preprocessing and are directly input into trained models for prediction; the prediction results serve as an indication of the mental fatigue states of construction workers.

The confusion matrix provides a specialized visualization of the classification performance for each model and enables a clear representation of the disparities between the predicted classifications and the ground-truth labels. Figure 9 illustrates the confusion matrices for the six classifiers using the test set, where the true mental fatigue levels are displayed along the x-axis, while the predicted mental fatigue levels are displayed along the y-axis. The values along the diagonal line, spanning from the top left to the bottom right, in the confusion matrices indicate the counts of correct classifications for mental fatigue levels achieved by the various models. It is evident from the confusion matrices that each model can accurately predict the correct mental fatigue level. The number of correctly classified samples significantly exceeds the number of misclassified samples. For example, as shown in Figure 9, the KNN model can accurately identify 900 samples from the test set of 1052 samples, with only 152 samples misclassified. Notably, the LSTM model exhibits the best predictive performance for mental fatigue, accurately classifying 937 samples from the test set. In summary, Figure 9 indicates that all models display exceptional capabilities in accurately classifying levels of mental fatigue.

5. Discussion and Limitations

Given the unique nature of construction work, construction workers commonly experience elevated mental workloads, which could lead to unsafety behavior and numerous other undesirable outcomes. From the human-centered and proactive control perspective, this study aimed to examine the early detection of adverse mental load among construction workers, prioritizing their well-being and safety. Based on the cognitive perspective, the mental fatigue of construction workers was chosen for the study of unsafety behavior management. Specifically, this research proposes a novel and portable monitoring method utilizing multimodal in-ear sensors.

5.1. Effectiveness of the Proposed Monitoring Method of Spontaneous Mental Fatigue

Mental fatigue can manifest through multiple aspects, including subjective evaluation, changes in cognitive performance, physiological responses, etc. In this research, neurophysiological measures were used for the portable assessment of mental fatigue among construction workers on construction sites. Referring to cognitive neuroscience and basic psychology, relationships between individual cognitive processes and external neurophysiological performance were analyzed. In particular, the neurophysiological metric system was established utilizing the in-ear EEG and ECG data considering the portability and feasibility of the above neurophysiological measures in data acquisition and analysis. The optimal and key data features selected for mental fatigue assessment were determined by considering the feature–feature correlations and evaluating their contributions in the classification models by using the correlation-based feature selection (CFS) method.

On this basis, this research empirically evaluated six supervised classification models, encompassing traditional machine learning approaches and advanced deep learning techniques. The proposed multimodal in-ear device exhibited promising performance in recognizing mental fatigue across various well-established classification models, with greater than 80% evaluation metrics (i.e., macroP, macroR, macroF₁, and Average Accuracy). The RF and LSTM models had an Average Accuracy above 90%. The LSTM model showcased exceptional prediction performance in classifying workers’ mental fatigue, with an Average Accuracy of 92.437%. In the confusion matrix, 115 misclassified instances occurred out of 1052 test samples. One potential explanation for this finding arises from the inherent characteristics of the model itself. The LSTM model is specifically designed to handle time series data, making it highly suitable for data that evolves over time, such as EEG and ECG data. The manifestation and progression of mental fatigue unfold over time, presenting long-term temporal dependencies that the LSTM model can effectively capture and retain through its built-in memory cells and gating mechanisms (input gate, forget gate, output gate). This capability allows the LSTM model to store past information and leverage it in its predictions, thereby enhancing the accuracy of its assessments regarding mental fatigue. This research finding coincides with conclusions from other scholars. For example, Sarkar et al. (2022) [77] reported that the LSTM model obtained leading classification accuracy compared to the CNN model and the multilayer perceptron (MLP) model by keeping 20%, 30%, and 40% data. Similarly, experimental results from Wang et al. (2021) [78] illustrated that the accuracy of the LSTM model for mental fatigue classification was 11.87% higher than that of the CNN model. Moreover, Rastgoo et. al. (2019) [79], Hu et. al. (2021) [80], and Nishtha et. al. (2022) [81] concluded that the LSTM model can be a promising option for the classification of stress and fatigue. This research substantiated the feasibility of the developed multimodal in-ear device as a means to monitor the occurrence of spontaneous mental fatigue among construction workers. The device would be adopted and deployed massively on construction sites due to its portability for automatic monitoring of mental fatigue among construction workers.

5.2. Research Implications

The utilization of BCI technology and biological measures has demonstrated promising applications in safety and security management fields [82]. In practice, effective safety management based on the BCI and biological measures requires a closed-loop system configured for three specific processes: sensing, processing, and actuation [82]. This research focused on mental fatigue monitoring for behavior safety management of construction sites based on the idea of the above closed-loop system. This study applied a control system framework to monitor mental fatigue, similar to the real-time data processing methods used in healthcare, such as the control strategies employed in managing insulin titration in Type 2 diabetes patients [83]. This demonstrates the applicability of such approaches across different fields.

Specifically, a method of multimodal neurophysiological signal integration and visualization was first proposed considering the characteristics of the construction environment and construction activities (i.e., the multimodal in-ear sensors). The efficiency and stability of the multimodal signal integration were analyzed. A monitoring method of spontaneous mental fatigue was proposed based on its correlation and reliability in reflecting endogenous brain activity. In this way, the research outcomes can provide new theoretical supports and practical tools for the effective management of mental fatigue among construction workers. In the future, the beneficiaries of the research will first be workers and companies. For project managers, the proposed mental fatigue monitoring method can visualize and quantify the mental statuses of workers. Individuals in special construction operations at inappropriate mental fatigue levels can be found.

In this context, the “actuation” process refers to the interventions or responses triggered by the analysis results of mental fatigue. Proactive and precise mental fatigue intervention can be provided in time. Moreover, subsequent safety training could be conducted to enhance mental fatigue management. For instance, by incorporating immersive virtual environments [84], workers can better cope with demanding construction tasks by simulating real-world scenarios, thereby improving their mental resilience and overall safety on construction sites. In this way, potential damages can be avoided. The long-term mental status data of workers on sites can be collected based on the portability of the proposed method; the data are meaningful for mental fatigue research and risk safety management in the future. By revealing the generation and development rules of construction worker’s mental load, targeted industry-wide guidance can be developed and promoted [3]. Furthermore, advancements can be made in enhancing the security and reliability of the mental fatigue monitoring system. For example, integrating blockchain technology ensures data integrity and accessibility [85], thereby establishing a more robust foundation for real-time decision-making in construction safety management. By preventing the occurrence of construction safety accidents and avoiding economic and personnel losses, sufficient and healthy human resources can be provided for the long-term stability and growth of the construction industry.

5.3. Applicability in Construction Environments

While the proposed method shows promise for monitoring mental fatigue in construction workers, scaling the proposed method to larger and more diverse populations in construction environments presents several key issues.

• Although this study was conducted in a controlled laboratory environment, the design and preliminary testing of the in-ear device has demonstrated that it maintains stability and data integrity during typical physical movements. The snug fit and elastic rubber earpieces of the device ensure that the electrodes remain securely in place, preventing displacement and minimizing artifacts caused by movements. Furthermore, the use of Bluetooth low-energy technology ensures seamless data transmission even in dynamic environments. This wireless communication protocol is specifically designed for the efficient and secure transfer of brainwave signals to connected smartphones or computers, which is crucial in real-world construction settings where physical movement and environmental conditions may vary. The robustness of the Bluetooth connection further enhances the reliability of data collection, ensuring accurate and consistent information gathering. Given these design considerations and successful results from the preliminary tests, the device is capable of accurately collecting and transmitting data in real-world construction environments.
• In this study, we employed band-pass filtering and ICA to address extrinsic artifacts such as environmental noise, and intrinsic artifacts like eye movements. It should be noted that this study was conducted in a controlled laboratory environment without motion artifacts, which were not considered in the current preprocessing framework. This controlled setting enabled us to focus on developing and evaluating the subsequent classification model. Motion artifacts are a significant factor affecting EEG signal quality in real-world construction scenarios. Future work should involve developing advanced preprocessing techniques, potentially leveraging deep learning models, and addressing motion artifacts when transitioning to more dynamic environments. This approach will enhance the robustness and applicability of EEG-based mental fatigue monitoring under practical on-site conditions.
• Moreover, the successful application of this method at a larger scale brings additional considerations. The cost of widespread adoption could be substantial, potentially limiting its feasibility. Additionally, although the monitoring method is designed for ease of use, ensuring that all workers can comfortably and effectively utilize the technology, particularly over extended periods, may require additional training and support. Integrating the proposed monitoring method with existing safety protocols could also necessitate adjustments to current practices and the development of new guidelines. Addressing these issues with cost-effective solutions, user training programs, and protocol integration strategies will be essential for maximizing the utility of the proposed monitoring method in construction environments.

5.4. Limitations and Future Work

Certain limitations were identified in this research, underscoring the need for further exploration and improvement in future endeavors. First, the machine learning and deep learning models utilized in this research were batch-learning models, which were trained and learned offline. In other words, these models must be trained on all available data at regular intervals, such as weekly, bi-weekly, monthly, quarterly, etc. For the neurophysiological data of workers arriving continually and accumulating for a long time on real construction sites, the process of repeatedly training the models from scratch can become expensive and time-consuming, posing challenges to achieving real-time monitoring of construction workers’ mental fatigue. In future research, an online learning model should be explored to meet the need for monitoring workers’ mental fatigue in real time. Second, the effectiveness and applicability of the developed multimodal in-ear sensors and the mental fatigue assessment model were studied and evaluated in a controlled laboratory environment. However, this setting does not fully replicate the complex and dynamic conditions of actual construction sites. Factors on sites such as noise, temperature, and physical exertion can play a significant role in influencing mental fatigue. Additionally, gender and experience differences can impact mental fatigue, especially in high-stress environments. This study exclusively included male participants with a background in construction engineering, which may introduce bias and limit the generalizability of the findings. Future research should focus on expanding the participant pool to include females and individuals from diverse educational and professional backgrounds, as well as conducting field experiments in real-world construction environments. It will allow for a more accurate assessment of mental fatigue across diverse conditions and enhance the external validity of the research findings.

6. Conclusions

Workers on construction sites are often in a state of high tension and high pressure and have irregular work and rest time. In these situations, mental fatigue, as a typical mental load state, is pervasive and affects an individual’s cognitive ability. Mental fatigue is one of the occupational concerns that reduces production, efficiency, and occurrence of safety accidents. In practical terms, the management of fatigue among construction workers remains a relatively underexplored domain within the realm of health and safety management on construction sites. Improving the unsafe behavior management of construction workers in the dimension of mental fatigue serves as the starting point of this study. By considering the unique characteristics of the construction environment and construction activities, a novel and non-invasive monitoring method of construction workers’ mental fatigue based on multimodal in-ear EEG sensors was proposed in this research. From a theoretical perspective, the mental fatigue neurophysiological metric system of construction workers was analyzed and optimized. From the perspective of application, the multimodal in-ear sensors were designed to realize the multimodal neurophysiological signal integration and visualization, based on the BCI technology. Accordingly, adaptable assessment models of the mental fatigue of construction workers, leveraging machine learning and deep learning techniques, were proposed and verified. All the models exhibited outstanding performance in detecting mental fatigue, with all evaluation metrics, including macroP, macroR, macroF₁, and Average Accuracy, surpassing 80% or higher. Notably, the LSTM model achieved exceptional results, with all computed metric values exceeding an impressive 89%. Combined with the multimodal in-ear device, the proposed model was proven to be effective in detecting mental fatigue. To make the research findings actionable for industry practitioners, it is recommended to integrate these technologies into their existing safety management systems. This integration would involve training personnel, establishing continuous monitoring protocols, and utilizing real-time data for proactive intervention. These applications will help bridge the gap between research findings and practical applications. In summary, the research outcomes are valuable in occupational health and construction site safety management, from a pre-control point of view.