Proposal of a Cost-Effective and Adaptive Customized Driver Inattention Detection Model Using Time Series Analysis and Computer Vision

Abstract

Advanced Driver Assistance Systems, such as Forward Collision Warning and Lane Departure Warning, play a crucial role in accident prevention by alerting drivers to potential hazards. With the advent of fully autonomous driving technology that requires no driver input, there is now a greater emphasis on monitoring the state of vehicle occupants. This is particularly important because, in emergency situations where control must suddenly be transferred to an unprepared occupant, the risk of accidents increases significantly. To mitigate this risk, new monitoring technologies are being developed to analyze driver behavior and detect states of inattention or drowsiness. In response to the emerging demands of driver monitoring technology, we have developed the Customized Driver Inattention Detection Model (CDIDM). This model employs video analysis and statistical techniques to accurately and rapidly classify information on drivers’ gazes. The CDIDM framework defines the components of inattentive or drowsy driving based on the Driver Monitoring System (DMS) safety standards set by the European New Car Assessment Programme (EuroNCAP). By defining six driving behavior-related scenarios, we have improved the accuracy of driver inattention assessment. The CDIDM estimates the driver’s gaze while simultaneously analyzing data in real-time. To minimize computational resource usage, this model incorporates a series of preprocessing steps that facilitate efficient time series data analysis, utilizing techniques such as DTW Barycenter Averaging (DBA) and K-means clustering. This results in a robust driver attention monitoring model based on time series classification.

1. Introduction

The main types of Information Technology in vehicles include Advanced Driver Assistance Systems (ADASs) such as Lane Departure Warning, Adaptive Cruise Control, and Forward Collision Warning. Additionally, along with autonomous driving technology, the importance of Driver Monitoring Systems in enhancing driver safety is increasingly acknowledged.

Autonomous driving technology is currently at SAE (Society of Automotive Engineers) Level 3, a stage of conditional autonomy that requires driver intervention in dangerous situations. This stage is still transitional and imperfect, which could cause driver inattention and lead to dangerous traffic situations. For example, in the United States, while there were 89 autonomous driving-related accidents over the three years from 2019 to 2021, this number significantly increased to 87 cases in 2022 alone [1].

Driver Monitoring Systems have become a crucial element in the automotive industry due to the advancement of autonomous driving technology. DMS plays a vital role in preventing accidents in autonomous driving by continuously assessing the driver’s state, attention, and readiness for control transfer [2]. These systems collect and analyze data from cameras, vehicle sensors, and other physiological indicators, including brain wave signals [3,4,5].

Recent developments in DMS technology focus on addressing the challenges posed by Level 3 autonomous vehicles, where the transition between automated and manual driving is critical [6]. State-of-the-art DMS can detect driver distraction, drowsiness, and cognitive load, providing real-time warnings and interventions when necessary. Recent advancements in artificial intelligence technology have significantly improved the accuracy and reliability of these systems [7,8].

In response to these trends, automotive regulatory bodies and safety organizations are incorporating various standards and regulations for managing control transfer in autonomous vehicles into their assessment protocols, as outlined below.

UN Regulation No.157—This regulation covers vehicle safety systems, fail-safe responses, human–machine interfaces, and incident detection. In particular, it specifies the requirements for driver availability recognition systems, assessing driver availability based on various criteria such as vehicle control inputs, eye blinking, and conscious head movements. If a driver fails to meet at least two of these criteria within 30 s, they are considered unavailable [3].

ISO/CD TS 5283-1—This standard provides driver monitoring guidelines based on the concept of driver availability, emphasizing safe control transfer between autonomous driving levels 2 and 3. It addresses ergonomic factors, measurable criteria, threshold setting, and the effectiveness of system interventions [9].

EuroNCAP (European New Car Assessment Programme)—EuroNCAP integrates driver monitoring systems into its safety assessment protocol, focusing on occupant state monitoring. It distinguishes between indirect monitoring, which evaluates the driver’s state through vehicle operation data, and direct monitoring, which observes the driver’s face using cameras. In particular, as shown in

Table 1. Driver state detection requirements for DMS evaluation.

Inattention Types		Distraction Scenarios	Criteria
Distraction	Long Distraction	Away from the forward road/non-driving task	When the gaze consistently fixates on a location other than the forward direction for more than three seconds
	Short Distraction (VATS)	Away from the forward road /non-driving task	When the gaze diverges for a cumulative duration of 10 s within a 30-s timeframe
		Away from road (multi-location)
Fatigue	Drowsiness		KSS (Karolinska Sleepiness Scale) level > 7
	Microsleep		When there is a blink duration of less than 3 s
	Sleep		When there is sustained eye closure lasting more than 3 s
Unresponsive Driver			When the gaze is averted from the forward direction or when the eyes are closed for more than 6 s

, it defines DMS evaluation criteria by categorizing the driver’s state into distraction, fatigue, and unresponsiveness [10].

Distraction while driving refers to anything that reduces the driver’s attention. To assess distraction, the driver’s gaze movements are classified into two types: Long Distraction, which is when the driver does not look ahead for more than 3 s, and Short Distraction, which is when the cumulative time of not looking ahead within 30 s exceeds 10 s.

Fatigue is a behavior that increases as driving time lengthens and is primarily evaluated by observing signs of drowsiness, microsleep, and sleep through changes in eye state. While EuroNCAP defines drowsiness separately from inattention, in the CDIDM, drowsiness is also classified and treated as a form of inattentive behavior. However, since gaze tracking is impossible when the eyes are closed, it was not included in the inattention scenarios and was implemented to be classified through PERCLOS. To measure drowsiness, the Stanford Sleepiness Scale, Karolinska Sleepiness Scale, and Visual Analogue Scale are generally used. EuroNCAP has adopted the Karolinska Sleepiness Scale as the standard for measuring drowsiness.

Table 2. Karolinska Sleepiness Scale (KSS).

Level	Description
1	Extremely alert
2	Very alert
3	Alert
4	Rather alert
5	Neither alert nor sleepy
6	Some signs of sleepiness
7	Sleepy, but no effort is required to stay awake
8	Sleepy, but some effort is required to stay awake
9	Very sleepy, great effort required to stay awake, fighting sleep
10	Extremely sleepy and cannot stay awake

below describes the Karolinska Sleepiness Scale, which is classified into 10 distinct levels [11].

In the field of Driver Monitoring Systems (DMS), EuroNCAP defines drowsiness as Karolinska Sleepiness Scale (KSS) level 7, and at level 8, a system warning is required. However, no criteria are provided for microsleep, characterized by eye closure lasting more than 3 s. If eye closure exceeds 6 s or the driver does not look ahead within 3 s after a warning, the system classifies the driver as unresponsive. In this paper, when the driver’s gaze could not be measured due to eye closure, we calculated PERCLOS using the eye closure ratio and classified the Karolinska Sleepiness Scale according to EuroNCAP’s DMS evaluation criteria. To comply with safety standards and effectively integrate DMS into vehicles, the following challenges must be addressed:

Technical precision: It is important to accurately monitor the driver’s eye state, which is difficult due to typical vibrations and shaking in a driving environment.

Diversity in device placement: The positions of the center console, side mirrors, and rear-view mirror vary by vehicle, which can affect the assessment of driver behavior based on their physical conditions and interactions with these devices.

Optimization and emergency response: The driver state assessment algorithm must be optimized from preprocessing to final evaluation, ensuring prompt warnings in emergencies while meeting the reliability and resource constraints of recording devices.

The methodology proposed in this paper tracks and evaluates the driver’s eye movements through real-time video analysis based on EuroNCAP’s DMS requirements. This method uses detailed eye and face direction data to detect inattention and includes the following innovations:

• Detailed eye movement tracking: Accurate measurement of gaze movement is crucial for assessing driver attention.
• Customized monitoring: This is a customized system that combines eye tracking with the empirical analysis of repeated interactions with key vehicle devices, such as the side mirrors, rear-view mirror, and center console.
• Hybrid Model: By utilizing deep learning models for video analysis to collect visual information about the driver and applying this to a machine learning-based time series classification model, we can strengthen the classification model in a device environment.
• Performance verification: The evaluation algorithm follows EuroNCAP’s DMS criteria and is tested under actual driving conditions, considering various practical variables and providing real performance results.

This paper first reviews DMS-related research, then introduces realistic scenarios for DMS, outlines some problem-solving methods, and concludes by presenting our results.

2. Related Work

There are two methods for monitoring a driver’s state: indirect monitoring and direct monitoring. Indirect monitoring is a method of estimating the driver’s state through vehicle sensor data such as steering wheel operation, speed, braking, and lane position. This method has the advantage of not disturbing the driver while monitoring them, but it also has the critical disadvantage of not being able to determine the driver’s state during autonomous driving. Direct monitoring methods include assessing the driver’s state by analyzing brain activity through the collection of Electroencephalogram (EEG) signals and judging the driver’s state through an analysis of image data collected via a camera.

Regarding EEG signal analysis-based driver monitoring technology, Abdulhamit Subasi [12] proposed a method of analyzing EEG signals through wavelet transform and then classifying arousal levels using ANN. Jiao, Yingying, et al. [13] achieved 98% accuracy in detecting driver drowsiness using GAN and LSTM neural networks. This method of analyzing EEG signals has the advantage of accurately detecting the driver’s state, but the driver has to wear a device to collect brain wave signals. To overcome this disadvantage, studies have been conducted on the collection of EEG signals using wireless or wearable devices [14,15]. Lin, Chin-Teng, et al. [14] devised a mobile wireless brain wave system based on dry brain-wave sensors that can predict the driver’s state in real-time, and Li, Gang, et al. [8] proposed a fully wearable EEG system that detects driver drowsiness using a Bluetooth-enabled EEG headband and a commercial smartwatch. However, its accuracy is somewhat low, between 84 and 91%, and the issue of driver discomfort is still not resolved as they still need to wear a headband.

Recently, with the technological advancement of artificial intelligence, image analysis solutions have become increasingly abundant. Driver video analysis can now be conducted with a certain level of accuracy while minimizing discomfort compared to the EEG signal analysis method, and research on image analysis-based driver state monitoring is being actively conducted.

The driver monitoring system using direct monitoring to analyze driver video is largely divided into determining driver inattention and drowsiness.

Driver inattention detection is mainly based on facial expressions and gaze tracking. Tango, Fabio, and Marco Botta [16] compared and verified the performance of various ML models in Real-Time Driver Distraction Detection (DDD) and confirmed that SVM outperforms all other ML methods. Liang et al. [17] detected drivers’ cognitive distraction using Bayesian networks and verified that dynamic models considering the drivers’ time dependency perform better than static models. However, these studies have limitations in that they do not reflect various environmental conditions, such as camera angles and the driver’s physical condition and behavior patterns.

Drowsiness detection judges drowsiness based on eye blinking. The indicator used here is PERCLOS, which represents the ratio of eye closure. Hu Shuyan et al. [18] predicted drowsiness using SVM (Support Vector Machine) from eyelid-related parameters extracted from EOG (Electrooculography) data collected in the EU Project SENSATION’s driving simulator. Dasgupta et al. [19] constructed a real-time driver drowsiness detection system reflecting various conditions of actual driving situations, such as lighting, facial rotation, and vehicle vibration. They confirmed a processing speed of 9.5 fps and drowsiness detection accuracy of over 97% in actual driving environments, proving that it is possible to achieve an accuracy level close to that of EEG signal analysis-based DDS, even with the driver video analysis method. However, recent driver monitoring system evaluation criteria also require the classification of driver inattention status.

Jo Jaeik et al. [20] proposed a method to minimize computational costs and eye detection errors while enabling the detection of both drowsiness and inattention by designing a system to detect drowsiness when the driver is looking ahead and distraction when they are not looking ahead. However, classifying drowsiness and driver inattention status based on whether they are looking ahead has limitations in terms of reliability. Grace, Richard, et al. [21] presented a method of estimating driver status by combining PERCLOS, which is the ratio of eyelid closure, with driver performance-based data such as steering, lane, acceleration, and video-based data analysis results; however, driving data cannot be used for vehicle control transfer as they cannot determine the driver’s status during autonomous driving.

Recent studies [22,23,24,25] have attempted to detect driver inattention using deep learning models, but they work by learning and detecting images of inattentive behavior itself, such as mobile phone operation or vehicle button operation. This method has the limitation of not being able to detect undefined, inattentive behavior patterns.

In this paper, we propose a driver-customized monitoring system that is robust to various variables, such as the physical condition of the vehicle and driver, while following EuroNCAP’s DMS evaluation criteria. Specifically, we extract one-dimensional gaze vector information through real-time preprocessing from the driver’s video information and utilize DTW Barycenter Averaging (DBA) and K-means clustering techniques to classify both the gaze vector information and sequence information of each frame. Finally, to classify inattentive behavior, we define driving-related behavior scenarios and realistically improve the accuracy of driver inattention detection by excluding the deviation of the driver’s gaze from the road ahead due to driving-related behavior from the inattention judgment criteria.

3. Proposed System

3.1. Workflow

Figure 1 provides an overview of the CDIDM used to determine driver inattention or drowsiness, explaining the analysis process in four stages.

In the first stage, key edge points are detected based on the center of the screen, and these points are used to stabilize the image. Subsequently, using the facial area extracted by the YOLO model, the face angle is estimated, and the ROI is extracted. The face rotation angle is also calculated in this process.

The second stage involves face landmark detection. This identifies the coordinates of the eyelids and pupils. From the pupil coordinates, the gaze direction is estimated, and vector values for eye movement are extracted. Additionally, the PERCLOS calculation is performed at this stage.

In the third stage, dimensionality reduction is carried out. The vector values extracted earlier undergo a preprocessing stage that facilitates time series classification. During this process, common pattern movement data are recalculated for improved learning. Moving averaging techniques are also applied to smooth the data.

The final stage determines the driver’s inattention and drowsiness state. The driver’s inattention is ultimately determined by evaluating its similarity with the classified common pattern movement data. Drowsiness is assessed by calculating PERCLOS using the facial landmark values used for inattention detection and classifying it according to the criteria defined by EuroNCAP.

The key point is that this system continuously evolves. As data on the common pattern movements of the driver’s gaze are reinforced through the accumulation of repeated driving data, a driver-customized monitoring model is established. This adaptive approach allows for increasingly accurate and personalized detection of the driver’s state over time.

Step 1 is the stage in which source data are secured for the extraction of gaze information, where the direction of the face is calculated based on the extracted key facial landmark values. The direction of the face is determined by calculating the rotation matrix based on a total of six values: the left/right eye corners, lip ends, chin, and nose, as shown in Figure 2b [26]. The process for calculating the rotation matrix is as follows: First, a rotation vector is generated using the solvePnP function of OpenCV-python v4.9.0, and this is passed to the Rodrigues function to calculate the rotation matrix. Finally, the rotation matrix is decomposed through the RQDecomp3x3 function to calculate the Euler angles, as shown in Figure 2a, thereby determining the direction of the face [27].

Step 2 involves extracting gaze information based on pupil position data along with the facial direction data calculated in Step 1 and calculating PERCLOS to determine drowsiness.

Gaze information extraction begins with detecting the facial region (Face ROI) from the input image using a YOLO model and identifying key landmark points. Next, a 3D face model is fitted based on the detected 2D facial landmarks, taking into account camera parameters to estimate the face’s position and orientation in 3D space. Through this 3D face model, the head’s 3D rotation (r) and the eyes’ 3D positions (t) are calculated. This extracted 3D information then undergoes a normalization process to ensure consistent results across various input conditions [28]. Based on the normalized data, the left eye angle vector (le), right eye angle vector (re), and head angle vector (h) are each derived. These calculated eye angle vectors (le, re) and head angle vectors (h) are then combined with consideration of their respective weights, reflecting the importance of each element in this process. Finally, as a result of this vector summation, a gaze vector (g) is generated, representing the driver’s gaze direction in 3D space. Figure 3 visually explains this process.

The determination of driver drowsiness is achieved by calculating PERCLOS using the eye coordinate values extracted in Step 1. PERCLOS is a representative indicator that can measure drowsiness, defined as the ratio of time when the eyes are more than 80% closed over a period of 3 min [29]. The equation for calculating the Eye Aspect Ratio (EAR) is as follows, and the key points used are the 12 coordinate values for the left (LP1~6) and right (RP1~6) eyes, shown as red dots in Figure 4. In the figure, yellow arrows indicate the vertical distances between the upper and lower eyelids, while blue arrows represent the horizontal distance between the outer and inner corners of each eye. These vertical and horizontal distances are used to calculate the EAR through the formula below.

$$EAR=\left\{{\displaystyle \frac{\left|\left|RP2-RP6\right|\right|+\left|\left|RP3-RP5\right|\right|}{2\left|\left|RP4-RP1\right|\right|}}\right\}$$

$$PERCLOS={\displaystyle \frac{Cumulative time of eye closure}{Measurement time}}$$

Finally, if the driver’s PERCLOS value is 0.15 or higher, it is determined to be a drowsiness state of level 7 or higher based on the Karolinska Sleepiness Scale [30]. Additionally, according to EuroNCAP’s DMS criteria, if a state where the EAR is 0.8 or higher continues for 3 s or more, it is classified as a sleep state.

3.2. Data Collection

To determine whether the driver’s behavior is related to driving based on visual information, we defined six scenarios and collected data accordingly. As depicted in Figure 5, we defined behavior related to driving as cases where the gaze deviates from the front due to checking the rear-view mirror or console or actions performed for vehicle operation.

In Figure 5, S1 through S6 represent scenarios where the driver is looking at the rear-view mirror, instrument cluster, center console, steering wheel, left side mirror, and right side mirror, respectively.

These scenarios represent common situations where a driver’s attention may shift from the road ahead for legitimate driving-related reasons. The inset image shows the camera setup used to capture the driver’s behavior. The aim of this classification was to ensure the reliability of the driver monitoring system by reducing unnecessary driver warnings through the recognition of driving-related behaviors that involve not looking ahead. By understanding these scenarios, the system can differentiate between necessary driving actions and potentially dangerous distractions, thus improving the accuracy and usefulness of driver alerts.

As shown in

Table 3. Face tracking dataset for gaze estimation.

Automobile Information				Driver Information
Type	Segment	Manufacturer	Model	Gender	Age	Height (cm)	Weight (kg)
SUV	D	KIA	SORRENTO	Male	43	174	68
SUV	C	KIA	Niro	Male	37	171	65
Truck	1 ton	KIA	Bongo3	Male	44	182	82
Sedan	C	Hyundai	Elantra	Female	30	162	53
Sedan	C	KIA	K3	Male	44	168	62
SUV	J	Hyundai	Tucson	Male	26	176	84
Sedan	F	BMW	720 d	Male	52	177	80
SUV	J	Hyundai	Tucson	Female	40	160	58
Van	M	KIA	Carnival	Male	41	175	83
Compact	A	KIA	Morning	Female	38	156	61

, data were collected by filming actual driving videos in four types of vehicles, including a full-size SUV, a compact SUV, a truck, and a sedan, with ten drivers. The device used for video recording was a Raspberry Pi 4 Model B product connected to a NOIR camera module, collecting video at a resolution of 640 × 480 p. From the collected videos, landmark data for key facial positions were extracted and secured for each vehicle type and driver.

Table 4. Facial Euler angles and iris position information for gaze estimation.

Seq	Pitch	Yaw	Roll	Iris_Lx	Iris_Ly	Iris_Rx	Iris_Ry
1	0.0761	−0.0562	−0.0248	575.2844	327.6482	654.3644	336.1082
2	0.0788	−0.0484	−0.0267	583.3813	324.1810	660.4034	333.9849
3	0.0769	−0.0465	−0.0343	589.4077	321.4966	664.8054	332.6090
4	0.0798	−0.0394	−0.0304	593.2281	319.6493	667.5866	331.8580
5	0.0604	−0.0697	−0.0244	595.3186	318.7085	669.1718	331.6109
6	0.0540	−0.0439	−0.0273	596.2318	318.4063	669.9681	331.7222
7	0.0722	−0.0409	−0.0347	596.6299	318.4297	670.4521	331.8593
8	0.0616	−0.0389	−0.0263	596.8526	318.5858	670.6852	331.9431
9	0.0614	−0.0466	−0.0285	596.7946	318.7040	670.8551	331.9021
10	0.0652	−0.0356	−0.0300	596.7021	318.6856	670.9465	331.7718

,

Table 5. Eye tracking and gaze estimation coordinate value data.

Seq	Gaze_Lx	Gaze_Ly	Gaze_Rx	Gaze_Ry
1	575.28	327.65	654.36	336.11
2	583.38	324.18	660.40	333.98
3	589.41	321.50	664.81	332.61
4	593.23	319.65	667.59	331.86
5	595.32	318.71	669.17	331.61
6	596.23	318.41	669.97	331.72
7	596.63	318.43	670.45	331.86
8	596.85	318.59	670.69	331.94
9	596.79	318.70	670.86	331.90
10	596.70	318.69	670.95	331.77

and

Table 6. Nummerical Encoding of Gaze Data Using Projected Vectors on a Pixel-Based Grid System.

Seq	Vector_Lx	Vector_Ly	Vector_Rx	Vector_Ry	SUM_x	SUM_y	GridValue	NORM
1	49	1	66	−21	115	−20	226,935	0.812
2	65	−8	82	−35	147	−43	209,766	0.738
3	76	−14	82	−45	158	−59	197,822	0.686
4	79	−22	80	−53	159	−75	185,878	0.634
5	82	−28	78	−56	160	−84	179,160	0.605
6	82	−30	82	−55	164	−85	178,413	0.602
7	83	−30	85	−53	168	−83	179,906	0.608
8	83	−26	93	−49	176	−75	185,878	0.634
9	84	−22	97	−43	181	−65	193,343	0.667
1	82	−21	102	−45	184	−66	192,597	0.663

below show the datasets used to estimate the drivers’ gaze. First, following Step 1 in Figure 1, we obtained facial direction data, including the Euler angles (pitch, yaw, and roll).

Using the eyelid landmark values used for PERCLOS calculation in Figure 4 and the pupil coordinate values from Table 4, we calculated the ratio indicating how far the pupil was positioned from the center of the eye in the up/down and left/right directions. The results are listed as the gaze values for the left and right pupils in Table 5. Subsequently, by applying an arctangent to the ratio values of pupil position from the eye center, we obtained the angle of gaze direction for both the left and right pupils. Then, using the Euclidean distance between the eye center and the pupil, we estimated the final coordinate values ($G_x,G_y)$ where the gaze was directed.

$$G_x{=I}_x+D•A•\cos \left(\mathrm{atan}\left(C_y,C_x\right)\right)•H_y+H_p$$

$$G_y=I_y+D•A•\sin \left(\mathrm{atan}\left(C_y,C_x\right)\right)•H_p+H_y$$

• I represents the specific coordinate values within the eye. It provides precise positions relative to the x and y coordinate axes.
• D is the measured distance from the geometric center of the eye to the pupil, indicating how far the eyeball is from the center of the eye.
• A is a constant representing the physical distance from the camera to the subject’s face. It is used to scale measurements within the image.
• C represents the length of one side of the Region of Interest (ROI) bounding box surrounding the face. It is important for normalizing eye positions to a standard size across various images and scenarios.
• H represents the angle of the head, where Hy denotes the cosine value of the yaw angle and Hp denotes the cosine value of the pitch angle.

Table 5 shows the data obtained from Scenario 1. These data display the relative position ratios of the eye to the eye center and the gaze values calculated with this formula. These ratios and gaze values help us to understand the direction and intensity of the gaze in the given scenario.

The magnitudes of the gaze vector values for the left and right eyes listed in Table 6 were reduced to a single gaze vector value through vector summation operations. The resulting values are represented as SUM_x and SUM_y in Table 6. For each gaze vector value, a Cartesian coordinate system in pixel units was applied, and a single value was derived by numerating from the top left corner. This grid number value is listed as GridValue in Table 6, and its normalized form is indicated as NORM.

Figure 6 visually represents the single gaze vector values from Table 6 on the x and y axes, illustrating the characteristics of the collected driver gaze data. In the left image, the red arrows indicate the gaze vector (dx, dy). These arrows represent the direction and magnitude of the driver’s gaze. The right image shows the gaze movements for six scenarios (S1 through S6) depicted as a color-coded scatter plot. Each point in the scatter plot corresponds to a gaze vector, with its position indicating the horizontal (x) and vertical (y) components of the driver’s gaze direction.

The image presented in Figure 7 illustrates the distribution of gaze vector components (dx, dy) for six distinct driver behavior scenarios, each represented by a different color. This visualization provides compelling evidence for the differentiation of driver attention states based on eye movement patterns. The scenarios are labeled as S1 through S6, where S1 represents the driver looking at the rear-view mirror, S2 indicates the driver focusing on the instrument cluster, S3 shows the driver’s gaze directed at the center console, S4 depicts the driver looking at the steering wheel, S5 signifies the driver observing the left side mirror, and S6 represents the driver checking the right side mirror. The right panel of the figure highlights these distinct clusters, demonstrating how each scenario results in a unique pattern of eye movements and gaze directions.

As observed in the scatter plot, each scenario (S1 through S6) occupies a relatively distinct region in the two-dimensional space defined by the horizontal (dx) and vertical (dy) components of gaze vectors. The left panel shows the raw data points, while the right panel overlays approximate boundaries for each scenario cluster, enhancing the visual separation between behaviors.

The clear clustering of data points corresponding to each scenario demonstrates that driver gaze patterns exhibit characteristic signatures depending on the specific driving-related task or attention state. For instance, scenario S3 (shown in green) appears to involve more upward and rightward gaze movements, possibly corresponding to checking the gear shift or center console. In contrast, scenarios S5 and S6 show concentrations of gaze vectors on the left and right sides of the central gaze point, respectively. This distribution likely indicates behaviors such as checking side mirrors or looking at the vehicle’s periphery, suggesting the driver’s attention to the lateral surroundings of the vehicle.

Importantly, while there is some overlap between clusters, particularly in central regions where multiple behaviors may share common gaze directions, the overall separation is substantial. This separation suggests that these gaze vector distributions can serve as a robust basis for classifying driver behaviors and attention states.

The distinguishability of these clusters supports the hypothesis that when a driver’s gaze deviates from the straight-ahead position, the nature of this deviation can be used to infer whether the eye movement is related to a specific driving task or potentially indicative of distraction or inattention.

Figure 8 presents a comprehensive visualization of gaze vector distributions for six distinct driving-related behavior scenarios, each representing data from ten different drivers. This multifaceted representation offers valuable insights into the consistency and variability of gaze patterns across different individuals and driving conditions.

The scatter plots for each scenario demonstrate characteristic clustering patterns, corroborating the findings observed in Figure 7. This consistency across drivers and vehicles reinforces the robustness of scenario-specific gaze behaviors.

Importantly, while each scenario exhibits a characteristic overall pattern, there is noticeable variability between drivers, as indicated by the color-coded data points. This inter-driver variability likely stems from differences in physical characteristics, driving habits, and potentially the specific vehicles used in the study.

The consistency of overall patterns across drivers, despite individual variations, strongly suggests that these scenario-specific gaze behaviors are generalizable. This finding has significant implications for the development of robust driver monitoring systems. It indicates that a generalized model based on these scenario-specific patterns could provide a solid foundation for detecting and classifying driver behaviors, while personalized data learning could further enhance accuracy.

Moreover, the observed patterns typically show a deviation from the forward position followed by a return, which aligns with expected driving behaviors where drivers periodically check various areas of the vehicle or surroundings before returning their gaze to the road ahead.

In conclusion, Figure 8 provides compelling evidence that driver gaze patterns, while subject to individual variability, exhibit consistent scenario-specific characteristics. This supports the feasibility of developing sophisticated driver monitoring systems capable of accurately classifying various attention states across a diverse driver population. This study also suggests that accumulating more data over time for individual drivers could further refine these classifications, potentially leading to highly accurate, personalized driver attention monitoring systems.

Figure 9 provides a comprehensive visualization of the correlation distribution between dx and dy coordinate values, representing gaze movement vectors for four drivers across six driving behavior scenarios. The figure consists of graphs showing dx and dy scatter plots for each driver, along with individual kernel density estimation (KDE) curves for dx and dy. These KDE curves, displayed on the sides of each scatter plot, illustrate the distinct density patterns of gaze movements along the x and y axes respectively. The varying shapes and peaks of these curves reveal unique concentration patterns of gaze for each scenario and driver. By analyzing these distinct patterns, it becomes possible to characterize and differentiate the gaze behaviors associated with each driving scenario. Each driver is distinguished by a unique color in both the scatter plots and the corresponding KDE curves, allowing for easy comparison across individuals and scenarios.

Key observations from this figure include:

• Inter-driver variability: Within each scenario, the scatter plots reveal distinct clustering patterns for each driver, indicating persistent individual differences in gaze behavior even under similar driving conditions. This is further supported by the varied shapes and peaks of the KDE curves for each driver.
• Scenario-specific patterns: The overall distribution of gaze vectors shows marked differences across scenarios, suggesting that different driving tasks elicit distinct gaze behaviors. For instance, Scenario 3 exhibits a more diffuse pattern compared to the tighter clustering in Scenario 1.
• Bimodal distributions: In several scenarios (e.g., S3, S5), the KDE curves for dy exhibit bimodal distributions for some drivers. This could indicate alternating attention between two vertical points of interest, such as the road ahead and the dashboard.
• Density hotspots: Areas of high point density in the scatter plots, often corresponding to peaks in the KDE curves, represent frequently occurring gaze positions. These hotspots could be indicative of key areas of visual attention for specific driving tasks.

The observed patterns and variations across drivers and scenarios underscore the complexity of driver gaze behavior and the potential for data-driven approaches in developing sophisticated driver monitoring systems. The consistent presence of identifiable patterns within scenarios, despite individual variations, suggests that scenario-specific models could form a robust foundation for gaze behavior classification. Furthermore, the clear inter-driver differences highlight the potential value of personalized calibration in such systems to enhance accuracy and reliability.

To classify eye movement vectors while preserving sequence information, these vectors must be converted into one-dimensional time series data. For this purpose, as described in Table 6, we constructed one-dimensional time series data using the synthesis results of gaze vector values gridded and enumerated in pixel units within a Cartesian coordinate system. These data were preprocessed through normalization.

Table 7. Variance ratios of Principal Component Analysis results.

	Eigen Vector	Explained Variance	Explained Variance Ratio
S1	[−0.97133103, −0.23773099]	736.9242905	0.93258419
S2	[0.75592334, −0.65466014]	1067.50632338	0.92779933
S3	[−0.55115528, 0.8344027 ]	1954.73997671	0.94382174
S4	[0.21681727, −0.97621221]	587.61989058	0.91274915
S5	[0.98644338, 0.16410198]	883.008934	0.83827288
S6	[0.97293209, −0.23109122]	1431.98554755	0.85216792

shows that the conversion to one-dimensional time series data effectively represents the original data without significant loss. The explained variance ratio ranges from a minimum of 0.83 to a maximum of 0.94, demonstrating that the integrity of the one-dimensional time series data can be maintained with minimal data loss.

3.3. Data Preprocessing

Data converted into a one-dimensional time series undergo preprocessing to enhance accuracy and speed. This process aims to emphasize inherent patterns within the data, making analysis easier. The preprocessing sequence consists of several key steps. Figure 10 visually illustrates how the data changes through this process.

First, Figure 10a displays seven different gaze information sets, each represented by a distinct color, which represent data collected from the same driving scenario repeated multiple times.

Next, change extraction is performed. This step calculates the change from one data point to the next within the sequence. Figure 10b illustrates this process, showing how it emphasizes significant changes in the data, which is crucial for pattern recognition. The graph demonstrates the transformation of subtle variations in the original gaze vector dataset Figure 10a into more pronounced spikes, representing changes.

Next, a moving average is applied to mitigate data fluctuations, reduce noise, and reveal underlying trends. This is followed by standardization, which adjusts the mean of the data to 0 and its standard deviation to 1. Figure 10c shows the result of these processes, where the data now display clear patterns with normalized amplitudes, facilitating easier comparison across different sequences.

The final preprocessing step involves extracting common patterns using techniques such as Dynamic Behavior Analysis (DBA). Figure 10d presents a derived common pattern that represents characteristic behavior shared across multiple sequences. This pattern serves as a template for classification and further analysis.

Upon completion of these preprocessing steps, the refined data are re-circulated into the learning process. This recirculation strengthens the classification model, enabling it to recognize and respond more adeptly to similar patterns in future datasets. Through this iterative process, the reinforced pattern data are expanded and undergo the same series of preprocessing steps, ensuring they maintain the integrity and clarity necessary for accurate classification in each new iteration.

Figure 10 provides a visual summary of this preprocessing workflow, illustrating the transformation from raw gaze vector data to refined, analyzable patterns. This systematic approach enhances the model’s ability to detect and classify complex gaze behaviors, potentially improving the accuracy and reliability of driver monitoring systems.

3.4. Driver Inattention Estimation

Time Series Classification (TSC) has been recognized as one of the most challenging problems in data mining [26]. With the advancement of IoT technology, the explosive increase in data generation has led to a corresponding increase in demand for analysis solutions. As a result, research and development in the TSC field have become active, and hundreds of algorithms have been proposed to date [31,32].

Conventional machine learning (ML) algorithms rely on processing on cloud or centralized servers [33]. However, this approach is not appropriate in cases where data processing speed is crucial, such as in driver monitoring systems in the automotive industry. To address this issue, we propose a driver monitoring system that uses on-device learning technology [4]. This system incorporates and optimizes the driver’s behavior patterns directly into the ML model, including the DBA (DTW Barycenter Averaging) and K-means clustering models proposed by Petitjean and François [5].

The CDIDM finalizes time series data on the rate of change in gaze vectors. When these data are classified using the K-means clustering model, they are effectively divided into three distinct categories. Figure 11 shows the results of classifying gaze data through the CDIDM. The data on the right show a total of seven sets of gaze data to be classified, consisting of two sets of gaze data from scenario 5, two sets of forward gaze data, and three sets of other gaze movement data. The graph on the left shows the results of the seven analyzed data classified into three time series consisting of driving-related gaze data, forward gaze data, and driver inattention data.

Figure 12 shows the results of classifying the drivers’ gaze data into six predefined driving-related gaze movements based on time series analysis techniques. Figure 12 shows the results of classifying the drivers’ gaze data into six predefined driving-related gaze movements based on time series analysis techniques, categorized into six scenarios. The test was conducted by selecting and labeling videos corresponding to the scenarios from footage of ten drivers operating eight types of vehicles, extracting gaze information from the videos, and dividing it to build a total of 12,648 Test DataSet items. Based on this, the classification accuracy was derived, resulting in a final ROC Score of 0.92, achieving satisfactory classification results.

The AUC values for scenarios such as checking the vehicle infotainment system (S2), operating the center console (S3), and checking the instrument panel (S4) are 0.92, 0.88, and 0.88, respectively, showing relatively lower accuracy compared to the other scenarios. This lower accuracy is attributed to the complexity of the classification process due to the proximity of the related locations. Furthermore, there are still limitations in accurately recognizing gaze positions due to the low resolution of 480 p. However, when the gaze information is dichotomized into non-driving-related and driving-focused categories, we confirm that the accuracy improves to 97%.

Table 8. Inattention Classification Results by Scenario and Driver.

Driver	Amount	S1	S2	S3	S4	S5	S6	AVG
Driver1	2610	99	98	94	95	97	99	97
Drvier2	1914	97	95	93	94	96	95	95
Driver3	870	96	92	90	91	93	90	92
Driver4	1130	97	93	91	92	94	94	93
Driver5	1031	93	97	95	96	95	97	95
Driver6	1016	94	96	98	97	95	96	95
Driver7	965	93	94	96	95	93	92	93
Driver8	1185	96	95	97	98	96	98	95
Driver9	904	92	94	93	95	92	91	93
Driver10	1023	93	96	94	95	94	93	95
AVG		98	94	92	92	95	95	94

shows the classification accuracy between gaze data for each scenario and inattention data for each driver. The results demonstrate that as the amount of data increases, the classification accuracy for each scenario improves. Furthermore, it can be observed that the classification accuracy between scenarios and inattention data has improved more significantly than the classification between scenarios, achieving up to 97% accuracy.

Table 9. Comparison of accuracy in driver negligence classification models.

Remarks	Accuracy
Customized Gaze Estimation Model (CDIDM)	97%
EEG signal analysis [3]	98%
EEG headband and smartwatch [10]	84%
Gaze estimation (PERCLOS) [15]	98%
Driving performance and eye movement [34]	93%
Steering wheel sensor and speed [35]	98%

presents a comparison of accuracy among driver inattention classification models. Our Customized Gaze Estimation Model (CDIDM) achieves 97% accuracy, which is slightly lower than some other models that reach 98%. However, it’s important to note that unlike other models, CDIDM can classify not only driver drowsiness but also inattentiveness. Moreover, considering our model’s potential for enhancement through data accumulation, we believe it remains competitive in the field. While the table shows that EEG signal analysis [3] and gaze estimation (PERCLOS) [15] methods achieve 98% accuracy, our model offers a more comprehensive approach to driver monitoring. Furthermore, we anticipate that CDIDM’s classification accuracy will improve even further as hardware performance advances, enabling real-time analysis of high-resolution video images. This potential for enhancement underscores the scalability and future-proof nature of our proposed model.

4. Results and Discussion

To demonstrate the effectiveness of CDIDM’s continuous learning capability, we conducted a longitudinal study using a subset of participant data. Figure 13 illustrates the improvement in accuracy as the system accumulates data from individual drivers.

As shown in the figure, the system initially achieved an average accuracy of 92% in detecting driver inattention with only 100 gaze datasets per participant. We then incrementally added 100 datasets at each stage and evaluated their accuracy. The results show that when the accumulated data reached 1000 sets, the system achieved 95% accuracy. Ultimately, as evidenced by Driver1′s classification results in Table 8, we confirmed that an accuracy of 97% could be attained with over 2000 datasets. This demonstrates significant performance improvement as more data are collected and the system continues to learn.

This trend clearly illustrates the system’s ability to adapt and enhance its performance over time, effectively addressing the initial limitation of a small participant pool. The personalized nature of this learning process ensures that the system becomes increasingly robust to each individual driver’s behaviors and patterns, resulting in more accurate and reliable inattention detection. These data not only validate the effectiveness of our continuous learning approach but also highlight the potential for even further improvements with extended use.

5. Conclusions

In this paper, we validated the performance of the Customized Driver Inattention Detection Model (CDIDM) using a Raspberry Pi 4 Model B equipped with a Broadcom BCM2711 chipset and 8 GB LPDDR4-3200 SDRAM, integrated with an intelligent camera module featuring Intel’s Myriad X Vision Processing Unit (VPU) with 4 TOPS processing capability. This hardware configuration achieved processing speeds exceeding 20 fps at 640 x480p resolution, confirming the feasibility of real-time driver monitoring services.

CDIDM utilizes the YOLO v8 model, optimized for edge devices, to perform efficient facial tracking. When deployed on the Myriad X VPU, our system demonstrated robust performance across various driving scenarios, maintaining real-time processing capabilities essential for prompt driver inattention detection and warnings.

Although the current study was conducted with a limited number of participants, it shows promising potential for real-world application by confirming accuracy improvements for individuals through the accumulation of empirical data. The combination of affordable hardware (Raspberry Pi 4) with specialized AI accelerators (Myriad X VPU) presents a cost-effective yet powerful solution for advanced driver monitoring systems. This approach bridges the gap between high-cost specialized equipment and the need for accessible real-time driver safety technologies.

The CDIDM proposed in our research offers a practical and scalable solution for real-time driver monitoring and warning provision. Its ability to perform complex visual analysis tasks while maintaining high frame rates demonstrates its potential for integration into actual vehicular systems. Future research will focus on three key areas: First, expand the participant pool and test the system under diverse real-world driving conditions to enhance reliability and generalizability. Second, explore the system’s adaptability to various vehicle types and embedded automotive platforms to ensure broad applicability. Lastly, we plan to develop a next-generation integrated driver monitoring system by incorporating vehicle internal state information via on-board diagnostics (OBD) and real-time road condition data using V2X communication protocols. This comprehensive approach will enable CDIDM to consider driver state, vehicle performance, and environmental factors holistically. By integrating these multifaceted data sources, we anticipate significantly improving CDIDM’s accuracy, reliability, and practicality, further advancing its potential as a crucial component in advanced driver assistance systems and autonomous driving technologies.

In conclusion, our research contributes to the field of automotive safety by presenting an innovative and cost-effective approach to driver inattention detection. The performance of CDIDM and its potential for widespread deployment position it as a promising technology that we anticipate will enhance road safety and support the development of more sophisticated driver assistance systems.