Author Contributions
Conceptualization, R.H. and Y.X. (Yong Xu); methodology, R.H., Y.X. (Yong Xu) and C.-Y.H.; software, Y.X. (Yunjie Xiang); validation, Y.X. (Yunjie Xiang); formal analysis, R.H. and Y.X. (Yong Xu); investigation, Y.X. (Yunjie Xiang), C.-Y.H. and C.D.; resources, Y.X. (Yong Xu); data curation, C.-Y.H. and C.D.; writing—original draft preparation, Y.X. (Yunjie Xiang); writing—review and editing, R.H. and Y.X. (Yong Xu); visualization, R.H. and Y.X. (Yong Xu); supervision, C.-Y.H.; project administration, R.H. and Y.X. (Yong Xu); funding acquisition, R.H. All authors have read and agreed to the published version of the manuscript.
Figure 1.
Fatigue-driving detection methods.
Figure 1.
Fatigue-driving detection methods.
Figure 2.
The framework of the fatigue-driving detection system.
Figure 2.
The framework of the fatigue-driving detection system.
Figure 3.
The network structure of YOLOv4-tiny.
Figure 3.
The network structure of YOLOv4-tiny.
Figure 4.
The network structure of IDLN.
Figure 4.
The network structure of IDLN.
Figure 5.
Variation of eye state degree with video frames when the driver is in different states and different scenes. (a,c,e,g) are drivers in the normal state, and (b,d,f,h) are drivers in the fatigue state. (a,b) indicate that the driver is in the bare face scene, (c,d) indicate that the driver is in the mask-covered scene, (e,f) indicate that the driver is in the glasses wearing scene, and (g,h) indicate that the driver is in the scene with mask and glasses cover.
Figure 5.
Variation of eye state degree with video frames when the driver is in different states and different scenes. (a,c,e,g) are drivers in the normal state, and (b,d,f,h) are drivers in the fatigue state. (a,b) indicate that the driver is in the bare face scene, (c,d) indicate that the driver is in the mask-covered scene, (e,f) indicate that the driver is in the glasses wearing scene, and (g,h) indicate that the driver is in the scene with mask and glasses cover.
Figure 6.
Variation of Gaussian weighting scores with eye state degree for different Gaussian weighting factors.
Figure 6.
Variation of Gaussian weighting scores with eye state degree for different Gaussian weighting factors.
Figure 7.
Internal scene of the data collection vehicle.
Figure 7.
Internal scene of the data collection vehicle.
Figure 8.
Some of the EFSD samples.
Figure 8.
Some of the EFSD samples.
Figure 9.
The curve of loss value of the YOLOv4-tiny model.
Figure 9.
The curve of loss value of the YOLOv4-tiny model.
Figure 10.
Detection results of the YOLOv4-tiny model on EIMDSD.
Figure 10.
Detection results of the YOLOv4-tiny model on EIMDSD.
Figure 11.
Comparison of Gaussian weighting scores for normal and fatigue states with different Gaussian weight factors. (a–j) denote the Gaussian weighting scores at 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, and 1.0, respectively.
Figure 11.
Comparison of Gaussian weighting scores for normal and fatigue states with different Gaussian weight factors. (a–j) denote the Gaussian weighting scores at 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, and 1.0, respectively.
Figure 12.
Selected infrared face images from the CASIA-Iris-Distance infrared face image dataset.
Figure 12.
Selected infrared face images from the CASIA-Iris-Distance infrared face image dataset.
Figure 13.
Eye detection model results on CASIA-Iris-Distance infrared face image dataset.
Figure 13.
Eye detection model results on CASIA-Iris-Distance infrared face image dataset.
Figure 14.
Sample data of some infrared eye images.
Figure 14.
Sample data of some infrared eye images.
Figure 15.
Results of eye semantic segmentation of infrared images.
Figure 15.
Results of eye semantic segmentation of infrared images.
Table 1.
Segmentation results of IDLN in EIMDSD.
Table 1.
Segmentation results of IDLN in EIMDSD.
Model | Images | mIoU | mPA | FPS |
---|
ANN | 600 | 0.863 | 0.957 | 15.15 |
BiSeNetv2 | 600 | 0.840 | 0.950 | 14.93 |
DANet | 600 | 0.818 | 0.944 | 19.61 |
DeepLabv3 | 600 | 0.866 | 0.959 | 18.52 |
DeepLabv3+ | 600 | 0.868 | 0.961 | 28.50 |
Fast-SCNN | 600 | 0.859 | 0.955 | 16.95 |
FCN | 600 | 0.858 | 0.955 | 5.85 |
ISANet | 600 | 0.858 | 0.955 | 16.13 |
OCRNet | 600 | 0.862 | 0.957 | 6.25 |
PSPNet | 600 | 0.802 | 0.939 | 14.29 |
UNet | 600 | 0.866 | 0.957 | 27.78 |
Ours | 600 | 0.883 | 0.962 | 34.10 |
Table 2.
Weighting scores corresponding to different Gaussian weighting factors when the driver is in a normal state.
Table 2.
Weighting scores corresponding to different Gaussian weighting factors when the driver is in a normal state.
Video | σ = 0.1 | σ = 0.2 | σ = 0.3 | σ = 0.4 | σ = 0.5 | σ = 0.6 | σ = 0.7 | σ = 0.8 | σ = 0.9 | σ = 1 |
---|
1 | 1.159 | 0.671 | 0.551 | 0.570 | 0.647 | 0.727 | 0.790 | 0.836 | 0.870 | 0.895 |
2 | 0.519 | 0.311 | 0.238 | 0.241 | 0.319 | 0.426 | 0.529 | 0.614 | 0.681 | 0.734 |
3 | 0.305 | 0.180 | 0.149 | 0.178 | 0.273 | 0.393 | 0.503 | 0.593 | 0.665 | 0.721 |
4 | 0.151 | 0.105 | 0.110 | 0.171 | 0.284 | 0.410 | 0.522 | 0.611 | 0.681 | 0.735 |
5 | 4.103 | 2.552 | 2.045 | 1.770 | 1.580 | 1.442 | 1.343 | 1.271 | 1.219 | 1.179 |
6 | 2.113 | 1.308 | 1.088 | 1.052 | 1.060 | 1.066 | 1.065 | 1.060 | 1.054 | 1.047 |
7 | 4.371 | 2.619 | 2.047 | 1.748 | 1.560 | 1.411 | 1.314 | 1.246 | 1.197 | 1.160 |
8 | 3.279 | 2.215 | 1.942 | 1.784 | 1.636 | 1.506 | 1.402 | 1.323 | 1.163 | 1.217 |
9 | 2.887 | 1.928 | 1.598 | 1.450 | 1.359 | 1.290 | 1.235 | 1.192 | 1.158 | 1.131 |
10 | 1.998 | 1.273 | 1.083 | 1.062 | 1.071 | 1.081 | 1.079 | 1.072 | 1.064 | 1.056 |
Table 3.
Weighting scores corresponding to different Gaussian weighting factors when the driver is in a fatigue state.
Table 3.
Weighting scores corresponding to different Gaussian weighting factors when the driver is in a fatigue state.
Video | σ = 0.1 | σ = 0.2 | σ = 0.3 | σ = 0.4 | σ = 0.5 | σ = 0.6 | σ = 0.7 | σ = 0.8 | σ = 0.9 | σ = 1 |
---|
1 | 4.925 | 2.768 | 2.061 | 1.742 | 1.553 | 1.423 | 1.331 | 1.263 | 1.214 | 1.176 |
2 | 6.254 | 3.532 | 2.543 | 2.052 | 1.756 | 1.562 | 1.429 | 1.337 | 1.269 | 1.220 |
3 | 4.873 | 2.560 | 1.843 | 1.517 | 1.346 | 1.247 | 1.188 | 1.142 | 1.113 | 1.092 |
4 | 7.417 | 3.878 | 2.732 | 2.184 | 1.856 | 1.638 | 1.487 | 1.383 | 1.307 | 1.251 |
5 | 7.857 | 4.181 | 2.941 | 2.306 | 1.920 | 1.671 | 1.504 | 1.391 | 1.310 | 1.252 |
6 | 8.292 | 4.416 | 3.126 | 2.456 | 2.038 | 1.762 | 1.576 | 1.447 | 1.356 | 1.289 |
7 | 7.583 | 4.017 | 2.817 | 2.232 | 1.882 | 1.652 | 1.496 | 1.387 | 1.310 | 1.253 |
8 | 10.578 | 5.419 | 3.649 | 2.757 | 2.227 | 1.889 | 1.666 | 1.515 | 1.408 | 1.331 |
9 | 8.752 | 4.621 | 3.183 | 2.448 | 2.010 | 1.732 | 1.548 | 1.423 | 1.335 | 1.271 |
10 | 7.572 | 3.935 | 2.729 | 2.138 | 1.791 | 1.571 | 1.427 | 1.329 | 1.261 | 1.211 |
Table 4.
Gaussian weight factors and the number of misclassified samples under different thresholds.
Table 4.
Gaussian weight factors and the number of misclassified samples under different thresholds.
Gaussian Weighting Factors | State Determination Threshold (T) | Number of Misclassified Samples |
---|
σ = 0.1 | 4.749 | 0 |
σ = 0.2 | 2.624 | 1 |
σ = 0.3 | 1.924 | 4 |
σ = 0.4 | 1.593 | 4 |
σ = 0.5 | 1.408 | 4 |
σ = 0.6 | 1.295 | 4 |
σ = 0.7 | 1.221 | 5 |
σ = 0.8 | 1.172 | 5 |
σ = 0.9 | 1.137 | 5 |
σ = 1 | 1.111 | 5 |
Table 5.
Detection results of our proposed method on the EFSD test set.
Table 5.
Detection results of our proposed method on the EFSD test set.
Number of Video Samples | Gaussian Weighting Score (S) | State Determination Threshold (T) | Actual Driving State | Predicted Driving State |
---|
1 | 2.126 | | Normal | Normal |
2 | 1.288 | | Normal | Normal |
3 | 5.051 | | Normal | Normal |
4 | 4.130 | | Normal | Normal |
5 | 4.460 | | Normal | Normal |
6 | 0.603 | | Normal | Normal |
7 | 0.430 | | Normal | Normal |
8 | 8.276 | | Normal | Fatigue |
9 | 7.198 | | Normal | Fatigue |
10 | 5.508 | | Normal | Normal |
11 | 1.295 | | Normal | Normal |
12 | 0.820 | | Normal | Normal |
13 | 0.589 | | Normal | Normal |
14 | 3.436 | | Normal | Normal |
15 | 0.067 | | Normal | Normal |
16 | 1.121 | | Normal | Normal |
17 | 0.827 | | Normal | Normal |
18 | 5.064 | | Normal | Normal |
19 | 8.424 | | Fatigue | Fatigue |
20 | 7.942 | | Fatigue | Fatigue |
21 | 6.222 | | Fatigue | Fatigue |
22 | 6.848 | | Fatigue | Fatigue |
23 | 6.160 | | Fatigue | Fatigue |
24 | 6.222 | | Fatigue | Fatigue |
25 | 9.527 | | Fatigue | Fatigue |
26 | 10.150 | | Fatigue | Fatigue |
27 | 10.653 | | Fatigue | Fatigue |
28 | 10.747 | | Fatigue | Fatigue |
29 | 8.249 | | Fatigue | Fatigue |
30 | 5.974 | | Fatigue | Fatigue |
31 | 8.251 | | Fatigue | Fatigue |
32 | 8.318 | | Fatigue | Fatigue |
33 | 9.274 | | Fatigue | Fatigue |
34 | 8.925 | | Fatigue | Fatigue |
35 | 6.949 | | Fatigue | Fatigue |
36 | 6.160 | | Fatigue | Fatigue |
Table 6.
Statistical results of our proposed method tested on the EFSD test set.
Table 6.
Statistical results of our proposed method tested on the EFSD test set.
Method | Accuracy (%) | Flops (M) | Params (M) | FPS |
---|
Proposed | 94.4 | 74,520.89 | 11.62 | 33.5 |
Table 7.
Detection statistics of the YOLOv4-tiny model on the EIMDSD dataset.
Table 7.
Detection statistics of the YOLOv4-tiny model on the EIMDSD dataset.
Method | AP_0 (%) | AP_1 (%) | mAP (%) | Flops (M) | Params (M) | FPS |
---|
YOLOv4-tiny | 81.43 | 79.91 | 80.67 | 34,734.37 | 5.78 | 37.86 |
Table 8.
Detection statistics of the YOLOv4-tiny model on the CASIA-Iris-Distance infrared face image dataset.
Table 8.
Detection statistics of the YOLOv4-tiny model on the CASIA-Iris-Distance infrared face image dataset.
Method | AP_0 (%) | AP_1 (%) | mAP (%) | Flops (M) | Params (M) | FPS |
---|
YOLOv4-tiny | 99.31 | 97.83 | 98.57 | 34,734.37 | 5.78 | 33.2 |
Table 9.
Experimental results of IDLN model for eye semantic segmentation on EIMDSD.
Table 9.
Experimental results of IDLN model for eye semantic segmentation on EIMDSD.
Method | mIoU | mPA | Flops (M) | Params (M) | FPS |
---|
IDLN | 0.883 | 0.962 | 39,786.52 | 5.84 | 34.1 |
Table 10.
Segmentation performance of the eye semantic segmentation model in the infrared eye image test set.
Table 10.
Segmentation performance of the eye semantic segmentation model in the infrared eye image test set.
Method | mIoU | mPA | Flops (M) | Params (M) | FPS |
---|
IDLN | 0.902 | 0.943 | 39,786.52 | 5.84 | 32.6 |
Table 11.
Comparison of experimental results with other fatigue-driving detection algorithms.
Table 11.
Comparison of experimental results with other fatigue-driving detection algorithms.
Method | Accuracy (%) |
---|
Pandey et al. [22] | 92.5 |
Kaur et al. [23] | 91 |
Miah et al. [24] | 93 |
Proposed | 97.5 |