ICONet: A Lightweight Network with Greater Environmental Adaptivity

Abstract

With the increasing popularity of artificial intelligence, deep learning has been applied to various fields, especially in computer vision. Since artificial intelligence is migrating from cloud to edge, deep learning nowadays should be edge-oriented and adaptive to complex environments. Aiming at these goals, this paper proposes an ICONet (illumination condition optimized network). Based on OTSU segmentation algorithm and fuzzy c-means clustering algorithm, the illumination condition classification subnet increases the environmental adaptivity of our network. The reduced time complexity and optimized size of our convolutional neural network (CNN) model enables the implementation of ICONet on edge devices. In the field of fatigue driving, we test the performance of ICONet on YawDD and self-collected datasets. Our network achieves a general accuracy of 98.56% and our models are about 590 kilobytes. Compared to other proposed networks, the ICONet shows significant success and superiority. Applying ICONet to fatigue driving detection is helpful to solve the symmetry of the needs of edge-oriented detection under complex illumination condition environments and the scarcity of related approaches.

1. Introduction

Considering that 14–20% of traffic accidents are caused by fatigue driving [1], fatigue driving detection is facing urgent needs and with high research significance. An effective fatigue driving detection approach will significantly reduce the consequent traffic accidents.

At present, there are mainly three types of methods to monitor fatigue driving. The first type of method is based on vehicle parameters. These methods focus on the rotation speed of the steering wheel, the change of the offset angle, and the changing frequency of the pedal [2]. The second type is based on physiological characteristics. These methods distinguish the driver’s mental state based on the driver’s multiple physiological characteristics [3], including blood pressure, pulse, heart rate, EMG (electromyography) signal, and EEG (electroencephalogram) signal. The involved algorithms include logistic regression, support vector machine, the k-nearest neighbor classifier [4,5], and artificial neural network [6]. The third type is based on computer vision. The driver’s driving behavior data is collected by a camera installed in the vehicle. These methods are based on relevant algorithms to comprehensively analyze the video data. They mainly focus on the eyes and mouth of the driver [7,8,9].

There are some noteworthy drawbacks to the first two types of methods. The methods based on vehicle parameters require a lot of installed sensors, which increases the cost. Meanwhile, the sensing and processing lag of related sensors may not provide a real-time result. The methods based on physiological characteristics require physical contact with the driver and the collected information is private, which may be difficult to be widely utilized due to various individual reasons. Considering the drawbacks of these two types of methods, most research nowadays focuses on the proper utilization of computer vision.

Deep learning has recognized many advances in the past several decades. Artificial intelligence is migrating from the cloud to the edge. Based on deep learning, future fatigue driving detection methods should have high accuracy, great environmental adaptivity, and orient to edge devices.

The motivation of this paper is twofold. First, compared to fatigue detection approaches based on vehicle parameters and physiological characteristics, as well as considering the recognized success of deep learning, the deep learning-based approaches are with less cost and able to be widely spread. Second, the current deep learning-based approaches do not consider both great environmental adaptivity and edge-orientation. This is because most deep learning frameworks are too complicated to be loaded on resource-limited edge devices [10].

In this paper, we propose a novel framework named ICONet (illumination condition optimized network). The main contributions of our work are shown as follows:

• We propose an illumination condition classification subnet based on the OTSU segmentation algorithm and fuzzy c-means clustering algorithm. This subnet focuses on classifying input pictures into three types, including normal daylight, weak daylight, and night.
• We propose a convolutional neural network subnet based on Haar-like features, AdaBoost algorithm, and the modified LeNet-5 network model. This subnet focuses on drivers’ face extraction and behavior classification.
• We design the two subnets in the ICONet with high modularity. Not limited to fatigue driving detection, ICONet can be applied to other classification problems under various illumination conditions.

The rest of this paper is organized as follows. Section 2 describes the related work. Section 3 introduces the methods used in the ICONet. Section 4 describes the experimental results. Section 5 provides a discussion. Section 6 concludes the whole paper.

2. Related Work

In this section, we explore the current research of fatigue driving detection in the field of deep learning, with regard to the aspect of environmental adaptivity and edge-orientation.

2.1. Deep Learning Approaches Considering Environmental Adaptivity

In the aspect of environmental adaptivity, the interference of changing illumination conditions must be considered. Fatigue driving usually occurs at night or dusk when the illustration condition is weak daylight or night.

Ma, Chau et al. [11] presented a convolutional three-stream network architecture, which integrated current-infrared-frame-based spatial information and achieved an accuracy of 94.68%. Hao et al. [12] presented a parallel convolutional neural network (CNN). The proposed method was based on the different detection characteristics of the same image. CNN was used to automatically complete the feature learning. It was claimed with high robustness to a complex environment. Villanueva et al. [13] used a deep learning algorithm. The proposed system used images captured by the camera to detect patterns in the driver’s facial features (eyes closed, nodding/head tilt, and yawning). A deep neural network named SqueezeNet was used for faster model development and retraining. There existed an alarm when drowsy driving was detected. Garcia et al. [14] presented a non-intrusive approach including three stages. The first stage was face detection and normalization. The second stage performed pupil position detection and characterization. The final stage calculated the percentage of eyelid closure (PERCLOS) based on closed eyes information. Songkroh et al. [15] used both vehicle speed and driver behavior to analyze and to determine the risk level of the driver. The proposed system included facial image preprocessing, facial feature detection, feature classification, and an analysis module. The risk alert yielded an accuracy of 86.30% at any vehicle speed. Yu et al. [16] proposed a condition-adaptive representation learning framework. The framework combined spatio-temporal representation and estimated scene conditions were merged to enhance the discriminative power. Spatio-temporal representation learning extracted features that could simultaneously describe motions and appearances. Scene condition understanding classified the scene conditions related to various situations. Memon et al. [17] built a non-intrusive constant checking framework based on OpenCV. Ma et al. [18] presented a two-stream CNN structure focusing on the night situation and achieved an accuracy of 91.57%.

Despite some good results in methods considering environmental adaptivity, these networks have high computational complexity and cannot be applied to resource-limited edge devices.

2.2. Deep Learning Approaches Considering Edge-Orientation

In the aspect of edge-oriented methods, most previous works faced embedded systems including Raspberry Pi and Android devices including smartphones.

On embedded systems, Gu et al. [19] proposed a convolutional neural network model with multi-scale pooling (MSP-Net) and implemented it on an NVIDIA JETSON TX2 development board. On Raspberry Pi, Sharan et al. [20] proposed a driver fatigue system based on eye states using a convolutional neural network. Ghazal et al. [21] used CNN to perform embedded fatigue detection and achieved an accuracy of 95%. Their proposed approach included video signal spatial processing and deep convolutional neural network classification.

On Android devices including smartphones, Xu et al. [22] presented the Sober-Drive system based on a neural network and achieved an accuracy of 90%. Dasgupta et al. [23] proposed a three-stage drowsiness detection with an accuracy of 93.33%. The three stages included PERCLOS calculation, the voiced to the unvoiced ratio, and touch response, which could generally detect drowsy driving and subsequently raised an alarm. Galarza et al. [24] proposed a surveillance system for real-time driver drowsiness detection and with an accuracy of 93.37%. Jabbar et al. [25] proposed a real-time driver drowsiness detection system. Their saved CNN model was within 75 kilobytes and had an accuracy of 83%.

These edge-oriented methods and systems can effectively perform fatigue driving detection and have been realized on edge devices. However, they are not adaptive to various environmental conditions and have relatively low accuracy. Several seconds of fatigue driving may directly cause a fatal traffic accident. An optimized method must detect fatigue driving with high accuracy in whatever environmental condition.

3. Methods

The structure of the illumination condition optimized network is shown in Figure 1. ICONet includes two subnets. The first subnet classifies illumination conditions, and the second subnet classifies related behaviors based on a modified LeNet-5 CNN model. The final stage of ICONet is a comprehensive judgement based on the output results of the two subnets. The whole network is designed to aim at greater environmental adaptivity and implementation on edge devices. For the first goal, our idea focuses on an effective classification. An optimized network is expected to accurately classify images under different illumination conditions, then accordingly call the specific pre-trained model. Each illumination condition type corresponds to a CNN model, which correlates with the symmetry concept. Instead of involving image correction, directly calling the pre-trained models under various conditions will reduce the real-time computing power, which is suitable for edge devices. For the second goal, considering the limited resources on edge devices, we optimize the CNN network structure, remove unnecessary layers, and reduce the convolutional kernel size. The network parameters are carefully adjusted to guarantee high accuracy when applied to fatigue driving detection.

3.1. Illumination Condition Classification Subnet

This subnet is based on OTSU segmentation algorithm and fuzzy c-means clustering algorithm.

3.1.1. OTSU Segmentation Algorithm

The OTSU segmentation algorithm [26] is used to determine the image binary segmentation threshold value. Based on the one-dimensional histogram of the gray image, it selects the best threshold value and uses this threshold value to divide the entire image into two parts, including the target and the background. The optimal threshold value can maximize the variance between the two parts of the image. The detail of the OTSU segmentation algorithm is described as follows.

Consider a threshold gray value $Thre$ splits the pixels in an image into two groups: $C_1$ (with pixel gray value < $Thre$) and $C_2$ (with pixel gray value < $Thre$). The average gray values of $C_1, C_2$ pixels are $m_1, m_2$. The average global gray value of the image is $m_G$. The probabilities of a pixel classified as $C_1$ or $C_2$ are $p_1$ and $p_2$.

$$p_1={\displaystyle \sum }_{i=0}^{Thre}{p}_i$$

(1)

$$p_2={\displaystyle \sum }_{i=Thre+1}^{255}{p}_i$$

(2)

$$p_i=\frac{n_i}{n}$$

(3)

where $n_i$ is the number of pixels with gray value $i$.

$$m_1=\frac{1}{p_1}{\displaystyle \sum }_{i=0}^{Thre}i{p}_i$$

(4)

$$m_2=\frac{1}{p_2}{\displaystyle \sum }_{i=Thre+1}^{255}i{p}_i$$

(5)

$$m_G={\displaystyle \sum }_{i=0}^{255}i{p}_i$$

(6)

with constraints in Equations (7) and (8).

$$p_1{m}_1+p_2{m}_2=m_G$$

(7)

$$p_1+p_2=1$$

(8)

The variance between these two classes is

$${\sigma }^2=p_1{\left(m_1-m_G\right)}^2+p_2{\left(m_2-m_G\right)}^2$$

(9)

By substituting, we have

$${\sigma }^2=p_1{p}_2{\left(m_1-m_2\right)}^2$$

(10)

The optimized threshold value will obtain a maximum variance ${\sigma }^2$ among all the 256 gray values. The whole process of the OTSU segmentation algorithm is described in Algorithm 1.

Algorithm 1 OTSU segmentation algorithm

4. Initial: $Thre=0$, ${\sigma }_{Thre}^2=0$

5.  for $i=0$ to 255

6.   Calculate $p_1, p_2, m_1, m_2, m_G$

7.   Calculate ${\sigma }_i^2$

8.   if ${\sigma }_i^2>{\sigma }_{Thre}^2$

9.     ${\sigma }_{Thre}^2={\sigma }_i^2$

10.     $Thre=i$

11.   end if

12. end for

13. return $Thre, {\sigma }_{Thre}^2$

3.1.2. Fuzzy C-Means Clustering Algorithm

The fuzzy c-means clustering algorithm [27] introduces the fuzzy theory to provide more flexible clustering results than normal hard clustering. The algorithm assigns a weight to each object and each cluster. It indicates the degree to which the object belongs to the cluster. Pixels in the image are divided into many disjoint sets based on the characteristic distance of each pixel. This distance reflects the similarity among all the pixels. Consider $X=\left\{x_1,\dots ,x_n\right\}$ as a set of $n$ objects, and $V=\left\{v_1,\dots ,v_k\right\}$ is the set of centers of the $k$ clusters. $U$ is a $k\times n$ partition matrix, where $u_{ij}$ is the membership degree of a sample $x_j$ to the cluster center $v_i.$ All cluster centers and the membership of the pixels can be obtained by minimizing the target function. The target function is

$$J\left(U, V\right)={\displaystyle \sum }_{i=1}^k{\displaystyle \sum }_{j=1}^n{u}_{ij}^m{d}_{ij}^2={\displaystyle \sum }_{i=1}^k{\displaystyle \sum }_{j=1}^n{u}_{ij}^m\Vert x_j-v_i\Vert {}^2$$

(11)

$${\displaystyle \sum }_{j=1}^k{u}_{ij}=1, 0\le u_{ij}\le 1$$

(12)

where $n$ represents the number of samples, $k$ represents the number of clusters, fuzzy weight index $m\in \left(1, +\infty \right)$. $m$ usually equals to 2. $d_{ij}=\Vert x_j-v_i\Vert$ is the spatial Euclidean distance from the pixel $x_j$ to the cluster center $v_i$.

$$v_i=\frac{{{\displaystyle \sum }}_{j=1}^n{u}_{ij}^m{x}_j}{{{\displaystyle \sum }}_{j=1}^n{u}_{ij}^m}, 1\le i\le k$$

(13)

$$u_{ij}=\frac{1}{{{\displaystyle \sum }}_{p=1}^k{\left(\frac{d_{ij}}{d_{pj}}\right)}^{\frac{2}{m-1}}}, 1\le i\le k, 1\le j\le n$$

(14)

The membership degrees are initially randomly assigned. The cluster center and membership in each following iteration are calculated based on Equations (13) and (14). The iterative process will stop if and only if

$$\left|U^{\left(t\right)}-U^{\left(t-1\right)}\right|=\max \left\{\left|u_{ij}^{\left(t\right)}-u_{ij}^{\left(t-1\right)}\right|\right\}>\epsilon , 1\le i\le k, 1\le j\le n$$

(15)

where $\epsilon >0$ and is preassigned. The whole process of fuzzy c-means clustering is described in Algorithm 2.

Algorithm 2 Fuzzy C-Means Clustering algorithm

1.  Initial: $m, \epsilon ,k$

2. Randomly initialize the partition matrix $U$

3.  while $\left|U^{\left(t\right)}-U^{\left(t-1\right)}\right|<\epsilon$ do

4.   Calculate $v_i, 1\le i\le k$

5.   Calculate $u_{ij}, 1\le i\le k, 1\le j\le n$

6. end while

After performing a lot of tests, we observed that the OTSU threshold value, the ratio of OTSU threshold and average gray value, and the minimum of the target function reflects the illumination condition of an image.

3.2. Convolutional Neural Network Classification Subnet

When applied in the field of fatigue driving detection, this subnet involves face detection and the extraction of the region of interest at the beginning. Face detection in our model is based on Haar-like features and the AdaBoost algorithm [28]. The Haar-like feature is a simple rectangular feature in the face detection system. It is defined as the difference of the pixel’s global gray value in adjacent areas of an image. The rectangular feature can reflect the gray changes of the local features of the detected object. The introduction of the integral images accelerates the feature acquisition speed of the detector. The basic idea of the AdaBoost algorithm is to superpose lots of weak classifiers to be a strong classifier with strong classification ability. Then, several strong classifiers are connected in series to complete image retrieval.

We modified the traditional LeNet-5 model, and the structure of the proposed CNN is shown in Figure 2. For the input image, our model uses thirty-two convolution kernels with 5 $\times$ 5 size, one pooling layer with 2 $\times$ 2 size, sixteen convolution kernels with 5 $\times$ 5 size, one pooling layer with 2 $\times$ 2 size, and three full connection layers.

There is a SoftMax layer after the last full connection layer. The SoftMax function is defined in Equation (16).

$$S{\left(z\right)}_i=\frac{e^{z_i}}{{{\displaystyle \sum }}_{j=1}^j{e}^{z_j}} , i=1,2,\dots ,j$$

(16)

The loss function based on SoftMax cross-entropy is defined in Equation (17).

$$Loss=-{\displaystyle \sum }_i{y}_{i}ln{s}_i$$

(17)

where $y_i$ represents the label, and $s_i$ represents the predicted probability.

We performed regularization in loss function to improve the generalization ability of the model and avoid over-fitting problems [29]. Indicators that can reflect the complexity of the model are added to the loss function. If the loss function used to describe the performance of the model on training data is $J\left(\theta \right)$, then the optimized target function is $J\left(\theta \right)+\lambda R\left(w\right)$. $R\left(w\right)$ represents the complexity of the model, and $\lambda$ represents the proportion of the model’s complexity loss in the total loss. $\theta$ represents all the parameters in a neural network, including the weight $w$ and the bias term $b$. The L2-norm regularization formula [30] used in this paper is shown in Equation (18).

$$R\left(w\right)={\left|\left|w\right|\right|}_2^2={\displaystyle \sum }_i\left|w_i^2\right|$$

(18)

The time complexity (floating-point operations per second) of convolutional layers in a CNN model is defined in Equation (19).

$$FLOP{s}_{CONV}=O({\displaystyle \sum }_{l=1}^D{M}_l^2{K}_l^2{C}_{l-1}{C}_l)$$

(19)

where $D$ is the depth of the network; $l$ is the $l^{th}$ convolution layer; $M$ is the length of the feature map belonged to each convolution kernel; $K$ is the size of convolution kernel; $C_l$ is the number of output channels of the $l^{th}$ convolution layer.

As for a fully connected layer, consider the dimension of input data is $\left(N, D\right)$, the weight dimension of a hidden layer is $\left(D, out\right)$, and the dimension of output data is $\left(N, out\right)$. Then, the time complexity (FLOPs) of a fully connected layer in a CNN model is defined in Equation (20). We compare the time complexity of ICONet with other approaches in Section 4.3.2.

$$FLOP{s}_{FC}=\left(2D-1\right) ·out$$

(20)

3.3. Comprehensive Judgement

Applying in the field of fatigue driving detection, the comprehensive judgement stage in ICONet is about fatigue judgement. Fatigue is closely related to the frequency of eyes and mouth closing.

PERCLOS [31] calculates the ratio of the eye closure frames within a certain period and then infers the driver’s eye closure frequency. PERCLOS can be calculated by:

$$f_{PERCLOS}=\frac{n_{close}}{n_{close}+n_{open}}\times 100\%$$

(21)

Similar to PERCLOS, FOM (frequency of mouth) [32] calculates the ratio of mouth open within a certain period and then infers the driver’s yawn frequency. FOM is calculated by:

$$f_{FOM}=\frac{n_{open}}{n_{close}+n_{open}}\times 100\%$$

(22)

For an input video, a certain frequency range is required for accurately calculating fatigue parameters. After a certain frame is detected by ICONet, according to the first-in-first-out principle, the latest result is added to the first place of the queue and the last value of the queue is removed. The total number of frames in the queue remains constant. In Figure 3, “1” represents a closed mouth or eye, and “0” represents an opened mouth or eye.

Combining the frequency queue with PERCLOS and FOM parameters, we can judge whether a driver is fatigued driving.

4. Experimental Results

All experiments were conducted on a computer with Intel(R) Core™ i7-10750H CPU @2.6GHz, 16.0GB RAM, NVIDIA GeForce RTX 2060, and Windows 10. The algorithms were developed in Python 3.6 via OpenCV 3.3.1 and TensorFlow 1.13.1.

4.1. Dataset

4.1.1. YawDD Dataset

YawDD dataset [33] includes two video datasets of car drivers’ behaviors in the car. Fatigue driving behaviors are involved. These drivers include males and females, with and without glasses, and from different races. In the 322 videos of the first video set, the camera is installed under the front mirror of the car. In the 29 videos of the second video set, the camera is installed on the dashboard.

4.1.2. CEW (Closed Eyes in the Wild) Dataset

The CEW dataset [34] is about eye state detection under normal daylight. It includes 2423 volunteers and 1192 of them are with eyes closed. There are 2462 images with open eyes and 2384 images with closed eyes.

4.1.3. Self-Collected Dataset

Our network focuses on environments with complex illumination conditions, however, YawDD and CEW datasets are about various driving behaviors under normal daylight. There are currently no open-access datasets focusing on environments with different illumination conditions. Thus, we collected our dataset by an infrared camera including fatigue driving behaviors under weak daylight and night.

The self-collected data includes 15 drivers. Each driver has 1 or 2 videos, depending on whether they wear glasses or not. Normal driving and yawning behaviors are involved in each video. The video is captured at 30 frames per second, and the frame height and width are 1920 × 1080.

4.2. Illumination Condition Classification Subnet

We have mentioned in Section 3.1 that four threshold values can represent the illumination condition of a picture. The four threshold values include the OTSU threshold value, average gray value, the ratio of OTSU threshold value and average gray value, and a minimum of target function in fuzzy c-means clustering. We classified the illumination condition types of pictures into normal daylight, weak daylight, and night. Based on YawDD and the self-collected dataset, we captured the videos according to a certain frame rate. We randomly picked 70% of the pictures to calculate the threshold ranges. The remaining 30% were used to verify the accuracy. The distribution of related parameters in the set of pictures used for calculation is shown in Figure 4.

Figure 4 indicates that under three types of illumination condition, the related parameters are within a certain range. After comparing the distribution of related parameters, we can obtain that the condition for normal daylight is $Average Gray value\in \left[70, 150\right] \&\& OTSU Threshold value\in$ $\left[100, 160\right] \&\& ratio\in \left[0.8, 1.6\right] \&\& Minimum of target function\in \left[0, 150\right]$, the condition for weak daylight is $Average Gray value\in \left[25, 70\right] \&\& Minimum of target function\in \left[200, 550\right]$, the condition for the night is $Average Gray value\in \left[0, 25\right] \&\& OTSU Threshold value\in \left[40, 90\right] \&\& ratio\in \left[2, 20\right]$. To test the accuracy of the method, we performed this method on the test datasets mentioned in Section 4.1. The results are shown in

Table 1. The accuracy of illumination condition classification.

Illumination Condition	Accuracy	General Accuracy
Normal daylight	98.55%	98.31%
Weak daylight	96.61%
Night	99.76%

.

According to Table 1, the proposed illumination condition classification subnet achieves a general accuracy of 98.31% when classifying the various illumination conditions. This subnet works as a pre-classification stage and leads to a well-directed behavior classification in the next subnet.

4.3. Convolutional Neural Network Classification Subnet

4.3.1. Experimental Results

In the CNN classification subnet, we first performed face detection and region of interest extraction based on the AdaBoost algorithm and Haar-like feature. The process is shown in Figure 5. In the classification process, the related CNN models are pre-trained. We used CEW and self-collected datasets to train eye and mouth models based on the proposed CNN model. The ratio of the training set and testing set was 70% to 30%.

In the training process, our model first resizes the input picture to 24 × 24, then optimizes the model based on the stochastic gradient descent (SGD) method and updates the parameters of the neural network. The batch size of the neural network is 120 and the learning rate is 0.001. Based on the CEW dataset, we trained the eye models under different illumination conditions. The mouth models were trained based on YawDD and self-collected datasets. The models’ loss and accuracy of training and testing are shown in Figure 6 and Figure 7.

Besides, we mixed the dataset under different illumination conditions and trained a hybrid model. The involvement of this hybrid model works as an ablation study. We expected to earn a higher accuracy of our subnet than the hybrid model. It will demonstrate the necessity and effectiveness of the involvement of the illumination condition classification subnet.

Under the circumstances of inputting pictures captured from a video, we compared the accuracy of this hybrid model with models involving illumination condition classification. The result is shown in

Table 2. Accuracy comparison of the hybrid model and models under different illumination conditions.

Illumination Condition	Eyes Classification Accuracy	Mouth Classification Accuracy	General Accuracy
Normal daylight	97.43%	99.65%	98.56%
Weak daylight	98.21%	98.57%
Night	98.93%	98.67%
Hybrid	94.24%	92.45%	93.33%

.

According to Table 2, the classification of illumination conditions earns a 5.23% superiority of accuracy. In other words, the involvement of the first subnet can guarantee at least one more correct behavior classification in every twenty detections. It should be noted that an accident may occur only after several seconds of fatigue driving. A more accurate driving behavior classification will result in an earlier notification if the proposed model is implemented on the vehicular devices, which will reduce the possibility of a tragic accident.

4.3.2. Comparison with Other Approaches

Additional to the ablation study, we performed comparisons between the proposed ICONet and other approaches. Since the training of all the models involves a self-collected dataset, we first performed a comparison of the eye models on the public CEW dataset, in order to strengthen the persuasiveness and demonstrate the superior ability of our proposed network. As is introduced in Section 4.1.2, the CEW dataset only includes eye images under normal daylight. The comparative results are shown in Figure 8 and

Table 3. Accuracy comparison of testing on the CEW dataset.

Author or Network	Year	Accuracy
Sharma [35]	2018	97.80%
Sharan [20]	2019	96.56%
ICONet	2020	98.30%

.

We reproduced the network proposed by Sharma [35] and Sharan [20]. According to Figure 8 and Table 3, our network earns a superior accuracy of at least 0.5%. The results indicate that regardless of the illumination condition classification, simply utilizing the second subnet of our network can comparatively obtain a better result. This proves that our improvements to the traditional LeNet-5 framework are effective and essential.

In the literature [12,13], the authors noted that their networks could be applied under different illumination conditions. To prove the superiority of the proposed ICONet, we reproduced several models in the two pieces of literature and compared them with ICONet. The results are shown in Figure 9.

Figure 9 compares the changing of test and train accuracies among the three networks. It can be observed that previous approaches are with high training cost, while ICONet requires approximately 100 steps to reach a stable accuracy. When comparing the final stable accuracy, the results demonstrate that although previous approaches can be utilized in various illumination conditions, they are not designed with superior environmental adaptivity compared to ICONet.

Verification of the proposed network with high accuracy on a computer or a server is not the final step. Instead, the resources-limited vehicular devices are closer to a driver in a real scenario. Aiming at the network implementation on edge devices, we compare the model size and time complexity (FLOPs) of ICONet with other approaches. A network with lower time complexity and smaller model size can perform better on edge devices and comparatively without serious lag. The time complexity (FLOPs) is calculated based on Equations (19) and (20). The results are shown in

Table 4. Time complexity and model size comparison.

Approach or Network	Involved Dataset		Time Complexity FLOPsCONV (Convolutional Layers)	Time Complexity FLOPsFC (Fully Connected Layers)	Model Size (Kilobytes)
	CEW	YawDD and Self-Collected
Sharma [35]	✓		3,888,000	1,801,498	3624.96
Sharan [20]	✓		15,206,400	1,704,958	14,725.12
Hao [12]		✓	1,525,678,080	26,210,044	65,873.92
Villanueva [13]		✓	835,794,912	/	4915.20
ICONet	✓	✓	3,225,600	270,498	592.72

.

According to Table 4, ICONet has superiority in convolutional layer time complexity of at least 17.04%, and a fully connected layer time complexity of at least 84.13%. Working as a lightweight network, ICONet is capable to be loaded on edge devices.

4.4. Comprehensive Judgement

The illumination condition classification subnet and the convolutional neural network subnet can only provide the classification results of mouth and eyes behavior under various illuminations. The comprehensive judgement stage is designed to combine these classification results and determine whether a driver is in fatigue driving. This determination is based on the PERCLOS and FOM threshold values. The process is shown in Figure 10.

In the process of fatigue driving detection, only focusing on one characteristic is inaccurate. For example, a driver may blink with high frequency or close eyes for a long time under intense illumination. Our approach combines the characteristics of both eyes and mouth.

Figure 11 shows the eye and mouth results during yawning, where “1” represents eye or mouth closing, and “0” represents eye or mouth opening. From the 30th captured frame to the 50th captured frame, the driver can be considered to be fatigue driving. According to related literature [23] and experiments, we set the threshold value as $f_{PERCLOS\_Threshold}=0.25, f_{FOM\_Threshold}=0.2$. A driver can be considered fatigue driving if $f_{PERCLOS}\ge 0.25$ or $f_{FOM}\ge 0.2$, i.e., in 100 continuously captured frames, there are at least 25 frames closing eyes or 20 frames opening mouth. Based on the threshold value range, we performed tests on YawDD and self-collected datasets. The result is shown in Figure 12.

Figure 12 demonstrates that under the high accuracy of the previous two subnets, ICONet can effectively judge fatigue driving.

5. Discussion

In the field of deep learning-based fatigue driving detection, previous works have achieved some significant success. However, with the migration of artificial intelligence from the cloud to the edge, fatigue driving detection is required to be effectively loaded on edge devices and have high accuracy under various environments. Most previous works mainly focus on one aspect and perform undesirably when being applied to other aspects.

Aiming at a greater environmental adaptivity, some pieces of literature design complicated network structures [12,13], while other approaches attempt to obtain a desirable result based on a large training set. The proposed illumination condition classification subnet in this paper is based on traditional image processing algorithms. The experimental results prove the effectiveness of our framework.

Aiming at the implementation on edge devices, we modified the LeNet-5 model, which is the most lightweight framework among all the CNN frameworks. The convolutional layers and kernels are optimized to obtain a lower time complexity. The experimental results comparatively demonstrate the compactness of ICONet.

Additional to the presented results, it should be noted that ICONet is designed to be a universal network, which works as a general solution towards classification problems in various illumination condition environments. Besides fatigue driving detection, it has the potential to be applied to other fields including but not limited to traffic classification, human activity classification, and classification problems in medical science and agriculture.

By comparing with other proposed networks [14,36,37,38], ICONet has several limitations, which guides the direction of our future study.

In the illumination condition classification subnet, we focused on the natural illumination conditions and classified the input pictures into normal daylight, weak daylight, and night. However, the complex illumination conditions may also include different luminance and location of the light source. Especially at night, the suddenly exerted intense light may affect the classification result. When considering these factors, the subnet may fail to classify well. In future work, we will focus on the classification of unnatural illumination conditions.

In the convolutional neural network classification subnet, the model focuses on a single person, which is the driver. However, there may exist more than one person in real scenarios, including the copilot, passengers, and people outside the vehicle. The subnet may fail to correctly detect the driver’s face under these circumstances. We will involve additional image processing algorithms in our future work.

Despite the mentioned limitations, ICONet provides a reference for designing a multi-subnet framework. With the sharp increase of edge devices, ICONet works as an attempt towards the future development of edge-oriented deep learning.

6. Conclusions

Artificial intelligence is migrating from cloud to edge, and deep learning is required to be edge-oriented and adaptive to complex environments. In this paper, we proposed an illumination condition optimized network (ICONet) and applied it to fatigue driving detection. Based on the OTSU segmentation algorithm and fuzzy c-means clustering algorithm, the illumination condition classification subnet classifies pictures under normal daylight, weak daylight, and night. After the face detection and the extraction of the region of interest, the CNN classification subnet provides the classification results of eyes and mouth based on the modified LeNet-5 model. According to indicators including PERCLOS and FOM, ICONet can comprehensively judge fatigue driving. ICONet achieves a general accuracy of 98.56% and time complexity is reduced by at least 17.04% compared to the previous work. The size of all the CNN models is about 590 kilobytes. Experimental results demonstrate the feasibility of applying ICONet on edge devices under various illumination conditions in fatigue driving detection.

In our future work, besides solving the mentioned limitations, we will transplant the ICONet to the Android platform and test it on onboard devices. Additionally, we will add more driving behaviors and further optimize our model to improve its environmental adaptivity.