1. Introduction
Autonomous driving is a technology that evaluates the external environment of a vehicle along with the driver’s condition and controls the vehicle based on the collected information without the direct operation of the driver. Although the commercialization of completely autonomous driving technology has yet to be implemented, Lv.3 technology has been utilized in some mass-produced vehicles. While the safety of autonomous driving has been negatively impacted because of various technical barriers, the technology has gradually developed with reinforced safety functions; however, developing Lv.4 or better technologies and ensuring driving stability requires considerable time. A partially autonomous driving technology, referred to as Lv.2, has already been commercialized and applied to mass-produced vehicles. For the commercialization of the autonomous driving phase of Lv.4 without driver intervention, consistent technology development and safety evaluations are required to improve the reliability of autonomous driving.
The three key sensors of autonomous driving are LiDAR, radar, and cameras. Various autonomous driving technologies have been developed by effectively combining the characteristics of each sensor. Object detection algorithms [
1,
2,
3,
4] have been consistently developed through the combination of cameras and LiDAR, with the limitations of these devices supplemented using the benefits of radar. The camera sensor has been essentially applied to the combination of sensors and operates as the eyes of autonomous vehicles by supplementing the limitations of LiDAR and radar sensors. A camera is a key component in automobiles, future home appliances, and robots and is the only sensor that can capture information on the texture, color, and contrast of the subject and recognize lanes and signals on roads, read signs, and classify objects, such as pedestrians, bicycles, and surrounding vehicles, with high performance. A camera sensor can obtain higher-level visual information than other sensors and is also cheaper.
When a camera detects an object in front of the vehicle, the vehicle applies automatic emergency braking and automatically keeps its lane. The camera also collects traffic information in front, maintains a distance from the vehicle ahead, recognizes traffic signs, and automatically controls high beams. Cameras have been established as major sensors for advanced driver assistance systems’ functions and have been applied in various technologies, such as forward collision prevention, lane departure avoidance, and surround view.
Table 1 lists the types and quantities of the sensors used in autonomous vehicles by manufacturer.
Tesla, a representative company that realizes autonomous driving technology using only cameras, has been collecting considerable amounts of data from more than one million vehicles on the roads—a completely different and rather innovative approach, considering that autonomous driving companies, such as Waymo, that attach expensive sensors (e.g., LiDAR) to small vehicles and other companies that use cameras also apply radar; however, Mobileye, a subsidiary of Intel, is credited with first implementing autonomous vehicle camera sensor technology. Mobileye leads the development of autonomous driving technology with cameras using the EyeQ chip and has achieved significant growth to occupy more than 60% of the initiative of the camera market. Recently, however, Mobileye has been focusing on EyeQ Ultra technology that uses LiDAR and radar and has been developing technology for mass production by 2025.
Along with the technical development in the autonomous driving industry, various studies related to camera technology have been conducted for autonomous driving. First, studies on algorithms required to address environmental pollution sources caused by raindrops, snow, dust, and sand have been reported [
5]. Moreover, technology for accurately measuring the pollution level of surrounding view cameras, installed on the outside and thus vulnerable to pollution, has been developed [
6]. In addition, the effects of geography, environment, and weather on sensor detection have been analyzed [
7], and studies on multitask learning for the adaptability of polluted sensors to recognition algorithms [
8] and on filtering algorithms have been conducted [
9]. A camera sensor is vulnerable to external environments, such as dust, sunlight, rain, snow, and darkness, and its performance can also be degraded by visual obstructions (blockage), such as dust, because it has smaller lens-type geometry than radar and LiDAR sensors, significantly affecting the safety of autonomous driving [
10,
11,
12,
13].
Cameras have been established as essential elements for autonomous driving; however, their pollution has severe consequences. First, in March 2018, the Tesla Model X in autonomous driving mode caused a major accident in a tunnel at a speed of approximately 60 km/h. In January 2019, an autonomous vehicle of Google Waymo hit a person, with camera pollution found to be the cause. In November 2018, an autonomous vehicle of Intel Zero Waste hit a person on the sidewalk. The cause of the accident was estimated to be the dust accumulated on the vehicle, which resulted in the pedestrian not being recognized.
Therefore, for camera sensors with performance degraded by blockage during autonomous driving, studies on the correlation between camera performance degradation and the risk of accidents should be conducted by identifying data collected from the surrounding road environment. In addition, manufacturers have attempted to propose the need for software solutions, sensor fusion, and sensor cleaning technology applications by evaluating the performance of camera recognition algorithms in solving the performance degradation by blockages.
This study developed technology to evaluate the object recognition performance of camera sensors in autonomous vehicles and verify the efficiency of the camera recognition algorithm under blockage. Through this study, the blockage effect on the camera during autonomous vehicle driving can be evaluated while identifying and supplementing the benefits and limitations of object recognition algorithms.
4. Results and Discussions
4.1. Analysis of the Importance of the Major Variables
Machine learning can handle complex and highly nonlinear relationships between dependent and independent variables. As this method randomly selects both data and variables in each tree, each tree has diverse and abundant expressions as an uncorrelated model (≒independent). For random forest parameter optimization, hyperparameters were entered in sequence, GridSearchCV was used to derive the optimal parameters [
23], and the optimal model was constructed using these hyperparameters.
Ensemble models, such as Random Forest, which belongs to the tree-based family of models, generally provide a measure called feature importance, which can be interpreted as the importance of the variables or features. The underlying mechanism is based on the concept of reducing impurity through information gain. In decision trees, nodes are split using a method that minimizes impurity. In ensemble methods composed of multiple decision trees, variable importance is determined by averaging the importance values from each tree [
24].
Here, represents the number of samples at node R, and denotes the set of nodes in the individual tree where the j-th variable is chosen for splitting. Additionally, and refer to the left and right nodes of node R, respectively.
The importance
for the
j-th variable is calculated as follows.
Finally, it is transformed through normalization as follows.
The feature importance analysis results show that the blockage concentration (feature importance value: 0.602) has the largest impact on object recognition, followed by the object type (0.179), blockage color (0.152), and object color (0.067), as shown in
Figure 9. For the concentration variable with the highest feature importance, a certain concentration range was specified to examine its impact on the remaining variables.
4.2. Analysis of the Effects of the Object and Dust Colors on Object Recognition
The blockage concentration, which has the largest impact on object recognition, should be specified to analyze the effects of dust and object colors on object recognition. Considering the characteristics of blockage that gradually accumulated from the clean condition, the effects of the blockage and object colors on the recognition rate were examined based on a 5% dust concentration, which is the lowest concentration in this study and the most easily accessible in the initial stage of pollution.
A histogram and QQ plot were drawn to verify the normality of the data, but bias was detected. To convert the distribution into a normal distribution, “Box–Cox”, a function conversion method that adjusts the skew of the distribution, was used [
25]. Consequently, the data distribution was normalized by adjusting the lambda value in Equation (5) and searching for the optimal lambda value at which the distribution became a normal distribution, as shown in
Figure 10.
Using six factors from three dust and two object colors, descriptive statistics that could identify differences between the individual groups were applied for 4172 data in the 5% blockage concentration range.
Table 4 shows the average and standard deviation of each group.
The normality was satisfied in the Shapiro–Wilk normality test results (W = 0.958,
p < 2.2 × 10
−16), and the histogram and Q-Q plot for the residual analysis for the six groups, which show the difference between the Box–Cox conversion values and representative values, are shown in
Figure 11. As the skewness value was
0.4276 and the kurtosis value was 1.9385, normality was assumed [
26], and an ANOVA analysis was conducted.
Levene’s test of variance homogeneity showed significant results (Levene statistic 38.366,
p < 0.05), indicating that the variances of the six groups were different. As homoscedasticity was not satisfied, Welch and Brown–Forsythe mean tests were conducted on the groups.
Table 5 shows the robust test results for the homogeneity of means. The significance probability confirmed significant differences among the six groups [
27].
Table 6 shows Dunnett’s T3 post hoc test results for fifteen items among the six groups.
Black_Dark was
3.33 lower than Black_Light but was higher than Gray_Dark (17), Yellow_Dark (9.89), and Yellow_Light (3.32); therefore, Black_Light exhibited the highest object score, followed by Black_Dark, Gray_Light, Yellow_Light, Yellow_Dark, and Gray_Dark. In the post hoc test results, most of the mutual significance results of the six factors were satisfied. The significance probability between Black_Dark and Gray_Light and that between Gray_Light and Yellow_Light exceeded 0.05, implying that the average difference between the two groups was not significant, as shown in
Figure 12.
As shown in
Figure 12, the effects of the dust and object colors on the recognition rate can be visually confirmed. Overall, the light objects exhibited higher object scores than the dark objects. This shows that the camera recognition algorithm is insufficient in recognition learning for dark objects compared to bright objects. For the colors, high Box–Cox conversion mean scores were observed for black blockage regardless of the object color. Basically, camera object recognition learning involves object recognition in the nighttime state; therefore, it seems to be a result of the darkness of the nighttime state being similar to the effect of black dust. In addition, the fact that black dust had a particularly high recognition score for certain objects, such as signals and cyclists, also had an impact. Because gray and yellow are relatively bright, contrasting dark objects showed lower recognition scores than bright objects. In particular, the combination of blockage gray and a dark object color exhibited the lowest object score. For the camera object recognition algorithm, improvements are needed to increase the recognition rate for dark objects and gray and yellow dust.
4.3. Effects of Blockage on Different Objects
Under static scenarios, five object types were distinguished, with various object shapes arranged at different positions for each object type. The blockage concentration was increased by 5% to 70%.
Figure 13 shows the object score by object color according to the blockage concentration for all the data. Notably, a difference in the object score was observed depending on the dust color for a concentration up to 25%. An analysis was conducted for the 5–25% concentration range to examine the object score tendency by object type according to the blockage concentration. As a large difference in the object score of each object in the normal state was observed, K-means clustering was conducted on the object scores of light and dark object colors for 15 objects to classify objects with similar patterns into groups [
28,
29]. The initial number of clusters was set to three.
Figure 14 shows the final clustering result.
For the objects in the green group (A-1, P-2, and P-3), the score was low when the object color was dark and relatively high when it was light. For the objects in the red group (P-1, P-4, and P-5), the object score was low regardless of the light and dark object colors. In the case of the objects in the blue group (C-1, C-2, C-3, P-6, V-1, V-2, V-3, V-4, and S-1), the object score was high regardless of the light and dark object colors.
For pedestrians, the object size affected the object score. For P-1, the overall object score was low under the influence of the background (V-1 and traffic signal). The vehicles, cyclists, and signals mostly exhibited high object scores regardless of color.
To examine the effect of blockage on the recognition of object types, the object score values in the normal state without blockage were summarized for 30 objects, as shown in
Figure 15.
The effects of the blockage concentration on the object types were analyzed for 12 objects (dark: C-1, C-2, P-6, S-1, and V-4; light: A-1, C-1, C-2, P-3, P-6, S-1, and V-1), corresponding to object scores of 85–93 with little difference in the individual object scores, as shown in
Figure 15. The twelve objects were grouped into five object types.
Figure 16 shows the object score trend in the 5–25% concentration range.
A noticeable difference in the object score between black and both gray and yellow was observed. For black, the object score decreased in the order of the signal, cyclist, vehicle, pedestrian, and animal. For yellow, it decreased in the order of the animal, pedestrian, vehicle, cyclist, and signal. For gray, it decreased in the order of the pedestrian, animal, vehicle, cyclist, and signal. Compared with black, yellow and gray exhibited almost the opposite object score results. In other words, object recognition was significantly affected by the blockage color. Particularly, for the signal, the pollution resistance was high for black but was significantly low for gray and yellow. The signal exhibited the largest difference in recognition tendency depending on the color, followed by the cyclist and vehicle. Pedestrians and vehicles, which can be considered representative objects, had object scores within similar categories regardless of color. Since learning from various cases was greater than that of other objects, it does not seem to have been greatly affected by blockage color. Nevertheless, the yellow and gray colors need improvement, particularly for the low object scores of the signal and cyclist.
Figure 17 shows the object scores of five individual objects with respect to the blockage concentration.
For the animal, black and gray exhibited similar tendencies, with object recognition failing at a blockage concentration of 15%. For yellow, recognition was impossible at 25%. For the cyclist, the object score sharply decreased from 20% for black but was maintained at approximately 40. For yellow and gray, object recognition failed between 15–20%. For the pedestrian, the object score slowly decreased from approximately 70 to 20 as the blockage concentration increased for gray and yellow. For black, it sharply decreased from 10% to 15%, and object recognition was infeasible at 20%. For the signal, a noticeable difference in the object score between black and both gray and yellow was observed. Particularly, for gray and yellow, the signal recognition was almost infeasible from 5%, indicating that the algorithm for blockage needs to be supplemented for this range in the future. Finally, in the case of the vehicle, no significant difference in color compared with other objects was observed, with recognition failing at 15% to 20%. Based on these results, when the manufacturer managed the object score at 40 or higher in the case of 10% blockage, 11 objects out of 15 met the management target; however, for the remaining four objects, the recognition algorithm needed to be improved through learning. In terms of autonomous driving sensor cleaning, this 10% blockage can be presented as the basis for camera lens cleaning to remove blockages.
5. Conclusions
In this study, the object recognition performance of an autonomous driving camera algorithm was evaluated from different angles by applying virtual blockages. The implications and limitations derived from the research results are as follows.
First, the concentration of the blockage was the most significant factor affecting object recognition. When the blockage concentration was 10% or less, most objects could be accurately recognized. If there is no additional improvement in cognitive performance owing to dust contamination, the standard for cleaning the camera is when contamination exceeds 10% concentration. Therefore, the appropriate timing of camera lens cleaning could be determined.
Second, as for the blockage color, yellow and gray were found to be more unfavorable for object recognition than black, indicating different tendencies depending on the object type. For important objects, such as traffic signals, the recognition algorithm needed to be improved.
Third, through the developed interface between the evaluation system and autonomous driving camera, a general-purpose methodology capable of comparing and evaluating the performance of camera algorithms from various manufacturers in virtual environments was presented.
Fourth, the costs and risks could be reduced by reproducing vehicle driving situation cases not easily implementable in a virtual environment. In addition, various scenarios other than dust, such as weather conditions, pedestrians, vehicles, and buildings, could be utilized for camera algorithm training.
Fifth, autonomous driving camera recognition technology manufacturers should set and comply with reference points for object recognition even at a certain level of blockage during technical development and mass production; these reference points could improve the safety of autonomous driving technology and user confidence.
In this study, we investigated object recognition based on dust concentration and color. In the future, performance evaluations of various cameras are necessary. The type of camera covering must additionally consider the impact of not only dust, but also weather conditions, such as fog and rain. In addition, studies on the recognition rate depending on the background and object colors and differences by object type will be conducted by arranging road environments and objects in a strictly controlled environment. In addition, an experiment in which dust is sprayed onto an actual camera and compared with the dust in a virtual environment will be conducted to enhance the validity of the virtual environment test. Finally, studies on the comparison of the performances of various camera manufacturers and an analysis of the differences depending on blockage will be conducted.