1. Introduction
With the continuous advancement of computer vision [
1], there is an increasing demand for higher standards in autonomous driving object detection. Autonomous vehicles [
2,
3] must have the capability of fast and accurate object detection in real-world conditions in order to ensure safe and correct driving behavior and decisions [
4]. In other words, an object detector intended for autonomous driving applications must exhibit attributes such as high accuracy, robustness, and light weight. However, adverse weather conditions, such as the presence of heavy snow, fog, rain, dust, or sandstorm conditions, can lead to a decrease in camera resolution, resulting in unclear object features and details, which can impact the accuracy of automatic driving detectors [
5]. Given the current technological constraints, enhancing the camera resolution in such conditions within a short timeframe is challenging. Therefore, refining the algorithm for improving the object detector in autonomous driving systems represents the most viable approach.
In the field of computer vision, numerous methods for detecting vehicle objects have emerged [
6], with deep learning-based object detection becoming a crucial technology in autonomous driving systems. Object detection models are typically categorized into two main approaches: single-stage and two-stage methods. The utilization of the Region Proposal Network (RPN) in two-stage object detectors like the Fast R-CNN (region convolutional neural network) [
7,
8] has been demonstrated to efficiently extract object details while integrating global information, thereby mitigating the effects of environmental conditions and achieving high-precision detection. However, slow inference speed hinders its application in real-time systems due to heavy computational costs in RPN. In contrast, single-stage detectors (YOLO [
9,
10,
11] and SSD [
12]) have extremely fast detection speeds in adverse conditions that formulate object detection as a simple regression problem, which refers to the simultaneous computation of bbox regression and classification on convolutional feature maps. Wang et al. [
13] used an enhanced YOLOv4 model for road object detection, incorporating SENet modules into the PANet structure to enhance channel attention and improve the overall detection performance of the model. Cao et al. [
14] developed an improved YOLOv5 model and introduced a multi-scale small object detection structure to enhance the sensitivity to dense small objects. Mao et al. [
15] improved YOLOv7 by introducing the SIoU [
16] loss function, which takes into account the angles, distances, and shapes between targets, resulting in better detection performance. Wang et al. [
17] proposed an improved YOLOv8 model to address the challenges of long-range detection in driving scenarios. Their approach included structural reparameterization techniques and the Bidirectional Feature Pyramid Network (BiFPN) [
18], which effectively reduces the model’s complexity and computational load while maintaining high performance. Undoubtedly, these studies provide significant assistance for autonomous driving. However, these methods face challenges in adverse driving conditions. Specifically, in environments such as rain, snow, fog, and sandstorms, image clarity is reduced and object features and details are obscured, leading to decreased detection accuracy and significantly limiting their widespread application in autonomous driving.
In recent years, there has been growing interest in enhancing object detection capabilities for autonomous driving applications. Numerous methods have been suggested by scholars to achieve improvement in the performance of detectors. Zeller et al. [
19] utilized Radar point clouds in lieu of images and introduced a full-resolution Backbone network to enable instance segmentation in challenging weather conditions. Piroli et al. [
20] employed deep learning networks to preprocess LiDAR point cloud data for improved robustness of object detection. Clearly, they have shown significant improvements in both speed and accuracy compared to the traditional algorithm. However, the substantial volume of data in radar and LiDAR point clouds poses a hindrance to real-time detection speed when compared to images. Similarly, while two-stage object detectors have high detection accuracy, their speed falls short of meeting the requirements for autonomous driving applications, hindering their further development in the field of autonomous driving [
21]. Wibowo et al. [
22] developed YOLOv7 MOD to achieve real-time detection in congested traffic, yet its accuracy diminishes significantly in snowy weather conditions. Hassaballah et al. [
5] proposed a visibility restoration scheme to enhance image quality and object detection performance in adverse weather conditions, but this requires additional image processing prior to network input. To balance speed and accuracy, many industries use dilated convolutions to address this issue. Zhang et al. [
23] proposed LS-YOLO, which improves the decoupled head using dilated convolutions, enhancing accuracy in landslide detection. Deng et al. [
24] designed a weighted feature pyramid using dilated convolutions to improve YOLOv7’s performance in detecting maritime targets under foggy conditions. Qian et al. [
25] improved YOLOv4’s architecture with dilated convolutions and skip connections, enhancing polyp detection accuracy in medical imaging. Chen et al. [
26] used multiple parallel dilated convolutions to create convolutional kernels with varying receptive fields to capture multi-scale object information. Despite these advancements, the Atrous Spatial Pyramid Pooling (ASPP) architecture did not fully utilize HDC to optimize the receptive field. However, the use of HDC during the sampling process has gained prominence in object detection applications, as it helps to mitigate information loss [
27,
28,
29]. This architecture enhances the model’s ability to preserve spatial resolution and capture intricate details across different scales, which is particularly beneficial in challenging environments. Therefore, Tian et al. [
30] also attempted to integrate dilated convolutions and the Self-Attention Module (SAM) into a single-stage object detector to balance detection accuracy and speed, but did not account for driving under adverse weather conditions. Despite the existence of various techniques to optimize detector performance, there has been a lack of research specifically targeting the improvement of autonomous vehicle object detectors in adverse weather conditions using image algorithms. Currently, algorithms for driving in adverse weather conditions face challenges in balancing detection accuracy and speed, hindering the widespread adoption of vehicle target detectors. Therefore, enhancing the detection accuracy of single-stage object detectors holds promise for advancing the field of autonomous driving.
In response to the challenges posed by detecting objects in adverse driving conditions, this study drew inspiration from the rational utilization of object details in the two-stage model to aggregate global information. The architecture of the original YOLOv7-Tiny model was improved to enhance its performance under such conditions. The main improvements are as follows: Firstly, SPP was replaced with SPPF in the YOLOv7-Tiny network, leading to the creation of SPPFCSPC, which resulted in speed improvements. Additionally, HDC was employed to expand the RF of the SPPCSPC and ELAN-T structures. Next, the EMA mechanism was integrated into the feature fusion process to combine spatial and channel information, thereby enhancing the effectiveness of feature fusion. Furthermore, Wise-IoUv1 was implemented in place of CIoU to improve convergence speed and bbox positioning accuracy. Finally, various evaluation experiments were conducted on the BDD100K and DAWN datasets. The model showed a 3.4% increase in mAP0.5 in the BDD100K dataset while achieving real-time detection at 142.86 FPS. In the DAWN dataset, the mAP0.5 value increased by 6.3%. Compared to other state-of-the-art detectors, DAN-YOLO achieved a favorable trade-off between detection accuracy and speed under adverse weather conditions, making it well-suited for applications in autonomous driving.
3. DAN-YOLO Network Model Architecture
The network architecture of the enhanced YOLOv7-tiny model, named DAN-YOLO, is shown in
Figure 4. Compared to the original YOLOv7-tiny network (
Figure 1), several improvements have been made, as indicated by the modules filled with green, blue, and brown boxes in
Figure 4. Firstly, the improved SPPFCSPC-E structure is employed in the Backbone network to achieve the lossless capture of different receptive field feature information. Subsequently, the improved DAN structures supplant ELAN-T modules in the Neck network to effectively aggregate global information. The EMA mechanism is then utilized prior to feature fusion to enhance feature representation during the Neck fusion process layers. Finally, the network utilizes Wise-IoUv1 loss [
34] to train the model, regardless of the original loss function. In the following subsections, a detailed description of each module will be provided.
3.1. SPPFCSPC-E Structure and DAN Structure
Figure 5 illustrates SPPF and SPPFCSPC, which are based on SPP, but offer significantly improved speed. SPPF sequentially produces the output of the upper branch by applying three 5 × 5 MP operations on the features. By concatenating the receptive fields from each MP layer, SPPF effectively reduces the computational cost associated with pooling scales. This structure improves detection speed while preserving the same RF as before the enhancement.
Figure 1 and
Figure 6 illustrate the original SPPCSPC structure and SPPFCSPC-E. Initially, SPP [
35] in SPPCSPC was replaced with SPPF to create SPPFCSPC, which improves speed while preserving the same RF. Additionally, the serial structure of SPPF is advantageous for stacking HDC, effectively maintaining internal data structures and spatial hierarchy, as shown in
Figure 5. In the SPPF structure of SPPFCSPC, the repeated use of the MP in multiscale fusion has raised concerns about the potential loss of internal data structure and spatial hierarchical information, which is critical for objects that have become unclear due to environmental influences. Therefore, this paper replaced the MP layer in the SPPF module with an HDC layer, incorporating a dilation factor of [1, 2, 5], as depicted in
Figure 6. The modified SPPFCSPC-E accelerates network inference speed while expanding the RF, thereby enhancing detection in adverse weather conditions. By replacing MP with HDC, the network’s ability to accurately capture features across different receptive fields is improved, leading to more precise localization of detailed target objects.
Figure 1 and
Figure 7 illustrate the original DAN structure and ELAN-T. The ELAN-T module in YOLOv7-Tiny is recognized for its simple and flexible structure. This module incorporates multi-level feature maps using the concat layer, which improves the network’s capacity to learn from input features and accomplish deep feature utilization objectives. However, the limitation of insufficient feature utilization in ELAN-T, which employs regular convolutions with relatively small RF, restricts its ability to effectively integrate global information while capturing object details. To address this issue, this study introduced a DAN structure aimed at expanding RF while maintaining network flexibility. Firstly, a 3 × 3 convolution layer was added to the fifth layer of the ELAN-T structure, after which the dilation factors of the third, fourth, and fifth layers were set to 1, 2, and 5, respectively. Given that the Neck plays a crucial role in enhancing the spatial detail control of global information, the ELAN-T structure in the Neck was replaced with the DAN structure to effectively aggregate global information. Both methods can significantly reduce the impact of adverse weather conditions and improve detection accuracy.
3.2. Efficient Multi-Scale Attention Mechanism
In the network, initial feature extraction has been accomplished within the Backbone, with subsequent feature integration carried out in the Neck to enhance the comprehensiveness of object features. Nevertheless, the features derived from the Backbone fail to fully account for spatial and channel information. During the fusion process, there is a lack of enhancement in the representational capacity of spatial and channel information, and the pixel-level correlation between features remains unimproved. Consequently, this paper incorporated the EMA attention mechanism to address the features extracted from the Backbone prior to feature fusion. This enables the network to emphasize spatial and channel information within features, leading to a more comprehensive understanding of contextual semantic relationships. Then, enhancing the feature representation during the Neck fusion process lays the foundation for the integration of deep features in subsequent stages.
The EMA mechanism [
36], known for its efficiency as a multi-scale attention mechanism, distinguishes itself from conventional channel and spatial attention mechanisms by its flexible and lightweight structure. This mechanism reduces network depth through parallel substructures, requiring less computational cost to achieve the same effect on a feature map without diminishing channel dimensions. Drawing inspiration from the Coordinate Attention (CA) mechanism [
37], the EMA mechanism introduces a multi-scale parallel sub-network to address dependencies within network modules. As illustrated in
Figure 8, the input features are first divided into G sub-features along the channel dimension (X ∈ R
C×h×w), with each sub-feature represented as X = [X
0, X
i, …, X
G−1]. Feature extraction for each sub-feature X
i is performed using learned weights to capture key information. These sub-features are then fed into three parallel subnetworks, enabling dynamic weight allocation to the grouped feature maps. In the parallel subnetworks, the first two sets, X Avg Pool and Y Avg Pool, perform average pooling in the horizontal and vertical directions, respectively, and are referred to as pooling networks. The third subnetwork applies a 3 × 3 convolution and is called a 3 × 3 network. The pooling network is defined as follows:
represents pooling along the horizontal direction, and
represents pooling along the vertical direction.
In the pooling network, the batch dimensions (C//G × h × w) of the input are combined through feature concatenation using the pooling network described earlier. This approach avoids standard convolution downsampling and incorporates direction-related positional information, effectively utilizing spatial information. The sigmoid function is then applied to activate and produce outputs in both horizontal and vertical directions. These two parallel networks perform the first cross-channel interaction of features through multiplication, as defined in Equation (6), where X
i represents the i-th feature in the G sub-features, while X
w and X
h are the convolution results in the horizontal and vertical directions, respectively.
In the 3 × 3 network, 3 × 3 convolution and global average pooling are used for encoding, extracting local key information within the channel, and expanding the spatial feature map. The results from the three parallel networks are combined to achieve the fusion of overall cross-spatial feature information. Specifically, there are two stages of cross-spatial feature information fusion between the pooling networks and the 3 × 3 network. The first stage involves encoding global information through 2D global average pooling in one output of the pooling network, which is then matrix-multiplied with the output of the 3 × 3 network. This operation is represented as , where N1×1 is the output of the pooling network and N3×3 is the output of the 3 × 3 network.
The second time is to multiply the global average pooling result of the 3 × 3 network output with the result of the pooling network in a matrix, expressed as
. Finally, the weight values of the cross-spatial feature information are aggregated to obtain pixel-level global key information features, achieving the purpose of emphasizing global contextual information [
38]. The experiments in this paper demonstrate the effectiveness of this method.
3.3. Wise-IoU Loss Function
In adverse driving conditions, the model’s generalization ability and localization accuracy play a crucial role. For example, in heavy snow, a white car covered by snow while driving may be inaccurately localized and misclassified as part of the road, leading to reduced generalization. Therefore, high-quality fitting during training is necessary, which requires calculating the bbox regression loss based on the geometric elements of the predicted box and the ground truth box. During data fitting, the loss function should appropriately reduce the focus on the center point distance and weaken the geometric penalty when the anchor box and the target box (high-quality anchor box) overlap well, thus reducing training interference and effectively improving the model’s generalization ability and localization accuracy. However, the CIoU loss function only considers three key geometric factors: overlap area, center distance, and aspect ratio, without distinguishing between ordinary anchor boxes and high-quality anchor boxes [
39]. To enhance the model’s generalization ability and localization accuracy, this paper replaced CIoU in the model with Wise-IoUv1.
Wise-IoU is known as a bbox regression loss function with a dynamic non-monotonic focusing mechanism, and Wise-IoUv1 is its first-generation version. In traditional IoU, the penalty for fitting bboxs with high-quality samples differs from that for low-quality samples. However, geometric metrics such as distance and aspect ratio often impose higher penalties on low-quality samples, which can potentially hinder the model’s generalization ability and localization accuracy. To address this issue, Wise-IoUv1 reduces the geometric penalty for overlapping anchor boxes and target boxes (representing high-quality samples), constructs distance attention, successfully reduces the emphasis on low-quality samples, minimizes training interference, and enhances the focus on typical samples, ultimately improving the model’s generalization ability and localization accuracy. The formula for Wise-IoUv1 is as follows:
where x and y represent the predicted bounding box’s center coordinates, x
gt and y
gt represent the center coordinates of the ground truth bounding box, W
g and H
g represent the width and height of the minimum bounding box enclosing the predicted and ground truth boxes, and the superscript * represent the computation separation.
denotes the intersection over Union (IoU) ratio of the predicted box and the ground truth box overlap area.
4. Experimentation Results and Discussion
To validate the effectiveness of the proposed method, object detection experiments and tests were performed using PyTorch based on Python 3.8 on a Win10-64bit operating system equipped with 24 GB NVIDIA RTX3090 GPU memory. The training datasets were sourced from the BDD100K dataset [
40] and the DAWN dataset [
41], for which detailed descriptions can be found in
Table 2, where the BDD100K dataset mainly selected images captured in snowy driving conditions and used data augmentation techniques to add 382 images.
4.1. Basic Testing and Implementation Details
The training configuration includes parameters such as epoch, batch size, image size, and workers, set at 200, 16, 640, and 8, respectively. Hyperparameters are established based on the official default file, and the cosine annealing learning rate scheduling algorithm is employed for updates. Additionally, both the YOLOv7-Tiny and DAN-YOLO models were trained from scratch to ensure impartiality in the evaluation process.
This paper evaluated the effectiveness of improving object detection algorithms by selecting Precision, Recall, Mean Average Precision (mAP), Frames Per Second (FPS), Params, and FLOPs as evaluation metrics. Predicted boxes were categorized into TP, TN, FP, and FN based on IoU, where T or F indicated whether a sample was correctly classified, and P or N indicated whether a sample was predicted as positive or negative. The calculations for Precision and Recall are as follows:
In the formula for mAP, n and AP represent the number of detection categories and the area under the Precision–Recall (P–R) curve for a specific category, respectively. The FPS formula calculates the frames per second by dividing the number of image frames by the time taken for processing.
4.2. Performance Analysis of DAN-YOLO
The training and validation were performed using an NVIDIA RTX3090 to demonstrate the effectiveness of the DAN-YOLO model in edge computing. And the study compared the proposed object detector with various state-of-the-art used ones, tracking computational complexity in terms of floating-point operations (FLOPs) and measuring the processing time per image to assess efficiency.
Table 3 details the detection performance of the algorithms on the BDD100K benchmark; all models follow official settings and are trained from scratch. From the perspective of mAP0.5,
Table 3 shows that compared to DAN-YOLO, other models have lower mAP0.5 values, indicating that this model has better detection capabilities. Notably, DAN-YOLO (15.6 G) has fewer computational parameters than YOLOv5s (16.3 G), YOLOv5m (50.3 G), YOLOv8s (28.4 G), and YOLOv10s (21.6 G), yet its mAP0.5 is higher by 5%, 2.2%, 0.5%, and 1.4%, respectively. In terms of real-time performance, DAN-YOLO achieves 142.86 FPS, fully meeting the real-time requirement (exceeding 60 FPS), demonstrating that DAN-YOLO excels in balancing detection accuracy and lightweight design. Compared to YOLOv7-Tiny (mAP0.5 of 48.6%), the mAP0.5 improved by 3.4%, with only an 18.87% sacrifice in FPS. In terms of precision for each category, compared to other models, DAN-YOLO shows improvements in the car, person, traffic sign, and traffic light categories, achieving 71.6%, 45.3%, 44.4%, and 46.6%, respectively, indicating that the model has stronger compatibility with objects in autonomous driving scenarios.
Additionally,
Table 4 presents the evaluation results of the algorithm on the DAWN validation set. From the perspective of mAP0.5, except for YOLOv8s, other models have lower mAP0.5, indicating that DAN-YOLO (52.8%) has stronger detection capabilities. Although YOLOv8s secured first place in the DAWN benchmark object detection challenge with an mAP0.5 of 54.9%, which is 2.1% higher than DAN-YOLO’s 52.8%, it came at the cost of an additional 12.8 G in computational load. Compared to YOLOv7-Tiny, DAN-YOLO made significant progress, specifically improving mAP0.5 by 6.3% with only a 2.6 G increase in FLOPs. This again demonstrates that DAN-YOLO excels in balancing detection accuracy and lightweight design.
Moreover,
Table 5 shares the evaluation results of transfer learning on the DAWN dataset, using the weights of DAN-YOLO and YOLOv7-Tiny trained on the BDD100K dataset as pre-trained weights. In the weight column, “√” indicates the use of the corresponding weights, while “×” indicates that the corresponding weights were not used. The results indicate that, when the pre-trained weights are used, the mAP0.5 and mAP0.5:0.95 of both models improve by 4.7%, 2.5%, and 1.7%, 4.2%, respectively. Moreover, DAN-YOLO outperforms YOLOv7-Tiny in terms of mAP0.5 and mAP0.5:0.95 by 3.3% and 1.6%, respectively, based on the outcomes derived from the pre-trained weights. This demonstrates that the performance and generalization ability of the DAN-YOLO model are superior to those of YOLOv7-Tiny.
Consequently, experimental results validate the superiority and competitiveness of this paper’s proposed DAN-YOLO, and better trade-off between accuracy and inference speed makes it more suitable for autonomous driving in adverse weather conditions.
4.3. Ablation Study
This paper included ablation studies on the DAWN dataset to explore the contributions of different components in the used structure, including SPPFCSPC-E, DAN, EMA, and the Wise-IoUv1 loss function.
Table 6 indicates that applying the methods from this study to YOLOv7-Tiny resulted in improvements of 2.6%, 3.6%, 3.6%, and 2.9% in mAP0.5, respectively, with corresponding improvements in the mAP0.5 for each class. This indicates the effectiveness of deploying individual modules. Furthermore, assigning all the proposed structural enhancements to YOLOv7-Tiny resulted in increases of 6.3% in mAP0.5 and 11.3% in precision, with parameters and FLOPs increasing by only 1.98M (Million) and 2.6 G, respectively, achieving an effective trade-off without excessive growth. Notably, the motorcycle+bicycle class, which only accounted for 1.36% of the total annotations, achieved a significant improvement of 30.9% in mAP0.5, indicating that the proposed method for this model shows stronger compatibility in adverse conditions and is more conducive to unpredictable autonomous driving scenarios in adverse conditions.
4.4. Visualization Results
To better understand the detection performance of DAN-YOLO, this paper provides several visualizations, including P-R curves, Confusion Matrix Difference Heatmaps, and object detection results.
Figure 9 provided in this paper shows the P–R curves on two datasets, which more intuitively display the recall and precision performance of the detector. As shown in
Figure 9a, the P–R curves of DAN-YOLO have completely enclosed the P–R curves of YOLOv7-Tiny. Also, the improved network has a higher value at the Balance Error Rate (BEP), as shown by the intersection points of the two curves with the x = y line in
Figure 9b. Therefore, the improved network model can achieve better results.
To visually demonstrate the performance improvement of the DAN-YOLO model, this paper provides the confusion matrix differential heatmap. Specifically, DAN-YOLO and YOLOv7-Tiny were benchmarked on the BDD100K and DAWN datasets, generating two confusion matrices containing the data, respectively. By subtracting the data in YOLOv7-Tiny’s matrix from that in DAN-YOLO’s matrix, two new differential heatmaps were produced, as shown in
Figure 10. In the figure, the blue areas indicate positive values, and the orange areas indicate negative values. It can be observed that the improved DAN-YOLO achieves performance gains in each category while reducing the impact of background and other categories. In
Figure 10a, the probability of each category being correctly identified increases by 0.04, 0.05, 0.06, and 0.09, respectively, while the probability of misclassifying targets as background decreases by 0.04, 0.04, 0.06, and 0.09. Notably, in
Figure 10b, the probability of correctly identifying motorcycle+bicycle increases by 0.3, while the probabilities of misclassifying motorcycle+bicycle and bus as background decrease by 0.22 and 0.19, respectively. Ultimately, these findings validate the effectiveness of the enhanced DAN-YOLO model, highlighting its exceptional performance in autonomous driving scenarios.
The paper presents the visualization of detection outcomes from YOLOv7-Tiny and DAN-YOLO on the test dataset, depicted in
Figure 11, where column one represents the detection results of YOLOv7-Tiny, and column two represents the detection results of DAN-YOLO. As observed, when there is image clarity decreases and object occlusion (driving in adverse conditions), the prediction confidence of YOLOv7-Tiny decreases and starts to generate numerous results of incorrect, false positive, and missed detections. On the contrary, following structural enhancements, the DAN-YOLO with improved feature extraction capabilities demonstrates outstanding precision in object localization, accuracy, and detection rates. It excels in identifying challenging samples that YOLOv7-Tiny may overlook, further underscoring the effectiveness of the proposed DAN-YOLO model outlined in the paper.