LLE-STD: Traffic Sign Detection Method Based on Low-Light Image Enhancement and Small Target Detection

Abstract

With the continuous development of autonomous driving, traffic sign detection, as an essential subtask, has witnessed constant updates in corresponding technologies. Currently, traffic sign detection primarily confronts challenges such as the small size of detection targets and the complexity of detection scenarios. This paper focuses on detecting small traffic signs in low-light scenarios. To address these issues, this paper proposes a traffic sign detection method that integrates low-light image enhancement with small target detection, namely, LLE-STD. This method comprises two stages: low-light image enhancement and small target detection. Based on classic baseline models, we tailor the model structures by considering the requirements of lightweight traffic sign detection models and their adaptability to varying image qualities. The two stages are then coupled to form an end-to-end processing procedure. During experiments, we validate the performance of low-light image enhancement small target detection, and adaptability to images of different qualities using the public datasets GTSDB, TT-100K, and GLARE. Compared to classic models, LLE-STD demonstrates significant advantages. For example, the mAP results tested on the GLARE dataset show that LLE-STD outperforms RetinaNet by approximately 15%. This research can facilitate the practical application of deep learning-based intelligent methods in the field of autonomous driving.

1. Introduction

Autonomous driving is a crucial component in achieving high-level intelligent transportation systems. Among its various aspects, traffic sign detection plays a significant role in guiding vehicles to travel in an orderly manner, enhancing driving safety, and providing vital assistance for seamless vehicle control [1]. With the continuous advancement of deep learning, traffic sign detection research has also shifted towards data-driven methods that automatically extract features, attracting widespread attention from scholars [2].

However, the small size and complex backgrounds of traffic signs pose significant challenges in processing. In computer vision, small objects typically refer to those occupying minimal regions in an image, lacking the rich texture, color, and other detailed features of regular-sized objects, thus making detection more difficult. The MS COCO dataset [3] defines small objects as those with a resolution less than 32 px × 32 px. The TT-100K dataset [4], curated by Wei et al., is the most commonly used domestic traffic sign detection dataset, where traffic signs exhibit clear characteristics of small objects. As a result, small object detection has emerged as the primary challenge in traffic sign detection.

Small object detection algorithms can be categorized into two main types: single-stage and two-stage algorithms. The two-stage approach is characterized by high detection accuracy but relatively slow speed. Notable examples of two-stage small object detection algorithms include R-CNN [5], Fast R-CNN [6], and Faster R-CNN [7]. In 2017, Lin et al. proposed the feature pyramid network (FPN) algorithm [8], which fuses high-level semantic feature maps with low-level semantic feature maps. This fusion ensures that the output feature maps contain both high-level semantic information and information about small objects, making them suitable for small object detection. In 2018, Cai et al. introduced the Cascade R-CNN algorithm [9]. Moreover, Zhao proposed a multi-stage structure that progressively raises the intersection over union (IoU) threshold to obtain a high-quality detector for small object detection. The single-stage approach aims to balance detection speed and accuracy. Representative single-stage small object detection algorithms include the you only look once (YOLO) series proposed by J. Redmon et al. [10,11] and the single-shot multiBox detector (SSD) algorithm by W. Liu et al. [12]. However, due to the smaller size and limited information content of small objects, combined with the constraints of the YOLO network architecture, the original YOLO and YOLO 9000 algorithms are not suitable for small object detection tasks. In 2018, J. Redmon addressed this issue with YOLOv3 [13], which, through structural reconfiguration, successfully applies the YOLO series to small object detection while maintaining high detection accuracy. In 2020, A. Bochkovskiy et al. introduced YOLOv4 [14], which significantly improves detection accuracy while maintaining the same speed as YOLOv3. YOLOv4 achieves mAP of 43.5% on the COCO dataset. Additionally, fully convolutional one-stage object detection (FCOS) [15], based on the concept of fully convolutional networks (FCN) [16], is a single-stage anchor-free detector that uses pixel-level predictions to solve the object detection problem. By avoiding anchor box calculations and region of interest (RoI) generation, FCOS is also frequently used for small object detection tasks.

In nighttime conditions, the lack of sufficient lighting can lead to low visibility and color distortion in captured dark images, significantly impacting traffic sign detection. To address this issue, deep learning has emerged as the dominant approach for low-light image enhancement (LLE) [17]. In this paper, we propose to integrate lightweight low-light image enhancement techniques with small object detection methods specifically designed for traffic sign detection. The goal is to improve the detection of small traffic signs in complex low-light scenarios, thereby enhancing the overall performance of traffic sign detection systems under challenging nighttime conditions.

Based on the above, the main contributions of this paper are as follows:

(1)

A cascaded model architecture combining LLE and small object detection is proposed to improve the detection performance of small traffic signs in complex low-light scenarios.

(2)

By addressing the real-time requirements of autonomous driving scenarios, especially for traffic sign detection tasks, this paper investigates the use of blueprint separable convolutions (BSConv) [18] as a means to reduce model complexity by replacing standard convolutions.

(3)

To address the issue of insufficient detection accuracy for small traffic signs, this paper introduces attention mechanisms and modifies activation functions during the object detection stage of the LLE-STD model. These enhancements effectively prevent feature loss and improve the overall detection accuracy.

In summary, aiming at the problems of small detection target size and complex detection scenarios faced by the traffic sign detection task, this paper proposes the LLE-STD model to balance the effectiveness and reliability of detection as much as possible.

The remainder of this paper is organized as follows: The related works in small object detection and LLE models are discussed in Section 2. The brief description of the proposed approach reviewed in Section 3. In Section 4, we show some assessment criteria used to evaluate the effectiveness of the proposed method. The results of our experiments are illustrated and compared with other state-of-the-art methods in Section 5. Finally, in Section 6 and Section 7, discussion and the conclusions are drawn.

2. Related Works

In intelligent driving systems, traffic sign detection plays a critical role in supporting real-time decision-making. However, traffic signs are typically smaller than other objects in natural scenes, and the actual traffic environment is complex and highly variable. Therefore, traffic sign detection faces challenges posed by small objects and complex scenes. This paper specifically focuses on the issue of detecting small traffic signs under low-light conditions.

2.1. LLE Models

The first CNN-based LLE model [19] utilizes an autoencoder to simultaneously learn denoising and enhancement. Inspired by the Retinex theory, several LLE models have been proposed [20,21,22,23,24], which typically decompose the low-light image input into reflectance and illumination maps, then adjust the illumination map to enhance the image intensity. Most of these methods integrate denoising modules into the reflectance map to suppress noise in the enhanced results. For instance, Zheng et al. [25] propose an unfolded total variation network to estimate the noise level for LLE. To improve generalization capabilities, unsupervised methods have also been introduced. The Enlightenment GAN [26], an attention-based U-Net, is trained using adversarial losses. Zero-DCE [27] and Zero-DCE++ [28] approach image enhancement as an image-specific curve estimation task, demonstrating robust enhancement effects. Given that autonomous driving systems require real-time information acquisition, processing, and decision-making, traffic sign detection must also satisfy real-time requirements. Consequently, optimizing network structures to reduce computational complexity is crucial.

The network architecture of Zero-DCE [27] is illustrated in Figure 1. It mainly comprises two parts: the deep curve estimation network (DEC-Net) and the light enhancement curve (LE-curve). Different from traditional supervised learning models, Zero-DCE learns a high-order curve through DEC-Net and applies this curve to perform a nonlinear dynamic range adjustment for each pixel in the input image. This adjustment expands the brightness range and increases the contrast, thereby enhancing the low-light image.

The BCE-Net takes low-light images as input and outputs a pixel-wise mapping of high-order curve parameters. This model consists of seven convolutional layers, with the first six layers each containing 32 convolutional kernels of size 3 × 3 and a stride of 1, followed by ReLU activation functions. The final layer contains 24 convolutional kernels of size 3 × 3 and a stride of 1, using a Tanh activation function. As a result, the model’s output, the curve parameter mapping, comprises 24 channels. These channels can be split into groups based on the RGB color channels, yielding eight sets of iterative curve parameter mappings. Each set of parameters corresponds to a different stage or iteration in the curve estimation process, enabling a more refined and adaptive adjustment of the image’s brightness and contrast.

In addition, the design of LE-curve includes the following:

(1)

The intensity value of each pixel in the enhanced image should be within the range of [0, 1] to avoid information loss caused by overflow truncation.

(2)

The LE-curve needs to maintain monotonicity to ensure stable contrast between adjacent pixels.

(3)

The gradient form of the LE-curve should be as simple and differentiable as possible.

LE-curve can be designed as follows:

$$LE\left(I;\alpha \right)=I+\alpha \cdot I\cdot \left(1-I\right)$$

(1)

where I is the input image; $LE\left(I;\alpha \right)$ is enhanced image; $\alpha$ is the parameter of curve learned by BCE-Net, whose range is [0, 1]. Performing multiple iterations on the LE-curve results in the following:

$$L{E}_n\left(x\right)=L{E}_{n-1}\left(x\right)+{\mathcal{A}}_n\left(x\right)\cdot L{E}_{n-1}\left(x\right)\cdot \left(1-L{E}_{n-1}\left(x\right)\right)$$

(2)

where n is the number of iterations. According to the structure of BCE-Net, n = 8. $\mathcal{A}$ is the parameter mapping of the same size as the given image. Applying the LE-curve separately to the RGB color channels can better preserve the inherent colors and reduce the risk of oversaturation.

2.2. Small Traffic Sign Detection Methods

Focusing specifically on traffic sign detection tasks, Huang et al. [29] employ Faster R-CNN as the detection model and utilized generative adversarial networks (GANs) for data augmentation. Zhang et al. [30] combine cascade R-CNN with sample balancing to detect traffic signs, achieving accuracies of 99.7% and 98.7% on the CCTSDB [31] and GTSDB [32] datasets, respectively. Zhao et al. [33] utilize the Libra R-CNN algorithm in conjunction with a balanced feature pyramid to detect traffic signs, reporting a detection accuracy of 77.3% in their experiments. The detection results of other typical traffic signs are shown in

Table 1. The existing traffic sign detection methods.

Method	Dataset	Accuracy (%)	Time
Huang et al. [29]	TT-100K	89.65	-
Zhang et al. [30]	CCTSDB/GTSDB	99.7/98.7	-
Zhao et al. [33]	MS COCO	77.3	-
Aghdam et al. [34]	GTSDB	99.89	26.506 ms
Chen et al. [35]	STSD	80.85	-
Houben et al. [36]
HOG + LDA	GTSDB	69.2	-
Hough-like	GTSDB	65.1	-
Viola-Jones	GTSDB	67.3	-
Liu et al. [37]	GTSDB	93.5	-
Liu et al. [38]	GTSDB	98.57	192 ms
Zang et al. [39]	GTSDB	96.50	35 ms

. Two-stage traffic sign detection algorithms generally exhibit higher detection accuracy than one-stage algorithms but are slower in speed. Currently, few studies explicitly address the challenge of small object detection as a crucial issue in traffic sign detection. Instead, the majority of research focuses on applying existing classical models. This paper, however, takes a different approach by enhancing the representation of small object features based on classic small object detection models as a baseline, aiming to improve the detection accuracy of traffic signs. By doing so, we aim to contribute to the development of more effective and efficient traffic sign detection systems.

Here, FCOS [15] is a single-stage anchor-free detection model proposed based on the idea of FCN. It primarily addresses the problem of object detection through pixel-level prediction, thereby avoiding the need for anchor box computations and RoI generation. This algorithm first feeds a preprocessed image into a backbone network for feature extraction, resulting in an initial feature layer. Then, this feature layer is input into a feature pyramid network (FPN) for multi-level predictions. Finally, the detection head outputs the detection results. The network structure is illustrated in Figure 2.

As depicted in Figure 2, FCOS processes the input image through a backbone network for feature extraction, yielding feature maps C3, C4, and C5. These feature maps are then passed through 1 × 1 convolutions to generate P3, P4, and P5 feature maps, which are connected in a top-down fashion. Additionally, P5 undergoes two downsampling steps to obtain P6 and P7 feature maps. Finally, each of these five feature maps with different scales is processed by a detection head, leveraging shared feature information to achieve multi-level object detection. For each pixel point (x, y) on the feature maps, it can be mapped back to a corresponding pixel point (x′, y′) in the original input image using the following formula:

$$\left(x^{\prime },y^{\prime }\right)=\left(⌊{\displaystyle \frac{s}{2}}⌋+x\cdot s,⌊{\displaystyle \frac{s}{2}}⌋+y\cdot s\right)$$

(3)

where s is step; $⌊\cdot ⌋$ is round down. If the pixel (x, y) falls within the target anchor box and corresponds to a specific category, then that pixel is considered a positive sample; otherwise, it is considered a negative sample. The regression status of each pixel can be expressed as follows:

$$l^{*}=x-x_0^{\left(i\right)},t^{*}=y-y_0^{\left(i\right)},r^{*}=x_1^{\left(i\right)}-x,b^{*}=y_1^{\left(i\right)}-y$$

(4)

where (x₀⁽ⁱ⁾, y₀⁽ⁱ⁾) and (x₁⁽ⁱ⁾, y₁⁽ⁱ⁾) are the coordinates of the pixels in the upper left and lower right corners of the bounding box, respectively. $l^{*}$, $t^{*}$, $r^{*}$, and $b^{*}$ represent the distance of the pixel to the top, bottom, left, and right sides of the bounding box, respectively. In the hypothesis that a feature point on the feature map falls within a target anchor box, this feature point is deemed as an ambiguous sample. To determine which class this ambiguous sample belongs to, multi-level predictions are made through FPN, and the prediction box with the smallest area is selected as its regression target. When a feature point in feature map satisfies the condition in Equation (5), the sample is called a negative sample and does not require regression prediction.

$$\max \left(l^{*},t^{*},r^{*},b^{*}\right)>m_i\hspace{1em}or\hspace{1em}\max \left(l^{*},t^{*},r^{*},b^{*}\right)<m_{i-1}$$

(5)

where m_i is the maximum regression distance of the ith effective feature layer. In FCOS, we set m₂~m₇ as 0, 64, 128, 256, and ∞. If there are two target anchor boxes that meet the requirements, the one with the smallest area is selected for regression prediction. In addition, since the FCOS algorithm performs pixel-level regression operations, it generates a large number of predicted boxes, including some low-quality predicted boxes generated by pixels far from the target center. These predicted boxes will have a certain impact on the detection performance. To address this issue, the FCOS algorithm introduces a “Center-ness” to suppress these low-quality bounding boxes. The definition of Center-ness is:

$$Center-ness=\sqrt{{\displaystyle \frac{\min \left(l^{*},r^{*}\right)}{\max \left(l^{*},r^{*}\right)}}\times {\displaystyle \frac{\min \left(t^{*},b^{*}\right)}{\max \left(t^{*},b^{*}\right)}}}$$

(6)

where, the range of Center-ness is [0, 1]. It serves as a single-layer branch that runs in parallel with the classification branch.

3. Methodology

This section introduces LLE-STD, including its overall structure, core module structure, and the design of its loss function.

3.1. The Overall Structure

The overall structure of LLE-STD is shown in Figure 3. It can be intuitively seen that LLE-STD consists of two stages, namely, the low-light image enhancement stage and the small traffic sign detection stage. The baseline models for these two stages are Zero-DCE and FCOS, respectively. The obvious differences between it and the classic model are as follows:

(1)

The two stages in the LLE-STD model are coupled and connected, enabling end-to-end learning. This means that during the inference stage, when a low-light image is input into LLE-STD, it can directly output the traffic sign detection results corresponding to the enhanced image. There is no need to separately perform low-light image enhancement and traffic sign detection tasks.

(2)

Given the requirement for real-time processing in autonomous driving scenarios, especially for traffic sign detection tasks, the proposed method should aim for high computational efficiency and a small model size. Therefore, in the low-light image enhancement stage of the LLE-STD model, we utilize BSConv [18] to replace standard convolutions. This modification reduces the number of multiplication operations in the model, thereby lowering the computational complexity.

(3)

To address the issue of insufficient detection accuracy for small traffic signs, we incorporate an attention mechanism into the object detection stage of the LLE-STD model. Additionally, we replace the classic ReLU and Sigmoid activation functions with the Swish activation function. This effectively avoids the loss of features and enhances the accuracy of object detection.

3.2. The Improved DCE-Net Model

As can be seen from Figure 3, the low-light image enhancement stage in LLE-STD uses Zero-DCE as the baseline model, but with improvements made to the DCE-Net component by replacing the classic convolutional processing with BSConv. The introduction of BSConv [18] is based on the theoretical foundation of strong correlation in the depth dimension. Through extensive experiments, the authors have verified that the proposed BSConv can significantly enhance the performance of MobileNet and other architectures based on depthwise separable convolutions, without introducing additional complexity. For fine-grained problems, the proposed method achieves a 13.7% performance improvement. In the ImageNet classification task, BSConv achieves a 9.5% performance improvement when “plugged and played” with ResNet.

The first layer of BSConv is a pointwise convolution, while the second layer is a depthwise convolution per channel. This order effectively exploits the correlations within the convolution kernels. The pointwise convolution uses a filter size of 1 × 1, which performs an element-wise weighting operation on each pixel in the input feature map. During the pointwise convolution, the number of input feature map channels needs to be the same as the number of channels in the convolution kernel, and the number of output feature map channels is determined by the number of convolution kernels. The depthwise convolution process per channel requires that the number of input feature map channels be the same as the number of convolution kernels, which also determines the number of output feature map channels.

After applying a standard convolution with a stride of 1 to the input feature map $U\in R^{M\times Y\times X}$, the resulting feature map is $V\in R^{N\times Y\times X}$. Here, the convolutional kernels are $F^{\left(1\right)},F^{\left(2\right)},\cdots ,F^{\left(N\right)}$, whose sizes are $M\times K\times K$. It is obvious that the number of trained parameters is $M\times N\times K^2$. For BSConv, the convolutional kernels are as follows:

$$F_{m,;,;}^n={\omega }_{n,m}\cdot B^{\left(n\right)},m\in \left\{1,\cdots ,M\right\},n\in \left\{1,\cdots ,N\right\}$$

(7)

where $\omega$ is weight; $B$ is convolutional kernels with size of $K\times K$. In view of that, the output feature maps can be obtained with size of $N\times Y\times X$. The number of trained parameters is $M\times N+K^2\times N$ in BSConv. There will be a significant reduction compared to standard convolution.

3.3. The Improved FCOS Model

Taking images from the classic public traffic sign dataset TT-100K [4] as an example, the image size is 2048 px × 2048 px. According to statistics, traffic signs generally occupy 0.2% to 1% of the entire image, indicating that traffic sign detection falls into the category of small object detection problems. The difficulty of small object detection lies in the small size of the objects, which may lead to the loss of object region features after multiple layers of processing, resulting in missed detections. To address this issue, LLE-STD introduces the convolutional block attention module (CBAM) [39] and employs the Swish activation function, which can both enhance feature representation and effectively prevent feature loss.

CBAM is an integration of channel attention and spatial attention. It generates attention feature map information on two dimensions, then multiplies these two types of feature map information with the input feature maps to complete adaptive feature refinement, and finally produces the refined feature maps. CBAM is a lightweight module that can be embedded into any backbone network to improve performance. The structure is illustrated in Figure 4.

CBAM is divided into a channel attention module and a spatial attention module. This process can be described mathematically as follows:

$$F^{\prime }=M_C\left(F\right)\otimes F$$

(8)

$$F″=M_S\left(F^{\prime }\right)\otimes F^{\prime }$$

(9)

where $M_C\left(\cdot \right)$ and $M_S\left(\cdot \right)$ are channel attention and spatial attention processing, respectively. $F$ is the input feature maps of CBAM; $F^{\prime }$ is the spatial attention processing input feature map; $F″$ is the output feature maps of CBAM; $\otimes$ is multiplication pixel by pixel multiplication. The channel attention module aggregates spatial information from the input feature maps through max pooling and average pooling, generating two different spatial descriptors. Then, these two descriptors are passed through a shared network consisting of a multilayer perceptron (MLP) and a hidden layer to generate a channel attention map. After applying the shared network to each descriptor, an element-wise summation operation is performed. The process can be described as follows:

$$\begin{array}{cc}{M}_C& =\sigma \left(MLP\left(MeanPool\left(F\right)\right)+MLP\left(MaxPool\left(F\right)\right)\right)\hfill \\ & =\sigma \left(W_1\left(W_0\left(F_{\mathrm{mean}}^c\right)\right)+W_1\left(W_0\left(F_{\max }^c\right)\right)\right)\hfill \end{array}$$

(10)

where $\sigma \left(\cdot \right)$ is the activation function, i.e., Sigmoid; $W_1$ and $W_0$ are two input shared weight coefficients; $MeanPool\left(\cdot \right)$ and $MaxPool\left(\cdot \right)$ are mean- and max-pooling; $MLP\left(\cdot \right)$ is MLP processing. It can be observed that after channel attention processing, although the spatial dimension is significantly compressed, the channel dimension remains unchanged.

The spatial attention module takes the feature maps output by the channel attention module as input. First, it performs average and max pooling along the channel dimension, then concatenates the results to generate an effective feature descriptor. Finally, a convolutional layer is utilized to generate a spatial attention map. The process can be described as follows:

$$\begin{array}{cc}{M}_S& =\sigma \left(f\left(\left[MeanPool\left(F\right);MaxPool\left(F\right)\right]\right)\right)\hfill \\ & =\sigma \left(f\left(\left[F_{\mathrm{mean}}^S;F_{\max }^S\right]\right)\right)\hfill \end{array}$$

(11)

where f(·) is convolution process. After the operation of the spatial attention module, the channel dimension is compressed, while the spatial dimension remains unchanged.

In addition, the classical ResNet50 is selected as the backbone network for the LLE-STD object detection stage in this paper. To achieve better detection results, the Swish activation function is used to replace the ReLU activation in the classical ResNet50. The definition of Swish activation function is as below:

$$f\left(x\right)=x\cdot \sigma \left(\beta x\right)$$

(12)

where $\beta$ is a trainable parameter. When $\beta =0$, the Swish function becomes a linear function; when $\beta =1$, the Swish function becomes SiLU function; when $\beta \to \infty$, the Swish function near to ReLU function. It is obvious that the Swish function is a smooth function lying between a linear kernel function and a ReLU function. The Swish function possesses the characteristics of having no upper bound but a lower bound, being smooth, and being non-monotonic. It helps reduce gradient explosion and gradient vanishing.

3.4. Loss Function

As mentioned earlier, LLE-STD enables end-to-end connection. Simultaneously, during the parameter training process, the backpropagation should propagate the gradients from the detection results section gradually towards the input end of the entire network. Based on this, the loss function for the LLE-STD model is designed in this paper. In view of that the loss function of LLE-STD is designed as below:

$$L_{total}=L_{LLE}+\gamma \cdot L_{STD}$$

(13)

where $L_{LLE}$ and $L_{STD}$ are used to drive the loss terms for low-light image enhancement and traffic sign recognition, respectively. $\gamma$ is the weight used to adjust the influence of both on the total loss function. Drawing on the experience of designing the loss function in the Zero-DCE model [31], $L_{LLE}$ is designed as follows:

$$L_{LLE}=L_{spa}+L_{\exp }+W_{col}{L}_{col}+W_{\mathcal{A}}{L}_{\mathcal{A}}$$

(14)

where, $L_{spa}$, $L_{\exp }$, $L_{col}$, and $L_{\mathcal{A}}$ are spatial consistency loss, exposure control loss, color constancy loss, and lighting smoothness loss, respectively. $W_{col}$ and $W_{\mathcal{A}}$ are weight. Firstly, the spatial consistency loss term is used to measure the difference in intensity values between the low-light image before and after enhancement in their respective adjacent regions. The expression is as follows:

$$L_{spa}={\displaystyle \frac{1}{K}}{\displaystyle {\sum }_{i=1}^K{\displaystyle {\sum }_{j\in \mathrm{\Omega }\left(i\right)}{\left(\left|Y_i-Y_j\right|-\left|I_i-I_j\right|\right)}^2}}$$

(15)

where K is the number of pre-allocated local areas; Y is enhanced image; I is original low-light image; $\mathrm{\Omega }\left(i\right)$ represents the four (top, bottom, left, and right) adjacent regions centered on region i. Obviously, the smaller the difference in intensity values between the neighborhoods of the images before and after enhancement, the smaller the $L_{spa}$ value will be. In addition, $L_{\exp }$ is used to evaluate the distance between the average intensity value of local regions and a threshold to avoid overexposure of the enhanced image. Its expression is the following:

$$L_{\exp }={\displaystyle \frac{1}{M}}{\displaystyle {\sum }_{k=1}^M\left|\overline{Y_k}-E\right|}$$

(16)

where M represents the number of non-overlapping regions, and in this paper, each region is set to have a size of 16 × 16; $\overline{Y}$ is the average intensity of enhanced image in the part; E is the pre-set intensity threshold, and in this paper, E = 0.6. $L_{col}$ is used to evaluate the degree of deviation between the three color channels. A smaller value indicates that the three color channels are more balanced. The definition is as follows:

$$L_{col}={\displaystyle {\sum }_{\forall \left(p,q\right)\in \epsilon }{\left(J^p-J^q\right)}^2},\epsilon =\left\{\left(R,G\right),\left(R,B\right),\left(G,B\right)\right\}$$

(17)

where $J^p$ represents the average intensity value of p color channel in the enhanced image. $L_{\mathcal{A}}$ is used to improve the parameter mapping of DCE-Net output to make it smoother. The expression is as below:

$$L_{\mathcal{A}}={\displaystyle \frac{1}{N}}{\displaystyle {\sum }_{n=1}^N{\displaystyle {\sum }_{c\in \xi }{\left(\left|{\nabla }_x{\mathcal{A}}_n^c\right|+\left|{\nabla }_y{\mathcal{A}}_n^c\right|\right)}^2}},\xi =\left\{R,G,B\right\}$$

(18)

where N is the number of iterations to improve DCE-Net, which is set to 8 in this paper. ${\nabla }_x$ and ${\nabla }_y$ are the horizontal and vertical gradients of the parameter mapping, respectively. The smaller the gradient value, the smaller $L_{\mathcal{A}}$ it is.

In addition, the loss function design of FCOS model [15] is mainly used for reference in setting target detection tasks, which is specifically expressed as follows:

$$L_{STD}={\displaystyle \frac{1}{N_{pos}}}{\displaystyle {\sum }_{x,y}{L}_{cls}\left(p_{x,y},c_{x,y}^{*}\right)}+{\displaystyle \frac{\lambda }{N_{pos}}}{\displaystyle {\sum }_{x,y}\chi \left(c_{x,y}^{*}>0\right)L_{reg}\left(t_{x,y},t_{x,y}^{*}\right)}$$

(19)

where $N_{pos}$ is the positive sample number; $\lambda$ is the weight parameter; $\chi \left(\cdot \right)$ is indicator function. That is, if the condition is met, the value is 1; otherwise, it is 0. $L_{cls}\left(\cdot \right)$ and $L_{reg}\left(\cdot \right)$ are FL loss term of RetinaNet and IoU loss term of UnitBox, respectively.

4. Assessment Criteria

In this experimental section, to quantitatively evaluate the enhancement effect of low-light images, we introduce the classical peak signal-to-noise ratio (PSNR), structure similarity index measure (SSIM) and learned perceptual image patch similarity (LPIPS). Their definitions are as follows:

(1)

PSNR

PSNR is an objective and quantitative evaluation criterion for image quality. It compares the gray-scale differences between the processed image and the original image. The definition is as follows:

$$PSNR=10\times {\log }_{10}({\displaystyle \frac{{(2^n-1)}^2}{MSE}})$$

(20)

where n is the number of sampling points. In this paper, we process the RGB images, so n = 24. MSE stands for mean squared error, which is defined as follows:

$$MSE={\displaystyle \frac{1}{H\times W}}{\displaystyle \sum _{i=1}^H{\displaystyle \sum _{j=1}^W{(X(i,j)-Y(i,j))}^2}}$$

(21)

where H × W is the number of pixels in the image; H and W are the length and width of the image; X is the enhanced image; Y is the real clear image.

(2)

SSIM

SSIM evaluates the similarity between a processed image and the ground truth image by considering three aspects: image luminance, image contrast, and image structural information. The definition of SSIM is as follows:

$$SSIM=l(X,Y)\times c(X,Y)\times s(X,Y)$$

(22)

where $l(X,Y)$, $c(X,Y)$, and $s(X,Y)$ are defined as follows:

$$\begin{array}{l}l(X,Y)={\displaystyle \frac{2{\mu }_x{\mu }_y+C_1}{{\mu }_x^2+{\mu }_y^2+C_1}}\\ c(X,Y)={\displaystyle \frac{2{\delta }_x{\delta }_y+C_2}{{\delta }_x^2+{\delta }_y^2+C_2}}\\ s(X,Y)={\displaystyle \frac{{\delta }_{xy}+C_3}{{\delta }_x{\delta }_y+C_3}}\end{array}$$

(23)

where ${\mu }_x$ is the mean of the enhanced image; ${\mu }_y$ is the mean of the clear image. Moreover, ${\delta }_x^2$ and ${\delta }_y^2$ are the variances of enhanced image and clear image, respectively. ${\delta }_{xy}$ is typically not used to directly represent the covariance between two images. C₁, C₂, and C₃ are constants. In order to avoid the situation where the denominator is zero, we usually set the following:

$$C_1={(K_1\times L)}^2, C_2={(K_2\times L)}^2, C_3={\displaystyle \frac{C_2}{2}}$$

(24)

Let $K_1=0.01$, $K_2=0.03$, and $L=255$. Clearly, the value of SSIM ranges from [0, 1], with a higher value indicating that the enhanced image is closer to the original clear image, resulting in better enhancement performance. Conversely, a lower SSIM value suggests a greater disparity between the two images, indicating less ideal enhancement results.

(3)

LPIPS

The design philosophy of LPIPS is to measure the perceptual similarity between an enhanced image and its original, authentic counterpart during the process of inversely recovering the original image from the enhanced one. Its definition is formulated as follows:

$$LPIPS={\displaystyle {\sum }_L\frac{1}{H_l{W}_l}{\displaystyle {\sum }_{h,w}{‖{\omega }_l\odot \left(y^l-y_o^l\right)‖}_2^2}}$$

(25)

where $y^l$ is the lth feature maps. Normalize it with respect to the initial feature map $y_o^l$ in the channel dimension using unit normalization and scale the number of activated channels using ${\omega }_l$, then calculate the L2 distance value. Here, $\odot$ is dot product operation. Finally, by averaging over the spatial dimension and summing over the channel dimension, the LPIPS metric can be obtained. A lower value of this metric indicates better image enhancement performance.

Furthermore, to quantitatively evaluate the effectiveness of object detection, this paper employs mAP and F1 measure as evaluation metrics. Among them, average precision (AP) is used to measure the precision for a specific type of object, representing the area under the precision–recall (PR) curve. mAP, typically serving as an indicator for quantitatively assessing the overall precision of a detection model, is the average result of AP across different categories and is one of the commonly used metrics for evaluating object detectors. The definition of mAP is as follows:

$$mAP={\displaystyle \frac{1}{C}}{\displaystyle {\sum }_{i=1}^{C}A{P}_i}$$

(26)

where C is the type number. Additionally, the F1 measure is the weighted harmonic mean of precision and recall. A higher F1 measure indicates better detection performance, and its definition is as follows:

$$F1-measure={\displaystyle \frac{2PR}{P+R}}$$

(27)

where P is precision; R is recall. In addition, to evaluate the significant difference in the accuracy of training and testing detection results, we use t-test indicator. It is defined as follows:

$$t-test={\displaystyle \frac{1}{N}}{\displaystyle {\sum }_{i=1}^N\frac{mean\left(X_i\right)-mean\left(Y_i\right)}{std\left(X_i-Y_i\right)}}$$

(28)

where X_i and Y_i are ith detection results in training and test stages, respectively; N is the number of samples; mean(·) is the average of detection results; std(·) is the standard deviation. If the value is in the range of [−2, 2], we judge that there is no overfitting phenomenon.

5. Experiment and Results

5.1. Experimental Data and Experimental Settings

This paper conducts experimental verification based on data from multiple public traffic sign datasets, mainly selecting three datasets: GTSDB [20], TT-100K [4], and GLARE [40]. The basic information of these three datasets is shown in

Table 2. The condition of public traffic sign datasets.

Dataset	Image Number	Resolution	Types	Country	Year
GTSDB [20]	900	1360 × 1024	43	Germany	2013
TT-100K [4]	100,000	2048 × 2048	221	China	2016
CLARE [41]	2157	720 × 480~1920 × 480	41	USA	2022

. Additionally, to increase the proportion of low-light images, an exponential transformation function is applied to over half of the images in each dataset to synthesize low-light images. Specifically, each original image corresponds to four synthesized low-light images, and by adjusting the parameters of the exponential transformation, the image quality varies accordingly. Figure 5 shows an original high-quality image from each of the three datasets and its corresponding four synthesized low-light images. The synthesized images are merged with the original images to form new datasets, which are then divided into training, validation, and test data according to a ratio of 6:2:2.

In this experiment, two NVIDIA 3090 ti GPUs are selected. The synchronous stochastic gradient descent (SGD) optimizer is used with an initial learning rate set to 0.01. The batch size is set to 8. At iterations 60 K and 80 K, the learning rate is reduced by a factor of 10. The weight decay and momentum were set to 0.0001 and 0.9, respectively. In order to ensure the referenceability of results, these experiments use fourfold cross-validation. Moreover, the following model loss functions have reached a minimum and stable.

5.2. Performance Analysis of Low-Light Image Enhancement Module

Firstly, to validate the processing effect of the low-light image enhancement stage in the LLE-STD model, an experimental analysis of low-light enhancement effects was conducted on the merged traffic sign datasets GTSDB, TT-100K, and GLARE, which simulate low-light images. Here, classical models including LIME, RetinexNet, EnlightenGAN, RUAS, Zero-DCE, Zero-DCE++, Retinexformer, as well as the proposed method, were selected for quantitative and qualitative comparisons. Among them, the quantitative comparison results are shown in

Table 3. Quantitative comparison of enhancement results in low-light images.

Method	PSNR	SSIM	LPIPS
LIME [41]	14.84	0.55	0.61
RetinexNet [42]	15.53	0.50	0.58
EnlightenGAN [26]	15.88	0.51	0.55
RUAS [24]	16.16	0.54	0.54
Zero-DCE [27]	16.09	0.55	0.51
Zero-DCE++ [28]	16.11	0.54	0.52
Retinexformer [43]	16.09	0.53	0.53
The proposed	16.18	0.55	0.52
(a) GTSDB
LIME [41]	14.88	0.44	0.65
RetinexNet [42]	15.02	0.45	0.69
EnlightenGAN [26]	15.22	0.49	0.71
RUAS [24]	15.30	0.48	0.68
Zero-DCE [27]	15.44	0.50	0.65
Zero-DCE++ [28]	15.20	0.50	0.68
Retinexformer [43]	15.35	0.49	0.66
The proposed	15.40	0.51	0.66
(b) TT-100K
LIME [41]	13.15	0.41	0.66
RetinexNet [42]	13.89	0.45	0.65
EnlightenGAN [26]	13.95	0.50	0.60
RUAS [24]	14.11	0.52	0.55
Zero-DCE [27]	14.52	0.52	0.56
Zero-DCE++ [28]	14.14	0.50	0.58
Retinexformer [43]	14.30	0.49	0.57
The proposed	14.38	0.50	0.55
(c) GLARE

.

As can be seen from Table 3, the proposed method ensures that most image enhancement metrics are at a relatively high level when processing data from the three different datasets. In particular, it has significant advantages compared to the LIME, RetinexNet, and EnlightenGAN methods. Compared to the RUAS and Zero-DCE methods, the processing effect is roughly the same. Even for the GTSDB dataset, the image enhancement metrics of the proposed method are slightly better than those of the two aforementioned methods. Additionally, the low-light image enhancement effect can also be compared qualitatively. Comparing with the aforementioned classical models across multiple datasets, the enhanced images are shown in Figure 6. It can be observed that each low-light image enhancement method can produce enhanced images with more detailed information compared to the original low-light images. However, the processing results of the proposed method, as well as the RUAS and Zero-DCE methods, are closer to the original clear images.

Compared to RUAS and Zero-DCE, the method proposed in this paper has certain advantages in terms of computational resource utilization due to the replacement of classical convolutional processing with BSConv. Specifically, this can be compared by examining the model runtime (RT) and floating-point operations (FLOPs), as shown in

Table 4. Comparison of running time and floating point computation of different networks.

Methods	RT (s)	FLOPs (G)
RUAS [24]	0.0186	411.35
Zero-DCE [27]	0.0093	102.61
The proposed	0.0046	64.14

. It is evident that the runtime of the proposed method is reduced by 0.0140 s and 0.0047 s, respectively, compared to RUAS and Zero-DCE, while the FLOPs are reduced by 347.21 G and 38.47 G, respectively. This demonstrates the significant lightweight effect of the proposed model. In other words, while ensuring that the low-light image enhancement performance remains at a near-optimal level, the model is lightweight and saves computational resources.

5.3. Performance Analysis of Small Target Detection Module

This section focuses on the performance of the object detection stage model in LLE-STD. To evaluate the reliability of the proposed method, it is compared with classical object detection models. The comparison models include Faster-RCNN [7], SSD [12], RetinaNet [42], Yolov3 [13], Yolov4-tiny [14], Yolov4 [14], and FCOS [15]. To ensure the persuasiveness of the comparison results, it is crucial to train different models using the same training data and test them on the same test data during the comparative experiments. Figure 7 first presents the detection results of eight methods on the same image from the TT-100K dataset. For intuitive comparison, this paper also enlarges the regions that mainly contain traffic signs. The detection types and their corresponding probability values are also clearly displayed.

The results in Figure 7 show that the traffic sign targets are very small, further indicating that traffic sign detection in intelligent transportation scenarios indeed belongs to the task of small object detection. Among the eight methods mentioned above, Faster-RCNN, SSD, and RetinaNet exhibit significant missed detections, which is due to the fact that the missed traffic signs are close to the edges of the entire image, potentially leading to information loss during the detection process. From the results of the other five detection methods, it can also be observed that the detected types are basically correct, but in terms of probability values, FCOS and the proposed method have obvious advantages, thereby demonstrating the superior detection performance of FCOS and the proposed method.

Furthermore, this paper also compares the object detection performance of some SOTA methods quantitatively by training and testing the models on three public datasets. The comparison results of the detection outcomes are shown in

Table 5. Quantitative comparison of detection results of different detection models.

Detection Method	mAP (%)			F1-Measure (%)
	GTSDB	TT-100K	GLARE	GTSDB	TT-100K	GLARE
Faster-RCNN [7]	70.85	71.28	68.51	58.86	70.59	67.51
SSD [13]	70.47	72.22	67.94	69.05	70.41	65.38
RetinaNet [44]	72.55	78.51	68.88	71.10	75.38	68.00
Yolov3 [13]	73.28	81.05	71.89	71.11	76.88	69.95
Yolov4-tiny [14]	75.85	81.10	72.04	72.94	80.94	71.05
Yolov4 [14]	75.41	81.38	72.28	72.85	81.18	70.98
AFPN [45]	78.06	81.75	75.45	75.87	82.05	72.22
Attention-Yolov4 [46]	78.55	81.59	75.40	75.81	81.95	72.08
FCOS [15]	78.08	81.41	75.33	75.75	81.88	72.10
FCOS-self-attention	78.15	81.84	75.40	75.30	82.08	72.19
The proposed	78.88	82.23	75.44	75.44	82.30	72.31

. Here, to verify the effectiveness of CBAM, the FCOS improved by self-attention model is also trained. The detection results are shown in Table 5, too. It can be clearly seen that the proposed method only underperforms FCOS in terms of mAP on the CLARE dataset and F1 measure on the GTSDB dataset, and the differences are relatively close. In other cases, the proposed method has significant advantages. Under the mAP metric, the proposed method achieves at least a 6.93% improvement over the classic Faster-RCNN model. Under the F1 measure metric, the proposed method achieves at least a 1.12% improvement over the classic Yolo series methods. In addition, we use t-test values to determine the differences in target detection results between the proposed method during the training and testing stages. The t-test values for the mAP metrics under the GTSDB, TT-100K, and GLARE datasets are 0.89, 1.32, and −0.44, respectively. The t-test values for the F1 measure metrics are 0.54, −1.00, and 0.87, respectively. Obviously, these values are all within the range of [−2, 2], indicating that there is no significant difference in the accuracy of training and testing object detection results, proving that the method does not exhibit significant overfitting.

In summary, the object detection stage in the LLE-STD method proposed in this paper exhibits strong adaptability to the task of small object detection for traffic signs, effectively enhancing detection accuracy and reducing missed detections.

5.4. Adaptability Analysis of the Proposed Model

The LLE-STD proposed in this paper does not simply cascade low-light image enhancement with small object detection but ensures end-to-end execution of the inference process through coupled connections. Meanwhile, during the backpropagation of parameter training, the gradient values at different stages will mutually influence the direction of parameters in both stages, thereby enhancing the adaptability of LLE-STD to images under different illumination conditions. This section verifies this adaptability through experiments.

Firstly, we use the Grad-CAM [46] to visualize the heatmap of the prediction process of the LLE-STD network, helping us better understand the key points of interest in LLE-STD. Figure 8 shows the heat maps of three groups of real traffic sign images. Obviously, LLE-STD can focus on the small target region of traffic signs in the image more specifically. Therefore, it is proved that the features learned by LLE-STD match the processing task.

Moreover, Figure 9 shows the detection results of an image from the TT-100K dataset after applying different levels of low-light simulation processing, using the proposed LLE-STD method. We also locally enlarge the regions containing traffic signs and their detection results. Comparing the results, it can be observed that as the image quality degrades, most of the detection results remain unchanged, with only slight decreases in probability values. The only missed detection occurs in the image with the weakest quality shown in Figure 9a, where the traffic sign corresponding to p11 is not detected. This is due to the reduced difference between small objects and the background caused by the deterioration of image quality. However, overall, the impact of low light on LLE-STD is within an acceptable range.

In addition, Figure 10 shows the LLE-STD detection results of three images with different light intensities in the real public dataset. It is evident that although light still has an impact on the detection of traffic sign targets. However, the LLE-STD method can ensure accurate detection results in real night scenes.

Furthermore, to further analyze the adaptability of LLE-STD to different image qualities from a quantitative perspective, this section also conducts statistics on mAP and F1 measure for data in the three datasets and compares them with three other classic object detection models. The results are shown in Figure 11. In these results, we need to pay attention not only to the detection results corresponding to different methods but also to the rate of change in detection results as image quality increases, i.e., the gradient situation. In Figure 11, levels 1 to 4 correspond to the low-light image synthesis levels described in Section 5.1, with image quality gradually improving. However, even at level 4, the image quality is still inferior to the original images in the public datasets.

As can be seen from the results in Figure 11, the proposed LLE-STD achieves the optimal values of mAP and F1 measure for data from all datasets. In particular, the advantage of LLE-STD is more pronounced under level 1 condition, with the mAP tested on the GLARE dataset being approximately 15% higher than that of RetinaNet. Additionally, as the horizontal axis moves to the right, the values of LLE-STD change most smoothly, indicating the smallest gradient. These results demonstrate that LLE-STD has the strongest adaptability to images of different qualities. This phenomenon occurs because the other three classic object detection models exhibit significant overfitting and can only adapt to high-quality original images. However, LLE-STD enhances adaptability by coupling the two stages of low-light image enhancement and small object detection.

6. Discussions

LLE-STD, proposed in this paper, can balance the effectiveness and reliability of detection as much as possible. In the Experiment and Results Section of this paper, comparative experiments are conducted on data from three public datasets: GTSDB, TT-100K, and GLARE. Firstly, in the performance analysis of the low-light enhancement module, the proposed method is qualitatively and quantitatively compared with five classic low-light image enhancement methods to demonstrate its advantages in image enhancement processing. It is also compared with two classic methods with better image enhancement effects in terms of runtime and FLOPs, proving that the proposed method is lightweight in the low-light image enhancement stage. Furthermore, in the performance analysis of the small object detection module, the performance of LLE-STD is compared with seven other classic object detection methods using both qualitative and quantitative methods. It is found that the proposed method achieves lower mAP values on the CLARE dataset and F1 measure values comparable to FCOS on the GTSDB dataset but outperforms other methods under most conditions. Specifically, under the mAP metric, the proposed method achieves at least a 6.93% improvement over the classic Faster-RCNN model across different datasets. Under the F1 measure metric, it achieves at least a 1.12% improvement over the classic Yolo series methods. Finally, in the analysis of the adaptability of the LLE-STD model to images, images of different qualities, no matter in simulation and actual images, are used for object detection tests, and the results of LLE-STD are compared with other classic models. It is found that the advantage of LLE-STD is more pronounced, especially in low-light images. For example, the mAP results tested on the GLARE dataset show that LLE-STD outperforms RetinaNet by approximately 15%, demonstrating that LLE-STD enhances image adaptability due to its coupled connection mode. In addition, we also conduct tests on the embedded segments of the proposed method. We selected the Zynq UltraScale+ MPSoCs CG series chip with model number XCZU2CG-1SFVC784E for testing. The proposed method can meet the real-time requirements of the autonomous driving field with a processing time of about 30 ms per frame image.

7. Conclusions

This paper focuses on the problem of small object detection in low-light complex scenes for traffic sign detection tasks and proposes a two-stage coupled model for low-light enhancement and small object detection, namely, LLE-STD. This method takes the classic low-light image enhancement model and small object detection model, Zero-DCE and FCOS, as baselines, and makes corresponding improvements by reducing model computation and enhancing small object feature representation. The two models are then coupled to form an end-to-end processing mode. Through the design of loss functions, the gradients of both stages jointly influence the training of model parameters, thereby enhancing the adaptability of the model to images of different qualities. Moreover, some experiments verify the effectiveness and reliability of LLE-STD. In summary, the proposed LLE-STD model in this paper can well adapt to low-light images and effectively detect small objects such as traffic signs, which can promote the application of related intelligent algorithms in the field of autonomous driving.