Advancing Nighttime Object Detection through Image Enhancement and Domain Adaptation

Abstract

Due to the lack of annotations for nighttime low-light images, object detection in low-light images has always been a challenging problem. Achieving high-precision results at night is also an issue. Additionally, we aim to use a single nighttime dataset to complete the knowledge distillation task while improving the detection accuracy of object detection models under nighttime low-light conditions and reducing the computational cost of the model, especially for small targets and objects contaminated by special nighttime lighting. This paper proposes a Nighttime Unsupervised Domain Adaptation Network (NUDN) based on knowledge distillation to address these issues. To improve the detection accuracy of nighttime images, high-confidence bounding box predictions from the teacher and region proposals from the student are first fused, allowing the teacher to perform better in subsequent training, thus generating a combination of high-confidence and low-confidence pseudo-labels. This combination of feature information is used to guide model training, enabling the model to extract feature information similar to that of source images in nighttime low-light images. Nighttime images and pseudo-labels undergo random size transformations before being used as input for the student, enhancing the model’s generalization across different scales. To address the scarcity of nighttime datasets, we propose a nighttime-specific augmentation pipeline called LightImg. This pipeline enhances nighttime features, transforming them into daytime features and reducing issues such as backlighting, uneven illumination, and dim nighttime light, enabling cross-domain research using existing nighttime datasets. Our experimental results show that NUDN can significantly improve nighttime low-light object detection accuracy on the SHIFT and ExDark datasets. We conduct extensive experiments and ablation studies to demonstrate the effectiveness and efficiency of our work.

1. Introduction

In recent years, significant progress has been made in object detection technology. However, under complex lighting conditions, particularly in nighttime environments, object detection still faces significant limitations. Nighttime object detection is crucial in many applications [1,2,3]. With the development of deep learning technology, the ability to learn object recognition features in a single domain has made significant advances [4,5,6]. However, due to the substantial domain shift between daytime and nighttime environments, models trained during the day often do not generalize well to other domains [5,6,7]. This is often due to unavoidable environmental factors such as insufficient lighting and high dynamic range, as well as technical limitations such as inadequate exposure time and low gain, which pose great challenges to nighttime recognition. Additionally, images usually cannot be captured under optimal lighting conditions and are affected by unavoidable environmental factors such as backlighting and uneven illumination, which often lead to significant degradation in the model’s actual detection performance [7,8,9,10]. Furthermore, objects in nighttime images are often smaller and harder to recognize, increasing the difficulty of detection [11,12]. Collecting sufficient nighttime data for manual annotation via cameras is also costly. The results of the LightImg image enhancement, compared to the unenhanced images, are shown in Figure 1. This has prompted researchers to focus on improving the model’s adaptability across domains. Domain adaptation is an effective solution that allows us to use existing datasets.

Unsupervised domain adaptation (UDA) methods attempt to adapt the model learned in the source domain to the target domain without annotated labels. Most existing cross-domain methods directly learn in the target domain by using labeled images from the source domain or pseudo-label estimates for unlabeled images from the target domain [11,12,13,14]. These methods do improve the accuracy of models in the target domain [2,5,10]. However, the learned models still struggle to generalize to unseen domains and require a large number of unlabeled target domain images for adaptation [15,16,17,18], which is clearly impractical in the context of day-to-night adaptation. In the source domain, a large amount of labeled image data is required, while in the target domain, the style or resolution of the images may differ. Adversarial learning is then used with image and instance-level classifiers. However, these methods isolate the domain adaptation task purely within the feature extractor, suppressing the features of the target data to maintain domain invariance. Recent unsupervised domain adaptation methods have utilized knowledge distillation. Since the student undergoes supervised learning during the pre-training phase, it is sensitive to data variations. To address this issue, adversarial learning methods are used. In day-to-night tasks, methods that make daytime image features more similar to nighttime features are also used. However, in day-to-night tasks, such methods can be contaminated by a large number of low-quality pseudo-labels generated by the teacher. In our investigation, the problem is mainly due to the significant changes between day and night environments. These erroneous pseudo-labels are repeatedly used as benchmarks during training, leading to poor object detection performance.

To address this problem, we propose the NUDN method, as shown in Figure 2, a nighttime unsupervised cross-domain object detection approach based on knowledge distillation. Our network is the first to use enhanced images based on nighttime environmental images for cross-domain training. Our NUDN first converts nighttime images to daytime images. Then, NUDN merges the bounding boxes of pseudo-labels produced by the teacher with the regions proposed by the student’s Region Proposal Network (RPN). The teacher then uses the merged results to generate new pseudo-labels. These pseudo-labels are then matched with the student’s predictions. The knowledge conveyed by Teacher RPN to students is shown in Figure 2. We can then use a weighted approach to ensure that our network is based on high-confidence pseudo-labels [19,20].

To enhance nighttime images, daytime images exhibit various features absent in nighttime scenes, such as rich colors, detailed lighting, and low noise levels. Since the student is trained on daytime images, nighttime images cannot be overly enhanced to prevent the network from overly favoring daytime features. At the same time, existing low-light image enhancement methods are based on Retinex theory [21], using either network or model-based approaches. However, both network-based and model-based methods require significant computation time, which is intolerable. To address this issue, we propose a nighttime enhancement method called LightImg (Figure 3). LightImg is designed to brighten images quickly, flexibly, and robustly in real low-light scenarios. As a fundamental module, it significantly reduces computational costs, enabling us to conduct cross-domain research using only nighttime datasets.

To address the issue of misidentifying small and medium-sized objects at night, we have designed a technique that can randomly scale images across various sizes. This technique resizes images and pseudo-labels to random sizes, enabling the model to robustly detect objects of various sizes and preventing overfitting at any single scale. To avoid affecting the teacher’s performance, the images used by the teacher retain their original size. This allows large objects to be disguised as smaller ones, thereby improving the student network’s ability to detect small and medium-sized objects. Overall, we achieve qualitative improvements in our results.

In summary, the contributions of this paper include:

• We propose NUDN, a teacher-student learning method with co-learning capabilities. NUDN utilizes a combination of high-confidence teacher labels and low-confidence student proposals. This strategy significantly enhances the utilization of pseudo-labels.
• To enable cross-domain research using existing nighttime datasets, we propose the image enhancement technique LightImg, which can convert nighttime features into daytime features.
• The use of random image scaling mitigates data limitations.
• We conducted comprehensive evaluations on two nighttime datasets, demonstrating that our NUDN exhibits strong discriminative capabilities in the source domain, robust adaptability to single-target domains, and strong generalization abilities to unseen domains.

2. Related Work

2.1. Image Enhancement

Image enhancement is a key task in the field of computer vision, primarily aiming to improve the visual quality of images, making them more suitable for human observation or computer processing [22,23,24]. In recent years, deep learning methods have made significant progress in the field of image enhancement, achieving this by learning the mapping relationships from low-quality to high-quality images through deep neural networks. Common visual disturbances include low light, glare, blur, and rain streaks.

For low-light problems, traditional low-light enhancement methods mainly include histogram equalization [25,26] and retina model-based methods [27,28,29]. Histogram equalization enhances contrast by adjusting the grayscale distribution of an image, while the retina model simulates the visual processing mechanisms of the human retina to improve image quality. However, these methods often have limited effectiveness in handling complex nighttime lighting environments. With the popularity of neural networks, some models enhance images in an end-to-end manner, while others use unsupervised methods to adapt to complex nighttime lighting. Additionally, many models have been designed based on physically interpretable retina theory. For example, Sharma et al. [30] used camera response function (CRF) estimation and HDR imaging methods to suppress glare; Jin et al. [31] performed unsupervised learning through integrated layer decomposition and light effect suppression. Although these methods improve image quality to some extent, they often perform poorly in nighttime scenes where low light and glare occur simultaneously.

Under low-light conditions, nighttime images may become blurry, especially in adverse weather such as rainy days, where rain streaks and puddle reflections can cause significant interference. To address these issues, some methods attempt to enhance complex rainy scenes [21,30,31,32,33,34,35,36]. For example, deep learning methods can effectively remove rain streaks and suppress reflections. However, although these methods can improve human visual quality, they do not consider the significant differences between human perception and machine vision, nor the applicability of these methods in other visual tasks.

Existing methods for processing images under low-light conditions still have some shortcomings. Specifically, they find it difficult to quickly process global illumination while using minimal computational resources, and they may not effectively suppress noise under complex lighting conditions. Furthermore, the applicability of these methods in different visual tasks is also limited.

To solve these problems, we propose a lightweight image enhancement method. This method not only quickly processes global illumination but also effectively suppresses noise under complex lighting conditions while using minimal computational resources. Through this method, we can achieve efficient and robust image enhancement in practical applications, thereby improving the performance and reliability of nighttime object detection.

2.2. Unsupervised Domain Adaptation (UDA) for Object Detection

To address the fundamental problem of domain shift, past work has primarily focused on exploring the transfer of rich knowledge from the source domain to the target domain, which helps reduce domain shift at the feature level [1,10,12,18]. Common methods in this category include adversarial feature learning and semi-supervised Mean Teacher (MT) frameworks [37]. Adversarial feature learning aims to generate domain-invariant features through a min-max game to produce domain-invariant feature maps. Chen et al. [1] first proposed adversarial training for image-level and instance-level features of multiple bounding boxes. Some methods enforce consistency regularization on instances of multiple categories. This approach has been applied to image classification. Our focus is on nighttime object detection, which is more complex as it involves low-light image recognition. The semi-supervised Mean Teacher framework [37] improves the performance of semi-supervised learning models by distilling pseudo-labels from unlabeled data [2,4]. The quality of pseudo-labels is crucial. Zhu et al. [2] proposed a domain-adaptive object detection algorithm based on CycleGAN to help improve the quality of pseudo-labels. Zheng et al. [32] proposed a hierarchical feature alignment method, which effectively addresses domain adaptability issues in cross-domain object detection through step-by-step feature adaptation from global to local levels. Knowledge distillation techniques have been utilized to improve detection accuracy by transferring knowledge from a larger, well-trained teacher model to a smaller student model [38,39,40]. Mengzhe He et al. [33] proposed a dual branch distillation method for cross-domain object detection, which is particularly effective for nighttime scenarios [33]. Another notable approach is the two-phase consistency training for day-to-night domain adaptation by Kennerley et al. [41]. Jin et al. [4] improved the generalization performance of object detection models in unknown target domains by increasing sample diversity between the source and target domains (Diversify) and enforcing distribution matching in feature space (Match) [42]. Research on nighttime object detection has also made significant progress. Li et al. [34] proposed a dynamic convolutional neural network for nighttime object detection, improving detection accuracy and robustness [7,8,16,18]. Wang et al. [35] used unsupervised image translation techniques to enhance object detection in nighttime traffic scenes [10,14,19,43]. Moreover, low-light image enhancement techniques have played a key role in nighttime object detection [44,45,46,47]. Some methods also combine adversarial feature learning to improve the quality of pseudo-labels. Although these methods have achieved promising results in most domain adaptation tasks, their adaptability is limited when domain shift is significant or when datasets are small. This is mainly because it is difficult to obtain high-quality pseudo-labels in extremely adverse environments [3,6,17].

In Unsupervised Domain Adaptation (UDA) methods, the goal of UDA is to learn transferable features so that the model can exhibit better adaptability and performance in tasks [1,12]. Unlike traditional UDA methods that minimize the feature distribution discrepancy between the source and target domains, most cross-domain methods use “labeled images” in the target domain for supervised learning, as different domains often have entirely different characteristics. For example, CycleGAN translates source domain images into the target domain [20], and the style-translated labeled images are then used for supervised learning in the target domain. Other methods iteratively estimate pseudo-labels for unlabeled target images through clustering and learn the model in the target domain through supervised learning via a self-training scheme. Some methods use additional labels or external models for learning. For instance, TJ-AIDL and MMFA use additional personnel attributes to learn attribute semantics and identity-discriminative feature representation spaces transferable to the target domain, while EANet uses partial segmentation constraints from external pose estimation models to enhance alignment. Although effective in the target domain, they still require a large amount of unlabeled data for adaptation.

3. Methodology

This section provides a detailed description of the proposed algorithm, the Nighttime Unsupervised Domain Adaptation Network (NUDN) for object detection based on distillation. Let $\mathrm{D}$ denote the daytime source data ${\mathrm{D}}_{\mathrm{d}}=\{{\mathrm{I}}_{\mathrm{d}},{\mathrm{C}}_{\mathrm{d}},{\mathrm{B}}_{\mathrm{d}}\}$, where C_d represents source data images, ${\mathrm{C}}_{\mathrm{d}}$ denotes source data class labels, and B_d denotes bounding box labels. The subscript d indicates the daytime source. The nighttime target data is denoted by ${\mathrm{D}}_{\mathrm{n}}$, where ${\mathrm{D}}_{\mathrm{n}}=\left\{{\mathrm{I}}_{\mathrm{n}}\right\}$, as the target domain contains unlabeled samples. The subscript indicates the nighttime target.

The architecture of our NUDN is illustrated in Figure 2. Our network comprises a student network and a teacher network. The student network is a multi-domain model trained with LightImg-enhanced daytime images and unlabeled nighttime images. The teacher network only needs to learn from nighttime images and subsequently generate pseudo-labels for the student, with weights being the exponential moving average (EMA) of the student [8,13]. Following the pre-training phase, the teacher begins generating pseudo-labels while the student initializes feature extractors and detectors.

Model initialization is achieved through pre-training, which involves generating candidate object proposals via the Region Proposal Network (RPN). The feature vectors obtained from ROI pooling are used by the ROI-level classifier to predict category labels. The training loss is composed of the losses from both the RPN and the ROI classifier:

$$L_{Pre}=L_{rpn}+L_{roi}$$

(1)

During each iteration, the teacher first generates pseudo-labels from the nighttime images. These pseudo-labels are filtered through a confidence threshold to ensure the student learns from high-quality pseudo-labels. Subsequently, the bounding boxes in the pseudo-labels are combined with regions proposed by the student’s Region Proposal Network (RPN). The combined region proposals are then used to generate predictions from the student’s RoI network.

3.1. Consistency Learning

Due to the cross-domain nature of the learning process, there exists a significant gap between source and target domain images, hindering the teacher’s ability to perform its tasks effectively. This phenomenon is prevalent throughout the network, particularly in challenging areas such as low light, glare, and flare. However, the teacher network tends to produce high-confidence labels that correspond to daytime features, gradually biasing towards daytime characteristics. This cyclical process results in the teacher generating simple samples with daytime attributes, thereby preventing the student from learning from more challenging areas.

Due to the limited understanding of positive samples, the teacher starts predicting highly confident but false-positive pseudo-labels. When the teacher provides these false-positive pseudo-labels to the student, a negative feedback loop begins, causing the teacher to subsequently update with incorrect knowledge. Consequently, errors propagate and accumulate throughout the training process. Our study shows that these errors are particularly pronounced in areas with nighttime features and small objects.

To address this issue, we designed a consistent learning approach. This approach allows for the coexistence of high-confidence and low-confidence pseudo-labels [6,20,48]. The teacher first predicts pseudo-labels from the unlabeled nighttime images. These pseudo-labels are filtered through a predefined confidence threshold, retaining only those with confidence higher than this threshold as the high-confidence pseudo-label set. At the initial stage, the confidence threshold was determined experimentally, with the initial value selected based on our performance on the validation set. A higher confidence threshold helps ensure the accuracy of the retained pseudo-labels, thereby reducing the risk of error propagation. As the model’s capability improves, the threshold can be gradually lowered, allowing the model to access a more diverse set of data samples, including those with lower confidence. Moreover, low-confidence pseudo-labels are not discarded; instead, at the initial stage, they are assigned a very low weight and continue to participate in the training process. As the model’s performance improves, the weight of low-confidence labels can be gradually increased, allowing the model to better manage complex nighttime scenes. The bounding boxes of the pseudo-labels are then used as input for the student network. These bounding boxes are merged with the candidate boxes generated by the student’s RPN module to form new candidate boxes:

$${CB}_p={RPN}_{student} \cup B_p$$

(2)

Here, ${CB}_p$ represents the combined candidate boxes, which are then sent to the RoI module to predict the category $\left(C_{student}\right)$ and the corresponding bounding box $\left(B_{student}\right)$ of each candidate. Next, the same candidate boxes ${CB}_p$ are used as input for the teacher’s RoI module, which, through regression, generates the pseudo-label information:

$$\left\{C_s, B_s\right\}={Rol}_{teacher}\left({CB}_p\right)$$

(3)

The fully connected layer (FC) then obtains feature maps from the ROI Pooling, flattening them into a one-dimensional vector x. A weight matrix $W$ and a bias vector $b$ are applied to compute the linear transformation $Z$. The result $z$ of the linear transformation is passed through an activation function, yielding the output $Y$ after the nonlinear transformation.

$$Z=W_x+b$$

(4)

Our fully connected layer (FC) contains two fully connected layers, denoted as FC1 (Equation (5)) and FC2 (Equation (6)). The computation for each fully connected layer is as follows:

$$Z_1=W_{1}x+b_1$$

(5)

$$Y_1=MAX\left(0,Z_1\right)$$

(6)

The feature vector output from the fully connected layer (FC) is sent to two different branches for different tasks. The first branch applies a Softmax activation function for the classification task:

$$\mathit{cls\_prob}=Softmax\left(W_{cls}{Y}_2+b_{cls}\right)$$

(7)

The second branch uses a regression layer to obtain the bounding box predictions for each candidate region:

$$\mathit{bbox\_pred}=W_{bbox}{Y}_2+b_{bbox}$$

(8)

To introduce more diversity and complexity, aiding the model’s adaptation to nighttime features, we use low-confidence pseudo-labels during training while controlling their impact with a weighting mechanism. Each pseudo-label is assigned a weight based on its confidence. High-confidence pseudo-labels have a higher weight and greater impact on training, while low-confidence pseudo-labels have a lower weight and smaller impact. We employ the Total Variation (TV) Distance as our consistency loss:

$$L_{cons}=W·TV\left(C_{student} , C_s\right)$$

(9)

Here, $W$ denotes the highest confidence of $C_s$, represented as $W=max\left(C_s\right)$, and $TV\left(·\right)$ is the TV Distance function.

$$L_{sum}=L_{pre}+L_{cons}$$

(10)

Our total loss is represented by Equation (10), where $L_{pre}$ is given by Equation (1) and $L_{cons}$ is given by Equation (9).

3.2. Label Randomization

Our experiments revealed that detecting small objects at night is highly challenging. This challenge arises from the fact that small object features are easily obscured by night-specific environmental factors such as glare or low illumination. To enable the student network to overcome this obstacle, we enhance the input images and pseudo-labels at multiple scales. ${\mathrm{C}}_{\mathrm{i}\mathrm{l}\mathrm{p}}$ represents the original size of the input image. At the start of training, we concentrate on small features and then determine a new scaling ratio ${\mathrm{W}}_{\mathrm{u}\mathrm{n}}$ based on the number of network iterations. Additionally, to prevent the student network from overfitting due to prolonged exposure to one scale, we empirically allow the actual scaling ratio to fluctuate within 0.15 of ${\mathrm{W}}_{\mathrm{u}\mathrm{n}}$:

$${\mathrm{C}}_{\mathrm{s}\mathrm{t}\mathrm{u}\mathrm{d}\mathrm{e}\mathrm{n}\mathrm{t}}={\mathrm{C}}_{\mathrm{i}\mathrm{l}\mathrm{p}}+{\mathrm{W}}_{\mathrm{u}\mathrm{n}}\left(-\mathrm{0.15,0.15}\right)$$

(11)

Simultaneously, to prevent imbalances caused by excessively small sizes and misleading models due to excessively large sizes, labels with sizes below or above the threshold will be discarded.

3.3. LightImg

Because the purpose of this study is cross-domain recognition, we propose LightImg, as shown in Figure 3, a module that can directly convert existing nighttime datasets into daytime datasets, enabling the use of existing nighttime datasets to complete tasks. This is a lightweight module that does not require excessive computational resources [14,19]. The enhancement by LightImg aims to transform the features of nighttime images into those of daytime images. Daytime images generally have higher brightness, clearer light sources, and more detailed backgrounds compared to nighttime images. The dynamic range (HDR) of daytime images may also be higher. Additionally, outside city centers, the influence of artificial light sources is also reduced.

Let $Y\left(x, y\right)$ be the original nighttime image, where $L\left(x, y\right)$ and $R\left(x, y\right)$ represent the luminance and reflectance components, respectively. The relationship is given by

$$Y\left(x, y\right)=L\left(x, y\right)R\left(x, y\right)$$

(12)

We know that the relationship between the input nighttime image $\mathrm{y}$ and the output clear image $\mathrm{R}$ is given by $\mathrm{y}=\mathrm{Z}⨂\mathrm{R}$, where $\mathrm{Z}$ represents the illumination component. Generally, illumination is considered the core component that needs to be optimized for low-light image enhancement. According to the Retinex theory [21], the input image consists of two parts: the luminance component $\mathrm{L}$ and the reflectance component $\mathrm{R}$.

The enhanced results at different scales are linearly combined, taking into account both local and overall illumination components. This multi-scale algorithm ultimately produces images with good dynamic range compression, color stability, and color restoration. First, we divide the nighttime image $\mathrm{Y}(\mathrm{x},\mathrm{y})$ into several scale levels based on grayscale values and separate the three color components. Then, different Gaussian surround functions ${\mathrm{G}}_{\mathrm{k}}(\mathrm{x},\mathrm{y})$ are constructed according to the ratios of 15:80:250. These Gaussian surround functions are used to perform convolution filtering on the R, G, and B color channels, and the illumination components are obtained by weighted averaging:

$$L\left(x, y\right)={\sum }_{k=1}^N{\omega }_k\left(I_i\left(x, y\right) \ast G_k\left(x, y\right)\right)$$

(13)

Here, ${\sum }_{\mathrm{k}=1}^{\mathrm{N}}\mathsf{\omega }=1$, ${\mathrm{I}}_{\mathrm{i}}\left(\mathrm{x},\mathrm{y}\right)$ is the pixel value of the original image, with $\mathrm{N}=3$ in the RGB color space, and iii represents the three color channels. Then, we take the logarithm and subtract the illumination component from the original image:

$$\log {\mathrm{R}}_{\mathrm{i}}={\sum }_{\mathrm{k}=1}^{\mathrm{N}}{\mathsf{\omega }}_{\mathrm{k}}\left(\log \left({\mathrm{I}}_{\mathrm{i}}\left(\mathrm{x},\mathrm{y}\right)\right)\right)-\mathrm{L}\left(\mathrm{x},\mathrm{y}\right)$$

(14)

The previous steps may result in local detail color distortion and fail to reveal the true colors of objects. Therefore, we add a color restoration step to address this issue:

$$C_i\left(x, y\right)={\displaystyle \frac{I_i\left(x, y\right)}{\sum _{j=1}^N{I}_j\left(x, y\right)}}$$

(15)

$\mathrm{j}$ also represents the three color channels. By combining Equations (14) and (15), we obtain the calculation formula for our LightImg:

$$\log {\mathrm{R}}_{\mathrm{L}\mathrm{i}\mathrm{g}\mathrm{h}\mathrm{t}\mathrm{I}\mathrm{m}\mathrm{g}}=C_i\left(x, y\right)·\log {\mathrm{R}}_{\mathrm{i}}\left(\mathrm{x},\mathrm{y}\right)$$

(16)

Here, $\mathsf{\alpha }$ is used to adjust the nonlinear transformation, and $\mathsf{\beta }$ is the gain constant. Algorithm 1 demonstrates the process of a single enhancement using LightImg. The pseudocode for Equations (1)–(16) is presented in Algorithm 2.

Algorithm 1 LightImg Image Enhncement

Require: Nighttime Image $I$

Ensure: Enhanced Image $I_{enhanced}$

$R, G, B \leftarrow decompos\mathit{e\_c}hannels\left(I\right)$

$L_R \leftarrow Gaussian\left(R\right), L_G \leftarrow Gaussian\left(G\right), L_B \leftarrow Gaussian\left(B\right)$

$R_R \leftarrow \log \left(R\right)-\log \left(L_R\right), R_G \leftarrow \log \left(G\right)-\log \left(L_G\right), R_B \leftarrow \log \left(B\right)-\log \left(L_B\right)$

$C_R \leftarrow R_R\times \left(R+G+B\right)÷3, C_G \leftarrow R_G\times \left(R+G+B\right)÷3, C_B \leftarrow R_B\times \left(R+G+B\right) ÷3$

$I_{enhanced} \leftarrow merg\mathit{e\_c}hannels\left(C_R, C_G, C_B \right)$

Return $I_{enhanced}$

Algorithm 2 NUDN

Initialize $S$ with daytime images

Initialize $T$ as EMA of $S$

for each iteration do

Enhanced_Night_Images $\leftarrow$LightImg(Dn)

High_Confidence_Labels $\leftarrow$ GenerateAndFilterLabels(T,Enhanced_Night_Images)

Candidate_Boxes $\leftarrow$ Merge(Candidate_Boxes, High_Confidence_Labels)

ClassifyAndRegress(S, Combined_Boxes)

Consistency_Loss $\leftarrow$ComputeConsistencyLoss(S, T, Combined_Boxes)

UpdateNetworks(S, T, Consistency_Loss)

end for

return Trained Student Network S

4. Discussion

In this section, the proposed nighttime unsupervised cross-domain object detection method has been extensively evaluated on nighttime datasets. First, we describe the dataset and implementation settings, including hyperparameter values. Second, we compare the nighttime unsupervised cross-domain object detection method with state-of-the-art domain adaptation methods to evaluate its performance. Finally, we analyze the performance by adopting various trade-off parameter values and present ablation studies to understand the impact of each component on the overall performance of the proposed Knowledge Distillation-based Nighttime Unsupervised Cross-Domain Object Detection Network.

4.1. Experimental Setup or Environment

To evaluate our method, we compared its domain adaptation capabilities in object detection with state-of-the-art methods, including 2PCNet, TDD, and UMT [47]. Additionally, we demonstrated the robustness of our nighttime-to-daytime image translation module.

4.1.1. ExDark [49] Datasets

The ExDark [49] (Extreme Darkness) dataset is a large-scale low-light object detection dataset. It encompasses 10 different low-light scenarios, ranging from extremely low light to twilight, including environments such as city roads, residential areas, and parking lots. The ExDark [49] dataset contains 7363 images with 12 object categories, including dog, motorbike, person, cat, chair, table, car, bicycle, bottle, bus, cup, and boat, totaling 18,103 annotated instances. To address the current lack of real nighttime datasets, the ExDark [49] dataset provides a rich experimental foundation for researching object detection algorithms in low-light environments. By offering a substantial number of low-light images, the ExDark [49] dataset helps improve the accuracy of object detection in low-light conditions. In this experiment, the dataset was divided into 80% for the training set and 20% for the validation set. No additional preprocessing was performed.

4.1.2. SHIFT [50] Datasets

The SHIFT [50] (Synthetic and High-Resolution Image for Target Detection) dataset is a large-scale, high-resolution, synthetic image object detection dataset. It covers a variety of scenes, including urban roads, rural roads, residential areas, and commercial areas, simulating different weather conditions and lighting variations such as sunny, rainy, foggy, daytime, and nighttime. The SHIFT [50] dataset contains 50,000 high-resolution synthetic images with various object types, such as cars, pedestrians, and traffic signs, totaling 150,000 annotated instances. There are similarities between the categories in SHIFT [50] and ExDark [49]. Due to the difficulty of acquiring high-resolution images in real-world scenarios, the SHIFT [50] dataset provides extensive multi-scene, multi-lighting condition image data through synthetic techniques, greatly aiding the research and development of object detection algorithms. By offering a substantial number of high-resolution synthetic images, the SHIFT [50] dataset helps address detail recognition and detection accuracy issues in adverse environments. For our evaluation, we used images labeled “night” as our data. We further ensured that the weather label was “clear” to isolate other weather conditions from the evaluation. In this experiment, the SHIFT dataset was manually filtered to isolate the “nighttime” data. The dataset was then split into 80% for the training set and 20% for the validation set. No additional preprocessing was performed.

4.2. Experimental Setup

The proposed NUDN method is implemented using Faster-RCNN [7]. The hardware configuration includes 96 GB RAM, an Nvidia RTX A4000 (16 GB) graphics card, and an Intel Xeon processor. There are two reasons for selecting Faster-RCNN as the backbone network for NUDN. First, it can generate high-quality candidate regions (RPN) that are classified and subjected to bounding box regression by the network, significantly reducing computational overhead and increasing detection speed. Second, other existing methods also use Faster-RCNN as the backbone network. Thus, using Faster-RCNN allows for a fair comparison of the proposed NUDN method with other state-of-the-art methods. The main parameters and environment of the experiment are the same as those in

Table 1. Environment and parameterization of the experiment.

Parameters	Configuration
CPU	Intel Xeon Gold 5320 @ 2.20 GHz
GPU	RTX A4000(16 GB) ×3
System	Ubuntu 20.04
Deep learning architecture	PyTorch1.11.0 + Python 3.8 + Cuda11.3
Training Epochs	60,000
Batch size	12
exponential moving average (EMA)	0.999
Base	0.04
threshold	0.8

.

We utilized all labeled source samples and unlabeled target samples. The network was trained for 60 k iterations with a batch size of 12, and the exponential moving average (EMA) was set to 0.999. The base learning rate was set to 0.04 and then decayed by a factor of 0.5 to find the optimal solution (0.04, 0.02, 0.01, 0.005). The threshold of 0.8 was used to filter out low-confidence predictions. The size of the training and testing images was adjusted to a length of 600 pixels.

4.3. Evaluation Indicators

In this study, we evaluate the models using AP, APl, APm, and APs. These metrics reflect the average precision, large object precision, medium object precision, and small object precision, respectively. AP measures the overall accuracy of the model in predicting positive classes, with higher values indicating a better capability to capture positive samples. AP is one of the primary evaluation metrics used in object detection tasks, as it reflects the model’s accuracy across multiple categories. An increase in AP indicates improved detection performance across all categories. We use the strictest metric by default, setting all IoU (Intersection over Union) thresholds to 0.5:0.95. Calculating the average AP over IoU thresholds from 0.5 to 0.95 (with intervals of 0.05) allows for a more accurate assessment of the model’s overall performance under different IoU conditions. APl: Large objects with an area greater than 96 × 96 pixels. APm: Medium objects with an area between 32 × 32 and 96 × 96 pixels. APs: Small objects with an area less than 32 × 32 pixels. The following formulas demonstrate how different metrics are calculated:

$$AP={\displaystyle \frac{ TP+TN }{TP+TN+FP+FN}}$$

(17)

$$APl,APm,APs={\displaystyle \frac{1 }{n}}{\int }_0^1{p}_i\left(r\right)$$

(18)

True Positives (TP) represent the number of positive samples correctly predicted as positive. False Positives (FP) represent the number of negative samples incorrectly predicted as positive. False Negatives (FN) represent the number of positive samples incorrectly predicted as negative. True Negatives (TN) represent the number of negative samples correctly predicted as negative.

In the image enhancement comparison experiment, we employed three scoring criteria: PSNR, SSIM, and LPIPS, which represent peak signal-to-noise ratio, structural similarity, and learned perceptual image patch similarity, respectively. For PSNR and SSIM, higher values indicate better performance. For LPIPS, lower values are preferred. Their calculation methods are detailed as follows:

$$PSNR=20· {\log }_{10}{\displaystyle \frac{ {MAX}_I }{\sqrt{MSE}}}$$

(19)

${\mathrm{M}\mathrm{A}\mathrm{X}}_{\mathrm{I}}$ represents the maximum value of the image pixels, while $\mathrm{M}\mathrm{S}\mathrm{E}$ denotes the mean squared error between the original image and the compressed image.

$$SSIM\left(x , y\right)={\displaystyle \frac{ \left({2\mu }_x{\mu }_{y }+C_1\right)\left(2{\sigma }_{xy}+C_2\right)}{\left({\mu }_x^2+{\mu }_y^2+C_1\right)\left({\sigma }_x^2+{\sigma }_y^2+C_2\right)}}$$

(20)

where ${\mathsf{\mu }}_{\mathrm{x}}$ and ${\mathsf{\mu }}_{\mathrm{y}}$ are the mean values of image blocks $\mathrm{x}$ and $\mathrm{y}$, ${\mathsf{\sigma }}_{\mathrm{x}}^2$ and ${\mathsf{\sigma }}_{\mathrm{y}}^2$ are the variances of image blocks $\mathrm{x}$ and $\mathrm{y}$, and ${\mathsf{\sigma }}_{\mathrm{x}\mathrm{y}}$ is the covariance of image blocks $\mathrm{x}$ and $\mathrm{y}$.

${\mathrm{C}}_1$ and ${\mathrm{C}}_2$ are small constants used to stabilize the denominator, ${\mathrm{C}}_1={\left({\mathrm{K}}_1\mathrm{L}\right)}^2$, ${\mathrm{C}}_2={\left({\mathrm{K}}_2\mathrm{L}\right)}^2$, where ${\mathrm{K}}_1$ = 0.01 and ${\mathrm{K}}_2$ = 0.03; $\mathrm{L}$ represents the dynamic range of pixel values.

$$LPIPS\left(x , y\right)={\sum }_l{\omega }_l · d_l\left({\varphi }_l\left(x\right), {\varphi }_l\left(y\right)\right)$$

(21)

${\mathsf{\varphi }}_{\mathrm{l}}\left(\mathrm{x}\right)$ and ${\mathsf{\varphi }}_{\mathrm{l}}\left(\mathrm{y}\right)$ represent the features of images $\mathrm{x}$ and $\mathrm{y}$ at the $\mathrm{l}$th layer, ${\mathrm{d}}_{\mathrm{l}}$ denotes the L2 distance between the features, and ${\mathsf{\omega }}_{\mathrm{l}}$ is the weight assigned to each layer’s features. By integrating these three mainstream indicators, we can objectively assess the quality of the LightImg module.

4.4. Datasets

4.4.1. Comparative Experiments on the ExDark [49] Datasets

We compared our method with state-of-the-art domain adaptation methods, including TDD [33], 2PCNet [41], and UMT [47], is shown in

Table 2. The results of cross-domain adaptation on the ExDark datasets are reported. The average precision (AP) of various methods is presented, along with the average precision for small objects (APs), medium objects (APm), and large objects (APl).

Method	AP	APs	APm	APl
TDD [33]	34.6	12.1	28.3	39.1
2PCNet [41]	39.7	8.3	25.0	36.6
UMT [47]	36.2	10.8	27.4	34.6
NUDN (Ours)	37.4	14.1	28.8	42.7

. Compared to the TDD [33] method, our method improves the average precision (AP) by 8.09%. Although our AP is 6.15% lower than last year’s SOTA algorithm 2PCNet [41], we outperform 2PCNet [41] in terms of average precision for small objects (APs), medium objects (APm), and large objects (APl). Notably, we achieve a substantial improvement of 69.88% in APs for small objects. We also conducted a visual comparison of the ExDark dataset, as shown in Figure 4.

Based on the visual results shown in Figure 4, we observe clear advantages of our method in detecting small objects and reducing false positives. The TDD method exhibits a significant number of false positives in this environment, while 2PCNet demonstrates complete insensitivity to small objects at night. In comparison to TDD and 2PCNet, our approach exhibits stronger adaptability in low-light conditions, particularly in accurately identifying small objects within complex backgrounds. This is attributed to the effective image enhancement provided by the LightImg module and the application of consistency learning strategies, which enable the model to maintain high detection accuracy across different scenarios.

4.4.2. Comparative Experiments on the SHIFT Datasets

To further compare our method with other methods, we conducted evaluations on the SHIFT [50] datasets. Due to the synthetic nature of the datasets, many nighttime image characteristics present in real datasets, such as glare, noise, illumination, and blur, are not reflected. The experimental results are shown in

Table 3. This paper reports the comparative results of TDD, 2PCNet, UMT, and our Method in cross-domain adaptation using the nighttime dataset from SHIFT.

Method	AP	APs	APm	APl
TDD [33]	33.2	10.1	29.2	37.2
2PCNet [41]	44.7	11.3	28.0	39.6
UMT [47]	32.4	9.1	24.8	29.3
NUDN (ours)	39.6	16.2	30.4	45.2

. We observe that the UMT [47] and TDD [33] methods both exhibit a decline in performance, likely due to error propagation. Although our method has a lower AP performance compared to the 2PCNet [41] method, we outperform 2PCNet [41] in APs, APm, and APl. The visualization results are shown in Figure 5.

As illustrated in Figure 5, our method demonstrates superior robustness compared to other methods in the testing phase of the SHIFT dataset, particularly excelling in the detection of small objects. Despite the synthetic nature of the nighttime images in the SHIFT dataset, which lack certain features present in real-world scenarios (such as glare and noise), our method continues to maintain high detection accuracy while reducing false positives. This further validates the adaptability and generalizability of our approach across various scenarios.

4.4.3. Ablation Studies

In this section, we systematically investigate the impact of three enhancement techniques on the network model. The experimental results are shown in

Table 4. Ablation studies were conducted on the ExDark dataset to evaluate the contributions of different components of our method. The components tested include consistency learning (CL), LightImg (I), and Label Randomization (LR).

Structure	I	CL	LR	AP	APs	APm	APl
Single				31.7	4.8	16.5	30.4
	✓			36.8	9.1	23.3	41.4
	✓	✓		36.2	9.4	24.7	42.9
NUDN(ours)	✓	✓	✓	37.4	14.1	28.8	42.7

. Our model is based on the Faster-RCNN network. First, we evaluate the effectiveness of the Consistency Learning backbone network on the entire network. Second, we verify the impact of LightImg on the network. Finally, we examine the combined effect of Consistency Learning, LightImg, and Label Randomization on the network.

Consistency Learning: As shown in Table 4, there is a significant difference before and after adding Consistency Learning (CL) to the base model, with an improvement of +5.1 AP (16.09%).

LightImg: This method transforms nighttime images into daytime images. As shown in Figure 1, the addition of LightImg (I) not only converts nighttime objects into daytime objects but also increases the detection rate of small objects by 6%.

Label Randomization: This module scales pseudo-labels and images processed by LightImg (I). It is particularly beneficial for small object detection, showing an improvement of 4.7 AP (50%) over Faster-RCNN + Consistency Learning + LightImg.

Further analysis revealed that LightImg not only directly improves the quality of low-light images but also indirectly enhances the effectiveness of consistency learning by increasing the reliability of input data, thereby improving the efficiency of label randomization. This indicates that LightImg serves not only as an independent image enhancement module but also as a means to optimize the overall synergy of various components, significantly enhancing the model’s detection capability under low-light conditions at night.

4.4.4. LightImg Time and Performance Comparison

To further compare the image enhancement capabilities of our LightImg method, we evaluated the performance of the LightImg component on the ExDark dataset. Figure 6 and

Table 5. Quantitative comparison was conducted between the current state-of-the-art low-light image enhancement methods. The best results are in bold. The second best results are underlined. Each method has its own advantages.

Method	ExDark			Times (ms)
	PSNR↑	SSIM↑	LPIPS↓
KinD++ [33]	13.44	0.570	0.205	281.4
LIME [41]	11.99	0.593	0.193	828.6
SCI [47]	10.25	0.470	0.306	8.7
LightImg (Ours)	11.36	0.522	0.258	42.6

jointly present the performance of LightImg in low-light image enhancement. Compared to LIME and KinD++, LightImg not only excels in enhancing image brightness but also demonstrates significant advantages in maintaining natural colors and enhancing local contrast. LIME often leads to overexposure, while KinD++ is less effective at enhancing dark regions. LightImg, through precise illumination control, avoids these issues, resulting in images that are more balanced both overall and in detail after enhancement.

The visual comparison results in Figure 6 further demonstrate the effectiveness of our proposed LightImg method in enhancing low-light images. Compared to the LIME and KinD++ methods, LightImg not only excels in brightness enhancement but also exhibits significant advantages in preserving natural colors and improving local contrast. Notably, the LIME method often results in overexposure, while KinD++ performs poorly in enhancing dark areas. In contrast, LightImg, through precise illumination control, effectively avoids these issues, ensuring that the enhanced images exhibit higher visual quality both overall and in detail.

Table 5 provides further quantitative performance metrics. The PSNR and SSIM values demonstrate LightImg’s superiority in maintaining high image quality, while the lower LPIPS values indicate that its output images are more natural in visual perception. Although the processing time is slightly higher than that of the SCI method, LightImg strikes an excellent balance between time and quality, with a processing time significantly lower than that of LIME, meeting both practical and performance requirements. These results indicate the applicability and efficiency of LightImg in real-world scenarios, particularly in situations requiring high-quality image enhancement.

Although LightImg performs excellently in various low-light nighttime scenarios, its ability to generate effective features may be limited under extremely challenging conditions such as very low lighting or strong backlighting (Figure 7). Experimental results in these scenarios indicate a decline in image enhancement effectiveness, which may also impact the quality of pseudo-labels [52]. This analysis not only reveals potential limitations of the method but also provides important directions for future optimization [53,54].

5. Conclusions

This paper proposes a network, NUDN, capable of performing cross-domain training using only existing low-light images. Specifically, during the pre-training phase, NUDN employs LightImg to enhance night-time images into daytime images, which are then fed to the teacher network for learning. In the formal learning phase, pseudo-labels generated by the teacher network are ranked by confidence level and assigned to the student network for learning. Additionally, we designed a Label Randomization technique to guide the model in identifying target locations across various images. NUDN significantly reduces the workload associated with existing day-night cross-domain tasks, demonstrating its effectiveness and applicability.

Through the analysis of Figure 4, Figure 5 and Figure 6, it is clear that the proposed NUDN method exhibits exceptional adaptability and performance across various nighttime scenes and low-light conditions. This success is not only attributed to the outstanding performance of the LightImg module in image enhancement but also to the synergistic combination of consistency learning and label randomization strategies, allowing the model to sustain high detection accuracy and robustness in diverse scenarios. In summary, these experimental results validate the effectiveness of our method and its potential for practical applications.

Extensive quantitative and qualitative experiments demonstrate that our proposed method effectively addresses the issue of sparse datasets in cross-domain day-night recognition, significantly outperforming existing methods. In future work, we will explore research on day-night object re-identification.