Low-Light Image Contrast Enhancement with Adaptive Noise Attenuator for Augmented Vehicle Detection

Abstract

The rapid progress in deep learning technologies has accelerated the use of object detection models, but most models do not operate satisfactorily in low-light environments. As a result, many studies have been conducted on image enhancement techniques aiming to make objects more visible by increasing contrast, but the process of image enhancement may negatively impact detection as it further strengthens unwanted noises due to indirect factors of light reflection such as overall low brightness, streetlamps, and neon signboards. Therefore, in this study, we propose a technique for improving the performance of object detection in low-light environments. The proposed technique inverts a low-light image to make it similar to a hazy image and then uses a haze removal algorithm based on entropy and fidelity to increase image contrast, clarifying the boundary between the object and the background. In the next step, we used the adaptive 2D Wiener filter (A2WF) to attenuate the noise accidentally strengthened during the image enhancement process and reinforced the boundary between the object and the background to increase detection performance. The test evaluation results showed that the proposed image enhancement scheme significantly increased image perception performance with the perception-based image quality evaluator being 12.73% lower than existing image enhancement techniques. In a comparison of vehicle detection performance, the proposed technique for enhancing nighttime images combined with the detection model proved its effectiveness by increasing the average precision by up to 18.63% against existing detection methods.

1. Introduction

As deep learning-based object detection models are continuously improving their performances, they are being deployed in various transportation fields, such as intelligent transport systems, emergency vehicle detection, and road-monitoring systems. Such systems must always operate in various environmental conditions during the daytime, nighttime, and in rain. However, most studies on object detection have been conducted in sufficiently bright environments and would not work satisfactorily in low-light environments, such as during nighttime. Generally, low-light images have low contrast, which blurs the boundary between an object and its background, which decreases detection performance. This prompted many studies to focus on techniques for enhancing the contrast of images in low-light environments [1,2,3].

A classic technique for improving the contrast of a low-light image is histogram equalization (HE) [4]. Generally, low-light images have very low contrast, and their pixels are clustered around certain values. The HE spreads out the skewed pixel values to increase the contrast, and its simple structure allows for easy implementation. However, brighter areas tend to get saturated, leading to a loss of detailed information. Lee et al. [5] proposed a layered difference representation of 2D histograms, which improved HE performance by adding a different normalization term to the histogram; however, it still risks color distortion due to over- or under-enhancement of contrast. Another major method for improving a low-light image occurs from the retinex theory [6], which was used by Jobson et al. [7] as a basis to propose the single-scale retinex (SSR). The SSR constrains an illumination map smoothly through a Gaussian filter. They also proposed multiscale retinex (MSR) [8] that extends the SSR through multiscale Gaussian filters and color restoration to enhance images. Guo et al. [9] proposed a method for estimating an initial illumination map from the maximum pixel values for each R, G, and B channel of an image to design a final stable map. However, in some cases, the contrast enhancement was too high, making the image look unnatural. Recently, research has been conducted to tackle such problems using a deep learning approach.

Lore et al. [10] presented, for the first time, a deep auto-encoder-based approach that adaptively enhances a low-light image without over-amplifying the image’s brighter areas. Pham et al. [11] used the retinex theory to design a two-stage neural network: In stage 1, a low-light image was separated into illumination and reflection maps; in stage 2, the illumination map was fine-tuned in a wide area as the image quality was enhanced. Fu et al. [12] applied contrastive learning to enhance low-light images with a reflectance module to extract reflectance features between low-light images and normal images, an illumination module to extract illumination features using four transformer layers and two convolution layers, and a module to fuse features from each module. Fully training a deep learning model requires a large amount of high-quality data; however, obtaining high light ground truth images for low-light images is very difficult and makes training costly. Zhao et al. [13] proposed to use deep image priors to decompose an image into components of the retinex theory, and then enhance each component through a deep learning network, followed by fusion to enhance the image. Liu et al. [14] proposed a network that enhances image contrast firstly through stretching coefficient-based enhancement and finally through a fusion network with two encoders that combine the low-light image and the initial enhanced image. Zhang et al. [15] proposed a DCC-Net consisting of a G-Net to recover gray images, a C-Net to identify color distributions, and an R-Net to enhance image contrast through fusion of the G-Net and C-Net. Zhou et al. [16] removed the noise present in low-light images through a low-exposure denoiser, enhanced the image contrast through a multi-branch retinex process, and fused the features through an ECA block to finally obtain an image with improved contrast. These methods also designed image enhancement models in consideration of noise that may occur in low-light environments, which is what we are interested in, but since artificially generated data were used in the performance evaluation, there is no way to check their performance in real low-light environments. In addition, since they are not being extended by linking with object detection networks, there is no guarantee that detection performance will be improved through these image enhancement techniques.

As a novel approach for enhancing low-light images, Dong et al. [17] recently proposed a method of image enhancement derived from an observation that an inverted low-light image is similar to a hazy image. The new method applied a haze removal algorithm based on the dark channel prior (DCP) to an inverted low-light image to increase image contrast. In an efficient image enhancement technique proposed by Zhang et al. [18], a bilateral filter was added to the method [17] to remove noise enhanced during the image enhancement process. Lin et al. [19] proposed another technique for enhancing images in which block-matching and 3D filtering (BM3D) and a structural filter were used to remove noise before image processing. Additionally, several other methods have been proposed to further enhance visual quality by effectively removing noise through various filters.

As such, despite many studies on low-light image enhancement primarily aiming to improve visual quality, there have been very limited endeavors to incorporate these enhanced images with real-world detection models to attain superior detection performance. Furthermore, datasets used to evaluate the performance of an object detection algorithm in low-light environments are often limited to artificial data with arbitrarily darkened images. Artificial data are clearly different from images taken in actual low-light environments because actual low-light images contain various noises that cannot be seen in bright environments. For this reason, noise in low-light images may get stronger during the process of image enhancement, causing the boundaries between objects and backgrounds to blur and partially obscure the objects. As a result, it leads to a degradation in the performance of the detection model. Therefore, in this paper, we proposed an image enhancement technique that contributes to an object detection model instead of simply improving image quality. The proposed model first applies a dehazing algorithm based on image entropy to an inverted low-light image to increase contrast between an object and its background for image enhancement. Here, the DCP-based dehazing technique applied in previous works such as [17] often causes dehazing errors due to white objects, buildings, bright light sources, etc., during image processing, and to solve this drawback, we changed to an entropy-based approach for image enhancement. However, the technique also enhanced noise in the image during the image enhancement process as attempted in previous works such as [18,19]; therefore, it is not yet directly helpful to the object detection model. Consequently, we used an adaptive 2D Wiener filter for the enhanced low-light image to suppress noise effects and preserve the boundary between the object and its background, resulting in improved performance of object detection. Our proposed technique offers the following contributions:

• The technique enhanced the contrast of low-light images more naturally than a commonly used DCP-based method by using an object function that consists of image entropy and fidelity;
• The application of the adaptive 2D Wiener filter to the enhanced low-light image and suppression of increased noise from the enhancement process preserved the contrast of the boundary between the object and the background and improved the performance of object detection;
• This image enhancement technique can be used as a preprocessor for various deep learning-based detection models, thus having general-purpose utility.

2. Image Contrast Enhancement

In low-light environments during nighttime, there is little or no lighting, making the visual range very short. This blurs the boundary between an object and the background, which is the main culprit for degrading the performance of an object detection model. As shown in Figure 1, compared to the low-light image in Figure 1a, the inverted low-light image in Figure 1b and the hazy image in Figure 1c appear very similar due to the high pixel intensity in the background and the low pixel intensity in at least one of the 3 RGB channels of the colors of the main objects, such as vehicles and people [17].

Such similarity prompted recent research to propose new methods for enhancing low-light images by applying the DCP-based dehazing technique to inverted low-light images [9,10,11]. However, the DCP-based dehazing technique for image enhancement was limited because DCP estimation was not performed properly on certain colors such as white objects or buildings. Therefore, in this study, we proposed a method to use an object function composed of image entropy and fidelity to avoid certain color constraints in a commonly used DCP-based dehazing technique, while also enhancing low-light images efficiently and naturally.

2.1. Atmospheric Scattering Model

Most haze removal techniques by default use the atmospheric scattering model to de-haze, representing a blurry image $\mathrm{L}\left(\mathrm{x},\mathrm{y}\right)$ as Equation (1):

$$\mathrm{L}\left(\mathrm{x},\mathrm{y}\right)=\mathrm{B}\left(\mathrm{x},\mathrm{y}\right)\mathrm{T}\left(\mathrm{x},\mathrm{y}\right)+\mathrm{A}(1-\mathrm{T}\left(\mathrm{x},\mathrm{y}\right))$$

(1)

Here, $\mathrm{B}\left(\mathrm{x},\mathrm{y}\right)$ is the ${\left(\mathrm{x},\mathrm{y}\right)}^{\mathrm{t}\mathrm{h}}$ pixel value of the clean image, $\mathrm{A}$ is the atmospheric light, and $\mathrm{T}\left(\mathrm{x},\mathrm{y}\right)$ is the ${\left(\mathrm{x},\mathrm{y}\right)}^{\mathrm{t}\mathrm{h}}$ pixel value of the transmission map, which corresponds to a proportion of the light from an object reaching the camera. If we rearrange Equation (1) with $\mathrm{B}\left(\mathrm{x},\mathrm{y}\right)$ as the subject, a clean image is obtained from estimates of the atmospheric light and the transmission map such as $\overline{\mathrm{A}}$ and $\overline{\mathrm{T}}\left(\mathrm{x},\mathrm{y}\right)$ as shown in Equation (2):

$$\mathrm{B}\left(\mathrm{x},\mathrm{y}\right)=\frac{\mathrm{L}\left(\mathrm{x},\mathrm{y}\right)-\overline{\mathrm{A}}}{\overline{\mathrm{T}}(\mathrm{x},\mathrm{y})}+\overline{\mathrm{A}}$$

(2)

2.2. Atmospheric Light Estimation

Haze in images develops from color distortion, which occurs when the colors are not balanced due to light scattering, caused by particulate matter in the atmosphere. Therefore, to restore a clean image, we must estimate the atmospheric light. Park et al. [17] hypothesized that the atmospheric light is scattered across a hazy image, with the light intensity being highest in a local region. From this concept, they proposed a method for estimating the atmospheric light through quad-tree subdivision. First, a blurry color image $\mathrm{L}\left(\mathrm{x},\mathrm{y}\right)$ is converted to a grayscale image ${\mathrm{L}}_{\mathrm{g}\mathrm{r}\mathrm{y}}\left(\mathrm{x},\mathrm{y}\right)$ as shown in Equation (3):

$${\mathrm{L}}_{\mathrm{g}\mathrm{r}\mathrm{y}}\left(\mathrm{x},\mathrm{y}\right)=0.299\times {\mathrm{L}}_{\mathrm{R}}\left(\mathrm{x},\mathrm{y}\right)+0.587\times {\mathrm{L}}_{\mathrm{G}}\left(\mathrm{x},\mathrm{y}\right)+0.114\times {\mathrm{L}}_{\mathrm{B}}(\mathrm{x},\mathrm{y})$$

(3)

where the subscripts R, G, B indicate a color channel. If we divide ${\mathrm{L}}_{\mathrm{g}\mathrm{r}\mathrm{y}}\left(\mathrm{x},\mathrm{y}\right)$ into a block of size $\mathrm{M}\times \mathrm{M}$, and define the ${\alpha }^{\mathrm{t}\mathrm{h}}$ segment of the block ${\mathrm{L}}_{\mathrm{g}\mathrm{r}\mathrm{y}}^{\mathrm{b}\mathrm{l}\mathrm{k}}$, subdivided into a total of $\beta$, as ${\mathrm{L}}_{\mathrm{g}\mathrm{r}\mathrm{y}}^{\mathrm{b}\mathrm{l}\mathrm{k}}$, we get Equation (4) for minimizing the error from local objects as:

$${\mathrm{H}}_{\mathrm{g}\mathrm{r}\mathrm{y}}^{\mathrm{b}\mathrm{l}\mathrm{k}}\left(\alpha \right)=\underset{\alpha =1,\cdots ,\beta }{\min }{\mathrm{L}}_{\mathrm{g}\mathrm{r}\mathrm{y}}^{\mathrm{b}\mathrm{l}\mathrm{k}}\left(\alpha \right)$$

(4)

Here, ${\mathrm{H}}_{\mathrm{g}\mathrm{r}\mathrm{y}}^{\mathrm{b}\mathrm{l}\mathrm{k}}\left(\alpha \right)$ is the ${\mathsf{\alpha }}^{\mathrm{t}\mathrm{h}}$ converted segment of the size-$\beta$ block and has a lower intensity value than ${\mathrm{L}}_{\mathrm{g}\mathrm{r}\mathrm{y}}^{\mathrm{b}\mathrm{l}\mathrm{k}}\left(\alpha \right)$ overall, allowing for the minimization of the error caused by white objects. As a result, it is selected as a candidate region for estimating the atmospheric light. We repeat the above process multiple times to finally select the sky region. Among the pixels in the selected region, we use a color vector of the ${\mathrm{k}}^{\mathrm{t}\mathrm{h}}$ pixel that minimizes the Euclidean norm to estimate the atmospheric light.

$$\overline{\mathrm{A}}=‖\left({\mathrm{R}}_{\mathrm{k}},{\mathrm{G}}_{\mathrm{k}},{\mathrm{B}}_{\mathrm{k}}\right)-(\mathrm{1,1},1)‖$$

(5)

2.3. Transmission Map Estimation

The transmission map indicates the proportion of light transmission between the object and the camera and is an important parameter along with the atmospheric light in the haze removal process. It can be estimated using two (2) objective functions: ${\mathrm{B}}_{\mathrm{e}\mathrm{n}\mathrm{t}}$ account for the image contrast and ${\mathrm{B}}_{\mathrm{f}\mathrm{i}\mathrm{d}}$ represent the fidelity of the haze removal [20]. Blurry images tend to have lower entropy than clean images because the objects are hazy and obscured by fog. The image entropy can be expressed using a function of the transmission value $\mathsf{\gamma }$, as shown in Equation (6):

$${\mathrm{B}}_{\mathrm{e}\mathrm{n}\mathrm{t}}\left(\mathsf{\gamma }\right)=-\sum _{\mathrm{j}=0}^{255}\frac{\mathrm{G}{\mathrm{B}}_{\mathrm{j}}\left(\mathsf{\gamma }\right)}{\mathrm{N}}\mathrm{l}\mathrm{o}\mathrm{g}\frac{\mathrm{G}{\mathrm{B}}_{\mathrm{j}}\left(\mathsf{\gamma }\right)}{\mathrm{N}}$$

(6)

Here, $\mathrm{N}$ is the total number of pixels in the image and ${\mathrm{G}\mathrm{B}}_{\mathrm{j}}\left(\mathsf{\gamma }\right)$ is the number of pixels with intensity $\mathrm{j}$ in the grayscale image of $\mathrm{B}(\mathrm{x},\mathrm{y})$ estimated using Equation (2), with the value $\overline{\mathrm{T}}(\mathrm{x},\mathrm{y})$ estimated as $\mathsf{\gamma }$. If $\mathsf{\gamma }$ value becomes small, the $\mathrm{B}(\mathrm{x},\mathrm{y})$ value will be less than 0 or greater than 255 by Equation (2), causing an excessive color distortion. To prevent such an underflow or overflow, and to obtain reliable dehazing results, we defined ${\mathrm{B}}_{\mathrm{f}\mathrm{i}\mathrm{d}}$ as shown in Equation (7):

$${\mathrm{B}}_{\mathrm{f}\mathrm{i}\mathrm{d}}\left(\mathsf{\gamma }\right)=\underset{\mathrm{c}\in \{\mathrm{R},\mathrm{G},\mathrm{B}\}}{\min }{\mathrm{P}}_{\mathrm{c}}\left(\mathsf{\gamma }\right)$$

(7)

Here, ${\mathrm{P}}_{\mathrm{c}}\left(\mathsf{\gamma }\right)$ is the ratio of pixels between 0 and 255 in each color channel of $\mathrm{B}(\mathrm{x},\mathrm{y})$, estimated using Equation (2), when $\overline{\mathrm{T}}(\mathrm{x},\mathrm{y})$ is estimated to be $\mathsf{\gamma }$. Therefore, ${\mathrm{B}}_{\mathrm{f}\mathrm{i}\mathrm{d}}$ represents a minimum ratio of pixels in the RGB color channels that do not have an underflow or overflow, and the larger the number, the less is the color distortion. To summarize, the objective function for estimating a transmission map is written as shown in Equation (8):

$${\mathrm{B}}_{\mathrm{o}\mathrm{b}\mathrm{j}}\left(\mathsf{\gamma }\right)={\mathrm{B}}_{\mathrm{e}\mathrm{n}\mathrm{t}}\left(\mathsf{\gamma }\right)\times {\mathrm{B}}_{\mathrm{f}\mathrm{i}\mathrm{d}}\left(\mathsf{\gamma }\right)$$

(8)

Finally, we estimated $\overline{\mathrm{T}}(\mathrm{x},\mathrm{y})$ for Equation (8) to have the maximum, but $\mathrm{T}(\mathrm{x},\mathrm{y})$ from an actual hazy image is not uniform, allowing the estimation error to get bigger due to certain regions with or without fog. For this reason, we divided the blurry image into $\mathrm{M}\times \mathrm{M}$ sub-blocks with no overlap and estimated a local optimal transmission for each sub-block in Equation (9):

$${\mathrm{T}}_{\mathrm{n}}^{\mathrm{b}\mathrm{l}\mathrm{k}}=\underset{0.01\le \mathsf{\gamma }\le 1}{\mathrm{argmax}}{\mathrm{B}}_{\mathrm{o}\mathrm{b}\mathrm{j}}\left(\mathsf{\gamma }\right)$$

(9)

Here, ${\mathrm{T}}_{\mathrm{n}}^{\mathrm{b}\mathrm{l}\mathrm{k}}$ is the nth sub-block with the estimated local optimal transmission, and $\overline{\mathrm{T}}$ expressed as in Equation (10). Using the final estimates $\overline{\mathrm{A}}$ and $\overline{\mathrm{T}}$ along with Equation (2), we restored a clean image.

$$\overline{\mathrm{T}}=\left[\begin{array}{ccc}{\mathrm{T}}_1^{\mathrm{b}\mathrm{l}\mathrm{k}}& \cdots & {\mathrm{T}}_{\mathrm{M}}^{\mathrm{b}\mathrm{l}\mathrm{k}}\\ ⋮& \ddots & ⋮\\ & \cdots & {\mathrm{T}}_{\mathrm{M}\mathrm{M}}^{\mathrm{b}\mathrm{l}\mathrm{k}}\end{array}\right]$$

(10)

2.4. Image Contrast Enhancement

As mentioned earlier, an inverted low-light image is similar to a hazy image. Therefore, to make a low-light image $\mathrm{I}(\mathrm{x},\mathrm{y})$ look similar to a blurry image $\mathrm{L}(\mathrm{x},\mathrm{y})$ in Equation (1), we first converted it to an inverted low-light image ${\mathrm{I}}_{\mathrm{i}\mathrm{n}\mathrm{v}}(\mathrm{x},\mathrm{y})$ as shown in Equation (11):

$${\mathrm{I}}_{\mathrm{i}\mathrm{n}\mathrm{v}}\left(\mathrm{x},\mathrm{y}\right)=255-\mathrm{I}(\mathrm{x},\mathrm{y})$$

(11)

We applied ${\mathrm{I}}_{\mathrm{i}\mathrm{n}\mathrm{v}}(\mathrm{x},\mathrm{y})$ along with $\overline{\mathrm{A}}$ and $\overline{\mathrm{T}}(\mathrm{x},\mathrm{y})$ to Equation (2) to get an enhanced image $\mathrm{B}(\mathrm{x},\mathrm{y})$ with higher contrast. The final enhanced image ${\mathrm{I}}_{\mathrm{E}}(\mathrm{x},\mathrm{y})$ restored to the RGB color channels by color inversion is expressed by the following equation:

$${\mathrm{I}}_{\mathrm{E}}\left(\mathrm{x},\mathrm{y}\right)=255-\mathrm{B}(\mathrm{x},\mathrm{y})$$

(12)

The image contrast enhancement process for low-light images is depicted in Figure 2.

2.5. Noise Attenuation with an Adaptive 2D Wiener Filter

Images taken outdoors are susceptible to noise because of numerous factors. Especially, images taken in low-light environments have poor visual range and are more susceptible to noise. Applying image enhancement techniques to such noisy images can also unintentionally strengthen the noise, interfering with subsequent data processing, such as object detection. To reduce these noise effects, Lim et al. [21] used an adaptive 2D Wiener filter (A2WF) that works adaptively to the image per pixel and is particularly effective in removing the additive white Gaussian noise.

$${\mathrm{I}}_{\mathrm{D}}\left(\mathrm{x},\mathrm{y}\right)=\mathsf{\mu }+\frac{{\mathsf{\sigma }}^2-{\mathrm{v}}^2}{{\mathsf{\sigma }}^2}({\mathrm{I}}_{\mathrm{E}}\left(\mathrm{x},\mathrm{y}\right)-\mathsf{\mu })$$

(13)

$$\mathsf{\mu }=\frac{1}{\mathrm{P}\mathrm{Q}}\sum _{\mathrm{x},\mathrm{y}\in \mathrm{M}}{\mathrm{I}}_{\mathrm{E}}(\mathrm{x},\mathrm{y})$$

(14)

$${\mathsf{\sigma }}^2=\frac{1}{\mathrm{P}\mathrm{Q}}\sum _{\mathrm{x},\mathrm{y}\in \mathrm{M}}{\mathrm{I}}_{\mathrm{E}}{\left(\mathrm{x},\mathrm{y}\right)}^2-{\mathsf{\mu }}^2$$

(15)

Here, ${\mathrm{I}}_{\mathrm{D}}$ is the image that we obtained after suppressing the noise from image enhancement; $\mathsf{\mu }$ and ${\mathsf{\sigma }}^2$ are the local mean and the local variance with the size $\mathrm{P},\mathrm{Q}$ of the input image pixel $(\mathrm{x},\mathrm{y})$; and ${\mathrm{v}}^2$ is the variance of the noise present in the image. If the variance of the noise is not given, we must use the mean of all local variances. In other words, if the local variance of the A2WF is large, the pixels in the local region are smoothed significantly; otherwise, very little smoothing occurs. Therefore, the filter functions adaptively to local variance and has higher selectivity than a linear filter. As a result, the edges and other high-frequency regions with high rates of change are well preserved. The image-denoising process is depicted in Figure 3.

3. Object Detection in Low-Light Images

As the demand for improved autonomous driving technology continues, the field of object detection has established itself as one of the most actively researched areas in computer vision. Object detection consists of classification and localization, each responsible for classifying objects and identifying the location of the objects. Classification separates the objects from the background in the image and classifies them into appropriate classes, while localization finds the location of the separated objects from the classification using the bounding box and identifies where they are located in the image. The advent of Convolutional Neural Networks (CNNs) has significantly improved detection performance, catapulting CNN-based object detection technology into a crucial method in this field. CNN-based object detection networks can be categorized into two types: 2-stage detection networks and 1-stage detection networks.

The 1-stage detection network ushered in several object detection models, starting with You Only Look Once (YOLO), followed by the Single Shot Multibox Detector and the Retina-Net. The detection process now has one integrated stage in which the classification and localization of the objects are performed simultaneously. Consequently, the 1-stage detection model guarantees a faster detection speed than the 2-stage detection model but has lower accuracy. To improve the performance of 1-stage detection networks, J. Redmon et al. [22] proposed YOLOv2 in which the bounding box inference uses more candidate boxes and a higher-resolution feature map than the existing YOLO. Subsequently, they proposed a method called YOLOv3 for improving detection performance by using a feature map with various sets of sizes to detect objects of varied sizes [23] A. Bochkovskiy et al. [24] proposed YOLOv4 by combining new techniques for the loss function, activation function, and skip connection that are derived from the latest deep learning technology to achieve a 10% higher average precision (AP) than YOLOv3, demonstrating a performance similar to or better than the existing 2-stage detection model.

Active research in developing CNN-based object detection technology has resulted in fast execution speed and stable detection performance, and the technology is being used in various fields such as intelligent transport systems and security/surveillance systems. However, most detection models are designed for well-lit daytime environments, making it difficult to expect stable performance in low-light environments such as nighttime. Although many researchers have focused on image enhancement techniques for enhancing low-light images, most studies aimed to increase the quality of images only, with no direct link to object detection. Furthermore, the datasets used in the test evaluation have images that are artificially darkened to evaluate the performance of image enhancement, which tend to differ from images taken in actual low-light environments. Images in an actual low-light environment are subject to various factors such as uneven lighting and vision camera exposed to noise. As a result, it is less likely to get stable performance of object detection using conventional image enhancement techniques. Therefore, in this paper, we propose an image enhancement technique that offers a more stable performance of vehicle detection in actual low-light driving environments. The overall flow is shown in Figure 4.

To make the low-light image I look similar to a hazy image, the proposed image enhancement technique first converts it to an inverted low-light image ${\mathrm{I}}_{\mathrm{i}\mathrm{n}\mathrm{v}}$, using Equation (11). Here, ${\mathrm{I}}_{\mathrm{i}\mathrm{n}\mathrm{v}}$ is structurally similar to $\mathrm{L}$ in Equation (2), so we apply ${\mathrm{I}}_{\mathrm{i}\mathrm{n}\mathrm{v}}$ instead of $\mathrm{L}$, and estimate $\overline{\mathrm{A}}$ and $\overline{\mathrm{T}}$ using Equations (5) and (10). For the stable estimation of $\overline{\mathrm{A}}$ and $\overline{\mathrm{T}}$, we divided the image into 30 × 30 sub-blocks. To prevent the division of the first term by zero (0) in Equation (11), we set the minimum value of $\mathsf{\gamma }$ to 0.01 for estimating ${\mathrm{T}}_{\mathrm{n}}^{\mathrm{b}\mathrm{l}\mathrm{k}}$. Finally, the enhanced image ${\mathrm{I}}_{\mathrm{E}}$ is obtained through the image inversion process in Equations (11) and (12). In actual environments, however, the image enhancement process can also enhance various noises, which can degrade detection performance if it is linked directly to an object detector. For this reason, we use the A2WF in Equation (13) to minimize the effects of noise to get a denoised image ${\mathrm{I}}_{\mathrm{D}}$, which can be connected, as the output of the preprocessor, to a CNN-based object detection model for detecting objects.

The proposed model uses an image entropy-based dehazing algorithm to increase image contrast and make the contrast between the object and the background clearer while significantly reducing the noise contained in the low-light image using the A2WF, providing better detection performance than detection models with existing image enhancement techniques.

4. Experimental Results

We used Nvidia RTX 3060 and Intel Core i7-12700F CPU to implement the proposed model for vehicle detection in low-light environments and assumed that the model would be applied to an intelligent transport system, and trained the model to detect vehicles using the FLIR RGB/Thermal dataset [25]. To improve the reliability of the test evaluation, we compared its performance against the detection models with existing image enhancement techniques, using the BDD100k [26] and Zju [27] datasets. We then compared detection performance again, using the custom dataset. All datasets used for testing consist of images taken in real low-light environments, rather than artificially created low-light images, which contain various noises that can occur in real low-light environments. The BDD100k and Zju datasets are day and night road-driving footage in various conditions, and 715 and 202 nighttime images, respectively, were extracted and used for testing. Finally, the custom dataset consists of driving footage taken directly at night and consists of a total of 1752 images.

Evaluating the performance of image enhancement techniques can be performed in two ways: full reference and no reference. Most image enhancement studies evaluated the model performance using the Peak-Signal-to-Noise-Ratio and Structural-Similarity-Index-Map, which are full reference-based quality tests. However, this requires a clean image as a reference, and it is not easy to obtain such data in a low-light environment, which is a prerequisite for this study. Therefore, we used the perception-based image quality evaluator (PIQE) [28], which, as a no-reference-based image performance test, does not require a reference image. To evaluate the perception quality, the PIQE estimates noise in key areas of the image, calculates the noise level at a local block level by dividing the image into 16 × 16 sizes, and finally gives a low score for better perception quality and a high score for worse perception quality.

When the proposed image enhancement technique was linked to an object detection model, the evaluation of detection performance was performed with the AP as the yardstick and verified by the YOLO and the Faster R-CNN, which are the two major object detection models. The AP is a performance indicator often used to evaluate detection performance and refers to the area under the precision-recall curve, where the larger the area, the higher the reliability of the object detection model. Precision refers to the ratio of the correct detection to the detection results, and the recall refers to the ratio of correctly detected objects to all the detectible objects, as written below:

$$\mathrm{P}\mathrm{r}\mathrm{e}\mathrm{c}\mathrm{i}\mathrm{s}\mathrm{i}\mathrm{o}\mathrm{n}=\mathrm{T}\mathrm{P}/(\mathrm{T}\mathrm{P}+\mathrm{F}\mathrm{P})$$

(16)

$$\mathrm{R}\mathrm{e}\mathrm{c}\mathrm{a}\mathrm{l}\mathrm{l}=\mathrm{T}\mathrm{P}/(\mathrm{T}\mathrm{P}+\mathrm{F}\mathrm{N})$$

(17)

Here, the TP is a true positive, the FP a false positive, and the FN a false negative.

4.1. Comparison of Image Quality by Image Enhancement Technique

To enhance low-light images, we proposed a method in which an entropy-based haze removal algorithm was used to increase the image contrast, followed by the A2WF to weaken the noise unintentionally enhanced during the image enhancement process, and to strengthen the contrast between the object and the background. Figure 5a is a low-light image source, and Figure 5b–d are results from the existing image enhancement techniques, all of which show higher brightness and contrast than the source image. However, they show a bit of haze as well as light bleeding around light sources such as streetlamps. In the second and fourth columns, the objects detected from the first and third columns are displayed as enlarged. To examine the degree of image enhancement, we passed the images through the Roberts filter to detect the edges and strengthened the boundaries using dilation. With the low-light source image, it is almost impossible to identify the vehicle shape, which is restored to a considerable degree by the existing image enhancement techniques. However, as shown below, the image enhancement process generated a lot of unwanted noise, making it difficult to identify the objects.

For this reason, we applied a noise-reducing bilateral filter and the A2WF to obtain Figure 5e,f, which shows that the colors are restored more naturally, and the noise is considerably reduced, making the objects look clearer. Thus, an additional noise-reducing filter can be used for post-processing after image enhancement to suppress the noise and strengthen the edges, effectively separating the object from the boundary.

We used the PIQE for an additional comparison of the image quality and summarized the results in

Table 1. Comparison of image quality by image enhancement technique and filter.

Image Enhancement	Type of Filter	PIQE
		BDD100K	Zju	Custom
N/A	N/A	69.77	47.49	57.06
KinD	N/A	73.73	52.68	65.27
DCP	N/A	76.73	54.39	68.07
Entropy	N/A	73.38	50.98	62.27
Entropy	Bilateral	72.77	52.15	62.78
Entropy	A2WF	63.99	49.12	56.90

. When the existing image enhancement techniques KinD [29], DCP [17], and entropy [20] were applied, noise also increased, resulting in a higher PIQE than the original image. We added the bilateral filter to the existing entropy-based image enhancement technique for noise suppression, but the effect was minimal, still showing a higher PIQE than the original image. However, by combining the entropy-based image enhancement technique with the A2WF, we could reduce the noise enhanced during the image enhancement process, achieving a lower PIQE of up to 12.73 against the existing enhancement technique DCP, and a similar or lower PIQE of up to 5.77 against the original low-light image. The Zju dataset tends to be brighter than the other datasets because of light-emitting vehicles in the dark as well as several streetlamps, showing a barely improved score for the image quality, while the typically dark BDD100k and the severely dark custom dataset show the improved image quality, with a quality score that is similar to the original image. Thus, the proposed technique shows a greater effect of image enhancement in a dark environment with fewer light sources where only streetlamps are present.

4.2. Comparison of Object Detection Performance by Image Enhancement Technique

We conducted a comparative evaluation to verify how useful the proposed method is in actual object detection; that is, the utility of an image enhancement technique that increases the contrast of a low-light image and strengthens the boundary between the objects and the background by reducing the noise generated during the enhancement process through the A2WF. To confirm the general-purpose utility of the proposed algorithm, we used it as a preprocessor for YOLOv4, a major 1-stage detection network model, and Faster R-CNN, a major 2-stage detection network model. We summarized the test results of the usage or non-usage of various image enhancement techniques and filters based on the AP when the Intersection of Union (IoU) is 0.5, as shown in

Table 2. Comparison of detection performance by image enhancement technique.

Image Enhancement	Type of Filter	Detection Model	AP(%)
			BDD100K	Zju	Custom
N/A	N/A	YOLO	62.32	58.45	28.32
		Faster R-CNN	28.01	16.94	7.72
KinD	N/A	YOLO	56.80	51.64	27.93
		Faster R-CNN	38.46	38.88	19.26
DCP	N/A	YOLO	52.20	46.33	28.57
		Faster R-CNN	37.43	34.74	23.50
Entropy	N/A	YOLO	51.42	40.19	26.85
		Faster R-CNN	35.02	34.46	20.49
Entropy	Bilateral	YOLO	62.89	51.96	30.80
		Faster R-CNN	38.53	38.03	20.33
Entropy	A2WF	YOLO	70.05	60.83	33.99
		Faster R-CNN	41.30	41.50	19.15

.

With the object detection results from YOLOv4 and Faster R-CNN being used alone as the baseline, we compared object detection performance using the images enhanced by the existing KinD, DCP, and entropy-based techniques. The performance of YOLOv4 was lower than that without preprocessing by image enhancement, while the performance of Faster R-CNN improved somewhat, but the baseline detection performance of Faster R-CNN is low, which cannot have too significant a meaning. Next, we applied the noise-reducing bilateral filter and the A2WF, achieving improved detection performance. In particular, YOLOv4, along with the preprocessing through entropy + A2WF, improved detection performance by at least 2.38% up to 7.73% from before the preprocessing. Although the detection performance score from the Faster R-CNN is low, preprocessing significantly improved detection performance, proving its general-purpose utility in various object detection models.

Figure 6 illustrates a comparison of YOLOv4-based vehicle detection in low-light images from three (3) datasets. We can see that the image enhancement technique increased the contrast of low-light images, but the enhanced contrast did not lead to an improvement in detection performance; the detected bounding box (green color) and the ground truth (white color). Subsequently, we confirmed that applying noise-reducing filters, the A2WF in particular, resulted in weakening of the enhanced noise, thereby strengthening the boundary between the object and the background, and in turn, significantly improving detection performance. Additional detection results and effects of image contrast enhancement can be further confirmed in Figure 7.

5. Conclusions

CNN-based object detection technology ensures excellent performance in typical daytime environments that have no light constraints, thus being used in various systems such as intelligent transport systems, autonomous driving, and road monitoring. These systems require the stable performance of a detection model even in low-light environments, such as nighttime, but detection performance deteriorates as the boundary between objects and a background becomes blurry in low-light environments. To solve this problem, many studies have been conducted on image enhancement techniques, but images taken in low-light environments may generate a lot of unexpected noise during the image enhancement process due to the reflection of light by numerous factors. Such noise can be directly linked to the object detection process and potentially have a negative impact. Therefore, we have proposed a technique for improving object detection performance in low-light environments, and image contrast was enhanced by applying a dehazing algorithm to an inverted low-light image. Additionally, we applied the A2WF to reduce the unwanted noise enhanced along with the contrast during the image enhancement process and strengthen the boundary between the object and the background. To evaluate detection performance, we connected the proposed image enhancement and noise reduction technique as preprocessing for the object detection models from YOLOv4 and Faster R-CNN. We showed that preprocessing significantly improved detection performance in terms of AP by up to 18.63%, proving the proposed technique’s utility.

Since the proposed scheme may require more processing time than expected for image enhancement, it tends to be somewhat unsuitable for real-time use in high-speed monitoring systems where the situation changes rapidly, such as traffic accident risk detection, jaywalking pedestrian detection, and lane violation detection. Therefore, in future research, we plan to optimize the image enhancement process to reduce its computational complexity and shorten processing time so that it can be applied to high-speed monitoring systems.