Image Quality Enhancement with Applications to Unmanned Aerial Vehicle Obstacle Detection

Abstract

Aiming at the problem that obstacle avoidance of unmanned aerial vehicles (UAVs) cannot effectively detect obstacles under low illumination, this research proposes an enhancement algorithm for low-light airborne images, which is based on the camera response model and Retinex theory. Firstly, the mathematical model of low-illumination image enhancement is established, and the relationship between the camera response function (CRF) and brightness transfer function (BTF) is constructed by a common parameter equation. Secondly, to solve the problem that the enhancement algorithm using the camera response model will lead to blurred image details, Retinex theory is introduced into the camera response model to design an enhancement algorithm framework suitable for UAV obstacle avoidance. Thirdly, to shorten the time consumption of the algorithm, an acceleration solver is adopted to calculate the illumination map, and the exposure matrix is further calculated via the illumination map. Additionally, the maximum exposure value is set for low signal-to-noise ratio (SNR) pixels to suppress noise. Finally, a camera response model and exposure matrix are used to adjust the low-light image to obtain an enhanced image. The enhancement experiment for the constructed dataset shows that the proposed algorithm can significantly enhance the brightness of low-illumination images, and is superior to other similar available algorithms in quantitative evaluation metrics. Compared with the illumination enhancement algorithm based on infrared and visible image fusion, the proposed algorithm can achieve illumination enhancement without introducing additional airborne sensors. The obstacle object detection experiment shows that the proposed algorithm can increase the AP (average precision) value by 0.556.

1. Introduction

With the development of unmanned aerial vehicle (UAV) technology, UAVs are widely used in aerial photography, smart agriculture, package delivery, remote sensing, and power line inspection. These widespread applications have led to an increasing number of UAVs in low-altitude airspace, increasing the risk of collisions between UAVs. Therefore, it is vital to ensure the safety of UAVs during their missions, i.e., avoid collisions between different UAVs.

Machine vision, as an intelligent technology, has been widely researched and succeeded in the fields of space robots [1], unmanned vehicles [2], micro air vehicles [3], and fixed-wing aircraft automatic landing [4].

Research on machine vision in unmanned aerial vehicles (UAVs) focuses on the following aspects

(1)

Vision-based positioning method [5];

(2)

Vision-based autonomous UAV landing [6];

(3)

Vision-based obstacle avoidance [7].

Vision-based obstacle avoidance is considered a viable technique for avoiding low-altitude collisions between UAVs. Generally, vision-based obstacle avoidance contains two parts: (1) obstacle perception, which aims to detect obstacles (other UAV) that may threaten the safety of UAV flight; (2) avoidance trajectory planning, which aims to replan a safety trajectory that can avoid obstacles.

Figure 1 presents a typical configuration of a common vision-based obstacle perception and avoidance system.

As shown in Figure 1, high-quality images are the prerequisite to ensure the effectiveness of obstacle avoidance. However, many missions require UAVs to perform at night, and poor lighting in night-time environments can severely affect their ability to sense obstacles and increase the risk of flight. Therefore, it is usually necessary to preprocess the image to enhance image brightness. In present days, image defogging, image dust removal, and brightness enhancement are widely studied algorithms for image preprocessing. Li et al. [8] proposed a method to eliminate the effect of dust on visual navigation during a Mars landing. González et al. [9] processed the low-resolution image acquired by a UAV based on the super-resolution technology of convolutional neural network (CNN) to obtain the high-resolution image. Zhao et al. [10] investigated the application of image-defogging technology in UAV visual navigation. Zhang et al. [11] studied a defogging method based on significance-guided dual-scale transmission correction. Wang et al. [12] utilized the double tangent curve to enhance the brightness of pedestrian photos acquired by UAV in night-time environments. Gao et al. [13] presented a new atmospheric scattering model to defog UAV remote sensing images obtained in hazy weather.

In general, image defogging and image dedusting algorithms, despite some differences in implementation details, can both be grouped into the same class of problems, i.e., eliminating the impact of suspended atmospheric particles (e.g., water droplets, sand, and dust) on image quality. In contrast, the image brightness enhancement algorithm has its own particular goals. Its core purpose is to adjust the exposure of the low-brightness pixel position to achieve image brightness enhancement. Existing low-brightness image enhancement algorithms can be grouped into two categories: brightness enhancement using a single visible image, and brightness enhancement by fusing infrared and visible images.

Brightness enhancement algorithms based on a single visible image contain the following aspects

(1)

Histogram equalization-based algorithms, its core idea is to extend the image dynamic range by adjusting the histogram, so that the darker areas of the image are visible. Its main advantages are simple and high in efficiency, whereas the disadvantage is that this kind of algorithm is not flexible enough for adjusting the local area of the image, which can easily lead to under-exposure/over-exposure and noise amplification in the local area of the image. Representative algorithms include contrast-accumulated histogram equalization (CAHE) [14], brightness-preserving dynamic histogram equalization (BPDHE) [15], etc.

(2)

Defogging-based algorithms, these algorithms first invert the image, then apply the defogging algorithm to the inverted image, and finally invert the defogged image to obtain an enhanced image. The basic model used by this kind of algorithm lacks reasonable physical explanation, and its use of denoising techniques as post-processing will result in the blurring of image details. Representative algorithms include adaptive multiscale retinex (AMSR) [16], ENR [17], etc.

(3)

Statistical model-based algorithms, this kind of algorithm utilizes a statistical model to characterize the ideal attributes of an image. The effectiveness of such algorithms relies on prior knowledge of the statistical model. When the assumptions are exceeded, such as strong noise in the input image, the adaptability of such algorithms is insufficient. Representative algorithms include bio-inspired multi-exposure fusion (BIMEF) [18].

(4)

Retinex-based algorithms, this kind of algorithm breaks down the image into two components, a reflectance map and an illumination map, and further processes these two components to obtain the enhancement image. The main advantage of such algorithms is that they can dynamically process images and achieve adaptive enhancement for various images. However, their disadvantage is that such algorithms remove illumination by default and do not limit the range of reflectance, so they cannot effectively maintain the naturalness of the image. Representative algorithms include joint enhancement and denoising (JED) [19], low-light image enhancement (LIME) [20], multiple image fusion (MF) [21], Robust [22], etc.

(5)

Deep learning-based algorithms, this kind of algorithm enhances an image by the relationship between the poor brightness image and the well-exposed image which is obtained by training a deep neural network. The extraction of powerful prior information from large-scale data gives this algorithm a general performance advantage. However, those algorithms have high computational complexity, are time-consuming, and require large datasets. Representative algorithms include enhancement via edge-enhanced multi-exposure fusion network (EEMEFN) [23], Zero-reference deep curve estimation (Zero DCE) [24], etc.

Although the aforementioned algorithms have achieved varying degrees of success on open datasets such as LIME [20] and ExDark [25], their limitations are as follows. (1) The above algorithms rely on iterative optimization in calculating the illumination map, which is time-consuming and therefore cannot satisfy the requirements of real-time processing for UAV obstacle avoidance. (2) Deep learning-based algorithms need to rely on a high volume of paired low/normal brightness image datasets for network training. For the application scenarios of UAV obstacle avoidance, there is currently a lack of large pairs of low/normal brightness datasets containing UAV obstacle objects, so this kind of algorithm cannot be applied to UAV obstacle avoidance scenarios. (3) The camera model lacks a reasonable physical explanation, which can easily result in blurred image details, and some algorithms are not flexible enough to adjust the low signal-to-noise ratio (SNR) area of the image, which can easily lead to under-exposure/over-exposure and noise expansion. Images with blurred details and noise expansion are not conducive to accurate object recognition in the UAV obstacle avoidance process.

Brightness enhancement algorithms based on infrared and visible image fusion include multi-scale transform-based algorithms [26,27], sparse representation-based algorithms [28,29], deep learning-based algorithms [30,31], and so on. Its core idea is to integrate the superiority of infrared and visible images by complementing each other, to achieve brightness enhancement. The brightness enhancement algorithm via fusing the infrared and visible images needs to introduce additional sensors that cannot meet the load requirements of small UAVs.

To eliminate the limitation of a low-illumination environment on the vision-based obstacle avoidance of unmanned aerial vehicles, this paper proposes a low-brightness airborne image enhancement algorithm utilizing the camera response model and the Retinex theory. While comparing with the existing brightness enhancement algorithms, the proposed algorithm has the following advantages.

(1) Solving the problem of blurred image details caused by existing enhancement algorithms, a low-light image enhancement algorithm framework for UAV obstacle avoidance is designed and combined with Retinex theory.

(2) Solving the problem that the pixel position with a low SNR value is easy to generate noise in the enhancement process of the existing algorithm, the maximum exposure threshold is set for the low SNR pixel position.

(3) Solving the problem that the calculation process of the illumination map depends on iterative optimization and is time-consuming, this study utilizes an accelerated solver to compress the solution space to reduce the time consumption.

The remainder of this article is structured below. The mathematical model of the airborne image enhancement algorithm is presented in Section 2. Section 3 presents the improved image enhancement algorithm designed for UAV obstacle avoidance. In Section 4, simulation results and analysis are presented. Section 5 summarizes the work of this paper.

2. Mathematical Model of Scene Brightness Described by Image

The core work of this paper is to improve the brightness of low-light images. Therefore, it is necessary to first establish a mathematical model for describing scene brightness, as shown in Equation (1). For a single image, the relationship between the image irradiance I and the pixel value V of each pixel can be expressed by a nonlinear function f, which is the camera response function (CRF).

$$V=f\left(I\right)$$

(1)

Generally, the irradiance I reaching the camera sensor changes linearly with the change of the camera exposure parameters. Because the camera performs nonlinear processing on the response of irradiance, the image pixel values V do not change linearly with the irradiance I, so it is needed to use a nonlinear function to characterize the mapping relationship between images with different irradiance. We use the brightness transfer function (BTF), which is represented by g, to define the mapping relationship. As shown in Equation (2), g defines the nonlinear relationship between the image $V_0$ and image $V_1$ under different irradiance scenes $I_0$ and $I_1$.

$$V_1=g(V_0,e)$$

(2)

where e is the exposure ratio.

As shown in Figure 2, in the actual imaging process, Equations (1) and (2) together determine the correspondence between image pixel values V and irradiance I for the same imaging scene. Substituting Equation (2) into Equation (1) gives the comparametric equation [32] shown in Equation (3), which describes the relationship between the functions f and g.

$$g\left(f\right(I),e)=f\left(eI\right)$$

(3)

To facilitate the subsequent solution, the following three assumptions are made about the function.

Assumption 1. 

All points of the camera imaging sensor have the same range of values of f.

Assumption 2. 

$f(.)$ can be normalized to $[0,1]$.

Assumption 3. 

f is a monotonically increasing function. On the basis of the assumptions, define F as the theoretical space of f.

$$\begin{array}{c}F:=\left\{f\right|f\left(0\right)=0,\\ f\left(1\right)=1,x_1>x_2\iff f\left(x_1\right)>f\left(x_2\right)\}\end{array}$$

(4)

From Equation (3), the BTF and CRF have common properties.

3. The Improved Image Enhancement Algorithm

To address the problem that camera response model-based enhancement algorithms have limited adaptability and can lead to blurred image details, Retinex theory [33] is now introduced to obtain a brightness-enhanced image with sharp details.

Retinex theory decomposes the amount of light reaching the observer into the following two components,

$$\mathbf{I}=\mathbf{R}\circ \mathbf{L}$$

(5)

where $\mathbf{R}$ and $\mathbf{L}$ denote the reflectance and light maps of the scene, respectively. The operator ∘ denotes multiplication by an element, and $\mathbf{I}$ is the image irradiance representing the amount of light reaching the camera imaging sensor.

Substituting Equation (5) into Equation (1) yields Equation (6), where $\mathbf{V}$ denotes the low-light image with image irradiance $\mathbf{I}$ mapped to the camera response model.

$$\mathbf{V}=f\left(\mathbf{I}\right)=f(\mathbf{R}\circ \mathbf{L})$$

(6)

The enhanced image ${\mathbf{V}}^{\prime }$ can be expressed as

$${\mathbf{V}}^{{}^{\prime }}=f(\mathbf{R}\circ \mathbf{1})$$

(7)

where $\mathbf{1}$ is an all-1 matrix. Equation (8) is derived from Equations (3) and (5), which gives the relationship between the low-light image $\mathbf{V}$ and the enhanced image ${\mathbf{V}}^{\prime }$.

$$\begin{array}{c}{\mathbf{V}}^{{}^{\prime }}=f\left(\mathbf{R}\right)=f(\mathbf{I}\circ (\mathbf{1}\oslash \mathbf{L}))=\hfill \\ g\left(f\right(\mathbf{I}),\mathbf{1}\oslash \mathbf{L})=g(\mathbf{V},\mathbf{1}\oslash \mathbf{L})\hfill \end{array}$$

(8)

The operator ⊘ indicates element-by-element division. According to Equation (8), by adjusting the exposure of the low-light image $\mathbf{V}$, an enhanced image ${\mathbf{V}}^{\prime }$ can be acquired. Therefore, the relationship between the low-light image and the enhanced image can be expressed via Equation (9)

$${\mathbf{V}}^{{}^{\prime }}=g(\mathbf{V},\mathbf{1}\oslash \mathbf{L})=g(\mathbf{V},\mathbf{e})$$

(9)

As large dark areas exist in UAV airborne images under low-illumination conditions, it is important to tailor the image to the actual brightness distribution and make the most appropriate enhancement for each pixel position to avoid significant noise in low SNR areas. Therefore, instead of using a fixed exposure ratio e to enhance the image, we calculate the exposure ratio matrix $\mathbf{e}$ of the whole image, which stores the appropriate exposure value for every pixel point in the image.

Thus, in contrast to the fixed exposure ratio e in Equation (2), $\mathbf{e}$ in Equation (9) is an exposure value matrix that defines every pixel point’s required exposure value in the image. The exposure ratio matrix $\mathbf{e}$ can be derived from Equation (10).

$$\mathbf{e}=\mathbf{1}\oslash \mathbf{L}$$

(10)

Equation (9) shows that there are two components of the proposed image brightness enhancement algorithm: (1) selecting an appropriate camera response model and setting the reasonable model parameters; (2) calculating the exposure ratio matrix $\mathbf{e}$ to ensure that all pixel positions achieve the required exposure. Equation (10) further shows that the exposure ratio matrix $\mathbf{e}$ can be found by calculating $\mathbf{L}$, i.e., the low brightness areas in the UAV airborne image would be adjusted using a high exposure value and the light areas in the UAV airborne image would be adjusted using a low exposure value.

3.1. Camera Response Model Determination

To improve the performance of the proposed algorithm, it is first necessary to determine the mathematical expression for the camera response model f.

This paper uses the two-parameter camera response model Sigmoid, which is based on the human visual system.

$$f\left(I\right)=(1+\sigma )\frac{I^m}{I^m+\sigma }$$

(11)

where $\sigma$ is the weight factor and m is a fixed value of 0.9.

From Equations (1) and (3), the BTF scattering for a specified exposure ratio e can be acquired by Equation (12).

$$g(V,e)=f\left(e{f}^{-1}\left(V\right)\right)$$

(12)

The BTF of Sigmoid can then be computed from Equations (11) and (12).

$$g(V,e)=\frac{e^{m}V(1+\sigma )}{(e^m-1)V+1+\sigma }$$

(13)

3.2. Exposure Ratio Matrix Calculation

After determining the camera response model, it is further necessary to determine the exposure ratio matrix $\mathbf{e}$. According to Equation (10), $\mathbf{e}$ is inversely proportional to the illumination map $\mathbf{L}$. Therefore, $\mathbf{L}$ is calculated first and then $\mathbf{e}$ is solved by Equation (10). To solve the problem that existing algorithms rely on iterative optimization and calculation time-consuming for calculating illumination map, we adopt the accelerated solver proposed in [20] to solve $\mathbf{L}$. The principle is as follows.

$$\underset{\mathbf{L}}{min}{∥\widehat{\mathbf{L}}-\mathbf{L}∥}_F^2+\alpha {∥\mathbf{W}\circ \nabla \mathbf{L}∥}_1$$

(14)

Equation (14) is the objective function for solving the illumination map, where $\widehat{\mathbf{L}}$ is the initial illumination map, $\alpha$ is the coefficient used to balance the two terms, and ${∥·∥}_F$ and ${∥·∥}_1$ denote the Frobenius norm and ${\ell }_1$ norm, respectively. $\mathbf{W}$ is the weight matrix and $\nabla \mathbf{L}$ is the first order derivative filter. The ${\ell }_1$ norm in Equation (14) and the gradient operation on $\mathbf{L}$ complicate the process of solving Equation (14), i.e., the presence of the sparse weighted gradient term ${∥\mathbf{W}\circ \nabla \mathbf{L}∥}_1$ makes the computational process dependent on iterative optimization.

$$\underset{ϵ\to 0^{+}}{lim}\sum _x\sum _{d\in \left\{h,v\right\}}\frac{{\mathbf{W}}_d\left(x\right){\left({\nabla }_d\mathbf{L}\left(x\right)\right)}^2}{\left|{\nabla }_d\mathbf{L}\left(x\right)\right|+ϵ}={∥\mathbf{W}\circ \nabla \mathbf{L}∥}_1$$

(15)

where h is the horizontal of pixel x, and v is the vertical of pixel x. Based on Equation (15), approximating ${∥\mathbf{W}\circ \nabla \mathbf{L}∥}_1$ by ${\displaystyle \sum _x}{\displaystyle \sum _{d\in \left\{h,v\right\}}}\frac{{\mathbf{W}}_d\left(x\right){\left({\nabla }_d\mathbf{L}\left(x\right)\right)}^2}{\left|{\nabla }_d\widehat{\mathbf{L}}\left(x\right)\right|+ϵ}$. Then Equation (14) can be written in the following form

$$\underset{\mathbf{L}}{min}{∥\widehat{\mathbf{L}}-\mathbf{L}∥}_F^2+\alpha \sum _x\sum _{d\in \left\{h,v\right\}}\frac{{\mathbf{W}}_d\left(x\right){\left({\nabla }_d\mathbf{L}\left(x\right)\right)}^2}{\left|{\nabla }_d\widehat{\mathbf{L}}\left(x\right)\right|+ϵ}$$

(16)

Although Equations (14) and (16) are different, we can still extract the illumination map $\mathbf{L}$ from the initial illumination map $\widehat{\mathbf{L}}$. This is because if $\left|{\nabla }_d\widehat{\mathbf{L}}\left(x\right)\right|$ is small then $\left|{\nabla }_d\mathbf{L}\left(\mathbf{x}\right)\right|$ and $\frac{{\left({\nabla }_d\mathbf{L}\left(\mathbf{x}\right)\right)}^2}{\left|{\nabla }_d\widehat{\mathbf{L}}\left(x\right)\right|+ϵ}$ will be inhibited, i.e., because the illumination map $\mathbf{L}$ is constrained, no gradient is generated when the initial estimated $\widehat{\mathbf{L}}$ exhibits a small gradient. Conversely, when $\left|{\nabla }_d\widehat{\mathbf{L}}\left(x\right)\right|$ is big, the above inhibition is mitigated as the position is more likely to be on a structural boundary rather than on a regular texture. Equation (16) only involves a quadratic term and can be solved by the following formula

$$\left(\mathbf{M}+\sum _{d\in \left\{h,v\right\}}{\mathbf{D}}_d^{T}Diag\left({\tilde{\mathbf{w}}}_d\right){\mathbf{D}}_d\right)\mathbf{l}=\widehat{\mathbf{l}}$$

(17)

where $\mathbf{M}$ is the unit matrix. $\mathbf{D}$ consists of ${\mathbf{D}}_h$ and ${\mathbf{D}}_v$, which are the Toeplitz matrices deduced from the discrete gradient operators with a forward difference. ${\tilde{\mathbf{w}}}_d$ is the vector form of ${\tilde{\mathbf{W}}}_d$, ${\tilde{\mathbf{W}}}_d\left(x\right)\leftarrow \frac{{\mathbf{w}}_d\left(x\right)}{\left|{\nabla }_d\widehat{\mathbf{L}}\left(x\right)\right|+ϵ}$. Equation (17) reduces the computational complexity of Equation (14) from $\mathcal{O}\left(tNlogN\right)$ to $\mathcal{O}\left(N\right)$, where t is the number of iteraions required to converge.

After obtaining the illumination map $\mathbf{L}$, the exposure ratio matrix $\mathbf{e}$ is calculated via Equation (10). It is worth noting that setting a high exposure value for a low SNR position will result in a high noise level in a low-light image. Although we can utilize image denoising techniques for noise reduction, it can decrease the efficiency of the enhancement process. As the brightness of UAV vision images is extremely low under low-illumination conditions, this paper sets a maximum exposure value for the extremely low-brightness pixel position to avoid excessive noise in the enhanced image.

$$\mathbf{e}\left(x\right)=\frac{1}{max(\mathbf{L}\left(x\right),L_{min})}$$

(18)

where $L_{min}$ is a threshold value. In this paper, we suppose that when the illumination value of a pixel is below $L_{min}$, the SNR of that pixel is very low and set a maximum exposure value of $1/L_{min}$ for it.

Figure 3 shows the effect of setting the maximum exposure value on noise suppression; Figure 3a is a low-illumination image; Figure 3b is the exposure ratio matrix estimated using Equation (10); Figure 3c is the exposure ratio matrix estimated using Equation (18) ($1/L_{min}=1/7$); Figure 3d is the enhanced image by the use of Figure 3b; Figure 3e is the enhanced image by the use of Figure 3c. It can be seen that when the maximum exposure ratio value is not set, the enhanced image becomes noticeably noisy, resulting in a deterioration in image quality. The enhanced image has better visual quality due to less image noise when setting the maximum exposure value, despite the reduced visibility in extremely low-illumination areas.

After calculating the exposure ratio matrix, the low-light image is enhanced using Equation (9). So far, the process of the proposed low-light image enhancement algorithm can be summarized as follows. (1) The low-light image $\mathbf{V}$ is inputted, and the camera response model parameter is determined according to the sensor characteristics. (2) To meet the real-time requirements of online processing of airborne images, the accelerated solver in [20] is introduced to obtain the illumination map $\mathbf{L}$ corresponding to the low-light image and the exposure ratio matrix $\mathbf{e}$ is calculated using Equation (18). (3) Equation (9) is used to enhance the low-light image $\mathbf{V}$ to acquire the enhanced image ${\mathbf{V}}^{\prime }$. Figure 4 visually illustrates the processing flow of the proposed algorithm.

4. Experimental Results and Analysis

4.1. Dataset Construction

The three types of UAV obstacle objects used to construct the dataset are shown in Figure 5, namely Align 700L V2, DJI F450, and DJI Mavic 2 Pro. The camera properties of another DJI Mavic2 Pro for collecting images are listed in

Table 1. Camera properties of the other DJI Mavic 2 Pro.

UAV	Image Sensor	Pixels	FOV	Aperture	ISO Range	Shutter Speed	Video Resolution
DJI Mavic 2 Pro	1 inch CMOS	20 MP	77${}^{\circ }$	f/2.8– f/11	100–6400	8–1/8000 s	1920 × 1080

, and the properties of the three UAV objects are listed in

Table 2. Properties of the three UAV obstacle objects.

UAV	Align 700L V2	DJI F450	DJI Mavic 2 Pro
Dimensions	1320 × 220 × 360 mm	450 × 450 × 350 mm	322 × 242 × 84 mm
Weight	5100 g	1357 g	907 g

. For the validation of the proposed algorithm, three video datasets with a resolution of 1920 × 1080 were taken at 6:30 pm (after sunset) on 21 October, in Nanjing University of Aeronautics and Astronautics ($31.{940402}^{\circ }$ N, $118.{788513}^{\circ }$ E), Jiangning District, Nanjing, China using another DJI Mavic2 Pro for three types of UAV objects (acting as obstacles), as shown in Figure 6.

4.2. Enhancement When $\sigma$ Changes

For the proposed algorithm, since the value of the camera model parameters directly determines the effectiveness of the enhancement algorithm, the camera model parameter $\sigma$ is first analyzed. It is important to note that the optimal values of the model parameters vary with the application scenario, but the general trend of the algorithm effectiveness due to parameter changes is consistent. Therefore, the model parameter specified in this research could be considered to serve as reference values when applying the algorithm to other datasets.

When enhancing the low-light image, the parameter $\sigma$ of the camera response model Sigmoid affects the effectiveness of the algorithm. Generally, if the parameter $\sigma$ is too small, the enhancement effect of the low-light image will be insufficient. On the contrary, if $\sigma$ is too large, the enhancement effect is too strong. However, it may lead to poor effects such as color distortion of the image. In this paper, we set the $\sigma$ values to increase from 0.1 to 1 in a step of 0.1 to analyze the effect on the enhancement when $\sigma$ takes different values.

The quantitative evaluation metrics the natural image quality evaluator (NIQE) [34], no-reference free energy based robust metric (NFERM) [35], and information entropy (EN) [36] are used to objectively evaluate the enhancement effect. The smaller the values of the NIQE and NFERM, the better the enhancement. The larger the EN, the more information is retained in the image and the higher quality of the enhanced image.

Figure 7, Figure 8 and Figure 9 show the visualization of the enhancement results of Video 1 containing the Align 700L V2 object, Video 2 containing the DJI F450 object, and Video 3 containing the DJI Mavic 2 Pro object using the proposed algorithm, respectively, when $\sigma$ is taken to different values. It can be seen from Figure 7, Figure 8 and Figure 9 that the image brightness is improving as the $\sigma$ gradually increased. The quantitative evaluation results of the enhanced images of Video 1, Video 2, and Video 3 at different $\sigma$ values using the NIQE, NFERM, and EN are shown in Figure 10, Figure 11 and Figure 12, respectively. From Figure 10, Figure 11 and Figure 12, the NIQE and NFERM decrease as the $\sigma$ increases, while the EN metric increases. Considering these three objective evaluation indicators and the actual enhanced results comprehensively, the $\sigma$ was kept constant at 0.8 in this study to achieve good enhanced effects in most cases.

4.3. Comparison Experiments

4.3.1. Time-Consumption Comparison with Similar Brightness Enhancement Algorithms

To validate the superiority of the proposed algorithm in terms of the enhancement process time consumption, the proposed algorithm was compared with several state-of-the-art algorithms, including BIMEF, BPDHE, ENR, MF, and JED. A total of 500 low-light images (1920 × 1080) were selected from the dataset and enhanced using the above algorithms. For fairness, all the experiments were performed in the MATLAB R2020b installed on a desktop equipped with an Intel i7- 9700KF CPU and 32 GB RAM. We find their average processing time for comparison. The time consumption results are shown in

Table 3. Comparison of time consumption of the six algorithms.

Algorithm	BIMEF	BPDHE	ENR	MF	JED	Ours
Mean time consumption (s)	0.53	1.99	2.96	3.94	4.11	0.88

.

From the time consumption results in Table 3, the proposed algorithm has the shortest time consumption and the fastest enhancement speed compared to other algorithms, except for the BIMEF algorithm. This is due to the introduction of an accelerated solver, which compresses the illumination map solution space and reduces the computation amount, thus shortening the time consumption.

4.3.2. Comparison with Similar Brightness Enhancement Algorithms

To further test the enhancement effect of the proposed algorithm, the proposed algorithm was compared with several state-of-the-art algorithms, including BIMEF, BPDHE, ENR, MF, and JED. Video 1, Video 2, and Video 3 were enhanced by BIMEF, BPDHE, ENR, MF, JED, and the proposed algorithm, respectively. The intuitive enhancement results are shown in Figure 13. Additionally, the quantitative comparison results of the enhanced images using the NIQE, NFERM, and EN metrics are shown in Figure 14.

Figure 13 shows that the brightness enhancement of the BIMEF algorithm is limited. The BPDHE algorithm has a lot of noise in the image. However, the ENR algorithm has a good overall effect, there is some noise in the sky area. In addition, the MF and JED algorithms have too much noise, color distortion, and blurred details in the buildings (shown in the green box). In contrast, the enhanced results of the proposed algorithm have less noise, while retaining more detail and better visual quality of the image. The NIQE and NFERM results of the proposed algorithm in Figure 14a,b are smaller than the compared algorithms, and the proposed algorithm has higher EN results than the other algorithms, as shown in Figure 14c. These quantitative evaluation results show that the proposed algorithm outperforms the compared algorithms in the three metrics of NIQE, NFERM, and EN. Therefore, compared with other algorithms, this algorithm has better enhancement performance.

4.3.3. The Effect of Brightness Enhancement on oBstacle Object Detection

The purpose of image enhancement is to raise the detection accuracy of obstacle objects for vision-based obstacle avoidance in low-illumination conditions. Therefore, it is necessary to conduct experiments on obstacle object detection to verify the effectiveness of the proposed low-light image enhancement algorithm. We use YOLO-V3 as the detection model and build a dataset containing UAV obstacle objects to retrain YOLO-V3. The images in the dataset are derived from video streams captured by a Mavic 2 Pro and UAV images from the Internet. For fairness, the training dataset contains images of diverse types of UAVs at different viewing angles, normal light images, low-light images, clear images, and blurred images. The 500 enhanced images containing UAV obstacle objects were used as a test dataset for obstacle detection.

Figure 15 shows the visualization of the obstacle detection results. The number annotated at the top of the yellow frame is the confidence coefficient (Cc) of the obstacle object detection. As can be seen from the first row of Figure 15, the obstacle detection Cc value is low on the original low-light images, and there are cases where the obstacle object is not detected. After being enhanced by BIMEF, BPDHE, ENR, MF, JED, and the proposed algorithm, the detection accuracy is improved and the Cc for obstacle object detection of the enhanced images is all higher than those of the original low-light images. This is shown in rows 2–7 of Figure 15.

For further analysis, the detection average precision (AP) values of the images enhanced by the above enhancement algorithms were calculated, respectively, and those results are summarized in

Table 4. The AP results of the images enhanced by different algorithms.

Image	Low-Light Image	BIMEF	BPDHE	ENR	MF	JED	Our
AP	0.301	0.572	0.507	0.740	0.786	0.749	0.857

. The results show that the images enhanced by the proposed algorithm have the highest AP value for obstacle object detection. As can be seen from Figure 15 and Table 4, the proposed enhancement algorithm can improve the accuracy of obstacle detection.

4.3.4. Comparison with the Infrared and Visible Images Fusion Algorithm

To further validate the superiority of the brightness enhancement algorithm based on a single visible image proposed in this paper, it was compared with the infrared and visible images fusion algorithm CSR [37].

To compare the enhancement effect with CSR, three low-light scenarios were created in the FlightGear [38] simulation software, as shown in Figure 16a–c, respectively. Figure 16a is a Boeing 737-300 taken at a distance of 2000 m, Figure 16b is a Boeing 737-300 taken at a distance of 1000 m, and Figure 16c is an MQ-9 taken at a distance of 750 m. The infrared images of the scenes corresponding to Figure 16a–c are Figure 16d–f, respectively, the enhancement results of Figure 16a–c using the proposed algorithm are Figure 16g–i, respectively, and the fusion results of the CSR are shown in Figure 16j–l. From Figure 16g–i, the proposed algorithm can not only effectively improve the image brightness to make the object clearly visible, but also retain the color information to the maximum extent. As shown in Figure 16j–l, although the CSR algorithm can enhance image brightness, the method cannot retain more color information and detail information. The lack of such information will seriously affect the object detection algorithm’s ability to detect and identify the object.

Quantitative evaluation metrics such as NIQE, NFERM, and EN were used to compare and analyze the enhancement results of the proposed algorithm and the CSR algorithm. The results are shown in Figure 17, with horizontal coordinates 1, 2, and 3 indicating the three scenes (a), (b), and (c) in Figure 16, respectively. From Figure 17, the proposed algorithm has better results in three metrics, so the proposed algorithm has better performance than CSR.

5. Conclusions

This paper presents a low-light airborne image enhancement algorithm based on the camera response model and Retinex theory, aiming at the need to overcome the limitations of low-light environments on UAV obstacle avoidance. The contributions of this paper can be summarized as follows.

(1) To address the problem that the existing algorithms can lead to blurred details of the enhanced image, Retinex theory is introduced and combined with the camera response model.

(2) To address the problem that the existing algorithms generate significant noise at low SNR pixel positions during enhancement, the exposure matrix of low-illumination images is obtained by calculating the illumination map, and the maximum exposure ratio value is set for the extremely low SNR pixel position.

(3) The proposed algorithm can increase the AP value of obstacle detection by 0.556 in low-illumination environments, which increases the safety of UAVs flying at night.

However, the proposed algorithm cannot run in real time, GPU parallel acceleration will be considered later as a solution to improve the real-time performance.