3.1. Regional Brightness Perception
The shortcomings of commonly used low-illumination image enhancement methods are still observed, such as the inability of traditional algorithms to effectively solve the problem of uneven brightness. Additionally, the contextual information of the image cannot be fully considered by deep learning-based algorithms [
36,
37,
38], leading to the inability to establish pixel dependencies. This defect makes it difficult to resolve the problem of uneven illumination in each local region. To address these problems, our method of regional luminance perception is proposed, aiming to solve the problem of uneven luminance generated by local regions, accurately divide regional luminance, and eliminate the influence of the division of luminance range on the suppression effect of the algorithm.
While the image segmentation algorithm [
3] is capable of dividing image’s brightness, it fails to fully consider the contextual information of the image. Additionally, a separate algorithm is needed to smooth the region boundaries. This becomes particularly challenging in real-life situations with complex backgrounds, where multiple light sources are present and need to be accurately taken into account for proper brightness region division. Although the JinY algorithm [
18] demonstrates good performance in dealing with halo light, it does not explicitly differentiate between a single light source and multiple light sources, as illustrated in
Figure 1 below. Obviously, it is evident that the JinY algorithm’s performance is not as effective in suppressing only multiple light sources, as our approach exceled in this particular task. In view of a multi-light source and a single light source under a complex background, we proposed to divide the brightness region into the dark light region, halo region, glare region, and other regions. The range of pixel brightness of the halo, glare, and dark light regions of an image is shown in
Figure 2 below, where
Figure 2a,e are the original frame of the image,
Figure 2b,f are the glare region of the image,
Figure 2c,g are the glare region plus the halo region of the image, and
Figure 2d,h are the visual representation of the division of the halo, glare, and dark light regions of the image.
In the case of a single light source in a relatively dark background, the object’s brightness and contrast are more prominent, making details such as its shape and texture easier to observe and recognize. However, with multiple light sources, complex reflection effects may occur, making object recognition more challenging. To address this issue, we leveraged the powerful adaptability of region luminance perception to maintain object perception in different luminance environments, enabling recognition in a variety of lighting conditions. The convenience of region brightness perception in dividing image brightness was fully presented by exclusively applying the suppression algorithm to suppress the original frames of both single and multiple light sources in the image, as depicted in
Figure 3—the complex light source suppression effect—where
Figure 3a,c represent the original frame of the image and
Figure 3b,d represent the resultant image suppression frame.
Before the luminance processing of low-illumination images, the limitations of image segmentation algorithms in dividing the luminance range were addressed. This was inspired by the luminance perception function of the human eye [
39], which was utilized to assist in selecting the luminance thresholds. This was done by leveraging the fact that the luminance perception function of the human eye covers a wider range of luminance sensations, thus aiding in the selection of appropriate thresholds for brightness division. Due to the existence of two regions with opposite brightness in dark light region brightness and glare region brightness, to select the current most suitable utilization algorithm for the respective region, we made use of the human eye luminance perception function
R(
L) [
39] for the luminance adjustment of each region, which was defined as follows:
where
R(
L) is the retinal responsiveness of the input grayscale image,
L is the original pixel intensity of the grayscale map,
is the pixel intensity normalized to [0, 1], and
k is a sensitivity parameter that determines how fast the retinal responsiveness grows with increasing luminance intensity. Since human retinal responsiveness is adaptively determined by background luminance, the sensitivity k can be expressed in the following functional form depending on luminance perception:
where
CDFG(
L) is the cumulative distribution function of the input histogram and
p(
x) denotes the normalized gray-level histogram.
In Equation (4), the human eye luminance perception function covers the full dynamic range of the low-illumination image [
R(
Lmin) = 0,
R(
Lmax) = 255], and the value range of
L is [0, 255];
is the pixel intensity normalized to [0, 1], specifically:
where
Lmax is the maximum pixel brightness of the grayscale image.
To obtain a threshold parameter
that is suitable for the brightness of the current region, a one-dimensional search algorithm was utilized in the optimization method, specifically, the golden section search algorithm [
40,
41]. The reason was that the golden section search algorithm is not only fast but also has a wide range of adaptability. Based on the golden section method, the shortening rate of each retention interval was typically considered as 0.618. By calculations and comparisons of the function values of a finite number of points, the original search interval was systematically diminished to meet the allowed inter-cell error. Consequently, the objective was to progressively decrease the interval of the optimal solution until the length of the interval was reduced to 0. In this paper, the iterative process of the algorithm used the initial space [0, 50], [50, 150], [150, 200], and [200, 255] in four iterations. The process continued until it reached the maximum number of iterations, which stopped at 100. The thresholds in the [0, 50], [50, 150], [150, 200], and [200, 255] luminance intervals were then found in turn using just the function values
,
,
, and
.
was derived in the same way as the other three luminance thresholds, and the derivation of
was taken as an example. Its luminance threshold acquisition method is shown in Algorithm 1.
Algorithm 1 Brightness threshold acquisition: |
- 1:
Set a = 0 as the initial left interval and b = 50 as the initial right interval. - 2:
Calculation . - 3:
Calculation . - 4:
If , it means that the minima are at . - 5:
If , update , and output . - 6:
Otherwise , and . - 7:
If , update , and output . - 8:
Otherwise , and . - 9:
Repeat steps 1–8 above to output .
|
3.2. GS-AGC
In this paper, the proposed algorithm is based on the regional brightness perception function and utilizes the exploits the nonlinear characteristics of the Gamma correction algorithm. Instead of employing region segmentation, the algorithm suggests controlling the change in image brightness data by identifying different brightness regions based on the perceived brightness results of various types of regions. This is achieved through adaptive adjustment using the brightness threshold parameter . By doing so, the algorithm enables the adaptive adjustment of image brightness without relying on traditional region segmentation methods.
The detail information of the light and dark light regions was enhanced in this paper by extracting the brightness information of the image through the corresponding grayscale map I
gray of the original image. Additionally, the algorithm expressed the brightness of the glare region that needed to be suppressed through the attention map Reg(I
gray), where the white portion represents the regions that need to be suppressed, while the blue portion represents the regions that need to be enhanced. Afterwards, the luminance information presented in the attention map Reg(I
gray) was integrated into the layer decomposition result of the algorithm by utilizing an iterative method to select the multi-level transition luminance threshold parameter. Additionally, a better visual processing effect for low-illumination images was achieved by processing the pixel luminance of different regions. It is important to note that the processing flow of this paper remained the same for both single and multiple light sources. In order to illustrate the detailed processing of GS-AGC,
Figure 4 below showcases the layer decomposition results of the algorithm in the scenarios of both single and multiple light sources.
In order to address the issue of uneven illumination caused by multiple light sources, in this paper, the brightness region of the image was divided into four parts based on the perceived brightness. The specific process involved traversing the overall grayscale image, starting from the grayscale value that corresponded to the brightness of the first pixel in each row, and ending at the grayscale value that corresponded to the brightness of the last pixel in each column. The GS-AGC algorithm was then applied to perform the necessary processing in different brightness regions. As a result, the algorithm enhanced the brightness in dark light regions and suppressed the brightness in glare regions, thereby improving the overall illumination uniformity of the image.
The core algorithm of this paper primarily comprised four parts:
In the formula,
R1,
R2, and
R3 are pre-defined parameters that are constants. The values for these parameters are set as follows:
R1 = 2,
R2 = 3, and
R3 = 3. These parameters represent the power to which the corresponding expressions are raised. Specifically, the
Gdark expression is raised to the second power, the
Gglare expression is raised to the third power, and the
Ghalo expression is raised to the third power. The
Gother expression has a function value of 1, indicating that it does not directly modify image brightness, and therefore, no parameter value is assigned to it. It was revealed from the experiments that the parameter mentioned above can drastically enhance the image processing result. Specifically, the image processing result remained unchanged once the value of parameter
R exceeded 5. The luminance region division was based on I
gray to carry out, on the overall gray range [0, 255] of the image, the gray value 0, 255, and the four luminance thresholds,
,
,
,
, will directly divide the image grayscale into four regions, which are the luminance region of the image I. They included the dark light region (T
0,
), the halo region(
,
), and the glare region (
, T
255), respectively, and the other regions excluding the dark light, halo, and glare regions(
,
), (
,
). To enhance the visual effect of low-illumination images with a narrow grayscale range, a nonlinear grayscale transformation was performed. This was necessary due to the limited visual distinction and high spatial correlation among adjacent pixels in such images. By adjusting the grayscale values, the image’s brightness range was expanded within a range more suitable for human eye observation. This process involved the following pixel-level operations:
where
R1 = 2,
is a dark light region threshold parameter,
is a medium-bright scene threshold parameter,
is a bright scene threshold parameter, T
0 is a gray-level value of 0, and its dark light enhancement function
Gdark indicates that the gray scale image pixels are stretched from the dark light region
by stretching a narrow gray range to the medium-bright scene
, and that all the corresponding Gamma values between the
Gdark regions (T
0,
) are less than 1.
In the image visual region in addition to focusing on the dark light region part, the high gray value of the region also needs to inhibit the halo, so that the brightness of the image is maintained in the same range, retaining the image with more information on the halo region for the processing:
wherein
R3 = 3,
is a highlight scene threshold parameter whose halo suppression
Ghalo function indicates that the gray scale image pixels are lowered from the highlight scene
to the mid-brightness scene
, and the high gray scale values are corrected by the
Ghalo function to make the corresponding Gamma value of the region between the highlight scene threshold parameter and the mid-brightness scene threshold parameter (
,
) both greater than 1.
Uneven distribution of brightness in images is primarily caused by glare, a phenomenon that occurs in various forms and can be complex in real-life scenarios. Glare imposes a tremendous level of visual burden due to the significant increase in brightness, making glare inhibition a core issue for us to solve. Therefore, it is particularly important to consider the size and number of gray values that span the pixel values of the glare region. We treat the glare region as follows:
where
R = 3,
T255 is the gray level value 255, the glare suppression function
Gglare makes the grayscale image pixels close to the maximum gray value of the bright scene
close to the mid-bright scene
to reduce the visual difference, and the
Gglare function makes the region between the threshold parameter of the bright scene and the threshold parameter of the mid-bright scene (
, T
255) and its corresponding Gamma value both greater than 1.
There is another function to control the removal of dark light, halo, and glare of the other brightness region, (
,
), (
,
). The main role of the other regions of brightness is to remain unchanged:
The flowchart of the GS-AGC algorithm proposed by us is displayed in
Figure 5, while the steps of the algorithm are exhibited in Algorithm 2. It is worth mentioning that the suppression algorithm proposed by us can be adjusted in a specific luminance region by regional luminance pixels, resulting in the enhancement of the overall low-light image based on local glare suppression and the maintenance of a balanced state of image luminance.
Algorithm 2 GS-AGC: |
- 1:
Get the grayscale map Igray of the input image I. - 2:
Obtain the brightness threshold of the image, , according to Equation (4). - 3:
Calculate the pixels in the dark light region of the processed image according to Equation (8), , where are derived from the luminance threshold of Equation (4). - 4:
Adjustment of the image halation region pixels according to Equation (9), , where the luminance threshold parameters are also derived from Equation (4). - 5:
The image glare region pixels are processed according to Equation (10), , the rest of the luminance is adjusted by Equation (11), and the luminance threshold parameters are also derived from Equation (4). - 6:
Output the image after all processing according to Equation (7).
|
3.3. FPGA Implementation of the Algorithm
3.3.1. Image Processing Framework on FPGA Platforms
FPGA, a programmable logic device, was utilized in this study to accelerate the algorithms on the hardware platform. It offers programmability, high performance, low power consumption, and robust security. Leveraging FPGA’s hardware architecture characteristics, the entire system’s input and output were facilitated via a USB interface. The development methodology employed comprised High-Level Synthesis (HLS) description, along with co-programming using Python, C, and Verilog hardware description language. These approaches harnessed FPGA’s high data throughput, computational prowess, and parallelization capabilities to achieve accelerated execution of the algorithms discussed in this paper.
Figure 6 illustrates the image processing framework employed in the glare suppression system. The framework utilizes the AXI4 Stream protocol, which enables stream-oriented transmission without the need for memory addresses and without restricting burst length. This protocol is well-suited for high-speed big data applications. Consequently, the entire image processing framework is built upon the ARM AMBA AXI4 protocol. This choice ensures that the overall framework possesses high bandwidth, low latency, and low power consumption, among other characteristics. These qualities are crucial for meeting the real-time requirements of the algorithm.
3.3.2. Modular Design of the Algorithm
Figure 7 presents the general architectural framework of the FPGA accelerator designed for the GS-AGC algorithm. The framework consists of two main components: the processing system and the FPGA acceleration core system. The processing system is responsible for access and control tasks and comprises ARM CPU cores and external memory. On the other hand, the FPGA acceleration core system is dedicated to implementing the algorithm’s acceleration tasks and is divided into three modules: the access module, the Gamma_trans image processing module, and the parameter cache module. To achieve higher processing speed in Vivado_HLS global processing and image processing, the framework employs a pipelined operation. This approach incorporates concurrent caching and processing of image data within the FPGA, enabling the implementation of IP cores with low latency and high bandwidth.
- (1)
Memory access module
During runtime, the CPU initializes the off-chip memory module and stores both the low-illumination image and the corresponding grayscale image data in the external DDR memory. Additionally, the CPU is in charge of transmitting the input image for processing to the access memory module via the AXI bus. The access memory module retrieves the input image from the external storage and caches it into the cache module.
- (2)
Parameter Cache Module
During transmission, to ensure the integrity of the transmitted data, the parameter cache module incorporates block random access memory (BRAM) located on the chip. Within the FPGA, the BRAM resources consist of individual RAMs that are primarily responsible for caching the image data and the results obtained from the Gamma_trans module after processing.
- (3)
Image processing module
The image processing module nonlinearizes the image through a brightness-based calculation of the pixels in the image, as well as by evaluating the brightness of the pixels in the surrounding region. The output of this operation is transferred to the image processing module to achieve the overall enhancement and regional suppression of the image.
To accelerate the algorithm, custom computation combined with parallel computation was utilized. A dedicated circuit was designed using Vivado 2017.4 to perform mathematical calculations for the custom computation, with multiple circuits performing parallel computations simultaneously. During the data parallel computation stage, a pipeline was used to fetch data from the cache module, perform calculations, and write the results back to the cache module. When traversing the pixels, the #Pragma HLS PIPELINE directive in Xilinx HLS guided the synthesis process and added the PIPELINE to the J loop, which was unrolled along the input image’s width direction. This enabled HLS to process one pixel per clock cycle. The HLS completely expanded the window calculation, allowing for the read of nine data at once and increasing the bandwidth. The algorithm is shown in Algorithm 3.
Algorithm 3 calulate loop |
Input: in_channel, rows, cols Output: gamma_val_1
- 1:
for r = 0→ rows do - 2:
for j = 0→ cols do #pragma HLS PIPELINE II = 1 - 3:
if ( >= && <=) - 4:
gamma_table[i][j] = gamma_val_1 - 5:
out gamma_val_1 = pow((log10((t1 + t2)/2)/log10(t1/2)), ratio_1)
|
For this purpose, the Gamma_trans nonlinearized operation was implemented as a pipelined multiply-add operation with the help of DSP Slice. The pipeline consists of fetching data from the cache module, performing the arithmetic operation, and writing the result back to the cache module. The pipelining cycle is determined by the execution time of the segment of the task with the longest execution time. By designing the pipelined system, the FPGA’s throughput could be improved, and the logic resources could be better utilized for the parallel processing of the two tasks of parameter caching and image processing.