High-Performance Enhancement of SWIR Images

Abstract

Short-wave infrared imaging technology has rapidly developed over the past decade. However, image detail and contrast enhancement techniques, which are crucial for infrared imaging systems, are rarely dedicated to SWIR imaging systems. Moreover, the existing IR image enhancement methods generally have difficulty providing real-time performance, which plays a significant role in imaging systems with high data rates. In this paper, we propose a simple and real-time SWIR image enhancement approach based on the difference of Gaussian filter and histogram remapping, and illustrate the implementation of the proposed method on FPGA. The experimental results demonstrated that our method achieves promising detail and contrast enhancement performance with a frame rate of around 50 fps for high-definition images, and that extremely high frame rates can be achieved with pipelined architecture.

1. Introduction

Short-wave infrared (SWIR) cameras have increasingly become popular due to their long-acting distance, controllable cost, and better fog transmission. These advantages are important in optoelectronic pod [1] applications, which typically integrate imaging systems with a long focal length and large F-number. Infrared (IR) images are generally processed by nonuniformity correction (NUC), bad pixel correction (BPC), detail and contrast enhancement, and dynamic range compression before being displayed [2]; detail as well as contrast enhancement are crucial to improving image quality. It is well-established that detailed information is represented in the form of edges (dramatic intensity changes within a spatial neighborhood) [3], so detail improvement can be regarded as edge enhancement in IR image processing. The dominant IR image detail improvement methods can be divided into mapping-based approaches, gradient-domain algorithms, and decomposition-based methods [3]. The typical disadvantages of gradient-domain algorithms [4] typically include being computationally expensive and having poor stability, while the mapping-based approaches [5,6] provide relatively limited improvements in the details. Consequently, the existing high-performance detail enhancement methods for IR image processing are decomposition-based methods [7], which separate a detail layer from the input image and assign higher weight before merging, providing outstanding detail and contrast enhancement performance.

In 2009, Branchitta et al. [8] first applied the bilateral filter to the IR image processing and proposed decomposition-based dynamic range partitioning (BF&DRP), but a careful tuning of several parameters is required in different applications, and serious gradient reversal artifacts caused by the bilateral filter are inevitable in some scenarios. To optimize the work in [8], Zuo et al. [9] proposed introducing the bilateral filter [10,11] for display and detail enhancement (BF&DDE), and an adaptive Gaussian filter was applied following the bilateral filter to suppress noise and artifacts. The BF&DDE achieved impressive enhancement performance, but at a high computational cost. In 2014, Liu et al. [12] proposed a method known as GF&DDE, which has a similar framework, but with the bilateral filter replaced by the guided image filter [13]. In our previous work [7], we achieved a good balance between detail enhancement and blinking pixel suppression by combining the fast guided filter [14] and plateau equalization [6] (FGF&PE), with the running time being superior to those of the baseline methods, including the BF&DDE and the GF&DDE, by a large margin.

However, the characteristics of SWIR images are different from those of medium-wave infrared (MWIR) and long-wave infrared (LWIR) images, especially images based on indium gallium arsenide (InGaAs) detectors [15]. The MWIR and LWIR images in long- range imaging systems are typically acquired by mercury cadmium telluride (MCT) IR detectors, which are susceptible to flickering noise (blinking pixels) [16]. The flickering noise of the detector is a characteristic of random telegraphic signals (RTSs), caused by the presence of defects in semiconductor crystals. The resulting location of the blinking pixels changes with time, and the response state is unstable, meaning it cannot be detected and corrected as defective pixels. Consequently, the existing decomposition-based IR image enhancement approaches focus on the performance of blinking pixels suppression. SWIR images are typically acquired by InGaAs detectors, which have fewer blinking pixels than MCT detectors. Blinking pixels are generally suppressed at the cost of detail enhancement performance, meaning that the existing decomposition-based methods [7,8,9,12] used for MWIR and LWIR enhancement cannot yield suitable SWIR image processing results.

More importantly, SWIR images have more detail information than MWIR and LWIR images with the same pixel resolution and optical parameters due to the optical diffraction limit (radius of the Airy disk), which is strongly related to the wavelength; the requirement for SWIR detail enhancement is higher than for MWIR and LWIR images.

In addition, the full-scale frame rate of SWIR (InGaAs) detectors (up to around 300 fps) is much higher than that of MCT detectors (up to around 100 fps). The existing IR image enhancement methods [7,8,9,12] are typically based on edge-preserving filters, including the bilateral [10,11] and guided filters [13], which have difficulty meeting the requirements of applications with high data rates [17].

In this paper, we propose a simple and robust SWIR image-enhancement method based on the difference of Gaussian (DoG) filter [18] and plateau equalization. Compared with our previous work [19], this paper focuses on the description and analysis of strategies for FPGA porting, which is helpful for the application of imaging systems. The primary contributions are given as follows:

To the best of our knowledge, there has been no implementation of the DoG filter to IR image enhancement. We demonstrate that the DoG filter has many advantages and potential for edge and detail extraction in SWIR imaging systems.

The running speed of the proposed method is faster than those of the methods used for comparison by a large margin, with a frame rate of around 50 fps for high-definition (HD) SWIR images, and can be simply and significantly accelerated by pipeline architecture, which we describe in Section 3.3.

The results of FPGA implementation and laptop CPU processing demonstrate that our method has the potential for engineering applications.

This paper is organized as follows: Section 2 provides background information about the DoG filter and plateau equalization. Section 3 describes the details of the proposed method, and provides application guidance for FPGA-based system. Section 4 discusses the results with different parameters and running modes, and the proposed method is compared with other methods. Finally, we conclude with the merits and limitations of our study, and outline future directions in Section 5.

2. Related Work

2.1. Difference of Gaussian Filter

The DoG filter has been widely used in computer vision tasks as an edge detector [20,21], and the DoG filter can be explicitly formulated by:

$$DoG(i,j,{\sigma }_1,{\sigma }_2)=\frac{1}{2\pi {\sigma }_1^2}{e}^{-(i^2+j^2)/2{\sigma }_1^2}-\frac{1}{2\pi {\sigma }_2^2}{e}^{-(i^2+j^2)/2{\sigma }_2^2},$$

(1)

where $(i,j)$ denotes the pixel coordinates in the image; ${\sigma }_1$ and ${\sigma }_2$ represent the standard deviations of the Gaussian filter. In the case of ${\sigma }_2/{\sigma }_1=1.6$, the DoG filter can be seen as an ideal approximation of the Laplacian of Gaussian (LoG) filter [21], which is one of the most commonly used edge detectors. Ref. [21] demonstrated that the DoG filter can be regarded as a narrow-center Gaussian filter subtracted by a wider surround Gaussian filter, which is equivalent to a band-pass filter consisting of two low-pass filters with different cut-off frequencies, as illustrated in Figure 1.

2.2. Plateau Equalization

The PE [6] is a widely used histogram equalization method that remaps the original histogram by applying a clipping limit to the bins with large grayscale values, which are usually regarded as the background in the image. Previous work [7] demonstrated that the PE robustly performs in IR image histogram equalization and remapping. Let $H$ and $C_{TH}$ represent the histogram of the input image and the clipping threshold, respectively, and let the variable m denote the grayscale distributed from 0 to $(L-1)$; then, the clipped histogram can be formulated by:

$$H_c(m)=\{\begin{array}{l}{C}_{TH}, H_c(m)>C_{TH}\\ H(m), \mathrm{others}\end{array}.$$

(2)

The cumulative distribution function (CDF) based on the clipped histogram can be expressed as follows:

$$CDF(m)=\{\begin{array}{ll}\frac{H_c(0)}{N},& m=0\\ \frac{{\displaystyle {\sum }_{n=0}^m{H}_c(n)}}{N},& 0<m\le L-1\end{array},$$

(3)

where $N$ represents the total number of pixels. The CDF value ranges from 0 to 1, and the output grayscale after remapping can be calculated by the following formula:

$$I_{PE\_out}(i,j)=CDF(I_{PE\_in}(i,j))\times (L^{\prime }-1),$$

(4)

where $(i,j)$ denotes the pixel coordinates; $L^{\prime }$ denotes the grayscale range of the output image; and $I_{PE\_in}$ and $I_{PE\_out}$ represent the input and output images of the PE processing, respectively. The proportion of grayscales dominated by detail information significantly increases due to the reshaped histogram, which means that the details account for a more dynamic range. Previous work [7] demonstrated that the PE works well in simultaneously enhancing the dynamic range and preserving the gradient distribution.

3. Principle of the Proposed Method

In general, edges in images are defined by significant intensity changes within a spatial neighborhood and high local contrast. Enhancement processing further increases the visual distinctiveness of these locations. The DoG filter was demonstrated to extract edge lines with high quality without post-processing, particularly when synthesizing line drawings and cartoons [21]. Given that the key principles of image detail enhancement are edge sharpness and dynamic range improvement, and being inspired by previous conclusions [7,21], we designed a fast and robust method for SWIR image detail enhancement. The framework of our proposed method is illustrated in Figure 2. We efficiently and accurately extracted the edge (detail layer) by using the DoG filter, and assigned a higher weight before fusion, before finally remapping the fused image by performing plateau equalization. It should be noted that our primary aim was to provide an FPGA implementation strategy; for the convenience of readers, we introduce the enhancement method in detail, which was the main content of our previous work [19].

Note that the input image at the enhancement stage was processed by nonuniformity correction and defective pixels correction. The image enhancement procedure is fairly simple, and we illustrate how to enhance image details by the DoG filter in the following sections.

3.1. Enhancement via Fast Edge Extraction

The key parameters of the DoG operator are the standard deviation of the two Gaussian functions and the window size, both of which significantly influence detail extraction performance. Because the DoG filter can be regarded as an approximation of the LoG filter in the case of ${\sigma }_2/{\sigma }_1=1.6$, the LoG filter has the highest response on a blob-like target with a size of $2\sqrt{2}\sigma$ [22], which also works for edge extractions with that size. Given the aforementioned issues, the equations set for the definition of the two standard deviations are established as follows:

$$\{\begin{array}{l}s=(2\sqrt{2}{\sigma }_1+2\sqrt{2}{\sigma }_2)/2\\ {\sigma }_2/{\sigma }_1=1.6\end{array}.$$

(5)

The size of the edge in SWIR images generally ranges from 3 to 6 pixels. We take the value of $s=4$ for demonstration, and compare the performance of different edge and operator window sizes in the next section. The solution to Equation (5) is ${\sigma }_1=1.088$ and ${\sigma }_2=1.741$ when s = 4. Taking the window size of 7 as an example, it is easy to formulate two Gaussian filters with a window size of 7 and standard deviations with value of ${\sigma }_1$ and ${\sigma }_2$. Consequently, the DoG operator can be simply formulated according to Equation (1). The explicit matrix of the DoG operator is given as follows:

$$F_{DoG}=\left[\begin{array}{ccccccc}-0.0029& -0.0061& -0.0090& -0.0099& -0.0090& -0.0061& -0.0029\\ -0.0061& -0.0107& 0.0088& -0.0047& -0.0088& -0.0107& -0.0061\\ -0.0090& 0.0088& 0.0168& 0.0398& -0.0168& -0.0088& -0.0090\\ -0.0099& -0.0047& 0.0398& 0.0775& 0.0398& -0.0047& -0.0099\\ -0.0090& 0.0088& 0.0168& 0.0398& -0.0168& -0.0088& -0.0090\\ -0.0061& -0.0107& 0.0088& -0.0047& -0.0088& -0.0107& -0.0061\\ -0.0029& -0.0061& -0.0090& -0.0099& -0.0090& -0.0061& -0.0029\end{array}\right].$$

(6)

3.2. Equalization and Remapping

According to the framework illustrated in Figure 2, detail information can be obtained by applying the DoG operator to the input image, and sharpened by gain assignment and refusion with the input image. Give that IR image detail sharpening parameters in applications, including photoelectric pods, are generally divided into several different grades, we set the customer-selected gain coefficient to 6 grades in our applications, which starts at 4 and increases with a step size of 4. We also found that 16 robustly works as a default gain coefficient. Letting $I_{in}$ and $I_d$ represent the input image and the DoG filtering result, the sharpened image is calculated by:

$$I_s=I_{in}+k\times I_d.$$

(7)

where k is a gain coefficient. The difference between the sharpened image $I_s$ and the input image $I_{in}$ is the edge information, and further histogram equalization and dynamic range compression are still required for improve the resulting visual effects. More importantly, the sharpened image may have pixels with negative grayscale due to the DoG filter processing. To facilitate equalization calculation, we propose processing the sharpened image as follows:

$$I_s^{\prime }=R(I_s-\min (I_s)).$$

(8)

where the operatorand min and R represent the minimum gray value calculation and rounding operation, respectively. We then perform plateau equalization, which was described in Equations (2) and (3), with a clipping threshold $C_{TH}$ of 0.01% of the total pixel number because it performs well in most experiments. The output grayscale $L$ is typically set to 256, which is consistent with the grayscale range of most commonly used monitors.

A demonstration of the input image, detail image using DoG filtering, sharpened image, and the output image after equalization and reprojection are shown in Figure 3. The edge in the sharpened image is sharper than in the input image due to the DoG filter efficiently extracting edge information, and the dynamic range is substantially improved by PE processing.

3.3. Guidance for Implementation on FPGA

The image processing in IR imaging systems is basically performed on field-programmable gate arrays (FPGAs), which have difficulty in processing floating-point numbers. To facilitate implementation on FPGAs, we multiply the parameters in Equation (6) by the integer 1024, and generate a new filter ${F_{DoG}}^{\prime }$ by performing the rounding operation:

$${F_{DoG}}^{\prime }=\left[\begin{array}{ccccccc}-3& -6& -9& -10& -9& -6& -3\\ -6& -11& -9& -5& -9& -11& -6\\ -9& -9& 17& 41& 17& -9& -9\\ -10& -5& 41& 79& 41& -5& -10\\ -9& -9& 17& 41& 17& -9& -9\\ -6& -11& -9& -5& -9& -11& -6\\ -3& -6& -9& -10& -9& -6& -3\end{array}\right].$$

(9)

We first perform the filter illustrated in Equation (9) to extract details, and then divide the result by 1024 (1024 times can be simply calculated by 10-bit shifting left on FPGAs) for subsequent refusion processing. Assume that the number of rows and columns of the input image are m and n, respectively, and let w represent the size of the filter. The architecture of the filter implemented on the FPGA is shown in Figure 4. The coefficient $C_{ij}$ is the element in the filter ${F_{DoG}}^{\prime }$, and $A_{ij}$ represents the result obtained by traversing all of the flip-flops and multipliers for each pixel. The pixel DoG filtering result is calculated by Equation (10). The gray value of all pixels after filtering does not drastically change due to the processing architecture in Figure 4.

$$I_d(i,j)=\frac{{\displaystyle \sum _{x=1}^w{\displaystyle \sum _{y=1}^w(A_{xy})}}}{sum({F_{DoG}}^{\prime })}$$

(10)

Notably, IR image processing generally requires synchronous dynamic random access memory (SDRAM) with a data width of 32 bits to facilitate multiplication calculations. For single-frame image processing, which is insensitive to running time, we sequentially perform NUC, BPC, detail extraction, refusion, and PE processing. For video processing, which requires high real-time performance, we apply pipeline architecture to achieve a high frame rate, as shown in Figure 5. Let $w$ represent the window size of the DoG operator and $t_1$ denote the delay of the DoG filtering for detail extraction processing. $t_1$ equals $(w-1)/2$ due to the symmetric padding of the input image. because the default gain factor is set to 16, pixel-wise refusion can be simply calculated by applying 4-bit left shift to the detail layer and performing an addition operation. We adopt the CDF of the previous frame for PE processing because the difference between two consecutive frames in high-frame-rate videos is slight. Basing the CDF calculation on the current frame results in a delay of one frame, which is unacceptable for some high-performance systems used for tracking tasks.

Image enhancement using the architecture in Figure 5 can achieve extremely high frame rates at a low computational cost and without any limitation on frame rate (frame rate is not limited by the enhancement method), so is suitable for embedded systems. More importantly, the framework in Figure 5 can be further accelerated by parallel computation including dividing the image into several blocks.

4. Experimental Results Comparison and Discussions

To evaluate the enhancement performance of the proposed method, we applied it to HD SWIR images (including the image shown in Figure 3a) that were acquired by a self-developed SWIR camera with a resolution of 1280 × 1024 and a focal length of 600 mm, which is shown in Figure 6. The brief characteristics of the test images are described in

Table 1. Summary of test images.

Samples	Resolution	Capture Condition	Image Characteristics
Image 1	1280 × 1024	Outdoor/noon/low integration time	Electric box at a distance of 400 m
Image 2	1280 × 1024	Outdoor/afternoon/long integration time	Tower cranes around 4.5 km away
Image 3	1280 × 1024	Outdoor/afternoon/medium integration time	Building with fence around 1 km away
Image 5	1280 × 1024	Outdoor/noon/medium integration time	Building around 1 km away

; the original test images were blurry due to the atmospheric turbulence and point spread function (PSF) of the optical lens.

Because there is no universal objective criterion for IR image quality assessment, and several blind image quality assessment metrics perform inconsistently on IR images [3], we performed a comparison using the average gradient (AG), which is widely used as an indicator for the evaluation of the edge detail contrast and sharpness characteristics of an image [4]. The AG can be calculated as follows:

$$AG=\frac{1}{(R-1)\times (C-1)}\times {\displaystyle \sum _{x=1}^{R-1}{\displaystyle \sum _{y=1}^{C-1}\sqrt{\frac{{(\frac{\partial I(x,y)}{\partial x})}^2+{(\frac{\partial I(x,y)}{\partial y})}^2}{2}}}}$$

(11)

where $I(x,y)$ is the pixel value of the image; R and C denote the number of rows and columns of the image, respectively; and $\frac{\partial I(x,y)}{\partial x}$ and $\frac{\partial I(x,y)}{\partial y}$ represent the horizontal and vertical gradients, respectively.

4.1. Performance Comparison of Different Parameters

To compare the edge extraction performance of different DoG filter parameters, we compared the filtering results obtained using different window and edge size (defined by the two standard deviations in Equation (5)), with the gain factor and PE clipping threshold fixed to 16 and 0.01%, respectively. The DoG filtering result and the final enhancement result produced by the proposed method on test image 1 are shown in Figure 7 and Figure 8, respectively.

Figure 7 and Figure 8 demonstrate that the window size of the DoG operator has a significantly stronger impact on image sharpness than the standard deviation values, while the extraction result did not show a notable difference among different preset edge sizes. As the operator with a small window size is sensitive to noise, as shown in Figure 7, and a large window size may cause a wider edge transition area and have negative impacts on image quality, we set the window size to seven in the subsequent experiments. The edge scale, which defines the standard deviation, was set to four because it robustly worked in most of our experiments.

4.2. Performance between CPU and FPGA

To evaluate the implementation performance on an FPGA, we applied the DoG filter to the test images with floating-point numbers and integers, which are shown in Equations (6) and (7), respectively. A comparison between the enhanced results on the test images by using CPU (floating-point numbers) and FPGA (integers) is shown in Figure 9. It can be seen that the difference is slight, proving that the proposed method does not suffer from serious performance degradation during implementation on an embedded system such as an FPGA. The average gradient values of different filtering parameters are provided in

Table 2. Average gradient results between enhancement by CPU and FPGA.

	CPU	FPGA
Image 1	25.4599	25.0362
Image 2	23.1785	23.9471
Image 3	31.7146	33.2149
Image 4	37.6565	37.5983
Mean value	29.5024	29.9491

, which further illustrates that the difference is slight, meaning the proposed method has application potential.

To further evaluate the resource requirement of the proposed method, we implemented our method on Xilinx Artix-7 FPGA (XC7A100T-2SBG484I); and the hardware use (with a clock frequency of 108 MHz), including the lookup table (LUT), slices, flip-flops (FFs), block random access memory (BRAM), and digital signal processing (DSP) slices, is shown in

Table 3. Hardware used on FPGA.

	LUT	Slices	FF	BRAM	DSP Slices
Hardware use	946	311	982	6	31
Total resources of the FPGA	101,440	15,850	126,800	135	420

.

4.3. Performance among Different Methods

For the enhancement performance comparison among the proposed and other methods, we performed classic decomposition-based methods, including BF&DDE [9], GF&DDE [12], and FGF&PE [7], as well as PE [6] as comparison methods, to demonstrate the effect of detail extraction and fusion on image quality.

The enhancement results on Image 1 are shown in Figure 10. The input image was acquired in low-light conditions, and all of the comparison methods significantly improved the dynamic range. The proposed method better enhanced detail at the cost of slightly higher noise. The result processed by BF&DDE had obvious artifacts, and the PE, GF&DDE, and FGF&PE produced limited detail enhancement. The proposed method achieved a good balance between detail enhancement and noise, with a better visual effect.

Image 2 was captured under strong sunlight. The distance of the tower crane was around 4.5 km, and the original image was obviously blurry due to the point spread function of the optical lens and atmospheric turbulence. The results in Figure 11 demonstrate that the dynamic range improvement performance of the five methods were not significantly different, but our method produced a significant sharpening effect on edges, especially in areas with fonts (zoom in on Figure 11 for details), being superior to the methods used for comparison by a large margin.

Image 3 was acquired with a short integration time, and the signal-to-noise ratio (SNR) of the input image is relatively low. The proposed method yielded the best performance at an acceptable noise level, and its improvement in detail was significantly higher than that produced by all comparison methods, as shown in Figure 12.

Image 4 was captured under strong sunlight, and there were distinct bright and dark areas in the image. The results in Figure 13 demonstrate that our method had a promising enhancement performance on dense edges in both bright and dark areas.

The average gradient results of different methods are given in

Table 4. The average gradient results among different methods.

	PE	BF&DDE	GF&DDE	FGF&PE	Proposed
Image 1	13.5015	18.5207	11.4644	15.2999	25.0362
Image 2	13.1319	19.3775	11.6492	13.0507	23.9471
Image 3	15.7237	18.5607	10.9612	16.3301	32.2149
Image 4	18.3325	24.8741	15.3778	20.0062	37.5983
Mean value	15.1724	20.3333	12.3632	16.1717	29.6991

. The decomposition-based methods typically produced better enhancement performance than PE, while the proposed method yielded the best AG result, with a mean value much higher than that of the other methods. More importantly, the enhancement performance provided by using the DoG filter before PE (the proposed method) was significantly better than that achieved by using PE only, with clear improvements in both visual effect and average gradient.

We performed different methods by running them on a laptop with Apple M1 Pro and 16 GB RAM for a fair comparison, and the mean values of running time were obtained by running all the methods 10 times. The results shown in

Table 5. Running time comparison of different methods (ms).

	PE	BF&DDE	GF&DDE	FGF&PE	Proposed
Image 1	14.3	207.6	252.1	106.3	18.9
Image 2	14.5	202.3	249.6	104.1	20.0
Image 3	14.6	201.4	261.4	101.2	19.7
Image 4	14.6	196.4	247.6	102.5	20.6
Mean value	14.5	201.9	252.7	103.5	19.8

demonstrate that the proposed method ran quickly, with the running time being slightly longer than that of PE, but significantly less than those of the other decomposition-based methods. Notably, we performed our method in serial mode, which takes much longer than pipelined architectures. Although the frame rate of the proposed method is around 50 fps, we could accelerate it according to the operational guidance of the pipelined architecture in Section 3.3, which can be simply applied in FPGA-based imaging systems. The frame rate is only limited by the output data rate of the IR detector and bandwidth of the imaging system.

4.4. Limitations

Although our method is simple, efficient, and practical, it has some drawbacks, which can be the focus of future research. First, DoG filters are susceptible to noise, especially flickering noise caused by blinking pixels. The current InGaAs SWIR detectors have fewer blinking pixels, but the proposed method cannot be directly performed for MWIR and LWIR image enhancement because he MCT-based detectors suffer from blinking pixels. Second, the dynamic range reprojection performance of PE can be further optimized and improved in the future.

5. Conclusions

In this paper, we proposed a fast and simple method to enhance SWIR images. The experimental results demonstrated that the proposed method yielded the best performance with a low computational cost, being superior to the methods used for comparison by a large margin. More importantly, we demonstrated that the proposed method can be simply accelerated by applying a pipeline architecture.

Although the proposed method works well in SWIR image processing, it has limitations in terms of the MWIR and LWIR image processing. In addition, the proposed method is sensitive to flickering noise and defective pixel clusters. Potential future directions include further improving the performance by accurately extracting detail information and suppressing the flickering noise simultaneously.