1. Introduction
The infrared sensor can capture the thermal radiation emitted by the objects, which is less impacted by the dark condition. It is widely applied in detection, scene surveillance, reconnaissance, and navigation, etc. due to its ability to operate 24 h a day. However, IR images do have many obvious shortcomings, including low contrast, weak details, and blurred resolution, when compared with visible images, which may trigger much inconvenience when people observe the images. Consequently, infrared sensors in high dynamic range (>8 bit) are always applied in practical application to capture more details in these years. If displaying the HDR images on normal facilities (8 bit) directly, some information in the original image could not be represented. The procedure to achieve high-quality visualization of HDR infrared image must take the following problems into consideration. First and foremost, the dynamic range of the output should be mapped to be acceptable for the display device. Meanwhile, in order to take advantage of the HDR sensor and guarantee convenience for the following work, weak details should be enhanced. Last but not least, the output should be as visually pleasing as possible.
The core idea of many conventional approaches is adjusting the distribution of the gray level. Those methods usually include linear stretching and curve stretching (logarithmic, gamma, and sigmoid) methods based on histogram equalization [
1] (HE) and gradient domain methods.
Linear stretching can compress the dynamic range to be acceptable for the display equipment, while it leads to detail losing. Curve stretching like Gamma transformations can increase the image contrast, while the fitting parameters change when the images are different. For each image, manual selection of parameters with experience is required.
HE enlarges the contrast of the image by redistributing the image pixel values so that the number of pixels in each gray level is approximately equal. Researchers have made plenty of efforts based on HE [
2,
3,
4,
5,
6,
7,
8], including global HE-based approaches and local HE-based approaches. The global HE-based approaches like Plateau HE [
3,
4] and its improved ones [
2,
7] can make the gray levels distribute more reasonably, but the ability to preserve the details is insufficient. The local HE-based approaches like the partially overlapped sub-block histogram equalization (POSHE) [
8] and contrast-limited adaptive histogram equalization (CLAHE) [
5] can generate more details, but they are intended to induce artifacts, over-enhancement, and blocking effects. In short, most of the existing HE-based methods cannot avoid the contradiction between the maintaining of local details and the global consistency of the entire image, because they only take the histogram information into consideration.
The main idea of gradient domain [
9,
10] methods is attenuating the large gradient while expanding the small gradient to produce the modified gradient field, and then reconstruct the result image by solving a Poisson equation. Generally, gradient domain operators are capable of achieving appropriate dynamic range compression and avoiding artifacts like halo and gradient reversal. However, they may be limited in enhancing the local details effectively, and cautious selection of parameters is necessary, which limits the practical application of these methods.
Recently, multiscale decomposition methods are critical and prevalent in the domain of HDR image display. These methods introduce filters to decompose the image into several components followed by processing each component separately, then compose the processed components to yield the result image. F.Durand et al. [
11] introduced a method to display HDR images. In their method, the image is decomposed into the base layer and the detail layer by the bilateral filter. Then, they attenuate the contrast of base layer, while the contrast of the detail layer is kept, thereby the details are preserved. Nowadays, an increasing number of researchers have adopted the use of a Guided Filter [
12,
13] to process the image for its simplicity and efficiency without the gradient reversal artifacts. B. Gu et al. [
14] presented an edge-preserving filter with locally adaptive property, which is particularly effective in preserving or enhancing local details. Besides, researchers also applied hat-top transform [
15,
16] or wavelet transform [
17] to decompose the image, then process the layers individually followed by composing them. Although multiscale decomposition methods have strength on details enhancing, they sometimes trigger halo artifacts in strong edges.
Currently, based on the human visual system (HVS), Retinex theory [
18,
19], researchers have presented various methods [
20,
21,
22,
23,
24,
25,
26]. The critical challenge of those algorithms is the contradiction between calculation speed and the processing effect. On the one hand, better results can be obtained by constructing more complex models; on the other hand, the intricate structure of the model increases the calculation complexity, which limits the widespread application of the algorithm. Meanwhile, HVS-based approaches are more appropriate for the visual images with sufficient details rather than the infrared images lack of details in general.
In conclusion, the inadequacies of the existing methods mainly include (1) the contradiction between the maintaining of local details and the global consistency of the entire image, (2) too many parameters need to be selected manually with experience, and (3) poor robustness for dim images lack of details.
Nowadays, infrared and visible light images fusion [
27,
28] and the fusion of multi-exposure images [
29,
30] are both research hotspots in image processing. The visible light sensor mainly captures the reflected light so that the visible light image has abundant background information. In contrast, the infrared sensor can capture the thermal radiation emitted by the object, and it is less impacted by the dark condition or the dim weather. Therefore, the fusion of infrared and visible light images can guarantee more complex and detailed scene information. Similarly, each of multi-exposure images has its own unique details. If these details are well fused into one image, a high-quality image with multiple details can be produced.
This paper presents a novel approach based on adaptive transform and image fusion to overcome the problems above and display HDR infrared images on LDR display equipment with appropriate contrast and clear and abundant detail information. Inspired by the idea of image fusion, we transform the original image into multiple brightness by gamma transformation followed by multiscaled guided filter enhancement to keep and enhance the details in the entire image. In order to simplify the selection of parameters, we adapt the energy of the gradient (EOG) to guide the transformation, and the entropy is utilized to guide the multiscaled guided filter enhancement. The experimental results can prove that our method can achieve acceptable results with the fixed parameters. For typical HDR infrared images of various scenes, the effect of our method is robust.
The rest of this paper can be chiefly described as follows.
Section 2 describes the fundamental theory and specific steps of our proposed method. In
Section 3, our experiment comparison of the methods are described in detail. In
Section 4, the conclusion of the paper is presented. Finally, the acknowledgment is made in
Section 5.
4. Discussion
Image group
Figure 4 is an example of infrared images of rich scene information including human, bicycles, benches, ground, and so on. HE and CLAHE enhanced the contrast, while a large amount of local details lost. The AHPBC and MSR can enhance the details to some extent, while the dynamic range of the result image is so small that the visibility is poor. The result of Reinhard is visually comfortable but the some texture information is still ambiguous. LEP can enhance the image well in general, but generates a halo. Compared with other six approaches, our method meets the best performance.
Image group
Figure 5 and
Figure 6 are examples of low contrast image, which contains many details about the texture. The dynamic range of the original IR image is so narrow that HE and CLAHE fail in the enhancement of the details, while some regions in their results are over-enhanced, and some noises are generated. AHPBC, MSR, and Reinhard can preserve the global contrast but has a relatively weak compatibility in the enhancement of the local details. LEP can successfully enhance the edge of the humans in the image, but some tiny details like the texture of the road are still dim. Our method yields the best enhancement results, producing global detail enhancement without noise generation.
Image group
Figure 7 and
Figure 8 are examples of foggy images. Details like outlines of trees and human are unobservable in the results of HE and CLAHE, and the backgrounds are distorted. Compared with the original linear mapped image, the results of AHPBC and Reinhard are still blurred, even though there might be a great change in brightness. MSR can increase the contrast to some extent, but the effect on local detail enhancement is relatively weak. The noise in the result of LEP is obvious. It can be indicated by the comparison of the results in
Figure 7 and
Figure 8 that the proposed method creates the most visually comfortable results, which reveal the details most fully.
Image group
Figure 9 and
Figure 10 are examples of image with blurred details. Due to low contrast and weak details in the original image, HE and CLAHE not only fail to reproduce the details, but also generate noises. Objects such as trees, buildings, and pedestrians in AHPBC’s results are blurred. The results of MSR and Reinhard are too dark to observe the information. Relatively, the results of LEP and the proposed method are visually pleasing; comparing with the results of LEP, noise in the proposed results is weaker.
The results of the Tenengrad for the test images are shown in the
Table 2. In theory, the higher value of Tenengrad, the clearer the entire image. In accordance with the result of visual comparisons, the proposed method and LEP achieve higher Tenengrads.
As being reported in
Table 3, comparing about the entropy, the proposed method and LEP have the robust result, and our method obtains the slightly better value than LEP does. Practically, there are more details in the results of our proposed method.
As being reported in
Table 4, comparing about the NIQE, lower value of NIQE reflect better perceptual quality of image. the overall difference of AHPBC, MSR, Reinhard, LEP, and our proposed method is not obvious.
As being reported in
Table 5, lower value of NIQE reflect better perceptual quality of image. In general, the proposed method and LEP achieve better results, but the average result of our proposed method is the best.
As being reported in
Table 6, as our approach introduce multiscale analysis and image fusion, the calculation time of the proposed algorithm is much more than the conventional and famous methods HE, CLAHE, MSR, and Reinhard. Our method runs relatively slower than LEP, but more quickly than AHPBC. How to accelerate our algorithm is one of the key points of our future work. Hardware acceleration is one of our choices. After optimization, our method is very likely to process image on real-time application.
All in all, performance of the proposed algorithm is verified by experiments with images with various characteristics. The above analysis of the results shows that the proposed method has strength in detail enhancement of the HDR infrared image. The dynamic range compression and detail enhancement results are visually comfortable without excessively obvious noise.
5. Conclusions
In this paper, a novel high dynamic range infrared image enhancement method is introduced. This method is capable of compressing the dynamic range, adjusting the gray levels, and enhancing the details effectively. The proposed approach is mainly based on adaptive Gamma correction, multiscale guided filter, and image fusion. First, in order to keep the weak global details in different area, we adopt an EOG-guided Gamma transformation, which is adaptive to adjust the original normalized image into multiple brightness levels. Second, the multiscale guided filter is utilized iteratively to decompose each brightness image into a base layer and several detail layers. Details in each image are enhanced separately and composed adaptively. Third, to obtain the image with global details of the input image, enhanced image in each brightness is fused together. Last, we filter out the bad pixels and adjust the dynamic range before outputting the image. Tested on HDR IR images of different scenes with sundry details and background, the experiment result indicates that the proposed method can compress the dynamic range while higher the contrast, enhance the details effectively, and generate a visually pleasing result. It should be pointed out that in the step of guided transformation, the EOG function is just chosen to guarantee the simplicity and correctness of the algorithm. That is to say, the function could be changed according to the case with flexibility in the future work. Meanwhile, the method of the enhancement of the decomposed layers could also be extended, which also provides new point for the research.