Ghost-Free Multi-Exposure Image Fusion Technology Based on the Multi-Scale Block LBP Operator

Abstract

This paper proposes a ghost-free multi-exposure image fusion technique based on the multi-scale block LBP (local binary pattern) operator. The method mainly includes two steps: first, the texture variation, brightness, and spatial consistency weight maps of the image are computed, and then these three image features are used to construct the initial weight map. Finally, the multi-resolution method is used to fuse the images to obtain the resulting image. The main advantage of this technique lies in the step of extracting the details of the source image based on the multi-scale block LBP operator, which is used to preserve the details of the brightest and darkest areas in high dynamic range scenes and preserve the texture features of the source image. Another advantage is that a new LBP operator-based motion detection method is proposed for fusing multi-exposure images in dynamic scenes containing moving objects. In addition, this paper also studies two spatially consistent weight distribution methods and compares and discusses the effects of these two methods on the results of dynamic image fusion. Through a large number of experimental comparisons, the superiority and feasibility of this method are proved.

1. Introduction

One of the basic functions of digital cameras is to obtain real-world information naturally and vividly, which is comparable to the information captured by the human eye. However, current digital camera technology still has many limitations in obtaining rich texture details, vivid colors and brightness. This is because the dynamic range of the real scene is larger (${10}^{-6}-{10}^{6}cd/m^2$ [1]), while the dynamic range captured by the sensor in ordinary cameras [2] is small compared to this. A photo can only capture the smaller dynamic range of the real world. Therefore, in HDR (high dynamic range) scenes, the picture is often underexposed or overexposed. The most direct way to deal with this problem is to use high dynamic range devices [3] to acquire and display real scenes. However, these devices are often expensive and cannot be used universally [4].

In recent years, software-based solutions have gradually entered people’s field of vision. Compared with hardware-based methods, software-based methods are easy to implement, inexpensive, and suitable for ordinary cameras [5]. Existing software-based solutions fall into two main categories: HDR imaging techniques and multi-exposure image fusion (MEF). HDR imaging technology uses multiple exposures of images captured with ordinary cameras to estimate the camera response function (CRF) [6,7], resulting in a high dynamic range image. Then, the image is compressed, and the tone mapping method [8] is used to convert the high dynamic range image into a low dynamic range (LDR) image so that the image can be visualized on common display devices [9,10,11].

However, the complexity of HDR imaging technology is high and the time required is long, which is not suitable for ordinary cameras [12]. Different from HDR imaging technology, the multi-exposure fusion method does not need to construct a HDR image. It extracts pixels with more information, better exposure, and higher image quality from the input multiple-exposure low dynamic range image, and then fuses them. The resulting fused image does not need to be processed and can be displayed on an ordinary display device [13]. Compared with HDR imaging technology, the multi-exposure image fusion method has lower computational complexity and faster speed, so this type of method is the first choice for ordinary cameras [12].

In the existing multi-exposure image fusion techniques, the source image sequence is obtained under different exposures, and the difference in exposure causes the loss of image details. When there are moving objects in the captured image, if the images are fused directly, ghosting artifacts can appear in the resulting image. The elimination of ghosting artifacts is a major difficulty in the current multi-exposure fusion technology.

As early as the 1980s, Brut et al. [14] proposed Laplacian pyramid decomposition, which can effectively fuse two images. Subsequently, this method is used in many image fusion techniques. Mertens et al. [12] proposed a multi-resolution fusion method. This method uses contrast, saturation, and exposure as three quality measures to construct a weight map, and then fuses the images with a Laplacian pyramid decomposition method. The images fused by this method have good saturation but lack detailed information, which is not suitable for processing images in dynamic scenes.

Shen [15] proposed a method based on the generalized random walk framework, which deals with image color distortion where the fused image lacks correct color information.

The method proposed by Gu et al. [16] fuses multi-exposure images in a gradient field, generates the gradient value of each pixel by maximizing the structure tensor, and derives the pixel value from the gradient field that best represents the geometric features in the scene. This method can extract some detailed information, but the resulting image is darker, and the details in the dark areas are lost.

Li and Kang et al. [17] used a multi-exposure fusion method based on median filtering and recursive filtering. This method uses color dissimilarity to eliminate the influence of moving objects, and to a certain extent, it can eliminate ghosting caused by motion, but it does not process the color information effectively, resulting in color distortion of the fused image.

Shen [18] proposed a new hybrid exposure weight metric. This method is characterized by the use of mixed exposure weights to guide Laplacian pyramid enhancement. His method can maintain the color appearance and texture structure of the image, but cannot maintain the edge well, and the algorithm has high complexity and low efficiency.

Bruce et al. [19] calculated the information entropy and the neighborhood radius of the pixel in the logarithmic domain and then set the weight according to the value of the information entropy of each pixel point. The result of this method is that although the information entropy is high, the overall color of the image is dark and distorted.

The main feature of Li and Wang [20] et al.’s method is the use of structure tensors to extract image details. The resulting images have better color saturation and better preservation of image texture details, but this method handles image sequences in dynamic scenes, and there are ghosting artifacts in the resulting images.

Hayat et al. [21] proposed a multi-exposure image fusion technique based on multi-resolution fusion, which estimates the weight map using contrast, saturation, exposure, and color dissimilarity, and finally uses a multi-resolution method of pyramid decomposition to obtain the fused image. The benefit of this technique is that it can avoid the seam problem well, but when moving objects appear in multiple images, ghosting artifacts can still be generated.

The characteristics of the method of Huang [22] et al. are that the resulting image has rich colors and appropriate exposure, but some texture details are lost in the overexposed and underexposed areas of the image, and when fusing image sequences in dynamic scenes, results in significant ghosting in the image.

Our method solved the problems of brightness imbalance, insufficient retention of detail information in light and dark areas, and ghosting artifacts that occurred when fusing dynamic images. To solve the above problems, this paper constructed a new ghost-free multi-exposure image fusion model based on the multi-scale block LBP operator. In this method, for multi-exposure image sequences containing moving objects, the multi-scale block LBP operator [8,23,24,25,26,27] was used for local texture extraction in bright and dark areas and ghost removal caused by moving objects. On this basis, a new brightness adaptive method was also proposed to ensure the fused image had better visibility. After constructing the weight map, the discontinuous and noisy initial weight map was refined using filters, and finally, the weight map was fused by a multi-resolution method to obtain the final result image. The main contributions of this paper can be summarized as the following three points:

• This method took advantage of the multi-scale block LBP operator (MBLBP) and applied it to the multi-exposure image fusion method for the first time. We designed two quality metrics based on the multi-scale block LBP for local texture extraction and ghost removal in images, respectively.
• According to the brightness characteristics of the image, a new brightness adaptive method was proposed, which could adaptively adjust the brightness weight of the pixels in the source image sequence according to the brightness and darkness of the pixels.
• A new method of multi-exposure fusion based on the multi-scale block LBP was proposed. This method could fuse multi-exposure image sequences captured in static scenes and dynamic scenes.

The rest of this article is organized as follows. Section 2 discusses a detailed explanation of the proposed technique. Section 3 contains the comparison of experimental results, as well as parameter evaluation. Section 4 discusses the conclusions, limitations, and future work of this methodology.

2. Proposed Method

In this paper, a ghost-free multi-exposure image fusion technique based on the multi-scale block LBP operator was proposed. Our method constructed an initial weight map by computing three image features: texture features, luminance features, and spatial consistency features. We used the multi-scale LBP operator for local texture measurement, which could compute spatial details from LDR source image sequences. A new motion detection method based on the LBP operator was also proposed for fusing image sequences of dynamic scenes containing moving objects. Finally, the method used fast-guided filters to initially refine the weights, and a pyramid decomposition method to fuse images.

The above quality metric calculation methods and experimental results are discussed in detail in the following subsections. Figure 1 shows an overall flow chart of our method.

2.1. Texture Change Weight

When fusing image sequences captured in static scenes, two image features were considered: contrast and brightness. We used contrast as one of the weights so that the fused image had more texture and edge information. When using the multi-resolution method [12] to fuse multi-exposure images, it retained enough detailed information in the normally-exposed regions. However, since the texture details of the bright and dark areas were affected by the brightness, some of the details were lost in the bright and dark areas. Aiming at this problem, we innovatively proposed the subregion texture detail extraction method based on the multi-scale block LBP.

We let $I_i\left(x,y\right)$ denote the source image sequence, where $i$ = 1, 2, …, K denoted the ith image in the image sequence. In a set of multi-exposure image sequences, the average brightness of each pixel $\left(x,y\right)$ represented the brightness of the pixels in the HDR scene, which was calculated as follows:

$$L\left(x,y\right)=\frac{{{\displaystyle \sum }}_{i=1}^K{l}_i\left(x,y\right)}{K}$$

(1)

where $L\left(x,y\right)$ was the normalized average brightness of pixels at position $\left(x,y\right)$ in the image sequence, and $l_i\left(x,y\right)$ represented the brightness value of pixels at position $\left(x,y\right)$ of the ith image.

Next, we used the average brightness $L\left(x,y\right)$ at each pixel $\left(x,y\right)$ to divide the bright region $B_i\left(x,y\right)$, the dark region $D_i\left(x,y\right)$, and the normally exposed region $N_i\left(x,y\right)$ of each image. The calculation was as follows:

$$I_i^{g}x,y=\{\begin{array}{l}{N}_i\left(x,y\right),\alpha \le L\left(x,y\right)\le 1-\alpha \\ B_i\left(x,y\right),1-\alpha \le L\left(x,y\right)\\ D_i\left(x,y\right),L\left(x,y\right)\le \alpha \end{array}$$

(2)

where $I_i^{g}x,y$ was the grayscale image and $\alpha$ was the luminance threshold, which usually took a value of 0.04–0.12 [28]. In this paper, we also set $\alpha$ to be 0.1 (as proposed by previous methods [17,29]). When the average brightness of the pixel at $\left(x,y\right)$ was less than 0.1, the pixel belonged to the dark area, when the average brightness of the pixel at $\left(x,y\right)$ was greater than 0.9, the pixel belonged to the bright area, otherwise, it belonged to the normal exposure area. For the convenience of calculation, here, the bright and dark regions in the image were denoted as $I{N}_i\left(x,y\right)$ together.

For the normal exposure area in the image, the Scharr operator was used to extract the texture detail information, and the contrast of each pixel was calculated as follows:

$$G_x=\left(\begin{array}{ccc}-3& 0& 3\\ -10& 0& 10\\ -3& 0& 3\end{array}\right)\ast N_i\left(x,y\right)$$

(3)

$$G_y=\left(\begin{array}{ccc}-3& -10& -3\\ 0& 0& 0\\ 3& 10& 3\end{array}\right)\ast N_i\left(x,y\right)$$

(4)

$G_x$ and $G_y$ represented the texture changes in the x-axis and y-axis directions of the image, respectively, and the symbol * was the convolution operator. Then, we used the above convolution calculation results to calculate the texture change weight in the normal area, the calculation was as follows:

$$W_{I{N}_i}^C\left(x,y\right)=\sqrt{G_x{}^2+G_y{}^2}$$

(5)

To preserve the details of the bright and dark regions in the input image sequence, we used the multi-scale block LBP operator to extract the texture details of the region. Because this operator had rotation invariance and grayscale invariance, and had strong robustness to illumination, it could better extract the texture detail information of this area, and its calculation was as follows:

$$S_i\left(x,y\right)=MBLBP\left(I{N}_i\left(x,y\right)\right)$$

(6)

$MBPLBP(·)$ was an operator for extracting local texture features of images. For more details on LBPs, please refer to [25,26,27]. $S_i\left(x,y\right)$ was used as the encoded value of the pixel at $\left(x,y\right)$, that is, the LBP eigenvalue, which could reflect the texture information of the central pixel $\left(x,y\right)$ and its neighborhood.

Then, the fast local Laplacian filter [30] was used to enhance the detail information in $S_i\left(x,y\right)$, while retaining the information of the edge parts. The texture change weights in the bright and dark areas were as follows:

$$W_{N_i}^C\left(x,y\right)=L{F}_{\mu 1,\mu 2}\left(S_i\left(x,y\right)\right)$$

(7)

$L{F}_{\mu 1,\mu 2}(·)$ stood for the fast local Laplacian filtering operation, and $\mu 1$ and $\mu 2$ were the parameters of this filter. $\mu 1$ represented the magnitude of the edge in the image: the intensity percentage less than $\mu 1$ was considered as detail, the percentage change greater than $\mu 1$ was considered as edge and was preserved, and when $\mu 2$ was less than 1 it meant enhancement of detail.

Finally, by combining the above two texture change weights, the final contrast weight map was obtained, as follows:

$$W_i^C\left(x,y\right)=W_{N_i}^C\left(x,y\right)+W_{I{N}_i}^C\left(x,y\right)$$

(8)

2.2. Brightness

If the image was underexposed/overexposed, the image was very dark/bright, and the brightness of the image was 0 (or close to 0)/1 (or close to 1), which caused the image to lose a lot of information. Therefore, the basic idea of constructing the brightness weight in this paper was to set a small brightness weight for the overexposed and underexposed areas in the image and assign a higher brightness weight to the pixels with a brightness close to 0.5 [21]. The formula for the brightness weight in this paper was as follows:

$$W_i^B\left(x,y\right)=R_l\times G_l\times B_l$$

(9)

$R_l$, $G_l$, and $B_l$ represented the Gaussian curve of the red, green, and blue channels [21], respectively, calculated as follows:

$$X_l=exp\left(\frac{{\left(X-0.5\right)}^2}{{\delta }^2}\right),X\in \left\{R,G,B\right\}$$

(10)

To simplify the formula, the variable $X$ was used to represent the red, green, and blue channels (for example, when the value of $X$ was $R$, the above formula represented the Gaussian curve of the red channel).

However, some pixels in the image were possibly inherently bright and dark rather than being too bright or too dark due to over or underexposure. In this case, we needed to choose appropriate weights for the pixels in the bright and dark areas to better preserve the characteristics of the source image. This paper developed a novel and flexible bell curve function, and innovatively proposed a luminance adaptive function to adjust the weight of pixels in light and dark areas. The improved method was calculated as follows:

$$X_l=\frac{\gamma }{\left({\left(X-\eta \left(rl,X\right)\right)}^2+\gamma \right)}exp\left(\frac{{\left(X-\eta \left(x\_l,X\right)\right)}^2}{{\delta }^2}\right),X\in \left\{R,G,B\right\},x\_l\in \left\{r\_l,g\_l,b\_l\right\}$$

(11)

The usage of variables $X$ and $x\_l$ here was similar to Formula (10) (for example, when $X$ took the value R, $x\_l$ was $r\_l$, and the formula represented the brightness weight curve of the red channel), where $\delta$ was 0.2 [12] and $\gamma$ was 1. $\eta \left(r\_l,R\right)$, $\eta \left(g\_l,G\right)$, and $\eta \left(b\_l,B\right)$ were adaptive functions, and $r\_l$, $g\_l$, and $b\_l$ were the average luminance values of the pixels in the red, green, and blue channels at the position $\left(x,y\right)$ in the source image, respectively. The calculation method used Formula (1). According to the average brightness of the pixel in each channel, according to the previous method, the bright, dark, and normal areas in each channel were divided, and the specific calculation method used Formula (2).

Take the red channel $\eta \left(r\_l,R\right)$ as an example: $B_i^r\left(x,y\right)$ was the bright area in the channel, $D_i^r\left(x,y\right)$ was the dark area in the channel, $N_i^r\left(x,y\right)$ was the normal area of exposure, and the adaptive function was:

$$\eta =\left\{\begin{array}{c}0.5-\lambda \times \log \left(1+\frac{1}{K}{{\displaystyle \sum }}_{i=1}^K{l}_{r,i}\left(x,y\right)\right),\left(x,y\right)\in B_i^r\left(x,y\right)\hfill \\ 0.5,\left(x,y\right)\in N_i^r\left(x,y\right)\\ 0.5+\lambda \times \log \left(1+\frac{1}{K}{{\displaystyle \sum }}_{i=1}^K{l}_{r,i}\left(x,y\right)\right),\left(x,y\right)\in D_i^r\left(x,y\right)\hfill \end{array}\right.$$

(12)

$l_{r,i}\left(x,y\right)$ represented the luminance value of the ith image pixel $\left(x,y\right)$ in the red channel in the image sequence. The adaptive functions $\eta \left(g\_l,G\right)$ and $\eta \left(b\_l,B\right)$ in the green and blue channels were calculated in the same way as in the red channel.

2.3. Spatial Consistency

When the image sequence contained moving objects (such as moving people), the weight map also needed to consider the effect of moving objects, otherwise, ghosting could appear in the resulting image. To solve this problem, this paper proposed a new method for constructing spatially consistent weight terms based on the multi-scale block LBP (MDLBP). Figure 2 shows a flow diagram of the motion detection based on the multi-scale LBP.

First, we computed the LBP feature for each pixel in the source image:

$$T_i\left(x,y\right)=MBLBP\left(I_i^g\left(x,y\right)\right)$$

(13)

$I_i^g\left(x,y\right)$ was a grayscale image. For any two different images $I_i\left(x,y\right)$ and $I_j\left(x,y\right)$ $\left(i\ne j\right)$ in the image sequence, the local similarity between them was measured by calculating the Euclidean distance between $T_i\left(x,y\right)$ and $T_j\left(x,y\right)$ at the pixel $\left(x,y\right)$. The calculation was as follows:

$$d_{i,j}{\left(x,y\right)}^2=||T_i\left(x,y\right)-T_j{\left(x,y\right)||}_2^2$$

(14)

where ${\left||·|\right|}_2$ was the 2-norm operation that computed the matrix.

Then, the spatial consistency weight term of the image in the moving scene was constructed in the following way, and the formula was as follows:

$$W_i^{S1}\left(x,y\right)={\displaystyle \sum }_{i,j=1,i\ne j}^{K}exp\left(-\frac{d_{i,j}{\left(x,y\right)}^2}{2{\delta }_d^2}\right)$$

(15)

Among them, $W_i^{S1}\left(x,y\right)$ represented the initial spatial consistency weight calculated using Formula (15) (S1 was just a symbol used to distinguish the spatial consistency weight from the texture weight and brightness weight mentioned above and had no other practical significance). The standard deviation ${\delta }_d$ controlled the influence of the local similarity $d_{i,j}\left(x,y\right)$ on the weight $W_i^{S1}\left(x,y\right)$, and ${\delta }_d$ was also set to 0.05 (the value of 0.05, which has been used in another multi-exposure method [29], and the rationality of this value has been proved by previous work) in the method. The design idea here, is that if pixel $\left(x,y\right)$ belonged to the motion region in image $I_i$, the local similarity $d_{i,j}\left(x,y\right)$ between the image $I_i$ and all $I_j$ ($i\ne j$) at $\left(x,y\right)$ was increased. The spatial consistency weight $W_i^{S1}\left(x,y\right)$ at the pixel point was decreased, resulting in a decrease in the weight value of the image $I_i$ at the pixel point $\left(x,y\right)$.

When the moving frequency of moving objects in the image sequence was high, to remove the influence of ghosting artifacts more effectively, the method of “split-channel” was adopted. We calculated the LBP eigenvalues ($T^r{}_i\left(x,y\right)$, $T^g{}_i\left(x,y\right)$, and $T^b{}_i\left(x,y\right)$) of the red, green, and blue channels at the pixel point $\left(x,y\right)$ of the source image, respectively, and then calculated the $d^r{}_{i,j}{\left(x,y\right)}^2$, $d^g{}_{i,j}{\left(x,y\right)}^2$, and $d^b{}_{i,j}{\left(x,y\right)}^2$ of the pixel $\left(x,y\right)$ in each channel, respectively. The local similarity between the ith and jth images was calculated as follows:

$$D_{i,j}\left(x,y\right)=d^r{}_{i,j}{\left(x,y\right)}^2\times d^g{}_{i,j}{\left(x,y\right)}^2\times d^b{}_{i,j}{\left(x,y\right)}^2$$

(16)

The spatial consistency weight term of the image constructed by the method proposed in this paper was calculated as follows:

$$W_i^{S2}\left(x,y\right)={{\displaystyle \sum }}_{i,j=1,i\ne j}^{K}exp\left(-\frac{D_{i,j}\left(x,y\right)}{2{\delta }_d^2}\right)$$

(17)

where $W_i^{S2}\left(x,y\right)$ was the initialized spatial consistency weight calculated using Equation (17). Formula (17) was improved from Formula (15), and Formula (17) had only one more step than Formula (15), that was, the calculation of $D_{i,j}\left(x,y\right)$. When the motion frequency of moving objects in the source image sequence was relatively high, the effect of removing ghost images using Equation (17) was better than using Equation (15). The effect of using Formulas (15) and (17) was the same in other cases, but the complexity of Formula (17) was higher than that of Formula (15). For simplicity, the following discussion of spatial consistency weights was based on Formula (17) unless otherwise stated (including the flow charts of spatial consistency weights in Figure 1 and Figure 2).

Finally, the weight map was refined by the morphological operator to remove the influence of noise, and the final spatial consistency weight $W_i^S\left(x,y\right)$ was obtained (among them, S in $W_i^S\left(x,y\right)$ and S2 in $W_i^{S2}\left(x,y\right)$ were similar to the above-mentioned S1. They were symbols used to distinguish them from other weights and had no other practical significance).

Figure 3 shows the weight map for each quality measure of the method, and Figure 3a shows the source image sequence “Arches”. Figure 3b shows a local contrast weight map, a texture detector that assigned higher weights to pixels with more detail. Figure 3c shows a luminance weight map that assigned higher weights to pixels with an average luminance value between 0.1 and 0.9. Pixels in light and dark areas used a brightness adaptation function to adjust their weights. The spatial consistency of Figure 3d detected moving objects by computing the texture differences between different input images in the source image sequence to remove the ghosting artifacts that affected the fused images.

2.4. Weight Map Estimation

According to the previous calculation, three image features (texture change weight, brightness feature, and spatial consistency) were obtained. In this step, these weight terms needed to be combined to obtain the initial weight map. For the proposed method to extract the highest quality regions from the weight map, this paper used pixel multiplication to combine different weight maps, calculated as follows:

$$W_i^{intial}\left(x,y\right)=\{\begin{array}{c}{W}_i^C\left(x,y\right)\times W_i^B\left(x,y\right),forstaticscene,\\ W_i^C\left(x,y\right)\times W_i^B\left(x,y\right)\times W_i^S\left(x,y\right),fordynamicscene,\end{array}$$

(18)

Among them, $W_i^C\left(x,y\right)$, $W_i^B\left(x,y\right)$, and $W_i^S\left(x,y\right)$ represented the texture change weight, brightness weight, and spatial consistency weight of image $i$, respectively, and $W_i^{intial}\left(x,y\right)$ was the initial weight map of this method.

In a static scene, the initial weight map did not need to consider the spatial consistency weight, which was only used for dynamic scenes with moving objects, to eliminate ghosting artifacts in the fused images caused by moving objects. After generating the initial weight map, to ensure that the sum of the weights of each pixel $\left(x,y\right)$ was 1, we also needed to normalize the weight map, calculated as follows:

$$W_i^I\left(x,y\right)=\frac{W_i^{initial}\left(x,y\right)+\epsilon }{{{\displaystyle \sum }}_{i=1}^K\left(W_i^{initial}\left(x,y\right)+\epsilon \right)}$$

(19)

ε was a small positive number (for more details about ε, please refer to [21,29]).

2.5. Weight Map Refinement

Since the initial weight map was usually affected by noise, we needed to denoise it with a filter before fusing the weight map. When we used the smoothing filter to refine the weight map, we needed to keep the sharpness of the image edge texture while removing the influence of noise, so the edge preservation smoothing filter was selected. Many existing filters could be used to optimize the weight map, such as recursive filters [31], bilateral filters [32], etc. Due to the low complexity of the fast-guided filter [33], it was used in this method to refine the initial weight map, which was calculated as follows:

$$W_i^F\left(x,y\right)=FG{F}_{r,ep,ϵ}\left(W_i^I\left(x,y\right),W_i^I\left(x,y\right)\right)$$

(20)

$W_i^F\left(x,y\right)$ represented the refined weight map and $FG{F}_{r,ep,ϵ}\left(I,W\right)$ represented the fast-guided filtering operation. $r$, $ep$, and $ϵ$ were the parameters of the filter, where $r$ was the window radius of the filter and $ep$ was the regularization parameter of the filter which controlled the smoothness of the filter. $ϵ$ was the subsampling rate and $I$ and $W$ represented the guide image and the image to be filtered, respectively. In this method, the weight map $W_i^I\left(x,y\right)$ was used as both the guide image and the input image. Finally, the weight map was normalized to obtain the final weight map $W_i\left(x,y\right)$.

2.6. Image Fusion

If the image was fused at a single resolution (that is, the weight map was directly fused with the image), due to the different exposure of each input image, they had different absolute intensities, which led to seam problems in the fused image. To solve this problem, a pyramid-based multi-resolution method [12] was used to fuse images in this paper. In this method, the source image was decomposed into a Laplacian pyramid, the above weight map was decomposed into a Gaussian pyramid, and then the Laplacian pyramid and the Gaussian pyramid were fused at each level, and the calculation was as follows:

$$L{\left\{F\right\}}^l={{\displaystyle \sum }}_{i=1}^{K}G{\left\{W_i\left(x,y\right)\right\}}^{l}L{\left\{I_i\left(x,y\right)\right\}}^l$$

(21)

$G{\left\{W_i\left(x,y\right)\right\}}^l$ meant decomposing the weight map into a Gaussian pyramid, $L{\left\{I_i\left(x,y\right)\right\}}^l$ meant decomposing the input image into a Laplacian pyramid, $L{\left\{F\right\}}^l$ was the new Laplacian pyramid after fusion, and l meant the number of layers of the pyramid. Finally, we reconstructed $L{\left\{F\right\}}^l$ to obtain the final image. Since the multi-resolution method was used to fuse images, it was the image features that were fused, not the image intensities. Therefore, when using this method to fuse images, there was no seam problem in the image fusion process.

3. Experimental Analysis

In this section, to verify the effectiveness of this method, 14 sets of multi-exposure image sequences (including static scenes and dynamic scenes) shot in different scenes were collected for experiments. Details about each set of image sequences are described in

Table 1. Information on image sequences relevant to this article.

Image Sequence	Size	Image Origin
Arno	339 × 512 × 3	Ma K [34]
Arch	669 × 1024 × 5	Ma K [34]
Belgium House	384 × 512 × 9	Ma K [34]
Cave	384 × 512 × 4	Ma K [34]
Chinese Garden	340 × 512 × 3	Ma K [34]
Candle	364 × 512 × 10	Ma K [34]
Forrest	683 × 1024 × 5	Ma K [34]
Garden	754 × 1024 × 5	Ma K [34]
House	340 × 512 × 4	Ma K [34]
Kluki	341 × 512 × 3	Ma K [34]
Laurenziana	356 × 512 × 3	Ma K [34]
Lighthouse	340 × 512 × 3	HDRsoft [35]
Office	340 × 512 × 6	Ma K [34]
Studio	341 × 512 × 5	HDRsoft [35]

.

This paper compared the method with eight existing techniques [12,16,18,19,20,21,22,36], and selected five representative multi-exposure image sequences from them for subjective evaluation, as shown in Figure 4. Among them, three groups (Figure 4a–c) were multi-exposure image sequences of static scenes, and the other two groups (Figure 4d,e) were multi-exposure image sequences of dynamic scenes. Figure 5, Figure 6, Figure 7, Figure 8 and Figure 9 show subjective evaluations of our method.

3.1. Subjective Evaluation

To demonstrate the effectiveness of the method proposed in this paper, we conduct a detailed evaluation of the method in static and dynamic scenarios, respectively, in this section.

3.1.1. Static Scene

Figure 5 shows the processing results of different methods on the “Lighthouse” image sequence. To more clearly compare the processing results of the image sequence “Lighthouse” by different methods, we used the form of a list to compare the main features (well-defined edges, correct brightness or color, and exposure) of the fused images. A detailed comparison of Figure 5 is shown in

Table 2. Detailed comparison of Figure 5.

Method	Well-Defined Edges	Correct Brightness or Color	Well-Exposed
Mertens et al. [12]	No	Yes	Yes
Gu et al. [16]	No	No	No
Shen et al. [18]	No	No	Yes
Bruce et al. [19]	No	Yes	No
Photomatix [36]	No	No	Yes
Hayat et al. [21]	No	Yes	Yes
Huang et al. [22]	Yes	No	No
Proposed method	Yes	Yes	Yes

below.

Figure 6 shows the processing results of the “House” image sequence, which captured an indoor room with an outdoor garden outside the window. A detailed comparison of Figure 6 is shown in

Table 3. Detailed comparison of Figure 6.

Method	Well-Defined Edges	Correct Brightness or Color	Well-Exposed
Mertens et al. [12]	No	Yes	Yes
Gu et al. [16]	Yes	No	No
Shen et al. [18]	Yes	No	Yes
Bruce et al. [19]	No	No	No
Photomatix [36]	No	No	No
Hayat et al. [21]	No	Yes	Yes
Huang et al. [22]	No	Yes	Yes
Proposed method	Yes	Yes	Yes

below.

Figure 7a–h show the processing results of the previous multi-exposure fusion methods and the method proposed in this paper on the image sequence “Studio”. A detailed comparison of Figure 7 is shown in

Table 4. Detailed comparison of Figure 7.

Method	Well-Defined Edges	Correct Brightness or Color	Well-Exposed
Mertens et al. [12]	No	Yes	Yes
Gu et al. [16]	No	No	No
Shen et al. [18]	No	No	No
Bruce et al. [19]	No	No	No
Photomatix [36]	No	No	No
Hayat et al. [21]	No	Yes	Yes
Huang et al. [22]	No	Yes	Yes
Proposed method	Yes	Yes	Yes

below.

3.1.2. Dynamic Scene

This subsection presents experimental results of processing image sequences captured in dynamic scenes, discusses related results, and shows close-ups of some of the results for better comparison.

The LBP-based fusion method proposed in this paper had a variable parameter: the scale factor. This parameter was used to control the size of each pixel neighborhood of the LBP, and this parameter was set to 15. When constructing the spatial consistency weight item, to distinguish the two methods of constructing the spatial consistency weight, we denoted the method using Formula (15) as MBLBP-1, and the method using Formula (17) as MBLBP-2.

The processing results of the “Candle” image sequence by different methods are shown in Figure 8a–h. To more clearly compare the processing results of the image sequence “Candle” by different methods, we used the form of a list to compare the main features (well-defined edges, correct brightness or color, and effects of ghosting artifacts) of the fused images. A detailed comparison of Figure 8 is shown in

Table 5. Detailed comparison of Figure 8.

Method	Well-Defined Edges	Correct Brightness or Color	Whether to Eliminate Ghosting
Mertens et al. [12]	Yes	Yes	No
Shen et al. [18]	No	No	No
Li et al. [20]	Yes	No	No
Photomatix [36]	No	No	No
Hayat et al. [21]	No	No	Yes
Huang et al. [22]	Yes	Yes	No
MBLBP-1	Yes	Yes	Yes
MBLBP-2	Yes	Yes	Yes

below.

Figure 9a–h shows the processing results of the previous multi-exposure fusion methods and the method proposed in this paper on the image sequence “Garden”. When the moving frequency of moving objects in an image sequence was high, it was difficult to completely remove ghosting artifacts because there were no moving objects in some areas of a small part of the image. Figure 9a–d,f shows Mertens [12], Shen [18], Li and Wei [20] et al.’s method, software Photomatix [36] processed images, and Huang et al.’s [22] method results, respectively. It could be seen from the local enlarged area in the figure that these fused images all had very obvious ghosting artifacts. In Figure 9e, although the method of Hayat et al. [21] partially removed the ghosting effect in the image, obvious shadows could still be observed in the enlarged area of the image. The results of our method are shown in Figure 9g,h. As could be seen from the figure, our method was better than the previous methods in removing ghosting artifacts. From the comparison of Figure 9g,h, it could be seen that when the moving objects in the image sequence moved frequently, the effect of using the MBLBP-2 method as the spatial consistency weight was better than that of the MBLBP-1 method.

3.2. Objective Evaluation

In this subsection, we objectively evaluate and compare the proposed method and seven existing techniques using two advanced quality metrics, $Q^{AB/F}$ [37] and the structural similarity index MEF-SSIM [38], respectively.

Table 6. The test result of indicator $Q^{AB/F}$.

Dataset	Mertens [12]	Gu [16]	Shen [18]	Bruce [19]	Li [20]	Hayat [21]	Huang [22]	Proposed
Arno	0.687	0.336	0.486	0.617	0.680	0.687	0.629	0.693
Arch	0.692	0.527	0.460	0.623	0.652	0.683	0.697	0.716
Belgium House	0.731	0.609	0.572	0.322	0.724	0.731	0.735	0.758
Cave	0.809	0.743	0.592	0.722	0.804	0.809	0.719	0.854
Chinese Garden	0.774	0.619	0.499	0.715	0.882	0.499	0.825	0.796
Candle	0.624	0.603	0.551	0.586	0.610	0.583	0.625	0.642
Forrest	0.392	0.533	0.346	0.671	0.393	0.616	0.574	0.698
Garden	0.566	0.512	0.428	0.567	0.677	0.622	0.672	0.665
House	0.674	0.566	0.447	0.564	0.651	0.674	0.707	0.707
Kluki	0.745	0.631	0.524	0.682	0.722	0.745	0.759	0.763
Laurenziana	0.746	0.365	0.538	0.727	0.745	0.746	0.814	0.767
Lighthouse	0.708	0.634	0.515	0.647	0.706	0.708	0.702	0.712
Office	0.671	0.529	0.513	0.550	0.691	0.671	0.675	0.677
Studio	0.710	0.395	0.478	0.521	0.662	0.710	0.741	0.748

Note: Bold data in the table represent optimal data.

and

Table 7. The test result of indicator MEF-SSIM.

Dataset	Mertens [12]	Gu [16]	Shen [18]	Bruce [19]	Li [20]	Hayat [21]	Huang [22]	Proposed
Arno	0.903	0.890	0.846	0.946	0.961	0.903	0.949	0.973
Arch	0.842	0.901	0.723	0.848	0.817	0.941	0.869	0.938
Belgium House	0.893	0.897	0.709	0.859	0.778	0.893	0.918	0.902
Cave	0.961	0.894	0.788	0.944	0.950	0.908	0.935	0.956
Chinese Garden	0.950	0.876	0.767	0.859	0.975	0.932	0.920	0.989
Candle	0.895	0.912	0.859	0.901	0.921	0.891	0.897	0.928
Forrest	0.791	0.877	0.744	0.898	0.821	0.901	0.894	0.930
Garden	0.835	0.855	0.809	0.817	0.865	0.867	0.827	0.867
House	0.915	0.878	0.717	0.863	0.903	0.898	0.919	0.937
Kluki	0.929	0.911	0.815	0.876	0.924	0.894	0.931	0.936
Laurenziana	0.985	0.755	0.881	0.963	0.991	0.982	0.990	0.991
Lighthouse	0.863	0.838	0.711	0.891	0.866	0.939	0.857	0.893
Office	0.977	0.916	0.756	0.954	0.976	0.973	0.979	0.983
Studio	0.920	0.774	0.880	0.911	0.883	0.920	0.911	0.928

Note: Bold data in the table represent optimal data.

give the $Q^{AB/F}$ [37] and MEF-SSIM [38] values of Mertens et al. [12], Gu et al. [16], Shen et al. [18], Bruce et al. [19], Li and Wang et al. [20], Hayat et al. [21], Huang et al. [22], and the proposed method (MBLBP-2), respectively. The higher the value of MEF-SSIM and $Q^{AB/F}$, the better the image quality. From Table 6 and Table 7, it could be concluded that in most cases, compared with the existing algorithms, the method in this paper had better results.

3.2.1. Evaluation Using $Q^{AB/F}$

The image quality metric $Q^{AB/F}$ is a novel objective quality assessment metric for fused images, which reflects the quality of the information retained after the fusion of source image sequences. $Q^{AB/F}$ represented the degree of preservation of image edge detail information, calculated as follows:

$$Q^{AB/F}=\frac{{{\displaystyle \sum }}_{x=1}^{H1}{{\displaystyle \sum }}_{y=1}^{H2}\left(Q^{AF}\left(x,y\right){\omega }^A\left(x,y\right)+Q^{BF}\left(x,y\right){\omega }^B\left(x,y\right)\right)}{{{\displaystyle \sum }}_{x=1}^{H1}{{\displaystyle \sum }}_{y=1}^{H2}({\omega }^A\left(x,y\right)+{\omega }^B\left(x,y\right))}$$

(22)

Among them, $Q^{AF}\left(x,y\right)=Q_g^{AF}\left(x,y\right)Q_0^{AF}\left(x,y\right)$, $Q_g^{AF}\left(x,y\right)$ and $Q_0^{AF}\left(x,y\right)$ were the edge strength and direction retention values at the pixel point $\left(x,y\right)$, respectively, and ${\omega }^A\left(x,y\right)$ and ${\omega }^B\left(x,y\right)$ represented the weight coefficient of each edge, respectively. The higher the $Q^{AB/F}$ value, the more the edge detail information in the input image sequence was preserved by the fused image.

In this paper, seven different algorithms were used to compare the $Q^{AB/F}$ value of the method, and the details are shown in Table 6. It could be observed from Table 6 that our method preserved image details well in most cases. The $Q^{AB/F}$ value of this method was lower than other methods when dealing with the image sequences “Chinese Garden”, “Garden”, “Kluki” and “Laurenziana”. When refining the weight map in this paper, to prevent the contour details of the image from being blurred as much as possible, the weight map was used as the guide image of the filter, which might have partially affected the $Q^{AB/F}$ of the fused image.

3.2.2. Evaluation Using MEF-SSIM

The MEF-SSIM is one of the objective quality indicators widely used to evaluate fused images. This indicator could detect the overall similarity between the resulting image of this method and the corresponding source image sequence. The maximum value of MEF-SSIM was one and the minimum value was zero. The higher the value, the higher the similarity between the resulting image and the source image sequence, and the better the image quality. The specific calculation method of this indicator was as follows:

$$Q\left(Y\right)={{\displaystyle \prod }}_{m=1}^M{\left[Q_m\left(Y\right)\right]}^{{\beta }_i}$$

(23)

where $Q\left(Y\right)$ was the final MEF-SSIM value, M was the maximum number of scales that the image could be decomposed, $Q_m\left(Y\right)$ was the average image quality when the image scale was m, and ${\beta }_i$ was the corresponding weight.

From Table 7, we could see that the proposed method could achieve the maximum MEF-SSIM value in most cases. When dealing with the image sequences “Arch”, “Belgian House”, “Cave”, and “Lighthouse”, the MEF-SSIM value of this method was lower relative to other methods. This was because when the algorithm extracted the texture details of the overexposed and underexposed areas in the image, to better preserve the detail information of the area. This method enhanced the detail information of the area to a certain extent. Therefore, when the proportion of brighter and darker regions in the source image sequence was relatively large, it had a certain impact on the MEF-SSIM of its fused image.

3.2.3. Analysis of Free Parameter

Free parameters were set to 15 for all experiments as previously described. We used evaluation metrics (MEF-SSIM and $Q^{AB/F}$) to examine the influence of this parameter on the experimental results of this paper. We set the free parameters to 12, 15, 18, 21, 24, and 28, and for each free parameter, tested the MEF-SSIM and $Q^{AB/F}$ values of the method (MBLBP-2) on the image sequence, respectively. Then, under each of the above free parameters, the average values of these two quality indicators of the image sequence were calculated separately. Figure 10 shows the trend diagram of quality indicators MEF-SSIM and $Q^{AB/F}$ when our method took different scale factors, and Figure 10a,b demonstrates that when this parameter was set to 15, the proposed method achieved the highest MEF-SSIM and $Q^{AB/F}$ values.

3.2.4. Comparing the Performance of the LBP and the Multi-Scale LBP

In this section, our method applies the LBP to a fixed-size neighborhood and compares its results with those of the multi-scale block LBP-based method proposed in this paper, as shown in Figure 11.

From Figure 11, it could be concluded that when the method applied the LBP operator with a fixed neighborhood size, the evaluation indicators MEF-SSIM and $Q^{AB/F}$ of the obtained image sequence processing results were generally lower than the processing results of the method proposed in this paper. The method based on the multi-scale block LBP in this paper exhibited great advantages.

4. Conclusions and Future Work

In this paper, a novel ghost-free multi-exposure image fusion method was proposed. In this model, we used a novel multi-scale LBP operator-based method to extract local texture features of bright and dark regions from source images. If the captured image contained moving objects, we adopted the method based on the MDLBP operator to remove ghosting artifacts in fused images. Through quantitative and qualitative analysis of this method, the experimental results showed that this method was superior to many representative image fusion methods in both visual quality and objective analysis. This method could obtain images with good contrast, color and brightness, and could remove ghosting artifacts in fused images. In addition, this paper also studied two spatially consistent weight distribution methods and compared and discussed the effects of these two methods on the results of dynamic image fusion.

The methods proposed in this paper followed the weighted sum-based framework for multi-exposure image fusion without ghosting, but these methods still suffer from some common limitations. When processing an image sequence containing moving objects, if moving objects appear in most positions of the image sequence, and the same moving object appears in different positions in different images, the effect of this method in eliminating ghosting artifacts may not be good enough. For example, in the image sequence “Garden”, there were many moving people in the scene. Since a certain pixel position in most of the source images was covered by different moving objects or the same moving objects, it was difficult to eliminate all the ghosting in this scenario. To solve this limitation, it is one of our future tasks to improve the multi-exposure image fusion method and develop a method that can eliminate ghosting more effectively.

In the future, I intend to further utilize the advantages of the multi-scale LBP operator to improve the ghost removal method in this paper to address the limitations mentioned above.