1. Introduction
A single image cannot fully display all the information of the target scene. For example, computed tomography (CT) images mainly shows hard bone tissues of the body, while magnetic resonance imaging (MRI) mainly shows soft tissues. Multifocus images are formed by focusing on different objects because the imaging equipment cannot achieve focused imaging of all the objects in the same scene. Infrared images can capture thermal radiation information, but the resolution is poor, while visible images have higher resolution. Therefore, it is essential to perform image fusion. The current research of image fusion is mainly applied in the fields of medical science [
1], remote sensing [
2], monitoring [
3], etc.
The current image fusion algorithms are mainly based on the principal component analysis method [
4], the pyramid transform-based method [
5], the wavelet transform-based method [
6], etc. The article [
7] elaborates the theoretical knowledge behind the different fusion algorithms. The method based on principal component analysis is easy-to-understand research. However, it is computationally intensive and has poor real-time performance. The methods based on pyramid transform mainly include the Laplacian pyramid [
8], gradient pyramid [
9], etc. However, the pyramid decomposition is redundant and lacks direction selectivity, which cannot effectively present the structural information of the fused image and leads to blurred boundaries of the fused image. The main methods based on wavelet transform are dual-tree complex wavelet transform [
10], contourlet transform [
11], shearlet transform [
12], etc., but the relationship between the decomposition layers, fusion quality, and time efficiency need to be balanced. The article [
13] describes the latest research progress in image fusion.
Because the variational method has a mature theoretical system and it is easy to design and analyze [
14], it is widely used in image recovery [
15], image denoising [
16], and image fusion [
17]. The authors in the literature [
15] proposed a variational model for local non-texture inpainting based on the total variational model and the Mumford–Shah model. The authors in the literature [
16] proposed a fractional-order total variational model for image denoising. In the literature [
17], a three-step method for fusing clear images is proposed, as shown in
Figure 1. In the image decomposition step, the images are processed using a variational model based on the data-driven tight frame (DDTF):
where
are the parameters corresponding to the two balances.
are the clear source images, and
are the sparse representation coefficients of
, respectively.
is the analysis operator associated with the tight frame generated by
two-dimensional filters
.
I is the unit matrix. Unfortunately, clear images can be handled by this method; images containing noise do not work.
In addition, noise is also an important factor to be considered in the image fusion process. In the process of actual image generation, noise interference is inevitable. For example, Poisson noise is often generated when using photon counting devices, which mainly depends on the number of photons. The common noises mainly include Gaussian noise, salt and pepper noise, Poisson noise, etc. The first two kinds of noise have been studied by a large number of researchers [
18,
19,
20,
21]. Image denoising is performed using artificial neural networks in article [
22], and a new hybrid filter technique is proposed by combining anisotropic diffusion with a Butterworth band-pass filter to overcome over-filtering of the image in article [
23] for image denoising. Hence, in this paper, we focus on the image disturbed by Poisson noise. The Tikhonov regularization model [
22], the total variational model (TV) [
23], and the higher-order total variational model [
24] are common methods that use variational models for removing Poisson noise. The total variational-based denoising model is as follows:
where
is a positive parameter,
f is the source image, and
u is the denoised image. This model performs well on piecewise constant images, but it causes staircasing artifacts for piecewise smooth images. High-order total variational models such as the PDE-based model [
25] and the Lysaker–Lundervold–Tai (LLT) model [
26] introduce speckle artifacts.
Different from other types of models, the fractional-order total variational (FOTV) model has shown that it can suppress staircasing artifacts and speckle artifacts [
27]. By considering the intensity of adjacent images, their local geometric features can be maintained [
28,
29]. The literature [
16] proposes a Poisson denoising model based on fractional-order total variation, which can maintain the high-order smoothness of the image. The model is as follows:
where
is the fraction that is greater than or equal to 1,
f is the noise image,
u is the denoised image, and
is the parameter of the fidelity term.
Before the fusion of images containing noise, the images are usually pre-processed with noise removal; however, this operation also reduces the efficiency of image fusion. Consider that each term in the variational model (
1) for image fusion and the variational model (
3) for image denoising is independent and indispensable to each other. The addition of an item to the variational model is feasible and easy to interpret. Many methods have been proposed to simultaneously denoise and fuse images disturbed by Gaussian noise [
30,
31,
32], and these methods have achieved excellent results. Inspired by them, we want to fuse images disturbed by Poisson noise using the variational model.
In this paper, a new three-step method for image denoising and fusion is proposed by combining the two variational models (
1) and (
3), and its work flowchart is shown in
Figure 2. Firstly, in the image decomposition step, an improved variational model is proposed to process the images with noise, and the split Bregman method is used to solve it. Secondly, the coefficients are constructed according to the fusion rules. Finally, the fused image is obtained using the variational model in the image reconstruction step, which maintains the image smoothness and significant features.
Our contributions can be summarized as follows:
Motivated by a fractional-order total variational denoising model and a data-driven tight frame variational model for image fusion, a variational fusion model capable of handling noisy images is constructed. The denoised images and analysis operator are obtained by this model.
The new three-step method is constructed. The method combines FOTV and DDTF models for simultaneously denoising and fusing images, and it can find the complementary information from the noisy source images to obtain the final fused images and suppress the noise output. This is the first time that a fractional-order variational model is used to denoise and fuse images disturbed by Poisson noise.
We evaluate this method on different types of images. The experiments show that the proposed method is more effective.
The rest of this paper is organized as follows. In
Section 2, a new three-step method is proposed. In
Section 3, the solving procedure using the split Bregman algorithm is described in detail. In
Section 4, by numerical experiments, the advantages of the proposed method are illustrated. This paper concludes with a brief summary in
Section 5.
4. Numerical Experiments
In this section, to verify the effectiveness of the method, numerical experiments of the proposed three-step method are performed. All experimental results were achieved using Matlab (R2016b) on a laptop with Intel(R) Core(TM) i5-1035G1 CPU @ 1.00 GHz 1.19 GHz, 16 GB RAM, Windows 11. Different types of images are selected, including synthesized images [
17], medical images [
32], multifocused images [
38], and infrared and visible images [
39], all of which are widely used in image processing.
4.1. Evaluation Metrics
It is important to choose the metrics to evaluate the quality of the denoised image and the fused image. In general, the metrics of the denoising and fusion effect can be divided into two aspects: subjective visual effect and objective quantitative evaluation. The subjective visual effect is that people evaluate the image quality by visual observation, which cannot be judged very accurately. Usually, some objective evaluations are described in the literature. There are many metrics for image evaluation quality, such as an intelligent model proposed in article [
40] to evaluate noise in ultrasound images. In this paper, the peak signal-to-noise ratio (PSNR), mutual information (MI) [
41], edge strength (
) [
42], and structural similarity (
) [
43] are chosen to evaluate the image quality. The PSNR is used to measure the denoising effect of the image. MI is used to measure the amount of information of an image containing another image.
is used to determine the relative amount of edge information that is transferred from the input images into the fused image.
is used for determining the structural similarity between two images. It is characteristic that the larger these values are, the better the results.
4.2. Selection of Parameters
The selection of parameters is crucial in the implementation of the algorithm. In the image decomposition step, the stop criterion is , and in the image reconstruction step, the number of iterations is 1000. The method proposed in this paper mainly involves the following parameters. Basic guidance for setting these parameters is discussed. are the coefficients of the approximation term. The larger , the closer the approximation solution obtained is to the exact solution. The parameters are the coefficients of the regular term. The larger , the smoother the image obtained; the smaller , the denoising effect on the image is not significant. and control the penalty function terms of . and control the iterations of .
4.3. Numerical Experimental Results and Analysis
At present, most researchers have studied the simultaneous denoising and fusion of images disturbed by Gaussian noise [
36,
44,
45], and some others have studied the problem of Poisson denoising. There are few studies on the simultaneous denoising fusion of images interfered with by Poisson noise. We found a paper [
46] on the simultaneous denoising and fusion of images with Poisson noise, which proposes an online convolutional coding model to train noisy images. The fusion was performed on multifocused images with a PSNR value of 29.46. Because the authors ran the model in GPU, this paper is not compared with it.
Because Poisson noise depends on the pixel intensity, the noise level can be controlled by the peak intensity of the original image. The original image with a preset peak value is scaled, before adding Poisson noise. Specifically, we consider three peak values: 55, 155, and 255.
It is obvious from
Figure 3 that the image with peak value 55 is more noisy than the image with peak values 155 and 255.
Next, the validity of fractional-order
is verified. In
Figure 4, we present the denoising results for the image with peak 55 using different fractional orders
, 1.6, and 2.4. It is obvious that when
, i.e., TV, the first column still contains some noise. When
, the last column image is too smooth and blurs the boundary. This means that the fractional-order
has an effect on the final result.
In all the experiments, the most appropriate parameters are chosen from the set of parameters
,
, and
,
. Firstly, we process the two medical images at peaks 55 and 155 and set
= 1.6,
K = 16, and the results are shown in
Figure 6.
In
Figure 6, the proposed method is very effective for medical images. The Poisson noise in the image is effectively removed and most of the feature information is retained without edge blurring. Information is provided on objective evaluation metrics, i.e., the values of
,
,
, and
in
Table 1, which show that the proposed method is more effective in most cases. Note that both ‘Noisy’ and ‘PSNR’ in
Table 1,
Table 2,
Table 3 and
Table 4 represent the values of the PSNR for the two noisy images and the denoised images, respectively.
Then, the effect of different values of the parameters on the fusion results are discussed, as shown in
Figure 7. It is observed that the selection of
will affect the final fusion effect, and these parameters are sensitive. The parameters
will affect the time of the whole calculation process.
The denoising results and the final fusion results of the multifocus images are described in
Figure 8 and
Figure 9.
Figure 8 shows the multifocus image at peaks of 155 and 255.
Figure 9 shows the denoising results and the final fusion results of the multifocus image with peaks of 155 and 255, respectively.
It is not difficult to find that for (a,e,b,f) in
Figure 9, although some of the noise can be removed from this image, there is still some left, and the fusion results show that the tiny texture features are blurred, which is consistent with the results in
Table 2.
Medical images and multifocal images are used to compare the proposed method with the method in the literature [
17], and the results are shown below. In
Figure 10, the first row presents the fused images using the proposed method, and the second row presents the fused images using the DDTF method. By looking at these images, it is easy to see that our proposed method has better performance with respect to denoising, which is also illustrated by the values in
Table 3.
At the end of the experiment, we show the denoising and fusion results of the infrared and visible images.
Figure 11 shows the images at peaks of 155 and 255.
Figure 12 shows the denoising results and fusion results for the infrared and visible images at peaks of 155 and 255, respectively.
As can be seen from
Figure 12, the proposed method does not seem to be too effective in denoising infrared and visible light, and the fused image still contains some noise, which is consistent with the results in
Table 4.