Multi-Focus Image Fusion via PAPCNN and Fractal Dimension in NSST Domain

Abstract

Multi-focus image fusion is a popular technique for generating a full-focus image, where all objects in the scene are clear. In order to achieve a clearer and fully focused fusion effect, in this paper, the multi-focus image fusion method based on the parameter-adaptive pulse-coupled neural network and fractal dimension in the nonsubsampled shearlet transform domain was developed. The parameter-adaptive pulse coupled neural network-based fusion rule was used to merge the low-frequency sub-bands, and the fractal dimension-based fusion rule via the multi-scale morphological gradient was used to merge the high-frequency sub-bands. The inverse nonsubsampled shearlet transform was used to reconstruct the fused coefficients, and the final fused multi-focus image was generated. We conducted comprehensive evaluations of our algorithm using the public Lytro dataset. The proposed method was compared with state-of-the-art fusion algorithms, including traditional and deep-learning-based approaches. The quantitative and qualitative evaluations demonstrated that our method outperformed other fusion algorithms, as evidenced by the metrics data such as $Q_{AB/F}$, $Q_E$, $Q_{FMI}$, $Q_G$, $Q_{NCIE}$, $Q_P$, $Q_{MI}$, $Q_{NMI}$, $Q_Y$, $Q_{AG}$, $Q_{PSNR}$, and $Q_{MSE}$. These results highlight the clear advantages of our proposed technique in multi-focus image fusion, providing a significant contribution to the field.

1. Introduction

In real life, when people use cameras to shoot, they hope to obtain clear images of all the scenery in the same scene. However, the camera lens is limited by the depth of field and cannot focus on all targets simultaneously, so some areas in the captured photos are clear and some areas are blurry [1]. Multi-focus image fusion technology can fuse multiple images with different focus areas in the same scene into a fully clear image, effectively solving this problem and improving the information utilization rate of the image [2]. Accurately identifying and extracting the focus area in the source image is a challenge in multi-focus image fusion algorithms. Figure 1 shows the example of multi-focus image fusion; from the results, we can see that if the focus area is not fully extracted, it can lead to artifacts and loss of edge contours in the fusion results.

Image fusion technology has undergone rapid development in recent years, which can be divided into traditional-based and deep-learning-based algorithms. The traditional-based fusion methods include DWT [4], DTCWT [5], curvelet [6], contourlet [7], nonsubsampled contourlet transform [8], shearlet [9], and nonsubsampled shearlet transform (NSST) [10]. These wavelet-based algorithms and improved techniques for image fusion can typically be divided into three key steps, which are outlined below: image decomposition, coefficient fusion, and image reconstruction. Edge-preserving filtering algorithms have become an integral part of image fusion due to their high efficiency [11]. Their fusion structure shares similarities with wavelet-based methods, involving image decomposition into basic- and detail-layers, coefficient fusion, and eventual reconstruction. Additionally, methods based on sparse representation have also made significant strides in the field of image fusion.

Deep learning has proven to be highly successful in various image and visual tasks, mainly due to its remarkable feature learning capabilities [12]. In the field of multi-focus image fusion, deep-learning-based methods have rapidly become a prominent and popular research area since their initial proposal by Liu et al. in 2017 [13]. These methods leverage the power of convolutional neural networks (CNNs) [14] and generative adversarial networks (GANs) [15] to achieve impressive results in image fusion. The utilization of deep learning models such as CNNs and GANs in multi-focus image fusion has opened up new possibilities and achieved state-of-the-art results in the field. The ability of these models to automatically learn and extract relevant features from input images has led to significant advancements, providing researchers and practitioners with powerful tools to tackle complex image fusion challenges. As deep learning continues to evolve, it is expected to further drive the progress and innovation in multi-focus image fusion and other image-related tasks.

NSST, being a multi-scale transformation model, offers essential features such as translation invariance, making it highly effective in extracting precise edge details from images. Furthermore, its ability to correct registration errors to a certain extent plays a significant role in minimizing fusion errors. One of the remarkable advantages of NSST is that it imposes no restrictions on the number of directions after shearing and the inverse transformation does not require a synthetic direction filter. This unique property enables better sparse representation of images, ultimately reducing computational complexity and improving overall efficiency.

To achieve information complementarity and obtain crystal-clear fully focused images, we propose a novel multi-focus image fusion method based on a combination of the parameter-adaptive pulse coupled neural network and fractal dimension, all operating within the nonsubsampled shearlet transform domain. This method integrates the strengths of each component to deliver exceptional fusion results. The key contributions of our proposed method are as follows:

• The nonsubsampled shearlet transform is applied to decompose and reconstruct the image.
• The parameter-adaptive pulse coupled neural network is introduced to fuse the low-frequency bands.
• The fractal dimension-based fusion rule via multi-scale morphological gradient is used to process the high-frequency bands.
• The public Lytro dataset [16] is used to test the proposed method and the experiment also verified the effectiveness of our algorithm in multi-focus image fusion.

The rest of this paper is organized as follows. In Section 2, the related works are introduced. In Section 3, the background is described. In Section 4, the proposed multi-focus image fusion is constructed. In Section 5, the experimental results and discussions are described, and the advantages of the proposed method are verified. The conclusions with a brief summary are found in Section 6.

2. Related Works

2.1. Traditional-Based Fusion Methods

Zhang et al. [17] introduced an image fusion algorithm based on tensor decomposition integrating joint static and dynamic guidance in the nonsubsampled shearlet transform domain. Panigrahy et al. [18] introduced the image fusion using adaptive unit-linking pulse coupled neural network and distance-weighted regional energy in the nonsubsampled shearlet transform domain. Li et al. [19] introduced the sparse representation model into shearlet domain for multi-focus image fusion. Luo et al. [20] introduced the multi-state contextual hidden Markov model for image fusion via regional energy in the nonsubsampled shearlet transform domain, and this method has demonstrated superior performance compared to other state-of-the-art image fusion techniques, as evident from both subjective and objective assessments. Lu et al. [21] introduced the dual bilateral least squares hybrid filter for infrared and visible image fusion; the residual network ResNet50-based fusion rule and structure tensor-based fusion rule are used to process the base- and detail-layers, respectively. Zhao et al. [22] introduced the image fusion method using the fractional-order variational method and data-driven tight frame, and this method has been tested on medical images, infrared and visible images, and multi-focus images, wherein the effectiveness of the algorithm has been verified. Kong et al. [23] introduced the image fusion method based on side window filter and framelet transform. The methods mentioned above are all based on three steps: image decomposition, coefficient fusion, and image reconstruction. Indeed, the selection of rules for coefficients fusion is a crucial aspect of the image fusion process, as different fusion rules can have a significant impact on the final fusion results.

2.2. Deep-Learning-Based Fusion Methods

Bouzos et al. [24] introduced a convolutional-neural-network-based conditional random field model for structured multi-focus image fusion. Yang et al. [25] introduced a local binary pattern (LBP)-based proportional input generative adversarial network for image fusion. Jin et al. [26] introduced the unsupervised multi-focus image fusion algorithm based on Transformer and U-Net. Zhou et al. [27] introduced the unsupervised dense network with multi-scale convolutional block attention for multi-focus image fusion. Zhang et al. [28] proposed a fast unified image fusion network based on proportional maintenance of gradient and intensity (PMGI). Fang et al. [29] introduced the deep-learning-based threshold post-processing multi-focus image fusion method. Yang et al. [30] proposed an image fusion method based on latent low-rank representation and convolutional neural network. These deep learning-based methods have shown promising results in the field of image fusion, achieving relatively good performance compared to traditional techniques. However, it is important to acknowledge that these deep-learning-based methods often demand a substantial amount of computational time, mainly due to the requirement for large volumes of training data.

3. Background

3.1. Nonsubsampled Shearlet Transform

The nonsubsampled shearlet transform (NSST) is a multi-scale transform with translation invariance, which has significant effects in the field of image fusion [10,31]. The non-subsampled pyramid filter bank (NSP) and the shearlet filter (SF) are used for the multi-scale decomposition and multi-directional decomposition, respectively. Figure 2 shows an example of NSST decomposition performed on one source image with three levels [10,31]. NSST decomposes the image into high-frequency and low-frequency parts, and they are processed separately. Wavelet transform has limited ability in representing data features in the high-dimensional space of images; NSST solves this problem, and it can effectively extract edge details of images.

3.2. Parameter-Adaptive Pulse-Coupled Neural Network

The simplified pulse-coupled neural network (SPCNN) model was introduced by Chen et al. [32] in 2011 for image segmentation, and the SPCNN model is depicted as follows:

$$F_{ij}\left[n\right]=S_{ij}$$

(1)

$$L_{ij}\left[n\right]=V_L{\displaystyle \sum _{kl}{W}_{ijkl}}{Y}_{kl}\left[n-1\right]$$

(2)

$$U_{ij}\left[n\right]=e^{-{\alpha }_f}{U}_{ij}\left[n-1\right]+F_{ij}\left[n\right]\left(1+\beta L_{ij}\left[n\right]\right)$$

(3)

$$Y_{ij}\left[n\right]=\left\{\begin{array}{l}1 \mathrm{if} U_{ij}\left[n\right]>E_{ij}\left[n-1\right]\\ 0 \mathrm{else}\end{array}\right.$$

(4)

$$E_{ij}\left[n\right]=e^{-{\alpha }_e}{E}_{ij}\left[n-1\right]+V_E{Y}_{ij}\left[n\right]$$

(5)

where $F_{ij}\left[n\right]$ and $L_{ij}\left[n\right]$ show the feeding input and linking input of the neuron at position $\left(i,j\right)$ in iteration $n$, respectively. $F_{ij}\left[n\right]$ is fixed to the intensity of input image $S_{ij}$ during the whole iteration. $L_{ij}\left[n\right]$ is associated with the previous firing status of eight neighboring neurons through the synaptic weights.

$$W_{ijkl}=\left[\begin{array}{ccc}0.5& 1& 0.5\\ 1& 0& 1\\ 0.5& 1& 0.5\end{array}\right]$$

(6)

The structure of the SPCNN model is shown in Figure 3 [3]. The SPCNN model is initialized as $Y_{ij}\left[0\right]=0$, $U_{ij}\left[0\right]=0$ and $E_{ij}\left[0\right]=0$. In the improved model parameter-adaptive pulse-coupled neural network (PAPCNN), the parameters ${\alpha }_f$, $\lambda$, $V_E$, and ${\alpha }_e$ can be adaptively computed by

$${\alpha }_f=\log \left(1/\sigma \left(S\right)\right)$$

(7)

$$\lambda =\frac{\left(S_{\max }/S^{\prime }\right)-1}{6}$$

(8)

$$V_E=e^{-{\alpha }_f}+1+6\lambda$$

(9)

$${\alpha }_e=\ln \left(\frac{\frac{V_E}{S^{\prime }}}{\frac{1-e^{-3{\alpha }_f}}{1-e^{-{\alpha }_f}}+6\lambda e^{-{\alpha }_f}}\right)$$

(10)

where $\sigma \left(S\right)$ indicates the standard deviation of the input image $S$ of range $\left[0, 1\right]$. $S^{\prime }$ and $S_{\max }$ show the normalized Otsu threshold and the maximum intensity of the input image, respectively. More details can be seen the references [3,32].

4. Proposed Fusion Method

In this section, the multi-focus image fusion method based on parameter-adaptive pulse-coupled neural network and fractal dimension in nonsubsampled shearlet transform domain is constructed. The proposed method consists of four steps: NSST decomposition, low-frequency sub-bands fusion, high-frequency sub-bands fusion, and inverse NSST. Figure 4 shows the structure of the proposed fusion method. More details can be concluded as follows:

4.1. NSST Decomposition

A $L$-level NSST decomposition is applied on source images A and B to generate the corresponding decomposed sub-bands $\left\{L_A,H_A^{l,k}\right\}$ and $\left\{L_B,H_B^{l,k}\right\}$, respectively. Here, $L_A$ shows the low-frequency sub-band of A and $H_A^{l,k}$ shows the high-frequency sub-band of A generated at the $l$-th decomposition level with the direction $k$. The meanings of $L_B$ and $H_B^{l,k}$ are the same with respect to B.

$$A\stackrel{NSST}{\Rightarrow }\left\{L_A,H_A^{l,k}\right\}$$

(11)

$$B\stackrel{NSST}{\Rightarrow }\left\{L_B,H_B^{l,k}\right\}$$

(12)

4.2. Low-Frequency Sub-Bands Fusion

The low-frequency sub-bands contain the main energy information and background information. In this subsection, the PAPCNN model is used to process the low-frequency sub-bands, and the feeding input is $F_{ij}\left[n\right]=\left|L_S\right|,S\in \left\{A,B\right\}$. The activity level of the low-frequency coefficient is generated by the total firing times during the whole iteration. Based on the PAPCNN model, the firing times can be accumulated by adding the following step at the end of each iteration:

$$T_{ij}\left[n\right]=T_{ij}\left[n-1\right]+Y_{ij}\left[n\right]$$

(13)

Here, the firing times of each neuron is $T_{ij}\left[N\right]$, where N depicts the total number of iterations. For the low-frequency sub-bands $L_A$ and $L_B$ of source images A and B, respectively, the PAPCNN firing times are computed and marked by $T_{A,ij}\left[N\right]$ and $T_{B,ij}\left[N\right]$, respectively. The fused low-frequency sub-band $L_F\left(i,j\right)$ is computed by

$$L_F\left(i,j\right)=\left\{\begin{array}{l}{L}_A\left(i,j\right), \mathrm{if} T_{A,ij}\left[N\right]\ge T_{B,ij}\left[N\right]\\ L_B\left(i,j\right), \mathrm{else}\end{array}\right.$$

(14)

which denotes that the coefficient with larger firing times is chosen as the fused coefficient.

4.3. High-Frequency Sub-Bands Fusion

The high-frequency sub-bands contain textures, details, and some noise. The fractal-dimension-based focus measure (FDFM) is constructed [33], and it is simple and easy to compute the clarity of a pixel; they are defined as follows:

$$FDF{M}^{H_A^{l,k}}\left(i,j\right)=g_{\max }^{H_A^{l,k}}\left(i,j\right)-g_{\min }^{H_A^{l,k}}\left(i,j\right)$$

(15)

$$FDF{M}^{H_B^{l,k}}\left(i,j\right)=g_{\max }^{H_B^{l,k}}\left(i,j\right)-g_{\min }^{H_B^{l,k}}\left(i,j\right)$$

(16)

where $g_{\max }^X\left(i,j\right)$ and $g_{\min }^X\left(i,j\right)$ are the maximum and minimum intensities, respectively, over a $3\times 3$ window centered at the ${\left(i,j\right)}^{th}$ pixel of $X\in \left\{H_A^{l,k},H_B^{l,k}\right\}$.

The multi-scale morphological gradients (MSMG) of a pixel can be effectively used as a clarity measure of that pixel [34]. In the first step, the multi-scale structuring elements are constructed:

$$S{E}_t=\underset{t \mathrm{times}}{\underbrace{S{E}_1\oplus S{E}_1\cdots \oplus S{E}_1}},t\in \left[1, 2,\cdots , n\right]$$

(17)

where $n$ depicts the number of scales and $S{E}_1$ shows the basic structuring element at radius $r$. In this subsection, the radius $r$ is 5.

In the next step, the morphological gradient operators are utilized to extract the gradient features $G_t$ from image $X$ according to the following:

$$G_t\left(i,j\right)=X\left(i,j\right)\oplus S{E}_1-X\left(i,j\right)\Theta S{E}_1,t\in \left[1, 2,\cdots , n\right]$$

(18)

where $\oplus$ and $\Theta$ denote the morphological dilation and erosion operators, respectively.

In the third step, the gradients of all scales are merged into the multi-scale morphological gradients (MSMG):

$$MSMG\left(i,j\right)={\displaystyle \sum _{t=1}^n{w}_t}{G}_t\left(i,j\right)$$

(19)

$$w_t=\frac{1}{2\times t+1}$$

(20)

In this subsection, the FDFM combined with MSMG is used to fuse the high-frequency sub-bands, and the corresponding equation is defined as follows:

$$H_F^{l,k}\left(i,j\right)=\left\{\begin{array}{l}{H}_A^{l.k}\left(i,j\right), \mathrm{if} FDF{M}^{H_A^{l,k}}\left(i,j\right)\times MSM{G}^{H_A^{l.k}}\left(i,j\right)\\ \ge FDF{M}^{H_B^{l,k}}\left(i,j\right)\times MSM{G}^{H_B^{l,k}}\left(i,j\right)\\ H_B^{l,k}\left(i,j\right), \mathrm{else}\end{array}\right.$$

(21)

where $H_F^{l,k}\left(i,j\right)$ denotes the fused coefficient and $MSM{G}^X\left(i,j\right)|X\in \left\{H_A^{l,k},H_B^{l,k}\right\}$ indicates the MSMG of $X$ at the $\left(i,j\right)\mathrm{th}$ pixel.

4.4. Inverse NSST

The fused image $F$ is generated by inverse NSST performed on the fused coefficients $\left\{L_F,H_F^{l,k}\right\}$:

$$F\stackrel{Inverse NSST}{\Leftarrow }\left\{L_F,H_F^{l,k}\right\}$$

(22)

where $F$ denotes the fused image.

The main steps of the proposed multi-focus image fusion approach are summarized in Algorithm 1.

Algorithm 1 Proposed multi-focus image fusion method

Input: the source images: A and B. Parameters: the number of NSST decomposition levels: L, the number of directions at each decomposition level: $K\left(l\right)$, $l\in \left[1,L\right]$, the number of PAPCNN iterations: N, the radius $r$   Step 1: NSST decomposition For each source image $X\in \left\{A,B\right\}$ Perform NSST on $X$ to generate $\left\{L_X,H_X^{l,k}\right\}$, $l\in \left[1,L\right]$, $k\in \left[1,K\left(l\right)\right]$; End   Step 2: Low-frequency sub-bands fusion For each source image $X\in \left\{A,B\right\}$ Initialize the PAPCNN model: $Y_{ij}\left[0\right]=0,$ $U_{ij}\left[0\right]=0$, $E_{ij}\left[0\right]=0$, $T_{ij}\left[0\right]=0$ and $F_{ij}\left[n\right]=\left|L_X\right|$, $n\in \left[1,n\right]$; Estimate the PAPCNN parameters via Equations (7)–(10); For each iteration $n=1:N$ Compute the PAPCNN model using Equations (2)–(5) and (13); End End Merge $L_A$ and $L_B$ via Equation (14) to generate $L_F$;   Step 3: High-frequency sub-bands fusion        For each level $l=1:L$              For each direction $k=1:K\left(l\right)$                     For each source image $X\in \left\{A,B\right\}$        Estimate $FDF{M}^{H_X^{l,k}}\left(i,j\right)$ via Equations (15) and (16)                     Compute $MSM{G}^{H_X^{l,k}}\left(i,j\right)$ via Equation (19)        End        Merge $H_A^{l,k}$ and $H_B^{l,k}$ via Equation (21) to generate $H_F^{l,k}$;              End        End   Step 4: Inverse NSST Perform inverse NSST on $\left\{L_F,H_F^{l,k}\right\}$ using Equation (22) to generate $F$; Output: the fused image $F$.

5. Experimental Results and Discussion

5.1. Experimental Setting

In this subsection, we utilized the Lytro dataset [16] for evaluating the performance of the proposed method, along with several comparison algorithms. Figure 5 illustrates the examples of images from the Lytro dataset. These comparative algorithms include parameter-adaptive pulse coupled neural network and nonsubsampled shearlet transform (NSSTPA) [3], proportional maintenance of gradient and intensity (PMGI) [28], three-layer decomposition and sparse representation (TLDSR) [35], convolutional simultaneous sparse approximation (CSSA) [36], local extreme map guided multi-modal image fusion (LEGFF) [37], unified unsupervised image fusion network (U2Fusion) [38], distance-weighted regional energy and nonsubsampled shearlet transform (NSSTDW) [18], and zero-shot multi-focus image fusion (ZMFF) [2]. Qualitative and quantitative evaluations were used to evaluate the results of fusion; the quantitative evaluation indicators include the edge-based similarity measurement $Q_{AB/F}$ [39,40], the structural similarity based metric $Q_E$ [41], the feature mutual information metric $Q_{FMI}$ [42], the gradient based metric $Q_G$ [41], the nonlinear correlation information entropy $Q_{NCIE}$ [41], the phase-congruency-based metric $Q_P$ [41], the mutual information metric $Q_{MI}$ [39,40], the normalized mutual information $Q_{NMI}$ [41], the structural-similarity-based metric $Q_Y$ introduced by Yang et al. $Q_Y$ [41,43] and the average gradient metric $Q_{AG}$ [44,45], peak signal-to-noise ratio $Q_{PSNR}$ [46], and mean square error $Q_{MSE}$ [47]. The algorithms with larger indicator values (except for metric $Q_{MSE}$) are better. In our method, the “maxflat” filter for pyramidal decomposition was adopted in NSST transform; the parameter N was set to 110.

5.2. Analysis of NSST Decomposition Levels

In this subsection, the number of NSST decomposition levels L was set to 1–4, correspondingly. In this paper, the numbers of directions were empirically set to 16, 16, 8, and 8 for the first four scales from fine to coarse, and the detailed direction settings for different values of L can be shown in

Table 1. The number of NSST decomposition levels and the corresponding direction settings for parameter analysis.

Number of Decomposition Levels	Number of Directions at Each Level
1	16
2	16, 16
3	16, 16, 8
4	16, 16, 8, 8

. The average metrics data of our method tested on the Lytro dataset with different NSST decomposition levels are shown in

Table 2. The average metrics data of our method tested on the Lytro dataset with different NSST decomposition levels.

Levels	${\mathit{Q}}_{\mathit{A}\mathit{B}/\mathit{F}}$	${\mathit{Q}}_{\mathit{E}}$	${\mathit{Q}}_{\mathit{F}\mathit{M}\mathit{I}}$	${\mathit{Q}}_{\mathit{G}}$	${\mathit{Q}}_{\mathit{N}\mathit{C}\mathit{I}\mathit{E}}$	${\mathit{Q}}_{\mathit{P}}$	${\mathit{Q}}_{\mathit{M}\mathit{I}}$	${\mathit{Q}}_{\mathit{N}\mathit{M}\mathit{I}}$	${\mathit{Q}}_{\mathit{Y}}$	${\mathit{Q}}_{\mathit{A}\mathit{G}}$	${\mathit{Q}}_{\mathit{P}\mathit{S}\mathit{N}\mathit{R}}$	${\mathit{Q}}_{\mathit{M}\mathit{S}\mathit{E}}$
1	0.4986	0.4843	0.8655	0.4941	0.8279	0.4845	6.6189	0.8859	0.8463	8.9285	33.3592	33.8391
2	0.6415	0.7508	0.8863	0.6356	0.8257	0.6480	6.3287	0.8451	0.8943	11.0215	33.3365	33.9162
3	0.7185	0.8600	0.8972	0.7149	0.8272	0.7749	6.5738	0.8766	0.9381	11.6322	33.8198	30.3342
4	0.7384	0.8756	0.8989	0.7359	0.8289	0.8114	6.8598	0.9142	0.9544	11.7205	34.2875	27.0097

. From the results, we can denote that when the NSST decomposition levels L was set to 4 and the corresponding number of directions were 16, 16, 8, and 8 from fine to coarse, the proposed method can achieve the best values.

5.3. Qualitative and Quantitative Evaluation

In this subsection, five pairs of multi-focus images selected from the Lytro dataset were used as a qualitative evaluation display, and the fusion results are shown in Figure 6, Figure 7, Figure 8, Figure 9 and Figure 10, and the corresponding metric data can be found in

Table 3. The quantitative evaluation of the different methods shown in Figure 6.

	NSSTPA	PMGI	TLDSR	CSSA	LEGFF	U2Fusion	NSSTDW	ZMFF	Proposed
$Q_{AB/F}$	0.6526	0.5466	0.7524	0.7205	0.6923	0.6575	0.6448	0.7342	0.7524
$Q_E$	0.8543	0.6316	0.8604	0.8805	0.8205	0.7952	0.8488	0.8779	0.8849
$Q_{FMI}$	0.9362	0.9169	0.9333	0.9341	0.9306	0.9206	0.9367	0.9312	0.9379
$Q_G$	0.6167	0.5156	0.7384	0.6977	0.6658	0.6338	0.6095	0.7134	0.7386
$Q_{NCIE}$	0.8180	0.8169	0.8232	0.8215	0.8158	0.8176	0.8180	0.8222	0.8239
$Q_P$	0.4842	0.3925	0.7915	0.7187	0.6937	0.6640	0.4783	0.7673	0.8000
$Q_{MI}$	5.3646	5.1347	6.2537	6.0385	4.8919	5.2894	5.3439	6.1505	6.4258
$Q_{NMI}$	0.7717	0.7410	0.8995	0.8658	0.6971	0.7472	0.7696	0.8827	0.9227
$Q_Y$	0.8680	0.7656	0.9739	0.9463	0.9164	0.8832	0.8473	0.9644	0.9781
$Q_{AG}$	7.4028	5.9436	7.9407	8.0460	10.2919	9.4564	7.1469	8.0149	8.2592
$Q_{PSNR}$	35.3776	44.7844	34.6425	37.1009	33.2880	31.8668	34.7133	37.4030	36.3369
$Q_{MSE}$	18.8504	2.1609	22.3270	12.6763	30.4987	42.3060	21.9657	11.8245	15.1144

,

Table 4. The quantitative evaluation of the different methods shown in Figure 7.

	NSSTPA	PMGI	TLDSR	CSSA	LEGFF	U2Fusion	NSSTDW	ZMFF	Proposed
$Q_{AB/F}$	0.5955	0.3491	0.7169	0.6729	0.6639	0.5988	0.6078	0.6780	0.7223
$Q_E$	0.7067	0.3959	0.8387	0.8592	0.8327	0.7853	0.7599	0.8539	0.8621
$Q_{FMI}$	0.8988	0.8947	0.9123	0.9089	0.9106	0.9007	0.9036	0.9065	0.9146
$Q_G$	0.5944	0.3491	0.7179	0.6734	0.6649	0.5985	0.6055	0.6774	0.7236
$Q_{NCIE}$	0.8308	0.8285	0.8356	0.8340	0.8279	0.8282	0.8305	0.8340	0.8382
$Q_P$	0.6179	0.3640	0.8270	0.7261	0.7240	0.6573	0.6068	0.7762	0.8404
$Q_{MI}$	7.0354	6.7140	7.7276	7.5562	6.5996	6.6513	7.0030	7.5359	7.9702
$Q_{NMI}$	0.9355	0.9185	1.0265	1.0026	0.8754	0.8853	0.9306	1.0015	1.0581
$Q_Y$	0.8837	0.6784	0.9639	0.9114	0.8700	0.7763	0.8795	0.9196	0.9623
$Q_{AG}$	7.0544	3.8431	7.8369	7.6230	9.4770	8.3623	7.1343	7.5959	8.0239
$Q_{PSNR}$	35.1474	33.6623	34.6805	37.4187	34.1485	29.4442	34.6818	37.7744	34.8978
$Q_{MSE}$	19.8764	27.9799	22.1327	11.7819	25.0165	73.9023	22.1257	10.8553	21.0523

,

Table 5. The quantitative evaluation of the different methods shown in Figure 8.

	NSSTPA	PMGI	TLDSR	CSSA	LEGFF	U2Fusion	NSSTDW	ZMFF	Proposed
$Q_{AB/F}$	0.6676	0.2819	0.7110	0.6580	0.6718	0.5919	0.6436	0.6702	0.7199
$Q_E$	0.8093	0.3429	0.8025	0.8339	0.8159	0.7646	0.8186	0.8290	0.8405
$Q_{FMI}$	0.8921	0.8835	0.8943	0.8921	0.8924	0.8863	0.8917	0.8898	0.8974
$Q_G$	0.6714	0.2829	0.7111	0.6613	0.6757	0.5957	0.6485	0.6736	0.7206
$Q_{NCIE}$	0.8280	0.8251	0.8311	0.8294	0.8252	0.8240	0.8280	0.8298	0.8321
$Q_P$	0.7641	0.4029	0.8204	0.7476	0.7804	0.6907	0.7152	0.7763	0.8283
$Q_{MI}$	6.7720	6.2935	7.2642	7.0010	6.3104	6.0734	6.7152	7.0675	7.3742
$Q_{NMI}$	0.8780	0.8489	0.9414	0.9089	0.8168	0.8123	0.8701	0.9189	0.9555
$Q_Y$	0.9146	0.6042	0.9566	0.9037	0.8768	0.6872	0.9021	0.9146	0.9463
$Q_{AG}$	9.9009	4.3240	10.0226	9.6842	12.1493	10.0790	9.6911	9.8506	10.3422
$Q_{PSNR}$	33.5968	28.0630	33.3634	35.8596	32.6296	30.4292	33.0496	36.3403	33.8703
$Q_{MSE}$	28.4055	101.5741	29.9736	16.8703	35.4914	58.9055	32.2200	15.1025	26.6719

,

Table 6. The quantitative evaluation of the different methods shown in Figure 9.

	NSSTPA	PMGI	TLDSR	CSSA	LEGFF	U2Fusion	NSSTDW	ZMFF	Proposed
$Q_{AB/F}$	0.6164	0.3863	0.6944	0.6437	0.6415	0.5668	0.6036	0.6532	0.6995
$Q_E$	0.8039	0.5027	0.8259	0.8322	0.8085	0.7162	0.8088	0.8297	0.8413
$Q_{FMI}$	0.8839	0.8687	0.8795	0.8785	0.8790	0.8733	0.8848	0.8771	0.8852
$Q_G$	0.6233	0.3959	0.6973	0.6500	0.6467	0.5741	0.6123	0.6588	0.7032
$Q_{NCIE}$	0.8256	0.8242	0.8279	0.8260	0.8222	0.8230	0.8259	0.8268	0.8295
$Q_P$	0.7314	0.4936	0.8645	0.8005	0.8208	0.7143	0.6986	0.8218	0.8760
$Q_{MI}$	6.4511	6.2386	6.8508	6.5525	5.8741	6.0200	6.4853	6.6908	7.1167
$Q_{NMI}$	0.8380	0.8306	0.8906	0.8504	0.7609	0.7818	0.8425	0.8701	0.9244
$Q_Y$	0.9029	0.7067	0.9624	0.9079	0.8891	0.7638	0.8908	0.9211	0.9630
$Q_{AG}$	9.2647	5.1823	9.8975	9.7875	11.8714	11.3723	9.0774	9.8887	10.2103
$Q_{PSNR}$	33.9717	30.7537	33.5135	35.7005	32.8602	28.8632	33.7087	36.3876	35.0734
$Q_{MSE}$	26.0559	54.6657	28.9554	17.4995	33.6561	84.4809	27.6826	14.9391	20.2181

and

Table 7. The quantitative evaluation of the different methods shown in Figure 10.

	NSSTPA	PMGI	TLDSR	CSSA	LEGFF	U2Fusion	NSSTDW	ZMFF	Proposed
$Q_{AB/F}$	0.7248	0.2478	0.7619	0.7349	0.7210	0.6604	0.7042	0.7442	0.7869
$Q_E$	0.9004	0.3656	0.8443	0.9143	0.8320	0.8484	0.8916	0.9088	0.9161
$Q_{FMI}$	0.8892	0.8762	0.8834	0.8886	0.8826	0.8763	0.8894	0.8856	0.8914
$Q_G$	0.7115	0.2410	0.7559	0.7257	0.7140	0.6536	0.6938	0.7355	0.7829
$Q_{NCIE}$	0.8167	0.8155	0.8201	0.8198	0.8143	0.8166	0.8166	0.8191	0.8204
$Q_P$	0.7127	0.4395	0.7713	0.7410	0.7494	0.6647	0.6570	0.7629	0.7979
$Q_{MI}$	5.1954	4.9412	5.8485	5.7667	4.6668	5.1743	5.1316	5.6763	5.9240
$Q_{NMI}$	0.6834	0.6672	0.7695	0.7605	0.6153	0.6856	0.6763	0.7487	0.7794
$Q_Y$	0.9178	0.5950	0.9629	0.9179	0.8941	0.8039	0.8902	0.9433	0.9719
$Q_{AG}$	14.6990	6.7097	14.7668	14.6963	20.5244	14.7662	14.2147	14.7826	15.5686
$Q_{PSNR}$	31.3848	26.9100	31.0631	34.0886	30.5288	31.8019	31.0528	34.1800	32.4951
$Q_{MSE}$	47.2717	132.4599	50.9067	25.3644	57.5702	42.9431	51.0270	24.8357	36.6079

.

From Figure 6, we can observe that the NSSTPA method produced a relatively blurry image, failing to achieve a full focus effect; the PMGI method resulted in a dark image, leading to some loss of information in the fused image; the TLDSR method generated a fully focused fusion image, but the overall brightness of the image was slightly darker; both CSSA and LEGFF methods yielded similar fusion results, but the LEGFF method provided clearer details; the U2Fusion method enhanced sharpness across the entire image but caused darkening in some areas; the fused image generated by the NSSTDW method appeared blurry, with some details lost; the ZMFF method achieved a relatively good fusion effect, but when compared to our method, the fused image generated by our approach maintained a moderate brightness and full focus information. From Table 3, we can conclude that our method outperformed others, as the metrics data $Q_{AB/F}$, $Q_E$, $Q_{FMI}$, $Q_G$, $Q_{NCIE}$, $Q_P$, $Q_{MI}$, $Q_{NMI}$, and $Q_Y$ demonstrated superior results of 0.7524, 08849, 0.9379, 0.7386, 0.8239, 0.8000, 6.4258, 0.9227, and 0.9781, respectively.

From Figure 7, we can see that the NSSTPA algorithm generated blurry images, for example, the woman on the right side of the image was not in focus; the PMGI method generated a dark and fuzzy image; the TLDSR method generated a full-focused fusion image; the CSSA and LEGFF methods generated full-focused images, but a small portion of these images appeared darkened; the U2Fusion method produced a fused image with some areas darker, such as the glasses area; the NSSTDW method also resulted in a blurry image; the ZMFF method obtained a full-focused image; in contrast, the fused image computed by the proposed method exhibited full-focused and clear information. Based on the data in Table 4, we can conclude that the metrics $Q_{AB/F}$, $Q_E$, $Q_{FMI}$, $Q_G$, $Q_{NCIE}$, $Q_P$, $Q_{MI}$, and $Q_{NMI}$ produced by the proposed method were the best among all the evaluated algorithms at 0.7223, 0.8621, 0.9146, 0.7236, 0.8382, 0.8404, 7.9702, and 1.0581, respectively. Comparatively, the metrics $Q_Y$ and $Q_{AG}$ obtained through the TLDSR and LEGFF algorithms yielded the best results, with values of 0.9639 and 9.4770, respectively. However, our corresponding metrics ranked as the second and third best performers, with values of 0.9623 and 8.0239, respectively. The metrics $Q_{PSNR}$ and $Q_{MSE}$ generated by the ZMFF method yielded the best results, with values of 37.7744 and 10.8553, respectively.

From Figure 8, it is evident that the NSSTPA and PMGI methods produced blurry images, particularly the PMGI algorithm, which exhibited the most severe information loss; the TLDSR algorithm introduced blurry artifacts into the fused image; the CSSA, LEGFF, and ZMFF methods had full-focused information in the fusion images, but the roof part of the building in these images appeared somewhat dark; the U2Fusion approach darkened certain areas in the fused image; the NSSTDW method resulted in a blurry image, particularly noticeable in the tree information in the lower left corner of the image. In contrast, our method generated a fused image with the best fusion effect, displaying moderate brightness and clear information. Referring to Table 5, it is evident that our method achieved the best performance based on the metrics $Q_{AB/F}$, $Q_E$, $Q_{FMI}$, $Q_G$, $Q_{NCIE}$, $Q_P$, $Q_{MI}$, and $Q_{NMI}$, achieving scores of 0.7199, 0.8405, 0.8974, 0.7206, 0.8321, 0.8283, 7.3742, and 0.9555, respectively.

From Figure 9, it is evident that the NSSTPA, TLDSR, and NSSTDW methods generated blurry images with unfocused background areas; for example, the building information on the left side of the images appeared blurry; the PMGI method caused distortion in the fused image and severe information loss; the CSSA method resulted in ambiguous building information on the right side of the fused image; the LEGFF and ZMFF methods generated fused images with all-focused information, and the LEGFF method produces a brighter fused image; the U2Fusion method led to some areas becoming darker and others brighter, resulting in potential information loss. However, the fused image achieved by our method exhibited the best performance, with fully focused information. Referring to Table 6, we can denote that the metrics data $Q_{AB/F}$, $Q_E$, $Q_{FMI}$, $Q_G$, $Q_{NCIE}$, $Q_P$, $Q_{MI}$, $Q_{NMI}$, and $Q_Y$ generated by our method were the best, with scores of 0.6995, 0.8413, 0.8852, 0.7032, 0.8295, 0.8760, 7.1167, 0.9244, and 0.9630, respectively.

From Figure 10, we can denote that artifacts were generated around the bottle in the fused images computed by the NSSTPA and TLDSR methods; the PMGI method caused distortion in the fused image; the fusion results achieved by the CSSA, LEGFF, and ZMFF methods looked similar, with the LEGFF method showing clearer details; the U2Fusion method created varying degrees of darkness on the bottle, cabin, and distant mountain scenery in the fused image; the fused image calculated by the NSSTDW method was blurry. However, the fused image generated by our method exhibited all-focused information without distortion and artifacts. Referring to Table 7, it is evident that the metrics $Q_{AB/F}$, $Q_E$, $Q_{FMI}$, $Q_G$, $Q_{NCIE}$, $Q_P$, $Q_{MI}$, $Q_{NMI}$, and $Q_Y$ obtained by our method were the best, with scores of 0.7869, 0.9161, 0.8914, 0.7829, 0.8204, 0.7979, 5.9240, 0.7794, and 0.9719, respectively.

Figure 11 presents a line chart illustrating the fluctuation trends of various algorithms on 20 data groups, along with the calculation of their respective average metrics. From

Table 8. The average metrics data of the different methods shown in Figure 11.

	NSSTPA	PMGI	TLDSR	CSSA	LEGFF	U2Fusion	NSSTDW	ZMFF	Proposed
$Q_{AB/F}$	0.6720	0.3901	0.7320	0.6897	0.6810	0.6143	0.6554	0.7087	0.7384
$Q_E$	0.8247	0.4736	0.8452	0.8706	0.8195	0.7835	0.8327	0.8687	0.8756
$Q_{FMI}$	0.8931	0.8815	0.8947	0.8948	0.8937	0.8844	0.8943	0.8925	0.8989
$Q_G$	0.6655	0.3857	0.7291	0.6853	0.6754	0.6093	0.6482	0.7030	0.7359
$Q_{NCIE}$	0.8254	0.8225	0.8287	0.8264	0.8214	0.8221	0.8255	0.8271	0.8289
$Q_P$	0.6932	0.4620	0.7995	0.7345	0.7565	0.6657	0.6519	0.7853	0.8114
$Q_{MI}$	6.3212	5.8641	6.8226	6.5378	5.6138	5.7765	6.3023	6.6271	6.8598
$Q_{NMI}$	0.8431	0.8004	0.9103	0.8716	0.7473	0.7725	0.8406	0.8838	0.9142
$Q_Y$	0.8955	0.6738	0.9563	0.8911	0.8817	0.7912	0.8767	0.9313	0.9544
$Q_{AG}$	10.8470	5.8684	11.3823	11.1252	14.8183	12.0343	10.5661	11.2183	11.7205
$Q_{PSNR}$	34.0329	32.4782	33.5788	35.9086	32.6160	31.2098	33.5706	36.4143	34.2875
$Q_{MSE}$	28.5567	75.3956	31.7463	18.0173	39.1523	59.4424	31.9315	16.3895	27.0097

, it becomes evident that our method outperformed the others, as it achieved the highest average metrics for $Q_{AB/F}$, $Q_E$, $Q_{FMI}$, $Q_G$, $Q_{NCIE}$, $Q_P$, $Q_{MI}$, and $Q_{NMI}$, with scores of 0.7384, 0.8756, 0.8989, 0.7359, 0.8289, 0.8114, 6.8598, and 0.9142, respectively. The average metrics $Q_Y$ and $Q_{AG}$ computed by the TLDSR and LEGFF methods were the best, with scores of 0.9563 and 14.8183, respectively, and the corresponding metrics computed by the proposed method ranked second and third, with scores of 0.9544 and 11.7205, respectively. The average metrics $Q_{PSNR}$ and $Q_{MSE}$ generated by the ZMFF method were the best, with scores of 36.4143 and 16.3895, respectively, and the corresponding metrics computed by the proposed method ranked in third, with scores of 34.2875 and 27.0097, respectively. This substantiates the superiority and universality of our fusion algorithm.

A thorough analysis led to the conclusion that our algorithm demonstrates superior fusion performance and effectively aligns subjective and objective analyses. The results indicate that our approach strikes a harmonious balance between the two, further affirming its efficacy and applicability.

For a more comprehensive experimental comparison, we added three fusion algorithms, namely, NSCTMSF [48], NSCTRPCNN [49], and GD [50], published in 2012, 2013, and 2016, respectively. The average metrics data comparison is provided in

Table 9. The average metrics data of other different methods.

	NSCTMSF	NSCTRPCNN	GD	Proposed
$Q_{AB/F}$	0.7131	0.7103	0.7034	0.7384
$Q_E$	0.8615	0.8644	0.7874	0.8756
$Q_{FMI}$	0.8971	0.8972	0.8887	0.8989
$Q_G$	0.7087	0.7058	0.6987	0.7359
$Q_{NCIE}$	0.8263	0.8280	0.8139	0.8289
$Q_P$	0.7631	0.7616	0.7466	0.8114
$Q_{MI}$	6.4660	6.7075	3.8521	6.8598
$Q_{NMI}$	0.8616	0.8945	0.5113	0.9142
$Q_Y$	0.9239	0.9208	0.8608	0.9544
$Q_{AG}$	11.4225	11.3477	11.6844	11.7205
$Q_{PSNR}$	33.5139	34.7486	26.5742	34.2875
$Q_{MSE}$	32.6256	24.5159	150.1382	27.0097

. From the metrics data, it can be seen that our algorithm had more advantages than previous fusion algorithms.

5.4. Sequence Multi-Focus Image Fusion

In this subsection, we proceed with the testing of our algorithm on four sets of sequence multi-focus images obtained from the Lytro dataset [16]. The results in Figure 12 clearly demonstrate the effectiveness of our approach in producing fully focused images. As a result, we can confidently assert that our algorithm is equally adept at handling sequence multi-focus image fusion.

5.5. Further Discussion

We conducted a comprehensive analysis of the impact of NSST decomposition levels on image fusion performance and found that increasing the number of decomposition levels appropriately significantly improved the image fusion results. Our proposed multi-focus image fusion model based on NSST was thoroughly validated through simulation experiments. When compared to traditional and deep-learning-based algorithms, our method exhibited substantial advantages in terms of preserving edge details and effectively utilizing complementary image information, as indicated by both subjective and objective evaluation criteria. However, despite these remarkable achievements, there are still certain limitations that warrant further investigation. Notably, the average gradient metric of our algorithm did not reach its optimal value, although it remains competitive with other approaches. As a result, our next step involves delving deeper into enhancing the average gradient information of the fused images. Figure 13 showcases the fusion results of our algorithm applied to medical images, infrared and visible images. For anatomical and functional image fusion, we utilized the RGB to YUV color space conversion, where the Y channel of the functional image was fused with the anatomical image using our proposed algorithm. Subsequently, the YUV to RGB space conversion was applied to obtain the final fusion result. The results demonstrate the effectiveness of our algorithm in achieving information complementarity.

6. Conclusions

To achieve clear and fully focused images while minimizing information loss, we proposed a novel parameter-adaptive pulse coupled neural network-based multi-focus image fusion method, operating in the nonsubsampled shearlet transform domain, with the incorporation of fractal dimension. This sophisticated approach allows for better preservation of background, brightness, and detail information in the fused images.

The fusion process involves two key components. First, we employed the parameter-adaptive pulse coupled neural network-based fusion rule to fuse the low-frequency sub-bands, which optimally combined information from different focus levels. Second, for the high-frequency sub-bands, we utilized the fractal dimension integrated multi-scale morphological gradient fusion rule, enabling us to effectively merge the fine details from the input images.

To validate the effectiveness of our proposed method, we conducted extensive testing and validation using the publicly available Lytro dataset, which comprises 20 pairs of images and four sets of sequence images. The results were compared with state-of-the-art fusion algorithms published both domestically and internationally from 2012 to 2023. Through this thorough evaluation, we demonstrated that our algorithm stands at the forefront of technology in the field of multi-focus image fusion.

Moreover, to enhance the practical usability and efficiency of the proposed method, the implementation of a Python or C++ version may significantly accelerate the fusion process.