MRI Image Fusion Based on Sparse Representation with Measurement of Patch-Based Multiple Salient Features

Abstract

Multimodal medical image fusion is a fundamental, but challenging, problem in the fields of brain science research and brain disease diagnosis, as it is challenging for sparse representation (SR)-based fusion to characterize activity levels with a single measurement and not lose effective information. In this study, the Kronecker-criterion-based SR framework was applied for medical image fusion with a patch-based activity level, integrating salient features of multiple domains. Inspired by the formation process of vision systems, the spatial saliency was characterized by textural contrast (TC), composed of luminance and orientation contrasts, to promote the participation of more highlighted textural information in the fusion process. As a substitute for the conventional l₁-norm-based sparse saliency, the sum of sparse salient features (SSSF) was used as a metric for promoting the participation of more significant coefficients in the composition of the activity level measurement. The designed activity level measurement was verified to be more conducive to maintaining the integrity and sharpness of detailed information. Various experiments on multiple groups of clinical medical images verified the effectiveness of the proposed fusion method in terms of both visual quality and objective assessment. Furthermore, this study will be helpful for the further detection and segmentation of medical images.

1. Introduction

Over the past several decades, a variety of information processing technologies have led to major achievements in clinical diagnosis research [1], such as image classification [2,3], image fusion [4], and image segmentation [5]. The main purpose of medical image fusion is to combine the complementary information from various sensors to construct a new image with which to assist medical experts with diagnosis. Despite the simplicity of the idea, there are many challenges related to the theoretical background and the nature of medical images that need to be resolved. For instance, computed tomography (CT) imaging is informative regarding dense tissues, but lacks soft tissue information. In contrast, magnetic resonance imaging (MRI) is more suitable for soft tissues, but is short on dense tissue information. More crucially, single imaging tends to be ineffective at characterizing the symptoms of different diseases.

To overcome these challenges, a variety of image fusion methods have been proposed. The content of the image can be either visual (i.e., color, shape, or texture) or textual (i.e., to identify datasets appearing within an image). Some new advances in the fusion field consider these two aspects simultaneously [6,7]. To further improve fusion performance, some new features, such as different image moments [8,9,10], have also been used in image fusion.

The mainstream directions of image fusion mainly focus on the visual content, including the spatial domain [11,12] and transform domain [13,14]. The former usually addresses the fusion issue via image blocks or pixel-wise gradient information for multi-focus fusion [15,16,17] and multi-exposure fusion [18,19,20] tasks. The latter merges the transform coefficients relevant to source images with different reconstruction algorithms to obtain a fused image, which is recognized as being effective for multimodal image fusion [21,22]. Multi-scale transform (MST)-based medical image fusion is a mainstream research direction. Dual-tree complex wavelet transform (DTCWT) [23], non-subsampled shearlet transform (NSST) [24], and non-subsampled contourlet transform (NSCT) [25] are conventional MST methods for image fusion. In recent years, some novel MST-based methods have been proposed. Xia combined sparse representation with a pulse-coupled neural network (PCNN) in the NSCT domain for medical image fusion [26]. Yin proposed a parameter-adaptive pulse-coupled neural network as part of an NSST domain (NSST-PAPCNN)-based medical image fusion strategy [27]. Dinh proposed a Kirsch compass operator with a marine-predator-algorithm-based method for medical image fusion [28].

Differently from MST, the principle of SR is more in accordance with the human visual system (HVS), and compared to the MST-based methods, SR-based methods have two main distinctions. For the first distinction, the fixed basis limits the MST-based methods to expressing significant features, while the SR-based methods are flexible and can procure more intrinsic features by means of dictionary learning. For the second distinction, the MST-based methods are sensitive to noise and misregistration with large decomposition level settings, while the SR-based methods with overlapping patch-wise modes are robust for misregistration, which guarantees the accuracy of the spatial location of tissues. Therefore, a wide range of research on SR-based medical image fusion has been conducted in recent years [29,30,31].

However, there are still drawbacks to these SR-based methods. Firstly, they may be insufficient to handle fine details due to an over-complete dictionary, and this highly redundant dictionary will lead to visual artifacts in the fused result [32]. Secondly, the dictionary atoms are updated in column vector form, resulting in the loss of correlation and structural information. In addition to the drawbacks of SR itself, the fusion weight accuracy will inevitably be reduced with an unreasonable fusion strategy of coefficients. One issue relates to the activity level measure, which helps to recognize distinct features in the fusion process, and another issue concentrates on the integration of coefficients into the counterparts of the fused image. For the former issue, the l₁-norm mode is a conventional solution to describe the detailed information contained in sparse vectors [33]; however, the solution is insufficient to express the sparse saliency well, since detailed information that characterizes the activity level with the same weight cannot be highlighted. Furthermore, as SR is an approximate technique, it tends to fail to reflect the salient features of sparse coefficient maps accurately with a single measurement of the activity level, thereby further leading to the loss of detailed information. For this issue, it may reduce the contrast of the fused image with the weighted averaging rule, and the maximum absolute rule enables the fused image to absorb the main visual information of source images at the cost of minor information loss.

Based on the above discussion, we adopted a promising signal decomposition model, known as Kronecker-criterion-based SR [34], to solve the medical image fusion problem. The main contributions of this work are illustrated as follows:

(a)

Kronecker-criterion-based SR, with a designed activity level measure integrating the salient features of multiple domains, will effectively reduce the loss of structural detailed information in the fusion process.

(b)

Inspired by the formation process of the vision system, the spatial saliency by textural contrast consists of luminance and orientation contrasts that can promote more highlighted textural information in order to participate in the fusion process.

(c)

Compared with the l₁-norm-based activity level measure in sparse vectors, the transform saliency by the sum of sparse salient features can highlight more coefficients to measure the composite activity level through the sum of differences in the adjacent areas.

The rest of this paper is organized as follows: Section 2 provides a brief description of conventional sparse representation theory and the Kronecker-criterion-based SR, i.e., the separable dictionary learning algorithm. The detailed fusion scheme is described in Section 3. The experimental results and a discussion are given in Section 4. Finally, Section 5 concludes the paper.

2. Related Work

2.1. SR-Based Image Fusion

SR reflects the sparsity of natural signals with minimal sparse coefficients, and this is consistent with the principle of HVS [35]. Given $y\in R^m$ in the vector mode of signal sample $Y\in R^{\sqrt{m}\times \sqrt{m}}$ and an over-complete dictionary $D\in R^{m\times n}$ (m < n), the objective function of dictionary learning consisting of the fidelity term and penalty term is defined as

$$\arg \underset{D,\alpha }{\min }\frac{1}{2}{\Vert y-D\alpha \Vert }_2^2+\lambda R(\alpha )$$

(1)

where $\alpha$ represents the sparse vector, ${\Vert •\Vert }_2^2$ represents the l₂-norm, and $\lambda$ represents the regularization parameter of the penalty term $R(\alpha )$. SR can be roughly divided into two categories, the greedy scheme (e.g., matching pursuit (MP) [36] and orthogonal matching pursuit (OMP) [37]) with $R(\alpha )={\Vert \alpha \Vert }_0$ and the convex optimization scheme (e.g., alternating direction method of multipliers (ADMM) [38]) with $R(\alpha )={\Vert \alpha \Vert }_1$. The extremely high complexity inhibits the practicality of the convex optimization scheme, while the greedy scheme has superiority in this regard.

In the process of conventional SR-based image fusion, its ability to handle fine details with an over-complete dictionary may be insufficient, since atoms (i.e., vectors) of the pre-trained over-complete dictionary are updated one-by-one with either the method of optimal directions (MOD) or k-singular value decomposition (K-SVD). This can be understood as extracting image textural information from only a one-dimensional direction; this breaks the potential correlations within the image, thus causing the obtained pre-trained dictionary to become unstructured. Meanwhile, the highly redundant dictionary is sensitive to random noise and may cause visual artifacts to appear. Therefore, there is a deviation between the source and fused images to some extent.

2.2. Separable Dictionary Learning Algorithm

To overcome the aforementioned deficiencies of SR for image fusion, the Kronecker-criterion-based separable structure has received significant attention [34]. On the premise of ensuring the quality of image reconstruction, the penalty in l₀-norm with $R(\alpha )={\Vert \alpha \Vert }_0$ and the corresponding objective function of separable dictionary learning [39] is defined as

$$\arg \underset{D_A,S,D_B}{\min }{\Vert S\Vert }_0 \mathrm{such} \mathrm{that} D_{A}S{D}_B^T=Y,S\in R^{n\times n}$$

(2)

where S represents a sparse matrix. As the cross-product of the over-complete dictionary D, the sub-dictionaries $D_A\in R^{\sqrt{m}\times n}$ and $D_B\in R^{\sqrt{m}\times n}$ were obtained by the Kronecher product criterion. For simplicity, we set both sub-dictionaries to the same size.

The steps of the separable dictionary-learning algorithm include sparse coding and dictionary update. The dictionary optimization problems were found using the extensional two-dimensional OMP (2D-OMP) greedy algorithm and the ASeDiL (analytic separable dictionary learning) algorithm to obtain the sparse coefficients and the pre-trained sub-dictionaries $\{D_A,D_B\}$, respectively, using the method described by [40]. The dictionary pre-training model is shown in Figure 1.

The process of sparse coding consists of a four-step iterative loop, including the determination of the most relevant dictionary atom, updates to the support set, updates to the sparse matrix S, and refactoring residual updates. To obtain the sparsest representation under the current dictionary, the objective function of sparse coding is expressed as

$$\arg \underset{S}{\min }{\Vert S\Vert }_0 \mathrm{such} \mathrm{that} {\Vert D_{A}S{D}_B^T-Y\Vert }_2^2<\epsilon$$

(3)

where $\epsilon$ represents the tolerance of reconstruction error, and when ${\Vert D_A{S}_j{D}_B^T-Y_j\Vert }_F^2>\epsilon$, the condition of iterations is terminated.

Combining the constraints with the l₂-norm of the dictionary atoms equaling 1, and with no correlation of atoms in the dictionary, the log function was employed to fit the full rank and the column irrelevance of sub-dictionaries. Then, the objective function of the dictionary update was written as

$$\arg \underset{D_A,D_B}{\min }{\Vert D_{A}S{D}_B^T-Y\Vert }_2^2+\omega [p(D_A)+p(D_B)]+\psi [h(D_A)+h(D_B)]$$

(4)

where $\omega$ and $\psi$ represent the fitting parameters, and $h(D_A){,}_{}p(D_A),$ $h(D_B){,}_{}p(D_B)$ are defined as

$$h(D_A)=-\frac{1}{m\log (m)}\log \det (\frac{1}{n}{D}_A{D}_A^T),p(D_A)=-\log (1-{({(D_A)}^T(D_A))}^2)$$

(5)

$$h(D_B)=-\frac{1}{m\log (m)}\log \det (\frac{1}{n}{D}_B{D}_B^T),p(D_B)=-\log (1-{({(D_B)}^T(D_B))}^2)$$

(6)

By means of geodesics on the Riemannian manifold, the dictionary update adopted the conjugate gradient method to correct the most rapid descent direction of the iteration point of the dictionary update, ensuring the rapid convergence of the cost function, and improved the efficiency of the dictionary update.

With the aforementioned separable structure, the obtained sparse matrix composed of correlation coefficients becomes able characterize more textural and structural information. This can not only increase the dimensions of texture extraction without adding dictionary redundancy, but also ameliorate the accuracy of texture extraction with effective noise suppression performance. Through the above separable dictionary learning algorithm, the pre-trained sub-dictionaries can be obtained. Then, the pre-trained sub-dictionaries will participate in the subsequent transform saliency measure characterization process to extract features from the source images.

3. Proposed Fusion Method

The framework of the proposed method is shown in Figure 2. Supposing that there are K pre-registered source images denoted by $I_k,k\in \{1,2,\dots ,K\}$, the r-th overlapping image patch of the k-th source image $I_k^r$ was obtained through the sliding window technique, and the corresponding sparse coefficient map $S_k^r$ was learned through the sparse coding process in the separable dictionary-learning algorithm with the pre-trained sub-dictionaries $\{D_A,D_B\}$. The proposed SR-based medical image fusion with measurement integrating spatial saliency and transform saliency consisted of the following two steps:

3.1. The Measurement of Activity Level for Fusion

3.1.1. Spatial Saliency by Textural Contrast

In general, the salient area is recognized by the vision system from the retina to the visual cortex. Some of the early information received by the retina is luminance contrast, and orientation contrast in the visual cortex is involved in understanding the context at higher levels. We were inspired to attempt to express spatial saliency by allowing textural contrast to be defined by luminance contrast and orientation contrast.

First of all, the luminance contrast was defined by considering the distinctiveness of the intensity attributes between each pixel and the corresponding image patch [41]. To increase the useful dynamic ranges and to suppress high contrast effectively in the background, the n-th order statistic was applied as

$$L{C}_k^r(x,y)={\left|{\widehat{\mu }}_k^r-\frac{1}{M}{\displaystyle \sum _{(x,y)\in {\Phi }^{\prime }}{I}_k^r(x,y)}\right|}^n$$

(7)

where ${\widehat{\mu }}_k^r$ denotes the mean luminance values over the r-th patch in the k-th source image $I_k^r$, and ${\Phi }^{\prime }$ and M represent a $3\times 3$ neighborhood with pixel $(x,y)$ centered and its size, respectively.

Along with luminance contrast, the local image structure was captured by orientation contrast through a weighted structure tensor. It is worth noting that we focused on weighted gradient information rather than the gradient itself, and this highlights the main features of the source images. The weighted structure tensor [42] was able to effectively summarize the dominant orientation and the energy along this direction based on the weighted gradient field:

$$G_{I_k^r}(x,y)=\left[\begin{array}{cc}{\displaystyle {\sum }_{k=1}^K{\left({\omega }_k(x,y)\frac{\partial I_k^r}{\partial x}\right)}^2}& {\displaystyle {\sum }_{k=1}^K{\omega }_k^2(x,y)\frac{\partial I_k^r}{\partial x}\frac{\partial I_k^r}{\partial y}}\\ {\displaystyle {\sum }_{k=1}^K{\omega }_k^2(x,y)\frac{\partial I_k^r}{\partial x}\frac{\partial I_k^r}{\partial y}}& {\displaystyle {\sum }_{k=1}^K{\left({\omega }_k(x,y)\frac{\partial I_k^r}{\partial y}\right)}^2}\end{array}\right]$$

(8)

where $\partial I_k^r/\partial x$ and $\partial I_k^r/\partial y$ denote the gradients along the x and y directions, respectively, at the given pixel $(x,y)$. The weight function ${\omega }_k(x,y)$ is calculated by

$${\omega }_k(x,y)=\frac{LS{M}_k^r(x,y)}{\sqrt{{\displaystyle {\sum }_{k=1}^K{(LS{M}_k^r(x,y))}^2}}}$$

(9)

where $LS{M}_k^r(x,y)$ represents the local salient metric, which reflects the importance of the pixel $(x,y)$ by computing the sum of intensity around it, and is calculated by

$$LS{M}_k^r(x,y)={\displaystyle \sum _{(x,y)\in {\Phi }^{\prime }}\left(\left|\frac{\partial I_k^r(x,y)}{\partial x}\right|+\left|\frac{\partial I_k^r(x,y)}{\partial y}\right|\right)}$$

(10)

where $|•|$ denotes the absolute value operator. The local salient metric is sensitive to the edges and texture while being insensitive to the flat part. To express the local image structure, the weighted structure tensor as a semi-definite matrix can be decomposed by eigenvalue decomposition as

$$G_{I_k^r}=V\left(\begin{array}{cc}{\beta }_1^2& \\ & {\beta }_2^2\end{array}\right)V^T$$

(11)

The orientation contrast related to the eigenvalues of ${\beta }_1$ and ${\beta }_2$ of this matrix is calculated as

$$O{C}_k^r(x,y)=\sqrt{{({\beta }_1+{\beta }_2)}^2+\eta {({\beta }_1-{\beta }_2)}^2}$$

(12)

where $\eta >-1$. This parameter can determine the relative emphasis of the orientation contrast to the corner-like structures effectively.

Since it is assumed that the salient area contains luminance contrast and orientation contrast, as mentioned, we then attempted to define the texture contrast with two parts:

$$T{C}_k^r(x,y)=L{C}_k^r(x,y)\times O{C}_k^r(x,y)$$

(13)

Here, each part was smoothed by Gaussian filtering, as in [43], and $T{C}_k^r$ was normalized to [0, 255] for gray-scale representation.

3.1.2. Spatial Saliency by Textural Contrast

Compared with the conventional transform-based activity level measure, which uses the l₁-norm to describe the detail information contained in sparse vectors, the SSSF metric [44] is able to highlight more significant coefficients to participate in the composition of activity level measures through the sum of differences in adjacent areas, which is defined as

$$SSS{F}_k^r(x,y)={\displaystyle \sum _{p=-P}^P{\displaystyle \sum _{q=-Q}^Q{[LSS{F}_k^r(x+p,y+q)]}^2}}$$

(14)

where P and Q determine a sliding window, with the size of the sparse matrix equal to the r-th patch in the k-th source image $I_k^r$. The local sparse salient feature (LSSF) metric represents the sparse saliency diversity of adjacent pixels, and is calculated as

$$LSS{F}_k^r(x,y)=\sqrt{{\displaystyle \sum _{(m,n)\in \Phi }{[S_k^r(x,y)-S_k^r(m,n)]}^2}}$$

(15)

where $\Phi$ denotes a square window centered with a certain sparse coefficient that corresponds to pixel $(x,y)$ in the source image patch $I_k^r$.

3.2. Fusion Scheme

Combining the transform saliency and the spatial saliency of the image patch, the proposed activity level measure was defined as

$${\varpi }_k^r(x,y)=SSS{F}_k^r(x,y)\times T{C}_k^r(x,y)$$

(16)

where ${\varpi }_k^r$ is the measurement result of the source image patch $I_k^r$. Then, the maximum weighted activity level measure was used to achieve the fused coefficient map:

$$S_F^r(x,y)=S_{k^{*}}^r(x,y){,}_{}{k}^{*}=\arg {\underset{k}{\max }}_{}[{\varpi }_k^r(x,y)]$$

(17)

Then, the fusion result was obtained by sparse reconstruction as

$$I_F^r=D_A{S}_F^r{D}_B^T$$

(18)

The final fused image $I_F$ was constituted by the overall fused image patches.

4. Experiments

4.1. Experimental Setting

4.1.1. Source Images

In our experiments, three categories of “Acute stroke”, “Hypertensive encephalopathy”, and “Multiple embolic infarctions” from clinical multimodal image pairs in the Whole Brain Atlas Medical Image (WBAMI) database were used as test images, which can be downloaded at: http://www.med.harvard.edu/aanlib/home.htm, accessed on 8 June 2023. The database covers a variety of modal combination types, and it is widely applied for the verification of medical image fusion performance [22,23,24]. The spatial resolution was set to $256\times 256$ for all test images. To make sure the registration was realized, we took the feature-based registration algorithm for each pair, i.e., the method of complementary Harris feature point extraction based on mutual information [45], which has strong robustness and is able to adapt to various image characteristics and variations.

4.1.2. Objective Evaluation Metrics

Since there are limitations of a single objective metric in terms of reflecting the fusion result accurately, the six popular objective metrics, namely, the Xydeas–Petrovic index [46], the structural similarity index Q_S [47], the universal image quality index Q_U [48], the overall image quality index Q₀ [48], the weighted fusion quality index Q_W [49], and the mutual information index Q_NMI [50], were adopted to evaluate the fusion performance in this study. The higher the scores of the above metrics, the better the fusion result of the corresponding fusion method. A classification of these metrics is shown in

Table 1. Classification of different objective assessment metrics.

Category	Metric	Symbol	Description
Textural-feature-preservation-based metrics	Normalized mutual information	QNMI	It measures the mutual information of a fused image and source images.
Edge-dependent-sharpness-based metrics	Normalized weighted performance index	QAB/F	It measures the amount of edge and orientation information of the fused image using the Sobel edge detection operator.
	Overall image quality index	Q0	It evaluates structural distortions in the fused image.
Comprehensive-evaluation-based metrics	Weighted fusion quality index	QW	It values the structural similarity by addressing the coefficient correlation, illumination, and contrast.
	Structural similarity index	QS	It determines the structural similarity by taking comparisons of luminance, contrast, and structure.
	Universal image index	QU	It is designed by modeling image distortion as a combination of the loss of correlation, luminance distortion, and contrast distortion.

.

4.1.3. Methods for Comparison

Since it was inspired by the transform-domain-based method [33] to design the activity level measure and fusion scheme in our method, to carry out a fair and clear comparison, the conventional l₁-norm-based scheme [33] and the sum of sparse salient features (SSSF)-based scheme were used to verify the advantages of the proposed activity level measurement. Meanwhile, in each of the medical image fusion categories, some newly published representative medical image fusion methods, such as LRD [51], NSST-MSMG-PCNN [52], and CSMCA [53], were used for comparison with the proposed method. The competitors adopted the default parameters in the corresponding literature.

4.1.4. Algorithm Parameter Setting

For the proposed method, to obtain the pre-trained sub-dictionaries, we chose 10⁴ patches with sizes of 8 × 8 from different uncorrupted images to be included in the training dataset. Furthermore, the training patches were normalized with a zero mean and unit l₂-norm, and the initial sub-dictionaries were obtained by the MATLAB function randn with normalized columns. Following the experimental setup detailed in [40], the spatial size of the sliding window was set to 8 × 8, the patch-wise step size was set to 1 to keep the shift invariant of SR, the two Kronecker-criterion-based separable dictionaries were set to the same size of 8 × 16, and the tolerance of the reconstruction error $\epsilon$ was set to 0.01.

In addition to the above general settings, variable n and variable $\eta$ were the key parameters affecting the luminance contrast and the orientation contrast, respectively, and the parameters set through quantitative experiments are shown in Figure 3. It can be seen that variable n would affect the luminance contrast and the retention of effective information in subsequent fusion results. On this basis, we set n = 3 as a compromise. With an increase in variable $\eta$, the texture structure of the source image was clearer, and it was conducive for extracting orientation contrast information. On this basis, we set $\eta =0.5$.

4.2. Comparison to Other Fusion Methods

The subjective visual and objective metrics were used to evaluate the proposed method. The comparison experiment contained three categories of clinical multimodal medical images, including “Acute stroke”, with 28 pairs of CT/MR-PD and CT/MR-T2; “Hypertensive encephalopathy”, with 28 pairs of CT/MR-Gad and CT/MR-T2; and “Multiple embolic infarctions”, with 60 pairs of CT/MR-PD, CT/MR-T1, and CT/MR-T2.

4.2.1. Subjective Visual Evaluation

In the experiments using multimodal medical image fusion, CT and MRI image fusion were the most common. This is because the information provided by CT and MRI images can act as a good supplement, while the multimodal combination category can be expanded to other types of fusion, such as the method used in this paper. Figure 4 shows the nine randomly selected groups of multimodal-fused medical images in subjective visual experiments. The first three groups belong to “Acute stroke”, the second three groups belong to “Hypertensive encephalopathy”, and the last three groups belong to “Multiple embolic infarctions”. To more intuitively reflect the superiority of the proposed method, one group showing typical fusion was selected from each of the three WBAMI categories as an example with which to conduct a detailed analysis on the amplification of representative regions, as shown in Figure 5, Figure 6 and Figure 7.

The CT/MR-T2 fusion results and the red box selections of the proposed method and its competitors are shown in Figure 5. The fusion results of LRD and NSST-MSMG-PCNN are blurred since artificial interference is unsuppressed (in Figure 5c,d), while CSMCA, l₁-norm, SSSF, and the proposed method, as SR-based methods, are robust to artificial interference, and the fused edges are more distinct (see in Figure 5e–h). However, the luminance loss of CSMCA caused a reduction in contrast (see in Figure 5e), and CSMCA, l₁-norm, and SSSF showed partial reductions in detail (see in Figure 5e–g). In contrast, more details from the source images were extracted via the proposed method, with artificial interference suppressed effectively (see in Figure 5h).

The CT/MR-T2 fusion results and the red box selections of the proposed method and its competitors are shown in Figure 6. We can clearly see that the results of LRD and NSST-MSMG-PCNN were disturbed by noise (see in Figure 6c,d). CSMCA, l₁-norm, and SSSF lost a significant amount of structural information (see in Figure 6e–g). In contrast, the proposed method performed better in terms of structural integrity and robustness to artificial interference (see in Figure 6h).

The CT/MR-T2 fusion results and the red box selections of the proposed method and competitors are shown in Figure 7. It is clear that artifacts appeared when using the LRD method (see in Figure 7c). NSST-MSMG-PCNN, CSMCA, l₁-norm, and SSSF caused losses of luminance, and all led to partial reductions in detail (see in Figure 7d–g). In contrast, the proposed method was obviously superior to its competitors in terms of luminance and detail retention (see in Figure 7h).

Through these subjective comparison experiments, it was difficult to contain the completed information for SR-based image fusion with a single measurement of the activity level, such as the CSMCA, l₁-norm, or SSSF. Moreover, the proposed method was not only able to retain luminance and detail information from the source images, but also performed better in terms of robustness to artificial interference, keeping the fused edges more distinct. Therefore, the proposed method offered better subjective visual performance than the competitors.

4.2.2. Objective Quality Evaluation

Objective quality evaluation is an important approach with which to evaluate fusion performance.

Table 2. Objective assessment of different fusion methods.

WBAMI	Metric	NSST-
		LRD	MSMG-	CSMCA	l1-norm	SSSF	Proposed
		PCNN
Acute stroke (28 pairs of CT/MR -PD, CT/MR-T2)	QAB/F	0.4821	0.5187	0.5513	0.5863	0.5844	0.5880
	QS	0.7244	0.6972	0.7254	0.7359	0.7366	0.7418
	QU	0.6709	0.4628	0.5862	0.6803	0.6809	0.6866
	Q0	0.3008	0.2984	0.3038	0.3271	0.3270	0.3319
	QW	0.5633	0.5791	0.5873	0.6035	0.6061	0.6090
	QNMI	0.7466	0.6693	0.7097	0.8554	0.8357	0.8827
Hypertensive encephalopathy (28 pairs of CT/MR-Gad, CT/MR-T2)	QAB/F	0.5062	0.5343	0.5840	0.6242	0.6248	0.6290
	QS	0.6974	0.6699	0.7165	0.7144	0.7163	0.7211
	QU	0.6283	0.4506	0.5825	0.6395	0.6413	0.6474
	Q0	0.3152	0.3051	0.3130	0.3540	0.3563	0.3541
	QW	0.5984	0.6254	0.6419	0.6607	0.6671	0.6736
	QNMI	0.6883	0.6240	0.6680	0.7091	0.7040	0.7464
Multiple embolic infarctions (60 pairs of CT/MR -PD, CT/MR-T1, CT/MR-T2)	QAB/F	0.4584	0.5140	0.5545	0.5850	0.5784	0.5840
	QS	0.6893	0.6785	0.7002	0.6939	0.6952	0.7016
	QU	0.6146	0.4438	0.6146	0.6331	0.6343	0.6412
	Q0	0.3211	0.3158	0.3111	0.3449	0.3458	0.3488
	QW	0.5562	0.5851	0.5842	0.5962	0.5977	0.5994
	QNMI	0.6951	0.6327	0.6536	0.7204	0.7095	0.7575

reports the objective assessment results of the proposed method and its competitors. The average scores of all of the test examples from each of the three WBAMI categories were calculated, and the highest value of each row, shown in bold, indicates the best fusion performance. It can be seen that the proposed method performed best in all six metrics in the “Acute stroke” category, which included 28 pairs of multimodal medical images. In the “Hypertensive encephalopathy” category with 28 pairs of multimodal medical images, except for Q₀ ranking second, the other five metrics of the proposed method were the best. In the “Multiple embolic infarctions” category with 60 pairs of multimodal medical images, this metric ranked second, and the other five metrics of the proposed method were the best. On the whole, the average results of the six metrics of the proposed method were the best in the three clinical category experiments. Therefore, based on the above subjective analysis and objective evaluation, the proposed method has considerable advantages over the most recently published methods of LRD, NSST-MSMG-PCNN, and CSMCA.

Furthermore, without changing the fusion framework, the ablation experiments were carried out to verify the universal advantages of the proposed method over the l₁-norm- and SSSF-based schemes, which only considered the transform domain situation of the activity level measurement. Through the six commonly used fusion metrics, the Q_NMI metric of the proposed method had the most obvious advantage over the two experiments in which ablation was competed. This indicates that the proposed new activity level measure plays a significant role in the retention of the textural information of the source images. Furthermore, it is worth noting that the SSSF-based scheme showed slightly significant superiority over the l₁-norm-based scheme in all test examples; this reveals the justification for using SSSF as a substitute for l₁-norm in order to participate in the construction of the activity level measure in the transform domain.

5. Conclusions

In this study, a multi-modal medical image fusion method with Kronecker-criterion-based SR was proposed. The main contribution of the proposed method is summarized in three parts. Firstly, a novel activity level measure integrating spatial saliency and transform saliency was proposed to represent more abundant textural structure features. Secondly, inspired by the formation process of the vision system, the spatial saliency was characterized by the textural contrast consisting of luminance contrast and orientation contrast to induce more highlighted textural information to participate in the fusion process. Thirdly, as a substitution for conventional l₁-norm-based sparse saliency, the sum of the sparse salient features metric characterizes the transform saliency to promote more significant coefficients and to participate in the composition of the activity level measure. The experimental results of different clinical medical image categories demonstrated the effectiveness of the proposed method. Extensive experiments have demonstrated the state-of-the-art performance of the proposed method in terms of visual perception and objective assessment. Taking into account the influence of computational efficiency, some measures can attempt to obtain a more compact and adaptive dictionary, such as by taking source images as the training sample and testing samples simultaneously, and some feature selection rules can be used to exclude unfeatured image patches.