A Hybrid Pansharpening Algorithm of VHR Satellite Images that Employs Injection Gains Based on NDVI to Reduce Computational Costs

Abstract

The objective of this work is to develop an algorithm for pansharpening of very high resolution (VHR) satellite imagery that reduces the spectral distortion of the pansharpened images and enhances their spatial clarity with minimal computational costs. In order to minimize the spectral distortion and computational costs, the global injection gain is transformed to the local injection gains using the normalized difference vegetation index (NDVI), on the assumption that the NDVI are positively or negatively correlated with local injection gains obtained from each band of the satellite data. In addition, the local injection gains are then applied in the hybrid pansharpening algorithm to optimize the spatial clarity. In particular, in the proposed algorithm, a synthetic intensity image is determined using block-based linear regression. In experiments using imagery collected by various satellites, such as KOrea Multi-Purpose SATellite-3 (KOMPSAT-3), KOMPSAT-3A and WorldView-3, the pansharpened results obtained using the proposed Hybrid Pansharpening algorithm using NDVI and based on the spectral mode (HP-NDVI_spectral) provide a better representation of the values of the Erreur Relative Globale Adimensionnelle de Synthèse (ERGAS), the spectral angle mapper (SAM) and the Q4/Q8 than those produced by existing pansharpening algorithms. In terms of spatial quality, the pansharpened images obtained using the proposed pansharpening algorithm based on the spatial mode (HP-NDVI_spatial) have higher average gradient (AG) values than those obtained using existing pansharpening methods. In addition, the computational complexity of our method is similar to that of a pansharpening algorithm that is based on a global injection model, although our methodology has characteristics that are similar to those of a local injection gain-based model that has a very high computational cost. Thus, the quantitative and qualitative assessments presented here indicate that the proposed algorithm can be utilized in various applications that employ spectral information or require high spatial clarity.

1. Introduction

With the development of remote sensing satellites, various algorithms for integrating or fusing optical or synthetic aperture radar (SAR) satellite images have been proposed [1,2,3]. Because various satellite sensors used for remote sensing, including very high resolution (VHR) satellite sensors such as IKONOS, QuickBird, GeoEye, WorldView-2/3 and the Korea Multi-Purpose Satellite (KOMPSAT)-2/3/3A, provide simultaneous multispectral and panchromatic images that have different spatial resolutions, pansharpening algorithms represent an essential element of the utilization of remote sensing data in data fusion frameworks. Such algorithms sharpen the spatial resolution of multispectral images using high spatial resolution panchromatic images obtained by identical satellite sensors [1,4,5]. In particular, pansharpening methods function as pre-processors and are used in various applications, such as visual interpretation, image classification, change detection and digital mapping [5,6,7,8].

Pansharpening algorithms can be classified as multi-resolution analysis (MRA) or component substitution (CS) methods, depending on the characteristics of each algorithm [9]. In MRA-based algorithms, pansharpened images are generated by injecting spatial details found in the original and spatially-reduced panchromatic images into the multispectral image [9,10]. Various multi-resolution decomposition techniques, such as a generalized version of the additive wavelet luminance proportional (AWLP) model that uses the spectral response function (SRF), ridgelets, curvelets and shearlets, have been proposed in order to enhance the quality of pansharpened images [4,11,12,13,14]. Meanwhile, in recent studies of pansharpening algorithms, a version of the generalized Laplacian pyramid (GLP) in which a Gaussian filter is matched with the modulation transfer function (MTF) of the satellite sensor being considered, is referred to as MTF-GLP [15,16]. In performing pansharpening with MTF-GLP, various injection gains are applied [5,17]. In addition, Yang et al. [18] proposed a new pansharpening algorithm based on a matting model and a multiscale transform. Recently, morphological operators have been applied to extract spatial details from panchromatic images [19]. Meanwhile, some researchers have indicated that spatial enhancement using MRA-based pansharpening algorithms that employ decimated analysis may not be satisfactory due to the occurrence of aliasing effects, artifacts and the blurring of textures, although MRA-based algorithms can minimize the spectral distortion of pansharpened images [11,20,21]. This issue is important in that the interpretation of pansharpened images is one of the most representative applications in the field of remote sensing.

Therefore, CS-based pansharpening algorithms have been developed in order to optimize the spatial and spectral quality of pansharpened images. General CS-based algorithms sharpen the spatial resolution of multispectral images through the injection of spatial differences between synthetic intensity images generated from multispectral images, instead of the degradation of panchromatic images. In particular, the development of the fast intensity-hue-saturation (FIHS) fusion method, which can quickly fuse large volumes of remote sensing data and can be extended to four or more spectral bands, has accelerated the advancement of pansharpening technology [22,23]. Subsequently, Dou et al. [24] showed that many pansharpening algorithms can be generalized to the CS framework. Based on the FIHS method and the generalized CS framework, various modifications of the FIHS method have been developed [25,26,27]. Laben and Brower [28] proposed the Gram–Schmidt (GS) method, which has been implemented in the Environment for Visualizing Images (ENVI) software environment. Aiazzi et al. [29] proposed various versions of the GS method that use different means of producing intensity images and experimented with carrying out the pansharpening process using global and local injection models. This study argued that the pansharpening results obtained using a local injection model are of higher quality than those obtained using a global model. However, they demonstrated that the local injection model has a high computational cost compared with the global model [15,29]. Xu et al. [30] suggested the use of a data fitting scheme to minimize the spectral distortion associated with CS-based methods. The band-dependent spatial detail (BDSD) algorithm, which is stated to be the one of most efficient algorithms in [9], generates spectrally-optimized pansharpened images using injection gains. These gains can be applied globally or locally, based on the minimum mean square error (MMSE) joint estimator [31]. In addition, Zhong et al. [32] generalized the BDSD algorithm using a combination of the CS and MRA methods.

The hybrid pansharpening algorithm, which employs two types of spatial detail in its injection model, was developed in order to improve the spatial quality of pansharpened images [33]. The nearest-neighbor diffusion (NNDiffuse) algorithm aims to enhance spatial details while preserving spectral fidelity, and the partial replacement adaptive component substitution (PRACS) method is designed to optimize the preservation of spectral information in pansharpened images through the construction of high- and low-resolution component images [21,34]. In addition, pansharpening algorithms of clustered multispectral and panchromatic images considering mixed pixels is proposed using the fuzzy c-means algorithm [35]. However, the spectral distortion produced by most CS-based algorithms needs to be addressed, because the injection models used to minimize spectral distortion in some CS-based methods cause a loss of spatial clarity during the pansharpening process. In particular, the moving window-based local injection models used in [9,15,33] are not practical in terms of their computational costs, although these methods show good performance in terms of spectral preservation.

Therefore, in this work, we attempted to reduce the computational costs associated with pansharpening through the generation of local injection gains. In particular, the optimal local injection gains for hybrid pansharpening are newly developed based on the normalized difference vegetation index (NDVI), as motivated in [36,37], except that sliding windows and clustering algorithms are not used. The developed local injection gains are optimized through the similarity between the NDVI and the injection gain parameter model of the pansharpening algorithm in order to minimize the spectral distortion of the pansharpened images. In contrast, existing algorithms, which are proposed in [36,37], require the generation of the global injection gain using the NDVI or an image clustering method. In addition, block-based intensity images are applied in the hybrid pansharpening process in order to enhance the spectral and spatial quality of the pansharpened images.

The manuscript is organized as follows. In Section 2, we provide a brief overview of the CS-based pansharpening algorithm and describe its characteristics. We then propose our new pansharpening algorithm, which is intended to reduce computational costs, in Section 3. In Section 4, we compare the quantitative and qualitative quality of the pansharpened images obtained using our algorithm with those obtained using existing state-of-the-art algorithms. Section 5 includes a discussion, followed by the conclusions of the paper.

2. Overview and Characteristics of CS-Based Pansharpening Methods

Some researchers have demonstrated a general pansharpening framework that can be defined by Equation (1) [9,15].

$${\widehat{MS}}_k={\widetilde{MS}}_k+g_k\left(P-I_L\right),k=1,\dots ,N$$

(1)

where $P$ is a panchromatic image with high spatial resolution, $N$ is the number of spectral bands, ${\widehat{MS}}_k$ is a pansharpened image that corresponds to the k-th spectral band, ${\widetilde{MS}}_k$ is an interpolated version of the original multispectral image $M{S}_k$ at the scale of $P$, $g_k$ indicates the injection gains of the k-th spectral band and $I_L$ is a synthetic intensity image with a low spatial resolution that is identical to the spatial resolution of ${\widetilde{MS}}_k$. Generally, the original multispectral image is interpolated to the same pixel size as the panchromatic image using a polynomial interpolator with 23 coefficients [16]. In addition, in pansharpening, CS- and MRA-based algorithms are classified in terms of how $I_L$ is generated, as mentioned in Section 1. $I_L$ is determined through spatial degradation of panchromatic images in MRA-based methods; however, CS-based methods generate $I_L$ using multispectral images with low spatial resolution. In the CS-based algorithms, the combination of multispectral images by using empirical formulas using the SRFs of satellite-based sensors or a multiple linear regression model between ${\widetilde{MS}}_k$ and $P$ is applied for producing $I_L$.

Meanwhile, the determination of $g_k$ is another key step in CS-based pansharpening. Except for empirical estimation of the parameters for each sensor based on trial and error, injection gains are most often extracted using a statistical model. In particular, some researchers have noted that spectral distortion of CS-based pansharpened images can occur due to the use of global injection gain models. As optimal values for different regions that have different spectral characteristics are applied to satellite images, local parameter values based on moving windows about overlapping image blocks can minimize spectral distortion [15,16,31,33]. In addition, due to the high computational costs of moving window-based algorithms, methods that employ segment- and non-overlapping block-based processing have been proposed [38,39].

3. The Proposed Methodology

In this work, we developed a new pansharpening algorithm based on optimal injection gains and the hybrid pansharpening framework. The overall workflow of the proposed pansharpening algorithm is shown as Figure 1. We first extract the optimal local injection gains based on NDVI. In addition, the intensity image, which is based on the non-overlapping block-based algorithm, is then generated. Finally, the injection gains and the intensity image are applied in the hybrid pansharpening framework. The details are as follows.

3.1. Similarities between the NDVI and Injection Gain Parameter Models

In general, in CS-based pansharpening methods, the major reason why spectral distortion occurs is the difference of spectral characteristics between the panchromatic image and each multispectral band. Spectral distortion is caused by mismatches between the dynamic range of the SRF associated with the panchromatic sensor and that of the multispectral sensor. If a satellite image is obtained from a region that includes various land cover types, such as vegetated areas, bodies of water, agricultural areas and buildings, the spectral dissimilarity between panchromatic and multispectral images may be intensified. For the reasons mentioned above, most pansharpening algorithms are evaluated using various satellite images containing different land cover types. Moreover, several methodologies for estimating the injection gain parameter have been proposed that are based on the spectral characteristics of land cover types. The NDVI is a representative index that quantifies the biophysical characteristics of vegetation. Given that major spectral distortion of pansharpened images occurs within vegetated areas, the NDVI may provide an effective means to estimate injection gain parameters. For example, a simple relationship between mean NDVI values and weighting factors that is based on trial and error and 200 IKONOS-2 images was determined in [36]. Moreover, only global injection gains with the same value for all bands were extracted. Xu et al. [37] classified panchromatic and multispectral images into two classes, vegetated and non-vegetated regions, using the k-means algorithm and then applied pansharpening to the classified pixel groups separately. This study indicates that injection gains can be determined according to the spectral characteristics of the land cover types of satellite images. Therefore, in this manuscript, we propose algorithms that identify optimized values of the local gain parameter using NDVI for hybrid pansharpening [40]. This algorithm is novel compared to previous studies in that it extends the global injection gains to local injection gains using NDVI, based on the assumption that the NDVI displays high similarity with the local injection gains obtained from each band of the satellite imagery. In particular, we tried to optimize the local gain parameter through standardization and adjustment of the dynamic range of the NDVI in order to minimize the spectral distortion of the pansharpened image.

First, we analyze the similarity between local injection gains and NDVI. Local injection gains are obtained from the Gram–Schmidt adaptive (GSA) method and the hybrid pansharpening algorithm, which is representative of pansharpening algorithms that employ local window processing to extract local gains [15,33].

Figure 2 represents the NDVI and the local injection gain image from GSA and hybrid pansharpening. As shown in Figure 2b,c,e, for the blue band, the values of the local injection gain are low within the vegetated areas, where the NDVI values are high. On the other hand, the values of the local injection gain are relatively high in the unvegetated areas, which contain low NDVI values. In contrast, in the case of the near-infrared (NIR) band, the local injection gain values display a positive correlation with the NDVI, as shown in Figure 2b,d,f. To enable a quantitative comparison between local injection gains and NDVI values, the correlation coefficient between the NDVI values and the local injection gains of GSA and hybrid pansharpening was calculated. As shown in

Table 1. Correlation coefficients between an NDVI and representative local injection gain models.

Correlation Coefficient	Local Injection Gain by GSA	Local Injection Gain by Hybrid Pansharpening
Band 1 (blue)	−0.3120	−0.4460
Band 2 (green)	−0.4114	−0.5104
Band 3 (red)	−0.3363	−0.4772
Band 4 (NIR)	0.4694	0.5095

, the injection gains of the blue, green and red bands display weak (0.2–0.4) to moderate (0.4–0.6) correlations with the NDVI. Moreover, the injection gains of the NIR band display a positive and moderate correlation value. Therefore, the local injection gain of the visible bands can be transformed based on this positive correlation, whereas that of the NIR band can be transformed based on the negative correlation. In particular, it appears that the local injection gains can be obtained through transformation of the NDVI without performing image processing based on overlapping image blocks and moving windows, which are used to extract the local injection gains in general pansharpening methods.

3.2. Optimization of Pansharpening Parameter Using the NDVI

In general, terms, the NDVI can be defined using Equation (2) [41]:

$$NDVI=\frac{NIR-red}{NIR+red}$$

(2)

where $NIR$ and $red$ are the reflectance values of the NIR and red bands of ${\widetilde{MS}}_k$. However, in our algorithm, it is possible to use a digital number (DN) in Equation (2) instead of a reflectance value because we only consider the NDVI in terms of the relative spectral characteristics of vegetated and non-vegetated areas. After determining the NDVI, it should be adjusted so that it contains values that are similar to those of the general local injection gains. To meet this requirement, the NDVI is reconstructed using the dynamic range and a histogram of the NDVI values obtained using existing global injection gain algorithms. Initially, $I_L$ is determined through linear regression between ${\tilde{P}}_L$ and ${\widetilde{MS}}_k$, where ${\tilde{P}}_L$ indicates a panchromatic image with low spatial resolution to which the Starck and Murtagh filter has been applied [42]. The initial injection gains $g_k^G$ of band $k$ can then be calculated using Equation (3) [33]:

$$g_k^G=\sqrt{\frac{\sigma \left({\widetilde{MS}}_k\right)}{\sigma \left(I_L\right)}\times {\left(S_k\right)}^3}$$

(3)

where $S_k$ is the correlation coefficient between the high-frequency information of $I_L$ and ${\widetilde{MS}}_k$ obtained by Laplacian filtering, and $\mathsf{\sigma }\left(A\right)$ is the standard deviation of image $A$. Considering that the global gain is similar to the mean of the local injection gains, the local injection gains $g_k$ based on NDVI values are expanded using Equation (4).

$$g_k={\left(-1\right)}^a\times NDVI+\overline{NDVI}+g_k^G$$

(4)

where $\overline{NDVI}$ is the mean value of the overall NDVI values, and $a$ indicates the adjustment variable for the dynamic range of the local injection gains. NDVI is negatively correlated with the local injection gains of the visible wavelength, and NDVI and the local injection gains of the NIR band are positively correlated. Therefore, the sign of NDVI values is transformed based on $a$ using Equation (5).

$$a=\{\begin{array}{c}1,if\mathrm{corr}\left({\widetilde{MS}}_k,NDVI\right)<0\\ 0,if\mathrm{corr}\left({\widetilde{MS}}_k,NDVI\right)>0\end{array}$$

(5)

where $\mathrm{corr}\left(\mathrm{A},\mathrm{B}\right)$ is the correlation coefficient between $A$ and $B$. Finally, the value of $g_k$ is revised to avoid over- or under-estimation of $g_k$, as described by Equation (6).

$$g_{k_{\left(i,j\right)}}=\{\begin{array}{c}0,if{g}_{k_{\left(i,j\right)}}<0\\ 1.5\times g_k^G,if{g}_{k_{\left(i,j\right)}}>1.5\times g_k^G\end{array}$$

(6)

Here, $g_{k_{\left(i,j\right)}}$ indicates the local injection value at an image position of $\left(i,j\right)$. In Equation (6), the minimum value of $g_k$ is set to zero because injection gains should be greater than zero. In addition, then, the maximum value of $g_k$ is considered to be $1.5\times g_k^G$ in order to prevent the over-injection of spatial details. Figure 2 shows $g_k$ corresponding to the blue and NIR bands. As shown in Figure 2 and Figure 3, the $g_k$ image obtained using our algorithm is qualitatively similar to the general local injection gains obtained by convolution. Using Equations (3)–(6), the optimal local injection gains according to the spectral characteristics of each wavelength range and land cover type were determined without the use of overlapping blocks-based convolution.

3.3. Construction of a Modified Hybrid Pansharpening Model

To apply $g_k$, we adapted the hybrid pansharpening framework, instead of the general CS-based pansharpening algorithm that is based on Equation (1). This revised framework can be defined as Equation (7) [33]:

$${\widehat{MS}}_k={\widetilde{MS}}_k+g_k\left(H_k+\alpha H_k^{\prime }\right)={\widetilde{MS}}_k+g_k\left[\left(P-I_L^B\right)+\alpha \left({\nabla }^2\left(P-I_L^B\right)\right)\right]$$

(7)

where ${\widehat{MS}}_k$ represents the pansharpened images, $H_k$ and $H_k^{\prime }$ indicate the spatial details obtained using the primary and secondary high-frequency information and $\mathsf{\alpha }$ is a decision parameter for selecting the hybrid pansharpening mode. First, $H_k$ is the difference between $P$ and the non-overlapping block-based intensity image $I_L^B$. Let $P$, which has a pixel size of $\mathrm{R}\times \mathrm{C}$, be partitioned into $\frac{\mathrm{R}\times \mathrm{C}}{S^2}$ blocks. Here, $\mathrm{S}$ is the block size. The intensity image corresponding to the b-th block can be obtained using Equation (8):

$$I_L^{B\left(b\right)}={\displaystyle \sum }_{k=1}^N{\omega }_k^{B\left(b\right)}{\widetilde{MS}}_k^{B\left(b\right)}$$

(8)

where ${\widetilde{MS}}_k^{B\left(b\right)}$ indicates the multispectral image (k-th spectral band) corresponding to the b-th block. Using a linear regression between $P_L^{B\left(b\right)}$ and ${\widetilde{MS}}_k^{B\left(b\right)}$, the weight parameter ${\omega }_k^{B\left(b\right)}$ can be determined. In addition, for pansharpening, $I_L^B$ is integrated using Equation (9).

$$I_L^B=\left\{I_L^{B\left(b\right)}\right\},b=1,\dots \dots \frac{\mathrm{R}\times \mathrm{C}}{S^2}$$

(9)

In Equations (8) and (9), the value of the block size $\mathrm{S}$ is important in determining the performance of our algorithm. If $\mathrm{S}$ is similar to the original image size, the spectral residual between $I_L^B$ and $P$ can be increased, whereas a small block size would cause spatial decline. If $\mathrm{S}$ is set to one, the pansharpening framework is identical to MRA-based pansharpening because $I_L^B$ with $\mathrm{S}=1$ is equal to ${\tilde{P}}_L$. For our algorithm, $\mathrm{S}=256$ has been determined to be the optimal value in terms of spatial and spectral quality through trial and error using various types of satellite images. This value maintains the spatial quality of CS-based pansharpening and minimizes the spectral distortion caused by the spectral residual in linear regression. Based on an intensity image obtained with Equations (8) and (9), the spatial details $H_k$ and $H_k^{\prime }$ can be extracted. Meanwhile, $H_k^{\prime }$ can be obtained from the application of a Laplacian filter, such as that shown in Equation (10), to $H_k$, which is defined as the differencing between the panchromatic and intensity images.

$${\nabla }^2=\left[\begin{array}{ccc}-1& -1& -1\\ -1& 8& -1\\ -1& -1& -1\end{array}\right]$$

(10)

$H_k$ plays a role that is similar to that of high boost filtering in order to improve the level of spatial detail and maximize the spectral clarity of the pansharpened image [33]. Meanwhile, the injection of spatial detail may cause spectral distortion, due to the tradeoff between spectral and spatial quality. Therefore, in this work, we set the decision parameter $\mathsf{\alpha }$ in order to control the spectral and spatial quality of the pansharpened image. Our algorithm can be classified as having a spectral and spatial mode, and the mode used is determined by the value of $\mathsf{\alpha }$. If $\mathsf{\alpha }$ is set to zero, the pansharpening framework is identical to the original CS-based pansharpening framework of Equation (1), and $H_k^{\prime }$ is not used in the pansharpening. This means that only primary high-frequency information is applied. In this manuscript, this mode is called the spectral mode, which does not reflect the additional spatial details contained in $H_k^{\prime }$. The spatial mode considers the injection of secondary high-frequency information $H_k^{\prime }$ using Equation (11) [33]:

$$\mathsf{\alpha }=\frac{\mathsf{\sigma }\left(H_k\right)}{2\mathsf{\sigma }\left(H_k^{\prime }\right)}$$

(11)

The spatial mode is designed to generate a pansharpened image that contains high quality spatial details when compared to pansharpened images generated using the spectral mode. Therefore, users can choose the spatial or spectral mode, depending on the type of application.

4. Experimental Results

4.1. Quality Assessment of Pansharpened Images

In evaluating the quality of pansharpened images, the method used to determine the reference image for comparison with the pansharpened image is an important issue because the original multispectral and panchromatic images do not include reference pansharpened datasets. To solve this problem, the synthesis and consistency properties have been used to conduct quality assessments of pansharpened images [9,43]. In the case of the synthesis property, the pansharpening framework is applied to degraded multispectral and panchromatic images, as shown in Figure 4. Thus, the spatial and spectral resolutions of the pansharpened image obtained from the degraded datasets will be identical to those of the original multispectral image. Therefore, the multispectral image in the original dataset can be used as a reference.

However, evaluations performed using the synthesis property may not guarantee identical spectral and spatial quality when the pansharpening algorithm is applied to the original datasets. Therefore, some researchers have proposed the quality estimation methodology, which evaluates the consistency property [43]. To evaluate the consistency property, pansharpened images are produced from the original multispectral and panchromatic datasets, as shown in Figure 5. Subsequently, the spatial resolution of the pansharpened image is degraded to the resolution of the original multispectral image. The spatial quality is assessed using the original panchromatic image and the pansharpened image.

Meanwhile, the quality no reference (QNR) metric, which is a representative quality index used in assessing pansharpened images, examines the cross similarity between each pair of pansharpened images [44]. However, QNR is not strongly correlated with the results of quality evaluations based on the synthesis and consistency properties [43]. In particular, the QNR index does not efficiently reflect spatial quality; artifacts and saturation of spatial details are visible in some pansharpened images with high QNR values. Therefore, in this work, we employ the consistency property to evaluate the quality of pansharpened images.

To estimate the quality of pansharpened images based on the consistency property, we employ the Erreur Relative Globale Adimensionnelle de Synthèse (ERGAS), the spectral angle mapper (SAM) algorithm, the Q4/Q8 metric, the spatial correlation coefficient (sCC) and the average gradient (AG) [9,12,43,45,46,47]. Particularly when assessing spatial quality, visual analysis was prioritized over quantitative evaluation [9,43]. The quality indices used in this experiment are specified as follows.

1. ERGAS

This metric calculates the global relative spectral/spatial error of pansharpened images [9,43]. It is given by Equation (12).

$$\mathrm{ERGAS}=\frac{100}{\mathrm{R}}\sqrt{\frac{1}{\mathrm{N}}{\displaystyle \sum }_{\mathrm{k}=1}^{\mathrm{N}}{\left(\frac{\mathrm{RMSE}\left({\mathrm{I}}_{\mathrm{k}},{\mathrm{J}}_{\mathrm{k}}\right)}{\mathsf{\mu }\left({\mathrm{I}}_{\mathrm{k}}\right)}\right)}^2}$$

(12)

where ${\mathrm{I}}_{\mathrm{k}}$ and ${\mathrm{J}}_{\mathrm{k}}$ are the reference multispectral and pansharpened images, respectively, and $\mathrm{R}$ is the spatial resolution ratio. The closer the ERGAS value is to zero, the less distorted the pansharpened image is.

2. SAM

This metric calculates the average difference in angle between the corresponding pixels of the reference and pansharpened images [9,43]. It is defined as Equation (13).

$$\mathrm{SAM}=\arccos \left(\frac{\langle {\mathrm{I}}_{\left\{\mathrm{k}\right\}},{\mathrm{J}}_{\left\{\mathrm{k}\right\}}\rangle }{\Vert {\mathrm{I}}_{\left\{\mathrm{k}\right\}}\Vert \Vert {\mathrm{J}}_{\left\{\mathrm{k}\right\}}\Vert }\right)$$

(13)

where ${\mathrm{I}}_{\left\{\mathrm{k}\right\}}$ indicates a pixel vector of image $\mathrm{I}$ in the k-th band. The SAM index approaches zero when the pansharpened image is similar to the reference image.

3. Q4/Q8

Wang and Bovik proposed the Q-index [45]. It quantifies three factors, loss of correlation, luminance distortion and contrast distortion, using Equation (14).

$$\mathrm{Q}=\frac{{\sigma }_{IJ}}{{\sigma }_I{\sigma }_J}\frac{2\overline{IJ}}{{\left(\overline{I}\right)}^2+{\left(J\right)}^2}\frac{2{\sigma }_I{\sigma }_J}{\left({\sigma }_I^2+{\sigma }_J^2\right)}$$

(14)

where ${\sigma }_{IJ}$ is the covariance of $I$ and $J$. The Q-index has a range of $\left[-1,1\right]$ and the optimal value of the Q-index is one. The Q4 and Q8 metrics are hypercomplex vector extensions of the Q-index that can be applied to multispectral images with $2^n$ bands, where $n$ is an integer [46]. In the case of the Q4 index, each pixel in a multispectral image with four bands is modeled as a quaternion, whereas each pixel in multispectral images with eight bands, such as those produced by WorldView-2 and -3, is modeled as an octonion.

4. sCC

This metric is based on the spatial similarity between reference and pansharpened images. The correlation coefficient between the spatial details of the reference and pansharpened images, as determined using the Laplacian filter of Equation (10), is used to measure sCC [12]. The closer the sCC is to one, the greater the spatial similarity of the pansharpened image and the reference dataset is. In this manuscript, we extracted the sCC using the original panchromatic images as reference datasets.

5. AG

This metric is defined as the amount of high-frequency information in the pansharpened image. The AG reflects the spatial sharpness in terms of the spatial difference between the pixels in the pansharpened image. It is calculated using Equation (15) [33,47,48].

$$\mathrm{AG}=\frac{1}{N}{\displaystyle \sum }_{k=1}^N\left(\frac{1}{R\times C}{\displaystyle \sum }_{x=1}^{R-1}{\displaystyle \sum }_1^{C-1}\sqrt{\frac{{\left(\frac{\partial F_k\left(x,y\right)}{\partial x}\right)}^2+{\left(\frac{\partial F_k\left(x,y\right)}{\partial y}\right)}^2}{2}}\right)$$

(15)

where F is a pansharpened image of pixel size $\mathrm{R}\times \mathrm{C}$, and $\frac{\partial F_k\left(x,y\right)}{\partial x}$ indicates the pixel differential of F in the k-th band at position (x, y). High AG values indicate pansharpened images with high spatial sharpness. However, high AG values may affect the degree of spectral distortion because there is a tradeoff between spectral and spatial distortion.

4.2. Test Data

In this work, datasets obtained from three different sensors are applied. Their characteristics are summarized in

Table 2. Specifications of the satellite sensors and datasets used in this study. KOMPSAT, Korean Multi-Purpose Satellite.

Sensor		KOMPSAT-3	KOMPSAT-3A	WorldView-3
location		Sejong Korea	Sejong Korea	Beolgyo Korea
date		25 March 2015	28 October 2015	26 July 2015
multispectral resolution/size		2.8 m	2.2 m	1.2 m
		2048 × 2048	2048 × 2048	2048 × 2048
panchromatic resolution/size		0.7 m	0.55 m	0.3 m
		8192 × 8192	8192 × 8192	8192 × 8192
wavelength	panchromatic	450–900 nm		448–808 nm
	coastal blue	-		397–454 nm
	blue	450–520 nm		445–517 nm
	green	520–600 nm		507–586 nm
	yellow	-		580–629 nm
	red	630–690 nm		626–696 nm
	red edge	-		698–749 nm
	NIR1	760–900 nm		765–899 nm
	NIR2	-		857–1039 nm

, and an overview of the datasets is provided in Figure 6. The details of the test data are as follows.

1. KOMPSAT-3

The Korean Multi-Purpose Satellite (KOMPSAT)-3 was launched by the Korea Aerospace Research Institute (KARI) on 17 May 2012. The site of the experiment includes various land cover types, such as buildings and mountainous and vegetated areas of the Sejong region in Korea. The spatial resolution of KOMPSAT-3 is 0.7 m (panchromatic) at nadir.

2. KOMPSAT-3A

KOMPSAT-3A was launched by KARI on 26 May 2015. The characteristics of KOMPSAT-3A are similar to those of KOMPSAT-3. However, the images collected by KOMPSAT-3A have a higher spatial resolution than the images acquired by the KOMPSAT-3 sensor, as shown in Table 2. As with the site examined using KOMPSAT-3 data, the site examined using KOMPSAT-3A data is located in the Sejong region in Korea. This site contains mainly mountainous areas with urban areas. Therefore, it can be used for the evaluation of pansharpening algorithms using data from different sensors.

3. WorldView-3

The site examined using the WorldView-3 sensor is a rural area that contains farmland and wetlands within the Beolgyo region in Korea. The WorldView-3 sensor provides one panchromatic and eight multispectral bands. Using this dataset, we assessed the performance of the algorithm when various multispectral bands are sharpened.

4.3. Experimental Results and Analysis

To evaluate the performance of the proposed pansharpening algorithm, the pansharpening results obtained using our method are compared to several representative pansharpening algorithms. Primarily CS-based algorithms are selected because our algorithm is based on CS for the injection of spatial details, and the tendency of the evaluation indices resulting from CS-based algorithms is different from that of MRA-based algorithms. An overview of the algorithms employed follows.

• EXP: expanded multispectral image. EXP is interpolated to the image size of the pansharpened image using a polynomial kernel with 23 coefficients [16].
• GIHS: generalized intensity-hue-saturation image fusion [21].
• GSA: Gram–Schmidt adaptive [29]. GSA is subdivided into GSA_G and GSA_L, depending on whether local injection gains are used. GSA_G uses global injection gains, whereas GSA_L utilizes local injection gains determined using overlapping image blocks.
• BDSD: band-dependent spatial detail with local parameter estimation [9,31]. The BDSD algorithm is generally applied using non-overlapping image blocks. In this manuscript, image blocks with a window size of 256, which are the same size as those applied in our algorithm, are used.
• NNDiffuse: nearest-neighbor diffusion-based pansharpening algorithm [34]. This approach has been implemented in the ENVI software package.

4.3.1. Quantitative Analysis

Table 3. Comparative pansharpening result corresponding to each satellite sensor. ERGAS: erreur relative globale adimensionnelle de synthèse; SAM, the spectral angle mapper; sCC, spatial correlation coefficient; AG, average gradient; EXP: expanded multispectral image; GIHS: generalized intensity-hue-saturation; GSA: Gram–Schmidt adaptive; BDSD: band-dependent spatial detail; NNDiffuse: nearest-neighbor diffusion; HP-NDVI_spectral: hybrid pansharpening using NDVI based on the spectral mode; HP-NDVI_spatial: hybrid pansharpening using NDVI based on the spatial mode.

Dataset	Algorithm	ERGAS	SAM	Q4/Q8	sCC	AG
KOMPSAT-3	EXP	0.8118	0.6409	0.9870	0.3036	63.82
	GIHS	1.6505	1.0255	0.9088	0.9886	110.48
	GSAG	1.0023	0.7336	0.9671	0.9858	113.68
	GSAL	1.1029	0.9480	0.9608	0.9176	103.75
	BDSD	0.9532	0.8223	0.9791	0.9439	117.73
	NNDiffuse	1.1865	0.8244	0.9398	0.9366	119.04
	HP-NDVIspectral	0.8766	0.6627	0.9828	0.9737	101.67
	HP-NDVIspatial	0.8926	0.6694	0.9816	0.9471	146.95
KOMPSAT-3A	EXP	0.5288	0.5147	0.9885	0.1027	49.07
	GIHS	1.8083	1.0259	0.7142	0.9952	167.29
	GSAG	0.9206	0.7343	0.9121	0.9968	175.17
	GSAL	1.0500	1.1713	0.9301	0.7616	149.74
	BDSD	0.8479	0.9122	0.9695	0.8688	159.34
	NNDiffuse	3.0593	0.7103	0.5551	0.9451	216.92
	HP-NDVIspectral	0.5355	0.5221	0.9769	0.9471	106.40
	HP-NDVIspatial	0.5574	0.5298	0.9744	0.9382	168.62
WorldView-3	EXP	0.7030	0.6709	0.9845	0.0526	4.88
	GIHS	3.2953	1.7482	0.6327	0.9784	9.00
	GSAG	1.7884	0.9160	0.8865	0.9787	8.93
	GSAL	2.8504	2.3264	0.8070	0.6309	9.24
	BDSD	1.7385	1.5836	0.9248	0.7629	8.77
	NNDiffuse	2.7860	0.9571	0.6412	0.8481	9.91
	HP-NDVIspectral	1.5924	0.8933	0.9474	0.8257	7.27
	HP-NDVIspatial	1.7045	0.9205	0.9403	0.8331	11.21

represents the quantitative results of the spatial and spectral quality indices based on the consistency property corresponding to the datasets obtained using the three different sensors. In Table 3, HP-NDVI_spectral indicates hybrid pansharpening using NDVI based on the spectral mode, and HP-NDVI_spatial is based on the proposed spatial mode.

As shown in Table 3, the pansharpened images obtained using the GIHS algorithm generally show poor values of the quantitative indices; however, the results of using GIHS to pansharpen the KOMPSAT-3 data are associated with the highest sCC value. The results of applying GSA_G to the KOMPSAT-3A and WorldView-3 data are associated with the highest sCC values. Moreover, the spectral quality of GSA_G is associated with low ERGAS and SAM values compared to those of GSA_L, although GSA_L employs local injection gains. The result of pansharpening the KOMPSAT-3A data using GSA_L shows a higher Q4 value than that obtained using GSA_G. This result indicates that the results of the GSA algorithm based on global and local injection gains may depend on the spectral and spatial characteristics of the original data. The pansharpening result obtained using the BDSD algorithm shows the best ERGAS, SAM and Q4/Q8 values compared to those obtained using GIHS, GSA and NNDiffuse. A large amount of spectral distortion occurs in the case of NNDiffuse, whereas the AG value associated with NNDiffuse is very large. This result indicates that NNDiffuse may frequently produce over-sharpened images during the injection of high-frequency information. The pansharpened images obtained using our algorithms, HP-NDVI_spectral and HP-NDVI_spatial, have lower ERGAS and SAM values than those obtained with other pansharpening methods for all of the datasets examined. The Q4 and Q8 values obtained from pansharpened images produced by our algorithms are also higher than those associated with any of the other algorithms. The pansharpened images obtained using HP-NDVI_spatial tend to have lower spectral quality indices compared to those of HP-NDVI_spectral, whereas the highest AG values are associated with the application of HP-NDVI_spatial to the KOMPSAT-3 and Worldview-3 datasets (though not the KOMPSAT-3A dataset). However, in terms of the pansharpening results obtained using the KOMPSAT-3A dataset, the higher AG value obtained using NNDiffuse is due to the over-injection of high-frequency information. This result indicates that HP-NDVI_spectral has advantages in minimizing spectral distortion, and HP-NDVI_spatial is efficient in producing spatial sharpness. In particular, HP-NDVI_spatial showed the best spectral quality and AG values compared to the results from GSA_G, GSA_L, BDSD and NNDiffuse, though not HP-NDVI_spectral.

4.3.2. Qualitative Analysis

Figure 7, Figure 8 and Figure 9 represent the visual results for detailed $400\times 400$ subregions of the pansharpened images obtained from each satellite sensor. The pansharpening results obtained using GIHS show the largest spectral distortion in vegetated areas and forests (Figure 7c and Figure 8c). However, the sharpened images obtained using GIHS show good visual quality in terms of spatial sharpness, whereas most of the vegetated area is over-sharpened. The GSA-based technique shows relatively high spectral distortion. In addition, in the case of the result obtained using GSA_L, the spatial clarity of the fused image decreases due to the error in the boundary portion of the local process, as shown in Figure 8e and Figure 9e. In the case of NNDiffuse, the spatial clarity was represented effectively, but spectral distortion was also generated. Visual inspection shows that the pansharpened image obtained using NNDiffuse displays greatly improved spatial sharpness; however, the spectral distortion caused by the tradeoff between spectral and spatial quality has also increased substantially. As shown in Figure 8g in particular, the pansharpened image obtained using the KOMPSAT-3A dataset shows that excessive high-frequency information has been injected, which indicates that NNDiffuse cannot be applied independently to the data from the various sensors. The pansharpening results from BDSD confirm that, although less spectral distortion occurs overall, the spatial clarity in some areas has decreased. This observation indicates that the injection gains based on non-overlapping block processing differ significantly from each other in the BDSD algorithm. Therefore, some differences in pixel values occur at the boundaries between the blocks (Figure 8f). In addition, as seen in the results obtained using the WorldView-3 dataset, blurring or saturation occurs in some areas (Figure 9f). Although the method proposed in this manuscript is a block-based injection gain technique that employs NDVI, the errors associated with spectral distortion occurring at the boundaries of the blocks are relatively small compared with those seen in the results of pansharpening performed using BDSD. It can be confirmed that, of the proposed methods, HP-NDVI_spectral causes the least spectral distortion, and the results of this method show spectral fidelity to the original multispectral image and have characteristics that are similar to those of the original image. In particular, the results of the HP-NDVI_spatial method have the best spatial clarity, compared to the existing algorithms (Figure 7i, Figure 8i and Figure 9i). Meanwhile, the sCC values associated with HP-NDVI_spatial are relatively low because the pansharpened images obtained using HP-NDVI_spatial are based on the injection of secondary high-frequency information, which is not highly correlated with the high-frequency information contained in the original panchromatic image. However, the pansharpened image obtained using HP-NDVI_spatial has the best visual clarity. In addition, then, HP-NDVI_spatial has the best AG value among the results obtained using the KOMPSAT-3 and Worldview-3 datasets.

5. Discussion

This manuscript proposes a new pansharpening algorithm that applies local injection models to NDVI to minimize spectral distortion while maintaining spatial clarity. Quantitative and qualitative assessments of the results of the proposed algorithm yield the following findings.

(1) In quantitative assessments performed by applying spectral quality indices to the pansharpened images, the pansharpened images produced by HP-NDVI_spectral are associated with the lowest ERGAS and SAM values and the highest Q4/Q8 values in all of the datasets obtained using three different satellite sensors. In addition, except for the results obtained using HP-NDVI_spectral, the pansharpening results obtained using HP-NDVI_spatial show better spectral quality than those obtained using existing state-of-the-art pansharpening algorithms. Therefore, these results indicate that the proposed technique for determining injection gains based on NDVI minimizes the spectral distortion of pansharpened images.

(2) In the evaluation of spatial quality, the sCC values of the pansharpened images obtained using HP-NDVI_spatial are lower than those of the pansharpening results obtained using HP-NDVI_spectral; however, the pansharpened images obtained using HP-NDVI_spatial display the best spatial clarity when inspected visually. In addition, the pansharpened images obtained using HP-NDVI_spatial have higher sCC values compared to those obtained using GSA_L, BDSD and NNDiffuse. This result indicates that the AG index provides a more efficient assessment of the spatial quality and clarity of pansharpened images than the sCC index, because the sCC index evaluates only the similarity between the original panchromatic and pansharpened images. However, the spatial characteristics of each multispectral band used in pansharpening differ from those of the panchromatic image. Therefore, in the experimental analysis, the AG index and visual estimation provide more effective evaluations of the spatial quality and clarity of pansharpened images. From this point of view, we conclude that the proposed hybrid pansharpening algorithm based on the spatial mode increases the spatial clarity while maintaining the spectral characteristics of HP-NDVI_spectral.

(3) The algorithm proposed in this work are intended to generate pansharpened images with optimal spectral and spatial quality and to reduce the computational costs associated with image processing. For example, because algorithms that employ overlapping block-based processes, such as hybrid pansharpening and GSA_L, have very high computational costs, they are difficult to apply to large volumes of satellite imagery. Therefore, we compare the computational complexity of GSA_G and GSA_L, which are representative algorithms that are based on global and local injection gains, in order to evaluate the efficiency of the extraction of local injection gains using our algorithms. The experiment was carried out using a 64 bit quad-core CPU (3.50 GHz processor). As seen in

Table 4. Comparison of the computational cost of the proposed algorithm with those of GSA-based algorithms.

Pansharpening Algorithm	Computational Cost (s)
GSAG	240 s
GSAL	25,246 s
HP-NDVIspatial	245 s

, HP-NDVI_spatial has a computational cost that is similar to that of GSA_G, which is based on global injection gains, and its processing time is much faster than that of GSA_L, which is based on local injection gains.

Therefore, it is confirmed that the processing speed associated with the use of local injection gains in our proposed algorithms, which employ NDVI, is similar to those of pansharpening algorithms based on global injection gains.

(4) In pansharpening algorithms based on non-overlapping block processing, such as BDSD, some spatial and spectral distortion may occur at block boundaries. Our proposed algorithms employ non-overlapping block processing to generate optimal intensity images; however, it has been confirmed that the proposed method does not cause spectral and spatial distortion at the block boundaries when compared with the results obtained using the BDSD method.

(5) Based on the above discussion, we make the following suggestions. The proposed pansharpening algorithms can be effectively applied to data collected by various satellite sensors with various spectral and spatial resolutions. In particular, based on quantitative and qualitative assessments, HP-NDVI_spectral can be utilized in applications that employ spectral information, such as image classification and change detection, whereas HP-NDVI_spatial can be applied to image interpretation and feature extraction.

6. Conclusions

In this work, a new pansharpening algorithm that applies local injection models to NDVI to minimize spectral distortion while maintaining spatial clarity and that have a low computational cost is developed. This algorithm is based on the assumption that the general local injection gains obtained from each band of satellite imagery are positively or negatively correlated with the NDVI. Two variants of this algorithm, which are named HP-NDVI_spectral and HP-NDVI_spatial, according to their spectral and spatial characteristics, are applied to the satellite imagery obtained using various sensors. The results of these experiments show that the HP-NDVI_spectral algorithm displays the least spectral distortion. In addition, the HP-NDVI_spatial algorithm displays the best spectral and spatial quality compared to existing algorithms, whereas the computational cost of the proposed algorithms is similar to that of traditional pansharpening algorithms based on global injection models. Therefore, spectral-based analysis involving image classification can be performed using pansharpened images obtained with HP-NDVI_spectral, whereas image interpretation can be performed using pansharpened images obtained with HP-NDVI_spatial.