Sharpening of Worldview-3 Satellite Images by Generating Optimal High-Spatial-Resolution Images

Abstract

Compared to using images in the visible and near-infrared (VNIR) wavelength range only, remotely sensed satellite imagery from the spectral wavelengths of both VNIR and shortwave infrared (SWIR), such as Sentinel-2A and Worldview-3, is more effective for analyzing various types of information for tasks such as land cover mapping, environmental monitoring and land use change detection. In this manuscript, a new sharpening technique to enhance the spatial resolution of Worldview-3 satellite imagery with various spatial and spectral resolutions is proposed. Selected and synthesized band schemes were used to produce optimal panchromatic images; then, sharpened images were generated by applying the Gram-Schmidt adaptive (GSA) and Gram-Schmidt 2 (GS2) techniques, which are component substitution (CS)- and multiresolution analysis (MRA)-based algorithms, respectively. In addition, to minimize the spectral distortion of the initial sharpened image, a postprocessing methodology for spectral distortion reduction was developed. Qualitative and quantitative evaluation of the sharpened images showed that the pansharpening performance using the GS2 technique based on the selected band scheme and spectral distortion reduction was the best. To confirm the usability of the SWIR band, supervised classification based on machine learning was performed on the pansharpened images obtained by applying the technique proposed in this study and on the pansharpened images obtained by the VNIR bands only. The classification accuracy of the results using SWIR bands was higher than that of VNIR bands only. In particular, it was confirmed that the accuracy of the classification of artificial facilities known to be effective for SWIR bands was greatly improved.

1. Introduction

The development of satellite sensors with wide wavelength ranges has engendered numerous remote sensing fields. In particular, the development of satellite sensors with high spatial resolution, such as those on the IKONOS, QuickBird, Geoeye, Worldview-2/3, and KOMPSAT-2/3/3A satellites, has promoted the utilization of such images in various fields, including surveying, defense, environmental monitoring, and urban analysis [1]. Generally, high-spatial-resolution satellite sensors provide panchromatic images with high spatial resolution and multispectral images with low spatial resolution. However, it is difficult to identify or extract various small objects in urban regions using multispectral images with low spatial resolution. Consequently, the need for multispectral images with higher spatial resolutions has increased, and efforts to produce high-spatial-resolution multispectral images that fuse panchromatic images and multispectral images have developed various algorithms to accomplish these tasks. Such techniques are termed pansharpening or image fusion in the remote sensing field [2,3]. The pansharpening technique can be divided into component substitution (CS)- or multiresolution analysis (MRA)-based techniques according to the method used to extract the spatial details [4]. The CS-based technique produces synthetic intensity images similar to the spectral characteristics of panchromatic images through weighted combination with multispectral images. Carper et al. [5] proposed an intensity–hue–saturation (IHS) pansharpening technique using an inverse transform method after first transforming the red, green, and blue (RGB) components of images into the IHS color system and replacing the intensity band with panchromatic image. However, applications for this technique were limited due to the subsequent development of sensors that could acquire multispectral images of more than four bands [6]. Other techniques were proposed that transformed images into other spectral systems through various spectral transformation methods, such as principal component analysis (PCA) and Gram-Schmidt (GS), and the images that showed the greatest similarity to panchromatic images were applied in pansharpening [7,8].

In contrast to CS-based techniques, which produce optimal synthetic images using multispectral images, the MRA-based technique employs high-resolution panchromatic images whose resolution is reduced in the optimal panchromatic images. MRA-based techniques can be divided into wavelet-based image filtering and image pyramid utilization. The discrete wavelet transform (DWT) and à trous wavelet transform (ATWT) are typical wavelet-based pansharpening techniques that achieve fusion by extracting high-frequency information from panchromatic images [9,10]. Because the DWT and ATWT techniques maintain the size of the decomposed image, they can create a sharpened image with higher spatial quality than the traditional wavelet-based techniques. Pansharpening techniques have recently been developed that employ image pyramids to process panchromatic images whose spatial resolution has been degraded. In particular, techniques that utilize the Laplacian pyramid have been proposed [11,12]. However, in these techniques, spatial distortions such as blurring or aliasing occur to a greater degree than when using CS-based fusion techniques. To reduce spatial distortion, Garzelli et al. [13] and Aiazzi et al. [4] proposed an ATWT context-based decision (CBD) technique and a generalized Laplacian pyramid and context-based decision (GLP-CBD) technique.

The high-spatial-resolution multispectral images produced through pansharpening techniques distort the spatial and spectral information of the original image. This distortion occurs because the bands of the panchromatic images and those of the multispectral images have different spectral characteristics; the resulting sharpened images with distorted spectral information are difficult to utilize in analyses that rely on spectral characteristics, such as land cover mapping [14]. Recently, images with more than three spatial resolutions or images of spectral wavelengths outside the visible and near-infrared (VNIR) spectral range can be acquired in contrast to only high-spatial-resolution satellite images consisting of panchromatic and multispectral images. Compared to using only images in the visible and near-infrared (VNIR) wavelength range, remotely sensed satellite imagery from the spectral wavelengths of VNIR and shortwave infrared (SWIR) is more effective for analyzing various types of information for tasks such as land cover mapping, cloud detection, environmental monitoring, and change detection. Generally, pansharpening techniques assume that panchromatic images include the spectral wavelength range of multispectral images. However, some panchromatic images of satellite sensors may not include the spectral wavelength range of multispectral images, such as Worldview-2 and WorldView-3. And some satellite sensors may not obtain panchromatic images with high spatial resolution, such as Sentinel-2A. To sharpen images that do not provide panchromatic images, images with a band that contains the highest spatial resolution should be utilized along with panchromatic images, or the optimal combination should be selected to produce an image that contains the best spectral characteristics of the image to be sharpened when many images are available that all have equally high spatial resolution.

The above pansharpening techniques were developed for use with typical high-spatial-resolution satellite images that provide panchromatic images; however, they cannot be used with satellite images that are not panchromatic images. To improve the spatial resolution of satellite images that do not provide panchromatic images, the ability to produce optimal panchromatic images is essential; several related studies have been undertaken in recent years. To improve the spatial resolution of hyperspectral images that contained no panchromatic images, Selva et al. [15] proposed both a selected band scheme and a synthesized band scheme to produce optimal panchromatic images. The selected band scheme used correlation analysis, while the synthesized band scheme relied on multiple regression analysis. Vaiopoulos and Karantzalos [16] proposed a pansharpening technique that replaced the calculated mean results of images with a 10 m spatial resolution with optimal panchromatic images to improve the spatial resolution of Sentinel-2A satellite images and compared the images resulting from various pansharpening techniques. Wang et al. [17] sharpened Sentinel-2A satellite images by applying the selected and synthesized band schemes proposed by Selva et al. [15] and conducted a quantitative evaluation of the pansharpening results according to the algorithms used. Du et al. [18] aimed to develop the modified normalized difference water index (MNDW), a normalized index of surface ground water, using Sentinel-2A satellite images. To do this, they improved the spatial resolution of the shortwave infrared (SWIR) bands by utilizing the near-infrared (NIR) bands of Sentinel-2A along with panchromatic images. Belfiore et al. [19] sharpened the visible and near-infrared (VNIR) bands of Worldview-3 satellite images using various pansharpening techniques and compared them. Kwan et al. [20] proposed three pansharpening methods that utilized a hypersharpening technique to improve the spatial resolution of Worldview-3 satellite images.

However, few studies have been conducted regarding improvements to the spatial resolution of Worldview-3 satellite images. In particular, very few studies have been conducted on the SWIR bands. Thus, this study aimed to develop a technique that could fuse 1.24 m spatial resolution VNIR images with 7.5 m spatial resolution SWIR images from Worldview-3 satellite images of various spatial and spectral resolutions into images with 0.31 m spatial resolution. In this manuscript, we propose a pansharpening technique for enhancing the spatial resolution of VNIR and SWIR bands while minimizing the spectral distortion of sharpened images. First, optimal panchromatic images were generated through a combination of high-spatial-resolution images and then applied in the sharpening process of SWIR bands based on the hypersharpening algorithm using the selected or synthesized scheme of Selva et al. [15] and Park et al. [21]. Then, to minimize the spectral distortion of the initial sharpened image, a postprocessing methodology for spectral distortion reduction was developed. To prove the performance of the proposed pansharpening technique, its results were compared with the results of existing applicable pansharpening techniques. Then, to confirm the usability of the proposed technique, accuracy was evaluated by applying the supervised classification.

The remainder of this paper is organized as follows. Section 2 summarizes the theories of the selected band scheme and the synthesized band scheme, which are pansharpening techniques based on the CS and MRA algorithms, evaluation indexes for quantitative evaluation, and theories about the spectral distortion reduction technique. Section 3 provides an analysis and discussion of the experimental results. Conclusions are presented in Section 4.

2. Materials and Methods

2.1. Specification and Experimental Data of Worldview-3 Satellite Images

The Worldview-3 satellite sensor provides a panchromatic image with a 0.3 m spatial resolution, eight bands with a 1.2 m VNIR wavelength range, and eight SWIR bands with a 7.2 m wavelength range. Although Worldview-3 provides the same VNIR and SWIR bands as Sentinel-2A, it provides images with higher spatial resolution, which not only is an advantage for environmental monitoring and change detection but also allows the analysis of small objects in urban areas.

Table 1. Specification of Worldview-3 satellite imagery.

Band		Central Wavelength (nm)	Bandwidth (nm)	Spatial Resolution (m)
Panchromatic		625	350	0.31
VNIR	Band 1-Coastal	425	50	1.24
	Band 2-Blue	480	60
	Band 3-Green	545	70
	Band 4-Yellow	605	40
	Band 5-Red	660	60
	Band 6-Red Edge	725	40
	Band 7-Near-IR 1	833	125
	Band 8-Near-IR 2	950	180
SWIR	Band 9-SWIR 1	1210	30	7.5
	Band 10-SWIR 2	1570	40
	Band 11-SWIR 3	1660	40
	Band 12-SWIR 4	1730	40
	Band 13-SWIR 5	2165	40
	Band 14-SWIR 6	2205	40
	Band 15-SWIR 7	2260	50
	Band 16-SWIR 8	2330	70

presents spectral information by the specifications of the Worldview-3 satellite image.

In this manuscript, the digital number (DN) value of Worldview-3 was used for pansharpening. To apply the pansharpening technique to Worldview-3 satellite images, panchromatic images are converted to 0.3 m spatial resolution, and the VNIR and SWIR bands are geometrically corrected by 1.2 m and 7.2 m, respectively. The images used in the study involved two (2.88 × 2.88 km) regions, each consisting of 9600 × 9600 pixels based on the 0.3 m spatial resolution band. To verify the pansharpening performance with regard to various types of land cover, one image over Dongducheon, Gyeonggi-do, Korea (site 1) was selected to represent urban areas, and another image over Namhae-gun, Gyeongsangnam-do, Korea (site 2) was selected to represent natural features, such as vegetation, agricultural land, and water systems. Figure 1 shows two images of the experimental regions used to sharpen Worldview-3 satellite images.

2.2. Optimal High-Spatial-Resolution Image Generation

Pansharpening involves the injection of high-frequency information from panchromatic images into multispectral images and can be defined by Equation (1) [21,22,23]:

$${\widehat{MS}}_n={\widetilde{MS}}_n+g_n\left(P-I_L\right), n=1,\dots ,N$$

(1)

where ${\widehat{MS}}_n$ is the sharpened multispectral image of the nth band, ${\widetilde{MS}}_n$ is the interpolated image of the multispectral image on the scale of $P$, $g_n$ is the vector of injection gains, $P$ is the panchromatic image with a high spatial resolution, $I_L$ is the synthetic intensity image with a low spatial resolution, and $N$ is the number of spectral bands.

The quality of the sharpened image varies according to the method used to calculate the injection gain $g_n$, and there is a trade-off relationship between spatial and spectral accuracy. Pansharpening techniques can be divided into CS- and MRA-based techniques based on the method used to produce the low-spatial-resolution image $I_L$ [24,25]. However, to sharpen satellite images that do not provide panchromatic images, it is necessary to produce artificial panchromatic images. These images can be produced using the selected and synthesized band schemes proposed by Selva et al. [15].

2.2.1. Selected Band Scheme

In the selected band scheme, the high-spatial-resolution band whose spectral characteristics are most similar to that of the low-spatial-resolution band to be sharpened is defined as the artificial high-spatial-resolution band to produce an optimal panchromatic image. To find this optimal high-spatial-resolution band, a correlation analysis is conducted, and the high-spatial-resolution image with the highest correlation is selected. To perform the correlation analysis, high-spatial-resolution images must be transformed into images with the same number of pixels as the low-spatial-resolution images by applying the modulation transfer function (MTF) filter; then, the similarity between the pixels of the two images is evaluated. The optimal panchromatic image is selected through the correlation analysis shown in Equations (2) and (3) [15,21]:

$$\overline{n}=\arg ma{x}_n\mathrm{corr}\left(HR{B}_n^L,LR{B}_m\right)$$

(2)

$$P_m^{sel}={\displaystyle \sum }_{n=1}^N{\alpha }_{m_n}HR{B}_n, \mathrm{where} {\alpha }_{m_n}=\{\begin{array}{c}1, if n=\overline{n}\\ 0, otherwise\end{array}$$

(3)

In Equation (9), $HR{B}_n^L$ refers to the $n$th high-spatial-resolution image whose spatial resolution is transformed into the number of pixels of the low-spatial-resolution image, and $LR{B}_m$ refers to the $m$th low-spatial-resolution image. In Equation (10), $P_m^{sel}$ refers to the $m$th high-spatial-resolution band that has the highest correlation with the $n$th low-spatial-resolution band.

The selected high-spatial-resolution band is utilized along with the panchromatic image during pansharpening, and weighted mean or multiple regression analysis is applied when the CS-based technique is applied, thereby producing $I_L$. When the MRA-based technique is applied, the images whose spatial resolution is reduced by applying the MTF filter may be utilized as $I_L$. Pansharpening is performed using the optimal panchromatic image $P_m^{sel}$ and the optimal low-spatial-resolution image $I_L$ produced in the selected band scheme via Equation (1). The injection gain $g_n$ is calculated and applied according to the CS- and MRA-based technique used.

2.2.2. Synthesized Band Scheme

The synthesized band scheme is a method for producing the optimal panchromatic image that has spectral characteristics similar to those of the low-spatial-resolution bands with which it will be sharpened [15]. In the selected band scheme, the existing high-spatial-resolution bands are defined as optimal panchromatic images for pansharpening, whereas multiple regression analysis is applied to the low- and high-spatial-resolution bands that will be sharpened, thereby producing new images that have similar spectral characteristics to the low-spatial-resolution band in the synthesized band scheme [15,21].

In the first phase of the synthesized band scheme, to produce optimal panchromatic images, the high-spatial-resolution bands are transformed to have the same number of pixels as the low-spatial-resolution bands with which they will be sharpened. Then, multiple regression analysis is applied to the low-spatial-resolution images to calculate the regression coefficient before sharpening them with the transformed bands. The regression coefficient calculation is presented in Equation (4):

$$LR{B}_m={\omega }_{m_0}+{\displaystyle \sum }_{n=1}^N{\omega }_{m_n}HR{B}_n^L$$

(4)

where ${\omega }_{m_0}$ refers to the regression coefficient calculated through the multiple regression analysis and ${\omega }_{m_n}$ refers to the regression coefficient that corresponds to the $n$th panchromatic image.

In the second phase, optimal panchromatic images are produced that have similar spectral characteristics to those of the low-spatial-resolution bands with which they will be sharpened by applying the regression coefficient calculated through the multiple regression analysis to the original high-spatial-resolution bands. The process that produces the optimal high-spatial-resolution image is defined by Equation (5):

$$P_m^{syn}={\omega }_{m_0}+{\displaystyle \sum }_{n=1}^N{\omega }_{m_n}HR{B}_n$$

(5)

where $P_m^{syn}$ refers to the optimal panchromatic image that has spectral characteristics similar to that of the $m$th low-spatial-resolution band generated through the regression coefficient. $I_L$ is generated from the produced optimal panchromatic image $P_m^{syn}$ and the low-spatial-resolution bands by applying Equation (1). The injection gain $g_n$ is determined and calculated according to the pansharpening technique used, as with the selected band scheme.

2.3. Proposed Pansharpening Technique

This study proposes a pansharpening technique to improve the spatial resolution of Worldview-3 satellite images that is performed in two phases [21]. The first phase sharpens the SWIR bands with 7.2 m spatial resolution using VNIR bands with 1.2 m spatial resolution, and the second phase sharpens the Worldview-3 bands with 1.2 m spatial resolution using panchromatic images with 0.3 m spatial resolution. The flowchart in Figure 2 describes the pansharpening technique proposed in this study and used for Worldview-3 satellite images.

2.3.1. Sharpening of SWIR Bands with 7.2 m Spatial Resolution

The first phase of the pansharpening technique for Worldview-3 satellite images aims to improve the spatial resolution of SWIR bands with a 7.2 m spatial resolution using VNIR bands with a 1.2 m spatial resolution. Because panchromatic images could not be utilized in this phase, optimal panchromatic images were produced using the selected and synthesized band schemes. To do this, the MTF filter was applied to the VNIR bands first to produce $VNI{R}^L$, which was then transformed into the number of pixels of the SWIR band.

A correlation analysis was conducted to generate an optimal panchromatic image through the selected band scheme using the SWIR band, which will be sharpened, and $VNI{R}^L$. The $VNI{R}^L$ band that showed the highest correlation with the SWIR band was set as the optimal panchromatic image, and the pansharpening technique was applied. The process to generate an optimal panchromatic image through the selected band scheme is presented in Equations (6) and (7):

$$\overline{n}=\arg ma{x}_n\mathrm{corr}\left(VNI{R}_n^L,SWI{R}_n\right), t=1,\dots ,8$$

(6)

$$P_n^{sel}={\displaystyle \sum }_{n=1}^8{\alpha }_{n}VNI{R}_n, \mathrm{where} {\alpha }_n=\{\begin{array}{c}1, if n=\overline{n}\\ 0, otherwise\end{array}$$

(7)

where $n$ refers to the number of bands in VNIR and SWIR; $SWI{R}_n$ refers to the $n$th SWIR band, which will be sharpened; $VNI{R}_n^L$ refers to the $n$th VNIR band, which is transformed into the number of pixels of the SWIR band; and $P_n^{sel}$ refers to the optimal panchromatic image generated through the selected band scheme.

To produce an optimal panchromatic image that has a spectral characteristic similar to that of a SWIR band to be sharpened, in the synthesized band scheme, multiple regression analysis was conducted with $VNI{R}^L$ to obtain the regression coefficient. This coefficient was then applied to the VNIR band again to produce the optimal panchromatic image $P^{syn}$. The process to generate an optimal panchromatic image through the synthesized band scheme is presented in Equations (8) and (9).

$$SWI{R}_n={\omega }_0+{\displaystyle \sum }_{n=1}^8{\omega }_{n}VNI{R}_n^L$$

(8)

$$P_n^{syn}={\omega }_0+{\displaystyle \sum }_{n=1}^8{\omega }_{n}VNI{R}_n$$

(9)

SWIR band sharpening was performed using the optimal panchromatic image generated through the selected and synthesized band schemes. Because the pansharpening technique of Worldview-3 satellite images proposed in this study was performed in two phases, spectral information distortion could be accumulated. Thus, GS2, which is a pansharpening technique that generates relatively little spectral distortion, was chosen in this study. The image whose spatial resolution was reduced by applying the MTF filter to the optimal panchromatic image and generated in the selected and synthesized band schemes was utilized as the optimal low-spatial-resolution image $I_{L_n}$ to which the GS2 technique was applied. The injection gain $g_n$ was calculated using the covariance of the optimal low-spatial-resolution image $I_{L_n}$ and SWIR bands and the variances of $I_{L_n}$, as presented in Equation (10):

$$g_n=\frac{cov\left(I_{L_n},M{S}_n\right)}{var\left(I_{L_n}\right)}$$

(10)

The GS2 technique process of the SWIR bands using the generated $I_{L_n}$ and $g_n$ is presented in Equation (11):

$${\widehat{SWIR}}_n={\widetilde{SWIR}}_n+g_n\left(P-I_{L_n}\right), n=1,\dots ,8$$

(11)

2.3.2. Sharpening of VNIR Bands with 1.2 m Spatial Resolution

In the second phase of the proposed pansharpening technique, the VNIR bands and SWIR bands with 1.2 m spatial resolution produced in the first phase were sharpened with the panchromatic image, which had a 0.3 m spatial resolution. To do this, the VNIR bands with 1.2 m spatial resolution and the SWIR bands were stacked. The stacking of the two bands is defined by Equation (12):

$$VNI{R}^{\prime }=\left[VNIR, \widehat{SWIR}\right]$$

(12)

After the bands were stacked, general pansharpening using panchromatic images was conducted using the CS-based GSA technique and the MRA-based GS2 technique. For the GSA technique, multiple regression analysis was applied to a panchromatic image that was transformed into the number of pixels of the VNIR image and the $VNI{R}^{\prime }$ bands to calculate the regression coefficient to generate the optimal low-spatial-resolution image $I_L$ using Equations (13) and (14):

$$P_L={\omega }_0+{\displaystyle \sum }_{n=1}^{16}{\omega }_{n}VNI{R}_n^{\prime }$$

(13)

$$I_L={\omega }_0+{\displaystyle \sum }_{n=1}^{16}{\omega }_{n}VNI{R}_n^{\prime }$$

(14)

To apply the GSA technique, the injection gain $g_n$ was generated through the covariance of $I_L$ and $VNI{R}^{\prime }$ and the variance of $I_L$, as presented in Equation (15):

$$g_n=\frac{cov\left(I_L,VNI{R}^{\prime }\right)}{var\left(I_L\right)}$$

(15)

The optimal low-spatial-resolution image $I_L$ used in the GS2 technique can utilize images whose spatial resolution has been reduced by applying the MTF filter to the panchromatic image. The injection gain $g_n$ is calculated using Equation (22) in the same way as the GSA technique. The result of applying the GSA and GS2 techniques to the Worldview-3 satellite images is defined by Equation (16).

$$\widehat{VNIR\prime }=\widetilde{VNIR\prime }+g\left(P-I_L\right)$$

(16)

The final sharpened image was produced by applying the spectral distortion reduction technique using the sharpened and original images generated through the technique proposed in this study. After applying the existing pansharpening techniques, the results were used in comparative evaluations to assess the performance of the sharpening results of the Worldview-3 satellite images. The existing method sharpens the VNIR and SWIR bands using panchromatic images by applying the GSA and GS2 algorithms.

2.4. Spectral Distortion Reduction Method of Sharpened Images

Generally, high-spatial-resolution satellite images provide panchromatic images and multispectral images that have a fourfold difference in spatial resolution. Because the existing pansharpening techniques were developed for high-spatial-resolution satellite images whose spatial resolution differs by fourfold, they may introduce additional spectral information distortion when applied to Worldview-3 satellite images, which have greater differences in spatial resolution. Thus, this study aimed to develop and apply a technique that could reduce the spectral distortion based on the spectral difference between the final sharpened image and the original image. Figure 3 shows a flowchart describing the spectral distortion reduction method.

The spectral distortion reduction method has two phases: The first phase generates images with reduced distortion, and the second phase produces the final sharpened images by injecting high-frequency information. The first phase of the spectral distortion reduction method is performed based on the difference between the sharpened image and the original image. The sharpened image is transformed into an image with the same number of pixels as the original image, and an image that has the distortion information is produced through the difference between the two images. The image containing the distortion information is defined by Equation (17):

$$D_L=SWIR-{\widehat{SWIR}}_L$$

(17)

In Equation (24), $SWIR$ refers to the original SWIR image, ${\widehat{SWIR}}_L$ refers to the sharpened SWIR image converted into the size of the original SWIR image, and $D_L$ refers to the image that has the distortion information with size of SWIR image.

The generated image containing the distortion information is then transformed into an image with the same number of pixels as the sharpened image, thereby creating the image $DR$, which is used to remove distortion from the sharpened image through differencing with the original sharpened image. The distortion-removed image $DR$ is generated via Equation (18):

$$DR=\widehat{SWIR}-D$$

(18)

where $D$ is the resized version of $D_L$, which is the resampled image converted into the size of $\widehat{SWIR}$. Because the spectral distortion-removed image generated in the first phase is produced based on image differencing, spatial information such as boundaries between objects in the image is likely to be distorted. In the second phase, high-frequency information is extracted from the original sharpened image and injected into the spectral distortion-removed image, thereby producing the final spectral distortion-removed-sharpened image. The method for extracting the high-frequency information from the original sharpened image is similar to the MRA-based technique, in which the result of applying the MTF filter to the original sharpened image is subtracted from the original sharpened image to extract the high-frequency information. This extraction is defined in Equation (19):

$$HF=\widehat{SWIR}-{\widehat{(SWIR}}_L{)}_H$$

(19)

In Equation (19), ${\widehat{(SWIR}}_L{)}_H$ refers to the SWIR image converted into the size of the sharpened image after the MTF filter is applied to the original sharpened image, and $HF$ refers to the high-frequency information extracted through differencing between the original sharpened image and ${\widehat{(SWIR}}_L{)}_H$.

.

Finally, the extracted high-frequency information is injected into the spectral distortion-removed image, thereby producing the final sharpened image $\widehat{\widehat{SWIR}}$. The final sharpened image can be calculated using Equation (20):

$$\widehat{\widehat{SWIR}}=DR+HF$$

(20)

2.5. Index for the Quality Evaluation of Sharpened Images

2.5.1. Erreur Relative Globale Adimensionnelle de Synthèse (ERGAS)

In this study, an evaluation index was applied to perform a quantitative comparative analysis of the spectral and spatial information contained in the original and sharpened images. ERGAS is an evaluation index that measures the quality of spectral information [26]. ERGAS quantifies the size difference between two vectors; values approaching zero indicate greater similarity in the spectral characteristics of the original and sharpened images. ERGAS is calculated as shown in Equation (21).

$$\mathrm{ERGAS}=100\frac{h}{l}\sqrt{\frac{1}{K}{\displaystyle \sum }_{l=1}^L\left(\frac{RMS{E}^2\left(i\right)}{MEAN\left(i\right)}\right)}$$

(21)

where $h$ refers to the spatial resolution of the sharpened image, $l$ is the spatial resolution of the original image, $K$ is the number of bands in the sharpened image, $MEAN\left(i\right)$ is the $l-$th band’s mean value, and $RMSE\left(i\right)$ refers to the mean square error between the original and sharpened images. $RMSE\left(i\right)$ can be calculated using Equation (22):

$$\mathit{RMSE}=\frac{1}{M\times N}\sqrt{{\displaystyle \sum }_{i,j=1}^{MN}{\left(\widehat{M{S}^L}\left(i,j\right)-MS\left(i,j\right)\right)}^2}$$

(22)

where $M\times N$ refers to the image size, $\widehat{M{S}^L}\left(i,j\right)$ is the $i,j$th pixel value of the sharpened image, and $MS\left(i,j\right)$ refers to the $i,j$th pixel value of the original image.

2.5.2. Spectral Angle Mapper (SAM)

The SAM is an evaluation index for spectral information distortion that calculates the angle of pixels between the sharpened and original images to quantify the angle difference between two vectors [27]. Values approaching zero indicate greater similarity between the original and sharpened images. The SAM is defined by Equation (23):

$$\mathrm{SAM}(v,\widehat{v}) = \arccos (\frac{\langle v,\widehat{v}\rangle }{{\Vert v\Vert }_2{\Vert \widehat{v}\Vert }_2})$$

(23)

where $v$ refers to the spectral vector of the original image’s pixel, $\widehat{v}$ refers to the spectral vector of the sharpened image’s pixel, $\langle v,\widehat{v}\rangle$ refers to the inner product of the two vectors, and ${\Vert v\Vert }_2$ and ${\Vert \widehat{v}\Vert }_2$ refer to the sizes of the two vectors.

2.5.3. Universal Image Quality Index (UIQI)

The UIQI is an index of similarity between two images that correlates luminosity and intensity distortions between two images [28]. Values approaching one indicate a higher quality of spectral information. The UIQI is defined by Equation (24):

$$\mathrm{UIQI}\left(x,y\right)=\frac{4{\sigma }_{xy}\overline{x}\overline{y}}{\left({\sigma }_x^2+{\sigma }_y^2\right)\left({\overline{x}}^2+{\overline{y}}^2\right)}$$

(24)

where $x$ and $y$ refer to the original and sharpened images; ${\sigma }_{xy}$ refers to the covariance of $x$ and $y$; ${\sigma }_x$ and ${\sigma }_y$ refer to the variances of $x$ and $y$, respectively; and $\overline{x}$ and $\overline{y}$ refer to the means of $x$ and $y$, respectively.

2.5.4. Spatial Correlation Coefficient (sCC)

This study also used the sCC index, which evaluates the similarity of spatial information between panchromatic and sharpened images. The sCC applies the Laplacian filter to two images and calculates the correlation between the results. Values approaching one indicate greater similarity in the spatial information of the two images [29].

3. Results and Discussion

3.1. Quality Evaluation of Sharpened Image

To evaluate the quality of the sharpened images, sharpened images generated by applying existing pansharpening techniques were utilized for comparative evaluations. Existing pansharpening techniques employing general pansharpening techniques were used to fuse the VNIR and SWIR bands using panchromatic images of Worldview-3 satellite images. In the comparative evaluation method, both a quantitative evaluation utilizing an evaluation index that quantifies spectral and spatial qualities and a qualitative evaluation performed through image interpretation were conducted simultaneously.

Table 2. Quantitative evaluation results of sharpened Worldview-3 satellite images.

Site 1
Quality Index	Spatial Resolution	Existing Technique		ProposedTechnique
		GSA	GS2	Selected BandScheme		Synthesized BandScheme
				GSA	GS2	GSA	GS2
ERGAS	1.2 m (VNIR)	1.2972	0.8981	0.8219	0.6688	0.8296	0.6688
	7.2 m (SWIR)	1.7000	0.7183	0.7589	0.4413	0.6932	0.4410
SAM	1.2 m (VNIR)	1.3672	1.2246	0.9155	0.8950	0.9160	0.8950
	7.2 m (SWIR)	4.9748	4.6347	2.7408	1.3620	1.8703	1.2305
UIQI	1.2 m (VNIR)	0.9656	0.9764	0.9823	0.9859	0.9822	0.9859
	7.2 m (SWIR)	0.7748	0.9522	0.9290	0.9777	0.9385	0.9774
sCC	1.2 m (VNIR)	0.9883	0.9855	0.9821	0.9784	0.9822	0.9784
	7.2 m (SWIR)	0.9992	0.9998	0.9990	0.9906	0.9081	0.9104
Site 2
Quality Index	Spatial Resolution	Existing Technique		ProposedTechnique
		GSA	GS2	Selected BandScheme		Synthesized BandScheme
				GSA	GS2	GSA	GS2
ERGAS	1.2 m (VNIR)	1.0183	0.7854	0.6574	0.5732	0.6587	0.5732
	7.2 m (SWIR)	1.7766	0.8019	0.5451	0.3472	0.5678	0.3460
SAM	1.2 m (VNIR)	1.0950	1.0582	0.7785	0.7745	0.7774	0.7745
	7.2 m (SWIR)	6.6940	5.4152	2.0588	0.8097	1.6268	0.7674
UIQI	1.2 m (VNIR)	0.9559	0.9688	0.9763	0.9803	0.9763	0.9803
	7.2 m (SWIR)	0.7759	0.9511	0.9428	0.9802	0.9369	0.9803
sCC	1.2 m (VNIR)	0.9816	0.9787	0.9724	0.9688	0.9724	0.9688
	7.2 m (SWIR)	0.9983	0.9997	0.9884	0.9905	0.9564	0.9625

and Figure 4 and Figure 5 present the quantitative evaluation results and sharpened images applying the existing and proposed pansharpening techniques to two sites.

For the quantitative evaluation of the results of applying the proposed and existing pansharpening techniques, the results of the two sites were similar. The quality of the spectral information of the sharpened images is calculated to be improved by applying the proposed technique rather than the existing technique. Among the proposed schemes, the sharpening quality of the selected band scheme and synthesized band scheme is similar. It was calculated that the spectral quality improved by GS2, which is an MRA-based technique, compared to GSA, which is a CS-based technique.

The results of the qualitative evaluation of sharpened images through visual inspection were different from those of quantitative evaluation. The existing pansharpening technique was interpreted to reflect the spatial characteristics of the panchromatic image as well as the quantitative evaluation effectively, but the spectral distortion was more than that of the original SWIR image. In addition, the quantitative evaluation results showed that the sharpened image quality applied with the synthesized band scheme was excellent, but the results of visual reading showed that various spatial distortion occurred in the results of applying the synthesized band scheme. These distortions are considered to be caused by inconsistent homogeneity of object such as buildings and roads in the process synthesizing the bands. Among the techniques proposed in this study, the sharpened images from the selected band scheme showed high spectral and spatial quality. In particular, the quantitative evaluation was calculated to show lower spatial quality than the existing pansharpening technique. However, the visual inspections were interpreted to have similar spatial clarity, and objects such as traffic, buildings, and vegetation could be identified. On the other hand, the results of applying the synthesized band scheme tended to differ from those of quantitative evaluation. As a result of using the synthesized band scheme, the results of the image inspection had different spectral characteristics from the result of applying the selected band scheme, and the spatial information of the panchromatic image was not reflected properly. In terms of the pansharpening technique applied in this study, it is confirmed that the spectral distortion is lower when applying GS2 than when applying GSA. In particular, spectral distortion occurring in the roof of the building is decreased as GS2 is applied. As a result of the above quantitative and qualitative analysis, it is considered that the selected band scheme and the GS2 technique have the highest performance among the techniques proposed in this study. In addition, in order to analyze the efficiency of the proposed algorithm for reducing the spectral distortion, images before and after applying the spectral distortion reduction technique was compared. As shown in Figure 6, the proposed technique can be reduced the global spectral distortion while maintaining the spatial clarity of the pansharpened image.

To further verify the performance of each proposed method, the spectral profile of eight SWIR bands of the original images and that of the sharpened images are compared, as shown in Figure 7. Spectral profiles were generated using the average values of 9 pixels of 3×3 for vegetation and building objects. Similar to the outcome of the quantitative and qualitative evaluation, applying the selected band scheme and GS2 technique yields results that are most similar to the spectral information of the original images.

3.2. Usability Verification of the Sharpened Image

Supervised classification was performed to verify the usability of the sharpened image produced by the method proposed in this study, and the supervised classification technique applied a support vector machine (SVM), one of the machine learning techniques. For SVM classification, 16 bands produced by the proposed method and 8 VNIR bands produced by the existing sharpening method were applied, and quantitative and qualitative comparison accuracy evaluations were performed. Training and reference data for classification were acquired from images, and classification classes were selected as building roofs, roads, vegetation, bare soil, water bodies, and cultivation facilities. In particular, the building roof was classified into three types according to color.

Table 3. Number of pixels in the training and reference data.

Class	Site 1		Site 2
	Training Data	Reference Data	Training Data	Reference Data
Blue Roof	521	1017	588	1045
Green Roof	568	1091	556	1073
Black Roof	575	1074	514	1069
Road	522	1127	545	1120
Vegetation	563	1036	530	1128
Bare Soil	521	1101	584	1095
Water Bodies	525	1024	576	1063
Cultivation Facility	541	1107	534	1023

shows the number of pixels of training and reference data, and Figure 8 shows the results of SVM classification.

The classification results using 16 bands and 8 bands were quantitatively and qualitatively compared and evaluated. As a result of qualitative evaluation through visual reading, it was read that the classification result to which SWIR was additionally applied showed improved performance for both study areas. In particular, it was confirmed that the classification accuracy of buildings, roads, and cultivation facilities, which are artificial structures, is improved. The quantitative evaluation results showed similar results to the qualitative evaluation.

Table 4. Results of the quantitative accuracy evaluation.

Class	Site 1		Site 2
	8 Bands	16 Bands	8 Bands	16 Bands
Blue Roof	100%	100	99.90%	100%
Green Roof	84.21%	93.49	99.85%	90.73%
Black Roof	96.35%	96.59	87.67%	96.88%
Road	77.05%	95.72	92.96%	92.57%
Vegetation	100%	100	82.07%	85.83%
Bare Soil	93.11%	93.08	99.01%	96.66%
Water Bodies	98.59%	94.75	99.17%	98.37%
Cultivation Facility	67.53%	89.26	64.08%	76.42%
Overall Accuracy	88.56%	95.29	88.89%	91.73%
Kappa Accuracy	0.8692	0.9461	0.8729	0.9055

shows the quantitative evaluation results.

The classification accuracy for site 1 using 16 bands was 95.29%, and the classification result using 8 bands was 88.56%. Thus, it was confirmed that the classification accuracy is improved when SWIR bands are additionally applied to classification. In addition, as a result of classification for each class, the classification accuracy of artificial structures such as building roofs, roads, and cultivation facilities was improved compared to that of natural features. For site 2, the classification results using 16 bands and 8 bands were calculated with 91.73% and 88.89% accuracy, respectively. Site 2 also showed high accuracy when classification was performed using 16 bands. When comparing each class, it was confirmed that the classification accuracy of the blue-roof buildings, black-roof buildings, and cultivation facilities among artificial structures was improved. However, the accuracy for the green-roof buildings decreased. Through the classification results using the sharpened image, when the SWIR bands are used with the VNIR bands, the superiority was confirmed in terms of image utilization, such as classification. In particular, this approach is considered to be appropriate for the classification of artificial structures, etc. for urban areas.

4. Conclusions

This study proposed a two-phase pansharpening technique to improve the spatial resolution of Worldview-3 satellite images with various spatial and spectral resolutions. Specifically, when applying the panchromatic images was difficult, selected and synthesized band schemes were used to produce optimal panchromatic images; then, sharpened images were generated by applying the GSA and GS2 techniques, which rely on CS- and MRA-based algorithms, respectively. The sharpening results were compared with the results of existing pansharpening techniques. Through both quantitative and qualitative evaluation, among the proposed techniques of this study, and the sharpening performance using the selected band scheme and GS2 was found to be the best. This study shows that sharpened images can be generated that minimize the spatial and spectral distortion of satellite images containing various spatial and spectral resolutions. Thus, the proposed techniques are expected to be useful in land cover mapping, change detection, and monitoring with satellite images that contain various spatial and spectral resolutions. Furthermore, sharpened images can be produced utilizing the proposed techniques with other satellite images that are similar to those of Worldview-3.