*3.2. Experimental Results*

Table 4 shows the effect of the back-projection (BP). The results of the intensity *Isr* obtained using the sparse representation (SR) and the intensity obtained by repeating BP ten times with a size 5 filter are shown. These were compared by CC, UIQI, and ERGAS, and applying BP was better in all cases. Table 5 shows the effect of the tradeoff process (TP). The intensity images generated by SCMP, SR, and TP are shown. TP was the best in all evaluations. Table 6 shows the comparison of BP and TP. TP was better in every case.

Tables 7 and 8 show numerical evaluations of the existing methods and the proposed method. The existing methods include fast IHS [5], Gram–Schmidt method (GS) [7], band-dependent spatial detail (BDSD) [41], weighted least-squares (WLS)-filter-based method (WLS) [42], multiband images with adaptive spectral-intensity modulation (MDSIm) [43], spectrum correction using modeled panchromatic image (SCMP) [35], image super-resolution via sparse representation (ISSR) [13] using natural images (Dict-natural) and corresponding images (Dict-self), the sparse representation of injected details (SR-D) [30], and the method of sparse representation described by Ayas et al. (SRayas) [34].

**Table 4.** Numerical evaluation of image intensities with or without the back-projection process of the intensity obtained via sparse representation. The best values are given in bold.



**Table 5.** Numerical evaluation of image intensities obtained by SCMP, SR, and the tradeoff process. The best values are given in bold.

**Table 6.** Comparison of the back-projection and the tradeoff process. The best values are given in bold.


**Table 7.** Numerical evaluation of the existing methods and the proposed method by CC and UIQI. The highest scores are printed in bold, and the second highest scores are underlined.


The size of the local estimation of the distinct block of BDSD was 256 × 256 for Nihonmatsu and 448 × 448 for Yokohama. For the ISSR settings, the training images for the dictionary included training PAN images and natural images, the number of training times was 40, the number of atoms in the dictionary was 1024, the atom size of the dictionary was 4 × 4 × 5, the sparsity regularization parameter was 0.1, randomly selected 1000 training image patches were used, the upscale was 4 (ratio of resolution of PAN images and MS images of IKONOS), overlap pixel of patches in the reconstruction process was R1×*<sup>p</sup>* in horizontal direction and R*p*×<sup>1</sup> in vertical direction, the size of the back-projection filter was 5 × 5, and the number of iterations was 10. For SR-D, high-resolution and low-resolution dictionaries were constructed from the original PAN images without training. The atom size of the high-resolution dictionary was R28×28×<sup>4</sup> and the overlapping areas of the adjacent atoms were R16×*p*, R*p*×16; the atom size of the low-resolution dictionary was R7×7×<sup>4</sup> and and the overlapping area of the adjacent atoms were R4×*p*, R*p*×4.For the SRayas setting, the original IKONOS MS images were used as the training images for the dictionary, the number of training times was 20, the number of atoms in the dictionary was 4096, the size of the dictionary atom was 8 × 8 × 4, the sparsity regularization parameter was λ = 0.15, the number of training image patches was 2000, the upscale was 4 (ratio of resolution of PAN images and MS images of IKONOS), β = 0.25, the weight of each spectral band of IKONOS was *w* = [0.1071, 0.2646, 0.2696, 0.3587], the overlap pixel in the patch of reconstruction process was 0, the back-projection filter size was 3 × 3, and the number of repetitions was 20. These settings of the existing methods followed those described in the original papers except the distinct block size of BDSD. The code of Vivone et al. [2] was used for GS and BDSD, and the code of Yang et al. [13] was used for ISSR.


**Table 8.** Numerical evaluation of the existing methods and the proposed method by ERGAS and SAM. The highest scores are printed in bold, and the second highest scores are underlined.

**Figure 6.** Scores of quality metrics.

In Tables 7 and 8, the highest scores are printed in bold, and the second highest scores are underlined. GS was generally not good except for the SAM of Yokohama. The results of BDSD were unremarkable but stable. SCMP was stable and gave good results. In ISSR, differences in training images had little effect on results. SRayas did not perform as well overall as ISSR and SRayas. Although some results, such as the CC of SR-D of Table 7, MDSIm of Table 8, and SAM of GS were better in part than the proposed method, in many other cases they were less accurate than the proposed method. The results of the proposed method were generally good, although there are some differences due to the tradeoff parameter.

Figure 6 shows the ranking of the quality metric of the numerical evaluations of Tables 7 and 8, except for the proposed method. For each test, the best result was worth three points, the second-best result was worth two points, and the third-best result was worth one point. The highest score was 24 points. This figure shows that only three methods, WLS, SCMP, and the proposed method, were good for both of the images, and the proposed method got the highest score.

**Figure 7.** Reference and pansharpened Nihonmatsu images: (**a**) original PAN image (reference image), (**b**) original RGB image, (**c**) ground truth RGB image, (**d**) RGB image upsampled by bicubic interpolation, (**e**) fast IHS, (**f**) GS, (**g**) BDSD, (**h**) WLS, (**i**) MDSIm, (**j**) SCMP, (**k**) ISSR (natural images were used for training), (**l**) ISSR (Nihonmatsu images were used for training), (**m**) SR-D, (**n**) SRayas method, (**o**) proposed method (τ = 0.3).

**Figure 8.** Reference and pansharpened Yokohama images: (**a**) original PAN image (reference image), (**b**) original RGB image, (**c**) ground truth RGB image, (**d**) RGB image upsampled by bicubic interpolation, (**e**) fast IHS, (**f**) GS, (**g**) BDSD, (**h**) WLS, (**i**) MDSIm, (**j**) SCMP, (**k**) ISSR (natural images were used for training), (**l**) ISSR (Yokohama images were used for training), (**m**) SR-D, (**n**) SRayas method, (**o**) proposed method (τ = 0.4).

Figures 7 and 8 show the reference and PS images from the Nihonmatsu and Yokohama datasets, respectively. Since the images were small, the enlarged image surrounded by the yellow frame of the original RGB image (b) is shown in (c)–(o). In Figure 7, in GS (f), the color of green was darker in the rice field area (indicated by the green arrow), while the forest area was whitish. BDSD (g) was more blurred than other images. In MDSIm (i), the color of the forest area was also whitish (indicated by the yellow arrow). In Figure 8, the vegetation area was whitish in GS (f) and MDSIm (i) (indicated by the red arrow). WLS (h) looked hazy. SR-D (m) had lower resolution than the other methods using sparse representation with back-projection. In both Figures 7 and 8, the results of ISSR (k) (l) and SRayas (n) had ringing artefacts. Other images appeared to be reproduced without problems.

#### **4. Discussion**

From the results in Table 4, it was found that the back-projection (BP) was effective from the viewpoint of improving spectral distortion. From the results in Table 5, it was found that the tradeoff process (TP) is effective in improving spectral distortion. Furthermore, since the TP was better than the individual methods of sparse representation (SR) and SCMP, it was clarified that these methods complementarily improve the spectral distortion. From the results in Table 6, it was found that the TP improved spectral distortion more than BP. In the results shown in Tables 7 and 8, the method using GIHS (WLS and SCMP) was better than the existing methods using SR. In addition, it was found that the proposed method, the linear combination of SR and SCMP, gave better results than SCMP alone. One of the problems to be solved in PS processing is the independence of the processed image. In other words, it is important to obtain stable and good results rather than obtaining good results only on a specific image. Although there are some methods shown in Tables 7 and 8 that gave better results than the proposed method, comparing the other evaluation results shows that the results were inconsistent. The reason why the evaluation results were so different could be that these processing methods depend on the processed image.

As shown in Figures 7 and 8, it was found that the reproduction of vegetation area by the GS was unstable. Since the image quality differs depending on the size of the local estimation on the distinct blocks used in BDSD, we evaluated the visibly good images with good numerical values, but the resolution of the image was low. The WLS gave good numerical results with two images, but the images were blurred. Both ISSR and SRayas using BP generated ringing artifacts. Other images seemed to be reproduced without problems in resolution and color. Among them, the proposed method gave the best results in the numerical evaluation.

In this method, resolution enhancement was achieved by using the visible and NIR regions. On the other hand, it can be applied only to the resolution enhancement of RGB images, and not NIR images.

### **5. Conclusions**

In this paper, we proposed a method for pansharpening based on CS theory. In the proposed method, the intensity obtained from the component substitution method and the intensity obtained via the method based on CS theory are fused to reproduce the intensity close to the original. We introduced SCMP as the intensity substitution method and used the tradeoff process for image fusion. Experimental results showed that the proposed method outperformed existing methods in terms of numerical and visual evaluation. The proposed method was also effective for satellites with panchromatic sensors (observed areas are visible and NIR regions) and multispectral sensors (observed areas are red, blue, green, and NIR bands) like IKONOS.

Generally, the intensity image generated by a CS-based method is blurrier than the intensity image generated by the component substitution method, because component substitution captures the intensity of the PAN image. On the other hand, it is expected that the spectral distortion of the intensity image generated by the CS-based method will be lower than that of the image generated by the component substitution method. Since complete restoration is not guaranteed, in order to get the image close to a complete reproduction, back-projection methods can be used. However, they may cause ringing artifacts. Based on these considerations, our proposed method combines the intensities generated by the CS-based method and the component-substitution-based method via the tradeoff process instead of the back-projection to achieve both an improvement of spatial resolution and a reduction of spectral distortion. Experimental results show that the tradeoff process was more effective than the back-projection in generating a pansharpened image of which the spatial resolution was equivalent to that of the PAN image and reducing spectral distortion. Improvement of the accuracy by parameter tuning is important future work.

**Author Contributions:** Methodology, N.T. and Y.S.; Software, N.T.; Validation, Y.S.; Writing—Original Draft Preparation, N.T.; Writing—Review and Editing, Y.S. and S.O.; Supervision, S.O.; Project Administration, S.O.; Funding Acquisition, S.O. All authors have read and agreed to the published version of the manuscript.

**Funding:** This work was partially supported by the JSPS KAKENHI Grant Number 18K19772 and the Yotta Informatics Project by MEXT, Japan.

**Acknowledgments:** The authors thank the Japan Space Imaging Corporation and Space Imaging, LLC for providing the images.

**Conflicts of Interest:** The authors declare no conflict of interest.
