Full-Resolution Quality Assessment for Pansharpening

Abstract

A reliable quality assessment procedure for pansharpening methods is of critical importance for the development of the related solutions. Unfortunately, the lack of ground truths to be used as guidance for an objective evaluation has pushed the community to resort to two approaches, which can also be jointly applied. Hence, two kinds of indexes can be found in the literature: (i) reference-based reduced-resolution indexes aimed to assess the synthesis ability; (ii) no-reference subjective quality indexes for full-resolution datasets aimed to assess spectral and spatial consistency. Both reference-based and no-reference indexes present critical shortcomings, which motivate the community to explore new solutions. In this work, we propose an alternative no-reference full-resolution assessment framework. On one side, we introduce a protocol, namely the reprojection protocol, to take care of the spectral consistency issue. On the other side, a new index of the spatial consistency between the pansharpened image and the panchromatic band at full resolution is also proposed. Experimental results carried out on different datasets/sensors demonstrate the effectiveness of the proposed approach.

1. Introduction

Image pansharpening is the process of merging two observations of the same scene, a low-resolution multispectral (MS) component and a high-resolution panchromatic (PAN) component, to generate a new multispectral image that displays both the rich spectral content of the MS and the high resolution of the PAN. By following the taxonomy proposed in [1], pansharpening methods can be roughly grouped into four main categories: component substitution (CS) [2], multiresolution analysis (MRA) [3], variational optimization (VO) [4,5], and machine/deep learning (ML) [6,7].

In the CS approach, the multispectral image is transformed in a suitable domain, where one of its components is replaced with the PAN. In the particular case of three spectral bands, the Intensity–Hue–Saturation (IHS) transform is an option where the intensity component can be replaced with the PAN band [8]. This method has been generalized in [9] (GIHS) to handle a larger number of bands. Other useful transforms to implement a CS solution include the principal component analysis [10], the Brovey transform [11], and the Gram–Schmidt (GS) decomposition [12]. More recently, adaptive CS methods have also been proposed, such as the advanced versions of GIHS and GS [13], the partial replacement CS method (PRACS) [14], or the band-dependent spatial detail (BDSD) injection method and its variants [15,16,17].

In MRA approaches [3], the pansharpening task is addressed from the perspective of a pyramidal decomposition to separate low-frequency content from detail components. The high-frequency spatial details are extracted by means of multi-resolution decomposition, such as decimated or undecimated wavelet transforms [3,18,19,20], Laplacian pyramids [21,22,23,24,25], or other nonseparable transforms, e.g., contourlet [26]. Extracted details are then properly injected into the upscaled MS component.

A further set of methods address the pansharpening problem through the variational optimization of suitable models of the fused image. In [4], the optimization target involves the degradation filters mapping high-resolution to low-resolution images, while [27] leverage sparse representations for detail injection. Palsson et al. proposed several methods of this class. A total variation regularized least square formulation is provided in [28]. The same research team has framed the pansharpening as a maximum a posteriori problem in [29] and, more recently, explored the use of low-rank representations of the joint PAN-MS [5]. Other methods do not fit the above categories and can be roughly classified as statistical [30,31,32,33,34], dictionary-based [35,36,37,38,39,40], or matrix factorization approaches [41,42,43]. The reader is referred to [1,44] for a more comprehensive review.

In recent years, a paradigm shift from model-based to data-driven approaches has revolutionized all fields of image processing, from computer vision [45,46,47,48,49] to remote sensing [7,50,51,52]. In pansharpening, the first method based on convolutional neural networks (CNN) was proposed by Masi et al. in 2016 [6], after which many more followed in the span of a few years [7,53,54,55,56,57,58,59,60,61,62,63,64]. It seems safe to say that deep learning is currently the most popular approach for pansharpening. Nonetheless, it suffers from a major problem: the lack of ground truth data for supervised training. In fact, multi-resolution sensors can only provide the original PAN-MS data, downgraded in space or spectrum, and never their high-resolution versions, which remain to be estimated.

Based on this brief and certainly not exhaustive overview, it appears that pansharpening is a very active research field, with many new methods proposed every year. Reliable quality assessment procedures are of critical importance for correctly advancing the state of the art, and an incorrect evaluation paradigm may negatively impact the design or tuning of any new solution. Unfortunately, by the very nature of pansharpening, no ground truth (GT) data are available to perform a reference-based assessment. As a consequence, two kinds of quality assessments are usually employed:

(i)

reference-based reduced-resolution assessment (synthesis check);

(ii)

no-reference full-resolution assessment (consistency check).

Lacking GTs, the synthesis capabilities of any pansharpening method can only be assessed on “synthetic” data. In particular, Wald’s protocol [65] suggests taking the real PAN-MS data and applying a proper resolution downgrade process for scale reduction. The scaled PAN-MS pair is the synthesized image whose GT is now given by the original MS component. The way in which the resolution downgrade has to be performed has been the object of intense research in the last two decades and has a non-negligible impact on the reliability of the consequent quality assessment. However, no matter how a GT is obtained, there exist plenty of reference-based image quality indicators. The spectral angle mapper was introduced in [66] and assesses the balance among the spectral bands. On the contrary, the spatial correlation coefficient [67] computes the correlation coefficient across the high-pass-filtered bands of the pansharpened image and of the GT. It is therefore oriented to the assessment of the spatial quality. The Erreur Relative Globale Adimensionnelle de Synthése [68] generalizes the root mean squared error, introducing band-wise correction weights. The universal image quality index [69] compares image statistics to take into account local correlation, intensity, and contrast. Based on this general-purpose image quality index, domain-specific variants suitably adapted to the pansharpening have been proposed in [70,71]. In addition to these indexes, other popular general-purpose options are sometimes considered for pansharpening, e.g., the peak signal-to-noise ratio or the structural similarity (SSIM) index.

Furthermore, spectral or spatial consistency indexes suited for full-resolution images that do not require any GT have also been developed. A first spectral distortion index proposed by Zhou et al. [67] compares a rescaled version of the pansharpened image with the MS image. Other spectral distortion indexes follow a similar procedure [72,73]. In [74], the Quality for Low Resolution (QLR) index based on SSIM was proposed and later updated, exchanging SSIM with the composite image quality measure CMSC based on means, standard deviations, and correlation coefficient [75]. Basically, most of the known spectral distortion indexes follow this protocol: degradation of the pansharpened image and comparison with the MS image. The different behavior stems from the degradation model and the error measure. For example, [74] is based on the structural similarity indexes, while [75] leverages a composite image quality measure based on statistics such as mean, variance, and correlation coefficient. On the other hand, the spatial consistency can be checked through the cross-scale invariance of some statistics [72]. A similar approach is proposed in [73], where only high-frequency components are concerned. In [74], it is proposed a Quality for High Resolution (QHR) index, which assesses the SSIM index between the panchromatic band and a projection of the pansharpened image in the PAN domain. A variant of this approach with a different error term was proposed in [75]. Another similar approach is proposed in [76], where the coefficient of determination is used to compare the PAN image with its projection from the pansharpened image. It is also worth mentioning other no-reference quality indexes, such as the Natural Image Quality Evaluator (NIQE) [77,78], which seeks to assess the image quality by itself, rather than checking consistency.

Finally, often, the spectral and the spatial consistency indexes are somehow combined, e.g., through geometric means, to provide a single hybrid consistency index representing the overall quality no-reference assessment of the fused images  [72,73,74,75,78,79,80,81,82,83].

Both reference-based and no-reference indexes present inherent limitations, which are analyzed in the next section.

In this work, we propose new full-resolution quality indexes that overcome some of these problems and provide more reliable guidance for the development of ever more accurate pansharpening methods. To this purpose, the developed indexes have been made available to the community through a web repository at https://github.com/matciotola/fr-pansh-eval-tool/ (accessed on 5 April 2022).

The remainder of the paper is organized as follows. Section 2 provides a brief critical survey on pansharpening assessment. Section 3 introduces the proposed approach. Section 4 discusses the experimental results and, finally, Section 5 draws conclusions.

2. A Review of Panshapening Indexes

Due to the lack of ground truths, visual inspection by human experts is the ultimate benchmark for pansharpening methods. However, this is a lengthy and tedious task, and numerical indexes are essential for a viable development process. Reduced-resolution (RR) indexes are computed, according to Wald’s protocol [65], by reducing the scale of both MS and PAN components. Then, pansharpening is performed on these rescaled data, and the original MS is used as GT to compute reference-based indexes. Instead, full-resolution (FR) indexes are computed on the original data and try to assess their spectral and/or spatial consistency with the pansharpened output. In a comprehensive evaluation, following Wald’s protocol [65], usually, both types of indexes are considered together, in addition to visual inspection [1]. In order to highlight the critical points of the current approaches to the quality assessment, we provide a brief overview of some of the most representative indexes in the following.

2.1. Reduced-Resolution Assessment

Let M and P be the original MS and PAN components, respectively, and $\widehat{M}$ be the pansharpened image. The synthesis properties cannot be directly verified on the full-resolution pair $(M,P)$ due to the lack of GT with which to compare $\widehat{M}$. Leveraging on a scale invariance hypothesis, it is, however, possible to synthesize datasets with GT by properly downgrading the available $(M,P)$ pairs by a factor R (PAN-MS resolution ratio). The resulting reduced-resolution pair $(M_{\downarrow },P_{\downarrow })$ can therefore be used as the source input for the pansharpening task, whereas the original MS image M acts as GT. Once moved to the reduced-resolution space, many different quality indexes can be computed to assess the mismatch between the pansharpened image and its reference GT, such as root mean square error, structural similarity, and so on. In particular, regarding pansharpening, the most common are the following.

(a)

SAM (Spectral Angle Mapper) [66]. It determines the spectral similarity in terms of the pixel-wise average angle between spectral signatures. If $\mathbf{v}$ and $\widehat{\mathbf{v}}$ are two corresponding pixel spectral responses to be compared, SAM is obtained by averaging over all image locations the following “angle” among vectors:

$$\mathrm{SAM}(\mathbf{v},\widehat{\mathbf{v}})=arccos\left(\frac{〈\mathbf{v},\widehat{\mathbf{v}}〉}{{\parallel \mathbf{v}\parallel }_2·{\parallel \widehat{\mathbf{v}}\parallel }_2}\right).$$

(1)

(b)

ERGAS (Erreur Relative Globale Adimensionnelle de Synthése) [68]. This is one of the most popular indexes to assess both spectral and structural fidelity between a synthesized image and a target GT. Such an index presents interesting invariance properties. Indeed, it is insensitive to the radiometric range, number of bands, and resolution ratio. If B is the number of spectral bands, it is defined as

$$\mathrm{ERGAS}=\frac{100}{R}\sqrt{\frac{1}{B}{\sum }_{b=1}^B{\left(\frac{{\mathrm{RMSE}}_b}{{\mu }_b^{\mathrm{GT}}}\right)}^2},$$

(2)

where ${\mathrm{RMSE}}_b$ is the root mean square error over the b-th spectral band, and ${\mu }_b^{\mathrm{GT}}$ is the average intensity of the b-th band of the GT image.

(c)

$Q{2}^n$ [71]. This is a multiband extension of the universal image quality index (UIQI) [69]. Each pixel of an image with B spectral bands is accommodated into a hypercomplex (HC) number with one real part and B– 1 imaginary parts. Let $\mathbf{z}$ and $\widehat{\mathbf{z}}$ denote the HC representations of a generic GT pixel and its prediction, respectively, and then $Q{2}^n$ can be written as the product of three terms:

$$Q{2}^n=\frac{|{\sigma }_{\mathbf{z}\widehat{\mathbf{z}}}|}{{\sigma }_{\mathbf{z}}{\sigma }_{\widehat{\mathbf{z}}}}·\frac{2{\sigma }_{\mathbf{z}}{\sigma }_{\widehat{\mathbf{z}}}}{{\sigma }_{\mathbf{z}}^2+{\sigma }_{\widehat{\mathbf{z}}}^2}·\frac{2{\mu }_{\mathbf{z}}{\mu }_{\widehat{\mathbf{z}}}}{|{\mu }_{\mathbf{z}}{|}^2+{\left|{\mu }_{\widehat{\mathbf{z}}}\right|}^2}.$$

(3)

The first factor provides the modulus of the HC correlation coefficient between $\mathbf{z}$ and $\widehat{\mathbf{z}}$. The second and the third terms measure contrast changes and mean bias, respectively, on all bands simultaneously. Statistics are typically computed on 32 × 32 pixel blocks, and an average over the blocks of the whole image provides the global $Q{2}^n$ score, which takes values in the $[0,1]$ interval, being 1 the optimal value achieved if and only if $\mathbf{z}=\widehat{\mathbf{z}}$ in each location.

Behind its appealing simplicity, regardless of the specific error measurement employed, the reduced-resolution assessment hides some major subtle pitfalls that undermine its usefulness. On one hand, its accuracy depends critically on the method used for scaling, which may not correspond to the actual sensing conditions. More fundamentally, it relies on the arbitrary assumption that a method optimized for the RR scale will keep its good behavior at the FR scale. Experimental evidence shows this not to be the case. Furthermore, optimizing parameters for a given scale may lead to a sort of “scale overfitting”, with the perverse effect of degrading performance on a different scale. For these reasons, many recent studies focus on no-reference full-resolution quality indexes [76,78,80,82]. It is also worth recalling that the assessment of spectral and spatial consistencies is not necessarily linked to a fusion task and is of broader interest, with applications in the medical [84], agricultural [85], and food [86] sciences, to mention a few.

2.2. Full-Resolution No-Reference Assessment

As a consequence of the above limitations, many studies have moved toward a no-reference approach to the pansharpening quality assessment. In particular, the same Wald’s protocol [65] recognizes the problem of invoking a complementary consistency quality check in addition to the synthesis capacity assessment. In practice, for consistency-based assessment, the input data are pansharpened in their original (full) resolution space. The outcome is then compared with both input components, MS and PAN, for spectral and consistency assessment, respectively. The spectral consistency check requires a spatial resolution degradation of the pansharpened image, which is typically done by means of a low-pass filtering (LPF), followed by decimation, using LPFs that approximate the sensor Modulation Transfer Function (MTF) [87]. Other approaches have also been explored, such as wavelet transforms [88,89], averaging operators [90], and pyramid decomposition [21]. The spatial (or structural) consistency check is indeed more controversial, lacking a clear relationship that allows one to project the pansharpened MS in the PAN domain [91]. No-reference indexes are also often referred to as FR indexes, as they do not require any resolution degradation of the input data, contrarily to the reference-based ones, which do need it because of the lack of GTs. For this reason, the latter are also referred to as RR indexes.

Among the several FR quality assessment approaches proposed in the last few years [26,67,72,73,76], we will recall a few of them, which are the most frequently used. In particular, Alparone et al. [72] proposed a spectral distortion index $D_{\lambda }$ defined as

$$D_{\lambda }=\sqrt[p]{\frac{1}{B(B-1)}\sum _{i=1}^B\sum _{j=1,j\ne j}^B{\left|d_{i,j}(M,\widehat{M})\right|}^p},$$

(4)

where $d_{i,j}(M,\widehat{M})=Q(M_i,M_j)-Q({\widehat{M}}_i,{\widehat{M}}_j)$, with subscripts indicating selected bands and $Q(·,·)$ being the UIQI [69]. Such an index was also enclosed in the pansharpening benchmark toolbox [44]. As can be observed, $D_{\lambda }$ compares the inter-band relationship through the UIQI, separately for the input MS M and the pansharpened image $\widehat{M}$, and then quantifies their divergence over each coupling of spectral bands. As can be assumed, lacking a direct measurement of the distance between M and $\widehat{M}$, and leveraging an assumption of cross-scale inter-band correlation invariance only, there can be important spectral aberrations that can remain unseen by $D_{\lambda }$. The recently published revisited version [1] of the same toolbox [44], indeed, makes use of an approximated version of Khan’s index proposed in [73] to monitor the spectral consistency. In particular, Khan’s index is defined as follows

$$D_{\lambda }^{(\mathrm{K})}=1-Q{2}^n\left({\widehat{M}}_{\downarrow }^{\mathrm{lp}},M\right),$$

(5)

where ${\widehat{M}}_{\downarrow }^{\mathrm{lp}}$ indicates the MTF-based low-pass-filtered and decimated version of the pansharpened image $\widehat{M}$, while M is the original input MS. Its approximation, used in the toolbox [1], named ${\tilde{D}}_{\lambda }^{(\mathrm{K})}$, avoids the decimation of $\widehat{M}$ by using an upscaled version $\tilde{M}$ of M:

$${\tilde{D}}_{\lambda }^{(\mathrm{K})}=1-Q{2}^n\left({\widehat{M}}^{\mathrm{lp}},\tilde{M}\right),$$

(6)

being ${\widehat{M}}^{\mathrm{lp}}$ the low-pass version of $\widehat{M}$. These indexes range between 0 (optimal value) and 1, and clearly relate to the spectral consistency, since the resolution downgrade process applied to $\widehat{M}$ eliminates the high-pass content retrieved from the PAN. However, two critical points should be taken into account:

(a)

dependence on the accuracy of the estimated MTF;

(b)

sensitivity to the PAN-MS alignment.

In particular, as a consequence of (b), methods such as several CS approaches [12,13,14,25,92] that intrinsically address the coregistration problem are penalized by both $D_{\lambda }^{(\mathrm{K})}$ and ${\tilde{D}}_{\lambda }^{(\mathrm{K})}$ on misaligned datasets. Hence, in order to fairly use Khan’s index, it is usually recommended to keep separate the registration and pansharpening problems by means of a prior coregistration of the datasets to be used for testing [1,44].

Several indexes have also been proposed for FR spatial consistency check [72,73,74,76]. In particular, the spatial distortion index proposed in [72] is used in the most recent pansharpening assessment toolbox [1] and is defined as:

$$D_{\mathrm{S}}={∥\frac{1}{B}\sum _{i=1}^{B}Q({\widehat{M}}_i,P)-Q(M_i,P_{\downarrow })∥}_q,$$

(7)

where $P_{\downarrow }$ is the original PAN component resized to the MS scale and $Q(·,·)$ is the UIQI quality index [69]. This index is somehow similar to $D_{\lambda }$ as it checks the cross-scale invariance of the PAN-MS spectral correlation. Therefore, it inherits similar limitations, which are:

• no direct comparison between the pansharpened image $\widehat{M}$ and the PAN P;
• a cross-scale invariance assumption for which there are no guarantees.

3. Proposed Full-Resolution Indexes

We now propose some new indexes for assessing the quality of pansharpening, aiming to overcome some limitations of traditional ones. Given the weakness of the scale invariance assumption of reduced-resolution quality assessment indexes, we focus on full-resolution ones, moving our perspective from synthesis to consistency. In the following, we will discuss spectral and spatial assessment separately.

3.1. Reprojection Protocol for Spectral Accuracy Assessment

The assessment of the spectral consistency is not as controversial as the assessment of the spatial one. In particular, as far as we know, Khan’s approach is the most trusted one for this purpose. Nonetheless, as remarked above, it has some limitations and, in particular, in order to work properly, the datasets must be well aligned.

To cope with this issue without the need to operate any coregistration on the available datasets, let us now consider a variation of Khan’s index where the decimation of ${\widehat{M}}^{\mathrm{lp}}$ is performed using a band-wise shift that maximizes the correlation coefficient between the low-pass-filtered PAN $P^{\mathrm{lp}}$ and each ideally interpolated MS band ${\tilde{M}}_b$. In the case of aligned datasets, it returns the usual Khan’s index because no shifts are applied. Otherwise, before decimation, we apply to $\widehat{M}$ the same shift that would align $\tilde{M}$ and $P^{\mathrm{lp}}$. Let us indicate as $D_{\lambda ,\mathrm{align}}^{(\mathrm{K})}$ such an index that is formally given by

$$D_{\lambda ,\mathrm{align}}^{(\mathrm{K})}=1-Q{2}^n\left({\widehat{M}}_{\downarrow a}^{\mathrm{lp}},M\right),$$

(8)

where $\downarrow a$ indicates decimation with band-wise alignment.

Let $\mathcal{M}$ indicate a generic reference-based distortion index, such as $Q{2}^n$, SAM, or ERGAS. We define the reprojection index R-$\mathcal{M}$ as

$$R\text{-}\mathcal{M}=\mathcal{M}\left({\widehat{M}}_{\downarrow a}^{\mathrm{lp}},M\right),$$

(9)

which, in the particular case of $\mathcal{M}=Q{2}^n$, reduces to the complement of the $D_{\lambda ,\mathrm{align}}^{(\mathrm{K})}$ variant defined above (Equation( 8)), i.e.,

$$R\text{-}Q{2}^n=Q{2}^n\left({\widehat{M}}_{\downarrow a}^{\mathrm{lp}},M\right)=1-D_{\lambda ,\mathrm{align}}^{(\mathrm{K})}.$$

(10)

In other words, the pansharpened image $\widehat{M}$ is projected in the original MS space prior to being compared with the MS image M (reference) using the error index $\mathcal{M}$. The projection accounts for both eventual misalignment and sensor MTF using matched low-pass filters. Of course, because of the low-pass filtering, the high-frequency image content (spatial details) is discarded and some further indexes will be necessary to assess spatial quality. Note also that, under the hypothesis that the sensor MTF is correctly simulated, the reprojection indexes ensure that an ideal pansharpening (GT) would achieve the best score as expected. This will be experimentally analyzed in Section 4.3.

A pictorial representation of the proposed reprojection indexes, which relates them to the synthesis assessment protocol proposed by Wald [65], is given in Figure 1. The common starting point (top-left) is the available $(P,M)$ pair. Indexes based on Wald’s protocol follow the counterclockwise path: decimation–pansharpening–measure. The proposed reprojection protocol, instead, consists in following the clockwise path: pansharpening–decimation–measure, where the actual pansharpened image is first computed, and then reprojected on the low-resolution domain for quality assessment. Eventually, in both cases, the original MS is used as a reference for any error measurement. In our proposal, however, pansharpening precedes the resolution downgrading, thereby exploiting the most valuable source of spatial information, the original PAN. Furthermore, the overall evaluation no longer depends on how the PAN is downscaled. Since the the reprojection protocol involves only the low-resolution component of the synthesized image, it has to be considered a spectral consistency rather than a synthesis check. A pseudo-code of the reprojection indexes is given in Algorithm 1.

Algorithm 1 Reprojection error assessment

Require:  $\widehat{M}$, M, P, ${\mathbf{g}}_{\mathrm{MTF}}$ (MTF parameters), $\mathcal{M}$ (error function) Ensure:  R-$\mathcal{M}$ 1: $\tilde{M}=\mathrm{upscale}(M,R)$                                                                          ▹ polynomial interpolation 2: $P^{\mathrm{lp}}=\mathrm{LPF}(P,{\mathbf{g}}_{\mathrm{MTF}})$                                                             ▹ MTF-tailored low-pass filtering 3: for$b=1:B$do 4:     $\mathbf{s}(b)=\underset{(m,n)\in {[1,R]}^2}{argmax}\mathrm{corrcoef}\left(P_{i,j}^{\mathrm{lp}},\tilde{M}{(b)}_{i+m,j+n}\right)$            ▹ band-wise shift for alignment 5: end for 6: ${\widehat{M}}^{\mathrm{lp}}=\mathrm{LPF}(\widehat{M},{\mathbf{g}}_{\mathrm{MTF}})$ 7: ${\widehat{M}}_{\downarrow a}^{\mathrm{lp}}=\mathrm{decimation}({\widehat{M}}^{\mathrm{lp}},R,\mathbf{s})$ 8: $R\text{-}\mathcal{M}=\mathcal{M}\left({\widehat{M}}_{\downarrow a}^{\mathrm{lp}},M\right)$                                                                                             ▹ Equation (9)

3.2. Correlation-Based Spatial Consistency Index

In order to complement the reprojection-based assessment, we also propose a new index to evaluate the spatial consistency with the aim of obtaining indications better correlated to human judgment than those provided by $D_{\mathrm{S}}$. In particular, to assess the preservation of fine-scale spatial structures, we compute the average local correlation between the pansharpened image and the PAN.

Let $X_{ij}^{\sigma }$ indicate a $\sigma \times \sigma$ patch of X centered on location $(i,j)$. We compute the correlation field ${\rho }_{P\widehat{M}}^{\sigma }$ given by the local correlation coefficients between P and each band b of $\widehat{M}$:

$${\rho }_{P\widehat{M}}^{\sigma }(i,j,b)=\mathrm{corrcoef}\left(P_{ij}^{\sigma },\widehat{M}{(b)}_{ij}^{\sigma }\right)$$

(11)

Then, we reduce it to its average value $\overline{{\rho }_{P\widehat{M}}^{\sigma }}$ over space and spectral bands. The final index is then defined as

$$D_{\rho }\triangleq 1-\overline{{\rho }_{P\widehat{M}}^{\sigma }},$$

(12)

such that $D_{\rho }=0$ corresponds to perfect correlation. From a different point of view, we are studying to what extent a $\sigma \times \sigma$ patch of any band of $\widehat{M}$ can be linearly predicted from the corresponding PAN patch. Therefore, $D_{\rho }$ is strictly related to the matching between the spatial layouts of $\widehat{M}$ and P at a fine scale.

The choice of the scale parameter $\sigma$ is of critical importance, as we are interested in the exploitation of the complementary information discarded by the reprojection indexes, i.e., details appearing only at the PAN scale but not at the MS scale. Therefore, if R is the resolution ratio, by choosing $\sigma =R$ (equal to 4 for our datasets), we can place the focus only on that spatial content that is not visible at the MS scale. By doing so, we reduce the dependence between the proposed spatial ($D_{\rho }$) and spectral (R-$\mathcal{M}$) distortion indexes, assigning to them clear but well-separated evaluation objectives.

As for $D_{\mathrm{S}}$ and $D_{\lambda }^{(\mathrm{K})}$, or its variants, $D_{\rho }$ computed for $\widehat{M}$ = GT does not reach the zero value, given the slightly different spatial layout of different bands in natural images and the complex relationship between the high-resolution image and its panchromatic projection [91]. Nonetheless, we expect good-quality pansharpened images to present relatively small values of $D_{\rho }$. A detailed discussion in this regard, supported by our experimental results, will be provided in the following.

4. Experimental Results and Discussion

In this section, we present several experimental analyses. Preliminarily, a summary of the datasets employed and the involved methods is provided. The first experiment focuses on the dependence of the spectral consistency indexes on the alignments of the MS bands with the PAN. Next, we move to the reduced-resolution domain to cross-check reference-based and no-reference indexes thanks to the availability of the ground truth. Then, it follows an experimental analysis focused on the assessment of the spatial consistency, before closing with an overall comparison.

4.1. Datasets and Methods

The experimental validation relies on 25 methods provided in the benchmark toolbox [1] belonging to the four main categories recalled in Section 1, CS (8), MRA (9), VO (3), and ML (4), plus an ideal interpolator (EXP). The dataset is composed of two WorldView-2 (WV2) and two WorldView-3 (WV3) large images, courtesy sample products of DigitalGlobe©. In total, 20 512 × 512 tiles were extracted from the WV2 images (Washington and Stockholm) and 20 from the WV3 images (Adelaide and Mexico City) for the experiments at full resolution. Likewise, 20 + 20 2048 × 2048 tiles were extracted and downscaled to size 512 × 512 for the experiments at reduced resolution.

Table 1. Bandwidths of the multispectral channels (left-hand side) and Ground Sample Distance (GSD) [m] at Nadir (rightmost column) for WorldView-2 and WorldView-3 images.

Sensor	Bandwidths of the MS Channels [nm]								GSD [m]
	Coastal	Red	Blue	Red Edge	Green	Near-IR1	Yellow	Near-IR2	PAN/MS
WV2	396-458	624-694	442-515	699-749	506-586	765-901	584-632	856-1043	0.46/1.84
WV3	400-450	630-690	450-510	705-745	510-580	770-895	585-625	860-1040	0.31/1.24

summarizes the main spectral and spatial characteristics of the WV-2 and WV-3 sensors.

4.2. Spectral Distortion Dependence on PAN-MS Misalignment

The impact of band misregistration on the quality of the fused products has been already recognized in the past [93,94,95]. Here, we propose an ad hoc experimental analysis to show the robustness of the proposed reprojection indexes to the data misregistration. The starting point is a WV3 dataset composed of ten 2048 × 2048 tiles extracted from a larger image of Adelaide (DigitalGlobe© sample product). This dataset presents band misalignment that we have corrected to produce a companion aligned dataset. In

Table 2. Misregistration impact on $D_{\lambda }$, $D_{\lambda }^{(\mathrm{K})}$, ${\tilde{D}}_{\lambda }^{(\mathrm{K})}$ and $D_{\lambda ,\mathrm{align}}^{(\mathrm{K})}$. Detailed information about the methods can be found in [1].

		${\mathit{D}}_{\mathit{\lambda }}$		${\mathit{D}}_{\mathit{\lambda }}^{(\mathbf{K})}$		${\tilde{\mathit{D}}}_{\mathit{\lambda }}^{(\mathbf{K})}$		${\mathit{D}}_{\mathit{\lambda },\mathbf{align}}^{(\mathbf{K})}$
	Aligned Dataset	Yes	No	Yes	No	Yes	No	Yes	No
	EXP (interpolator)	0.0000	0.0000	0.0249	0.0336	0.0394	0.0479	0.1238	0.0670
CS	BT-H [92]	0.0698	0.0823	0.1231	0.0696	0.2310	0.1324	0.0874	0.0697
	BDSD [15]	0.0429	0.0377	0.0989	0.0867	0.1834	0.1561	0.1511	0.1064
	C-BDSD [16]	0.0514	0.0557	0.1436	0.1195	0.2158	0.1836	0.2346	0.1664
	BDSD-PC [17]	0.0141	0.0160	0.0782	0.0896	0.1464	0.1530	0.0705	0.0892
	GS [12]	0.0196	0.0177	0.1494	0.0824	0.2611	0.1414	0.0824	0.0839
	GSA [13]	0.0576	0.0573	0.1182	0.0604	0.2457	0.1334	0.0760	0.0624
	C-GSA [25]	0.0309	0.0333	0.0929	0.0583	0.1925	0.1240	0.0741	0.0625
	PRACS [14]	0.0178	0.0193	0.0728	0.0468	0.1464	0.0889	0.0834	0.0610
MRA	AWLP [96]	0.0332	0.0416	0.0282	0.0273	0.0415	0.0320	0.0920	0.0495
	MTF-GLP [96]	0.0679	0.0759	0.0231	0.0195	0.0434	0.0352	0.0912	0.0428
	MTF-GLP-FS [97]	0.0544	0.0669	0.0222	0.0199	0.0400	0.0347	0.0944	0.0439
	MTF-GLP-HPM [96]	0.0620	0.0722	0.0310	0.0210	0.0508	0.0376	0.0909	0.0411
	MTF-GLP-HPM-H [92]	0.1019	0.1177	0.0270	0.0200	0.0524	0.0401	0.0885	0.0402
	MTF-GLP-HPM-R [98]	0.0497	0.0620	0.0272	0.0210	0.0453	0.0362	0.0928	0.0420
	MTF-GLP-CBD [87]	0.0551	0.0657	0.0222	0.0200	0.0402	0.0346	0.0942	0.0441
	C-MTF-GLP-CBD [25]	0.0089	0.0375	0.0224	0.0225	0.0360	0.0347	0.1129	0.0495
	MF [99]	0.0585	0.0711	0.0421	0.0281	0.0681	0.0441	0.0733	0.0371
VO	FE-HPM [4]	0.0579	0.0666	0.0340	0.0210	0.0529	0.0355	0.0784	0.0406
	SR-D [27]	0.0225	0.0381	0.0043	0.0064	0.0117	0.0190	0.1254	0.0450
	TV [28]	0.0166	0.0238	0.0225	0.0173	0.0459	0.0373	0.0583	0.0252
ML	PNN [6]	0.0589	0.0553	0.0740	0.0629	0.1477	0.1307	0.1815	0.0878
	PNN-IDX [6]	0.0837	0.0848	0.2671	0.0546	0.1570	0.1005	0.3344	0.0889
	A-PNN [100]	0.0527	0.0636	0.0371	0.0312	0.0773	0.0607	0.0963	0.0522
	A-PNN-FT [100]	0.0196	0.0203	0.0414	0.0308	0.0815	0.0579	0.0830	0.0489

, we summarize the average spectral distortion scores for each method for both datasets. For a convenient reading of these numbers, in Figure 2, we show the impact of data misregistration on $D_{\lambda }^{(\mathrm{K})}$ and ${\tilde{D}}_{\lambda }^{(\mathrm{K})}$ in differential terms.

Each bar indicates the difference between the values of the given indicator computed on aligned and misaligned datasets, respectively. As can be clearly noticed, traditional CS methods such as BT-H, GS, GSA, C-GSA, PRACS, which, by construction, provide fused images that are strongly anchored to the PAN geometry, show a considerable spectral loss according to $D_{\lambda }^{(\mathrm{K})}$ or ${\tilde{D}}_{\lambda }^{(\mathrm{K})}$. These results are not aligned with our expectations, as these CS methods are expected to be more robust to misregistration. The reader can refer to [93] for a theoretical comparison between CS and MRA methods in the presence of misregistration, with the former category being superior to the latter.

In Figure 3, instead, we compare Khan’s index $D_{\lambda }^{(\mathrm{K})}$ with its variant $D_{\lambda ,\mathrm{align}}^{(\mathrm{K})}$ with alignment. Again, bars indicate their variations due to dataset misalignment.

By inspecting the figure, it can be noticed that misregistered data generally cause an increase in $D_{\lambda ,\mathrm{align}}^{(\mathrm{K})}$, except for some CS methods, such as BT-H, GS, GSA, C-GSA, and PRACS, which tend to preserve the PAN geometry operating an intrinsic alignment. On the contrary, other methods, notably those belonging to the MRA, VO, and ML categories, which are oriented to the spectral preservation but do not operate alignment, register an increase in the spectral distortion when the metric takes into account the misalignment.

Concluding, the proposed experiment suggests that Khan’s index with alignment correction, which is the complement of the proposed reprojection index R-$Q{2}^n$, is more robust to misalignment or, at least, more consistent with the theoretical expectation [93] than the original Khan’s index $D_{\lambda }^{(\mathrm{K})}$ or its variant ${\tilde{D}}_{\lambda }^{(\mathrm{K})}$.

4.3. Reference vs. No-Reference Index Cross-Checking in the Reduced-Resolution Space

In order to assess the consistency between no-reference and reference-based indicators, we have designed a set of experiments in the reduced-resolution space. Although no-reference indexes are conceived to work in the full-resolution framework, their use on (simulated) reduced-resolution datasets allows us to carry out studies on the correlation between them and objective error measurements (reference-based indexes) thanks to the availability of GTs. In particular, for this experiment, we have resorted to our WV2 dataset composed of twenty 2048 × 2048 images at full resolution (for the sake of brevity, analogous results obtained on WV3 images will not be presented). These images come from two larger images of Washington (13) and Stockholm (7), respectively, courtesy samples of DigitalGlobe©. Each such tile was resized to 512 × 512 pixels using the usual Wald’s downgrading protocol. This dataset was already well coregistered; therefore, we have proceeded to create a misregistered counterpart operating a simple modification of the downgrading process: a 1-pixel shift (in both directions) has been introduced in the decimation (after LPF) of the MS bands.

Then, 25 pansharpening algorithms provided by the toolbox [1] were run on each RR sample image, generating 500 results, for each of the two datasets (registered and not), for which all the indexes of interest have been computed. Eventually, we have obtained hundreds of points in a multi-dimensional evaluation space, which enable plenty of analyses, with some dimensions corresponding to the reference-based indexes (SAM, ERGAS and $Q{2}^n$, SSIM, CMSC), and the remaining ones associated with the proposed indexes (R-SAM, R-ERGAS, R-$Q{2}^n=1-D_{\lambda ,\mathrm{align}}^{(\mathrm{K})}$, $D_{\rho }$) and other state-of-the-art no-reference indexes ($D_{\lambda }$, $D_{\lambda }^{(\mathrm{K})}$, ${\tilde{D}}_{\lambda }^{(\mathrm{K})}$, $D_{\mathrm{S}}$, QLR${}_1$, QLR${}_2$). In particular, by construction, we expect to observe a good correlation between Khan’s index variants and $Q{2}^n$. Therefore, we start from the scatter plots shown in Figure 4 with such variants, in turn, vs. $Q{2}^n$.

Both the “aligned” (top) and the “misaligned” (bottom) datasets are considered. For the aligned case, by visual inspection, we can appreciate that $D_{\lambda }^{(\mathrm{K})}$ correlates better than ${\tilde{D}}_{\lambda }^{(\mathrm{K})}$ with $Q{2}^n$. Moreover, $D_{\lambda ,\mathrm{align}}^{(\mathrm{K})}$ behaves similarly to $D_{\lambda }^{(\mathrm{K})}$ because no alignment is operated. For both $D_{\lambda }^{(\mathrm{K})}$ and $D_{\lambda ,\mathrm{align}}^{(\mathrm{K})}$, the GTs (black dots), for which $Q{2}^n$ is always 1, obtain the ideal value (0) of spectral distortion. Rarely, the alignment process (based on the maximization of the correlation) embedded in the computation of $D_{\lambda ,\mathrm{align}}^{(\mathrm{K})}$ fails for some spectral band, giving rise to non-zero results for the GT even if the test image is aligned. Moreover, ${\tilde{D}}_{\lambda }^{(\mathrm{K})}$ is not minimized by the GT, coherently with its definition (Equation (6)) based on the comparison between the smoothed GT and the upscaled MS. It is also worth noticing that the correlation degree between $Q{2}^n$ and the different variants of spectral distortion grows when assessed by category (by colors), supporting the idea that no-reference indexes are more reliable for intra-class assessment [1]. Moving to the misaligned dataset (bottom scatters), it can be clearly recognized a bias for both ${\tilde{D}}_{\lambda }^{(\mathrm{K})}$ and $D_{\lambda }^{(\mathrm{K})}$, which does not occur for $D_{\lambda ,\mathrm{align}}^{(\mathrm{K})}$, represented by the shift of the GT scores, which no longer achieve the ideal value. In general, the distortion scores in terms of ${\tilde{D}}_{\lambda }^{(\mathrm{K})}$ and $D_{\lambda }^{(\mathrm{K})}$ register a degradation, with CS methods (blue) much more penalized than others. This last observation gives further support to the interpretation provided above about the results of Figure 2 and Figure 3. A similar behavior is also registered for other state-of-the-art indexes. For example, in Figure 5, the score scatters in the QLR${}_1$-SSIM (left) and QLR${}_2$-CMSC (right) planes, with registered (top) and misaligned (bottom) datasets.

Let us now leave the registration problem out, considering aligned datasets only and looking at the relationship between the spectral consistency indexes and the three most used reference-based indexes, i.e., SAM, ERGAS, and $Q{2}^n$, with the help of Figure 6. From top to bottom, we can see how different compared spectral consistency indexes agree with the three (column-wise distributed) reference-based indexes.

From the top row scatters, it clearly appears that $D_{\lambda }^{(\mathrm{K})}$ agrees with SAM and ERGAS much less than with $Q{2}^n$. This is because $D_{\lambda }^{(\mathrm{K})}$ is based on the same $Q{2}^n$ index but also because SAM and ERGAS encode a different concept of quality. Similar considerations apply for QLR${}_1$ and QLR${}_2$, which, as well as $Q{2}^n$, are both based on the comparison among local statistics. For these reasons, we believe that, in addition to R-$Q{2}^n$, it makes sense to also provide R-SAM and R-ERGAS for a more comprehensive evaluation of pansharpening methods. On the bottom row, we show all three ($\mathcal{M}$, R-$\mathcal{M}$) scatter plots, which speak in favor of a good level of agreement between each objective index $\mathcal{M}$ and the reprojected counterpart R-$\mathcal{M}$.

Besides the qualitative interpretation of the score scatters, we can quantify the level of agreement among the reference-based indexes and the compared no-reference indexes in terms of correlation coefficients. These are shown in the usual matrix form, for both the aligned (

Table 3. Correlation coefficient among different indexes (some have been complemented for a uniform polarization) obtained on the aligned RR WV2 datasets.

	R-SAM	R-ERGAS	R-$\mathit{Q}{2}^{\mathit{n}}$	${\mathit{D}}_{\mathit{\lambda }}$	${\mathit{D}}_{\mathit{\lambda }}^{(\mathbf{K})}$	${\tilde{\mathit{D}}}_{\mathit{\lambda }}^{(\mathbf{K})}$	${\mathit{D}}_{\mathit{\lambda },\mathbf{align}}^{(\mathbf{K})}$	${\mathbf{QLR}}_1$	${\mathbf{QLR}}_2$
SAM	0.595	0.353	0.096	0.372	0.101	$-0.012$	0.096	0.409	0.348
ERGAS	0.141	0.743	0.169	0.050	0.169	0.078	0.169	0.173	0.493
$Q{2}^n$	0.121	0.306	0.384	−0326	0.382	0.417	0.384	0.486	0.356
SSIM	0.204	0.338	0.199	−0.106	0.199	0.179	0.199	0.603	0.359
CMSC	0.144	0.390	0.318	−0.095	0.320	0.211	0.318	0.466	0.556

) and misaligned (

Table 4. Correlation coefficient among different indexes (some have been complemented for a uniform polarization) obtained on the misaligned RR WV2 datasets.

	R-SAM	R-ERGAS	R-$\mathit{Q}{\mathbf{2}}^{\mathit{n}}$	${\mathit{D}}_{\mathit{\lambda }}$	${\mathit{D}}_{\mathit{\lambda }}^{(\mathbf{K})}$	${\tilde{\mathit{D}}}_{\mathit{\lambda }}^{(\mathbf{K})}$	${\mathit{D}}_{\mathit{\lambda },\mathbf{align}}^{(\mathbf{K})}$	${\mathbf{QLR}}_{\mathbf{1}}$	${\mathbf{QLR}}_{\mathbf{2}}$
SAM	0.745	0.200	0.119	0.336	0.054	−0.066	0.119	0.238	0.253
ERGAS	0.022	0.932	0.221	0.041	0.171	0.062	0.221	0.028	0.361
$Q{2}^n$	0.023	0.152	0.439	−0.329	0.287	0.293	0.439	0.226	0.110
SSIM	0.164	0.168	0.255	−0.142	0.066	0.035	0.255	0.218	0.046
CMSC	0.151	0.298	0.429	−0.106	0.238	0.079	0.429	0.159	0.294

) datasets, respectively. As expected, the reprojected indexes show a relatively high correlation with the non-reprojected counterpart, both for aligned and misaligned datasets. $D_{\lambda }^{(\mathrm{K})}$ and ${\tilde{D}}_{\lambda }^{(\mathrm{K})}$ also correlate well with $Q{2}^n$, but only in the aligned case. Likewise, QLR${}_1$ and QLR${}_2$ also correlate well with different reference-based indexes in ideal conditions but register a drop on misaligned data. It is also worth remarking that GT scores have been discarded so as not to penalize excessively the competitors on the misaligned dataset (see GT score distribution in Figure 4 and Figure 5).

4.4. A Qualitative Assessment of the Proposed Spatial Distortion Index

While objective synthesis quality indexes such as $Q{2}^n$ and ERGAS account for both spectral and spatial inaccuracies, their reprojected versions clearly limit their scope to the spectral component as they are computed on low-pass versions of the fused products, ignoring image “details”. It is therefore necessary to complement these indexes with some spatial quality indicator.

The assessment of the spatial quality of a pansharpened image in the full-resolution framework is very difficult and somewhat controversial, lacking a degradation model to describe the relationship between the full-resolution spatial–spectral data cube (fused image) and the panchromatic image. In fact, while the spectral degradation can be reasonably modeled through the sensor MTF, allowing for a spectral consistency check, the spatial degradation cannot be modeled through a simple global weighted band average [91]. Indeed, in addition to the obvious spatial dependency of the spectral mixing process that provides the PAN image, there is also a mismatch between the PAN spectral coverage and the MS coverage [91]. This means that there could be details “seen” by the PAN but not by the virtual full-resolution MS counterpart, and vice versa. This is the origin of the ill-posedness of the pansharpening problem, mostly residing in the spatial reconstruction, which makes the spatial consistency assessment subjective to some extent. On the basis of this observation, we provide here an interpretation of some sample results through visual inspection, comparing the proposed index $D_{\rho }$ with the state-of-the-art spatial distortion index $D_{\mathrm{S}}$. Of course, given the subjective nature of this kind of analysis, we will focus on clearly visible phenomena, leaving to the reader the final say based on his/her visual perception of more subtle patterns, if any. In particular, we propose two experiments: a “horizontal” comparison among the pansharpening toolbox methods and a “vertical” comparison where a single machine learning method is optimized using the proposed $D_{\rho }$ (varying the related scale parameter $\sigma$) jointly with a term controlling the spectral consistency.

Let us start with the first experiment. Figure 7 shows some FR pansharpening results for crops extracted from a single tile of the WV3 Adelaide image (For visualization purposes, hereinafter, all displayed crops extracted from multispectral images are obtained combining the red, green, and blue channels (see Table 1)). The PAN component, used as a reference, is shown in the middle of different groups of results. In the top half of the figure, the top four solutions according to $D_{\mathrm{S}}$ (left) and the top four according to $D_{\rho }$ (right) are shown with $D_{\mathrm{S}}$ and $D_{\rho }$ computed on the whole tile. Similarly, in the bottom half, the top four results according to QHR and NIQE are shown with related scores. It appears clearly that images selected according to the $D_{\rho }$ index ensure better agreement with the reference in terms of spatial layout and, in general, better quality, with sharp contours, accurate textures, and a lack of annoying patterns such as those present in some top $D_{\mathrm{S}}$ and top NIQE images. It is interesting to observe that, among the top four $D_{\mathrm{S}}$ results, the most convincing one from visual inspection corresponds also to the relatively lowest $D_{\rho }$ (AWLP). It is also worth observing a certain degree of agreement between $D_{\rho }$ and QHR. Similar phenomena are observed on all other tiles.

In the previous experiment, we set the scale parameter $\sigma =R=4$ according to the above-discussed theoretical motivations (Section 3.2). To gain insight into the effectiveness of this choice, we have designed an additional ad hoc experiment to provide pansharpening results with controlled spectral and spatial qualities, with the latter quantified through $D_{\rho }$, configurable using different scale settings. This is achieved by leveraging a CNN model working in target-adaptive modality [101] and using a combination of the wanted consistency measures as a loss. In this context, the choice of the specific CNN model is not critical as we work in adaptive modality, by running unsupervised tuning iterations on each test image until the loss terms reach the desired level (overfitting). In particular, the optimized loss is defined as

$$\mathcal{L}=max(\parallel {\widehat{M}}_{\downarrow a}^{\mathrm{lp}}-M{\parallel }_1,{\mathcal{L}}_{\lambda }^{\ast })+\beta max(D_{\rho }^{(\sigma )},D_{\rho }^{\ast }),$$

(13)

where ${\mathcal{L}}_{\lambda }^{\ast }$ and $D_{\rho }^{\ast }$ are two prefixed target levels for the spectral ($\parallel {\widehat{M}}_{\downarrow a}^{\mathrm{lp}}{-M\parallel }_1$) and spatial ($D_{\rho }^{(\sigma )}$) quality indicators, respectively. In practice, the network parameters are pushed to overfit the test image so that the corresponding loss reaches the target quality $({\mathcal{L}}_{\lambda }^{\ast },D_{\rho }^{\ast })$. These two threshold values could be ideally set to zero, but this would lead to extremely long tuning and may also generate instability because of a conflicting interaction between the two loss terms. In particular, we have set ${\mathcal{L}}_{\lambda }^{\ast }=0.015$, quite a low value considering the dynamic range of the pixel values, and $D_{\rho }^{\ast }=0.1$, which is also quite low according to our experiments. By doing so, each test requires a different number of tuning iterations. We have therefore oversized the number (experimentally set to 5000) of iterations to ensure convergence for all. This process is repeated for several choices of the scale parameter $\sigma$ ranging from 2 to 64. In Figure 8, we show some crops from the WV3 dataset. These sample results reflect a general behavior that we have observed in a wide range of experiments, which is a relatively good response in the range of $\sigma \in [4,8]$. For smaller ($\sigma =2$) or larger ($\sigma >8$) scale values, noisy patterns arise. These are particularly noticeable on roofs and roads for the selected samples. The above observations provide experimental validation of the choice $\sigma =R=4$ proposed in this work on the basis of the theoretical motivations discussed in Section 3.2.

The present experiment gives us the opportunity to also make some considerations about the computational load related to the proposed indexes. In general, in the context of quality assessment, the computation time is not a critical issue as it is carried out offline. However, with the advent of deep learning, researchers have started to use such quality indexes as loss functions for training purposes. In this regard, it is worth distinguishing pixel-based indexes (e.g., SAM, ERGAS, ${\ell }_{\alpha }$-norms), which are relatively light to compute, from other indexes that involve local statistics computed at a certain scale for each location (e.g., $Q{2}^n$, $D_{\rho }$, $D_{\lambda }^{(\mathrm{K})}$). When using the latter as loss, the training may slow down. In the particular case of $D_{\rho }$, which has been included in the loss function to fine-tune the sample CNN employed in the experiment, we have actually registered a moderate impact on the training time. In fact, using a training batch composed of a single 2048 × 2048 image, the time consumption per iteration, without or with the additional $D_{\rho }$ loss component (second term in Equation (13)), shifts from 1.27 s to 1.6 s on an NVIDIA P6000 GPU.

4.5. Comparative Results

To conclude, we present here an overall comparison among the available pansharpening methods provided by the toolbox [1], using the proposed quality indicators and supported by the visual inspection of some sample results. In Figure 9 and Figure 10, we gather the average numerical results obtained on our WV3 and WV2 datasets, respectively.

As expected, the numerical results clearly show a good level of agreement among the three reprojection error indexes, all essentially linked to the spectral consistency. However, some exceptions can be observed, particularly for the WV2 dataset (Figure 10), where some CS methods register a performance loss in terms of $D_{\lambda ,\mathrm{align}}^{(\mathrm{K})}$ not observed on R-SAM and R-ERGAS. We also recognize good agreement with some literature findings, particularly on WV3 (Figure 9), such as the spectral accuracy gap between MRA and CS methods [44] or the competitiveness of some MRA, VO, and ML solutions. Moreover, for the ML solutions, it is worth noticing a performance gap moving from one dataset (WV3) to another (WV2). Indeed, such variability is not unusual for data-driven approaches, which can suffer generalization limits when the training dataset is not sufficiently representative. In this particular situation, it could likely be the case that the WV2 test images used are too different from the images used for the training of the involved ML methods.

To gain insight into the effectiveness of the proposed indexes, let us take a look at some sample results for a complementary subjective analysis. In Figure 11, we show some clips from a single WV3 tile of the Adelaide dataset. On the leftmost column are gathered the PAN and MS input components on two consecutive rows. Then, the corresponding pansharpening results of the best-performing solutions according to R-$Q{2}^n$ are shown in decreasing order from left to right, next to the PAN. Next to the MS, we also show the “reprojection” error map, which is the difference between the input MS and the reprojection (LPF plus decimation) of the output. On the bottom lines are gathered the R-$Q{2}^n$ and $D_{\rho }$ scores. Therefore, the spectral quality of the results decreases left to right. This can also be appreciated through the inspection of the reprojection error maps. Moreover, the best methods from the spatial perspective ($D_{\rho }$) follow a different ordering. This partially reflects a tradeoff between spectral and spatial features. Particularly interesting is the case of the method TV, which scores first in terms of R-$Q{2}^n$ (actually, it is the best according to R-SAM and R-ERGAS, as well) but shows the worst $D_{\rho }$ value, 0.215, corresponding to a 78.5% average correlation between the PAN and the pansharpened bands. The impact of this low correlation level is clearly visible in the pansharpening results of TV, which show underlying noisy patterns. Such patterns disappear with the reprojection, explaining why the reprojection indexes do not worsen. The other methods achieve much lower values of $D_{\rho }$, ranging from 0.084 to 0.115 (around 90% of correlation), which corresponds to a higher coherence between the spatial features of the results and the PAN, easily appreciable through visual inspection. For completeness, in Figure 12, we show analogous results for a WV2 tile, which basically confirms the same considerations made above for WV3.

5. Conclusions

To cope with the limitations of the reference-based reduced-resolution procedures for the quality assessment of pansharpening methods, in this work, we propose a new full-resolution no-reference evaluation framework. By following Wald’s protocol [65], the full-resolution assessment must be carried out checking the “consistency” of the fused products with the MS and PAN input components rather than the “synthesis” capacity, which would require the availability of GTs. In particular, inspired by Khan’s index [73], we have proposed the use of reprojection-based indexes with embedded alignment to handle misregistered datasets for the assessment of the spectral consistency between the pansharpened image and the input MS. Moreover, the spatial consistency between the fused image and the PAN is quantified by averaging, spatially and spectrally, the fine-scale local correlation of individual super-resolved bands with the high-resolution PAN.

A key qualifying aspect of the proposed indexes is the absence of any resolution downgrading of the input data, which frees the assessment from the effect of scale-dependent phenomena. Experiments on reduced-resolution datasets show that the reprojection indexes are reliable predictors of image quality as quantified by reference-based indexes, supporting their use in the full-resolution domain. On the other hand, experiments on full-resolution data make clear that the local correlation-based index provides indications on image quality that largely agree with the judgement of human experts. The proposed approach can be readily generalized to fusion tasks other than pansharpening, such as, for example, the combination of low-resolution hyperspectral and high-resolution multispectral images.

On the other hand, the user must also be aware of some limitations. In particular, the reprojection requires the knowledge of an accurate estimation of the sensor MTF for a correct low-pass filtering. A wrong estimation would lead to inaccurate spectral consistency assessment, a problem shared with preexisting solutions. Moreover, the proposed correlation distortion index is based on the assumption that the PAN correlates well with all spectral bands. Such a hypothesis is globally acceptable but there can be rare cases, spatially and spectrally well localized, for which it may be too strong.

In order to ensure reproducible research, the code for the proposed indexes can be found at https://github.com/matciotola/fr-pansh-eval-tool/ (accessed on 5 April 2022).