Automatic Fine Co-Registration of Datasets from Extremely High Resolution Satellite Multispectral Scanners by Means of Injection of Residues of Multivariate Regression

Abstract

This work presents two pre-processing patches to automatically correct the residual local misalignment of datasets acquired by very/extremely high resolution (VHR/EHR) satellite multispectral (MS) scanners, one for, e.g., GeoEye-1 and Pléiades, featuring two separate instruments for MS and panchromatic (Pan) data, the other for WorldView-2/3 featuring three instruments, two of which are visible and near-infra-red (VNIR) MS scanners. The misalignment arises because the two/three instruments onboard GeoEye-1 / WorldView-2 (four onboard WorldView-3) share the same optics and, thus, cannot have parallel optical axes. Consequently, they image the same swath area from different positions along the orbit. Local height changes (hills, buildings, trees, etc.) originate local shifts among corresponding points in the datasets. The latter would be accurately aligned only if the digital elevation surface model were known with sufficient spatial resolution, which is hardly feasible everywhere because of the extremely high resolution, with Pan pixels of less than 0.5 m. The refined co-registration is achieved by injecting the residue of the multivariate linear regression of each scanner towards lowpass-filtered Pan. Experiments with two and three instruments show that an almost perfect alignment is achieved. MS pansharpening is also shown to greatly benefit from the improved alignment. The proposed alignment procedures are real-time, fully automated, and do not require any additional or ancillary information, but rely uniquely on the unimodality of the MS and Pan sensors.

1. Introduction

The ever increasing availability of image data featuring a spectral diversity (visible and near-infra-red (VNIR), short-wave infra-red (SWIR), thermal infra-red (TIR), X-band, C-band and L-band microwave data, also with multiple polarizations) and complementary spectral and spatial resolutions, together with the characteristics of the various imaging modalities, have promoted the study of fusion methodologies tailored to remote-sensing images of the Earth. Fusion aims to produce an added value besides those available from the individual data sets. Although fusion results are often interpreted by humans to solve application-oriented tasks (detection of landslides, floods, and burning events, to mention some examples), semi-supervised and even fully automated systems (e.g., thematic classifiers) benefit from fusion products instead of separate datasets.

Extensive studies on the fusion of remotely sensed images for Earth observation (EO) have been accomplished in recent decades, and a significant number of methods has been developed [1]. Image fusion can be labeled according to several criteria. One is based on sensors’ homogeneity. Homogeneous image fusion means that the images that shall be combined are provided by instruments that exploit the same physical mechanism. This class is also known as unimodal image fusion. The fusion of a multispectral (MS) and a panchromatic (Pan) image is referred to as MS pansharpening, and is a typical unimodal fusion. The images subject to unimodal fusion are results of measurements of solar radiation reflected by the scene, though they may span different wavelengths and exhibit different spatial resolutions. Conversely, the fusion of heterogeneous image data, also known as multimodal fusion, pertains to cases in which the data are captured by instruments that do not share the same physical imaging features.

A further way to discriminate among fusion methods is the content level subject to fusion: (a) pixel level (or area-based), (b) feature level, or (c) decision level [1]. Fusion at the pixel level combines the values of pixels in the images that are merged; the goal is to produce a fused image. Feature-level fusion combines specific descriptors, or features, extracted from the individual images that are to be merged. For the fusion of optical and synthetic aperture radar (SAR) images [2], a direct combination of the two datasets is not recommended, in order to avoid contaminating the fused image with the poor signal-to-noise ratio (SNR) of the SAR data. To do so, features calculated from the SAR images (e.g., texture and spatial heterogeneity [3]) can be implanted into the optical images, thus relaxing the tight co-registration requirements peculiar of pixel-level fusion. Decision-level fusion combines the classification maps obtained from each dataset separately, or from different classification algorithms working on the same dataset. The output of decision-based fusion is a classification map.

Among pixel-based methods, MS pansharpening is an established methodology, and is still receiving an ever increasing attention [1,4], especially after the launch of very high resolution (VHR) and extremely high resolution (EHR) spaceborne imaging systems. Pansharpening benefits from the complementary spatial and spectral resolutions of MS and Pan, due to the physical SNR constraints of broad and narrow bands [5]. The goal of pansharpening is the synthesis of a unique product featuring as many spectral bands as are those the original MS image, each having spatial resolution same as that of the Pan image.

After the interpolated MS bands have been overlapped to the Pan image [6], the first step is to check the co-registration between the datasets. This issue is crucial because pixel-based fusion is generally sensitive to misalignment, feature-based fusion is to a lower extent [7], and decision-based fusion is even less sensitive [8]. Then, the geometric details are extracted from the Pan image and added to the MS bands, following a certain injection model. The first step of detail extraction may follow the spectral criterion, originally referred to as component substitution (CS), or the spatial criterion, relying on separable [9,10] or non-separable [11] multiresolution analysis (MRA). Following the spectral criterion, the detail is given as a difference between the original Pan image and a little-detailed intensity component, generated by a pixel-based combination of interpolated MS bands. According to the spatial criterion, the detail is calculated as the difference between the original Pan image and the Pan image smoothed by a lowpass filter, retaining the same content of spatial frequency as the MS bands. Spectral and spatial methods are complementary concerning their tolerance to spatial and spectral artifacts, respectively [12,13]. In principle, all CS methods are intrinsically shift-tolerant [12], though the same property may be obtained by sophisticated convex optimization techniques [14]. It should be noted that, for multimodal fusion, an intensity component makes no sense and, therefore, cannot be defined [15], and the registration of the datasets that will be merged cannot be implicitly achieved by the fusion itself, and should be carried out otherwise [16].

Before fusion is accomplished, the Pan image is radiometrically transformed by an affine grayscale transformation, such that the lowpass-filtered Pan exhibits mean and variance equal to those of the component that shall be substituted [17]. The detail injection model rules the combination of the latter with the interpolated MS image. The injection model is stated between each resampled MS band and a lowpass version of the Pan image. The multiplicative injection model with haze correction [18,19] is essential for improving the fusion performance by leveraging atmospheric imaging mechanisms [20]. The design of injection models is a key topic for multimodal fusion, where the two dataset are the outcomes of different physical mechanisms, like for thermal sharpening [21].

The basic categorization of CS and MRA has been updated considering other methods that have recently been introduced [4], relying on Bayesian inference [22], total variation (TV) regularization [23], and sparse representations [24]. Recently, machine learning paradigms have been exploited: from the pioneering approach based on convolutional neural networks (CNN) [25], to more sophisticated architectures, such as generative adversarial networks (GAN) [26]. For learning-based methods, histogram matching and detail injection modeling are automatically learned from the training data and implicitly performed by the network. However, GANs are capable of controlling each other and, thus, are invaluable, for example, for multimodal fusion [27].

Here, we present two totally unsupervised procedures for the fine co-registration of interpolated MS bands to Pan image. The former is suitable for systems featuring a unique instrument for all MS bands, e.g., GeoEye-1 and Pléiades, as well as GeoEye-2 (which ceased operations in 2019), IKONOS-2, and QuickBird-2 (both dismissed in 2015). The latter is suitable for the two MS instruments of WorldView-2/3. This occurs because the Pan channel is statistically correlated with the MS channels. The fact that a unimodal fusion, as pansharpening is, was potentially capable of overcoming MS-to-Pan misalignment was recognized several years ago [28]. However, until 2020, no alignment of spectral diversity (color hues) was feasible, only of spatial structures [29]. A previous approach is totally different because it is based on the model of the instrument [30], and works with four MS bands and Pan. Another valuable effort is fully automatic, and works with submetric Pan [31], but requires considerable processing resources. The proposed method is real-time, fully automatic, may be used for any satellite system and, in principle, extended to more than three onboard instruments, e.g., WorldView-3.

The organization of the article is the following. Section 2 investigates the causes of misalignment in VHR and EHR satellite MS scanners, and lays the foundation of the fine alignment procedures, one suitable for most systems, featuring two separate instruments, one for MS and another for Pan, and another suitable for WorldView-2/3, having two separate VNIR scanners. Section 3 presents extensive results on the analysis of residues, correction of shifts, and pansharpening for the GeoEye-1 and WorldView-2 datasets. Section 4 discusses the main issues encountered, and explains the advantages of the alignment procedure. Section 5 suggests possible application scenarios.

2. Materials and Methods

2.1. Materials

Since the pioneering very high resolution (VHR) imager IKONOS-2, launched in 1999, MS scanners onboard satellites have attained submetric resolutions in the VNIR wavelengths of the Pan image. The present generation of WorldView satellites for EO features extremely high resolution (EHR) acquisition capability: up to 1.24 m for MS and 0.31 m for Pan, at nadir. The resolution capability of less than one meter from an orbit of heights greater than 600 km is obtained thanks to extremely cumbersome and heavy optics. Constraints in the system design have motivated the adoption of a unique optics for both the MS and Pan scanners. MS bands have a lower spatial resolution, and are acquired with a somewhat elevated value of the modulation transfer function (MTF) at Nyquist frequency (half the sampling frequency). Conversely, the Pan image, typically having four times greater Nyquist frequency, for an MS-to-Pan scale ratio equal to four, is captured with a lower MTF value at Nyquist, but a higher SNR, thanks to the time-delayed integration (TDI), directly implemented on the charge coupled device (CCD). The on-orbit Pan image, being only a little noisy and practically aliasing-free, is then postprocessed for restoration of the MTF to a value similar to those of the MS bands. An edge-crispening filter, i.e., an all-pass filter plus a high-pass filter, is usually applied during the geocoding process, which includes a resampling. The characteristics of the MTF, with shape of a Chinese hat with broad flaps, produces typical values at Nyquist frequency around 0.3 for the MS channels, and less than 0.1, on orbit, for the Pan. Figure 1 illustrates the laboratory MTF of an MS channel of the Pléiades instrument. Note that the spatial frequency response is squeezed along track due to the motion of the platform [32]. Due to diffraction of the optics, the MTF is slightly shrunk as the wavelength increases. Thus, the spatial resolution is slightly better for the leftmost part of the visible spectrum, corresponding to the violet color [33].

In parallel to spatial resolution, the number of spectral channels of new-generation MS scanners has been increased to finely cover the VNIR wavelengths. Figure 2 displays the spectral responsivity channels of a typical 4-band MS + Pan system (GeoEye-1) and the nine spectral channels (4 + 4 + Pan) of WorldView-2.

The MS and Pan instruments share the same optics and, hence, a unique focal plane. Thus, they cannot have strictly parallel axes, but have a small angle that is contained in the orbit plane. The consequence is that the instruments scan different areas at the same instant; equivalently, they scan the same area at different times. Thus, moving objects, e.g., vehicles, appear displaced in the MS and in the Pan images. The MS instrument, however, simultaneously captures all of the bands in a bundle under the same perspective. So, the MS bands are rigidly shifted from one another, a misalignment easily corrected before the data are distributed, but not locally warped by an extent depending on the relief of the terrain.

The Pan instrument images the same scene as the MS instrument does, from a different viewpoint along the orbit, which determines a parallax view. In the presence of discontinuities of the ground surface, and of terrain relief in general, the same object is imaged at different positions in the MS and Pan images. Figure 3, left panel, illustrates acquisition of two separate instruments across the swath. The small angle between the axes of the two instruments is magnified along the orbit, for displaying convenience. The across-track angle encompasses the width of the swath captured by the CCD detector.

The displacement of the same point imaged from different viewpoints is referred to as disparity, and depends on the quote of the point. Whereas in computer stereo vision, disparity is exploited to estimate the quote, in order to reset the disparity to align the same point framed in the two images, its ground height must be previously known. This operation, inverse to stereo restitution, is performed by orthorectification, whose goal is to yield an image same as if it were acquired with the zenith satellite position at infinity distance, which can be exactly superimposed onto a chart. Once the MS and Pan images have been orthorectified, perfect overlap is possible with a rigid shift. The precision of orthorectification, however, depends on the accuracy of the digital elevation model (DEM) used for its calculation [34,35,36]. If the spatial sampling of DEM is too coarse with respect to the pixel size, the quote-driven local warping is insufficient, and a residual shift survives. Another possible cause of error is that DEM usually represents the height of the bare soil, and not of possible natural and human-made structures, typically buildings.

Another viable solution for MS and Pan co-registration is the use of a coordinate transformation based on ground control points (GCP), which are points whose positions in the two images are easily detectable with supervised or semi-supervised techniques. This procedure does not require an accurate DEM, but requires a large number of GCPs, especially if the scene is large and the terrain relief is variable. The sensitiveness to operator’s skill is mitigated by the use of semi-supervised techniques [37]. In any case, these procedures are scarcely reproducible, and may be insufficient for the extremely high-resolution characteristics of modern satellite scanning systems.

The situation described before is even more complicated for satellites equipped with more than two instruments, such as the WorldView-2/3 platforms, in which eight MS bands are taken by two separate instruments. In WorldView-2, three instruments are present on the platform: the classical 4-band MS instrument (MS1), and the new MS instrument (MS2) featuring four more bands, partially interleaved with the others. WorldView-3 is also equipped with a short-wave infra-red (SWIR) scanner. Now, the three instruments acquire different areas at the same instant, or equivalently, they scan the same area at different times. Consequently, moving objects and points at different heights in the scene appear not only shifted in the two sets of MS bands towards the Pan images [38], but the two sets of MS bands are also captured at different moments from distinct positions along the orbit; hence, under different viewing perspectives. Figure 3, right panel, illustrates the acquisition geometry of the WorldView-2 platform. For the sake of clarity, the angles between the optical axes have been intentionally exaggerated.

2.2. Methods

Whenever pansharpening is accomplished, co-registration is usually assumed between the datasets. An inaccurate MS-to-Pan overlap may severely impair the quality of the fusion, depending on the adopted class of methods. CS methods damp small shifts and prevent the geometry of the scene from fading [12,39]. Depending on the injection model, there may be a small residual color change, but not in the structures [29,40]. Instead, the class of methods based on MRA is sensitive to spatial mismatches that cause a loss of synchronization of lowpass and highpass spatial frequency components and the consequent fading of sharp edges and structures. The duality of the classes of spectral (CS) and spatial (MRA) pansharpening algorithms makes the latter robust to spectral impairments, i.e., MS and Pan datasets taken from different satellites and/or at different times [13]. This property is invaluable for multiplatform fusion [41]. Generally speaking, the fusion of heterogeneous datasets (multi-modal fusion), such as thermal and optical datasets, cannot be approached as a problem of CS [21].

For a unimodal fusion, as pansharpening is, a multivariate linear regression between lowpass-filtered Pan, ${\mathbf{P}}_L$, and interpolated MS bands, ${\tilde{\mathbf{M}}}_k$, in which the phase of the linear interpolating kernel should match the acquisition geometry to avoid introducing systematic shifts [6], has been used to model the spectral and spatial relationships between the channels of the two instruments [42]:

$${\mathbf{P}}_L={\widehat{w}}_0+\sum _{k=1}^N{\widehat{w}}_k·{\tilde{\mathbf{M}}}_k+ϵ\triangleq {\widehat{\mathbf{I}}}_L+ϵ$$

(1)

in which ${\tilde{\mathbf{M}}}_k$ is the kth MS band interpolated to the size of Pan, ${\widehat{\mathbf{I}}}_L$ is the LS intensity component, and $ϵ$ the space-varying residue. The set of LS weights, ${\left\{{\widehat{w}}_k\right\}}_{k=0,\dots ,N}$, is calculated as the LS solution of Equation (1). A measure of the matching success in Equation (1) is provided by the coefficient of determination, $R^2$, defined as

$$R^2\triangleq 1-{\displaystyle \frac{{\sigma }_{ϵ}^2}{{\sigma }_{{\mathbf{P}}_L}^2}}$$

(2)

in which ${\sigma }_{ϵ}^2$ and ${\sigma }_{{\mathbf{P}}_L}^2$ indicate the variance of LS residue and of lowpass-filtered Pan image, respectively.

2.2.1. Alignment of GeoEye-1 Data (4-Bands + Pan)

In the seminal article on the spatial properties of MS pansharpening methods [12], the shift-tolerance of CS methods has been mathematically proven. However, no idea was presented of how to exploit these properties in contexts other than those of fusion. It was also unclear how to extend the shift-invariance to the spectral component (color hues), and not only to the spatial structures. In a subsequent short paper [28], the properties of CS methods were analyzed in depth. It was found that the spatial detail injected by CS methods has a lowpass component that has the effect of aligning the expanded MS bands over the Pan image and a highpass component that provides the actual sharpening. In MRA methods, the lowpass component, and hence the alignment, are missing. If the CS method features an intensity component, like that defined in Equation (1), the lowpass component is exactly given by the LS residue, which can be calculated as difference between ${\mathbf{P}}_L$ and ${\widehat{\mathbf{I}}}_L$. The problem was finding an injection model suitable for the shift of both structures and colors. The multiplicative injection model, derived from the transfer model of solar radiation through the atmosphere [20], was first proposed in [29].

With the aim of correcting the interpolated MS bands, we used the space-varying multiplicative gain [43], instead of the projection-based injection gain used in [40], to drive the injection of the residue, $ϵ$, into the interpolated MS bands:

$$\begin{array}{ccc}\hfill {\tilde{\mathbf{M}}}_k^{*}& =& {\tilde{\mathbf{M}}}_k+{\displaystyle \frac{{\tilde{\mathbf{M}}}_k}{{\widehat{\mathbf{I}}}_L}}·ϵ={\tilde{\mathbf{M}}}_k·\left(1+{\displaystyle \frac{ϵ}{{\widehat{\mathbf{I}}}_L}}\right)\hfill \\ & =& {\tilde{\mathbf{M}}}_k·\left(1+{\displaystyle \frac{{\mathbf{P}}_L-{\widehat{\mathbf{I}}}_L}{{\widehat{\mathbf{I}}}_L}}\right)={\tilde{\mathbf{M}}}_k·{\displaystyle \frac{{\mathbf{P}}_L}{{\widehat{\mathbf{I}}}_L}}\hfill \end{array}$$

(3)

in which ${\tilde{\mathbf{M}}}_k^{*}$ is the corrected ${\tilde{\mathbf{M}}}_k$ that overlaps onto ${\mathbf{P}}_L$. Equation (3) highlights that the injection of the residue, $ϵ$, multiplied by the multiplicative gain, i.e., the ratio ${\tilde{\mathbf{M}}}_k/{\widehat{\mathbf{I}}}_L$, into the resampled MS band is equal to its multiplication by ${\mathbf{P}}_L/{\widehat{\mathbf{I}}}_L$.

2.2.2. Alignment of WorldView-2/3 Data (4 + 4 + Pan)

Though never explicitly addressed, the set of MS bands in Equation (1) are implicitly supposed to be aligned to each other. This condition was satisfied until the first satellite of the WorldView MS scanner generation was successfully launched in 2009 [44]. Whenever WorldView-2/3 data are concerned (WorldView-4, which is no longer operational since 2019, featured a unique MS instrument), the geometry of the view allows for the eight spectral bands to be imaged from two different positions along the orbit, both different from Pan, as shown in Figure 3. The lack of an accurate digital surface model (DSM) can cause local shifts between the old bands (blue (B), green (G), red (R), and near-infra-red 1 (NIR1)), and the four bands imaged by the new MS instrument, coastal (C), i.e., violet, yellow (Y), red-edge (RE), and outermost NIR, namely, NIR2. The presence of desynchronization among the components of Equation (1) is expected to locally increase their mismatch with the lowpass Pan.

The problem can be solved by considering the eight MS bands of WorldView-2/3 as produced by two separate instruments, and by splitting Equation (1) between the datasets while, in principle, Equation (1) might be written for all eight bands together. Let $S1$ and $S2$ denote the subscripts of the bands of the old and new MS scanners, respectively. Based on the sequence of the bands arranged by ascending wavelength, $S1=\{2,3,5,7\}$ and $S2=\{1,4,6,8\}$. Equation (1) can be divided into

$$\begin{array}{cc}{\mathbf{P}}_L& ={\widehat{w}}_{01}+\sum _{k\in S1}{\widehat{w}}_k·{\tilde{\mathbf{M}}}_k+{ϵ}_1\triangleq {\widehat{\mathbf{I}}}_{L1}+{ϵ}_1\\ \\ {\mathbf{P}}_L& ={\widehat{w}}_{02}+\sum _{k\in S2}{\widehat{w}}_k·{\tilde{\mathbf{M}}}_k+{ϵ}_2\triangleq {\widehat{\mathbf{I}}}_{L2}+{ϵ}_2\end{array}$$

(4)

in which ${\widehat{w}}_k$, including the offsets ${\widehat{w}}_{01}$ and ${\widehat{w}}_{02}$, are the LS space-constant spectral weights, and ${ϵ}_1$ and ${ϵ}_2$ are the space-varying residues of the matching of old and new instruments towards the unique Pan.

In the case of two separate MS instruments, the correction in Equation (3) is split across MS1 and MS2 and the residues in Equation (4), either ${ϵ}_1$ or ${ϵ}_2$, are injected into the interpolated MS bands of the old and new MS scanners, respectively:

$$\begin{array}{ccc}\hfill {\tilde{\mathbf{M}}}_{k,k\in Si}^{*}& =& {\tilde{\mathbf{M}}}_{k,k\in Si}+{\displaystyle \frac{{\tilde{\mathbf{M}}}_{k,k\in Si}}{{\widehat{\mathbf{I}}}_{Li}}}·{ϵ}_i={\tilde{\mathbf{M}}}_{k,k\in Si}·\left(1+{\displaystyle \frac{{ϵ}_i}{{\widehat{\mathbf{I}}}_{Li}}}\right)\hfill \\ & =& {\tilde{\mathbf{M}}}_{k,k\in Si}·\left(1+{\displaystyle \frac{{\mathbf{P}}_L-{\widehat{\mathbf{I}}}_{Li}}{{\widehat{\mathbf{I}}}_{Li}}}\right)={\tilde{\mathbf{M}}}_{k,k\in Si}·{\displaystyle \frac{{\mathbf{P}}_L}{{\widehat{\mathbf{I}}}_{Li}}}\hfill \end{array}$$

(5)

in which $i\in \{1,2\}$ and ${\tilde{\mathbf{M}}}_k^{*}$ is the corrected version of ${\tilde{\mathbf{M}}}_k$ overlapped onto ${\mathbf{P}}_L$. Equation (5) states that the residue of each scanner, ${ϵ}_i$, should be multiplied by the ratio ${\tilde{\mathbf{M}}}_{k,k\in Si}/{\widehat{\mathbf{I}}}_{Li}$ before being injected into the interpolated MS band. This equals the multiplication of ${\tilde{\mathbf{M}}}_{k,k\in Si}$ by the ratio ${\mathbf{P}}_L/{\widehat{\mathbf{I}}}_{Li}$.

3. Experimental Results

3.1. GeoEye-1 Dataset

A GeoEye-1 image has been acquired over the area of Trenton NJ, USA, on 27 September 2019. The on-orbit limit sampling interval at nadir (half the system resolution) is 1.64 m for MS and 0.41 m for Pan. The spatial sampling interval (SSI) of the resampled geocoded product is 2 m for MS and 0.5 m for Pan; thus, the scale ratio is equal to four. The MS image comprises four bands: blue (B), green (G), red (R), and near-infra-red (NIR). The test image size is 512 × 512 pixels for MS, and 2048 × 2048 for Pan. The radiometric resolution of the packed DN format is 11 bits; the conversion coefficients to top-of-atmosphere (TOA) spectral radiance have been extracted from the metadata. The Pan image and a true color composition of the MS bands (B3, B2, B1) are displayed in Figure 4.

Figure 5 presents the residue of the multivariate regression of the four MS bands towards the lowpass-filtered Pan, both before and after the four bands have been corrected, as in Equation (3). The residue is mostly concentrated on the top of buildings, whose height is presumably unspecified in the available DEM used for the geometric corrections. After correction, the coefficient of determination of the multivariate regression increases from $R^2=0.93914$ to $R^2=0.99810$.

Once the MS bands have been overlapped over the Pan image, simulations of MS pansharpening have been carried out. The goal is to check possible benefits of the alignment for the main class of MRA or spatial methods. From Ref. [12], CS (or spectral) methods are known to be insensitive to spatial impairments of the data. The popular and performing fusion methods that are compared are:

• Gram–Schmidt with spectral adaptivity (GSA) [42], perhaps the most popular among CS methods;
• Brovey transform with haze correction (BT-H) [17], an optimized version of the popular Brovey transform CS method;
• Adaptive wavelet luminance proportional with haze correction (AWLP-H) [45], an optimized version of a popular MRA method;
• Modulation transfer function generalized Laplacian pyramid with full scale (MTF-GLP-FS) injection gains [46], an optimized version of an MRA method based on GLP.

Implementations of pansharpening methods have been taken from the Pansharpening Toolbox.

The results of the fusion of the original (uncorrected) dataset are depicted in Figure 6. What immediately stands out is that, due to the presence of MS-to-Pan shifts on the top of buildings, the two CS methods yield clearer and sharper images than those of the two MRA methods, which are known to be unable to compensate for local misalignment. Instead, once the data have been corrected following Equation (3), the fused images are almost identical to one another. Figure 7 highlights that the two MRA methods recover their top performance, which had been diminished by local shifts.

To provide a deeper understanding of the nature of fusion,

Table 1. GE-1 Trenton: full scale quality indexes without alignment for fusion/assessment.

	QNR	HQNR	FQNR	RQNR
EXP	0.9222	0.8478	0.7131	0.8068
BT-H	0.8792	0.7749	0.8467	0.8575
GSA	0.8634	0.7470	0.8203	0.8303
AWLP-H	0.9125	0.9037	0.9175	0.9057
MTF-GLP-FS	0.8894	0.9038	0.9159	0.8973

and

Table 2. GE-1 Trenton: full scale quality indexes with alignment for fusion/assessment.

	QNR	HQNR	FQNR	RQNR
EXP	0.8974	0.8518	0.7393	0.8504
BT-H	0.8818	0.9085	0.9485	0.9648
GSA	0.8672	0.9050	0.9568	0.9668
AWLP-H	0.8912	0.9212	0.9571	0.9707
MTF-GLP-FS	0.8660	0.9193	0.9537	0.9653

report the values of established full-scale quality indexes [47] on the fused datasets. Full-scale quality assessment means that there is no reference, as in reduced resolution assessment [48], and quality measurement is brought back to the fulfillment of the two consistencies between the fused image and the original MS and Pan, respectively [49]. In fact, it has recently been noted that full-scale quality indexes may be sensitive to shifts in the original datasets [50]. Spatial consistency is crucial in the presence of misalignment [51], depending on the class of methods, either CS or MRA. When the MS bands are overlapped over Pan, they become the new starting point for fusion, but also for carrying out the consistency checks. Therefore, for uncorrected data, the quality reference is provided by the original MS bands that are fused; for corrected data, the new MS bands overlapped to Pan are implicitly the quality assessment reference.

The entry labeled as EXP denotes the absence of injected details. In the case of no alignment, the original data are simply resampled with bicubic interpolation. In the case of realignment, the corrected data are generated at the spatial scale of Pan, and resampling is implicit in the correction procedure. The values of the quality indexes for the fusion of uncorrected data in Table 1 reveal that all CS methods are poorer than the MRA methods, despite the fact that the visual appearance is exactly the opposite. The fusion of corrected MS data with the latter as a reference of consistency turns the situation upside down: all methods, except EXP, are pretty comparable to each other, with AWLP-H being slightly better, according to all indexes. The explanation of this is very simple. If CS methods force the fused image to overlap over Pan, they shift the fused image from the original MS, which is the reference for spectral consistency. Hence, the latter becomes poorer. Instead, with the automatic pre-alignment, both CS and MRA methods do not change the alignment of the data, which is the same as the alignment of the data that are fused. Therefore, consistency measurements are not altered, because all MS data (initial, fused, and reference) are aligned. Eventually, we cannot help noting that the index QNR, dating back to 2008, is moderately sensitive to shifts, and has the asset of recognizing AWLP-H as the best method, but also, the drawback is that the perfect spectral consistency of EXP conceals its poor spatial consistency; hence, EXP is always the best fusion method, which is puzzling. We wish to recall that each index is normalized in [0,1], but in such an interval, the mapping of scales is nonlinear. The indexes are suitable for relatively ranking methods, rather than for providing an absolute quality measure.

3.2. WorldView-2 Dataset

A WorldView-2 image was acquired on the area of Sydney, Australia, on 21 August 2012. The spatial sampling interval (SSI) is 2 m for MS and 0.5 m for Pan. The area is 2048 × 2048 Pan and 512 × 512 MS. Both MS and Pan have been orthonormalized using a DEM available at 10 m. Figure 8 shows the 0.5 m Pan, the true color composition of the bicubically interpolated bands acquired by the MS1 instrument, and the false color composition of the interpolated MS2 bands. The three icons display the data captured by each of the three instruments in Figure 3. There is no misalignment of bands within the same bitmap, corresponding to a unique instrument, but there is misregistration between bands taken by different instruments. The roofs of the buildings are framed with slightly different perspectives, partially corrected during cartographic resampling.

Figure 9 shows the map of the LS residue of the multivariate regression of MS to lowpass-filtered Pan and the related values of matching success, given by $R^2$, defined in Equation (2). The residue is mostly concentrated in built-up structures, for which the DEM used for the orthonormalization was not of adequate spatial resolution. What immediately stands out, however, is the presence of moving vehicles along the main road, whose motion can be detected by looking at Figure 8. The variance of the zero-mean residue is large for the match of MS2, and slightly lower for MS1, which better encompasses Pan, while the outermost bands of MS2 lie outside the bandwidth of Pan. If regression is carried out on the corrected bands of Equation (3), the residue almost vanishes: more for MS1 only, less for MS2 only, due to the different layouts of the bands of the two instruments. Using the eight corrected bands in Equation (5), the regression achieves a value of $R^2$ of 0.99999. Few non-zero residues, due to interpolation errors of moving cars, appear on the road.

The fusion results of BT-H [43] are presented for the original MS data and the MS data corrected for misalignment of the MS instruments. Figure 10 shows two color compositions of the uncorrected and corrected MS data where all eight bands have been fused together. When the fused data are displayed in true color, only data from MS1 are displayed and the shift towards Pan is almost completely absorbed by BT-H (a CS method), analogously to what happened in [29]. The effect of the correction is hardly perceivable in the magnified detail of the car. When the bands displayed are taken from both instruments, the intersensor shift is not completely canceled by the unique residue calculated on all eight bands. The surviving misalignment is visible in the magnified box as a desynchronization of the color hues of the car. After applying the correction in Equation (3), the alignment is recovered. The problem originating with the presence of two separate MS scanners imaging from different parallaxes is that the alignment of MS over Pan cannot be implicitly recovered by CS pansharpening methods, as it happens with only two onboard instruments.

The comparison of the four methods in Section 3.1 on the detail of the main road with moving vehicles is reported in Figure 11, Figure 12, Figure 13 and Figure 14. Without correction, MRA is far poorer than CS on moving cars. With the full correction (see Equation (5)), all methods perform well. In the challenging case of inter-instrument color composition, the bleeding of colors is restrained. According to the proof stated in [12], only shifts of moderate extents can be recovered. The high speed of cars on the main road originates shifts of MS towards Pan of extent up to five-six pixels at the scale of Pan. Such a shift cannot be well approximated by the first-order development of the mathematical function representing the moving vehicle.

4. Discussion

After presenting the results on the two datasets of two different instruments, in which the acquisition geometry is progressively complicated by the presence of two and then three separate instruments, a series of considerations can be made. For scanners with decametric resolution, e.g., Sentinel-2 and the operational land imager (OLI), the problem of misalignment does not exist, either with a unique instrument capturing all the MS bands (Sentinel-2) or also with three separate instruments (OLI), because the pixel size, SSI, is comparable with the spatial scale at which the DEM is generally available (10 m). If the scale of DEM is 10 m, and the scale of Pan is less than 1 m, it becomes impossible to correct the parallaxes with sufficient accuracy. In the presence of discontinuities of the ground surface, the registration error turns out to be a fraction of the scale of DEM, say 25%, with an optimistic view. Thus, a misalignment of up to three pixels of Pan is produced, spread over the height discontinuities of the surface. This problem has been recognized since full-scale quality assessments have become widespread. With fusion and assessment performed on spatially degraded data, typically by a factor of four, all pixel shifts are divided by four and do not produce appreciable misalignment effects. This is the reason why the proposed approach regards datasets that are at least VHR, corresponding to 1 m Pan.

Once the resampled MS bands have been automatically and accurately overlaid onto the Pan image, established MRA pansharpening algorithms recover their original top performance, which had been diminished in the passage from HR to VHR and EHR datasets. Performance is intended both in the visual sense and also according to widespread statistical quality indexes based on consistency measurements of fusion products towards unfused MS and Pan. In other words, the same resampled MS bands corrected for shift compensation should be used as input to the fusion and quality assessment procedures.

According to the theoretical development reported in [12], the proposed pre-processing patches are intrinsically capable of removing possible aliasing artifacts, originated by insufficient sampling step size, from the resampled MS bands, provided that the Pan image is aliasing-free. This is true because the Pan image is taken by sampling the same MTF in Figure 1 with a step size four times greater.

It should be noted that preprocessing patches are independent of the measure unit of the data, e.g., top of atmosphere (TOA) spectral radiance, TOA reflectance, surface reflectance, in floating-point or packed fixed-point formats, due to multivariate regression, whose degree of matching, namely $R^2$, does not change with the data units or formats [52]. In other words, the dataset are correctly realigned, whichever is their initial format, and the corrected datasets retain their original formats.

The proposed method does not require either training datasets or ancillary data (DEM, GCP, etc.), and may work on a unique MS + Pan image. Last but not least, its computational complexity is quite moderate, comparable to that of pansharpening methods that exploit multivariate regression, such as GSA [42] and BT-H [17].

We wish to note that this is a tool to improve the usability of MS and Pan datasets for a variety of possible applications. The datasets are provided by the distributor in a geocoded format. The geo-localization accuracy is stated to be 3 m for GeoEye-1 (the same as for WorldView-2), and is obtained using “a GPS receiver, a gyroscope and a star tracker, without any GCP (Ground Control Points)”. Since an error of 3 m (six Pan pixels!) is too large for crucial tasks, like MS pan-sharpening, we devised a procedure for the fine co-registration of the MS and Pan datasets. We wish to stress the fact that the data have been already co-registered by the provider by using a digital elevation model (DEM) and the geo-localization accuracy measured, e.g., by means of a GPS receiver, is 3 m for both MS and Pan. Unfortunately, a DEM may not contain buildings, or trees, on the top of which most of shifts are concentrated. So, unless a high spatial resolution digital surface model (DSM) calculated via laser scanners is available, as happens for the skyscrapers of the down-towns of most cities, local shifts due to uncompensated parallaxes are unavoidable. In order check whether the procedure is capable of compensating shifts of 3 m, we used information of moving cars, whose shifts can be measured with a certain accuracy. We are not concerned about measuring or improving the geo-localization accuracy, but we want to have interpolated MS and Pan that overlap everywhere as much as possible, because the performance of fusion depends on that.

5. Conclusions

The study has presented a simple idea that has lead to two fully automated real-time procedures. One for the fine co-registration of MS and Pan datasets produced by systems featuring two separate onboard instruments, like GeoEye-1 and Pléiades, among systems that are currently operative. Another, for the MS1, MS2 and Pan datasets produced by the three instruments onboard WorldView-2/3. The key point to detect and correct shifts is the space-varying residue of the multivariate linear regressions between the interpolated MS bands and the lowpass-filtered Pan. Simulations performed on a 4-band GeoEye-1 image and an 8-band WorldView-2 image, both having 2 m MS and 50 cm Pan, show that local shifts up to at least 2 m are compensated, both in geometry and in colors.

The proposed strategy relies on the unimodality of MS and Pan instruments, thanks to which the lowpass filtered Pan image can be synthesized as weighted combination of the four interpolated MS bands of each MS scanner, if there is more than one. The spatial residue of the multivariate regression locally measures the misalignment between the intensity component of the MS bands captured by one instrument and the lowpass-filtered Pan. If such a residue is weighted by the pixel ratio of the interpolated MS band to the synthetic intensity and injected into the interpolated MS band, the latter is aligned to the Pan image, in terms of both spatial structures and color hues. In practice, the correction consists of pixel-by-pixel multiplying the interpolated MS band by the ratio of lowpass-filtered Pan and synthetic LS intensity component.

As a possible development, the effects of haze correction, beneficial for pansharpening [19,43,53], are worth investigating. Specifically, the correction of atmospheric path radiance should be applied to the multiplicative injection gain by which the residue is modulated before being injected to correct the shift of the current MS band.

The proposed automatic fine co-registration procedure can be extended to a fourth onboard instrument: the SWIR MS scanner mounted on WorldView-3. The residue-based alignment procedure might in principle be utilized to finely co-register hyperspectral (HS) and Pan; in addition to fusion, this issue is also relevant for a spectral unmixing that is driven by Pan [54].

As a final remark, it would be intriguing to investigate the behaviors of fusion methods based on learning, when they are trained on correctly aligned MS and Pan data. In fact, there are methods that are capable of learning and compensating for small misalignments [55,56], as happens to all CS methods that operate without learning, but nobody has ever checked the improvements in learning carried out on perfectly aligned MS and Pan datasets.