A Robust Tie-Points Matching Method with Regional Feature Representation for Synthetic Aperture Radar Images

Abstract

The precise tie-points (TPs) on synthetic aperture radar (SAR) images are a critical cornerstone in the global digital elevation model (DEM) and digital ortho map (DOM) production process. While there are abundant studies on SAR TPs matching, improvement opportunities persist in large areas. The correspondences have pixel-level errors during geocoding, which result in misalignment between global products. Consequently, this paper proposed a robust method for SAR images TPs matching, which consists of three key steps: (1) interest point extraction based on the dynamic Harris area entropy (DHAE) grid; (2) adaptive determination of template size; (3) normalized cross correlation (NCC) template matching. DHAE is a regional texture information grid based on the SAR-Harris map, and it is achieved through dynamic block division. Generating the DHAE grid over SAR images enables the extraction of interest points that have regional feature representation and distribution uniformity. A variable-size matching template is adaptively determined based on DHAE to enhance template quality while maintaining computational efficiency. Subsequently, the NCC algorithm is employed to find subpixel-precise correspondences. The proposed method is applied on TPs matching in 57 Terra-SAR images, which cover a large geographical area. Furthermore, the overlapping area is partitioned into five segments according to different coverage types. The experimental results demonstrate that the proposed method outperforms other template matching methods. For all coverage types, the proposed method exhibits high-precision sub-pixel results that reach up to 38.64% in terms of the relative positioning error (RPE), particularly in texture-weak and large areas.

1. Introduction

SAR plays a pivotal role in the observation of the Earth across the air, space, and sea domains [1,2]. Its large-viewing angle satellite platform and strong penetrating microwave provide a unique advantage in generating DEM, DOM, and other products [3,4,5]. In the process of generating DEM and DOM, a localization error may occur, attributed to factors such as variations in attitude, orbital positioning, and terrain undulation [6,7,8]. This can result in misalignment and offsets when stitching large-scale products. To address this, TPs matching is used to find homonymous points on adjacent products [9,10]. This ensures high accuracy of DEM, DOM, and other products generated via SAR [11].

There are two types of method for TPs matching: DEM-aided and registration-based methods [12]. DEM-aided methods utilize terrain constraints to improve the quality of correspondences in areas of terrain relief [13]. Teo et al. [14] selected TPs directly based on geocoded information on overlapping regions of SPOT 5 images with weakly convergent geometric features. Huber et al. [15] validated a TPs selection scheme for the block adjustment for DEM generated by TanDEM-X and proposed the TP chips method. Wang et al. [16] used the DEM produced by TanDEM-X in the proposed CSS-based DEM block adjustment model with a TP-chips approach to select TPs. However, the method of selecting TP-chips on the DEM highly depends on the accuracy of geocoding of SAR images. Due to a variety of complex factors, the accuracy of geocoding is not always accurate. Therefore, Ravanbakhsh et al. [17] used the NCC-DEM method to extract the TPs. Wang et al. [18] developed a NCC-CSM method to solve the NCC-DEM mismatch problem in flat areas.

In contrast to the DEM-aided method, the registration-based TPs matching technique solely relies on the SAR intensity image’s attributes. This approach eliminates the errors that could be caused by additional sources and is particularly ideal for TPs matching with SAR orthoimages. Image registration methods fall into two categories: area-based and feature-based methods [19]. The area-based approach, also known as template matching, involves selecting a template on the reference image and selecting the corresponding sliding window on the sensed image [20]. Area-based algorithms like NCC, phase correlation (PC), mutual information (MI), sum of squared differences (SSD), and structure similarity index measure (SSIM) have shown excellent matching results [21,22,23]. Feature-based matching techniques are also extensively employed in TPs matching since they do not require a priori coarse offset and are computationally efficient [24]. In general, compared with feature-based methods, area-based methods have stronger terrain adaptability because they focus more on the statistical characteristics of the image itself than structural features, which is very desirable in TPs matching over large areas. The area-based method needs to start from two established initial points, the location of which will directly affect the quality of the matching results. The traditional template matching method uses regular grid points as interest points, which are in locations that have no real matching meaning, reducing the number of correct matches. Furthermore, due to the speckle noise on SAR images, unstable image features are misused for template matching. Conventional interest point detectors are not immune to speckle noise, which increases the number of mismatches [25]. Heinrich et al. [26] used the Harris corner point detector to select interest points in a proposed modality-independent neighborhood descriptor (MIND) for multimodal image registration. Ye et al. [27] used the block Harris method to extract feature points and ensure their uniform distribution on optical and SAR images, and they matched them using a histogram of orientated phase congruency (HOPC) similarity metrics.

However, the traditional template matching method does not consider the structural features at the interest point or only considers the feature response at a single point, which will bring many problems, such as uneven matching results, mismatching on the weak-texture area, etc. Extracting the interest points which have regional feature representation is one way to improve the quality of template matching.

After analyzing the current methods, it has been discovered that there are several issues present. First, the interest points’ lack of regional feature representation. Second, mismatching occurs in weak-texture regions. Almost all template matching methods use fixed-size matching patches, which will not be applicable in weakly textured areas of the image. Weakly textured areas require a larger perceptual range, while only a small patch size is required at the obvious structural information to match successfully. Third, there are still improvements in automaticity and computational efficiency. The template matching method has high computational complexity, requiring hardware acceleration when applied to large-scale image stitching.

To address these issues, a new and robust SAR images TPs matching method has been proposed in this paper. First, a dynamic block Harris area entropy (DHAE) method has been introduced to identify the main structures on the image using the SAR-Harris method. The dynamic division idea guarantees that the interest points have an overall uniform distribution and parameter adaptivity. The information entropy approach ensures the structural robustness of the region around the interest points. Second, an adaptive NCC template matching window, based on the DHAE grid, has been developed to select the TPs template window containing the closest neighboring structures. This enhances the reliability of the matching results in the weak-texture region.

The remainder of the paper is organized as follows: Section 2 outlines the proposed methodology, specific operational details, and accuracy evaluation metrics; Section 3 describes the experiments that were designed and carried out for validation; the proposed method is discussed in Section 4; lastly, Section 5 summarizes the findings and draws conclusions.

2. TPs Extraction Strategy and NCC Matching

2.1. Overall Process

This paper proposed a new method for TPs matching that is different from the co-registration used in InSAR processing. The method is designed for stitching large-scale SAR images or DEMs produced by InSAR technology and aims to provide reliable planar control TPs for subsequent block adjustment. To ensure the reliable operation of large-scale image stitching, the method focuses on solving the mismatching problem caused by weak-texture and complex terrain conditions, taking into account the robustness and efficiency of large-scale computation.

The first step in the process is to extract overlapping regions from the geocoded SAR images sequence. As shown in Figure 1. The images acquired by the along-track and adjacent-track orbits of the SAR satellite cover a large area, and there is a certain overlapping area between the neighboring images. The TPs’ matching work is carried out on the overlapping area, and the positions of the homonymous point on different images should be one-to-one correspondence after matching. The TPs must be evenly distributed on the overlap area to provide accurate geometric constraints for the block adjustment. Selecting regular grid points directly can maximize uniformity. However, this is not a good choice in the case of complex terrain, because the matching effect is not ideal in weakly textured areas and rugged surfaces due to the imaging modes of the SAR images. After the geocoding step of the SAR images, each pixel has the corresponding latitude and longitude coordinates under the geographic coordinate system. Although this result is not very accurate due to the baseline error, terrain undulation, and simulated SAR images error, we can still find the initial correspondence of the interest points with the help of this positional relationship. In the initial fine matching of interest points, the same size of the template window for all points is not suitable for large-scale matching applications because for some structurally rich areas with large terrain undulations or SAR images acquired from the same orbit, a smaller search window is usually able to achieve higher matching accuracy. However, for the images from adjacent orbits, due to the long imaging interval, inconsistent imaging angles geocoding distortions due to perspective shrinkage, etc., all lead to the need for a larger search window in order to reach the reliable threshold of the correlation coefficient.

The processing of the proposed SAR TPs matching method is represented as follows. First, the overlapping region is extracted from geocoded SAR images, and subsequently, the matching list of reference and sense image pairs is generated. For an image pair matching process, the proposed robust TPs matching method is applied, which is represented as follows. The TPs matching scheme is represented in Figure 2.

• Extracting uniform interest points in the reference image, and the specific method is introduced in Section 2.2.
• Calculating a suitable template matching window for each interest point, and the design scheme of the window is introduced in Section 2.3.
• Applying the NCC sub-pixel offset search method for each overlap. After all image matching is executed, a list of correspondences for the overall region is generated.

2.2. Extraction of Interest Points

Currently, most of the template matching methods use a regular grid to generate interest points. Although they are uniformly distributed, they may fall in some weakly textured areas without actual features inevitably, and these interest points will then bring errors to the subsequent matching process.

2.2.1. SAR-Harris Detector

SAR-Harris detector can effectively suppress speckle noise. Reference [28] has shown that corner points on images can be accurately extracted using Harris features. However, this method is not necessarily applicable to SAR images, and its main problems are as follows: First, compared to regular grid points, the Harris method makes it difficult to control the uniform distribution of the extracted interest points on the image space and can achieve good results in some respects, such as specific target detection, but it is difficult to ensure the stability of the accuracy in calculating the overall offset of the image. Second, the widespread multiplicative scattering noise brings great trouble for Harris corner detection, and the detected corners are often localized at the noise points. In their proposed SAR-SIFT algorithm, Dellinger et al. [29] redefined the computation of Harris values by horizontal and vertical gradient as

$$\left\{\begin{array}{c}{G}_{x,\alpha }=\log \left(R_{1,\alpha }\right)\\ G_{y,\alpha }=\log \left(R_{3,\alpha }\right),\end{array}\right.$$

(1)

where $G_{x,\alpha }$ is the vertical gradient and $G_{y,\alpha }$ is the horizontal gradient. $R_{1,\alpha }$ and $R_{3,\alpha }$ denote the ROA on the vertical and horizontal gradients, and $\alpha$ is the scale parameter in the ROEWA operator proposed by Fjortoft [30]. The SAR-Harris matrix $C\left(x,y,\alpha \right)$, after redefining the gradient, with the detection function $R\left(x,y,\alpha \right)$, is denoted as

$$\left\{\begin{array}{c}C\left(x,y,\alpha \right)={Gs}_{\sqrt{2}\alpha }\star \left[\begin{array}{cc}{\left(G_{x,\alpha }\right)}^2& \left(G_{x,\alpha }\right)·\left(G_{y,\alpha }\right)\\ \left(G_{x,\alpha }\right)·\left(G_{y,\alpha }\right)& {\left(G_{y,\alpha }\right)}^2\end{array}\right]\\ R\left(x,y,\alpha \right)=det\left(C\left(x,y,\alpha \right)\right)-d·{tr\left(C\left(x,y,\alpha \right)\right)}^2,\end{array}\right.$$

(2)

where ${Gs}_{\sqrt{2}\alpha }$ is a Gaussian convolution kernel with $\sqrt{2}\alpha$ as standard deviation, the operator $\star$ denotes convolution, and $d$ is an empirical parameter.

The SAR-Harris features in the form of multi-scale parameters are expressed in (2). Multi-scale feature points are critical in processing SAR images in a radar coordinate system or with different zoom and rotation characteristics. However, for SAR image stitching in the geographic coordinate system, the scale parameter is not necessary, which introduces matching errors and brings additional computational burden. Therefore, we use the single-scale form of the SAR-Harris. It is expressed as

$$\left\{\begin{array}{c}C\left(x,y\right)={Gs}_{\sigma }\star \left[\begin{array}{cc}{\left(G_{x,\sigma }\right)}^2& \left(G_{x,\alpha }\right)·\left(G_{y,\sigma }\right)\\ \left(G_{x,\sigma }\right)·\left(G_{y,\sigma }\right)& {\left(G_{y,\sigma }\right)}^2\end{array}\right]\\ R\left(x,y\right)=det\left(C\left(x,y\right)\right)-d·{tr\left(C\left(x,y\right)\right)}^2,\end{array}\right.$$

(3)

where ${Gs}_{\sigma }$ denotes the standard Gaussian convolution kernel with standard deviation $\sigma$, where $\sigma$ is a constant. Similarly, the ROEWA operator for horizontal and vertical gradients uses the constant $\sigma$ as the scale parameter.

2.2.2. Dynamic Block Division

For conventional SAR-Harris corner point detectors based on gradient amplitude variation, interest points appear at locations with large gradients in the response values. However, the distribution of these points is spatially correlated. Consequently, the direct application of the SAR-Harris detector does not foster the uniform extraction of interest points. This non-uniform distribution of interest points frequently leads to localized distortions within the transformed model. In their work [31], Ye et al. introduced the concept of block Harris on optical remote sensing images. This approach involves dividing the Harris results of the original image into $m\times n$ blocks. Within each block, the top $k$ results are selected based on their ascending order of Harris response values. These selected results serve as the interest points for feature matching, aiming to achieve a more uniform distribution effect. Nonetheless, when applying this blocking method in the practical context of large-scale SAR image stitching, controlling the parameters becomes a challenging task. Additionally, this method may not effectively distinguish against the presence of abnormal reflection values within SAR images. These abnormalities often manifest as the aggregation of local control points.

Therefore, we adopt the following scheme: First, fix the parameter $k$, take only the most regionally representative SAR-Harris response point in each block, i.e., $k=1$, and the judging method of regional representativeness will be introduced in Section 2.2.3. Second, set the number of blocks $m\times n$ according to the size of overlapping regions, and usually, we set the initial size of the block to invert $m$ and $n$; the initial size of block is set to a smaller value $B_s\times B_s$ so that the initial number of interest points in overlap area is $m\times n$. Third, when extracting the regionally representative interest points in the block, if there are no interest points in the block that satisfy the condition, the block merges with the neighboring $l$ block and inherits the $l\times k$ of interest points.

By adopting the dynamic division approach, we can effectively enhance the uniformity of the initial distribution of interest points and the representation of regional features. Simultaneously, this approach leads to a reduction in the number of parameters, rendering it better suited for adaptive operation in large-scale SAR image TPs matching programs. This refined approach is especially valuable for managing the complexities inherent in handling a substantial volume of SAR images.

2.2.3. Area Entropy

While the block Harris method provides feature representation, it remains rooted in capturing gradients in the vicinity of a given point. However, in image template matching, achieving a higher success rate relies on the diversity of features and textures within the template. To address this, we propose that with dynamic block division processing, the extraction of interest points should prioritize template surface features over point-based features. The quantity of information within an image serves as an indicator of the template region’s complexity. Consequently, when identifying a template window with increased information in the SAR-Harris map, we can consider the center point of that window as the focal point for matching. This approach ensures a more comprehensive consideration of features, potentially enhancing the effectiveness of the matching process.

An effective metric for calculating the amount of information in the image is entropy. Image entropy is a discrete 2D expression of Shannon entropy, which represents a measure of the degree of information present in a target image by defining the probability of a pixel appearing in the image [32]. In a 2D image, the greater the information uncertainty, the greater the entropy value, and the more information the image may contain.

For an $n$ state system, its Shannon entropy is defined as

$$H=-\sum _{i=1}^n{p}_i\log \left(p_i\right),$$

(4)

where $p_i$ denotes the random probability that event $i$ occurs. There is

$$\sum _{i=1}^n{p}_i=1,0\le p_i\le 1.$$

(5)

Image entropy is the discrete Shannon entropy in two dimensions. For a digital image of size $m\times n$, suppose its gray level is $L$. Then, there exists

$$p_l=\mathrm{Pr}\left(X=l\right)=\frac{n_l}{T,}$$

(6)

where $p_l$ denotes the probability of occurrence of a pixel in a digital image, $l$ is a certain gray value in an image with gray level $L$ ($l\in \left\{\mathrm{0,1},\cdots ,L,L-1\right\}$), $n_l$ denotes the number of pixels with pixel intensity level $l$ in this image, and $T$ denotes the total number of pixels in the image ($T=m\times n$). According to (4), there is

$$H_{im}=-\sum _{i=1}^{L-1}{p}_l\log \left(p_l\right),$$

(7)

where $H_{im}$ denotes the information entropy of the digital image, obtained by bringing (6) into (7). There is

$$H_{im}=-\sum _{i=1}^{L-1}\frac{n_l}{T}\log \left(\frac{n_l}{T}\right).$$

(8)

Image entropy is widely used in the field of image information quantity estimation, uncertainty measure, and image encryption [33]. Therefore, based on applying block Harris processing to the image, the image entropy is calculated by selecting a sub-window of matching template size in each block. Combined with the idea of dynamically dividing blocks mentioned in Section 2.2.1, this DHAE grid can accurately respond to the amount of information in the vicinity of the target and improve the feature representativeness of the interest points. The step of the sliding window on the block determines the resolution of the DHAE grid.

2.3. Adaptive NCC Template Matching

2.3.1. Principle of NCC Algorithm

As a classical homologous image matching algorithm, NCC is characterized by high accuracy, simple structure, and strong robustness, especially in the task of image registration with only planar position offset. Whether in the block adjustment of InSAR-DEM or SAR-DOM, the task of relative position correction between intensity images requires accurate and stable sub-pixel level matching results. Therefore, we adopt the NCC algorithm to perform relative position correction between overlapping regions of homologous SAR images.

In the image registration algorithm, the NCC algorithm is essentially an area-based searching algorithm, which calculates the NCC coefficients between two templates, slides the template window over the search area to obtain the location where the largest NCC coefficient is located, and considers the center point of the window with the largest NCC coefficient value as the pixel-level matching result.

NCC coefficient representation is

$$NCC=\frac{1}{n}\sum _{x,y}\frac{1}{{\sigma }_f{\sigma }_t}\left(f\left(x,y\right)-u_f\right)\left(t\left(x,y\right)-u_t\right),$$

(9)

where $n$ denotes the number of window pixels, ${\sigma }_f$ and ${\sigma }_t$ denote the standard deviation of the template window and the search window, respectively, $u_f$ and $u_t$ denote the mean value of the template window and the search window, respectively, and $f\left(x,y\right)$ and $t\left(x,y\right)$ are the pixel values of the template window and the search window at the $\left(x,y\right)$ position.

2.3.2. Template Size Adaptive Determination

In Section 2.2.3, we give the computation method of the DHAE grid and select an example region to show its DHAE grid results. In the NCC matching phase of the proposed method in this paper, we determine the size of the matching template based on the DHAE grid. The interest points extracted via DHAE do not necessarily have a very strong structure in a flat area, although it has been considered as the most representative area in this block, at this point, the variable template window size is very favorable for the computational accuracy of NCC matching. The process of determining this window is fully adaptive, relying only on the DHAE grid.

The specific operation method is as follows. First, calculate the position of the largest value in the DHAE of a block and record. Second, calculate the position of the next largest value in this block and record. Third, if the ratio of largest value and next largest value is smaller than the peak-side-lobe ratio (PSLR) proposed in reference [18], re-adjust the size of the window to be the smallest window that can include the largest value and the next largest value.

2.3.3. Subpixel Localization

Subpixel results for template matching methods can generally be obtained via either interpolation or fitting, where interpolation is achieved by targeting a small window (usually 3 × 3 or 5 × 5) in the region near the peak of the NCC in the template matching and up-sampling it to the target accuracy of the subpixel matching [34]. The fitting method obtains the subpixel location of the extreme value of the subpixel NCC by fitting a quadratic surface to the 2D plane where the NCC value is located using, for example, a known regular grid point. The method of obtaining subpixel alignment results using quadratic surface fitting has a great efficiency improvement compared with interpolation, and when the subpixel accuracy is required to be high, for example, 0.1 pixel for a 3 × 3 window, it needs to be interpolated to a size of 30 × 30, which makes the interpolation results very imprecise. Therefore, we use quadratic surface fitting to obtain the subpixel-level offsets. The corresponding fitting function is denoted as

$$P\left(x,y\right)=a_0+a_{1}x+a_{2}y+a_3{x}^2+a_{4}xy+a_5{y}^2.$$

(10)

The extreme points of the surface $P\left(x,y\right)$ are satisfied as follows:

$$\left\{\begin{array}{c}\frac{\partial P(x,y)}{\partial x}=a_1+2a_{3}x+a_{4}x=0\\ \frac{\partial P(x,y)}{\partial y}=a_2+2a_{4}x+a_{5}y=0.\end{array}\right.$$

(11)

Solving (11) yields the location of the extreme point of the NCC coefficients on the surface, expressed as

$$\left\{\begin{array}{c}∆x=\frac{2a_1{a}_5-a_2{a}_4}{{a_4}^2-4a_3{a}_5}\\ ∆y=\frac{2a_2{a}_3-a_1{a}_4}{{a_4}^2-4a_3{a}_5}\end{array}\right..$$

(12)

Thus, the final sub-pixel level alignment offset is expressed as

$$\left\{\begin{array}{c}nx=x_0+∆x=x_0+\frac{2a_1{a}_5-a_2{a}_4}{{a_4}^2-4a_3{a}_5}\\ ny=y_0+∆y=y_0+\frac{2a_2{a}_3-a_1{a}_4}{{a_4}^2-4a_3{a}_5,}\end{array}\right.$$

(13)

where $x_0$ and $y_0$ are the initial pixel-level offsets, and $nx$ and $ny$ are the final sub-pixel-level alignment offsets, respectively. Since the NCC values are known for the regular positions on the extracted window, such as (1, 1), (2, 2), etc., the surface fitting coefficients $a_0,a_1\cdots a_5$ can be obtained using the least squares method to fit the surface equations in (10).

2.4. Evaluation Criterion of TP Matching

For the needs of subsequent block adjustment and the stitching of large-scale SAR images, it is worth paying attention to the following in the TPs matching:

• Reliability: The refined correspondences obtained via template matching can be used to initially measure the quality of matching based on the peak value of the similarity map and the standard deviation except for the peak value, and the threshold value of the correlation coefficient in NCC similarity matching is usually 0.2.
• Accuracy: In InSAR-DEM or SAR-DOM production, it is difficult to ensure that each image area has a stable ground control point on the surface. The manual selection of control point check will be affected by factors such as point selection error and radiation error. It cannot achieve sub-pixel accuracy check, so usually, it calculates the least squares polynomial fitting accuracy to match the relative accuracy of the results.
• Uniformity of distribution: Since the TPs need to provide robust plane position constraint relations for the neighboring images with overlapping regions where they are located, distribution uniformity needs to be ensured over the overlapping regions at the TPs in order to avoid the impact of local distortion on the transformation relations of the overall region.

According to the above requirements, this paper will evaluate the accuracy of the matching results of the TPs through four metrics: stable ratio (SR), stable uniformity (SU), standard deviation (STD), and relative positioning error (RPE), as proposed in [35]. SR represents the proportion of reliable match points (determined by the cross-correlation threshold) to the total number of match points on an overlapping image pair, calculated as

$$SR=\frac{N_s}{N},$$

(14)

where $N_s$ is the number of correspondences with a correlation coefficient threshold greater than 0.2, and $N$ is the total number of TPs. This ratio can represent the reliability of matching TPs.

SU stands for distribution uniformity of stable correspondences. This is reflected by the distribution of reliable TPs over overlapping areas. Here, the method proposed in [35] is adapted in this paper to divide the overlapping area into 10 × 10 blocks, and if more than one stable correspondence exists on each block, the number of stable blocks is counted. The specific calculation method is

$$SU=\frac{B_s}{B},$$

(15)

where $B_s$ is the number of more than 1 stable TP contained in the divided block, and $B$ is the total number of blocks.

STD represents the standard deviation of the fit result of the least squares polynomial of TPs and is often used to measure the relative accuracy. Here, in order to increase the robustness of the accuracy of the results, we choose to extract half of the correspondences to participate in the least squares polynomial fitting, and the other half will be substituted into the fitting polynomial to calculate the RPE of the offset result. The polynomial fit is expressed as

$$\left\{\begin{array}{c}{O}_{lat}=T{\times P}_{lat}\\ O_{lon}=T\times P_{lon},\end{array}\right.$$

(16)

where $O_{lat}$ and $O_{lon}$ represent the latitude and longitude direction offsets column vector, and the dimension is $n\times 1$; $P_{lat}$ and $P_{lon}$ represent the fitted polynomial coefficient matrix of latitude and longitude, and the dimension is $MA\times 1$; and $MA$ is the number of coefficients of the polynomial. $T$ is the corresponding coordinate matrix in polynomial fitting, and the dimension is $n\times MA$, when $MA=4$. It is expressed as

$$T=\left[\begin{array}{cc}\begin{array}{cc}\begin{array}{c}0\\ ⋮\\ 0\end{array}& \begin{array}{c}{x}_1\\ ⋮\\ x_n\end{array}\end{array}& \begin{array}{cc}\begin{array}{c}{y}_1\\ ⋮\\ y_n\end{array}& \begin{array}{c}{x}_1{y}_1\\ ⋮\\ x_n{y}_n\end{array}\end{array}\end{array}\right].$$

(17)

The STD of the fitted offsets is expressed as

$$\left\{\begin{array}{c}{E}_{lat}=\sqrt{\frac{{\sum }_{i=1}^n{({Offlat}_i-{O_{lat}}_i)}^2}{n-1}}\\ E_{lon}=\sqrt{\frac{{\sum }_{i=1}^n{({Offlon}_i-{O_{lon}}_i)}^2}{n-1,}}\end{array}\right.$$

(18)

where $E_{lat}$ and $E_{lon}$ represent the standard deviation of latitude and longitude direction, and $Offlat$ and $Offlon$ are the latitude and longitude offsets of match the original result.

We bring the other half of the matching offset into the polynomial parameters calculated in (16) and calculate the RPE, which is expressed as

$$\left\{\begin{array}{c}{R}_{lat}=\sqrt{\frac{{\sum }_{i=1}^n{({{Offlat}^{\prime }}_i-{{O_{lat}}^{\prime }}_i)}^2}{n}}\\ R_{lon}=\sqrt{\frac{{\sum }_{i=1}^n{({{Offlon}^{\prime }}_i-{{O_{lon}}^{\prime }}_i)}^2}{n,}}\end{array}\right.$$

(19)

where $R_{lat}$ and $R_{lon}$ represent the RPE of latitude and longitude direction, and ${Offlat}^{\prime }$, ${Offlon}^{\prime }$ are the latitude and longitude direction offsets of the other half of the check points. ${O_{lat}}^{\prime }$ and ${O_{lon}}^{\prime }$ are the fitting offsets calculated by bringing the other half of the checkpoints into the fitted polynomial coefficient.

3. Experiment Results

3.1. Research Areas and Experimental Environments

To test the performance of the proposed method for TPs matching in large-scale SAR images, 57 SAR images acquired by the TanDEM/TerraSAR dual-star X-band SAR are selected in this paper, which roughly covers 5° × 5°, as shown in Figure 3. Image parameters are listed in

Table 1. Parameters of Terra-SAR images in this experiment.

Items	Parameter
Sensor	TSX-1
Sensing area	Spain Baleares
Image number	57
Coordinate	Geo Lat/Lon
Data format	Geotiff
Datatype	Float
Coverage	0°54′ to 4°31′E, 38°30′ to 40°16′N
Time span	21 March 2011 to 6 November 2013

. Due to the large number of selected images, the large coverage area, and the wide distribution of feature types, it is possible to more effectively test the performance of the proposed method in texture-rich areas (mountainous areas, cities) as well as weakly textured areas (plains, farmlands). There are both along-track and adjacent-track images, and the latter requires more robust matching algorithms than the former to solve the distortion after geocoding from different viewpoints. Moreover, the large scale of the images poses a great challenge to computational efficiency, which is very suitable for testing the efficiency and feasibility of the GPU-parallel NCC algorithm used in this paper.

In terms of the experimental environment, the CPU of the workstation for data processing in this paper is the AMD Ryzen Threadripper 3970X 32-Core Processor with 256 GB of RAM, and the GPU is Nvidia RTX 2080Ti with 11 GB of Global RAM. The external DEM used for geocoding is the 30 m × 30 m resolution SRTM global DEM provided by NASA.

3.2. Analysis of Matching Accuracy and Distribution Uniformity of TPs

3.2.1. Experiment Settings

The geocoded SAR images in this experiment have a total of 79 valid overlap areas. In order to determine the matching robustness of the proposed method in different regions and facilitate the display of the experimental results, this paper divides the overlapping regions with reliable matching numbers into five parts according to the characteristics of overlapping regions. A: steep area; B: flat area; C: mixed area; D: small area; E: large area. The areas illustrate overlapping pairs of imagery, as shown in Figure 4.

We select the NCC and MI algorithms that are usually used with regular grid (expressed as RG) points as the interest points, and the NCC method that uses the block Harris (expressed as BH) method to select interest points and compare them with the DHAE-NCC method proposed in this paper. To ensure that the number of points of interest in the four methods is as consistent as possible, the experimental parameters are set as follows. The regular grid point interval is 256 pixels, the block size of the BH method is 512 × 512 pixels, five points of interest are extracted from each block, the Harris response threshold is 0.5, the size of the DHAE grid is 256 × 256 pixels, and the regional entropy template size is 64 × 64 pixels. The threshold for NCC peaks is 0.2, and for MI peaks, it is 7.0.

3.2.2. Experimental Results

Since the DHAE grid method dynamically adjusts the window size of template matching when determining the interest point, the influence of template size on matching accuracy is analyzed first. Matching experiments were performed on the SAR images of data pair A with overlapping areas, which incorporates more image texture detail so that the matching template can range from small to large. Four methods—RG-NCC, RG-MI, BH-NCC, and DHAE-NCC—were used for comparative experiments, which changed the template size. SAR images of matching experimental areas under different template sizes are shown in Figure 5. The matching error is represented by the STD of the fitted polynomial, and Figure 6 shows the matching accuracy of the methods at different template sizes.

As shown in Figure 6, the accuracy of the matching methods gradually increases when the matching template gradually increases and tends to stabilize when the template size achieves 64. The matching accuracy of the overall longitude direction is better than the latitude direction because the azimuth direction of SAR satellite imaging resolution is better than the range direction. Since DHAE is a matching TPs selection framework for variable templates, the DHAE method shown here does not change as the window is raised.

The DHAE method is calculated only once in the window experiment, so it is represented as a constant. Observing the distribution of DHAE windows ratio, it is easy to find that windows of different sizes are distributed in matching, and 64 is the peak of window size. Compared with the NCC method, the accuracy of the MI method is poor, and the interest points selected via the block Harris criterion show little difference in NCC matching accuracy from the NCC matching of regular grid points. However, the DHAE method accurately selects the interest points with actual matching significance and reasonably determines the matching template according to the real feature information near the TPs. Therefore, in the case of relatively small templates used, the accuracy that can only be achieved by other methods under large matching templates is achieved. The matching accuracy and distribution of the proposed DHAE-NCC method in the overlapping region were analyzed in detail via SR, SU, STD, and RPE. And we compared RG-NCC, RG-MI, and BH-NCC. Based on the results of the window experiment, all algorithms, except DHAE-NCC, used a template size of 64 × 64 pixel and searched on the sliding area of 64 × 64 pixel. The specific experimental results are shown in Figure 7 and Figure 8 and

Table 2. Reliability, accuracy, and distribution uniformity results of four matching methods on different datasets.

Pair	Number	Method	SR (%)	SU (%)	STD (pixel)	RPE (pixel)	Time (s)
Pair A	29	RG-NCC	99.66	75.73	0.2649	0.3039	170.25
		RG-MI	77.67	71.16	0.2863	0.3252	226.75
		BH-NCC	99.93	47.47	0.2537	0.2770	394.25
		DHAE-NCC	99.89	65.87	0.2163	0.2639	132.51
Pair B	23	RG-NCC	79.06	84.99	0.4124	0.6535	145.38
		RG-MI	41.73	76.90	0.6464	0.9046	195.70
		BH-NCC	76.40	68.28	0.4055	0.6345	327.89
		DHAE-NCC	81.85	77.14	0.3393	0.5550	108.96
Pair C	27	RG-NCC	87.42	83.44	0.3482	0.4727	220.56
		RG-MI	63.67	79.11	0.4728	0.5760	303.55
		BH-NCC	85.06	60.23	0.3556	0.4527	418.14
		DHAE-NCC	88.57	73.15	0.2805	0.4807	143.15
Pair D	45	RG-NCC	97.21	80.97	0.2854	0.3268	397.02
		RG-MI	66.22	72.38	0.3330	0.3747	562.26
		BH-NCC	96.28	56.15	0.2621	0.2782	705.84
		DHAE-NCC	97.78	69.62	0.2264	0.3237	244.25
Pair E	34	RG-NCC	79.25	81.19	0.4036	0.6441	279.64
		RG-MI	57.39	79.75	0.6161	0.8508	483.82
		BH-NCC	77.03	60.20	0.4262	0.6568	595.24
		DHAE-NCC	81.49	74.31	0.3372	0.5538	212.93

. According to the matching experimental results of the five image pairs of the dataset, the DHAE-NCC algorithm proposed in this paper shows a good matching effect. In all terrains, the average matching accuracy can reach the subpixel level. The overall matching results performed better in steep areas, with lower STD and RPE, which is caused by the high geocoding accuracy in mountainous areas. However, flat areas have relatively poor matching results. This is due to the lack of texture in the simulated SAR images generated via DEM.

During the geocoding process, flat areas lead to a shift in matching with the SAR images in the radar coordinate system. When looking for the interest points, the DHAE-NCC algorithm selects the interest point with relatively strong characteristics based on the SAR-Harris map and increases the number of reliable matches without greatly changing the number of TPs; so, it has a high SR.

DHAE uses the principle of dynamic regional entropy to ensure the uniform distribution of TPs under the overall image coordinate system, and the SU is significantly improved compared with the block Harris method. However, compared with the regular grid method, the DHAE method has a slight disadvantage. In terms of internal and external accuracy checking, the MI method belonging to the RG selection strategy has a large error between STD and RPE due to the limitation that the mutual information entropy is greatly affected by the speckle noise of SAR. The performance of RG-NCC and BH-NCC is the same because the BH selection idea only relies on the Harris response value of a single interest point, so the template matching where the interest point is located is not necessarily the regional optimal matching template. The variable template size of DHAE plays a key role in weak-texture areas such as flat areas, which solves the problem that it is difficult to balance matching accuracy and matching efficiency with traditional fixed-size templates. Therefore, the STD and RPE errors in Pair B and Pair E are small. However, we noticed that BH-NCC slightly outperformed our method on RPE on datasets C and D. It is speculated that BH-NCC is able to extract feature information more accurately in small regions where features are more concentrated. In terms of computational consumption, the regular grid method does not need to spend too much time on point selection, and the MI method takes more time than the NCC method because the joint histogram of the template is calculated in the matching template. The BH method constructs the SAR-Harris space to increase the computational cost, and in DHAE, the regional entropy is calculated to build the matching point selection framework. Fortunately, since most of the SAR image matching operations can be run independently, the coupling calculation amount is small, which is very suitable for GPU-CUDA parallel computing. Therefore, in the proposed DHAE-NCC method, we use the GPU to accelerate the construction of the DHAE grid and NCC matching operation, which has been greatly improved in computational efficiency.

4. Discussion

In this section, the generation of interest points is discussed first. Then, the parameter values of DHAE and the distribution of variable templates are discussed. Finally, the limitations of DHAE are discussed.

4.1. Discussion on Interest Points Generation and Template Size Determination

We analyze the generation of TPs for a real SAR image in the dataset. A farmland area containing structural information was selected, and the feature extraction effects of different methods were compared, as shown in Figure 9.

Based on the comparison results in Figure 9, it is clear that SAR-Harris is more effective than Harris feature extraction on SAR images feature extraction. SAR-Harris suppresses speckle noise by redefining the gradient calculation method, which is sensitive to Harris. Although SAR-Harris is accurate in the expression of image point features, in template matching, it is difficult to ensure sufficient matching information in the template area by relying only on the gradient of pixel. Figure 10 illustrates different feature representations of SAR intensity images with structure. In Figure 10a, the original SAR image is corrupted by multiplicative speckle noise with blurred feature outlines. The SAR-Harris plot demonstrated in Figure 10b retains the stabilized structural information. In Figure 10c, the location of the largest response value in the dense DHAE grid reflects the most representative region of the feature. In fact, we usually set the calculation step size to the size of the template window, as shown in Figure 10d. The DHAE grid with template window size as the resolution reduces the computational effort.

As shown in Figure 11, in (a) and (d), it is easy to find out based on the intensity information of the SAR image that in the weak-texture region, the DHAE grid determines a larger matching template that encompass the most representative boundary features in the area. In Figure 11b,e, there are almost no noteworthy feature points on the SAR image, and the DHAE grid selects the optimal local small template. In Figure 11c,f, it is obvious that the pond region is the most suitable matching template on the SAR image, and the DHAE grid adaptively adjusts the template size to fit the region and minimize computational consumption. The complete DHAE grid on the SAR image is shown in Figure 12, which can provide a robust reference framework for the extraction of interest points.

In the block Harris method, it is essential to configure various parameters, including block size in both row and column directions, the quantity of points to be extracted from each block, the threshold for identifying interest points, and other related parameters. Accurately predicting the expected number of interest points remains challenging within this approach. Conversely, with the DHAE method, the is focus primarily on extracting a predetermined number of interest points solely within specified rows and columns.

As illustrated in Figure 13, where the regular grid interval is 512 pixels, the block size of the block Harris method is 512 × 512 pixels, the number of points per block is 5, the Harris threshold is 0.5, and the dynamic block SAR-Harris is expected to have 16 × 24 points. According to the comparison results, it is easy to find that interest points extracted via DHAE have uniformity and fall on the textured area, while the regular grid only maintains uniformity, and block Harris only guarantees the strong texture where the interest points are located.

4.2. Discussion on DHAE Parameter Values

According to the results of the experiment in Section 3, DHAE has higher accuracy than other commonly used template matching methods, but the uniformity of distribution at TPs is inferior to that of the regular grid. The effect of parameter selection on the experimental results for other matching methods involved in the comparison is the size of the template window and the search window. For large-scale TPs matching and accuracy analysis of five data pairs, we used the template patch and search window with a size of 64 × 64. This is due to the fact that the sizes of the DHAE block and entropy are calculated to be 256 × 256 and 64 × 64 for the same image match, respectively. This leads to the adaptive template size distribution of the interest points extracted via DHAE on the interval [32, 128], while 64 is the peak of the template size distribution. The DHAE method brings an adaptive variable-size matching template, which makes the matching results more focused on areas with rich textures. In the selection of parameters for the accuracy evaluation, the number of coefficients of a polynomial is generally 3, 4, or 6. Here, we take 4 as the polynomial coefficient. This is because too many or too few parameters will cause the polynomial to be distorted.

4.3. Discussion on Limitations of DHAE

For SAR images with particularly uneven image feature distribution, the matching effect still has certain deficiencies. Although the advantages of the DHAE method are obvious, there are still some limitations in multi-view images. In the matching of multi-view and ascending and descending orbits images, the selection of TPs is not the most important factor affecting the results. Due to the significantly different structural characteristics, the matching quality improvement brought by DHAE is very limited. Different imaging angles cause different degrees of foreshortening and shadows, and the traditional template matching methods, like NCC, SSID, MI, etc., make it difficult to resolve such structural differences. It can be improved by robust structural extraction methods such as phase consistency.

Another limitation is that due to the lack of ground control points, it is difficult to verify the absolute accuracy of the proposed method in the experiments. However, the SAR TPs matching task will serve the block adjustment of large-scale SAR products; so in most cases, only the relative accuracy of the tie-points on the SAR images needs to be guaranteed. The absolute accuracy can be verified from the subsequent regional block adjustment task.

In addition, the calculation of the entropy of the regional image is also a very expensive step for computing resources. Computational efficiency can be improved by constructing fast region descriptors. Information entropy provides feature representation to a certain extent, but it also brings limitations. Since DHAE represents feature information in a certain spatial range and ignores the feature value of the point itself, it is only suitable for template matching. For feature matching, key-points require higher positioning accuracy, so the interest points determined via the DHAE method have no actual matching significance in this case. Furthermore, due to the limitations of template matching, DHAE-NCC also needs to perform rough registration or geocoding in advance to provide initial correspondence information.

5. Conclusions

In this paper, a robust method for large-scale SAR image TPs matching is proposed, which is divided into two parts: DHAE grid and adaptive template window. First, according to the SAR-Harris criterion, the local area entropy is calculated and the DHAE grid is generated according to the dynamic block selection strategy, and the interest points that are more representative of the regional template compared with the traditional regular grid are extracted to control the uniformity of distribution and reduce the mismatching rate. In the second stage, the optimal matching template size is adaptively calculated on the DHAE grid, which improves the matching accuracy of key areas and optimizes the calculation efficiency. Through the experimental results on a large range of SAR image datasets, it is shown that the accuracy and run time of the DHAE-NCC algorithm are better than those of the existing SAR image template matching methods, especially on the overlapping area images of flat areas and large areas.