A Robust Algorithm Based on Phase Congruency for Optical and SAR Image Registration in Suburban Areas

Abstract

Automatic registration of optical and synthetic aperture radar (SAR) images is a challenging task due to the influence of SAR speckle noise and nonlinear radiometric differences. This study proposes a robust algorithm based on phase congruency to register optical and SAR images (ROS-PC). It consists of a uniform Harris feature detection method based on multi-moment of the phase congruency map (UMPC-Harris) and a local feature descriptor based on the histogram of phase congruency orientation on multi-scale max amplitude index maps (HOSMI). The UMPC-Harris detects corners and edge points based on a voting strategy, the multi-moment of phase congruency maps, and an overlapping block strategy, which is used to detect stable and uniformly distributed keypoints. Subsequently, HOSMI is derived for a keypoint by utilizing the histogram of phase congruency orientation on multi-scale max amplitude index maps, which effectively increases the discriminability and robustness of the final descriptor. Finally, experimental results obtained using simulated images show that the UMPC-Harris detector has a superior repeatability rate. The image registration results obtained on test images show that the ROS-PC is robust against SAR speckle noise and nonlinear radiometric differences. The ROS-PC can tolerate some rotational and scale changes.

1. Introduction

The rapid development of sensor technology provided multiple remote sensing images for the observation of the Earth. Optical images ensure facilitated interpretation and are similar to human vision; however, they are affected easily by the weather. The synthetic aperture radar (SAR) is an active microwave imaging system that effectively compensates for the shortcomings of optical imaging systems and operates irrespective of the time of day and weather conditions. Optical and SAR images can be used together to form complementary information, which has important application value, such as image fusion [1,2], pattern recognition [3], and change detection [4,5]. Image registration is a preliminary work of these applications. It refers to aligning two or more images of the same scene acquired by different times, viewpoints, or sensors. Registration accuracy seriously affects these applications. Optical and SAR registration is still a challenging task owing to the speckle noise of SAR and the large radiation differences between optical and SAR images [6,7].

Generally, image registration methods can be roughly divided into two categories—namely, area-based methods and feature-based methods [8]. In area-based methods, which are also known as intensity-based methods, first, a template is defined, and subsequently, the geometric transformation model is estimated by optimizing a similarity measurement between the SAR and optical images, such as mutual information [9,10], normalized cross-correlation [11], and cross-cumulative residual entropy [12]. Area-based methods deliver high accuracy, as the entire intensity information is utilized. However, due to its high computational loads and sensitivity to the geometry and radiation differences, they are limited in their applications of optical and SAR image registration.

Feature-based methods usually first extract features such as points [13], edges [14,15], and contours [16] from input images. Then, a distinctive feature descriptor is designed. Finally, the transformation model is estimated by establishing the corresponding relationship between the features. Feature-based methods are recommended for optical and SAR image registration because they process images with their significant features rather than all intensity information, thereby achieving high precision and robustness to geometry and radiation differences. Feature-based methods are mainly composed of three steps: feature detection, feature description, and feature matching.

The most representative feature-based method is the scale-invariant feature transform (SIFT), owing to its efficient performance and invariance to scale and rotation [17]. Subsequently, a variety of improved methods have been reported. To improve matching efficiency, principal component analysis (PCA) is applied to reduce the dimension of the descriptor [18]. To reduce time, a speeded-up robust feature uses the determinant value of the Hessian matrix to detect feature points and an integral graph to accelerate the operation [19]. Affine SIFT simulates the parameters of affine transformation to achieve full affine invariance and considerably expands the scope of application of image registration [20]. A uniform robust SIFT is proposed to extract uniformly distributed and robust feature points [21]. Adaptive binning SIFT is proposed to increase the particularity and robustness of descriptors [22].

However, speckle noise in SAR images and the intensity difference between optical and SAR images make it difficult to obtain good results when these methods are applied directly to image registration. Numerous scholars have proposed improved methods for optical and SAR image registration. An improved SIFT is realized using optical and SAR satellite image registration by exploring their spatial relationship [23]. An automatic SAR and optical image registration method, from rough to accurate, is proposed with the use of SIFT features [24]. A novel gradient definition, yielding an orientation and a magnitude that is robust to speckle noise, is specifically dedicated to SAR images [25]. Further, to overcome the difference in image intensity between remote image pairs and increase the number of correct correspondences, a new gradient definition and an enhanced feature matching method by combining the position, scale, and orientation of each keypoint are proposed [26]. The gradients in the descriptor are computed by a multiscale Gabor odd filter (GOF)-based ratio operator, and the proposed GOF-based descriptor is formed for the SIFT features [27]. Xiang et al. proposed a robust SIFT-like algorithm (OS-SIFT) to register high-resolution optical and SAR images, in which the consistent gradient magnitudes in the SAR and optical images are computed using a multi-scale ratio of exponentially weighted averages (ROEWA) operator and a multi-scale Sobel operator, respectively [28].

Although numerous methods have achieved improvements in gradient redefinition and descriptor construction when encountering optical and SAR images with large nonlinear radiation differences, the matching performance of feature descriptors based on gradient information is not ideal, and there are still many mismatches. Recently, various registration methods based on phase congruency (PC) information have been widely used in multi-sensor images, because PC has been confirmed as an illumination and contrast invariant measure of the features [29,30,31].

An image descriptor, namely, the histogram of oriented phase (HOP) based on the PC concept and PCA is present, and it is more robust to image scale variations and contrast and illumination changes [32]. Ye et al. proposed a novel feature descriptor named the histogram of oriented phase congruency (HOPC) for multimodal image registration [33]. Further, they proposed a local phase-based invariant feature for remote sensing image matching, which consists of a feature detector called minimum moment of PC (MMPC)-Lap and a feature descriptor called the local HOPC (LHOPC) [34]. Similar to gradients, PC also reflects the significance of the features of local image regions. Chen et al. proposed an optical and SAR image registration method by combining a new Gaussian-Gamma-shaped bi-windows-based gradient operator and the histogram of oriented gradient pattern [35]. To address large geometric differences and speckle noise in SAR images, a novel optical-to-SAR image registration algorithm was proposed using a new structural descriptor [36]. A dense descriptor named the histograms of oriented magnitude and phase congruency was proposed to register multi-sensor images. It is based on the combination of the magnitude and PC information of local regions, and successfully captures the common features of images with nonlinear radiation changes [37]. A novel image registration method, which combines nonlinear diffusion and PC structural descriptors, has been proposed for the registration of SAR and optical images [38]. To overcome nonlinear radiation distortions, Li et al. [39] proposed a radiation invariant feature transform (RIFT) algorithm to register multi-sensor images, including optical and SAR images. The RIFT uses PC instead of image intensity for feature point detection and it proposes a maximum index map (MIM) for feature description. Further, the RIFT not only largely improves the stability of feature detection but also overcomes the limitation of gradient information for feature description.

Although a number of PC-based image registration methods have been proposed in the past few years, there are limitations that cannot be ignored when these methods are applied to optical and SAR image registration with large radiation differences. These limitations are listed below.

• Several methods detect keypoints directly from a PC map (PCM) or the moment of the PCM for feature matching. However, because of SAR speckle noise, some unreasonable points are detected in the SAR image; further, because of significant nonlinear intensity differences, the feature detection result of one image generally has no corresponding feature points in the other image. Several classical methods are tested using a pair of optical and SAR images, as shown in Figure 1. These limitations lead to the low repeatability of the feature point, which is not conducive to feature matching.
• The extracted points are not uniformly distributed. When calculating the PC of a whole image, the noise threshold $T$ is estimated using the Rayleigh distribution mode, which is a fixed value. This leads to a noise threshold larger than the actual noise in the dark region. The feature information is drowned by the noise. As shown in Figure 1, the features are always concentrated in the bright region, especially in the SAR image. The nonuniform distribution of feature points leads to limited registration accuracy on large images or high-contrast images.
• Because of the different imaging mechanisms for optical and SAR sensors, the acquired images have different expressions for the same objects, thereby resulting in large radiation differences between image pairs. Such nonlinear radiation differences reduce the correlation between corresponding points, which often leads to difficulties in feature description.

In this paper, we address the above limitations by developing a robust optical and SAR image registration method based on PC (ROS-PC). The proposed method mainly contains the following two works.

First, a uniform Harris feature detection method based on multi-moment of the PCM (UMPC-Harris) is proposed. In the UMPC-Harris, we take the corners and edge points as keypoints. The edge structure feature has a high similarity and better resistance to radiation difference between optical and SAR images [30,36,39], thus, extracting feature points on the edge can ensure enough number of features and robustness to radiation difference. Besides, corner features can increase the number of homologous points. Therefore, the multi-moment of the PCM is constructed by using maximum and minimum moment maps. Harris operator is used on the multi-moment to detect corners and edge points. Finally, the overlapping block and voting strategy are introduced to detect uniformly distributed and reliable keypoints.

Second, since PC is not suitable for constructing descriptors directly, the feature descriptor is derived for a keypoint by utilizing the histogram of phase congruency orientation on multi-scale max amplitude index maps (HOSMI). The proposed HOSMI descriptor is utilizing the MIM instead of the PCM because it is more robust to intensity radiation distortions than the PCM [39]. Furthermore, in remote sensing images, many salient features usually appear in different scales [38]. Therefore, we construct the phase congruency orientation maps and max amplitude index maps, respectively. In the local region of each keypoint, the histograms of phase congruency orientation on multi-scale max amplitude index maps are calculated. Finally, the descriptor is constructed by combining the feature vectors of all patched in order. Compared with state-of-the-art, the main contribution of this study can be summarized as follows:

• The UMPC-Harris feature detection method is proposed based on the multi-moment of the PCM, a voting strategy, and an overlapping block strategy. The detector can obtain enough reliable and uniformly distributed feature points.
• The HOSMI feature description method is proposed based on the histograms of phase congruency orientation on multi-scale max amplitude index maps. The descriptor is more robust against nonlinear radiation variation and speckle noise.

The rest of this paper is organized as follows: Section 2 starts with a review of PC theory, and followingly introduces the ROS-PC in detail, including the UMPC-Harris feature detector and HOSMI feature descriptor. In Section 3, through several experiments, the repeatability rate of keypoints by UMPC-Harris, the robustness of ROS-PC, and the sensitivity of ROS-PC to scale and rotation changes are evaluated and discussed. Finally, the conclusions are provided in Section 4.

2. Methodology

The PC has been confirmed to be robust to nonlinear radiometric differences, which can capture the common features between multi-sensor images [37,39,40]. The ROS-PC method is based on PC. This section first reviews the PC theory briefly and then presents the design processing of the UMPC-Harris detector and HOSMI descriptor.

2.1. Review of PC Theory

According to Kovesi’s approach, PC can be computed by convolving an image with a log-Gabor filter (LGF) to extract local phase information. The LGF is efficient for detecting features over multiple scales and orientations. In the frequency domain, LGF is defined as:

$$LGF\left(\omega \right)=\exp \left(\frac{-{\left(\log \left(\omega /{\omega }_0\right)\right)}^2}{2\left(\log \left(\kappa /{\omega }_0\right)\right)}\right),$$

(1)

where ${\omega }_0$ is the central frequency of the filter, $\kappa$ is the related-width parameter of the filter that varies with ${\omega }_0$, which ensures that $\kappa /{\omega }_0$ is a constant.

The filter is transformed from the frequency to the spatial domain using an inverse Fourier transform. In the spatial domain, the 2-D LGF is represented as:

$$LGF\left(x,y\right)=LG{F}_{s,o}^{even}\left(x,y\right)+i\times LG{F}_{s,o}^{odd}\left(x,y\right).$$

(2)

Considering the coordinates of an input image $I(x,y)$, the convolution responses $e_{s,o}(x,y)$ and $o_{s,o}(x,y)$ at scale s and orientation o are obtained, and then, the convolution results of even and odd symmetric wavelets form the response arrays as follows:

$$\left[e_{s,o}\left(x,y\right),o_{s,o}\left(x,y\right)\right]=\left[I\left(x,y\right)\ast LG{F}_{s,o}^{even},I\left(x,y\right)\ast LG{F}_{s,o}^{odd}\right],$$

(3)

where $LG{F}_{s,o}^{even}$ and $LG{F}_{s,o}^{odd}$ refer to the even-symmetric (cosine) and odd-symmetric (sine) wavelets of the LGF at scale s and orientation o, respectively. Further, $e_{s,o}\left(x,y\right)$ and $o_{s,o}\left(x,y\right)$ are the convolution responses of $LG{F}_{s,o}^{even}$ and $LG{F}_{s,o}^{odd}$ at scale $s$ and orientation $o$, respectively.

The corresponding amplitude $A_{s,o}\left(x,y\right)$ and phase ${\phi }_{s,o}\left(x,y\right)$ at scale s and orientation o are given by:

$$A_{s,o}\left(x,y\right)=\sqrt{e_{s,o}{\left(x,y\right)}^2+o_{s,o}{\left(x,y\right)}^2},$$

(4)

$${\phi }_{s,o}\left(x,y\right)=\arctan \left(o_{s,o}\left(x,y\right), e_{s,o}\left(x,y\right)\right).$$

(5)

Considering the negative effect of image noise, the improved PC (called PC₂) and the phase deviation function are, respectively, defined as [41]:

$$P{C}_2=\frac{{\displaystyle \sum _o{\displaystyle \sum _s{W}_o\left(x,y\right)⌊A_{s,o}\left(x,y\right)\Delta {\phi }_{s,o}\left(x,y\right)-T⌋}}}{{\displaystyle \sum _o{\displaystyle \sum _s{A}_{s,o}\left(x,y\right)+\epsilon }}},$$

(6)

$$\Delta {\phi }_{s,o}(x,y)=\cos ({\phi }_{s,o}(x,y)-{\overline{\phi }}_{s,o}(x,y))-\left|\sin ({\phi }_{s,o}(x,y)-{\overline{\phi }}_{s,o}(x,y))\right|,$$

(7)

where $W_o\left(x,y\right)$ is the weighting function, $T$ is the estimated noise threshold, $\epsilon$ is a small constant to prevent division by zero, and ${\overline{\phi }}_{s,o}(x,y)$ is the mean phase angle. The function $⌊\cdot ⌋$ denotes that the enclosed quantity is equal to itself when its value is positive, and zero otherwise. $P{C}_2$ denotes the PC magnitude map of the input image.

Further, to obtain the information of PC varying with orientation $o$ in the image, phase congruency is calculated independently in each orientation. Thus, serval PCMs according to the orientation angle are obtained [42].

$$P{C}_2={\displaystyle \sum _{o}PC({\theta }_o)},$$

(8)

where ${\theta }_o$ denotes the angle corresponding to orientation $o$, and $PC({\theta }_o)$ represents a PCM at orientation angle ${\theta }_o$. The moment of PC is calculated using these intermediate quantities as:

$$a={{\displaystyle \sum _o(PC({\theta }_o)\cos ({\theta }_o))}}^2,$$

(9)

$$b=2{\displaystyle \sum _o(PC({\theta }_o)\cos ({\theta }_o))\cdot (PC({\theta }_o)\sin ({\theta }_o))},$$

(10)

$$c={{\displaystyle \sum _o(PC({\theta }_o)\sin ({\theta }_o))}}^2.$$

(11)

The maximum moment $ma{x}_{\psi }$ and the minimum moment $mi{n}_{\psi }$ of PC are defined as:

$$ma{x}_{\psi }=\frac{1}{2}(a+c+\sqrt{b^2+{(a-c)}^2}),$$

(12)

$$mi{n}_{\psi }=\frac{1}{2}(a+c-\sqrt{b^2+{(a-c)}^2}).$$

(13)

The maximum and minimum moments of the PCM represent the edge and corner strength map, respectively.

2.2. The Proposed UMPC-Harris Feature Detector

Keypoints with high repeatability and uniform distribution can obtain sufficient matches, thus improving the image registration accuracy [38]. The subsection presents a novel feature detector UMPC-Harris, which is based on voting strategy, Harris on the multi-moment of PCMs, and overlapping block strategy for the detection of corners and edge points. The purpose of this UMPC-Harris detector is to detect sufficient, reliable, and well-distributed keypoints in optical and SAR images. Figure 2 presents the main process of the UMPC-Harris detector, which contains three steps.

First, the input image is divided into $S_n\times S_m$ blocks. Further, to avoid missing feature information on the block boundary, an overlap region with $n_{op}$ pixels is added between adjacent blocks. The choice of parameters $S_n$ and $S_m$ is a tradeoff between the amount of computation and the uniform distribution of the keypoints. When more blocks are divided, the keypoints will become more uniform, while increasing the number of calculations. The size and local complexity of the image should be considered for the selection of $S_n$ and $S_m$.

Second, according to the description in Figure 2, we take block (1,2) as an example to illustrate the construction of multi-moment of the PCM. According to the definition of the maximum and minimum moments, the moment of the PCM $M_k$ is defined as:

$$M_k=\frac{1}{2}\left(a+c\right)+\frac{k_t}{2}\sqrt{b^2+{\left(a-c\right)}^2},$$

(14)

where $k_t$ is a variable between −1 and 1. The moment map contains the maximum and minimum moment map, and we can use $ma{x}_{\psi }$ and $mi{n}_{\psi }$ to describe the above equation as:

$$M_k=\frac{1}{2}\left(ma{x}_{\psi }+mi{n}_{\psi }\right)+\frac{k_t}{2}\left(ma{x}_{\psi }-mi{n}_{\psi }\right),$$

(15)

where $M_k$ represents the moment of the PCM with parameter $k_t$, and it is obvious that if $k_t$ is set to $-1$, $M_k$ is the minimum moment map $M_k=mi{n}_{\psi }$, and if $k_t$ is set to $1$, $M_k$ is the maximum moment map $M_k=ma{x}_{\psi }$. The number of moments is $n$, and the step $h$ is $\frac{2}{n-1}$.

Third, the points detected by Harris on the maximum and minimum moments of the PCM represent edge points and corners, respectively. Because the edge feature has a high similarity and better resistance to radiation difference between optical and SAR images, thus, extracting feature points on the edge can ensure enough number of features and robustness to radiation difference. Besides, corner features can increase the number of homologous points. Thus, we combine the corners and edge points as keypoints. However, corner features are sensitive to SAR speckle noise and the repeatability rate of the edge points is poor, and therefore, if all of them are considered as keypoints, there could be some unreasonable keypoints. Therefore, we extract Harris corners on the multi-moment of the PCMs, respectively, and we consider the points appearing many times as the final keypoints. Stable and reliable keypoints are found based on the voting strategy.

2.3. Feature Description

After a set of keypoints are detected by the UMPC-Harris method, feature descriptors need to be designed for each keypoint to achieve image registration. The orientation and maximum amplitude index of PC have been proved suitable for describing the similar local features of multi-sensor images [34,37]. In this subsection, we first introduce the construction process of the multi-scale max index maps and the orientation map of phase congruency. Finally, the HOSMI descriptor is established to increase the distinction of the features.

2.3.1. Multi-Scale Max Index Maps

Max index map is more suitable for multimodal image registration and is more robust to intensity radiation distortions compared to the gradient amplitude map. Furthermore, in remote sensing images, many salient features appear in different scales [38]. Inspired by this, multi-scale MIMs are formed by calculating the index of the maximum amplitude at each scale to improve the significance of descriptors. Figure 3 shows the construction process of the four-scale MIMs.

First, the input image $I(x,y)$ is convoluted with LGF to obtain $s\times o$ PC amplitude maps in four scales and six orientations. Second, in the six amplitude maps of the same scale, we can find the maximum amplitude $max{\{A_{s,o}(i,j)\}}_1^o$ and the corresponding orientation $o$, where the superscripts and subscripts indicate orientations ranging from 1 to $o$ for a pixel $p$ with coordinates $A_{s,o}(i,j)$. Third, the coordinate of the pixel $p$ in MIM is represented by the corresponding orientation $o$. Thus, the MIM is an image with all elements from 1 to $o$. Finally, multi-scale MIMs are constructed by calculating the index of the maximum amplitude on four scales. Therefore, the information in the fine scale and coarse scale of the input image can be obtained, which can effectively enhance the saliency of features in the image.

2.3.2. Orientation of Phase Congruency

The orientation of PC represents the important directions of feature variation, and it has been proved robust to nonlinear radiation distortions [32]. Therefore, similar to the gradient and gradient orientations in the SIFT algorithm, we need to find orientation information in addition to multiscale MIMs.

The PC is calculated by the odd and even symmetric wavelets of LGF, wherein the odd-symmetric wavelet is a smooth derivative filter, which can compute the image derivative in a single direction [34]. For the calculation of PC, the convolution results of the odd-symmetric are obtained according to six orientations. The six convolution results are projected into $x$ and $y$ axes, respectively, and the projections of the x and y axes are obtained. The orientation of the PC can be calculated by the arctangent functions defined as:

$$O_x={\displaystyle \sum _o(o_{s,o}({\theta }_o)\cos ({\theta }_o))},$$

(16)

$$O_y={\displaystyle \sum _o(o_{s,o}({\theta }_o)\sin ({\theta }_o))},$$

(17)

$$O_{pc}=\arctan (O_y,O_x),$$

(18)

where ${\theta }_o$ represents the angle corresponding to the orientation $o$, and $e_{s,o}({\theta }_o)$ is the convolution result of the odd-symmetric in angle ${\theta }_o$. Further, $O_x$ and $O_y$ are the sum of the projection of the convolution result in the $x$ and $y$ directions, respectively. The PC orientation ${\mathrm{O}}_{pc}$ can be obtained by the arctangent function. Notably, the PC orientation is limited to $\left[0^{°},{180}^{°}\right)$, which can handle gradient inversion in optical and SAR images. Figure 4 shows the calculation process of the PC orientation.

2.3.3. The Proposed HOSMI Feature Descriptor

The proposed HOSMI descriptor is constructed using the histograms of the PC orientation on the multi-scale MIM. Figure 5 presents the main processing chain of the proposed HOSMI descriptor.

As shown in Figure 5, HOSMI is calculated based on a grid of patches, where local histograms of PC orientation are formed on each scale MIM. The main steps of the feature descriptor are listed below:

• Apply the LGF to the local region $Lx$ of each keypoint, and then, calculate the odd and even convolution results of four scales and six orientations.
• Calculate the amplitude map over four scales and six orientations. In each scale, the corresponding orientation to the maximum amplitude forms the multi-scale MIMs; the detailed calculation process is shown in Figure 3.
• Obtain the PC orientation using the odd convolution results; the detailed calculation process is shown in Figure 4. The PC orientation is restricted to an interval $\left[0^{°},{180}^{°}\right)$, which can handle gradient inversion in optical and SAR images, and large intensity differences between the optical and SAR images can be reduced.
• Divide the PC orientation map and the multi-scale MIMs of each keypoint into $n_p\times n_p$ patches. If the local region $Lx$ is selected with a size of $m\times m$ pixels, the size of the patch is $(m/n_p)\times (m/n_p)$ pixels. The feature vector of each patch is calculated in order, and then a descriptor is constructed by combining the feature vectors of all patches.
  • To calculate the feature vector of a patch, PC orientation is formed using $n_o$ bins covering the 180 degrees range of orientations. The sample added to the histogram is the element of the corresponding location on the MIM. To interpolate the peak position for better accuracy, a parabola is fitted to the three histogram values closest to each peak. The feature vector of patch $P$ is calculated on four scales; therefore, the dimension of the feature vector of a patch is $s\times n_o$. In Figure 5, we take the first patch $P_1$ as an example. The scale used in the PC method is set to 4, and the feature vector of the patch is constructed as $P_1=\left[H_1,H_2,H_3,H_4\right]$, where $H_1$~$H_4$ are the histograms of the four scales.
  • To obtain the feature descriptor of a keypoint, the feature vectors of all patches are combined into one feature vector. The feature descriptor is normalized by the $L2$ norm to achieve better invariance to illumination and shadowing. The dimension of the feature descriptor of a keypoint is $s\times n_o\times n_p\times n_p$. As shown in Figure 5, if the local region of a keypoint is divided into 4 $\times$ 4 patches, the feature descriptor is constructed by the 16 patches, as in $HOSMI=\left[P_1,P_2,\cdot \cdot \cdot ,P_{16}\right]$.
• Construct a local feature descriptor HOSMI for optical and SAR image registration.

A pair of corresponding points in the optical and SAR images are selected to construct descriptors. To verify the similarity of descriptors, we draw descriptors into stem images in Figure 6.

Figure 6 shows the HOSMI descriptors of a pair of keypoints between the optical and SAR images. This pair of optical and SAR images has a strong difference in intensity and in gradient inversion, and there is obvious scattering in the SAR image. These differences introduce great challenges to the robustness of the descriptors. The square region represents the local region (96 × 96 pixels) around the keypoint, which is used for computing the feature descriptor. As shown in Figure 6, the similarity of the feature vector is high and radiation changes have a low effect on the proposed descriptor.

3. Experimental Results and Discussion

In this section, we evaluate the performance of the feature detector on simulated images with different SAR noise levels and radiometric (non-uniform intensity) changes. Then, eight pairs of optical and SAR images are used to test the ROS-PC and analyze the experimental results. The registration performances are evaluated via objective and subjective approaches. One approach is to use the evaluation criteria, and the other is to use a chessboard mosaic image and enlarged submaps. Finally, experiments are conducted to evaluate the tolerance of rotation and scale changes from the ROS-PC method. All experiments were conducted with the MATLAB R2017b software on a computer with an Intel Core i5-7200U CPU and 16.0 GB memory.

3.1. Performance Experiments of Proposed UMPC-Harris Detector

We test the performance of the proposed UMPC-Harris detector on simulated images with different noise levels and radiometric (non-uniform intensity) changes. The UMPC-Harris is compared to three other state-of-the-art detectors, Harris, SAR-Harris, and m+M-Harris.

3.1.1. Evaluation Criteria of Feature Detector

Repeatability rate: Given a pair of simulated optical and SAR images to be registered, the keypoints are detected on the two images. Further, two points are regarded as a pair of corresponding keypoints, only if their coordinates are satisfied:

$${‖p_{so}(x,y)-p_{ss}(x,y)‖}_2\le T,$$

(19)

where $p_{so}(x,y)$ and $p_{ss}(x,y)$ denote the coordinates of the corresponding keypoints in the simulated optical and SAR images, respectively. The function ${‖\cdot ‖}_2$ denotes the Euclidean distance between points $p_{so}(x,y)$ to $p_{ss}(x,y)$. $T$ is the threshold of the Euclidean distance, which is set to 2 pixels in this experiment. The repeatability rate is defined as:

$$R_{rep}=\frac{2N_{cor}}{n_{so}+n_{ss}},$$

(20)

where $N_{cor}$ is the number of pairs of the corresponding keypoints, and $n_{so}$ and $n_{ss}$ are the number of keypoints in the simulated optical and SAR images, respectively. The repeatability rate is a number between 0 and 1. The larger the repeatability rate, the better is the robustness of the detector.

3.1.2. Experimental Data and Parameter Settings of Feature Detector

• Experimental Data

We used the high-resolution (HR) optical images from the official website of Changguang Satellite Technology Company as the experimental data. The resolution of these images is better than 1 m/pixel. We selected three images with 1000 × 1000 pixels, captured at the Kabul International Airport, Afghanistan, in June 2018; these images are named as Group 1 to 3, as shown in Figure 7.

b.

Parameter Settings

For the Harris detector, the threshold of the Harris operator is normalized between 0 and 1, and it is set to 0.1 in the following experiments. For the SAR-Harris detector, the first scale is set as $\sigma =2$, the constant between two adjacent scales is set as $k=2^{1/3}$, the number of the scale layer is set to 8, and the arbitrary parameter is set as $d=0.04$. Based on our previous experience, the threshold in the keypoint detection of the simulated optical image is set between 1 and 5, and that of the SAR image is set between 5 and 10. Other parameters used in the experiments follow the parameter settings suggested in Reference [25]. The m+M-Harris and UMPC-Harris are both based on PC. For fairness, the parameters of the PC method are tuned to the same value at each noise level. The PC is calculated in four scales and six orientations. The wavelength of the smallest filters is set from 3 to 5 pixels, according to different images. The scaling factor between successive filters is set to 1.6. In the experiment, the parameters are selected as $S_n=4$, $S_m=5$, $n_{op}=20$, and the number of moments of the PCM is selected as $n=5$ and $h=0.5$. The array of parameters $k_t$ is $\left[-1,-0.5,0,0.5,1\right]$.

3.1.3. Influence of Noise Level on Proposed UMPC-Harris Detector

To assess the robustness of the feature detector to noise, the HR optical images are utilized to generate simulated optical and SAR images by adding Gaussian noise and speckle noise, respectively. The simulated optical image is obtained by adding Gaussian white noise with 0 mean and 0.01 variance. In the simulated SAR images, the noise level is defined to describe the degree of multiplicative noise. The multiplicative noise with a different number of looks is simulated from one-look to nine-look in the simulated SAR image. It decreases with an increase in the SAR number of looks. A high number of looks refers to a small noise level in SAR images. The simulated images are shown in Figure 8.

With the SAR noise level ranging from 1 to 9, the repeatability rate of the UMPC_Harris detector is compared to three other detectors, Harris, Sar-Harris, and m+M-Harris. For fairness, each detector extracts approximately 600 pairs of points by adjusting the threshold, and the average value of ten calculations is taken as the experimental result. The curves of the repeatability rate with the SAR noise level in the three groups are shown in Figure 9.

The repeatability rate of the UMPC-Harris is the highest among the four detectors in the three groups, and it is more robust to noise than the other detectors. SAR-Harris and m+M-Harris have similar repeatability rates as that of the SAR noise level. Their repeatability rates are between 0.5 and 0.3. When the SAR noise level is high, the repeatability rate of the UMPC-Harris is still higher than 0.4. When the SAR noise level is small, the differences between the two simulated images caused by noise are small, and hence, the three methods show good performance, except for the Harris detector. The repeatability rate of the Harris detector is lower than 0.4, and it decreases rapidly with an increase in the SAR noise level. It is difficult for the Harris detector to deal with multiplicative noise in SAR images directly.

3.1.4. Influence of Radiometric Changes on Proposed UMPC-Harris Detector

To assess the robustness of the feature detector to nonlinear radiometric differences, HR optical images are utilized to generate an image with non-uniform radiometric differences. This is achieved by multiplying the HR optical image by a variable coefficient according to the change of the image column. The results are shown in Figure 10.

The four detectors are tested on these images. Each detector extracts approximately 600 pairs of keypoints by adjusting the threshold properly, and the experiment results are shown in Figure 11. Furthermore, the repeatability rate of the four detectors are presented in

Table 1. Repeatability rate of the detectors on the images with non-uniform radiometric differences.

Method	Harris	SAR-Harris	m+M-Harris	UMPC-Harris
Repeatability (%)	59.98	71.83	68.80	90.96

.

The repeatability rate of the UMPC-Harris is the highest among the four detectors. One can see that the keypoints detected by UMPC-Harris are distributed more uniformly over the image than other detectors, which illustrates that the UMPC-Harris is more robust to radiometric variation and it further indicates that the UMPC-Harris can be applied to keypoint detection for images with radiometric differences.

3.1.5. Results and Discussion of the Proposed UMPC-Harris Detector

The results of corner detection by UMPC-Harris on a pair of optical and SAR images are shown in Figure 12. They include images of a suburban in Weinan, Shaanxi, China. The size of the optical image and SAR image are 863 × 761 and 858 × 761, respectively. The optical image is obtained by Google Earth, and the SAR image is obtained by Airborne SAR. The resolutions are both 3.2 m/pixel. The comparison results indicate that the UMPC-Harris detector is able to extract uniformly distributed in the entire image and avoid missing keypoints at the border of the block.

The comparison of the repeatability rates of the four detectors indicates that the proposed UMPC-Harris detector has the highest repeatability rate. It is more robust to noise and it has better resistance to radiation differences. The reasons are listed below.

• The UMPC-Harris aims to extract feature points on the multi-moment of the PCM. Stable and valuable keypoints are selected by voting on the Harris corner, which appears repeatedly on the PCM. The combination of the effective corners and edge points not only ensures the high repeatability of the features, but also a large number of features, which lays a foundation for subsequent feature matching.
• Keypoints are well-distributed in the entire image, further points with obvious local features in the dark regions can be detected. This ensures that the keypoints are not limited to the bright region, thereby improving the accuracy of image registration.

3.2. Performance Experiments of Proposed ROS-PC Registration Algorithm

To evaluate the performance of the ROS-PC, it is compared with two other state-of-the-art algorithms, namely OS-SIFT [28] and RIFT [39]. The OS-SIFT is a feature-based method, and ROEWA and Sobel operators are used to calculate consistent gradients for optical and SAR images. The RIFT is a PC-based method that detects corners and edge points on the PC map, and it proposes a MIM for feature description. Both the OS-SIFT and RIFT exhibit good performances on multi-sensor image registration. The comparative programs are obtained through their respective academic home pages. Subjective and objective criteria are used to evaluate the performance of the registration algorithm.

3.2.1. Evaluation Criteria of the Registration Algorithm

The checkboard mosaic image and enlarged sub-images are displayed to observe the effect and details of image registration. For each test image pair, each algorithm is executed ten times, and the average of the ten results is computed as the final result. The following evaluation criteria are used to analyze the performance of the algorithm objectively and quantitatively.

Root mean square error (RMSE): This criterion is used to measure the accuracy of the image registration algorithm and is computed by the following method.

First, approximately 20 pairs of corresponding points are manually selected from the optical and SAR images to estimate the affine transformation matrix H. The coordinates of $ith$ correctly matched keypoints are $\left\{(x_i^o,y_i^o),(x_i^s,y_i^s)\right\}$. The RMSE is computed as [26]:

$$RMSE=\sqrt{\frac{1}{N_{cor}}{\displaystyle \sum _{i=1}^{N_{cor}}{(x_i^o-{(x_i^s)}^{\prime })}^2+{(y_i^o-{(y_i^s)}^{\prime })}^2}}$$

(21)

where $N_{cor}$ is the number of correctly matched keypoints after the fast sample consensus (FSC) [43], $({(x_i^s)}^{\prime },{(y_i^s)}^{\prime })$ and it denotes the transformed coordinates of $(x_i^s,y_i^s)$ by the estimated transformation matrix H.

Number of correct matches (NCM): The NCM is the number of correctly matched keypoints after the FSC. If the NCM of an image pair is less than four, the matching is considered to have failed.

A small RMSE indicates that the accuracy of optical and SAR image registration is high. A large NCM indicates that there exist more correctly matched keypoints, thereby resulting in a more accurate transformation matrix H.

3.2.2. Datasets and Parameter Settings of the Registration Algorithm

• Datasets

In our experiment, eight pairs of optical and SAR images are used to test the ROS-PC; these pairs are referred to as Pairs A-H.

Table 2. Information for the test images.

Pair	Sensor	Resolution	Date	Size (Pixel)
A	Google Earth	1 m	9 October 2012	923 × 704
	TerraSAR-X	1 m	23 December 2010	900 × 795
B	Google Earth	1 m	27 March 2020	932 × 684
	Airborne SAR	1 m	June 2020	867 × 740
C	Google Earth	3 m	24 June 2020	1019 × 699
	Airborne SAR	3 m	April 2018	1016 × 697
D	Google Earth	3 m	1 July 2017	1741 × 1075
	Airborne SAR	3 m	April 2018	1744 × 1078
E	Google Earth	3.2 m	25 April 2020	874 × 768
	Airborne SAR	4 m	April 2018	692 × 612
F	Google Earth	2.5 m	19 February 2020	1019 × 701
	Airborne SAR	2.5 m	April 2018	1020 × 711
G	Google Earth	2.2 m	19 February 2020	968 × 662
	Airborne SAR	2.2 m	April 2018	1010 × 676
H	Google Earth	2.5 m	19 February 2020	858 × 758
	Airborne SAR	2.5 m	April 2018	863 × 761

lists the information for the test images.

Numerous factors are considered in the selection of test images, including different SAR sensors, date, resolution, and size. The optical images of the eight pairs are obtained from Google Earth, and the SAR image contains a satellite SAR image and seven airborne SAR images. To verify the robustness of the ROS-PC, the image contains different features, as shown in Figure 13.

Pair A includes images of an airport in Tucson, AZ, USA; in this pair, there exists a slight rotation and translation difference. Pair B is obtained in Zhengzhou, Henan, China, and it includes images of a small village. There is a slight rotation and translation in this pair, and because of the arc-shaped roof, there is a large radiation difference over the houses. Some houses even are difficult to recognize on SAR images. Pair C is also obtained in Zhengzhou, Henan, China, and it includes images of a field, and the features of the field vary significantly with the date. Some obvious features exist in the SAR image but not in the optical image. The remaining five pairs of images are obtained in Weinan, Shaanxi, China. Pair D includes images of a large scene, that includes a river, small buildings, multiple fields, and several roads. There are no scale and rotation changes in this pair, and the features exhibit a little temporal difference. This pair of images is used for the rotation and scale variation experiments. Pair E mainly includes images of a lake. There are certain scale variations and time differences. Pair F includes images of a terrace. The fields and terraces in the image are divided into two parts by a road. The feature intensity of the terraces is stronger than that of the fields. Pair G includes images of a scene including a river and some fields, there exists a large time span between the two images. Pair H includes images of a complex scene with some different structure buildings.

b.

Parameter Settings

The parameter settings of the UMPC-Harris detector are described in Section 3.1.2. The proposed ROS-PC method contains three parameters $n_o$, $n_p$ and $m$, respectively. Parameter $n_o$ and $n_p$ are related to the dimensions of the descriptor, so they should not be too large. Parameter $m$ is the size of the local region used for feature description. If the local region is too small, it contains less information, which does not reflect the difference of features. On the contrary, if the local region is too large, not only the amount of calculation will increase but also the effect of geometric distortion will be received. In the feature descriptor, the parameters are set as $n_o=6$, $n_p=4$, and $m=96$. Therefore, the dimension of the feature descriptor of a keypoint is 384. The parameter settings of comparative algorithms OS-SIFT and RIFT follow the References [28,39]. For a fair, the thresholds of keypoints detection are properly adjusted to obtain similar numbers of keypoints (approximately 1000~1200).

For the feature matching, the sum of squared differences (SSD) is selected for the feature matching metric. If the distance between the two feature vectors is less than the threshold, a pair of keypoints is considered as a potential match, and the threshold is set to 3 pixels. Generally, the matching pairs contain many false matches. The FSC algorithm is used to remove false matches.

3.2.3. Comparison of Experimental Results and Discussion

To evaluate the optical and SAR image registration performance of the ROS-PC, the algorithm is compared with OS-SIFT and RIFT. The OS-SIFT utilizes two different operators to calculate the gradients for SAR and optical images. Multiple image patches are aggregated to construct a gradient location orientation histogram-like descriptor. It is an advanced gradient-based method. The RIFT is a radiation-insensitive feature matching method based on PC and MIM, which is considerably more robust to nonlinear radiation distortions than traditional gradient maps. The registration results of eight pairs of optical and SAR images are shown in Figure 14, Figure 15, Figure 16, Figure 17, Figure 18, Figure 19, Figure 20 and Figure 21.

Eight groups of images with different features are selected to verify the robustness of ROS-PC algorithm. It can be found that the ROS-PC has the best performance among the three methods, owing to the advantages of the proposed UMPC-Harris detector and HOSMI descriptor. For images with date and season differences, shown in Figure 14 and Figure 16, the ROS-PC shows better robustness and obtains some matched keypoints with the time difference. For images with large radiation differences, shown in Figure 15 and Figure 21, ROS-PC can still obtain some correctly matched keypoints, which are well-distributed in the image. For images with multiple objects, shown in Figure 17, Figure 18, Figure 19 and Figure 20, ROS-PC can extract matching keypoints from each object, and they are uniformly distributed, which ensures the accuracy of registration. To sum up, the ROS-PC is a robust algorithm, which is suitable for optical and SAR image registration.

To further observe the registration accuracy of the ROS-PC, the checkboard mosaic images and enlarged sub-images of each pair are displayed in Figure 22.

The sub-image is an enlarged view of the intersection of the checkboard mosaic images, where the common features in the optical and SAR images are displayed clearly. In each pair, three sub-images with different features are selected.

Comparisons of RMSE, NCM, and the running time of the eight pairs are presented in

Table 3. Comparison of root mean square error (RMSE), number of correct matches (NCM), and time for different methods on eight pairs of test images.

	Method	OS-SIFT			RIFT			ROS-PC
Pair		RMSE	NCM	Time(s)	RMSE	NCM	Time(s)	RMSE	NCM	Time(s)
A		1.8507	30	52.84	2.2953	39	10.4	1.9326	41	73.46
B		16.1443	6	65.74	---	---	---	2.8296	17	77.36
C		---	---	---	5.6334	17	11.94	2.6961	31	74.45
D		4.8541	9	81.12	2.6507	33	24.33	1.6741	58	97.49
E		20.7450	4	52.90	2.7754	38	9.93	1.9116	38	75.56
F		22.2591	5	67.62	4.6117	19	13.58	1.7773	30	74.12
G		5.6215	5	57.6	2.9563	23	12.25	1.5264	35	71.26
H		---	---	---	---	---	---	1.7612	33	70.59

.

In the eight pairs of test images, most features of suburban areas, such as airports, houses, fields, terraces, roads, rivers, and lakes are included. The optical and SAR images exhibit nonlinear radiation distortion, which leads to intensity differences or gradient inversion. Next, we analyze and discuss the registration results of each pair of images.

For Pair A, all three methods can successfully achieve optical and SAR image registration because of the HR and less noise. The ROS-PC performs best on the RMSE and NCM, benefiting from the high repeatability rate of keypoints and the robustness of the feature description method. For Pair B, the RIFT fails to register the two images correctly because of the serious nonlinear radiometric difference. This is because the houses in the image have arc-shaped roofs, which induce the optical and SAR images with intensity differences. Although several correctly matched keypoints are detected by the OS-SIFT, the number and accuracy are significantly lower than the ROS-PC. For Pair C, the OS-SIFT fails to register the two images because of the gradient difference and obvious scattering in the SAR image. However, the ROS-PC is suitable for describing similar local information based on the PC orientation and the multi-scale MIMs of the keypoints. For Pair D, the image contains many features, and there are little scale, rotation, and date difference. The ROS-PC remains superior to the other two methods, as it is robust to nonlinear radiation differences and noise. For pair E and F, the image contains two types of features. ROS-PC can obtain correctly matching keypoints from each object, and they are well-distributed. For pair G, owing to the time difference, there are many unmatched keypoints of the river in the image, which causes extra difficulties in feature matching. However, the ROS-PC still successfully completes more NCM and achieves higher accuracy. For pair H, the radiation difference between optical and SAR images is large, because there are many buildings in the image, which leads to the failure of the other two algorithms. To sum up, the ROS-PC has the most uniform distribution, the largest NCM, and the best RMSE among the three algorithms. Therefore, the ROS-PC is more robust to noise and scattering in SAR images and the radiation difference between optical and SAR images.

Gradient-based descriptors such as OS-SIFT are more sensitive to nonlinear radiation differences because the gradient-based descriptors rely on a linear relationship between images, and therefore, they are not appropriate for significant nonlinear intensity differences caused by radiation distortion. The speckle noise and scattering in SAR images pose significant challenges in image registration. The RIFT performs better than the gradient-based descriptors because it uses PC to capture MIM. The RIFT uses the MIM to express the shape and structure information of objects and, therefore, it is robust to nonlinear radiation distortion. However, the repeatability rate of the corner detector in RIFT is not as good as that of UMPC-Harris, and the descriptor in RIFT has limited significance and robustness to noise and scattering in SAR images. Therefore, our ROS-PC yields the smallest RMSE and the largest NCM among all eight pairs because of two reasons, which are listed below.

• The UMPC-Harris can obtain a higher repeatability rate of keypoints than SAR-Harris and m + M-Harris between SAR and optical images.
• The HOSMI descriptor uses four-scale and six-orientation LGFs to capture the multi-scale max index and orientation feature information of PC, which is robust to nonlinear radiation variations of optical and SAR images. Further, it can effectively overcome the noise and scattering of SAR images.

After comparisons of the running time in Table 3, it can be found that the ROS-PC is the most time-consuming. The reason is that the algorithm is based on the principle of PC, which is slow by nature. Second, in the process of feature detection, the overlapping block and voting strategies need to additional calculation than the other methods. Third, the descriptor is constructed over the four scales and the dimension is larger. This paper only focuses on a robust registration method for optical and SAR images, and the running time is not the focus. Therefore, reducing the computation time and improving the efficiency of the algorithm is a problem we need to study in the future. Moreover, computational efficiency can be further improved by optimizing the algorithm and implementing the ROS-PC in C/C++.

3.3. Influence of Rotation and Scale Variations on the Proposed ROS-PC

The previous experimental results show that the algorithm is robust to the radiation distortion between optical and SAR images; however, the ROS-PC is not designed for scale and rotation deformations. The large-angle rotation between remote sensing images can be corrected using sensor geographic information. Further, by employing remote sensing image ground resolution information, remote sensing images can be assigned to the same scale by resampling. Then, the ROS-PC could be used for fine matching, which can handle slight rotation and scale differences between optical and SAR images. In this subsection, the influence of rotation and scale variation on our algorithm is evaluated based on the NCM for Pair D.

3.3.1. Rotation Experiments of the Proposed ROS-PC

We tested the effect of rotation changes on the ROS-PC. Assuming the optical image remains unchanged, the SAR image is rotated from −12° to 16°. The optical and SAR image registration results of the rotation variation are shown in Figure 23. The relationship between the NCM and the rotation angle is listed in

Table 4. NCM with different rotation angles.

Rotation Angle	−12°	−9°	−6°	−3°	0°	4°	8°	12°	16°
NCM	8	19	28	35	58	32	31	20	15

.

Figure 23 and Table 4 indicate that the ROS-PC can tolerate rotations between optical and SAR images below 9°, which is sufficient for images that have been corrected by sensor geographic information.

3.3.2. Scale Experiments of the Proposed ROS-PC

We test the robustness of the ROS-PC to scale changes. The optical image in Pair-D remains unchanged, and the SAR image is resized from 0.6 to 1.4 with an interval of 0.1. The optical and SAR image registration results of the scale variation are shown in Figure 24. The relationship between the NCM and the scale factor is listed in

Table 5. NCM with different scale factors.

Scale	0.6	0.7	0.8	0.9	1	1.1	1.2	1.3	1.4
NCM	16	27	39	55	58	39	23	13	13

.

Figure 24 and Table 5 indicates that the ROS-PC can tolerate the scale difference between optical and SAR images in the range of 0.7–1.2, which is sufficient for images that have been assigned a similar scale by resampling.

4. Conclusions

In this study, a PC-based optical and SAR image registration algorithm ROS-PC is proposed to address the matching difficulties caused by complex nonlinear radiation differences and speckle noise.

We designed a novel feature detector named UMPC-Harris, comprising an overlapping block strategy, Harris on multi-moment of the PCM, and the vote strategy to obtain uniformly distributed keypoints and increase the repeatability rate of the keypoints. The experimental results on simulated images demonstrated that the proposed UMPC-Harris method achieved a good performance in keypoints detection. The proposed HOSMI descriptor is constructed using the histograms of the PC orientation on the multi-scale MIM. The image registration experiments prove that the ROS-PC is robust to nonlinear radiation variations of optical and SAR images and it can tolerate a small amount of rotation and scale changes.