Optical and SAR Image Registration Based on the Phase Congruency Framework

Abstract

The improved phase congruency (PC) algorithms have been successfully applied to optical and synthetic aperture radar (SAR) image registration since they are insensitive to nonlinear radiometric and geometric differences. However, most of the algorithms are sensitive to large-scale differences and rotation differences between optical and SAR images. To tackle this, we propose a PC framework to register optical and SAR images. It is compatible with large-scale and rotation invariance. Firstly, a multi-scale Harris keypoint extraction method based on the maximum moment of PC (named PC-Harris) is proposed. The scale space is constructed by combining PC with the log-Gabor filter. Secondly, we propose a PC model to construct the feature descriptors. The orientation and amplitude responses are obtained based on the PC model. Meanwhile, the novel descriptor is constructed based on the polar coordinate system and thus can handle the scale and rotation differences between optical and SAR images. Finally, outliers are removed by the fast sample consensus (FSC). The experiments conducted on several optical and SAR images verify the effectiveness of the proposed framework.

1. Introduction

With the development of remote sensing techniques and the increasing demand for aerial remote sensing image processing, multi-sensor information processing technology, including image registration and fusion, has greatly increased [1,2]. Optical images have such advantages as easy understanding, rich content, obvious structural features, high resolution, and a large field of view angle, but they are greatly influenced by illumination, cloud, season, and shadow. SAR images have the advantages of working all day and in all weather conditions since they are not easily affected by illumination and weather. However, SAR images also have disadvantages, such as the ambiguity of target details and the insufficient detection range. The registration of airborne SAR and optical images is very important, and it is the foundation of image fusion and image mosaic. Therefore, multi-sensor image registration as a key technology of multi-sensor image fusion is crucially important [3,4]. In general, optical-to-SAR image registration has two main categories: area-based and feature-based methods. Compared with area-based methods, feature-based methods can successfully solve the scale and rotation differences between multi-sensor images [5]. Feature-based algorithms can be composed of feature detection and descriptor building. In the stage of feature detection, points and lines are commonly used. First, the point-based feature detection methods are more sophisticated. Based on SIFT [6], SURF [7], and other traditional algorithms, many researchers have achieved scale invariance in the keypoint detection stage. Ma et al. [8] replaced the traditional gradient with position, scale, and orientation information on the basis of the SIFT algorithm. It was robust to nonlinear intensity transformation, but it was sensitive to geometric and radiometric differences between optical and SAR images. Fan et al. [9] extended the nonlinear diffusion method to extract uniformly distributed keypoints. They proposed a structural descriptor based on the PC information. Xiang et al. [10] proposed a SIFT-like algorithm for optical and SAR images (OS-SIFT). It used a multi-scale Harris detector to improve the keypoint repeatability between optical and SAR images. These above algorithms provided the registration framework based on the keypoint-based method. In addition, for the line-based method, Zhang et al. [11] proposed the features and spectral graph theory to perform SAR and optical image registration. The algorithm extracted line segments, and defined the intersection points between different line segments as keypoints. However, the algorithm was inapplicable to the situation with little structural information. At present, the most popular keypoint extraction method is to extract corner points from the image composed solely of edge information. In optical and SAR images, the edge features obtained by the PC maximum moment are consistent, so the corner point detection methods based on the PC maximum moment are popular with researchers. Ye et al. [12] used the orientation histogram based on the PC model to deal with the nonlinear radiometric difference between optical and SAR images. It used PC information to reflect the structural properties. However, it was sensitive to scale and geometric differences. Li et al. [13] proposed a radiation-variation insensitive feature transform method. It replaces the image intensity values with the PC information to extract keypoints, which improves the compatibility with the radiometric difference between optical and SAR image pairs. However, the algorithm does not consider the scale invariance, so it cannot match the optical and SAR images with large-scale differences. Xie et al. [14] used the complexity analysis method to obtain keypoints, and combined the PC maximum moment with the binary mode to build the novel descriptor. Although the porposed algorithm can overcome the geometric and radiometric differences between optical and SAR images, it is only compatible with small-scale and rotation differences. Wang et al. [15] proposed a 3D descriptor based on the oriented gradients, and it improved the robustness of the registration results. However, it cannot solve large-scale and rotation differences. Meanwhile, Jia et al. [16] used the novel nonlinear diffusion scale space to obtain the keypoints, and it improved the computational efficiency by optimizing the number of matching correspondences.

For the optical and SAR remote sensing image registration, PC-based algorithms are more robust to nonlinear radiometric and geometric differences. However, most of them are sensitive to scale and rotation differences between optical and SAR images. To solve this problem, we propose an algorithm based on the PC framework, which takes the scale and rotation differences between optical and SAR images into account. The main contributions of the proposed algorithm are as follows:

(1)

To solve the problem that the scale invariance is not considered in the PC-based algorithms, we propose a novel multi-scale space based on the PC algorithm, which is constructed by the convolution of the maximum moment map and log-Gabor filter. Then, the Harris detector is used to extract keypoints in the novel multi-scale space and we name the method PC-Harris. PC-Harris is compatible with large-scale differences between optical and SAR images.

(2)

In order to solve the problem that most of the descriptor construction methods based on the PC algorithm are not suitable for large-scale and rotation differences between optical and SAR images, we propose a PC-based descriptor (named PCLG), which combines the PC maximum moment and the log-Gabor filter.

The remainder of this paper is organized as follows: Section 2 describes the PC-based feature detector and the novel descriptor. Section 3 analyzes the performance of the proposed algorithm. Section 4 presents the conclusion and the recommendations for future work.

2. Proposed Method

This section first introduces the PC algorithm [17] and then illustrates the proposed PC-Harris detector and the proposed PCLG descriptor. Figure 1 shows the framework of the proposed algorithm.

2.1. Review of PC Theory

The 2D log-Gabor filter plays an important role in the PC theory, and it can be defined as follows:

$$LG(w,\theta )=\exp (\frac{-{(\ln (w/w_0))}^2}{2{(\ln (\sigma /w_0))}^2})\cdot \exp (\frac{-{(\theta -{\theta }_0)}^2}{2{\sigma }_{\theta }^2})$$

(1)

where $w_0$ is the centre frequency of the 2D log-Gabor filter; $\sigma$ is the bandwidth; $\sigma /w_0$ is a constant; ${\theta }_0$ is the filtering orientation; and ${\sigma }_{\theta }$ is the standard deviation of the Gaussian function. In the spatial domain, the 2D log-Gabor filter can be represented as follows:

$$LG(x,y)=L{G}_{s,o}^{even}(x,y)+i\times L{G}_{s,o}^{odd}(x,y)$$

(2)

where $L{G}_{s,o}^{even}$ and $L{G}_{s,o}^{odd}$ stand for the even-symmetric and the odd-symmetric log-Gabor wavelets in $s$ scale and $o$ orientation, respectively.

Assuming that $I(x,y)$ is an image, convolving $I(x,y)$ with $L{G}_{s,o}^{even}$ and $L{G}_{s,o}^{odd}$ yields the response components $e_{s,o}(x,y)$ and $o_{s,o}(x,y)$ as follows:

$$\left[e_{s,o}(x,y),o_{s,o}(x,y)\right]=\left[I(x,y)\ast L{G}_{s,o}^{even}(x,y),I(x,y)\ast L{G}_{s,o}^{odd}(x,y)\right]$$

(3)

Then, the amplitude component $A_{s,o}(x,y)$ and the phase component ${\varphi }_{s,o}(x,y)$ of $I(x,y)$ at scale $s$ and orientation $o$ can be obtained by the following:

$$A_{s,o}(x,y)=\sqrt{e_{s,o}{(x,y)}^2+o_{s,o}{(x,y)}^2}$$

(4)

$${\varphi }_{s,o}(x,y)=\arctan (o_{s,o}(x,y)/e_{s,o}(x,y))$$

(5)

where $A_{s,o}(x,y)$ is the function that will be used in the descriptor construction process. Finally, the 2D PC model $PC(x,y)$ can be defined as follows:

$$PC(x,y)=\frac{{\displaystyle \sum _s{\displaystyle \sum _o{w}_0(x,y)⌊A_{s,o}(x,y)\Delta {\Phi }_{s,o}(x,y)-T⌋}}}{{\displaystyle \sum _s{\displaystyle \sum _o{A}_{s,o}(x,y)+\xi }}}$$

(6)

where $w_0(x,y)$ is the weighting factor; $⌊\cdot ⌋$ refers to the fact that the enclosed quantity is equal to zero when its value is negative; $T$ denotes the noise threshold; $\Delta {\Phi }_{s,o}(x,y)$ is a phase deviation on the scale $s$ and orientation $o$; and ξ is a small value.

The PC maximum moment M to be used in the proposed algorithm is calculated as follows:

$$a={\displaystyle \sum _{\theta }{(PC(\theta )cos\theta )}^2}$$

(7)

$$b=2{\displaystyle \sum _{\theta }(PC(\theta )cos\theta )(PC(\theta )\sin \theta )}$$

(8)

$$c={\displaystyle \sum _{\theta }{(PC(\theta )sin\theta )}^2}$$

(9)

$$M=\frac{1}{2}(c+a+\sqrt{b^2+{(a-c)}^2})$$

(10)

Thus, the PC maximum moment M and the amplitude component $A_{s,o}(x,y)$ that will be used in the process of keypoint detection and descriptor construction are all obtained.

2.2. The Proposed Algorithm

2.2.1. The Feature Detector PC-Harris

At present, the feature detectors based on the PC maximum moment of the edge images become more prevalent since the PC-based algorithms are not sensitive to the geometric and radiometric differences between optical and SAR images. However, PC-based feature detectors are not compatible with the scale differences. Thus, to solve the scale difference problem between optical and SAR images, we propose a scale-insensitive feature detection method. It convolves the PC maximum moment with the log-Gabor filter to construct the scale space.

First, the PC maximum moment M of the optical or SAR image $I(x,y)$ is calculated. Second, the response map ${A^{\prime }}_{s,o}(x,y)$ in the orientation $o$ and the scale $s$ is calculated by combining the log-Gabor filter and M according to Formulas (3) and (4). The response map ${A^{\prime }}_{s_i}(x,y)$ of the maximum moment M on the scale $s_i$ is obtained as follows:

$${A^{\prime }}_{s_i}(x,y)={\displaystyle \sum _{j=1}^o{A^{\prime }}_{s_i,o_j}(x,y)}$$

(11)

where the scale factor is $s_i\in \left[1,s\right]$. The gradient amplitude $D(s_i)$ and orientation $\theta (s_i)$ are calculated by the gradient components $D_X(x,y)$ and $D_Y(x,y)$ as follows:

$$D(s_i)=\sqrt{D_X{(x,y)}^2+D_Y{(x,y)}^2}$$

(12)

$$\theta (s_i){=\tan }^{-1}(D_Y(x,y)/D_X{(x,y)}^2)$$

(13)

The proposed PC-based Harris scale space is obtained by $D_X(x,y)$ and $D_Y(x,y)$ as follows:

$$M_{s_i}(x,y)=g_{\sqrt{2}{s}_i}\ast \left[\begin{array}{cc}{(D_X(x,y))}^2& D_X(x,y)D_X(x,y)\\ D_Y(x,y)D_X(x,y)& {(D_Y(x,y))}^2\end{array}\right]$$

(14)

$$R_{s_i}(x,y)=\det (M_{s_i}(x,y))-d\cdot tr{(M_{s_i}(x,y))}^2$$

(15)

where g stands for a Gaussian kernel; ∗ represents the convolution operator; d is an arbitrary parameter; and det and tr denote the values of the matrix determinant and the matrix trace, respectively.

Here, experiments are conducted to analyze the performance of the proposed PC-Harris between the optical and SAR images. Figure 2 and

Table 1. Comparisons on repeatability rates.

Method	Multi-Harris	PC-Harris
γ (%)	13.62	15.71

show the quantitative results of the keypoint repeatability based on the multi-scale Harris [12] and the proposed PC-Harris. Among them, the scale factor $s$ is set to 4 and the threshold between repeatable keypoints is set to 3. On the premise that the two detectors extract similar numbers of keypoints, Table 1 shows that the proposed PC-Harris gains approximately 2% of points compared with the multi-scale Harris. Therefore, the proposed PC-Harris can improve the keypoint repeatability between optical and SAR images.

2.2.2. Feature Description

It is well known that the PC-based descriptor is prevalent because of its robustness to nonlinear radiation. However, most of them are sensitive to scale and rotation differences. Thus, we propose a novel PC-based descriptor, which is compatible with large-scale and rotation differences between optical and SAR images. Here, we use the previously calculated response map $A_{s,o}(x,y)$, which contains both orientation and amplitude information, to construct a novel descriptor. Above all, the amplitude and orientation are calculated under the condition that $s$ and $o$ are set to 4 and 6, respectively. The calculated 24 response maps as shown in Figure 3. We sum the response map value $A_{s_i,o_j}(x,y)$ in each scale $s_j$, and obtain the response map $A_{o_j}(x,y)$ in each orientation $o_j$ as follows:

$$A_{o_j}(x,y)={\displaystyle \sum _{i=1}^4{A}_{s_i,o_j}(x,y)}$$

(16)

$$A_o(x,y)=\{A_{o_1}(x,y),\dots ,A_{o_j}(x,y),\dots ,A_{o_6}(x,y)\}$$

(17)

As for each keypoint $(x,y)$, we obtain the maximum response map $A_{o_i}^{\max }(x,y)$ as follows, and the index value $o_i(x,y)$ represents the orientation of $A_{o_i}^{\max }(x,y)$:

$$A_{o_i}^{\max }(x,y)=\max \{A_{o_1}(x,y),\dots ,A_{o_j}(x,y),\dots ,A_{o_6}(x,y)\}$$

(18)

Then, we construct the descriptor vectors based on the above method. On the one hand, we expect that the descriptor is unique. On the other hand, it is better to be insensitive to the geometric and radiometric differences between optical and SAR images. However, the two aspects are contradictory. To solve this problem, we use the novel descriptor construction framework to obtain the orientation histogram. As shown in Figure 4, the inner circle improves the descriptor uniqueness by the histogram statistic of the six orientations based on the four location bins, and the outer circle reduces the sensitivity to nonlinear differences by mapping the six orientations to three orientations. In detail, the steps of the proposed descriptor are as follows:

Step 1: The descriptor framework is constructed in the polar coordinate system, and the main orientation is obtained in the region where one keypoint is located. The framework is rotated to the main orientation to ensure the rotation invariance of the framework. Among them, the main orientation is statistically obtained from the $\theta (x,y)$ corresponding to Formula (13).

Step 2: As shown in Figure 4, two radial areas are composed of 4 and 8 orientation bins, respectively. $o_i(x,y)\in \left[1 6\right]$ and the amplitude of each location bin is counted up in the six orientations. Each location bin corresponds to a 6-dimensional orientation histogram, and 12 groups of 6-dimensional vectors are obtained totally.

Step 3: For the rotation invariance of the elements in one eigenvector, the orientation histogram in each location bin is rotated to the main orientation $\Theta$ by the process in Figure 5 to ensure the rotation invariance of the elements.

Step 4: In the outer circle, the 6-dimensional orientation is mapped to the 3-dimensional orientation. In detail, the amplitudes in two orientations are added together to form the outer circle vector. It is a 24-dimensional vector which consists of 8 location bins and 3 orientations. In the inner circle, a 24-dimensional vector is counted up from the 4 location bins and 6 orientations. The two vectors are concatenated into a 48-dimensional descriptor.

In Figure 6a,b, the yellow circles represent the local regions used for the descriptor construction, and the size is set to $153\times 153$ pixels. Note that the dimension of the proposed descriptor is 48, which is smaller than the descriptor of OS-SIFT. Therefore, only the first 48-dimensional vectors are displayed in Figure 6c,d. It can be seen that the two curves of the optical and SAR images in Figure 6d have a higher fitting degree. Meanwhile, the dimension of the proposed descriptor is smaller than that of the OS-SIFT. It is more conducive to the subsequent calculation efficiency.

3. Experiments

In this Section, to estimate the optical-to-SAR image registration capability of the proposed algorithm, we use the proposed algorithm to compare with OS-SIFT [12] and RIFT [15]. Both of the two comparison algorithms display a good performance for optical and SAR image registration. All methods in this paper are conducted via MATLAB R2020b software.

3.1. Parameter Settings

In the experiments, the thresholds used for the keypoint detection are set to 0.01 and 0.02, respectively. In the process of descriptor construction, to obtain more information, the circle radius of the support region is empirically set to $30s_j$. Settings of the parameters in the two comparison algorithms are the same as described by their authors’ instructions. At the same time, in order to obtain a similar number of keypoints, we fine-turned the thresholds of the keypoint detection.

3.2. The Performance of the Proposed Algorithm

The proposed algorithm is carried out on more than 30 image pairs, and the performance is evaluated using four optical and SAR image pairs that have significant intensity and geometric differences.

Table 2. Details of the test images.

Pair	Image Source	Resolution/m	Date	Size/Pixel
A	Google Earth	2 m	25 July 2020	1707 × 1321
A	Airborne SAR	2 m	5 November 2019	1724 × 1384
B	Google Earth	2 m	25 July 2020	1853 × 979
B	Airborne SAR	2 m	5 November 2019	1612 × 925
C	Google Earth	1 m	9 May 2021	789 × 696
C	Airborne SAR	1 m	1 October 2020	785 × 679
D	Google Earth	1 m	9 May 2021	1454 × 643
D	Airborne SAR	1 m	1 October 2020	1460 × 608

lists the details of the four test image pairs, and Figure 7 shows the matching results of the test images. The four SAR images are from the 23rd institute of the second Academy of China Aerospace Science and Industry Corporation, and the four optical images are from Google Earth.

Four pairs of images describe the suburbs with geometric and intensity differences. Note that the scale and rotation differences of the test image pairs are small. Therefore, the scale factor is set to one. In particular, it is a challenging task since there are roofs and a mountain in Pair B. The side-looking mechanism of SAR sensors makes the roof regions suffer from shadows and strong scattering. The buildings in optical and SAR images have significant geometric and radiometric differences, which leads to a low structural information consistency of the homonymy points. In sum, for the optical and SAR images with small-scale and rotation differences, the matching points of the four image pairs are all located in the correct locations. The proposed algorithm obtains the ideal numbers of matching point pairs, and it achieves the most satisfactory matching results among the three algorithms.

3.3. Experiment Analysis

In

Table 3. CMNs and RMSEs of several methods for test images.

Method	P-A		P-B		P-C		P-D
	CMN	RMSE	CMN	RMSE	CMN	RMSE	CMN	RMSE
OS-SIFT	6	9.2547	11	7.6965	8	6.3563	22	5.0750
RIFT	26	5.7254	43	4.7385	32	4.6788	40	3.7918
Proposed	75	2.0676	36	4.3173	64	2.5721	54	2.1136

, the correct matching number (CMN) and the root mean square error (RMSE) are used to quantitatively evaluate the performances of the three algorithms. We used FSC [18] to obtain CMNs by removing the outliers from the initial correspondences and obtain CMN. RMSE is calculated as follows:

$$RMSE=\sqrt{\frac{1}{m}{\displaystyle \sum _{i=1}^m\left[{(x_i^{o_1}-x_i^s)}^2+{(y_i^{o_1}-y_i^s)}^2\right]}}$$

(19)

We manually selected 20 checkpoint pairs from each set of correspondences to evaluate the matrix H based on the affine transformation model. $(x_i^o,y_i^o)$ and $(x_i^s,y_i^s)$ are the coordinates of the ith correspondence. $(x_i^{o_1},y_i^{o_1})$ represents the transformed coordinates of $(x_i^o,y_i^o)$ through the transformation matrix H, and m denotes the number of the correspondences in optical and SAR images after FSC.

Table 3 shows the quantitative analysis of the three comparison algorithms. Since the geometric information is obvious and there are almost no dense regions, it is relatively easy to register Pair A. The OS-SIFT algorithm describes the image information based on the gradient. The gradient orientations corresponding to Pair A are single, and the representativeness and uniqueness of the descriptors are poor. Therefore, the number of correspondences is few. Both the RIFT and the proposed algorithm use PC to describe the structural information, and the PC-based information of the homonymous regions is consistent between optical and SAR images. Thus, the proposed algorithm and RIFT perform better than OS-SIFT. However, RIFT only extracts corner points from the maximum and minimum moment maps, while the proposed algorithm uses the convolution of the maximum moment map and log-Gabor filter to extract keypoints, which reduces the interference of the speckle noises in SAR images. Therefore, the proposed algorithm has a larger CMN and smaller RMSE than RIFT.

Pair B contains fields, roads, buildings, and mountains, where the structural information is dense and unevenly distributed. In the regions where buildings and mountains are located, OS-SIFT only obtains a few correspondences with low matching accuracy due to the obvious gradient differences and scattering in SAR images. Since the RIFT uses the PC method to capture structural information, RIFT performs better than OS-SIFT. RIFT is insensitive to radiometric differences because it is based on the PC maximum moment information. Although the number of correspondences obtained by RIFT is superior to the proposed algorithm, the proposed descriptor framework is unique and has a better discrimination ability for the farmland with single structural features. Therefore, the proposed algorithm has a more uniform correspondences distribution and a smaller RMSE than RIFT.

For Pair C with a time differences, the corresponding field is divided into two by a road. The cars on the road can be clearly seen in the optical image, which are not present in the corresponding SAR image. The gradient difference between optical and SAR images is not obvious. Thus, the correspondences obtained by OS-SIFT are few, and they are concentrated in the areas where the gradient differences are significant. Since the proposed descriptor is representative, the proposed algorithm is superior to RIFT in both CMN and RMSE.

Pair D corresponds to rural areas which consist of roads and fields. Although the intensity difference is significant, the structural features are rich and the radiometric difference is small. The three compared algorithms can overcome the intensity difference and achieve ideal matching results. The RMSE of the proposed algorithm is above two pixels. However, it is superior to OS-SIFT and RIFT both in CMN and RMSE. Note that there are obvious geometric distortions between optical and SAR images, and the affine transformation model used cannot effectively solve these distortions. Therefore, the RMSEs in Table 3 are relatively large. In sum, the experiments show that the proposed algorithm performs best among the three state-of-the-art algorithms.

3.4. Scale and Rotation Variations Experiments of the Proposed Algorithm

We tested the compatibility of the proposed algorithm with the scale and rotation differences between optical and SAR images.

Figure 8 and

Table 4. CMNs with different scale factors between optical and SAR image pair.

Scale	0.3	0.4	0.6	0.8	1.0	1.2	1.4	1.6	1.8
CMN	7	12	26	56	75	32	23	19	8

show the registration results under the several scale differences for Pair A. We can see that when the scale difference increases, the value of CMN gradually decreases. We verified that the algorithm in this paper is well-behaved with scale differences in the range of 0.3–1.8 times between optical and SAR images.

In addition, for the optical and SAR image pairs with rotation differences,

Table 5. CMNs with different rotation angles between optical and SAR image pair.

Rotation Angle/°	−150	−120	−90	−60	−30	0	30	60	90	120	150
CMN	16	18	19	23	35	75	43	32	42	26	20

and Figure 9 show the registration results under the various rotation differences for Pair A. From Table 5, excepting the fact that the CMN with the 90° rotation difference is larger than that of the 60° rotation difference, CMN decreases with the increase in the rotation difference. As can be seen from Figure 9, there are many correspondences between optical and SAR images with different rotation angles. Table 5 shows that the proposed algorithm can be compatible with the rotation differences between optical and SAR images, ranging from −150° to 150°.

In summary, the proposed algorithm is insensitive to geometric and radiometric differences, and can successfully register optical and SAR images with large-scale and rotation differences. However, the proposed algorithm in this paper has not fully covered the full-scale and full-angle differences between optical and SAR images, and we will focus on solving this deficiency in future work.

4. Conclusions and Future Work

This paper proposed an optical and SAR image registration algorithm based on the PC framework, which is novel and insensitive to scale and rotation differences between optical and SAR image pairs. The multi-scale PC-Harris is constructed based on the log-Gabor filter and the PC maximum moment. Furthermore, we constructed a discriminative descriptor based on the PC model. Experiments verify that the proposed algorithm is compatible with large-scale and rotation differences between optical and SAR images. It effectively solved the problem that the traditional PC-based registration algorithms cannot be compatible with large-scale and rotation differences between optical and SAR images. The limitations are that it has not fully covered the full-scale and full-angle differences, and the matching accuracy needs to be further improved. In the future, we will combine the local and global PC information to improve the matching performance.