Range–Visual–Inertial Odometry with Coarse-to-Fine Image Registration Fusion for UAV Localization

Abstract

In Global Navigation Satellite System (GNSS)-denied environments, image registration has emerged as a prominent approach to utilize visual information for estimating the position of Unmanned Aerial Vehicles (UAVs). However, traditional image-registration-based localization methods encounter limitations, such as strong dependence on the prior initial position information. In this paper, we propose a systematic method for UAV geo-localization. In particular, an efficient range–visual–inertial odometry (RVIO) is proposed to provide local tracking, which utilizes measurements from a 1D Laser Range Finder (LRF) to suppress scale drift in the odometry. To overcome the differences in seasons, lighting conditions, and other factors between satellite and UAV images, we propose an image-registration-based geo-localization method in a coarse-to-fine manner that utilizes the powerful representation ability of Convolutional Neural Networks (CNNs). Furthermore, to ensure the accuracy of global optimization, we propose an adaptive weight assignment method based on the evaluation of the quality of image-registration-based localization. The proposed method is extensively evaluated in both synthetic and real-world environments. The results demonstrate that the proposed method achieves global drift-free estimation, enabling UAVs to accurately localize themselves in GNSS-denied environments.

1. Introduction

Owing to small size, high agility, and low cost, UAVs have been widely applied in search and rescue [1], environmental monitoring [2], aerial photography, and other fields [3,4,5]. However, to successfully carry out these tasks, precise localization of UAVs is of utmost importance. Although the GNSS is the most commonly used localization system, providing the geographic coordinates of UAVs, its reliability is compromised by challenges such as multipath reception [6] and Non-Line-Of-Sight (NLOS) reception [7], which can introduce inaccuracies. In addition, the increasing threat of signal spoofing poses a significant risk to UAVs, particularly in defense and security applications [8]. Therefore, it is essential to develop methods for locating UAVs in GNSS-denied environments.

In GNSS-denied environments, UAVs can utilize onboard sensors, including cameras and Inertial Measurement Units (IMUs), to infer their position without depending on external infrastructure. These two sensors can be integrated to form Visual-Inertial Odometry (VIO), which estimates the UAV’s position and orientation by processing data acquired from both the camera and the IMU. However, when the UAV moves in uniform linear motion, the acceleration is constant, resulting in VIO’s inability to estimate the scale. To address this issue, an alternative approach involves incorporating a 1D LRF, which is a lightweight and accurate sensor that assists VIO in observing scale information. On the other hand, it is imperative to emphasize that without global information, odometry methods are susceptible to drift and can only provide the local pose of the UAV rather than a more generic global pose. This limitation highlights the need for additional techniques to mitigate drift and obtain a more generic global pose estimation for UAVs operating in GNSS-denied environments.

Recently, image-registration-based localization has been used to attain the global pose of a UAV [9,10,11,12]. This method involves registering the images taken from the UAV to the pre-existing maps, thereby enabling the acquisition of accurate global positioning information for the UAV. The most-used pre-existing maps are orthorectified satellite imagery, which is easily accessible and GNSS-aligned. However, the primary challenge associated with this method lies in matching images from different sources. These images often exhibit variations in perspective, lighting conditions, seasons, and other factors, making traditional image matching methods such as template matching and feature matching susceptible to failure [13].

In this study, we present a novel method for performing global localization by utilizing RVIO to track UAV motion, registering UAV images to geo-referenced satellite imagery, and fusing measurements with global pose graph optimization. As depicted in Figure 1, our method comprises two modules: the RVIO module and the image-registration-based geo-localization module. The RVIO module utilizes a downward-facing monocular camera, an LRF, and an IMU to estimate the relative motion between consecutive frames of the UAV. The image-registration-based geo-localization module comprises two steps: coarse matching and fine matching. During the coarse matching step, aided by the LRF, the suitable map tiles are retrieved from the database generated through satellite image segmentation. These suitable map tiles serve as potential candidates for further registration. Subsequently, in the fine matching step, the image captured by the UAV is registered with the selected candidates from the coarse matching step. This registration process enables us to acquire global position measurements for the UAV. To obtain a globally consistent localization estimate, the UAV ego-motion measurements derived from RVIO and the global position measurements are fused through a global pose graph optimization.

This work makes the following contributions:

• We propose an optimization-based RVIO method which utilizes range measurements to effectively suppress scale drift from VIO, thereby enhancing localization accuracy.
• We propose a coarse-to-fine image-registration-based localization method that provides the global pose of the UAV. With the assistance of LRF, the retrieval efficiency of the coarse matching step is improved. Then, the fine matching step calculates the accurate geographic position of the UAV.
• The proposed method is evaluated on both synthesized and real-world datasets. The results demonstrate the effectiveness of our method.

The rest of this paper is organized as follows. In Section 2, we discuss the related works. Section 3 illustrates the proposed method used for UAV geo-localization. In Section 4, we describe the experiments we conducted on both synthesized and real-world datasets. The conclusions are summarized in Section 5.

2. Related Works

2.1. Range–Visual–Inertial Odometry

VIO frameworks can be classified into two main categories: tightly coupled and loosely coupled methods. Delmerico et al. [14] conducted a comprehensive investigation into various VIO frameworks and reported that the tightly coupled approach exhibited superior performance compared with the loosely coupled alternative. Tightly coupled VIO frameworks can be further divided into two categories: filter-based algorithms and optimization-based algorithms. Examples of the former include MSCKF [15], ROVIO [16], etc. However, filter-based algorithms have a theoretical limitation: they must linearize nonlinear measurements before processing, which can lead to significant linearization errors that may affect the accuracy of the estimation. In contrast, optimization-based algorithms [17,18,19,20] can perform relinearization during each iteration step, resulting in more accurate estimates and are, therefore, generally preferred.

In addition, range sensors can effectively reduce the scale drift of odometry. Various methods have been proposed using 2D or 3D LiDARs [21], RGB-D cameras [22], or stereo cameras, among others. However, the size and measurement distance of these sensors can limit their applicability to UAVs. Conversely, 1D LRFs offer a compelling solution, as they are compact, lightweight, and well-suited for integration with most UAV platforms. For instance, Giubilato et al. [23] used an LRF to recover and maintain the correct metric scale for visual odometry. Based on this, they proposed a range-enhanced monocular system and an extrinsic calibration pipeline [24]. For VIO applications, Delaune et al. [25] proposed an RVIO system that leverages a range measurement update model to assign depth to features and eliminate scale drift. Hu et al. [26] extended the RVIO method by fully exploiting distance measurements to constrain all coplanar features. Simultaneously, they conducted online extrinsic calibration between the LRF and camera. However, these methods only constrain the depth of feature points, without taking into account the constraint of UAV flight altitude. Furthermore, unlike the filter-based methods they use, our proposed method implements RVIO using an optimization-based framework.

2.2. Image-Registration-Based Localization

There have been various methods developed for UAV image registration and localization in GNSS-denied environments. As reviewed in [13], the works in the field of image registration and localization can be divided into three classes depending on the different image matching technologies: template matching, feature matching, and deep learning matching.

The key to the template matching method is the choice of a suitable similarity metric function. In early works, the Sum of Squared Differences (SSD) was the chosen metric, which involves a comparison of the luminance of each pixel [27]. However, it is sensitive to changes in scenarios. Recent works utilize more robust similarity measures, such as Mutual Information (MI) [28], Normalized Cross-Correlation (NCC) [29], and Normalized Information Distance (NID) [30]. Nevertheless, the template matching method is inefficient, and the computational burden is correlated to the size of the satellite image.

In addition, feature matching stands as another approach to accomplish image matching. The typical feature descriptors include SIFT [31], ORB [32], etc. However, the performance of these traditional descriptors decreases in the case of larger scenario changes and geometric transformations.

Semantic features have been used to accomplish the image-matching task. Nassar et al. [33] utilized semantic segmentation to extract the meaningful shapes of both UAV images and satellite imagery. A semantic shape-matching pipeline is performed to find correspondences between UAV images and satellite imagery. Choi et al. [34] constructed a feature descriptor called the building ratio feature by extracting the building areas from images. The matching algorithm based on the building ratio feature provides the global position estimation to correct the local odometry. Xu et al. [35] proposed a point–line–patch feature descriptor with rotation and scaling invariance. Based on the descriptors, the geo-localization is calculated and refined through the application of the Inverse Compositional Lucas–Kanade (ICLK) algorithm. However, the semantic features require the presence of buildings or roads in the scene, which limits the practical applications.

Moreover, utilizing the high-level features from deep learning methods is a natural choice. Goforth et al. [36] employed a deep CNN with an ICLK layer to align UAV images with satellite images. Then, they developed an optimization to refine the UAV’s pose. Chen et al. [37] proposed a two-stage image-based geo-localization pipeline that adapts to downward-tilted camera configurations. However, this method requires offline preparation of the dataset in advance. Kinnari et al. [38] performed orthorectification of the UAV image based on VIO and planarity assumption, which makes the UAV camera orientation not strictly required. The ortho-projected image is used to match with the satellite imagery for geo-localization, and the results and the tracking pose from VIO are fused in a particle filter framework. Further, they used learned features to replace the classical image-matching metrics to obtain seasonally invariant localization [39]. However, the works mentioned above directly employ the satellite map with suitable regions and resolutions, while in practice, the area covered by satellite images is much larger than that of UAV images. Successfully aligning UAV images with larger satellite maps becomes an impractical task. Therefore, the interested area needs to be retrieved in the satellite image. In contrast to the aforementioned methods, we present a novel image-registration-based geo-localization approach that follows a coarse-to-fine strategy, eliminating the need for any initial prior information. Moreover, our method fuses both global position measurements and local position measurements obtained from RVIO using a global pose graph optimization. This fusion process facilitates the attainment of a globally consistent localization estimate.

3. Method

In this section, we elaborates on the specifics of our proposed localization method. The method comprises the RVIO module and the image-registration-based geo-localization module that executes the ensuing operations: (1) estimating the local pose of the UAV using RVIO, (2) registering the UAV image with satellite imagery for geo-localization, and (3) fusing the global position measurements and local pose through a global pose graph optimization to yield the global pose estimation of the UAV.

3.1. Range–Visual–Inertial Odometry

3.1.1. Visual–Inertial Odometry

The architecture of our RVIO is based on [17], which is a sliding-window keyframe-based nonlinear optimization framework. All states in the sliding window are defined as

$${\gamma }_{VIO}=\left[{\gamma }_I,{\gamma }_{\lambda }\right],$$

(1)

where ${\gamma }_{\lambda }=[{\lambda }_0,{\lambda }_1,\cdots ,{\lambda }_n]$ represents the inverse depth of the landmarks upon their initial observation in the camera frame, and ${\gamma }_I=\left[{\mathit{x}}_0,{\mathit{x}}_1,\cdots ,{\mathit{x}}_m\right]$ represents m IMU states. Specifically, the k-th IMU state is defined as the UAV’s position ${}^L{\mathit{p}}_{I_k}={[x_{I_k},y_{I_k},z_{I_k}]}^T$, velocity ${}^L{\mathit{v}}_{I_k}$, orientation quaternion ${}^L{\mathit{q}}_{I_k}$, and IMU bias ${\mathit{b}}_a,{\mathit{b}}_g$. The transformation between the IMU frame $\left\{I\right\}$ and the camera frame $\left\{C\right\}$, as well as the transformation between $\left\{C\right\}$ and the LRF frame $\left\{F\right\}$, have been previously calibrated and are regarded as known. Without loss of generality, these frames are considered to be coincident in what follows.

For each camera image, features are detected and tracked between consecutive frames using an optical flow algorithm [40]. Meanwhile, the IMU measurements between two consecutive frames undergo preintegration [41]. After initialization, all the measurements are placed in a sliding window for nonlinear optimization. The states ${\gamma }_{VIO}$ are estimated through the minimization of the cost function, which takes into account the marginalization information, inertial residuals, and visual residuals. Specifically, the inertial residuals ${\mathit{e}}_I^k$ are calculated by leveraging IMU preintegration between two successive frames within the sliding window. The visual residuals ${\mathit{e}}_C^{h,j}$ delineate the reprojection error by reprojecting the landmark ${}^L{\mathit{P}}_h$ onto keyframe $K_j$ and subsequently comparing it with the original raw visual measurements ${\widehat{\mathit{z}}}_{h,j}$. ${\mathit{e}}_P$ stands as the marginalization residuals that encompass information pertaining to previously marginalized states. The cost function is expressed as follows:

$$\begin{array}{c}\hfill J_{VIO}=\sum _{h,j\in \mathcal{C}}{\parallel {\mathit{e}}_C^{h,j}\parallel }_{{\mathit{W}}_C^{h,j}}^2+\sum _{k\in \mathcal{I}}{\parallel {\mathit{e}}_I^k\parallel }_{{\mathit{W}}_I^k}^2+{\parallel {\mathit{e}}_P\parallel }^2,\end{array}$$

(2)

where $\mathcal{I}$ represents the set of all IMU measurements, while $\mathcal{C}$ represents the set of all features that have been observed at least twice within the sliding window. To calculate the norm of the residuals, the Mahalanobis distance weighted by covariance $\mathit{W}$ is used. The notation $\left|\right|·{\left|\right|}_{\mathit{W}}$ denotes the Mahalanobis distance.

3.1.2. Range Measurements

Whenever a new frame is added to the sliding window, the most recent range measurement ${\mathit{d}}_i^F$ is stored. As the optical center of the camera and the origin of the LRF are coincident, the range measurement can be represented as a 3D landmark of depth $d_i^F$ in the camera frame $\left\{C\right\}$. In the ideal scenario, LRF would emit rays for measurement. However, due to the detection angle, the LRF emits a rectangular spot of light instead. The size of the spot varies at different distances, with the spot becoming larger as the distance increases. If the detected object’s length fall short of the dimensions of the detection area’s edges, the LRF is unable to provide valid data. Therefore, we adopt the assumption that the detection area is locally flat when the data from the LRF is valid.

Figure 2 illustrates the association of range measurement and features. Based on the flat assumption, the features associated with a range measurement are those for which the corresponding landmark is within the detection area. Let ${\mathit{f}}_i^F$ denote the projection of the range measurement onto the image plane. We can formulate the range measurement as ${\mathit{d}}_i^F={\left[{\mathit{f}}_i^F,d_i^F\right]}^T$, where ${\mathit{f}}_i^F=\left[u_i,v_i\right]$ is the pixel coordinate of the projection point. Each associated feature ${\mathit{f}}_j=\left[u_j,v_j\right]$ must satisfy the following formula:

$$\begin{array}{c}\hfill \left\{\begin{array}{c}\left|\right|u_j-u_i\left|\right|<d_u\hfill \\ \left|\right|v_j-v_i\left|\right|<d_v,\hfill \end{array}\right.\end{array}$$

(3)

where $d_u$ and $d_v$ are determined by edge length of the detection area of the LRF. To avoid incorrect constraints from landmarks that do not satisfy the planar assumption, we perform a standard deviation test. Only features that meet the following condition are optimized:

$$\begin{array}{c}\hfill \sqrt{\frac{1}{n_i}\sum _{j=1}^{n_i}\left|\right|d_j-\mu {\left|\right|}^2}<\sigma ,\end{array}$$

(4)

where $n_i$ is the number of associated features, $d_j$ is the depth of the associated landmark, $\mu$ is the average depth of the associated landmarks, and $\sigma$ is an empirical threshold.

3.1.3. Joint Optimization

VIO triangulation suffers from scale ambiguity. By the range measurements from LRF, there is an additional constraint for the depth of the landmarks. Therefore, we can formulate the residuals of the j-th feature in the i-th frame as

$$\begin{array}{c}\hfill {\mathit{e}}_D^{i,j}=d_i^F-\frac{1}{{\lambda }_j}.\end{array}$$

(5)

Furthermore, the downward-facing LRF can measure the height of the UAV directly. By setting the first measurement as the origin height, subsequent measurements of height can be obtained. The height residuals ${\mathit{e}}_H^i$ are expressed as

$$\begin{array}{c}\hfill {\mathit{e}}_H^i=d_i^F-h^L\left({\mathit{x}}_i\right),\end{array}$$

(6)

where $h^L\left({\mathit{x}}_i\right)=z_{I_i}$.

To fuse all the measurements in the sliding window, we construct a pose graph, as shown in Figure 3. The cost function (2) needs to be extended with additional terms, resulting in the following equation:

$$\begin{array}{c}\hfill J_{RVIO}=J_{VIO}+\sum _{i\in \mathcal{F}}{\parallel {\mathit{e}}_H^i\parallel }_{{\mathit{W}}_H^i}^2+\sum _{i\in \mathcal{F}}\sum _{j\in \mathcal{D}}{\parallel {\mathit{e}}_D^{i,j}\parallel }_{{\mathit{W}}_D^{i,j}}^2,\end{array}$$

(7)

where $\mathcal{F}$ represents the set of all range measurements within the sliding window, and $\mathcal{D}$ denotes the set of landmarks for which the depth $d_j$ is constrained by the range measurements $d_i^F$. To solve this nonlinear problem, we employ the Google Ceres Solver [42], which utilizes the Levenberg–Marquadt approach.

3.2. Coarse-to-Fine Image-Registration-Based Localization

3.2.1. Coarse Matching

Registering UAV imagery directly within large satellite imagery is a highly challenging task. Therefore, it is necessary to crop the satellite imagery into smaller map tiles. These map tiles are intentionally cropped with suitable overlap to ensure a successful registration. The goal of coarse matching is to identify the most appropriate map tile from the database of cropped map tiles for a subsequent fine matching step.

Rather than storing the original images of the map tiles, we extract the descriptors from the tiles to construct a database. We use NetVLAD [43] to extract the descriptors of the map tiles. NetVLAD is a deep learning implementation of the VLAD [44] process, which extracts convolutional feature maps and estimates both the cluster centers and the residual vectors in an end-to-end manner. We encode each map tile as a 4096-dimensional descriptor and add it to the database. Furthermore, we include two geographical coordinates for each map tile in the database, representing the latitude and longitude of the upper-left corner and lower-right corner of the map tile, respectively. Since the area of the map tile is small, we can posit a linear correlation between the pixel coordinates and the corresponding geographical coordinates. Therefore, the geographical coordinates of each point within the map tile can be calculated utilizing the two geographical coordinates stored within the database. For a $W\times H$ pixels map tile, the geographical coordinates $(lat,lon)$ of point $(c_x,c_y)$ are calculated as

$$\begin{array}{c}\hfill lon=c_x·\frac{lo{n}_r-lo{n}_l}{W}+lo{n}_l\\ \hfill lat=c_y·\frac{la{t}_r-la{t}_l}{H}+la{t}_l,\end{array}$$

(8)

where $(la{t}_l,lo{n}_l)$ and $(la{t}_r,lo{n}_r)$ represent the geographical coordinates of the upper-left corner and lower-right corner of the map tile, respectively.

It is noteworthy to mention that the UAV operates at different heights, resulting in varying ranges of the photographed areas. These differences in scale pose a challenge to the matching process. To address this issue, it is necessary to store map tiles of different sizes that simulate the different heights at which the UAV operates. However, including additional data of varying heights in the database not only expands the vertical working range of the UAV but also increases the complexity of retrieval. Consequently, to enhance retrieval efficiency, we leverage LRF measurements to limit the search to the data corresponding to the height obtained from the LRF measurements stored in the database.

Similar to the construction of the map tile database, NetVLAD is also utilized to extract the descriptor of the UAV image. The similarities between the descriptor of the UAV image and every descriptor in the database are computed and sorted in descending order. To identify potential candidates for further registration, we select the top n similarities from the sorted list. It should be noted that the selected similarities must undergo a threshold test. Only the map tiles associated with the selected similarities that exceed the predefined threshold are considered as candidates for the subsequent fine matching step. Empirically, we set $n=3$ for the experiment.

3.2.2. Fine Matching

After the coarse matching step, we need to find the correct retrieval from the candidates. We rerank the candidates by using local feature extraction and matching. For robustness in handling scene changes, we select SuperPoint [45] as the local descriptor. To match the local descriptors extracted from two images, we leverage SuperGlue [46] due to its exceptional proficiency in matching features in significantly different images. The map tile with maximum number of matched points is considered the best candidate and is used in the subsequent registration process.

Consequently, we can establish the correspondences between the local keypoints of the UAV image and those of the best candidate. As the UAV images are captured by a downward-facing camera and the satellite imagery is orthorectified, we can describe the position of UAV using the planar homography by flat-world assumption. The homography transformation is calculated using the correspondences, and RANdom SAmple Consensus (RANSAC) is applied to avoid outliers. The homography transformation allows us to compute the corresponding point to the center of the UAV image in the best candidate map tile, from which we can obtain the geographical coordinates of the UAV by Equation (8). Then, we convert the geographical coordinate into ENU (East North Up) coordinates, representing the global position measurement denoted as ${}^G{\widehat{\mathit{p}}}_{I_i}={{[}^G{x}_{I_i}{,}^G{y}_{I_i},0]}^T$.

However, due to the vibration of the UAV, we cannot guarantee that the camera remains strictly pointing vertically downward during UAV flight, which results in deviations of the global position measurements. For instance, at a flight height of 150 m, an angle of $5^{\circ }$ results in a deviation of 13 m. Therefore, we compensate for the deviation to obtain more accurate global positions. During the initialization procedure of RVIO, the z-axis of the local RVIO frame, denoted as $\left\{L\right\}$ (the first camera frame), is aligned with the direction of the gravity. We utilize the intersection vector between the plane where the LRF measurement hits and is perpendicular to the direction of gravity and the plane formed by the optical axis and gravity to compensate for the deviation

3.2.3. Global Pose Graph Optimization

Once we obtain the geographical coordinates, we fuse them with the local odometry estimates from RVIO into a global pose graph optimization scheme. The estimated local poses of the UAV are added to the global pose graph and serve as vertices after the RVIO process. Each vertex is connected to others by a sequential factor. Upon successful registration of the UAV image to the map tile, we obtain the UAV’s global position to derive a global positional factor. This enables us to incorporate the global factors to the global pose graph. Moreover, the global frame $\left\{G\right\}$ where the global position measurements are located and the local RVIO frame $\left\{L\right\}$ need to be aligned. We introduce the transformation ${}^G{\mathit{T}}_L=\left[{}^G{\mathit{q}}_L{,}^G{\mathit{p}}_L\right]$ into the states. As a result, the states subjected to optimization are defined as

$$\begin{array}{c}\hfill \gamma =\left[{}^L{\mathit{p}}_{I_0},{}^L{\mathit{q}}_{I_0},\dots {,}^L{\mathit{p}}_{I_n},{}^L{\mathit{q}}_{I_n}{,}^G{\mathit{p}}_L,{}^G{\mathit{q}}_L\right],\end{array}$$

(9)

where n is the number of all poses of frames that are included in the graph. Figure 4 illustrates the structure of the global pose graph.

The sequential factor between two frames $i+1$ and i is derived using their relative pose. The residual of the sequential factor between frames $i+1$ and i can be expressed as

$$\begin{array}{c}\hfill {\mathit{e}}_S^i{(}^L{\mathit{p}}_{I_i}{,}^L{\mathit{q}}_{I_i}{,}^L{\mathit{p}}_{I_{i+1}}{,}^L{\mathit{q}}_{I_{i+1}})=\left[\begin{array}{c}{}^L{\mathit{R}}_{I_{i+1}}^{-1}{(}^L{\mathit{p}}_{I_i}{-}^L{\mathit{p}}_{I_{i+1}}){-}^{I_{i+1}}{\widehat{\mathit{p}}}_{I_i}\\ {}^{I_{i+1}}{\widehat{\mathit{q}}}_{I_i}{}^L{{\mathit{q}}_{I_i}^{-1}}^L{\mathit{q}}_{I_{i+1}}\end{array}\right],\end{array}$$

(10)

where the notation $\widehat{(·)}$ represents the noisy measurement. The residual of the global factor for frame i is defined as

$$\begin{array}{c}\hfill {\mathit{e}}_G^i{(}^L{\mathit{p}}_{I_i}{,}^L{\mathit{q}}_{I_i}{,}^G{\mathit{p}}_L{,}^G{\mathit{q}}_L)={}^G{\mathit{R}}_L{}^L{\mathit{p}}_{I_i}-{}^G{\widehat{\mathit{p}}}_{I_i}+{}^G{\mathit{p}}_L.\end{array}$$

(11)

Hence, the cost function can be formulated as

$$\begin{array}{c}\hfill J=\sum _{i,i+1\in \mathcal{S}}{\parallel {\mathit{e}}_S^i\parallel }_{{\mathit{W}}_S^{i,i+1}}^2+\sum _{i\in \mathcal{G}}{\parallel {\mathit{e}}_G^i\parallel }_{{\mathit{W}}_G^i}^2,\end{array}$$

(12)

where $\mathcal{S}$ denotes the set of all sequential factors, and $\mathcal{G}$ denotes the set of all global factors.

3.2.4. Image Registration Evaluation

In order to effectively perform the global pose graph optimization, it is crucial to evaluate the quality of the global position measurements derived from image registration. We, therefore propose an evaluation strategy that considers four factors jointly: the number of matched points ${\psi }_n$, the angle between the optical axis of the camera and the direction of gravity ${\psi }_a$, the discrepancy between the global position measurement and the previous estimation from the optimization ${\psi }_d$, and prior information ${\psi }_p$ (inherent errors in satellite imagery, etc.).

We define the quality of the global position measurement as

$$\begin{array}{c}\hfill Q={\beta }_n{\psi }_n+{\beta }_a{\psi }_a+{\beta }_d{\psi }_d+{\beta }_p{\psi }_p,\end{array}$$

(13)

where ${\beta }_n$, ${\beta }_a$, ${\beta }_d$, and ${\beta }_p$ are the balance factors. Consequently, the residual weights of the global factor ${\mathit{W}}_G^i$ can be represented in terms of the quality Q as ${\mathit{W}}_G^i=Q·\mathit{I}$. This approach ensures that the optimization process takes into account the quality of the global position measurements to obtain more accurate results.

4. Experiments

In this section, we conduct evaluations of the proposed method on both synthesized dataset and real-world datasets. The implementation of our proposed method relies on the open-source framework VINS-Fusion [47]. All experiments were performed on an Intel NUC Mini PC equipped with a 4.70 GHz Intel Core i7 processor and Nvidia Geforce RTX 2060 discrete graphics.

4.1. Synthesized Dataset

4.1.1. Setup

To validate the proposed method, we develop a simulation environment using the Unreal Engine 4 as the development engine. We import real physics world data into the engine using the Cesium for Unreal plugin. The data comprises 3D topographic scenes with realistic geographic coordinates. To collect data in the simulation environments, we use AirSim [48] to control a UAV equipped with multiple sensors, including a downward-facing camera, IMU, and a downward-facing LRF. All the measurements are recorded through a Robot Operating System (ROS). GNSS data are recorded as the ground truth.

4.1.2. RVIO Performance

To evaluate the performance of our proposed RVIO, we compare it with other methods, including the method without range measurements (marked as VIO [17]) and the method that only constrains height by the range measurements (marked as HVIO [47]). We conduct the evaluation on sequence 1, which is collected in the simulation environment. The trajectories of different methods on this sequence are shown in Figure 5. All estimated trajectories are aligned with the ground truth using [49]. As shown in Figure 6, the estimated height of UAV from VIO is inaccurate, while the heights estimated by HVIO and RVIO are closer to the ground truth. We also calculated the root mean square error of the Absolute Trajectory Error (ATE) on sequence 1, and the results are presented in

Table 1. RMSE(m) of ATE for different methods on synthesized datasets.

Sequence	VIO	HVIO	RVIO	Proposed Method
1	14.846	4.566	4.288	-
2	49.503	27.479	21.453	16.149

. The results demonstrate that RVIO outperforms HVIO. These findings suggest that the constraints provided by the range measurements for the depth of landmarks can improve localization performance.

4.1.3. Geo-Localization Performance

We collected another sequence in the simulation environment to evaluate the performance of our proposed method, which fuses the RVIO estimation and the image-registration-based geo-localization results. In this sequence, we fly the UAV along a route that is 3.49 km long and 300 m high. The geo-referenced satellite imagery is obtained from Google Earth Pro, and we used a 3.2 km × 2.1 km satellite map that was captured in January 2019. Figure 7 illustrates the trajectories obtained from different methods on sequence 2.

An example of successfully matched UAV image and map tile is shown in Figure 8. The results of this experiment, presented in Table 1, show that our proposed method suppresses the drifts of the odometry thanks to image-registration-based geo-localization.

4.2. Real-World Dataset

4.2.1. Setup

The real-world experimental setup is displayed in Figure 9. A GNSS receiver is attached to the UAV to allow for ground-truth comparison. The camera, IMU, and LRF are calibrated offline and integrated to place under the UAV. All the measurements are time-synchronized and recorded on an Intel NUC Mini PC through ROS. Based on the setup, we collected two real-world datasets in Tsingtao in March 2023. The trajectories of the UAVs are plotted on the satellite map in Figure 10, encompassing a variety of terrains, such as fields, residential zones, pools, and roads. Additional characteristics of these two datasets are listed in

Table 2. Characteristics of the real-world datasets.

Dataset	Color	Length (m)	Altitude (m)	Duration (s)
1	orange	4426	160	663
2	green	3371	160	477

. The satellite imagery employed for these experiments is 3.31 km × 5.27 km in size and was obtained from Google Earth Pro in November 2021.

4.2.2. Results and Discussion

To simulate different flight heights, the satellite imagery is categorized into three distinct sizes. Each size consists of a specific number of map tiles: 4701, 4420, and 4150 tiles, corresponding to flight heights of 120 m, 240 m, and 320 m, respectively. Despite the extensive size of the database, we leverage LRF measurements to efficiently access a specific subset of the database. This enables us to retrieve relevant information more efficiently and streamline the retrieval process. Additionally, we conducted tests to evaluate the recall rate for retrieving the best candidate from the dataset. In dataset 1, the recall rate for the images is measured at 73.12%, while in dataset 2, the recall rate is recorded as 70.83%.

Similarly, we compare our proposed method to other methods outlined above. The ground-truth trajectory is provided by GNSS. The trajectories of different methods on the real-world dataset are shown in Figure 11. The black square points indicate the geo-localization results obtained from successful image registration. As shown in Figure 11a, the trajectory of the proposed method more closely matches the ground truth. Figure 11b reveals a scale drift in the results of VIO and HVIO in the initial stage of constant acceleration. However, with the additional constraints for the depths of feature points, the scale drift can be effectively suppressed.

To ensure efficiency, the image-registration-based localization process is performed every five seconds, considering its time-consuming nature. The average runtime of the image-registration-based localization on the two datasets is presented in

Table 3. Average runtime (s) of the image-registration-based localization on real-world datasets.

Dataset	Encoding Time	Retrieval Time
1	0.11	2.96
2	0.13	2.93

. As a result, the coarse-to-fine image-registration-based localization module does not impact the global optimization thread. The mean errors of the image-registration-based localization on the datasets are reported as 21.012 m and 23.594 m, respectively. Figure 11 illustrates that image-registration-based localization tends to encounter challenges when the UAV is positioned above the field area. This is attributed to the similarity in global descriptors formed by the map tiles within the fields, leading to incorrect matches. For comparative analysis, we reimplemented the state-of-the-art approach [36]. Despite providing the initial GNSS coordinates of the UAV as prior information, this approach still fails to deliver accurate results on the datasets.

The results of geo-localization are fused with the RVIO estimates through global optimization. It is observed that our method exhibits a slight shift toward the image-registration-based localization results. The extent of this offset is influenced by the quality of the global position measurements. To evaluate the performance, we present the Root Mean Square Error (RMSE) for each case in

Table 4. RMSE (m) of ATE for different methods on real-world datasets.

Dataset	VIO	HVIO	RVIO	[36]	Proposed Method
1	71.944	65.442	38.820	-	12.744
2	101.395	87.541	29.064	-	23.099

. Overall, the proposed method provides global position measurements and effectively bounds the drift in odometry, thus enhancing the accuracy and reliability of the system.

5. Conclusions

In this paper, we propose a geo-localization method for UAVs by fusing RVIO estimates and image-registration-based localization results. In particular, the proposed optimization-based RVIO utilizes range measurements to ensure scale consistency. The coarse-to-fine image-registration-based geo-localization module provides an accurate geographical coordinate of the UAV. Then, the global pose graph optimization adaptively adjusts the weights of global position measurements to obtain the global estimation of the UAV. From the experimental results, we can draw these conclusions:

• Based on the planar measurement characteristics of the LRF detection area, it is possible to achieve data association between range measurements and visual feature point depths, thereby achieving accurate and scale-consistent estimation for RVIO.
• The coarse-to-fine image-registration-based geo-localization enables global localization of the UAV and eliminates the drift of odometry methods. Additionally, with the assistance of LRF, the retrieval efficiency can be improved.
• By employing global graph optimization, the results of image-registration-based geo-localization and the outputs from odometry can be effectively fused. Experimental results demonstrate that the proposed method exhibits better localization performance compared with state-of-the-art image-registration-based geo-localization methods.

It should be noted that the proposed method is more suitable for UAV localization in scenarios where the flight distance is in the range of tens to hundreds of meters. In higher-altitude or lower-altitude scenarios, the image-registration-based localization method may not be applicable for UAV localization. Furthermore, the limitation of our method lies in its inability to achieve localization in low-texture environments, such as deserts and oceans. In future work, we will explore more robust features to enable UAV localization in complex environments.