A Robust and Lightweight Loop Closure Detection Approach for Challenging Environments

Abstract

Loop closure detection is crucial for simultaneous localization and mapping (SLAM), as it can effectively correct the accumulated errors. Complex scenarios put forward high requirements on the robustness of loop closure detection. Traditional feature-based loop closure detection methods often fail to meet these challenges. To solve this problem, this paper proposes a robust and efficient deep-learning-based loop closure detection approach. We employ MixVPR to extract global descriptors from keyframes and construct a global descriptor database. For local feature extraction, SuperPoint is utilized. Then, the constructed global descriptor database is used to find the loop frame candidates, and LightGlue is subsequently used to match the most similar loop frame and current keyframe with the local features. After matching, the relative pose can be computed. Our approach is first evaluated on several public datasets, and the results prove that our approach is highly robust to complex environments. The proposed approach is further validated on a real-world dataset collected by a drone and achieves accurate performance and shows good robustness in challenging conditions. Additionally, an analysis of time and memory costs is also conducted and proves that our approach can maintain accuracy and have satisfactory real-time performance as well.

1. Introduction

With the research and development of autonomous driving and drone swarm systems, the application scenarios for unmanned ground vehicles (UGVs) and unmanned aerial vehicles (UAVs) have gradually expanded from the laboratory to complex scenarios in the real world. Visual SLAM technology has been widely applied on UAVs/UGVs to complete series of complex tasks in the real world. The complexity of real-world scenarios is mainly reflected in factors such as large spatial ranges, dynamic scene changes, and variations in lighting. Loop closure detection plays a crucial role in reducing cumulative errors during the completion of complex tasks [1]. Visual loop closure detection mainly relies on the associations between visual features, but large viewpoint changes, variations in lighting, occlusions, blurriness, and lack of texture pose particularly challenging factors for 2D-to-2D data association [2]. Therefore, such scenarios impose high demands on the robustness of visual SLAM loop closure detection, which is a significant challenge in SLAM [3]. To address the challenges faced by visual SLAM in complex real-world scenarios, we conduct research on robust loop closure detection.

Most existing loop closure detection methods use traditional visual bag-of-words (BoW) for place recognition, such as VINS-Fusion [4], ORB-SLAM3 [5], and Kimera [6]. The BoW typically requires a training stage wherein visual features are extracted from training images, such as SIFT [7], SURF [8], BRIEF [9], ORB [10], etc., and these features are then clustered to generate a vocabulary, which is a set of quantized visual features known as visual words [11]. However, traditional feature-based BoW methods are invalid in long-term SLAM applications where the scene changes over time. Moreover, the static vocabularies used in pre-trained BoW methods are not well-suited for long-term place recognition because the scenes may differ from the training images [12].

With the rapid development of convolutional neural networks (CNNs), many works have shown robust performance on tasks such as image retrieval and feature matching, especially under extremely challenging conditions [13]. In terms of image retrieval, NetVLAD [14], and DenseVLAD [15] have achieved good results. For image matching, the combination of SuperPoint [16] and SuperGlue [2] has been proven to perform robust feature matching. However, applying deep-learning-based methods to practical SLAM systems still presents challenges. With better and more complex models emerging, ensuring real-time model inference is a crucial problem. If the inference time of a model is too long, it may not be able to meet the real-time requirements of the SLAM system.

To address the problems mentioned above, this paper proposes a robust and lightweight loop closure detection approach for challenging environments that is based on a state-of-the-art deep learning approach. Building upon the success of the combination of SuperPoint and SuperGlue in the SLAM field, in our proposed approach, SuperPoint is used for the feature extraction component to enhance the robustness of the extracted feature points, and a more lightweight and effective method LightGlue [17] is employed in the feature matching part. For place recognition, we utilize MixVPR [18], which is a method based on multi-layer perceptrons (MLPs) that can more stably represent the global features of images. Furthermore, we deploy the models utilized in our approach using the TensorRT framework to guarantee the real-time performance of loop closure detection.

The main contributions of this paper are as follows:

• A novel deep loop closure detection method is proposed to address challenging scenarios; it incorporates the latest methods for both feature matching and place recognition into the visual SLAM framework. Performance evaluations are conducted on multiple datasets and achieve promising results.
• The inference of the models is accelerated using TensorRT. Pre-processing and post-processing by the models are optimized based on the concept of parallel programming, ensuring the real-time performance of the loop closure detection approach.
• Our code is open-source and is available at https://github.com/kajo-kurisu/D_VINS (accessed on 24 April 2024).

The structure of this paper is as follows: In Section 2, we review the existing image matching and loop closure detection methods. In Section 3, we provide a detailed introduction to the proposed approach. Section 4 describes the implementation details of the algorithm. In Section 5, we test the proposed approach on several datasets and compare it with other approaches. Finally, in Section 6, we present the conclusions drawn from our research.

2. Related Works

2.1. Robust Feature Matching

In SLAM, the correspondence between points in images is vital for estimating 3D structures and camera poses. This correspondence is typically estimated through the matching of local features: a process known as data association. The key to this step is to identify the best features that are invariant to orientation, angle, and contrast in images, as this results in higher precision and faster outcomes for data association [19]. In visual SLAM, feature point extraction and matching methods are mainly divided into two categories: traditional methods and deep methods. Traditional methods include algorithms such as SIFT, SURF, and ORB. The SIFT and SURF algorithms cannot meet the real-time requirements of SLAM. The ORB algorithm achieves real-time performance, but its matching accuracy is not as high as SIFT, and it is significantly affected by changes in lighting conditions.

Recent studies on deep learning for image matching have demonstrated the advanced capabilities of deep local features. Representative works include SuperPoint [16], R2D2 [20], and DISK [21]. SuperPoint is a local feature extraction method based on a fully convolutional neural network (FCNN) that is self-supervised. On the HPatches [22] dataset, SuperPoint’s performance in homography estimation significantly outperforms LIFT [23] and ORB, and in some cases, it can even surpass SIFT’s performance when using various thresholds of correctness. R2D2 is a method that jointly learns the description and detection of feature points and is capable of finding reliable and repeatable keypoints. DISK is a method based on reinforcement learning that differs from the previously mentioned approaches by providing a very dense set of feature points.

At the same time, the performance of feature matching algorithms has also been significantly improved. Representative works include SuperGlue [2], which is a method for feature matching based on attention graph neural networks that robustly matches the features extracted from two images. Another work is LoFTR [24], which is a transformer-based [25] method that directly matches two images without feature extraction (detect-free) and outputs matching pairs. Experiments using SuperGlue have shown that when combined with SuperPoint, it achieves leading performance in pose estimation tasks in challenging indoor and outdoor scenes. Some works have integrated these methods into SLAM, such as [26,27], which apply them to visual odometry by using SuperPoint as the front-end feature extraction method and SuperGlue for feature matching to achieve feature tracking. SP-LOOP [28] utilizes this combination for loop closure detection: performing local feature matching between the current frame and loop frame. In [29], the authors not only apply this combination to visual odometry but also to loop closure detection.

Recently, LightGlue [17], which is a matcher that combines attention mechanisms with insight about the matching problem and recent innovations in transformers, has been proposed. It has the ability to stop at earlier layers based on the amount of visual overlap, appearance changes, or discriminative information. Compared to SuperGlue, LightGlue is lighter and more accurate, and it also demonstrates good compatibility with SuperPoint. Experimental results using LightGlue show that the combination of SuperPoint and LightGlue almost reaches the state-of-the-art (SOTA) level in terms of accuracy. Therefore, this combination has great potential for application in SLAM. Inspired by the above methods, our work focuses on designing a combination based on SuperPoint and LightGlue in order to achieve a robust and lightweight feature matching method that is then applied to the field of SLAM.

2.2. Robust Loop Closure Detection

In order to effectively perform SLAM tasks in the real world, the loop closure detection approach must be robust enough to overcome disturbances in the real environment, such as significant changes in viewpoint and illumination, occlusions, blurring, and lack of texture.

With the proposal of the bag of words (BoW), many works have used the BoW for loop closure detection. DBoW [11] utilizes traditional handcrafted features (such as SIFT [7], SURF [8], ORB [10], etc.) to generate a BoW for training; then it constructs a visual vocabulary tree, which is finally used for loop closure detection. Systems such as VINS-Fusion [4], Kimera [6], ORB-SLAM2 [30], and ORB-SLAM3 [5] all utilize DBoW2. SP-LOOP [28] combines the CNN-based descriptor SuperPoint with BoW, using SuperPoint to train the vocabulary tree and apply it to loop closure detection. However, a main issue with BoW methods is the requirement for a training phase. Depending on the number of descriptors and clustering techniques used, this phase can be quite time-consuming. Higher descriptor counts generally result in better performance but require more training time. Additionally, various BoW-based methods do not perform ideally under varying lighting conditions and viewpoints. Moreover, the static vocabulary used in pre-trained BoW methods may not be suitable for long-term place recognition, as the scene can differ from the training images.

There is another method for loop closure detection based on vectors of locally aggregated descriptors (VLADs) [31,32]. With the development of deep learning, many CNN variants based on VLADs have emerged and have been used in various excellent projects. Omni-Swarm [33] uses MobileNetVLAD [34] to extract global descriptors for loop closure detection in drone swarms. PanoNetVLAD [35] shows that the NetVLAD-based method is more effective at loop closure detection in complex scenes than DBoW2. ESA-VLAD [36] integrates a second-order attention module with EfficientNetB0 as the backbone network for global feature extraction and combines it with NetVLAD to form a lighter and more effective network for loop closure detection. Ref. [37] achieves precise hierarchical localization using HF-Net.

However, while methods based on the aggregation of local features can address some issues of viewpoint transformation, they are easily compromised under severe scene changes, such as drastic illumination variations and seasonal changes. Recently, MixVPR [18], which is a new holistic feature aggregation technique based on MLPs, has shown the ability to robustly describe the global features of images. It achieves this through a stack of isotropic blocks referred to as a FeatureMixer, which consists solely of multi-layer perceptrons (MLPs). This method based on MLPs is more effective at focusing on the global features of an image compared to previous methods based on local feature aggregation, which has been demonstrated in MixVPR’s experiments. Moreover, MixVPR is lighter than other approaches, which aligns with the requirements of SLAM systems. Inspired by these approaches, our research aims to create a more robust and lightweight loop closure detection method by combining holistic feature aggregation methods based on MLPs with vector retrieval methods based on k-nearest neighbors (kNNs).

3. Methods

In this section, we first provide an overview of the system, followed by an introduction of each component of the system, including feature extraction, keyframe retrieval, feature matching, and relative pose estimation.

3.1. System Overview

The loop closure detection approach we designed is scalable and can be easily applied under the keyframe-based SLAM framework. The pipeline is shown in Figure 1.

When the VIO system receives a new keyframe, we first calculate its global descriptors, local descriptors, and local features. The global descriptor is obtained through the MixVPR network and is further used to construct a global descriptor database for keyframe retrieval. To ensure the accuracy of keyframe retrieval, we use the inner product between global descriptors to evaluate the similarity between keyframes and sort them, selecting the top k frames with the highest similarity as candidate frames. To accelerate the retrieval process of the global descriptor database, we employ the Faiss library [38], which is an efficient library for similarity search and dense vector clustering.

Local features and local descriptors are image features that are extracted and encoded by the SuperPoint network and are used for feature matching after the completion of keyframe retrieval. We use the LightGlue network to match the local features between the current frame and the final loop frame. Then, using this matching information, we employ the Perspective-n-Point (PnP) algorithm [39] to calculate the relative pose between the current frame and the loop closure candidate frame. Finally, this loop information is used as a constraint in the pose graph optimization (PGO) process to generate more accurate pose estimation results.

3.2. Feature Extraction

The process of feature extraction is shown in Figure 2. Whenever a new keyframe is received, if the image is in color, it is initially converted to grayscale before being processed by the SuperPoint network.

The image first passes through the encoder, the primary function of which is to map the input image $I\in {\mathbb{R}}^{H\times W}$ to an intermediate tensor $\mathcal{B}\in {\mathbb{R}}^{H_c\times W_c\times F}$ with smaller spatial dimensions and larger channel depth through max-pooling layers, where $H_c=H/8$, $W_c=W/8$, and $F>1$. This intermediate tensor is then used by the interest point decoder to compute the feature points, while the descriptor decoder also calculates the local descriptors corresponding to the feature points. Through the inference process of the SuperPoint network, we obtain the keypoints, local descriptors, and corresponding scores.

Additionally, the feature points tracked by the VIO system can also serve as the local features of the image. To integrate these features and ensure consistency with the features output by the original SuperPoint network, we modified the original SuperPoint network by retaining its shared encoder and descriptor decoder components. We input the keyframe images and feature points tracked by VIO into the adjusted network to obtain the SuperPoint descriptors for these tracked feature points. This modification enhances the scalability of our loop closure approach, making it compatible with feature points extracted from various visual front-ends.

3.3. Keyframe Retrival

As previously mentioned, we design our keyframe retrieval module based on the MixVPR network. Specifically, we first use MixVPR to extract global descriptors from the images. For an input image $\mathcal{I}$ with a format of $C\times H\times W$, it first passes through a CNN backbone network. MixVPR extracts the holistic feature map $F\in {\mathbb{R}}^{c\times h\times w}$ from the middle layer of this backbone network, where F can be represented as $F=\mathrm{CNN}\left(\mathcal{I}\right)$. Then, this three-dimensional tensor is divided into a set of 2D feature maps of size $h\times w$:

$$F=\left\{X^i\right\},i=\{1,\dots ,c\}$$

(1)

Here, $X^i$ corresponds to the $i^{th}$ activation map within F, which contains a certain amount of image information. Then, the 2D feature $X^i$ is further reshaped into a 1D representation (flattening), resulting in a flattened feature map $F\in {\mathbb{R}}^{c\times n}$, where $n=h\times w$. Subsequently, these flattened features are input to a set of L MLP blocks (Feature Mixer) with identical structures. It is the fully connected structure that allows the network to have a complete receptive field, enabling it to better focus on the global features of the entire image. After the MLP layers are processed, two mapping operations, depth-wise projection and row-wise projection, are performed to reduce the dimensionality of the obtained global descriptor to $(d,r)$. After one more flattening and normalization operation, we finally obtain a $d\times r$ dimensional global descriptor with good global feature representation capabilities. The network structure of MixVPR is shown in Figure 3.

However, MixVPR supplies only the descriptor itself and does not directly support loop detection functionality. To fill this gap, we designed a real-time global database based on global descriptors. The process of this part is shown in Figure 4.

In this database, each keyframe $k{f}_i$ has a corresponding global descriptor $gdi$, where i represents the index of the keyframe, and n is the total number of keyframes. Therefore, our database is represented as:

$$DB=\left\{g{d}_i\right\},i=\{1,\dots ,n\}$$

(2)

Whenever a new keyframe is obtained, we first extract its global descriptor and then integrate it into the database $DB$. Subsequently, we perform a similarity query for this new global descriptor to detect potential loops. To reduce the impact of recent data on query results, we introduce a window threshold $thres$. Thus, the query set Q is defined as:

$$Q=\left\{gdi\right|i<n-thres\}$$

(3)

This ensures that the query set only contains keyframe descriptors that are far enough from the current frame, thereby improving the accuracy and efficiency of detection.

The retrieval of loop closure keyframes is achieved through the global descriptor database mentioned earlier. The process of this part is shown in Figure 5. Whenever a new keyframe $KF$ arrives, we select a subset Q of vectors from the database for the similarity search, and we sort these vectors by similarity to identify the top k frames with the highest similarity as loop candidate frames; these are denoted as $\{k{f}_1,k{f}_2,\dots k{f}_k\}$.

To select the best loop closure candidate frames, we set two similarity thresholds: $th1$ and $th2$. If the highest similarity among the top k candidate frames is below $th1$, it is concluded that the new keyframe $KF$ does not have a corresponding loop frame. In this case, we only add its global descriptor to the database and do not perform further loop closure detection operations. On the other hand, if the highest similarity among the top k candidate frames exceeds $th1$, we continue searching and further filter out frames with similarities above $th2$ from the candidate frames. The earliest appearing keyframe is then selected as the loop closure candidate, as earlier keyframes often have more-reliable pose estimation. Through this process, we ultimately determine the best loop closure candidate frame for the current keyframe.

3.4. Feature Matching and Relative Pose Estimation

After identifying the best loop closure frame, the next step is to perform local feature matching between the current frame and the loop frame. We design the feature matching and pose computation module of our approach based on LightGlue. Specifically, we first employ LightGlue to process the previously mentioned local features and their descriptors that were extracted by SuperPoint. The procedure for this step is illustrated in Figure 6.

The first half of the LightGlue network is similar to SuperGlue. First, the features and local descriptors of the two images are input into the network. After self-attention and cross-attention processing, correspondence prediction is carried out. This prediction evaluates all pairs of points for matching and similarity. The matching prediction score is used to assess whether a point can be matched with others, while similarity prediction assesses the reliability of the matching relationship between points. The pair of points $(i,j)$ that is predicted to be matchable and has the highest similarity score is considered the correct matching point.

Unlike SuperGlue, LightGlue has an adaptive exit mechanism and a pruning mechanism, which can reduce unnecessary computations in the model and speed up inference time. Due to the enhancement of the visual features in the backbone, if the input image is relatively simple, i.e., it has high field-of-view overlap and minimal appearance changes, then the matching predictions in the early layers tend to be reliable and are essentially the same as those in the later layers. At this point, inference can be stopped to save time. The matching prediction is as follows:

$$c_i=\mathrm{Sigmoid}\left(\mathrm{MLP}\left({\mathbf{x}}_i\right)\right)\in [0,1]$$

(4)

Here, $MLP(·)$ is part of each attention unit, and ${\mathbf{x}}_i^I\in {\mathbb{R}}^d$ represents the state corresponding to feature i in image I. The greater the value of this prediction, the more similar the predictions for feature i are in the earlier and later layers (these features can be either matchable or non-matchable). After each layer is processed, an exit decision is made. When a sufficient number of reliable points are identified, LightGlue exits the inference and presents the current result. Otherwise, it performs point pruning by discarding points with high reliability but marked as non-matchable, and then it continues inference in the next layer.

Once network inference is complete, we utilize the set $M=\left\{\begin{array}{c}{P}_i,P_j\end{array}\right\}$ to represent the matching pairs obtained from LightGlue’s inference. Within this set, we filter out matching pairs with scores above threshold $cl$, but we refrain from setting this threshold too high because we believe that the reprojection error handled by the subsequent RANSAC algorithm [40] is more critical than the precision of the network’s output. When the number of matching pairs exceeds a predetermined threshold N-loop, we use the PnP algorithm to determine the relative transformation T between the two frames. During this computation, the VIO tracking points of the current frame serve as the input 3D points, while the SuperPoint feature points and tracking points of the loop frame serve as the input 2D points. Finally, the relative pose transformation T obtained through the PnP calculation is applied as a loop closure constraint in the PGO to optimize the pose estimation.

4. Implement Details

We replaced the loop closure detection module VINS-Loop in VINS-Fusion with the proposed module and deployed it using C++ and ROS. For the parts of the algorithm related to neural networks, the original SuperPoint, LightGlue, and MixVPR were used in Python (v3.8.18). To facilitate deployment in C++, the C++ interface of Torch [41] is a convenient conversion method. However, in practical engineering scenarios involving GPUs, the inference speed based on the Torch framework is often slow, which can severely affect algorithm performance. Therefore, we chose to use TensorRT as our inference framework. Specifically, we first converted the PyTorch model of the mentioned networks into an ONNX model, simplified the ONNX model, and then converted it to an engine suitable for TensorRT inference. Considering the balance between precision and time, we chose FP16 as the quantization precision. Additionally, we optimized the pre-processing and post-processing of the models based on the concept of parallel programming to make them more efficient. The other parts of the algorithm are executed on the CPU. We will analyze the time and memory costs in Section 5.

5. Experiment

To demonstrate the robustness and accuracy of our approach, we deployed it on the VINS-Fusion framework and conducted experimental validations on both static and dynamic scene datasets. We also analyzed the time cost to prove the real-time performance of our approach. In our experiments, we used the absolute trajectory error (ATE) as the evaluation metric. ATE is used to evaluate the deviation between the estimated pose of the approach and the groundtruth at the same time, which can intuitively show the performance of the algorithm throughout the pose estimation process. However, the poses estimated by VI-SLAM and the groundtruth are usually not in the same coordinate system, so we align the two trajectories first. This alignment process calculates the transformation matrix $S\in SE\left(3\right)$ that best fits the two trajectories by considering the temporal correspondence. Then, the estimated trajectory can be converted to the true trajectory’s coordinate system through this transformation matrix. Since the VI-SLAM evaluated in this paper has absolute scale information, the uncertainty of the scale does not need to be considered. Therefore, the ATE for the ith frame can be expressed as follows:

$${\mathbf{F}}_i:={\mathbf{Q}}_i^{-1}{\mathbf{SP}}_i$$

(5)

where $Q_1,...,Q_n\in SE\left(3\right)$ represent the groundtruth poses, with the subscript indicating the timestamp (frame n). $P_1,...,P_n\in SE\left(3\right)$ represent the poses estimated by the algorithm. When comparing to datasets, the timestamps of the estimated poses and true poses are aligned. To assess the overall performance of the algorithm as much as possible, the root mean squared error (RMSE) [42] is usually used to calculate the ATE:

$$\mathrm{RMSE}\left({\mathbf{F}}_{1:n},\mathrm{\Delta }\right):={\left({\displaystyle \frac{1}{m}}\sum _{i=1}^m{∥\mathrm{trans}\left({\mathbf{F}}_i\right)∥}^2\right)}^{\frac{1}{2}}$$

(6)

In our experiments, we used evo as our evaluation tool. Evo is an open-source evaluation toolkit that provides tools for measuring ATE, relative pose error (RPE), and other metrics, making it easy to use.

5.1. EuRoC

The EuRoC dataset [43] is a visual-inertial dataset collected by drones and is mainly static scenes. The dataset was collected in two kinds of scenarios: narrow room and factory. Our approach is experimentally validated on this dataset and compared with VINS-Loop and SP-LOOP because the front-ends of these works are consistent with our algorithm. The experimental results are presented in

Table 1. Comparison with other approaches in the EuRoC dataset using RMSE (m). The best results are highlighted in bold.

Sequence	VINS-VIO	VINS-Loop	SP-LOOP	Ours
V1_01_easy	0.098	0.048	0.042	0.044
V1_02_medium	0.098	0.055	0.034	0.050
V1_03_difficult	0.103	0.103	0.082	0.057
V2_01_easy	0.127	0.053	0.038	0.059
V2_02_medium	0.123	0.088	0.054	0.056
V2_03_difficult	0.345	0.085	0.10	0.075
MH_01_easy	0.160	0.049	0.070	0.058
MH_02_easy	0.177	0.064	0.044	0.033
MH_03_medium	0.315	0.065	0.068	0.057
MH_04_difficult	0.319	0.108	0.10	0.079
MH_05_difficult	0.177	0.095	0.09	0.081
Average	0.186	0.074	0.066	0.059

.

The experiments demonstrate that our approach is competitive in more challenging sequences. In the six sequences of V1 and V2, which are narrow indoor environments, our approach has a clear advantage in the relatively difficult V1_03 and V2_03 sequences. In V1_01 and V2_02, the error of our approach is only 0.02 higher than the smallest one. However, in the open factory scenarios, our algorithm outperforms both VINS-Loop and SP-LOOP in all sequences except MH_01. Figure 7 shows some matching illustrations from the EuRoC dataset, showing that our approach can stably match even when faced with large viewpoint changes.

5.2. KAIST

The KAIST urban dataset [44] was collected from highways in South Korean cities using a car driving under normal conditions and provides data from various sensors. The data used for experiments in this section consist of 10 Hz stereo images (with a resolution of 1280 × 560) and 100 Hz IMU data. This dataset is highly challenging for loop closure detection due to its complex environment, including high speeds, long routes, and prolonged durations. We conducted real-world tests of the proposed approach on this dataset and compare it with other approaches. The route diagrams for the sequences used in the experiments are illustrated in Figure 8.

The accuracy evaluation of these sequences from the KAIST dataset is presented in

Table 2. Comparison with other approaches in the KAIST urban dataset using RMSE (m). The best results are highlighted in bold.

Sequence	VINS-VIO	VINS-Loop	SP-LOOP	Ours
urban26	14.126	11.663	14.122	5.445
urban27	21.359	17.655	21.204	7.256
urban28	12.024	6.209	11.432	5.217
urban38	8.879	7.833	7.356	8.119
urban39	12.831	5.772	7.310	5.623
Average	13.844	9.826	12.285	6.332

. We conducted multiple experiments on the utilized approaches and took the average values to obtain the experimental result.

It can be seen that our approach also demonstrates good performance on the KAIST urban dataset. In sequence 26, where loop closure scenarios are infrequent, both VINS-Loop and SP-LOOP fail to detect loops, while our approach can still reliably identify the same place and accurately execute the loop closure. The complexity of the scenes for sequences 27–28 has increased, yet our approach still significantly outperforms VINS-Loop and SP-LOOP. Sequences 38–39 are the most challenging in the KAIST visual-inertial dataset, and while our approach has lower precision than VINS-Loop and SP-LOOP on sequence 38, it still demonstrates good performance on sequence 39. In terms of the average error, our proposed approach’s is much lower compared to VINS-Loop and SP-LOOP, proving its robustness in the presence of numerous dynamic objects interfering with the urban scenes and achieving good results. Figure 9 shows some loop closure scenarios from the KAIST dataset using our approach.

5.3. Extremely Challenging Test

To further validate the performance of our approach in extreme scenarios, we conducted multi-session experiments on the 4Seasons dataset [45]. This dataset was collected by a vehicle and contains sequences with extreme changes such as seasonal variations and changes between morning and afternoon within the same scene, posing high demands on the robustness of loop detection algorithms. The three sequences we tested have environmental conditions of spring sunny, summer sunny, and winter snowy, respectively, with significant variations between them, making them highly challenging. The results of our multi-session experiments on the 4Seasons dataset are shown in

Table 3. Comparison with other approaches for the 4Seasons dataset using RMSE (m). The best results are highlighted in bold.

Sequence	VINS-VIO	VINS-Loop	Ours
1_spring _sunny	88.413	49.764	20.735
3_summer _sunny	73.317	34.887	16.968
5_winter_snowy	57.252	16.913	19.681
Average	72.994	33.855	19.128

.

It can be seen that our approach maintains accuracy under extreme stress tests, demonstrating sufficient robustness. Figure 10 shows some examples of loop detection for the extremely challenging test.

5.4. Real-World Experiments

Apart from public datasets, we conducted experiments in real-world outdoor scenarios. The platform we employed was a self-designed drone, as illustrated in Figure 11. The drone was equipped with an Intel Realsense D455 camera (Intel, Santa Clara, CA, USA) and an NVIDIA Jetson Orin Nano (NVIDIA, Santa Clara, CA, USA). The experimental scenarios were divided into simple and difficult sequences. In the simple sequence, the drone flew through the gaps between low-rise buildings with a limited field-of-view depth for a short route distance and a duration of approximately two minutes. In the difficult sequence, the drone navigated along a road on the spacious grounds in front of a library with a greater field-of-view depth, for a longer route distance, and for a duration of about 12 min. The overall trajectory diagrams for the simple and difficult sequences are shown in Figure 12.

In the experiments, we aligned the starting point of the drone’s takeoff with the ending point after landing. We then evaluated the overall performance of the approach based on the starting and ending positions of the trajectory estimated by the loop closure approach. We conducted experiments using VINS-Loop, SP-LOOP, and our approach and compared their performance. The specific experimental data are presented in

Table 4. Comparison with other approaches in the real-world experiment using ATE (m). The best results are highlighted in bold.

Sequence	VINS-VIO	VINS-Loop	SP-LOOP	Ours
outdoor_easy	1.337	1.266	0.122	0.115
outdoor_difficult	41.745	29.807	2.681	0.473

.

In the outdoor simple sequence, due to the scarcity of loop closure scenes, VINS-Loop fails to find any loops. However, both SP-LOOP and our approach are capable of effectively identifying loop closures and obtaining accurate results. In this sequence, the precision of SP-LOOP is comparable to that of our approach, with our approach only being 0.007 m more precise than SP-LOOP. In the outdoor difficult sequence, where the trajectory length and duration significantly increase, VINS-Loop introduces considerable errors, and SP-LOOP is only able to find some accurate loop closures, while our approach handles the challenges of the difficult sequence well and performs accurate loop closure detection.

Figure 13 shows the performance of the three algorithms used in the simple sequence. Figure 13a shows that loop closure cannot be found using VINS-Loop. In Figure 13b, SP-LOOP is almost able to return to the starting point, with some deviation in the vertical direction. In Figure 13c, our approach can also return to the starting point, with some deviation in the forward direction.

Figure 14 shows the performances for the difficult sequence. In Figure 14a, the trajectory obtained using VINS-Loop shows significant drift from the starting point. In Figure 14b, the trajectory obtained using SP-LOOP shows significant improvement compared to VINS-Loop but still exhibits some drift from the starting point. In Figure 14c, our approach, compared to the previous two algorithms, is capable of returning to the starting point more accurately, demonstrating the best performance.

Figure 15 presents some loop closure examples from our approach, showing that it can stably detect accurate loop closures in real-world scenarios, especially in challenging environments.

5.5. Time and Memory Costs

Our platform is divided into two types: a desktop and an embedded system. The desktop is equipped with an RTX 2070 SUPER GPU and an Intel i7-8700 CPU, while the embedded system utilizes the NVIDIA Jetson Orin Nano 8 GB. On the desktop, the average inference time for our SuperPoint network on a 752 × 480 input image is about 6 ms, the average matching time for LightGlue is about 4 ms, and the global descriptor computation for MixVPR takes about 1.5 ms, with GPU memory usage of about 422 MB. On the CPU, to ensure the lightness of the algorithm, we use Faiss as the vector retrieval tool, with an average retrieval time of about 2 ms per global vector retrieval. Overall, the algorithm proposed in this paper can balance high-precision computation with low time consumption. The time cost and GPU memory cost are shown in

Table 5. The runtime (ms) for each part of our approach on the desktop.

Dataset	Global Feature Extraction	Local Feature Extraction	Keyframe Retrieval	Feature Matching
EuRoC	1.5	6.1	0.8	4.1
KAIST	2.3	10.3	1.7	3.8
4Seasons	2	5.4	2.2	4.1

and

Table 6. The time cost (ms) and GPU memory cost (MB) of our approach on the desktop.

Total Cost	Dataset
	EuRoC	KAIST	4Seasons
Time cost	12.5	18.1	13.7
GPU memory cost	422	600	400

.

On the NVIDIA Jetson Orin Nano 8GB platform, our approach can also ensure real-time performance. The primary factor affecting real-time performance on the Orin Nano is feature extraction, as we use SuperPoint to extract 512 feature points in our experiments. Despite this, our approach can still run at a frequency greater than 10 Hz for both the EuRoC and 4Seasons datasets. For the KAIST dataset, due to the higher image resolution, our approach operates at 7 Hz. However, since the image frame rate is only 10 fps, it still meets the real-time requirement. The time cost and GPU memory cost are shown in

Table 7. The runtime (ms) for each part of our approach on the Orin Nano.

Dataset	Global Feature Extraction	Local Feature Extraction	Keyframe Retrival	Feature Matching
EuRoC	8.48	53	0.7	28.3
KAIST	10.2	98.48	1.8	33.4
4Seasons	8.71	57.88	2.3	26.7

and

Table 8. The time cost (ms) and GPU memory cost (MB) of our approach on the Orin Nano.

Total Cost	Dataset
	EuRoC	KAIST	4Seasons
Time cost	90.48	143.88	95.59
GPU memory cost	753	824	734

.

6. Discussion

This paper proposes a robust and lightweight loop closure detection approach suitable for challenging scenes in the real world with high dynamics and large-scale scenes. The approach employs SuperPoint for feature extraction of keyframes, then performs loop keyframe retrieval based on MixVPR, finally uses LightGlue for feature matching, and computes the relative pose using the PnP method. Moreover, to ensure real-time performance, we deploy the networks with TensorRT, enabling them to inference faster on the GPU.

We have conducted experimental validations on multiple datasets and compared our algorithm with others. We first tested on the EuRoC dataset, demonstrating better performance than VINS-Loop and SP-LOOP for general indoor scenes. Subsequently, we tested on the KAIST urban dataset, showing excellent performance for urban scenes. Moreover, we conducted extreme stress testing on the 4Seasons dataset, proving the algorithm’s robustness even under extreme seasonal changes. Experiments in the real world further confirm the robust performance of our approach. Finally, we performed a cost analysis, demonstrating that our approach is very lightweight and can ensure real-time performance on modern CPUs and GPUs.

Actually, two different feasible methods can be applied to further enhance the real-time performance of our approach, which will be included in our future research. In scenarios with high precision requirements, non-destructive acceleration optimization can be carried out. In scenarios with more relaxed precision requirements, the thresholds can be dynamically adjusted in our approach based on the platform’s computational power and precision requirements in order to achieve a balance between performance and accuracy.