GMS-RANSAC: A Fast Algorithm for Removing Mismatches Based on ORB-SLAM2

Abstract

This paper presents a new method of removing mismatches of redundant points based on oriented fast and rotated brief (ORB) in vision simultaneous localization and mapping (SLAM) systems. On the one hand, the grid-based motion statistics (GMS) algorithm reduces the processing time of key frames with more feature points and greatly increases the robustness of the original algorithm in a complex environment. On the other hand, aiming at the situation that the GMS algorithm is prone to false matching when there are few symmetry feature point pairs, the random sample consensus (RANSAC) algorithm is used to optimize and correct it. Experiments show that the method we propose has an average error correction rate of 28.81% for individual GMS while the time consumed at the same accuracy threshold is reduced by 72.18% on average. At the same time, we compared it to locality preserving matching (LPM) and progressive sample consensus (PROSAC), and it performed the best. Finally, we integrated GMS-RANSAC into the ORB-SLAM2 system for monocular initialization, which results in a significant improvement.

1. Introduction

With the proposal and popularization of intelligence in robotics, this also accelerates the development of simultaneous localization and mapping (SLAM), which plays an important role in applications such as service robots, VR, 3D reconstruction and autonomous navigation platforms. Depending on the kinds of sensors, we usually divide SLAM into two main categories. One is based on a laser. The distance and angle of markings in adjacent keyframes is used to estimated position and motion with Rao-Blackwellized Particle Filter in the robot coordinate system [1]. The other category is vision SLAM, which can be carried out by using just a monocular camera. With good robustness and accurate recognition in position, these cameras which provide rich environmental information are also cheap.

MonoSLAM [2] is the first real-time monocular vision SLAM system, in which the Extended Kalman Filter (EKF) [3] is used as the back-end to track the sparse feature points at the front-end. In 2007, Klein et al. proposed Parallel Tracking and Mapping (PTAM) [4] that realized the parallelization of the tracking and drawing process and used nonlinear optimization instead of conventional filters at the back-end. ORB-SLAM [5] was put forward in 2015. It has good versatility for supporting monocular, stereo and RGB-D. In the traditional feature point method of the SLAM process, feature matching is a critical step in visual SLAM [6], which fixes an issue with data association in SLAM. This means the correspondence was determined between a road sign that is currently seen and before it was seen. By accurately matching the description subs between the image and the image or the map [7], we can reduce a lot of burden for subsequent attitude estimation, optimization and other operations. However, due to the local characteristics of image features, mismatches are widespread, and this has not been effectively solved for a long time [8]. At present, it has become a bottleneck in visual SLAM that restricts performance improvement [9].

As Figure 1 shows, researchers usually estimate the Fundamental matrix and Homography matrix [10], and then choose the one which has smaller reprojection error to perform the exercise estimation. We prefer to use random sample consensus (RANSAC) [11] techniques when there may be mismatches. To meet the corresponding accuracy requirements when processing large amounts of data, the number of iterations must be increased, which greatly increases the time it takes to match images [12]. The response time has a great impact on real-time performance [13], and its reduction will sacrifice the accuracy of the SLAM system. Therefore, it is difficult to improve the response time and accuracy at the same time when the amount of data is large.

This paper presents a new method in which grid-based motion statistics (GMS) [14] and RANSAC techniques are introduced to eliminate the incorrect matches. The GMS algorithm reduces the processing time of key frames with more feature points and greatly increases the robustness of the original algorithm in a complex environment. On the other hand, aiming at the situation that the GMS algorithm is prone to false matching when there are few feature points, the proposed algorithm can correct it. By experimental comparison, GMS-RANSAC improved the accuracy of GMS by about 28.81% and saved RANSAC about 72.18% of the time by setting the RANSAC threshold instead of setting the number of iterations.

Finally, we projected the method to create a new initializer based on the ORB-SLAM2 [15] system for monocular initialization, in which the number of the RANSAC model is decreased significantly. Figure 2 presents the initialization process for new design. The number of models is rapidly controlled within a corresponding range by GMS and canceling the number of iterations makes optimization global to achieve a better model, which results in a significant improvement.

The remainder of the paper is organized as follows: Section 2 describes related work. In Section 3, we present our algorithm. The results are evaluated in Section 4. Section 5 concludes the paper. The source code of the demo is at https://github.com/fycie/demo.git, accessed on 10 March 2022.

2. Related Work

2.1. RANSAC

RANSAC is used to randomly find a few points from all observations to fit the model and calculate the residuals of the model and all observation data in turn. All points will be judged as intrinsic points and be counted when they are less than a given threshold. On the contrary, a point could be judged as an outer point. Then, a few points are randomly selected to fit the model iteration again. If the number of fit inner points is larger than the previous model, the old is iterated into the new one.

The traditional RANSAC algorithm needs to set an upper bound of iteration value and error threshold, calculate the transformation matrix $H_1$ through four symmetry pairs of sample data, and record it as a data detection model $M_1$ which is used to measure all of the points and calculate the reprojection error. When the error is less than the set threshold, this point is classified as an inner point and added to the set of inner points. If the current set of inner points is more than the optimal, the current set is updated to the optimal set of inner points. Then, we update the number of iterations $n$. The process ends when the current is greater than the set number of iterations, otherwise the above steps are repeated and iterations are set to $n+1$. It is expressed as:

$$N=\frac{\lg (1-p)}{\lg (1-{\varphi }^m)}$$

(1)

If a maximum overlap is not set in advance generation value, the iterations will continue in this process. The optimal solution obtained at the end is determined by the limitation of the number of iterations, and the upper limit of the number of iterations is strongly consistent with the probability of obtaining the best model, which means the greater the upper limit of the number of iterations, the greater the probability of obtaining the best model. Although the iterations are increased at the same time, it slows down the algorithm. In the ORB-SLAM2 system, RANSAC rudely iterates a smaller number of times to ensure that the response time meets the real-time requirements. The proposed method is not to limit the number of iterations, but to set the threshold to ensure the accuracy of the results. No matter how large the amount of data is, in our method the GMS algorithm can quickly filter data to prepare for the RANSAC algorithm. Figure 3 shows the example of RANSAC algorithm.

2.2. GMS

Combining smoothness constraints into feature matching is known to enable super robust matching. However, it was complex and time-consuming for us to succeed in this process in the past. GMS proposes a simple method to illustrate motion smoothness as a number of matching statistical possibilities in an area, which makes high-quantity matching with displacement have a high match quality. This provides a real-time, robust matching system. Evaluation of a video with low texture, blur and wide baselines (a pair of images that can be thought of as rotating displacements) shows that GMS exceeds other real-time matching methods and achieves the same effect as more complex, slower matching methods.

The key idea of GMS technology is to combine motion smoothing constraints (adjacent pixels of images from different viewpoints share similar motion) [12] into a statistical framework to reject wrong matching. The motion smoothing assumption means that the real matching neighborhood view is the same 3D region. Formally, the number of similar neighbors is used to identify good correspondences. The flaw in the model is that when there are few points within each grid, confidence of neighborhoods is low, which may lead to a large number of error results. By fusing RANSAC, the proposed approach is more robust than GMS when faced with a dataset containing errors.

2.3. LPM

Locality preserving matching (LPM) [16] is to maintain the local neighborhood structures of those potential true matches. It derives a closed-form solution with linearithmic time and linear space complexities in which the problem was formulated into a mathematical model [17]. It utilizes a more relaxed geometric constraint, yet it achieves surprisingly excellent performance and becomes the most popular performer considering the time cost [18]. Compared with LPM, our method obtains better performance in accuracy and runtime.

3. Methodology

3.1. Workflow of the Method

Figure 4 shows the workflow of the proposed method. Specifically, the main workflow is as follows:

(1)

Firstly, the ORB feature points of two images are extracted, respectively;

(2)

Then, the ORB features are matched by Hamming distance of descriptor;

(3)

Thirdly, the results of the previous step are roughly screened by GMS algorithm, which makes the number of matches greatly reduced;

(4)

Finally, the outliers are further removed by setting the random sampling consistency of the threshold.

As shown in Figure 4, first, the same number of ORB feature points was created to match two images $I=\{I_1,I_2\}$. We recorded their horizontal and vertical coordinates as $k{p}_i$, so that $K{P}_i=\{k{p}_1,k{p}_2,\dots ,k{p}_n\}$. Secondly, by calculating the Hamming distance of ORB features, we found the corresponding matching points through $I_2$ in the order in which the feature points were stored in $K{P}_1$ and then recorded their serial numbers in the $K{P}_2$. $m_i$ is the number of $K{P}_2$, which is matched to $k{p}_i$ in $K{P}_1$. If there were no points matched, $m_i=-1$. After that, we would obtain $M_1=\left\{m_1,m_2,\dots ,m_n\right\}$ and then

$$F_{gms}(i,m_i)=\left\{\begin{array}{l}-1,m_i\le 0\\ f_{gms}(i,m_i),m_i>0\end{array}\right.$$

(2)

We used Equation (2) that is GMS to set the value of $m_i$ which did not meet the requirements to −1.

According to the idea of GMS algorithm, the image was divided to be matched into $G\times G$ grids, and we set G to 20 in this paper. $M_2=\left\{m_1,m_2,\dots ,m_n\right\}$ was obtained through GMS. Then, we sorted out the data. When $m_i\ne -1$, we set

$$p_n=f(i,m_i)$$

(3)

As a result, $P_g=\{p_1,p_2,\dots ,p_n\}$, which was prepared for RANSAC. After normalizing the points, we used RANSAC algorithm to calculate the Homography matrix, and set the threshold of the algorithm to 5. According to the same form before, the outer points were eliminated by re-projection error. Finally, we obtained $P_{g+r}=\{p_1,p_2,\dots ,p_n\}$, which are excellent matches.

3.2. GMS Mismatch Correction

Figure 5 presents false matches of GMS in left picture when 2000 features are created, and the matches are corrected in the right image after RANSAC. RANSAC takes only 0.68 ms and improves 42% percent of accuracy at the criterion of the RANSAC threshold of 5.

Then, GMS was compared with GMS-RANSAC at different numbers of ORB feature points. Figure 6 shows that GMS-RANSAC becomes more effective with the number of points increasing, and the great gap from 1000 to 2000 is the weaknesses of the GMS. The trend of optimization also reflects changes in GMS accuracy. We processed the data according to the threshold value of 5. The highest point occurs when the number of ORB feature points is 5000, and the accuracy is 59.76%. Each of feature points were matched for record, and then set as $Q_{\mathrm{i}}=\{X_1,X_2,\dots ,X_n\}$. The maximum and minimum values in ${\overline{Q}}_i$ were removed for analysis.

$${\overline{Q}}_i=\frac{{\displaystyle \sum _{i=1}^n{\overline{X}}_i}}{n}$$

(4)

The average improvement efficiency A is about 54.24%.

$$A=\frac{{\displaystyle \sum _{i=1}^n{\overline{Q}}_i}}{n}$$

(5)

3.3. The Acceleration of RANSAC

GMS for continuity-based neighborhood search matching can quickly reduce a large number of coarse matches. Time of different matching methods was measured by taking same number of ORB feature points from the same two pictures, and numbers of fine matches were recorded.

As Figure 7 shows, RANSAC takes more time with ORB feature points increasing at the same RANSAC threshold. For example, after rough matching, outliers are removed by the RANSAC algorithm. It takes 64.23 ms to reach the standard algorithm with a threshold of 5 based on 1000 ORB features, but the method proposed only takes 1.86 ms in total. GMS consumed 1.18 ms and other times are RANSAC In addition, through the calculation from 1000 to 10,000 feature points at 1000 intervals, the average time consumed was reduced by 94% in parallel to RANSAC. Because GMS rapidly reduces the number of matches, the number of RANSAC reaching the 5 threshold is greatly reduced. This is the reason why the time required by RANSAC algorithm was reduced.

4. Experiments

The experiments were conducted on a 4-threaded, 4-RAM virtual machine ubuntu16.04 installed by VMware Workstation Pro under intel i5-8400 CPU, 16 G RAM, 256 G SSD desktops and C++ code.

4.1. GMS-RANSAC for Correspondence Selection

In order to evaluate GMS-RANSAC performance comprehensively, we experimented with different local features and different number of points. Two frames were selected from a range of datasets for experimental comparison to simulate the operation of scenarios.

4.1.1. Datasets and Metrics

It consists of 2 datasets, including TUM [19] and KITTI [20], which provide indoor scenes and street views. We calculated the ratio by counting the time spent by the algorithm and the number of good matches obtained in Section 3. Figure 8 presents the comparison of RANSAC and GMS-RANSAC in different datasets at 3000 feature points, and a.1, b.1 and c.1 represent the results of RANSAC under TUM and KITTI. The other results are GMS-RANSAC’s. We evaluate the two methods by comparing the number of matches and the time it takes.

4.1.2. Experimental Results

It can be seen from Figure 9a that the matching numbers of RANSAC are more than GMS and GMS-RANSAC. As the number of ORB points increases, all three methods gradually increase, and the gap between RANSAC and the other two methods increases. GMS-RANSAC is slightly higher than GMS around 8000 points of ORB features on KITTI in Figure 9b. Figure 9c shows the results on TUM dataset in which the total number filtered by the method is lower than that of the other two datasets.

Figure 9a–c shows that the number of GMS-RANSAC results is similar to the number of high-quality matches obtained by the original RANSAC method. Using Equations (4) and (5), we summarize the correction rate of the above GMS-RANSAC compared to the GMS. The data show that the number of matches obtained after GMS processing is greater than RANSAC. However, when further filtered with the same threshold RANSAC, the resulting number of fine matches is not much different from the original.

As can be seen in Figure 10a, the cost time of three methods increases with the increasing number of ORB features on TUM desk. All three algorithms reach their maximum values at 10,000 ORB features. It can be seen in Figure 10b that there are several inflection points in RANSAC on KITTI. The extremum of RANSAC is 32.735 ms at 3000, and the time consumed by the three algorithms is relatively close when ORB feature points are 1000–2000 (Figure 10c). The gap becomes increasing gradually with the number of ORB features rising.

Figure 10a–c present that as the ORB feature points increase, the time consumed by RANSAC increases linearly. In contrast, the time required for GMS stabilized at 1–2ms. GMS-RANSAC, where the growth of GMS-RANSAC is stabilizing.

As shown in

Table 1. Data obtained from three sets of experiments.

Datasets	Correction Rate for GMS GMS-RANSAC	Time Reduction for RANSAC GMS-RANSAC
TUM desk [19]	28.28%	67.94%
KITTI [20]	27.23%	64.07%
TUM [19]	30.94%	84.54%
Average value	28.81%	72.18%

, the method we propose has an average error correction rate of 28.81% for individual GMS. On the other hand, there is a reduction of about 72.18% in time for individual RANSAC. Overall, it is a significant improvement in feature matching. The effect of GMS-RANSAC is most pronounced on the TUM dataset, and the maximum of correction rate between GMS and GMS-RANSAC is 30.94%. The time reduction maximum between GMS and GMS-RANSAC is 84.54%. It plays an important role in both ways.

4.1.3. Comparisons

Table 2. Experimental results of three algorithms under 2000 ORB features.

Datasets		PROSAC [21]	LPM [16]	GMS-RANSAC
TUM desk [19]	%Precision	88.05	71.94	88.74
	%Recall	100.00	84.77	93.98
	Cost time (ms)	28.39	25.93	3.23
KITTI [20]	%Precision	91.53	57.55	86.86
	%Recall	100.00	84.77	87.18
	Cost time (ms)	27.89	23.94	2.89
TUM [19]	%Precision	93.29	77.42	89.27
	%Recall	90.41	88.29	90.93
	Cost time (ms)	29.47	24.96	3.17

reports the experimental results for PROSAC [21], LPM and GMS-RASAC. Here, we mainly compare algorithms in terms of %Precision, %Recall and time cost after RANSAC-based outlier removal.

All the above results demonstrate that GMS-RANSAC can show better performance with LPM using the same feature correspondences as input. Although PROSAC is outstanding, it is less stable at runtime with different number of points and consumes more time.

4.2. The New Initializer Based on ORB-SLAM2

To save computing resources, we inserted GMS-RANSAC in $S\mathrm{e}archForInitialization$. According to its original practice,$vmatched12[i]$ at the outer point was set to −1 to mark it. Furthermore, the number of iterations of the original group was replaced by the threshold method, which was integrated into the initialization of ORB-SLAM2. The number of iterations and the parallax angle of the initialization decision were changed to keep system stability. We set the number of iterations of the original system from 200 to 300 and adjusted the parallax angle limit from 0.36 to 0.2 to test the initializer.

4.2.1. Datasets and Metrics

Because the monocular initialization in KITTI datasets was easy, we chose TUM dataset freiburg1_xyz and freiburg1_desk as the datasets for this experiment, and this was beneficial for us to identify the ability of initializers. In each test, we measured (1) how many 3D points were triangulated; (2) how much time the algorithm cost. In addition, we calculated the ratio of the number of points to the initialization time to evaluate the efficiency of the initializers.

4.2.2. Experimental Results

Table 3. Modified ORB-SLAM2 initialization results on the TUM dataset freiburg1_xyz (R:RANSAC G-R:GMS-RANSAC #3D Points: points by triangulation).

ORB Numbers	#3D Points R G-R		Initialization Time (ms) R G-R		Rate (Points/Time) R G-R
1000	109	141	17.4	13.9	6.3	10.1
2000	127	217	30.9	30.7	4.1	6.5
3000	143	326	55.3	50.2	2.6	6.5
4000	107	392	74.5	69.1	1.4	5.7
5000	131	454	85.5	84.8	1.5	5.4
6000	179	970	104.6	133.6	1.7	7.3
7000	672	979	110.9	130.2	6.1	7.5
8000	209	973	122.6	132.8	1.7	7.3
9000	210	973	128.9	139.4	1.6	6.9
10000	210	973	129.1	133.9	1.6	7.3

shows the results, in which RANSAC is in parallel to GMS-RANSAC based on the original initializer. The results show that the proposed initializer can significantly improve the initialization efficiency and make the 3D map denser. This is of great significance for the ORB-SLAM2 system, especially when they meet the complex texture environment. The proposed method can greatly increase the robustness of the system.

Table 4. Modified ORB-SLAM2 initialization results on the TUM dataset freiburg1_desk (R:RANSAC G-R:GMS-RANSAC #3D Points: points by triangulation).

ORB Numbers	#3D Points Numbers R G-R		Initialization Time (ms) R G-R		Rate (Points/Time) R G-R
1000	102	108	21.8	13.6	4.7	8.0
2000	166	171	28.9	27.4	5.7	6.2
3000	125	314	33.9	50.0	3.7	6.3
4000	103	411	32.5	76.6	3.2	5.4
5000	103	451	42.6	97.0	2.4	4.6
6000	103	556	34.9	132.1	3.0	4.2
7000	103	610	34.9	146.9	2.9	4.2
8000	103	665	31.8	158.3	3.2	4.2
9000	103	616	33.3	138.6	3.1	4.4
10000	103	616	32.6	138.7	3.1	4.4

presents that the original initializer tends to be stable in the number of triangulated map points while our proposed method increases steadily. Although the time will increase significantly with the increase of 3D points after 3000 ORB points, the rate is always higher than the original. Figure 11 presents the experimental scenarios under different datasets.

5. Conclusions

This paper proposes a fast correspondence selection algorithm which combines GMS and RANSAC. Consequently, it can make up for the shortcomings of the original two algorithms used alone and make them more robust, especially when the ORB features are too many or too little. Moreover, experiments show that the method we propose has an average error correction rate of 28.81% for individual GMS while the time consumed at the same accuracy threshold is reduced by 72.18% on average. This was also verified in the ORB-SLAM2 system, in which it shows its great potential in real-time applications.

We think that the RANSAC in ORB-SLAM2 is quantitative from the method. As experiments show, it can handle a peak of 3000, which does not fully utilize the acquired image information. This paper proposes a new method of global optimization, which can greatly improve the robustness of the system in a complex environment. We are simply using the Homography matrix for experiments at present, and a new contrast structure will be designed on the Homography matrix and the Fundamental matrix to improve the monocular initialization in the future. In addition, we are considering replacing bag of words (BOW) with GMS-RANSAC to make great progress in real time of the SLAM system. Furthermore, mapping needs to be further developed, and building more reliable and dense maps to support navigation is also the core pursuit of the SLAM system.