Point-Line Visual Stereo SLAM Using EDlines and PL-BoW

Abstract

Visual Simultaneous Localization and Mapping (SLAM) technologies based on point features achieve high positioning accuracy and complete map construction. However, despite their time efficiency and accuracy, such SLAM systems are prone to instability and even failure in poor texture environments. In this paper, line features are integrated with point features to enhance the robustness and reliability of stereo SLAM systems in poor texture environments. Firstly, method Edge Drawing lines (EDlines) is applied to reduce the line feature detection time. Meanwhile, the proposed method improves the reliability of features by eliminating outliers of line features based on the entropy scale and geometric constraints. Furthermore, this paper proposes a novel Bags of Word (BoW) model combining the point and line features to improve the accuracy and robustness of loop detection used in SLAM. The proposed PL-BoW technique achieves this by taking into account the co-occurrence information and spatial proximity of visual words. Experiments using the KITTI and EuRoC datasets demonstrate that the proposed stereo Point and EDlines SLAM (PEL-SLAM) achieves high accuracy consistently, including in challenging environments difficult to sense accurately. The processing time of the proposed method is reduced by 9.9% and 4.5% when compared to the Point and Line SLAM (PL-SLAM) and Point and stereo Point and Line based Visual Odometry (sPLVO) methods, respectively.

1. Introduction

Simultaneous Localization and Mapping (SLAM) was initially proposed by Smith in 1987 [1]. Since then, diverse methods systems that can simultaneously estimate the position of the onboard sensors and construct the surrounding environment map via the captured scene information have been extensively developed. This has been especially important in the field of robot navigation using diverse types of camera systems (e.g., monocular, stereo and panoramic) [2,3,4,5].

In recent years, most of the research has focused on improving the accuracy of point-feature SLAM, and many encouraging breakthroughs have been made [6,7]. Such research includes the development of the ORB-SLAM2 system based on Parallel Tracking and Mapping (PTAM) ideas that improve the place recognition and the loop closure modules [8]. The invariant of ORB features to different viewpoints and illuminations has been employed to improve tracking, mapping, and loop closure. As a result, ORB-SLAM2 has become one of the most popular systems in rich-textured scenes. Despite the success of the ORB-SLAM2 framework, the expanding applications of mobile robots, among many other applications, such as augmented reality and autonomous driving, impose new challenges, which still remain to be resolved, related to low-textured or structured engineered environments [9,10]. Traditional point-based strategies are unstable and even fail in some scenarios where the point features are uneven or not well distributed. Fortunately, Cadena et al. pointed out that a way to solve this problem is to find an alternative feature that is abundant in the environment [3]. The low sensitivity of line features to light changes and motion blurring is also a prevalent topic in SLAM research. Lu et al. proposed an RGB-D SLAM system that combines line features and depth information to solve the problem of indoor illumination changes [11]. Scaramuzza et al. focused on extracting the vertical lines in the wider field image information captured by the omnidirectional camera to improve the accuracy and robustness of the mobile robot visual odometer system [12,13]. Pumarola et al. improved the initial accuracy of the system by synchronously calculating point and line features in the monocular SLAM initialization thread [14]. Ma et al. utilized vanishing points (VP) to constrain line features, which greatly reduced the mismatching of line features [15]. Despite the improvements, the above line segment detection and matching methods are time-consuming, making the line-based SLAM methods difficult to be processed in real-time.

To enhance the effectiveness of line detection methods, Gomez-Ojeda et al. combined a Line Segment Detector (LSD) with a Line Band Descriptor (LBD) algorithm to develop an enhanced stereo version of Point and Line SLAM (PL-SLAM). Such a method has shown higher translation accuracy than that of ORB-SLAM2 in rich-feature scenes, which proved that stereo systems are more accurate and resistant to interference than other monocular systems in line detection [16]. Berenguer et al. proposed a method to estimate the relative attitude by using Holistic descriptors of the omnidirectional image. This method solves the problem that the omnidirectional image can not deal with the height change of mobile robots, but it requires further research to apply in estimating movements with six degrees of freedom [17]. Inspired by the Scale Invariant Feature Transform (SIFT) algorithm, Li et al. proposed the scale-invariant mean-standard deviation LSD (SMLSD) to extract line features faster without sacrificing detection accuracy [18]. However, Zuo et al. proved that the performance of LSD line detection is still unsatisfactory for real-time applications [19]. There is an urgent need to develop line detection methods that can accurately extract line features at a fast rate regardless of the environment geometrical complexity. To realize the real-time SLAM for point and line features, Zhang et al. proposed the line detector based on Canny edges to obtain line features iteratively [20]. In contrast to [20], Vakhitov et al. focused on improving the accuracy and robustness of line feature extraction by training a deep yet lightweight full-convolutional neural network [21]. Gomez-Ojeda et al. [22] and Luo et al. [23] introduced the Fast Line Detector (FLD) algorithm to reduce the detecting time of the LSD method. The fast detection speed and straightforward logic composition of the FLD approach makes it suitable in the field of point-line SLAM. Although effective, it requires prior information about the scene to determine the needed parameters, which limits its usefulness in prior unknown environments, such as those encountered inside collapsed buildings.

Another hindrance that prevents the line feature from being extensively used is the complex outlier culling technique that makes their implementation nontrivial. Line features have a number of characteristics that change with the angle of view and occlusions, so the method of removing inappropriate features from the detected lines has been a focus of constant research. Shao et al. utilized a coplanar constraint of line features to filter mismatched line features, but the method lacks processing real-time data [24]. Ma et al. trained a general classifier to judge the correctness of any hypothetical matching [25]. This method converts the mismatch elimination into a two-class classification problem, but the performance of the algorithm in unknown scenes still requires further experimental verification. Lim et al. proposed a structural model based on epipolar and vanishing points to remove degenerate LSD line features intended to improve the reliability of line features [26].

Although the above-mentioned approaches present many practical solutions to issues existing in line feature detection and outliers elimination, they are still unable to effectively address the SLAM problem of point-line loop detection. Pumarola et al. proposed that only point features were detected in the PL-SLAM loop closure to reduce the computational complexity [14]. Gomez-Ojeda et al. built a line Bag of Words (BoW) model that had a parallel relationship with the point BoW to achieve the independent line loop detection [22]. However, the point-line detection method without considering the geometric proximity between point and line features does not significantly improve the accuracy of loop detection. Ma et al. modeled a BoW containing VPs and their connecting lines to improve the accuracy of the SLAM system in the corridor [15]. For the problem of false loop detection caused by the existence of a similar point map, researchers have pointed out that the accuracy and robustness of loop detection can be improved by building a Bag of Words (BoW) model that considers the spatial proximity and co-occurrence information of visual vocabulary [27,28]. Under this context, an effective Point and EDlines SLAM framework, named PEL-SLAM, is proposed to solve the above-mentioned issues, including the line feature detection, line outliers elimination, and loop detection.

Specifically, a point-line stereo SLAM system based on Edge Drawing lines (EDlines) [29] and the improved BoW model is proposed to realize localization and mapping in human-made environments. The proposed approach takes advantage of line features in low-textured environments and PL-BoW for loop detection. Point and line features of the input stereo image are detected by ORB and EDlines, followed by calculating the uncertainty entropy that describes the accuracy of the detected line features. The PL-BoW is then obtained using a PL pair selection technique following the ORB point and LBD line descriptors. Finally, a loop detection mechanism based on PL-BoW, similarity score, and space consistency detection of the keyframe is employed to determine the best matching keyframe. The three main contributions of this paper are:

(1)

A stereo SLAM system based on the integration of point and line features. Such a method employs an EDlines algorithm to improve the speed of line feature detector in the front-end of the system. In addition, the comprehensive representation and transformation of line features are also derived.

(2)

A method using entropy scale and geometric constraints is proposed to eliminate the outliers of line features. The strategy of removing the mismatched features in the front-end ensures the reliability of the extracted lines without increasing the additional algorithm complexity. The application of this method improves the accuracy of camera pose estimation and map construction.

(3)

A novel Point and Line Bags of Word (PL-BoW) model combining the point and line features is proposed to improve the accuracy and robustness of loop detection. Unlike popular methods of evaluating the BoW of point and line features independently, the proposed PL-BoW model takes into account the constraints of the extracted point and line features. Such a model improves the reliability of the loop detection process under the interference of weak texture and light changes, which typically exist in structured engineered environments.

The remainder part of this paper is as follows. Section 2 provides the geometric expression, detection, and matching methods of line features. The graph optimization for line features and the improved loop detection mechanism are presented in Section 3. Section 4 compares and analyzes the experimental results of the proposed algorithm with existing methods. The conclusions of the paper are presented in Section 5.

2. Representation and Detection of Line Features

The schematic diagram of the PEL-SLAM system proposed here is illustrated in Figure 1. Since the PEL-SLAM system is improved on the basis of ORB-SLAM2, most modules of the system are the same as the ORB-SLAM2 system except for the green part in Figure 1.

Improvements of the line representation, line feature detection, and line feature matching in the front-end of the proposed method are presented in this section, while improvements of Bundle Adjustment (BA) optimization and loop closure in the back-end of the system are described in the following sections.

2.1. Geometric Representation of Lines

A spatial line can be expressed via its two endpoints in the image plane, ${\mathit{s}}_k={\left[\begin{array}{ccc}{u}_s\hfill & v_s\hfill & 1\hfill \end{array}\right]}^{\mathrm{T}}$ and ${\mathit{e}}_k={\left[\begin{array}{ccc}{u}_e\hfill & v_e\hfill & 1\hfill \end{array}\right]}^{\mathrm{T}}$ (see Figure 2). This is an essential operation of the visual odometer needed to transform 3D lines in the world reference to the image plane. The Pl$\ddot{\mathrm{u}}$cker parameter has the characteristic of observability and computational simplicity, and it is used for the transformation and projection of line features. However, the over parameterization of the Pl$\ddot{\mathrm{u}}$cker parameter hinders the performance of the system in back-end optimization. To address this, an orthonormal representation is introduced for optimization. The spatial line $\mathcal{L}$ in the Pl$\ddot{\mathrm{u}}$cker coordinate is expressed by $\mathcal{L}={\left({\mathit{n}}^{\mathrm{T}},{\mathit{d}}^{\mathrm{T}}\right)}^{\mathrm{T}}\in {\mathbb{R}}^6$, where $\mathit{n}\in {\mathbb{R}}^3$ is the normal vector, and $\mathit{d}\in {\mathbb{R}}^3$ represents the line direction vector of the plane ${\pi }_k$. The plane ${\pi }_k$ is composed of the line $\mathcal{L}$ and the $k_{th}$ camera coordinate origin ${\mathit{C}}_k$.

As shown in Figure 2, the Pl$\ddot{\mathrm{u}}$cker line coordinate is usually constructed by triangulation of two different camera frames. The parameters of the ${\pi }_1={\left(\begin{array}{cccc}{\pi }_x\hfill & {\pi }_{\mathrm{y}}\hfill & {\pi }_z\hfill & {\pi }_w\hfill \end{array}\right)}^{\mathrm{T}}$ plane can be determined by ${\mathit{s}}_1$, ${\mathit{e}}_1$, and the camera origin ${\mathit{C}}_1={\left[\begin{array}{ccc}{x}_1\hfill & y_1\hfill & z_1\hfill \end{array}\right]}^{\mathrm{T}}$ in the world reference as follows:

$${\pi }_w=\left[\begin{array}{ccc}{\pi }_x\hfill & {\pi }_y\hfill & {\pi }_z\hfill \end{array}\right]\left[\begin{array}{c}x\hfill \\ y\hfill \\ z\hfill \end{array}\right]$$

(1)

$$\left[\begin{array}{c}{\pi }_x\hfill \\ {\pi }_y\hfill \\ {\pi }_z\hfill \end{array}\right]={\left[{\mathit{s}}_1\right]}_{\times }{\mathit{e}}_1,{\pi }_w={\pi }_x{x}_1+{\pi }_{\mathrm{y}}{y}_1+{\pi }_z{z}_1$$

(2)

where the []${}_x$ is the skew-symmetric matrix of a vector in ${\mathbb{R}}^3$. Other planes (e.g., ${\pi }_k$) can be obtained following the same calculation.

Given planes ${\pi }_1$ and ${\pi }_2$, the coordinate of the line feature in the world reference is determined via the following dual Pl$\ddot{\mathrm{u}}$cker matrix ${\mathit{L}}^{*}$:

$${\mathit{L}}^{*}=\left[\begin{array}{cc}{\left[\mathit{d}\right]}_{\times }\hfill & \mathit{n}\hfill \\ -{\mathit{n}}^{\mathrm{T}}\hfill & 0\hfill \end{array}\right]={\pi }_1{\pi }_2^{\mathrm{T}}-{\pi }_2{\pi }_1^{\mathrm{T}}\in {\mathbb{R}}^4$$

(3)

With the Pl$\ddot{\mathrm{u}}$cker coordinates known, the transformation of the line becomes convenient in 3D Euclidean space. In what follows, we represent a 3D rigid body motion $\mathit{T}=\left[\begin{array}{cc}\mathit{R}& \mathit{t}\\ 0& 1\end{array}\right]\in SE\left(3\right)$ by a rotation matrix $\mathit{R}\in SO\left(3\right)$ and a translation vector $\mathit{t}\in {\mathbb{R}}^3$. Given the transformation matrix ${\mathit{T}}_{cw}$ from the world reference to the camera reference, the corresponding Pl$\ddot{\mathrm{u}}$cker transformation matrix is defined by:

$${\mathcal{T}}_{cw}=\left[\begin{array}{cc}{\mathit{R}}_{cw}& {\left[{\mathit{t}}_{cw}\right]}_{\times }{\mathit{R}}_{cw}\\ \mathbf{0}& {\mathit{R}}_{cw}\end{array}\right]$$

(4)

Then the Pl$\ddot{\mathrm{u}}$cker representation of the given line feature can be transformed by:

$${\mathcal{L}}^c=\left[\begin{array}{c}{\mathit{n}}^c\hfill \\ {\mathit{d}}^c\hfill \end{array}\right]={\mathcal{T}}_{cw}{\mathcal{L}}^w$$

(5)

After calculating the representation of the line feature in the camera reference, the spatial line can be projected to the image plane through:

$$\mathit{I}=\left[\begin{array}{c}{l}_1\hfill \\ l_2\hfill \\ l_3\hfill \end{array}\right]=\mathcal{K}{\mathit{n}}^c=\left[\begin{array}{ccc}{f}_y& 0& 0\\ 0& f_x& 0\\ -f_y{c}_x& -f_x{c}_y& f_x{f}_y\end{array}\right]{\mathit{n}}^c$$

(6)

where $\mathcal{K}$ denotes the projection matrix of a given line, and $f_x,f_y,c_x,c_y$ represent the intrinsic parameters of the calibrated camera. It should be noted that when a line feature is projected onto a normalized image plane, $\mathcal{K}$ is an identity matrix.

Since a spatial line in 3D space only has four degrees of freedom, using the Pl$\ddot{\mathrm{u}}$cker representation increases the computational complexity of the optimization process. The de-coupled orthonormal representation $(\mathit{U},\mathit{W})\in SO\left(3\right)\times SO\left(2\right)$ employed in the unconstrained optimization problem is obtained from the $QR$ decomposition of the Pl$\ddot{\mathrm{u}}$cker coordinates.

$$\left[\begin{array}{cc}\mathit{n}\hfill & \mathit{d}\hfill \end{array}\right]=\left[\begin{array}{ccc}\frac{\mathit{n}}{\parallel \mathit{n}\parallel }& \frac{\mathit{d}}{\parallel \mathit{d}\parallel }& \frac{\mathit{n}\times \mathit{d}}{\parallel \mathit{n}\times \mathit{d}\parallel }\end{array}\right]\left[\begin{array}{cc}\parallel \mathit{n}\parallel & 0\\ 0& \parallel \mathit{d}\parallel \\ 0& 0\end{array}\right]=\mathit{U}\left[\begin{array}{cc}{w}_1& 0\\ 0& w_2\\ 0& 0\end{array}\right]$$

(7)

Furthermore, $\mathit{U}$ and $\mathit{W}$ are expressed by:

$$\mathit{U}=R\left(\phi \right)=\left[\begin{array}{ccc}\mathit{n}\hfill & \mathit{d}\hfill & \mathit{n}\times \mathit{d}\hfill \\ \parallel \mathit{n}\parallel \hfill & \parallel \mathit{d}\parallel \hfill & \parallel \mathit{n}\times \mathit{d}\parallel \hfill \end{array}\right]$$

(8)

$$\mathit{W}=W\left(\varphi \right)=\frac{1}{\sqrt{{\parallel \mathit{n}\parallel }^2+{\parallel \mathit{d}\parallel }^2}}\left(\begin{array}{cc}\parallel \mathit{n}\parallel \hfill & -\parallel \mathit{d}\parallel \hfill \\ \parallel \mathit{d}\parallel \hfill & \parallel \mathit{n}\parallel \hfill \end{array}\right)$$

(9)

where $\phi ={\left[\begin{array}{ccc}{\phi }_1\hfill & {\phi }_2\hfill & {\phi }_3\hfill \end{array}\right]}^{\mathrm{T}}$ denotes the angle of rotation from the camera reference to the line reference (see Figure 2). Since $\mathit{W}$ implies one-dimensional scale information, the minimum parameterization of a spatial line can be defined by a four-dimensional vector $\mathcal{O}={\left[\begin{array}{cc}{\phi }^T\hfill & \varphi \hfill \end{array}\right]}^T\in {\mathbb{R}}^4$. Once the orthogonal representation of the optimized feature is obtained, the corresponding Pl$\ddot{\mathrm{u}}$cker coordinates are calculated by:

$${\mathcal{L}}^{\prime }={\left[\begin{array}{cc}{w}_1{\mathit{u}}_1^T\hfill & w_2{\mathit{u}}_2^T\hfill \end{array}\right]}^T=\frac{1}{\sqrt{{\parallel \mathit{n}\parallel }^2+{\parallel \mathit{d}\parallel }^2}}\mathcal{L}$$

(10)

in which ${\mathit{u}}_i$ denotes the $i_{th}$ column of $\mathit{U}$.

2.2. Extraction and Description of Line Features

The line segment detector LSD, which gives an accurate line detection and requires no parameter tuning (widely applied in state-of-the-art PL-SLAM systems), is designed to extract line segments on noisy images with sub-pixel detection accuracy [30]. However, the computational complexity of LSD causes extra time spending on the system in the front-end feature extraction. Since the limitation of LSD directly affects the tracking and matching of line features, this aspect leads to the failure of the visual odometer.

To improve the real-time performance of the line feature extraction, the EDlines algorithm is used instead of the LSD to detect line features from images captured by the stereo camera. The EDlines algorithm has been proven to run faster than the LSD, while its output tends to contain irrelevant lines [15]. The experiment in [31] shows that EDlines can achieve a similar performance when compared with LSD without sacrificing the detection accuracy by using an appropriate line outlier elimination method.

In this work, the individual performance of the LSD and EDlines mechanisms was validated on three different scenes. Referring to Figure 3, the median three figures show the line features detected by LSD, whereas the three figures on the right side of the figure show the detected results of EDlines. The results show that both detecting algorithms are able to detect the line features of interest that correspond to real straight lines in the given environments. However, it is noted that EDlines is more likely to detect curves that are near straight lines. The detected curves may degrade the performance of the system. However, these curves usually possess bad spatial positions and thus can be easily recognized from different camera views. Thus, from the testing performed, it was identified that it is possible to use EDlines to achieve effective line detection without corrupting the systems by leveraging some line renting strategies.

To make the line features detected by EDlines more recognizable, the LBD descriptors [32] are computed to represent each line feature. Similar to the ORB descriptor for point feature, the LBD descriptor contains geometric attributes and appearance descriptions of the corresponding line features. For two consecutive stereo frames, the similarity of line features is measured by calculating the consistency of LBD descriptors between line pairs.

2.3. Extraction and Description of Line Features

After exacting line features, LBD descriptors are generated to describe the corresponding line features in two consecutive captured adjacent frames. The similarity of descriptors is calculated to find the matching line features. The extracted line feature is considered to be a good match only if such feature is the best match in both images of a stereo frame. Before eliminating outliers, the preprocessing of matched pairs is required to improve the accuracy of the system matching. Given the occlusion and perspective changes that might exist in real-world environments, in this work, the line pair is not considered a match if their lengths are more than twice as different. At the same time, the line pair is considered to be mismatched if the distance between the midpoints of the two lines on the image is greater than a given threshold.

For a stereo frame, a spatial line is represented as ${\mathit{l}}_l$ and ${\mathit{l}}_r$ in the left and right images, respectively, where ${\mathit{l}}_r$ can be represented as ${\left(\begin{array}{ccc}{l}_{r1}\hfill & l_{r2}\hfill & l_{r3}\hfill \end{array}\right)}^{\mathrm{T}}$ by the homogeneous coordinates of its endpoints. Given a stereo frame captured by a stereo camera, the matching points in the two rectified images have the same horizontal position. Therefore, the endpoints in the right image can be obtained through ${\mathit{l}}_r$ and line ${\mathit{l}}_{v=v^{\prime }}$, as illustrated in Figure 4.

The endpoints of the line ${\mathit{l}}_r$ can be denoted by the endpoints $\mathit{s}={\left(\begin{array}{cc}{u}_s\hfill & v_s\hfill \end{array}\right)}^{\mathrm{T}}$ and $\mathit{e}={\left(\begin{array}{cc}{u}_e\hfill & v_e\hfill \end{array}\right)}^{\mathrm{T}}$ of the left line.

$$u_s^{\prime }=-\frac{l_{r2}{v}_s+l_{r3}}{l_{r1}},u_e^{\prime }=-\frac{l_{r2}{v}_e+l_{r3}}{l_{r1}}$$

(11)

The depth parameters of $\mathit{s}$ and $\mathit{e}$ are calculated by the disparities as follows:

$$d_s=\frac{b{f}_x}{\Delta u_s^{\prime }},d_e=\frac{b{f}_x}{\Delta u_e^{\prime }}$$

(12)

where $\Delta u_s^{\prime }=u_s-u_s^{\prime }$, $\Delta u_e^{\prime }=u_e-u_e^{\prime }$, and b denotes the baseline of the stereo camera. Then the 3D position of ${\mathit{P}}_s$ is obtained by the back-projection of the endpoint $\mathit{s}$ as follows:

$${\mathit{P}}_s=\left[\begin{array}{c}{X}_s\\ Y_s\\ Z_s\end{array}\right]=\left[\begin{array}{c}\left(u_s-c_x\right)d_s/f_x\\ \left(v_s-c_x\right)d_s/f_y\\ d_s\end{array}\right]$$

(13)

Since line features are more sensitive to image noise and mismatching than point features, the uncertainty of line features is modeled to quantify the reliability of line features. Considering the spatial properties of line features, the covariance propagation method is used to construct the uncertainty matrix of the endpoint, which can be expressed as follows:

$${\Sigma }_{P_s}={\mathit{J}}_{P_s}cov\left(\mathit{s}\right){\mathit{J}}_{P_s}^{\mathrm{T}}$$

(14)

in which ${\mathit{J}}_{P_s}$ represents the Jacobian matrix of ${\mathit{P}}_s$, and $cov\left(s\right)$ is the uncertainty matrix of $\mathit{s}$. Based on the properties of ${\mathit{p}}_s$ in the image plane, $cov\left(s\right)$ is modeled as a bi-dimensional Gaussian set with standard deviation ${\sigma }_u={\sigma }_v=1$ pixel. The matrix ${\mathit{J}}_{P_s}$ can be derived from Equation (13) as follows:

$${\mathit{J}}_{P_s}=\frac{\partial {\mathit{P}}_s}{\partial \mathit{s}}=\left[\begin{array}{cc}{d}_s/f_x& 0\\ 0& d_s/f_y\\ 0& 0\end{array}\right]$$

(15)

Due to the fact that the scale of uncertainty between image pairs varies, the values are not directly comparable. The entropy of multivariate normal distribution is thus introduced to abstract the uncertainty in the covariance matrix into a scalar value, which is defined as:

$$H_s=0.5m(ln2\pi +1)+0.5ln\left(\left|{\Sigma }_{{\mathit{P}}_s}\right|\right)$$

(16)

where m is the dimension of ${\mathit{P}}_s$. The uncertainty entropy of ${\mathit{P}}_e$ can be calculated in the same way.

In the methods of [20,24], the processing of outliers is to remove matching pairs that do not meet a set constant threshold. However, we found that a constant threshold is not applicable because the uncertainty depends on the changes of motion and scene. A threshold determination method based on the uncertainty entropy of all line features of the current frame is thus applied to the outliers eliminating process. After calculating the average entropy ${\overline{H}}_s$ for all line features of the current frame, the threshold $T{d}_c$ is set to 0.85 ${\overline{H}}_s$ according to experiments. If the uncertainty entropy $H_s$ and $H_e$ of a spatial line reconstructed from the stereo frame are greater than $T{d}_c$, the line can be regarded as an accurate line feature, and vice versa.

3. Bundle Adjustment and Loop Closure with Points and Lines

The BA module in the back-end optimization consists of two main aspects: local BA of local mapping thread and global BA of loop closure thread. The original BA module of ORB-SLAM2 includes variables of camera poses and point landmarks. However, the addition of line features complicates the optimization process as a minimization process of the cost function in the co-visibility graph needs to be performed. Therefore, a graph BA optimization strategy considering line and point features is adopted in this paper. In addition, the accuracy of the loop closure depends on the calculation of the global BA and the loop detection. A novel point and line BoW is thus proposed to improve the stability and accuracy of the loop detection.

3.1. Graph Optimization with Point and Line Features

Due to the fact that the length and angle of the same line in the two images are different, projection errors cannot be obtained directly from the corresponding two frames. In this work, the projection errors are computed by re-projecting the matched lines from the world reference back to the current image reference. Given the spatial line $\mathcal{L}$ in the world reference and its orthogonal expression $\mathcal{O}$, the corresponding line feature is transformed to the camera frame of reference by $T_{cw}$. Then the projected line ${\mathit{I}}^{c_i}$ is obtained by projecting the line features onto the normalized plane of the current frame. The reprojection error is then defined as the distance between the projected line ${\mathit{I}}^{c_i}$ and the endpoints of the detected line ${\mathit{l}}^{c_i}$, which is computed via Equation (17).

$${\mathit{r}}_l=\left[\begin{array}{c}d\left({\mathit{l}}_s^{c_i},{\mathit{I}}^{c_i}\right)\\ d\left({\mathit{l}}_e^{c_i},{\mathit{I}}^{c_i}\right)\end{array}\right],d(\mathit{s},\mathit{I})=\frac{{\mathit{s}}^{\mathrm{T}}\mathit{I}}{\sqrt{l_1^2+l_2^2}}$$

(17)

where ${\mathit{l}}_s^{c_i}$ and ${\mathit{l}}_e^{c_i}$ are the endpoints of the detected line ${\mathit{l}}^{c_i}$. With the reprojection error ${\mathit{r}}_l$ defined, a global loss function C containing point and line features is formulated as:

$$C=F_{ep}+F_{el}=\sum _{i,k}{\rho }_l\left({\mathit{r}}_l^{\mathrm{T}}(i,k){\Sigma }_l^{-1}{\mathit{r}}_l(i,k)\right)+\sum _{i,j}{\rho }_p\left({\mathit{r}}_p^{\mathrm{T}}(i,j){\Sigma }_p^{-1}{\mathit{r}}_p(i,j)\right)$$

(18)

where $\rho$ denotes the robust Huber cost functions, and ${\Sigma }_l^{-1}$, ${\Sigma }_p^{-1}$ denote the inverse information matrices of the reprojection error of points and lines, respectively. Compared to the original loss function of the ORB-SLAM2 system, the reprojection error function $F_{el}$ about the line feature is added to Equation (18). The camera pose and landmark are calculated by minimizing the loss function C. The Jacobians of ${\mathit{r}}_p$ are already derived in ORB-SLAM2, and the Jacobians ${\mathit{J}}_l$ concerning camera pose and line landmark can be expressed by the following matrix chain multiplication:

$${\mathit{J}}_l=\frac{\partial {\mathit{r}}_l}{\partial {\mathit{I}}^{c_i}}\frac{\partial {\mathit{I}}^{c_i}}{\partial {\mathrm{L}}^{c_i}}\left[\frac{\partial {\mathrm{L}}^{c_i}}{\partial \delta {\mathit{x}}^i}\frac{\partial {\mathrm{L}}^{c_i}}{\partial {\mathrm{L}}^w}\frac{\partial {\mathrm{L}}^w}{\partial \delta \mathcal{O}}\right]$$

(19)

The Jacobian matrix of the line reprojection error relative to ${\mathit{I}}^{c_i}$ is expressed as:

$$\begin{array}{cc}\hfill & \frac{\partial {\mathit{r}}_l}{\partial {\mathit{I}}^{c_1}}=\left[\begin{array}{ccc}\frac{\partial r_1}{\partial l_1}& \frac{\partial r_1}{\partial l_2}& \frac{\partial r_1}{\partial l_3}\\ \frac{\partial r_2}{\partial l_1}& \frac{\partial r_2}{\partial l_2}& \frac{\partial r_2}{\partial l_3}\end{array}\right]=\hfill \\ \hfill & {\left[\begin{array}{ccc}\frac{-l_1{\mathit{s}}_l^{\mathrm{T}}\mathit{I}}{{\left(l_1^2+l_2^2\right)}^{\frac{3}{2}}}+\frac{u_s}{{\left(l_1^2+l_2^2\right)}^{\frac{1}{2}}}\hfill & \frac{-l_2{\mathit{s}}_l^{\mathrm{T}}\mathit{I}}{{\left(l_1^2+l_2^2\right)}^{\frac{3}{2}}}+\frac{v_s}{{\left(l_1^2+l_2^2\right)}^{\frac{1}{2}}}\hfill & \frac{1}{{\left(l_1^2+l_2^2\right)}^{\frac{1}{2}}}\hfill \\ \frac{-l_1{\mathit{e}}_l^{\mathrm{T}}\mathit{I}}{{\left(l_1^2+l_2^2\right)}^{\frac{3}{2}}}+\frac{u_e}{{\left(l_1^2+l_2^2\right)}^{\frac{1}{2}}}\hfill & \frac{-l_2{\mathit{e}}_l^{\mathrm{T}}\mathit{I}}{{\left(l_1^2+l_2^2\right)}^{\frac{3}{2}}}+\frac{v_e}{{\left(l_1^2+l_2^2\right)}^{\frac{1}{2}}}\hfill & \frac{1}{{\left(l_1^2+l_2^2\right)}^{\frac{1}{2}}}\hfill \end{array}\right]}_{2\times 3}\hfill \end{array}$$

(20)

According to the 3D line projection, Equation (21) is obtained.

$$\frac{\partial {\mathit{I}}^{c_i}}{\partial {\mathcal{L}}^{c_i}}=\left[\begin{array}{cc}\frac{\partial \mathit{I}}{\partial \mathit{n}}\hfill & \frac{\partial \mathit{I}}{\partial \mathit{d}}\hfill \end{array}\right]={\left[\begin{array}{cc}\mathcal{K}\hfill & 0\hfill \end{array}\right]}_{3\times 6}$$

(21)

The term $\frac{\partial {\mathcal{L}}^{c_i}}{\partial \delta {\mathit{x}}^i}$ represents the Jacobian matrix of line features concerning translation and rotation errors in camera reference in which $\delta {\mathit{x}}^i=\left[\begin{array}{cc}\delta \mathit{p}\hfill & \delta \theta \hfill \end{array}\right]$. The line features only optimize the translation $\mathit{p}$ and the rotation $\theta$ in the state variable, and the Jacobian matrix is calculated as follows:

$$\begin{array}{cc}\hfill & \frac{\partial {\mathcal{L}}^{c_i}}{\partial \delta {\theta }_{b{b}^{\prime }}}={\mathcal{T}}_{bc}^{-1}\left[\begin{array}{c}\partial \left(\mathit{I}-{\left[\delta {\theta }_{b{b}^{\prime }}\right]}_{\times }\right){\mathit{R}}_{wb}^T\left({\mathit{n}}^w+{\left[{\mathit{d}}^w\right]}_{\times }{\mathit{p}}_{wb}\right)\partial \delta {\theta }_{b{b}^{\prime }}\\ \frac{\partial \left(\mathit{I}-{\left[\delta {\theta }_{b{b}^{\prime }}\right]}_{\times }\right){\mathit{R}}_{wb}^T{\mathit{d}}^w}{\partial \delta {\theta }_{b{b}^{\prime }}}\end{array}\right]\hfill \\ \hfill & ={\mathcal{T}}_{bc}^{-1}{\left[\begin{array}{c}{\left[{\mathit{R}}_{wb}^T\left({\mathit{n}}^w+{\left[{\mathit{d}}^w\right]}_{\times }{\mathit{p}}_{wb}\right)\right]}_{\times }\\ {\left[{\mathit{R}}_{wb}^T{\mathit{d}}^w\right]}_{\times }\end{array}\right]}_{6\times 3}\hfill \end{array}$$

(22)

$$\frac{\partial {\mathcal{L}}^{c_i}}{\partial \delta {\mathit{p}}_{b{b}^{\prime }}}={\mathcal{T}}_{bc}^{-1}\left[\begin{array}{c}\frac{\partial {\mathit{R}}_{wb}^T\left({\mathit{n}}^w+{\left[{\mathit{d}}^w\right]}_{\mathrm{x}}\left({\mathit{p}}_{wb}+\delta {\mathit{p}}_{b{b}^{\prime }}\right)\right)}{\partial \delta {\mathit{p}}_{b{b}^{\prime }}}\\ \frac{\partial {\mathit{R}}_{wb}^T{\mathit{d}}^w}{\partial \delta {\mathit{p}}_{b{b}^{\prime }}}\end{array}\right]={\mathcal{T}}_{bc}^{-1}{\left[\begin{array}{c}{\mathit{R}}_{wb}^T{\left[{\mathit{d}}^w\right]}_{\mathrm{x}}\\ \mathbf{0}\end{array}\right]}_{6\times 3}$$

(23)

The Jacobian matrix of the line in the camera reference concerning the line in the world reference is the inverse of the transformation matrix represented as Equation (24).

$$\frac{\partial {\mathcal{L}}^{c_i}}{\partial {\mathcal{L}}^w}={\mathcal{T}}_{wc}^{-1}$$

(24)

According to the orthogonal representation of a spatial line, the last term of Equation (19) can be defined as:

$$\begin{array}{cc}\hfill & \frac{\partial {\mathcal{L}}^w}{\partial \delta \mathcal{O}}=\left[\begin{array}{cccc}\frac{\partial {\mathcal{L}}^w}{\partial {\phi }_1}\hfill & \frac{\partial {\mathcal{L}}^w}{\partial {\phi }_2}\hfill & \frac{\partial {\mathcal{L}}^v}{\partial {\phi }_3}\hfill & \frac{\partial {\mathcal{L}}^w}{\partial \varphi }\hfill \end{array}\right]\hfill \\ \hfill & =\left[\begin{array}{cccc}\frac{\partial {\mathcal{L}}^v}{\partial \mathit{U}}\frac{\partial \mathit{U}}{\partial {\phi }_1}\hfill & \frac{\partial {\mathcal{L}}^v}{\partial \mathit{U}}\frac{\partial \mathit{U}}{\partial {\phi }_2}\hfill & \frac{\partial {\mathcal{L}}^w}{\partial \mathit{U}}\frac{\partial \mathit{U}}{\partial {\phi }_3}\hfill & \frac{\partial {\mathcal{L}}^v}{\partial \mathit{W}}\frac{\partial \mathit{W}}{\partial \varphi }\hfill \end{array}\right]\hfill \\ \hfill & ={\left[\begin{array}{cccc}0& -w_1{u}_3& w_1{u}_2& -w_2{u}_1\\ w_2{u}_3& 0& -w_2{u}_1& w_1{u}_2\end{array}\right]}_{6\times 4}\hfill \end{array}$$

(25)

With the analysis of the Jacobians completed, iterative algorithms, such as Gaussian–Newton, can be employed to solve the local graph optimization of local mapping and global graph optimization of loop closure.

3.2. Loop Closure with Points and Lines

In human-made environments, the existence of weak textures (i.e., white wall) and frequent light changes leads to the false detection of the traditional BoW-based loop closure. The insufficient recognition of point features leads to the mismatch results between frames. To address this aspect, PL-BoW, a BoW that combines point and line features, is proposed, which utilizes the co-occurrence information and spatial proximity of visual words.

The visual words of point features are generated from the ORB descriptor, including the position ${\mathit{p}}_{point}$ and direction ${\theta }_p$ of point features. At the same time, the direction ${\theta }_l$ and position information of the corresponding line feature is obtained from its endpoints. As shown in Figure 5, the combination of a point and a line is defined as a PL pair only when the direction and distance of the line and the point are close enough. To improve the search speed of PL pairs, a K-D tree is constructed by using the position of each point feature. Then the line features satisfying the PL pair are selected to build the point and line K-D tree.

Given a current keyframe $f_u$ and a candidate keyframe $f_c$, the corresponding BoW vectors are defined as $\left[{\mathit{v}}_{p1},\dots ,{\mathit{v}}_{ps},{\mathit{v}}_{l1},\dots ,{\mathit{v}}_{lt}\right]$ and $\left[{\mathit{w}}_{p1},\dots ,{\mathit{w}}_{p{s}^{\prime }},{\mathit{w}}_{l1},\dots ,{\mathit{w}}_{l{t}^{\prime }}\right]$, respectively. In loop closure detection, the similarity of two keyframes is calculated through the BoW vectors as follows:

$$\begin{array}{cc}\hfill & S=-\frac{N_p}{2\left(N_p+N_l\right)}\sum _{i,j}\left(\left|{\mathit{v}}_{pi}-{\mathit{w}}_{pj}\right|-\left|{\mathit{v}}_{pi}\right|-\left|{\mathit{w}}_{pj}\right|\right)\hfill \\ \hfill & -\frac{N_l}{2\left(N_p+N_l\right)}\sum _{i,j}\left(\left|{\mathit{v}}_{li}-{\mathit{w}}_{lj}\right|-\left|{\mathit{v}}_{li}\right|-\left|{\mathit{w}}_{lj}\right|\right)\hfill \end{array}$$

(26)

where $N_p$ and $N_l$ represent the total number of point and line features in the image, respectively. The mismatched keyframes that have the lower score are removed from the set of candidate keyframes. The procedure of the improved PL-BoW based loop detection is given in Algorithm 1.

Algorithm 1: PL-BoW Based Loop Detection.
Input:	The keyframes set $F=\left\{f_1\dots f_i\right\}$, the KD-tree associated with $f_i$ and the current keyframe $f_u$;
Output:	A revisit matching keyframe $f_{bm}$;
1	Select candidate keyframes through retrieving the words of points and lines in the
	image using Term Frequency-Inverse
	Document Frequency (TF-IDF);
2	$n_{Wi}=$ NumberOfCommonView Words $\left(f_i\right)$;
3	$n_{PLi}=$ NumberOfCommonViewPLpairs $\left(f_i\right)$;
4	for each $f_i\in F_c$ do
5	if $n_{Wi}<{Max}_{f_i\in F_c}\left\{n_{Wi}\right\}\&\&n_{PLi}<{Max}_{f_i\in F_c}\left\{n_{PLi}\right\}$ $\mathit{then}$
6	$f_i\cup F_{cm}\to F_{cm}$;
7	Calculate the similarity $S_i$;
8	$S_{max}=Max\left\{S_i\right\}$;
9	end
10	end
11	for each $f_i\in F_c$ do
12	Remove $f_i$ with $S_i<0.8S_{max}$;
13	end
14	Perform space consistency detection on $F_{cm}$ to obtain $f_{bm}$.

4. Experimental Verification

To test and analyze the proposed PEL-SLAM, a set of experimental tests were performed with an Intel CPU i5-10060 [email protected], 32GB RAM, without a dedicated GPU. OpenCV and g2o were mainly used as the environment running on an Ubuntu16.04 desktop. The proposed PL visual stereo SLAM algorithm was compared with popular methods, including ORB-SLAM2 [8], PL-SLAM [16], and stereo Point and Line based Visual Odometry (sPLVO) [23]. The above algorithms were tested on the KITTI stereo dataset [34] and EuRoC micro aerial vehicle (MAV) dataset [33], which provide several challenging sequences of images in indoor and outdoor environments. The absolute Root Mean Square Errors (RMSEs) between the estimated translation and rotation of the SLAM system and the groundtruth given in the dataset were computed by the EVO [35] evaluation tool.

4.1. Stereo SLAM on KITTI Dataset

The test results in terms of the RMES of ORB-SLAM2, PL-SLAM, and the proposed algorithm on the KITTI dataset are shown in

Table 1. Absolute RMSE errors of three methods on KITTI.

Seg.	ORB-SLAM2		PL-SLAM		PEL-SLAM
	Trans. (m)	Rot. (deg)	Trans. (m)	Rot. (deg)	Trans. (m)	Rot. (deg)
0	$1.303313$	$0.018447$	$3.323087$	$0.068646$	$\mathbf{1}.\mathbf{187783}$	$\mathbf{0}.\mathbf{017726}$
1	$\mathbf{9}.\mathbf{231962}$	$0.025892$	$10.009194$	$0.033958$	$9.509255$	$\mathbf{0}.\mathbf{024612}$
2	$5.177579$	$0.027988$	$7.724952$	$0.065647$	$\mathbf{4}.\mathbf{427225}$	$\mathbf{0}.\mathbf{026923}$
3	$\mathbf{0}.\mathbf{760564}$	$0.014225$	$0.895579$	$\mathbf{0}.\mathbf{011213}$	$0.083256$	$0.013563$
4	$0.203649$	$0.457889$	$0.260442$	$0.431715$	$\mathbf{0}.\mathbf{153068}$	$\mathbf{0}.\mathbf{322527}$
5	$\mathbf{0}.\mathbf{748097}$	$\mathbf{0}.\mathbf{008962}$	$1.983561$	$0.022194$	$0.876509$	$0.009753$
6	$0.736519$	$0.011604$	$0.891829$	$0.031371$	$\mathbf{0}.\mathbf{725683}$	$\mathbf{0}.\mathbf{010336}$
7	$0.540521$	$0.010325$	$0.871499$	$0.038998$	$\mathbf{0}.\mathbf{503406}$	$\mathbf{0}.\mathbf{010071}$
8	$3.23028$	$0.028896$	$4.97701$	$0.030086$	$\mathbf{3}.\mathbf{094443}$	$\mathbf{0}.\mathbf{025968}$
9	$2.962183$	$0.029326$	$3.588782$	$0.031642$	$\mathbf{2}.\mathbf{499952}$	$\mathbf{0}.\mathbf{027685}$

. Since the strategy applied in the sPLVO is not appropriate for outdoor environments, sPLVO was not tested for comparison on KITTI dataset.

The lowest absolute translation and rotation errors for each test are marked in bold. Overall, the proposed method performs better than the other two methods. The translation and rotation errors of the proposed method are about 24.8% and 29.6% lower than that of ORB-SLAM2, respectively. However, it can be seen that the application of line features in the environment with insufficient line features hinders the accuracy of the SLAM process. As shown in Figure 6, the condition of the KITTI 05 dataset is mostly foresting with fewer line features, and the translation and rotation accuracy of the proposed method are reduced by 17.1% and 8.8%, respectively. Table 1 also shows that the proposed method outperforms PL-SLAM, which also employs both point and line features. One reason for this is that the use of PL-BoW suppresses the long-term positioning drift of the system in large outdoor scenes. Another reason for this result is that the proposed line feature outliers elimination method reduces the front-end mismatching and further improves the accuracy of the constraints constructed in the back-end optimization of the system.

4.2. Stereo SLAM on EuRoC Dataset

The accuracy of the four algorithms (PEL-SLAM, PL-SLAM, sPLVO, and ORB-SLAM2) was also compared on the EuRoC dataset, which includes indoor environments images of machine halls and rooms.

Table 2. Absolute RMSE errors of four methods on EuRoC.

Seg.	PL-SLAM		ORB-SLAM2		sPLVO		PEL-SLAM
	Trans. $\left(\mathbf{m}\right)$	Rot. (deg)	Trans. $\left(\mathbf{m}\right)$	Rot. (deg)	Trans. $\left(\mathbf{m}\right)$	Rot. $\left(\mathbf{deg}\right)$	Trans. $\left(\mathbf{m}\right)$	Rot. $\left(\mathbf{deg}\right)$
MH-01	$0.156799$	$6.039926$	$0.038788$	$0.784221$	$0.039265$	$0.735803$	$\mathbf{0}.\mathbf{037170}$	$\mathbf{0}.\mathbf{713027}$
MH-02	$0.142146$	$2.541990$	$0.051815$	$0.786260$	$\mathbf{0}.\mathbf{042640}$	$0.636756$	$0.047628$	$\mathbf{0}.\mathbf{613111}$
MH-03	$0.146580$	$3.376991$	$0.039685$	$0.782628$	$\mathbf{0}.\mathbf{037920}$	$\mathbf{0}.\mathbf{665110}$	$0.043620$	$0.671286$
MH-04	$0.123971$	$6.755803$	$0.131072$	$0.777158$	$0.063646$	$0.711528$	$\mathbf{0}.\mathbf{059120}$	$\mathbf{0}.\mathbf{673853}$
MH-05	$0.554628$	$9.947981$	$0.091573$	$0.781631$	$0.054628$	$0.792583$	$\mathbf{0}.\mathbf{047690}$	$0.693294$
V1-01	$0.168556$	$4.452180$	$0.087874$	$0.712945$	$0.087200$	$0.947981$	$\mathbf{0}.\mathbf{082576}$	$\mathbf{0}.\mathbf{712721}$
V1-02	$0.168729$	$5.623589$	$\mathbf{0}.\mathbf{064295}$	$0.776845$	$0.067039$	$0.798271$	$0.064460$	$\mathbf{0}.\mathbf{756860}$
V1-03	$0.419889$	$9.123150$	$\mathbf{0}.\mathbf{069812}$	$\mathbf{0}.\mathbf{767960}$	$0.070297$	$0.771652$	$0.085186$	$0.861888$
V2-01	$0.194298$	$2.280268$	$0.085005$	$0.781473$	$0.072192$	$0.780698$	$\mathbf{0}.\mathbf{063510}$	$\mathbf{0}.\mathbf{770080}$
V2-02	$0.251842$	$4.635829$	$0.056297$	$0.791928$	$0.065607$	$0.701221$	$\mathbf{0}.\mathbf{054290}$	$\mathbf{0}.\mathbf{653020}$
V2-03	$0.567585$	$6.001996$	$\mathbf{0}.\mathbf{272255}$	$0.787116$	$0.372658$	$0.800124$	$0.405792$	$\mathbf{0}.\mathbf{774410}$

shows the absolute translation and rotation errors of these four algorithms.

It is clear from Table 2 that the strategy used by sPLVO for indoor line features makes it competitive in indoor environments. Nevertheless, the performance of the proposed method on the EuRoC dataset was generally superior to PL-SLAM, ORB-SLAM2, and sPLVO. Compared with the ORB-SLAM2, the translation and rotation accuracy of the proposed method are improved by nearly 5.7% and 7.3%, respectively. These improvements demonstrate that the incorporation of line features increases the accuracy of pose estimation and map construction. Although both sPLVO and the proposed method introduce line feature outliers elimination, the proposed method shows better accuracy in most testing cases. Compared with the state-of-the-art sPLVO, the maximum translation and rotation accuracy of the proposed method improves the results by 17.2% and 24.8%, respectively. These results indicate that using the entropy scale to measure the uncertainty of line features and applying PL-BoW to loop detection are beneficial to the accuracy of the SLAM process.

The tests also demonstrate that the proposed method environments with drastic changes and rapid motion of carriers. Figure 7 illustrates the estimated trajectory comparisons of MH-02 and V1-01, which contains the results with typical rapid motion and changing light. The accuracy of the two methods based on point and line features (i.e., sPLVO and PEL-SLAM) is higher than the methods that use only-point-features, ORB-SLAM2. Meanwhile, the average rotation accuracy of the proposed method is nearly 14.3% higher than that of sPLVO.

4.3. Comparison of Processing Time

Due to the fact that the real-time performance is one of the important indicators of the SLAM process, a comparison between the average processing time per frame of different methods was performed (

Table 3. Average processing time per frame on KITTI and EuRoC.

KITTI	ORB-SLAM	PL-SLAM	PEL-SLAM	EuRoC	ORB-SLAM	PL-SLAM	sPLVO	PEL-SLAM
Seg.	Time $\left(\mathrm{s}\right)$	Time (s)	Time $\left(\mathrm{s}\right)$	Seg.	Time $\left(\mathrm{s}\right)$	Time $\left(\mathrm{s}\right)$	Time $\left(\mathrm{s}\right)$	Time $\left(\mathrm{s}\right)$
0	$0.05932$	$0.09126$	$0.07917$	MH-01	$0.04481$	$0.08069$	$0.08553$	$0.07410$
1	$0.08233$	$0.09912$	$0.09119$	MH-02	$0.04333$	$0.08232$	$0.08238$	$0.07798$
2	$0.06338$	$0.08637$	$0.07817$	MH-03	$0.04346$	$0.08327$	$0.07756$	$0.07635$
3	$0.06470$	$0.08653$	$0.07846$	MH-04	$0.03610$	$0.07563$	$0.07299$	$0.06956$
4	$0.06627$	$0.09610$	$0.08687$	MH-05	$0.03747$	$0.07267$	$0.06968$	$0.06660$
5	$0.06738$	$0.10634$	$0.08765$	V1-01	$0.03360$	$0.07085$	$0.06423$	$0.06133$
6	$0.07303$	$0.09923$	$0.09506$	V1-02	$0.03548$	$0.08065$	$0.05815$	$0.06585$
7	$0.06103$	$0.08712$	$0.08899$	V1-03	$0.03299$	$0.07695$	$0.06187$	$0.05910$
8	$0.06534$	$0.09138$	$0.08847$	V2-01	$0.03232$	$0.06924$	$0.06611$	$0.06172$
9	$0.06156$	$0.09175$	$0.08749$	V2-02	$0.03731$	$0.07138$	$0.06765$	$0.05856$

). Table 3 shows that the operating time of the SLAM process is directly affected by the image resolution, which means that the process needs more time to process the KITTI images. At the same time, the application of line features increases the running time of the system, especially the detection of line features. The PEL-SLAM reduces the time consumption during the line features detection. Compared with PL-SLAM, the proposed method saves approximately 7.7% and 12.2% of the average running time in KITTI and EuRoC datasets, respectively. The running time of the proposed method is 4.5% faster than that of sPLVO without sacrificing the accuracy of the pose estimation and map construction.

5. Conclusions

In this paper, a method that can extract point and line features with the purpose of using them to improve the positioning, mapping, and loop detection is proposed. The PEL-SLAM system solves the problem of point-based failure in poorly textured scenes. The proposed method uses the faster EDlines instead of the widely used LSD and leverages the entropy scale to measure the uncertainty of line features. Meanwhile, the proposed PL-BoW is constructed and applied to the loop detection, which improves the accuracy of loop keyframe matching. Such mechanisms system enables SLAM to be performed in real-time without loss of accuracy while producing a complete point-linemap. Finally, the detailed experiments on KITTI and EuRoC datasets show that the proposed method outperforms many well-known methods with respect to translation and rotation accuracy in challenging environments, with the additional benefit of using less time consumption compared with the state-of-the-art PL SLAM systems. In the proposed algorithm, the accuracy of the system depends on the performance of the detected visual features. The way of combining other sensors information to improve the robustness of the point-line SLAM will be the focus of future research.