StereoVO: Learning Stereo Visual Odometry Approach Based on Optical Flow and Depth Information

Abstract

We present a novel stereo visual odometry (VO) model that utilizes both optical flow and depth information. While some existing monocular VO methods demonstrate superior performance, they require extra frames or information to initialize the model in order to obtain absolute scale, and they do not take into account moving objects. To address these issues, we have combined optical flow and depth information to estimate ego-motion and proposed a framework for stereo VO using deep neural networks. The model simultaneously generates optical flow and depth information outputs from sequential stereo RGB image pairs, which are then fed into the pose estimation network to achieve final motion estimation. Our experiments have demonstrated that our combination of optical flow and depth information improves the accuracy of camera pose estimation. Our method outperforms existing learning-based and monocular geometry-based methods on the KITTI odometry dataset. Furthermore, we have achieved real-time performance, making our method both effective and efficient.

1. Introduction

Simultaneous localization and mapping (SLAM) technology has found widespread application in numerous domains, such as autonomous driving, robotics, and augmented reality. Visual SLAM techniques play a significant role in this field and have garnered considerable attention from both the computer vision (CV) and robotics communities over the past few decades, as they are based on a low-cost and small sensor system [1,2], which guarantees these advantages compared to other sensor-based SLAM techniques. As an essential module of a visual SLAM system, VO aims to accurately estimate camera motion from frame sequences.

In recent decades, there has been a significant amount of research focused on creating precise, dependable, and robust VO methods. Typically, these methods fall into two main categories, VO based on geometry and VO based on learning. In geometry-based VO approaches, the final outcome is achieved by using a coarse-to-fine iteration procedure [3,4]. With careful initialization or relocalization within the generated maps, these methods have demonstrated a remarkable level of accuracy. Geometry-based VO methods heavily rely on manual intervention to detect, analyze, and address failures; track faults; and refine localization results. Alternatively, learning-based VO methods leverage neural networks to extract features from entire RGB input frames and then regress the features to obtain the camera’s ego-motion [5,6,7,8]. Some of these methods incorporate intermediate parameters such as optical flow maps [9,10,11], while others use additional information such as semantic segmentation [12,13]. Moreover, some treat VO as a subset of a larger multi-task framework [14,15,16,17]. VO methods can be broadly classified into monocular and stereo VO based on the type of camera used. Monocular VO methods recover absolute scale by utilizing additional information [3,4,18], such as the height of the camera or labels. Stereo frames, with their depth information, provide an inherent scale reference. Our approach can effectively eliminate the initial operation required in ORB-SLAM2 [18] when working with stereo frames. Several methods [15,16,17,19] estimate depth and ego-motion together, but they fail to account for the presence of moving objects, leading to suboptimal results. In this paper, our goal is to investigate the feasibility of estimating the frame-to-frame ego-motion of a camera solely from stereo image pairs.

We introduce a state-of-the-art network that utilizes stereo image pairs to estimate camera ego-motion by utilizing optical flow and depth information as intermediate quantities. Optical flow enables the network to discern between static and moving objects, while depth information provides an absolute scale. The primary contributions of our study are as follows: (1) we present a new approach to learning VO that effectively leverages optical flow and depth information to estimate ego-motion; (2) we put forth a novel structural ssim cost volume (SSIMCV) for extracting depth information from stereo image pairs; and (3) our method achieves state-of-the-art results when compared to existing geometry-based and learning-based VO methods on the KITTI visual odometry dataset. The remainder of this paper is structured as follows: §2 discusses relevant VO studies, §3 provides an overview of our StereoVO architecture, §4 compares the performance of our approach to other state-of-the-art methods, and §5 concludes the paper.

2. Related Work

Previous studies of VO methods are reviewed in this section. These methods can be classified into two main categories: geometry-based and learning-based methods.

2.1. Methods Based on Geometry

VO methods based on geometry rely on extracting geometric constraints from images to estimate motion. These methods can be further categorized into two types: feature-based methods and direct methods. In feature-based methods such as MonoSLAM [20], which is a typical filter-based approach [20,21], the state vector size can increase substantially with a rapid increase in feature points, making real-time calculations challenging. PTAM [22] improved based on MonoSLAM by splitting tracking and mapping into different threads, resulting in real-time camera motion tracking. ORB-SLAM [3] inherited the framework of PTAM while achieving real-time performance on CPU. Later, ORB-SLAM2 extended ORB-SLAM to support stereo and RGB-D cameras, while ORB-SLAM3 [23] added support for fisheye cameras and visual–inertial SLAM.

In contrast to feature-based methods, direct methods do not rely on artificially designed sparse features but instead build an optimization problem that estimates camera motion directly from pixel information, typically photometric errors. DTAM [24] calculates a dense depth map for each key-frame by minimizing global spatially regular energy functionals. The camera’s position is determined by aligning the entire image with the depth map directly. This method is computationally intensive. To reduce the amount of calculation involved, SVO [25] only extracts features when a key-frame is selected to initialize new 3D points. DSO [26] combines photometric and geometric errors and jointly optimizes all model parameters, and the method demonstrates robust performance in certain featureless environments.

However, all these geometry-based methods are complex, and fine adjustment is required for each module to achieve good performance. Additionally, current frameworks are basically fixed, and algorithms are approaching their limits.

2.2. Methods Based on Learning

In recent years, learning-based VO has gained substantial attention for its promising prospects of learning capability and robustness to challenging environments.

DeepVO [27] introduced an end-to-end framework for monocular VO that utilizes deep Recurrent Convolutional Neural Networks and requires five sequential frames for estimation, which results in greater memory consumption and poorer real-time performance. Zhan et al. [16] proposed a parallel CNNs framework where joint training for single view depth and VO improved depth prediction by imposing additional constraints on depth, achieving competitive results for VO. UnDeepVO [7] estimates the 6-DoF pose of a monocular camera using deep neural networks and leverages stereo image pairs to recover absolute scale. However, its inability to generalize to other datasets due to changes in absolute scale limit its application. Similarly, D3VO [28] also employs stereo image pairs to train a monocular VO network and models photometric uncertainties to enhance VO accuracy, which results in a fixed absolute scale and cannot be generalized to other datasets. MLF-VO [29] estimates camera ego-motion by using paired monocular frames in an unsupervised manner, focusing on different depth and image feature fusion strategies, which utilizes Multi-Layer Fusion to leverage RGB and inferred depth information to predict ego-motion after predicting depth images for each source frame; however, dynamic objects are not considered. P-CNN [9] estimates frame-to-frame ego-motion from dense optical flow generated by BROX [30], which can be time-consuming. Flowdometry [10], trained by optical flow generated by FlowNet [31], achieved better results than P-CNN but did not consider absolute scale. TartanVO [32] utilizes monocular image sequences, optical flow labeling, and camera matrix from the TartanAir dataset [33], which offers a wide range of synthetic data, for training and can be applied to other real-world datasets such as KITTI [34] and EuRoC [35] by adjusting the intrinsic matrix. In addition to pure VO methods, there are learning visual–inertial odometry methods [36,37] that combine visual and inertial together, which are not in the scope of this work.

Although recent monocular VO methods [19,27,28] employ stereo frames for training to obtain absolute scale, they still cannot fully solve the issue of absolute scale that persists in all monocular methods. Additionally, most VO methods based on learning do not consider the effect of dynamic objects. In this work, we leverage stereo image pairs to recover scale and employ optical flow and depth information to consider the effects of dynamic objects.

3. Approach

In this section, we provide a detailed description of our method. We first present the structure of our proposed network. Next, we introduce the SSIMCV, and finally, the loss function of our network is described.

3.1. The Structure of StereoVO

The proposed StereoVO comprises three main components: the Flow-Net, the Depth-Net, and the VO-Net. For the Flow-Net, FlowNet2 [38] is employed to estimate optical flow due to its superior generalization capability in realistic scenes and fast processing speed. In the Flow-Net, the optical flow of each pair of consecutive stereo frames are generated and then down-sampled to 1/8 of the original resolution using Average Pooling. In the Depth-Net, we utilized SSIMCV to acquire depth information. Different from previous methods, such as [7,8,15,16], we use down-sampled features to calculate depth information without up-sampling to obtain a full resolution depth map, which allows for a simpler structure of the network. The resulting optical flow and extracted depth information were used as input to the VO-Net. The architecture of the proposed StereoVO method is illustrated in Figure 1.

In the Flow-Net, the FlowNet2 network takes two continuous stereo frames and produces raw optical flow. These optical flows and frame at time t + 1 are used to warp the frame at time t. The structural similarity index (SSIM index) [39] between the warped frame at t and the original frame at t is utilized as the confidence of the optical flow. The SSIM index for two frames, x and y, is calculated as follows:

$$ssim\left(x,y\right)=\frac{\left(2{\mu }_x{\mu }_y+C_1\right)\left(2{\sigma }_{xy}+C_2\right)}{\left({\mu }_x{}^2+{\mu }_y{}^2+C_1\right)\left({\sigma }_x{}^2+{\sigma }_y{}^2+C_2\right)},$$

(1)

where μ and σ are the mean intensity and standard deviation of frames, respectively, C₁ and C₂ are constant while we set C₁ = 9 × 10⁻⁴, and C₂ = 10⁻⁴ follows [39]. Then, the SSIM index is concatenated with optical flow and fed into Conv Block1, whose architecture is shown in Figure 2. Additionally, depth information is obtained from the left and right frame using the SSIMCV block, as described in Section 3.2. The architecture of the Conv and FC blocks is shown in Figure 3, and the parameters for all convolution and pooling layers can be found in

Table 1. Configuration of the convolution layers with learnable parameters. Larger kernel size is used for the beginning three Conv Blocks to capture more information. The table provides information on kernel size (k), stride (s), padding (p), and the number of input and output channels (chns) for each layer. As the depth of the network increases, the size of the feature maps decreases.

Block	Layer	k	s	p	chns
Conv Block	Conv1_1	7	2	3	3/16
	Conv1_2	3	1	1	48/16
	Conv Redir1	7	2	3	3/2
	Conv2_1	5	3	2	18/32
	Conv2_2	3	1	1	96/32
	Conv Redir2	5	2	2	18/2
	Conv3_1	5	2	2	34/64
	Conv3_2	3	1	1	192/64
	Conv Redir3	5	2	2	34/2
Conv Block4	Conv4_1	3	2	1	157/256
	Conv4_2	3	1	1	471/256
Conv Block5	Conv5_1	3	2	1	256/512
	Conv5_2	3	1	1	768/512
	Conv Redir5	3	2	1	256/2
Conv Block6	Conv6_1	3	2	1	514/1024
	Conv6_2	3	1	1	1542/1024
	Conv Redir6	3	2	1	514/2
FC	FC1	-	-	-	4104/1024
	FC2	-	-	-	1024/6

.

In the FlowNet, the left-side and right-side optical flows, generated by FlowNet2, are concatenated with ssim right and ssim left, respectively. The SSIM index (Equation (1)) is computed between the destination image I_d and the warped destination image W_d,

$$ssi{m}_s=SSIM\left({\mathit{W}}_{\mathit{d}},{\mathit{I}}_{\mathit{d}}\right),$$

(2)

and we warp the source image I_s(x,y) via flow from source image to destination image flow_sd(u, v):

$${\mathit{W}}_{\mathit{d}}={\mathit{I}}_{\mathit{s}}\left(x+u_x,y+v_y\right).$$

(3)

3.2. SSIM Cost Volume

A cost volume stores the matching costs for pairing a pixel from feature map f₁ with its corresponding pixels in feature map f₂. In stereo matching and optical flow estimation, three common types of cost volumes are used: Difference, Concatenation, and Correlation cost volume. The Difference cost volume computes the disparity between two patches, the Concatenation cost volume simply concatenates features f₁ and f₂, whereas FlowNet2 uses a window size of one to compute the Correlation cost volume between two patches. However, each of these cost volumes can be easily affected by outliers.

To assist the network to match left and right features accurately, we introduce the SSIM cost volume, which employs a sliding window patch comparison between two feature maps, f₁ and f₂, each with width, height, and number of channels (w, h, and c). Now, we consider a single comparison of two patches for sample, using a sliding window of size k and a maximum displacement of d_max. The SSIM cost volume is defined as follows:

$$\mathit{c}\mathit{o}\mathit{s}\mathit{t}\left(x_0,y_0\right)=SSIM\left({\mathit{f}}_1\left(x_0,y_0\right),{\mathit{f}}_2\left(x_0+d,y_0\right)\right),$$

(4)

where $x_0\in \left. [0,w\right],y_0\in \left. [0,h\right],d\in \left. [0,d_{\max }\right]$, f(x,y) denotes the feature window whose center is (x₀, y₀). Then, the cost volume is averaged over channel dimension, and finally, a 3D matrix is achieved.

3.3. Loss Function

We take advantage of the sequential stereo frames and the ground truth camera motions in our task. Our objective is to minimize the camera motion loss l. The end-to-end loss is defined as:

$$l=a{‖{\mathit{P}}_{\mathit{o}}-{\mathit{L}}_{\mathit{o}}‖}_2+b{‖{\mathit{P}}_{\mathit{p}}-{\mathit{L}}_{\mathit{p}}‖}_2+c\times l_f,$$

(5)

$$l_f=d‖{\mathit{W}}_{\mathit{f}}-\mathit{f}‖+e\times SSIM\left({\mathit{W}}_{\mathit{f}},f\right),$$

(6)

$${\mathit{W}}_{\mathit{f}}=\mathit{f}\left(x+\mathit{d}\mathit{i}\mathit{s}\mathit{p}(x,y),y\right),$$

(7)

where P and L represent predicted and ground truth motion, respectively, (o,p) indicates relative Euler angels o and translation p of left camera from t + 1 frame to t frame, o consists of roll, pitch, and yaw, p is composed of translation along the x, y, and z axles, $\Vert ·{\Vert }_2$ denotes mean square error, and a to e are the factors of losses; similarly to [27], we set a = 100, b = 1, and c = 0.01; following [8], we set d = 0.15 and e = 0.85 while training.

4. Experimental Results

In this section, we present the experimental results to verify the effectiveness of our proposed method. We begin by detailing the experimental datasets and implementation specifics, followed by a comparative analysis of our results with state-of-the-art methods.

4.1. Dataset and Metrics

Dataset: The KITTI odometry benchmark [34] comprises twenty-two image sequences captured by two cameras mounted on a car during driving scenes in suburban and highway environments. Sequences 00–10 provide ground truth information on camera position, whereas sequences 11–21 only offer stereo images. The KITTI dataset records at a 10 frames-per-second rate during driving and includes numerous dynamic objects, such as cars, motorcycles, bicycles, and pedestrians, with speeds between 0 km/h and 90 km/h, which makes it a challenging dataset for the VO algorithm. To train our proposed network, we leveraged sequences 00–06 and 08, while our validation process selected frames randomly from the training set. Meanwhile, sequences 07, 09, and 10 served as our testing datasets.

Table 2. Sequence detail of KITTI odometry benchmark.

Phase	Train								Test
Sequence	00	01	02	03	04	05	06	08	07	09	10
Frames	4541	1101	4661	801	271	2761	1101	4071	1101	1591	1201
Total	19,308								3893

shows the number of frames in each sequence. The stereo images are resized to a resolution of 1024 × 256. Labels from the KITTI dataset are absolute position according to the initial frame of the sequence; we transform labels to relatively frame-to-frame Euler angle and translation along the x, y, and z axles.

Evaluation Metrics: We adopt the KITTI odometry criterion, which reports the average translational error T_rel(%) and rotational errors R_rel(deg/100 m) of possible sub-sequences of length (100, 200, ···, 800) in meters.

4.2. Implementation Details

For our experiments, we employed the PyTorch framework for implementing our work and trained it using a single Nvidia RTX2080S GPU. The parameters of Flow-Net remained fixed during training. To optimize our approach, we utilized the Adam optimizer with a batch size of 100 and a learning rate of 10⁻³ for 1000 epochs. Subsequently, we adjusted the learning rate to 10⁻⁴ for 200 epochs, followed by another 50 epochs at a learning rate of 10⁻⁵. We eventually fine-tuned our network via a batch size of 10 and a learning rate of 10⁻⁵.

4.3. Comparison with Other Methods

Table 3. Odometry results comparison with the state-of-the-art VO methods. The best results of each block are highlighted in bold. M: monocular; St: stereo; G: geometry based; S: supervised; U: unsupervised. ‘-’ means the results are not available from that paper.

Method		Seq. 07		Seq. 09		Seq. 10		Average
		Trel (%)	Rrel (deg/100 m)	Trel (%)	Rrel (deg/100 m)	Trel (%)	Rrel (deg/100 m)	Trel (%)	Rrel (deg/100 m)
MG	VISO2-M [4]	23.61	19.11	4.04	1.43	25.20	3.80	17.62	8.11
	ORB-SLAM2-M [18]	10.96	0.37	15.30	0.26	3.71	0.30	9.99	0.31
StG	ORB-SLAM2-S [18]	0.59	0.20	1.8	0.18	1.67	0.37	1.35	0.25
	ORB-SLAM3-S [23]	0.62	0.21	1.16	0.19	1.02	0.34	1.24	0.25
MU	GeoNet [8]	-	-	30.57	9.89	45.14	10.37	37.86	8.17
MU	D3VO [28]	-	-	10.40	0.75	14.24	1.21	12.32	0.98
MU	MLF-VO [29]	6.11	2.59	3.64	1.33	6.39	1.14	5.38	1.68
MS	DeepVO [27]	3.91	4.60	-	-	8.11	8.83	6.01	6.72
MS	PCNN [9]	-	-	6.42	2.52	19.70	3.62	13.06	7.79
MS	Flowdometry [10]	-	-	12.64	8.04	11.65	7.28	12.15	7.66
MS	TartanVO [32]	8.51	4.55	7.24	2.99	9.20	2.80	8.32	3.45
StS	StereoVO (our)	3.42	1.87	5.11	1.39	5.78	1.62	4.77	1.63

presents a comparison between our best model and several state-of-the-art geometry-based [3,4] algorithms as well as learning methods [8,9,10,27,28,29,32]. VISO2 is one of the most widely used geometry-based VO algorithms commonly employed as a baseline for comparison, which uses sparse features for stereo matching to achieve efficient monocular and stereo VO. Monocular VISO2 leverages a fixed camera height to recover absolute scale. ORB-SLAM2, another geometry-based algorithm, has demonstrated impressive results utilizing monocular, stereo, and RGB-D cameras. In our experiments, we denote the monocular and stereo versions of ORB-SLAM2 as “ORB-SLAM2-M” and “ORB-SLAM2-S”, loop-closure detection is disabled while comparison. Among the learning-based approaches, we selected both supervised [9,10,27,32] and unsupervised [8,28,29] methods.

In the supervised methods, DeepVO takes image sequences as inputs and uses RNN to refer to former ego-motion, while PCNN employs calculated optical flow pairs as input and a sampling network. TartanVO estimates optical flow for subsequent estimation. In the unsupervised methods, GeoNet estimates depth maps and ego-motion to generate rigid flow, which is then compared with predicted optical flow to filter out moving objects in the scene. MLF-VO analyzes how different fusion strategies affect the results and performs ego-motion estimation using RGB and inferred depth information in a Multi-Layer Fusion manner. In addition, for ORB-SLAM2, GeoNet, MLF-VO, and D3VO, the trajectories are aligned with the ground truth using [40], with 20 frames used for alignment. The results of VISO2-M, PCNN, DeepVO, Flowdometry, and TartanVO come from their paper or source code.

Figure 4 illustrates the trajectories of the stereo version of ORB-SLAM2 and ORB-SLAM3, PCNN, GeoNet, D3VO, TartanVO, MLF-VO, our StereoVO, and ground truth on sequences 07, 09, and 10 (if available). Our StereoVO achieves the best average performance in both translation and rotation compared to other learning-based methods. In comparison to PCNN, which employs optical flow calculated by BROX [39] as input for their network, our method significantly outperforms them in the task of VO. Flowdometry and TartanVO use similar strategies as ours, where optical flow is utilized as the middle procedure. However, Flowdometry uses FlowNetS [31] to generate optical flow, while TartanVO employs ground truth optical flow as a part of their label. In our StereoVO, we use a fixed-parameter FlowNet2 without ground truth optical flow, yet our method still demonstrates superior results. Similarly to our method, MLF-VO employs depth net and pose net in a series manner. However, MLF-VO did not consider the influence of moving objects. On the other hand, GeoNet uses separate components to learn the rigid flow and optical flow by rigid structure reconstructor and non-rigid motion localizer, respectively. They estimate rigid flow and optical flow in parallel, but in this case, the pose estimator in the rigid parts cannot become fully informed about moving objects. D3VO employs a back-end non-linear optimization module to optimize final poses. However, their T_rel error is more than twice that of ours.

We further compare the RPE (relative pose error) with MLF-VO, which has a similar average translational and rotation RMSE drift, shown in Figure 5, while resultes in

Table 4. Average RPE results on KITTI Seq. 07, 09, and 10. Our method outperforms MLF-VO in all metrics.

Method	RMSE	Mean	Std.
MLF-VO	0.105	0.078	0.068
StereoVO	0.089	0.070	0.055

.

We compare the running time of our method with ORB-SLAM2, ORB-SLAM3, MLF-VO, and TartanVO, and the results are shown in

Table 5. The average processing time (seconds) for each image on KITTI sequence 07, 09, and 10. ‘-’ means not needed.

Method	Data	GPU (G)	Speed (s/Frame)
ORB-SLAM2-S [18]	stereo	-	0.1
ORB-SLAM3-S [23]	stereo	-	0.12
MLF-VO [29]	monocular	1.7	0.03
StereoVO	stereo	1.9	0.05

. The frames in the original KITTI VO dataset were resized to 192 × 640 for accurate comparison. The experiments were performed on Ubuntu 18.04 with AMD Ryzen7 3700x 3.6GHz CPU and NVIDIA RTX2080S 8G GPU. Our stereo VO method was found to be superior to both ORB-SLAM2 and ORB-SLAM3, exhibiting half the inference runtime to deliver precise results. Furthermore, our method uses 1.9 GB memory, which is almost the same as MLF-VO, making it efficient and practical for real-world applications.

5. Discussion

Previous learning-based visual odometry methods have paid little attention to the effect of dynamic objects on prediction results; in this paper, we consider dynamic objects implicitly, and in future research, we will consider the effect of dynamic objects explicitly. Although our method surpasses the stereo versions of ORB-SLAM2 and ORB-SLAM3 in terms of speed, it still falls short in terms of accuracy; more training data and new networks may help to improve it. In the future, we plan to focus on unsupervised stereo VO.

6. Conclusions

We propose a novel data-driven Deep Convolutional Neural Network (StereoVO) that encodes geometrical features in stereo image pairs to estimate camera motion. The main idea is to use CNN to extract features of stereo image pairs in image sequences, then match the features with proposed SSIM cost volume to obtain depth information, fuse depth information with optical flow, and finally achieve the regression of the camera motion among consecutive stereo images. The performance of our approach is better than the monocular versions of ORB-SLAM2, ORB-SLAM3, and DeepVO, PCNN, GeoNet, Flowdometry, and TartanVO. In terms of the memory occupation, our approach is close to the learning-based monocular VO. The proposed method achieves real-time performance and a higher running rate than the stereo versions of ORB-SLAM2 and ORB-SLAM3.