Self-Supervised Monocular Depth Estimation Based on Channel Attention

Abstract

Scene structure and local details are important factors in producing high-quality depth estimations so as to solve fuzzy artifacts in depth prediction results. We propose a new network structure that combines two channel attention modules in a deep prediction network. The structure perception module (spm) uses a frequency channel attention network. We use frequencies from different perspectives to analyze the channel representation as a compression process. This enhances the perception of the scene structure and obtains more feature information. The detail emphasis module (dem) adopts the global attention mechanism. It improves the performance of deep neural networks by reducing irrelevant information and magnifying global interactive representations. Emphasizing important details effectively fuses features at different scales to achieve more accurate and clearer depth predictions. Experiments show that our network produces clearer depth estimations, and our accuracy rate on the KITTI benchmark has improved from 98.1% to 98.3% in the δ < 1.25³ metric.

1. Introduction

Accurately estimating the depth from a single image is a basic task in computer vision, which enables computers to understand real scenes. It has been widely used in robotics navigation [1], autonomous driving [2] and augmented reality [3] for generating high-quality depth from colors instead of using expensive LIDAR sensors. Although monocular cameras are cheap and lightweight, the task of depth estimation is still challenging to the traditional SfM (Structure from motion) algorithms. Recently, supervised methods [4,5,6,7,8] have achieved some success. Nevertheless, they rely on a large amount of ground truth, which can only be obtained sparingly by expensive LiDAR sensors [9]. Self-supervised methods use geometric constraints as the only source of supervision for monocular videos [10] or simultaneous stereo image pairs [11].

In recent years, due to its powerful ability to extract data features and express complex relationships, deep learning has attracted more attention in solving traditional monocular depth estimation problems. Deep learning [12,13,14,15,16,17,18,19] is committed to mining the hidden rules of data from large datasets and then using the learned rules to predict the results. It is expected that the models obtained through learning will have a good generalization ability. Convolutional neural networks are used to extract semantic features that contain structural information of the scene.

The local detail feature is another feature that emphasizes the boundaries of objects and tries to produce clear depth maps. U-Net [20] is the framework used by most depth estimation networks, in which the decoder fuses features based on concatenation and basic convolution. We find that these operations fail to retain sufficient detail and accurately obtain spatial information, resulting in the inefficient integration of features at different levels and blurred artifacts in regions of depth discontinuity.

We propose a new monocular depth estimation network based on the attention mechanism, which has different channel attention modules above the depth network to enhance the performance of features by capturing more contextual information about the scene geometry. In this paper, a deep convolutional neural network model for monocular depth estimation is proposed, and the overall architecture of the network has the form of an encoder–decoder. The encoder completes the feature extraction process, and the decoder completes the feature output process. The main contributions of our work are as follows:

(1)

A new network architecture is proposed, which combines two channel attention modules in the depth prediction network to capture more contextual information of the scene and emphasize detailed features.

(2)

The spm is based on frequency channel attention to enhance the perception of the scene structure and obtain more feature information. The dem is based on the channel attention mechanism to efficiently fuse features at different scales and emphasize important details to obtain clearer depth estimates.

(3)

The superior performance of the proposed method is validated on the KITTI benchmark and the Make 3D dataset.

2. Related Work

2.1. Supervised Depth Estimation

Deep learning has been continuously developed, and it has shown a relatively high performance in image processing, such as image classification [21], target detection [22], semantic segmentation [23,24,25,26,27,28,29], etc. Estimating depth is an inherently uncertain problem because there may be multiple seemingly reasonable depths for the pixels in an image. Recently, supervised methods have shown their ability to predict models. They can correctly estimate the depth of color images. Various supervised methods based on deep learning are continuously being explored. However, they require high-quality real depth of the ground truth, which can be expensive.

Eigen et al. [4] used multiscale neural networks for dense pixel depth estimation, with one estimating a coarse global depth prediction and the other estimating a local fine prediction generated by the first network. Eigen et al. [5] improved the framework of prediction depth, surface normal vectors and semantic tags and deepened the network structure. Laina et al. [6] proposed a full convolution network which uses a residual network to extract global information and designed residual upsampling blocks to improve the quality of depth estimation. Many methods have turned depth estimation into a classification problem. Cao et al. [7] divided the continuous depth values into several boxes and classified them by CNN. Fu et al. [8] proposed a discretization strategy of increasing distance to discretize the depth and reformulated deep network learning as an ordered regression problem to prevent the loss of over-reinforcement. The methods of supervision require annotated datasets, and the cost of obtaining these datasets is very high.

2.2. Self-Supervised Monocular Depth Estimation

Due to the limitations of supervised methods, the self-supervised approach was developed. Godard [11] extended the reconstruction constraint by the loss of parallax smoothness and the loss of left–right depth consistency. Zhou et al. [10] proposed a method for learning depth and self-motion from monocular videos by training a depth estimation network and an individual attitude network. Tosi et al. [30] introduced a traditional depth estimation method, which uses the inverse Huber loss and image reconstruction loss. Wong et al. [31] proposed a new objective function to elucidate the bilateral cyclic relationship between the left and right parallax and introduced an adaptive regularization scheme to deal with co-visible and occluded regions in stereo pairs. The full CNN framework proposed in [32] uses monocular images and the corresponding optical flow to estimate the exact depth map. In [33], to improve the consistency of the estimated depth and the self-motion between consecutive frames, 3D constraints were proposed. Guizilini et al. [34] learned to compress and unpack in order to decompress by symmetric packing, thus preserving detailed features. The state-of-the-art framework proposed by Godard et al. [35] is Monodepth2, which introduces minimal reprojection loss to handle occlusion and automatic masking schemes to robustly remove invalid pixels.

2.3. Self-Attention Mechanism

Self-attention mechanisms are used in monocular depth estimation. Wang et al. [36] modeled spatiotemporal correlations between video sequences and images by aggregating global contexts specific to each location of the query. Zhang et al. [37] learned a better image generator by incorporating the self-attention mechanism into the GAN framework. Fu et al. [38] designed two types of attention modules to enhance the feature representation ability of scene segmentation. Johnston et al. [39] captured the background of similar disparity values in discontinuous areas by exploring the similarity of features in spatial dimensions.

Compared to previous work, we demonstrate that more relative depth information is obtained from more distant regions due to different channels. We can obtain a better depth estimation performance by capturing globally relevant information along the channel dimension and distinguishing different features.

3. Self-Supervised Depth Estimation and Network Models

3.1. Network Model

3.1.1. Attention U-Net Architecture

In this paper, the network architecture combining a depth network with a pose network is adopted. In this case, the depth network uses an encoder and decoder architecture integrated with U-Net [20] and ResNet18 [40]. The input is a single frame of an image at a certain moment. The pose network uses the same encoder as the depth network, and three neighboring frames are used as inputs. The pose network adopts PoseNet without mask prediction [10]. The output is the predicted depth of each pixel, and the overall structure of the network model is shown in Figure 1.

As shown in Figure 2, the U-Net [20] network architecture is a lightweight all-convolution neural network which includes two parts: the encoder and decoder. We use a residual network. The last layer of the encoder connects the spm, and skip connections are used to facilitate the gradient and information flow throughout the model. The encoder connects the dem through skip connections to obtain a clear depth map. Finally, we use the nearest neighbor interpolation to successively sample the predicted depth map at multiple scales until the resolution of the original input is reached. In addition, the training loss is calculated using the higher input resolution.

3.1.2. Depth Network

Table 1. The structure of the encoder network.

Layers	k	s	c	Activation
convl	7	2	64	ReLU
maxpool	3	2	64	-
econv1	3	1	64	ReLU
econv2	3	2	128	ReLU
econv3	3	2	256	ReLU
econv4	3	2	512	ReLU
spm	3	-	512	-

and

Table 2. The structure of the decoder network.

Layers	k	s	c	Activation
upconv6	3	1	512	ELU
dem	3	1	512	ReLU
iconv6	3	1	512	ELU
upconv5	3	1	256	ELU
dem	3	1	256	ReLU
iconv5	3	1	256	ELU
disp5	3	1	1	Sigmoid
upconv4	3	1	128	ELU
dem	3	1	128	ReLU
iconv4	3	1	128	ELU
disp4	3	1	1	Sigmoid
upconv3	3	1	64	ELU
dem	3	1	64	ReLU
iconv3	3	1	64	ELU
disp3	3	1	1	Sigmoid
upconv2	3	1	32	ELU
dem	3	1	32	ReLU
iconv2	3	1	32	ELU
disp2	3	1	1	Sigmoid
upconv1	3	1	16	ELU
dem	3	1	16	ReLU
iconv1	3	1	16	ELU
disp1	3	1	1	Sigmoid

show the encoder and decoder structure of the depth network. Every two convolutional blocks form a building block with the following structure: Conv (3 × 3)-BN-ReLU-Conv (3 × 3)-BN. The BN (Batch Normalization) layer is the batch normalization layer. In the whole model, each convolutional kernel has a convolutional step of 2. The output part of the convolutional kernel uses the ReLU activation function uniformly. Corresponding to the encoder, the decoder part of the depth network uses a convolutional kernel with a step size of 1 and a size of 3 × 3. Here are the symbols in the tables: k, kernel size; s, step size; c, number of output channels; activation, activation function. The encoder block is represented by econv.

3.2. Structure Perception Module

GAP (global average pooling) is an average operation which can be regarded as the simplest frequency spectrum of the input, that is, the component with a frequency of 0. However, it is not good to only use separated gap information in the channel attention. Since the separate spectrum is not good, several more frequency components can be considered, which introduce a multispectral channel attention mechanism.

Based on theoretical analysis, GAP is a special form of 2DDCT (two-dimensional discrete cosine transform), which is proportional to the lowest frequency component in 2DDCT. We can see that using GAP in the channel attention mechanism means that only the information from the lowest frequency is kept. All the components from the other frequencies are discarded, which also encode useful information patterns representing the channel and should not be lost.

In order to use more information in the input characteristic graph X, we can use multiple frequency components, including GAP. First, we take the input characteristic graph X and divide it into multiple groups according to the channel: [X₀, X₁, ..., X_n−1]. The original number of channels is C. After the division, the number of channels in each group is C’, and C’ = C/n, C should be a multiple of n. For each group, a specific frequency component of 2DDCT is assigned, and this specific frequency component needs to be chosen in advance.

Thus, the 2DDCT can be used as a preprocessing of channel attention. Inspired by [41], we apply the channel attention module to weight different local features. As shown in Figure 3, the channel attention module tries to learn and model the correlations between different channel mappings. We put this attention module at the beginning of the decoder part to integrate global information into local features and improve the representation of local features.

$$\begin{gathered} Fre{q}^i=2DDC{T}^{u.v}\left(X^I\right), \\ =\sum _{h=0}^{H-1}\sum _{w=0}^{W-1}{X}_{:,h,w}^i{B}_{h,w}^{u,v} \\ s.t.i\in \left\{0,1,\dots ,n-1\right\}, \end{gathered}$$

(1)

Here, [u, v] is the 2D index of the frequency component, $Fre{q}^i$ is a C’-dimensional vector. The overall preprocessing vector is the one that splices all:

$$Freq=cat([Fre{q}^0,Fre{q}^1,\dots ,Fre{q}^{n-1}]),$$

(2)

Here, Freq is the obtained multispectral vector, and the channel attention of this multispectral can be written as:

$$ms\_att=sigmoid(fc(Freq)).$$

(3)

It can be seen that different combinations of [u, v] may be used for different groups, that is, different frequency components can be used for each group. In this way, the single spectrum of the GAP is expanded into multiple spectra.

3.3. Detail Emphasis Module

The jump connects the low-level information of the encoder, while the high-level information contains richer spatial details. Further processing of local details is crucial and cannot be simply fused, which contains semantic differences between features of different levels. It leads to blurred artifacts in the predicted depth map. We predict some sharp edges by processing local details. The network can easily recover accurate depth predictions. Therefore, by using a channel focus mechanism, the network is enabled to focus on specific channel features. Inspired by [42], the detail emphasis module allows us to highlight some important details and effectively integrate features at different scales. 3D alignment is used to retain 3D information. Then, two-layer MLP (Multilayer Perceptron) is used to magnify the spatial correlation of the cross-dimensional channel. (MLP is an encoder–decoder structure with a compression ratio of r, which is the same as for BAM.) As shown in Figure 4, given an input feature map F₁ ∈ R^CxHxW and an output, F₂ is defined as:

$$F_2=M_C(F_1)\otimes F_1$$

(4)

3.4. Loss Function

The minimized photometric reprojection is used to deal with the object occlusion problem, while the automatic masking loss is used to deal with the artifacts caused by motion. The per-pixel minimum photometric reprojection error L_p is experssed as:

$$L_p=\underset{t^{\prime }}{\min }pe\left(I_t,I_{t^{\prime }\to t}\right)$$

(5)

$$I_{t^{\prime }\to t}=I_{t^{\prime } }\langle proj\left(D_t,T_{t^{\prime }\to t},K\right)\rangle$$

(6)

where $I_t$ is the target image, $I_{t^{\prime }}$ is the source image, $T_{t^{\prime }\to t}$ represents the camera pose of each source image with respect to the target image and pe is the photometric error, consisting of SSIM (Structural Similarity index) and L1 loss. The expression is:

$$pe(I_t,I_{t^{\prime }\to t})=\partial \frac{1-SSIM(I_t,I_{t^{\prime }\to t})}{2}+(1-\partial )||I_t-I_{t^{\prime }\to t}|{|}_1$$

(7)

The SSIM is used to measure the similarity of the target image to the estimated image. The L1 loss is used to make the difference between the pixel values in the target image and those in the estimated image and take the absolute value. The value of $\partial$ is set to 0.85.

In this paper, an automatic masking method is used to solve the artifact problem of processing images. The pixel mask loss μ is applied to the masking loss by selectively weighting the pixels, μ ∈ {0,1} is calculated automatically during the forward pass of the network. The function expression is:

$$\mu =[\min pe(I_{t,}{I}_{t^{\prime }\to t})<\min pe(I_t,I_{t^{\prime }})]$$

(8)

where [] is the Iverson bracket.

Furthermore, an edge-aware smoothness regularization term L_s is used in order to regularize the differences in the texture-free regions.

$$L_s=\left|{\partial }_x{d}_t{}^{*}\right|e^{-\left|{\partial }_x{I}_t\right|}+\left|{\partial }_y{d}_t{}^{*}\right|e^{-\left|{\partial }_y{I}_t\right|}$$

(9)

where d_t^* is the average normalized inverse depth to stop the estimated depth from shrinking.

The final loss function L is a weight sum of photometric loss L_p and smoothness loss L_s:

$$L=\mu L_p+\lambda L_s$$

(10)

where λ is a constant with a value of 0.001. The value of λ is referred to Godard’s setting, which is the most commonly used optimal parameter setting, and L_p is the combination of SSIM and L1.

4. Experiments and Analysis

Our approach is compared with previous related works on the KITTI dataset [9] to demonstrate that our proposed approach can improve the accuracy of model prediction and reduce the error in model estimation. An ablation study of the attention mechanism is also studied based on monodepth2.

4.1. Implementation Details

Our network model is implemented via PyTorch, and during training, 20 epochs are trained. The batchsize is set to 12, and the resolution of the input and output is 640 × 192. We use ResNet-18 pre-trained on ImageNet [43] as the encoder. For the first 15 epochs, we use the Adam [44] optimizer, with a learning rate of 10⁻⁴. The learning rate for the last five epochs is 10⁻⁵. The GPU is NVIDIA GeForce GTX 1080Ti, and the training duration is 12 h.

4.2. KITTI Results

We use the KITTI dataset for all experiments, using the data segmentation of Eigen et al. [4] to increase the distance to 80 m. Before training, the same preprocessing as that performed by Zhou et al. [10] is performed to remove some static frames. In the end, 39,810 frames are used for training, 4424 frames are used for verification and 697 frames are used for testing. When training these networks, the same camera inherent focal length and average focal length are used for all frames. As shown in Figure 5, for road signs and utility poles, our model yields clearer results. As shown in

Table 3. Quantitative results. Comparison of our method with existing methods for the intrinsic splitting of methods used in KITTI 2015. The best effect categories for each method are shown in bold. Lower values are better for Abs Rel, Sq Rel, RMSE and RMSE log, and higher values are better for δ < 1.25, δ < 1.25² and δ < 1.25³. In the table, M: Self-supervised mono supervision.

Method	Train	Abs Rel	Sq Rel	RMSE	RMSE Log	δ < 1.25	δ < 1.252	δ < 1.253
DDVO [36]	M	0.151	1.257	5.583	0.228	0.810	0.936	0.974
DF-Net [45]	M	0.150	1.124	5.507	0.223	0.806	0.933	0.973
Ranjan [46]	M	0.148	1.149	5.464	0.226	0.815	0.935	0.973
EPC++ [47]	M	0.141	1.029	5.350	0.216	0.816	0.941	0.976
Struct2depth [48]	M	0.141	1.026	5.291	0.215	0.816	0.945	0.979
Monodepth [11]	M	0.124	1.388	6.125	0.217	0.818	0.929	0.966
SGDdepth [49]	M	0.117	0.907	4.844	0.196	0.875	0.954	0.979
Monodepth2 [35]	M	0.115	0.903	4.863	0.193	0.877	0.959	0.981
PackNet-SfM [34]	M	0.111	0.785	4.601	0.189	0.878	0.960	0.982
HR-Depth [50]	M	0.109	0.792	4.632	0.185	0.884	0.962	0.983
Johnston [39]	M	0.106	0.861	4.699	0.185	0.889	0.962	0.982
Ours	M	0.107	0.765	4.532	0.184	0.893	0.963	0.983

, our network outperforms other self-supervised methods in most metrics.

4.3. Make 3D Results

Make3D [51] consists of RGB monocular images and their depth maps, but stereo images are not available. Therefore, it is not possible to train an unsupervised depth estimation model on this dataset. Our model is equally valid when tested on other datasets. We tested Make3D using the model trained at KITTI 2015, employing the same camera parameters as those provided by the KITTI dataset. We cropped the input images according to the aspect ratio requirements of the images in the model. As shown in

Table 4. The error measure results on the Make3D dataset. In the table, M: Self-supervised mono supervision, S: Self-supervised stereo supervision.

Method	Train	Abs Rel	Sq Rel	RMSE	RMSE Log
Monodepth [11]	S	0.544	10.94	11.760	0.193
Zhou [10]	M	0.383	5.321	10.470	0.478
DDVO [36]	M	0.387	4.720	8.090	0.204
Monodepth2 [35]	M	0.322	3.589	7.417	0.163
Ours	M	0.314	3.112	7.048	0.159

, our method outperforms other self-supervised methods.

4.4. Ablation Study

Table 5. Comparison of prediction accuracy before and after adding the structure perception module and the detail emphasis module, respectively, on the KITTI dataset using ResNet 18 (R18).

Method	Backbone	Abs Rel	Sq Rel	RMSE	RMSE Log	δ < 1.25	δ < 1.252	δ < 1.253
Baseline (monodepth2)	R18	0.115	0.903	4.863	0.193	0.877	0.959	0.981
Baseline + spm	R18	0.111	0.833	4.768	0.191	0.881	0.961	0.982
Baseline + dem	R18	0.110	0.812	4.733	0.190	0.882	0.961	0.982
Ours	R18	0.110	0.810	4.678	0.190	0.882	0.962	0.983

and

Table 6. Comparison of prediction accuracy before and after adding the structure perception module and the detail emphasis module, respectively, to the KITTI dataset using ResNet 50 (R50).

Method	Backbone	Abs Rel	Sq Rel	RMSE	RMSE Log	δ < 1.25	δ < 1.252	δ < 1.253
Baseline (monodepth2)	R50	0.110	0.831	4.642	0.187	0.883	0.962	0.982
Baseline + spm	R50	0.109	0.768	4.554	0.183	0.885	0.963	0.983
Baseline + dem	R50	0.109	0.772	4.593	0.185	0.886	0.962	0.982
Ours	R50	0.107	0.765	4.532	0.184	0.893	0.963	0.983

show the ablation study based on Monodepth2, and the backbones are ResNet 18 and ResNet 50, respectively. It shows that all of our contributions achieve a steady improvement in almost all evaluation metrics and obtain a consistent performance gain on different backbones. A visualization of the improvement of different components is shown in Figure 6.

4.5. Discussion

Based on monodepth2, our approach adds spm and dem. The experimental results show that incorporating the attention mechanism is effective. We use a pre-trained residual network as the backbone to extract semantic features, which are then fed into spm and generate new features to explicitly enhance the perception of the scene structure. In the decoding phase, we gradually recover the spatial resolution, use skip connections to facilitate the gradient and information flow throughout the model and use dem to produce fine details. While our addition of the attention mechanism improves the performance of the monocular depth estimation network, there are some shortcomings: the network becomes more complex and takes longer of a cycle.

5. Conclusions

In this paper, a new network structure is proposed based on a self-supervised monocular depth estimation model. Two different channel attention modules are added. The spm uses a frequency channel attention network to enhance the perception of the scene structure and obtain more feature information. The dem employs a channel attention mechanism. It emphasizes some important details and can effectively fuse features at different scales to achieve more accurate and clearer depth prediction. Our network has achieved advanced results on the KITTI dataset.