A Deep Joint Network for Monocular Depth Estimation Based on Pseudo-Depth Supervision

Abstract

Depth estimation from a single image is a significant task. Although deep learning methods hold great promise in this area, they still face a number of challenges, including the limited modeling of nonlocal dependencies, lack of effective loss function joint optimization models, and difficulty in accurately estimating object edges. In order to further increase the network’s prediction accuracy, a new structure and training method are proposed for single-image depth estimation in this research. A pseudo-depth network is first deployed for generating a single-image depth prior, and by constructing connecting paths between multi-scale local features using the proposed up-mapping and jumping modules, the network can integrate representations and recover fine details. A deep network is also designed to capture and convey global context by utilizing the Transformer Conv module and Unet Depth net to extract and refine global features. The two networks jointly provide meaningful coarse and fine features to predict high-quality depth images from single RGB images. In addition, multiple joint losses are utilized to enhance the training model. A series of experiments are carried out to confirm and demonstrate the efficacy of our method. The proposed method exceeds the advanced method DPT by 10% and 3.3% in terms of root mean square error (RMSE(log)) and 1.7% and 1.6% in terms of squared relative difference (SRD), respectively, according to experimental results on the NYU Depth V2 and KITTI depth estimation benchmarks.

1. Introduction

Monocular depth estimation (MDE) is a challenging task in the field of autonomous driving that aims at recovering the depth map from a given single image (see Figure 1). The estimated depth map is valuable in various intelligent transportation applications, including scene understanding [1], 3D mapping [2], object recognition [3], and obstacle avoidance [4]. Traditionally, the method of obtaining high-precision target depth information is usually to use LiDAR or structured light reflection on the surface of the object to obtain the depth point cloud, but because of its high price and the difficulty of synchronization, there is still a certain distance to be applied and deployed on a large scale in the field of autonomous driving. With the success of Tesla’s pure vision program [5] and the stunning effect of Tesla AI Day [6], the camera has become one of the more popular sensor technologies in the field of autonomous driving because of its low price, rich content of acquired information, and compact size. Accordingly, monocular vision depth estimation has also been hotly anticipated by research and has received more attention.

With the advancements in convolutional neural networks (CNNs) [7,8], researchers have recently adopted various CNN models to enhance the precision of monocular depth estimation (MDE). Compared to traditional methods, like the approach presented in Saxena et al. [9], CNNs generally achieve higher accuracy. These CNN studies can be categorized into three main groups: (1) unsupervised MDE, (2) supervised MDE, and (3) self-supervised MDE. For instance, supervised MDE methods [10,11,12,13,14] have shown promising results by utilizing ground-truth labels for training. On the other hand, common unsupervised methods [15,16] and self-supervised strategies [17,18,19] aim to reduce reliance on ground-truth annotations, thereby enabling depth estimation without direct supervision.

CNNs have been the primary tool for depth estimation, using encoder–decoder architectures  [20,21,22]. While most of the work has focused on the design of decoders [20,22], recent studies have shown that encoders are more important for accurate depth estimation [21,22,23]. A competent encoder must be able to efficiently leverage both local information, which refers to consistency within an item, and long-range dependencies, which refer to distance relationships between objects, because of the lack of depth information [24]. The encoder, where convolutional operators are almost unable to simulate long-range correlations in a constrained receptive field, may, therefore, be the bottleneck of existing depth estimation approaches. As an alternative to CNNs, the Vision Transformer (ViT) [25] has achieved great success in image recognition tasks, where it demonstrates the advantages of being a depth estimation encoder. By utilizing attention mechanisms, the transformer excels in establishing long-range dependency models with global receptive fields. However, ViT encoders lack spatial inductive bias when modeling local information [26], resulting in unsatisfactory performance in near-range depth estimation. In contrast, models that employ convolutional encoders yield better predictions for these regions. Thus, a significant research problem lies in successfully integrating Transformers and convolutional encoders for improved performance. In addition, since the depth estimation problem is a standard regression problem, the loss function is typically the mean square error (MSE) in log space or one of its variants. Although the optimized regression network produced reasonable solutions, the convergence rate was found to be rather slow, and the final solution was far from satisfactory. Therefore, this paper focuses on addressing two problems in existing monocular depth estimation methods. One is how to effectively integrate the transformer and convolutional encoder to improve performance. The other is how to design a matching loss function for the constructed model for better optimization.

To address these challenges, a depth estimation method is proposed based on an encoder–decoder architecture that combines pseudo-depth estimation and a depth estimation network. The proposed method enables end-to-end learning of the mapping from a single image to a depth map. The proposed method starts by encoding the input image into a feature representation. Then, the encoded feature is decoded to generate the corresponding depth map. In order to maintain dense pixel-wise output, the spatial information from the encoder is directly preserved and transferred to the corresponding decoder, without the need for additional parameters or operations. This helps preserve the spatial information and improve the accuracy of the depth estimation process.

(1) Our method introduces a combined pseudo-depth and depth module network that aims to provide both coarse and fine features to accurately predict high-quality depth images from a single RGB image.

(2) The pseudo-depth network utilizes upsampling mapping, residual modules, and an improved codec to directly obtain a depth map prior. The depth network employs an effective global Transformer strategy and Unet depth network to enhance the training of the model, which greatly improves estimation accuracy.

(3) Our method makes collaborative use of several loss functions to improve network training. By using multiple losses, the model can capture different aspects of depth estimation and improve overall performance.

The structure of the paper is as follows. In Section 2, the pertinent literature and earlier studies in this area are covered. The proposed method is fully explained in Section 3. In Section 4, the experimental findings are discussed. Section 5 and Section 6 conclude with a summary of the research and conclusions made in this work.

2. Related Work

2.1. Traditional Methods

MDE is a mature and challenging research area in computer vision and autonomous driving. Initially, researchers heavily relied on manual feature engineering and probabilistic graphical models to tackle this problem. For instance, Saxena et al. [24] successfully employed a combination of absolute and relative depth features, along with the utilization of Markov random fields (MRFs), to accurately predict the depth of monocular images. Building upon this work, Saxena et al. [9] extended their approach to 3D scene reconstruction. Liu et al. [27] used semantic labels as contextual information. Their approach involved employing a learned MRF to infer the semantic category of each pixel in the image, followed by the application of the L-BFGS technique to create a pixel-depth image.

Unlike the previously mentioned parameter-dependent approaches, Karsch et al. [28] treated the MDE as a non-parametric problem. In the pixel shift–based approach proposed by Karsch et al. [28], given an input image, similar images are first searched for within the dataset by comparing GIST features. Then, by transferring the labels from the input image to the matched image, a range of potential depth values for the scene is created. To address the challenges of over-smoothing and preserving occlusion boundaries in the predicted depth map, Liu et al. [29] devised a discrete-continuous condition matrix (CRF) approach. By utilizing this technique, they were able to generate a depth map that avoided excessive smoothing while maintaining accurate boundaries between occluded regions.

Admittedly, the above methods rely heavily on manually created features to predict depth values. These features are carefully designed to capture specific characteristics and patterns in the input data. However, one drawback of such pre-designed features is that they may not generalize well when applied to new and unfamiliar environments.

2.2. Deep Learning–Based Methods

The success of CNN in various computer vision tasks has also prompted the exploration of CNN-based depth prediction methods. Several notable studies [10,30,31,32,33] have addressed the CNN-based depth prediction problem. A monocular depth estimation method based on deep learning was pioneered by Eigen et al. [10]. Two networks were employed in their approach: the first one used the complete input image to predict a global depth map, and the second network improved the global prediction locally. Building upon their initial work, Eigen and Fergus [31] extended the method to incorporate multitask learning, which involved jointly learning depth prediction with other related tasks. Liu et al. developed DCNF that combines the strengths of the deep CNN and continuous conditional random fields (CRFs) within a unified framework to achieve accurate depth prediction. It is worth noting that the above methods [10,32] use fully connected (FC) layers, which involve numerous parameters and lead to high computational costs. This motivates further research into exploring more efficient and lightweight network architectures for MDE tasks.

Laina et al. [11] suggested a fully convolutional residual network (FCRN) for MDE in order to circumvent the delay brought on by the FC layer. Based on the fully convolutional portion of ResNet-50, the FCRN encoder creates feature maps at a 1/32 scale relative to the input image. The final depth map is produced by the FCRN decoder by combining these feature maps. Further research built upon the FCRN framework by exploring the impact of the depth of the encoder network on depth estimation accuracy. Studies by [34,35,36,37,38] increased the depth of the encoder network to over 100 layers. It was found that deeper networks with a larger reception range resulted in improved performance compared to using a variant of ResNet-50. Additional enhancements were made by Hu et al. [36] and Chen et al. [37], who incorporated multiscale features and reconstruction modules into the network. In order to further enhance performance, Cao et al. [34] developed depth estimation as a pixel classification job and used a fully connected conditional random field (CRF) as a post-processing technique. Li et al. [35] utilized multi-scale characteristics to estimate depth based on the outputs of several layers. Godard et al. [39] utilized the loss of left–right disparity coherence for the stereo dataset. Bian et al. [16,17,40,41] explored self-supervised approaches with extensions for scale inconsistency, rotation, dynamic objects, and object boundary blurring. Klingner et al. [42] improved depth estimation for moving objects using a novel semantic mask. Lee et al. [21] designed a multiscale local plane guidance layer. Yang et al. [26] combined transformers and CNNs, while Bhat et al. [22] proposed a transformer-based architecture for adaptive depth estimation. Ranftl et al. [23] introduced the dense prediction transformer that utilizes markers from different stages to improve predictions. These approaches aim to enhance depth estimation results and incorporate global information. Overall, these research contributions demonstrate the ongoing efforts to advance monocular depth estimation and address its limitations through innovative algorithmic strategies and network architectures.

The proposed method aims to reduce complexity and the number of parameters while improving the depth estimation results. The backbone network plays a crucial role by constructing a hierarchical representation of linear complexity for the input image. This hierarchical representation allows for multiscale high-level feature extraction, which is then fed into a back-end network. The back-end network leverages a multilevel localized planar bootstrap layer to predict depth. This layer effectively combines the benefits of convolutional neural networks (CNNs) and transformers. It utilizes a transformer representation learning approach with linear complexity, which helps reduce computational requirements while maintaining strong inference capabilities. By combining the strengths of CNNs and transformers, our method aims to achieve accurate depth estimation with reduced complexity and improved efficiency. This approach demonstrates the ongoing exploration of hybrid techniques to optimize network architectures for depth estimation tasks.

3. The Proposed Method

3.1. Framework Overview

Figure 2 provides an overview of the proposed method, which involves training two networks: the Pseudo-depth Net and the Depth Net. These networks are trained jointly using a large dataset of monocular videos. Given an RGB input I, the pseudo-depth and depth CNNs first estimate their pseudo-depth maps $PD$ and depth maps D, respectively. Eachnetwork is then supervised based on the loss between the true depth image $D^{prime}$ and the generated image D. In addition, the normal matching loss $L_N$ and the edge-aware loss $L_{edge}$ between the pseudo-depth image $PD$ and the depth image D provide additional supervised information to the network. Next, the Pseudo-depth Net (see Figure 3) and Depth Net networks (see Figure 4) are described separately.

3.2. Pseudo-Depth Net

Pseudo-Depth-Encoder Block. Figure 3 illustrates an overview of the pseudo-depth network. Pseudo-depth Net uses pre-trained ResNet-50 as an encoder block, which inputs $304\times 228$ pixels. To overcome the gradient vanishing problem, ResNet-50 constructs a multilayer structure using jump connections. In addition, ResNet-50 has a large receptive field to capture a wide range of spatial information from input images, which is lightweight but has superior performance. The details are listed in

Table 1. The architecture of pseudo-depth-encoder block.

Layer	Input	Output	Details
Input_image	-	In	Input size: 304 × 228 × 3
Convolution_1	In	Conv_1	Kernel number: 64, Kernel size: 7 × 7, stride: 2
Batch Norm_1	Conv_1	BN_1
ReLU_1	BN_1	ReLU_1
Projection_1	ReLU_1	Pro_1	Kernel number: 256
Skip_1	Pro_1	S_1	Kernel number: 256
Skip_2	S_1	S_2	Kernel number: 256
Projection_2	S_2	Pro_2	Kernel number: 512
Skip_3	Pro_2	S_3	Kernel number: 512
Skip_4	S_3	S_4	Kernel number: 512
Skip_5	S_4	S_5	Kernel number: 512
Projection_3	S_5	Pro_3	Kernel number: 1024
Skip_6	Pro_3	S_6	Kernel number: 1024
Skip_7	S_6	S_7	Kernel number: 1024
Skip_8	S_7	S_8	Kernel number: 1024
Skip_9	S_8	S_9	Kernel number: 1024
Skip_10	S_9	S_10	Kernel number: 1024
Projection_4	S_10	Pro_4	Kernel number: 2048
Skip_11	Pro_4	S_11	Kernel number: 2048
Skip_12	S_11	S_12	Kernel number: 2048
Convolution_2	S_12	Conv_2	Kernel number: 1024, Kernel size: 7 × 7, stride: 1
Batch Norm_2	Conv_2	BN_2
Up-projection_1	BN_2	U_1	Kernel number: 512
Up-projection_2	U_1	U_2	Kernel number: 256
Up-projection_3	U_2	U_3	Kernel number: 128
Up-projection_4	U_3	U_4	Kernel number: 64
Convolution_3	U_4	Conv_3	Kernel number: 1, Kernel size: 3 × 3, stride: 1
ReLU_3	Conv_3	Out

. The encoder block first performs $7\times 7$ convolution and $3\times 3$ max-pooling on the input image with a stride of 2. Then, the encoder block contains a repeated application of residual learning, including skip modules and projection modules. The skip module exploits a shortcut connection to skip three layers and performs residual mapping. As shown in Figure 5a, the skip module performs identity mapping without extra parameters, which is formulated as

$$z_{out}=W_3\sigma \left(W_2\sigma \left(W_1{z}_{in}\right)\right)+z_{in},$$

(1)

where $z_{in}$ and $z_{out}$ are the input and output features, respectively. $\left\{W_1,W_2,W_3\right\}$ denote three convolution operations, and $\sigma$ is the rectified linear unit (ReLU). The details are listed in

Table 2. The skip module.

Layer	Input	Output	Details
Input_feature	-	In	Input size: M × N × C
Convolution_1	In	Conv_1	Kernel size: 1 × 1, stride: 1
Batch Norm_1	Conv_1	BN_1
ReLU_1	BN_1	ReLU_1
Convolution_2	ReLU_1	Conv_2	Kernel size: 3 × 3, stride: 1
Batch Norm_2	Conv_2	BN_2
ReLU_2	BN_2	ReLU_2
Convolution_3	ReLU_2	Conv_3	Kernel size: 1 × 1, stride: 1
Batch Norm_3	Conv_3	BN_3
Skip Connection	In, BN_3	SC
ReLU_3	SC	Out

. As depicted in Figure 5b, the projection module performs a shortcut connection with linear projection $W_4$, which is formulated as

$$z_{out}=W_3\sigma \left(W_2\sigma \left(W_1{z}_{in}\right)\right)+W_4{z}_{in},$$

(2)

where $z_{in}$ and $z_{out}$ are the input and output features, respectively. $\left\{W_1,W_2,W_3,W_4\right\}$ denote the convolution operations, and $\sigma$ is the ReLU. The specific details can be found in

Table 3. The projection module.

Layer	Input	Output	Details
Input_feature	-	In	Input size: M × N × C
Convolution_1	In	Conv_1	Kernel size: 1 × 1, stride: 1
Batch Norm_1	Conv_1	BN_1
ReLU_1	BN_1	ReLU_1
Convolution_2	ReLU_1	Conv_2	Kernel size: 3 × 3, stride: 1
Batch Norm_2	Conv_2	BN_2
ReLU_2	BN_2	ReLU_2
Convolution_3	ReLU_2	Conv_3	Kernel size: 1 × 1, stride: 1
Batch Norma_3	Conv_3	BN_3
Convolution_4	In	Conv_4	Kernel size: 1 × 1, stride: 1
Batch Norma_4	Conv_4	BN_4
Skip Connection	BN_3, BN_4	SC
ReLU_3	SC	Out

. The original ResNet-50 contains a series of convolutions and pooling operations, which decrease the resolution of the feature map. To retain the spatial information, the original ResNet-50 architecture eliminates both the FC layer and the last pooling layer. The encoder block can be formulated as

$$u=f_{en}\left(x\right)$$

(3)

where $u\in {\mathbb{R}}^{10\times 8\times 2048}$ is the output feature map and $f_{en}$ encodes the input image x. The encoder block produces 2048 feature maps with a spatial resolution of $10\times 8$ pixels.

Pseudo-Depth-Decoder Block. The encoder block reduces the spatial resolution of the input image, which is $304\times 228$ pixels, to $10\times 8$ pixels. Reversing the pooling operation with unpooling layers improves the spatial resolution of the feature map u. The unpooling layer upscales the feature map by mapping the element into the top-left corner and fills the holes with zeros. After the $2\times 2$ unpooling layer, $5\times 5$ convolution is performed to avoid zero elements, followed by a ReLU activation function. Inspired by residual learning, skip connections are considered to propagate context information, which is named the up-projection module. According to

Table 4. The up-projection module.

Layer	Input	Output	Details
Input_feature	-	In	Input size: M × N × C
Up-pooling	In	Up	2 × 2 upsampling
Convolution_1	Up	Conv_1	Kernel size: 5 × 5, stride: 1
ReLU_1	Conv_1	ReLU_1
Convolution_2	ReLU_1	Conv_2	Kernel size: 3 × 3, stride: 1
Convolution_3	Up	Conv_3	Kernel size: 5 × 5, stride: 1
Skip Connection	Conv_2, Conv_3	SC
ReLU_2	SC	Out

, a $3\times 3$ convolution is followed by a $5\times 5$ convolution.

Additionally, a projection connection is established from the unpooling layer to the $3\times 3$ convolution. The up-projection module is formulated as

$$z_{out}=\sigma \left(W_2\sigma \left(W_{1}up\left(z_{in}\right)\right)+W_{3}up\left(z_{in}\right)\right),$$

(4)

where $up$ represents the $2\times 2$ unpooling operation, $\left\{W_1,W_2,W_3\right\}$ denote three convolution operations, and $\sigma$ is the ReLU. $z_{in}$ and $z_{out}$ are the input and output features, respectively. To estimate the depth maps, four up-projection modules are stacked to upscale the feature map u (2× resolution per block). As shown in Figure 6, the up-projection module allows large feature channels, which can propagate context information to the depth map. Our model uses four up-projection modules to predict an output map with $160\times 128$ pixels, which is approximately half the input resolution $304\times 228$. The prediction maps are up-sampled using bilinear interpolation to their original size before being compared to the provided ground truth. The decoder block can be formulated as

$$\hat{y}=f_{de}\left(u\right),$$

(5)

where $f_{de}$() decodes the feature map $u\in {\mathbb{R}}^{10\times 8\times 2048}$ to reconstruct the depth map $\hat{y}$.

3.3. Depth Net

For depth prediction, the Transformer Conv block and Depth-Unet architecture are adopted. From a tensor $X\in {\mathbb{R}}^{\hat{H}\times \hat{W}\times \hat{C}}$, the initial step in our Transformer Conv module (as shown in Figure 7) is to create query (Q), key (K), and value (V) projections, which are improved with local context. The cross-channel context in terms of pixels is collected by a $1\times 1$ convolutional layer. Then, to represent the spatial context of the channel, a $3\times 3$ deep convolutional layer is used. This process ultimately produces the desired output.

$$\begin{array}{c}\hfill Q=F_d^Q{F}_p^{Q}X,\end{array}$$

(6)

$$\begin{array}{c}\hfill K=F_d^K{F}_p^{K}Y,\end{array}$$

(7)

$$\begin{array}{c}\hfill V=F_d^V{F}_p^{V}X,\end{array}$$

(8)

where $1\times 1$ point-wise convolution is represented by $F_p^{·}$. The $3\times 3$ depth-wise convolution is denoted by the symbol $F_d^{·}$. The network uses bias-free convolutional layers. The query and key projections are then rearranged to form a transposed attention map A of size ${\mathbb{R}}^{\hat{C}\times \hat{C}}$. The transformer conv module is generally described as

$$\begin{array}{c}\hfill \hat{X}=W_p·\hat{V}·Softmax(\hat{K}·\hat{Q}/\alpha ),\end{array}$$

(9)

where the output feature map is represented by $\hat{X}$. The learnable parameters are denoted by the expression $W\in {\mathbb{R}}^{C\times C}$. After reshaping tensors from the original size, the following matrices are produced: $\hat{Q}\in {\mathbb{R}}^{\hat{H}\hat{W}\times \hat{C}}$, $\hat{K}\in {\mathbb{R}}^{\hat{H}\hat{W}\times \hat{C}}$, and $\hat{V}\in {\mathbb{R}}^{\hat{H}\hat{W}\times \hat{C}}$. Before using the softmax function, the magnitude of the dot product between $\hat{K}$ and $\hat{Q}$ is modified using the learnable scaling parameter $\alpha$.

As illustrated in Figure 4, the Depth-Unet architecture uses a skip connection encoder–decoder design with multi-scale side predictions. ReLU activation is applied to all convolutional layers, aside from the prediction layers. The function $1/(\alpha \ast \mathrm{sigmoid}(x)+\beta )$, where $alpha=10$ and $beta=0.1$, is used to constrain the depth values in the prediction layers to be positive within a tolerable range. The Depth-Unet consists of multiple contraction parts and extension parts that are linked together over a long range. Convolutional layers with occasional strides of 2 make up the contraction portion, which results in a 64-downsampling factor overall. This makes it possible for the network to estimate significant depth map displacements. By hopping connections, the network’s expansion portion gradually and nonlinearly up-samples the feature map and includes data from the contraction. A number of up-convolutional and convolutional layers are used to accomplish this. Since information can also move through long-distance links between the contraction and expansion layers, there are no data bottlenecks in the network. See

Table 5. Specification of the Depth-UNet structure. Convolutions conv1 through conv6b make up the contracting portion. Up convolutions (upconvN), convolutions (iconvN, prN), and loss layers alternate throughout the expanding portion. Higher-layer features are concatenated with features from lower levels. pr1 produces the predicted disparity image.

Layer	Input	Output	Details
conv1	-	conv1	Kernel size: 7 × 7, stride: 2
conv2	conv1	conv2	Kernel size: 5 × 5, stride: 2
conv3a	conv2	conv3a	Kernel size: 5 × 5, stride: 2
conv3b	conv3a	conv3b	Kernel size: 3 × 3, stride: 1
conv4a	conv3b	conv4a	Kernel size: 3 × 3, stride: 2
conv4b	conv4a	conv4b	Kernel size: 3 × 3, stride: 1
conv5a	conv4b	conv5a	Kernel size: 3 × 3, stride: 2
conv5b	conv5a	conv5b	Kernel size: 3 × 3, stride: 1
conv6a	conv5b	conv6a	Kernel size: 3 × 3, stride: 2
conv6b	conv6a	conv6b	Kernel size: 3 × 3, stride: 1
pr6+loss6	conv6b	conv6b	Kernel size: 3 × 3, stride: 1
upconv5	conv6b	upconv5+pr6+conv5b	Kernel size: 4 × 4, stride: 2
iconv5	upconv5+pr6+conv5b	iconv5	Kernel size: 3 × 3, stride: 1
pr5+loss5	iconv5	iconv5	Kernel size: 3 × 3, stride: 1
upconv4	iconv5	upconv4+pr5+conv4b	Kernel size: 4 × 4, stride: 2
iconv4	upconv4+pr5+conv4b	iconv4	Kernel size: 3 × 3, stride: 1
pr4+loss4	iconv4	iconv4	Kernel size: 3 × 3, stride: 1
upconv3	iconv4	upconv3+pr4+conv3b	Kernel size: 4 × 4, stride: 2
iconv3	upconv3+pr4+conv3b	iconv3	Kernel size: 3 × 3, stride: 1
pr3+loss3	iconv3	iconv3	Kernel size: 3 × 3, stride: 1
upconv2	iconv3	upconv2+pr3+conv2	Kernel size: 4 × 4, stride: 2
iconv2	upconv2+pr3+conv2	iconv2	Kernel size: 3 × 3, stride: 1
pr2+loss2	iconv2	iconv2	Kernel size: 3 × 3, stride: 1
upconv1	iconv2	upconv1+pr2+conv1	Kernel size: 4 × 4, stride: 2
iconv1	upconv1+pr2+conv1	iconv1	Kernel size: 3 × 3, stride: 1
pr1+loss1	iconv1	output	Kernel size: 3 × 3, stride: 1

for more information.

3.4. Loss Function

There are also several other well-established loss functions [39,43] that have been found to perform accurate depth estimation and reconstruction tasks using CNN. It has been demonstrated that structural similarity index measure [44] (SSIM) loss performs well in these tasks. The anticipated and actual depth maps’ structural and luminance similarities are both taken into account by the SSIM loss. The loss function offers further in-depth details on the application and efficiency in complex estimate and reconstruction jobs.

$$L_{SSIM(y,\hat{y})}=\frac{(2·Av{g}_y·Av{g}_{\hat{y}}+a)(Converg{e}_{y\hat{y}}+b)}{(Av{g}_y^2+Av{g}_{\hat{y}}^2+a)(Va{r}_y^2+Va{r}_{\hat{y}}^2+b)},$$

(10)

where the values for the anticipated depth map and the ground depth map, respectively, are y and $haty$. The average and variance are denoted as $Avg$ and ${Var}^2$, respectively. The network’s convergence constants are a and b.

Another common loss function is the mean squared error:

$$L_2={∥y-\hat{y}∥}_2^2,$$

(11)

where the predicted map $\hat{y}$ should be at approximately the ground truth map y. The network is trained using stochastic gradient descent using the input images x and the accompanying depth maps y.

The normal matching loss is used to compute the loss between the surface normals of the predicted depth map and the pseudo-depth map. Formally,

$$L_N=\frac{1}{K}\sum _{k=1}\parallel n_i-n_i^{*}{\parallel }_1+\frac{1}{K}\sum _{k=1}{\parallel n_i-{\hat{n}}_i\parallel }_1,$$

(12)

where the surface normals based on the predicted depth, pseudo-depth, and ground-truth map are $n_i$, $n_i^{*}$, and ${\hat{n}}_i$, respectively. The entire number of pixels in the image is represented by N. The overall depth structure is strongly supervised by the pixel loss function.

Normals, essential geometric features, constitute a complimentary modality to depth. Edge-aware loss is utilized for structure-oriented ordering to improve the clarity of edges and thus ensure accurate depth estimation. We use surface normal maps to find planar regions where the normals are nearly identical and regions where the normals change significantly. We then track and sample paired points [45] on either side of these significantly varying edges. Within a small distance of the edge points, two points are randomly selected on each side, ensuring that these four points lie on a line orthogonal to the sampled edge points. This results in three pairs of points for sorting loss in Figure 8: (a,b), (b,c), and (c,d). After converting an image to grayscale, the gradient mappings $M_y$ and $M_{\hat{y}}$ are obtained, along with the gradient size map M. The gradient size map is then thresholded in order to compute the edge E.

$$E=[M\ge \alpha ·max(M\left)\right],$$

(13)

where $\alpha$ is used to control the threshold of E density. For each edge point $e=(x,y)$ sampled from E, four edge points $e=(x,y)$$[(x_k,y_k),k=a,b,c,d]$ are sampled by

$$\left\{\begin{array}{ccc}\hfill x_k& =\hfill & \hfill x+{\delta }_k{M}_x\left(e\right)/M\left(e\right)\\ \hfill y_k& =\hfill & \hfill y+{\delta }_k{M}_y\left(e\right)/M\left(e\right),\end{array}\right.$$

(14)

We sample them within a moderate distance $\beta$ from the edge point e to obtain ${\delta }_a<{\delta }_b<0<{\delta }_c<{\delta }_d$. The values of $\alpha$ and $\beta$ are set to 0.1 and 30, respectively. A margin of two pixels on each side of the edge is also provided in order to avoid sampling points too close to the edge point e, which would make it difficult to determine the ground-truth depth value. The whole sampling process is summarized in Algorithm 1.

In planar regions, paired points are also sampled in the same plane. In addition, to improve the global geometric quality, we globally randomize the sampling of paired points. Thus, the overall structure is enhanced, and the boundary regions of the object are also emphasized. More specifically, the approach involves sampling point pairs around the edges of the image and ensuring that the relative angle of the normal vectors of these sampled points is consistent with the pseudo-depth. To achieve this, the technique known as edge-guided sampling [45] is employed to create the point pairs <$A,B$>. Edge-aware loss is defined as follows:

$$L_{edge}=\frac{1}{N}\sum _{i=1}^N\parallel n_{A_i}·n_{B_i}-n_{A_i}^{*}·n_{B_i}^{*}{\parallel }_1+\frac{1}{N}\sum _{i=1}^N{\parallel n_{A_i}·n_{B_i}-{\hat{n}}_{A_i}·{\hat{n}}_{B_i}\parallel }_1,$$

(15)

where the normals of the sampled points from the predicted depth map are represented by $n_A$ and $n_B$. $n_A^{*}$ and $n_B^{*}$ stand for the normals of sampled points from the pseudo-depth map. ${\hat{n}}_{A_i}$ and ${\hat{n}}_{B_i}$ are the normal of sampled points from ground-truth map. Edge-guided sampling and the relative normal loss can be used together to constrain the depth estimation in the object border region effectively. This approach ensures that the predicted depth values accurately represent the depth variations at the edges of the object. By utilizing the information from the edge-guided sampling, the network can better understand the object boundaries and refine the depth estimation accordingly. The relative normal loss further strengthens the consistency of the surface normal between the predicted depth map and the pseudo-depth map, helping to maintain accurate depth estimation, specifically in the boundary region of the object. Overall, this combined approach provides a more precise and reliable depth estimation in the object boundary region.

In summary, our objective function is defined as follows:

$$L=L_{SSIM}+L_2+L_N+L_{edge}.$$

(16)

Algorithm 1: The procedure for edge-guided sampling

4. Experiments

4.1. Experimental Setup

(1) Datasets

We evaluated the proposed method on three benchmark datasets (NYU-Depth-v2 [46], KITTI [47], and DDAD [48]) in both indoor and outdoor scenarios. About 240,000 RGB and depth image pairs from 464 distinct indoor settings captured with a Microsoft Kinect camera make up the NYU-Depth-v2 dataset. Around 48,000 synchronized RGB and depth image pairs from this dataset were used to train our method. Additionally, 654 images were used for testing. To align with previous literature [10,11], the original images were first reduced to half their size from $640\times 480$ pixels. Then, a region of $304\times 228$ pixels was centered and cropped from the images to serve as the input to the network. The KITTI dataset, on the other hand, focuses on real-world outdoor scenes and consists of high-resolution outdoor images with dimensions of $375\times 1241$ pixels. Similar to Eigen et al. [10], we utilized only the left-side images for our method. The KITTI dataset contains 22,600 training images and 697 test images. The associated Velodyne data points are projected onto the left image plane to create ground-truth depth maps. Missing depth values in the ground-truth depth maps were not taken into account during training or testing. For this study, only the bottom $228\times 912$ pixel region was utilized as the LiDAR measurements could not capture the upper part of the image. By using these two diverse datasets, spanning indoor and outdoor scenes, we ensured a comprehensive evaluation of the proposed method’s performance in different environments and scenarios. The DDAD dataset contains 200 driving videos taken in urban scenes. The point clouds that were scanned by LiDAR are given. Compared to the KITTI dataset, the DDAD dataset has almost all vehicles traveling on the road and fewer parked vehicles, which made it more challenging for the model training. We divided the dataset according to the standard training/testing subset, which contained 150 training scenarios (totaling 12,650 images) and 50 validation scenarios (totaling 3950 images). We used the validation scenes for model performance evaluation. During training, the resolution of the images was scaled to 640 pixels × 384 pixels.

(2) Training details

In our method, the network was trained to predict depth maps using RGB inputs. The network architecture was implemented using the PyTorch framework. We utilized Python 3.7 and CUDA 11.6 on Ubuntu 20.04 to create an MDE model. The model was trained on the NYU-Depth-v2 and KITTI datasets using an NVIDIA RTX 3090 with 24 GB of RAM. Except for the first layer, which included a variable number of input channels, the ResNet weights in the encoder block were initialized using models that had already been trained on the ImageNet [49] dataset. The network was trained using the AdamW [50] optimizer with 100k iterations on each dataset, with the learning rate set to ${10}^{-4}$.

(3) Data Augmentation

Several data augmentation techniques were employed to increase the diversity of the training samples and the network’s robustness. These techniques, inspired by Eigen et al. [10], included rotation, scaling, color transformation, flipping, and small translations. By applying these data augmentation techniques, we could generate a more diverse set of training samples, which helped improve the network’s generalization capability and robustness. These techniques contributed to the network’s ability to estimate depth accurately in various real-world scenarios.

4.2. Evaluation Metrics

We tested and compared the outcomes of our method with various depth estimation methods on the NYU-Depth-v2, KITTI, and DDAD datasets. Some of the results are shown in Figure 8 and Figure 9. A colored map is used to visualize the different depths. Blue indicates a shorter distance, and yellow indicates a longer distance. A well-recognized set of assessment methods proposed by Eigen et al. [10] was used, which included four assessment indicators: root mean square error (RMSE), RMSE (log), absolute relative difference (ARD) and squared relative difference (SRD). The evaluation metrics are calculated using the following:

$$RMSE\left(linear\right)=\sqrt{\frac{1}{\left|N\right|}\sum _{i=1}^N{\parallel y_i-{\hat{y}}_i\parallel }^2},$$

(17)

$$RMSE\left(log\right)=\sqrt{\frac{1}{\left|N\right|}\sum _{i=1}^N{\parallel log{y}_i-log{\hat{y}}_i\parallel }^2},$$

(18)

$$ARD=\sqrt{\frac{1}{\left|N\right|}\sum _{i=1}^N\frac{|y_i-{\hat{y}}_i|}{{\hat{y}}_i}},$$

(19)

$$SRD=\sqrt{\frac{1}{\left|N\right|}\sum _{i=1}^N\frac{\parallel y_i-{\hat{y}}_i{\parallel }^2}{{\hat{y}}_i}},$$

(20)

where the sum of the pixels is denoted by N. The ith pixel value of the predicted depth image is $y_i$. The ground truth depth image’s ith pixel value is denoted by ${\hat{y}}_i$.

4.3. Evaluation Results

Our method is compared with previous state-of-the-art methods, including DORN [20], AdaBins [22], DPT [23], FCRN [11], SC-DepthV1 [40], SC-DepthV2 [16], and SC-DepthV3 [17]. The NYU-Depth-v2 dataset was used to evaluate the proposed method. The performance comparison of the NYU-Depth-V2 dataset is presented in

Table 6. Objective metrics for depth estimation on the NYU-Depth-v2 [46] dataset. The best results are bolded.

Method	Param (M)	RMSE (linear)	RMSE (log)	ARD	SRD	GPU 3090(s)
FCRN	63.6	0.573	0.195	0.152	0.121	83.52
SC-DepthV2	40.9	0.554	0.186	0.142	0.112	37.48
DORN	110.3	0.509	0.172	0.115	0.082	925.22
SC-DepthV3	28.7	0.486	0.165	0.123	0.090	44.37
AdaBins	78	0.364	0.122	0.103	0.070	1377.6
DPT	123.1	0.357	0.121	0.110	0.077	81.85
Ours	19.8	0.342	0.110	0.095	0.064	36.24

. As shown in the table, the RMSE (linear), RMSE (log), ARD, and SRD improvements using the proposed method with the NYU-Depth-v2 dataset are more than 4%, 9%, 13%, and 16%, respectively, compared to the published best architecture DPT. The advanced performance of our proposed model on most evaluation measures is due to the architecture and loss function we propose. Additionally, the proposed model outperforms other advanced methods with fewer parameters, such as AdaBins and DPT. The proposed compact pseudo-depth estimation network, in combination with the transformer depth estimation network, successfully helps achieve the accurate and efficient estimation of depth maps. The visualization results are displayed in Figure 9. In comparison to previous methods, our method estimates the depth values of the example image given accurately and is more resistant to changing lighting circumstances.

The depth estimation results for KITTI are demonstrated in

Table 7. Objective metrics for depth estimation on the KITTI [47] dataset. The best results are bolded.

Method	Param (M)	RMSE (linear)	RMSE (log)	ARD	SRD	GPU 3090(s)
SC-DepthV1	27.9	4.997	0.196	0.118	0.870	38.39
SC-DepthV3	28.7	4.699	0.188	0.119	0.756	38.51
DORN	110.3	2.727	0.120	0.072	0.307	985.59
DPT	123.1	2.573	0.092	0.062	0.221	96.91
AdaBins	78	2.360	0.088	0.058	0.190	1504.35
Ours	19.8	2.351	0.085	0.058	0.187	36.27

. The RMSE (linear), RMSE (log), and SRD improvements using the proposed method with the KITTI dataset are more than 8%, 7%, and 15%, respectively, compared to the DPT MDE method. Furthermore, Figure 10 provides the visualization results for the KITTI dataset. SC-DepthV1 and SC-DepthV3 present fuzzy depth maps at the object boundaries, while the methods of DORN, DPT, and AdaBins fail to preserve fine details, such as distant tiny objects. In contrast, our method has richer structural and object details and produces visually pleasing results.

Comparisons are made with SC-DepthV1 and SC-DepthV3 methods on DDAD.

Table 8. Objective metrics for depth estimation on the DDAD [48] dataset. The best results are bolded.

Method	Param (M)	RMSE (linear)	RMSE (log)	ARD	SRD	GPU 3090(s)
SC-DepthV1	27.9	16.118	0.279	0.168	3.825	88.04
SC-DepthV3	28.7	15.702	0.248	0.143	3.008	88.27
Ours	19.8	13.427	0.218	0.118	2.861	82.18

lists the performance comparison results on the DDAD dataset. Compared with SC-DepthV1 and SC-DepthV3 methods, the proposed method outperforms all competitors in all metrics by a significant margin. Specifically, in the DDAD dataset, the proposed method improves the RMSE (linear), RMSE (log), ARD, and SRD by 14%, 12%, 17%, and 4%, respectively, compared to SC-DepthV3. A qualitative comparison is shown in Figure 11. The proposed method achieves more accurate and clearer depth estimation results. The cars in all four images in Figure 11 are better recognized by our method, and the reconstructed scenes are satisfactory with clear object boundaries and reasonable depth times, which can be better applied in the field of autonomous driving.

4.4. Ablation Study

To confirm the efficacy of the proposed method, comparison tests were carried out on the NYU-Depth-V2 dataset, i.e., Pseudo-Depth Net and Depth Net. A network architecture ablation study was conducted.

Table 9. Ablation experiments for the proposed architecture on the NYU-Depth-v2 [46] dataset. The best results are bolded.

Method	RMSE (linear)	RMSE (log)	ARD	SRD
Ours	0.342	0.110	0.095	0.064
Ours w/o Pseudo-Depth Net	0.378	0.129	0.194	0.095
Ours w/o Depth Net	0.461	0.142	0.253	0.142

presents the results. The first row indicates the proposed method. The second row indicates the results obtained using the method without Pseudo-Depth Net. As shown in the table, the absence of Pseudo-Depth Net degraded the performance of the model. The second row represents the results obtained using the method without Depth Net. The performance of the network without Depth Net was significantly reduced. Table 9 demonstrates that joint Pseudo-Depth Net and Depth Net could greatly improve the accuracy of depth estimation.

4.5. Running Time

In this subsection, we show the results of testing the runtime on NYU-Depth-v2, KITTI, and DDAD datasets. For the NYU-Depth-v2 dataset, the experiments were performed on 654 images of size $304\times 228$. For the KITTI dataset, the experiments were performed on 697 images of size $228\times 912$. For the DDAD dataset, the experiments were performed on 3950 images of size $640\times 384$. The test time results are shown in Table 6, Table 7 and Table 8. All CNN-based methods were implemented using PyTorch on GPU (GPU was NVIDIA RTX 3090), belonging to NVIDIA of CA, USA, USA. Among all the methods, the proposed method was faster than all the competing methods, which shows that real-time depth estimation is possible.

5. Discussion

In the final section, several aspects of the computational efficiency of our model and its portability to current network architectures are discussed. The transferability of deep networks has garnered increasing interest in recent times. Currently, many methods achieve satisfactory results by training and testing on the same dataset. However, when confronted with diverse datasets from various fields or captured using different cameras, the performance often suffers a significant decline. To tackle this challenge, researchers have turned their focus toward improving the portability of depth networks. One approach involves incorporating camera parameters into the depth estimation framework. By considering specific camera information, like intrinsic and extrinsic parameters, the network can better adapt to different camera setups, resulting in more accurate depth estimation. Furthermore, employing domain adaptation techniques during the training process has emerged as a promising research direction. Domain adaptation aims to bridge the disparity between different datasets by aligning their feature distributions. This enables the network to generalize better and enhance performance on unseen datasets. Through the application of domain adaptation methods, the network learns to extract more robust and transferable features, leading to improved portability across diverse datasets or camera configurations.

Indeed, while deep networks have demonstrated impressive performance, their high computational requirements pose a significant challenge for practical application, particularly in real-time scenarios. The ability to achieve real-time performance with deep estimation networks holds great importance in their practical usability. Lightweight networks are characterized by their reduced number of parameters, which can impact their overall performance. Thus, a significant research focus lies in improving the accuracy of these networks while ensuring real-time performance. Striking a balance between accuracy and efficiency is a topic that warrants further exploration. Furthermore, questions such as the depth cues learned by deep networks and the specific cues utilized in the estimation process have received little attention in the literature. Investigating these aspects can deepen our understanding of the inner workings of deep networks for depth estimation tasks. To summarize, the development of lightweight networks for real-time performance and further research on the mechanisms underlying monocular depth estimation using deep learning are both worthwhile directions. These areas of study hold promise in advancing the practical application and understanding of deep estimation networks.

6. Conclusions

A deep joint network, which consists of a pseudo-depth and a depth network, is proporsed to provide meaningful coarse and fine features to predict high-quality depth images from a single RGB image. The pseudo-depth network utilizes upsampling mapping, residual modules, and a modified codec to obtain high-resolution depth maps directly. The depth network employs an effective global transformer strategy and Unet depth network to improve the performance, which greatly improves the estimation accuracy. In addition, multiple losses are used jointly to improve the training of the network, and the proposed method attains advanced performance on KITTI and NYU-Depth-v2 datasets.