Depth Map Super-Resolution Based on Semi-Couple Deformable Convolution Networks

Abstract

Depth images obtained from lightweight, real-time depth estimation models and consumer-oriented sensors typically have low-resolution issues. Traditional interpolation methods for depth image up-sampling result in a significant information loss, especially in edges with discontinuous depth variations (depth discontinuities). To address this issue, this paper proposes a semi-coupled deformable convolution network (SCD-Net) based on the idea of guided depth map super-resolution (GDSR). The method employs a semi-coupled feature extraction scheme to learn unique and similar features between RGB images and depth images. We utilize a Coordinate Attention (CA) to suppress redundant information in RGB features. Finally, a deformable convolutional module is employed to restore the original resolution of the depth image. The model is tested on NYUv2, Middlebury, Lu, and a Real-Sense real-world dataset created using an Intel Real-sense D455 structured-light camera. The super-resolution accuracy of SCD-Net at multiple scales is much higher than that of traditional methods and superior to recent state-of-the-art (SOTA) models, which demonstrates the effectiveness and flexibility of our model on GDSR tasks. In particular, our method further solves the problem of an RGB texture being over-transferred in GDSR tasks.

1. Introduction

Depth information is widely applicable in augmented reality [1], pose estimation [2], autonomous driving [3], and robot navigation [4]. The industrial depth estimation sensors include time-of-fly (ToF), binocular cameras, and structured-light cameras to accurately obtain real-world depth information. Owing to the limitations of this technology, depth maps collected from consumer-oriented sensors have a low-resolution (LR) and sparse depth information, and they contain significant noise. Traditional interpolation methods, such as bilinear and bicubic interpolation, perform exceptionally poorly in handling sparse and noisy depth images. Meanwhile, high-resolution (HR) RGB images are more accessible to acquire compared to depth maps; however, both have remarkably similar low-frequency features. For example, depth maps of binocular cameras are obtained based on the principle of left and right visual differences in RGB images. He Kaiming proposed the concept of guided filtering in Guided Image Filtering [5] and discovered the local linear relationship between the guided and original images. Based on this theory, subsequent studies discovered a correlation between the discontinuous regions in depth maps and edges of RGB images [6]. Therefore, applying RGB images in guided depth map super-resolution (GDSR) has become a crucial research topic in cross-modal image processing. Recently, GDSR has been applied to reconstruct the collected LR depth maps (LRDMs) and assist with the visual depth estimation tasks [7].

The three main traditional GDSR methods are as follows: the filter method represented by joint bilateral filtering [8,9,10,11], optimizer method represented by Markov random fields [12,13,14,15,16], and learning method represented by sparse dictionary learning [17,18,19]. Filter methods use predesigned filters to guide the depth maps using RGB images to preserve deep edges. However, owing to the rich textural information in RGB images, redundant edge features migrate to depth maps. Furthermore, artificially designed filters, which exhibit inferior generalization performance, cannot be applied to most scenarios. The optimizer method uses a predesigned global energy equation to fit the depth map super-resolution problems; however, these tasks are difficult to express owing to their complexities. Learning-based methods, which have high computational complexities, are limited to low-dimensional signals.

Recently, numerous deep convolutional neural network (CNN)-based methods have been proposed [20,21,22], including the coarse-to-fine method progressive multi-branch aggregation network (PMBANet) [23] that surpasses the conventional accuracy and speed methods. These CNN-based methods have significantly improved the performance in GDSR tasks; however, they have limitations. During feature extraction, the convolution checks the entire input for the same operation and transfers the redundant features. The datasets used by these models are limited and augmented by random cropping or resizing of the NYUv2 dataset [24,25]. This dataset primarily comprises cluttered indoor data and a small collection of complicated objects in the GDSR tasks, which significantly affect the generalization ability of the model.

To this end, we propose a novel semi-coupled deformation convolutional network (SCD-Net) containing a semi-coupled feature extraction module and a learnable deformation convolution filter. Figure 1 shows the workflow of this model.

First, we describe the semi-coupled dictionary learning [26,27]. For feature extraction, a semi-coupled feature module was used to learn the connections between the edges in the RGB images and drastic changes in depth values at the interactions of the front and back scenes in the depth maps. Private kernels extract unique information from the depth and RGB images in each semi-couple block, whereas shared kernels extract common information. The CA attention [28] was included after extracting the RGB features to highlight the valuable features. Therefore, this module can adaptively extract useful feature information for the GDSR.

Second, we propose a deformation convolution module to process the feature images obtained in the feature extraction section. Herein, the working mechanism of the deformation convolution [29] is transferred to the feature map processing, facilitating the element-wise learning of weights and offsets for paired RGB and depth features. To restore an HR depth map, we calculate the weighted average using the weight and original LRDM. The final depth result is obtained by concatenating it with the original LRDM.

Finally, we created the Real-sense dataset to address data scarcity. This dataset was collected using an Intel Real-sense D455 structured-light camera to match 1100 RGB depth-image pair sets. The Real-sense dataset supplements the missing data about human body postures, gestures, and plants in the NYUv2 dataset.

The proposed model was evaluated using the following datasets: NYU v2 [25], Middlebury [30], and Lu [31], and the collected data were sampled using Intel real-sense cameras. Our findings indicate that SCD-Net can achieve excellent results in GDSR tasks while maintaining its speed.

We analyze the current GDSR methods and datasets in the “Section 2”, pointing out their advantages and disadvantages. Following this, in the “Section 3”, we introduce the dataset we have created to address the issues of scarcity and poor quality of datasets in current GDSR tasks. In the “Section 4”, we provide a detailed explanation of our approach. In the “Section 5”, we validate the effectiveness of our method and dataset through qualitative and quantitative analyses, as well as multiple sets of ablation experiments. The “Section 6”, summarizes the entire work, highlights the existing issues, and proposes ideas for future improvements.

2. Related Work

2.1. Benchmark Datasets

Currently, multiple RGB-D datasets are used to train and test models. These datasets can be divided into synthetic and real scenes. The depth errors of synthetic deep datasets, which are obtained using a computer graphics software, approach zero. Commonly used methods include SceneFlow [32], Sintel [33], and New Tsukuba [34]. Virtual scenes cannot satisfy the demands of real-world situations; therefore, deep datasets of natural scenes have been constructed. The Middlebury [30] dataset provides high-quality RGB images and corresponding depth map samples containing noise from 2001 to 2021. The New York University created the NYUv2 dataset [25] containing 407,024 pairs of color and raw depth images of various indoor scenes obtained using Kinectv2 depth cameras (Microsoft corporation, Redmond, WA, USA). The team annotated 1449 RGB images and provided the corresponding depth maps.

2.2. GDSR Algorithm

This section discusses the GDSR algorithm based on the traditional GDSR and deep learning-based methods. We summarize the comparison of four main current GDSR algorithms from three directions. The details of various methods are as follows (

Table 1. Summary of the performance of the four methods.

	Methods	Filter-Based	Optimizer-Based	Learning-Based	Deep Learning
Directions
Computational speed		Fast	Slowest	Fast	Fastest
Accuracy		Lowest	Low	High	highest
Interpretability		Strong	Strong	Poor	Poorest

).

2.2.1. Traditional GDSR Method

Filter-based methods: Kopf et al. [35] learned from bilateral filtering [36] and proposed that an HR input image can provide contextual a priori information during upsampling to yield better reconstruction results. In 2009, Kaiming et al. [5] proposed a guided filtering method that has a local linear transformation relationship between the filtering output and guided image. This overcomes the gradient flipping phenomenon of bilateral filtering and yields good results in applications such as image denoising and joint upsampling. Lo et al. [37] proposed a GDSR-based novel joint trilateral filtering method that integrates the local gradient information of depth maps while reconstructing their HR outputs, thus overcoming the limitations of edge discontinuity. Li et al. [38] proposed a joint example-based super-resolution method for depth maps that learns mapping functions from a set of training samples and improves the resolution of the depth maps through sparse encoding. Lu et al. [10] proposed a method for generating detailed HR depth maps using subsampled depth maps to solve texture transfer problems. These filter-based methods are limited in recovering high-frequency details. Furthermore, artificially designed filters exhibit inferior generalization.

Optimizer-based methods: These methods model the interactions between pixels in RGB images and depth maps represented by Markov random fields [12]. Optimization methods, such as the derived conditional random fields [39], total variation regularization [40], limited memory Broyden–Fletcher–Goldfarb–Shanno (L-BFGS) [41], approximate message passing [42], and non-negative matrix factorization [43], exhibit excellent performance in GDSR. However, these methods have the following limitations: high computational complexity, difficulty in adjusting parameters, and tendency to generate visual artifacts.

Learning-based methods: Tosic et al. [44] proposed a joint dictionary-learning method to identify shared information and correlations between cross-modal data to accomplish the GDSR tasks. Zhang et al. [45] introduced multiscale dictionary learning into the SR method, integrating local and nonlocal priors to suppress reconstruction artifacts and enrich the visual details. Wang et al. [46] proposed a semi-coupled dictionary-learning method that can enhance the shared information and correlation between data domains. Meanwhile, it preserves the independence and original features of each data domain and reduces the complexity of the model. These learning-based methods have the following limitations: high computational complexity and difficulty in hyperparameter selection.

2.2.2. Deep Learning GDSR Methods

Owing to the limitations of the conventional methods, GDSR tasks have been transferred to deep learning.

In 2016, Li et al. designed a directed filtering method, Deep Joint Filtering (DJF) [47], which encodes the target and guided images, concatenates their feature maps, and convolutes these maps to obtain HR depth maps. This method outperforms the guided filtering methods. Zeng et al. [26] modeled the GDSR problem as a neural implicit image-interpolation problem and proposed using joint implicit image functions (JIIF) to represent the learned interpolation weights and values. Tang et al. [48] proposed a joint learning network called BridgeNet, which designs a high-frequency attention bridge to guide GDSR tasks by learning high-frequency information from monocular depth-estimation tasks. Gu et al. [19] proposed a weighted analysis representation model for guided depth image enhancement. This model contains a guided weight function and task-driven learning formulas to optimize and guide the GDSR tasks. He et al. [49] proposed a fast and adaptive depth map super-resolution model that can adaptively decompose high-frequency components from RGB images to guide depth maps in accomplishing super-resolution tasks. Guo et al. [21] employed a multiscale-guided method using residual U-net to learn the residual information between bilateral interpolation upsampling and ground truth values. Wu et al. [50] proposed a fast end-to-end trainable direct-filtering model, emphasizing the requirement for HR input-to-output mapping. Herein, the input is downsampled, and the task is completed at LR. This method significantly improves the speed and use of computing resources.

Recently, Kim et al. [24] proposed a deformation convolution joint upsampling method, Deformable Kernel Network (DKN), that achieves adaptive filtering and improves the super-resolution accuracy. The image generated using this method exhibits a high overall accuracy. However, the technique is limited by simplistic feature extraction, and the receptive field is returned to the original image by each convolutional kernel. The depth image restored using this method contains numerous pixel points with fitting errors. Zhao et al. proposed a Discrete Cosine Transform Network (DCTNet) [51], which shifts the GDSR task from the spatial domain to a frequency domain and uses the discrete cosine transform method to solve an optimization problem channel-wise. This method outperforms the previous state-of-the-art methods in various datasets; however, it overwhelms the GPU.

2.3. Comparison with Existing Methods

The proposed SCD-Net learns from the conventional semi-coupled dictionary learning methods [46], compensates for the limitations of DKN [24], replaces the feature extraction section with a semi-coupled feature extraction module, and solves the problem of RGB texture error transfer. Furthermore, CA [28] attention modules were implemented with deformable convolution modules to enhance the ability of the model to select correct cross-modal features adaptively. Finally, bicubic interpolation was used to restore the sampled subpixels, further improving the accuracy of the results.

3. Real-Sense Dataset

In the NYU dataset, approximately six-pixel areas at the four boundaries of the image contained no information. While cropping them can prevent the impact on the model, it increases the workload. Additionally, the dataset is expected to be sufficiently standardized and user-friendly.

Collection device: The NYUv2 dataset used a Kinect v2 ToF camera, which is disadvantageous in that the errors for close-range objects can only be accurate to the order of centimeters. The resolution of the generated depth map was low, and the edge information feedback for the sampled objects was poor. The depth map generated by the structured-light camera has relatively straightforward edges. Furthermore, it can attain a resolution of 1080 × 720 pixels with a minimum sampling error of 0.01 mm for nearby objects. Hence, an Intel RealSense D455 camera (Intel corporation, Santa Clara, CA, USA) was used as the acquisition device. The camera was recalibrated to ensure that the depth information coverage of the collected depth images was sufficiently high. Finally, the depth-map pair resolution was set according to that of NYUv2 at 640 × 480 pixels.

Data processing: The obtained depth maps and RGB images cannot be directly used for training. Figure 2 shows that the collected depth maps often contain many holes, indicating missing information. Furthermore, the RGB and depth images are not aligned. To obtain aligned data, internal and external parameter matrices of the camera were used to process the image pairs. Then, the RGB pixels correspond to those of the depth map. After that, the following methods used in the NYUv2 dataset were employed:

(i)

Implement the over-segmentation algorithm [27] to obtain rich color image boundary information. Then, use the coloring method [52] to fill the holes under the edge information supervision of the RGB image.

(ii)

A depth completion model [53] is employed to restore the collected images directly.

The final ground truth (GT) depth map was used as the optimal image selected between the two methods after human evaluation.

Data composition: Depth acquisition was performed for various scenarios. The deficiencies present in conventional depth datasets were supplemented, explicitly addressing the absence of specific targets:

(i)

Attitudes of people from different backgrounds.

(ii)

Various models, such as chairs and backpacks.

(iii)

Densely intertwined plants, such as flowers.

To ensure the accuracy of the data, we collected the data within the recommended range of 0.6–4 m for the camera. Figure 2 shows the data composition.

Figure 2b,c indicate that the depth coverages of most of the original HR depth maps in our dataset exceeded 85%. The prediction of depth values in depth-missing areas achieved higher accuracy during the depth completion process. Therefore, the accuracy of the GT image was greater than that of the NYUv2 dataset.

4. Methods

4.1. Problem Formula

Essentially, the GDSR tasks establish a relationship between LR depth maps (LRDMs) and HR-guided RGB images through the optimal objective function comparing the difference between them. In this relationship, RGB images of HR are used for guidance. The LR and GT depth maps are defined as D_L and D_H, respectively (D_L has H/K and W/K sizes). The HR-guided RGB image is defined as R_H (with image sizes H and W), where K is set as the scaling factor. The mapping between D_L and R_H can be written as

$$D_H=G(D_L,R_H;\partial )$$

(1)

where G represents the established network model and $\partial$ represents the parameters learned by the network. Using the energy equation to represent G, we have

$$G=\frac{1}{2}{‖D_H-\widehat{D_L}‖}_2^2+{\frac{\gamma }{2}‖F\left(D_H\right)-F\left(\widehat{R_H}\right)·W\left(\widehat{R_H}\right)‖}_2^2,$$

(2)

where $\widehat{D_L}$ is the upsampled image of $D_L$; $\gamma$ controls the contribution of the second term; $F\left(\right)$ is a learnable filter; $\widehat{R_H}$ is the grayscale image of $R_H$; ∙ is the matrix multiplication operator; and $W\left(\right)$ is the given threshold function that selects the useful edges for the GDSR problem. According to Equation (2), the proposed method overcomes the following limitations:

(i)

Learning a suitable threshold function to select the beneficial edges for the task;

(ii)

Modifying the model to adaptively select the value of $\gamma$. The following section presents a novel SCD-Net model to address these issues and prevent the excessive RGB texture migration in the cross-modal image processing.

4.2. Overall Network Architecture

The proposed SCD-Net addresses the above three issues sequentially, as shown in Figure 3. It first utilizes a semi-coupled approach for feature extraction; then, it applies a CA attention for feature processing; it finally employs deformable convolution modules for feature fusion. Figure 1 shows a detailed diagram of SCD-Net with the following modules:

(i)

A set of HR RGB images and LRDMs is presented. The semi-coupled residual module extracts the shared and private features from the source image.

(ii)

For each element in the feature map, a set of matching weights and offsets are learned, enabling the filter to extract information beneficial to the task from different images.

(iii)

The obtained weights and offsets are multiplied and concatenated with the original LRDM to obtain an HR feature map.

4.2.1. Semi-Couple Feature Extractor

Based on the fundamentals of GDSR tasks, a set of corresponding cross-modal images comprising correlated and independent features is created. For instance, the edges in RGB images are height-correlated with the depth-discontinuities in depth maps, whereas detailed textures in RGB images are absent in depth maps. However, in current deep learning-based GDSR methods, it is impossible to effectively extract cross-modal features by cascading RGB and depth image features. Therefore, the semi-coupled feature extraction module emulates the coupled dictionary-learning method to achieve cross-modal extraction between the shared and private features. Figure 4 shows that the input RGB and depth images were resampled to obtain a feature pair (H/4, W/4, 16).

Figure 5 illustrates the sampling method, wherein the samples were fed into the semi-coupled feature extraction module. In this module, the convolution kernels for RGB images and depth images comprise their respective private convolution kernels and a portion of shared convolution kernels.

The outputs of the resample operation were set to $S_0^R$ and $S_0^{LD}$. The corresponding inputs of each layer can be represented as $S_{i,j}^R$ and $S_{i,j}^{LD}$, where i and j represent the i-th semi-coupled block and j-th kernel group in the i-th semi-coupled block, respectively. The corresponding features outputs of each group are denoted as ${\phi }_{i,j}^R$ and ${\phi }_{i,j}^{LD}$. The feature size is (C, H, W), where C represents the number of convolutional kernels within the kernel group. In Section 5, we determine the value of i and number of convolution kernels in each group during the ablation experiment. The correspondence between the output features and inputs is represented as

$$S_{i,j}^R\left({\phi }_{i,j-1}^R\right)={\phi }_{i,j-1}^R\ast Concat\left(K_{i,j}^{shared},K_{i,j}^{R_{private}}\right),$$

(3)

$$S_{i,j}^{LD}\left({\phi }_{i,j-1}^{LD}\right)={\phi }_{i,j-1}^{LD}\ast Concat\left(K_{i,j}^{shared},K_{i,j}^{{LD}_{private}}\right),$$

(4)

where ∗ represents the convolution operation; $K_{i,j}^{shared}$, $K_{i,j}^{R_{private}}$, and $K_{i,j}^{{LD}_{private}}$ represent the shared kernel corresponding to the j-th kernel group of the i-th semi-coupled block, private kernel for RGB feature extraction, and private kernel for depth map feature extraction, respectively. Concat represents the tensor-splicing operation in PyTorch. Based on Figure 4, the RGB feature output of each module can be represented as

$${\phi }_i^R=ReLU\left\{S_{i,3}^R\left(ReLU\left(S_{i,2}^R\left(ReLU\left(S_{i,1}^R\left({\phi }_{i-1}^R\right)\right)\right)\right)\right)+{\phi }_{i-1}^R\right\}.$$

(5)

Equation (5) represents the same depth map output features. The CA attention module [28] was used to highlight the useful information in color features. Through this semi-coupled learning approach, the final output features include common and unique features in the cross-modal image pairs.

4.2.2. Deformable Kernel for Guided Edge

In the previous section, RGB and depth features were obtained. Their corresponding unique features were determined using a semi-coupled feature extraction module. In order to suppress the redundant RGB features, a CA attention was used to highlight the information that improves up-sampling. Thereafter, a deformation convolution module was employed as a learnable filter. Consequently, deformable convolutional kernels can adapt to irregular image structures. Compared to traditional convolutional kernels with fixed shapes, they better capture textures and features in RGB images that help recover depth information. Additionally, using deformable convolutions in this context allows for the representation of a broader range of features with fewer parameters, thus reducing the model’s parameter count to some extent and enhancing computational efficiency. Finally, deformable convolutions excel in handling image boundaries, thereby minimizing the impact of boundary effects on recovering depth-discontinuities in GDSR tasks. Figure 6 shows the process described herein.

For general deformation convolution operations, the offsets should be learned along the x- and y-axis for each element of the feature map based on the following formula:

$$S\left(p_0\right)={\sum }_{p_{n\in R}}\omega \left(P_n\right)\ast \alpha (p_0+P_n+(∆x+∆y))$$

(6)

where $p_0$ is the feature map point on which the convolution operation is performed; $P_n$ is the standard offset of each point in the convolutional kernel relative to the center point; $\omega \left(P_n\right)$ is the weight at the corresponding position of the convolutional kernel; $\alpha (p_0+P_n)$ is the element value at position $p_0+P_n$ on the input feature map; $S\left(p_0\right)$ is the element value at position $p_0$ on the output feature map; $∆x$ and $∆y$ denote the offsets of the additionally learned convolutional kernel elements along the x- and y-axis, respectively. This method outputs sets of neighbors and the corresponding weights adaptively for each pixel. The process is illustrated in Figure 7.

After understanding the working principle of deformable convolution, we need to combine it with the GDSR task. To cover the entire feature map with the learned deformable convolutional kernel, a resample operation should be performed on the input feature map. To achieve this, we refer to the DKN approach [24].

K*K 1×1 convolutional kernels and 2K*K 1×1 convolutional kernels are used to process input features to learn the weights and offsets for each element of the assumed deformable convolutional kernel, respectively. Here, K represents the assumed deformable convolution kernel size. The two offset features are concatenated by multiplying the corresponding matrix points. The two weight features, after being activated using the ReLU function, are concatenated by multiplying the corresponding matrix points. The final weight is obtained by subtracting the mean values of RGB features and depth map features, respectively. Finally, the weight and offset features are combined to obtain the output, and the residual based on the LR depth images is calculated; then, the super-resolution depth map is obtained by adding it to the original low-resolution depth map.

Because most of the resulting offset elements were floating points, the sampled points were represented as subpixels. Therefore, bicubic interpolation was used to solve for the pixel values corresponding to the subpixel points. In Equation (6), the part representing the position is denoted as $F_g$, where j represents the sixteen points closest to subpixel point g, and $f_d$ is the input target feature. B represents a bicubic interpolation function.

$$\alpha (p_0+P_n+\left(∆x+∆y\right))=F_g={\sum }_{j\in R\left(g\right)}^{}B(g,j)f_d$$

(7)

4.2.3. Training Loss

Our training model utilizes smooth Huber loss (L1-loss). In GDSR tasks, compared to L2-loss, L1-loss has been proven to be more robust in depth-discontinuities regions [24,49], and L1-loss can significantly suppress the influence of noise. The formula is as follows: the model output is set to $D_H$; for a true HR depth map $D_{gt}$, we have

$$L\left(D_H,D_{gt}\right)={\sum }_{i=1}^N{\left|D_H-D_{gt}\right|}_1^1.$$

(8)

5. Experiments

This section describes ablation experiments conducted with diverse datasets, showcasing the proposed model’s effectiveness.

5.1. Setup

Datasets: The proposed model was trained based on recent studies [24,49]. We utilized an equal number of training samples to ensure fairness in the subsequent model comparison experiments. Of 1000 sheets selected from the NYUv2 dataset as the training set, 900 and 100 pairs were used for training and validation, respectively. The remaining 449 pairs were used as the test sets.

To verify its generalization ability, SCD-Net was tested using Middlebury and Lu datasets, the test set, and 100 pairs of RealSense data that did not participate in the training.

5.2. Experimental Details

The training samples were inputted using the original image (640 × 480). The depth images were normalized based on the maximum and minimum values in each depth image. For the color images, the standard deviation provided by ImageNet was used for normalization. The network was trained with 200 epochs, the mini-batch size was set to eight, and the Adam optimizer was employed at a learning rate of 0.0001. The proposed LRDM adhered to the DCT-Net [51] approach. HR images were down-sampled and restored to their original scale using bicubic interpolation. The processed image was used as the corresponding LRDM for the RGB image of the input model.

In the testing section, the conventional super-resolution tasks were emulated, and the root mean square error (RMSE) was used to evaluate the error between the GT and output images. The RMSE value was inversely proportional to the image quality. PyTorch was used to perform all the experiments, and an Nvidia GeForce RTX 3090 GPU was used for training and testing.

5.3. Comparison with Other Methods

SCD-Net was evaluated on the NYUv2, Middlebury, Lu, and Real-sense datasets and compared to the conventional bicubic interpolation methods and current depth map super-resolution models, such as DJF [47], DJFR [54], pixel-adaptive convolutional neural network (PAC) [55], common and unique information splitting network (CUNet) [56], and DKN [24].

To verify the superiority of the proposed model, we used ×4, ×8, ×16, and three amplification coefficients to display the output images of various models trained using the same dataset, i.e., NYUv2, as shown in Figure 6.

Table 2. Quantitative comparison between the proposed SCD-Net and existing state-of-the-art approaches on the three benchmark datasets (Bold indicates the best RMSE result).

Methods	Middlebury			NYU v2			Lu
	×4	×8	×16	×4	×8	×16	×4	×8	×16
Bicubic	4.44	7.58	11.87	8.16	14.22	22.32	5.07	9.22	14.27
DJF [47]	1.68	3.24	5.62	2.80	5.33	9.46	1.65	3.96	6.75
DJFR [54]	1.32	3.19	5.57	2.38	4.94	9.18	1.15	3.57	6.77
PAC [55]	1.32	2.62	4.58	1.89	3.33	6.78	1.20	2.33	5.19
CUNet [56]	1.10	2.17	4.33	1.92	3.70	6.78	0.91	2.23	4.99
DKN [24]	1.08	2.17	4.50	1.86	3.58	6.96	0.82	2.10	5.05
ours	1.13	2.13	4.39	1.68	3.45	6.88	0.86	1.92	4.88

shows the RMSE comparison of these models after applying the super-resolution models on different test sets. Our method outperforms traditional models significantly at the most demanding ×8 and ×16 scales, and it also surpasses recent state-of-the-art models. Compared to the parameter-heavy CUNet and PAC methods, our model achieves optimal or near-optimal results across the three datasets, demonstrating the robustness of our approach.

In order to verify the generalization ability of the model for various datasets, five ×8 scale restoration models trained on the NYUv2 dataset were selected, as shown in Figure 8. The super-resolution was performed on the Lu, Middlebury, and Real-sense images obtained using bicubic interpolation. The complicated parts of the GDSR task are marked and enlarged in red boxes. Compared to DJF and DJFR, the proposed model can restore deep discontinuous edges in depth maps. Compared to DKN, which can also recover depth discontinuities, the proposed model suppresses the RGB texture error migration (the image restored using the DKN model contains erroneous depth points). It is evident from this that the semi-coupled feature extraction module we have proposed, in combination with the CA attention-based feature processing approach, contributes to improving the model’s robustness.

5.4. Ablation Study

In order to determine the structure of the proposed semi-coupled feature extraction method, ablation experiments were performed on the number of convolutional kernels in each layer and that of residual layers. The proposed model was uniformly trained using the NYUv2 dataset. We defined i and j as the number of convolutional kernels within a layer and that of semi-coupled layers, respectively. The results are summarized in

Table 3. Results of ablation experiments on the NYUv2 test set. Bold figures indicate the best score in terms of RMSE.

Number of Kernel-Group i (j = 3)
Setting	8	16	32	64	128
×4	1.9146	1.8432	1.8247	1.6816	1.6729
×8	4.1318	3.8965	3.8424	3.4579	3.4413
×16	8.3753	7.8421	7.6025	6.8872	6.9037
Number of Kernel-Group j (i = 64)
Setting	2	3	4	5	6
×4	2.0121	1.6816	1.6732	1.6553	1.6599
×8	3.8937	3.4579	3.4599	3.4603	3.4623
×16	7.5524	6.8872	6.8551	6.8324	6.8121

.

Based on Table 3, the accuracy of the generated depth map exhibits an upward trend at higher convolutional kernels and a greater number of semi-coupling layers. However, a small accuracy improvement significantly increases the parameter. (For example, when j = 4, increasing the size of i from 64 to 128 causes the model’s parameter to change from 714,608 to 2,789,992. However, the improvement in performance is marginal.) The speed and accuracy of the model were weighed, and a 64 × 4 semi-coupled feature extraction structure was selected.

In order to verify the effectiveness of the proposed dataset, the proposed model was retrained using the Real-sense dataset. Before training, the dataset format was preprocessed to ensure consistency with NYUv2. The training parameter settings were consistent with those described in Section 4.2. The results are presented in

Table 4. Quantitative depth map SR results on the Real-sense dataset. SCD_R is SCD-Net trained on the Real-sense dataset. (Bold indicates the best RMSE result).

RMSE	DJF	DJFR	DKN	SCD	SCD_R
×4	2.66	2.23	1.75	1.58	1.47
×8	5.22	4.61	3.35	3.22	3.01
×16	9.11	8.89	6.56	6.34	6.03

and Figure 8. The performance of the model trained using the proposed dataset improved significantly, and the overall RMSE and accuracy of the object edge depth values increased.

As shown in Figure 9, we selected a pair of human-pose depth data that did not participate in the training to test the generalization ability of all models for natural scenes. Among them, _N and _R represent the models trained using the NYU and Real-sense datasets, respectively. Figure 9e indicates a slight texture error migration, and the target edge is not sufficiently flat in the area marked by the orange box. After retraining the proposed model using the Real-sense dataset, super-resolution was performed on the LR image, which returned typical depth values in this area.

6. Conclusions

This study proposes SCD-Net with semi-coupled feature extraction and deformable convolution to overcome the limitation of degradation in depth discontinuities when using traditional interpolation methods for depth map super-resolution and RGB texture over-transfer in GDSR tasks. Furthermore, we addressed the issue of limited datasets and poor quality of the current depth-map super-resolution tasks. However, the deformable convolution module needs a resampled input, which makes it difficult to migrate it to other models. At the same time, fully convolutional neural networks find it difficult to interpret the working mechanism. For GDSR datasets, existing methods are unable to establish a genuine correspondence between HR depth maps and LRDMs. This is crucial for real-world GDSR tasks. In future studies, we will improve the deformable convolution module structure to allow it to participate flexibly in other models. We will further shift the GDSR task from the spatial domain to the frequency domain to replace the fully convolutional network and enhance the interpretability of the model. To further enhance the quality of our dataset, we will delve into the principles of structured-light camera imaging. We plan to construct a collection system that combines low-resolution and high-resolution structured-light cameras in order to achieve a genuine correspondence between HR depth maps and LRDMs.