Stage-Aware Interaction Network for Point Cloud Completion

Abstract

Point cloud completion aims to restore full shapes of objects from partial scans, and a typical network pipeline is AutoEncoder, which has coarse-to-fine refinement modules. Although existing approaches using this kind of architecture achieve promising results, they usually neglect the usage of shallow geometry features in partial inputs and the fusion of multi-stage features in the upsampling process, which prevents network performances from further improving. Therefore, in this paper, we propose a new method with dense interactions between different encoding and decoding steps. First, we introduce the Decoupled Multi-head Transformer (DMT), which implements and integrates semantic prediction and resolution upsampling in a unified network module, which serves as a primary ingredient in our pipeline. Second, we propose an Encoding-aware Coarse Decoder (ECD) that compactly makes the top–down shape-decoding process interact with the bottom–up feature-encoding process to utilize both shallow and deep features of partial inputs for coarse point cloud generation. Third, we design a Stage-aware Refinement Group (SRG), which comprehensively understands local semantics from densely connected features across different decoding stages and gradually upsamples point clouds based on them. In general, the key contributions of our method are the DMT for joint semantic-resolution generation, the ECD for multi-scale feature fusion-based shape decoding, and the SRG for stage-aware shape refinement. Evaluations on two synthetic and three real-world datasets illustrate that our method achieves competitive performances compared with existing approaches.

1. Introduction

Point clouds are sets of points in 3D Cartesian coordinates. They are the typical outputs of 3D scanners such as depth cameras or LiDARs in intelligent robots and are usually used to describe the positions and discretized shapes of object surfaces. As concise and effective shape representations of objects and scenes, point clouds are widely used in various visual perception tasks [1,2]. However, real-world raw point clouds acquired by scanners are usually incomplete due to view angles, occlusions, surface materials, etc. Incomplete point clouds usually cause structural losses of target object shapes and might lead to ambiguous results in downstream tasks such as semantic segmentaion, object detection, path planning, robot grasping, etc. Therefore, with the help of recent advances in deep learning, many point cloud completion approaches elaborate networks to restore complete shapes from the learned features of partial scans [3,4,5,6,7,8,9,10,11,12].

Point cloud completion approaches usually use 2048 [13,14] or 16,384 [3,8] (i.e., multiples of 2048) points to represent each single object; in this paper, we choose to generate a dense point cloud that contains 16,384 points for high resolutions of the object shape. A typical dense point cloud completion architecture is AutoEncoder-based coarse-to-fine pipeline [3], which first generates low-resolution shape skeletons and then gradually upsamples points. For coarse shape generators, some approaches use a Multilayer Perceptron (MLP) [3,4,6] to predict flattened point coordinates from global features, while other methods [10] use transformers to derive shapes from point-wise features of sampled partial points to avoid the capacity limitation of global features when describing local details. For upsamplers, folding-based networks [3,7,11] try to let canonical grids fit the local topologies of coarse points, while some transformer-based methods [8,10] learn attention scores to aggregate point feature vectors and use them to estimate new points.

Although existing approaches achieve promising results, there is still some room for improvement. First, their coarse shapes are usually generated from the deep features of inputs [3,8,10], while the shallow features are neglected. Specifically, their decoders are directly placed behind the last layer of encoder, where the features are obtained from either global pooling or several sampling and encoding modules. Although shallow features have proven useful in semantic segmentation since they include more fine-grained local shape information [15,16], such information is not comprehensively used in point cloud completion approaches. Under such circumstances, the decoder still faces a loss of local details in its early generation steps, which can hardly be fully solved by the refinement modules (i.e., upsamplers). Second, their upsamplers are typically cascaded in one chain, where each upsampler (e.g., in the third stage) takes the features and shapes from the last upsampler in front of it (e.g., in the second stage) to generate new points. Under such circumstances, features with potential noises or errors in early stages are less representative and informative, which might influence all the stages in feedfoward network [17]. A possible solution is a feedback mechanism [17], which may require several time steps to execute the network and increases the computation complexity.

Before introducing our method, we need to revisit two targets in point cloud completion: predicting new semantics that are lost or incomplete in partial scans, and upsampling the density of sparse points to refine local details. Semantics can be defined as the meanings of components in objects, and they are used to precisely depict specific object shapes. Thus, ‘new semantics’ refers to the semantics that are unavailable or incomplete in partial scans but should be included in complete shapes. Taking the chair in Figure 1 as an example, its complete semantics are as follows: a chair back that is thicker on the top, four legs with the same thickness, a cushion covering four legs, and four continuous beams that connect neighboring legs. In a partial scan, we can find that ‘thicker on the top’, ‘covering four legs’, and ‘continuous’ are to some extent lost in the chair back, cushion and one beam, respectively, so they are the semantics that need to be predicted and reflected in the output shape.

Our method introduces three main modules to address the problems in existing approaches and realize the two targets above. First, we introduce Decoupled Multi-head Transformer (DMT), which can jointly predict new semantics and upsample local feature resolutions; DMT acts as the basic ingredient in shape-generation modules. Second, we propose an Encoding-aware Coarse Decoder (ECD), which compactly interacts the top–down shape skeleton-decoding process with the bottom–up feature-encoding process. In this way, it can predict missing semantics using the deep features that describe the observed semantics of partial inputs and further decode points after introducing the shallow features that include more detailed local information, which would also facilitate the follow-up refinement process. Third, we design a Stage-aware Refinement Group (SRG) that consists a group of upsamplers with dense connections across each other, which makes each upsampling module aware of the features and points from multiple decoding stages with different resolutions and scales. In this way, potential errors or noises in one stage can be amended after fusing the features of other stages and have less influence on the next stages.

In general, the main contributions of our method can be summarized as follows:

• A Decoupled Multi-head Transformer (DMT), which integrates semantic generation and point upsampling in one module.
• A Encoding-aware Coarse Decoder (ECD), which parallels the top–down decoding with bottom–up encoding process to generate shape skeletons based on both shallow and deep features of partial inputs.
• A Stage-aware Refinement Group (SRG), which defines a coarse-to-fine pipeline with dense feature fusion across different point decoding and upsampling stages.

Our method is evaluated on two synthetic datasets MVP [6] and PCN [3], as well as three real-world datasets, Matterport3D [18], ScanNet [2] and KITTI [1]. Both quantitative and qualitative results illustrate the competitive performances of our method compared with existing representative approaches.

2. Related Work

In this section, we will mainly discuss recent advances of learning-based generative models on 3D shapes and scenes.

2.1. Shape Generation

Three-dimensional objects can be described by different data structures, such as voxels [19], point clouds [20], and fields [21]. We will mainly discuss the last two representations. For points, pioneering approaches use MLP [20] to estimate point coordinates or folding-based networks [22] to parameterize object surfaces. Recent approaches improve generation qualities with more diversified techniques. For example, some folding-based methods substitute parameterization priors with a sphere [23] or cube [24] and use adversarial training to obtain more realistic outputs. Flow-based methods model the distribution transformations from standard Gaussian priors to points with two or more normalization flows [25,26] conditioned on shape latents. Diffusion-based methods learn the inverse diffusion process to let noise points form desired shapes, which can be applied in point cloud auto-encoding [27], image-conditioned generation [28], and shape editing [29]. For fields, the pioneering method [30] utilizes 3D convolutions in its AutoEncoder pipeline predicting distance values in voxels, which are used for mesh extraction. Recent neural field-based approaches tend to let network estimate the occupancy value of any spatial position. Such neural representation can be derived from global latent [31], multiscale latent grids [32], sampled points with features [21], or sets of vectors [33].

2.2. Point Cloud Completion

Point clouds are sets of points that represent discrete object surfaces, and point cloud completion methods typically design AutoEncoders to restore shapes from the extracted features of partial scans. The pioneering method [3] utilizes stacked PointNet [34] layers as the encoder, MLP as Coarse Decoder, and folding-based layers as the upsampler. This kind of pipeline has been widely adopted by the following methods. For encoders, some methods [4,6,8] extract global features from inputs, some methods [7,10,11] extract local features of downsampled points to achieve better results, and other methods [12] extract features from the rendered images of inputs to facilitate shape generation. For decoders, coarse shapes can be restored by MLP [3,4,6] or Transformer [7,10], and shape refinement can be realized by folding layers [7,11] or Transformer [8,10]. Besides generating the whole shapes, some methods mainly focus on the missing parts [7,11,35]. In addition, another branch of methods try to train networks without partial-complete point cloud pairs. These methods typically establish mappings between the distributions of partial and complete shapes, which can be achieved by latent transformation [36], cycle GAN [13], GAN inversion [14], energy function [37], knowledge transfer [38], and symmetric preserving [39]. Adversarial learning is widely adopted in these unsupervised methods to improve generation quality when specific ground truths are not available.

2.3. Scene-Level Completion

Besides object-level completion, scene-level completion aims to restore the whole partially scanned scenes with multiple instances. One typical topic is Semantic Scene Completion (SSC) [40], which represents indoor [40] or outdoor [41] scenes using occupancy semantic voxels and predicts them from depth images or point clouds. Some effective SSC methods include Sketch-aware Feature Embedding [42], empty voxel removal and point–voxel fusion [43], local deep implicit functions [44], the knowledge distillation with noise-free model [45], the Dual-path Transformer [46], knowledge distillation from multi-frame to single-frame [47], etc. Another recent topic in scene-level completion is Point Scene Understanding [48,49], which reconstructs certain categories of objects in scenes simultaneously. This branch of methods usually integrate object detection or instance segmentation with shape completion. Their restored objects are usually represented as meshes.

3. Method

A shape-completion example of our method is illustrated in Figure 1, and the specific completion pipeline of our method is illustrated in Figure 2. In Section 3.1, we will first introduce the Decoupled Multi-head Transformer (DMT), which acts as a basic ingredient in our network. In Section 3.2, we will discuss the Encoding-aware Coarse Decoder (ECD) that generates shape skeletons (i.e., $Y_c$ in Figure 2). Taking the first three blocks in Figure 1 as an example, when estimating points and semantics in one side of the chair, ECD queries the related points and areas (i.e., neighbors and similar shapes in the other side) from different encoding and downsampling stages. In Section 3.3, we will describe the Stage-aware Refinement Group (SRG) (i.e., the upsampling process in Figure 2). Taking the last upsampler in Figure 1 as an example, with dense connections, SRG simultaneously analyzes the shape structures represented by earlier low-resolution outputs and the more detailed locals provided by later high-resolution outputs.

3.1. Decoupled Multi-Head Transformer (DMT)

Given input points $X={\left\{x_i\right\}}_{i=1}^n$ (with point-wise features $Q={\left\{q_i\right\}}_{i=1}^n$), DMT aims to find their relationships with reference point cloud $Y={\left\{y_i\right\}}_{i=1}^m$ (with features $R={\left\{r_i\right\}}_{i=1}^m$) and fuse their features. Here, X and Y can be certain middle outputs of network or partial inputs, which will be specified in Section 3.2 and Section 3.3. The output features of DMT can have different resolutions, represented as Q, which will be used to either generate new shape parts (in Coarse Decoder) or refine current shapes (in upsamplers).

Inspired by [16], DMT uses vector attention to fuse the features of input and reference points according to their embeddings and relative locations. For each point $x_i\in X$, the original vector attention is described as

$$z_i=\sum _{y_j\in \mathcal{N}\left(x_i\right)}A(H\left(q_i\right)-K\left(r_j\right)+P(x_i-y_j))\odot (V\left(r_i\right)+P(x_i-y_j))$$

(1)

where $\mathcal{N}\left(x_i\right)$ is the k-Nearest Neighbor (kNN) of $x_i$ in reference point cloud Y. Function H maps the feature $q_i$ of input point $x_i$ to query embedding $H\left(q_i\right)$, K maps the feature $r_j$ of neighboring point $y_j\in \mathcal{N}\left(x_i\right)$ to key embedding $K\left(r_j\right)$, and V is also applied on $y_j$ to obtain value embedding $V\left(r_i\right)$. The above embeddings are based on the absolute position of points (i.e., Cartesian coordinates), so function P directly maps the relative position $x_i-y_j$ to positional embedding. Then, function A measures the relevance of $y_j$ to $x_i$ by analyzing the joint effects of ‘embedding differences on absolute positions’ and ‘embedding on relative positions’ and maps such relations to point-wise attention scores. With the Hadamard product ⊙, these scores are assigned to the value and positional embeddings of $y_j$ as weights, and the summation notation ∑ aggregates these embeddings. In this way, vector attention learns the features of a local area that lies in reference point cloud Y and is centered at query point $x_i$. All these functions are implemented by networks. Typically, there is also a residual link that adds the aggregated embeddings with input features.

However, the original vector attention learns a weighted average of local embeddings for each input point, which can hardly predict the missing parts or generate new points. To this end, as illustrated in Figure 3, we apply two independent groups of attention heads on different stages of DMT to decouple semantic prediction and resolution upsampling.

Specifically, functions H, K, V and P all map input features or coordinates to new embedding space, so different groups of these functions will be able to estimate different related semantics for each input point, and we define such group as a semantic head. After that, function A calculates the weights of local embeddings for each input, if these weights are normalized via softmax, they will serve as interpolations inside local embeddings, and different groups of A will be able to return different interpolations conditioned on specific semantics; we define these groups as resolution head. If these weights are not normalized, they will also be able to generate new shape patterns [10] like the semantic head.

In this way, it will be convenient to control the shape generation and point upsampling process by adjusting the number of semantic head $u_s$ and resolution heads $u_r$, so the final upsampling rate of a DMT module is $u_s\times u_r$. We define the function of DMT as

$$Z=\mathrm{DMT}(X,Q,Y,R;u_s,u_r)$$

(2)

Now, we summarize the inputs, parameters and outputs of function DMT according to Equation (2). X is the input point cloud; it can either be a partial scan that requires semantic prediction or middle output of the network that requires upsampling. Y and R are the reference point cloud and point-wise features, which provide information (i.e., relative positions and local features) for X. DMT then derives new feature set Z from the point-wise features Q of X. Z is used to generate new points, and its resolution is controlled by parameters $u_s$ and $u_r$. $u_s$ mainly controls the generation of new semantics (e.g., the third block in Figure 1), and $u_r$ mainly controls the upsampling within input point cloud (e.g., the seventh block in Figure 1).

3.2. Encoding-Aware Coarse Decoder (ECD)

Then we introduce the Coarse Decoder, which takes the encoded features of partial point cloud $X_p$ as inputs and generates a complete skeleton $Y_c$. Inspired by the design of U-Net [50], we stack the top–down shape generation process (i.e., Decoder) with the bottom–up feature extraction process (i.e., Encoder) compactly to make full use of features at different scales.

Specifically, as illustrated in Figure 4, we borrow the encoder of [10] that leverages two Set Abstraction (SA) modules [15] to gradually downsample $X_p$ to $X_p^1={\left\{x_{p,i}^1\right\}}_{i=1}^{n_1}$ and $X_p^2={\left\{x_{p,i}^2\right\}}_{i=1}^{n_2}$, and encode their corresponding point-wise features $Z_p^1={\left\{z_{p,i}^1\right\}}_{i=1}^{n_1}$ and $Z_p^2={\left\{z_{p,i}^2\right\}}_{i=1}^{n_2}$ using vector attention, respectively. After that, global feature vector $Z_p^3$ is obtained by applying MLP and maxpooling on $Z_p^2$. This bottom–up encoder extracts features that represent local shape geometries with different scales at different stages.

Our decoder contains two DMTs that are in parallel with the SA modules. The first DMT takes point cloud $X_p^2$ with its features as both inputs and references:

$$Z_c^2=F^s\circ {\mathrm{DMT}}^d(X_p^2,F^d(Z_p^2,Z_p^3),X_p^2,F^d(Z_p^2,Z_p^3);u_s^d,u_r^d)$$

(3)

where the superscript d of $\mathrm{DMT}$ means it analyzes deeper encoded features. Because the AutoEncoder structure of the point-cloud-completion pipeline can implicitly cluster the global features of partial shapes according to their categories [3], we wish to derive new features from $Z_p^2$ conditioned on global feature $Z_p^3$. Therefore, $F^d$ is a single mapping layer that concatenates $Z_p^2$ and $Z_p^3$ as input and outputs with fused point-wise features on $X_p^2$. According to the definition of Equation (2), the output of ${\mathrm{DMT}}^d$ is $Z_c^2={\left\{z_{c,i}^2\right\}}_{i=1}^{n_2\times u_s^d\times u_r^d}$, which contains $n_2\times u_s^d\times u_r^d$ embeddings; then, we use another single layer $F^s$ to fuse it with global feature $Z_p^3$ again and let it have the same number of dimensions as $Z_p^1$. Given that DMT assigns the same number of heads to each input query point and feature, it is intuitive that each point $x_p^2\in X_p^2$ equally derive $u_s^d\times u_r^d$ embeddings in $Z_c^2$. Therefore, we can consider that $Z_c^2$ is associated with a point cloud ${\mathbb{X}}_p^2$ that repeats $X_p^2$ by $u_s^d\times u_r^d$ times. On that basis, the second DMT further fuses $Z_c^2$ with $Z_p^1$:

$$Z_c^1={\mathrm{DMT}}^s({\mathbb{X}}_p^2,Z_c^2,X_p^1,Z_p^1;1,1)$$

(4)

We set the number of heads to 1 to ensure that the output embedding set $Z_c^1$ has the same resolution as $Z_c^2$ (i.e., $Z_c^1={\left\{z_{c,i}^1\right\}}_{i=1}^{n_2\times u_s^d\times u_r^d}$). Finally, we concatenate $Z_c^1$ with $Z_c^2$ and fuse them via a shared MLP to generate coarse point cloud features $Z_c$; then, another MLP is used to decode $Z_c$ to point cloud $Y_c$ with $n_c$ points, where $n_c=n_2\times u_s^d\times u_r^d$.

3.3. Stage-Aware Refinement Group (SRG)

The generated coarse point clouds require further refinement and upsampling for better quality and higher resolution. Therefore, we introduce SRG to gradually upsample points based on their coordinates and features at different decoding stages. In order to realize this purpose, as illustrated in Figure 2, SRG adopts a DMT-based architecture with dense connections across different refinement stages.

In general, a typical SRG contains three upsamplers, $U^1$, $U^2$, and $U^3$, with upsample rates $r_1$, $r_2$, and $r_3$, respectively. In addition to Figure 2, a summarization of input and output point clouds with the inner upsampling process (in the first upsampler as an example) is illustrated in Figure 5.

${\mathit{U}}^{\mathbf{1}}$: Given coarse point cloud $Y_c={\left\{y_{c,i}\right\}}_{i=1}^{n_c}$ with point-wise features $Z_c={\left\{z_{c,i}\right\}}_{i=1}^{n_c}$, we merge $Y_c$ with input point cloud $X_p$ and downsample it using Furthest Point Sampling (FPS). The features of points in $X_p$ are obtained by making interpolations [15] in $Z_c$. In this way, we have point cloud $Y_m^1={\left\{y_{m,i}^1\right\}}_{i=1}^{m_1}$ with features $Z_m^1={\left\{z_{m,i}^1\right\}}_{i=1}^{m_1}$. In addition, given that partial and complete point clouds usually have similar local semantics that can facilitate shape refinement (e.g., the seen leg of a partial chair might provide templates for the unseen leg), we encode the local patterns of both $Y_m$ and $X_p$ using one MLP and obtain features $Z_{ma}={\left\{z_{ma,i}\right\}}_{i=1}^{m_1}$ and $Z_{xa}={\left\{z_{xa,i}\right\}}_{i=1}^n$:

$$\begin{array}{c}{z}_{ma,i}=\mathrm{maxpool}\left\{\mathrm{MLP}(y_{c,i}-\mathcal{N}\left(y_{c,i}\right))\right\}\hfill \\ z_{xa,i}=\mathrm{maxpool}\left\{\mathrm{MLP}(x_i-\mathcal{N}\left(x_i\right))\right\}\hfill \end{array}$$

(5)

where $\mathcal{N}$ means collecting the neighbors of points using a ball query. In order to ensure that the semantics of similar local shapes in partial point cloud $X_p$ and coarse point cloud $Y_c$ are also similar, the MLPs for $Y_m^1$ and $X_p$ share weights.

After that, the features of $Y_m^1$ are further fused based on the reference points from three sources: partial inputs $X_p$, coarse points $Y_c$, and $Y_m^1$ itself:

$$\begin{array}{cc}\hfill & Z_{m,1}^1={\mathrm{DMT}}_{u,1}^1(Y_m^1,Z_{ma},X_p,Z_{xa};1,1)\hfill \\ \hfill & Z_{m,2}^1={\mathrm{DMT}}_{u,1}^1(Y_m^1,Z_m^1,Y_c,Z_c;1,1)\hfill \\ \hfill & Z_{m,3}^1={\mathrm{DMT}}_{u,3}^1(Y_m^1,Z_m^1,Y_m^1,Z_m^1;1,1)\hfill \end{array}$$

(6)

unlike other DMTs, ${\mathrm{DMT}}_{u,1}^1$ tries to find local areas in partial shapes that have similar semantics with the points in $Y_m^1$, so it queries neighbors based on the L2 distance of point features instead of coordinates. The output features $Z_{m,1}^1$, $Z_{m,2}^1$, and $Z_{m,3}^1$ all contain $m_1$ vectors, and they are generated based on point clouds and features at different stages with different shapes and resolutions. In this way, our network can comprehensively use such information by simply concatenating these features in the feature channel:

$$Z_t^1=\mathrm{cat}(Z_{m,1}^1,Z_{m,2}^1,Z_{m,3}^1)$$

(7)

and upsample them using another DMT with shared MLP:

$$Z_u^1=\mathrm{MLP}\circ {\mathrm{DMT}}_{u,4}^1(Y_m^1,Z_t^1,Y_m^1,Z_t^1;u_s^1,u_r^1)$$

(8)

In this way, we obtain upsampled features $Z_u^1={\left\{z_{u,i}\right\}}_{i=1}^{u_1}$ and use them to fuse with global feature $Z_p^3$ to generate the upsampled points $Y_u^1={\left\{y_{u,i}\right\}}_{i=1}^{u_1}$ using a shared MLP, where $u_1=u_s^1\times u_r^1\times m_1$, the order and inputs of MLP and DMT in Equation (8) are visualized by the first block of Figure 5.

${\mathit{U}}^{\mathbf{2}}$: Similar to $U_1$, we first fuse $Y_u^1$ with $X_p$ and downsample it to $Y_m^2$ (and features $Z_m^2$) with resolution $m_2$, and then we further fuse features based on three references $Y_c$, $Y_m^1$ and $Y_m^2$ itself:

$$\begin{array}{cc}\hfill & Z_{m,1}^2={\mathrm{DMT}}_{u,1}^2(Y_m^2,Z_m^2,Y_c,Z_c;1,1)\hfill \\ \hfill & Z_{m,2}^2={\mathrm{DMT}}_{u,2}^2(Y_m^2,Z_m^2,Y_m^1,Z_m^1;1,1)\hfill \\ \hfill & Z_{m,3}^2={\mathrm{DMT}}_{u,3}^2(Y_m^2,Z_m^2,Y_m^2,Z_m^2;1,1)\hfill \end{array}$$

(9)

The new features $Z_{m,1}^2$, $Z_{m,2}^2$ and $Z_{m,3}^2$ are all derived from the point cloud and features at the current stage (i.e., $Y_m^2$ and $Z_m^2$), while their reference point clouds and features come from different generation stages (i.e., two previous stages and one current stage). In order to comprehensively use them like $U^1$, we concatenate these acquired features as $Z_t^2$:

$$Z_t^2=\mathrm{cat}(Z_{m,1}^2,Z_{m,2}^2,Z_{m,3}^2)$$

(10)

We then use another DMT to upsample the resolution of $Z_t^2$ and use a shared MLP to predict the point cloud conditioned on global feature $Z_p^3$, which is similar to Equation (8) in $U^1$:

$$Z_u^2=\mathrm{MLP}\circ {\mathrm{DMT}}_{u,4}^2(Y_m^2,Z_t^2,Y_m^2,Z_t^2;u_s^2,u_r^2)$$

(11)

the upsampled points $Y_u^2={\left\{y_{u,i}\right\}}_{i=1}^{u_2}$ are generated from features $Z_u^2={\left\{z_{u,i}\right\}}_{i=1}^{u_2}$ using a shared MLP, where $u_2=u_s^2\times u_r^2\times m_2$.

${\mathit{U}}^{\mathbf{3}}$: Similar to the above upsamplers, we obtain merged points $Y_m^3$ (and features $Z_m^3$) with resolution $m_3$; fuse $Z_m^3$ with $Y_m^1$, $Y_m^2$ and $Y_m^3$ as references (using ${\mathrm{DMT}}_{u,1}^3$, ${\mathrm{DMT}}_{u,2}^3$, ${\mathrm{DMT}}_{u,3}^3$); use ${\mathrm{DMT}}_{u,4}^3$ to obtain upsampled feature $Z_u^3$ with resolution $u_3=u_s^3\times u_r^3\times m_3$; and generate the final point cloud $Y={\left\{y_i\right\}}_{i=1}^{u_3}$.

For all the three upsamplers in SRG, the raw points generated from upsampled features are actually displacements, and they are added to the input points they are derived from. This operation is commonly applied in coarse-to-fine networks [3,8,10].

3.4. Losses

Following previous methods [3,8,10], we use Chamfer Distance (CD) as training loss and evaluation metric of our network, which defines the similarity between two point clouds by calculating the lowest point distances:

$$L(X,Y)={\displaystyle \frac{1}{\left|X\right|}}\sum _{x\in X}\underset{y\in Y}{\min }\left|\right|x-{y\left|\right|}_2+\sum _{y\in Y}\underset{x\in X}{\min }\left|\right|y-{x\left|\right|}_2$$

(12)

The form of CD is borrowed from PCN [3] and VAPNet [51], where $\left|\right|y-{x\left|\right|}_2$ means the L2-Norm of $y-x$. The loss $\mathcal{L}$ of our network is the sum of CDs between the generated outputs and the ground truth G at different stages:

$$\mathcal{L}=L(Y_c,G)+L(Y_u^1,G)+L(Y_u^2,G)+L(Y,G)$$

(13)

in the first three terms of Equation (13), G is downsampled by FPS to meet the resolutions of $Y_c$, $Y_u^1$, and $Y_u^2$, respectively.

4. Experiments

In this section, we will illustrate the performances of our method on several datasets. We first specify the details and hyperparameters of the network in Section 4.1, then we implement tests on synthetic and real-world objects in Section 4.2 and Section 4.3, respectively. Finally, we discuss the effect of different modules in the network with ablation studies in Section 4.4.

4.1. Datasets and Network Details

Datasets. We train and evaluate our network on two synthetic datasets created from ShapeNet [52]: PCN [3] and MVP [6]. The PCN dataset is introduced by the pioneering approach [3] and contains eight categories of objects; it generates partial point clouds using virtual rendering on each mesh from eight (in training set) or one (in test set) view angles, and complete point clouds are uniformly sampled from meshes. MVP is another dataset that contains 16 categories objects and 26 view angles for both the training and test set. Each complete point cloud in these two datasets contains 16,384 points.

Besides synthetic datasets, we also evaluated methods on chairs and tables in real-world datasets Matterport3D [18] and ScanNet [2]. Because the objects in these two datasets are usually reconstructed from a different view angle, the raw partial point clouds are quite dense; therefore, their resolutions are downsampled to 2048 using FPS before entering the Encoder of the network, and for a fair comparison, we apply the same settings for existing methods. In addition, we test network performances on the sparse outdoor cars from KITTI [1], which are preprocessed by [3].

Network details. The details of each module are listed as follows: For DMT in Section 3.1, functions Q, K and V are all a single layer that takes d-dimensional features as inputs and outputs 64-dimensional embeddings. The positional and attentional embedding function P and A are both MLPs with output layers $\{64,64\}$ and $\{256,64\}$, respectively. Finally, another single layer transfers embedding dimensions from 64 to d. The weights in these functions could be different in different semantic and resolution heads.

For the encoder in Section 3.2, the first SA encodes input $X_p$ to $X_p^1$ with $n_1=512$ points and 128-dimensional point features $Z_p^1\in {\mathbb{R}}^{n_1\times 128}$ using MLP with layers $\{64,128\}$; then, the vector attention aggregates local features for each point from its 20 nearest neighbors. The second SA encodes $X_p^1$ to $X_p^2$ with $n_2=128$ points and 256-dimensional features $Z_p^2\in {\mathbb{R}}^{n_2\times 256}$ using layers $\{128,256\}$; then, the vector attention aggregates features from 16 neighbors. Another MLP encodes $Z_p^2$ to 512-dimensional global feature $Z_p^3$ using layers $\{512,512\}$. For the Coarse Decoder, the single-layer network $F^l$ transfers $n_2$ point feature vectors to 256 dimensions; based on them, ${\mathrm{DMT}}^d$ generates embeddings $Z_c^2\in {\mathbb{R}}^{2n_2\times 256}$ with semantic head $u_s^d=2$, and resolution head $u_r^d=1$, $F^d$ then decreases the dimensions to 128. After $Z_c^1\in {\mathbb{R}}^{2n_2\times 128}$ is achieved from ${\mathrm{DMT}}^s$ and concatenated with $Z_c^2$, three cascaded MLP blocks with output layers $\{512,256\}$, $\{128,128\}$, $\{256,128\}$ and residual links generate feature $Z_c\in {\mathbb{R}}^{2n_2\times 128}$, and then $Z_c$ is decoded to point cloud $Y_c$ using another MLP with layers $\{64,3\}$.

For SRG in Section 3.3, given coarse point cloud $Y_c$ with 256 points, we merge it with input partial point cloud and downsample it to 512 points (with corresponding features), so the upsampling rates for 16,384 output points are set to 2, 2, and 8. In this way, the heads in three upsamplers ${\mathrm{DMT}}_{u,4}^1$, ${\mathrm{DMT}}_{u,4}^2$ and ${\mathrm{DMT}}_{u,4}^3$ are $\{u_s^1,u_r^1\}=\{1,2\}$, $\{u_s^2,u_r^2\}=\{1,2\}$ and $\{u_s^3,u_r^3\}=\{2,4\}$, respectively. Inside SRG, after merging upsampled point clouds (i.e., $Y_u^1$ and $Y_u^2$) with partial input $X_p$, we downsample them to have the same resolutions as the generated ones. The points are predicted using the same MLP structures as the Coarse Decoder above, and the ball query used in Equation (5) is 0.1 for MVP and 0.05 for the PCN dataset.

Our network is trained on one Nvidia A6000 GPU using PyTorch 1.8 with a batch size 24, and Adam [53] is used as an optimizer. For the PCN dataset, the initial learning rate is 0.001 with 10 warm-up steps, and it exponentially decays by 0.1 every 150 epochs. Following [5,10], each epoch only trains one of eight partial scans for each object, so we train our network for 400 epochs. For the MVP dataset, the initial learning rate is 0.0005, and it exponentially decays by 0.1 every 50 epochs, and we train our network for 40 epochs.

4.2. Evaluation on Synthetic Datasets

We first evaluate our network on MVP and PCN datasets and compare its performances with several representative existing approaches. Following the mainstream settings, we report the CD-$l2$ (on squared point distance) and F1 score [54] on the MVP benchmark in

Table 1. Comparison of network performances on the MVP dataset (16,384 points) in terms of CD-$l2$ ($\times {10}^4$); lower is better. The bold value means the best.

Method	Air	Cab	Car	Cha	Lam	Sof	Tab	Wat	Bed	Ben	Boo	Bus	Gui	Mot	Pis	Ska	Avg
PCN [3]	2.95	4.13	3.04	7.07	14.93	5.56	7.06	6.08	12.72	5.73	6.91	2.46	1.02	3.53	3.28	2.99	6.02
TopNet [55]	2.72	4.25	3.40	7.95	17.01	6.04	7.42	6.04	11.56	5.62	8.22	2.37	1.37	3.90	3.97	2.09	6.36
MSN [56]	2.07	3.82	2.76	6.21	12.72	4.74	5.32	4.80	9.93	3.89	5.85	2.12	0.69	2.48	2.91	1.58	4.90
CRN [4]	1.59	3.64	2.60	5.24	9.02	4.42	5.45	4.26	9.56	3.67	5.34	2.23	0.79	2.23	2.86	2.13	4.30
ECG [57]	1.41	3.44	2.36	4.58	6.95	3.81	4.27	3.38	7.46	3.10	4.82	1.99	0.59	2.05	2.31	1.66	3.58
GRNet [5]	1.61	4.66	3.10	4.72	5.66	4.61	4.85	3.53	7.82	2.96	4.58	2.97	1.28	2.24	2.11	1.61	3.87
NSFA [58]	1.51	4.24	2.75	4.68	6.04	4.29	4.84	3.02	7.93	3.87	5.99	2.21	0.78	1.73	2.04	2.14	3.77
VRCNet [6]	1.15	3.20	2.14	3.58	5.57	3.58	4.17	2.47	6.90	2.76	3.45	1.78	0.59	1.52	1.83	1.57	3.12
Snowflake [8]	0.96	3.19	2.27	3.30	4.10	3.11	3.43	2.29	5.93	2.29	3.34	1.81	0.50	1.72	1.54	2.13	2.73
FBNet [17]	0.81	2.97	2.18	2.83	2.77	2.86	2.84	1.94	4.81	1.94	2.91	1.67	0.40	1.53	1.29	1.09	2.29
VAPCNet [51]	0.78	3.19	2.10	3.05	3.16	3.14	3.26	2.15	5.36	1.92	3.08	1.68	0.33	1.39	1.34	0.95	2.40
CP3 [59]	0.74	2.94	2.25	2.78	2.54	2.87	2.84	2.00	5.24	1.98	2.87	1.67	0.45	1.45	1.23	0.92	2.27
Ours	0.79	3.06	1.98	2.60	2.90	2.80	2.73	1.93	5.02	1.91	2.96	1.56	0.32	1.28	1.32	0.71	2.23

and

Table 2. Comparison of network performances on the MVP dataset (16,384 points) in terms of F1 ($\times {10}^2$); higher is better. The bold value means the best.

Method	Air	Cab	Car	Cha	Lam	Sof	Tab	Wat	Bed	Ben	Boo	Bus	Gui	Mot	Pis	Ska	Avg
PCN [3]	86.1	64.1	68.6	51.7	45.5	55.2	64.6	62.8	45.2	69.4	54.6	77.9	90.6	66.5	77.4	86.1	63.8
TopNet [55]	79.8	62.1	61.2	44.3	38.7	50.6	63.9	60.9	40.5	68.0	52.4	76.6	86.8	61.9	72.6	83.7	60.1
MSN [56]	87.9	69.2	69.3	59.9	60.4	62.7	73.0	69.6	56.9	79.7	63.7	80.6	93.5	72.8	80.9	88.5	71.0
CRN [4]	89.8	68.8	72.5	67.0	68.1	64.1	74.8	74.2	60.0	79.7	65.9	80.2	93.1	77.2	84.3	90.2	74.0
ECG [57]	90.6	68.0	71.6	68.3	73.4	65.1	76.6	75.3	64.0	82.2	70.6	80.4	94.5	78.0	83.5	89.7	75.3
GRNet [5]	86.1	64.1	68.6	51.7	45.5	55.2	64.6	62.8	45.2	69.4	54.6	77.9	90.6	66.5	77.4	86.1	63.8
NSFA [58]	90.3	69.4	72.1	73.7	78.3	70.5	81.7	79.9	68.7	84.5	74.7	81.5	93.2	81.5	85.8	89.4	78.3
VRCNet [6]	92.8	72.1	75.6	74.3	78.9	69.6	81.3	80.0	67.4	86.3	75.5	83.2	96.0	83.4	88.7	93.0	79.6
Snowflake [8]	92.0	70.0	73.0	73.0	78.0	70.0	80.0	79.0	68.0	85.0	73.0	82.0	95.0	80.0	87.0	92.0	78.2
FBNet [17]	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	82.2
VAPCNet [51]	94.2	76.2	75.8	78.6	84.4	75.9	84.5	82.4	73.6	89.2	80.8	85.4	97.8	84.5	90.2	94.4	82.9
CP3 [59]	94.0	74.0	75.0	77.0	84.0	74.0	82.0	82.0	72.0	87.0	77.0	84.0	97.0	85.0	90.0	93.0	81.4
Ours	94.9	75.6	76.7	80.0	85.5	75.1	85.3	83.3	74.2	89.6	80.5	84.6	98.0	85.4	90.8	94.4	83.3

, as well as these two metrics on different output resolutions in

Table 3. Network performances of different resolutions on the MVP dataset in terms of CD-$l2$ ($\times {10}^4$) and F1 score ($\times {10}^2$). The bold value means the best.

Points	2048		4096		8192		16,384
Metric	CD	F1	CD	F1	CD	F1	CD	F1
PCN [3]	9.77	32.0	7.96	45.8	6.99	56.3	6.02	63.8
TopNet [55]	10.11	30.8	8.20	44.0	7.00	53.3	6.36	60.1
MSN [56]	7.90	43.2	6.17	58.5	5.42	65.9	4.90	71.0
CRN [4]	7.25	43.4	5.83	56.9	4.90	68.0	4.30	74.0
ECG [57]	6.64	47.6	5.41	58.5	4.18	69.0	3.58	75.3
GRNet [5]	7.61	35.3	5.73	49.3	4.51	61.6	3.54	70.0
VRCNet [6]	5.96	49.9	4.70	63.6	3.64	72.7	3.12	79.1
PoinTr [7]	5.79	49.9	4.29	63.8	3.52	72.5	2.95	78.3
PDR [60]	5.66	49.9	4.26	64.9	3.35	75.4	2.61	81.7
Snowflake [8]	5.71	50.3	4.45	64.8	3.48	74.3	2.69	79.6
FBNet [17]	5.06	53.2	3.88	67.1	2.99	76.6	2.29	82.2
VAPCNet [51]	5.40	52.1	3.96	65.8	3.02	76.3	2.40	82.9
CP3 [59]	5.10	52.6	3.49	68.2	3.14	75.6	2.27	81.4
Ours	5.03	52.6	3.66	68.6	2.85	77.9	2.23	83.3

. For the PCN benchmark, we report the completion qualities using CD-$l1$ (on unsquared point distance) in

Table 4. Network performances on PCN (16,384 points) dataset in terms of CD-$l1$ ($\times {10}^3$); lower is better. The bold value means the best.

Method	Air	Cab	Car	Cha	Lam	Sof	Tab	Wat	Avg
FoldingNet [22]	9.49	15.80	12.61	15.55	16.41	15.97	13.65	14.99	14.31
TopNet [55]	7.61	13.31	10.90	13.82	14.44	14.78	11.22	11.12	12.15
AtlasNet [3]	6.37	11.94	10.10	12.06	12.37	12.99	10.33	10.61	10.85
PCN [3]	5.50	22.70	10.63	8.70	11.00	10.34	11.68	8.59	9.64
GRNet [55]	6.45	10.37	9.45	9.41	7.96	10.51	8.44	8.04	8.83
SK-PCN [61]	5.09	9.98	8.22	9.29	8.39	10.80	7.84	8.02	8.49
CRN [4]	4.79	9.97	8.31	9.49	8.94	10.69	7.81	8.05	8.51
PMP-Net [62]	5.65	11.24	9.64	9.51	6.95	10.83	8.72	7.25	8.73
ECG [57]	5.23	10.12	8.36	9.43	8.53	10.94	7.98	8.16	8.63
NSFA [58]	5.03	10.51	9.11	9.16	7.45	10.46	7.56	7.28	8.32
PoinTr [7]	4.75	10.47	8.68	9.39	7.75	10.39	7.78	7.29	8.38
SnowflakeNet [8]	4.29	9.16	8.08	7.89	6.07	9.23	6.55	6.40	7.21
PMP-Net++ [63]	4.39	9.96	8.53	8.09	6.06	9.82	7.17	6.52	7.56
FBNet [17]	3.99	9.05	7.90	7.38	5.82	8.85	6.35	6.18	6.94
SeedFormer [10]	3.85	9.05	8.06	7.06	5.21	8.85	6.05	5.85	6.74
ProxyFormer [11]	4.01	9.01	7.88	7.11	5.35	8.77	6.03	5.98	6.77
VAPCNet [51]	4.10	9.28	8.15	7.51	5.55	9.18	6.28	6.10	7.02
CP3 [59]	4.34	9.02	7.90	7.41	6.35	8.52	6.32	6.26	7.02
SVDFormer [12]	3.62	8.79	7.46	6.91	5.33	8.49	5.90	5.83	6.54
WalkFormer [64]	3.73	9.17	8.26	7.28	5.35	8.69	6.12	5.74	6.79
Ours	3.69	8.74	7.61	6.88	5.10	8.60	5.86	5.74	6.53

. We can find that our method is able to achieve competitive results on both datasets and reach the best in most cases.

Specifically, for quantitative evaluation, our method achieves the best average CD and F1 on these datasets with full point cloud resolution (16,384 points). The overall best CD means that the point clouds completed by our method are the closest to the ground truths. The overall best F1 means that our outputs can effectively balance shape completeness and fidelity; in other words, our method is less likely to miss the required parts of objects or excessively generate unnecessary parts. For category-level comparison, our method also reaches the best results in the most categories for both two datasets. Besides the full point cloud resolution, Table 3 illustrates that our method reaches the best results in terms of both CD and F1 when generating 8192 points and in terms of one metric when generating 2048 and 4096 points. Therefore, we can find that the advantage of our method is more obvious when the completed point clouds are denser.

For qualitative evaluation, several examples of completion results on the PCN dataset are visualized in Figure 6, and we have the following two main findings according to these examples. First, our method can reasonably infer the missing parts of objects based on their partial observations. For example, in an airplane, our method can clearly restore the unobserved fuel tank that is similar to the observed one on the other side, while other methods may either be unable to generate fuel tank or include noises. In the first lamp, only our method restores the unobserved stairs with shapes and resolutions comparable to the observed stairs. Second, our method can generate shapes more smoothly with less noise. For example, points in our car are more uniformly distributed. The vertical beams in our chair back are more complete and distinguishable. Our second lamp, sofa and tables contain less noise or unnecessary parts. In the top and front sides of our submarine, the boundaries between the observed and unobserved parts are more smoothly merged.

Besides reporting average results, Figure 7 tries to describe the variances and estimate the confidence intervals. Specifically, we collect all the completion results in terms of CD on the PCN dataset and plot their variances with respect to category-level means using histograms. The bars on the left side of zero record the numbers of outputs with CDs that are less than their means. In addition, we use different colors to represent different percentiles. Green bars represent the number of completion results whose percentiles in CDs are less than 50%, and gray bars mean those between 50% and 80%. We can define the confidence intervals around [−0.004, 0.002] since most CDs are within this range.

4.3. Evaluation on Real-World Datasets

Our method was also tested on chairs and tables from Matterport3D [18] and ScanNet [2] and was compared with existing methods [5,10,12]. We use the network weights trained on the PCN dataset in evaluation, and directly take those of baseline methods [5,10,12] from their released pre-trained models. Because there are no ground truths for these partial objects, we follow previous approaches to use the Fidelity and Minimal Matching Distance (MMD) for quantitative assessment. Fidelity applies Unidirectional Chamfer Distance (UCD) to calculate the average distance from input points to their nearest points in the output, while MMD aims to assess the quality of each output point cloud by calculating the lowest CDs between it and the instances in ShapeNet test set. The quantitative results are reported in

Table 5. Network performances on Matterport3D and ScanNet datasets in terms of Fiderlity (Fid $\times {10}^4$) and MMD ($\times {10}^3$). The bold value means the best.

Datasets	Matterport3D				SacnNet
Category	Chair		Table		Chair		Table
Metric	Fid.	MMD	Fid.	MMD	Fid.	MMD	Fid.	MMD
GRNet [5]	0.356	1.541	0.341	1.916	0.355	1.633	0.353	1.586
SeedFormer [10]	0.150	1.721	0.129	1.702	0.158	1.662	0.121	1.734
SVDFormer [12]	0.236	1.780	0.205	1.795	0.253	1.575	0.209	1.506
Ours	0.113	1.758	0.071	1.707	0.148	1.573	0.111	1.478

. In fact, it is usually not easy to achieve optimal performances in terms of Fidelity and MMD at the same time. This is because real-world partial shapes might contain noise or introduce errors when normalizing scales and poses, so high fidelity to partial inputs may bring about such errors and challenge the qualities of outputs assessed by MMD. However, our method still achieves the best performances in terms of both Fidelity and MMD in ScanNet and the best Fidelity in Matterport3D, which means that it is suitable for real-world completion.

The qualitative results are visualized in Figure 8; we can find that the two findings summarized in Section 4.2 are also valid here. Specifically, our method is able to reasonably recover the unobserved object parts. For example, our method better completes the missing right side of the second table and generates a right arm that is comparable to the left arm in the first chair. In addition, our outputs usually include less noise or unnecessary parts, such as the bottoms of the first table and two chairs.

For the KITTI dataset, given that the cars scanned by LiDAR are usually sparse, we substituted the KNN query of SA modules in our encoder with a ball query, with radii of 0.05 and 0.1 in the two SAs, respectively, while the other network parameters, including the weights of neurons (pretrained on the PCN dataset), were kept unchanged. Then, we fine-tuned and augmented the network on the cars of ShapeNet with the new query method. We also used fidelity and MMD to evaluate our method and existing ones. The quantitative and qualitative results are illustrated in

Table 6. Network performances on KITTI dataset in terms of Fidelity ($\times {10}^3$) and MMD ($\times {10}^3$). The bold value means the best.

	PCN [3]	TopNet [55]	NSFA [58]	CRN [4]	GRNet [5]	PoinTr [7]	Seed. [10]	Proxy. [11]	Walk. [64]	Ours
Fidelity	2.235	5.354	1.281	1.023	0.816	0.000	0.151	0.000	0.093	0.135
MMD	1.366	0.636	0.891	0.872	0.568	0.526	0.516	0.508	0.504	0.496

and Figure 9, respectively. According to the quantitative results in Table 6, we can find that our method achieved the best performance in terms of MMD, which means our generated cars are realistic and close to the shape distributions in the dataset. On the other hand, some methods, such as [7,11], could achieve 0 on Fidelity, which is mainly because they only generate the missing parts of objects, while our method estimates the whole shapes and does not directly take take input points as parts of outputs. Under this circumstance, our method still achieves competitive performance in terms of Fidelity, which is close to them. The qualitative results visualized in Figure 9 also prove that our method can restore cars with higher quality, even when the input partial scans are sparse. Specifically, we can easily distinguish each component of our cars, while the other cars seem ambiguous.

In addition, a possible pipeline that deals with some over-sparse inputs is first using ECD to ‘pre-complete’ the partial scans, then taking the estimated coarse points as inputs and using the whole network to complete them. We find that this pipeline could work well on partial inputs with high sparsity but may degrade the peformances on others with normal sparsity. Therefore, a future improvement could be automatically determining the sparsities of partial scans and using different pipelines in completion.

4.4. Ablation Studies

In order to experimentally prove the effects of the designs in our method, we retrained and tested the ablated versions of it on the MVP dataset. The results on the MVP dataset in terms of CD and F1 are reported in

Table 7. Ablated versions of our method and their performances on the MVP dataset.

	ECD	Con.1	Con.2	Con.3	CD	F1
V1		√	√	√	2.315	82.40
V2	√		√	√	2.268	82.74
V3	√	√		√	2.266	82.55
V4	√	√	√		2.293	83.61
Ours	√	√	√	√	2.233	83.33

. For ECD, we removed the fusion with shallow features $Z_c^2$ and only generated shapes from deep features (i.e., V1 in Table 7). For SRG, we removed the connections with previous stages in each of the three upsamplers, respectively, and kept the other two upsamplers unchanged. Specifically, V2, V3, and V4 in Table 7 represent the network with only ${\mathrm{DMT}}_{\mathrm{u},3}^1$, ${\mathrm{DMT}}_{\mathrm{u},3}^2$ and ${\mathrm{DMT}}_{\mathrm{u},3}^3$ in $U^1$, $U^2$ and $U^3$, respectively (i.e., the colored links in Figure 2 that skip the last module are removed). We can find that our pipeline with all these components achieves the best CD. Although version V4 achieves a little higher F1, the discussion below shows that it is quite unstable.

In addition, as discussed in Section 1, applying dense connections on the decoding and upsampling process may alleviate potential error accumulation compared with the cascaded pipeline. In order to experimentally prove this assumption, we take the second and the third upsampling modules (i.e., $U^2$ and $U^3$ in Section 3.3) as examples and arbitrarily add Gaussian noises on the points they need to upsample. We change the levels of noises by adjusting the standard deviation from $0.0025$ to $0.03$, and then record the performances of networks with or without dense connections (i.e., Ours and V3/V4 in Table 7). As illustrated in Figure 10, dense connections can effectively alleviate network degradations in terms of both CD and F1, and this phenomenon is more obvious when the level of noises are higher.

5. Conclusions

In this paper, we propose a new method that mainly consists of three designs for point cloud completion: DMT, ECD and SGR. DMT is designed to estimate new points in both coarse shape generation and dense shape refinement, so it decouples semantic prediction and resolution upsampling in a unified Transformer-based architecture. ECD aims to estimate coarse shapes based on both semantics and local patterns of partial inputs, so it compactly parallels top-down decoding with bottom–up feature encoding network. SGR tries to refine points with monitoring on previous upsampling processes, so it replaces the conventional cascaded modules with a densely connected pipeline to make full use of features with different scales in different stages. Experiments on both synthetic and real-world datasets illustrate the competitive performances of our method when compared with existing approaches. Therefore, we hope our proposed method can contribute to future high-level 3D perception tasks such as scene reconstruction and robot navigation.