An Efficient Bidirectional Point Pyramid Attention Network for 3D Point Cloud Completion

Abstract

Point cloud completion is a necessary task in real-world applications of recovering a complete geometry from missing regions of 3D objects. Furthermore, model efficiency is of vital importance in computer vision. In this paper, we present an efficient encoder–decoder network that predicts missing point clouds on the basis of incomplete point clouds. There are several advantages to this approach. First, a Mixed Attention Module (MAM) was implemented to obtain the correlational information of points. Second, the proposed Bidirectional Point Pyramid Attention Network (BiPPAN) can achieve simple and fast multiscale feature fusion to capture important features. Lastly, the designed encoder–decoder framework comprises skip connections to capture long-distance dependencies and structural information. We can conclude from the results of the experiments that the proposed network is an efficient and effective method to accomplish point cloud completion tasks.

1. Introduction

Three-dimensional vision is a major research area in computer vision. Recently, due to the rapid development of 3D sensing technology, 3D vision has been one of the research hotspots. Three-dimensional data can emerge in different forms, including depth images, point clouds, meshes, and volumetric grids. The point cloud is a common display form because it preserves the original geometric information in 3D space and requires less memory to store. However, point clouds obtained from laser scanners or other devices are usually incomplete, which brings some difficulties to the subsequent processing of point clouds. Therefore, as a point cloud preprocessing method, completing point clouds from missing and sparse raw data has become an important task [1,2].

Deep learning technology has dominated numerous research areas and shown significant advantages in image recognition [3], speech recognition [4], natural language processing [5] and other fields [6]. With the widespread application of 3D data, many available large 3D datasets have emerged, such as ModelNet [7] and ShapeNet [8]. At the same time, deep learning on 3D shape classification, 3D object detection and tracking, and 3D point cloud completion is receiving increasing attention [9,10].

As a pioneering work, PointNet [11] developed by Qi et al. directly takes point clouds as input and output that respect the permutation invariance of points. The major contribution is that PointNet uses a max pooling layer as a symmetric function to extract global features from all the points, and multi-layer perceptron (MLP) layers to learn pointwise features independently. It is able to carry out object classification, part segmentation, and scene semantic parsing. After PointNet, many other methods sprang up to generate complete point clouds [12,13]. The challenge of point cloud completion is to extract structural information from unordered and unstructured point cloud data. Obviously, learned features directly identify the quality of completion. Therefore, it is very important to effectively extract and exploit the features of point clouds to raise the accuracy of prediction. We set the goal of capturing the correlational information of points and selectively convey geometric information from the local regions of incomplete point clouds, so that we could reconstruct complete point clouds. The designed modules enable our model to learn structural features better, and preserve detailed information for point cloud completion to improve prediction accuracy.

In this work, we demonstrate a method to only generate the missing part of point clouds. Our main contributions are reflected in three aspects and are summed up in Figure 1.

(1)

We offer an encoder–decoder network for point cloud completion. A set of experiments proved the effectiveness and robustness of the proposed method on several challenging datasets.

(2)

We applied a Channelwise Attention Module (CAM) and a Mixed Attention Module (MAM) to introduce a reasonable weight to learn the importance of distinct features. While fusing different input features, the contributions of the fused output features are not equal. The attention mechanism allows for the network to infer the missing regions from incomplete point clouds, exploiting more effective geometric information.

(3)

We designed a simple and highly effective decoder that consists of skip connections and a Bidirectional Point Pyramid Attention Network (BiPPAN). BiPPAN applies top–down and bottom–up multiscale feature fusion. In fact, a multilayer pyramid network model was used on PF-Net [14]. However, a point cloud generation method propagating low-level features to improve the entire feature hierarchy has not yet been invented.

To verify whether the proposed network acted effectively, we evaluated our network with extensive experiments on public ShapeNet datasets of three different sizes (ShapeNet-13, ShapeNet-16, ShapeNet-55). We can conclude from the experimental results that our proposed method outperformed PF-Net as the baseline method, whether it was on large or small datasets.

The remainder of the paper is organized as follows. Section 2 demonstrates the related works. Section 3 presents our network and loss function. Section 4 describes our experiments on the ShapeNet dataset to perform completion in point clouds. Lastly, Section 5 summarizes our paper.

2. Related Work

2.1. 3D Shape Completion

In this section, we review recent work applying deep learning technology to shape completion, and then introduce the feature pyramid network and its application to point cloud completion.

In the field of shape completion, PCN [15] is the first deep learning network using point clouds without any voxelization. It presents an encoder–decoder architecture and FoldingNet to generate a dense point cloud in a coarse-to-fine fashion. Subsequently, many models have been developed to achieve high-resolution completion and strong generalization performance. By combining reinforcement learning with a generative adversarial network (GAN), RL-GAN-Net [16] and Render4Completion [17] are a novel set of architectures for 3D point cloud representation learning and generation whose reconstruction results had better robustness. Huang et al. [18] proposed a new recurrent forward network (RFNet) for point cloud completion and achieved the most advanced performance, which consisted of three separate modules: recurrent feature extraction (RFE), forward dense completion (FDC), and raw shape protection (RSP).

2.2. Multiscale Feature Representations

The feature pyramid network (FPN) is a top–down pathway to combine multiscale features. It can effectively represent multiscale features and is widely used in various tasks [19,20]. Huang et al. [14] designed a point fractal network (PF-Net) to hierarchically generate a missing point cloud that utilizes a multilayer point pyramid decoder (PPD) to predict the completion results of different layers.

3. Methods

In this part, we state the detailed information of our network. Figure 2 shows the full architecture of our method, which only predicts the missing parts from incomplete inputs. It consists of an encoder with Attention Combined Multi-Layer Perceptron (ACMLP) and a decoder with a Bidirectional Point Pyramid Attention Network (BiPPAN). The attention module was applied to the encoder and decoder as detailed in Section 3.1. The encoder shows its talent not only with excellent feature extraction, but also in the presentation of feature aggregation. We detail the point feature encoder in Section 3.2. Between the input and the decoder, skip connections connect the geometric information of the point cloud with the point features in the decoder. BiPPAN performs the prediction of missing point clouds with different resolutions. We describe the decoder in detail in Section 3.3. To optimize the network parameters, multistage completion loss was applied. The multistage completion loss was set as the Chamfer Distance (CD) error between the predicted point cloud and ground truth, which was composed of three items. We detail the loss function in Section 3.4.

3.1. Mixed Attention Module

Because of its ability to select features, the attention mechanism has been applied in a series of tasks, such as machine translation, object recognition, visual question answering and so on [21,22]. Recent studies showed that the attention mechanism has been widely used in 3D shape completion. Wen et al. [23] proposed a skip-attention network (SA-NET) for point cloud completion tasks, and designed a skipping-attention mechanism to effectively utilize the local structural details of a point cloud. Yu et al. [24] presented a novel model called PoinTr for point cloud completion that fully uses an encoder–decoder architecture with the adapted transformer blocks. Inspired by squeeze-and-excitation networks (SENets) [25] that adaptively recalibrate channelwise feature responses, we adopted channelwise and pointwise attention in our encoder–decoder architecture to conquer the problem of point cloud completion. The structure of the mixed attention module is illustrated in Figure 3.

Channelwise attention: Given a point feature matrix $\mathrm{z}$ with the size of $\mathrm{C}\times \mathrm{L}$, we applied channelwise average pooling to obtain a global feature vector with the size of $\mathrm{C}\times 1$. Then, we operated two fully connected (FC) layers around ReLU activation units. The layers were able to fully capture channelwise dependencies.

$$\mathrm{C}\mathrm{A}=\mathrm{F}(\mathrm{z},\mathrm{W})={\mathrm{W}}_2\mathsf{\delta }\left({\mathrm{W}}_1\mathrm{p}\mathrm{o}\mathrm{o}\mathrm{l}\right(\mathrm{z}\left)\right)$$

(1)

where the $\mathrm{p}\mathrm{o}\mathrm{o}\mathrm{l}$ denotes the global average pooling, ${\mathrm{W}}_1\in {\mathbb{R}}^{\mathrm{C}/16\times \mathrm{C}}$ and ${\mathrm{W}}_2\in {\mathbb{R}}^{\mathrm{C}\times \mathrm{C}/16}$ denote parameters in the two FC layers, and $\mathsf{\delta }$ refers to the ReLU activation units. $\mathrm{C}\mathrm{A}\in {\mathbb{R}}^{\mathrm{C}\times 1}$ is the channelwise attention of $\mathrm{z}$.

Pointwise attention: Similar to the strategy for predicting the channelwise attention for a point feature matrix $\mathrm{z}\in {\mathbb{R}}^{\mathrm{C}\times \mathrm{L}}$, a pointwise average pooling operation was performed to aggregate the pointwise feature vector with the size of 1 × L. Then, we computed $\mathrm{P}\mathrm{A}={\mathrm{F}}^{\prime }(\mathrm{z},{\mathrm{W}}^{\prime })={\mathrm{W}}_2^{\prime }\mathsf{\delta }\left({\mathrm{W}}_1^{\prime }\mathrm{p}\mathrm{o}\mathrm{o}\mathrm{l}\right(\mathrm{z}\left)\right)$. The weight parameters of two fully connected layers are ${\mathrm{W}}_1^{\prime }\in {\mathbb{R}}^{\mathrm{L}/16\times \mathrm{L}}$ and ${\mathrm{W}}_2^{\prime }\in {\mathbb{R}}^{\mathrm{L}\times \mathrm{L}/16}$. $\mathrm{P}\mathrm{A}\in {\mathbb{R}}^{1\times \mathrm{L}}$ is the pointwise attention of $\mathrm{z}$.

Lastly, the output feature vector $\tilde{\mathrm{z}}$ of the mixed attention module was obtained by weighting the feature $\mathrm{z}$ with CA and PA as follows. Using the sigmoid function $\mathsf{\sigma }$, the attention matrix was normalized to [0,1].

$$\tilde{\mathrm{z}}=\mathsf{\sigma }(\mathrm{C}\mathrm{A}\times \mathrm{P}\mathrm{A})\times \mathrm{z}+\mathrm{z}$$

(2)

Through the operations above, the feature representation $\tilde{\mathrm{z}}$ could learn discrimination and robust representation for point clouds.

3.2. Point Feature Encoder

The size of the incomplete point cloud input was N = 2048 with its 3-dimensional coordinates. The encoder network was utilized to extract features from the incomplete input. Motivated by PointNet, we designed an Attention Combined Multi-Layer Perceptron (ACMLP) as our point cloud feature encoder. As shown in Figure 4, the structure of the encoder comprised a shared multi-layer perceptron (MLP) with LeakyReLU activation, which consisted of five layers with neuron sizes 64, 128, 256, 512, and 1024. Different dimensions of MLP can extract low, mid-, and high-level features, each of which contains rich point cloud information. In order to utilize these features effectively, we used max pooling to obtain the global latent representation $f_i$ of the output of the last four layers, whose size was $f_i:=128,256,512,1024,$ for i = 1,...,4, and concatenate $f_i$ to produce a 1920-dimensional feature vector. Then, we adopted a channelwise attention module for the 1920-dimensional features to obtain the weighted multilevel features $\tilde{f}$ that contained low-, mid-, and high-level feature information, and could help in generating effective features for point cloud prediction. Since we used max pooling for the 1920-dimensional feature vector, we did not take the pooling layer into the channelwise attention module. Eventually, we generated the final feature vector ${\mathrm{F}}_1$ through a fully connected layer.

3.3. Point Pyramid Decoder

The decoder structure is composed of two modules: skip connections and Bidirectional Point Pyramid Attention Network (BiPPAN).

As shown in Figure 2, we concatenated the input incomplete point clouds with the ${\mathrm{F}}_1-{\mathrm{F}}_3$ via the long-range skip connections. We obtained ${\mathrm{F}}_2$ and ${\mathrm{F}}_3$ by passing ${\mathrm{F}}_1$ through fully connected layers. We also employed the mixed attention module to fuse these features and generate ${\mathrm{F}}_1^{\prime }-{\mathrm{F}}_3^{\prime }$ as the input of BiPPAN. There are two benefits to such skip connections. One is to provide long-range information compensation, so the raw incomplete point cloud geometry information is still available in the decoder architecture. The other is that residual learning can facilitate gradient backpropagation.

BiPPAN is based on the feature pyramid network (FPN) that contains a top–down pathway to aggregate multiscale features from Level 1 to 3 (${\mathrm{F}}_1^{\prime }=512\times 3$, ${\mathrm{F}}_2^{\prime }=256\times 3$ and ${\mathrm{F}}_3^{\prime }=128\times 3$), as shown in Figure 5a. Levels 1–3 represent low-to-high levels. The conventional FPN fuses multiscale features in a top–down manner. Formally,

$${\mathrm{F}}_3^{\mathrm{o}\mathrm{u}\mathrm{t}}=\mathrm{C}\mathrm{o}\mathrm{n}\mathrm{v}\left({\mathrm{F}}_3^{\prime }\right)$$

(3)

$${\mathrm{F}}_2^{\mathrm{o}\mathrm{u}\mathrm{t}}=\mathrm{C}\mathrm{o}\mathrm{n}\mathrm{v}({\mathrm{F}}_2^{\prime }+\mathrm{R}\mathrm{e}\mathrm{s}\mathrm{i}\mathrm{z}\mathrm{e}({\mathrm{F}}_3^{\mathrm{o}\mathrm{u}\mathrm{t}}\left)\right)$$

(4)

$${\mathrm{F}}_1^{\mathrm{o}\mathrm{u}\mathrm{t}}=\mathrm{C}\mathrm{o}\mathrm{n}\mathrm{v}({\mathrm{F}}_1^{\prime }+\mathrm{R}\mathrm{e}\mathrm{s}\mathrm{i}\mathrm{z}\mathrm{e}({\mathrm{F}}_2^{\mathrm{o}\mathrm{u}\mathrm{t}}\left)\right)$$

(5)

where $\mathrm{R}\mathrm{e}\mathrm{s}\mathrm{i}\mathrm{z}\mathrm{e}$ is an upsampling/downsampling or reshape operation for size matching, and $\mathrm{C}\mathrm{o}\mathrm{n}\mathrm{v}$ is a convolutional operation. ${\mathrm{F}}_1^{\mathrm{o}\mathrm{u}\mathrm{t}}$, ${\mathrm{F}}_2^{\mathrm{o}\mathrm{u}\mathrm{t}}$ and ${\mathrm{F}}_3^{\mathrm{o}\mathrm{u}\mathrm{t}}$ (${\mathrm{F}}_1^{\mathrm{o}\mathrm{u}\mathrm{t}}=512\times 3$, ${\mathrm{F}}_2^{\mathrm{o}\mathrm{u}\mathrm{t}}=128\times 3$ and ${\mathrm{F}}_3^{\mathrm{o}\mathrm{u}\mathrm{t}}=64\times 3$) are the predicted missing regions of the incomplete point cloud in different resolutions.

Following this idea, we designed a bidirectional point pyramid attention network to optimize multiscale feature fusion on top of FPN, as depicted in Figure 5b. First, we added not only a bottom–up path aggregation network, but also cross-level connections. By fully fusing high-level features with rich global information and low-level features with many fine details of local geometry information, we could predict the detailed structure of the missing point cloud. Second, we introduced the mixed attention module to learn the importance of different input features whose purpose is to selectively aggregate the features.

3.4. Loss Function

The loss function in our method is multistage completion loss ${\mathrm{L}}_{\mathrm{c}\mathrm{o}\mathrm{m}}$. Because we set the pyramid level N = 3 in our network, the multistage completion loss ${\mathrm{L}}_{\mathrm{c}\mathrm{o}\mathrm{m}}$ was composed of three parts (${\mathrm{d}}_{\mathrm{C}\mathrm{D}1}$, ${\mathrm{d}}_{\mathrm{C}\mathrm{D}2}$, ${\mathrm{d}}_{\mathrm{C}\mathrm{D}3}$):

$${\mathrm{L}}_{\mathrm{c}\mathrm{o}\mathrm{m}}={\mathrm{d}}_{\mathrm{C}\mathrm{D}1}({\mathrm{F}}_1^{\mathrm{o}\mathrm{u}\mathrm{t}},{\mathrm{F}}_{\mathrm{g}\mathrm{t}})+{\mathsf{\alpha }\times \mathrm{d}}_{\mathrm{C}\mathrm{D}2}({\mathrm{F}}_2^{\mathrm{o}\mathrm{u}\mathrm{t}},{\mathrm{F}}_{\mathrm{g}\mathrm{t}}^{\prime })+2\mathsf{\alpha }\times {\mathrm{d}}_{\mathrm{C}\mathrm{D}3}({\mathrm{F}}_3^{\mathrm{o}\mathrm{u}\mathrm{t}},{\mathrm{F}}_{\mathrm{g}\mathrm{t}}^{″})$$

(6)

where $\mathsf{\alpha }$ is the hyperparameter, and ${\mathrm{d}}_{\mathrm{C}\mathrm{D}}$ is the Chamfer Distance proposed by Fan [26]. The mean Chamfer Distance can measure the average nearest squared distance between prediction point cloud ${\mathrm{F}}^{\mathrm{o}\mathrm{u}\mathrm{t}}$ and ground truth ${\mathrm{F}}_{\mathrm{g}\mathrm{t}}$, which is calculated with:

$${\mathrm{d}}_{\mathrm{C}\mathrm{D}}({\mathrm{F}}^{\mathrm{o}\mathrm{u}\mathrm{t}},{\mathrm{F}}_{\mathrm{g}\mathrm{t}})=\frac{1}{{\mathrm{F}}^{\mathrm{o}\mathrm{u}\mathrm{t}}}{\sum }_{\mathrm{x}\in {\mathrm{F}}^{\mathrm{o}\mathrm{u}\mathrm{t}}}\underset{\mathrm{y}\in {\mathrm{F}}_{\mathrm{gt}}}{\min }{‖\mathrm{x}-\mathrm{y}‖}_2^2+\frac{1}{{\mathrm{F}}_{\mathrm{g}\mathrm{t}}}{\sum }_{\mathrm{y}\in {\mathrm{F}}_{\mathrm{g}\mathrm{t}}}\underset{\mathrm{x}\in {\mathrm{F}}^{\mathrm{out}}}{\min }{‖\mathrm{y}-\mathrm{x}‖}_2^2$$

(7)

In detail, ${\mathrm{d}}_{\mathrm{C}\mathrm{D}1}$ calculates the squared distance between predicted detailed points ${\mathrm{F}}_1^{\mathrm{o}\mathrm{u}\mathrm{t}}$ and the ground truth of the missing region ${\mathrm{F}}_{\mathrm{g}\mathrm{t}}$. Items ${\mathrm{d}}_{\mathrm{C}\mathrm{D}2}$ and ${\mathrm{d}}_{\mathrm{C}\mathrm{D}3}$ calculate the squared distance among secondary central points ${\mathrm{F}}_2^{\mathrm{o}\mathrm{u}\mathrm{t}}$, predicted primary central points, ${\mathrm{F}}_3^{\mathrm{o}\mathrm{u}\mathrm{t}}$ and ground truth ${\mathrm{F}}_{\mathrm{g}\mathrm{t}}^{\prime }$, ${\mathrm{F}}_{\mathrm{g}\mathrm{t}}^{″}$.

4. Experimental

In this part, we present the three benchmarks for point cloud completion and the implementation details of our network. Then, we show the results of our method and some other baseline methods on the benchmark. Lastly, we provide an ablation study and robustness test of our network. We evaluate how our model works both quantitatively and qualitatively.

4.1. Data Generation and Model Training

We tested our method on a commonly used benchmark for 3D point cloud completion, i.e., ShapeNet. We used 13 and 16 categories of different objects in the benchmark dataset ShapeNet-Part [27]. Since a more diverse dataset can comprehensively test the capabilities of the model, we also conducted experiments using all 55 categories in ShapeNet. Similar to the dataset of PoinTr, we randomly sampled 80% of the objects from each category of ShapeNet-55 to form the training set, and 20% for evaluation, resulting in 41,952 models for training and 10,518 models for testing. Complete point clouds were generated by uniformly sampling 2048 points from 5 randomly distributed viewpoints. During training and testing, we sampled 512 points as the ground truth ${\mathrm{F}}_{\mathrm{g}\mathrm{t}}$ of the incomplete point cloud, that is, 25% of the original data were missing. We applied iterative farthest point sampling (IFPS) [28] to extract ground truths ${\mathrm{F}}_{\mathrm{g}\mathrm{t}}^{\prime }$ and ${\mathrm{F}}_{\mathrm{g}\mathrm{t}}^{″}$ with 128 and 64 points, respectively.

We implemented all of our models with the PyTorch deep learning framework, and used the Adam optimizer to update the parameters of the network during training. The learning rate was initially set to 0.0001, the model was trained for 200 epochs, and the continuous learning rate decreased by 0.2 every 40 epochs. We defined the batch size to be 24. The models were trained and tested on a NVIDIA GeForce RTX 3090 GPU with CUDA 11.7. The mixed attention module can be used flexibly in convolutional neural networks (CNNs) to capture feature correlations effectively, and the bidirectional point pyramid attention network (BiPPAN) can be used as a decoder in the network.

4.2. Completion Results on ShapeNet

We applied the mean Chamfer Distance as the evaluation metric, which contained two items in our experiments:$\mathrm{P}\mathrm{r}\mathrm{e}\mathrm{d}\to \mathrm{G}\mathrm{T}$ and $\mathrm{G}\mathrm{T}\to \mathrm{P}\mathrm{r}\mathrm{e}\mathrm{d}$.

The Chamfer Distance from a predicted point cloud to the ground truth ($\mathrm{P}\mathrm{r}\mathrm{e}\mathrm{d}\to \mathrm{G}\mathrm{T}$) measures the difference between the predicted and real point clouds, which is calculated with:

$${\mathrm{d}}_{\mathrm{C}\mathrm{D}}({\mathrm{F}}_{\mathrm{P}\mathrm{r}\mathrm{e}\mathrm{d}},{\mathrm{F}}_{\mathrm{G}\mathrm{T}})=\frac{1}{{\mathrm{F}}_{\mathrm{P}\mathrm{r}\mathrm{e}\mathrm{d}}}{\sum }_{\mathrm{x}\in {\mathrm{F}}_{\mathrm{P}\mathrm{r}\mathrm{e}\mathrm{d}}}\underset{\mathrm{y}\in {\mathrm{F}}_{\mathrm{GT}}}{\min }{‖\mathrm{x}-\mathrm{y}‖}_2^2$$

(8)

The Chamfer Distance from the ground truth to a predicted point cloud ($\mathrm{G}\mathrm{T}\to \mathrm{P}\mathrm{r}\mathrm{e}\mathrm{d}$) manifests the extent to which the shape of the prediction covers the real point cloud, which is calculated with:

$${\mathrm{d}}_{\mathrm{C}\mathrm{D}}({\mathrm{F}}_{\mathrm{G}\mathrm{T}},{\mathrm{F}}_{\mathrm{P}\mathrm{r}\mathrm{e}\mathrm{d}})=\frac{1}{{\mathrm{F}}_{\mathrm{G}\mathrm{T}}}{\sum }_{\mathrm{x}\in {\mathrm{F}}_{\mathrm{G}\mathrm{T}}}\underset{\mathrm{y}\in {\mathrm{F}}_{\mathrm{Pred}}}{\min }{‖\mathrm{x}-\mathrm{y}‖}_2^2$$

(9)

Results on ShapeNet-13: We compared the results of our model with those of L-GAN [29], PCN [15], 3D-point Capsule Networks [30] and PF-Net [14] on standard metrics. In

Table 1. Results of a comparison between our method and most advanced methods on the ShapeNet-13 dataset. The numbers in pairs are Pred → GT and GT → Pred Chamfer Distance × 10³ (lower is better).

Category	LGAN-AE	PCN	3D Capsule	PF-Net	BiPPAN
Airplane	3.357/1.130	5.060/1.243	2.676/1.401	1.091/1.070	0.964/0.875
Bag	5.707/5.303	3.251/4.314	5.228/4.202	3.929/3.768	2.864/2.823
Cap	8.968/4.608	7.015/4.240	11.04/4.739	5.290/4.800	4.371/3.489
Car	4.531/2.518	2.741/2.123	5.944/3.508	2.489/1.839	2.28/1.681
Chair	7.359/2.339	3.952/2.301	3.049/2.207	2.074/1.824	1.758/1.442
Guitar	0.838/0.536	1.419/0.689	0.625/0.662	0.456/0.429	0.377/0.376
Lamp	8.464/3.627	11.61/7.139	9.912/5.847	5.122/3.460	3.731/2.513
Laptop	7.649/1.413	3.070/1.422	2.129/1.733	1.247/0.997	1.118/0.864
Motorbike	4.914/2.036	4.962/1.922	8.617/2.708	2.206/1.775	1.931/1.562
Mug	6.139/4.735	3.590/3.591	5.155/5.168	3.138/3.238	3.036/2.914
Pistol	3.944/1.424	4.484/1.414	5.980/1.782	1.122/1.055	0.929/0.831
Skateboard	5.613/1.683	3.025/1.740	11.49/2.044	1.136/1.337	1.007/1.029
Table	2.658/2.484	2.503/2.452	3.929/3.098	2.235/1.934	1.744/1.626
Mean	5.395/2.603	4.360/2.661	5.829/3.008	2.426/2.117	2.008/1.694

, the results of the above four methods are cited from [14]. Our method surpassed the other ones on 13 categories in terms of $\mathrm{P}\mathrm{r}\mathrm{e}\mathrm{d}\to \mathrm{G}\mathrm{T}$ and $\mathrm{G}\mathrm{T}\to \mathrm{P}\mathrm{r}\mathrm{e}\mathrm{d}$ errors, and achieved the best average Chamfer Distance in the missing region. This suggests that our network was able to reconstruct the missing point cloud with higher precision.

Results on ShapeNet-16: We experimented with our method and the most advanced PF-Net method on ShapeNet-16. Since PF-Net was only trained on the ShapeNet-13 dataset, we trained and tested it again using the same dataset according to their open-source code for fair comparison and quantitative analysis.

Table 2. Results of a comparison between our method and PF-Net on ShapeNet-16 dataset. We also provide the number of parameter (Params) in the last rows.

Category	PF-Net		BiPPAN
	Pred → GT	GT → Pred	Pred → GT	GT → Pred
Airplane	1.084	1.119	0.963	0.872
Bag	3.979	4.668	2.83	2.624
Cap	5.254	4.897	4.004	3.614
Car	2.548	1.914	2.316	1.634
Chair	2.154	2.019	1.738	1.428
Earphone	6.003	8.058	3.194	3.690
Guitar	0.464	0.546	0.377	0.382
Knife	0.555	0.563	0.464	0.465
Lamp	4.943	3.883	3.863	2.557
Laptop	1.309	1.072	1.135	0.888
Motorbike	2.328	1.836	1.944	1.581
Mug	3.080	3.580	3.147	2.893
Pistol	1.284	1.053	0.920	0.811
Rocket	1.052	0.762	0.853	0.620
Skateboard	1.196	1.362	1.025	1.004
Table	2.305	2.123	1.760	1.629
Mean	2.471	2.466	1.908	1.668
Params (×106)	76.571		11.763

shows that our method resulted in a lower average Chamfer Distance error when increasing the dataset from 13 to 16 categories. The three new categories, namely, Earphone, Knife, Rocket, on both $\mathrm{P}\mathrm{r}\mathrm{e}\mathrm{d}\to \mathrm{G}\mathrm{T}$ and $\mathrm{G}\mathrm{T}\to \mathrm{P}\mathrm{r}\mathrm{e}\mathrm{d}$ errors were all smaller than those of PF-Net. These results justify that the proposed network could handle a larger dataset and effectively reconstruct the full shape. Figure 6 shows the qualitative results of our method and PF-Net on ShapeNet-16. Our method preserved the original input geometry while better computing and refining the missing parts of the completion for the shown examples.

Furthermore, we compared the number of trainable parameters in PF-Net and our method. As shown in Table 2, our method performed significantly better with fewer parameters than those of PF-Net.

Results on ShapeNet-55: To further investigate how PF-Net and our method performed with more dataset categories, we conducted experiments on ShapeNet-55. Due to the imbalanced number of shapes in the dataset, we classified the 55 categories into simple, moderate, and hard classes on the basis of their number. Specifically, more than 2500 training shapes were considered to be of the simple class, such as table, chair, and airplane. Fewer than 80 samples were classified as of the hard class, such as birdhouse, bag, and keyboard. Bed, camera, and rifle were classified as of the moderate class. We chose 3 categories from each class as examples to demonstrate the results. The average CD results for the three classes are shown in

Table 3. Results of a comparison between our method and PF-Net on the ShapeNet-55 dataset. The numbers in pairs are Pred → GT and GT → Pred Chamfer Distance × 10³ (lower is better). CD-S, CD-M, and CD-H were used to represent CD errors in the simple, moderate, and hard classes.

	Simple			Moderate			Hard			CD-S	CD-M	CD-H	CD- Ave
	Table	Chair	Airplane	Bed	Camera	Rifle	Birdhouse	Bag	Keyboard
PF-Net	1.963 /1.882	2.101 /1.975	1.162 /1.414	3.327 /4.742	4.827 /5.613	1.235 /1.029	3.689 /3.650	3.896 /3.107	1.911 /1.006	1.845 /1.737	2.464 /2.142	2.744 /2.362	2.463 /2.150
BiPPAN	1.386 /1.156	1.591 /1.177	0.884 /0.874	2.741 /1.987	3.258 /2.321	0.541 /0.488	2.015 /2.770	1.889 /1.289	0.766 /0.594	1.449 /1.134	1.801 /1.301	1.647 /1.245	1.738 /1.274

with details. The category errors and average errors of our method were smaller than those of PF-Net. The results of our method on ShapeNet-55 were especially better than those on ShapeNet-13 and ShapeNet-16. Due to the imbalanced number of training samples, the average CD errors of the moderate and hard classes with insufficient samples for PF-Net were higher than those of the simple class. For our BiPPAN, there were no significant differences in the average errors of the three classes. In Figure 7, we present the qualitative results for 9 categories on Shapenet-55. As shown in the example, our method was able to complete the missing point cloud with higher accuracy and more details for the various incomplete categories.

Regardless of quantitative comparison or qualitative analysis, our method could improve the quality of the completed point cloud on a diverse dataset and achieve good results in the point cloud completion task.

4.3. Ablation Study

To test the effectiveness and necessity of the three designed modules of the Attention Mechanism (AM, including the Channelwise Attention Module of the encoder and the Mixed Attention Module of the decoder), the Bidirectional Point Pyramid Network (BiPPN), and skip connections (Skip-C), we exhaustively compared and tested our method BiPPAN with its four variants on ShapeNet-16. The results of the ablation study are shown in

Table 4. Ablation study on different components of our proposed encoder–decoder network framework, including Attention Module (AM), Bidirectional Point Pyramid Network (BiPPN), and skip connections (Skip-C).

Model	AM	BiPPN	Skip-C	Pred → GT (×103)	GT → Pred (×103)
A				2.498	2.300
B		√	√	2.125	1.869
C	√		√	2.242	1.837
D	√	√		2.004	2.011
BiPPAN	√	√	√	1.908	1.668

.

Model A is a very basic baseline that includes the standard CMLP and FPN as the encoder and decoder architectures, respectively. Models B, C, D had the same structure as that of BiPPAN (our entire network) except for the removed/replaced module. Model B was the variant where AM was removed from the BiPPAN model. Adding AM reduced $\mathrm{P}\mathrm{r}\mathrm{e}\mathrm{d}\to \mathrm{G}\mathrm{T}$ and $\mathrm{G}\mathrm{T}\to \mathrm{P}\mathrm{r}\mathrm{e}\mathrm{d}$ errors. This confirmed the effectiveness of AM. Model C focused on demonstrating the effectiveness of introducing BiPPN into network design. After changing the decoder from FPN into BiPPN, $\mathrm{P}\mathrm{r}\mathrm{e}\mathrm{d}\to \mathrm{G}\mathrm{T}$ and $\mathrm{G}\mathrm{T}\to \mathrm{P}\mathrm{r}\mathrm{e}\mathrm{d}$ errors were smaller, as expected. When removing Skip-C from the BiPPAN model, the $\mathrm{G}\mathrm{T}\to \mathrm{P}\mathrm{r}\mathrm{e}\mathrm{d}$ error of Model D increased markedly. This clearly shows that Skip-C alone can bring significant performance improvement. Notably, our BiPPAN method performed better than its variants. A visual comparison is presented in Figure 8. For the shown examples, Baseline Model A and Models B, C, D suffered from some obvious fairness coarsening and exhibited impotence for most of the details. This further confirmed the effectiveness of the BiPPAN design.

4.4. Robustness Test

In order to analyze the robustness of our method, we evaluated the point cloud completion of missing regions in different degrees on ShapeNet-16. We varied the number of incomplete point clouds from 512 to 1024 and 1536. In other words, we extended the missing part from 25% to 50% and 75% of the original shape.

Table 5. Robustness results of different extent of incomplete input point cloud using our method.

Missing Ratio	25%		50%		75%
CD (×103)	Pred → GT	GT → Pred	Pred → GT	GT → Pred	Pred → GT	GT → Pred
Airplane	0.963	0.872	0.971	0.809	0.968	0.896
Car	2.316	1.634	2.494	1.786	2.479	1.841

and Figure 9 show the experimental results of the network on the Airplane and Car categories. Table 5 shows that the errors between the predicted point cloud and the ground truth did not differ much when 25%, 50%, and 75% of the points were missing. Figure 9 displays that, although the partial data were quite tough, our predictions were sensible. In the case of the 75% missing area, the complete car shape could be defined and completed only on the basis of the front of the car. This experiment verified the strong completion robustness of our method when dealing with input point clouds with distinct missing degrees.

5. Conclusions

In this paper, we recommended a novel Bidirectional Point Pyramid Attention Network framework (BiPPAN) for point cloud completion. Our method produces missing point clouds via an encoder–decoder structure. The attention module was introduced into the encoder and decoder to raise the correlational information of point clouds, which extracts the local details of object features better [31]. The point pyramid attention decoder was designed not only to fully use hierarchical features from the bidirectional connection, but also to capture long-distance structural information by employing skip connections as well [32]. Experimental results on ShapeNet-13, ShapeNet-16, and ShapeNet-55 demonstrate that, no matter the size of the dataset, our proposed method was effective and efficient in comparison with other methods. An ablation and robustness test validated our claim that the attention module, bidirectional connection, and skip connections could more effectively extract and exploit features of point clouds to improve prediction accuracy. Our method tended to reduce the error in challenging classes.

Hence, this method can be widely applied to the task of generating target point clouds with both rich semantic profiles and detailed features while preserving existing contours. The good results of BiPPAN on many datasets demonstrate its ability to repair and complete 3D shapes, and improve the accuracy of downstream tasks such as 3D recognition, and 3D object detection and tracking. As a point cloud preprocessing method, our method has tremendous potential in autonomous vehicles, 3D reconstruction, and remote sensing research.