Group-in-Group Relation-Based Transformer for 3D Point Cloud Learning

Abstract

Deep point cloud neural networks have achieved promising performance in remote sensing applications, and the prevalence of Transformer in natural language processing and computer vision is in stark contrast to underexplored point-based methods. In this paper, we propose an effective transformer-based network for point cloud learning. To better learn global and local information, we propose a group-in-group relation-based transformer architecture to learn the relationships between point groups to model global information and between points within each group to model local semantic information. To further enhance the local feature representation, we propose a Radius Feature Abstraction (RFA) module to extract radius-based density features characterizing the sparsity of local point clouds. Extensive evaluation on public benchmark datasets demonstrate the effectiveness and competitive performance of our proposed method on point cloud classification and part segmentation.

1. Introduction

Point cloud processing tasks have attracted much attention due to their wide application in remote sensing [1,2], autonomous driving [3], robotics [4], and augmented reality [5]. Unlike 2D images, point clouds are unordered, irregular, and sparse. Therefore, designing an effective approach for point cloud learning is an important yet challenging task.

Currently, with the continuous development of deep learning, there has been much work trying to tackle this challenge [6,7,8], and existing methods can be divided into three categories. The first category relies on the successful application of convolutional neural networks in regular grids. Some of them transform unordered point clouds into regular voxels [9] or multi-view images [10] and then extract features by sparse 3D and 2D convolutions, respectively. Although these methods achieve promising results, they inevitably cause the loss of important geometric information during the transform process, and their high computational cost and memory consumption further hinder the performance improvement. The second category is processing the raw point cloud data directly, thus avoiding the information loss in transforming the point cloud into a regular data structure. The pioneering work PointNet [11] learns point-based global information by a shared multi-layer perceptron (MLP) and global aggregation. Some of its subsequent work [12] uses local aggregation mechanisms and hierarchical structures to further improve the network performance, and others [13,14,15] learn deeper semantic structure information by defining new specific convolution-like operations or constructing graph structures [16,17]. However, due to the irregularity of point clouds, point-based methods must satisfy permutation invariance. Therefore, it is unavoidable to use symmetric functions (e.g., pooling functions), which limits their feature representation capability to some extent [18], and neither of them is sufficient enough to capture the local relations of the points. The third category is the hybrid-data based methods, which attempt to combine voxel-based and point-based approaches. For example, two studies [19,20] integrate voxel features and PointNet features at the scene level. However, as these two point cloud representations are fundamentally different, the current methods in this category are not very effective in extracting point cloud features.

In order to develop a more efficient point cloud learning network, PCT [21] recently introduced Transformer to point cloud processing tasks. Transformer has been extremely successful in natural language processing [22,23] and computer vision [24]. Transformer is not affected by the order of the input sequence and thus satisfies the permutation invariance requirement of point clouds. PCT focuses on using offset-attention layers to extract global features. Although PCT utilizes the neighbor feature embedding in EdgeConv [17], it still cannot learn the relationship within local point clouds effectively to model local geometric features, and thus the extraction of semantic features for local point clouds is not sufficient.

Inspired by transformer-in-transformer (TNT) [25], we propose a point cloud feature extraction network that efficiently learns both global and local information through a group-in-group relation-based Transformer architecture (GiG). Specifically, we divide a point cloud into a series of groups, and use an inner-group Transformer to extract local features within each group and use a cross-group Transformer to modeling interaction among groups and extract global features. To further enhance local shape information, we propose a Radius Feature Abstraction (RFA) module to extract radius-based density features to represent the sparsity of the local point cloud.

The main contributions of this paper are as follows:

•

We propose a new Transformer-based point cloud learning architecture GiG. By dividing a point cloud into a set of small groups, GiG can not only learn the relationship between groups to model global information but also model local semantic information by learning the relationship between points within each group. Therefore, it is possible to extract effective object-related global information as well as shape-related local information.

•

We propose a RFA module to enhance the representation of local semantic information by extracting radius-based density features to characterize the sparsity of local point clouds.

•

Extensive experiments demonstrate that our method achieves new state-of-the-art performance in object classification and part segmentation.

Our paper is structured as follows. In Section 2, we briefly review related work. In Section 3, the proposed method is introduced in detail. Section 4 introduces the experimental results and analysis. In the last section, we give the conclusions of our paper.

2. Related Work

2.1. Deep Learning on Point Clouds

2.1.1. Multi-View Based Methods

The multi-view based methods [10,26,27] mainly map an original 3D point cloud to 2D images from different viewing angles, and then extract view-wise features using a 2D convolutional neural network. Pioneering work MVCNN [5] aggregates the multi-view features into a global descriptor by max-pooling. How to integrate the features from different views into discriminative global features is still the main challenge. Despite the good results of these methods, the mapping process inevitably results in loss of shape information, and the high performance often requires a large number of views, which is also a time-consuming process.

2.1.2. Voxel-Based Methods

The voxel-based methods [9,28,29] voxelize irregular point clouds into a regular volumetric grid structure, which can then be directly used for feature extraction using 3D convolutional neural networks. Although there are encouraging results, these methods tend to suffer from low resolution due to computational cost, and thus lose geometric structural information.

2.1.3. Point-Based Methods

Point-based methods process the raw point cloud data directly. According to the network architecture used to extract the features, point-based methods can be further classified into point-wise MLP methods, convolution-based methods, and graph-based methods.

Point-wise MLP Methods: As a pioneering work, PointNet [11] processes each point individually by a shared MLP and then aggregates global features by a symmetric function. Its follow-up work further aggregates local features to extract local geometric information [12,30,31].

Convolution-based Methods: Convolution-based methods [13,14,15,32,33,34,35,36] extract local features of point clouds by a convolution-like strategy. Due to the irregularity of point clouds, the current convolution-based methods can be further classified into continuous convolution methods [14,15,32,33] and discrete convolution methods [13]. Continuous methods define the convolution kernel in continuous space, and the weights of its nearest neighbors are related to the spatial distribution of the corresponding centroids, whereas discrete methods define the convolution kernel in regular grids, and the weights of its nearest neighbors are related to the offsets of the corresponding centroids.

Graph-based Methods: Graph-based methods [16,17,37,38,39] model the original point clouds as graphs and then utilize graph-based learning methods. For example, FeastNet [37] and DGCNN [16] model the geometric information by constructing different graph convolution operators to aggregate the neighboring features to model geometric information.

2.1.4. Hybrid-Data Based Methods

The hybrid-data methods [29,40,41] are proposed based on different point cloud data structures (e.g., octree and kd-tree) and integrate voxel features and PointNet features at the scene level [19,20]. However, the effectiveness of extracting point cloud features is further limited by the fact that these two point cloud representations are fundamentally different.

2.2. Transformer in Point Clouds

Transformer has achieved promising results in the field of natural language [22,23] and computer vision [24]. Compared with convolution and MLP-like operations, Transformer can model effective long-range dependency while directly processing irregular point clouds. However, there are relatively few studies on Transformer-based point cloud processing. PCT [21] mainly uses offset-attention self-attentive layers to extract global features while using neighbor feature embedding in EdgeConv [17] to assist in extracting local features. However, PCT does not make full use of Transformer to extract local features and cannot model local geometric features effectively.

3. Methods

3.1. Preliminaries

To make it straightforward to understand our method, we first describe the basic components of the transformer [42], including scaled dot-product self-attention (SA), multi-head self-attention (MSA), feed-forward networks (FFN), and layer normalization (LN).

SA: Given the inputs $X\in {\mathbb{R}}^{N\times D}$, in the SA module, they are first linearly mapped into three vectors (queries $Q\in {\mathbb{R}}^{N\times D_k}$, keys $K\in {\mathbb{R}}^{N\times D_k}$, and values $V\in {\mathbb{R}}^{N\times D_V}$), where N is the sequence length, and $D,D_k,D_V$ are the dimensions of the input, queries (keys), and values, respectively. Each word in the sequence is treated as a query and all of the words are treated as keys. We calculate the similarity score between each query and key, then the similarity score is normalized and multiplied by the corresponding values to obtain a weighted sum, i.e., the attention score of the word, which is defined as follows:

$$Attention(Q,K,V)=softmax\left(\frac{Q{K}^T}{\sqrt{D_k}}\right)V$$

(1)

MSA: If we only perform single self-attention, all the information is aggregated in the same level. Study [42] stacks multiple self-attention modules (i.e., multi-head self-attention) to learn stronger discriminative representation. Specifically, MSA applies h different linear functions for $Q,K,V$ in parallel and finally concatenates the output values of each head, which are defined as follows:

$$MultiHead(Q,K,V)=Concat{({head}_1,\cdots ,{head}_h)W}^O$$

(2)

$${head}_i=Attention\left(Q{W}_i^Q,K{W}_i^K,V{W}_i^V\right)$$

(3)

where $W^Q,W^K,W^V,and{W}^O$ are four learnable matrixes that perform the linear mapping operation for queries, keys, values, and the final output, respectively.

FFN: The FFN module (achieved by several MLPs) is used between MSA layers for feature mapping:

$$FFN\left(x\right)=\sigma \left(x{W}_1+b_1\right)W_2+b_2$$

(4)

where W and b are the weights and bias of the MLP, and $\sigma (·)$ is the activation function such as Gaussian error linear units (GELU) [43].

LN: Layer normalization is applied in Transformer to stabilize the training process effectively and reduce the training time [44]. For each input $x\in {\mathbb{R}}^D$, LN is performed as follows:

$$\widehat{x}\leftarrow \frac{x-{\mu }_L}{\sqrt{{\sigma }_L^2+ϵ}}$$

(5)

$$\widehat{y}\leftarrow \gamma {\widehat{x}}_i+\beta \equiv {LN}_{\gamma ,\beta }\left(x\right)$$

(6)

where ${\mu }_L$ and ${\sigma }_L$ are the mean and standard deviation of the input, respectively, and $\gamma$ and $\beta$ are two learnable transform parameters.

3.2. Framework Overview

The overall framework is shown in Figure 1. In this paper, we propose GiG to efficiently learn global and local information for point cloud feature extraction. The network is able to focus not only on object-related global information by cross-group transformer, but also on shape-related local semantic information by inner-group transformer. In addition, we further propose the RFA module to enhance the local semantic information by radius-based density features.

Formally, we are given an input point cloud $p\in {\mathbb{R}}^{N\times 3}$ with N points. First, a global feature extraction module (E) is used to extract point features $f_p\in {\mathbb{R}}^{N\times d}$, and the original point cloud p and the features $f_p$ are input to the Group Abstraction ($GA$) module. $GA$ samples $M_1$ representative points from the original point cloud p and constructs a local neighboring point group for each of them, so that we have $M1$ groups denoted as $G1\in {\mathbb{R}}^{M_1\times K\times 3}$, where K (K = 32) is the number of points in each group. We use $f_{G1}\in {\mathbb{R}}^{M_1\times K\times d}$ to denote the features of the point groups. The radius-related density features $f_{R1}\in {\mathbb{R}}^{M_1\times K\times d_1}$ are extracted from $G1$ by the RFA module. The features $f_{G1}$ are then aggregated by the local feature aggregation module ($LA$, consisting of cascaded MLPs and MaxPooling) and concatenated with $f_{R1}$ to obtain the local group features ${\psi }_{G1}\in {\mathbb{R}}^{M_1\times d_G}$. Subsequently, all groups $G_1$ and group features ${\psi }_{G1}$ are input to another $GA$ module and sampled into $M_2$ groups $G2\in {\mathbb{R}}^{M_2\times K\times 3}$. Similarly, local features are integrated to obtain group features ${\psi }_{G2}\in {\mathbb{R}}^{M_2\times d_G}$, and $f_{G2}$ is reshaped to obtain inner-group point features ${\phi }_{G2}\in {\mathbb{R}}^{M_2\times K\times d_p}$. Finally, the group features ${\psi }_{G2}$ and the inner point features ${\phi }_{G2}$ within group $G2$ are input to the GiG module to extract cross-group and inner-group features to learn the global information and local semantic information, respectively, and the point cloud feature F is obtained by MaxPooling. The GA, RFA, and GiG module will be introduced in detail in the following subsections.

Classification: After obtaining the global feature F, in order to classify the point cloud p into $N_c$ classes, we input F to a classification head, which consists of two cascaded classification layers. Each classification layer consists of a linear layer, a BatchNorm, a LeakyReLU layer, and a Dropout layer with probability 0.5. Finally, the final classification result is predicted by a linear layer.

Segmentation: The segmentation network is a little different from the classification network because the segmentation task requires higher resolution. Therefore, we do not perform a downsampling operation before the GiG module. We apply DGCNN as the backbone of the segmentation network, and the GiG module is introduced after the second graph feature extraction layer of DGCNN to model local and global information, which effectively improves the performance of segmentation. This shows the effectiveness and practicality of the proposed GiG module.

3.3. Group Abstraction (GA) and Radius-Based Feature Abstraction (RFA)

First, given a point cloud $p\in {\mathbb{R}}^{N\times 3}$, we group it by the GA module and perform linear projection. Specifically, we obtain M representative points $p_1,p_2,\dots ,p_M$ for the original point cloud p by farthest distance sampling (FPS) [45], and then for each representative point $p_M\in p$, we collect its K nearest neighbors to form a group and obtain M groups $G\in {\mathbb{R}}^{M\times K\times 3}$, and the features corresponding to all the points in the groups are retained. The features of all the points in each group are denoted as $f_p^i(i=1,2,\cdots ,M)$ and it is augmented by a local feature enhancement module ($LE$, consisting of cascaded MLP) to obtain $f_G^i\in {\mathbb{R}}^{K\times d}(f_G^i\in f_G{,f}_G\in {\mathbb{R}}^{M\times K\times d})$ by linearly mapping the local features. After two GAs, we reshape the features $f_G$ to obtain inner-group feature ${\phi }_G\in {\mathbb{R}}^{M\times K\times d_p}$.

For each group, we apply the RFA module to enhance local semantic information, where the farthest distance between the centroid $p_M$ from other inner-group points is mapped as the feature representation $f_R^i\in {\mathbb{R}}^{K\times d}(f_R^i\in f_R{,f}_R\in {\mathbb{R}}^{M\times K\times d})$ to determine if the group is sparse or dense. The features $f_G$ are then aggregated by the LA and concatenated with $f_R$ to obtain the local group features ${\psi }_G\in {\mathbb{R}}^{M\times d_G}$.

3.4. GiG (Group-in-Group Relation-Based Transformer)

The GiG module is described in detail below, and its architecture is shown in Figure 1. The encoded inner-group point features $\phi$ and group features $\psi$ are input to the GiG module. To enhance the inner-group feature representation, we use an inner-group transformer block to learn the relationship between inner points $p^{i,j}$ in each group, where $p^{i,j}$ represents the jth point in the ith group. The inner-group transformer block is defined as follows:

$${\phi }_l^{\prime i}={\phi }_{l-1}^i+MSA\left(LN\left({\phi }_{l-1}^i\right)\right)$$

(7)

$${\phi }_l^i={\phi }_l^{\prime i}+FFN\left(LN\left({\phi }_l^{\prime i}\right)\right)$$

(8)

where l = 1, 2 …L represent the lth block, and L represents the number of transformer blocks.

For the extraction of cross-group features, we use another standard transformer to learn the relationship between different groups. In each layer, the point feature ${\phi }_l^i$ after linear projection within the group is added to the group feature, which is defined as follows:

$${\psi }_{l-1}^i={\psi }_{l-1}^i+Vec\left({\phi }_{l-1}^i\right)W_{l-1}+b_{l-1}$$

(9)

where $Vec(·)$ represents flattening the input into a one-dimensional vector, and $W_{l-1}$,$b_{l-1}$ represent the weights and bias, respectively. Cross-group transformer block is defined as follows:

$${\psi }_l^{\prime i}={\psi }_{l-1}^i+MSA\left(LN\left({\psi }_{l-1}^i\right)\right)$$

(10)

$${\psi }_l^i={\psi }_l^{\prime i}+FFN\left(LN\left({\psi }_l^{\prime i}\right)\right)$$

(11)

The inner-group transformer block is used to model the relationship between points within a group and then extract shape-related local semantic features. The cross-group transformer block is applied to model the relationship between groups and then extract object-related global features. By stacking L layers of GiG blocks, we build the GiG module.

$${\phi }_l,{\psi }_l=GiG\left({\phi }_{l-1},{\psi }_{l-1}\right)$$

(12)

4. Results

The comparison results are detailed as follows. In Section 4.1, we compare our proposed method with the existing methods in the classification task on the ModelNet40 dataset. In Section 4.2, we further compare our method in the part segmentation task on ShapeNet dataset.

4.1. Classification on ModelNet40 Dataset

Dataset: ModelNet40 is currently the most widely used dataset for point cloud classification. It includes 12311 CAD models with 40 categories. For a fair comparison, we used the official data splitting, with 9843 point clouds as the training set and 2468 point clouds as the test set. The point clouds were uniformly downsampled to 1024 points. No data augmentation was used during either training or testing. The training batch size and epoch are set as 16 and 250, respectively, and an initial learning rate of 0.01 is set according to DGCNN [16] and decayed by using cosine annealing strategy for each epoch. In addition, as the point cloud data already contains the location information, it is not necessary to perform position encoding, and class token is not used.

Performance Comparison: The experimental results of classification are shown in Table 1, and our method achieves the best accuracy over existing state-of-the-art methods. We conducted an ablation study on the proposed RFA module. We can see that our method without RFA can still achieve the same accuracy as PCT. After adding the RFA module, our method achieves 93.9% classification accuracy. Compared with PointNet, the accuracy of our method is 4.7% higher.

To have an intuitive understanding of the classification performance of our models, we visualize the learned features on the test set of ModelNet40 in Figure 2. For better visualization, we randomly select 10 classes to map the high-level features to 2D space by applying t-SNE [46]. From Figure 2, we can clearly see that features from different categories are separated fairly well, which reflects the strong discriminative power of the learned representation by our proposed method.

Table 1. Comparison of classification accuracy on ModelNet40.

Method	Input	#Points	Accuracy (%)
MVCNN [5]	multi-view	_	90.1
OctNet [29]	octree	_	86.5
PointwiseCNN [34]	points	1K	86.1
PointNet [11]	points	1K	89.2
PointNet++ [12]	points + normal	5K	91.9
SpecGCN [47]	points + normal	2K	92.1
PCNN [33]	points	1K	92.3
SpiderCNN [47]	points + normal	1K	92.4
DGCNN [16]	points	1K	92.9
PointCNN [13]	points	1K	92.5
PointWeb [30]	points + normal	1K	92.3
PointConv [14]	points + normal	1K	92.5
RS-CNN [32] w/o vot.	points	1K	92.4
KPConv [22]	points	1K	92.9
3D-GCN [35]	points	1K	92.1
FPConv [36]	points	1K	92.5
PCT [21]	points	1K	93.2
Ours w/o RFA	points	1K	93.4
Ours	points	1K	93.9

Table 1. Comparison of classification accuracy on ModelNet40.

Method	Input	#Points	Accuracy (%)
MVCNN [5]	multi-view	_	90.1
OctNet [29]	octree	_	86.5
PointwiseCNN [34]	points	1K	86.1
PointNet [11]	points	1K	89.2
PointNet++ [12]	points + normal	5K	91.9
SpecGCN [47]	points + normal	2K	92.1
PCNN [33]	points	1K	92.3
SpiderCNN [47]	points + normal	1K	92.4
DGCNN [16]	points	1K	92.9
PointCNN [13]	points	1K	92.5
PointWeb [30]	points + normal	1K	92.3
PointConv [14]	points + normal	1K	92.5
RS-CNN [32] w/o vot.	points	1K	92.4
KPConv [22]	points	1K	92.9
3D-GCN [35]	points	1K	92.1
FPConv [36]	points	1K	92.5
PCT [21]	points	1K	93.2
Ours w/o RFA	points	1K	93.4
Ours	points	1K	93.9

4.2. Part Segmentation on ShapeNet Dataset

Dataset: Part segmentation of point clouds is more challenging compared to point cloud classification. We perform experiments in the commonly used dataset ShapeNet [45], which consists of 16,880 3D models with 16 classes and 50 parts, with each model consisting of 2–5 parts. We use DGCNN data splitting with 14,006 models in the training set and 2874 models in the test set. The training batch size and epoch are set as 16 and 200, respectively. The same experimental settings as in classification task are used for training.

Evaluation Metric: We use the intersection-over-union (IoU) to quantitatively evaluate the segmentation results of our proposed method and compare with other existing methods. According to the configuration of PointNet [11], we define the IoU of each category as the average of IoUs for all shapes belonging to each category. Furthermore, the overall mean IoU (mIoU) is finally calculated by the average of IoUs among all the shape instance.

Performance Comparison: The results of part segmentation of the proposed method and other comparison methods are shown in

Table 2. Comparison of part segmentation results.

Method	mIoU	Plane	Bag	Cap	Car	Chair	Earph-	Guitar	Knife	Lamp	Laptop	Motor	Mug	Pistol	Rocket	Skate	Table
ShapeNet [45]	$81.4$	$81.0$	$78.4$	$77.7$	$75.7$	$87.6$	$61.9$	$92.0$	$85.4$	$82.5$	$95.7$	$70.6$	$91.9$	$85.9$	$53.1$	$69.8$	$75.3$
PointNet [11]	$83.7$	$83.4$	$78.7$	$82.5$	$74.9$	$89.6$	$73.0$	$91.5$	$85.9$	$80.8$	$95.3$	$65.2$	$93.0$	$81.2$	$57.9$	$72.8$	$80.6$
PointNet++ [12]	$85.1$	$82.4$	$79.0$	$87.7$	$77.3$	$90.8$	$71.8$	$91.0$	$85.9$	$83.7$	$95.3$	$71.6$	$94.1$	$81.3$	$58.7$	$76.4$	$82.6$
KD-Net [40]	$82.3$	$80.1$	$74.6$	$74.3$	$70.3$	$88.6$	$73.5$	$90.2$	$87.2$	$81.0$	$94.9$	$57.4$	$86.7$	$78.1$	$51.8$	$69.9$	$80.3$
SO-Net [41]	$84.9$	$82.8$	$77.8$	$88.0$	$77.3$	$90.6$	$73.5$	$90.7$	$83.9$	$82.8$	$94.8$	$69.1$	$94.2$	$80.9$	$53.1$	$72.9$	$83.0$
RGCNN [39]	$84.3$	$80.2$	$82.8$	$\mathbf{92.6}$	$75.3$	$89.2$	$73.7$	$91.3$	$\mathbf{88.4}$	$83.3$	$96.0$	$63.9$	$95.7$	$60.9$	$44.6$	$72.9$	$80.4$
PCNN [6]	$85.1$	$82.4$	$80.1$	$85.5$	$79.5$	$90.8$	$73.2$	$91.3$	$86.0$	$85.0$	$95.7$	$73.2$	$94.8$	$83.3$	$51.0$	$75.0$	$81.8$
SRN [8]	$85.3$	$82.4$	$79.8$	$88.1$	$77.9$	$90.7$	$69.6$	$90.9$	$86.3$	$84.0$	$95.4$	$72.2$	$94.9$	$81.3$	$62.1$	$75.9$	$83.2$
DGCNN [16]	$85.2$	$84.0$	$83.4$	$86.7$	$77.8$	$90.6$	$74.7$	$91.2$	$87.5$	$82.8$	$95.7$	$66.3$	$94.9$	$81.1$	$63.5$	$74.5$	$82.6$
P2Sequence [7]	$85.2$	$82.6$	$81.8$	$87.5$	$77.3$	$90.8$	$77.1$	$91.1$	$86.9$	$83.9$	$95.7$	$70.8$	$94.6$	$79.3$	$58.1$	$75.2$	$82.8$
PointConv [14]	$85.7$	−	−	−	−	−	−	−	−	−	−	−	−	−	−	−	−
PointCNN [13]	$86.1$	$84.1$	$\mathbf{86.5}$	$86.0$	$80.8$	$90.6$	$\mathbf{79.7}$	$92.3$	$\mathbf{88.4}$	$85.3$	$\mathbf{96.1}$	$77.2$	$95.2$	$84.2$	$64.2$	$\mathbf{80.0}$	$83.0$
PointASNL [31]	$86.1$	$84.1$	$84.7$	$87.9$	$79.7$	$\mathbf{92.2}$	$73.7$	$91.0$	$87.2$	$84.2$	$95.8$	$74.4$	$95.2$	$81.0$	$63.0$	$76.3$	$83.2$
PCT [21]	$86.4$	$85.0$	$82.4$	$89.0$	$\mathbf{81.2}$	$91.9$	$71.5$	$91.3$	$88.1$	$\mathbf{86.3}$	$95.8$	$64.6$	$\mathbf{95.8}$	$83.6$	$62.2$	$77.6$	$83.7$
Ours	$\mathbf{86.6}$	$\mathbf{85.6}$	$84.6$	$88.0$	$79.3$	$91.8$	$\mathbf{79.7}$	$\mathbf{92.5}$	$86.5$	$83.6$	$95.3$	$\mathbf{77.9}$	$94.9$	$\mathbf{84.9}$	$\mathbf{65.6}$	$77.4$	$\mathbf{83.8}$

. The visualization of segmentation results of several models are shown in Figure 3. Compared with other methods, our method achieves the highest mean IoU. In addition, by inserting the proposed GiG module, the mIoU of DGCNN is improved from 85.2% to 86.6%, and the performance is improved in 13 of 16 classes. PointCNN achieves the best IoU in 5 of 16 classes, including “bag”, “earphone”, “laptop”, “knife”, and “skateboard”. These categories are small in shape and relatively simple in structure, which shows that PointCNN with CNN can better learn local information but fails to capture long distance global information. In contrast, our method with Transformer can perform better with complicated and relatively big shapes, such as “plane” and “motor”. As we can see in Figure 3, the segmentation results show the robustness of our method to diverse shapes.

5. Conclusions

In this paper, we propose a new transformer-based point cloud learning network that can efficiently extract both local and global information of point clouds. The GiG module proposed in this paper extracts object-related global information and shape-related local information by modeling the relationship between cross-group and inner-group points. In addition, to further enhance the local semantic information characterization, the RFA module is proposed to represent the sparsity of local regions of point clouds based on radius-based density features. The proposed method achieves the best results in both the classification and segmentation experiments of point clouds. In the future, the method proposed in this paper has potential in other research efforts on point clouds, such as point cloud generation and completion. Furthermore, more effective architecture could be proposed to extract the global and local information of point clouds.