REGNet: Ray-Based Enhancement Grouping for 3D Object Detection Based on Point Cloud

Abstract

Currently, 3D objects are usually represented by 3D bounding boxes. Much research work has focused on detecting 3D objects directly from point clouds, and significant progress has been made in this field. However, we find there are there is still room for improvement in three aspects. First is point cloud feature extraction. Many successful methods are based on PointNet/PointNet++, which uses multi-layer perceptrons (MLP) to extract features to generate seed points, without considering foreground and background clues. The second aspect is grouping. The “vote-based cluster” grouping method defined by the pioneering VoteNet ignores shape information that is very important in the object detection field. The final aspect is the modeling ability of grouped clusters. Most successful methods treat grouped clusters separately, regardless of their different contributions to the final detection. To address these challenges, we propose three modules to address them: the foreground-aware module, the voting-aware module, and the cluster-aware module. Extensive experiments on two large datasets of real 3D scans, ScanNet and SUN RGB-D, demonstrate the effectiveness of our method for 3D object detection on point clouds.

1. Introduction

Three-dimensional point cloud data provide precise geometric and spatial information, which is crucial for computer vision applications such as autonomous driving [1,2], augmented reality [3,4], and domestic robots [5,6]. The development of commercial depth cameras and LiDARs has made this task an active research field, attracting increasing attention. The goal of this task is to obtain 3D bounding box orientation and semantic label information for each object in the input scene. However, detecting objects in point clouds may be more challenging than in 2D due to their non-structured nature. Although current CNN networks have been proven to be very effective in 2D object detection, applying these CNN networks directly to point clouds is very difficult.

Compared to grid-structured images, 3D point clouds can provide pure shape and geometric information, unaffected by lighting and reflectance. However, point clouds are irregular, disordered, and sparse, thus how to apply the current successful CNN methods to 3D point clouds is the main challenge. Early work attempted to transform point clouds into “grid” structured data and process them using 2D-based detectors. Song [7] proposed a 3D ConvNet formulation for deep sliding shapes with a 3D region proposal network and an object recognition network to obtain 3D bounding boxes of the input scene. Hou [8] extended the standard 2D object detection framework R-CNN [9,10] to 3D. The above methods try to project point clouds into voxelization spaces, which would obtain a plausible performance; however, this kind of method suffers from high computation costs due to expensive 3D convolutions. Alternatively, some researchers have focused on projecting point clouds into a bird’s-eye-view space [11,12]. However, bird’s-eye-view methods sacrifice depth in the geometric clues that are crucial for clustering indoor scenes, which decrease detection performance. Some other researchers have adopted two-stage cascade approaches to address the issues [13,14]. In the first stage, 2D object detectors will be used to obtain 2D object bounding boxes. Then, in the second stage, these 2D bounding boxes will be extruded to form the final 3D bounding boxes.

Another technical branch is point-based methods. Point clouds have emerged as a great powerful representation for 3D deep learning tasks, such as classification [15,16,17,18,19,20], semantic segmentation [21,22,23], point cloud normal estimation [24], 3D reconstruction [25,26,27], and 3D object detection [28,29,30,31]. Most of these works adopt raw point clouds to extract expressive representations based on pioneering work PointNet/PointNet++ [15,16].

The first end-to-end point-based work, VoteNet [32], utilizes PointNet++ as the backbone to extract point features, followed by a neural network that reproduces the classic traditional Hough voting scheme. It consists of three main components: a PointNet++-based point feature extraction module, a voting module, and a cluster generation module for object proposal and classification. Based on these three modules, many subsequent works attempt to improve performance by enhancing or modifying each module. Some works propose a combined MLP (CMLP) [33] and attention MLP (AMLP) [28] to enhance the modelling ability of the PointNet++ backbone. Some works consider the voting and clustering generation module not powerful enough, such as MLCVNet [29], which proposes a patch-to-patch context module and object-to-object context modules to capture contextual information to obtain more accurate vote centers and clusters, respectively. RBGNet [34] proposes a ray-based feature grouping module to improve the grouping scheme of VoteNet as well as a foreground-based feature extraction module to enhance feature representation ability.

Compared with VoteNet, although the aforementioned methods can achieve many improvements, we find that there is still room for improvement in these three modules. For feature extraction in the backbone network, most methods focus on obtaining more reasonable feature representations. Although RBGNet [34] proposes a foreground-based module to consider different contributions of the foreground points and background points in the input point clouds, it did not consider the different contributions of foreground and background points in the input point cloud, and it did not take into account the relationship between foreground and background points. There is also much work to be done to improve the voting generation part, such as RBGNet [34], which utilizes a ray-based module to learn better representations of object shape. However, the authors did not consider the relationship among the votes on each ray. For the cluster generation and classification part, most of the works ignore the surrounding clusters’ contextual information, which is crucial for the current clustering classification.

To address the aforementioned issues, this paper discusses the task of indoor scene 3D object detection via point cloud. Specifically, we propose a foreground-based module to calculate the different contributions of the foreground and background points in the backbone. In this paper, we adopt PointNet++ as our backbone network. In this module, we produce a two-channel weighted map for both foreground and background points separately. Towards the second problem, inspired by [34], we introduce a vote-aware module to simulate the spatial relationship between votes along rays. For the third issue, we propose a cluster-aware module to build the spatial dependencies among clusters to utilize the rich contextual information for the final 3D bounding box classification. With these three modules, we propose a unified ray-based enhancement network (REGNet) to incorporate into VoteNet for 3D object detection.

The framework of REGNet is shown in Figure 1; the three main modules are shown in red font bounding boxes. In summary, the contributions of this paper include:

• We propose a ray-based enhancement 3D object detection network that exploits contextual information at foreground background, voting patch, and cluster.
• We design three sub-modules, involving a foreground-aware module, a voting-aware module, and a cluster-aware module. The new modules nicely fit into thde VoteNet framework.
• Experiments on ScanNet V2 and SUN RGB-D datasets demonstrate the effectiveness and superiority of the proposed modules in improving detection accuracy.

The rest of the article is organized as follows. Section 2 briefly reviews the most relevant work. Section 3 provides detailed information on the proposed REGNet. We present experimental results and in-depth analysis in Section 4. Finally, the conclusions are drawn in Section 5.

2. Related Work

Grid Projection/Voxelization-based Detection. 3D object detection is a challenging task due to the irregular, sparse, and orderless characteristics of 3D points. Most current existing work could be classified into three categories in terms of point cloud representations, i.e., voxel-based, bird’s-eye-view-based, and point-based. Thanks to the success of the deep neural network, marvelous progress has been achieved in 2D object detection. However, the 2D object detection task ignores the depth information, which is important to understand the whole scene. Early 3D object detection methods project point clouds to 2D grids [35,36,37] or 3D voxels [8,38], so that the most successful convolutional networks could be directly applied. Concerning bird’s-eye-view based methods, most of them are applied to autonomous driving. In outdoor scenes, most objects are distributed on the same plane, so there is little mutual occlusion of objects in the top-down view. However, in indoor scenes, many objects are on top of each other, such as photos on walls or tables covering sofas. Therefore, some works project the point cloud onto a frontal view and use 2D ConvNets to tackle the problem. However, self-occlusion of objects in indoor scenes poses many challenging issues to address. Voxel-based methods transform the point cloud into 3D voxels, which have been shown to yield more reasonable performance. However, these methods suffer from high memory and computational costs, as well as quantization errors.

Point based-Detection. To tackle the problems noted above, recently, most methods process point clouds directly for 3D object detection. Due to the point clouds being irregular and sparse, how to extract feature representations from them is the core task of these methods. The pioneering works PointNet/PointNet++ [15,16] provide a powerful and robust backbone for point cloud based tasks. VoteNet [32] adopts PointNet++ as the backbone and reproduces a Hough voting strategy to propose an end-to-end framework for 3D object detection. This is the first end-to-end framework based on point clouds for this task. There are numerous successors based on this work, such as BGNet [39], which further improves the traditional Hough voting mechanism [40]. The authors proposed a back-tracing strategy that generatively backtracked representative points from the vote centers and then revisited the seed point. H3DNet [30] and MCGNet [41] recognize the lack of modeling ability with only feature extraction from a single backbone branch. They utilized a four-way backbone to extract more plausible feature representations. MLCVNet [29] proposes three sub-modules to capture multi-level contextual information in point cloud data to boost performance. VENet [28] improves the voting procedure in “before, during and after” stages to address the limitations of current voting schemes.

Attention Mechanism/Transformer-based Detection. The transformer scheme is the dominant network architecture for the tasks of neural language of processing (NLP). Due to the powerful ability of feature modelling, it is applied in the field of 2D image recognition [42,43,44]. Most recently, many works apply the transformer scheme into 3D object detection [45,46]. GFNet [31] proposes a group-free strategy that adopts a powerful transformer module to replace the proposal head in VoteNet [32]. In this paper, we also resorts to the transformer scheme to build spatial context dependencies among different vote centers and clusters to boost prediction performance.

Point Cloud Sampling. Since the point cloud is sparse and irregular, it cannot be calculated by conventional grid-based methods. Therefore, in order to facilitate training and inference, sampling operation [47,48,49] plays a very key role in the task of point cloud analysis. Examples include farthest point sampling (FPS) and k-closest points sampling (KPS), which have been widely leveraged in the task of object detection in point clouds. However, these downsampling strategies are class-agnostic, treating all points equally without considering their different contributions. Therefore, some redundant points may inadvertently be retained, while important information may be lost after downsampling. 3DSSD [50] proposed F-FPS sampling strategy based on feature distance to preserve interior points. BGRNet [34] proposes a foreground-biased sampling to keep the foreground point during sampling. However, these methods still do not consider the spatial relationship between foreground points and background points. To address this issue, we proposed a foreground-aware module to sample more points on the foreground and establish relationships among the sampled points.

3. Our Approach

The proposed REGNet is based on VoteNet and inspired by the RGBNet for the ray-aware module. This method aims to establish relationships between foreground and background points to enhance seed point representation capability as well as model spatial relationships between voting centers on rays to obtain better object feature representation. Furthermore, it computes contextual information for all clusters to improve the final prediction performance. To achieve this goal, we have developed three new modules based on VoteNet, namely the foreground-aware module, voting-aware module, and cluster-aware module. This section elaborates on the learning details of the proposed REGNet, and the overall framework is shown in Figure 1.

3.1. Backgrounds

PointNet++ is a pioneering work applied in 3D point cloud learning tasks. It is a hierarchical neural network that processes a set of points sampled in metric space in a layered manner. This work provides more powerful feature extraction capabilities. Therefore, it is often used as a backbone network for downstream tasks.

The original VoteNet can be summarized as three modules, namely the feature extraction module, voting module, and voting aggregation module. The feature extraction module generates seed points and corresponding features based on PointNet++. The voting module regresses object centers from each seed point, and the voting aggregation module combines features from different seed points to vote for the object center. Then, the object proposals are classified and the accurate position and size of the 3D object are regressed based on the aggregated features.

3.2. Foreground-Aware Module

Current sampling methods, such as FPS and KPS, lack consideration for foreground and background information. However, sometimes foreground points and background points can provide different contributions. For example, if too many points are sampled from a chair, it may result in incorrect classification as a table, as shown in Figure 5a. Foreground points provide rich clues about object shape, such as position and orientation information, which are important for 3D object detection tasks. However, it is also unreasonable to completely ignore background points while only considering foreground points. The background points could provide some important contextual information for the final prediction. Most sampling methods, such as FPS employed in the backbone, are class-agnostic. They do not consider foreground and background, but rather randomly sample an initial point from the dataset and iteratively select the furthest point from the previously selected point as the next point in the subset. Although these methods can capture the basic characteristics of the data, they bring about some uncertainty of information since the selection of points is solely based on the properties of the dataset, rather than any specific location information such as foreground and background. This is because the selection is based on the spatial distribution of points and their distance from previously selected points, without considering any class-specific information.

Although the FPS method can effectively solve the problem of distinguishing foreground and background information in point cloud sampling, it does not consider the interaction between different foreground and background information. Although foreground and background information is distributed in different locations and expresses different information in the scene, there is still a certain correlation between them. For example, the information around a chair can provide certain background knowledge for the classification of objects within the bounding box.

To address the above problem, we propose a foreground-aware module. For simplicity, the point cloud data in this paper are denoted as $\mathbf{P}$$=\{{\mathbf{p}}_1,\dots ,{\mathbf{p}}_N\}\in {\mathcal{R}}^{N\times 3}$. To be specific, there are four SA layers in the standard PointNet++. After the first SA layer, the input point cloud is downsampled to $\mathbf{P}=\{{\mathbf{p}}_1,\dots ,{\mathbf{p}}_{2048}\}\in {\mathcal{R}}^{2048\times 3}$ points and corresponding features $f_i^{128}$, where $i=\{1,2,\dots ,2048\}$. To better separate foreground and background from the obtained point cloud data, this article employs an additional 2-class segmentation network for foreground and background point acquisition. The segmentation network obtains a 2-dimensional score map through a standard softmax network, and the argmax operation can be used to determine the category to which each point belongs. The detail is as follows:

$$\begin{array}{c}\hfill Seg(p_i,f_i^{128})=1,if:argmax\left(softmax\left(Seg(·)\right)\right)=1\\ \hfill Seg(p_i,f_i^{128})=0,if:argmax\left(softmax\left(Seg(·)\right)\right)=0\end{array}$$

(1)

where $p_i$ denotes a point, $Seg(·)$ denotes a 2-class point cloud segmentation network, and $f_i\in f_i^{128}$ is the the point feature corresponding to $p_i$. After the operation noted above, all 2048 points have a label; the point with label 1 is grouped to $P^{fore}$, and point with label 0 is grouped to $P^{back}$. Based on these two point clouds, we applied farthest point sampling to the foreground and background sets, respectively, and combined them into the final sample set, as shown below:

$$\begin{array}{c}\hfill P^{\check{fore}}=FPS\left(P^{fore}\right),P^{\check{back}}=FPS\left(P^{back}\right)\end{array}$$

(2)

The next issue is how to combine $P^{\check{fore}}$ and $P^{\check{back}}$ into the final sample set, which is very important. Ref. [34] combines them directly without considering each contribution to the next stage. We propose a weighted map for all 2048 sampled points. Each item in the weighted map denotes the contribution of each point. The weighted map is generated by a 2-layer fully connected layer with a softmax layer, and then we build the relationship of the $P^{\check{fore}}$ and $P^{\check{back}}$ to form the final sample set. The detail is as follows:

$$\begin{array}{c}\hfill wp=softmax({\mathrm{FC}}_2({\mathrm{FC}}_1(P^{\check{fore}},P^{\check{back}})))\\ \hfill P^{final}=(P^{\check{fore}}\oplus P^{\check{back}})\odot wp\end{array}$$

(3)

where $wp$ denotes the learned weighted map, the operator ⊙ represents the Hadamard product, ⊕ represents sum operation, and FC₂, FC₁ denote the 2-layer fully connected layer.

3.3. Voting-Aware Module

The vote scheme in the VoteNet does not consider relationships between point patches; however, this information is very important to the location and classification of the final prediction. MLCVNet proposes patch-patch context (PPC) to capture the relationships between point patches. It adopts the compact generalized non-local network (CGNL) [51] to explicitly build rich correlations between any pair of point patches. The detail is as follows:

$$\begin{array}{c}\hfill Re=sim\left(\theta \right(f),\varphi (f\left)\right)g\left(f\right)\end{array}$$

(4)

where $\theta (·)$, $\varphi (·)$, $g(·)$ denote three transform functions, and $sim(a,b)$ calculates the similarity between the two positions of a and b. However, CGNL only has one head and one layer to build the relationship among the vote centers. Moreover, CGNL can only model the vote center and cannot consider the relationship between the vote center and the clusters. Additionally, in VoteNet, vote centers and cluster centers are generated through sampling and grouping from seed points. However, this method lacks consideration for the appearance information of objects. For [34], a ray-based feature grouping method is employed, which can learn a better feature representation of the surface geometry of foreground objects. However, the relationship between the vote center and the cluster has yet to be considered. With the development of transformer technology, more and more researchers are trying to use a transformer scheme to model the interconnections between different modules.

In this paper, we adopt the ray-based feature grouping method proposed in [34] to generate vote centers and clusters, and the stacked multi-head self-attention and multi-head cross-attention modules are leveraged to establish the relationship between the voting center and the cluster. The multi-head self-attention module is used to simulate the relationship between votes, and the multi-head cross-attention module is used to simulate the interaction between votes and clusters. After stacking the attention modules, a feed-forward network (FFN) is used to obtain more reasonable transformation features for each object. The structural description of this module is shown in Figure 2. Denote the point features of the voting center as ${f_v}_{i=1}^M$ and the cluster feature as ${f_c}_{i=1}^K$. The self-attention module build the relationship between votes, formulated as follows:

$$\begin{array}{c}\hfill self-att(\theta \left(f_v\right),\varphi \left(f_v\right))g\left(f_v\right)\end{array}$$

(5)

and the cross-attention module adopts point features to compute object features, formulated as follows:

$$\begin{array}{c}\hfill cross-att(f_{v_i},f_{c_i})\end{array}$$

(6)

where $f_{v_i}\in {f_v}_{i=1}^M$ and $f_{c_i}\in {f_c}_{i=1}^K$.

3.4. Cluster-Aware Module

VoteNet detects each object class and bounding box by feeding the generated votes to MLP layers in the scene. However, this grouping method lacks consideration of surrounding information. MLCVNet proposes object–object context (OOC) to combine features of surrounding objects to provide more information about object relationships. It inputs the grouped voting centers to an MLP with max-pooling to form a single vector representing the cluster, and then introduces a self-attention module to establish relationships between these clusters instead of processing them separately. It adopts the CGNL attention module to generate a weighted map to calculate the affinity between all clusters. The detail is as follows:

$$\begin{array}{c}\hfill R{e}_{OOC}=self-att(\underset{i=1,2,\dots ,n}{max}\left\{MLP\left(c_i\right)\right\})\end{array}$$

(7)

where $self-att(·)$ is the CGNL attention module, and $c_i$ is the feature of the cth cluster.

However, as noted above, the CGNL attention module can only simulate the affinity between clusters. Yet, a voting center can also provide useful information. In this paper, we adopt stacked self-attention and cross-attention modules to establish relationships. The detail of the self-attention module is as follows:

$$\begin{array}{c}\hfill self-att({\theta }_1\left(f_v\right),{\varphi }_1\left(f_v\right))g_1\left(f_v\right)\end{array}$$

(8)

and the cross-attention module is formulated as follows:

$$\begin{array}{c}\hfill cross-att(f_{c_i},f_{v_i})\end{array}$$

(9)

After the stacked self-attention and cross-attention modules, a feed-forward neural network (FFN) is utilized to extract transformation features for the final prediction task.

By combining these three modules, the final 3D bounding box inference incorporates all surrounding contextual information while also taking into account the distinct contributions of foreground and background points. This results in a more accurate and reasonable final prediction.

4. Experiments and Discussions

4.1. Dateset

We evaluated our REGNet on two large datasets, ScanNet V2 and SUN RGB-D, both of which have been captured from real 3D scans. ScanNet V2 is a well-known public indoor scene dataset that comprises 1513 indoor scenes with detailed annotations, including instance points, semantic labels for 40 categories, and 3D bounding boxes categorized into 18 categories.

SUN RGB-D contains 10,335 indoor RGB and depth images with per-point semantic labels and object bounding boxes. The data in this dataset were captured by four consumer-grade depth cameras, namely Intel Realsense, Asus Xtion, Kinect v1, and Kinect v2. A total of 5285 pairs of data are used for training, and the rest are used for validation. Prior to training and validation, each RGB-D scene is first converted into a point cloud representation by aligning the RGB and depth images and then projecting the depth information onto the RGB image.

4.2. Training Details

Our network is optimized end-to-end using the Adam optimizer with a batch size of 8. The base learning rate for the ScanNet V2 dataset is set to 0.01 and for the SUN RGB-D dataset it is set to 0.001. The network is trained for 350 epochs on ScanNet V2 and 300 epochs on SUN RGB-D. The learning rate decay steps are set to {100, 200, 300} for ScanNet and {100, 200, 250} for SUN RGB-D, and the decay rates are {0.1, 0.1, 0.1} for both datasets. Gradnorm clipping was used to ensure stable training dynamics. Due to unstable and difficult-to-train transformer blocks in the voting-aware module and the cluster-aware module, we use the same setting as in [31], performing three training runs for each experiment and also testing each training experiment three times. To illustrate the randomness of the algorithm, the average performance of nine trials is reported. All experiments are implemented by the PyTorch platform using RTX P6000 GPU.

Following [32], we report the detection performance on the validation set by calculating the average precision (AP) at IoU thresholds of 0.25 ([email protected]) and 0.5 ([email protected]) in 3D space. The detection performance and average results are displayed for each class. As shown in [32], there are four set abstraction (SA) layers in the network. In this paper, we use the same approach as [34], using a foreground bias sampling (FBS) module before each SA layer to enhance the network’s ability to distinguish foreground and background points during the learning process.

4.3. Comparisons with the State-of-the-Art Methods

Experiments with ScanNet V2. To demonstrate the effectiveness of the proposed REGNet, we compared it with the most recent state-of-the-art techniques. The results are summarized in

Table 1. Performance comparisonon the ScanNet V2 validation set with state-of-the-art methods. The top two results are shown in red and blue, respectively.

Methods	Presented at	Cabinet	Bed	Chair	Sofa	Table	Door	Window	Bookshelf	Picture	Counter	Desk	Curtain	Refrigerator	Showercurtain	Toilet	Sink	Bathtub	Ofurn	[email protected]	[email protected]
VoteNet [32]	ICCV’19	36.27	87.92	88.71	89.62	58.77	47.32	38.1	44.62	7.83	56.13	71.69	47.23	45.37	57.13	94.94	54.7	92.11	37.2	58.6	33.5
GRNet [52]	ISPRS’20	39.45	88.78	89.18	88.34	58.16	48.46	32.7	46.97	4.94	63.48	69.81	48.46	49.06	66.37	94.07	49.7	90.9	35.6	59.1	39.1
Griffiths [53]	ECCV’20	43.0	70.8	58.3	16.0	44.6	28.0	13.4	58.2	4.9	69.9	74.0	75.0	36.0	58.9	79.0	47.0	77.9	48.2	50.2	–
SPOT [54]	ECCV’20	–	–	–	–	–	–	–	–	–	–	–	–	–	–	–	–	–	–	59.8	40.4
GSDN [55]	ECCV’20	41.58	82.5	92.14	86.95	61.05	42.41	40.66	51.14	10.23	64.18	71.06	54.92	40.0	70.54	99.97	75.5	93.23	53.07	62.8	34.8
H3DNet [30]	ECCV’20	49.4	88.6	91.8	90.2	64.9	61.0	51.9	54.9	18.6	62.0	75.9	57.3	57.2	75.3	97.9	67.4	92.5	53.6	67.2	48.1
HGNet [56]	CVPR’20	–	–	–	–	–	–	–	–	–	–	–	–	–	–	–	–	–	–	61.3	34.4
SESS [57]	CVPR’20	–	–	–	–	–	–	–	–	–	–	–	–	–	–	–	–	–	–	62.1	–
DOPS [58]	CVPR’20	53.2	83.3	91.6	82.6	60.5	54.8	45.2	41.0	26.3	51.9	73.7	53.9	49.2	64.7	98.0	71.3	86.6	59.2	63.7	38.2
MLCVNet [29]	CVPR’20	42.45	88.48	88.98	87.4	63.5	56.93	46.98	56.94	11.94	63.94	76.05	63.94	60.86	65.91	98.33	59.18	87.22	47.89	64.5	41.4
VENet [28]	ICCV’21	50.4	87.7	92.7	88.1	68.6	60.7	46.0	55.2	18.2	70.2	77.5	59.9	58.4	75.9	95.1	67.2	92.3	54.4	67.7	–
GFNet (L6, O256) [31]	ICCV’21	–	–	–	–	–	–	–	–	–	–	–	–	–	–	–	–	–	–	67.3 (66.3)	48.9 (48.5)
GFNet (L12, O512) [31]	ICCV’21	52.1	92.9	93.6	88.0	70.7	60.7	53.7	62.4	16.1	58.5	80.9	67.9	47.0	76.3	99.6	72.0	95.3	56.4	69.1 (68.6)	52.8 (51.8)
Hyper3D [59]	CVPR’22	–	–	–	–	–	–	–	–	–	–	–	–	–	–	–	–	–	–	70.9	57.2
RBGNet (R66, O256) [34]	CVPR’22	–	–	–	–	–	–	–	–	–	–	–	–	–	–	–	–	–	–	70.2 (69.6)	54.2 (53.6)
RBGNet (R66, O512) [34]	CVPR’22	–	–	–	–	–	–	–	–	–	–	–	–	–	–	–	–	–	–	70.6 (69.9)	55.2 (54.7)
Our REGNet	–	55.1	93.0	93.0	90.3	70.9	61.8	55.0	62.7	24.7	69.1	81.3	68.1	56.2	73.1	98.1	70.0	95.1	55.9	70.7 (69.9)	55.3 (53.9)

. As shown, our proposed REGNet outperforms its baseline VoteNet and achieves comparable results with the current state-of-the-art techniques. Furthermore, REGNet performs the best in 8 out of 18 categories, which is almost three times more than the second-ranked GFNet, which only achieves the best results in 3 categories. This indicates that the three proposed enhancement modules can effectively improve subsequent object localization and classification tasks.

We also provide the qualitative comparisons in Figure 3. In the figure, we give the raw point cloud, the visualization results of VoteNet, the current GFNet, the visualization results of our REGNet, and the corresponding ground truth. To ensure a fair comparison, we only compare the results of methods using only 3D geometric information. The comparison results show that our approach can obtain more accurate location results for objects such as beds and sofas, especially the bounding boxes indicated by the red dashed lines in the figure.

Experiments with SUN RGB-D. We also evaluate our REGNet on the SUN RGB-D dataset; the quantitative results are provided in

Table 2. Performance comparison on the SUN RGB-D validation set with state-of-the-art methods. The top two results are shown in red and blue, respectively.

Methods	Presented at	Table	Sofa	Bookshelf	Chair	Desk	Dresser	Nightstand	Bed	Bathtub	Toilet	[email protected]
F-PointNet [14]	CVPR’18	51.1	61.1	33.3	64.2	24.7	32.0	58.1	81.1	43.3	90.9	54.0
VoteNet [32]	ICCV’19	47.3	64.0	28.8	75.3	22.0	29.8	62.2	83.0	74.4	90.1	57.7
GRNet [52]	ISPRS’20	51.1	64.8	29.3	76.2	26.0	26.1	59.2	84.3	76.8	90.4	58.4
MLCVNet [29]	CVPR’20	50.4	66.3	31.9	75.8	26.5	31.3	61.5	85.8	79.2	89.1	59.8
ImVoteNet [60]	CVPR’20	51.1	70.7	41.3	76.7	28.7	41.4	69.9	87.6	75.9	90.5	63.4
H3DNet [30]	ECCV’20	50.8	66.5	31.0	76.7	29.6	33.4	65.5	85.6	73.8	88.2	60.1
GFNet [31]	ICCV’21	53.8	70.0	32.5	79.4	32.6	36.0	66.7	87.8	80.0	91.1	63.0 (62.6)
Hyper3D [59]	CVPR’22	–	–	–	–	–	–	–	–	–	–	63.5
RGBNet [34]	CVPR’22	–	–	–	–	–	–	–	–	–	–	64.1 (63.6)
Our GFENet	–	54.0	69.1	34.1	79.7	33.9	41.0	70.8	88.1	79.2	91.7	64.1 (63.7)

. In it, we compare our method with many other state-of-the-art methods, such as VoteNet, H3DNet, and many others on the SUN RGB-D benchmark. From the comparison results, we find that our method achieves the best performance in six classes, whereas almost all of the rest achieve the second-best performance. We also illustrate the qualitative comparisons of SUN RGB-D in Figure 4. In the figure, from left to right, are the image, raw point cloud, the prediction results of VoteNet and GFNet, our results, and the corresponding ground truth.

4.4. Ablation Study and Qualitative Results

In this section, a set of ablative studies are conducted on the ScanNet V2 and SUN RGB-D datasets to demonstrate the effectiveness of the three proposed modules of our REGNet.

Effectiveness of the foreground-aware module. To demonstrate the effectiveness of the foreground-aware module, we provide its performance across different architecture methods, as shown in

Table 3. Performance on different architecture methods of our foreground-aware module (FAM) module.

Methods	[email protected]
VoteNet + FAM	63.2
H3DNet + FAM	68.1
GFNet(L6, O256) + FAM	67.2

. In the experiments, we replace the standard backbone (PointNet++) in three state-of-the-art architectures: VoteNet, H3DNet, and GFNet (L6, O256). Due to the fact that VoteNet and GFNet both use a branch of PointNet++ to extract feature representations, we only replace the backbone part. For H3DNet, there are four branches in the backbone used to extract four different geometric primitives, and we replaces all four branches.

To better validate the effectiveness of the module, we visualize the results of VoteNet with and without the module, as shown in Figure 5. From the visual comparison of (a) and (b), it can be observed that without the module, the localization of object bounding boxes is prone to error, and misclassification can occur. With the module, both localization and classification have been improved accordingly.

Effectiveness of the voting-aware module. To validate the advantages of the voting-aware module, we provide visual results with and without using the module on the validation dataset of ScanNet V2, as shown in Figure 6. In Figure 6a, we visualize some voting centers, shown as green points around the table and red points on the table. In VoteNet, the relationship between voting centers is not considered, resulting in many incorrect bounding boxes. However, when the voting perception module is used in VoteNet, the boundary boxes of the table are more concise, and the predicted results are more reasonable.

Effectiveness of the cluster-aware module. Finally, to demonstrate the effectiveness of our cluster-aware module, we suggest conducting experiments on ScanNet V2 with and without the module. The qualitative results are shown in Figure 7. From left to right, we provide the raw point cloud, the visualization results of VoteNet, the results of VoteNet with this module, and the ground truth. From the raw visualization, we can see that the object within the red dashed circle on the right and the chair within the red dashed circle on the left appear very similar in color and shape. VoteNet is unable to distinguish between them. With this module, we believe that it can extract more reasonable feature representations based on more surrounding information and provide more reasonable prediction results.

Quantitative depth analysis. In

Table 4. Ablation study of the proposed REGNet model on ScanNet V2 dataset; F, V, and C denote foreground-aware module, voting-aware module, and cluster-aware module, respectively.

	Methods	mIoU
a.	VoteNet	58.6
b.	VoteNet + F	63.2
c.	VoteNet + F + V	69.1
d.	VoteNet + W + C	68.3
e.	VoteNet + F + V + C	70.7

, we show detailed ablation studies on the proposed REGNet model on the ScanNet V2 dataset. First, we provided the results of VoteNet on ScanNet V2, which is 58.6% in [email protected]. To demonstrate the effectiveness of our proposed modules, we conduct ablation experiments on the VoteNet model. We incrementally add the modules one by one and compare the results to the baseline. As shown in Table 4, the results clearly indicate that all three proposed modules improve the final prediction results. Furthermore, we compare our proposed method to other state-of-the-art methods on the ScanNet V2 and SUN RGB-D datasets. The quantitative comparison results are shown in Table 1 and Table 2. Our method achieve a 12.1 improvement on the ScanNet V2 dataset based on VoteNet and a 6.4 improvement on the SUN RGB-D dataset compared to VoteNet. These results demonstrate that the proposed modules are effective for improving 3D object detection tasks. We believe that the inclusion of these modules can significantly enhance the performance of existing models.

Although our method has achieved significant benefits compared to VoteNet, we have not increased the model size. VoteNet has a model size of 11.2 M, and ours is 37.6 M. Compared to H3DNet and MCGNet, which are four-branch networks, our method is smaller yet more accurate. We provide detailed comparisons of backbone and model sizes in

Table 5. Comparison with VoteNet [31], GFNet, MLCVNet, H3DNet, and MCGNet with various configurations on ScanNet V2. The main comparison is based on the best results of multiple experiments between different methods, and the number within the bracket is the average result. Here, 4×PointNet++ denotes that this method adopted four-branch individual PointNet++ as backbone; PointNet++w2× denotes that the backbone width in the method is expanded by 2; L denotes the depth of decoder, and O denotes the number of the proposal.

Model	Backbone	Model Size	[email protected]
VoteNet [32]	PointNet++	11.2 M	58.6
GFNet (L6, O256) [31]	PointNet++	14.5 M	67.3
GFNet (L12, O512) [31]	PointNet++w2×	29.6 M	69.1
MLCVNet [29]	PointNet++	13.9 M	64.5
H3DNet [30]	4×PointNet++	56.0 M	67.2
MCGNet [41]	4×PointNet++	57.2 M	67.5
REGNet	PointNet++w2×	37.6 M	70.7

to demonstrate the efficiency and compactness of our model.

To better visualize the effectiveness of the three proposed modules in this paper, we provide a comparison of our method and two classic methods, VoteNet and VENet, on the ScanNet V2 dataset in terms of category-wise performance, as shown in Figure 8. We also include a comparison of our method and VoteNet, as well as ImVoteNet, on the SUN RGB-D dataset in terms of the [email protected] criteria to further demonstrate the effectiveness and stability of our proposed method, as shown in Figure 9. Through comparison, we find that although our method decreases performance in some categories, there is an increase in performance to varying degrees in the majority of categories.

The realistic inference speed of our method is competitive with other state-of-the-art methods. The results are shown in

Table 6. Comparison on realistic inference speed on ScanNet V2 with MLCVNet, BRNet, H3DNet, GFNet, MCGNet, and RBGNet.

Model	[email protected]	[email protected]	Frames/s
MLCVNet [29]	64.5	41.4	5.37
BRNet [39]	66.1	50.9	7.37
H3DNet [30]	67.2	48.1	3.75
GFNet [31]	66.3	48.5	6.64
MCGNet [41]	67.5	50.1	3.59
RBGNet [34]	70.2	54.2	4.75
REGNet	70.7	55.3	4.69

. Our method achieves better performance with a competitive speed.

5. Conclusions

In this paper, we propose a novel REGNet to tackle the challenging task of 3D object detection in indoor scenes. Our method is inspired by VoteNet, based on which we propose the foreground-aware module, voting-aware module, and cluster-aware module to improve the detection performance. The foreground-aware module establishes relationships between foreground and background points to obtain more expressive seed point features. The voting-aware module proposes ray-based voting with a transformer strategy to calculate the contributions of each vote centers. The cluster-aware module is used to adaptively aggregate multiple clusters to achieve better performance. Qualitative and quantitative experimental results on the public ScanNet V2 and SUN RGB-D datasets show that our REGNet achieves reasonable performance improvement in 3D object detection.