Voxel- and Bird’s-Eye-View-Based Semantic Scene Completion for LiDAR Point Clouds

Abstract

Semantic scene completion is a crucial outdoor scene understanding task that has direct implications for technologies like autonomous driving and robotics. It compensates for unavoidable occlusions and partial measurements in LiDAR scans, which may otherwise cause catastrophic failures. Due to the inherent complexity of this task, existing methods generally rely on complex and computationally demanding scene completion models, which limits their practicality in downstream applications. Addressing this, we propose a novel integrated network that combines the strengths of 3D and 2D semantic scene completion techniques for efficient LiDAR point cloud scene completion. Our network leverages a newly devised lightweight multi-scale convolutional block (MSB) to efficiently aggregate multi-scale features, thereby improving the identification of small and distant objects. It further utilizes a layout-aware semantic block (LSB), developed to grasp the overall layout of the scene to precisely guide the reconstruction and recognition of features. Moreover, we also develop a feature fusion module (FFM) for effective interaction between the data derived from two disparate streams in our network, ensuring a robust and cohesive scene completion process. Extensive experiments with the popular SemanticKITTI dataset demonstrate that our method achieves highly competitive performance, with an mIoU of 35.7 and an IoU of 51.4. Notably, the proposed method achieves an mIoU improvement of 2.6 % compared to previous methods.

1. Introduction

Three-dimensional scene understanding is a crucial task in autonomous driving, which requires a detailed understanding of both the three-dimensional geometry and semantic information of an environment. In this domain, light detection and ranging (LiDAR) sensors play a pivotal role due to their superior capability to capture accurate distance measurements over longer ranges when compared to traditional visual cameras. However, despite their advantages, the data collected by LiDAR are characteristically sparse, which poses significant challenges for machines to achieve a complete understanding of the environment. To address this, the task of 3D semantic scene completion aims at reconstructing and semantically categorizing the entire environment using the limited data available, thereby bridging the gap in scene understanding capabilities between machines and humans.

Three-dimensional point cloud semantic scene completion is a considerably challenging task, particularly for large-scale outdoor scenarios. Nevertheless, there has been a remarkable interest in research in this direction recently [1,2,3,4,5,6]. The underlying efforts aim to enhance the precision and efficiency of reconstructing and semantically interpreting large-scale outdoor environments from sparse 3D point cloud data, which marks a significant stride towards improving the environmental comprehension of autonomous systems. Still, existing scene completion techniques face obvious limitations [1,2,3,4,5,6]. For instance, Yan et al. [3] introduced JS3C-Net, which combines a segmentation network and a scene completion module to achieve semantic scene completion. Initially, the segmentation network is utilized to conduct preliminary semantic segmentation, and then the scene completion module utilizes the outcomes of the segmentation network to construct a comprehensive voxel representation of the entire scene. Additionally, a point-voxel interaction (PVI) module is introduced to facilitate the transfer of shape-specific knowledge between the incomplete point clouds and the completed voxels generated in the initial steps. Nevertheless, this approach introduces the risk of perpetuating initial segmentation errors throughout the completion phase, potentially resulting in the significant corruption of valuable information.

More recently, Cheng et al. [4] presented S3CNet, a network leveraging sparse convolutions to effectively learn features from sparse point cloud data. They also developed a 2D version of S3CNet, incorporating a multi-view fusion strategy aimed at improving the 3D semantic scene completion performance. Later, Xia et al. [6] designed SCPNet to improve the representation learning of the single-frame model by transferring dense, relation-based semantic knowledge from a multi-frame teacher to a single-frame student via a dense-to-sparse knowledge distillation. A completion label rectification is further proposed to remove traces of dynamic objects in the completion labels. However, the approach initially utilizes a completion network to derive completion results, followed by employing a segmentation network to obtain the segmentation outcomes, which considerably increases the model size. This is further exacerbated by the absence of downsampling operations in the model. These aspects collectively contribute to significantly increasing the memory and computational footprint of the model.

In this work, we propose an integrated approach that merges a 3D semantic scene completion network (3D SSCNet) with a 2D semantic scene completion network (2D SSCNet), enhancing semantic scene completion from both 3D and 2D perspectives. For the proposed network, as shown in Figure 1, our design includes an efficient multi-scale convolution block (MSB) within the 3D SSCNet that aggregates features across multiple scales, leveraging rich contextual information for enhancing accuracy in small objects, distant objects, and dense scenes within outdoor scenarios. A layout-aware semantic block (LSB) is also developed in the 3D SSCNet to grasp the overall layout of the scene to precisely guide the reconstruction and recognition of features. Additionally, the introduction of a 2D SSCNet complements this framework by providing additional semantic scene information, further bolstering the capabilities of the model. A pivotal component of our approach is the feature fusion module (FFM), designed for the effective integration of the 3D and 2D features from their respective networks. This combination facilitates superior scene completion. We conduct extensive experiments with the major outdoor scene completion benchmark, SemanticKITTI [7]. Experimental results on the SemanticKITTI validation dataset indicate that our method achieves highly competitive results, outperforming current state-of-the-art methods by a significant margin of 2.6 mIoU.

Our contributions are summarized as follows:

•

We propose an integrated network that merges a 3D SSCNet with a 2D SSCNet. For the former, a highly efficient MSB is devised to segment small, distant, and dense objects. Moreover, an LSB is developed to grasp the overall layout information of the outdoor scenes.

•

We propose the 2D SSCNet to process bird’s-eye-view (BEV) features of the scene, which deliver precise spatial layout information in the two-dimensional space, thereby enhancing the overall performance of 3D semantic scene completion.

•

We propose FFM for an improved interaction of the information from the 3D SSCNet and the 2D SSCNet, where the strengths of the other can enhance each set of features.

The organization of the paper is as follows: Section 2 presents a comprehensive review of the related works, outlining the key contributions and methodologies previously employed in this research area. Section 3 details the methodology applied in our study, describing the experimental design. Section 4 discusses the experimental results, providing a thorough analysis of the data and highlighting significant findings. Finally, Section 5 presents the conclusion, summarizing the main outcomes of the research.

2. Related Work

In the field of 3D semantic scene completion, existing techniques can be categorized as follows: (i) image-based methods, (ii) point-based methods, (iii) voxel-based methods, and (iv) multi-modality-based methods. Each category represents a unique methodology for processing and interpreting 3D data, with distinct advantages and challenges.

2.1. Image-Based Methods

In image-based techniques for semantic scene completion, many proposed methods in literature employ convolutional neural networks or transformers to process multi-modal data, like RGB and depth images [1,8,9,10,11,12,13,14,15,16,17,18,19,20]. For instance, Song et al. [1] proposed an end-to-end 3D convolutional network, introducing a dilation-based context module for efficient large-receptive-field context learning. Guo et al. [8] introduced VVNet to bridge the gap between 2D geometric features and 3D context by projecting 2D CNN-extracted features onto a volumetric space. Wang et al. [9] introduced the first generative adversarial network (GAN) for 3D semantic scene completion. This innovative approach leverages adversarial learning in both the output and latent spaces to improve results. Building on this, Wang et al. [10] presented ForkNet, which utilizes a unique architecture that features a single encoder and three generators that share a latent space for further incorporating geometric information by employing strategic connections. This design addresses inaccuracies in ground truth annotations, while multiple discriminators refine the details and realism of the completed scenes. Zhang et al. [11] developed the Cascaded Context Pyramid Network, aiming to limit the large-memory requirement and capture multi-scale spatial contexts by integrating local geometry details and multi-scale 3D contexts in a self-cascaded manner. Similarly, Cao et al. [14] developed MonoScene, which performs semantic completion from single RGB images by projecting 2D features into 3D and enhancing context awareness. Lastly, Li et al. [15] introduced VoxFormer, which employs occupancy prediction and self-attention mechanisms for depth correction and semantic segmentation, respectively. Though effective, these techniques are not tailored to deal with raw point cloud data, which limits their applicability in practical outdoor settings, especially those related to autonomous driving applications.

2.2. Point-Based Methods

These methods [2,21,22,23,24,25,26,27,28] directly apply convolution or transformer operations to original point clouds, leveraging their distance measurements and fine semantic descriptions for semantic scene completion. Roldao et al. [2] introduced LMSCNet, which uses lighter 2D convolutions to process point clouds, thus reducing the heavy computation caused by voxelization. Wang et al. [22] constructed octree-based CNNs (O-CNNs) with U-Net-like structures, where a novel output-guided skip connection is introduced to better preserve the geometric information in the input while also being robust to the input noise. This method reports high computational and memory efficiency and supports the construction of a deep network structure for 3D CNNs. Subsequently, Zhang et al. [24] developed a point cloud semantic scene completion network (PCSSC-Net), which utilizes a patch-based contextual encoder to learn point-level, patch-level, and scene-level geometric features, facilitating comprehensive information extraction from partial inputs through a divide-and-conquer strategy. Rist et al. [25] designed Deep Implicit Functions (DIFs) with spatial support derived from a 2D multi-resolution grid to produce a representation for both the geometry and semantics of 3D scenes. Note that this continuous representation avoids a trade-off between the achievable spatial output resolution and the extent of the 3D scene. Furthermore, Xiong et al. [26] proposed UltraLiDAR, which uses a sparse-to-dense data reconstruction pipeline to enhance data density and a zero-shot scheme to improve the generalization ability of the trained detection models. Xu et al. [27] presented CasFusionNet, deploying the following: (i) a global completion module (GCM) to produce an unsampled and completed but coarse point set, (ii) a semantic segmentation module (SSM) to predict the per-point semantic labels for the completed points, and (iii) a local refinement module (LRM), which further refines the coarse completed points and the associated labels. While point-based methods can effectively process point clouds, their high computational load and unstructured data processing remain open challenges.

2.3. Voxel-Based Methods

These methods [3,4,5,6,29,30,31,32] partition point clouds into 3D voxels and then perform convolution or transformer (or both) operations on the voxels to perform 3D semantic scene completion. Zhang et al. [29] introduced the concept of Spatial Group Convolution (SGC), utilizing sparse convolutions to efficiently process voxels by grouping them spatially, significantly reducing computational demands. Dai et al. [30] developed ScanComplete, which leverages a fully convolutional completion network for processing 3D scenes with arbitrary spatial extents, and employs a coarse-to-fine strategy to capture both local and global structures accurately. Zou et al. [31] proposed an Up-to-Down network (UDNet) to achieve large-scale semantic scene completion with an encoder–decoder architecture for voxel grids. The UDNet aggregates multi-scale context information to improve labeling coherence by UD blocks and expands the receptive field while preserving detailed geometric information through the atrous spatial pyramid pooling module. Furthermore, the designed multi-scale fusion mechanism efficiently aggregates global background information and improves semantic completion accuracy. Yan et al. [3] proposed JS3C-Net, which combines semantic segmentation and semantic scene completion: initially segmenting the scene semantically and then completing the scene’s voxel representation. This method includes a PVI module for transferring shape-aware knowledge between incomplete point clouds and their voxel counterparts. Li et al. [32] proposed a novel hybrid architecture combining a discriminative model and a generative model, along with a tailored training paradigm for implicit representation. The learned shape embeddings are treated as dense boundary values that constrain the semi-supervised signed distance learning function. Recently, Xia et al. [6] designed SCPNet, aiming to enhance single-frame model representations by distilling dense relational semantic knowledge from a multi-frame teacher model to the single-frame student by introducing a label rectification technique to remove the traces of dynamic objects in completion labels. Whereas voxel-based techniques have computational advantages over point-based methods, these gains are offset by a loss of information due to the discretization of the 3D space.

2.4. Multi-Modality-Based Methods

Semantic scene completion is achieved in [33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50] by adopting convolutions, transformers, or a combination of both, on different modalities like RGB images, depth images, and truncated signed distance functions (TSDFs), where the distance to the closest TSDF surface is computed at given 3D locations (usually voxel centers). Liu et al. [33] proposed a disentangled framework, sequentially carrying out 2D semantic segmentation, 2D–3D projection, and 3D semantic scene completion. In their technique, explicit semantic segmentation boosts performance and flexible fusion of sensor data brings extensibility. Guedes et al. [34] fused color and depth data to correct depth image inaccuracies. Garbade et al. [36] proposed a two-stream approach that leverages depth information and semantic information for 3D semantic scene completion. This method designs three-channel encoding for the semantic volume, which is not only memory efficient but also results in higher accuracies compared to single-channel encoding. It remains competitive with memory-expensive one-hot encoding. Chen et al. [38] devised a geometry-based strategy to embed depth information with low-resolution voxel representation, where a 3D sketch-aware feature embedding is proposed to explicitly encode geometric information. In addition, a lightweight 3D Sketch Hallucination module within an effective semantic scene completion framework is also devised to guide the inference of occupancy and semantic labels via a semi-supervised structure prior to the learning strategy. Cai et al. [42] developed the Scene-Instance-Scene Network (SISNet) to iteratively perform scene-to-instance and instance-to-scene conversions to encode object surrounding context and capture fine-grained object details within 3D scenes. Wang et al. [45] devised FFNet to address inconsistencies in RGB-D data and the uncertainty measurements of depth data during the fusion process of these two. Their method uses a frequency fusion module and a correlation descriptor to obtain the explicit correlation of the RGB-D feature and enhance its structural information, respectively. Recently, Wang et al. [48] proposed a Cleaner Self (CleanerS) framework, which addresses the noise problem in depth cameras by distilling intermediate supervision from a teacher network to a student network. Overall, multi-modal methods can benefit from the strengths of different data modalities. However, it is crucial to control the computational complexity of such methods for pragmatic reasons.

3. Methodology

The primary objective of 3D semantic scene completion is to infer complete geometric and semantic scene information from data that are both incomplete and sparse. The goal here is to reconstruct a complete scene by utilizing merely a single frame of point cloud data. Given an input point cloud $P\in {\mathbb{R}}^{N\times 3}$, we partition it into a grid of 3D voxels with the proportional dimensions $L\times W\times H$, indicating length, width, and height, respectively. The task is to accurately assign a class to each voxel within this specified grid. This means that, for each voxel, it needs to be determined whether the voxel is vacant or contains an object belonging to one of C semantic categories, represented as $c\in 0,1,2,\dots ,C-1$.

3.1. Methodology Overview

Our proposed network architecture for the task, presented in Figure 1, synergistically fuses 2D and 3D data representations through an integrated approach. It combines a 2D semantic scene completion network (SSCNet) and a 3D SSCNet to elevate the precision of semantic scene completion. The 3D SSCNet leverages multi-scale convolutional blocks (MSBs) that are not only parameter-efficient but are also capable of capturing rich local and global geometric features while maintaining a low memory footprint. Furthermore, our proposed layout-aware semantic blocks (LSBs) within the network are specifically tailored to comprehend the overall layout of the scene, which serves to accurately direct the feature reconstruction and recognition processes. Concurrently, the 2D SSCNet employs standard techniques found in existing image-based networks. Moreover, we present an innovative feature fusion module that is designed to integrate voxel-based characteristics from the 3D SSCNet with the 2D BEV attributes sourced from the 2D SSCNet. This strategic fusion facilitates the incorporation of essential structural and layout information from the 2D domain into the 3D semantic scene completion process, thereby enhancing the overall accuracy and depth of scene understanding. In the forthcoming sections, we will provide an in-depth discussion of the 3D SSCNet. This will be complemented by a comprehensive exploration of the feature fusion module, elucidating its role and significance in bridging the gap between two-dimensional and three-dimensional scene analysis.

3.2. 3D Semantic Scene Completion Network

While advancements in 3D semantic scene completion have been notable, existing methodologies exhibit certain limitations. For instance, SCPNet [6] adopts a sequential approach where scene completion precedes semantic segmentation. Although this strategy improves the overall semantic scene completion fidelity, it also substantially increases the number of network parameters. Additionally, the absence of downsampling layers in SCPNet necessitates considerable memory allocation, which is a bottleneck for efficient processing. In contrast, JS3C-Net [3] inverts this order by initiating the process with semantic segmentation, subsequently using the segmented features to infer the scene completion. This approach, while computationally economical, introduces the risk of perpetuating initial segmentation errors throughout the completion phase, potentially amplifying the loss of valuable information. Furthermore, S3CNet [4] innovatively employs sparse tensor operations to effectively construct the representation of the scene through integrated 2D and 3D modalities. This technique aims to achieve semantic completion across both dimensions. However, despite its computational efficiency, the 3D module of S3CNet may not capture the full geometric information effectively, from local details to global context, which is critical for a nuanced and comprehensive semantic scene completion. To address these challenges, we propose the incorporation of the 3D layout-aware semantic block (LSB) and the multi-scale convolutional block (MSB), which can augment the capability of the network for the reconstruction of both the semantic content and structural integrity of various scenes. The structures of the LSB and MSB are shown in Figure 2.

3.2.1. Layout-Aware Semantic Block

Recent studies [51,52] have highlighted the importance of capturing comprehensive contextual and layout information for effective 3D semantic segmentation, pointing to the challenges posed by the inherent sparsity of 3D voxels and their computational demands. To deal with these challenges and enhance context acquisition within a scene, our approach introduces the LSB. This block incorporates a set of dimensional decomposition residual (DDR) blocks [37], each employing a distinct dilation rate to capture extensive scene layout information efficiently, which was inspired by prior research [38]. Specifically, the DDR block is designed to offer a computational advantage by decomposing the conventional computation in 3D CNNs. In detail, the computational cost for a traditional block in a 3D CNN is quantified as $C^{in}\times C^{out}\times k\times k\times k$ for $C^{in}$ and $C^{out}$ input and output channels, respectively, with $k\times k\times k$ specifying the kernel size. By breaking down kernel operations into three successive layers, i.e., $1\times 1\times k$, $1\times k\times 1$, and $k\times 1\times 1$, the DDR block significantly reduces the computational memory demands from $C^{in}\times C^{out}\times \{k\times k\times k\}\to C^{in}\times C^{out}\times \left\{3k\right\}$. This reduces the parameter count to one-third of that required by a standard $3\times 3\times 3$ kernel in a 3D CNN while maintaining the capacity of the network for detailed spatial layout recognition. This strategic architectural decision enables our framework to utilize parameters more efficiently, ensuring that the network can expand its receptive field without significantly increasing the computational memory load. The detailed formula for the LSB is as follows:

$$\begin{array}{c}\hfill F_{out1}=\sigma (W_{d1}{F}_{3D}+b_1),\end{array}$$

(1)

$$\begin{array}{c}\hfill F_{out2}=\sigma (W_{d2}{F}_{out1}+b_2),\end{array}$$

(2)

$$\begin{array}{c}\hfill F_{out3}=\sigma (W_{d3}{F}_{out2}+b_3).\end{array}$$

(3)

where $d1$, $d2$, and $d3$ are dilation rates to modify the receptive fields of the layers, represented by $W_{d1}$, $W_{d2}$, and $W_{d3}$, respectively. The input feature map $F_{3D}$ is processed through these layers, with biases $b_1$, $b_2$, and $b_3$ added at each stage. The nonlinear activation function $\sigma$ is applied to produce outputs $F_{out1}$, $F_{out2}$, and $F_{out3}$, with $F_{out3}$ being the final output of the block. The increasing dilation factor enhances the ability of each layer to capture global contextual information from the input, expanding the effective receptive field and improving the capacity of the model to interpret the scene comprehensively.

3.2.2. Multi-Scale Convolutional Block

In SCPNet [6], the approach to 3D semantic scene completion involves a multi-path block featuring convolution kernels of varying sizes, $3\times 3\times 3$, $5\times 5\times 5$, and $7\times 7\times 7$, to capture multi-scale information. While this design allows for a wider receptive field, the deployment of larger convolutions, i.e., greater than $5\times 5\times 5$, significantly increases computational demands. To address this challenge and improve efficiency, we introduce the multi-scale convolutional block (MSB) based on the multi-path block within the SCPNet [6], which can extract multi-scale context in a computationally efficient manner, drawing inspiration from the principles underlying the VGG architecture [53]. Our solution involves substituting the larger convolution kernels with sequential applications of the $3\times 3\times 3$ convolution layers; specifically, two iterations mimic the effect of a single $5\times 5\times 5$ convolution, and three iterations approximate the effect of a $7\times 7\times 7$ convolution. This adjustment dramatically reduces computational costs from $5^3\times C^2=125C^2$ for a single $5\times 5\times 5$ convolution to $2\times \left(3^3\right)\times C^2=54C^2$ with our method, and from $7^3\times C^2=343C^2$ for a $7\times 7\times 7$ to $3\times \left(3^3\right)\times C^2=81C^2$. By adopting this strategy, not only does our framework efficiently gather critical local and global contextual information but it also ensures no significant increase in computational overhead and memory usage. Therefore, this adaptation permits a more nuanced and comprehensive analysis of semantic scenes, leveraging the benefits of multi-scale processing while circumventing the limitations associated with the use of large convolutional kernels. The MSB in our model incorporates multi-scale convolution (MC) layers that utilize convolutions with varying kernel sizes to extract features from different receptive fields. The formula of the MC is shown below:

$$\begin{array}{c}\hfill F_{out}={\sum }_{i=1}^N{W}_i{F}_{in}.\end{array}$$

(4)

where $W_i$ is the kernel weights at scale i, $F_{in}$ and $F_{out}$ are the input and output feature maps.

3.3. 2D Semantic Scene Completion Network

To enhance the semantic scene completion performance of our proposed 3D semantic scene completion framework, we integrate a specialized lightweight 2D semantic scene completion network, inspired by S3CNet [5]. This network leverages a 2D encoder–decoder architecture optimized for semantic scene completion. Utilizing a bird’s-eye-view (BEV) approach enables the transformation of the 3D scene into a 2D feature map with dimensions $L\times W$. This transformation effectively converts the complex 3D spatial data into an image-like format, making them amenable to conventional 2D convolutional neural network (CNN) methodologies. Thus, the process initiates with the deployment of one convolutional layer, specifically designed to extract preliminary voxel features from the 3D data. This is followed by a crucial feature dimension reduction phase, where the initial voxel features are compressed into BEV features. These BEV feature representations not only retain spatial and semantic information but also significantly reduce computational load and memory requirements. Therefore, the 2D SSCNet can facilitate data processing and enhance the overall capability of our framework to achieve precise and computationally efficient semantic scene completion.

In the 2D BEV network framework, we strategically deploy a series of 2D convolution layers, each developed to expand the receptive field of the network. The architecture undergoes a systematic progression through four downsampling stages, wherein each stage halves the resolution, thereby incrementally simplifying the level of detail for efficient feature processing. To further augment the capability of the network and ensure a more comprehensive feature integration, we integrate skip connections across the architecture. Subsequently, the refined outputs generated by the 2D SSCNet are put into an advanced feature fusion module.

3.4. Feature Fusion Module

We implement a novel, two-stage feature fusion module (FFM) to combine data from the two previously derived feature maps. The FFM fuses the detailed 2D SSCNet outputs with the geometric 3D SSCNet features. This fusion process leverages the spatial and semantic information richness of 3D data and the spatial layout information of 2D data to achieve improved performance in semantic scene completion tasks. The structure of the FFM network is illustrated in Figure 3. The first stage, termed the feature exchange stage, facilitates separate pathways for the features derived from each prior network. This architecture is crucial for enabling a thorough and efficient exchange of information between the two- and three-dimensional domains, ensuring that the strengths of each pathway are used to enhance the other. The second stage facilitates the feature fusion, where previously segregated features in the feature exchange stage are intricately recombined into their original dimensional contexts. This recombination is achieved through a channel mixing and embedding strategy, which is essential to enhance model comprehension and to represent complex spatial relationships more accurately.

This two-stage approach ensures that our framework not only maintains the integrity of the distinct dimensional features but also leverages the unique advantages of each. By facilitating an extensive exchange of information in the feature exchange stage and by employing a sophisticated fusion technique in the feature fusion stage, our model achieves a more holistic and nuanced understanding of the scene. Notably, these two stages are designed to complement each other and cannot function independently. Consequently, this enhances the overall capacity of the model to perform semantic scene completion tasks with greater accuracy and efficiency, capturing the full complexity and richness of spatial relationships within the scene.

3.4.1. Feature Exchange Stage

Our proposed FFM is critical for integrating multi-dimensional feature sets. It is designed to enhance the semantic scene completion by enabling an efficient exchange and fusion of feature maps from different dimensions. In the feature exchange stage, features from the 3D SSCNet ($F_{3d}$) and the 2D SSCNet ($F_{2d}$) undergo a structured exchange. Here, each set of features is bifurcated into two pathways facilitated by a linear embedding, which generates feature vectors $F_{nd}^{\prime }$ and $F_{nd}^{″}$ given an n-dimensional feature set. Mathematically, the transformation for the n-dimensional features can be articulated as follows:

$$\begin{array}{c}\hfill F_{nd}^{\prime },F_{nd}^{″}=Linear\left(F_{nd}\right),\end{array}$$

(5)

where the function $Linear$ embodies the linear embedding process. The feature vectors derived from the 3D and 2D SSCNets are combined into $F_{merge}$, creating a robust platform for dynamic information exchange. This structured exchange is instrumental in fostering a synergistic relationship between the modalities, enhancing the effectiveness of the model. Consequently, the vectors from each modality ($F_{2d}^{\prime }$ and $F_{2d}^{″}$, $F_{3d}^{\prime }$ and $F_{3d}^{″}$) are combined, serving as the input into a linear projection layer, followed by normalization to yield an integrated feature representation $F_{merge}$. Then, the feature maps $F_{merge}$, $F_{3d}$, and $F_{2d}$ are finally combined to form the input of the feature fusion stage $F_{merge}^{\prime }$, i.e.,

$$\begin{array}{cc}\hfill F_{merge}& =Add(F_{nd}^{\prime },F_{nd}^{″}),\hfill \end{array}$$

(6)

$$\begin{array}{cc}\hfill F_{merge}^{\prime }& =N\left(P\left(F_{3d}+F_{2d}+F_{merge}\right)\right).\hfill \end{array}$$

(7)

where P signifies the linear projection operation that maps the combined feature vectors to a higher-level representation, and N represents the normalization operation that stabilizes the learning process by normalizing the feature distributions. By fusing the detailed spatial geometric information features captured by the 3D SSCNet with the textural and spatial layout contextual information identified by the 2D SSCNet, our framework ensures that the output is not only detail-rich but also guarantees the accuracy of the semantic structure of the scene.

3.4.2. Feature Fusion Stage

During the feature fusion stage, our methodology employs a sophisticated approach to ensure the accurate fusion of features from both 3D and 2D SSCNets. This stage first uses a $3\times 3\times 3$ convolution layer, which is adept at efficiently extracting and combining features across different channels. This step improves segmentation precision by extracting contextual information from neighboring regions. Drawing upon the principles established in advanced neural network architectures [54,55], this stage incorporates an element: a depth-wise $3\times 3\times 3$ convolution layer. We then establish a skip-connection structure that significantly augments the feature integration process and enables the accurate merging of feature vectors. By adopting this, our network capitalizes on the wealth of contextual information inherent in the scene, thus fostering a more robust and comprehensive representation of semantic information. This fusion strategy amplifies the descriptive capabilities of the model while maintaining computational efficiency, which is essential for the scalability of real-world applications.

3.5. Overall objective

The overall loss function of our network intuitively combines two pivotal components: the cross-entropy loss (CE) and the Lovasz-softmax loss. The Lovasz-softmax loss is designed to optimize the mean intersection-over-union (mIoU), a key performance metric in semantic segmentation, and can be mathematically represented as

$${\mathcal{L}}_{Lovasz}=\frac{1}{\left|C\right|}\sum _{c=1}^{C}J\left(e\left(c\right)\right),$$

(8)

where J signifies the Lovasz extension of the IoU, manifesting as a piecewise linear function that directly targets the minimization of the mIoU error, and $e\left(c\right)$ denotes the vector of errors for each class c within the set of classes C. The cross-entropy loss function, a staple in classification tasks, is utilized in parallel, minimizing the prediction error and thus improving accuracy, and is given as

$${\mathcal{L}}_{CE}=-\sum _i{y}_{i}log\left(\widehat{y_i}\right),$$

(9)

where ${\widehat{y}}_i$ and $y_i$ represent the predicted probability distribution and the ground truth label distribution for the ith element. Integrating these loss functions, the overall objective function for our model is formulated as

$$\mathcal{L}={\mathcal{L}}_{CE}+\beta {\mathcal{L}}_{Lovasz}.$$

(10)

where $\beta$ is the balancing coefficient that modulates the influence of each loss component. This calibration allows us to maintain a balance between optimizing for overall accuracy and directly improving the mIoU metric, thereby leading to an accurate model that is also aligned with the core evaluation criteria for semantic scene completion tasks. Through this dual-component loss function, the network is trained to achieve an understanding of the scene, enabling it to classify various elements accurately within the semantic landscape.

4. Experiments

The proposed network is rigorously tested on the SemanticKITTI dataset [7], a comprehensive LiDAR-based benchmark for semantic scene completion. This section outlines the characteristics of the dataset and the metrics utilized for evaluation. Next, we delve into the specific implementation details of the proposed network, laying the groundwork for its operational understanding. Then, we report our detailed experimental results and compare them to the existing methods that are also validated on SemanticKITTI. Finally, we conduct a series of ablation studies designed to evaluate the impact of individual components of the proposed approach on overall performance. Through this systematic exploration, we aim to substantiate the efficacy of our approach and its contribution to the advancement of semantic scene completion tasks.

4.1. Datasets and Evaluation Metrics

Datasets. In our study, we rigorously assess the efficacy of our proposed methodology using the widely recognized semantic scene completion benchmark, i.e., SemanticKITTI [7]. SemanticKITTI presents a comprehensive dataset composed of LiDAR (Velodyne HDL-64E laser scanner, Velodyne Lidar, San Jose, CA, USA) point clouds, annotated at the point level. Specifically tailored for semantic scene completion tasks, SemanticKITTI provides ground truth semantic labels across a sequence of successive point cloud frames. Adhering to the established guidelines, we select individual frames from the raw point cloud data, confined within $\left[0\sim 51.2,\mathrm{m},-25.6\sim 25.6,\mathrm{m},-2\sim 4.4,\mathrm{m}\right]$ spatial dimensions. This selected frame is subsequently partitioned into voxels with a resolution of 0.2 m, resulting in a detailed scene representation encapsulated within a grid of $256\times 256\times 32$ voxels. Our objective is to reconstruct an exhaustive scene model within these incomplete and partial scenes. The SemanticKITTI dataset encompasses a total of 22 sequences and has 19 distinct semantic categories designated for both training and evaluation purposes. For training, we utilize Sequences 0–7 and 9–10, which collectively offer a compilation of 3834 scans. Sequence 8, comprising 815 scans, is allocated as our validation dataset. Sequences 11–21, including 3901 scans, served as test datasets. This follows the same protocol as LMSCNet [2].

Evaluation metrics. To evaluate our proposed L3D-SCN framework and compare our results to existing works, we deploy a set of evaluation metrics for semantic scene completion, as established by Song et al. [34]. Our evaluation focuses on two critical aspects: (i) the accurate geometry reconstruction of the scene, and (ii) the precise semantic segmentation of each voxel. Thus, the core metrics we deploy for our assessment of semantic scene completion performance are the intersection-over-union (IoU), reflecting the geometric completion performance of our reconstructions, and the mean intersection-over-union (mIoU), evaluating the semantic accuracy across multiple categories. These metrics are computed as follows:

$$\begin{array}{cc}\hfill Io{U}_i& =\frac{T{P}_i}{T{P}_i+F{N}_i+F{P}_i},\hfill \end{array}$$

(11)

$$\begin{array}{cc}\hfill mIo{U}_i& =\frac{1}{C}\sum _{c=1}^{C}Io{U}_i,\hfill \end{array}$$

(12)

where $T{P}_i$, $F{P}_i$, and $F{N}_i$ represent the true positives, false positives, and false negatives for the ith class, providing an exact measure of our model per class. C is the total number of classes. These metrics provide a holistic evaluation of the ability of our framework to not only complete but also semantically understand scenes, thus offering a robust evaluation of our contribution to the semantic scene completion domain.

4.2. Implementation Details

For the experimental evaluation of our proposed method, we employed one Nvidia GeForce 4090 GPU (Nvidia, Santa Clara, CA, USA), equipped with 24 GB of graphical RAM. Our training was conducted on a single GPU, enabled by the resource efficiency of our method. Throughout the experiments, each model was trained for a total of 40 epochs. The training protocol for achieving the best performance with our proposed network incorporated the Adam optimization algorithm, starting with a learning rate of 0.01 and a weight decay of 0.0001 to mitigate overfitting. Additionally, we integrate an exponential learning rate scheduler to iteratively reduce the learning rate.

4.3. Results

Our comparative analysis highlights the performance of our proposed method against six state-of-the-art methods that use the SemanticKITTI validation dataset: LMSCNet [2], UDNet [31], Local-DIFs [25], SSA-SC [5], JS3C-Net [3], and SSC-RS [56]. The results are summarized in

Table 1. Quantitative results of semantic scene completion on the SemanticKITTI validation set. Best-performing single-frame results in each column are boldfaced. w/o denotes “without”, and M denotes “millions”.

Methods	mIoU	Completion	Precision	Recall	Parameters (M)	Car	Bicycle	Motorcycle	Truck	Other Vehicle	Person	Bicyclist	Motorcyclist	Road	Parking	Sidewalks	Other Ground	Building	Fence	Vegetation	Trunk	Terrain	Pole	Traffic Sign
LMSCNet-SS [2]	16.8	54.2	-	-	0.4	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
UDNet [31]	20.7	58.9	78.5	70.9	-	42.1	1.8	2.3	25.7	11.2	2.5	1.2	0.0	67.0	20.3	37.2	2.2	36.0	11.9	40.1	18.3	45.8	23.0	3.8
Local-DIFs [25]	26.1	57.8	-	-	-	51.3	4.3	3.3	32.3	10.6	15.7	24.7	0.0	71.2	31.8	43.8	3.3	38.6	13.6	40.1	19.6	50.6	25.7	14.0
SSA-SC [5]	24.5	58.2	78.5	69.3	41.0	47.0	9.2	7.4	39.7	19.1	6.3	3.2	0.0	72.8	21.0	44.3	4.1	41.5	15.2	41.9	22.0	49.5	17.9	4.4
JS3C-Net [3]	24.0	57.0	71.5	73.5	3.1	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
SSC-RS [56]	24.8	58.6	78.5	69.8	23.0	46.8	1.5	6.9	41.5	19.8	6.2	1.5	0.0	73.8	26.6	45.3	2.1	41.0	15.8	42.6	22.2	50.6	17.9	4.6
S3CNet [4]	33.1	57.1	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
SCPNet [6] w/o downsampling	37.2	49.9	-	-	-	50.5	28.5	31.7	58.4	41.4	19.4	19.9	0.2	70.5	60.9	52.0	20.2	34.1	33.0	35.3	33.7	51.9	38.3	27.5
SCPNet [6] w downsampling	33.1	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
Ours	35.7	51.4	83.5	57.2	40.0	46.8	20.7	26.7	49.6	41.1	8.3	4.4	0.0	73.7	58.8	54.0	27.6	36.4	38.2	40.3	35.7	56.6	37.7	21.4

. Note that the performance of SCPNet on the validation set depends on the implementation of downsampling within its network architecture. Its mIoU is 33.1 when incorporating downsampling layers. In contrast, our proposed network architecture, which integrates two layers of downsampling, attains an mIoU of 35.7 on the same validation set. Although SCPNet achieves a higher mIoU of 37.2 without downsampling, surpassing our results, it requires significantly greater memory resources, rendering deployment on a single RTX 4090 GPU impractical.

Our proposed semantic scene completion method outperforms existing works in the key metric of mIoU. However, it exhibits lower performance in IoU when compared to these methods. This discrepancy can be attributed primarily to the enhanced ability of the method to accurately segment small, distant, and dense objects. These types of objects, while crucial for detailed scene analysis, constitute a relatively minor proportion of the total scene composition. Consequently, their improved segmentation does not substantially influence the overall IoU value. In detail, we demonstrate superior performance in accurately completing and segmenting smaller objects like bicycles, motorcycles, poles, and traffic signs. Thus, these results support the enhanced capabilities of our model in capturing and interpreting the nuances of small-scale features within the scene. Moreover, the proposed method exhibits remarkable improvements in segmenting larger object categories, including trucks, parking spaces, sidewalks, other ground types, fences, trunks, and terrain. This balanced proficiency indicates that our approach does not singularly prioritize small objects but extends its efficacy across objects of varying sizes and complexities.

We also employ the pre-trained models and official implementations of LMSCNet [2], SSA-SC [5], and SSC-RS [56] to generate visual, qualitative results on the SemanticKITTI validation set, which we use to compare our proposed approach against. The qualitative results are presented in Figure 4. The visual analysis reinforces the advantages of our model in segmenting planar categories like parking areas, other ground types, and terrain, which is supported by our quantitative findings. Furthermore, while other comparative methods occasionally produce distorted representations, our proposed approach consistently delivers accurate and recognizable scene reconstructions. Both the qualitative and quantitative results emphasize the comprehensive advancement of the proposed method in semantic scene completion, highlighting its potential in the field. Additionally, we present the failure cases in Figure 5. The inability of the proposed methods to accurately complete objects such as bicycles and bicyclists can be attributed to the insufficient number of examples in the training dataset. This limitation restricts the capacity of the model to learn the diverse features and variations associated with these objects, which is essential for accurate object completion.

In terms of parameter efficiency, LMSCNet [2], JS3C-Net [3], and SSC-RS [56] demonstrate higher efficiency compared to our proposed method. However, it is crucial to recognize that the performance of these models (LMSCNet [2], JS3C-Net [3], and SSC-RS [56]) falls below acceptable thresholds. This indicates that despite their lower parameter counts, their practical applicability might be constrained. Conversely, our method outperforms these models and maintains fewer parameters than SSA-SC [5], which has 41 million parameters. Additionally, while JS3C-Net [3] records the longest runtime of 0.28 s per sample on the same GPU (NVIDIA RTX 4090), our method processes each sample in just 0.15 s. This suggests that our method not only delivers higher accuracy, but does so with greater parameter efficiency and speed. Such advantages are particularly valuable in scenarios computational resources are limited, highlighting the practical superiority of our approach.

However, there are two shortcomings in the current methodology. The first shortcoming is the lack of specific datasets. Currently, there is only one dataset, SemanticKITTI [7], in the 3D point cloud semantic scene completion. This limitation restricts the ability to train and test our models under varied conditions and may affect the generalizability of our approach. To address this, we plan to develop comprehensive datasets for 3D point cloud semantic scene completion. These datasets will support the advancement of our research and will be valuable resources for the community. By publishing these datasets, we aim to enhance model performance and achieve scene completions that are closer to real-world scenarios. Another shortcoming is network efficiency. Our current model architecture has more parameters compared to LMSCNet [2], which could hinder its applicability in scenarios where computational resources are limited. To overcome this, we are in the process of designing a more lightweight network. This revision will aim to maintain or improve the accuracy of our model while significantly reducing its computational demands, thus facilitating easier deployment and application in practical settings.

4.4. Ablation Studies

Through a series of methodical experiments, we analyze the impact of distinct blocks proposed in our framework, aiming to quantify their respective roles in enhancing the overall performance of the model. We detail the results of our ablation studies in

Table 2. Experimental results for removing different modules from our overall scene completion framework, evaluated on the SemanticKITTI validation dataset. Note that w/o denotes “without”.

Method	mIoU	Completion	Car	Bicycle	Motorcycle	Truck	Other Vehicle	Person	Bicyclist	Motorcyclist	Road	Parking	Sidewalks	Other Ground	Building	Fence	Vegetation	Trunk	Terrain	Pole	Traffic Sign
w/o LSB	28.9	44.9	41.8	8.9	3.2	38.8	19.8	7.8	3.1	7.3	70.6	56.8	49.3	30.6	32.4	37.9	33.8	28.5	53.3	17.4	5.8
w/o MSB	26.7	39.1	39.0	6.9	13.7	42.3	22.0	5.2	0.0	0.0	66.4	48.7	46.2	18.5	26.5	37.1	27.6	34.7	44.2	23.2	4.6
w/o 2D SSCNet	31.9	46.4	44.8	14.3	29.8	45.9	18.8	9.3	2.5	2.7	72.5	57.2	52.2	32.6	32.2	37.3	34.8	35.0	54.0	22.9	7.5
w/o FFM	33.9	49.0	45.2	30.1	33.3	50.6	38.5	9.4	1.6	0.0	71.4	56.3	50.6	27.6	34.2	35.6	36.7	36.3	55.6	25.2	5.2
Full Model	35.7	51.4	46.8	20.7	26.7	49.6	41.1	8.3	4.4	0.0	73.7	58.8	54.0	27.6	36.4	38.2	40.3	35.7	56.6	37.7	21.4

. Additionally, we complement our analysis with qualitative results, as shown in Figure 6. These visual representations allow us to illustrate the practical effect of each block on the semantic scene completion task, offering intuitive insights into how each component contributes to refining the model output. Through these evaluations, we report the significance of each proposed block within the overall proposed architecture, showcasing their collective synergy in achieving superior performance for the scene completion task.

Effect of layout-aware semantic blocks. In our comprehensive analysis, we explore the impact of incorporating LSB within our framework, specifically examining its performance on the SemanticKITTI validation set. A detailed comparison is provided, contrasting scenarios with and without integrating the LSB. As illustrated in Table 2, the inclusion of LSBs markedly enhances the ability of our model to classify accurately and segment “planar” categories such as roads, sidewalks, other ground types, and vegetation. This improvement highlights the pivotal role of the LSB in enabling the model to effectively capture and interpret spatial and semantic scene information. Through this focused investigation, we demonstrate the significant value added by these blocks to our overall methodology, reinforcing the performance of our model for semantic scene completion.

Effect of multi-scale convolutional blocks. As part of our ablation studies, we scrutinized the impact of integrating the MSB into our framework. The comparative analysis, as detailed in

Table 3. Impact of multi-scale information blocks on the performance with SemanticKITTI val dataset.

Methods	mIoU	Completion	Car	Bicycle	Motorcycle	Truck	Other Vehicle	Person	Bicyclist	Motorcyclist	Road	Parking	Sidewalks	Other Ground	Building	Fence	Vegetation	Trunk	Terrain	Pole	Traffic Sign
Our Method w/o MSB	26.7	39.1	39.0	6.9	13.7	42.3	22.0	5.2	0.0	0.0	66.4	48.7	46.2	18.5	26.5	37.1	27.6	34.7	44.2	23.2	4.6
Our Method w MSB	35.7	51.4	46.8	20.7	26.7	49.6	41.1	8.3	4.4	0	73.7	58.8	54.0	27.6	36.4	38.2	40.3	35.7	56.6	37.7	21.4

, distinctly highlights the significant enhancements brought by these blocks. With the incorporation of MSBs, the performance of our model in semantic scene completion saw an increase from 26.7 mIoU to 35.7 mIoU, alongside an improvement in geometric scene completion from 39.1 IoU to 51.4 IoU. This substantial uplift in performance is particularly noteworthy across smaller object categories, such as bicycles, motorcycles, persons, and bicyclists, and “plane” categories like roads, parking areas, sidewalks, and other ground types. These improvements highlight the effectiveness of MSBs in capturing both nuanced local geometric details and global contextual scene information. Thus, the introduction of MSBs represents a significant advancement in our methodology and demonstrates their pivotal role in achieving superior performance, particularly in processing complex scenes with a diverse range of object sizes and spatial relationships.

Effect of 2D SSCNet. In our evaluation, we assess the impact of integrating a 2D SSCNet within our proposed framework. This comparative analysis, detailed in

Table 4. Impact of 2D SSCNet on the performance with SemanticKITTI val dataset.

Methods	mIoU	Completion	Car	Bicycle	Motorcycle	Truck	Other Vehicle	Person	Bicyclist	Motorcyclist	Road	Parking	Sidewalks	Other Ground	Building	Fence	Vegetation	Trunk	Terrain	Pole	Traffic Sign
Our Method w/o 2D SSCNet	31.9	46.4	44.8	14.3	29.8	45.9	18.8	9.3	2.5	2.7	72.5	57.2	52.2	32.6	32.2	37.3	34.8	35.0	54.0	22.9	7.5
Our Method w 2D SSCNet	35.7	51.4	46.8	20.7	26.7	49.6	41.1	8.3	4.4	0	73.7	58.8	54.0	27.6	36.4	38.2	40.3	35.7	56.6	37.7	21.4

, highlights the performance differential when incorporating the 2D SSCNet versus its absence. The inclusion of 2D SSCNet notably enhances the performance of our method, yielding a 3.8 mIoU increase and a 5.0 IoU improvement for geometric completion. Table 2 demonstrates that the integration of 2D SSCNet significantly improves performance across various categories, including cars, bicycles, trucks, other vehicle types, poles, and traffic signs. These improvements suggest that the 2D SSCNet effectively complements the three-dimensional analysis by providing spatial layout information from the two-dimensional space in the semantic scene completion process. As a result, the addition of this component demonstrates the importance of leveraging two-dimensional insights to enrich the perception and interpretation of the model in three-dimensional spaces.

Effect of feature fusion model. The comparative analysis, as summarized in

Table 5. Impact of Feature Fusion Model on the performance with SemanticKITTI val dataset.

Methods	mIoU	Completion	Car	Bicycle	Motorcycle	Truck	Other Vehicle	Person	Bicyclist	Motorcyclist	Road	Parking	Sidewalks	Other Ground	Building	Fence	Vegetation	Trunk	Terrain	Pole	Traffic Sign
Our Method w/o FFM	33.9	49.0	45.2	30.1	33.3	50.6	38.5	9.4	1.6	0.0	71.4	56.3	50.6	27.6	34.2	35.6	36.7	36.3	55.6	25.2	5.2
Our Method w FFM	35.7	51.4	46.8	20.7	26.7	49.6	41.1	8.3	4.4	0	73.7	58.8	54.0	27.6	36.4	38.2	40.3	35.7	56.6	37.7	21.4

, showcases the improvements made when integrating the FFM in our method. This integration elevates the performance for semantic scene completion from 33.9 mIoU to 35.7 mIoU and for geometric completion from 49.0 IoU to 51.4 IoU. This evidences the effectiveness of our proposed FFM, which is designed with feature exchange and feature fusion stages. The feature exchange stage facilitates a comprehensive interchange of attributes between different modal inputs, while the feature fusion stage adeptly integrates these attributes, optimizing the overall semantic and geometric understanding of the scene. Thus, it is clear that these two stages are not only pivotal in enhancing the capability of the model to process multidimensional information fusion but also illustrate the substantial benefits of two such stages in improving the accuracy and detail of semantic scene completion tasks.

5. Conclusions

This work introduces an integrated framework for 3D semantic scene completion tasks, which intuitively combines the strengths of both 3D and 2D semantic scene completion networks (SSCNets). The parameter-efficient and effective multi-scale convolutional block (MSB) within 3D SSCNet is developed to aggregate multi-scale features. This enhances the capability of the network to accurately segment small, distant, and crowded objects, addressing a common shortcoming of the existing methods. Additionally, the 3D SSCNet incorporates a layout-aware semantic block (LSB), which plays a crucial role in understanding the layout of the overall scene, facilitating the precise guidance of feature reconstruction and recognition. Concurrently, the 2D SSCNet is developed to process bird’s-eye-view (BEV) features, augmenting the layout information for three-dimensional semantic scene completion. The FFM is designed to foster effective fusion between the 2D SSCNet and 3D SSCNet data. This module ensures a robust scene completion process by leveraging the combined strengths of 3D and 2D data, thereby overcoming the challenges associated with fusing disparate multimodalities. Our extensive evaluation, conducted on the widely recognized SemanticKITTI dataset, demonstrates the superior performance of our method when compared to the state-of-the-art.