Lightweight Semantic Architecture Modeling by 3D Feature Line Detection

Abstract

Existing architecture semantic modeling methods in 3D complex urban scenes continue facing difficulties, such as limited training data, lack of semantic information, and inflexible model processing. Focusing on extracting and adopting accurate semantic information into a modeling process, this work presents a framework for lightweight modeling of buildings that joints point clouds semantic segmentation and 3D feature line detection constrained by geometric and photometric consistency. The main steps are: (1) Extraction of single buildings from point clouds using 2D-3D semi-supervised semantic segmentation under photometric and geometric constraints. (2) Generation of lightweight building models by using 3D plane-constrained multi-view feature line extraction and optimization. (3) Introduction of detailed semantics of building elements into independent 3D building models by using fine-grained segmentation of multi-view images to achieve high-accuracy architecture lightweight modeling with fine-grained semantic information. Experimental results demonstrate that it can perform independent lightweight modeling of each building on point cloud at various scales and scenes, with accurate geometric appearance details and realistic textures. It also enables independent processing and analysis of each building in the scenario, making them more useful in practical applications.

1. Introduction

Accurate 3D understanding and presentation technology of complex urban scenes play an important role in addressing issues of unmanned localization, unmanned aerial vehicle (UAV) autonomous navigation, and urban scene planning. Among them, the 3D semantic modeling of buildings is particularly necessary for 3D scene perception because the architectures occupy a large proportion of urban scenes. However, existing mainstream 3D semantic modeling methods, such as PointNet by Qi et al. [1] and PointNet++ by Qi et al. [2], rely heavily on a large amount of semantic labeled training data, which are severely lacking in urban 3D point cloud scenes. In addition, the lack of fine-grained semantic information of buildings cannot meet the requirements of high-precision scene localization and perception (the robots need to be parked accurately in front of a specific unit door of a building). Furthermore, the number of point clouds in a real urban scene is generally beyond the million level. Even if accurate semantic information can be extracted, the vast amount of discrete point cloud data need to occupy large storage resources, which are not suitable for low-power embedded robot devices. Therefore, further lightweight processing of point cloud data is necessary. Existing lightweight modeling techniques, such as Poisson reconstruction Kazhdan and Hoppe [3] and VSA Cohen-Steiner et al. [4], utilize 3D point clouds to produce a lightweight mesh of the scene. However, building edges directly extracted from discrete point clouds are serrated and not smooth enough. Additionally, given that these methods construct a unified mesh of the whole scene, each building target in the mesh is connected and indistinguishable, thereby interrupting the generation of fine-grained semantics for the target building. None of them can directly generate lightweight models with semantic information and smooth building edges. In summary, the existing methods cannot achieve high-precision building lightweight modeling with fine-grained semantic information. Hence, they are still far from being used in practical applications.

Focusing on the above issues, this paper proposes a fine-grained segmentation-based lightweight semantic modeling framework. Combining the semantic information of multi-view images with the geometric structure of the 3D point cloud, a complex urban scene is first segmented coarse-grained to extract each building point cloud in the scene. Second, constrained by 3D feature lines, a vectorized 3D model that contains building element details is generated by using the multi-view fine-grained segmentation-based lightweight modeling method. In practical applications, our lightweight fine-grained semantic model cannot only significantly reduce the storage load but also improve the understanding of semantic details of 3D architecture. The main contributions can be summarized as follows:

• Joint 2D–3D Semi-supervised Semantic Object Segmentation: The existing dataset that can be applied to the semantic segmentation of large-scale complex urban scenes is extremely limited, thus seriously affecting the generalization ability and segmentation accuracy of networks. Moreover, urban scene data sets are accompanied by a large number of manual semantic annotations, and constructing new data sets is also highly labor intensive. To solve the aforementioned issues, we propose a semi-supervised 2D-3D joint semantic segmentation method that learns robust 3D scene semantic representations without requiring semantic labeling of the point cloud. Our method enforces multi-view consistency and geometric co-planarity constraints, ensuring that pixels obtained by projecting the same 3D point or belonging to the same plane onto multiple views have the same semantics. Using only a limited amount of image semantic labeling, our semi-supervised method effectively trains a robust semantic segmentation model. Our approach effectively addresses the problem of limited labeled data and outperforms complex 3D or KNN convolutions, as it relies on more efficient 2D convolutions.
• Plane Constrained Multi-view Accurate 3D Feature Lines Extraction and Optimization: For vectorization modeling, a highly accurate plane fitting of the building point cloud should be constrained by an accurate geometric boundary. However, 3D point cloud data are discrete and sparse, and are not smooth enough to extract the boundary directly. In addition, boundaries extracted from 2D images are smooth but discontinuous due to occlusions and shadows. To solve the above problems, we project the multi-view edge information to the 3D reference plane to generate continuous and smooth 3D feature lines of the building based on the edge detection of multi-view images and point cloud reference plane parameters. Among them, the introduction of multi-view image consistency not only ensures the smooth edge but also solves the boundary discontinuity caused by occlusion. Finally, point clouds are replaced with vector planes generated by the shortest loop algorithm, which achieves a highly accurate 3D vector modeling of building targets in complex scenes.
• Lightweight Semantic Modeling Framework in Fine-grained Elements Segmentation Level: The existing semantic modeling approach for the urban scene is a global meshing of the overall scene, where each element (e.g., buildings, road, trees, etc.) mesh is interconnected and indistinguishable. However, this kind of modeling cannot process and analyze individual objects separately, thereby significantly limiting the flexibility of semantic modeling. The worst factor is that more complex geometric representations and a large number of planes and points are required, which will waste more storage resources and are not suitable for practical applications. To solve the above problems, we propose a novel semantic modeling framework for buildings. Neglecting the fact that we use the plain modeling method, the accurate building models are generated provided with the aforementioned precise point cloud segments and 3D feature lines.In the building modeling stage, multi-view images and 3D point cloud clustering information are referenced to extract each building and vectorized individually. This not only guarantees the accuracy of the 3D geometric structure of the building but also reduces modeling complexity. Finally, a feature line that constraints fine-grained element segmentation method is proposed to obtain detailed semantics of building elements (doors and windows) and integrated with the lightweight model to achieve a highly accurate 3D vectorization modeling of individual buildings with fine-grained semantic information.

2. Related Work

We explore generate accurate point cloud segmentation and detect precise 3D feature lines to aid semantic and lightweight architecture modeling. Given the emphasis on building modeling and semantic segmentation, we view the related works on two main aspects: lightweight modeling and semantic segmentation of buildings.

2.1. Vectorized Modeling

Building modeling is a popular research topic because of its various applications and development of point cloud generation techniques.

Several early works concentrate on robust and accurate modeling. Kazhdan and Hoppe [3] proposed a faster and higher-quality surface reconstruction method by extending the Poisson surface reconstruction algorithm. Driven by geometric error, Cohen-Steiner et al. [4] and Garland and Heckbert [5] tended to simplify surface mesh to reduce model complexity. However, relying on accurate surface meshes, these methods are sensitive to outliers.

Considering that building models are composed of fixed geometric primitives, several works use bounding boxes to represent building models under the Manhattan World assumption. Li et al. [6] fitted buildings with bounding boxes and formulated building modeling as a linear integer programming problem. Nan et al. [7] adopted a similar strategy and solved the building modeling problem in a way that minimized energy.

Viewing the building modeling reconstruction as a labeling problem, Li et al. [8] selected a subset of candidate boxes in Markov random field formulation, which could be easily solved by the graph-cut algorithm. Although most buildings can be reconstructed by the above-mentioned bounding box-based methods, modeling for buildings with complex architecture remains a challenge.

To solve this problem, few works emphasized the modeling of complex building roofs. Zhou and Neumann [9], under the hypothesis of vertical walls, extended classic dual contour into a 2.5D method to recover buildings with a complex roof structure. By solving an energy minimization problem of 2.5D roof section arrangement, Lafarge and Mallet [10] modeled complex roof and vertical walls with a hybrid representation that is combined with mesh patches and regular geometric primitives. Nevertheless, under the vertical wall assumption, building facade details, which are important in building modeling, are neglected.

PolyFit [11], which was developed by Nan and Wonka [11], proposed a method for modeling random building surfaces that were composed of planes. In PolyFit, taking organized point clouds as input, the building surface is composed of faces that are selected from a large set of face candidates by solving an energy minimization problem using binary linear programming. In spite of its robustness on most building models, the time cost of binary linear programming is unacceptable for buildings with a large number of faces.

2.2. Semantic Segmentation

Semantic segmentation has been a popular topic in computer vision, robotics, and remote sensing for decades and plays a vital role in building reconstruction. Segmentation on point clouds is a common method to reconstruct buildings with semantic information, and most algorithms can be categorized into supervised and unsupervised formula.

2.2.1. Unsupervised Point Cloud Semantic Segmentation

Region growing [12] was first proposed to segment 2D images and is widely used in point cloud segmentation today. It grows regions from seed points and determines whether near points belong or not to the region according to the criteria that combine features between two points to measure distances among points. Gorte [13] applied the region-growing algorithm to clustering LiDAR generating building point clouds into planar facets. Vo et al. [14] proposed a building point cloud segmentation algorithm, which first applied a region-growing step and then refined cluster in a coarse to fine manner, based on voxelized point clouds.

Random sample consensus (RANSAC) by Fischler and Bolles [15] is a popular modeling fitting method. Considering that buildings are made up of parameterized 3D geometric primitives, RANSAC-based algorithms are suitable for segment building point clouds. In application, RANSAC-based algorithms are widely used to detect planar building faces. Efficient RANSAC developed by Schnabel et al. [16] aims to detect basic geometric primitives from unorganized point clouds and show fine robustness in the presence of many outliers and a high degree of noise. Adam et al. [17] took point clouds produced by overlapped images as inputs and leveraged RANSAC on 3D geometry and 2D segmentation to achieve more accurate results.

Even for unsupervised segmentation algorithms, determining the semantic label of point clouds directly is difficult, and it can be applied in the preprocessing step in building semantic segmentation and modeling.

2.2.2. Supervised Point Cloud Segmentation

Compared with unsupervised point cloud segmentation algorithm that focuses on cluster points with co-geometric primitives, supervised segmentation methods aim to classify point clouds into different semantic labels.

Traditional machine learning has been commonly used in point cloud semantic segmentation. Wang et al. [18] and Lodha et al. [19] applied Adaboost classifier to segment point clouds. Chehata et al. [20] used random forest algorithm to classify urban LiDAR point clouds. These classifiers depend on hand-crafted features and neglect contextual information, which is useful for classification. Additionally, these methods take point clouds whose ground is in parallel with the x–y plane as inputs. This approach is unusual for multi-view generating point clouds. In contrast to the above classifiers, conditional random field (CRF) can capture more specific relations of object classes and model contextual information. Niemeyer et al. [21] adopted CRF classifier in complex urban LiDAR point clouds. To speed up the problem solving of CRF, Lim and Suter [22] over-segmented and classified point clouds into supervoxels. Restricted by domain-specific parameters of CRF, these methods show fine robustness in LiDAR point clouds, yet failed to hold generalization on multi-view generated point clouds.

With the increasing popularity of deep learning techniques, a growing number of works have been leveraging deep learning models on the point cloud segmentation task. PointNet, which was proposed by Qi et al. [1], incorporated non-local information into point cloud segmentation by maxpooling operation on point feature vectors. Based on PointNet, Qi et al. [2] used hierarchical network to capture more local features and proposed PointNet++. To recover the topological connection between points, DGCNN, which was developed by Wang et al. [23], leveraged EdgeConv for point cloud segmentation. Our closest related work is Genova et al. [24], who generated sparse 3D point cloud semantic labels by combining image semantic labeling with pose information, and densified scene semantics further using 3D convolutions. However, this approach still requires a significant amount of labeled image data to obtain a robust semantic segmentation model, and its multi-stage segmentation method poses deployment challenges in practical applications. Deep-learning-based methods are enjoying increasing popularity on point cloud segmentation. However, limited by the need for massive training data, applying deep learning techniques in building point cloud segmentation is difficult.

3. Methodology

3.1. Overview

We propose a unified framework for producing accurate segmentation results and extracting precise 3D feature lines for building models from complex urban scenes. Notably, the input data, including 2D multi-view images and 3D dense point cloud, are multi-source and heterogeneous, and the output is a lightweight building model with fine-grained semantic annotations. To address the challenges of extracting object-level segmentation results from noisy and cluttered point clouds, we propose a novel joint 2D-3D semi-supervised object segmentation algorithm. Our approach leverages the complementary strengths of multi-view images and 3D point clouds by combining photometric consistency with geometric consistency. However, the point cloud is memory costly. To generate lightweight 3D building models, we generate a lightweight contour for each building facet based on the 3D plane constrained multi-view accurate feature line extraction and optimization algorithm. Finally, to enrich the fine-grained semantic information of the acquired lightweight model, we designed a lightweight semantic modeling framework to produce lightweight models with fine-grained and lightweight semantic annotations. The detailed pipeline of our framework is shown in Figure 1.

3.2. Joint 2D-3D Semi-Supervised Semantic Object Segmentation

Previous works mainly focus on segment point clouds by supervised learning methods, which require a large amount of labeled training data [18,19,20,21,22,25,26]. However, constructing a training dataset for urban scene semantic segmentation is labor intensive for the complexity and discreteness of urban scene point clouds. When confronted with dense and complex urban point cloud semantic segmentation tasks, no methods are deemed effective yet.

A joint 2D-3D semi-supervised semantic object segmentation method for dense point clouds is designed to overcome the aforementioned issues. To extract target building pixels from multi-view images, we utilize the HRNet [27] as our backbone for object segmentation. Additionally, we employ the unsupervised clustering algorithm proposed in [28] to perform coplanar clustering of point clouds. This approach enables us to obtain coplanar image pixels through multiview projection of coplanar point clouds. These pixels are then bounded by semantic consistency during the training process. To ensure multi-view photometric consistency, we enforce the constraint that the projections of the same point cloud on different images belong to the same semantic class. By combining the constraints of multi-view segmentation information and point cloud clustering information, we effectively reduce the errors that may arise from discrete point clouds. This approach improves the accuracy and quality of the segmentation results and enables us to obtain more reliable and robust segmentation output for various downstream applications. Notably, 2D multi-view segmentation only needs to provide a handful of labeled 2D images as the training set, which reduces the dependence on semantic annotation of the point cloud greatly. Meanwhile, it can provide photometric consistency to reduce the occlusion problem. Then, co-planar geometric consistency from point cloud clustering can also guarantee the details of the accuracy of the dense point cloud segmentation, which does not need any supervised information.

Notably, we propose a semi-supervised high-accuracy architecture modeling framework. To achieve more accurate building modeling, segmentation and meshing modules in our framework, not limited by methods in this work, are upgradeable and alternative.

3.2.1. Multi-View Image Segmentaion

A lack of training data remains one of the main challenges of existing segmentation methods. Most open-source segmentation datasets are captured in front views. However, the input images in our tasks are mostly bird-eye views, which cannot directly utilize the existing data set for training. To overcome this problem, we use some manually annotated images from Dataset I to train an HRNet [27] from scratch. We annotated collected images with two categories, namely, building, and background clutter. Given an image as input, our network outputs a probability $P\left(l\right|\mathit{p})$ for every pixel $\mathit{p}\in$ image i, where $l\in \{building,backgroundclutter\}$. Thus, our loss function can be defined as follows:

$$\begin{array}{c}\hfill {\mathcal{L}}_{labeled}=\sum _{p\in i}\mathrm{CrossEntropy}(P\left(\mathit{p}\right),\mathbf{y})\end{array}$$

(1)

y is the ground truth label represented by a vector of 0 or 1.

3.2.2. Photo-Metric Semantic Constraint

By providing image location and point visibility information, we can easily incorporate photometric consistency and semantic constraints from multi-view images into the training framework. This intuitive approach enhances the accuracy and reliability of the segmentation results and allows for more effective training of the algorithm. With known image intrinsic matrix $K_i$, rotation matrix $R_i$ and translation vector ${\mathit{t}}_{\mathit{i}}$, projection matrix is defined as $M_i=K_i\left[R_i\right|{\mathbf{t}}_{\mathbf{i}}]$, $\mathit{i}$ denotes the image from image sets $\mathit{S}$, 2D images pixel ${\mathit{p}}_{i,x}$ location ${\mathit{v}}_{\mathit{i},\mathit{x}}$ on image $\mathit{i}$ according to $\mathit{x}$ as:

$$\begin{array}{c}\hfill {\mathit{v}}_{\mathit{i},\mathit{x}}=M{\mathit{X}}_{\mathit{x}}\end{array}$$

(2)

where ${\mathit{X}}_{\mathit{x}}$ is the location of $\mathit{x}$. ${\mathit{X}}_{\mathit{x}}$ and ${\mathit{v}}_{\mathit{i},\mathit{x}}$ are the homogeneous coordinates. Visibility is denote as an image set ${\mathit{H}}_x$ for each 3D point $\mathit{x}$. The photometric loss is computed as:

$$\begin{array}{c}\hfill {\mathcal{L}}_{photometric}=\sum _{i,j\in {\mathit{H}}_x}\parallel P\left(p_{i,x}\right)-P\left(p_{j,x}\right)\parallel \end{array}$$

(3)

3.2.3. Co-Planar Geometric Constraint

The simplest way to decide the final classification for point clouds is by fetching the label $l_x$ with max probability $P\left(l\right|\mathit{x})$ for point $\mathit{x}$. Nevertheless, these methods neglect the facts that multi-view image segmentation results are inevitably noisy. As such, the training framework can be further improved combined with 3D geometric consistency constraints. Building on the assumption that co-planar points belong to the same object, we introduce a novel loss function that is based on self-supervised signals. Our proposed loss function enhances the performance of the algorithm by enforcing the constraint that co-planar points are assigned to the same semantic class.

Clustering point clouds with similar geometric features is an unsupervised segmentation problem that has been heavily researched for decades. Commonly used algorithms for segmentation point clouds include region-growing-based [12] and RANSAC-based [15] approaches. These methods are efficient for detecting clusters with similar features. However, these algorithms are designed to neglect hypothesized outliers, and the segmentation completeness is not ensured. Our inputs are noisy multi-view generated point clouds, and segmentation completeness is critical to this task. For consideration, we adopt a newly proposed method called pairwise-linkage by Lu et al. [28], which can detect co-planar segmentation and guarantee the segmentation completeness.

We used the pairwise-linkage algorithm to segment point clouds into different co-planar clusters. For each cluster $\mathit{c}$ and projected image i, we compute geometric loss as follows:

$$\begin{array}{c}\hfill {\mathcal{L}}_{geometric}=\sum _{z,x\in \mathit{c}}\parallel P\left(p_{i,x}\right)-P{(}_{i,z})\parallel \end{array}$$

(4)

Thus, our final training loss is a linear combination of the above three losses, expressed as follows:

$$\begin{array}{c}\hfill \mathcal{L}={\mathcal{L}}_{labeled}+{\mathcal{L}}_{photometric}+{\mathcal{L}}_{geometric}\end{array}$$

(5)

In this way, the background and noise point clouds can be removed in the later facet plane fitting and facet contour generating steps. In addition, after removing the background point clouds, the connection among different target buildings is eliminated, which is more beneficial to model and generate fine-grained semantics for target building independently.

3.3. Plane Constrained Multi-View Accurate 3D Feature Line Extraction and Optimization

After determining the architecture point clouds, our target in this step is to generate lightweight architecture polygon contour to replace the original point cloud architecture facet. In this way, we can avoid the use of memory costly point cloud on lower energy cost applications. Several early works focus on lightweight modeling based on point clouds. Cohen-Steiner et al. [4] and Kazhdan and Hoppe [3] attempted to detect building edges from 3D point clouds. However, for the discrete and noise of point clouds, the obtained building edges are generally jagged and non-smooth. The building edges extracted from the images are smooth but discontinuous and hence cannot compose valid 3D building edges. In summary, generating accurate and smooth edges and being robust to noisy point clouds are the main challenges on the architecture modeling of urban dense point cloud scenes.

To relieve the aforementioned issues, we proposed a 3D plane constrained multiview accurate feature line extraction and optimization algorithm. In our methods, architecture facet contour is constrained and generated from 3D feature lines. First, leveraging multi-view image information to extract edges can avoid the generation of architecture facet contour from point clouds, which eliminates the noise of major edges. Second, multi-view images have more 3D edge details than point clouds, which can considerably boost robustness and precision on architecture modeling.

3.3.1. Ground Calibration

In the urban scene segmentation stage, point clouds are classified into coplanar clusters and labeled with different classes. Before extracting architecture facets, the world coordinates of point clouds must be calibrated to satisfy the Manhattan-world assumption. We take point clouds belonging to ground and use RANSAC [15] to estimate the normal of ground point clouds. Given the estimated ground normal vector ${\mathit{n}}_{\mathit{1}}$ and center point ${\mathit{x}}_{\mathit{cen}}$, we generate two vectors, namely, ${\mathit{n}}_{\mathit{2}}$, ${\mathit{n}}_{\mathit{3}}$ and $\{{\mathit{n}}_{\mathit{1}},{\mathit{n}}_{\mathit{2}},{\mathit{n}}_{\mathit{3}}\}$ are pair-wise orthogonal. The rotation matrix $R_{cal}$ and translation vector ${\mathit{t}}_{\mathit{cal}}$ is defined as:

$$\begin{array}{c}\hfill R_{cal}=\left[\begin{array}{ccc}{\mathit{n}}_{\mathit{1}}^T& {\mathit{n}}_{\mathit{2}}^T& {\mathit{n}}_{\mathit{3}}^T\end{array}\right]\end{array}$$

(6)

$$\begin{array}{c}\hfill {\mathit{t}}_{cal}=-R_{cal}{\mathit{x}}_{cen}\end{array}$$

(7)

By applying rotation and translation, point clouds are calibrated to the Manhattan-world coordinate.

3.3.2. Inner 3D Feature Line Generation

The parameterization of roof facets is implemented using RANSAC [15]. We define inner 3D feature edges as the intersection line segments of parameterized architecture facets. To determine the intersection line segments of facets, hypothesis intersection lines are computed on the basis of parametrized facets. After obtaining a hypothesis line, we select two sets of intersection facet points, which are under the distance threshold t to the intersection line. A line segment can be acquired by reprojecting one set of points to an intersection line and fetching endpoints. Intersection line segments are obtained by computing the common interval of two line segments. Considering that most facets have common intersection points, we refine the endpoint intersection segments to the common intersection point.

3.3.3. Multi-View 2D Feature Line Detection

In this stage, our main goal is to extract edge line segments of facets on 2D images. We apply LSD Von Gioi et al. [29] to detect candidate 2D feature lines. For each facet, two steps must be followed to obtain the final facet contour, view selection, and multi-view segment fusion.

• View Selection: For each architecture facet, a subset of visible images ${\mathit{H}}_x$ is selected according to the visibility file. With known visibility image set ${\mathit{H}}_x$ for each point $\mathit{x}$, we define visibility set ${\mathit{H}}_f$ for each facet $\mathit{f}$ as:

  $$\begin{array}{c}\hfill \begin{array}{c}\hfill H_f=\left\{\mathit{i}\right|Count\left(\mathit{i}\right)>0.5\}\\ \hfill Count\left(\mathit{i}\right)=\frac{1}{\left|\mathit{f}\right|}\sum _{\mathit{x}\in \mathit{f}}{\mathbb{I}}_{{\mathit{H}}_{\mathit{x}}}\left(\mathit{i}\right)\end{array}\end{array}$$

  (8)
• Multi-view 2D Feature Line Fusion: We use LSD to extract a large amount of candidate line segments from ${\mathit{H}}_f$. Aiming to transfer 2D line segments to 3D facet plane, end points of the line segments extracted from image $\mathit{i}$ are projected to 3D roof facet using projection Matrix $M_i$ and parametrized facet plane equation $p\left(\mathit{f}\right)$ as:

  $$\begin{array}{c}\hfill \begin{array}{c}\hfill p\left(\mathit{f}\right){\mathit{v}}_{\mathit{e}}^T=\mathit{0}\\ \hfill \mathit{e}=M_i{\mathit{v}}_{\mathit{e}}\end{array}\end{array}$$

  (9)

  where $\mathit{e}$ is the endpoint of the line segment, and ${\mathit{v}}_e$ is the 3D coordinate location of projected $\mathit{e}$. Through this projection method, the 2D line segments from different images are gathered to the target facet 3D plane.
• 3D Feature Line Filtering: However, most of these line segments are not valid contour of the target facet, and the filtering process is essential to reduce the solving time of contour arrangement optimization. In common sense, edge line segments are always close to point clouds, and the point cloud densities between two sides of the line segment have a huge difference. Based on the two aforementioned assumptions, we design a robust and simple strategy to filter out invalid line segments. Facet point clouds are projected to the facet plane. Given a line segment with length L, we construct a bounding box parallel to the line segment and centered on the middle point of the line segment with an aspect ratio of $\sqrt{L}:1$. All points located in the bounding box are counted and divided into the negative side ${\mathit{x}}_{neg}$ and positive side ${\mathit{x}}_{pos}$. Line segments with $max(\frac{|{\mathit{x}}_{neg}|}{|{\mathit{x}}_{pos}|}>2,\frac{|{\mathit{x}}_{pos}|}{|{\mathit{x}}_{neg}|})>2$ and $|{\mathit{x}}_{pos}\bigcup {\mathit{x}}_{neg}|>\tau$ are retained. The intuitive explanation is shown in Figure 2.
• 3D Feature Line Merging: Although the majority of invalid 3D feature lines are filtered out, a large number of candidate edge line segments of target facet is conserved yet. To further speed up the 2D contour arrangement optimization problem, merging is vital to duplicate line segments. When determining whether line segments are duplicated, parallel and proximity are our main considerations. Given two line segments $Lin{e}_1$, $Lin{e}_2$ with middle points ${\mathit{d}}_{\mathit{1}}$, ${\mathit{d}}_{\mathit{2}}$, and direction vector ${\mathit{n}}_{\mathit{1}}$ and ${\mathit{n}}_{\mathit{2}}$, parallel is defined as:

  $$\begin{array}{c}\hfill |{\mathit{n}}_{\mathit{1}}^T{\mathit{n}}_{\mathit{2}}|>0.95\end{array}$$

  (10)
  The defined $Lin{e}_1$ and $Lin{e}_2$ are parameterized by $a_1,b_1,c_1$ and $a_2,b_2,c_2$. Proximity can be defined as:

  $$\begin{array}{c}\hfill \frac{\left|{\mathit{d}}_{\mathit{2}}^T\left[\begin{array}{c}{a}_1\\ b_1\\ c_1\end{array}\right]\right|}{\sqrt{a_1^2+b_1^2+c_1^2}}+\frac{\left|{\mathit{d}}_{\mathit{1}}^T\left[\begin{array}{c}{a}_2\\ b_2\\ c_2\end{array}\right]\right|}{\sqrt{a_2^2+b_2^2+c_2^2}}<0.1m\end{array}$$

  (11)

With the definition of parallel and proximity, we cluster all the duplicated segments. We determine the direction and middle point of the new line segment merged from segment clusters as the average direction and middle point of original segment clusters. Moreover, the endpoints of the new segment are the max interval of endpoints of original segment clusters. With the average operation, the noise is eliminated, and boundary precision is improved.

3.3.4. Geometric-Based Contour Optimization

In this step, our main goal is to construct a complete facet polygon contour from inner intersection line segments and outer candidate edge line segments for each facet. We emphasize the generation of low complexity polygons and fitted the shape to the real roof facet. Viewing the contour arrangement problem as the shortest circle path problem of directed graph which is natural to roof contour, we developed an efficient algorithm to solve it.

We define the Graph as $\mathit{G}$.

• Node definition: Given the inner and edge line segments, we define their intersection points as node $\mathit{g}$.
• Edge definition: After generating nodes, we define the directed edge $\mathit{e}$ between two nodes located on the same line segment ${\mathit{l}}_g$ in the same direction. To determine the edge direction, we extract the rough 2D contour of target facet using the alpha-shape algorithm implemented by CGAL [30]. Then, a simplified contour ${\mathit{B}}_{rough}$ based on the extracted contour is computed by Douglas-Peucker algorithm. For each edge in ${\mathit{B}}_{rough}$, we select a direction to guarantee that only one circular path exist. The direction of the line segment ${\mathit{l}}_g$ in $\mathit{G}$ is consistent with the direction of the edge ${\mathit{e}}_b$ in ${\mathit{B}}_{rough}$ with maximum matching score. The matching score between edge ${\mathit{l}}_g$ and ${\mathit{e}}_b$ is defined as:

  $$\begin{array}{c}\hfill \frac{|{\mathit{n}}_{\mathit{g}}^T{\mathit{n}}_{\mathit{b}}|}{\parallel {\mathit{d}}_{\mathit{g}}-{\mathit{d}}_{\mathit{b}}{\parallel }_2}\end{array}$$

  (12)

  where ${\mathit{n}}_{\mathit{g}}$ and ${\mathit{n}}_{\mathit{b}}$ are the normal vectors of ${\mathit{l}}_g$ and ${\mathit{e}}_b$, respectively. Moreover, $d_g$ and $d_b$ are the middle points of ${\mathit{l}}_g$ and ${\mathit{e}}_b$, respectively. See Figure 3 for a straight-forward explanation.
• Edge weight definition: Edge weight definition is auxiliary for guiding the algorithm to solve the correct contour shape of the roof facet. Edges with accurate contour, compact, and edge preserving have two main features. For compactness, we designed a factor to measure the compactness reduction for adding the edge $\mathit{e}$ to the contour, $1+\alpha \parallel e\parallel$ where $\parallel e\parallel$ s the length of edge $\mathit{e}$. Edge $\mathit{e}$ located on line le is defined. For edge preserving, line segments with more confidence semantic boundary will be detected in more images. We calculate $\beta \frac{1}{\parallel Lin{e}_e\parallel }$ as the edge-preserving factor. Because line ${\mathit{l}}_g$ is merged from a number line segments $Lin{e}_e$, $\parallel Lin{e}_e\parallel$ is the number of set $Lin{e}_e$. The final edge weight for edge $\mathit{e}$ is defined as:

  $$\begin{array}{c}\hfill weigh{t}_e=1+\alpha \parallel e\parallel +\beta \frac{1}{\parallel Lin{e}_e\parallel }\end{array}$$

  (13)
• Contour Rearrangement: We view the contour arrangement as the shortest circular path algorithm. The algorithm to solve this problem is described as Algorithm 1.

With the building facet contour composed of 3D feature lines, the smoothness and lightweight of the building edge are guaranteed, thereby avoiding the use of dense point clouds or complex mesh to represent the building model. Further, lightweight models are the fundamental of lightweight fine-grained semantic representation of buildings. In practice, lightweight models can reduce computation and storage consumption of applications significantly and is beneficial to energy conservation.

Algorithm 1 Shortest Circular Path Algorithm

Input: Graph $G(V,E)$, Adjacent Table $adjacent\left(V\right)$ Output: Shortest Circular Path Initialization: Set $dist\left[i\right]\left[j\right]\leftarrow \infty$ for each i, j, $dist\left[i\right]\left[i\right]\leftarrow 0$ for each i, dist[i][j] is the shortest distance from node i to node j. Step 1. Use Floyd-Warshall Algorithm to compute dist[i][j]. Step 2. $lengt{h}_{min}\leftarrow \infty$, $u\leftarrow 0$, $v\leftarrow 0$ for $\mathit{e}$ in E do:  i, j, w = start node of $\mathit{e}$, end node of $\mathit{e}$, weight of e.  if w + dist[j][i] $<lengt{h}_{min}$ then:   $lengt{h}_{min}$ = w + dist[j][i]   $u\leftarrow j$   $v\leftarrow i$ Step 3. $path\leftarrow \left\{\right\}$ while $u\ne v$ do:  for g, w in $adjacent\left(u\right)$ do:   if dist[g][v] + w = dist[u][v] then:    $path$ = $path$ + g    $u\leftarrow g$    break; $path$ = $path$ + v Return $path$

3.4. Light Weight Semantic Modeling Framework in Fine–Grained Element Segmentation Level

Currently, many applications not only require accurate 3D building model parameters but also fine-grained building semantic information. However, to be utilized by lower-cost devices, a lightweight representation of semantic information is necessary. Therefore, our main goal is to generate lightweight fine-grained semantics of the target building. To enrich the semantic information of building models, we proposed a lightweight semantic modeling framework with fine-grained element segmentation.

The majority of recent works [3,4,11] on modeling building mainly produce a unified mesh of the whole scene where each target building is inseparable. Recently proposed methods [31,32,33] for segmentation of building facades show excellent results. However, the outputs of these works are pixel-wise, which is not compact enough for vectorized model representation. To eliminate these disadvantages, we train a DeepLabv3 [34] network to output pixel-wise labeled results and adopt Algorithm 1 to achieve the polygonal segmentation. Finally, we combine the polygonal segmentation with the obtained lightweight model to generate lightweight semantic models.

3.4.1. Candidate View Selection

A candidate image must be selected for each architecture facet to detect fine-grained semantic instances. To extract complete fine-grained semantics, we attempt to select the image that contains the majority of points of the target architecture facet. The candidate view image ${\mathit{i}}_f$ for facet $\mathit{f}$ is defined as follows.

$$\begin{array}{c}\hfill {\mathit{i}}_f=\underset{\mathit{i}\in H_f}{arg\; max}Count{\left(\mathit{i}\right)}_f\end{array}$$

(14)

3.4.2. Fine-Grained Semantic Instance Detection

We first detect fine-grained semantic instance $\mathit{k}$ on candidate view $i_f$ using a trained neural network. ${\mathit{p}}_k$ is defined as the 2D semantic pixel belonging to $\mathit{k}$, the 3D back projection location ${\mathit{X}}_{pk}$ to facet $\mathit{f}$ of ${\mathit{p}}_k$ can be computed as follows. ${\mathit{v}}_{pk}$ is the 2D location of pixel ${\mathit{p}}_k$.

$$\begin{array}{cc}\hfill & p\left(\mathit{f}\right){\mathit{X}}_{pk}^T=0\hfill \end{array}$$

(15)

$$\begin{array}{cc}\hfill & {\mathit{v}}_{pk}=M_i{\mathit{X}}_{pk}\hfill \end{array}$$

(16)

After obtaining the fine-grained semantic 3D point cloud ${\mathit{X}}_k$, we use the method in Section 3.3.3 to collect the 3D feature lines and apply algorithm in Section 3.3.4 to generate light weight polygonal fine-grained semantic representation.

3.4.3. Framework Designation

The designed framework can solve the fine-grained semantic modeling problem for urban scenes. The concise data stream and independent modules of our framework bring fine extensibility and utility. For utility, our framework can take reconstruction results of most multi-view stereo algorithms as input data. For extensibility, our current fine-grained semantic elements can extend to more classes, such as stairs and balconies, by replacing the fine-grained segmentation network. Based on our framework, developing more semantic modeling applications, such as indoor semantic modeling or urban street semantic modeling, is possible.

Through the above steps, the semantic and 3D structure of the produced model is lightweight, thereby extending the application range and enabling the lower-cost devices to process fine-grained building semantics effectively.

4. Results

We have applied our methods on several datasets of real-world buildings and conducted qualitative and quantitative measurements of the proposed method. It should be noted that we use the vanilla modeling approach to show the superior benefits of point cloud segmentation and 3D feature line detection for architecture modeling. Evaluation Dataset I is a collection of multi-view images captured from an unmanned aerial vehicle (UAV). The point clouds are generated using COLMAP [35,36,37]. The dataset [38] used is an open dataset for fine-grained building facade image segmentation. The point clouds inputted to our methods are noisy with incomplete building facades and uneven densities. Results show that our method is robust to complex building structures and meanwhile with fine-grained semantic information. We evaluate our methods around our three tasks, namely, object-level point cloud semantic segmentation, elements level vectorized semantic segmentation and vectorized building modeling.

4.1. Point Cloud Segmentation

In these experiments, we test point cloud semantic segmentation performance in different models. We compare our results with the SOTA method, PointNet [1], PointNet++ [2], and RandLA-Net [39] on our test data. The test scenes are selected from Dataset I with an average of million points. We selected 9 scenes from Dataset I, and 6 of them were selected as the training data of PointNet [1], PointNet++ [2], RandLA-Net [39], Point Transformer [40] and PointNeXt [41], and three of them were selected as the test data. For our methods that take multi-view images as input data, we manually annotated 40 images to train HRNet [42] to produce segmentation results on 2D images.

Table 1. Comparison of different point cloud segmentation methods for PointNet [1], PointNet++ [2], RandLA-Net [39], Point Transfromer [40], PointNeXt [41] and our method on Dataset I.

Metric		PointNet	PointNet++	RandLA-Net	PointFormer	PointNeXt	Our
Acc	Background	0.917	0.475	0.887	0.880	0.953	0.973
	Building	0.583	0.975	0.903	0.992	0.937	0.967
	Overall	0.712	0.828	0.893	0.958	0.945	0.971
IoU	Background	0.552	0.449	0.671	0.866	0.844	0.916
	Building	0.554	0.800	0.697	0.936	0.914	0.951
	Mean	0.553	0.625	0.685	0.901	0.879	0.931

shows the accuracy and IoU of our multi-view fusion model, PointNet, PointNet++, RandLA-Net, Point Transformer and PointNeXt. We achieve the best performance and dense semantic segmentation results (Figure 4).

As shown in Figure 5, the top left and bottom left of the figure is the collected multi-view images and generated dense scene point clouds, and the bottom right of the figure is the semantic segmentation result of the scene point clouds where redpoint clouds denote building, and blue point clouds denote background. Finally, the top right of the figure corresponds to the produced fine-grained semantic included vectorized building models after extracting the building point clouds from the scene point clouds.

4.1.1. Fine-Grained Semantic Segmentation

We mainly focus on generating precise and lightweight fine-grained semantic contour from pixel-wise segmentation results, combined with the lightweight architecture model.

In this experiment, we set DeepLabv3+ [34] as our base model and train it on Dataset from the 3D Semantic Segmentation and Procedural Modelling Challenge [38] to detect pixel-wise segmentation results on 2D images.

In

Table 2. Overview of fine-grained semantic segmentation results of PSPNet [43], DANet [44], DeepLabv3+ [34] and our GCO optimization based on DeepLabv3+ on Dataset [38].

Metric	PSPNet	DANet	DeepLabv3+	After GCO
mean Acc	81.1	83.14	83.68	85.01
mean IoU	72.03	73.17	73.35	74.59

, we train our models using the training set labeling information of dataset [38], including windows and doors. The results in the table correspond to the performance of our model on the test set of the dataset. We show that we can generate more accurate contour and improve the segmentation accuracy based on rough pixel-wise segmentation results with the geometric-based contour optimization algorithm. As shown in Figure 6, through our fine-grained segmentation-based lightweight semantic modeling framework, the architecture model can be enhanced with richer and more lightweight fine-grained semantic annotation. Figure 7 shows that the fine-grained semantic contour generated by our optimization is more lightweight and compact than the neural network segmentation result.

Figure 7 shows that our boundary optimization algorithm can extract accurate and compact semantic contour from raw pixel-wise segmentation results. After applying our optimization algorithm, the semantic segmentation can be represented by a polygon which is more lightweight than pixel-wise information. Simultaneously, the edge of the optimized segmentation is smoother and more similar to the true semantic annotation.

4.1.2. Vectorized Modeling

In this section, our main goal is to evaluate the generated model in terms of accuracy and lightweight. We selected Dataset I as the test data. In our model generation experiments, weight parameters $\alpha$ and $\beta$ are set to 1.0 and 1.0, thresholds t and $\tau$ are set to 0.001 and 200 for both datasets, respectively. We conduct our experiment in terms of quality and quantity. For quantity evaluation, we select six scenes from dataset I and compute the errors and facet numbers of the model among different methods.

Table 3. Modeling results on Dataset I for our method, PolyFit [11], VSA [4], and QEM [5]. For each modeling result, the mean Hausdorff distance $\mathit{e}$(m) is computed from the output to the original point cloud. Furthermore, the $\mathit{n}$ is the number of facets in the generated model. For the largely missing point clouds in our datasets, PolyFit failed to generate the model on several buildings, which are marked with green cross.

Our
$\mathit{e}$/$\mathit{n}$	0.0437/33	0.0900/167	0.1183/70	0.0821/70	0.0192/52	0.0461/54
PolyFit
$\mathit{e}$/$\mathit{n}$	0.2610/44	0.1837/76	-	0.0821/14	-	-
VSA
$\mathit{e}$/$\mathit{n}$	0.0707/254	0.1248/521	0.4935/336	0.0798/320	0.0274/869	0.0447/425
QEM
$\mathit{e}$/$\mathit{n}$	0.0549/544	0.1739/581	0.5162/619	0.0944/523	0.0968/466	0.0333/753

shows that our results, which are combined with few lightweight facets, are more compact than models generated from PolyFit [11], VSA [4], and QEM [5]. Two main metrics are $\mathit{e}$, the distance from the origin point cloud to the building mesh, and $\mathit{n}$, the facet number of the building mesh. Generally, smaller $\mathit{e}$ represent higher modeling precision. As an indicator of lightweight, $\mathit{n}$ should be smaller under the constraint of accurate modeling (smaller $\mathit{e}$). Considering the complex structure and limited precision of the point cloud, the $\mathit{e}$ should be less than 0.1, and $\mathit{n}$ should range from 50 to 200. Furthermore, our algorithm is robust to simple and complex roof structures. Nevertheless the raw point clouds with sparse and incomplete walls can still acquire good vectorized models using our algorithm.

In Figure 5, we show the modeling results applied on Dataset I, large downtown area. Compared with the original point clouds, the models produced by our method are more lightweight, and each facet of the buildings can be described by a polygon. Besides, the edge of the generated model is smooth, and no intersection exists between different buildings. We also extend our experiment to Dataset [38]. Figure 8 shows the model reconstruction results on Dataset [38] with vectorized semantic segmentation. Compared with the original mesh, our reconstruction result is preferable with lightweight architecture facades.

5. Discussion

Regarding the segmentation results, we observed that PointNet [1] and PointNet++ [2] performed poorly in semantic segmentation experiments of building point clouds. The networks failed to obtain effective semantic information of the buildings, possibly due to heavy downsampling during the segmentation process (Figure 4). However, our approach, which combines multi-view information and 2D convolutional neural networks, outperformed PointNeXt [41] and RandLANet [39] in the segmentation of building geometric corners, producing more precise and sharper segmentation results (Figure 4, highlighted in orange).

PointFormer showed clear segmentation results of building corners. However, this was based on a global attention mechanism that required $O\left(n^2\right)$ computational complexity and consumed large amounts of memory. In contrast, our approach achieved good segmentation results even in cases of severe point cloud loss. For example, in Figure 4 highlighted in green, most of the building points were not reconstructed, and only a corner of the building was visible. However, our approach accurately segmented these points into buildings, showing its robustness to loss and occlusion. This robustness could be attributed to the complementary nature of the missing parts in multi-view images.

Regarding the lightweight modeling results (Table 3), our approach outperformed other methods in terms of modeling accuracy. The high accuracy was due to the extraction of multi-view 3D feature lines, which avoided direct surface fitting on the point cloud, leading to better results than methods based on pure point clouds. In terms of modeling lightweightness, our approach was comparable to the PolyFit [11] method, while the VSA [4] and QEM [5] methods did not consider prior knowledge about building modeling, such as the fact that buildings are typically composed of geometric planes. Our approach effectively utilized this prior knowledge, strengthening the 3D plane constraint during building face generation, resulting in more lightweight models.

Concerning the smoothness of the models, our approach and PolyFit [11] outperformed the other methods. However, it is important to note that PolyFit [11] generated building models that did not match real building shapes. This was due to the optimization problem of constructing a watertight geometry, which requires complete point clouds without significant occlusion or loss; otherwise, the optimization problem becomes ill-conditioned. However, buildings reconstructed using multi-view geometry collection methods are usually incomplete and occluded, making our approach more suitable for lightweight modeling.

6. Conclusions

In this paper, a novel lightweight semantic modeling framework is designed to achieve a highly accurate 3D vectored modeling of buildings with fine-grained semantic information. A joint 2D-3D semi-supervised semantic object segmentation framework is proposed to improve the semantic accuracy and richness of buildings through 2D segmentation and unsupervised clustering. Meanwhile, semi-supervised building semantic segmentation can also reduce the labeling work of training data without affecting the quality of segmentation significantly. Subsequently, a 3D plane-constrained multi-view accurate feature line extraction and optimization method is proposed to achieve continuous, multi-view, consistent, and smooth 3D lines of the buildings. The introduction of multi-view image consistency not only solves the boundary discontinuity caused by occlusion but also improves the accuracy of single building surface vector modeling. Finally, a lightweight semantic modeling framework with fine-grained element segmentation is designed carefully. Each building in complex urban scenes will be extracted accurately and finely vectorized independently, thereby not only improving the details of the 3D geometric structure of the building but also solving the issue of buildings in urban scene mesh models that cannot be processed and analyzed separately. A feature line that constraints fine-grained element segmentation is also introduced into the modeling framework to achieve a highly accurate 3D vectorization modeling of individual buildings with fine-grained semantic information (doors and windows). Of course, our framework has shortcomings.

In the current reconstruction process, we only make use of segmentation and 3D feature lines by using a plain approach to generate 3D building mesh. By extending our framework to a more complex and well-designed modeling method, we may attain better meshing results. Another interesting problem is that the piece-wise planar roof structure assumption becomes overly restrictive when dealing with atypical architectures. Moreover, the point clouds used in this work, mostly generated by multi-view reconstruction, have some missing parts, which will slightly affect the meshing process. Furthermore, the 3D feature line detection also requires high-accuracy camera poses from multi-view reconstruction. In future work, we would like to enhance our method to incorporate more types of geometric primitives, such as cylinders and spheres, to describe more complex and diverse building vector models. We also expect to extend our framework to more fine-grained semantic elements, such as exterior building decorations, geometric structural details of doors, windows, balconies and others, and apply our method to more practical applications, such as animation, movie making, and autonomous driving.