Three-Dimensional Reconstruction of Structural Surface Model of Heritage Bridges Using UAV-Based Photogrammetric Point Clouds

Abstract

Three-dimensional (3D) digital technology is essential to the maintenance and monitoring of cultural heritage sites. In the field of bridge engineering, 3D models generated from point clouds of existing bridges is drawing increasing attention. Currently, the widespread use of the unmanned aerial vehicle (UAV) provides a practical solution for generating 3D point clouds as well as models, which can drastically reduce the manual effort and cost involved. In this study, we present a semi-automated framework for generating structural surface models of heritage bridges. To be specific, we propose to tackle this challenge via a novel top-down method for segmenting main bridge components, combined with rule-based classification, to produce labeled 3D models from UAV photogrammetric point clouds. The point clouds of the heritage bridge are generated from the captured UAV images through the structure-from-motion workflow. A segmentation method is developed based on the supervoxel structure and global graph optimization, which can effectively separate bridge components based on geometric features. Then, recognition by the use of a classification tree and bridge geometry is utilized to recognize different structural elements from the obtained segments. Finally, surface modeling is conducted to generate surface models of the recognized elements. Experiments using two bridges in China demonstrate the potential of the presented structural model reconstruction method using UAV photogrammetry and point cloud processing in 3D digital documentation of heritage bridges. By using given markers, the reconstruction error of point clouds can be as small as 0.4%. Moreover, the precision and recall of segmentation results using testing date are better than 0.8, and a recognition accuracy better than 0.8 is achieved.

1. Introduction

Ancient bridges in China are an essential part of the world’s material and cultural heritage, but due to their age, archival information on ancient bridges is seriously scarce, and the number is sharply decreasing due to natural and human-made disasters. Therefore, each ancient bridge needs a fast, efficient, and accurate record for permanent preservation, particularly for endangered bridges.

Detailed three-dimensional (3D) reconstruction of ancient bridges provides a new way to solve this problem. This kind of work can be achieved using either range-based techniques, such as terrestrial laser scanners (TLS), or image-based techniques, mainly photogrammetry including structure-from-motion (SfM) [1,2]. Compared to the range-based approach, the image-based approach has several advantages, including easy acquisition and low costs. Recently, UAVs have been widely used to provide a general view for earth sensing reconnaissance and scientific data collection purposes [3,4]. Compared to terrestrial acquisition and the classic manned aerial acquisition, UAV imaging possesses the merits of flexibility and limited costs. These merits make these flying platforms an attracting choice for data acquisition in diverse applications such as 3D modeling [5,6], urban planning [7,8], natural environment mapping [9,10,11], and in general, several scene inspection and monitoring applications [12]. UAVs can acquire high-resolution and close-range images that can subsequently be processed using SfM and multiview stereopsis workflows (MVS) to generate detailed 3D point clouds and surface models. The emergence of ready-to-use consumer-grade drones as well as the development of photogrammetry and computer vision techniques further lead to significant research interest in 3D modeling and 3D mapping using UAV images in various applications, such as heritage building documentation [1], landslide monitoring [13], glacial geomorphology [14], building modeling [15] and change detection [16]. Among all these applications, the UAV-based 3D reconstruction. Currently, benefiting from photogrammetric triangulation/SfM techniques, 3D point clouds can be reconstructed from UAV images with satisfactory accuracy. In recent works like [17], with the help of bundle adjustment-based exterior orientation parameters, the accuracy of reconstructed points with consumer-grade drones can reach as high as 0.05 m, which is sufficient for the majority of engineering projects. However, to achieve such a high accuracy of reconstruction, two vital points should be considered and investigated. The first one is the relative orientation of images [18], which is essential to the rectification of the image. The quality of relative orientation will directly influence the success rate of extracting sparse corresponding points and the further rectification of images [19]. The second point is the dense matching technique [2,20], which is a critical step to generate dense point clouds from the orientated images. Only when problems involving these two points are solved can qualified point clouds be obtained with high accuracy for further applications. In the field of bridge engineering, several studies have successfully demonstrated the power of UAV photogrammetry for bridge inspection and maintenance. Khaloo et al. [21] and Morgenthal et al. [22] produced a 3D model of a bridge using UAV-captured images and a SfM algorithm to perform structural condition assessment. Chen et al. [23] proposed a bridge inspection process using point clouds generated from UAV images and compared this with TLS-based inspection. However, few published works in this area focus on the reconstruction of a structural 3D model of bridges. As the semantic and structural information is essential for heritage bridge documentation, automatic recognition of structural elements from the UAV photogrammetric bridge models using advanced point cloud processing methods is necessary.

However, to use 3D point clouds for further applications, we need to identify and separate individual objects from the entire scene [24]. To achieve this, segmentation of the point cloud is usually required before the recognition [25]. The segmentation of point clouds, aggregating 3D points into multiple homogeneous groups with common characteristics [26], has been explored for decades. Conventional segmentation methods like region growing or clustering examine points in the vicinity of initial seeds or origins and check if they belong to the same group or not according to given criteria. Euclidean distance [27], density [28,29], normal vector deviation [30,31], smoothness of surfaces [32], and curvatures [33] of points are representative criteria. Besides, the segmentation can also be conducted in the feature space. Geometric features or RGB colorinformation with distinctiveness are also introduced as segmentation criteria [34]. However, all these segmentation methods are easily influenced by noise and outliers in the dataset, resulting in over- or under-segmentation with different granularities of the obtained segments. Besides, complex segmentation criteria will significantly increase the computational cost. All these drawbacks should be solved if we want to achieve a good partition of objects from the entire scene.

In this paper, we attempt to bridge the gap between the engineering techniques of photogrammetry and remote sensing and applications from bridge engineering. To this end, an automatic framework for reconstructing structural surface models of heritage bridges is developed using UAV photogrammetry and point cloud processing. Detailed point clouds of a heritage bridge are generated from the captured UAV images through the SfM and MVS workflows. A novel segmentation method based on supervoxel structure and global graph optimization are proposed to separate the entire point cloud into consistent segments. Sequentially, the different structural elements are clustered using a classification based on a classification tree and bridge geometry, and the structural surface model is finally created by a Poisson surface reconstruction algorithm. To be specific, we utilize the graph-based segmentation method for the application of bridge engineering. The reason for using the graph-based method is that graph structure, which is a statistical context model, is commonly used for modeling the geospatial relationship between neighboring 2D/3D points. Since the graph structure has a natural relation with 3D topology, compared with other data structures (e.g., regular 3D grids for structured point clouds), a graphical model can encode not only the features of points in the local context but also the interactions between the point and its surrounding neighbors when constructing the weighted edges of the graph. The weights of edges encapsulate the dependency and affinity between connecting nodes. Using the optimization (e.g., partition) of the graphical model of the point cloud, the segmentation of the bridge point cloud can be easily achieved.The performance of the presented framework is validated via experiments with two bridges in China.

2. Methodology

The framework of 3D structural model generation is illustrated in Figure 1, which consists of five main steps, i.e., UAV flight path planning and image acquisition, image rectification, 3D reconstruction, point cloud segmentation, and structural elements recognition. The point cloud segmentation and structural elements recognition, which involves classifying different structural components of the bridge from the obtained point clouds, are the main contributions of our structural model generation framework. Details of these five steps are briefly presented in the following.

2.1. UAV Flight Path Planning and Image Acquisition

Planning a flight path requires an awareness of multiple factors such as distance from target, speed, overlap, and pattern. According to the principle of photogrammetry and 3D reconstruction, higher information redundancy can improve the quality of the solution. In order to obtain better reconstruction results, the flight path should be properly planned before take-off to ensure a high overlap rate between images. According to the recommendation in [21], the following rules are initially determined:

• Shooting according to a serpentine route;
• The overlap ratio of the heading should be greater than 60%, and 90% is recommended;
• The side overlap ratio should be greater than 30%, and 60% is recommended.

The distance between the UAV and the bridge is determined by factors including camera field of view, sensor resolution, and safety. In this study, the distances from the bridge are divided into three scales: near, middle, and far, which are about 2 m, 5 m and 8 m, respectively. Taking the 2 m scale as an example, the flight path planning is shown in Figure 2.

2.2. Image Rectification

This step includes scale estimation and camera calibration. Since the 3D model normally obtained by SfM is initially generated in an arbitrary reference system, it is necessary to transform this initial arbitrary datum into a predefined coordinate reference system. For the purpose of heritage bridge digitization, the absolute position and orientation are not critical, but the absolute scale is indispensable. In this study, three pairs of mark points with the known distance are arranged on the bridge. As shown in Figure 3, Mark-1 and Mark-2 are located on the bridge deck, and Mark-3 is located on the bridge side railing. According to the ratio of the model distance and the actual distance between the mark points, the whole model can be synchronously scaled.

Camera calibration is the process of estimating the internal geometry of a camera and has the most significant influence on the accuracy and reliability of photogrammetric measurements. Typically, there are two general strategies for calibration, i.e., pre-calibration and self-calibration. The influence of the calibration method on the quality of point clouds derived using UAV photogrammetry has been investigated in [35]. In this study, we pre-calibrate the camera shortly before the flight, by modeling the focal length, principal point coordinates, and distortion coefficients using Brown’s distortion model [36]. The estimated interior camera parameters are used to undistort and thus correct the raw images prior to the following process. To be specific, the camera was calibrated immediately prior to capturing the surface of interest. The pre-calibration was realized using a plane-based calibration method implemented by EOS PhotoModeler Scanner software. The planar calibration pattern with circle targets and coded targets is supplied by PhotoModeler Scanner software, as show in Figure 4. It is a flexible, robust and low-cost method to obtain stable camera parameters.

2.3. 3D Reconstruction

The 3D reconstruction to generate the photogrammetric point clouds is achieved via SfM and MVS workflows [13]. To be specific, this process includes two major steps, i.e., sparse reconstruction and dense matching [37]. The sparse reconstruction involves the identification and matching of homologous feature points as well as the reconstruction of the geometric image acquisition configuration and generation of the sparse point clouds. Dense matching is then conducted to increase the density of the 3D point cloud using the projection relationship recovered by sparse reconstruction. In this study, all processes are implemented with Agisoft PhotoScan software. A description of the SfM and MVS workflows in PhotoScan and commonly used parameters is provided in [38]. Moreover, a statistical outlier removal filtering (SOR) [39] is applied to these point clouds before the point cloud processing. The SOR filter we used is provided by the PCLlibrary. It computes first the average distance of each point to its neighbors, considering k-nearest neighbors for each. Then, it rejects the points that are farther than the average distance considering the standard deviation.

2.4. Point Cloud Segmentation

The segmentation step groups all the points into multiple consistent regions having one or several common characteristics [26], playing a vital role for separating the entire point cloud into meaningful geometric primitives in the presented framework. A novel segmentation method for point clouds based on the supervoxel structure and global graph optimization is proposed. The workflow of the proposed segmentation method is shown in Figure 5 and can be divided into three steps: supervoxelization, global graph construction, and graph-based clustering.

2.4.1. Supervoxelization of Point Clouds

For the supervoxelization of point clouds, we adopt the octree-based data structure to discretize the entire point cloud with 3D voxels, which allows the indexing of the unorganized point cloud with octree structure and simplifies the dataset with a grid-based representation [30]. As stated in [40], the selection of the voxel size should be carefully considered since we need to have a good balance between the processing time and the quality of preserved details.

Based on the voxel structure, supervoxels are further generated using the VCCS method [41], which groups candidate voxels according to their distance to seed points within a feature space comprising centroid positions, normal vectors, geometrical features, and RGB colors. The major advantage of VCCS is its ability to preserve boundaries, through which we can obtain supervoxels whose boundaries coincide with the edges of objects in the scene. For the performance of VCCS, parameters like the voxel size and the seed resolution will significantly influence the quality of segments. It is noteworthy that in this study, when generating the supervoxel, no color information is considered (i.e., only the spatial distance and normal vectors are used). The parameters are empirically set according to densities of points and distances between the sensor and objects within the scene.

2.4.2. Construction of Global Graph

We use a supervoxel graph-based method of point cloud segmentation. The supervoxel structure of a point cloud can be represented by an undirected k-nearest neighbors graph $G=(V,E)$, in which the nodes represent the supervoxels and the edges encode their relationship. Each edge $(v_{\mathrm{i}},v_{\mathrm{j}})\in E$ has a corresponding weight $w_{\mathrm{ij}}$, which is a non-negative measure of the similarity between neighboring supervoxels $v_{\mathrm{i}}$ and $v_{\mathrm{j}}$. To identify the weight of the edge, each node v is assigned with an attribute vector representing geometric attributes formed by all points within v, relating to three groups of attributes: spatial positions, geometric features, and normal vectors [42]. To be specific, the spatial positions are the spatial coordinates $\mathbf{X}$ of the centroid p of supervoxel v. The geometric features are eigenvalue-based features representing the 3D distribution of points inside v, including linearity $L_{\mathrm{e}}$, planarity $P_{\mathrm{e}}$, scattering $S_{\mathrm{e}}$, and change in curvature $C_{\mathrm{e}}$ [43]. The normal vector $\mathbf{N}$ is estimated using points within v. Based on the attributes of nodes, three types of geometric cues (i.e., proximity, similarity, continuity) representing binary features between neighboring supervoxels $v_{\mathrm{i}}$ and $v_{\mathrm{j}}$ are estimated. The proximity $W_{\mathrm{ij}}^p$ relates to the spatial distance between $v_{\mathrm{i}}$ and $v_{\mathrm{j}}$. The similarity $W_{\mathrm{ij}}^s$ measures the consistency between geometric features of $v_{\mathrm{i}}$ and $v_{\mathrm{j}}$. The continuity $W_{\mathrm{ij}}^c$ combines the smoothness $W_{\mathrm{ij}}^m$ and the convexity $W_{\mathrm{ij}}^o$ of supervoxel surfaces, and is estimated by normal vectors and direction vectors between $v_{\mathrm{i}}$ and $v_{\mathrm{j}}$ [40,44]. To judge the continuity, we assume four types of connected surfaces between supervoxels: smooth, “stair-like”, convex, and concave. Sketches of these four types of connections are shown in Figure 6. The smoothness $W_{\mathrm{ij}}^m$ relates to the difference in angles between normal vectors ${\mathbf{N}}_{\mathrm{i}}$ and ${\mathbf{N}}_{\mathrm{j}}$. The convexity $W_{\mathrm{ij}}^o$ relies on the local configuration of the surfaces of adjacent supervoxels. If the local configuration is convex, adjacent supervoxels are considered to be highly connective. The local configuration (i.e., convex or concave) is judged by angles ${\alpha }_{\mathrm{i}}$ and ${\alpha }_{\mathrm{j}}$ (see Figure 6c,d), and is calculated by the angle between ${\mathbf{N}}_{\mathrm{i}}$ and ${\mathbf{N}}_{\mathrm{j}}$ and the vector ${\mathbf{D}}_{\mathrm{ij}}$ linking ${\mathbf{X}}_{\mathrm{i}}$ and ${\mathbf{X}}_{\mathrm{j}}$, where ${\mathbf{D}}_{\mathrm{ij}}=({\mathbf{X}}_{\mathrm{j}}-{\mathbf{X}}_{\mathrm{i}})/\left|\right|{\mathbf{X}}_{\mathrm{i}}-{\mathbf{X}}_{\mathrm{j}}{\left|\right|}_2$. As stated in [44], if ${\alpha }_{\mathrm{i}}-{\alpha }_{\mathrm{j}}>\theta$, the surface connectivity is defined as a convex connection, and vice versa. Here $\theta$ is the threshold for judging convexity, which is a given threshold. The surface continuity $W_{\mathrm{ij}}^c$ is calculated according to Equation (1) [42], giving a higher continuity to convex or smooth connected surfaces:

$$W_{\mathrm{ij}}^c=\left\{\begin{array}{c}{\left({\alpha }_{\mathrm{i}}-{\alpha }_{\mathrm{j}}\right)}^2+{\pi }^2,{\alpha }_{\mathrm{i}}>{\alpha }_{\mathrm{j}}+\theta \hfill \\ {\left({\alpha }_{\mathrm{i}}-{\alpha }_{\mathrm{j}}\right)}^2+{\left({\alpha }_{\mathrm{i}}+{\alpha }_{\mathrm{j}}-\pi \right)}^2,{\alpha }_{\mathrm{i}}\le {\alpha }_{\mathrm{j}}+\theta .\hfill \end{array}\right.$$

(1)

Finally, the weight $w_{\mathrm{ij}}$ can be defined by considering all $W_{\mathrm{ij}}^k,k\in [\mathrm{p},\mathrm{s},\mathrm{c}]$ as:

$$w_{\mathrm{ij}}=\prod _{k\in [\mathrm{p},\mathrm{s},\mathrm{c}]}\exp (-\frac{{(W_{\mathrm{ij}}^k)}^2}{2{\lambda }^2}),$$

(2)

where $\lambda$ controls the weight of the spatial distance, the similarity, and the continuity. In this work, all of these three lambda parameters are set to one.

2.4.3. Global Graph-Based Clustering

To aggregate supervoxels into an entire segment, we formulate the aggregation process into a clustering work based on the graphical model. As stated in [46], the graphical model can explicitly represent 3D points with a mathematically sound structure and employs contextual information to deduce hidden information from given observations [47]. Thus, by constructing and partitioning the graphical model, we can obtain the connection information between supervoxels. To be specific, graph-based clustering aims to divide a dataset into disjointed subsets with members similar to each other from the affinity matrix. In [42], the use of the local graph structure for the description of the 3D geometry with the supervoxel structure was tested. The use of the local graph model can make the clustering process quite efficient and available for parallel computing when combined with a region-growing strategy. However, the local graph structure can merely encode the local geometry information, which can hardly represent the optimal in the global scale, so that over-segmentation frequently occurs when dealing with surfaces with irregular geometric shapes (e.g., points of vegetation). To tackle the drawbacks of the local graph model, we developed global graph-based clustering, which constructs a global graph model to describe the regional characteristics of 3D scenes with different complexities, and details of objects are preserved among the clustered nodes. Specifically, once the global graph of all supervoxels is constructed, we can optimize the connection between supervoxels by partitioning the global graph into several subgraphs. To solve this graphical model, we utilize the method introduced in [48], in which the Min-cut algorithm achieves a foreground–background separation. Different from the original work of [48], in our solution each supervoxel is regarded as the seed of the foreground and is compared with the background consisting of the local neighborhood. The foreground and background of the supervoxel builds a local graphical structure that is solved by the Min-cut algorithm. In Figure 7, we illustrate the built local graphical structure in the entire graphical model. Once the Min-cut algorithm is applied, the local graphical structure is separated into the foreground and background, and the edges linking these two parts are regarded as disconnected. When all the supervoxels have been checked, the entire global graphical model is separated into disconnected subgraphs by considering the connection of edges. For the supervoxels V in the same subgraph C, these are merged into a single segment S of points.

2.5. Structural Element Recognition

Once the segments are achieved, a rule-based classification method is applied to recognize different structural elements from segments. Here, the recognition step consists of the refinement of segments, the recognition of elements, and surface modeling.

2.5.1. Refinement of Segments

The refinement of obtained segments is needed for the supervoxel-based segmentation, the results of which always suffer from the “zig-zag” effect since the basic element of segments is the cubic-shaped voxel [49]. To overcome this drawback, we propose a boundary refinement of the achieved segment, consisting of two major steps: the detection of points of boundary supervoxels and the refinement of these boundary points. In the first step, for each segment containing several supervoxels, if one of the supervoxels is adjacent to the supervoxels of other segments, this supervoxel will be identified as the boundary one, and all of its points will be regarded as boundary points. Then, in the second step, the normal vectors of the boundary points and the supervoxel at the boundary are estimated. Based on the estimated normal vectors, a local k-mean clustering is conducted between the boundary point and the centers of neighboring supervoxels (see Figure 8).

Here, the clustering is governed by a distance measure calculated in the feature space, considering the normal vectors $\mathbf{N}$ and the spatial distance between centroids $\mathbf{X}$:

$$D=\sqrt{w_{\mathrm{n}}\frac{{\mathbf{N}}_{\mathrm{i}}·{\mathbf{N}}_{\mathrm{b}}}{|{\mathbf{N}}_{\mathrm{i}}\left|\right|{\mathbf{N}}_{\mathrm{b}}|}+w_{\mathrm{d}}\left(\right|{\mathbf{X}}_{\mathrm{i}}|-|{\mathbf{X}}_{\mathrm{b}}\left|\right)},$$

(3)

where $w_{\mathrm{n}}$ and $w_{\mathrm{d}}$ are weight factors controlling the contribution of normal vectors and spatial distances.

2.5.2. Recognition of Elements

For recognizing structural elements from the segments, a rule-based classification method is introduced that utilizes the saliency of segments to determine the label of segments (i.e., the type of object that it belongs to) [50]. The saliency of the segment is defined by its geometric properties, which include the following aspects:

• The height $S_{\mathrm{h}}$ indicating the spatial position;
• The angle $S_{\mathrm{v}}$ between the horizontal direction and the normal vector of the segment;
• The size $S_{\mathrm{s}}$ of segments relating to the spatial length, width, and height.

To be specific, the saliency for a given segment $S_{\mathrm{sal}}$ is a vector that is computed as follows:

$$S_{\mathrm{sal}}=[\frac{S_{\mathrm{h}}}{{\max }_{k=1,\dots ,n}\left(S_{\mathrm{h}}^k\right)},(1-\frac{S_{\mathrm{s}}}{{\max }_{k=1,\dots ,n}\left(S_{\mathrm{s}}^k\right)}),\frac{S_{\mathrm{v}}}{\pi /2}],$$

(4)

where n is the total number of segments in the point clouds. The saliency of each segment is ranked in decreasing order as the input to the decision for recognizing structural elements. In this work, three types of structural elements are considered, namely decks, fences, and walls of the bases. Once the saliency of all of the segments is calculated, a sequential classification is applied with given thresholds in a classification tree (see Figure 9). The segments with saliency values falling into particular branches are recognized as particular structural elements.

2.5.3. Surface Modeling

A Poisson surface reconstruction algorithm [51] is adopted for surface modeling. The input for Poisson surface reconstruction is the point cloud and its normal vector. The point represents the position of the object surface, and its normal vector stands for the direction. Surface reconstruction is achieved using an implicit function framework, which computes a 3D indicator function and considers all of the points at once. Considering a region M with its boundary $\delta M$ and an indicator function ${\mathcal{X}}_M$, for any point $p\in \delta M$, we define ${\overrightarrow{N}}_{\delta M}\left(p\right)$ as the inward surface normal vector. $\tilde{F}$ is a Gaussian smoothing filter, and ${\tilde{F}}_p\left(q\right)=\tilde{F}(q-p)$ is a translation of $\tilde{F}$ to the point p. As ${\mathcal{X}}_M$ is generally not well-derived, the available gradient of ${\mathcal{X}}_M\ast \tilde{F}$ is approximated as

$$▽({\mathcal{X}}_M\ast \tilde{F})\left(q_0\right)={\int }_{\delta M}{\tilde{F}}_p\left(q_0\right){\overrightarrow{N}}_{\delta M}\left(p\right)dp.$$

(5)

The indicator function can be calculated by Equation (5), and the reconstructed surface M is obtained by extracting an appropriate isosurface. Finally, a 3D mesh model with vertices and faces is achieved.

3. Results and Discussion

In the experiments section, we first introduce our bridge testing data, with two bridge field sites introduced. The generation of point clouds for these two bridges is also stated. Then, experiments using benchmarks are carried out to assess the performance of our proposed segmentation methods. Field tests using point clouds of the two bridges mentioned above are conducted, and the corresponding results are analyzed and discussed.

3.1. Testing Data of Bridges

To test the performance of the presented framework, two bridges were selected as testing sites. The first one is the Hongde Bridge, located in Hongsan Village, Tang Town, Shanghai, China. It was built in the Qianlong period of the Qing dynasty and has been around for more than 250 years. It is a single-span stone bridge with a span of 4.9 m, a bridge length of 14.5 m, and a bridge deck width of 2 m. The second one, the Tongxin Bridge, is located at Tongji University, Shanghai, China. It is a brick double-arch bridge, that was built over 50 years ago, with a length of 9.5 m and a bridge deck width of 5 m. In Figure 10, views of the testing bridges are shown.

The UAV used was a DJI Phantom 3 Professional, which is a consumer-grade quadcopter. A total of 763 and 575 images, respectively, were collected with a 1/2.3${}^{″}$ complementary metal oxide semiconductor sensor with 12 megapixels, for each of the two bridges. The camera was mounted on a three-axis stabilization gimbal. Although more advanced UAVs and cameras are available, the purpose of this flight was to demonstrate the presented framework for bridge digitization in a real scenario.

3.2. Generated Point Clouds

According to the matching results of the feature points and the orientation parameters, point clouds of the Hongde Bridge and Tongxin Bridge were generated, as shown in Figure 11a–d. It is clear that the sparse point cloud reflects the main contour of the Hongde Bridge, including the outer edges of the bridge deck, the pier, the stone lion, and so on. The number of sparse points was about 300,000. Then, using the results of MVS, all of the pixels were re-projected to obtain the dense point cloud model shown in Figure 11b,d. It can be seen that the overall structure and texture of the bridges was fully restored. The point number of the final point cloud is around 18 million. The point cloud of the Tongxin Bridge was generated in the same way. The number of points in the sparse point cloud was 37,000, which is much smaller than that of the Hongde Bridge. For the dense point cloud, there was a total of 2.5 million points generated. The root mean square (RMS) of reprojection errors was 0.715 pixels and 1.010 pixels for the point clouds of the Hongde Bridge and the Tongxin Bridge, respectively.

To evaluate the quality of the point cloud reconstructions, in the point clouds of the Hongde Bridge, we measured the distance between three pairs of marker and compared them with the ground truth distance. The result is given in

Table 1. Reconstruction accuracy of different scales.

Scale	UAV Distance to Bridge (m)	Measured Distance (m)	Ground Truth Distance (m)	Error
Small	2.0	0.504	0.5	0.8%
Middle	5.0	0.515	0.5	3.0%
Large	8.0	0.523	0.5	4.6%
All	–	0.502	0.5	0.4%

. As seen from the table, we find that when using all the UAV images from different scales, the error of reconstruction can be as small as 0.4%, which could be regarded as sufficient for the documentation of heritage bridges.

3.3. Quality of Segmentation

The performance of the segmentation plays an essential role in the recognition of structural elements. For this reason, we assessed the segmentation method by conducting a comparison between the achieved segments and the manually generated segments from reference data. For the reference dataset, we utilized the one used in [42], which is a manually segmented point cloud of a single building. The results generated by three point- or voxel-based segmentation algorithms, namely the smoothness-based Region Growing (RG) [32], Supervoxel- and Graph-based Segmentation (SVGS) [42], and Locally Convex Connected Patches (LCCP) [44] algorithms, were used as baselines for further comparison.

In these experiments, the voxel size (used in our method, SVGS, and LCCP) and the neighborhood size (for estimating normal vectors) of RG were set to 0.15 m, ensuring that the normal vectors used in all the methods were calculated by almost the same number of points. For the supervoxelization process used in our segmentation method, SVGS, and LCCP, the seed resolution of VCCS was set to 0.5 m. The threshold for tolerating the angle difference between normal vectors was set to 0.26 rad, empirically. For SVGS, the threshold for efficient graph-based segmentation was 0.75.

The $precision$ and $recall$ were selected as the basic evaluation metrics for assessing the performance of our method, which are calculated via Equations (6) and (7), by the use of true positive (TP), true negative (TN), false positive (FP), and false negative (FN). Here, $precision$ is used to estimate the ratio of correctly segmented points in the segmentation results, while $recall$ is used to assess the ration of correctly segmented points in the reference data [30].

$$precision=\frac{\left|TP\right|}{\left|TP\right|+\left|FP\right|}.$$

(6)

$$recall=\frac{\left|TP\right|}{\left|TP\right|+\left|FN\right|}.$$

(7)

In Figure 12, the segmentation results using different baseline algorithms is given. It is noted that to handle outliers and isolated points, we removed the supervoxels that had too few points, namely, those that did not satisfy the minimum number (at least three points) of required points when estimating the normal vector. As seen from the figure, it is clear that our segmentation method can generate outstanding results. To be specific, not only are planar surfaces (e.g., roofs and walls) well separated, but irregular shapes (e.g., bushes and fences) are also segmented. By contrast, the results of LCCP are more like over-segmentation, with small patches appearing at edges. With RG, the areas of planes are well segmented, but for curved surfaces and irregular objects, they are more likely to be over-segmented as well. As for the results of SVGS, irregular objects like bushes are well segmented, but it seems that under-segmentation frequently happens for nearby objects.

To thoroughly investigate the potential of our segmentation method, we created the precision–recall (PR) curves (see Figure 13) for all baseline algorithms by changing thresholds based on the reference data. The tendencies of the PR curves show that our method can achieve better segmentation and a good compromise between accuracy and recall. Specifically, according to the PR curve, we find that when the recall value is greater than 0.7, the proposed method has better performance than other methods. However, since the RG method is a point-based method that tends to generate over-segmentation, the results of the RG method can achieve better precision values than our method, but with smaller recall values. For the results of the LCCP method, since the smoothness and convexity criteria used in the LCCP method are more suitable to segment planes and box-shaped structures [40], when dealing with rough and irregular-shaped surfaces (e.g., roofs) and linear structures (e.g., fences), we can observe a similar over-segmentation like that of RG, showing discontinuous patches of segments. In this work, the size of voxels is determined according to the demands of application, namely the subdivision of the octree is stopped according to the divided size of voxels. In each voxel, there should remain at least three points to estimate the eigenvectors. In Figure 14, we illustrate the relation between the voxel sizes and the segmentation performance. Here, the segmentation result is evaluated via the $F_1$ score. In this test, with a fixed threshold for partitioning the graphical model, the voxel size ranges from 0.1 m to 1.0 m, and the seed resolution of corresponding supervoxels is three times larger than the voxel size. The test results shows that with appropriate voxel sizes (i.e., 0.2 m–0.4 m), our segmentation method can achieve good results, with $F_1$ scores ranging from around 0.8 to 0.67.

The segmentation results of point clouds of real bridges are given in Figure 15 and Figure 16, while the number of generated segments is given in

Table 2. Segmentation results for the Hongde Bridge and Tongxin Bridge.

Name	Number of Points	Number of Segments
Hongde Bridge	18,114,743	30
Tongxin Bridge	5,613,422	15

. We find that the major components of the bridges are well separated from the entire point cloud. However, for the boundaries of the obtained segments, there are still some errors, especially for the area without a good quality of points (e.g., the points are too sparse or include too many outliers). In some area, over-segmentation also happens, for example, in Figure 15b, the wall on the base of the bridge is over-segmented into two parts. Moreover, for some connection areas between the decks and fences, the edges are not clear and incorrect. However, for the majority of the segments, the edges are satisfactory.

3.4. Recognition and Modeling of Structural Elements

Once the segmentation of the point cloud is achieved. The segments are classified into three groups by the saliency values. In Figure 17, the saliency values of segments from two different datasets are given. As seen from these saliency values, it is evident that segments of different structural elements will generate significantly different saliency values; for example, the decks and bases of the bridge will have totally different height and direction values.

Based on these saliency values, the classification tree mentioned above was applied to group the segments. The results are shown in

Table 3. Recognition results for Hongde Bridge and Tongxin Bridge.

Name	Total Segments	Segments of Decks	Segments of Bases	Segments of Fences
Hongde Bridge	30	8	8	14
Tongxin Bridge	15	2	4	9

and the corresponding saliency values of each segment are shown in Figure 18. Concerning the saliency values, we find that the height and direction values dominate the distinguishing process. As seen from the recognition results, it is clear that the majority of the major structures have been correctly recognized, and especially large elements. However, small patches belonging to the decorations are occasionally recognized incorrectly. To be specific, for Hongde Bridge, there are eight segments recognized as decks, while for Tongxin Bridge, only two segments are recognized as decks. For the bases, there are eight and four segments recognized as bases of the two bridges, respectively. For the rest of the segments, they are all regarded as parts of fences. Nevertheless, there are still some segments of the fences that are recognized incorrectly as parts of the base, which can be seen in Figure 19b. In

Table 4. Evaluation of the recognition result.

		Hongde Bridge			Tongxin Bridge
	Predict	Decks	Bases	Fences	Decks	Bases	Fences
Truth
Decks		6	0	2	2	0	0
Bases		1	8	1	0	3	2
Fences		1	0	11	0	1	7
Overall accuracy (OA)			0.83			0.8

, we also give an evaluation of the recognition results compared with the ground truth from manual recognition. The evaluation result reveals that our recognition method can reach an overall accuracy (OA) greater than 0.8 for both bridges.

In Figure 19, we illustrate the modeling results of the recognized structural elements. As seen from the figure, it is apparent that the quality of segments significantly influences the modeling quality. For segments with clear and accurate boundaries, the surfaces are well reconstructed. The generated surface models provide a comprehensive documentation of these historical bridges, with the size, the orientation, and the geometric properties estimated and recorded. However, we should also note that in this work, we only categorize the structural elements of the bridge into three types, namely decks, fences, and bases, which is not sufficient to fully describe the 3D information of historical bridges.

4. Conclusions

In this work, we present an automatic method for generating surface models of structural elements of heritage bridges. To be specific, we tackle this challenge via a novel top-down method for segmenting main bridge components, combined with rule-based classification, to generate labeled models from point clouds. This study focuses on using UAV photogrammetric point clouds associated with heritage bridges in China. The segmentation method is developed based on voxel structure and global graph optimization, which can effectively separate bridge components based on geometric features. Then, classification based on a classification tree and bridge geometry is utilized to recognize different structural elements from the obtained segments. Finally, surface modeling is conducted to generate surface models of the recognized segments. Experiments using Hongde Bridge and Tongxin Bridge demonstrate the potential for using UAV photogrammetry in 3D digital documentation of cultural heritage sites, with promising results achieved. To be specific, primary structural elements are recognized from 30 and 15 segments of the two bridges, respectively. Surface models are also created based on the recognized segments, which can provide solid documentation for the further preservation of these bridges. Moreover, experimental results using benchmark datasets also reveal that our proposed segmentation algorithm is promising for the separation of structural elements in the field of civil engineering. In the future, we will focus on optimizing 3D reconstruction algorithms for more complicated bridge structures and developing better noise removal techniques. In addition, the improvement of the quality of UAV-based point clouds should also be considered, which could be helped by acquiring better position and orientation information based on internal (GPS) and inertial measurement units systems.