Semi-Automatic Registration of Airborne and Terrestrial Laser Scanning Data Using Building Corner Matching with Boundaries as Reliability Check

Abstract

Data registration is a prerequisite for the integration of multi-platform laser scanning in various applications. A new approach is proposed for the semi-automatic registration of airborne and terrestrial laser scanning data with buildings without eaves. Firstly, an automatic calculation procedure for thresholds in density of projected points (DoPP) method is introduced to extract boundary segments from terrestrial laser scanning data. A new algorithm, using a self-extending procedure, is developed to recover the extracted boundary segments, which then intersect to form the corners of buildings. The building corners extracted from airborne and terrestrial laser scanning are reliably matched through an automatic iterative process in which boundaries from two datasets are compared for the reliability check. The experimental results illustrate that the proposed approach provides both high reliability and high geometric accuracy (average error of 0.44 m/0.15 m in horizontal/vertical direction for corresponding building corners) for the final registration of airborne laser scanning (ALS) and tripod mounted terrestrial laser scanning (TLS) data.

1. Introduction

With the development of laser scanning technology, the family of such applications has grown to include airborne laser scanning (ALS), tripod mounted terrestrial laser scanning (TLS), mobile laser scanning (MLS), and boat-based laser scanning. The performance, as well as application scope, of different platforms is variable, although they generally complement one another [1,2]. Along with the progression of laser scanning technology, the integration of multi-platform laser scanning is an evident trend [3]. Airborne laser scanning is quickly able to obtain three-dimensional information about objects over a large area [4]. Despite abundant top-surface information, the corresponding façade information is lacking. Terrestrial laser scanning can obtain detailed façade information, with high geometric precision, but its scanning range is limited and obtaining top-surface information is difficult [5,6]. As ALS and TLS have their own advantages and disadvantages, the integration of these two technologies will collect a full set of information about object surfaces, both on the top and the façade sides [7]. Recently, the integrated application of these two technologies has begun to appear in fields such as: (1) geological exploration, such as topographic mapping [8], erosion calculation [5], landslide monitoring [9], and rockfall monitoring [10]; (2) forestry applications, such as biomass assessment [11] and canopy structure investigations; (3) hydrological studies, such as flood models [12] and studies of the fluvial environment [13]; (4) 3D reconstructions, such as surface model reconstruction [14] and urban model reconstruction [15,16].

Registration is a prerequisite in the integration of multi-platform laser scanning for various applications, especially in detailed three-dimensional reconstruction. The second part of this paper specifically introduces related studies on the registration of ALS and TLS data, which often uses manual operations or auxiliary data (e.g., GPS and images). Much work on registration of LiDAR data and images has been reported [17]. Until now, little has been reported on the registration of ALS and TLS data. Indeed, due to the differences between ALS and TLS, registration of data from these two platforms is very difficult. This is a result of: (1) Different perspectives and covering areas. Airborne laser scanning systems obtain a wide range of data by overlooking an area, and, thus, the top-surface information is abundant with very sparse façade points; on the contrary, terrestrial laser scanning systems obtain limited data, horizontally and upwards, with relatively detailed façade information and little top-surface information. This means there is a problem of adequate commonality between overlapping scans to allow for a match. (2) Different spatial resolutions. The distance between airborne laser scanning systems and the scanning targets ranges from hundreds to thousands of meters, data resolutions being in the meter- or decimeter-scale; the distance between terrestrial laser scanning systems and the scanning targets is often tens of meters, with data resolutions being on the millimeter scale. Therefore, the registration of a single point to a single point may theoretically be changed to the situation of a one-to-many mapping. (3) The discrete nature of point cloud data. Point cloud data is discrete in nature, and, thus, it is difficult to automatically obtain corresponding points and/or features from the data given by these two platforms.

Considering the disparity between two datasets, it is better to conduct the registration with registration primitives. In an urban environment, the boundary of building roofs and building corners can serve as good registration primitives. In Section 2, registration using points may bring about reliability concerns, whereas that using lines is a complex task. It is meaningful to combine points and lines for the fast generation of reliable matching relationships. Based on this, the present study proposes a new approach using building boundaries and corners as registration primitives to automatically conduct the registration of ALS and TLS data. Using building boundaries for the reliability check, conjugate building corners are reliably matched in a fully automatic way from the building corners derived from the two platforms’ data. On this basis, the final registration is achieved with the conjugate corners using least mean square (LMS) method. Due to the extremely small coverage of a single scan of TLS where few building features can be extracted, multiple scans of TLS are registered manually or automatically beforehand. After that, the TLS data is registered with a single or several strips of ALS data using the proposed method. As buildings may have eaves, leading to the non-correspondence of building contours from ALS and TLS data, large errors may exist. Thus, this paper mainly focuses on the registration of buildings without eaves.

The rest of the paper is organized as follows: Section 2 gives a brief overview on registration and feature extraction of point clouds. Section 3 presents our method, including building boundary segment extraction, building corner extraction, and point cloud registration using corner matching with a reliability check. Section 4 presents the experimental case and performance evaluation of the proposed algorithm. Section 5 draws the conclusion.

2. Related Work

2.1. Review of Registration of Point Clouds from the Same Platform

In the last few decades, the registration of 3D point clouds has drawn enormous attention from various fields, including computer vision, artificial intelligence, and photogrammetry. In photogrammetry, relevant studies can mainly be found in the registration of point clouds from the same platform, i.e., (1) registration of multi-scan terrestrial laser scanning, and (2) strip matching of airborne laser scanning. Here, we mainly review the registration of multi-scan TLS.

Point cloud registration has two main approaches, depending on the features utilized, i.e., range-based registration and image-based registration. In later techniques, spectrum information (e.g., intensity and RGB values) is used to provide additional information for the registration [18]. In Al Manasir, and Fraser and Han et al. [18,19], digital images are used to solve the transformation relationship with scale-invariant feature transform (SIFT) and speeded up robust features (SURF) features, respectively. Both, camera images and reflectance images, are good supplementary information for point cloud registration. Based on an algorithm for the least-squares matching of overlapping 3D surfaces, Akca provided an extension algorithm that can simultaneously match the surface geometry and its attribute information under a combined estimation model [20]. Reflectance images have also been used by Kang et al. [21] and Eo et al. [22], where SIFT features are extracted from images as registration primitives. Image-based registration handles point clouds with spectrum information well, but such information is not always accessible.

More often, geometric features such as points, lines, planes, and curves are used for point cloud registration. The iterative closest point (ICP) algorithm, developed by Besl and McKay [23], is a widely used point-based registration method. ICP estimates the transformation parameters between two overlapping point clouds by an iterative process, which converges to a local minimum. Later, various variant approaches were also proposed to improve the robustness of ICP. Nevertheless, these approaches require a good initial alignment of the point clouds to enable convergence to the local minimum, which is not always easy to access. Local normal and curvature features are another source of useful information for point-based registration approaches. Relevant research can be found in [24–26]. In these approaches, changes in geometric curvature and the approximate normal vector of the local surface around each point are evaluated and used for the registration of two partially overlapping point clouds. Due to the very large amount of features, the matching of conjugate features becomes a difficult problem.

Lines are an essential feature in urban scenes, especially in man-made structures. Eo et al. showed that line-based matching may give better results than point-based matching [22]. Stamos et al. [27] extracted planar and linear features from raw range scans and suggested a line- and plane-based registration technique. Jaw and Chuang [28] provided a line-based registration approach consisting of a 3D line feature extractor, a 3D line feature-matching engine, and spatial 3D line features. Of particular concern is the matching of 3D line features, which utilizes angle and distance constraints. Lee et al. [29] employed lines to measure and adjust the discrepancies between overlapping data strips, in which triangulated irregular networks are created for the segmentation of points and line equations are determined from the intersection of neighboring planar patches. More recently, after the global registration of point clouds, Bucksch et al. extracted skeletons from tree branches and exploited the matching of these skeletons [30]. As line features present geometric evidence of edges, which are quite prominent and extraordinary, the matching of conjugate line features can be performed correctly and reliably. Nevertheless, due to the completeness and precision of extracted line features, their direct matching is no easy task.

Plane features for registration can be found in von Hansen [31], where single plane correspondences are used to find the transformation parameters with some additional assumptions. Zhang et al. [32] fitted planes from segmented point clouds and utilized characteristics of the Rodriguez Matrix to establish a mathematical model of registration. Brenner et al. [33] showed that three plane matches can help to determine the transformation parameters between two point clouds, and their method was later compared with the point-based Normal Distributions Transform method. A comparison of two open source LiDAR surface classification algorithms was report in [34]. The results of this comparison showed that the plane-based method tends to be more accurate and the point-based method is conceptually simple and fast.

The above features are comprehensively applied in other methods. For instance, Chen and Medioni proposed a point-to-plane registration method for two overlapping point clouds [35] that improves the convergence property of ICP. Since then, many variant point-to-plane approaches have appeared, and a useful review can be found in Rusinkiewicz and Levoy [36]. Similar to the above methods, Huang et al. presented a method utilizing multiple geometric features for the registration of TLS data [37]. In their method, conjugate planes were used to find the rotation, and the intersections of the axes of the cylinders and planes were used to determine the translation. Their experiments showed a reliable and satisfactory registration solution. The comprehensive use of multi-features has been found to effectively enhance the reliability and precision of registration.

2.2. Review of the Registration of ALS and TLS Data

Although point cloud registration has become a hot topic, there is little in the literature on the registration of point clouds from different platforms, particularly the registration of ALS and TLS data. Al-Durgham and Habib [7] introduced a valuable methodology for the optimal registration and segmentation of heterogeneous Lidar data, and gave an example of integrating airborne and terrestrial laser scans. Hohenthal et al. used global navigation satellite system (GNSS) to locate a terrestrial laser scanner as reference points for the registration of these two platforms’ data [13]. However, as GNSS is manually located on the scanners for their locations, slight deviations of their relative location will lead to large registration errors. Based on this consideration, after using GNSS data for registration, a multi-station registration method can be applied to minimize the surface error of point cloud data [5,10]. Fruh and Zakhor, however, believe this is not sufficiently reliable to permit the use of GPS data in urban street canyons [38]. Auxiliary third-party data (i.e., GNSS and images) is required in the registration procedures of the above-mentioned work. Taking into account the availability and precision of the auxiliary data, it is strongly desirable to eliminate this auxiliary data and find reliable matching primitives from the point cloud.

Some researchers have tried to utilize points, lines, and planes for the registration of the two datasets. Von Hanson et al. extracted line features from airborne and terrestrial LiDAR data, and determined the matching relations of the lines by calculating the amounts of translation and rotation [6]. The use of line primitives provides a good basis for the registration of point cloud data. However, in many complicated situations, the calculation is extremely difficult due to the completeness and precision of the extracted line features. Jaw and Chuang, utilizing the conversion relationship among points, lines, and polygons, constructed a feature-based transformation model [39]; nevertheless, this method must determine, in advance, the matching relations among the points, lines, and polygons. Bang et al. attempted to calibrate the parameters of airborne LiDAR with terrestrial LiDAR as a reference, using point, point-to-plane, and plane transformation models. This paper focuses on the registration of features, rather than their extraction [40]. The realization of high-reliability, high-precision, and fully automated ALS and TLS data registration remains to be studied.

2.3. Review of Building Extraction from Terrestrial Laser Scanning

Many studies are reported for building extraction from airborne LiDAR data [41–43]. Procedures described in this paper are concerned with building extraction from TLS, thus, we give a brief review of this topic.

Segmentation-based methods, which adopt certain criteria to measure the similarity between a given set of points to make grouping decisions, are often used in the classification of point clouds. According to the work presented by Vosselman et al. [44] and Rabbani et al. [45], segmentation-based approaches have three varieties: (1) edge-based segmentation, (2) surface-based segmentation, and (3) scanline-based segmentation. Edge-based segmentation methods, e.g., Belton and Lichti [46], extract edges based on changes in local surface properties and group the points inside the boundaries. Several surface-based methods can be found in Schmitt and Vogtle [47] and Pu and Vosselman [48], which use local surface properties as a similarity measure for the grouping of points. Due to the huge volume of data, redundancy, and occlusion, these methods have difficulties in extracting the point clouds of individual buildings from point clouds that include trees and buildings. Another surface-based segmentation adopts a top-down strategy, which mainly refers to the Random Sample Consensus (RANSAC) algorithm and the Hough transform. RANSAC algorithms present better-fitting results than the Hough transform, although the distance threshold for partitioning inliers and outliers must be determined empirically. Scanline-based methods, such as that of Manandhar and Shibasaki [49], classify surfaces with scanline profiles. These methods require a few preferred directions to be chosen for the scanlines, making the segmentation results orientation-dependent.

As segmentation-based methods usually deal with every point in a cloud, they are extremely time-consuming when a huge number of points are encountered. The density of projected points method is introduced to extract buildings [50,51]. In this approach, the survey region was divided into regular mesh grids with all the points projected onto the grids. The number of points inside each grid was regarded as density, and a certain threshold was used to extract buildings. In Hammoudi et al. [52], similar work is undertaken for street wall models on a large scale. As all of the above methods project 3D point clouds onto 2D grids or rasters, they are collectively known as projection-based methods. Compared with segmentation-based methods, an obvious advantage of projection-based methods is their fast calculation speed when encountering a huge data volume. Nevertheless, several thresholds must be determined in order to obtain good extraction results.

3. Method

The proposed approach includes the following steps. (1) Boundary segment extraction. A hierarchical density of projected points method is introduced to extract accurate boundary segments from TLS data (hereinafter referred to as the “terrestrial boundary”). The key to this is that the proposed theoretical estimation method is able to determine an appropriate density threshold. In addition, the maximum height difference is used to extract boundary segments from ALS data (hereinafter referred to as the “airborne boundary”). (2) Building corner extraction. A boundary self-extending algorithm is developed to segment and recover the extracted boundaries, which then intersect to form 3D building corners. (3) Point cloud registration using corner matching with boundaries as reliability check. The key idea of this study is our strategy of using corner matching with reliability check. The building corners extracted from ALS data (hereinafter referred to as the “airborne corners”), and those extracted from TLS data (hereinafter referred to as the “terrestrial corners”), are reliably matched during an automatic iterative process by taking the distances between airborne boundaries and terrestrial boundaries for the reliability check. Then conjugate building corners are used for the generation of registration relationship with LMS method, which finally guides the registration of ALS and TLS data.

3.1. Building Boundary Segment Extraction

3.1.1. Boundary Segment Extraction from Terrestrial Laser Scanning Data

Review 2.3 shows some methods for building contour extraction from TLS data. The density of the projected points (DoPP) method [50,51] shows an obvious advantage in terms of its fast calculation speed, and so this is selected for the boundary segment extraction from TLS. In DoPP, as shown in Figure 1a, the LiDAR points are projected onto the grids in the plane, and the number of points inside each grid is recognized as a density. Due to the distinguished number of points along the walls, the density of walls stands out. In this paper, based on DoPP, a hierarchical strategy is used for the extraction of boundaries. Due to the large amount of points, the computation is extremely time-consuming, large grids are set first to extract the rough boundary regions. Here, the grid size is usually set larger than the width of window hoods. Then, smaller girds are used to extract accurate boundaries, in which the size is set smaller than the width of window hoods.

The specific steps are as follows.

• Step 1: Large boundary grid selection. A grid with a relatively large size (1 m × 1 m in this study), named a large grid, is constructed in the XY plane. All terrestrial points are projected onto the XY plane. The point density of each grid is calculated by obtaining the number of points in each grid based on the point-grid spatial relationship. A point density threshold is estimated (as discussed later), and this is used to identify the grids belonging to the building walls. All rough grids are separated into two sets: wall grids (with point densities above the threshold) and non-wall grids (with point densities below the threshold).
• Step 2: Small boundary grid selection. The wall grids selected in Step 1 are used to construct grids of relatively small size (0.2 m × 0.2 m in this study), named small grids. The point density of each grid is recalculated according to the point-grid relationship. Similarly, based on a point density threshold, all grids are separated into two sets: wall grids and non-wall grids.
• Step 3: Boundary segment extraction. The Hough transform is used to convert the selected small boundary grids into vector boundary segments.

The point density threshold is the key factor in the above processes, as this is directly related to the results of grid selection. This study proposes a theoretical estimation method to determine the threshold. First, the building with the minimal density in the whole area is determined. After that the boundary density of the found building is calculated using a theoretical estimation method and its density is set as the final filtering threshold. In Figure 1a,c, Point O is the center of the terrestrial laser scanning instrument, and Point A is the wall point horizontally perpendicular to this instrument. Line CD is the edge of a grid (Grid A in Figure 1a) in the horizontal direction, in which Point C is the start point of this grid and Point D is the end point (the intersections of Line AD and dotted lines in Figure 1c). The length of Line CD is D_G. The distance between Point D and Point O is D_M, and that between Point A and Point O is D_V. D_R and D_C are, respectively, the point spacing around Point A along the horizontal and vertical direction. As shown in the figure, if D_M and D_V decrease, the number of columns will increase. Assuming the increased number of columns is n₁, then the density will increase by n₁ × N_row, which is a considerable influence on grid density. Nevertheless, if H_B increases, the number of rows will increase. Supposing that the increased number of rows is n₂, then the density will increase by n₂N_col. It is obvious that N_row is significantly larger than N_col, so D_M and D_V determine the contour with minimal density. In our method, the building farthest away from the apparatus, and with the lowest height, is considered to be the building with the most minimal density and the distance and height of the building is determined manually.

After finding the building with the most minimal density, its point density threshold is estimated as follows.

Step 1: Calculating the number of points between Point C and Point D in the horizontal direction, i.e., the column number of point clouds in a grid. Letting 2α denote the angle of each rotation of the laser scanner along horizontal direction, so:

$$\alpha =\text{arctan}(\frac{D_R}{2\times D_V})$$

(1)

Point E is the last point of the point cloud data on Line AC, i.e., the next LiDAR point along Line AD locates inside Line CD as shown in Figure 1c, then:

$$\angle \mathit{AOE}=\left[\frac{\angle \mathit{AOC}-\alpha }{2\times \alpha }\right]+\alpha ,\hspace{1em}\angle \mathit{AOC}=\text{arctan}((\sqrt{D_{\text{M}}^2-D_{\text{V}}^2}-D_{\text{G}})/D_{\text{V}})$$

(2)

Point B is the start point of the point cloud data on Line CD during the laser scanning operation. Letting β denote the intersection angle between Line OA and Line OB, so:

$$\beta =(2*\left[\frac{\text{arctan}((\sqrt{D_M^2-D_V^2}-D_G)/D_V)-\alpha }{2\times \alpha }\right]+3)\times \alpha$$

(3)

Then, the total number of points along Line CD is:

$$N_{\mathit{col}}=\left[\frac{\text{arccos}(D_M/D_V)-\beta }{\alpha }\right]+1$$

(4)

Step 2: Calculating the number of points in each column parallel to Line AF. Setting the vertical rotating angle as 2θ, so:

$$\theta =\text{arctan}(\frac{D_{\text{C}}}{2\times D_{\text{V}}})$$

(5)

Assuming this is the i-th column of the N_col columns, whose intersection point with Line AC is Point I. Thus, the angle from this column to Line AF is:

$$\angle \mathit{AOI}=\beta +i\times 2\times \alpha$$

(6)

As the i-th column of points includes those above Line AD (recorded as $N_{\mathit{Above}}^i$) and those below Line AD (recorded as $N_{\mathit{Below}}^i$), then:

$$N_{\text{row}}^i=N_{\text{Above}}^i+N_{\text{Below}}^i$$

(7)

The number of points in Line MI is:

$$N_{\text{Above}}^i=\left[\frac{\angle \mathit{MOI}-\theta }{2\times \theta }\right]+1,\hspace{1em}\angle \mathit{MOI}=\text{arctan}(\frac{H_{\text{B}}-H_{\text{L}}}{D_{\text{V}}/\text{cos}(\beta +i\times 2\times \alpha )})$$

(8)

The number of point in Line NI is:

$$N_{\text{Below}}^i=\left[\frac{\angle \mathit{NOI}-\theta }{2\times \theta }\right]+1,\hspace{1em}\angle \mathit{NOI}=\text{arctan}(\frac{H_{\text{L}}}{D_{\text{V}}/\text{cos}(\beta +i\times 2\times \alpha )})$$

(9)

Step 3: Adding the number of point clouds in each column, thus obtaining the total number of points in a grid, i.e.,

$$N=\sum _{i=1}^{N_{\mathit{col}}}{N}_{\mathit{row}}$$

(10)

As is well known, building façades are not regular planes, but complex entities with local uneven features. The strong reflection of window glass often leads to a sparsity of laser points on the walls. In this case, the point density of each grid corresponding to the same wall may exhibit large differences. As shown in Figure 1d, the point density around the windows is obviously lower than in other regions, with some portions totally lost. In addition, the “occlusions” may be introduced by trees or other objects standing between the building and the TLS impacts on point density. Therefore, it is necessary to set a rate value representing the influence of the unevenness of walls, the proportion of windows, and the roundness of the façade. The final threshold is obtained by multiplying the theoretical value N by the rate value, i.e.,

$${\mathit{den}}_{\text{thre}}=\mathit{rate}\times N$$

(11)

A smaller value should be set if the wall is uneven with many windows or sheltered by other objects; otherwise, a larger value is set. In this study, we use a rate value of 0.5. Several experiments have been carried out to verify the accuracy of the determined building, producing good results.

3.1.2. Boundary Segment Extraction from Airborne Laser Scanning Data

Airborne laser scanning systems collect data by overlooking a region. Although they can obtain a certain number of wall points, the number, in comparison with dense non-wall points, is extremely limited. The grid density method is therefore not appropriate for boundary extraction. Compared with TLS, ALS has one specific advantage, i.e., it collects a large number of points corresponding to building roofs, and these have obvious stepwise changes of elevation with terrestrial points. Utilizing this feature, boundaries are extracted from ALS data using the height difference of points inside a grid.

Step 1: Height difference calculation. Grids (1 m × 1 m) are constructed according to the ALS data. Points in each grid are determined according to the point-grid spatial relationship. The maximum height difference between the highest and lowest points within each grid is calculated.

Step 2: Grid selection. According to the height of the lowest building in the experimental area, a threshold for grid selection is determined. A grid is retained if its maximum height difference is larger than the threshold; otherwise, it is removed.

Step 3: Boundary segment extraction. For the retained grids, the Hough transform is used to extract vector boundary segments. As different sizes of buildings may exist leading to various length of building contours, it is difficult to extract all the building contours with only one Hough transform. In this study, the whole area is divided into several parts equally and then as for the parts Hough transform is conducted respectively.

3.2. Building Corner Extraction

In this section, a boundary self-extending algorithm is proposed to recover the extracted boundary segments (both airborne and terrestrial), after which the boundaries intersect to form building corners. The boundary segment extraction of the previous section may make it improbable that a perfect approach to extract all the wall grids can be found, particularly in extreme cases, e.g., full glass coverage (Place A in Figure 2a) or large-area aisles around the building (Place B in Figure 2a). Meanwhile, the Hough transform can hardly detect vector segments without any loss. The idea of our boundary self-extending method is to extend the extracted boundary segments and evaluate the point density along the extended segments. The extension procedure stops when a large density variation appears. By comparing the point density of the cylinder in the to-be-extended direction with that of the existing segment, we can determine whether the extension should be conducted, as shown in Figure 2b.

Step 1: Segmentation of building contours. As building contours are extracted from 2D grids, isolated contours may be extracted as a single contour, as shown in Figure 2c. A contour segmentation method with point elevations is introduced. As in Figure 2d, a plane perpendicular to the XY plane is constructed, and point clouds inside the contour grids are projected onto this plane. The projected points are divided into blocks in a small interval, which is usually slightly larger than the average point spacing (0.2 m in this study). On this basis, the elevation and gradient difference of neighboring blocks is calculated from the highest point in each block. If the differences are small enough, the neighboring blocks are clustered; otherwise, a new cluster is generated. Finally, the RANSAC approach is used to fit each cluster and 3D building contours are obtained.

Step 2: Cylinder construction. A 3D buffer zone for boundary segments, forming a cylinder, is constructed. The buffer area is usually several times the average spacing of the point cloud, so that there are a sufficient number of points inside the cylinder volume. The number of points in the cylinder is determined, and this is divided by the cylinder volume to obtain the point density of the original boundary.

Step 3: Self-extending procedure. The boundary segment along its direction is extended, and the point density in the to-be-extended direction in the same way is calculated. As the density would decrease sharply when the extended segment encounters an actual edge of the building roof patches, a density difference threshold can easily be set to prevent the extension of segments. Extension is conducted if the density decrease is within a certain percent (20% in this study) of that of the original boundary; otherwise, extending is stopped.

Step 4: Intersection for building corners. During the calculation of the 2D boundary segment elevation, errors may prevent the 3D boundary from intersecting in the 3D space. The recovered boundaries are projected to the XY plane for pairs of 2D cross-points. If the elevation difference of the cross-point pair is smaller than a threshold (0.5 m in this study), it is considered that there is a 3D cross-point corresponding to the 2D cross-point pair and the average elevation value of the 2D cross-point pair is used as the elevation of the corner. As gable roofs may exist, then situations exist, as shown in Figure 2c. In order to extract the corners of gable roofs, 3D boundaries are also projected to a plane perpendicular to XY plane. The intersection point of two lines is calculated and 3D corners are obtained.

Steps 1–4 are repeated for all boundary segments; and the building corners are extracted.

3.3. Point Cloud Registration Using Corner Matching with Boundaries as Reliability Check

As TLS collects data using sensor own coordinate system while ALS usually uses global coordinate system, the registration is needed. In a 3D coordinate transformation, if a coordinate o-xyz is transformed into O − XYZ, then a possible transformation model is:

$$\left[\begin{array}{c}X\\ Y\\ Z\end{array}\right]=\lambda R(\alpha ,\beta ,\gamma )\left(\begin{array}{c}x\\ y\\ z\end{array}\right)+\left(\begin{array}{c}{x}_0\\ y_0\\ z_0\end{array}\right),R(\alpha ,\beta ,\gamma )=R_{\alpha }{R}_{\beta }{R}_{\gamma }$$

(12)

where α, β, γ are the rotation values around the Z, Y, and X axes, respectively, x₀, y₀, z₀ are the translation values along the X, Y, and Z axes, and λ is the scale factor.

Despite the errors of the extracted corners, the distances between corners approximately correspond to their actual distances, making this a rigid matching issue. Thus, the transformation between ALS and TLS data is a six-parameter model in which a transformation matrix can be produced by choosing three points from two corner data sets. A common method of determining a reliable transformation matrix is to calculate the number of other successfully matched points according to this transformation matrix—the higher the number, the more reliable the matrix. However, when the number of corners is limited, errors are likely to be produced.

A large amount of line features exist in both ALS and TLS data. Accurate line features can provide reliable information for registration. However, as segments are prone to vary in length, pose, etc., the direct use of line segments for matching is difficult. Considering the problems in line matching and point matching, the extracted building corner information and boundary segment information are combined. A corner matching method with boundaries as reliability check is proposed. In this method, corners are used to calculate transformation matrices, and boundary segments are regarded as a check to evaluate the reliability of the matrices. Reliable conjugate corners are achieved through the matching process and least mean square method is finally used for the registration matrices.

Assume the two sets of corner points are A= {A_i, i=0,1,2,…,u} and B= {B_i, i=0,1,2,…,v}, and the corresponding boundary segments are LA= {LA_i, i=0,1,2,…,m} and LB= {LB_i, i=0,1,2,…,n}.

Step 1: Three pairs of points from corner sets A and B are randomly chosen and the translation matrix T_ini and rotation matrix R_ini are calculated.

Step 2: T_ini and R_ini, to convert the boundary set LB are used. The converted boundary set LC= {LC_i, i=0,1,2,…,n} is obtained.

Step 3: The number of matched segment pairs in LA and LC are determined. The details of this part are in the following part.

Step 4: If the number of matched segments in LA and LC exceeds a predefined threshold (5 in this study), then the transformation matrix is considered reliable; otherwise, Steps 1–4 are repeated. An incorrect transformation matrix will achieve few matched segments, and, thus, the threshold is easy to set.

Step 5: The reliable transformation matrices R_rel and T_rel to transfer all the points of B to F= {F_i, i=0,1,2,…,v} are used.

Step 6: The successfully matched points from A and F are chosen and recorded as P= {P_i, i=0,1,2,…,m} and U= {U_i, i=0,1,2,…,n} (m=n), thus, finishing the initial registration procedure.

Step 7: Using the least mean square (LMS) method, the registration relationships between set P and set U are calculated, and the registration transformational matrices R and T, where R is a 3 × 3 rotation matrix and T is a 3 × 1 translation matrix are obtained.

As for the determination of matched line pairs in LA and LC (Step 3), details are shown below. The procedure commences by taking an unmatched segment from LA and judging whether it matches with a segment in LC. The two segments are considered as successfully matched, if following conditions are satisfied:

$$\{\begin{array}{c}l\mathit{Ang}\le {\mathit{Thre}}_{\mathit{ang}}\\ l\mathit{Dist}\le {\mathit{Thre}}_{\mathit{dist}}\\ l\mathit{Dif}\le {\mathit{Thre}}_{\mathit{dif}}\end{array},\{\begin{array}{c}l\mathit{Ang}=\text{arccos}(\frac{\overrightarrow{l_1}\hspace{0.17em}\hspace{0.17em}\overrightarrow{l_2}}{\left|\overrightarrow{l_1}\right|\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\left|\overrightarrow{l_2}\right|})\\ l\mathit{Dist}=|\frac{(\overrightarrow{l_1}\times \overrightarrow{l_2})\hspace{0.17em}\overrightarrow{P_1{P}_2}}{|\overrightarrow{l_1}\times \overrightarrow{l_2}|}|\\ l\mathit{Dif}=||\overrightarrow{l_1}|-|\overrightarrow{l_2}||\end{array}$$

(13)

where lAng is the angle between the straight lines in which the two boundary segments are located, lDist is the distance between the straight lines in which the two boundary segments are located, lDif is the length difference of the two boundary segments and Thre_ang, Thre_dist, Thre_dif are the angle, distance, and length difference thresholds, respectively. $\overrightarrow{l_1}$ and $\overrightarrow{l_2}$ refer to the directions of the two boundary segments, and are given by Equation (14). P₁ and P₂ are the perpendicular foots in l₁ and l₂, respectively.

$$\begin{array}{l}\overrightarrow{l_1}=(x_{l_{12}}-x_l{}_{{}_{11}},y_{l_{12}}-y_{l_{11}},z_{l_{12}}-z_{l_{11}})\\ \overrightarrow{l_2}=(x_{l_{22}}-x_{l_{21}},y_{l_{22}}-y_{l_{21}},z_{l_{22}}-z_{l_{21}})\end{array}$$

(14)

where (x_l11, y_l11, z_l11), (x_l12, y_l12, z_l12) refer to the coordinates of the start and end points of a segment in LA, respectively, and (x_l21, y_l21, z_l21), (x_l22, y_l22, z_l22) refer to the coordinates of the start and end points of a segment in LC, respectively.

4. Experiments and Analysis

4.1. Experimental Data

The campus of Nanjing University, China (Figure 3a), is taken as the experimental area. This covers a total area of about 1 km × 1 km. Figure 3b shows the ALS data obtained from above the campus, which is cut out from part of the ALS data covering Nanjing City. The point number of this ALS data is about 11 million, with an average point spacing of 0.30 m, a vertical accuracy of 0.15 m, and a horizontal accuracy of 0.30 m. In the northwest part of the campus (yellow box in Figure 3a), a Leica ScanStation 2 is used to collect TLS data at nine stations. Considering the limited spatial resolution of ALS data, the spatial resolution of the terrestrial scanning operation is set to give a point spacing of 0.20 m over a distance of 100 m. The point cloud data from these nine stations are manually registered based on the targets, which cover an area of about 300 m × 200 m (Figure 3c). The total point number of the nine stations is approximately 30 million. The objective of this experiment is to automatically register the ALS and TLS data.

4.2. Registration Results

4.2.1. Boundary Segment Extraction

Figure 4 illustrates the process of extracting the boundaries from TLS data using the point density method. Grids with a size of 1 m × 1 m are established in Figure 4a. The farthest building is about 50 m away from the apparatus and has a height of 40 m, the height of the equipment is 1.5 m, the minimum horizontal distance is 25 m, and the ratio of window to wall is set to 0.5. Based on our theoretical estimation, the point density of a grid (1 m × 1 m) corresponding to a building wall is about 2214 pts/m². The point density threshold is set to 1107 pts/m². The 1 m × 1 m grids retained after selection based on this point density threshold are shown in Figure 4b. Using the same parameters, the point density threshold for the 0.2 m × 0.2 m grid is calculated to be 389 pts/m², and the grids retained after selection with this threshold are shown in Figure 4c. From this, we can see that almost all of the buildings’ boundary structures are effectively retained, and a large number of non-wall grids have been removed. Figure 4d shows the 0.2 m × 0.2 m grids retained after height selection; it is clear that the remaining grids accurately express the boundary structure of the actual buildings with fewer noise points. Figure 4e shows the vector boundary segments from the application of the Hough transformation on Figure 4d. Almost all the boundary information in Figure 4d is retained in these vector boundaries, despite the presence of some line segment breakage and non-intersection, which can be effectively processed through subsequent measures.

In applying the boundary segment extraction to the ALS data, the height difference threshold, based on the height value of the lowest building in this experimental area, is determined to be 20 m. The grids retained after selection based on this threshold. By applying the Hough transform, we get the vector boundary segments shown in Figure 5b. It can be seen that the main building structures have been effectively extracted, and that they correspond to the terrestrial boundaries, providing sufficient information for subsequent corner extraction and registration.

4.2.2. Building Corner Extraction

The boundary self-extending method is used to cover the airborne and terrestrial boundaries, as shown in Figure 5a,b, respectively. The boundaries within the dotted line box in Figure 5b correspond to the boundaries in Figure 5a. The boundaries share many parts, which effectively supports the concept of boundaries as a reliability check in the subsequent process. Figure 5c,d shows the building corners, which are formed by the line-intersection operation. There are 20 terrestrial corners (dark gray triangles) and 58 airborne corners (light gray circles), respectively, among which the number of corresponding building corners is 13. As the accuracy of corresponding building corners directly relates to the final registration accuracy, we evaluate the geometric accuracy of the building corners, as listed in

Table 1. Geometric accuracy of extracted building corners.

	Horizontal Error (m)			Vertical Error (m)			Total Error (m)
	Average	Max	RMSE	Average	Max	RMSE	Average	Max	RMSE
Airborne corner	0.66	1.22	0.73	0.20	0.29	0.21	0.70	1.24	0.77
Terrestrial corner	0.25	0.38	0.26	0.11	0.21	0.11	0.27	0.39	0.29

. The horizontal positions of real building corners are manually obtained from an orthophoto, and the elevations are determined through the averaging of neighboring LiDAR points.

4.2.3. Registration Results of ALS and TLS Data

Figure 6 illustrates the final results of the registration of airborne and terrestrial corners by the proposed approach. Figure 6a shows the automatic matching results of airborne corners (circles) and terrestrial corners (triangles) from the corner matching method with boundaries as a reliability check. We can see 13 pairs of successfully matched corners, with a matching correctness of 100%. Figure 6b shows the results of boundaries under the matching relationship of Figure 6a. In Figure 6b, the black dotted lines are terrestrial boundaries and the grey solid lines are airborne boundaries. It can be seen that the two sets of boundaries are well matched. Based on the matching results of Figure 6c, LMS is used to refine the registration, as shown in Figure 6d. After accurately matching the corners through the LMS algorithm, we obtain the final transformational relationship for point cloud registration. Guided by this transformation, the registration between the airborne point cloud (dark gray) and the terrestrial point cloud (light gray) is complete, as shown in Figure 7a. Thus, it is clear that the terrestrial point cloud falls exactly on its actual location with a good registration result. Figure 7b–e shows zoom-in views of four buildings in Figure 7a. For each building, the airborne point cloud and terrestrial point cloud are registered very well.

4.3. Reliability Analysis

In this study, the reliability check strategy ensures the reliability of the matching. To check the registration reliability, we designed an experimental analysis by comparing the proposed approach (registration with boundaries as a reliability check) with the maximum consensus approach (registration based on maximum consensus of two point sets without reliability check).

The reliability is calculated as follows:

$$\mathit{reliability}=\{\begin{array}{c}\frac{C_n^2}{M},C_n^2<M\\ 1\end{array}$$

(15)

where n is the number of actual conjugate point pair, $C_n^2$ is the number of transformational parameters that may lead to the correct correspondence of points, and M is the total number of transformational relationships that can be obtained using different algorithms.

As shown in Figure 8a, three of the terrestrial corners are selected for the reliability test. Figure 8b shows the possible matching results using the proposed approach (with boundary as a reliability check), the reliability of which is 100%. Figure 8c shows two possible matching results using the maximum consensus approach. Label A refers to the correctly matched pairs and Label B refers to the incorrectly matched pairs; thus, the reliability is 50%. Due to the distance threshold of the maximum consensus approach directly determines the matching results, it has a great effect on the reliability evaluation. This study therefore sets the distance threshold of the maximum consensus approach to range from 1 m to 10 m, enabling the calculation of reliability under different thresholds. The results are shown in Figure 8d. It can be seen that, with an increase in the distance threshold, the matching reliability declines sharply.

To prove the reliability of the proposed approach under a different number of boundaries, 10 different boundaries are selected, as shown in Figure 8e. The reliability results are shown in Figure 8f. Under the reliability check, we can see that the reliability of the registration obtained through the proposed method remains 100%. Compared with the maximum consensus approach, the use of a reliability check to determine matching corner pairs has higher reliability.

4.4. Geometric Accuracy Analysis

The evaluation on geometric accuracy of registration includes two aspects: building corner accuracy and common ground point accuracy.

Here, the building corner accuracy refers to the residual error of conjugate airborne and terrestrial corners. The evaluation result is listed in

Table 2. Geometric accuracy of matched corners.

	Horizontal Error (m)			Vertical Error (m)			Total Error (m)
	Average	Max	RMSE	Average	Max	RMSE	Average	Max	RMSE
Proposed method	0.44	0.77	0.49	0.15	0.34	0.18	0.50	0.79	0.53

. From the table, we can see that a precise registration result is obtained, especially along the vertical direction.

As there are errors in the extracted building corners (both airborne and terrestrial corners), it is not so objective to evaluate the registration accuracy using these corners. A further evaluation is conducted with some sample points in common regions to evaluate the geometric accuracy of the two registered point clouds in the vertical direction. We randomly select 25 sample points in this common region to compare the height difference between the interpolation results of ALS and TLS data. The average error, max error, and RMSE of the sample points are 0.28 m, 0.43 m, and 0.31 m, respectively.

5. Conclusion

An approach for the semi-automatic registration of ALS and TLS data has been proposed. The building corners and boundary segments extracted from ALS and TLS data, respectively, are combined as the registration elements. Based on an experimental analysis of our method, the following conclusions can be drawn.

(1)

The grid density method for boundary extraction from TLS data is feasible. The key to this method is the ability of the proposed theoretical estimation to determine an appropriate threshold.

(2)

Airborne corners and terrestrial corners can be reliably matched in a fully automatic way, and the reliability check strategy ensures the reliability of this automatic process. It is meaningful to combine points and lines for the fast generation of reliable matching relationships.

(3)

The proposed method can result in both high reliability and high geometric accuracy of the final registration of ALS and TLS data.

Despite of the advantages of the proposed methodology, limitations still exist. First of all, it comes to the existing of buildings with eaves. As building contours are used for the registration, building contours from ALS should correspond to those from TLS; nevertheless, due to the existing of eaves, the contours from ALS and TLS may deviate at a certain distance, which may lead to the large error of registration. Second, during the extraction of contours from TLS, as a global threshold is calculated with the height of building and scanning point density, the scanning point density of multiple TLS scans should be the same, which is to be noticed in the methodology.

The registration results of ALS and TLS data provide a new dataset for accurate and detailed 3D reconstruction, as well as many other applications. More importantly, the integration of multi-platform laser scanning data may inspire new applications. Due to the large differences between ALS and TLS data, there exist gaps in the spatial scale and geometric accuracy of the features (corners and boundaries) extracted from airborne data and terrestrial data. Therefore, further study should be conducted into the fusion of these two kinds of data to bridge existing gaps and improve the quality of feature extraction.