Global Registration of Terrestrial Laser Scanner Point Clouds Using Plane-to-Plane Correspondences

Abstract

Registration of point clouds is a central problem in many mapping and monitoring applications, such as outdoor and indoor mapping, high-speed railway track inspection, heritage documentation, building information modeling, and others. However, ensuring the global consistency of the registration is still a challenging task when there are multiple point clouds because the different scans should be transformed into a common coordinate frame. The aim of this paper is the registration of multiple terrestrial laser scanner point clouds. We present a plane-based matching algorithm to find plane-to-plane correspondences using a new parametrization based on complex numbers. The multiplication of complex numbers is based on analysis of the quadrants to avoid the ambiguity in the calculation of the rotation angle formed between normal vectors of adjacent planes. As a matching step may contain several matrix operations, our strategy is applied to reduce the number of mathematical operations. We also design a novel method for global refinement of terrestrial laser scanner data based on plane-to-plane correspondences. The rotation parameters are globally refined using operations of quaternion multiplication, while the translation parameters are refined using the parameters of planes. The global refinement is done non-iteratively. The experimental results show that the proposed plane-based matching algorithm efficiently finds plane correspondences in partial overlapping scans providing approximate values for the global registration, and indicate that an accuracy better than 8 cm can be achieved by using our global fine plane-to-plane registration method.

1. Introduction

In recent years, the rising role of building information modeling (BIM) has driven the market for static terrestrial laser scanners (TLS). TLS is an efficient and cost-effective technology for rapid and accurate collection of 3D point clouds. TLS point clouds are the most suitable data source for the generation of as-built BIMs, which are a crucial tool for many construction and architecture professionals. To derive globally consistent 3D point cloud models from TLS with high positional accuracy, registration is a mandatory task. Typically, in the existing frameworks, the registration of multiple point clouds is divided into a pairwise registration step and a global fine registration step. Pairwise registration involves finding feature correspondences between pairs of point clouds and minimizing the sum of residuals over all such feature correspondences for the estimation of transformation parameters (3D rotation matrix and 3D translation vector), which establish the relative orientation for each pair of scans in a common coordinate system. In practice, pairwise registration using free-form correspondences (e.g., iterative closest point algorithm [1]), feature point-based (e.g., keypoints) methods [2,3,4,5,6,7], or primitive-based (e.g., lines or planar surfaces) approaches [8,9,10,11,12,13] should be applied first to obtain the transformation parameters. However, a problem which arises in the registration of multiple point clouds is that corresponding scan features may still present significant residual errors from pairwise registration task. As a direct consequence, most of the pairs of registered point clouds will contain loop closing errors, thus reducing their local metric accuracy. Then, the global refinement should be used to minimize the influence of pairwise alignment errors on the transformation parameters, by distributing the residual errors evenly across the scan project [14]. Our present contribution is twofold. First, a plane matching strategy by decoupling rotation and translation parameters is proposed. The proposed matching algorithm makes use of a new parametrization based on complex numbers to find the correspondence between pairs of segmented planes. Second, a new global closed-form solution is proposed via graph-based formulation adapted for plane-to-plane correspondences. The proposed solution refines the sensor positions by treating the 3D points and their corresponding surface normal as observations. To the best of our knowledge, the proposed method is the first using this approach.

The rest of the paper is organized as follows. Section 2 provides the related works in point cloud registration, helping the reader to gain an insight into the registration problem. The studied area, our proposed plane-based matching approach, and our global fine registration solution based on plane correspondences are presented in Section 3. Experimental evaluation of the proposed solution on real datasets and a discussion about the potential and limitation of the proposed method are given in Section 4. Finally, the paper is concluded in Section 5.

2. Related Work

Typically, standard global refinement solutions first find the transformation parameters using [1], then evenly redistribute loop closing errors with the help of a graph-based optimization such as the one proposed by [15], which has been extended to 3D by [16]. This task is formulated as a scan graph, in which each scan denotes a node, and each edge highlights the spatial constraint between the pairs of nodes. A globally consistent registration of multiple point clouds via graph optimization was proposed by [14]. The authors find the coarse alignments using an unambiguous geometric keypoint configuration able to reduce the number of candidates that are sampled in the iteration process, called on K-4PCS [6], and the pairwise candidate solutions are filtered using a discrete graphical model of the scan network. Then, least-squares optimization [15] is used to evenly distribute the residual error. Ji et al. [13] generated a globally consistent 3D map of high-speed viaduct point clouds using closing condition and external geometric constraints. Their proposed method uses artificial targets for the iterative pairwise registration step. The pairwise transformations are also refined with [1], followed by global refinement to evenly spread the residual errors, as described in [15]. Yang et al. [17] use a branch-and-bound strategy to globally solve the objective function. It first uses the iterative closest point algorithm for a coarse registration task and refines the transformation parameters with the algorithm of [15]. The approach proposed by Huber and Hebert [2] uses surfaces and a Bayesian filter for alignment of multi-view 3D point clouds. The main disadvantage of this method is its high computational cost. Other global refinement approaches rely on general graph optimization [18], bundle adjustment [19], low-rank sparse decomposition strategy [20], kernel-based energy function [21] and visual bag of words [22], which are computationally less attractive.

The aforementioned existing methods for registration of multiple point clouds are prominently iterative free-form or point-based correspondences. Despite their popularity, iterative solutions have several limitations. For instance, they are not effective for sparse point clouds, require good approximate values to avoid convergence to weak local minima, they are sensitive to noise, and involve a matching step which incurs a high computational cost. Furthermore, when a downsampling step is applied, as in [6], the details of objects in the scene may be lost [12]. In contrast, closed-form solutions estimate the transformation parameters in one-step and they do not require approximate values. Additionally, plane-based approaches can achieve alignments with higher accuracy, are less influenced by the presence of outliers [23], and are more robust in identifying corresponding feature pairs [24]. In [25] a closed-form solution for pose-graph relaxation is introduced for enhancing the consistency of 3D maps. First, planar surfaces are extracted from the point clouds and matched. In the subsequent step, the pairwise registration of scans is performed. Assuming that the rotation parameters are accurately estimated, the sensor pose vectors are then refined using the least-squares solution of [15]. Pavan and dos Santos [26] introduce a global closed-form refinement of multiple point clouds with local consistency of planes. They use the similarity of plane properties and the geometric constraint formed by planar surfaces to identify correspondences and place all the rotation parameters into a common coordinate system by exploiting a property of quaternions. Moreover, the global refinement procedure is done using a combination of [1] and the global refinement algorithm of [15]. Yan et al. [27] presented a framework for global registration of building point clouds using portals (windows and doors) of rooms to connect scans with limited overlap, while the global refinement step is formulated based on linear integer programming. First, they classify all the points into horizontal and vertical groups according to the normals of the points. Then, portals are extracted by detecting boundary points in the vertical segments. After, pairwise matching are encoded in a connection graph. The globally consistent registration of the point clouds is obtained by choosing an optimal subset of the pairwise matchings from the graph. To avoid enumerate an extremely large number of constraints, an acceleration scheme by iterative adding constraints is proposed. During the registration step, local alignments remain conflicting after each optimization step. Thus, the set of constraints are added and the optimization is realized again. This process is repeated until the conflict free registration is obtained. The registration is refined using [1].

In general, closed-form solutions are more efficient because they provide the best transformation in one-step, without requiring initial approximations for each pair of scans. Closed-form solutions avoid convergence issues of iterative methods and have been shown to achieve more efficient and accurate registration results [28]. Since urban environments are characterized by an abundance of planar surfaces and current point cloud registration methods are inefficient for large-scale data [29], a closed-form global registration method based on planes have clear advantages for urban mapping applications [28,30]. While closed-form solutions are more suitable for pairwise registration of point clouds, there is a lack of global closed-form solutions in the literature, especially for plane-based approaches. In this article, we propose a new closed-form solution for global registration of point clouds from plane-to-plane correspondences.

3. Data and Method

3.1. Studied Area

We evaluate our proposed method on three different TLS datasets. We use one outdoor and two indoor scan projects for our experiments. Detailed information of the experiments is listed in

Table 1. The core specifications of the TLS sensors and outdoor/indoor experiments.

ID	TLS Sensor	Area (m)	Average Overlap (%)	Mean Point Density Points/m2	Mean Size (Million Points)	Number of Scans	Environment
Patio Batel	Faro LS 880	150 × 170	45	12	1.5	9	Urban (outdoor)
Lape	Faro LS 880	15 × 15 × 3	70	6	2.7	4	Office (indoor)
Royal exhibition building	Faro Focus S120	70 × 30 × 15	70	11	4.5	16 (for each floor)	Building (indoor)

.

The Faro 880 TLS offers a near-spherical field of view made possible by a 320° vertical angle scanning range and a 180° horizontal field with a linearity error in the rangefinder less than 3 mm at a range of 10 m, a ranging error about 1 mm and the measurement speed of 122,000 pts/sec. The Faro Focus S120 offers a distance accuracy up to $\pm$ 2 mm, a range from 0.6 m up to 120 m, a measurement rate up to 976,000 pts/sec, vertical field of view (vertical/horizontal) of 305°/360°, and a maximal vertical scan speed of 97 Hz. Figure 1 shows the marker position for each dataset.

Our outdoor scan project “Patio Batel” (located in a urban area in the south of Brazil, –30°04′09″S, –51°24′20″E) is part of a densely urban environment, in which there are cars, pedestrians, trees, poles, streets and building façades, as shown in Figure 1a. The “Lape” (located in a urban area in the south of Brazil, –30°04′09″S, –51°24′20″E) and the “Royal exhibition building” ” (located in an urban scene in the south of Australia, –38°34′06″S, 145°18′07″E) experiments were conducted to evaluate the performance of the method in an indoor environment. The first dataset contains features as tables, computers, chairs and façade constructions, as shown in Figure 1b. The “Royal exhibition building” experiment was carried out to evaluate the performance of the method with scans of a historical building with complex interior structures. The indoor area features as chairs and façade constructions, as shown in Figure 1c. Figure 2 shows some detailed views of unregistered point clouds for each datasets.

3.2. Method

In this section, we first describe the general prerequisites for TLS scans acquisition and preprocessing with a brief description of plane segmentation using the random sample consensus (RANSAC) algorithm [31]. Then, we present the closed-form registration method that first finds a set of plane correspondences with complex numbers and minimizes the sum of residuals over all such plane correspondences. Finally, the closed-form solution, which uses a set of plane-to-plane correspondences to consistently redistribute the residual errors across the whole project, is presented. In Figure 3, the proposed framework is shown, and the involved steps are illustrated. The details of these steps are explained in the subsequent sections.

3.2.1. Plane Segmentation with the RANSAC Algorithm

Planar surfaces are abundant in human-made environments, and their extraction requires less storage than voxel-grid representations. However, due to the varying point density of TLS data, measurement errors can lead to sparse outliers. These can influence the plane extraction task, and reduce the registration accuracy. To avoid these weaknesses, we start by removing outliers from the raw point cloud ($\aleph$) using the statistical outlier removal technique proposed by [32]. The filtered point cloud ${\aleph }^{\prime }$ is obtained based on a statistical analysis of a query point $p_i$ with respect to its surrounding neighbors $k$ $\in \aleph$. Thus, given a filtered reference point cloud $\aleph r^{\prime }$ and a filtered target point cloud $\aleph t^{\prime }$ with both inliers and outliers, the RANSAC algorithm is used for plane extraction task. This is done by randomly selecting three non-collinear points in ${\aleph }^{\prime }$, followed by a voting scheme to find the optimal fitting plane model. The best fit plane is estimated by finding the eigenvector corresponding to the smallest eigenvalue of the covariance of weighted point coordinates, as described in [17].

3.2.2. Estimation of Transformation Parameters from Plane Correspondences

Typically, pairwise registration of point clouds follows a two-step strategy. First, corresponding features are required in overlapping scans, and second, transformation parameters are estimated to align a pair of point clouds relative to each other in a common coordinate system. Our proposed strategy is based on plane correspondences and estimates the rotation and translation parameters separately. Given two sets of segmented planar surfaces $S_i$ $\subset$ $\aleph r^{\prime }$ and $S_i^{\prime }$ $\subset$ $\aleph t^{\prime }$, the transformation from $p_i\in$ $\aleph r^{\prime }$ to $p_i^{\prime }\in$ $\aleph t^{\prime }$ consists of a 3D rigid motion, as follows:

$$p_i^{\prime }=R{p}_i+t$$

(1)

where $R$ denotes the 3D rotation, $t$ represents a 3D translation and $p_i={\left[x_i,y_i,z_i\right]}^{\mathrm{T}}$ a point in 3D space. The condition that a point $p_i^{\prime }$ lies on a plane ${\pi }^{\prime }={\left[{{\mathrm{u}}^{\prime }}^{\mathrm{T}}, d^{\prime }\right]}^T$ in $\aleph r^{\prime }$, where ${\mathrm{u}}^{\prime }={\left[a^{\prime },b^{\prime },c^{\prime }\right]}^{\mathrm{T}}$ is the normal vector of plane $\pi$ and d its distance to the origin, can be written as [26]:

$${{\mathrm{u}}^{\prime }}^{\mathrm{T}}{p}_i^{\prime }+d^{\prime }=0$$

(2)

where u′ is the unit normal vector of the plane ${\pi }^{\prime }$ and d′ its distance to the origin. By substituting Equation (1) into (2), we can obtain the following expression:

$${{\mathrm{u}}^{\prime }}^{\mathrm{T}}R{p}_i+{{\mathrm{u}}^{\prime }}^{\mathrm{T}}t+d^{\prime }=0$$

(3)

from Equation (3), we have:

$${{\mathrm{u}}^{\prime }}^{\mathrm{T}}t=-({{\mathrm{u}}^{\prime }}^{\mathrm{T}}R{p}_i)-d^{\prime }=d-d^{\prime }$$

(4)

note that the rotation $R$ from vector $\mathrm{u}$ to the vector ${\mathrm{u}}^{\prime }$ is expressed as:

$${\mathrm{u}}^{\prime }=R\mathrm{u}$$

(5)

We now have two separate sets of equations per plane correspondence, one for t, as presented in Equation (4), and one for R, as can be seen in Equation (5). Given a set of putative correspondences, we can estimate the transformation parameters R and t using a closed-form solution. As described by [26], a system of equations $At=l+v$ can be formed to estimate the parameter t, where $A$ and $l$ are obtained by stacking ${\mathrm{u}}^{\prime }$ and $d-d^{\prime }$ for all plane-to-plane correspondences and $v$ contains the residual values. The least-squares solution for t can be obtained as: $\widehat{t}={\left(A^{\mathrm{T}}A\right)}^{-1}{A}^{\mathrm{T}}l$. The estimate of R is obtained using Horn’s solution [33]. In the subsequent section, we describe our proposed method to find corresponding planes using complex numbers.

3.2.3. Plane Matching with Complex Numbers

In its basic form, the estimation of the transformation parameters relies on the presence of pairs of corresponding planes. Dold and Brenner [34] give a plane matching method for TLS point clouds. They investigated the combinatorial complexity of the search for corresponding planes. Our plane matching algorithm extends this method by adding a false positive detection task. This is done by inferring the relative angle between patch plane vectors using complex numbers. It works in three steps: (1) Classification of the segmented planes; (2) Determination of the rotation between pairs of candidate planes; and (3) Calculation of approximate translation.

• Classifying the segmented planes

First, we assume that the TLS is mounted on a tripod and is levelled using a bubble level. Consequently, planes extracted from TLS data have small deviations from vertical and horizontal planes [35]. Thus, given two sets of segmented planar surfaces $S_i$ $\subset$ $\aleph r^{\prime }$ and $S_i^{\prime }$ $\subset$ $\aleph t^{\prime }$, our algorithm separates these in different classes, i.e., horizontal planes (representing the planes parallel to the floor) and vertical planes (representing the wall planes of buildings) using the third component of the normal vector of $S_i$ or $S_i^{\prime }$. The horizontal planes are those whose normal vectors and the z axis form an angle smaller than 3 degrees, while the vertical planes are those whose normal vectors and the z axis form an angle larger than 89 degrees. Such classification procedure is essential to reduce the search space, and for the optimization of the matching performance.

• Calculating the rotation between normal vectors of pairs of candidate planes

Second, given a set of segmented planes in both the reference point cloud ($\aleph r^{\prime }$) and the target point cloud ($\aleph t^{\prime }$), as presented in Figure 4a, in order to calculate the rotation angle ($\theta$) between the normal vectors (Figure 4b) of each pair of candidate planes for correspondence (Figure 4c), we propose the use of complex numbers. Figure 4 shows an example of segmented planes, the normal vectors and the relative roation angle between two segmented planes in $\aleph r^{\prime }$ and $\aleph t^{\prime }$.

As shown in Figure 4c, ${\theta }_1$ and ${\theta }_2$ represent the rotation angles of ${\mathrm{u}}_1$ to ${\mathrm{u}}_1^{\prime }$ and ${\mathrm{u}}_2$ to ${\mathrm{u}}_2^{\prime }$, respectively, where ${\mathrm{u}}_1$ and ${\mathrm{u}}_1^{\prime }$ are the normals of the pair of segmented planes ${\pi }_1$ in $\aleph r$ and ${\pi }_1^{\prime }$ in $\aleph r^{\prime }$, and ${\mathrm{u}}_2$ and ${\mathrm{u}}_2^{\prime }$ are the normals of the pair of segmented planes ${\pi }_2$ in $\aleph t$ and ${\pi }_2^{\prime }$ in $\aleph t^{\prime }$. For vertical planes, we can assume that $R$ is only applied around the $z$ axis; then, for every pair of adjacent non-parallel vertical planes, we have:

$${\mathrm{u}}^{\prime }=\mathrm{Ru} \therefore \left[\begin{array}{c}{a}^{\prime }\\ b^{\prime }\\ c^{\prime }\end{array}\right]=\left[\begin{array}{ccc}\cos \theta & -\sin \theta & 0\\ \sin \theta & \cos \theta & 0\\ 0& 0& 1\end{array}\right] \left[\begin{array}{c}a\\ b\\ c\end{array}\right]$$

(6)

Equation (6) can be rewritten as:

$$\begin{array}{c}{\left[\begin{array}{cc}{a}^{\prime }& b^{\prime }\end{array}\right]}^T=\mathrm{S}\left(\theta \right){\left[\begin{array}{cc}a& b\end{array}\right]}^T\\ c^{\prime }=c\end{array}$$

(7)

where $\mathrm{S}\left(\theta \right)$ is the rotation matrix in the $xy$ plane.

For all angles $\theta$, the matrices $\mathrm{S}\left(\theta \right)$ form a special orthogonal group $\mathbb{M}\left(2\right)$. All rotation $\mathrm{S}\left(\theta \right)$ in the plane can be represented by a complex number. Then, the rotation in the plane of a vector $\left(a, b\right)\in {ℝ}^2$ can be rewritten by the multiplication of complex numbers, as follows [36]:

$$a^{\prime }+i{b}^{\prime }=e^{i\theta } \left(a+ib\right),$$

(8)

where $e^{i\theta }=\cos \theta +i\sin \theta$. Thus, the complex number $e^{i\theta }$ can be calculated by multiplying Equation (8) by the inverse of the complex number $\left(a+ib\right)$, as follows:

$$e^{i\theta }=\frac{\left(a{a}^{\prime }+b{b}^{\prime }\right)-i\left(a{b}^{\prime }-b{a}^{\prime }\right)}{a^2+b^2}$$

(9)

Then, the rotation angle between $\left(a,b\right)$ and $\left(a^{\prime },b^{\prime }\right)$ can be calculated using the following expression:

$$\theta =atan\left(cos\theta ,sen\theta \right)$$

(10)

Subsequently, the algorithm checks the correspondence for every two pairs of non-parallel vertical planes calculating the difference between their relative angle ($\epsilon$), as follows:

$$\epsilon =\left|{\theta }_1-{\theta }_2\right|$$

(11)

The pseudo-correspondences between the two pairs of non-parallel vertical planes are established if $\left|\epsilon \right|\le 1 \mathrm{degree}$, otherwise, the correspondence hypothesis is rejected. Please note that, to avoid the singularity problem, the planes must be non-parallel vertical planes, since vertical planes are influenced by the rotation of the TLS sensor.

• Selection of pairwise plane correspondences

In the third step, the components $t_x$ and $t_y$ are calculated using the pairs of corresponding non-parallel vertical planes. For this step, we can reduce the search space by initially assuming that $t_z=0$. Thus, we can use two corresponding planes to estimate an approximation of $t_x$ and $t_y$, as follows:

$$\begin{array}{c}{a}_1{t}_x+b_1{t}_y={\mathrm{u}}_1^{\mathrm{T}}t=d_1-d_1^{\prime }\\ a_2{t}_x+b_2{t}_y={\mathrm{u}}_2^{\mathrm{T}}t=d_2-d_2^{\prime }\end{array}$$

(12)

which can be further decomposed to:

$$\left[\begin{array}{cc}{a}_1& b_1\\ a_2& b_2\end{array}\right]\left[\begin{array}{c}{t}_x\\ t_y\end{array}\right]=\left[\begin{array}{c}{d}_1-d_1^{\prime }\\ d_2-d_2^{\prime }\end{array}\right]$$

(13)

where ${\mathrm{u}}_1={\left[\begin{array}{ccc}{a}_1& b_1& 0\end{array}\right]}^{\mathrm{T}}$, ${\mathrm{u}}_2={\left[\begin{array}{ccc}{a}_2& b_2& 0\end{array}\right]}^{\mathrm{T}}$, ${{\mathrm{u}}_1}^{\prime }={\left[\begin{array}{ccc}{a}_1^{\prime }& b_1^{\prime }& 0\end{array}\right]}^{\mathrm{T}}$, ${{\mathrm{u}}_2}^{\prime }={\left[\begin{array}{ccc}{a}_2^{\prime }& b_2^{\prime }& 0\end{array}\right]}^{\mathrm{T}}$ and $\mathrm{t}={\left[\begin{array}{ccc}{t}_x& t_y& t_z\end{array}\right]}^{\mathrm{T}}$. The solution for Equation (13) is given as follows:

$$\left[\begin{array}{c}{t}_x\\ t_y\end{array}\right]=\frac{1}{a_1{b}_2-b_1{a}_2}\left[\begin{array}{cc}{b}_2& -b_1\\ -a_2& a_1\end{array}\right]\left[\begin{array}{c}{d}_1-d_1^{\prime }\\ d_2-d_2^{\prime }\end{array}\right],$$

(14)

An issue which requires attention in this solution is the correspondence between non-parallel vertical planes. The choice of parallel vertical planes can lead to an inconsistent system of equations with no solution. Similarly, the solution for component $t_z$ is obtained using the third pair of corresponding non-parallel vertical planes, as follows:

$$t_z={\mathrm{u}}_3^{\mathrm{T}}t=d_3-d_3^{\prime }$$

(15)

where ${\mathrm{u}}_3={\left[\begin{array}{ccc}0& 0& 1\end{array}\right]}^{\mathrm{T}}$ and ${{\mathrm{u}}_3}^{\prime }={\left[\begin{array}{ccc}0& 0& 1\end{array}\right]}^{\mathrm{T}}$ are parallels to the $z$ axis. Next, all pairs of vertical planes considered as pseudo-corresponding are combined to find the correct correspondences. This is done by using the approximated translation values and the rotation angle, as follows:

$$\begin{array}{c}erro{r}_{xy}={{\mathrm{u}}^{\prime }}^{\mathrm{T}}t+d^{\prime }-d\\ \delta =\left|{\theta }_1-\beta \right| \end{array}$$

(16)

where $\beta$ is the rotation angle from normal vector $\mathrm{u}$ in $S_i$ to the combined normal vector ${\mathrm{u}}^{\prime }$ in ${S_i}^{\prime }$, $d$ and $d^{\prime }$ are the perpendicular distances between the origin of the coordinate system and the combined planes, respectively.

For $\delta$ values less than a predefined threshold and for the smallest $erro{r}_{xy}$ value, the pair of corresponding planes is added to the set of pseudo-correspondences with respect to the planes ${\pi }_1\leftrightarrow {{\pi }_1}^{\prime }$ and ${\pi }_2\leftrightarrow {{\pi }_2}^{\prime }$. Then, the two steps of rotation angle estimation and the approximated estimation of $t_x$ and $t_y$ components are repeated using other planes and a new set of pseudo-corresponding planes is found. The combinations inserted in this set are regarded as true correspondences. In addition, the third component of translation parameters ($t_z$) is recalculated using corresponding horizontal planes. Thus, pairs of pseudo-corresponding horizontal planes are combined to find the correct correspondences using the following expression:

$$erro{r}_z={{\mathrm{u}}^{\prime }}^{\mathrm{T}}t+d^{\prime }-d,$$

(17)

When $erro{r}_z$ is less than a predefined threshold, the pair of corresponding horizontal planes is added to the set of pseudo-correspondences with respect to the planes ${\pi }_3\leftrightarrow {{\pi }_3}^{\prime }$. The combinations inserted in this set are also regarded as true correspondences. After all verifications, the potential corresponding planes are used to refine the translation parameters.

3.2.4. Proposed Global Plane-to-Plane Refinement Solution

Due to the accumulated residual errors resulting from the consecutive pairwise registration steps, a global refinement task should be used to minimize the loop closure error across all scan pairs in the project [14]. Given a discrete set of putative pairwise registrations, we develop a global refinement solution based on a plane-to-plane approach. The global refinement is done with the help of a pose-graph structure.

According to [37], a graph $G\left(\mathbb{X},\mathbb{T}\right)$ is formed by nodes ($\mathbb{X}$) and edges ($\mathbb{T}$), where nodes represent the pose of each point cloud and edges denote each pairwise registration (transformation of pairs points cloud) with sufficient mutual overlap. An example of a graph structure is shown in Figure 5. The variables $X_i\in \mathbb{X}$ denote point clouds and the each pairwise registration $T_{ij}\in \mathbb{T}$ are represented by transformation parameters $M_{ij}$ (rotation) and $t_{ij}$ (translation) between the point clouds $X_i$ and $X_j$, respectively, with $j, i\in \left\{0,1, 2, \dots , n\right\}$.

• Global refinement of rotation parameters

The rotation parameters are globally refined ($R_j$) using our previous work [26], as follows:

$$R_j=M_{ij}{R}_i$$

(18)

where $R_i$ represents the 3D rotation matrix from node $X_i$ to the node $X_0$ (reference) and $M_{ij}$ denotes the 3D rotation matrix from $X_i$ to $X_j$ with $j$ and $i=0,$ $1$, $2$,$\dots$, $n$ (see Figure 4). Using quaternions, Equation (18) can be rewritten as follows:

$$R_j=M_{ij}{R}_i-{\mathsf{\Delta }}_{ij}$$

(19)

where $R_i$ represents the 3D rotation matrix from node $X_i$ to the node $X_0$ (reference) and ${\mathrm{M}}_{ij}$ denotes the 3D rotation matrix from $X_i$ to $X_j$ and ${\mathsf{\Delta }}_{ij}$ is the residual error, with $j, i\in \left\{0,1, 2, \dots , n\right\}$. Thus, Equation (19) is rewritten as follows:

$$\left[L_{q_{ij}}\right]Q_j-Q_i={\mathsf{{\rm E}}}_{ij}$$

(20)

where $\left[L_{q_{ij}}\right]$ is a 4 × 4 matrix associated with left product of the quaternion ${\dot{q}}_{ij}$, $Q_i$, $Q_j$ and ${\mathsf{{\rm E}}}_{ij}$ are 4 × 1 vectors that represent the quaternions ${\widehat{\dot{q}}}_i$, ${\widehat{\dot{q}}}_j$ and ${ϵ}_{ij}$, respectively. Equation (20) can be rewritten as follows:

$$\mathsf{\Omega }\mathsf{\Psi }=\left[\begin{array}{cc}\begin{array}{cc}\begin{array}{cc}\begin{array}{c}\begin{array}{c}\left[L_{q_{01}}\right]\\ -I\end{array}\\ \begin{array}{c}⋮\\ 0\\ -I\end{array}\end{array}& \begin{array}{c}\begin{array}{c}-I\\ \left[L_{q_{02}}\right]\end{array}\\ \begin{array}{c}⋮\\ 0\\ 0\end{array}\end{array}\end{array}& \begin{array}{cc}\begin{array}{c}\begin{array}{c}\cdots \\ \cdots \end{array}\\ \begin{array}{c}\ddots \\ \cdots \\ \cdots \end{array}\end{array}& \begin{array}{c}\begin{array}{c}0\\ 0\end{array}\\ \begin{array}{c}⋮\\ \left[L_{q_{n-1.n}}\right]\\ 0\end{array}\end{array}\end{array}\end{array}& \begin{array}{c}\begin{array}{c}0\\ 0\end{array}\\ \begin{array}{c}⋮\\ -I\\ \left[L_{q_{n0}}\right]\end{array}\end{array}\end{array}\right]\left[\begin{array}{c}\begin{array}{c}{Q}_0\\ Q_1\end{array}\\ \begin{array}{c}⋮\\ Q_{n-1}\\ Q_n\end{array}\end{array}\right]=\left[\begin{array}{c}\begin{array}{c}{\mathrm{E}}_0\\ {\mathrm{E}}_1\end{array}\\ \begin{array}{c}⋮\\ {\mathrm{E}}_{n-1}\\ {\mathrm{E}}_n\end{array}\end{array}\right]=E,$$

(21)

where $\mathsf{\Omega }$ represents a $4n\times 4m$ matrix formed by partial derivatives with respect to all components of $Q_j$, $Q_i$ is the column vector $4n\times 1$ formed by concatenation of all matrices $Q_i$ and $Q_j$, $n$ is the number of the nodes and $m$ denotes the number of the edges on the graph. Then, the total least square method [38] can be used to estimate the matrix $\mathsf{\Psi }$ (or $R_r$), as follows:

$${\mathsf{\Psi }}^{\mathrm{T}}{\mathsf{\Omega }}^T\mathsf{\Omega }\mathsf{\Psi }=E^{T}E\underset{}{\to }min,$$

(22)

Thus, the matrix $\mathsf{\Psi }$ that minimizes the Equation (22) is the eigenvector corresponding to the smallest eigenvalue of the matrix ${\mathsf{\Omega }}^T\mathsf{\Omega }$. To obtain ${\widehat{\dot{q}}}_i$ it is essential to normalize each $Q_i$. As this solution is direct, it does not require initial approximations.

• Proposed global refinement of translation parameters

By combining a pair of point clouds, a difference $R_r{t}_i$ between two sensor poses ${\mathrm{x}}_i$ and ${\mathrm{x}}_j$ can be obtained, as follows:

$$R_r{t}_{ij}={\mathrm{x}}_i-{\mathrm{x}}_j$$

(23)

where $R_r$ represents the globally refined rotation parameters, $t_{ij}$ is the 3D translation vector from node $X_i$ to $X_j$, ${\mathrm{x}}_i$ is a column matrix that represents the 3D position of the node ${\mathrm{X}}_i$ with ${\mathrm{x}}_0={\left[\begin{array}{ccc}0& 0& 0\end{array}\right]}^{\mathrm{T}}$ and ${\mathrm{x}}_j$ is a column matrix that represents the 3D position of the node ${\mathrm{X}}_j$. As described, the translation parameters can be estimated from plane correspondences:

$${\mathrm{u}}_{jk}^{\mathrm{T}}{t}_{ij}=d_{ik}-d_{jk}$$

(24)

where ${\mathrm{u}}_{jk}^{\mathrm{T}}$ denotes the normal vector from segmented plane ${\pi }_{jk}$ $\in$ $X_j$ and $d_{jk}$ its perpendicular distance from the origin, and $d_{ik}$ the distance from the origin to plane ${\pi }_{ik} \in$ $X_i$. Multiplying Equation (24) by ${\mathrm{u}}_{ijk}^{\mathrm{T}}{R}_r^{\mathrm{T}}$ and substituting it into Equation (23) yields:

$$d_{ik}-d_{jk}={\mathrm{u}}_{jk0}^{\mathrm{T}}{R}_j^{\mathrm{T}}\left({\mathrm{x}}_i-{\mathrm{x}}_j\right)$$

(25)

For each pair of point clouds within the network, the proposed method calculates the residual errors based on plane-to-plane correspondences. As the Equation (25) is linear with respect to the sensor positions ${\mathrm{x}}_1$, ${\mathrm{x}}_2$,$\dots$, ${\mathrm{x}}_n$ and assuming that $d_{ik}-d_{jk}$ is the vector of observations and ${\mathrm{u}}_{jk}^{\mathrm{T}}$ is the normal vector for all plane-to-plane correspondences, respectively, the solution $\left({\mathrm{x}}_i-{\mathrm{x}}_j\right)$ can be globally refined using the least squares method (LSM). Thus, the set of liner equations $A\mathrm{x}+v=l$ can be formed to estimate the sensor positions, where $A$ is the coefficient matrix containing the partial derivatives in Equation (25), $l$ is the vector of constants containing $d-d^{\prime }$ for all plane-to-plane correspondences and $v$ is the residual vector. The least-squares solution for $\left({\mathrm{x}}_i-{\mathrm{x}}_j\right)$ can be obtained as: $\widehat{\left({\mathrm{x}}_i-{\mathrm{x}}_j\right)}={\left(A^{\mathrm{T}}A\right)}^{-1}{A}^{\mathrm{T}}l$. Please note that the proposed method has been adapted to use plane parameters to estimate ${\mathrm{x}}_2$,$\dots$, ${\mathrm{x}}_n$, instead of feature points as used by [15]. To obtain a solution, at least four pairs of corresponding planes are needed. Fortunately, this is not a major issue in urban environments, since a large number of planar surfaces are available.

4. Experiments and Results

First, the plane matching method is evaluated in terms of matching success rate (SR), by simply counting the number of correctly matched planes over all pairs of point clouds and the matching time (MT). Its performance also is compared to the K-4PCS registration algorithm [6], which is available in the point cloud library (PCL) [39]. Secondly, the global refinement method is evaluated in terms of mean registration accuracy. The residual root means square errors (RMSE) before and after global refinement are calculated based on the plane-to-plane distances between the corresponding planes. The performance of the proposed global refinement method is evaluated by comparison with three other approaches. The first is a combination of iterative closest point algorithm [1] and the global refinement method proposed by Lu and Milios [15]. The second approach is the global registration using our pipeline but omitting the global refinement of the rotation parameters. The third approach is the global registration proposed by Theiler et al. [14]. The first approach is available in the point cloud library (PCL) [39], while the source code from the third approach can be found in [40].

4.1. Pre-Processing Data

For all the datasets, the outliers were detected and removed using the statistical outlier removal algorithm. With the removal of the outliers, the number of points contained in the point clouds was reduced by 25%. Basically, this step of the method removes from the data sample all points outside the interval ${\mu }_k\pm \alpha ·{\sigma }_k$. In this paper, the values assumed for the variables ${\mu }_k$ = 0.050 m (mean distance between neighbouring points) and $\alpha =0.10$ (restriction factor) were determined empirically and were those that best represented the expected sampling of the object on the surface. The surface planes were extracted with RANSAC algorithm using a distance threshold 0.01 m, and 100 iterations. Figure 6 shows an example of the segmented planes for pairs of point clouds obtained by using the RANSAC algorithm. As can be seen, ground planes could not be correctly classified because small inclinations exist due to the different ground levels in all scanned regions. However, this issue does not affect the performance of the matching algorithm.

The planes were classified and segmented into horizontal and vertical planes, with their normals and perpendicular distances calculated with respect to the origin. The $erro{r}_{xy}$ and $erro{r}_z$ threshold values were respectively 0.15 and 0.01 m for Patio batel and Lape datasets, while for the Royal exhibition building dataset, the value 0.01 m was used for $erro{r}_{xy}$ and $erro{r}_z$.

4.2. Experimental Evaluation

4.2.1. Plane Matching Evaluation

As previously mentioned, we tested our plane-based matching algorithm against the K-4PCS algorithm proposed by [6]. For the tests conducted with K-4PCS, we used the following matching algorithm and parameter settings: 3D scale feature transform (SIFT) algorithm for keypoints detection, and voxel size $\tau =50$ mm, respectively.

Table 2. Evaluation of our proposed plane-based matching algorithm and K-4PCS algorithm.

K-4PCS					Proposed Plane-Based Matching Algorithm
Dataset	#Mean extracted keypoints	Mean SR (%)	Mean MT (s)	Mean of RMSER (m)	# Mean corresponding planes	Mean SR (%)	Mean MT (s)	Mean of RMSER (m)
Patio Batel	16.840	98.90	995	0.60 ± 0.31	19	84.4	200	0.38 ± 0.15
Lape	13.850	95.82	789	0.49 ± 0.24	9	75.5	101	0.25 ± 0.11
Royal Building	25.742	99.67	1045	0.28 ± 0.21	35	95.2	127	0.16 ± 0.07

shows the results obtained with both our plane-based matching algorithm and the K-4PCS method. For the “Patio batel” dataset, a sufficiently large number of corresponding planes was obtained for the pairs of point clouds containing salient structures. For these pairs of point clouds, the method achieves high SR (>80%). The SR is generally high for point clouds with large overlap (about 40%). The lower SR for a few of the pairs is due to the smaller overlap (30%) and the high degree of symmetry of planar surfaces in the scene, which reduces the chance for finding correct matches. Table 2 also shows the metric accuracy of the estimated transformation parameters represented by the root mean square error between true correspondences (RMSE_R). The RMSE_R obtained with the K-4PCS algorithm is about 0.60 m, while our method achieved a RMSE_R around 0.40 m. Probable reason for a large RMSE_R value found is the lower overlap between the pairs of point clouds. Note also that both algorithms use plane correspondences, instead of targets. Thus, to improve the pairwise registration accuracy is necessary to apply the global refinement.

For the “Lape” dataset, the proposed plane matching has been carried out with the optimal parameters, $erro{r}_{xy}$ and $erro{r}_z$, set to 0.15 m and 0.01 m, respectively. From the test area, nine evenly distributed corresponding pairs of planes were found, which indicates a matching success rate above 75% (see Table 2). The reason for the moderate success rate is the high number of parallel planes discarded by the proposed plane matching algorithm to avoid shift errors. For the “Royal building exhibition” dataset, the optimal parameters are set to 0.01 m for both the planimetric and altimetric values. This setting is based on the overlap area between the point clouds: where for larger overlaps smaller values for the parameters are used. As can be observed, the reference and target point clouds are well registered. The mean SR for both the experiments is around 95%, as shown in Table 2. However, some failed cases with a large degree of scene symmetry occur. The results of the successful cases reveal that our proposed plane matching algorithm works well. For plane pairwise registration methods, the translational errors are larger than rotational errors due the estimate of normal vectors. Errors of orientation from normal vectors should produce large translation errors. For these experiments, the results reveal that the pairwise registration obtains an RMSE higher than 0.3 degrees and 0.2 m for the rotation error and translation error, respectively. As can be observed in Table 2, the K-4PCS achieve highest matching SR; however, the matching time is considerably higher. Examples of the results obtained with our proposed pairwise registration method can be seen in Figure 7.

4.2.2. Global Refinement Evaluation

As previously mentioned, for comparisons with state-of-the-art research, the renowned global refinement approach presented by [15] and the approach of Theiler et al. [14] are introduced. The results of the 3D globally consistent point clouds obtained with the proposed method can be seen in Figure 8.

Figure 9 shows the mean residual RMSE obtained for each dataset using the proposed method and the baseline approaches. The RMSE values shown that the proposed global refinement method achieves more accurate results than both the method of [15] and omitting the global refinement of the rotation parameters. It can also be seen that scans with more geometric constraints (plane correspondences with other scans) are registered more accurately after global refinement. The lowest registration accuracy is obtained for sensor poses III and VIII which have fewer corresponding planes.

Figure 10 shows the RMSE value of the registration before and after the global refinement. As expected, the proposed global refinement method significantly reduces the misalignments. After the global refinement we achieved accuracies better than 8 cm, that is close to the expected measurement accuracy of the scanning device.

4.3. Discussion

The proposed point cloud registration algorithm involves three prerequisite algorithms, including outlier removal, plane segmentation, and plane-based matching. The complexity of the proposed algorithm is centred on the plane-based matching algorithm. In practice, by calculating $\theta$ using the dot product of the two normal, only internal angles are verified, because such a process is limited to the interval from 0 to 180 degrees, producing ambiguity between the angles. However, a normal vector can be rotated from −180 to 180 degrees, that is, internal angles and external angles should be verified. To overcome this limitation, we use complex numbers to find the angle between the normal vectors of each pair of candidate planes referencing to the point at which this plane is rotated. Using complex numbers helps to avoid ambiguity between the angles through an analysis of the quadrants (see Equation (11)). Note also that we constraint the sign of the parameter d to be positive to orient all $\mathrm{u}$ towards the viewpoint. Using complex numbers to calculate the rotation angles between adjacent non-parallel vertical planes, we can decrease the number of mathematic operations comparisons with matrix operations. For the plane-based matching algorithm, plane extraction and manual setting of parameters is mandatory, as these arguments reduce the number of false positives and the number of candidate features. A close inspection of false-positive detection step revealed that the proposed plane-based mathing algorithm can reduce the number of false positives about 79%. As expected, failure cases are caused by the symmetry of the scenes, which are often in the indoor areas. As limitations of the plane-based scheme, we can point out the requirement for horizontal and vertical planes. Although surface planes are abundant in urban scenes, the behavior of the proposed strategy is not consistent for forest environments and under occlusion or very low overlap. The pairwise registration performance is better with high valid overlapping area such as for indoor environments. However, indoor environments remind us that parallel planes will increase the risk of failure for the proposed plane matching algorithm. In light of the proposed plane-based matching algorithm a minimum of three non-parallel pairs of correct plane correspondences must be found at the matching step. Fortunately, this is not a major issue in man-made environments (e.g., cities) and indoor environments that can be used in several applications, such as indoor navigation, infrastructure inspection, façade 3D modelling, cultural heritage documentation, mapping, augmented reality, 3D modelling of industrial objects, and others. With respect to the matching performance, as the rotation of the TLS is around z axis, the horizontal planes are less influenced by rotation of the TLS, while vertical planes are influenced by the rotation of the TLS. Thus, we assume that horizontal planes are those whose normal vectors and the z axis form an angle smaller than 3 degree and vertical planes are those whose normal vectors and the z axis form an angle larger than 89 degree. This is essential for reducing the search space, and for the optimization of the matching performance. As a direct consequence of the TLS scans, different plane parameters may be obtained from different views. As a result, the plane-based registration is influenced by the plane extraction accuracy. The $d$ parameter is defined as a function of the $u$ (normal plane) and point coordinates, while the precision of plane fitting is dependent on both the noise level of the points and the choice of the coordinate system, as presented by [28]. Since the estimation of translation parameter ($t$) in the plane-based solution is dependent on the d parameter of the noisy planes, a solution is normalized point coordinates to improve the estimate of $d$ parameter in plane fitting, and consequently uses the variance of $d$ in plane fitting to improve the estimate of translation parameters ($t$) during the pairwise registration solution. However, this has not been tested, and is left for future work.

Compared with other global refinement methods, our method is a closed-form solution and refines both rotational and translational parameters. From our point of view, the refinement of the rotations is essential for the correct refinement of the sensor positions. Thus, as previously mentioned, we first refine the rotations using the approach indicated by [26], while the TLS positions are corrected with our plane-to-plane refinement method. In practice, the pairs of point clouds with lowest overlapping point clouds present worst results for the pairwise registration task and the alignment is not good enough to be refined by our method. To disambiguate the pairwise transformations, geometric constraints should be added in the pose-graph optimization. Moreover, pairs of point clouds that lack salient structures, and contain more occlusions and vegetation present the largest errors. Although our global refinement method is compatible in terms of accuracy compared to the state-of-the-art, it is much faster, corresponding to 55% less computation time, as can be seen in see Figure 11.

5. Conclusions

In this paper, we presented a framework for automatic marker-less registration of multiple terrestrial laser scanner point clouds. First, we introduced a plane-based matching scheme that relies on a new parametrization using complex numbers to find the correspondence between pairs of segmented planes. The RANSAC algorithm is used to segment plane surfaces in TLS data. The segmented planes are then classified into vertical and horizontal planes, and their correspondences are obtained using our proposed plane-based matching algorithm. The key novel aspect of our plane-based matching algorithm is the multiplication of complex numbers based on analysis of the quadrants to avoid the ambiguity in the calculation of the rotation angle formed between normal vectors of adjacent planes. It also is able to reduce the number of mathematic matrix operations during the correspondence task. Secondly, we formulated the global fine registration as a graph-based formulation adapted for plane-to-plane correspondences. The main characteristic of this proposed solution is that the refinement of TLS positions by treating the 3D points and their corresponding surface normal as observations. By globally refining the rotation parameters and the translation parameters, TLS position can be accurately obtained. Since our global registration method is non-iterative, multiple point clouds can be quickly registered. In our results, we demonstrated the potential of our method in registering point clouds of outdoor and indoor urban environments with reasonable overlapping. Future work involves conducting additional experiments using dataset of different sources, i.e., photogrammetry and RGB-D data. In addition, the loop closing detection can be executed automatically.