^{1}

^{2}

^{*}

^{3}

^{4}

^{5}

^{6}

This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution license (http://creativecommons.org/licenses/by/3.0/).

This paper presents a new approach to the automatic registration of terrestrial laser scanning (TLS) point clouds using panoramic reflectance images. The approach follows a two-step procedure that includes both pair-wise registration and global registration. The pair-wise registration consists of image matching (pixel-to-pixel correspondence) and point cloud registration (point-to-point correspondence), as the correspondence between the image and the point cloud (pixel-to-point) is inherent to the reflectance images. False correspondences are removed by a geometric invariance check. The pixel-to-point correspondence and the computation of the rigid transformation parameters (RTPs) are integrated into an iterative process that allows for the pair-wise registration to be optimised. The global registration of all point clouds is obtained by a bundle adjustment using a circular self-closure constraint. Our approach is tested with both indoor and outdoor scenes acquired by a FARO LS 880 laser scanner with an angular resolution of 0.036° and 0.045°, respectively. The results show that the pair-wise and global registration accuracies are of millimetre and centimetre orders, respectively, and that the process is fully automatic and converges quickly.

Terrestrial laser scanning (TLS) has been proven effective in urban mapping for applications ranging from as-built documentation to the three-dimensional (3D) reconstruction of architectural details and building facades. However, one of the biggest problems encountered while processing the scans is TLS point cloud registration in which rigid transformation parameters (RTPs) are determined in order to bring one dataset into alignment with the other. RTPs are computed by common points, obtained by correspondence tracking, in scenes from different viewpoints. Commercial software typically uses separately scanned markers that can be automatically identified as corresponding points. Several surface matching algorithms have been proposed to avoid the use of artificial markers. The most popular method is the iterative closest point (ICP) algorithm developed by Besl and McKay [

In recent years, a great deal of effort has been devoted to developing approaches based on segmenting TLS point clouds and thereby matching extracted primitives. Primitives are derived from point clouds and are matched in a semi-automatic or fully automatic way (e.g., [

Today, most TLS manufacturers offer the option of a high-resolution digital camera mounted on the scanner for users to capture digital imagery while TLS point clouds are collected, so as to generate photorealistic 3D object and scene models. The generally higher resolution of the optical images and the well-established image processing algorithms offer attractive possibilities for automatically aligning the TLS point clouds. Several methods can be found in the literature and are referred to as image-based registration (IBR). For example, Wendt [

Our earlier work described an algorithm for automatically registering TLS point clouds using reflectance images [

Many applications require more than two scans of any one object or scene. The global registration of multiple scans is more difficult because of the large nonlinear search space and the huge number of raw TLS data involved. Several useful approaches to this problem have been proposed in recent years [

This paper extends our previous work [

The remainder of this paper is organised as follows. The following two sections detail the proposed approach, including pair-wise registration and global registration, respectively. Section 4 discusses our test results, after which we offer our conclusions, together with suggestions for further research.

In this paper, we distinguish between three types of correspondences, namely, pixel-to-pixel, pixel-to-point, and point-to-point. Pixel-to-pixel correspondence is between image pixels of two overlapping images, while pixel-to-point correspondence involves image pixels and 3D points from a TLS scan. The latter is known when either reflectance imagery is employed or optical images are available. The TLS units with a built-in digital camera usually provide this correspondence. Alternatively, it can be computed by knowing the camera calibration and orientation parameters. Recent research (e.g., [

As pixel-to-point correspondence is inherent to reflectance images, we only explain how the reflectance images are constructed and elaborate on the two other processes, namely, pixel-to-pixel and point-to-point correspondence. Image matching (which allows us to establish the pixel-to-pixel correspondence) has been widely explored in photogrammetry and computer vision communities for years. Many mature algorithms for automatic pixel-to-pixel correspondence have been proposed to date (e.g., [

Throughout this paper, in identifying a pair of scans as being co-registered, the scan that defines the coordinate system is termed the

In general, the registration process requires RTPs. Our method computes RTPs using a two-stage iterative process that includes both pixel-to-pixel and point-to-point correspondence (see

The entire iterative process consists of five steps: (1) point extraction, (2) point correspondence, (3) outlier detection, (4) computing RTPs, and (5) correspondence prediction. The first two steps are performed on the reflectance images and output all possible candidate points for the subsequent computation of RTPs. The subsequent three steps are performed on the TLS points. 3D corresponding points are taken on the basis of the known pixel-to-point correspondence. When only image information is used, some correspondences may be incorrect; outlier detection should be implemented to remove such incorrectly matched points. The RTPs computed from the 3D corresponding points at the beginning may remain after outlier detection. Generally speaking, most of the corresponding points that exhibit repetitive patterns in those regions are likely to be falsely accepted by the image-based matching method. Thus, they should be removed after outlier detection so that the distribution of corresponding point pairs in the overlapping area is unlikely to be homogeneous. The iterative process can track out the correct correspondences. Using the RTPs computed in earlier iterations, 2D correct correspondences on the moving image can be better determined, which results in an increased number of matched points. Then, the new matches can be added into the computation of new transformation parameters for the subsequent iteration. The iterative process continues until the transformation parameters reach the predefined accuracy threshold.

In general, each instrument observation comprises four values, namely, horizontal and vertical angles, range and reflectance value. 3D points are calculated as a combination of the horizontal and vertical angles of the laser beam plus the measured distance. The reflectance imagery can be created directly from scans using the angular coordinates and reflectance value of each point (see

Concerning the TLS scans, the angular resolution Δ

The pixel-to-pixel correspondence is critical for our proposed method. The image matching algorithm employed to find corresponding points between the adjacent images is explained below.

Since 360° reflectance images are used, it is generally quite difficult to make any assumption regarding the set of possible correspondences for a given feature point that can be extracted by normal corner detectors, which merely conform to focal plane array optical cameras. We therefore use the SIFT method [

Similar to Lowe [

Scale-space extrema detection: employing a difference-of-Gaussian function to identify potential points of interest that are invariant to scale and orientation,

Key-point localisation: fitting an analytical model (mostly in the form of a parabola) at each candidate location to determine the location and scale,

Orientation assignment: assigning one or more orientations to each key-point location based on local image gradient directions, and

Key-point descriptor: measuring the local image gradients at the selected scale in the region around each key-point.

When using reflectance images, gray values depend heavily on the viewpoint and the angle of illumination. Moreover, the gradients may vary between different scans. However, the SIFT method can cope with these problems by reducing the effects of variable illumination conditions. First, the descriptor vector is normalised to unit length so that the image contrast change is cancelled by vector normalisation. A brightness change in which a constant is added to each image pixel will not affect the gradient values, as they are computed from pixel differences. Therefore, the key-point descriptor is invariant to affine changes in illumination. Variable non-linear illumination can cause a change in relative magnitudes for some gradients, but is less likely to affect the gradient orientations. Therefore, the influence of large gradient magnitudes is reduced by thresholding the values in the unit feature vector, and then renormalising to unit length. This means that matching the magnitudes for large gradients is no longer important, and the distribution of orientations carries greater weight.

To identify correspondences, the invariant descriptor vector for the key-point is given as a list of 128 integers in the range [0,255]. Correspondences can generally be tracked out among images. Key-points from a given image can be matched to those from previous images by simply looking for the descriptor vector that exhibits the closest Euclidean distance among all vectors from previous images. To identify matches, we use Lowe’s [

This approach can be used to easily identify false matches from the panoramic reflectance images of buildings, as building facades are likely to exhibit repetitive patterns. The rigid geometric invariance derived from TLS point clouds is used to prune false correspondences in the next step, namely, point-to-point correspondence.

In practice, it is impossible to correctly match all 3D correspondences. Most likely, some incorrect corresponding points also get inadvertently accepted. A rigid geometric invariance is used to detect and remove these false correspondences.

_{AB} = _{A′B′}.

However, this verification process may increase the computation time considerably. To avoid this, we construct a Triangulated Irregular Network (TIN) using Delaunay triangulation and use the relations between the points in the triangles to determine the distances. The TIN model is selected because of its simplicity and computational economy. The 2D Delaunay triangulation is computed in the image plane and then mapped to TLS point clouds. As the TIN is constructed to verify distance invariance between correspondences, we merely need to construct the TIN model for 3D points in one scan and replicate it in the other using the corresponding points. Consequently, only those point-pairs that are connected in the TIN model are verified for distance invariance.

The distance between the two points in Scan (a) should equal the distance between the two corresponding points in Scan (b), i.e., _{AB} = _{A′B′} (_{AB}_{A′B′}_{A}_{A}_{A}_{B}_{B}_{B}_{A′}_{A′}_{A′}_{B′}_{B′}_{B′}

The error of the distance invariance σ_{ΔS} can be estimated using the error propagation law in light of the location error of the two corresponding point pairs. Because the location error of each point is independent, the error estimation can be obtained by:
_{(XYZ)i} is the variance matrix of point

According to the error propagation law, the variance matrix of point _{R}_{θ}_{φ}_{(XYZ)i} is computed in terms of σ_{R}_{θ}_{φ}

Three times the error of distance invariance σ_{ΔS} is chosen as a threshold to determine the correct correspondence, so

In _{ΔS} is a variant and is related to each of the two corresponding pairs. Therefore, the chosen threshold is self-adaptive, instead of a constant. If the above condition is satisfied, those two point pairs are said to be corresponding. Otherwise, there should be an outlier among the point pairs. According to

If the four point pairs shown in _{ΔS}.

The coordinates of each single TLS scan are recorded in a local coordinate frame defined by the instrument. Using the corresponding points detected in the previous step, it is possible to compute transformation parameters between different coordinate frames and thus register the two point clouds. The least-squares adjustment for the absolute orientation in photogrammetry [

We note that, after outlier detection, the incorrectly matched points have been removed and RTPs can be computed only with the correct ones. However, the outlier detection may remove so many points that the remaining correspondences are unlikely to be uniformly distributed in the overlapping areas. Therefore, RTPs determined from these cannot be considered final. To improve them, more matching-appropriate points must be found. We use an iterative process to seek candidate points from amongst the feature points already extracted in the step pixel-to-pixel correspondence.

Using the initial RTPs and inherent pixel-to-point correspondence, the feature points in the fixed reflectance image can be projected onto the moving image for the purpose of correspondence prediction. As presented earlier, based on the image coordinates (

Each iteration process accordingly consists of four steps (see

In general, global registration considers the network of views as a whole and minimises the registration errors across all views simultaneously. The network topology can be star-shaped (e.g., [

Given six scans of a 3D object (see _{2-1}, _{2-1}), (_{3-2}, _{3-2}),_{6-5}, _{6-5}), corresponding to the motion of each slave into the master reference frame. The last-to-first closure constraint can be expressed as follows:
_{1}, _{1}, _{1}) and (_{m}_{m}_{m}_{i,X}_{i,Y}_{i,Z}_{i} is the rotation from

Intuitively, the point coordinates (_{m}, _{m}, _{m}) in the last scan _{1}, _{1}, _{1}) in the first scan using m−1 rigid transformations. The discord of this transformation process in practical terms is referred to as the closure error. However, in the case of

To tackle this imperfection, we now extend the closure constraint into a self-closure form. An additional transformation from S1 to S6 is therefore computed as (_{1-6}, _{1-6}). As shown in _{1-6}, _{1-6}) enables the first self-closure scan S1, which can be expressed as follows:

Compared with _{m}_{m}_{m}_{1}, _{1}, _{1}) and accordingly one more transformation (_{m}, _{m}) is added, which means that after m rigid transformations (_{1}, _{1}, _{1}) maps back to itself in the ideal case. The closure constraint is error-free. In practical terms, the closure error can reach least squares after a bundle adjustment that merely consists of parameter errors. In addition, the self-closure form can be further extended to a circular one, as clearly the transformation ring ((_{2-1}, _{2-1}),_{1-6}, _{1-6})) allows each of the six scans to form a self-closure model (see

The proposed approach was tested with datasets acquired by a FARO LS 880 laser scanner, which can produce panoramic reflectance images. Dataset 1 contains 24 million TLS points for an office environment. Datasets 2 and 3, respectively, contain 33 million and 20 million TLS points for the Aula Conference Centre of TU Delft (Delft University of Technology), The Netherlands. Dataset 4 contains 26 million TLS points for a building on the TU Delft campus.

The angular resolution selected for Datasets 1 and 2 in both the horizontal and the vertical directions is a quarter of the full resolution given by the manufacturer, while the angular resolution of Datasets 3 and 4 is one-fifth of the full resolution.

Our experiments were conducted to evaluate the proposed approach using panoramic reflectance images from both the indoor and the outdoor scene. Our algorithm was implemented in C++. All of the tests were performed on a PC with an Intel Pentium IV CPU running at 3 GHz with 1 GB RAM.

Various error metrics have been defined to measure the registration accuracies of point clouds (e.g., [

As mentioned in Section 2.2, the panoramic reflectance images are generated in terms of the angular coordinates and reflectance value of each 3D point of the FARO LS 880 point clouds. Usually, the panoramic reflectance images are used to generate a photorealistic texture of the scanned object or surface. Similar to a black and white image, the user does not need much experience to interpret these data. Example applications of image matching and texture mapping using panoramic reflectance images can be found in the literature, as in construction analysis [

We use the SIFT method to extract distinctive invariant features from the panoramic reflectance images and then identify the matches from the extracted features by looking for the descriptor vector with the closest Euclidean distance. The probability that a match is correct can be determined by taking the ratio of the distance from the closest neighbour to the distance of the second closest. This ratio was selected and used as a threshold for correspondence identification. The predefined threshold can be increased to select more matches or reduced to select only the most reliable match. We chose 0.8 as the threshold because experiments show that this can normally eliminate 90% of false matches [

The threshold of distance invariance verification was defined before outlier detection. _{ΔS} in terms of the angular resolution and angular and range accuracies listed in _{ΔS}, rather than 3σ_{ΔS}, as the threshold to remove outliers. As a result, only 110 correct correspondences were preserved, instead of the 655 correspondences shown in _{ΔS} to accept more correct correspondences because most of the matching for point pairs became more accurate with increasing numbers of iterations.

As shown in

Note: _{i} is the total number of TLS points in scan

The registration of Dataset 1 was implemented with these correct corresponding points. In addition, 24 pairs of corresponding points were selected as check points for the purpose of accuracy comparison (see

The registration function of Cyclone 5.5 using point-to-surface-based ICP was employed to implement the Dataset 1 registration. Compared to the results from our method, the accuracy is 2.4 mm; i.e., both of the accuracies are on the order of millimetres. The major difference is in the automation level and computational requirements. While the presented approach is completely automatic, the ICP requires a manual determination of initial corresponding point pairs. Moreover, the time cost of the whole process of ICP is 21 minutes, which is three times greater than that of our proposed method.

For comparison purposes, Datasets 3 and 4, which have lower angular resolutions, were also tested. The angular resolution selected for Datasets 3 and 4 is four-fifths of that of Datasets 1 and 2. Twenty-four pairs of corresponding points were selected as check points to measure the registration accuracy. The registration result listed in

ICP was also used to implement the outdoor dataset registration (i.e., Datasets 2–4). The registration accuracies are on the order of millimetres as well. However, the results of Datasets 2 and 4 were output after a few failures, as the selected corresponding point pairs failed to provide good initial parameters for registration. This result clearly shows that the implemented ICP approach is sensitive to the initial values.

Three bundle adjustment models were used in our study, named Model A (imposed by the circular self-closure constraint), Model B (imposed by the self-closure constraint of a single scan), and Model C (imposed by the last-to-first closure constraint of a single scan). All of the bundle adjustment models were implemented in C++ by the authors.

A total of 20 scans acquired by a FARO LS 880 laser scanner for the Aula Conference Centre of TU Delft were used to verify the three models. The method presented in Section 2 was used to implement pair-wise registration on these scans. Before the pair-wise registration, panoramic reflectance images were generated from the scans, and from each of them a key point database was extracted using SIFT. The corresponding scans for the pair-wise registration were automatically chosen by correspondence tracking among the key point databases.

Before global registration, the closure error from the last to the first scan was 5.1621 m, and the self-closure errors of the 20 scans ranged from 1.5460 to 6.0452 m, which suggests that although the residual error of each individual pair-wise registration was low, a propagation and an accumulation of the registration error are present such that large discrepancies exist between the last and first scans. The global registration was implemented based on the corresponding points and 20 sets of RTPs acquired from previous pair-wise registrations. The results of global registration are listed in

As shown in

The error accumulation before global registration is visualised by the twisting building shape (the left part of

In this paper, we have presented a fully automatic, iterative point cloud registration method based on panoramic reflectance imagery. This two-stage method consists of pair-wise and global registration. It makes use of the images to automatically find 2D correspondences and then generate 3D corresponding points from those correspondences. The 3D corresponding points can be used to compute RTPs, which are required for registering, using known pixel-to-point correspondences. The use of panoramic reflectance imagery, to which the pixel-to-point correspondence is inherent, realises the registration free of camera and camera-to-instrument calibrations and resolution differences between the images and the scans. Moreover, the panoramic reflectance imagery can be theoretically generated, in terms of the angular coordinate and reflectance value of each 3D point, directly from TLS scans. Rather than viewing correspondence tracking and the computation of the RTPs as a unilateral and ordinal procedure, we have integrated these two steps into an iterative process that allows for the optimisation of pair-wise registration, as it significantly improves the initial matches in three ways: (1) it removes outliers, (2) corrects false established correspondences, and (3) ensures a good distribution of 3D corresponding points in the overlapping regions. As for global registration, the self-closure constraint that forms is free of correspondence error. It can be extended to a circular model in order to minimise the self-closure errors across all scans through simultaneous least-squares adjustment.

Our tests with panoramic reflectance images show that the registration accuracy is of millimetre order and that the process is fully automatic and converges quickly. Since the registration uses only the corresponding points obtained by image matching, processing of the entire point clouds is avoided, which in turn significantly shortens the processing time. Furthermore, our method can compute RTPs in difficult cases, as for example when the normal vectors of the planes are parallel, which in turn may be problematic for primitive-based approaches.

The experiments with global registration with real TLS scan data prove that the closure error with respect to the adjustment model based on the last-to-first closure constraint consists of not only transformation error but also correspondence error. The results highlight that the bundle adjustment model imposed by the circular self-closure constraint may undoubtedly lead to the global optimised result.

Our future investigations will include efforts to improve the registration accuracy and the distance invariance error. Laser scanner accuracy (e.g., incident angle, edge effects) will be investigated to estimate the distance invariance error. In addition, the proposed method will be extended to an integrated change detection/registration approach for TLS applications in emergency response.

The comments of anonymous reviewers regarding this paper are gratefully acknowledged. This research was supported by the Natural Science Foundation of China under Grant No. 40801191 and the BSIK Project of The Netherlands “Virtual reality for urban planning and safety.” The authors appreciate the technical support that was kindly offered by Leica Geosystems.

Flowchart of our pair-wise registration strategy.

Procedure to construct the reflectance image from a 3D point cloud.

Four corresponding TLS points from two overlapping scans. The distance between the two points in Scan (a) should equal the distance between the two corresponding points in Scan (b), i.e., _{AB} = _{A′B′}.

Circular self-closure constraint. The solid rectangle S

Floor plans of the test data sets. (a) Dataset 1; the scanned scene is the inside of an office. (b) Dataset 2; the scanned scene is the area between two buildings. (c) Dataset 3; the scanned scene is the area in front of the Aula Conference Centre of TU Delft. (d) Dataset 4; the scanned scene is an area surrounded by two buildings. The FOVs of the test datasets are 360° in all cases.

Corresponding points on two 360° reflectance images, which are identified by nearest neighbour search in terms of the Euclidean distance from the descriptor vector of the key point in the fixed image to that in the moving image. The numbers in the image are the indices of corresponding point pairs.

Corresponding points preserved after pruning from Dataset 1.

Corresponding points on the two reflectance images acquired after the iterative corresponding process. The numbers in the image are indices of corresponding point pairs.

Check points used to measure registration accuracy. To ensure the proper distribution of check points, in each 60° segment of the 360° FOV four pairs of corresponding points were manually selected as check points in order to compare accuracies.

Pair-wise registered point clouds. (a) Dataset 1. (b) Dataset 2. (c) Dataset 3. (d) Dataset 4.

Scale image of the distances between 24 corresponding check points in the registered point cloud.

Corresponding points kept after pruning from Dataset 2.

Evenly distributed corresponding points on the building facade after the iterative corresponding process.

Closure errors of real scans after global registration.

Registered real scans

Description of the test point clouds.

Datasets 1 and 2 | 0.036° | ±0.009° | ±3 mm |

Datasets 3 and 4 | 0.045° | ±0.009° | ±3 mm |

Results of the pair-wise registration.

_{1}_{2} |
|||||||
---|---|---|---|---|---|---|---|

Dataset 1 | 11987424 | 2 | 0.0037 | 0.0053 | 0.0009 | 0.0035 | 5.0 |

11974976 | |||||||

Dataset 2 | 16726500 | 2 | 0.0044 | 0.0145 | 0.0018 | 0.0066 | 6.0 |

16713375 | |||||||

Dataset 3 | 10006564 | 3 | 0.0038 | 0.0185 | 0.0025 | 0.0078 | 5.5 |

10046768 | |||||||

Dataset 4 | 13266288 | 4 | 0.0039 | 0.0083 | 0.0010 | 0.0041 | 6.3 |

13259400 |

Results of the pair-wise registration of 20 scans.

| |||||||
---|---|---|---|---|---|---|---|

Scan 2-1 | 3 | 0.0058 | 5.4 | Scan 12-11 | 3 | 0.0073 | 5.3 |

Scan 3-2 | 3 | 0.0067 | 5.5 | Scan 13-12 | 5 | 0.0098 | 6.6 |

Scan 4-3 | 4 | 0.0079 | 5.9 | Scan 14-13 | 5 | 0.0085 | 6.7 |

Scan 5-4 | 5 | 0.0074 | 6.7 | Scan 15-14 | 3 | 0.0023 | 5.3 |

Scan 6-5 | 4 | 0.0036 | 6.1 | Scan 16-15 | 4 | 0.0075 | 6.1 |

Scan 7-6 | 4 | 0.0082 | 6.0 | Scan 17-16 | 4 | 0.0056 | 6.1 |

Scan 8-7 | 5 | 0.0098 | 6.6 | Scan 18-17 | 5 | 0.0069 | 6.8 |

Scan 9-8 | 5 | 0.0095 | 6.7 | Scan 19-18 | 4 | 0.0039 | 6.0 |

Scan 10-9 | 4 | 0.0037 | 6.2 | Scan 20-19 | 3 | 0.0015 | 5.4 |

Scan 11-10 | 4 | 0.0051 | 6.0 | Scan 1–20 | 5 | 0.0079 | 6.7 |

Results of global registration.

| |||
---|---|---|---|

A | 6 | 0.0340 | 1.98 |

B (Scan 20) | 4 | 0.0351 | 0.62 |

B (Scan 14) | 4 | 0.0352 | 0.62 |

C | 4 | 0.0381 | 0.63 |