*7.2. Evaluation Measurements*

For the evaluation of the mobile laser scanning system, five test runs were carried out in November 2018 and August 2019 using both a trolley and a van as carrier platform. Table 4 gives an overview.



**Figure 15.** Evaluation of the mobile laser scanning system: Van trajectory (**left**), trolley trajectory (**middle**), control points with BOTA8 targets (**right**). The yellow boxes indicate the test sites A and B for an area-based evaluation of the mobile laser scanning system (Source: Google Earth, modified).

During the test runs, a total distance of 13.5 km was covered. The trajectory was different for each run and lasted between 18 and 43 min. As an example, the cyan dots in Figure 15 indicate the trajectories of run 4 with a trolley and run 5 with a van (cf. Table 4). For each test run, the mobile laser scanning system was reinitialized. For a point-based evaluation, BOTA8 targets were adapted to the control points and scanned with the mobile laser scanning system. Each control point was scanned multiples times leading to a total number of 148 target scans that can be used for a point-based evaluation of the system. For an area-based evaluation using TLS reference point clouds, two selected test sites A and B in the evaluation environment were repeatedly scanned with the mobile laser scanning system (Figure 15). Test site A contains a face concrete wall with dimensions of 9.5 m × 50 m (cf. Figure 14), and test site B is a road passage with houses. The mobile scans of the test sites A and B were also performed during the five test runs in parallel to the scans of the control points.

#### *7.3. Point-Based Evaluation Using Control Points*

For the point-based evaluation, the coordinates of the control points were extracted from the point clouds of the mobile laser scanning system and compared to the reference values. That way, an evaluation of the entire processing chain including the calibration is possible. Figure 16 shows the histograms of the differences for east, north, and height component for all 148 target scans. An ideal Gaussian distribution is added to the histograms for better visual interpretation. In addition, the mean

values ∆*E*, ∆*N*, and ∆*H* as well as the empirical standard deviations *σ* are stated on top of the histograms. The histograms in Figure 16 have the approximate shape of a Gaussian distribution with absolute mean values of <2 mm and standard deviations of *σ* < 9 mm. The differences between the horizontal and vertical components is marginal. These results indicate a high accuracy of the mobile laser scanning system with a satisfactory extrinsic calibration. Moreover, these results also indicate the high quality of the control points in our evaluation environment.

**Figure 16.** Point-based evaluation of the mobile laser scanning system for east, north, and height component. A total number of 148 target scans were compared to the reference values. The mean values ∆*E*, ∆*N*, and ∆*H* as well as the standard deviations *σ* are stated on top of the histograms.

#### *7.4. Area-Based Evaluation Using TLS Reference Point Clouds*

For an area-based evaluation, the 3D point clouds of the mobile laser scanning system were compared with TLS reference point clouds. For this purpose, we made use of the M3C2 algorithm (Multiscale-Model-to-Model-Cloud) in the software CloudCompare. The M3C2 algorithm allows for a robust distance computation and comparison between two point clouds [85,86]. Both test sites A and B (cf. Figure 15) were scanned five times with the mobile laser scanning system. In order to demonstrate the potential of an area-based evaluation, we calculated the mobile point clouds with two different sets of calibration parameters, i.e., with an approximate calibration that was taken from the construction plan of the system as well as with the calibration that we determined in our plane-based calibration field. For the plane-based calibration, we utilized the mean values of all 98 calibration runs (cf. Figure 10) as these are the best available calibration parameters.

The results of the area-based evaluation are listed in Table 5, which shows the mean, median, standard deviation (STD), and root mean square error (RMS) of the deviations between the mobile point clouds and the TLS reference point clouds that were obtained from the M3C2 comparison. When using the approximate calibration, mean and median values between −18.5 mm and 8.7 mm are obtained (Table 5, left column). In contrast, the plane-based calibration considerably reduces the mean and median values to −6.3 mm to 4.0 mm (Table 5, right column). Thus, the unbiasedness of the point clouds is improved by the plane-based calibration. When using the approximate calibration, we obtain standard deviations and RMS values between 8.5 mm and 33.4 mm. For the plane-based calibration, these values are reduced to 4.1 mm to 9.9 mm. In general, when using the plane-based calibration, the results of the area-based evaluation are in good accordance with the results of the point-based evaluation based on the control points (cf. Figure 16).

Please note that the standard deviations and RMS values are generally smaller in test site A than in test site B. This might be caused by the fact that test site A is not sensitive to all types of errors. In test site A, for example, errors that cause a shift of the mobile point cloud that is parallel to the wall cannot be detected by the M3C2 point cloud comparison. Contrary to this, test site B is more adequate to evaluate the overall system performance, since test site B shows more variation w.r.t. surface position and orientation. This is a drawback of the area-based evaluation strategy using algorithms for the direct comparison of point clouds.


**Table 5.** Area-based evaluation of the mobile laser scanning system using calibration parameters from the construction plan (**left**) and from the plane-based calibration field (**right**). The table shows the mean, median, standard deviation (STD), and root mean square error (RMS) of an M3C2 comparison to a TLS reference point cloud. The rows marked with \* correspond to the histograms in Figure 17.

**Figure 17.** M3C2 comparison between point clouds of the mobile laser scanning system and TLS reference point clouds by using calibration parameters taken from the construction plan (A1, B1, **left column**) and from the plane-based calibration field (A2, B2, **right column**).

However, the area-based evaluation strategy also carries huge potential for an examination of mobile laser scanning systems. In order to demonstrate this, Figure 17 shows the scatter plots and histograms of the M3C2 point cloud comparison at both test sites A and B for the measurements that are marked with an asterisk (∗ ) in Table 5. A1 and B1 correspond to the approximate calibration taken from the construction plan, and A2 and B2 correspond to the plane-based calibration. At test site A, the mobile laser scanning system was moved parallel to a wall (Figure 17, top row). This wall is also shown in Figure 14. When using the approximate calibration parameters (A1), deviations of up to −40 mm are visible increasing from the bottom up. This pattern indicates a tilting error of the point cloud of the mobile laser scanning system. When using the plane-based calibration (A2), such deviations are vanished. The improvement is also visible in the related histograms. The distribution of A2 is considerably smaller and more unbiased in comparison to A1. When having a closer look to the calibration parameters, which are stated at the beginning of Table 5, we can see that there is a big difference of 0.1420◦ for the boresight angle *α*. This calibration parameter is mostly aligned with the roll axis of the mobile platform. Hence, the tilting error of the wall in A1 can probably be traced back to an erroneous boresight angle *α*. This example demonstrates that certain errors can be revealed by an area-based evaluation, when using an appropriate object in combination with a specific driving maneuver. This strategy was already proposed in previous publications [34,42–44]. However, it is difficult to unambiguously trace back deviations to its origin. For instance, the pattern in A1 could also be caused by an error in the roll angle. In order to clarify this, it would be conceivable to frequently repeat such measurements within a certain period of time making the assumptions that a rolling error is changing, but the calibration error is constant.

Test site A is not sensitive for all kinds of errors, e.g., height errors or errors parallel to the wall. Thus, we extended our investigations to test site B, where the mobile laser scanning system passes a heterogeneous scene with vertical, horizontal, and inclined surfaces (Figure 17, middle row). Test site B is more sensitive to different types of errors and, thus, more adequate to evaluate the mobile laser scanning system. Clearly, the erroneous boresight angle *α* can be detected at the rooftop (B1). When using the plane-based calibration (B2), the deviations have a mean value of −0.3 mm and a standard deviation of 7.6 mm. Due to the diversity of the scene, such results indicate a satisfactory quality of the calibration parameters. As indicated by the numbers in Table 5 and the histograms in Figure 17, the deviations are not perfectly Gaussian distributed and not completely unbiased, even after applying the plane-based calibration. This is due to the error characteristics of the mobile laser scanning system, which is mostly affected by systematic errors of the trajectory estimation. However, the 2D laser scanner and the M3C2 algorithm also contribute to the error budget.

#### **8. Conclusions and Outlook**

This paper presents the design and evaluation of a plane-based calibration field for determining the lever arm and boresight angles of a 2D laser scanner w.r.t. a GNSS/IMU unit on a mobile platform. In addition to this, the calibration field is sensitive for the estimation of the range finder offset of the 2D laser scanner. The calibration field was designed on the basis of a geodetic configuration analysis and Monte Carlo simulations. In this respect, the impact of random, systematic, and gross observation errors on the calibration was analyzed leading to a plane setup that provides accurate and controlled calibration parameters with a standard deviation of ≤1 ... 1.5 mm for the lever arm components and ≤0.005 ◦ for the boresight angles. This was empirically verified by calibration measurements with our own mobile laser scanning system. Using tools from geodetic configuration analysis to analyze both the accuracy and the controllability of the parameter estimation process in the context of calibrating mobile laser scanning systems has only been addressed to a limited extent so far and is the major scientific contribution of this work. The plane-based calibration field was realized outdoors being permanent, stable, weather-resistant, and cost-effective. The associated calibration procedure in the calibration field takes less than one minute and, thus, can be repeated frequently. In addition to the configuration analysis, repetitive calibrations also increase the controllability of the calibration parameters and allow for a realistic empirical quantification of their accuracy and stability.

In order to evaluate the mobile laser scanning system and the calibration, a dense network of control points and TLS reference point clouds of diverse building structures was installed. These facilities allow for a point-based as well as an area-based evaluation of the overall performance

of mobile laser scanning systems. We could demonstrate that the TLS reference point clouds can also partially be utilized to evaluate individual components of the system. Due to the complexity of the systems, however, the evaluation of individual components is challenging. Moreover, the rigorous modeling of the system accuracy by considering an error budget for each component is hardly feasible. Therefore, we argue that an empirical evaluation of the overall system, as conducted within this work, is the most effective strategy. According to our tests, both the point-based and the area-based evaluation indicate that the accuracy of our mobile laser scanning system can be specified with an RMS of <10 mm for the east, the north, and the height component, separately. This accuracy applies to the measuring conditions in our evaluation environment and might change for other fields of applications. Please note that the obtained evaluation results for our mobile laser scanning system are in good accordance with the results that were obtained in a prior case study on a motorway [59].

In the future, we plan to calibrate and evaluate systems other than the one used within this work using our facilities. Moreover, initial tests have been accomplished regarding the expansion of the plane-based calibration field to extrinsic camera calibration (e.g., [87–89]).

**Author Contributions:** E.H. and C.H. designed the plane-based calibration field; E.H., L.K., and H.K. installed the plane-based calibration field and the evaluation environment; E.H. performed the experiments; E.H. and L.K. analyzed the data; E.H. wrote the paper; All authors have read and agreed to the published version of the manuscript.

**Funding:** This research was partially funded by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) under Germany's Excellence Strategy - EXC 2070 – 390732324.

**Conflicts of Interest:** The authors declare no conflict of interest.

## **References**


© 2020 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).
