Performance Analysis of Mobile Laser Scanning Systems in Target Representation

Abstract

The technology of mobile laser scanning (MLS) has developed rapidly in recent years. This speedy development is evidenced by the emergence of a variety of MLS systems in commercial market and academic institutions. However, the producers tend to supply the specifications of the individual sensors in a generic sense, and this is not enough for guiding the choice of a MLS system for a specific application case. So far, the research efforts comparing the efficacy ranges of the existing MLS systems have been little reported. To fill this gap, this study examined the performance of three typical MLS systems (Riegl VMX-250, Roamer and Sensei) in terms of target representation. Retrievals of window areas and lighting pole radiuses served as representative cases, as these parameters correspond to the spatial scales from meter to centimeter. The evaluations showed that the VMX-250 with highest sampling density did best, and thus, it was preferred in the scenario of this study. If both the cost and efficacy were regarded, Roamer was a choice of compromise. Therefore, an application-oriented scheme was suggested for selecting MLS systems to acquire the desired performance.

1. Introduction

As a state-of-the-art technology for mapping and remote sensing, mobile laser scanning (MLS) can serve as an effective solution for surveying complex situations, such as urban environments and transportation corridors. In fact, the MLS concept was proposed a long time ago, but its thriving progress mainly spanned the last decade [1]. During that phase, a large number of mature MLS systems aimed at various applications were introduced into the market. These commercial MLS systems include the Riegl VMX-250, StreetMapper, Optech LYNX and Trimble Cougar, as summarized in [2]. There are also plenty of MLS systems developed by academic institutions, and the VLMS [3], Roamer [4], and Sensei [5] are the representative cases of this type.

Along with the MLS system developments, there were a lot of research efforts on assessing and improving their performance. The performance of an MLS system depends on numerous factors [6]. Many previous studies were dedicated to exploring these factors via, e.g., analysis of StreetMapper relative and absolute mapping accuracies [7], assessment of StreetMapper range accuracy [8], test of StreetMapper georeferencing accuracy [9], estimation of Optech LYNX planimetric precision [10], investigation of the effect of XP-1 scanner configuration and vehicle velocity on its scan profiles [11], calibration of Velodyne HDL-64E and analysis of its temporal stability [12–14], and simulation-based comparative analysis of MLS design plans [15]. Some non-commercial MLS systems and their error sources were also studied [16,17].

With the explosive emergence of diverse MLS systems and the enhancement of their functionality, the expected paradigm shift for MLS from mobile mapping to telegeoinformatics [18] has gradually become true. At the same time, the extension of MLS usage requires various tools for data processing and analyzing. Correspondingly, a large number of MLS-based information extraction methods have been developed for, e.g., geographical database update [19], vegetation bio-properties retrieval [20–25], transportation landmark surveying [26–30], historical remains documentation [31], roadside environment investigation [32–37], and rock-fall hazard monitoring [38]. All of these endeavors have effectively expanded the application ranges of MLS systems.

However, MLS producers tend to distribute the specifications of the individual sensors in a generic sense, and this is not enough for guiding the choice of a MLS system for a specific application case. This issue was induced from the fact that a high ratio of the application cases listed above seemed to be involve just a technical try by the researchers with an MLS system coming in handy. In this situation, the rules derived merely based on the utilized MLS system can only feature its own capability, and they are far from sufficient to guide the choice of a more appropriate MLS system for such an application. What is more, practice suggests that different applications highlight different technical requirements. Selection of an optimal MLS system for an application case involves a complexity of influence factors ranging from data collection to end-product generation [39], e.g., sampling density, ranging accuracy, echo georeferencing precision and laser divergence. Moreover, the criteria for assessing MLS systems are also diverse, including mapping efficiency, spatial resolution and object representation correctness and completeness, etc. Thus, an intuitive strategy is to select an MLS system with all of the influence factors comprehensively balanced in order to ensure the key criteria reached or the application-specific demands satisfied. To our best knowledge, few such studies on comparing MLS system performance have been conducted for the purpose of application optimization.

To fill this technical gap, this study was devized to compare the performance of the state-of-the-art MLS systems (Riegl VMX-250, Roamer and Sensei) in terms of target representation. The analyses were deployed based on the typical cases of object parameter extractions at different scales. With the relationships between the performance and the dominant impact factors exploited, it was attempted to derive the basic rules for guiding the selection of MLS systems.

2. Materials

2.1. Mobile Laser Scanning Systems

This study assumed the Riegl VMX-250, Roamer and Sensei (Figure 1) as the target MLS systems for performance analysis. These three MLS systems were chosen because they can characterize MLS system specifications at different performance levels. That is, the Riegl VMX-250, Roamer and Sensei perform with high, intermediate and low sampling densities and point geo-referencing accuracies respectively. The general specifications of the three MLS systems are listed in

Table 1. Specifications of the Riegl VMX-250, Roamer and Sensei mobile laser scanning (MLS) systems for fully-kinematic mapping.

	Riegl	Roamer	Sensei
Min range (m)	1.5	0.6	0.3
Max range (m)	200	76	100
Range for 10% reflectance (m)	75	25	50
Max sampling rate (points/second)	600,000	120,000	38,000
Ranging accuracy for 10% reflectance (cm)	1	2	4
Ground density at a same place (points/m2)	4,000	500	30

. There are also other factors that can decide their efficacies in target representation, and the dominant ones are described as follows.

The Riegl VMX-250 MLS system (Figure 1(a)) comprises two Riegl VQ-250 laser scanners, which are calibrated and integrated with a high-precision inertial measurement unit/global navigation satellite system (IMU/GNSS) pose/positioning module and a Riegl software package for data post-processing. The Riegl VQ-250 scanner has a laser beam divergence of 0.36 mrad, and can run high-performance pulsed ranging with high penetrability through obstructions (e.g., plants and fences). This penetration stems from the unique echo signal digitization technology of Riegl and its online pulse waveform processing. In addition, the VMX-250 has the capability of self-contained calibration of its individual subsystems. Its data acquisition and operator control is fulfilled through the compact control unit box, optimized for easy transportation. More details about the Riegl VMX-250 MLS system can be found in [40].

The Roamer MLS system (Figure 1(b)) adopts a FARO LS 880HE80 laser scanner for 3D mapping, with its spatial trajectory derived by the NovAtel SPAN (Synchronized Position Attitude Navigation) technology. FARO LS 880HE80 with beam divergence of 0.2 mrad utilizes a phase-difference ranging technique, facilitating Roamer in high-speed data collection. At the same time, the SPAN technology limits the standard deviation of Roamer ranging accuracies to two centimeters. Details on the Roamer MLS system calibration can be found in [41]. In addition, Roamer can perform panoramic scans when the mobile platform is held static (termed as the stop-and-go mapping mode hereafter), since FARO LS 880HE80 originally was designed as a tripod-based terrestrial laser scanner with a field-of-view of 320°× 360°. More details on the Roamer system can be found in [42].

The Sensei MLS system (Figure 1(c)) contains an Ibeo Lux laser scanner and a NovAtel SPAN-CPT tightly-coupled GPS/INS positioning module. The Ibeo Lux laser scanner can simultaneously receive laser echoes from four vertical scan layers, between which an angle of 0.8°exists. This scanner can also record up to three echoes per pulse. These two specialties enable Sensei to collect the backscattered points from the obscured portions of trees, which allows Sensei to somewhat overcome the issue of vegetation occlusion. The divergence of the laser beams from the Ibeo Lux laser scanner is 1.4 mrad horizontally and 14 mrad vertically with respect to the scanner body. More details of the Sensei system can be found in [5].

2.2. Test Site and Data Collection

The test data was collected at the Espoonlahti district in Southern Finland, a typical urban street environment [21]. The Roamer, Sensei and Riegl VMX-250 based campaigns for data collections were deployed on 10 June 2009, 6 May 2009 and 21 March 2010 respectively. The reference data was measured on 7 May 2009 using Roamer, but in its stop-and-go mode with the scan resolution set to 1/8 of the value specified in its kinematic collection. Namely, its maximum sampling rate is 960,000 hits per second. In view of object representation in the general sense, the feasibility of using the Roamer in its stop-and-go mode and manually data processing to supply the reference data has been recognized [43]. Then, targets with distances over 50 m from the MLS scanners were excluded, since few echoes of >50 m were recorded due to tree obstruction. The resulting datasets are all dense 3D point clouds, which are restrained by the corridor along the driving route. A plot is illustrated in Figure 2 to give an intuitive overview of the performance of the four scan modes. Note that the data output from only one VQ-250 laser scanner of the Riegl VMX-250 was examined in this study, and the rule for keeping its analysis consistent with Roamer and Sensei, both with only one scanner, was also adapted in the following procedure of scanning geometry analysis. The advantages of integrating two laser scanners, of course, will be discussed later. In addition, note that the Riegl data was collected along the road lane behind the lighting poles (closer to building), and this explained why the Riegl and Sensei data did not show tree trunk occlusion (Figure 2(a,c)) as Roamer did (Figure 2(b)). Overall, it can be recognized that although the point densities decline from the Riegl VMX-250, Roamer to Sensei, the geometries of objects such as tree crowns, buildings, lighting poles and even their details still can be distinguished in the lower density datasets.

2.3. Object Segmentation

As the subject of this study was explored from the perspective of target representation, the point clusters corresponding to the representative objects need to be segmented into four sets of point clouds for post-processing. In this study, object segmentation was implemented by using TerraScan (TerraSolid, Helsinki, Finland), which is commercial software effective for laser scanning data processing. TerraScan can easily manipulate millions of scattered laser points, since its procedures are all tweaked for optimum performance. It is also valid to display point clouds three dimensionally, define user-categorized point groups such as ground, vegetation, buildings and wires according to their 3D morphologies, and delete the unnecessary or erroneous points by drawing fences around them. Fence here is a segmentation tool of TerraScan, and a closed fence can define a local space like a cylinder, with its axis collinear with the line of sight. Thus, an object needs to be determined using three perpendicular closed fences. The fence-surrounded points can be isolated and resaved into a separate file for further refinement. By this means, target points can be reserved and non-target points can be removed. This interactive routine of fence-based segmentation can ensure the completeness of target-associated point clusters for performance analysis of the three MLS systems.

3. Methodologies

3.1. Performance Analysis Plan

Given that this is a study in target representation, the analysis of MLS performance starts by selecting the objects with representative significance. From the literature review, it can be learnt that the objects surveyed by MLS systems typically include vegetation, buildings and transportation landmarks. Therefore, two kinds of objects, i.e., window and lighting pole, are chosen for analysis. Further, for the convenience of numerical assessment, their metrics are extracted, i.e., window area and lighting pole cross-section radius respectively. They can feature the geometries of the targeted objects. By statistical analysis of these metrics, the performance of the three MLS systems on the representation of these two target classes can be evaluated.

The reasons for selecting these two categories of objects are: (1) Windows and pole cross-sections can characterize the spatial scales from meter to centimeter; (2) Lighting poles and windows represent two kinds of scenarios in MLS scanning, i.e., their morphologies are represented by the laser hits on their outer and inner boundaries; (3) The conclusions drawn based on windows and lighting poles can be applied to other objects. For example, the process of high-precision window corner reconstruction is beneficial for examining the errors in building corner representation. The procedures established for lighting pole reconstruction can enlighten the development of the algorithms for tree stem modeling. Hence, the selected study objects are representative choices to evaluate target representation.

3.2. Scanning Geometry Analysis

Different MLS systems even with the same sampling densities can represent the same target differently. One dominant reason is that different MLS systems generally present different scanning geometries, and their responses to target structures are consequently inconsistent. Specifically for the three MLS systems, their data collections are all by means of parallel scan profiles. In this scenario, the spatial distribution between any two adjacent scan profiles cannot be directly sensed. Therefore, the spacing between scan profiles can indicate the completeness of object reconstruction. When the scan profiles are vertical, it is unavoidable for an MLS system with large spacing to miss lighting poles in its laser echo recordings. However, if its scan profiles are obliquely set, multiple tilt cross-sections of a lighting pole can be acquired thanks to its intersections with the scan profiles. The basic features of that lighting pole can be retrieved from these cross-sections. This case shows that different scanning geometries indeed can render varied performance. Thus, pre-analysis of MLS scanning geometry can indicate whether or not a specific MLS system will reliably be able to represent a selected target type.

The scanning geometries for the Riegl VMX-250, Roamer and Sensei are manifested in Figure 3. Specifically for this test, the spacings between their adjacent scan profiles are on average 4 cm, 16 cm and 40 cm respectively. Riegl VMX-250 transmits laser beams in a plane with a full 360° field-of-view. Its scan profiles each have the angles δ and θ relative to the side plane and the bottom plane of the road corridor respectively. Roamer scans in a similar mode, except that its angle δ is set to 90°. So, its parallel scan lines on the road are perpendicular to the driving direction. Sensei emits laser pulses and receives echoes in a 180°field-of-view, and both of the angles δ and θ are set as 90°. Based on these detailed analyses of scanning geometries, it can be inferred that Sensei tends to miss lighting poles and vertical boundaries of windows, while Riegl VMX-250 and Roamer can represent them relatively well.

3.3. Analysis of Sampling Density Impact

In target representation based on point clouds, the phenomena of “feature variations” [44] generally exist at all spatial scales. That is, some “feature variations” cannot be accurately represented from the discrete sampling points. This can be illustrated by an example of target representation with different sampling densities in Figure 4, wherein the planimetric accuracies after plane fitting are evaluated by the correctness of plane extraction [8].

Obviously, low sampling densities are more likely to give incomplete plane representations. This conclusion can be evidenced by Figure 4(c), in which the building corners B and C are missed. With the increased point densities, the vertexes B and C are then captured in Figure 4(b) by intersecting two adjacent lines. Nonetheless, higher sampling densities still cannot ensure generating the absolutely-correct geometric models, as illustrated by corner A in Figure 4(b) and the fitted line BC in Figure 4(a). This example suggests that it is important to take the scale of the target into account when selecting an appropriate MLS system. When targets are diverse and simultaneously cover a variety of scales, higher sampling densities are a major advantage and a good choice to better target representation in a general sense.

3.4. Geometrical Modeling of Targets

The kernel procedure for MLS performance analysis is to tackle the two quantitative features—window area and pole radius. The calculation for each is a process of geometrical modeling of the segmented point clusters. For the two kinds of objects, different geometrical modeling methods are assumed.

Window modeling is based on the 2D Alpha shape method [45,46]. Before the step of Delaunay triangulation, the surrounding-window points attributed with 3D coordinates are projected into the window-associated planes using the method of principal component analysis [47]. Then, the resulting points attributed with 2D coordinates are geometrically modeled using the 2D alpha shape method. Generally, two boundaries can be sought for each window in this scenario due to the ring-like distribution of laser points. The inner boundaries of windows are extracted, and the window areas can be calculated.

Lighting poles are geometrically modeled using 3D cylinder fitting. The specific operations are based on a vertical section of points, which are segmented out at breast height (1.4 m) from each of the lighting poles. Based on the Gauss-Newton algorithm [48], the non-linear least-square cylinder fitting is fulfilled. After the iterative procedures of estimating the rotation and translation parameters for the best fit, the cylinder models with the minimum bias residuals are derived for all of the lighting poles. Then, the radiuses of the fitted cylinders can be retrieved to give the radiuses of the lighting pole cross-sections.

3.5. Performance Comparison

The comparisons of the three MLS systems are primarily based on their accuracies in the target representations. The accuracies are acquired by comparing their results with the reference ones, and their individual evaluations are based on statistics. For the two measured variables, the coefficient of determination R² and the levels of significance are explored to quantify the global stability of the three MLS systems in target representation. As well, the boxplots in terms of the biases from the reference data are displayed to characterize the global correctness of the MLS systems. In particular, for the process of lighting pole reconstruction, the standard deviations of the distances between the laser points and the fitted cylinder surfaces are also investigated by boxplots. This can infer the echo-georeferencing accuracies of the three MLS systems. To quantitatively characterize the performance of the MLS systems in target representation, the medium values of the associated boxes in the boxplots can be used as the indices. For example, if the medium values of the boxplots about the biases from the reference data are less than the threshold, the related MLS system can be applied.

4. Results

4.1. Window Area

The point clusters relative to 11 sample windows were extracted. After the principal component analysis and 2D Alpha shape calculation, the inside boundaries of the windows were obtained. They are illustrated by the reference-related window edges (Figure 5(a)) and the Roamer-acquired window edges (Figure 5(b)). With the window edges determined, the window areas were resolved. The values are displayed in Figure 5(c). The coefficients of determination R² between the MLS-derived areas and the reference values were calculated (see Figure 5(c)). The levels of significance are 0.16, 0.33 and 0.50 for the Riegal, Roamer and Sensei respectively.

A comparison of the R² values indicates that the data collected by the Riegl VMX-250 has the best correlation with the reference data in terms of window representation, while Sensei has the worst. For the area differences between the MLS-derived windows and the reference ones, their boxplots are shown in Figure 5(d). Overall, the Riegl VMX-250 models the windows most accurately, while the Sensei system performs the poorest.

4.2. Pole Radius

The point clusters for 12 sample lighting poles were segmented. The effects of pole representation based on the echoes are individually illustrated in Figure 6(a). After 3D cylinder fitting, the cross-section radiuses of these poles were retrieved and the values are displayed in Figure 6(b). The coefficients of determination R² between the MLS-derived radiuses and the reference values were calculated (listed in Figure 6(b) as well).

The R² values show that the data collected by the Riegl VMX-250 has the best correlation with the reference data, and Sensei has the poorest. The boxplots of the radius differences between the MLS-derived pole cross-sections and the reference ones are shown in Figure 6(c). The pole representations derived from the Riegl VMX-250 data collections best represent the real ones, while the Sensei does the worst. Moreover, the standard deviations of the distances between the laser points and the fitted cylinder surfaces were also explored using boxplots (Figure 6(d)). Here, the reference data collected by Roamer in its stop-and-go mode was also regarded. The results show that the Riegl VMX-250 performs with the lowest dispersion, even better than the statistical results from the reference data.

5. Discussions and Suggestions

In view of the illustrations of the collected data (Figure 2), the Riegl VMX-250 MLS system gives a representation of the test area (Figure 2(a)) as clear as that of the reference one (Figure 2(d)). If the second VQ-250 scanner is used, its overlapping data can render the representation more intact. Roamer suffers from the impact of obstruction, which is shown by the building façades represented in an incomplete way. However, if the parts without undergoing tree occlusion are examined, it can be realized that Roamer has the capability of generating a relatively-intact representation of the test area as well. As regards Sensei, its representation is not influenced by tree occlusions as seriously as Roamer, although its point clouds are acquired in a much lower density. The reasons are that this scanner can record up to three echoes per pulse, and this can help reduce tree foliage occlusion. It also can record four synchronous scan layers with an angle of 0.8°between them. These strengths can help Sensei to overcome the tree trunk and foliage occlusions to a large extent. In other words, Sensei has its own appropriate application domains.

For window representation, the results indicate that Roamer overestimates the window areas while Sensei underestimates the window areas (Figure 6(d)). This is triggered by two factors—MLS sampling density and the parameter setting of R in the 2D Alpha shape calculation. The spacing between two adjacent scan profiles in the Roamer data is larger than the opposite in the reference data. This tends to make the laser hits outside of the real window boundaries also extracted as the window edges (see Figure 6(b)). When it comes to Sensei, the spacing becomes far larger and the parameter R needs to be enlarged to find the points on the boundaries. In this way, fewer points on the edges were extracted. Hence, the window corners are reconstructed incompletely, and the Sensei-resulting window areas are less than the actual ones. If the a priori knowledge of windows with rectangular shapes is considered, the window areas retrieved from Roamer and Sensei point clouds can both be improved.

The lighting pole modeling shows that the Riegl VMX-250 performs with the lowest dispersion of the distances between its echoes and the fitted geometrical surfaces (Figure 6(d)). The narrow dispersion shows that the Riegl VMX-250 is generally more appropriate for centimeter-level object representation. The reason is that the performance of the IMU/GNSS module in the Riegl VMX-250 is better than the opposites in the other two, and the better ranging precision of the VQ-250 also plays a positive role. All of these comparisons suggest that compared to the other two MLS systems, the Riegl VMX-250 is the best choice for representation of the lighting poles in the study area. Note that Roamer also has low radius estimation errors (Figure 6(b)). In fact, if more restrictions like data-processing efficiency are considered, Roamer would be a good alternative means for pole reconstruction.

From the above analyses, it can be learnt that the elements impacting the performance of MLS systems are complicated. A quantitative criterion for deciding which MLS system is optimal for an application case is hard to reach. Let us take the task of surveying lighting poles in different projects for example. If all of the information of the lighting poles has already been stored in the transportation geographical information database, the selection of Sensei seems to be very appropriate for merely confirming their existence. If the locations of all of the lighting poles need to be re-checked, Roamer is suitable for supplying their positions. However, if a complete update (including e.g., deformations) is required, then the Riegl VMX-250 is the best one to provide the finest details. If the broken degrees of the lighting poles in all directions are demanded, multi-scan Roamer mapping in its stop-and-go mode is the right choice. In a global view, the “feature variations” of the targets need to be pre-examined in accordance of the explicit requirements of different applications, and the assumed MLS systems need to ensure that the dominant technical requirements are satisfied.

It is also apparent, that especially for urban environments, target reconstruction in fine scales still cannot be absolutely fulfilled using MLS, even with its positioning accuracy and sampling density largely improved. This issue becomes more serious with targets lying farther from the MLS scanners. As illustrated in Figure 4(a), modeling the short segment of BC with bias is unavoidable even if the sampling density is set high. These kinds of “feature variations” exist in the MLS measurements at all scales. To solve this problem, in addition to the common solution plan of enhancing sampling densities, combination of different MLS measurement patterns is also proposed. One potential plan is to fuse the MLS-mapped data and the relative data collected in its stop-and-go mode. Specifically, an MLS with moderate sampling density can be applied initially to extract the outlines of targets, and then, the local susceptible areas can be re-investigated by the same MLS with higher sampling density tuned in its stop-and-go mode, e.g., with the mapping vehicle stopped on the pavement. These methods can efficiently overcome the problem of MLS blind spots to some extent. In the three MLS systems, Roamer is the only one currently capable of deploying this function.

It is further worth mentioning that apart from the inherent parametric specifications, external factors like object obstruction can also influence the performance of MLS systems in target representation. The previous endeavors mostly focused on the “clean” scenarios, e.g., regular buildings along streets without trees disturbing laser beams in the façade-relevant surveys. On the contrary, the scenes with vegetation growing in front of buildings are often-encountered in practice. In this situation, Sensei with four synchronous scan profiles that have angles of 0.8°between them may work well for some objects. There are also many other occlusion cases, e.g., snow piled along the curbs in the Riegl RMX-250 data collected on 21 March 2010. In this case, the roadside curbs cannot be recognized, even though the Riegl VMX-250 has high positioning accuracy and sampling density. To solve this issue, other remote sensing techniques and more effective information extraction methodologies need to be incorporated or developed.

6. Conclusion

Based on the results and the discussions, the performance of the three MLS systems in terms of target representation can be basically ascertained. In the case of window area retrieval, the coefficients of determination R² between the MLS-derived areas and the reference ones are 0.36, 0.31, and 0.10 for the Riegl VMX-250, Roamer and Sensei respectively. As regards pole radius estimation, the R² values are correspondingly 0.42, 0.21, and 0.01. That is, in the scenario of this study, the Riegl VMX-250 presented the highest accuracies in the geometry extraction. If more criteria such as cost-effectiveness are involved, Roamer can be theoretically reckoned as an effective compromise for these two tasks. In consequence, for the goal of selecting the best MLS system for an application case, this study can give inferences. That is, high sampling density and high point geo-referencing accuracy are, in general, preferred for fine-scale target representation. When some special technical requirements are encountered, the MLS systems with their corresponding potentialities are recommended. In summary, an application-oriented scheme is suggested as the strategy for selecting MLS systems with the required performance.