Assessment of NavVis VLX and BLK2GO SLAM Scanner Accuracy for Outdoor and Indoor Surveying Tasks

Abstract

The Simultaneous Localization and Mapping (SLAM) scanner is an easy and portable Light Detection and Ranging (LiDAR) data acquisition device. Its main output is a 3D point cloud covering the scanned scene. Regarding the importance of accuracy in the survey domain, this paper aims to assess the accuracy of two SLAM scanners: the NavVis VLX and the BLK2GO scanner. This assessment is conducted for both outdoor and indoor environments. In this context, two types of reference data were used: the total station (TS) and the static scanner Z+F Imager 5016. To carry out the assessment, four comparisons were tested: cloud-to-cloud, cloud-to-mesh, mesh-to-mesh, and edge detection board assessment. However, the results of the assessments confirmed that the accuracy of indoor SLAM scanner measurements (5 mm) was greater than that of outdoor ones (between 10 mm and 60 mm). Moreover, the comparison of cloud-to-cloud provided the best accuracy regarding direct accuracy measurement without manipulations. Finally, based on the high accuracy, scanning speed, flexibility, and the accuracy differences between tested cases, it was confirmed that SLAM scanners are effective tools for data acquisition.

1. Introduction

Three-dimensional laser scanners are instruments that capture a vast number of observations with the use of Light Detection and Ranging (LiDAR). LiDAR is an active sensor that sends laser rays that hit surfaces in the environment and reflect (backscatter) to the sensor to determine the position of the surface in 3D space relative to the sensor to create a cloud of observations, commonly known as a point cloud [1]. Panoramic images are mainly used for the association of RGB information to 3D data. [2]. These scanners can be used to create digital models and maps for deformation monitoring, and for many other applications [3,4].

Using Global Navigation Satellite System (GNSS) and Inertial Measurement Unit (IMU) data, the scanner localization and orientation can be measured. The position and orientation data are stored as a function of the GNSS time. As the laser scanner data are also stored with timestamps generated from the received GNSS signal, the scanner, the GNSS, and IMU datasets can be synchronized. As a result, the scanner accuracy is primarily determined by the accuracy of the laser sensor, GNSS, and IMU [5]. Furthermore, the accuracy of the LiDAR reflectance can be influenced by factors such as angle of incidence, type of material, and environmental factors [6]. The angle of incidence can affect the amount of backscatter, with smoother surfaces having a higher reflectance being more sensitive (the greater the angle, the less backscatter) than rough surfaces, with the angle of incidence having little effect on the amount of backscatter [7]. Moreover, the surface type, whether the color, how rough/smooth, how shiny/matte, or how transparent, can influence the intensity of backscatter. The instruments used to make these types of acquisitions are mounted on tripods and are referred to as static terrestrial laser scanners (TLS).

A survey-grade static TLS is known to have high precision and can capture a vast number of observations. Each scan produces a point cloud that needs to be registered together with sufficient overlap to create one overall point cloud in post-processing software [8]. Due to these devices being static, some surveys, such as of indoor environments, can be difficult to plan and complete. Indoor environments with short-sight distances, multiple levels with stairs, and other obstacles can be difficult to navigate and can require multiple setups to ensure everything is acquired, which can increase the time of the survey both in the field and office. A solution to this problem may be the use of a Mobile Laser Scanner (MLS). MLS is a technology that can take observations while moving. For terrestrial use, these devices can be moved using a vehicle, trolley, or a person. These instruments have four main hardware components: optical sensors, LiDAR system, navigations/positioning sensors, and a control synchronization unit [9]. MLS offers advantages, such as reduced field and office work due to the absence of multiple setups. These devices are more capable of navigating indoor environments due to their portability and access to space-restricted environments.

This paper investigates man-portable Simultaneous Localization and Mapping (SLAM) scanners. This system uses LiDAR and panoramic cameras to capture observations. An IMU and SLAM are used to discern where the device is. SLAM has an algorithm software that defines the trajectory of the device and the three-dimensional reconstruction of the recorded sensors [9]. This algorithm allows the device to estimate its location concerning the location of landmarks. In this context, different applications of the SLAM system in different areas have been raised, such as indoor, outdoor, underwater, and airborne systems [10]. The main two sections of the SLAM system are the localization and the mapping, which may be separately considered. Of course, localization and mapping are particularly dependent on each other. The map is necessary for accurate localization, whereas the localization is vital for mapping. The position and the map in early classical SLAM algorithms were cooperatively estimated. Later, modern performances process the localization and mapping as two parallel duties, such as Parallel Tracking and Mapping (PTAM) [11]. Regarding the mapping, maps are required to assist the path planning, obstacle, and avoidance, and may represent the objective of scanning. Moreover, the accuracy of localization depends notably on the mapping accuracy [12]. On the other hand, the algorithms for resolving the positioning troubles may be categorized into the probabilistic and non-probabilistic methods. Probabilistic approaches are a typical classification. The probability algorithms are based on the Bayesian estimation approach, where mainly particle filters and Kalman filters are employed.

Acquisitions are taken at every epoch, and SLAM calculates the correlation between the instrument and the environment to update its trajectory equation in conjunction with the IMU’s measurements [13]. This enables the device to be used indoors, where the GNSS is ineffective. The accuracy of the SLAM can be affected by drift and tracking errors. Drift errors are the accumulation of minor measurement errors during a survey that can cause the scan data to drift. Furthermore, tracking errors are when the environment does not have enough discernible features and the SLAM algorithm gets confused about where it is [14], e.g., a long homogenous corridor. Both types of errors are amplified in areas with poor distribution of 3D features [15], outside environments, and areas where objects move [13]. These errors can be mitigated by selecting an appropriate trajectory. An appropriate trajectory is the use of loop closures and not rotating too quickly around corners. Loop closures are the scanning overlap of an area from a different perspective. The user conducts loops around an area, similar to a closed traverse in the surveying practice to ensure this overlap [16]. This is why it is recommended that a survey should start and finish at the same spot as one big loop closure [17]. Also, control points are another method used to reduce these types of errors by providing alignment adjustments during post-processing [16].

In the same context, speed is another factor that can influence an MLS survey, where the increase in speed can lead to a decrease in the number of observations [18] and can decrease the accuracy of acquisitions [19]. Kaartinen et al. [20] tested the elevation accuracy of the produced point cloud of a road with an MLS. The best results achieved were a planimetric accuracy of 25 mm over a range of 45 m. Similar accuracies were obtained from mapping topography changes and elevation of erosion changes of riverbanks by Vaaja et al. [21], where the obtained RMSE value was between 23 mm and 76 mm. This may suggest that the NavVis VLX could potentially be utilized in such applications, where the main intent is to map elevation changes or deformation, which could be useful for monitoring surveys. Also, Barba et al. [22] used SLAM in the documentation of Architectural, Engineering, and Construction (AEC) or built heritage to assess the accuracy differences between photogrammetry, TLS, and a Wearable Mobile Laser System (WMLS) based on a SLAM approach. For this purpose, a cloud-to-cloud comparison was employed.

One of the central challenges in SLAM is data association. This involves linking measurements captured by the sensor at different instances and locations with mapped features to determine if they originate from the same physical place [23]. Unlike photogrammetry, where distinguishing features in each dataset is comparatively straightforward, SLAM depends heavily on recognizing complex features, such as changes in wall orientation, window outlines, or building and furniture edges. Thus, the accuracy of this process is pivotal in generating a precise model.

Lauterbach et al. [24] and Chiappini et al. [25] investigated the use of MLS in an outdoor setting, in which the focus of Building Information Modeling (BIM) was the main goal. Results indicated that the SLAM scanner can be successfully utilized in an outdoor environment to extract building information of buildings. This is of importance, as this component needs to be considered when extracting features in a residential topographic survey. Fassi and Perfetti [26] and Vatandaşlar and Zeybek [27] investigated Digital Terrain Model (DTM) production from the derived data of SLAM in outdoor environments. Both studies indicated that with the capture point cloud data, a DTM was successfully produced to the required accuracies. The main differences were that these studies were applied within a natural landscape and did not have features that would be typically found in an urban residential area, such as buildings. Further research has indicated that from an application use, SLAM scanners in the form of backpack devices are mainly utilized in applications from cultural heritage preservation, environmental monitoring and forestry, and agriculture, which mainly covers the main purposes of 3D modeling, erosion change detection, and vegetation inventory [28]. A similar study was conducted by Yiğit et al. [29], where a comparative analysis of MLS and TLS for indoor mapping was conducted. Within the study, data were collected from a TLS to an MLS, and the total station (TS) of a large multistory building was compared.

At this point, as geometric feature extraction will be used in this research (see Section 3), and it is important to highlight the employed applied principles. Scanners provide 3D LiDAR point clouds, and the accuracy assessment operation may use data classification and geometric feature extraction. Nevertheless, automatic feature extraction is widely used to implement on point clouds to obtain more accurate results on the position of features in a point cloud dataset [30]. Many studies have investigated automatic feature extraction of the use of a RANdom SAmple Consensus (RANSAC) algorithm and Hough transform to obtain roof and building features at high levels of accuracy relative to baseline observations [31,32,33]. Another automatic feature extraction explored by Xiong and Wang [34] and Dey et al. [35] investigated the deep learning framework for BIM construction, finding that the stated method could efficiently and accurately segment the point cloud to list important features.

At this point, it is beneficial to highlight the novel contributions of this paper, as follows:

• Assessment of the efficiency of a SLAM scanner in the application of detailed surveys in an outdoor setting to address the envisaged challenges.
• A comparative study is conducted on residential areas, comparing different workflow methods and different reference datasets.
• Assessment of the capabilities of the SLAM scanner for indoor and outdoor data collection.
• Assessment of whether SLAM can achieve accuracies to a standard fit for conducting surveys.

2. Assessment Design and Used SLAM Scanners

There are many factors that can impact the point cloud accuracy, not only hardware performance but also software capability. When 3D reconstruction works with point cloud data, it is always concerned with how precise the model will be, how much power it will consume, and how fast the model will be built. This paper will not discuss the accuracy of the constructed model starting from LiDAR data, it will focus only on the obtained point cloud accuracy. The assessment in this paper consists of two sections: outdoor assessment and indoor assessment.

In outdoor assessment, first, the NavVis SLAM Scanner data were compared to reference datasets measured by the TS. Second, the BLK2GO SLAM Scanner point cloud was compared to reference TS datasets. Third, edge detection board assessment was carried out using BLK2GO data. Fourth, the TS dataset was used as a reference model for the BLK2GO SLAM Scanner point cloud. Finally, in indoor assessment, the NavVix SLAM Scanner data were compared to Z+F static scanner datasets (

Table 1. Comparison of products obtained from survey activities with different equipment.

	Outdoor				Indoor
	NavVis	TS	BLK2GO	TS	NavVix	Z+F
Number of points	10,340,402	365	54,668,797	573	10,827,312	117,776,549
Number of mesh triangles	11,280,274	691	13,637,817	1118	-	-
Acquisition time (minutes)	50	180	42	190	2.45	70
Post-processing time (minutes)	7		12		32

). These experiments allow an understanding of the behaviors of the two employed scanners and to compare the efficacity of the two kinds of reference data. Moreover, three types of data comparison were tested: cloud-to-cloud, cloud-to-mesh, and mesh-to-mesh. However, in indoor data assessment, based on the obtained results of the last experiments, only one test was realized, which was the NavVix SLAM data comparison to static scanner datasets.

For this study, the NavVis VLX as well as the BLK2GO, from Leica SLAM Scanners, were chosen to realize the SLAM accuracy assessment. The first one is a wearable device, whereas the second one is portable. At this stage, we present the used SLAM scanners before starting the assessment stage.

Finally, it is important to underline that in the context of error estimation, we used the absolute values of errors and the positive half of the Gaussian curve, because most discrepancies in our dataset were positive. This method aligns with the data distribution and focus of this paper. Additionally, using absolute errors is common in evaluating the accuracy with tools such as CloudCompare, as it measures the magnitude of discrepancies without considering directions. Similar metrics, such as Mean Absolute Error (MAE) and Mean Root Square Error (MRSE), are often used for this purpose. This approach effectively reflects overall accuracy and meets our evaluation goal.

2.1. NavVis VLX SLAM Scanner

The NavVis VLX 2nd-generation scanner is a wearable and walkable mobile scanner (Figure 1a). It is placed on the user’s shoulders, as seen in Figure 1a. The instrument has two LiDAR sensors, four 20-megapixel cameras, an IMU, and SLAM software (NavVis IVION). It offers both indoor and outdoor performance, with absolute accuracies of 6 mm at a 68% confidence level and 15 mm at a 95% confidence level in a dedicated test environment of 500 m² [14]. Considering the limited existing research with this device in a residential outdoor environment and the known efficiency of the device to capture point cloud datasets at high speeds while moving [14], this device was selected to be used in the data acquisition. NavVis recommends conducting a loop and walking at a normal pace when performing a survey [14] as it reduces SLAM/drift error by providing sufficient overlap.

The dual LiDAR sensors use the safest laser class (class 1 lasers), have a maximum range of 100 m, and are positioned with one vertical and the other horizontal, as seen in Figure 1a, which allows the device a greater field of view. Also, LiDAR sensors use a wavelength of either ~905 nm or ~1550 nm. For capturing images, the cameras have a 360-degree view in static mode. The IMU with the SLAM technology allows the instrument to discern where it is and where it has been. The NavVis can geo-reference marks either on the wall or ground with a hook on the bottom of the instrument. The device has a touch screen that provides information and shows an overview map of what the device has scanned [36].

2.2. BLK2GO SLAM Scanner

In 2019, Leica Geosystems introduced the BLK2GO (Figure 1b), a portable mobile mapping device capable of collecting 420,000 points per second, with a claimed accuracy of 10 mm within indoor environments [37]. Leica has introduced an app named BLK2GO Live. This app allows users to observe the mapped regions and areas awaiting data collection in real time, thereby enhancing efficiency, and reducing the requirements of mission planning.

The backbone of the BLK2GO system, the LiDAR, utilizes a class 1 laser to gather data. This LiDAR system captures a field of view covering 360° horizontally and 270° vertically, with effective ranges spanning from 0.5 m to 25 m [38]. The laser operates at a wavelength of 830 nm and is accompanied by a stated noise range of 3 mm [37]. The scanner comes with three cameras, each with a resolution of 4.8 megapixels. These cameras capture a panoramic image spanning 300° horizontally and 135° vertically. This imagery is then utilized to detect similarities among successive images, aiding in calculating the scanner’s movement throughout the environment. Through the utilization of the Grand SLAM Technology, Leica aims to address the challenges encountered in traditional SLAM. In conventional SLAM, which suffers from low demand nowadays, where images are not captured, the precision of maps and real-time instrument positioning relies on how effectively data from various locations can be linked together [39].

3. Outdoor Assessment

As shown in Section 2, four experiments of SLAM scanner measurements will be realized in this section. The next section will detail the first assessment, which was the NavVis SLAM Scanner and TS datasets.

3.1. NavVis SLAM Scanner and TS Datasets

The goal was to assess the efficiency of a wearable NavVis VLX MLS (Figure 1a) in the application of a residential contour and detail surveys in an outdoor setting to address the challenges mentioned. A comparative study was conducted on a residential house in Queensland, Australia (Figure 2a), comparing two different workflow methods. Initially, a detailed survey was executed under conventional methods with the use of a TS and GNSS unit for control.

The initial field task consisted of a control survey to establish control around the site on the local coordinate system of Map Grid of Australia (MGA) GDA2020 and on Australian Height Datum (AHD). This was completed through a standard traverse from two known permanent survey marks located near the site combined with the placement of three control marks around the front kerbing and one at the rear of the property, in the form of retro targets. Four control points were established, as previous studies have indicated that a minimum of three control points is required for effective georeferencing of point cloud datasets. Once the control survey was completed, the topographic survey was completed with the conventional approach with a TS. At this point, the time to complete the survey in conventional methods was recorded as three hours. In this context, Magnet office was the software utilized to reduce the TS observations to align the coordinates established from the control survey (Figure 2c) as well as a height reduction to make all observed points relative to AHD.

Subsequently, that same area was surveyed using the NavVis VLX (Figure 2b), where the measured point cloud was georeferenced using NavVis Ivion software. Both datasets, derived from the TS and the point cloud, were superimposed together after generating two DTMs, which will be used for NavVis VLX data assessment (see Section 3.1.2). The horizontal and positional accuracies of the produced point cloud from the NavVis VLX were compared to the baseline of the TS with three different data analysis techniques, consisting of a cloud-to-cloud, cloud-to-mesh, and mesh-to-mesh comparison. For this purpose, CloudCompare 2.12.4 software was used to carry out this comparison. To analyze the horizontal position (x and y axis) of the point cloud regarding the baseline TS data, a list of features, such as retaining walls, service utilities, such as power poles and water meters, fences, concrete driveways, slight sloping elevation, and also a house located on the site, were compared with computation of the Root Mean Square Error (RMSE). These features were labeled manually on the LiDAR point cloud, whereas they were assigned during a survey by TS.

3.1.1. Data Assessments

The site survey was conducted using a TS to provide reference data. The instrument used to complete the TS survey was a Leica TS16. This instrument has a stated angular accuracy of 3 s of ark and a distance accuracy of 2 mm + 2 ppm when using the reflectorless feature [40].

To compare the LiDAR point cloud with the survey data, the point cloud was filtered by the removal of duplicate points at a tolerance of 5 mm. This tolerance value was considered regarding the mean laser spot radius value (3 mm) added to the tolerance amount. The purpose of the removal of duplicate points in the point cloud is to keep the useful points and make them more manageable to work with, and reduce the noise in the point cloud, which in turn will enhance the visualization of the cloud [41]. This has also been proven to speed up processing times due to the reduction in point cloud size [42]. After this filtering, the point cloud was compared to the points from the exported Triangulated Irregular Network (TIN) using an Iterative Closest Point (ICP) algorithm [5].

In the second test, a cloud-to-mesh comparison was utilized for data analysis, in which the LiDAR point cloud was classified into the ground and off-ground points. Thereafter, a mesh generated by Delaunay 2.5D [43] was created of the filtered ground points (Figure 3), which was compared to the points exported from the TIN produced from the TS point clouds. The third form of data analysis involved a mesh-to-mesh comparison of the scan data compared to the TS point clouds. A mesh was produced, also using Delaunay 2.5D, on these two datasets, which were then compared to one another. This chosen test was utilized because it has been shown to ensure the triangles are well shaped and avoids creating any irregular or elongated triangles in the mesh, which can result in some inaccuracy in interpolation [43]. At this stage, it is important to note that the last three comparisons have been proven to be the main forms of comparison utilized in studies on point cloud comparison [44].

The last conducted assessment was the vertical and horizontal positional comparison between the two datasets of extracted geometric features located on the site. The main points of interest selected for comparison consisted of features that would not have much deviation in X and Y coordinates, for example, the building corners of the house located on the site, and for the vertical component, features such as finished levels of the buildings and decks located on the site. Comparisons were computed between the differences in X, Y, and Z coordinates. To investigate the standard deviation of the residuals computed, the RMSE was calculated for these data (Equation (1)):

$$RMSE=\sqrt{{\displaystyle \frac{\sum V^2}{n}}}$$

(1)

where v is the deviation, and n is the sample size.

3.1.2. Results and Discussion

In the context of the LiDAR data assessment, even though the aim of surveying or laser scanning a given structure is generally to extract sections, planes, or 3D models, the direct comparison of the raw data (the point clouds) can provide more faithful results regarding avoiding modeling errors [44]. This is why Antova [44] suggested three kinds of data analysis for the vertical or Z coordinate comparison between the TS point clouds and the NavVis VLX scanner data, which are a cloud-to-cloud, cloud-to-mesh, and mesh-to-mesh comparison. To provide a statistical analysis of the two datasets, the Gaussian distribution, otherwise known as the normal distribution, was utilized, as this measure provided information on how far the data were from the mean, as well as the standard deviation. The standard deviation from the three conditions tested is the main important value derived from the analysis because it represents the average distance of data points from the mean as well as quantifies the spread of the data. The Gaussian distribution analysis was calculated at three confidence levels: 68%, 95%, and 100 %, i.e., the 68% confidence level is where approximately 68% of the data falls within one standard deviation of the mean. Figure 4a shows the histogram of the cloud-to-cloud comparison, with a mean of 35 mm and a standard deviation of 22 mm. Figure 4b shows the histogram of the cloud-to-mesh comparison, with a mean of 110 mm and a standard deviation of 180 mm. Finally, Figure 4c presents the histogram of the mesh-to-mesh comparison, with a mean of 27 mm and a standard deviation of 65 mm.

Concerning the feature comparison, the main selected features from the site are presented in

Table 2. Height differences between the MLS data and TS for measured features.

Feature	TS Z (m)	MLS Z (m)	Δ Z (mm)
Water meter	37.449	37.453	−4
Storm water pit	37.327	37.312	15
FFL front deck	40.674	40.678	−4
FFL back deck	40.66	40.674	−14
Roof heights	44.673	44.698	−25
			RMSE (mm)
	Confidence level	100%	15
		95%	14
		68%	10

and

Table 3. X and Y coordinate differences between the MLS data and TS for measured features.

	TS		MLS
Feature	X (m)	Y (m)	X (m)	Y (m)	Δ X (mm)	Δ Y (mm)
Front deck corner	501,452.516	6,970,295.245	501,452.520	6,970,295.22	−4	25
Building corner	501,452.131	6,970,292.727	501,452.100	6,970,292.789	31	−62
Building corner	501,456.724	6,970,291.919	501,456.710	6,970,291.920	14	−1
Building corner	501,456.977	6,970,293.345	501,456.960	6,970,293.290	17	55
Building corner	501,466.178	6,970,291.649	501,466.19	6,970,291.55	−12	99
Building corner	501,464.789	6,970,283.685	501,464.85	6,970,283.68	−61	5
Building corner	501,451.046	6,970,286.189	501,451.01	6,970,286.23	36	−41
Back deck corner	501,450.14	6,970,281.271	501,450.13	6,970,281.27	10	1
Back deck corner	501,456.101	6,970,280.18	501,456.09	6,970,280.19	11	−10
Electrical pole	501,443.919	6,970,279.769	501,443.856	6,970,279.676	63	93
Electrical pole	501,447.283	6,970,301.2	501,447.289	6,970,301.224	−6	−24
Street sign	501,446.139	6,970,293.845	501,446.12	6,970,293.684	19	161
Street sign	501,453.381	6,970,306.063	501,453.405	6,970,306.111	−24	−48
Gully pit corner	501,458.671	6,970,307.661	501,458.688	6,970,307.709	−17	−48
Gully pit corner	501,459.563	6,970,307.532	501,459.573	6,970,307.53	−10	2
					RMSE_X (mm)	RMSE_Y (mm)
			Confidence level	100%	28	63
				95%	27	60
				68%	19	43

, where the Z coordinate is compared to features measured on the site, such as finished floor levels on the front and back deck. This was carried out by calculating the elevation difference between the baseline measurements from the TS compared to the LiDAR point cloud. The RMSE value was 15 mm (Table 2). For the horizontal positional accuracies, the selected features in the two datasets are displayed in Table 3. For this comparison, the X and Y coordinates of the MLS were compared to the baseline of the TS observations. The calculated RMSE for the X coordinate was 28 mm, and 63 mm for the Y coordinate. The greatest difference in the horizontal positions was found for the electrical pole feature, with a difference of 63 mm in X and 93 mm in Y.

As presented in Figure 4, from the histograms of the Gaussian distribution of the three comparison tests, the cloud-to-cloud comparison produced the best results based on the lowest standard deviation score of 22 mm. The main reason for testing the three different comparative methods was to define which method would be most accurate for comparing the DTM extracted points from the TS observations as the baseline. The cloud-to-cloud comparison primarily involved calculating distances between similar points of two point cloud datasets. It was the most efficient and simple to utilize as it does not require another model to be built, such as a Digital Elevation Model (DEM), meaning less manipulation and filtering of the point cloud dataset [45]. Indeed, it calculates the nearest-neighbor distance, which is the distance between two points in the cloud, which is then compared to the two similar points in the other observed cloud, and then their Euclidean distance is calculated [46]. In this paper, the DTM model from the TS data was utilized for comparison, as it is one of the main requirements typically provided for design purposes, hence why this dataset was compared to the point cloud. The proven result of a standard deviation of 22 mm suggests that the DTM would be able to be produced quite accurately with the scanned point cloud data from the NavVis VLX, with the tested feature extraction of point picking in the cloud of ground points of features, such as surface levels, footpaths, and the top of bottom of walls, to form the ground classification. When comparing this standard deviation result for elevation to other studies with the use of an MLS, it was evident that the NavVis performed quite well.

Concerning the horizontal accuracy, the calculated RMSE for the X and Y coordinates identified some evident variances from the baseline of the TS observations. Certain features compared on a horizontal plane differed in tolerance, which would make it questionable to be utilizing these data to produce an accurate topographic map. This is the case, as a certain tolerance must be met when the horizontal position of features is crucial to define in the residential contour and detailed survey, as they are typically relative to cadastral boundaries. The results showed a combined RMSE of 30 mm in the abscissa and 60 mm in the ordinate, and when considering these values, it can indicate that these tolerances would be not suitable for residential topographic surveys. These results do, however, fall upon the method utilized in this study of manual feature extraction from point picking in the cloud and largely rely upon the interpretation of the point cloud data. This provides a suggestion for implementing different methods of feature extraction on the point cloud to investigate if the accuracies are improved on the horizontal comparison. To conduct this experimentation again, the use of automatic feature extraction algorithms, as mentioned in Section 1, could be advantageous to obtain more accurate points that represent certain features on the site, for example, building corners of the house, which then could be compared to baseline observations.

The NavVis XLV produced a point cloud that was vertically accurate after georeferencing, with a standard deviation of 22 mm from the computed cloud-to-cloud comparison. However, the horizontal accuracies, i.e., X and Y coordinates, did not seem to be as accurate, which may be due to an inaccurate feature extraction method and lack of control points. This suggests that further investigation should be realized on feature extraction, and the enhanced algorithms can be used to improve feature extraction. Finally, from an efficiency and flexibility standpoint, the NavVis VLX produced great results, which was reflected in the time to scan the site, which was 50 min, compared to the 3 h it took with the TS.

3.2. NavVix SLAM Data and Static Scanner Dataset

In this section, a new SLAM scanner assessment of the outdoor area (Figure 5a) was carried out using a static scanner to measure a reference point cloud. The equipment used to measure the point clouds included the Z+F 5016 static TLS (Figure 5b) and the NavVix VLX 2 SLAM Scanner (Figure 1a,b). The NavVix VLX 2 SLAM scanner was presented in Section 2. Concerning Z+F 5016 static TLS, the field of view is 360° × 320°, and the maximum laser range of class 1 “eye-safe” equals 360 m, in addition to a spot diameter of ~3.5 mm @ 1 m/~0.3 mrad. Also, it is supported by a full-panorama (80 MPixel) camera to calculate the point cloud RGB colors.

At this stage, it is important to note that the scanning accuracy is mainly affected by the laser spot diameter, which can be changed according to the laser range. This is why the LiDAR point accuracy was located around ±4 mm. In this context, the TLS data were filtered to remove erroneous points, such as duplicate points and point cloud noise. For this purpose, two filters were applied. The first filter was the Statistical Outlier Removal (SOR), which eliminates the noise points (Rusu et al., 2007 [47]). This filter supposes that the distance between one LiDAR point and its neighborhood is normally distributed. The second filter removes the duplicate points at a given distance tolerance by testing each point neighborhood [3]. Thereafter, depending on the 50% overlap between neighboring stations, as well as the used target points, a registration process between neighboring station clouds was successfully completed, with a connection standard deviation of 3.4 mm and a target standard deviation of 0.3 mm.

Discussion

To complete the comparison of SLAM and static datasets, a cloud-to-cloud comparison was applied (Figure 6). The outdoor data accuracy achieved was 3 mm at a 68% confidence level, with a maximum distance from the TLS of 6 mm at 68%. In comparison to the stated relative accuracy of 8 mm at 68%, and the white paper outdoor building façade scenario accuracy of 6 mm at 68%, the assessment results achieved better accuracy of the outdoor area. The resulting point cloud seen in Figure 6a displays higher distances in red, located on the sandstone wall adjacent to the water feature and front entrance. Most of the data can be seen to be within the range of 0 mm and 5 mm, colored in blue. Promising results were obtained from the comparison with the terrestrial laser scanner, which undoubtedly had greater accuracy, confirming the efficiency of the SLAM scanner regarding the static scanner within the outdoor areas.

3.3. BLK2GO SLAM Scanner Accuracy Assessment

It is necessary for the data measurement to grasp the potential levels of accuracy attainable from instruments, such as the BLK2GO. This section aims to assist in evaluating the accuracy of this instrument and determining whether the instrument can deliver satisfactory levels of accuracy in survey works. It includes the use of an edge detection board as well as a comparison with TS data to assess the SLAM LiDAR accuracy. At this stage, it is important to underline that some accuracy assessments in this section will apply different strategies from those applied in Section 3.1 that can present a richness of data assessment approaches.

3.3.1. Edge Detection Board Assessment

Without any human intervention, the determination of object edges, and subsequently the accuracy of measured objects, relies on the density of the point grid at specific distances. As highlighted in Section 1, the precise identification of objects is paramount in the SLAM process. Though this test could be influenced by the type of materials of the object surveyed, this paper focused on edge detection regardless of the material type, and this topic will be deeply studied in future research.

Thus, accurate data association plays a critical role in combining datasets and, ultimately, generating a precise model. Typically, scanning systems calculate measurements by computing the angular and distance differences between successive points. Consequently, targets situated closer to the scanning unit will exhibit a denser concentration of closely spaced points, while targets positioned farther away will display a lower point density [48]. The test apparatus consisted of a 12 mm ply board, with eight Medium-Density Fiberboard (MDF) panels of different thicknesses attached to it (Figure 7). A steel machinist ruler and digital calipers were used to directly confirm the dimensions of the MDF pieces attached to the board. Once all the measurements were confirmed, multiple scans of the board were executed. This approach enabled the assessment of distance measurements and the determination of object edges concerning the sensor’s proximity.

The scans were taken at distances of 2 m, 5 m, 7 m, 10 m, 15 m, and 25 m, each lasting 40 s. Running the scan for 40 s provided sufficient time for the instrument to establish its position and ensure a comprehensive saturation of points on the board. To assess the collected data, the horizontal and vertical values of each token were measured at varying distances. To assess the depth measurement of the tokens, sections of the edge detection board were considered, allowing for an assessment of the depth measurements when using a top view. Figure 8 shows the results of the edge detection board scan, where the tokens were 120 mm × 105 mm and varied in depth from 3 mm to 24 mm.

3.3.2. Discussion

From Figure 8, the instrument consistently delivered high-quality results up to a distance of 7 m. However, the mean standard deviation in the x-axis was 4.7 mm, and in the y-axis, 4.8 mm. The data obtained from the edge detection board played a crucial role in evaluating the accuracy of the LiDAR unit and determining the ideal proximity to a subject for obtaining reliable results. The calculated distance throughout the scans provided accurate results. The increase in the standard deviation aligns with the findings of Harrap and Lato [48].

From

Table 4. Edge detection board depth results.

Scan Distances to Target (m)	Confirmed Depth (mm)	3	6	9	12	15	18	21	24
2	Scanned Depth (mm)	7	14	14	17	20	23	32	31
	Delta (mm)	4	8	5	5	5	5	11	7
5	Scanned Depth (mm)	5	8	11	16	20	25	28	29
	Delta (mm)	2	2	2	4	5	7	7	5
7	Scanned Depth (mm)	8	12	13	20	20	20	27	29
	Delta (mm)	5	6	4	8	5	2	6	5
10	Scanned Depth (mm)	-	-	20	17	15	23	25	30
	Delta (mm)	-	-	14	8	3	8	7	9

, the results from the Z-axis were far less impressive, with errors of up to 14 mm recorded in the 10 m scan. However, the data obtained from these scans were uninterpretable at distances beyond 10 m. Table 4 shows the differences in definition between the scans taken at 2 m to 10 m of cross-section. The weaker outcomes of in-depth measurements primarily stemmed from the scanner’s stationary position during the scan. Since the scanner was arranged perpendicular to the board, it resulted in the creation of small shadows by the tokens. To address this problem and potentially enhance the results, the instrument should be moved laterally and vertically while maintaining the desired offset. Implementing these adjustments would allow the instrument to perform more effectively by enabling better scanning of the corners.

3.3.3. TS Datasets as a Reference

A survey of the site was conducted using TS (Leica TS16) to provide reference data. The TS used had a stated angular accuracy of 3 s of the ark and a distance accuracy of 2 mm + 2 ppm when using the reflectorless feature. A traverse was executed around the site using the TS, following the same trajectory as the data collected from the scanner. Data were collected at three different points: along the base of the walls, halfway up the walls, and at the intersections of the ceiling and walls, to enable a three-dimensional model to be constructed. From these data, approximately 550 positions were recorded for comparison to the scan data. The data collected from the TS were imported into GeoCivil for reduction. Within this program, the data were assessed for outliers and the strings were completed. Once the data were assessed, they were moved to the same coordinate system as the data collected from the scanner. Finally, a data exchange file (dxf) was produced to enable import into CloudCompare. This software was also used to calculate areas of the leases from the TS data.

After filtering the scan data (Section 3.2), the cloud-to-cloud comparison strategy was then employed to compare the surveys from the subject area. This analysis visually highlighted any disparities that emerged between the surveys and provided a statistical summary of the quality of the scan data. Subsequently, a comprehensive investigation was undertaken to ascertain the underlying causes of these discrepancies and determine their magnitude. Figure 9 shows the results of the comparison between the data collected from the TS and the BLK2GO. It can be noted that the data obtained from the BLK2GO yielded a mean value of 21 mm, with a standard deviation of 12 mm. Although it fell slightly short of the manufacturer’s stated accuracy claims, the difference was not dramatic. The mean and standard deviation could potentially be improved by removing less of the point cloud.

4. Indoor Assessment

This section focuses on the indoor assessment of the SLAM scanner’s accuracy. This choice was adopted regarding the wide employment of SLAM scanners for data acquisition in indoor projects. In this context, the NavVix VLX 2 scanner data were compared with reference datasets measured by the Z+F static scanner.

To assess the NavVix scanner’s accuracy, a point cloud measured by the Z+F Imager 5016 scanner was used as reference data because this scanner is much more accurate than NavVix [49]. To meet the aim of this research, the Ipswich City Council Art Gallery in Queensland, Australia, was chosen to conduct the scan experiment. The test area was set, and data were collected by the TLS and the MLS, performing three separate scans: normal walking speed in a loop closure (loop normal; Figure 10a), fast walking speed in a loop closure (loop fast), and normal walking speed in a straight path with no loop closure (straight normal; Figure 10b) [14]. The results of the loop normal were used to provide the best accuracy against the TLS data and to compare the other MLS scans. The second scan (loop fast) was chosen to test the effects of speed on the device by walking at a fast pace. The third survey (straight normal) was chosen to test the effects of not conducting a loop in the scan. The TLS point cloud was used as the reference model (ground truth) for the MLS point clouds for carrying out a cloud-to-cloud distance comparison [50].

Discussion

The analysis was conducted to assess the relative accuracy of each of the MLS scans compared to the TLS scan. Further analysis was conducted to assess if speed and performing a loop closure with the MLS affected its relative accuracy. Speed was calculated using the metadata of the MLS scans, and the relative accuracy was calculated. To analyze each of the MLS point clouds, the MLS point clouds were compared to the TLS point clouds using a cloud-to-cloud comparison with a quadratic local model calculated for the TLS to yield a better approximation of the test site’s surface. Thereafter, a distribution-fitting statistical test was realized to conduct a Gaussian test. Finally, these data were used to determine the mean, standard deviation, and RMS between the two point clouds, as shown in Figure 11.

Despite the high similarity between the obtained accuracies, it can be noted that each point cloud aligned better with the loop normal, producing a result approximately 5 mm better. Comparing all point cloud results showed that the loop normal point cloud produced the most accurate results, as seen in Figure 11, with a mean of 4 mm, standard deviation at 2σ (95% C.I.) of 8 mm, maximum deviation at 2σ (95% C.I.) of 12 mm, and an RMSE of 5 mm. This is shown in Figure 11, with loop normal producing the lowest mean, standard deviation, and RMSE. This point cloud was followed by straight normal and then loop fast in terms of accuracy. The Gaussian histograms for each scan showed that they had a normal distribution with a high peak, and each was skewed to the left on the lower end of the distance. This indicates that some of the higher distance values caused the mean to shift to the right of where most of the values lay. The high peak of the histograms also showed that most of the cloud-to-cloud distances were within a small range. The maximum deviation for all points clouds at 2σ (95% C.I.) was 16 mm.

These results showed a significant difference in the number of points observed between the loop normal and loop fast (10,371,499 points). The loop normal also produced a more accurate result, with a maximum at 95% C.I. of 12 mm. When comparing the number of points observed, the TLS point cloud had over 100 million more points observed than the loop normal, which had the highest number of points for an MLS survey. The TLS survey took approximately 70 min from start to finish, whereas the loop normal scan took 2.45 min.

5. Conclusions

This comprehensive assessment of the NavVis VLX and BLK2GO SLAM scanners yielded significant insights into their performance across various conditions and environments. The findings demonstrated the influence of multiple factors on the accuracy and capabilities of these scanners, highlighting the critical roles of speed, flexibility, surface type, and scanning conditions. Speed and flexibility were found to have a more substantial impact on accuracy than loop closures, suggesting that prioritizing these factors can lead to better outcomes when using SLAM scanners in survey work. Additionally, the ease with which the instrument can scan different surfaces, especially those with high angles of incidence, played a major role. Surfaces that are difficult to scan may not provide the necessary backscatter to produce an accurate point cloud. The time required for a SLAM scanner to collect data in the field was significantly shorter compared to traditional methods, such as total station (TS) and terrestrial laser scanner (TLS). Despite this efficiency, TLS and TS are better suited for high-accuracy scans, whereas SLAM scanners are more appropriate for scenarios where lower accuracy is acceptable in exchange for speed and flexibility.

Indoor environments showed higher accuracy levels, with SLAM scanners achieving up to 5 mm accuracy. For outdoor environments, accuracy varied between 10 mm and 60 mm depending on the conditions and factors involved. The cloud-to-cloud comparison method provided the most direct and accurate measurement of point cloud data, with the loop normal scan achieving a mean accuracy of 4 mm, standard deviation of 8 mm, and RMSE of 5 mm. The maximum deviation observed was 16 mm at the 95% confidence interval (C.I.). The NavVis VLX scanner exhibited robust performance, with an absolute accuracy of 6 mm at the 68% confidence level and 15 mm at the 95% confidence level in controlled environments. The BLK2GO scanner, although portable, achieved a stated accuracy of 10 mm within indoor environments.

The results indicated that SLAM scanners hold significant promise for various sectors, particularly in indoor environments and applications, such as underground mining, where regular, frequent surveys can detect movements early. However, for tasks requiring high precision, such as cadastral surveys, TLS and TS remain more suitable. Further research should focus on assessing the impact of environmental factors, such as elevation changes, on SLAM scanner performance, conducting real-world surveys, such as building final plans or digital twins to evaluate the practical applicability and efficiency gains of SLAM scanners compared to traditional methods, and enhancing feature extraction algorithms to improve horizontal accuracy, particularly in residential topographic surveys. In conclusion, while SLAM scanners offer significant advantages in terms of speed and flexibility, their accuracy must be carefully considered relative to the specific requirements of the survey task. These findings provide a foundation for understanding the strengths and limitations of SLAM technology and guide future developments and applications in the surveying field.