A Framework of Wearable Sensor-System Development for Urban 3D Modeling

Abstract

Recently, with the expansion of the smart city and autonomous driving-related technologies within complex urban structures, there has been an increase in the demand for precise 3D modeling technology. Wearable sensor systems can contribute to the construction of seamless 3D models for complex urban environments, as they can be utilized in various environments that are difficult to access using other sensor systems. Consequently, various studies have developed and utilized wearable sensor systems suitable for different target sites and purposes. However, studies have not yet suggested an overall framework for building a wearable system, including a system design method and an optimal calibration process. Therefore, this study aims to propose a framework for wearable system development, by presenting guidelines for wearable sensor system design and a calibration framework optimized for wearable sensor systems. Furthermore, calibration based on point–plane correspondences is proposed. A wearable sensor system was developed based on the proposed guidelines and it efficiently acquired data; the system calibration and data fusion results for the proposed framework showed improved performance in a comparative evaluation.

1. Introduction

The demand for detailed 3D modeling of urban landscapes has recently increased in tandem with the growing demand for smart cities and autonomous driving technologies [1]. Precise 3D modeling can be used to solve urban problems [2], as well as for autonomous driving and vehicle localization in GNSS-denied environments [3]. Furthermore, 3D modeling that reflects spatiotemporal changes facilitates the implementation of digital twins and smart cities [4]. Thus, it is considered an essential element of future technologies, with applications in the city and for transportation.

Multi-sensor systems have been considered among the important research topics of the past few years, owing to their ability to increase the efficiency of 3D modeling [5,6]. In particular, the usability of wearable multi-sensor systems equipped with LiDAR and cameras is increasing with the demand for 3D modeling of complex urban structures, such as indoor and underground spaces, urban canyons, alleys, and mixed traffic streets [3]. Wearable systems have several advantages, the system offers sufficient 3D information even in environments with insufficient texture or discontinuous depth [7,8,9]. Wearable systems mounted on the human body are available in various environments. More specifically, wearable systems can be operated in complex environments where the accessibility of mobile platforms is severely restricted, such as indoors, underground spaces, narrow alleys, mixed traffic streets, and historical buildings [10,11,12].

Although there are significant advantages of the wearable multi-sensor system for urban 3D modeling, for accurate data acquisition and fusion, the following must be carefully considered in system development. First, the sensor arrangement method for efficient data acquisition must be reviewed. Wearable sensor systems have a small maximum payload, thereby limiting the number and types of sensors used. Therefore, the configuration and arrangement of sensors must be reviewed, with the aim of acquiring data more efficiently using a relatively small number of sensors. Second, self-calibration of the on-board sensors is required. Self-calibration is the process of estimating the interior orientation parameters (IOPs) and the distortion parameters of the camera by determining the mathematical model equation related to the data acquisition of each sensor. Finally, for accurate data fusion, precise system calibration should be conducted to determine the geometric relationships between sensors. In particular, the system calibration for wearable systems should be conducted especially carefully for the following reasons. For efficient data acquisition, wearable systems have generally been designed to minimize the overlapping of FOV (Field Of View) between sensors. However, such a sensor arrangement obstructs the acquisition of common control features between sensors and limits the accuracy of system calibration. Therefore, the design and characteristics of the wearable system should be considered during the system calibration procedure.

1.1. Previous Studies

Previous studies related to wearable systems have developed systems suitable for the purpose of each study; however, details have been lacking regarding the placement method for the sensors of the wearable system for efficient data acquisition. Gong et al. [3] developed a backpack LiDAR system comprising two 3D LiDARs for 3D modeling of underground parking in a GNSS-denied environment. Filippo et al. [11] developed a wearable mapping system using a hand-held laser scanner for 3D modeling of a historical site. Chahine et al. [10] performed data fusion for data acquired in the forest area using a backpack-type wearable system equipped with LiDAR and multiple optical cameras. Liu et al. [13] developed a system to help visually impaired people move indoors using a wearable system composed of LiDAR and a camera. However, these studies did not provide specific criteria or reasons for the arrangement and design of their sensors and sensor systems. One previous study [14] related to this research indicated the method for LiDAR sensor arrangement within the backpack system; however, that study dealt only with LiDAR.

Self-calibration of optical cameras and LiDAR sensors has been sufficiently verified through various previous studies. The calibration methodology for conventional and fish-eye lens cameras has been established through various studies [15,16,17,18,19,20,21,22,23,24] from the 1960s to recent years. Furthermore, regarding LiDAR self-calibration, various methodologies have been proposed and verified through numerous studies [25,26,27,28,29,30]. Therefore, the self-calibration of optical cameras and LiDARs can be performed reliably and accurately via the approaches of previous studies.

Various studies have proposed multi-sensors system calibration approaches; however, most of the previous studies focused on the characteristics of the sensor without considering the platform used in the development of the system calibration approach. In other words, previous approaches mainly considered the correspondences between control features (points, lines, and planes) of sensor data, not the characteristics of the sensor arrangement according to the platform.

The point–point correspondence-based approach has been the most commonly used methodology of system calibration. This approach allows stable calibration, because several matched points robustly limit the relative position of the sensors. Therefore, many previous studies used the point–point correspondence-based approach to conduct system calibration between an optical camera and a LiDAR, using the camera images and intensity images [31,32,33,34,35,36] or LiDAR point clouds [37,38,39,40,41,42,43,44]. However, this approach has the disadvantage that the accuracy is dependent on the resolution of the data and the FOV overlap between the sensors [45,46,47,48,49].

The conventional point–plane correspondence-based approach is a calibration method for an optical camera and a LiDAR. The constraint condition of this approach is that LiDAR point clouds for a plane should be on the corresponding plane derived from the optical images. This approach has the advantage that it requires nothing but checkerboard calibration targets in the testbed configuration [50]. Thus, several studies have employed the conventional point–plane correspondence-based approach [51,52,53,54,55]. However, the approach has a disadvantage in that (i) the application is limited to a system consisting of an optical camera and a LiDAR with sufficient FOV overlap, and (ii) it requires an additional process for extracting 3D plane information from optical images.

The line–plane and point–line correspondence-based approaches are generally used for the systems with 2D LiDAR and cameras. These approaches offer the advantage that the positional relationship between control features is strongly constrained, and they derive accurate results [56,57]. Therefore, they have been utilized in various studies dealing with 2D LiDAR [56,57,58,59,60,61,62,63,64]. However, the line–plane and point–line correspondence-based approaches are not suited for 3D LiDAR systems, and require an additional process to extract the edge points or lines of the calibration target.

The common disadvantages of previous system calibration approaches include their unsuitability for systems consisting of sensors with insufficient FOV overlap. Furthermore, previous system calibration methods for multi-sensor systems generally required pre-processing to measure GCP (Ground Control Point) coordinates. The procedure for GCP measurement increases the time required for system calibration and makes periodic system calibration difficult.

1.2. Purpose of Study

The purpose of this study is to propose a framework for building a wearable system that generates accurate RGB-D data. More specifically, the objectives of this study are: (1) to suggest guidelines for the design of a wearable multi-sensor system, (2) to propose a framework that can accurately conduct calibration of a wearable sensor system, and (3) to propose an efficient system calibration method.

In this study, criteria for sensor placement of wearable systems are suggested for efficient and accurate RGB-D data generation. For efficient data acquisition and accurate data fusion, sensor placement and system design should be determined carefully. However, most previous studies have not expanded upon the details of the sensor placement for wearable systems. Therefore, this study suggests the criteria to be considered in the sensor arrangement of a wearable sensor system.

This study proposed a calibration framework optimized for wearable sensor systems. Because the number of mounted sensors in a wearable system is limited, the FOV overlap between the same modal sensors is generally kept small for efficient data acquisition. Therefore, previous calibration methods for sensors with high FOV overlap are not suitable for wearable systems calibration. The proposed calibration framework is optimized for wearable sensor systems and can be applied to various sensor systems. Sensor-bundle configuration proceeds based on interpretation of FOV overlap between sensors, and subsequently conducts system calibrations for each bundle.

Finally, in this study, a calibration method based on point–plane correspondence is proposed for more efficient system calibration. Previous calibration methods for multi-sensor systems require (i) the coordinates of GCPs or the length of control lines, or (ii) processes to extract points or lines on the edge of the calibration target of the test-bed. On the other hand, the proposed method does not require the pre-processed coordinates of GCPs for point or line extraction on the edge of the calibration target. Therefore, the proposed method contributes to efficient system calibration. Furthermore, the proposed method has high utility because it is applicable to sensors with low FOV overlap. It is based on bundle adjustment (BA) using a co-linearity condition, which is commonly used in optical camera calibration.

This paper describes the research in the following order. Section 2 presents the sensor arrangement method for the backpack-based wearable system used in this study, and describes the actual fabricated sensor system. Section 3 provides a detailed description of the proposed calibration framework. The contents of self-calibration for the optical camera and LiDAR sensor (mounted in the wearable system) in the previous study are briefly introduced, and the details of the proposed system calibration framework are described. Section 4 evaluates the proposed framework through the calibration of a wearable sensor system and the analysis of data fusion results. Finally, Section 5 considers the significance of this study and presents suggestions for future research.

2. Design and Production of Wearable Sensor System

In this study, a wearable sensor system was designed based on efficient data acquisition, system calibration, and data fusion accuracy. The wearable sensor system was designed in two major steps. First, the platform to be used and the type of sensor to be used were determined; second, the number of sensors and the sensor arrangement were determined through simulation.

2.1. Platform and Sensors

The platform and sensors used for building the wearable sensor system were the backpack platform, optical camera, and LiDAR, respectively. The reasons for selecting the backpack as the platform of the system were as follows. First, the backpack platform can accommodate more sensors than other wearable platforms. Also, the backpack-based wearable system was easier to move through various environments, leaving the wearer’s hands free. Moreover, it could be used in a variety of ways owing its capacity if necessary to be mounted on a wheel-based platform with a simple modification.

This study used an optical camera and 3D LiDAR for RGB-D data generation, and the sensor to be used was determined as shown in

Table 1. Specifications of used sensors.

Sensor		Model	Specifications
Camera	Lens	Sennex DSL315	Projection model		Fish-eye lens (equisolid angle projection)
	Body	Cameleon3 5.0 MP Color USB3 Vision	Image size (pixel size)		$2448\times 2448$ (0.00345 mm)
LiDAR		Velodyne HDL-16E	Channel number		16
			FOV	Horizontal	360°
				Vertical	30° (±15°)
			Resolution	Horizontal	0.5° (10 Hz)
				Vertical	2°

. First, fisheye lens cameras were employed to acquire RGB data efficiently. The advantage of a fisheye camera is its ability to acquire data efficiently through a wide FOV compared to a conventional camera. Therefore, the system in this study was developed using fisheye cameras to minimize the number of mounted sensors. The LiDAR sensor was adopted in the system for accurate 3D depth data acquisition. In other words, because 3D modeling using only optical images could result in low accuracy in an environment where feature points were insufficient or similar textures were repeated, in this study a multi-modal sensor system was constructed composed of an optical camera and LiDAR.

2.2. Design of Wearable Sensor System

Sensor placement criteria were prepared for efficient data acquisition and accurate data fusion, and the design of the wearable sensor system was agreed. The wearable sensor system design criteria determined in this study were as follows:

• Criterion 1. One LiDAR sensor is installed to acquire data about the surrounding environment without gaps, while one LiDAR is installed horizontally on the ground to enable data fusion using algorithms such as SLAM (Simultaneous Localization And Mapping) or ICP (Iterative Close Point).
• Criterion 2. For efficient data acquisition, the overlapping of FOV is minimized between cameras of the same type, thereby allowing data for a wider area to be acquired with one shot.
• Criterion 3. The FOV of the optical camera and LiDAR allows sufficient overlap to enable efficient data fusion between different types of sensors.
• Criterion 4. All sensors are positioned to exhibit a sufficient overlapping of FOV with at least one other sensor to ensure system calibration accuracy.

In this study, a wearable sensor system was designed based on the above criteria, followed by a review of the design created through simulation. Figure 1 shows the final determined sensor arrangement and the data acquisition simulation results obtained. As shown in Figure 1a, the sensor system comprised two fisheye lens cameras and two LiDARs. According to Criterion 1, LiDAR1 was installed horizontally and LiDAR2 was installed diagonally on the back of the backpack. The diagonal placement of LiDAR2 was determined through a previous study [14] on Li-DAR placement method for efficient data acquisition. According to Criterion 2, the fisheye cameras were installed to face the left and right of the backpack, respectively. Figure 1b,c shows the data acquisition simulation results. Here, the red and green crosses in the figure indicate the areas covered by fisheye cameras 1 and 2, respectively, and the cyan and purple dots indicate the areas covered by LiDARs 1 and 2, respectively. The figure shows that the design of the wearable system satisfies each criterion as follows. First, the fisheye cameras were placed such that their FOVs did not overlap to acquire image data for all areas around the system. Next, the LiDAR sensors seamlessly acquired 3D information of the surrounding area, while point cloud data (light blue dots) that can be used in algorithms such as SLAM was acquired through LiDAR1. Finally, with respect to Criteria 3 and 4, the FOV of camera 1 overlapped with that of LiDARs 1 and 2 in the left direction of the system movement direction (yellow arrow), while the FOV of camera 2 overlapped with that of LiDARs 1 and 2 in the right direction.

Figure 2 shows the backpack-based wearable sensor system that was produced based on the design. First, a reference LiDAR (LiDAR1 in Figure 1) was installed at the top of the platform, and depth data was obtained horizontally. The second LiDAR (LiDAR2 in Figure 1) was installed at low overlap ratio with LiDAR 1 on the back of the platform in a 60-degree diagonal direction. Two fish-eye lens cameras were installed at the top-left (camera 1) and top-right (camera 2) of the platform to acquire images for the left and right sides of the platform, respectively.

3. Calibration Framework for Wearable Sensor Systems

The calibration framework for the wearable sensor system proceeded in two stages: sensor self-calibration and system calibration. In the sensor self-calibration phase, the IOPs and distortion parameters of the fisheye camera and LiDAR embedded in the system were calculated. Next, in the system calibration step, ROPs that describe the relative geometric positions between the sensors were derived.

Self-calibrations for onboard sensors were performed using the methodology proposed in the previous study paper of this research. The self-calibration of the fisheye cameras was performed using the approach proposed by Choi et al. [24]. Next, for self-calibration of LiDAR, the method proposed by Kim et al. [30] was applied. Further details regarding self-calibration for the fisheye camera and LiDAR can be found in related previous studies [22,23,24,30].

In the proposed system calibration, the relative location between the sensors exhibiting a relatively high overlap ratio was first estimated through system calibration, based on which the relative locations for all sensors were derived. The method comprised three steps: (i) sensor-bundle configuration, (ii) system calibration for each sensor-bundle, and (iii) global optimization for the entire sensor system. First, in the sensor-bundle configuration stage, a camera and LiDAR with a relatively high FOV overlap ratio were checked and set as one sensor-bundle. Next, system calibration was performed for each sensor-bundle set employing the point–plane correspondence-based approach. Finally, in the global optimization step, the ROPs of all other sensors based on the reference sensor were calculated using ROPs between sensors derived through system calibration.

3.1. Sensor-Bundle Configuration

In this study, a multi-sensor-bundle was constructed by selecting sensors with a high FOV overlap ratio among sensors installed in the multi-sensor system. In case of insufficient FOV overlap ratio between a sensor and the reference sensor, accuracy of estimated ROPs can be reduced owing to insufficient control feature correspondences. Therefore, in this study, calibration of sensors with high FOV overlap was first carried out, and a sensor-bundle was constructed more stably to obtain the final relative orientation parameters (ROPs) using the result.

Figure 3 describes the sensor-bundle configuration method for the multi-sensor system used in this study. Figure 3a shows the ROPs to be derived from the wearable system used in this study. LiDAR 1 was set to reference sensor, and the ROPs of the two cameras and LiDAR 2 were targets to be estimated from system calibration. However, regarding the manufactured wearable system, extracting a common control object and securing the accuracy of the ROPs was challenging owing to the very low FOV overlap ratio of LiDAR1 and LiDAR2. Therefore, in this study, four sensor bundles (LiDAR1-Camera1, LiDAR1-Camera2, LiDAR2-Camera1, and LiDAR2-Camera2) were composed of an optical camera with a high FOV overlap ratio and LiDAR, as shown in Figure 3b. Subsequently, in the system calibration step, the ROPs (${\mathrm{rop}}_{l1}^{C1}$, ${\mathrm{rop}}_{l1}^{C2}$, ${\mathrm{rop}}_{l2}^{C1}$, and ${\mathrm{rop}}_{l2}^{C2}$) of each sensor-bundle were estimated, whereas in the global optimization step, the final ROPs (${\mathrm{ROP}}_{l1}^{c1}$, ${\mathrm{ROP}}_{l1}^{C2}$, and ${\mathrm{ROP}}_{l1}^{l2}$) were derived using the ROPs of sensor bundles.

3.2. Point-Plane Correspondences Based System Calibration for Sensor Bundles

3.2.1. Point-Plane Correspondence of the Proposed System Calibration Method

The system calibration approach proposed in this study is based on the point–plane correspondence condition that “all control points estimated using camera images should be on the plane extracted from the LiDAR point cloud.” The manner in which the point-plane correspondences of the proposed method determine the geometric relationship between the sensor data is indicated in Figure 4; where the green, black, and blue dots represent LiDAR point clouds, GCPs, and image control points corresponding to GCPs corresponding to the yellow plane, respectively, the white dots are the initial GCPs to be input into bundle adjustment, and the red ellipses are the error ellipses of the initial GCPs.

When a planar target with GCPs was captured through LiDAR and a camera, the GCPs (black dots) positioned on the target plane were positioned on a planar patch (yellow rectangle) obtained from the LiDAR data. Therefore, any point on the planar patch (white point) can be set as the initial coordinates of GCPs, and the set GCPs yield an error ellipse that exhibit large and very small errors in the plane (U-V plane) and vertical directions, respectively, as shown in Figure 4. Exterior orientation parameters (EOPs) of each image expressed in the LiDAR coordinate system can be derived by performing BA using the collinearity condition between the image control point (blue dot) and the set initial GCPs; where the coordinates of the initial GCPs are arbitrarily selected on the plane, and those of the initial GCPs are adjusted in the BA process by reflecting the size of the error ellipse in the BA weight matrix.

3.2.2. Application of the Proposed System Calibration Method

In the proposed system calibration methodology, three or more planes installed perpendicular to each other are required to accurately adjust the coordinates of GCPs while securing BA accuracy. Figure 5 shows three patches arranged perpendicular to each other, and the white and black dots show the positions of the initial GCPs and the actual GCPs, respectively. The position (plane direction coordinates) of each GCP in the plane direction is limited by the constraint that “GCPs must be located on the corresponding plane” for GCPs on different patches. For example, the GCPs in patches 2 and 3 cannot be adjusted in directions perpendicular to the plane ($z_{p2}$ and $z_{p3}$ respectively), as each of them must be located on a plane, and the vertical orientations of patches 2 and 3 coincide with the plane directions ($x_{p1}$ and $y_{p1}$) in patch 1, respectively. Hence the constraint that the GCPs of patches 2 and 3 must be on each patch serves as a condition for positioning the GCPs of patch 1 on the plane.

The proposed system calibration method was applied to each sensor bundle as shown in Figure 6; where the green dots are LiDAR point clouds in Epoch m, and the black dots are the GCPs. When the proposed system calibration was performed using the entire image (n images) and the point cloud obtained from Epoch m, the EOPs of each image acquired in n epochs were calculated based on the LiDAR coordinate system of Epoch m. Moreover, as the relative position between the camera and LiDAR was fixed, EOPs of the image m (${\mathrm{EOPs}}_l^m$) are the same as the ROPs (${\mathrm{ROPs}}_l^C$) between the LiDAR and the camera. Therefore, through the point–plane correspondence-based approach, system calibration between LiDAR and camera can be performed and the ROPs between sensors can be estimated.

3.2.3. Advantages of the Proposed System Calibration Method

The proposed point–plane correspondence-based approach offers four advantages as follows:

• First, the proposed methodology only requires information regarding the target plane from the point cloud. It does not require an additional complicated process to accurately extract the edge points or lines of the target plane and is less affected by the density of the point cloud.
• Second, arbitrarily determined initial GCPs employing the BA process are used instead of the coordinates of previously measured GCPs. Therefore, the need for prior work measuring the coordinates of GCPs using a total station, etc., is eliminated and the time required to perform system calibration can be reduced.
• Third, because the proposed method is based on BA using the collinearity condition, implementing it utilizing various existing open-source codes is easy.
• Fourth, the proposed method offers the advantage that when sensor data for part of the target plane is acquired, information on the entire target can be used for calibration.

Figure 7 shows further specific details for the fourth advantage of the proposed system calibration method. In the proposed method, the coordinates of the initial GCPs corresponding to the control points (red and blue points in Figure 7a) were set to random points on the plane (white points in Figure 7a), which were then adjusted to their actual positions through BA. Therefore, control points outside the plane patch without corresponding point clouds (red dots in Figure 7a) can also be used for calibration. In particular, the above advantage becomes more prominent when the FOV overlap ratio between the LiDAR and the camera is low, as shown in Figure 7b. The proposed method utilizes all the control points shown in Figure 7b (red and blue dots), thereby allowing more stable calibration than when only GCPs inside the planar patch (blue dots) are used.

3.3. Global Optimization for Wearable System

In the global optimization step, the ROPs of all other sensors based on the reference sensor were derived and adjusted using the ROPs of each sensor bundle estimated in the system calibration step. In the multi-sensor system used in this study, the sensor-bundle was set as shown in Figure 3b, and the ROPs of each sensor-bundle ${\mathrm{rop}}_{l1}^{c1}$, ${\mathrm{rop}}_{l2}^{c1}$, ${\mathrm{rop}}_{l1}^{c2}$, and ${\mathrm{rop}}_{l2}^{c2}$ were derived in the calibration stage, where, ${\mathrm{rop}}_{l1}^{c1}$-${\mathrm{rop}}_{l2}^{c1}$-${\mathrm{ROP}}_{l1}^{l2}$ and ${\mathrm{rop}}_{l1}^{c2}$-${\mathrm{rop}}_{l2}^{c2}$-${\mathrm{ROP}}_{l1}^{l2}$ must satisfy the closed loop condition. These two closing conditions can be expressed as Equations (1)–(4), where $T_{l1}^{l2}$ and $M_{l1}^{l2}$ are the translation and rotation matrices of ${\mathrm{ROP}}_{l1}^{l2}$, and $T_{li}^{cj}$ and $M_{li}^{cj}$ are the translation and rotation matrices of the ROPs of the j-th camera with respect to the i-th LiDAR, respectively. In this study, while deriving ${\mathrm{ROP}}_{l1}^{l2}$ based on the least squares method using Equations (1)–(4), adjustments were made to the estimated ROPs in the calibration step, and the variance value of the orientation parameters calculated in the system calibration step was used as the weight for each ROP.

$$M_{l1}^{l2}(T_{l1}^{c1}-T_{l1}^{l2})-T_{l2}^{c1}=0$$

(1)

$$M_{l1}^{c1}-M_{l2}^{c1}{M}_{l1}^{l2}=0$$

(2)

$$M_{l1}^{l2}(T_{l1}^{c2}-T_{l1}^{l2})-T_{l2}^{c2}=0$$

(3)

$$M_{l1}^{c2}-M_{l2}^{c2}{M}_{l1}^{l2}=0$$

(4)

3.4. Test Bed Configuration and Data Acquisition for Calibration

In this study, using the proposed methodology, a test-bed was built to perform system calibration for the wearable sensor system, and data for calibration were acquired. Figure 8a shows the calibration target used for the test-bed construction. Each calibration target consisted of two planes forming an angle of 120°, with multiple grid patterns to be used as control points printed on each plane. Figure 8b shows the test-bed constructed using the calibration targets. A total of nine calibration targets (18 planes in total) were used in the test-bed configuration, and the target planes were installed at various angles to one another to ensure the accuracy of system calibration. Figure 9 shows the location and direction where the data for calibration were acquired. The red dots and blue arrows in the figure indicate the optical camera positions and shooting directions of each sensor bundle.

4. Evaluation of System Calibration Results

In this section, the results of system calibration through proposed framework are presented and evaluated. First, the performance of the proposed point-plane correspondences-based approach is presented through precision and accuracy of sensor-bundle ROPs. Next, a visual analysis of the accuracy of ROPs between the estimated LiDAR and the camera is performed. Finally, the accuracy of ROPs between LiDAR is estimated via global optimization and then comparatively evaluated with other approaches.

The results of system calibration for each sensor bundle using the proposed method were derived as presented in

Table 2. System calibration results for sensor bundles.

	X0 (mm)	Y0 (mm)	Z0 (mm)	ω (°)	φ (°)	κ (°)
${\mathrm{rop}}_{l1}^{c1}$	−232.89	−121.40	−16.28	89.28	79.66	−87.85
${\mathrm{rop}}_{l2}^{c1}$	−226.61	−281.42	−406.70	−157.07	100.42	97.56
${\mathrm{rop}}_{l1}^{c2}$	210.83	−122.16	−18.32	273.90	−100.73	274.78
${\mathrm{rop}}_{l2}^{c2}$	214.32	−281.23	−404.43	36.17	−80.89	98.75

. In this study, the proposed approach was judged to exhibit high accuracy based on the standard deviation results, residuals of image control points, and correlation between orientation parameters.

Table 3. Standard deviation of system calibration for sensor bundles.

	Standard Deviation of ROPs						RMS-Residuals for Image Points (pixel)
	X0(mm)	Y0(mm)	Z0(mm)	ω (°)	φ (°)	κ (°)
${\mathrm{rop}}_{l1}^{c1}$	2.25	3.67	2.57	0.0033	0.0011	0.0033	0.56
${\mathrm{rop}}_{l2}^{c1}$	0.77	1.50	2.28	0.0058	0.0008	0.0058	0.48
${\mathrm{rop}}_{l1}^{c2}$	2.78	4.55	4.11	0.0089	0.0019	0.0081	0.59
${\mathrm{rop}}_{l2}^{c2}$	4.95	4.86	4.89	0.0117	0.0008	0.0119	0.55

presents the standard deviation of the estimated ROPs and the residuals of the image control points. The maximum standard deviation of estimated ROPs was found at X0 and κ of ${\mathrm{rop}}_{l2}^{c2}$, which were 4.95 mm and 0.0119°, respectively. Further, the RMSE of the image control points were found to be a maximum and minimum of 0.68 and 0.79 pixels, respectively. In addition, the correlation between the ROPs was generally low (

Table 4. Correlation between the ROPs.

	${\mathit{r}\mathit{o}\mathit{p}}_{\mathit{l}1}^{\mathit{c}1}$						${\mathit{r}\mathit{o}\mathit{p}}_{\mathit{l}2}^{\mathit{c}1}$
	X0	Y0	Z0	ω	φ	κ	X0	Y0	Z0	ω	φ	κ
X0	-	−0.27	0.08	−0.28	0.79	0.21	-	0.19	0.17	−0.26	0.02	0.28
Y0	−0.27	-	−0.15	0.95	0.10	−0.84	0.19	-	−0.08	−0.86	−0.42	0.87
Z0	0.08	−0.15	-	−0.20	0.36	−0.02	0.17	−0.08	-	−0.37	0.92	0.37
ω	−0.28	0.95	−0.20	-	0.06	−0.79	−0.26	−0.86	−0.37	-	−0.02	−0.79
φ	0.79	0.10	0.36	0.06	-	−0.20	0.02	−0.42	0.92	−0.02	-	0.01
κ	0.21	−0.84	−0.02	−0.79	−0.20	-	0.28	0.87	0.37	−0.79	0.01	-
	${\mathrm{rop}}_{l1}^{c2}$						${\mathrm{rop}}_{l2}^{c2}$
	X0	Y0	Z0	ω	φ	κ	X0	Y0	Z0	ω	φ	κ
X0	-	0.03	−0.05	0.23	−0.01	0.23	-	−0.01	0.02	0.02	0.08	0.02
Y0	0.03	-	0.21	0.01	0.90	−0.08	−0.01	-	0.00	−0.55	−0.10	−0.54
Z0	−0.05	0.21	-	−0.89	0.21	−0.91	0.02	0.00	-	−0.08	0.51	−0.07
ω	0.23	0.01	−0.89	-	−0.01	0.78	0.02	−0.55	−0.08	-	−0.05	0.78
φ	−0.01	0.90	0.21	−0.01	-	−0.07	0.08	−0.10	0.51	−0.05	-	−0.05
κ	0.23	−0.08	−0.91	0.78	−0.07	-	0.02	−0.54	−0.07	0.78	−0.05	-

). The highest correlation (0.95) was that between Y0 and ω of ${\mathrm{rop}}_{l1}^{c1}$; however, it was not at a level that had a significant effect on the accuracy of system calibration.

Global optimization was performed using the estimated ROPs of each sensor bundle (Table 2) and the standard deviation of ROPs (Table 3). The results are presented in

Table 5. Estimated ROPs of backpack based wearable sensor system.

	X0 (mm)	Y0 (mm)	Z0 (mm)	ω (°)	φ (°)	κ (°)
${\mathrm{ROP}}_{l1}^{c1}$	−232.81	−121.88	−16.13	89.28	79.66	−87.85
${\mathrm{ROP}}_{l1}^{c2}$	210.51	−121.09	−18.66	273.9	−100.73	274.78
${\mathrm{ROP}}_{l1}^{l2}$	2.49	−340.55	422.79	61.28	1.05	−0.26

; where ${\mathrm{ROP}}_{l1}^{c1}$ and ${\mathrm{ROP}}_{l1}^{c2}$ show the corrected ROPs of cameras 1 and 2, respectively, and ${\mathrm{ROP}}_{l1}^{l2}$ shows the ROPs of LiDAR 1 derived through global optimization. In this study, the fusion data generated using the final ROPs was analyzed to verify the accuracy of the final system calibration results.

First, in this study, the accuracy of the final ROPs was analyzed through a visual analysis of the fusion accuracy between the camera image and the point cloud. To clearly determine the accuracy of the fusion data, a color that was easily identifiable was provided to the main plane of the image (a relatively wide plane with the edges of the plane easily identifiable on the image), and the fusion accuracy was confirmed at the corners where each plane met. The fusion accuracy visual analysis was performed using sensor data acquired at two test-sites. Figure 10 shows the optical images acquired at the test-sites; Figure 10a–d are the original images obtained at test-site 1 and the images with new colors, (e–h) are the original images obtained at test-site 2 and the images with new colors.

Figure 11 and Figure 12 show the data fusion results for test-sites 1 and 2, respectively. As evident, in both test-sites, the color of the plane corresponding to the point cloud corresponding to each plane was properly assigned and overall accurate data fusion was achieved. Initially, at test-site 1, a data fusion error was confirmed at position 1. However, considering that the error appeared on both sides of the wall simultaneously, the error may have been due to a LiDAR multi-path error at the edge and not because of an ROP error. At test-site 2, an error appeared at position 2 in Figure 11b, and the level of error was approximately 1.5 cm at a distance of approximately 4 m. However, as the error appeared along the edge of the boundary between the white and yellow planes, it did not appear significantly to affect the overall accuracy of data fusion.

To clarify the accuracy of ROPs (${\mathrm{ROP}}_{l1}^{l2}$) between the LiDARs, fusion error between point clouds was evaluated based on the distance between the point clouds being acquired from LiDARs 1 and 2, which correspond to the same plane in this study. With more accurate ROPs between the LiDARs, the planes scanned simultaneously by both LiDARs can be visualized at the same plane through data fusion. Conversely, with less accurate ROPs, each plane exhibits different plane shapes and the distance between the planes becomes larger. Figure 12 shows the calculation methodology for the fusion error between point clouds. The gray and green points are the point clouds of LiDAR 1 and LiDAR 2, respectively, and the black lines are planar patches extracted from the point clouds of LiDAR 1. In addition, $P_i$ is the i-th point cloud of LiDAR 2, $D_i$ is the distance (m) between LiDAR 1 and $P_i$, and $e_i$ is the distance from $P_i$ to the planar patch. The fusion accuracy between LiDARs was evaluated based on the mean bias error (MBE), the mean absolute error (MAE), and the root mean square error (RMSE) of $e_i$ and $\widetilde{e_i}$; $\widetilde{e_i}$ was calculated using Equation (5), implying that $e_i$ was normalized based on the distance between the reference sensor (LiDAR 1) with $P_i$ set as 1 m.

$$\widetilde{e_i}=\frac{e_i}{D_i}$$

(5)

In the accuracy evaluation of ROPs between LiDARs, ROPs obtained through the proposed method (${\mathrm{ROP}}_{l1}^{l2})$ were comparatively evaluated with nominal ROPs and ROPs derived through ICP (iterative closest point).

Table 6. Used ROPs for comparative evaluation of LiDAR data fusion accuracy.

	X0 (mm)	Y0 (mm)	Z0 (mm)	ω (°)	φ (°)	κ (°)
${\mathrm{ROPs}}_{nom}$	0.00	−350.00	400.00	60.00	0.00	0.00
${\mathrm{ROPs}}_{icp}$	10.50	−341.00	415.00	59.10	1.49	−1.15

presents the nominal ROPs (${\mathrm{ROP}}_{nom}$) used in the wearable system design and ROPs obtained through ICP (${\mathrm{ROP}}_{icp})$. To evaluate the accuracy of ROPs between LiDARs, planes were used located in the section where the FOVs of LiDARs 1 and 2 overlapped. Figure 13 shows the positions of the used planes (No. 1–4).

Table 7. LiDAR data fusion accuracy by test-site.

ROPs	Location	${\mathit{e}}_{\mathit{i}}$			$\widetilde{{\mathit{e}}_{\mathit{i}}}$
		$$\begin{gathered} \mathbf{M}\mathbf{B}\mathbf{E} \\ \left(\mathbf{mm}\right) \end{gathered}$$	$$\begin{gathered} \mathbf{M}\mathbf{A}\mathbf{E} \\ \left(\mathbf{mm}\right) \end{gathered}$$	$$\begin{gathered} \mathbf{R}\mathbf{M}\mathbf{S}\mathbf{E} \\ \left(\mathbf{mm}\right) \end{gathered}$$	$$\begin{gathered} \mathbf{M}\mathbf{B}\tilde{\mathbf{E}} \\ \left(\mathbf{mm}\right) \end{gathered}$$	$$\begin{gathered} \mathbf{M}\mathbf{A}\tilde{\mathbf{E}} \\ \left(\mathbf{mm}\right) \end{gathered}$$	$$\begin{gathered} \mathbf{R}\mathbf{M}\mathbf{S}\tilde{\mathbf{E}} \\ \left(\mathbf{mm}\right) \end{gathered}$$
${\mathrm{ROP}}_{l1}^{l2}$	1	−3.11	29.08	34.82	−1.48	13.85	16.58
	2	−1.23	12.17	15.32	−0.82	8.11	10.21
	3	18.80	18.87	20.12	4.47	4.49	4.79
	4	−1.70	16.40	20.19	−0.55	5.29	6.51
	Overall	4.10	17.71	20.87	0.41	7.94	9.52
${\mathrm{ROP}}_{nom}$	1	46.43	46.44	50.34	22.11	22.11	23.97
	2	20.24	23.00	28.61	13.49	15.33	19.07
	3	49.81	49.81	50.68	11.86	11.86	12.07
	4	26.78	28.86	34.04	8.64	9.31	10.98
	Overall	34.30	35.68	39.57	14.02	14.65	16.52
${\mathrm{ROP}}_{icp}$	1	−20.18	20.33	23.37	−9.61	9.68	11.13
	2	−34.43	34.43	35.94	−22.95	22.95	23.96
	3	117.44	117.44	119.00	27.96	27.96	28.33
	4	−30.87	30.90	32.80	−9.95	9.97	10.58
	Overall	12.01	55.12	56.98	−3.64	17.64	18.50

lists the fusion error between point clouds based on ROP type and test-site location, confirming that the most accurate ROPs were derived employing the proposed methodology. The overall error based on the $\mathrm{RMS}\tilde{\mathrm{E}}$ was the lowest for ${\mathrm{ROPs}}_{l1}^{l2}$, followed by ${\mathrm{ROPs}}_{nom}$ and ${\mathrm{ROPs}}_{icp}$. In particular, the fusion data using ${\mathrm{ROP}}_{l1}^{l2}$ derived using the proposed method showed the lowest fusion error at all positions. However, fusion errors for ${\mathrm{ROP}}_{nom}$ and ${\mathrm{ROP}}_{icp}$ showed very similar absolute values of MBE and MAE at each location, implying that the two ROPs used were not fused on the same plane but arranged adjacent to each other on different planes.

Figure 14 shows the fusion data for each location of the test-site, visually confirming the results in Table 7. When data fusion was performed using ${\mathrm{ROPs}}_{l1}^{l2}$, the point clouds obtained from both LiDARs formed a single plane regardless of location. In contrast, where ${\mathrm{ROP}}_{nom}$ and ${\mathrm{ROP}}_{icp}$ were used, point clouds obtained from different LiDARs cannot be fused into one plane and were rather expressed as two different planes.

5. Conclusions

This study aimed to propose a framework for building a wearable sensor system optimized for complex urban structures. Thus, this study presents a workflow for wearable sensor system design and a calibration framework optimized for wearable sensor systems. Furthermore, as part of the framework, a calibration method based on point–plane correspondences is proposed for efficient system calibration of wearable systems.

The wearable sensor system design workflow was based on criteria prepared in consideration of the efficiency of data acquisition and the accuracy of system calibration and data fusion. In this study, a wearable sensor system was manufactured through the proposed design workflow, and the manufactured sensor system efficiently acquired images and point cloud data about the surrounding environment.

The proposed framework and system calibration method contributed to the accurate calibration and data fusion of the wearable sensor system, and showed better performance than the other method. The advantages of the proposed system calibration method stem from (i) a calibration process designed in two steps, and (ii) a calibration method based on point–plane correspondences. First, through the two-step system calibration method, ROPs between sensors with low FOV overlap can be more stably derived, avoiding a decrease in accuracy obtained by calculating ROPs between same-modal sensors with low FOV overlap ratio, by instead using ROPs between a LiDAR and an optical camera, with relatively high FOV overlap ratio. Second, the system calibration method proposed in this study accurately and efficiently derived ROPs between sensors in a wearable system. The proposed method does not require any prior work to acquire GCP coordinates, and it is easy to implement as a code or program based on the conventional colinear equation by modifying the weight matrix setting method. In addition, it offers the advantage that does not require estimation of a 3D point or line segment located at the edge of the target object in the lidar point cloud. The data fusion results through the proposed framework showed better performance in comparative evaluation with the results of methods using ICP and nominal values.

In conclusion, this study offers a method of efficient and accurate data acquisition using a wearable system suitable for 3D modeling in a complex urban environment. The proposed framework contributes to accurate data fusion through calibration optimized for wearable multi-sensor systems. Furthermore, the proposed point–plane correspondences-based system calibration method supports efficient calibration because it does not require a GCP coordinate measurement process. This advantage of the point–plane correspondences-based method contributes to the implementation of in situ system calibration by using various planar objects of the site. On the other hand, the limitation of the proposed method is that it requires three or more planar targets installed perpendicular to each other. Therefore, when the directions of planar objects for calibration are similar, the performance of the proposed system calibration method may decline. However, for complex urban environments, it is expected that the limitations of the proposed method will have little effect on the accuracy of system calibration, because the urban environment generally includes planar objects in various directions.

This study will contribute to accurate 3D modeling of urban spaces, the generation of the high-definition 3D road maps, and the development of smart cities, in that it supports efficient 3D data acquisition and fusion using wearable sensor systems. In the future, an in situ system calibration methodology and an accurate 3D modeling methodology for the wearable sensor system will be developed by modifying, evolving, and automating the proposed method, based on the advantage that the proposed method does not require prior work to acquire GCP coordinates.