Real-Time Drivable Region Mapping Using an RGB-D Sensor with Loop Closure Refinement and 3D Semantic Map-Merging

Abstract

Drivable region maps, created using a visual sensor, are essential for autonomous navigation because off-the-shelf maps do not reflect contemporary real-world conditions. This study presents a large-scale drivable region mapping system that is capable of capturing large-scale environments in real-time, using a single RGB-D sensor. Whereas existing semantic simultaneous localization and mapping (SLAM) methods consider only accurate pose estimation and the registration of semantic information, when loop closure is detected, contemporaneous large-scale spatial semantic maps are generated by refining 3D point clouds and semantic information. When loop closure occurs, our method finds the corresponding keyframe for each semantically labeled point cloud and transforms the point cloud into adjusted positions. Additionally, a map-merging algorithm for semantic maps is proposed to address large-scale environments. Experiments were conducted on the Complex Urban dataset and our custom dataset, which are publicly available, and real-world datasets using a vehicle-mounted sensor. Our method alleviates the drift errors that frequently occur when the agents navigate in large areas. Compared with satellite images, the resulting semantic maps are well aligned and have proven validity in terms of timeliness and accuracy.

1. Introduction

When autonomous mobile platforms such as autonomous vehicles and mobile robots navigate external environments, the difference in road representation between off-the-shelf maps and actual drivable conditions can lead to potential confusion and accidents.These incidents include scenarios where robotic agents descend stairs, fall off cliffs, or navigate into unexpected construction sites. Therefore, real-time estimates of drivable regions (surfaces where a mobile robot can travel, such as roads and sidewalks) are crucial for safe and reliable autonomous navigation systems. Drivable region mapping has two key requirements. The first is simultaneous localization and mapping (SLAM) [1,2,3], which allows agents to determine their pose and the structure of the surrounding environment. The second is semantic segmentation [4,5,6], which involves understanding what constitutes the surrounding environment. Based on these two technologies, semantic mapping not only maps 3D space but also its meaning. The most basic map representation method for expressing 3D space is to use point clouds, and, thus, the semantic point cloud map contains a semantic label for each point. When considering mapping a larger area, two key factors must be considered for large-scale semantic mapping. First, it is essential to create accurate onsite maps in real time. Second, multiple onsite maps must be effectively merged through postprocessing.

For the first key factor, the most important component in creating accurate onsite maps is loop closure detection. Loop closure refers to the identification of a loop when a specific building or area is circumnavigated and the agent returns to the visited area. When a loop is detected, a globally matched edge is created. This step adjusts the entire recorded trajectory to reduce errors, following the principles of global matching. In addition, the map is transformed according to the trajectory changes. Unsuccessful loop creation or a map in need of modification potentially hinders the collection of large volumes of data for map construction. Moreover, the absence of semantic information, such as roads or sidewalks, can lead to inaccurate path planning. Thus, studies on semantic SLAM [7,8,9] are also being conducted to address this issue. Semantic SLAM extracts semantic information from images and uses it within SLAM. However, in these methods, loop closure is either not applied, or the segmented information is not refined after loop closure.

For the second key factor, when dealing with large-scale tasks, an additional map-merging process is needed. Map-merging combines two or more submaps into a single global map [10,11,12,13,14,15,16]. Using partially constructed maps to generate a large-scale map is highly efficient. Map-merging methods vary significantly depending on the type of maps, such as grid, topological, feature-based, or point cloud maps. This research adopts a point cloud registration method [17,18,19]. In map-merging via these registration methods, there is a risk of failure in complex environments because it relies solely on the geometric information of the whole point cloud. Additionally, it requires using all the points within the submaps, resulting in higher computational costs.

To resolve the two aforementioned issues, our proposed method considers two stages. The first stage is the onsite drivable region mapping stage, and the second stage is the map-merging stage. In the first stage, drivable region maps are created in real time. Additionally, the map-merging stage allows us to combine multiple segmented drivable region maps into a large-scale map. The main goal of the first stage is to estimate drivable regions using both visual SLAM and semantic segmentation. Our approach addresses the misalignment issue arising from semantic SLAM. The loop closure refinement is implemented by considering the semantic label encoded error function. Consequently, our method prioritizes minimizing the gap between semantic information and world/agent data, including maps and six degrees of freedom (6-DOF) poses. In the second stage, for large-scale environments, the map-merging stage is required to construct a global map, i.e., the multiple drivable region maps are integrated into a single map. During the map-merging stage, our proposed merging method is performed based on overlapping regions specified by semantic information. This approach avoids the drift problem that frequently occurs in large-scale drivable region mapping. In these two stages, a large-scale drivable region map is obtained.

In summary, this study proposes a new method to create a large-scale map of drivable regions through semantic map generation and map-merging algorithms. Initially, input images captured by RGB-D sensors undergo semantic segmentation to distinguish between roads, vehicles, and non-drivable areas. We improved the accuracy of the semantic location information through loop closure refinement and map-merging. Finally, leveraging the camera pose derived from the SLAM system, the drivable regions are visualized in real time. To the best of our knowledge, this study represents the first instance of using map-merging and semantic visual SLAM to construct a large-scale semantic map. The main contributions of our proposed method are as follows:

• Semantic visual SLAM and map-merging methods are proposed to identify drivable areas in large-scale environments. Our method allows for the visualization of drivable areas that traditional visual SLAM cannot depict;
• A loop closure refinement for semantic information is developed to build seamless semantic 3D maps. This approach allows for the simultaneous use of semantic segmentation and visual SLAM results to create a unified semantic map, minimizing the gap between semantic information and world/agent data;
• A 3D point cloud registration-based semantic map-merging is performed not only on 3D point descriptors but also on semantic labels to create a large-scale map. Our proposed method uses fewer points in overlapping areas, thereby accelerating the map-merging process;
• Our proposed system was tested in real-world experiments by attaching a single RGB-D camera to a vehicle. We constructed a large-scale drivable map from multiple segmented datasets, demonstrating higher accuracy than existing methods.

2. Related Work

Visual SLAM [20,21,22,23,24,25] is a computational method for building or updating a map of an unknown environment and estimating the current pose using visual sensors. Visual SLAM extracts visual features and optimizes poses and map points through correspondence matching by calculating geometric errors. Feature-based SLAM systems, such as VINS-MONO [24] or ORB-SLAM [25], extract key features from images, find correspondences between images, and estimate the camera pose and the 3D information of the environment. However, this process inevitably leads to drift errors over time. To address this issue, most SLAM systems employ pose graph optimization [26]-based loop closure techniques to minimize accumulated errors. Alternatively, Elhashash et al. [27] proposed the use of satellite images as an external data source to correct drift, while Arun et al. [28] utilized Wi-Fi data for error correction. Similarly, previous works [29,30] have proposed SLAM methods that address trajectory drift by leveraging external data rather than relying on loop closure. In contrast, this study adopts an RGB-D camera-based approach with loop closure as a key component. RGB-D cameras are commonly used as visual sensors for 3D mapping because they provide more stable localization in SLAM processes and face fewer challenges with scale ambiguity compared to monocular or stereo cameras.

For exploiting semantic information of the scene, semantic segmentation has been extensively studied in the field of computer vision. Prior research has investigated recognizing drivable regions from segmented images with deep learning-based methods. The results generated from the training of semantic segmentation [31,32,33,34,35] or instance segmentation [36,37,38], which are well-known tasks for representing semantics, can be utilized for drivable region detection. Recently, Xu et al. [39] employed the semantic segmentation method to determine the real-time drivable region from an image. It applies the concept of a proportional–integral–derivative (PID) controller to a deep learning model and is designed for real-time segmentation.

Gosala et al. [40] proposed a self-supervised learning system trained on monocular images for classification, which performs semantic mapping for roads and objects through bird’s-eye-view transformation. Some studies [41,42,43] have introduced loop closure methods that leverage semantic information. While these approaches demonstrate the potential of semantic data in mapping and localization, they primarily focus on consecutive frame-to-frame semantic mapping. As a result, a gap between the 3D map and the semantic map appears during loop closure since the frame-to-map correspondence has not been considered.

Regarding the second key factor, map-merging process, for large-scale drivable mapping, the descriptor of features extracted from point clouds and the registration method that uses it are important. Point feature histogram (PFH) [44] is a method that finds four features using the normal vector of a plane obtained through principal component analysis (PCA) and the point itself by considering one point and its k-neighbors. Fast point feature histogram (FPFH) [45] is an improved algorithm based on PFH. The original PFH has a complexity of $O\left(n{k}^2\right)$ with n points and k neighboring points. FPFH addresses the complexity issue by first calculating the relationships between points and neighbors and then assigning weights, reducing the complexity to $O\left(nk\right)$ and enabling faster computations. Radius-based surface descriptor (RSD) [46] focuses on the shape of surfaces among features. It classifies surfaces into types, such as plane, edge, corner, cylinder, and sphere. Signatures of a histogram of orientations (SHOT) [47] collect information based on the direction and curvature of the object’s surface in point cloud data, creating a histogram that represents unique features.

For 3D point clouds registration, iterative closest point (ICP) [48] is a well-known algorithm that finds correspondences between target and source point clouds, calculates rotation and translation based on corresponding points, and iteratively refines the alignment. Generalized iterative closest point (G-ICP) [49] is an algorithm that combines ICP and point-to-plane ICP. G-ICP uses point correspondences and also incorporates the normal vectors of planes from the target and source point clouds. However, existing registration methods using point cloud descriptors lack a comprehensive consideration of semantic information for each point or feature descriptors. In this paper, we propose a method that considers semantic information to find overlapped region and effectively merges multiple local semantic maps.

3. Methodology

3.1. Problem Formulation

As shown in Figure 1, our proposed method consists of two steps: drivable region mapping and map-merging.

In the drivable region mapping process, as the RGB-D camera navigates an unknown environment, the series of input images, denoted by $\mathcal{I}={\left\{I_t\right\}}_{t=1:T}$ from time $t=1$ to $t=T$, represents the collection of images captured. As $I_t$ is an RGB-D image, it can be converted to point clouds $P_t={\left\{{\mathbf{p}}_i\right\}}_{i=1:N}$, where N denotes the number of points. A camera pose that corresponds to $P_t$ is denoted as ${\mathbf{x}}_t=\left[\begin{array}{cc}\mathbf{R}& \mathbf{t}\\ 0& \mathbf{1}\end{array}\right]\in \mathrm{SE}\left(3\right)$, where $\mathbf{R}\in \mathrm{SO}\left(3\right)$ represents orientation and $\mathbf{t}\in {\mathbb{R}}^3$ represents position. A camera pose can also be represented with quaternion vectors as ${\mathbf{x}}_t={\left({\mathbf{x}}_t^p,{\mathbf{x}}_t^q\right)}^T$, where ${\mathbf{x}}_t^p=\left(x_t{y}_t{z}_t\right)$ and ${\mathbf{x}}_t^q=\left(q_{x,t}{q}_{y,t}{q}_{z,t}{q}_{w,t}\right)$ are the position and quaternion vectors, respectively. The camera trajectory $\mathcal{X}={\left\{{\mathbf{x}}_t\right\}}_{t=1:t}$ can be obtained using visual SLAM [25] methods. Semantic segmentation outputs labels ${\xi }_t\in \left\{1,\cdots ,cl\right\}$ for each pixel of an image $I_t$, where $cl$ indicates the number of classes. The main goal of this step is to build fused point clouds ${\tilde{P}}_t$, which are composed of $P_t$ and ${\xi }_t\in \left\{road,sidewalk\right\}$. A 3D point ${\mathbf{p}}_i$ in the camera coordinates is projected into a 2D pixel point ${\mathbf{u}}_i$ on the image plane by

$$\left[\begin{array}{c}{\mathbf{u}}_i\\ 1\end{array}\right]=\left[\begin{array}{c}\pi \left({\mathbf{p}}_i\right)\\ 1\end{array}\right]={\displaystyle \frac{1}{z}}\mathbf{K}{\mathbf{p}}_i$$

(1)

where $\mathbf{K}$ is the intrinsic camera matrix and $\pi \left(·\right)$ is a projection function. Conversely, 2D point $\mathbf{u}$ and a class label $\tilde{\xi }$ are converted into 3D point clouds with a depth value of each point [50].

$${\mathbf{p}}_i={\pi }^{-1}\left({\mathbf{u}}_i\right)=D\left({\mathbf{u}}_i\right){\mathbf{K}}^{-1}\left[\begin{array}{c}{\mathbf{u}}_i\\ 1\end{array}\right]$$

(2)

where $D\left({\mathbf{u}}_i\right)$ is the depth value of image pixel ${\mathbf{u}}_i$. Thus, ${\tilde{P}}_t=\left\{P_t,\tilde{\xi }\right\}\in {\mathbb{R}}^4$ is composed of point clouds and each point’s class label. The world coordinate point cloud map ${\tilde{P}}_t^W$ can be generated by the camera pose $\mathbf{R}$ and $\mathbf{t}$.

$${\tilde{P}}_t^W=\left\{\mathbf{R}{\tilde{P}}_t+\mathbf{t},\tilde{\xi }\right\}$$

(3)

The 3D feature point clouds map ${\mathcal{M}}_j$. Here, j indicates the j-th submap index, which is constructed by combining the SLAM results with the semantic point clouds.

$${\mathcal{M}}_j\leftarrow {\tilde{P}}_t^W\cup {\mathcal{M}}_j$$

(4)

The next step is map-merging. The separately built maps ${\mathcal{M}}_{j=1:N}$ across various regions are combined into a single map $\tilde{\mathcal{M}}$. To make $\tilde{\mathcal{M}}$, the collection of feature points ${\mathcal{F}}_j={\left\{{\mathbf{f}}_{k,j}\right\}}_{k=1:K}$ is extracted using FPFH [45] from ${\mathcal{M}}_j$. To integrate maps developed at various times and locations, the 3D point-registration technique was used [51].

3.2. System Overview

Drivable Region Mapping: For drivable region mapping, both semantic information and the 3D map are required. We employed deep learning-based semantic segmentation method and visual SLAM with a single RGB-D Camera such as Asus Xtion and Intel Realsense. When the camera captures $I_t$, ${\tilde{P}}_t$ is generated via deep-learning-based semantic segmentation results ${\mathbf{u}}_i$ and $D\left({\mathbf{u}}_i\right)$. PIDNet [39] is used for semantic segmentation, and ${\tilde{P}}_t$ represents a collection of points in the camera coordinates. The results of visual SLAM include $\mathcal{X}={\left\{{\mathbf{x}}_t\right\}}_{t=1:t}$ and ${\mathcal{M}}_j$. ORB-SLAM3 is used for visual SLAM. Subsequently, ${\tilde{P}}_t$ is transformed to ${\tilde{P}}_t^W$ according to ${\mathbf{x}}_t$ and combined with ${\mathcal{M}}_j$. This sequence is repeated to create ${\mathcal{M}}_j$. When this transformation occurs, our method effectively addresses the mismatch between the 3D map and semantic information.

Map-merging: In this step, two ${\mathcal{M}}_j$ are merged. First, the overlapping region is specified, and the corresponding point clouds are extracted. Then, ${\mathcal{F}}_j$ is extracted from the extracted point clouds using FPFH [45]. The KNN and KDTree [52] algorithms are subsequently used to find correspondences and perform 3D point cloud registration via TEASER++ [51]. By applying the transformation matrix from TEASER++, the two maps are merged to generate a global map. This process is repeated for all ${\mathcal{M}}_j$, creating final global map $\tilde{\mathcal{M}}$.

3.3. Drivable Region Mapping

The results of semantic segmentation ${\mathbf{u}}_i$ with depth value $D\left({\mathbf{u}}_i\right)$ enable the transformation of pixels to ${\mathbf{p}}_i$. For every ${\mathbf{u}}_i$ in $I_t$, ${\tilde{P}}_t$ is obtained, and Figure 2 shows the result of Equation (2). Additionally, ${\tilde{P}}_t$ is in the camera coordinates. The generation of excessive point clouds from $I_t$ leads to computational memory constraints, which prevent the direct utilization of ${\tilde{P}}_t$. Therefore, the number of point clouds should be reduced through a voxel grid filter. For the simplicity, we denote ${\tilde{P}}_t$ as filtered point clouds.

Integrated Semantic Segmentation and 3D Mapping: As we mentioned in system overview section, the semantic segmentation result ${\tilde{P}}_t$ and the visual SLAM results (${\mathcal{M}}_j$, $\mathcal{X}$) are obtained. To consolidate ${\tilde{P}}_t$ into a single ${\mathcal{M}}_j$, it is necessary to transform camera coordinates ${\tilde{P}}_t$ into world coordinates, following Equation (3). The update is subsequently performed according to Equation (4). However, during this update, many duplicated ${\tilde{P}}_t^W$ can be generated at the same location, which degrades the computing performance. To prevent this, KDTree [52] is utilized to remove duplicate ${\tilde{P}}_t^W$ within a specified threshold.

Additionally, loop closure is necessary. When a loop is detected in ORB-SLAM3, the system enters the loop closure phase. In the loop closure phase, $\mathcal{X}$ is modified through pose graph optimization and global bundle adjustment (BA). Consequently, ${\mathcal{M}}_j$ is also changed; thus, ${\tilde{P}}_t^W$ must be adjusted accordingly.

According to the full BA of ORB-SLAM3, the optimization problem is as follows:

$$\left\{{\mathbf{x}}_t,{\mathcal{M}}_j\right\}=\underset{{\mathbf{x}}_t,{\mathcal{M}}_j}{argmin}{E}_{point}$$

(5)

$$E_{point}=\sum _{k\in \mathcal{K}}\sum _{j\in {\chi }_k}\rho \left(E\left(k,j\right)\right)$$

(6)

where $E\left(k,j\right)={∥{\mathbf{u}}^j-\pi \left({\mathbf{R}}_k{P}^j+{\mathbf{t}}_k\right)∥}_{\Sigma }^2$ is the re-projection error between matched 3D points $P^j\in {\mathbb{R}}^3$ in keyframes $\mathcal{K}$ and key points ${\mathbf{u}}^j\in {\mathbb{R}}^2$ with $j\in {\chi }_k$, the set of all matches. Additionally, $\pi \left(·\right)$ is the projection function mentioned in Equation (1), and $\rho \left(·\right)$ is the robust Huber cost function [53]. Further, the label information of ${\tilde{P}}_t^W$ is considered. The label-encoded error function $E_c$, which represents how correctly the current class label aligns with existing point cloud labels, is defined as

$$E_c=\sum _k\sum _j\left[\mathcal{L}\left({\mathcal{M}}_j\left(P_t^{*}\right)\right)=\mathcal{L}\left({\mathbf{R}}_k{P}^j+{\mathbf{t}}_k\right)\right]$$

(7)

where $\left[·\right]$ indicates the Iverson bracket. Point ${\mathcal{M}}_j\left(P_t^{*}\right)$ is the nearest point from ${\mathbf{R}}_k{P}^j+{\mathbf{t}}_k$ within the existing 3D point map ${\mathcal{M}}_j$. Additionally, $\mathcal{L}\left(·\right)$ indicates the label value of the input points. Therefore, the final BA is formulated as a nonlinear least-squares optimization problem as follows:

$$\left\{{\mathbf{x}}_t,\tilde{\mathcal{M}}\right\}=\underset{{\mathbf{x}}_t,\tilde{\mathcal{M}}}{argmin}{E}_{point}+E_c$$

(8)

Finally, the output of BA $\left\{\mathbf{R},\mathbf{t}\right\}$ is used to adjust ${\tilde{P}}_t^W$.

$${\mathcal{M}}_j\leftarrow \left(\mathbf{R}{\tilde{P}}_t^W+\mathbf{t}\right)\cup {\mathcal{M}}_j$$

(9)

As shown in Figure 3, the keyframe with the corresponding ${\tilde{P}}_t^W$ is matched in real time. If $I_t$ is not a keyframe, ${\tilde{P}}_t^W$ will match with the closest keyframe before $t=T$. When the loop closure phase begins, changes to the keyframe pose are applied to ${\tilde{P}}_t^W$. This ensures that the drivable region created through semantic segmentation helps create a more accurate map during the loop closure phase.

3.4. Map-Merging

Algorithm 1 is the pseudo-code of the proposed map-merging algorithm. The map-merging process starts with preparing the data. Figure 4a,b shows example maps ${\mathcal{M}}_1$ and ${\mathcal{M}}_2$. Before map-merging, information about overlapping buildings or roads between multiple maps is needed. Overlapping regions are selected before data collection and considered in the 3D mapping process. When selecting overlapping regions, the following priorities are established: (1) a building-centric region, (2) a region where both roads and buildings coexist, and (3) a road-centric region. Additionally, when selecting an overlapping area, it is better to consider the point cloud of buildings rather than focusing on roads, especially in the case of straight segments. The overlapping region $P_{OR,j}={\left\{{\mathbf{p}}_{l,j}\right\}}_{l=1:L}$ is extracted using Algorithm 2.

Algorithm 1 Map-merging

Require: Input point clouds $P_{OR,a}$, $P_{OR,b}$, map ${\mathcal{M}}_a$, ${\mathcal{M}}_b$ Ensure: Output global map $\tilde{\mathcal{M}}$  ${\widehat{P}}_{OR,a}$, ${\widehat{P}}_{OR,b}$← Downsampled $P_{OR,a}$, $P_{OR,b}$ by a voxel grid filter  ${\mathcal{F}}_a$, ${\mathcal{F}}_b$← FPFH(${\widehat{P}}_{OR,a}$, ${\widehat{P}}_{OR,b}$)  ${\left\{C_i\right\}}_{i=1:N_m}$← Estimate correspondences  for  $i=1:N_m$  do      $\mathbf{R},\mathbf{t}\leftarrow \underset{\mathbf{R},\mathbf{t}}{argmin}\left\{{\displaystyle \sum _i}{∥n_i·(\mathbf{T}·{\widehat{P}}_{OR,a}-{\widehat{P}}_{OR,b})∥}^2\right\}$  end for   ${\widehat{\mathcal{M}}}_a=\mathbf{R}{\mathcal{M}}_a+\mathbf{t}$   $\tilde{\mathcal{M}}\leftarrow {\widehat{\mathcal{M}}}_a\cup {\mathcal{M}}_b$  return$\tilde{\mathcal{M}}$

Algorithm 2 Extracting point clouds in the overlapping region

Require: input map ${\mathcal{M}}_j$, overlapping region’s edge points $P_{Edge}={\left\{{\mathbf{p}}_i\right\}}_{i=1:N}$ Ensure: overlapping region point clouds $P_{OR,j}={\left\{{\mathbf{p}}_{l,j}\right\}}_{l=1:L}$  for  ${\mathcal{M}}_j$  do      Set $counter=0$, ${\mathbf{p}}_{\mathbf{N}+\mathbf{1}}={\mathbf{p}}_{\mathbf{1}}$      for $i=1:N$ do           if point of ${\mathcal{M}}_j$’s x-axis between ${\mathbf{p}}_i$, ${\mathbf{p}}_{i+1}$ and y-axis lower than $max({\mathbf{p}}_i,{\mathbf{p}}_{i+1})$ and y-axis above line by ${\mathbf{p}}_i$, ${\mathbf{p}}_{i+1}$ then              $counter=counter+1$           end if       end for       if $counter$ = odd number then           point of ${\mathcal{M}}_j\in P_{OR,j}$       end if   end for   return  $P_{OR,j}$

In Algorithm 2, lines 1 to 7, each point in ${\mathcal{M}}_j$ compares the x-axis and y-axis with $P_{Edge}={\left\{{\mathbf{p}}_i\right\}}_{i=1:N}$. $P_{Edge}$ is a set of edge points in the overlapping region. It is specified and represented in the form of an N-polygon. A polygonal shape is selected to best preserve the information of the overlapping region, and z-axis information is not considered in this process. For $i=1:N$, if the point’s x-axis is between ${\mathbf{p}}_i$ and ${\mathbf{p}}_{i+1}$ and the y-axis is lower than $max({\mathbf{p}}_i,{\mathbf{p}}_{i+1})$ but above the line created by ${\mathbf{p}}_i$, ${\mathbf{p}}_{i+1}$, then the counter is incremented by 1. In lines 8 to 11, if the counter is an odd number, then the point is added to $P_{OR,j}$.

The red box in Figure 4c,d indicates an overlapping region $P_{OR,1}$, $P_{OR,2}$. In Algorithm 1, line 1 downsamples $P_{OR,1}$, $P_{OR,2}$ via a voxel grid filter. Figure 4e is an example of this. This process helps to prevent point cloud registration from being dominated by road points and provides computational advantages. We found that a voxel size of 0.1 was the best parameter empirically, as it preserves the features of the point clouds without significant degradation. In Algorithm 1 line 4, ${\mathcal{F}}_a$, ${\mathcal{F}}_b$ are extracted from downsampled point clouds ${\widehat{P}}_{OR,a}$, ${\widehat{P}}_{OR,b}$ by FPFH and, in line 3, correspondences ${\left\{C_i\right\}}_{i=1:N_m}$ are estimated via the KNN algorithm. Figure 4f shows the estimated correspondences and their connections from ${\mathcal{F}}_6$ and ${\mathcal{F}}_8$. In Algorithm 1 lines 4 to 6, the registration step estimates the relative transformation between two point clouds: $\mathbf{R},\mathbf{t}\leftarrow \underset{\mathbf{R},\mathbf{t}}{argmin}\left\{{\displaystyle \sum _i}{∥n_i·(\mathbf{T}·P_{OR,a}-P_{OR,b})∥}^2\right\}$, which aligns $P_{OR,a}=\mathbf{T}·P_{OR,b}$. The 3D point cloud registration tool TEASER++ was used in this process. In line 7, the result of the estimation is aligned with ${\widehat{\mathcal{M}}}_a$. This process can be observed in Figure 4g. If registration does not proceed correctly, $P_{OR,1}$, $P_{OR,2}$ are redefined. Finally, in line 8, ${\widehat{\mathcal{M}}}_a$ is merged with ${\mathcal{M}}_b$ to produce $\tilde{\mathcal{M}}$.

4. Experiments and Results

4.1. Datasets

For semantic segmentation, 4093 labeled images—3475 from the Cityscapes [54] dataset and 618 from a custom dataset—were used for training. The Cityscapes dataset provided sufficient labels for vehicles and roads. The custom dataset focused on capturing roads near Jeonbuk National University, especially pathways with trees and stairs. For annotation, the Computer Vision Annotation Tool was utilized.

For the drivable region mapping, a total of eight paths were captured, as shown in Figure 5. Data collection was performed using a Zed2i RGB-D camera installed on a vehicle. Further details about the dataset are provided in

Table 1. The 3D map information.

	Map1	Map2	Map3	Map4	Map5	Map6	Map7	Map8
Length (m)	491	754	1062	911	1482	531	374	510
Area (m2)	15,947	37,046	38,689	33,237	76,572	11,966	9215	5030
Images	2472	3119	4755	3447	6423	3361	1495	3987
Time (s)	164	211	326	231	434	168	106	156

. Additionally, the Complex Urban dataset [55], a benchmark dataset taken in an urban environment, was also used to evaluate the performance of the proposed method. Among the datasets, the Urban 28 sequence was used, as shown in Figure 6.

Table 2. Complex Urban dataset results.

	Map1	Map2	Map3	Merged Map
Length (m)	3179	2219	5861	11,285
Images	5673	3563	11,671	19,745
RMSE	5.674	11.20	14.75	18.79

provides further details about the dataset. Unfortunately, the Complex Urban dataset does not contain depth images. Therefore, SLAM was performed in stereo mode, and the drivable region was confirmed through depth estimation. In the depth-estimation process, the Depth Anything [56] model was utilized. Through the evaluation of these two datasets, we demonstrated that the proposed method is effective in large-scale environments.

4.2. Drivable Region Mapping Results

PIDNet [39] has three variants based on model size: PIDNet_S, PIDNet_M, and PIDNet_L. Among these, PIDNet_L and PIDNet_M were considered unsuitable for real-time usage due to their higher number of parameters, so PIDNet_S was selected. Compared to other models, the PIDNet_S model achieves a final accuracy of 92.3% and a mean intersection over union (mIoU) of 80.2%, as shown in

Table 3. Training results of the semantic segmentation model using the Cityscapes dataset and our custom dataset.

Segmentation Model		Accuracy (%)	Loss	mIoU (%)	FPS	Params
PIDNet_S	[39]	92.3	0.111	80.2	45.2	7.6 M
DeeplabV3	[57]	92.4	0.769	65.6	5.2	58.3 M
PsPUnet	[58]	90.1	0.312	75.2	24.9	46.7 M
BiSeNetV2	[59]	90.7	0.766	74.8	33.7	12.8 M
DDRNet_23	[60]	96.1	0.212	73.4	35.7	20.1 M

. According to Equation (10), accuracy is calculated by dividing the number of correctly classified pixels by the total number of pixels in the image, meaning 92.3% of the pixels were correctly classified. With a frame rate of 45.2 FPS, PIDNet_S demonstrated optimal performance for real-time applications. Although DDRNet_23 offers slightly higher accuracy and supports real-time execution, we prioritized mIoU as the key evaluation metric. For this reason, PIDNet_S was selected. In the future, the semantic segmentation model may be upgraded to a higher-performing alternative.

$$Accuracy={\displaystyle \frac{NumberofCorrectlyClassifiedPixels}{TotalNumberofPixelsintheImage}}$$

(10)

The implementation results of loop closure for the drivable region mapping are shown in Figure 7. In Figure 7a, the point clouds of buildings overlapped with the drivable region, and the trajectory was not updated because of the unsuccessful loop closure of the drivable region. By contrast, Figure 7b exhibits successful loop closure of the drivable region.

In Figure 8a–h, the results of the drivable region mapping using data captured along the paths presented in Figure 5 are shown. Sequentially, they correspond to Maps 1 to 8. The cyan point clouds represent the drivable region, and the white point clouds indicate features from ORB-SLAM3.

Figure 9 shows the result of the drivable region mapping using the Complex Urban dataset. Owing to the use of stereo images rather than RGB-D images, it was necessary to align the scale difference between the map results. Thus, alignment using evo [61] was performed, followed by performance evaluation, i.e., Root Mean Square Error (RMSE) and qualitative result evaluation. The results are shown in Table 2.

4.3. Map-Merging Results

For map-merging, the process described in Figure 4 was sequentially applied to the data captured in Figure 8a–h, resulting in Figure 10c. For comparison, map-merging was also performed using the trajectory provided by ORB-SLAM3, yielding Figure 10b. The frame of the most visually similar image in each map was manually identified. The quaternion values specified in the trajectory were then used to generate the transformation matrix for each map, which was applied to the world coordinate system. Figure 10a represents the best result obtained by visually comparing and aligning the point clouds of each map for the optimal merging outcome.

A comparison of Figure 10a,b reveals that Figure 10b has significant errors. Discrepancies are particularly noticeable in the central circular section. By contrast, Figure 10c demonstrates precise alignment, indicating the successful execution of our proposed method.

In Figure 11a,b, the increase in data distorted the overall map, even in regions where the original loop closure performed well. Additionally, despite using the same dataset and settings, large-scale map generation consistently produces different results. By contrast, Figure 11c is the result of data captured in separate regions to create maps, avoiding issues during map creation with multiple loops. This approach produced more stable results during large-scale mapping.

In Figure 12a,b, the map and trajectory of the provided ground truth are shown. Figure 12b shows the trajectories of maps 1, 2, and 3, which are displayed in the same way as those in Figure 6. As shown in Figure 12c,d, ORB-SLAM3 failed to localize the camera pose. Therefore, the mapping result also failed even though we reran the SLAM after the initialization process. Figure 12e,f shows the results of the proposed method. Compared with the ORB-SLAM3 result, our proposed method produces better results. It is also evident that, in a large-scale area, dividing the map region and generating a map within those areas yields better results. These findings are summarized in Table 2.

As shown in Figure 11a,b and Figure 12c,d, the results of existing methods that perform mapping without dividing the path do not consistently yield the same outcomes because of drift error, resulting in failures in achieving proper mapping. However, as shown in Figure 11c and Figure 12e, the results obtained via the proposed method demonstrate significant improvement.

The proposed method successfully performs drivable region mapping, as demonstrated in Figure 11c and Figure 12e. It accurately detects roads where vehicles can travel, enabling the creation of maps at a large scale.

Moreover, to the best of our knowledge, this study represents the first instance of large-scale drivable region mapping using only a camera.

5. Conclusions

This study proposed a method for mapping large-scale drivable regions that uses semantic segmentation, visual simultaneous localization and mapping, and map-merging. Semantic visual simultaneous localization and mapping identifies regions where robots can safely navigate, using semantic segmentation and visual localization to create maps that ensure safe movement. The created maps are merged into a single large-scale map using our proposed map-merging method, which uses 3D point cloud registration.

The semantic segmentation and visual simultaneous localization and mapping used for real-time drivable region mapping demonstrated adequate real-time performance, enabling the creation of drivable region maps in real-world scenarios. Semantic segmentation and visual simultaneous localization and mapping were integrated through pose transformation and loop closure.

In the process of map-merging to create large-scale maps, a method that uses 3D point cloud registration based on the feature descriptors of overlapping regions’ 3D point clouds was proposed. This method yielded positive results, particularly with respect to the drift error of visual simultaneous localization and mapping, which tends to increase as the number of images increases. Moreover, our method demonstrated good results compared with real satellite maps.

The proposed method successfully generated a large-scale map of drivable regions. Our method was verified with both our large-scale custom dataset and publicly available benchmark dataset. Furthermore, the proposed map-merging method mitigated drift errors during simultaneous localization and mapping (SLAM) in large-scale environments. On average, only 19% of the data—the ratio of images that correspond to the overlapping area among the entire images—was used for map-merging. Despite utilizing a relatively small amount of data, the process was highly effective. These findings highlight the efficiency and accuracy of our approach to drivable region mapping, a critical component for path planning in autonomous driving systems.

Our future work will focus on visual localization and map updates on large-scale drivable maps composed of 3D point clouds. Additionally, we aim to explore visual-based large-scale drivable mapping in areas with limited features, such as rural environments, as opposed to university campus settings.