Image Preprocessing with Enhanced Feature Matching for Map Merging in the Presence of Sensing Error

Abstract

Autonomous robots heavily rely on simultaneous localization and mapping (SLAM) techniques and sensor data to create accurate maps of their surroundings. When multiple robots are employed to expedite exploration, the resulting maps often have varying coordinates and scales. To achieve a comprehensive global view, the utilization of map merging techniques becomes necessary. Previous studies have typically depended on extracting image features from maps to establish connections. However, it is important to note that maps of the same location can exhibit inconsistencies due to sensing errors. Additionally, robot-generated maps are commonly represented in an occupancy grid format, which limits the availability of features for extraction and matching. Therefore, feature extraction and matching play crucial roles in map merging, particularly when dealing with uncertain sensing data. In this study, we introduce a novel method that addresses image noise resulting from sensing errors and applies additional corrections before performing feature extraction. This approach allows for the collection of features from corresponding locations in different maps, facilitating the establishment of connections between different coordinate systems and enabling effective map merging. Evaluation results demonstrate the significant reduction of sensing errors during the image stitching process, thanks to the proposed image pre-processing technique.

1. Introduction

Autonomous exploration involves the process of robots constructing maps of unfamiliar environments by sensing their surroundings while navigating. Simultaneous Localization and Mapping (SLAM) is a widely adopted approach in autonomous exploration as it enables robots to simultaneously build maps of the environment and determine their own location. The concept of SLAM was initially proposed and discussed in 1986, and it has since gained widespread recognition as a valuable technique [1,2]. Over the years, several commonly used methods have been proposed, starting from the 1990s until the present day. These include EKF-SLAM [3], Fast-SLAM [4], GraphSLAM [5], and others, which have all contributed significantly to the field of SLAM.

Despite advancements in SLAM technology, a single robot is not sufficient to conduct a comprehensive environmental exploration of a large area. As a result, researchers have focused on the development of multi-robot systems (MRS) for the decentralized exploration of environments. MRS involve collaboration among distributed robots, which enhances both work efficiency and system robustness when compared to single-robot systems. However, local maps generated by multiple robots may have different coordinates, making it challenging to integrate them into a global map. Therefore, researchers have explored methods for combining various local maps, which is known as the map merging problem.

This study presents a map merging method that utilizes image features to address the issue of sensing errors, which has received little attention in the literature. Our approach involves using image processing techniques to correct local maps before merging them into a global map. This paper is organized as follows: Section 2 reviews previous methods proposed to address the map merging problem. Section 3 presents a detailed problem statement. In Section 4, we describe our proposed method. Section 5 presents the simulation results and provides a discussion. Finally, Section 6 concludes the study and offers suggestions for future work.

2. Literature Review

Robot mapping utilizes different types of maps, and the methods for merging them may vary. The commonly used maps include occupancy grid maps, feature maps, and topological maps [6]. Occupancy grid maps are widely used and can be easily implemented in any environment [7]. Feature maps require less memory space, but their complex feature extraction process consumes computational resources. Topological maps depict a network structure and are useful for specific tasks such as navigation and path planning. In this study, we use occupancy grid maps as they have abundant applications in the literature.

Map merging methods can be classified as direct or indirect, with most recent methods using indirect approaches. Lee’s survey reviewed several merging methods and classified them into these two types based on their logical structure [8]. Direct methods involve establishing correlations between the sensing data of multiple robots to obtain their relative positions for merging. They can be based on direct observation, where robots use their sensors to observe cooperating robots and determine their relative positions [9], or on identifying common environmental objects, such as ceiling features, to establish relative positions [10]. However, direct methods have the limitation of requiring robots to be in the same field, which restricts their activity range and time allocation.

Indirect methods offer a solution to the limitations of direct methods but come at the expense of increased computational complexity. These methods treat maps as images or 2D matrices, allowing them to overcome temporal and spatial constraints and reduce computational requirements by discarding certain sensing data [8]. Indirect methods often extract and match features to establish correlations between the maps. Once these correlations are obtained, a relative transformation can be calculated by scaling, rotating, and translating the grid cells of the maps. Indirect methods effectively align maps with significant deviations in position or orientation, enhancing their reliability for map merging purposes.

The map images exhibit a diverse range of features, encompassing spectral features of line segments [11] and local extreme features [12]. Depending on the extracted features from these maps, different methods are employed. Carpin [11] proposed a non-iterative and fast algorithm that utilizes the Hough transform to determine the necessary rotation and translation parameters for map merging. This approach performs well in spatially structured environments, such as square rooms. However, it is important to note that this method has certain limitations, such as the requirement for local maps to have the same scale and a sufficient number of overlapping areas for successful merging. On the other hand, Ferrão [12] utilized the scale invariant feature transformation (SIFT) algorithm [13] to extract robust features from occupancy grid maps. This widely-used algorithm identifies extreme grid points in map images and performs feature matching. Although SIFT is commonly employed in computer vision and image stitching problems [14], it is worth noting that typical red-green-blue (RGB) images exhibit a higher diversity of texture, brightness, and color characteristics, enabling the extraction of more features compared to occupancy grid maps with pixel values represented by only three integers denoting free, unknown, and occupied areas. Furthermore, the presence of sensor errors and uncertainties in occupancy grid maps poses additional challenges for feature extraction and matching, leading to a decreased success rate of map merging solutions, as discussed in Jiang’s works [7,15].

In order to overcome the limitations in feature extraction and matching from occupancy grid maps, we present a novel image pre-processing operation aimed at reducing the impact of sensor errors and uncertainty. Our objective is to improve the effectiveness of feature extraction and ultimately address the challenge of map merging. Our proposed map merging approach involves two key steps. Firstly, we apply the image pre-processing technique to mitigate the impact of uncertainty and enhance the efficiency of feature extraction. This step aims to extract significant features from the occupancy grid maps. Secondly, we utilize an image stitching method to merge the pre-processed occupancy grid maps. By combining these steps, we aim to achieve a more effective and reliable map merging solution.

3. Basic Definitions on Map Merging

Consider multiple map images obtained by robots. Figure 1 illustrates the process defining the map merging problem in this work. After the maps are obtained, occupancy grid maps are generated based on the availability of each grid element. Two maps are associated by transforming them onto the same basis. The association results in a merging index, which determines whether the maps can be merged into one. If the index is acceptable, the maps are merged, and if not, the merging attempt fails. The rest of this section describes the occupancy grid map, the transformation between two maps, and the index used to evaluate the merged result in detail.

• Create occupancy grid maps: In the context of map merging, we define M as a 2D occupancy grid map with size $r\times c$ as shown in Equation (1). Each cell $m_{ij}$ contains a binary value that indicates whether the cell is occupied ($m_{ij}=1$) or free ($m_{ij}=0$).

  $$M=\left[\begin{array}{ccc}{m}_{11}& \cdots & m_{1c}\\ ⋮& \ddots & ⋮\\ m_{r1}& \cdots & m_{rc}.\end{array}\right]$$

  (1)
  Let $p(m_{ij}=1)$ be the probability that the cell is occupied. A grid cell could be classified into three conditions:

  $$m_{ij}=\left\{\begin{array}{cc}\hfill \mathrm{occupied},& \mathrm{if}p(m_{ij}=1)\ge p_{\mathrm{occupied}}.\hfill \\ \hfill \mathrm{free},& \mathrm{if}p(m_{ij}=1)\le p_{\mathrm{free}}.\hfill \\ \hfill \mathrm{unknown},& \mathrm{else},\hfill \end{array}\right.$$

  where $p_{\mathrm{occupied}}$ and $p_{\mathrm{free}}$ are two predefined thresholds.
  In addition to a binary value, if M represents an image, each cell element in Equation (1) will have a pixel value that can also be used to represent the cell conditions.

  $$m_{ij}=\left\{\begin{array}{cc}\hfill 0& \mathrm{if}\mathrm{cell}\mathrm{type}\mathrm{is}“\mathrm{occupied}.”\hfill \\ \hfill 255& \mathrm{if}\mathrm{cell}\mathrm{type}\mathrm{is}“\mathrm{free}.”\hfill \\ \hfill 128& \mathrm{if}\mathrm{cell}\mathrm{type}\mathrm{is}“\mathrm{unknown}.”\hfill \end{array}\right.$$
• Associate maps for merging: Consider two maps $M_1$ and $M_2$ with different coordinates, as shown in Figure 2. The transformation matrix between them is denoted as $T^{12}(s,\theta ,t_x,t_y)$, as defined in Equation (2).

  $$T^{12}(s,\theta ,t_x,t_y)=\left[\begin{array}{ccc}s·cos\theta & -s·sin\theta & t_x\\ s·sin\theta & s·cos\theta & t_y\\ 0& 0& 1\end{array}.\right]$$

  (2)
  In Equation (2), s represents the scale difference between the coordinates of the maps, $\theta$ represents the angle difference, and $t_x$ and $t_y$ represent the translation difference. Using this transformation, map $M_1$ can be transformed into $M_1^{\prime }$ and overlapped with parts of map $M_2$, as illustrated in Figure 2.
• Determine acceptance of the merged result: In Equation (3), the acceptance index [11], denoted as $\omega (M_1,M_2)$, serves as a means to evaluate the merging result when the transformed map $M_1^{\prime }$ is overlaid onto map $M_2$. The acceptance index considers two types of grids: agreement and disagreement. Agreement represents correctly paired grid cells between $M_1$ and $M_2$, indicating that both maps have the same type (occupied-occupied or free-free) in the overlapping areas. On the other hand, disagreement signifies incorrect pairings, indicating that the two maps have different types (occupied-free) in those regions. Due to its higher uncertainty, the unknown type is not discussed here.
  When the transformation $T^{12}$ accurately aligns the grids of the transferred map $M_1^{\prime }$ with those of map $M_2$ in the overlapping regions, the number of agreement grids increases while the number of disagreement grids decreases. As a result, the acceptance index $\omega (M_1^{\prime },M_2)$ approaches 1. Conversely, if the transformation $T^{12}$ is incorrect, the overlapping areas will contain numerous disagree grids, leading to a smaller or even 0 acceptance index $\omega (M_1^{\prime },M_2)$. Thus, the acceptance index effectively determines the correctness of the map transformation and reflects the quality of the merging result. In our study, we define an index threshold, as shown in Equation (4), to judge the accuracy of the map transformation.

  $$\begin{array}{cc}\hfill \omega & (M_1,M_2)=\left\{\begin{array}{ccc}\hfill 0,& \hfill & \hfill \mathrm{if}agr(M_1,M_2)=0\\ \hfill \frac{agr(M_1,M_2)}{agr(M_1,M_2)+dis(M_1,M_2)},& \hfill & \hfill \mathrm{if}agr(M_1,M_2)\ne 0\end{array}\right.\hfill \end{array}$$

  (3)

  $$\omega (M_1^{\prime },M_2)\ge {\omega }_{\mathrm{thres}}.$$

  (4)

With an acceptable merged result, we can successfully combine multiple images from robots into a single map. In this study, we use a threshold of ${\omega }_{\mathrm{thres}}=0.95$ and test it in various scenarios. In the next section, we will present our method that builds on the concept flow shown in Figure 1 to handle images that may contain sensing errors in reality.

4. Map Merging Method

In this section, we present our novel image pre-processing method, which effectively mitigates the impact of sensing errors and uncertainties in maps prior to their stitching. The proposed pre-processing operation focuses on eliminating noise and distortion issues in the occupancy grid map through several image processing steps. By applying these steps, we ensure that the existing image-stitching method is unaffected by noise and can seamlessly merge the maps. In what follows, we will describe the existing process of imaging stitching in Section 4.1, and our proposed operation in Section 4.2.

4.1. Existing Method in Image Stitching

We utilize an image-stitching method [16] proposed by Mills for our stitching operation. However, when combining two occupancy grid maps, it is essential to consider their probabilistic information, which differs from regular images. Therefore, we incorporate an entropy filter [17] after calculating the relative transformation using Mills’ method. The complete operation is illustrated in Figure 3. In this operation, two maps $M_1$ and $M_2$ are merged, as shown in Figure 2. First, the scale-invariant feature transform (SIFT) [18,19] is used to extract features from the occupancy grid maps. SIFT applies Gaussian filters to build the scale space of the image and then finds the extreme points that can be used as features under the difference of Gaussians (DoG) in the scale space. The gradient vector around each feature is calculated, and a descriptor with the histogram of the gradient vector is obtained. Therefore, SIFT features are invariant to scale, rotation, and brightness [18,19]. The SIFT algorithm obtains features $P=\{p_1,\dots ,p_n\}$ from $M_1$ and features $Q=\{q_1,\dots ,q_m\}$ from $M_2$. These features are matched as pairs $\{p_i=(x_{pi},y_{pi}),q_j=(x_{qj},y_{qj})\}$, $i=1,\dots ,n$ and $j=1,\dots ,m$ by using the Euclidean distance to find the feature $q_j$ in Q that is closest to the feature $p_i$ in P.

Matching pairs are further processed by applying the random sample consensus algorithm (RANSAC) [20] to eliminate outliers and estimate a relative transformation between the maps. RANSAC effectively exclude outliers and calculate a model that fits the input data by randomly sampling the data. If the inliers (that conform to the real data model) are selected, the correct model can be calculated and outliers can be effectively excluded. Specifically, in our method, RANSAC includes the following three steps.

Step 1

Define a ‘Guess’ model.

$$\begin{array}{cc}\hfill T^{12,\mathrm{guess}}& (s,\theta ,t_x,t_y)=\left[\begin{array}{ccc}scos\theta & -ssin\theta & t_x\\ ssin\theta & scos\theta & t_y\\ 0& 0& 1.\end{array}\right]\hfill \end{array}$$

(5)

Randomly sample two matched pairs of SIFT features from the input data.

$$\begin{array}{c}\hfill \{p_a=(x_{pa},y_{pa}),q_a=(x_{qa},y_{qa})\}\\ \hfill \{p_b=(x_{pb},y_{pb}),q_b=(x_{qb},y_{qb})\}.\end{array}$$

(6)

Calculate the four variables in the model $T^{12,\mathrm{guess}}$ with the following equations.

$$s=\frac{|\sqrt{{(x_{qb}-x_{qa})}^2+{(y_{qb}-y_{qa})}^2}|}{|\sqrt{{(x_{pb}-x_{pa})}^2+{(y_{pb}-y_{pa})}^2}|}$$

(7)

$$\begin{array}{c}\hfill \theta =arctan\left(\frac{y_{qb}-y_{qa}}{x_{qb}-x_{qa}}\right)-arctan\left(\frac{y_{qb}-y_{pa}}{x_{pb}-x_{pa}}\right)\end{array}$$

(8)

$$\left[\begin{array}{c}{t}_x\\ t_y\end{array}\right]=\left[\begin{array}{c}{x}_{qa}\\ y_{qa}\end{array}\right]-s\left[\begin{array}{cc}cos\theta & -sin\theta \\ sin\theta & cos\theta \end{array}\right]\left[\begin{array}{c}{x}_{pa}\\ y_{pa}.\end{array}\right]$$

(9)

Step 2

Test other feature pairs ${\{p_i=(x_{pi},y_{pi}),q_i=(x_{qi},y_{qi})\}}_{i=1,2,\cdots ,N}$ and transform the pairs by Equations (5)–(9) as follows.

$$\begin{array}{cc}\hfill v_{qi}& =\left[\begin{array}{c}{x}_{qi}\\ y_{qi}\end{array}\right]\hfill \\ \hfill v_{pi}^{\prime }& =s\left[\begin{array}{cc}cos\theta & -sin\theta \\ sin\theta & cos\theta \end{array}\right]\left[\begin{array}{c}{x}_{pi}\\ y_{pi}\end{array}\right]+\left[\begin{array}{c}{t}_x\\ t_y.\end{array}\right]\hfill \end{array}$$

(10)

If the transformed pair satisfies

$$|v_{qi}-v_{pi}^{\prime }|<ϵ,$$

(11)

then the pair conforms to the guess model. $ϵ$ is a parameter for selecting data. The number of pairs that match the guess model is recorded. If that number exceeds the highest number previously recorded, the current model is updated as the ‘best’ model $T^{12,\mathrm{best}}$.

Step 3

Repeat Steps 1 and 2 until the maximum number of iterations is reached and output the best model $T^{12,\mathrm{best}}$ as $T_{\mathrm{RANSAC}}$.

Once we have obtained $T_{\mathrm{RANSAC}}$, the two maps can be overlapped through the transformation. The quality of the transformation is then evaluated using an acceptance index and a threshold ${\omega }_{\mathrm{thres}}$. If the index exceeds the threshold (in this study, we use 0.95), an entropy filter [17] is applied to adjust the map probabilities and combine the maps into a global map. A higher entropy value corresponds to a higher level of uncertainty. Thus, the entropy filter selects probabilities with low uncertainty [17].

Let us assume that the occupied probabilities of a grid $m_{ij}=M(x,y)$ in the overlapping areas of two maps $M_1$ and $M_2$ are $p(M_1(x,y)=1)$ and $p(M_2(x,y)=1)$, respectively. We can calculate the combined probability $p(M_{\mathrm{combined}}(x,y)=1)$ by updating the occupancy grid mapping in terms of the log probabilities [21]. This can be done using Equation (12). The corresponding entropy is calculated using Equation (13), and the three entropy values for $p(M_1(x,y)=1)$, $p(M_2(x,y)=1)$, and $p(M_{\mathrm{combined}}(x,y)=1)$ are listed in Equation (14).

$$\begin{array}{cc}\hfill p(M_{\mathrm{combined}}(x,y)=1)& =\frac{Odd(M_{\mathrm{combined}}(x,y)=1)}{1+Odd(M_{\mathrm{combined}}(x,y)=1)}\hfill \\ \hfill logOdd(M_{\mathrm{combined}}(x,y)=1)& =logOdd(M_1(x,y)=1)+logOdd\left(M_2(x,y)\right)\hfill \\ \hfill Odd(M_1(x,y)=1)& =\frac{p(M_1(x,y)=1)}{1-p(M_1(x,y)=1)}\hfill \\ \hfill Odd(M_2(x,y)=1)& =\frac{p(M_2(x,y)=1)}{1-p(M_2(x,y)=1)}\hfill \end{array}$$

(12)

$$\begin{array}{c}\hfill H\left(M\right(x,y)=1)=-p\left(M\right(x,y)=1)logp\left(M\right(x,y)=1)-(1-p(M(x,y)=1\left)\right)(log(1-p(M(x,y)=1\left)\right))\end{array}$$

(13)

$$\begin{array}{cc}\hfill H(M_1(x,y)=1)=& -p(M_1(x,y)=1)logp(M_1(x,y)=1)\hfill \\ \hfill & -(1-p(M_1(x,y)=1))(log(1-p(M_1(x,y)=1)))\hfill \\ \hfill H(M_2(x,y)=1)=& -p(M_2(x,y)=1)logp(M_2(x,y)=1)\hfill \\ & -(1-p(M_2(x,y)=1))(log(1-p(M_2(x,y)=1)))\hfill \\ \hfill H(M_{\mathrm{combined}}(x,y)=1)=& -p(M_{\mathrm{combined}}=1)logp(M_{\mathrm{combined}}=1)\hfill \\ & -(1-p(M_{\mathrm{combined}}=1))(log(1-p(M_{\mathrm{combined}}=1))).\hfill \end{array}$$

(14)

In overlapping areas, the occupancy probability of grids from different maps are considered together and a grid with low uncertainty is selected as the output.

4.2. Proposed Image Pre-Processing Operation

Failed map merging can occur due to uncertainty in the map and distortion caused by sensing errors. To tackle this issue, we applied image correction to the maps, which eliminates uncertainty and distortion, and improves the merge result after feature extraction and matching. Our image correction process involves five steps, which are illustrated in the image correction block of Figure 4.

4.2.1. Image Correction Process

To extract features with matching relationships from occupancy grid maps, it is important to avoid using pixel values of unknown regions with high uncertainty. In order to get around this, we use a method called binarization that separates the map’s regions into “occupied” and “non-occupied” sections. The “occupied” section includes the original occupied area, while the “non-occupied” section consists of the original free and unknown areas.

In binarization, each grid is classified using a threshold value $B_{\mathrm{thres}}$. If the pixel value of a grid $M(x,y)$ is less than $B_{\mathrm{thres}}$, it is classified as occupied and assigned a new value of 255. If $M(x,y)$ is greater than or equal to $B_{\mathrm{thres}}$, it is classified as non-occupied, and assigned a new value of 0, as shown in Equation (15). After binarization, the map only contains two types of grids, as illustrated in Figure 5.

$$\begin{array}{c}\hfill M(x,y)=\left\{\begin{array}{cccc}\hfill 255& \left(\mathrm{Occupied}\right)\hfill & \hfill & \mathrm{if}M(x,y)<B_{\mathrm{thres}}\hfill \\ \hfill 0& \left(\mathrm{Non-occupied}\right)\hfill & \hfill & \mathrm{else}.\hfill \end{array}\right.\end{array}$$

(15)

Figure 6b illustrates what can happen when barriers like linear walls on a map are not parallel or perpendicular to the coordinate axes: they can take the form of stairs. We rotate the maps to align linear features with the coordinate axes in order to improve feature extraction. The Radon transform, which is applied to the map, is used to determine the rotation’s angle (ref. [22]). By performing line integration on f, the Radon transform turns the map image into a two-dimensional linear function $Rf$, $M\stackrel{f}{\to }Rf$, as shown in Equation (16).

$$\begin{array}{c}\hfill Rf=R(s,\alpha )=\int {\int }_{R^2}f(x,y)\delta (xcos\left(\alpha \right)+ysin\left(\alpha \right)-s)dxdy.\end{array}$$

(16)

The longest line in the original map image corresponds to the greatest value of the linear function $Rf$ following the Radon transform on the occupancy grid map, as shown in Figure 7. The rotation angle for the map can be calculated as the angle between this longest line and the coordinate axis. However, as illustrated in Figure 6c, some grids’ pixel values may no longer be equal to 0 or 255 as a result of rotation’s interpolation calculations. Therefore, as shown in Figure 6d, we apply extra binarization to make sure that the map image only comprises occupied and unoccupied areas. We use methods from image morphology [23] to fix instances where the map image has uneven burrs and imperfections as a result of prior operations.

The dilation operation from image morphology moves the structuring element S across each grid $M(x,y)$ of the image M when given an image M and S. When S intersects with the grids surrounding $M(x,y)$, the value of $M(x,y)$ is set to 255; otherwise, it is set to 0. While the foreground regions of image M are expanded by the dilation operation, the erosion operation, reduces the foreground regions of the image M. Erosion process sets the value of $M(x,y)$ to 255 only if S belongs to grids surrounding $M(x,y)$; otherwise, it is set to 0.

To improve the image, we performed a closing operation as shown in Figure 8, which applies dilation followed by erosion. This operation expands the foreground regions (i.e., the occupied image areas), fills gaps and defects by dilation, and then restores the area to its original size by erosion. The closing operation can be represented mathematically by Equation (17), where the ⊕, ⊖, and • operators represent dilation, erosion, and closing, respectively.

$$\begin{array}{cc}\hfill M\oplus S& =\left\{z\right|\widehat{{\left(S\right)}_z}\cap M\ne \varphi \}\hfill \\ \hfill M\ominus S& =\left\{z\right|\widehat{{\left(S\right)}_z}\subseteq M\}\hfill \\ \hfill M•S& =(M\oplus S)\ominus S.\hfill \end{array}$$

(17)

After performing the closing operation, the preliminary correction of the lines in the occupied areas has been completed. However, to further improve the quality of the lines and increase their straightness, remove defects, and normalize their widths, we applied another correction method before using the feature extraction algorithm. This correction method consists of two parts—the first part involves sampling interest points from the current lines as depicted in Figure 9 and the second part involves redrawing new lines based on these interest points as shown in Figure 10.

The extraction of interest points from the lines is performed by Algorithm 1, which creates a new image $M^{\prime }$ with all interest points. To achieve this, the structuring element S is moved across the original image M in a similar way to previous image morphology operations. If S intersects with the grids surrounding $M(x,y)$, a marker value of 255 is assigned to the grid $M^{\prime }(x,y)$, and the original grids surrounding $M(x,y)$ are deleted by setting their values to 0. After the movement of the structuring element S is completed on each grid of M, all interest points are identified and marked on image $M^{\prime }$.

Algorithm 1 Extraction of interest points of map images.

Input: M: map image; S: structuring element; Output: $M^{\prime }$: map image with interest points 1: r, c ← size(M) 2: $M^{\prime }\leftarrow$ new matrix(r,c) 3: for $x=1$ to r do 4:       for $y=1$ to c do 5:             $M_s\leftarrow$ surrounding of $M(x,y)$ 6:             if $S\subseteq M_s$ then 7:                   $M^{\prime }(x,y)\leftarrow 255$ 8:                   surrounding of $M(x,y)\leftarrow 0$ 9:             end if 10:       end for 11: end for 12: return  $M^{\prime }$

Algorithm 2 is the final step of our proposed image correction process, presented in Figure 4, which aims to correct the line segments in the occupied area. The algorithm takes the image $M^{\prime }$ with interest points and the image M after closing as inputs, along with the distance parameter L, to ensure that the new line segments resemble the original data with some uncertainty and are parallel to the coordinates. The algorithm finds points on the map that form line segments and corrects them using the following five steps:

Step 1

Find the reference point $M^{\prime }(x_i,y_i)$ closest to the origin using the Manhattan distance in Equation (18).

$$(x_i,y_i)=arg\underset{x,y}{min}|x-x_o|+|y-y_0|;$$

(18)

Step 2

Find the target point $M^{\prime }(x_t,y_t)$ closest to the reference point as presented in Equation (19).

$$(x_t,y_t)=arg\underset{x,y}{min}|x-x_i|+|y-y_i|;$$

(19)

Step 3

Connect $M^{\prime }(x_i,y_i)$ and $M^{\prime }(x_t,y_t)$ with a line, and all grids ($M^{\prime }(x_1,y_1),\cdots ,M^{\prime }(x_n,y_n)$) that the line intersects with are recorded. The position of these n grids is then used to calculate the ratio $R_{it}$ of occupied areas on the original image M, as shown in Equation (20).

$$R_{it}=\frac{{\sum }_{i=1}^{n}M(x_i,y_i)}{255\times n}.$$

(20)

The line connecting $M^{\prime }(x_i,y_i)$ and $M^{\prime }(x_t,y_t)$ was present in the original image and can be corrected if the calculated ratio $R_{it}$ exceeds the defined threshold $R_{thres}$, $R_{it}>R_{thres}$. If not, the algorithm picks a new target point and repeats Step 2 in that case.

Step 4

Calculate a vector $v_{it}$ as Equation (21) if the line between $M^{\prime }(x_i,y_i)$ and $M^{\prime }(x_t,y_t)$ can be corrected.

$$v_{it}=(x_t-x_i,y_t-y_i).$$

(21)

According to $v_{it}$, we then take $M^{\prime }(x_i,y_i)$ as a reference and translate $M^{\prime }(x_t,y_t)$. The translation follows the following rule.

(1)

If $|x_t-x_i| < |y_t-y_i|$, translate grid $M^{\prime }(x_t,y_t)$ in x-direction. For example, moves $M^{\prime }(x_t,y_t)$ to $M^{\prime }(x_i,y_t)$;

(2)

If $|x_t-x_i| > |y_t-y_i|$, translate grid $M^{\prime }(x_t,y_t)$ in y-direction. For example, moves $M^{\prime }(x_t,y_t)$ to $M^{\prime }(x_t,y_i)$;

(3)

If $|x_t-x_i| = |y_t-y_i|$, add an additional grid as a target point according to the previous direction and perform the translation again. For example, if the previous direction is:

• x-direction, then add a new grid $M^{\prime }(x_i,y_t)$ as a new target point;
• y-direction, then add a new grid $M^{\prime }(x_t,y_i)$ as a new target point;
• none (no previous direction), then abandon the translation and select another reference and target points.

In the aforementioned translation, shifts or additions of grids are performed. To ensure that the correction does not differ excessively from the original image, we again connect the grid $M^{\prime }(x_i,y_i)$ and the shifted (or added) grid $M^{\prime }(x_i,y_t)$ (or $M^{\prime }(x_t,y_i)$) on a line, record all n grids that the line passes through, and calculate the ratio $R_{i{t}^{\prime }}$ of occupied areas at the same positions on original image M. This calculation is the same as Equation (20). If ratio $R_{i{t}^{\prime }}$ is greater than $R_{thres}$, the correction is accepted and the occupied grids on the connected line will be stored to corrected image $M_{\mathrm{correct}}$. If not, the correction is rejected and the original information from image M will be stored to corrected image $M_{\mathrm{correct}}$.

Step 5

According to the result of Step 4, the correction point $M^{\prime }(x_i,y_t)$ (or $M^{\prime }(x_t,y_i)$) and can be used as a new reference point to find a new adjacent target point. Steps 3 to 5 are repeated until the correction is completed. Once the correction is completed, the corrected image, $M_{\mathrm{correct}}$, is output as the result.

Algorithm 2 Corrected occupancy grid maps with interest points.

Input: M: map image; $M^{\prime }$: map image with interest points; L: distance parameter; Output: $M_{\mathrm{correct}}$: map image after correction 1: r, c ← size(M) 2: $M_{\mathrm{correct}}\leftarrow$ new matrix(r,c) 3: direction ← none 4: $p_{start}^{\prime }\leftarrow$ search for the nearest point of interest to the origin on $M^{\prime }$ 5: while interest points exist do 6:         $p_{target}^{\prime }\leftarrow$ search for the nearest point of interest to $p_{start}^{\prime }$ on $M^{\prime }$ within distance L 7:         if $p_{target}^{\prime }$ exists then 8:               $p_{start},p_{target}\leftarrow$ find points with the same position of $p_{start}^{\prime }$ and $p_{target}^{\prime }$ on M 9:               $l_{st}\leftarrow$ make a line from $p_{start}$ to $p_{target}$ 10:             if all points on $l_{st}$ = 255 then 11:                   $(dx,dy)\leftarrow (p_{target}.x-p_{start}.x,p_{target}.y-p_{start}.y)$ 12:                   if  $\left|dx\right|>\left|dy\right|$ then 13:                         $p_{correct}\leftarrow (p_{start}.x,p_{target}.y)$ 14:                         $l_{sc}\leftarrow$ make a line from $p_{start}$ to $p_{correct}$ 15:                         if all points on $l_{sc}$ = 255 then 16:                               $M_{\mathrm{correct}}\leftarrow$ set the value of all points on $l_{sc}$ to 255 17:                               direction ← toward_y 18:                               $M^{\prime }\leftarrow$ delete $p_{start}^{\prime }$ and $p_{target}^{\prime }$ on $M^{\prime }$ 19:                               $p_{start}^{\prime }\leftarrow p_{correct}$ 20:                         else 21:                               $M_{\mathrm{correct}}\leftarrow$ set the value of all points on $l_{st}$ to 255 22:                               direction ← None 23:                         end if 24:                   else if  $\left|dx\right|<\left|dy\right|$ then 25:                         $p_{correct}\leftarrow (p_{target}.x,p_{start}.y)$ 26:                         $l_{sc}\leftarrow$ make a line from $p_{start}$ to $p_{correct}$ 27:                         if all points on $l_{sc}$ = 255 then 28:                               $M_{\mathrm{correct}}\leftarrow$ set the value of all points on $l_{sc}$ to 255 29:                               direction ← toward_x 30:                               $M^{\prime }\leftarrow$ delete $p_{start}^{\prime }$ and $p_{target}^{\prime }$ on $M^{\prime }$ 31:                               $p_{start}^{\prime }\leftarrow p_{correct}$ 32:                         else 33:                               $M_{\mathrm{correct}}\leftarrow$ set the value of all points on $l_{st}$ to 255 34:                               direction ← None 35:                         end if 36:                   else 37:                         if direction = toward_y then 38:                               $p_{correct}\leftarrow (p_{start}.x,p_{target}.y)$ 39:                               $l_{sc}\leftarrow$ make a line from $p_{start}$ to $p_{correct}$ 40:                               if all points on $l_{sc}$ = 255 then 41:                                     $M_{\mathrm{correct}}\leftarrow$ set the value of all points on $l_{sc}$ to 255 42:                                     direction ← toward_y 43:                                     $M^{\prime }\leftarrow$ delete $p_{start}^{\prime }$ on $M^{\prime }$ 44:                                     $p_{start}^{\prime }\leftarrow p_{correct}$ 45:                               else 46:                                     $M_{\mathrm{correct}}\leftarrow$ set the value of all points on $l_{st}$ to 255 47:                                     direction ← None 48:                               end if 49:                         else if direction = toward_x then 50:                               $p_{correct}\leftarrow (p_{target}.x,p_{start}.y)$ 51:                               $l_{sc}\leftarrow$ make a line from $p_{start}$ to $p_{correct}$ 52:                               if all points on $l_{sc}$ = 255 then 53:                                     $M_{\mathrm{correct}}\leftarrow$ set the value of all points on $l_{sc}$ to 255 54:                                     direction ← toward_x 55:                                     $M^{\prime }\leftarrow$ delete $p_{start}^{\prime }$ on $M^{\prime }$ 56:                                     $p_{start}^{\prime }\leftarrow p_{correct}$ 57:                               else 58:                                     $M_{\mathrm{correct}}\leftarrow$ set the value of all points on $l_{st}$ to 255 59:                                     direction ← None 60:                               end if 61:                         else 62:                               $M^{\prime }\leftarrow$ delete $p_{start}^{\prime }$ on $M^{\prime }$ 63:                               $p_{start}^{\prime }\leftarrow$ for the nearest point of interest to the origin on $A^{\prime }$ 64:                         end if 65:                   end if 66:             end if 67:       else 68:             $M^{\prime }\leftarrow$ delete $p_{start}^{\prime }$ on $M^{\prime }$ 69:             $p_{start}^{\prime }\leftarrow$ for the nearest point of interest to the origin on $M^{\prime }$ 70:       end if 71: end while 72: return  $M_{\mathrm{correct}}$

4.2.2. Image Stitching with ICP

After the image correction, the map images only contain line segments with consistent width that are parallel to the coordinate axes. Features from these maps can then be extracted to enable image stitching. However, feature extraction involves the extraction of the extreme points in the scale space of the image, a certain occupancy proportion is required to effectively detect the extreme points in the scale space. In our experiments, an additional dilation operation might be required before extracting features when we have insufficient proportion of occupied areas in the map. In addition, most extracted features are roughly located at the corner or the end of the line segments where their neighborhoods are overly similar, causing the feature descriptors to be similar and preventing matching using the Euclidean distance. Instead, we used brute-force matching, as in Equation (22).

$$\{\{p_1,q_1\},\dots ,\{p_1,q_m\},\dots ,\{p_n,q_1\},\dots ,\{p_n,q_m\}\},$$

(22)

where ${\left\{p_i\right\}}_{i=1,\dots ,n}$ and ${\left\{q_j\right\}}_{j=1,\dots ,m}$ are features from maps $M_1$ and $M_2$, respectively.

When using RANSAC to merge corrected map images, slight deviations may occur. Therefore, we first performed an initial alignment of the maps using the RANSAC model $T_{\mathrm{RANSAC}}$ and then applied the Iterative Closest Point (ICP) algorithm to achieve a more precise alignment.

ICP is a point cloud matching and alignment method that utilizes the least square difference to calculate a spatial transformation and minimize the distance between two input point clouds to achieve alignment [24,25]. In our approach, we use the occupied grids ($P_{\mathrm{occ}}=p_{\mathrm{occ},1},\dots ,p_{\mathrm{occ},n}$, $Q_{\mathrm{occ}}=q_{\mathrm{occ},1},\dots ,q_{\mathrm{occ},m}$) from the original maps ($M_1$, $M_2$) as point clouds and applied the ICP algorithm to obtain a model for the second alignment. To ensure good initial conditions, we transformed the point clouds, $P_{\mathrm{occ}}$, as described in Equation (23).

$$P_{\mathrm{occ}}^{\prime }=T_{\mathrm{RANSAC}}\times P_{\mathrm{occ}}=\{p_{\mathrm{occ},1}^{\prime },\dots ,p_{\mathrm{occ},n}^{\prime }\}.$$

(23)

During each iteration of the ICP algorithm, every point in the set $P_{\mathrm{occ}}^{\prime }$ is matched with its corresponding point in the point set $Q_{\mathrm{occ}}$ to establish a correspondence relationship of ${p_{\mathrm{occ},i}^{\prime },q_{\mathrm{occ},i}}_{i=1,\dots ,N_p}$. Using this correspondence relationship, the ICP model for the second alignment can be obtained by solving Equation (24).

$$\begin{array}{cc}\hfill R^{*},t^{*}& =arg\underset{R,t}{min}\sum _{i=1}^{N_p}\left|\right|q_{\mathrm{occ},i}-(R{p}_{\mathrm{occ},i}^{\prime }+t){\left|\right|}^2\hfill \\ \hfill T_{\mathrm{ICP}}& =\left[\begin{array}{c}\begin{array}{cc}{R}^{*}& t^{*}\\ \mathbf{0}& 1\\ .\end{array}\end{array}\right]\hfill \end{array}$$

(24)

$R^{*}$ and $t^{*}$ are rotation and translation matrices, respectively. With $T_{\mathrm{RANSAC}}$ and $T_{\mathrm{ICP}}$, a final model is obtained by combining these two models:

$$T^{12}(s,\theta ,t_x,t_y)=T_{\mathrm{ICP}}\times T_{\mathrm{RANSAC}}.$$

(25)

5. Results & Discussion

In this section, we provide a comprehensive performance comparison between the utilization of our proposed image pre-processing method and the absence of such pre-processing in two distinct cases, as detailed and analyzed in Section 5.1 and Section 5.2, respectively. Additionally, we present the corresponding merging results for each case to support our findings. Additionally, both of these cases are carried out within the Robotic Operating System (ROS) [26] and the Gazebo virtual world [27]. Moreover, we utilize various open-source algorithms, including OpenCV [28] for feature extraction and Gmapping-based SLAM for mapping the environment.

5.1. Scenario 1: Performance of Our Proposed Method

A simulation environment was created as shown in Figure 11. In this environment, we created four local maps with different parameters, as illustrated in Figure 12, and the specific parameters are listed in

Table 1. Parameters used in mapping.

Local Maps	Resolution (Number of Grids/Meter)	Lidar Sensing Error (Standard Deviation, Meter)
Map 1 (Figure 12a)	20	0.005
Map 2 (Figure 12b)	20	0.005
Map 3 (Figure 12c)	10	0.005
Map 4 (Figure 12d)	20	0.05

.

In Figure 12, the displayed maps exhibit variations in coordinates due to differences in rotation and translation resulting from mapping from various origin positions. Additionally, we manipulated the resolutions of the maps to create variations in their scales. Moreover, we investigated the impact of sensing errors on the map quality, as depicted in Figure 13. In Figure 13a, the robot is positioned at the center, and the blue area represents the LiDAR scanning range. The long object on the right depicts the wall being measured, with the red line indicating the reference distance of 4 m from the robot to the wall. We adjusted the LiDAR signal with different sensor errors, represented by standard deviation, and mapped the wall into occupancy grid maps. The effects of sensing errors can be observed in Figure 13b–e. Mapping with a lower standard deviation of LiDAR measurement (Figure 13) resulted in a more accurate representation of the wall (Figure 13c). Conversely, mapping with a higher standard deviation of LiDAR measurement (Figure 13d) led to a distorted representation of the wall (Figure 13e). These findings highlight the importance of low standard deviation in LiDAR measurements for constructing accurate maps.

Table 2. Sets of the local maps to be merged.

	Input Map 1	Input Map 2	Relation of Variables
Set 1	Local map 1 (Figure 12a)	Local map 2 (Figure 12b)	Rotation Translation
Set 2	Local map 1 (Figure 12a)	Local map 3 (Figure 12c)	Rotation Translation Scale
Set 3	Local map 1 (Figure 12a)	Local map 4 (Figure 12d)	Rotation Translation Sensing Error

outlines the relationships between the local maps to be merged, encompassing three distinct data sets. The first set involves merging maps with differing coordinates, while the second set introduces an additional scale difference. The third set focuses on the impact of sensing errors. In the subsequent sections, we will discuss the merging outcomes with and without the utilization of our proposed method.

5.1.1. Results without Image Pre-Process—Mills’ Method [16]

The merging results for the three data sets without the image pre-process (i.e., using Figure 3) are shown in Figure 14, and the corresponding acceptance indices are listed in

Table 3. Corresponding acceptance indices for merged sets in test case 1 by using the existing method.

	Set 1	Set 2	Set 3
Acceptance Index	${\omega }_1=0.966292$	${\omega }_2=0.935030$	${\omega }_3=0.903026$
Human judgment	Success	Failure	Failure

.

Only data set 1 (Figure 14a) produces a matching result with the environment. To evaluate the merging performance, Table 3 shows the acceptance indices for the three data sets. The results suggest that a successful merging threshold should fall between 0.966 and 0.935. That is the reason we set the merging threshold, ${\omega }_{\mathrm{thres}}$, to 0.95 throughout the study.

Mills’ method showed ineffective results when merging these occupancy grid maps, mainly due to feature matching errors. During feature extraction, extreme points are calculated based on the value and distribution of pixels, leading to the inclusion of unknown areas with high uncertainty and generating unnecessary features without matching relationships between maps. This issue is highlighted in Figure 15 where red circles indicate these errors. Furthermore, sensing errors can also result in the extraction of features without matching relationships. A smaller number of features were extracted in the occupied area of Figure 15a, while a larger number of features were extracted in the occupied area of Figure 15b (marked in green circles). These feature extraction errors lead to ineffective map merging, as evidenced by our results.

Consequently, inaccurate matching relations can lead to errors in calculating the relative transformation and ultimately result in unsuccessful merges. However, our proposed method provides an effective solution to address this issue. We performed tests on the same datasets, applying our image pre-processing technique before utilizing Mills’ method. The results of these tests will be thoroughly examined and discussed in Section 5.1.2.

5.1.2. Results with Image Pre-Process

Figure 16 presents the results of feature extraction after applying our proposed image pre-processing. It is notable that features are selectively extracted only at the corners of occupied areas. Unlike Figure 15, this approach effectively excludes the influence of uncertainties and sensing errors on feature extraction, making it more efficient for true matches of features and the subsequent stitching method. Furthermore, Figure 17 displays the merging results for the three datasets obtained by applying our image pre-processing operation before utilizing Mills’ method (as depicted in Figure 4). The corresponding acceptance indices are presented in

Table 4. Corresponding acceptance indices for merged sets in test case 1 by using the reference method.

	Set 1	Set 2	Set 3
Acceptance Index	${\omega }_1^{\prime }=0.992072$	${\omega }_2^{\prime }=0.989327$	${\omega }_3^{\prime }=0.979052$
Human judgment	Success	Success	Success

. The results demonstrate a close resemblance to the characteristics of the simulated environment. Moreover, acceptance indices surpassing the predetermined threshold (${\omega }_{\mathrm{thres}}=0.95$) signify successful merges. These findings validate that our proposed operation effectively eliminates the impact of sensing errors and uncertainties in occupancy grid maps, resulting in accurate feature matching and correct map merging. Furthermore,

Table 5. Comparison between existing method and our method.

	Existing Method	Our Method	Increased Performance
Set 1	${\omega }_1=0.966292$	${\omega }_1^{\prime }=0.992072$	$\frac{{\omega }_1^{\prime }-{\omega }_1}{{\omega }_1}=+2.67\%$
Set 2	${\omega }_2=0.935030$	${\omega }_2^{\prime }=0.989327$	$\frac{{\omega }_2^{\prime }-{\omega }_2}{{\omega }_2}=+5.81\%$
Set 3	${\omega }_3=0.903026$	${\omega }_3^{\prime }=0.979052$	$\frac{{\omega }_3^{\prime }-{\omega }_3}{{\omega }_3}=+8.42\%$

illustrates the superior performance of the stitching method when coupled with our image pre-processing operation compared to the method without such an operation.

Additionally, to demonstrate that our proposed method can merge multiple maps simultaneously, we created tests in various environments, which will be discussed in greater detail in the subsequent section.

5.2. Scenario 2: Merging Multiple Maps

To assess the capability of our method to merge multiple maps simultaneously, we conducted a test in our research office situated in Yonglin Biomedical Engineering Hall, Taipei, Taiwan. In order to expedite the experiment, we simulated the environment using floor plans that faithfully replicated the real-world setting, as shown in Figure 18. Within this simulated environment, we created three distinct local maps, as depicted in Figure 19, and the corresponding parameters can be found in

Table 6. Parameters for local maps in test case 2.

Local Maps	Resolution (Number of Grids/Meter)	Lidar Sensing Error (Standard Deviation, Meter)
Map 1 (Figure 19a)	20	0.005
Map 2 (Figure 19b)	20	0.005
Map 3 (Figure 19c)	20	0.005

and

Table 7. Sets of the local maps to be merged.

	Input Map 1	Input Map 2	Relation of Variables
Set 1	Local map 1 (Figure 19a)	Local map 2 (Figure 19b)	Rotation Translation
Set 2	Local map 1 (Figure 19a)	Local map 3 (Figure 19c)	Rotation Translation

.

Results with the Proposed Method

The merging results for these two sets are illustrated in Figure 20, and it can be observed that most of the overlapping grids have accurate pairings (green areas), with only few grids having incorrect ones (red areas). These red grids are caused by minor errors due to image rotation. The results indicate that the merged maps are consistent with the upper and lower parts of the environment. Additionally, the corresponding acceptance indices are listed in

Table 8. Corresponding acceptance indices for merged sets in test case 2 by using our proposed method.

	Set 1	Set 2
Acceptance Index	${\omega }_1=0.984169$	${\omega }_2=0.984952$
Human judgment	Success	Success

, both of which exceed the threshold value (${\omega }_{\mathrm{thres}}=0.95$), indicating successful merges.

Finally, the merging operations generate transformations that allow for the combination of the three local maps. By transforming these maps into the same coordinate system and processing their occupancy probabilities through the entropy filter, a global map can be obtained. The resulting global map is presented in Figure 21.

These two cases demonstrate the effectiveness of our proposed method in solving the problem of merging occupancy grid maps.

6. Conclusions

In conclusion, we have introduced a novel method for merging occupancy grid maps by utilizing image features to calculate the relative relationships between the maps. Our approach combines the proposed image pre-processing operation with an existing image stitching method, allowing for seamless integration with multi-robot systems without the need for physical interactions, thus enhancing flexibility.

Furthermore, we have identified that occupancy grid maps are susceptible to sensing errors and possess low pixel values, which can result in a reduced number of extracted features or inaccurate feature matching. To address this, we have implemented a pre-processing operation to rectify the map images prior to feature extraction and matching. This operation effectively eliminates the impact of uncertainty and sensing errors, enabling the extraction of features at specific positions. Consequently, this facilitates the accurate calculation of the relative transformation and ensures successful map merging.

We conducted rigorous testing of our method in two distinct environments that resemble typical living spaces. The results of these tests demonstrate the effectiveness of our method in merging maps with varying coordinates, scales, and the ability to merge multiple local maps into a global map simultaneously. Overall, our proposed method offers a robust and flexible approach to map merging for multi-robot systems, providing valuable advancements in this field.