Online Unmanned Ground Vehicle Path Planning Based on Multi-Attribute Intelligent Reinforcement Learning for Mine Search and Rescue

Abstract

Aiming to improve the efficiency of the online process in path planning, a novel searching method is proposed based on environmental information analysis. Firstly, a search and rescue (SAR) environmental model and an unmanned ground vehicle (UGV) motion model are established according to the characteristics of a mining environment. Secondly, an online search area path-planning method is proposed based on the gray system theory and the reinforcement learning theory to handle multiple constraints. By adopting the multi-attribute intelligent (MAI) gray decision process, the action selection decision can be dynamically adjusted based on the current environment, ensuring the stable convergence of the model. Finally, experimental verification is conducted in different small-scale mine SAR simulation scenarios. The experimental results show that the proposed search planning method can capture the target in the search area with a smoother convergence effect and a shorter path length than other path-planning algorithms.

1. Introduction

UGVs can replace humans in performing hazardous tasks with their superior speed, mobility, and atmospheric independence [1]. In recent years, research into UGV control has concentrated on navigation, SAR, area coverage, and military tasks. As an important function of UGV, SAR has received significant attention in academia, with the key focus areas including search area prediction [2,3], SAR force configuration [4], and rescue path planning [5,6,7]. The complex structure of mining caves lacks environmental information, and there is a risk of secondary explosions and collapses during the rescue process. To address the above dilemma, UGVs (see more detailed information about mining UGVs in [8]) have been considered to replace humans in dangerous SAR missions due to their advantages of low cost and high efficiency. Additionally, their path planning is a prerequisite for subsequent rescue missions. Therefore, the path-planning problem derived from this has always been an important topic in the field of UGV applications [9,10,11].

Previously, the research focus of SAR was on point-to-point path planning. According to the difference in target information, it can be categorized as known and unknown target path planning [12]. When in different application backgrounds, it can be classified as indoor planning, outdoor planning, or maritime planning [13]. In addition, according to the types of application platforms, robots can be classified as isomorphic path planners or heterogeneous path planners [14,15,16]. What is noteworthy is that most of the above methods plan paths based on precise or fuzzy target location information, which has high dependence on the in-variance of the demand. However, path planning without the targets’ location information is more in line with the general scenario and more challenging.

To address the dimension explosion caused by large amounts of data, most online POMDP methods are based on forward-looking searches, and these technologies reduce the complexity from different angles [17,18]. Among them, the most classic is PRIMAL [19]. These methods can usually be classified into three categories: Branch-and-Bound Pruning [20,21], Monte Carlo Sampling [22,23,24,25,26], and Heuristic Search [27,28,29,30]. However, most research using these methods has the following main shortcomings: (1) agents require much training before completing path-planning tasks, namely offline learning, and spend many resources on that; (2) it is also difficult for agents to respond when task maps differ significantly from training maps.

When the environmental structure is unfixed, observations are discrete through the sensor system’s data acquisition process. In order to study path-planning methods, modeling the environmental information of mining caves is necessary. In addition, obviously, a path-planning problem with real-time reactions to environmental uncertainty is NP-hard, which has been proven by Ryan A. MacDonald and Stephen L. Smith using informative path planning [31]. And the partial observable Markov decision process (POMDP) search balances short-term tracking performance and the long-term final cost [32]. The environmental information objectively exists in a stable state of change, in which the subject generally lacks awareness. The external world is a generally steady resource, and stable information is considered to not require storage [33]. Targeting the challenges above, this paper proposes a gray Q-Learning (GQL) search area path-planning method using UGV as the agent. This method combines the even gray model (EGM) prediction model, the MAI gray decision process, and the Q-Learning (QL) action planning model. The main contributions of this paper are as follows:

• A method for constructing an environmental model and an information feature mining method based on EGM are proposed to address the lack of environmental information in mining SAR.
• An agent-centered path-planning model based on the RL theory is proposed for the online path-planning problem. The optimization reward function designed for multiple scenarios effectively solves the conflict problem between paths, obstacles, and traps.
• A heuristic decision-making strategy based on the gray system theory is proposed for our SAR problem, which helps the model accelerate convergence towards the target and improve the robustness of the intelligent agent decision-making process.

The remaining sections of this paper are organized as follows: Section 2 introduces the constraints in the search area path-planning process used in Section 3. Section 4 provides a detailed description of the algorithm, a GQL search area path-planning method for the mine SAR problem. The proposed model is compared with the A* search algorithm [34], rapidly exploring the random tree (RRT) algorithm [35] and QL algorithm [36]. The simulation results and their discussion follow in Section 5, and the paper is concluded with a summary and an outlook for future work in Section 6.

2. Constraints in Search Area Path Planning

Although the natural environment undergoes constant changes, subtle changes occur in adjacent environments. Therefore, ground information is usually a stable resource that is always continuous when there are no obstacles [33]. Environmental information objectively exists in a stable state of change, but the subject generally lacks awareness of this fact. In new environments, it is impossible to obtain environmental data, but the observations with the scope measured by the sensor system describe the UGV’s understanding of the environment. The assumptions are as follows:

Assumption 1.

The SAR environmental information is described as the measured value of the sensor system.

Assumption 2.

The SAR environmental state always continues slightly.

There may be multiple coupled constraints for point-to-point path planning with an unknown target position, and the constraints that this paper focuses on are the resource and time expenditure of the search target. Based on the above analysis, it is reasonable to evaluate the efficiency of search area path-planning methods from the following aspects.

2.1. Probability of Detection (POD)

The $POD$ calculates the similarity between the characteristic values of the current position and the target position, as shown in (1), and describes the probability of a successful search at the current position of the target.

$$POD=\frac{1}{4}\frac{{(1+e^{0.1(c_T-c_a)})}^2}{e^{0.1(c_T-c_a)}}$$

(1)

2.2. Relative Distance (RD)

The $RD$ is defined as follows, and refers to the Euclidean distance between the agent’s and target’s current positions in real time.

$$RD=|p_t-p_a|$$

(2)

$p_t$ is the grid unit of the target, and $p_a$ is the grid unit of the agent.

2.3. Characteristic Distance (CD)

The $CD$ is defined in (3) and it describes the ability of agents to approach targets autonomously. The faster the characteristic distance converges, the higher the search efficiency.

$$CD=POD\times RD$$

(3)

3. SAR Environment Modeling

3.1. Environment Model

The environment is formulated as 2D grid worlds of size $m\times m\in N^2$ with a cell size of c and a set of all possible positions M. The buildings and walls that the UGV cannot occupy are given by the set

$$W=\left\{{\left[x_i^W,y_j^W\right]}^T:{\left[x_i^W,y_j^W\right]}^T\in M\right\}$$

(4)

The traps where the UGV will be punished are given by the set

$$B=\left\{{\left[x_i^B,y_j^B\right]}^T:{\left[x_i^B,y_j^B\right]}^T\in M\right\}$$

(5)

The environment can be described by functions $f_1,f_2,\dots ,f_e$, and the eigenvector at position $p\in M$ is

$$F_p=\left(f_1\left(p\right),f_2\left(p\right),\cdots ,f_e\left(p\right)\right)$$

(6)

The state of the target is described through its position:

$$p_{\mathrm{target}}={\left[x_{\mathrm{target}},y_{\mathrm{target}}\right]}^T\in M-W$$

(7)

The target tends to appear in positions within the feature interval, such as

$$F_{\mathrm{low}}\le F_{p_{\mathrm{target}}}\le F_{\mathrm{up}}$$

(8)

The UGV moves within the limits of the grid world, and the state of the UGV is described through the following:

• Its position;
• Its operational status, either inactive or active;
• Its field of view (FOV), the grid world size with the cell size, and the set of all possible positions.

The distance that the UGV walks in a mission’s online time slot ${\delta }_t$ is equivalent to the cell size c. The position of the UGV evolves according to the motion model given by

$$\begin{array}{c}\hfill p\left(t+1\right)=\left\{\begin{array}{cc}p\left(t\right)+a\left(t\right),\hfill & \hfill \phi \left(t\right)=1\\ p\left(t\right),\hfill & \hfill \mathrm{otherwise}\end{array}\right.\end{array}$$

(9)

The evolution of the operational status $\phi \left(t\right)$ of the UGV is given by

$$\begin{array}{c}\hfill \phi \left(t+1\right)=\left\{\begin{array}{cc}\hfill 0,\hfill & \hfill \phi \left(t\right)=0\vee p\left(t\right)=p_{\mathrm{target}}\\ \hfill 1,\hfill & \hfill \mathrm{otherwise}\end{array}\right.\end{array}$$

(10)

The end time point T is defined as the time slot when the UGV reaches its terminal state without actively operating. The following constraints restrict the UGV mobility model:

$$\begin{array}{c}\hfill \left\{\begin{array}{cc}p\left(t\right)\notin W,\hfill & \hfill \left(a\right)\\ t\le T,\hfill & \hfill \left(b\right)\\ \phi \left(0\right)=1,\hfill & \hfill \left(c\right)\end{array}\right.\end{array}$$

(11)

3.2. Optimization Problem

Using the environment model, the central aim of the UGV path-planning problem is to search the position of the target, given as H, while adhering to mobility constraints (8) and (11). The optimal problem is given over joint actions $\times a\left(t\right)$ by

$$\begin{array}{c}\hfill \begin{array}{c}{\max }_{\times a\left(t\right)}H\\ \mathrm{s}.\mathrm{t}.\left(8\right)\left(11\right)\end{array}\end{array}$$

(12)

4. Search Area Path-Planning Model for Mine SAR

Considering practical applications, the model work is performed as a two-step mapping process based on the visual and position sensors. Due to the state transition of the agent with the Markov property, the POMDP can provide a general framework [18]. Based on the above processes, the workflow of the gray online method is shown in Figure 1.

4.1. Local Path-Planning Reward Function

The state space $s\left(t\right)\in S$ of search area path-planning problem consists of the environment and the agent. Meanwhile, the states in the observation space $\Omega$ are given as $f_{\mathrm{map}}=f_{\mathrm{information}}\times f_{\mathrm{position}}:\mathrm{Environment}\to \Omega$ and $f_{\mathrm{position}}\in N^2$ and $f_{\mathrm{information}}\in R$. Observations $o\left(t\right)$ (5 × 5 FOV in mission) are defined through the tuple

$$o\left(t\right)=\left(M\left(t\right),V\left(t\right),W\left(t\right),B\left(t\right),p_{\mathrm{target}}\left(t\right),\left\{\phi \left(t\right)\right\}\right)$$

(13)

A security controller is introduced into the system to implement an obstacle avoidance constraint (a). The security controller evaluates the point $V_{i,j}\left(t\right)$ in the map and determines whether it can be reached. If not, it treats this point as ∅.

$$\begin{array}{c}\hfill V_{ij}\left(t\right)=\left\{\begin{array}{cc}\hfill \varnothing ,\hfill & (x_i,y_j)\in W\\ \hfill V_{i,j}\left(t\right),\hfill & \mathrm{otherwise}\end{array}\right.\end{array}$$

(14)

In addition, a safety controller evaluates the agent’s action $a\left(t\right)$ and determines whether the action should be punished. Based on the conclusion, the corresponding safety penalty value $\beta$ is offered:

$$\begin{array}{c}\hfill \beta \left(t\right)=\left\{\begin{array}{cc}-1,& p\left(t\right)+a\left(t\right)\in B\\ 0,& \mathrm{otherwise}\end{array}\right.\end{array}$$

(15)

The reward function $R:S\times A\times S\to R$ of the POMDP is defined as

$$r\left(t\right)=\tau \times \Phi \left(t\right)+\beta \left(t\right)+\epsilon$$

(16)

The agent reaches a new state $s(t+1)$ which may depend on the state $s\left(t\right)$ and action $a\left(t\right)$, and updates value Q.

4.2. Environmental Data Prediction Process

In order to highlight the attribute of the uncertain information, the environment data are treated as gray numbers [37], that is, more attention is paid to the nature of the data than the values. Therefore, it is necessary to preprocess the data. However, the prediction system with limited information may be affected by interference. Firstly, for the information system with the gray attribute, the gray number and buffer operator [38] are combined to describe the data of the information system. Secondly, the data analyzed by the gray algorithm are preprocessed by the even gray model (as shown in Figure 2).

The FOV data sequence $d^{\left(0\right)}\left(k\right)$ is described by generating the function $G\left(o\right(k\left)\right)$, and its one time accumulating generation operator(1-AGO) sequence $D^{\left(1\right)}$ can be calculated as

$$D^{\left(1\right)}=\left(d^{\left(1\right)}\left(1\right),d^{\left(1\right)}\left(2\right),\dots ,d^{\left(1\right)}\left(n\right)\right)$$

(17)

Thus, the sequence $Z^{\left(1\right)}$

$$Z^{\left(1\right)}=\left(z^{\left(1\right)}\left(1\right),z^{\left(1\right)}\left(2\right),\dots ,z^{\left(1\right)}\left(n\right)\right)$$

(18)

can be obtained from

$$z^{\left(1\right)}\left(k\right)=\frac{1}{2}\left(d^{\left(1\right)}\left(k\right)+d^{\left(1\right)}(k-1)\right)$$

(19)

So, the even gray model of the environmental data prediction process is

$$d^{\left(0\right)}\left(k\right)+a{z}^{\left(1\right)}\left(k\right)=b$$

(20)

in which $-a$ is the development coefficient and b is the gray actuating quantity. The prediction sequence is the basis for the subsequent algorithm to determine the optimal decision.

4.3. The Multi-Attribute Gray Decision Process

This paper presents a multi-attribute gray decision-making method to find solutions in exploration and development to enable UGVs to independently complete search tasks. In the new model, the following constraints of the observation space are regarded: feature constraints, update constraints, exclusion constraints, and attraction constraints. Four data maps are formed from the constraints, and the target map and the obstacle map constitute a complete observation space.

The feature constraint $U^{\left(1\right)}=(u_{ij}^{\left(1\right)})$ is regarded as an interval number, and is a moderate-type objective with the upper limit $G_{\max }$ and lower limit $G_{\min }$ assumed.

When $u_{ij}^{\left(1\right)}\in \left[G_{\min },\frac{1}{2}\left(G_{\max }+G_{\min }\right)\right]$, the lower effect measurement function for a moderate objective is

$$r_{ij}^{\left(1\right)}=\frac{2\left(u_{ij}^{\left(1\right)}+G_{\min }\right)}{G_{\max }-G_{\min }}$$

(21)

When $u_{ij}^{\left(1\right)}\in \left[\frac{1}{2}\left(G_{\max }+G_{\min }\right),G_{\max }\right]$, the upper effect measurement function for a moderate objective is

$$r_{ij}^{\left(1\right)}=\frac{2\left(G_{\max }-u_{ij}^{\left(1\right)}\right)}{G_{\max }-G_{\min }}$$

(22)

The update constraint $U^{\left(2\right)}=\left(u_{ij}^{\left(2\right)}\right)$ is a benefit-type objective; the more significant the effect sample value, the better. The effect measurement function for the benefit-type objective is

$$r_{ij}^{\left(2\right)}=\frac{u_{ij}^{\left(2\right)}-\frac{5}{2}}{{\max }_i{\max }_j\left\{u_{ij}^{\left(2\right)}\right\}-\frac{5}{2}}$$

(23)

The exclusion constraint $U^{\left(3\right)}=\left(u_{ij}^{\left(3\right)}\right)$ is a benefit-type objective. The effect measurement function for this objective is

$$r_{ij}^{\left(3\right)}=\frac{u_{ij}^{\left(3\right)}}{{\max }_i{\max }_j\left\{u_{ij}^{\left(3\right)}\right\}}$$

(24)

Attraction constraint reflects the agent’s preference for the map after starting. If the agent has no preference, the map should be searched evenly. The attraction constraint $U^{\left(4\right)}=\left(u_{ij}^{\left(4\right)}\right)$ is a cost-type objective. The effect measurement function for this objective is

$$r_{ij}^{\left(4\right)}=\frac{u_{ij}^{\left(4\right)}}{{\min }_i{\min }_j\left\{u_{ij}^{\left(4\right)}\right\}}$$

(25)

Furthermore, the decision weight of the k-th objective is ${\eta }_k(k=1,2,3,4)$. The uniform effect measurement matrix of the decision strategy $s_{ij}\in S$ under the k-th objective is $R^{\left(k\right)}=\left(r_{ij}^{\left(k\right)}\right)$. The synthetic effect measurement of decision strategy $s_{ij}$ is

$$r_{ij}=\sum _{k-1}^s{\eta }_k{r}_{ij}^{\left(k\right)}$$

(26)

The synthetic effect measurement matrix is $R=\left(r_{ij}\right)$. Using the synthetic effect measure matrix, effect values of different significance, dimensions, and characters can be comprehensively considered to determine the optimal decision strategy.

5. Experimental Verification and Result Analysis

5.1. Simulating Scenarios and Experimental Setup

The environment of the simulated case models and randomly selects three scenarios based on the mining SAR environment model. In order to verify the ability of the UGV agent to avoid obstacles online, the scene is randomly set by a map generation code (based on the code in [19]), and only kept consistent in the comparative experiment. Because the experiment involves online path-planning verification, the UGV does not conduct offline learning in advance. All algorithms in this paper are coded and implemented in Python version 3.9. The training environment for this model is Intel Core i7-4710MQ CPU @2.50 GHz four cores Chinese-made laptop.

5.2. Simulation and Comparison Experiment

The model is compared with the A* search algorithm, RRT algorithm, and QL algorithm, while the rolling online algorithm is used to supplement the traditional algorithm. The comparison results are shown in Figure 3. In order to adapt to a four-way proxy, the A* search algorithm combines a linear function and the Manhattan distance as the heuristic function. The RRT algorithm selects the nearest node to join the tree after obstacle judgment. The QL algorithm combines the average rolling algorithm to complete the test, which is different from the proposed method. All algorithms only search online on the map without being trained beforehand. From Figure 3, it can be seen that the A* search algorithm cannot adapt to real-time scenarios, and the agent is easily trapped in deadlock situations, resulting in its inability to complete tasks. Due to the lack of a target decision process, the generalization ability of the RRT algorithm could be improved in different scenarios, and the algorithm often cannot continue due to stuttering during runtime. The QL algorithm and the proposed method can complete planning and produce coverage search paths, but the proposed method performs better when searching for the target.

Statistical analyses are conducted on the path-planning results for final coverage, repeated coverage, and step size to evaluate the above algorithms.

Table 1. Quantitative evaluation of planning results for different scenarios.

Scenario	Algorithms	Step	Coverage (%)	Repeated Coverage (%)
Scenario 1	A*	-	9.81	-
	RRT	68	31.31	1.40
	QL	143	57.94	9.35
	Ours	45	21.50	0
Scenario 2	A*	-	5.96	-
	RRT	80	37.16	0
	QL	66	30.73	0
	Ours	40	15.60	3.21
Scenario 3	A*	-	9.95	-
	RRT	56	27.01	0
	QL	73	32.23	2.37
	Ours	42	20.38	0

shows the quantitative evaluation results. It shows that the search path produced by the proposed method performs better when searching for the target. In addition, the algorithm is evaluated from the perspective of searchability. As shown in Figure 4, the proposed method has the fastest convergence speed. This indicates that the agent in the proposed method catches the target preferentially. Traditional path-planning algorithms are prone to becoming stuck in local optima, especially when dealing with autonomous navigation tasks in unknown maps. In contrast, reinforcement learning algorithms have more substantial autonomous planning capabilities. Our improved algorithm has a faster convergence speed in the autonomous planning process and can effectively handle searching for dynamic targets in unknown maps.

5.3. Policy Evaluation

According to the planning results of sub-section A, the improved algorithm is smaller than the conventional QL planning algorithm in terms of running step size. In addition, the QL algorithm has a high repetition coverage rate. The target decision algorithm can play a good role in assisting with decision making. To evaluate whether the corresponding parameters of the target decision algorithm can effectively assist in the decision problem when the target position is uncertain, 20 simulation experiments were carried out in the mine SAR scenario in Figure 5. Figure 5 shows that the median set of step size of the guidance scheme is smaller than that of the average scheme. In addition, the step sizes of guidance schemes is concentrated below 200, and the data fluctuation is not significant. In contrast, the step sizes of the average scheme are discretely distributed between 225 and 375, resulting in significant data fluctuation. The above results illustrate that the guided solution has a better adjustment effect on the target decision algorithm.

The search results under different schemes are shown in Figure 6. In the early and medium stages, all schemes remain explored and developed to obtain the best search path. In the later stage, however, the average scheme will still maintain a high probability of exploration, reducing data use. The above results indicate that improved policy has advantages from the perspective of action selection.

5.4. Reward Function Evaluation

The data for different solutions are shown in

Table 2. The principle parameters of the target-tracking algorithm.

Solution	Parameter	Value
Improved Solution	learning rate	0.8
	reward decay	0.9
	$ϵ$ greedy	0.9
	$\Phi \left(t\right)$	5
		0
	$\beta \left(t\right)$	−1
Unimproved Solution	learning rate	0.4
	reward decay	0.8
	$ϵ$ greedy	0.9
	$\Phi \left(t\right)$	1
		0

. Figure 7 shows the performance of planning operation steps under different reward function solutions. The same curve trend can be observed between the unimproved and improved solutions, indicating that these two solutions play a role in solving obstacle avoidance and goal orientation. However, the convergence speed of the improved solution is faster. Meanwhile, the search path-planning results under different schemes are shown in Figure 8. During the learning process, the learning speed of the unimproved solution is slower, resulting in inaccurate planning multiple times. The improved solution provides more accurate results than the unimproved one during the learning process. It reduces the repetitive path rate of the agent and improves the efficiency of online planning. This proves that the reward function is goal-oriented for the intelligent agent, enabling it to learn gradually.

6. Conclusions and Future Work

This paper studies the online path planning of mine SAR. Based on simulating an SAR environment using the characteristics of a mining cave field, a path-planning model for POMDP-based online search areas is proposed. The gray objective decision function guides this model and plans the optimal search path through information analysis, making it more flexible and intelligent. The comparative experimental results show that the model can complete target search tasks online in different maps of the same region and generate search paths to achieve faster convergence to dynamic target positions.

The future potential trends that can be integrated with the planning issues of UGVs will include, but are not limited to, the following:

• The interaction of UGVs with Cyber–Physical systems could enhance UGV’s adaptability to dynamic environments and validate the effectiveness and feasibility of path-planning algorithms through the use of 3D virtual reality models [39].
• The strategic application of human–machine interaction technology is not just a possibility, but a promising avenue that is poised to play a pivotal role in the rapid advancement of UGV intelligence [40].
• Swarm Intelligence technology can expand UGVs’ application scope and significantly improve its efficiency in completing tasks [41].
• Another important research direction to enhance the autonomy of UGVs is to combine them with Internet of Things technology to improve their flexibility in different locations [42,43].

Given the aforementioned topics, further in-depth research will be carried out within the above-specified fields.