A Long-Term Target Search Method for Unmanned Aerial Vehicles Based on Reinforcement Learning

Abstract

Unmanned aerial vehicles (UAVs) are increasingly being employed in search operations. Deep reinforcement learning (DRL), owing to its robust self-learning and adaptive capabilities, has been extensively applied to drone search tasks. However, traditional DRL approaches often suffer from long training times, especially in long-term search missions for UAVs, where the interaction cycles between the agent and the environment are extended. This paper addresses this critical issue by introducing a novel method—temporally asynchronous grouped environment reinforcement learning (TAGRL). Our key innovation lies in recognizing that as the number of training environments increases, agents can learn knowledge from discontinuous trajectories. This insight leads to the design of grouped environments, allowing agents to explore only a limited number of steps within each interaction cycle rather than completing full sequences. Consequently, TAGRL demonstrates faster learning speeds and lower memory consumption compared to existing parallel environment learning methods. The results indicate that this framework enhances the efficiency of UAV search tasks, paving the way for more scalable and effective applications of RL in complex scenarios.

1. Introduction

Due to the low cost and flexibility of unmanned aerial vehicles (UAVs), they are frequently employed in various search missions. Depending on the type of target motion, these problems can be categorized into static target search problems and dynamic target search problems. In static target search problems, common scenarios include resource exploration in harsh environments and rescuing individuals trapped in earthquakes. In dynamic target search problems, application scenarios encompass rescuing lost tourists in forests and searching for enemy targets cruising in warfare.

Based on the differences in modeling approaches, the existing methods for UAV search can be primarily categorized into four types: traditional methods, heuristic methods, probability-based methods, and deep reinforcement learning-based methods. Traditional methods can be further divided into exact algorithms and approximation algorithms. The notable representatives of exact algorithms are integer programming algorithms and A* search algorithms [1]. Approximation algorithms include Voronoi diagram methods [2] and visibility graph-based methods [3]. Traditional methods have good interpretability and exhibit excellent performance on simple problems. However, they often lack scalability and are difficult to model and solve when dealing with complex issues. Heuristic algorithms encompass genetic algorithms [4,5], ant colony optimization [6], wolf pack algorithms [7], and particle swarm optimization [8,9]. These algorithms can be applied to search problems of various scales and are often easier to model. Probability-based methods [10,11] often exhibit superior interpretability compared to intelligent algorithms, and in some instances, they may also demonstrate enhanced performance. Nevertheless, these methods yield fixed solutions. When the scenario undergoes slight changes, the solutions obtained by these algorithms cannot automatically adapt to the variations in the scenario. In contrast, deep reinforcement learning-based algorithms can produce an intelligent agent capable of real-time decision-making, thereby possessing a better ability to adapt to dynamic scenarios.

The characteristics of the four categories of methods are presented in

Table 1. Comparison of four methodological approaches.

Method	Scalability	Accuracy	Adaptability
Traditional method	Inferior	Satisfactory	Inferior
Heuristic method	Superior	High variance, dependent on iteration time	Inferior
Probability-based method	Superior	Satisfactory	Inferior
DRL-based method	Superior	High variance, dependent on training time	Superior

:

In conclusion, deep reinforcement learning methods are emerging as a focal point for researchers due to their ease of modeling, scalability to problems of varying magnitudes, and adaptability to changing scenarios. These attributes make deep reinforcement learning an increasingly attractive approach in the field.

Researchers have proposed numerous methods for guiding search using deep reinforcement learning algorithms. Imanberdiyev et al. [12] solved the problem of autonomous drone path planning in a positional environment using model-based reinforcement learning methods, while Wang et al. [13] proposed a reinforcement learning method for this problem that does not require building a planning model. They investigated drone search strategies in scenarios with limited communication spectrum. Wang Tian et al. [14] studied the search and tracking problem for multiple drones under energy constraints. They explored drone search schemes in scenarios with potential threats. Shen et al. [15] proposed an improved multi-agent reinforcement learning algorithm for the scenario of multiple UAVs searching for multiple static targets. Su et al. [16] investigated a multi-agent reinforcement learning algorithm with constraints on sensing range and communication range. Zhang et al. [17] investigated a multi-UAV search method that combines belief probability graphs and an improved DDPG algorithm.

In these studies, reinforcement learning demonstrated its powerful adaptability and scalability. However, its performance is dependent on extended training times. To address this issue, researchers have proposed numerous agent–environment parallel interaction frameworks. These frameworks leverage the multi-core advantages of modern processors, transforming the original serial multi-round interactions between agents and environments into parallel single-round interactions.

For instance, Ray [18] and RLlib [19] can harness the hardware processing capabilities of distributed systems. Dalton et al. [20] demonstrated that running the Atari environment on GPUs significantly accelerates the agent’s learning speed by increasing the number of environments. Subsequently, Makoviychuk et al. [21] proposed Isaac Gym as a GPU-parallel simulation platform for reinforcement learning, serving as a new benchmark environment for reinforcement learning. Weng et al. introduced Envpool [22], which supports not only the Atari environment but also complex environments such as StarCraft. In various subfields, parallel agent–environment interaction methods have been extensively utilized. For instance, Liu et al. [23] developed a high-performance GPU-parallel underwater robot learning environment. Freeman et al. [24] proposed a differentiable physics simulation tool utilizing GPU parallel computing.

However, when addressing the long-duration search problem for UAVs, these methods still face two primary challenges. First, regardless of the number of processor cores utilized, agents still require at least one complete round of interaction with the environment. When the application scenario of drones involves searching for unknown dynamic targets, the uncertainty of target movement information can lead to lengthy search times. As a result, each round of simulation consumes considerable time, thereby slowing the learning speed of the agent. Secondly, a large number of simulated environments consume substantial GPU memory. Since all exploration data of the agents need to be stored prior to updating the agent network, both the increase in the number of environments and the prolongation of agent exploration time will lead to increased memory usage. Consequently, when addressing the problem of long-term drone search, memory constraints may prevent the creation of a sufficient number of simulated environments for agent training.

In our previous work [25], we proposed the OC-MAPPO method, which integrates optimal control with DRL (deep reinforcement learning) to address the multi-UAV cooperative search problem. The experimental results demonstrated its enhanced adaptability to dynamic environments. Additionally, we employed the method of environment pool proposed in [22], utilizing parallel environments to accelerate agent learning. During experiments, we observed that increasing the number of environments from 1 to 10 significantly improved the agent’s learning speed, as the interaction time with the environment changed from multiple rounds to a single round. Yet, further increasing the number of environments to several hundreds or even thousands did not alter the learning speed. This is because the data collected before policy updates originate from the same distribution, and increasing the number of environments merely results in more homogeneous data. Then, how to overcome such a challenge and speed up DRL training by quantitatively increasing the parallel training environment become a problem area of interest.

In this paper, through analysis and experimentation, we discover that in the context of UAV cooperative search tasks, it is possible to designate different time segments for different environments during the parallel process. This approach allows agents to engage in minimal forward interactions with each environment, further expediting the learning speed of the agents. Our findings suggest a novel approach to optimizing the efficiency of reinforcement learning in multi-agent UAV scenarios, potentially overcoming existing limitations in the field.

The remainder of this article is structured as follows: Section 2 delineates the problem scenario under investigation. Section 3 elucidates and analyzes extant methodologies. Section 4 expounds upon the proposed approach. Section 5 empirically validates the efficacy of the proposed method through experimental evaluation.

2. Problem Formulation

In our research, we examine a search scenario where the target is lost at a specific time point. Given the known parameters of the target’s maximum velocity, last known position, and time of loss, UAVs are deployed to locate the missing target. The primary objective of this study is to determine the optimal search trajectory for each UAV.

2.1. Scenario Description

In our study, the search scenario can be represented by Figure 1. The primary assumptions for this scenario are as follows:

• UAVs’ scanning radius is fixed, and the scanning radius is an integral multiple of the unit grid edge length within the environment.
• The targets exhibit stochastic movement patterns within the designated operational zone, demonstrating no intentional evasive behavior in response to UAVs’ reconnaissance activities.
• UAVs communicate with one another via satellite-based systems, enabling them to share their respective positional data and reconnaissance information.

2.2. Update Model and Move Model

In [25], we employ a target probability map to represent the probability of target presence in each unit area within the task domain. The update of the target probability map is conducted in two steps:

• Diffusion update due to target movement.
• Bayesian update based on UAV reconnaissance.

These two steps can be described by the following formula:

$$TP{M}_{i,j}^{t+1}={\displaystyle \frac{1}{9}}\ast \sum _{m=i-1}^{i+1}\sum _{j-1}^{j+1}TP{M}_{m,n}^t$$

(1)

$$TP{M}_{i,j}^{t+1}={\displaystyle \frac{TP{M}_{i,j}^t\ast tp}{tp\ast TP{M}_{i,j}^t+(1-tp)(1-TP{M}_{i,j}^t)}}$$

(2)

In this context, $tp$ represents the probability of missed detection by the sensor at position $i,j$.

In the practical application of the algorithm, two scenarios are considered. The first scenario involves an environment sufficiently large that the target does not reach the boundary when the maximum detection time elapses. Alternatively, boundary conditions must be separately addressed. When the target appears at corners or edges, it exhibits different diffusion probabilities for the surrounding areas. The diffusion probability at corners changes from 1/9 to 1/4, while at edges, it shifts from 1/9 to 1/6. In the experimental results presented in this paper, we adopt the second design approach.

This article primarily focuses on the rapid learning of search strategies for UAVs rather than on UAV modeling itself. Consequently, the motion model of the UAV is simplified, employing Newton’s Second Law of Motion to describe the UAV’s movement. Let ${\alpha }_u$ denote the orientation of drone u, $po{s}_x^u$ and $po{s}_y^u$ represent the position of drone u, $ve{l}^u$ indicate the velocity, $a^u$ signify the acceleration, and ${\omega }_u$ represent angular acceleration. Consequently, the UAV’s movement can be described as follows:

$$\left\{\begin{array}{c}\dot{po{s}_x^u}=spee{d}^{u}cos{\alpha }_u\hfill \\ \dot{po{s}_y^u}=spee{d}^{u}sin{\alpha }_u\hfill \\ \dot{ve{l}^u}=a_u\hfill \\ \dot{{\alpha }_u}={\omega }_u\hfill \end{array}\right.$$

(3)

2.3. Reconnaissance Model

Let the sensor model of UAV u be denoted as $Senso{r}_u(·)$ and $Precis{e}_u$, where $Senso{r}_u$ characterizes the reconnaissance range of the sensor, and $Precis{e}_u$ represents the reconnaissance precision of the sensor. The reconnaissance area of UAV u at time t is designated as $D_t^u$, and $p_t^u$ denotes the probability of UAV u detecting the target at time t. The reconnaissance model of the UAV can be described using the following formula:

$$D_t^u=Senso{r}_u\left(\overrightarrow{c_t^u}\right)$$

(4)

$$p_t^u={\int }_{(x,y)\in D_t^u}Precis{e}_u(\overrightarrow{c_t^u},x,y)\ast TP{M}_{x,y}^{t}dxdy$$

(5)

2.4. Optimal Control Model Construction

In [25], we proposed an optimal control model to formalize the UAV search problem. In this model, the search objective for the unmanned aerial vehicle (UAV) is to minimize the logarithm of the product of probabilities that the target remains undetected. The mathematical formulation is as follows:

$$mi{n}_{\mathbf{A}}J\left(\mathbf{A}\left(T\right)\right)={\int }_0^T\sum _{u\in U}ln(1-p_t^u)dt$$

(6)

In this equation, $u_p$ denotes the probability of a drone detecting the target. U represents the set of all drones. T signifies the maximum search time. And $\mathbf{A}\left(t\right)$ denotes the flight trajectory of a drone within the time interval $[0,t]$.

3. Preliminary and Analysis

Existing reinforcement learning methods typically involve an agent interacting with the environment over multiple episodes, denoted as C. The agent improves its behavior based on the experiences gathered. Without any parallelization, the time taken for the agent to update its policy after interacting with the environment is $C\times T$. If multiple environments are initiated simultaneously for training, as in the OC-MAPPO method, the time required for the agent to update its policy can be reduced to T. However, in OC-MAPPO, the number of environments initiated far exceeds C, yet the time expenditure cannot be further reduced. Based on these considerations, this section reanalyzes the principles of reinforcement learning operations and proposes a new architecture to improve the training process.

When applying reinforcement learning to solve this problem, the cumulative return and single-step reward are defined as follows:

$$\begin{array}{cc}\hfill return& =-J\left(\mathbf{A}\right(T\left)\right)\hfill \\ \hfill R_t& =J\left(\mathbf{A}\right(t-1\left)\right)-J\left(\mathbf{A}\right(t\left)\right)\hfill \end{array}$$

(7)

Let $S_t$ denote the state of UAVs at time t. $S_t$ encompasses the UAV’s position, its orientation, and the probabilistic map of target information. Let $V\left(S_t\right)$ represent the maximum expected reward attainable when the UAV is in state $S_t$. The mathematical formulation of $V\left(S_t\right)$ is as follows:

$$V\left(S_t\right)=minJ\left(\mathbf{A}\left(T\right)\right)-J\left(\mathbf{A}\left(t\right)\right)$$

(8)

In essence, the original problem can be reformulated as follows:

$$minJ\left(\mathbf{A}\left(T\right)\right)=J\left(\mathbf{A}\left(t\right)\right)+V\left(S_t\right)$$

(9)

If we know the form of the function $V(·)$, then we can solve the original problem by simply solving the aforementioned problem, in which the agent only needs to interact with the environment for t steps. This formula delineates the interactive process between the agent and the environment within the time interval $[0,t]$. The value of $V\left(S_t\right)$ can be obtained by solving the following problem:

$$V\left(S_t\right)=min{\int }_t^T\sum _{u\in U}ln(1-p_t^u)dt$$

(10)

This formula corresponds to the interaction process between the agent and the environment during the time interval $[t,T]$. It can be observed that this problem shares the same structure as the original problem, thus allowing for the application of identical solution methods. In this case, one could directly employ reinforcement learning techniques to solve the problem, or alternatively, partition it into two intervals: from t to $t_1$ and from $t_1$ to T. The optimal policy for the t to $t_1$ interval can then be determined by estimating $V\left(S_{t_1}\right)$, ultimately yielding the value of $V\left(S_t\right)$.

This question arises: is it feasible to concurrently execute the processes of solving (9) and (10)? Intuitively, it would seem that the second process must invariably follow the first, as reaching state $S_t$ necessarily requires first attaining state $S_{t-1}$, precluding parallel execution. However, if we initiate both processes simultaneously, with the first process remaining idle during the preparation phase while the second process allowing for the agent to interact with the environment $t_1$ times, we can then simultaneously complete one simulation corresponding to (9) and (10) in the subsequent interaction (assuming $t=t_1$). This methodology can be delineated through Algorithm 1:

Algorithm 1 Asynchronous Exploration Algorithm

Warm up stage:     For $e_1$         do nothing     For $e_2$         For t in $1\dots t_1$             $A_t\sim Actor\left(S_t\right)$             $S_{t+1}=Environment(S_t,A_t)$             $S_t<-S_{t+1}$ Train stage: For each i in $1\dots n$     For $e_1$,$e_2$         For t in 1…$t_1$         $A_{t}Actor\left(S_t\right)$         $S_{t+1}=Environment(S_t,A_t)$         $S_t<-S_{t+1}$

It is important to note that this approach differs from using a single process to perform T simulations. Owing to the agent’s policy exhibiting some randomness in each interaction with the environment, the environment itself may possess inherent stochasticity. Consequently, the terminal state of the first process and the initial state of the second process may not be identical. Then, what impact does this difference have on the learning strategies of intelligent agents?

To illustrate this issue, we will use the Proximal Policy Optimization (PPO) [26] algorithm, a prominent example from the field of reinforcement learning algorithms. Consider two solution outcomes in the solving process: actor and critic. The actor is responsible for the agent’s behavioral policy, $\pi$, accepting the drone’s state as input and outputting the drone’s action, i.e., $\pi \left(S_t\right)\to a\left(t\right)$. The critic is responsible for fitting $V(·)$. If the PPO algorithm is employed to train the actor and critic, the parameter update formulas for the actor and critic are as follows:

For the actor:

$$\begin{array}{cc}\hfill L\left({\theta }_t\right)& ={\mathbb{E}}_t\left[min\left({\displaystyle \frac{{\pi }_{\theta }\left(a_t\right|s_t)}{{\pi }_{{\theta }_{old}}\left(a_t\right|s_t)}}Ad{v}^{{\pi }_{{\theta }_{old}}}(s_t,a_t),\right.\right.\hfill \\ \hfill & \hfill \left.\left.\mathrm{clip}\left({\displaystyle \frac{{\pi }_{\theta }\left(a_t\right|s_t)}{{\pi }_{{\theta }_{old}}\left(a_t\right|s_t)}},1-ϵ,1+ϵ\right)Ad{v}^{{\pi }_{{\theta }_{old}}}(s_t,a_t)\right)\right]\hfill \\ \hfill {\theta }_{t+1}& ={\theta }_t+{\displaystyle \frac{\partial L\left({\theta }_t\right)}{{\theta }_t}}\ast lr\hfill \end{array}$$

(11)

In this context, $\theta$ represents the parameters of the actor network, and $lr$ denotes the learning rate.

$$\begin{array}{cc}\hfill L\left(v_t\right)& ={\mathbb{E}}_{s_t\sim {\pi }_{\theta }}\left[{\left(Criti{c}_v\left(s_t\right)-Criti{c}_t^{target}\right)}^2\right]\hfill \\ Criti{c}_t^{target}\hfill & =r_t+\gamma Criti{c}_v\left(s_{t+1}\right)\hfill \\ \hfill v_{t+1}& =v_t+{\displaystyle \frac{\partial L\left(v_t\right)}{\partial v_t}}\ast lr\hfill \end{array}$$

(12)

wherein v denotes the parameters of the critic network, and $\gamma$ represents the discount factor, which is a commonly used parameter in reinforcement learning. In this particular problem, to ensure correspondence with the original formulation, $\gamma$ is set to 1. Let us first consider the learning process of the critic. Consider the fitting target of the critic as follows:

$$Critic\left(S_t\right)-Critic\left(S_{t+1}\right)=r$$

(13)

Although the ultimate goal of the $Critic$ is to fit $V(·)$, it first attempts to fit the expected reward under the current policy. Then, it improves the $Actor$’s policy by letting the $Actor$ choose actions that can generate more expected rewards than the current policy. When the $Actor$’s policy is improved to the optimal policy, the $Critic$ network will be able to fit the true $V(·)$ value. From Equation (13), it can be seen that the update of the $Critic$ depends on the r value obtained from the interaction process between the $agent$ and the environment. Whether it is the (9) process or the (10) process, the intelligent agent can obtain the true r from the interaction with the environment, thereby improving the accuracy of the $Critic$’s fitting of the expected return $V(·)$.

Nonetheless, a challenge persists for (9). Due to the original process, the following equation holds:

$$Environment(S_t,A\left(t\right))=S_{t+1}\mathrm{for}t\in \{1,2,\dots ,T\}$$

(14)

where $Environment(·)$ denotes the environmental model. However, for Algorithm 1, at $t=t_1$, it holds that $S_{t_1+1}^{e_2}\ne Environment(S_{t_1}^{e_1},A{\left(t_1\right)}^{e_1})$. This discrepancy leads to the inability of $e_1$ to generate the labels for the critic, because for each state $S_t^{e_1}$ of $e_1$, the corresponding critic label $V\left(S_t^{e_1}\right)$ is computed as follows:

$$V\left(S_t^{e_1}\right)=\sum _{i=t}^{i=t_1}V\left(S_i\right)+\sum _{i=t_1+1}^{T}V\left(S_i\right)$$

(15)

Since $S_i$ is interrupted at $t_1$, the data for ${\sum }_{i=t_1+1}^{T}V\left(S_i\right)$ cannot be obtained.

Fortunately, this issue is improved when the agent interacts simultaneously with multiple environments. Consider the set $\mathbb{S}$, which represents the set of all states reached after $t_1$ time steps from the initial state $S_1$. Thus, $S_{t_1+1}^{e_2}\in \mathbb{S}$ and $Environment(S_t^{e_1},A{\left(t\right)}^{e_1})\in \mathbb{S}$. Although both are in the same set, due to the often large number of elements in $\mathbb{S}$, the critic cannot infer $V\left(S_{t+1}^{e_1}\right)$ solely from $V\left(S_{t_1+1}^{e_2}\right)$. However, if we have a large amount of data, suppose m samples, we can get their label through the following equation.

$$\begin{array}{cc}\hfill V\left(S_{t_1+1}^{e_2}\right)& \hfill =\sum _{i=t_1+1}^{T}V\left(S_i^{e_2}\right)\hfill \\ \hfill V\left(S_{t_1+1}^{e_3}\right)& \hfill =\sum _{i=t_1+1}^{T}V\left(S_i^{e_3}\right)\hfill \\ \hfill & \hfill \dots \hfill \\ \hfill V\left(S_{t_1+1}^{e_m}\right)& \hfill =\sum _{i=t_1+1}^{T}V\left(S_i^{e_m}\right)\hfill \end{array}$$

(16)

Then, we can use these m labeled data points to fit $V\left(s\right)$ using the neural network critic, where $s\in \mathbb{S}$. The fitted critic can then be used to obtain the value of $V\left(S_{t_1+1}^{e_1}\right)$.

$$\begin{array}{cc}\hfill \widehat{V}\left(s\right)& =\mathrm{Critic}\left(s\right),s\in \mathbb{S}\hfill \\ \hfill \widehat{V}\left(S_{t+1}^{e_1}\right)& \approx \mathrm{Critic}\left(S_{t+1}^{e_1}\right)\hfill \end{array}$$

(17)

Regarding the actor, as per Equation (11), the update of the actor is related to ${\pi }_{\theta }$ and $Adv$, where ${\pi }_{\theta }$ denotes the parameters of the actor, and $Adv(S_t,a_t)$ represents the advantage of executing action $a_t$ in state $S_t$, which can be calculated by the following equation:

$$\begin{array}{c}\hfill Adv\left(s_t\right)=r_t+\gamma ·Critic\left(s_{t+1}\right)-Critic\left(s_t\right)\end{array}$$

(18)

From the above equation, it is evident that the value of $Adv$ is related to the output of the critic and the actual reward r obtained by the agent. When the critic can be updated normally, the actor can also be updated accordingly.

Through the aforementioned analysis, we have demonstrated that when the number of environments is large, the interaction between the agent and the environment can be divided into two parts: the $0-t_1$ segment and the $t_1-T$ segment, which can run concurrently. By partitioning the interaction process, the $t_1-T$ process exhibits the same structure as the original problem. This suggests that if the number of environments is sufficiently large, the $t_1-T$ process can be further subdivided. Consequently, we propose the temporally asynchronous grouped environment reinforcement learning (TAGRL) method. This approach divides the environments into several groups. After a preparation phase lasting T, each group executes different time segments of an interaction episode.

Let E denote the total number of environments, and G denote the number of environment groups. Through the aforementioned design, the algorithm can allow the agent to interact with the environment $T/G$ times before each update instead of T times. Furthermore, compared to existing parallel methods (such as OC-MAPPO), the TAGRL method shows lower memory consumption. The key memory consumption component for both algorithms is the buffer, which stores the experiential data generated during the training process. In the OC-MAPPO method, the data storage requirement is as follows:

$$Memor{y}_{OC-MAPPO}=E\times T\times (Size\left(obs\right)\times N+Size\left(state\right))$$

(19)

In contrast, the memory requirement for TAGRL is as follows:

$$Memor{y}_{TAGRL}=E\times {\displaystyle \frac{T}{G}}\times (Size\left(obs\right)\times N+Size\left(state\right))$$

(20)

where N represents the number of agents. In the aforementioned calculations, we disregard the action and reward, as their memory usage is significantly less than that of obs and states (the latter being multidimensional tensors while the former are vectors). It is evident that, with the same number of environments, the TAGRL method requires less memory compared to the OC-MAPPO method.

4. Method

Based on the preceding analysis, we propose an asynchronous parallel reinforcement learning framework.

Suppose we have E environments, which we divide into G groups (in Figure 2, we use $G=5$ as an example to illustrate the method). During the preparation phase, the first four groups of environments interact with the agent $T/G$ times each, while the fifth group remains inactive. The preparation phase consumes the time equivalent to one complete round of interactions. Following the preparation phase, before each update of the agent’s policy, each environmental group interacts with the agent $T/G$ times. During the Warm Up phase, for the i-th environment, the following formula can be used to determine the number of steps the agent needs to interact with that environment.

$$GT\left(i\right)=t_{max}\times {\displaystyle \frac{\lfloor \frac{i\ast G}{E}\rfloor }{G}}$$

(21)

In this context, G represents the number of environmental groups, and E represents the total number of environments. For environments that have not been terminated, the advantage value obtained by the agent is calculated as follows:

$$\begin{array}{cc}\hfill \delta \left(t\right)& =r\left(t\right)+Critic\left(S_{t+1}\right)-Critic\left(S_t\right)\hfill \\ \hfill Adv\left(t\right)& =\delta \left(t\right)+\tau \times \gamma \times Adv(t+1)\hfill \end{array}$$

(22)

where $r\left(t\right)+V\left(S_{t+1}\right)$ represents the expected reward obtained by the agent after executing an action, while $Critic\left(S_t\right)$ denotes the expected reward for the agent prior to action execution. $\delta \left(t\right)$ signifies the advantage value of a single step action taken by the agent, and $A\left(t\right)$ represents the cumulative advantage value of the agent.

For terminated environments, the advantage value obtained by the agent is calculated as follows:

$$Adv\left(T\right)=r\left(T\right)-Critic\left(T\right)$$

(23)

After determining the A-values required for updating the agent, these can be substituted into the original formula to continue the agent’s training process.

Figure 3 below illustrates the advantages of this method and its distinctions from the original approach.The asterisk (*) denotes multiplication in figure.

In the most common practice, an agent sequentially interacts with a single environment for n rounds, wherein the time spent on agent–environment interactions is $n\times T$. If parallel environments are employed, utilizing more than n environments to simultaneously interact with the agent and collect data, the time spent on agent–environment interactions is reduced to T. However, adopting this approach, whether using n environments or $100n$ environments, the agent–environment interaction time remains T.

In the TAGRL method, environments are divided into different groups. For instance, if there are $100n$ environments, divided into 5 groups, each group contains $20n$ environments. After the warm-up phase, each environment only needs to advance $0.2T$ steps, thereby further reducing the interaction time between the agent and the environments.

This approach significantly enhances the efficiency of data collection and potentially accelerates the learning process. By strategically grouping environments and implementing a phased interaction strategy, TAGRL optimizes the trade-off between the breadth of environmental exploration and the depth of learning within each environment.

The following outlines the process of the TAGRL search Algorithm 2. The symbols utilized in this process are elucidated in

Table 2. Variable list.

Variable	Type	Size	Explanation
Width	Scalar	1	The number of environment grids in the x-axis direction
Height	Scalar	1	The number of environment grids in the y-axis direction
$TP{M}^t$	Matrix	$Height\times Width$	Each value of TPM expresses the probability of a target in each environment grid at time t
$a_t^u$	Scalar	1	The action of the UAV u at time t
${\alpha }_t^u$	Scalar	1	The orientation angle of the UAV u at time t
${\psi }_{max}^u$	Scalar	1	The maximum steering angle of the UAV u
$\overrightarrow{c_t^u}$	Vector	2	The position of the UAV u at time t
${\mathrm{speed}}^u$	Scalar	1	The speed of the UAV u
$Senso{r}_u(·)$	Function	1	The sensor function of the UAV u
$Precis{e}_u(·)$	Function	1	The precise function of the UAV u sensor
$D_t^u$	Plane	1	The detection range of the UAV u at time t
$p_t^u$	Scalar	1	The probability of the UAV u detecting the target at time t
T	Scalar	1	The total reconnaissance time
$N_u$	Scalar	1	The number of UAVs
$\mathbf{A}$	Tensor	$T\times N_u\times 2$	The action of the UAVs at each time
$return$	Scalar	1	The total reward of the UAVs
$R_t$	Scalar	1	The reward of the UAVs at time t
$J(·)$	Function	1	The objective function of the UAVs
$V(·)$	Function	1	The value function of the UAVs
$Critic$	Net&Function	1	Fitting the expected reward of the actor
$Actor|\pi$	Net&Function	1	The policy of the UAVs
$Criti{c}_t^{target}$	Scalar	1	The target value of the critic at time t
$Adv(·)$	Function	1	The advantage function of the UAVs
$S_t$	tuple	(TPM, ${\mathbf{c}}_{\mathbf{t}}$, $\overrightarrow{{\alpha }_t}$)	The state of the UAVs at time t
$GT(·)$	Function	1	The maximum simulation time of the grouped environment
$O_t$	tuple	(TPM, ${\mathbf{c}}_{\mathbf{t}}$, $\overrightarrow{{\alpha }_t}$, id:Scalar)	The observation of the UAVs at time t

.

Algorithm 2 TAGRL Search Algorithm

Initialize the parameter $\theta$ of actor network $\pi$ using orthogonal initialization Initialize the parameter v of $Critic$ network using orthogonal initialization Initialize total iterations I Let us denote $t_{max}$ as the time limit for drone search operations. Let us denote E as the number of environments Let us denote g as the number of environment groups Let us denote n as the maximum simulation turns for a drone Let us denote $D^u$ is u reconnaissance coverage. $\mathbf{Warm}\mathbf{up}\mathbf{stage}$: $\mathbf{For}$ each environment e in 1 …E     Get $\overrightarrow{O_0}$ from environments     Calculate $GT\left(i\right)$ by Equation (21)     $\mathbf{For}$ each t in 1 …$GT\left(i\right)$          execute average filtering on $TP{M}^e$ by Equation (1)          $\mathbf{For}$ each UAV u                 $distributio{n}_t^u$ = $\pi (o_t^u,\theta )$                 $a_t^u\sim distributio{n}_t^u$                 update $\overrightarrow{c_t^u}$ and ${\alpha }_t^u$ according to $a_t^u$ by Equation (3)                 update $TP{M}^e$ by Equation (2) $\mathbf{Train}\mathbf{stage}$: $\mathbf{For}$ each i in 1 …n     Get $S_0$,$\overrightarrow{O_0}$ from environments     Set buffer b = [ ]     $\mathbf{For}$ each t in 1 …${\displaystyle \frac{t_{max}}{G}}$         $\mathbf{For}$ each environment e             $r_t$ = 0             execute average filtering on $TP{M}^e$ by Equation (1).             $\mathbf{For}$ each UAV u                 $distributio{n}_t^u$ = $\pi (O_t^u,\theta )$               $a_t^u$∼$distributio{n}_t^u$               calculate $p_t^u$ by (5)               $r_t$ += $-ln(1-p_t^u)$               update $TP{M}^e$ by Equation (2)            $valu{e}_t$ = $Critic(S_t,v)$            b+=[$S_t$,$valu{e}_t$,$O_t$,$\overrightarrow{a_t}$,$r_t$]        Compute advantage estimate $Adv$ via Equations (22) and (23) on b        Compute reward-to-go R on b and normalize     Compute loss of $\pi$ by b,$Adv$ and Equation (11)     Compute loss of V by Equation (12)     Update $\pi$,V

5. Experiment

The experimental setup for this study consisted of a single NVIDIA GeForce RTX 4090 graphics card (NVIDIA Corporation, Santa Clara, CA, USA) and an Intel Core i9-14900K processor (Intel Corporation, Santa Clara, CA, USA). The software environment utilized Python 3.11.9. Neural networks were constructed using PyTorch 2.3.1, while the unmanned aerial vehicle (UAV) search environment was implemented using Numba 0.60.0.

To validate the effectiveness of the proposed method, we conducted comparative experiments with traditional approaches. In these experiments, both our method and the conventional approach utilized 256 environments to ensure equal computational resources. Both methods employed MAPPO [27] to solve the established optimal control model(OC-MAPPO) [25]. The primary difference was that our method (TAGRL) incorporated environmental grouping and asynchronous temporal interactions, while all other settings and net architecture remained consistent. To ensure a fair comparison, when evaluating the OC-MAPPO method, a discrete action design was implemented, wherein the UAVs’ movement patterns were limited to three options: forward, left, or right at each step. In contrast, for experiments demonstrating UAV trajectories, we employed the aforementioned continuous action model, where the UAVs’ movement patterns at each time point were determined by acceleration and angular acceleration. The

Table 3. TAGRL parameters.

Parameter	Value
$\gamma$	0.99
Horizon length	819
Update epochs	4
Learning rate	0.0003
Vector environment numbers	10
GAE lambda parameter	0.95
Learning rate	0.001
Value loss coefficient	0.05
Entropy coefficient	0.005
Eps clip	0.2
Hidden size	128
Environment number	256
Environment group number	5

presents the key parameter settings:

To mitigate the impact of random factors, each experiment was repeated 30 times, and error bands were plotted. The experimental results are illustrated in Figure 4.

In Figure 4, the orange error band represents the performance of the proposed method, while the blue curve indicates the performance when only using the parallel environment. It can be observed that agents in the TAGRL method learn faster and achieve higher average rewards. Furthermore, the TAGRL method and the OC-MAPPO method utilize the same number of computing cores, but the TAGRL method requires significantly less memory. This is because TAGRL stores only one-fifth of the interaction data per environment, whereas the OC-MAPPO method needs to store all the interaction data.

Subsequently, we conducted a comparative analysis of our method’s performance under varying numbers of parallel environments.

From the Figure 5, it is observed that when the number of parallel environments is relatively low, the learning curve of the agent exhibits greater fluctuations and achieves lower reward values. This phenomenon may be attributed to the fact that during the learning process in (9), the agent requires a stable V function as a foundation. When the number of environments is limited, the agent obtains less data from each round of interaction with the environment, making it challenging to learn an accurate and stable $V(·)$ function. Conversely, when the agent interacts with a large number of environments, the $V(·)$ function learned by the agent more closely approximates the true expected reward, and the policy learned through the (9) process more accurately reflects the correct policy.

To validate the comparative performance of our method under different search durations, we conducted experiments for four scenarios: $T=50,T=100,T=200$, and $T=500$. Each method was executed for 300 s, with the TAGRL method consistently employing five environmental groups. The optimal reward generated during the training process was selected as the representative reward for each method. The experimental results are shown in Figure 6.

The OC-MAPPO method lacks values for the case of $T=500$ and $E=512$ due to memory overflow. As observed in the figure, when $T=50$, the OC-MAPPO method performs comparably to TAGRL with fewer environments. However, as the number of environments increases, the MAPPO method demonstrates certain advantages. With further increases in the number of environments, TAGRL again exhibits performance similar to the OC-MAPPO method. When $T=100$, TAGRL shows slightly superior performance to OC-MAPPO, with TAGRL significantly outperforming OC-MAPPO when $E=32$, although this may be attributed to the stochasticity inherent in reinforcement learning. At $T=200$, TAGRL demonstrates a more pronounced advantage, likely due to its shorter simulation time per round, as the OC-MAPPO method can complete fewer rounds in the same time frame. When $T=500$, both the memory and time advantages of TAGRL become evident. The increase in simulation environments does not significantly constrain TAGRL in terms of memory usage, whereas the OC-MAPPO method experiences memory overflow at $E=512$. Due to the further reduction in simulation rounds, the performance of the OC-MAPPO method becomes more unstable. Overall, when the reconnaissance time for UAV is longer, TAGRL exhibits superior performance compared to the OC-MAPPO method.

To further validate the robustness of the algorithm, we conducted a comparative analysis of its performance under varying numbers of UAVs. We selected scenarios with 2, 3, and 5 UAVs for our comparative experiments. The results of these experiments are illustrated in Figure 7.

To validate the search strategy learned by the agent, we visualized the agent’s search trajectory and the changes in the target probability map during the search process. The drone’s search trajectory is composed of 200 discrete segments. The Figure 8 below illustrates the search trajectories learned by the agent.

To better comprehend the search trajectory of UAVs, we segmented the UAVs’ route for display and selected probability maps of the target at various time points for presentation.

Figure 9 illustrates that initially, both UAVs move towards the target point. Subsequently, the UAV closer to the target loss point begins to rotate approximately around a point to the right and behind the loss point, while the other UAV circles around the left front. Following this, the UAVs continue to rotate on either side of the target loss point. During this period, the Figure 10 indicates a significant decrease in the likelihood of the target appearing at or near the loss point. The red arrows in the figure indicate the forward direction of the UAV. However, after $t=100$, despite higher target probabilities in the upper and right regions of TPM, the UAVs fail to investigate these areas promptly. This observation highlights potential areas for improvement in the current method. Increasing training duration and expanding network parameters could potentially enhance the UAVs’ performance.

Despite the ability of agents to autonomously learn effective search strategies in dynamic long-term search tasks, current algorithms still have potential areas for improvement in application. Firstly, there exists an approximation error between the discrete TPM and the continuous world. Increasing the density of the grid in the TPM can reduce this error but also increases the algorithm’s memory consumption. A potential solution may be to focus on regions of interest within the TPM while compressing other areas. Secondly, the current research problems do not involve adversarial strategies for the target. The design of drone search strategies in scenarios where the target intentionally employs evasive tactics also deserves further consideration.Third, the current study assumes that UAVs fly at a fixed altitude, resulting in a constant scanning radius. However, in practical applications, UAVs can adjust their altitude to change their reconnaissance radius and precision. Removing this limitation may facilitate the design of more flexible search strategies.

6. Conclusions

This paper proposes a novel temporally asynchronous grouped environment reinforcement learning method. This approach enables UAVs to rapidly acquire long-term search strategies while reducing memory requirements. Agents trained on long-term search strategies may exhibit enhanced robustness in their policies compared to those focused on short-term strategy learning. Further investigation into the impact of this methodology on the robustness of UAVs is warranted. Once UAV policies demonstrate strong resilience across temporal and two-dimensional spatial domains, it becomes feasible to extend the search scenario to a three-dimensional space, ultimately facilitating deployment on physical UAV platforms for practical applications.