A Two-Stage Target Search and Tracking Method for UAV Based on Deep Reinforcement Learning

Abstract

To deal with the complexities of decision-making for unmanned aerial vehicles (UAVs) in denial environments, this paper applies deep reinforcement learning algorithms to search and rescue (SAR) tasks. It proposes a two-stage target search and tracking method for UAVs based on deep reinforcement learning, which divides SAR tasks into search and tracking stages, and the controllers for each stage are trained based on the proposed deep deterministic policy gradient with three critic networks (DDPG-3C) algorithm. Simulation experiments are carried out to evaluate the performance of each stage in a two-dimensional rectangular SAR scenario, including search, tracking, and the integrated whole stage. The experimental results show that the proposed DDPG-3C model can effectively alleviate the overestimation problem, and hence results in a faster convergence and improved performance during both the search and tracking stages. Additionally, the two-stage target search and tracking method outperforms the traditional single-stage approach, leading to a more efficient and effective decision-making ability in SAR tasks.

1. Introduction

Unmanned aerial vehicles (UAVs) are a class of aircraft that can fly autonomously without a need for an onboard pilot, which are notable for their convenience, flexibility, low cost, and wide range of applications [1]. UAVs operate without a pilot on board, allowing them to perform a variety of high-risk or inaccessible tasks in complex and changing environments, including terrain mapping, precision agriculture (PA), surveillance and reconnaissance, power line inspections, search and rescue (SAR), film shooting, aerial delivery, and military fields [2,3,4,5,6,7,8].

In the field of UAVs, the execution of numerous tasks fundamentally relies on target tracking. For instance, in security management tasks, it is imperative to track, encircle, and apprehend criminals when a terrorist event occurs. Under such circumstances, UAVs can be used to track criminals and even carry light weapons to assist police. Another vivid case is the search and rescue (SAR) tasks after a disaster happened. When natural disasters occur, the terrain in the affected area is complex, and communication is interrupted, posing significant challenges to SAR tasks. In such scenarios, UAVs can quickly enter these areas with advanced sensors like high-definition cameras and infrared thermal imagers to provide valuable information for SAR tasks, conduct aerial patrols, and perform target tracking with their aerial superiority.

With the rapid development of technology, UAVs have achieved significant improvements in intelligence, endurance, payload capacity, and resistance to interference, broadening their applicability across diverse scenarios. Despite these improvements, UAVs operating in SAR tasks often encounter unique challenges, such as infrastructure damage and severe natural conditions, which can disrupt communications. This disruption severely impacts the UAVs’ ability to perceive their environment, diminishing the success rate of rescues. Furthermore, in denial environments, the communication between UAVs and the ground command center will be disrupted, which cannot guarantee fast and safe communication. Therefore, each UAV is required to have the ability to independently and intelligently complete tasks under these severe environments, achieving a highly autonomous ability as much as possible.

The investigation into how unmanned aerial vehicles (UAVs) can make rapid and effective maneuvering decisions based on real-time observation information to accomplish tasks in unknown and complex environments efficiently has emerged as a significant area of research interest. In recent years, advanced methods such as model predictive control, optimization-based methods, and particle swarm optimization (PSO) algorithms have been successfully applied in UAVs. These algorithms empower UAVs to navigate and complete complex tasks within unfamiliar environments by improving route planning and decision-making processes.

Faced with denial environments without external positioning systems, Abraham et al. [9] presented a method to enable a quadrotor helicopter equipped with a laser rangefinder to autonomously explore and map unstructured indoor environments. Mac et al. [10] introduced a novel method combining sensor fusion for localization and an improved potential field method for UAVs in obstacle avoidance tasks, alongside an optimally tuned PID controller using multi-objective particle swarm optimization. However, their methods relied heavily on onboard sensors, which may limit performance in more complex or dynamically changing environments. Rothmund et al. [11] utilized scenario-based model predictive control for inspection drones to avoid obstacles in unknown environments. Although the UAV would change speed or take detours to avoid flying in potentially dangerous areas to mitigate risk, the system was limited by its reliance on a pre-planned path. At the same time, the high computational cost of this method made it difficult to guarantee real-time performance requirements. Similarly, Kulathunga et al. [12] introduced an optimization-based, trajectory-tracking approach for multi-rotor aerial vehicles operating in unknown environments. The method utilized a dual-planner system consisting of a global planner and a local planner. The global planner adjusted the initial reference trajectory to avoid obstacles, while the local planner focused on generating optimal control policies for following this adjusted trajectory. This approach was based on precise mathematical models, which could be a limitation in unknown environments where model parameters are often difficult to obtain accurately. Saccani et al. [13] introduced a multi-trajectory model predictive control for UAV navigation in unknown and static environments by using LiDAR for obstacle detection and path planning that balanced safety and goal achievement. To address the problem of coordinating USVs and UAVs for maritime parallel search, Li et al. [14] proposed a dynamic event-triggered control mechanism to reduce communication load and a sensor-tolerant control to handle sensor faults. This method was validated through simulation experiments, demonstrating its effectiveness in maintaining formation and achieving full coverage of the search area. However, this method depended on threshold tuning and assumed the states of the USV and UAV were known, which is difficult to implement in practice. Additionally, it did not consider obstacles, limiting its applicability in complex environments.

In recent years, with the rapid development of artificial intelligence (AI), reinforcement learning (RL), as a part of machine learning, has achieved remarkable results due to its excellent performance and wide application in the field of decision-making of complex tasks, such as autonomous vehicles, robot control, game strategy optimization, and financial transactions [15,16,17]. Unlike traditional optimal control methods, the agent in reinforcement learning learns a policy through interaction with the environment, aiming to maximize cumulative rewards over time. It does not necessarily require a known model of the system dynamics and focuses on adaptability through trial and error, which is more adaptable to handling uncertainties and unknown dynamics.

Pham et al. [18] integrated a PID controller with Q-learning, a type of reinforcement learning algorithm, to manage the UAV’s trajectory. This combination enabled the UAV to dynamically adjust its path based on real-time inputs, learning from the environment to enhance both navigation precision and efficiency. Li et al. [19] enhanced a UAV’s capability to swiftly adapt to unpredictable target movements by integrating deep reinforcement learning (DRL) with meta-learning. This approach significantly improved tracking accuracy and efficiency, making it particularly effective in diverse scenarios such as wildlife protection and emergency aid. When it comes to aerial robots in search and rescue tasks within unknown environments, Ramezani et al. [20] introduced an energy-aware hierarchical reinforcement learning approach by utilizing a predictive energy consumption algorithm to enhance the sustainability and endurance of missions. To address the real-time challenges in the decision-making process of UAVs, Zhao et al. [21] proposed adaptive and random exploration methods, ensuring that UAVs reach their target positions via reasonable and safe paths, enhancing their operational effectiveness and safety in dynamic environments. Kun et al. [22] utilized the deep deterministic policy gradient (DDPG) algorithm to direct UAVs toward fixed-point targets; however, their method did not address the guidance and tracking of moving targets. In response to this limitation, Alejandro et al. [23] developed a Gazebo-based reinforcement learning framework that effectively applied the DDPG algorithm for continuous UAV landing on a moving ship, which improved the UAV’s adaptability and performance in dynamic environments. However, the initial position of the UAV was fixed in their simulation, which could not effectively reflect the autonomy of guiding the maneuvering decision-making process. To address UAV flight control in dynamic environments with random wind turbulence, Ma et al. [24] introduced an incremental reinforcement learning (IRL)-based algorithm, which employed policy relief (PR) for exploration and significance weighting (SW) to enhance learning. This method was validated through simulations and real-world tests, demonstrating good tracking performance under wind disturbances though it was operated under the assumption of a fully known environment and did not account for collision avoidance. Xia et al. [25] proposed an end-to-end cooperative multi-agent reinforcement learning (MARL) scheme for UAV target tracking, addressing limitations such as unknown trajectories and limited flight performance. This paper modeled communication between UAVs in the swarm, improving coordination and efficiency. The method enhanced tracking with an energy-saving strategy and spatial information entropy, outperforming deep reinforcement learning baselines in simulations. However, it requires position information, assumes half-duplex communication, and may cause difficulties in communication-denied environments.

Faced with the complexities of autonomous decision-making for unmanned aerial vehicles (UAVs) in unknown environments, and to enable UAVs to autonomously make decisions based on limited observation information to adapt to unknown settings while maintaining strong generalization capabilities, this paper applies deep reinforcement learning algorithms to UAV search and rescue (SAR) tasks. A two-stage target search and tracking method for UAVs based on deep reinforcement learning is proposed. The novelties of this paper are listed as follows.

(1)

A deep deterministic policy gradient with three critic networks (DDPG-3C) algorithm is proposed to alleviate the overestimation problem in the critic network of the traditional DDPG algorithm. This method enhances both training speed and effectiveness by adopting three critic networks and introducing the experience replay buffer mechanism.

(2)

A two-stage target search and tracking method is introduced to enhance the search success rate and tracking performance of UAVs in SAR tasks within unknown environments. This method divides SAR tasks into the search stage and tracking stage, and the controllers for each stage are trained based on the proposed DDPG-3C algorithm.

(3)

A simplified two-dimensional SAR scenario is designed to demonstrate the practical application of the proposed methods and algorithms.

By introducing the DDPG-3C algorithm, this paper effectively alleviates the overestimation problem found in the traditional DDPG algorithm, leading to faster convergence and improved decision-making ability. The proposed two-stage target search and tracking method significantly improves the efficiency of UAV operations in SAR tasks, which not only provides a more efficient search strategy but also better adaptability to the target’s movements. Additionally, the development of a simplified two-dimensional SAR simulation environment provides a solid foundation for future research into more complex 3D environments.

The structure of this paper is as follows: Section 2 describes the search and rescue task and a simplified model of the SAR scenario is designed. Section 3 introduces the proposed deep deterministic policy gradient with three critic networks (DDPG-3C) and its training process. Section 4 introduces the proposed two-stage search and tracking method in SAR tasks. In Section 5, the effectiveness of the proposed DDPG-3C model is validated through numerous experiments, as well as the proposed two-stage target search and tracking method. Finally, some conclusions are drawn in Section 6.

2. Problem Description

2.1. Scenario Description

After a natural disaster, such as an earthquake, flood, or fire, the landscape is often devastated and full of challenges. Firstly, the terrain may change due to shaking or flooding, and roads and buildings may be severely damaged, with new obstacles such as fissures, craters, and inundated areas becoming prevalent. Additionally, the post-disaster environment may pose various potential hazards, such as communication disruptions, natural gas leaks, or secondary disasters triggered by unstable ground. These factors not only make it difficult to access affected areas but also significantly exacerbate the complexity of search and rescue (SAR) operations. In response, rescue teams have deployed advanced unmanned aerial vehicles (UAVs) for SAR tasks, as shown in Figure 1. These UAVs, such as the upgraded DJI Matrice 350 RTK (DJI, Shenzhen, China) [26], are equipped with sophisticated thermal imaging and night vision technologies, enabling effective operation in diverse and challenging conditions.

The task of these UAVs is to conduct aerial patrol and rescue. Once a UAV identifies trapped individuals or signs of life, it can swiftly switch to target tracking mode, which allows the UAV to continuously monitor the target to assist survivors or transport food, medicine, and other supplies.

Due to communication disruptions in disaster areas, these UAVs often operate in denial environments where communication with the command center is limited. These UAVs must rely on their onboard systems to perceive environments and execute tasks effectively. Furthermore, debris scattered around the disaster area, such as metal fragments and other materials, can pose threats to the radar and other systems of the UAVs. Therefore, the UAVs must avoid flying over areas that might disrupt their operational capabilities when executing search and rescue tasks.

2.2. Simplified Model

By simplifying the search and rescue scenario of the UAV in denial environments described in Section 2.1 to a two-dimensional search and rescue (SAR) scenario, the simplified model can be obtained, which is shown in Figure 2.

The simplified SAR scenario of the UAV is represented by a two-dimensional rectangular area with a length of $L$ and a width of $W$, where our UAV is represented by a dark blue UAV icon, and the target waiting to be rescued is represented by a blue circular icon. The green lines located at the boundary of the square area represent the barriers of the area. There are dark green circular areas with a radius of $R$, representing obstacles or danger zones. The UAV is equipped with sensors to observe the velocity and direction of approaching objects, whose sensing range is indicated by a red circle in Figure 2.

The positions and velocities of the UAV and the target are all random at the beginning. The target works at the initial velocity, and it is assumed that it will act on a completely elastic collision and continue to move when colliding with an obstacle or barrier. Our UAV acts by choosing a thrust vector to add their current velocity. The motion model of our UAV can be written as

$$\left\{\begin{array}{l}{v}_x(k+1)=v_x\left(k\right)+{\displaystyle \frac{F_x}{m}}△t\hfill \\ v_y(k+1)=v_y\left(k\right)+{\displaystyle \frac{F_y}{m}}△t\hfill \end{array}\right.,$$

(1)

where $\left[F_x,F_y\right]$ is the force vector generated by the thrust and is applied to the body in the horizontal $x$ and vertical $y$ coordinates with restricted value $F_x,F_y\in \left[-F_{\max },F_{\max }\right]$; $\left[v_x,v_y\right]$ is the velocity vector in the horizontal and vertical direction with a restricted value $v_x,v_y\in \left[-v_{\max },v_{\max }\right]$; $m$ is the mass of the UAV and $△t$ is the sampling interval.

The mission of our UAV is to go to great lengths to search for the target while avoiding obstacle areas and barriers. Once a target is detected, the UAV needs to approach it as quickly as possible and continuously track this target. Due to the communication denial environment being unable to communicate with the controlling center, the UAV can only make decisions independently based on the information obtained from sensors.

2.3. State and Action Space

In the SAR scenario, the UAV needs to perceive the current situation and make decisions independently based on the information obtained from sensors. A complicated state space will reduce the efficiency of the agent in extracting and understanding important information and weaken the generalization ability of the intelligent algorithm, while an overly simple state space cannot provide sufficient information for the agent to make optimal decisions. When designing the state space, the information including UAV, target, and obstacle is considered, and the state space is shown in Equation (2):

$$S=\left[{\mathrm{p}}_{\mathrm{UAV}},{\mathrm{v}}_{\mathrm{UAV}},{\mathrm{d}}_{\mathrm{target}},{\mathrm{v}}_{\mathrm{target}},{\mathrm{d}}_{\mathrm{obs}}\right],$$

(2)

where ${\mathrm{p}}_{\mathrm{UAV}},{\mathrm{v}}_{\mathrm{UAV}},{\mathrm{d}}_{\mathrm{target}},{\mathrm{v}}_{\mathrm{target}},{\mathrm{d}}_{\mathrm{obs}}\in {ℝ}^2$.

Table 1. Meaning of each dimension in the state space.

Symbol	Description	Normalized Values
${\mathrm{p}}_{\mathrm{UAV}}$	Position of UAV in the horizontal and vertical direction	$\left[-1,1\right]\times \left[-1,1\right]$
${\mathrm{v}}_{\mathrm{UAV}}$	Velocity of UAV in the horizontal and vertical direction	$\left[-1,1\right]\times \left[-1,1\right]$
${\mathrm{d}}_{\mathrm{target}}$	Horizontal and vertical distances of target detected by sensors	$\left[0,1\right]\times \left[0,1\right]$
${\mathrm{v}}_{\mathrm{target}}$	Horizontal and vertical velocities of target detected by sensors	$\left[0,1\right]\times \left[0,1\right]$
${\mathrm{d}}_{\mathrm{obs}}$	Horizontal and vertical distances of obstacle detected by sensors	$\left[0,1\right]\times \left[0,1\right]$

explains the meaning of each symbol in the state space. Sensors that do not sense any objects within their range report 0 for velocity and 1 for distance, which means that, if the objects (target, obstacles, or barriers) are not within the sensor range of the UAV, their velocity will be marked as 0 and their distance will be marked as 1.

The UAV has a continuous action space represented as a two-element vector $\left[F_x,F_y\right]$, which corresponds to the force generated by the horizontal and vertical thrust. Action values must be in the range $\left[-F_{\max },F_{\max }\right]$ by taking into account the maneuverability of the UAV.

3. Improved DDPG Algorithm

3.1. DDPG

Deep deterministic policy gradient (DDPG) is a reinforcement learning algorithm based on policy gradients [27] that combines the ideas from Deep Q-network (DQN) and Actor–Critic (AC) framework and can be used to solve decision-making problems in high-dimensional continuous action spaces. The main idea is to use deep neural networks (actor network and critic network) to fit behavior strategy functions and state-action value functions, which are called policy networks and value networks, respectively.

The actor network is responsible for mapping states $s_t$ to actions $a_t$:

$$a_t=\mu \left(s_t\left|{\theta }^{\mu }\right.\right),$$

(3)

where $s_t$ is the state at time $t$ and ${\theta }^{\mu }$ are the parameters of the actor network. This network outputs a deterministic action for any given state, aiming to maximize future rewards. The actions that it selects are based on its current policy, and, during the training period, it adjusts its parameters to improve the policy based on the gradients received from the critic network.

The critic network estimates the value of the state-action pairs:

$$Q\left(s_t,a_t\left|{\phi }^Q\right.\right)$$

(4)

where $s_t$ is the state, $a_t$ is the action, and ${\phi }^Q,$ are the parameters of the critic network. The critic network evaluates the expected return in a given state and takes a specific action according to the actor’s policy.

The actor network is trained to maximize the Q-values predicted by the critic network. The policy gradient used for updating the actor’s parameters is calculated as:

$${\nabla }_{{\theta }^{\mu }}J\approx {\displaystyle \frac{1}{N}}{\displaystyle \sum _i{\left.{\nabla }_{a}Q\left(s,a\left|{\phi }^Q\right.\right)\right|}_{s=s_i,a=a_i}}{\left.{\nabla }_{{\theta }^{\mu }}\mu \left(s\left|{\theta }^{\mu }\right.\right)\right|}_{s_i},$$

(5)

where ${\nabla }_{{\theta }^{\mu }}\mu \left(s\left|{\theta }^{\mu }\right.\right)$ is the gradient of the policy with respect to its parameters; ${\nabla }_{a}Q\left(s,a\left|{\phi }^Q\right.\right)$ is the gradient of the action-value function with respect to the action; $N$ represents the batch size, which is the number of experience tuples sampled from the experience replay buffer during each training iteration.

The critic network is updated by minimizing the squared Bellman error:

$$L={\displaystyle \frac{1}{N}}{\displaystyle \sum _i{\left(y_i-Q\left(s_i,a_i\left|{\phi }^Q\right.\right)\right)}^2},$$

(6)

where $y_i$ is the target value calculated using the rewards obtained and the next state values are estimated using the target critic network.

3.2. DDPG-3C

The DDPG algorithm, while widely applied in decision-making problems with continuous action spaces, is known to exhibit an overestimation bias in Q-values, which can hinder the convergence of the policy. This issue primarily resulted from maximization bias and target network update delays [28,29]. To alleviate the overestimation problem, this paper imitates the double Q-learning approach and applies it to the DDPG algorithm.

In this paper, an improved deep deterministic policy gradient with three critic networks (DDPG-3C) is proposed. The framework of the DDPG-3C decision model is shown in Figure 3.

As is shown in Figure 3, the DDPG-3C uses an Actor–Critic (AC) framework, and there are eight networks: one actor network ${\mu }^{\theta }$, one target actor network ${\mu }^{{\theta }^{\prime }}$, three critic networks $\left\{Q_1^{\phi },Q_2^{\phi },Q_3^{\phi }\right\}$, and three target critic networks $\left\{Q_1^{{\phi }^{\prime }},Q_2^{{\phi }^{\prime }},Q_3^{{\phi }^{\prime }}\right\}$. Our UAV can be treated as an agent containing a DDPG-3C decision model. When the agent obtains a state $s_i$ from the environment, the actor network outputs an action $a_i$ according to the input state and takes the next step $s_i^{\prime }$. Meanwhile, the environment returns a reward $r_i$. Then, the agent stores the data $\left(s,a,r,s^{\prime }\right)$ in the experience replay buffer. The agent updates its critic and actor networks by sampling mini-batches from the experience replay buffer.

The actor network is responsible for mapping states to actions. It attempts to learn a policy that maximizes future expected returns:

$$Loss=J\left({\theta }^{\mu }\right)=E_{s~{\rho }^{\mu }}\left[R_1\left|s_0=s,{\theta }^{\mu }\right.\right],$$

(7)

where $R_1$ denotes the reward from the state $s_0$; ${\rho }^{\mu }$ is the state distribution under the policy $\mu$; ${\theta }^{\mu }$ are the parameters of the actor network. To optimize this policy, the gradient of $J\left({\theta }^{\mu }\right)$ with respect to the parameters ${\theta }^{\mu }$ is computed. This is typically obtained using the policy gradient theorem [30], leading to the gradient for updates, as shown in Equation (5). It should be noted that when calculating the loss of the actor network, only the first critic network is used to predict the Q-value. This allows to reduce computational costs when all the estimates are not needed. The parameters of the target actor network are updated using a soft update rule weighted by a factor $\tau$:

$${\theta }^{{\mu }^{\prime }}=\tau {\theta }^{\mu }+(1-\tau ){\theta }^{{\mu }^{\prime }}.$$

(8)

Unlike traditional DDPG that uses a single critic network, DDPG-3C employs three critic networks $\left\{Q_1^{\phi },Q_2^{\phi },Q_3^{\phi }\right\}$ and three target critic networks $\left\{Q_1^{{\phi }^{\prime }},Q_2^{{\phi }^{\prime }},Q_3^{{\phi }^{\prime }}\right\}$ to independently estimate the Q-values. Three critic networks are identical, each trying to estimate the Q-value for a given state and action pair $\left(s,a\right)$. To reduce the impact of overestimation, DDPG-3C removes the highest of the three Q-value estimates and uses the average of the remaining two Q-values as the target Q-value:

$$y_t=r_t+{\displaystyle \frac{1}{2}}\gamma {\displaystyle \sum _{i=1}^2{Q}_i^{{\phi }^{\prime }}\left(s^{\prime },a^{\prime }\right)},$$

(9)

where $r_t$ is the reward obtained from the environment at step $t$; $\gamma$ is the discount factor, used to calculate the present value of future rewards; $a^{\prime }$ represents the action output by the target actor network for the next state $s^{\prime }$; $Q_i^{{\phi }^{\prime }}$ are the outputs from the two remaining target critic networks. The loss of three critic networks is

$$Loss=MSE\left(Q_1^{\phi },y_t\right)+MSE\left(Q_2^{\phi },y_t\right)+MSE\left(Q_3^{\phi },y_t\right),$$

(10)

where $MSE$ is short for mean squared error; $Q_i^{\phi }$ are the outputs from critic networks. Once the loss is computed, gradients are propagated back through each critic network to update the network parameters ${\phi }_i^Q$, leading to the following gradient for updates:

$${\nabla }_{{\phi }^Q}J({\phi }^Q)\approx {\displaystyle \frac{1}{N}}{\displaystyle \sum _i{\left.{\nabla }_{{\phi }^Q}\left(MSE\left(Q_1^{\phi },y_t\right)+MSE\left(Q_2^{\phi },y_t\right)+MSE\left(Q_3^{\phi },y_t\right)\right)\right|}_{s=s_i,a=a_i}},$$

(11)

where $N$ represents the batch size. ${\nabla }_{{\phi }^Q}$ is the gradient operator with respect to the network parameters ${\phi }^Q$. The parameters of three target critic networks are updated using a soft update rule weighted by a factor $\tau$:

$${\phi }^{Q^{\prime }}=\tau {\phi }^Q+(1-\tau ){\phi }^{Q^{\prime }}.$$

(12)

3.3. Training Procedure of DDPG-3C

The training procedures of the proposed DDPG-3C decision algorithm are shown in Algorithm 1.

Algorithm 1: Training process of the DDPG-3C decision algorithm

$\mathbf{Initialize}:\mathrm{actor}\mathrm{network}\mathrm{parameters}{\theta }^{\mu }$, $\mathrm{target}\mathrm{network}\mathrm{parameters}{\theta }^{{\mu }^{\prime }}$,         $\mathrm{critic}\mathrm{networks}\mathrm{parameters}\left\{{\phi }_1^Q,{\phi }_2^Q,{\phi }_3^Q\right\}$,         $\mathrm{target}\mathrm{critic}\mathrm{networks}\mathrm{parameters}\left\{{\phi }_1^{Q^{\prime }},{\phi }_2^{Q^{\prime }},{\phi }_3^{Q^{\prime }}\right\}$; $\mathbf{Input}:{\theta }^{\mu }$$,{\theta }^{{\mu }^{\prime }}$$,{\phi }_1^Q,{\phi }_2^Q,{\phi }_3^Q$$,{\phi }_1^{Q^{\prime }},{\phi }_2^{Q^{\prime }},{\phi }_3^{Q^{\prime }}$, learning rate $\beta$,      discount factor $\gamma$, $\mathrm{soft}\mathrm{update}\mathrm{factor}\tau$ $\mathbf{Output}:\mathrm{optimal}\mathrm{parameters}{\theta }^{\mu *},{\phi }_1^{Q*},{\phi }_2^{Q*},{\phi }_3^{Q*}$; $\mathbf{for}episode=1$ to $M$ do   $\mathrm{Receive}\mathrm{observation}{s}_0$;   $\mathbf{for}t=1$ $\mathrm{to}maxepisodelength$ do    $\mathrm{Select}{a}_t$ $\mathrm{w}.\mathrm{r}.\mathrm{t}\mathrm{the}\mathrm{current}\mathrm{actor}\mathrm{network}{\mu }^{\theta }\left(s_t\right)$;    $\mathrm{Execute}\mathrm{actions}{a}_t$ $\mathrm{and}\mathrm{observe}\mathrm{reward}{r}_t$ $\mathrm{and}\mathrm{new}\mathrm{state}{s}_{t+1}$;    Store $\left(s_t,a_t,r_t,s_{t+1}\right)$ in experience replay buffer $D$;    Sample a random minibatch from $D$;    Calculate the target Q-value as Equation (9);    Update critic by minimizing the loss as Equation (10);    Update actor using the sampled policy gradient as Equation (5);    Update target network parameters as Equations (8) and (12);   end for end for

4. Two-Stage Target Search and Tracking Method

This paper proposes a two-stage target search and tracking method, which divides SAR tasks into search stage and tracking stage. In the search stage, the specific optimization of the search algorithm aims to cover broader areas and quickly find targets. Once a target is detected, the focus shifts to optimizing the ability to track dynamic targets, including adapting to changes in the target’s speed and direction. The controllers for each stage are trained based on the proposed DDPG-3C algorithm. The diagram of the proposed two-stage target search and tracking method is shown in Figure 4. When implementing, if the target is not within the radar range of the UAV, the UAV will make decisions according to the model trained in the cover scenario to search for targets. If the target is detected by the radar, the UAV will turn to tracking mode.

By separating the task into two stages, the complexity of the state and action spaces for each stage is reduced, making the training process more efficient and faster to converge. Each controller can be optimized for a relatively smaller and more specific set of tasks, which improves learning efficiency. Additionally, a two-stage method allows each stage to be independently adjusted and optimized, making it easier to adapt to environmental changes and task requirements. What’s more, if an issue arises in either the search or tracking models, this method makes it easier to pinpoint and adjust the problem without affecting other parts of the system.

4.1. Search Stage

In the search stage, the goal is to quickly and efficiently cover the entire two-dimensional rescue area using the radar while avoiding obstacles and barriers in the area. The DDPG-3C decision model described in Section 3.2 is utilized for training the UAV (agent). The structure of the critic network and actor network is depicted in Figure 5.

The training procedure is described in Algorithm 1. In the search stage, the state is set as

$$S=\left[{\mathrm{p}}_{\mathrm{UAV}},{\mathrm{v}}_{\mathrm{UAV}},{\mathrm{d}}_{\mathrm{obs}}\right],$$

(13)

where ${\mathrm{p}}_{\mathrm{UAV}},{\mathrm{v}}_{\mathrm{UAV}},{\mathrm{d}}_{\mathrm{target}}\in {ℝ}^2$ and Table 1 explains the meaning of each symbol in the state space. The UAV has a continuous action space represented as a two-element vector $\left[F_x,F_y\right]$, which corresponds to the force generated by the horizontal and vertical thrust.

The rewards in this search scenario are comprised of step rewards and terminal rewards aiming to cover the entire two-dimensional rescue area. Each step incurs a step reward $r_{step}=-0.1$ to encourage the agent to complete the coverage task quickly. At the same time, each step incurs a coverage reward $r_{coverage}$ meaning that covering new areas grants additional rewards. The coverage reward is calculated as

$$r_{coverage}=D_{sensor}\cap {\overline{D}}_{covered},$$

(14)

where $D_{sensor}$ is the area currently covered by the UAV’s sensor, and ${\overline{D}}_{covered}$ is the complement of the previously covered area (i.e., areas that have not yet been covered). The logical AND ($\cap$) ensures that only the areas currently covered by the sensor, but not previously covered, are considered—these are the newly covered areas. At the end of every episode, the agent (UAV) receives a reward $r_{complete}=300$ and marks the task as completed when the total covered area reaches 95%. Meanwhile, if the agent (UAV) is too close to barriers or obstacles, the agent will be penalized with a reward $r_{collision}=-200$, and this episode is terminated. The reward function is shown as follows:

$$R={\displaystyle \sum r_{step}}+p{\displaystyle \sum r_{coverage}}+r_{complete}+r_{collosion},$$

(15)

where $p=0.001$ is the reward ratio; $r_{step}$ and $r_{coverage}$ are step rewards; $r_{complete}$ and $r_{collosion}$ are terminal rewards.

When implementing the decision model, it only needs to load the structure and parameters of the actor network. The actor network receives a state from the environment. This state should be in the same format and should be preprocessed in the same way as Equation (13) used during training. Subsequently, the action is output by the actor network and then executed by the UAV in the environment. The implementation process of the search operation is shown in Figure 6.

It should be noted that there is no target in the search scenario when the decision model is training. The goal of the UAV is to quickly and efficiently cover the entire two-dimensional rescue area. During implementation, a target is moving and the UAV will go to great lengths to search for this target.

4.2. Tracking Stage

Once the target is identified during the search stage, the system transfers to the tracking stage. Here, the controller focuses on maintaining continuous surveillance of the target, adjusting to its movements. The DDPG-3C decision model described in Section 3.2 is utilized for training the UAV (agent). The structure of the critic network and actor network is depicted in Figure 5, which is similar to the model in the search stage.

In the tracking stage, to better adapt to the target’s dynamics and maintain consistent tracking, the state is set as

$$S=\left[{\mathrm{p}}_{\mathrm{UAV}},{\mathrm{v}}_{\mathrm{UAV}},{\mathrm{d}}_{\mathrm{target}},{\mathrm{v}}_{\mathrm{target}},{\mathrm{d}}_{\mathrm{obs}}\right],$$

(16)

where ${\mathrm{p}}_{\mathrm{UAV}},{\mathrm{v}}_{\mathrm{UAV}},{\mathrm{d}}_{\mathrm{target}},{\mathrm{v}}_{\mathrm{target}},{\mathrm{d}}_{\mathrm{obs}}\in {ℝ}^2$ and Table 1 carefully explains the meaning of each symbol in the state space. Similarly, the UAV has a continuous action space represented as a two-element vector $\left[F_x,F_y\right]$.

To encourage the UAV to move closer to the target, with each step, the agent receives a reward $r_{pro}$ for proximity to the target, calculated as the Euclidean distance between the UAV and the target:

$$r_{pro}={\displaystyle \frac{5}{\Vert {\mathrm{d}}_{\mathrm{target}}\Vert +0.5}},$$

(17)

where ${\mathrm{d}}_{\mathrm{target}}\in {ℝ}^2$ represents the horizontal and vertical distances of the target detected by the UAV’s sensor. Adding 0.5 to the denominator prevents excessively high rewards and division by zero errors when the distance is very small. If the agent is too close to barriers or obstacles, it will receive a penalty $r_{collision}=-200$, and this episode is terminated. The reward function is shown as follows:

$$R={\displaystyle \sum r_{pro}}+r_{collosion}.$$

(18)

It should be noted that when the model is training, it is assumed that the UAV can perceive the states of the environment (target, obstacles, and barriers), regardless of whether objects are within the radar range of the UAV because the goal of this period is to train the tracking capacity of the UAV. When implementing this trained model, once the target enters the radar range of the UAV, the UAV switches from search mode to tracking mode. The implementation process of the tracking operation is similar to that of the searching stage.

5. Experimental Simulations

In order to validate the effectiveness of the proposed DDPG-3C model that utilizes three critic networks to alleviate the overestimation problem, the performance of the proposed model is compared with two popular reinforcement learning baselines in the two-dimensional search and rescue (SAR) scenario described in Section 2: DDPG [27] and TD3 (twin delayed deep deterministic policy gradient) [29]. At the same time, to validate the advantages of the proposed two-stage target search and tracking method, this paper compares it with the traditional single-stage reinforcement learning model in the SAR scenario.

When training the DDPG decision model, TD3 decision model, and the DDPG-3C decision model, the hyperparameters are set according to

Table 2. Hyperparameters of the network.

Name	Value
Input vector size of the critic network	search stage: 8; tracking stage: 12; traditional single-stage: 12
Hidden layer size of the critic network	400, 300
Output vector size of the critic network	1
Activation function of the critic network	ReLU, ReLU
Learning rate of the critic network	0.001
Optimizer of the critic network	Adam
Input vector size of the actor network	search stage: 6; tracking stage: 10; traditional single-stage: 10
Hidden layer size of the actor network	400, 300
Output vector size of the actor network	2
Activation function of the actor network	ReLU, ReLU, Tanh
Learning rate of the actor network	0.001
Optimizer of the actor network	Adam
Train step size	500
Batch size	1024

.

In the following section, the DDPG decision model, the TD3 decision model, and the DDPG-3C decision model will be applied to the search stage and tracking stage of the SAR task, as well as the traditional single-stage method for comparison. All of these algorithms are implemented in Python and run on a desktop computer with a GeForce RTX 4090 GPU and a 13th 64-Core i9-13900KS CPU.

5.1. Search Stage Simulations

In the search stage, there is no target in this scenario when the decision model is training. The designed SAR scenario includes a single UAV with a sensor range of 200 m and two obstacles, each with a radius of 50 m. Both the obstacles and the UAV are positioned at randomly generated locations within a two-dimensional rescue area that spans 0 to 800 m. The obstacles are static, while the UAV operates based on the force generated by the horizontal and vertical thrust, with a maximum action value of 400 N. The hyperparameters for the DDPG, TD3, and DDPG-3C decision models are listed in Table 2.

The training results of the DDPG-3C decision model, TD3 decision model, and the DDPG decision model in the search stage of the SAR task in denial environments are shown in Figure 7. The results are calculated using an exponential moving average (EMA) to help visualize trends more clearly by reducing noise from the raw data according to Equation (19):

$$y_t=\left\{\begin{array}{ll}{x}_t,\hfill & t=1\hfill \\ s\times y_{t-1}+\left(1-s\right)\times x_t,\hfill & t>1\hfill \end{array}\right..$$

(19)

where $s=0.95$ is the smoothing factor where closer to 1 implies heavier smoothing, and closer to 0 implies lighter smoothing. Moreover, the detailed convergence rewards and convergence steps are listed in

Table 3. Evaluation metrics about training process in search stage.

Model	Convergence Rewards	Convergence Steps
DDPG	~600 *	~2.8 M *
TD3	~560 *	~4 M *
DDPG-3C	~560 *	~7 M *

* Tilde (“~”) symbol conveys that the values are approximate and 1 M represents 1,000,000.

.

It can be seen from Figure 7a and Table 3 that the DDPG-3C model, which utilizes three critic networks, shows a reward trajectory that steadily increases and stabilizes at approximately 600, indicating robust learning and performance stability. At the same time, the DDPG-3C model achieves optimal performance more rapidly than its counterparts. In contrast, the traditional DDPG model, which uses a single critic network, reaches a lower peak reward of around 560 and takes approximately 7 million training steps to converge, showing a slower learning rate compared with the DDPG-3C model. Similarly, the TD3 model, which uses twin critic networks, stabilizes at a reward of approximately 560 but requires about 4 million training steps to converge, demonstrating slower convergence compared to DDPG-3C but faster than DDPG.

When it comes to the estimation of target Q-values, the red line in Figure 7b represents the target Q-value for the DDPG-3C model. This value is calculated by discarding the highest of the three Q-value estimates and averaging the remaining two. Figure 7b shows that the target Q-value for DDPG-3C is consistently lower than that of the traditional DDPG and TD3, indicating that the proposed decision model can effectively alleviate the problem of overestimation.

To compare the performance of these three models during the search stage, each model will be tested across 10,000 episodes. The average coverage ratio and average collision rate will be calculated to evaluate their effectiveness and safety. All models are initialized under the same conditions to maintain fairness in testing. The results are shown in

Table 4. Comparison of collision rate and coverage rate in search stage.

Model	Collision Frequency	Collision Rate	Coverage Rate
DDPG	543	5.43%	87.92%
TD3	461	4.61%	89.14%
DDPG-3C	349	3.49%	90.17%

. For the simulation video of the search stage, please click here (https://easylink.cc/5r3njl, accessed on 25 September 2024) to view (Supplementary Materials).

According to the data presented in Table 4, the DDPG model recorded a collision frequency of 543, meaning that collisions occurred in approximately 5.43% of the episodes. The TD3 model demonstrated an improvement with a collision frequency of 461 (4.61%). However, the DDPG-3C model demonstrated a significantly lower collision frequency with a total number of 349 collisions (3.49%). This improvement indicates that the DDPG-3C decision model is more effective at avoiding obstacles, which is safer during the search stage of SAR tasks. In terms of area coverage, the traditional DDPG model achieved a coverage rate of 87.92%, while the TD3 model achieved a slightly better coverage rate of 89.14%. However, the DDPG-3C model outperformed both, achieving a higher coverage rate of 90.17%. This indicates not only a more thorough coverage per episode but also improved efficiency in searching.

5.2. Tracking Stage Simulations

In the tracking stage, the goal is to maintain continuous surveillance of a moving target within a defined two-dimensional area spanning 0 to 800 m. The scenario includes a single UAV that has complete perception of the environment’s states. The environment also contains two static obstacles, each with a radius of 50 m, randomly placed in the area. The UAV operates based on the horizontal and vertical thrust forces, with a maximum action value of 400 N, to keep track of the target without colliding with the obstacles. By providing the UAV with full environmental awareness, the model can learn the optimal strategies for tracking and obstacle avoidance without the constraints of sensor range.

The DDPG, TD3, and DDPG-3C models are trained under the same conditions using hyperparameters listed in Table 2. Training results are analyzed using an exponential moving average (EMA) according to Equation (19) to smooth the data and highlight trends more effectively. Moreover, the detailed convergence rewards and convergence steps are listed in

Table 5. Evaluation metrics about training process in tracking stage.

Model	Convergence Rewards	Convergence Steps
DDPG	~200 *	~3 M *
TD3	~180 *	~4 M *
DDPG-3C	~160 *	~7 M *

* Tilde (“~”) symbol conveys that the values are approximate and 1 M represents 1,000,000.

.

Figure 8a,b displays the rewards and target Q-value trajectories for the DDPG, TD3, and DDPG-3C decision models. Similar to the results from the search stage, the DDPG-3C, with its three critic networks, is expected to converge more quickly, achieving this at 3 million training steps, and gain a higher reward of approximately 200. In contrast, the TD3 model takes about 4 million training steps to converge, with a reward of approximately 180, and the traditional DDPG model converges at around 7 million training steps with a lower reward of roughly 160. These results suggest that DDPG-3C shows better adaptability to the target’s movements compared to the other models. The target Q-values for DDPG-3C, calculated by discarding the highest and averaging the remaining two estimates, are relatively lower than those of the traditional DDPG and TD3. This approach offers a more conservative and accurate estimation, thus reducing the risks of overestimation.

To compare the performance of these three models during the tracking stage, each model will be tested across 10,000 episodes. The average collision frequency and the tracking success rate, defined as the percentage of episodes where the UAV successfully tracks and maintains the target within a range of 20 m, will be calculated and compared. All models are initialized under the same conditions to maintain fairness in testing. The results are shown in

Table 6. Comparison of collision rate and tracking success rate in tracking stage.

Model	Collision Frequency	Collision Rate	Tracking Success Frequency	Tracking Success Rate
DDPG	316	3.16%	9357	93.57%
TD3	274	2.74%	9397	93.97%
DDPG-3C	189	1.89%	9512	95.12%

. For the simulation video of the tracking stage, please click here (https://easylink.cc/5r3njl, accessed on 25 September 2024) to view.

According to the data presented in Table 6, collisions occurred in approximately 3.16% of the episodes when implementing the DDPG decision model in the tracking stage, while the TD3 model showed an improvement with a collision frequency of 274 (2.74%). The DDPG-3C decision model demonstrated an even lower collision frequency of 189, with a collision rate of 1.89%. In terms of tracking success, the DDPG model achieved a tracking success frequency of 9357 out of 10,000 episodes, while the TD3 model performed slightly better with a tracking success frequency of 9397 (93.97%). However, the proposed DDPG-3C decision model outperformed both, achieving a tracking success rate of 95.12%. This higher success rate for DDPG-3C suggests that the model, with its additional critic networks, provides more accurate value estimates and decision-making ability compared to DDPG and TD3.

5.3. Whole Period

In this experiment, the whole period of the SAR tasks was considered, which combines the search stage and tracking stage. Initially, the UAV operates in search mode, aiming to extensively cover a defined two-dimensional area (0 to 800 m) to detect any potential targets. During this stage, the UAV lacks target and obstacle information and relies on its sensor system to detect obstacles and other objects. Once a target enters the UAV’s radar range (within 200 m), the UAV switches to tracking mode and continuously tracks the moving target. The search stage and tracking stage use the DDPG-3C decision model trained in Section 5.1 and Section 5.2, respectively. The implementation process of the whole period is shown in Figure 4.

Figure 9 shows the experimental results of the proposed two-stage target search and tracking method for SAR tasks, which include avoiding obstacles, searching for targets, and tracking the target. In Figure 9a–c, the UAV’s initial position is (700, 700), with a random initial angle, and an initial speed of 0. The target’s initial position is (100, 100), with a speed of 30 m/s and an initial angle is $\pi /8$. The positions of the obstacles are (200, 200) and (400, 400). In Figure 9d–f, the UAV’s initial position is also (700, 700), with a random initial angle and an initial speed of 0. The target’s initial position is (100, 100), with a speed of 30 m/s and an initial angle of $\pi /4$, while the obstacles are located at (200, 80) and (350, 200).

It can be seen from Figure 9a,d that the UAV is positioned randomly and continues to search the environment for potential targets. After the UAV detects a target, as illustrated in Figure 9b,e, it transitions into tracking mode. The UAV adjusts its path to approach the target while avoiding obstacles. In the tracking stage, as shown in Figure 9c,f, the UAV not only maintains its course towards the target but also adapts to changes in the target’s speed. Additionally, it consistently avoids obstacles, as demonstrated in Figure 9c. For the simulation video of the whole stage, please click here (https://easylink.cc/5r3njl, accessed on 25 September 2024) to view.

Figure 10 presents the experimental results of the proposed two-stage target search and tracking method for SAR tasks, featuring scenarios with 5, 9, and 13 obstacles, respectively. In each scenario, the UAV’s initial position is (700, 700), with a random initial angle and an initial speed of 0, and the target’s speed is 30 m/s. In Figure 10a, the target’s initial position is (400, 500), with an angle of $17\pi /16$, and the obstacles are located at (100, 100), (100, 700), (700, 100), (700, 700), and (400, 400). In Figure 10b, four additional obstacles are added, positioned at (400, 100), (100, 400), (700, 400), and (400, 700), and the target’s initial position is (400, 500), with an angle of $13\pi /16$. In Figure 10c, the number of obstacles is increased to 13, which are evenly distributed throughout the SAR scenario, and the target moves at a speed of 30 m/s along a predetermined trajectory.

It can be seen from the figure that in SAR scenarios with varying numbers of obstacles, the UAV successfully avoids obstacles during the search stage while covering the SAR area to locate the target. Once the UAV’s radar detects the target, it switches to tracking mode. During the tracking stage, the UAV not only effectively tracks the target but also continuously avoids obstacles. For the simulation video of the whole stage, please click here (https://easylink.cc/5r3njl, accessed on 25 September 2024) to view.

To better illustrate the effectiveness of the proposed two-stage target search and tracking method for SAR tasks, this paper compares models trained using the two-stage target search and tracking method with the model trained using a traditional single-stage reinforcement learning method. In traditional single-stage SAR tasks, the UAV is required to search for and track the target as soon as possible while avoiding obstacles. The traditional single-stage method utilizes the DDPG-3C model. The hyperparameters of DDPG-3C model for the traditional single-stage method are similar to the DDPG-3C decision models in the search stage and tracking stage. The state is set as Equation (2). The reward function is shown as follows:

$$R={\displaystyle \sum r_{pro}}+r_{search}+r_{collosion},$$

(20)

where $r_{pro}$ is the proximity reward, calculated as the Euclidean distance between the UAV and the target, as shown in Equation (17); $r_{search}=100$ represents the reward for finding the target; $r_{collision}=-200$ represents the collision reward for being too close to barriers or obstacles.

When comparing the performance of the two-stage target search and tracking method with the traditional single-stage method, each model was tested across 20,000 episodes. The average number of steps taken to find the target, defined as the target being detected by the UAV’s sensor, and the average steps for successfully tracking the target, defined as the UAV maintaining the target within a 20-m range, were calculated and compared. Additionally, collision rates were compared. Both models are initialized under the same conditions to maintain fairness in testing. The results are presented in

Table 7. Comparison of average steps and collision rate in the whole period.

Method	Average Steps (Finding the Target)	Average Steps (Tracking the Target)	Collision Rate
Two-stage method	171.27	354.54	4.07%
Traditional single stage	313.78	481.68	4.51%

. To view the comparison video of these two methods, please click here (https://easylink.cc/5r3njl, accessed on 30 September 2024).

According to the data presented in Table 7, when using the two-stage target search and tracking method, which divides SAR tasks into search and tracking stages, requires an average of 171.27 steps to find the target. This is significantly fewer than the traditional single-stage method, which takes an average of 313.78 steps. A more efficient search strategy of the two-stage target search and tracking method, due to specialized training in the search stage, is suggested. Once the target is detected, the two-stage target search and tracking method takes an average of 354.54 steps to track the target, maintaining it within a 20-m range. However, the traditional single-stage method requires more steps, averaging 481.68, to maintain tracking of the target. This means that the UAV is able to operate more effectively in dynamic environments by using the two-stage target search and tracking method. In terms of collision rates, the traditional single-stage method has a slightly higher collision rate of 4.51% compared with 4.07% for the two-stage target search and tracking method, indicating not only faster but also safer operations compared with the traditional method.

It should be noted that although the two-stage target search and tracking method shows overall better efficiency in finding and tracking targets compared with the traditional single-stage method, there is no significant difference in the number of steps taken from detecting to successfully tracking the target between these two methods. To be more specific, when measuring the steps from finding the target to tracking it, the traditional single-stage method requires fewer steps (167.9 steps) than the two-stage method (183.27 steps). This can be attributed to a state transition process from the search stage to the tracking stage. For the two models trained with the two-stage method, during this transition, the state of the UAV in the search stage acts like noise, which the tracking stage must overcome, requiring additional steps to stabilize and effectively track the target.

6. Conclusions

To deal with the complexities of decision-making for unmanned aerial vehicles (UAVs) in unknown environments, this paper applies deep reinforcement learning algorithms to search and rescue (SAR) tasks. A two-stage target search and tracking method for UAVs based on deep reinforcement learning is proposed, which divides SAR tasks into search and tracking stages, and the controllers for each stage are trained based on the proposed deep deterministic policy gradient with three critic networks (DDPG-3C) algorithm. Simulation experiments are carried out to evaluate the performance of each stage in a two-dimensional rectangular SAR scenario, including search, tracking, and the integrated whole stage. Some concluding remarks are drawn below:

(1)

The deep deterministic policy gradient (DDPG) algorithm, while effective for continuous action spaces, exhibits overestimation bias in Q-values in the search and tracking stages of SAR tasks. This overestimation slows the convergence of the policy and leads to less reliable decision-making.

(2)

The proposed DDPG-3C model, which incorporates three critic networks, addresses the overestimation problem by removing the highest of the three Q-value estimates and uses the average of the remaining two Q-values as the target Q-value. This improvement leads to more accurate Q-value estimations, resulting in faster convergence and improved performance during both the search and tracking stages. Therefore, the proposed method not only provides a more efficient search strategy but also better adaptability to the target’s movements, improving overall efficiency.

(3)

The twin delayed deep deterministic policy gradient (TD3) algorithm, which uses twin critic networks, can also alleviate the overestimation problem to some extent and demonstrates good performance in both avoiding collisions and achieving a high coverage rate. However, the DDPG-3C model surpasses TD3 in these areas, offering even better obstacle avoidance and coverage rate, as well as more robust performance.

(4)

The two-stage target search and tracking method proposed in this paper outperforms the traditional single-stage approach. According to the average steps to finding and tracking the target and the collision rates, the proposed method offers not only faster and safer operations but also simplifies the process by eliminating the need for intricate reward function design. Additionally, it reduces the state vector size, which results in a more efficient and effective decision-making ability in SAR tasks.

Despite these improvements, the current study only considers simulations in a two-dimensional area, which limits its applicability to more complex real-world scenarios. Additionally, the UAV’s obstacle detection focuses on the nearest obstacle within the radar range when using the DDPG-3C decision model. This simplification may increase the likelihood of collisions when multiple obstacles are present within the radar’s detection range. Future work could explore the expansion of the model to consider target search and tracking in three-dimensional environments. Additionally, developing reinforcement learning methods with state embeddings could allow the state vector to better adapt to varying numbers of targets and obstacles. Furthermore, research could focus on increasing the number of UAVs and targets by using multi-agent reinforcement learning algorithms, enabling multiple UAVs to collaboratively search for and track targets.