Searching and Tracking an Unknown Number of Targets: A Learning-Based Method Enhanced with Maps Merging

Abstract

Unmanned aerial vehicles (UAVs) have been widely used in search and rescue (SAR) missions due to their high flexibility. A key problem in SAR missions is to search and track moving targets in an area of interest. In this paper, we focus on the problem of Cooperative Multi-UAV Observation of Multiple Moving Targets (CMUOMMT). In contrast to the existing literature, we not only optimize the average observation rate of the discovered targets, but we also emphasize the fairness of the observation of the discovered targets and the continuous exploration of the undiscovered targets, under the assumption that the total number of targets is unknown. To achieve this objective, a deep reinforcement learning (DRL)-based method is proposed under the Partially Observable Markov Decision Process (POMDP) framework, where each UAV maintains four observation history maps, and maps from different UAVs within a communication range can be merged to enhance UAVs’ awareness of the environment. A deep convolutional neural network (CNN) is used to process the merged maps and generate the control commands to UAVs. The simulation results show that our policy can enable UAVs to balance between giving the discovered targets a fair observation and exploring the search region compared with other methods.

1. Introduction

In the past decade, unmanned aerial vehicles (UAVs) have been widely used in military and civilian applications due to their low cost and high flexibility. Especially in search and rescue (SAR) missions, multiple UAVs working together can reduce mission execution time and provide timely relief to targets [1,2,3]. In a SAR mission, UAVs need to search out targets in an unknown region and continuously track them to monitor their status. However, in general, the number of targets is unknown and available UAVs are limited, which requires multiple UAVs to work together to keep track of the discovered targets while finding more unknown targets [4]. The problem of using robot teams to cooperatively observe multiple moving targets has been formalized first by Parker and Emmons [5], who termed this problem as Cooperative Multi-Robot Observation of Multiple Moving Targets (CMOMMT) and showed it is NP-hard.

Since the CMOMMT problem was raised, there has been a great deal of work to address it. A classical approach is the local force vector proposed by Parker and Emmons [5], in which a robot is subject to the attractive forces of nearby targets and the repulsive forces of nearby robots, and the direction of the robot’s motion is determined by the combined force of the two. However, this method will cause overlapping observations on the same target. Thus, Parker [6] proposed an improved method called A-CMOMMT to solve this phenomenon, where the robots are controlled by the weighted local force vectors for tracking targets. Additionally, in [7], the authors proposed B-CMOMMT, in which a help behavior is added to reduce the risk of losing a target. In [8], the authors proposed an algorithm called P-CMOMMT, considering the uniformity of the observation of the targets through the information entropy of targets’ observations. The methods based on local force vector lack the prediction of the targets’ behaviors and do not make full use of the targets’ historical position information, resulting in a low efficiency for searching and tracking the targets.

A large number of optimization-based methods have been proposed to solve CMOMMT-like problems [9,10,11,12]. In [13], the authors used group of vision-based UAVs to search for multiple ground targets. The objective is to optimize the collective coverage area and the detection performance based on a distributed probability map updating model, in which the probability of target existence is calculated through the measurement information and information sharing among neighboring agents. In [14], the authors proposed a multi-objective optimization approach based on genetic algorithm (GA) to minimize the mission completion time for a team of UAVs finding a target in a bounded area. In [15], the UAVs’ task sequence for a reconnaissance task assignment problem is considered, where the problem is formulated as a multi-objective, multi-constraint, nonlinear optimization problem solved with a modified Multi-Objective Symbiotic Organisms Search algorithm (MOSOS). In [16], searching and tracking an unknown ground moving target by multiple UAVs in an urban environment was modeled as a multi-objective optimization problem with preemptive priority constraints. The authors proposed a fuzzy multi-objective path planning method to solve this problem with target behavior predicted by extended Kalman filter (EKF) and probability estimation. In [17], the authors proposed a real-time path-planning solution enabling multiple UAVs to cooperatively search a given area. The problem is modeled as a Model Predictive Control (MPC) problem solved with Particle Swarm Optimization (PSO) algorithm. In [18], the authors emphasize the fairness of observations among different targets compared with the initial CMOMMT problem. They proposed an integer linear programming model to solve this problem where the motion of the targets is estimated in a Bayesian framework.

The above-mentioned approaches fail to balance between target searching and target tracking, which will make it difficult for UAVs to keep searching for undiscovered targets when the number of UAVs is less than the number of targets. To solve this problem, Li et al. [19] proposed a profit-driven adaptive moving targets search algorithm, which considers the impact of moving targets and collaborating UAVs in a unified framework through a concept called observation profit of cells. However, this approach assumes that the total number of targets is known, which is impractical in some complex environments. In [20], Dames proposed a method to enable multiple robots to search for and track an unknown number of targets. The robots use the Probability Hypothesis Density (PHD) filter to estimate the number of targets and the positions of the targets, and a Voronoi-based control strategy to search and track targets. This method assumes that each robot has a unique ID for creating a globally consistent estimate, which will limit the scalability of the robot team.

Recently, the development of deep reinforcement learning (DRL) [21] provides an alternative way to deal with the CMOMMT problem. DRL learns control policies through interacting with the environment, and it has reached or exceeded human levels in some game tasks [22,23]. There have been some studies using DRL to solve the targets search and tracking problem. In [24], the authors proposed a framework for searching for multiple static targets through a group of UAVs. The framework consists of a global planner based on a modern online Partially Observable Markov Decision Process (POMDP) solver and a local continuous-environment exploration controller based on a DRL method. In [25], the authors proposed a target following method based on deep Q-networks, considering visibility obstruction from obstacles and uncertain target motion. In [26], the authors proposed a DRL-based method to enable a robot to explore unknown cluttered urban environments, in which a deep network with convolutional neural network (CNN) [27] was trained by asynchronous advantage actor-critic (A3C) approach to generate appropriate frontier locations. In [28], the authors constructed a framework for automatically exploring unknown environments. The exploration process is decomposed into the decision, planning, and mapping modules, in which the decision module is implemented by a deep Q-network for learning exploration policy from the partial map.

In this paper, we focus on the problem of Cooperative Multi-UAV Observation of Multiple Moving Targets (CMUOMMT), where a UAV team needs to search and track an unknown number of targets in a search region. Our objective is to enable UAVs to give the discovered targets a fair observation and meanwhile maximize the exploration rate of the environment to discover more targets. To achieve this objective, the problem is formulated as a POMDP and solved with a DRL method. During the mission, each UAV maintains four observation history maps, which can reduce the partial observability of the environment. Furthermore, maps merging among UAVs can further improve awareness of the environment. To extract environmental features, a deep network with CNN is used to process each UAV observation map. A modern DRL method is used to train the shared policy with a centralized training, decentralized execution paradigm. The main contributions of this work are as follows:

• The average observation rate of the targets, the standard deviation of the observation rates of the targets, and the exploration rate of the search region are simultaneously optimized to enable multiple UAVs to cooperatively achieve fair observation of discovered targets and continuous search for undiscovered targets.
• Each UAV maintains four observation maps recording observation histories, and a map merging method among UAVs is proposed, which can reduce the partial observability of the environment and improve awareness of the environment.
• A DRL-based multi-UAV control policy is proposed, which allows UAVs to learn to balance tracking targets and exploring the environment by interacting with the environment.

The remainder of this paper is organized as follows. In Section 2, the problem is formulated and the optimization objectives are introduced. In Section 3, the details of our method are proposed, including the maps merging method and the key ingredients of the DRL method. In Section 4, simulation experiments are conducted and the results are discussed. Finally, we conclude this paper in Section 5.

2. Problem Formulation

In this paper, we consider the problem of CMUOMMT described in [6,19], which is shown in Figure 1 and defined as follows:

• A bounded two-dimensional rectangular search region S discretized into $C_L\times C_W$ equally sized cells, where $C_L$ and $C_W$ represent the number of cells in the length and width directions of the search region, respectively.
• The time step is discretized and denoted by t within a mission time duration T.
• A set of N moving targets $\mathcal{V}$ in S. For target ${\nu }_j({\nu }_j\in \mathcal{V},j=1,2,\cdots N)$, the cell that lies at time step t is denoted by $c_t\left({\nu }_j\right)\in S$. The mission is to observe these targets using multiple UAVs. To simplify this mission, we assume that the maximal speed of the targets is smaller than that of the UAVs.
• A team of M homogeneous UAVs $\mathcal{U}$ deployed in S to observe the targets. For UAV $u_i(u_i\in \mathcal{U},i=1,2,\cdots M)$, the cell that lies at time step t is denoted by $c_t\left(u_i\right)\in S$. Each UAV can observe the targets through its onboard sensor. The sensing range of each UAV is denoted by $d_s$. We assume that the UAVs are flying at a fixed altitude, and the size of the field of view (FOV) of each UAV is the same and remains constant. The term $FO{V}_t\left(u_i\right)$ denotes the FOV of the UAV $u_i$ at time step t. In addition, each UAV is equipped with a communication device to share information to coordinate with other UAVs. The communication range is denoted by $d_c$, which is assumed to be larger than the sensing range $d_s$. The UAVs can only share information with UAVs within a communication range. We further assume that all UAVs share a known global coordinate system.

The target ${\nu }_j$ is monitored when it is within the FOV of at least one UAV, which can be defined as

$$O_t\left({\nu }_j\right)=\left\{\begin{array}{c}1,\mathrm{if}\exists u_i,c_t\left({\nu }_j\right)\in FO{V}_t\left(u_i\right)\hfill \\ 0,\mathrm{else}\hfill \end{array}\right.$$

(1)

where $O_t\left({\nu }_j\right)$ indicates the observation state of target ${\nu }_j$.

During the mission, the observation rate of the target ${\nu }_j$ can be defined as

$$\eta \left({\nu }_j\right)=\frac{1}{T}\sum _{t=1}^T{O}_t\left({\nu }_j\right)$$

(2)

where $\eta \left({\nu }_j\right)$ represents the observation rate of the target ${\nu }_j$, which represents the proportion of time elapsed under the observation of at least one UAV during the mission.

The first objective for the UAV team is to maximize the average observation rate of N targets, which can be characterized by the metric $\overline{\eta }$:

$$\overline{\eta }=\frac{1}{N}\sum _{j=1}^N\eta \left({\nu }_j\right)$$

(3)

Maximizing $\overline{\eta }$ alone is unfair, especially when the number of UAVs is less than the number of targets, which may result in some targets not being observed during the mission. To solve this problem, the second objective for the UAV team is to minimize the standard deviation ${\sigma }_{\eta }$ of the observation rates of N targets:

$${\sigma }_{\eta }=\sqrt{\frac{1}{N}\sum _{j=1}^N\left(\eta \left({\nu }_j\right)-\overline{\eta }\right)}$$

(4)

A low value of ${\sigma }_{\eta }$ means that all targets are observed relatively uniformly during the mission. In addition, since the UAV team does not know the total number of targets, it needs to continuously explore the search region to discover new targets. Thus, the third objective for the UAV team is to maximize the exploration rate $\beta$ of the search region, which is defined as

$$\beta =\frac{1}{T}\frac{1}{C_L{C}_W}\sum _{k=1}^{C_L}\sum _{l=1}^{C_W}{t}_{stamp}\left(c_{kl}\right)$$

(5)

where $t_{stamp}\left(c_{kl}\right)$ represents the latest observed time for cell $c_{kl}$. k and l represent the indexes of the cell $c_{kl}$ in the length and width directions of the search region, respectively. The maximum value of $\beta$ is 1, which means that all cells in the search region are being observed by the UAV team at time step T. However, this is unrealistic since the maximum region observed by the UAV team is less than the total search region to be observed. That is,

$$\underset{u_i\in \mathcal{U}}{\cup }FOV\left(u_i\right)<S$$

(6)

The ultimate objective is a combination of $\overline{\eta }$, ${\sigma }_{\eta }$ and $\beta$, which is different from [6,19], whose objectives only consider the average observation rate $\overline{\eta }$. In this study, the UAV team needs to balance between giving the known targets a fair observation and exploring the search region through an efficient method.

3. Methods

3.1. Overview

We formulate the CMUOMMT problem as a POMDP and solve it with a DRL method. In this method, all UAVs share a global control policy $\pi$ to decide actions. The action is selected according to the observation from the environment, i.e., ${\mathbf{a}}_t\sim \pi \left({\mathbf{a}}_t\right|{\mathbf{o}}_t)$, ${\mathbf{o}}_t\sim \mathcal{O}\left({\mathbf{s}}_t\right)$, where ${\mathbf{s}}_t$ is the global state of the environment, ${\mathbf{o}}_t$ is the local observation of the environment state, $\mathcal{O}\left({\mathbf{s}}_t\right)$ is the observation function determined by the UAVs’ sensing range and communication range, and ${\mathbf{a}}_t$ is the selected action. The observation ${\mathbf{o}}_t$ includes four observation maps about the environment, which will be given in Section 3.2.

In a reinforcement learning (RL) framework, an RL agent learns an optimal policy ${\mathbf{a}}_t\sim {\pi }^{*}\left({\mathbf{a}}_t\right|{\mathbf{o}}_t)$ through interacting with the environment. The goal of the RL agent is to maximize a long-term accumulated reward

$$G_t=\sum _{k=0}^{\infty }{\gamma }^k{r}_{t+k+1}$$

(7)

where $r_{t+k+1}$ is the reward the RL agent received at time step $t+k+1$, $\gamma (0<\gamma <1)$ is the discount factor to make $G_t$ a bounded value.

In the proposed DRL method, we use a deep neural network ${\pi }_{\theta }\left({\mathbf{a}}_t\right|{\mathbf{o}}_t)$ parameterized by $\theta$ to approximate the UAVs’ control policy. The objective is to use a DRL method to find the optimal parameters ${\theta }^{*}$, which can make the UAV team balance between giving the known targets a fair observation and exploring the search region. The system architecture is shown in Figure 2.

As shown in Figure 2, each UAV first gets the observations from the environment through its onboard sensor to update its local observation maps. Then, each UAV receives the local maps of the other UAVs through its communication device, and the local maps are merged to provide the deep network ${\pi }_{\theta }$ an observation ${\mathbf{o}}_t$. Finally, the deep neural network ${\pi }_{\theta }$ outputs the action ${\mathbf{a}}_t$ to control the UAV and receives the reward $r_{t+1}$ at the next time step. The maps merging method is introduced in the next subsection, and the ingredients of deep reinforcement learning are introduced in Section 3.3.

3.2. Maps Merging

During the mission, each UAV maintains four observation maps:

(1) The observation map of the UAV’s position in the search region, denoted by a $C_L\times C_W$ matrix $M{S}_t\left(u_i\right)$, $M{S}_t\left(u_i\right)\in {\mathbb{R}}^2$, defined as follows:

$$M{S}_t\left(u_i\right)={\left[m{s}_t^{kl}\left(u_i\right)\right]}_{C_L\times C_W},m{s}_t^{kl}\left(u_i\right)=\left\{\begin{array}{c}1,\mathrm{if}{c}_t\left(u_i\right)=c_{kl}\hfill \\ 0,\mathrm{else}\hfill \end{array}\right.$$

(8)

(2) The observation history map of the cells, which records the latest observed time for each cell. This map is denoted by a $C_L\times C_W$ matrix $M{C}_t\left(u_i\right)$, $M{C}_t\left(u_i\right)\in {\mathbb{R}}^2$, defined as follows:

$$M{C}_t\left(u_i\right)={\left[m{c}_t^{kl}\left(u_i\right)\right]}_{C_L\times C_W}$$

(9)

The map $M{C}_t\left(u_i\right)$ is obtained in two steps. At each time step t, the map $M{C}_t\left(u_i\right)$ is first updated by the observation of the UAV $u_i$ on the subset of cells within $FO{V}_t\left(u_i\right)$, that is,

$$m{c}_t^{kl}\left(u_i\right)=t,\mathrm{for}{c}_{kl}\in FO{V}_t\left(u_i\right)$$

(10)

In addition, the observation history maps from other UAVs within a communication range will also update the local maps. The values of corresponding cells in the observation history map will be updated with the latest observation time as follows:

$$m{c}_t^{kl}\left(u_i\right)=m{c}_t^{kl}\left(u_j\right),\mathrm{if}m{c}_t^{kl}\left(u_j\right)>m{c}_t^{kl}\left(u_i\right)\mathrm{and}{d}_t(u_i,u_j)<d_c,u_j\in \mathcal{U},j\ne i$$

(11)

where $d_t(u_i,u_j)$ represents the distance between UAV $u_i$ and UAV $u_j$.

(3) The position history map of the other UAVs, which records the history positions of the other UAVs. This map is denoted by a $C_L\times C_W$ matrix $M{U}_t\left(u_i\right)$, $M{U}_t\left(u_i\right)\in {\mathbb{R}}^2$, defined as follows:

$$M{U}_t\left(u_i\right)={\left[m{u}_t^{kl}\left(u_i\right)\right]}_{C_L\times C_W},$$

(12)

$$m{u}_t^{kl}\left(u_i\right)=\left\{\begin{array}{c}1,\mathrm{if}{c}_t\left(u_j\right)=c_{kl}\mathrm{and}{d}_t(u_i,u_j)<d_c,u_j\in \mathcal{U},i\ne j\hfill \\ 0,\mathrm{if}t=0\hfill \end{array}\right.$$

(13)

At each time step, the map $M{U}_t\left(u_i\right)$ is updated by the observation history maps from other UAVs within a communication range as follows:

$$m{u}_t^{kl}\left(u_i\right)=m{u}_t^{kl}\left(u_j\right),\mathrm{if}m{u}_t^{kl}\left(u_j\right)>m{u}_t^{kl}\left(u_i\right)\mathrm{and}{d}_t(u_i,u_j)<d_c,u_j\in \mathcal{U},j\ne i$$

(14)

In addition, the knowledge about the positions of the other UAVs at the last observation time might be outdated. Thus, at the next time step, the values of cells in $M{U}_t\left(u_i\right)$ are decayed as follows:

$$m{u}_{t+1}^{kl}\left(u_i\right)=m{u}_t^{kl}\left(u_i\right)-\frac{1}{t_U},\mathrm{if}m{u}_t^{kl}\left(u_i\right)\ge \frac{1}{t_U}$$

(15)

where $t_U$ is a time constant, representing the decay period of the value of $m{u}_t^{kl}\left(u_i\right)$.

(4) The position history map of the targets, which records the historical positions of the targets. This map is denoted by a $C_L\times C_W$ matrix $M{T}_t\left(u_i\right)$, $M{T}_t\left(u_i\right)\in {\mathbb{R}}^2$, defined as follows:

$$M{T}_t\left(u_i\right)={\left[m{t}_t^{kl}\left(u_i\right)\right]}_{C_L\times C_W},$$

(16)

$$m{t}_t^{kl}\left(u_i\right)=\left\{\begin{array}{c}1,\mathrm{if}{c}_t\left({\nu }_j\right)=c_{kl}\mathrm{and}{c}_t\left({\nu }_j\right)\in FO{V}_t\left(u_i\right),{\nu }_j\in \mathcal{V}\hfill \\ 0,\mathrm{if}t=0\hfill \end{array}\right.$$

(17)

The map $M{T}_t\left(u_i\right)$ is also updated by the observation history maps from other UAVs within a communication range, that is,

$$m{t}_t^{kl}\left(u_i\right)=m{t}_t^{kl}\left(u_j\right),\mathrm{if}m{t}_t^{kl}\left(u_j\right)>m{t}_t^{kl}\left(u_i\right)\mathrm{and}{d}_t(u_i,u_j)<d_c,u_j\in \mathcal{U},j\ne i$$

(18)

Same as the map $M{U}_t\left(u_i\right)$, the values of cells in $M{T}_t\left(u_i\right)$ are decayed as follows:

$$m{t}_{t+1}^{kl}\left(u_i\right)=m{t}_t^{kl}\left(u_i\right)-\frac{1}{t_T},\mathrm{if}m{t}_t^{kl}\left(u_i\right)\ge \frac{1}{t_T}$$

(19)

where $t_T$ is a time constant, representing the decay period of the value of $m{t}_t^{kl}\left(u_i\right)$.

One example of four observation maps for UAV $u_0$ is shown in Figure 3.

3.3. Deep Reinforcement Learning

In this section, we introduce the key elements of the proposed DRL method, consisting of observation space, action space, network architecture, reward function, and training algorithm.

3.3.1. Observation Space

At time step t, the observation of UAV $u_i$ consists of four parts, i.e., ${\mathbf{o}}_t\left(u_i\right)=[{\mathbf{o}}_t^1\left(u_i\right),{\mathbf{o}}_t^2\left(u_i\right),{\mathbf{o}}_t^3\left(u_i\right),{\mathbf{o}}_t^4\left(u_i\right)]$.

• The observation ${\mathbf{o}}_t^1\left(u_i\right)$ is a part of the map $M{S}_t\left(u_i\right)$ centered in the UAV’s current cell $c_t\left(u_i\right)$, with length $C_{\mathrm{input}}$ and width $C_{\mathrm{input}}$. That is, ${\mathbf{o}}_t^1\left(u_i\right)$ is a $C_{\mathrm{input}}\times C_{\mathrm{input}}$ matrix, representing the positional relationship of UAV $u_i$ relative to the boundary of the search area S, which is defined as follows:

  $${\mathbf{o}}_t^1\left(u_i\right)={\left[{}^1{o}_t^{kl}\left(u_i\right)\right]}_{C_{\mathrm{input}}\times C_{\mathrm{input}}},$$

  (20)

  $${}^1{o}_t^{kl}\left(u_i\right)=\left\{\begin{array}{c}1,\mathrm{if}{c}_t\left(u_i\right)=c_{kl}\mathrm{or}{c}_{kl}\notin S,k=1,2,\cdots C_{\mathrm{input}},\hfill \\ l=1,2,\cdots C_{\mathrm{input}}\hfill \\ 0,\mathrm{else}\hfill \end{array}\right.$$

  (21)
• The observation ${\mathbf{o}}_t^2\left(u_i\right)$ is a part of the map $M{C}_t\left(u_i\right)$ centered in the UAV’s current cell $c_t\left(u_i\right)$, with length $C_{\mathrm{input}}$ and width $C_{\mathrm{input}}$. Similarly, ${\mathbf{o}}_t^2\left(u_i\right)$ is a $C_{\mathrm{input}}\times C_{\mathrm{input}}$ matrix, representing the observation state of the cells around UAV $u_i$, which is defined as follows:

  $${\mathbf{o}}_t^2\left(u_i\right)={\left[{}^2{o}_t^{kl}\left(u_i\right)\right]}_{C_{\mathrm{input}}\times C_{\mathrm{input}}},$$

  (22)

$${}^2{o}_t^{kl}\left(u_i\right)=\left\{\begin{array}{c}m{c}_t^{mn}\left(u_i\right)/t,\mathrm{if}t>0\mathrm{and}{c}_{kl}\in S\mathrm{and}{c}_{kl}=c_{mn},k=1,2,\cdots C_{\mathrm{input}},\hfill \\ l=1,2,\cdots C_{\mathrm{input}},m=1,2,\cdots C_L,n=1,2,\cdots C_W\hfill \\ 1,\mathrm{elseif}{c}_{kl}\notin S\hfill \\ 0,\mathrm{else}\hfill \end{array}\right.$$

(23)

• The observation ${\mathbf{o}}_t^3\left(u_i\right)$ is a part of the map $M{U}_t\left(u_i\right)$ centered in the UAV’s current cell $c_t\left(u_i\right)$, with length $C_{\mathrm{input}}$ and width $C_{\mathrm{input}}$. Like ${\mathbf{o}}_t^2\left(u_i\right)$, ${\mathbf{o}}_t^3\left(u_i\right)$ is a $C_{\mathrm{input}}\times C_{\mathrm{input}}$ matrix, representing historical position information of other UAVs around UAV $u_i$, which is defined as follows:

  $${\mathbf{o}}_t^3\left(u_i\right)={\left[{}^3{o}_t^{kl}\left(u_i\right)\right]}_{C_{\mathrm{input}}\times C_{\mathrm{input}}},$$

  (24)

$${}^3{o}_t^{kl}\left(u_i\right)=\left\{\begin{array}{c}m{u}_t^{mn}\left(u_i\right),\mathrm{if}{c}_{kl}\in S\mathrm{and}{c}_{kl}=c_{mn},k=1,2,\cdots C_{\mathrm{input}},\hfill \\ l=1,2,\cdots C_{\mathrm{input}},m=1,2,\cdots C_L,n=1,2,\cdots C_W\hfill \\ 0,\mathrm{else}\hfill \end{array}\right.$$

(25)

• The observation ${\mathbf{o}}_t^4\left(u_i\right)$ is a part of the map $M{T}_t\left(u_i\right)$ centered in the UAV’s current cell $c_t\left(u_i\right)$, with length $C_{\mathrm{input}}$ and width $C_{\mathrm{input}}$. Similarly, ${\mathbf{o}}_t^4\left(u_i\right)$ is a $C_{\mathrm{input}}\times C_{\mathrm{input}}$ matrix, representing historical position information of targets around UAV $u_i$, which is defined as follows:

  $${\mathbf{o}}_t^4\left(u_i\right)={\left[{}^4{o}_t^{kl}\left(u_i\right)\right]}_{C_{\mathrm{input}}\times C_{\mathrm{input}}},$$

  (26)

$${}^4{o}_t^{kl}\left(u_i\right)=\left\{\begin{array}{c}m{t}_t^{mn}\left(u_i\right),\mathrm{if}{c}_{kl}\in S\mathrm{and}{c}_{kl}=c_{mn},k=1,2,\cdots C_{\mathrm{input}},\hfill \\ l=1,2,\cdots C_{\mathrm{input}},m=1,2,\cdots C_L,n=1,2,\cdots C_W\hfill \\ 0,\mathrm{else}\hfill \end{array}\right.$$

(27)

One example of observations for UAV $u_0$ is shown in Figure 4, which is consistent with the scenario shown in Figure 3.

3.3.2. Action Space

The UAV’s action space is a set of target cells around the UAV, that is, each UAV can move into one of its eight neighbor cells or stay at its current cell. Thus, the action space has a total of nine command actions. The actual command action is selected according to the selection probabilities calculated by the deep neural network.

3.3.3. Network Architecture

In this study, a deep neural network is used to process the observation ${\mathbf{o}}_t$, and its outputs are the selection probabilities of actions, denoted by $P\left({\mathbf{a}}_t\right|{\mathbf{o}}_t)$. The deep neural network architecture is shown in Figure 5.

As shown in Figure 5, we use four hidden layers to process the observation ${\mathbf{o}}_t$. The first hidden layer uses the CNN to process the input data, which has 4 two-dimensional filters with kernel size = (2, 2) and stride = 1, and its activation function is ReLU [29]. The second and third hidden layers are two fully connected layers with 200 rectifier units. The last hidden layer contains nine nonlinear units with an activation function of Softmax, limiting the output to (0, 1), whose outputs are the selection probabilities of each action.

3.3.4. Reward Function

The design of the reward function is closely related to our objective, which is to enable the UAV team to balance between giving the known targets a fair observation and exploring the search region. Thus, a reward function is designed to achieve this objective:

$$r_t\left(u_i\right)=r_t^1\left(u_i\right)+r_t^2\left(u_i\right)+r_t^3\left(u_i\right)+r_t^4\left(u_i\right)$$

(28)

where $r_t\left(u_i\right)$ is the reward received by UAV $u_i$ at time step t, which is a sum of four different rewards.

The reward $r_t^1\left(u_i\right)$ encourages UAV $u_i$ to track targets that have been discovered, which consists of the following three terms:

$$r_t^1\left(u_i\right)={\lambda }_1{}^l{r}_t^1\left(u_i\right)+{\lambda }_2{}^g{r}_t^1\left(u_i\right)+{\lambda }_3{}^h{r}_t^1\left(u_i\right)$$

(29)

where ${}^l{r}_t^1\left(u_i\right)$ represents the local reward for tracking the discovered targets, ${}^g{r}_t^1\left(u_i\right)$ represents the global reward for tracking the discovered targets, ${}^h{r}_t^1\left(u_i\right)$ represents the reward for recording the historical positions of the targets, ${\lambda }_1$, and ${\lambda }_2$ and ${\lambda }_3$ are the positive coefficients. The rewards ${}^l{r}_t^1\left(u_i\right)$, ${}^g{r}_t^1\left(u_i\right)$ and ${}^h{r}_t^1\left(u_i\right)$ are designed as follows:

$$\left\{\begin{array}{c}{}^l{r}_t^1\left(u_i\right)=\min ({\displaystyle \sum _{j=1}^N}\frac{d_s}{20\ast d_t(u_i,{\nu }_j)},1),\mathrm{for}{d}_t(u_i,{\nu }_j)<d_s\hfill \\ {}^g{r}_t^1\left(u_i\right)={\overline{\eta }}_t-{\overline{\eta }}_{t-1}\hfill \\ {}^h{r}_t^1\left(u_i\right)=min(\mathrm{sum}\left(M{T}_t\left(u_i\right)\right),1)\hfill \end{array}\right.$$

(30)

where $d_t(u_i,{\nu }_j)$ represents the distance between UAV $u_i$ and target ${\nu }_j$ at time step t, ${\overline{\eta }}_t$ represents the average observation rate of targets at time step t, $\mathrm{sum}\left(M{T}_t\left(u_i\right)\right)$ represents the sum of the values of the elements in matrix $M{T}_t\left(u_i\right)$, $min(x,y)$ returns the minimum value of x and y.

The reward $r_t^2\left(u_i\right)$ encourages UAV $u_i$ to explore the search region, which consists of the following two terms:

$$r_t^2\left(u_i\right)={\lambda }_4{}^l{r}_t^2\left(u_i\right)+{\lambda }_5{}^g{r}_t^2\left(u_i\right)$$

(31)

where ${}^l{r}_t^2\left(u_i\right)$ is the local reward for exploring the search region, ${}^g{r}_t^2\left(u_i\right)$ is the global reward for exploring the search region, ${\lambda }_4$ and ${\lambda }_5$ are the positive coefficients. The rewards ${}^l{r}_t^2\left(u_i\right)$ and ${}^g{r}_t^2\left(u_i\right)$ are designed as follows:

$$\left\{\begin{array}{c}{}^l{r}_t^2\left(u_i\right)=\min (\frac{C_L{C}_W({\beta }_t\left(u_i\right)-{\beta }_{t-1}\left(u_i\right))}{M\ast d_s},1)\hfill \\ {}^g{r}_t^2\left(u_i\right)=min(\frac{C_L{C}_W({\beta }_t-{\beta }_{t-1})}{M\ast d_s},1)\hfill \end{array}\right.$$

(32)

where ${\beta }_t\left(u_i\right)$ represents the local exploration rate of the search region known by UAV $u_i$ at time step t, ${\beta }_t$ represents the global exploration rate of the search region at time step t. ${\beta }_t\left(u_i\right)$ and ${\beta }_t$ are calculated as follows:

$$\left\{\begin{array}{c}{\beta }_t\left(u_i\right)=\frac{1}{t}\frac{1}{C_L{C}_W}{\displaystyle \sum _{k=1}^{C_L}}{\displaystyle \sum _{l=1}^{C_W}}m{c}_t^{kl}\left(u_i\right)\hfill \\ {\beta }_t=\frac{1}{t}\frac{1}{C_L{C}_W}{\displaystyle \sum _{k=1}^{C_L}}{\displaystyle \sum _{l=1}^{C_W}}{t}_{stamp}\left(c_{kl}\right)\hfill \end{array}\right.$$

(33)

The reward $r_t^3\left(u_i\right)$ penalizes UAV $u_i$ for approaching other UAVs too close, which is designed as follows:

$$r_t^3\left(u_i\right)=\sum _{j=1,j\ne i}^M{r}_t^3(u_i,u_j),r_t^3(u_i,u_j)=\left\{\begin{array}{c}-0.2,\mathrm{if}0.8d_s\le d_t(u_i,u_j)<d_s\hfill \\ -0.5,\mathrm{else}\mathrm{if}0.5d_s\le d_t(u_i,u_j)<0.8d_s\hfill \\ -1.0,\mathrm{else}\mathrm{if}{d}_t(u_i,u_j)<0.5d_s\hfill \\ 0.0,\mathrm{else}\hfill \end{array}\right.$$

(34)

The reward $r_t^4\left(u_i\right)$ penalizes UAV $u_i$ for leaving the search region, which is designed as follows:

$$r_t^4\left(u_i\right)=\left\{\begin{array}{c}-5,\mathrm{if}{c}_t\left(u_i\right)\notin S\hfill \\ 0,\mathrm{else}\hfill \end{array}\right.$$

(35)

The reward function designed above can make UAVs receive dense rewards in the training process, which can reduce the difficulty of learning. In addition, we set ${\lambda }_1=0.6$, ${\lambda }_2=0.2$, ${\lambda }_3=0.2$, ${\lambda }_4=0.7$, and ${\lambda }_5=0.3$ in the training process.

3.3.5. Training Algorithm

In this study, we used a policy-based DRL algorithm, proximal policy optimization (PPO) [30], to train the deep neural network. The PPO has the benefits of optimizing control policies with guaranteed monotonic improvement and high sampling efficiency, and it has been widely used in the control of robots [31,32].

The algorithm flow is shown in Algorithm 1. In the training process, a centralized training, decentralized execution paradigm is used. Specifically, at each time step, each UAV independently obtains the observation and selects action through the shared policy, and the policy is trained with experiences collected by all UAVs during network training. The collected experience is used to construct the loss function $L^{CLIP}\left(\theta \right)$ for the policy network ${\pi }_{\theta }$ and the loss function $L^V\left(\varphi \right)$ for the value network $V_{\varphi }$. The value network structure is the same as the policy network structure, except that it has only one linear unit in its last layer. In each episode, the policy network ${\pi }_{\theta }$ is optimized $E_{\pi }$ times, and the value network $V_{\varphi }$ is optimized $E_V$ times on the same minibatch data sampled from the collected experience with Adam optimizer [33].

4. Results

In this section, simulation experiments are performed to evaluate the effectiveness of our proposed policy. We first describe the simulation setup and introduce the training process. Then, we compare our policy with other methods in various scenarios to validate its performance. Finally, we discuss the results.

4.1. Simulation Setup and Training Results

We conduct simulation experiments in a Python environment. The deep neural networks are implemented with Pytorch [34]. In the training process, we consider a search region of size $50\times 50$ cells, i.e., $C_L=C_W=50\mathrm{cells}$. The numbers of UAVs and targets in the search region are set to $M=5$ and $N=10$, respectively. The sensing range of each UAV is set to $d_s=5\mathrm{cells}$ and the communication range of each UAV is set to $d_c=10\mathrm{cells}$. In addition, the maximum UAV speed is set to 1 cell per time step, and the maximum target speed is set to 0.5 cells per time step. The total mission time step is 200, i.e., $T=200$. We set $t_U=5$ and $t_T=8$ for the decay period of the position history map of the UAVs and that of the position history map of the targets, respectively. The parameters in Algorithm 1 are listed in

Table 1. Training parameters in Algorithm 1.

Parameters	Values
T	200
M	5
$\gamma$	0.99
$E_{\pi }$	10
$\epsilon$	0.1
B	64
$l_{r\theta }$	0.00005
$E_V$	10
$l_{r\varphi }$	0.001

. In addition, the observation input size is set to $C_{\mathrm{input}}=21\mathrm{cells}$.

Algorithm 1: PPO with multiple UAVs for CMUOMMT

The training process took 3000 episodes. At the beginning of each training episode, the positions of the UAVs and the targets are randomly reset. The speed of each target is randomly generated between [0, 0.5] cells per time step and remains unchanged during a training episode. We recorded the average and variance of each episode’s cumulative reward every 100 episodes. The cumulative reward for each training episode is the average of the cumulative rewards received by all UAVs. The training results are shown in Figure 6. As training progresses, each UAV receives progressively larger rewards, which means that the control policy gradually converges, allowing each UAV to track discovered targets and explore unknown environments. It is worth noting that in the early stages of training, the UAVs receive negative rewards due to leaving the search region.

4.2. Comparison with Other Methods

In this subsection, we compare our policy with other methods, including A-CMOMMT [6], P-CMOMMT [8], PAMTS [19], and Random policy. A-CMOMMT is a traditional approach for solving the CMOMMT problem, which uses weighted local force vectors to control UAVs. P-CMOMMT considers the uniformity of the observation of the targets compared with A-CMOMMT. PAMTS is a novel distributed algorithm, considering tracking the targets and exploring the environment in a unified framework. Random policy serves as a baseline of the CMOMMT problem.

In each set of comparative simulation experiments, we ran 50 random test experiments for each method and calculated the average of the following three metrics:

• the average observation rate of the targets $\overline{\eta }$,
• the standard deviation ${\sigma }_{\eta }$ of the observation rates of the N targets, and
• the exploration rate $\beta$ of the search region.

We first compared our policy against other methods with different numbers of UAVs while the number of targets was fixed to 10. As shown in Figure 7a, the average observation rates of the targets continued to increase as the number of UAVs increased across all methods. Our policy had the best performance when the number of UAVs was 2, 10, or 15, and it was the second best method when the number of UAVs was 5 or 20. In addition, Figure 7b shows that our policy had the minimum standard deviation of the observation rates compared with A-CMOMMT and PAMTS in most cases, which shows that our policy can give the targets relatively fair observations. It is worth noting that the standard deviation of the observation rates gradually increased with the increase in the number of UAVs when using P-CMOMMT and Random policy. This is because the number of targets being observed increases when the number of UAVs increases, so that the standard deviation of the observation rates also increases with it. Figure 7c shows the exploration rate of the search region with the various number of UAVs. It can be seen that our policy had a high exploration rate in most cases relative to other methods except for the random policy. Overall, our policy can give targets a high and fair observation while maintaining a high exploration rate of the search region.

The impact of the total mission time on the three metrics was also studied. Figure 8a shows that the observation rates with A-CMOMMT, PAMTS, and our policy continued to improve as the total mission time increased. It is because that the increased mission time allows UAVs to search the environment sufficiently to find the targets. In addition, the observation rates of the A-CMOMMT, PAMTS, and our policy gradually approached as the total mission time increased, which means all three methods can find the targets in the search region with enough mission time. P-CMOMMT had a low observation rate because it tries to give a uniform observation to the targets, which can also be seen from Figure 8b, where P-CMOMMT had a relatively low standard deviation of the observation rates. As shown in Figure 8b, our policy had a medium standard deviation of the observation rates. Similarly, as shown in Figure 8c, our policy had a medium exploration rate compared to the other methods. The results show that our policy can increase the observation rate of the targets when the mission time increases, while reducing the standard deviation of the observation rates and increasing the exploration rate of the search region.

In addition, the impact of the size of the search region on the three metrics is studied. Figure 9a,c shows that the observation rate of the targets and the exploration rate of the search region decreased as the size of the search region increased. It is obvious that targets were more scattered in a larger search region, which makes it difficult for UAVs to find targets and explore the entire search region. As shown in Figure 9b, the increase in the standard deviation of the observation rates from $C_L=C_W=25$ to $C_L=C_W=50$ was due to the number of the discovered targets decreasing as the size of the search region increased. However, the decrease in the standard deviation of the observation rates from $C_L=C_W=50$ to $C_L=C_W=125$ was due to the difficulty for UAVs to find the targets in a large search region.

Finally, we studied the impact of the communication range on the three metrics. As shown in Figure 10, for our policy, the observation rate and the exploration rate continued to improve, and the standard deviation of the observation rate continued to decrease as the communication range increased, until the communication range was greater than 10 cells, where all three metrics basically no longer changed. The impact of the communication range on the three metrics under PAMTS was consistent with our policy, except when there was no communication among UAVs, i.e., $d_c=0$ cells. The results show that the information from the nearby UAVs can bring significant improvements, and the information from the remote UAVs has little impact on this mission. In addition, because A-CMOMMT and P-CMOMMT only consider the impact of UAVs within the sensing range, the variation in communication range has no effect on the three metrics. Like the above results, our policy has a high observation rate just below PAMTS and a low standard deviation of the observation rates and a high exploration rate of the search region compared with A-CMOMMT and PAMTS.

4.3. Discussion

The above comparison results show that our policy can find a balance between giving the known targets a fair observation and exploring the search region. Though our policy has a low observation rate compared with PAMTS in most cases, it can give a fair observation to the targets with a low standard deviation of the observation rates and continue a high exploration rate of the search region, which can enable UAVs to find more targets when the total number of the targets is unknown. It is worth noting that PAMTS assumes that the total number of targets is known, and we do not have this assumption.

5. Conclusions

In this paper, a DRL based approach is proposed to solve the CMUOMMT problem. Unlike traditional CMOMMT approaches, we considered the average observation rate of the targets, the standard deviation of the observation rates, and the exploration rate of the search region at the same time under the assumption that the total number of the targets is unknown. To achieve this objective, we used four observation maps to record the historical positions of targets and other UAVs, exploration status of the search region, and the UAV’s position relative to the search region. In addition, UAVs’ maps were merged from the maps of different UAVs within a communication range. The merged maps were then cropped and processed with a deep neural network to obtain the selection probabilities of actions. The reward function was designed carefully to provide UAVs with dense rewards in the training process. The results of the extensive comparison simulation experiments prove that our policy can give the targets a fair observation and meanwhile maintain a high exploration rate of the search region. Future work will study the CMUOMMT problem in a search region with obstacles and targets with evasive movements. This is a more challenging problem that requires smarter collaboration between UAVs.