Online Unmanned Aerial Vehicles Search Planning in an Unknown Search Environment ^†

Abstract

Unmanned Aerial Vehicles (UAVs) have been widely used in localized data collection and information search. However, there are still many practical challenges in real-world operations of UAV search, such as unknown search environments. Specifically, the payoff and cost at each search point are unknown for the planner in advance, which poses a great challenge to decision making. That is, UAV search decisions should be made sequentially in an online manner thereby adapting to the unknown search environment. To this end, this paper initiates the problem of online decision making in UAV search planning, where the drone has limited energy supply as a constraint and has to make an irrevocable decision to search this area or route to the next in an online manner. To overcome the challenge of unknown search environment, a joint-planning approach is proposed, where both route selection and search decision are made in an integrated online manner. The integrated online decision is made through an online linear programming which is proved to be near-optimal, resulting in high information search revenue. Furthermore, this joint-planning approach can be favorably applied to multi-round online UAV search planning scenarios, showing a great superiority in first-mover dominance of gathering information. The effectiveness of the proposed approach is validated in a widely applied dataset, and experimental results show the superior performance of online search decision making.

1. Introduction

With the rapid development of unmanned technology, Unmanned Aerial Vehicles (UAVs) have been widely applied in information search [1], environmental monitoring [2], border reconnaissance [3], supply delivery [4], etc. This is mainly because of the superiority on maneuverability, portability and flexibility [5,6]. Generally, an operation of UAV includes two aspects, i.e., routing and searching, which are both technical foundations for the application of UAV and directly affect the efficiency of operation execution [7,8,9].

UAV search planning, an important choice for information collection, has become an attractive issue because of the superiority of UAV applications over the traditional manned aircraft approach. Recent research studies have made efforts to address the challenge of joint optimization considering both routing and searching processes in the UAV search planning problem [10,11]. Traditional approaches follow an offline two-stage paradigm that considers path selection and searching time allocation sequentially, in which all decisions are made before action. In the first stage, only the problem of routing is solved separately using methods such as greedy search [12], Ant Colony Optimization (ACO) [13], and the information including the searching payoff and cost at each search point is not considered. Then, the second stage focuses on the allocation of time or cost, where both the difficulty and potential value of searching a specific point are taken into account. To solve this problem, a series of approaches have been proposed, including the quick score method based on the significance of information revenue [14], the heuristic method based partially on the observable decentralized Markov process [10], and time allocation based on the Newton Method [15,16]. These methods all have the premise that the prior parameters of searching payoff and cost need to be known before making decisions.

Unfortunately, the searching payoff and cost are usually revealed in an online manner rather than being known before action in many real-world UAV search operations. However, few studies have investigated this challenge of online revealed information in UAV search planning from the unknown environment. For instance, in the aftermath of an earthquake, utilizing UAVs for search and rescue operations involves the challenge of unknown costs and benefits associated with each search location. The information regarding the cost and benefit of each search location becomes available only upon reaching that specific search point. To this end, this paper first addresses the problem of online decision making in UAV search planning, where the payoff and cost at each search point are unknown in advance. Furthermore, considering the fact that many real-world UAV search missions require multiple trips with battery replacement/charging and data offloading in between because of the limited energy budget of UAVs [17], the issue of multi-round online UAV search planning is addressed in this paper. The main contributions of this work are as follows:

• A novel online UAV search planning problem and the multi-round planning problem are proposed to address the challenge of online revealed payoff and cost of search actions in unknown environment.
• A joint-planning approach is proposed to solve this problem, where both route selection and search decision are made in an integrated online manner which is near-optimal to the omniscient offline solution.
• The proposed approach can help planners achieve an online policy for the multi-round UAV search planning problem, which shows great superiority in the first-mover dominance of gathering information.

The effectiveness of the proposed approach is validated in a widely applied dataset [18], and experimental results show the superior performance of online search decision making.

2. Related Works

In recent years, a large number of scholars have been attracted to studying UAV planning problems. We here give a brief review of UAV planning problems from the following three aspects: UAV path planning, UAV search planning, and online planning in unknown environments.

2.1. UAV Route Planning

UAV path planning is one of the core technologies of UAV planning, in which the planner tries to find the best path for the UAV to reach a specified location under the constraints of the hardware of a drone, flight environment, and performance. Essentially, UAV path planning is an optimization problem under various constraints [19].

Traditional path planning algorithms include Dijkstra, A*, and artificial potential field algorithms. The Dijkstra algorithm [20] is based on the greedy approach to realize path planning for UAVs [21]. Building upon the Dijkstra algorithm, the A* algorithm was proposed by [21] to introduce global information for the estimation and evaluation of the shortest path, thus enhancing efficiency and rendering the path more efficient. In the realm of path planning, [22] put forward an artificial potential field algorithm composed of attractive and repulsive forces. This algorithm achieves smooth and secure paths; however, it is susceptible to local minima and fails to attain the optimal solution.

Moreover, the problem addressed in this study can similarly be regarded as an optimization problem with stochastic rewards. Reference [23] presented a precise algorithm for solving the directed problem with stochastic rewards, as well as a bi-objective genetic algorithm. ESA determines the convex Pareto frontier by solving multiple deterministic Orienteering Problem with Stochastic Profits (OPSP) problems, while GA is employed to generate approximate Pareto frontiers. For the optimization problem with stochastic rewards in a population, reference [24] addressed a problem known as Team Orienteering Coverage Planning with Uncertain Reward (TOCPUR). This problem considers the uncertain and increasing costs at each location and permits multiple visits to the same vertex. An iterative approach is proposed, which estimates the cumulative cost of each vertex and iteratively solves a variant of the Team Orienteering Problem (TOP). This method surpasses precise TOP solutions and greedy algorithms in terms of cost optimization.

2.2. UAV Search Planning

UAV search planning can be divided into dynamic and static types according to the probability modeling method in the search process [25]. For the static search method, since the probability of each search point is known, the search planning can be transformed into the optimal path planning in the search area. On the other hand, dynamic search, because the probability that other search points will search time and search order changes in the process of search, is a process that requires UAV online access to the environment change and decision making, so choosing reasonable environment modeling is the key. For instance, reference [26] proposed an online replanning approach for utilizing autonomous unmanned aerial vehicles (UAVs) to map agricultural fields. This method employs the Markov random field framework to represent the perception of uncertain maps and computes optimized pest sampling strategies. In addition, according to the number of search UAVs, it can be divided into two types, i.e., single UAV and multi-UAV search planning. Reference [11] proposed a multi-UAV-based search time allocation and cooperative search algorithm to obtain the optimal solution for time allocation under a given path. Studies on UAV search planning have been extended from single-target to multi-target missions. Reference [17] proposed the Gen-Path algorithm based on the multi-objective problem to improve the efficiency of UAVs searching for multiple targets.

2.3. Online Planning in Unknown Environments

Online decision making in unknown environments is critical to improve the effectiveness of UAV applications. Most UAV planning nowadays targets globally known environments or locally known environments and pre-plans search trajectories offline. However, when facing an unknown environment, these methods become less efficient in search, while lacking dynamic resilience and mostly transforming the target search problem into a simple area coverage problem [27]. Reference [28] proposed a real-time motion planning framework for mobile robots based on random sampling for dynamically uncertain environments. The robust asymptotically optimal RRT* algorithm is proposed for bounded uncertain linear systems. The CC-RRT algorithm was proposed for unbounded strongly uncertain systems, which can obtain probabilistic feasible path solutions in linear systems satisfying Gaussian distribution assumptions. References [29,30] proposed a series of PSO algorithms based on path planning and objective search. An improved particle swarm optimization method was proposed that applies the collaboration rules of the robot system to the potential field function and applies it as the fitness function of PSO to the unknown environment of the robot collaborative search [31]. Reference [32] proposed a distributed PSO-based algorithm to overcome the limitation of the robot workspace so that the PSO-based algorithm can achieve the overall optimum in the global mechanism.

3. Problem Description and Notations

In the problem of UAV search planning, the planner aims to maximize the information collection, i.e., the accumulated searching payoff over a set of search points (areas), with a limited energy supply constraint. Both actions of routing and searching consume energies of the UAV during the operations, which is referred to as cost in this paper. Traditionally, both the costs of traveling a edge and searching a point are known as prior knowledge; hence, plans can be made in an offline manner before taking actions. However, this is not the case in many real-world applications. In searching operations, the cost of searching a point to collect information is usually not known in advance before the UAV reaches and evaluates the search point, although the cost of traveling a path of a given distance is generally prior knowledge. Therefore, this paper focuses on the problem of UAV search planning, where the searching cost is revealed sequentially online.

Specifically, $G=(V,E)$ denotes a directed network with a set of vertices $V=\{v_1,v_2,\cdots ,v_n\}$ and a set of edges $E=\left\{\right(i,j\left)\right|i,j\in V\}$. At each search point $i\in V$, the searching cost of $a_i$ is assigned to represent the energy consumption (vector form $\mathit{a}$), while $c_{ij}$ (vector form $\mathit{c}$) denotes the energy cost of traveling through edge $(i,j)\in E$. The budget of energy for a UAV denoted by b is limited, which cannot support searching all points in G. The UAV aims to plan a traveling route considering all search points, decide on which points to reach, and conduct the search operations. Let $x_i\in {\{0,1\}}^n$ be the binary decision variables (vector form $\mathit{x}$) of the search selection, and $x_i=1$ if the UAV chooses to spend energy $a_i$ to search vertex i. The corresponding payoff of searching from the vertices is presented as $\mathit{r}={\left(r_i\right)}_{i\in V}\in {\mathbb{R}}^n$. We denote by $y_{ij}\in {\{0,1\}}^{\left|E\right|}$ (vector form $\mathit{y}$) the binary decision variables of the path selection, and $y_{ij}=1$ if the edge $(i,j)$ is selected to traverse by the planner. The UAV should select a route that allows it to visit each vertex only once, starting and ending in the same vertex, i.e., the UAV base. The objective of the planner is to maximize the accumulated searching payoff over V, i.e., ${\sum }_{i\in V}{r}_i{x}_i$, while the total cost of operations is within the budget, i.e.,

$$\sum _{i,j\in E}{c}_{ij}{y}_{ij}+\sum _{i\in V}{a}_i{x}_i\le b,$$

(1)

where ${\sum }_{i,j\in E}{c}_{ij}{y}_{ij}$ denotes the accumulated cost of traveling, and ${\sum }_{i\in V}{a}_i{x}_i$ represents the accumulated cost of searching.

A descriptive schematic of the UAV search planning problem is shown in Figure 1. The start point of the UAV is $v_1$, where the pair of searching cost and payoff is $(a_1,r_1)=(0,0)$ since the UAV does not need to search for information at its base. By traversing the edge $(1,2)\in A$ and consuming the corresponding energy cost $c_{12}$ for traveling, the UAV can reach vertex $V_2$, where the pair of searching cost and payoff $(a_2,r_2)$ is first revealed to the planner in the online version. The UAV should choose the route and decide online whether to search for the vertices it encounters.

To formulate the Online UAV Search Planning problem (OnUAVSP) formally, we define the history of search planning before stage $t-1$ as

$${\mathcal{H}}_{t-1}={\{a_j,r_j,\mathit{c},x_j,{\mathit{y}}^{t-1}\}}_{j=1}^{t-1},$$

(2)

where ${\mathit{y}}^t$ denotes the accomplished route of UAV at stage t. Hence, solving OnUAVSP is equivalent to obtaining the policy functions of the history and the coefficients observed at the current time period, i.e.,

$$(x_t,{\mathit{y}}^t)={\psi }_t(a_t,r_t,{\mathcal{H}}_{t-1}).$$

(3)

Denote by $\mathit{\psi }={\left({\psi }_t\right)}_{t\in \{1,2,\cdots ,|\mathcal{V}\left|\right\}}$ the vectors of their policy functions, which can be time dependent.

Due to the energy budget of UAVs, many real-world UAV search missions require multiple trips with battery replacement/charging and data offloading in between [17]. As shown in Figure 2, since the energy budget is limited and can only support the search mission of partial vertexes, the UAV needs to return back to the base vertex for battery replacement/charging. The UAV search mission is divided into two trips (i.e., trip 1: $v_1\to v_2\to v_4\to v_5\to v_3\to v_1$ and trip 2: $v_1\to v_6\to v_8\to v_7\to v_1$) with battery replacement/charging at $v_1$ in between.

Therefore, we consider a Multi-round Online UAV Search Planning problem (MOUAVSP) in which k rounds of sub-missions are assigned to a UAV for searching for the specified region. At each round $\tau \in K=\{1,2,\cdots ,k\}$, the accumulated searching payoff ${\pi }_{\tau }$ can be obtained by solving the OnUAVSP problem at this round.

Accordingly, we can formulate problem MOUAVSP as follows:

$$\left[\text{P-MOUAVSP}\right]{\Lambda }_k^{*}=\underset{\mathcal{P}}{max}\sum _{\tau \in K}{\pi }_{\tau },$$

where ${\mathcal{V}}_{\tau }$ denotes the vertex set selected by UAV to traverse at round $\tau$, and $\mathcal{P}=\{P_1,P_2,\cdots ,P_k\}$ presents a partition of $V/v_1$ containing k sets of vertexes, which are chosen by the UAV to traverse at different rounds of the search mission. That is, solving MOUAVSP is directly finding an optimal partition $\mathcal{P}*$ to maximize ${\sum }_{\tau \in K}{\pi }_{\tau }$.

4. An Online Joint-Planning Approach

In the traditional two-stage approach, the UAV route planning is performed offline and consumes an energy budget to traverse the planned vertexes preferentially. Although the UAV search planning process can be performed online, only the remaining energy budget is considered to optimize the decisions of data collection and information search. This might lead to a local–optimal solution to a large extent. Hence, a joint-planning approach is proposed, where both the route selection and search decision are made in an integrated online manner so as to improve the overall revenue of the information search.

4.1. Online UAV Greedy Route Planning

When the energy budget is limited, exploring more valuable vertexes and reducing the energy consumption on routing are two fundamental options to improve the revenue of the information search. We first model the route selection at each stage t as an online decision making problem in which the UAV decides which vertex is to be the next search point in an online manner. We denote by $\mathcal{V}$ the final selected set of vertexes in this mission round. Let $S_t\in \mathcal{V}$ be the location state of UAV, and $b{l}_t$ be the left energy budget at stage t. We can denote by $\phi$ the online route planning function at each stage as follows:

$$S_t=\phi (S_{t-1},b{l}_{t-1},\mathit{c}),$$

(4)

where

$$b{l}_t=b-\sum _{i=1}^t{a}_i{x}_i-L\left({\left\{S_i\right\}}_{i=1}^t\right),$$

(5)

and $L\left({\left\{S_i\right\}}_{i=1}^t\right)$ denotes the energy consumption traversing the vertex sequence ${\left\{S_i\right\}}_{i=1}^t$.

The final aim of the online route planning is to optimally reduce the energy consumption on routing, thereby maximizing the accumulated searching payoff with the limited budget of the UAV energy. Meanwhile, since a UAV needs to return back to its base before its energy is exhausted, the following constraint should be considered during the route planning:

$$b{l}_{t-1}-L\left(\{S_{t-1},S_t\}\right)\ge L\left(\{S_t,S_1\}\right).$$

(6)

As shown in Figure 3, the UAV is now located at vertex $v_4$. According to constraint (6), only when the total energy consumption traverses from $v_4$ to $v_6$, and $v_6$ to $v_1$ is smaller than the current remaining energy budget, is vertex $v_6$ is feasible for search planning.

Different from the offline approach of solving a traveling salesman problem, an online greedy route selection method is proposed, in which the UAV selects the nearest vertex that has not been traversed as the next search point greedily at each stage t, i.e.,

$$S_t=arg\underset{v_i}{min}L\left(\{S_{t-1},v_i\}\right),$$

(7)

where $v_i\notin {\left\{S_j\right\}}_{j=1}^{t-1}$ and the cost of traversing from $S_{t-1}$ to $S_t$ can be denoted by ${\overline{c}}_t$.

4.2. Integrated Online UAV Search Planning

To solve OnUAVSP, the UAV planner should make irrevocable online decisions on which vertex to access and whether to search for it or not at each stage. After obtaining the online greedy UAV route planning strategy in (7), we next integrate it with the online UAV search planning process.

Similarly, the objective is to maximize the accumulated searching payoff over greedily selected vertexes $\mathcal{V}$, i.e., ${\sum }_{i\in \mathcal{V}}{r}_i{x}_i$. When assuming that all online data are known in advance, we can formulate the integrated offline UAV search planning problem (InOffSP) as follows:

$$\left[\text{P-InOffSP}\right]R_n^{*}=\underset{\mathit{x}\in {\{0,1\}}^{\left|\mathcal{V}\right|}}{max}\sum _{i\in \mathcal{V}}{r}_i{x}_i$$

$$\mathrm{s}.\mathrm{t}.\sum _{i\in \mathcal{V}}{a}_i{x}_i\le b-\sum _{i\in \mathcal{V}}{\overline{c}}_i,$$

(8)

where $\mathcal{V}$ and ${\overline{c}}_i$ are determined by the greedy route planning above.

P-InOffSP is a linear integer programming. Since $\mathit{a}$, $\overline{\mathit{c}}$ and $\mathit{r}$ are revealed in an online manner, we cannot obtain the optimal solution $R_n^{*}$ and ${\mathit{x}}^{*}$ of problem InOffSP. In order to achieve an near-optimal online policy ${\psi }_t$ for OnUAVSP, we can first relax the integer constraint of $\mathit{x}$ in P-InOffSP and take the linear dual of it, thereby obtaining the following programming:

$$\left[\text{DLP-InOffSP}\right]\underset{p\ge 0,\mathit{s}\ge \mathbf{0}}{min}p(b-\sum _{i\in \mathcal{V}}{\overline{c}}_i)+{\mathbf{1}}^{\top }\mathit{s}$$

$$\mathrm{s}.\mathrm{t}.p·\mathit{a}+\mathit{s}\ge \mathit{r},$$

(9)

by introducing dual decision variables as $p\in \mathbb{R}$ and $\mathit{s}\in {\mathbb{R}}^n$.

We denote by ${\mathit{x}}^{*}$, $p_n^{*}$ and ${\mathit{s}}^{*}$ the optimal solution of the linear relaxation of InOffSP and DLP-InOffSP. According to the complementary condition, the following is valid:

$$x_j^{*}=\left\{\begin{array}{cc}\hfill 1,& r_j>a_j{p}_n^{*}\hfill \\ \hfill 0,& r_j<a_j{p}_n^{*}\hfill \end{array}\forall j\in \{1,2,\cdots ,n\}\right.,$$

(10)

and when $r_j=a_j{p}_n^{*}$, the decision variable $x_j^{*}$ may be a non-integer. Based on Equation (10), we can design an online algorithm (as shown in Algorithm 1) to solve the integrated online UAV search planning problem [33].

To evaluate the quality of this algorithm, the expected optimality gap, i.e., the average regret, between OffSP and OnSP can be calculated as

$${\Delta }_n=\mathbb{E}[R_n^{*}-R_n],$$

(11)

where $R_n^{*}={\mathit{r}}^{\top }{\mathit{x}}^{*}$ and $R_n={\mathit{r}}^{\top }\mathit{x}$ denote the accumulated payoff under the optimal offline solution and online Algorithm 1, respectively. When we assume that online parameters $\mathit{r}$, $\overline{\mathit{c}}$ and $\mathit{a}$ are independent and identically distributed sampled from unknown distribution, which is usually reasonable in real-world applications, Algorithm 1 is near optimal and achieves $O({\displaystyle \frac{1}{\sqrt{n}}})$ average regret [33] of problem InOffSP.

In this online joint-planning framework, the UAV planner make an online decision on which vertex to access in step 4 and whether to search for it in step 5 at each stage.

Algorithm 1 Online joint-planning algorithm for InOnSP

Input: n, online revealed $(\mathit{r},\mathit{a})$ and $\mathit{c}$

Parameter: learning rate ${\gamma }_t={\displaystyle \frac{1}{\sqrt{n}}}$

Output: an online searching sequence $\mathit{x}$

1: Initialize $p_1=0$ 2: While $t\le n$ 3: Let $e={\displaystyle \frac{b-{\sum }_{i=1}^t{\overline{c}}_i}{n}}$ 4:   Select $S_t$ and obtain ${\overline{c}}_t$ through $$S_t=arg\underset{v_i}{min}L\left(\{S_{t-1},v_i\}\right).$$ 5:   if $b{l}_{t-1}-L\left(\{S_{t-1},S_t\}\right)\ge L\left(\{S_1,S_t\}\right)$ then 6:   Set $$x_t=\left\{\begin{array}{cc}1,\hfill & r_t>a_t{p}_t\\ 0.& r_t\le a_t{p}_t\end{array}\right.$$ 7:   Compute $$p_{t+1}=max\{p_t+{\gamma }_t(a_t{x}_t-e),0\}$$ 8:   update $$t=t+1$$ 9:   else 10:    break 11:   endif 12: end while 13: return  $\mathit{x}={\left(x_i\right)}_{i=\{1,2,\cdots ,t\}}$

4.3. Multi-Round Online UAV Search Planning

After obtaining an online search planning algorithm for a single round of the search mission in an unknown environment, we then apply it to the multi-round scenarios.

As mentioned above, the route selection and search decision are made in an online manner together in the joint-planning framework. That is, the planner does not need to determine the subset of vertexes in advance that can be covered using the limited energy budget b at each trip. Hence, this framework makes it possible to achieve a policy for the problem MOUAVSP. At each round $\tau$, the UAV will receive a searching payoff ${\pi }_{\tau }$ and a set $P_{\tau }$ of vertexes which have been traversed in an online manner. By removing the searched vertexes in $P_{\tau }$ from the entire set V, the online UAV search planning can be applied in multi-round scenarios apparently.

To evaluate the performance of the planning approach on MOUAVSP, the summation of the searching payoff at each round, i.e., $\Lambda ={\sum }_{\tau \in K}{\pi }_{\tau }$ is an intuitive criterion. However, this summation criterion cannot reflect the process characteristics of information collection in multi-round scenarios. For example, the payoff summations from two planning approaches may be the same, but the one that collects more information in the early stage is preferred by mission planners, especially for emergency scenarios such as search-and-rescue operations [1,5]. Therefore, we then design another criterion to evaluate the ability of the first-mover information collection. We denote by ${\Pi }_i=\{{\pi }_1^{\left(i\right)},{\pi }_2^{\left(i\right)},\cdots ,{\pi }_k^{\left(i\right)}\}$ the sequence of searching payoffs under plan i during the multi-round trips.

Definition 1. 

(First-mover Dominance) The plan i dominates plan j at ratio α as a first-mover if

$$\left\{\begin{array}{cc}{\pi }_{\tau }^{\left(i\right)}>{\pi }_{\tau }^{\left(j\right)},\hfill & \hfill \forall \tau \le \alpha k\\ {\pi }_{\tau }^{\left(i\right)}\le {\pi }_{\tau }^{\left(j\right)},\hfill & \hfill \forall \tau >\alpha k\end{array}\right.,$$

(12)

where $\alpha \in [0,1]$ denotes the magnitude of first-mover dominance over plan j.

5. Experimental Results and Discussion

In this section, we evaluate the performance of the proposed joint-planning approach to the online UAV search planning problem and the multi-round problem on a commonly used dataset named TSPLIB [18]. The proposed programming and algorithm are tested on a Windows10 (64) computer with Intel Core-i7 CPU and 16.0 GB RAM using the Gurobi 9.0.1 solvers.

5.1. Performance Analysis on Single-Round Online Search Planning

In this experiment, we compare the performance of different approaches to solving problem OnUAVSP. We first analyze the common situation that the energy budget of the UAV is limited and not capable of traversing all vertexes. As shown in

Table 1. Performance comparison of two approaches.

Case Name	Offline Two-Stage Approach			Online Joint-Planning Approach
	${\mathit{R}}_{\mathit{n}}^{*}$	$\mathit{\rho }$ (%)	${\mathit{T}}^{*}$	${\mathit{R}}_{\mathit{n}}$	${\mathit{R}}_{\mathit{n}}^{*}$	$\mathit{\rho }$ (%)	$\mathit{T}$	${\mathit{T}}^{*}$
att48	46.98	45.16	8.61	58.16	65.26	88.99	1.97	2.53
ch130	210.66	75.66	72.70	181.25	196.66	92.21	7.06	7.83
tsp225	373.13	86.57	309.93	324.19	345.51	93.81	14.82	15.61
gr431	590.08	83.90	1707.88	513.56	549.24	93.49	36.54	37.30
pr1002	1068.86	84.89	63702.50	1074.30	1136.48	94.52	97.79	98.63

, we test both the joint-planning and offline two-stage approaches on the same cases from TSPLIB with the same energy budget $b=75$. In the offline two-stage approach, the routing plan over all vertexes is made first and then the remaining energy of UAV is optimally used to select vertexes to search in an omniscient way. In this case, the offline two-stage approach is more likely to converge to a local optimum, where energy is wasted on the trail, though the omniscient information about search costs is known, while the online joint-planning approach suffers from the unknown environment. However, the proposed online joint-planning approach still outperforms the offline one in the case of ‘att48’ and ‘pr1002’, achieving a higher online revenue as shown in Table 1. Meanwhile, the online joint-planning approach receives a high competitive ratio in all cases manifestly (reporting from 88.99% over 94.52%).

Secondly, the time efficiency of both approaches is tested. We denote by T and $T^{*}$ the online policy and offline policy time consumption of different approaches. Note that the time consumption of the two-stage approach includes two part, i.e., the time for offline route planning (solving a TSP problem through ACO algorithm [13] in this paper) and offline search planning (solving an linear integer programming). As shown in Table 1, it is clear that the proposed online joint-planning approach shows magnificent superiority over the offline two-stage approach in time efficiency, reporting 97.79 versus 63,702.50 CUP seconds in the case ‘pr1002’ (i.e., around three orders of magnitude difference). These results illustrate that the proposed online joint-planning approach in Algorithm 1 is superior in time efficiency over the offline two-stage one when the energy budget of the UAV is limited.

Finally, we analyze the impact of the energy budget on the performance of both approaches. We record the accumulated search payoffs under both approaches with the increase in energy budget b from 25 to 350 in the case ‘ch130’ shown in Figure 4. When the amount of energy budget is relatively small (lower than 125), the payoff under the online joint-planning approach is apparently larger than that under the offline two-stage one. To intuitively illustrate the above phenomenon in Figure 4, we randomly select two specific UAV search planning solutions when the energy budget can only support the search mission of partial vertexes and the UAV needs to return back to the base vertex for battery replacement/charging. The results of the case ‘ch130’ are shown in Figure 5. The star symbol represents the base of the UAV, and the square denotes the vertex which is determined to be searched for. The straight line with the arrow represents the flight route of the UAV under different policies. With the same energy budget $b=75$, the solution under the offline two-stage approach first fulfills the consumption of traversing all vertexes and then assigns the remained energy budget to valuable vertexes so as to accomplish the search tasks (cf. Figure 5a), reporting a revenue of 94.45. In Figure 5b, the total energy budget is allocated to route and search tasks as a whole in an integrated online manner, reporting a revenue of 141.80 in online joint-planning approach. Due to the advantage of integrated allocation, more of the energy budget is assigned to the fulfillment of search tasks, and less energy is wasted on useless routing missions.

5.2. Performance Analysis on Multi-Round Online Search Planning

This experiment focuses on the performance of different approaches on multi-round online search planning. The summation of the searching payoff over multi-rounds $\Lambda$ and the criterion of first-order dominance are both discussed in the case of ‘ch130’ with an energy budget of $b=75$ for each round. Due to the limitation on energy for each round and the number of points to be searched being relatively large, we set the round number as $k=10$. As shown in Figure 6, the summations of the searching payoff over 10 rounds are 968.98 under the online joint-planning approach and 866.16 under the offline two-stage approach. Moreover, they show different progress in the multi-round mission. The online joint-planning approach tends to receive higher payoff on information collection in early rounds of the mission, while that under offline two-stage approach fluctuates randomly. Specifically, the online joint-planning approach consistently achieves higher information collection benefits within the first six rounds compared to the offline two-stage method. This observation suggests that, under similar energy constraints, the online joint-planning approach is capable of acquiring a greater amount of information at a faster rate than the offline two-stage method. Quantitatively, the plan derived from the online joint-planning approach dominates the plan under the offline two-stage approach at a ratio of 0.6 as a first-mover, which shows great superiority in the first-mover dominance of gathering information. In conclusion, the online joint-planning approach surpasses the offline two-stage method in terms of both overall information collection benefits and the speed of acquiring those benefits.

6. Conclusions

In this paper, we address the problem of online decision making in UAV search planning as well as the problem in multi-round scenarios, where the payoff and cost at each search point are unknown for the planner in advance. UAV search decisions should be made sequentially in an online manner, thereby adapting to the unknown search environment.

• We propose a novel online UAV search planning problem and the multi-round planning problem to address the challenge of dynamically revealed payoffs and costs associated with search actions in an unknown environment.
• To overcome the uncertainty of the search environment, we introduce an online joint-planning approach. This approach integrates route selection and search decision making in a near-optimal manner, closely approximating the omniscient offline solution.
• Our proposed approach enables planners to develop an online policy for the multi-round UAV search planning problem, demonstrating significant advantages in terms of first-mover dominance and information gathering capabilities.

The effectiveness of the proposed approach is validated in a widely applied dataset. Experimental results show the superior performance in both time efficiency and the solution quality of online search decision making, reporting several orders of magnitude improvement in time efficiency and superiority in terms of first-mover information collection. In the subsequent stages, we will delve deeper into intricate environments, taking into account precise factors such as wind direction and obstacles. Our objective is to deploy the algorithm in practical scenarios that are both relevant and consequential. These scenarios include post-earthquake UAV rescue missions and UAV data acquisition within wireless sensor networks.