Model-Reference Reinforcement Learning for Safe Aerial Recovery of Unmanned Aerial Vehicles

Abstract

In this study, we propose an aerial rendezvous method to facilitate the recovery of unmanned aerial vehicles (UAVs) using carrier aircrafts, which is an important capability for the future use of UAVs. The main contribution of this study is the development of a promising method for online generation of feasible rendezvous trajectories for UAVs. First, the wake vortex of a carrier aircraft is analyzed using the finite element method, and a method for establishing a safety constraint model is proposed. Subsequently, a model-reference reinforcementearning algorithm is proposed based on the potential function method, which can ensure the convergence and stability of training. A combined reward function is designed to solve the UAV trajectory generation problem under non-convex constraints. The simulation results show that, compared with the traditional artificial potential field method under different working conditions, the success rate of this method under non-convex constraints is close to 100%, with high accuracy, convergence, and stability, and has greater application potential in the aerial recovery scenario, providing a solution to the trajectory generation problem of UAVs under non-convex constraints.

1. Introduction

Aerial recovery is a new technology that can quickly and efficiently capture unmanned aerial vehicle (UAV) in the air when there are no suitable platforms foranding onand or sea [1,2]. Swarm UAVs areow-cost and reusable devices that can be quickly deployed and have recently attracted considerable research interest [3]. A carrier aircraft transports a swarm of UAVs to the edge of a combat zone and releases them. Swarm UAVs then return to the carrier aircraft after completing their tasks. Aerial recovery can improve combat performance, reuse rate, and flexibility of UAVs. However, this involves docking the probe of the UAVs with a cable-drogue assembly towed by the carrier aircraft and then pulling it into the cabin with a winch. The idea of “aerial recovery” has gained interest since theaunch of the Defense Advanced Research Projects Agency (DARPA)’s “Gremlins” project (2016) [4]. On 29 October 2021, DARPA successfully recovered an X-61 “Gremlin” drone in mid-air for the first time [5,6]. Studies have shown that wake vortices have a significant impact on the flight safety of aircrafts [7]. If a following aircraft breaks into the wake area of aeading aircraft, the aerodynamic force of the following aircraft is significantly disturbed [8].

In this case, the main challenges of UAV aerial recovery can be summarized as follows: (1) describing the unsafe area for rendezvous and docking missions while accounting for the effects of wake vortices and turbulence from the carrier aircraft; (2) obtaining closer proximity to the carrier while avoiding unsafe areas, which will reduce theength of the recovery rope, thereby reducing the difficulty of dragging the drone back into the cabin and increasing efficiency; (3) the development of an online real-time control algorithm that ensures safety and accessibility.

Current research methods for wake vortices can be roughly divided into experimental measurement methods, theoretical analyses, and numerical simulation methods based on computational fluid dynamics. The experimental methods of wake simulation include artificial simulation of experimental environment devices represented by on-site observation methods, such as wind tunnels [9]. However, these methods are time consuming, expensive, andack flexibility. In addition, some characteristics of the wake vortex flow field cannot be accurately resolved and captured through experiments owing to theimitations of the data collection and measurement methods. The theoretical analysis methods are primarily used for rapid modeling and simulation of wake vortices [10]. They cannot satisfy the computational requirements of complex shapes and high precision. Numerical simulations are based on the theory of computational fluid dynamics. With the rapid development of high-performance computer technology, these methods have been widely used in aircraft flow fields, such asarge eddy simulations (LES) and simulations based on the Reynolds-averaged Navier–Stokes (RANS) theory [11,12]. Numerical simulation methods are widely recognized for their ability to simulate wake flows in a cost-effective, convenient, and scalable manner.

Some studies have been conducted on the aerial recovery of UAVs [4,13]. Different theoretical approaches have been explored to solve the guidance and control problem [14]. These can be divided into three categories: optimization, hybrid, and analytical methods [15,16]. Examples include direct collocation [17], indirect (variational) methods [18], rapidly exploring random trees (RRTs) [19], sequential convex optimization [20], waypoint following using a PD or H-infinity controller [21], artificial potential functions (APFs) [22], second-order cone programming [23], model predictive control (MPC) [24], and adaptive control techniques [25]. However, many of these methods are offline, time-consuming, and unsuitable for non-convex optimization problems, whereas deep reinforcementearning (RL), which enables autonomous real-time, collision-free rendezvous, and proximity operations, has the potential to overcome these disadvantages [26]. One method of using deep RL is to obtain solutions for systems with complex dynamics and constraints [27]. It has the key benefit of performing well for problems that do not have a suitable nonlinear programming formulation, can generate real-time and dynamic solutions for online applications [28], and has been used in applications in real-world UAV trajectory generation and control, achieving the highest performance far beyond that of humans [29]. This can also be beneficial when the underlying problem is stochastic [30]. However, solving the potential reward sparseness when RL is used to solve UAV navigation problems is difficult [31]. In the early exploration stage of training, UAVs without expert experience do not always receive rewards after each action, rendering it difficult for the algorithm to converge [32]. Existing methods primarily include exploiting intrinsic curiosity [33,34] or information gain [35]; however, these methods mayead to performance degradation. Another method is to imitate expert experience [36]; however, the upperimit of the algorithm performance may be affected by experts, and expert data may be difficult to obtain.

In this study, we considered the aerial recovery of UAV as a reach-avoidance problem to establish a safety constraint model by analyzing the flow field around the carrier aircraft. We present an online trajectory generation method for UAVs in a three-dimensional (3D) space. The contributions of this study can be summarized as follows:

(1) The equipotential airflowines of the flow field around a carrier aircraft were depicted by finite element analysis, and a method for establishing safe constraints was proposed.

(2) A new potential function-based reinforcementearning control method (PRL) was proposed, utilizing the advantages of a model-reference RL algorithm. It can be used for trajectory generation under non-convex constraints, satisfying the safety constraints while guaranteeing convergence. This may be more efficient than RL algorithmsearned from scratch.

(3) A reward function based on a potential function was designed to ensure security while reducing the impact of sparse rewards. Simultaneously, as the training of reinforcementearning is completed in an offline environment, the call model can meet the requirements of online and real-time response in the actual application process. In addition, the convergence and stability of PRL algorithm are proved and the feasibility of the algorithm is verified through simulation experiments.

2. Problem Formulation

When performing an air-recovery mission, it is assumed that the carrier aircraft is flying at the intended cruise speed. The UAV moves to the recovery drogue, which is connected to the carrier through a cable and remains relatively stationary with the carrier. During this process, the UAV must avoid the flow field in an unsafe area around the carrier aircraft. When rendezvous occurs and the relative speed is similar, recovery drives dock with the UAV and finally return to the cabin. In this study, both UAV and drogue are regarded as mass points. A schematic diagram is shown in Figure 1.

2.1. Dynamic Model for The UAV

In this study, we considered a simplified dynamic model for a UAV [37], which can be written as

$$\left[\begin{array}{c}\dot{x}\\ \dot{y}\\ \dot{z}\\ \dot{v}\\ \dot{\gamma }\\ \dot{\psi }\end{array}\right]=\left[\begin{array}{c}vcos\left(\psi \right)cos\left(\gamma \right)\\ vsin\left(\psi \right)cos\left(\gamma \right)\\ vsin\left(\gamma \right)\\ u_1\\ u_2\\ u_3\end{array}\right]$$

(1)

where ${\displaystyle (x,y,z)}$ is the position of UAV in space, ${\displaystyle v}$ is the ground speed, ${\displaystyle \psi }$ is the yaw angle, ${\displaystyle \gamma }$ is the flight path angle, and ${\displaystyle {\mathit{u}}_t={[u_1,u_2,u_3]}^{\top }}$ is the control of UAV.

As the drogue is considered to be relatively stationary with the carrier, we use ${\displaystyle {\mathit{p}}_{Dt}={[x_D,y_D,z_D]}^{\top }}$ to express the state vector of its position ${\displaystyle {\mathit{p}}_{Ct}={[x_C,y_C,z_C]}^{\top }}$ to express the position of the carrier and use ${\displaystyle {\mathit{p}}_{Ut}={[x_U,y_U,z_U]}^{\top }}$ to express the position of the UAV. ${\displaystyle {\mathit{v}}_{Ut}={[{\dot{x}}_U,{\dot{y}}_U,{\dot{z}}_U]}^{\top }}$, ${\displaystyle {\mathit{v}}_{Dt}={[{\dot{x}}_D,{\dot{y}}_D,{\dot{z}}_D]}^{\top }}$ is the velocity of UAV and the velocity of the drogue at time step ${\displaystyle t}$, respectively. The speed deviation can be expressed as ${\displaystyle {\mathit{e}}_v={\mathit{v}}_{Dt}-{\mathit{v}}_{Ut}={[e_{vx},e_{vy},e_{vz}]}^{\top }}$, the flight path angle error between the drogue and the UAV is defined as ${\displaystyle e_{\gamma }={\gamma }_U-{\gamma }_D}$, and the yaw angle error between them is defined as ${\displaystyle e_{\psi }={\psi }_U-{\psi }_D}$. The position of the UAV relative to that of the drogue is defined as ${\displaystyle {\mathit{d}}_{Dt}={\mathit{p}}_{Dt}-{\mathit{p}}_{Ut}={[e_x,e_y,e_z]}^{\top }}$.

2.2. Hazard Area Concept

The wake vortex created by the movement through the carrier can cause a variety of responses, as the air circulation can causearge rolling torques on the aircraft, which can be dangerous. Air swirling in the wake vortex has a tangential velocity that depends on its relative distance from the vortex center. A vortex has a core in which the tangential velocity increasesinearly with the radius. Outside the vortex core, the tangential velocity begins to decrease due to an inverse relationship with the radius. In order to simplify this problem, an area can be defined around the wake vortex, which is called the Hazard area. Outside this area, tangential velocityess than ${\displaystyle V_0}$ pose no danger to aircraft. It should be noted that the tolerance of tangential velocity is different for an aircraft with different aerodynamic parameters. The tangential velocity can be expressed as

$$\begin{array}{cc}\hfill V_{\omega }& =\omega y_{\omega },y_{\omega }\le \alpha \hfill \\ \hfill V_{\omega }& =\frac{\Gamma }{2\pi y_{\omega }},y_{\omega }>\alpha \hfill \end{array}$$

(2)

where ${\displaystyle \alpha }$ is the core radius, ${\displaystyle \omega }$ is the angular velocity, ${\displaystyle y_{\omega }}$ is theocation of the wing relative to the vortexocation, and ${\displaystyle \Gamma }$ is the initial vortex strength. It is related to the weight, speed, and altitude of the aircraft generating the vortex and can be expressed as

$$\begin{array}{cc}\hfill {\Gamma }_0& =\frac{4M_C}{\rho v_C{b}_C\pi }{M}_c\hfill \end{array}$$

(3)

where ${\displaystyle M_C}$, ${\displaystyle v_C}$, ${\displaystyle b_C}$ denote the mass, speed, and wingspan of the carrier aircraft inevel flight, respectively. This means that the initial strength of the vortex can be treated as a constant. Theocal variation in the aircraft’s angle of attack caused by the eddy’s current velocity field causes a rolling moment around the centerline of the wing and affects the flight of the aircraft [7], particularly for the outer region of the wake vortex, which is relevant for sizing the hazard zone because the core region must be avoided in any case [38].

The NASA Ames Research Center utilizes a numerical integration technique known as the strip theory, which uses the Biot–Savartaw to determine the induced velocity normal to the surface [39]. It determines the final rolling moment coefficient induced by the eddy currents to find aircraft danger, indicating that the numerical simulation method is feasible for establishing an unsafe area of the aircraft wake vortex. The rolling moment coefficient can be expressed as

$$C_{\omega }=\frac{2}{T{S}_C{b}_U}{\int }_0^{\frac{b_U}{2}}{y}_{\omega }{C}_{l_{\alpha }}\frac{V_{\omega }}{V_{\infty }}Tc\mathrm{d}{y}_{\omega }$$

(4)

where ${\displaystyle T}$ is the dynamic pressure, ${\displaystyle S_C}$ is the wing area of the carrier aircraft, ${\displaystyle b_U}$ is the span of the wake-penetrating aircraft, ${\displaystyle C_{l_{\alpha }}}$ is the wingift curve slope, ${\displaystyle V_{\infty }}$ is the freestream velocity, and ${\displaystyle c}$ is the cordength.

The simplified danger zone can help overcome the wake vortex problem. As the distance between the aircraft and the wake vortex increases, the impact on the aircraft decreases. The spatial characteristics of this phenomenon indicated the formation of a dangerous space around the wake vortex. Avoiding these areas enables safe and undisturbed flight operations [40]; thus, it can be assumed that outside the hazardous area, vortices pose no danger to the aircraft [38]. The dangerous area is based on the vortex-induced roll coefficient, which is caused by the tangential velocity. Comparing the vortex-induced roll factor to the maximum coefficient of roll supplied by the airplane control system can determine the safety of a UAV flight [7]. In this study, the non-safe area was determined using numerical analysis. When the tangential velocity of the wake vortex wasess than ${\displaystyle V_{\omega }=V_0}$, the vortex-induced roll coefficient was smaller than the roll coefficient provided by the aircraft control system, as described in detail in Section 3.

3. Establishment of Safety Constraint Model

3.1. Finite Element Analysis

We consider a carrier aircraft with a wingspan of 40 m as an example to perform finite element analysis using ANSYS FLUENT 2021 R2. The transport aircraft model used in the experiment is shown in Figure 2. An area of block-oriented inversion encryption (BOI) was set near the aircraft to refine the grid.

A watertight geometric process was applied to divide the mesh. The minimum size of the surface grid, the maximum size, the grid of the encrypted area, and the growth rate were set to 20 mm, 10,000 mm, 500 mm, and 1.1, respectively. To capture the surface accurately, we used curvature and proximity methods to identify the curvature and microstructure. ${\displaystyle far}$ is set as the pressure far-field boundary condition, ${\displaystyle sym}$ is set as the symmetrical surface boundary condition, and the other boundaries are set as ${\displaystyle wall}$ boundaries. Finally, the volume grid is divided, and a polyhedral grid with a maximum grid size of 10,000 mm is selected. Simultaneously, a boundary–layer grid is applied to the aircraft’s surface. The final grid division results are shown in Figure 3.

In this study, the density-based steady-state solver was selected as the solution, the turbulence model was selected as the reliable k-e model, the energy equation was activated, the air density was set as an ideal gas, the viscosity was set as Sutherland, and the boundary condition was the pressure far field. This work analyzes the trailing vortex condition of the carrier aircraft at cruising altitude and cruising speed, and carries out aerial recovery of the fixed wing UAV under these conditions. The flight surface speed is 580 km/h and the flight altitude is 6 km. After the calculation, the Mach number was 0.51, the temperature was 249.18 K, and the far-field pressure was 47,217.5 Pa. The numerical solution method used in this study is a coupled algorithm in which the discrete format is second-order upwind. Theift and drag were monitored during the calculation until convergence. The flow field around a carrier aircraft is shown in Figure 4. Figure 5a shows the velocity contours and Figure 5b shows the pressure contours on the symmetry plane. Figure 6 shows the wake vortex cloud diagrams of different flow field sections, and the vector diagram of wake vortex tangential velocity in different flow field sections is shown in Figure 7.

3.2. Establishment of the Safety Constraint Model

Because the ideal docking state of aerial recovery is performed on the symmetry plane (${\displaystyle xoz}$ plane) of the carrier aircraft, we used a multi-segment second-order Bezier curve with ${\displaystyle n}$ segments to establish safety constraints in this plane. Figure 6 and Figure 7 illustrate that the wake vorticity and the corresponding tangential velocity vector of the vortex center decrease as the wake vortex gradually moves away from the carrier aircraft. In addition, owing to the influence of the separation flow created by theow-pressure areas around the carrier, there are changes in the flow velocity (as shown in Figure 5), where the tangential velocity vector isess than ${\displaystyle V_{\omega }=V_0}$, and the corresponding points in the flow field are projected onto the symmetry plane to obtain the contourines at the points farthest from the aircraft in all directions. We establised a polar coordinate system with the center of the wing as the origin and the speed direction of the carrier machine in the ${\displaystyle x}$ direction, and set ${\displaystyle n}$ to ${\displaystyle 5}$ to create safety constraints, as shown in Figure 8, where points ${\displaystyle N_a(l_a,{\beta }_a)}$ are the control points of the second-order Bezier curve and ${\displaystyle a}$ is an integer from ${\displaystyle 0}$ to ${\displaystyle 2n}$. Notably, the polar coordinate system is only established in the symmetry plane to facilitate the expression of safety constraints, and the conversion with the Cartesian coordinate system will not be expanded in detail.

We define a positive integer ${\displaystyle h}$ that satisfies ${\displaystyle h\le n}$, and points ${\displaystyle M_j(l_j,{\beta }_j)}$ on the boundary of the constraints can be expressed as

$$\begin{array}{cc}\hfill l_j& =\sum _{i=0}^2{l}_{2h-i}\left(\genfrac{}{}{0pt}{}{2}{i}\right)b^{2-i}{(1-b)}^i\hfill \\ \hfill {\beta }_j& =\sum _{i=0}^2{\beta }_{2h-i}\left(\genfrac{}{}{0pt}{}{2}{i}\right)b^{2-i}{(1-b)}^i\hfill \end{array}$$

(5)

where ${\displaystyle b\in [0,1)}$, ${\displaystyle {\beta }_{2h-2}<{\beta }_j<{\beta }_{2h}}$.

The coordinates of the recovery target point (the point where the recovery drogue isocated) are determined as ${\displaystyle D_0(l_{D_0},{\beta }_0)}$, the coordinates of the point on the equipotentialine in this direction are determined as ${\displaystyle A_0(l_{A_0},{\beta }_0)}$, and the parameters in Figure 8 are set to satisfy the following conditions:

$$\begin{array}{cc}\hfill {\beta }_0& =arctan\frac{2m_{D}g}{C\rho S_D{v_d}^2}-\frac{\pi }{2}\hfill \\ \hfill \frac{l_0}{l_{A_0}}& =k_1\hfill \\ \hfill \frac{l_{D_0}}{l_0}& =k_2\hfill \end{array}$$

(6)

where ${\displaystyle m_D}$ is the mass of the recovery drogue, ${\displaystyle g}$ is gravity, ${\displaystyle C}$ is the air resistance coefficient, ${\displaystyle \rho }$ is the air density, ${\displaystyle S_D}$ is the windward area of the drogue, ${\displaystyle v_d}$ is the relative velocity of the drogue and air, and ${\displaystyle k_1}$ and ${\displaystyle k_2}$ are the expansion coefficients, usually greater than 1.

4. PRL Algorithms

In the UAV navigation problem, there are situations in which rewards are sparse, and only when the UAV completes the target task can it obtain a nonzero reward [32]. The distance between the two aircrafts always increases further in the early stages of training, owing to the high speed of the carrier aircraft. Hence, it is difficult for the UAV to obtain rewards, eading to a poor convergence of the algorithm. This difficulty is particularly evident in this problem, where at the early stage of exploration, owing to aack of expert experience and understanding of the environment, the agent will randomly choose actions, and it will take aong time to reach the goal or fail to converge at all. To solve the above problems, we propose a potential function reinforcementearning method thatearns the potential function by interacting with the environment, ensuring convergence without expert experience while avoiding the problems of being unable to reach and falling into theocal optimum.

First, the corresponding potential field function is constructed based on the hazard area established previously and the position of the carrier aircraft. By designing the corresponding reward function, the output value of the action network in the reinforcementearning algorithm is used to adjust the size of the potential function to make the UAV move toward the target position, and the purpose of avoiding the hazard area is achieved at the same time. The specific details of the algorithm will be introduced in detail in this section.

4.1. Construction of Potential Function

The APF method uses potential field gradients comprising attractive and repulsive potentials to derive the necessary control inputs to achieve the desired or target positions [22,41]. Considering this idea combined with the safety boundary model established in Section 3.2, we determine a potential field function for aerial recovery. The attractive potential ${\displaystyle {\tau }_a}$ can be expressed as

$$\begin{array}{cc}\hfill {\tau }_a& =\frac{1}{2}{\mathit{d}}_{Dt}^{\top }{\mathit{W}}_{\mathit{a}}{\mathit{d}}_{Dt}\hfill \end{array}$$

(7)

where ${\displaystyle {\mathit{W}}_{\mathit{a}}\subset {\mathbb{R}}^{3\times 3}}$ is a symmetric positive-definite attractive potential shaping matrix.

The repulsive potential ${\displaystyle {\tau }_b}$ can be expressed as

$$\begin{array}{cc}\hfill {\tau }_b& =\frac{{\mathit{d}}_{Dt}^{\top }{\mathit{W}}_{\mathit{b}}{\mathit{d}}_{Dt}}{2exp[{\mathit{d}}_{St}^{\top }{\mathit{W}}_{\mathit{c}}{\mathit{d}}_{St}-1]}\hfill \end{array}$$

(8)

where ${\displaystyle {\mathit{W}}_{\mathit{b}},{\mathit{W}}_{\mathit{c}}\subset {\mathbb{R}}^{3\times 3}}$ is a positive-definite attractive potential shaping matrix and ${\displaystyle {\mathit{d}}_{St}={\mathit{p}}_{Ut}-{\mathit{p}}_{St}}$ is the position of the UAV relative to the target safety boundary constraint inertial position, ${\displaystyle {\mathit{p}}_{St}}$.

From a geometric perspective, shaping matrix ${\displaystyle \mathit{W}}$ can be regarded as theength, width, and height of a potential function. The total potential function ${\displaystyle {\tau }_{\mathrm{sum}}}$ is defined as the sum of the attractive and repulsive potential functions as follows:

$$\begin{array}{cc}\hfill {\tau }_{\mathrm{sum}}& ={\tau }_a+{\tau }_b\hfill \end{array}$$

(9)

4.2. Markov Decision Process

Generally, the RL problem is formalized using a discrete-time Markov decision process (MDP): ${\displaystyle =(\mathcal{S},\mathcal{A},\mathcal{P},R,\delta )}$. ${\displaystyle {\mathit{s}}_t\in \mathcal{S}\subset {\mathbb{R}}^{\dim \left(\mathit{s}\right)}}$ is the state vector at time ${\displaystyle t}$ and ${\displaystyle \mathcal{S}}$ denotes the state space. Then, the agent takes an action ${\displaystyle {\mathit{a}}_t\in \mathcal{A}\subset {\mathbb{R}}^{\dim \left(\mathit{a}\right)}}$ according to a stochastic policy/controller ${\displaystyle \pi \left({\mathit{a}}_t\mid {\mathit{s}}_t\right)}$. The state transition probability ${\displaystyle \mathcal{P}}$ defines the transition probability. In the problem studied herein, the state of the next step can be updated by inputting the state and action of the current step into the model. We assumed that the input and state data can be sampled from the system (1) at discrete time steps due to the fact that RL trains the policy network from data samples. Based on MDP, an actor–critic algorithm was proposed as theearning control framework. The detailed derivations are discussed below.

4.3. State and Action Settings

Let ${\displaystyle {\mathit{s}}_t={[e_x,e_y,e_z,e_{vx},e_{vy},e_{vz}]}^{\top }\subset {\mathbb{R}}^6}$ be the state of the system at step ${\displaystyle t}$ and the actions of RL as the coefficients of the potential function established in (9). The action of system at step ${\displaystyle t}$ is defined as ${\displaystyle {\mathit{a}}_t={[{\mathit{a}}_t^a,{\mathit{a}}_t^b]}^{\top }}$. Because a higher precision control is desired at the end positions, a continuous feedback control rate is chosen and defined as

$$\begin{array}{cc}\hfill {\mathit{u}}_t& ={\mathit{K}}_a(\dot{{\mathit{u}}_t}+{\mathit{a}}_t^a\nabla {\tau }_a+{\mathit{a}}_t^b\nabla {\tau }_b)\hfill \end{array}$$

(10)

where ${\displaystyle {\mathit{K}}_a\subset {\mathbb{R}}^{3\times 3}}$ is the positive gain matrix and ${\displaystyle \nabla {\tau }_a}$ can be expressed as

$$\begin{array}{c}\hfill \nabla {\tau }_a={\mathit{d}}_{Dt}{\mathit{W}}_{\mathit{a}}\end{array}$$

(11)

${\displaystyle \nabla {\tau }_b}$ can be expressed as

$$\begin{array}{c}\hfill \nabla {\tau }_b=exp[1-{\mathit{d}}_{St}^{\top }{\mathit{W}}_{\mathit{c}}{\mathit{d}}_{St}]({\mathit{W}}_{\mathit{b}}{\mathit{d}}_{Dt}-\left({\mathit{d}}_{Dt}^{\top }{\mathit{W}}_{\mathit{b}}{\mathit{d}}_{Dt}\right){\mathit{W}}_{\mathit{c}}{\mathit{d}}_{St})\end{array}$$

(12)

4.4. Setup of the RL

For standard RL, the objective is to maximize an expected accumulated return described by a value function ${\displaystyle V_{\pi }\left(s_t\right)}$ with

$$\begin{array}{c}\hfill V_{\pi }\left({\mathit{s}}_t\right)=\sum _t^{\infty }\sum _{{\mathit{a}}_t}\pi \left({\mathit{a}}_t\right|{\mathit{s}}_t)\sum _{{\mathit{s}}_{t+1}}\mathcal{P}\left({\mathit{s}}_{t+1}\mid {\mathit{s}}_t,{\mathit{a}}_t\right)(R_t+\delta V_{\pi }\left({\mathit{s}}_{t+1}\right))\end{array}$$

(13)

where ${\displaystyle \mathcal{P}\left({\mathit{s}}_{t+1}\mid {\mathit{s}}_t,{\mathit{a}}_t\right)}$ is theikelihood of the next state ${\displaystyle {\mathit{s}}_{t+1}}$, ${\displaystyle R_t=R({\mathit{s}}_t,{\mathit{a}}_t)}$ is the reward function, ${\displaystyle \delta \in [0,1)}$ is a constant discount factor, and ${\displaystyle \pi \left({\mathit{a}}_t\right|{\mathit{s}}_t)}$ is called control policy in RL. Policy ${\displaystyle \pi \left({\mathit{a}}_t\right|{\mathit{s}}_t)}$ is the probability of choosing an action ${\displaystyle {\mathit{a}}_t\in \mathcal{A}}$ at a state ${\displaystyle {\mathit{s}}_t\in \mathcal{S}}$. In this paper, a Gaussian policy is used, which is

$$\begin{array}{c}\hfill \pi \left(\mathit{a}\right|\mathit{s})=\mathcal{N}(\mathit{a}\left(\mathit{s}\right),\sigma )\end{array}$$

(14)

where ${\displaystyle \mathcal{N}(·,·)}$ stands for a Gaussian distribution with ${\displaystyle \mathit{a}\left(\mathit{s}\right)}$ as the mean and ${\displaystyle \sigma }$ as the covariance matrix which controls the exploration during theearning process. The Q-function used to evaluate the action can be expressed as

$$\begin{array}{c}\hfill Q_{\pi }({\mathit{s}}_t,{\mathit{a}}_t)=R_t+\delta {\mathbb{E}}_{{\mathit{s}}_{t+1}}\left[V_{\pi }\left({\mathit{s}}_{t+1}\right)\right]\end{array}$$

(15)

where ${\displaystyle {\mathbb{E}}_{{\mathit{s}}_{t+1}}\left[·\right]=\sum _{{\mathit{s}}_{t+1}}{P}_{t+1|t}\left[·\right]}$ is an expectation operator.

By introducing the entropy of policy ${\displaystyle \mathcal{H}\left(\pi \left({\mathit{a}}_{t+1}\right|{\mathit{s}}_{t+1})\right)=-\sum _{a_t}\pi \left({\mathit{a}}_t\right|{\mathit{s}}_t)ln\left(\pi \left({\mathit{a}}_t\right|{\mathit{s}}_t)\right)}$ and the temperature parameter ${\displaystyle \eta }$ [42], the modified Q-function can be expressed as

$$\begin{array}{c}\hfill Q_{\pi }({\mathit{s}}_t,{\mathit{a}}_t)=R_t+\delta {\mathbb{E}}_{{\mathit{s}}_{t+1}}[V_{\pi }\left({\mathit{s}}_{t+1}\right)+\eta \mathcal{H}\left(\pi \left({\mathit{a}}_{t+1}\right|{\mathit{s}}_{t+1})\right)]\end{array}$$

(16)

There are two steps that needed to be executed repeatedly in theearning process: policy evaluation and policy improvement. In the policy evaluation, the Q-value in Equation (16) is computed by applying a Bellman operation ${\displaystyle Q_{\pi }({\mathit{s}}_t,{\mathit{a}}_t)={\mathcal{T}}^{\pi }{Q}_{\pi }({\mathit{s}}_t,{\mathit{a}}_t)}$ where

$$\begin{array}{c}\hfill {\mathcal{T}}^{\pi }{Q}_{\pi }({\mathit{s}}_t,{\mathit{a}}_t)=R_t+\delta {\mathbb{E}}_{{\mathit{s}}_{t+1}}\left\{{\mathbb{E}}_{\pi }[Q_{\pi }({\mathit{s}}_{t+1},{\mathit{a}}_{t+1})-\eta ln\left(\pi \left({\mathit{a}}_{t+1}\right|{\mathit{s}}_{t+1})\right)]\right\}\end{array}$$

(17)

and the policy is improved by

$$\begin{array}{c}\hfill {\pi }_{\mathrm{new}}=arg\underset{{\pi }^{\prime }\in \Pi }{min}{\mathcal{D}}_{KL}({\pi }^{\prime }\left(·\right|{\mathit{s}}_t)\parallel Z^{{\pi }_{\mathrm{old}}}{e}^{\frac{1}{\eta }{Q}_{{\pi }_{\mathrm{old}}}({\mathit{s}}_t,·)})\end{array}$$

(18)

where ${\displaystyle \Pi }$ is a policy set and ${\displaystyle {\pi }_{\mathrm{old}}}$ is theast updated policy. ${\displaystyle Q_{{\pi }_{\mathrm{old}}}}$ is the Q-value of ${\displaystyle {\pi }_{\mathrm{old}}}$, ${\displaystyle {\mathcal{D}}_{KL}}$ is the Kullback–Leiber (KL) divergence [43], and ${\displaystyle Z^{{\pi }_{\mathrm{old}}}}$ is the normalization factor. Through mathematical deduction [42], Equation (18) can be written as

$$\begin{array}{c}\hfill {\pi }^{*}=arg\underset{\pi \in \Pi }{min}{\mathbb{E}}_{\pi }[\eta ln\pi \left({\mathit{a}}_t\right|{\mathit{s}}_t)-Q({\mathit{s}}_t,{\mathit{a}}_t)]\end{array}$$

(19)

4.5. Rewards

In the actual process, the velocity direction of the carrier aircraft can be selected arbitrarily in the ${\displaystyle xoy}$ plane. In this subsection, we assume that the carrier moves along the x-direction to facilitate the description of the content of the reward function.

The essence of the reward function in reinforcementearning is to communicate the goal to the agent and motivate the agent to select a strategy to achieve the goal. The goal of the agent in the action is to maximize the reward value. For UAV trajectory generation, reinforcementearning algorithms are trained using discrete reward functions. Because the agent is initially trained withittle or no knowledge of the environment, it chooses actions at random and may reach the goal over aong period [44]. However, excessively dense reward values may violate safety constraints or reduce accuracy. To address these problems, we designed a reward function for safe-RL. The reward ${\displaystyle R_t}$ is defined as

$$R_t=C_{\mathrm{R}}{[R_{\mathrm{dis}},R_{\mathrm{sync}},R_{\mathrm{done}},R_{\mathrm{safe}}]}^{\top }$$

(20)

where

$$\begin{array}{cc}\hfill R_{\mathrm{dis}}& =-\frac{\mathrm{d}∥{\mathit{d}}_t∥}{\mathrm{d}t}\hfill \\ \hfill R_{\mathrm{sync}}& =\left\{\begin{array}{c}-\frac{\mathrm{d}∥{\mathit{v}}_{Ut}-{\mathit{v}}_{Dt}∥}{\mathrm{d}t},e_t\le e_{\mathrm{threshold}}\hfill \\ 0,e_t>e_{\mathrm{threshold}}\hfill \end{array}\right.\hfill \end{array}$$

(21)

are performance indicators related to distance and speed, respectively, ${\displaystyle C_{\mathrm{R}}=[C_{\mathrm{dis}},C_{\mathrm{sync}},}$${\displaystyle C_{\mathrm{done}},C_{\mathrm{safe}}]}$ is the designed parameter, which is the coefficient of the rewards. ${\displaystyle e_t=∥{\mathit{d}}_t∥}$ is the distance between UAV and the drogue at step ${\displaystyle t}$, and ${\displaystyle e_{\mathrm{threshold}}}$ is the threshold that can be adjusted, after which the two agents start to synchronize their speed. ${\displaystyle R_{\mathrm{done}}}$ can be expressed as follows:

$$\begin{array}{cc}\hfill R_{\mathrm{done}}& =\left\{\begin{array}{c}{R}_d,d_t<e_d\hfill \\ 0,d_t\ge e_d\hfill \end{array}\right.\hfill \end{array}$$

(22)

where ${\displaystyle R_d}$ is aarge positive number, but its value cannot be too high; otherwise, it may cause the algorithm to fall into theocal optimum. ${\displaystyle e_d}$ is the error range that the recovery mechanism can cover, which means that entering this range can be considered a successful recovery. Distance ${\displaystyle l_U}$, which is the distance of the carrier aircraft relative to the safe boundary constraints in the symmetry plane, can be derived from Equation (5), which is expressed as

$$\begin{array}{cc}\hfill l_U& =∥e_x,e_z∥-\sum _{i=0}^2{l}_{2h-i}\left(\genfrac{}{}{0pt}{}{2}{i}\right)b_U^{2-i}{(1-b_U)}^i\hfill \end{array}$$

(23)

where ${\displaystyle b_U}$ satisfies

$$\begin{array}{c}\hfill {\beta }_U=arctan\frac{e_z}{e_x},{\beta }_{2h-2}<{\beta }_U<{\beta }_{2h}\\ \hfill \sum _{i=0}^2{\beta }_{2h-i}\left(\genfrac{}{}{0pt}{}{2}{i}\right)b_U^{2-i}{(1-b_U)}^i={\beta }_U,b_U\in [0,1)\end{array}$$

(24)

Safety is judged when the relative distance in the ${\displaystyle y}$ direction isess than that in the safetyine. ${\displaystyle R_{\mathrm{safe}}}$ is the penalty incurred when the UAV violates safety constraints. Because the distance from the UAV to the carrier aircraft can be expressed as ${\displaystyle l_C=∥([x_U-x_C,z_U-z_C]∥}$, we use the difference between ${\displaystyle l_C}$ and ${\displaystyle l_U}$ to describe the relative position between UAV and the safe boundary. ${\displaystyle R_{\mathrm{safe}}}$ can be expressed as

$$\begin{array}{cc}\hfill R_{\mathrm{safe}}& =\left\{\begin{array}{c}-\frac{c_1}{exp[c_2{(l_C-l_U)}^2-1]},e_y\le e_{\mathrm{safe}}\hfill \\ 0,e_y>e_{\mathrm{safe}}\hfill \end{array}\right.\hfill \end{array}$$

(25)

where ${\displaystyle c_1,c_2\in {\mathbb{R}}_{+}}$ and ${\displaystyle e_{\mathrm{safe}}}$ are the safe distances in the y-direction.

4.6. Algorithm Design and Implementation

In PRL, the deep neural network for ${\displaystyle Q_{\theta }}$ and ${\displaystyle {\pi }_{\chi }}$ is called “critic” and “actor”, respectively. The entire algorithm training process is shown in Figure 9, and the training process will be performed offline. At each time step ${\displaystyle t+1}$, we collect the state ${\displaystyle {\mathit{s}}_t}$, action ${\displaystyle {\mathit{a}}_t}$, and reward ${\displaystyle R_t}$ from theast step and the current state ${\displaystyle {\mathit{s}}_{t+1}}$, which will be stored as a tuple ${\displaystyle ({\mathit{s}}_t,{\mathit{a}}_t,R_t,{\mathit{s}}_{t+1})}$ at the replay buffer ${\displaystyle \mathcal{D}}$ [45]. At each step of policy evaluation or improvement, we randomly select a batch of historical data ${\displaystyle \mathcal{B}}$ from the replay buffer ${\displaystyle \mathcal{D}}$ to train the parameters ${\displaystyle \theta }$ of ${\displaystyle Q_{\theta }\left(·\right)}$ and parameters ${\displaystyle \chi }$ of ${\displaystyle {\pi }_{\chi }}$.

At the policy evaluation step, the parameters ${\displaystyle \theta }$ are obtained by minimizing the Bellman residual. Furthermore, we introduce a copy of the actor and critic networks to calculate target values, which can slowly change the goal value to improve the stability ofearning.

$$\begin{array}{c}\hfill J_Q\left(\theta \right)={\mathbb{E}}_{({\mathit{s}}_t,{\mathit{a}}_t)\sim \mathcal{D}}\left[\frac{1}{2}{(Q_{\theta }({\mathit{s}}_t,{\mathit{a}}_t)-\widehat{Q}({\mathit{s}}_t,{\mathit{a}}_t))}^2\right]\end{array}$$

(26)

with

$$\begin{array}{c}\hfill \widehat{Q}({\mathit{s}}_t,{\mathit{a}}_t)=R_t+\delta Q_{\overline{\theta }}-\delta \eta ln\left({\pi }_{\chi }\right)\end{array}$$

(27)

in which ${\displaystyle \overline{\theta }}$ is the parameter of the target critic network.

Parameter ${\displaystyle \chi }$ is updated by adaptive momentum-estimation technique, and Equation (19) can be rewritten as

$$\begin{array}{c}\hfill J_{\pi }\left(\chi \right)={\mathbb{E}}_{({\mathit{s}}_t,{\mathit{a}}_t)\sim \mathcal{D}}[-\eta ln{\pi }_{\chi }-Q_{\theta }({\mathit{s}}_t,{\mathit{a}}_t)]\end{array}$$

(28)

Th equation used to update the temperature parameter ${\displaystyle \eta }$ is given as

$$\begin{array}{c}\hfill J_{\eta }={\mathbb{E}}_{\pi }[-\eta ln\pi \left({\mathit{s}}_t\right|{\mathit{a}}_t)-\eta \overline{\mathcal{H}}]\end{array}$$

(29)

in which ${\displaystyle \overline{\mathcal{H}}}$ is target entropy. If training is done, optimal parameters will be output by Algorithm 1, where ${\displaystyle {\lambda }_Q,{\lambda }_{\pi },{\lambda }_{\alpha }>0}$ areearning rates and ${\displaystyle \tau >0}$ is constant scalar.

Algorithm 1 PRL control algorithm

Initialize parameters ${\displaystyle \theta }$, ${\displaystyle \eta }$ and ${\displaystyle \chi }$

for all each iteration do

for all each environment step do

${\displaystyle {\mathit{a}}_t\sim {\pi }_{\chi }\left({\mathit{a}}_t\right|{\mathit{s}}_t)}$

${\displaystyle {\mathit{s}}_{t+1}\sim P\left({\mathit{s}}_{t+1}\right|{\mathit{s}}_t,{\mathit{a}}_t)}$

${\displaystyle \mathcal{D}\stackrel{}{\leftarrow }\mathcal{D}\bigcup \left\{({\mathit{s}}_t,{\mathit{a}}_t,R_t,{\mathit{s}}_{t+1})\right\}}$

end for

for all each gradient step do

Sample a batch of data ${\displaystyle \mathcal{B}}$ from ${\displaystyle \mathcal{D}}$

${\displaystyle \theta \stackrel{}{\leftarrow }\theta -{\lambda }_Q{\widehat{\nabla }}_{\theta }{J}_Q\left(\theta \right)}$

${\displaystyle \chi \stackrel{}{\leftarrow }\chi -{\lambda }_{\pi }{\widehat{\nabla }}_{\chi }{J}_{\pi }\left(\chi \right)}$

${\displaystyle \eta \stackrel{}{\leftarrow }\chi -{\lambda }_{\pi }{\widehat{\nabla }}_{\chi }{J}_{\pi }\left(\chi \right)}$

${\displaystyle \overline{\theta }\stackrel{}{\leftarrow }\tau \theta +(1-\tau )\overline{\theta }}$

end for

end for

until training is done

Output optimal parameters ${\displaystyle {\chi }^{*}}$, ${\displaystyle {\theta }^{*}}$

5. Performance Analysis

5.1. Converge Analysis

According to (21) and (25), ${\displaystyle R_{\mathrm{dis}}}$, ${\displaystyle R_{\mathrm{safe}}}$, and ${\displaystyle R_{\mathrm{sync}}}$ are all non-positive. In addition, the reward function ${\displaystyle R_{\mathrm{done}}}$ is bounded. Therefore, the total return ${\displaystyle R_t}$ is bounded, that is ${\displaystyle R_t\in [R_{\min },R_{\mathrm{done}}]}$, where ${\displaystyle R_{\min }}$ is theowest bound of reward. For the convergence analysis of the PRL algorithm, the following Theorems 1 and 2 can be given.

Theorem 1.

Let ${\displaystyle {\mathcal{T}}^{\pi }}$ be the Bellman backup operator under policy ${\displaystyle \pi }$, sequence ${\displaystyle Q^{i+1}(\mathit{s},\mathit{a})={\mathcal{T}}^{\pi }{Q}^i(\mathit{s},\mathit{a})}$ will coverage to Q-function ${\displaystyle Q^{\pi }}$ while ${\displaystyle i\to \infty }$.

Proof. 

The specific proof process is detailed in Appendix A. ${\displaystyle \square }$

Theorem 2.

According to (18),et ${\displaystyle Q_{{\pi }_{\mathrm{old}}}}$ be the old policy ${\displaystyle \pi }$, it exists ${\displaystyle Q_{{\pi }_{\mathrm{old}}}(\mathit{s},\mathit{a})\le Q_{{\pi }_{\mathrm{new}}}(\mathit{s},\mathit{a})}$,${\displaystyle \forall \mathit{s}\in \mathcal{S}}$ and ${\displaystyle \forall \mathit{a}\in \mathcal{A}}$.

Proof. 

The specific proof process is detailed in Appendix B. ${\displaystyle \square }$

According to Theorems 1 and 2, the following theorem can be proposed, in which superscript ${\displaystyle k}$ represents the result of the ${\displaystyle k}$-th iteration.

Theorem 3.

Suppose ${\displaystyle {\pi }_k}$ is the strategy obtained at the ${\displaystyle k}$-th iteration, if the strategy evaluation and improvement steps are applied repeatedly, there exists ${\displaystyle {\pi }_k\to {\pi }_{optimal}}$ as ${\displaystyle k\to \infty }$ such that ${\displaystyle Q_{{\pi }_{optimal}}(\mathit{s},\mathit{a})\le Q_{{\pi }_{\mathrm{new}}}(\mathit{s},\mathit{a})}$, ${\displaystyle \forall \mathit{s}\in \mathcal{S}}$ and ${\displaystyle \forall \mathit{a}\in \mathcal{A}}$, where ${\displaystyle {\pi }_{optimal}}$ is the optimal policy.

Proof. 

The specific proof process is detailed in Appendix C. ${\displaystyle \square }$

5.2. Stability Analysis

According to the following theorem, there is stability of the PRL algorithm in the closed-loop tracking system.

Theorem 4.

To prove that the closed-loop tracking system is stable, the Q-functionearned in the PRL training process needs to satisfy.

$$\begin{array}{c}\hfill {\xi }_1{R}_t\le Q_{\pi }(\mathit{s},\mathit{a})\le {\xi }_2{R}_t\end{array}$$

(30)

$$\begin{array}{c}\hfill {\mathbb{E}}_{{\mathit{s}}_t\sim {\zeta }_{\pi }}[{\mathbb{E}}_{{\mathit{s}}_{t+1}\sim {\mathcal{P}}_{\pi }}\left[Q_{\pi }(\mathit{s}+1,\mathit{a}+1)\right]-Q_{\pi }(\mathit{s},\mathit{a})]\le -\upsilon {\mathbb{E}}_{{\mathit{s}}_t\sim {\zeta }_{\pi }}\left[R_t\right]\end{array}$$

(31)

$$\begin{array}{c}\hfill ∥{\mathbb{E}}_{{\mathit{s}}_t\sim {\zeta }_{\pi }}\left[{\mathit{s}}_t\right]∥\le {ϵ}^t\frac{{\xi }_1}{{\xi }_2}∥{\mathbb{E}}_{{\mathit{s}}_0\sim {\zeta }_{\pi }}\left[{\mathit{s}}_0\right]∥\end{array}$$

(32)

where ${\displaystyle {\xi }_1>0}$, ${\displaystyle {\xi }_2>0}$, ${\displaystyle \upsilon >0}$, and ${\displaystyle ϵ\in (0,1)}$. ${\displaystyle {\zeta }_{\pi }\left({\mathit{s}}_t\right)}$ can be expressed as

$$\begin{array}{c}\hfill {\zeta }_{\pi }\left({\mathit{s}}_t\right)\triangleq \underset{T\to \infty }{lim}\frac{1}{T}\sum _{t=0}^T\mathcal{P}({\mathit{s}}_t\mid \pi ,t)\end{array}$$

(33)

Proof. 

The specific proof process is detailed in Appendix D. ${\displaystyle \square }$

6. Simulations and Comparisons

The numerical simulation results demonstrate the effectiveness of the proposed method in solving the rendezvous problem in UAV aerial recovery. First, the convergence and accuracy of the PRL algorithm were verified in a task scenario in which the recovery drogue moved along a straightine. Then, the algorithm was extended to a more complex rendezvous situation, in which the recovery drogue moved along a circle, and the performance of the proposed method was further investigated. The APF method, which enables real-time on-board execution [46,47], was used to solve this rendezvous problem with non-convex constraints for comparison [15,48] to demonstrate the performance of the PRL. Recycling in our simulation was considered successful if the following conditions were met:

$$\begin{array}{cc}\hfill ∥{\mathit{d}}_{Dt}∥& <0.3\hfill \\ \hfill ∥{\mathit{e}}_v∥& <0.1\hfill \end{array}$$

(34)

Note that ${\displaystyle l_U-l_C>0}$ should be satisfied at every step during the entire approach; otherwise, it is regarded as a failure.

Reinforcementearning training was conducted in an offline environment. In the actual simulation process, a real-time response can be achieved after the network parameters areoaded. In the modeling of safety constraints, the coordinates of the safety boundary control points ${\displaystyle N_a}$ established in the symmetric plane relative to the carrier after being converted to the the Cartesian coordinate system are given as ${\displaystyle N_0(-10,-10)}$, ${\displaystyle N_1(-80,-15)}$, ${\displaystyle N_2(-120,-3)}$, ${\displaystyle N_3(80,15)}$, ${\displaystyle N_4(2,15)}$, ${\displaystyle N_5(30,15)}$, ${\displaystyle N_6(30,-4)}$, ${\displaystyle N_7(30,-15)}$, ${\displaystyle N_8(2,-15)}$, ${\displaystyle N_9(-8,-13)}$, and ${\displaystyle D_0(-15,-15)}$.

Table 1. Hyperparameters of PRL.

Hyperparameters	Value
Sample batchsize ${\displaystyle \left|\mathcal{B}\right|}$	512
Learning rate ${\displaystyle {\mu }_{\pi }}$	${\displaystyle 1\times {10}^{-3}}$
Learning rate ${\displaystyle {\mu }_V}$	${\displaystyle 2\times {10}^{-3}}$
Discount factor ${\displaystyle \delta }$	0.99
Neural network structure of policy	(256, 256)
Neural network structure of Value	(256, 256)
Time step (ms)	10
${\displaystyle ϵ}$	0.2
${\displaystyle C_{\mathrm{R}}}$	[50.0, 5.0, 20.0, 20.0]

summarizes the hyperparameters used in reinforcementearning. In combination with Algorithm 1, a training architecture of reinforcementearning was established.

6.1. Rendezvous on Straight-Line Path

In the first case, the carrier plane is assumed to move along the ${\displaystyle x}$-direction at a constant speed of 161 m/s and the UAV approaches from above with an initial position deviation ${\displaystyle {\mathit{d}}_{D0}={[-1000,1000,-1000]}^{\top }}$ and initial speed ${\displaystyle {\mathit{e}}_{v0}={[80,0,0]}^{\top }}$. Figure 10 shows the global trajectories of the different methods with the same initial state. Figure 11 shows the converged solutions for ${\displaystyle e_x}$, ${\displaystyle e_y}$, ${\displaystyle e_z}$, ${\displaystyle ∥{\mathit{v}}_{Ut}∥}$, ${\displaystyle e_{\gamma }}$, ${\displaystyle e_{\psi }}$. The red curves represent the solution obtained by the PRL, the blue curves represent the solution obtained by the APF, and the black dashedine represents the desired state curve.

As shown in Figure 11a, the position deviation in the ${\displaystyle x}$-direction always decreases throughout the process, and the APF converges faster in this direction. According to Figure 11b,c, in the y and z directions, the position deviation of the PRL in the direction steadily decreases, whereas the position deviation of the APF oscillatingly decreases. After a period, the relative position converges to a small value, and rendezvous occurs. Figure 11d shows that the overshoot of PRL was significantly smaller than that of APF as the ground speed approached the target value. Figure 11e,f shows that the velocity direction deviation of PRL quickly converged to the target value, whereas APF slowly converged. Combined with the global trajectory shown in Figure 10, both algorithms can generate trajectories that bring the drone closer to the target.

Table 2. Comparison of average final state of rendezvousing on straight-line path.

Method	PRL	APF
${\displaystyle ∥{\mathit{d}}_{Dt}∥}$, m/s	${\displaystyle 1.47\times {10}^{-1}}$	${\displaystyle 1.93\times {10}^{-1}}$
${\displaystyle v_U}$, m/s	${\displaystyle 161}$	${\displaystyle 161}$
${\displaystyle e_{\gamma }}$, rad	${\displaystyle -3.5\times {10}^{-15}}$	${\displaystyle -9.98\times {10}^{-4}}$
${\displaystyle e_{\psi }}$, rad	${\displaystyle 5.7\times {10}^{-40}}$	${\displaystyle -8.07\times {10}^{-4}}$
${\displaystyle P_{\mathrm{suc}}}$, Success rate	${\displaystyle 1.00}$	${\displaystyle 0.43}$

presents a comparison of the numerical results, which shows the average of 100 experiments of the two algorithms in random initial state, where the probability of reaching the allowable deviation range under the condition of satisfying the safety constraints is ${\displaystyle P_{\mathrm{suc}}}$. Based on the table, both methods can bring the UAV close to the target point, and the ending accuracy of Safe-RL is slightly higher.

6.2. Rendezvous on Circular Path

We consider a recovery drogue moving along a circular path in another intersection case, which is more complex than the straight-line intersection problem discussed in the previous subsection, to further investigate the performance of the proposed method. In this case, the speed of recovery of the drogue was 161 m/s, and the path radius was 72,000 m. The UAV approaches from above with an initial position deviation ${\displaystyle {\mathit{d}}_{D0}={[-2500,2500,-1000]}^{\top }}$ and initial speed ${\displaystyle {\mathit{e}}_{v0}={[80,40,0]}^{\top }}$. Figure 12 shows the global trajectories of the different methods with the same initial state. In these cases, the PRL method provides a smoother trajectory than the APF. Figure 13 shows the converged solutions for ${\displaystyle e_x}$, ${\displaystyle e_y}$, ${\displaystyle e_z}$, ${\displaystyle ∥{\mathit{v}}_{Ut}∥}$, ${\displaystyle e_{\gamma }}$, ${\displaystyle e_{\psi }}$. The red curves represent the solution obtained by the PRL, the blue curves are the solution from APF, the black dashedine represents the desired state curve, and the purple dashedine indicates the upper bound of the control.

Based on the same parameter training used in Section 6.1, the PRL algorithm is implemented and the results are given as follows. Compared with the APF, similar to the solution observed in the previous example, the converged solutions of the two do not completely overlap; however, they follow a similar trend: the speed of the UAV reaches its upper bound at the beginning of the flight, as shown in Figure 11d, and hardly touches theower bound. As shown in Figure 11a–c, during the entire process, the position deviation continued to converge, and the PRL converged faster in the ${\displaystyle z}$ direction. Figure 11e,f shows that at the end of the mission, the speed of the UAV obtained by the PRL algorithm is aligned with the heading direction of the recovery drogue, whereas the results of the APF algorithm still have a certain deviation, although the deviation isess than 0.01 rad. The 3D intersection trajectory shown in Figure 12 indicates that the recovery drogue completed a circular arc before the intersection occurred. Similarly, the PRL method provides a smoother trajectory than the APF.

Table 3. Comparison of average final state of rendezvous on circular path.

Method	PRL	APF
${\displaystyle ∥{\mathit{d}}_{Dt}∥}$, m/s	${\displaystyle 1.97\times {10}^{-1}}$	${\displaystyle 2.22\times {10}^{-1}}$
${\displaystyle v_U}$, m/s	${\displaystyle 161}$	${\displaystyle 161}$
${\displaystyle e_{\gamma }}$, rad	${\displaystyle 1.00\times {10}^{-5}}$	${\displaystyle -1.79\times {10}^{-3}}$
${\displaystyle e_{\psi }}$, rad	${\displaystyle -3.91\times {10}^{-4}}$	${\displaystyle -8.07\times {10}^{-3}}$
${\displaystyle P_{\mathrm{suc}}}$, Success rate	${\displaystyle 0.99}$	${\displaystyle 0.33}$

presents the comparison of the numerical results; it shows the average of 100 experiments of the two algorithms in the same random initial state. The recovery success rate of the PRL was significantly higher than that of the APF. Section 6.3 analyzes the safety of the recovery process.

6.3. Security Analysis

The comparisons of the success rates of PRL and APF are shown in Table 2 and Table 3. The success rate is the rate at which the UAV does not violate safety constraints and satisfies reachability. The results shown in the table prove the ability of PRL to solve the intersection problem with security constraints. Figure 14a shows typical cases of an approaching process with safety constraint violations of APF. Under the same initial state, the APF violates the safe constrains and breaks into the unsafe area, which frequently occurs in our test of 100 randomly initial states. This is because of the overshoot of the APF whose trajectory is not as smooth as PRL, which also causes it to touch the safety boundary at some point. Only when the trajectory is relatively flat can the target position be reached while ensuring safety. However, after training, owing to the penalty of security constraints, PRL has a significantly higher probability of satisfying reachability without violating the security constraints.

The results of the comprehensive simulation experiments can be used to easily compare the advantages of the proposed safe-RL algorithm. First, the convergence of PRL hasess overshoot and is more stable than APF. Second, PRL has a smaller terminal deviation. Finally, under the condition of satisfying the safety constraints, the task success rate of PRL is higher. In summary, the online feasible trajectory generation method has significant application potential for aerial recovery tasks.

7. Conclusions

To address a keyimitation in the current use of UAVs, this study considers rendezvous and docking problems based on aerial recovery. The main contributions of this study are as follows. First, a method of establishing safety constraints based on finite element analysis was proposed to model the non-safety region. Second, a reinforcementearning method based on a potential function was proposed to realize UAV online trajectory generation, while the trajectory satisfies the requirements of non-convex safety constraints. Finally, a reward function was designed such that the UAV can overcome the problem of reward sparsity even without expert experience. The simulation experiment results show that the performance of this algorithm is better than APF. The proposed algorithm has a smoother trajectory, higher efficiency, and can get closer to the targetocation. In terms of more important safety, in the two working conditions of the experiment, the success rates of this algorithm in avoiding hazard areas reached 100${\displaystyle \%}$ and 99${\displaystyle \%}$, respectively, while APF was only 43${\displaystyle \%}$ and 33${\displaystyle \%}$. Therefore, the proposed algorithm can be generalized to the rendezvous and docking problem as a baseline technology, demonstrating that autonomous real-time, collision-free rendezvous and aircraft approaches are feasible.