Online Safe Flight Control Method Based on Constraint Reinforcement Learning

Abstract

UAVs are increasingly prominent in the competition for space due to their multiple characteristics, such as strong maneuverability, long flight distance, and high survivability. A new online safe flight control method based on constrained reinforcement learning is proposed for the intelligent safety control of UAVs. This method adopts constrained policy optimization as the main reinforcement learning framework and develops a constrained policy optimization algorithm with extra safety budget, which introduces Lyapunov stability requirements and limits rudder deflection loss to ensure flight safety and improves the robustness of the controller. By efficiently interacting with the constructed simulation environment, a control law model for UAVs is trained. Subsequently, a condition-triggered meta-learning online learning method is used to adjust the control raw online ensuring successful attitude angle tracking. Simulation experimental results show that using online control laws to perform aircraft attitude angle control tasks has an overall score of 100 points. After introducing online learning, the adaptability of attitude control to comprehensive errors such as aerodynamic parameters and wind improved by 21% compared to offline learning. The control law can be learned online to adjust the control policy of UAVs, ensuring their safety and stability during flight.

1. Introduction

Against the backdrop of continuously increasing flight mission requirements, the flight environment is becoming increasingly complex. UAVs inherently possess characteristics such as strong nonlinearity, complex coupling effects, rapid time-varying features, and significant uncertainties. Control design methods require precise knowledge or identified internal dynamic information about the aircraft. The high-order derivatives of the output variables needed in these methods are difficult to measure in practical applications [1,2]. This necessitates the intelligent upgrade and transformation of various key links in the control system, enabling the aircraft to possess intelligent learning capabilities. Reinforcement learning is a type of intelligent method that achieves end-to-end flight control based on data without the need for operations such as model decoupling and linearization.

Currently, reinforcement learning has received widespread attention and research in various decision-making tasks of artificial intelligence [3,4,5,6,7,8,9,10]. Reinforcement learning can typically be described as a Markov decision process (MDP), meaning that the state at the next moment depends only on the current state and the action taken by the agent, with rewards determined by two consecutive states and the action of the agent between them [11]. Introducing constraints into the MDP, that is, adding loss to describe the degree of violation of constraints by the agent’s behavior in the elements of reinforcement learning, forms the constrained Markov decision process (CMDP) [12]. In CMDP problems, the goal of the agent is to maximize cumulative rewards in a task while keeping the loss within certain constraints. Losses include cumulative losses and real-time losses; the former includes expectations and averages of long-term losses or the probability of exceeding a certain threshold, while the latter refers to explicit or implicit losses at each time step.

According to the objective of reinforcement learning, reinforcement learning is essentially solving an optimization problem, and constrained reinforcement learning transforms the unconstrained optimization problem of classical reinforcement learning into an optimization problem with inequality constraints. Therefore, many methods in the field of optimization can be borrowed for constrained reinforcement learning. Researchers at home and abroad have carried out some studies on the application of reinforcement learning to aircraft control [13,14,15,16,17,18]. Hao et al. [14] proposed a model reference output feedback reinforcement learning control algorithm with a broader application scope. Its learning process only relies on the output of the object and can obtain an output feedback control policy that enables the closed-loop system to have the desired dynamic performance. The algorithm constructs a reward function based on the reference model, which can effectively describe the desired closed-loop dynamic performance of the system. Huang et al. [15] utilized the deep deterministic policy gradient (DDPG) algorithm, using state information from multiple data frames as the agent’s observation state, and rudder angle and engine thrust commands as the agent’s output actions. After training, they obtained a generalized and robust intelligent flight controller. Wang et al. [19] proposed a deep deterministic policy gradient-based reference model for quadrotor UAV attitude controller design, which further incorporates a reference model in the DDPG structure to circumvent the system overshoot caused by too much control. Rui et al. [20] proposed a PPO-based RL controller for attitude control during the transition process of tilt rotor unmanned aerial vehicles (TRUAVs). Through direct interaction with the environment to learn control strategies, they designed and improved the reward function to adapt to the transition process. Burak et al. [21] proposed a design method for flight control systems based on reinforcement learning, aiming to improve the transient response performance of closed-loop reference model adaptive control systems. This method implements reinforcement learning in the feedback path gain matrix of the reference model to generate dynamic adjustment strategies, providing the possibility of learning multiple adaptation strategies, thereby improving the transient response performance of traditional model reference adaptive control system designs. Ma et al. [22] proposed an incremental reinforcement learning-based UAV tracking control algorithm in dynamic environments, using a policy relief approach to enable UAVs to explore appropriately in new environments, and a significance weighting approach to increase the plot with higher significance and richer information utilization.

Traditional reinforcement learning methods lack theoretical guarantees in terms of safety and credibility, making the trained systems unable to meet practical application requirements. Several studies have combined Lyapunov theory with deep reinforcement learning to improve the robustness and stability of control systems. Chow et al. [23] proposed a safety policy optimization method based on Lyapunov functions that trains neural network policies via a deep deterministic policy gradient and proximal policy optimization algorithms and ensures that the set of feasible solutions induced by linearized Lyapunov constraints is satisfied at each policy update. Yu et al. [24] proposed an adaptive control method for mobile robots based on Lyapunov reward shaping that optimizes the control parameters through environmental feedback to achieve real-time stable control. Therefore, this article proposes a new framework for online safe flight control based on constrained reinforcement learning. To the best of our knowledge, this is the first time that a constrained reinforcement learning approach has been applied to aircraft control research. The main contributions of this article are as follows:

• A new framework is proposed for online safe flight control. The core idea is to first design a constrained reinforcement learning algorithm based on extra safety budget, which introduces Lyapunov stability requirements to ensure flight safety and improves the robustness of the controller, and then an online condition-triggered meta-learning method is used to adjust the control raw online to complete the attitude angle tracking task.
• A novel flight control simulation environment is built based on the Python Flight Mechanics Engine (PyFME) [25] for offline training and online learning.
• This work proves that this method not only ensures the safety and stability of the aircraft during flight but also adapts the control law to various environmental changes through online learning.

The rest of this article is structured as follows: Section 2 describes the aircraft model, controller model, and simulation environment. Section 3 introduces the proposed method. Section 4 presents the experimental results and analysis. Finally, Section 5 concludes this article.

2. Mathematic Model

To facilitate research on online trajectory planning based on the characteristics of the aircraft, three reasonable assumptions are made:

• The aircraft is a rigid body.
• The ground is flat and stationary, ignoring the influence of the earth’s curvature and rotation.
• The deformation of the landing gear is neglected.

We only focus on the aircraft model after takeoff. The taxiing phase during takeoff and the landing phase during landing are temporarily not considered. Due to the large range of changes in the attitude motion parameters of the aircraft, quaternions are used to describe the attitude in the simulation to avoid singularities. After numerical integration of the differential equations, the attitude angles $\theta , \psi$, and $r$, which represent pitch, yaw, and roll angles, are calculated using the conversion relationship between attitude angles and quaternions.

2.1. Aircraft Model

The differential equations for the state of the aircraft [26,27] are as follows:

$$\left[\begin{array}{c}\dot{X}\\ \dot{Y}\\ \dot{Z}\end{array}\right]=\left[\begin{array}{c}{V}_x\\ V_y\\ V_z\end{array}\right]$$

(1)

$$\left[\begin{array}{c}\dot{V_x}\\ \dot{V_y}\\ \dot{V_z}\end{array}\right]=\left[A\right]\begin{array}{c}{\dot{W}}_{x1}\\ {\dot{W}}_{y1}\\ {\dot{W}}_{z1}\end{array}+\left[\begin{array}{c}0\\ -g\\ 0\end{array}\right]$$

(2)

$$\left[\begin{array}{c}\dot{q_0}\\ \dot{q_1}\\ \dot{q_2}\\ \dot{q_3}\end{array}\right]={\displaystyle \frac{1}{2}}\left[\begin{array}{ccc}-q_1& -q_2& -q_3\\ q_0& -q_3& q_2\\ q_3& q_0& -q_1\\ -q_2& q_1& q_0\end{array}\right]\left[\begin{array}{c}{\omega }_{x1}\\ {\omega }_{y1}\\ {\omega }_{z1}\end{array}\right]$$

(3)

$$\left[\begin{array}{c}{\dot{\omega }}_{x1}\\ {\dot{\omega }}_{y1}\\ {\dot{\omega }}_{z1}\end{array}\right]={\left[\begin{array}{ccc}{I}_x& -I_{xy}& -I_{xz}\\ -I_{yx}& I_y& -I_{yz}\\ -I_{zx}& -I_{zy}& I_z\end{array}\right]}^{-1}\left\{\left[\begin{array}{ccc}{I}_{xz}{\omega }_{y1}-I_{xy}{\omega }_{z1}& I_{yz}{\omega }_{y1}-I_z{\omega }_{z1}& I_y{\omega }_{y1}-I_{yz}{\omega }_{z1}\\ I_z{\omega }_{z1}-I_{zx}{\omega }_{x1}& I_{xy}{\omega }_{z1}-I_{yz}{\omega }_{x1}& I_{xz}{\omega }_{z1}-I_x{\omega }_{x1}\\ I_{xy}{\omega }_{x1}-I_y{\omega }_{y1}& I_x{\omega }_{x1}-I_{xy}{\omega }_{y1}& I_{yz}{\omega }_{x1}-I_{zx}{\omega }_{y1}\end{array}\right]\left[\begin{array}{c}{\omega }_{x1}\\ {\omega }_{y1}\\ {\omega }_{z1}\end{array}\right]+\left[\begin{array}{c}{M}_{x1}\\ M_{y1}\\ M_{z1}\end{array}\right]\right\}$$

(4)

where $X,Y, \mathrm{a}\mathrm{n}\mathrm{d} Z$ represent the positions along the three axes in the ground coordinate system, while $V_x,V_y, \mathrm{a}\mathrm{n}\mathrm{d} V_z$ stand for the velocities along the three axes in the ground coordinate system. $q_0,q_1,{ q}_2, \mathrm{a}\mathrm{n}\mathrm{d} q_3$ are the attitude quaternions, and ${\omega }_{x1},{\omega }_{y1}, \mathrm{a}\mathrm{n}\mathrm{d} {\omega }_{z1}$ represent the angular velocities of rotation along the three axes in the body coordinate system. ${\dot{W}}_{x1},{\dot{W}}_{y1}, \mathrm{a}\mathrm{n}\mathrm{d} {\dot{W}}_{z1}$ are the visual accelerations along the three axes in the body coordinate system. $M_{x1},M_{y1}, \mathrm{a}\mathrm{n}\mathrm{d} M_{z1}$ are the combined external moments of the three axes under the machine system, which are composed of pneumatic moments and inertia moments. $\left[A\right]$ is the transformation matrix from the body coordinate system to the ground coordinate system, and $g$ represents the acceleration of gravity. The corresponding calculation diagram is shown in Figure 1.

In the body coordinate system, the aerodynamic forces and moments are as follows:

$$\begin{array}{c}{F}_{x1}^{OD}=q{S}_{ref}{C}_{x1}\\ F_{y1}^{OD}=q{S}_{ref}{C}_{y1}\\ \begin{array}{c}{F}_{z1}^{OD}=q{S}_{ref}{C}_{z1}\\ M_{x1}^{QD}=q{S}_{ref}{L}_{ref}{C}_{mx1}\\ \begin{array}{c}{M}_{y1}^{QD}=q{S}_{ref}{L}_{ref}{C}_{my1}\\ M_{z1}^{QD}=q{S}_{ref}{L}_{ref}{C}_{mz1}\end{array}\end{array}\end{array}$$

(5)

where $q$ represents dynamic pressure; aerodynamic force coefficients include axial force coefficient $C_{x1}$, normal force coefficient $C_{y1}$, and lateral force coefficient $C_{z1}$; aerodynamic moments include rolling moment coefficient $C_{mx1}$, yawing moment coefficient $C_{my1}$, and pitching moment coefficient $C_{mz1}$; Lref represents the aerodynamic reference length; and Sref represents the aerodynamic reference area.

2.2. Controller Model

To control the aircraft using reinforcement learning algorithms, the controller serves as the reinforcement learning action network, primarily responsible for executing the aircraft attitude angle control tasks. The inputs (states s) to the controller include observable quantities such as the current altitude $H$, climbing speed $dH$, flight speed $V$, pitch angle $\theta$, yaw angle $\psi$, roll angle $\Phi$, sideslip angle $\beta$, attack angle $\alpha$, roll rate $p$, pitch rate $q$, yaw rate $r$, and the desired yaw angle ${ \psi }_{des}$ and pitch angle ${\theta }_{des}$. The outputs (action a) are ${\delta }_e, {\delta }_a$, and ${\delta }_r$. In the implementation process, a neural network is used to take all the above states as inputs and produce three control commands for the control surfaces as outputs. The block diagram of the controller system is shown in Figure 2.

In the action space, the control commands for the control surfaces are continuous values. During the simulation process, a Gaussian action model is adopted to control the rotation of the control surfaces. The design of the action network is shown in Figure 3.

2.3. Flight Simulation Environment Model

Gazebo’s expertise and depth in flight mechanics simulation are not as good as PyFME, and it consumes more resources and has higher hardware requirements in complex simulations [28]. AirSim is not as focused on the depth and breadth of flight mechanics simulation as PyFME in terms of functional scope. PyFME provides more comprehensive flight mechanics models and related parameters, which may be more suitable for highly specialized flight mechanics research [29]. FlightGear’s customization capabilities are not as flexible and powerful as those in PyFME, especially in research that requires highly customized simulation scenarios [30]. Compared to PyFME, which focuses on the depth and accuracy of flight mechanics simulation, X-Plane, as a commercial software program, has relatively limited customization capabilities, limiting users’ ability to deeply modify and expand the software [31]. After considering the existing flight control simulation environment, we found that although they perform well in some respects, there are limitations in terms of flexibility, user customization, and integration of specific algorithms. To overcome these limitations, we chose PyFME as our research tool.

We established a flight simulation environment based on PyFME, whose main idea is to model the physical scenarios involved in aircraft flight, including aircraft models, atmospheric models, dynamic models, aerodynamic models, and so on. It is capable of simulating the movement of aircraft in the air and, accordingly, replicating all the physical environments pertinent to flight. For further details, please refer to https://github.com/AeroPython/PyFME/wiki, accessed on 12 April 2024. Next, we introduce it from three perspectives: state quantity design, action quantity design, and the design of reward and cost functions.

2.3.1. State Quantity Design

The observables available to the aircraft include flight speed, altitude, climbing rate, pitch angle, yaw angle, roll angle, and the angular rates of these three angles. In reinforcement learning algorithms, the inputs to the agent’s action network include these nine state variables and control variables. During the learning process, the agent needs to utilize the deflection of the control surfaces to calculate the loss function. Therefore, in the experiment, we consider adding the deflection of the control surfaces to the state space, creating an inner feedback loop. Additionally, we utilize state stacking to allow for the agent to capture more dynamic information, obtaining high-order and integral quantities from past states, thereby generating better control outputs. To facilitate model training, during data processing, the mean and standard deviation of the data are dynamically calculated and updated based on the data in the state variables, rather than calculating the mean and standard deviation of all the data at once. This online computing method can better adapt to the dynamic changes of data, especially suitable for situations that require real-time updating of statistical information.

2.3.2. Action Quantity Design

In the attitude control task, the actions of the aircraft involve elevator deflection, rudder deflection, and aileron deflection. The objective of attitude control is to enable the aircraft to follow the target yaw angle and target pitch angle, as shown in Figure 4. In the task design, the target yaw angle and target pitch angle vary sinusoidally with simulation time, allowing for the aircraft to learn to track curved trajectories. The expressions for the target pitch angle ${\theta }_{des}$ and target yaw angle ${ \psi }_{des}$ are as follows:

$$\begin{array}{c}{\theta }_{\mathrm{des} }=A_{\theta }sin\left({\omega }_{\theta }t\right)\\ {\psi }_{\mathrm{des} }={\psi }_0+A_{\psi }sin\left({\omega }_{\psi }t\right)\end{array}$$

(6)

where $A_{\theta }$ and $A_{\psi }$ are the amplitudes of change for the target pitch angle and target yaw angle, respectively, while ${\omega }_{\theta }$ and ${\omega }_{\psi }$ represent the angular frequencies of change for the target pitch angle and target yaw angle, respectively. $A_{\theta }$ randomly takes values between the positive and negative maximum pitch angles, while $A_{\psi }$ randomly takes values between −π/2 and π/2. ${\omega }_{\theta }$ and ${\omega }_{\psi }$ randomly take values within a certain frequency range, enabling the aircraft to learn following and attitude control strategies for different frequencies of trajectory changes.

2.3.3. Reward Function Design

We designed the reward function based on the incremental satisfaction of the target using the aircraft. Let the target state quantity in a certain task of the aircraft be $T_i$, the current corresponding state quantity be $S_i$, and the corresponding state quantity at the previous moment be $S_i^{-}$. Then, the reward value corresponding to the current state $S_i$ is as follows:

$$R_i=k_{Ri}\left(\left|T_i-S_i^{-}\right|-\left|T_i-S_i\right|\right)$$

(7)

where $k_{Ri}$ is the proportionality coefficient, related to the dimensions.

After the aircraft reaches the target attitude, the value of the aforementioned reward calculation method is 0, which is not conducive to the agent learning the policy of maintaining the attitude. Therefore, when the aircraft reaches the target attitude, we allow for the agent to obtain additional positive rewards. At this time, any action that causes the aircraft to deviate from the target attitude results in the aircraft receiving negative rewards, whereas maintaining the flight attitude at the target attitude enables the aircraft to accumulate positive rewards.

For the attitude angle control task, due to the high precision requirements, adopting a constant reward based on a threshold for the angle would make it difficult for the aircraft to meet the reward conditions, which is not conducive to learning control strategies for small errors. Therefore, we adopt a smoother reward function model similar to the Laplace distribution, which is expressed as follows:

$$R_{goal}=k_{Rg}\exp \left(-{\displaystyle \frac{\left|e\right|}{b}}\right)$$

(8)

where $k_{Rg}$ is the overall scaling factor for the target reward, used to balance the reward for approaching the target posture and the reward for maintaining the target posture; $b$ is the precision control factor, used to adjust the sharpness of the Laplace function.

To ensure the safe operation of the aircraft, the maximum angle of attack during flight is set to ±0.5 rad. When the angle of attack exceeds the maximum angle of attack, the turn ends and a reward of −10 is generated. Set the minimum flight altitude of the aircraft to 2 m. If the flight altitude is below 2 m, the aircraft is considered to have crashed, the turn ends, and a reward of −10 is generated.

2.3.4. Cost Function Design

In the aircraft control task, we aim for the deflection of the control surfaces to be as smooth as possible, ensuring that the aircraft’s attitude remains stable and energy consumption is reduced. The cost function, designed specifically based on the deflection of the control surfaces, is formulated as follows:

$$C\left(s,a,s^{\prime }\right)=k_c|a-a^{-}|$$

(9)

where $k_c$ is the amplification factor, s is the current state of the aircraft, a is the action at the current moment, $s^{\prime }$ is the state of the aircraft at the previous moment, and $a^{-}$ is the action at the previous moment.

Therefore, the optimization problem for reinforcement learning in aircraft control becomes achieving the target attitude while ensuring that the control surface deflections are within a certain limit.

3. Methodology

The online safe flight control method based on constrained reinforcement learning employs a two-stage process, which is divided into offline virtual trial-and-error training and online reinforcement learning model calibration. In the offline stage, constrained reinforcement learning with an additional safety budget is used to train and optimize the policy function and value function. Finally, the trained policy function is used to drive the aircraft in the simulation to complete the flight mission. At the online stage, the control law model trained in the offline stage is first used to construct a task set through online flights in a virtual environment. Then, real-time interactive data are used to update the task set and continue fine-tuning the model using a conditional trigger-based meta-learning online reinforcement learning method. A schematic diagram of the method is shown in Figure 5.

3.1. Constrained Policy Optimization Algorithm with Extra Safety Budget

The constrained policy optimization with extra safety budget (ESB-CPO) [32] algorithm first samples from the environment, calculates the normalized safety state and constraint equation gradients for each time step based on the sampled losses, and updates the factors ${\alpha }_{i\theta }$ and ${\beta }_{i\theta }(s_t)$ based on the normalized safety state and constraint equation gradients. After that, it calculates the Lyapunov Advantage estimation (LAE) corresponding to each time step. Finally, it solves the approximate constrained policy optimization (CPO) [33] problem using the first-order approximation of the optimization objective function and constraint equation, and the second-order approximation of the KL divergence, to obtain a new policy. Adaptive factors control safety constraints enabling the aircraft to initially ignore the constraints of unsafe states and quickly converge to a trajectory to complete the task. Subsequently, it can gradually meet the requirements of a safe state, ultimately obtaining the optimal trajectory. The LAE $A_{\theta }^{{C_i}^{\prime }}(s,a)$ is as follows:

$$A_{\theta }^{C_i^{\prime }}(s,a)=\underset{s^{\prime }\sim P_a^s}{E}\left[V_{\theta }^{C_i}\left(s^{\prime }\right)-V_{\theta }^{C_i}(s)+\alpha \left(V_{\theta }^{C_i}(s)-\beta V_{\theta }^{C_i}\left(s^{\prime }\right)\right)\right]$$

(10)

where $\alpha \in (\mathrm{0,1}),\beta \in [0,1]$ are adaptive factors, $s$,$a$ are the state and action at the current moment, $s^{\prime }$ is the state at the next moment, and $V_{\theta }^{C_i}$ is the cost value function. Further, $P_a^s$ is the distribution of the next state of $(s,a)$.

Describe the parameterized policy in terms of ${\pi }_{\theta }$. The expected discounted cumulative return of the policy is as follows:

$$J^R(\theta )={\mathbb{E}}_{\tau ～{ \pi }_{\theta }}\left[{\sum }_{t=0}^{\mathrm{\infty }}{\gamma }^{t}R(s,a,s^{\prime })\right]$$

(11)

where $\tau ～{\pi }_{\theta }$ is the trajectory sampled from ${\pi }_{\theta }$. The expected discounted cumulative cost of the policy is as follows:

$$J^{C_i}(\theta )={\mathbb{E}}_{\tau ～{ \pi }_{\theta }}\left[{\sum }_{t=0}^{\mathrm{\infty }}{\gamma }^t{C}_i(s,a,s^{\prime })\right]$$

(12)

The optimization objective of the safe RL algorithm is to find the optimal policy ${\pi }_{{\theta }^{*}}$ that maximises $J^R$ and guarantees that $J^{C_i}\le d_i$, where $d_i$ is the upper cost limit of the ith constraint.

Thus, the optimization problem can be defined:

$$\begin{array}{c}\underset{\theta }{max}{J}^R(\theta )\\ s.t.J^{C_i}(\theta )\le d_i\end{array}$$

(13)

The commonly used advantage functions are as follows:

$$\begin{array}{c}{A}_{\theta }^R(s,a)=Q_{\theta }(s,a)-V_{\theta }(s)\\ A_{\theta }^{C_i}(s,a)=Q_{\theta }^{C_i}(s,a)-V_{\theta }^{C_i}(s)\end{array}$$

(14)

where $V_{\theta }$ is the value function, $Q_{\theta }$ is the state-action value function, $V_{\theta }^{C_i}$ is the cost value function, and $Q_{\theta }^{C_i}$ is the state-action cost value function.

Using the LAE to derive the optimization problem, the policy can be updated:

$$\begin{array}{c}{\theta }^{\prime }=\mathrm{a}\mathrm{r}\mathrm{g}\underset{\tilde{\theta }}{\mathrm{m}\mathrm{a}\mathrm{x}}\underset{\begin{array}{c}s～{\rho }_{\theta }\\ a～{\pi }_{\theta }\end{array}}{\mathbb{E}}\left[{\displaystyle \frac{{\pi }_{\tilde{\theta }}\left(a∣s\right)}{{\pi }_{\theta }\left(a∣s\right)}}{A}_{\theta }^R\left(s,a\right)\right]\\ s.t. J^{C_i}(\theta )+{\displaystyle \frac{1}{1-\gamma }}\underset{\begin{array}{c}s～{\rho }_{\theta }\\ a～{\pi }_{\theta }\end{array}}{\mathbb{E}}\left[{\mathsf{\Delta }}_{\theta ,{\theta }^{\prime }}\left(s,a\right){\displaystyle \frac{A_{\theta }^{C_i^{\prime }}\left(s,a\right)}{1-{\alpha }_{i\theta }}}\right]\le d_i\\ \underset{s～{\rho }_{\theta }}{\mathbb{E}}\left[D_{KL}\left({\pi }_{\tilde{\theta }}(· \mid s)\parallel {\pi }_{\theta }(· \mid s)\right)\right]\le \delta \end{array}$$

(15)

where ${\alpha }_{i\theta }$ decreases from $1^{-}$ to 0 with updating and ${\mathsf{\Delta }}_{\theta ,{\theta }^{\prime }}(s,a)={\displaystyle \frac{{\pi }_{\tilde{\theta }}(a\mid s)}{{\pi }_{\theta }(a\mid s)}}-1$. ${\mathsf{\Delta }}_{\theta ,{\theta }^{\prime }}(s,a)$ denotes the tendency of the policy to update from ${\pi }_{\theta }$ to ${\pi }_{\tilde{\theta }}$. If the new policy tries to avoid choosing an action $a$ under $s$, then ${\mathsf{\Delta }}_{\theta ,{\theta }^{\prime }}(s,a)<0$; conversely, ${\mathsf{\Delta }}_{\theta ,{\theta }^{\prime }}(s,a)>0$.

The relationship between $A_{\theta }^{C_i}(s,a)$ and $A_{\theta }^{C_i^{\prime }}\left(s,a\right)$ is as below:

$$\begin{array}{c}{\displaystyle \frac{A_{\theta }^{C_i^{\prime }}(s,a)}{1-{\alpha }_{i\theta }}}=A_{\theta }^{C_i}(s,a)+B_{1\theta }^i(s,a)+B_{2\theta }^i(s^{\prime })\\ B_{1\theta }^i(s,a)=(1-\gamma )V_{\theta }^{C_i}(s,a)-C_i(s,a,s^{\prime })\\ B_{2\theta }^i(s^{\prime })={\displaystyle \frac{{\alpha }_{i\theta }(1-{\beta }_{i\theta }(s))}{1-{\alpha }_{i\theta }}}{V}_{\theta }^{C_i}(s)\end{array}$$

(16)

Therefore, adding two gaps to the constraint function of (16) yields the following:

$$\begin{array}{c}{J}^{C_i}(\theta )+{\displaystyle \frac{1}{1-\gamma }}\underset{\begin{array}{c}s～{\rho }_{\theta }\\ a～{\pi }_{\theta }\end{array}}{\mathbb{E}}[{\mathsf{\Delta }}_{\theta ,{\theta }^{\prime }}(s,a){\displaystyle \frac{A_{\theta }^{C_i^{\prime }}(s,a)}{1-{\alpha }_{i\theta }}}]\le d_i\\ ⟺J^{C_i}(\theta )+{\displaystyle \frac{1}{1-\gamma }}\underset{\begin{array}{c}s～{\rho }_{\theta }\\ a～{\pi }_{\theta }\end{array}}{\mathbb{E}}\left[{\displaystyle \frac{{\pi }_{\tilde{\theta }}(a\mid s)}{{\pi }_{\theta }(a\mid s)}}{A}_{\theta }^{C_i}(s,a)\right]+G_{1\theta }^i(s,a)+G_{2\theta }^i(s,a)\le d_i\\ \end{array}$$

(17)

where $G_{1\theta }^i(s,a)={\displaystyle \frac{1}{1-\gamma }}\underset{\begin{array}{c}s～{\rho }_{\theta }\\ a～{\pi }_{\theta }\end{array}}{\mathbb{E}}\left[{\mathsf{\Delta }}_{\theta ,{\theta }^{\prime }}(s,a)B_{1\theta }^i(s,a)\right]$; $G_{2\theta }^i(s,a)={\displaystyle \frac{1}{1-\gamma }}\underset{\begin{array}{c}s～{\rho }_{\theta }\\ a～{\pi }_{\theta }\end{array}}{\mathbb{E}}\left[{\mathsf{\Delta }}_{\theta ,{\theta }^{\prime }}(s,a)B_{2\theta }^i(s,a)\right]$. If these gaps are negative, they relax the constraints; otherwise, they tighten them.

The normalized safe state $z_{i\theta }(s_t)$ is a sample-based internal state that directly shows the safety of the state at step t. $z_{i\theta }(s_t)$ is defined as follows:

$$z_{i\theta }(s_t)={\displaystyle \frac{d_i-\sum _{l=0}^t{\gamma }^l{C}_i(s_l,a_l,s_{l+1})}{{\gamma }^t{d}_i}}$$

(18)

where $s_l$, $a_l$, and $s_{l+1}$ are in the trajectory sampled by ${\pi }_{\theta }$. When the sum of costs is greater than the cost limit $d_i$, $z_{i\theta }(s_t)$ is less than 0.

$z_{i\theta }(s_t)$ has an initial value of 1 before t = 0, and its update formula is as follows:

$$z_{i\theta }(s_{t+1})={\displaystyle \frac{z_{i\theta }(s_t)-\frac{C_i(s_l,a_l,s_{l+1})}{d_i}}{\gamma }}$$

(19)

Considering the range [0, 1], ${\beta }_{i\theta }(s_t)$ is calculated as follows:

$${\beta }_{i\theta }(s_t)=1+\mathrm{m}\mathrm{i}\mathrm{n}(\mathrm{t}\mathrm{a}\mathrm{n}\mathrm{h}(z_{i\theta }(s_t)), 0)$$

(20)

When $z_{i\theta }(s_t)$ is less than 0, ${\beta }_{i\theta }(s_t)$ decreases to 0.

The policy gradient directly reflects the effect of constraint on the policy, so the Lagrange multiplier ${\lambda }_i$ can be introduced to compute ${\alpha }_{i\theta }$ based on the policy gradient of the constraint function, constructing the following local optimization problem:

$$\underset{{\lambda }_i}{\mathrm{m}\mathrm{i}\mathrm{n}} \underset{\tilde{\theta }}{\mathrm{m}\mathrm{a}\mathrm{x}}{\lambda }_i{P}_{i\theta }(\tilde{\theta })$$

(21)

where $P_{i\theta }(\tilde{\theta })=\underset{\begin{array}{c}s\mathrm{～}{\rho }_{\theta }\\ a\mathrm{～}{\pi }_{\theta }\end{array}}{\mathbb{E}}\left[{\displaystyle \frac{{\pi }_{\tilde{\theta }}(a\mid s)}{{\pi }_{\theta }(a\mid s)}}{A}_{\theta }^{C^i}(s,a)\right]$.

The dual problem of the above equation is as below:

$$\begin{array}{c}\underset{{\lambda }_i}{\mathrm{m}\mathrm{i}\mathrm{n}} \underset{\tilde{\theta }}{\mathrm{m}\mathrm{a}\mathrm{x}}{\lambda }_i{P}_{i\theta }\left(\tilde{\theta }\right)\\ \mathrm{s}.\mathrm{t}. {\lambda }_i\ge 0\end{array}$$

(22)

The update formula for ${\lambda }_i$ is as follows:

$${\lambda }_{i,t+1}=\mathrm{m}\mathrm{a}\mathrm{x}({\lambda }_{i,t}+\eta P_{\theta }(\tilde{\theta }),0)$$

(23)

where $\eta$ is the step size.

Considering the range of values of ${\alpha }_{i\theta }$, the formula can be defined:

$${\alpha }_{i\theta }=\mathrm{t}\mathrm{a}\mathrm{n}\mathrm{h}({\displaystyle \frac{k_i}{e^{-{\lambda }_i}}})$$

(24)

where $k_i$ is a hyperparameter that globally controls the rate of decrease in ${\alpha }_{i\theta }$. As ${\lambda }_i$ decreases, ${\alpha }_{i\theta }$ changes from 1 to 0.

For small step sizes $\delta$, the optimization problem can be solved approximately by first-order approximation of the objective and constraints and second-order approximation of the KL divergence. Let the objective gradient be $\mathbf{g}$, the constraint gradient be $\mathbf{b}$, and the Hessian of the KL divergence be $H$. Define ${\mathbf{c}}_i\dot{=}{J}^{C_i}(\theta )-d_i$; then, the approximation of Equation (15) is as follows:

$$\begin{array}{c}{\theta }^{\prime }=arg \underset{\tilde{\theta }}{\mathrm{m}\mathrm{a}\mathrm{x}}{\mathbf{g}}^{\mathrm{T}}(\tilde{\theta }-\theta )\\ s.t. {\mathbf{c}}_i+{\mathbf{b}}_i^{\mathrm{T}}(\tilde{\theta }-\theta )\le 0\\ {\displaystyle \frac{1}{2}}(\tilde{\theta }-\theta {)}^{\mathrm{T}}H(\tilde{\theta }-\theta )\le \delta \end{array}$$

(25)

The dual problem of the above equation is as below:

$$\underset{\begin{array}{c}{\mu }_1\ge 0\\ {\mu }_2\ge 0\end{array}}{\mathrm{m}\mathrm{a}\mathrm{x}}{\displaystyle \frac{-1}{2{\mu }_1}}({\mathbf{g}}^{\mathrm{T}}{H}^{-1}\mathbf{g}-2{\tau }^{\mathrm{T}}{\mu }_2+{\mu }_2^{\mathrm{T}}S{\mu }_2)+{\mu }_2^{\mathrm{T}}\mathbf{c}-{\displaystyle \frac{{\mu }_1\delta }{2}}$$

(26)

where $\mathbf{c}=\left[{\mathbf{c}}_0,{\mathbf{c}}_1,...\right],\tau \dot{=}{\mathbf{g}}^{\mathrm{T}}{H}^{-1}B,S\dot{=}{B}^{\mathrm{T}}{H}^{-1}B,$ and $B=[{\mathbf{b}}_0,{\mathbf{b}}_1,...].$

Equation (25) can be solved by approximating CPO update as follows:

If Equation (26) is feasible:

$$\widehat{\theta }=\theta +{\displaystyle \frac{1}{{\mu }_1^{*}}}{H}^{-1}\left(\mathbf{g}-{\mu }_2^{*}\mathbf{b}\right)$$

(27)

else:

$${ \widehat{\theta }}^{\prime }=\theta -\sqrt{{\displaystyle \frac{2\sigma }{{\mathbf{b}}^{\mathrm{T}}{H}^{-1}\mathbf{b}}}}{H}^{-1}\mathbf{b}$$

(28)

where ${\mu }_1^{*}$ and ${\mu }_2^{*}$ are solutions of Equation (26).

In summary, the aircraft attitude control based on ESB-CPO is shown in Algorithm 1. The corresponding framework is shown in Figure 6.

Algorithm 1 Aircraft attitude control based on ESB-CPO

Initialize the policy network ${\pi }_{\theta }$, the value network $V_{\varnothing }$

Initialize the replay buffer $\mathcal{B}$ and step counter t = 0

for k < in 0, 1, 2, … do

Use policy ${\pi }_{{\theta }_k}$ to carry out the flight mission and collect a batch of samples

${\mathcal{D}=\left\{\tau \right\}=\{(s}_t,a_t,r_t^e,s_{t+1})\}$

According to the FIFO principle, update replay buffer B with $\mathcal{D}$

Update step counter t = t + len($\mathcal{D}$)

for $\tau$ in $\mathcal{D}$ do

for s in $\tau$ do

${\beta }_{{\theta }_k}\left(s\right)=1+\min \left(\tanh \left(z_{{\theta }_k}\left(s\right)\right),0\right)$

end for

end for

Compute ${\alpha }_{{\theta }_k}$ by solving the local dual problem

Estimate $\widehat{\mathbf{g}}$, $\widehat{\mathbf{b}}$, $\widehat{\mathbf{c}}$ and $\widehat{H}$ using the sample constructed with $\mathcal{D}$     if approximate ESB-CPO is feasible then

$\widehat{\theta }=\theta +{\displaystyle \frac{1}{{\mu }_1^{*}}}{H}^{-1}\left(\mathbf{g}-{\mu }_2^{*}\mathbf{b}\right)$     else

${ \widehat{\theta }}^{\prime }=\theta -\sqrt{{\displaystyle \frac{2\sigma }{{\mathbf{b}}^{\mathrm{T}}{H}^{-1}\mathbf{b}}}}{H}^{-1}\mathbf{b}$     end if

Obtain ${\theta }_{k+1}$ by backtracking line search to enforce satisfaction of constraint function in (15)

Update $V_{\varnothing }$ by TD-like critic learning end for

3.2. Condition-Triggered Meta-Learning Online Learning Method

Meta-learning, also known as learning to learn, is characterized by the fact that the trained deep model’s structure is not designed to complete a specific task in a particular scenario, but rather to rapidly adapt and accomplish new tasks in different scenarios after only a few training samples and one or a few iterations. This fully embodies the idea of enabling machines to learn to learn.

Assuming the existence of a sample set related to the training task $T_i$, also known as a task set, where each task set contains its own training data and testing data, in meta-learning, these are referred to as the Support Set and Query Set, respectively. We define the initial parameters of the network as $\varphi$, and the model parameters trained on the i-th test task as ${\widehat{\theta }}^i$. Therefore, the overall loss function can be defined:

$$L(f_{\varphi })={\sum }_{i=1}^n{l}^i\left(f_{{\widehat{\theta }}^i}\right),$$

(29)

using gradient descent to update $\varphi$.

As a result, an optimized model can be obtained. When a new task $T_i$ arrives, a small training sample set can be used to train the network, enabling the rapid acquisition of the corresponding network model parameters for that task.

We apply meta-learning to the safe flight control method based on constrained reinforcement learning, using real-time interactive data to update the task set based on conditional triggers and continue fine-tuning the model. The online flight control is shown in Algorithm 2.

Algorithm 2 Online flight control based on meta-learning

Loading offline model parameters

Initialize the replay buffer $\mathcal{B}$, step counter t = 0, learning rates α, β, and the batch counter l = 0.

while not done do

Use policy ${\pi }_{{\theta }_l}$ to carry out the flight mission and collect a batch of samples ${\mathcal{D}=\{(s}_t,a_t,r_t^e,s_{t+1})\}$

According to the FIFO principle, update playback buffer B with $\mathcal{D}$.

Update step counter t = t + len($\mathcal{D}$)

if attitude angle error exceeds threshold, then

l = l + 1

Using the samples in B to construct a task set, and dividing the task set into a support set and a query set         Utilize the support set to compute adaptive parameters

${\theta }_i^{\prime }=\theta -\alpha {\nabla }_{\theta }L\left(f_{\theta }\right)$         Utilize the query set to update the policy network parameters $\theta =\theta -\beta {\nabla }_{\theta }L\left(f_{{\theta }_i^{\prime }}\right)$

end if end

4. Results and Discussion

The experiments were conducted on a hardware platform equipped with AMD Ryzen 9 7945HX CPU using Windows 11 OS, version 23H2 and were based on the open-source deep learning framework PyTorch, version 1.10.1. Our experiments are also applicable to the TensorFlow framework and can run stably on an Ubuntu system.

The limitation on the deflection of control surfaces has greater practical significance in attitude control tasks. In the experiments involving attitude control tasks at this stage, the loss limit is set to 8, and the training results of the controller are shown in Figure 7.

As can be seen from the above figure, TRPO [34] has a larger rudder deflection loss, reflecting its high-frequency rudder control policy. CPO and PPO cannot complete training to learn a policy that achieves superior reward. SAC [35] has a greater rudder deflection loss than ESB-CPO, and its reward convergence is slower than TRPO and ESB-CPO. ESB-CPO’s performance is close to TRPO’s in the early stages of the training, and the rewards converge rapidly. Thereafter, rudder deflection loss decreases gradually, which is beneficial to the safe flight of the aircraft, but due to the limitation of the amount of rudder deflection, the return is close to that of TRPO. The training process of the controller significantly illustrates how our method works. By adding rudder deflection constraints to the attitude control task, in the early stage, the controller effectively explores under very loose constraints, with both high rewards and losses. In the later stage, the controller gradually tries to avoid unsafe situations, and the rudder deflection loss gradually decreases. The results show that this method allows for early loss overturning and violation of constraints, and the constraint policy ultimately returns to the safe zone, ensuring smooth rudder deflection and improving flight safety.

4.1. Assessment Method

To assess the control quality of the control algorithm, it is first necessary to determine whether it is stable or not based on the simulation results. If it is stable, the time domain control index is calculated, and then the control quality score of the algorithm is calculated; if it is unstable, the control quality score of the algorithm is recorded as 0.

The control quality score is calculated based on the satisfaction of time-domain indicators. For each indicator, if it meets the set scoring criteria, it can obtain the corresponding score for that indicator; otherwise, no score will be awarded for that indicator. The cumulative score of each indicator is the control quality score ${\mathrm{q}}_{\mathrm{i}}$ of the algorithm, where i is the simulation case number. The scoring criteria and values for each indicator are shown in

Table 1. Scoring criteria and values for each indicator.

Indicator Subjects		Scoring Criteria	Indicator Score
Pitch channel First incentive	Steady state error	$e_{ss,pitch,1}\le 1.0°$	0.1
	Adjusting time	$t_{s,pitch,1}\le 4.0 \mathrm{s}$	0.05
	Overshoot	${\sigma }_{pitch,1}\le 3.0°$	0.1
Pitch channel Second incentive	Steady state error	$e_{ss,pitch,2}\le 1.0°$	0.1
	Adjusting time	$t_{s,pitch,2}\le 2.0 \mathrm{s}$	0.05
	Overshoot	${\sigma }_{pitch,2}\le 1.5°$	0.1
Yaw channel First incentive	Steady state error	$e_{ss,yaw,1}\le 1.0°$	0.1
	Adjusting time	$t_{s,yaw,1}\le 5.0 \mathrm{s}$	0.05
	Overshoot	${\sigma }_{yaw,1}\le 5.0°$	0.1
Yaw channel Second incentive	Steady state error	$e_{ss,yaw,2}\le 1.0°$	0.1
	Adjusting time	$t_{s,yaw,2}\le 5.0 \mathrm{s}$	0.05
	Overshoot	${\sigma }_{yaw,2}\le 5.0°$	0.1
$\mathrm{Total}$			1.0

.

Under the disturbances of aerodynamic parameters, wind, and other parameters, a total of N simulation cases were run. Based on the stability, control quality of each case, and the distribution of the quality of a group of cases, the total score S of the algorithm was obtained according to the following method:

$$\mathrm{S}={\mathrm{C}}_1{\mathrm{S}}_1+{\mathrm{C}}_2{\mathrm{S}}_2+{\mathrm{C}}_3{\mathrm{S}}_1{\mathrm{S}}_3$$

(30)

${\mathrm{C}}_1,{\mathrm{C}}_2$, and ${\mathrm{C}}_3$ represent the full scores for different individual items, as detailed in

Table 2. Full marks for each individual item.

Symbol	Meaning	Value
${\mathrm{C}}_1$	Full marks for the stabilization ratio term	30
${\mathrm{C}}_2$	Full marks for control quality items	40
${\mathrm{C}}_3$	Full marks for indicator dispersal items	30
Total		100

below.

${\mathrm{S}}_1$ is the scoring coefficient for the stability proportion item, which represents the proportion of stable cases among all the cases:

$${\mathrm{S}}_1={\displaystyle \frac{{\mathrm{N}}_{\mathrm{s}}}{\mathrm{N}}}$$

(31)

where ${\mathrm{N}}_{\mathrm{s}}$ is the number of stable cases and $\mathrm{N}$ is the total number of cases.

${\mathrm{S}}_2$ is the scoring coefficient for the control quality item, which represents the average score of the control quality of each case:

$${\mathrm{S}}_2={\displaystyle \frac{{\sum }_{\mathrm{i}=1}^{\mathrm{N}}{\mathrm{q}}_{\mathrm{i}}}{\mathrm{N}}}$$

(32)

${\mathrm{S}}_3$ is the scoring coefficient for the indicator dispersion item, which is calculated based on the dispersion degree of the control quality scores of the stable cases:

$${\mathrm{S}}_3=1-\mathrm{V}$$

(33)

where V is the dispersion coefficient of the control quality scores for the stable cases. Let ${\mathrm{q}}_{\mathrm{s},\mathrm{i}}$ represent the control quality score of each stable case, and the array ${\mathrm{q}}_{\mathrm{s}}=\left\{{\mathrm{q}}_{\mathrm{s},1},{\mathrm{q}}_{\mathrm{s},2},...\right\}$. The dispersion coefficient V is the ratio of the standard deflection to the mean of ${\mathrm{q}}_{\mathrm{s}}$:

$$\mathrm{V}={\displaystyle \frac{\mathrm{s}\mathrm{t}\mathrm{d}({\mathrm{q}}_{\mathrm{s}})}{\mathrm{m}\mathrm{e}\mathrm{a}\mathrm{n}({\mathrm{q}}_{\mathrm{s}})}}$$

(34)

Then, to evaluate the real-time performance of the algorithm, a total of N examples were run under the disturbance of aerodynamic parameters and wind. During online learning, when the pitch angle or yaw angle deviation exceeds the set threshold, the condition for triggering learning is met, and timing is started. When the next action ends, timing is stopped. The definition of the control algorithm’s single calculation time is as follows:

$${\mathrm{T}}_i=t_{offi}-t_{oni}$$

(35)

where ${\mathrm{T}}_i$ is the single calculation time of the control algorithm when the i-th condition is triggered; $t_{oni}$ is the time when the i-th condition is triggered and the timer starts counting; $t_{offi}$ is the time at which the timing stops when the i-th condition is triggered and the next action ends.

So, the average time consumption of the control algorithm is as follows:

$$\overline{\mathrm{T}}={\displaystyle \frac{{\sum }_{i=1}^{\mathrm{N}}{\mathrm{T}}_i}{\mathrm{N}}}$$

(36)

4.2. Experimental Details

During the simulation process, we obtain the state quantities of the aircraft by numerical integration using the fourth-order Runge–Kutta method with a fixed step size of $∆t=0.002 \mathrm{s}$. The model parameters and initial state of the aircraft are shown in

Table 3. Model parameters and initial state of the aircraft.

Parameter	Symbol	Value	Dimension
Mass	m	30	kg
Wingspan	${\mathrm{S}}_w$	3	$\mathrm{m}$
Reference area	${\mathrm{S}}_{\mathrm{r}\mathrm{e}\mathrm{f}}$	1.5	${\mathrm{m}}^2$
Reference chord length	${\mathrm{L}}_{\mathrm{r}\mathrm{e}\mathrm{f}}$	0.469	$\mathrm{m}$
Centre of mass in the theoretical vertex system X-axis coordinates	${\mathrm{x}}_{\mathrm{c}}$	0.632	$\mathrm{m}$
Centre of mass in the theoretical vertex system Y-axis coordinates	${\mathrm{y}}_{\mathrm{c}}$	0.0473	$\mathrm{m}$
Centre of mass in the theoretical vertex system Z-axis coordinates	${\mathrm{z}}_{\mathrm{c}}$	0.0014	$\mathrm{m}$
Initial position random ranges	$H_0$	(200, 400)	$\mathrm{m}$
Initial speed random ranges	$V_0$	(25, 40)	$\mathrm{m}/\mathrm{s}$
Initial pitch angle	${\theta }_0$	0.0	$°$
Initial yaw angle random ranges	${\psi }_0$	(−180, 180)	$°$
Initial roll angle	${\Phi }_0$	0.0	$°$
Initial pitch rate	$q_0$	0.0	$°/\mathrm{s}$
Initial yaw rate	$r_0$	0.0	$°/\mathrm{s}$
Initial roll a rate	$p_0$	0.0	$°/\mathrm{s}$

.

During the training phase, the target attitude angles were set as shown in Equation (6). The goal of training is to complete the transition process within 20 s, and the simulation step size is 0.02 s, i.e., 1000 steps. Specific network hyperparameters are shown in

Table 4. Hyperparameters used in training.

Name	Value	Name	Value
check_freq	25	min_rel_budget	1.0
cost_limit	8	safety_budget	15
entropy_coef	0.01	saute_discount_factor	0.99
epochs	500	test_rel_budget	1.0
gamma	0.99	unsafe_reward	−1.0
lam	0.95	save_freq	10
lam_c	0.95	seed	0
max_grad_norm	0.5	steps_per_epoch	10,000
num_mini_batches	16	target_kl	0.01
pi_lr	0.0003	train_pi_iterations	80
max_ep_len	1000	train_v_iterations	40
max_rel_budget	1.0	vf_lr	0.001

.

During the testing phase, the goal of testing is to complete the transition process within 30 s, and the simulation step size is 0.02 s, i.e., 1500 steps. Specific network hyperparameters are shown in

Table 5. Hyperparameters used in testing.

Name	Value	Name	Value
max_ep_len	1500	qry_size	80
buffer_size	1000	dist_angle	0.8
batch_size	200	learning_rate	1 × 10−5
minimal_size	200	gamma	0.99
sup_size	120	lam	0.95

. The target attitude angles, specifically the target pitch angle and the target yaw angle, were set to vary piecewise over time. The equations for these changes are as follows:

$${\theta }_{cx}=\left\{\begin{array}{cc}10.0°& 0.0 \mathrm{s} \le t<3.0 \mathrm{s}\\ 15.0°& 3.0 \mathrm{s}\le t<10.0 \mathrm{s}\\ 12.0°& 10.0 \mathrm{s}\le t\le 30.0 \mathrm{s}\end{array}\right.$$

(37)

$${\psi }_{cx}=\left\{\begin{array}{cc}0.0°& 0.0 \mathrm{s}\le t<15.0 \mathrm{s}\\ 20.0°& 15.0 \mathrm{s}\le t<22.0 \mathrm{s}\\ 0.0°& 22.0 \mathrm{s}\le t\le 30.0 \mathrm{s}\end{array}\right.$$

(38)

Assuming that the components of the wind velocity vector $\overrightarrow{\omega }$ in the ground coordinate system are ${\left[V_{w_x},V_{w_y},V_{w_z}\right]}^T$, and the aircraft is in a horizontal steady wind field, the wind velocity vector lies within the horizontal plane, and the wind direction angle $A_w$ is defined as follows: According to the right-hand rule, starting from the ground coordinate system’s O_gY_g axis, the angle rotated around the O_gY_g axis to the wind velocity vector, the wind speed in the ground coordinate system is calculated using the following formula:

$$\left[\begin{array}{c}{V}_{w_x}\\ V_{w_y}\\ V_{w_z}\end{array}\right]=\left[\begin{array}{c}{v}_{w}cos{A}_w\\ 0\\ {-v}_{w}sin{A}_w\end{array}\right]$$

(39)

We conducted 100 simulations, with the wind direction angle $A_w$ randomly selected from the interval [0°, 360°] and the wind speed $v_w$ randomly selected from the interval [5, 60] (in m/s). Meanwhile, we added random white noise with mean 0 and variance from 1 to 10 to the aerodynamic parameters. The total scores for both the offline flight control method based on ESB-CPO and the meta-learning online flight control method based on ESB-CPO are presented in

Table 6. Attitude control mission assessment results.

Symbol	Meaning	Value
${\mathrm{S}}_{\mathrm{o}\mathrm{f}\mathrm{f}}$	The total score of the offline flight algorithm	82.5
${\mathrm{S}}_{\mathrm{o}\mathrm{n}}$	The total score of the online flight algorithm	100

.

As shown in Table 6, the overall score for executing the aircraft attitude angle control task using the offline control law is 82.5 points, while the overall score for performing the same task using the online control law is 100 points. By introducing online learning, the adaptability of attitude control to comprehensive errors such as aerodynamic parameters and wind has improved by 21% compared to offline learning.

We calculated the computation time of the control algorithm 1000 times, and the average computation time per time is 0.6 ms. The CPU frequency is 2.5 GHz, which is converted to 1 GHz. The online stage control algorithm has a single computation time of about 1.5 ms, which meets the real-time requirements.

The attitude angle control test results for the offline flight control method based on ESB-CPO and the online flight control method based on meta-learning with ESB-CPO are shown in Figure 8 and Figure 9, respectively.

As can be seen from the above figures, the flight control method based on ESB-CPO can reduce the frequency of control surface movements by incorporating control surface losses to limit their rotation. Except when step commands arrive, the control surfaces will quickly deflect to track the attitude angles, and the control surface deflection remains stable during other time periods. In the offline phase of attitude control tasks, due to the large variation in the step commands for the attitude angles, the control law cannot quickly complete the attitude angle tracking task. However, in the online phase of attitude control tasks, the control law can learn online to adjust the aircraft control policy to complete the attitude angle control task and reduce the rudder deflection, which ensures the safety and stability of the aircraft during the flight process.

As CPO and PPO cannot perform the attitude angle tracking task, only TRPO and SAC test results are shown in Figure 10 and Figure 11.

As can be seen from Figure 10 and Figure 11, the flight control method based on TRPO achieves the purpose of controlling attitude accuracy by increasing the frequency and amplitude of the rudder deflection, which would lead to a greater perturbation of the aircraft during the flight process and cause a major potential hazard to flight safety. In contrast, with the SAC-based flight control method, the elevator rudder deflection frequency is significantly faster, leading to a decrease in its tracking stability.

5. Conclusions

To address the issues of inefficiency, safety, and stability in the intelligent flight process of UAVs, we propose a new architecture for online safe flight control based on constrained reinforcement learning. Firstly, a flight control simulation environment is established based on PyFME for offline training and online learning. Secondly, to avoid flight accidents or mission failures caused by online learning, the ESB-CPO algorithm is used for flight control, and a constrained optimization problem is constructed based on the trust region method, which ensures the safety and stability of the aircraft during flight by introducing the Lyapunov stability requirement and limiting the rudder deflection loss. Finally, meta-learning is combined with the ESB-CPO algorithm to perform attitude angle tracking tasks. Experimental results show that the overall score of the aircraft attitude angle control task is 100 points, and the adaptability of attitude control to comprehensive errors, including aerodynamic parameters and wind, improves by 21% compared to offline learning after introducing online learning, indicating that the online control law can adapt the control law according to the environment to ensure the safety and stability of the aircraft during flight. In the current study, research is conducted only in a simulated environment, which may not fully capture the complexities and uncertainties present in the real world. In future work, experiments need to be conducted in more complex scenarios, such as communication interference or motor faults [36,37,38], to explore and improve control strategies. At the same time, we will challenge various aircraft models in order to translate research results into practical applications.