Reinforcement Learning for Dual-Control Aircraft Six-Degree-of-Freedom Attitude Control with System Uncertainty

Abstract

This article proposes a near-optimal control strategy based on reinforcement learning, which is applied to the six-degree-of-freedom (6-DoF) attitude control of dual-control aircraft. In order to solve the problem that the existing reinforcement learning is difficult to apply to the high-dimensional multiple-input multiple-output (MIMO) systems, the Long Short-Term Memory (LSTM) neural network is introduced to replace the polynomial network in the adaptive dynamic programming (ADP) technique. Meanwhile, based on the Lyapunov method, a novel online adaptive updating law of LSTM neural network weights is given, and the stability of the system is verified. In the simulation process, the algorithm proposed in this article is applied to the six-degree-of-freedom attitude control problem of dual-control aircraft with system uncertainty. The simulation results show that the algorithm can achieve near-optimal control.

1. Introduction

Aircraft attitude control is an important part of the design of aircraft autopilot. With the increase in aircraft flying altitude and speed, pure aerodynamic control has been unable to meet the tracking requirements of attitude control commands. Therefore, some scholars have proposed a dual-control strategy of direct force and aerodynamic force. When the aerodynamic force cannot meet the required overload, direct force is provided by the reaction jet to assist the aircraft in establishing the required attitude and improve system dynamic response performance [1,2,3,4,5,6].

In general, the aerodynamic force is generated by the attitude angle and tail fins of the aircraft, and the direct force is generated by the reaction jet. Due to the limitations of aircraft layout, the volume of the attitude control engine is generally small, and the fuel carried is also limited (the disposable solid fuel rocket is usually used). During the flight process of the aircraft, it is always accompanied by attitude adjustment. How to reduce the fuel consumption of the attitude control engine is the key problem in the design of the controller. Once the fuel is exhausted in advance, the dynamic response of the aircraft will decline, and even the controller will diverge. That is to say, how to ensure the optimality of control input is a key point of aircraft attitude control.

Since the optimal control theory was proposed in the 1950s, it has been widely used in the field of aircraft control. For linear systems, the most common method is to design a quadratic cost function and solve the linear Riccati equation to obtain the optimal control law. However, for nonlinear systems, solving the nonlinear partial differential Hamilton–Jacobi–Bellman (HJB) equation is a very complex problem, especially when considering external interference and system uncertainty, which make it more difficult to solve the equation and limit the practical application of the optimal control theory to a certain extent.

With the development of neural network techniques, reinforcement learning is a recently emerging near-optimal control method. Reinforcement learning algorithms are mainly divided into model-dependent and model-independent. Adaptive dynamic programming (ADP) is widely used as a model-based reinforcement learning algorithm. This method was first proposed by Werbos [7]. Its basic logic is to use a neural network to approximate the optimal cost function, so as to avoid the “disaster of dimensionality” problem in dynamic programming calculation and provide a convenient and effective solution for the optimal control problem of high-dimensional nonlinear systems. This method combines modern control theory with an intelligent control algorithm, which is not a complete “black box” strategy, ensuring the credibility of the algorithm. Considering that the offline iterative ADP algorithm has difficulty in ensuring stability when the system structure changes or there is external interference, the online iterative ADP algorithm is gradually recognized by scholars and has been widely developed and applied [8,9,10,11,12]. Pang [13] used the ADP algorithm to solve the optimal control problem of continuous-time linear periodic systems. Rizvi [14] used the ADP algorithm to solve linear zero-sum differential game problems, obtained complete system measurement values by introducing an observer, and proposed two ADP algorithms, namely the policy iteration and value iteration algorithms. For linear time-varying systems, Xie [15] proposed an ADP algorithm that introduced a virtual system to replace the original system, thus avoiding the integration problem in iterative operation. In the article [16], Jia designed a data-driven ADP algorithm to suppress the Pogo vibration of liquid rockets. Nie [17] designed an ADP algorithm based on a model-free single-network adaptive critic method for non-affine systems such as solid-rocket-powered vehicles, which can achieve optimal control of trajectory tracking for solid-rocket-powered vehicles. In the article [18], Xue designed a novel integral ADP scheme for input-saturated continuous-time nonlinear systems, and through event-triggered control law, the computational burden and communication cost were reduced. In the articles [19,20], the ADP scheme was applied to aircraft guidance law design. However, in the existing ADP algorithm, how to deal with the MIMO system is a problem to be solved. In the articles [21,22,23,24,25,26], the ADP technique was applied to the application control system, such as attitude control of hypersonic aircraft [21], satellite control allocation [22], multi-target cooperative control [23], formation of quadrotor UAVs [24], attitude control of morphing aircraft [25], and air-breathing hypersonic vehicle tracking control [26]. In these articles, although the processing system was a high-dimensional system, without exception, they all used a single control input, that is, a single-input multiple-output (SIMO) system. At present, the ADP algorithm for the MIMO system rarely appears. In the process of the author’s reproduction of the existing ADP algorithm, the MIMO system will cause the polynomial neural network to easily fall into saturation, and the convergence speed is very slow, even unable to converge. In this context, how to improve the network depth is an important problem to be solved to promote the application of the ADP algorithm.

To solve this problem, some scholars proposed to use other more complex neural networks instead of polynomial neural networks to improve the fitting ability of the ADP algorithm, such as RBF neural networks. In the article [27], Zhang designed an ADP algorithm based on a sliding mode surface for nonlinear switched systems. The algorithm uses the integral sliding mode term to combat the disturbance of the system and ensure the stability of the system during the switching process and uses the ADP algorithm to ensure the optimality of the control input. In the ADP algorithm, the RBF neural network is used instead of the polynomial network to realize the optimal control of the MIMO-switched system. The simulation process uses a dual-inverted pendulum system instead of the actual application system.

In fact, there is no essential difference between the chained neural network and the polynomial neural network, but the activation function is replaced, so the fitting ability of the network is slightly increased. In the existing article, the ADP algorithm based on the chained neural network has not been applied to the actual MIMO system. Considering the shortcomings of chained neural networks, this paper introduces a kind of recurrent network with gating units, namely the LSTM neural network [28,29,30] instead of the polynomial network. As a complex network, the LSTM neural network has a strong fitting ability, which can effectively solve the problem of insufficient fitting ability of multinomial networks. An additional term is introduced into the optimal control law to ensure the boundedness of the closed-loop system.

When the neural network is replaced, the following problem is how to design the weight update law. In the existing ADP algorithm, the design method of weight update law is to take the value of the Hamilton function as the error and perform a partial derivative operation on the network weights, respectively, so as to obtain the gradient of weight decreasing along the error, and then obtain the update law of each weight of the network. This method is intuitive and effective, but when the complexity of the network increases, the calculation of the gradient becomes very complex, resulting in the gradient descent method no longer being applicable. At this time, a new design method of network weight update law is needed to replace the gradient descent method.

In this paper, an adaptive dynamic programming algorithm based on the LSTM neural network (ADP-LSTM) is proposed to solve the optimal 6-DoF attitude control problem of dual-control aircraft. The main contributions of this paper are as follows:

(1)

A reinforcement learning near-optimal control method based on the LSTM neural network is proposed, which is applied to the 6-DoF attitude control of dual-control aircraft. Different from the existing algorithms, this algorithm does not need to decouple the nonlinear aircraft attitude dynamics model, and retains the internal characteristics of the system as much as possible. The algorithm can effectively solve the optimal control problem of the MIMO nonlinear control system.

(2)

Based on the nonlinear optimal control theory, an additional term based on output feedback is introduced to ensure that the closed-loop system with disturbance is bounded and converges in the small neighborhood of the control command.

(3)

Based on the Lyapunov method, the online adaptive updating law of LSTM neural network weights is given. All the updating laws are analytical, which avoids the excessive burden of system operation caused by large-scale real-time operation and proves the stability of the system.

(4)

In the simulation analysis, it is verified that the algorithm can effectively solve the optimal 6-DoF attitude control problem of dual-control aircraft.

The rest of this paper is arranged as follows: In the Section 2, the 6-DoF attitude dynamics model of dual-control aircraft is established. In the Section 3, based on the nonlinear optimal control theory, the nonlinear partial differential HJB equation is designed, and the optimal controller is designed. In the Section 4, the design method of a near-optimal controller based on the ADP-LSTM technique is given, the novel online update law of LSTM neural network weights is designed based on the Lyapunov method, and the stability of the system is proved. In the Section 5, the ADP-LSTM is applied to the 6-DoF attitude control problem of dual-control aircraft, and the simulation process is analyzed. The Section 6 is the conclusion.

2. Attitude Dynamics Model of Dual-Control Aircraft

As shown in Figure 1, dual-control aircraft’s pitch and yaw channels have two control inputs, i.e., tail fins and reaction jets. Since the direct force is perpendicular to the axis of the aircraft, it will not affect the rolling channel, so there is only one control input in the roll channel, i.e., tail fins. Among them, the aircraft has four tail fins in a cross layout. Two vertical fins provide ${\delta }_y$, two horizontal fins provide ${\delta }_z$, and ${\delta }_x$ is generated by the differential between horizontal fins and vertical fins.

The missile body coordinate system $o{x}_1{y}_1{z}_1$ and the missile velocity coordinate system $o{x}_3{y}_3{z}_3$ are defined in Figure 1. The axis $o{x}_1$ is the longitudinal axis of the missile and the axis $o{x}_3$ is along $V$, the velocity of the missile. The axis $o{y}_1$ is in the plane of symmetry of the missile. The relationship between the two coordinate systems is determined by two angles, i.e., the angle of attack $\alpha$ and the sideslip angle $\beta$. Let ${\omega }_z$ denote the pitch rotational rate. We define the aerodynamic parameters in the elevation loop of the dual-control system as $a_1=-M_z^{{\omega }_z}/J_z$, $a_2\left(\alpha \right)=-M_z^{\alpha }\left(\alpha \right)/J_z$, $a_3=-M_z^{{\delta }_z}/J_z$, $a_4=Y^{\alpha }/\left(mV\right)$, $a_5=Y^{{\delta }_z}/\left(mV\right)$, $a_6=-(J_x-J_y)/J_z$, $b_1=-M_y^{{\omega }_y}/J_y$, $b_2=-M_y^{\beta }/J_y$, $b_3=-M_y^{{\delta }_y}/J_y$, $b_4=-Z_{ }^{\beta }/\left(mV\right)$, $b_5=-Z_{ }^{{\delta }_y}/\left(mV\right)$, $b_6=-(J_z-J_x)/J_y$, $c_1=-M_x^{{\omega }_x}/J_x$, $c_3=-M_x^{{\delta }_x}/J_x$, $k_y=1/\left(mV\right)$, $k_z=-1/\left(mV\right)$, $l_y=l/J_y$, $l_z=-l/J_z$.

Where ${{\rm M}}_z^{{\omega }_z}$ is the partial derivative of the pitching moment ${{\rm M}}_z$ with respect to the pitch rate ${\omega }_z$, ${{\rm M}}_z^{\alpha }$ is the partial derivative of the pitching moment ${{\rm M}}_z$ with respect to the angle of attack $\alpha$, $M_z^{{\delta }_z}$ is the partial derivative of the pitching moment ${{\rm M}}_z$ with respect to the rudder deflection angle ${\delta }_z$, ${{\rm M}}_y^{{\omega }_y}$ is the partial derivative of the yaw moment ${{\rm M}}_y$ with respect to the yaw rate ${\omega }_y$, ${{\rm M}}_y^{\beta }$ is the partial derivative of the yaw moment ${{\rm M}}_y$ with respect to the sideslip angle $\beta$, $M_y^{{\delta }_y}$ is the partial derivative of the yaw moment ${{\rm M}}_y$ with respect to the rudder deflection angle ${\delta }_y$, ${{\rm M}}_x^{{\omega }_x}$ is the partial derivative of the roll moment ${{\rm M}}_x$ with respect to the roll rate ${\omega }_x$, $M_x^{{\delta }_x}$ is the partial derivative of the roll moment ${{\rm M}}_x$ with respect to the rudder deflection angle ${\delta }_x$, $J_x$, $J_y$ and $J_z$ are the component of moment of inertia on axis $o{x}_1$, $o{y}_1$ and $o{z}_1$ respectively, and $l$ is the distance from the point of the lateral thrust to the mass center of the missile.

Considering entering the terminal guidance stage, the main engine of the aircraft is shut down, the aircraft mass and velocity are constant, and the attitude dynamics model of the dual-control aircraft is established through the above aerodynamic parameters.

$$\dot{\alpha }={\omega }_z-{\omega }_x\beta -a_4\alpha -a_5{\delta }_z-k_y{F}_{Ty}$$

(1)

$$\dot{\beta }={\omega }_y+{\omega }_x\alpha -b_4\beta -b_5{\delta }_y-k_z{F}_{Tz}$$

(2)

$${\dot{\omega }}_x=-c_1{\omega }_x-c_3{\delta }_x$$

(3)

$${\dot{\omega }}_y=-b_1{\omega }_y-b_2\beta -b_3{\delta }_y-b_6{\omega }_z{\omega }_x-l_y{F}_{Tz}$$

(4)

$${\dot{\omega }}_z=-a_1{\omega }_z-a_2\alpha -a_3{\delta }_z-a_6{\omega }_x{\omega }_y-l_z{F}_{Ty}$$

(5)

The external forces on the missile are gravity, aerodynamic force, and direct force, so the missile overload dynamic model of the pitch channel can be written as

$$n_y=\frac{V}{g}{a}_4\alpha +\frac{V}{g}{a}_5{\delta }_z+\frac{V}{g}{k}_y{F}_z$$

(6)

The derivative of Equation (6) is

$${\dot{n}}_y=\frac{V}{g}{a}_4\dot{\alpha }+\frac{V}{g}{a}_5{\dot{\delta }}_z+\frac{V}{g}{k}_y{\dot{F}}_z$$

(7)

By substituting Equations (1) and (3) into Equation (7), we can obtain

$${\dot{n}}_y=\frac{V}{g}{a}_4({\omega }_z-{\omega }_x\beta -a_4\alpha -a_5{\delta }_z-k_y{F}_{Ty})+\frac{V}{g}{a}_5{\dot{\delta }}_z+\frac{V}{g}{k}_y{\dot{F}}_z$$

(8)

With further simplification, we obtain

$${\dot{n}}_y=\frac{V}{g}{a}_4{\omega }_z-\frac{V}{g}{a}_4\beta -a_4\left(\frac{V}{g}{a}_4\alpha +\frac{V}{g}{a}_5{\delta }_z+\frac{V}{g}{k}_y{F}_z\right)+\frac{V}{g}{a}_5{\dot{\delta }}_z+\frac{V}{g}{k}_y{\dot{F}}_z$$

(9)

$${\dot{n}}_y=\frac{V}{g}{a}_4{\omega }_z-\frac{V}{g}{a}_4{\omega }_x\beta -a_4{n}_y+\frac{V}{g}{a}_5{\dot{\delta }}_z+\frac{V}{g}{k}_y{\dot{F}}_z$$

(10)

The dynamic response of the controller actuator is considered as the inertial system, i.e.,

$${\dot{\delta }}_z=-\frac{1}{{\tau }_1}{\delta }_z+\frac{1}{{\tau }_1}{\delta }_{zc}$$

(11)

$${\dot{F}}_z=-\frac{1}{{\tau }_2}{F}_z+\frac{1}{{\tau }_2}{F}_{zc}$$

(12)

where ${\tau }_1$ and ${\tau }_2$ are the mechanical constants of the actuator, respectively.

By substituting Equation (6) into Equation (5), we obtain

$$\begin{array}{c}{\dot{\omega }}_z=-a_1{\omega }_z-a_2\left(\frac{g}{V{a}_4}{n}_y-\frac{a_5}{a_4}{\delta }_z-\frac{k_y}{a_4}{F}_z\right)-a_6{\omega }_x{\omega }_y-a_3{\delta }_z-l_z{F}_z\\ =-a_1{\omega }_z-\frac{a_{2}g}{V{a}_4}{n}_y+\frac{a_2{a}_5}{a_4}{\delta }_z+\frac{a_2{k}_y}{a_4}{F}_z-a_6{\omega }_x{\omega }_y-a_3{\delta }_z-l_z{F}_z\\ =-a_1{\omega }_z-\frac{a_{2}g}{V{a}_4}{n}_y-a_6{\omega }_x{\omega }_y+\left(\frac{a_2{a}_5}{a_4}-a_3\right){\delta }_z+\left(\frac{a_2{k}_y}{a_4}-l_z\right)F_z\end{array}$$

(13)

Similarly, we can obtain the dynamic model of the yaw channel as

$${\dot{n}}_z=\frac{V}{g}{b}_4{\omega }_y-\frac{V}{g}{b}_4{\omega }_x\alpha -b_4{n}_z+\frac{V}{g}{b}_5{\dot{\delta }}_y+\frac{V}{g}{k}_z{\dot{F}}_y$$

(14)

$${\dot{\omega }}_y=-b_1{\omega }_y-\frac{b_{2}g}{V{b}_4}{n}_z-b_6{\omega }_x{\omega }_z+\left(\frac{b_2{b}_5}{b_4}-b_3\right){\delta }_y+\left(\frac{b_2{k}_z}{b_4}-l_y\right)F_y$$

(15)

The attitude dynamics model of rolling channel considering three-channel coupling is

$$\dot{\gamma }={\omega }_x-\tan \vartheta \left({\omega }_y\cos \gamma -{\omega }_z\sin \gamma \right)$$

(16)

$${\dot{\omega }}_x=-c_1{\omega }_x-c_3{\delta }_x$$

(17)

Finally, the aircraft attitude dynamics model is obtained as

$$\dot{\mathit{X}}=\left[\begin{array}{c}{\omega }_x-\tan \vartheta \left({\omega }_y\cos \gamma -{\omega }_z\sin \gamma \right)\\ -c_1{\omega }_x-c_3{\delta }_x\\ -b_1{\omega }_y-\frac{b_{2}g}{V{b}_4}{n}_z-b_6{\omega }_x{\omega }_z+\left(\frac{b_2{b}_5}{b_4}-b_3\right){\delta }_y+\left(\frac{b_2{k}_z}{b_4}-l_y\right)F_y\\ -a_1{\omega }_z-\frac{a_{2}g}{V{a}_4}{n}_y-a_6{\omega }_x{\omega }_y+\left(\frac{a_2{a}_5}{a_4}-a_3\right){\delta }_z+\left(\frac{a_2{k}_y}{a_4}-l_z\right)F_z\\ \frac{V}{g}{a}_4{\omega }_z-\frac{V}{g}{a}_4{\omega }_x\beta -a_4{n}_y+\frac{V}{g}{a}_5\left(-\frac{1}{{\tau }_1}{\delta }_z+\frac{1}{{\tau }_1}{\delta }_{zc}\right)+\frac{V}{g}{k}_y\left(-\frac{1}{{\tau }_2}{F}_z+\frac{1}{{\tau }_2}{F}_{zc}\right)\\ \frac{V}{g}{b}_4{\omega }_y-\frac{V}{g}{b}_4{\omega }_x\alpha -b_4{n}_z+\frac{V}{g}{b}_5\left(-\frac{1}{{\tau }_1}{\delta }_y+\frac{1}{{\tau }_1}{\delta }_{yc}\right)+\frac{V}{g}{k}_z\left(-\frac{1}{{\tau }_2}{F}_y+\frac{1}{{\tau }_2}{F}_{yc}\right)\\ -\frac{1}{{\tau }_1}{\delta }_x+\frac{1}{{\tau }_1}{\delta }_{xc}\\ -\frac{1}{{\tau }_1}{\delta }_y+\frac{1}{{\tau }_1}{\delta }_{yc}\\ -\frac{1}{{\tau }_1}{\delta }_z+\frac{1}{{\tau }_1}{\delta }_{zc}\\ -\frac{1}{{\tau }_2}{F}_z+\frac{1}{{\tau }_2}{F}_{zc}\\ -\frac{1}{{\tau }_2}{F}_y+\frac{1}{{\tau }_2}{F}_{yc}\end{array}\right]$$

(18)

We define $\mathit{X}={\left[\begin{array}{ccccccccccc}\gamma & {\omega }_x& {\omega }_y& {\omega }_z& n_y& n_z& {\delta }_x& {\delta }_y& {\delta }_z& F_z& F_y\end{array}\right]}^T$ as the state vector, where $\mathit{u}={\left[\begin{array}{ccccc}{\delta }_{xc}& {\delta }_{yc}& {\delta }_{zc}& F_{zc}& F_{yc}\end{array}\right]}^T$ is the control vector, then Equation (18) can be written as the following state space model:

$$\dot{\mathit{X}}=\mathit{f}\left(\mathit{X}\right)+\mathit{g}\left(\mathit{X}\right)\mathit{u}+\mathit{d}$$

(19)

where

$$\mathit{f}\left(\mathit{X}\right)=\left[\begin{array}{c}{\omega }_x-\tan \vartheta \left({\omega }_y\cos \gamma -{\omega }_z\sin \gamma \right)\\ -c_1{\omega }_x-c_2\beta -c_3{\delta }_x\\ -b_1{\omega }_y-\frac{b_{2}g}{V{b}_4}{n}_z-b_6{\omega }_x{\omega }_z+\left(\frac{b_2{b}_5}{b_4}-b_3\right){\delta }_y+\left(\frac{b_2{k}_z}{b_4}-l_y\right)F_y\\ -a_1{\omega }_z-\frac{a_{2}g}{V{a}_4}{n}_y-a_6{\omega }_x{\omega }_y+\left(\frac{a_2{a}_5}{a_4}-a_3\right){\delta }_z+\left(\frac{a_2{k}_y}{a_4}-l_z\right)F_z\\ \frac{V}{g}{a}_4{\omega }_z-\frac{V}{g}{a}_4{\omega }_x\beta -a_4{n}_y-\frac{V}{g{\tau }_1}{a}_5{\delta }_z-\frac{V}{g{\tau }_2}{k}_y{F}_z\\ \frac{V}{g}{b}_4{\omega }_y-\frac{V}{g}{b}_4{\omega }_x\alpha -b_4{n}_z-\frac{V}{g{\tau }_1}{b}_5{\delta }_y-\frac{V}{g{\tau }_2}{k}_z{F}_y\\ -\frac{1}{{\tau }_1}{\delta }_x\\ -\frac{1}{{\tau }_1}{\delta }_y\\ -\frac{1}{{\tau }_1}{\delta }_z\\ -\frac{1}{{\tau }_2}{F}_z\\ -\frac{1}{{\tau }_2}{F}_y\end{array}\right],$$

$$\mathit{g}\left(\mathit{X}\right)=\left[\begin{array}{ccccc}0& 0& 0& 0& 0\\ -c3& 0& 0& 0& 0\\ 0& 0& 0& 0& 0\\ 0& 0& 0& 0& 0\\ 0& 0& \frac{V}{g{\tau }_1}{a}_5& \frac{V}{g{\tau }_2}{k}_y& 0\\ 0& \frac{V}{g{\tau }_1}{b}_5& 0& 0& \frac{V}{g{\tau }_2}{k}_z\\ \frac{1}{{\tau }_1}& 0& 0& 0& 0\\ 0& \frac{1}{{\tau }_1}& 0& 0& 0\\ 0& 0& \frac{1}{{\tau }_1}& 0& 0\\ 0& 0& 0& \frac{1}{{\tau }_2}& 0\\ 0& 0& 0& 0& \frac{1}{{\tau }_2}\end{array}\right],$$

where $\mathit{d}$ is the external disturbance.

3. Design of Optimal Control Law Based on HJB Equation

Consider continuous affine nonlinear systems with a class of uncertainties

$$\dot{\mathit{X}}\left(t\right)=\mathit{f}\left(\mathit{X}\right)+\mathit{g}\left(\mathit{X}\right)\left(\mathit{u}\left(t\right)+\mathit{d}\left(t\right)\right)$$

(20)

where $\mathit{X}\left(t\right)\in {ℝ}^n$ is the state vector of the system, $\mathit{u}\left(t\right)\in {ℝ}^m$ is the control input vector, and $\mathit{f}\left(\mathit{X}\right)\in {ℝ}^n$ and $\mathit{g}\left(\mathit{X}\right)\in {ℝ}^{n\times m}$ are the system function and control matrix, respectively.

Assumption 1. 

The nonlinear function $\mathit{f}\left(\mathit{X}\right)$ satisfies the local Lipschitz condition in the set containing the origin and f(0) = 0. The control matrix $\mathit{g}\left(\mathit{X}\right)$ is bounded.

Consider reference systems without uncertainties

$$\dot{\mathit{X}}\left(t\right)=\mathit{f}\left(\mathit{X}\right)+\mathit{g}\left(\mathit{X}\right)\mathit{u}\left(t\right)$$

(21)

Assumption 2. 

There is a symmetric positive definite matrix $\mathit{R}\in {ℝ}^{m\times m}$ such that the system uncertainty satisfies $\mathit{d}\left(t\right)={\mathit{R}}^{0.5}\overline{\mathit{d}}\left(t\right)$ and the system uncertainty $\mathit{d}\left(t\right)$ is bounded, that is, $‖\mathit{d}\left(t\right)‖\le d_e$.

Based on the above assumptions, the control system cost function is defined as

$$J\left(\mathit{X}\right)={\displaystyle {\int }_0^{\infty }\left[Q\left(\mathit{X}\right)+{\mathit{u}}^T\left(t\right)\mathit{R}\mathit{u}\left(t\right)\right]}dt$$

(22)

where $Q\left(\mathit{X}\right)={\mathit{e}}^T{\mathit{Q}}_1\mathit{e}+d_e^2$, ${\mathit{Q}}_1\in {ℝ}^{n\times n}$ is a symmetric positive definite matrix, and $\mathit{e}=\mathit{X}-{\mathit{X}}_d$ is defined as the tracking error.

Remark 1. 

The minimum value of the cost function is achieved by searching for the optimal control term ${\mathit{u}}^{\ast }\left(t\right)$. When designing the cost function, the tracking error of the system and the upper bound of the uncertainty of the system are described, and it can be seen from the definition that $\forall \mathit{X}\ne$0$, \mathit{Q}\left(\mathit{X}\right)> 0$. Therefore, when the minimum cost function is obtained, the closed-loop system state can converge to a sufficiently small neighborhood of the control command, and the interference of uncertainty is considered to achieve the effect of interference suppression.

According to the optimal control theory of nonlinear systems, the Hamiltonian function of the design reference system is

$$H\left(\mathit{X},\mathit{u},\Delta \mathit{J}\left(\mathit{X}\right)\right)=Q\left(\mathit{X}\right)+{\mathit{u}}^T\mathit{R}\mathit{u}+\Delta {\mathit{J}}^T\left(\mathit{X}\right)\left(\mathit{f}\left(\mathit{X}\right)+\mathit{g}\left(\mathit{X}\right)\mathit{u}\right)$$

(23)

where $\Delta \mathit{J}\left(\mathit{X}\right)\in {ℝ}^n$ is the partial derivative of the cost function $J\left(\mathit{X}\right)$ with respect to the state of the system $\mathit{X}$, i.e., $\Delta \mathit{J}\left(\mathit{X}\right)=\partial J\left(\mathit{X}\right)/\partial \mathit{X}$.

The optimal cost function $J^{\ast }\left(\mathit{X}\right)$ can be obtained by solving the following Hamilton–Jacobi–Bellman (HJB) Equation (24):

$$\underset{\mathit{u}}{\min }H\left(\mathit{X},\mathit{u},\Delta {\mathit{J}}^{\ast }\left(\mathit{X}\right)\right)=0$$

(24)

According to the necessary conditions $\partial H/\partial \mathit{u}=$ 0, the optimal control law is

$${\mathit{u}}^{\ast }=-\frac{1}{2}{\mathit{R}}^{-1}{\mathit{g}}^T\left(\mathit{X}\right)\Delta {\mathit{J}}^{\ast }\left(\mathit{X}\right)$$

(25)

By substituting Equation (25) into Equation (24), the HJB equation can be rewritten as

$$Q\left(\mathit{X}\right)+{\left(\Delta {\mathit{J}}^{\ast }\left(\mathit{X}\right)\right)}^T\mathit{f}\left(\mathit{X}\right)-\frac{1}{4}{\left(\Delta {\mathit{J}}^{\ast }\left(\mathit{X}\right)\right)}^T\mathit{g}\left(\mathit{X}\right){\mathit{R}}^{-1}{\mathit{g}}^T\left(\mathit{X}\right)\Delta {\mathit{J}}^{\ast }\left(\mathit{X}\right)=0$$

(26)

Remark 2. 

It can be seen that in order to obtain the optimal control law ${\mathit{u}}^{\ast }$, the above HJB equation needs to be solved to obtain the optimal cost function $J^{\ast }$ and its partial derivative to the system state $\Delta {\mathit{J}}^{\ast }\left(\mathit{X}\right)$. However, for nonlinear systems, it is very difficult to solve the HJB equation, especially in the case of considering external disturbance, so the difficulty of solving the equation further increases. On the other hand, if we can find the cost function to ensure the Hamiltonian function $H^{\ast }=0$, we can obtain the optimal control law ${\mathit{u}}^{\ast }$. In other words, through this idea, the optimal control problem can be transformed into the problem of how to obtain the optimal cost function $J^{\ast }$. In the next section, a reinforcement learning algorithm based on the LSTM neural network is proposed, which uses the LSTM neural network to fit the optimal cost function, so as to achieve approximate optimal control.

4. Design of Approximate Optimal Control Law Based on Reinforcement Learning

4.1. LSTM Neural Network

The LSTM neural network shown in Figure 2 contains three parts, i.e., a forget gate, an input gate, and an output gate. Except for the same hidden state ${\mathit{h}}_t$ as the RNNs, it also introduces a cell state for keeping the long-term memory information. At the current time t, the cell state is updated through the data from the forget gate and the input gate to achieve a long-term memory update. Then, the cell state ${\mathit{c}}_t$, the last step hidden state ${\mathit{h}}_{t-1}$, and the current step input ${\mathit{x}}_t$ are mixed via the output gate to obtain a network output. The details of the LSTM neural network can be obtained from another article [29]. The LSTM neural network is expressed as follows:

$${\mathit{o}}_t=\sigma (\mathit{n}e{\mathit{t}}_{o,t})$$

(27)

$$\mathit{n}\mathit{e}{\mathit{t}}_{o,t}={\mathit{W}}_{oh}{\mathit{h}}_{t-1}+{\mathit{W}}_{ox}{\mathit{X}}_t+{\mathit{b}}_o$$

(28)

$${\mathit{f}}_t=\sigma (\mathit{n}e{\mathit{t}}_{f,t})$$

(29)

$$\mathit{n}e{\mathit{t}}_{f,t}={\mathit{W}}_{of}{\mathit{h}}_{t-1}+{\mathit{W}}_{fx}{\mathit{X}}_t+{\mathit{b}}_f$$

(30)

$${\tilde{\mathit{c}}}_t=tanh(\mathit{n}e{\mathit{t}}_{o,t})$$

(31)

$$\mathit{n}e{\mathit{t}}_{\tilde{c},t}={\mathit{W}}_{ch}{\mathit{h}}_{t-1}+{\mathit{W}}_{cx}{\mathit{X}}_t+{\mathit{b}}_c$$

(32)

$${\mathit{i}}_t=\sigma (\mathit{n}e{\mathit{t}}_{i,t})$$

(33)

$$\mathit{n}e{\mathit{t}}_{i,t}={\mathit{W}}_{ih}{\mathit{h}}_{t-1}+{\mathit{W}}_{ix}{\mathit{X}}_t+{\mathit{b}}_i$$

(34)

$${\mathit{c}}_t={\mathit{f}}_t\odot {\mathit{c}}_{t-1}+{\mathit{i}}_t\odot {\tilde{\mathit{c}}}_t$$

(35)

$${\mathit{h}}_t={\mathit{o}}_t\odot tanh\left({\mathit{c}}_t\right)$$

(36)

where $\odot$ denotes the element-wise multiplication; ${\mathit{c}}_t\in {ℝ}^{n_s}$ is the cell state vector; ${\mathit{x}}_t\in {ℝ}^{n_x}$ is the input vector; and ${\mathit{i}}_t$, ${\mathit{f}}_t$ and ${\mathit{o}}_t$ are the input, forget, and output gates, respectively. The sigmoid function $\sigma \left(\cdot \right)$ and the hyperbolic tangent function $tanh\left(\cdot \right)$ apply point-wise to the vector elements. Furthermore, ${\mathit{W}}_{oh}\in {ℝ}^{n_s\times n_s}$, ${\mathit{W}}_{ox}\in {ℝ}^{n_s\times n_x}$, ${\mathit{W}}_{fh}\in {ℝ}^{n_s\times n_s}$, ${\mathit{W}}_{fx}\in {ℝ}^{n_s\times n_x}$, ${\mathit{W}}_{ch}\in {ℝ}^{n_s\times n_s}$, ${\mathit{W}}_{cx}\in {ℝ}^{n_s\times n_x}$, ${\mathit{W}}_{ih}\in {ℝ}^{n_s\times n_s}$, ${\mathit{W}}_{ix}\in {ℝ}^{n_s\times n_x}$, ${\mathit{b}}_o\in {ℝ}^{1\times n_s}$, ${\mathit{b}}_f\in {ℝ}^{1\times n_s}$, ${\mathit{b}}_c\in {ℝ}^{1\times n_s}$, and ${\mathit{b}}_i\in {ℝ}^{1\times n_s}$ are the weighting matrices.

According to Equation (36), we can know that the dimension of the output value of the LSTM neural network ${\mathit{h}}_t$ is related to the number of cell states. To obtain the output dimension we need, the usual method is introducing a scaling matrix $\mathit{W}$, i.e., ${\mathit{y}}_t=\mathit{W}{\mathit{h}}_t$. However, when the scale difference between the input and output of the network is large, the value of each element in the scaling matrix $\mathit{W}$ will be too large or too small, which will affect the update of other weights of the network. Therefore, we introduce a full connection layer as the scaling matrix of the network to increase the depth of the network and improve the fitting accuracy of the network, i.e.,

$${\mathit{y}}_t=\mathit{W}{\mathit{L}}_h$$

(37)

$${\mathit{L}}_h=\sigma \left(\mathit{U}{\mathit{h}}_t+{\mathit{b}}_h\right)$$

(38)

4.2. Design of Near-Optimal Control Law Based on LSTM Neural Network and Output Feedback

The optimal output value of the neural network is defined as

$$\frac{\partial {\mathit{J}}^{\ast }}{\partial \mathit{X}}={\mathit{y}}^{\ast }={\mathit{W}}^{\ast }{\mathit{L}}_h^{\ast }+\mathit{\epsilon }$$

(39)

Assumption 3. 

There exist an optimal weight ${\mathit{W}}^{\ast }$, standard bias terms ${\mathit{b}}_i^{\ast }$, ${\mathit{b}}_f^{\ast }$, ${\mathit{b}}_c^{\ast }$, ${\mathit{b}}_o^{\ast }$, and optimal network weights ${\mathit{W}}_{ih}^{\ast },{\mathit{W}}_{ix}^{\ast },{\mathit{W}}_{fh}^{\ast },{\mathit{W}}_{fx}^{\ast },{\mathit{W}}_{ch}^{\ast },{\mathit{W}}_{cx}^{\ast },{\mathit{W}}_{oh}^{\ast },{\mathit{W}}_{ox}^{\ast }$ in approximating the unknown function ${\mathit{y}}^{\ast }=\partial {\mathit{J}}^{\ast }/\partial \mathit{X}$ which can be expressed as ${\mathit{y}}^{\ast }={\mathit{W}}^{\ast }{\mathit{L}}_h^{\ast }+\mathit{\epsilon }$, where ${\mathit{L}}_h^{\ast }$ presents the optimal output value of ${\mathit{L}}_h$, and $\mathit{\epsilon }$ is the mapping error uniformly bounded as $‖\mathit{\epsilon }‖\le {\epsilon }_b$, where ${\epsilon }_b$ is an arbitrarily small positive constant. The terms of the optimal weight matrices are all constant.

The detailed structure of the ADP-LSTM algorithm can be seen in Figure 3.

The difference between the estimated value and the optimal value of the Hamilton function is defined as the training error of the LSTM neural network, i.e.,

$$e_c=\widehat{H}\left(\mathit{X},\widehat{\mathit{u}},\Delta \widehat{\mathit{J}}\left(\mathit{X}\right)\right)-H^{\ast }\left(\mathit{X},{\mathit{u}}^{\ast },\Delta {\mathit{J}}^{\ast }\left(\mathit{X}\right)\right)$$

(40)

As we know from the last section, $H^{\ast }\left(\mathit{X},{\mathit{u}}^{\ast },\Delta {\mathit{J}}^{\ast }\left(\mathit{X}\right)\right)=0$, and Equation (40) can be rewritten as

$$e_c={\left(\widehat{\mathit{y}}-{\mathit{y}}^{\ast }\right)}^T\mathit{f}\left(\mathit{X}\right)-\frac{1}{4}{\left(\widehat{\mathit{y}}-{\mathit{y}}^{\ast }\right)}^T\mathit{\chi }\left(\widehat{\mathit{y}}-{\mathit{y}}^{\ast }\right)$$

(41)

where $\mathit{\chi }=\mathit{g}\left(\mathit{X}\right){\mathit{R}}^{-1}{\mathit{g}}^T\left(\mathit{X}\right)$ is a positive definite matrix.

The difference between the neural network output and the optimal value is

$$\widehat{\mathit{y}}-{\mathit{y}}^{\ast }=\tilde{\mathit{W}}{\widehat{\mathit{L}}}_h+\widehat{\mathit{W}}{\tilde{\mathit{L}}}_h+{\mathit{\epsilon }}_0=\tilde{\mathit{y}}$$

(42)

where $\tilde{\mathit{W}}={\mathit{W}}^{\ast }-\widehat{\mathit{W}}$. By substituting Equation (42) into Equation (41), Equation (41) can be rewritten as

$$e_c={\tilde{\mathit{y}}}^{T}f\left(\mathit{X}\right)-\frac{1}{4}{\tilde{\mathit{y}}}^T\mathit{\chi }\tilde{\mathit{y}}$$

(43)

The time derivative of Equation (43) can be obtained as

$$\begin{array}{l}{\dot{e}}_c={\dot{\tilde{\mathit{y}}}}^T\mathit{f}\left(\mathit{X}\right)+{\tilde{\mathit{y}}}^T\dot{\mathit{f}}\left(\mathit{X}\right)-\frac{1}{2}{\tilde{\mathit{y}}}^T\mathit{\chi }\dot{\tilde{y}}\\ ={\dot{\tilde{\mathit{y}}}}^T\mathit{f}\left(\mathit{X}\right)+{\tilde{\mathit{y}}}^T\Delta \mathit{f}\left(\mathit{X}\right)\left(\mathit{f}\left(\mathit{X}\right)+\mathit{g}\left(\mathit{X}\right)\widehat{\mathit{u}}\right)-\frac{1}{2}{\tilde{\mathit{y}}}^T\mathit{\chi }\dot{\tilde{\mathit{y}}}\end{array}$$

(44)

where $\Delta \mathit{f}\left(\mathit{X}\right)=\frac{\partial \mathit{f}\left(\mathit{X}\right)}{\partial \mathit{X}}$.

Theorem 1. 

The following control law can ensure that the closed-loop system converges to a sufficiently small neighborhood of the control command, and the control law is approximately optimal.

$$\widehat{\mathit{u}}={\widehat{\mathit{u}}}_1+{\widehat{\mathit{u}}}_2$$

(45)

$${\widehat{\mathit{u}}}_1=-\frac{1}{2}{\mathit{R}}^{-1}{\mathit{g}}^T\left(\mathit{X}\right)\widehat{\mathit{y}}$$

(46)

$$\begin{array}{ll}{\widehat{\mathit{u}}}_2=& {\left(e_c{\tilde{\mathit{y}}}^T\Delta \mathit{f}\left(\mathit{X}\right)\mathit{g}\left(\mathit{X}\right)+{\mathit{e}}^T\overline{\mathit{Q}}\mathit{g}\left(\mathit{X}\right)\right)}^{†}·\\ & \left(-e_c{\tilde{\mathit{y}}}^T\Delta \mathit{f}\left(\mathit{X}\right)\mathit{f}\left(\mathit{X}\right)-e_c{\dot{\widehat{\mathit{y}}}}^T\mathit{f}\left(\mathit{X}\right)-{\mathit{e}}^T\overline{\mathit{Q}}\left(\mathit{f}\left(\mathit{X}\right)+{\dot{\mathit{X}}}_d\right)-K{e}_c\right)\end{array}$$

(47)

where $\overline{\mathit{Q}}=2\mathit{Q}$.

Remark 3. 

The above control law is an improved form of the optimal control law (25) in the previous section, in which ${\widehat{\mathit{u}}}_1$ ensures that the system control input is approximately optimal. As an additional term, ${\widehat{\mathit{u}}}_2$ ensures that the tracking error of the closed-loop system finally converges to a sufficiently small neighborhood of the control command. The stability analysis will be given in the next section.

4.3. Design of Online Weight Update Law for LSTM Neural Networks

For brevity, we define $\mathit{\kappa }=\Delta \mathit{f}\left(\mathit{X}\right)\mathit{f}\left(\mathit{X}\right)$, and by substituting it into the control law, we can obtain

$${\dot{\mathit{e}}}_c={\tilde{\mathit{y}}}^T\mathit{\kappa }-\frac{1}{2}{\tilde{\mathit{y}}}^T\mathit{\chi }\dot{\tilde{y}}-\frac{1}{2}{\tilde{\mathit{y}}}^T\Delta \mathit{f}\left(\mathit{X}\right)\mathit{\chi }\widehat{\mathit{y}}-K{e}_c$$

(48)

According to the definition of $\tilde{\mathit{y}}$ and Assumption 3, we know that $\dot{\tilde{\mathit{y}}}=\dot{\widehat{\mathit{y}}}$, so Equation (48) can be rewritten as

$$\begin{array}{l}{\dot{\mathit{e}}}_c={\tilde{\mathit{y}}}^T\mathit{\kappa }-\frac{1}{2}{\tilde{\mathit{y}}}^T\mathit{\chi }\dot{\widehat{\mathit{y}}}-\frac{1}{2}{\tilde{\mathit{y}}}^T\Delta \mathit{f}\left(\mathit{X}\right)\mathit{\chi }\widehat{\mathit{y}}-K{e}_c\\ ={\left(\tilde{\mathit{W}}{\widehat{\mathit{L}}}_h+\widehat{\mathit{W}}{\tilde{\mathit{L}}}_h\right)}^T\mathit{\kappa }-\frac{1}{2}{\left(\tilde{\mathit{W}}{\widehat{\mathit{L}}}_h+\widehat{\mathit{W}}{\tilde{\mathit{L}}}_h\right)}^T\mathit{\chi }\dot{\widehat{\mathit{y}}}-\frac{1}{2}{\tilde{\mathit{y}}}^T\Delta \mathit{f}\left(\mathit{X}\right)\mathit{\chi }\widehat{\mathit{y}}-K{e}_c\\ ={\widehat{\mathit{L}}}_h^T{\tilde{\mathit{W}}}^T\mathit{\kappa }+{\tilde{\mathit{L}}}_h^T{\widehat{\mathit{W}}}^T\mathit{\kappa }-\frac{1}{2}\left({\widehat{\mathit{L}}}_h^T{\tilde{\mathit{W}}}^T+{\tilde{\mathit{L}}}_h^T{\widehat{\mathit{W}}}^T\right)\mathit{\chi }\dot{\widehat{\mathit{y}}}-\frac{1}{2}{\tilde{\mathit{y}}}^T\Delta \mathit{f}\left(\mathit{X}\right)\mathit{\chi }\widehat{\mathit{y}}-K{e}_c\\ ={\widehat{\mathit{L}}}_h^T{\tilde{\mathit{W}}}^T\mathit{\kappa }+{\tilde{\mathit{L}}}_h^T{\widehat{\mathit{W}}}^T\mathit{\kappa }-\frac{1}{2}{\widehat{\mathit{L}}}_h^T{\tilde{\mathit{W}}}^T\mathit{\chi }\dot{\widehat{\mathit{y}}}-\frac{1}{2}{\tilde{\mathit{L}}}_h^T{\widehat{\mathit{W}}}^T\mathit{\chi }\dot{\widehat{\mathit{y}}}-\frac{1}{2}\left({\widehat{\mathit{L}}}_h^T{\tilde{\mathit{W}}}^T+{\tilde{\mathit{L}}}_h^T{\widehat{\mathit{W}}}^T\right)·\\ \Delta \mathit{f}\left(\mathit{X}\right)\mathit{\chi }\widehat{\mathit{y}}-K{e}_c\\ ={\widehat{\mathit{L}}}_h^T{\tilde{\mathit{W}}}^T\left(\mathit{\kappa }-\mathit{\lambda }\right)+{\tilde{\mathit{L}}}_h^T{\widehat{\mathit{W}}}^T\left(\mathit{\kappa }-\mathit{\lambda }\right)-\frac{1}{2}{\widehat{\mathit{L}}}_h^T{\tilde{\mathit{W}}}^T\mathit{\chi }\dot{\widehat{\mathit{y}}}-\frac{1}{2}{\tilde{\mathit{L}}}_h^T{\widehat{\mathit{W}}}^T\mathit{\chi }\dot{\widehat{\mathit{y}}}-K{e}_c\end{array}$$

(49)

Also for brevity, we define $\mathit{\omega }=\frac{1}{2}\mathit{\chi }\dot{\widehat{\mathit{y}}}$ and $\mathit{\lambda }=\frac{1}{2}\Delta \mathit{f}\left(\mathit{X}\right)\mathit{\chi }\widehat{\mathit{y}}$, where, by the difference method, we can obtain

$$\dot{\widehat{\mathit{y}}}=\frac{{\mathit{y}}_t-{\mathit{y}}_{t-1}}{\Delta t}$$

(50)

Thus, Equation (49) can be further simplified as

$${\dot{\mathit{e}}}_c={\widehat{\mathit{L}}}_h^T{\tilde{\mathit{W}}}^T\left(\mathit{\kappa }-\mathit{\lambda }-\mathit{\omega }\right)+{\tilde{\mathit{L}}}_h^T{\widehat{\mathit{W}}}^T\left(\mathit{\kappa }-\mathit{\lambda }-\mathit{\omega }\right)-K{e}_c$$

(51)

Consider that ${\widehat{\mathit{L}}}_h$ is the output of the LSTM neural network, which can be expressed as

$${\widehat{\mathit{L}}}_h={\widehat{\mathit{L}}}_h\left(\mathit{e},\widehat{\mathit{W}},{\widehat{\mathit{W}}}_{ih},{\widehat{\mathit{W}}}_{ix},{\widehat{\mathit{W}}}_{fh},{\widehat{\mathit{W}}}_{fx},{\widehat{\mathit{W}}}_{ch},{\widehat{\mathit{W}}}_{cx},{\widehat{\mathit{W}}}_{oh},{\widehat{\mathit{W}}}_{ox},{\widehat{\mathit{b}}}_i,{\widehat{\mathit{b}}}_f,{\widehat{\mathit{b}}}_c,{\widehat{\mathit{b}}}_o\right)$$

(52)

According to the Taylor expansion formula, we can obtain

$$\begin{array}{l}{\tilde{\mathit{L}}}_h={\left.\frac{\partial {\mathit{L}}_h}{\partial {\mathit{\theta }}_1}\right|}_{{\mathit{\theta }}_1={\widehat{\mathit{\theta }}}_1}{\tilde{\mathit{\theta }}}_1+{\left.\frac{\partial {\mathit{L}}_h}{\partial {\mathit{\theta }}_2}\right|}_{{\mathit{\theta }}_2={\widehat{\mathit{\theta }}}_2}{\tilde{\mathit{\theta }}}_2+{\left.\frac{\partial {\mathit{L}}_h}{\partial {\mathit{\theta }}_3}\right|}_{{\mathit{\theta }}_3={\widehat{\mathit{\theta }}}_3}{\tilde{\mathit{\theta }}}_3+{\left.\frac{\partial {\mathit{L}}_h}{\partial {\mathit{\theta }}_4}\right|}_{{\mathit{\theta }}_4={\widehat{\mathit{\theta }}}_4}{\tilde{\mathit{\theta }}}_4\\ +{\left.\frac{\partial {\mathit{L}}_h}{\partial {\mathit{\theta }}_5}\right|}_{{\mathit{\theta }}_5={\widehat{\mathit{\theta }}}_5}{\tilde{\mathit{\theta }}}_5+{\left.\frac{\partial {\mathit{L}}_h}{\partial {\mathit{\theta }}_6}\right|}_{{\mathit{\theta }}_6={\widehat{\mathit{\theta }}}_6}{\tilde{\mathit{\theta }}}_6+{\left.\frac{\partial {\mathit{L}}_h}{\partial {\mathit{\theta }}_7}\right|}_{{\mathit{\theta }}_7={\widehat{\mathit{\theta }}}_7}{\tilde{\mathit{\theta }}}_7+{\left.\frac{\partial {\mathit{L}}_h}{\partial {\mathit{\theta }}_8}\right|}_{{\mathit{\theta }}_8={\widehat{\mathit{\theta }}}_8}{\tilde{\mathit{\theta }}}_9\\ +{\left.\frac{\partial {\mathit{L}}_h}{\partial {\mathit{b}}_i}\right|}_{{\mathit{b}}_i={\widehat{\mathit{b}}}_i}{\tilde{\mathit{b}}}_i+{\left.\frac{\partial {\mathit{L}}_h}{\partial {\mathit{b}}_f}\right|}_{{\mathit{b}}_f={\widehat{\mathit{b}}}_f}{\tilde{\mathit{b}}}_f+{\left.\frac{\partial {\mathit{L}}_h}{\partial {\mathit{b}}_c}\right|}_{{\mathit{b}}_c={\widehat{\mathit{b}}}_c}{\tilde{\mathit{b}}}_c+{\left.\frac{\partial {\mathit{L}}_h}{\partial {\mathit{b}}_o}\right|}_{{\mathit{b}}_o={\widehat{\mathit{b}}}_o}{\tilde{\mathit{b}}}_o+{\mathit{O}}_h\end{array}$$

(53)

where $\tilde{\mathit{\theta }}={\mathit{\theta }}^{\ast }-\widehat{\mathit{\theta }}$, $\tilde{\mathit{b}}={\mathit{b}}^{\ast }-\widehat{\mathit{b}}$.

Proof of Theorem 1. 

To ensure that the tracking error of the Hamiltonian function can converge to 0, the Lyapunov function is defined as follows:

$$V_1=\frac{1}{2}{e}_c^2$$

(54)

$$\begin{array}{l}{V}_2=\frac{1}{2{\eta }_w}tr(\tilde{\mathit{W}}{\tilde{\mathit{W}}}^T)+\frac{1}{2{\eta }_{\theta }}{\tilde{\mathit{\theta }}}_1^T{\tilde{\mathit{\theta }}}_1+\frac{1}{2{\eta }_{\theta }}{\tilde{\mathit{\theta }}}_2^T{\tilde{\mathit{\theta }}}_2+\frac{1}{2{\eta }_{\theta }}{\tilde{\mathit{\theta }}}_3^T{\tilde{\mathit{\theta }}}_3+\frac{1}{2{\eta }_{\theta }}{\tilde{\mathit{\theta }}}_4^T{\tilde{\mathit{\theta }}}_4\\ +\frac{1}{2{\eta }_{\theta }}{\tilde{\mathit{\theta }}}_5^T{\tilde{\mathit{\theta }}}_5+\frac{1}{2{\eta }_{\theta }}{\tilde{\mathit{\theta }}}_6^T{\tilde{\mathit{\theta }}}_6+\frac{1}{2{\eta }_{\theta }}{\tilde{\mathit{\theta }}}_7^T{\tilde{\mathit{\theta }}}_7+\frac{1}{2{\eta }_{\theta }}{\tilde{\mathit{\theta }}}_8^T{\tilde{\mathit{\theta }}}_8\\ +\frac{1}{2{\eta }_b}{\tilde{\mathit{b}}}_i^T{\tilde{\mathit{b}}}_i+\frac{1}{2{\eta }_b}{\tilde{\mathit{b}}}_f^T{\tilde{\mathit{b}}}_f+\frac{1}{2{\eta }_b}{\tilde{\mathit{b}}}_c^T{\tilde{\mathit{b}}}_c+\frac{1}{2{\eta }_b}{\tilde{\mathit{b}}}_o^T{\tilde{\mathit{b}}}_o\end{array}$$

(55)

$$V_3={\mathit{e}}^T\mathit{Q}e$$

(56)

$$V=V_1+V_2+V_3$$

(57)

The time derivative of Equations (54) and (55) can be obtained as

$$\begin{array}{l}{\dot{V}}_1=e_c{\dot{e}}_c\\ =e_c\left({\widehat{\mathit{L}}}_h^T{\tilde{\mathit{W}}}^T\left(\mathit{\kappa }-\mathit{\lambda }-\mathit{\omega }\right)+{\tilde{\mathit{L}}}_h^T{\widehat{\mathit{W}}}^T\left(\mathit{\kappa }-\mathit{\lambda }-\mathit{\omega }\right)-K{e}_c\right)\end{array}$$

(58)

$$\begin{array}{l}{\dot{V}}_2=\frac{1}{{\eta }_w}tr(\tilde{\mathit{W}}{\dot{\tilde{\mathit{W}}}}^T)+\frac{1}{{\eta }_{\theta }}{\dot{\tilde{\mathit{\theta }}}}_1^T{\tilde{\mathit{\theta }}}_1+\frac{1}{{\eta }_{\theta }}{\dot{\tilde{\mathit{\theta }}}}_2^T{\tilde{\mathit{\theta }}}_2+\frac{1}{{\eta }_{\theta }}{\dot{\tilde{\mathit{\theta }}}}_3^T{\tilde{\mathit{\theta }}}_3+\frac{1}{{\eta }_{\theta }}{\dot{\tilde{\mathit{\theta }}}}_4^T{\tilde{\mathit{\theta }}}_4\\ +\frac{1}{{\eta }_{\theta }}{\dot{\tilde{\mathit{\theta }}}}_5^T{\tilde{\mathit{\theta }}}_5+\frac{1}{{\eta }_{\theta }}{\dot{\tilde{\mathit{\theta }}}}_6^T{\tilde{\mathit{\theta }}}_6+\frac{1}{{\eta }_{\theta }}{\dot{\tilde{\mathit{\theta }}}}_7^T{\tilde{\mathit{\theta }}}_7+\frac{1}{{\eta }_{\theta }}{\dot{\tilde{\mathit{\theta }}}}_8^T{\tilde{\mathit{\theta }}}_8\\ +\frac{1}{{\eta }_b}{\dot{\tilde{\mathit{b}}}}_i^T{\tilde{\mathit{b}}}_i+\frac{1}{{\eta }_b}{\dot{\tilde{\mathit{b}}}}_f^T{\tilde{\mathit{b}}}_f+\frac{1}{{\eta }_b}{\dot{\tilde{\mathit{b}}}}_c^T{\tilde{\mathit{b}}}_c+\frac{1}{{\eta }_b}{\dot{\tilde{\mathit{b}}}}_o^T{\tilde{\mathit{b}}}_o\end{array}$$

(59)

$$\begin{array}{l}{\dot{V}}_3={\mathit{e}}^T\overline{\mathit{Q}}\dot{\mathit{e}}\\ ={\mathit{e}}^T\overline{\mathit{Q}}\left(\mathit{f}\left(\mathit{X}\right)+\mathit{g}\left(\mathit{X}\right)\widehat{\mathit{u}}-{\dot{\mathit{X}}}_d\right)\end{array}$$

(60)

To ensure that $\dot{V}$ is negative, we set the $e_c{\widehat{\mathit{L}}}_h^T{\tilde{\mathit{W}}}^T\left(\mathit{\kappa }-\mathit{\lambda }-\mathit{\omega }\right)+\frac{1}{{\eta }_w}tr({\tilde{\mathit{W}}}^T\dot{\tilde{\mathit{W}}})=0$

According to matrix operation rules, we can obtain

$$\frac{1}{{\eta }_w}tr({\tilde{\mathit{W}}}^T\dot{\tilde{\mathit{W}}})=\frac{1}{{\eta }_w}({\tilde{\mathit{W}}}_{l1}^T{\dot{\tilde{\mathit{W}}}}_{l1}+{\tilde{\mathit{W}}}_{l2}^T{\dot{\tilde{\mathit{W}}}}_{l2}+\cdots +{\tilde{\mathit{W}}}_{ln}^T{\dot{\tilde{\mathit{W}}}}_{ln})$$

(61)

where ${\tilde{\mathit{W}}}_{li}$ is the i-th line of $\tilde{\mathit{W}}$.

Expand the terms on the right side of Equation (58) to obtain

$$e_c{\widehat{\mathit{L}}}_h^T{\tilde{\mathit{W}}}^T\left(\mathit{\kappa }-\mathit{\lambda }-\mathit{\omega }\right)=e_c\left(\begin{array}{l}{\widehat{\mathit{L}}}_h^T{\tilde{\mathit{W}}}_{l1}^T\left({\kappa }_1-{\lambda }_1-{\omega }_1\right)+\\ {\widehat{\mathit{L}}}_h^T{\tilde{\mathit{W}}}_{l2}^T\left({\kappa }_2-{\lambda }_2-{\omega }_2\right)+\cdots \\ +{\widehat{\mathit{L}}}_h^T{\tilde{\mathit{W}}}_{ln}^T\left({\kappa }_n-{\lambda }_n-{\omega }_n\right)\end{array}\right)$$

(62)

where ${\kappa }_i$, ${\lambda }_i$ and ${\omega }_i$ are i-th elements of $\mathit{\kappa }$, $\mathit{\lambda }$ and $\mathit{\omega }$ respectively.

By eliminating the corresponding term, the update law of the weight $\mathit{W}$’s i-th line is obtained as

$${\dot{\tilde{\mathit{W}}}}_{li}=-{\dot{\widehat{\mathit{W}}}}_{li}=-{\eta }_w{e}_c\left({\kappa }_i-{\lambda }_i-{\omega }_i\right){\widehat{\mathit{L}}}_h^T$$

(63)

Similarly, in order to obtain the update law of other weights, set

$$\begin{array}{l}{e}_c{\tilde{\mathit{h}}}^T{\widehat{\mathit{W}}}^T\left(\mathit{\kappa }-\mathit{\lambda }\right)+\frac{1}{{\eta }_{\theta }}{\dot{\tilde{\mathit{\theta }}}}_1^T{\tilde{\mathit{\theta }}}_1+\frac{1}{{\eta }_{\theta }}{\dot{\tilde{\mathit{\theta }}}}_2^T{\tilde{\mathit{\theta }}}_2+\frac{1}{{\eta }_{\theta }}{\dot{\tilde{\mathit{\theta }}}}_3^T{\tilde{\mathit{\theta }}}_3+\frac{1}{{\eta }_{\theta }}{\dot{\tilde{\mathit{\theta }}}}_4^T{\tilde{\mathit{\theta }}}_4+\frac{1}{{\eta }_{\theta }}{\dot{\tilde{\mathit{\theta }}}}_5^T{\tilde{\mathit{\theta }}}_5+\frac{1}{{\eta }_{\theta }}{\dot{\tilde{\mathit{\theta }}}}_6^T{\tilde{\mathit{\theta }}}_6\\ +\frac{1}{{\eta }_{\theta }}{\dot{\tilde{\mathit{\theta }}}}_7^T{\tilde{\mathit{\theta }}}_7+\frac{1}{{\eta }_{\theta }}{\dot{\tilde{\mathit{\theta }}}}_8^T{\tilde{\mathit{\theta }}}_8+\frac{1}{{\eta }_b}{\dot{\tilde{\mathit{b}}}}_i^T{\tilde{\mathit{b}}}_i+\frac{1}{{\eta }_b}{\dot{\tilde{\mathit{b}}}}_f^T{\tilde{\mathit{b}}}_f+\frac{1}{{\eta }_b}{\dot{\tilde{\mathit{b}}}}_c^T{\tilde{\mathit{b}}}_c+\frac{1}{{\eta }_b}{\dot{\tilde{\mathit{b}}}}_o^T{\tilde{\mathit{b}}}_o=0\end{array}$$

(64)

We can obtain

$${\dot{\tilde{\mathit{\theta }}}}_i=-{\dot{\widehat{\mathit{\theta }}}}_i=-{\eta }_{\theta }{e}_c{\left(\frac{\partial {\mathit{L}}_h}{\partial {\mathit{\theta }}_i}\right)}^T{\widehat{\mathit{W}}}^T\left(\mathit{\kappa }-\mathit{\lambda }-\mathit{\omega }\right) i=1,2,\dots ,8$$

(65)

$${\dot{\tilde{\mathit{b}}}}_i=-{\dot{\widehat{\mathit{b}}}}_i=-{\eta }_b{e}_c{\left(\frac{\partial {\mathit{L}}_h}{\partial {\mathit{b}}_i}\right)}^T{\widehat{\mathit{W}}}^T\left(\mathit{\kappa }-\mathit{\lambda }-\mathit{\omega }\right)$$

(66)

$${\dot{\tilde{\mathit{b}}}}_f=-{\dot{\widehat{\mathit{b}}}}_f=-{\eta }_b{e}_c{\left(\frac{\partial {\mathit{L}}_h}{\partial {\mathit{b}}_f}\right)}^T{\widehat{\mathit{W}}}^T\left(\mathit{\kappa }-\mathit{\lambda }-\mathit{\omega }\right)$$

(67)

$${\dot{\tilde{\mathit{b}}}}_c=-{\dot{\widehat{\mathit{b}}}}_c=-{\eta }_b{e}_c{\left(\frac{\partial {\mathit{L}}_h}{\partial {\mathit{b}}_c}\right)}^T{\widehat{\mathit{W}}}^T\left(\mathit{\kappa }-\mathit{\lambda }-\mathit{\omega }\right)$$

(68)

$${\dot{\tilde{\mathit{b}}}}_o=-{\dot{\widehat{\mathit{b}}}}_o=-{\eta }_b{e}_c{\left(\frac{\partial {\mathit{L}}_h}{\partial {\mathit{b}}_o}\right)}^T{\widehat{\mathit{W}}}^T\left(\mathit{\kappa }-\mathit{\lambda }-\mathit{\omega }\right)$$

(69)

□

Remark 4. 

By substituting the weight update law (63), (65)~(69), and the control law (45) into Equation (57), it can be obtained that $\dot{V}=-K{e}_c^2$. $K$ is a positive constant, so $\dot{V}\le 0$; obviously, $V$ is positive, and according to the Lyapunov stability theory, closed-loop systems are bounded and $\underset{t\to \infty }{\lim }{e}_c\to 0$. According to the optimal control theory, when $e_c\to 0$, $\widehat{H}\to H^{\ast }$ and $\widehat{J}\to J^{\ast }$; at this point, the cost function is optimal. According to the definition of the cost function in Section 3, the system tracking error and control input are both considered; that is to say, the optimal performance cost function can ensure system tracking error $\underset{t\to \infty }{\lim }\mathit{e}\to$0, and the closed-loop control system is bounded and can converge to the small neighborhood of the control command.

Remark 5. 

In this article, ADP-LSTM is described as a model-based reinforcement learning algorithm, rather than a data-based form. During the process of updating the weights of the LSTM neural networks, the algorithm does not rely on training data and loss functions, such as common stochastic gradient descent algorithms. Instead, it is based on the Lyapunov method, which directly provides an analytical solution for the network weight update law (63) and (65)~(69). In other words, ADP-LSTM does not require training data and loss functions. This direct update law eliminates a large number of iterative operations in the network training process, significantly reducing the system’s computational burden and single-step computation time.

5. Simulation Analysis

In the simulation process, the 6-DoF attitude dynamics model of the dual-control aircraft mentioned in the Section 2 is applied to verify the performance of the control law which is presented in this paper. The command signal is ${\mathit{X}}_d={\left[\begin{array}{cccccccc}{\gamma }_d& n_{zd}& n_{yd}& {\delta }_{xd}& {\delta }_{yd}& {\delta }_{zd}& F_{zd}& F_{yd}\end{array}\right]}^T$, and all aerodynamic parameters are designed based on an aircraft flying altitude of 30 km.

When the aircraft flies at an altitude of 30 km, the thin atmospheric density results in a decrease in aerodynamic force. At this time, relying on pure aerodynamic control will seriously reduce the dynamic characteristics and control quality of the control system. Usually, the dual-control strategy is used to design the aircraft autopilot, because the direct force is generated by the reaction jet, which is not affected by the flight altitude, and can effectively compensate for the lack of control input caused by insufficient aerodynamic force.

Due to the volume limitation of the aircraft, it is impossible to place the orbit control engine with a large volume and weight. Therefore, the attitude control engine is used in this simulation, which only affects the attitude and has a very small direct impact on the overload. Therefore, the overload establishment of the aircraft still depends on the aerodynamic force; that is to say, in order to obtain enough overload, the aircraft will make a large angle of attack or sideslip angle maneuver. At this time, the assumption that the aerodynamic parameters are considered as constant or slow time variables is no longer tenable. Therefore, taking the aerodynamic parameters $a_2$ and $b_2$ as examples, we consider $a_2$ and $b_2$ as functions of the AOA and sideslip angle, i.e.,

$$a_2=a_{20}+k_{a_2}\alpha$$

(70)

$$b_2=b_{20}+k_{b_2}\beta$$

(71)

We consider other aerodynamic parameters as perturbation parameters, i.e.,

$$\begin{array}{l}{a}_1=a_{10}+k_1\ast \sin ({\omega }_{1}t)\\ a_3=a_{30}+k_2\ast \sin ({\omega }_{2}t)\\ a_4=a_{40}+k_3\ast \sin ({\omega }_{3}t)\\ a_5=a_{50}+k_4\ast \sin ({\omega }_{4}t)\end{array}$$

(72)

$$\begin{array}{l}{b}_1=b_{10}+k_1\ast \sin ({\omega }_{1}t)\\ b_3=b_{30}+k_2\ast \sin ({\omega }_{2}t)\\ b_4=b_{40}+k_3\ast \sin ({\omega }_{3}t)\\ b_5=b_{50}+k_4\ast \sin ({\omega }_{4}t)\end{array}$$

(73)

As the roll angle and roll rate are both small, we consider $c_1$ and $c_3$ as constants. We consider that the external disturbance vector $\mathit{d}$ is Gaussian white noise.

The initial weights of the LSTM neural network are randomly selected in the closed interval $\left[-0.2,0.2\right]$. According to the practical application, the tail fin angle and the magnitude of the direct force are subject to saturation constraints, i.e., $\left|{\delta }_{xc}\right|\le 30 \mathrm{deg}$, $\left|{\delta }_{yc}\right|\le 30 \mathrm{deg}$, $\left|{\delta }_{zc}\right|\le 30 \mathrm{deg}$, $\left|F_z\right|\le 3000 \mathrm{N}$, and $\left|F_y\right|\le 3000 \mathrm{N}$. The initial state of the system is $\mathit{X}={\left[\begin{array}{cccccccc}3/57.3& 0& 0& 0& 0& 0& 0& 0\end{array}\right]}^T$.

To verify the optimality of ADP-LSTM, the adaptive sliding mode control method (SMC-RNN) proposed in reference [31] was applied to the attitude control model during the simulation process and compared with ADP-LSTM. The algorithm in reference [31] combines sliding mode control and Recurrent Neural Networks (RNN), using RNNs to fit system terms and external disturbances to achieve adaptive control of the system model and external disturbances. However, it is important to note that the algorithm in reference [31] did not consider energy optimization during its design process. Therefore, comparing it with ADP-LSTM can effectively reflect the energy-optimal control effect of ADP-LSTM.

The LSTM neural networks used in ADP-LSTM have eight input nodes, eight output nodes, and 10 cell states, making the system state $\mathit{X}$ the input vector of the network. The RNNs also have eight input nodes, eight output nodes, and 10 hidden states; the structure of RNNs can be found in the article [31].

To verify the control effectiveness of ADP-LSTM, two simulation scenarios are designed: tracking a fixed overload command and tracking a time-varying overload command. Both scenarios represent common target maneuver forms and can demonstrate the general applicability of the algorithm presented in this paper.

The aircraft parameters of the pitch channel can be seen in

Table 1. Aircraft parameters.

$\mathit{V}$	$\mathit{m}$	${\mathit{a}}_{10}$	${\mathit{a}}_{20}$	${\mathit{a}}_{30}$	${\mathit{a}}_{40}$	${\mathit{a}}_{50}$	${\mathit{c}}_1$	${\mathit{c}}_3$	$\mathit{l}$
1200 m/s	500 kg	0.048	1.500	12.101	0.166	0.004	0.024	0.604	1 m

. Considering that the aircraft has an axisymmetric shape, the pitch channel parameters are consistent with the yaw channel parameters.

5.1. Scenario 1: Tracking a Fixed Overload Command

The curve of the angle of attack, sideslip angle, and roll angle are shown in Figure 4. According to the previous introduction, the overload of the aircraft is established by the aerodynamic force, so in order to track the overload command as soon as possible, the angle of attack and sideslip angle of the aircraft need to respond quickly. In Figure 4a,b, we can see that the angle of attack and sideslip angle both enter the steady state quickly. Like other STT aircraft, the controller designed in this paper ensures that the aircraft body axis does not roll; that is, the roll angle command is 0 deg. To verify the effect of the controller, the initial roll angle is set to 3 degrees. As can be seen from Figure 4c, the controller can ensure that the roll angle converges to 0 deg. Both ADP-LSTM and SMC-RNN can achieve control of roll angle, among which ADP-LSTM has a faster convergence speed but some overshoot, while SCM-RNN, although it has no overshoot, has a slower convergence speed.

The curve of roll rate, yaw rate, and pitch rate are shown in Figure 5. This state intuitively reflects the attitude agility of the aircraft. It can be seen from Figure 5 that the aircraft has strong agility and fast attitude response speed under the effect of the dual-control strategy.

The overload curves of the two controllers are shown in Figure 6. It can be seen from Figure 6 that the aircraft overload can track the command signal by ADP-LSTM, but it has to be admitted that the convergence rate is slow for two reasons. First, according to the nonlinear system optimal control theory, when the Hamilton function tends to zero, the optimal control input ${\mathit{u}}^{\ast }$ obtained at this time can only ensure that the tracking error $\mathit{e}$ converges to 0 in infinite time, i.e., $\underset{t\to \infty }{\lim }\mathit{e}\left(t\right)\to$0. Second, there are coupling terms between the pitch, yaw, and roll channels of the aircraft, which reduce the control quality. Usually, when designing the autopilot, it is completed after decoupling the three channels. However, this will ignore some characteristics of the system, and obviously, this will reduce the robustness of the algorithm in practical applications. The advantage of the ADP-LSTM in this paper is that it does not need three-channel decoupling, retains the characteristics of the system, and does not need the necessary assumptions when decoupling, which widens the application scope of the algorithm and is more general. Meanwhile, we can observe that the control effect of SMC-RNN is better than that of ADP-LSTM. This is an unavoidable trade-off. In order to achieve optimal energy consumption for the system, some sacrifice in control effectiveness is inevitable. However, the control effectiveness of ADP-LSTM has not significantly decreased and still maintains steady-state error, with only a slight increase in convergence time.

The control inputs of pitch, yaw, and roll channels of ADP-LSTM and SMC-RNN are shown in Figure 7. And low-pass filters were introduced to better display the specific details of the curve. It can be seen from Figure 7a,c,e that when the system enters the steady state, the control input of the tail fins has a chattering phenomenon, which is caused by external disturbance, and the control input shows a sinusoidal trend, which is caused by the perturbation of aerodynamic parameters. The direct force input will tend to a fixed value, and this value is very small. Intuitively, this avoids the waste of control energy. However, it is still impossible to determine whether the control input is optimal from the results of this figure alone. It is necessary to refer to whether the Hamiltonian function $\widehat{H}$ converges to zero. In Figure 7b,d,f, we can see that the control input of SMC-RNN is higher than that of ADP-LSTM, especially for direct force control input. The chattering phenomenon in the control input is more severe, and it does not significantly weaken after passing through the low-pass filter. This is due to the inherent defect of sliding mode control. Controlling the input chattering phenomenon can cause serious energy waste, and it is also very unfriendly to the actuator.

The outputs of the LSTM neural network are shown in Figure 8. In this paper, the LSTM neural network is aimed to fit the partial derivative of the cost function $J\left(\mathit{X}\right)$ with respect to the state of the system $\mathit{X}$, i.e., $\Delta \mathit{J}$. According to the definition of the system, we know that $\Delta \mathit{J}\in {ℝ}^8$, and there are eight output values of the LSTM neural network, i.e., $Y_1~Y_8$. It can be seen from Figure 7 that after a short dynamic process, the output value of the neural network is nearly stable, which shows that under the effect of the adaptive weight update law, the output value $\widehat{\mathit{y}}$ gradually tends to the optimal value ${\mathit{y}}^{\ast }$.

The training process is shown in Figure 9. It can be seen from the figure that under the effect of the adaptive weight update law, most neural network weights converge in 1 s, which shows that the training efficiency is very high. Because the updated law of network weights is derived from the Lyapunov function, the training trend of network weights is very clear, which must have obvious advantages over the stochastic gradient descent (SDG) method, and there are no problems such as local optimization in the training process.

The curve of the Hamiltonian function is shown in Figure 10. According to the nonlinear system optimal theory mentioned in Section 3, the necessary condition for the optimal control input is that the Hamiltonian function tends to zero, i.e., $e_c\to 0$ and then $\widehat{H}\to H^{\ast }\to 0$ It can be seen from the figure that under the action of the LSTM neural network, the Hamiltonian function converges to 0 quickly, indicating that $\widehat{J}\to J^{\ast }$ and $\Delta \widehat{\mathit{J}}\to \Delta {\mathit{J}}^{\ast }$, and at this time, $\widehat{\mathit{u}}\to {\mathit{u}}^{\ast }$.

The energy consumption of the two control algorithms is shown in Figure 11. To quantify energy consumption, the energy consumption indicator is defined as $Q_e={\displaystyle {\int }_0^t{\mathit{u}}^2\left(t\right)}dt$. Figure 7a illustrates the energy consumption of the tail fins, while Figure 7b shows the energy consumption of the direct force. It is important to note that the values after low-pass filtering were used when calculating energy consumption. From Figure 7, it is evident that the energy consumption of both the tail fins and direct force using ADP-LSTM is superior to that of SMC-RNN. Particularly in the case of direct force energy consumption, ADP-LSTM demonstrates clear advantages, effectively avoiding energy waste. While ADP-LSTM may be slightly inferior to SMC-RNN in terms of control effectiveness, it holds significant advantages in energy consumption. As previously introduced, the energy of direct force is limited, and the aircraft will encounter multiple attitude adjustments and overload command tracking in a complete working environment. Without limiting energy consumption, early depletion of aircraft fuel can occur, leading to a loss of partial tracking ability. Therefore, this article focuses on studying the optimal control of aircraft attitude.

The average single-step time of the two algorithms is ADP-LSTM 1.105 ms, and SMC-RNN 0.751 ms (simulation environment: Intel 12^th i7-12700).

5.2. Scenario 2: Tracking a Time-Varying Overload Command

The simulation results for Scenario 2 are shown in Figure 12, Figure 13, Figure 14, Figure 15, Figure 16, Figure 17, Figure 18 and Figure 19. Similar to Scenario 1, both ADP-LSTM and SMC-RNN can track time-varying overload commands. However, the control effectiveness of ADP-LSTM is slightly weaker than that of SMC-RNN. This can be attributed to two reasons: 1. The convergence speed of the LSTM neural network is slightly slower than that of a traditional RNN, especially under time-varying commands, which becomes more apparent. 2. To achieve energy-optimal control, it is necessary to sacrifice some control effectiveness, especially in terms of command tracking speed. From Figure 14, it can be observed that although ADP-LSTM has a slightly slower convergence speed than SMC-RNN, there is no significant difference in their tracking accuracy, which is consistent with the performance in Scenario 1.

Figure 15 illustrates the control inputs of the two control algorithms. It is evident from the figure that SMC-RNN’s control input exhibits significant oscillations, similar to Scenario 1. This is unfriendly to the control execution mechanism and results in a significant waste of energy. In contrast, ADP-LSTM does not exhibit such oscillations. Furthermore, from Figure 19, it can be seen that the energy consumption of ADP-LSTM is significantly lower than that of SMC-RNN (after the filtering of SMC-RNN’s control input). This demonstrates the significant advantage of ADP-LSTM in energy-optimal control.

Through the above two simulation scenarios, it is evident that ADP-LSTM can handle common aircraft overload commands and has a certain degree of generality.

6. Conclusions

This article presents a reinforcement learning near-optimal control algorithm based on an LSTM neural network, which can be applied to solve the 6-DoF attitude control problem of dual-control aircraft. For the first time, the reinforcement learning-based near-optimal control algorithm is applied to the complex MIMO system.

Compared with the existing reinforcement learning algorithm, this algorithm has an obvious advantage:

(1)

It can deal with high-dimensional MIMO systems, rather than ideal simple systems such as inverted pendulum, slider car, spring damper, and other SIMO or SISO systems, which benefit from the strong fitting ability of LSTM neural network;

(2)

The ADP-LSTM does not need to decouple the nonlinear aircraft 6-DoF attitude dynamics model, and it retains the internal characteristics of the system as much as possible and the assumptions of necessity when system decoupling is no longer needed, making the algorithm more universal.

(3)

Based on the Lyapunov method, the novel adaptive online update law of LSTM neural network weights is given. Compared with the stochastic gradient descent method, this method has higher training efficiency and can ensure that the closed-loop system is uniformly asymptotically stable.

(4)

During the simulation process, we designed two kinds of scenarios to prove the commonality of ADP-LSTM, which was compared with SMC-RNN in terms of control effectiveness and energy consumption. The comparison results revealed that ADP-LSTM had a significant advantage in energy consumption, albeit at the expense of sacrificing some control effectiveness. This reflects the overall advantage of ADP-LSTM over algorithms that do not consider energy optimization.

As for the future research direction, I think the main direction in the near future is how to solve the optimal control problem when the control input is saturated, and a longer-term goal is how to realize the finite-time convergence of the system.