Adaptive Fuzzy Fault-Tolerant Attitude Control for a Hypersonic Gliding Vehicle: A Policy-Iteration Approach

Abstract

In this paper, adaptive fuzzy fault-tolerant control (AFFTC) for the attitude control system of a hypersonic gliding vehicle (HGV) experiencing an actuator fault is proposed. Actuator faults of the HGV are considered with respect to its actual structure and actuator characteristics. The HGV’s attitude system is firstly represented by a T–S fuzzy model, and then a normal T–S fuzzy controller is designed. A reinforcement learning (RL)-based policy iterative solution algorithm is proposed for the solving of the T-S fuzzy controller. Then, based on the normal T–S controller, a fuzzy FTC controller is proposed in which the control matrices can improve themselves according to the special fault. An integral reinforcement learning (IRL)-based solving algorithm is proposed to reduce the dependence of the design methods on the HGV model. Simulations on three different kinds of actuator faults show that the designed IRL-based FTC can ensure a reliable flight by the HGV.

1. Introduction

A hypersonic gliding vehicle (HGV) is a vehicle with high flight speed and a large flight envelope, so its flight dynamics are really complex. As a basic element of aircraft control, the modeling and realization of HGV attitude control have received extensive attention in recent years. Various control methods are used to solve this problem, such as trajectory-linearization-based active disturbance rejection control [1], fuzzy logic-based adaptive control [2], reinforcement-learning-based nonlinear control [3], and so on. Linear parameter varying-based attitude control for the attitude system of an HGV is proposed in [4], and the sources of uncertainties in HGV control are analyzed, which are uncertainty of aerodynamic parameters and external disturbance. Robust nonlinear controllers for the attitude system of an HGV suffering from model disturbance have also received widespread attention.

In practice, faults may exist in the actuator of the HGV, and such failures necessarily reduce the control accuracy of the HGV, so FTC is an important research area in the attitude control of HGVs. FTC of HGVs has been widely studied in recent years. In ref. [5], actuator faults of an HGV’s attitude control system are considered, and adaptive FTC is proposed. In ref. [6], actuator faults of an HGV’s attitude control system are given in a more general model, and then FTC that can ensure the robustness of the system under actuator fault is given. In ref. [7], fixed-time-quantized fault-tolerant attitude control is presented for an HGV. In the above literature, the model of the HGV’s attitude system is assumed to be known, while in practice the nonlinear dynamics of the HGV may be unknown. In ref. [8], adaptive FTC is proposed for an HGV’s attitude system with an unknown inertial matrix and state constraints. In ref. [9], time-varying fault control of an HGV together with an adaptive FTC are proposed. Actuator faults and model uncertainties are considered synchronously and then sliding-mode-control-based FTC is proposed in ref. [10].

In the above literature, the specific form and occurrence time of the faults are all supposed to be already known, and then FTC is designed according to these known faults. This is unreasonable in practice, as the specific form and occurrence time of faults are difficult to obtain. In ref. [11], online computing-method-based FTC is proposed for the attitude control of HGV, but this method can deal with, only fixed-form faults. In ref. [12], iterative learning fault-tolerant control is proposed for time-varying industrial processes with actuator faults, and in ref. [13], stochastic actuator failures are considered for Markovian jump time-delayed systems. The FTC of the constrained system is solved by model predictive control in ref. [14]. The proposed controller design strategy can deal with some given faults but cannot adjust itself according to a specific fault online, so this is still an open problem. It should be noted that in order to facilitate practical use, the form of a given adaptive FTC should be as simple as possible.

The Takagi–Sugeno (T–S) fuzzy-based control design strategy can approach a nonlinear system with arbitrary accuracy, and the resulting T–S model is a combination of linear systems, so classical linear control methods can still be utilized by a parallel-distributed compensation (PDC) scheme [15]. T–S fuzzy technology is viewed as an efficient way for analyzing and designing the control of nonlinear systems and has been widely applied, such as type-2 T–S fuzzy-based tracking control of a saturation system [16] and T–S fuzzy control for semi-Markov jumps [17]. T–S fuzzy FTC of HGVs has also been utilized in [18]. In ref. [18], the upper and lower bounds of the faults are already known. How to design FTC for HGVs with faults of unknown specific form and unknown occurrence time is still an open problem.

Furthermore, in order to improve the self-learning ability of the FTC, an intelligent algorithm is needed. Commonly used intelligent algorithms include deep learning [19] and reinforcement learning (RL) [20]. The learning process of RL is similar to that of human beings. By interacting with the environment and updating the reward value in time, RL can obtain an optimal control law. RL does not need a specific model of the nonlinear system and can handle high-dimensional task scenarios, so it has been widely utilized by all trades and professions [21].

As a branch of RL, integral reinforcement learning (IRL) obtains its reinforcement signal by integrating the value function, and thereby can be utilized by unknown systems. Using only data from a completely unknown system, IRL can complete the assigned studying task, so it has been successfully applied to the optimal control of discrete-time multiagent systems [22], motion planning of autonomous vehicles [23], nuclear systems [24], and linear systems with input delay [25]. IRL-based fault-tolerant adaptive tracking control of Euler–Lagrange systems are also presented to improve the tracking performance of fault-tolerant control [26]. In a word, IRL has proved to be an effective means of solving FTC design of complex nonlinear systems [27].

Based on the above understanding, in this paper, an RL-based policy-iteration (PI) algorithm is utilized for the design of FTC of an HGV’s attitude system. A nonlinear model of the HGV’s attitude system is firstly, represented by a T–S fuzzy model, and then, the actuator fault model is built. A PI-based normal T–S fuzzy controller solving method is proposed without considering the actuator fault model. Based on the normal fuzzy controller, IRL-based fuzzy adaptive FTC is proposed. The control gains of the adaptive fuzzy FTC controller can self-adjust according to the special model of actuator fault. Finally, three simulation results are given to prove the effectiveness of the presented controller under different faults.

To sum up, the paper’s contributions are:

(1) A policy-iteration (PI) algorithm is utilized for the optimal controller design of a T–S fuzzy system.

(2) IRL-based adaptive fuzzy FTC is built for a T–S fuzzy system with actuator faults. With this FTC controller, the controller can be utilized online.

(3) This method is successfully applied to the attitude-tracking control of an HGV’s attitude system.

2. Problem Description

In this section, the nonlinear attitude model of an HGV is discussed, and then the nonlinear mode is represented by a T–S fuzzy system. Based on the attitude tracking control objective and the actuator fault model of the HGV, the control objective of the proposed T–S fuzzy model is discussed.

2.1. Nonlinear Model of an HGV’s Attitude System

The attitude model of an HGV used in this paper is

$$\left\{\begin{array}{c}\dot{\alpha }={\omega }_z-{\omega }_{x}cos\alpha tan\beta +{\omega }_{y}sin\alpha tan\beta \hfill \\ -sec\beta \left(\left(Y\frac{cos{\gamma }_v}{mV}-D\frac{sin{\gamma }_v}{mV}-\frac{mgcos\theta }{mV}\right)cos{\gamma }_v\right.\hfill \\ \left.+\left(Y\frac{sin{\gamma }_v}{mVcos\theta }-D\frac{cos{\gamma }_v}{mVcos\theta }\right)cos\theta sin{\gamma }_v\right)\hfill \\ \dot{\beta }={\omega }_{x}sin\alpha +{\omega }_{y}cos\alpha \hfill \\ -\left(Y\frac{cos{\gamma }_v}{mV}-D\frac{sin{\gamma }_v}{mV}-\frac{mgcos\theta }{mV}\right)sin{\gamma }_v\hfill \\ +\left(Y\frac{sin{\gamma }_v}{mVcos\theta }-D\frac{cos{\gamma }_v}{mVcos\theta }\right)cos\theta cos{\gamma }_v\hfill \\ \dot{{\gamma }_v}=sec\beta ({\omega }_{x}cos\alpha -{\omega }_{y}sin\alpha )\hfill \\ +\left(Y\frac{cos{\gamma }_v}{mV}-D\frac{sin{\gamma }_v}{mV}-\frac{mgcos\theta }{mV}\right)tan\beta cos{\gamma }_v\hfill \\ +\left(Y\frac{sin{\gamma }_v}{mVcos\theta }-D\frac{cos{\gamma }_v}{mVcos\theta }\right)\hfill \\ •(sin\theta +tan\beta cos\theta sin{\gamma }_v)\hfill \\ J_x\frac{d{\omega }_x}{dt}+(J_x-J_y){\omega }_z{\omega }_y=M_x\hfill \\ J_y\frac{d{\omega }_y}{dt}+(J_x-J_z){\omega }_x{\omega }_z=M_y\hfill \\ J_z\frac{d{\omega }_z}{dt}+(J_y-J_x){\omega }_y{\omega }_x=M_z\hfill \end{array}\right.$$

(1)

where V is the velocity of the HGV, $\theta$ is the flight path angle and ${\psi }_v$ is the heading angle, $L,D,$ and Y are the lift force, drag force, and side force, respectively. $\alpha$ is the attack angle, $\beta$ is the sideslip angle, and ${\gamma }_v$ is the velocity inclination angle, ${\omega }_x$, ${\omega }_y$, and ${\omega }_z$ are angular velocity of three axes, respectively. $M_x$, $M_y$, $M_z$ are the corresponding aerodynamic torques, and they are functions of $\alpha$, $\beta$, and ${\gamma }_v$, and ${\delta }_e$, ${\delta }_r$, and ${\delta }_a,$ where ${\delta }_e$, ${\delta }_r$, and ${\delta }_a$ are the elevator angle, yaw angle, and aileron angle, respectively. More details of the utilized mode can be found in ref. [5].

For the convenience of description, the nonlinear mode (1) can be rewritten as

$$\dot{x}\left(t\right)=F(x\left(t\right),u\left(t\right))$$

where

$$\begin{array}{ccc}\hfill x\left(t\right)& =& {\left[\begin{array}{cccccc}\alpha ,& \beta ,& {\gamma }_v,& {\omega }_x,& {\omega }_y,& {\omega }_z\end{array}\right]}^T,\hfill \\ \hfill u\left(t\right)& =& {\left[\begin{array}{ccc}{\delta }_e,& {\delta }_r,& {\delta }_a\end{array}\right]}^T\hfill \end{array}$$

and the expression of $F(x\left(t\right),u\left(t\right))$ can be found in (1).

2.2. T–S Fuzzy Modeling of an HGV

Since the parameter variation range of an HGV is big, a simple linear model derived by linearization of the equilibrium points is unsuitable for controller design. Considering the application convenience of the control system, T–S fuzzy modeling technology is utilized here.

For the application of T–S fuzzy modeling to an HGV’s attitude system, the premise variable is chosen as $\alpha$. Then three levels for $\alpha$ are chosen, the specific values of which are listed in

Table 1. Fuzzy Rules.

Rule NO.	Premise Variables $\mathit{\alpha }$
1	S
2	M
3	B

. Then, the HGV attitude system T–S model is built. The details are as follows:

(Rule 1) If $\alpha$ is ${\alpha }_S$, then

$$\dot{x}\left(t\right)=A_{1}x\left(t\right)+B_{1}u\left(t\right),$$

(Rule 2) If $\alpha$ is ${\alpha }_M$, then

$$\dot{x}\left(t\right)=A_{2}x\left(t\right)+B_{2}u\left(t\right),$$

(Rule 3) If $\alpha$ is ${\alpha }_B$, then

$$\dot{x}\left(t\right)=A_{3}x\left(t\right)+B_{3}u\left(t\right),$$

$A_i$ and $B_i$$(i=S,M,B)$ are system matrices and

$$\begin{array}{ccc}\hfill A_i& =& {\frac{\partial F}{\partial x}|}_{x=x_i,u=u_i},\hfill \\ \hfill B_i& =& {\frac{\partial F}{\partial u}|}_{x=x_i,u=u_i},\hfill \end{array}$$

where $x_i=({\alpha }_{\mathit{i}},{\beta }_{\mathit{i}},{\gamma }_{vi}),$$i=(S,M,B)$, $u_i$ is the equilibrium point associated with $x_i$.

Based on the T–S fuzzy model, the overall T–S fuzzy model of the HGV’s attitude model can be represented by:

$$\begin{array}{ccc}\hfill \dot{x}\left(t\right)& =& \stackrel{3}{\sum _{i=1}}{\mu }_i\left(t\right)\left(A_{i}x\left(t\right)+B_{i}u\left(t\right)\right)\hfill \\ & =& \overline{A}x\left(t\right)+\overline{B}u\left(t\right)\hfill \end{array}$$

(2)

where

$$\overline{A}=\stackrel{3}{\sum _{i=1}}{\mu }_i\left(t\right)A_i,\overline{B}=\stackrel{3}{\sum _{i=1}}{\mu }_i\left(t\right)B_i$$

and ${\mu }_i\left(t\right)$ is the membership function, ${\mu }_i\left(t\right)\ge 0,i=1,2,3$ and ${\displaystyle \sum _{i=1}^3}{\mu }_i\left(t\right)=1$.

2.3. Actuator Fault Model

In considering the actuator faults of an HGV, the following actuator fault is utilized:

$$u^F\left(t\right)={\mathrm{\Gamma }}_{F}u\left(t\right)-{\mathrm{\Lambda }}_F,$$

(3)

where ${\mathrm{\Gamma }}_F$ is the failure matrix, which represents the failure coefficient and is unknown, and ${\mathrm{\Lambda }}_F$ is the loss matrix, which denotes a bias fault, and is also unknown. $u\left(t\right)$ is the normal control signal, and

$${\mathrm{\Gamma }}_F=\left[\begin{array}{ccc}{\rho }_{{\delta }_e}& 0& 0\\ 0& {\rho }_{{\delta }_r}& 0\\ 0& 0& {\rho }_{{\delta }_a}\end{array}\right],{\mathrm{\Lambda }}_F=\left[\begin{array}{c}{\chi }_{{\delta }_e}\\ {\chi }_{{\delta }_r}\\ {\chi }_{{\delta }_a}\end{array}\right]$$

${\mathrm{\Gamma }}_F$ and ${\mathrm{\Lambda }}_F$ have different values, which means that different fault cases occur. ${\rho }_j$$(j={\delta }_e,{\delta }_r,{\delta }_a)$ represents the actual available proportion of the normal actuator, ${\chi }_j$$(j={\delta }_e,{\delta }_r,{\delta }_a)$ represents the actuator float variable caused by friction, piezoelectric effects, and so on. ${\chi }_j$$(j={\delta }_e,{\delta }_r,{\delta }_a)$ is a slowly changed value or is invariant; so it can be regarded as a constant in the controller design.

Expression (3) represents almost all fault models, more specifically:

(1) If ${\mathrm{\Gamma }}_F=I$, ${\mathrm{\Lambda }}_F=0$, the actuator fault is in free fault mode;

(2) If $0<{\mathrm{\Gamma }}_F<I,$ ${\mathrm{\Lambda }}_F=0,$ the actuator fault is loss of effectiveness;

(3) If ${\mathrm{\Gamma }}_F=I,$ ${\mathrm{\Lambda }}_F\ne 0,$ the actuator fault is a drift fault;

(4) If $0<{\mathrm{\Gamma }}_F<I,$ ${\mathrm{\Lambda }}_F\ne 0,$ the actuator fault is a combined loss of effectiveness and drift fault fault.

Remark 1.

The faults described by (3) are a general form of HGV actuator faults. In the previous literature, one or more faults were considered, but there was no unified description of HGV actuator faults. This paper gives a general description as shown in (3) and then presents a controller design method according to this fault mode.

2.4. Control Objective

The problem mainly considered in this paper is the design of an attitude-tracking controller for an HGV, so the main objective of the optimal tracking problem is to seek the control policy $u\left(t\right)$, so as to make the system (2) track a desired trajectory $r_d\left(t\right)$, where $r_d\left(t\right)={\left[\begin{array}{cccccc}{\alpha }_d,& {\beta }_d,& {\gamma }_{vd},& 0,& 0,& 0\end{array}\right]}^T$. Then, defining the tracking error as

$$e\left(t\right)\triangleq x\left(t\right)-r_d\left(t\right)$$

(4)

by (2) and (4), $e\left(t\right)$ can be rewritten as:

(Rule i) If $\alpha$ is ${\alpha }_i$, then

$$\dot{e}\left(t\right)=A_{i}e\left(t\right)+A_i{\dot{r}}_d\left(t\right)+B_i{u}^F\left(t\right)$$

(5)

where ${\dot{r}}_d\left(t\right)$ is the the differential of the reference command $r_d\left(t\right)$. Then, the whole system is:

$$\begin{array}{ccc}\hfill \dot{e}\left(t\right)& =& \stackrel{k}{\sum _{i=1}}{\mu }_i\left(t\right)\left(A_{i}e\left(t\right)+B_i{u}^F\left(t\right)+A_i{\dot{r}}_d\left(t\right)\right)\hfill \\ & =& \overline{A}e\left(t\right)+\overline{B}{u}^F\left(t\right)+\overline{A}{\dot{r}}_d\left(t\right)\hfill \end{array}$$

(6)

where ${\dot{r}}_d\left(t\right)$ is assumed to be bounded. Then, considering the actuator faults proposed in (3), the designed controller for (6) should track a command $r_d\left(t\right)$ under unknown actuator faults $u^F\left(t\right)$.

3. Main Results

In this section, a PI-based fuzzy adaptive FTC strategy is proposed based on only a little information about the actuator faults.

3.1. PI-Based Normal Controller

For the T–S fuzzy model (2) without actuator faults (3), the comprehensive control performance index for the ith rule is

$$J_i(e\left(t\right),u_i\left(t\right),{\dot{r}}_d\left(t\right))={\int }_t^{\infty }{u}_i^T\left(\tau \right)R{u}_i\left(\tau \right)+e{\left(\tau \right)}^{T}Qe\left(\tau \right)-{\gamma }^2{\dot{r}}_d{\left(t\right)}^T{\dot{r}}_d\left(t\right)d\tau ,$$

(7)

where $R\in {\mathbb{R}}^3$ and $Q\in {\mathbb{R}}^6$ are given positive definite symmetric matrices, and $\gamma$ is a prescribed constant that respects the performance of interference suppression for the designed controller. Defining the following value function for the ith rule:

$$V_i(e\left(t\right),u_i\left(t\right),{\dot{r}}_d\left(t\right))={\int }_t^{\infty }e{\left(\tau \right)}^{T}Qe\left(\tau \right)+u_i{\left(\tau \right)}^T{R}_i{u}_i\left(\tau \right)-{\gamma }^2{\dot{r}}_d{\left(t\right)}^T{\dot{r}}_d\left(t\right)d\tau$$

(8)

then, the above value function can be viewed as a zero-sum game between the control policy $u_i\left(t\right)$ and the derivation of the reference command ${\dot{r}}_d\left(t\right)$[28]. In this zero-sum game, the control policy $u_i\left(t\right)$ wants to seek an optimal controller to minimize the performance index (7), while ${\dot{r}}_d\left(t\right)$ wants to maximize it. According to the definition of a zero-sum game, we make the following definition of the Nash equilibrium.

Definition 1.

Nash equilibrium: The zero-sum game (8) has a unique Nash equilibrium if the following conditions are satisfied:

1. For each fixed control $u_i\left(t\right)$, there is always a unique ${\dot{r}}_d\left(t\right)$ that can maximize $V_i(e\left(t\right),u_i\left(t\right),{\dot{r}}_d\left(t\right))$; which means that, there exists a ${\dot{r}}_d^{*}\left(t\right)$ such that

$$V_i(e\left(t\right),u_i\left(t\right),{\dot{r}}_d^{*}\left(t\right))\ge V_i(e\left(t\right),u_i\left(t\right),{\dot{r}}_d\left(t\right));$$

2. The optimal game solution $u_i^{*}\left(t\right)$ can minimize $V_i(x,u,{\dot{r}}_d^{*}\left(t\right))$, which means that

$$V_i(e\left(t\right),u_i\left(t\right),{\dot{r}}_d^{*}\left(t\right))\ge V_i(e\left(t\right),u_i^{*}\left(t\right),{\dot{r}}_d^{*}\left(t\right));$$

3. $u_i^{*}\left(t\right)$ is the optimal control input, and ${\dot{r}}_d^{*}\left(t\right)$ is the optimal disturbance policy.

Remark 2.

In the above zero-sum game, we assume that the reference command ${\dot{r}}_d\left(t\right)$ is a player and can be changed freely. In practice, the reference command must combine some flight requirements, but this assumption is more conducive to us obtaining the optimal strategy of the game. This assumption can also cope with various changes in actual instructions, so this assumption is necessary.

For the above game, according to the Bellman equation, we choose the Hamiltonian function for the ith rule to be

$$H_i(e\left(t\right),u_i\left(t\right),{\dot{r}}_d\left(t\right))=e{\left(t\right)}^{T}Qe\left(t\right)+u_i{\left(t\right)}^T{R}_i{u}_i\left(t\right)-{\gamma }^2{\dot{r}}_d{\left(t\right)}^T{\dot{r}}_d\left(t\right)+\frac{\partial V_i\left(e\left(t\right)\right)}{\partial e\left(t\right)}$$

For convenience of description, using the fuzzy state feedback controller, we use the following rule

(Rule i) If $\alpha$ is ${\alpha }_i$, then

$$\begin{array}{ccc}\hfill u_i\left(t\right)& =& -K_{i}e\left(t\right),\hfill \\ \hfill {\dot{r}}_d\left(t\right)& =& L_{i}e\left(t\right)\hfill \end{array}$$

(9)

to describe the control policy and derivative policy of the reference command. Then, the game value function for the ith rule can be respected as

$$V_i\left(e\left(t\right),u_i\left(t\right),{\dot{r}}_d\left(t\right)\right)={\int }_t^{\infty }e{\left(\tau \right)}^T{P}_{i}e\left(\tau \right)d\tau$$

where $P_i$ is a solution of the following Algebraic Riccati Equation (10):

$$P_i(A_i+B_i{K}_i)+{(A_i+B_i{K}_i)}^T{P}_i+Q+K_i^T{R}_i{K}_i+{\gamma }^{-2}{P}_i{A}_i{A}_i^T{P}_i=0$$

(10)

According to the Pontryagin minimum principle, the optimal control policy for (9) is

$$\begin{array}{ccc}\hfill \frac{\partial H_i(e\left(t\right),u_i^{*}\left(t\right),{\dot{r}}_d\left(t\right))}{\partial u_i^{*}\left(t\right)}& =& 0\hfill \\ \hfill \frac{\partial H_i(e\left(t\right),u_i\left(t\right),{\dot{r}}_d^{*}\left(t\right))}{\partial {\dot{r}}_d^{*}\left(t\right)}& =& 0\hfill \end{array}$$

which implies that

$$\begin{array}{ccc}\hfill u_i^{*}\left(t\right)& =& -R_i^{-1}{B}_i^T{P}_i^{*}\zeta \left(t\right)=-K_{i}e\left(t\right)\hfill \end{array}$$

(11)

$$\begin{array}{ccc}\hfill {\dot{r}}_d^{*}\left(t\right)& =& {\gamma }^{-2}{A}_i^T{P}_i^{*}\zeta \left(t\right)=L_{i}e\left(t\right)\hfill \end{array}$$

(12)

where $u_i^{*}\left(t\right)$ is the optimal control for the ith rule of the T–S model (2),

$$K_i=R_i^{-1}{B}_i^T{P}_i^{*}$$

${\dot{r}}_d^{*}\left(t\right)$ is the optimal derivation of the reference command, and

$$L_i={\gamma }^{-2}{A}_i^T{P}_i^{*}$$

With the above optimal control $u_i^{*}\left(t\right)$ and ${\dot{r}}_d^{*}\left(t\right),$ the optimal game value function for the ith rule is

$$V_i^{*}\left(e\left(t\right),u_i^{*}\left(t\right),{\dot{r}}_d^{*}\left(t\right)\right)={\int }_t^{\infty }e{\left(\tau \right)}^T{P}_i^{*}e\left(\tau \right)d\tau$$

where $P_i^{*}$ is the optimal value of $P_i$. Then, for the T–S model of the HGV, the total controller is constructed as:

$$u\left(t\right)=-\stackrel{3}{\sum _{j=1}}{\mu }_j\left(t\right)K_{j}e\left(t\right)$$

Theorem 1.

For system (2) without actuator faults, if there exist positive-definite matrices $P_i>0$, and for the ith rule of the T–S fuzzy system, under controller (11) and (12), satisfying

$$P_i(A_i+B_i{K}_i)+{(A_i+B_i{K}_i)}^T{P}_i+Q+K_i^T{R}_i{K}_i+{\gamma }^{-2}{P}_i{A}_i{A}_i^T{P}_i=0$$

(13)

then, the T–S fuzzy system (2) is stable.

Proof. 

The overall performance of (6) is

$$\begin{array}{ccc}\hfill J_T& =& {\int }_t^{\infty }e{\left(\tau \right)}^{T}Pe\left(\tau \right)d\tau \hfill \\ & =& {\int }_t^{\infty }u{\left(\tau \right)}^{T}Ru\left(\tau \right)+e{\left(\tau \right)}^{T}Qe\left(\tau \right)d\tau \hfill \end{array}$$

(14)

where P is a positive definite symmetric matrix for the overall system. Then,

$$\begin{array}{ccc}\hfill {\dot{J}}_T& =& -u{\left(t\right)}^{T}Ru\left(t\right)-e{\left(t\right)}^{T}Qe\left(t\right)\hfill \\ & =& e^T\left(t\right)\left[\stackrel{3}{\sum _{i=1}}{\mu }_i\left(t\right)\stackrel{3}{\sum _{j=1}}{\mu }_j\left(t\right)P(A_i+B_i{K}_j)+{(A_i+B_i{K}_j)}^{T}P\right]e\left(t\right)+{\gamma }^2{\dot{r}}_d{\left(t\right)}^T{\dot{r}}_d\left(t\right)\hfill \\ & =& 2e^T\left(t\right)\left[\stackrel{3}{\sum _{i=1}}{\mu }_i\left(t\right)\stackrel{3}{\sum _{j=1}}{\mu }_j\left(t\right)\left(P{A}_i-P{B}_i{R}^{-1}{B}_j^T{P}_j\right)e\left(t\right)\right]+{\gamma }^2{\dot{r}}_d{\left(t\right)}^T{\dot{r}}_d\left(t\right)\hfill \end{array}$$

Since P and $P_j$ are all positive definite symmetric matrices, considering the matrices $B_i$ for controller design, there exist positive constants ${\varrho }_M\ge {\varrho }_m>0$,

$${\varrho }_m{e}^T\left(t\right)P_j{B}_j{R}^{-1}{B}_j^T{P}_{j}e\left(t\right)\le e^T\left(t\right)P{B}_i{R}^{-1}{B}_j^T{P}_{j}e\left(t\right)\le {\varrho }_M{e}^T\left(t\right)P_j{B}_j{R}^{-1}{B}_j^T{P}_{j}e\left(t\right)$$

and positive definite symmetric matrix $M_j$, and positive constant $\eta >0,$ such that

$$P=\eta P_j{M}_j.$$

Then,

$$\begin{array}{ccc}\hfill {\dot{J}}_T& =& 2e^T\left(t\right)\left[\stackrel{3}{\sum _{j=1}}{\mu }_j\left(t\right)\left(\eta P_j{\overline{A}}_j-\eta P_j{B}_i{R}^{-1}{B}_j^T{P}_j\right)e\left(t\right)\right]+{\gamma }^2{\dot{r}}_d{\left(t\right)}^T{\dot{r}}_d\left(t\right)\hfill \\ & =& 2\eta e^T\left(t\right)\left[\stackrel{3}{\sum _{j=1}}{\mu }_j\left(t\right)\left(P_j{\overline{A}}_j-P_j{B}_i{R}^{-1}{B}_j^T{P}_j\right)e\left(t\right)\right]+{\gamma }^2{\dot{r}}_d{\left(t\right)}^T{\dot{r}}_d\left(t\right)\hfill \\ & \le & 2\eta e^T\left(t\right)\left[\stackrel{3}{\sum _{j=1}}{\mu }_j\left(t\right)\left(P_j{\overline{A}}_j-{\varrho }_M{P}_j{B}_j{R}^{-1}{B}_j^T{P}_j\right)e\left(t\right)\right]+{\gamma }^2{\dot{r}}_d{\left(t\right)}^T{\dot{r}}_d\left(t\right)\hfill \\ & \le & -\eta {\int }_0^t{e}^T\left(t\right)\left(\stackrel{3}{\sum _{i=1}}{\mu }_j\left(t\right)\left(Q+\left(2{\varrho }_M-1\right)K_j^{T}R{K}_j\right)\right)e\left(\tau \right)d\tau +{\gamma }^2{\dot{r}}_d{\left(t\right)}^T{\dot{r}}_d\left(t\right)\hfill \end{array}$$

In this case, ${\dot{J}}_T<0$ if ${\varrho }_M>\frac{1}{2}$ and $∥e\left(t\right)∥>\sqrt{{\gamma }^2{\dot{r}}_d{\left(t\right)}^T{\dot{r}}_d\left(t\right)/{\lambda }_{\min }\left({\displaystyle \sum _{i=1}^3}{\mu }_j\left(t\right)(Q+\left(2{\varrho }_M-1\right)K_j^{T}R{K}_j)\right)}.$ The proof process is completed.    □

Based on Theorem 1, the online policy-iteration algorithm is as Algorithm 1:   

Algorithm 1: Model-based PI algorithm

1. Initialization: Select $i=0$, choose any reasonable policy $u_i\left(t\right)=K_i^{\left(0\right)}e\left(t\right)$; 2. Policy evaluation: Solve the following equations for $P_i^{(i+1)}$ $$P_i^{(i+1)}(A_i+B_i{K}_i^{\left(i\right)})+{(A_i+B_i{K}_i^{\left(i\right)})}^T{P}_i^{(i+1)}+Q+K_i^{T\left(i\right)}{R}_i{K}_i^{\left(i\right)}+{\gamma }^{-2}{P}_i{A}_i{A}_i^T{P}_i=0$$ 3. Policy improvement: $$\begin{array}{ccc}\hfill K_i^{\left(i+1\right)}& =& -R_i{\overline{B}}_i^T{P}_i^{(i+1)}\hfill \end{array}$$ (15) $$\begin{array}{ccc}\hfill L_i^{(i+1)}& =& {\gamma }^{-2}{A}_i^T{P}_i^{(i+1)}\hfill \end{array}$$ (16) 4. If the convergence condition is satisfied, stop; else, go to step 2.

In Algorithm 1, the superscript $\left(i\right)$ represents the ith iteration.

Remark 3.

In the proof of the above Theorem, overall performance $J_T$ in (14) is not equal to the simple sum of $J_i$ proposed in (7). This proof process for Theorem 1 has taken into account the PDC of fuzzy systems.

3.2. PI-Based Fuzzy FTC Control

Just as described in (3), ${\mathrm{\Gamma }}_F$ and ${\mathrm{\Lambda }}_F$ are unknown, but they are all bounded. Then, adaptive fuzzy FTC is constructed: for the T–S fuzzy model (6) with actuator faults (3), the fuzzy state feedback FTC controller is:

(Rule i) If $\alpha$ is ${\alpha }_i$, then

$$u_i\left(t\right)=-K_{i}e\left(t\right)+{\widehat{\mathrm{\Gamma }}}_{Fi}^{-1}{\widehat{\mathrm{\Lambda }}}_{Fi},$$

(17)

where ${\widehat{\mathrm{\Lambda }}}_{Fi}$ is the estimation of ${\mathrm{\Lambda }}_F$ at Rule i, and ${\widehat{\mathrm{\Gamma }}}_{Fi}$ is the estimation of ${\mathrm{\Gamma }}_F$ at Rule i.

Theorem 2.

For system (2) with actuator fault (3), if there exist positive-definite matrices $P_i>0$, and for the ith rule of the T–S fuzzy system and a FTC controller $K_i$ satisfying

$$P_i(A_i+B_i{K}_i)+{(A_i+B_i{K}_i)}^T{P}_i+Q+K_i^T{R}_i{K}_i+{\gamma }^{-2}{P}_i{A}_i{A}_i^T{P}_i=0$$

(18)

where

$$\begin{array}{ccc}\hfill K_i& =& -{\widehat{\mathrm{\Gamma }}}_{Fi}^{-1}{R}_i{B}_i^T{P}_i,\hfill \\ \hfill u_i\left(t\right)& =& K_{i}e\left(t\right)+{\widehat{\mathrm{\Gamma }}}_{Fi}^{-1}{\widehat{\mathrm{\Lambda }}}_{Fi},\hfill \end{array}$$

and the updating law ${\widehat{\mathrm{\Gamma }}}_{Fi}$ and ${\widehat{\mathrm{\Lambda }}}_{Fi}$ are

$$\begin{array}{ccc}\hfill {\dot{\widehat{\mathrm{\Lambda }}}}_{Fi}& =& 2q_1{B}_i^T{P}_{i}e\left(t\right)\hfill \end{array}$$

(19)

$$\begin{array}{ccc}\hfill {\dot{\widehat{\mathrm{\Gamma }}}}_{Fi}& =& diag\left(2q_{2}e{\left(t\right)}^T{P}_i{B}_i{u}_i\left(t\right)\right)\hfill \end{array}$$

(20)

where $q_1$ and $q_2$ are given constants, then system (2) is semi-globally uniformly ultimately bounded.

Proof. 

Choosing a Lyapunov function for tracking system (6) as

$$V^F\left(t\right)=V_1\left(t\right)+V_2\left(t\right)+V_3\left(t\right),$$

where

$$\begin{array}{ccc}\hfill V_1\left(t\right)& =& {\int }_0^{t}e{\left(\tau \right)}^{T}Pe\left(\tau \right)d\tau ,\hfill \\ \hfill V_2\left(t\right)& =& \frac{1}{2q_1}tr\left({\tilde{\mathrm{\Gamma }}}_{Fi}^T{\tilde{\mathrm{\Gamma }}}_{Fi}\right),\hfill \\ \hfill V_3\left(t\right)& =& \frac{1}{2q_2}{\tilde{\mathrm{\Lambda }}}_{Fi}{\tilde{\mathrm{\Lambda }}}_{Fi},\hfill \end{array}$$

and

$${\tilde{\mathrm{\Gamma }}}_{Fi}={\mathrm{\Gamma }}_{Fi}-{\widehat{\mathrm{\Gamma }}}_{Fi},{\tilde{\mathrm{\Lambda }}}_{Fi}={\mathrm{\Lambda }}_{Fi}-{\widehat{\mathrm{\Lambda }}}_{Fi}$$

Then, considering the differential of the proposed Lyapunov function

$$\begin{array}{ccc}\hfill {\dot{V}}_1\left(t\right)& =& 2e^T\left(t\right)\stackrel{3}{\sum _{i=1}}{\mu }_i\left(t\right)\stackrel{3}{\sum _{j=1}}{\mu }_j\left(t\right)\left[P{\overline{A}}_{i}e\left(t\right)+P{\overline{B}}_i{u}_j^F\right]e\left(t\right)+{\gamma }^2{\dot{r}}_d{\left(t\right)}^T{\dot{r}}_d\left(t\right)\hfill \\ & =& 2e^T\left(t\right)\stackrel{3}{\sum _{i=1}}{\mu }_i\left(t\right)\stackrel{3}{\sum _{j=1}}{\mu }_j\left(t\right)\left[P{\overline{A}}_{i}e\left(t\right)+P{\overline{B}}_i\left({\mathrm{\Gamma }}_{F_j}{u}_j\left(t\right)-{\mathrm{\Lambda }}_{Fj}\right)\right]+{\gamma }^2{\dot{r}}_d{\left(t\right)}^T{\dot{r}}_d\left(t\right)\hfill \\ & =& 2e^T\left(t\right)\stackrel{3}{\sum _{i=1}}{\mu }_i\left(t\right)\stackrel{3}{\sum _{j=1}}{\mu }_j\left(t\right)\left[P{\overline{A}}_{i}e\left(t\right)+P{\overline{B}}_i\left(\left({\mathrm{\Gamma }}_{Fi}-{\widehat{\mathrm{\Gamma }}}_{Fi}\right)u_j\left(t\right)+{\widehat{\mathrm{\Gamma }}}_{Fi}{u}_j\left(t\right)-{\mathrm{\Lambda }}_{Fj}\right)\right]\hfill \\ & & +{\gamma }^2{\dot{r}}_d{\left(t\right)}^T{\dot{r}}_d\left(t\right)\hfill \\ & =& 2e^T\left(t\right)\left[\stackrel{3}{\sum _{i=1}}{\mu }_i\left(t\right)\stackrel{3}{\sum _{j=1}}{\mu }_j\left(t\right)\left(P{\overline{A}}_i-P{\overline{B}}_i{R}^{-1}{B}_j^T{P}_j\right)\right]e\left(t\right)\hfill \\ & & +2e^T\left(t\right)\stackrel{3}{\sum _{i=1}}{\mu }_i\left(t\right)\stackrel{3}{\sum _{j=1}}{\mu }_j\left(t\right)\left[P{\overline{B}}_i\left({\tilde{\mathrm{\Gamma }}}_{Fi}{u}_j\left(t\right)-{\tilde{\mathrm{\Lambda }}}_{Fi}\right)\right]\hfill \\ & & +{\gamma }^2{\dot{r}}_d{\left(t\right)}^T{\dot{r}}_d\left(t\right)\hfill \end{array}$$

and considering the updating law of ${\widehat{\mathrm{\Gamma }}}_{Fi}$ and ${\widehat{\mathrm{\Lambda }}}_{Fi},$

$${\dot{V}}_2\left(t\right)=-tr\left({\tilde{\mathrm{\Gamma }}}_{Fi}^T{\widehat{\mathrm{\Gamma }}}_{Fi}\right).$$

$${\dot{V}}_3\left(t\right)=-{\tilde{\mathrm{\Lambda }}}_{Fi}{\widehat{\mathrm{\Lambda }}}_{Fi}.$$

and

$$\begin{array}{ccc}& & e^T\left(t\right)\stackrel{3}{\sum _{i=1}}{\mu }_i\left(t\right)\stackrel{3}{\sum _{j=1}}{\mu }_j\left(t\right)\left[P{\overline{B}}_i\left({\tilde{\mathrm{\Gamma }}}_{Fi}{u}_j\left(t\right)-{\tilde{\mathrm{\Lambda }}}_{Fi}\right)\right]\hfill \\ & =& e^T\left(t\right)P\stackrel{3}{\sum _{i=1}}{\mu }_i\left(t\right)\stackrel{3}{\sum _{j=1}}{\mu }_j\left(t\right)\left[{\overline{B}}_i\left({\tilde{\mathrm{\Gamma }}}_{Fi}{u}_j\left(t\right)-{\tilde{\mathrm{\Lambda }}}_{Fi}\right)\right]\hfill \end{array}$$

then,

$$\begin{array}{ccc}& & {\dot{V}}_1\left(t\right)+{\dot{V}}_2\left(t\right)+{\dot{V}}_3\left(t\right)\hfill \\ & \le & -\eta {\int }_0^t{e}^T\left(t\right)\left(\stackrel{3}{\sum _{i=1}}{\mu }_j\left(t\right)\left(Q+\left(2{\varrho }_M-1\right)K_j^{T}R{K}_j\right)\right)e\left(\tau \right)d\tau +{\gamma }^2{\dot{r}}_d{\left(t\right)}^T{\dot{r}}_d\left(t\right)\hfill \\ & & +2{\varrho }_M^{\prime }{e}^T\left(t\right)P_j{B}_j{\tilde{\mathrm{\Gamma }}}_{Fi}{u}_j\left(t\right)-tr\left({\tilde{\mathrm{\Gamma }}}_{Fi}^T{\widehat{\mathrm{\Gamma }}}_{Fi}\right)-2{\varrho }_m^{\prime }{e}^T\left(t\right)P_j{B}_j{\tilde{\mathrm{\Lambda }}}_{Fi}-{\tilde{\mathrm{\Lambda }}}_{Fi}{\widehat{\mathrm{\Lambda }}}_{Fi}\hfill \\ & \le & -\eta {\int }_0^t{e}^T\left(t\right)\left(\stackrel{3}{\sum _{i=1}}{\mu }_j\left(t\right)\left(Q+\left(1-2(1-\eta )\right)K_j^{T}R{K}_j\right)\right)e\left(\tau \right)d\tau +{\gamma }^2{\dot{r}}_d{\left(t\right)}^T{\dot{r}}_d\left(t\right)\hfill \end{array}$$

Based on the analysis of Theorem 1, the whole system is semi-globally uniformly ultimately bounded.    □

3.3. IRL-Based Fuzzy FTC Control

For solving Theorem 2, all information on $A_i$ and $B_i$ are needed, and the ARE is difficult to solve. To reduce the dependence on the system model in the solution process, and give a more easily solved method, a new data-based PI fuzzy FTC controller is developed from Theorem 2 and Algorithm 1, in which the system matrices $A_i$ can be unknown and only $B_i$ are utilized for the proposed FTC [25].

The derivative of the ith value function is

$$\begin{array}{ccc}\hfill {\dot{V}}_i(e\left(t\right),u_i\left(t\right),{\dot{r}}_d\left(t\right))& =& e^T\left(t\right)P_i\dot{e}\left(t\right)\hfill \\ & =& -e{\left(t\right)}^{T}Qe\left(t\right)-u_i{\left(t\right)}^T{R}_i{u}_i\left(t\right)+{\gamma }^2{\dot{r}}_d{\left(t\right)}^T{\dot{r}}_d\left(t\right)\hfill \end{array}$$

(21)

Integrating both sides of (21) from t to $t+T$, results in

$$e{(t+T)}^T{P}_i^{(i+1)}e(t+T)-e{\left(t\right)}^T{P}_i^{(i+1)}e\left(t\right)=-{\int }_t^{t+T}\left[e{\left(\tau \right)}^T\left(Q+K_i^{\left(i\right)T}{R}_i{K}_i^{\left(i\right)}\right)e\left(\tau \right)-{\gamma }^2{\dot{r}}_d{\left(t\right)}^T{\dot{r}}_d\left(t\right)\right]d\tau .$$

Since the value of $e(t+T),$ $e\left(t\right)$ and the integral of the right side can all be computed, the positive defined matrices $P_i$ can be solved by the above equation.

Then, the following theorem is proposed (Algorithm 2):    

Algorithm 2: IRL-based fuzzy FTC control algorithm

1. Initialization: Select $i=0$, choose any reasonable policy $u_i\left(t\right)=K_i^{\left(0\right)}e\left(t\right)$, $P^{\left(i\right)}>0$; 2. Policy evaluation: Solve the following equations for $P_i^{(i+1)}$ $$e{(t+T)}^T{P}_i^{(i+1)}e(t+T)-e{\left(t\right)}^T{P}_i^{(i+1)}e\left(t\right)=-{\int }_0^t\left[e{\left(\tau \right)}^T\left(Q+K_i^{\left(i\right)T}{R}_i{K}_i^{\left(i\right)}\right)e\left(\tau \right)-{\gamma }^2{\dot{r}}_d{\left(t\right)}^T{\dot{r}}_d\left(t\right)\right]d\tau .$$ 3. Policy improvement: $$K_i^{\left(i+1\right)}=-{\widehat{\mathrm{\Gamma }}}_{Fi}^{-1}{R}_i{B}_i^T{P}_i^{(i+1)}.$$ and $$\begin{array}{ccc}\hfill {\widehat{\mathrm{\Lambda }}}_{Fi}& =& 2q_1{B}_i^T{P}_{i}e\left(t\right)\hfill \\ \hfill {\widehat{\mathrm{\Gamma }}}_{Fi}& =& diag\left(2q_{2}e{\left(t\right)}^T{P}_i{B}_i{u}_i\left(t\right)\right)\hfill \end{array}$$ 4. If the convergence condition is satisfied, stop; else, go to step 2.

A flowchart of the proposed AFFTC is given in Figure 1. From Figure 1, one can see that, the desired trajectory $r_d\left(t\right)$ is firstly added to system (1), and then, the tracking error $e\left(t\right)$ is generated. After that, the adaptive parameter estimator updates the parameters according to (19) and (20), and the controller updates the controller according to (18). Finally, the overall updating of the fuzzy controller is completed.

Remark 4.

The proposed FTC method is an IRL-based adaptive fuzzy FTC controller. Although adaptive control methods can adjust the parameters of a controller, the efficiency of self-adaptation adjustment is too low for time-varying faults or sudden faults. Reinforcement learning can quickly obtain new control parameters according to the changes in HGV states. Combining reinforcement learning with self-adaptation, the proposed method can quickly adjust the parameters of the controller when the HGV has sudden or time-varying faults.

4. Simulation Results

According to the T–S approximation method proposed in Section 2, the fuzzy model of the HGV’s attitude system can be constructed as follows:

(Rule 1) If $\alpha$ is ${\alpha }_S$, then

$$\dot{x}\left(t\right)=A_{1}x\left(t\right)+B_{1}u\left(t\right),$$

(Rule 2) If $\alpha$ is ${\alpha }_M$, then

$$\dot{x}\left(t\right)=A_{2}x\left(t\right)+B_{2}u\left(t\right),$$

(Rule 3) If $\alpha$ is ${\alpha }_B$, then

$$\dot{x}\left(t\right)=A_{3}x\left(t\right)+B_{3}u\left(t\right),$$

Then, based on the adaptive fuzzy FTC strategy given in Section 3.2, the FTC controller can be driven from different faults. The parameters and initial states of the proposed simulation are similar to ref. [5]. For testing the effectiveness of the proposed controller, three simulations under three different fault modes are given as follows:

Case I: loss of effectiveness. In this simulation, the actuator faults are

$${\rho }_F=\left[\begin{array}{ccc}0.6& 0& 0\\ 0& 0.6& 0\\ 0& 0& 0.6\end{array}\right],{\beta }_F=\left[\begin{array}{c}0\\ 0\\ 0\end{array}\right].$$

According to the proposed method, the loss of effectiveness fault is carried on the nonlinear attitude model of the HGV. The fault is set to occur 40 s after the simulation starts. For comparison, the T–S controller proposed in ref. [29], marked as $u_{T-S}$, is applied to the attitude control of the HGV, and the proposed adaptive fuzzy control is marked as $u_{FTC}$. The tracking simulation results are presented in Figure 2 and Figure 3. Specifically, the tracking results of the given command are presented in Figure 2, and the input is presented in Figure 3. From the simulation figures, we can see that, when a fault occurs, the tracking error of $u_{FTC}$ is smaller than that of $u_{T-S}$, and the input of $u_{FTC}$ is also smoother than that of $u_{T-S}$. So, the proposed adaptive fuzzy FTC can adjust itself and obtain a better performance.

Case II: Drift fault. In this simulation, the following faults are considered:

$${\rho }_F=\left[\begin{array}{ccc}1& 0& 0\\ 0& 1& 0\\ 0& 0& 1\end{array}\right],{\beta }_F=\left[\begin{array}{c}0.1\\ 0.1\\ 0.1\end{array}\right].$$

The fault is set to occur 40 s after the simulation starts. The tracking simulation results are presented in Figure 4 and Figure 5. Specifically, the tracking results of the given command are presented in Figure 4, and the input is presented in Figure 5. From the simulation figures, we can see that, under drift fault, the proposed adaptive fuzzy FTC can also guarantee the tracking performance of the HGV, and the smoothness of the input under $u_{FTC}$ is better than that under $u_{T-S}$.

Case III: Combined fault. In this simulation, the following faults are considered:

$${\rho }_F=\left[\begin{array}{ccc}0.6& 0& 0\\ 0& 0.6& 0\\ 0& 0& 0.6\end{array}\right],{\beta }_F=\left[\begin{array}{c}0.1\\ 0.1\\ 0.1\end{array}\right].$$

The faults are set to occur 40 s after the simulation starts. The tracking simulation results are presented in Figure 6 and Figure 7. Specifically, the tracking results of the given command are presented in Figure 6, and the input to the HGV is presented in Figure 7. Under the combined fault, the input of $u_{FTC}$ and $u_{T-S}$ are all non-smooth, but the oscillation amplitude of $u_{FTC}$ is much smaller than that of $u_{T-S}$.

5. Conclusions

FTC of an HGV’s attitude control system is discussed in this paper. Actuator faults are considered and the mode of the faults is the general model of the actuator. Based on the T–S fuzzy approach, a nonlinear attitude model is firstly modeled using a T–S fuzzy model, and then, a normal T–S controller without considering the actuator fault is designed utilizing RL technology. Then, based on the normal fuzzy controller, an improved adaptive FTC is designed in which the FTC can be adjusted online according to the failure mode. Finally, simulation results on different kinds of actuator faults are given to show the good performance of the proposed FTC.

The simulation results given in this paper are based on a mathematical model. Future work will consist of engineering the proposed algorithm and verification of the proposed method by combining the algorithm with an actual HGV.