Fixed-Point Control of Airships Based on a Characteristic Model: A Data-Driven Approach

Abstract

Factors such as changes in the external atmospheric environment, volatility in the external radiation, convective heat transfer, and radiation between the internal surfaces of the airship skin will cause a series of changes in the motion model of an airship. The adaptive control method of the characteristic model is proposed to extract the relationship between input and output in the original system, without relying on an accurate dynamic model, and solves the problem of inaccurate modeling. This paper analyzes the variables needed for two-dimensional path tracking and combines the guidance theory and the method of wind field state conversion to determine specific control targets. Through the research results, under the interference of wind, the PD control method and the reinforcement learning-based method are compared with a characteristic model control method. The response speed of the characteristic model control method surpasses the PD control method, and it reaches a steady state earlier than the PD control method does. The overshoot of the characteristic model control method is smaller than that of the PD control method. Using the control method of the characteristic model, the process of an airship flying to a target point will be more stable under the influence of an external environment. The modeling of the characteristic model adaptive control method does not rely on a precise model of the system, and it automatically adjusts when the parameters change to maintain a consistent performance in the system, thus reflecting the robustness and adaptability of the characteristic model adaptive control method in contrast with reinforcement learning.

1. Introduction

The high-altitude airship is a kind of unmanned aerial vehicle lighter than air. It uses electric drive propellers or new ion propulsion. It can hover or fly at low speed for a long time at a fixed point in the high-altitude range from 20 to 30 km. The high altitude airship has the characteristic of hovering at a fixed point for a long time, which makes it play an irreplaceable role in the mission [1]. Moreover, the airship has the characteristics of a low production cost, a long dwell time, and a large load capacity. In the military, the stratospheric airship can be used for military reconnaissance, gathering important intelligence, and monitoring secret targets. In terms of civilian use, it can carry out observations on the ground and measurements in land meteorology. It can also be used as an ideal platform for high-altitude communication, with the advantage of a wide coverage area for transmitted and received signals [2,3]. When studying the motion control of the airship, the change in airship position caused by atmospheric circulation is generally considered, while the changes in parameter changes brought about by the external environment of the airship are ignored [4]. The external atmospheric environment, external radiation, convective heat transfer, and radiant heat transfer inside the airship skin can cause changes in the skin’s temperature [5]. In order to achieve the safe flight of the airship, the difference between the gas pressure in the airbag and the outside atmospheric pressure is maintained within a certain range [6]. However, the adaptive control method based on the characteristic model extracts the main relationship between input and output on the original dynamic model without relying on the precise dynamic model [7]. This design of the controller is more in line with the actual engineering. Liang Dong et al. used the Lyapunov stability theorem to design the fixed point mode of a stratospheric airship, but this method required an accurate dynamic model and did not consider the influence of the wind field on the airship [6]. Wang Xiaoliang et al. achieved the fixed point control of an airship based on sliding mode variable structure control, but only considered the influence of the horizontal wind field [5]. Gao Wei et al. used the neural network dynamic inverse control method to design the fixed-point control law of an airship’s horizontal plane and only considered the one-dimensional motion model on the horizontal plane [3].

The uncertainty of airship parameters in control has been studied extensively in the past few years. Considering the special structure of some nonlinear objects, such as the upper triangle and the lower triangle, relevant research has solved the adaptive control of these systems [8,9]. In order to facilitate online execution, decentralized adaptive control of these special structures has been investigated [10]. Practical factors (such as structural damage and unmeasurable states) that affect robustness have also been considered [11,12,13]. Reinforcement learning (RL) algorithms are being increasingly applied to control airships [14,15]. Obviously, all of the above relevant adaptive methods involve prior system information.

In this paper, a six-degree-of-freedom dynamic model of an airship is established based on the Newton–Euler equation. By analyzing the variables required by the airship model in two-dimensional path tracking, the dynamic model of an airship is simplified into a three-degree-of-freedom motion model, and specific control targets are given by integrating guidance theory with the wind field state conversion method. Based on characteristic modeling of the kinematics model, combined with the golden section adaptive control law and the logical differential control law, this work achieves the fixed-point control of an airship. For comparison purposes, a PD controller is often selected as an object of comparison [16,17,18,19,20,21]. In the mathematical simulation, the control result of the controller is compared with the control result of a traditional PD controller, which shows the effectiveness and superiority of the controller designed in this study.

With the improvement of hardware computing power and the emergence of various numerical calculation methods, data-driven approaches have emerged in a variety of fields [22,23,24]. Their advantage lies in the ability to interact with the environment online, which reduces the involvement of prior information [25]. In a control field, the datasets will be part of the controller and participate in the control in an adaptive manner [26]. Offline methods compute a control policy in the form of a parametric form or a table lookup prior to applying it to the process. Although the online computing needed might be minimal, robustness against uncertainty decreases. From the perspective of control theory, ML or RL achieves stability and a transient performance of the system by determining the dynamic programming problem, which shows a fitting process that identifies the HJB (Hamilton–Jacobi–Bellman) equation with a posteriori approach [27]. Making the most feasible choice in accordance with certain desired criteria is never easy. In airship control, relevant research can be found in [28,29,30,31]. Such solutions can only be made when the system model is unknown or there are uncertainties in the system dynamics, which make them even more difficult and sometimes impossible [32]. Nevertheless, owing to the difficulty of the computational problem (even if existence and other theoretical questions are settled), this is not always possible. In addition, the practical control problem is not usually posed as a well-formulated mathematical optimization problem. In this work, we will provide a data-driven approach without pursuing an optimal index by identifying the characteristic model online, which is instrumental in practice. Meanwhile, we provide the results of comparison with the RL algorithm, which can be found in [33,34].

The organization of this paper is as follows. In Section 2, we provide a description of the dynamic system. In Section 3, some control loops are provided, and corresponding characteristic models are divided. The main results of this paper are then presented. Based on the model given in Section 3, the fixed-point control method is presented in Section 4 and Section 5. Some necessary proofs are provided. In Section 6, we compare some experiments by numerical simulation. For a comparison with existing data-driven methods, in Section 7, we provide a policy iteration algorithm via reinforcement learning, and a comprehensive comparative analysis is given. Finally, we provide some concluding remarks in Section 8.

2. The Dynamic Model of a Stratospheric Airship

The structure of the airship is shown in Figure 1. The thrust system of the airship is composed of two propellers at the rear of the airship. The direction of the airship is controlled by changing the deflection of the rudder of the airship. In this paper, the ground coordinate system and airship coordinate system are established. In the ground coordinate system, the origin of coordinate $o_e$ is selected as any point on the ground. The axis $o_e{x}_e$ points to the east direction, and the axis $o_e{y}_e$ points to the north direction. The origin of the airship coordinate system $o_b$ is set at the center of the airship’s body. The coordinate axis $o_b{x}_b$ is arranged in the plane of symmetry, parallel to the axis of the body of the airship and pointing to the head of the airship. The coordinate axis $o_b{z}_b$ is set in the plane of symmetry perpendicular to the coordinate axis $o_b{x}_b$, pointing downwards. $o_b{y}_b$ is set perpendicular to the plane $x_b{o}_b{y}_b$ and points to the right. This article studies the fixed-point control of the stratosphere airship in the horizontal plane. The roll motion and the motion in the vertical plane need not be considered. Therefore, there is no need to consider the pitch angle $\theta$ and the roll angle $\mathsf{\Phi }$ only the change in the yaw angle $\psi$. T is the thrust vector. The thrust components in the directions of $o_b{x}_b$, $o_b{y}_b$ can be expressed as $T_x=Tcos\mu$, $T_y=Tsin\mu$, where $\mu$ is used as the angle between the thrust vector and the plane $x_b{o}_b{y}_b$ The six-degree-of-freedom motion model of the airship can be simplified into a three-degree-of-freedom dynamic equation. The three-degree-of-freedom dynamic equation can be expressed as follows:

$$\left\{\begin{array}{c}{\dot{u}}_b=\frac{m+m_{22}}{m+m_{11}}{v}_{b}r+\frac{Q{V}_f^{\frac{2}{3}}}{m+m_{11}}(-C_{x}cos\beta +C_{y}sin\beta )+\frac{T_x}{m+m_{11}}\hfill \\ {\dot{v}}_b=\frac{m+m_{11}}{m+m_{22}}{u}_{b}r+\frac{Q{V}_f^{\frac{2}{3}}}{m+m_{22}}(C_{x}sin\beta +C_{y}cos\beta )+\frac{T_y}{m+m_{22}}\hfill \\ \dot{r}=\frac{1}{I_z+m_{66}}Q{V}_f{C}_n+\frac{l_x}{I_z+m_{66}}{T}_y\hfill \end{array}\right.$$

(1)

The simplified kinematic equation of the airship in the horizontal plane can be expressed as

$$\left\{\begin{array}{c}{\dot{x}}_e=u_{b}cos\psi -v_{b}sin\psi +V_{\omega }cos{\psi }_{\omega }\hfill \\ {\dot{y}}_e=u_{b}sin\psi +v_{b}cos\psi +V_{\omega }sin{\psi }_{\omega }\hfill \\ \dot{\psi }=r\hfill \end{array}\right.$$

(2)

where m is the central mass of the airship, $m_{ii}(i=1,2,3,\cdots ,6)$ is additional mass; $V_f$ is the volume of the airship; Q is the dynamic pressure, $Q=\frac{1}{2}{\rho }_a{V}_f^2$, where ${\rho }_a$ is the atmospheric density; $u_b,v_b,r$ are the forward, lateral, yaw speed, respectively; $C_x$, $C_y$, and $C_n$ are the drag coefficient, side force coefficient, and yaw moment coefficient, respectively; $l_z$ is the distance from the thrust point $p_0$ to the coordinate axis $o_b{x}_b$; $\beta$ is expressed as the side slip angle. $I_z$ is expressed as the moment of inertia about the coordinate axis $o_b{z}_b$; $x_e$ and $y_e$ are, respectively, expressed as the axial displacement and lateral displacement of the airship.

Remark 1. 

For the idea of the characteristic model, it is assumed that the dynamic characteristics of the controlled system are known. Based on these characteristics, one can establish a mathematical model that is different from ordinary differential equations, and then use appropriate parameter estimation methods to obtain the coefficients of the characteristic model equation, i.e., determine quantitatively the system model of the characteristic equation in a data-driven way. Its goal is to facilitate the control objectives and performance to achieve the controller design. We cannot acquire precise system matrices, so a characteristic model is a good method for describing the system evolution, which is different from the dynamics model identified by the first principle.

3. The Target of the Control

The speed of the airship relative to the ground is equal to the speed of the airship relative to the air plus the wind speed, so the plane position kinematics equation in the stratospheric airfield can be expressed as

$${\dot{\zeta }}_p=\left[\begin{array}{cc}cos\psi & -sin\psi \\ sin\psi & cos\psi \end{array}\right]\left[\begin{array}{c}{u}_b\\ v_b\end{array}\right]+\left[\begin{array}{c}{V}_{\omega }cos{\psi }_{\omega }\\ V_{\omega }sin{\psi }_{\omega }\end{array}\right]$$

(3)

where ${\zeta }_p$ represents the position of the airship, $V_{\omega }$ represents the wind speed, and $V_{\omega }>0$; ${\psi }_{\omega }$ represents the wind direction. The position of the airship in the wind field can be represented as

$$\left\{\begin{array}{c}{x}_b=x_e-{\int }_0^t{V}_{\omega }cos{\psi }_{\omega }dt\hfill \\ y_b=y_e-{\int }_0^t{V}_{\omega }sin{\psi }_{\omega }dt\hfill \end{array}\right.$$

(4)

Thus, Equation (3) can be expressed as

$${\dot{\zeta }}_a=\left[\begin{array}{c}{\dot{x}}_b\\ {\dot{y}}_b\end{array}\right]=\left[\begin{array}{cc}cos\psi & -sin\psi \\ sin\psi & cos\psi \end{array}\right]\left[\begin{array}{c}{u}_b\\ v_b\end{array}\right]$$

(5)

where ${\zeta }_a$ represents the position error of the airship. In this paper, the straight path that passes through the desired point ${\zeta }_d={\left[x_d,y_d\right]}^T$ and has a slope of ${\psi }_{\omega }$ which can be given as

$${\zeta }_c\left(\varpi \right)=\left[\begin{array}{c}{x}_c\left(\varpi \right)\\ y_c\left(\varpi \right)\end{array}\right]=\left[\begin{array}{c}-\varpi cos{\psi }_{\omega }+x_d\\ -\varpi sin{\psi }_{\omega }+y_d\end{array}\right]$$

(6)

where $\varpi$ is the path parameter. Based on the definition of ${\zeta }_c$, we can see that, if ${\zeta }_a$ tracks ${\zeta }_c$, when $\dot{\varpi }>0$, the airship heading angle is directly opposite to the wind direction. At the same time, if the control airship’s forward airspeed $u_a$ is equal to the wind speed, fixed-point hovering can be achieved.

3.1. The Design of the Guidance Loop

In this paper, guidance theory is applied to solve the fixed point problem of an airship on a two-dimensional plane. We define the path coordinate system with the point ${\zeta }_c\left(\varpi \right)$ on the desired path as the origin and with the axis $x_c$ along the tangential direction of ${\zeta }_c\left(\varpi \right)$. $y_c$ is perpendicular to $x_c$ and points to the right. The rotation angle from the airship’s body coordinate system to the path coordinate system can be expressed as

$${\phi }_p=arctan({\dot{y}}_c/{\dot{x}}_c)={\psi }_{\pi }-\pi$$

(7)

Differentiating Equation (6) and combining Equation (7), we have

$${\dot{\zeta }}_c=\left[\begin{array}{cc}cos{\psi }_p& -sin{\psi }_p\\ sin{\psi }_p& cos{\psi }_p\end{array}\right]\left[\begin{array}{c}\dot{\varpi }\\ 0\end{array}\right]=R\left({\psi }_p\right)\left[\begin{array}{c}\dot{\varpi }\\ 0\end{array}\right]$$

(8)

In the path coordinate system, the position error between ${\zeta }_a$ and ${\zeta }_c$ can be expressed as

$$\epsilon =\left[\begin{array}{cc}cos{\psi }_p& -sin{\psi }_p\\ sin{\psi }_p& cos{\psi }_p\end{array}\right]\left[\begin{array}{c}{x}_b-x_d\\ y_b-y_d\end{array}\right]=R^T\left({\psi }_p\right)({\zeta }_a-{\zeta }_c\left(\varpi \right))$$

(9)

Therefore, the target of the guidance loop is converted to the design expected yaw angle such that $\epsilon \to 0$. The definition of the Lyapunov function can be expressed as

$$V_{\epsilon }=\frac{1}{2}{\epsilon }^T\epsilon$$

(10)

We assume that, when $u_a=V_{\omega }$, $\psi ={\psi }_c$, ${\psi }_c$ is a given value. the stratosphere airship reaches the desired path ${\zeta }_c\left(\varpi \right)$. Differentiating $V_{\epsilon }$ with Equations (5) and (8), we have

$$\begin{array}{c}\hfill {\dot{V}}_{\epsilon }={\epsilon }^T({\dot{R}}^T\left({\psi }_p\right)({\zeta }_a-{\zeta }_c)+R^T\left({\psi }_p\right)({\dot{\zeta }}_a-{\dot{\zeta }}_c))\end{array}$$

(11)

$\dot{\varpi }=V_{\omega }cos({\psi }_c-{\psi }_p)-v_{a}sin({\psi }_c-{\psi }_p)+k_{s}s$, ${\psi }_c={\psi }_p+arctan\left(\frac{-e}{k_e}\right)-arctan\left(\frac{v_b}{V_{\omega }}\right)$, where $\left\{k_s,k_e\right\}>0$ are the control parameters, which can be substituted into Equation (11) to obtain

$${\dot{V}}_{\epsilon }=-k_s{s}^2-\frac{\sqrt{V_{\omega }^2+v_a^2}}{\sqrt{e_2+k_e^2}}{e}^2\le 0$$

(12)

where s is the forward tracking error, and e is the lateral tracking error. Therefore, the path following error $\epsilon$ is convergent.

3.2. The Design of Expectation Speed

To achieve the control goal, the airship has the same speed as the wind speed and flies along the path. The defined expected forward airspeed can be expressed as

$$u_{bc}=V_{\omega }-k_p(cos{\psi }_p(x_b-x_d)+sin{\psi }_p(y_b-y_d))$$

(13)

where $k_p>0$ is the controller parameter. The desired airspeed value can be adjusted by the position error. If the airship is in front of the desired position, the desired airspeed is reduced; otherwise, the airship is accelerated. Therefore, the desired forward speed can be expressed as

$$u_c=u_{bc}-V_{\omega }=-k_p{s}_p$$

(14)

where $s_p$ is the forward tracking error between ${\zeta }_p$ and ${\zeta }_d$.

4. Fixed-Point Control of an Airship Based on a Multiple-Input Multiple-Output Characteristic Model

The stratospheric airship is a nonlinear coupling system with multiple inputs and outputs. The analyses for a precise dynamic model is not a trivial task, even if a Lyapunov equation can be found by trial and error. In this section, We apply a characteristic model approach which is built on a discrete state space. Adaptive learning method will be employed in the following policy design, which includes feedforward compensation and feedback stabilization in the method of characteristic model. Specifically, golden section adaptive controller is used to learn airship parameters online, which is subject to prescribed coefficients $L_1=0.382$ and $L_2=0.618$. It is worth mentioning that minimum variance combined with $L_1$, $L_2$ can guarantee system stability at given equilibrium state under some mild limits on the range of parameters learnt. For avoiding redundancies, we recommend the literature [4,35] for the details of the characteristic model. In addition, we ignore the feedforward compensation part of the controller design. Firstly, a four channel coupling model of forward velocity, lateral velocity, yaw velocity, and yaw angle are established. We then parameterize these known models by the characteristic model. In practice, one can adopt it directly.

4.1. The Characteristic Model Derivation of the Forward Velocity Channel

According to Equation (1), where $Q=\frac{1}{2}{\rho }_a(u_b^2+v_b^2)$ is dynamic pressure, $\beta =arctan\left(\frac{v_b}{u_b}\right)$ in Equation (1) is the slip angle, we then obtain a more compact expression:

$$\begin{array}{cc}\hfill {\dot{u}}_b& =\frac{1}{2(m+m_{11})}{\rho }_a·V_f^{\frac{2}{3}}(u_b^2+v_b^2)[-C_x·cos(arctan\frac{v_b}{u_b})\hfill \\ \hfill & +(C_{y0}+C_{ybeta}·arctan\frac{v_b}{u_b}+C_{yr}·r)sin(arctan\frac{v_b}{u_b})]\hfill \\ \hfill & +\frac{1}{m+m_{11}}{T}_x+\frac{m+m_{22}}{m+m_{11}}{v}_{b}r\hfill \\ \hfill & =f_1(u_b,v_b,r)+\frac{1}{m+m_{11}}{T}_x\hfill \end{array}$$

(15)

where $C_{y0}$, $C_{ybeta}$, $C_{yr}$ are the different drag model parameters. One can then obtain the derivative of Equation (15), which will be used for the parameterization design of the characteristic model:

$${\ddot{u}}_b={f^{\prime }}_{1,u_b}·{\dot{u}}_b+{f^{\prime }}_{1,v_b}·{\dot{v}}_b+{f^{\prime }}_{1,r}·\dot{r}+\frac{1}{m+m_{11}}{\dot{T}}_x$$

(16)

The expression of ${f^{\prime }}_{1,u_b}$ in Equation (16) can be expressed as

$$\begin{array}{cc}\hfill {f^{\prime }}_{1,u_b}& =\frac{1}{m+m_{11}}{\rho }_a·V_f^{\frac{2}{3}}·u_b[-C_{x}cos\beta +C_{y}sin\beta ]\hfill \\ \hfill & +\frac{1}{2(m+m_{11})}{\rho }_a·V_f^{\frac{2}{3}}\hfill \\ \hfill & \times [-C_{x}sin\beta ·v_b-C_{ybeta}sin\beta ·v_b-C_{y}cos\beta ·v_b]\hfill \\ \hfill & -C_{ybeta}sin\beta ·v_b-C_{y}cos\beta ·v_b]\hfill \end{array}$$

(17)

In the same way, the expression of ${f^{\prime }}_{1,v_b}$ in Equation (16) can be expressed as

$$\begin{array}{cc}\hfill {f^{\prime }}_{1,v_b}& =\frac{1}{m+m_{11}}{\rho }_a·V_f^{\frac{2}{3}}·v_b[-C_{x}cos\beta +C_{y}sin\beta ]\hfill \\ \hfill & +\frac{1}{2(m+m_{11})}{\rho }_a·V_f^{\frac{2}{3}}\hfill \\ \hfill & \times [-C_{x}sin\beta ·u_b-C_{ybeta}sin\beta ·u_b-C_{y}cos\beta ·u_b]\hfill \\ \hfill & +\frac{m+m_{22}}{m+m_{11}}r\hfill \end{array}$$

(18)

As ${f^{\prime }}_{1,u_b}$ and ${f^{\prime }}_{1,v_b}$ above, ${f^{\prime }}_{1,r}$ in Equation (16) can be acquired:

$${f^{\prime }}_{1,r}=\frac{m+m_{22}}{m+m_{11}}{v}_b+\frac{{\rho }_a·V_f^{\frac{2}{3}}}{2(m+m_{11})}(u_b^2+v_b^2)·C_{yr}sin\beta$$

(19)

Equation (15) can then be simplified to

$${\ddot{u}}_b={f^{\prime }}_{1,u_b}·{\dot{u}}_b+\frac{1}{m+m_{11}}{\dot{T}}_x+{\Delta }_u$$

(20)

where ${\Delta }_u={f^{\prime }}_{1,v_b}·{\dot{v}}_b+{f^{\prime }}_{1,r}·\dot{r}$. Equation (20) after discretization can be expressed as

$$\begin{array}{cc}\hfill \frac{u_b(k+1)-2u_b\left(k\right)+u_b(k-1)}{h^2}& ={f^{\prime }}_{1,u_b}\frac{u_b\left(k\right)-u_b(k-1)}{h}\hfill \\ \hfill & +\frac{1}{m+m_{11}}\frac{T_x\left(k\right)-T_x(k-1)}{h}\hfill \\ \hfill & +{\Delta }_u\left(k\right)\hfill \end{array}$$

(21)

Equation (21) is written in the form of a characteristic model [36]:

$$\begin{array}{cc}\hfill u_b(k+1)& =f_{11}·u_b\left(k\right)+f_{12}·u_b(k-1)\hfill \\ \hfill & +g_{11}\left(k\right)·T_x\left(x\right)+g_{12}\left(k\right)·T_x(k-1)+{\Delta }_u\left(k\right)\hfill \end{array}$$

(22)

For normalization, Equation (22) is written as a parameter estimation equation:

$$\begin{array}{cc}\hfill u_b(k+1)& ={\overline{f}}_{11}\left(k\right)·u_b\left(k\right)+{\overline{f}}_{12}\left(k\right)u_b(k-1)+{\overline{g}}_{11}\left(k\right)·T_x\left(x\right)\hfill \\ \hfill & +{\overline{g}}_{12}\left(k\right)·T_x(k-1)\hfill \end{array}$$

(23)

Here,

$$\left\{\begin{array}{c}{\overline{f}}_{11}\left(k\right)=2+h·{f^{\prime }}_{1,u_b}+\frac{h^2{\Delta }_u\left(k\right)u_b\left(k\right)}{u_b^2\left(k\right)+u_b^2(k-1)+T_x{\left(k\right)}^2+T_x{(k-1)}^2}\hfill \\ {\overline{f}}_{12}\left(k\right)=-1-h·{f^{\prime }}_{1,u_b}+\frac{h^2{\Delta }_u\left(k\right)u_b(k-1)}{u_b^2\left(k\right)+u_b^2(k-1)+T_x{\left(k\right)}^2+T_x{(k-1)}^2}\hfill \\ {\overline{g}}_{11}\left(k\right)=h·\frac{1}{m+m_{11}}+\frac{h^2{\Delta }_u\left(k\right)T_x\left(k\right)}{u_b^2\left(k\right)+u_b^2(k-1)+T_x{\left(k\right)}^2+T_x{(k-1)}^2}\hfill \\ {\overline{g}}_{12}\left(k\right)=-h·\frac{1}{m+m_{11}}+\frac{h^2{\Delta }_u\left(k\right)T_x(k-1)}{u_b^2\left(k\right)+u_b^2(k-1)+T_x{\left(k\right)}^2+T_x{(k-1)}^2}\hfill \end{array}\right.$$

(24)

where h is the sampling period.

4.2. The Characteristic Model Derivation of the Lateral Velocity Channel

By the same token, according to Equation (1), the lateral velocity of the airship can be derived as

$${\dot{v}}_b=f_2(u_b,v_b,r)+\frac{1}{m+m_{22}}{T}_y$$

(25)

By taking the derivative of Equation (25), we have

$${\ddot{v}}_b={f^{\prime }}_{2,v_b}·{\dot{v}}_b+{f^{\prime }}_{2,u_b}·{\dot{u}}_b+{f^{\prime }}_{2,r}·\dot{r}+\frac{1}{m+m_{22}}{\dot{T}}_y$$

(26)

Similarly, the expression of ${f^{\prime }}_{2,v_b}$ can be expressed as

$$\begin{array}{cc}\hfill f_{2,v_b}^{\prime }& =\frac{1}{m+m_{22}}{\rho }_a·v_f^{\frac{2}{3}}·v_b(C_{x}sin\beta +C_{y}cos\beta )\hfill \\ \hfill & +\frac{{\rho }_a{v}_f^{\frac{2}{3}}·u_b}{2(m+m_{22})}(C_{x}cos\beta -C_{y}sin\beta +C_{ybeta}cos\beta )\hfill \end{array}$$

(27)

Further, the expression of ${f^{\prime }}_{2,u_b}$ can be given as

$$\begin{array}{cc}\hfill {f^{\prime }}_{2,u_b}& =\frac{1}{m+m_{22}}{\rho }_a·v_f^{\frac{2}{3}}·u_b(C_{x}sin\beta +C_{y}cos\beta )\hfill \\ \hfill & +\frac{{\rho }_a{v}_f^{\frac{2}{3}}·u_b}{2(m+m_{22})}(-C_{x}cos\beta +C_{y}sin\beta -C_{ybeta}cos\beta )\hfill \\ \hfill & -\frac{m+m_{11}}{m+m_{22}}·r\hfill \end{array}$$

(28)

The expression of ${f^{\prime }}_{2,r}$ can be expressed as

$${f^{\prime }}_{2,r}=\frac{{\rho }_a{v}_f^{\frac{2}{3}}·u_b}{2(m+m_{22})})(u_b^2+v_b^2)C_{yr}cos\beta -\frac{m+m_{11}}{m+m_{22}}·u_b$$

(29)

Equation (26) can be expressed as

$${\ddot{v}}_b={f^{\prime }}_{2,v_b}·{\dot{v}}_b+\frac{1}{m+m_{22}}{\dot{T}}_y+{\Delta }_v$$

(30)

where ${\Delta }_v={f^{\prime }}_{2,u_b}·{\dot{u}}_b+{f^{\prime }}_{2,r}·\dot{r}$. The above formula after discretization, like Equation (20), can be expressed as

$$\begin{array}{cc}\hfill \frac{v_b(k+1)-2v_b\left(k\right)+v_b(k-1)}{h^2}& ={f^{\prime }}_{2,v_b}·\frac{v_b\left(k\right)-v_b(k-1)}{h}\hfill \\ \hfill & +\frac{1}{m+m_{22}}\frac{T_y\left(k\right)-T_y(k-1)}{h}\hfill \\ \hfill & +{\Delta }_v\left(k\right)\hfill \end{array}$$

(31)

Similarly, Equation (31) is written in the form of a characteristic model [36]:

$$\begin{array}{cc}\hfill v_b(k+1)& =f_{21}\left(k\right)·v_b\left(k\right)+f_{22}\left(k\right)v_b(k-1)\hfill \\ \hfill & +g_{21}\left(k\right)·T_y\left(k\right)+g_{22}\left(k\right)·T_y(k-1)+{\Delta }_v\left(k\right)\hfill \end{array}$$

(32)

For normalization, Equation (32) is written as a parameter estimation equation:

$$\begin{array}{cc}\hfill v_b(k+1)& ={\overline{f}}_{21}\left(k\right)·v_b\left(k\right)+{\overline{f}}_{22}\left(k\right)v_b(k-1)\hfill \\ \hfill & +g_{21}\left(k\right)·T_y\left(k\right)+g_{22}\left(k\right)·T_y(k-1)\hfill \end{array}$$

(33)

Here,

$$\left\{\begin{array}{c}{\overline{f}}_{21}\left(k\right)=2+h·{f^{\prime }}_{2,v_b}+\frac{h^2{\Delta }_v\left(k\right)v_b\left(k\right)}{v_b^2\left(k\right)+v_b^2(k-1)+T_y{\left(k\right)}^2+T_y{(k-1)}^2}\hfill \\ {\overline{f}}_{12}\left(k\right)=-1-h·{f^{\prime }}_{2,v_b}+\frac{h^2{\Delta }_v\left(k\right)v_b(k-1)}{v_b^2\left(k\right)+v_b^2(k-1)+T_y{\left(k\right)}^2+T_y{(k-1)}^2}\hfill \\ {\overline{g}}_{21}\left(k\right)=h·\frac{1}{m+m_{22}}+\frac{h^2{\Delta }_v\left(k\right)T_y\left(k\right)}{v_b^2\left(k\right)+v_b^2(k-1)+T_y{\left(k\right)}^2+T_y{(k-1)}^2}\hfill \\ {\overline{g}}_{22}\left(k\right)=-h·\frac{1}{m+m_{22}}+\frac{h^2{\Delta }_v\left(k\right)T_y(k-1)}{v_b^2\left(k\right)+v_b^2(k-1)+T_y{\left(k\right)}^2+T_y{(k-1)}^2}\hfill \end{array}\right.$$

(34)

where h is the sampling period.

4.3. The Characteristic Model Derivation of the Yaw Angular Velocity Channel

Analogously to forward and lateral velocity channel parameterization process, according to Equation (1), one can represent the yaw angular velocity channel as follows:

$$\dot{r}=f_3(u_b,v_b,r)+\frac{l_x}{I_z+m_{66}}{T}_y$$

(35)

where $I_z$, $l_x$ is the regularized mass about the corresponding axis. The derivative of Equation (35) can be obtained:

$$\ddot{r}={f^{\prime }}_{3,r}·\dot{r}+{f^{\prime }}_{3,u_b}·{\dot{u}}_b+{f^{\prime }}_{3,v_b}·{\dot{v}}_b+\frac{l_x}{I_z+m_{66}}{\dot{T}}_y$$

(36)

Similarly, the expression of ${f^{\prime }}_{3,r}$ can be expressed as

$${f^{\prime }}_{3,r}=\frac{1}{I_z+m_{66}}{V}_f·Q·C_{nr}$$

(37)

The expression of ${f^{\prime }}_{3,u_b}$ can be expressed as

$${f^{\prime }}_{3,u_b}=\frac{{\rho }_a·V_f·u_b}{I_z+m_{66}}-\frac{1}{2(I_z+m_{66})}{\rho }_a·V_f·C_{nbeta}·v_b$$

(38)

The expression of ${f^{\prime }}_{3,v_b}$ can be expressed as

$${f^{\prime }}_{3,v_b}=\frac{{\rho }_a·V_f·v_b}{I_z+m_{66}}+\frac{1}{2(I_z+m_{66})}{\rho }_a·V_f·C_{nbeta}·u_b$$

(39)

Finally, Equation (36) can be simplified to

$$\ddot{r}={f^{\prime }}_{3,r}·\dot{r}+\frac{l_x}{I_z+m_{66}}{\dot{T}}_y+{\Delta }_r$$

(40)

where ${\Delta }_r={f^{\prime }}_{3,u_b}·{\dot{u}}_b+{f^{\prime }}_{3,v_b}·{\dot{v}}_b$. Equation (40) after discretization can be expressed as

$$\begin{array}{cc}\hfill \frac{r(k+1)-2r\left(k\right)+r(k-1)}{h^2}& ={f^{\prime }}_{3,r}·\frac{r\left(k\right)-r(k-1)}{h}\hfill \\ \hfill & +\frac{l_x}{I_z+m_{66}}\frac{T_y\left(k\right)-T_y(k-1)}{h}\hfill \\ \hfill & +{\Delta }_r\left(k\right)\hfill \end{array}$$

(41)

The above equation is written in the form of a characteristic model [36]:

$$\begin{array}{cc}\hfill r(k+1)& =f_{31}\left(k\right)·r\left(k\right)+f_{32}\left(k\right)r(k-1)\hfill \\ \hfill & +g_{31}\left(k\right)·T_y\left(k\right)+g_{32}\left(k\right)·T_y(k-1)+{\Delta }_r\left(k\right)\hfill \end{array}$$

(42)

For normalization, Equation (42) is written as a parameter estimation equation:

$$\begin{array}{cc}\hfill r(k+1)& ={\overline{f}}_{21}\left(k\right)·r\left(k\right)+{\overline{f}}_{32}\left(k\right)r(k-1)\hfill \\ \hfill & +g_{31}\left(k\right)·T_y\left(k\right)+g_{32}\left(k\right)·T_y(k-1)\hfill \end{array}$$

(43)

Here,

$$\left\{\begin{array}{c}{\overline{f}}_{31}\left(k\right)=2+h·{f^{\prime }}_{3,r}+\frac{h^2{\Delta }_r\left(k\right)r\left(k\right)}{r^2\left(k\right)+r^2(k-1)+T_y{\left(k\right)}^2+T_y{(k-1)}^2}\hfill \\ {\overline{f}}_{32}\left(k\right)=-1-h·{f^{\prime }}_{3,v_b}+\frac{h^2{\Delta }_r\left(k\right)r(k-1)}{r^2\left(k\right)+r^2(k-1)+T_y{\left(k\right)}^2+T_y{(k-1)}^2}\hfill \\ {\overline{g}}_{31}\left(k\right)=h·\frac{l_x}{I_z+m_{66}}+\frac{h^2{\Delta }_r\left(k\right)T_y\left(k\right)}{r^2\left(k\right)+r^2(k-1)+T_y{\left(k\right)}^2+T_y{(k-1)}^2}\hfill \\ {\overline{g}}_{32}\left(k\right)=-h·\frac{l_x}{I_z+m_{66}}+\frac{h^2{\Delta }_r\left(k\right)T_y(k-1)}{r^2\left(k\right)+r^2(k-1)+T_y{\left(k\right)}^2+T_y{(k-1)}^2}\hfill \end{array}\right.$$

(44)

where h is the sampling period.

4.4. The Characteristic Model Derivation of Yaw Angle Channel

According to Equations (1) and (2), we have the following:

$$\ddot{\psi }={f^{\prime }}_{41}\dot{\psi }+{g^{\prime }}_{41}{T}_y+{\Delta }_{\psi }$$

(45)

Here,

$$\left\{\begin{array}{c}{f^{\prime }}_{41}=0\hfill \\ {g^{\prime }}_{41}=\frac{l_x}{I_z+m_{66}}\hfill \\ {\Delta }_{\psi }=\frac{l_x}{I_z+m_{66}}Q{V}_f{C}_n\hfill \end{array}\right.$$

(46)

Equation (45) can be written in the form of a characteristic model as shown above:

$$\begin{array}{cc}\hfill \psi (k+1)& =f_{41}\left(k\right)·\psi \left(k\right)+f_{42}\left(k\right)\psi (k-1)\hfill \\ \hfill & +g_{41}\left(k\right)·T_y\left(k\right)+{\Delta }_{\psi }\left(k\right)\hfill \end{array}$$

(47)

For normalization, the above equation can be written as a parameter estimation equation:

$$\psi (k+1)={\overline{f}}_{41}\left(k\right)·\psi \left(k\right)+{\overline{f}}_{42}\left(k\right)\psi (k-1)+{\overline{g}}_{41}\left(k\right)·T_y\left(k\right)$$

(48)

Here,

$$\left\{\begin{array}{c}{\overline{f}}_{41}\left(k\right)=2+h·{f^{\prime }}_{41}+\frac{h^2{\Delta }_{\psi }\left(k\right)\psi \left(k\right)}{{\psi }^2\left(k\right)+{\psi }^2(k-1)+T_y{\left(k\right)}^2}\hfill \\ {\overline{f}}_{42}\left(k\right)=-1-h·{f^{\prime }}_{41}+\frac{h^2{\Delta }_{\psi }\left(k\right)\psi \left(k\right)}{{\psi }^2\left(k\right)+{\psi }^2(k-1)+T_y{\left(k\right)}^2}\hfill \\ {\overline{g}}_{41}\left(k\right)=h^2·{g^{\prime }}_{41}+\frac{h^2{\Delta }_{\psi }\left(k\right)\psi \left(k\right)}{{\psi }^2\left(k\right)+{\psi }^2(k-1)+T_y{\left(k\right)}^2}\hfill \end{array}\right.$$

(49)

5. The Design of a Fixed-Point Control Law for an Airship

A golden section adaptive controller and a logic differential controller are proposed in [12]. In this section, we focus only on the stability issues of the airship via an adaptive method.

The golden section adaptive control law of the forward speed channel can be expressed as

$$\begin{array}{cc}\hfill u_1\left(k\right)& =\frac{-1}{{\widehat{g}}_{11}\left(k\right)+{\lambda }_1}[L_1{\widehat{f}}_{11}\left(k\right)(u_b\left(k\right)-u_c\left(k\right))\hfill \\ \hfill & +L_2{\widehat{f}}_{12}\left(k\right)(u_b(k-1)-u_c(k-1))]\hfill \end{array}$$

(50)

where $L_1=0.382$, $L_2=0.618$. ${\lambda }_1$ is a small positive number. ${\widehat{f}}_{11}\left(k\right)$, ${\widehat{f}}_{12}\left(k\right)$, and ${\widehat{g}}_{11}\left(k\right)$, respectively, represent the identification values of the characteristic model parameters. The logic differential control law of the forward speed channel can be expressed as

$$u{d}_1\left(k\right)=k_{1,d}[u_b\left(k\right)-u_c(k-1)]$$

(51)

where $k_{1,d}=c_1\sqrt{{\sum }_{i-1}^N\left|u_b(k-i)\right|}$, and $c_1$ is an adjustable parameter.

The golden section adaptive control law of the yaw angle channel can be expressed as

$$\begin{array}{cc}\hfill u_2\left(k\right)& =\frac{-1}{{\widehat{g}}_{41}\left(k\right)+{\lambda }_2}[L_1{\widehat{f}}_{41}\left(k\right)(\psi \left(k\right)-{\psi }_c\left(k\right))\hfill \\ \hfill & +L_2{\widehat{f}}_{42}\left(k\right)(\psi (k-1)-{\psi }_c(k-1))]\hfill \end{array}$$

(52)

where $L_1=0.382$,$L_2=0.618$. ${\lambda }_2$ is a small positive number. ${\widehat{f}}_{41}\left(k\right)$, ${\widehat{f}}_{42}\left(k\right)$, and ${\widehat{g}}_{41}\left(k\right)$, respectively, represent the identification values of the characteristic model parameters. The logic differential control law of the forward speed channel can be expressed as

$$u{d}_2\left(k\right)=k_{2,d}[\psi \left(k\right)-{\psi }_c(k-1)]$$

(53)

$k_{2,d}=c_2\sqrt{{\sum }_{i-1}^N\left|\psi (k-i)\right|}$. $c_2$ is an adjustable parameter.

Remark 2. 

In off-policy optimal control, to maintain system stabilization, PI (policy iteration) needs an initial controller that can stabilize the system [34,37]. Similarly, in the initial transition stage of adaptive control, when the parameter estimation has not yet converged to the “true value”, the above methods cannot easily guarantee the stability of the closed-loop system theoretically. The golden section adaptive controller restricts f to the specified parameter range. The feedback control law can guarantee the stability of the closed-loop system regardless of whether the parameter estimation converges to the “true value”. Next, we show that the stabilization of the controlled system can be guaranteed under some conditions.

Online Parameter Learning

The difference of Equation (23), taken as an example, can be transformed into

$$u_b(k+1)=U^T\left(k\right)\theta (k+1)$$

(54)

where $U\left(k\right)=\left[\begin{array}{cccc}{u}_b\left(k\right)& u_b(k-1)& T_x\left(k\right)& T_x(k-1)\end{array}\right]$, and input parameters $\theta \left(k\right)=\left[\begin{array}{cccc}{\overline{f}}_{11}\left(k\right)& {\overline{f}}_{12}(k-1)& {\overline{g}}_{11}\left(k\right)& {\overline{g}}_{12}(k-1)\end{array}\right]$. The least square method can be used to estimate the parameters of the difference equation.

Remark 3. 

The main characteristics of the system are compressed into several characteristic variables. We can obtain a characteristic model with unknown coefficients, and this depends on the basic structural characteristics of the system. The input and output variable information data are obtained through online or offline operations, and the coefficients of the characteristic model equation are obtained by using appropriate parameter estimation methods. The least-squares problem can be solved in real time after a sufficient number of data points are collected along a single state trajectory under the regular presence of an excitation requirement.

$$\left\{\begin{array}{c}K(k+1)=\frac{P\left(k\right)\psi \left(k\right)}{\rho +{\psi }^T\left(k\right)P\left(k\right)\psi \left(k\right)}\hfill \\ \widehat{\theta }(k+1)=\widehat{\theta }\left(k\right)+K(k+1)[u_b(k+1)-{\psi }^T\left(k\right)\widehat{\theta }\left(k\right)]\hfill \\ P(k+1)=\frac{1}{\rho }[I-K(k+1){\psi }^T(k+1)]P\left(k\right)\hfill \end{array}\right.$$

(55)

where $P\left(k\right)=\alpha I$, I is the identity matrix, and α is a large positive number, i.e., $0.95\le \rho \le 0.99$.

Remark 4. 

In order to ensure that the datasets are sufficiently rich (i.e., informative) and linearly independent, there is a need to add probing noise (excitation signal) to the control input in RL. However, the persistent excitation condition will lead to a bias away from true value if output feedback is employed. In practice, it is necessary to collect a sufficient amount of data samples for ensuring the existence of a recursive least squares solution, but our method obviate extra input singal.

6. Numerical Experiments

In this section, we compare our approach with a frequently chosen PID controller. Furthermore, trajectory tracking of the airship is converted to corresponding fixed point control. Numerical simulation will be performed on the Matlab platform to simulate the fixed-point hovering control of the stratosphere airship. Controller parameters are $k_s$, $k_e$, and $k_p$. In simulation, $k_s=0.01$ (when disturbance exists), $k_e=100$, and $k_p=0.1$. We select the desired hover trajectory ${\zeta }_d={[sin\left(t\right),cos\left(t\right)]}^T$, and the initial position is ${\zeta }_p={[0,0]}^T$. We define the airship initial state as $u_{b0}=1(m/s)$, $v_{b0}=1(m/s)$, and $r_0=0(rad/s)$, respectively. Wind speed is $V_{\omega }=5(m/s)$, and wind direction is ${\psi }_{\omega }=\frac{7\pi }{6}\left(rad\right)$. The reference trajectory is a set of triangular signals. The parameters and initial values of the airship are shown in

Table 1. Airship model parameters.

Parameter	Value
m	329 kg
$I_z$	5264 kg·m2
${\rho }_a$	1.29 kg/m3
$m_{11}$	16.45 kg
$m_{22}$	493.5 kg
$m_{66}$	7896 kg

and

Table 2. Airship model parameters.

Coefficients	Value
$c_x$	0.0437
$c_{y_0}$	0
$c_{ybeta}$	−1.2534
$c_{yr}$	0.644
$c_{n0}$	0
$c_{nbeta}$	0.323
$c_{nr}$	−0.1659

. In simulation results, the notion “ideal” represents ideal fixed-point (trajectory) and “Characteristic model” and “RL” denote corresponding controller tracking results.

In practice, the displacement of the airship can be measured by GPS. The traditional PD control method and the adaptive control proposed in this paper are used to compare the control effects. In the ground coordinate system, the simulation results of the airship position are shown in Figure 2 and Figure 3. The simulation result of the speed of the airship relative to the speed of the air are shown in Figure 4. According to the above research results, under the interference of wind, both the PD and the characteristic model control schemes can achieve the control targets. It can be seen from the first result shown in Figure 4 that the characteristic model control method responds slightly more slowly than the PD control method in the initial stage. After a period of time, the response speed of the characteristic model exceeds that of the PD and reaches the steady state before the PD method. As shown by Figure 5 and the second result shown in Figure 4, the overshoot amount of the characteristic model control method is smaller than that of the PD control, and the PD control has a stronger shock than the characteristic model control method. An integrated system trajectory is shown in Figure 6. Using the control method of the characteristic model, the process of the airship flying to the target point will be more stable.

We have tried our best to find the optimal parameters of the PD controller. Parameter tuning is a difficult process in practice, impossible in some cases, especially with some complex equipment. Thus, an adaptive controller is necessary for a changing environment.

7. Fix-Point Control by RL

In this section, we solve the airship fixed-point control problem using well-known adaptive dynamic programming (ADP) with the goal of achieving stability and optimality and show results, as in [15]. The constraint of the performance index on the system is the main characteristic of this algorithm (RL, ADP) [38,39]. For simplicity, we assume the plant operates in a small neighborhood of the operating point. We then linearize the original system on this fixed point. Namely, one can acquire an optimal policy online with the linear model of the original system. On the other hand, a nonlinear optimal policy can also be determined with the help of a neural network. To this end, we have the following system, which can be acquired by calculating the Jacobian matrix on the system.

$$\dot{x}=Ax+Bu$$

(56)

The design objective is to find a linear optimal control law $u=-kx$ that can stabilize the system and minimize the following performance index:

$$J(t,x_0)={\int }_t^{\infty }\left[x{\left(t\right)}^{T}Qx\left(t\right)+u{\left(t\right)}^{T}Ru\left(t\right)\right]dt$$

(57)

where $Q=Q^T\ge 0$, with $(A,\sqrt{Q})$ observable, and $R=R^T>0$.

By optimal control theory and the reinforcement learning method, one can obtain optimal gain without accessing both A and B by identifying the Lyapunov equation $A_k^T{P}_k+P_k{A}_k=-Q-K_k^{T}R{K}_k$, with a symmetric positive definite $P_k$, where $A_k=A-B{K}_k$, and a policy update equation $K_{k+1}=R^{-1}{B}^T{P}_k$ at iteration step k, which is known as policy iteration, shown in Algorithm 1. For this purpose, some datasets are collected along the single system trajectory before the system is stable. The following datasets are needed:

$$\mathsf{\Lambda }={\left[\overline{x}\left(t_1\right)-\overline{x}\left(t_0\right),\overline{x}\left(t_2\right)-\overline{x}\left(t_1\right),\cdots ,\overline{x}\left(t_l\right)-\overline{x}\left(t_{l-1}\right)\right]}^T,$$

$$\begin{array}{cc}\hfill & {\mathsf{\Phi }}_{xx}=\hfill \\ \hfill & {\left[{\int }_{t_0}^{t_1}x\left(t\right)\otimes x\left(t\right)dt,{\int }_{t_1}^{t_2}x\left(t\right)\otimes x\left(t\right)dt,\cdots ,{\int }_{t_{l-1}}^{t_l}x\left(t\right)\otimes x\left(t\right)dt\right]}^T,\hfill \end{array}$$

$$\begin{array}{cc}\hfill & {\mathsf{\Phi }}_{xu}=\hfill \\ \hfill & {\left[{\int }_{t_0}^{t_1}x\left(t\right)\otimes u\left(t\right)dt,{\int }_{t_1}^{t_2}x\left(t\right)\otimes u\left(t\right)dt,\cdots ,{\int }_{t_{l-1}}^{t_l}x\left(t\right)\otimes u\left(t\right)dt\right]}^T.\hfill \end{array}$$

where symbol ⊗ denotes the Kronecker product, and we obtain the following operators $\overline{x}={\left[x_1^2,x_1{x}_2,\cdots ,x_1{x}_n,x_2^2,x_2{x}_3,\cdots ,x_{n-1}{x}_n,x_n^2\right]}^T$.

Algorithm 1 On-Policy Iteration for Airship Control.

1: Initialization: Select any initial admissible control policy $u_0=-K_{0}x$ 2: Let $k=0$ and choose stop condition $\epsilon >0$ 3: repeat 4:  Update $P_k$ based on Equation  ${(A-B{K}_k)}^T{P}_k+P_k(A-B{K}_k)+Q+K_{{}^k}^{T}R{K}_k=0$ 5:  Update $K_{i+1}$ based on equation  $K_k=R^{-1}{B}^T{P}_{k-1}$ 6:  Update u based on Equation  $u=-K_{k}x$ 7:  $k\leftarrow k+1$ 8: until$∥P^{k+1}-P^k∥<\epsilon$ Output:  Obtain the optimal control policy $u^{*}$ and the optimal value function $V^{*}$

Enough datasets with sufficient information about system dynamics are obtained in order to determining Equation (57). It is worth pointing out that the algorithm requires data with a high quality; i.e., measuring errors will spoil the system’s robustness to some extent. When rank condition [34] is satisfied, the following key equation in RL admits a unique solution:

$$\begin{array}{cc}\hfill & x{(t+\Delta t)}^T{P}_{k}x(t+\Delta t)-x{\left(t\right)}^T{P}_{k}x\left(t\right)\hfill \\ \hfill & =-{\int }_t^{t+\Delta t}x{\left(t\right)}^T(Q+K_k^{T}R{K}_k)x\left(t\right)dt\hfill \\ \hfill & +2{\int }_t^{t+\Delta t}{[u\left(t\right)+K_{k}x\left(t\right)]}^{T}R{K}_{k+1}x\left(t\right)dt\hfill \end{array}$$

(58)

In (58), an exploration input v will generally be added in $u\left(t\right)$ so as to make the datasets linearly independent.

Remark 5. 

Exploration conditions are essential in RL to learn improved policies [12]. However, the choice of exploration noise v is not a trivial task for general reinforcement learning issues and machine learning, particularly for high-dimensional systems. There are some peculiarities in the control of the airship, such as a large volume, a special environment, communication delay, and so on, and this requires that the controller is able to handle a wide variety of problems. RL can overcome the majority of complicated control problems when we do not have a model, but some prior information is necessary. For example, the dimensions of the system matrix ($A,B$) need to be identified beforehand. Moreover, an initial stable policy may be difficult to acquire in practice.

In numerical stimulation, we compared RL policy and PD controller with our work in the uncertainty environment. The unpredicted wind speed is the main disturbance that the airship needs to reject. To this end, we will consider wind speeds that have different amplitudes and data uncertainties that are caused by sensor errors. From the perspective of practice, the considerations of these uncertainties are relatively sufficient to meet the robustness requirements of the controller. It should be noted that exploration input v can make datasets informative, but RL shows worse robustness against sensor errors under input v. Surprisingly, the data-driven strategy performs better than both the high-order model-based design and the model-based control method utilizing a low-order model, as predicted by the theory of this study. In fact, this comparison result with the low-order model-based approach is application-specific rather than universal. However, it can be argued that the data-driven approach is a reliable alternative for controller design to address many real-world issues, because even small modeling errors can have a large impact on the overall control performance. We can see that the controller based on the characteristic model outperforms the RL-based control policy and the PD controller.

The results of the simulation in the absence of uncertainty are displayed in Figure 7, Figure 8, Figure 9, Figure 10 and Figure 11. It is obvious that the characteristic model outperforms the RL and PD in terms of accuracy and output smoothness in the presence of disturbance. Figure 11 displays the comparison of the complete trajectory among PD, RL, and the proposed method. As a result of our research on adaptive control, we discovered that, while the optimal control algorithm has strong stability, its output is not smooth. This shows that the characteristic model-based control suggested in this study is superior. To verify the adaptability to different trajectories, we simulated a relatively complex trajectory (astroid $x={cos}^{3}t$, $y={sin}^{3}t$) as shown in Figure 12 and Figure 13. One can see that the proposed approach is insensitive to certain disturbances from Figure 12.

Remark 6. 

Integral reinforcement learning (IRL) plays an important role in the control theory community, as it makes it possible to obtain the optimal and stable control policy online. However, there are many integral operations in plants, such as IMU and the calculation of (58), so errors will accumulate unpredictably. Although some filter methods can be employed to improve data quality, it will inevitably put higher demands on computing power. When there are no disturbances, one can see learning convergence as shown by Figure 14 and Figure 15.

8. Conclusions

In this paper, we propose a new golden section adaptive controller based on path tracking theory. The characteristic motion model of a high-altitude airship is build and achieves control targets of fixed point hovering. Through the study, it is shown that characteristic model control achieves a superior control effect when solving the tracking control problem of a complex system, with a fast response, a small overshoot, and a high robustness, which is helpful for the further popularization and application of the characteristic model control method. This method was also compared to reinforcement learning algorithms to a relatively conservative degree and showed advantages in online running from a practical point of view. In contrast with the classic dynamic adaptive controllers, the proposed algorithm shows a superior performance. Issues such as the neural network approximation methods and the choice of a persistent excitation condition for RL will be addressed in future work.