Deep Deterministic Policy Gradient-Based Active Disturbance Rejection Controller for Quad-Rotor UAVs

Abstract

Thanks to their hovering and vertical take-off and landing abilities, quadrotor unmanned aerial vehicles (UAVs) are receiving a great deal of attention. With the diversified development of the functions of UAVs, the requirements for flight performance with higher stability and maneuverability are increasing. Aiming at parameter uncertainty and external disturbance, a deep deterministic policy gradient-based active disturbance rejection controller (DDPG-ADRC) is proposed. The total disturbances can be compensated dynamically by adjusting the controller bandwidth and the estimation of system parameters online. The tradeoff between anti-interference and rapidity can be better realized in this way compared with the traditional ADRC. The process of parameter tuning is demonstrated through the simulation results of tracking step instruction and sine sweep under ideal and disturbance conditions. Further analysis shows the proposed DDPG-ADRC has better performance.

1. Introduction

Quadrotor unmanned aerial vehicles (UAVs) have attracted attention thanks to their ability to hovering and to take off and landing vertically. Due to their under-actuated nature, quadrotors’ position control is performed by controlling the attitude angles [1]. For this reason, attitude control of quadrotors has been a hot research topic in recent years. However, quadrotors are subject to parameter uncertainty and external disturbance, which threaten flight safety and pose huge challenges to the design of controllers [2]. In addition, with the popularity of quadrotors, higher requirements are being placed on the controllers. Thus, it is urgent to design an advanced controller to improve reliability and rapidity.

In the literature, plenty of approaches have been studied for the quadrotor attitude control problem. As a classical controller, proportion integration differentiation (PID) is widely used because of its simple structure and good control effect [3,4,5]. Taybe et al. [6] developed an augmented proportion differentiation (PD) attitude controller that guarantees exponential stability. Cao et al. [7] focused on the position control of quadrotors using an inner–outer loop control structure. The outer loop generates a saturated thrust, reference roll, and pitch angles, while the inner loop is designed to follow these reference angles using a traditional PID controller.

Due to nonlinearity and disturbances, the control effect of PID is unsatisfactory. As one of the most important control techniques, sliding mode control (SMC) is able to handle nonlinear systems with external disturbances. Based on second-order SMC, Zheng et al. [8] designed a controller for a small quadrotor unmanned aerial vehicle (UAV). Xiong et al. [9] designed a highly coupled and nonlinear controller for a fully actuated UAV through a novel robust terminal sliding mode control algorithm. Nevertheless, the oscillation caused by SMC is the main obstacle restricting its application.

To achieve robust performance and stabilization, the robust $H\infty$ control method of George Zames has been widely studied [10]. Due to the uncertain nature of aircraft systems, Babar et al. [11] improved the traditional inner–outer loop strategy and adopted a robust controller for the inner control loop. Liu et al. [12] designed a distributed robust controller consisting of a position controller and an attitude controller for multiple quadrotors with nonlinearities and disturbances.

To deal with nonlinearities and disturbances, the main idea of active disturbance rejection control (ADRC) is to reduce the state feedback, whether linear or non-linear, to a cascade of integrators [13,14]. To solve the problem that UAV tracking control relies too much on mathematical modeling and the accuracy of measurements, Niu et al. [15] proposed a longitudinal pitching angle control system based on a nonlinear ADRC. Lotufo et al. [16] combined ADRC with embedded model control (EMC), relying on the disturbance rejector to bridge the gap between model and reality.

However, there are issues remaining that deserve attention [17].

• The classical controller design relies on understanding the physics of flight, and has difficulty to handling the coupling multiple loops design task. In other words, the classical one-loop-at-a-time design cannot guarantee success when more loops are added and coupled.
• Modern control techniques often require exact knowledge of models and are sensitive to parameter uncertainty and external disturbances [18]. However, different loads in each flight mission lead to uncertainty in system parameters. Meanwhile, parameters may be difficult to obtain, especially aerodynamic parameters. This sometimes leads to unstable behaviors, limiting the application of model-based controllers.
• For modern robust controllers [12], it is usually difficult to obtain the upper bounds of external disturbance and parameter uncertainty, which causes unsatisfactory performance.
• In the ADRC algorithm, the predefined bandwidth of the closed-loop system is unable to guarantee the tradeoff between robustness and transient tracking performance. Meanwhile, the estimation of parameters affects the ability of the controller to resist disturbances [14].

Aiming at the controller parameters tuning problem, many optimization algorithms have been used, including genetic algorithms (GA) [19], particle swarm optimization (PSO) [20], and grey wolf optimization (GWO) [21]. Bolandi et al. [22] used an analytical optimization method to tune a conventional PID controller for stabilization and disturbance rejection of quadrotors.

With the development of computer science and technology, reinforcement learning (RL) is able to autonomously learn optimal strategies through continuous interaction with the environment and is considered one of the most likely approaches for achieving general artificial intelligence [23]. Lee et al. [24] proposed an RL-based adaptive PID controller for dynamic positioning systems. The results showed that the system had better station-keeping performance without any deterioration in its control efficiency. Gheisarnejad et al. [25] proposed a deep deterministic policy gradient (DDPG)-based supplementary controller to enhance the adaptive capability of the tracking control problem. Zhao et al. [26] employed RL to update the optimal control weights in the fault-tolerant formation control law design. Zheng et al. [27] used the Q-learning algorithm to select the adaptive parameters for ADRC. However, as Q-learning can only deal with discrete problems, the states need to be stored in the Q table, and the action must be discrete. By itself, Q-learning cannot deal with complex continuous problems such as attitude control of UAVs. RL, which can solve the nonlinear optimal consensus control problem, is widely used in fault-tolerant control. Ma et al. [28] presented an adaptive model-free fault-tolerant control scheme based on integral RL by introducing the integral of the tracking error. Li et al. [29] designed direct adaptive optimal controllers by combining the backstepping technique with RL. The critic network is used to approximate the strategic utility functions and the action network is used to approximate the unknown and desired control input signals.

Motivated by the above discussions, ADRC based on DDPG is proposed in this paper. The main contributions of this paper are as follows:

• A realistic and nonlinear model of quadrotors is established, considering parameter uncertainty and external disturbances.
• Online continuous adjustment of the bandwidth of the closed loop is realized by DDPG, and is beneficial for balancing the robustness and transient tracking performance.
• DDPG is adopted to achieve fast and accurate compensation for the total disturbance of the system, leading to the response speed and control accuracy being further improved.

The remainder of this paper is organized as follows. In Section 2, the proposed dynamic quadrotor model with internal and external disturbances is provided. The proposed DDPG-based ADRC is presented in Section 3. The simulation results are provided and analyzed in Section 4. Finally, Section 5 presents our conclusions.

2. Nonlinear Model of Quadrotors

In this section, a nonlinear dynamic model with internal and external disturbances is provided. Figure 1 shows the structure and coordinate system of the quadrotor.

2.1. Ideal Model of Quadrotors

The ideal dynamic model of quadrotors is established in Formula (1).

$$\begin{array}{c}\hfill \begin{array}{c}m\ddot{E^I}=R_b^I{F}_b\hfill \\ J\ddot{\mathsf{\Theta }}=C(J,\dot{\mathsf{\Theta }})+M_b\hfill \end{array},\end{array}$$

(1)

where m is the quadrotor mass, $E^I={[E_x^I,E_y^I,E_z^I]}^T$ is the position expressed in the Earth-inertial coordinate, $R_b^I\in SO\left(3\right)$ denotes the rotation matrix from the body-fixed coordinate to the Earth-inertial coordinate, and $F_b={[0,0,f_t]}^T-{R_b^I}^T{[0,0,mg]}^T$ is the force established in the body-fixed coordinate, where $f_t=C_w{\sum }_{i=1}^4{w}_i^2$. $C_w$ is the lift coefficient and $w_i(i=1,2,3,4)$ denotes the rotational speed of the $ith$ rotor. Above, $J=diag\left\{J_{\varphi },J_{\theta },J_{\psi }\right\}$ denotes the inertia matrix, while $\mathsf{\Theta }={[\varphi ,\theta ,\psi ]}^T$ indicates the Euler angles, i.e., the roll, pitch, and yaw angles, respectively; thus, the rotation matrix can be rewritten using the Euler angles [12].

$$R_b^I=\left[\begin{array}{ccc}cos\theta cos\psi & cos\psi sin\varphi sin\theta -cos\varphi sin\psi & sin\varphi sin\psi +cos\varphi cos\psi sin\theta \\ cos\theta sin\psi & cos\varphi cos\psi +sin\varphi sin\theta sin\psi & cos\varphi sin\theta sin\psi -cos\psi sin\varphi \\ -sin\theta & cos\theta sin\varphi & cos\varphi cos\theta \end{array}\right]$$

$C(J,\dot{\mathsf{\Theta }})$ denotes the Coriolis term, where $C(J,\dot{\mathsf{\Theta }})=\left[\begin{array}{c}(J_y-J_z)\dot{\theta }\dot{\psi }\\ (J_z-J_x)\dot{\varphi }\dot{\psi }\\ (J_x-J_y)\dot{\varphi }\dot{\theta }\end{array}\right]$;

$M_b={[M_{xb},M_{yb},M_{zb}]}^T$ represents the torque in the body-fixed coordinate

$$\left[\begin{array}{c}{M}_{xb}\\ M_{yb}\\ M_{zb}\end{array}\right]=\left[\begin{array}{c}{U}_2+J_{r}q(-w_1+w_2-w_3+w_4)\\ U_3-J_{r}q(-w_1+w_2-w_3+w_4)\\ U_4\end{array}\right],$$

where $\left[\begin{array}{c}{U}_2\\ U_3\\ U_4\end{array}\right]=\left[\begin{array}{c}l(F_2+F_3-F_1-F_4)\\ l(F_3+F_4-F_1-F_2)\\ ld(w_2^2+w_4^2-w_1^2-w_3^2)\end{array}\right]$. $l_c$ and d represent the distance from the motor to the center of mass and the anti-torque coefficient, respectively, while $J_r$ is the moment of inertia of the motors and propellers. For now, the normal model of quadrotors has been established.

2.2. Internal and External Disturbances

Quadrotors usually carry various mission payloads to perform different missions, resulting in changes in parameters such as mass or moment of inertia. This can be modeled as $m^{*}=k_{m}m$ and $J^{*}=k_{J}J$, where $m^{*}$ and $J^{*}$ are the actual mass and inertia matrix, respectively, and $k_m$ and $k_J$ are the scaling factors of uncertainty. At the same time, quadrotors are inevitably disturbed by the environment, $M_e$.

Thus, the actual dynamic model of quadrotors is expressed as follows:

$$\begin{array}{c}\hfill \begin{array}{c}{m}^{*}\ddot{E^I}=R_b^I{F}_b\hfill \\ J^{*}\ddot{\mathsf{\Theta }}=C^{*}(\mathsf{\Theta },\dot{\mathsf{\Theta }})+M_b+M_e\hfill \end{array}\end{array}$$

(2)

3. Construction of DDPG-Based ADRC

3.1. ADRC-Based Attitude Controller Design

Only the attitude control of quadrotors is considered here, and the construction of ADRC is designed as in Figure 2. To facilitate the control system design, the quadrotor is reduced to a second-order system with perturbations, which can be written in state equation form:

$$\begin{array}{c}\hfill \begin{array}{c}\dot{x}=Ax+Bu+Eh\hfill \\ y=Cx\hfill \end{array}\end{array}$$

(3)

where

$$A=\left[\begin{array}{ccc}0& 1& 0\\ 0& 0& 1\\ 0& 0& 0\end{array}\right],B=\left[\begin{array}{c}0\\ b\\ 0\end{array}\right],C=\left[\begin{array}{ccc}1& 0& 0\end{array}\right].$$

h is the unknown disturbance, and $E={\left[\begin{array}{ccc}0& 0& 1\end{array}\right]}^T$.

Figure 2. Structure of ADRC; $\overline{\theta }$ is the desired angle of pitch and b is the estimation of system parameters.

Figure 2. Structure of ADRC; $\overline{\theta }$ is the desired angle of pitch and b is the estimation of system parameters.

The extended state observer (ESO) is designed from the ideal model of the quadrotor, which can be established as follows:

$$\begin{array}{c}\hfill \begin{array}{c}\dot{z}=Az+Bu+L(y-\widehat{y})\hfill \\ \widehat{y}=Cz\hfill \end{array}\end{array}$$

(4)

where y is the state of the system and L is the observer gain vector, $L={\left[\begin{array}{ccc}{\beta }_1& {\beta }_2& {\beta }_3\end{array}\right]}^T$.

Let $e_i=x_i-z_i$ and combine Equations (3) and (4); then, the error can be rewritten as

$$\begin{array}{c}\hfill \dot{e}=A_{e}e+Eh\end{array}$$

(5)

where $A_e=A-LC=\left[\begin{array}{ccc}-{\beta }_1& 1& 0\\ -{\beta }_2& 0& 1\\ -{\beta }_3& 0& 0\end{array}\right]$.

It is obvious that the ESO is bounded-input bounded-output stable if the roots of the characteristic polynomial of $A_e$

$$\lambda \left(s\right)=s^3+{\beta }_1{s}^2+{\beta }_{2}s+{\beta }_3$$

are all in the left half plane and h is bounded [14,30].

Thus, ${\beta }_1$, ${\beta }_2$, ${\beta }_3$ can be designed using the pole placement technique. Let $\lambda \left(s\right)=s^3+{\beta }_1{s}^2+{\beta }_{2}s+{\beta }_3={(s+w_o)}^3$. Therefore, it can be obtained that

$$\begin{array}{c}\hfill {\beta }_1=3w_0,{\beta }_2=3w_o^2,{\beta }_3=w_o^3,\end{array}$$

(6)

where $w_o$ is the bandwidth of the observer.

For the controller, the ideal system can be written as $\ddot{y}=x_3+bu$, where $\dot{x_3}=h$. According to the proof above, the appropriate value (6) can make $e\to 0$; in other words, $z_1\to x_1,z_2\to x_2,z_3\to x_3$.

The controller is designed as $u=\frac{-z_3+u_0}{b}$. Thus, $\ddot{y}=x_3+bu=(x_3-z_3)+u_0\approx u_0$, where $u_0$ is designed as a PD controller, $u_0=k_p(r-z_1)+k_d(\dot{r}-z_2)$. It can be assumed that $\dot{r}=0$. Then, $\ddot{y}=k_p(r-z_1)-k_d{z}_2$ and the closed loop transfer function can be rewritten as

$$G_{cl}=\frac{k_p}{s^2+k_{d}s+k_p}.$$

When $K_p=w_c^2,k_d=2\xi w_c$, the closed-loop system is simplified into a standard second-order system.

Taken together, the effectiveness of the involved ESO and controller is demonstrated. Normally, $w_o\approx 5\sim 10w_c$, where $w_c$ is the bandwidth of the controller. In this paper,

$$\begin{array}{c}\hfill w_o=5w_c.\end{array}$$

(7)

However, as described in Section 2.2, when there are internal disturbances, B = ${\left[\begin{array}{ccc}0& b& 0\end{array}\right]}^T$ in Formula (3) turns into $B_0={\left[\begin{array}{ccc}0& b_0& 0\end{array}\right]}^T$. The difference between b and $b_0$ reduces the robustness of the system. The presence of the observer in ADRC allows the total disturbances to be observed, which means that $z_3\to x_3+(b-b_0)u$. Then, the internal disturbance can be compensated for.

In practice, a step signal has a great impact on the system. To balance the contradiction between increasing the rapidity of the system and reducing the overshoot, a tracking differentiator (TD) can be adopted to track the desired signal, and a smooth tracking signal can be obtained and further used in the controller. In this paper, a standard second-order system is designed,

$$G_{TD}=\frac{w_n^2}{s^2+2\xi w_{n}s+w_n^2},$$

where $w_n=20$ is the natural frequency and $\xi =1$ is the damping ration.

3.2. Reinforcement Learning Theory

In this paper, the attitude control is regarded as a Markov decision process (MDP), which can be modeled as $(S,A,T,R,\gamma )$, where S represents the state space, A is the action space, $T:T(s_{t+1}=s^{{}^{\prime }}|s_t=s,a_t=a)$ is the state transition model, $R:S\times A\to R$ signifies the reward function, and $\gamma$ is the discount factor. The MDP means that, at every timestep, the agent in state $s_t$ takes action $a_t$, reward $r_t$ is obtained, and the state is transited to $s_{t+1}$. A generic flowchart of the process is shown in Figure 3.

RL discusses how an agent can maximize its rewards in a complex and uncertain environment. The goal is to learn an optimal policy ${\pi }^{*}$, which in all states enables the agent to obtain the maximum discount return $G_t={\sum }_{i=t}^T{\gamma }^{i-t}r(s_i,a_i);\gamma \in [0,1]$; the action-value function is called Q function, and can be rewritten using the Bellman equation:

$$\begin{array}{c}\hfill Q_{\pi }(s_t,a_t)={\mathbb{E}}_{r_t,s_{t+1}\sim E}[r(s_t,a_t)+\gamma {\mathbb{E}}_{a_{t+1}\sim \pi }\left[Q_{\pi }(s_{t+1},a_{a+1})\right]],\end{array}$$

(8)

where policy $\pi$ maps a state $s_t$ to action $a_t$, which can be learned by an off-policy learning algorithm called Q-learning [31].

The strategy used in this paper is the DDPG algorithm, which is an extension of the deep Q network (DQN). A model-free algorithm that is able to operate over continuous action spaces has previously been presented in [32] based on the deterministic policy gradient. The structure is shown in Figure 4.

Such a structure is called Actor–Critic. The policy network, which is called the Actor, outputs actions based on states $a=g_{\vartheta }\left(S\right)$. The Q network is employed to export the action value $q_w(s,a)$, which is named Critic, and a replay buffer is used to eliminate correlations between inputs. Compared with the Actor network, the Critic network usually has a more complex structure to infer the underlying state from the measurements and deal with the state transition [33].

3.3. Structure of DDPG-Based ADRC

As a feedback-based controller, the inputs of DDPG-ADRC include the control command and the tracking error. Then, the outputs of DDPG are used to update the parameters in ADRC, which means the estimation of system parameters b and the bandwidth of the controller $w_c$. On one hand, the parameter b reflects the gain from the input to the output of the system, which is related by the system parameters. On the other hand, to compensate for the total disturbance, $-\frac{z_3}{b}$ is added to the PD controller, which means that b affects the compensation for disturbances. Meanwhile, the bandwidth of the controller, $w_c$, directly determines the performance of the PD controller, and the bandwidth of the observer, $w_o$, determines the performance of the ESO, where $w_o=5w_c$. An overall structure of the proposed fault-tolerant controller is shown in Figure 5.

The reward function is a key element in RL, supervising agents to learn and obtain the optimal policy. In order to solve the difficulty of training caused by sparse rewards, the reward function is designed as follows:

$$\begin{array}{c}\hfill R_1=-\sqrt{{(\overline{\varphi }-\varphi )}^2+{(\overline{\theta }-\theta )}^2+{(\overline{\psi }-\psi )}^2},\end{array}$$

(9)

To solve the problem of slow convergence under small errors, step rewards are designed.

$$\begin{array}{c}\hfill R_2=\left\{\begin{array}{cc}{R}_1+5& if|\overline{\varphi }-\varphi |,|\overline{\theta }-\theta |,|\overline{\psi }-\psi |\le 0.5^{\circ }\\ R_1+3& elseif|\overline{\varphi }-\varphi |,|\overline{\theta }-\theta |,|\overline{\psi }-\psi |\le 1^{\circ }\end{array}\right.\end{array}$$

(10)

At the same time, a sparse penalty function is considered. When the attitude of the agent is too far away from the target, the current round of training is terminated in advance. To reduce ineffective exploration, a large penalty is introduced. Thus, the total reward is

$$\begin{array}{c}\hfill R=\left\{\begin{array}{cc}{R}_2-1000& ifthetrainisterminatedinadvance\\ R_2& else\end{array}\right.\end{array}$$

(11)

Above all, the algorithm flow presented in this paper is shown in Algorithm 1.

The state input of DDPG is a two-dimensional vector, namely, control command, $\overline{\theta }$, and tracking error, e. The action output is a two-dimensional variable, i.e., b and $w_c$.

Algorithm 1 DDPG-based ADRC controller

Randomly initialize Q network $q_w$ and policy network $g_{\vartheta }$ parameters Initialize the target network parameters $q_w^t$ and $g_{\vartheta }^t$ Initialize the experience pool for$episode=1,2\cdots N$do    Random initialization of control command and initial state    for $i=1,2\cdots T$ do      State $s_t$ is obtained      Select the action based on the current state and exploration noise $a_t=g_{\vartheta }\left(s_t\right)+{\xi }_i$      Perform the action $a_t$,observe the return $r_t$, get the next state $s_{t+1}$      Put the sample $(s_t,a_t,r_t,s_{t+1})$ in the experience pool D      Sample random mini-batch of $(s_t,a_t,r_t,s_{t+1})$ from D      Optimize critic network parameters w:      $Loss=MSE[q_w(s,a),r+\gamma q_{\overline{w}}(s^{\prime },a^{\prime })]$      Optimize actor network parameters $\theta$:      $Loss=-q_w(s,a)$      Every C steps update $\overline{w},\overline{\theta }$:      $\overline{w}=\tau w+(1-\tau )\overline{w}$,$\overline{\theta }=\tau \theta +(1-\tau )\overline{\theta }$    end for end for

4. Simulation and Results

To verify the effectiveness of the proposed controller, simulations with ideal conditions and under conditions of internal and external disturbance are presented. The parameters of the quadrotor are shown in

Table 1. Quadrotor model parameters.

Variable	Value	Measuring Unit
mass	$m=1.4$	kg
acceleration of gravity	$g=9.8$	m/s2
moment of inertia $J_{xx}$ and $J_{yy}$	$J_{xx}=J_{yy}=0.01724$	kg·m2
radius of the quadrotor	$r=0.24$	m
thrust coefficient $C_T=T_p/w^2$	$C_T=1.227\times {10}^{-5}$	N/(rad/s)${}^2$
moment coefficient $C_M=M_p/w^2$	$C_M=2.215\times {10}^{-7}$	N·m/(rad/s)${}^2$
moment of inertia of motor and propeller $J_r$	$J_r=2.13\times {10}^{-4}$	kg·m2

.

An Intel Xeon(R)W-2123 CPU @ 3.60 GHz, NVIDIA GeForce RTX1080Ti GPU, and Windows 10 64 bit were used in the experiments. To evaluate the performance of the proposed method several common evaluation indicators were adopted, such as integrated time and absolute error (ITAE), integrated time and square error (ITSE), and integrated absolute error (IAE).

$$\begin{array}{c}\hfill \begin{array}{c}\mathrm{ITAE}={\int }_{t_0}^{t_f}t\left|e\left(t\right)\right|dt\\ \mathrm{ITSE}={\int }_{t_0}^{t_f}t{e}^2\left(t\right)dt\\ \mathrm{IAE}={\int }_{t_0}^{t_f}\left|e\left(t\right)\right|dt\end{array}\end{array}$$

(12)

These indicators take into account both the control accuracy and convergence speed; smaller values indicate better controller performance.

4.1. Simulations in the Presence of Internal Disturbances

In order to verify the effectiveness of the proposed DDPG-ADRC method, simulations under internal disturbance conditions are presented. The pitch channel of the quadrotor tracks a step command of ${10}^{\circ }$, and the command reduces to $8^{\circ }$ 2 s later. The response is shown in Figure 6a, and the outputs of RL, i.e, the bandwidth of the controller $w_c$ and the estimation of the system b are displayed in Figure 6b.

It can be seen that with DDPG-ADRC the quadrotor can accurately track the instruction. In addition, the controller bandwidth $w_c$ and the system parameter b can be adaptively adjusted according to the observations.

In order to demonstrate the advantages of dynamic parameter adjustment, the steady results are used as fixed parameters, i.e, $w_c=22.2,b=12$. Figure 7a shows the response of the system and Figure 7b reveals the differences between traditional ADRC and DDPG-ADRC in compensating for total disturbances.

By dynamically adjusting parameters, DDPG-ADRC can compensate for disturbances more accurately and quickly, which is the advantage of DDPG-ADRC compared to traditional ADRC.

In order to explore the influence of parameter uncertainty on controllers, simulations were carried out with different parameter estimates b; the results are shown in Figure 8a, and are compared with model predictive control (MPC), shown in Figure 8b. In the design of the MPC controller, the same second-order system with a gain b is used. ITAE, ITSE, and IAE are adopted to evaluate the tracking process, and are shown in

Table 2. Evaluation indicators of MPC, traditional ADRC, and DDPG-ADRC.

Parameter and Indicator	MPC				Traditional ADRC				DDPG ADRC
b	12	13	14	15	12	13	14	15	/
ITAE	9.5016	13.047	10.629	14.049	9.4512	9.4512	9.4512	9.4512	9.357
ITSE	0.0664	0.2393	0.2504	0.2781	0.19384	0.19337	0.19293	0.19252	0.1848
IAE	13.895	22.555	20.976	24.614	18.868	18.868	18.868	18.868	18.273
Rewards	24,470	22,230	22,010	21,459	21,943	21,949	21,956	21,966	22,010

.

It can be seen from Table 2 and Figure 8 that, under nominal conditions, all three controllers can achieve satisfactory control effect. Meanwhile, with the selection of appropriate parameters, with lower ITAE, ITSE, and IAE and higher rewards, the control effect of MPC is the best. However, MPC is less robust against parameter uncertainty compared with ADRC.

4.2. Simulations in the Presence of External Disturbances

In order to verify the performance of the proposed controller in the face of external disturbances, disturbance torque caused by windblast was added in the simulation time 1∼1.5 s. Figure 9a shows the control instruction and the response, and the action of RL is displayed in Figure 9b.

It can be seen that the designed DDPG-ADRC can respond in time when faced with external disturbances. The performance is compared with the traditional ADRC in Figure 10. The evaluation indicators are shown in

Table 3. Evaluation indicators of traditional ADRC and DDPG-ADRC in the presence of disturbances.

Evaluation Indicator	Traditional ADRC	DDPG-ADRC
ITAE	11.971	11.87
ITSE	0.2064	0.1848
IAE	20.73	20.131
Total rewards	21,829	21,904

.

Compared with the case of internal disturbances shown in Table 3, external disturbances have a greater affect on controller performance, although both traditional ADRC and DDPG-ADRC can counteract the disturbances in time. Similarly, the performance of DDPG-ADRC is more prominent in both control accuracy and convergence speed. Under the ITSE indicator, DDPG-ADRC is improved by $10.4\%$ compared to ADRC in the presence of external disturbances. This means that DDPG-ADRC can achieve better performance than ADRC with fixed parameters in practice, which is demonstrated in Figure 11a as well. Although the MPC has better control performance under nominal conditions, it diverges when there are large external disturbances, as Figure 11b shows.

4.3. Simulation under Sine Sweep

In designing a control system, in order to know the response of the system under different frequency commands it is necessary to carry out frequency sweep experiments. A sine sweep is often used to measure the time-frequency characteristics of the system. Figure 12 shows the control instruction and response, while the evaluation indicators are shown in

Table 4. Evaluation indicators of MPC, ADRC, and DDPG-ADRC.

Parameter and Indicator	MPC				Traditional ADRC				DDPG ADRC
b	12	13	14	15	12	13	14	15	/
ITAE	375.85	361.27	351.32	344.28	288.38	288.62	288.89	289.18	286.3
ITSE	41.985	38.145	35.623	33.911	23.772	23.811	23.856	23.905	23.456
IAE	192.85	188.61	186.37	185.33	155.2	155.25	155.31	155.38	154.25
Rewards	−19,203	−18,821	−18,513	−18,267	−16,996	−17,027	−17,071	−17,104	−16,946

.

It can be seen from Figure 12 and Table 4 that the uncertainty of the parameters affects the control effect of MPC; generally speaking, ADRC has better performance and lower phase delay in the high-frequency bound than MPC. Meanwhile, from the point of view of indicators, DDPG-ADRC has stronger tracking ability thanks to the adaptive adjustment of compensation.

5. Conclusions

In this paper, a novel DDPG-based ADRC is proposed for the attitude control of quadrotors. First, a nonlinear mathematical model of quadrotors with internal disturbance and external disturbance is established. Then, by properly setting the reward function, online continuous adjustment of the bandwidth is realized to balance the robustness and transient tracking performance. Meanwhile, fast and accurate compensation for the total disturbance is achieved, further improving the response speed and control accuracy. Simulation results show that DDPG-ADRC has advantages on all indicators; in other words, it has advantages in terms of both control accuracy and convergence speed. This paper provides a new solution to the attitude control of quadrotors in the presence of disturbances. In the future, the proposed controller will be used to conduct hardware-in-the-loop simulation experiments to further verify the stability of the algorithm. However, the gap between the simulation and the real world presents additional challenges, such as the oscillation of the controller [34].