Deep Reinforcement Learning Based Active Disturbance Rejection Control for ROV Position and Attitude Control

Abstract

Remotely operated vehicles (ROVs) face challenges in achieving optimal trajectory tracking performance during underwater movement due to external disturbances and parameter uncertainties. To address this issue, this paper proposes a position and attitude control strategy for underwater robots based on a reinforcement learning active disturbance rejection controller. The linear active disturbance rejection controller has achieved satisfactory results in the field of underwater robot control. However, fixed-parameter controllers cannot achieve optimal control performance for the controlled object. Therefore, further exploration of the adaptive capability of control parameters based on the linear active disturbance rejection controller was conducted. The deep deterministic policy gradient (DDPG) algorithm was used to optimize the linear extended state observer (LESO). This strategy employs deep neural networks to adjust the LESO parameters online based on measured states, allowing for more accurate estimation of model uncertainties and environmental disturbances, and compensating the total disturbance into the control input online, resulting in better disturbance estimation and control performance. Simulation results show that the proposed control scheme, compared to PID and fixed parameter LADRC, as well as the double closed-loop sliding mode control method based on nonlinear observers (NESO-DSMC), significantly improves the disturbance estimation accuracy of the linear active disturbance rejection controller, leading to higher control precision and stronger robustness, thus demonstrating the effectiveness of the proposed control strategy.

1. Introduction

Remotely operated vehicles (ROVs) play an important role in underwater inspection, marine salvage, and deep-sea mining. Currently, the requirements for the control accuracy and robustness of underwater robots are also increasing. However, due to the nonlinear dynamics, external disturbances, and parameter uncertainties present in the underwater movement of ROVs, designing a reliable tracking controller is challenging. In recent decades, an increasing number of scholars have conducted extensive research on the control stability of ROVs. The control methods they used are elaborated upon below.

Proportional-integral-derivative (PID) control is the most widely used approach in industrial control. Guerrero et al. proposed a saturation function-based nonlinear PID controller, effectively addressing the control instability issues in underwater vehicles caused by actuator saturation and complex environmental disturbances [1]. Sarhadi et al. proposed a model reference adaptive PID control structure with an anti-saturation compensator to address the issue of model uncertainty in autonomous underwater vehicle systems [2].

Fuzzy control is a control method similar to expert systems. Han et al. proposed a fuzzy logic system to address the issue of unknown inertia matrices in AUV systems [3]. Li et al. proposed a fuzzy adaptive controller that considers the dynamics of ROV thrusters to improve the trajectory tracking performance of work-class ROVs, using a fuzzy adaptive control algorithm to compensate for changes in system parameters and disturbances [4]. Yang et al. proposed a fuzzy logic system (FLS) to replace the discontinuous switching terms in CSMC to reduce chattering phenomena [5].

Sliding Mode Control (SMC) is often used for trajectory tracking of underwater robots due to its resistance to external disturbances and parameter variations. Cheng et al. proposed a method that combines a finite-time observer with adaptive sliding mode control to achieve high-precision robust tracking of underwater vehicles [6]. Long et al. proposed an Adaptive Sliding Mode Control (ASMC) to construct a dynamic controller that calculates the optimal force and torque based on the output virtual speed. This approach is robust to parameter uncertainties and addresses the issue of flutter [7]. Luo et al. proposed an improved sliding surface has been proposed to address the finite selection problem of exponential parameters, resulting in a controller with better robustness [8]. Huang et al. introduced a double closed-loop sliding mode controller for trajectory tracking control of working-class ROVs, which uses the arctangent function as the switching function of the controller, effectively reducing chattering phenomena [9].

Neural network control (NNC) has emerged as a potent tool for crafting controllers for nonlinear and uncertain systems. Wen et al. proposed a predefined time control strategy using Radial Basis Function Neural Networks (RBFNNs), which effectively approximates external disturbances, thereby enhancing the robustness of the system [10]. Chu et al. proposed an adaptive control scheme based on radial basis function (RBF) neural networks for ROV trajectory tracking. To ensure system stability under actuator saturation, a first-order auxiliary state system was constructed [11]. Shojaei et al. proposed a neural network-enhanced feedback linearization control framework, which effectively addresses the performance guarantee issues of underactuated AUVs under model uncertainties and disturbances [12].

Each of the methodologies mentioned above has specific limitations. As the complexity of the Remotely Operated Vehicle (ROV) model increases, the effectiveness of proportional-integral-derivative (PID) control decreases significantly. Fuzzy control heavily relies on a fuzzy rule base constructed from expert experience, while SMC (Sliding Mode Control) has a very high dependency on the model and is prone to high-frequency chattering issues [13,14]. Neural network control (NNC) is particularly influenced by the number of nodes in the neural network; while increasing the nodes can improve control accuracy, it also leads to a significant rise in computational complexity, posing challenges for practical engineering applications [15]. Furthermore, these approaches do not adequately address the constraints related to the ROV’s state, potentially compromising tracking precision and risking damage to the thrusters.

To overcome the limitations associated with the status of the Remotely Operated Vehicle (ROV), this research employs Linear Active Disturbance Rejection Control (LADRC), an optimal control approach. LADRC preserves the essential characteristics of the proportional-integral-derivative (PID) algorithm without requiring an accurate model of the controlled system [16]. Instead, it treats the unmodeled dynamics and external disturbances as “total disturbances,” which are then estimated and compensated for. This methodology offers significant advantages in engineering applications due to its ease of use and robust resistance to interference [17,18]. Zhao et al. introduced a trajectory tracking control method for a dual-joint robotic arm system, integrating an extended state observer to estimate both the disturbances and states of the system. Additionally, they applied a state error feedback controller, and experimental findings indicate that the proposed control approach effectively meets control requirements under various conditions, including low-frequency, high-frequency, load, and disturbance scenarios [19,20]. Despite the successful implementation of LADRC in nonlinear systems, its control effectiveness is limited by its fixed structure and parameters.

The adaptive adjustment of parameters for control methods has been a prominent subject of interest, with various optimization algorithms being utilized to improve the robustness and control efficacy of these methods [21,22]. Different operational contexts necessitate varying control parameters, posing challenges for controllers with fixed optimization parameters to achieve optimal control performance. Drawing inspiration from artificial intelligence technologies, reinforcement learning (RL) algorithms have been amalgamated with control theory to devise novel control strategies that augment the adaptability of control systems and uphold optimal control performance in real time. Chen introduced a Q-learning-based adaptive tuning technique for LADRC parameters [23], which identifies optimal control parameters through iterative updates of the Q-value table and applies it to the heading control of ships. Nevertheless, the Q-learning algorithm mandates manual partitioning of the states of the controlled object and the specification of discrete actions, rendering it arduous to train and learn efficiently as the number of states and specified actions escalates. Furthermore, due to the discrete actions specified, the controller parameters can only assume predetermined values rather than varying continuously, thereby constraining the controller’s flexibility. To solve this problem [24], this study employs the Deep Deterministic Policy Gradient (DDPG) RL algorithm to dynamically generate optimal control gains online for the designed LADRC within the Linear Extended State Observer (LESO), thereby determining the optimal parameters of the extended state observer under diverse unknown disturbance conditions. This methodology circumvents the issue of inaccurate disturbance estimation stemming from fixed parameters, and the efficacy of the algorithm is ultimately corroborated through simulations.

The main contributions of this article include:

• This article presents a nonlinear model for underwater robots that considers parameter uncertainty in the dynamic model. It also proposes a linear active disturbance rejection controller for controlling the position and attitude of the underwater robot based on this model. The convergence of the extended state observer in the active disturbance rejection controller and the stability of the closed-loop control system are demonstrated using the Lyapunov method.
• A novel control method, named DDPG-LADRC, has been introduced to address disturbances in linear systems by integrating the Deep Deterministic Policy Gradient (DDPG) algorithm with an active disturbance rejection control approach. This method focuses on optimizing the extended state observer through the DDPG algorithm, enabling the observer to sustain optimal performance under varying external disturbances during both position and attitude control of Remotely Operated Vehicles (ROVs). Through real-time adjustments of control parameters, the performance of the extended state observer (LESO) is enhanced, thereby improving the system’s resilience to disturbances and enhancing control accuracy in intricate underwater settings.
• Based on a nonlinear underwater robot model, numerical simulations have confirmed the efficacy of the approach. The simulation results first compared three control algorithms: PID, fixed-parameter LADRC, and DDPG-LADRC, and finally included NESO-DSMC for comparison. Through analysis, the proposed method has been verified to have significant advantages in terms of transient performance, control accuracy, and robustness.

The subsequent sections of this article are structured as follows: Section 2 introduces a nonlinear model for the underwater robot, Section 3 outlines the design of the LADRC controller for the robot, and Section 4 introduces the DDPG-LADRC algorithm and analyzes the convergence of the extended state observer and the stability of the closed-loop control. Section 5 presents the results and analysis of numerical simulations, and lastly, a conclusion is offered.

2. Dynamics Model of a Robot

This section investigates the dynamic model of an ROV for offshore underwater structure marine growth cleaning and structural inspection independently developed by the Hadal Science and Technology Center of Shanghai Ocean University. The robot is equipped with eight thrusters, allowing it to execute three-dimensional spatial maneuvers. Figure 1 shows the coordinate system of the robot and defines the inertial coordinate system ($x_0, y_0,z_0$) and the motion coordinate system ($x, y,z$). The state variables relative to the motion coordinate system are represented as $V=[u,v,w,p,q,r{]}^T$, where $u,v,w$ represent linear velocity, and $p,q,r$ represent angular velocity. The state variables relative to the inertial coordinate system are represented as $[x,y,z,\varphi ,\theta ,\psi {]}^T$, where $x,y,z$ indicate the position of the ROV, and $\varphi ,\theta ,\psi$ represent the attitude of the ROV. The kinematic equations of the ROV can be expressed as $\dot{\eta }=J\left(\eta \right)\nu$ [25]. The roll and pitch are passively stabilized by the buoyancy system, requiring no active control, so $\varphi =\theta =0$. Therefore, the six degrees of freedom motion of the ROV can be simplified to four degrees of freedom motion.

The dynamic equations in the Remotely Operated Vehicle (ROV) motion coordinate system are presented below [25]:

$$M\dot{\nu }+C\left(v\right)v+D\left(v\right)v+g\left(\eta \right)=\tau +∆f$$

(1)

In the dynamics Equation (1) of the ROV, $M$ represents the inertia matrix of the ROV, $v$ represents the linear velocities and angular velocities of the ROV, $\dot{\nu }$ represents the linear and angular accelerations of the ROV, $C\left(v\right)$ represents the Coriolis-centrifugal matrix, $D\left(v\right)$ represents the hydrodynamic damping matrix, $g\left(\eta \right)$ represents the restoring force vector, $\tau$ is the control input provided by the main thrust and torque from the ROV’s thrusters, and $∆f$ represents external water flow disturbances and uncertainties such as unmodeled dynamics. The movement of an ROV can be conceptualized as the general motion of a rigid body influenced by gravity and hydrodynamics in a water flow. Usually, the fluid dynamics parameters are derived from experiments or fluid simulations. However, due to the complexity of real-world ocean conditions, ensuring the accuracy of these parameters is a significant challenge. Consequently, the parameters $M$, $C\left(v\right)$, and $D\left(v\right)$ in the equation remain indeterminate. The $M,C\left(v\right),D\left(v\right)$ in Equation (1) can be represented as the combination of the nominal parameters $M_0,C_0\left(\nu \right),D_0\left(\nu \right)$ and the dynamic uncertainties $∆M_0, ∆C_0\left(v\right), ∆D_0\left(v\right)$, as follows:

$$\left\{\begin{array}{c}M=M_0+∆M\hfill \\ C\left(v\right)=C_0\left(v\right)+∆C\left(v\right)\hfill \\ D\left(v\right)=D_0\left(v\right)+∆D\left(v\right)\hfill \end{array}\right.$$

(2)

Then, Equation (1) can be rewritten as:

$$M_0\dot{\nu }+C_0\left(v\right)v+D_0\left(v\right)v+g\left(\eta \right)=\tau +∆f+{\tau }_{\Delta }$$

(3)

$$\left.M_{RB}=\left[\begin{array}{cccc}m& 0& 0& -m{y}_G\\ 0& m& 0& m{x}_G\\ 0& 0& m& 0\\ -m{y}_G& m{x}_G& 0& I_z\end{array}\right.\right] M_A=-\left[\begin{array}{cccc}{X}_{\dot{u}}& X_{\dot{v}}& X_{\dot{w}}& X_{\dot{r}}\\ Y_{\dot{u}}& Y_{\dot{v}}& Y_{\dot{w}}& Y_{\dot{r}}\\ Z_{\dot{u}}& Z_{\dot{v}}& Z_{\dot{w}}& Z_{\dot{r}}\\ N_{\dot{u}}& N_{\dot{v}}& N_{\dot{w}}& N_{\dot{r}}\end{array}\right]$$

(4)

By assuming that the origin $O$ of the dynamic system coincides with the centroid $G$ and that the coordinate axes coincide with the three principal inertia axes, and ignoring the off-diagonal elements in the $M_{RB}$ and $M_A$ matrices [25], we can obtain the $M_0$ matrix:

$$M_0=M_{RB}+M_A=\left[\begin{array}{cccc}m-X_{\dot{u}}& 0& 0& 0\\ 0& m-Y_{\dot{v}}& 0& 0\\ 0& 0& m-Z_{\dot{w}}& 0\\ 0& 0& 0& I_Z-N_{\dot{r}}\end{array}\right]$$

(5)

In Formula (5), $M_0\in R^{4\times 4}$ represents the inertia matrix, which is composed of the sum of the rigid body mass matrix $M_{RB}$ and the added mass matrix, $M_A$. $m$ is the mass of the ROV, $I$ is the moment of inertia, and $X_{\dot{u}},Y_{\dot{v}},Z_{\dot{w}}$ represent the hydrodynamic forces induced by the added mass in the $x$, $y$, and $z$ directions, respectively, with unit acceleration along the $\dot{u}, \dot{v},$ and $\dot{w}$ axes. ${ N}_{\dot{r}}$ represents the additional inertial force generated by the unit angular acceleration $\dot{r}$ in the direction of the z-axis.

$$\begin{gathered} \left.C_A\left(\nu \right)=\left[\begin{array}{cccc}0& 0& 0& Y_{\dot{v}}\nu \\ 0& 0& 0& -X_{\dot{u}}u\\ 0& 0& 0& 0\\ -Y_{\dot{v}}\nu & X_{\dot{u}}u& 0& 0\end{array}\right.\right]\left. C_{RB}\left(\nu \right)=\left[\begin{array}{cccc}0& 0& 0& -m\nu \\ 0& 0& 0& mu\\ 0& 0& 0& 0\\ m\nu & -mu& 0& 0\end{array}\right.\right] \\ \left.C_0(\nu )=C_{RB}+C_A=\left[\begin{array}{cccc}0& 0& 0& -(m-Y_{\dot{v}})\nu \\ 0& 0& 0& (m-X_{\dot{u}})u\\ 0& 0& 0& 0\\ (m-Y_{\dot{v}})\nu & -(m-X_{\dot{u}})u& 0& 0\end{array}\right.\right] \end{gathered}$$

(6)

In Formula (6), $C_0\left(v\right)\in R^{4\times 4}$, $C_0\left(\nu \right)=C_{RB}+C_A$, where $C_{RB}$ represents the matrix encompassing the rigid body Coriolis force and centripetal force, while $C_{A }$ denotes the matrix accounting for the Coriolis force and centripetal force resulting from the added mass of the inertial fluid dynamics [25].

$$D_0\left(v\right)=-\left[\begin{array}{cccc}{X}_u+X_{u\left|u\right|}\left|u\right|& 0& 0& 0\\ 0& Y_v+Y_{v\left|v\right|}\left|v\right|& 0& 0\\ 0& 0& Z_w+Z_{w\left|w\right|}\left|w\right|& 0\\ 0& 0& 0& N_r+N_{r\left|r\right|}\left|r\right|\end{array}\right]$$

(7)

In the Formula (7), $D_0\left(V\right)\in R^{4\times 4}$ denotes the damping matrix, which arises from the effects of viscous fluid dynamics on the robot. The symbols $X_u,Y_v,Z_w,N_r$ and $X_{u\left|u\right|},Y_{v\left|v\right|},Z_{w{|}_w|},N_{r\left|r\right|}$ correspondingly denote the primary and secondary hydrodynamic damping coefficients that emerge during the motion of the underwater robot [25].

$$g\left(\eta \right)={\left[\begin{array}{cccc}0& 0& -(W-B)& 0\end{array}\right]}^T$$

(8)

In Formula (8), $g\left(\eta \right)$ is represented as the restoring force and moment caused by gravity and buoyancy [25]. $W$ is the gravity of the underwater robot, and $B$ is the buoyancy. In the physical structure design of the ROV, buoyancy is equal to gravity, so it can be further expressed as: $g\left(\eta \right)={\left[\begin{array}{cccc}0& 0& 0& 0\end{array}\right]}^T$.

In Formula (3), $\tau \in R^{4\times 4}$ represents the thrust generated by the propeller, expressed as $\tau ={\left[F_x,F_y,F_Z,N_Z\right]}^T$, where $F_x,F_y,F_Z$ are the thrusts generated by the propeller along the three coordinate axes, and $N_Z$ are the moments generated by the propeller around the coordinate axes.

In Formula (3), $\Delta f\in R^{4\times 4}$ represents external disturbances such as ocean currents in the working environment, and ${\tau }_{\Delta }\in R^{4\times 4}$ represents the uncertainty of dynamic lumped parameters, where ${\tau }_{\Delta }=-\Delta M\dot{\nu }-\Delta C\left(v\right)v-\Delta D\left(v\right)v$. For the subsequent design of LADRC, we unify the model parameter uncertainty terms $\left(\Delta M,\Delta C\left(V\right),\Delta D\left(V\right)\right)$ and external disturbances as total disturbances.

3. LADRC Controller Design

In addressing the challenges posed by uncertainties in ocean current disturbances, process noise, and hydrodynamic damping coefficients, the Linear Active Disturbance Rejection Control (LADRC) method is utilized. LADRC comprises a Linear Extended State Observer (LESO) and Linear State Error Feedback Control Law (LSEFC). Of particular significance is the development of the LESO, which is implemented to estimate and counteract external disturbances and uncertainties in model parameters. The LSEFC determines the virtual control signal $u_0$ by evaluating the system’s state error. The control block diagram is illustrated in Figure 2.

3.1. Linear Extended State Observer

LESO improves the performance of control systems by estimating and compensating for disturbances and unknown state variables. This section outlines the design process of the LESO scheme based on the mathematical model of the ROV system. The kinematics expression of the ROV is $\dot{\eta }=J\left(\eta \right)\nu$ [25], from which we can derive Formula (9):

$$=\begin{array}{c}\ddot{\eta }=J\left(\eta \right)\dot{\nu }+\dot{J}\left(\eta \right)\nu \\ J\left(\eta \right)M_0^{-1}\left(\Delta f+{\tau }_{\Delta }-C_0\left(\nu \right)v-D_0\left(\nu \right)v+g\left(\eta \right)\right)+\dot{J}\left(\eta \right)\nu +J\left(\eta \right)M_0^{-1}\tau \\ =f+X+J\left(\eta \right)M_0^{-1}\tau \end{array}$$

(9)

In Formula (9), $X=\dot{J}\left(\eta \right)\nu -J\left(\eta \right)M_0^{-1}\left(C_0\left(\nu \right)v-D_0\left(\nu \right)v+g\left(\eta \right)\right),f=J\left(\eta \right)M_0^{-1}(\Delta f+{\tau }_{\Delta })$ represents the total uncertainty of the unmodeled dynamics and external disturbances. For the convenience of designing the LESO, let $f=\left[f_x,f_y,f_z,f_{\psi }\right]$, $\tau =\left[\begin{array}{c}{\tau }_x,{\tau }_y,{\tau }_z,{\tau }_{\psi }\end{array}\right]$, then Formula (9) can be written as [26]:

$$\begin{array}{c}\ddot{x}=a_1{\tau }_x+X_1+f_x\\ \ddot{y}=a_2{\tau }_y+X_2+f_y\\ \ddot{z}=a_3{\tau }_z+X_3+f_z\\ \ddot{\psi }=a_6{\tau }_{\psi }+X_6+f_{\psi }\end{array}$$

(10)

$$\left\{\begin{array}{c}{X}_1={\displaystyle \frac{cos\psi }{m-X_{\dot{u}}}}\left(X_u+X_{u\mid u\mid }\mid u\mid -\left(m-Y_{\dot{v}}\right)ur\right)\\ -{\displaystyle \frac{sin\psi }{m-Y_{\dot{v}}}}\left(Y_v+Y_{v\mid v\mid }\mid v\mid -\left(m-X_{\dot{u}}\right)ur\right)\\ -usin\psi -vcos\psi \\ \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}{a}_1=\left({\displaystyle \frac{cos\psi }{m-X_{\dot{u}}}}-{\displaystyle \frac{sin\psi }{m-Y_{\dot{v}}}}\right)\\ X_2={\displaystyle \frac{sin\psi }{m-X_{\dot{u}}}}\left(X_u+X_{u\mid u\mid }\mid u\mid -\left(m-Y_{\dot{v}}\right)ur\right)+{\displaystyle \frac{cos\psi }{m-Y_{\dot{v}}}}\\ \left(Y_v+Y_{v\mid v\mid }\mid v\mid -\left(m-X_{\dot{u}}\right)ur\right)+ucos\psi -vsin\psi \\ a_2=\left({\displaystyle \frac{sin\psi }{m-X_{\dot{u}}}}+{\displaystyle \frac{cos\psi }{m-Y_{\dot{v}}}}\right)\\ X_3={\displaystyle \frac{\left(Z_w+Z_{w\left|w\right|}\left|w\right|\right)w}{m-Z_{\dot{w}}}}\\ a_3={\displaystyle \frac{1}{m-Z_{\dot{w}}}}\\ X_4={\displaystyle \frac{\left(N_r+N_{r\left|r\right|}\left|r\right|\right)r}{I_z-N_{\dot{r}}}},a_4={\displaystyle \frac{1}{I_z-N_{\dot{r}}}}\end{array}\right.$$

(11)

The total uncertainty $f$, which represents the unmodeled dynamics and external disturbances, is defined as the total disturbance. To achieve an accurate estimation of the total disturbance $f$ experienced in the control of underwater robots [19], the dynamics model of the ROV is rewritten as follows.

$$\ddot{\psi }={\displaystyle \frac{1}{I_z-N_{\dot{r}}}}{\tau }_{\psi }+{\displaystyle \frac{\left(N_r+N_{r\left|r\right|}\left|r\right|\right)r}{I_z-N_{\dot{r}}}}+f_{\psi }$$

(12)

Using the dynamic expression of heading angle $\psi$ from Equations (10) and (11) as an example for controller design, we provide a detailed design explanation for LESO and LSEFC.

In Formula (12), ${\displaystyle \frac{1}{I_z-N_{\dot{r}}}}$ is a constant term whose value is determined by the system’s inertia parameter $I_z$ and damping coefficient $N_{\dot{r}}$. To simplify the formula, $b={\displaystyle \frac{1}{I_z-N_{\dot{r}}}}$ is used to replace this complex coefficient. The second term ${\displaystyle \frac{\left(N_r+N_{r\left|r\right|}\left|r\right|\right)r}{I_z-N_{\dot{r}}}}$ describes the nonlinear disturbances caused by hydrodynamics, which are typically regarded as a type of interference or unmodeled dynamics. In the design of LESO, this part is incorporated into the total uncertainty $f_{\psi }$ for unified treatment. Therefore, Formula (12) can be simplified to:

$$\ddot{\psi }=b{\tau }_{\psi }+f_{\psi }$$

(13)

Taking the state variables ${\psi }_1,{\psi }_2,{\psi }_3$, among them ${\psi }_3=f_{\psi }$ as the extended state, the ROV heading angle $\psi$ control model can be expressed as:

$$\left\{\begin{array}{c}\begin{array}{c}\psi ={\psi }_1\\ {\dot{\psi }}_1={\psi }_2\end{array}\\ {\dot{\psi }}_2=b{\tau }_{\psi }+{\psi }_3\\ {\dot{\psi }}_3=D\end{array}\right.$$

(14)

Let $D={\dot{f}}_{\psi }$, ${\psi }_2=\dot{\psi }$. A linear expansion state observer can be established for the system (13) [19]:

$$\left\{\begin{array}{c}{e}_1=\psi -z_1\\ {\dot{z}}_1=z_2+{\beta }_1{e}_1\\ {\dot{z}}_2=z_3+{\beta }_2{e}_1+b{\tau }_{\psi }\\ {\dot{z}}_3={\beta }_3{e}_1\end{array}\right.$$

(15)

In reference Formula (15), $z_1,z_2$ represent the estimated values of the state variables of the controlled object (in this example, $z_1$ represents the observation value of $\psi$, and $z_2$ is the observation value of the $\psi$ derivative), while $z_3$ represents the real-time estimated value of the total disturbance (unknown external disturbances and uncertain models). ${\beta }_1,{\beta }_2,{\beta }_3$ are the gains of the LESO. If the observer gains are chosen appropriately, LESO can achieve precise tracking of each state variable of the controlled object. To facilitate parameter tuning, the values of ${\beta }_1,{\beta }_2,{\beta }_3$ are determined by ${\omega }_0$. By reasonably selecting the parameter ${\omega }_0$, the observed value of the “total disturbance” can be made closer to the true value. The Laplace transform of the LESO equation yields [19]:

$$\begin{array}{c}{z}_1={\displaystyle \frac{{\beta }_1{s}^2+{\beta }_{2}s+{\beta }_3}{L^{\prime }\left(s\right)}}Y\left(s\right)+{\displaystyle \frac{bs}{L^{\prime }\left(s\right)}}U\left(s\right)\\ z_2={\displaystyle \frac{{\beta }_2{s}^2+{\beta }_{3}s}{L^{\prime }\left(s\right)}}Y\left(s\right)+{\displaystyle \frac{bs\left(s+{\beta }_1\right)}{L^{\prime }\left(s\right)}}U\left(s\right)\\ z_3={\displaystyle \frac{{\beta }_3{s}^2}{L^{\prime }\left(s\right)}}Y\left(s\right)-{\displaystyle \frac{b{\beta }_3}{L^{\prime }\left(s\right)}}U\left(s\right)\end{array}$$

(16)

$Y\left(S\right)$ is the Laplace transform of the system output y(t) in the time domain, and $U\left(S\right)$ is the Laplace transform of the system input u(t) in the time domain. The characteristic equation corresponding to LESO is [18]:

$$L^{\prime }\left(\begin{array}{c}s\end{array}\right)=s^3+{\beta }_1{s}^2+{\beta }_{2}s+{\beta }_3$$

(17)

To stabilize the system, the roots of the characteristic equation must be located in the left half of the s-plane. Therefore, the three poles of the observer are uniformly placed on the left half of the real axis at $-{\omega }_o$ (where ${\omega }_o$ is the bandwidth of the observer, and ${\omega }_o>0$). Therefore, the observer gain can be obtained as ${ \beta }_1=3{\omega }_o, {\beta }_2=3{\omega }_o^2, {\beta }_3={\omega }_o^3$. The estimated error of LESO can be expressed as [18]:

$$\left\{\begin{array}{c}{\dot{e}}_1=e_2-{\beta }_1{e}_1\\ {\dot{e}}_2=e_3-{\beta }_2{e}_1\\ {\dot{e}}_3=D-{\beta }_3{e}_1\end{array}\right.$$

(18)

Among them, $e_i={\psi }_i-z_i (i=1,2,3)$ provides the conditions for the estimation error of LESO, which will be used for the stability proof of LESO in the following text.

3.2. Linear State Error Feedback Controller

Traditional PID controllers use error integration to eliminate static errors, but the feedback from error integration can make the system prone to oscillation. In contrast, the LESO (Linear Extended State Observer) employs real-time compensation for total disturbances, avoiding the negative effects of integral feedback. The result is shown in Equation (19) [18]:

$$\left\{\begin{array}{c}{e}_1={\psi }_d-z_1\\ e_2={\dot{\psi }}_d-z_2\\ u_0=k_p{e}_1+k_d{e}_2\end{array}\right.$$

(19)

In the equation, ${\psi }_d$ is the reference heading angle input, $u_0$ is the error feedback control quantity, and $k_p$, $k_d$ are the controller gains. According to Equation (19), the transfer function of $u_{0 }$ concerning $r$ can be obtained:

$${\displaystyle \frac{U_0\left(s\right)}{R\left(s\right)}}={\displaystyle \frac{k_p+k_{d}s}{s^2+k_{d}s+k_p}}$$

(20)

By setting both poles of the controller on the real axis at $-{\omega }_c$ (where ${\omega }_c$ represents the control bandwidth) in the left half of the $s$-plane, the controller gain can be determined as [18].

$$\left\{\begin{array}{c}{k}_p={\omega }_c^2\\ k_d=2{\omega }_c\end{array}\right.$$

(21)

Based on $u_0$, an additional compensation term for the total disturbance estimate is added, so the control quantity can be taken as:

$${\tau }_{\psi }={\displaystyle \frac{-z_3+u_0}{b_0}}$$

(22)

In Formula (22), $u_0$ is the error feedback control quantity, and ${\tau }_{\psi }$ is the actual input of the controlled object.

4. DDPG Optimization of Control Parameters for Active Disturbance Rejection Controller

For the controlled system, when the range of unknown disturbances is too large and the rate of change is too fast, using a fixed-parameter active disturbance rejection controller results in low control accuracy. Therefore, by combining deep reinforcement learning, an improved active disturbance rejection controller has been designed, in which the active disturbance rejection control parameters will vary with the environment.

4.1. Deep Deterministic Policy Gradient-Active Disturbance Rejection Controller Algorithm Framework

Reinforcement learning is an algorithm that allows an agent to adjust its behavioral strategies based on observations made during interactions with the environment, to maximize cumulative rewards. The schematic diagram is shown in Figure 3.

In response to the online adjustment problem of fixed parameter active disturbance rejection controllers, this paper employs the Deep Deterministic Policy Gradient algorithm (DDPG), which can handle continuous action control. The proposed control strategy, DDPG-LADRC, treats the entire underwater robot control system as the environment, using the system’s control performance as the reward evaluation criterion. The DDPG-LADRC agent determines actions based on the current environment, and then the environment provides a new state based on the output value and calculates the reward value. The DDPG-LADRC agent makes judgments, optimizes and updates the next action, and interacts with the environment until the reward converges.

4.2. Deep Deterministic Policy Gradient Algorithm Principles

DDPG is a deep reinforcement learning algorithm based on the actor-critic framework. The actor-network outputs deterministic actions in a continuous action space based on environmental state feedback, while the critic-network calculates the corresponding Q-value based on the current state and action, which is used to evaluate the long-term expected return of the action. By adjusting the weights of the critic network according to the error between the reward output by the critic network and the actual received reward, the output estimates of the critic network can become more accurate. Using the policy gradient algorithm, the parameters of the actor-network are updated in the direction of increasing the action value. During the interaction between the agent and the environment, the learning parameters of both networks will be continuously updated until the policy converges [27].

At time $t$, the mapping from state $s$ to action $a$ is referred to as policy $\pi$.

$$a_t=\pi \left(s_t\right)$$

(23)

According to the actions generated by strategy $\pi$, new states and reward values $r$ are continuously obtained. The formula for calculating cumulative rewards is:

$$G_t=\sum _{t=1}^T{\gamma }^t{r}^t$$

(24)

The Bellman equation for the state value function is represented as:

$$V_{\pi }\left(s\right)=E_{\pi }[{\sum }_{k=0}^{\infty }{G}_t|s=s_t]=E_{\pi }\left[r_{t+1}+\gamma V_{\pi }\left(s_{t+1}\right)|s=s_t\right]$$

(25)

Considering the impact of actions on the value function, the Bellman equation for the state-action value function is represented as:

$$Q_{\pi }\left(s,a\right)=E_{\pi }\left[r_{t+1}+\gamma Q_{\pi }\left(s_{t+1},a_{t+1}\right)|s=s_t,a=a_t\right]$$

(26)

The optimal Bellman equation can be expressed as:

$$Q^{\ast }\left(s_t,a_t\right)=r_{s_t}^{a_t}+\gamma Q_{\pi }^{\ast }\left(s_{t+1},a_{t+1}\right)$$

(27)

The optimal strategy ${\pi }^{\ast }$ is obtained by maximizing the cumulative reward and its corresponding optimal Bellman equation:

$$\pi \left(s\right)\to a_t={\mathit{argmax}Q}_{\pi }^{\ast }\left(s_t,a_t\right)$$

(28)

We calculate the loss function for the target $Q$ value, using $y_{t }$ to represent it:

$$\left\{\begin{array}{c}{y}_t=r_t+\gamma Q^{\prime }\left(s_{t+1},{\pi }^{\prime }\left(s_{t+1}∣{\theta }^{{\pi }^{\prime }}\right)∣{\theta }^{Q^{\prime }}\right)\\ L\left({\theta }^Q\right)=E(y_t-Q(s_t,a_t\mid {\theta }^Q){)}^2\end{array}\right.$$

(29)

By calculating the gradient of the loss function, we update the current value network [28].

$$\left\{\begin{array}{c}{\theta }_k^Q={\theta }_{k-1}^Q-{\mu }_Q{\nabla }_{{\theta }^Q}L\left({\theta }_{k-1}^Q\right)\\ {\nabla }_{{\theta }^Q}L\left({\theta }_{k-1}^Q\right)=E\left(2\right(y_t-Q(s,a|{\theta }_{k-1}^Q\left){|}_{s=s_t,a=a_t}\right)\\ \nabla Q_{{\theta }^Q}(s,a|{\theta }_{k-1}^Q\left){|}_{s=s_t,a=a_t}\right)\end{array}\right.$$

(30)

The strategy network uses the $Q$ function output from the value network as the loss function. By taking the policy gradient of the $Q$ function, the update formula is obtained [28].

$$\left\{\begin{array}{c}{\theta }_k^{\pi }={\theta }_{k-1}^{\pi }-{\mu }_{\pi }{\nabla }_{{\theta }^{\pi }}L\left({\theta }_{k-1}^{\pi }\right)\hfill \\ {\nabla }_{{\theta }_{k-1}^{\pi }}\pi ={\nabla }_{a}Q(s,a\mid {\theta }_{k-1}^{\pi })\hfill \\ {|}_{s=s_t,a_t=\pi \left(s_t\right)}{\nabla }_{{\theta }^{\pi }}\pi (s\mid {\theta }_{k-1}^{\pi }){|}_{s=s_t})\hfill \end{array}\right.$$

(31)

The target network uses a soft update method as follows [29].

$$\left\{\begin{array}{c}{\theta }_k^{Q^{\prime }}=\tau {\theta }_{k-1}^Q+\left(1-\tau \right){\theta }_{k-1}^{Q^{\prime }}\\ {\theta }_k^{{\pi }^{\prime }}=\tau {\theta }_{k-1}^{\pi }+\left(1-\tau \right){\theta }_{k-1}^{{\pi }^{\prime }}\end{array}\right.$$

(32)

The DDPG algorithm is a deterministic policy that adds noise to the deterministic policy, as shown in Equation (33), allowing the agent to explore the environment more effectively and preventing it from getting stuck in local optima. The deterministic policy gradient helps the critic converge and updates the network parameters [29]. The meanings of the various parameters in the above analysis are shown in

Table 1. DDPG algorithm parameter meaning.

Algorithm Parameters	Meaning
$Q(s_t,a_t\mid {\theta }^Q)$	The $Q$ value output by the current value network at time $t$
$Q^{\prime }(s_{t+1},{\pi }^{\prime }(s_{t+1}\mid {\theta }^{{\mu }^{\prime }})\mid {\theta }^{Q^{\prime }})$	Input $Q$ value of the target network
${\pi }^{\prime }\left(S_{t+1}∣{\theta }^{{\pi }^{\prime }}\right)$	Action variables output by the target strategy network
${\theta }_k^Q,{\theta }_k^{\pi }$	The parameters of the network at the k round of learning
${\mu }_{\mathcal{Q}}$	Learning rate of value networks
${\nabla }_{{\theta }^Q}L\left({\theta }_{k-1}^Q\right)$	A gradient of the loss function concerning the parameters
${\mu }_{\pi }$	Learning rate of the strategy network
$\begin{array}{c}{\theta }_{k-1}^{\pi }\end{array}$	Strategy gradient
${\theta }_k^{Q^{\prime }},{\theta }_k^{{\pi }^{\prime }}$	The parameters of the target network at the k learning iteration
$\tau$	Soft update coefficient

.

$$a_t={\mu }_{\theta }\left(s_t\right)+N$$

(33)

The structure of the DDPG algorithm model is shown in Figure 4. The DDPG agent stores the sample data obtained from interacting with the LADRC control system in the experience pool. During the learning process, it randomly samples m pieces of data from the experience pool and continuously iterates to update the network gradient values to optimize the algorithm [29,30].

Combining deep reinforcement learning with an active disturbance rejection controller, the control system is designed to obtain environmental state data through the interaction between the agent and the environment (underwater robot control system). Here, LSEFC represents the state error feedback controller, and LESO represents the linear expanded observer. The structural block diagram of the active disturbance rejection controller based on deep reinforcement learning is shown in Figure 5.

According to the control system structure block diagram designed in the above figure, we set the various parameters of the deep reinforcement learning agent.

If the LADRC control has the Markov property, then when optimizing with DDPG, the future state transitions of the system depend only on the current state and control actions, without the need to explicitly model the state transition probabilities. The optimization problem of the LESO observation capability is modeled as a reinforcement learning task in a continuous action space. The DDPG algorithm is used to dynamically adjust the key parameter $w_0$ of the LESO, enabling it to adapt to changes in external disturbances, thereby improving the disturbance estimation accuracy of the LESO and the robustness of the controller. The actor network is responsible for generating the adjustment of the LESO observer bandwidth $w_0$, based on the current state. The critic network outputs the action value Q based on the current state and the action generated by the actor network, guiding the policy update of the actor network. The ROV studied in this paper is under umbilical cable control, effectively avoiding the issue of limited resources for the ROV, and the ROV’s controller can meet the high computational demands of DDPG training.

For the state space, select $\left(e,\dot{e}\right)$, corresponding to the errors $\left(e_x,e_y,e_z,e_{\psi }\right)$, and the differential of the error in each degree of freedom.

For the action space, select the poles of the expanded state observer, that is $w_0$, ${\beta }_1=3{\omega }_o,{\beta }_2=3{\omega }_o^2,{\beta }_3={\omega }_o^3$.

To reduce the final error, a reward function is set based on the error between the output of each degree of freedom and the expected value: $R=-\left[\left(\sqrt{{e_x}^2+{e_y}^2+{e_z}^2+{e_{\psi }}^2}\right)\right]$.

In terms of the discount factor, the degree to which future rewards influence current decisions is determined, and in this article, the chosen discount factor $\gamma$ = 0.98 is used to ensure that accurate trajectory tracking is given high priority. After multiple adjustments, the final selected DDPG parameters are shown in

Table 2. DDPG algorithm parameters.

Hyperparameter	Value
Actor-network learning rate	0.001
Critics’ online learning rate	0.0005
Small batch sampling sample size	64
Discount factor	0.98
Noise variance	0.2
Noise attenuation coefficient	0.00001
Experience pool size	100,000

.

The reward curve of the Deep Deterministic Policy Gradient (DDPG) algorithm is generally used to determine whether the agent’s training has converged. The curve showing the change in rewards after training over the training iterations is shown in Figure 6.

4.3. Stability Analysis

To conduct the analysis, the following hypothesis is proposed based on engineering practice: the total disturbance observed by the observer in the self-disturbance rejection control is bounded within $H$, $H=\left\{D\right|\left|D\right|⩽F_h\}$, where $F_h$ is a positive constant.

Theorem 1. 

The estimation error of the observer constructed in Equation (15) is bounded [18]. $\underset{{\omega }_0\to \infty ,t\to \infty }{lim}||e||=0$.

Proof. 

Let $q_i={\displaystyle \frac{e_i}{{\omega }_o^i}}(i=1,2,3)$, then Equation (18) can be rewritten as:

$$\left\{\begin{array}{c}{\dot{q}}_1={\omega }_o(q_2-3q_1)\hfill \\ {\dot{q}}_2={\omega }_o(q_3-3q_1)\hfill \\ {\dot{q}}_3={\omega }_o\left({\displaystyle \frac{D}{{\omega }_o^4}}-q_1\right)\hfill \end{array}\right.$$

(34)

The reference Formula (34) can be rewritten as:

$$\dot{\eta }={\omega }_{o}Aq+B{\displaystyle \frac{D}{{\omega }_o^3}}$$

(35)

For simplicity, let $q=[q_1,q_2,q_3{]}^T$, therefore:

$$A=\left[\begin{array}{ccc}-3& 1& 0\\ -3& 0& 1\\ -1& 0& 0\end{array}\right], B=\left[\begin{array}{c}0\\ 0\\ 1\end{array}\right]$$

(36)

Observing the above equation, for any positive ${\omega }_o, A$ is Hurwitz, therefore, there exists a unique positive definite symmetric matrix $P_{\eta }$ that satisfies the Lyapunov equation $A^T{P}_{\eta }+P_{\eta }A=-Q_{\eta }$. By choosing the Lyapunov function as $V\left(q\right)=q^T{P}_{\eta }q$, we can derive the following for $V\left(q\right)$:

$$\begin{array}{c}\dot{V}\left(q\right)=q^T{P}_{\eta }\dot{q}+{\dot{q}}^T{P}_{\eta }q\hfill \\ =q^T{P}_{\eta }({\omega }_{0}Aq+B{\displaystyle \frac{D}{{\omega }_0^3}})+({\omega }_{0}Aq+B{\displaystyle \frac{D}{{\omega }_0^3}}{)}^T{P}_{\eta }q\hfill \\ =-{\omega }_0{q}^T{Q}_{\eta }q+2q^T{P}_{\eta }B{\displaystyle \frac{D}{{\omega }_0^3}}\hfill \end{array}$$

(37)

Referring to Equation (37), using the Cauchy inequality and the property of the minimum eigenvalue of positive definite matrices, $\dot{V}\left(q\right)$ can be rewritten as:

$$\begin{array}{c}\dot{V}(q)=-{\omega }_0{q}^T{Q}_{\eta }q+2q^T{P}_{\eta }B{\displaystyle \frac{D}{{\omega }_o^3}}\\ \le {\omega }_o{\lambda }_{min}\left(Q_{\eta }\right)\parallel q{\parallel }^2+{\displaystyle \frac{2F_h{\lambda }_{max}\left(P_{\eta }\right)\parallel q\parallel }{{\omega }_o^3}}\end{array}$$

(38)

Using the eigenvalues of the matrix, we can obtain the following bounds on the quadratic form: ${\lambda }_{\mathrm{m}\mathrm{i}\mathrm{n}}\left(P_{\eta }\right)\parallel q{\parallel }^2\le q^T{P}_{\eta }q\le {\lambda }_{\mathrm{m}\mathrm{a}\mathrm{x}}\left(P_{\eta }\right)\parallel q{\parallel }^2$. This can be rewritten as: ${\displaystyle \frac{V\left(q\right)}{{\lambda }_{\mathrm{m}\mathrm{a}\mathrm{x}}\left(P_{\eta }\right)}}\le \parallel q{\parallel }^2\le {\displaystyle \frac{V\left(q\right)}{{\lambda }_{\mathrm{m}\mathrm{i}\mathrm{n}}\left(P_{\eta }\right)}}$. Therefore, inequality (38) can be rewritten as:

$$\begin{array}{c}\dot{V}(q)\le {\omega }_o{\displaystyle \frac{{\lambda }_{min}\left(Q_{\eta }\right)}{{\lambda }_{max}\left(P_{\eta }\right)}}V(q)\hfill \\ +{\displaystyle \frac{2F_h{\lambda }_{max}\left(P_{\eta }\right)}{{\omega }_o^3\sqrt{{\lambda }_{min}\left(P_{\eta }\right)}}}\sqrt{V\left(q\right)}\hfill \end{array}$$

(39)

To obtain the linear differential inequality, let $W=\sqrt{V\left(q\right)}$, then $\dot{W}={\displaystyle \frac{\dot{V}\left(q\right)}{2\sqrt{V\left(q\right)}}}$ can be obtained, and inequality (39) can be rewritten as:

$$\dot{W}\le {\omega }_o{\displaystyle \frac{{\lambda }_{min}\left(Q_{\eta }\right)}{2{\lambda }_{max}\left(P_{\eta }\right)}}W+{\displaystyle \frac{F_h{\lambda }_{max}\left(P_{\eta }\right)}{{\omega }_o^3\sqrt{{\lambda }_{min}\left(P_{\eta }\right)}}}$$

(40)

When studying the state equation $\dot{W}$, it is often necessary to obtain the boundary of its solution $W$, rather than the solution itself. The Gronwall–Bellman method is one of the approaches used for this purpose [31]. First, let $\beta =-w_0{\displaystyle \frac{\lambda min\left(Q_{\eta }\right)}{2\lambda max\left(P_{\eta }\right)}},\alpha ={\displaystyle \frac{F_h{\lambda }_{max}\left(P_{\eta }\right)}{{\omega }_o^3\sqrt{{\lambda }_{min}\left(P_{\eta }\right)}}}$. By applying the inequality, we can obtain the following.

Assume $\dot{ W}=\beta W+\alpha$. Thus, there is $W\le (W(t_0)-{\displaystyle \frac{\alpha }{\beta }})e^{\beta (t-t_0)}+{\displaystyle \frac{\alpha }{\beta }}$. By organizing, we can obtain the following expression:

$$\begin{array}{c}W\le ({\displaystyle \frac{2F_h{\lambda }_{\mathrm{m}\mathrm{a}\mathrm{x}}^2\left(P_{\eta }\right)}{{\omega }_o^4\sqrt{{\lambda }_{\mathrm{m}\mathrm{i}\mathrm{n}}\left(P_{\eta }\right)}{\lambda }_{\mathrm{m}\mathrm{i}\mathrm{n}}\left(Q_{\eta }\right)}}-W(t_0))e^{-{\omega }_0{\displaystyle \frac{{\lambda }_{\mathrm{m}\mathrm{i}\mathrm{n}}\left(Q_{\eta }\right)}{2{\lambda }_{\mathrm{m}\mathrm{a}\mathrm{x}}\left(P_{\eta }\right)}}\left(t-t_0\right)}\hfill \\ +{\displaystyle \frac{2F_h{\lambda }_{\mathrm{m}\mathrm{a}\mathrm{x}}^2\left(P_{\eta }\right)}{{\omega }_o^4\sqrt{{\lambda }_{\mathrm{m}\mathrm{i}\mathrm{n}}\left(P_{\eta }\right)}{\lambda }_{\mathrm{m}\mathrm{i}\mathrm{n}}\left(Q_{\eta }\right)}}\hfill \end{array}$$

(41)

From $W=\sqrt{V\left(q\right)}=\sqrt{q^T{P}_{\eta }q}$, ${\displaystyle \frac{V\left(q\right)}{{\lambda }_{max}\left(P_n\right)}}\le {∥q∥}^2\le {\displaystyle \frac{V\left(q\right)}{{\lambda }_{min}\left(P_n\right)}}$, we can obtain Equation (42):

$$∥q∥\le {\displaystyle \frac{\sqrt{V}}{\sqrt{{\lambda }_{min}\left(P_{\eta }\right)}}}={\displaystyle \frac{W}{\sqrt{{\lambda }_{min}\left(P_{\eta }\right)}}}$$

(42)

When $t\to \infty$, the Expression (43) can be obtained:

$$\begin{array}{c}∥q∥\le {\displaystyle \frac{\sqrt{V\left(q\right)}}{\sqrt{{\lambda }_{min}\left(P_{\eta }\right)}}}\\ \le {\displaystyle \frac{2F_h{\lambda }_{max}^2\left(P_{\eta }\right)}{{\omega }_o^4{\lambda }_{min}\left(P_{\eta }\right){\lambda }_{min}\left(Q_{\eta }\right)}}\\ ={\displaystyle \frac{k}{{\omega }_0^4}}\end{array}$$

(43)

In Formula (43), $k={\displaystyle \frac{2F_h{\lambda }_{max}^2\left(P_{\eta }\right)}{{\lambda }_{min}\left(P_{\eta }\right){\lambda }_{min}\left(Q_{\eta }\right)}}$ is a normal constant, and because $P_{\eta }$, $Q_{\eta }$ are independent of ${\omega }_0$, the Equation (43) can demonstrate that $\underset{{\omega }_0\to \infty ,t\to \infty }{lim}||q||=0$, which, together with $q_i={\displaystyle \frac{e_i}{{\omega }_o^i}}(i=1,2,3)$, is produced. Therefore, $\underset{{\omega }_0\to \infty ,t\to \infty }{lim}||e||⩽||q||{\omega }_0^3⩽{\displaystyle \frac{k}{{\omega }_o}}=0$, thus completing the proof. Define $H_e$ as $H_e=\left\{e\right|\left|\right|e\left|\right|⩽E\}$, where $E$ is a positive constant. By adjusting ${\omega }_0$ to ensure ${\displaystyle \frac{k}{{\omega }_o}}⩽E$, the estimated error of LESO will remain within $H_e$, Theorem 1 has been proven. □

Theorem 2. 

According to the error feedback control law given by Equation (19), we can ensure the closed-loop stability of the control system. According to Theorem 1, the convergence of LESO can be guaranteed by carefully selecting  ${\omega }_0$,$b$, and the estimation error of LESO will be constrained within${ H}_e$. Translate Equation (19) into the Equation (14) to obtain:

$$\left\{\begin{array}{c}{\dot{e}}_{{\psi }_1}=e_{{\psi }_2}\hfill \\ {\dot{e}}_{{\psi }_2}=-k_p{e}_{{\psi }_1}-k_d{e}_{{\psi }_2}+k_p{e}_1+k_d{e}_2+e_3\hfill \end{array}\right.$$

(44)

Referencing Equation (44), $e_{\psi 1}={\psi }_r-{\psi }_1$ and $e_{\psi 2}={\dot{\psi }}_r-{\psi }_{2 }$ are defined as tracking errors. The above equation can be rewritten as:

$$\begin{array}{c}{\dot{e}}_{\psi }=C{e}_{\psi }+Ge\\ e_{\psi }={\left[\begin{array}{c}{e}_{\psi 1},e_{\psi 2}\end{array}\right]}^T,C=\left[\begin{array}{cc}0& 1\\ -k_p& -k_d\end{array}\right]\\ G=\left[\begin{array}{ccc}{k}_p& k_d& 1\end{array}\right]\end{array}$$

(45)

Since the controller output is positive, ensure that $k_p,k_d⩾0$. Then, the characteristic roots of $C$ can be expressed as:

$${\lambda }_{1,2}=-{\displaystyle \frac{k_d}{2}}\pm \sqrt{-k_p+{\displaystyle \frac{k_d^2}{4}}}$$

(46)

Therefore, the designed linear active disturbance rejection controller is stable, and Theorem 2 has been proven. The tracking error of the observer is also bounded.

5. Simulation Analysis

The underwater intelligent cleaning and inspection robot is specifically designed for the safety inspection of marine oil platform risers and the removal of marine organisms attached to the risers. It is equipped with an Ultra-Short Baseline positioning system (USBL), an attitude sensor, a depth sensor, and a compass, enabling precise positioning, attitude awareness, and depth perception. In addition, its propulsion system includes four horizontal thrusters and four vertical thrusters. The model parameters of the underwater robot are shown in

Table 3. ROV model parameters.

Parameter	Values	Parameter	Values
m	197 kg	$Y_{v\left|v\right|}$	−245.2 N2/m2
$I_z$	25.1 Nms2	$Z_w$	−12.6 N/S
$X_u$	−5.24 N/S	$Z_{\dot{w}}$	−367.8 kg
$X_{\dot{u}}$	−135.1 kg	$Z_{w\left|w\right|}$	−547.4 N2/m2
$X_{u\left|u\right|}$	−109.1 N2/m2	$N_r$	−1.52 N/S
$Y_v$	−11.1 N/S	$N_{\dot{r}}$	−34.3 kg
$Y_{\dot{v}}$	−390.6 kg	$N_{r\left|r\right|}$	−26.2 N2/m2

, and the physical prototype and thruster layout are illustrated in Figure 7.

To verify that DDPG-LADRC has stronger robustness, this paper proposes two experimental simulation scenarios.

(a)

Section 5.1 introduces a simple time-varying external disturbance, and the tracked trajectory is also relatively simple, to evaluate the improvement of the DDPG-LADRC control strategy on the transient performance during the motion of the ROV.

(b)

The time-varying disturbances introduced in Section 5.2 are related to the motion state of the ROV and track different trajectories, aiming to verify that the DDPG-LADRC control strategy has stronger robustness when the ROV is in a dynamic marine environment.

5.1. Scenario 1

To verify the enhanced effect of combining reinforcement learning DDPG with a linear active disturbance rejection controller in terms of disturbance suppression capability and control accuracy, the position and attitude of the underwater robot are tracked under time-varying external disturbances. The transient performance of the control system under perturbations is evaluated to validate the disturbance rejection and robustness of the DDPG-LADRC control scheme. Disturbances are introduced during the movement of the ROV as follows:

$$\begin{array}{c}f={\left[\begin{array}{c}{f}_x,f_y,f_z,f_{\psi }\end{array}\right]}^T\\ =[20\mathit{sin}\left(0.4\ast t\right) 20 sin(0.4\ast t)\\ 20 sin\left(0.4\ast t\right) 20 sin\left(0.4\ast t\right){]}^T\end{array}$$

(47)

The initial conditions for the underwater robot are set as $\left[X\right(0),Y(0),Z(0),Phi(0\left)\right]=0$, with the velocity and angular velocity set as $u\left(0\right)=\nu \left(0\right)=w\left(0\right)=r\left(0\right)=0$. Additionally, for the controller parameters, the PID parameters are set as:

$$\left\{\begin{array}{c}{K}_p=\{150,150,300,370\}\hfill \\ K_i=\{15,15,60,15\}\hfill \\ K_d=\{300,300,150,300\}\hfill \end{array}\right.$$

(48)

The parameters for the Active Disturbance Rejection Control are set as follows: $b_0=10, w_0=5$, because ${ \beta }_1=3{\omega }_o, {\beta }_2=3{\omega }_o^2,{\beta }_3={\omega }_o^3$ which means ${ \beta }_1$ = 15, ${\beta }_2=75$, ${\beta }_3=125.$ The relevant DDPG setting parameters are shown in Table 2 above. The underwater robot simulation is designed to run for 100 s, with a simulation step size of 0.01 s. The proposed control algorithm is mainly compared with PID and LADRC under fixed parameters through three-dimensional trajectory tracking, and planar tracking, to verify the degree of improvement in the system’s transient performance by the DDPG-LADRC control strategy. The trajectory tracking curve in the inertial coordinate system is:

$$\left\{\begin{array}{c}{x}_d=2\mathit{sin}(0.1\pi t) \mathrm{m}\\ y_d=2\mathit{cos}(0.1\pi t) \mathrm{m}\\ z_d=0.2t \mathrm{m}\\ {\psi }_d=0.03\pi t \mathrm{r}\mathrm{a}\mathrm{d}\end{array}\right.$$

(49)

First, a feasibility analysis of the parameter optimization for LESO is conducted. Figure 8 compares the observation errors of the LESO optimized by DDPG with those of the fixed-parameter LESO. It can be observed that the fixed-parameter LADRC controller is not precise in tracking total disturbances. In contrast, the DDPG-LADRC can maintain better performance with a shorter time under the constraints of model parameter uncertainty and strong unknown external disturbances in underwater robot trajectory tracking control. The DDPG-LADRC can quickly respond to changes in disturbances and adjust its control strategy promptly to adapt to these changes, thereby enhancing the system’s dynamic performance. This indicates that the optimized observer parameters of DDPG-LADRC are effective.

The three-dimensional trajectory tracking performance of the ROV under different control schemes, as well as the tracking curves in the XY, XZ, and YZ planes shown in Figure 9, can be observed. It can be seen that even in the presence of disturbances, the DDPG-LADRC control scheme can achieve precise trajectory tracking, with control performance superior to that of the PID controller and the fixed parameter LADRC controller, demonstrating stronger robustness. Therefore, parameter optimization based on DDPG can enhance the control performance of LADRC.

The selected evaluation indicators for transient performance are overshoot, settling time, and peak time.

In underwater robot control, overshoot is an important indicator used to describe the dynamic performance of a system. Overshoot is typically measured by the difference between the maximum output value and the steady-state value, and it can also be expressed as a percentage of this difference relative to the steady-state value. The system without overshoot typically stabilizes at the setpoint without deviating too much from the target value, indicating that there is no significant overreaction or oscillation during the response process. From

Table 4. Comparison of transient performance of different control methods: overshoot.

Comparison	$X$	$Y$	$Z$	$\mathrm{Phi}$
PID	6%	12.5%	10%	5%
LADRC	2%	25%	5%	0
DDPG-LADRC	0	0	0	0

, we can see that DDPG-LADRC maintains response speed without overshoot, while PID and LADRC exhibit overshoot. When the overshoot is too large, the control system is prone to oscillation. The results indicate that DDPG-LADRC ensures the dynamic response process of the system, maintaining high robustness even in the face of model uncertainty or external disturbances. The parameter optimization effect of DDPG-LADRC is evident, effectively meeting the dynamic performance requirements of the system.

In underwater robot trajectory tracking control, the adjustment time is an important dynamic performance indicator. It reflects the robot’s sensitivity to changes in control signals and its ability to respond quickly, defined as the time required for the ROV to respond and maintain within a certain allowable error range (usually ±2% or ±5% of the final value) after initially reaching the target value. A shorter adjustment time means that the ROV can stabilize more quickly around the target value, reducing oscillations or instability during the transition process. Additionally, a rapid response can better handle external disturbances and changes in the internal parameters of the ROV, enhancing the system’s robustness and stability. Referring to

Table 5. Comparison of transient performance of different control methods: settling time/s.

Comparison	$X$	$Y$	$Z$	$Phi$
PID	50	71	10	11
LADRC	14	40	3	35
DDPG-LADRC	1	5	1	4

, it can be seen that the adjustment time of DDPG-LADRC for the ROV in the $X$-direction is significantly better than the other two control strategies, reducing by 93% and 98%, respectively. In the $Y$-direction, the reductions are 93% and 86%, respectively, and in the $Z$-direction, the reductions are 66.7% and 90%, respectively. The attitude angles $Phi$ were reduced by 64% and 89%, respectively.

Even if the overshoot is 0, the system response may still have a “peak,” which does not refer to a deviation exceeding the steady-state value, but rather to the maximum value during the response process. In the underwater robot trajectory tracking control system, the peak time is an important dynamic performance indicator that describes the time required for the system response to exceed its steady-state value and reach the first peak. Referring to

Table 6. Comparison of transient performance of different control methods: peak time/s.

Comparison	$X$	$Y$	$Z$	$Phi$
PID	50	34	5	61
LADRC	15	62	50	35
DDPG-LADRC	1	6	1	4

, the comparison of peak times shows that in the $X$-direction, DDPG-LADRC significantly outperforms the other two control strategies, reducing by 93% and 98%, respectively. In the $Y$-direction, it reduces by 82% and 90%, respectively, and in the $Z$-direction, it reduces by 80% and 98%, respectively. The attitude angles $Phi$ are reduced by 93% and 89%, respectively.

In summary, through a comparative analysis of transient performance under different control methods, the results indicate the superiority of the DDPG-LADRC control strategy in terms of transient performance. Compared to PID controllers and traditional LADRC controllers, the proposed DDPG-LADRC is more suitable for underwater robotic systems that are multivariable, strongly coupled, have significant randomness, and are subject to unknown disturbances.

The tracking error of the ROV trajectory tracking in Figure 9 is shown in Figure 10. Compared to the PID controller and the fixed parameter LADRC controller, the proposed DDPG-LADRC controller has a smaller steady-state error. The PID and fixed-parameter LADRC control schemes are unable to eliminate steady-state errors in a short time, which leads to an inability to track the desired trajectory. However, the DDPG-LADRC significantly improves the control accuracy of the system by introducing DDPG to achieve online tuning of LADRC parameters in response to environmental changes. This ensures that the ROV can maintain satisfactory control performance even in the presence of inaccurate model parameters and significant uncertain disturbances.

After 60 s, data from 1000 sampling points should be collected to calculate the root mean square error for determining the stable accuracy of the control method, as presented in

Table 7. Stable accuracy (X, Y, Z = m, Phi = rad).

Comparison	$X$	$Y$	$Z$	$Phi$
PID	0.351	0.497	0.381	0.004
LADRC	0.0285	0.3587	1.43 × 10−5	0.003
DDPG-LADRC	5.43 × 10−5	1.14 × 10−4	1.29 × 10−14	6.56 × 10−9

.

In underwater robot control, better stability accuracy means that the robot can precisely reach the target position. Simulation results indicate that the designed DDPG-LADRC controller not only has robust performance but also possesses the ability to quickly track commands and suppress disturbances. Further comparisons show that the performance of DDPG-LADRC surpasses that of PID and conventional fixed-parameter LADRC. Therefore, parameter optimization based on DDPG can enhance the control performance of LADRC.

5.2. Scenario 2

To further verify the robustness of the controller, the anti-interference capability of different control methods under strong interference conditions was compared. The most representative tracking trajectory during the ROV’s motion was selected (Formula (51)). A dual closed-loop sliding mode control scheme based on a nonlinear extended state observer (NESO-DSMC) was added for the comparison of control methods [26], to validate the superiority of the DDPG-LADRC controller’s performance.

The parameters for the Active Disturbance Rejection Control are set as follows: $b_0=10, w_0=5$, Because ${\beta }_1=3{\omega }_o, {\beta }_2=3{\omega }_o^2, {\beta }_3={\omega }_o^3$ which means ${\beta }_1=15$, ${\beta }_2=75$, ${\beta }_3=125.$ The relevant DDPG setting parameters are shown in Table 2 above. In addition, the controller parameters proposed in the NESO-DSMC are chosen as follows: $\delta =0.01$, ${ϵ}_{11}={ϵ}_{21}={ϵ}_{31}=0.5, \beta =0.1, {\rho }_{11}=100, {\rho }_{21}=300, {\rho }_{31}=1000, {\gamma }_1={\gamma }_2={\gamma }_3={\gamma }_4=0.01, m=5, q=2, K_{\eta }=diag\{0.3, 0.3, 0.3, 0.3\},K_{\nu }=\mathrm{d}\mathrm{i}\mathrm{a}\mathrm{g}\{10, 10, 10, 10\}$ [26].

The external interference added is shown in Equation (50). The added disturbance signal is related to the state of the ROV, and this signal is constantly changing.

$$\left\{\begin{array}{c}{f}_x=40-1.65X-0.3Y^2-1.22Z^2-8Z\\ f_y=2.2X^2-2.5Y+0.3Z\\ f_z=18-2.1X^2-0.88Y^2-0.5Z^2\end{array}\right.$$

(50)

The tracked trajectory is shown in Formula (51). This trajectory indicates that the ROV first descends vertically, then performs linear back-and-forth and spiral movements on a horizontal plane, accompanied by changes in depth and adjustments in heading, ultimately returning to a horizontal straight path. The initial position and attitude of the ROV are set as: $X\left(0\right)=0 \mathrm{m}, Y\left(0\right)=1 \mathrm{m}, Z\left(0\right)=0 \mathrm{m}, Phi\left(0\right)=0 \mathrm{r}\mathrm{a}\mathrm{d}$.

$$\begin{array}{c}{x}_d(t)=\left\{\begin{array}{cc}0 \mathrm{m},0\le t\le 20 \mathrm{s}& \\ 0.2\left(t-20\right) \mathrm{m},20\le t<40 \mathrm{s}& \\ \sin \left(0.04\pi \left(t-40\right)\right)+4 \mathrm{m},40\le t<60 \mathrm{s}& \\ -0.2\left(t-60\right)+4 \mathrm{m},60\le t<80 \mathrm{s}& \\ - \mathrm{s}\mathrm{i}\mathrm{n}\left(0.05\pi \right(t-80\left)\right),80\le t<100 \mathrm{s}& \\ 0.2\left(t-100\right) \mathrm{m},100\le t\le 120 \mathrm{s}& \end{array}\right.\hfill \\ y_d(t)=\left\{\begin{array}{cc}1 \mathrm{m},0\le t\le 20 \mathrm{s}& \\ 1 \mathrm{m},20\le t<40 \mathrm{s}& \\ -\mathrm{c}\mathrm{o} \mathrm{s}\left(0.05\pi \right(t-40\left)\right)+2 \mathrm{m},40\le t<60 \mathrm{s}& \\ 3 \mathrm{m},60\le t<80 \mathrm{s}& \\ -\cos \left(0.05\pi \left(t-80\right)\right)+4 \mathrm{m},80\le t<100 \mathrm{s}& \\ 5 \mathrm{m},100\le t\le 120 \mathrm{s}& \end{array}\right.\hfill \\ z_d(t)=\left\{\begin{array}{cc}0.3t \mathrm{m},0\le t<20 \mathrm{s}& \\ 6-4\mathrm{c}\mathrm{o} \mathrm{s}\left(0.1\pi \right)+5 \mathrm{s}\mathrm{i}\mathrm{n}\left(0.1\pi x\right)+4\cos \left(0.1\pi y\right)\mathrm{m},20\le t\le 120 \mathrm{s}& \end{array}\right.\hfill \\ {\psi }_{\mathrm{d}}(t)=\left\{\begin{array}{cc}0 \mathrm{r}\mathrm{a}\mathrm{d},0\le t<20 \mathrm{s}& \\ 0 \mathrm{r}\mathrm{a}\mathrm{d},20\le t<40 \mathrm{s}& \\ 0.05\pi \left(t-40\right) \mathrm{r}\mathrm{a}\mathrm{d},40\le t<60 \mathrm{s}& \\ \pi \mathrm{r}\mathrm{a}\mathrm{d},60\le t<80 \mathrm{s}& \\ \pi -0.05\pi \left(t-80\right) \mathrm{r}\mathrm{a}\mathrm{d},80\le t<100 \mathrm{s}& \\ 0 \mathrm{r}\mathrm{a}\mathrm{d},100\le t\le 120 \mathrm{s}& \end{array}\right.\hfill \end{array}$$

(51)

The simulation results shown in Figure 11 demonstrate that the DDPG-LADRC can achieve accurate disturbance estimation for the perturbation observations and corresponding disturbance observation error curves of the three state variables $f_x,f_y,f_z$. The maximum observation error value for the observer in the $X$-direction is 0.00141, the maximum observation error in the $Y$-direction is 0.0016, and the maximum observation error in the $Z$-direction is 0.0021. DDPG-optimized LESO has achieved the estimation accuracy for disturbances that meet our requirements.

From Figure 12, it can be seen that LADRC, due to the issue of fixed parameters in the controller, is unable to eliminate steady-state errors in a short time. Under continuously changing external disturbances, LADRC cannot achieve optimal control performance. In the presence of significant uncertain disturbances, NESO-DSMC cannot reach the same level of error convergence accuracy as DDPG-LADRC.

Table 8. Root Mean Square Error (X, Y, Z = m, Phi = rad).

Comparison	$X$	$Y$	$Z$	$Phi$
LADRC	0.01	0.021	0.014	0.003
NESO-DSMC	0.005	0.0003	0.006	0.0006
DDPG-LADRC	0.0011	0.0002	0.0012	0.00029

and

Table 9. Mean Absolute Error (X, Y, Z = m, Phi = rad).

Comparison	$X$	$Y$	$Z$	$Phi$
LADRC	0.005	0.015	0.012	0.008
NESO-DSMC	0.001	0.0003	0.0007	0.00028
DDPG-LADRC	0.0006	0.0001	0.0005	0.00011

show the RMSE and MAE under different control methods, indicating that DDPG-LADRC has better robustness compared to LADRC and NESO-DSMC. DDPG-LADRC can eliminate steady-state errors within 5 s because it incorporates DDPG for online adjustment of LADRC parameters in response to uncertain disturbances caused by environmental changes, significantly improving the control accuracy of the system. This ensures that the ROV can maintain satisfactory control performance even in the presence of inaccurate model parameters and significant uncertain disturbances.

6. Conclusions

In response to the issue of underwater robots facing difficulties in determining model parameters and external disturbances, and the inability of traditional fixed-parameter controllers to achieve optimal control performance for the controlled object, an online parameter tuning strategy based on active disturbance rejection control has been proposed: the DDPG-LADRC algorithm.

1.

Based on the nonlinear model of underwater robots, dynamic parameter uncertainty was considered, and a linear active disturbance rejection controller was designed. The convergence of the extended state observer in the linear active disturbance rejection controller and the stability of the closed-loop control were proven using the Lyapunov method. To address the issue that fixed parameter controllers in nonlinear systems cannot achieve optimal control performance, a DDPG-LADRC control strategy was designed, which improved the performance of the LESO by online adjusting control parameters, resulting in the reward curve of DDPG. A feasibility analysis of parameter optimization for LESO was conducted in numerical simulations, demonstrating the effectiveness of the DDPG-LADRC strategy.

2.

Compared to PID, fixed-parameter LADRC, and the latest nonlinear observer-based double closed-loop sliding mode control method (NESO-DSMC), the DDPG-LADRC method can generate optimal parameters for the controller, thereby improving control accuracy. Experiments show that this control strategy outperforms PID, fixed-parameter LADRC, and NESO-DSMC control strategies in terms of transient performance and anti-interference capability. Therefore, it can be said that DDPG-LADRC has significant advantages in tracking and anti-interference capabilities.

3.

The algorithm, although demonstrating good performance in simulations, still faces significant challenges when being translated into practical engineering applications. For instance, the accurate determination of an ROV’s rotational inertia and hydrodynamic coefficients presents a notable challenge. In the future, the parameter adaptation concept based on DDPG can be combined with other control methods to achieve asymptotic stability and optimal control performance.