Reference Model-Based Deterministic Policy for Pitch and Depth Control of Autonomous Underwater Vehicle

Abstract

The Deep Reinforcement Learning (DRL) algorithm is an optimal control method with generalization capacity for complex nonlinear coupled systems. However, the DRL agent maintains control command saturation and response overshoot to achieve the fastest response. In this study, a reference model-based DRL control strategy termed Model-Reference Twin Delayed Deep Deterministic (MR-TD3) was proposed for controlling the pitch attitude and depth of an autonomous underwater vehicle (AUV) system. First, a reference model based on an actual AUV system was introduced to an actor–critic structure, where the input of the model was the reference target, the outputs were the smoothed reference targets, and the reference model parameters can adjust the response time and the smoothness. The input commands were limited to the saturation range. Then, the model state, the real state and the reference target were mapped to the control command through the Twin Delayed Deep Deterministic (TD3) agent for training. Finally, the trained neural network was applied to the AUV system environment for pitch and depth experiments. The results demonstrated that the controller can eliminate the response overshoot and control command saturation while improving the robustness, and the method also can extend to other control platforms such as autonomous guided vehicle or unmanned aerial vehicle.

1. Introduction

With the development of ocean exploration, the applications of autonomous underwater vehicles (AUVs) have become more extensive, and they include ocean resource exploration, military applications, and mapping. The complex application environment raises the demand for the control of AUVs [1,2]. Owing to the highly complex and time-varying working environment that is heightened by climate change, AUVs are subject to various uncertainties, which affect their motion stability and reliability [3,4]. Therefore, designing a robust controller that considers uncertainty factors has attracted the attention of many researchers.

The complex and changing environment, along with the underactuated Multi-Input–Multi-Output (MIMO) controlled objects, places intense demands on the controller design. As a neural-network controller, DRL has natural processing advantages. DRL algorithms are increasingly being adopted in control fields. DRL methods are divided into two branches: value-based [5] and policy-based [6,7]. Standard algorithms in these two methods include Deep Q-Networks (DQN) [8], PPO [9], SAC [10], Deep Deterministic Policy Gradient (DDPG) [7], and Twin Delayed DDPG (TD3) [11]. The Proximal Policy Optimization (PPO) algorithm was applied in the AUV attitude control field and was proved to be efficient [12]. One study introduced multi-state space and multi-action space schemes to control the AUV attitude [13], and another mentioned DDPG to maintain the attitude of the UAV [14,15]. Reference [16] used DDPG algorithms to control the depth of an AUV and compared it with a model-based method. Here, it was found that the DRL method performed better than Nonlinear Model Predictive Control (NMPC) and Linear–Quadratic–Gaussian (LQG) approaches. The DQN algorithm is applied to the path tracking of multi-formation AUV controllers on the water surface, and the result is good in practical applications [17]. The deterministic artificial intelligence developed for UUVs in JMSE’s best paper in 2020 by Sands was also applied to UUV actuator DC motors by Shah in 2021 and improved by Koo in 2022 [18]. Zhai (2022) directly compared deterministic artificial intelligence to deep learning with neural networks in [19] and then proposing signal-encoded deep learning in [20]. Hui Wu used the DRL method to manipulate the depth and test the proposed method on a seafloor data set sampled from the South China Sea [16]. An integral reinforcement learning-based adaptive neural network (NN) tracking control is developed in [21]. All these studies approved that DRL can control the MIMO model perfectly, but none of these considered the disturbance.

This algorithm is also used in unmanned aerial vehicle (UAV) attitude control, and the DRL controller is optimized for unseen disturbances in the form of wind and turbulence even under conditions of severe disturbances [22]. This study also introduced a double loop for the controller, whereby the inner DRL controller provids stability and control, while an “outer loop” is responsible for the mission-level objectives. Huang in [23] proposed an RL method considering actuator faults. As an optimal controller, the DRL controller in the former literature just considered the ultimate result; the aforementioned study illustrates that DRL algorithms are suitable for controlling unknown disturbances and complex-model systems. However, neither the amount of overshoot nor saturation of the actuators during the control process is described, which reduces the capacity to deal with external disturbances. If a machine runs in a saturated state for a long time, it can easily damage the actuator [24].

Mai The Vu proposed a robust technique for anti-disturbance [25], and Ref. [26] proposed the robust station-keeping (SK) control algorithm based on a sliding mode control (SMC) theory to guarantee stability and better performance despite the existence of model uncertainties and ocean current disturbance. Model reference control refers to an adaptive control method that enables the dynamic characteristics of the controlled object and the known reference model to be as close as possible using a designed adaptive mechanism [27]. A new model reference adaptive control algorithm provides robust stability for unmodeled plant uncertainties. It is designed to cope with the uncertainty and time variability of system parameters [28]. The model-reference self-adaptive controller provides robust stability concerning unmodeled plant uncertainties. The goal of a robust and adaptive controller is to eliminate the uncertainty of the dynamic system model and external disturbances [29] such that self-adaptive control can enable an AUV system to adapt to uncertainties and disturbances in the environment. In [30], an adaptive nonlinear controller for the diving control of an AUV was presented. A simple modification is proposed to break the restricting condition on the vehicle’s pitch angle during a diving motion such that the AUV can perform free pitch motion. The anti-integral saturation adaptive controller plays a vital role in the pitch and yaw attitude angle control of AUVs [31]. In [32,33], the proposed algorithm incorporating adaptive control with anti-windup compensators provided a convenient combination to counteract the challenge. Composite model reference adaptive control offers better tracking, more robust interference rejection, and faster recovery from thruster failures, and it works well when the AUV system fails [34]. The above methods have robustness and anti-jamming capability but are not sufficient for exploring and controlling significant unknown disturbances.

To solve the above problem, this study combines the DRL algorithm and the model reference algorithm, retains the processing capability of the DRL algorithm deep neural network for complex nonlinear coupled systems, and introduces the smoothing of the reference model such that the new algorithm has the dual advantages of eliminating the amount of overshooting control and generalizing against disturbances. The main contributions of this study are as follows:

• A reference model is proposed in this paper. The model is established based on the actual system model. The input is the target, and the output is the reference target, which can be seen as a baseline for the agent controller to follow, while the observed state of the reference model should be considered the reference process value. The parameters of the reference model can adjust the smoothness of the reference target and the response time. When adjusting the response time, the control command must be left with some redundancy and must not reach the saturation domain to deal with disturbances.
• We formulated MR-TD3 structure, combining the reference model and reinforcement learning agent to establish the MR-TD3 controller. The inputs of the TD3 controller are the reference target state and the actual state, respectively, and the corresponding mapped command objective of the controller is to converge the deviation of the reference target and the actual value to zero.
• Multiple simultions, including a step and sine response and overshoot experiment as well as a various mass and initial depth state roubust experiment, are designed to verify the performance of the proposed method.

The remainder of this paper is organized as follows. In Section 2, the kinematic and dynamic models of the AUV system are presented along with the mathematical equations for the pitch and depth. The algorithmic structure of MR-TD3 is explained in Section 3. We test the MR-TD3 method in various simulated environments and compare the results with those of other algorithms in Section 4. Conclusions and an outlook on improvement directions are presented in Section 5.

2. Model Description and Problem Formulation

A mathematical model of the AUV was required for the controller designing. For this reason, the AUV model is presented in this section.

2.1. Assumption

As the application scenario is docking [35], the control objectives were pitch and depth, so we were only interested in the vertical plane of the x/z axis. Therefore, only the pitch and depth control problems are described in this section. To simplify the model of the system, we assumped the AUV worked under the following situations [36]:

• The motion plane of the AUV is the x/z axis, and the three DOFs are surge, heave, and yaw.  Roll, pitch, and sway motion were neglected.
• The AUVs are symmetrical, and the buoyant center coincides with the origin of the body coordinate system.
• The AUV equation did not consider the disturbance forces (waves, winds, and ocean currents).

2.2. Coordinate Systems of AUVs

In this study, we modeled an underactuated AUV system developed by Hydroid in the United States [37,38], which was defined in two sets of coordinate systems: a body-fixed reference (B-frame) and an earth coordinate system (North–East–Down Reference, NED-frame) [39,40], as shown in Figure 1. The AUV system has six-degrees-of-freedom in the environment and Equations (1)–(7) can be used to describe the dynamic characteristics of the system.

$$M\dot{\nu }+C\left(\nu \right)v+D\left(\nu \right)v+g\left(\eta \right)=\tau \left(\eta \right),$$

(1)

$$M=M_{rb}+M_a$$

(2)

$$C\left(\nu \right)=C_{rb}\left(\nu \right)+C_a\left(\nu \right)$$

(3)

$$D\left(\nu \right)=D_P\left(\nu \right)+D_S\left(\nu \right)+D_W\left(\nu \right)+D_M\left(\nu \right)$$

(4)

$$g\left(\eta \right)=\left[\begin{array}{c}(W-B)sin\theta \\ -(W-B)cos\theta sin\phi \\ -(W-B)cos\theta cos\phi \\ -\left(y_{G}W-y_{G}B\right)cos\theta cos\phi +\left(z_{G}W-z_{B}B\right)cos\theta sin\phi \\ \left(z_{G}W-z_{B}B\right)sin\phi +\left(x_{G}W-x_{B}B\right)cos\theta cos\phi \\ -\left(x_{G}W-x_{B}B\right)cos\theta sin\phi -\left(y_{G}W-y_{B}B\right)sin\theta \end{array}\right]$$

(5)

$$\tau ={\tau }_p+{\tau }_r,$$

(6)

where M is the mass matrix, which consists of a rigid body mass matrix and additional masses of the AUV system, $C\left(\nu \right)$ is the Coriolis force matrix and additional mass-induced Coriolis force matrix, and $D\left(v\right)$ is the damping matrix. Here, $D_P$ is potential flow damping, $D_S$ is friction damping, $D_W$ is hinged wave damping, and $D_M$ is vortex shedding damping, where $D_P$ and $D_M$ can be neglected in practical applications. $g\left(\eta \right)$ is the reversion moment matrix, which consists of gravity G and buoyancy W. $r_G={[x_g,y_g,z_g]}^T$ and $r_B={[x_b,y_b,z_b]}^T$ represent the center of gravity and center of buoyancy in the coordinate system. $\tau$ is the drive force of the AUV, ${\tau }_p$ is the propulsion of the propeller and ${\tau }_r$ is the rudder force.

The position state of the AUV vector is described by $\eta ={[x,y,z,\phi ,\theta ,\psi ]}^T$, where the position vector $\eta$ is used to describe the absolute position $(x,y,z)$ and attitude angle $(\varphi ,\theta ,\psi )$ [21], which are termed surge, sway, heave, roll, pitch, and yaw, respectively. The velocity vector $v={[u,v,w,p,q,r]}^T$ represents the three-axis linear velocity $(u,v,w)$ and the three-axis angular velocity $(p,q,r)$ in the body coordinate system. The kinematic relationship between the body coordinate system and the inertial coordinate system of the underwater AUV is

$$\dot{\eta }=J\left({\eta }_2\right)\dot{v},$$

(7)

The above two vectors can be transformed into each other, and the transformation formula of the velocity vector of the motion in a two-coordinate system is as follows:

$$\left[\begin{array}{c}\dot{x}\\ \dot{y}\\ \dot{z}\end{array}\right]=J_1\left({\eta }_2\right)\left[\begin{array}{c}u\hfill \\ v\hfill \\ w\hfill \end{array}\right].$$

(8)

The attitude vector can be expressed as follows:

$$\left[\begin{array}{c}\dot{\phi }\\ \dot{\theta }\\ \dot{\psi }\end{array}\right]=J_2\left({\eta }_2\right)\left[\begin{array}{c}p\\ q\\ r\end{array}\right],$$

(9)

where

$$\begin{array}{c}{J}_1\left({\eta }_2\right)=\left[\begin{array}{ccc}cos\psi cos\theta & -sin\psi cos\phi +cos\psi sin\theta sin\phi & sin\psi sin\phi +cos\psi cos\phi sin\theta \\ sin\psi cos\theta & cos\psi cos\phi +sin\phi sin\theta sin\psi & -cos\psi sin\phi +sin\theta sin\psi cos\phi \\ -sin\theta & cos\theta sin\phi & cos\theta cos\phi \end{array}\right]\end{array},$$

(10)

$$\begin{array}{c}{J}_2\left({\eta }_2\right)=\left[\begin{array}{ccc}1& sin\phi tan\theta & cos\phi tan\theta \\ 0& cos\phi & -sin\phi \\ 0& \frac{sin\phi }{cos\theta }& \frac{cos\phi }{cos\theta }\end{array}\right]\end{array}.$$

(11)

2.3. AUV Vertical Plane Control Description

To simplify the system model, we consider only the motion of the AUV platform in the vertical-axis model and the relationship between the world frame state $(x,z,\theta )$ and body frame state $(u,w,q)$.

The motion state variables can be defined as $s=[u,w,q,x,z,\theta ]$, where u is the speed of the x-axis in the horizontal direction, z is the depth position, w is the velocity in the z-axis direction, $\theta$ is the pitch attitude angle, and q is the pitch attitude angular velocity. The motion relationship can be described by Equation (12):

$$\begin{array}{cc}\hfill & \dot{x}\left(t\right)=u\left(t\right),\hfill \\ \hfill & \dot{z}\left(t\right)=w\left(t\right),\hfill \\ \hfill & \dot{\theta }\left(t\right)=q\left(t\right).\hfill \end{array}$$

(12)

The equation below describes the relationship between the linearized dynamic equation and the linearized external force and moment,

$$\begin{array}{cc}\hfill & \dot{u}\left(t\right)=\frac{-X_{u}u\left(t\right)+X_T\left(t\right)}{m+X_{\dot{u}}},\hfill \\ \hfill & \dot{w}\left(t\right)=\frac{-Z_{w}w\left(t\right)-Z_{\dot{q}}\dot{q}\left(t\right)-Z_{q}q\left(t\right)+Z_s\left(t\right)+Z_b\left(t\right)}{m+Z_{\dot{w}}},\hfill \\ \hfill & \dot{q}\left(t\right)=\frac{-M_{q}q\left(t\right)-M_{\dot{w}}\dot{w}\left(t\right)-M_{w}w\left(t\right)-x_s{Z}_s\left(t\right)+x_b{Z}_b\left(t\right)}{I_y+M_{\dot{q}}},\hfill \end{array}$$

(13)

where $X_u,Z_w,Z_q,M_q,M_u,$ and $M_w$ are linear drag coefficients, $X_T,Z_s,$ and $Z_b$ are outside drivers, $X_{\dot{u}},Z_{\dot{w}},$ and $M_{\dot{q}}$ are added mass matrices, $I_y$ is the mass moment of inertia about the y axis, and the other parameters are the same as before. Among the above variables, those that can be measured by the outside world are only x, z, and $\theta$. Therefore, the complete AUV dynamic model in the form of state space equations is

$$\begin{array}{c}\dot{x}=Ax+Bu,\\ y=Cx,\end{array}$$

(14)

where A is the state space matrix, B is the control matrix, and C is the output matrix. All the state space parameters can be described in detail as Equations (15) and (16). Here, the parameters ${\sigma }_1=I_y{Z}_{\dot{w}}+M_{\dot{q}}{Z}_{\dot{w}}-M_{\dot{w}}{Z}_{\dot{q}}+I_{y}m+M_{\dot{q}}m$, and all the AUV system parameters are shown in

Table 1. Parameters of the AUV.

Parameter	Value	Parameter	Value	Parameter	Value	Parameter	Value
m	50 kg	$X_{\dot{u}}$	$16.6667$ kg	$K_p$	$-0.19$ kgms${}^{-1}$	$K_{p\left|p\right|}$	$-32.2$ kgm${}^{-1}$
L	$2.5$ m	$Y_{\dot{v}}$	$-12.23$ kg	$M_q$	$-0.23$ kgms${}^{-1}$	$M_{q\left|q\right|}$	$-96.2$ kgm${}^{-1}$
r	$0.45$ m	$Z_{\dot{w}}$	$-15.87$ kg	$N_r$	$-0.23$ kgms${}^{-1}$	$N_{r\left|r\right|}$	$-15.3$ kgm${}^{-1}$
$I_x$	$0.297$ kgm${}^2$	$K_{\dot{p}}$	$-0.63$ kg	$X_u$	94 kgms${}^{-1}$	$X_{u\left|u\right|}$	$-15.7$ kgm${}^{-1}$
$I_y$	$1.0417$ kgm${}^2$	$M_{\dot{q}}$	$-0.78$ kg	$Y_v$	$-0.54$ kgms${}^{-1}$	$Y_{v\left|v\right|}$	$-21.2$ kgm${}^{-1}$
$I_z$	$1.304$ kgm${}^2$	$N_{\dot{r}}$	$-0.56$ kg	$Z_w$	$-0.54$ kgms${}^{-1}$	$Z_{w\left|w\right|}$	$-17.2$ kgm${}^{-1}$

. Due to materials and assembly, the parameters of the system may change, and the intuitive indicator is the mass, which leads to the changes of the system model. From Equation (15), we can find that if the mass changed, the state space model of the AUV system will be changed. The open loop system’s matrices variables A, B, and C can be given, and the state space equation can be written as Equation (17).

$$\begin{array}{cc}\hfill & \left[\begin{array}{c}\dot{u}\\ \dot{w}\\ \dot{q}\\ \dot{x}\\ \dot{z}\\ \dot{\theta }\end{array}\right]=\left[\begin{array}{cccccc}\frac{-X_u}{x_{\dot{u}}+m}& 0& 0& 0& 0& 0\\ 0& \frac{-\left(I_y{z}_w+M_{\dot{q}}{z}_w-M_w{z}_{\dot{q}}\right)}{{\sigma }_1}& \frac{-\left(I_y{z}_q-M_q{z}_{\dot{q}}+M_{\dot{q}}{z}_q\right)}{{\sigma }_1}& 0& 0& 0\\ 0& \frac{-\left(M_w{z}_w-M_w{z}_w+M_{w}m\right)}{{\sigma }_1}& \frac{-\left(M_q{z}_{\dot{w}}-M_{\dot{w}}{z}_q+M_{q}m\right)}{{\sigma }_1}& 0& 0& 0\\ 1& 0& {\sigma }_1& 0& 0& 0\\ 0& 1& 0& 0& 0& 0\\ 0& 0& 1& 0& 0& 0\end{array}\right]\left[\begin{array}{c}u\\ w\\ q\\ x\\ z\\ \theta \end{array}\right]+\hfill \\ \hfill & \left[\begin{array}{ccc}\frac{1}{x_{\dot{u}}+m}& 0& 0\\ 0& \frac{\left(I_y+M_{\dot{q}}+Z_{\dot{q}}{x}_s\right)}{{\sigma }_1}& \frac{\left(I_y+M_{\dot{q}}+Z_{\dot{q}}{x}_b\right)}{{\sigma }_1}\\ 0& \frac{-\left(M_w+Z_w{x}_s+m{x}_s\right)}{{\sigma }_1}& \frac{\left(Z_w{x}_b-M_w+m{x}_b\right)}{{\sigma }_1}\\ 0& 0& 0\\ 0& 0& 0\\ 0& 0& 0\end{array}\right]\left[\begin{array}{c}{X}_T\hfill \\ Z_s\hfill \\ Z_b\hfill \end{array}\right]\hfill \end{array},$$

(15)

$$\left[\begin{array}{c}x\hfill \\ z\hfill \\ \theta \hfill \end{array}\right]=\left[\begin{array}{cccccc}0\hfill & 0\hfill & 0\hfill & 1\hfill & 0\hfill & 0\hfill \\ 0\hfill & 0\hfill & 0\hfill & 0\hfill & 1\hfill & 0\hfill \\ 0\hfill & 0\hfill & 0\hfill & 0\hfill & 0\hfill & 1\hfill \end{array}\right]\left[\begin{array}{c}u\hfill \\ w\hfill \\ q\hfill \\ x\hfill \\ z\hfill \\ \theta \hfill \end{array}\right].$$

(16)

$$\begin{array}{cc}\hfill & \left[\begin{array}{c}\dot{u}\\ \dot{w}\\ \dot{q}\\ \dot{x}\\ \dot{z}\\ \dot{\theta }\end{array}\right]=\left[\begin{array}{cccccc}-0.1819& 0& 0& 0& 0& 0\\ 0& -0.8422& -1.6830& 0& 0& 0\\ 0& -0.2725& -0.9656& 0& 0& 0\\ 1& 0& 0& 0& 0& 0\\ 0& 1& 0& 0& 0& 0\\ 0& 0& 1& 0& 0& 0\end{array}\right]\left[\begin{array}{c}u\\ w\\ q\\ x\\ z\\ \theta \end{array}\right]+\left[\begin{array}{ccc}0.0019& 0& 0\\ 0& 0.0015& 0.0022\\ 0& 0.0076& -0.0077\\ 0& 0& 0\\ 0& 0& 0\\ 0& 0& 0\end{array}\right]\left[\begin{array}{c}{X}_T\hfill \\ Z_s\hfill \\ Z_b\hfill \end{array}\right]\hfill \\ \hfill & \left[\begin{array}{c}x\hfill \\ z\hfill \\ \theta \hfill \end{array}\right]=\left[\begin{array}{cccccc}0\hfill & 0\hfill & 0\hfill & 1\hfill & 0\hfill & 0\hfill \\ 0\hfill & 0\hfill & 0\hfill & 0\hfill & 1\hfill & 0\hfill \\ 0\hfill & 0\hfill & 0\hfill & 0\hfill & 0\hfill & 1\hfill \end{array}\right]\left[\begin{array}{c}u\\ w\\ q\\ x\\ z\\ \theta \end{array}\right]\hfill \end{array}.$$

(17)

2.4. DRL Algorithm TD3 Description

DRL aims to discuss how agents can maximize their rewards in complex and uncertain environments. Reinforcement learning consists of agents and the environment interacting continuously during the learning process [16]. In this section, the DRL framework controls the errors between the actual system and reference model to converge the deviation to zero. Because the AUV system is a continuous control system, and various control variables are continuous, this study uses the strategic gradient algorithm for control. Both TD3 and the DDPG algorithms are typical applications of the strategic gradient algorithm, and the TD3 algorithm is a modified version of the DDPG algorithm. Because a comparison of the two is required later, only the difference between the two, that is, the improvement of TD3 is introduced here.

The TD3 algorithm, as a gradient policy with actor–critic, was used to solve the problem of Q overestimation, in which the following essential methods were adopted. First, the delayed update strategy function method was adopted to solve the coupling problem between the valuation function critic and actor policy functions. Second, the actor network was updated only once, whereas the critic network was updated every n times. The target network was then used to reduce the accumulation of errors, reduce variance reduction, and calculate the Q value, adding less noise to the action. The high variance build-up and the loss function are given by,

$$L\left({\theta }^{Q_{predict\_1}}\right)={\mathbb{E}}_{\left(s,a,s^{{}^{\prime }}\right)\sim \mathcal{B}}\left[\frac{1}{2}{\left(R\left(s^{{}^{\prime }}\right)+\gamma \underset{i=1,2}{min}{Q}_{\mathrm{target}\_i}\left(s^{{}^{\prime }},{\theta }^{{\mu }_{target\_i}}(s^{{}^{\prime }};{\theta }^{{\mu }_{target\_i}})+{ϵ}_r;{\theta }^{Q_{target\_i}}\right)-Q_{preject\_1}(s,a;{\theta }^{Q_{preject\_1}})\right)}^2\right],$$

(18)

$$L\left({\theta }^{Q_{predict\_2}}\right)={\mathbb{E}}_{\left(s,a,s^{{}^{\prime }}\right)\sim \mathcal{B}}\left[\frac{1}{2}{\left(R\left(s^{{}^{\prime }}\right)+\gamma \underset{i=1,2}{min}{Q}_{\mathrm{target}\_i}\left(s^{{}^{\prime }},{\theta }^{{\mu }_{target\_i}}(s^{{}^{\prime }};{\theta }^{{\mu }_{target\_i}})+{ϵ}_r;{\theta }^{Q_{target\_i}}\right)-Q_{preject\_2}(s,a;{\theta }^{Q_{preject\_2}})\right)}^2\right].$$

(19)

If mod d, then

$$L\left({\theta }^{{\mu }_{\mathrm{predict}}}\right)=-J\left({\theta }^{{\mu }_{\mathrm{predict}}}\right)=-{\mathbb{E}}_{S\sim \epsilon }\left[Q_{\mathrm{prodict}\_1}\left(s,{\mu }_{\mathrm{predict}}\left(s;{\theta }^{{\mu }_{\mathrm{predict}}}\right);{\theta }^{Q_{\mathrm{prodict}\_1}}\right)\right],$$

(20)

where $\mathcal{B}$ denotes the replay buffer, ${ϵ}_r$ represents random noise, d denotes the policy frequency, and $R\left(s^{\prime }\right)$ denotes the immediate reward r.

It is easy to overestimate the Q value because $argmaxQ\left(s^{\prime }\right)$ is used to replace $V\left(s^{\prime }\right)$ and evaluate $Q\left(s\right)$. In TD3, two sets of networks were used to estimate the Q value, where the smaller one is used as the update target. When a value function is updated, a minor $ϵ$ disturbance is added to the input action each time. This operation is a target policy smoothing regulation. Policy noise was added to the target network when a parameter became available to make the assessment more accurate. TD3 is a deterministic policy algorithm, and the policy gradient function is

$${\nabla }_{\varphi }J\left(\varphi \right)={\left.N^{-1}\sum {\nabla }_a{Q}_{{\theta }_1}(s,a)\right|}_{a={\pi }_{\varphi }\left(s\right)}{\nabla }_{\varphi }{\pi }_{\varphi }\left(s\right).$$

(21)

3. Proposed Method

The kinematics model was introduced in detail in Section 2. In this section, we introduce the implementation of acquiring the reference model from the basic kinematic model. In this study, to simplify the control model, the higher-order term near the working point was ignored. We can approximate the linear model equation in Section 3.1.1. We show the design process of the state space, action space, and reward function. Thereafter, the reference model was combined with the TD3 agent to formalize the MR-TD3 controller. From the perspective of the control theory, the architecture of the control system is shown in the block diagram in Figure 2, and each component is described in the following sections.

3.1. Control Method

Considering the nonlinearity and strong coupling of the system, a DRL neural network plays a critical role. The controller’s ability to prevent saturation and eliminate overshoot is also vital, and the part of the reference model serves as a baseline to guide the controlled target forward, thus smoothing the target to avoid an excessive overshoot. The maximum control command interval was considered in the design process of the reference model, which can prevent command control saturation and thus leave a control margin for external disturbances. Suppose that the target of the controller is a direct input target. This variable will be massive, which will cause the corresponding control variable to be relatively large, thus reaching the upper limit of the controller actuator. If there is any disturbance at this time, there will be no redundant control commands to deal with interference, and the robustness will be very poor. The role of the reference model was to track the target of the input. The parameters of the reference model can adjust the response time, and the output contains all observed variables, which is the derivation of the target and the reference target. The deviations between these values are relatively small, and the goal of the controller was to drive the deviation to zero.

3.1.1. Design of the Reference Model

Since the pitch and depth are a pair of coupling values in the underdriven system, the coupling quantities of the two channels are removed in the process of designing the reference model; that is, the coupling quantities can be regarded as external disturbances and removed to avoid the disturbance of the coupling quantities. So, the AUV system model without coupled values can be written as follow state space equation

$$\begin{array}{ccc}\hfill & \left[\begin{array}{c}\dot{u}\\ \dot{w}\\ \dot{q}\\ \dot{x}\\ \dot{z}\\ \dot{\theta }\end{array}\right]=\left[\begin{array}{cccccc}-0.1819& 0& 0& 0& 0& 0\\ 0& -0.8422& 0& 0& 0& 0\\ 0& 0& -0.9656& 0& 0& 0\\ 1& 0& 0& 0& 0& 0\\ 0& 1& 0& 0& 0& 0\\ 0& 0& 1& 0& 0& 0\end{array}\right]\left[\begin{array}{c}u\\ w\\ q\\ x\\ z\\ \theta \end{array}\right] +\hfill & \hfill \left[\begin{array}{ccc}0.0019& 0& 0\\ 0& 0.0015& 0\\ 0& 0& -0.0077\\ 0& 0& 0\\ 0& 0& 0\\ 0& 0& 0\end{array}\right]\left[\begin{array}{c}{X}_T\hfill \\ Z_s\hfill \\ Z_b\hfill \end{array}\right]\end{array}.$$

(22)

A reference model was designed, whose output was the reference target variable that corresponded to the physical meaning of the input target variable. The reference model of the corresponding AUV system was designed according to the state space Equations (14)–(16)

$$\begin{array}{c}{\dot{x}}_m=A_m{x}_m+B_{m}r\\ y_m=C_m{x}_m\end{array}.$$

(23)

Here, r is the reference input target, where $C_m=C$, $e=x-x_m$ is set, and the state variable deviation can be calculated using the above formula,

$$\dot{e}=A_{m}e+\left(A-A_m\right)x+Bu-B_{m}r$$

(24)

We make the system satisfy the following conditions:

$$A_m-A=B{K}_1,B_m=B{K}_2,$$

(25)

such that,

$$\dot{e}=A_{m}e-B\left(K_{1}x+K_{2}r-u\right).$$

(26)

To ensure that output $y_m$ tracks the target variable r, the DC gain should be adjusted to one. Assuming infinite time, we drive the following formula

$$A_m{x}_m+B_{m}r=0.$$

(27)

Output $y_m$ can be expressed as,

$$y_m=-C_m{A}_m^{-1}{B}_{m}r=-C_m{A}_m^{-1}B{K}_{2}r,$$

(28)

such that

$$K_2={\left(-C_m{A}_m^{-1}B\right)}^{-1},$$

(29)

and

$$B_m=B{K}_2=B{\left(-C_m{A}_m^{-1}B\right)}^{-1}.$$

(30)

The parameter $A_m$ can be adjusted using Matlab.

3.1.2. Simulation of the Reference Model

After adjustment and verification, considering both the response time of the reference model output value to the actual target and the constraints of the first-order derivation and control command of the mechanism in the actual system, the reference model state matrix $A_m$ and the control matrix $B_m$ can be written as follows:

$$A_m=\left[\begin{array}{cccccc}-2.919& 0& 0& -1.515& 0& 0\\ 0& -0.9022& 0& 0& -0.08& 0\\ 0& 0& -0.9656& 0& 0& -0.055\\ 1& 0& 0& 0& 0& 0\\ 0& 1& 0& 0& 0& 0\\ 0& 0& 1& 0& 0& 0\end{array}\right],$$

(31)

$$B_m=\left[\begin{array}{ccc}1.515& 0& 0\\ 0& 0.08& 0\\ 0& 0& 0.055\\ 0& 0& 0\\ 0& 0& 0\\ 0& 0& 0\end{array}\right].$$

(32)

The reference model’s depth and pitch step response are shown in Figure 3, where the depth response time is 30 s without overshoot, the maximum first-order derivative dive velocity w is $0.4$ m/s, and the peak second-order reference derivative dive acceleration ${\dot{w}}_r$ is $0.45$ m/s${}^2$, because when the command is full, the maximum acceleration is $0.5$ m/s${}^2$. This indicated that in the tracking process, no matter what happend, the control command will not reach the saturation zone, because the reference model state is the biggest controlled value in this process. At the same time, pitch $\theta$, as the other channel, is the indicator to be referred to for step response. The maximum pitch is $\pi /3$ rad, the response time is about 25 s, the maximum angular velocity is $0.16$ rad/s, and the maximum pitch acceleration is $0.2$ rad/s${}^2$, which is also within the range of the actuator. Both of them are in the acceptable domain and will not get into the input saturation zone.

3.2. Neural Network Approximators

Just as described in Section 2.4, we construct a three-layer neural network to train the AUV system. The evaluation network has 12 states as the input, and there are three driver commands as the output commands.

The control strategy consists of three layers of the neural network: the input state quantity is s, and the output control quantity is u. Due to the limitation and constraint of the control quantity range of the actual AUV system, the output control quantity must be controlled within the actual range, so the tanh unit is chosen as the output activation layer, and the output range after activation is $[-1,1]$, based on which the actual extuator range is scaled up and down. The hidden layer is also an important layer of the neural network, so the ReLu unit is chosen as the activation function of the hidden layer. The final structure of the neural network is shown in Figure 4.

3.3. Integrated Controller of MR-TD3

In this study, an AUV control system with a reference model-based controller is proposed in Section 3.1. The DRL agent TD3 is described in Section 2.4, where the reference signal was generated according to the control objective, the AUV system was analyzed, and the position, surge velocity, depth, dive velocity, pitch angle, and angular velocity were selected as the control variables. The errors are used as the state variables to form the reference model of the AUV system. Because our proposed controller consists of the TD3 algorithm and the model reference control method, it is termed the MR-TD3 controller, and the block diagram of MR-TD3 is shown in Figure 5.

In reinforcement learning, such as pendulum, the reward function includes the angle, angular velocity, and control volume, which does not consider the longtime command range. In contrast, for the AUV control system, we should consider not only the absolute error of the system but also the response time, overshoot, and actuator command scope. There are many types of AUV system states, such as depth, velocity, acceleration, attitude, angle, angular velocity, and angular acceleration. The AUV control system should consider both the response time and system stability. From the controller principle structure in Figure 2 and the DRL agent shown in Figure 5, we can see that the input state consists of the state of the reference model and the reference target variable. Therefore, the reward function is designed in the following form in Figure 2 and Figure 5; one is represented by the block diagram of the control system and the other is represented by the block diagram of reinforcement learning. Both names are slightly different, but the meanings are the same.

$$r_i=-\left[{\lambda }_1\left(\left|y_{Rm}-\mathrm{y}\right|\right)+{\lambda }_2|x_{Rm}|+{\lambda }_3\left|u\right|\right],$$

(33)

where $y_{Rm}$ is the output of the reference model, y is the output of the actual controlled object, $x_{Rm}$ is the observed state output of the reference model, and u is the control command, in the DRL agent, that is, the output action of TD3. In the system control process, there is an intuitive understanding that the less energy consumed, the better. Therefore, a reward function is established for the power that directly corresponds to the control value u to make the system controller move toward the target direction and avoid spinning in place.

The coefficients of all the reward variables are ${\lambda }_1=1,{\lambda }_2=0.1,\mathrm{and} {\lambda }_3=0.01$.

3.4. Flowchart and Pseudocode

Unlike stochastic strategies, deterministic strategies require additional control variables as opposed to a random exploration of state actions. Our approach was to add noise to the action output, and the actuator range determined the max command value of the action.

$$\overline{A}\left(s_t\right)=A^{\mu }\left(s_t\right)+n_t,$$

(34)

where $n_t$ represents normally distributed noise $N\left(0,{\sigma }^2\right)$. The actor–critic network structure of MR-TD3 is shown in Figure 5. When optimizing the evaluation network, the goal of the gradient strategy is no longer to track the target angle but to keep the state of the AUV close to the output state of the reference model, which avoids saturation of the control commands. The reference model state vector $\left[\theta ,q,{\delta }_b\right]$ was used for pitch control, whereas $\left[z,w,\theta ,q,{\delta }_s,{\delta }_b\right]$ was used for depth control. The neural network structure was designed according to the actor–critic and TD3 structures [11].

The control objective of MR-TD3 was to optimize the control volume output with an action layer of three, a state layer of twelve, and a hidden layer of 256 × 256. In AUV control applications, the actions must be restricted in the reinforcement learning action space; that is, the agent control instructions are within the action space of the execution structure. In practical applications, a sigmoid function was used at the output level to restrict the control signal to a reasonable range and to introduce deviations into the AUV controller. The purpose of this approach is to let the error integral be considered as the control input to the policy network. The actor–network input includes the AUV reference model state, the actual AUV state, and the integration error, and the critic network input is the action of the actor output. During the optimal evaluation of the network, the goal of the gradient strategy is no longer to track the target angle but to make the AUV output state follow the output state value of the reference model, and this process eliminates the saturation of the control commands.

The MR-TD3 algorithm was designed in episode mode. First, the parameters of the critic network were cloned as the target critic network whereafter the state was initialized. Each state must be within a suitable range, and all state transition models were stored in the replay buffer. When the buffer storage capacity was sufficient, the minimum min-batch was collected and the training process was initiated. The  algorithm flow is shown in Algorithm 1.

Algorithm 1 MR-TD3 for the Depth and Pitch controller of the AUV

1: Initialize critic networks $Q_{{\theta }_1},Q_{{\theta }_2}$ and actor network ${\pi }_{\varphi }$ with random parameters ${\theta }_1,{\theta }_2,\varphi$ 2: Initialize target networks ${{\theta }_1}^{\prime }\leftarrow {\theta }_1,{{\theta }_2}^{\prime }\leftarrow {\theta }_2,{\varphi }^{\prime }\leftarrow \varphi$ 3: Initialize replay buffer $\mathcal{B}$ 4: for $t_1=1$ to $T_1$ do 5:    for $t_2=1$ to $T_2$ do 6:      Select action with exploration noise $a\sim {\pi }_{\varphi }\left(s\right)+ϵ,ϵ\sim \nu (0,\sigma )$ 7:      Interact with environment $(r,s^{{}^{\prime }})\leftarrow ENV(s,a)$ 8:      Save the transition $\mathcal{B}.save(s,a,r,s^{{}^{\prime }})$ 9:      Sample mini-batch of N transition from $\mathcal{B}$: $(s,a,r,s^{{}^{\prime }})\leftarrow \mathcal{B}.sample\left(batc{h}_{num}\right)$ 10:      $\tilde{a}\leftarrow {\pi }_{\varphi }^{{}^{\prime }}\left(s^{{}^{\prime }}\right)+ϵ,ϵ\sim clip(\mathcal{N}(0,\tilde{\sigma }),-c,c)$ 11:      $y\leftarrow r+\gamma \underset{i=1,2}{min}{Q}_{{\theta }_i^{{}^{\prime }}}\left(s^{{}^{\prime }},\tilde{a}\right)$ 12:      Update critics ${\theta }_i\leftarrow {argmin}_{{\theta }_i}{N}^{-1}\sum {\left(y-Q_{{\theta }_i}(s,a)\right)}^2$ 13:    end for 14:    Update $\varphi$: ${\nabla }_{\varphi }J\left(\varphi \right)={\left.N^{-1}\sum {\nabla }_a{Q}_{{\theta }_1}(s,a)\right|}_{a={\pi }_{\varphi }\left(s\right)}{\nabla }_{\varphi }{\pi }_{\varphi }\left(s\right)$ 15:    Update target networks: ${\theta }^{{}^{\prime }}\leftarrow \tau \theta +(1-\tau ){\theta }^{{}^{\prime }}$ 16:    Update target networks: ${\varphi }^{{}^{\prime }}\leftarrow \tau \varphi +(1-\tau ){\varphi }^{{}^{\prime }}$ 17: end for

4. Computational Experiment

In this section, we set the specific parameters of the selected model and state parameters of the controller designed above. Thereafter, we simulated the set target values, compared the performance in various experiments, and finally analyzed and discussed their performance.

4.1. Experimental Setting

We selected an AUV model to simulate the performance of the proposed algorithms. All model parameters are listed in Table 1 [37].

The experimental hardware is an 11th Gen Intel(R) Core(TM) i9-11900H @ 2.50 GHz and the operating system is Ubuntu18.04. The network was developed by Python with pytorch. In the training process, we use Adam optimizer, so there is no GPU or parallel computing technique needed in our experiments.

The linear system model follows Equation (17); the DRL neural network and training parameters are shown in

Table 2. Parameters of Training.

Parameters	Value
Replay buffer size M	$1\times {10}^6$
Batch size N	256
Discount factor $\gamma$	$0.99$
Soft update rate $\eta$	$0.001$
Noise variance	$0.1$
Exploration noise	$0.1$
Policy noise	$0.2$
Noise clip N	$0.2$
Policy update frequency	2
Episode max step	1000
Max train episode	1000
Evaluate Interval	$1\times {10}^4$
Actor learning rate $l_{r\pi }$	$1\times {10}^{-4}$
Critic learning rate $l_{rQ}$	$1\times {10}^{-4}$
Decay rate of explore noise	0.998

.

4.2. Training Result

The strategy gradient optimization algorithm is described in Section 2.4. Because TD3 adds a dual critic and actor network to DDPG, it solves the problem of overestimation of Q values, delays the update of actors, stabilizes the training process, and adds noise to the output of the target network of actors. Thus, the proposed algorithm was more stable.

From the reward curve in Figure 6, the green curve is the MR-TD3 algorithm and the red curve is the MR-DDPG algorithm. In the curve of MR-TD3, the peak variance is reached first, and it is stable, whereas DDPG is overestimated and dramatically fluctuates during the run process. These behaviors are the same in the case of the MR-DDPG training process, which is consistent with the characteristics of DDPG, and it can also be seen that the TD3 is better trained. During this process, the training process was not stabilized. However, the difference between MR-DDPG and MR-TD3 is not very large, and both are relatively good DRL algorithms.

4.3. Simulation Result

In this section, we test the performance of the depth and pitch controllers and discuss all these simulation results. The experiments consisted of the following parts: Firstly, the TD3 and MR-TD3 controllers were used to control the depth and pitch angle, respectively, as well as verify the effect of the reference model on the control volume overshoot and actuator saturation. Secondly, the mass and the initial position of the AUV was adjusted to confirm that the designed controller is robust.

4.3.1. Depth Response and Overshoot Experimental Result

In the field of AUV control, the response time and overshoot are the two primary performance indicators of the controller. To test the performance of the DRL controller, we designed comparative experiments with TD3 and proposed the MR-TD3 method to discuss the step response and sine wave response. For the pitch channel, the target is the attitude angle and attitude angular velocity; the command of elevator rudder control should be concerned as the process reference value. For the depth channel, the target was the depth, and the process reference values were the diving speed and corresponding control commands. The depth and pitch angle are a pair of coupled values; these two can affect each other, so all the process variables of pitch and depth must be observed during the control process. In these experiments, the surge velocity is a fixed value of 2 m/s.

We first test the depth step response and sine response, in which all the initial state is zero, the step target depth is $1.5$ m, the sinusoidal amplitude target is 1 m, and the deviation of the sine curve is positive one. Figure 7 illustrates the states of the depth step response and sine wave response of the AUV system and the control command of the two channels.

In Figure 7a,b, the classical TD3 agent always tends to choose the maximum control command to follow the target, the maximum depth reaches 1.6 m (overshoot about $6.67\%$) in 5 s, and the elevator rudder comes into the full command 200 N and stays in this zone about 3 s, just as for the MR-TD3 agent. Because the reference target is not changed dramatically to $1.5$ m, the MR-TD3 command can smoothly follow the reference model depth target and reach the reference depth in 30 s; this time is the same as shown in Figure 3. However, since the ultimate control object is depth, not the reference model depth, upon comparing the two algorithms, just as Figure 8 shows, we can find that at the start time, the deviation of the two algorithms is 1.5 m, but the TD3 agent can drive the error to zero quickly in 4 s and the MR-TD3 agent can drive the error to zero in 30 s. To make it easier to compare the two results, the response time and overshoot are listed in

Table 3. Comparison of response time and overshoot in depth channel.

	Step RT (s)	Step OS (%)	Sin RT (s)
TD3	6	$6.67$	6
MR-TD3	30	0	∞

RT, OS denote the response time, overshoot.

. As the first-order derivative of depth control, the TD3 agent’s peak velocity of the depth step response $\omega$ is almost 1 m/s; this is almost three times that of the MR-TD3 agent. In addition, the second-level cascade system, pitch angle $\theta$, angular velocity q and the corresponding control command, as seen in Figure 7b, have the same response time and depth control trend.

Figure 7c,d are the depth sine response result. The TD3 agent can generate full command at 200 N and keeps about 4 s; this can track the reference depth in about 6 s, and then, the command can be changed to follow the target value. However, as for the MR-TD3 agent, the command can track the reference model value tightly, but it is always 5 s slower than the reference depth value.

From Figure 7 and Figure 8 and Table 3, we can come to the conclusions that:

(1).

In a step test scenario, the TD3 agent has a better response time but the MR-TD3 agent has better overshoot capacity.

(2).

In a sine test scenario, the MR-TD3 agent always has a 5 s delayed reaction.

(3).

The response time of the MR-TD3 agent is the same as that of the reference model.

4.3.2. Pitch Response and Overshoot Experimental Result

Figure 9 illustrates the pitch step response and pitch sine wave response of the two methods, in which all the initial state is $1/3$$\pi$, the sinusoidal amplitude target is 0.75 $\pi$, and the deviation of the sine curve is positive.

We can see that TD3 algorithms always tend to choose the maximum control command to track the target as close as possible; the tracking speed is so fast that the pitch angular velocity can reach $0.6$ rad/s in 1 s, and the pitch can reach the peak in about 3 s and has a $4.41\%$ overshoot but becomes stable at nearly 10 s (Figure 9a). Although the pitch control command ${\delta }_b$ did not reach the full zone, the depth control command ${\delta }_s$ reached the peak command 200 N for about 5 s (Figure 9b). The MR-TD3 agent can reach the target pitch in 25 s, and all the transition states are very smooth. Figure 9c,d are the pitch sine response simulation, which is almost the same as the depth sine response, the TD3 agent response time is 4 s, and MR-TD3 always has a nearly 7 s delay response behind the reference target.

From Figure 9 and Figure 10 and

Table 4. Comparison of the response time and overshoot in pitch channel.

	Step RT (s)	Step OS (%)	Sin RT (s)
TD3	4	$4.41$	3
MR-TD3	25	0	∞

RT, OS denote the response time, overshoot.

, we can come to the conclusion that:

(1).

In the pitch step test scenario, the TD3 agent has better response time but MR-TD3 is smoother.

(2).

In the pitch sine test scenario, the MR-TD3 agent always has a 7 s delayed reaction.

(3).

The response time of MR-TD3 is the same as that of the reference model.

4.3.3. Analysis

From Figure 7 and Figure 9, during the training process, to track the target, the actor of DRL will keep accumulating rewards, and faster tracking will lead to more rewards, so the TD3 method easily enters the saturation domain. It can also be seen that the response speed of the TD3 controller is faster than that of the MR-TD3 controller. This result is acceptable in the simulation process. Still, in the actual application, this control command is strong enough to follow the target as closely as possible. The control command will reach the maximum value (command saturation). Any external noise will lead to oscillation and instability, and the long-time control command saturation will also lead to the actuator overheating and damage. In contrast, the MR-TD3 controller chooses the reference target as the guide baseline instead of tracking the target directly, and the reference baseline will make a balance between the response time and the maximum control command.

The above figures also verify the first contribution proposed in this study, where the DRL agent of the MR-TD3 controller relative to the direct control can constrain the control commands to avoid the overshoot of the actuator and thus enter the saturation space.

The depth and pitch step response time of MR-TD3 are 30 s and 25 s, respectively, just in Table 3 and Table 4; this is the same as that in Figure 3. This phenomenon shows that the response time of the reference model control is determined by the parameters of the reference model, and it also proved the importance of the model parameters selection; these ultimately determine MR-TD3’s performance.

From Figure 7, Figure 8, Figure 9 and Figure 10 and Table 3 and Table 4, in the TD3 method, when the controlled object is the depth, the pitch angle and depth are a pair of coupling quantities. The response time of the pitch angle (25 s) is shorter than the depth (30 s) due to the cascading system and maintaining the rolling balance of the system. In the MR-TD3 method, as these two have been decoupled in the reference model, according to the actual intermediate state quantity of the system, that is, the dive velocity and pitch angular velocity, the pitch response time of the reference model is less than that of the depth. The above can be summarized as:

(1).

The TD3 agent has better response time but has overshoot.

(2).

The MR-TD3 state is much more smoother; all the commands will not reach the saturation zone, and there is a fixed delay reaction in the sine test.

(3).

The MR-TD3 response time is the same as that of the reference model.

(4).

As cascading AUV systems, pitch is the first level and depth is the second level; pitch always has a faster response than depth.

4.3.4. Robust Test of MR-TD3 Controller

This section tests the robustness of the trained agent. The purpose of the test is to change some initial state of the controlled object and check whether the controller can still keep the object stable.

(1)

Case I: Model Generalization.

From Equation (15), it is most convenient to change the mass m of the system to change the AUV model. Therefore, the weight of the AUV system is altered here to perform the robust test. The task is the robustness step response test of the MR-TD3 controller for the depth with initial position 0 m to reference position $1.5$ m. We changed the mass of the AUV system by decreasing it by $5\%$ ($47.5$ (kg)) and $20\%$$(40$ (kg)) and then increasing it by $5\%$$(52.5$ (kg)), $10\%$$(55$ (kg)) and $20\%$$(60$ (kg)), while all the other parameters remain the same. From Figure 11, we can observe that the controller remained stable when the weight changed within $10\%$.

However, for the AUV system with a $20\%$ weight decrease, the peak overshoot (about $23\%$) of the depth is $1.85$ m at the time of the 7 s, and the phenomenon of oscillation appears again until about the 15 s. Correspondingly, when the mass was $20\%$ increased, the peak overshoot (about $6.7\%$) appears at the time of the 6 s. We consider this experimental result reasonable because the disturbance or robustness must be a small proportion compared to the control value. If the percentage is too large compared with the control amount and dominates, the controller must be redesigned.

(2)

Case $II$: Random Initial State

Furthermore, we test whether the MR-TD3 controller can stabilize the AUV system with different initial positions. The control object is used to drive the AUV from a different initial depth back to 2 m; the other states and parameters are kept unchanged, each episode lasts for 10 s, and the MR-TD3 agent convged all the states to 2 m in 6 s. Figure 12 verifies that the proposed controller has the stability to drive the AUV to the target depth smoothly.

(3)

Conclusion

From the above two cases, we can come to the conclusion that:

(1).

MR-TD3 has the capacity of robustness, but the controlled object can only be changed by 10%.

(2).

MR-TD3 can converge from different initial positions, and the converge time is the same.

5. Conclusions

This study proposed a new control algorithm, the MR-TD3 algorithm, for controlling AUV systems. We analyzed the system model and set up the state space equation of the AUV system. Thereafter, we designed the reference model and system state variables to follow the target. Finally, we conducted pitch and depth control experiments using MR-TD3 and TD3 controllers to test the response time and overshoot performance.

The results show that the MR-TD3 algorithm can reduce the overshoot of control commands and eliminate the saturation of control commands. This can maintain the stability and anti-disturbance ability of the system. Moreover, this algorithm combines reinforcement learning with a reference model to provide a robust and applicable control method.

However, we also found that the reference model had a significant impact on the saturation and robustness of the instructions, indicating that the parameters of the reference model response time and the control magnitude play a crucial role. Additionally, these parameters may lead to a lack of robustness, thus leading to the control instruction of system saturation being made again. On the other hand, the designed MR-TD3 algorithm improves the saturation and robustness of the instructions but at the expense of the response speed, which is an issue to be considered in future research. In future work, we hope to solve the problem of response time and test the method in an actual platform. We also hoped that the method can be used in other control scenarios, such as autonomous guided vehicles and UAV.