Fuzzy Reinforcement Learning Disturbance Cancellation Optimized Course Tracking Control for USV Autopilot Under Actuator Constraint

Abstract

Unmanned surface vehicles (USVs) course control research constitutes a vital branch of ship motion control studies and serves as a key technology for the development of marine critical equipment. Aiming at the problems of model uncertainties, external marine disturbances, performance optimization, and actuator constraints encountered by the autopilot system, this paper proposes a composite disturbance cancellation optimized control method based on fuzzy reinforcement learning. Firstly, a coupling design of the finite-time disturbance observer and fuzzy logic system is conducted to estimate and reject the composite disturbance composed of internal model uncertainty and ocean disturbances. Secondly, a modified backstepping control technique is employed to design the autopilot controller and construct the error system. Based on the designed performance index function, the fuzzy reinforcement learning is utilized to propose an optimized compensation term for the error system. Meanwhile, to address the actuator saturation issue, an auxiliary system is introduced to modify the error surface, reducing the impact of saturation on the system. Finally, the stability of the autopilot system is proved using the Lyapunov stability theory. Simulation studies conducted on the ocean-going training ship “Yulong” demonstrate the effectiveness of the proposed algorithm. Under the strong and weak ocean conditions designed, this algorithm can ensure that the tracking error converges within 7 s.

1. Introduction

With the deepening promotion of intelligent marine vessel technologies by global maritime powers, the Unmanned Surface Vehicle (USV) has emerged as a pivotal platform for marine environmental monitoring, patrol search-and-rescue, and resource exploration, owing to its autonomy, mobility, and cost effectiveness [1,2,3]. As a fundamental guarantee for USV mission execution, course control has become a critical branch in the investigation of ship motion control. However, course control necessitates high-precision tracking under complex marine conditions and constrained mechanical equipment [4,5,6]. The propulsion system of USVs is inherently challenged by strong nonlinearity and environmental disturbances. Meanwhile, how to implement low-cost control for USV has become an urgent issue to maintain their long-term operational status. In addition, the autonomous driving system is also subject to physical constraints; exceeding the rated limits will inevitably lead to actuator failure, causing the vehicle to deviate from the predetermined route and even endangering navigation safety. Under such circumstances, constructing a coordinated control scheme that integrates disturbance elimination, performance optimization, and rudder angle constraints has become a key challenge in improving the safety of USVs [7,8,9,10].

The USV autopilot system typically relies on feedback from gyrocompasses for heading measurement. Heading rate measurements can be obtained from rate sensors, gyroscopes, the numerical differentiation of heading measurements, or state estimators. The first control scheme implemented on ships was a PID-based autopilot [11], followed by successive proposals of control techniques such as backstepping [12]. However, early control algorithms showed limited effectiveness in resisting various disturbances. To enhance USV adaptability in complex sea conditions and achieve stable course control, methods have been developed including adaptive control [13,14,15,16,17,18], disturbance observer-based control [19,20], and H_infinity robust control [21,22]. For example, Zhu et al. [23] designed an adaptive backstepping controller using fuzzy approximators to handle unknown internal uncertainties, while [21] proposed a fault-tolerant control scheme for autopilots based on robust techniques to suppress disturbances and address actuator failures. In particular, both approaches focus on disturbance suppression rather than estimation and elimination. In contrast, references [19,20] utilize disturbance estimation and cancellation techniques, which are better suited for high-precision control requirements. Concurrently, research on composite disturbances (integrating internal uncertainties and external perturbations) has evolved [24], coupling neural networks or fuzzy logic systems (FLSs) with finite-time disturbance observers to co-design disturbance cancellation control schemes. However, these studies prioritize anti-disturbance control for USV safety while neglecting optimal performance indices, which represent critical engineering demands.

For nonlinear autopilot systems, traditional linear optimal control struggles to directly solve for optimal solutions, whereas solving the Hamilton–Jacobi–Bellman (HJB) partial differential equation remains computationally intractable [25]. Reinforcement learning (RL) has emerged as an effective approach to address this challenge [26,27,28], fostering the development of RL-based autopilot controller designs in recent years [29,30]. References [31,32] proposed nonlinear optimal control schemes for USV autopilots using function approximators to construct critic and actor networks. However, these studies, despite addressing nonlinear optimization, overlook anti-disturbance design, compromising adaptability in complex marine environments. Additionally, USV autopilot actuators are subject to physical constraints (for example, 35° angle limits), highlighting the importance of a coordinated control scheme that integrates composite disturbance cancellation, performance optimization, and actuator saturation handling.

Building on the above analyses, this paper presents a composite disturbance cancellation optimal saturation control scheme for USVs based on fuzzy reinforcement learning. The approach involves coupling finite-time disturbance observers with fuzzy logic systems, designing optimal controllers for the performance index functions via fuzzy RL, and introducing an auxiliary system to modify the error surface and mitigate saturation effects. The comparison between the proposed USV RL scheme and the existing method is shown in

Table 1. Comparison of different control methods.

Control Method	Optimization	Adaptation	Disturbance Cancellation	Finite-Time Disturbance Cancellation	Actuator Constraint
Ref. [5]			✓
Ref. [19]			✓		✓
Ref. [23]		✓
Ref. [33]		✓	✓
Ref. [29]	✓	✓			✓
Proposed control	✓	✓	✓	✓	✓

. The novel contributions of this work include the following.

• Composite Disturbance Cancellation with RL Optimization. This study presents a composite disturbance cancellation optimized autopilot control scheme by integrating an FLS with a finite-time disturbance estimator and incorporating this coupled design into an RL framework. The proposed strategy not only achieves precise disturbance rejection but also minimizes performance indices, thereby enabling potential energy consumption reduction for USVs.
• RL-Optimized Control with Actuator Constraint. In USV control systems, actuator constraints (e.g., rudder angle limits) pose critical challenges for maintaining both control precision and system safety. This study proposes the RL framework that integrates the tanh function to address actuator saturation issues, complemented by an auxiliary system for error surface modification.

The remainder of this paper is structured as follows. Section 2 formulates the problem statement, while Section 3 details the design of the disturbance cancellation optimized controller with actuator constraints. Section 4 demonstrates the simulation results. Finally, conclusions are drawn in Section 5.

2. Problem Formation and Preliminaries

This section details the description of the USV autopilot model along with FLS.

2.1. Problem Formulation

The planar motion diagram of the USV shown in Figure 1, where u and v represent the forward speed and sway speed, respectively. $\Psi$ and $\phi$ represent the heading angle and rudder angle. $O_B{X}_B{Y}_B$ is the attached coordinate system, with $O_B$ being the center of gravity of the USV; $O_E{X}_E{Y}_E$ is the geodetic coordinate system, with $O_E$ as the origin and $O_E{X}_E$ as the north direction.

The model of the USV autopilot control system is established as [34]

$$\left\{\begin{array}{c}\dot{\phi }=r\hfill \\ \dot{r}=H\left(r\right)+gc\left(\tau \right)+d\hfill \end{array}\right.$$

(1)

Among them, $\phi$ represents the heading angle of USV; r represents the heading angular velocity; and $g=K/T$ represents the gain of the control system. The parameters K and T are the rotation index and the following index, both of which are maneuverability indicators of the USV. $c\left(\tau \right)$ indicates a restricted actuator. $H\left(r\right)$ is a nonlinear function, which can be approximated as

$$H\left(r\right)=-a_{1}r-a_2{r}^3-a_3{r}^5-\cdots$$

(2)

$a_i$ is the nonlinearity coefficient of the ship and is a constant. Due to limitation of physical conditions, the rudder is constrained by a constraint that can be constructed as

$$c\left(\tau \right)=\left\{\begin{array}{c}\mathrm{sign}\left(\tau \right){\tau }_M,\mathrm{if}\left|\tau \right|\ge {\tau }_M\hfill \\ \tau ,\mathrm{if}\left|\tau \right|<{\tau }_M\hfill \end{array}\right.$$

Clearly, the relationship between the applied control $c\left(\tau \right)$ and the control input $\tau$ has a sharp corner when $\left|\tau \right|={\tau }_M$. Then, the non-smooth saturated control signals enter the control system. To avoid damage to the stability of the system and increase the depletion of the rudder, a smooth function can be used to approximate the constraint in (1) as

$$u\left(\tau \right)={\tau }_M\times tanh\left({\displaystyle \frac{\tau }{{\tau }_M}}\right)$$

(3)

where ${\tau }_M$ is the upper bound of $\tau$. $e\left(\tau \right)=c\left(\tau \right)-u\left(\tau \right)$ is the approximation error, and $\left|e\left(\tau \right)\right|=\left|c\left(\tau \right)-u\left(\tau \right)\right|\le {\tau }_M(1-tanh\left(1\right))=e_M$ means that the upper bound of $e\left(\tau \right)$ is $e_M$.

The following assumptions are provided as follows.

Assumption 1 

([23]). Desired signal $y_r$ is smooth, bounded, and has a 2-order continuous bounded derivative.

Assumption 2 

([33]). The unknown environmental disturbance d is characterized as time-varying processes. Additionally, its first-order derivative is bounded, with the maximum value denoted as $d_M$.

The core control objective of this study is to develop a fuzzy reinforcement learning-based composite disturbance cancellation optimal control strategy for Unmanned Surface Vehicle (USV) autopilots subject to actuator constraints. This strategy is designed to enable the USV heading to not only track the desired signal accurately but also minimize the performance index function. Simultaneously, it ensures that all signals within the closed-loop system remain bounded.

2.2. Fuzzy Logic Systems (FLSs)

FLSs are used to approximate the uncertainties. Based on [33,35,36], the FLS is composed of four fundamental components: a fuzzy rule base that encodes domain knowledge, a singleton fuzzifier responsible for converting crisp inputs into fuzzy sets, a fuzzy reasoning engine that applies inference mechanisms to generate fuzzy outputs, and a defuzzifier that transforms these fuzzy results back into crisp values for practical use. The fuzzy rule base is shown via IF-THEN rules as

$R^L$: if ${\mathcal{X}}_1$ is ${{\mathcal{F}}_1}^L$, …, ${\mathcal{X}}_n$ is ${{\mathcal{F}}_n}^L$, then $\mathcal{Y}$ is ${\mathcal{G}}^L$, where $\mathcal{X}={\left[{\mathcal{X}}_1,\dots ,{\mathcal{X}}_n\right]}^{\top }$ is the input and $\mathcal{Y}$ is the output. ${{\mathcal{F}}_n}^L$ and ${\mathcal{G}}^L$ denote the fuzzy sets. The FLS can be presented as

$$\mathcal{Y}={\displaystyle \frac{{\sum }_{L=1}^{N}L{\prod }_{i=1}^n{s}_{{\mathcal{F}}_i^L}\left({\mathcal{X}}_i\right)}{{\sum }_{L=1}^N{\prod }_{i=1}^n{s}_{{\mathcal{F}}_i^L}\left({\mathcal{X}}_i\right)}},i=1,\dots ,n$$

(4)

where $w_L={max}_{\mathcal{Y}\in \mathcal{R}}{s}_{{\mathcal{F}}^L}\left(\mathcal{Y}\right)$. The fuzzy basis function is

$${\mathcal{S}}_L\left(\mathcal{X}\right)={\displaystyle \frac{{\prod }_{i=1}^n{\mathcal{S}}_{{\mathcal{F}}_i^L}\left({\mathcal{X}}_i\right)}{{\sum }_{L=1}^N\left[{\prod }_{i=1}^n{\mathcal{S}}_{{\mathcal{F}}_i^L}\left({\mathcal{X}}_i\right)\right]}}$$

(5)

Letting $w={[w_1,\dots ,w_N]}^{\top }$ and $s\left(\mathcal{X}\right)={[s_1\left(\mathcal{X}\right),\dots ,s_N\left(\mathcal{X}\right)]}^{\top }$, the FLS is shown as $\mathcal{Y}=w^{\top }s\left(\mathcal{X}\right)$.

Lemma 1 

((Universal Approximation Property of Fuzzy Logic Systems) [37]). Given a continuous function $f(·)$ defined over a compact set $\mathcal{R}$, and for an arbitrarily small positive real number $ϵ>0$, a fuzzy logic system (FLS) exists such that the following inequality holds:

$$\underset{·\in \Re }{sup}\left|f(·)-w^{\top }s(·)\right|\le ϵ$$

(6)

3. Rl-Composite Disturbance Cancellation Optimized Tracking Control Design

The FLS is deployed to approximate system uncertainties, effectively handling unknown dynamics. Complementing this, a finite-time disturbance observer is constructed to accurately estimate external disturbances in a timely manner. For actuator constraint, an auxiliary system is integrated into the control framework, ensuring that control inputs remain within the physical limits. The optimal controller $\tau$ is proposed via the RL optimized term ${\tau }^{\Xi }$ and backstepping-based controller ${\tau }^{\Theta }$ as the sum of $\tau ={\tau }^{\Theta }+{\tau }^{\Xi }$. The stability of the entire autopilot control system is rigorously proven using Lyapunov theory.

3.1. Construction of Finite-Time Disturbance Observer (FTDO)

A FTDO is constructed for the estimation. Define the $p_1=r$ and $p_2=d$. The system (1) can be extended, rewritten as

$$\left\{\begin{array}{c}{\dot{p}}_1=H\left(r\right)+gc\left(\tau \right)+p_2\hfill \\ {\dot{p}}_2=d\hfill \end{array}\right.$$

(7)

The FTDO can be established as [19]

$$\left\{\begin{array}{c}{\dot{\widehat{p}}}_1=\widehat{H}\left(r\right)+gu\left(\tau \right)+{\widehat{p}}_2-{\mu }_1{sig}^{1/2}({\widehat{p}}_1-p_1)\hfill \\ {\dot{\widehat{p}}}_2=-{\mu }_{2}sgn({\widehat{p}}_1-p_1)\hfill \end{array}\right.$$

(8)

where ${\mu }_1$ and ${\mu }_2$ are positive constants. ${\mathrm{sig}}^{1/2}({\widehat{p}}_1-p_1)=\mathrm{sgn}({\widehat{p}}_1-p_1){|{\widehat{p}}_1-p_1|}^{1/2}$

The observation error system along with (8) and (7) can be shown as

$$\left\{\begin{array}{c}{\dot{\tilde{p}}}_1={\tilde{p}}_2+{\mu }_1{sig}^{1/2}({\widehat{p}}_1-p_1)+\tilde{H}\left(r\right)+ge\left(\tau \right)\hfill \\ {\dot{\tilde{p}}}_2=d+{\mu }_{2}sgn({\widehat{p}}_1-p_1)\hfill \end{array}\right.$$

(9)

where the $\widehat{H}\left(r\right)$ and $\tilde{H}\left(r\right)$ denote the estimation and the estimation error of $H\left(r\right)$ via FLS.

The proposed FTDO finite-time stability can be proved based on the [38] and will not be elaborated further here.

3.2. Composite Disturbance Cancellation Control Design

A composite disturbance cancellation controller is designed by the adaptive backstepping with an auxiliary system. The following error surfaces are defined firstly

$$z_1=\phi -y_r$$

(10)

$$z_2=r-\alpha -\lambda$$

(11)

where $\alpha$ is the optimized virtual term, which can be defined as $\alpha ={\alpha }^{\Theta }+{\alpha }^{\Xi }$, ${\alpha }^{\Theta }$ is the virtual control, and ${\alpha }^{\Xi }$ is the optimal virtual term. $\lambda$ is the auxiliary system’s state, which can be designed as

$$\lambda =-\lambda +g\left(u\right(\tau )-\tau )$$

(12)

Step 1: Taking the derivative of $z_1$ with respect to (10) and (11) results in

$$\begin{array}{cc}\hfill {\dot{z}}_1& =\dot{\phi }-{\dot{y}}_r\hfill \\ & =r-{\dot{y}}_r\hfill \\ & =z_2+{\alpha }^{\Theta }+{\alpha }^{\Xi }+\lambda -{\dot{y}}_r\hfill \end{array}$$

(13)

Choose the Lyapunov function candidate as

$$V_1={\displaystyle \frac{1}{2}}{z}_1^2$$

(14)

The time derivative of $V_1$ is

$$\begin{array}{cc}\hfill {\dot{V}}_1& =z_1{\dot{z}}_1\hfill \\ & =z_1\left(z_2+{\alpha }^{\Theta }+{\alpha }^{\Xi }+\lambda -{\dot{y}}_r\right)\hfill \end{array}$$

(15)

Design the virtual control as

$${\alpha }^{\theta }=-k_1{z}_1-\lambda +{\dot{y}}_r$$

(16)

where $k_1$ is a design positive constant.

Taking (16) into (15), it yields

$${\dot{V}}_1=-k_1{z}_1^2+z_1{\alpha }^{\Xi }+z_1{z}_2$$

(17)

Step 2: Consider the second error surface (11) with the time derivative shown as

$$\begin{array}{cc}\hfill {\dot{z}}_2& =\dot{r}-\dot{\alpha }-\dot{\lambda }\hfill \\ & =H\left(r\right)+gc\left(\tau \right)+d+\lambda -gu\left(\tau \right)+g\tau -\dot{\alpha }\hfill \\ & =g\tau +ge\left(\tau \right)+f\left(\theta \right)+d+\lambda -\dot{\alpha }+{\omega }^{\top }\delta \left(\alpha \right)+{\eta }_{\alpha }\hfill \end{array}$$

(18)

where $\theta =[r,\alpha ],f\left(\theta \right)=H\left(r\right)-H\left(\alpha \right)$. the uncertainty $H\left(r\right)$ and $H\left(\alpha \right)$ can be handled by FLS as $H(·)={\omega }^{\top }\delta (·)+{\eta }_{·}$. ${\eta }_{·}$ are the approximation errors. The actual control is represented as $\tau ={\tau }^{\Theta }+{\tau }^{\Xi }$.

Considering the following Lyapunov function candidate as

$$V_2=V_1+{\displaystyle \frac{1}{2}}{z_2}^2+{\displaystyle \frac{1}{2}}{\tilde{\omega }}^{\top }\tilde{\omega }$$

(19)

The time derivative of $V_2$ with (18) is

$$\begin{array}{cc}\hfill {\dot{V}}_2& =-k_1{z}_1^2+z_1{\alpha }^{\Xi }+z_1{z}_2+z_2[g({\tau }^{\Theta }+{\tau }^{\Xi })+ge\left(\tau \right)+f\left(\theta \right)+d+\lambda -\dot{\alpha }\hfill \\ & +{\omega }^{\top }\delta \left(\alpha \right)+{\eta }_{\alpha }]+{\displaystyle \frac{1}{2}}{\tilde{\omega }}^{\top }\dot{\widehat{\omega }}\hfill \end{array}$$

(20)

The design of the actual controller with the adaptation law is presented as follows:

$${\tau }^{\Theta }=g^{-1}\left[-z_1-k_2{z}_2-{\widehat{\omega }}^{\top }\delta \left(\alpha \right)-\lambda -\widehat{d}+\dot{\alpha }\right]$$

(21)

$$\dot{\widehat{\omega }}=z_2\delta \left(\alpha \right)-k_3\widehat{\omega }$$

(22)

where $k_3$ and $k_2$ are the designed constants. Substituting (21) and (22) into (20) yields

$$\begin{array}{cc}\hfill {\dot{V}}_2& =-k_1{z}_1^2-k_2{z}_2^2+z_1{\alpha }^{\Xi }+z_{2}g{\tau }^{\Xi }+z_{2}ge\left(\tau \right)\hfill \\ & +z_2\tilde{d}+z_{2}f\left(\theta \right)+z_2{\eta }_{\alpha }+k_3{\tilde{\omega }}^{\top }\widehat{\omega }+z_2{\tilde{\omega }}^{\top }\delta \left(\alpha \right)\hfill \end{array}$$

(23)

and the following inequalities can be obtained:

$$z_{2}e\left(\tau \right)\le {\displaystyle \frac{1}{2}}{z}_2^2+{\displaystyle \frac{1}{2}}{e}_M^2$$

(24)

$$z_2\tilde{d}\le {\displaystyle \frac{1}{2}}{z}_2^2+{\displaystyle \frac{1}{2}}{\tilde{d}}_M^2$$

(25)

$$z_2{\eta }_{\alpha }\le {\displaystyle \frac{1}{2}}{z}_2^2+{\displaystyle \frac{1}{2}}{\eta }_{\alpha M}^2$$

(26)

$$z_2{\tilde{\omega }}^{\top }\delta \left(\alpha \right)\le {\displaystyle \frac{1}{2}}{z}_2^2+{\displaystyle \frac{1}{2}}{\tilde{\omega }}^{\top }\tilde{\omega }$$

(27)

$${\tilde{\omega }}^{\top }\widehat{\omega }\le -{\displaystyle \frac{1}{2}}{\tilde{\omega }}^{\top }\tilde{\omega }+{\displaystyle \frac{1}{2}}{\omega }^{\top }\omega$$

(28)

Substituting (24)–(28) into (23) yields

$$\begin{array}{cc}\hfill {\dot{V}}_2& \le -k_z{∥Z∥}^2+{\displaystyle \frac{1}{2}}{k}_3{\omega }_M^2+{\displaystyle \frac{1}{2}}{k}_3{\tilde{\omega }}_M^2+{\displaystyle \frac{1}{2}}{\eta }_{\alpha }^2+{\displaystyle \frac{1}{2}}{\tilde{d}}_M^2+{\displaystyle \frac{1}{2}}g{e}_M^2\hfill \\ & +Z^{\top }\left(\left[\begin{array}{c}0\\ f\left(\theta \right)\end{array}\right]+\left[\begin{array}{cc}1& 0\\ 0& g\end{array}\right]{\Phi }^{\Xi }\right)\hfill \end{array}$$

(29)

where $k_z=min\{k_1,k_2\}$ and $Z={[z_1,z_2]}^{\top }$. ${\Phi }^{\Xi }={[{\alpha }^{\Xi },{\tau }^{\Xi }]}^{\top }$ is the optimized controller which contains the optimal virtual and actual controls.

Based on [39], the boundedness of all variables within the entire closed-loop system can be ensured by formulating a controller that stabilizes the subsequent tracking error system. The optimized control law is designed to maintain system stability, thereby guaranteeing the bounded nature of all system variables:

$$\dot{Z}=\left[\begin{array}{c}0\\ f\left(\theta \right)\end{array}\right]+\left[\begin{array}{cc}1& 0\\ 0& g\end{array}\right]{\Phi }^{\Xi }$$

(30)

3.3. Fuzzy RL Optimized Compensator Design

For the tracking error system mentioned above, a fuzzy RL optimized compensator is devised. Subsequently, the error system (30) can be reformulated as

$$\dot{Z}=F\left(Z\right)+G{\Phi }^{\Xi }$$

(31)

where $F\left(Z\right)={\left[\begin{array}{c}0,f\left(\theta \right)\end{array}\right]}^{\top }$ and $G=\left[\begin{array}{cc}1& 0\\ 0& g\end{array}\right]$.

Define the performance index function as follows:

$$\rho ={\int }_T^{\infty }(Q\left(Z\right)+{\Phi }^{\Xi \top }R{\Phi }^{\Xi })dt$$

(32)

In this context, function $Q\left(Z\right)\in \mathcal{R}$ exhibits positive definiteness, while matrix $R\in {\mathcal{R}}^{2\times 2}$ is characterized by the same positive definite property.

For the (32) under the admissible $\Phi$, the corresponding Hamiltonian function is formulated as

$$\vartheta (Z,\Phi )=Q\left(Z\right)+{\Phi }^{\top }R\Phi +\nabla {\rho }_Z^{\top }(F\left(Z\right)+G\Phi ).$$

(33)

Here, $\nabla {\rho }_Z$ denotes the gradient of $\rho \left(Z\right)$ with respect to Z. Calculating the equation as $\partial \vartheta (Z,\Phi )/\partial \Phi =0$

$${\Phi }^{\Xi }=-{\displaystyle \frac{R^{-1}{G}^{\top }\nabla {\rho }_z^{\Xi }}{2}}$$

(34)

The HJB equation can be obtained with (34) and (33) as

$$0=Q\left(Z\right)+\nabla {\rho }_Z^{\Xi \top }f\left(\theta \right)-{\displaystyle \frac{1}{4}}\nabla {\rho }_z^{\Xi \top }G{R}^{-1}{G}^{\top }\nabla {\rho }_z^{\Xi }.$$

(35)

Considering the direct solution of the HJB equation is intractable, the FLS is employed to approximate the performance index function, which is expressed as

$$\rho \left(Z\right)={\varpi }^{\top }\zeta \left(Z\right)+\epsilon$$

(36)

where the weight matrix is denoted by $\varpi \in {\mathcal{R}}^{l\times 1}$, the activation function by $\varsigma \left(Z\right)\in {\mathcal{R}}^{l\times 1}$, and the approximation error by $\epsilon \in \mathcal{R}$.

The estimation forms of (34) and (36) are shown as

$$\widehat{\rho }\left(Z\right)={\widehat{\varpi }}^{\top }\varsigma \left(Z\right)+\epsilon$$

(37)

$${\widehat{\Phi }}^{\Xi }=-{\displaystyle \frac{R^{-1}{G}^{\top }{\widehat{\varpi }}^{\top }\varsigma \left(Z\right)}{2}}$$

(38)

In order to minimize ${\widehat{\Phi }}^{\Xi }$, the following weight update law can be designed based on the gradient descent method

$$\begin{array}{cc}\hfill \dot{\widehat{\mathit{\omega }}}& =-{\beta }_1\left[{\left(\nabla {\varsigma }_Z^{\top }\right)}^{\top }\widehat{f}\left(\theta \right)-{\displaystyle \frac{1}{2}}\times {\left(\nabla {\varsigma }_Z^{\top }\right)}^{\top }\Psi \nabla {\varsigma }_Z^{\top }\widehat{\varpi }\right]\hfill \\ & \times \left[Q\left(Z\right)+{\widehat{\varpi }}^{\top }{\left(\nabla {\varsigma }_Z^{\top }\right)}^{\top }\widehat{f}\left(\theta \right)-{\displaystyle \frac{1}{4}}\times {\widehat{\varpi }}^{\top }{\left(\nabla {\varsigma }_Z^{\top }\right)}^{\top }\Psi \nabla {\varsigma }_Z^{\top }\widehat{\varpi }\right]\hfill \end{array}$$

(39)

where $0<{\beta }_1<1$ denotes the learning rate.

The weight error dynamics are given by

$$\begin{array}{cc}\hfill \dot{\tilde{\varpi }}=& -{\beta }_1[{(\nabla {\zeta }_Z^{\top })}^{\top }Z-{(\nabla {\zeta }_Z^{\top })}^{\top }f\left(\theta \right)\hfill \\ & +{(\nabla {\varsigma }_Z^{\top })}^{\top }\Psi \nabla {\varsigma }_Z^{\top }\tilde{\varpi }+{\displaystyle \frac{1}{2}}{(\nabla {\varsigma }_Z^{\top })}^{\top }\Psi \nabla {\epsilon }_Z]\hfill \\ & \times [{\tilde{\varpi }}^{\top }{(\nabla {\zeta }_z^{\top })}^{\top }\dot{Z}+{\widehat{\varpi }}^{\top }{(\nabla {\zeta }_z^{\top })}^{\top }\tilde{f}\left(\theta \right)\hfill \\ & +{\displaystyle \frac{1}{4}}{\tilde{\mathit{\varpi }}}^{\top }{(\nabla {\zeta }_Z^{\top })}^{\top }\Psi \nabla {\zeta }_Z^{\top }\tilde{\mathit{\varpi }}+{\displaystyle \frac{1}{2}}{\tilde{\mathit{\varpi }}}^{\top }{(\nabla {\zeta }_Z^{\top })}^{\top }\Psi \nabla {\epsilon }_Z+\kappa ].\hfill \end{array}$$

(40)

The whole control scheme has been designed, and the final fuzzy RL-optimized control law is designed as $\tau ={\tau }^{\Theta }+{\tau }^{\Xi }$. The overall control diagram of the proposed RL autopilot control strategy is shown in Figure 2.

Theorem 1.

For the autopilot control system (1), with Assumptions 1 and 2, the proposed FTDO (8), disturbance cancellation controller (21), optimized compensator (38), adaptation laws (22) and (39), respectively, the proposed optimal strategy can guarantee that all the signals in the whole closed-loop USV autopilot control system are bounded via appropriate parameters.

Proof. 

The Lyapunov function candidate is selected as

$$V_{all}=V_3+{\displaystyle \frac{1}{2}}{\tilde{\varpi }}^{\top }\tilde{\varpi }.$$

(41)

Taking the time derivative of $V_{a}ll$ considering Equations (29) and (41) yields

$$\begin{array}{cc}\hfill {\dot{V}}_{all}& \le -k_z{∥Z∥}^2+{\displaystyle \frac{1}{2}}{k}_3{\omega }_M^2+{\displaystyle \frac{1}{2}}{k}_3{\tilde{\omega }}_M^2+{\displaystyle \frac{1}{2}}{{\eta }_{\alpha }}^2+{\displaystyle \frac{1}{2}}{{\tilde{d}}_M}^2+{\displaystyle \frac{1}{2}}g{e}_M^2\hfill \\ & +Z^{\top }\left(f\left(\theta \right)+G{\Phi }^{\Xi }\right)-{\beta }_1[\tilde{\varpi }{(\nabla {\varsigma }_Z^{\top })}^{\top }\dot{Z}-\tilde{\varpi }{(\nabla {\varsigma }_Z^{\top })}^{\top }\tilde{f}\left(\theta \right)\hfill \\ & +\tilde{\varpi }{(\nabla {\varsigma }_Z^{\top })}^{\top }\Psi \nabla {\varsigma }_Z^{\top }\tilde{\varpi }+{\displaystyle \frac{1}{2}}\tilde{\varpi }{(\nabla {\varsigma }_Z^{\top })}^{\top }\Psi \nabla {\epsilon }_Z]\hfill \\ & \times [{\tilde{\varpi }}^{\top }{(\nabla {\varsigma }_Z^{\top })}^{\top }\dot{Z}+{\widehat{\varpi }}^{\top }{(\nabla {\varsigma }_Z^{\top })}^{\top }\tilde{f}\left(\theta \right)\hfill \\ & +{\displaystyle \frac{1}{4}}{\tilde{\mathit{\varpi }}}^{\top }{(\nabla {\varsigma }_Z^{\top })}^{\top }\Psi \nabla {\varsigma }_Z^{\top }\tilde{\mathit{\varpi }}+{\displaystyle \frac{1}{2}}{\tilde{\mathit{\varpi }}}^{\top }{(\nabla {\varsigma }_Z^{\top })}^{\top }\Psi \nabla {\epsilon }_Z+\kappa ].\hfill \end{array}$$

(42)

According to [39], some assumptions are shown as $∥\nabla {\varsigma }_Z^{\top }∥\le {\varsigma }_M$, $∥\nabla {\epsilon }_Z∥\le {\epsilon }_M$, $f\left(\theta \right)+G{\Phi }^{\Xi }\le b_2\sqrt{∥Z∥}$, $∥\varpi ∥\le {\varpi }_M$ and $b_3\le ∥{\left(\nabla {\varsigma }_Z^{\top }\right)}^{\top }\Psi \nabla {\varsigma }_Z^{\top }∥\le b_4$. Here, ${\xi }_M,{\epsilon }_M,b_2,b_3,b_4,{\varpi }_M$ are all positive constants. Then, (42) can be rewritten as

$$\begin{array}{cc}\hfill {\dot{V}}_{all}& \le -k_z{∥Z∥}^2+{\displaystyle \frac{1}{2}}{k}_3{\omega }_M^2+{\displaystyle \frac{1}{2}}{k}_3{\tilde{\omega }}_M^2+{\displaystyle \frac{1}{2}}{\eta }_{\alpha }^2+{\displaystyle \frac{1}{2}}{\tilde{d}}_M^2+{\displaystyle \frac{1}{2}}g{e}_M^2\hfill \\ & +∥Z∥b_2\sqrt[ ]{∥Z∥}+{\beta }_1\left[{∥\tilde{\varpi }∥}^2\left({\zeta }_M^2-b_3+{\displaystyle \frac{b_3^2}{4}}{\zeta }_M^{-2}\right)+{\displaystyle \frac{b_2^2}{2}}∥Z∥+{\displaystyle \frac{1}{2}}{∥\tilde{\omega }∥}^2+{\displaystyle \frac{1}{4}}{\epsilon }_M^2\right]\hfill \\ & \times \left[{∥\tilde{\varpi }∥}^2{\displaystyle (}{\xi }_M^2+b_4+{\displaystyle \frac{b_3^2}{4}}{\xi }_M^{-2}{\displaystyle )}+b_2^2∥Z∥+{\displaystyle \frac{1}{2}}{∥\tilde{\omega }∥}^2+{\xi }_M^2+{\displaystyle \frac{b_3^2}{4}}{\xi }_M^{-2}+{\displaystyle \frac{5}{16}}{\epsilon }_M^2\right]\hfill \end{array}$$

(43)

By Young’s inequality, one obtains

$${\dot{V}}_{all}\le -{\gamma }_1{∥Z∥}^2-{\gamma }_2{∥\tilde{\omega }∥}^2-{\gamma }_3{∥\tilde{\varpi }∥}^2+{\gamma }_4$$

(44)

where ${\gamma }_1=k_z-{\displaystyle \frac{11b_2^2}{16}}-{\displaystyle \frac{b_5^2}{2}}-{\displaystyle \frac{3b_2^2}{2}}{\zeta }_M^2-{\displaystyle \frac{b_3^2}{4}}{\zeta }_M^{-2}$, ${\gamma }_2={\displaystyle \frac{b_1}{8}}-{\displaystyle \frac{3b_4^2}{4}}-{\displaystyle \frac{5}{2}}$, ${\gamma }_3=b_4{\epsilon }_M^2+b_3{\zeta }_M^2-{\displaystyle \frac{b_3^2}{4}}-{\displaystyle \frac{b_3^4}{32}}{\zeta }_M^4$, ${\gamma }_4={\displaystyle \frac{1}{2}}{\zeta }_M^2{\epsilon }_M^2+{\displaystyle \frac{7}{16}}{\epsilon }_M^2+{\displaystyle \frac{b_3^2}{32}}{\zeta }_M^{-2}+{\displaystyle \frac{1}{2}}{\eta }_{\alpha }^2+{\displaystyle \frac{1}{2}}{\tilde{d}}_M^2+{\displaystyle \frac{1}{2}}g{e}_M^2$, $b_5=b_2{b}_3-{\displaystyle \frac{3b_2}{32}}{\zeta }_M^2-{\displaystyle \frac{b_2{b}_3^2}{4}}{\zeta }_M^{-2}$. By selecting appropriate parameters, it implies that ${\dot{V}}_{all}<0$ if $\parallel Z\parallel >\sqrt{{\displaystyle \frac{{\gamma }_4}{{\gamma }_1}}}\mathrm{or}\parallel \tilde{w}\parallel >\sqrt{{\displaystyle \frac{{\gamma }_4}{{\gamma }_2}}}\mathrm{or}\parallel \tilde{\omega }\parallel >\sqrt{{\displaystyle \frac{{\gamma }_4}{{\gamma }_3}}}$.

This concludes the proof. □

4. Simulation Results

This section presents simulation results to verify the effectiveness of the proposed disturbance cancellation fuzzy RL-optimized control approach, using the ocean-going training vessel Yulong. The numerical simulations are made with the help of an experimental computer (Intel Core Ultra 7-U7-265CPU@5.30 GHz, RAM: 16.00 GB) and the simulation tool is MATLAB R2024a platform. The relevant parameters of the vessel are provided in [40].

To verify the effectiveness of the algorithm proposed in this paper, three cases are used for specific demonstration. Among them, Case 1 and Case 2 demonstrate the performance under weak and strong disturbances respectively. Case 1 (weak ocean disturbance): $d=(1/(\mathrm{K}0/\mathrm{T}0\left)\right)\times 0.01\times sin\left(0.05t\right)+0.007\times cos\left(0.03t\right).$ Case 2 (strong ocean disturbance): $d=5\times ((1/(\mathrm{K}0/\mathrm{T}0\left)\right)\times 0.01\times sin(0.05t)+0.007\times cos(0.03t\left)\right).$ Case 3 is the verification of the USV extended to 3 DOF. For Cases 1 and 2, three algorithms are employed for comparison, which are PID, RL with traditional disturbance observer [30], and adaptive neural network control considering input saturation [34]. It is particularly pointed out that, in order to ensure the validity of the algorithm comparison, the parameters and initial values in these control experiments are designed to be basically consistent. The experimental results are shown in Figure 3, Figure 4 and Figure 5. And the Root Mean Square Error (RMSE) comparison of different controllers in different cases is shown in

Table 2. RMSE comparison of different controllers in different cases.

Case	Case 1				Case 2
Controller	Proposed Controller	PID	[34]	[30]	Proposed Controller	PID	[34]	[30]
RMSE	0.004	0.226	0.122	0.009	0.007	0.464	0.293	0.143

.

When choosing the tracking signal, we select the mathematical model that can represent a certain actual performance requirement [34] as follows:

$${\ddot{y}}_m\left(t\right)+0.1{\dot{y}}_m\left(t\right)+0.0025y_m\left(t\right)=0.0025y_r\left(t\right)$$

(45)

Among them, $y_m$ the ideal system performance that characterizes the heading. $y_r$ is the command input signal, and its value ranges from 0 degrees to 30 degrees. The parameters of the mathematical model are obtained through calculation as $K=0.478$, $T=216$, $a_1=1$, $a_2=30$. In the simulation, the initial conditions are selected as follows: the initial state $\varphi \left(0\right)={15}^{\circ }$ (i.e., the heading deviation is +15°) and the initial state $r\left(0\right)=1^{\circ }/s$. The constraint limits of actuator is ${\tau }_M={35}^{\circ }$.

For the disturbance cancellation control scheme, the initial auxiliary states are selected as $\lambda \left(0\right)=0$, ${\widehat{p}}_1\left(0\right)=0.1$ and ${\widehat{p}}_2\left(0\right)=0.1$. The parameters are selected as $k_1=2$, $k_2=5$, ${\mu }_1=1$, ${\mu }_2=2.5$ and $k_3=0.1$. The initial adaptation law parameters are assigned random values within the interval (0,1). The fuzzy basis functions are constructed as

$$\delta \left(\alpha \right)={\displaystyle \frac{exp\left[-\frac{{\left(\alpha +3-l\right)}^2}{2}\right]}{{\sum }_{l=1}^{5}exp\left[-\frac{{\left(\alpha +3-l\right)}^2}{2}\right]}}\mathrm{and}\delta \left(r\right)={\displaystyle \frac{exp\left[-\frac{{\left(r+2-l\right)}^2}{2}\right]}{{\sum }_{l=1}^{5}exp\left[-\frac{{\left(r+2-l\right)}^2}{2}\right]}}.$$

For the optimal controller, in the performance index function, $Q\left(Z\right)=4Z^{\top }Z$ and $R=3$. The initial value of $\widehat{\varpi }\left(0\right)$ is assigned random values within the interval (0,1) under ${\beta }_1=0.02$. Additionally, the fuzzy basis function is selected as the following:

$$\varsigma \left(Z\right)={\displaystyle \frac{exp\left[-\frac{{\left(z_1+2-l\right)}^2}{2}\right]\times exp\left[-\frac{{\left(z_2+2-l\right)}^2}{2}\right]}{{\displaystyle \sum _{l=1}^5}exp\left[-\frac{{\left(z_1+2-l\right)}^2}{2}\right]\times exp\left[-\frac{{\left(z_2+2-l\right)}^2}{2}\right]}}.$$

4.1. Comparisons with Traditional PID Controller

The comparisons were conducted between the designed RL controller and the PID controller under nearly identical initial conditions and parameters.

The simulation results are presented in Figure 2 and Figure 3, respectively. Figure 2 shows that under weak marine disturbances, the tracking performance of PID and the proposed RL method exhibits little difference. However, the tracking error plot reveals that the PID algorithm yields relatively larger errors. Moreover, without constraints on the control inputs, the initial values of the PID control inputs reach approximately −200° to 500°, exceeding the physical limits of rudder angles. This is practically unacceptable in engineering applications. In Figure 3, under strong marine disturbances, a significant disparity in tracking performance is observed between PID and the proposed RL method. The PID-based control exhibits notable tracking errors of ±0.5°, whereas the RL controller maintains tighter error bounds. These results collectively demonstrate that the proposed optimized controller outperforms the PID controller in terms of control accuracy and constraint compliance.

4.2. Comparisons with Traditional NN Adaptive Control with Input Constraints [34]

These comparisons were conducted between the designed RL controller and NN controller under nearly identical initial states and parameters.

The simulation results are presented in Figure 2 and Figure 3, respectively. Figure 2 shows that under weak marine disturbances, it can be seen that the NN control has a larger overshoot compared with the proposed algorithm. The initial tracking error of NN control is relatively large, but due to the adaptive adjustment, the error gradually converges after 8 s. However, compared with the proposed algorithm with disturbance rejection and the method in reference [30], the disturbance suppression effect of NN control is relatively ordinary. This also leads to (c) where the NN control input reaches saturation multiple times, with a duration of about 8 s. In Figure 3, the problems encountered by NN control are further amplified: it can be observed that between 350 s and 500 s, the performance of NN control is obviously inferior to that of the disturbance rejection method. Under strong disturbances, the tracking error of NN control increases significantly to around 0.5°, and its control input remains in saturation for a long time starting at about 9 s, which imposes a heavy burden on the machinery. Therefore, the proposed optimized controller has certain advantages over the NN controller in terms of disturbance rejection and optimization.

4.3. Comparisons with RL Control with Traditional DO [30]

These comparisons were conducted between the designed RL finite-time disturbance rejection controller with input constraints and the reinforcement learning optimized controller with DO under nearly identical initial states and parameters.

The simulation results are presented in Figure 2 and Figure 3, respectively. Figure 2 shows that under weak marine disturbances, from (a) and (b), it can be seen that compared with the proposed algorithm, the RL control in reference [30] can track the desired signal within 3 s and enable the tracking error to converge faster than the proposed algorithm. However, (c) reveals that this is achieved by removing the constraints on the input. From (d), it can be observed that for the same disturbance, the convergence speed of the proposed FTDO is nearly 2 s faster than that of DO. In Figure 3, when there are no constraints on the control input, reference [30] exhibits better control performance, but this ignores the requirements of engineering practice. From (d), it can be seen that for the same strong disturbance, the estimation accuracy of the proposed FTDO is somewhat higher than that of DO. Respectively, (e) and (f) illustrate the convergence processes of the adaptive parameters of the proposed algorithm. Therefore, the proposed optimized controller has certain advantages over reference [30] in terms of disturbance rejection.

4.4. Extended Simulation on 3-DOF USV

To verify the universality of the proposed algorithm, it is extended to a fully actuated 3-DOF USV. For the specific parameters and turning trajectory of the USV, reference can be made to [41]. The detailed results are shown in Figure 5. It can be seen from (a) to (d) that the proposed algorithm is also applicable to the 3-DOF USV, with a considerable overshoot of tracking error and relatively accurate estimation of disturbances. Then, (e) and (f) represent the corresponding update rates of adaptive parameters. Therefore, the proposed optimized control algorithm can still be applied to the 3-DOF USV.

5. Conclusions

In this paper, a composite disturbance cancellation optimized control via fuzzy reinforcement learning has been proposed to handle the problems of model uncertainties, external marine disturbances, performance optimization and actuator constraints encountered by the USV autopilot system. The coupling design for the finite-time disturbance observer and fuzzy logic system has been conducted to estimate and reject the composite disturbance. The design of backstepping and RL techniques have been used to construct the optimized constraint controller with the designed performance function and the auxiliary system. The stability of the autopilot control system has been theoretically validated using Lyapunov stability theory. Simulation studies conducted on the ocean-going training ship “Yulong” have demonstrated the effectiveness of the proposed algorithm.

Currently, the work in this paper has been performed regarding the adaptive optimization control of rudders. Future research will consider the following directions. Simulation experiments have verified the effectiveness of this framework. Physical simulations can be conducted using real ships under sensor noise and other external factors, taking a step forward for real-world applications. In addition, as the computational burden of high-complexity algorithms gradually increases, exploring ways to reduce the computational load and complexity of algorithms is promising.