Output Feedback Adaptive Optimal Control of Multiple Unmanned Marine Vehicles with Unknown External Disturbance

Abstract

This paper presents an optimal output-feedback tracking control problem for multiple unmanned marine vehicles (UMVs) to track a desired trajectory. To guarantee the control objective in an optimal manner, adaptive dynamic programming (ADP) with optimal compensation terms is adopted. A neural velocity observer is designed based on a neural network (NN) to estimate the unmeasured system states and the unknown system dynamics. Furthermore, a disturbance observer (DO) is proposed to go against the effect of the unknown external disturbance of the sea environment. It is proved that the proposed controller can guarantee that all signals in the closed-loop system are bounded. Simulation results are given to demonstrate the effectiveness of the proposed control algorithm.

1. Introduction

In recent years, unmanned marine vehicles (UMVs) have garnered increasing attention for the exploration of natural resources in oceanic spaces due to their unique advantages, such as low energy costs and advanced intelligence. Furthermore, UMVs can complete dangerous tasks without putting human lives at risk [1,2]. However, a solitary UMV may not suffice to execute complex tasks in some situations. Thus, the control problem of multiple UMVs has garnered substantial interest. In recent years, the coordinated control of multiple UMVs has become a burgeoning research topic [3]. Some important theoretical results of coordinated control of multiple vehicles have been reported [4,5,6,7,8]. In [8], a time-varying formation control problem was presented. Connectivity preservation and collision avoidance were investigated with position-heading measurements. In [5], the path-tracking control of under-actuated unmanned surface vehicles was investigated. Model uncertainties and unknown disturbances over a wireless network were considered. This research has far-reaching implications; however, the control attempts can use high amounts of energy because the aforementioned control design strategies do not consider the optimization problem.

Optimal control is a fundamental design principle that can be adopted to enhance the control performance of dynamic systems. Given that ocean transportation or deep-sea exploration frequently necessitates substantial energy provision, it becomes imperative to incorporate optimization into vessel control design to reduce energy consumption. However, the optimal control of UMVs is a difficult problem due to the inherent nonlinearities of the UMV system. The optimization problems of nonlinear systems often need to solve the Hamilton–Jacobi–Bellman (HJB) equation, which does not have a closed-form solution. To handle this difficulty, a promising adaptive optimal method was presented based on the reinforcement learning (RL) method, namely, adaptive dynamic programming (ADP) [9,10,11,12], in which an RL system was designed to approximate the HJB equation [13,14,15]. Many studies on the control problems of vehicles by using the ADP method have been reported [16,17,18]. In [16], the optimal tracking control problem for the dynamic positioning of marine vessels was investigated. The observer based on a fuzzy logic system (FLS) was given to handle the problem of unmeasured states of the vessels. A disturbance observer (DO) was given to estimate the external disturbances. In [17], a data-driven RL-based controller was introduced to address the optimal control problem of the vehicle. Using a data-driven approach, a model-free control method was formulated to achieve control optimality and prescribed tracking accuracy concurrently. In [18], an optimal control scheme was presented for RL-based optimal tracking control applied to UMVs. The unknown dead-zone input nonlinearities and unknown disturbances were considered and handled by using a neural network (NN)-based identifier. This research shows how the optimal control problem of a single UMV can be solved. However, the optimal tracking control problem of multiple UMVs cannot be handled directly using these methods.

In practical applications, accurately obtaining velocity information from shipborne sensors is often challenging due to the potential impact of noise interference and sensor signal loss on its effectiveness [19]. To handle this difficulty, some research based on the observer has been reported [3,20,21,22]. In [20], a study on the distributed containment maneuvering problem was conducted. Adopting the input–output data from each vehicle, a novel approach utilizing an echo state network-based observer was introduced to address the issue of unmeasured states. In [3], the flocking control problem of vehicles was studied. The velocity information of the vehicles was obtained by employing an extended state observer. In [22], the path-following problem of vehicles was studied. Unmeasured states were handled by giving a state observer based on FLS. From the studies mentioned above, establishing an observer to handle the unmeasured velocity information of vehicles is an effective approach.

When working in a sea environment, the control effectiveness of a UMV can be influenced by external disturbances such as wind and waves, which may lead to a failure to achieve the control target [23,24]. Thus, it is important to adopt disturbance rejection techniques to reduce the effect of unknown external disturbances. In recent years, many disturbance observer (OB)-based controllers were reported [24,25,26]. In [24], the robust leader–follower synchronization navigation for the UMVs was presented. The problem of unknown external disturbance was solved by adopting OB. In [26], the trajectory tracking control problem of UMVs was solved using the dynamic surface control technique. The time-varying disturbances were considered and estimated by a proposed DO. These results are fruitful and inspiring.

Motivated by the observations described above, the formation tracking problem of multiple UMVs is studied in this paper. An ADP algorithm with an optimal compensation term is adopted to guarantee the control objective optimally. An adaptive controller is designed using the backstepping technique; the optimal tracking control problem is transformed into an equivalent optimal regulation problem. Subsequently, an optimal compensation term is designed by using the policy iteration method. The overall optimal control input is the adaptive controller plus the optimal compensation term. To handle the unknown time-varying external disturbances of the sea environment, a DO is designed. It is proved that the proposed controller can guarantee that all signals in the closed-loop system are bounded. Simulation results are given to demonstrate the effectiveness of the proposed control algorithm.

The main contribution of this work can be summarized as follows.

(1) Unlike the references [4,5,6,7,8] investigating the tracking control problem of multiple UMVs without consideration of optimality, this paper considers optimality for designing the consensus controller. This means that the control method proposed in this paper can achieve the tracking control target with less energy consumption.

(2) Compared with the works in [17,18], an advantage of this paper is that this paper investigated the optimal tracking control problem of multiple UMVs. In contrast, the authors in [17,18] only investigated the optimal tracking control problem of a single UMV. Therefore, the task we set out to achieve is more challenging than the existing studies.

The rest of the paper is organized as follows: Section 2 provides the problem formulation, Section 3.1 presents the DO, Section 3.2 and Section 3.3 introduce the adaptive controller and the optimal compensation term, Section 3.4 provides a stability analysis, Section 4 provides the simulation results, and Section 5 presents concluding remarks.

2. Problem Formulation

Two reference frames are adopted: a body-fixed reference frame (BF) and a north-east-down reference frame (NED), which can be found in Figure 1. These two reference frames are generally adopted in ship motion control, and readers can find more details in [27,28,29].

Considering the leader–follower formation control problem of m UMVs, the underactuated system dynamics of 3 degrees of freedom (3-DOF) with uncertain dynamics can be described as follows [28,29,30]:

$$\begin{array}{c}{\dot{\eta }}_i=R\left({\psi }_i\right){\nu }_i\hfill \\ M_i{\dot{\nu }}_i=-C_i\left({\nu }_i\right){\nu }_i-D_i\left({\nu }_i\right){\nu }_i+\Delta ({\eta }_i,{\nu }_i)+d_i+{\tau }_i\hfill \\ i=1,2,\dots ,m\hfill \end{array}$$

(1)

where ${\eta }_i={[x_i,y_i,{\psi }_i]}^{\top }$, $x_i,y_i$ is the position of UMV in the earth-fixed frame, ${\psi }_i$ is the heading angle in the earth-fixed frame, ${\nu }_i={[u_i,v_i,r_i]}^{\top }$ denotes the velocities of UMV in the body-fixed frame, and ${\tau }_i={[{\tau }_{i1},{\tau }_{i2},{\tau }_{i3}]}^{\top }$ denotes the control input. $\Delta ({\eta }_i,{\nu }_i)\in {\mathbb{R}}^{3\times 1}$ denotes the uncertain dynamics. $d_i$ is the unknown disturbance from the sea environment. $R\left({\psi }_i\right)\in {\mathbb{R}}^{3\times 3}$ is the rotation matrix from the earth-fixed frame to the body-fixed frame, which is given as follows:

$$R\left({\psi }_i\right)=\left[\begin{array}{ccc}cos\left({\psi }_i\right)& -sin\left({\psi }_i\right)& 0\\ sin\left({\psi }_i\right)& cos\left({\psi }_i\right)& 0\\ 0& 0& 1\end{array}\right]$$

(2)

$M\in {\mathbb{R}}^{3\times 3}$ is an inertia matrix including hydrodynamic added inertia; $D_i\left({\nu }_i\right)\in {\mathbb{R}}^{3\times 3}$ is the damping matrix; $C_i\left({\nu }_i\right)\in {\mathbb{R}}^{3\times 3}$ is a matrix of the centripetal and Coriolis terms. These three terms are shown as follows:

$$\begin{array}{c}{M}_i=\left[\begin{array}{ccc}{m}_{i11}& 0& 0\\ 0& m_{i22}& m_{i23}\\ 0& m_{i32}& m_{i33}\end{array}\right]\\ C_i\left({\nu }_i\right)=\left[\begin{array}{ccc}0& 0& c_{13i}({\nu }_i,r_i)\\ 0& 0& m_{i11}{u}_i\\ -c_{13i}({\nu }_i,r_i)& -m_{i11}{u}_i& 0\end{array}\right]\\ D_i\left({\nu }_i\right)=\left[\begin{array}{ccc}{d}_{i11}& 0& 0\\ 0& d_{i22}& d_{i23}\\ 0& d_{i32}& d_{i33}\end{array}\right]\end{array}$$

(3)

where $m_{i11}=m_i-X_{i\dot{u}}$, $m_{i22}=m_i-Y_{i\dot{v}}$, $m_{i32}=m_i{x}_{iG}-N_{i\dot{v}}$, $m_{i33}=I_{iz}-N_{i\dot{r}}$, and $c_{13i}({\nu }_i,r_i)=-m_{i22}{v}_i-m_{i23}{r}_i$. $m_i$ denotes the mass of the vessel, and $x_{iG}$ denotes the distance between the center of gravity of the vessel and the origin of the body-fixed frame. $I_{iz}$ is the moment of inertia. $d_{i11}=-X_{iu}-X_{i\left|u\right|u}\left|u_i\right|$, $d_{i22}=-Y_{iv}-Y_{i\left|v\right|v}|v_i|-Y_{i\left|r\right|v}\left|r_i\right|$, $d_{i23}=-Y_{ir}-Y_{i|v_i|r}|v_i|-Y_{i\left|r\right|r}\left|r_i\right|$, $d_{i32}=-N_{iv}-N_{i\left|v\right|v}|v_i|-N_{i\left|r\right|v}\left|r_i\right|$, and $d_{i33}=-N_{ir}-N_{i\left|v\right|r}|v_i|-N_{i\left|r\right|r}\left|r_i\right|$. $X_{•}$, $Y_{•}$, and $N_{•}$ are the corresponding hydrodynamic derivatives. Consider a virtual leader moving along a desired trajectory shown as ${\eta }_{0d}={\left[x_{0d},y_{0d},{\psi }_{0d}\right]}^{\top }$.

For simplicity, we can obtain the following equation by using System (1):

$$\begin{array}{c}{\dot{\eta }}_i={\upsilon }_i\hfill \\ {\dot{\upsilon }}_i=f_i\left({\eta }_i,{\upsilon }_i\right)+M_i^{-1}{R}_i\left({\psi }_i\right)({\tau }_i+d_i)\hfill \\ i=1,2,\dots ,m\hfill \end{array}$$

(4)

where

$${\upsilon }_i=R_i\left({\psi }_i\right){\nu }_i$$

(5)

and

$$\begin{array}{cc}\hfill f_i\left({\eta }_i,{\upsilon }_i\right)=& {M_i}^{-1}{R}_i\left({\psi }_i\right)(-C_i\left({\nu }_i\right){\nu }_i-D_i\left({\nu }_i\right){\nu }_i+\Delta \left({\eta }_i,{\nu }_i\right))+{\dot{R}}_i\left({\psi }_i\right){\nu }_i\hfill \end{array}$$

(6)

$f_i\left({\eta }_i,{\upsilon }_i\right)$ is an unknown function since $\Delta \left({\eta }_i,{\nu }_i\right)$ is unknown. To handle this problem, NN is adopted to obtain the approximation.

Control Problem Statement: This study aims to design an output feedback control algorithm that can handle the optimal tracking control problem of multiple UMVs with unknown external disturbance and uncertain dynamics. The controller can ensure that all the signals in the closed-loop are bounded.

3. Main Results

3.1. Neural Observer Design

A neural state observer (NSO) is adopted to handle the unmeasured velocities and unknown system dynamics. From (4), we can obtain the following equations:

$$\begin{array}{cc}\hfill {\dot{X}}_i& =A_i{X}_i+B_{i1}{f}_i({\eta }_i,{\upsilon }_i)+D_i{B}_{i1}({\tau }_i+d_i)\hfill \\ \hfill {\dot{Y}}_i& =A_i{Y}_i+B_{i2}{f}_i({\eta }_i,{\upsilon }_i)+D_i{B}_{i2}({\tau }_i+d_i)\hfill \\ \hfill {\dot{\Psi }}_i& =A_i{\Psi }_i+B_{i3}{f}_i({\eta }_i,{\upsilon }_i)+D_i{B}_{i3}({\tau }_i+d_i)\hfill \end{array}$$

(7)

where $X_i={[x_i,u_i]}^{\top }$, $Y_i={[y_i,v_i]}^{\top }$, ${\Psi }_i={[{\psi }_i,r_i]}^{\top }$, $A_i=\left[\begin{array}{cc}0\hfill & 1\\ 0\hfill & 0\end{array}\right]$, $B_{i1}=\left[\begin{array}{ccc}0\hfill & 0& 0\\ 1\hfill & 0& 0\end{array}\right]$, $B_{i2}=\left[\begin{array}{ccc}0\hfill & 0& 0\\ 0\hfill & 1& 0\end{array}\right]$, $B_{i3}=\left[\begin{array}{ccc}0\hfill & 0& 0\\ 0\hfill & 0& 1\end{array}\right]$, $D_i=R_i\left({\psi }_i\right)M^{-1}$. Since $f_i({\eta }_i,{\upsilon }_i)$ is unknown, NN is adopted to obtain the approximation, which can be described as

$$f_i({\eta }_i,{\upsilon }_i)={\theta }_i^{*}{\phi }_i({\widehat{\eta }}_i,{\widehat{\upsilon }}_i)+{\epsilon }_i$$

(8)

${\epsilon }_i$ denotes the minimum approximation error, i.e., $\parallel {\epsilon }_i\parallel \le {\epsilon }_{im}$, where ${\epsilon }_{im}\in {\mathbb{R}}^{3\times 1}$ is a constant vector, $\parallel {\phi }_i\parallel \le {\phi }_{im}$. We can obtain the approximation of $f_i({\eta }_i,{\upsilon }_i)$ as

$${\widehat{f}}_i({\widehat{\eta }}_i,{\widehat{\upsilon }}_i)={\widehat{\theta }}_i{\phi }_i({\widehat{\eta }}_i,{\widehat{\upsilon }}_i)$$

(9)

We can design the neural velocity observer as follows:

$$\begin{array}{cc}\hfill {\dot{\widehat{X}}}_i& =-A_{i1}{\widehat{X}}_i+C_i{X}_i+B_{i1}{\widehat{\theta }}_i{\phi }_i({\widehat{\eta }}_i,{\widehat{\upsilon }}_i)+D_i{B}_{i1}({\tau }_i+{\widehat{d}}_i)\hfill \\ \hfill {\dot{\widehat{Y}}}_i& =-A_{i2}{\widehat{Y}}_i+C_i{Y}_i+B_{i2}{\widehat{\theta }}_i{\phi }_i({\widehat{\eta }}_i,{\widehat{\upsilon }}_i)+D_i{B}_{i2}({\tau }_i+{\widehat{d}}_i)\hfill \\ \hfill {\dot{\widehat{\Psi }}}_i& =-A_{i3}{\widehat{\Psi }}_i+C_i{\Psi }_i+B_{i3}{\widehat{\theta }}_i{\phi }_i({\widehat{\eta }}_i,{\widehat{\upsilon }}_i)+D_i{B}_{i3}({\tau }_i+{\widehat{d}}_i)\hfill \end{array}$$

(10)

where ${\widehat{X}}_i={[{\widehat{x}}_i,{\widehat{u}}_i]}^{\top }$, ${\widehat{Y}}_i={[{\widehat{y}}_i,{\widehat{v}}_i]}^{\top }$, and ${\widehat{\Psi }}_i={[{\widehat{\psi }}_i,{\widehat{r}}_i]}^{\top }$. $A_{i1}=\left[\begin{array}{cc}{k}_{i11}\hfill & 1\\ k_{i12}\hfill & 0\end{array}\right]$, $A_{i2}=\left[\begin{array}{cc}{k}_{i21}\hfill & 1\\ k_{i22}\hfill & 0\end{array}\right]$, $A_{i3}=\left[\begin{array}{cc}{k}_{i31}\hfill & 1\\ k_{i32}\hfill & 0\end{array}\right]$. $C_i={[1,0]}^{\top }$$A_{i1}$, $A_{i2}$, and $A_{i3}$ are a Hurwitz matrix by choosing the suitable parameters $k_{i11}$, $k_{i12}$, $k_{i21}$, $k_{i22}$, $k_{i31}$, and $k_{i32}$. $P_{i1}$, $P_{i2}$, $P_{i3}$, $Q_{i1}$, $Q_{i2}$, and $Q_{i3}$ are positive definite matrices that satisfy $A_{i1}^{\top }{P}_{i1}+P_{i1}^{\top }{A}_{i1}=-Q_{i1}$, $A_{i2}^{\top }{P}_{i2}+P_{i2}^{\top }{A}_{i2}=-Q_{i2}$, and $A_{i3}^{\top }{P}_{i3}+P_{i3}^{\top }{A}_{i3}=-Q_{i3}$, where $Q_{i1}=Q_{i1}^{\top }>0$, $Q_{i2}=Q_{i2}^{\top }>0$, and $Q_{i3}=Q_{i3}^{\top }>0$. ${\widehat{d}}_i$ is the disturbance estimation that the DO will obtain. Let ${\tilde{d}}_i=d_i-{\widehat{d}}_i$ be the disturbance approximation error. Define the NSO error dynamics as ${\dot{\tilde{X}}}_i={\dot{X}}_i-{\dot{\widehat{X}}}_i$, ${\dot{\tilde{Y}}}_i={\dot{Y}}_i-{\dot{\widehat{Y}}}_i$, and ${\dot{\tilde{\Psi }}}_i={\dot{\Psi }}_i-{\dot{\widehat{\Psi }}}_i$. From (7) and (10), we can obtain the error dynamics of NSO as follows:

$$\begin{array}{cc}\hfill {\dot{\tilde{X}}}_i& =A_{i1}{\widehat{X}}_i+B_{i1}{\tilde{\theta }}_i{\phi }_i({\widehat{\eta }}_i,{\widehat{\upsilon }}_i)+D_i{B}_{i1}{\tilde{d}}_i+B_{i1}{\epsilon }_i\hfill \\ \hfill {\dot{\tilde{Y}}}_i& =A_{i2}{\widehat{Y}}_i+B_{i2}{\tilde{\theta }}_i{\phi }_i({\widehat{\eta }}_i,{\widehat{\upsilon }}_i)+D_i{B}_{i2}{\tilde{d}}_i+B_{i2}{\epsilon }_i\hfill \\ \hfill {\dot{\tilde{\Psi }}}_i& =A_{i3}{\widehat{\Psi }}_i+B_{i3}{\tilde{\theta }}_i{\phi }_i({\widehat{\eta }}_i,{\widehat{\upsilon }}_i)+D_i{B}_{i3}{\tilde{d}}_i+B_{i3}{\epsilon }_i\hfill \end{array}$$

(11)

3.2. Disturbance Observer Design

To implement the DO, we begin by defining the auxiliary vector $q_i$ for each vehicle as follows:

$$q_i=d_i-K_{id}{\upsilon }_i$$

(12)

where $q_i={[q_{i1},q_{i2},q_{i3}]}^{\top }$, and a positive definite design matrix $K_{id}\in {\mathbb{R}}^{3\times 3}$ is employed. We can obtain the time derivative of $q_i$ as

$$\begin{array}{cc}\hfill {\dot{q}}_i& ={\dot{d}}_i-K_{id}{\dot{\upsilon }}_i\hfill \\ \hfill & ={\dot{d}}_i-K_{id}(f_i({\eta }_i,{\upsilon }_i)+R_i\left({\psi }_i\right)M_i^{-1}({\tau }_i+q_i+K_{id}{\upsilon }_i))\hfill \end{array}$$

(13)

Since $d_i$ is unknown, $q_i$ is also unknown. We can obtain an approximation of $q_i$ using the following equation:

$${\dot{\widehat{q}}}_i=-K_{id}({\widehat{f}}_i({\widehat{\eta }}_i,{\widehat{\upsilon }}_i)+R_i\left({\psi }_i\right)M_i^{-1}({\tau }_i+{\widehat{q}}_i+K_{id}{\widehat{\upsilon }}_i))$$

(14)

Thus, we obtain ${\widehat{d}}_i$ as follows:

$${\widehat{d}}_i={\widehat{q}}_i+K_{id}{\widehat{\upsilon }}_i$$

(15)

Let ${\tilde{q}}_i=q_i-{\widehat{q}}_i$. The time derivative of ${\tilde{q}}_i$ is described as follows:

$$\begin{array}{cc}\hfill {\dot{\tilde{q}}}_i& ={\dot{q}}_i-{\dot{\widehat{q}}}_i\hfill \\ \hfill & ={\dot{d}}_i-K_{id}{\tilde{\theta }}_i{\phi }_i({\widehat{\eta }}_i,{\widehat{\upsilon }}_i)-K_{id}{R}_i\left({\psi }_i\right)M_i^{-1}\left({\psi }_i\right){\tilde{q}}_i\hfill \end{array}$$

(16)

3.3. Adaptive Controller Design

The proposed adaptive controller is designed utilizing the backstepping technique. First, the change of coordinates is given as follows:

$$z_{i1}={\eta }_i-{\eta }_{id}$$

(17)

$$z_{i2}={\widehat{\upsilon }}_i-{\alpha }_i$$

(18)

where ${\eta }_{id}={\eta }_d+R_i\left({\psi }_i\right)p_i$, and ${\eta }_d={\left[x_d,y_d,{\psi }_d\right]}^{\top }$ represents the tracking trajectory of the leader. Here, $p_i={[p_{ix},p_{iy},0]}^{\top }$, where $p_{ix}$ and $p_{iy}$ denote the relative position between the i-th ASV and the leader in the $X_E$ and $Y_E$ directions, respectively. ${\alpha }_i$ serves as the virtual control for design purposes, where ${\alpha }_i^a$ and ${\alpha }_i^{*}$ are the adaptive virtual control and the optimal compensation term, respectively. The actual control combines these two terms, i.e., ${\alpha }_i={\alpha }_i^a+{\alpha }_i^{*}$.

From (17), we can obtain the following:

$$\begin{array}{cc}\hfill {\dot{z}}_{i1}& ={\dot{\eta }}_i-{\dot{\eta }}_{id}\hfill \\ \hfill & ={\upsilon }_i-{\dot{\eta }}_{id}\hfill \\ \hfill & =z_{i2}+{\alpha }_i^a+{\alpha }_i^{*}-{\dot{\eta }}_{id}\hfill \end{array}$$

(19)

To obtain the control objective, we design the following equation:

$$\begin{array}{c}{V}_{i1}={\displaystyle \frac{1}{2}}{\tilde{X}}_i^{\top }{P}_{i1}{\tilde{X}}_i+{\displaystyle \frac{1}{2}}{\tilde{Y}}_i^{\top }{P}_{i2}{\tilde{Y}}_i+{\displaystyle \frac{1}{2}}{\tilde{\Psi }}_i^{\top }{P}_{i3}{\tilde{\Psi }}_i+{\displaystyle \frac{1}{2}}{z}_{i1}^{\top }{z}_{i1}\hfill \end{array}$$

(20)

We can obtain the following:

$$\begin{array}{cc}\hfill {\dot{V}}_{i1}& ={\displaystyle \frac{1}{2}}{\dot{\tilde{X}}}_i^{\top }{P}_i{\tilde{X}}_i+{\displaystyle \frac{1}{2}}{\tilde{X}}_i^{\top }{P}_i{\dot{\tilde{X}}}_i+{\displaystyle \frac{1}{2}}{\dot{\tilde{Y}}}_i^{\top }{P}_{i2}{\tilde{Y}}_i+{\displaystyle \frac{1}{2}}{\tilde{Y}}_i^{\top }{P}_{i2}{\dot{\tilde{Y}}}_i\hfill \\ \hfill & +{\displaystyle \frac{1}{2}}{\dot{\tilde{\Psi }}}_i^{\top }{P}_{i3}{\tilde{\Psi }}_i+{\displaystyle \frac{1}{2}}{\tilde{\Psi }}_i^{\top }{P}_{i3}{\dot{\tilde{\Psi }}}_i+z_{i1}^{\top }{\dot{z}}_{i1}\hfill \\ \hfill & \le -{\displaystyle \frac{1}{2}}{\lambda }_{min}\left(Q_{i1}\right)\parallel {\tilde{X}}_i{\parallel }^2-{\displaystyle \frac{1}{2}}{\lambda }_{min}\left(Q_{i2}\right)\parallel {\tilde{Y}}_i{\parallel }^2-{\displaystyle \frac{1}{2}}{\lambda }_{min}\left(Q_{i3}\right){\parallel {\tilde{\Psi }}_i\parallel }^2\hfill \\ \hfill & +{\tilde{X}}_i^{\top }{P}_{i1}\left(B_{i1}{\tilde{\theta }}_i^{\top }{\phi }_i({\widehat{\eta }}_i,{\widehat{\upsilon }}_i)+B_{i1}{\epsilon }_i+D_i{B}_{i1}{\tilde{d}}_i\right)\hfill \\ \hfill & +{\tilde{Y}}_i^{\top }{P}_{i2}\left(B_{i2}{\tilde{\theta }}_i^{\top }{\phi }_i({\widehat{\eta }}_i,{\widehat{\upsilon }}_i)+B_{i2}{\epsilon }_i+D_i{B}_{i2}{\tilde{d}}_i\right)\hfill \\ \hfill & +{\tilde{\Psi }}_i^{\top }{P}_{i3}\left(B_{i3}{\tilde{\theta }}_i^{\top }{\phi }_i({\widehat{\eta }}_i,{\widehat{\upsilon }}_i)+B_{i3}{\epsilon }_i+D_i{B}_{i3}{\tilde{d}}_i\right)\hfill \\ \hfill & +z_{i1}^{\top }\left(z_{i2}+{\alpha }_i^a+{\alpha }_i^{*}-{\dot{\eta }}_{id}\right)\hfill \end{array}$$

(21)

By using Young’s inequality, we can obtain the following:

$${\tilde{X}}_i^{\top }{P}_{i1}{B}_{i1}{\tilde{\theta }}_i^{\top }{\phi }_i({\widehat{\eta }}_i,{\widehat{\upsilon }}_i)\le {\displaystyle \frac{1}{2}}{\lambda }_{\max }^2\left(P_{i1}\right)\parallel {\tilde{X}}_i{\parallel }^2+{\displaystyle \frac{1}{2}}{\parallel {\tilde{\theta }}_i\parallel }^2$$

(22)

$${\tilde{Y}}_i^{\top }{P}_{i2}{B}_{i2}{\tilde{\theta }}_i^{\top }{\phi }_i({\widehat{\eta }}_i,{\widehat{\upsilon }}_i)\le {\displaystyle \frac{1}{2}}{\lambda }_{\max }^2\left(P_{i2}\right)\parallel {\tilde{Y}}_i{\parallel }^2+{\displaystyle \frac{1}{2}}{\parallel {\tilde{\theta }}_i\parallel }^2$$

(23)

$${\tilde{\Psi }}_i^{\top }{P}_{i3}{B}_{i3}{\tilde{\theta }}_i^{\top }{\phi }_i({\widehat{\eta }}_i,{\widehat{\upsilon }}_i)\le {\displaystyle \frac{1}{2}}{\lambda }_{\max }^2\left(P_{i3}\right)\parallel {\tilde{\Psi }}_i{\parallel }^2+{\displaystyle \frac{1}{2}}{\parallel {\tilde{\theta }}_i\parallel }^2$$

(24)

$${\tilde{X}}_i^{\top }{P}_{i1}{B}_{i1}{\epsilon }_i\le {\displaystyle \frac{1}{2}}{\lambda }_{\max }^2\left(P_{i1}\right){\parallel {\tilde{X}}_i\parallel }^2+{\displaystyle \frac{1}{2}}{\epsilon }_{im}^2$$

(25)

$${\tilde{Y}}_i^{\top }{P}_{i2}{B}_{i2}{\epsilon }_i\le {\displaystyle \frac{1}{2}}{\lambda }_{\max }^2\left(P_{i2}\right){\parallel {\tilde{Y}}_i\parallel }^2+{\displaystyle \frac{1}{2}}{\epsilon }_{im}^2$$

(26)

$${\tilde{\Psi }}_i^{\top }{P}_{i3}{B}_{i3}{\epsilon }_i\le {\displaystyle \frac{1}{2}}{\lambda }_{\max }^2\left(P_{i3}\right){\parallel {\tilde{\Psi }}_i\parallel }^2+{\displaystyle \frac{1}{2}}{\epsilon }_{im}^2$$

(27)

$${\tilde{X}}_i^{\top }{P}_{i1}{D}_i{B}_{i2}{\tilde{d}}_i\le {\displaystyle \frac{1}{2}}{\lambda }_{\max }^2\left(P_{i1}{D}_i{B}_{i1}\right)\parallel {\tilde{X}}_i{\parallel }^2+{\displaystyle \frac{1}{2}}{\parallel {\tilde{d}}_i\parallel }^2$$

(28)

$${\tilde{Y}}_i^{\top }{P}_{i2}{D}_i{B}_{i2}{\tilde{d}}_i\le {\displaystyle \frac{1}{2}}{\lambda }_{\max }^2\left(P_{i2}{D}_i{B}_{i2}\right)\parallel {\tilde{X}}_i{\parallel }^2+{\displaystyle \frac{1}{2}}{\parallel {\tilde{d}}_i\parallel }^2$$

(29)

$${\tilde{\Psi }}_i^{\top }{P}_{i3}{D}_i{B}_{i3}{\tilde{d}}_i\le {\displaystyle \frac{1}{2}}{\lambda }_{\max }^2\left(P_{i3}{D}_i{B}_{i3}\right)\parallel {\tilde{\Psi }}_i{\parallel }^2+{\displaystyle \frac{1}{2}}{\parallel {\tilde{d}}_i\parallel }^2$$

(30)

The adaptive controller ${\alpha }_i^a$ can be designed as follows:

$${\alpha }_i^a=-r_{i1}{z}_{i1}+{\dot{\eta }}_{id}$$

(31)

where $r_{i1}=\mathrm{diag}(r_{i11},r_{i12},r_{i13})$ is the positive design parameter vector. Thus, we can obtain the following:

$$\begin{array}{cc}\hfill {\dot{V}}_{i1}\le & -{\sigma }_{i1}\parallel {\tilde{X}}_i{\parallel }^2-{\sigma }_{i2}\parallel {\tilde{Y}}_i{\parallel }^2-{\sigma }_{i3}{\parallel {\tilde{\Psi }}_i\parallel }^2-r_{i1}{z}_{i1}^{\top }{z}_{i1}+z_{i1}^{\top }{z}_{i2}+z_{i1}^{\top }{\alpha }_i^{*}\hfill \\ \hfill & +{\displaystyle \frac{3}{2}}\parallel {\tilde{\theta }}_i{\parallel }^2+{\displaystyle \frac{3}{2}}{\parallel {\tilde{d}}_i\parallel }^2+{\displaystyle \frac{3}{2}}{\epsilon }_{im}^2\hfill \end{array}$$

(32)

where ${\sigma }_{i1}={\displaystyle \frac{1}{2}}{\lambda }_{min}\left(Q_{i1}\right)-{\lambda }_{\max }^2\left(P_{i1}\right){\parallel {\tilde{X}}_i\parallel }^2-{\displaystyle \frac{1}{2}}{\lambda }_{\max }^2\left(P_{i1}{B}_{i1}\right)$, ${\sigma }_{i2}={\displaystyle \frac{1}{2}}{\lambda }_{min}\left(Q_{i2}\right)-{\lambda }_{\max }^2\left(P_{i2}\right)$$\parallel {\tilde{Y}}_i{\parallel }^2-{\displaystyle \frac{1}{2}}{\lambda }_{\max }^2\left(P_{i2}{B}_{i2}\right)$, ${\sigma }_{i3}={\displaystyle \frac{1}{2}}{\lambda }_{min}\left(Q_{i3}\right)-{\lambda }_{\max }^2\left(P_{i3}\right){\parallel {\tilde{\Psi }}_i\parallel }^2-{\displaystyle \frac{1}{2}}{\lambda }_{\max }^2\left(P_{i3}{B}_{i3}\right)$.

We design the following Lyapunov function:

$$\begin{array}{c}{V}_{i2}=V_{i1}+{\displaystyle \frac{1}{2}}{z}_{i2}^{\top }{z}_{i2}+{\displaystyle \frac{1}{2}}{\tilde{\theta }}_i^{\top }{\tilde{\theta }}_i+{\displaystyle \frac{1}{2}}{\tilde{q}}_i^{\top }{\tilde{q}}_i\hfill \end{array}$$

(33)

Utilizing the fact that ${\tilde{\theta }}_i={\theta }_i^{*}-{\widehat{\theta }}_i$, we can obtain the following equation:

$$\begin{array}{cc}\hfill {\dot{V}}_{i2}=& {\dot{V}}_{i1}+z_{i2}^{\top }{\dot{z}}_{i2}+{\tilde{\theta }}_i^{\top }{\dot{\tilde{\theta }}}_i+{\tilde{q}}_i^{\top }{\dot{\tilde{q}}}_i\hfill \\ \hfill =& {\dot{V}}_{i1}+z_{i2}^{\top }({\widehat{f}}_i({\widehat{\eta }}_i,{\widehat{\upsilon }}_i)+R_i\left({\psi }_i\right)M_i^{-1}({\tau }_i+d_i)\hfill \\ \hfill & -{\dot{\alpha }}_i+{\tilde{\theta }}_i{\phi }_i({\widehat{\eta }}_i,{\widehat{\upsilon }}_i))-{\tilde{\theta }}_i^{\top }{\dot{\widehat{\theta }}}_i\hfill \\ \hfill & +{\tilde{q}}_i^{\top }({\dot{d}}_i-K_{id}{\tilde{\theta }}_i{\phi }_i({\eta }_i,{\upsilon }_i)-K_{id}{R}_i\left({\psi }_i\right)M_i^{-1}{\tilde{q}}_i)\hfill \end{array}$$

(34)

By applying Young’s inequality, the following inequalities can be obtained:

$${\tilde{q}}_i{\dot{d}}_i\le {\displaystyle \frac{1}{2}}{\parallel \widetilde{q_i}\parallel }^2+{\displaystyle \frac{1}{2}}{d}_{im}^2$$

(35)

and

$${\tilde{q}}_i^{\top }{K}_{id}{\tilde{\theta }}_i{\phi }_i({\eta }_i,{\upsilon }_i)\le {\displaystyle \frac{1}{2}}\parallel \widetilde{q_i}{\parallel }^2+{\displaystyle \frac{1}{2}}{K}_{id}^2{\phi }_{im}^2{\parallel {\tilde{\theta }}_i\parallel }^2$$

(36)

We define $h_i\left(Z_{i2}\right)\triangleq f_i({\eta }_i,{\upsilon }_i)-f_i\left({\alpha }_i\right)$, where $Z_{i2}={[{\eta }_i,{\alpha }_i]}^{\top }$, and one has

$$\begin{array}{cc}\hfill {\dot{V}}_{i2}\le & {\dot{V}}_{i1}+z_{i2}^{\top }(h_i\left(Z_{i2}\right)+{\widehat{\theta }}_i{\phi }_i\left({\alpha }_i\right)+{\tilde{\theta }}_i{\phi }_i\left({\alpha }_i\right)+{\epsilon }_i\hfill \\ \hfill & +R_i\left({\psi }_i\right)M_i^{-1}({\tau }_i^a+{\tau }_i^{*}+d_i)-{\dot{\alpha }}_i)\hfill \\ \hfill & -{\tilde{\theta }}_i^{\top }{\dot{\widehat{\theta }}}_i-(K_{id}{R}_i{M}_i^{-1}\left({\psi }_i\right)-1){\parallel \widetilde{q_i}\parallel }^2\hfill \\ \hfill & +{\displaystyle \frac{1}{2}}{d}_{im}^2+{\displaystyle \frac{1}{2}}{K}_{id}^2{\phi }_{im}^2{\parallel {\tilde{\theta }}_i\parallel }^2\hfill \end{array}$$

(37)

We can design the adaptive controller and adaptive law as follows:

$$\begin{array}{c}\hfill {\tau }_i^a=M_i{R}_i^{\top }\left({\psi }_i\right)(-z_{i1}-z_{i2}-r_{i2}{z}_{i2}-{\widehat{\theta }}_i{\phi }_i\left({\alpha }_i\right)+{\dot{\alpha }}_i)-{\widehat{q}}_i-K_{id}{\upsilon }_i\end{array}$$

(38)

and

$${\dot{\widehat{\theta }}}_i=z_{i2}^{\top }{\phi }_i\left({\alpha }_i\right)-{\widehat{\theta }}_i-{\widehat{\theta }}_i{\parallel {\widehat{\theta }}_i\parallel }^2$$

(39)

where $r_{i2}=\mathrm{diag}(r_{i21},r_{i22},r_{i23})$ is the positive design vector. According to Young’s inequality, we can obtain that

$${\tilde{\theta }}_i^{\top }{\widehat{\theta }}_i\le -{\displaystyle \frac{1}{2}}\parallel {\tilde{\theta }}_i{\parallel }^2+{\displaystyle \frac{1}{2}}{\parallel {\theta }_i^{*}\parallel }^2$$

(40)

and

$${\tilde{\theta }}_i^{\top }{\widehat{\theta }}_i\parallel {\widehat{\theta }}_i{\parallel }^2\le -{\displaystyle \frac{1}{10}}\parallel {\tilde{\theta }}_i{\parallel }^4+{\displaystyle \frac{1}{2}}{\parallel {\theta }_i^{*}\parallel }^4$$

(41)

From (17) and (18), the tracking error dynamics can be described as follows:

$${\dot{Z}}_i=F_i\left(Z_i\right)+G_i{U}_i^{*}$$

(42)

where $Z_i={[z_{i1}^{\top },z_{i2}^{\top }]}^{\top }$, $F_i\left(Z_i\right)={[0_{3\times 3},h_i\left(Z_{i2}\right)]}^{\top }$, $G_i=\mathrm{diag}(I_{3\times 3},R_i{M}_i^{-1}\left({\psi }_i\right))$, $U_i^{*}={[{\alpha }_i^{*\top },{\tau }_i^{*\top }]}^{\top }$. The cost function of the tacking error system can be described as

$$J_i={\int }_{t_0}^{\infty }{r}_i\left(Z_i\left(t\right),U_i^{*}\left(t\right)\right)dt$$

(43)

where $r_i(Z_i,U_i^{*})=Q_i\left(Z_i\right)+U_i^{*\top }{R}_i{U}_i^{*}$ is the optimal index, $Q_i\left(Z_i\right)$ is a positive definite matrix satisfying $\parallel Q_i\left(Z_i\right)\parallel = 0$ only if $Z_i = 0$ and ${\Gamma }_{\mathrm{imin}}\le \parallel Q_i\left(Z_i\right)\parallel \le {\Gamma }_{\mathrm{imax}}$ for ${\overline{Z}}_{\mathrm{imin}}\le \parallel Z_i\parallel \le {\overline{Z}}_{\mathrm{imax}}$, ${\Gamma }_{\mathrm{imin}},{\Gamma }_{\mathrm{imax}}$, ${\overline{Z}}_{\mathrm{imin}}$, and ${\overline{Z}}_{\mathrm{imax}}$ are positive bounds. $R_i=R_i^{\top }$ is a positive definite matrix. An infinitesimal equivalent of (43) can be written as follows [31]:

$${\dot{J}}_i=\nabla J_i^{\top }(F_i\left(Z_i\right)+G_i{U}_i^{*})=-Q_i\left(Z_i\right)-U_i^{*\top }{R}_i{U}_i^{*}$$

(44)

We define a Hamiltonian function as

$$H_i(Z_i,U_i^{\prime })=r_i(Z_i,U_i^{\prime })+{(\nabla J_i\left(Z_i\right))}^{\top }(F_i\left(Z_i\right)+G_i{U}_i^{\prime })$$

(45)

where $U_i^{\prime }$ is the associated admissible control, and $\nabla J_i\left(Z_i\right)$ is the gradient of $J_i\left(Z_i\right)$ with regard to $Z_i$. The optimal controller $U_i^{*}\left(Z_i\right)$ can be obtained by applying the condition $\partial H_i(Z_i,U_i^{\prime })/\partial U_i^{\prime }=0$, and one has

$$U_i^{*}\left(Z_i\right)=-{\displaystyle \frac{1}{2}}{R}_i^{-1}{G}_i^{\top }\nabla J_i^{*}\left(Z_i\right)$$

(46)

where $\nabla J_i^{*}\left(Z_i\right)$ represents the gradient of $J_i^{*}\left(Z_i\right)$ with respect to $Z_i$. The HJB equation can be described as follows:

$$\begin{array}{c}{Q}_i\left(Z_i\right)+{(\nabla J_i^{*}\left(Z_i\right))}^{\top }F\left(Z_i\right)-{\displaystyle \frac{1}{4}}{(\nabla J_i^{*}\left(Z_i\right))}^{\top }{G}_i{R}_i^{-1}(\nabla J_i^{*}\left(Z_i\right))=0\hfill \end{array}$$

(47)

with $J_i^{*}\left(0\right)=0$. It can be concluded that $J_i$ is a continuously differentiable and radially unbounded Lyapunov function, and the derivative with an optimal controller can be described as

$${\dot{J}}_i=\nabla J_i^{\top }{\dot{Z}}_i=\nabla J_i^{\top }(F_i\left(Z_i\right)+G_i{U}_i^{*})\le 0$$

(48)

By choosing suitable $Q_i\left(Z_i\right)$ satisfying ${\lim }_{{\mathrm{Z}}_{\mathrm{i}}\to \infty }{\mathrm{Q}}_{\mathrm{i}}\left({\mathrm{Z}}_{\mathrm{i}}\right)=\infty$ and $\nabla J_i^{*\top }{Q}_i\left(Z_i\right)\nabla J_i$$=Q_i\left(Z_i\right)+U_i^{*\top }{R}_i{U}_i^{*}$, the following equation can be obtained [32]:

$$\nabla J_i^{\top }(F_i\left(Z_i\right)+G_i{U}_I^{*})=-\nabla J_i^{\top }{Q}_i\left(Z_i\right)\nabla J_i$$

(49)

From (37), (40), and (41), we can obtain that

$$\begin{array}{cc}\hfill {\dot{V}}_{i2}\le & -{\sigma }_{i1}\parallel {\tilde{X}}_i{\parallel }^2-{\sigma }_{i2}\parallel {\tilde{Y}}_i{\parallel }^2-{\sigma }_{i3}\parallel {\tilde{\Psi }}_i{\parallel }^2-{\gamma }_i\parallel Z_i{\parallel }^2-{\displaystyle \frac{1}{10}}\parallel {\tilde{\theta }}_i{\parallel }^4-({\displaystyle \frac{1}{2}}{K}_{id}^2{\phi }_{im}^2-{\displaystyle \frac{1}{2}}){\parallel {\tilde{\theta }}_i\parallel }^2\hfill \\ \hfill & -(K_{id}{R}_i\left({\psi }_i\right)M_i^{-1}-{\displaystyle \frac{1}{2}}{R}_i\left({\psi }_i\right)M_i^{-1}{M}_i{R}_i^{\top }\left({\psi }_i\right)){\parallel \widetilde{q_i}\parallel }^2\hfill \\ \hfill & +{\displaystyle \frac{1}{2}}\parallel {\theta }_i^{*}{\parallel }^2+{\displaystyle \frac{1}{2}}{\parallel {\theta }_i^{*}\parallel }^4+{\displaystyle \frac{3}{2}}{\epsilon }_{im}^2+{\displaystyle \frac{1}{2}}{d}_{im}^2+Z_i^{\top }\left(F_i\left(Z_i\right)+G_i{U}_i^{*}\right)\hfill \end{array}$$

(50)

where ${\gamma }_i={[r_{i1}^{\top },r_{i2}^{\top }]}^{\top }$. According to (49), it can be seen that $Z_i^{\top }\left(F_i\left(Z_i\right)+G_i{U}_i^{*}\right)$ becomes negative when the controller $U_i^{*}$ is designed by adopting the optimal control method to stabilize the tracking error system (42), which implies that the tracking error $Z_i$ remains bounded and the cost function $J_i$ is minimized. However, the HJB Equation (47) is difficult to solve, so the ADP method is applied in the next section to obtain $U_i^{*}$.

3.4. Optimal Compensation Term Design

To obtain the approximation of the optimal cost function, NN is adopted as

$$J_i^{*}\left(Z_i\right)={\theta }_{ib}^{*\top }{\phi }_{ib}\left(Z_i\right)+{\epsilon }_{ib}$$

(51)

Here, ${\theta }_{ib}^{*}$ represents the ideal parameter, ${\phi }_{ib}\left(Z_i\right)$ denotes the basis function, and ${\epsilon }_{ib}$ signifies the minimum approximation error. The gradient of the optimal cost function is given by

$$\partial J_i^{*}\left(Z_i\right)/{\partial }_i{Z}_i=\nabla {\phi }_{ib}^{\top }\left(Z_i\right){\theta }_{ib}^{*}+\nabla {\epsilon }_{ib}$$

(52)

where $\nabla {\phi }_{ib}^{\top }\left(Z_i\right)$ and $\nabla {\epsilon }_{ib}$ are the gradients of ${\phi }_{ib}^{\top }\left(Z_i\right)$ and ${\epsilon }_{ib}$, respectively. The optimal controller and the Hamiltonian function are expressed below:

$$U_i^{*}\left(Z_i\right)=-{\displaystyle \frac{1}{2}}{R}_i^{-1}\left(\nabla {\phi }_{ib}^{\top }\left(Z_i\right){\theta }_{ib}^{*}+\nabla {\epsilon }_{ib}\right)$$

(53)

$$\begin{array}{cc}\hfill H_i\left(Z_i,{\theta }_{ib}^{*}\right)=& Q_i\left(Z_i\right)+{\theta }_{ib}^{*\top }{\left(\nabla {\phi }_{ib}^{\top }\left(Z_i\right)\right)}^{\top }{H}_i\left(Z_i\right)+{\epsilon }_{i\mathrm{HJB}}\hfill \\ \hfill & -{\displaystyle \frac{1}{4}}{\theta }_{ib}^{*\top }{\left(\nabla {\phi }_{ib}^{\top }\left(Z_i\right)\right)}^{\top }{G}_i{R}_i^{-1}{G}_i^{\top }\nabla {\phi }_{ib}^{\top }\left(Z_i\right){\theta }_{ib}^{*}\hfill \end{array}$$

(54)

We can obtain the following equation:

$$\begin{array}{cc}\hfill {\epsilon }_{i\mathrm{HJB}}=& {(\nabla {\epsilon }_{ib})}^{\top }\left(F_i\left(Z_i\right)+G_i{U}_i^{*}\right)\hfill \\ \hfill & +{\displaystyle \frac{1}{4}}{(\nabla {\epsilon }_{ib})}^{\top }{G}_i{R}_i^{-1}{G}_i^{\top }\nabla {\epsilon }_{ib}\hfill \end{array}$$

(55)

Describing the approximation of the optimal cost function achieved by NN, we have

$${\widehat{J}}_i\left(Z_i\right)={\widehat{\theta }}_{ib}^{\top }{\phi }_{ib}\left(Z_i\right)$$

(56)

where ${\widehat{\theta }}_{ib}$ is the approximation of ${\theta }_{ib}^{*}$. The optimal controller estimation can be reformulated as follows:

$${\widehat{U_i}}^{*}\left(Z_i\right)=-{\displaystyle \frac{1}{2}}{R}_i^{-1}{G}_i^{\top }\nabla {\phi }_{ib}^{\top }\left(Z_i\right){\widehat{\theta }}_{ib}$$

(57)

The approximate Hamiltonian function can be obtained as follows:

$$\begin{array}{cc}\hfill {\widehat{H}}_i\left(Z_i,{\widehat{\theta }}_{ib}\right)=& Q_i\left(Z_i\right)+{\widehat{\theta }}_{ib}^{\top }{\left(\nabla {\phi }_{ib}^{\top }\left(Z_i\right)\right)}^{\top }{\widehat{F}}_i\left(Z_i\right|{\widehat{\Theta }}_i)\hfill \\ \hfill & -{\displaystyle \frac{1}{4}}{\widehat{\theta }}_{ib}^{\top }{\left(\nabla {\phi }_{ib}^{\top }\left(Z_i\right)\right)}^{\top }{G}_i{R}_i^{-1}{G}_i^{\top }\nabla {\phi }_i^{\top }\left(Z_i\right){\widehat{\theta }}_{ib}\hfill \end{array}$$

(58)

where ${\widehat{F}}_i\left(Z_i\right|{\widehat{\theta }}_i)={\left[{\widehat{f}}_{i1}\left(z_{i1}\right|{\widehat{\theta }}_{i1}),{\widehat{f}}_{i2}\left(z_{i2}\right|{\widehat{\theta }}_{i2}),\dots ,{\widehat{f}}_{in}\left(z_{in}\right|{\widehat{\theta }}_{in})\right]}^{\top }$, ${\widehat{f}}_{i1}\left(z_{i1}\right|{\widehat{\theta }}_{i1})={\widehat{f}}_{i1}\left(x_{i1}\right|{\widehat{\theta }}_{i1})-{\widehat{f}}_{i1}\left(x_{i1d}\right|{\widehat{\theta }}_{i1})$, ${\widehat{f}}_{ij}\left(z_{ij}\right|{\widehat{\theta }}_{ij})={\widehat{f}}_{ij}\left(x_{ij}\right|{\widehat{\theta }}_{ij})-{\widehat{f}}_{ij}\left(x_{ijd}\right|{\widehat{\theta }}_{ij})j=2,3,\dots ,n$.

We choose the weight updating law of ${\widehat{\theta }}_{ib}$ as follows:

$$\begin{array}{cc}\hfill {\dot{\widehat{\theta }}}_{ib}=& -\left[{\left(\nabla {\phi }_{ib}^{\top }\left(Z_i\right)\right)}^{\top }{\widehat{F}}_i(Z_i\mid \widehat{{\theta }_i})\right.\hfill \\ \hfill & \left.-{\displaystyle \frac{1}{2}}{\left(\nabla {\phi }_{ib}^{\top }\left(Z_i\right)\right)}^{\top }{G}_i{R}_i^{-1}{G}_i^{\top }\nabla {\phi }_{ib}^{\top }\left(Z_i\right){\widehat{\theta }}_{ib}\right]\hfill \\ \hfill & \times \left[Q_i\left(Z_i\right)+{\widehat{\theta }}_{ib}^{\top }{\left(\nabla {\phi }_{ib}^{\top }\left(Z_i\right)\right)}^{\top }{\widehat{F}}_i(Z_i\mid {\Theta }_i)\right.\hfill \\ \hfill & \left.-{\displaystyle \frac{1}{4}}{\widehat{\theta }}_{ib}^{\top }{\left(\nabla {\phi }_{ib}^{\top }\left(Z_i\right)\right)}^{\top }{G}_i{R}_i^{-1}{G}_i^{\top }\nabla {\phi }_{ib}^{\top }\left(Z_i\right){\widehat{\theta }}_{ib}\right]\hfill \end{array}$$

(59)

The estimation error of the optimal cost function parameter is defined as ${\tilde{\theta }}_{ib}={\theta }_{ib}^{*}-{\widehat{\theta }}_{ib}$. From this definition, we can obtain that

$$\begin{array}{cc}\hfill {\widehat{H}}_i(Z_i,{\widehat{\theta }}_{ib})=& {\displaystyle \frac{1}{2}}{\tilde{\theta }}_{ib}^{\top }{\left(\nabla {\phi }_{ib}^{\top }\left(Z_i\right)\right)}^{\top }{G}_i{R}_i^{-1}{G}_i^{\top }\nabla {\phi }_{ib}^{\top }\left(Z_i\right){\theta }_{ib}^{*}\hfill \\ \hfill & -{\tilde{\theta }}_{ib}^{\top }{\left(\nabla {\phi }_{ib}^{\top }\left(Z_i\right)\right)}^{\top }{\tilde{F}}_i\left(Z_i\right)-{\epsilon }_{i\mathrm{HJB}}\hfill \\ \hfill & -{\displaystyle \frac{1}{4}}{\tilde{\theta }}_{ib}^{\top }{\left(\nabla {\phi }_{ib}^{\top }\left(Z_i\right)\right)}^{\top }{G}_i{R}_i^{-1}{G}_i^{\top }\nabla {\phi }_{ib}^{\top }\left(Z_i\right){\tilde{\theta }}_{ib}\hfill \end{array}$$

(60)

The error dynamics of Equation (59) can be expressed as follows:

$$\begin{array}{cc}\hfill {\dot{\tilde{\theta }}}_{ib}=& -\left[{\left(\nabla {\phi }_{ib}^{\top }\left(Z_i\right)\right)}^{\top }\dot{Z_i}-{\left(\nabla {\phi }_{ib}^{\top }\left(Z_i\right)\right)}^{\top }\widetilde{F_i}\left(Z_i\right)\right.\hfill \\ \hfill & +{\left(\nabla {\phi }_{ib}^{\top }\left(Z_i\right)\right)}^{\top }{G}_i{R}_i^{-1}{G}_i^{\top }\nabla {\phi }_{ib}^{\top }\left(Z_i\right){\tilde{\theta }}_{ib}\hfill \\ \hfill & \left.+{\displaystyle \frac{1}{2}}{\left(\nabla {\phi }_{ib}^{\top }\left(Z_i\right)\right)}^{\top }{G}_i{R}_i^{-1}{G}_i^{\top }\nabla {\epsilon }_{ib}\left(Z_i\right)\right]\hfill \\ \hfill & \times \left[{\tilde{\theta }}_{ib}^{\top }{\left(\nabla {\phi }_{ib}^{\top }\left(Z_i\right)\right)}^{\top }\dot{Z_i}+{\widehat{\theta }}_{ib}^{\top }{\left(\nabla {\phi }_{ib}^{\top }\left(Z_i\right)\right)}^{\top }\widetilde{F_i}\left(Z_i\right)\right.\hfill \\ \hfill & +{\displaystyle \frac{1}{4}}{\tilde{\theta }}_{ib}^{\top }{\left(\nabla {\phi }_{ib}^{\top }\left(Z_i\right)\right)}^{\top }{G}_i{R}_i^{-1}{G}_i^{\top }\nabla {\phi }_{ib}^{\top }\left(Z_i\right){\tilde{\theta }}_{ib}\hfill \\ \hfill & \left.+{\displaystyle \frac{1}{2}}{\tilde{\theta }}_{ib}^{\top }{\left(\nabla {\phi }_{ib}^{\top }\left(Z_i\right)\right)}^{\top }{G}_i{R}_i^{-1}{G}_i^{\top }\nabla {\epsilon }_{ib}\left(Z_i\right)+{\epsilon }_{i\mathrm{HJB}}\right]\hfill \end{array}$$

(61)

where ${\tilde{F}}_i\left(Z_i\right)=F_i\left(Z_i\right)-{\widehat{F}}_i\left(Z_i\right|{\widehat{\theta }}_i)$.

3.5. Stability Analysis

Theorem 1.

For the multiple UMV systems (System (1)), the weight updating law is given by (39), the adaptive controller is defined by (38), the optimal compensation term is provided by (57), and the updated law for the cost function is specified by (59). By selecting the design parameters appropriately, the entire control scheme ensures the boundedness of all signals in the closed-loop system, and the system outputs can optimally track the reference signal.

Proof. 

Consider the following Lyapunov function:

$$\begin{array}{cc}\hfill V_i=& {\displaystyle \frac{1}{2}}{\tilde{X}}_i^{\top }{P}_{i1}{\tilde{X}}_i+{\displaystyle \frac{1}{2}}{\tilde{Y}}_i^{\top }{P}_{i2}{\tilde{Y}}_i+{\displaystyle \frac{1}{2}}{\tilde{\Psi }}_i^{\top }{P}_{i3}{\tilde{\Psi }}_i+{\displaystyle \frac{1}{2}}{z}_{i1}^{\top }{z}_{i1}\hfill \\ \hfill & +{\displaystyle \frac{1}{2}}{z}_{i2}^{\top }{z}_{i2}+{\displaystyle \frac{1}{2}}{\tilde{\theta }}_i^{\top }{\tilde{\theta }}_i+{\displaystyle \frac{1}{2}}{\tilde{\theta }}_{ib}^{\top }{\tilde{\theta }}_{ib}+{\displaystyle \frac{1}{2}}{\tilde{q}}_i^{\top }{\tilde{q}}_i\hfill \end{array}$$

(62)

We can obtain that

$$\begin{array}{cc}\hfill {\dot{V}}_i=& {\displaystyle \frac{1}{2}}{\dot{\tilde{X}}}_i^{\top }{P}_{i1}{\tilde{X}}_i+{\displaystyle \frac{1}{2}}{\tilde{X}}_i^{\top }{P}_{i1}{\dot{\tilde{X}}}_i+{\displaystyle \frac{1}{2}}{\dot{\tilde{Y}}}_i^{\top }{P}_{i2}{\tilde{Y}}_i+{\displaystyle \frac{1}{2}}{\tilde{Y}}_i^{\top }{P}_{i2}{\dot{\tilde{Y}}}_i+{\displaystyle \frac{1}{2}}{\dot{\tilde{\Psi }}}_i^{\top }{P}_{i3}{\tilde{\Psi }}_i\hfill \\ \hfill & +{\displaystyle \frac{1}{2}}{\tilde{\Psi }}_i^{\top }{P}_{i3}{\dot{\tilde{\Psi }}}_i+z_{i1}^{\top }{\dot{z}}_{i1}+z_{i2}^{\top }{\dot{z}}_{i2}+{\tilde{\theta }}_i^{\top }{\dot{\tilde{\theta }}}_i+{\tilde{\theta }}_{ib}^{\top }{\dot{\tilde{\theta }}}_{ib}+{\tilde{q}}_i^{\top }{\dot{\tilde{q}}}_i\hfill \end{array}$$

(63)

Assume that $∥{\left(\nabla {\phi }_{ib}^{\top }\left(Z_i\right)\right)}^{\top }{R}_i^{-1}\nabla {\phi }_{ib}^{\top }\left(Z_i\right)∥\le {\ell }_{i2}$, $F_i\left(Z_i\right)+G_i{U}_i^{*}\le c_i\sqrt{∥Z_i∥}$, $∥\nabla {\epsilon }_i\left(Z_i\right)∥$$\le {\epsilon }_{ibm}$, and $∥\nabla {\phi }_{ib}^{\top }\left(Z_i\right)∥\le {\phi }_{ibm}$, where ${\ell }_{i2}$, $c_i$, ${\epsilon }_{im}$, and ${\phi }_{ibm}$ are positive constants. By applying Young’s inequality and the Cauchy–Schwartz inequality to (63), we can obtain the following inequality:

$$\begin{array}{cc}\hfill {\dot{V}}_i\le & -{\ell }_{i1}{∥Z_i∥}^2+{\ell }_{i2}∥Z_i∥-{\ell }_{i3}{∥{\tilde{\theta }}_i∥}^4+{\ell }_{i4}{∥{\tilde{\theta }}_i∥}^2\hfill \\ \hfill & -{\ell }_{i5}{∥{\tilde{\theta }}_{ib}∥}^4+{\ell }_{i6}{∥{\tilde{\theta }}_{ib}∥}^2-{\ell }_{i7}{∥{\tilde{q}}_i∥}^2-{\ell }_{i8}{\parallel {\tilde{X}}_i\parallel }^2\hfill \\ \hfill & -{\ell }_{i9}\parallel {\tilde{Y}}_i{\parallel }^2-{\ell }_{i10}{\parallel {\tilde{\Psi }}_i\parallel }^2-{\ell }_{i11}\hfill \end{array}$$

(64)

where

$$\begin{array}{c}{\ell }_{i1}={\gamma }_i-{\displaystyle \frac{11}{4}}{c}_i^4-{\displaystyle \frac{1}{2}}{c}_i^4{\phi }_{ibm}^4\hfill \end{array}$$

(65)

$$\begin{array}{c}{\ell }_{i2}=2c_i^2{\phi }_{ibm}^2\hfill \end{array}$$

(66)

$$\begin{array}{c}{\ell }_{i3}={\displaystyle \frac{1}{10}}-{\displaystyle \frac{7}{2}}{\phi }_{im}^4\hfill \end{array}$$

(67)

$$\begin{array}{c}{\ell }_{i4}={\displaystyle \frac{1}{4}}{\phi }_{im}^2+2{\theta }_{ib}^{*}{\phi }_{im}^4-{\displaystyle \frac{1}{2}}\hfill \end{array}$$

(68)

$${\ell }_{i5}={\displaystyle \frac{1}{4}}{\pi }_{i5}^2-{\displaystyle \frac{99}{32}}{\phi }_{im}^4{∥G_i∥}^4{∥R_i^{-1}∥}^2-{\displaystyle \frac{77}{16}}{\phi }_{im}^4$$

(69)

$${\ell }_{i6}={\displaystyle \frac{3}{8}}{\phi }_{im}^2{∥G_i∥}^4{∥R_i^{-1}∥}^2{\epsilon }_{ibm}^2+{\displaystyle \frac{1}{8}}{\phi }_{im}^2{∥G_i∥}^4{∥R_i^{-1}∥}^2$$

(70)

$${\ell }_{i7}=K_{id}{R}_i\left({\psi }_i\right)M_i^{-1}\left({\psi }_i\right)-{\displaystyle \frac{1}{2}}-2K_{id}^2{\phi }_{im}^2-{\displaystyle \frac{1}{2}}{A}_i$$

(71)

$${\ell }_{i8}={\sigma }_{i1}$$

(72)

$${\ell }_{i9}={\sigma }_{i2}$$

(73)

$${\ell }_{i10}={\sigma }_{i3}$$

(74)

$$\begin{array}{cc}\hfill {\ell }_{i11}=& {\displaystyle \frac{3}{32}}{∥G_i∥}^8{∥R_i^{-1}∥}^4{\epsilon }_{ibm}^4+{\displaystyle \frac{3}{32}}{∥G_i∥}^4{∥R_i^{-1}∥}^2{\epsilon }_{ibm}^4\hfill \\ \hfill & +{\displaystyle \frac{1}{2}}{\epsilon }_{im}^2+{\displaystyle \frac{1}{2}}{d}_{im}^2+{\displaystyle \frac{1}{2}}\parallel {\theta }_i^{*}{\parallel }^2+{\displaystyle \frac{1}{2}}{\parallel {\theta }_i^{*}\parallel }^4\hfill \end{array}$$

(75)

Assume the following equations hold:

$$∥Z_i∥>{\displaystyle \frac{-{\ell }_{i2}+\sqrt{{\ell }_{i2}^2+4{\ell }_{i1}{\ell }_{i11}}}{2{\ell }_{i1}}}$$

(76)

or

$$∥{\tilde{\theta }}_i∥>\sqrt{{\displaystyle \frac{-{\ell }_{i4}+\sqrt{{\ell }_{i4}^2+4{\ell }_{i3}{\ell }_{i11}}}{2{\ell }_{i3}}}}$$

(77)

or

$$∥{\tilde{W}}_i∥>\sqrt{{\displaystyle \frac{-{\ell }_{i6}+\sqrt{{\ell }_{i6}^2+4{\ell }_{i5}{\ell }_{i11}}}{2{\ell }_{i5}}}}$$

(78)

or

$$∥{\tilde{X}}_i∥>\sqrt{{\displaystyle \frac{{\ell }_{i11}}{{\ell }_{i8}}}}$$

(79)

or

$$∥{\tilde{Y}}_i∥>\sqrt{{\displaystyle \frac{{\ell }_{i11}}{{\ell }_{i9}}}}$$

(80)

or

$$∥{\tilde{\Psi }}_i∥>\sqrt{{\displaystyle \frac{{\ell }_{i11}}{{\ell }_{i10}}}}$$

(81)

or

$$∥{\tilde{q}}_i∥>\sqrt{{\displaystyle \frac{{\ell }_{i11}}{{\ell }_{i7}}}}$$

(82)

Thus, according to standard Lyapunov extension [31], we can obtain ${\dot{V}}_{i3}<0$, and all signals within the closed-loop system remain bounded. □

4. Simulations and Comparison Study

4.1. Simulation Experiment

The simulation is applied by using MatlabR2018a. The parameter of CyberShip II is adopted in the simulation part; the details of this simulation model can be found in [33]. Three UMVs are adopted in the simulation experiment, which are named ${\mathrm{UMV}}_1$, ${\mathrm{UMV}}_2$, and ${\mathrm{UMV}}_3$, respectively. $R_i$ is designed as $0.01\times I_{3\times 3}$. The initial position of the three UMVs are $x_1\left(0\right)=0,y_1\left(0\right)=0.7,{\psi }_1=10$, $x_2\left(0\right)=-0.3,y_2\left(0\right)=-0.5,{\psi }_2=-20$, $x_3\left(0\right)=0.3,y_3\left(0\right)=-0.5,{\psi }_3=10$, and $x_4\left(0\right)=0.4,y_4\left(0\right)=0,{\psi }_4=-20$, respectively. The parameters of related position are given as follows: $p_{1x}=\sqrt{2}/2$, $p_{1y}=0$, $p_{2x}=-1/2$, $p_{2y}=-1/2$, $p_{3x}=-1/2$, $p_{3y}=1/2$, $p_{4x}=-1/5$, and $p_{4y}=0$. $k_{i11}=k_{i12}=k_{i21}=k_{i22}=k_{i31}=k_{i32}=50$, and $k_{id}=10$. The applied disturbances are chosen as follows: $d_{i1}=20\left(0.5cos\left(0.25t\right)+0.5sin\left(0.15t\right)\right)$, $d_{i2}=20\left(-0.5cos\left(0.15t\right)-0.5sin\left(0.25t\right)\right)$, and $d_{i3}=2\left(0.5cos\left(0.25t\right)+0.5sin\left(0.15t\right)\right)$. The desired tracking signal of the virtual leader is given as ${\eta }_d{=[10\sin \left(0.02\mathrm{t}\right),10(1-\cos \left(0.02\mathrm{t}\right),0.02\mathrm{t}]}^{\top }$.

The simulation results are presented in Figure 2, Figure 3, Figure 4, Figure 5, Figure 6, Figure 7, Figure 8, Figure 9, Figure 10, Figure 11, Figure 12, Figure 13 and Figure 14. The tracking trajectory of the UMVs is shown in Figure 2, where it can be seen that the tracking control objective is achieved, and the UMVs maintain the desired formation. The tracking errors in the x and y directions are shown in Figure 3 and Figure 4, respectively. Initially, the errors are large due to the distance between the UMVs’ initial positions and the desired positions. However, once the UMVs approach the desired positions, the tracking errors tend to zero. Figure 5, Figure 6 and Figure 7 present the observer errors of the NSO. Since the initial value of the NSO is different from the actual value, the initial error is large. However, the designed updating law helps the NSO’s estimates converge to the true velocity values. The disturbance observer (DO) errors are shown in Figure 8, Figure 9 and Figure 10. As mentioned before, the applied disturbances $d_{i1}$, $d_{i2}$, and $d_{i3}$ oscillate within the range of (−20, 20), (−20, 20), and (−2, 2), respectively. From Figure 8, Figure 9 and Figure 10, it can be observed that the DO errors stabilize within the range of [(−2, 2); (−4, 4); (−0.5, 0.5)], demonstrating that the designed DO effectively can reduce the impact of disturbances. The controller outputs are shown in Figure 11, Figure 12 and Figure 13, while the designed optimal index is shown in Figure 14. It can be seen that the proposed controller minimizes the optimal index.

In conclusion, the simulation results demonstrate that the proposed optimal algorithm successfully handles the control task, and the NSO and DO can estimate the unmeasured velocities and unknown disturbances.

4.2. Comparison Study

To further demonstrate the disturbance rejection capability of the proposed algorithm, a comparative study is presented in this part. The controller adopted in this part is a fuzzy adaptive optimal controller without the disturbance rejection technique [34]. The experiment conditions, initial conditions, and applied disturbances are the same with the conditions given in Section 4.1.

The simulation results are presented in Figure 15, Figure 16, Figure 17, Figure 18, Figure 19, Figure 20 and Figure 21. As shown in Figure 15, the controller proposed in [34] achieves the tracking control objective. However, in Figure 16 and Figure 17, it is evident that the tracking errors are larger compared to Figure 3 and Figure 4. From Figure 18, Figure 19 and Figure 20, it can be observed that the controller output is larger compared to Figure 11, Figure 12 and Figure 13. This suggests that, under the applied disturbances, the lack of a disturbance rejection technique leads to larger tracking errors, which in turn requires a greater controller output to decrease tracking errors and achieve the control objective. This increased output results in a higher optimal index, as seen in Figure 21. Comparing Figure 21 to Figure 14, it becomes clear that the proposed controller achieves better performance in minimizing the optimal index. Therefore, the simulation results in Figure 15, Figure 16, Figure 17, Figure 18, Figure 19, Figure 20 and Figure 21 demonstrate that, under the disturbances $d_{i1}$, $d_{i2}$, and $d_{i3}$ given in Section 4.1, the proposed control method with the DO demonstrates better control performance compared to the existing optimal control method without a disturbance rejection technique [34].

5. Conclusions

This paper has presented an observer-based adaptive optimal controller for tracking control problems of multiple UMVs with uncertain dynamics and unmeasured velocities. NSOs have been developed to manage the unmeasured velocities and the uncertain dynamics of the UMVs. An adaptive controller with an optimal compensation term has been implemented to achieve the control objective optimally. Additionally, a DO has been proposed to address unknown external disturbances. It has been proven that the proposed controller ensures all signals in the closed-loop system remain bounded. Simulation results have been provided to demonstrate the effectiveness of the proposed algorithm. From these results, it can be concluded that the proposed control algorithm has successfully achieved the control objectives and minimized the optimal index. The NSO and DO have effectively estimated the unmeasured velocities and unknown disturbances. To further illustrate the disturbance rejection capability of the proposed algorithm, a comparison study has been conducted using an existing method [34]. The simulation results of this comparison study have shown that, under the appied external disturbances, the proposed control algorithm can achieve better control performance by adopting the disturbance rejection technique.