Optimal Robust Control of Nonlinear Systems with Unknown Dynamics via NN Learning with Relaxed Excitation

Abstract

This paper presents an adaptive learning structure based on neural networks (NNs) to solve the optimal robust control problem for nonlinear continuous-time systems with unknown dynamics and disturbances. First, a system identifier is introduced to approximate the unknown system matrices and disturbances with the help of NNs and parameter estimation techniques. To obtain the optimal solution of the optimal robust control problem, a critic learning control structure is proposed to compute the approximate controller. Unlike existing identifier-critic NNs learning control methods, novel adaptive tuning laws based on Kreisselmeier’s regressor extension and mixing technique are designed to estimate the unknown parameters of the two NNs under relaxed persistence of excitation conditions. Furthermore, theoretical analysis is also given to prove the significant relaxation of the proposed convergence conditions. Finally, effectiveness of the proposed learning approach is demonstrated via a simulation study.

1. Introduction

In the past several decades, much attention has been given to $H_{\infty }$ control problems, wherein the aim is to eliminate the influence of disturbance on the system. $H_{\infty }$ control mainly focuses on designing a robust controller to regulate and stabilize the system. In practice, we should not only focus on the control performance, but also consider the optimization of the system [1,2]. Therefore, optimal $H_{\infty }$ control problems will always be a hot research topic.

Adaptive dynamic programming (ADP), as one of the optimal control methods, has emerged as a powerful tool through which to deal with the optimal control problems of all kinds of dynamic systems [3]. The ADP framework combines dynamic programming and neural network approximation, and it has strong learning and adaptive ability. In this sense, ADP has rapidly developed in the control community in recent years. Generally speaking, the core of controller designs mainly concentrates on solving a Hamilton–Jacobi–Bellman (HJB) equation for nonlinear systems or an algebraic Riccati equation for linear systems [4]. Unfortunately, the HJB equation contains nonlinear, partial differential parts, which are difficult to solve directly [5]. Therefore, many efforts have been made for finding approximate solutions to the HJB equation using iterative or learning methods. Regarding the case of iterative methods, the ADP can be classed into two categories: value iteration (VI) [6,7] and policy iteration (PI) [8,9]. Regarding the case of learning-based methods, neural network (NN) approximation is generally utilized to learn the optimal or suboptimal solutions to the HJB equation. The standard learning frameworks include the following: actor–critic NNs and only-critic NNs. However, the abovementioned pieces of literature require partial or full model information in the controller design loop. To avoid relying on system models, many data-driven or model-free methods have been developed for improving the existing ADP frameworks, that is, data-driven RL [7], integral RL (IRL) [10,11], and system identification-based ADP methods [12,13,14].

More recently, excellent development has been realized with the use of ADP for the robust controller designs of optimal $H_{\infty }$ control problems [15,16,17]. The main way through which to solve optimal $H_{\infty }$ control problems is to model such problems as a two-player zero-sum game (min–max optimization problem), where the controller and the disturbance are viewed as players that try to find a controller to minimize the performance index function in worst-case disturbance conditions [18,19]. However, the disadvantage of zero-sum games is in judging the existence of the saddle point, which is generally difficult to judged. In order to overcome this issue, an indirect method motivated by [20] was developed by formulating an optimal regulation for a nominal system with new designs of the cost/value function [21]. For instance, Yang et al. proposed an event-triggered robust control strategy for nonlinear systems [22] using the indirect method. Xue et al. studied a tracking control problem for partial continuous-time systems with uncertainties and constraints [23] by transforming the robust control problem into an optimal regulation of nominal systems.

However, the existing results on $H_{\infty }$ optimal control designs have two main characteristics: (1) their controller designs are based on the assumption that the complete or partial knowledge of the system dynamics are known in advance; however, (2) to address this issue, some system identification methods have been proposed, such as the identifier–critic- or identifier–actor–critic-based designs of $H_{\infty }$ optimal control. However, it is generally required that the persistence of excitation (PE) condition must be satisfied to ensure the learning performance of the weight updating of neural networks, which is difficult to check online in practice [18,19,23]. Therefore, how to weaken the PE condition is also the research motivation of this paper.

From the abovementioned observations and considerations, in this paper, we propose a novel online parameter estimation method based on an identifier–critic learning control framework for the $H_{\infty }$ optimal control of nonlinear systems that have unknown dynamics with relaxed PE conditions. The contributions of our work can be summarized as follows:

• A new online identifier–critic learning control framework with a relaxed PE condition is proposed to address robust control for unknown continuous-time systems subject to unknown disturbances. To reconstruct the information of the system dynamics, neural networks combined with the linear regressor method are established to approximate the unknown system dynamics and disturbances.
• The approach in this paper is different from the existing weight adaption laws [18,19,23], where the PE condition is needed to ensure the learning performance of the NN’s weight parameters. However, such a condition is difficult to check online, and a general way through which to satisfy this condition is to add external noise to the controller, which may lead to the instability of the system. To overcome this issue, a Kreisselmeier regressor extension and mixing (KREM)-based weight adaption law is designed for identifier–critic NNs with new convergence conditions.
• Weak PE properties of new convergence conditions are analyzed rigorously compared to traditional PE conditions. Moreover, the theoretical results indicate that the closed-loop system’s stability and the convergence of identifier–critic learning are guaranteed.

The remainder of this article is organized as follows. In Section 2, some preliminaries are introduced and the optimal robust control problem of nonlinear continuous-time systems is given. Then, a system identifier design with a relaxed PE condition is constructed in Section 3. Section 4 gives the critic NN design for robust control under a relaxed PE condition. Theoretical analyses of the weak PE properties under new convergence conditions and the stability of the closed-loop systems are given in Section 5. The simulation results are provided in Section 6. Some conclusions are summarized in Section 7.

2. Preliminaries and Problem Formulation

In this section, some notation and definitions are first introduced. Then, the optimal robust control problem of the nonlinear continuous-time systems is described.

2.1. Preliminaries

To facilitate readability, some notations are listed.

$\lambda (·)$	Eigenvalue of a matrix
${\{·\}}^{*}$	Adjoint matrix
$I_n$	Identity matrix
$tr(·)$	Trace of a matrix
${\lambda }_M(·)$	Maximum eigenvalues
${\lambda }_m(·)$	Minimum eigenvalues

The following definitions will be used in the sequel.

Definition 1

($PersistenceofExcitation$ [24]). A bounded signal $\psi \left(t\right)$ is said to be PE, if there exist positive constants T and ${\delta }_1$ such that

$${\int }_t^{t+T}\psi \left(r\right){\psi }^T\left(r\right)dr\ge {\delta }_{1}I.$$

For clarity, we indicate that $\psi \left(t\right)$ satisfies the PE condition using the notation $\psi \left(t\right)\in PE$; otherwise, $\psi \left(t\right)\notin PE$.

Definition 2

($UniformlyUltimatelyBounded$ [24]). The time function $x\left(t\right)$ is said to be uniformly ultimately bounded (UUB) on a compact set ${\Omega }_x$, if, for all $x\left(t_0\right)=x_0\in {\Omega }_x$, there exists a ${\delta }_2>0$ and a number $T({\delta }_2,x_0)$ such that $\parallel x\left(t\right)\parallel <{\delta }_2$ for all $t\ge t_0+T$.

2.2. Problem Formulation

Consider the nonlinear continuous-time (NCT) systems with disturbances described by the following dynamics:

$$\begin{array}{cc}\hfill & \dot{x}\left(t\right)=f\left(x\right)+g\left(x\right)u\left(t\right)+G\left(x\right)d\left(t\right),\hfill \end{array}$$

(1)

where $x\left(t\right)\in {\mathbb{R}}^n$ and $u\left(t\right)\in {\mathbb{R}}^m$ denote the system state and control input, respectively. $d\left(t\right)\in {\mathbb{R}}^q$ represents the external disturbance. The terms $f\left(x\right)\in {\mathbb{R}}^n$, $g\left(x\right)\in {\mathbb{R}}^{n\times m}$, and $G\left(x\right)\in {\mathbb{R}}^{n\times q}$ are the drift dynamics, input dynamics, and disturbance injection dynamics, respectively. In this study, $f\left(x\right)$, $g\left(x\right)$, and $G\left(x\right)$ are assumed to be unknown. Furthermore, it is assumed that $f\left(x\right)$, $g\left(x\right)$, and $G\left(x\right)$ are Lipschitz continuous with $f\left(0\right)=0$, and that the system (1) is stabilizing and controllable.

The goal of this study is to solve an $H_{\infty }$ control problem for the system (1). This problem can be equivalently transformed into a two-player zero-sum game, where the control input $u\left(t\right)$ acts as the minimizing player and the disturbance $d\left(t\right)$ acts as the maximizing player. The solution to the $H_{\infty }$ control problem corresponds to a saddle point in the game, which stabilizes the equilibrium of the two-player zero-sum game.

Define the infinite-horizon performance index function as

$$\begin{array}{c}\hfill V(x,u,d)={\int }_t^{\infty }\left(x^{T}Qx+u^{T}Ru-{\kappa }^2{d}^{T}d\right)d\tau ,\end{array}$$

(2)

where $\kappa >0$, $V\left(0\right)=0$, and Q and R are symmetric positive-definite matrices with appropriate dimensions. Let $u^{★}$ be the optimal control input and $d^{★}$ be the worst disturbance. Our objective is to find the saddle point $(u^{★},d^{★})$ that optimizes the performance index (2), which can be more precisely clarified by the following inequality:

$$\begin{array}{c}\hfill V(u^{★},d)\le V(u^{★},d^{★})\le V(u,d^{★}).\end{array}$$

(3)

We then define the optimal performance index function $V^{★}$ as follows:

$$\begin{array}{c}\hfill V^{★}(x,u,d)=\underset{u}{min}\underset{d}{max}{\int }_t^{\infty }\left(x^{T}Qx+u^{T}Ru-{\kappa }^2{d}^{T}d\right)d\tau .\end{array}$$

(4)

The Hamiltonian of system (1) can be written as  

$$\begin{array}{cc}\hfill H(V_x,x,u,d)=& V_x^T[f\left(x\right)+g\left(x\right)u+G\left(x\right)d]+x^{T}Qx+u^{T}Ru-{\kappa }^2{d}^{T}d,\hfill \end{array}$$

(5)

where $V_x=\partial V/\partial x\in {\mathcal{R}}^n$. The Hamilton–Jacobi–Isaacs (HJI) equation related to this game has the form

$$\begin{array}{c}\hfill \underset{u}{min}\underset{d}{max}H(V_x^{★},x,u,d)=0,\end{array}$$

(6)

where $V_x^{★}=\partial V^{★}/\partial x\in {\mathcal{R}}^n$. Based on the stationarity condition, the $H_{\infty }$ control pair $(u^{★},d^{★})$ for (1) has the following form:

$$\begin{array}{c}\hfill u^{★}=-\frac{1}{2}{R}^{-1}{g}^T\left(x\right)V_x^{★}\left(x\right),\end{array}$$

(7)

$$\begin{array}{c}\hfill d^{★}=\frac{1}{2{\kappa }^2}{G}^T\left(x\right)V_x^{★}\left(x\right).\end{array}$$

(8)

Thus, according to (7) and (8), the HJI Equation (6) can be rewritten as

$$\begin{array}{cc}\hfill x^{T}Qx& +V_x^{★T}f\left(x\right)-\frac{1}{4}{V}_x^{★T}g\left(x\right)R^{-1}{g}^T\left(x\right)V_x^{★}+\frac{1}{4{\kappa }^2}{V}_x^{★T}G\left(x\right)G^T\left(x\right)V_x^{★}=0.\hfill \end{array}$$

(9)

Indeed, the HJI Equation (9) represents a highly nonlinear partial differential equation (PDE) and requires complete system information for its resolution. To address these challenges, a new IC framework with relaxed PE conditions will be proposed in the following sections. Furthermore, new adaptive update laws for the identifier and critic NNs are provided with the help of the KREM technique. The block diagram of the proposed control system is shown in Figure 1, and detailed theoretical analysis will be presented in subsequent sections.

Figure 1. Schematic of the proposed control system.

3. System Identifier Design with Relaxed PE Condition

In this section, an NN-based identifier is utilized to reconstruct the unknown system dynamics in (1). The KREM technique is introduced to adjust the identifier weights under relaxed PE conditions. We assume that the unknown system dynamics $f\left(x\right)$, $g\left(x\right)$, and $G\left(x\right)$ in (1) are continuous functions defined on compact sets. The NN-based identifier is designed as follows:

$$f\left(x\right)=W_f{\theta }_f\left(x\right)+{ϵ}_f,$$

(10)

$$g\left(x\right)=W_g{\theta }_g\left(x\right)+{ϵ}_g,$$

(11)

$$G\left(x\right)=W_G{\theta }_G\left(x\right)+{ϵ}_G,$$

(12)

where $W_f\in {\mathcal{R}}^{n\times d_f}$, $W_g\in {\mathcal{R}}^{n\times d_g}$ and $W_G\in {\mathcal{R}}^{n\times d_G}$ are the ideal NN weights; ${\theta }_f\left(x\right)\in {\mathcal{R}}^{d_f}$, ${\theta }_g\left(x\right)\in {\mathcal{R}}^{d_g\times m}$ and ${\theta }_G\left(x\right)\in {\mathcal{R}}^{d_G\times q}$ are the basis functions; and ${ϵ}_f\in {\mathcal{R}}^n$, ${ϵ}_g\in {\mathcal{R}}^{n\times m}$ and ${ϵ}_G\in {\mathcal{R}}^{n\times q}$ are the reconstruction errors. Then, according to the Weierstrass theorem and the statements in [10], the approximation errors ${ϵ}_f$, ${ϵ}_g$, and ${ϵ}_G$ can be shown to approach zero as the number of NN neurons $d_f$, $d_g$, and $d_G$ increases to infinity.

Before proceeding, it is essential to establish the following underlying assumption.

Assumption 1.

(1) 

The basis functions ${\theta }_f\left(x\right)$, ${\theta }_g\left(x\right)$ and ${\theta }_G\left(x\right)$ are bounded, that is, $∥{\theta }_f\left(x\right)∥\le b_{{\theta }_f}$, $∥{\theta }_g\left(x\right)∥\le b_{{\theta }_g}$, $∥{\theta }_G\left(x\right)∥\le b_{{\theta }_G}$, respectively.

(2) 

The reconstruction errors ${\epsilon }_f$, ${\epsilon }_g$ and ${\epsilon }_G$ are bounded, that is, $∥{\epsilon }_f∥\le b_{{\epsilon }_f}$, $∥{\epsilon }_g∥\le b_{{\epsilon }_g}$, $∥{\epsilon }_G∥\le b_{{\epsilon }_G}$, respectively.

Using (10)–(12), the system (1) can be rewritten as

$$\dot{x}=W_I^T{\theta }_I(x,u)+{ϵ}_T,$$

(13)

where $W_I={[W_f,W_g,W_G]}^T\in {\mathcal{R}}^{d\times n}$ is the augmented weight matrix with $d=d_f+d_g+d_G$, and ${\theta }_I(x,u)={[{\theta }_f^T\left(x\right),u^T{\theta }_g^T\left(x\right),d^T{\theta }_G^T\left(x\right)]}^T\in {\mathcal{R}}^d$ is the augmented regressor vector. ${ϵ}_T={ϵ}_f+{ϵ}_{g}u+{ϵ}_{G}d\in {\mathcal{R}}^n$ is the model approximation error.

Note that $\dot{x}$ and $W_I$ are unknown. Therefore, we define the filtered variables $x_f$ and ${\theta }_{If}$ as

$$\left\{\begin{array}{c}\rho {\dot{x}}_f+x_f=x,x_f\left(0\right)=0\hfill \\ \rho {\dot{\theta }}_{If}+{\theta }_{If}={\theta }_I,{\theta }_{If}\left(0\right)=0\hfill \end{array}\right.$$

(14)

where $\rho \in \mathcal{R}>0$ is the filter coefficient. From Equations (13) and (14), we can deduce that

$${\dot{x}}_f=\frac{x-x_f}{\rho }=W_I^T{\theta }_{If}+{ϵ}_{Tf},$$

(15)

where ${ϵ}_{Tf}$ denotes the filtered version of ${ϵ}_T$ as $\rho {\dot{ϵ}}_{Tf}+{ϵ}_{Tf}={ϵ}_T$. Clearly, (15) is a linear regression equation (LRE), where ${\dot{x}}_f$ and ${\theta }_{If}$ can be calculated from (14). In the following, we describe how the KREM technique is applied to estimate $W_I$ by using the measured information ${\dot{x}}_f$ and ${\theta }_{If}$.

To approximate the unknown weights $W_I$ in (15) such that the estimated weights ${\widehat{W}}_I$ converge to their true values under a relaxed PE condition, we aim to construct an extended LRE (E-LRE) based on (15). We define the matrices $P_I\in {\mathbb{R}}^{d\times d}$ and $Q_I\in {\mathbb{R}}^{d\times n}$ as follows:

$$\begin{array}{c}\hfill \{\begin{array}{c}{P}_I={\mathcal{H}}_I\left[{\theta }_{If}{\theta }_{If}^T\right],P_I\left(0\right)=0\hfill \\ Q_I={\mathcal{H}}_I\left[{\theta }_{If}{\left(\frac{x-x_f}{\rho }\right)}^T\right],Q_I\left(0\right)=0\hfill \end{array}\end{array}$$

(16)

where

$$\begin{array}{c}\hfill {\mathcal{H}}_I=\frac{1}{p+l_I}\left[s\right]\left(t\right)\end{array}$$

with $p=d/dt$, $l_I>0$ is a forgetting factor. From (16), we can derive its solution as

$$\{\begin{array}{c}{P}_I={\int }_0^t{e}^{-l_I(t-\tau )}{\theta }_{If}\left(\tau \right){\theta }_{If}^T\left(\tau \right)d\left(\tau \right),\hfill \\ Q_I={\int }_0^t{e}^{-l_I(t-\tau )}{\theta }_{If}\left(\tau \right){\left(\frac{x\left(\tau \right)-x_f\left(\tau \right)}{\rho }\right)}^{T}d\left(\tau \right).\hfill \end{array}$$

(17)

Note that it can be verified that $P_I$ and $Q_I$ are bounded for any given bounded ${\theta }_I$ and x due to the appropriate choice of $l_I$. Thus, an E-LRE is obtained

$$Q_I\left(t\right)=P_I\left(t\right)W_I+v_I,$$

(18)

where $v_I={\int }_0^t{e}^{-l_I(t-\tau )}{\theta }_{If}\left(\tau \right){ϵ}_{Tf}^T\left(\tau \right)d\left(\tau \right)$.

To construct an identifier weight error dynamics that achieves better convergence properties, we define the variables ${\mathcal{Q}}_I\left(t\right)\in {\mathbb{R}}^{d\times n}$, ${\mathcal{P}}_I\in {\mathbb{R}}^{d\times d}$, and ${\mathcal{V}}_I\in {\mathbb{R}}^{d\times n}$ as follows:

$$\begin{array}{c}\hfill \{\begin{array}{c}{\mathcal{Q}}_I=P_I^{*}{Q}_I,\hfill \\ {\mathcal{P}}_I=P_I^{*}{P}_I,\hfill \\ {\mathcal{V}}_I=P_I^{*}{v}_I.\hfill \end{array}\end{array}$$

(19)

Then Equation (18) becomes

$${\mathcal{Q}}_I\left(t\right)={\mathcal{P}}_I\left(t\right)W_I+{\mathcal{V}}_I.$$

(20)

Note that for any square matrix $M\in {\mathcal{R}}^{q\times q}$, we have $M^{*}M=\left|M\right|I_q$, even if M is not full rank. Thus, ${\mathcal{P}}_I=\left|P_I\right|I_d\in {\mathcal{R}}^{d\times d}$. Moreover, ${\mathcal{P}}_I$ is a scalar diagonal matrix, where (20) can be decoupled into a series of scalar LREs:

$${{\mathcal{Q}}_I}_{(i,j)}\left(t\right)=\left|P_I\right|\left(t\right){W_I}_{(i,j)}+{\mathcal{V}}_{I(i,j)},i=1,\dots ,d,j=1,\dots ,n,$$

(21)

where ${{\mathcal{Q}}_I}_{(i,j)}$ and ${W_I}_{(i,j)}$ indicate the ith row and jth column of ${\mathcal{Q}}_I$ and $W_I$, respectively.

Then, the estimation algorithm for the unknown identifier NN weights can be designed based on (21) as follows:

$${\dot{\widehat{W}}}_{I(i,j)}=-{\gamma }_1|P_I\left|\right[|P_I|{\widehat{W}}_{I(i,j)}-{\mathcal{Q}}_{I(i,j)}],$$

(22)

where ${\gamma }_1\in \mathcal{R}>0$ presents the adaptive learning gain.

The convergence of identifier (22) can be given as follows.

Theorem 1.

Consider the system (13) with the online update law (22); if $|P_I|\in PE$, then

(i) 

for ${ϵ}_T=0$, the estimator error ${\tilde{W}}_{I(i,j)}$ converges to zero exponentially;

(ii) 

for ${ϵ}_T\ne 0$, the estimator error ${\tilde{W}}_{I(i,j)}$ converges to a compact set around zero.

Proof. 

If $|P_I|\in PE$, according to Definition 1 we have ${\int }_t^{t+T}|P_I{|}^{2}dr\ge {\delta }_I>0$. Defining the estimation error ${\tilde{W}}_{I(i,j)}={\widehat{W}}_{I(i,j)}-W_{I(i,j)}$, $i=1,\dots ,d$, $j=1,\dots ,n$. Due to (21) and (22), the identifier weight error dynamics can be obtained

$${\dot{\tilde{W}}}_{I(i,j)}=-{\gamma }_1|P_I{|}^2{\tilde{W}}_{I(i,j)}+{\gamma }_1\left|P_I\right|{\mathcal{V}}_{I(i,j)}.$$

(23)

Considering the Lyapunov function $V_I=0.5{\gamma }_1^{-1}{\tilde{W}}_{I(i,j)}^2$, the derivation of $V_I$ can be calculated as

$$\begin{array}{cc}\hfill {\dot{V}}_I& =\frac{1}{{\gamma }_1}{\tilde{W}}_{I(i,j)}{\dot{\tilde{W}}}_{I(i,j)}\hfill \\ \hfill & =-|P_I{|}^2{\tilde{W}}_{I(i,j)}^2+\left|P_I\right|{\tilde{W}}_{I(i,j)}{\mathcal{V}}_{I(i,j)}.\hfill \end{array}$$

(24)

In fact, when ${ϵ}_T=0$, (24) can be rewritten as

$$\begin{array}{c}\hfill {\dot{V}}_I=-{\left|P_I\right|}^2{\tilde{W}}_{I(i,j)}^2<-{\mu }_I{V}_I,\end{array}$$

(25)

where ${\mu }_I=2{\gamma }_1{\delta }_I>0$. According to the Lyapunov theorem, the weight estimation error ${\tilde{W}}_{I(i,j)}$ exponentially converges to zero.

When ${ϵ}_T\ne 0$, (24) can be further presented as

$$\begin{array}{cc}\hfill {\dot{V}}_I& =-|P_I{|}^2{\tilde{W}}_{I(i,j)}^2+\left|P_I\right|{\tilde{W}}_{I(i,j)}{\mathcal{V}}_{I(i,j)}\hfill \\ \hfill & =-\left[\right|P_I{|}^2{\tilde{W}}_{I(i,j)}-|P_I|{\mathcal{V}}_{I(i,j)}]{\tilde{W}}_{I(i,j)}.\hfill \end{array}$$

(26)

According to Assumption 1, $|P_I|{\mathcal{V}}_{I(i,j)}$ is bounded, denoted as $\parallel |P_I|{\mathcal{V}}_{I(i,j)}\parallel <b_{P_I{\mathcal{V}}_I}$. Then,  

$$\begin{array}{cc}\hfill {\dot{V}}_I& \le -\left[\right|P_I{|}^2\left|\right|{\tilde{W}}_{I(i,j)}\left|\right|-b_{P_I{\mathcal{V}}_I}\left]\right||{\tilde{W}}_{I(i,j)}\left|\right|.\hfill \end{array}$$

(27)

According to the extended Lyapunov theorem, the estimation error ${\tilde{W}}_{I(i,j)}$ uniformly ultimately converges to a compact set $\left\{{\tilde{W}}_{I(i,j)}\right|\left|\right|{\tilde{W}}_{I(i,j)}\left|\right|\le b_{P_I{\mathcal{V}}_I}/p_I^2\}$.    □

Remark 1.

In [12], the update law for the unknown weight $W_I$ was designed based on (18), while the PE condition (i.e., ${\theta }_I\in PE$) was required to ensure convergence. However, satisfying the PE condition is generally challenging. In Theorem 1, we provide a new convergence condition $|P_I|\in PE$. Notably, this new condition is significantly superior to the conventional PE condition for two reasons. (1) We theoretically prove that $|P_I|\in PE$ is much weaker than ${\theta }_I\in PE$, as detailed in Section 5. (2) $|P_I|$ is directly related to the determinant of the matrix $P_I\left(t\right)$. Therefore, checking $|P_I|\in PE$ online becomes feasible by calculating the determinant of $P_I\left(t\right)$. In contrast, assessing the standard PE condition directly online is not possible [18,19,23].

Based on the above analysis, the unknown information $f\left(x\right)$, $g\left(x\right)$, and $G\left(x\right)$ can be estimated using (13) and (22). This allows for the reconstruction of the completely unknown system dynamics. In order to obtain the optimal $H_{\infty }$ control pair, the critic NN will be introduced to learn the solution of the HJB equation in the subsequent section.

4. Critic NN Design for ${\mathit{H}}_{\infty }$ Control under Relaxed PE Condition

In this section, the performance index will be approximated via a critic NN to obtain the optimal $H_{\infty }$ control pair. The KREM algorithm will be continually utilized to design the update law of critic NN under the relaxed PE condition. Firstly, based on the above identifier, the system (1) can be represented as

$$\dot{x}={\widehat{W}}_f{\theta }_f\left(x\right)+{\widehat{W}}_g{\theta }_g\left(x\right)u+{\widehat{W}}_G{\theta }_G\left(x\right)d\left(t\right)+{ϵ}_I+{ϵ}_T,$$

(28)

where ${\widehat{W}}_f$, ${\widehat{W}}_g$ and ${\widehat{W}}_G$ are the estimated values of $W_f$, $W_g$ and $W_G$, respectively. ${ϵ}_I={\tilde{W}}_I{\theta }_I$ denotes the identifier error. And, the Hamiltonian (5) can be further written as

$$\begin{array}{cc}\hfill H=& V_x^T[{\widehat{W}}_f{\theta }_f\left(x\right)+{\widehat{W}}_g{\theta }_g\left(x\right)u+{\widehat{W}}_G{\theta }_G\left(x\right)d\left(t\right)+{ϵ}_I+{ϵ}_T]+x^{T}Qx+u^{T}Ru-{\kappa }^2{d}^{T}d.\hfill \end{array}$$

(29)

Then, the HJI Equation (6) becomes

$$\begin{array}{cc}\hfill 0& =\underset{u}{min}\underset{d}{max}\left[H(V_x^{★},x,u^{★},d^{★})\right]\hfill \\ & =V_x^{★T}[{\widehat{W}}_f{\theta }_f\left(x\right)+{\widehat{W}}_g{\theta }_g\left(x\right)u^{★}+{\widehat{W}}_G{\theta }_G\left(x\right)d^{★}\left(t\right)+{ϵ}_I+{ϵ}_T]+x^{T}Qx+u^{★T}R{u}^{★}-{\kappa }^2{d}^{★T}{d}^{★}.\hfill \end{array}$$

(30)

Therefore, based on (30), the $H_{\infty }$ control pair $(u^{★},d^{★})$ for the estimated system (28) can be expressed as follows:

$$\begin{array}{c}\hfill u^{★}=-\frac{1}{2}{R}^{-1}{\left[{\widehat{W}}_g{\theta }_g\right]}^T{V}_x^{★},\end{array}$$

(31)

$$\begin{array}{c}\hfill d^{★}=\frac{1}{2{\kappa }^2}{\left[{\widehat{W}}_G{\theta }_G\right]}^T{V}_x^{★}\left(x\right).\end{array}$$

(32)

Since the HJI Equation (30) is a nonlinear PDE, similar to (6), we utilize a critic NN to estimate $V^{★}\left(x\right)$ and its gradient $V_x^{★}\left(x\right)$ as follows:

$$V^{★}\left(x\right)=W_c^T{\theta }_c\left(x\right)+{ϵ}_v,$$

(33)

$$V_x^{★}\left(x\right)=\nabla {\theta }_c^T\left(x\right)W_c+\nabla {ϵ}_v,$$

(34)

where $W_c\in {\mathcal{R}}^l$ is the unknown constant weight. ${\theta }_c\left(x\right)\in {\mathcal{R}}^l$ represents the independent basis function with $\nabla {\theta }_c\left(x\right)=\partial {\theta }_c/\partial x$. l is the number of neurons. The approximation error is presented as ${ϵ}_v$ with $\nabla {ϵ}_v=\partial {ϵ}_v/\partial x$. Note that as the number of independent basis functions increases, both the approximation errors and their gradients can approach zero.

Before proceeding, the following assumption is needed.

Assumption 2.

(1) 

The ideal critic NN’s weight $W_c$ is bounded, that is, $∥W_c∥<b_{W_c}$.

(2) 

The basis functions ${\theta }_c\left(x\right)$ and its gradients $\nabla {\theta }_c\left(x\right)$ are bounded, that is, $∥{\theta }_c∥\le b_{{\theta }_c}$, $∥\nabla {\theta }_c∥\le b_{\nabla {\theta }_c}$.

(3) 

The approximator reconstruction error ${ϵ}_v$ and its gradients $\nabla {ϵ}_v$ are bounded, that is, $∥{ϵ}_v∥\le b_{{ϵ}_v}$, $∥\nabla {ϵ}_v∥\le b_{\nabla {ϵ}_v}$.

Since the ideal critic NN weights $W_c$ are unknown, take ${\widehat{W}}_c$ as the estimated value of $W_c$ and $\widehat{V}$ as the estimated value of V, where the practical critic NN is given by

$$\begin{array}{cc}\hfill & \widehat{V}\left(x\right)={\widehat{W}}_c^T{\theta }_c\left(x\right),\hfill \\ \hfill & {\widehat{V}}_x\left(x\right)=\nabla {\theta }_c^T\left(x\right){\widehat{W}}_c.\hfill \end{array}$$

(35)

The estimated $H_{\infty }$ control pair $\widehat{u}$ and $\widehat{d}$ can be obtained as

$$\begin{array}{c}\hfill \widehat{u}=-\frac{1}{2}{R}^{-1}{\left[{\widehat{W}}_g{\theta }_g\right]}^T{\widehat{V}}_x=-\frac{1}{2}{R}^{-1}{\left[{\widehat{W}}_g{\theta }_g\right]}^T\nabla {\theta }_c^T{\widehat{W}}_c,\end{array}$$

(36)

$$\begin{array}{c}\hfill \widehat{d}=\frac{1}{2{\kappa }^2}{\left[{\widehat{W}}_G{\theta }_G\right]}^T{\widehat{V}}_x=\frac{1}{2{\kappa }^2}{\left[{\widehat{W}}_G{\theta }_G\right]}^T\nabla {\theta }_c^T{\widehat{W}}_c.\end{array}$$

(37)

To online estimate the unknown weights of the critic NN using KREM technology, we aim to construct a linear equation according to (30) and (34) as

$$\begin{array}{cc}\hfill & {ϵ}_{HJI}+x^{T}Qx+{\widehat{u}}^{T}R\widehat{u}-{\kappa }^2{\widehat{d}}^T\widehat{d}+W_c^T\nabla {\theta }_c{\widehat{W}}_f{\theta }_f\left(x\right)\hfill \\ \hfill & +W_c^T\nabla {\theta }_c{\widehat{W}}_g{\theta }_g\left(x\right)\widehat{u}+W_c^T\nabla {\theta }_c{\widehat{W}}_G{\theta }_G\left(x\right)\widehat{d}=0,\hfill \end{array}$$

(38)

where ${ϵ}_{HJI}=W_c^T\nabla {\theta }_c({ϵ}_I+{ϵ}_T)+\nabla {ϵ}_v^T({\widehat{W}}_f{\theta }_f+{\widehat{W}}_g{\theta }_g\widehat{u}+{\widehat{W}}_G{\theta }_G\widehat{d}+{ϵ}_I+{ϵ}_T)$ is a bounded residual HJI equation error. Let $\Theta =\nabla {\theta }_c[{\widehat{W}}_f{\theta }_f+{\widehat{W}}_g{\theta }_g\widehat{u}+{\widehat{W}}_G{\theta }_G\widehat{d}]$ and $\Sigma =x^{T}Qx+{\widehat{u}}^{T}R\widehat{u}-{\kappa }^2{\widehat{d}}^T\widehat{d}$, where a linear equation is obtained as follows:

$$\begin{array}{c}\hfill \Sigma =-W_c^T\Theta -{ϵ}_{HJI}.\end{array}$$

(39)

Similar to the previous section, we define the filtered regressor matrix $P_c\in {\mathcal{R}}^{l\times l}$ and the vector $Q_c\in {\mathcal{R}}^l$ as follows:

$$\begin{array}{c}\hfill \left\{\begin{array}{c}{P}_c={\mathcal{H}}_c[\Theta {\Theta }^T],P_c\left(0\right)=0\hfill \\ Q_c={\mathcal{H}}_c[\Theta \Sigma ],Q_c\left(0\right)=0\hfill \end{array}\right.\end{array}$$

(40)

where

$${\mathcal{H}}_c=\frac{1}{p+l_c}\left[s\right]\left(t\right),$$

and $l_c>0$ is the forgetting factor. Then, the solution of (40) can be deduced as

$$\begin{array}{c}\hfill \left\{\begin{array}{c}{P}_c={\int }_0^t{e}^{-l_c(t-\tau )}\Theta {\Theta }^{T}d\tau ,\hfill \\ Q_c={\int }_0^t{e}^{-l_c(t-\tau )}\Theta \Sigma d\tau .\hfill \end{array}\right.\end{array}$$

(41)

From (39) and (41), an E-LRE related to $P_c$ and $Q_c$ is obtained

$$Q_c\left(t\right)=-P_c\left(t\right)W_c-v_c,$$

(42)

where $v_c={\int }_0^t{e}^{-l_c(t-\tau )}\Theta \left(\tau \right){ϵ}_{HJI}^T\left(\tau \right)d\tau$ is bounded. To estimate the unknown parameter $W_c$ in (42) under a relaxed PE condition, define the variables ${\mathcal{Q}}_c\left(t\right)\in {\mathcal{R}}^l$, ${\mathcal{P}}_c\in {\mathcal{R}}^{l\times l}$, and ${\mathcal{V}}_c\in {\mathcal{R}}^l$ as  

$$\begin{array}{c}\hfill \left\{\begin{array}{c}{\mathcal{Q}}_c=P_c^{*}{Q}_c,\hfill \\ {\mathcal{P}}_c=P_c^{*}{P}_c,\hfill \\ {\mathcal{V}}_c=P_c^{*}{v}_c.\hfill \end{array}\right.\end{array}$$

(43)

Then Equation (42) becomes

$${\mathcal{Q}}_c\left(t\right)=-{\mathcal{P}}_c\left(t\right)W_c-{\mathcal{V}}_c.$$

(44)

Note that ${\mathcal{P}}_c=\left|P_c\right|I_l$. Since ${\mathcal{P}}_c$ is a scalar matrix, a series of scalar LREs is obtained as

$${{\mathcal{Q}}_c}_{\left(i\right)}\left(t\right)=-\left|P_c\right|\left(t\right)W_{c\left(i\right)}-{\mathcal{V}}_{c\left(i\right)},i=1,\cdots ,l,$$

(45)

where ${{\mathcal{Q}}_c}_{\left(i\right)}$, $W_{c\left(i\right)}$ and ${\mathcal{V}}_{c\left(i\right)}$ indicate the $ith$ rows of ${\mathcal{Q}}_c$, $W_c$, and ${\mathcal{V}}_c$, respectively.

Driven by the parameter error based on (45), the critic unknown weight $W_{c\left(i\right)}$ is designed as

$${\dot{\widehat{W}}}_{c\left(i\right)}=-{\gamma }_2|P_c\left|\right[|P_c|{\widehat{W}}_{c\left(i\right)}+{\mathcal{Q}}_{c\left(i\right)}],$$

(46)

where ${\gamma }_2\in \mathcal{R}>0$ presents the adaptive learning gain.

The convergence condition for the proposed critic NN adaptive law is provided in Theorem 2.

Theorem 2.

For adaptive law (46) of critic NN with the regressor matrix ${\mathcal{P}}_c$ in (44); if $|P_c|\in PE$, then

(i) 

for ${ϵ}_{HJI}=0$, the estimator error ${\tilde{W}}_{c\left(i\right)}$ converges to zero exponentially;

(ii) 

for ${ϵ}_{HJI}\ne 0$, the estimator error ${\tilde{W}}_{c\left(i\right)}$ converges to a compact set around zero;

Proof. 

Defining the estimation error ${\tilde{W}}_{c\left(i\right)}={\widehat{W}}_{c\left(i\right)}-W_{c\left(i\right)}$, $i=1,\dots ,l$. The proofs presented in Theorem 1 can be extended to establish similar results in the current context. Note that the Lyapunov function $V_c$ here is chosen as $0.5{\gamma }_2^{-1}{\tilde{W}}_{c\left(i\right)}^2$.    □

Remark 2.

According to Theorem 2, a new convergence condition for the estimation error of the critic neural network weights, denoted as ${\tilde{W}}_c$, is provided. This condition does not rely on the conventional parameter estimation (PE) condition, i.e., $\Theta \in PE$. In this paper, the additional exploration signal is not required to guarantee $\Theta \in PE$. Instead, the satisfaction of $|P_c|\in PE$ can be achieved by adjusting the forgetting factor $l_c$. It is worth noting that the new convergence condition is associated with the matrix $P_c$, and it can be verified online by calculating the determinant of $P_c$. The proof of the weak PE property for the new convergence condition will be presented in the following section.

Remark 3.

The convergence analysis of ${\tilde{W}}_{I(i,j)}$ and ${\tilde{W}}_{c\left(i\right)}$ are provided in Theorem 1 and Theorem 2, respectively. In fact, we can derive the convergence of ${\tilde{W}}_I$ and ${\tilde{W}}_c$ using simple matrix operations, which will be omitted in this paper.

Till now, the identifier–critic learning-based framework for $H_{\infty }$ optimal control under the relaxed PE condition is given. For clarity, the design details of the proposed method are shown in Algorithm 1, which can be considered the pseudocode for the simulation part.

Algorithm 1 Identifier–critic learning-based $H_{\infty }$ optimal control algorithm

1: Initialization 2: Initialize system parameters: $x\left(0\right)$, Q, R and running time T; 3: Set the identifier and critic filter operators: ${\mathcal{H}}_I$ and ${\mathcal{H}}_c$; 4: Set the basis functions of identifier and critic NNs: ${\theta }_I(x,u)$ and ${\theta }_c\left(x\right)$; 5: Initialize and set the filter operator parameters: $\rho$, $l_I$, $l_c$, $x_f\left(0\right)=0$, ${\theta }_{If}\left(0\right)=0$ and ${ϵ}_{If}\left(0\right)=0$; 6: Initialize identifier NNs parameters: ${\gamma }_1>0$, ${\widehat{W}}_I^{initial}\in (0,1]$; 7: Initialize critic NNs parameters: ${\gamma }_2>0$, ${\widehat{W}}_c^{initial}\in (0,1]$; 8: Initialize the control pair by $\left(36\right)$ and $\left(37\right)$; 9: while  $t\le T$do 10:    Calculate the filter processing of the identifier NNs by $\left(14\right)$; 11:    Calculate the dynamic regressor extension (DRE) of the identifier NNs by $\left(15\right)$; 12:    Calculate the regressor “mixing” of the identifier NNs by $\left(18\right)$; 13:    Update the weight parameters of the identifier NNs ${\widehat{W}}_{I(i,j)}$ by $\left(20\right)$; $${\dot{\widehat{W}}}_{I(i,j)}=-{\gamma }_1|P_I\left|\right[|P_I|{\widehat{W}}_{I(i,j)}-{\mathcal{Q}}_{I(i,j)}];$$ 14:    Compute the approximated HJB equation by $\left(39\right)$; 15:    Calculate the dynamic regressor extension (DRE) of the critic NNs by $\left(40\right)$; 16:    Calculate the regressor “mixing” of the critic NNs by $\left(42\right)$; 17:    Update the weight parameters of the critic NNs ${\widehat{W}}_{c\left(i\right)}$ by $\left(46\right)$; $${\dot{\widehat{W}}}_{c\left(i\right)}=-{\gamma }_2|P_c\left|\right[|P_c|{\widehat{W}}_{c\left(i\right)}+{\mathcal{Q}}_{c\left(i\right)}];$$ 18:    Update the control pair by $\left(36\right)$ and $\left(37\right)$; 19:    Update the system states x by $\left(28\right)$; 20: end while

5. Stability and Convergence Analysis

In this section, we present the main results, which include the theoretical analysis of weak PE properties under new convergence conditions proposed in Theorem 1 and Theorem 2. Furthermore, we provide a stability result for the closed-loop system under the proposed online learning optimal control method.

To facilitate the analysis, the following assumption is made.

Assumption 3.

The system dynamics in (1) satisfy $∥f\left(x\right)∥\le b_f∥x∥$, $∥g\left(x\right)∥\le b_g$ and $∥G\left(x\right)∥\le b_G$, where $b_f>0$, $b_g>0$ and $b_G>0$.

5.1. Weak PE Properties of New Convergence Conditions

As shown in Theorem 1, Theorem 2 and Remark 3, the convergence of ${\tilde{W}}_I$ and ${\tilde{W}}_c$ is established without the restrictive PE condition, i.e., ${\theta }_I\in PE$ and $\Theta \in PE$. These new convergence conditions can be easily checked online, as mentioned in Remark 1 and Remark 2. Furthermore, we will analyze the superiority of the new convergence conditions compared to the conventional PE condition from a theoretical standpoint.

Theorem 3.

Consider the system (13) with the online identifier NN adaptive law (22) and critic NN adaptive law (46),

(i) 

The convergence condition of estimation error ${\tilde{W}}_I$ in Theorem 1, that is, $|P_c|\in PE$, is weaker than ${\theta }_I\in PE$ in the following precise sense

$${\theta }_I\left(t\right)\in PE\Rightarrow \left|P_c\right|\in PE,$$

(47)

$$|P_c|\in PE\nRightarrow {\theta }_I\left(t\right)\in PE;$$

(48)

(ii) 

The convergence condition of estimation error ${\tilde{W}}_c$ in Theorem 2, that is, $|P_c|\in PE$, is weaker than $\Theta \in PE$ in the following precise sense

$$\Theta \in PE\Rightarrow |P_c|\in PE,$$

(49)

$$|P_c|\in PE\nRightarrow \Theta \in PE.$$

(50)

Proof. 

For $\left(i\right)$, suppose that ${\theta }_I\left(t\right)$ in (13) is PE, indicating that ${\theta }_{If}\left(t\right)\in PE$ [25]. From Definition 1, we have

$$\begin{array}{cc}\hfill & {\int }_t^{t+\tau }{\theta }_{If}\left(r\right){\theta }_{If}^T\left(r\right)dr\ge \delta I\iff {\int }_{t-\tau }^t{\theta }_{If}\left(r\right){\theta }_{If}^T\left(r\right)dr\ge \delta I\mathrm{for}t>\tau >0.\hfill \end{array}$$

(51)

Moreover, since $e^{-{\beta }_I(t-r)}\ge e^{-{\beta }_I\tau }>0$ with $r\in [t-\tau ,t]$, the following inequality holds

$$\begin{array}{cc}\hfill & {\int }_{t-\tau }^t{e}^{-{\beta }_I(t-r)}{\theta }_{If}^T\left(r\right){\theta }_{If}\left(r\right)dr\ge {\int }_{t-\tau }^t{e}^{-{\beta }_I\tau }{\theta }_{If}^T\left(r\right){\theta }_{If}\left(r\right)dr\ge e^{-{\beta }_I\tau }\delta I.\hfill \end{array}$$

(52)

Furthermore, for $t>\tau >0$, we also have

$$\begin{array}{cc}\hfill & {\int }_0^t{e}^{-{\beta }_I(t-r)}{\theta }_{If}^T\left(r\right){\theta }_{If}\left(r\right)dr>{\int }_{t-\tau }^t{e}^{-{\beta }_I(t-r)}{\theta }_{If}^T\left(r\right){\theta }_{If}\left(r\right)dr.\hfill \end{array}$$

(53)

From (17), (52) and (53), we conclude that

$$\begin{array}{cc}\hfill P_I& ={\int }_0^t{e}^{-{\beta }_I(t-r)}{\theta }_{If}^T\left(r\right){\theta }_{If}\left(r\right)dr>e^{-{\beta }_I\tau }{\int }_{t-\tau }^t{\theta }_{If}^T\left(r\right){\theta }_{If}\left(r\right)dr\ge e^{-{\beta }_I\tau }\delta I.\hfill \end{array}$$

(54)

Hence, the matrix $P_I$ in (16) is positive definite, that is, ${\lambda }_i\left(P_I\right)>0$, $i=1,\dots ,d$. Considering that the determinant of a matrix is equal to the product of all its eigenvalues, that is, $|P_I|={\lambda }_1\left(P_I\right){\lambda }_2\left(P_I\right)\dots {\lambda }_d\left(P_I\right)$, we obtain ${\lambda }_i\left(P_I\right)>0\Rightarrow {\displaystyle \prod _{i=1}^d}{\lambda }_i\left(P_I\right)>0\Rightarrow \left|P_I\right|>0$. Thus, (47) is true.

The proof of (48) is established by the following:

$$\begin{array}{cc}\hfill |P_I|\in PE& \iff {\int }_0^t|P_I{|}^2\left(\tau \right)d\tau >0\hfill \\ \hfill & \iff {\int }_0^t\prod _{i=1}^d{\lambda }_i^2\left(P_I\right)>0\hfill \\ \hfill & \nRightarrow {\lambda }_i\left(P_I\right)>0,i=1,\dots ,d\hfill \\ \hfill & \iff P_I>0\hfill \\ \hfill & \iff P_I\in PE.\hfill \end{array}$$

(55)

For (ii), the proof process can be referred to in (i). This finishes the proof. □

5.2. Stability and Convergence Analysis

The stability result for the closed-loop system under the proposed online learning optimal control method will be presented in the following theorem.

Theorem 4.

Let Assumptions 1 and 2 hold. Considering system (1) with the identifier weight tuning law given by (22), the $H_{\infty }$ control pair are computed by (36) and (37), respectively. The critic NN weight tuning laws are updated by (46). If $|P_I|\in PE$ and $|P_c|\in PE$, then the closed-loop system, system identifier estimation error ${\tilde{W}}_I$, and critic estimation error ${\tilde{W}}_c$ are uniformly ultimately bounded (UUB). Moreover, the approximated $H_{\infty }$ control pair given by (36) and (37) are close to the optimal control pair $u^{★}$ and $d^{★}$ within a small region $b_u$ and $b_d$, that is, $\parallel \widehat{u}-u^{★}\parallel \le b_u$ and $\parallel \widehat{d}-d^{★}\parallel \le b_d$, where $b_u$ and $b_d$ are positive constants.

Proof. 

We consider the Lyapunov function as follows:

$$\begin{array}{cc}\hfill \mathcal{J}\left(t\right)& =\frac{1}{2}tr\left\{{\tilde{W}}_I^T\left(t\right){\gamma }_1^{-1}{\tilde{W}}_I\left(t\right)\right\}+\frac{1}{2}{\tilde{W}}_c^T\left(t\right){\gamma }_2^{-1}{\tilde{W}}_c\left(t\right)+{\gamma }_3{x}^{T}x+{\gamma }_4{V}^{★}\left(x\right)\hfill \\ \hfill & +{\gamma }_{5}tr\left\{{\mathcal{V}}_I^T{\mathcal{V}}_I\right\}+{\gamma }_6{\mathcal{V}}_c^T{\mathcal{V}}_c\hfill \\ \hfill & ={\mathcal{J}}_1+{\mathcal{J}}_2+{\mathcal{J}}_3+{\mathcal{J}}_4+{\mathcal{J}}_5+{\mathcal{J}}_6,\hfill \end{array}$$

(56)

where ${\gamma }_3$, ${\gamma }_4$, ${\gamma }_5$ and ${\gamma }_6$ are positive constants.

By applying matrix operations, we can obtain the following:

$$\begin{array}{cc}\hfill & {\dot{\tilde{W}}}_I=-{\gamma }_1|P_I\left|\right[|P_I|{\tilde{W}}_I-{\mathcal{V}}_I],\hfill \\ \hfill & {\dot{\tilde{W}}}_c=-{\gamma }_2|P_c\left|\right[|P_c|{\tilde{W}}_c-{\mathcal{V}}_c].\hfill \end{array}$$

(57)

According to Definition 1, $|P_I|\in PE$ and $|P_c|\in PE$ imply that ${\int }_t^{t+T}|P_I{|}^{2}dr\ge {\delta }_I>0$ and ${\int }_t^{t+T}|P_c{|}^{2}dr\ge {\delta }_c>0$. Substituting (19), (43), and using Young’s inequality $ab\le a^2\eta /2+b^2/2\eta$ with $\eta >0$, we have

$$\begin{array}{cc}\hfill {\dot{\mathcal{J}}}_1& =tr\left\{-{\tilde{W}}_I^T|P_I{|}^2{\tilde{W}}_I+\left|P_I\right|{\tilde{W}}_I^T{\mathcal{V}}_I\right\}\le -\left({\delta }_I-\frac{1}{2\eta }\right){∥{\tilde{W}}_I∥}^2+\frac{\eta }{2}{b}_{P_I^{*}}^2{∥|P_I|v_I∥}^2,\hfill \end{array}$$

(58)

$$\begin{array}{cc}\hfill {\dot{\mathcal{J}}}_2& =-{\tilde{W}}_c^T|P_c{|}^2{\tilde{W}}_c+\left|P_c\right|{\tilde{W}}_c^T{\mathcal{V}}_c\le -\left({\delta }_c-\frac{1}{2\eta }\right){∥{\tilde{W}}_c∥}^2+\frac{\eta }{2}{b}_{P_c^{*}}^2{∥|P_c|v_c∥}^2.\hfill \end{array}$$

(59)

where $∥P_I^{*}∥\le b_{P_I^{*}}$, $∥P_c^{*}∥\le b_{P_c^{*}}$.

For ${\mathcal{J}}_3$ and ${\mathcal{J}}_4$,

$$\begin{array}{cc}\hfill {\dot{\mathcal{J}}}_3+{\dot{\mathcal{J}}}_4& =2{\gamma }_3{x}^T\dot{x}+{\gamma }_4{\dot{V}}^{★}\left(x\right)\hfill \\ & =2{\gamma }_3{x}^T\left[f\left(x\right)+g\left(x\right)\widehat{u}+G\left(x\right)\widehat{d}-g\left(x\right)u^{★}+g\left(x\right)u^{★}-G\left(x\right)d^{★}+G\left(x\right)d^{★}\right]\hfill \\ & +{\gamma }_4(-x^{T}Qx-u^{★T}R{u}^{★}+{\kappa }^2{d}^{★T}{d}^{★})\hfill \\ \hfill & =2{\gamma }_3{x}^T[f\left(x\right)+g\left(x\right)(-\frac{1}{2}{R}^{-1}{\widehat{g}}^T\left(x\right)\nabla {\theta }_c^T\left(x\right){\widehat{W}}_c+\frac{1}{2}{R}^{-1}{g}^T\left(x\right)(\nabla {\theta }_c^T\left(x\right)W_c\hfill \\ \hfill & +\nabla {ϵ}_v\left)\right)+g\left(x\right)u^{★}+G\left(x\right)(\frac{1}{2{\kappa }^2}{\widehat{G}}^T\left(x\right)\nabla {\theta }_c^T\left(x\right){\widehat{W}}_c-\frac{1}{2{\kappa }^2}{G}^T\left(x\right)(\nabla {\theta }_c^T\left(x\right)W_c\hfill \\ & +\nabla {ϵ}_v\left)\right)+G\left(x\right)d^{★}]+{\gamma }_4(-x^{T}Qx-u^{★T}R{u}^{★}+{\kappa }^2{d}^{★T}{d}^{★}).\hfill \end{array}$$

(60)

Since $g^T\nabla {\theta }_c^T{W}_c-{\widehat{g}}^T\nabla {\theta }_c^T{\widehat{W}}_c=g^T\nabla {\theta }_c^T{\tilde{W}}_c+{\tilde{g}}^T\nabla {\theta }_c^T{\widehat{W}}_c$, and $-G^T\nabla {\theta }_c^T{W}_c+{\widehat{G}}^T\nabla {\theta }_c^T{\widehat{W}}_c=$$-G^T\nabla {\theta }_c^T{\tilde{W}}_c-{\tilde{G}}^T\nabla {\theta }_c^T{\widehat{W}}_c$, (60) can be rewritten as

$$\begin{array}{cc}\hfill {\dot{\mathcal{J}}}_3+{\dot{\mathcal{J}}}_4=& 2{\gamma }_3{x}^T[f\left(x\right)+g\left(x\right)(\frac{1}{2}{R}^{-1}{g}^T\left(x\right)\nabla {\theta }_c^T\left(x\right){\tilde{W}}_c+\frac{1}{2}{R}^{-1}{\tilde{g}}^T\left(x\right)\nabla {\theta }_c^T\left(x\right){\widehat{W}}_c\hfill \\ & +\frac{1}{2}{R}^{-1}{g}^T\left(x\right)\nabla {\epsilon }_v)+g\left(x\right)u^{★}+G\left(x\right)(-\frac{1}{2{\kappa }^2}{G}^T\left(x\right)\nabla {\theta }_c^T\left(x\right){\tilde{W}}_c\hfill \\ & -\frac{1}{2{\kappa }^2}{\tilde{G}}^T\left(x\right)\nabla {\theta }_c^T\left(x\right){\widehat{W}}_c-\frac{1}{2{\kappa }^2}{G}^T\left(x\right)\nabla {ϵ}_v)+G\left(x\right)d^{★}]\hfill \\ & +{\gamma }_4(-x^{T}Qx-u^{★T}R{u}^{★}+{\kappa }^2{d}^{★T}{d}^{★})\hfill \\ \hfill \le & -\left[{\gamma }_4{\lambda }_m\left(Q\right)-2{\gamma }_3{b}_f-4\eta \right]{∥x∥}^2+[\frac{1}{2\eta }{\gamma }_3^2{b}_g^2{b}_{\omega }^2{b}_{\nabla {\theta }_c}^2{\lambda }_M^2\left(R^{-1}\right)\hfill \\ & +\frac{1}{2\eta {\kappa }^4}{\gamma }_3^2{b}_G^2{b}_{\omega }^2{b}_{\nabla {\theta }_c}^2]{∥{\tilde{W}}_I∥}^2+\left[\frac{1}{2\eta }{\gamma }_3^2{b}_g^4{b}_{\nabla {\theta }_c}^2{\lambda }_M^2\left(R^{-1}\right)+\frac{1}{2\eta {\kappa }^4}{\gamma }_3^2{b}_G^4{b}_{\nabla {\theta }_c}^2\right]{∥{\tilde{W}}_c∥}^2\hfill \\ & -\left[{\gamma }_4{\lambda }_m\left(R\right)-2{\gamma }_3^2{b}_g^2/\eta \right]{∥u^{★}∥}^2+\left[\frac{1}{2\eta }{\gamma }_3^2{b}_g^4{\lambda }_M^2\left(R^{-1}\right)+\frac{{\gamma }_3^2}{{\kappa }^4}{b}_G^4\right]{∥\nabla {ϵ}_v∥}^2\hfill \\ & +\left[{\gamma }_4{\kappa }^2+2{\gamma }_3^2{b}_G^2/\eta \right]{∥d^{★}∥}^2,\hfill \end{array}$$

(61)

where $b_{\omega }=∥{\widehat{W}}_c∥$ is a bounded variable.

Recall that ${\mathcal{V}}_I=P_I^{*}{v}_I$ and ${\dot{v}}_I=-l_I{v}_I+{\theta }_{If}{ϵ}_{Tf}^T$, thus

$$\begin{array}{cc}\hfill {\dot{\mathcal{J}}}_5\le & 2{\gamma }_5{b}_{P_I^{*}}^2∥v_I^T{\dot{v}}_I∥=2{\gamma }_5{b}_{P_I^{*}}^2∥v_I^T[-l_I{v}_I+{\theta }_{If}{ϵ}_{Tf}^T]∥\hfill \\ \hfill \le & -\left(2{\gamma }_5{b}_{P_I^{*}}^2{l}_I-b_{P_I^{*}}^2\eta \right){∥v_I∥}^2+\frac{1}{\eta }{\gamma }_5^2{b}_{P_I^{*}}^2{∥{\theta }_{If}{ϵ}_{Tf}^T∥}^2.\hfill \end{array}$$

(62)

Since ${\dot{v}}_c=-l_c{v}_c+\Theta {ϵ}_{HJI}^T$. Hence, the last term of (56) can be given as

$$\begin{array}{cc}\hfill {\dot{\mathcal{J}}}_6\le & 2{\gamma }_6{b}_{P_c^{*}}^2∥v_c^T{\dot{v}}_c∥\hfill \\ \hfill =& 2{\gamma }_6{b}_{P_c^{*}}^2∥v_c^T\left[-l_c{v}_c+\Theta {ϵ}_{HJI}^T\right]∥\hfill \\ \hfill \le & -\left(2{\gamma }_6{l}_c{b}_{P_c^{*}}^2-5b_{P_c^{*}}^2\eta \right){∥v_c∥}^2+\frac{1}{\eta }{\gamma }_6^2{b}_{P_c^{*}}^2{b}_{W_c}^2{b}_{\nabla {\theta }_c}^2{∥\Theta ∥}^2{∥{ϵ}_T∥}^2\hfill \\ & +\frac{1}{\eta }{\gamma }_6^2{b}_{P_c^{*}}^2{b}_{\nabla {\theta }_c}^2{b}_{W_c}^2{∥{\theta }_I∥}^2{∥\Theta ∥}^2{∥{\tilde{W}}_I∥}^2+\frac{1}{\eta }{\gamma }_6^2{b}_{P_c^{*}}^2{b}_f^2{b}_{\nabla {ϵ}_v}^2{∥\Theta ∥}^2{∥x∥}^2\hfill \\ & +\frac{1}{4\eta }{\gamma }_6^2{b}_{P_c^{*}}^2{b}_g^2{b}_{\varpi 1}^2{b}_{\nabla {\theta }_c}^2{b}_{\omega }^2{\lambda }_M^2\left(R^{-1}\right){∥\Theta ∥}^2{∥\nabla {ϵ}_v∥}^2\hfill \\ & +\frac{1}{4\eta {\kappa }^4}{\gamma }_6^2{b}_{P_c^{*}}^2{b}_G^2{b}_{\varpi 2}^2{b}_{\nabla {\theta }_c}^2{b}_{\omega }^2{∥\Theta ∥}^2{∥\nabla {ϵ}_v∥}^2.\hfill \end{array}$$

(63)

where $b_{\varpi 1}=∥{\widehat{W}}_g{\theta }_g∥$ and $b_{\varpi 2}=∥{\widehat{W}}_G{\theta }_G∥$ are bounded variables. Consequently, we substitute (58), (59), and (61)–(63) into (56); thus, we have

$$\begin{array}{cc}\hfill \dot{\mathcal{J}}\left(t\right)& ={\dot{\mathcal{J}}}_1+{\dot{\mathcal{J}}}_2+{\dot{\mathcal{J}}}_3+{\dot{\mathcal{J}}}_4+{\dot{\mathcal{J}}}_5+{\dot{\mathcal{J}}}_6\hfill \\ \hfill & \le -({\delta }_I-\frac{1}{2\eta }-\frac{1}{2\eta }{\gamma }_3^2{b}_g^2{b}_{\omega }^2{b}_{\nabla {\theta }_c}^2{\lambda }_M^2\left(R^{-1}\right)-\frac{1}{2\eta {\kappa }^4}{\gamma }_3^2{b}_G^2{b}_{\omega }^2{b}_{\nabla {\theta }_c}^2\hfill \\ \hfill & -\frac{1}{\eta }{\gamma }_6^2{b}_{P_c^{*}}^2{b}_{\nabla {\theta }_c}^2{b}_{W_c}^2{∥{\theta }_I∥}^2{∥\Theta ∥}^2){∥{\tilde{W}}_I∥}^2-\left({\gamma }_4{\lambda }_m\left(R\right)-\frac{2}{\eta }{\gamma }_3^2{b}_g^2\right){∥u^{★}∥}^2\hfill \\ \hfill & -\left({\delta }_c-\frac{1}{2\eta }-\frac{1}{2\eta }{\gamma }_3^2{b}_g^4{b}_{\nabla {\theta }_c}^2{\lambda }_M^2\left(R^{-1}\right)-\frac{1}{2\eta {\kappa }^4}{\gamma }_3^2{b}_G^4{b}_{\nabla {\theta }_c}^2\right){∥{\tilde{W}}_c∥}^2\hfill \\ \hfill & -\left({\gamma }_4{\lambda }_m\left(Q\right)-2{\gamma }_3{b}_f-4\eta -\frac{1}{\eta }{\gamma }_6^2{b}_{P_c^{*}}^2{b}_f^2{b}_{\nabla {ϵ}_v}^2{∥\Theta ∥}^2\right){∥x∥}^2+\frac{1}{\eta }{\gamma }_5^2{b}_{P_I^{*}}^2{∥{\varphi }_{If}{ϵ}_{Tf}^T∥}^2\hfill \\ & -\left(2{\gamma }_5{b}_{P_I^{*}}^2{l}_I-b_{P_I^{*}}^2\eta -\frac{\eta }{2}{b}_{P_I^{*}}^2{\left|P_I\right|}^2\right){∥v_I∥}^2-\left(2{\gamma }_6{l}_c{b}_{P_c^{*}}^2-5b_{P_c^{*}}^2\eta -\frac{\eta }{2}{b}_{P_I^{*}}^2{\left|P_I\right|}^2\right){∥v_c∥}^2\hfill \\ & +(\frac{1}{2\eta }{\gamma }_3^2{b}_g^4{\lambda }_M^2\left(R^{-1}\right)+\frac{{\gamma }_3^2}{{\kappa }^4}{b}_G^4+\frac{1}{4\eta }{\gamma }_6^2{b}_{P_c^{*}}^2{b}_g^2{b}_{\varpi 1}^2{b}_{\nabla {\theta }_c}^2{b}_{\omega }^2{\lambda }_M^2\left(R^{-1}\right){∥\Theta ∥}^2\hfill \\ & +\frac{1}{4\eta {\kappa }^4}{\gamma }_6^2{b}_{P_c^{*}}^2{b}_G^2{b}_{\varpi 2}^2{b}_{\nabla {\theta }_c}^2{b}_{\omega }^2{∥\Theta ∥}^2){∥\nabla {ϵ}_v∥}^2+\frac{1}{\eta }{\gamma }_6^2{b}_{P_c^{*}}^2{b}_{W_c}^2{b}_{\nabla {\theta }_c}^2{∥\Theta ∥}^2{∥{ϵ}_T∥}^2.\hfill \end{array}$$

(64)

We choose the parameters ${\gamma }_3$, ${\gamma }_4$, ${\gamma }_5$, ${\gamma }_6$ and $\eta$, fulfilling the following conditions

$$\begin{array}{cc}\hfill & \eta >max\left\{\right({\kappa }^4+{\kappa }^4{\gamma }_3^2{b}_g^2{b}_{\omega }^2{b}_{\nabla {\theta }_c}^2{\lambda }_M^2\left(R^{-1}\right)+{\gamma }_3^2{b}_G^2{b}_{\omega }^2{b}_{\nabla {\theta }_c}^2+2{\kappa }^4{\gamma }_6^2{b}_{P_c^{*}}^2{b}_{\nabla {\theta }_c}^2{b}_{W_c}^2\hfill \\ & {∥{\theta }_I∥}^2{∥\Theta ∥}^2)/2{\kappa }^4{\delta }_I,\left({\kappa }^4+{\kappa }^4{\gamma }_3^2{b}_g^4{b}_{\nabla {\theta }_c}^2{\lambda }_M^2\left(R^{-1}\right)+{\kappa }^4{\gamma }_3^2{b}_G^4{b}_{\nabla {\theta }_c}^2\right)/2{\kappa }^4{\delta }_c\},\hfill \\ \hfill & {\gamma }_3<\sqrt{{\gamma }_4\eta {\lambda }_m\left(R\right)/2b_g^2},\hfill \\ \hfill & {\gamma }_4>\left\{2{\gamma }_3{b}_f-4\eta -\frac{1}{\eta }{\gamma }_6^2{b}_{P_c^{*}}^2{b}_f^2{b}_{\nabla {ϵ}_v}^2{∥\Theta ∥}^2\right\}/{\lambda }_m\left(Q\right),\hfill \\ \hfill & {\gamma }_5>(\eta +\frac{\eta }{2}|P_I{|}^2)/2l_I,\hfill \\ \hfill & {\gamma }_6>(5\eta -\frac{\eta }{2}|P_c{|}^2)/2l_c.\hfill \end{array}$$

Then, (64) can be further presented as

$$\begin{array}{cc}\hfill \dot{\mathcal{J}}\left(t\right)& \le -k_1{∥{\tilde{W}}_I∥}^2-k_2{∥{\tilde{W}}_c∥}^2-k_3{∥x∥}^2-k_4{∥v_I∥}^2-k_5{∥v_c∥}^2+b_{\gamma },\hfill \end{array}$$

(65)

where $k_1$, $k_2$, $k_3$, $k_4$, $k_5$ and $b_{\gamma }$ are positive constants

$$\begin{array}{cc}\hfill k_1=& {\delta }_I-\frac{1}{2\eta }-\frac{1}{2\eta }{\gamma }_3^2{b}_g^2{b}_{\omega }^2{b}_{\nabla {\theta }_c}^2{\lambda }_M^2\left(R^{-1}\right)-\frac{1}{2\eta {\kappa }^4}{\gamma }_3^2{b}_G^2{b}_{\omega }^2{b}_{\nabla {\theta }_c}^2-\frac{1}{\eta }{\gamma }_6^2{b}_{P_c^{*}}^2{b}_{\nabla {\theta }_c}^2{b}_{W_c}^2{∥{\theta }_I∥}^2{∥\Theta ∥}^2,\hfill \\ \hfill k_2=& {\delta }_c-\frac{1}{2\eta }-\frac{1}{2\eta }{\gamma }_3^2{b}_g^4{b}_{\nabla {\theta }_c}^2{\lambda }_M^2\left(R^{-1}\right)-\frac{1}{2\eta {\kappa }^4}{\gamma }_3^2{b}_G^4{b}_{\nabla {\theta }_c}^2,\hfill \\ \hfill k_3=& {\gamma }_4{\lambda }_m\left(Q\right)-2{\gamma }_3{b}_f-4\eta -\frac{1}{\eta }{\gamma }_6^2{b}_{P_c^{*}}^2{b}_f^2{b}_{\nabla {ϵ}_v}^2{∥\Theta ∥}^2,\hfill \\ \hfill k_4=& 2{\gamma }_5{b}_{P_I^{*}}^2{l}_I-b_{P_I^{*}}^2\eta -\frac{\eta }{2}{b}_{P_I^{*}}^2|P_I{|}^2,k_5=2{\gamma }_6{l}_c{b}_{P_c^{*}}^2-5b_{P_c^{*}}^2\eta -\frac{\eta }{2}{b}_{P_c^{*}}^2{\left|P_c\right|}^2,\hfill \\ \hfill b_{\gamma }=& (\frac{1}{2\eta }{\gamma }_3^2{b}_g^4{\lambda }_M^2\left(R^{-1}\right)+\frac{{\gamma }_3^2}{{\kappa }^4}{b}_G^4+\frac{1}{4\eta }{\gamma }_6^2{b}_{P_c^{*}}^2{b}_g^2{b}_{\varpi 1}^2{b}_{\nabla {\theta }_c}^2{b}_{\omega }^2{\lambda }_M^2\left(R^{-1}\right){∥\Theta ∥}^2+\frac{1}{\eta }{\gamma }_5^2{b}_{P_I^{*}}^2{∥{\varphi }_{If}{ϵ}_{Tf}^T∥}^2\hfill \\ \hfill +& \frac{1}{4\eta {\kappa }^4}{\gamma }_6^2{b}_{P_c^{*}}^2{b}_G^2{b}_{\varpi 2}^2{b}_{\nabla {\theta }_c}^2{b}_{\omega }^2{∥\Theta ∥}^2){∥\nabla {ϵ}_v∥}^2+\frac{1}{\eta }{\gamma }_6^2{b}_{P_c^{*}}^2{b}_{W_c}^2{b}_{\nabla {\theta }_c}^2{∥\Theta ∥}^2{∥{ϵ}_T∥}^2.\hfill \end{array}$$

Thus, $\dot{\mathcal{J}}\left(t\right)$ is negative if

$$\begin{array}{c}\hfill ∥{\tilde{W}}_I∥>\sqrt{b_{\gamma }/k_1},∥{\tilde{W}}_c∥>\sqrt{b_{\gamma }/k_2},∥x∥>\sqrt{b_{\gamma }/k_3},∥{\Xi }_I∥>\sqrt{b_{\gamma }/k_4},∥{\Xi }_c∥>\sqrt{b_{\gamma }/k_5},\end{array}$$

which implies that the NN weight estimation errors ${\tilde{W}}_I$, ${\tilde{W}}_c$ and the system state x are all UUB.

Lastly, the error between the proposed $H_{\infty }$ control pair and the ideal one are written as

$$\begin{array}{cc}\hfill \widehat{u}-u^{★}=& -\frac{1}{2}{R}^{-1}{\left[{\widehat{W}}_g{\theta }_g\left(x\right)\right]}^T\nabla {\theta }_c^T\left(x\right){\widehat{W}}_c+\frac{1}{2}{R}^{-1}{g}^T\left(\nabla {\theta }_c^T\left(x\right)W_c+\nabla {ϵ}_v\right)\hfill \\ \hfill =& \frac{1}{2}{R}^{-1}{g}^T\nabla {\theta }_c^T\left(x\right){\tilde{W}}_c+\frac{1}{2}{R}^{-1}{\left[g-{\widehat{W}}_g{\theta }_g\left(x\right)\right]}^T\nabla {\theta }_c^T\left(x\right)W_c\hfill \\ \hfill & -\frac{1}{2}{R}^{-1}{\left[g-{\widehat{W}}_g{\theta }_g\left(x\right)\right]}^T\nabla {\theta }_c^T\left(x\right){\tilde{W}}_c+\frac{1}{2}{R}^{-1}{g}^T\nabla {ϵ}_v,\hfill \end{array}$$

$$\begin{array}{cc}\hfill \widehat{d}-d^{★}=& \frac{1}{2{\kappa }^2}{\left[{\widehat{W}}_G{\theta }_G\left(x\right)\right]}^T\nabla {\theta }_c^T\left(x\right){\widehat{W}}_c-\frac{1}{2{\kappa }^2}{G}^T\left(\nabla {\theta }_c^T\left(x\right)W_c+\nabla {ϵ}_v\right)\hfill \\ \hfill =& -\frac{1}{2{\kappa }^2}{G}^T\nabla {\theta }_c^T\left(x\right){\tilde{W}}_c+\frac{1}{2{\kappa }^2}{\left[G-{\widehat{W}}_G{\theta }_G\left(x\right)\right]}^T\nabla {\theta }_c^T\left(x\right)W_c\hfill \\ \hfill & +\frac{1}{2{\kappa }^2}{\left[G-{\widehat{W}}_G{\theta }_G\left(x\right)\right]}^T\nabla {\theta }_c^T\left(x\right){\tilde{W}}_c-\frac{1}{2{\kappa }^2}{\left[{\widehat{W}}_G{\theta }_G\left(x\right)\right]}^T\nabla {ϵ}_v,\hfill \end{array}$$

which further implies the following fact

$$\begin{array}{cc}\hfill \underset{t\to +\infty }{lim}∥\widehat{u}-u^{★}∥\le & \frac{1}{2}{\lambda }_M\left(R^{-1}\right)\{b_g\left(b_{\nabla {\theta }_c}∥{\tilde{W}}_c∥+b_{\nabla {ϵ}_v}\right)+b_{\nabla {\theta }_c}{b}_{W_c}\left(∥{\tilde{W}}_I∥+∥b_g∥\right)\hfill \\ \hfill & +b_{\nabla {\theta }_c}∥{\tilde{W}}_c∥\left(∥{\tilde{W}}_I∥+∥b_g∥\right)\}\le b_u,\hfill \end{array}$$

$$\begin{array}{cc}\hfill \underset{t\to +\infty }{lim}∥\widehat{d}-d^{★}∥\le & \frac{1}{2{\kappa }^2}\{b_G\left(b_{\nabla {\theta }_c}∥{\tilde{W}}_c∥+b_{\nabla {ϵ}_v}\right)+b_{\nabla {\theta }_c}{b}_{W_c}\left(∥{\tilde{W}}_I∥+∥b_G∥\right)\hfill \\ \hfill & +b_{\nabla {\theta }_c}∥{\tilde{W}}_c∥\left(∥{\tilde{W}}_I∥+∥b_G∥\right)\}\le b_d,\hfill \end{array}$$

where $b_u>0$ and $b_d>0$ are constants determined by the identifier NN estimation error ${\tilde{W}}_I$ and the critic NN estimation error ${\tilde{W}}_c$. It proves that the approximate $H_{\infty }$ control pair can converge to a set around the optimal solution.

This completes the proof. □

6. Numerical Simulation

This section aims to verify the effectiveness of the proposed KREM-based IC learning approach for optimal robust control. We consider the following NCT system [12]

$$\begin{array}{c}\hfill \dot{x}=f\left(x\right)+g\left(x\right)u+G\left(x\right)d,\end{array}$$

(66)

where $f\left(x\right)=\left[\begin{array}{c}-x_1+x_2\hfill \\ -0.5x_1-0.5x_2(1-{(cos\left(2x_1\right)+2)}^2)\hfill \end{array}\right]$, $g\left(x\right)=\left[\begin{array}{c}0\hfill \\ cos\left(2x_1\right)+2\hfill \end{array}\right]$, $G\left(x\right)=\left[\begin{array}{c}0\hfill \\ sin\left(4x_1\right)+2\hfill \end{array}\right]$.

We choose the regressor of identifier NN as

$${\theta }_I(x,u)={[x_1,x_2,x_2(1-{(cos\left(2x_1\right)+2)}^2),ucos\left(2x_1\right),u,dsin\left(4x_1\right),d]}^T,$$

with the unknown identifier weight matrix given by

$$W_I=\left[\begin{array}{ccccccc}-1\hfill & 1& 0& 0& 0& 0& 0\\ -0.5\hfill & 0& -0.5& 1& 2& 1& 2\end{array}\right].$$

The activation function in (33) for the critic NN is selected as

$${\theta }_c(x,u)={[x_1^2,x_1{x}_2,x_2^2]}^T.$$

The ideal critic NN weights were $W_c={[0.5,0,1]}^T$.

In this numerical example, several other parameters are set as follows: the initial values of the system states are $x_1\left(0\right)=3$ and $x_2\left(0\right)=-1$. $Q=I_2$ and $R=1$. The filter coefficients are $\rho =0.001$, $l_I=0.1$, $l_c=20$, ${\gamma }_1=800$, ${\gamma }_2=200diag\{0.3,1,1\}$. It is important to note that in this simulation, there is no need to add noise to the control input $u\left(t\right)$ to ensure the PE condition. This condition is often necessary for many existing ADP-based control methods to ensure that ${\theta }_I\left(t\right)\in PE$ and $\Theta \left(t\right)\in PE$.

For comparison, we consider the Kreisselmeier’s Regressor Extension (KRE) based identifier-critic network framework [12] for the system (66). Figure 2 and Figure 3 display the convergence of the identifier NN weights and the critic NN weights, respectively, under our KREM-based optimal robust control method and the KRE-based control method [12]. As illustrated in Figure 2, the KREM-based ADP method proposed in this paper exhibits faster convergence compared to the KRE-based ADP method. Furthermore, it demonstrates element-wise monotonicity, thus preventing oscillations and peaking in the learning curve. The trajectories of the approximate control input $\widehat{u}$ and the estimated disturbance $\widehat{d}$ are presented in Figure 4 and Figure 5, respectively. By applying the optimal $H_{\infty }$ control pair, the system states are stabilized, as depicted in Figure 6.

Figure 2. Comparison of the convergence of identifier NN’s weights ${\widehat{W}}_I$: (a) KREM-based method; (b) KRE-based method in [12].

Figure 3. Comparison of the convergence of critic NN’s weights ${\widehat{W}}_c$: (a) KREM-based method; (b) KRE-based method in [12].

Figure 4. Evolution of the approximate control input $\widehat{u}$.

Figure 5. Disturbance action d.

Figure 6. Trajectories of the system states $x={[x_1,x_2]}^T$.

7. Conclusions

This paper presents a novel adaptive learning approach using neural networks (NNs) to address the problem of optimal robust control for nonlinear continuous-time systems with unknown dynamics. The approach involves employing a system identifier that utilizes NNs and parameter estimation techniques to approximate the unknown system matrices and disturbances. Additionally, a critic NNs learning structure is introduced to obtain an approximate controller that corresponds to the optimal control problem. Unlike existing identifier-critic NNs learning control methods, this approach incorporates adaptive tuning laws based on a regressor extension and mixing technique. These laws facilitate the learning of unknown parameters in the two NNs under relaxed persistence of excitation conditions. The convergence conditions of the proposed approach have been theoretically demonstrated. Finally, the effectiveness of the proposed learning control approach is validated via a simulation study.