Near-Optimal Tracking Control of Partially Unknown Discrete-Time Nonlinear Systems Based on Radial Basis Function Neural Network

Abstract

This paper proposes an optimal tracking control scheme through adaptive dynamic programming (ADP) for a class of partially unknown discrete-time (DT) nonlinear systems based on a radial basis function neural network (RBF-NN). In order to acquire the unknown system dynamics, we use two RBF-NNs; the first one is used to construct the identifier, and the other one is used to directly approximate the steady-state control input, where a novel adaptive law is proposed to update neural network weights. The optimal feedback control and the cost function are derived via feedforward neural network approximation, and a means of regulating the tracking error is proposed. The critic network and the actor network were trained online to obtain the solution of the associated Hamilton–Jacobi–Bellman (HJB) equation within the ADP framework. Simulations were carried out to verify the effectiveness of the optimal tracking control technique using the neural networks.

1. Introduction

As is widely known, nonlinear system control is an important topic of control fields, especially for discrete-time nonlinear systems, and is difficult for traditional control methods. In recent decades, many different approaches to discrete-time system control have been proposed, such as adaptive control [1], fuzzy control [2], and PID control [3]. Optimal tracking control, one of the effective methods for nonlinear systems, has many practical engineering applications [4,5,6]. Its purpose is to design a control law that not only allows the system to track the desired trajectory but also minimizes a specific performance index. It is of great theoretical significance to explore the optimal tracking optimal control of nonlinear systems. Although dynamic programming is an effective method for solving optimal control problems, there is the problem of “curse of dimensionality” when dealing with relatively complex systems [7,8]. Moreover, it is difficult to solve the HJB equation derived from the optimal control of nonlinear systems, which has no analytical solution.

On the other hand, neural network control is used as a common control method for uncertainly nonlinear systems. In 1990, Narendra et al. first proposed an artificial neural network (ANN) adaptive control method for nonlinear dynamical systems [9]. Through neural network approximation, the uncertain system can be reconstructed using input and output data. Since then, multilayer neural networks (MNNs) have been successfully applied in pattern recognition and control systems [10]. This also has led to the generation of many types of neural networks, including the RBF-NN. In [11], Poggio et al. first proved that the RBF-NN is superior in approximating functions. Studies of RBF-NNs have also shown that these neural networks have the ability to approximate any nonlinear function with a compact ensemble and arbitrary accuracy [12,13]. Compared to other ANNs, the RBF-NN does not have the complex structure of neural networks such as back propagation (BP) networks or recurrent neural networks (RNNs), and it is easier to select parameters [11,14,15]. Its good generalization ability, simple network structure, and avoidance of unnecessarily lengthy computations are advantages that make RBF-NNs attract attention [15,16]. Many research results have been published on neural network control for nonlinear systems [17,18,19].

Benefiting from neural networks and reinforcement learning (RL), the difficult problem of solving nonlinear HJB partial differential equations is solved. The ADP algorithm was proposed by Powell to approximate the solution of the HJB equation [20], which combines the theory and methods of RL, neural networks, adaptive control and optimal control. As developed, ADP has not only been considered as one of the core methods for solving the diversity of optimal control problems but also has been successfully applied to both continuous-time systems [21,22,23] and discrete-time systems [24,25,26,27,28,29,30,31] to search for solutions of the HJB equations online. Particularly, several works have attempted to solve the discrete time nonlinear optimal regulation problem using the ADP algorithm such as robust ADP [32,33,34,35], iterative/invariant ADP [36,37,38,39], off-policy RL [40,41,42] and the Q-Learning Algorithm [40,43].

In the past decades, many relevant studies have been conducted on the optimal tracking control of discrete-time nonlinear system using the ADP algorithm. However, in the existing literature on optimal tracking of nonlinear discrete-time systems, there is no RBF neural network-based ADP algorithm. In this paper, an optimal tracking control method based on RBF-NNs for discrete-time partially unknown nonlinear systems is proposed. Two RBF neural networks are used to approximate the unknown system dynamic as well as the steady-state control. After transforming the tracking problem into a regulation problem, the critic network and the actor network are used to obtain the nearly optimal feedback control, which allows the online learning process to require only current and past system data.

The contributions of article are as follows: (1) Unlike the classical technique of NN approximation, we propose a near-optimal tracking control scheme for a class of partially unknown discrete-time nonlinear systems based on RBF-NNs and prove the stability of the system. (2) Compared with [35,39], we additionally used an RBF-NN to directly approximate the steady-state controller of the unknown system. It can solve the requirement for the priori knowledge of the controlled system dynamics and reference system dynamics. Moreover, we propose a novel adaptive law to update the weight of the steady-state controller.

The paper is organized as follows. The problem statement is shown in Section 2. The design of the optimal tracking controller of the system with partially unknown nonlinear dynamics is given in Section 3, which includes the RBF-NN identifier, the RBF-NN steady-state controller, near optimal feedback controller, and stability analysis. Section 4 provides simulation results to validate the proposed control method and details the method comparison. Section 5 draws some conclusions.

2. Problem Statement

Consider the following affine nonlinear discrete-time system [31]:

$$x(k+1)=f\left[x\right(k\left)\right]+g\left[x\right(k\left)\right]u\left(k\right)$$

(1)

where $x\left(k\right)\in {\mathbb{R}}^n$ is the measurable system state and $u\left(k\right)\in {\mathbb{R}}^m$ is the control input. Assume that the nonlinear smooth function $f\left[x\right(k\left)\right]\in {\mathbb{R}}^n$ is an unknown drift function, $g\left[x\right(k\left)\right]\in {\mathbb{R}}^{n\times m}$ is a known function, and ${\parallel g\left[x\left(k\right)\right]\parallel }_F\le g_1$ where the Frobenius norm ${\parallel ·\parallel }_F$ is applied. In addition, assume that $g\left[x\right(k\left)\right]$ has a generalized inverse matrix ${g\left[x\right(k\left)\right]}^{+}\in {\mathbb{R}}^{m\times n}$ such that $g\left[x\left(k\right)\right]{g\left[x\right(k\left)\right]}^{+}=I\in {\mathbb{R}}^{n\times n}$ where I is the identity matrix. Let $x\left(0\right)$ be the initial state.

The reference trajectory is generated by the following bounded command:

$$x_d(k+1)=\phi \left(x_d\left(k\right)\right)$$

(2)

where $x_d\left(k\right)$$\in {\mathbb{R}}^n$ and $\phi \left(x_d\left(k\right)\right)\in {\mathbb{R}}^n$, and $x_d\left(k\right)$ is the reference trajectory; it need only be a stable state trajectory or asymptotically stable.

The goal of this paper is to design a controller $u\left(k\right)$ that not only ensures the state of the system (1) tracks the reference trajectory but also minimizes a cost function. For the optimal tracking control technique, the cost functions are usually considered in quadratic form [4], that is

$$\begin{array}{cc}\hfill & J(e\left(k\right),u\left(k\right))=\sum _{k=0}^{\infty }{e}^T\left(k\right)Qe\left(k\right)+u^T\left(k\right)Ru\left(k\right)\hfill \end{array}$$

(3)

where $Q\in {\mathbb{R}}^{n\times n}$ and $R\in {\mathbb{R}}^{m\times m}$ are symmetric positive definite; $e\left(k\right)=x\left(k\right)-x_d\left(k\right)$ is tracking error. For common solutions of tracking problems, the control input consists of two parts, a steady-state input $u_d$ and a feedback input $u_e$ [24]. Next, we will discuss how to obtain each part.

The steady-state controller is used to ensure perfect tracking. This perfect tracking equation is realized under the condition $x\left(k\right)=x_d\left(k\right)$. For this condition to be fulfilled, the steady-state part of the control $u_d\left(k\right)$ must exist to make $x\left(k\right)$ equivalent to $x_d\left(k\right)$. By substituting $x_d\left(k\right)$ and $u_d\left(k\right)$ into system (1), the reference state is

$$x_d(k+1)=f\left[x_d\left(k\right)\right]+g\left[x_d\left(k\right)\right]u_d\left(k\right)$$

(4)

where $x_d\left(k\right)$ and $x_d(k+1)$ are bounded to be tracked by the reference trajectory. If the system dynamics (1) are known, $u_d\left(k\right)$ is acquired by

$$u_d\left(k\right)=g{\left[x_d\left(k\right)\right]}^{+}(x_d(k+1)-f\left[x_d\left(k\right)\right])$$

(5)

where $g{\left[x_d\left(k\right)\right]}^{+}={\left(g{\left[x_d\left(k\right)\right]}^{T}g\left[x_d\left(k\right)\right]\right)}^{-1}g{\left[x_d\left(k\right)\right]}^T$ is the generalized inverse of $g\left[x_d\left(k\right)\right]$ with $g{\left[x_d\left(k\right)\right]}^{+}g\left[x_d\left(k\right)\right]=I$.

Remark 1. 

In the subsequent discussion, the RBF network can be used to identify the unknown dynamics of system (1); hence, (5) can be computed.

By using (1) and (4), the tracking error dynamics $e\left(k\right)$ are given by

$$\begin{array}{cc}\hfill e(k+1)& =f\left[x\left(k\right)\right]+g\left[x\left(k\right)\right]u\left(k\right)-x_d\left(k+1\right)\hfill \\ \hfill & =f_e\left(k\right)+g_e\left(k\right)u_e\left(k\right)\hfill \end{array}$$

(6)

where $f_e\left(k\right)=g(e\left(k\right)+x_d\left(k\right))g{\left(x_d\left(k\right)\right)}^{+}(\phi \left(x_d\left(k\right)\right)-f\left(x_d\left(k\right)\right))+f(e\left(k\right)+x_d\left(k\right))-\phi \left(x_d\left(k\right)\right)$, $u_e\left(k\right)=u\left(k\right)-u_d\left(k\right)$, and $g_e\left(k\right)=g[x_d\left(k\right)+e\left(k\right)]$. $u_e\left(k\right)$$\in {\mathbb{R}}^m$ is the feedback control input. By minimizing the cost function, it is designed to stabilize the tracking error dynamics. For $e\left(k\right)$ in the control sequence, the cost function can be expressed as the following discrete time tracking HJB equation

$$\begin{array}{cc}\hfill J_e(e\left(k\right),u_e\left(k\right))& =\sum _{k=0}^{\infty }{e}^T\left(k\right)Qe\left(k\right)+u_e^T\left(k\right)R{u}_e\left(k\right)\hfill \\ \hfill & =e^T\left(k\right)Qe\left(k\right)+u_e^T\left(k\right)R{u}_e\left(k\right)+J_e(e(k+1),u_e(k+1))\hfill \\ \hfill & =r\left(k\right)+J_e(e(k+1),u_e(k+1))\hfill \end{array}$$

(7)

where $r\left(k\right)=e^T\left(k\right)Qe\left(k\right)+u_e^T\left(k\right)R{u}_e\left(k\right)$, $J_e(e\left(k\right),u_e\left(k\right))>0$ for $\forall e\left(k\right),u_e\left(k\right)\ne 0$ and $J_e(e(k+1)$, $u_e(k+1))$ denotes the cost function at the next tracking error dynamics $e(k+1)$. The tracking error $e\left(k\right)$ is used in the study of the cost function of the optimal tracking control problem.

In general, this feedback control $u_e\left(k\right)$ is found by minimizing (7) to solve the extremum condition in the optimal control framework [4]. This result is

$$u_e^{*}\left(k\right)=-\frac{1}{2}{R}^{-1}{g}_e\left(k\right)\frac{\partial J\left(e\right(k+1\left)\right)}{\partial e(k+1)}$$

(8)

Then, the standard control input is obtained

$$u^{*}\left(k\right)=u_d\left(k\right)+u_e^{*}\left(k\right)$$

(9)

where $u_d\left(k\right)$ is obtained from (5), and $u_e^{*}\left(k\right)$ is obtained from (8).

As detailed in the subsequent discussion, in order to acquire the unknown dynamics in system (1), we used the RBF neural networks to reconstruct system dynamics. Moreover, faced with the problem of unable to find the analytical solution of (7) and the curse of dimensionality, the ADP algorithm was used to approximately solve the HJB Equation (7).

The main results of this paper are based on the following definitions and assumptions [30].

Definition 1. 

A control law $u_e$ is admissible with respect to (7) on the set Ω if $u_e$ is continuous on a compact set ${\Omega }_u\in \mathbb{R}$ for $\forall e\left(k\right)\in \Omega$, $u_e\left(0\right)=0$, and $J(e\left(0\right),u_e(·))$ is finite.

Assumption 1. 

System (1) is controllable, and the system state $x\left(k\right)=0$ is in equilibrium under control $u\left(k\right)=0$. Input control $u\left(k\right)=u\left(x\right(k\left)\right)$ satisfies $u\left(x\right(k\left)\right)=0$ for $x\left(k\right)=0$, and the cost function is a positive definite function for any $x\left(k\right)$ and $u\left(k\right)$.

Lemma 1. 

For the tracking error system (6), assume that $u_e\left(k\right)$ is an admissible control, the internal dynamics $f_e\left(k\right)$ is bounded, and

$$\begin{array}{cc}\hfill \parallel f_e{\left(k\right)\parallel }^2\le & \Gamma {\lambda }_{min}{\left(Q\right)\parallel e\left(k\right)\parallel }^2/2+(\Gamma {\lambda }_{min}\left(R\right)-2g_1^2){\parallel u_e\left(k\right)\parallel }^2/2,\hfill \end{array}$$

(10)

where ${\lambda }_{min}\left(R\right)$ is the minimum eigenvalue of R, ${\lambda }_{min}\left(Q\right)$ is the minimum eigenvalue of Q, and $\Gamma >2g_1^2/{\lambda }_{min}\left(R\right)$ is a known positive constant. Then, the tracking error system (6) is asymptotically stable.

Proof. 

We consider the following Lyapunov function,

$$V\left(k\right)=e^T\left(k\right)e\left(k\right)+\Gamma J_e\left(k\right)$$

(11)

where $J_e\left(k\right)=J_e(e\left(k\right),u_e\left(k\right))$ is defined in (7). Differencing the Lyapunov function yields

$$\Delta V\left(k\right)=e^T(k+1)e(k+1)-e^T\left(k\right)e\left(k\right)+\Gamma (J_e(k+1)-J_e\left(k\right))$$

(12)

Using (6) and (7), we can obtain

$$\begin{array}{cc}\hfill \Delta V\left(k\right)=& {(f_e\left(k\right)+g_e\left(k\right)u_e\left(k\right))}^T(f_e\left(k\right)+g_e\left(k\right)u_e\left(k\right))\hfill \\ & -e^T\left(k\right)e\left(k\right)-\Gamma (e^T\left(k\right)Qe\left(k\right)+u_e^T\left(k\right)R{u}_e\left(k\right))\hfill \end{array}$$

(13)

Using the Cauchy–Schwarz inequality yields

$$\begin{array}{c}\Delta V\left(k\right)\le 2\parallel f_e{\left(k\right)\parallel }^2-(\Gamma {\lambda }_{\min }\left(R\right)-2g_1^2)\parallel u_e{\left(k\right)\parallel }^2-\Gamma {\lambda }_{\min }{\left(Q\right)\parallel e\left(k\right)\parallel }^2-{\parallel e\left(k\right)\parallel }^2\end{array}$$

(14)

For the purpose of asymptotically stabilizing the tracking system (6), i.e., $\Delta V\left(k\right)<0$, it is necessary to satisfy the following

$$\begin{array}{cc}\hfill 2\parallel f_e{\left(k\right)\parallel }^2\le & \Gamma {\lambda }_{min}{\left(Q\right)\parallel e\left(k\right)\parallel }^2+(\Gamma {\lambda }_{min}\left(R\right)-2g_1^2){\parallel u_e\left(k\right)\parallel }^2\hfill \end{array}$$

(15)

Thus, $\Delta V\left(k\right)<0$ and the asymptotic stability of the tracking error system (6) are proved if the bound in (10) is satisfied. □

Remark 2. 

Lemma 1 shows that under the condition that the internal dynamics $f_e\left(k\right)$ is bounded to satisfy (10), there exists an admissible control $u_e\left(k\right)$ that not only stabilizes the tracking error system (6) on Ω but also guarantees that the cost function $J_e\left(k\right)$ is finite.

3. Optimal Tracking Controller Design with Partially Unknown Dynamics

In this section, firstly, we use an RBF-NN to approximate the unknown system dynamics $f\left[x\right(k\left)\right]$ and use another RBF-NN to approximate the steady-state controller $u_d\left(k\right)$. Secondly, two feedback neural networks are introduced to approximate the cost function and the optimal feedback control $u_e\left(k\right)$. Finally, the system stability is proved by selecting an appropriate Lyapunov function.

3.1. RBF-NN Identifier Design

In this subsection, in order to capture the unknown dynamics of the system (1), an RBF-NN-based identifier is proposed. Without losses of generality, this unknown dynamics is assumed to be a smooth function within a compact set. Using an RBF-NN, this unknown dynamics (1) is identified as

$$\hat{f}\left(x\left(k\right)\right)=\widehat{w_f}{\left(k\right)}^{\mathrm{T}}h\left[x\left(k\right)\right]+{\Delta }_f\left(x\right)$$

(16)

where $\widehat{w_f}\left(k\right)$ is the matrix of ideal output weights of the neural network and $h\left[x\right(k\left)\right]$ is the vector of radial basis functions, ${\Delta }_f\left(x\right)$ is the bounded approximation error, and $\left|\right|{\Delta }_f\left(x\right)\left|\right|<{\epsilon }_f$, where ${\epsilon }_f$ is a positive constant.

For any non-zero approximation error ${\Delta }_f\left(x\right)$, there exists optimal weight matrix ${w_f}^{*}$ such that

$$f\left(x\left(k\right)\right)=\hat{f}(x,w_f^{*})-{\Delta }_f\left(x\right)$$

(17)

where $w_f^{*}$ is the optimal weight of identifier, and $\hat{f}(x,w_f^{*})=w_f^{*}{\left(k\right)}^{\mathrm{T}}h\left[x\left(k\right)\right]$. The output weights are updated, and the hidden weights remain unchanged when training, so the neural network model identification error is

$$\begin{array}{cc}\hfill \tilde{f}\left(x\left(k\right)\right)& =f\left[x\left(k\right)\right]-\hat{f}\left[x\left(k\right)\right]\hfill \\ \hfill & =\hat{f}(x,{w_f}^{*})-{\Delta }_f\left[x\left(k\right)\right]-\widehat{w_f}{\left(k\right)}^{\mathrm{T}}h\left[x\left(k\right)\right]\hfill \\ & =-\widetilde{w_f}{\left(k\right)}^{\mathrm{T}}h\left[x\left(k\right)\right]-{\Delta }_f\left[x\left(k\right)\right]\hfill \end{array}$$

(18)

where $-\widetilde{w_f}\left(k\right)={w_f}^{*}\left(k\right)-\widehat{w_f}\left(k\right)$.

The error function is defined as the following

$$E(k+1)=\frac{1}{2}{\left[\tilde{f}\left(x\left(k\right)\right)\right]}^T\left[\tilde{f}\left(x\left(k\right)\right)\right]$$

(19)

Using the gradient descent method, the weights are updated by

$$\begin{array}{cc}\hfill \Delta w_{f_j}(k+1)& =-\eta \frac{\partial E}{\partial w_{f_j}}\hfill \\ \hfill & =\eta (f(x(k))-\hat{f}(x(k)))h\left[x(k)\right]\hfill \\ \hfill & =\eta (\tilde{f}(x(k)))h\left[x(k)\right]\hfill \end{array}$$

(20)

and

$$w_{f_j}\left(k\right)=w_{f_j}\left(k-1\right)+\Delta w_{f_j}\left(k\right)$$

(21)

where $\eta >0$ is the learning rate of the identifier.

Inspired by the work in [36], we must state the following assumptions before proceeding.

Assumption 2. 

The neural network identifying error is assumed to have an upper bound, namely

$${\Delta }_f{\left(x\right)}^{\mathrm{T}}{\Delta }_f\left(x\right)\le \widetilde{w_f}{\left(k\right)}^{\mathrm{T}}\widetilde{w_f}\left(k\right)h{\left[x\left(k\right)\right]}^{\mathrm{T}}h\left[x\left(k\right)\right]$$

(22)

3.2. RBF-NN Steady-State Controller Design

We use the RBF-NN to approximate the steady-state control $u_d\left(k\right)$ directly, and the inverse dynamic NN is established to approximate [16,19].

We design the steady-state control $u_d\left(k\right)$ through the approximation of the RBF-NN

$$u_d\left(k\right)={\widehat{w_d}}^T\left(k\right)h\left[x_d\left(k\right)\right]$$

(23)

where $\widehat{w_d}$ is the actual neural network weights; $h\left[x_d\left(k\right)\right]$ is the output of the hidden layers; and $u_d\left(k\right)$ is the output of the RBF-NN.

Let the ideal steady-state control $u_d^{*}\left(k\right)$ be

$$u_d^{*}\left(k\right)=w_d^{*T}h\left[x_d\left(k\right)\right]+{\epsilon }_u$$

(24)

where $w_d^{*}$ is the optimal neural network weights and ${\epsilon }_u$ is the error vector. Assuming that $x_d(k+1)$ is the reference output of the system at the point $k+1$, without considering external disturbances, the control input $u_d^{*}\left(k\right)$ satisfies

$$L[x_d\left(k\right),u_d^{*}\left(k\right)]-x_d(k+1)=0$$

(25)

where $L[x_d\left(k\right),u_d^{*}\left(k\right)]=f\left[x_d\left(k\right)\right]+g\left[x_d\left(k\right)\right]u_d^{*}\left(k\right)$.

Thus, we can define the error $e_m\left(k\right)$ of the approximation state as

$$e_m(k+1)=L[x_d\left(k\right),u_d\left(k\right)]-x_d(k+1)$$

(26)

where $L[x_d\left(k\right),u_d\left(k\right)]=f\left[x_d\left(k\right)\right]+g\left[x_d\left(k\right)\right]u_d\left(k\right)$.

(24) subtracted from (23) yields

$$\begin{array}{c}{u}_d\left(k\right)-u_d^{*}\left(k\right)={\widehat{w_d}}^T\left(k\right)h\left[x_d\left(k\right)\right]-w_d^{*T}\left(k\right)h\left[x_d\left(k\right)\right]-{\epsilon }_u\\ ={\widetilde{w_d}}^T\left(k\right)h\left[x_d\left(k\right)\right]-{\epsilon }_u\end{array}$$

(27)

where $\widetilde{w_d}\left(k\right)=\widehat{w_d}\left(k\right)-w_d^{*}\left(k\right)$ is weight approximation error.

The weights are updated by the following update law of the weights

$$\widehat{w_d}(k+1)=\widehat{w_d}\left(k\right)-\gamma [h\left(x_d\left(k\right)\right)e_m(k+1)+\sigma \widehat{w_d}\left(k\right)]$$

(28)

where $\gamma >0$ and $\sigma >0$ are the positive constant.

Assumption 3. 

Within the set ${\Omega }_{\epsilon }$, the optimal neural network weights $w^{*}$ and the approximation error are bounded.

$$\Vert w_d^{*}\Vert ⩽w_m,\left|\right|{\epsilon }_u\left|\right|⩽{\epsilon }_l$$

(29)

3.3. Near-Optimal Feedback Controller Design

In this subsection, we present an ADP algorithm based on the Bellman optimality. The goal is to find the optimal approximate feedback control law that minimizes the approximate cost function.

First, considering the HJB Equation (7) and the optimal feedback control (8), the cost function $J_e(e\left(k\right),u_e\left(k\right))$ is rewritten as $V^i\left(e\left(k\right)\right)$. The initial cost function $V^0\left(e\left(k\right)\right)=0$ may not represent the optimal value function. Then, a single control vector $u_e^0\left(k\right)$ can be solved by

$$\begin{array}{cc}\hfill V^0\left(e\left(k\right)\right)& =\underset{u_e\left(k\right)}{min}\{e^T\left(k\right)Qe\left(k\right)+u_e^T\left(k\right)R{u}_e\left(k\right)+V^0\left(e(k+1)\right)\}\hfill \\ \hfill & =e^T\left(k\right)Qe\left(k\right)+{\left(u_e^0\left(k\right)\right)}^{T}R{u}_e^0\left(k\right)\hfill \end{array}$$

(30)

Updating the control law yields

$$\begin{array}{cc}\hfill u_e^1\left(k\right)& =arg\underset{u_e\left(k\right)}{min}\{e^T\left(k\right)Qe\left(k\right)+u_e^T\left(k\right)R{u}_e\left(k\right)+V^0\left(e(k+1)\right)\}\hfill \\ & =-\frac{1}{2}{R}^{-1}{g}_e^T\left(k\right)\frac{\partial V^0\left(e(k+1)\right)}{\partial e(k+1)}\hfill \end{array}$$

(31)

Hence, for $i=1,2,\dots$, the ADP algorithm can be realized in a continuous iterative process in

$$\begin{array}{cc}\hfill V^i\left(e\left(k\right)\right)& =\underset{u_e\left(k\right)}{min}\left\{e^T\left(k\right)Qe\left(k\right)+u_e^T\left(k\right)R{u}_e\left(k\right)+V^i\left(e(k+1)\right)\right\}\hfill \\ & =e^T\left(k\right)Qe\left(k\right)+{\left(u_e^i\left(k\right)\right)}^{T}R{u}_e^i\left(k\right)+V^i\left(e(k+1)\right)\hfill \end{array}$$

(32)

and

$$\begin{array}{cc}\hfill u_e^{i+1}\left(k\right)& =arg\underset{u_e\left(k\right)}{min}\{e^T\left(k\right)Qe\left(k\right)+u_e^T\left(k\right)R{u}_e\left(k\right)+V^i\left(e(k+1)\right)\}\hfill \\ & =-\frac{1}{2}{R}^{-1}{g}_e^T\left(k\right)\frac{\partial V^i\left(e(k+1)\right)}{\partial e(k+1)}\hfill \end{array}$$

(33)

where index i represents the number of iterations of the control law and the cost function, i.e., the update count of internal neuron to update the weight parameters, while index k represents time index of state. Moreover, it is worth noting in the iterative process of the ADP algorithm that the number of iterations of the cost function and the control law increases from zero to infinity.

To begin the development of the feedback control policy, we used neural networks to construct the critic network and the actor network.

The cost function $V_i\left(e\left(k\right)\right)$ is defined as the critic network.

The output of the critic network is denoted as

$${\hat{V}}^i\left(e\left(k\right)\right)=w_{ci}^{T}z\left({\nu }_{ci}^{T}e\left(k\right)\right)+{\epsilon }_c\left(k\right)$$

(34)

where $z\left({\nu }_{ci}^{T}e\left(k\right)\right)$ is the hidden layer function, $w_{ci}$ is the hidden layer weight of the critic network, ${\nu }_{ci}$ is the input layer weight of the critic network, and ${\epsilon }_c\left(k\right)$ is the approximation error.

So, we define the prediction error of the critic network as

$$e_{ci}\left(k\right)={\hat{V}}^i\left(e\left(k\right)\right)-V^i\left(e\left(k\right)\right)$$

(35)

The error function of the critic network is defined as

$$E_{ci}\left(k\right)=\frac{1}{2}{e}_{ci}^T\left(k\right)e_{ci}\left(k\right).$$

(36)

Using the gradient descent method, the weights of the critic network are updated,

$$w_{ci}(k+1)=w_{ci}\left(k\right)-{\alpha }_c\left[\frac{\partial E_{ci}\left(k\right)}{\partial w_{ci}\left(k\right)}\right]$$

(37)

where ${\alpha }_c>0$ is the learning rate of the critic network.

The inputs of the actor network is the system error $e\left(k\right)$, and the outputs of the actor network is the optimal feedback control $u_e\left(k\right)$. The output can be formulated as

$${\hat{u}}_e^i\left(k\right)=w_{ai}^{T}z\left(v_{ai}^{T}e\left(k\right)\right)+{\epsilon }_a\left(k\right),$$

(38)

where $z\left({\nu }_{ai}^{T}e\left(k\right)\right)$ is the hidden layer function, $w_{ai}$ is the hidden layer weight of the actor network, ${\nu }_{ai}$ is the input layer weight of the actor network, and ${\epsilon }_a\left(k\right)$ is the approximation error.

Similar to the critic network, the prediction error of the actor network is defined as

$$e_{ai}\left(k\right)={\hat{u}}_e^i\left(k\right)-u_e^i\left(k\right)$$

(39)

where ${\hat{u}}_e^i\left(k\right)$ is approximation optimal feedback control, and $u_e^i\left(k\right)$ is the optimal feedback control at the iterative number i.

The error function of the actor network is defined as

$$E_{ai}\left(k\right)=\frac{1}{2}{e}_{ai}^T\left(k\right)e_{ai}\left(k\right)$$

(40)

The weights of the actor network are also updated in the same way as the critic network; we use the gradient descent method

$$w_{ai}(k+1)=w_{ai}\left(k\right)-{\beta }_a\left[\frac{\partial E_{ai}\left(k\right)}{\partial w_{ai}\left(k\right)}\right],$$

(41)

where ${\beta }_a>0$ is the learning rate of the actor network, and i is the update count of the internal neuron to update the weight parameters.

3.4. Stability Analysis

In this subsection, we give the stability proof by Lyapunov’s stability theory.

Assumption 4. 

Radial basis function $h\left(t\right)=exp\left(-\frac{\Vert x\left(t\right)-c\left(t\right){\Vert }^2}{2b^2}\right)$ of the maximum value is $h_{max}=1$, where $c\left(t\right)$ is the center point and b is the width of radial basis function. Assuming the numbers of neurons is $l\in [l_f,l_d]$ for any radial basis function $h\in [h\left[x\left(k\right)\right],h\left[x_d\left(k\right)\right]]$, then

$$\begin{array}{cc}\hfill & \left|h_i\right|⩽1,∥h∥⩽\sqrt{l}⩽l,\hfill \\ \hfill & h^{T}h={∥h∥}^2⩽l\hfill \end{array}$$

(42)

We can know the maximum value ${∥h∥}^2$ of the hidden layer with l neurons is $l\in [l_f,l_d]$, then we assume the maximum value ${∥h\left[x\right(k\left)\right]∥}^2$ of the hidden layer for the identifier $\hat{f}\left(x\left(k\right)\right)$ is $l_f$, and the maximum value ${∥h_d\left[x\left(k\right)\right]∥}^2$ of the hidden layer for the steady-state controller $u_d\left(k\right)$ is $l_d$.

Lemma 2. 

The relationship between (25) and weight approximation error (27) satisfies the following equation.

$${\widetilde{w_d}}^T\left(k\right)h\left[x_d\left(k\right)\right]=\frac{e_m(k+1)}{L_u}+{\epsilon }_u$$

(43)

where $e_m\left(k\right)$ is the error of the approximation state $x_d\left(k\right)$, $L_u=\frac{\partial L}{\partial u}{|}_{u=\xi },\xi \in [u_d^{*}\left(k\right),u_d\left(k\right)]$, $g_1⩾\left|\frac{\partial L}{\partial u}\right|>ϵ>0$, $g_1$ and ϵ are positive constants.

Proof. 

Subtracting $w_d^{*}$ from both sides of (28), we obtain

$$\widetilde{w_d}(k+1)=\widetilde{w_d}\left(k\right)-\gamma [h\left[x_d\left(k\right)\right]e_m(k+1)+\sigma \widehat{w_d}\left(k\right)]$$

(44)

Combining (25) and (27) with the mean value theorem, we can obtain

$$\begin{array}{cc}\hfill L[x_d\left(k\right),u_d\left(k\right)]& =[L\left[x_d\left(k\right)\right],u_d^{*}\left(k\right)+{\widetilde{w_d}}^T\left(k\right)h\left[x_d\left(k\right)\right]-{\epsilon }_u]\hfill \\ \hfill & =[L{x}_d\left(k\right),u_d^{*}\left(k\right)]+[{\widetilde{w_d}}^T\left(k\right)h\left[x_d\left(k\right)\right]-{\epsilon }_u]L_u\hfill \\ \hfill & =x_d(k+1)+\left[{\widetilde{w_d}}^T\left(k\right)h\left[x_d\left(k\right)\right]-{\epsilon }_u\right]L_u\hfill \end{array}$$

(45)

Further combining (45) with (26), we can obtain

$$\begin{array}{cc}\hfill e_m(k+1)& =L[x_d\left(k\right),u_d\left(k\right)]-x_d(k+1)\hfill \\ \hfill & =[{\widetilde{w_d}}^T\left(k\right)h\left[x_d\left(k\right)\right]-{\epsilon }_u]L_u\hfill \end{array}$$

(46)

After rearranging, we can obtain

$${\widetilde{w_d}}^T\left(k\right)h\left[x_d\left(k\right)\right]=\frac{e_m(k+1)}{L_u}+{\epsilon }_u$$

(47)

The proof is completed. □

Lemma 3. 

For simplicity of analysis, ${\epsilon }_u$ and $e_m(k+1)$ have an inequality relation though using Assumption 3 and Young’s inequality.

$$-2{\epsilon }_u^T{e}_m(k+1)⩽k_0\parallel {\epsilon }_u{\parallel }^2+\frac{1}{k_0}{\parallel e_m(k+1)\parallel }^2$$

(48)

where $k_0$ is a positive constant.

Theorem 1. 

For the optimal tracking problem (1)–(3), the RBF-NN identifier (16) is used to approximate $f\left(x\right(k\left)\right)$, the steady-state controller $u_d\left(k\right)$ is approximated by the RBF-NN (23), and the feedforward networks (34), (38) are used to approximate the cost function $J\left(e\right(k),u(k\left)\right)$ and the feedback controller $u_e\left(k\right)$, respectively. Assume that the parameters satisfy the following inequality

$$\begin{array}{cc}\hfill & \left(a\right)0<\eta ⩽\frac{1}{l_f}\hfill \\ \hfill & \left(b\right)0<g_1⩽k_0\hfill \\ \hfill & \left(c\right)0<(1+\sigma )l_d\gamma ⩽\frac{1}{g_1}-\frac{1}{k_0}\hfill \\ \hfill & \left(d\right)0<(l_d+\sigma )\gamma ⩽1\hfill \\ \hfill & \left(e\right)a_c\le 2/{\parallel z\left(v_{ci}^{T}e\left(k\right)\right)\parallel }^2\hfill \\ \hfill & \left(f\right){\beta }_a\le 2/{\parallel z\left({\nu }_{ai}^{T}e\left(k\right)\right)\parallel }^2\hfill \end{array}$$

(49)

where η is the learning rate of the RBF-NN identifier, σ and γ are the update parameters of the steady-state controller approximation network weights, $a_c$ is the learning rate of the actor network, ${\beta }_a$ is the learning rate of the critic network, and $z\left(v_{ci}^{T}e\left(k\right)\right)$ and $z\left({\nu }_{ai}^{T}e\left(k\right)\right)$ are hidden layer functions of the actor network and the critic network. Then, the closed loop system (6) of approximation error is asymptotically stable when the parameter estimation errors are bounded.

Proof. 

Considering the following positive definite Lyapunov function candidate

$$\begin{array}{cc}\hfill J\left(k\right)& =J_1\left(k\right)+J_2\left(k\right)+J_3\left(k\right)+J_4\left(k\right)\hfill \\ \hfill & =\frac{1}{\eta }\widetilde{w_f}{\left(k\right)}^T\widetilde{w_f}\left(k\right)+\frac{1}{g_1}{e}_m{\left(k\right)}^T{e}_m\left(k\right)+\frac{1}{\gamma }\widetilde{w_d}{\left(k\right)}^T\widetilde{w_d}\left(k\right)+w_{ci}{\left(k\right)}^T{w}_{ci}\left(k\right)+w_{ai}{\left(k\right)}^T{w}_{ai}\left(k\right)\hfill \end{array}$$

(50)

where $J_1\left(k\right)=\frac{1}{\eta }\widetilde{w_f}{\left(k\right)}^T\widetilde{w_f}\left(k\right)$, $J_2\left(k\right)=\frac{1}{g_1}{e}_m{\left(k\right)}^T{e}_m\left(k\right)+\frac{1}{\gamma }\widetilde{w_d}{\left(k\right)}^T\widetilde{w_d}\left(k\right)$, $J_3\left(k\right)=w_{ci}{\left(k\right)}^T{w}_{ci}\left(k\right)$, $J_4\left(k\right)=w_{ai}{\left(k\right)}^T{w}_{ai}\left(k\right)$.

Firstly, differencing it according to the Lyapunov function of $J_1\left(k\right)=\frac{1}{\eta }\widetilde{w_f}{\left(k\right)}^T\widetilde{w_f}\left(k\right)$ yields

$$\begin{array}{cc}\hfill \Delta J_1\left(k\right)& =J_1\left(k+1\right)-J_1\left(k\right)\hfill \\ \hfill & =\frac{1}{\eta }\widetilde{w_f}{\left(k+1\right)}^T\widetilde{w_f}\left(k+1\right)-\frac{1}{\eta }\widetilde{w_f}{\left(k\right)}^T\widetilde{w_f}\left(k\right)\hfill \\ \hfill & =\frac{1}{\eta }{[\widetilde{w_f}\left(k\right)+\eta \tilde{f}\left(x\left(k\right)\right)h[x\left(k\right)]]}^T[\widetilde{w_f}\left(k\right)+\eta \tilde{f}\left(x\left(k\right)\right)h[x\left(k\right)]]-\frac{1}{\eta }\widetilde{w_f}{\left(k\right)}^T\widetilde{w_f}\left(k\right)\hfill \\ \hfill & =\frac{1}{\eta }[\widetilde{w_f}{\left(k\right)}^T\widetilde{w_f}\left(k\right)-\frac{1}{\eta }\widetilde{w_f}{\left(k\right)}^T\widetilde{w_f}\left(k\right)+{\eta }^2[\tilde{f}{\left(x\left(k\right)\right)}^T\tilde{f}\left(x\left(k\right)\right)h{[x\left(k\right)]}^{T}h[x\left(k\right)]\hfill \\ \hfill & +2\eta \tilde{f}\left(x\left(k\right)\right)\widetilde{w_f}{\left(k\right)}^{T}h\left[x\left(k\right)\right]\hfill \\ \hfill & =\eta [{[\widetilde{w_f}{\left(k\right)}^{T}h[x\left(k\right)]+{\Delta }_f[x]]}^T[\widetilde{w_f}{\left(k\right)}^{T}h[x\left(k\right)]+{\Delta }_f[x]]h{[x\left(k\right)]}^{T}h[x\left(k\right)]]\hfill \\ \hfill & +2[\widetilde{w_f}{\left(k\right)}^{T}h[x\left(k\right)]+{\Delta }_f[x]]\widetilde{w_f}{\left(k\right)}^{T}h[x\left(k\right)]\hfill \\ \hfill & =\eta [\widetilde{w_f}{\left(k\right)}^T\widetilde{w_f}\left(k\right)h{[x\left(k\right)]}^{T}h[x\left(k\right)]+2\widetilde{w_f}{\left(k\right)}^{T}h[x\left(k\right)]{\Delta }_f[x]-2\widetilde{w_f}{\left(k\right)}^{T}h[x\left(k\right)]\hfill \\ \hfill & -2\widetilde{w_f}{\left(k\right)}^T\widetilde{w_f}\left(k\right)h{[x\left(k\right)]}^{T}h[x\left(k\right)]+{\Delta }_f{[x]}^T{\Delta }_f[x]h{[x\left(k\right)]}^{T}h[x(k)]\hfill \end{array}$$

(51)

According to the Assumption 2, Assumption 4 and (42), (51) can be carried out to obtain

$$\begin{array}{cc}\hfill \Delta J_1\left(k\right)⩽& \eta l_f^2\Vert \widetilde{w_f}{\left(k\right)\Vert }^2-2l_f\Vert \widetilde{w_f}{\left(k\right)\Vert }^2+\eta l_f^2{\Vert \widetilde{w_f}\left(k\right)\Vert }^2\hfill \\ \hfill & +2\eta l_f\widetilde{w_f}{\left(k\right)}^{T}h\left[x\left(k\right)\right]{\Delta }_f\left[x\right]-2\widetilde{w_f}{\left(k\right)}^{T}h\left[x\left(k\right)\right]{\Delta }_f\left[x\right]\hfill \\ \hfill ⩽& \Vert \widetilde{w_f}{\left(k\right)\Vert }^2(2\eta l_f^2-2l_f)+(2l_f\eta -2)\widetilde{w_f}{\left(k\right)}^{T}h\left[x\left(k\right)\right]{\Delta }_f\left[x\right]\hfill \\ \hfill ⩽& \Vert \widetilde{w_f}{\left(k\right)\Vert }^2(4l_f^2\eta -4l_f)\hfill \end{array}$$

(52)

Next, differencing according to the Lyapunov function of $J_2\left(k\right)=\frac{1}{g_1}{e}_m{\left(k\right)}^T{e}_m\left(k\right)$$+\frac{1}{\gamma }\widetilde{w_d}{\left(k\right)}^T\widetilde{w_d}\left(k\right)$ yields

$$\begin{array}{cc}\hfill \Delta J_2\left(k\right)& =J_2\left(k+1\right)-J_2\left(k\right)\hfill \\ \hfill & =\frac{1}{g_1}[e_m{\left(k+1\right)}^T{e}_m\left(k+1\right)-e_m{\left(k\right)}^T{e}_m\left(k\right)]-\frac{1}{\gamma }\widetilde{w_d}{\left(k\right)}^T\widetilde{w_d}\left(k\right)+\frac{1}{\gamma }\widetilde{w_d}{(k+1)}^T\widetilde{w_d}(k+1)\hfill \\ \hfill & =\frac{1}{\gamma }{\langle \widetilde{w_d}\left(k\right)-\gamma [h[x_d\left(k\right)]e_m(k+1)+\sigma \widehat{w_d}\left(k\right)]\rangle }^T\langle \widetilde{w_d}\left(k\right)-\gamma [h\left[x_d\left(k\right)\right]e_m(k+1)+\sigma \widehat{w_d}\left(k\right)]\rangle \hfill \\ \hfill & -\frac{1}{\gamma }\widetilde{w_d}{\left(k\right)}^T\widetilde{w_d}\left(k\right)+\frac{1}{g_1}[e_m{\left(k+1\right)}^T{e}_m\left(k+1\right)-e_m{\left(k\right)}^T{e}_m\left(k\right)]\hfill \\ \hfill & =\frac{1}{g_1}[e_m{\left(k+1\right)}^T{e}_m\left(k+1\right)-e_m{\left(k\right)}^T{e}_m\left(k\right)]-2\widetilde{w_d}{\left(k\right)}^{T}h\left[x_d\left(k\right)\right]e_m(k+1)\hfill \\ \hfill & -2\sigma \widetilde{w_d}{\left(k\right)}^T\widehat{w_d}\left(k\right)+\gamma h^T\left[x_d\left(k\right)\right]h\left[x_d\left(k\right)\right]e_m{\left(k+1\right)}^T{e}_m\left(k+1\right)+\gamma {\sigma }^2\widehat{w_d}{\left(k\right)}^T\widehat{w_d}\left(k\right)\hfill \\ & +2\gamma \sigma \widehat{w_d}{\left(k\right)}^{T}h\left[x_d\left(k\right)\right]e_m(k+1)\hfill \end{array}$$

(53)

where

$$\begin{array}{cc}\hfill 2\sigma \widetilde{w_d}{\left(k\right)}^T\widehat{w_d}\left(k\right)& =\sigma \widetilde{w_d}{\left(k\right)}^T[\overline{w_d}\left(k\right)+w_d^{*}]+\sigma {[\widehat{w_d}\left(k\right)-{\omega }_d^{*}]}^T\widehat{w_d}\left(k\right)\hfill \\ \hfill & =\sigma \Vert \widetilde{w_d}{\left(k\right)\Vert }^2+{\Vert \widehat{w_d}\left(k\right)\Vert }^2+\widetilde{w_d}{\left(k\right)}^T{w}_d^{*}-w_d^{*}\widehat{w_d}{\left(k\right)}^T\hfill \\ \hfill & =\sigma [\Vert \widetilde{w_d}\left(k\right){\Vert }^2+\Vert \widehat{w_d}\left(k\right){\Vert }^2-\Vert w_d^{*}{\Vert }^2],\hfill \\ \hfill & \\ \hfill & \gamma h^T\left[x_d\left(k\right)\right]h\left[x_d\left(k\right)\right]e_m{\left(k+1\right)}^T{e}_m\left(k+1\right)⩽\gamma l_d{\parallel e_m\left(k+1\right)\parallel }^2,\hfill \\ \hfill & \\ \hfill & 2\gamma \sigma {\widehat{w_d}}^T\left(k\right)h\left[x_d\left(k\right)\right]e_m(k+1)⩽\gamma \sigma l_d[\Vert \widehat{w_d}{\left(k\right)\Vert }^2+\parallel e_m\left(k+1\right){\parallel }^2],\hfill \\ \hfill & \\ \hfill & \gamma {\sigma }^2{\widehat{w_d}}^T\left(k\right)\widehat{w_d}\left(k\right)=\gamma {\sigma }^2{\Vert \widehat{w_d}\left(k\right)\Vert }^2\hfill \end{array}$$

(54)

Considering (26) and $g_1⩾\left|\frac{\partial L}{\partial u}\right|>ϵ>0$, we can deduce

$$\begin{array}{cc}& \frac{1}{g_1}-\frac{2}{L_u}⩽\frac{1}{g_1}-\frac{2}{g_1}=-\frac{1}{g_1}<0\hfill \end{array}$$

(55)

Recall Lemmas 2 and 3; substituting (54) into (53) yields

$$\begin{array}{cc}\hfill \Delta J_2\left(k\right)& ⩽\left[-\frac{1}{g_1}+\gamma (1+\sigma )l_d+\frac{1}{k_0}\right]\parallel e_m\left(k+1\right){\parallel }^2+\sigma (\gamma l_d+\gamma \sigma -1){\Vert \widehat{w_d}\left(k\right)\Vert }^2\hfill \\ \hfill & -\frac{1}{g_1}\parallel e_m\left(k\right){\parallel }^2-\sigma {\Vert \overline{w_d}\left(k\right)\Vert }^2+\sigma {\omega }_m^2+k_0{\epsilon }_l^2\hfill \\ \hfill & =-\left[\frac{1}{g_1}-(1+\sigma )l_d\gamma -\frac{1}{k_0}\right]\parallel e_m\left(k+1\right){\parallel }^2+\sigma [\left(l_d+\sigma \right)\gamma -1]{\Vert \widehat{w_d}\left(k\right)\Vert }^2\hfill \\ \hfill & -\frac{1}{g_1}\left[\parallel e_m\left(k\right){\parallel }^2-\beta \right]-\sigma {\Vert \widetilde{w_d}\left(k\right)\Vert }^2\hfill \end{array}$$

(56)

where $\beta =g_1(\sigma w_m^2+k_0{\epsilon }_l^2)$ is a positive constant.

Next, we consider the following Lyapunov function

$$J_3\left(k\right)+J_4\left(k\right)=w_{ci}{\left(k\right)}^T{w}_{ci}\left(k\right)+w_{ai}{\left(k\right)}^T{w}_{ai}\left(k\right).$$

(57)

Then, differencing it according to the Lyapunov function of (57) yields

$$\begin{array}{cc}\hfill \Delta J_3\left(k\right)+\Delta J_4\left(k\right)& =\{w_{ci}{(k+1)}^T{w}_{ci}(k+1)+w_{ai}{(k+1)}^T{w}_{ai}(k+1)\}\hfill \\ \hfill & -\{w_{ci}{\left(k\right)}^T{w}_{ci}\left(k\right)+w_{ai}{\left(k\right)}^T{w}_{ai}\left(k\right)\}\hfill \\ \hfill & =a_c{\parallel e_{ci}\left(k\right)\parallel }^2(-2+a_c{\parallel z\left(v_{ci}^{T}e\left(k\right)\right)\parallel }^2)\hfill \\ \hfill & +{\beta }_a\Vert e_{ai}{\left(k\right)\Vert }^2(-2+{\beta }_a\Vert z\left(v_{ai}^{T}e\left(k\right)\right){\Vert }^2).\hfill \end{array}$$

(58)

Finally, $\Delta J\left(k\right)$ is derived from (52), (56), and (58)

$$\begin{array}{cc}\hfill \Delta J\left(k\right)& =\Delta J_1\left(k\right)+\Delta J_2\left(k\right)+\Delta J_3\left(k\right)+\Delta J_4\left(k\right)\hfill \\ \hfill & ⩽4\Vert \widetilde{w_f}{\left(k\right)\Vert }^2(l_f^2\eta -l_f)-\sigma \Vert \widetilde{w_d}\left(k\right){\Vert }^2-\left[\frac{1}{g_1}-(1+\sigma )l_d\gamma -\frac{1}{k_0}\right]{\parallel e_m\left(k+1\right)\parallel }^2\hfill \\ \hfill & +\sigma [\left(l_d+\sigma \right)\gamma -1]\Vert \widehat{w_d}\left(k\right){\Vert }^2-\frac{1}{g_1}\left[\parallel e_m\left(k\right){\parallel }^2-\beta \right]+a_c{\parallel e_{ci}\left(k\right)\parallel }^2(-2+a_c{\parallel z\left(v_{ci}^{T}e\left(k\right)\right)\parallel }^2)\hfill \\ \hfill & +{\beta }_a\Vert e_{ai}{\left(k\right)\Vert }^2(-2+{\beta }_a\Vert z\left(v_{ai}^{T}e\left(k\right)\right){\Vert }^2).\hfill \end{array}$$

(59)

Based on the above analysis, when the parameters are selected to fulfill the following condition with $\parallel e_m\left(k\right){\parallel }^2⩾\beta$,

$$\begin{array}{cc}\hfill & 0<\eta ⩽\frac{1}{l_f}\hfill \\ \hfill & 0<g_1⩽k_0\hfill \\ \hfill & 0<(1+\sigma )l_d\gamma ⩽\frac{1}{g_1}-\frac{1}{k_0}\hfill \\ \hfill & 0<(l_d+\sigma )\gamma ⩽1\hfill \\ \hfill & a_c\le 2/{\parallel z\left(v_{ci}^{T}e\left(k\right)\right)\parallel }^2\hfill \\ & {\beta }_a\le 2/{\parallel z\left({\nu }_{ai}^{T}e\left(k\right)\right)\parallel }^2\hfill \end{array}$$

(60)

we can obtain $\Delta J\left(k\right)⩽0$. □

The working process of the proposed control technique is shown in Figure 1. As shown in Figure 1, with $x\left(k\right)$, $u_d\left(k\right)$ and $u_e^i\left(k\right)$, the estimated error $e(k+1)$ can be obtained by using the RBF-NN identifier and the steady-state controller. Corresponding to the steady-state controller $u_d\left(k\right)$, we can obtain the reference trajectory $x_d\left(k\right)$. Using the ADP algorithm, we can obtain nearly optimal feedback controller ${\hat{u}}_e^i\left(k\right)$. Then, the actual controller $u\left(k\right)={\hat{u}}_e^i\left(k\right)+u_d\left(k\right)$ and system dynamics $x(k+1)$ can be obtained. In addition, by using $x_d\left(k\right)$ and $x\left(k\right)$ the estimated tracking error $e\left(k\right)$ can be obtained, $e(k+1)$ can be further obtained. Finally, we can reconstruct the system dynamics to track the reference trajectory.

4. Simulation

In this section, we give the simulation results of our method and compare it with other methods [36]. A discrete-time nonlinear system is introduced to demonstrate the effectiveness of the proposed tracking control method. The case is derived from [24]. We assume that the nonlinear smooth function $f\in {\mathbb{R}}^n$ is an unknown nonlinear drift function and $g\in {\mathbb{R}}^{n\times m}$ is a known function. The corresponding $f\left[x\right(k\left)\right]$ and $g\left[x\right(k\left)\right]$ are given as

$$\begin{array}{cc}\hfill & f\left[x\left(k\right)\right]=\left[\begin{array}{c}{f}_1\left[x\left(k\right)\right]\\ f_2\left[x\left(k\right)\right]\end{array}\right]=\left[\begin{array}{c}-sin\left(0.5x_2\left(k\right)\right)x_1^2\left(k\right)\\ -cos\left(1.4x_2\left(k\right)\right)sin\left(0.9x_1\left(k\right)\right)\end{array}\right]\hfill \\ \hfill & \\ \hfill & g\left[x\left(k\right)\right]=\left[\begin{array}{cc}{\left(x_1\left(k\right)\right)}^2+1.5& 0.1\\ 0& 0.2({(x_1\left(k\right)+x_2\left(k\right))}^2+1)\end{array}\right]\hfill \end{array}$$

(61)

The reference trajectory $x_d\left(k\right)$ for the above system is defined as

$$x_d\left(k\right)=\left[\begin{array}{c}0.25sin\left({10}^{-3}k\right)\\ 0.25cos\left({10}^{-3}k\right)\end{array}\right]$$

(62)

where $u\left(k\right)\in {\mathbb{R}}^2\in {[u_1\left(k\right),u_2\left(k\right)]}^T$, and $time\left(s\right)$ of y-axis is chosen to have a $k(1,\dots ,10,000)$ multiplied by $ts=0.001$ in the simulation.

4.1. Simulation Result of the Proposed Method

In this subsection, we give the simulation result for our proposed method.

Firstly, in order to deal with the unknown dynamics, we need to use two RBF networks to obtain the RBF identifier and the RBF steady-state controller. The RBF networks have a three-layer structure with two input neurons, hidden layers have nine neurons, and output layer have two neurons. The parameters $c_i$ and $b_j$ of the radial basis functions are chosen to be $c_i=\left[\begin{array}{ccccccccc}-2& -1.5& -1.0& -0.5& 0& 0.5& 1.0& 1.5& 2\\ -2& -1.5& -1.0& -0.5& 0& 0.5& 1.0& 1.5& 2\end{array}\right]$ and $b_j=[b_1,b_2]=[2,2]$, and the initial weights $w_0$ are chosen to be random numbers between $(0,1)$. For the RBF identifier with its weights updating law (21) to update the weights $\widehat{w_f}$, the unknown function f is identified by the input/output data $x\left(k\right)$. For the RBF steady-state controller, the reference trajectory data $x_d$ and the weights updating law (28) are used to update the weights $\widehat{w_d}$ to identify the steady-state controller $u_d$. Because $g_1⩾\frac{\partial L}{\partial u}=1$, we can select $g_1=5$. According to $0<g_1⩽k_0$ of Theorem 1, we can select $k_0=10$. For the control parameters $\eta$, because hidden layers have nine neurons, $l=9$, $0<\eta ⩽\frac{1}{l}⩽\frac{1}{9}$, we select $\eta =0.1$. With control parameters $\gamma ,\sigma$, we can know $0<(1+\sigma )9\gamma ⩽\frac{1}{5}-\frac{1}{10}=\frac{1}{10}=0.10$ and $0<(9+\sigma )\gamma ⩽1$ from Theorem 1 and thus select $\gamma =0.01,\sigma =0.001$. The initial state is set as $x\left(0\right)={[0,0]}^T$. We trained the RBF networks with 10,000 steps of acquired data. Figure 2 and Figure 3 show the RBF-NN identifiers to approximate the tracking curves of the unknown dynamics $\tilde{f}\in {[{\tilde{f}}_1\left[x\left(k\right)\right],{\tilde{f}}_2\left[x\left(k\right)\right]]}^T$.

Then, based on the ADP algorithm of Bellman optimality, Equation (6) was used to obtain the tracking error e and the optimal feedback control $u_e$ to train the critic network and the actor network, respectively. Meanwhile, the obtained standard control inputs $u={\hat{u}}_e^i+u_d$ were used in system (1), which keeps on looping until the value function $V^i\left(e\left(k\right)\right)$ converge and the tracking error $e\left(k\right)$ is zero, where the performance index is selected as $Q=I$ and $R=I$, where I is the identity matrix with appropriate dimension. For the actor network and the critic network, we used the same parameter settings. The initial weights of the critic networks and actor networks are randomly chosen between $(-10,10)$. The input layer has 2 neurons, the hidden layer has 15 neurons, the output layer has 2 neurons, and the learning rate is 0.1. The hidden layer uses the function $tansig$ and the function $purelin$, and the output layer uses the function $trainlm$. Though parameter settings, we trained the actor network and the critic network with 5000 training steps to reach the given accuracy $1\times {10}^{-9}$. Figure 4 shows the curves of the system control $u\in [u_1,u_2]$. In Figure 5 and Figure 6, we can see the curves of the state trajectory x and the reference trajectory $x_d$.

Based on above the results, the simulation results show that this tracking technique obtains a relatively satisfactory tracking performance for partially unknown discrete-time nonlinear systems.

4.2. Comparison with Other Methods

In this subsection, we will compare with the research results in [36], which use a BP neural network to approximate the unknown system dynamics. In the comparison, we use the same system dynamics and desired tracking trajectory as (61) and (62) with the initial state $x\left(0\right)={[0,0]}^T$ and the performance index $R=Q=I$.

To begin with, an NN identifier is established by a three-layer BP neural network, which is chosen to have a 4–10–2 structure with four input neurons, eight hidden neurons, and two output neurons. The feedforward-neuro-controller is also established by a three-layer BP NN, which is chosen to have a 2–10–2 structure with two input neurons, eight hidden neurons, and two output neurons. For the NN identifier and the feedforward-neuro-controller, the parameter settings of the neural networks are identical, where the hidden layers use the sigmoidal function $tansig$, the output layers use the linear function $purelin$, the learning rate is 0.1, and the initial weights are chosen to be random numbers between $(0,1)$.

For the actor network and the critic network, we also use the same parameter settings. A 2–15–2 structure is chosen for the critic networks and actor networks, the initial weights are randomly chosen between $(-10,10)$, and the learning rate is 0.1. The hidden layer uses the function $tansig$ and the function $purelin$, and the output layer uses the function $trainlm$. Then, the given accuracy is $1\times {10}^{-9}$. In Figure 7 and Figure 8, we can see the curves of the state trajectory x and the reference trajectory $x_d$ using tracking control methods for references.

Comparing the two methods, from Figure 5, Figure 6, Figure 7 and Figure 8, we can see that our method has better performance in tracking the reference trajectory.

5. Conclusions

This paper proposes an effective scheme to find the near-optimal tracking controller for a class of partially unknown discrete-time nonlinear systems based on RBF-NNs. In dealing with unknown variables, two RBF-NNs are used to approximate the unknown function and the steady-state controller. Moreover, the ADP algorithm is introduced to obtain the optimal feedback control for tracking the error dynamics, two feedforward neural networks are utilized as structures to approximate the cost function and the feedback controller. Finally, simulation results show a relatively satisfactory tracking performance, which verifies the effectiveness of the optimal tracking control technique. In future work, we may consider completely unknown dynamics and event-triggering conditions.