On Model Identification Based Optimal Control and It’s Applications to Multi-Agent Learning and Control

Abstract

This paper reviews recent progress in model identification-based learning and optimal control and its applications to multi-agent systems (MASs). First, a class of learning-based optimal control method, namely adaptive dynamic programming (ADP), is introduced, and the existing results using ADP methods to solve optimal control problems are reviewed. Then, this paper investigates various kinds of model identification methods and analyzes the feasibility of combining the model identification method with the ADP method to solve optimal control of unknown systems. In addition, this paper expounds the current applications of model identification-based ADP methods in the fields of single-agent systems (SASs) and MASs. Finally, some conclusions and some future directions are presented.

1. Introduction

In recent years, with the rapid development of communication and network technology, MASs have been deeply applied in many fields, such as transportation, industrial production, etc. Facing increasingly large-scale and complex systems, the integration solutions to single-agent systems (SASs) are often limited by various resources and conditions. The MASs can effectively improve the robustness, reliability, and flexibility of large-scale complex systems [1,2].

MASs are composed of multiple agents with particular capabilities of sensing, computation, communication and control, and agents can coordinate to complete some common tasks through local interactions among agents [3,4]. Compared with traditional SASs, MASs involve relatively simple agents and thus reduce costs while improving robustness. Meanwhile, distributed coordination mechanisms exerted on multiple agents can improve the operation efficiency and reduce resource consumption. MASs have been widely used in real applications, such as resource detection, safety monitoring, natural disaster preparedness, etc. In some scenarios, agents can replace humans to guarantee the safety of military or agricultural production. In industrial applications, using multiple agents instead of single-agent can reduce production costs. Especially via coordination, such as mobile multi-unmanned aerial vehicles (Multi-UAV) systems, multi-robot systems, and multi-agent supporting systems, agents can complete more complex and challenging tasks while safety and reliability can be guaranteed [5,6,7].

The concerns in system control have gradually shifted from stabilization and stability to high steady-state accuracy, rapidity, strong robustness, and anti-interference performances. In many engineering application fields, scientists and engineers usually not only want to ensure the stability of controllable systems, but also aim to optimize certain performances (energy consumption and cost) at the same time. In this way, considering optimization is a key topic with greater practical implications for MASs. That is, a group of autonomous agents set out to complete some difficult tasks while also optimizing their performance indices.

Recently, optimization and optimal control employing a preset performance criterion have become increasingly hot research topics in the system and control fields. By interacting with an environment, an agent or decision maker develops a strategy to maximize a long-term reward using reinforcement learning (RL), a goal-oriented learning technology, which has achieved great success in the field of artificial intelligence (AI) [8,9,10]. In this context, the ADP method with strong self-learning ability has become a promising intelligent optimization technology. At present, in the field of multi-agent optimal control, most existing ADP methods are partially model-dependent or completely model-dependent. Unfortunately, model uncertainties exist in most of actual control systems, which leads to inaccurate modeling. In order to solve this problem, model identification-based ADP methods have been developed to solve MAS optimal control problems.

Motivated by the observations mentioned above, this paper aims at giving a brief survey for important developments in model identification based optimal control and its applications to multi-agent learning and control. In particular, we mainly focus on adaptive dynamic programming based optimal control method, model identification method, and the combination of ADP and model identification for dealing with the kinds of control problems of unknown system dynamics.

2. Adaptive Dynamic Programming-Based Optimal Control Method

Adaptive Dynamic Programming (ADP) is a learning-based intelligent control method with capabilities of adaption and optimization, which has great potential in solving optimal control problems. This section mainly introduces the origin of ADP, its basic structures and the development in the field of optimal control of dynamical systems, respectively.

2.1. Basic Structures of ADP

ADP, as a fusion technology of AI and control theory, is based on the traditional optimal control theory and RL principle. ADP can effectively solve a series of complex optimal control problems by learning through the continuous interactions between the agent and the environment. It is noted that there are some synonyms for ADP, such as Approximate Dynamic Programming [11], Neuro-Dynamic Programming [12], Adaptive Critic Design [13].

In the early stage, ADP was mainly used in the fields of computer science and operational research [14] and then gradually integrated with RL technology to solve optimal control problems later. Theoretically, ADP borrows from the basic principle of RL. That is, an agent interacts with the environment and constantly adjusts its strategy to achieve the optimal cumulative feedback (return) to solve an optimal decision problem. In 1977, Werbos proposed four basic ADP structures [11,15]: Heuristic Dynamic Programming (HDP), Dual Heuristic Programming (DHP), Action Dependent HDP (ADHDP), and Action Dependent DHP (ADDHP). Generally speaking, these ADP structures mainly include an actor-critic framework with the use of neural network approximation structure, which significantly improves the online learning and adaptive abilities of ADP. The basic structure of ADP is given in Figure 1. The ADP method not only avoids the “dimensional disaster” problem in dynamic programming (DP) methods, but also provides an effective way to solve the decision control problem of complex nonlinear systems, which makes it become an important research direction in the fields of artificial intelligence and control theory [9,16].

2.2. Developments of ADP-Based Optimal Control

As an important optimal control method, ADP has been widely used in the field of optimal control. Particularly, many significant scientific research achievements have been made in early theoretical studies (including algorithm and convergence) [17]. In 2002, under the ADP framework proposed by Werbos, Murray et al. [8] firstly proposed an off-line iterative algorithm of the ADP strategy to solve an optimal control problem for nonlinear systems. At the same time, the authors offered rigorous proofs for the convergence of the iterative technique and the stability of the closed-loop system with an initial admissible control. This important theoretical result laid a solid theoretical foundation for the subsequent studies related to ADP.

The early groups engaged in ADP-related research mainly included Professor Frank L. Lewis’ Team from the University of Texas at Arlington, Professor Zhongping Jiang’s Team from New York University, Professor Huaguang Zhang’s Team from Northeastern University, Professor Derong Liu’s Team, etc. They have done much pioneering research in the field of optimal control based on ADP in the early stage. Frank L. Lewis [18] designed an ADP policy iterative algorithm to solve an input-constrained control problem for nonlinear systems. In [18], they introduced a special non-quadratic performance index function for the first time and proposed a Hamilton–Jacobi–Isaac (HJI) equation simultaneously. However, the limitation of this algorithm is that the controller design depended on the complete dynamics information of the system. To overcome this limitation, Vrabie [19] proposed a partially dynamics-dependent online optimal control algorithm based on a policy iteration, namely Integral Reinforcement Learning (IRL), for nonlinear systems with partially unknown dynamics. This algorithm parametrically represents the system’s control strategy and performance using an actor-critic neural network framework, which makes the algorithm converge to the optimal control solution without requiring the system’s internal dynamics, and guarantees the stability of the closed-loop system as well. After that, in order to solve a tracking control problem for partially unknown nonlinear systems, Hamidreza Modares [20] developed an IRL-based control method. The authors proposed an augmented system containing both error states and desired states, and used the augmented system to define a new non-quadratic discount performance index function.

In recent years, in order to improve the parameter updating efficiency of the actor-critic structure, Vamvoudakis [21] proposed an online policy iteration algorithm. In this algorithm, new parameter update laws were designed for the actor and critic networks, respectively, so that the two networks can realize online updates synchronously. In addition, Zhang [22] proposed a Greedy HDP iterative algorithm to solve a tracking control problem for discrete-time nonlinear systems by introducing a new tracking error performance index function. The above research results provided an essential theoretical basis for the developments of ADP methods.

In the following, we will describe the formulation of optimal control problems for two class of nonlinear dynamical systems, that is, discrete-time system and continuous-time system, respectively.

(1) For a continuous-time nonlinear system whose dynamics are modeled as follows

$$\dot{x}\left(t\right)=f\left(x\right)+g\left(x\right)u\left(t\right),$$

(1)

where $f\left(x\right)$ and $g\left(x\right)$ are the system matrices. $x\left(t\right)=[x_1\left(t\right),x_2\left(t\right),\cdots ,x_n\left(t\right)]\in R^n$ denotes the system state, and $u\left(t\right)=[u_1\left(t\right),u_2\left(t\right),\cdots ,u_m\left(t\right)]\in R^m$ is the control input. The objective is to find an optimal controller to stabilize the system (1) as well as minimize a pre-defined performance index function, which is given by

$$V(x\left(t\right),u\left(t\right))={\int }_t^{\infty }r\left(x\right(\tau ),u(\tau \left)\right)d\tau ,$$

(2)

where $r(x\left(t\right),u\left(t\right))=x_{}^{\top }\left(t\right)Q{x}_{}\left(t\right)+u_{}^{\top }\left(t\right)Ru\left(t\right)$ represents the utility function, and Q and R are symmetric positive definite matrices with appropriate dimension. It is important to assume that the control input must be admissible such that a finite performance index function can be ensured.

The Hamiltonian of the system (1) is defined as

$$H(x\left(t\right),V_x^{}\left(t\right),u\left(t\right))=r(x\left(t\right),u\left(t\right))+V_x^T(f\left(x\left(t\right)\right)+g\left(t\right)u\left(t\right)),$$

(3)

where $V_x=\partial V/\partial x$ is a partial derivative of x.

The optimal performance index function satisfies the continuous-time HJB (CT-HJB), i.e.,

$$0=\underset{u\left(t\right)}{\min }\left\{H(x\left(t\right),V_x^{*}\left(t\right),u\left(t\right))\right\}.$$

(4)

By applying the stationarity condition, the ideal optimal control is then given by

$$u_{}^{*}\left(t\right)=-\frac{1}{2}{R}_{}^{-1}g{\left(t\right)}_{}^{\top }\frac{\partial V_{}^{*}\left(x\left(t\right)\right)}{\partial x\left(t\right)}.$$

(5)

In order to obtain the optimal controller, it is necessary to solve the CT-HJB Equation (4). However, it is very difficult to solve (4) because it contains nonlinear and partial differential items, and requires knowledge of system dynamics model $g\left(x\right)$ (that is, it needs to be known in advance). Therefore, the CT-HJB is difficult to be solved directly.

(2) For a discrete-time nonlinear system, whose dynamics is given as follows

$$x(k+1)=f\left(x\right(k),u(k\left)\right),$$

(6)

where $x\left(k\right)$ is system state, $u\left(k\right)$ is control input, and $k=0,1,2,\dots$ denotes the sampling index. The goal is to design a controller $u\left(k\right)$ to minimize the following performance index function

$$J(x\left(k\right),u\left(k\right))=\sum _{j=k}^{\infty }r\left(x\right(j),u(j\left)\right),$$

(7)

where $r\left(x\right(j),u(j\left)\right)$ denotes the utility function. By using the performance index (7), the following Bellman Equation (nonlinear Lyapunov equation) can be obtained

$$J\left(x\right(k\left)\right)=r\left(x\right(k),u(k\left)\right)+J\left(x\right(k+1),u(k+1\left)\right).$$

(8)

According to the Bellman’s principle of optimality, the optimal performance index function satisfies the following discrete-time Hamilton–Jacobi–Bellman (DT-HJB) equation

$$J^{*}\left(x\left(k\right)\right)=\underset{u\left(k\right)}{\min }\{r(x\left(k\right),u\left(k\right))+J^{*}(x(k+1),u(k+1))\}.$$

(9)

Then, we can obtain the optimal controller as

$$u^{*}\left(k\right)=\arg \underset{u\left(k\right)}{\min }\{r(x\left(k\right),u\left(k\right))+J^{*}(x(k+1),u(k+1))\}.$$

(10)

It is noted from above process that the optimal controller relies on the performance index at next time step $J^{*}(x(k+1),u(k+1))$. No matter how, the HJB equation is the key part for computing optimal control for both discrete-time and continuous-time nonlinear systems. Thus, it is important to obtain the approximate solution to the HJB equation. In the past decades, many researchers have made great efforts to propose all kinds of iterative algorithms to deal with this issue.

2.3. ADP-Based Approximate Solution to HJB Equations

In fact, most of the research results discussed above are mainly obtained for optimal control of nonlinear systems. Theoretically, the solutions to optimal control problems for nonlinear systems usually rely on Hamilton Jacobi Bellman (HJB) Equations [18]. However, it is very difficult to compute the analytical solutions to HJB equations in general, and thus numerous researches are essentially dedicated to approximate HJB equations. Till now, from the perspective of approximate solution methods, ADP-based algorithms can be divided into two categories: Value Iteration (VI) [23,24] and Policy Iteration (PI) [18,25].

Policy Iteration (PI):

Step 1: Initialization: Initial an admissible control $u^0\left(t\right)$;

Step 2: Policy evaluation: For a given iterative control strategy $u^k\left(t\right)$, the cost function can be updated according to the following rules:

$$0=\underset{u\left(k\right)}{\min }\left\{H(x\left(t\right),V_x^k\left(t\right),u^k\left(t\right))\right\};$$

Step 3: Strategy improvement: the iterative control strategy is updated as follows:

$$u^{k+1}\left(t\right)=-\frac{\alpha }{2}{R}^{-1}g{\left(t\right)}^{\top }\frac{\partial V^k\left(x\left(t\right)\right)}{\partial x\left(t\right)},$$

where k is the iterative index, the policy evaluation and policy improvement are updated alternately until the performance function and control policy converge to the optimal value. In addition, for the above PI iterative algorithm, the convergence of the algorithm has been proved.

Value Iteration (VI):

Step 1: Initialization: given an any control $u^0\left(t\right)$ and $V^0\left(t\right)$;

Step 2: Policy evaluation: the control policy can be updated according to the following rule:

$$u^k\left(t\right)=\underset{u\left(k\right)}{\min }\left\{H(x\left(t\right),V_x^k\left(t\right),u^k\left(t\right))\right\};$$

Step 3: Value improvement: the index function is updated according to the following Bellman equation:

$$V^{k=1}\left(x\left(t\right)\right)=r(x\left(t\right),u^k\left(t\right))+V^{k+1}\left(x(t+1)\right),$$

where k is the iterative index and the policy evaluation and value improvement are updated alternately until the performance function and control converge to the optimal value.

The PI algorithm starts from an initial admissible control strategy and solves a series of HJB equations to obtain the optimal control strategy. In contrast, PI has a faster convergence rate than VI. The advantage of VI algorithm is that it does not require an initial admissible control. However, the iterative control during the iterative processing may not guarantee the stability of the closed-loop system. Al-Tamimi [23] presented a VI method (also known as greedy iterative ADP algorithm) for a discrete-time system and studied its convergence and stability under the approximation optimum controller. In [25], Liu et al. proposed a PI algorithm. Compared with other ADP algorithms, this paper presented a complete convergence analysis of the proposed PI algorithm for discrete-time nonlinear systems for the first time.

In recent decades, ADP methods have been widely concerned by academia and industry because of their theoretical research and practical application values. However, most ADP methods are partially model dependent or completely model dependent [26,27], so it is difficult to deal with the situation that accurate system information cannot be obtained. In most practical cases, the system model structure of the controlled object is unknown, or the model structure is known but the model parameters are unknown. Actually, the first consideration for the unknown model in the engineering field is to identify the model. Because accurate system models can reflect the system structure information, corresponding control strategies can then be better formulated.

From another perspective, in order to address the issue of unknown system dynamics, ADP can be divided into two main types: the indirect method and the direct method. In the direct method, the optimal control law is directly designed based on the measurable system data including the state information or input/output information without system identification process [28,29,30]. The indirect technique might be a significant new trend in the development of model-free optimal control, where the reconstructed system model is firstly established by approximate approaches such as neural networks (NNs) based identifiers. Then, an ADP algorithm is introduced to design an optimal controller for the approximate model. However, Modares et al. [31] have shown that the error of model identification directly affects the convergence effect of NN weights in the ADP algorithm. Therefore, the synthesis of model identification and ADP is an important trend and also a challenging issue, which has been widely attracted in this field very recently.

3. Model Identification

From the perspective of model structure, model identification methods can be divided into parametric model identification and non-parametric model identification, which will be introduced in the following, respectively.

3.1. Parametric Model Identification Method

A parametric model identification method needs to determine the model structure and order of the system in advance, and then estimates the unknown parameters of the system model. This method mainly includes the least squares method, the gradient method, the maximum likelihood estimation method, and expectation maximization method. The overview of the parametric model identification methods is illustrated in Figure 2.

Least squares methods have formed a complete theoretical system architecture and been widely applied in many model identification problems till now. Aiming at a parameter identification problem of linear-in-parameter systems with missing data, Ding et al. [32] developed an interval-varying auxiliary model based on the recursive least squares (AM-RLS) algorithm with the help of the auxiliary model identification idea. By introducing the forgetting factors, the parameter estimation accuracy and convergence rates can be improved. For the multivariable pseudo-linear autoregressive moving average (ARMA) systems, Ding et al. [33] proposed a decomposition-based least squares iterative identification algorithm. The key in the proposed algorithm is to transform the original system to a hierarchical identification model using a data filtering technique. The model was then divided into three subsystems, with each subsystem being identified separately. The proposed approach involves less processing effort than least squares-based iterative techniques. For the identification of bilinear forms, Camelia [34] proposed a recursive least-squares for bilinear forms (RLS-BF) algorithm. Two variations of the RLS-BF algorithm based on the dichotomous coordinate descent (DCD) approach were presented to lower the computing complexity of the process. Meanwhile, a regularized version of the RLS-BF method was created to increase the resilience of the RLS-BF technique in noisy situations.

Essentially, a gradient method is an iterative algorithm. Compared with the recursive least squares, it has a slower convergence rate and lager error variance of parameter estimation. However, the computation of each step in the recursive process of gradient methods is smaller. According to the different search steps, the gradient method can be divided into the stochastic approximation method and the stochastic gradient method. There are two commonly used stochastic approximation methods, the Robbins-Monro algorithm, and the Kiefer-Wolfpwitz algorithm. However, because of the slow convergence rates of these two algorithms near the extreme points, they have not received widespread attention.

On the basis of stochastic approximation method, the stochastic gradient method adjusts the search step and accelerates the convergence rate. Recently, this method is widely used in the identification of various systems. For multivariate output-error systems, Wu [35] developed an auxiliary model based stochastic gradient (AM-SG) method and a coupled AM-SG algorithm, which ensured the parameter estimation error converged to zero under the persistence excitation (PE) condition. For the bilinear system with white noise, Ding [36] introduced a stochastic gradient (SG) technique and a gradient-based iterative approach for estimating system parameters with known input-output data using an auxiliary model. Experimental results show that the proposed gradient-based iterative algorithm has higher estimation accuracy than the auxiliary model based stochastic gradient.

In recent years, a new class of algorithms has been derived in the field of adaptive control based on the gradient method. An important concern in developing parameter identification and adaptive control schemes is transforming the original system model to a linear regressor Equation (LRE), in which the unknown parameters are linearly related to the measurable data. Then the unknown parameter estimation problem of the original system is transformed to solving the LRE, which derives a series of parameter identification methods based on the LRE of the original system.

The classical LRE can be expressed as

$$y={\varphi }^T\theta ,$$

where $y\in R$ and $\varphi \left(t\right)\in R^q$ are measurable signals. $\theta \in R^q$ is an unknown constant signal. Herein, $\varphi \left(t\right)$ is also called the regression vector. Generally, we can use the least square method [37] or the gradient method [38] to solve the unknown parameters of the original system LRE. The gradient-descent based adaptive law is designed as

$$\dot{\widehat{\theta }}\left(t\right)=\alpha \varphi \left(t\right)[y\left(t\right)-{\varphi }^T\left(t\right)\widehat{\theta }\left(t\right)],$$

where $\widehat{\theta }$ is the estimation of $\theta$, $\alpha >0$ presents adaptive learning gain. The idea of these two methods is to generate a linear time-varying (LTV) dynamic equation, known as the parameter error Equation (PEE) that can describe the estimation error, and then design the parameter estimator based on the PEE. However, the fundamental disadvantage of these techniques is that parameter estimation convergence is dependent on the PE condition of the regression vector.

Mathematically, the PE condition means that there exist some constants $\mathfrak{t}>0$ and $\Delta >0$ such that

$${\int }_t^{t+\mathfrak{t}}\varphi \left(s\right){\varphi }^T\left(s\right)ds\ge \Delta I$$

for any time t. That is, the input signal should excite all kinds of system modality so that the measurable signal contains enough information about the system, and then the convergence of parameter estimation can be guaranteed. In practice, input signals need to be designed to satisfy the PE condition. However, this is seldom practicable and difficult to verify online. Even if the input signal meets the PE criteria, the adaptive control’s parameter convergence is largely reliant on the PE intensity, which leads to a slower convergence rate.

Moreover, the transient performance of these two methods is highly unpredictable and can only guarantee weak (vector norm) monotonicity of the estimation errors. Unfortunately, poor transient estimation error performance (such as significant overshoot and slow convergence in the first few seconds) may severely degrade the estimation response, resulting in identification and adaptive control instability. Therefore, engineering applications increasingly need fast, accurate, and robust parameter estimation method to maintain the security and reliability of control systems.

To improve the parameter convergence of the gradient method, most ideas are to convert the LRE of the original system into an alternative LRE to generate a new PEE with stronger convergence properties. By introducing multiple linear filter operators to apply on the LRE of the original system, Lion [39] piled up the filtered signals to generate an extended LRE. Then, a gradient estimator based on the extended LRE was proposed. The way of developing the extended LRE is called dynamic regression extension (DRE). Compared with the classical gradient estimation method, the parameter convergence rate of the DRE-based gradient estimator can be made arbitrarily fast by increasing the adaptive gain. Kreisselmeier [40] also proposed a filter method, namely memory regressor extension (MRE), to design new LREs. Unlike DRE, Kreisselmeier only applied one linear filter operator to $\varphi \left(s\right){\varphi }^T\left(s\right)$. In fact, DRE can be transformed into MRE by rationally choosing the filter operator in the DRE algorithm. That is, MRE is a particular case of DRE. Except the advantages of the DRE-based gradient estimator, the MRE-based gradient estimator has better estimation performance than traditional gradient estimators for systems which do not satisfy the PE conditions.

To improve the transient performance of parameter identification, some researchers advocated combining the tracking error in direct adaptive control and the identification error in indirect control to form a new PEE. Then, parameter estimation algorithms based on tracking and identification errors were successively proposed [41,42,43]. Duarte et al. [41] used such an approach for model reference adaptive control (MRAC) of linear time-invariant (LTI) systems and gave the name composite adaptive control. In [42], position tracking control of robot manipulators was considered with composite adaptive control. Panteley [43] applied the composite adaptive control algorithm to the adaptive control of a class of nonlinear systems with measurable states, and relaxed a rather restrictive–detectability assumption in the stability proof. Later, Lavretsky [44] applied the work of Panteley to linear systems.

The above two types of parameter estimation frameworks lay the foundation for LRE parameter estimation. Five new adaptive control methods have gradually evolved in recent years based on these two types of original system LRE parameter estimation frameworks.

For the adaptive control of linear LTI systems, Chowdhary [45] used recorded and current data concurrently to estimate unknown parameters when designing composite adaptive law. This technique is named concurrent learning. Notably, the technique does not rely on the PE condition but guarantees the global exponential stability (GES) of the closed-loop system under an interval excitation (IE) condition. Compared with the traditional PE condition, the IE condition focuses on the evolution of integrals within an interval which is strictly weaker than the PE condition.

Cho [46] and Roy [47] designed a new composite estimator by constructing residual signals. Smilar with Chowdhary, the proposed algorithm used an “offline data selection method”. That is, the incoming data are first accumulated to build the information matrix. A composite estimator is designed by the full rank information matrix after sufficient but not persistent excitation.

In [48,49,50], a variant algorithm of MRE is proposed, which selects the filter operator as a pure integral form. Actually, this improvement leads to a positive semi-definite open-loop integral in the parameter estimator, which affects the noise sensitivity and high-gain adaptive alertness of the parameter estimator. It will make the algorithm difficult to apply in practical engineering.

Adetola [51] proposed a finite-time parameter estimation algorithm for nonlinear systems. This algorithm combines the pure integrator based MRE technique with the “initialization” process proposed in [48], and the unknown parameters of the original system can be estimated in finite time under the condition that the regression vector satisfies IE.

Aranovskiy [52] proposed a modified algorithm for DRE and named it “DRE and mixing” (DREM). The DREM algorithm adds a key mixing step to DRE and decouples vector PEEs into scalar PEEs. The scalar PEE ensures the monotonicity of each element in the parameter estimation error, which is stronger than the norm monotonicity of the traditional parameter error vector. It means the parameter estimator designed based on the scalar PEE has stronger transient stability. At the same time, the algorithm guarantees the parameter convergence and proposes a new parameter convergence condition that does not depend on the PE condition.

The least squares method and the gradient method have been developed very well, but it is difficult to address the data with missing information. Since the maximum likelihood estimation method and the expectation maximization algorithm can deal with the problem of missing information, these two algorithms have received more and more attention. The maximum likelihood estimation method proposed by Panuskal [53] is the initial probabilistic model identification method, but it did not consider the situation of missing information at that time. To deal with the parameter estimation problem in the absence of data, Dempster [54] proposed the expectation maximization algorithm. This algorithm has been used for parameter estimation of the Gaussian mixture model [55], linear variable parameter model [56], and state space model [57], and a series of expectation maximization variants algorithms have been developed.

Notably, the parametric model identification method can describe the controlled object analytically and achieve better identification results. In the development of recent decades, a fairly complete theoretical system has been formed. However, these methods are mainly for the identification of linear systems. However, most of the controlled objects often contain many complex nonlinear uncertain items in the actual system, and their model structure parameters also show time-varying characteristics, making it impossible to obtain the accurate system dynamic model. Recently, since the non-parametric models can approximate the dynamics of arbitrary complex processes in infinite dimensions, the nonparametric identification methods have begun to become the focus of scholars.

3.2. Non-Parametric Model Identification Method

The model reconstructed by the non-parametric model identification method is called a non-parametric model. It does not mean that there are no parameters in the model but that it does not need to determine the structure and order of the model in advance, which is the advantage of the non-parametric model identification method. Non-parametric model identification methods include some classic identification methods, such as correlation analysis and spectral analysis, etc. It also includes neural network (NN) models which have been developed rapidly in recent years. A neural network has been widely used in nonlinear system control because of its high nonlinearity, approximation ability, and strong self-learning ability. At present, non-parametric model identification methods mainly include: Back-Propagation (BP) neural network non-parametric model identification and Radial Basis Function (RBF) neural network non-parametric model identification. The overview of the non-parametric model identification methods is illustrated in Figure 3.

For the non-parametric model identification method using the BP neural network, since the BP neural network can approximate any nonlinear mapping relationship, and the BP algorithm belongs to the global approximation algorithm, it has better generalization ability. Generally speaking, when using a neural network to identify nonlinear systems, it is often combined with classical parameter identification methods to optimize the weights of NN.

Coban [58] proposed a new recurrent neural network, the context layered locally recurrent neural network (CLLRNN), which is effective in the identification of input-output relationships in both linear and nonlinear dynamic systems. To maximize the weights of the neural network model, Nguyen [59] proposed a hybrid modified differential evolution plus back-propagation (MDE-BP) approach. The suggested training method was evaluated in comparison to existing algorithms, including the classic DE and BP algorithms. As a result, the proposed strategy can improve the identification process’s accuracy. In [60], Aguilar proposed a fractional order neural network (FONN) for system identification by combining neural network and fractional order calculus methodologies. When compared to existing techniques, the suggested FONN model achieved higher accuracy with fewer parameters. Li [61] developed a new bilevel learning paradigm for self-organizing feed-forward neural networks (FFNN). The hybrid binary particle swarm optimization (BPSO) algorithm is used as an upper level optimizer in this interactive learning algorithm to self-organize network architecture, while the Levenberg–Marquardt (LM) algorithm is used as a lower level optimizer to optimize the connection weights of an FFNN. When compared to conventional learning algorithms, experimental results show that the bilevel learning algorithm produces much more compact FFNNs with superior generalization capabilities. Singh [62] developed a gradient evolution-based counter propagation network (GE-CPN) for approximating the noncanonical form of a nonlinear system. Learning from nonlinear systems with parametric uncertainty is a key characteristic of GE-CPN networks. Furthermore, this demonstrated that reparameterization of neural network models is required and beneficial for approximation of noncanonical systems.

As a feedforward network, RBF neural network has attracted extensive attention recently because of its fixed basis function and linear parameter network structure, which can approximate any continuous function with arbitrary precision. For the identification and modeling of nonlinear dynamic systems, Qiao [63] designed a novel self-organizing radial basis function (SORBF) neural network. Based on the neuron activity and mutual information (MI), the SORBF neural network’s hidden neurons can be added or removed to reach the desired network complexity while maintaining overall computing efficiency for identification and modeling. Meanwhile, parameter adjustment can considerably increase model performance. Slimani [64] utilized the descent gradient and the genetic algorithm technique to developed an optimization technique of neural networks radial basis function multi-model identification of nonlinear system. Errachdi [65] developed a no-preprocessing online radial basis function (RBF) neural network technique. The suggested online RBF neural network approach is then combined with a kernel principal component analysis (KPCA), which made RBF neural network training efficient and fast by reducing memory requirements of the models. In [66], with the use of adaptive particle swarm optimization, a self-organizing radial basis function (SORBF) neural network was constructed to increase both accuracy and parsimony (APSO). The presented APSO-SORBF neural network is capable of producing a network model with a compact structure and outstanding accuracy. In [67], to self-organize the structure and parameters of the RBFNN, a distance concentration immune algorithm (DCIA) was devised. A sample with the most frequent occurrence of maximum error was constructed to govern the parameters of the new neuron in order to increase forecasting accuracy and reduce computation time.

The above studies have introduced many mature identification algorithms from linear system identification into the RBF network framework. At the same time, many scholars have extended the RBF network framework to solve the problem of parameter model identification, which makes up for the limitations of traditional parametric model identification methods for nonlinear system identification. When the structure and order of the system model are known, even if the controlled object contains many complex nonlinear uncertain terms, or its model parameters show time-varying characteristics, the RBF neural network framework can identify it accurately. This not only makes full use of the available system information but also maximizes the accurate feature description of the original system.

4. Model Identification-Based Optimal Control for SAS

In the optimal control community, researchers are trying to introduce the model identification methods to the classical optimal control for a single agent system (SAS) with an unknown system model.

Bhasin [68] proposed an actor-critic-identifier (ACI) ADP framework, in which an identifier NN is utilized to approximate the unknown system information and then embedded into the actor-critic NN architectures. The ACI ADP framework is shown in Figure 4. However, the input system dynamics are still assumed to be known. To further remove this assumption, Modares [31] designed an new ACI algorithm for the unknown constrained-input nonlinear systems. The proposed ACI algorithm contains an identifier NN with an experience replay technique (ERT) to fully approximate the unknown system information (including system dynamics and input dynamics). Then, a gradient method was used to estimate the weights of critic-actor NNs. Actually, the idea of ERT is very similar to concurrent learning, both of which use recorded historical data and current data to estimate the unknown information of the system. Although this technique can relax the PE condition for parameter convergence during the online learning, it requires more computation time and computer memory to store historical data. The algorithm was then generalized to solve many control problems, such as the IRL algorithm for constrained input systems [69], the $H_{\infty }$ tracking control problem [70], and so on.

To relax the PE condition for parameter convergence during the online learning, Zhao [71] used the ERT to estimate the unknown weights of the identifier NN and critic NN simultaneously, so that the conventional PE condition could be relaxed to a simplified condition on recorded data. However, the proposed algorithm also has the same drawbacks in [31]. Based on the ERT, Yang [72] proposed an event-triggered robust control policy for unknown continuous-time nonlinear systems. To improve the convergence rate of the ERT, a data-based feedback relearning (FR) algorithm for uncertain nonlinear systems with control channel disturbances and actuator faults was developed [73]. Furthermore, a data processing method based on experience replay technology is designed to improve data utilization efficiency and algorithm convergence. To achieve model-free fault compensation, a neural network (NN)-based fault observer is used. To reduce the difficulty of designing NNs for an unknown nonlinear system and improve generalization, the polynomial activation function is redesigned using the sigmoid function/hyperbolic tangent activation function.

To avoid excessive use of NNs and achieve faster convergence, Lv [74] proposed a new identifier-critic (IC) ADP structure with the MRE method. Since the algorithm did not use an actor NN, and it did not need to record historical data, the convergence rate is greatly improved. Later, this IC algorithm was used to solve a series of other control problems [75,76,77].

5. Model Identification-Based Optimal Control for MASs

More recently, few works on the model identification-based optimal control has been studied for MASs. Based on the work of Modares [31], Tatari [78] proposed an online optimal distributed learning algorithm to find the game theoretic solution of systems on graphs with completely unknown dynamics. In [79], Tatari introduced an online distributed optimal adaptive algorithm for continuous-time nonlinear differential graphical games with unknown systems subject to external disturbances. Shi [80] utilized the MRE filtering technique and designed an adaptive disturbance observer for a class of nonlinear systems with unknown disturbances where the disturbance is assumed to be generated by some unknown dynamics. Tan [81] proposed a novel event-triggered, model-free structure to address the optimal consensus control problem for MASs with unknown dynamics and input constraints.

In the following, as an example, we give the model identification-based optimal control of MASs with unknown dynamics.

Algebraic graph theory: The communication topology between agents in a MAS is described by a directed graph $\mathcal{G}=(\mathcal{V},\mathcal{E},\mathcal{A})$ where $\mathcal{V}=\{1,2,\cdots ,N\}$ is a nonempty set of vertices and $\mathcal{E}=\left\{\right(i,j\left)\right|i,j\in \mathcal{V}\}\subseteq \mathcal{V}\times \mathcal{V}$ is the set of edges. Define $\mathcal{A}=\left\{e_{ij}\right\}\in {\mathbb{R}}^{N\times N}$ as a weighted adjacency matrix, where $e_{ij}=1$ if and only if $(i,j)\in \mathcal{E}$, and $e_{ij}=0$, otherwise. The neighbor set of the agent i is denoted by ${\mathcal{N}}_i=\left\{j\right|(i,j)\in \mathcal{E}\}$. Define a diagonal matrix $\mathcal{D}=diag\left\{d_i\right\}$ as the in-degree matrix, where $d_i={\sum }_{j\in {\mathcal{N}}_i}{e}_{ij}$. The Laplacian matrix $\mathcal{L}$ is defined by $\mathcal{L}=\mathcal{D}-\mathcal{A}$.

In order to take a single leader into account, we introduce an augmented graph $\overline{\mathcal{G}}=(\overline{\mathcal{V}},\overline{\mathcal{E}})$, where $\overline{\mathcal{V}}=\{0,1,\cdots ,N\}$ and $\overline{\mathcal{E}}\subseteq \overline{V}\times \overline{V}$. A nonnegative number $e_{i0}$ is used to describe the interaction relationship between the leader and agent i. Specifically, $e_{i0}>0$ if agent i can receive information from the leader; otherwise, $e_{i0}=0$. A leader adjacency matrix $\mathcal{B}$ is defined by $\mathcal{B}=\mathrm{diag}(e_{10},\cdots ,e_{N0})\in {\mathbb{R}}^{N\times N}$.

Assumption 1. 

The communication interaction network $\overline{\mathcal{G}}$ has a spanning tree with the root vertex 0.

Problem formulation: Consider heterogeneous MASs described by a linear time-invariant system as follows

$${\dot{x}}_i\left(t\right)=A_i{x}_i\left(t\right)+B_i{u}_i\left(t\right),i=1,2,\dots ,N,$$

(11)

where $x_i\left(t\right)\in {\mathbb{R}}^n$ and $u_i\left(t\right)\in {\mathbb{R}}^m$ are the state vector and the control input vector, respectively. The system matrices $A_i\in {\mathbb{R}}^{n\times n}$ and input matrices $B_i\in {\mathbb{R}}^{n\times m}$ are assumed be unknown in this paper. Furthermore, we assume that the pairs $(A_i,B_i)(\forall i=1,\dots ,N)$ are controllable, and the state and the control input of each agent are available.

The dynamics of the leader agent is described by

$${\dot{x}}_0=A_0{x}_0,$$

(12)

where $x_0\in {\mathbb{R}}^n$.

The local tracking error ${\delta }_i\in {\mathbb{R}}^n,i=1,\dots ,N$ can be defined as

$${\delta }_i\left(t\right)=\sum _{j\in {\mathcal{N}}_i}{e}_{ij}(x_i-x_j)+e_{i0}(x_i-x_0),$$

(13)

where the pinning gain $e_{i0}\ge 0$. Then, the dynamics of the local tracking error are written by

$$\begin{array}{cc}\hfill {\dot{\delta }}_i\left(t\right)=& \sum _{j\in {\mathcal{N}}_i}{e}_{ij}({\dot{x}}_i-{\dot{x}}_j)+e_{i0}({\dot{x}}_i-{\dot{x}}_0)\hfill \\ \hfill =& \sum _{j\in {\mathcal{N}}_i}{e}_{ij}(A_i{x}_i-A_j{x}_j)+e_{i0}(A_i{x}_i-A_0{x}_0)+(d_i+e_{i0})B_i{u}_i-\sum _{j\in {\mathcal{N}}_i}{e}_{ij}{B}_j{u}_j.\hfill \end{array}$$

(14)

The overall tracking error vector is given by

$$\begin{array}{cc}\hfill \delta \left(t\right)=& ((\mathcal{L}+\mathcal{B})\otimes I_n)(x-{\widehat{x}}_0)\hfill \\ \hfill =& ((\mathcal{L}+\mathcal{B})\otimes I_n)\zeta ,\hfill \end{array}$$

(15)

where $\delta ={({\delta }_1^T,{\delta }_2^T,\cdots ,{\delta }_N^T)}^T$, $x={(x_1^T,x_2^T,\cdots ,x_N^T)}^T\in {\mathbb{R}}^{nN}$, ${\widehat{x}}_0={(x_0^T,x_0^T,\cdots ,x_0^T)}^T\in {\mathbb{R}}^{nN}$, $\zeta =x-{\widehat{x}}_0$ is the global synchronization error.

One of the objectives in this paper is to design a tracking strategy to ensure that all follower agents can follow the leader, that is, $\underset{t\to \infty }{\lim }\parallel x_i\left(t\right)-x_0\left(t\right)\parallel =0$. The second objective is to design a distributed controller that can minimize the performance index function.

In fact, under Assumption 1, $\mathcal{L}+\mathcal{B}$ is invertible. From (15), one can obtain that $\underset{t\to \infty }{\lim }\zeta \left(t\right)=0$ if and only if $\underset{t\to \infty }{\lim }\parallel \delta \left(t\right)\parallel =0$. Thus, once the local neighbor error approaches to zero, we can say that the tracking control problem is solved.

We define the local performance index (value function) for the agent i as follows

$$V_i\left({\delta }_i\left(t\right)\right)=\frac{1}{2}{\int }_0^{\infty }({\delta }_i^T{Q}_{ii}{\delta }_i+U\left(u_i\right)+\sum _{j\in {\mathcal{N}}_i}U\left(u_j\right))d\tau ,$$

(16)

where $Q_{ii}⩾0$ is a symmetric weight matrix, $U(·)=u_i{R}_{ii}{u}_i$ is a positive definite integrand function. We assume that (16) satisfies zero-state observability.

The tracking problem is aimed at finding the Nash equilibrium policies $u_i^{*}$ for the N player game. That is, for all agent i, there have $V_i^{*}=V_i({\delta }_i\left(0\right),u_i^{*},u_{{\mathcal{N}}_i}^{*})\le V_i({\delta }_i\left(0\right),u_i,u_{{\mathcal{N}}_i}^{*}),$$\forall u_i$, ($i=1,\dots ,N$). Therefore, the tracking problem of MASs with input constraint in this paper can be transformed to solving the N coupled optimization problems, that is

$$\begin{array}{c}\hfill V_i^{*}\left({\delta }_i\left(t\right)\right)=\underset{u_i}{\min }\frac{1}{2}{\int }_0^{\infty }({\delta }_i^T{Q}_{ii}{\delta }_i+U\left(u_i\right)+\sum _{j\in {\mathcal{N}}_i}U\left(u_j\right))d\tau ,\end{array}$$

(17)

with given (14) while the dynamic informations $A_i$ and $B_i$, $i=1,\dots ,N$ are considered completely unknown.

By differentiating each value function $V_i$, and using (16), the following Lyapunov equation is obtained

$$\begin{array}{cc}\hfill \nabla V_i^T(\sum _{j\in {\mathcal{N}}_i}{e}_{ij}(A_i{x}_i-A_j{x}_j)& +e_{i0}(A_i{x}_i-A_0{x}_0)+(d_i+e_{i0})B_i{u}_i-\sum _{j\in {\mathcal{N}}_i}{e}_{ij}{B}_j{u}_j)+\frac{1}{2}{\delta }_i^T{Q}_{ii}{\delta }_i\hfill \\ \hfill & +\frac{1}{2}U\left(u_i\right)+\frac{1}{2}\sum _{j\in {\mathcal{N}}_i}U\left(u_j\right)=0,\hfill \end{array}$$

(18)

where $\nabla V_i=\partial V_i/\partial {\delta }_i\in {\mathbb{R}}^n$ and $V_i\left(0\right)=0$.

Then one can get the Hamiltonian function as follows

$$\begin{array}{cc}\hfill & H_i({\delta }_i,\nabla V_i,u_i,u_{{\mathcal{N}}_i})=\nabla V_i^T(\sum _{j\in {\mathcal{N}}_i}{e}_{ij}(A_i{x}_i-A_j{x}_j)+e_{i0}(A_i{x}_i-A_0{x}_0)\hfill \\ \hfill & +(d_i+e_{i0})B_i{u}_i-\sum _{j\in {\mathcal{N}}_i}{e}_{ij}{B}_j{u}_j)+\frac{1}{2}{\delta }_i^T{Q}_{ii}{\delta }_i+\frac{1}{2}U\left(u_i\right)+\frac{1}{2}\sum _{j\in {\mathcal{N}}_i}U\left(u_j\right).\hfill \end{array}$$

(19)

According to the first-order stationary condition in the Hamiltonians, the optimal control policy for each agent can be obtained as

$$\begin{array}{c}\hfill \frac{\partial H_i}{\partial u_i}=0\to u_i^{*}=-\left(d_i+e_{i0}\right)\lambda R_{ii}^{-T}{B}_i^T\nabla V_i^{*}.\end{array}$$

(20)

System identifier using neural networks: Since the system matrices $A_i$ and input matrices $B_i$ are assumed to be unknown, the unknown dynamics of each agent are modeled by using NNs. Then the experience replay technique is used to formulate the identifier weights adaptive update law.

The NN-based identifiers are designed to approximate system dynamic, which is given as follows

$$A_i{x}_i=A_i^{*}{x}_i+{\epsilon }_{A_i},B_i{u}_i=B_i^{*}{u}_i+{\epsilon }_{B_i},i=1,\dots ,N,$$

(21)

where $A_i^{*}\in ,R^{n\times n}$,$B_i^{*}\in {\mathbb{R}}^{n\times m}$ are unknown weights, $x_i\in {\mathbb{R}}^n$,$u_i\in {\mathbb{R}}^m$ are the basis functions, and ${\epsilon }_{A_i}$ and ${\epsilon }_{B_i}$ are the reconstruction errors.

Combining (21) and (11), the system can be reformulated as follows

$${\dot{x}}_i={\vartheta }_{A_i{B}_i}^{*}{z}_i(x_i,u_i)+{\epsilon }_{A_i{B}_i},i=1,\dots ,N,$$

(22)

where ${\vartheta }_{A_i{B}_i}^{*}=\left[A_i^{*}{B}_i^{*}\right]\in {\mathbb{R}}^{n\times d}$, $z_i(x_i,u_i)={\left[x_i^T{u}_i^T\right]}^T\in {\mathbb{R}}^d$ is the regressor vector. ${\epsilon }_{A_i{B}_i}={\epsilon }_{A_i}+{\epsilon }_{B_i}$ is the model approximation error.

Assumption 2. 

On a given compact set $\Omega \subset {\mathbb{R}}^n$, the approximator reconstruction errors ${\epsilon }_{A_i}$ and ${\epsilon }_{B_i}$, $i=1,\dots ,N$ and their gradients are bounded, i.e., $∥{\epsilon }_{A_i}∥\le {\overline{\epsilon }}_{A_i}$,$∥{\epsilon }_{B_i}∥\le {\overline{\epsilon }}_{B_i}$, and the approximator basis functions and their gradients are bounded.

Remark 1. 

According to Assumption 2, the model approximation error ${\epsilon }_{A_i{B}_i}$ is bounded, that is, $∥{\epsilon }_{A_i{B}_i}∥\le {\overline{\epsilon }}_{A_i{B}_i}={\overline{\epsilon }}_{A_i}+{\overline{\epsilon }}_{B_i}$.

A filtered regressor is proposed for (22), which can be expressed as

$$x_i={\vartheta }_{A_i{B}_i}^{*}{h}_i\left(x_i\right)+c{l}_i\left(x_i\right)+{\epsilon }_{x_i},$$

(23)

$$\begin{array}{cc}\hfill & {\dot{h}}_i\left(x_i\right)=-c{h}_i\left(x_i\right)+z(x_i,u_i),h_i\left(0\right)=0,\hfill \\ \hfill & {\dot{l}}_i\left(x_i\right)=-C{l}_i\left(x_i\right)+x_i,l_i\left(0\right)=0,\hfill \end{array}$$

(24)

where $C=c{I}_{n\times n},c>0$, $h_i\left(x_i\right)\in {\mathbb{R}}^d$ is a filtered regressor version of $z(x_i,u_i)$, $l_i\left(x_i\right)\in {\mathbb{R}}^n$ is a filtered regressor version of the state $x_i$. ${\epsilon }_{x_i}=e^{-Ct}{x}_i\left(0\right)+{\int }_0^t{e}^{-C(t-\tau )}{\epsilon }_{A_i{B}_i}d\tau$ is bounded, since ${\epsilon }_{A_i{B}_i}$ is bounded. $x_i\left(0\right)$ is the initial state of (22).

To obtain the adaptive tuning law that does not affected by the system instability, both side of the filtered regressor (23) are divided by a normalizing signal $n_{s_i}=1+h_i^T{h}_i+l_i^T{l}_i$,

$${\overline{x}}_i={\vartheta }_{A_i{B}_i}^{*}{\overline{h}}_i\left(x_i\right)+c{\overline{l}}_i\left(x_i\right)+{\overline{\epsilon }}_{x_i},$$

(25)

where ${\overline{x}}_i=x_i/n_{x_i}$, ${\overline{h}}_i=h_i/n_{x_i}$, ${\overline{l}}_i=l_i/n_{x_i}$, ${\overline{\epsilon }}_{x_i}={\epsilon }_{x_i}/n_{x_i}$. Obviously, ${\overline{\epsilon }}_{x_i}$ is bounded.

Based on (21), (23) and (25), the form of the identifier weights estimator of agent i can be expressed as

$${\widehat{\overline{x}}}_i={\widehat{\vartheta }}_{A_i{B}_i}{\overline{h}}_i\left(x_i\right)+c{\overline{l}}_i\left(x_i\right),i=1,\dots ,N$$

(26)

where ${\widehat{\vartheta }}_{A_i{B}_i}=\left[{\widehat{A}}_i{\widehat{B}}_i\right]$ is the estimated value of the identifier weights matrix ${\vartheta }_{A_i{B}_i}^{*}$.

Thus, the state estimation error $e_i\in {\mathbb{R}}^n,i=1,\dots ,N$ can be defined as

$$e_i\left(t\right)={\widehat{\overline{x}}}_i-{\overline{x}}_i={\tilde{\vartheta }}_{A_i{B}_i}\left(t\right){\overline{h}}_i\left(x_i\right)-{\overline{\epsilon }}_{x_i},$$

(27)

where ${\tilde{\vartheta }}_{A_i{B}_i}\left(t\right)={\widehat{\vartheta }}_{A_i{B}_i}\left(t\right)-{\vartheta }_{A_i{B}_i}^{*}\left(t\right),i=1,\dots ,N$ is the parameter estimation error of agent i at time t.

The experience replay technique is utilized to formulate the identifier weights adaptive tuning law in the following. The idea of this technique is to store or record linearly independent historical data along with current data, so as to improve data utilization.

Then, we set

$$Z_i=[{\overline{h}}_i\left(x_i\left(t_1\right)\right),\dots ,{\overline{h}}_i\left(x_i\left(t_{p_i}\right)\right)]$$

(28)

to be the recorded historical data stack of each agent i at the past times $t_1,\dots ,t_{p_i}$.

Remark 2. 

It is noted that the number of linearly independent elements in $Z_i$ should be equal to the dimension of the $h_i\left(x_i\right)$ in (23), i.e., $rank\left(Z_i\right)=d$. This condition aims to satisfying the PE condition and can easily be checked online.

Then, based on the experience replay technique, a weight tuning law is designed for the identifier of agent i as follows

$$\begin{array}{cc}\hfill {\dot{\widehat{\vartheta }}}_{A_i{B}_i}\left(t\right)=& -{\Gamma }_i{e}_i\left(t\right){{\overline{h}}_i}^T\left(x_i\left(t\right)\right)-{\Gamma }_i\sum _{k=1}^{p_i}{e}_i\left(t_k\right){{\overline{h}}_i}^T\left(x_i\left(t_k\right)\right),\hfill \end{array}$$

(29)

where ${\Gamma }_i>0,i=1,\dots ,N$ is a positive definite learning rate matrix.

It is noted from Remark 2 that, with the aid of experience replay technique, the PE condition can be checked by monitoring the rank of the recorded historical data, but it usually consumes large computing resources, resulting in low learning efficiency. Therefore, how to design an identification method that can take into account the learning efficiency and relaxed the PE condition is an interesting and challenging research direction.

6. Conclusions and Future Work

In this paper, we have reviewed the development of ADP-based learning optimal control, several model identification techniques, and their applications to the learning and control of MASs. Based on these reviews, it is noted that the model identification-based ADP method has made significant progress in both theoretical research and practical applications. However, the model identification-based ADP methods still have many challenges in theory and algorithm design that have not yet been resolved. Through the above summary and analysis of the model identification-based ADP methods, some related issues for future research directions are outlined as follows:

• In fact, the model identification-based ADP method is mainly focused on the design of a single controller currently, but not so much on the design of multiple controllers. It will be a very beneficial work to use the model identification-based ADP method to realize the distributed coordinated control of MASs.
• Most of the existing model identification-based ADP methods need to satisfy the PE condition. However, PE conditions are difficult to verify in practical applications. How to design a novel identification-based ADP method such that the PE condition is easier to be checked and remain low pressure [82].
• For more complex MASs such as power grids and transportation, where their accurate models cannot be obtained, the model identification-based ADP method may be used to solve large-scale practical optimization problems, which have important practical applications.