Neural Networks Simulation of Distributed SEIR System ^†

Abstract

In this paper, a neural network-based optimal control synthesis is presented for distributed optimal control problems. We deal with solutions of systems controlled by parabolic differential equations with control and state constraints and discrete time delays. The given optimal control problem is transformed into a discrete nonlinear problem and then implemented into a feed-forward adaptive critic neural network. We propose a new algorithm to reach optimal control and an optimal trajectory using a feed-forward neural network. We present a concrete application of this simulation method on the SEIR (Susceptible—Exposed—Infectious—Recovered) optimal control problem of a distributed system for disease control. The results show that the adaptive-critic-based neural network approach is suitable for the solution of optimal distributed control problems with delay in state and control variables subject to control-state constraints and simulates the spread of disease in the SEIR system.

1. Introduction

In recent decades, the application of nonlinear dynamic systems has proven to be useful in many fields, for example, computer networks, robotics, transportation systems, epidemiology, and biology. These nonlinear dynamic systems were extended with control variables to control the dynamic of the systems. The design of controllers for nonlinear dynamic systems has also drawn intensive attention [1]. Many results have been obtained for the control of dynamic systems using adaptive control and neural networks (see, e.g., [2,3,4]).

According to Marino [5] (p. 55), “Adaptive controls for nonlinear systems are defined as nonlinear dynamic (state or output feedback) compensators capable of guaranteeing asymptotic tracking of an output reference signal for any unknown parameter vector and any initial condition for the closed loop system when time varying disturbances are not present.” Dynamic programming has been a useful technique to control nonlinear dynamic systems [6], where we want to attain the minimum or maximum of a given objective function, optimization criterion, or performance index representing the mathematical expression of a phenomenon.

The aim of our article, which is related to works [7,8,9,10], is to apply neural network methods in the study and simulation of nonlinear complex adaptive epidemiological systems, specifically a SEIR system. We consider an optimal distributed control problem for systems governed by parabolic differential equations with control and state constraints and a discrete time delay for the SEIR model, which contains the intensity of vaccination. There are some numerical methods based on necessary conditions of optimality [11,12], where the state and co-state equations are solved forward and backward in time. In the last decade, efficient techniques have been developed [13,14,15] to construct learning systems for the approximate solution of optimal control problems. Characterized by self-learning and adaptability, artificial neural networks are also a serviceable tool to implement controlled learning [16,17,18]. “Multilayer feedforward networks with as few as one hidden layer are indeed capable of universal approximation in a very precise and satisfactory sense” [19] (p. 260). Therefore, feedforward networks are used in adaptive dynamic programming (ADP) algorithms or adaptive critic design (ACD), which was proposed by Werbos [17,18] to deal with optimal control problems forward in time.

The finite element approximation approach has been thoroughly studied in [20,21,22] and [12] for parabolic and elliptic optimal control problems. This approach plays an important role in the numerical treatment of optimal control problems. The characteristic of this solution is that the optimal control problem is translated into a finite-dimensional nonlinear programming problem (NLP-problem) by discretization. These methods, basically, apply nonlinear programming techniques to the resulting finite-dimensional optimization problem, as one can see in [23,24] and [12]. Thereafter, neural networks are used as a universal approximation to solve finite-dimensional optimization problems forward in time with ACD [18,25,26,27].

In this paper, we use an adaptive critic neural network architecture based on [7] for two optimal distributed control problems. In Section 2, we introduce an optimal control problem for a parabolic differential equation where state and control variables are delayed in time and also a control problem with control and state constraints. In Section 3, we transform the initial continuous problem into a nonlinear programming problem using discretization. Section 4 presents a new algorithm using adaptive critic neural network design, where the adaptive neural network approximates optimal control and critic network co-state variables. Section 5 presents a short description of the SEIR model and investigates the optimal vaccine coverage threshold needed for disease control and eradication as the optimal control problem subject to control and state constraints. For the numerical experiments described in Section 5, Matlab 2021b software was used.

2. The Optimal Control Problem

We consider the nonlinear control problem governed by parabolic equations with delays in state and control variables for mixed control-state constraints. We denote $x(p,t)\in R^n$ and $u(p,t)\in R^m$ the state and control variable, respectively, in a given space-time domain $Q=\left[a,b\right]\times \left[t_0,t_f\right]$, where the final time $t_f$ is fixed. The optimal control problem is to minimize

$$\mathcal{J}(u)={\displaystyle {\int }_a^b}g(x(p,t_f))dp+{\displaystyle {\int }_a^b}{\displaystyle {\int }_{t_0}^{t_f}}{f}_0(x(p,t),x(p,t-{\tau }_x),u(p,t),u(p,t-{\tau }_u))dtdp,$$

(1)

subject to

$$\frac{\partial x(p,t)}{\partial t}=D\frac{{\partial }^{2}x(p,t)}{\partial p^2}+f(x(p,t),x(p,t-{\tau }_x),u(p,t),u(p,t-{\tau }_u)),$$

(2)

$$\frac{\partial x(a,t)}{\partial p}=\frac{\partial x(b,t)}{\partial p}=0,t\in [t_0,t_f],$$

$$x(p,t)={\varphi }_s(p,t),u(p,t)={\varphi }_c(p,t),p\in [a,b],t\in [t_0-{\tau }_u,t_0],$$

$$\psi (x(p,t_f))=0,c(x(p,t),u(p,t))\le 0,p\in [a,b],t\in [t_0,t_f].$$

where $c(x,u)$ formulates a mixed state-control constraint, ${\tau }_x\ge 0$ and ${\tau }_u\ge 0$ are discrete time delays in the state and control variable, respectively, and $\psi$ is a given terminal condition for state variable $x.$ $D$ is a diffusion matrix. The functions $g:R^n\to R$, $f_0:R^{2(n+m)}\to R$, $f:R^{2(n+m)}\to R^n$, $c:R^{n+m}\to R^q$ and $\psi :R^n\to R^r,$ $0\le r\le n$ are assumed to be sufficiently smooth on appropriate open sets, and the initial conditions ${\varphi }_s(p,t),{\varphi }_c(p,t)$ are continuous functions. Dynamic optimization objective functional $J(u)$ has Bolza form with scalar terminal cost and Equation (2) describes a dynamic of state variables. The theory of necessary conditions for the optimal control problem of form (1) is well developed, see, e.g., [24,28]. To simplify the notation, we introduce an additional state variable $x_0(p,t)={\displaystyle {\int }_0^t}{f}_0(x(p,s),x(p,s-{\tau }_x),u(p,s),$ $u(p,s-{\tau }_u)ds.$ Then the augmented Hamiltonian function for problem (1) is

$$\mathcal{H}(x,x_{{\tau }_x},u,u_{{\tau }_u},\lambda ,\mu )={\displaystyle \sum _{j=0}^n}{\lambda }_j{f}_j(x,x_{{\tau }_x},u,u_{{\tau }_u})+{\displaystyle \sum _{j=0}^q}{\mu }_j{c}_j(x,u),$$

where $\lambda \in R^{n+1}$ is the adjoint variable and $\mu \in R^q$ is a multiplier associated to the inequality constraints. Assume that ${\tau }_x,{\tau }_u\ge (0,{\tau }_x,{\tau }_u)\ne (0,0)$ and $\frac{{\tau }_x}{{\tau }_u}\in Q$ for ${\tau }_u>0$ or $\frac{{\tau }_u}{{\tau }_x}\in Q$ for ${\tau }_x>0.$ Let $(\widehat{x},\widehat{u})$ be an optimal solution for (1). Then the necessary optimality condition for (1) implies [24,28] that there exist a piecewise continuous and piecewise continuously differentiable adjoint function $\lambda :Q\to R^{n+1}$, a piecewise continuous multiplier function $\mu :Q\to R^q,$ $\widehat{\mu }(p,t)\ge 0$ and a multiplier $\sigma \in R^r$ satisfying

$$\frac{\partial \lambda }{\partial t}=D\frac{{\partial }^2\lambda }{\partial p^2}-\frac{\partial \mathcal{H}}{\partial x}\left(\widehat{x},{\widehat{x}}_{{\tau }_x},\widehat{u},{\widehat{u}}_{{\tau }_u},\lambda ,\mu \right)-{\chi }_{[t_0,t_f-{\tau }_x]}\frac{\partial \mathcal{H}}{\partial x_{{\tau }_x}}({\widehat{x}}_{+{\tau }_x},\widehat{x},{\widehat{u}}_{+{\tau }_x},{\widehat{u}}_{{\tau }_u+{\tau }_x},{\lambda }_{+{\tau }_x},{\mu }_{+{\tau }_x}),$$

(3)

$$\lambda (p,t_f)=g_x(\widehat{x}(p,t_f))+\sigma {\psi }_x(\widehat{x}(p,t_f)),\frac{\partial \lambda (a,t)}{\partial p}=\frac{\partial \lambda (b,t)}{\partial p}=0,$$

(4)

$$0=-\frac{\partial \mathcal{H}}{\partial u}\left(\widehat{x},{\widehat{x}}_{{\tau }_x},\widehat{u},{\widehat{u}}_{{\tau }_u},\lambda ,\mu \right)-{\chi }_{[t_0,t_f-{\tau }_u]}\frac{\partial \mathcal{H}}{\partial u_{{\tau }_u}}({\widehat{x}}_{+{\tau }_u},{\widehat{x}}_{{\tau }_x+{\tau }_u},{\widehat{u}}_{+{\tau }_u},\widehat{u},{\lambda }_{+{\tau }_u},{\mu }_{+{\tau }_u}).$$

(5)

In addition, the complementary conditions hold, i.e., in $p\in [a,b],t\in [t_0,t_f],$ $\mu (p,t)\ge 0,$ $c(x(p,t),u(p,t))\le 0$, and $\mu (p,t)c(x(p,t),u(p,t))=0$. The subscript $x,x_{{\tau }_x},u$ and $u_{{\tau }_u}$ denote the partial derivative with respect to $x,x_{{\tau }_x},u$ and $u_{{\tau }_u},$ respectively and $x_{+{\tau }_x}=x(p,t+{\tau }_x),$ $x_{{\tau }_x+{\tau }_u}=x(p,t-{\tau }_x+{\tau }_u).$ Numerical methods that solve the challenges of obtaining an optimal control $\widehat{u}$ and optimal trajectory $\widehat{x}$ are described in [29,30]. The first is based on solving the optimal system (2, 3) which consists of $2n$ partial differential equations and boundary conditions in time also. The second method starts with initial guess for the adjoint state given by Equation (3). We solve Equation (6) and then state Equation (2). Then, those control values $u$ and state values $x$ are used to solve the adjoint state $\lambda$ by backward method in time. We proposed a new method to solve the distributed optimal control problem (1) using adaptive critic neural networks, where state variable $x$ and adjoint variable $\lambda$ are solved forward in time.

3. Discretization of the Distributed Optimal Control Problem

In this section, we develop discretization techniques to transform the distributed control problem (1) into a nonlinear programming problem (see also in [11,12,20]). We suppose that ${\tau }_u=l\frac{{\tau }_x}{k}$ with $l,k\in N.$ Defining $h_{max}=\frac{{\tau }_x}{k}$ gives the maximum interval length for an elementary transformation interval that satisfies $\frac{{\tau }_x}{h_{max}}=k\in N$ and $\frac{{\tau }_u}{h_{max}}=l\in N.$ The minimum number of grid points for an equidistant discretization mesh is $N_{min}=\frac{t_f-t_0}{h_{max}}.$ Let $K\in N$, and set $N=K{N}_{min}.$ Let $t_j\in \langle t_0,t_f\rangle , j=0,\dots ,M,$ be an equidistant grid point with $t_j=t_0+j{h}_t,$ $j=0,\dots ,M$, where $h_t=\frac{b-a}{M}$ is a time step and $t_f=Nh+t_0$. Let us suppose that the rectangle $R=\{(p,t):a\le p\le b,t_0\le t\le t_f\}$ is subdivided into $N$ by $M$ rectangles, each with sides $h_t$ and $h_s=\frac{b-a}{M}$. For the bottom row, we have $t=t_0$, and the solution is $x(p_i,t_0)={\varphi }_s(p_i,t_0)$. For the grid points in successive rows, we have $\{x(p_i,t_j):i=0,1,\dots ,N,$ $j=0,1,\dots ,M\}$ for the approximations to $x(p,t)$. The difference formulae for $x_t(p,t),$ $x_p(p,t)$ and $x_{pp}(p,t)$ are

$$x_t(p,t)\approx \frac{x(p,t+h_t)-x(p,t)}{h_t},$$

$$x_p(p,t)\approx \frac{x(p+h_s,t)-x(p,t)}{h_s}$$

(6)

And

$$x_{pp}(p,t)\approx \frac{x(p-h_s,t)-2x(p,t)+x(p+h_s,t)}{h_s^2}.$$

(7)

Using the approximation $x_{i,j}$ in Equations (6) and (7), which are substituted into Equation (2), we obtain that

$$\frac{x_{i,j+1}-x_{i,j}}{h_t}=D\frac{x_{i-1,j}-2x_{i,j}+x_{i+1,j}}{h_s^2}+f_{i,j}.$$

(8)

Equation (8) is applied to create the $(j+1)$th row across the grid, assuming that approximations in the $j$th row are known. Let the vectors $x_{ij}\in R^n, u_{ij}\in R^m,$ $i=0,\dots ,N,j=0,\dots ,M,$ be an approximation of the state variable and control variable $x(p_i,t_j), u(p_i,t_j)$, respectively at the mesh point $(p_i,t_j).$ $z:=((x_{ij}),(u_{ij}),i=0,\dots ,N,j=0,\dots ,M)\in R^{N_s}, N_s=(n+m)NM.$ Instead of the optimal control problem, here we use the following discretised control problem in the form of nonlinear programming problem with inequality constraints: Minimise

$$\mathcal{J}(z)=h_s{h}_t{\displaystyle \sum _{(i,j)}}{f}_0(x_{ij},x_{{\tau }_{x}ij},u_{ij},u_{{\tau }_{u}ij})+h_s{\displaystyle \sum _{(i)}}g(x_{iM})$$

(9)

subject to

$$x_{i,j+1}=x_{ij}+h_{t}D\frac{x_{i-1,j}-2x_{ij}+x_{i+1,j}}{h_s^2}+h_t{f}_{ij},$$

(10)

$$x_{0j}=x_{1j},x_{Nj}=x_{N-1j}, i=0,\dots ,N, j=0,\dots ,M-1,$$

$$x_{i,-j}={\varphi }_x(p_i,t_0-jh),j=k,\dots ,0,u_{i,-j}={\varphi }_u(p_i,t_0-jh),j=l,\dots ,0,$$

$$\psi (x_{i,N})=0,c(x_{ij},u_{ij})\le 0, i=0,\dots ,N, j=0,\dots ,M-1,$$

Our goal is to find an admissible control in a discrete-time formulation which minimises objective function (9). The Lagrangian function for the nonlinear optimisation problem (9) is given by Equation (12).

$$\mathcal{L}(z,\lambda ,\sigma ,\mu )=h_s{h}_t{\displaystyle \sum _{(i,j)}}{f}_0(x_{ij},x_{{\tau }_{x}ij},u_{ij},u_{{\tau }_{u}ij})+$$

(11)

$$\begin{gathered} {\displaystyle \sum _{(i,j)}}{\lambda }_{i,j+1}(-x_{i,j+1}+x_{ij}+D\frac{x_{i-1,j}-2x_{ij}+x_{i+1,j}}{h_s^2}+h_t{f}_{ij})+{\displaystyle \sum _{(i,j)}}{\mu }_{ij}c(x_{ij},u_{ij})+ \\ {\displaystyle \sum _{(i)}}{\sigma }_i\psi (x_{iN})+h_s{\displaystyle \sum _{(i)}}g(x_{iM})+{\displaystyle \sum _{(j)}}{\lambda }_{0,j}\frac{x_{0,j}-x_{1,j}}{h_s}+{\displaystyle \sum _{(j)}}{\lambda }_{N,j}\frac{x_{N,j}-x_{N-1,j}}{h_s}, \end{gathered}$$

The first order optimality Karush–Kuhn–Tucker conditions [31] (p. 199) for problem (9) are given by Equations (12)–(14).

$$\begin{gathered} 0={\mathcal{L}}_{x_{ij}}(z,\lambda ,\sigma ,\mu )={\lambda }_{i,j+1}-{\lambda }_{ij}+h_{t}D\frac{{\lambda }_{i-1,j}-2{\lambda }_{ij}+{\lambda }_{i+1,j}}{h_s^2}+h_t{\lambda }_{i,j+1}{f}_{x_{ij}}(x_{ij},x_{i,j-k},u_{ij},u_{i,j-l})+ \\ {h}_t{\lambda }_{i,j+k+1}{f}_{x_{ij{\tau }_x}}(x_{i,j+k},x_{ij},u_{i,j+k},u_{i,j-l+k})+{\mu }_{ij}{c}_{x_{ij}}(x_{ij},u_{ij}), j=0,\dots ,M-k-1, \end{gathered}$$

(12)

$$\begin{gathered} 0={\mathcal{L}}_{x_{ij}}(z,\lambda ,\sigma ,\mu )={\lambda }_{i,j+1}-{\lambda }_{ij}+h_{t}D\frac{{\lambda }_{i-1,j}-2{\lambda }_{ij}+{\lambda }_{i+1,j}}{h_s^2}+ \\ {h}_t{\lambda }_{i,j+1}{f}_{x_{ij}}(x_{ij},x_{i,j-k},u_{ij},u_{i,j-l}), j=M-k,\dots ,M-1, \end{gathered}$$

$$0={\mathcal{L}}_{x_{iM}}(z,\lambda ,\sigma ,\mu )=g_{x_{iM}}(x_{im})+{\sigma }_i{\psi }_{x_{iM}}(x_{iM})-{\lambda }_{iM},{\lambda }_{0j}={\lambda }_{1j},{\lambda }_{Nj}={\lambda }_{N-1,j},$$

(13)

$$\begin{gathered} 0={\mathcal{L}}_{u_{ij}}(z,\lambda ,\sigma ,\mu )=h_t{\lambda }_{i,j+1}{f}_{u_{ij}}(x_{ij},x_{i,j-k},u_{ij},u_{i,j-l})+ \\ {h}_t{\lambda }_{i,j+l+1}{f}_{u_{ij{\tau }_u}}(x_{i,j+l},x_{i,j-k+l},u_{i,j+l},u_{i,j})+{\mu }_{ij}{c}_{u_{ij}}(x_{ij},u_{ij}), j=0,\dots ,M-l-1, \end{gathered}$$

(14)

$$0={\mathcal{L}}_{u_{ij}}(z,\lambda ,\sigma ,\mu )=h_t{\lambda }_{i,j+1}{f}_{u_{ij}}(x_{ij},x_{i,j-k},u_{ij},u_{i,j-l})+{\mu }_{ij}{c}_{u_{ij}}(x_{ij},u_{ij}), j=M-l,\dots ,M-1.$$

These equations represent the discrete version of the necessary conditions (3)–(6) for the optimal control problem (1).

4. Adaptive Critic Neural Network for a Distributed Optimal Control Problem with Control and State Constraints

Neural networks are useful for approximation of the smooth time-invariant functions. According to [32] (p. 2), “experience has shown that optimisation of functionals over admissible sets of functions made up of linear combinations of relatively few basis functions with a simple structure and depending nonlinearly on a set of “inner” parameters e.g., feedforward neural networks with one hidden layer and linear output activation units often provides surprisingly good suboptimal solutions”. In 1977, Werbos [18,33] introduced an approach to “approximate dynamic programming”, which later became known under the name adaptive critic design (ACD). “A typical design of ACD consists of three modules: action, model (plant), and critic” [8] (p. 587), also in [14,17,18].

In the following part describing Algorithm 1, we resolve how to adjust the critic network, the model network, and the action network. ACD is able to optimize over time under conditions of noise and uncertainty. For control actions to be performed sequentially for a given series, while the effect of the actions is not known up to the end of the sequence, one cannot design an optimal controller with the well-known supervised learning artificial neural network. Progressively adapting two artificial neural networks, we have an adaptive critic method in which the first neural network dispenses the control signals (this is the action network) and the second one (the critic network) learns the requested performance index for some action associated with the index. These two then approximate the optimal control and co-state equation. The adaptation process starts with an arbitrary selection, controlled by the action network. “The critic network then guides the action network towards the optimal solution at each successive adaptation. This method determines optimal control policy for the entire range of initial conditions and needs no external training, unlike other neurocontrollers” [34] (p. 293).

Algorithm 1: Algorithm for Solving the Optimal Control Problem
Input: Select $t_0,t_f,a,b,$$N,M$- number of steps, time and space steps $h_t,h_s,$       ${\epsilon }_a,{\epsilon }_c$- stopping tolerance for action and critic neural networks, respectively, $\hspace{1em}{x}_{i,-j}={\varphi }_s(p_i,t_0-j{h}_t),j=k,\dots ,0,$       $u_{i,-j}={\varphi }_c(i,t_0-j{h}_t),j=l,\dots \mathrm{,0}$- initial values.
Output: Set of final approximate optimal control $\widehat{u}(p_i,t_0+j{h}_t)={\widehat{u}}_{ij}$ and optimal trajectory $\widehat{x}(p_i,t_0+(j+1)h_t)={\widehat{x}}_{i,j+1},j=0,\dots ,M-1$,
1	Set the initial weight ${\mathbb{W}}^a=(V^a,W^a),{\mathbb{W}}^c=(V^c,W^c)$ for $j\leftarrow 0$ to $M-1$ do
2			for $i\leftarrow 1$ to $N-1$ do
3					while $er{r}_a\ge {\epsilon }_a$ and $er{r}_c\ge {\epsilon }_c$ do
4							for $s\leftarrow 0$ to $max(k,l)$ do
5									Compute $u_{i,s+j}^a,{\mu }_{i,s+j}^a$ and ${\lambda }_{i,s+j+1}^c$ using action (${\mathbb{W}}^a$) and critic (${\mathbb{W}}^c$) networks, and $x_{i,s+j+1}$ by (10)
6							Compute ${\lambda }_{ij}^t,u_{ij}^t,and {\mu }_{ij}^t$ using (12) and (14) $\mathbb{F}(u_{ij},{\mu }_{ij})=({\mathcal{L}}_{u_{ij}}(z,\lambda ,\sigma ,\mu ),-c(x_{ij},u_{ij}))=0$
7							if $j=M-1$ then
8									$\mathbb{F}(u_{i,M-1},{\mu }_{i,M-1},{\sigma }_i)=({\mathcal{L}}_{u_{i,M-1}}(z,\lambda ,\sigma ,\mu ),-c(x_{i,M-1},u_{i,M-1}),-\psi (x_{i,M}))$ with ${\lambda }_{iM}={\mathcal{G}}_{x_{iM}}(x_{iM})+{\sigma }_i{\psi }_{x_{iM}}(x_{iM})$
9							$er{r}_c=\parallel {\lambda }_{ij}^t-{\lambda }_{ij}^c\parallel$
10							$er{r}_a=\parallel {(u,\mu )}_{ij}^t-{(u,\mu )}_{ij}^a\parallel$
11							With data set $x_{ij},{\lambda }_{ij}^t$ update weight parameters ${\mathbb{W}}^c$
12							With data set $x_{ij},(u,\mu {)}_{ij}^t$ update weight parameters ${\mathbb{W}}^a$
13							Set ${\lambda }_{ij}^c={\lambda }_{i,j}^t,$ ${(u,\mu )}_{i,j}^a=(u,\mu {)}_{i,j}^t$
14					Set ${\widehat{\lambda }}_{i,j}={\lambda }_{i,j}^t,({\widehat{u}}_{i,j},{\widehat{\mu }}_{i,j})=(u,\mu {)}_{i,j}^t$
15					Compute ${\widehat{x}}_{i,j+1}$ using (10) and ${\widehat{u}}_{i,j}$
16			${\lambda }_{0j}={\lambda }_{1j},{\lambda }_{Nj}={\lambda }_{N-1,j}$
17			return ${\widehat{\lambda }}_{i,j},{\widehat{u}}_{i,j},{\widehat{\mu }}_{i,j},{\widehat{x}}_{i,j+1}$

Let $x=[x_1,\dots ,x_n{]}^{\prime }$ and $y=[y_1,\dots ,y_m{]}^{\prime }$ be the input and output vectors for this network. The matrix of synaptic weights between the input nodes and the hidden units is $V=[v_1,\dots ,v_r{]}^{\prime }$, where $v_k=[v_{k0},v_{k1}\dots ,v_{kn}];$ $v_{k0}$ is the bias of the $k$th hidden unit and $v_{ki}$ is the weight for $i$th input node connecting to the $k$th hidden unit. Similarly, $W=[w_1,\dots ,w_m{]}^{\prime }$, (where $w_j=[w_{j0},w_{j1}\dots ,w_{jr}];$ $w_{j0}$ is the bias of $j$th output unit, and $w_{jk}$ is the weight that connects the $k$th hidden unit to $j$th output unit) is the matrix of synaptic weights between the hidden and output units. The response of the $k$th hidden unit is given by $z_k=tanh\left({\displaystyle {\sum }_{i=0}^n}{v}_{ki}{x}_i\right),k=1,\dots ,r,$ where $\tanh (.)$ is the activation function for the hidden units (hyperbolic tangent), and of the $j$th output unit is given by $y_j={\displaystyle {\sum }_{k=0}^r}{w}_{jk}{z}_k,$ $j=1,\dots ,m.$

The number of input variables specifies the number of neurons in the input layer. Multiple layers of neurons with nonlinear transition functions provide the opportunity to learn nonlinear and linear relationships between input and output vectors. The number of output vectors is determined by the number of output variables. The multi-layered network is trained by back-propagation algorithm, where the steepest descent error is propagated backward from the output to the input layer. With the help of the adaptive critic and recurrent neural network we can specify the algorithm for solving the optimal control problem (Algorithm 1). In the maximum principle of Pontryagin for optimal control, the mutual dependence of the state, co-state and control dynamics becomes clear. Naturally, the optimal control $\widehat{u}$ and multiplier $\widehat{\mu }$ are given by Equation (14), while co-state Equations (12) and (13) progress backward in time and depend on the state and control. Adaptive critic neural networks based on this relationship are shown in [26,27]. There are two networks at each node. The first one is an action network, where the inputs are the current states and their outputs are the corresponding control $\widehat{u}$ and multiplier $\widehat{\mu },$ and the second one is a critic network, where the current states for the critic network are inputs and current co-states are outputs for normalising the inputs and targets. For a detailed explanation see [35].

The output of the critic network is given by ${\lambda }^{ij,c}=W^{c}f(V^c{x}^{ij}).$ The target function ${\lambda }^{ij,t}$ is expressed by Equation (12). Then, we define the error function of the critic network as $er{r}_c=\parallel {\lambda }^{ij,t}-{\lambda }^{ij,c}\parallel .$ The objective function to be minimised in the critic network is $E_c=1/2er{r}_c^2.$ The weights are updated by

$$W^c(s+1)=W^c(s)-\rho \frac{\partial E^c}{\partial W^c},$$

(15)

$$V^c(s+1)=V^c(s)-\rho \frac{\partial E^c}{\partial V^c},$$

where $\rho >0$ is the learning rate of the critic network; $j$ is the inner-loop iteration step for updating the weight matrix $W^c$ and $V^c.$ This is a gradient-based adaptation for training the critic network.

To get the approximate optimal control as an output, the state $x^i$ was used as an input in the action network $(u^{ij,a},{\mu }^{ij,a})=W^{a}f(V^a{x}^{ij}).$ The target function $(u^{ij,t},{\mu }^{ij,t})$ can be written as a solution of Equation (14). The error function of the action network is $er{r}_a=\parallel (u^{iu,t},{\mu }^{ij,t})-(u^{ij,a},{\mu }^{ij,a})\parallel .$ The objective function to be minimised in the action network is $E_a=1/2er{r}_a^2.$ The weights are updated by

$$W^a(s+1)=W^a(s)-\rho \frac{\partial E^a}{\partial W^a},$$

(16)

$$V^a(s+1)=V^a(s)-\rho \frac{\partial E^a}{\partial V^a},$$

where the meaning of $\rho >0$ and $j$ are the same as in (15) for the weight matrices of action network $W^a$ and $V^a.$

The whole procedure of the distributed optimal control problem is summarised in Algorithm 1. Now, we describe the condition for terminating the training of action and critic networks within each cycle $j=0,\dots ,M-1$ and $i=1,\dots ,N-1.$ We use stopping tolerance ${\epsilon }_a$ for action network to approximate optimal control $u$ and multipliers $\mu .$ Convergence condition is given as $er{r}_a=\parallel {(u,\mu )}_{ij}^t-{(u,\mu )}_{ij}^a\parallel ,$ where ${(u,\mu )}_{ij}^t$ and ${(u,\mu )}_{ij}^a$ are solutions of optimality condition (14), inequality multipliers and an action network output, respectively. Similarly, we use stopping tolerance ${\epsilon }_c$ for critic network to approximate co-state variable $\lambda .$ Convergence condition is given as $er{r}_c=\parallel {\lambda }_{ij}^t-{\lambda }_{ij}^c\parallel ,$ where ${\lambda }_{ij}^t$ and ${\lambda }_{ij}^c$ are a solutions of co-state Equation (12) and the critic network output, respectively. The proof of convergence of adaptive critic design for a class of parabolic partial differential equations is given in [25]. For nonlinear distributed systems with time delay we can examine only the numerical convergence of ACD.

The learning procedure for the action and critic network is shown in [27,35]. The action and critical networks in adaptive critic synthesis were chosen to consist of $n+m$ subnetworks, each constructed with $n-3n-1$ structure such that there are $n$, $3n$ and 1 neurons in the input, the hidden and the output layer, respectively.

5. SEIR Model with Spatial Diffusions

The classical SEIR (Susceptible—Exposed—Infectious–Recovered) model is one of the most used models in the simulation of infectious diseases [36]. In the SEIR model, the entire population is divided into four distinct groups of individuals. The first group is created by susceptible individuals (marked by $S,x_1$) who are able to catch the disease, the second group contains exposed persons (marked by $E,x_2$), who are infected but not yet infectious, infective people are in the third group ($I,x_3$), they are able to transmit the disease, and the last group contains recovered individuals (marked by $R,x_4$), who are permanently immune. The whole population size is $N$, while the letters S, E, I, R sign the number of individuals in each group at a particular time, so $S+E+I+R=N.$ We suppose that the population size $N$ is constant and that the populations are well mixed (the spatial mobility of individuals has been ignored in the model) [37]. The evolution of the groups of the population over time in the SEIR model is described by the following four non-linear partial differential equations:

$$\frac{\partial x_i(p,t)}{\partial t}=D_i\frac{{\partial }^2{x}_i(p,t)}{\partial p^2}+F_i\left(x(p,t),x(p,t-\tau ),u(p,t),t\right),$$

(17)

where $F_i,i=1,\dots \mathrm{,4}$ are given by Equation (20) with Neumann boundary condition

$$\frac{\partial x_i}{\partial p}(0,t)=\frac{\partial x_i}{\partial p}(1,t)=0$$

(18)

and initial conditions

$$x_i\left(p\mathrm{,0}\right)={\varphi }_i\left(p\right)\ge 0, 0\le p\le 1, i=1,\dots \mathrm{,4}$$

(19)

and

$$F_1\left(x(p,t),u(p,t),,t\right)=\mu \left(N(t)-x_1(p,t)\right)-\beta \frac{x_3(p,t-\tau )}{N(t)}{x}_1(p,t)-u(p,t)x_1(p,t)$$

$$F_2\left(x(p,t),u(p,t),,t\right)=\beta \frac{x_3(p,t-\tau )}{N(t)}{x}_1(p,t)-(\mu +\gamma )x_2(p,t)$$

(20)

$$F_3\left(x(p,t),u(p,t),,t\right)=\gamma x_2(p,t)-(\mu +\delta )x_3(p,t)$$

$$F_4\left(x(p,t),u(p,t),,t\right)=\delta x_3(p,t)-\mu x_4(p,t)+u(p,t)x_1(p,t)$$

Here, $t$ denotes the time, $p$ represents the spatial location, $D_i$ are the diffusion coefficients, $\mu$ is both birth and death rate, $\beta$ is the contact rate that indicates the probability of catching the disease between susceptible and infectious individuals with time delay $\tau$, $1/\delta$ is the mean latent period of the disease, $1/\gamma$ is the mean infectious period and $u(p,t)$ is the vaccination coverage of susceptible individuals. We set $N(t)={\displaystyle \underset{0}{\overset{1}{\int }}}\left({\displaystyle \sum _{i=1}^4}{x}_i(p,t)\right)dp$ for all $t\ge 0$, where $x(\varphi ,t,p)=\left(x_1(\varphi ,t,p),\dots ,x_4(\varphi ,t,p)\right)$ is the solution of Equation (17).

It follows from the model that

$$\begin{gathered} \frac{dN(t)}{dt}=\frac{d}{dt}{\displaystyle \underset{0}{\overset{1}{\int }}}\left({\displaystyle \sum _{i=1}^4}{x}_i(p,t)\right)dp={\displaystyle \underset{0}{\overset{1}{\int }}}\frac{d}{dt}\left({\displaystyle \sum _{i=1}^4}{x}_i(p,t)\right)dp={\displaystyle \underset{0}{\overset{1}{\int }}}\left({\displaystyle \sum _{i=1}^4}\frac{d}{dt}{x}_i(p,t)\right)dp= \\ {\displaystyle \sum _{i=1}^4}\left({\displaystyle \underset{0}{\overset{1}{\int }}}{D}_i\frac{{\partial }^2{x}_i(p,t)}{\partial p^2}dp\right)={\displaystyle \sum _{i=1}^4}{D}_i\left(\frac{\partial x_i}{\partial p}(1,t)-\frac{\partial x_i}{\partial p}(0,t)\right)=0. \end{gathered}$$

It follows easily that

$$N(t)={\displaystyle \underset{0}{\overset{1}{\int }}}\left({\displaystyle \sum _{i=1}^4}{x}_i(p,t)\right)dp=c$$

(21)

and $c={\displaystyle \underset{0}{\overset{1}{\int }}}\left({\displaystyle \sum _{i=1}^4}{x}_i(p,0)\right)dp,$ i.e., that the population size $N(t)$ is constant for all $t.$

Distributed Optimal Control Problem of Spread of Diseases

To control the spread of diseases, we introduce an optimal control in the SEIR model. The SEIR models with optimal control are currently being intensively studied, see [38,39,40]. Our objective is to find the optimal vaccine coverage threshold $u(p,t)$ for disease control and liquidation [41].

Our optimal control problem is to find a distributed control $u(p,t)$ to minimise the following objective function:

$$\mathcal{J}(u)={\displaystyle {\int }_a^b}{\displaystyle {\int }_{t_0}^{t_f}}\left(a_1{x}_1(p,t)+a_2{x}_2(p,t)+a_3{x}_3(p,t)+\frac{1}{2}u{(p,t)}^2\right)dtdp,$$

(22)

subject to the state system (17)–(18), under constraints

$$c_1(x,u)=u_{min}-u\le 0$$

$$c_2(x,u)=u-u_{max}\le 0,$$

where $a_1,a_2,a_3$ are positive constants to keep a balance in the size of S, E and I.

The augmented Hamiltonian function for problem (22) is given by

$$\mathcal{H}(x,x_{\tau },u,\lambda )=a_1{x}_1(p,t)+a_2{x}_2(p,t)+a_3{x}_3(p,t)+$$

(23)

$$\frac{1}{2}u{(p,t)}^2+{\displaystyle \sum _{j=1}^4}{\lambda }_j{F}_j(x,x_{\tau },u)=\frac{1}{2}u{(p,t)}^2-u(p,t)x_1(p,t)({\lambda }_1(p,t)-{\lambda }_4(p,t))+G(p,t),$$

where $\lambda \in R^4$ is the adjoint variable. Let $(\widehat{x},\widehat{u})$ be an optimal solution for (22). Then the necessary optimality condition for (22) implies [24] that there exists a piecewise continuous and piecewise continuously differentiable adjoint function $\lambda :Q\to R^4$, satisfying

$$\frac{\partial \lambda }{\partial t}=D\frac{{\partial }^2\lambda }{\partial p^2}-\frac{\partial \mathcal{H}}{\partial x}\left(x,x_{\tau },u,\lambda \right)-{\chi }_{[t_0,t_f-\tau ]}\frac{\partial \mathcal{H}}{\partial x_{\tau }}(x_{+\tau },x,u,{\lambda }_{+\tau }),$$

(24)

$$\lambda (p,t_f)=0,\frac{\partial \lambda (a,t)}{\partial p}=\frac{\partial \lambda (b,t)}{\partial p}=0,$$

(25)

$$0=\frac{\partial \mathcal{H}}{\partial u}\left(x,x_{\tau }u,\lambda \right).$$

(26)

By the optimality condition (26), we have

$$\frac{\partial \mathcal{H}}{\partial u}(x,x_{\tau },u,\lambda )=u(p,t)-x_1(p,t)({\lambda }_1(p,t)-{\lambda }_4(p,t))=0.$$

Now, using the property of the control space $u\in \langle u_{min},u_{max}\rangle$, we get

$$\widehat{u}(p,t)=u_{min},({\lambda }_1(p,t)-{\lambda }_4(p,t))\le u_{min},$$

(27)

$$\widehat{u}(p,t)={\widehat{x}}_1(p,t)({\lambda }_1(p,t)-{\lambda }_4(p,t)),$$

$$u_{min}<x_1(p,t)({\lambda }_1(p,t)-{\lambda }_4(p,t))<u_{max},$$

$$\widehat{u}(p,t)=u_{max},({\lambda }_1(p,t)-{\lambda }_4(p,t))\ge u_{max}.$$

Discretisation of Equations (17) and (22) using Equations (12)–(14) and (9) leads to minimize

$$\mathcal{J}(u)=-h_s{h}_t{\displaystyle \sum _{(i,j)}}{a}_1{x}_1(p_i,t_j)+a_2{x}_2(p_i,t_j)+a_3{x}_3(p_i,t_j)+\frac{1}{2}u{(p_i,t_j)}^2$$

subject to

$$x_{i,j+1}=x_{ij}+h_{t}D\frac{x_{i-1,j}-2x_{ij}+x_{i+1,j}}{h_s^2}+h_t{F}_{ij},$$

$$x_{0j}=x_{1j},x_{Nj}=x_{N-1j},$$

$$x_{i\mathrm{,0}}={\varphi }_x(p_i,t_0),i=0,\dots ,N,$$

$$u_{ij}-m_{max}\le 0,m_{min}-u_{ij}\le 0,$$

$$i=0,\dots ,N,j=0,\dots ,M-1$$

and Equations (12)–(14). The vector function $F(x,u)=(a_1{x}_1(p,t)+a_2{x}_2(p,t)+a_3{x}_3(p,t)+\frac{1}{2}u{(p,t)}^2,F_1(x,u),\dots ,F_4(x,u))$ is given by right-hand side of Equation (17). Discrete form of optimality condition (14) is

$$\widehat{u}(p_i,t_j)=u_{min},({\lambda }_1(p_i,t_j)-{\lambda }_4(p_i,t_j))\le u_{min},$$

(28)

$$\widehat{u}(p_i,t_j)={\widehat{x}}_1(p_i,t_j)({\lambda }_1(p_i,t_j)-{\lambda }_4(p_i,t_j)),$$

$$u_{min}<x_1(p_i,t_j)({\lambda }_1(p_i,t_j)-{\lambda }_4(p_i,t_j))<u_{max},$$

$$\widehat{u}(p_i,t_j)=u_{max},({\lambda }_1(p_i,t_j)-{\lambda }_4(p_i,t_j))\ge u_{max}.$$

The solution of distributed optimal control problem (22) using the algorithm for solving the optimal control problem are displayed in Figure 1, Figure 2, Figure 3 and Figure 4.

In this simulation, we have chosen a critic network consisting of 4 subnetworks and an action network consisting of one subnetwork. Each subnetwork has 4–12–1 structure, where the triple of numbers indicates the number of neurons in the input layer, hidden layer, and output layer, respectively. During the adaptive critic synthesis, the proposed neural network fulfils the chosen convergence tolerance values, which leads to satisfactory simulation results.

We have plotted immune, susceptible individuals with and without control by considering values of parameters as $a_2=0.3,a_3=0.2,\tau =0.5.$ We can see that the population of immune individuals with control and without control increasing and decreasing to its stable state, respectively, see Figure 2. In the case of susceptible individuals, we get the opposite trend; see Figure 3.

6. Conclusions

In this paper, we have presented a new algorithm of adaptive critic method for parabolic distributed optimal control problem. The method is based on feed-forward neural network approximation of co-state variables, where state variable $x$ and co-state variable $\lambda$ are solved forward in time and are able to simulate optimal feedback law. The new algorithm was applied to the SEIR model to minimize the infected and susceptible populations and to maximise recovered populations. The results of numerical simulation (using Matlab) have shown that our algorithm for solving a distributed optimal control problem is able to solve this practical problem successfully. We presented a comparison between optimal control and no control of the SEIR model (see Figure 2, Figure 3 and Figure 4). It is easily seen that the optimal control of the SEIR model is able to find the effective vaccination rate of the population to influence the population infection by decreasing the number of susceptible individuals.