Application of Migrating Optimization Algorithms in Problems of Optimal Control of Discrete-Time Stochastic Dynamical Systems

Abstract

The problem of finding the optimal open-loop control for discrete-time stochastic dynamical systems is considered. It is assumed that the initial conditions and external influences are random. The average value of the Bolza functional defined on individual trajectories is minimized. It is proposed to solve the problem by means of classical and modified migrating optimization algorithms. The modification of the migrating algorithm consists of cloning the members of the initial population and choosing different strategies of migratory behavior for the main population and for populations formed by clones. At the final stage of the search for an extremum, an intensively clarifying migration cycle is implemented with the participation of three leaders of the populations participating in the search process. Problems of optimal control of bundles of trajectories of deterministic discrete dynamical systems, as well as individual trajectories, are considered as special cases. Seven model examples illustrating the performance of the proposed approach are solved.

1. Introduction

Discrete system control problems have found wide application in solving engineering and economic problems [1,2,3,4]. For deterministic problems, as a rule, the discrete maximum principle or more general necessary conditions are used to search for optimal open-loop control [5] and the Bellman equation for the synthesis of systems with full feedback [1,6,7,8]. When solving problems with non-separable quality criteria, methodological problems arise in their use. As the dimension of the problem being solved increases, the computational difficulties can become insurmountable.

To solve stochastic problems in which the initial conditions and external influences are random, when searching for the optimal open-loop control, the stochastic discrete maximum principle [9,10,11] is used. To find the optimal control with full feedback on the state vector, the Bellman equation for discrete stochastic systems is usually applied [2,6,7]. Compared to deterministic problems, the complexity of the numerical solution increases significantly. Therefore, the direct search for optimal control for the described class of stochastic problems using various methods of global optimization, including metaheuristic algorithms, is relevant.

A special case of stochastic problems is the problem of optimal control of bundles of trajectories, in which it is assumed that there are no external influences on the system and the initial conditions belong to a given set of possible states. As a rule, the law of distribution of the initial states of the system is assumed to be uniform, which corresponds to the case of uncertainty in setting the initial conditions. Various necessary and sufficient conditions for the optimality of the control of bundles of trajectories of discrete systems, as well as computational procedures for their implementation, are considered in [12,13,14]. The problem of finding the optimal open-loop control of a separate trajectory of a discrete deterministic system starting from a given initial state can, in turn, be considered a special case of controlling bundles of trajectories.

Examples of solving discrete deterministic optimal control problems using various metaheuristic optimization algorithms, such as evolutionary methods, the Luus-Jaakola method, iterative dynamic programming, and methods that simulate the behavior of the tomtit flock and a flock of river perch, are described in [15,16,17,18]. Migrating optimization algorithms that use the processes of solution population evolution have been used in solving various applied problems [19,20]. The development of migrating algorithms was a modified migrating algorithm, which has proven itself to solve finite-dimensional technical problems [21].

The article proposes applying a modified migrating algorithm to solve the problem of finding the optimal open-loop control of nonlinear discrete stochastic systems. As special cases, we consider the solution of control problems for bundles of trajectories in the presence of uncertainty in setting the initial conditions and individual trajectories under a fixed initial condition.

The purpose of the article is to demonstrate the possibility of using migration metaheuristic optimization algorithms, including new ones, to solve a wide class of problems of finding optimal control of discrete dynamic systems. The solution procedure does not require the application of necessary optimality conditions and numerical methods for satisfying them. It is shown that the stated problems can be solved with relatively low computational costs. The completed research opens up the possibility of using a similar approach to solve more complex problems, namely the problems of finding control laws with complete and incomplete feedback.

The article has the following structure: In Section 2, the formulations of three optimal control problems for discrete dynamical systems are given. A description of the basic self-organizing migrating optimization algorithm and the modified one developed by the authors is presented. Section 3 contains solutions to seven model examples, two for each of the problem statements given in Section 2. Finally, conclusions are summarized in Section 4.

2. Materials and Methods

2.1. Statements of Optimal Control Problems

Problem 1

(average optimal open-loop control of a stochastic system). The behavior of a nonlinear discrete stochastic model of the control object is described using the following equation:

$$X(t+1)=f(t,X(t),u(t),W(t)),t=0,1,\dots ,N-1,$$

(1)

where $t$ is a discrete time, $t\in T=\{0,1,\dots ,N\}$ and the number $N$ is given; $X$ is a system state vector, $X\in R^n$; $u$ is a control vector; $u\in U\subseteq R^q;$ $U$ is the set of admissible control values, which is a direct product of segments $[a_i,b_i]$, $i=1,\dots ,q;$ $f(t,x,u,w)=\left(f_1(t,x,u,w)\right.,\dots ,{\left.f_n(t,x,u,w)\right)}^{\mathrm{T}}$ is a continuous vector function; and $W(t)$ is an $l$-dimensional random vector with a known probability density $p(W,t)$.

A special case of model (1) is the equation

$$X(t+1)=g(t,X(t),u(t))+\mathsf{\sigma }(t,X(t))W(t)),$$

in which a random external influence $W(t)$ on the control system enters additively; $g(t,x,u)$ is a vector function of dimensions $(n\times 1)$; and $\mathsf{\sigma }(t,x)$ is a matrix function of dimensions $(n\times l)$.

The initial conditions

$$X(0)=X_0$$

(2)

are given by the initial probability density $p_0(x)$.

It is assumed that only discrete-time information is used in the control law. The set $\mathbf{U}$ of admissible controls form sequences $u(·)=\{u(0),u(1),\dots ,u(N-1)\}$ such that $\forall t\in \{0,\dots ,N-1\}$ $u(t)\in U,$ and the function $f\left(t,x,u(t),w\right)$ is such that a solution to the system of Equation (1) with initial condition (2) exists and is unique.

Let us define the control quality functional for a separate trajectory in the Bolza form:

$$I\left(X_0,u(·)\right)={\displaystyle \sum _{t=0}^{N-1}{f}^0(t,X(t),u(t))}+F\left(X(N)\right)$$

(3)

where $f^0(t,x,u),F(x)$ are given continuous functions.

The quality of system control is proposed to be estimated by the value of the function

$$J\left[u(·)\right]=M[I\left(X_0,u(·)\right)],$$

(4)

where $M$ denotes the operation of finding the mathematical expectation.

One needs to find a control such that

$$J\left[u^{*}(·)\right]=\underset{u(·)\in \mathbf{U}}{\min }J\left[u(·)\right].$$

(5)

The desired control is called optimal on average since the average value of the functional (3) is minimized.

Problem 2

(average optimal control of a bundle of trajectories of a deterministic system). The behavior of the nonlinear deterministic discrete model of the control object is described by the difference equation as follows:

$$x(t+1)=f(t,x(t),u(t)),t=0,1,\dots ,N-1,$$

(6)

where $t$ is a discrete time, $x$ is a system state vector, $x\in R^n$; $u$ is a system control vector, $u\in U\subseteq R^q$; $U$ is the set of admissible control values, which is a direct product of segments $[a_i,b_i]$, $i=1,\dots ,q$; $f(t,x,u)={(f_1(t,x,u),\dots ,f_n(t,x,u))}^{\mathrm{T}}$ is a continuous vector function; and the number of discrete-time steps is given as $N$.

The initial state is given by a compact set of positive measures:

$$x(0)=x_0\in \mathsf{\Omega }\subset R^n.$$

(7)

The set $\mathbf{U}$ of admissible controls form the sequences $u(·)=\{u(0),u(1),\dots ,u(N-1)\}$, where $u(t)\in U$ $\forall t\in \{0,1,\dots ,N-1\}$.

Let us define the control quality functional for a separate trajectory in the Bolza form:

$$I(x_0,u(·))={\displaystyle \sum _{t=0}^{N-1}{f}^0\left(t,x(t),u(t)\right)+F\left(x(N)\right)}$$

(8)

or

$$I(x_0,u(·))=F(x(0),\dots ,x(N);u(0),\dots ,u(N-1)),$$

(9)

where $f^0\left(t,x,u\right),$ $F\left(x\right),$ $F(x(·),u(·))$ are given continuous functions, trajectory $x(·)=\{x_0,x(1),\dots ,x(N)\}$, and control $u(·)=\{u(0),u(1),\dots ,u(N-1)\}.$ Function (9) corresponds to the case of non-separable criteria.

To every admissible control $u(·)\in \mathbf{U}$ and set $\mathsf{\Omega }$, let us set in correspondence the bundle of trajectories of the system of Equation (6):

$$X\left(t,u(·)\right)=\cup \left\{x\left(t,u(·),x_0\right)|x_0\in \mathsf{\Omega }\right\},$$

(10)

i.e., the union of solutions $x\left(t,u(·),x_0\right)$ of Equation (6) over all possible initial states (7).

The quality of control of a bundle of trajectories is proposed to be estimated by the value of the functional

$$J[u(·)]={\displaystyle \underset{\mathsf{\Omega }}{\overset{}{\int }}I(x_0,u(·))}d{x}_0/\mathrm{mes}\mathsf{\Omega },$$

(11)

where $\mathrm{mes}\mathsf{\Omega }$ is a set $\mathsf{\Omega }$ measure.

It is required to find a control $u^{*}(·)\in \mathbf{U}$ such that

$$J\left[u^{*}(·)\right]=\underset{u(·)\in \mathbf{U}}{\min }J\left[u(·)\right].$$

(12)

The desired control is called optimal on average, since the average value of the functional (8) or (9) is minimized on the given set of initial states.

It is further assumed that set $\mathsf{\Omega }$ of initial states is a parallelepiped defined by the direct product of the segments $\left[{\mathsf{\alpha }}_i;{\mathsf{\beta }}_i\right],i=1,\dots ,n,$ i.e., $\mathsf{\Omega }=\left[{\mathsf{\alpha }}_1;{\mathsf{\beta }}_1\right]\times \cdots \times \left[{\mathsf{\alpha }}_n;{\mathsf{\beta }}_n\right]$. All segments with a step $\Delta x_i$ are divided into $K_i$ segments, and parallelepiped $\mathsf{\Omega }$ is divided into $K=K_1\cdots K_n$ elementary subsets ${\mathsf{\Omega }}_k,k=1,\dots ,K$. In each elementary subset ${\mathsf{\Omega }}_k$ the initial state $x_0{}^k$ (center of the parallelepiped) is set. These initial states and the applied control generate $K$ trajectories on which the value of the functional (8) or (9) is calculated. After averaging over all realizations, the value of functional (11) is found.

Problem 3

(optimal control of a separate trajectory of a dynamical deterministic system). The behavior of the nonlinear deterministic discrete model of the control object is described by the difference equation

$$x(t+1)=f(t,x(t),u(t)),t=\{0,1,\dots ,N-1\},$$

(13)

where $t$ is a discrete time, $x$ is a system state vector, $x\in R^n$; $u$ is a system control vector, $u\in U\subseteq R^q$; $U$ is the set of admissible control values, which is a direct product of segments $[a_i,b_i]$, $i=1,\dots ,q$; $f(t,x,u)={(f_1(t,x,u),\dots ,f_n(t,x,u))}^{\mathrm{T}}$ is a continuous vector function; and a number of discrete-time steps is given as $N$.

The initial state is given as follows:

$$x(0)=x_0.$$

(14)

The set of admissible processes $\mathit{D}(0,x_0)$ is a set of pairs $d=(x(·),u(·))$, including the trajectory $x(·)=\{x_0,x(1),\dots ,x(N)\}$ and admissible control $u(·)=\{u(0),u(1),\dots ,u(N-1)\}$, where $u(t)\in U$, satisfying the equation of state (13) and the initial condition (14).

On the set $\mathit{D}(0,x_0)$, let us define the control quality functional for a separate trajectory in the Bolza form as follows:

$$I(d)={\displaystyle \sum _{k=0}^{N-1}{f}^0\left(k,x(k),u(k)\right)+F\left(x(N)\right)}$$

(15)

or

$$I(d)=F(x(0),\dots ,x(N);u(0),\dots ,u(N-1)),$$

(16)

where $f^0(t,x,u),F(x)$, $F(x(·),u(·))$ are given continuous functions.

We need to find a pair $d^{*}=\left(x^{*}(·),u^{*}(·)\right)\in \mathit{D}(0,x_0)$ such that

$$I(d^{*})=\underset{d\in \mathit{D}(0,x_0)}{\min }I(d).$$

(17)

Elements of a pair $d^{*}$, i.e., desired trajectory $x^{*}(·)=\{x_0,x^{*}(1),\dots ,x^{*}(N)\}$ and control $u^{*}(·)=\{u^{*}(0),u^{*}(1),\dots ,u^{*}(N-1)\}$ are called the optimal trajectory and optimal control, respectively.

If any admissible control $u(·)$ match single pair $d$, then problem (17) can be rewritten in an equivalent form as the problem of finding the minimum of the functional on the set of admissible controls. Functional (16) is more general than the Bolza functional (15), and its application is typical for so-called non-separable problems.

2.2. Self-Organizing Migrating Optimization Algorithms

2.2.1. Optimization Problem Statement

Given an objective function $f(x)=f(x_1,x_2,\dots ,x_n),$ defined on the set of admissible solutions $D\subseteq R^n.$ It is required to find the constrained global minimum of the function $f(x)$ on the set $D,$ i.e., a point $x^{*}\in D,$ such that

$$f(x^{*})=\underset{x\in D}{\min }f(x),$$

where $x={(x_1,x_2,\dots ,x_n)}^T,$ $D=\left\{x\right|x_i\in [a_i,b_i],i=1,2,\dots ,n\}.$

2.2.2. Self-Organizing Migrating Algorithm (SOMA)

SOMA can be classified as the metaheuristic evolutionary optimization algorithm based on the self-organizing behavior of groups of individuals in a social environment. When solving the problem, finite sets $I=\left\{x^j={\left(x_1^j,x_2^j,\dots ,x_n^j\right)}^T,j=1,2,\dots ,NP\right\}\subset D$ of possible solutions, called populations, are used, where $x^j$ is an individual with number $j$, $NP$ is a population size.

The Self-Organizing Migrating Algorithm [19,20] simulates the evolution of the initial population $I_0=\{x^j,j=1,2,\dots ,NP|x^j={\left(x_1^j,x_2^j,\dots ,x_n^j\right)}^T\in D\}$ and is an iterative process that explores the set D.

Initially, a population is created from individuals with randomly generated coordinates $x_i$ from the segment $\left[a_i,b_i\right]$ using the uniform distribution law.

Then, the migration cycles begin. Before each cycle, a leader is identified, i.e., an individual with the best value of the objective function $f\left(x\right)$. In these cycles, all individuals sequentially move towards the leader along the straight line connecting them, including moving behind him for some distance along the same straight line. In this case, the leader remains motionless. Then, the position of the individual is selected, which corresponds to the best value of the objective function obtained in the search process, and it is recorded in the next population.

The algorithm continues until the difference between the values of the objective function corresponding to the leader and the worst individual of the population becomes less than a predetermined value, or until the maximum number of migration cycles is reached.

Algorithm SOMA parameters that must be specified are as follows: control parameter NStep, determining the number of steps until the end of the movement; control parameter PRT, determining whether the individual will move along the chosen coordinate to the leader; control parameter NP, determining the size of the population of individuals; stop parameter Migration, determining the maximum number of iterations; and stop parameter MinDiv, showing the allowable difference in the value of the objective function between the leader and the worst individual.

2.2.3. Modified Self-Organizing Migrating Algorithm

When developing a modified self-organizing migrating optimization algorithm (MSOMA) [21], the basic version of the SOMA algorithm [19,20] was used.

Modifications consist of the selection of three leaders among the individuals that form the population. For each of the members of the population, two clones are generated with the same position. Thus, three populations are generated, each of which implements a migration cycle relative to its leader. For all members of the population, the best positions achieved during the cycle are found. During the search, the population is regularly updated with new individuals generated on the set of feasible solutions. They replace outgoing individuals with the worst objective function values. After the termination conditions are met, a refinement search (migration cycle) is performed, in which the three remaining leaders of the population participate. The best result is presented as a solution.

When solving the problem, finite sets $I=\left\{x^j={\left(x_1^j,x_2^j,\dots ,x_n^j\right)}^T,j=1,2,\dots ,NP\right\}\subset D$ of possible solutions, called populations, are used, where $x^j$ is an individual with number j, $NP$ is a population size.

The Self-Organizing Migrating Algorithm simulates the evolution of the initial population $I_0=\{x^j,j=1,2,\dots ,NP|x^j={\left(x_1^j,x_2^j,\dots ,x_n^j\right)}^T\in D\}$ and is an iterative process that explores the set D. The initial population is created from individuals with randomly generated coordinates $x_i$ from the segment $\left[a_i,b_i\right],i=1,\dots ,n$. In this case, a uniform distribution law is used.

Further migration cycles are realized. Before each cycle, individuals are ordered, and three leaders are identified: individuals with the best values of the objective function $f\left(x\right)$. Each individual creates two individual clones with the same coordinates. Thus, three populations of total size $K=3NP$ are created. Each of the populations moves relative to one of the three leaders. In these cycles, all individuals sequentially, with a certain step, move in a straight line toward their leader and along the same straight line behind him for a certain distance. In this case, the leader remains motionless. Then, the position of the individual is selected, which corresponds to the best value of the objective function obtained in the search process. This position is written to the next population of size $K$. After all individuals are recorded in the new population, $2NP+\left[\frac{1}{3}NP\right]$ individuals with worse objective function values $f\left(x\right)$ are removed from it. Then, $\left[\frac{1}{3}NP\right]$ new individuals, according to the rules for the formation of the initial population, are generated to supplement the population to its original size NP in order to update the population and explore new areas in the set of feasible solutions.

The algorithm continues until the average deviation in the value of the function between the three leaders of the population becomes less than a predetermined value or until the maximum number of migration cycles is realized. Then, a clarifying migration cycle is implemented for the three leaders of the population relative to the absolute leader. As an approximate solution, the best position and the corresponding value of the objective function found during the search are presented.

• Algorithm MSOMA for solving the optimization problem

Step 1. Set the method parameters:

• Control parameter NStep, determining the number of steps until the end of the movement;
• Control parameter PRT, determining whether the individual will move along the chosen coordinate to the leader;
• Control parameter NP, determining the size of the population of individuals;
• Stop parameter Migration, determining the maximum number of iterations;
• Stop parameter MinDist, reflecting the value of the average deviation between the three leaders of the population in terms of the value of the function;
• Iteration counter MCount, necessary to stop the algorithm when it reaches the number Migration.

Set MCount = 1.

Step 2. Create an initial population.

Step 2.1. Create a population of $NP$ individuals with randomly generated coordinates $x_i$ from a segment $\left[a_i,b_i\right]$ using the uniform distribution law:

$$x_i^j=a_i+ran{d}_i\left[0,1\right]\left(b_i-a_i\right),i=1,\dots ,n;j=1,\dots ,NP.$$

Step 2.2. For each individual, calculate the objective function value and choose leaders (individuals with the best values). To do this, order the population in a non-decreasing order of the objective function. Then, select the first three individuals from it with the smallest values of the objective function.

Step 3. Realize a migration cycle regarding the three leaders.

Step 3.1. For all individuals, create two “clones”, i.e., the individuals with the same coordinates:

$$\mathrm{for}\mathrm{all}j=1,\dots ,NP:x_i^{Np+j}=x_i^j,x_i^{2Np+j}=x_i^j,i=1,\dots ,n.$$

In this case, the parent individual moves toward the first leader, the first clone moves towards the second leader, and the second clone moves toward the third leader.

Step 3.2. Before starting the movement of an individual to one of the leaders, generate a random number $rn{d}_i$ on the segment $[0,1]$ for each coordinate of the individual and then compare it with PRT:

$$\mathrm{if}rn{d}_i<PRT,\mathrm{then}PRTVecto{r}_i=1,\mathrm{else}0,i=1,\dots ,n.$$

Vector $PRTVector$ thus formed defines the set of coordinate directions along which the search will be performed.

Step 3.3. Implement the movement of individuals to the appropriate leader. The movement takes place in steps until the end position is reached, according to the following formula:

• for the first leader:

$$x^{j,m_1}=x^j+m_1\frac{\left(x^1-x^j\right)}{2NStep}\otimes PRTVector,m_1=0,1,\dots ,4NStep,j=1,\dots ,NP;$$

• for the second leader:

$$x^{j,m_2}=x^j+m_2\frac{\left(x^2-x^j\right)}{Nstep}\otimes PRTVector,m_2=0,1,\dots ,2NStep,j=\left(NP+1\right),\dots ,2NP;$$

• for the third leader:

$$x^{j,m_3}=x^j+m_3\frac{\left(x^3-x^j\right)}{\left[\frac{NStep}{2}\right]}\otimes PRTVector,m_3=0,1,\dots ,NStep,j=\left(2NP+1\right),\dots ,4NP,$$

where $\otimes$ is an element-wise product of vectors (according to Hadamard [22]), $\left[\frac{NStep}{2}\right]$ denotes the integer part of number $NStep/2$.

Step 3.4. After all the steps performed for each individual, find the best position (at which the objective function value was the smallest), and the individual to take this position, and then move to the next population:

• for the first leader:

$$x^{j,New}=\underset{m_1=0,1,\dots ,4NStep}{\mathrm{argmin}}f\left(x^{j,m_1}\right),j=1,\dots ,NP;$$

• for the second leader:

$$x^{j,New}=\underset{m_2=0,1,\dots ,2NStep}{\mathrm{argmin}}f\left(x^{j,m_2}\right),j=\left(Np+1\right),\dots ,2NP;$$

• for the third leader:

$$x^{j,New}=\underset{m_3=0,1,\dots ,NStep}{\mathrm{argmin}}f\left(x^{j,m_3}\right),j=\left(2NP+1\right),\dots ,3NP.$$

Step 3.5. Sort all individuals in non-decreasing order of the objective function value.

Step 4. Checking the conditions for completing the search.

If $\sqrt{\frac{1}{2}{\displaystyle \sum _{j=2}^3}{\left[f\left(x^j\right)-f\left(x^1\right)\right]}^2}<MinDist$ or if the migration limit (Migration) is reached, then go to step 6; otherwise, go to step 5.

Step 5. Update the current population.

Step 5.1. Sort all individuals in non-decreasing objective function value:

$$\{x^1,\dots ,x^{3NP}|f(x^k)\le f\left(x^{k+1}\right)\},k=1,\dots ,3NP-1.$$

Step 5.2. Delete last $2NP+\left[\frac{1}{3}NP\right]$ individuals.

Step 5.3. Create $\left[\frac{1}{3}NP\right]$ new individuals:

$$x_i^j=a_i+ran{d}_i\left[0,1\right]·\left(b_i-a_i\right),i=1,\dots ,n;j=\left[\frac{2}{3}NP\right]+2,\dots ,NP.$$

Step 5.4. Increment iteration counter: $MCount=MCount+1$ and go to step 3.

Step 6. Realize the intensively clarifying migration cycle.

Step 6.1. Increase parameter NStep ($NStep=10NStep$) and implement a migration cycle for the second and third leaders relative to the first leader. Before the individual moves towards the leader, generate a random number on the segment $[0,1]$ for each coordinate of the individual, and then compare with PRT:

$$\mathrm{if}rn{d}_i<PRT,\mathrm{then}PRTVecto{r}_i=1,\mathrm{else}0,i=1,\dots ,n.$$

Thus, to form $PRTVector$, defining the set of coordinate directions along which the search will be performed.

The movement of the second and third leaders is implemented according to the formula as follows:

$$x^{j,m}=x^j+m\frac{\left(x^1-x^j\right)}{\left[\frac{NStep}{2}\right]}\otimes PRTVector,m=0,1,\dots ,NStep;j=1,2,3.$$

Step 6.2. Find a new leader position:

$$x^{j,New}=\underset{m=0,1,\dots ,NStep}{\mathrm{argmin}}f\left(x^{j,m}\right),j=1,2,3.$$

Step 6.3. Return the best solution found during the search: $x^{ans}=\underset{j=1,2,3}{\mathrm{argmin}}f\left(x^j\right).$

The complete steps of the MSOMA algorithm are shown in the Algorithm 1.

Algorithm 1 MSOMA Algorithm

1: Initialize the algorithm parameters NStep, PRT, NP, Migration, MinDist, MCount 2: Set MCount = 1 3: Create an initial population: 4: for all $j=1,\dots ,NP$ do 5: $x_i^j=a_i+ran{d}_i\left[0,1\right]\left(b_i-a_i\right),$ $i=1,\dots ,n;$ 6: end for 7: for all $j=1,\dots ,NP$ calculate the objective function value 8: end for 9: Select the first three individuals with the smallest values of the objective function from the current population 10: Realize a migration cycles regarding the three leaders: 11: for all $j=1,\dots ,NP$ create two “clones” as 12: $x_i^{Np+j}=x_i^j,x_i^{2Np+j}=x_i^j,i=1,\dots ,n;$ 13: end for 14: Implement the movement of individuals to the first leader: 15: for all $j=1,\dots ,NP$ do 16: for all $i=1,\dots ,n$ do 17: generate a random number $rn{d}_i$ on the segment $[0,1]$; 18: $\mathrm{if}rn{d}_i<PRT,\mathrm{then}$ 19: $PRTVecto{r}_i=1$ 20: $\mathrm{else}$ 21: $PRTVecto{r}_i=0$ 22: end if 23: end for 24: for $m_1=0,1,\dots ,4NStep$ do 25: $x^{j,m_1}=x^j+m_1\frac{\left(x^1-x^j\right)}{2NStep}\otimes PRTVector$; 26: end for 27: Calculate $x^{j,New}=\underset{m_1=0,1,\dots ,4NStep}{\mathrm{argmin}}f\left(x^{j,m_1}\right)$ 28: end for 29: Implement the movement of individuals to the second leader: 30: for all $j=\left(NP+1\right),\dots ,2NP$ do 31: for all $i=1,\dots ,n$, do 32: generate a random number $rn{d}_i$ on the segment $[0,1]$; 33: $\mathrm{if}rn{d}_i<PRT,\mathrm{then}$ 34: $PRTVecto{r}_i=1$ 35: $\mathrm{else}$ 36: $PRTVecto{r}_i=0$ 37: end if 38: end for 39: for $m_2=0,1,\dots ,2NStep$ do 40: $x^{j,m_2}=x^j+m_2\frac{\left(x^2-x^j\right)}{Nstep}\otimes PRTVector$; 41: end for 42: Calculate $x^{j,New}=\underset{m_2=0,1,\dots ,2NStep}{\mathrm{argmin}}f\left(x^{j,m_2}\right)$ 43: end for 44: Implement the movement of individuals to the third leader: 45: for all $j=\left(2NP+1\right),\dots ,4NP$ do 46: for all $i=1,\dots ,n$, do 47: generate a random number $rn{d}_i$ on the segment $[0,1]$; 48: $\mathrm{if}rn{d}_i<PRT,\mathrm{then}$ 49: $PRTVecto{r}_i=1$ 50: $\mathrm{else}$ 51: $PRTVecto{r}_i=0$ 52: end if 53: end for 54: for $m_3=0,1,\dots ,NStep$ do 55: $x^{j,m_3}=x^j+m_3\frac{\left(x^3-x^j\right)}{\left[\frac{NStep}{2}\right]}\otimes PRTVector$; 56: end for 57: Calculate $x^{j,New}=\underset{m_3=0,1,\dots ,NStep}{\mathrm{argmin}}f\left(x^{j,m_3}\right)$ 58: end for 59: Sort all individuals in non-decreasing order of the objective function value 60: Check the conditions for completing the search 61: if $\sqrt{\frac{1}{2}{\displaystyle \sum _{j=2}^3}{\left[f\left(x^j\right)-f\left(x^1\right)\right]}^2}<MinDist$ ˅ Mcount = Migration, then 62: go to 75 63: else 64: go to 66 65: end if 66: Update the current population: 67: Sort all individuals in non-decreasing objective function value: $\{x^1,\dots ,x^{3NP}|f(x^k)\le f\left(x^{k+1}\right)\},k=1,\dots ,3NP-1$ 68: Delete last $2NP+\left[\frac{1}{3}NP\right]$ individuals and create $\left[\frac{1}{3}NP\right]$ new individuals: 69: for $j=\left[\frac{2}{3}NP\right]+2,\dots ,NP$ do 70: for $i=1,\dots ,n$ do 71: $x_i^j=a_i+ran{d}_i\left[0,1\right]·\left(b_i-a_i\right)$ 72: end for 73: end for 74: Set $MCount=MCount+1$ and go to 10 75: Realize the intensively clarifying migration cycle 76: Set $NStep=10·NStep$ 77: for $j=1,2,3$ do 78: for all $i=1,\dots ,n$, do 79: generate a random number $rn{d}_i$ on the segment $[0,1]$; 80: $\mathrm{if}rn{d}_i<PRT,\mathrm{then}$ 81: $PRTVecto{r}_i=1$ 82: $\mathrm{else}$ 83: $PRTVecto{r}_i=0$ 84: end if 85: end for 86: for $m=0,1,\dots ,NStep$ do 87: $x^{j,m}=x^j+m\frac{\left(x^1-x^j\right)}{\left[\frac{NStep}{2}\right]}\otimes PRTVector$; 88: end for 89: Calculate $x^{j,New}=\underset{m=0,1,\dots ,NStep}{\mathrm{argmin}}f\left(x^{j,m}\right)$ 90: end for 91: Return the best solution $x^{ans}=\underset{j=1,2,3}{\mathrm{argmin}}f\left(x^j\right)$

3. Model Examples

The solutions of all model examples are implemented using the Jupyter Notebook software in the Phyton language. Macbook Pro 16 gb RAM, and M1 pro processor with a frequency of 3.2 GHz were used.

3.1. Optimal Open-Loop Control of a Stochastic Discrete Systems

Example 1.

The behavior of the control object model is described by the difference equation:

$$X(t+1)=X(t)+u(t)+\mathsf{\sigma }W(t),$$

where $X\in R$,$t=0,1,\dots ,N-1;$ the control variable is constrained by $-2\times {10}^4\le u\le 0$ and $W(t)$ is the Gaussian random variable with zero mean and unit variance. We take the value of the parameter characterizing the intensity of external influences $\mathsf{\sigma }=20$. The initial states $X(0)$ are given by the uniform distribution law on the segment $[-200,200]$. Functional (3) has the form

$$I(X_0,u(·))=\frac{1}{2}{\displaystyle \sum _{t=0}^{N-1}{\mathsf{\gamma }}^{-t}{u}^2(t)+X(N),\mathsf{\gamma }>1.}$$

It is required to find a control minimizing the value of the functional (4).

Set the number of discrete-time steps: $N=50$, number of implementations $K=5$. SOMA parameters are given in

Table 1. SOMA parameters (Example 1).

Parameters
$NP$	$Nstep$	$PRT$	$Migrations$	$MinDist$
1500	100	0.4	100	${10}^{-5}$

. Figure 1 shows the obtained optimal control and the family of trajectories. The resulting value of the quality functional for $\mathsf{\gamma }=1.1$ is $\min J=-428.0443$; numerical results were obtained in 3173.59 s.

MSOMA parameters are given in

Table 2. MSOMA parameters (Example 1).

Parameters
$NP$	$Nstep$	$PRT$	$Migrations$	$MinDiv$
600	40	0.4	50	${10}^{-10}$

. The resulting value of the quality functional is $\min J=-593.329$; numerical results were obtained in 1783.98 s. Figure 2 shows the obtained optimal control and the set of trajectories.

Example 2.

The behavior of the control object model is described by a system of difference equations:

$$X_1(t+1)=X_2(t)+{\mathsf{\sigma }}_1{W}_1(t),$$

$$X_2(t+1)=2X_2(t)-X_1(t)+\frac{1}{N^2}u(t)+{\mathsf{\sigma }}_2{W}_2(t),$$

where $X\in R^2,u\in R$, $W_1(t),W_2(t)$ are the independent Gaussian random variables with zero mean and unit variance. We take the values of the parameters characterizing the intensity of external influences ${\mathsf{\sigma }}_1=0.02,{\mathsf{\sigma }}_2=0.03$. The initial states are described by the uniform distribution law on the set $[-2,2]\times [-2,2]$, the control variable is constrained by $0\le u\le 100$.

Functional (3) has the form

$$I(X_0,u(·))=-X_1(N)+\frac{1}{2N}{\displaystyle \sum _{t=0}^{N-1}{u}^2(t).}$$

It is required to find the optimal open-loop control and the minimum value of the functional (4).

Let us set $N=10$, number of implementations $K=$ 25. SOMA parameters are given in

Table 3. SOMA parameters (Example 2).

Parameters
$NP$	$Nstep$	$PRT$	$Migrations$	$MinDist$
900	70	0.3	100	${10}^{-5}$

. Figure 3 and Figure 4 show the resulting optimal control and trajectories. The resulting value of the quality functional is $\min J=-0.46146$; numerical results were obtained in 8504.02 s.

MSOMA parameters used are shown in

Table 4. MSOMA parameters (Example 2).

Parameters
$NP$	$Nstep$	$PRT$	$Migrations$	$MinDiv$
600	50	0.3	50	${10}^{-10}$

. Figure 5 and Figure 6 show the obtained optimal control and trajectories of the discrete dynamical system. The resulting value of the quality functional is $\min J=-0.52729$; numerical results were obtained in 14,270.38 s.

3.2. Optimal Open-Loop Control of Bundles of Trajectories

Example 3.

The behavior of the control object model is described by the difference equation

$$x(t+1)=x(t)+u(t),$$

where $x\in R$, $t=0,1,\dots ,N-1.$ The control variable is constrained by $-2\times {10}^4\le u\le 0$. The initial states are determined from the condition $x(0)\in \mathsf{\Omega }=[-200,200]$. Functional (8) has the form

$$I(x_0,u(·))=\frac{1}{2}{\displaystyle \sum _{t=0}^{N-1}{\mathsf{\gamma }}^{-t}{u}^2(t)+x(N),\mathsf{\gamma }>1.}$$

It is required to find a control minimizing the value of the functional (11).

Set the number of discrete-time steps $N=50$, number of implementations $K=5$.

SOMA parameters are given in

Table 5. SOMA parameters in Example 3.

Parameters
$NP$	$Nstep$	$PRT$	$Migrations$	$MinDist$
1000	100	0.3	70	0.001

. Figure 7 shows the resulting optimal control and bundle of trajectories. The resulting value of the quality functional for $\mathsf{\gamma }=1.1$ is $\min J=-581.8247$, numerical results were obtained in 520.52 s.

MSOMA parameters are given in

Table 6. MSOMA parameters in Example 3.

Parameters
$NP$	$Nstep$	$PRT$	$Migrations$	$MinDiv$
1000	100	0.3	150	0.001

. Figure 8 shows the resulting optimal control and bundle of trajectories. The resulting value of the quality functional for $\mathsf{\gamma }=1.1$ is $\min J=-581.9022$; numerical results were obtained in 730.24 s.

Example 4.

The behavior of the control object model is described by a system of difference equations:

$$x_1(t+1)=x_2(t),$$

$$x_2(t+1)=2x_2(t)-x_1(t)+\frac{1}{N^2}u(t),$$

where $x\in R^2,u\in R.$ The initial states are determined from the condition $x(0)\in \mathsf{\Omega }=[-2,2]\times [-2,2]$. The control variable is constrained by $0\le u\le 100$. Functional (8) has the form

$$I(x_0,u(·))=-x_1(N)+\frac{1}{2N}{\displaystyle \sum _{t=0}^{N-1}{u}^2(t).}$$

It is required to find the optimal program control and the minimum value of the functional (11).

Let us set $N=10$, number of implementations $K=K_1{K}_2=$ 25. The results of the SOMA algorithm (the parameters are given in Table 5). Figure 9 and Figure 10 show the resulting optimal control and corresponding bundle of trajectories. The resulting value of the quality functional: $\min J=-0.5424$; numerical results were obtained in 225.58 s.

MSOMA parameters are given in Table 6. The resulting value of the quality functional is $\min J=-0.5424$, numerical results were obtained in 868.34 s. Figure 11 and Figure 12 show the resulting optimal control and corresponding bundle of trajectories.

3.3. Optimal Open-Loop Control of Individual Trajectories

Example 5.

The behavior of the control object model is described by the difference equation

$$x(t+1)=x(t)+u(t),$$

where $x\in R$,$t=0,1,\dots ,N-1.$ The control variable is constrained by $-2\times {10}^4\le u\le 0$. The initial state is given by the condition $x(0)=0$.

On the set of admissible processes, the quality functional is defined as follows:

$$I=\frac{1}{2}{\displaystyle \sum _{t=0}^{N-1}{\mathsf{\gamma }}^{-t}{u}^2(t)+x(N),\mathsf{\gamma }>1.}$$

It is required to find the minimum value of the functional and the optimal process $(x^{*}(·),u^{*}(·)),$ at which this value is reached.

The problem under consideration has an analytical solution presented as: $x^{*}(t)=x_0+\frac{1-{\mathsf{\gamma }}^t}{\mathsf{\gamma }-1}$, $\min I=x_0+\frac{1-{\mathsf{\gamma }}^N}{2(\mathsf{\gamma }-1)}.$

Set the number of discrete-time steps: $N=50$. For $\mathsf{\gamma }=1.1$ the exact value of the quality functional is $I_{\min }=-581.95426$. SOMA parameters are given in Table 3. The resulting value of the quality functional is $\min I=-581.32586$, numerical results were obtained in 2.2 min. Figure 13 shows the resulting optimal control and corresponding trajectory.

MSOMA parameters are given in Table 4. The resulting value of the quality functional is $\min I=-581.80435$; numerical results were obtained in 10.7 min. Figure 14 shows the resulting optimal control and corresponding trajectory.

Example 6.

The behavior of the control object model is described by a system of difference equations:

$$x_1(t+1)=x_2(t),$$

$$x_2(t+1)=2x_2(t)-x_1(t)+\frac{1}{N^2}u(t),$$

where $x\in R^2,u\in R$. The initial state is given by the condition $x(0)={(0,0)}^T$. The control variable is constrained by $0\le u\le 100$.

On the set of admissible processes, a functional of the Bolza form

$$I=-x_1(N)+\frac{1}{2N}{\displaystyle \sum _{t=0}^{N-1}{u}^2(t)}$$

is defined. It is required to find the minimum value of the functional and the optimal process on which this value is reached.

The problem has an analytical solution: $u^{*}(t)=\frac{N-t-1}{N}$, $\min I=-\frac{1}{3}+\frac{3N-1}{6N^2}+\frac{1}{2N^3}{\displaystyle \sum _{t=0}^{N-1}{t}^2}$. If one set $N=10$, then $\min I=-0.1425$.

SOMA parameters are given in

Table 7. SOMA parameters in Example 6.

Parameters
$NP$	$Nstep$	$PRT$	$Migrations$	$MinDist$
500	50	0.4	100	${10}^{-10}$

. The resulting value of the quality functional is $\min I=-0.14256$, numerical results were obtained in 26 s. Figure 15 and Figure 16 show the resulting optimal control and trajectory.

MSOMA parameters are given in

Table 8. MSOMA parameters in Example 6.

Parameters
$NP$	$Nstep$	$PRT$	$Migrations$	$MinDiv$
100	50	0.3	100	0.0001

. The resulting value of the quality functional is $\min I=-0.14236$; numerical results were obtained in 12.7 s. Figure 17 and Figure 18 show the resulting optimal control and trajectory.

Example 7.

The behavior of the control object model is described by a difference equation:

$$x(t+1)=x(t)+u(t),$$

where $x\in R,u\in R$. The initial state is given by the condition $x(0)=3$, the control variable is constrained by $-100\le u\le 100$.

On the set of admissible processes, a functional of the form

$$I={\displaystyle \sum _{t=0}^{N-1}[u^2(t)+x^2(t)]}$$

is defined. It is required to find the minimum value of the functional and the optimal process on which this value is reached.

The problem has an analytical solution. If one set $N=10,$ then $\min I=14.56231$.

MSOMA parameters are given in Table 8. The resulting value of the quality functional:$\min I=14.58935$; numerical results were obtained in 5.7 s. The obtained control and state values are presented in

Table 9. Optimal control and state values in Example 7.

$\mathit{t}$	0	1	2	3	4	5	6
$u^{*}(t)$	−1.83466	−0.68896	−0.33061	−0.10169	0.01892	−0.05302	−0.05551
$x^{*}(t)$	3.00000	1.16534	0.47638	0.14576	0.04407	0.06299	0.00997
t	7	8	9	10
$u^{*}(t)$	0.05713	−0.04223	−0.04906	—
$x^{*}(t)$	−0.04554	0.01159	−0.03064	−0.07970

. Figure 19 shows the resulting optimal control and trajectory.

4. Discussion

Analysis of the results of solving Examples 5 and 6 indicates the ability of migrating optimization algorithms SOMA and MSOMA to find the optimal control and the corresponding trajectory for deterministic discrete dynamical systems with sufficiently high accuracy, which follows from a comparison with the exact solutions obtained by applying the discrete maximum principle. Example 7 shows that there are problems in which the control values tend to zero over time, and the coordinate values of the state vector tend to be stationary values.

The results of solving Examples 3 and 4 demonstrate the capabilities of metaheuristic migrating algorithms to solve optimal control problems under conditions of uncertainty in specifying initial conditions. In this case, the found control laws generate a fairly regular behavior of the trajectories forming the bundle. The disadvantages of using open loop control of trajectory bundles include the inability to concentrate trajectories in the desired neighborhood, which is demonstrated in Figure 10 and Figure 12 in Example 4. The optimal control and graphs in Figure 7 and Figure 8 obtained as a result of solving Example 3 demonstrate that the uncertainty generated by specifying a set of possible initial conditions when using open loop control at least does not decrease over time.

The results of solving Examples 1 and 2, presented in Figure 1, Figure 2, Figure 3, Figure 4, Figure 5 and Figure 6, demonstrate the influence of random external influences on the type of trajectories of a dynamical system. Comparing the graphs in Figure 1 and Figure 2 with Figure 7 and Figure 8 (Examples 1 and 3), as well as the graphs in Figure 3, Figure 4, Figure 5 and Figure 6 and Figure 9, Figure 10, Figure 11 and Figure 12 (Examples 2 and 4), one can conclude that the set of obtained trajectories of stochastic systems compared with a similar set of trajectories of deterministic systems are characterized by noticeable turbulence. At the same time, the patterns of behavior of controlled systems generally correspond to each other.

5. Conclusions and Future Work

The article demonstrates the application of metaheuristic migrating optimization algorithms to solving optimal control problems for discrete dynamical systems of different classes. Their use makes it possible to obtain a solution of acceptable quality at reasonable computational costs. It is noted that the proposed modified algorithm solves the problem of finding the optimal control of stochastic systems more successfully than the basic one. In problems of optimal control of bundles of trajectories and an individual trajectory, migrating algorithms are almost as accurate as each other.

Noting the shortcomings of the proposed approach, it should be emphasized that with a significant increase in the value of N, the complexity of the problems being solved also increases because of the number of variables on which the value of the functionals (4), (11), (15) and (16) depends increases. This fact leads to an increase in computational costs, which may be unjustified. One direction of development may be the search for the desired control law in parametric form with a relatively small number of characteristic parameters.

A further direction of research can be the search for an optimal control that depends on time and a set of coordinates of the state vector available for measurement, as well as solving problems of joint estimation and control. In this case, it is possible to use other metaheuristic optimization algorithms, namely methods of swarm intelligence, bio-inspired, multi-agent, and multi-start, as well as simulating physical and chemical processes.