A Multi-Population Mean-Field Game Approach for Large-Scale Agents Cooperative Attack-Defense Evolution in High-Dimensional Environments ^†

Abstract

As one of the important issues of multi-agent collaboration, the large-scale agents’ cooperative attack–defense evolution requires a large number of agents to make stress-effective strategies to achieve their goals in complex environments. Multi-agent attack and defense in high-dimensional environments (3D obstacle scenarios) present the challenge of being able to accurately control high-dimensional state quantities. Moreover, the large scale makes the dynamic interactions in the attack and defense problems increase dramatically, which, using traditional optimal control techniques, can cause a dimensional explosion. How to model and solve the cooperative attack–defense evolution problem of large-scale agents in high-dimensional environments have become a challenge. We jointly considered energy consumption, inter-group attack and defense, intra-group collision avoidance, and obstacle avoidance in their cost functions. Meanwhile, the high-dimensional state dynamics were used to describe the motion of agents under environmental interference. Then, we formulated the cooperative attack–defense evolution of large-scale agents in high-dimensional environments as a multi-population high-dimensional stochastic mean-field game (MPHD-MFG), which significantly reduced the communication frequency and computational complexity. We tractably solved the MPHD-MFG with a generative-adversarial-network (GAN)-based method using the MFGs’ underlying variational primal–dual structure. Based on our approach, we carried out an integrative experiment in which we analytically showed the fast convergence of our cooperative attack–defense evolution algorithm by the convergence of the Hamilton–Jacobi–Bellman equation’s residual errors. The experiment also showed that a large number of drones can avoid obstacles and smoothly evolve their attack and defense behaviors while minimizing their energy consumption. In addition, the comparison with the baseline methods showed that our approach is advanced.

1. Introduction

Cooperative control among multiple agent has always been an important research topic in swarm intelligence [1]. Game theory, as a branch of modern mathematics, studies optimal decision-making problems under conflicting adversarial conditions and can provide a theoretical basis for multi-agent cooperative decision-making problems. This paper is oriented toward the large-scale agents’ cooperative attack–defense evolution, one of the important issues of multi-agent collaboration, which requires a large number of agents to make stress-effective strategies to achieve their goals in complex environments [2]. The multi-player continuous attack–defense game, as the mainstream research method for multi-agent attack–defense evolution, is a differential game between two adversarial teams of cooperative players playing in an area with targets. The attacker attempts to reach the set destination. The goal of the defender is to delay or stop the attacker by catching it [3]. The attack–defense game problem can be described as an optimal decision-making problem under complex multi-constraint conditions [4].

The classic attack–defense games have long been studied in multi-agent cooperative control. The multiple-pursuer-one-evader problem has been well documented. In [5], the Voronoi diagram construct was used for the capture of an evader within a bounded domain. Based on differential game theory and optimal control theory, differential game models have been established to solve optimal strategies by setting some rules and assumptions. In [6], based on the geometric relationship between two pursuers and an evader, a differential game model was established through coordinate transformation to solve the optimal cooperative strategy. Paper [7] studied the optimal guidance law of two missiles intercepting a single target based on the hypothesis of the missile hit sequence. Reference [8] proposed an online decision-making technique based on deep reinforcement learning (DRL) for solving the environmental sensing and decision-making problems in the chase and escape games. For the case of N-pursuers and M-evaders, more complex interactions need to be analyzed. Paper [9] studied a multiplayer border-defense problem and extended classical differential game theory to simultaneously address weapon assignments and multiplayer pursuit–evasion scenarios. Papers [10,11] proposed some methods based on linear programming, and applied deep reinforcement learning methods to deal with a simplified version of the RoboFlag competition [4]. An approach to the task allocation of many agents was proposed in [12], where Bakolas and Tsiotras used the Voronoi diagram construct to control the system. To solve general attack–defense games, the Hamilton–Jacobi–Isaacs (HJI) approach is ideal when the game is low-dimensional. However, because its complexity increases exponentially with the number of agents, the HJI approach is only tractable for the two-player game [13]. However, the above methods cannot be directly applied to the large-scale attack–defense game we are concerned with, because these traditional optimization and control technologies deal with the dynamic interactions between individuals separately. Moreover, to conduct more accurate real-time control for agents, the state variables used to characterize their kinematics are usually high-dimensional. Thus, with the increase in the agents’ number, the modeling process of cooperative attack–defense problems tends to be complex, and the difficulty of solving the optimal strategy will increase significantly.

To solve the communication and calculation difficulties caused by agents’ interactions on a large scale, mean-field games (MFGs) were proposed by Lasry and Lions [14,15,16] and Huang, Malhame, and Caines [17,18,19] independently. MFGs have been widely used in industrial engineering [20,21,22], crowd motion [23,24,25,26], swarm robotics [27,28], epidemic modeling [29,30], and data science [31,32,33]. In the MFG, each agent can obtain the evolution of the global or macroscopic information (mean-field) by solving the Fokker–Planck–Kolmogorov (FPK) equation. The optimal strategy of each agent is found by solving the Hamilton–Jacobi–Bellman (HJB) equation [34]. Recently, a machine-learning-based method, named APAC-net, has been proposed to solve high-dimensional stochastic MFGs [35]. Intuitively, the large-scale attack–defense problems are more consistent with the multi-population model. The multi-population mean-field game is a critical subclass of mean-field games (MFGs). It is a theoretically feasible multi-agent model for simulating and analyzing the game between multiple heterogeneous populations of interacting massive agents. We proposed a numerical solution method (CA-Net) for multi-population high-dimensional stochastic MFGs in [36], and studied the large-scale attack–defense problem in a 3D blank scenario based on CA-Net in [37]. In this paper, we focus on a 3D obstacle scene. The presence of multiple obstacles in the 3D space brings qualitative changes to the attack and defense decisions of large-scale agents, i.e., a “diversion” phenomenon. How to model and solve the cooperative attack–defense evolution problem of large-scale agents in 3D obstacle scenarios has become a challenge.

Inspired by the above-mentioned cutting-edge works, in this paper, we jointly considered energy consumption, inter-group attack and defense, intra-group collision avoidance, and obstacle avoidance in their cost functions. Meanwhile, the high-dimensional state dynamics were used to describe the motion of agents under environmental interference. Then, we made the following main contributions:

• We formulated the cooperative attack–defense evolution of large-scale agents in high-dimensional environments as a multi-population high-dimensional stochastic mean-field game (MPHD-MFG), which significantly reduced the communication frequency and computational complexity.
• We propose ECA-Net, an extended nonlinear coupled alternating neural network composed of multiple generators and multiple discriminators. We tractably solved the MPHD-MFG with the ECA-Net algorithm using MFGs’ underlying variational primal–dual structure.
• We carried out an integrative experiment in which we analytically showed the fast convergence of our cooperative attack–defense evolution algorithm by the convergence of the Hamilton–Jacobi–Bellman equation’s residual errors. The experiment also showed that a large number of drones can avoid obstacles and smoothly evolve their attack and defense behaviors while minimizing their energy consumption. The comparison with the baseline methods showed that our approach is advanced.

Our approach accomplishes a breakthrough from few-to-few to mass-to-mass in terms of attack–defense game theory in 3D obstacle scenarios.

In Section 2, we model the cooperative attack–defense evolution of large-scale agents in a 3D obstacle scene and formulate it as an MPHD-MFG. In Section 3, we propose ECA-Net to tractably solve the MPHD-MFG. Section 4 shows the performance of our algorithm with numerical results, and in Section 5, we draw our conclusions.

2. Modeling and Formulating

The system model consists of the objective function and the kinematics equation, which describe how large-scale agents evolve paths through attack–defense relationships. We considered a 3D obstacle attack–defense scenario with N agents, as shown in Figure 1. The blue side ${\mathbb{N}}_1$ as the attacker with $N_1$ agents hopes to break through the red side’s interception and successfully reach the destination; the red side ${\mathbb{N}}_2$ as the defender with $N_2$ agents hopes to complete the interception against the blue side in the given area to prevent the blue side from penetrating. The state of agent i is denoted by ${\mathbf{x}}_i^p\left(t\right)\in {\mathbb{R}}^n$; $p=1$ represents the blue side; $p=2$ represents the red side, $i\in {\mathbb{N}}_1$ or ${\mathbb{N}}_2$, and $N_1+N_2=N$.

2.1. Kinematics Equation

Each agent $i\in \mathbb{N}={\mathbb{N}}_1\cup {\mathbb{N}}_2$ belongs to a population $p\left(i\right)\in \mathbb{Q}=\{1,2\}$ and is characterized by a state ${\mathbf{x}}_i^p\left(t\right)\in {\mathbb{R}}^n$ at time $t\in [0,T]$. We considered that the continuous-time state ${\mathbf{x}}_i^p\left(t\right)$ of player i of population p has the dynamic–kinematic equation of the following form

$$\mathrm{d}{\mathbf{x}}_i^p={\mathbf{h}}_i^p({\mathbf{x}}_i^p,{\mathbf{x}}_i^{p^{-}},{\mathbf{x}}_i^{-p^{-}},{\mathbf{u}}_i^p)\mathrm{d}t,$$

(1)

where ${\mathbf{u}}_i^p:[0,T]\to {\mathcal{U}}_p\subseteq {\mathbb{R}}^m$ is the control input (strategy) implemented by agent i in view of the state ${\mathbf{x}}_i^p$ at time t, ${\mathbf{x}}_i^{p^{-}}:{\mathbb{R}}^n\times [0,T]\to {\mathcal{P}}_2\left({\mathbb{R}}^n\right)$ is the states of all other intra-group agents, ${\mathbf{x}}_i^{-p^{-}}$ describes the states of all other inter-group agents, and ${\mathbf{h}}_i^p:{\mathbb{R}}^n\times {\mathcal{P}}_2{\left({\mathbb{R}}^n\right)}^2\times {\mathcal{U}}_p\to {\mathbb{R}}^n$ is the nonlinear evolution function, $p=1,2$.

2.2. Objective Function

For player i of population p, $p=1,2$, we considered the cost function of the following form:

$$\begin{array}{cc}\hfill J_i^p({\mathbf{x}}_i^p,{\mathbf{x}}_i^{p^{-}},{\mathbf{x}}_i^{-p^{-}},{\mathbf{u}}_i^p)={\int }_0^T{L}_i^p({\mathbf{x}}_i^p\left(t\right),{\mathbf{u}}_i^p\left(t\right))& +F_i^p({\mathbf{x}}_i^p\left(t\right),{\mathbf{x}}_i^{p^{-}}({\mathbf{x}}_i^p\left(t\right),t),{\mathbf{x}}_i^{-p^{-}}({\mathbf{x}}_i^{-p}\left(t\right),t))\mathrm{d}t\hfill \\ \hfill & +G_i^p({\mathbf{x}}_i^p\left(T\right),{\mathbf{x}}_i^{p^{-}}({\mathbf{x}}_i^p\left(T\right),T),{\mathbf{x}}_i^{-p^{-}}({\mathbf{x}}_i^{-p}\left(T\right),T)),\hfill \end{array}$$

(2)

where $L_i^p:{\mathbb{R}}^n\times {\mathcal{U}}_p\to \mathbb{R}$ is the running cost incurred by agent i based solely on its actions, $F_i^p:{\mathbb{R}}^n\times {\mathcal{P}}_2{\left({\mathbb{R}}^n\right)}^2\to \mathbb{R}$ is the running cost incurred by agent i based on its interactions with the rest of the same population and the other population, and $G_i^p:{\mathbb{R}}^n\times {\mathcal{P}}_2{\left({\mathbb{R}}^n\right)}^2\to \mathbb{R}$ is a terminal cost incurred by agent i based on its final state and the final states of the whole of the related populations.

2.2.1. Blue-Side Control Problem

In the 3D obstacle space, the blue side’s agents traveling from a common source hope to break through the red side’s interception while avoiding obstacles and successfully reach the destination, where a straight line segment in the area above the red side’s initial distribution is set as the destination. At time $t\ge 0$, the i-th agent controls its strategy ${\mathbf{u}}_i^1$ to minimize its: (1) motion energy, (2) red side capture, (3) blue side inter-agent collision, (4) collision of the blue side with the obstacles, and (5) travel time during the remaining travel to the destination. The running cost $L_i^1$ is given by

$$L_i^1=\underset{\left(1\right)\mathrm{motion}\mathrm{energy}\mathrm{minimization}}{\underbrace{c_1{\parallel {\mathbf{u}}_i^1\left(t\right)\parallel }^2}},$$

(3)

where $c_1$ is a constant. The running cost $L_i^1$ denotes the control effort implemented by agent i. The interaction cost $F_i^1$ is given by

$$\begin{array}{cc}\hfill F_i^1& =\underset{\left(2\right)\mathrm{red}\mathrm{side}\mathrm{capture}\mathrm{minimization}}{\underbrace{c_2\left(-\frac{1}{N_2}\sum _{k=1}^{N_2}∥{\mathbf{p}}_i^1\left(t\right)-({\mathbf{p}}_k^2\left(t\right)+e_r\mathbf{e})∥\right)}}+\underset{\left(3\right)\mathrm{blue}\mathrm{side}\mathrm{inter}-\mathrm{agent}\mathrm{collision}\mathrm{avoidance}}{\underbrace{c_3\left(\frac{1}{N_1}\sum _{j\ne i}^{N_1}{\mathbf{1}}_{\parallel {\mathbf{p}}_i^1\left(t\right)-{\mathbf{p}}_j^1\left(t\right)\parallel \le e_0}\right)}}\hfill \\ \hfill & +\underset{\left(4\right)\mathrm{blue}\mathrm{side}\mathrm{obstacle}\mathrm{collision}\mathrm{avoidance}}{\underbrace{c_4\left(\frac{1}{N_1}\sum _{i=1}^{N_1}\left\{\begin{array}{cc}{\sum }_{a=1}^A{\gamma }_{obs,a}\frac{1}{Q_a\left({\mathbf{p}}_i,t\right)}\hfill & \mathrm{if}{\mathbf{p}}_i\in {\mathsf{\Omega }}_{obs,trn}\hfill \\ 0\hfill & \mathrm{otherwise}\hfill \end{array}\right.\right)}},\hfill \end{array}$$

(4)

$${\mathsf{\Omega }}_{obs,trn}=\bigcup _{a=1}^A{\mathsf{\Omega }}_{obs,trn,a}$$

(5)

$${\mathsf{\Omega }}_{obs,trn,a}:Q_a(x,y,z)=\frac{1}{3v_{1,a}^2}{(x-x_{0,a})}^2+\frac{1}{3v_{2,a}^2}{(y-y_{0,a})}^2+\frac{1}{3v_{3,a}^2}{(z-z_{0,a})}^2\le 1.1,a=1,\dots ,A,$$

(6)

$${\mathsf{\Omega }}_{obs,a}=\left\{(x,y,z)\right||x-x_{0,a}|\le v_{1,a},|y-y_{0,a}|\le v_{2,a},|z-z_{0,a}|\le v_{3,a}\},a=1,\dots ,A.$$

(7)

where $\mathbf{p}=(x,y,z)$ is the agent’s usual Euclidean spatial coordinate position, $\mathbf{e}$ is the unit vector of the spherical coordinate system, $e_r$ is the capture radius of the red side agent, $e_0$ is the safe distance between agents, $c_2$, $c_3$, $c_4$ are constants, ${\gamma }_{obs,a}$, $a=1,\dots ,A$, is the repulsive force gain coefficient between the agent i and other obstacles, $(x_{0,a},y_{0,a},z_{0,a})$ is the center of the corresponding obstacle, and $(\pm v_{1,a},\pm v_{2,a},\pm v_{3,a})$ are its vertices, which are parallel with the $x,y,z$ axes, respectively. The interaction cost $F_i^1$ denotes the sum of the trajectory collision loss of the intra-group, inter-group, and with obstacles about agent i.

Remark 1.

In our 3D obstacle scene, obstacles were rectangular solids ${\mathsf{\Omega }}_{obs}$. For training, the cuboid obstacle repulsion is formulated as a differentiable unit ellipsoid repulsion function $\frac{1}{Q}$ [38]. To better reduce the collision between agents and obstacles, the radius of ${\mathsf{\Omega }}_{obs,trn}$ is 10% larger than that of the unit circumscribed ellipsoid of ${\mathsf{\Omega }}_{obs}$. Our algorithm was trained by using ellipsoidal repulsion, which can produce gradient information smoothly in obstacles, stimulating the model to learn trajectories to avoid obstacles [39].

The terminal cost $G_i^1$ is given by

$$G_i^1=\underset{\left(5\right)\mathrm{travel}\mathrm{time}\mathrm{minimization}}{\underbrace{c_5{\parallel {\mathbf{x}}_i^1\left(T\right)-{\mathbf{x}}_0\parallel }_2}},$$

(8)

where $c_5$ is a constant, ${\mathbf{x}}_i^1\left(T\right)$ is the final state of agent i, and ${\mathbf{x}}_0$ is the target state in which we want the agents to reach the objective. The terminal cost $G_i^1$ denotes the distance between agent i’s terminal state and the desired state.

In summary, by defining the cost function as (2), the optimal control problem faced by agent i of the blue side is given by

$$\begin{array}{cc}\hfill & \underset{{\mathbf{u}}_i^1\in {\mathcal{U}}_1}{inf}{J}_i^1({\mathbf{x}}_i^1,{\mathbf{x}}_i^{1^{-}},{\mathbf{x}}_i^{2^{-}},{\mathbf{u}}_i^1)\hfill \\ \hfill & \mathrm{s}.\mathrm{t}.\mathrm{d}{\mathbf{x}}_i^1={\mathbf{h}}_i^1({\mathbf{x}}_i^1,{\mathbf{x}}_i^{1^{-}},{\mathbf{x}}_i^{2^{-}},{\mathbf{u}}_i^1)\mathrm{d}t.\hfill \end{array}$$

(9)

2.2.2. Red Side Control Problem

The red side’s agents traveling from a common source avoid obstacles and hope to complete the interception against the blue side to prevent the blue side from penetrating. At time $t\ge 0$, the i-th agent controls its strategy ${\mathbf{u}}_i^2$, to minimize its: (1) motion energy, (2) distance to the blue side, (3) red side inter-agent collision, and (4) the collision of the red side with the obstacles during the remaining travel time. The running cost $L_i^2$ is given by

$$L_i^2=\underset{\left(1\right)\mathrm{motion}\mathrm{energy}\mathrm{minimization}}{\underbrace{l_1{\parallel {\mathbf{u}}_i^2\left(t\right)\parallel }^2}},$$

(10)

where $l_1$ is a constant. The running cost $L_i^2$ denotes the control effort implemented by agent i. The interaction cost $F_i^2$ is given by

$$\begin{array}{cc}\hfill F_i^2& =\underset{\left(2\right)\mathrm{distance}\mathrm{to}\mathrm{blue}\mathrm{side}\mathrm{minimization}}{\underbrace{l_2\left(\frac{1}{N_1}\sum _{k=1}^{N_1}∥{\mathbf{p}}_i^2\left(t\right)-{\mathbf{p}}_k^1\left(t\right)∥\right)}}+\underset{\left(3\right)\mathrm{red}\mathrm{side}\mathrm{inter}-\mathrm{agent}\mathrm{collision}\mathrm{avoidance}}{\underbrace{l_3\left(\frac{1}{N_2}\sum _{j\ne i}^{N_2}{\mathbf{1}}_{\parallel {\mathbf{p}}_i^2\left(t\right)-{\mathbf{p}}_j^2\left(t\right)\parallel \le e_0}\right)}}\hfill \\ \hfill & +\underset{\left(4\right)\mathrm{red}\mathrm{side}\mathrm{obstacle}\mathrm{collision}\mathrm{avoidance}}{\underbrace{l_4\left(\frac{1}{N_2}\sum _{i=1}^{N_2}\left\{\begin{array}{cc}{\sum }_{a=1}^A{\gamma }_{obs,a}\frac{1}{Q_a\left({\mathbf{p}}_i,t\right)}\hfill & \mathrm{if}{\mathbf{p}}_i\in {\mathsf{\Omega }}_{obs,trn}\hfill \\ 0\hfill & \mathrm{otherwise}\hfill \end{array}\right.\right)}},\hfill \end{array}$$

(11)

where $l_2$, $l_3$, $l_4$ are constants. The interaction cost $F_i^2$ denotes the sum of the trajectory collision loss of the intra-group, inter-group, and with obstacles about agent i.

In summary, by defining the cost function as (2), the optimal control problem faced by agent i of the red side is given by

$$\begin{array}{cc}\hfill & \underset{u_i^2\in {\mathcal{U}}_2}{inf}{J}_i^2({\mathbf{x}}_i^2,{\mathbf{x}}_i^{2^{-}},{\mathbf{x}}_i^{1^{-}},{\mathbf{u}}_i^2)\hfill \\ \hfill & \mathrm{s}.\mathrm{t}.\mathrm{d}{\mathbf{x}}_i^2={\mathbf{h}}_i^2({\mathbf{x}}_i^2,{\mathbf{x}}_i^{2^{-}},{\mathbf{x}}_i^{1^{-}},{\mathbf{u}}_i^2)\mathrm{d}t.\hfill \end{array}$$

(12)

To this end, the cooperative attack–defense evolution for large-scale agents in the 3D obstacle scene is formulated as a non-cooperative differential game. Thus, an N-player non-cooperative game is formed. Its obvious solution is the Nash equilibrium, that is no agent can unilaterally reduce its cost under this control decision [16].

2.3. Multi-Population High-Dimensional Mean-Field Game

With the increase of the number N of game participants, the complexity of solving differential games will increase significantly. For the current agent in the cooperative attack–defense problem (16), the neighborhood interactions of intra-group collision avoidance, the inter-group interactions of attack and defense, and the ecological interactions of obstacle avoidance will lead to the need for a large amount of communication and computing resources. Traditional optimization and control methods usually deal with the increasing interactions separately, leading to the dimension explosion problem. The mean-field game is able to overcome the communication and computational difficulties associated with a large scale, and its core technology is to inscribe a large number of interacting swarm intelligence problems as coupled sets of partial differential equations (PDEs). Therefore, we propose the multi-population high-dimensional stochastic MFG (MPHD-MFG) reformulation of the cooperative attack–defense problem (16). Under the framework of the MPHD-MFG, a generic player only reacts to the collective behaviors (mean field) of all players instead of the behavior of each player, which greatly reduces the amount of communication and computation. Here, “mean-field” means the states’ probability distribution. Now, we can drop the index i since players are indistinguishable within each population of the MPHD-MFG. Let ${\rho }^p({\mathbf{x}}^p,t):{\mathbb{R}}^n\times [0,T]\to {\mathcal{P}}_2\left({\mathbb{R}}^n\right)$ denote the probability density function of state ${\mathbf{x}}^p$ at time t, then the cost function (2) is transformed into

$$\begin{array}{cc}\hfill & J^p({\mathbf{x}}^p,{\rho }^p,{\rho }^{-p},{\mathbf{u}}^p)=\hfill \\ \hfill & \mathbb{E}\left[{\int }_0^T{L}^p({\mathbf{x}}^p\left(t\right),{\mathbf{u}}^p\left(t\right))\right.+F^p({\mathbf{x}}^p\left(t\right),{\rho }^p({\mathbf{x}}^p\left(t\right),t),{\rho }^{-p}({\mathbf{x}}^{-p}\left(t\right),t))\mathrm{d}t+\left.G^p({\mathbf{x}}^p\left(T\right),{\rho }^p({\mathbf{x}}^p\left(T\right),T),{\rho }^{-p}({\mathbf{x}}^{-p}\left(T\right),T))\right]\hfill \\ \hfill & ={\int }_0^T\left\{{\int }_{\mathsf{\Omega }}{L}^p({\mathbf{x}}^p\left(t\right),{\mathbf{u}}^p\left(t\right)){\rho }^p({\mathbf{x}}^p,t)\mathrm{d}\mathbf{x}\right\}\mathrm{d}t+{\int }_0^T\left\{{\int }_{\mathsf{\Omega }}{F}^p({\mathbf{x}}^p\left(t\right),{\rho }^p({\mathbf{x}}^p\left(t\right),t),{\rho }^{-p}({\mathbf{x}}^{-p}\left(t\right),t)){\rho }^p({\mathbf{x}}^p,t)\mathrm{d}\mathbf{x}\right\}\mathrm{d}t\hfill \\ \hfill & +{\int }_{\mathsf{\Omega }}{G}^p({\mathbf{x}}^p\left(T\right),{\rho }^p({\mathbf{x}}^p\left(T\right),T),{\rho }^{-p}({\mathbf{x}}^{-p}\left(T\right),T)){\rho }^p({\mathbf{x}}^p,t)\mathrm{d}\mathbf{x}\hfill \\ \hfill & ={\int }_0^T\left\{{\int }_{\mathsf{\Omega }}{L}^p({\mathbf{x}}^p\left(t\right),{\mathbf{u}}^p\left(t\right)){\rho }^p({\mathbf{x}}^p,t)\mathrm{d}\mathbf{x}\right\}\mathrm{d}t+{\int }_0^T{\mathcal{F}}^p({\mathbf{x}}^p\left(t\right),{\rho }^p({\mathbf{x}}^p\left(t\right),t),{\rho }^{-p}({\mathbf{x}}^{-p}\left(t\right),t))\mathrm{d}t\hfill \\ \hfill & +{\mathcal{G}}^p({\mathbf{x}}^p\left(T\right),{\rho }^p({\mathbf{x}}^p\left(T\right),T),{\rho }^{-p}({\mathbf{x}}^{-p}\left(T\right),T)),\hfill \end{array}$$

(13)

where $\mathsf{\Omega }\in {\mathbb{R}}^n$ is the state space containing all possible states of the generic agent and variational derivatives of functionals $\mathcal{F},\mathcal{G}$ for $\rho$ are the interaction and terminal costs F and G, respectively. Meanwhile, the state dynamic–kinematic equation in (1) is transformed into

$$\mathrm{d}{\mathbf{x}}^p={\mathbf{h}}^p({\mathbf{x}}^p,{\rho }^p,{\rho }^{-p},{\mathbf{u}}^p)\mathrm{d}t+{\sigma }^p\mathrm{d}{\mathbf{W}}_t^p,$$

(14)

where ${\sigma }^p\in {\mathbb{R}}^{n\times m}$ denotes a fixed coefficient matrix of a population-dependent volatility term and ${\mathbf{W}}^p$ means an m-dimensional Wiener process springs from the environment, in which each component ${\mathbf{W}}_k^p$ is independent of ${\mathbf{W}}_l^p$ for all $k\ne l$, $p=1,2$. According to Ito’s lemma [40], (14) can be expressed in terms of the mean field ${\rho }^p({\mathbf{x}}^p,t)$ and then will be equivalent to the Fokker–Planck (FPK) equation given by

$${\partial }_t{\rho }^p-\frac{{{\sigma }^p}^2}{2}\Delta {\rho }^p+\nabla ·\left({\rho }^p{\mathbf{h}}^p\right)=0.$$

(15)

With cost function (13) and FPK Equation (15), the multi-population high-dimensional MFG, which describes the cooperative attack–defense evolution of a large number of agents, is now summarized as

$$\begin{array}{cc}\hfill & \underset{{\rho }^p,{\mathbf{u}}^p}{inf}{J}^p({\mathbf{x}}^p,{\rho }^p,{\rho }^{-p},{\mathbf{u}}^p)\hfill \\ \hfill & \mathrm{s}.\mathrm{t}.{\partial }_t{\rho }^p-\frac{{{\sigma }^p}^2}{2}\Delta {\rho }^p+\nabla ·\left({\rho }^p{\mathbf{h}}^p\right)=0,{\rho }^p({\mathbf{x}}^p,0)={\rho }_0^p\left({\mathbf{x}}^p\right)\hfill \\ \hfill & {\partial }_t{\rho }^{-p}-\frac{{{\sigma }^{-p}}^2}{2}\Delta {\rho }^{-p}+\nabla ·\left({\rho }^{-p}{\mathbf{h}}^{-p}\right)=0,{\rho }^{-p}({\mathbf{x}}^{-p},0)={\rho }_0^{-p}\left({\mathbf{x}}^{-p}\right)\hfill \\ \hfill & p=1,2,\hfill \end{array}$$

(16)

where ${\rho }_0^p$ is the initial probability distribution of population p’s agents. To this end, the cooperative attack–defense evolution for large-scale agents in the 3D obstacle scene is formulated as a multi-population high-dimensional MFG. For every population $p=1,2$, each agent i of population $p\left(i\right)$ forecasts a distribution ${\left\{{\rho }^p(·,t)\right\}}_{t=0}^T$ and aims at minimizing its cost, which eventually reaches the Nash equilibrium, where no agent can decrease its individual cost by changing its control strategy unilaterally. The formulaic representation, for every ${\mathbf{x}}^p\in {\mathbb{R}}^n$:

$$J^p({\mathbf{x}}^p,{\rho }^p,{\rho }^{-p},\widehat{{\mathbf{u}}^p})\le J^p({\mathbf{x}}^p,{\rho }^p,{\rho }^{-p},{\mathbf{u}}^p),\forall {\mathbf{u}}^p:[0,T]\to {\mathcal{U}}_p,$$

(17)

where $\widehat{{\mathbf{u}}^p}$ is the agent’s equilibrium strategy at state ${\mathbf{x}}^p$. Here, we assumed that the agent is small, and its unilateral actions will not change the density ${\rho }^p$. Reference [41] provides a sufficient condition for the solution to the multi-population MFG PDEs, and Reference [42] provides the necessary one.

Remark 2.

Under appropriate assumptions, the MFG’s solution will offer an approximate Nash equilibrium (ϵ-NE) for the corresponding game with a large, but finite number of agents [41].

3. GAN-Based Approach for MPHD-MFG

In this section, we put forward a generative-adversarial-network (GAN)-based method for solving the multi-population high-dimensional MFG in (16). Inspired by Wasserstein GANs [43], APAC-Net [35], and CA-Net [36,37], we formulated (16) as a convex-concave saddle-point problem using MFG’s variational primal–dual structure. Then, we propose an extended coupled alternating neural network (ECA-Net) algorithm to solve the MPHD-MFG in (16).

3.1. Variational Primal–Dual Structure of MPHD-MFG

We reveal the underlying primal–dual structure of the MPHD-MFG, then deduce the convex–concave saddle-point problem equivalent to (16). Denote ${\Phi }_p:{\Phi }_p({\mathbf{x}}^p,t)={inf}_{{\mathbf{u}}^p}{J}^p$$({\mathbf{x}}^p,{\rho }^p,{\rho }^{-p},{\mathbf{u}}^p)$ as the Lagrange multiplier; we can add the differential constraint (15) (FPK equation) into the cost function (13) to obtain the extended cost function:

$$\begin{array}{cc}\hfill \underset{{\Phi }_p}{sup}\underset{{\rho }^p,{\mathbf{u}}^p}{inf}& \left\{{\int }_0^T{\int }_{\mathsf{\Omega }}{L}^p({\mathbf{x}}^p\left(t\right),{\mathbf{u}}^p\left(t\right)){\rho }^p({\mathbf{x}}^p,t)\mathrm{d}\mathbf{x}\mathrm{d}t\right.+{\int }_0^T{\mathcal{F}}^p({\mathbf{x}}^p\left(t\right),{\rho }^p({\mathbf{x}}^p\left(t\right),t),{\rho }^{-p}({\mathbf{x}}^{-p}\left(t\right),t))\mathrm{d}t\hfill \\ \hfill & +{\mathcal{G}}^p({\mathbf{x}}^p\left(T\right),{\rho }^p({\mathbf{x}}^p\left(T\right),T),{\rho }^{-p}({\mathbf{x}}^{-p}\left(T\right),T))\hfill \\ \hfill & -{\int }_0^T{\int }_{\mathsf{\Omega }}{\Phi }_p({\mathbf{x}}^p,t)({\partial }_t{\rho }^p-\frac{{{\sigma }^p}^2}{2}\Delta {\rho }^p+\nabla ·\left({\rho }^p({\mathbf{x}}^p,t){\mathbf{h}}^p({\mathbf{x}}^p,{\rho }^p,{\rho }^{-p},{\mathbf{u}}^p)\right))\mathrm{d}{\mathbf{x}}^p\mathrm{d}t\hfill \\ \hfill & -\left.{\int }_0^T{\int }_{\mathsf{\Omega }}{\Phi }_{-p}({\mathbf{x}}^{-p},t)({\partial }_t{\rho }^{-p}-\frac{{{\sigma }^{-p}}^2}{2}\Delta {\rho }^{-p}+\nabla ·({\rho }^{-p}({\mathbf{x}}^{-p},t){\mathbf{h}}^{-p}({\mathbf{x}}^{-p},{\rho }^{-p},{\rho }^p,{\mathbf{u}}^{-p})))\mathrm{d}{\mathbf{x}}^{-p}\mathrm{d}t\right\}.\hfill \end{array}$$

(18)

The Hamiltonian $H_p:{\mathbb{R}}^n\times {\mathcal{P}}_2{\left({\mathbb{R}}^n\right)}^2\times {\mathbb{R}}^n\to \mathbb{R}$, $p=1,2$ is defined as

$$H_p({\mathbf{x}}^p,{\rho }^p,{\rho }^{-p},z^p)=\underset{{\mathbf{u}}^p}{sup}\{-L_p({\mathbf{x}}^p,{\mathbf{u}}^p)-{z^p}^{\top }{\mathbf{h}}^p({\mathbf{x}}^p,{\rho }^p,{\rho }^{-p},{\mathbf{u}}^p)\}.$$

(19)

Utilizing (19) and integrating by parts, we can rewrite (18) as

$$\begin{array}{cc}\hfill \underset{{\rho }^p}{inf}\underset{{\Phi }_p}{sup}& \left\{{\int }_0^T{\int }_{\mathsf{\Omega }}({\partial }_t{\Phi }_p+\frac{{{\sigma }^p}^2}{2}\Delta {\Phi }_p-H_p({\mathbf{x}}^p,\nabla {\Phi }_p)){\rho }^p({\mathbf{x}}^p,t)\mathrm{d}{\mathbf{x}}^p\mathrm{d}t\right.+{\int }_0^T{\mathcal{F}}^p({\mathbf{x}}^p\left(t\right),{\rho }^p({\mathbf{x}}^p\left(t\right),t),{\rho }^{-p}({\mathbf{x}}^{-p}\left(t\right),t))\mathrm{d}t\hfill \\ \hfill & +{\mathcal{G}}^p({\mathbf{x}}^p\left(T\right),{\rho }^p({\mathbf{x}}^p\left(T\right),T),{\rho }^{-p}({\mathbf{x}}^{-p}\left(T\right),T))+{\int }_{\mathsf{\Omega }}{\Phi }_p({\mathbf{x}}^p,0){\rho }_0^p\left({\mathbf{x}}^p\right)\mathrm{d}{\mathbf{x}}^p-{\int }_{\mathsf{\Omega }}{\Phi }_p({\mathbf{x}}^p,T){\rho }^p({\mathbf{x}}^p,T)\mathrm{d}{\mathbf{x}}^p\hfill \\ \hfill & +{\int }_0^T{\int }_{\mathsf{\Omega }}({\partial }_t{\Phi }_{-p}+\frac{{{\sigma }^{-p}}^2}{2}\Delta {\Phi }_{-p}+\nabla {\Phi }_{-p}·{\mathbf{h}}^{-p}){\rho }^{-p}({\mathbf{x}}^{-p},t)\mathrm{d}{\mathbf{x}}^{-p}\mathrm{d}t+{\int }_{\mathsf{\Omega }}{\Phi }_{-p}({\mathbf{x}}^{-p},0){\rho }_0^{-p}\left({\mathbf{x}}^{-p}\right)\mathrm{d}{\mathbf{x}}^{-p}\hfill \\ \hfill & -\left.{\int }_{\mathsf{\Omega }}{\Phi }_{-p}({\mathbf{x}}^{-p},T){\rho }^{-p}({\mathbf{x}}^{-p},T)\mathrm{d}{\mathbf{x}}^{-p}\right\}.\hfill \end{array}$$

(20)

Here, our derivation path follows that of [44,45,46]. The formulation (20) is the cornerstone of our approach.

3.2. ECA-Net for Cooperative Attack–Defense Evolution

We solved (20) by training a GAN-based neural network. The solving network of the multi-population MFG in the 3D obstacle scene is an extended nonlinear coupled alternating neural network formed by multiple generators and multiple discriminators, named ECA-Net. We coupled the obstacle avoidance loss term in the design of the loss function of the ECA-Net algorithm. The structure and training process of ECA-Net are shown in Figure 2.

First, we initialized two pairs of neural networks $N_{{\omega }^p}^p({\mathbf{x}}^p,t)$ and $N_{{\theta }^p}^p({\mathbf{y}}^p,t)$, $p=1,2$. Let

$${\Phi }_{{\omega }^p}^p({\mathbf{x}}^p,t)=(1-t)N_{{\omega }^p}^p({\mathbf{x}}^p,t)+t{G}^p\left({\mathbf{x}}^p\right),{Ģ}_{{\theta }^p}^p({\mathbf{y}}^p,t)=(1-t){\mathbf{y}}^p+t{N}_{{\theta }^p}^p({\mathbf{y}}^p,t),$$

(21)

where ${\mathbf{y}}^p\sim {\rho }_0^p$ is a sample drawn from the initial distribution. The formulation of ${\Phi }_{{\omega }^p}^p$ (the value function for the generic agent of population p) and ${Ģ}_{{\theta }^p}^p$ (the density distribution of population p) in (21) automatically encodes the terminal condition $G^p(·,T)$ and initial distribution ${\rho }_0^p(·)$, respectively. Moreover, ECA-Net encodes the underlying structure of the MPHD-MFG by (20) and (21), exempting the neural network from learning the entire game solution from scratch.

Our approach for training this neural network includes parallel–alternate training of two pairs of ${Ģ}_{{\theta }^p}^p$ and ${\Phi }_{{\omega }^p}^p$, $p=1,2$. Intuitively, to gain the equilibrium of the MPHD-MFG of the cooperative attack–defense problem, we trained an extended coupled alternating neural network (ECA-Net) about multi-group distributions and agent controls. Specifically, we trained ${\Phi }_{{\omega }^p}^p$ by first sampling a batch ${\left\{{\mathbf{y}}_b^p\right\}}_{b=1}^B$ from given initial distribution ${\rho }_0^p$, another batch ${\left\{{\mathbf{y}}_b^{-p}\right\}}_{b=1}^B$ from given initial density ${\rho }_0^{-p}$, and ${\left\{t_b\right\}}_{b=1}^B$ uniformly from $[0,T]$. Then, we computed the push-forward states ${\mathbf{x}}_b^p={Ģ}_{{\theta }^p}^p({\mathbf{y}}_b^p,t_b)$, ${\mathbf{x}}_b^{-p}={Ģ}_{{\theta }^{-p}}^{-p}({\mathbf{y}}_b^{-p},t_b)$ for $b=1,\dots ,B$. The main loss term for training the discriminator ${\Phi }_{{\omega }^p}^p$ is given by

$${loss}_{{\Phi }^p}=\frac{1}{B}\sum _{b=1}^B{\Phi }_{{\omega }^p}^p({\mathbf{x}}_b^p,0)+\frac{1}{B}\sum _{b=1}^B\left\{{\partial }_t{\Phi }_{{\omega }^p}^p({\mathbf{x}}_b^p,t_b)+\frac{{{\sigma }^p}^2}{2}\Delta {\Phi }_{{\omega }^p}^p({\mathbf{x}}_b^p,t_b)-H_p\left({\nabla }_{{\mathbf{x}}^p}{\Phi }_{{\omega }^p}^p({\mathbf{x}}_b^p,t_b)\right)\right\}.$$

(22)

We need to consider adding a dual term for the distribution of the neighboring population:

$${\mathrm{penalty}}_{\mathrm{neighbors}}=\eta \left\{\frac{1}{B}\sum _{b=1}^B\left[{\partial }_t{\Phi }_{{\omega }^{-p}}^{-p}({\mathbf{x}}_b^{-p},t_b)+\frac{{{\sigma }^{-p}}^2}{2}\Delta {\Phi }_{{\omega }^{-p}}^{-p}({\mathbf{x}}_b^{-p},t_b)+{\nabla }_{{\mathbf{x}}^{-p}}{\Phi }_{{\omega }^{-p}}^{-p}({\mathbf{x}}_b^{-p},t_b)·{\mathbf{h}}^{-p}({\mathbf{x}}_b^{-p},{\mathbf{x}}_b^p,t_b)\right]\right\}$$

(23)

to correct for the density update of the other population (which tends to obey the FPK equation) [36]. We can optionally add a regularization term:

$$\begin{array}{cc}\hfill {\mathrm{penalty}}_{\mathrm{HJB}}& =\lambda \left\{\frac{1}{B}\sum _{b=1}^B\parallel {\partial }_t{\Phi }_{{\omega }^p}^p({\mathbf{x}}_b^p,t_b)+\frac{{{\sigma }^p}^2}{2}\Delta {\Phi }_{{\omega }^p}^p({\mathbf{x}}_b^p,t_b)-H_p\left({\nabla }_{{\mathbf{x}}^p}{\Phi }_{{\omega }^p}^p({\mathbf{x}}_b^p,t_b)\right)+F^p({\mathbf{x}}_b^p,{\mathbf{x}}_b^{-p},t_b)\parallel \right\}\hfill \end{array}$$

(24)

to penalize deviations from the HJB equations. Finally, we backpropagated the total loss ${\ell }_{\mathrm{total}}$ to update the weights of the discriminator ${\Phi }_{{\omega }^p}^p$.

To train the generator, we again sampled ${\left\{{\mathbf{y}}_b^p\right\}}_{b=1}^B$, ${\left\{{\mathbf{y}}_b^{-p}\right\}}_{b=1}^B$ and ${\left\{t_b\right\}}_{b=1}^B$, $p=1,2$ as before and computed

$${loss}_{{Ģ}^p}=\frac{1}{B}\sum _{b=1}^B\left\{{\partial }_t{\Phi }_{{\omega }^p}^p({Ģ}_{{\theta }^p}^p\left({\mathbf{y}}_b^p\right),t_b)+\frac{{{\sigma }^p}^2}{2}\Delta {\Phi }_{{\omega }^p}^p({Ģ}_{{\theta }^p}^p\left({\mathbf{y}}_b^p\right),t_b)-H_p\left({\nabla }_{{\mathbf{x}}^p}{\Phi }_{{\omega }^p}^p({Ģ}_{{\theta }^p}^p\left({\mathbf{y}}_b^p\right),t_b)\right)+F^p({Ģ}_{{\theta }^p}^p\left({\mathbf{y}}_b^p\right),{Ģ}_{{\theta }^{-p}}^{-p}\left({\mathbf{y}}_b^{-p}\right),t_b)\right\}.$$

(25)

Finally, we backpropagated the total loss ${\zeta }_{\mathrm{total}}$ to update the weights of the generator ${Ģ}_{{\theta }^p}^p$.

In conclusion, in each time slot $t\in [0,T]$, the ECA-Net will be trained. The generator ${Ģ}_{{\theta }^p}^p$ will generate the state distribution of population p at time t, and the discriminator ${\Phi }_{{\omega }^p}^p$ will obtain the result of the value function of population p at time t, $p=1,2$. Please refer to Algorithm 1 for the detailed operation flow.

Algorithm 1 ECA-Net for cooperative attack–defense evolution.

Require:${\sigma }^p$ diffusion parameter, $G^p$ terminal cost, $H_p$ Hamiltonian, $F^p$ interaction term, $p=1,2$. Require: Initialize neural networks $N_{{\omega }^p}^p$ and $N_{{\theta }^p}^p$, batch size B. Require: Set ${\Phi }_{{\omega }^p}^p$ and ${Ģ}_{{\theta }^p}^p$ as in (21). While not converged, do          train ${\Phi }_{{\omega }^p}^p$: Sample batch ${\left\{({\mathbf{y}}_b^p,t_b)\right\}}_{b=1}^B$, ${\left\{({\mathbf{y}}_b^{-p},t_b)\right\}}_{b=1}^B$, where ${\mathbf{y}}_b^p\sim {\rho }_0^p$, ${\mathbf{y}}_b^{-p}\sim {\rho }_0^{-p}$ and $t_b\sim Unif(0,T)$. ${\mathbf{x}}_b^p\leftarrow {Ģ}_{{\theta }^p}^p({\mathbf{y}}_b^p,t_b)$, ${\mathbf{x}}_b^{-p}\leftarrow {Ģ}_{{\theta }^{-p}}^{-p}({\mathbf{y}}_b^{-p},t_b)$ for $b=1,\dots ,B$. ${\ell }_0^p\leftarrow \frac{1}{B}{\sum }_{b=1}^B{\Phi }_{{\omega }^p}^p({\mathbf{x}}_b^p,0)$ ${\ell }_t^p\leftarrow \frac{1}{B}{\sum }_{b=1}^B\left\{{\partial }_t{\Phi }_{{\omega }^p}^p({\mathbf{x}}_b^p,t_b)+\frac{{{\sigma }^p}^2}{2}\Delta {\Phi }_{{\omega }^p}^p({\mathbf{x}}_b^p,t_b)-H_p\left({\nabla }_{{\mathbf{x}}^p}{\Phi }_{{\omega }^p}^p({\mathbf{x}}_b^p,t_b)\right)\right\}$ ${\ell }_{\mathrm{n}}^p\leftarrow \eta \left\{\frac{1}{B}{\sum }_{b=1}^B\left[{\partial }_t{\Phi }_{{\omega }^{-p}}^{-p}({\mathbf{x}}_b^{-p},t_b)+\frac{{{\sigma }^{-p}}^2}{2}\Delta {\Phi }_{{\omega }^{-p}}^{-p}({\mathbf{x}}_b^{-p},t_b)+{\nabla }_{{\mathbf{x}}^{-p}}{\Phi }_{{\omega }^{-p}}^{-p}({\mathbf{x}}_b^{-p},t_b)·{\mathbf{h}}^{-p}({\mathbf{x}}_b^{-p},{\mathbf{x}}_b^p,t_b)\right]\right\}$ ${\ell }_{\mathrm{HJB}}^p\leftarrow \lambda \left\{\frac{1}{B}{\sum }_{b=1}^B\parallel {\partial }_t{\Phi }_{{\omega }^p}^p({\mathbf{x}}_b^p,t_b)+\frac{{{\sigma }^p}^2}{2}\Delta {\Phi }_{{\omega }^p}^p({\mathbf{x}}_b^p,t_b)-H_p\left({\nabla }_{{\mathbf{x}}^p}{\Phi }_{{\omega }^p}^p({\mathbf{x}}_b^p,t_b)\right)+F^p({\mathbf{x}}_b^p,{\mathbf{x}}_b^{-p},t_b)\parallel \right\}$ ${\ell }_{\mathrm{total}}^p\leftarrow {\ell }_0^p+{\ell }_t^p+{\ell }_{\mathrm{n}}^p+{\ell }_{\mathrm{HJB}}^p$ Backpropagate total loss ${\ell }_{\mathrm{total}}={\sum }_p{\ell }_{\mathrm{total}}^p$ to $\omega ={\left({\omega }^p\right)}_{p=1,2}$ weights.          train ${Ģ}_{{\theta }^p}^p:$ Sample batch ${\left\{({\mathbf{y}}_b^p,t_b)\right\}}_{b=1}^B$, ${\left\{({\mathbf{y}}_b^{-p},t_b)\right\}}_{b=1}^B$, where ${\mathbf{y}}_b^p\sim {\rho }_0^p$, ${\mathbf{y}}_b^{-p}\sim {\rho }_0^{-p}$ and $t_b\sim Unif(0,T)$. ${\zeta }_t^p\leftarrow \frac{1}{B}{\sum }_{b=1}^B\left\{{\partial }_t{\Phi }_{{\omega }^p}^p({Ģ}_{{\theta }^p}^p\left({\mathbf{y}}_b^p\right),t_b)+\frac{{{\sigma }^p}^2}{2}\Delta {\Phi }_{{\omega }^p}^p({Ģ}_{{\theta }^p}^p\left({\mathbf{y}}_b^p\right),t_b)-H_p\left({\nabla }_{{\mathbf{x}}^p}{\Phi }_{{\omega }^p}^p({Ģ}_{{\theta }^p}^p\left({\mathbf{y}}_b^p\right),t_b)\right)\right.$ $\left.+F^p({Ģ}_{{\theta }^p}^p\left({\mathbf{y}}_b^p\right),{Ģ}_{{\theta }^{-p}}^{-p}\left({\mathbf{y}}_b^{-p}\right),t_b)\right\}$ Backpropagate total loss ${\zeta }_{\mathrm{total}}={\sum }_p{\zeta }_t^p$ to $\theta ={\left({\theta }^p\right)}_{p=1,2}$ weights. end while

4. Simulation Results

In this section, we carry out an integrative experiment based on Algorithm 1 and demonstrate the feasibility and effectiveness of our approach via the following numerical simulation results. To prove the advanced nature of our approach, we finally compare its performance with that of baseline methods.

4.1. Experimental Setup

For the attack–defense obstacle scenario in Figure 3, the initial density of the two populations is discrete uniform distributions ${\rho }_0^p$, $p=1,2$. The initial spatial coordinates $(x,y,z)$ of the blue and red sides are located in the cuboid areas $\left(\right(-5,5),(-9,-7),(-9,-7\left)\right)$ and $\left(\right(-5,5),(7,9),(-9,-7\left)\right)$, respectively. Note that we set all other initial coordinates to zero—initial angular position, initial velocity, and initial angular velocity were all set to zero. The coordinates of the terminal line were set to $(·,8,8)$. Two obstacles were placed between the attacking and defending groups. The obstacles were represented by cuboids, specified by the coordinates of their vertices. The first obstacle was placed at $\left(\right(-3,-1),(-2,2),(-10,10\left)\right)$, and the second obstacle was placed at $\left(\right(1,3),(-2,2),(-10,10\left)\right)$.

We examined with this high-dimensional scene where the dynamics was that of quadrotor crafts. The dynamic–kinematic equation of the generic quadrotor craft i of population p, $p=1,2$, is given by

$$\left\{\begin{array}{c}{\dot{x}}_p=v_{x_p}\hfill \\ {\dot{y}}_p=v_{y_p}\hfill \\ {\dot{z}}_p=v_{z_p}\hfill \\ {\dot{\psi }}_p=v_{{\psi }_p}\hfill \\ {\dot{\theta }}_p=v_{{\theta }_p}\hfill \\ {\dot{\varphi }}_p=v_{{\varphi }_p}\hfill \\ {\dot{v}}_{x_p}=\frac{u_p}{m_p}(sin\left({\varphi }_p\right)sin\left({\psi }_p\right)+cos\left({\varphi }_p\right)cos\left({\psi }_p\right)sin\left({\theta }_p\right))\hfill \\ {\dot{v}}_{y_p}=\frac{u_p}{m_p}(-cos\left({\psi }_p\right)sin\left({\varphi }_p\right)+cos\left({\varphi }_p\right)sin\left({\theta }_p\right)sin\left({\psi }_p\right))\hfill \\ {\dot{v}}_{z_p}=\frac{u_p}{m_p}(cos\left({\theta }_p\right)cos\left({\varphi }_p\right))-g\hfill \\ {\dot{v}}_{{\psi }_p}={\tilde{\tau }}_{{\psi }_p}\hfill \\ {\dot{v}}_{{\theta }_p}={\tilde{\tau }}_{{\theta }_p}\hfill \\ {\dot{v}}_{{\varphi }_p}={\tilde{\tau }}_{{\varphi }_p}\hfill \end{array}\right.,$$

(26)

which we compactly denote as $\dot{{\mathbf{x}}^p}={\mathbf{h}}^p({\mathbf{x}}^p,{\mathbf{u}}^p)$, where ${\mathbf{h}}^p$ is a 12-dimensional vector function in the right-hand side of (26), ${\mathbf{x}}^p={[x_p,y_p,z_p,{\psi }_p,{\theta }_p,{\varphi }_p,v_{x_p},v_{y_p},v_{z_p},v_{{\psi }_p},v_{{\theta }_p},v_{{\varphi }_p}]}^{\top }$ $\in {\mathbb{R}}^{12}$ is the state with velocities ${\mathbf{v}}^p=[v_{x_p},v_{y_p},v_{z_p},v_{{\psi }_p},v_{{\theta }_p},$ $v_{{\varphi }_p}{]}^{\top }\in {\mathbb{R}}^6$, and ${\mathbf{u}}^p={[u_p,{\tilde{\tau }}_{{\psi }_p},{\tilde{\tau }}_{{\theta }_p},{\tilde{\tau }}_{{\varphi }_p}]}^{\top }\in {\mathbb{R}}^4$ is the control. In the stochastic case, we added a noise term to the dynamics: $\mathrm{d}{\mathbf{x}}^p={\mathbf{h}}^p({\mathbf{x}}^p,{\mathbf{u}}^p)\mathrm{d}t+{\sigma }^p\mathrm{d}{\mathbf{W}}_t^p$, where $\mathbf{W}$ means a standard Brownian motion, meaning the quadcopter suffers from noisy measurements. The cost functions of the blue side and red side are given in Section 2.2.

For the model hyperparameters, we set $c_1=0.5$ (in (3)), $c_2=1,c_3=20,c_4=5$ (in (4)), $c_5=5$ (in (8)), $l_1=0.5$ (in (10)), and $l_2=1,l_3=20,l_4=5$ (in (11)). For ECA-Net, both networks have three linear hidden layers with 100 hidden units in each layer. Residual neural networks (ResNets) were used for both networks, with a skip connection weight of $0.5$. The tanh activation function was used in ${\Phi }_{{\omega }^p}^p$, while the ReLU activation function was used in ${Ģ}_{{\theta }^p}^p$. For training, we used ADAM with $\beta =(0.5,0.9)$, learning rate $2e-4$ for ${\Phi }_{{\omega }^p}^p$, learning rate $5e-5$ for ${Ģ}_{{\theta }^p}^p$, weight decay of $1e-4$ for both networks, batch size 50, $\lambda =2$ (the HJB penalty coefficient), and $\eta =2.5e-3$ (the neighbors’ penalty coefficient) in Algorithm 1. As in standard machine learning methods, all the plots in Section 4.3 and Appendix A.1 and Appendix A.2 were generated using validation data (data not used in training), to ensure the general adaptability of ECA-Net.

4.2. Convergence Analysis

The convergence of the MPHD-MFG method and the ECA-Net algorithm can be observed by checking the convergence of the HJB residual errors, along with the convergence of the total loss. In Figure 4a, we plot the HJB residual errors of the red and blue sides, i.e., ${\ell }_{\mathrm{HJB}}^p$ in Algorithm 1, which measures the deviation from the objective function (13) and shows the convergence of the theoretical model, the MPHD-MFG. Without an efficient strategy control, the HJB residuals under different stochasticity parameters ($\nu =\frac{{\sigma }^2}{2}=$ 0, 0.04, 0.08) were relatively high. The HJB residuals dropped fast after we applied a series of controls. After around $2\times {10}^5$ iterations, the error curves tended to be bounded and stable when we obtained the optimal control for the drones. In Figure 4b, we plot the total loss of the red and blue sides, i.e., ${\ell }_{\mathrm{total}}^p$ in Algorithm 1. After around $2\times {10}^5$ iterations, the total loss value curves under different stochasticity parameters ($\nu =\frac{{\sigma }^2}{2}=$ 0, 0.04, 0.08) tended to be bounded and stable, which means that the weight update of the neural network was completed, proving the good convergence of the solution algorithm, ECA-Net.

4.3. Performance Analysis

We set the same number for the blue side and red side, $N_1=N_2=100$. Now, we obtained the model from the above training and predicted the trajectories of the red and blue sides. Assuming that the blue side is not aggressive and the red side is aggressive, if the blue UAV is surrounded by two or more red UAVs within $e_r$, the blue individual will be captured. We give the evolutionary trajectories of the red and blue sides under different stochasticity parameters ($\nu =\frac{{\sigma }^2}{2}=$ 0, 0.04, 0.08) and different capture radii ($e_r=$ 0.7, 0.9, 1.1), as shown in Figure 5. When stochasticity parameter $\nu$ was fixed, observing Figure 5a–c over time t, respectively, the UAVs successfully avoided obstacles while minimizing their energy consumption; the smaller the capture radius of the red side, the higher the survival rate of the blue side is. At the same capture radius $e_r$ and at the same moment t, the larger the stochasticity parameter, the more scattered the distribution of the UAVs is, which increases the difficulty for the red side to completely capture the blue side. For example, in the case of $e_r=1.1$, comparing the first line of Figure 5a–c, the larger $\nu$ is, the higher the survival rate of the blue side. In particular, Figure 5c shows the diversion phenomenon. At $\nu =0.08$, the third moment, the UAVs are flying through the obstacles. In order to avoid collision with the obstacles, the UAVs choose three different paths, i.e., a diversion occurs, which demonstrates the adaptive nature of the UAV. In addition, the corresponding 3D run diagram of Figure 5 is placed in Appendix A.1. Appendix A.2 shows and analyzes the offensive and defensive effects of the UAV swarms in the asymmetric case.

4.4. Comparison with Baselines

We verified the progressiveness of our approach by comparing its performance with that of some typical baseline methods for attack–defense games in

Table 1. Comparison with baseline methods for attack–defense games.

Method	Scene	Scale of UAVs	Scene Complexity	Communication
[7]	3D blank scene	Small	0.67 1	$\mathcal{O}\left(N\right)$2
[8]	2D obstacle scene	Small	0.67	$\mathcal{O}\left(N\right)$
[9]	2D blank scene	Large	0.67	$\mathcal{O}\left(N\right)$
[37]	3D blank scene	Large	0.83	$\mathcal{O}\left(1\right)$
Ours	3D obstacle scene	Large	1	$\mathcal{O}\left(1\right)$

¹ The measurement method of scene complexity is as follows: here, it is a scoring system, [2D, 3D] = [1′, 2′]; [blank scene, obstacle scene] = [1′, 2′]; [small, large] = [1′, 2′]. We accumulated the scores for each literature experiment scene according to each item and finally normalized them. ² $\mathcal{O}\left(\right)$ is infinitesimal of the same order.

. From Table 1, it can be seen that our approach handled the most complex application scene and simplified the large-scale communication. For more details about the following baseline methods, please refer to the related References [7,8,9,37], etc.

5. Conclusions

In this paper, we formulated the cooperative attack–defense evolution of large-scale agents in high-dimensional environments as a multi-population high-dimensional stochastic mean-field game (MPHD-MFG), which significantly reduced the communication frequency and computational complexity of the swarm intelligence system. Then, we tractably solved the MPHD-MFG with a generative-adversarial-network (GAN)-based method using the MFGs’ underlying variational primal–dual structure. Based on our approach, we conducted a comprehensive experiment. The good convergence of the MPHD-MFG method and the ECA-Net algorithm was corroborated by checking the bounded stable convergence of the HJB residual error and the total loss. Through simulations, we saw that a large number of UAVs can avoid obstacles (even showing diversions) and smoothly evolve their attack and defense behaviors while minimizing their energy consumption.The comparison with the baseline methods showed that our approach is advanced. In the future, we will consider in-depth research in the asymmetric case of 3D obstacle scenarios, for example the evolution of a cooperative attack and defense between multiple (greater than or equal to three) large-scale swarms and the evolution of a cooperative attack and defense under the existence of individual performance (speed, acceleration, turning range) differences between attackers and defenders.