Research on Wargame Decision-Making Method Based on Multi-Agent Deep Deterministic Policy Gradient

Abstract

Wargames are essential simulators for various war scenarios. However, the increasing pace of warfare has rendered traditional wargame decision-making methods inadequate. To address this challenge, wargame-assisted decision-making methods that leverage artificial intelligence techniques, notably reinforcement learning, have emerged as a promising solution. The current wargame environment is beset by a large decision space and sparse rewards, presenting obstacles to optimizing decision-making methods. To overcome these hurdles, a Multi-Agent Deep Deterministic Policy Gradient (MADDPG) based wargame decision-making method is presented. The Partially Observable Markov Decision Process (POMDP), joint action-value function, and the Gumbel-Softmax estimator are applied to optimize MADDPG in order to adapt to the wargame environment. Furthermore, a wargame decision-making method based on the improved MADDPG algorithm is proposed. Using supervised learning in the proposed approach, the training efficiency is improved and the space for manipulation before the reinforcement learning phase is reduced. In addition, a policy gradient estimator is incorporated to reduce the action space and to obtain the global optimal solution. Furthermore, an additional reward function is designed to address the sparse reward problem. The experimental results demonstrate that our proposed wargame decision-making method outperforms the pre-optimization algorithm and other algorithms based on the AC framework in the wargame environment. Our approach offers a promising solution to the challenging problem of decision-making in wargame scenarios, particularly given the increasing speed and complexity of modern warfare.

1. Introduction

In the intricate field of contemporary joint warfare, the deployment of cutting-edge technologies such as unmanned and intelligent combat equipment has sparked an unprecedented escalation in the ferocity of the battlefield. In this context, the strain on mental resources for military leaders soars. In light of this, the implementation of auxiliary decision-making systems plays a critical role in relieving commanders of the burden of laborious manual tasks and cognitive overload. The capacity of these systems to process information in a refined manner and to present it in an intuitive form provides invaluable decision support for effective facilitation and scientifically grounded resource planning, program adjustment, and battlefield analysis [1,2,3,4]. With the breakneck development of computer technology and the simulation of war game systems reaching new heights, the integration of artificial intelligence technology into wargame systems and assisted decision-making is rapidly becoming a mainstream trend [5]. However, the bottlenecks facing wargames deduced from the success of AI are numerous and perplexing [6]. Among them, the bottleneck of combat intelligence situational awareness is a particularly challenging link that urgently requires a breakthrough.

Faced with the uniqueness of wargame deduction and the current situation, as well as core technologies of AI development, researchers [7] clarified the problems and solutions that arise when AI, such as deep learning, is applied to the field of wargame deduction. Specifically, Xiaoling et al. [8] proposed a training algorithm based on deep learning for the specific content of equipment damage in the process of equipment maintenance and guarantee, as well as the location point where the mobile maintenance detachment reaches the damaged equipment. This approach imbues wargame pieces with a certain sense of intelligence, resulting in more varied and bursty outcomes. To address the problem of inefficient and inaccurate target detection in indoor unmanned aerial vehicle (UAV) simulation searches, Peng [9] proposed a training technique based on a neural network algorithm. This technique effectively shortens the training period while improving search efficiency and accuracy, adding yet another layer of complexity and perplexity to the text. Wu [10] analyzed the construction of an urban flooding disaster emergency linkage system using wargame projection, highlighting the promising prospects for intelligent wargame projection in real-life applications. Meanwhile, the problems of handling incomplete information in computer wargames and the scarcity of open-source datasets for wargame replay make AI algorithms all the more challenging [11]. A new specific network model was designed to predict the enemy’s location and solved these challenges by using deep learning dataset processing, employing multi-headed input, multi-headed output, a convolutional neural network (CNN), and gated recurrent unit (GRU) layers to handle multi-agent and long-term memory problems. To overcome the problem of low efficiency, stability, and reliability for traditional intent recognition methods due to wargame fog, Chen [12] proposed a deep learning architecture consisting of a contrast predictive coding model, and a variable-length long and short-term memory network model with an attention weight allocator. This approach allows for the online intent recognition of incomplete information in wargames, increasing the perplexity and burstiness of the text by adding another layer of complexity and specificity to the topic at hand.

While there has been some success in the studies mentioned above, there are limitations in the application of traditional AI techniques to assist in decision-making regarding wargames. These limitations stem from three major factors: insufficient dynamic adaptation capability, insufficient global capability, and insufficient autonomous decision-making ability. Firstly, traditional AI techniques, such as ant colony algorithms [13], simulated annealing algorithms [14], and other heuristic algorithms, are based on existing problems to find optimal solutions. These generated paradigms or models are very efficient in solving similar scenarios. However, when a more complex scenario or scenario changes, the generated model becomes unusable. This limitation makes it difficult to adapt to the needs of a modern fast-paced society [15]. Secondly, most of the traditional intelligent decision-making technologies are oriented to local decision-making problems in a specific field under a single link or a specific objective, and the local optimal solution is obtained [16]. Traditional intelligent decision-making technologies are incapable of finding the relationships between these fields, and they consequently struggle to provide global decision-making support [17]. Lastly, there is insufficient autonomous decision-making ability. Machine learning technologies, represented by neural networks, have rapidly improved the speed of model training to meet the needs of many decision-making scenarios [18]. For example, traditional intelligent decision-making technologies are often unable to make quick judgments in the face of unprecedented unstructured environments [19].

The aforementioned studies identified limitations in the implementation of traditional AI methods, and to overcome these issues, the researchers explored the introduction of reinforcement learning techniques into wargame decision-making. One such study [20] employed reinforcement learning multi-agent deep deterministic policy gradient algorithms for dynamic decision-making in game AI, while also leveraging deep learning and natural language processing techniques to transform game context maps into textual suggestions during wargame confrontations. By combining reinforcement learning techniques, deep learning techniques, and natural language processing techniques, semantic text with state-of-the-art accuracy output enables generalization, thereby playing a crucial role in enhancing the human understanding of game AI behavior. Wu [21] found that reinforcement learning-based models are more robust and powerful than expert experience-based approaches in a wargame environment, but require more time to train. Choi [22] automated troop deployment in wargames using reinforcement learning. Boron [23] trained AI agents for optimal offensive behavior validated by the tactical principles of mass and economic power. Hung [24] used reinforcement learning to design strategies for wargame simulations and to provide commanders with decision-making support. Nevertheless, despite the significant achievements of these researchers, wargame decision-making models still suffer from large decision space, local optimal solution, and slow training convergence.

From the above investigations, it can be seen that although reinforcement learning has made some progress in the field of wargame decision-making, problems such as large decision space, local optimal solution, and slow training convergence still exist, which may reduce the wargame win rates. A wargame decision-making method which optimizes the MADDPG algorithm as well as divides the wargame decision-making method into supervised learning and reinforcement learning phases is proposed to address these challenges, hence enabling a faster training convergence, higher reward values, and achieving the global optimal solution and ultimately improved wargame win rates. The main contributions of this paper can be summarized as follows:

• The MADDPG was optimized to adapt the wargame environment. The POMDP, the joint action-value function, and the Gumbel-Softmax estimator were introduced to model the decision process, train centralized critics, and fit the discrete policies of wargames, respectively.
• The supervised learning was incorporated before the reinforcement learning to improve training efficiency and reduce the action space. The wargame decision-making method was structured by dividing it into a supervised learning phase and a reinforcement learning phase. In the supervised learning phase, the state-action information pair data were separated to obtain the training and testing sets, and the model was trained with the supervised learning algorithm to obtain the primary agent.
• In the reinforcement learning phase of the wargame decision-making method, the policy gradient estimator was adopted to achieve the reduction of action space and to obtain the global optimal solution, while the additional reward function was designed to solve the sparse reward problem.

The above contributions provide a method for solving the problems of large decision space, local optimal solution, and slow training convergence, which additionally provide some research ideas and solutions for future related studies.

This paper is organized as follows. Section 2 provides a comprehensive literature review. Section 3 introduces MARL and optimizes MADDPG for the wargame environment, and Section 4 presents the wargame decision-making methods. Section 5 conducts wargame experiments to verify the proposed method. Section 6 concludes this study and lists the future work.

2. Related Work

2.1. Labeled and Real Combat Data Shortage

Wargames and decision-making are intricate affairs that require labeled data to make predictions and decisions. The labeling process involves categorizing data such as terrain types, enemy units, friendly forces, resources, and objectives. Nonetheless, a deficiency in labeling data is ubiquitous, particularly regarding real combat data [25,26,27,28,29].

The complexity and unpredictability of combat scenarios illustrate the scarcity of labeled data in this context. In real-world conflicts, it is arduous to categorize data for every possible outcome or decision, and the available data may be incomplete or unreliable. Consequently, the application of traditional supervised learning methods that rely on labeled data is challenging. Reinforcement learning (RL) presents an opportunity to surmount these limitations. In wargame simulations, RL would be utilized to train models to make decisions based on the present game state and available options. The agent is allowed to test diverse strategies and learn from the outcomes to enhance decision-making over time. Similarly, in a decision-making scenario, an RL agent is trained to make decisions based on available information and feedback from previous decisions. The appeal of RL in these contexts is reflected in its ability to adapt to changing circumstances and to learn from experience, even in the absence of labeled data. RL allows for the handling of complex and uncertain environments with multiple possible outcomes, making it a well-suited approach for wargame simulations and decision-making scenarios.

The dearth of labeled data and real combat data in wargame simulations and decision-making scenarios renders traditional supervised learning methods challenging to employ. Nevertheless, RL is available to train models to make decisions based on experience and feedback, making it a promising approach in these intricate and unpredictable environments.

2.2. Markov Decision Process

The arcane and complex world of Markov processes has been the focus of relentless research since A.A. Mapkob’s seminal paper. The evolution of Markov processes is listed in

Table 1. The evolution of Markov processes.

Contributors	Contributed Content
A.A. MapkoB	Created Markov processes
Cogburn, R [30]	Gave formulas for Markov chains in stochastic environments
Chung, K.L [31]	Presented various limit theorem theories in the general context
Orey, S [32]	Compiled the limit theorems concerning the transfer probability of Markov chains
Cogburn, R [33]	Analyzed the dependence between environmental factors and controlled Markov chains
Cogburn, R [34]	Established the central limit theorem of the function of Markov chains in a stochastic environment

. The introduction of environmental factors into classical Markov chains was a watershed moment for scholars, as it opened up new avenues for exploration. In the groundbreaking work conducted by Cogburn [30], researchers gave formulas for Markov chains in stochastic environments, exploring special cases such as branching processes, queues, life and death chains, and random wandering in stochastic environments. Based on this study, Chung [31] presented various limit theorem theories for Markov processes in the general context, while Orey [32] compiled the limit theorems concerning the transfer probability of Markov chains. Cogburn [33] tackled the daunting task of providing general expressions for the stochastic model of Markov chains in stochastic environments, analyzing the dependence between environmental factors and controlled Markov chains. They also derived the ergodic theory of Markov chains in stochastic environments and established the conditions for the existence of finite invariant measures, a formidable undertaking to say the least. Under the premise of a finite state space and the existence of finite invariant ergodic measures and mixed conditions, Cogburn [34] established the central limit theorem for the function of a Markov chain in a stochastic environment. It has been demonstrated that these conditions are always satisfied when the state space is finite, which is a pioneering result that will undoubtedly determine the future of research in this field.

Underlying the above studies, the Markov decision process (MDP) was formulated, which consists of a tuple $M=<S,A,P,R,\gamma >$. Here, $s\in S$ describes the true state of the environment. In the MDP, the agent needs to select the action $a\in A$, which depends on the agent’s own policy. This may lead to the state of the environment changing to $s^{\prime }$, which is determined by the state transition function $P\left(s^{\prime }\right|s,a)$. In addition, the environment will give the reward $r=R(s,a)$. $\gamma \in [0,1)$ is a discount factor.

2.3. DDPG

A Deep Deterministic Policy Gradient (DDPG) is an algorithm developed by Lillicrap et al. of DeepMind in 2015 [35]. DDPG is based on an improvement of the DPG algorithm and is regarded as a combination of Actor-Critic (AC) and DQN. It learns both a Q-function and a policy: the Q-function is learned by Q-learning, and the policy is updated by the Q-function. The DDPG algorithm is an online deep reinforcement learning algorithm in the AC framework, and the algorithm internally consists of an Actor network and a Critic network, each of which is updated according to its own update law, thus maximizing the cumulative expected reward.

DDPG is under the paradigm of centralized training with decentralized execution (CTDE) [36], and the framework of DDPG is demonstrated in Figure 1. The agent puts the obtained empirical data $(s,a,r,s^{\prime })$ into the Replay Buffer $D$ and follows the batch sampling when updating the network parameters. In addition to the Actor network and Critic network, a set of Target Actor network and Target Critic network for estimating the target are used. When updating the target network, soft updates are used to avoid an excessive parameter update. Since the deterministic strategy outputs deterministic actions, it lacks the exploration of the environment. In the training phase, noise is added to the actions outputted by the Actor network to allow the agent to have some exploration ability.

3. Improved MADDPG for Wargame Decision-Making

3.1. Multi-Agent Reinforcement Learning

Single-agent systems are unable to realize the collaborative or competitive relationships among multiple decision makers when faced with large-scale, complex contextual decision-making problems. Therefore, the deep reinforcement learning model is extended to a multi-agent system in which multiple agents cooperate, communicate, and compete with each other, which is known as Multi-Agent Reinforcement Learning (MARL) [37].

Agents explore in MARL using the CTDE approach to maximize cumulative expectations, and the MARL framework is shown in Figure 2. All agents will be trained centrally to obtain the loss value to update the Critic network, and update the policy according to the action-value function from the Critic network. Each agent selects an action according to its current policy after training. For each agent, the MDP defines its interaction with the environment, which contains other agents with a single environment. Thus, the Joint State is denoted as $S_t^J=\{env,s_t^1,s_t^2,\dots ,s_t^n\}$, where $env$ denotes a single environment, and ${\{s}_t^1,s_t^2,\dots ,s_t^n\}$ denotes the set of all agent states. When any agent needs to use the environment parameter, it needs to extract $S_t^J$ (in addition to its own state). Each agent derives its respective action $a$ based on the current policy after obtaining $S_t^J$. During training, all actions of all agents are combined linearly or nonlinearly to obtain the Joint Action $A_t^J=\{a_t^1,a_t^2,\dots ,a_t^n\}$. The environment feeds the Joint Reward $R_t^J=\{r_t^1,r_t^2,\dots ,r_t^n\}$ according to $A_t^J$, each agent decomposes $R_t^J$ to get its own reward $r$, and the reward $r$ is used to update its own policy.

3.2. Improved MAPDDPG

3.2.1. Partially Observable Markov Decision Process

In wargames, agents are constrained to perceive all states $s\in S$ of the environment. Modeling with MDP in this case is not feasible, therefore POMDP [38] is introduced for modeling, and the POMDP framework is illustrated in Figure 3. A Partially Observable Markov Decision Process (POMDP) consists of a tuple $G=<S,A,P,R,\gamma ,N,\mathsf{\Omega },O>$, where $N=\left\{\mathrm{1,2},\cdots ,n\right\}$ denotes the set of agents. Due to the partial observability, each agent $x\in N$ draws an individual partial observation $o_x\in \Omega$ from the observation kernel $O\left(s,x\right)$ [39]. When the agent learns the deterministic strategy ${\mu }_x\left({\tau }_x\right)$, it is parameterized only by the local action and observation history ${\tau }_x\in T\equiv (\mathsf{\Omega }\times U)$.

3.2.2. Joint Action-Value Function

MADDPG is an extension of DDPG in a multi-agent environment, which is suitable for collaborative, competitive, or mixed mission settings [40]. In MADDPG, a separate actor and critic are learned for each agent, and each agent is therefore allowed to have its own arbitrary reward function.

In wargames, each agent $x$ has a deterministic policy ${\mu }_x({\tau }_x,{\theta }_x)$, where ${\theta }_x$ represents the parameter of ${\mu }_x$, and $\mu$ is the set of policies for all agents defined as ${\left\{{\mu }_x({\tau }_x,{\theta }_x)\right\}}_{x=1}^n$. In order to centralize the training of agents in the critic networks, it is necessary to estimate the Action-Value function $Q_x^{\mu }(\mathrm{o},a_1{,a}_2,\dots {,a}_n;{\varphi }_x)$, which is learned for each agent $x$ separately. In the Action-Value function $Q_x^{\mu }$, $o$ represents the observation set $\mathsf{\Omega }$, $\left\{a_1{,a}_2,\dots {,a}_n\right\}$ refers to the actions of agents in wargame, and ${\varphi }_x$ is the parameter of the Action-Value function. It trains the critic network by minimizing the loss function:

$$LOSS\left({\varphi }_x\right)={\mathbb{E}}_{\mathcal{D}}\left[{(y^x-Q_x^{\mu }(o,a_1{,a}_2,\dots {,a}_n;{\varphi }_x\left)\right)}^2\right],$$

(1)

where $y^x=r_x{+\gamma Q_x^{\mu }(o^{\prime },a_1^{\prime },a_2^{\prime },\dots a_n^{\prime }{|}_{{a_n^{\prime }=\mu }_x(a_x^{\prime },{\theta }_x^{\prime })};{\varphi }_x^{\prime })}^2$. In this equation, $r_x$ denotes the reward received by each agent $x$, ${\{a}_1^{\prime },a_2^{\prime },\dots a_n^{\prime }\}$ is the set of target policies, the delay parameters ${\theta }_x^{\prime }$ and ${\varphi }_x^{\prime }$ are the parameter of the target critic network, and the Replay Buffer $D$ consists of the set $[s,s^{\prime },a_1,...,a_n,r_1,...,r_n]$. The following policy gradients are calculated separately to update each agent’s policy:

$${\nabla }_{{\theta }_x}J({\mu }_x)={\mathbb{E}}_{\mathcal{D}}\left[{{\nabla }_{{\theta }_x}{\mu }_x\left({\tau }_x\right){\nabla }_{a_x}{Q}_x^{\mathit{\mu }}(s,a_1,\dots ,a_n)|}_{a_x={\mu }_x\left({\tau }_x\right)}\right],$$

(2)

where agent $x$’s current action $a_x$ is sampled from its current policy ${\mu }_a$ when evaluating the Action-Value function $Q_x^{\mu }$, while all other agents’ actions are sampled from the replay buffer $D$.

However, in the wargame environment, where the number of agents and actions is large, training a centralized critic using only a simple MADDPG is potentially difficult and may not converge. To meet this challenge, a Joint Action-Value function $Q_{tot}^{\mathit{\mu }}$ is introduced to train a centralized critic. Specifically, the Action-Value function of MADDPG is factorized, and the $Q_x^{\mu }$ of each agent is inputted into the mixing network, followed by setting a mixing network parameter, while the $Q_{tot}^{\mathit{\mu }}$ is obtained after estimation. The following is the joint action value function $Q_{tot}^{\mathit{\mu }}$:

$$Q_{tot}^{\mathit{\mu }}(\mathit{\tau },\mathit{a},s;\mathit{\varphi },\psi )=g_{\psi }(s,\{Q_x^{{\mu }_x}({\tau }_x,a_x;{\varphi }_x){\}}_{x=1}^n),$$

(3)

where $\mathit{\varphi }$ is the parameter of the Joint Action-Value function $Q_{tot}^{\mathit{\mu }}$, and ${\varphi }_x$ represents the parameter of the Action-Value function $Q_x^{{\mu }_x}$. In a wargame, $g_{\psi }$ denotes a linear monotonic function, where the parameter $\psi$ is a mixing network parameter. In order to compute the policy accurately, the centralized critic network needs to be trained to minimize the loss function. The loss function is presented as follows

$$LOSS(\mathit{\varphi },\psi )={\mathbb{E}}_D\left[{(y^{tot}-Q_{tot}^{\mathit{\mu }}(\mathit{\tau },\mathit{a},s;\mathit{\varphi },\psi \left)\right)}^2\right],$$

(4)

where $D$ refers to the Replay Buffer, and $y^{tot}=r+\gamma Q_{tot}^{\mathit{\mu }}({\mathit{\tau }}^{\prime },\mathit{\mu }({\mathit{\tau }}^{\prime };{\mathit{\theta }}^{\prime }),s^{\prime };{\mathit{\varphi }}^{\prime },{\psi }^{\prime })$. In the $y^{tot}$ equation, ${\mathit{\varphi }}^{\prime }$, ${\mathit{\theta }}^{\prime }$, ${\psi }^{\prime }$ represents the parameters of the mixing network, the target actor networks, and the target critic networks, respectively.

In the Actor network, in order to update each agent’s own policy, it is necessary to use the policy gradient to perform the calculation. The following is the gradient formula:

$${\nabla }_{{\theta }_x}J({\mu }_x)={\mathbb{E}}_{\mathcal{D}}\left[{{\nabla }_{{\theta }_x}{\mu }_x\left({\tau }_x\right){\nabla }_{a_x}{Q}_{tot}^{\mathit{\mu }}(\mathit{\tau },a_1,\dots ,a_n,s)|}_{a_x={\mu }_x\left({\tau }_x\right)}\right],$$

(5)

3.2.3. Gumbel-Softmax Estimator for Discrete Policy

Due to the limited space for agent action in wargames, it does not have a high degree of freedom as drones or robots. The action space in wargames is discrete, so it is necessary to use the discrete policy to decide which action to use.

However, the policy needs in MADDPG are differentiable, while the agent’s actions in a wargame are discrete, so direct sampling leads to non-differentiability. Therefore, the Gumbel-Softmax estimator was introduced to solve this problem. Gumbel-Softmax distribution [41] is defined by:

$$P_{\pi ,\tau }(y_1,\dots ,y_k)=\mathsf{\Gamma }(k){\tau }^{k-1}{({\sum }_{i=1}^k{\pi }_i/y_i^{\tau })}^{-k}{\prod }_{i=1}^k({\pi }_i/y_i^{\tau +1}),$$

(6)

When the parameter $\tau$ in softmax tends to zero, the samples that fit the Gumbel-Softmax distribution become single-peaked, and the sample distribution of Gumbel-Softmax becomes the same as the discrete distribution. Accordingly, in MADDPG, the discrete distribution is replaced with samples of Gumbel-Softmax. The specific process is that since the Actor network needs to output discrete actions, samplers are read into the Actor networks to sample discrete actions from continuous policies, and when the Actor network needs to be updated, the gradient ${\nabla }_{\theta }J\left(\mathit{m}\right)\approx {\mathbb{E}}_{\mathcal{D}}\left[{\nabla }_{\theta }\mathit{m}{\nabla }_{\mathit{m}}{Q}_{tot}^{\mathit{m}}\right(\mathit{\tau },m_1,\dots ,m_n,s\left)\right]$ is approximated by the Gumbel-Softmax sample $m_a$, where $m=\left\{m_1,\dots ,m_n\right\}$ denotes the set of consecutive samples, and then the policy of the Actor network is available to be updated. The improved MADDPG architecture is presented in Figure 4.

4. Wargame Decision-Making Method

Intelligent conditional wargame adversarial pushing is actually a confrontation of agents in a wargame, but there are huge challenges for current agent training. Wargames have a large map environment and a large action space, which is less efficient to train directly using the MADDPG algorithm [42]. As shown in Figure 5, the wargame decision-making method is divided into a supervised learning phase and a reinforcement learning phase, and then the rule-based supervised learning is introduced to train the initial agent before using the MADDPG algorithm, followed by using a policy gradient estimator to optimize the action space of the agent after entering the MADDPG algorithm, and ending with adjusting the reward function to suit the needs of the wargame.

4.1. Supervised Learning Phase

Since it is difficult to train directly using MADDPG, supervised learning is introduced to pre-train the agent to reduce its policy search space. Both rules of engagement data and human vs data for data pre-processing are utilized to obtain state-action information pairs, which are separated into a training set and a validation set to facilitate supervised learning.

The back propagation neural network [43] is used for the supervised learning of state-action information pairs in this paper. The neural network structure design is shown in Figure 6. The neural network is mainly composed of an input layer, two hidden layers, and an output layer. The activation functions of all three hidden layers use a Rectified Linear Unit (ReLU). The output layer uses the Softmax activation function and selects the one with the highest probability and outputs it as an action. In the training process, the winners of the pre-collected combat data from x-groups ($x>500$) for the specific rule agents are used as the training data set for decision learning. The training data set uses several randomized agents with different rules, strategies and degrees to fight against each other. The neural network parameters are trained by signal forward propagation and error back propagation. The cost function in the training process is as follows:

$$J(\mathsf{\Theta })=-\frac{1}{n}[{\sum }_{i=1}^n{\sum }_{k=1}^m{a}_k^{(i)}\log {(h_{\mathsf{\Theta }}(s^{(i)}))}_k+(1-a_k^{(i)})\mathrm{l}\mathrm{o}\mathrm{g}{(1-h_{\mathsf{\Theta }}(s^{(i)}))}_k],$$

(7)

where $n$ denotes the number of training samples, $K$ refers to dimensions of the output vector, $s^{\left(i\right)}$ represents the input of the $i$-th training sample, $a_k^{\left(i\right)}$ refers to the expected output of the $k$-th scalar value of the $i$-th training sample, and $h_{\mathsf{\Theta }}\left(s^{\left(i\right)}\right)$ indicates the actual output of the $k$-th scalar value of the $i$-th training sample.

In the process of parameter updating, the action decisions made according to the current situation are compared with the action decisions calculated by the current neural network through the pre-collected combat data of the specific rule agents. The neural network parameters are then updated by the reverse transfer. After the training is completed, it can be used as the initial agent network for deep reinforcement learning.

4.2. Reinforcement Learning Phase

4.2.1. Policy Gradient Estimator

The utilization of policy gradients in the MADDPG framework is always a widely discussed topic. The updating of individual deterministic policies for all agents now hinges upon a singular, shared factored critic, $Q_{tot}^{\mathit{\mu }}$. This is in stark contrast to the traditional approach of learning and employing a monolithic critic, $Q_n^{\mu }$, for each agent.

However, as elegant as this solution may seem, there are two fundamental issues that plague policy gradients. Firstly, each agent optimizes its own policy, whilst assuming that the actions of all other agents are fixed. This myopic approach leads to suboptimal policies, i.e., no agent wishes to unilaterally modify their actions. Secondly, these policy gradients are susceptible to overgeneralization, where agent a’s ascent up the gradient according to $Q_n^{\mu }$ or $Q_{tot}^{\mathit{\mu }}$, only involves the sampling of its own action, $a_x$, from its current policy, ${\mu }_a$. Meanwhile, all other agents’ actions are sampled from the Replay Buffer $D$, potentially resulting in their actions being drastically different from those dictated by their current policies. This leads agents to converge to suboptimal actions that seem to be a better choice when considering the impact of arbitrary actions from their collaborators.

Utilizing a revolutionary approach [44], a novel centralized gradient estimator is introduced to optimize in the complete joint action space. It not only reduces the action space, but also obtains the global optimal solution. Unlike the conventional methods in Equations (2) and (5), the estimator proposed in this study avoids optimizing each agent’s actions in isolation.

To overcome the issue of relative overgeneralization, a new sampling technique is incorporated in which $Q_{tot}^{\mathit{\mu }}$ is evaluated by sampling all actions from each agent’s current policies during policy gradient calculation. This ensures a more comprehensive evaluation that takes into account all possible actions, leading to a more significant improvement in overall performance. The following is the policy gradient formula:

$${\nabla }_{\theta }J\left(\mu \right)={\mathbb{E}}_D\left[{\nabla }_{\theta }\mu {\nabla }_{\mu }{Q}_{tot}^{\mathit{\mu }}\right(\mathit{\tau },{\mu }_1\left({\tau }_1\right),\cdot \cdot \cdot ,{\mu }_n\left({\tau }_n\right),s\left)\right],$$

(8)

where $\mathit{\mu }=\{{\mu }_1\left({\tau }_1;{\theta }_1\right),\dots ,{\mu }_n\left({\tau }_n;{\theta }_n\right)\}$ is the set of all agents’ current policies, and they share an Actor network with the same parameter $\theta$.

4.2.2. Additional Reward Function

The intricate issue of the sparse reward problem [45] has been a persistent obstacle in the practical implementation of deep reinforcement learning. It focuses on the problem that arises due to the training environment’s inability to supervise the updating of agent parameters during deep reinforcement learning. While supervised learning relies on human supervision, deep reinforcement learning relies on rewards to optimize the agent’s strategy. In the wargame deductive environment discussed in this paper, the situation is even more complex, as the environment is only able to make rule judgments and war decisions based on the action taken. Additionally, it does not provide any reward information after the maneuver or the battle. The environment only sends a victory message when our operators reach the point of control or when the enemy operators are entirely annihilated. Similarly, it sends a failure message after the enemy operator arrives at the point of control or our operators are entirely annihilated. Accordingly, as illustrated in Figure 7, every step in the training process is unrewarding. This issue of sparse reward is a considerable hindrance to the convergence of the algorithm, and in some cases it may cause the algorithm to fail to converge.

As demonstrated in Figure 8, to solve the convergence problem, additional rewards are engaged during the training process based on an analysis of the deductive environment. Specifically, the deductive environment mandates that the victory condition is met when the control point is reached or when the enemy operator is fully eliminated. When this victory condition cannot be fulfilled, the victory is assessed by calculating the remaining operator’s blood level. As a result, additional rewards are incorporated in the training process based on this experience, and each agent is rewarded after each action. Furthermore, to avoid the agent getting trapped in a local optimum during the exploration process, a penalty is applied to the agent for each turn taken before winning.

5. Experiments

5.1. Experiments Platform

Our experiments were performed on a computer with an Intel Core i7-8700K CPU and an NVIDIA GeForce RTX 3060. The experiments were run on the Ubuntu system, while the machine learning-related components made use of PyTorch. In this paper, the algorithm was designed and simulated based on the environment of the wargame-playing confrontation in the “Temple Calculation Wargame System” (http://wargame.ia.ac.cn/aidevelopment, accessed on 8 July 2022). The trained agent model was tested in the “Temple Calculation Wargame Replay System” (http://wargame.ia.ac.cn/newReplay, accessed on 8 July 2022) to obtain scores.

Our force was determined to rely on a plateau channel to launch a general attack on the enemy main Force depth. On the both sides of the channel, mountains towered high and the terrain was high. The valley in the middle channel is low-lying and open, and is not easy to hide. In an attempt to gain control of the channel, our force dispatched an armored infantry platoon and a tank platoon to mass in the area of 6048. The advance reconnaissance in our force required a rapid travel to occupy the settlement areas from 4435 to 430, to organize the fire reconnaissance, and to provide intelligence support for the follow-up main forces. The enemy dispatched two armored infantry platoons and a tank platoon to mass in the area of 3426. They were ordered to hold the target settlement ground, ambush and prevent our force to despoil this area and cover their own deep main forces. In the experiment, our own agents, according to the wargame decision-making methods, fought against enemies who simply followed the rules, and the ablation was added experiments. Figure 9 shows a part of the simulated environmental map. The basic topographic information of the map includes urban land, soft ground, and roads and elevations. The left side of the map is the red operator, and the hexagonal lattice with the green flag is the control point.

5.2. Experimental Settings

The POMDP (consisted of a tuple $G=<S,A,P,R,\gamma ,N,\mathsf{\Omega },O>$) model is investigated in the experimental settings. The state of the wargame is substituted into $S$, the set of actions that the agent can take into $A$ (a total of 10 actions can be taken), $P$ refers to the state transfer probability of the agents in the wargame, $R$ demonstrates the set of rewards after the agents interact with the wargame environment, $\gamma$ is the discount factor and here is taken as 0.9, $N$ represents the number of agents, in this case a total of three, and each agent $x\in N$ draws an individual partial observation $o_x\in \Omega$ from the observation kernel $O(s,x)$.

There are three agents on each side, and the details have been listed in

Table 2. Our side’s agents and their details.

Our Side’s Agent	Icon	Action Speed (s/Grid)	Initial Position
Tank		20	5947
Chariot		20	6048
Infantry Squad		144	6048

and

Table 3. The enemy’s agents and their details.

Enemy’s Agents	Icon	Action Speed (s/Grid)	Initial Position
Heavy Tank		15	3427
Heavy Chariot		15	3526
Infantry Squad		144	3526

, while the number corresponds to the agent’s action, as shown in

Table 4. The agents‘ action.

Number of Actions	Action of Agents
0	Move to the left
1	Move to the right
2	Move to the upper left
3	Move to the upper right
4	Move to the lower left
5	Move to the lower right
6	Attack enemy’s heavy tank
7	Attack enemy’s heavy chariot
8	Attack enemy’s infantry squad
9	Convert to covert status

. The parameters of MADDPG used to train the agents are shown in

Table 5. The parameters of MADDPG.

Parameters	Values
action_selector	“gumbel”
epsilon_start	0.5
epsilon_finish	0.05
epsilon_anneal_time	50,000
obs_last_action	True
batch_size_run	1
batch_size	32
buffer_size	5000
act_noise	0.1
gamma	0.9
target_update_interval	200
target_update_mode	‘hard’
target_update_tau:	0.001

.

5.3. Experimental Results and Analysis

The wargame decision-making method was abbreviated based on MADDPG as wdMADDPG. Experiments were conducted by using the following techniques: wdMADDPG, MADDPG alone (MADDPG-a), wdMADDPG without supervised learning (wdMADDPG-sl), wdMADDPG without policy gradient estimator (wdMADDPG-pge), and wdMADDPG without additional reward function (wdMADDPG-ar). It should be noted that MADDPG-a is not a pure MADDPG, as it includes the Gumbel-Softmax estimator to cope with the discrete policy in a wargame, otherwise a pure MADDPG would not work properly in a wargame. The DOP [46] and COMA [47] algorithms based on the AC framework were evaluated and compared with the proposed MADDPG.

Each method was trained 2 million times in the experiments, and the reward values and loss values of all seven methods are shown in Figure 10 and Figure 11, Seven methods were trained for each seven models, and each model was tested in 500 games in the “Temple Calculation Wargame Replay System”, and the test results are presented in Figure 12.

As shown in Figure 10a and Figure 11a, after 2 million training sessions, the mean reward values and mean loss values of MADDPG outperformed other methods based on the AC framework. As demonstrated in Figure 12a, the win rates of MADDPG are 69.9% and 34.9% higher than DOP and COMA, respectively.

As shown in Figure 10b, after 2 million training sessions, the mean reward values are wdMADDPG, wdMADDPG-pge, wdMADDPG-ar, wdMADDPG-sl, and MADDPG-a, in descending order. From the results, all the other methods are significantly better than MADDPG-a, indicating that the optimizations of the wargame decision-making method proposed in this paper are effective. wdMADDPG-sl is only a little higher than MADDPG-a in terms of mean reward values, indicating that supervised learning before reinforcement learning is the most helpful for wargame training, which effectively reduces the movement space and improves the training efficiency. An additional reward is also significantly helpful for the wargame decision-making method, which improves the training rate less effectively than the supervised learning, but it also significantly increases the mean reward values in training. The policy gradient estimator has the lowest improvement in training efficiency and the lowest improvement in the mean reward values.

As shown in Figure 11b, after 2 million training sessions, the mean loss values are in descending order: MADDPG-a, wdMADDPG-pge, wdMADDPG-sl, wdMADDPG-ar and wdMADDPG. From this ranking, all the remaining methods are more exploratory compared to MADDPG-a, which represents the usefulness of all the optimizations we performed on the wargame decision-making methods. The experimental results indicate that the policy gradient estimator effectively optimizes the action space, and in the critic networks, each agent extracts actions from the current policy to estimate $Q_{tot}^{\mu }$, and globally considers network parameter updates, which make the wargame decision-making methods more exploratory. Figure 11b demonstrates that the proposed wargame decision-making method is the most exploratory among these methods, with the highest probability of obtaining the global optimal solution.

As shown in Figure 12b, each model was tested for 500 games through the “Temple Calculation Wargame Replay System”. The models ranked from the highest to lowest win rate were wdMADDPG, wdMADDPG-pge, wdMADDPG-ar, wdMADDPG-sl and MADDPG-a, corresponding to t 413, 375, 343, 337, and 314 games won, respectively. From the results, the optimizations made in this paper all improved the win rate of the wargame. The wdMADDPG-based wargame decision-making method shows the best performance in a wargame, where it improved the win rate by 31.5% over pure MADDPG.

6. Conclusions and Future Work

A novel wargame decision-making approach that utilizes MADDPG is proposed in this paper. The proposed method surpasses traditional reinforcement learning methods applied to wargame-assisted decision-making scenarios. The POMDP model is leveraged to encapsulate the decision-making process within wargames. In addition, the joint action-value functions tailored to wargame environments are introduced in the MADDPG algorithm. With the purpose of overcoming the challenge of discrete policies, the Gumbel-Softmax estimator was integrated into MADDPG. Our approach includes a supervised learning phase and a reinforcement learning phase to tackle the problems of the large action space and sparse rewards. The experimental results demonstrate that the proposed wargame decision-making method improves the wargame win rate by 31.5% compared to pure MADDPG. Furthermore, our method outperforms the DOP with COMA based on the AC framework by 69.9% and 34.9%. The results indicate that the proposed approach holds great potential for decision-making in wargame scenarios given its superior performance and ability to overcome the challenges of the current wargame environment.

A high-performance intelligent decision-making approach for wargames that still has some limitations was presented in this paper. The joint value function $Q_{tot}$ and the Gumbel-Softmax estimator increased training time. Additionally, scalability is a crucial metric for evaluating multi-agent methods, but has not been investigated in this study. In future studies, a wider range of algorithms will be adopted to train the proposed wargame decision-making method, and the number of agents in the wargame will be increased to evaluate scalability. The discrete policy-based method will be directly used to avoid the detrimental effect of the Gumbel-Softmax estimator on the training speed in order to enhance the model’s training efficiency.