Deep Reinforcement Learning-Based Multi-Agent System with Advanced Actor–Critic Framework for Complex Environment

Abstract

The development of artificial intelligence (AI) game agents that use deep reinforcement learning (DRL) algorithms to process visual information for decision-making has emerged as a key research focus in both academia and industry. However, previous game agents have struggled to execute multiple commands simultaneously in a single decision, failing to accurately replicate the complex control patterns that characterize human gameplay. In this paper, we utilize the ViZDoom environment as the DRL research platform and transform the agent–environment interactions into a Partially Observable Markov Decision Process (POMDP). We introduce an advanced multi-agent deep reinforcement learning (DRL) framework, specifically a Multi-Agent Proximal Policy Optimization (MA-PPO), designed to optimize target acquisition while operating within defined ammunition and time constraints. In MA-PPO, each agent handles distinct parallel tasks with custom reward functions for performance evaluation. The agents make independent decisions while simultaneously executing multiple commands to mimic human-like gameplay behavior. Our evaluation compares MA-PPO against other DRL algorithms, showing a 30.67% performance improvement over the baseline algorithm.

1. Introduction

Recent advancements in fundamental physics have been facilitated by the implementation of artificial neural networks and machine learning methodologies [1]. In parallel, AlphaFold, an innovative system integrating deep learning and reinforcement learning methodologies, was honored with the Nobel Prize in Chemistry for its revolutionary advancements in the field of protein structure prediction [2]. The progression from AlphaStar and AlphaGo’s notable successes in gaming to AlphaFold’s groundbreaking applications in biology exemplifies the versatility of deep reinforcement learning (DRL) technology [3]. DRL, which integrates neural networks with reinforcement learning principles, has not only demonstrated remarkable efficacy in gaming scenarios but also continues to significantly advance various domains of scientific research [4,5].

DRL has achieved substantial attention across both industry and academia [6], demonstrating its capability as a model-free methodology for addressing complex decision-making challenges in industrial environments [7]. AlphaGo, an artificial intelligence (AI) system for the ancient game of Go, achieved a historic milestone by defeating human Go champions [8]. The system utilizes a sophisticated dual-phase approach, initially employing supervised learning on human data before transitioning to reinforcement learning through self-play iterations. Leveraging the powerful function approximation capabilities of deep neural networks, DRL agents demonstrate remarkable proficiency in acquiring knowledge through interactions with unknown, information-limited environments, ultimately leading to robust real-time decision-making processes. Building upon the exceptional adaptability of DRL agents in complex environments, DRL-based algorithms have demonstrated significant value across multiple domains, including intraday scheduling for home energy systems [9], flight trajectory optimization for unmanned aerial vehicles [10], and energy trading between electric vehicles and power grids [11]. The foundation for these achievements stems is built on the extensive testing and validation of DRL frameworks through diverse gaming environments and simulators, establishing it as an advanced and effective paradigm [12,13].

The journey commenced with DeepMind’s groundbreaking implementation of DRL technology in Atari games [14], where artificial intelligence agents demonstrated exceptional capabilities in mastering and subsequently surpassing human performance levels [15]. While AlphaStar showcased unprecedented mastery by outperforming 99.8% of human players in StarCraft through its precise real-time execution, it encountered limitations in certain scenarios due to deficiencies in cognitive reasoning and strategic planning capabilities. Currently, OpenAI Gym provides a diverse range of game interfaces for training and evaluating DRL algorithms, i.e., SC2LE [16], ViZDoom [17], and Mujoco [18], establishing a robust foundation for developing more sophisticated game AI [19]. Therefore, our research is focused on optimizing the implementation of advanced DRL frameworks within ViZDoom, a simulation environment that incorporates complex multi-dimensional action controls.

Moreover, collaborative interaction between multiple agents has been demonstrated to be an effective approach for converging toward near-optimal solutions [20,21]. With the advancement of single-agent DRL implementations, multi-agent DRL has sparked a wave of enthusiasm [22,23]. Multi-agent reinforcement learning enables collaboration by maximizing collective rewards, which expands traditional reinforcement learning from individual problem-solving to cooperative achievement [24]. The methodology has shown remarkable effectiveness in advanced applications including multiplayer games [25], coordinated UAV operations [26], and robot swarm control systems [27]. Since real-world scenarios often involve multiple interacting agents, we incorporate a multi-agent framework into the DRL paradigm to improve agent performance in ViZDoom.

In reinforcement learning, developing sophisticated agents capable of replicating human-like behavioral patterns within gaming environments remains a significant challenge. To address this challenge, we introduce MA-PPO, an advanced reinforcement learning algorithm framework specifically designed for complex gaming scenarios. Using the first-person shooter game ViZDoom as our experimental platform, we model environmental interactions as a Partially Observable Markov Decision Process. Our MA-PPO algorithm incorporates an advanced multi-agent architecture that overcomes a key limitation of traditional reinforcement learning: instead of executing only one action per decision, it enables agents to perform multiple commands simultaneously, much like human players. To ensure genuine intelligence rather than scripted behavior, we implement a distributed task allocation mechanism, where agents operate independently on parallel objectives, avoiding control conflicts. Through comprehensive empirical evaluation against other reinforcement learning algorithms, our framework achieves superior performance in the ViZDoom environment.

The main contributions of our work are summarized as follows:

• We transform the interactions between the ViZDoom environment and agents into a Partially Observable Markov Decision Process, and systematically define the corresponding states, actions, and reward functions. The reward functions serve as key indicators for assessing the effectiveness of each agent in executing their designated specialized tasks.
• We propose an innovative multi-agent reinforcement learning algorithm framework MA-PPO based on PPO. Our framework utilizes image information as input and outputs joint actions composed of multiple individual actions to interact with the ViZDoom environment, achieving predefined objectives in a manner that better approximates human players.
• Simulation experiments show that compared to original PPO, MA-PPO achieved a 30.67% reward gain, while compared to three other benchmark algorithms including DQN, MA-PPO achieves at least 32.00% performance improvement. Visual analysis shows that MA-PPO achieves optimal task completion with minimal steps required, and parameter experiment results indicate that all selected parameters are optimal.

The reminder of this paper is organized as follows. Section 2 introduces the related work. Section 3 first introduces the ViZDoom simulator and the multi-agent POMDP formulation. Section 4 describes the update process of the proposed MA-PPO algorithm and the network architecture. Section 5 evaluates the performance of the proposed algorithm through simulations and conducts a parameter experiment. Finally Section 6 concludes the paper. The main symbols used in this paper are summarized in

Table 1. List of notations.

Symbol	Definition
P	State transition probability function
$\gamma$	Discount factor
I, i	Set of agents, agent number
T, t	Termination time, time slot
R, $r_t$	Reward function, reward at time slot t
$r^{move}$, $r^{shoot}$	Penalty of movement and shooting
$r^{hit}$	Reward of hitting the target
S, $s_t$	State space set, state at time slot t
A, $a_t$	Action space set, action made by the agent at time slot t
$A^i$	Action space set of agent i
$a^{stay}$, $a^{left}$, $a^{right}$	Stay still, move to the left or right
$a^{hold}$, $a^{shoot}$	Hold a weapon without firing, fire
$r_t^i$, $s_t^i$, $a_t^i$	Reward, state or action at time slot t of agent i
$\tau$	Trajectory
$V_{\mathit{target}}$	Target value
$\Theta$	Parameter of critic network
$V_{\Theta }\left(s_t\right)$	Value calculated by critic network with $\Theta$
$L_{\mathit{VF}}(\Theta )$	Loss function of critic network with $\Theta$
${\widehat{A}}_t$	Estimated value of advantage function
${\delta }_t$	Measurement of the loss estimated by critic network
$\theta$	Parameter of actor network
${\theta }_{\mathit{old}}$	Old parameter of actor network before updating
${\pi }_{\theta }$	Policy of actor network
${\pi }_{{\theta }_{\mathit{old}}}$	Old policy of actor network before updating
${\pi }_{\theta }(a_t\mid s_t)$	Probability of policy ${\pi }_{\theta }$ taking action $a_t$ at state $s_t$
$p_t\left(\theta \right)$	Ratio of ${\pi }_{\theta }$ to ${\pi }_{{\theta }_{\mathit{old}}}$ in taking action $a_t$ at state $s_t$
$L^{\mathit{clip}}\left(\theta \right)$	Loss function for updating actor network
$p^{\mathit{clip}}\left(\theta \right)$	Value of $p_t\left(\theta \right)$ after clip operation

.

2. Related Work

In this section, we review the related works, including applications of the actor–critic framework in RL.

Although RL has demonstrated remarkable efficacy in practical applications, there remain several critical areas that need further exploration and research. Traditional tabular-based RL methods, i.e., Q-learning and Sarsa, employ a tabular structure to maintain states and actions, facilitating value function retrieval through systematic table consultation. Li et al. introduced Approximate Dynamic Programming (ADP) as an innovative methodology for optimizing long-term operational costs in microgrid systems [28]. The real-time scheduling problem was modeled as a finite-horizon Markov decision process. Through the implementation of neural networks for power generation forecasting, the ADP approach demonstrated effectiveness when evaluated using authentic power grid data simulations. Anderson et al. conducted research on modeling player interactions in StarCraft through Nash equilibrium theory, with findings indicating that the implementation of minimax-Q algorithms led to enhanced performance for AI agents in StarCraft tournaments [29].

However, value function-based tabular RL algorithms are limited to handling only discrete action and state spaces, and encounter the problem of decision fatigue. As a result, the implementation of the actor–critic framework presents an effective solution to address the operational limitations. Wang et al. implemented an actor–critic network in home energy systems to deal with continuous battery charging and discharging actions [9]. The actor–critic-based RL scheme enhances the economic efficiency of battery scheduling, while maintaining safety constraints throughout daily operations. Zha et al. introduced an actor–critic algorithm named Advanced Actor–Critic (AAC) that trains agents to match human-level skill in the complex strategy game StarCraft II [30]. The AAC-based AI demonstrates rapid convergence despite encountering vast state and action spaces.

Although reinforcement learning has advanced significantly across many fields, research has primarily focused on single-agent scenarios, even though real-world applications often involve multiple decision-makers working together [31]. Thus, multi-agent reinforcement learning has emerged as a powerful approach for tackling complex real-world problems [32]. Lowe et al. first introduced the multi-agent actor–critic framework, i.e., multi-agent deep deterministic policy gradient (MADDPG), which handles environmental non-stationarity through a system of multiple trained sub-policies, with random policy selection during each episode [33]. MADDPG, as a versatile multi-agent reinforcement learning framework, has made significant contributions through its centralized training and distributed execution strategy in multi-robot collaborative systems [34], human behavior imitation [35], and Data Center Digital Twins [36].

3. Modeling in ViZDoom

In this section, we introduce the ViZDoom simulator, a gaming platform widely used for training DRL agents. We then present the concept of Partially Observable Markov Decision Process (POMDP) and its extension to a multi-agent framework. Finally, we detail the construction process and theoretical foundations of multi-agent POMDP.

3.1. Preliminaries

3.1.1. ViZDoom

ViZDoom is a reinforcement learning research platform built on a popular open-source 3D first-person shooting game, allowing agents to play Doom by interacting with screen buffers and game variables [37]. From a technical performance perspective, the ViZDoom environment demonstrates excellent optimization results. At a resolution of 320 × 240, the environment achieves an average rendering rate of 2500 frames per second. At a lower resolution of 160 × 120, even single-threaded CPUs can achieve up to 7000 frames per second. For reinforcement learning research, the ViZDoom simulator runs in synchronous mode, where the game engine pauses to await decisions from the learning agent. As a result, the learning system can progress at its optimal pace without time restrictions. Additionally, the environment also comes with multiple pre-built scenarios specifically designed to train fundamental skills like shooting and navigation. Based on the technical advantages, the ViZDoom environment provides an ideal experimental platform for our research.

3.1.2. Reinforcement Learning with POMDP

In the reinforcement learning framework, the environment and agent continuously interact in order to maximize rewards [38]. The Partially Observable Markov Decision Process (POMDP) provides a formal mathematical model for analyzing and characterizing agent–environment interactions, denoted as a five-tuple $<S,A,R,P,\gamma >$.

In POMDP, we denote $s_t\in S$ as the state space set, where $s_t$ refers to the state variables obtained by the agent from the environment at time slot t. We denote $a_t\in A$ as the action space set, where $a_t$ refers to the specific action taken by the agent to interact with the environment at time slot t. Following the execution of an action, the agent receives environmental feedback, which is formally designated as the reward signal. We denote R as the reward function, where $R(s_t,a_t)$ refers to the immediate feedback received by the agent when executing action $a_t$ in state $s_t$ at time slot t. Thus, we denote P as the state transition probability function, where $P\left(s_{t+1}\right|s_t,a_t)$ characterizes the probability distribution of the system transitioning from state $s_t$ to the next state $s_{t+1}$ through action $a_t$. Finally, we denote $\gamma \in [0,1]$ as the discount factor, regulating the trade-off between immediate and long-term rewards. A larger discount factor $\gamma$ emphasizes long-term gains, while a smaller discount factor $\gamma$ focuses more on immediate returns. During the continuous interaction between the agent and environment, the system produces an ordered sequence of state–action pairs, which forms a temporal transition sequence parameterized by the time slot t. We define the sequence of state–action pairs as trajectory

$$\tau =\{s_1,a_1,s_2,a_2,\dots ,s_t,a_t,s_{t+1},a_{t+1},\dots s_T,a_T\}$$

(1)

where T is the termination time of the interaction.

In this paper, we consider a multi-agent POMDP, where I agents share information and collaborate to achieve a common goal. We denote $i\in I$ as the set of agents. All agents share the same state space S and discount factor $\gamma$. Due to different agents having unique action spaces, we redefine the action space A in multi-agent POMDP as a composite action space. We denote $a_t^i\in A^i\in A$ as the action taken by the agent i to interact with the environment at time t; thus, the action space for the whole system satisfies $a_t=\{a_t^{i=1},\dots ,a_t^{i=I}\}$ at time t. Ultimately, the agent i updates its policy ${\pi }^i$ according to different reward functions.

As shown in Figure 1, the environment interaction of the multi-agent system follows the process below: Based on the environment state $s_t$, each agent i independently selects action $a_t^i$, and the individual decisions collectively form the system’s joint action. After the agents execute the joint action $a_t$ and interact with the environment, the state transitions from state $s_t$ to state $s_{t+1}$ and receives a reward $r_t$. Based on the next state $s_{t+1}$ and reward $r_t$, the system continues making decisions, and the cycle ultimately generates a complete interaction trajectory $\tau$ in Equation (1).

3.2. Multi-Agent POMDP Formulation

In the multi-agent POMDP, the agent i first observes the state $s_t$ in the ViZDoom environment, then selects corresponding actions $a_t^{i=1}$ and $a_t^{i=2}$ from the action spaces $A^{i=1}$ and $A^{i=2}$, respectively. The system then executes the combined joint action $a_t$ in the ViZDoom environment. Following execution, the system receives both the next state $s_{t+1}$ and reward $r_{t+1}$ from the environment interaction before proceeding to the next decision cycle. Since the ViZDoom simulator updates all agent-observable information in real-time, the next state space $s_{t+1}$ is independent of previous states $s_t$ and actions $a_t$, satisfying the MDP property. The POMDP formulation is as follows.

3.2.1. State

We adopt the ViZDoom Basic environment as the experimental platform. The system is configured with a dual-agent architecture, i.e., $I=2$. Both agents share the same state observation information, satisfying

$$s_t^{i=1}=s_t^{i=2}$$

(2)

The state space is composed of game images directly obtained by the multi-agent system, which is a discrete state space. The state space consists of a series of post-interaction images obtained directly from the ViZDoom environment, which is a discrete state space containing key decision-making information, such as player weapon orientation, enemy positions, ammunition levels, terrain features, and player health status.

3.2.2. Action

To align with the ViZDoom environment, we define four basic operations for the agents, i.e., stay still, move left, move right, and fire. We construct a multi-agent system composed of two agents with differentiated action spaces to control the player unit in the environment. For the agents i, we establish a clear division of action spaces, denoted as

$$\begin{array}{cc}\hfill & a_t^{i=1}\in A^{i=1}=\{a^{stay},a^{left},a^{right}\}\hfill \\ \hfill & a_t^{i=2}\in A^{i=2}=\{a^{hold},a^{shoot}\}\hfill \end{array}$$

(3)

where the agent $i=1$ is responsible for controlling the player’s position movement, and its action space contains three actions: $a^{stay}$ represents the agent maintaining a stationary position without movement, $a^{left}$ represents the agent moving to the left, and $a^{right}$ represents the agent moving to the right. The second agent is specifically responsible for shooting decisions, with its action space containing two actions: $a^{hold}$ represents the agent holding a weapon without firing, and $a^{shoot}$ represents the agent holding a weapon and firing. Through the collaborative mechanism, the multi-agent system achieves complete control over player behavior and interacts with the environment.

3.2.3. Reward

The reward function settings adopted in our research represent the measurement of the multi-agent system’s actual performance in the game. To achieve differentiated behavior guidance, we establish different evaluation criteria for each agent in the system, as follows:

$$\begin{array}{cc}\hfill & r_t^{i=1}=r^{move}+r^{hit}\hfill \\ \hfill & r_t^{i=2}=r^{shoot}+r^{hit}\hfill \end{array}$$

(4)

The reward function adopted in this paper represents the measurement of the multi-agent system’s actual performance in the game. When the system successfully hits the target, it receives a positive reward $r^{hit}$. Meanwhile, we introduce $r^{move}$ and $r^{shoot}$ as behavioral penalty factors, corresponding to the costs of movement and shooting actions. These two penalties are designed to optimize the system’s behavioral strategy, encouraging agents to adopt more strategic movement patterns with the aim of achieving more efficient shooting performance under limited ammunition conditions.

4. Proposed Algorithm

The core scheme adopted in this paper is the Proximal Policy Optimization (PPO) algorithm. In this section, we first extend the PPO algorithm to a multi-agent architecture. Then, we present a comprehensive analysis of the optimization mechanisms underlying these algorithm variants, along with a detailed examination of their respective network architectures.

4.1. Proximal Policy Optimization

As an advanced method in DRL, PPO achieves high training efficiency while maintaining learning stability, making it particularly suitable for agent decision training in complex environments. PPO is based on an actor–critic architecture design, where the critic network is responsible for evaluating the state value function and plays an evaluative role in the policy optimization process. Within the ViZDoom environment, the agent i perceives the state $s_t^i$ at the beginning of the time slot t. Subsequently, the agent i engages with the environment by selecting and executing the action from a discrete set $a_t^i\in A^i$. Then, the agent i observes an immediate reward $r_t^i$ from the ViZDoom environment before transitioning to the next state $s_{t+1}^i$. Through agent–environment interactions, we obtain the interaction trajectory $\tau$. We input the state transition and corresponding reward from the trajectory $\tau$ into the critic network to calculate the state target value $V_{\mathit{target}}$:

$$V_{\mathit{target}}=r_t+{\gamma }^V·V_{\Theta }\left(s_{t+1}\right)$$

(5)

where ${\gamma }^V$ is the discount factor for value function and $\Theta$ represents the weight of the critic network. Based on the critic network’s estimated value for the current state and the target value $V_{\mathit{target}}$, we construct the following loss function:

$$L_{\mathit{VF}}(\Theta )={\widehat{\mathbb{E}}}_t\left[{\left(V_{\Theta }\left(s_t\right)-V_{\mathit{target}}\right)}^2\right]$$

(6)

By applying the Adam optimizer, we minimize the loss function $L_{\mathit{VF}}(\Theta )$ to update the weight of the critic network $\Theta$. The loss function $L_{\mathit{VF}}(\Theta )$ serves to minimize the disparity between the estimated value of critic model and the target value, thereby enhancing the model’s capacity to accurately assess the quality of the current state.

In terms of the actor network, its main responsibility is to generate action decisions. To optimize the actor network, we first need to use the Generalized Advantage Estimator (GAE) method to calculate the advantage estimate value ${\widehat{A}}_t$:

$${\widehat{A}}_t={\delta }_t+({\gamma }^A·\lambda ){\delta }_{t+1}+\cdots +{({\gamma }^A·\lambda )}^{T-t+1}{\delta }_{T-1}$$

(7)

where ${\delta }_t=r_t+{\gamma }^A·V_{\Theta }\left(s_{t+1}\right)-V_{\Theta }\left(s_t\right)$. During the estimation process, the state values $V_{\Theta }\left(s_{t+1}\right)$ and $V_{\Theta }\left(s\right)$ are provided by the critic network, ${\gamma }^A$ is the discount coefficient for the advantage function, and $\lambda$ is used to balance bias and variance.

We denote the weights of the actor network as $\theta$, and the network weights before updating as ${\theta }_{\mathit{old}}$. To prevent drastic gradient fluctuations during parameter updates, the PPO algorithm employs a clipping mechanism to constrain the difference between new and old policies ${\pi }_{\theta }$ and ${\pi }_{{\theta }_{\mathit{old}}}$. Let ${\pi }_{\theta }\left(a_t\mid s_t\right)$ denote the probability of policy ${\pi }_{\theta }$ selecting action $a_t$ under state $s_t$, and define the probability ratio between new and old policies as $p_t\left(\theta \right)={\displaystyle \frac{{\pi }_{\theta }\left(a_t\mid s_t\right)}{{\pi }_{{\theta }_{\mathit{old}}}\left(a_t\mid s_t\right)}}$. Therefore, we adopt the following loss function for the PPO scheme:

$$L^{\mathit{clip}}\left(\theta \right)={\widehat{\mathbb{E}}}_t\left[min\left(p_t\left(\theta \right){\widehat{A}}_t,p^{\mathit{clip}}\left(\theta \right){\widehat{A}}_t\right)\right]$$

(8)

where the clip function $p^{\mathit{clip}}\left(\theta \right)$ is denoted as

$$p^{\mathit{clip}}\left(\theta \right)=\left\{\begin{array}{c}1-ϵ, \mathit{if} p_t\left(\theta \right)<1-ϵ\hfill \\ p_t\left(\theta \right), \mathit{if} 1-ϵ<p_t\left(\theta \right)<1+ϵ\hfill \\ 1+ϵ, \mathit{if} 1+ϵ<p_t\left(\theta \right)\hfill \end{array}\right.$$

(9)

Figure 2 shows how the loss function $L^{\mathit{clip}}\left(\theta \right)$ varies with the probability ratio $p_t$ under different advantage functions. The loss function $L^{\mathit{clip}}\left(\theta \right)$ aims to provide the actor model with insights regarding policy advantages while limiting the magnitude of single updates to ensure controlled optimization. Here, $ϵ$ is a hyper-parameter used to limit the clipping range. Finally, we update the weight of the actor network $\theta$, utilizing the Adam optimizer based on the loss function $L^{\mathit{clip}}\left(\theta \right)$.

4.2. Multi-Agent-Based RL Framework for ViZDoom

We systematically preprocess the game screen output from the ViZDoom environment to optimize the model input. Specifically, the ViZDoom simulator outputs a three-channel color image with a resolution of $240\times 320$. After selective cropping and gray scale conversion, the image is resized to a single-channel feature of $84\times 84$. To enhance temporal information expression, we concatenate four frames of images together, ultimately forming a state input with four channels of $84\times 84$ features. The preprocessing scheme effectively preserves the dynamic characteristics of the environment while providing sufficient contextual information for the agent.

It is worth noting that only one frame of the image is available in the game’s initial state. Therefore, when processing the initial state, we replicate the first frame four times and concatenate them to construct the initial state $s_0$. It not only ensures the uniformity of input dimensions but also provides a stable and coherent learning foundation for the model. Through the above standardized processing, we achieve continuous and consistent state representations from the beginning to the end of the game, and provide the agent with a stable learning starting point.

In the actor network architecture responsible for shooting control, we use the preprocessed game information as the model input. Then, it goes through multiple convolutional network processing: in the first convolutional layer, a 5 × 5 kernel operates with a stride of 2, followed by a ReLU activation function; in the second convolutional layer, a 4 × 4 kernel operates with a stride of 2, followed by a ReLU activation function; in the third convolutional layer, a 3 × 3 kernel operates with a stride of 2, followed by a ReLU activation function; and in the fourth convolutional layer, a 3 × 3 kernel operates with a stride of 2, producing output dimensions of (batch size, 32, 9, 9). Subsequently, the output is flattened through a flatten layer, and then passes through three fully connected layers, i.e., outputting 512, 64, and 2 units, respectively. Each is followed by a ReLU activation function. Finally, a softmax layer generates an action probability distribution to determine whether to fire.

The actor network architecture for movement control is essentially identical to the shooting model, with only the output dimension in the final layer changed to 3, used to control left movement, right movement, or staying still. The two agents share the same critic network structure. It maintains the same input processing method and convolutional layer configuration, and through a similar network architecture, ultimately outputs a single state value estimate. Specifically, the critic network’s input will go through four convolutional layers. After flattening, it passes through fully connected layers with dimensions of 512, 64, and 1, each followed by a ReLU activation function, ultimately outputting the state value assessment. The specific process is shown in Figure 3.

According to the MA-PPO training process in Algorithm 1, we extend the original PPO algorithm to implement a dual-agent collaborative control system. Compared to the PPO pseudocode table Algorithm A1, we design the simultaneous training and updating of multiple agents in the MA-PPO training process. Two independently trained agents jointly control a game character, with their policy updates occurring relatively independently. The algorithm accepts several input variables, i.e., the maximum number of epochs $\eta$, maximum number of steps $\xi$, maximum number of training repetitions $\upsilon$, and environment state $s_t$, and then outputs the control strategy. During practical application, the agent accepts the same inputs to generate joint actions $a_t$ as output. The training procedure follows the process below as follows. After initializing the system parameters and model architecture for the two agents in the system, a training loop is entered into, consisting of $\eta$ epochs, with each epoch containing $\xi$ training steps. During the training process, the agent first interacts with the environment until the end of a game round, then the system records the interaction trajectory $\tau$ and updates the counter, followed by $\upsilon$ iterations of optimization on the collected data for both agents. During the optimization phase, the system first calculates the state target value $V_{\mathit{target}}$ based on trajectory $\tau$, constructs the critic network’s loss function $L_{\mathit{VF}}(\Theta )$, and updates the relevant parameters through the Adam optimizer. Next, the system evaluates the advantage function ${\widehat{A}}_t$ based on trajectory $\tau$, which is used to construct the actor network’s loss function $L^{\mathit{clip}}\left(\theta \right)$, and similarly employs the Adam optimizer for parameter updates. Finally, the control strategy $\pi$ is formed. These steps collectively form the core optimization mechanism of the MA-PPO algorithm.

Algorithm 1 Multi-Agent Proximal Policy Optimization.

Input: state from environment $s_t$, $\eta$, $\xi$, $\upsilon$;

Output: control strategy;

1: for agent $i=1,2$ do   2:       initialize critic parameters $\Theta$ and actor parameters $\theta$   3:       initialize the learning rate ${\beta }^{lr}$ and optimizer   4: EndFor   5: for epoch $k=1,2,\cdots ,\eta$ do   6:       for step $t=1,2,\cdots ,\xi$ do   7:             play the game under parameters $\Theta$ and $\theta$   8:             for game is not over do   9:                   the agent obtains environment information 10:                   the agent executes actions 11:             EndFor 12:             gain the trajectory $\tau$ 13:             record the game steps 14:             for agent $i=1,2$ do 15:                   for repeat training times $j=1,2,\cdots ,\upsilon$ do 16:                         compute $V_{\mathit{target}}$ according to Equation (5) 17:                         compute $L_{\mathit{VF}}(\Theta )$ according to Equation (6) 18:                         update parameters of critic network $\Theta$ 19:                         compute ${\widehat{A}}_t$ according to Equation (7) 20:                         compute $L^{\mathit{clip}}\left(\theta \right)$ according to Equation (8) 21:                         update parameters of actor network $\theta$ 22:                   EndFor 23:             EndFor 24:       EndFor 25: EndFor 26: form a control strategy $\pi$ for the whole system

5. Results and Analysis

5.1. Experiment Setup

To assess the efficacy of our multi-agent deep reinforcement learning framework in first-person shooting scenarios, we conduct comprehensive experiments utilizing the open-source platform, VizDoom. The simulation is conducted on a workstation with an Intel Core i7-9700K processor which is manufactured by Intel Corporation in Santa Clara, CA, USA. The operation system is Ubuntu 20.04.6 LTS (Focal Fossa).  The code is written in Python 3.6.0 using Pytorch 1.6.1 and ViZDoom 1.2.4. The parameters are listed in

Table 2. Parameters of the MA-PPO training settings.

Parameter	Description	Value
${\beta }^{lr}$	Learning rate	[1 × 10−7, 1 × 10−5]
${\gamma }^V$	Discount factor for calculating target value	0.98
${\gamma }^A$	Discount factor for calculating advantage function	0.98
$\lambda$	Bias between 0 and 1	0.95
$ϵ$	Range of clip operation	0.2
$\eta$	Training epoch	1000
$\xi$	Training step	2000
$\upsilon$	repeat training times	10

. The learning rate ${\beta }^{lr}$ gradually decreases from 1 × 10⁻⁵ to 1 × 10⁻⁷. The discount factors ${\gamma }^A$ and ${\gamma }^V$ are both 0.98 according to Section 5.3.2. Bias $\lambda$ is multiplied with discount factor ${\gamma }^A$ when calculating the advantage function. Following the recommendation in the PPO, we set the range of clip operation $ϵ$ = 0.2. For the training iteration variables $\eta$, $\xi$, and $\upsilon$, we select values that enable algorithm convergence during the actual training process.

5.2. Performance Evaluation

5.2.1. Baseline

We employ five baseline algorithms for experimental evaluation and comparative analysis, i.e., PPO, MA-PPO, A2C, MA-A2C, and DQN. The following will elaborate on the characteristics and implementation methods of each algorithm.

Proximal Policy Optimization (PPO) [39]: PPO was proposed by Schulman et al. in 2017, and is an advanced reinforcement learning method. PPO adopts an actor–critic architecture, which can be applied to both discrete and continuous action spaces, demonstrating strong versatility in reinforcement learning problems. The most distinctive feature of PPO lies in the clip operations, which limits the magnitude of single policy updates, greatly enhancing its stability. Therefore, we develop MA-PPO, extending the PPO algorithm for application within a multi-agent system framework. Compared to the original PPO algorithm, we change the task allocation structure and design two agents responsible for movement and shooting, respectively. The two agents are both updated according to the PPO algorithm and jointly control the player’s behavior. The specific algorithm details are elaborated on in Section 4.

Advantage Actor–Critic (A2C) [9]: The A2C algorithm is an efficient policy optimization method, which employs an actor–critic architecture. The algorithm integrates policy gradient methods with value function estimation, and introduces advantage functions to optimize the training process. The actor network is responsible for action decision-making and updates based on the advantage function, while the critic network evaluates action values and optimizes parameters in a manner similar to PPO. Correspondingly, MA-A2C is our improved version of the A2C algorithm after adapting it to a multi-agent framework.

Deep Q-Network (DQN) [40]: DQN is a deep reinforcement learning algorithm developed by the DeepMind team in 2013. DQN approximates the Q-value function through deep neural networks to handle complex high-dimensional input data. DQN introduces an experience replay mechanism, storing the agent’s interactions with the environment in a replay buffer, and conducts training through random sampling, effectively reducing data correlation and significantly improving training efficiency and stability.

5.2.2. Convergence

We evaluate the convergence performance of the proposed multi-agent architecture through a training period of 1000 epochs. We conduct systematic comparative analysis with PPO, MA-PPO, A2C, MA-A2C, and DQN to validate the performance advantages of our architecture.

In the analysis of Figure 4a, the results show that MA-A2C demonstrates optimal performance during the convergence phase, followed by DQN and A2C. DQN demonstrates significant rapid convergence characteristics, reaching a stable state at around 50 epochs. The efficient convergence performance can be attributed to its streamlined algorithm structure and lower parameter complexity. A2C converges and maintains stability around epoch 80. The MA-A2C learning curve exhibits unique dynamic characteristics. The convergence curve exhibits a significant drop at epoch 86, i.e., reward value −313, followed by a strong recovery. MA-A2C surpasses the performance levels of A2C at epoch 130 and DQN at epoch 140, before achieving stable convergence at epoch 150. The performance curve characteristics reflect the collaborative learning challenges of the multi-agent system in the initial phase and the effective cooperation mechanism subsequently established through continuous environmental interaction. The results in Figure 4b show that MA-PPO consistently maintains a leading advantage, while PPO performs better than DQN after convergence. Specifically, PPO achieves initial convergence at epoch 30, followed by a continuous optimization trend, surpassing DQN in overall performance at epoch 250. MA-PPO reaches convergence at epoch 50, demonstrating excellent performance stability.

The comprehensive evaluation results in Figure 4c indicate that MA-PPO demonstrate far superior convergence compared to the other four baselines, i.e., PPO, MA-A2C, DQN, and A2C. After convergence, the rewards for MA-PPO, PPO, MA-A2C, and DQN are 75, 52, 51, and 31, respectively. Quantitative analysis shows that MA-PPO achieves test performance improvements of 30.67%, 32.00%, and 58.67% compared to PPO, MA-A2C, and DQN, respectively. Meanwhile, compared to their original versions, both the MA-PPO algorithm versus the PPO algorithm, and the MA-A2C algorithm versus the A2C algorithm show highly significant improvements in reward data from the experiments. Therefore, the experimental data confirm that the algorithm variants based on the multi-agent architecture, i.e., MA-PPO and MA-A2C, achieve significant performance improvements compared to their original versions, fully validating the effectiveness of the proposed multi-agent architecture in optimizing reinforcement learning performance.

5.3. Performance Comparison

5.3.1. Rendering

Figure 5 shows the results of our visualization-based comparative analysis of the five baselines’ performance in the game environment. As shown in

Table 3. ViZDoom testing results for MA-PPO and other baselines.

Scheme	Average Reward ↑	Average Step ↓	Ammunition (AMMO) ↑
PPO	56	35	47
A2C	−109	165	40
DQN	36	55	47
MA-A2C	47	44	47
MA-PPO (Ours)	76	20	48

, the multi-agent architecture MA-PPO demonstrates optimal performance, efficiently completing the task with a reward of 76, 20 action steps, and 48 remaining ammunition. The PPO algorithm is ranked second, achieving a reward of 56, 35 action steps, and 47 remaining ammunition. MA-A2C achieves a reward of 47, takes 44 steps, and maintains 47 remaining ammunition. DQN achieves a reward of 36, took 55 steps, and has 47 ammunition remaining. The A2C algorithm shows relatively weak performance, obtaining a reward of −109, taking 165 steps, and having 40 ammunition remaining.

Then, we conduct a detailed analysis of the visualization-based results from the perspective of each agent’s actions. MA-PPO executes shooting actions at steps 7 and 20, successfully hitting the enemy at step 20, while rapidly approaching the target during the intermediate phases. PPO executes shooting actions at steps 12, 25, and 35, successfully hitting the enemy at step 35, while slowly approaching the target during the intermediate phases. MA-A2C executes shooting actions at steps 10, 21, and 44, successfully hitting the enemy at step 44, with a brief stationary period between steps 27 and 32, while slowly approaching the target during the intermediate phases. DQN executes shooting actions at steps 20, 37, and 55, successfully hitting the enemy at step 55, with a brief stationary period between steps 25 and 35, while slowly approaching the target during the intermediate phases. A2C executed a total of 10 shots, hitting the enemy at step 165, with the intermediate phases involving frequent lateral oscillations and stationary periods. The experimental data clearly demonstrate that algorithm variants based on multi-agent architecture, i.e., MA-PPO and MA-A2C, outperform their corresponding baselines across all performance metrics, with particularly notable improvements in the case of the A2C algorithm. The empirical results strongly confirm that the proposed multi-agent system has significant advantages in improving agent game performance.

5.3.2. Parameter Experiments

We conduct systematic parameter experiments on the discount factors ${\gamma }^A$, ${\gamma }^V$ for estimating target value $V_{\mathit{target}}$ and advantage function $\widehat{A_t}$ to evaluate how their different combinations affect model training. Through the analysis of 1000 training epochs, we obtain the following key findings.

Figure 6a shows the impact of different ${\gamma }^A$ values (0.98, 0.95, 0.90, and 0.85) on model convergence when ${\gamma }^V$ is fixed at 0.98. The experimental results show that when ${\gamma }^A=0.98$, it not only demonstrates optimal training stability but also achieves the highest reward value. The data show that as the ${\gamma }^A$ value approaches 1, the model performance becomes more superior.

Figure 6b shows the experimental results with different ${\gamma }^V$ values (0.98, 0.95, 0.90, and 0.85) when ${\gamma }^A$ is fixed at 0.98. The study finds that changes in ${\gamma }^V$ can lead to significant fluctuations in the model training process, but as the ${\gamma }^V$ value approaches 1, the model training becomes more stable and can achieve higher reward levels.

Figure 6c visualizes the effects of 16 combinations of ${\gamma }^A$ and ${\gamma }^V$ through a heatmap, where darker regions represent superior model performance. The experimental results clearly demonstrate that the parameter combination of ${\gamma }^A=0.98$ and ${\gamma }^V=0.98$ produces the best performance, indicating that the combination achieves an ideal balance between bias and variance in reward function estimation. Above all, we accordingly determine the final parameter configuration.

6. Conclusions

In this paper, we propose a multi-agent reinforcement learning framework called MA-PPO that achieves predetermined objectives through continuous interaction with the ViZDoom environment under specific constraints. Specifically, the primary objective is to achieve target acquisition with optimal efficiency while adhering to specified ammunition and time limitations. Comprehensive performance evaluation shows that the improved MA-PPO algorithm achieves a 30.67% improvement in convergence performance compared to the baseline algorithm. Through systematic parameter experiments, we validate the scientific basis and effectiveness of our chosen discount factor parameters. Through a series of simulation experiments, it is demonstrated that the proposed multi-agent architecture has good generality and can be integrated with various actor–critic frameworks to enhance overall performance.

Future work will be expanded in the following direction. For practice, we will explore the potential applications of multi-agent reinforcement learning algorithms in broader and more complex game environments to validate their generality and practical application value. We plan to conduct in-depth research on optimizing the decision-making mechanisms and develop algorithm architectures that better adapt to diverse gaming environments. Furthermore, we will focus on resource efficiency issues in multi-agent systems, emphasizing research on the optimal balance point between agent scale expansion and computational resource consumption, aiming to achieve an ideal trade-off between performance improvement and computational efficiency.