Research on Autonomous Manoeuvre Decision Making in Within-Visual-Range Aerial Two-Player Zero-Sum Games Based on Deep Reinforcement Learning

Abstract

In recent years, with the accelerated development of technology towards automation and intelligence, autonomous decision-making capabilities in unmanned systems are poised to play a crucial role in contemporary aerial two-player zero-sum games (TZSGs). Deep reinforcement learning (DRL) methods enable agents to make autonomous manoeuvring decisions. This paper focuses on current mainstream DRL algorithms based on fundamental tactical manoeuvres, selecting a typical aerial TZSG scenario—within visual range (WVR) combat. We model the key elements influencing the game using a Markov decision process (MDP) and demonstrate the mathematical foundation for implementing DRL. Leveraging high-fidelity simulation software (Warsim v1.0), we design a prototypical close-range aerial combat scenario. Utilizing this environment, we train mainstream DRL algorithms and analyse the training outcomes. The effectiveness of these algorithms in enabling agents to manoeuvre in aerial TZSG autonomously is summarised, providing a foundational basis for further research.

1. Introduction

Unmanned Aerial Systems (UASs) (see Glossary) are increasingly vital in modern life, primarily operating via ground station remote control with human oversight for tasks like reconnaissance, surveillance, and ground attacks. However, advancements in networked, information-based, and intelligent scenarios have reduced aerial response times and intensified manoeuvres beyond human capabilities [1]. Enhancing UAS intelligence is crucial to adapting and gaining an advantage in aerial engagements. This involves enabling UASs to autonomously generate control commands based on situational awareness for aerial combat, making UAS intelligence a key future research direction.

Autonomous decision making is essential for the effectiveness of UASs, involving the real-time generation of optimal tactical commands in complex aerial combat environments [2,3,4,5]. While various zero-sum game decision-making methods have shown success under specific conditions, comprehensive methods for different scenarios and intelligent agents remain undeveloped [6]. The current methods are categorised into traditional and intelligent decision-making approaches.

Traditional autonomous decision-making methods include differential and matrix game methods [7,8,9], expert systems [10,11,12,13], and influence diagrams [14]. Recent advancements include Soltanifar [15], Farajpour Khanaposhtani [16], Ismail et al. [17], and Hosseini Nogourani et al. [18], focusing on multi-attribute decision making, support vector machines, DEMATEL with Bonferroni mean aggregation, and decision making under uncertainty and risk. However, these methods often suffer from high computational demands, reliance on manually set rules, and inflexibility. Reinforcement learning (RL), based on Markov decision processes (MDP), addresses these issues by enabling agents to interact with the environment and optimize strategies through feedback. Though suitable for finite MDPs, RL struggles with continuous spaces. Deep reinforcement learning (DRL) [19,20,21,22,23] combines deep learning’s perception capabilities with RL’s decision making, proving effective for complex problems like aerial manoeuvre decision making. Pan Y et al. [24] used DRL for zero-sum game scenarios with a dual-network Deep Q-network (DQN). Liu P et al. [25] applied DQN for continuous state inputs. Zhang Q et al. [26] used DQN for beyond-visual-range scenarios, addressing the Nash equilibrium. B. Kurniawan et al. [27] introduced the Actor–Critic (AC) architecture for improved training efficiency. Yang Q et al. [28,29] employed the Deep Deterministic Policy Gradient (DDPG) algorithm for continuous action outputs, enhancing control precision and smoothness.

This paper particularly focuses on Witness Visual Range (WVR) (see Glossary) scenarios due to their unique challenges, such as real-time decision making and high-speed manoeuvring. These complexities necessitate advanced decision algorithms. This paper advances research in deep reinforcement learning for close-range aerial zero-sum games (WVR). Key contributions include (1) establishing a kinematic model and a coarse-grained effective capture region model for WVR aerial zero-sum games; (2) constructing an MDP model for WVR games, facilitating deep reinforcement learning with reward shaping to address sparse rewards in autonomous manoeuvre decision making; (3) incorporate aerial agent manoeuvring into flight control mechanisms modelled on a typical UAS to expand the operational space and enhance AI exploration. A typical WVR aerial two zero-sum games (TZSGs) (see Glossary) is designed, and mainstream DRL algorithms are applied to a realistic simulation environment. The experimental results are analysed to summarise the characteristics of these algorithms and provide insights for further optimisation.

2. A Deep Reinforcement Learning Model for Manoeuvre Decision Making in WVR Aerial Agent Zero-Sum Games

2.1. WVR Aerial Two-Player Zero-Sum Games Dynamics Modeling

Figure 1 illustrates the 1v1 WVR aerial TZSG model involving intelligent agents. Agent (r) represents our side, while agent (b) represents the opponent. The OXYZ coordinate system is a three-dimensional Cartesian coordinate system in which the origin O denotes the coordinate origin, typically located within the defined engagement area. In this context, the geometric centre of the aerial combat zone serves as the origin. The positive X-axis points east, the positive Y-axis points north, and the positive Z-axis points vertically upward.

As shown in Figure 1, assuming the zero angle of attack and sideslip angle, the agent’s velocity and body coordinate systems coincide, aligning the agent’s axis with its velocity direction. Agent (r) has velocity ${\overrightarrow{v}}_r$ and position $c_{\mathrm{R}}=\left(x_r,y_r,z_r\right)$. Agent (b) has velocity ${\overrightarrow{v}}_b$ and position $c_{\mathrm{B}}=\left(x_b,y_b,z_b\right)$. The distance d is the separation between agents (r) and (b). The target line connects their centres of mass. The target azimuth angle $\theta$ is the angle between ${\overrightarrow{v}}_r$ and the target line, positive for rightward deviation, with $0⩽\left|\theta \right|⩽{180}^{\circ }$. The target entry angle $\phi$ is the supplementary angle between ${\overrightarrow{v}}_b$ and the target line, also positive for rightward deviation, with $0⩽\left|\phi \right|⩽{180}^{\circ }$. The relative situation is described by $c_{\mathrm{R}}$, $c_{\mathrm{B}}$, $d$, $\theta$, and $\phi$:

$$d=\parallel c_{\mathrm{R}}-c_{\mathrm{B}}{\parallel }_2=\sqrt{{\left(x_r-x_b\right)}^2+{\left(y_r-y_b\right)}^2+{\left(z_r-z_b\right)}^2}$$

(1)

$$\left|\theta \right|=\left|\arccos \left(\frac{v_{\mathrm{R}}\cdot \left(c_{\mathrm{B}}-c_{\mathrm{R}}\right)}{\left|v_{\mathrm{R}}\right|\left|c_{\mathrm{B}}-c_{\mathrm{R}}\right|}\right)\right|\frac{180}{\pi }$$

(2)

$$\left|\phi \right|=\left|\arccos \left(\frac{v_{\mathrm{B}}\cdot \left(c_{\mathrm{B}}-c_{\mathrm{R}}\right)}{\left|v_{\mathrm{B}}\right|\left|c_{\mathrm{B}}-c_{\mathrm{R}}\right|}\right)\right|\frac{180}{\pi }$$

(3)

2.2. Capture Zone Modelling

The agent’s capture zone (see Glossary), which constrains manoeuvring decisions, is modelled using the radar radiation area centred on the agent, as shown in Figure 2. In Figure 2, $D_{max}$ and $q_{max}$ represent the maximum effective range and spread angle of the capture zone, respectively.

The maximum effective range $D_{max}$ and the maximum dispersion angle $q_{max}$ define the capture zone. It is independent of agent speed and target azimuth angle, changing only with the agent’s longitudinal axis. Both the distance and angle conditions must be met, as shown in Equation (4):

$$\left\{\begin{array}{c}d\le D_{max}\\ \theta \le q_{max}\end{array}\right.$$

(4)

In the equation, $d$ represents the distance between the red and blue agents, and $\theta$ denotes the target azimuth angle. Victory is achieved when any agent satisfies Equation (4).

2.3. WVR Aerial TZSG MDP Modelling

In an aerial TZSG, an agent’s autonomous manoeuvring is modelled using a Markov decision process (MDP) defined by the quintuple $〈S,A,P,r,\gamma 〉$. The agent interacts with the environment, obtaining state information $s_t\in S$. Based on $s_t$, the agent performs a manoeuvre $a_t\in A$, receives a reward $r\left(s_t,a_t\right)$, and the environment updates to $s_t+1\in S$. Assuming an ideal model with deterministic dynamics $P\left(\cdot \mid \left(s,a\right)\right)=1$, the agent seeks a strategy π to maximize the cumulative reward ${\left.\sum _0^{\mathrm{\infty }}{ɣ}^{t}r\left(s_t,a_t\right)\right|}_{a_t=\pi \left(s_t\right)}$ where $\gamma \in \left[0,1\right]$ is the discount factor.

2.3.1. State Space (S)

Based on the analysis of the agent’s WVR aerial TZSG model, as shown in Figure 1, the factors affecting the agent’s TZSG are the state characteristics of the fighters. These include the coordinates of the red agent $c_{\mathrm{R}}=\left(x_r,y_r,z_r\right)$, the coordinates of the blue agent $c_{\mathrm{B}}=\left(x_b,y_b,z_b\right)$, the speed of the red agent $v_r$, the speed of the blue agent $v_b$, the target azimuth $\theta$, and the target entry angle $\phi$. The state space of the agent’s WVR aerial TZSG manoeuvre decision making deep reinforcement learning model can be represented as follows:

$$S=\left(x_r,y_r,z_r,x_b,y_b,z_b,v_r,v_b,\theta ,\phi \right)$$

(5)

Since the state space of a fighter aircraft is continuous and infinite, a deep learning neural network is required to process these features.

2.3.2. Action Space (A)

References [30,31] define seven fundamental manoeuvre actions for an agent’s flight states, forming a manoeuvre action library: (1) steady flight, (2) max load factor increase, (3) max load factor decrease, (4) max throttle increase, (5) max throttle decrease, (6) max roll angle increase, and (7) max roll angle decrease. This paper uses this library for game-theoretic decision making. The figure below shows these actions.

Figure 3 depicts that in one time step $\mathsf{\Delta }t$, the seven basic flight manoeuvres that a UAV can execute within a time step are as follows:

Using seven basic manoeuvre combinations simplifies calculations and allows for a flexible representation of most flight attitudes. The core of manoeuvre decision making is choosing the appropriate basic manoeuvre to maximise combat advantage in the current state.

2.3.3. Rewards

Training a manoeuvre guidance agent using deep reinforcement learning (DRL) in aerial zero-sum games faces two main challenges: the sparse reward problem, where success is only measured by achieving an advantageous position, leading to slow convergence, and the need to consider both the time to reach this position and the quality of the trajectory in practical scenarios. A reward-shaping method is proposed to address these issues and improve training speed and strategy performance.

Before reward shaping, advantageous locations/areas need to be defined.

Figure 4 illustrates the dominant region (see Glossary) in games according to the definition provided in the reference [4].

As shown in Figure 2, the smaller the red agent’s target azimuth angle $\theta$, the easier it is for the blue agent to aim at and destroy the red agent, gaining a combat advantage. Conversely, when the red agent’s target entry angle $\phi$ is larger, it becomes harder for the red agent to aim at the blue agent, making the blue agent safer. Additionally, the capture zone of the agents is crucial. Reward shaping [32] is used to modify the reward function to promote learning while maintaining an optimal policy. In this study, the reward function is defined as follows:

(1)

Environmental reward function

The environmental reward function is shown below:

$${\mathrm{R}}_{env}\left(\mathrm{s}\right)={\mathrm{R}}_T\left(\mathrm{s}\right)+{\mathrm{R}}_C\left(\mathrm{s}\right)$$

(6)

where ${\mathrm{R}}_T\left(\mathrm{s}\right)$ is the termination reward function given to the AI at the end game moment and ${\mathrm{R}}_C\left(\mathrm{s}\right)$ are the reward functions given to the basic information of the matchmaking, which are, respectively:

$${\mathrm{R}}_T\left(\mathrm{s}\right)=\left\{\begin{array}{c}100 Red Win\\ -100 Blue Win\\ 0 \mathrm{others}\end{array}\right.$$

(7)

(2)

Shaping the reward function

A real-valued function $R_{situ}\left(s\right)$ is a function of the dominance of a state. The larger the value of this function, the more favourable the current situation is. This function can provide additional information to help the agent choose an action, which is better than using only termination rewards. It is defined as follows:

$$R_{situ}\left(s\right)=t\times \left(R_a\left(s\right)\times R_d\left(s\right)\right)$$

(8)

The T-value is tentatively set at 100. We need to tune the parameter.

• where $R_d\left(s\right)$ is the distance reward function, and $R_a\left(s\right)$ is the direction reward function, which are defined as

$$R_a\left(s\right)=1-\frac{\left|\mathrm{ATA}\right|+\left|\mathrm{AA}\right|}{{180}^{\circ }}$$

(9)

$$R_d\left(s\right)=\exp \left(\frac{-\left|d_t-\left(d_{max}+d_{min}\right)/6\right|}{{180}^{\circ }k}\right)$$

(10)

where k is in metres/degrees, used to adjust the relative effect of distance and angle; the value is tentatively set to 10 needed to adjust the reference; in this case, $d_{max}=1000$, $d_{min}=10$.

(3)

Total reward function

$$R\left(s\right)=R_{situ}\left(s\right)+{\mathrm{R}}_{env}\left(\mathrm{s}\right)$$

(11)

3. Deep Reinforcement Learning Algorithms

3.1. Research Proposal

This paper advances the field by implementing deep reinforcement learning algorithms for WVR aerial TZSG manoeuvre decision making in three-dimensional space, addressing the limitations of current two-dimensional studies in the literature. The intelligent decision module effectively enhances the autonomy of aerial agents, improving mission efficiency and adaptability. Figure 5 illustrates the module, which evaluates the situation of agents and targets, inputs the data, and generates control commands to guide our agents.

3.2. Principles of Reinforcement Learning

Reinforcement learning (RL) uses a “trial and error” approach to interact with the environment, characterised by a Markov decision process [33]. It evaluates actions based on the cumulative payoff expectation, considering long-term effects. RL does not require training samples; only the environment’s return value is required. Thus, building a WVR aerial TZSG manoeuvre decision-making model using RL is feasible for autonomous decision making, as it generates decision sequences through self-interactive training.

3.3. Introduction to Deep Reinforcement Learning

Deep learning enables feature extraction directly from raw data via deep neural networks, excelling in feature representation but lacking in decision making. Conversely, reinforcement learning excels in decision making but is limited in perception. Consequently, DRL, which merges these two approaches, has become a research hotspot. DRL leverages trial and error, using rewards and punishments to adjust neural networks for optimal policy models.

Deep learning [34] refines data through representation learning without manual feature selection, achieving distributed data representation by combining low-level features into abstract high-level ones. Reinforcement learning [35], rooted in optimal control theory, addresses time-series decision making to maximise cumulative expected returns through continuous environment interaction.

Researchers use function approximation to address the dimensionality issues in high-dimensional state-action spaces, replacing tabular methods with parametric functions represented by weight vectors. Adjusting these weights yields different functions.

DRL integrates deep learning’s feature representation capabilities with reinforcement learning’s decision-making abilities to solve high-dimensional tasks. It uses deep neural networks to fit reinforcement learning components, such as state-value functions, action-value functions, and policies, enabling end-to-end learning. DRL’s practicality in solving complex real-world problems was exemplified by the DQN algorithm [36], which outperformed human players in Atari video games by learning directly from image pixels, marking the rise of deep reinforcement learning.

3.4. Several Typical Deep Reinforcement Learning Algorithms Based on Discrete Action Spaces

3.4.1. DQN

The DQN algorithm [19] enhances Q-learning [37] by addressing the “dimensionality” catastrophe caused by using a Q-value table in complex environments. It combines deep learning with Q-learning, employing deep convolutional neural networks to approximate the value function, enabling the learning of control policies from high-dimensional inputs.

$$\left.\left.\left.\left.Q(S_t,a_t\right)=Q(S_t,a_t\right)+\alpha \left(r+\gamma Q(S_{t+1},a\right)-Q(S_t,a_t\right)\right)$$

(12)

Let $\left.Q(S_t,a_t\right)$ denote the reward for executing the action $a_t$ in state $S_t$, with $\alpha$ as the learning rate and ($\gamma$) as the discount factor. The $\epsilon -greedy$ algorithm to select action $a_t$ allows the value function to gradually converge to the optimal strategy.

$${\pi }^{*}\left(s\right)=\mathrm{a}\mathrm{r}\mathrm{g} \underset{a}{max}Q(s,a)$$

(13)

Given that the agent WVR aerial TZMG involves a high-dimensional state space, processing such data is challenging. Therefore, a neural network is employed to approximate the state-action value function. Specifically, a neural network with parameter θ is used to replace the state-action value function, as shown in Equation (14).

$$Q\left(s,a\right)\approx f\left(s,a,\theta \right)$$

(14)

where $f\left(s,a,\theta \right)$ can be any type of function that approximates the replacement of Q-value by a function.

In DQN, two networks—the estimation network and the target network—share the same structure but differ in parameters. The estimation network outputs $Q\left(s,a,\theta \right)$ to estimate the current state-action value, while the target network outputs $Q\left(s,a,{\theta }^{-}\right)$. The estimation network’s parameters are updated in real-time, and in every C step, these parameters are copied to the target network. Parameter updates in the DQN network are performed using TD-error.

$$L_t\left({\theta }_t\right)=E{\left(y_t-Q\left(s,a,{\theta }_t\right)\right)}^2$$

(15)

It is common to use $y_t$ to approximate the optimisation objective of the value function, which $y_t$ defined as follows:

$$y_t=r_{t+1}+\gamma \underset{a}{max}Q\left(s_{t+1},a_{t+1}\mid {\theta }_t^{-}\right)$$

(16)

Let ${\theta }_t^{-}$ and ${\theta }_t$ denote the parameters of the target network and the estimated network at time t, respectively. $s_{t+1}$ is the state at the next moment, $a_{t+1}$ is the action yielding the maximum Q-value for the agent at state $s_{t+1}$, and $r_{t+1}$ is the reward obtained after pacting $\gamma$ is the discount factor.

The DQN algorithm updates the network by stochastic gradient descent (SGD) and backpropagation algorithm with the update formula:

$${\nabla }_{\theta }L\left(\theta \right)=E\left[\left(Y_t-Q\left(s_t,a_t;\theta \right)\right){\nabla }_{\theta }Q\left(s_t,a_t;\theta \right)\right]$$

(17)

In DQN, the samples for training are drawn from an experience pool $D=\left\{e_1,e_2,\cdots ,e_n\right\}$, which stores interactions $\left(s_t,a_t,r_t\right.\left.,s_{t+1}\right)$ between the mobile robot and the environment. A subset of these samples is used for each training session to update the network parameters via stochastic gradient descent.

3.4.2. DDQN

The DeepMind team proposed Double DQN [38] based on DQN, which provides better performance and faster learning with a simple improvement of the algorithms for calculating the target action value and the dual network structure in DQN. DDQN optimises the problem of over-estimation in traditional DQN algorithms by decoupling the selection of the target Q-value action strategy from the calculation of the target Q-value. Specifically, DDQN passes each action and state in the action space to the online network θ to get the Q-value outputs corresponding to all the actions in the online network and then selects the corresponding action among the current Q-value outputs using the greedy method:

$$a_t=\left\{\begin{array}{cc}\underset{a}{\arg max}Q\left(s_{t+1},a_t\mid \theta \right)& e⩽{\epsilon }_{\mathrm{greedy} }\\ random\left(a_t\right)& e>{\epsilon }_{\mathrm{greedy}}\end{array}\right.$$

(18)

where ${\epsilon }_{\mathrm{greedy} }$ is the greedy coefficient of ($e$) obeys the interval $\left(0, 1\right)$ between which is uniformly distributed.

The neural network for deep Q learning can be viewed as a new neural network plus an old neural network; they have the same structure, but there is a time difference in updating the parameters within them. Its updated formula is as follows:

$$Y_{\mathrm{t}}^{\mathrm{DQN}}=R_{t+1}+\gamma maxQ\left(S_{t+1},\alpha ;{\theta }_t\right)$$

(19)

Double DQN [39,40] addresses the overestimation bias in traditional deep Q-learning by introducing a second neural network. In standard deep Q-learning, the Q-estimation network predicts the maximum action value (Q-max) in the Q-reality network, which can lead to overestimation errors. Double DQN mitigates this by using the Q-estimation network to select the action and the Q-reality network to evaluate it. The improved update formula is as follows [41,42]:

$$Y_t^{\mathrm{DoubleDQN} }=R_{t+1}+\gamma Q\left(S_{t+1},\mathrm{argmax}Q\left(S_{t+1},\alpha ;{\theta }_t\right),{\theta }_t\right)$$

(20)

3.4.3. Dueling DQN

Dueling DQN enhances DQN by decomposing the Q-value into Value and Advantage components. The Value represents the significance of the current state, while the Advantage indicates the relative value of each action. The final Q-value is computed as follows:

$$Q(s,a;\theta ,\alpha ,\beta )=V(s;\theta ,\beta )+(A(s,a;\theta ,\alpha )-\underset{a^{\prime }\in \left|A\right|}{max}A\left(s,a^{\prime };\theta ,\beta \right)$$

(21)

Equation (21): $\theta$ denotes the parameters of the convolutional layer of the network; $\alpha$ and $\beta$ denote the parameters of the Advantage and Value function fully connected layers, respectively; the state value function $V\left(s;\theta ,\beta \right)$ is related to the state only and does not consider the effect of the action on the state; the action value function $A\left(s,a;\theta ,\alpha \right)$ is related to both action and state.

3.4.4. D3QN

D3QN [43], an algorithm building upon DQN and DDQN, incorporates a duelling network structure. This structure modifies the neural network to output the state value function and the advantage function instead of the Q-value directly.

$$Q\left(s,a;\theta \right)=V\left(s;{\theta }^V\right)+\overline{A}\left(s,a;{\theta }^{\overline{A}}\right)$$

(22)

Equation (22) indicates that after adding the competitive network, the Q-value is obtained from the value function of the state and the dominance function of each action $\overline{A}\left(s,a;{\theta }^{\overline{A}}\right)$ obtained by adding them. $\theta$ For the shared network parameters, the ${\theta }^V$ with ${\theta }^{\overline{A}}$ denotes separate network parameters, where the state value function indicates how good or bad the state is, and the dominance function indicates how good or bad an action in that state is relative to the other actions, and the association yields the value of an action in that state, Q. The introduction of the dominance function serves the purpose that, unlike DQNs that learn all Q values directly, D3QNs are able to differentiate between whether the current reward is caused by the state itself or by the chosen action.

4. Simulation Experiment

4.1. Experimental Framework

This study investigates a two-player zero-sum game scenario involving agents operating in three-dimensional space. Under given environmental conditions and initial parameters, the controlled agents autonomously formulate strategies based on the situational awareness of both players. The goal is to gain an advantage over the opposing agent and effectively defeat the opponent to achieve victory. Manoeuvre decision making is assisted by DRL algorithms specifically tailored for aerial combat. Experiments compare different algorithms, including DQN, DDQN, Dueling DQN, and D3QN, all using fully connected (FC) neural networks to ensure consistent performance.

As shown in Figure 6, the game process consists of three main modules: environmental interaction, manoeuvre decision, and motion. The environment interaction module provides the gaming environment and observation data. The manoeuvre decision module uses DRL to generate manoeuvre control variables, which the motion module executes. The motion module updates position and attitude through agent motion equations, interacts with the opposing agent, and feeds updated situational information back to the manoeuvre decision module. This iterative process optimises the agent’s manoeuvring strategy, achieving effective and rapid victory in the game.

4.2. Experimental Preparation

4.2.1. Experimental Hardware Preparation

The experimental hardware is Microsoft Windows 11 Flagship 64-bit operating system CPU Inter i9-13900K@6.00 GHz, 32 GB of RAM, GPU RTX4080.

4.2.2. Experimental Software Preparation

The experimental software is the simulation environment is all based on Python3.9.16 language writing, using Pycharm Professional 2023.1.2 and the Anaconda3 platform; the deep reinforcement learning environment uses Pytorch2.1.0 and cuda12.1.

4.2.3. Simulation Experiment Scene Setting

As shown in Figure 6, agent 1 selects actions to execute using DRL algorithms, while agent 2, acting as the opponent, determines its actions based on the master state machine. In the 1v1 autonomous WVR air TZSG scenario, our side starts at a disadvantage, positioned in the enemy aircraft’s advantageous area, as illustrated in Figure 7. Throughout the training, both sides’ positions are fixed and initialised in 3D space, consistently maintaining the depicted posture. Our manoeuvre selection employs a deep reinforcement learning algorithm with a discrete action space (refer to Section 4.3), while the enemy uses an expert system optimised for air combat superiority.

To update the WVR air combat simulation parameters, set the WVR action update cycle to 100 Hz and the air combat decision cycle to 1 Hz. The initial conditions for the Red and Blue agents are as follows in

Table 1. Agent initial setting.

Parameter	Agent (r)	Agent (b)
Spatial coordinates	$c_r=\left(-1997.48, 0.04,-6000.19\right)$	$c_b=\left(-0.04, 2.50,-7000.19\right)$
Trajectory inclination	${\gamma }_r=0^{\circ }$	${\gamma }_b=0^{\circ }$
Trajectory declination	${\psi }_r=0^{\circ }$	${\psi }_{\mathrm{b}}={270}^{\circ }$
Velocity magnitude	$v_r=180 \mathrm{m}/\mathrm{s}$	$v_b=200 \mathrm{m}/\mathrm{s}$

.

In a zero-sum aerial combat scenario, both the red and blue factions engage in a single episode, with 1000 iterations of combat ensuing. Following this, neural network learning ceases, and both sides utilise network-derived values for decision making in manoeuvring. The blue faction employs state machine-based manoeuvre execution, while the red faction utilises a deep neural network for action selection. The initial aerial combat configuration is depicted in Figure 8. As a control experiment, the strategy of agent (b) remains constant, while agent (r) employs deep reinforcement learning for action selection.

4.2.4. Hyperparameter Settings for Each Algorithm

The hyperparameters used by each of our agents when performing algorithm training are shown below in

Table 2. List of hyperparameters of agent algorithm.

Hyper Parameterisation	(Be) Worth
random seed	1
Maximum number of steps in a round	50,000
training wheels	1000
Number of hidden layer nodes	256
Update Batch	512
optimiser	Adam
Experience pool capacity	1,000,000
Number of hidden layer nodes	128
Frequency of target network updates	1000 steps.
learning rate	0.001
probability of exploration	0.9
Incentive Discount Factor	0.95

.

4.2.5. Algorithm Convergence Criterion

The convergence criteria for the algorithm in this experiment adhere to two widely recognised methods in the academic field of deep reinforcement learning (DRL), as detailed below:

• Stability of Evaluation Metrics: During training, evaluation metrics gradually stabilise and reach a certain level, indicating that the model’s performance tends to be stable. In this experiment, the primary evaluation metric of interest is the cumulative reward. By plotting the change curve of the cumulative reward over time during training, we can observe the stabilisation of the evaluation metric.
• No Significant Performance Improvement: Within a certain period, if the model’s performance no longer shows significant improvement, it suggests that further training does not yield noticeable changes in performance metrics. In this experiment, we use the adversarial win rate of the trained model as the performance metric. By setting several predefined training intervals (iterations) and observing the changes in performance metrics during these intervals, we can conclude that the model has converged if there is no significant improvement in performance.

4.3. Experimental Results and Analyses

Four types of discrete action-based deep reinforcement learning (DRL) algorithms, namely DQN, DDQN, Dueling DQN, and D3QN, were trained based on the same environment, identical initial conditions, and partially shared network hyperparameters. The reward curves of the agents during the training process are depicted in Figure 8a–e.

Figure 8. The training results of the deep reinforcement learning (DRL) model.

Figure 8. The training results of the deep reinforcement learning (DRL) model.

From the above figure, it can be seen that all algorithms can have more frequent final round rewards of 200+ or more in large rounds of training, which means that the blue side controlled by the DRL algorithm can beat the red side of the expert system based on the dominance function more consistently in training. Compared to the DQN, shown in Figure 8a, algorithm, DDQN, Dueling DQN, and D3QN, shown in Figure 8b–d, all show superiority in convergence, where D3QN, shown in Figure 8d, has stronger convergence rates, with +100 final step rewards occurring much more frequently than −100 rewards in large rounds of training. The agent is trained due to exploration, so winning or losing a game during training does not indicate the algorithm’s performance. Therefore, it is necessary to conduct multiple rounds of random confrontation between the trained deep network model and the red expert system alone and evaluate its performance. In this experiment, each agent interacts with the environment for 1000 episodes under the control of the same DRL algorithm, with an average of 23,000 steps of agent decision making per episode and 23,000,000 interactions between the agent and the environment. In this paper, the agent will save the current neural network every 20 episodes, in the performance test, each DRL algorithm saved the neural network model by the time sequence was tested with the red side of the random initialisation of 100 battles, and the experimental results obtained are shown in Figure 9a–d

In Figure 9a, the average win rate of the DQN algorithm is about 40% and the wins do not increase significantly at about the 400th episode, which can be regarded as the convergence of the algorithm. In Figure 9b, the average win rate of the DDQN algorithm is about 43% and the algorithm converges at about the 300th episode. In Figure 9c, the Dueling DQN algorithm has an average win rate of about 45% and converges at about the 300th episode. In Figure 9d, the D3QN algorithm has an average win rate of about 52%, and the algorithm converges at about the 180th episode. From the above, the D3QN algorithm shows strong performance in the WVR air two zero-sum game scenario.

Under the same hardware conditions, training environment, and initial state, this experiment statistically analyses the training time of intelligent control algorithms for air combat manoeuvring decisions within a visual range, considering differences in algorithm scale and complexity. The time taken to reach the 1000th training iteration was recorded for each algorithm, as shown in Figure 10.

In the comprehensive above experiments, the DQN algorithm can achieve effective decision making of air manoeuvre strategy and higher efficiency, but its performance is more limited. The DDQN algorithm’s performance has improved, but efficiency has been reduced. Dueling the DQN algorithm and DDQN algorithm performance is similar to that of the DDQN algorithm; efficiency has been reduced slightly. The D3QN algorithm, combining the DDQN and Dueling DQN, has greatly improved performance and convergence, but training efficiency has been reduced.

In addition, all the above four algorithms learn to combine basic technical movements to form tactical manoeuvres in the training process to obtain tactical advantages. Figure 11 shows some screenshots of the simulation environment. The explanation of the manoeuvre can be found in the Glossary.

5. Conclusions

This paper begins by examining the characteristics of within-visual range (WVR) aerial two-player zero-sum games (TZSGs) and establishes a training mechanism for AI algorithms in this context. A reward-shaping method based on the advantage function is designed to address the issue of sparse rewards in WVR aerial TZSG. A benchmark deep reinforcement learning (DRL) algorithm that effectively handles discrete action spaces is selected. The hardware and hyperparameters are standardised, and algorithm training is conducted in an identical environment. Performance validation experiments, decoupled from the algorithm training, are designed to test and analyse the performance of the selected benchmark DRL algorithm. The results indicate that DRL can effectively handle manoeuvring decision problems in WVR aerial TZSG. However, due to the limitations of the benchmark DRL algorithm, there is room for improvement in decision-making performance and accuracy through modifications and adaptations of the DRL algorithm itself. Discussing the limitations of the current approach and potential future work would strengthen the paper’s conclusion.