Maneuver Decision-Making for Autonomous Air Combat Based on FRE-PPO

Abstract

Maneuver decision-making is the core of autonomous air combat, and reinforcement learning is a potential and ideal approach for addressing decision-making problems. However, when reinforcement learning is used for maneuver decision-making for autonomous air combat, it often suffers from awful training efficiency and poor performance of maneuver decision-making. In this paper, an air combat maneuver decision-making method based on final reward estimation and proximal policy optimization is proposed to solve the above problems. First, an air combat environment based on aircraft and missile models is constructed, and an intermediate reward and final reward are designed. Second, the final reward estimation is proposed to replace the original advantage estimation function of the surrogate objective of proximal policy optimization to improve the training performance of reinforcement learning. Third, sampling according to the final reward estimation is proposed to improve the training efficiency. Finally, the proposed method is used in a self-play framework to train agents for maneuver decision-making. Simulations show that final reward estimation and sampling according to final reward estimation are effective and efficient.

1. Introduction

With the development of artificial intelligence, autonomous air combat has become more and more attractive. In August 2020, the US Defense Advanced Research Projects Agency held the Alpha DogFight competition. The air combat agent developed by Heron systems defeated the human pilot by 5:0 in close air combat [1]. This competition totally reflects the great advantages and potential of artificial intelligence in autonomous air combat, and the key to the victory of air combat is the maneuver decision-making method. Therefore, it is very important to study maneuver decision-making methods based on artificial intelligence. However, artificial intelligence-based methods require numerous samples for training agents, and sometimes, even much more samples cannot guarantee performance or efficiency, which is an urgent problem needed to be solved. Therefore, we mainly focus on improving the performance and efficiency of the maneuver decision-making method based on reinforcement learning (RL).

In recent studies, most of the research on maneuver decision-making for air combat focused on within-visual-ranges air combat [2,3,4,5,6], which excludes the kinematic model and the dynamic model of missiles and thus is insufficient and defective for addressing the problems of maneuver decision-making. Wei et al. [7] proposed a cognitive control model with a three-layered structure for multi-UAV cooperative searching according to the cognitive decision-making mode of humans performing searching behavior. The mission attack area is based on cognitive matching, reduction, and division based on this model and the fuzzy cluster. The simulation experiments indicate the great performance of the fuzzy cognitive decision-making method for cooperative search. On the other hand, this method mainly focuses on cooperative search, which may not be effective for maneuver decision-making because these problems are different from each other. Zhang et al. [8] proposed a maneuver decision-making method based on the Q network and Nash equilibrium strategy, combining the missile attack area in the reward function to improve the efficiency of RL. However, the maneuver library of this method contains only five maneuvers, which cannot meet the requirements of air combat. Hu et al. [9] proposed to use the improved deep Q network [10] for maneuver decisions in unmanned air combat, constructed the relative motion model, missile attack model and maneuver decision-making framework, designed the reward function for training agents, and replaced the strategy network in deep Q network with the perception situation layer and value fitting layer. This method improves the winning rate of air combat, but the action space of the proposed method is discrete. However, the action space should be continuous to make the proposed method more reliable.

It is worth noting that deep RL has achieved professional performance in video games [11,12,13], board games such as GO [14,15,16], real-time strategy games such as StarCraft [17] and the magnetic control of Tokamak plasmas [18], which indicates the generality of deep RL for solving decision-making problems. Therefore, using deep RL to improve the level of air combat maneuver decision-making is a feasible potential direction. Ma et al. [19] described the cooperative occupation decision-making problem of multiple UAVs [20] as a zero-sum matrix game problem and proposed a solution of a double oracle algorithm combined with neighborhood search. However, the method did not incorporate the missile into cooperative occupation decision-making problems. Ma et al. [21] built an air combat game environment and trained the agent with deep Q-learning with a discrete action space. Eloy et al. [22] studied attacks against static high-value targets in air combat. They analyzed the confrontation process with game theory and put forward a differential game method of air combat combined with the missile attack area [23,24,25]. However, their flight model was two-dimensional rather than three-dimensional. He et al. [26] proposed a maneuver decision-making method based on Monte Carlo tree search (MCTS), which uses MCTS to find the action with the maximum air combat advantage function value among seven basic maneuvers. This method verifies the feasibility of MCTS for maneuver decision-making, however, its action space is discrete as well.

Although the above studies have made some progress, there are still some problems in current air combat maneuver decision-making. First, the action space of air combat maneuver decision-making is continuous, but existing methods usually adopt the method of continuous space discretization, which is a common drawback of existing methods. Second, existing decision-making methods determine whether it wins or not according to whether the target enters the missile attack area, namely, excluding the kinematic model and the dynamic model of the missile, which is another drawback. However, the target may not be shot down by the missile even if it enters the missile attack area. Therefore, it is necessary to determine whether it wins or not according to whether the missile hits the target. Third, RL algorithms often perform poorly in maneuver decision-making, so it is urgent to propose more effective and efficient RL algorithms address maneuver decision-making. These problems still require to be solved, and the goal of this article is to solve these problems.

To solve the above problems of air combat, we propose a method based on the final reward estimation and proximal policy optimization (FRE-PPO). Our main contributions are as follows: (1) we used a deep RL method, namely proximal policy optimization, to handle maneuver decision-making in continuous action space. (2) We propose to use final reward estimation to replace the original advantage estimation function of the surrogate objective of proximal policy optimization to improve training performance and efficiency. (3) The proposed method is conducted in a self-play framework to generate air combat agents. (4) Ablation studies are provided to support our claims. Section 2 introduces the basic knowledge of RL. In Section 3, an air combat simulation environment based on an aircraft model and missile model is constructed, and an intermediate reward and final reward are designed. And the air combat agent training framework based on FRE-PPO is proposed, which can improve the maneuver decision-making ability of air combat agents by means of self-play and FRE-PPO. In Section 4, ablation studies and simulation experiments are performed, and FRE-PPO is compared with the other four methods to verify its performance. Section 5 summarizes the experimental results and draws conclusions.

2. Background

The advantage of the maneuver decision-making method based on RL (e.g., PPO) is that it can directly use the continuous action space without discretization of the action space. At the same time, it does not rely on expert experience and can adapt to changes in the air combat environment. Meanwhile, RL can create agents outperforming human beings [14,15,16]. Therefore, RL has some advantages and potential in air combat maneuver decision-making. However, there are still some problems when using RL in air combat maneuver decision-making: it is difficult to train effective agents using RL; RL method usually performs well in Atari [27] and robot control tasks [28], but air combat maneuver decision-making is different from these tasks. Therefore, how to guarantee better and more stable performance in air combat maneuver decision-making using RL needs to be addressed.

The purpose of RL is to learn policies to maximize rewards in sequential decision-making [29]. Sequential decision-making is that, at each discrete time step t, an agent starts from a certain state, s_t, chooses an action, a_t, according to its policy and obtains a reward, r_t. Then, it takes the discounted sum of rewards (DSR) as its return with a discount factor. The aim of the agent acting in a stochastic environment by sequentially choosing actions over a sequence of time steps is to obtain more rewards or win the game. Air combat maneuver decision-making can be regarded as a sequential decision-making problem in continuous action space. Therefore, the maneuver decision-making problem is modeled as the Markov decision process (MDP), a sequential decision-making process, and the agent interacts with the air combat environment by its policies and then obtains a tuple (s_t, a_t, r_t).

RL can be divided into value-function-based RL (VBRL) and policy-based RL (PBRL). VBRL uses the temporal difference (TD) error to optimize the value function of the action and selects the action corresponding to the value function. Its disadvantage is that if the action space is continuous, the action space must be discretized at first, which results in the exponential growth of actions. Value-based algorithms include Q-learning [11], deep Q-network (DQN) [10] and its variants: (1) prioritized experience replay [30] weights sampling of data based on TD error to improve learning efficiency; (2) dueling DQN [31] improves the network structure by decomposing the action value function Q into the state value function and the advantage function to improve the function approximation ability; (3) double DQN [32] uses two independent action value functions to solve the problem of over estimation; (4) noisy DQN [33] adds noise to network parameters to increase the exploration ability of agents; (5) distributed DQN [34] refines the state action value estimation into the estimation of the state action value distribution; (6) rainbow [13] combines the characteristics of the above DQN based methods; (7) R2D2 [35] studies the effect of adding recurrent neural networks to RL; (8) NGU [36] uses short-term novel incentives in an episode and long-term novel incentives across multiple episodes to increase the exploration ability of agents; (9) agent57 [37], based on NGU, solves the problem of long-term credit allocation by improving the training stability and dynamically adjusting discount factors.

PBRL methods directly parameterize policies to maximize DSR by updating the policy using a policy gradient. Its advantage is its fast convergence speed and can be directly used in the continuous and high-dimensional action space without discretization. Policy-based algorithms include policy gradient algorithm [38], natural policy gradient algorithm [39], deterministic policy gradient algorithm [40], deep deterministic policy gradient algorithm [41], dual delay deep deterministic policy gradient algorithm [42], trust region policy optimization (TRPO) [43] and proximal policy optimization (PPO) [44]. TRPO and PPO limit the update of the policy. TRPO can guarantee its cost after policy update is non-increasing. PPO clips the probability ratio of the policy, making the algorithm more stable. Because VBRL cannot be directly used in continuous action space, this paper uses PBRL to solve the air combat maneuver decision-making problem.

3. Method

3.1. Aircraft Model and Missile Model

In maneuver decision-making, the aircraft model adopts normal overload, tangential overload and roll angle as control parameters. To simplify the complexity of the problem, the angles of attack and angle of side slip are regarded as 0, and the ground coordinate system is treated as the inertial system. Meanwhile, the effects of the rotation of the earth are overlooked. The kinematic and dynamic model is shown as follows [45]:

$$\begin{array}{l}\left\{\begin{array}{l}\dot{x}=v\cos \gamma \cos \psi \\ \dot{y}=v\cos \gamma \sin \psi \\ \dot{z}=v\sin \gamma \\ \dot{v}=g(n_x-\sin \gamma )\\ \dot{\gamma }=\frac{g}{v}(n_z\cos \mu -\cos \gamma )\\ \dot{\psi }=\frac{g}{v\cos \gamma }{n}_z\sin \mu \end{array}\right.\\ n_x\in [-0.5,1.5]\\ n_z\in [-3,9]\\ \mu \in [-\pi ,\pi ]\end{array}$$

(1)

where x, y, and z indicate the positions of the aircraft in the inertial coordinate system; $\gamma$ is the pitch angle, $\psi$ is the yaw angle, v is the velocity, and g is the acceleration of gravity. Roll angle $\mu$, tangential overload $n_x$, and normal overload $n_z$ are control parameters. The action space consists of $\mu$, $n_x$, and $n_z$, which is continuous, as shown in (1). This means that the control parameters are selected from the three intervals instead of using space discretization as in previous methods. The kinematic model of the missile is [46]:

$$\left\{\begin{array}{l}{\dot{x}}_m=v_m\cos {\gamma }_m\cos {\psi }_m\\ {\dot{y}}_m=v_m\cos {\gamma }_m\sin {\psi }_m\\ {\dot{z}}_m=v_m\sin {\gamma }_m\end{array}\right.$$

(2)

where x_m, y_m, and z_m indicate the positions of the missile in the inertial coordinate system; v_m is the velocity, ${\gamma }_m$ is the pitch angle, and ${\psi }_m$ is the yaw angle. The dynamic model of the missile is:

$$\left\{\begin{array}{l}{\dot{v}}_m=\frac{(P_m-Q_m)g}{G_m}-g\sin {\gamma }_m\\ {\dot{\psi }}_m=\frac{n_{mc}g}{v_m\cos {\gamma }_m}\\ {\dot{\gamma }}_m=\frac{n_{mh}g}{v_m}-\frac{g\cos {\gamma }_m}{v_m}\hspace{0.17em}\hspace{0.33em}\hspace{0.33em}\end{array}\right.$$

(3)

where P_m and Q_m are thrust and air resistance, respectively, G_m is the mass of the missile, and n_mc and n_mh are control overload in the yaw direction and pitch direction, respectively. P_m, Q_m and G_m can be calculated by the following formula:

$$P_m=\left\{\begin{array}{c}\begin{array}{cc}12000& t\le t_w\end{array}\\ \begin{array}{cc}0& t\end{array}>t_w\end{array}\right.$$

(4)

$$Q_m=\frac{1}{2}\rho v_m^2{S}_m{C}_{Dm}$$

(5)

$$G_m=\left\{\begin{array}{c}\begin{array}{cc}173.6-8.2t& t\le t_w\end{array}\\ \begin{array}{cc}108& t>t_w\end{array}\end{array}\right.$$

(6)

where t_w = 8.0 s, $\rho$ = 0.607, S_m = 0.0324, C_Dm = 0.9. It is assumed that the guidance coefficient of proportional guidance law is K in the control planes. The two overloads in the yaw and pitch directions are defined as:

$$\left\{\begin{array}{l}{n}_{mc}=K\cdot \frac{v_m\cos {\gamma }_t}{g}[\dot{\beta }+\tan \epsilon \tan (\epsilon +\beta )\dot{\epsilon }]\\ n_{mh}=\frac{v_m}{g}\frac{K}{\cos (\epsilon +\beta )}\dot{\epsilon }\end{array}\right.$$

(7)

$$\left\{\begin{array}{l}\beta =\arctan (r_y/r_x)\\ \epsilon =\arctan (r_z/\sqrt{r_x^2+r_y^2})\end{array}\right.$$

(8)

$$\left\{\begin{array}{l}\dot{\beta }=({\dot{r}}_y{r}_x-r_y{\dot{r}}_x)/(r_x^2+r_y^2)\hspace{1em}\hspace{1em}\\ \dot{\epsilon }=\frac{(r_x^2+r_y^2){\dot{r}}_z-r_z({\dot{r}}_x{r}_x+{\dot{r}}_y{r}_y)}{R^2\sqrt{r_x^2+r_y^2}}\end{array}\right.$$

(9)

where $\beta$ and $\epsilon$ are the yaw angle and pitch angle of line-of-sight, respectively. The line-of-sight vector is the distance vector r, where $r_x=x_t-x_m,r_y=y_t-y_m,r_z=z_t-z_m$ and $R=‖r‖=\sqrt{r_x^2+r_y^2+r_z^2}$.

The maximum overload of the missile is 40. When the minimum distance between the missile and the target is less than 12 m, the target is regarded as being hit; when the missile flight time exceeds 120 s, and it still fails to hit the target, the target is regarded as missed; during the midcourse guidance stage, the target is regarded as missed when its azimuth relative to the aircraft exceeds 85°; during the final guidance stage, the target is regarded as missed when its azimuth relative to missile axis exceeds 70°.

3.2. Reward Design

In RL, the agent maximizes its return through trial and error experience [47]. Due to sparse rewards, it is difficult for agents to obtain rewards in an air combat environment and complete the given task. Therefore, in order to avoid invalid training caused by sparse rewards, this paper proposes intermediate rewards and final rewards. The intermediate reward is determined by the situation information of air combat, and the agent can get an intermediate reward at each time step, so the reward is not sparse, and the agent may get the final reward according to the assistance of the intermediate reward. The final reward is determined by the result of air combat, namely win, lose or draw. As the agent can only get the final reward at the end of air combat, the final reward is sparse. The combination of intermediate reward and final reward can effectively avoid bad training performance when only using the final reward. The formula of intermediate reward r_M and the explanation of the meaning behind the formula is shown as follows:

$$\begin{array}{l}{r}_a={\omega }_{a1}({\psi }_{\max 1}-{\psi }_{r1})+{\omega }_{a2}({\gamma }_{\max 1}-{\gamma }_{r1})+{\omega }_{a3}({\psi }_{r2}-{\psi }_{r1})+{\omega }_{a4}({\gamma }_{r2}-{\gamma }_{r1})+\min (\frac{\mathrm{D}}{d},1)\\ r_b={\omega }_b\frac{\left|z_1-\mathrm{H}\right|}{\mathrm{H}}\\ r_c={\omega }_c{t}_{missile}\\ r_d=\left\{\begin{array}{l} 1, \mathrm{launched}\\ 0, \mathrm{not} \mathrm{launched}\\ -1, \mathrm{miss} \mathrm{the} \mathrm{target}\end{array}\right.\\ r_M=r_a+r_b+r_c+r_d\end{array}$$

(10)

where, ${\gamma }_{\max 1}$ and ${\psi }_{\max 1}$ are the maximum pitch and yaw angle of the missile, respectively, ${\gamma }_{r1}$ and ${\psi }_{r1}$ are, respectively, the pitch and yaw angles of the speed vector of one aircraft relative to the line of sight, ${\gamma }_{r2}$ and ${\psi }_{r2}$ are, respectively, the pitch and yaw angles of the speed vector of another aircraft relative to the line of sight. ${\omega }_{a1}$, ${\omega }_{a1}$, ${\omega }_{a3}$, and ${\omega }_{a4}$ are corresponding weights. As can be seen in (10), smaller ${\gamma }_{r1}$ and ${\psi }_{r1}$ mean larger r_a, which motivates the aircraft to fly to the target (it can be regarded as an attack); larger ${\gamma }_{r1}$ and ${\psi }_{r1}$ mean smaller r_a, which indicates that the target is approaching the aircraft (it can be regarded as being attacked). D is the typical attack distance of the missile, and d is the distance between both sides, so the item min(D/d, 1) motivates the aircraft to approach the target. Therefore, r_a is set as its opposite during the final guidance stage to make the aircraft fly away from the target since, during the final guidance stage, the missile no longer needs the information transited by the aircraft. z₁ is the altitude of the aircraft, H is the typical flight altitude, and ${\omega }_b$ is the weight. Therefore, r_b encourages the aircraft to stay at its typical flight altitude. t_missile is the flight time after the missile is launched and ${\omega }_c$ is the weight. This means that earlier launching corresponds to a larger reward. When the missile is launched, the agent gets a reward of 1. When the missile is not launched, the agent gets a reward of 0. When the missile loses its target, the agent gets a reward of −1. This item means launching the missile is encouraged, and missing the target is punished. The sum of the above four rewards is the intermediate reward r_M. The formula of the final reward r_F is:

$$r_F=\left\{\begin{array}{c}\begin{array}{l}30, \mathrm{win}\\ -30, \mathrm{lose}\end{array}\\ 0, \mathrm{draw} \end{array}\right.$$

(11)

The result of the air combat simulation is defined as follows: if one side’s missile hits its target, the side wins air combat, and the other side hit by the missile loses air combat; either the aircraft or the missile misses the target, it is regarded as missing the target, if both sides lose targets or the simulation reaches the time limit, this air combat is regarded as a draw. At the end of air combat, the winner gets a final reward of 30, and the loser gets a final reward of −30, and in other cases, the final reward is 0. Therefore, the reward obtained by the agent at each time step is r_t = r_M + r_F. Finally, we set ${\omega }_{a1}$, ${\omega }_{a1}$, ${\omega }_{a3}$ and ${\omega }_{a4}$ to 1 because the radar may lose its target if the azimuth angle of the target is bigger than the maximum pitch angle or maximum yaw angle. We set ${\gamma }_{\max 1}$ and ${\psi }_{\max 1}$ to π/3 because radar cannot detect the target from all directions simultaneously, and these parameters can be changed according to different radars. We set ${\omega }_b$ to −4 in order to reduce the altitude difference between the aircraft and its target. We set ${\omega }_c$ to 1/60 to encourage the maneuvers which make the target within radar detection range.

3.3. FRE-PPO Algorithm

TRPO repeatedly performs the following steps:

• Use the single path or vine procedures to collect a set of state-action pairs along with Monte Carlo estimates of their Q-values;
• By averaging over samples, construct the estimated objective and constraint in

  $$\begin{array}{l}\underset{\theta }{\mathrm{maximize}} {{\rm E}}_{s~{\rho }_{{\theta }_{old}},a~q}\left[\frac{{\pi }_{\theta }(a|s)}{q(a|s)}{Q}_{{\theta }_{old}}(s,a)\right]\\ \mathrm{subject} \mathrm{to} {{\rm E}}_{s~{\rho }_{{\theta }_{old}}}\left[D_{KL}({\pi }_{{\theta }_{old}}(\cdot |s)\left|\right|{\pi }_{\theta }(\cdot |s))\right]\le \delta .\end{array}$$

  (12)
• Approximately solve this constrained optimization problem to update the policy’s parameter vector θ by means of the conjugate gradient algorithm followed by a line search, which is altogether only slightly more expensive than computing the gradient itself. Where, ${\theta }_{old}$ is the parameter of the current strategy, ρ is the probability distribution of state transition, q is the sampling distribution, and D_KL is the KL divergence.

By establishing the Fisher information matrix, TRPO avoids storing a dense Hessian matrix or policy gradients from a batch of trajectories and improves computational efficiency. Theoretically, the penalty coefficient of larger KL divergence can lead to a small step size, so the penalty coefficient should be decreased. However, it is difficult to select a penalty coefficient that can ensure the stability of the effect, so TRPO uses a constraint term instead of a penalty term. At the same time, since the maximum KL divergence is not easy to be optimized and estimated, TRPO uses the average KL divergence.

PPO is an improved version of TRPO. TRPO may cause excessive policy updates when maximizing the objective function, so PPO modifies the objective function to punish the policy changes that make the probability ratio far from 1. The objective function of PPO is:

$$\begin{array}{l}{L}^{CLIP}(\theta )={\rm E}\left[\min (r_t(\theta )A_t,\mathrm{clip}(r_t(\theta ),1-\epsilon ,1+\epsilon )A_t)\right]\\ P_r(\theta )=\frac{{\pi }_{\theta }(a_t|s_t)}{{\pi }_{{\theta }_{old}}(a_t|s_t)}.\end{array}$$

(13)

by clipping the probability ratio P_r and removing the strategy change that makes the probability ratio exceed the interval [1 − ε, 1 + ε]. The formula of the DSR advantage function used in PPO is: $A_t{=\mathrm{r}}_t+\gamma {\mathrm{r}}_{t+1}+\cdots {\gamma }^{T-t+1}{\mathrm{r}}_{T-1}+{\gamma }^{T-t}V(s_T)-V(s_t)$. The formula of generalized advantage estimation (GAE) [48] used in PPO is: $A_t={\delta }_t+(\gamma \lambda ){\delta }_{t+1}+\cdots +(\gamma \lambda {)}^{T-t+1}{\delta }_{T-1}, \mathrm{where} {\delta }_t{=\mathrm{r}}_t+\gamma V(s_{t+1})-V(s_t)$. Since GAE and DSR are not effective for solving the air combat maneuver decision-making problem, this paper proposes a new function, namely, final reward estimation (FRE). The formula of FRE is defined as follows:

$$A_t=\left\{\begin{array}{l}{\mathrm{r}}_t+{\gamma }^{T-t}{\mathrm{r}}_T,t<T\\ {\mathrm{r}}_T,t=T\end{array}\right.$$

(14)

where T is the last time step in air combat. This method uses self-play samples obtained from the air combat environment to train the maneuver policy, which is represented by neural networks. The input of the neural networks is air combat situation information, and the output is the three control quantities of aircraft and whether to launch a missile. At the same time, in order to increase the exploration of the policy, Gaussian noise is added to each action. After obtaining training samples, a certain number of samples need to be selected from these samples to train the neural network. A common sampling method is randomly uniform sampling. In order to improve the efficiency and effect of training, the method of sampling according to FRE is proposed in order to collect training samples, that is, the higher FRE of the sample, the greater the probability of being collected. The probability of each sample is calculated by the following formula:

$$P(i)=\frac{A_i^{\alpha }}{{\displaystyle {\sum }_k{A}_k^{\alpha }}}$$

(15)

where A_i is calculated by Formula (14), and the index α represents the role of FRE in probability. When α = 0, it is identical to uniform choice; when α = 1, samples are completely sampled according to FRE. Figure 1 shows the FRE-PPO self-play architecture. First, run the policy in an air combat environment and acquire transitions (s_t, a_t, and r_t). Second, store these transitions in the experience pool and compute probabilities of transitions according to FRE. Then, sample transitions according to probabilities and use these transitions to train neural networks.

3.4. Air Combat Agent Training Framework Based on FRE-PPO

Combined with the aircraft model, missile model, reward design, and FRE-PPO algorithm, an air combat agent training framework based on FRE-PPO is proposed. The agent training framework consists of six parts: actor, experience pool, learner, player pool, evaluator and recorder. The framework is shown in Figure 2.

In each iteration, the actor first generates self-play data in the air combat environment and stores these data in the experience pool. After that, the experience pool calculates the FRE of each sample and the average returns of all data and stores the average returns in the recorder. Then, a batch of samples with the size of 1024 is collected from the experience pool according to FRE and sent to the learner. The learner trains the current agent using these received samples and then stores the trained agent in the player pool. Then, the player pool randomly selects 36 past agents and sends these agents to the evaluator, where each past agent is compared against the current agent 3 times. The initial angle, speed and distance of air combat are randomly selected within a certain range. After the simulations, the number of wins, losses, and draws are stored in the recorder. According to the results of evaluations, it can be seen whether the decision-making ability of the agent has increased after training. Finally, the player pool sends the current agent to the actor to start the next iteration.

Obviously, the more wins, the stronger the decision-making ability of the agent. It should be pointed out that more losses also mean more effective agent training because the agent is compared against its own past version. Therefore, more losses mean a stronger past version of the current agent, which can indicate that the training of the agent is effective. On the contrary, a large number of draws means that the training of the agent may be invalid (a draw represents a missile missing the target or simulation timeout). At the beginning of training, there are usually a large number of draws; at this time, the agent often maneuvers randomly, so its missile easily misses the target, which results in a large number of draws. Therefore, if there are still a large number of draws after training for a period of time, the agent training may be invalid.

3.5. Air Combat State

The input of the neural network is a one-dimensional vector with 44 elements, which are composed of the state of the current time step and the state of the first three time steps. As shown in

Table 1. Air combat state.

State	Symbol	Formula
yaw angle	$\psi$	$\psi ={\psi }_0+{\displaystyle \int \frac{g}{v\cos \gamma }{n}_z\sin \mu }\mathrm{dt}$
pitch angle	$\gamma$	$\gamma ={\gamma }_0+{\displaystyle \int \frac{g}{v}(n_z\cos \mu -\cos \gamma )} \mathrm{dt}$
velocity	v	$v=v_0+{\displaystyle \int g(n_x-\sin \gamma )} \mathrm{dt}$
altitude	z	$z=z_0+{\displaystyle \int v\sin \gamma } \mathrm{dt}$
distance between the two sides	d	$d=‖r_1-r_2‖$
launch missile	f1	0 or 1
yaw angle of the missile	${\psi }_1$	${\psi }_m={\psi }_{m0}+{\displaystyle \int \frac{n_{mc}g}{v_m\cos {\gamma }_m}} \mathrm{dt}$
pitch angle of the missile	${\gamma }_1$	${\gamma }_m={\gamma }_{m0}+{\displaystyle \int \frac{n_{mh}g}{v_m}-\frac{g\cos {\gamma }_m}{v_m}}\hspace{0.17em}\hspace{0.33em}\mathrm{dt}\hspace{0.33em}$
distance between the missile and the other side	d1	$d_1=‖r_{m1}-r_2‖$
heading crossing angle	$\beta$	$\beta =\arccos (\frac{v_1\cdot v_2}{‖v_1‖‖v_2‖})$
launch missile from the other side	f2	0 or 1

, each state contains 11 quantities: $\psi ,\gamma ,v,z,d,f_1,{\psi }_1,{\gamma }_1,d_1,\beta ,f_2$, where $\psi$ and $\gamma$ are the yaw angle and pitch angle of the velocity vector relative to the line of sight, respectively, v is the velocity of the aircraft, z is the flight altitude, d is the distance between the two sides in air combat, r₁ and r₂ are the coordinates of the two sides, respectively, f₁ represents whether the agent should launch a missile, ${\psi }_1$ and ${\gamma }_1$ are the yaw angle and pitch angle of the missile’s velocity vector relative to the line of sight, respectively, d₁ is the distance between the missile and the other agent, r_m1 is the coordinate of the missile in air combat, β is the heading crossing angle, that is the angle between the 2 velocity vectors of the 2 sides, represented by v₁ and v₂ in Table 1, and f₂ represents whether the other side launched a missile. The input layer is followed by 3 hidden layers. Finally, it outputs 4 quantities. The first 3 outputs are normal overload, tangential overload and roll angle, respectively, and the fourth output is whether to launch the missile. The activation function is tanh.

4. Experiments and Results

In order to verify the performance of the proposed method, a large number of experiments are performed in this section. The proposed method is compared with the other four methods from two aspects of statistics and simulation results. In each simulation, the initial states of two sides of air combat (i.e., azimuth angles, velocities and distance between the two sides) are selected from corresponding intervals in

Table 2. Hyperparameters.

Hyperparameter	Value
azimuth angle	(−π/4, π/4)
distance	(40,000 m, 100,000 m)
velocity	(250 m/s, 400 m/s)
batch size	1024
optimizer	Adam
actor learning rate	0.0002
critic learning rate	0.001
actor architecture	(256, 256, 4)
critic architecture	(256, 256, 1)
activate function	tanh
epoch	8
$\gamma$	0.995
$\lambda$	0.98
$\epsilon$	0.2

. Then the two sides begin to maneuver by means of neural networks to generate training samples. Five independent experiments were performed for each method. Each experiment consists of 40 iterations in total, and each iteration consists of 20 cycles, as shown in Figure 2. The experimental hyperparameters are set as follows:

4.1. Ablation Studies

In this section, five methods are used for experiments: FRE, FRE-U, FRE-S, GAE and DSR. FRE stands for using FRE in the objective function and collecting samples according to FRE, FRE-U stands for using FRE in the objective function and collecting samples randomly, FRE-S stands for using FRE in the objective function without intermediate rewards and collecting samples according to FRE, GAE stands for using GAE in the objective function, DSR stands for using DSR in the objective function. In each iteration, the evaluator records the evaluation results of the method. Figure 3 shows the change in the average return of FRE, FRE-U and FRE-S during the agent training process. The solid line represents the mean of average returns of the five experiments, and one point is drawn on the solid line every five gradient steps. The shaded part represents the standard deviation of average returns. As can be seen from Figure 3, since agents have not yet acquired any effective maneuver policy, average returns in the early training are negative. After several times of training, the average returns of FRE and FRE-U have an obvious upward trend, while the average return of FRE-S is always around 0.

The blue line represents the average return of FRE, the orange line represents the average return of FRE-U and the black line represents the average return of FRE-S. Two points about the change of average return need to be declared: 1. In air combat maneuver decision-making, the return value obtained by the agent may be related to whether it can win air combat without causality, that is, the maneuver decision-making ability of air combat agent cannot be compared only according to returns it obtained. A high return value does not mean more probability of a win, and a low return value does not mean less probability of a win. The increase in the return value only means that the agent gradually learns the policy to increase the reward. 2. FRE-S only includes the final reward, while FRE and FRE-U include not only the final reward but also the intermediate reward. Therefore, the possible return value of FRE and FRE-U is more than that of FRE-S. According to the above two points, to accurately evaluate the performance of different methods, we need to use the results of air combat confrontation. Therefore, the results of different methods are compared in the evaluator, where each evaluation contains 108 rounds. The variations of the number of wins, losses, and draws in each method with the number of iterations are shown in Figure 4. The solid line represents the mean of the number of wins, loses or draws of this iteration over the five experiments, and the shaded part represents the standard deviation of the number of wins, loses or draws.

Obviously, the more wins, the stronger the decision-making ability of the agent. It should be pointed out that more losses also mean more effective agent training because the agent is playing against its own past versions. Therefore, more losses represent the stronger the decision-making ability of the past versions of the agent, which can indicate that the training of the agent is effective. On the contrary, a large number of draws means that the training of the agent may be invalid. At the beginning of training, there are usually a large number of draws because, at this time, the agent often maneuvers randomly, so its missile is likely to miss the target, which results in a large number of draws. Therefore, if there are still a large number of draws after training for a period of time, it indicates that the training may be invalid.

As shown in Figure 4a, the number of wins of FRE is the most among the five methods, and it increases faster than the other four methods. The number of wins of FRE-U and GAE also gradually increase, but their increasing rates are significantly slower than that of FRE, and the numbers of wins are only half of that of FRE. At the same time, the number of wins of FRE-U is slightly more than that of GAE, which indicates that final reward estimation is better than generalized advantage estimation in air combat maneuver decision-making. In the first 14 iterations, the number of wins of FRE-S increases rapidly, but it gradually decreases from the 15th iteration, which indicates that the early training of FRE-S is effective, while the later training is invalid. In addition, the winning rate of the first 14 iterations of FRE-S is faster than that of FRE-U, which indicates that sampling according to FRE is an effective method to improve training efficiency. However, the sparse rewards can make training more difficult, so the number of wins of FRE-S in the later period gradually decreases. The number of wins of DSR does not increase significantly, which indicates that DSR is invalid in air combat maneuver decision-making. It should be noted that some shaded parts of FRE-S and DSR are less than zero, it is because the mean values of the winning games of FRE-S and DSR are small, and their variances are large, which results in the shaded parts being less than zero. However, this does not mean that the wins are less than zero.

As shown in Figure 4b, The number of losses of FRE is also the most among the five methods, and it gradually increases during training. At the same time, its increased rate of losses is also the fastest. Because the agent fights against its own past versions, a large number of losses of the current agent indicates that the number of wins by its past versions is large, which means that the training method based on FRE is effective. The number of losses of FRE-U and GAE also increases, which shows that the two methods are also effective. The number of losses of FRE-S first increases and then decreases, with the same trend as its number of wins, which shows that the early training of FRE-S is effective, while the late training is invalid. At the same time, the decrease in the number of losses in the later period of the FRE-S is smaller than that of the number of wins. This is because the opponents are randomly selected from the player pool during evaluations, so it is possible to select the opponents who have been effectively trained in the early training stage. Therefore, its number of losses increases. As for FRE-S, the decrease in the number of losses is smaller than that of the number of wins. Similar to the number of wins, the number of losses of DSR does not increase, which indicates that DSR is invalid in air combat maneuver decision-making.

As shown in Figure 4c, the number of draws of FRE decreases obviously during training, and its amplitude is also large. The decreasing rate of the number of draws of FRE-U is faster than that of GAE, which indicates that the performance of FRE-U is better than GAE. The number of draws of FRE-S first decreased and then increased, and the number of draws of DSR is almost not decreased, which is consistent with the results in Figure 4a,b.

Table 3. Statistical results of evaluations.

	GAE	DSR	FRE-U	FRE-S	FRE
maximum wins	32	14	32	28	55
average wins	9.20	3.75	12.45	10.68	26.22
average loses	7.96	4.37	8.96	9.03	18.08
average draws	90.84	99.88	86.59	88.29	63.70
average time of each decision-making	0.001 s	0.001 s	0.001 s	0.001 s	0.001 s

shows the maximum number of wins, average number of wins, average number of losses and average number of draws from each method’s overall evaluations. As can be seen from Table 3, the maximum number of wins for FRE over all evaluations was 55, the maximum number of wins for FRE-U and FRE-S were 32 and 28, respectively, and the maximum number of wins for GAE and DSR were 32 and 14, respectively. Therefore, the maximum number of wins for FRE was significantly more than that of the other four methods. The average number of wins for FRE was 26.22, 13.77 more than that of FRE-U, 15.44 more than that of FRE-S, 17.02 more than that of GAE and 22.47 more than that of DSR. The average number of losses for FRE was 18.08, 9.12 more than that of FRE-U, 9.05 more than that of FRE-S, 10.12 more than that of GAE, and 13.71 more than that of DSR. The average number of wins for DSR was less than the average number of losses, which shows again that DSR is invalid in air combat maneuver decision-making. In addition, the average wins and losses of FRE-S were more than that of GAE because FRE-S is more efficient in the early stage than GAE, and thus FRE-S accumulated more wins and losses. Table 3 also shows the time consumption of different methods. Because the average time of each decision-making is determined by the scale of neural networks, and the network scales of different methods are the same, so the average time of each decision-making is the same. Meanwhile, it can be seen that the average time of the proposed method is approximately 0.001 s, which meets the real-time requirements.

Above all, comparing the experimental results of FRE-U with GAE and DSR, it can be seen that the effect of the final reward estimation proposed in this paper is better than GAE and DSR. Comparing the experimental results of FRE-S with FRE-U, GAE and DSR, it can be seen that the method of sampling according to FRE can significantly improve the training efficiency. Comparing the experimental results of FRE with FRE-S, FRE-U, GAE and DSR, it can be seen that the intermediate reward proposed in this paper can avoid invalid training caused by sparse reward in the later training stage. Meanwhile, combining FRE with the method of sampling according to FRE can improve training efficiency and performance.

4.2. Simulation Experiments

The above experiments show the statistical results of agent training, and the following experiments show the simulation results of agent training. Figure 5 shows the air combat simulation results of the final agent against the first, eighth, 16th, 24th and 32nd agents starting from the equal initial situation. The initial position of the final agent is (0, 0, 10,000), the initial yaw angle is 0°, and the initial pitch angle is 0°. The initial positions of other agents are (70,000, 70,000, and 10,000), the initial yaw angle is 180°, and the initial pitch angle is 0°. Figure 5a,c,e,g,i are three-dimensional trajectories of the final agent against the first, eighth, 16th, 24th and 32nd agents, respectively. Figure 5b,d,f,h,j are top views corresponding to the three-dimensional trajectories. In Figure 5, the solid line represents the trajectory of the aircraft, the dotted line represents the trajectory of the corresponding missile, and the two dots represent the initial positions of both sides of air combat.

As can be seen from Figure 5a,b, because the first agent has not been trained, it maneuvers randomly, and it can be considered that the first agent cannot make effective maneuver decision-making. It can be seen from Figure 5a that the flight altitude of the first agent gradually rises with random maneuvers, and the final agent also gradually rises in order to defeat it. From the top view of the flight trajectory of the final agent, it can be seen that the final agent flies in the direction of the first agent and launches a missile, and the first agent is hit by the missile of the final agent at last, which indicates that the agent acquired how to attack its target after training of the proposed method.

As can be seen from Figure 5c,d, after launching its missile, the eighth agent soon loses its target, so it flies away from the final agent to avoid the missile of the final agent. At this time, the final agent does not lose its target by means of its maneuver decision-making. Finally, the eighth agent is hit by the missile of the final agent, and the simulation ends. According to the above analysis, the eighth agent does not acquire effective attack policies, but it acquires some evasive policies to get rid of missile tracking, although these policies are not effective enough.

5. Discussion

Trained from scratch, the agent can gradually improve its maneuver decision-making ability by means of the proposed method. Comparing Figure 5e,f with the above four figures, it can be seen that the flight time of the 16th agent’s missile increases, which indicates that the 16th agent has learned some attack policies. However, during the confrontation with the final agent, the missile of the 16th agent loses its target, so the attack capability of the 16th agent is not as good as that of the final agent. 16th agent performs evasive maneuvers after its missile lost the target, and successfully avoids the attack of the final agent’s missile, while the 8th generation agent does not avoid the attack of the final agent’s missile, which indicates that the maneuver decision-making capability of 16th agent is better than that of the eighth agent.

Comparing the trajectory of the 24th agent with that of the 16th agent, it can be seen that the 24th agent is farther away from the missile of the final agent during flight, while the 16th agent is closer to the missile of the final agent during flight, which indicates that the evasive ability of the 24th agent is better than that of the 16th agent. As can be seen from Figure 5i,j, the maneuvers of the 32nd are similar to that of the final agent. After launching a missile, it gradually approaches its target to maintain the detection of the target. At the same time, neither side flies directly to the target because direct flying to the target would make them more likely to be hit by the opponent’s missile. Finally, both sides lose their targets, and the simulation ends. According to the above analysis, the proposed FRE-PPO-based maneuver decision-making method is effective and efficient, which enables agents to learn evasive and attack policies. On the other hand, experiment results indicate that the final reward is more important than the intermediate reward in maneuver decision-making. A possible direction for future research is to improve the proposed method in order to increase the winning rate; for example, it can be combined with an evolution algorithm, which can enhance the exploration of reinforcement learning and generate more high-value samples.

6. Conclusions

Aiming at the problem of autonomous maneuver decision-making, an air combat agent training framework based on FRE-PPO is proposed. Based on the RL algorithm PPO, this method uses self-play to generate training data and evaluates the air combat ability of the trained agents during the training process. To address the problem of poor performance of PPO in maneuver decision-making, this paper proposes to use FRE to replace the advantage function in the PPO optimization objective and proposes to select training samples according to FRE.

Comparing the experimental results of FRE-U with GAE and DSR, it can be concluded that the final reward estimation proposed in this paper is more effective. Comparing the experimental results of FRE-S with that of FRE-U, GAE and DSR, it can be concluded that the proposed method of sampling by FRE can significantly improve the training efficiency. Comparing the experimental results of FRE with those of FRE-S, FRE-U, GAE and DSR, it can be concluded that the proposed intermediate reward can avoid invalid training in the later stage caused by sparse rewards and combining FRE with the method of sampling according to FRE can improve the training efficiency and make the training performance better. According to the air combat simulation results of the final agent against the 1st, 8th, 16th, 24th and 32nd agents, respectively, the agent gradually acquires evasive and attack maneuvers during the training process and meets the real-time requirements.

To sum up, the maneuver decision-making method proposed in this paper has certain effectiveness and superiority and meets the real-time requirements. In future work, enhancing the winning rate is the main research direction since the average wins of the proposed method are not enough, and plenty of draws still exist (fewer draws mean better performance), which indicates that the proposed method should be improved further. Meanwhile, RL algorithms can be combined with evolution algorithms, such as genetic algorithms, to improve the exploration ability, which may lead to better performance and more wins.