Exploring the Use of Invalid Action Masking in Reinforcement Learning: A Comparative Study of On-Policy and Off-Policy Algorithms in Real-Time Strategy Games

Abstract

Invalid action masking is a practical technique in deep reinforcement learning to prevent agents from taking invalid actions. Existing approaches rely on action masking during policy training and utilization. This study focuses on developing reinforcement learning algorithms that incorporate action masking during training but can be used without action masking during policy execution. The study begins by conducting a theoretical analysis to elucidate the distinction between naive policy gradient and invalid action policy gradient. Based on this analysis, we demonstrate that the naive policy gradient is a valid gradient and is equivalent to the proposed composite objective algorithm, which optimizes both the masked policy and the original policy in parallel. Moreover, we propose an off-policy algorithm for invalid action masking that employs the masked policy for sampling while optimizing the original policy. To compare the effectiveness of these algorithms, experiments are conducted using a simplified real-time strategy (RTS) game simulator called Gym-$\mu$RTS. Based on empirical findings, we recommend utilizing the off-policy algorithm for addressing most tasks while employing the composite objective algorithm for handling more complex tasks.

1. Introduction

Reinforcement Learning (RL) has attracted tremendous attention due to its human-level performance in challenging decision-making tasks [1,2,3,4]. However, in some more complex tasks, such as real-time strategy (RTS) games [5,6,7] and multiplayer online battle arena (MOBA) games [8,9,10], the action spaces are high-dimensional, and partial actions are usually invalid in some specific states. For instance, executing a skill on cool-down or buying an item exceeding existing gold will not be allowed. The action availability usually depends on the current state (e.g., skill cool-down finished or having enough gold). It is impractical to design variable action spaces for different states [11]. Therefore, finding an effective way to prevent the agent from choosing invalid actions is significant [12,13,14].

A common approach, called invalid action penalty, assigns a negative reward if the agent takes an invalid action so that the agent learns to avoid invalid actions by minimizing the expected penalty [15,16]. However, this technique cannot prevent the agent from taking invalid actions and can hardly scale on complex tasks [17,18]. An alternative method, action space shaping, works by reducing the full discrete action space to a simpler action space [19], which typically involves non-useful action removals and discretization of continuous action space. However, the tuning of action space shaping has been found to be challenging, and it can occasionally hinder the agent’s ability to successfully accomplish desired tasks [20]. Another popular technique is invalid action masking, which filters invalid actions by setting the probability of all invalid actions to near zero [7,8,9]. This technique ensures the agent can avoid selecting impossible or unavailable actions during training. In addition, it reduces the dimension of exploration space and improves exploration efficiency. Invalid action masking has been widely applied to many domains besides RTS or MOBA games, such as traffic signal control [21], unmanned vehicle [22], adaptive voltage control [12], etc.

Huang et al. [23] provides a closer look at invalid action masking for the first time, categorizing it into two types, invalid action policy gradient (utilizing the masked policy for gradient calculation) and naive invalid action policy gradient (utilizing the original policy for gradient calculation). Through a numerical example, it demonstrates that invalid action masking alters the policy gradient and that the invalid action policy gradient corresponds to a valid policy gradient. In addition, it is noted that employing the naive invalid action policy gradient may lead to instability. However, empirical results in [23] show that naive invalid action masking can outperform the invalid action policy gradient when the masking is removed. This finding challenges the existing theoretical analysis in [23], which has inspired our interest in understanding the mechanism behind naive invalid action masking.

Inspired by aforementioned analysis, this study provides a theoretical analysis and reveals why naive invalid action masking is an effective way to train the agent to avoid invalid actions explicitly. Specifically, the regular invalid action masking is to replace the logits of invalid actions with a large negative number [23], which we call logit-level masking. To derive the policy gradient, we present a novel masking called action-level masking. We prove that two types of masking are equivalent so that the conclusions derived by one also apply to another. We utilize the action-level masking to verify that invalid action masking indeed changes the policy gradient. More interestingly, we find that the difference between naive invalid action policy gradient and invalid action policy gradient is a state-dependent factor that directly impacts the probabilities associated with invalid actions. Based on this, we prove that the naive invalid action policy gradient (taking the masked policy as behavior policy and using the original policy to calculate the gradient) is actually a valid gradient that concurrently optimizes the masked policy and the original policy. From this conclusion, we design a Composite Objective Invalid Action Masking (CO-IAM) algorithm that concurrently optimizes the masked policy and reduces the probabilities of invalid actions. Furthermore, we present on-policy and off-policy algorithms for Invalid Action Masking, called On-PIAM and Off-PIAM, respectively. The On-PIAM algorithm aims to optimize the performance of the masked policy, which is the same as that of the invalid action policy gradient in [23]. The Off-PIAM adopts the Importance Sampling (IS) technique and optimizes the original policy. The main contributions of this paper are summarized as follows:

• We introduce the concept of action-level masking, which is proven to be equivalent to logit-level masking. The theoretical analysis based on action-level masking demonstrates that the difference between the invalid action policy gradient and the naive invalid action policy gradient is a state-dependent gradient that impacts the probability distribution of invalid actions. This contributes to a better understanding of the impact of invalid action masking on gradients.
• We prove that the naive policy gradient is indeed a valid gradient that optimizes both the masked policy and original policy concurrently. Although this is a counter-intuitive conclusion, it ensures that we can directly use the naive policy gradient for training without causing instability. In addition, we also propose an off-policy algorithm for invalid action masking that focuses on optimizing the original policy.
• We conduct experiments based on the Gym-$\mu$RTS platform [24] to compare the performance of the proposed algorithms and invalid action penalty. The results show that Off-PIAM outperforms other algorithms even if the masking is removed, and CO-IAM scales well on complex tasks. Based on these findings, we conclude that the Off-PIAM algorithm is suitable for addressing most tasks, while the CO-IAM algorithm is well-suited for handling more complex tasks.

2. Background

2.1. Reinforcement Learning

Consider a standard reinforcement learning problem in which an agent interacts with a stochastic environment and attempts to maximize cumulative reward [25]. The stochastic environment is formalized by a Markov Decision Process (MDP), which is denoted as the tuple $\mathcal{M}=\left(\mathcal{S},\mathcal{A},\mathcal{P},\rho ,R,\gamma \right)$, a state space $\mathcal{S}$, an action space $\mathcal{A}$, a transition probability distribution $\mathcal{P}:\mathcal{S}\times \mathcal{A}\times \mathcal{S}\to [0,1]$, an initial state distribution $\rho :\mathcal{S}\to [0,1]$, a reward function $R:\mathcal{S}\times \mathcal{A}\to \mathbb{R}$, and a discount factor $\gamma \in [0,1)$. Without prior knowledge about the transition probability distribution, the agent utilizes its policy ${\pi }_{\theta }$ to collect a trajectory $\tau =\{s_0,a_0,r_0,s_1,a_1,r_1,\cdots \}$ for training. The agent improves its policy ${\pi }_{\theta }$ to maximize the expected discounted return for each episode

$$\begin{array}{c}\hfill J\left({\pi }_{\theta }\right)={\mathbb{E}}_{s_0\sim \rho ,a_t\sim {\pi }_{\theta },s_{t+1}\sim \mathcal{P}(·|s_t,a_t)}\left[\sum _{t=0}^H{\gamma }^t{r}_t\right],\end{array}$$

(1)

where $r_t=R\left(s_t,a_t\right)$, and H is the maximal length of one episode. The fundamental result underlying this problem is the Policy Gradient Theorem [25], which derives the policy gradient ${\nabla }_{\theta }J\left({\pi }_{\theta }\right)$ with respect to the policy parameter $\theta$. Executing gradient ascent $\theta \leftarrow \theta +{\nabla }_{\theta }J\left({\pi }_{\theta }\right)$ results in policy improvement and maximizes the expected discounted return. The policy gradient estimate is

$$\begin{array}{c}\hfill g={\mathbb{E}}_{\tau \sim {\pi }_{\theta }}\left[\sum _{t=0}^H{\nabla }_{\theta }log{\pi }_{\theta }\left(a_t|s_t\right)G_t\right],\end{array}$$

(2)

where $G_t={\sum }_{t^{\prime }=t}^H{\gamma }^{t^{\prime }-t}{r}_{t^{\prime }}$ is the discounted return for a trajectory following step t. The General Advantage Estimation (GAE) technique [26] is proposed to reduce the variance of the policy gradient estimate at the cost of some bias

$$\begin{array}{c}\hfill \widehat{g}={\mathbb{E}}_{\tau \sim {\pi }_{\theta }}\left[\sum _{t=0}^H{\nabla }_{\theta }log{\pi }_{\theta }\left(a_t|s_t\right){\widehat{A}}_t\right],\end{array}$$

(3)

where ${\widehat{A}}_t$ is a GAE estimator of the advantage function [26]. The policy gradient algorithm requires on-policy trajectory for training (i.e., the target policy and behavior policy should be consistent).

2.2. Logit-Level Masking

In typical deep reinforcement learning algorithms, a neural network is commonly used to represent the policy. In the case of a discrete action space, the action probability distribution is produced by implementing the Softmax operation on the output logits of the neural network, which is the framework we will assume in the rest of the paper.

Invalid action masking is a practical technique in deep reinforcement learning to prevent the agent from taking impossible or unavailable actions. The most popular way for invalid action masking is to replace the logits of invalid actions with negative infinity (or a large negative number in practice) [23], which we call logit-level masking. Logit-level masking works by applying a state-dependent masking function $m(·|s)$ when calculating action probability distribution [23], as shown in Figure 1.

Specifically, let us define an MDP with the state space $\mathcal{S}$ and the discrete action space $\mathcal{A}=\{a_1,a_2,\cdots ,a_n\}$, where n is the number of discrete actions. The policy ${\pi }_{\theta }$ is parameterized by parameters $\theta$ of the neural network. Taking any state $s\in \mathcal{S}$ as input, the output logits of the neural network are represented by $l_{\theta }(·|s)=[l_{\theta }\left(a_1\right|s),l_{\theta }\left(a_2\right|s),\cdots ,l_{\theta }\left(a_n\right|s)]$. The logit-level masking function is defined as

$$\begin{array}{c}\hfill m\left(l_{\theta }\left(a_i\right|s)\right)=\left\{\begin{array}{cc}{l}_{\theta }\left(a_i\right|s)\hfill & \mathrm{if}{a}_i\mathrm{is}\mathrm{valid}\mathrm{in}s,\hfill \\ -\infty \hfill & \mathrm{if}{a}_i\mathrm{is}\mathrm{invalid}\mathrm{in}s.\hfill \end{array}\right.\end{array}$$

(4)

Then, we can obtain the re-normalized action probability distribution ${\pi }_{\theta }^m(·\mid s)$ by implementing the Softmax operation on the masked logits

$$\begin{array}{cc}\hfill {\pi }_{\theta }^m(·\mid s)& =Softmax\left(m\left(l_{\theta }(·|s)\right)\right)\hfill \\ \hfill & =\left[\frac{exp\left(m\left(l_{\theta }\left(a_1\right|s)\right)\right)}{{\displaystyle \sum _{a_i\in \mathcal{A}}}exp\left(m\left(l_{\theta }\left(a_i\right|s)\right)\right)},\cdots ,\frac{exp\left(m\left(l_{\theta }\left(a_n\right|s)\right)\right)}{{\displaystyle \sum _{a_i\in \mathcal{A}}}exp\left(m\left(l_{\theta }\left(a_i\right|s)\right)\right)}\right].\hfill \end{array}$$

(5)

Since the values of logits associated with invalid actions are $-\infty$, the sample probabilities of invalid actions are zero as ${lim}_{l\to -\infty }exp\left(l\right)=0$.

3. Invalid Action Masking

This section proposes a novel paradigm for invalid action masking, called action-level masking. Compared with logit-level masking, the key benefit of action-level masking is its suitability for conducting theoretical analyses of action masking. Furthermore, we verify the equivalence of logit-level masking and action-level masking. Thus, the conclusions derived by Huang et al. [23] are also applicable to action-level masking.

3.1. Action-Level Masking

Different from logit-level masking, action-level masking works by replacing the action probability of invalid actions to be masked by zero and re-normalizing the action probability distribution (see Figure 2).

Taking any state $s\in \mathcal{S}$ as input, the original policy ${\pi }_{\theta }$ outputs the action probability distribution $[{\pi }_{\theta }\left(a_1\right|s),{\pi }_{\theta }\left(a_2\right|s),\cdots ,{\pi }_{\theta }\left(a_n\right|s)]$, where

$$\begin{array}{c}\hfill {\pi }_{\theta }\left(a_i\right|s)=\frac{exp\left(l_{\theta }\left(a_i\right|s)\right)}{{\sum }_{a_j\in \mathcal{A}}exp\left(l_{\theta }\left(a_j\right|s)\right)}.\end{array}$$

(6)

The action-level masking function $\varpi (·|s)$ is a state-dependent binary function

$$\begin{array}{cc}\hfill \varpi (·|s)& =[\varpi \left(a_1\right|s),\varpi \left(a_2\right|s),\cdots ,\varpi \left(a_n\right|s)],\hfill \\ \hfill \mathrm{where}\varpi \left(a_i\right|s)& =\left\{\begin{array}{cc}1,\hfill & \mathrm{if}{a}_i\mathrm{is}\mathrm{valid}\mathrm{in}s,\hfill \\ 0,\hfill & \mathrm{if}{a}_i\mathrm{is}\mathrm{invalid}\mathrm{in}s.\hfill \end{array}\right.\hfill \end{array}$$

(7)

Then, we can re-normalize the Hadamard product of the original policy and action-level masking function and obtain the masked policy

$$\begin{array}{cc}\hfill {\pi }_{\theta }^{\varpi }(·|s)& =Normalize\left({\pi }_{\theta }(·|s)\circ \varpi (·|s)\right)\hfill \\ \hfill & =\left[\frac{{\pi }_{\theta }\left(a_1\right|s)\varpi \left(a_1\right|s)}{{\displaystyle \sum _{a_i\in \mathcal{A}}}{\pi }_{\theta }\left(a_i\right|s)\varpi \left(a_i\right|s)},\cdots ,\frac{{\pi }_{\theta }\left(a_n\right|s)\varpi \left(a_n\right|s)}{{\displaystyle \sum _{a_i\in \mathcal{A}}}{\pi }_{\theta }\left(a_i\right|s)\varpi \left(a_i\right|s)}\right].\hfill \end{array}$$

(8)

We assume that for any state $s\in \mathcal{S}$, at least one action in the action space should be valid (i.e., ${\sum }_{a_i\in \mathcal{A}}\left(\varpi \left(a_i\right|s)\right)\ne 0,\forall s\in \mathcal{S}$). However, it is important to note that this assumption may not hold true in certain scenarios, such as when a passive unit is surrounded from all sides. To address this, we propose incorporating a no-operation action into the action space so that at least one action is always legal in any given state. The sample probability of any invalid action $a_{\mathrm{inv}}$ is zero since the corresponding value of the action-level masking function $\varpi \left(a_{\mathrm{inv}}\right|s)$ is equal to zero. We design action-level masking because it is conducive to the theoretical analysis of the policy gradient algorithm.

3.2. Equivalence Verification

This section will be concerned with verifying the equivalence of logit-level masking and action-level masking, based on which the conclusions derived by Huang et al. [23] are also applicable to action-level masking. The equivalence holds if two types of invalid action masking satisfy that:

$$\begin{array}{c}\hfill {\pi }_{\theta }^m\left(a\right|s)={\pi }_{\theta }^{\varpi }\left(a\right|s),\forall s\in \mathcal{S},\forall a\in \mathcal{A}.\end{array}$$

(9)

For any state $s\in \mathcal{S}$, the action space can be divided into two parts, valid action space ${\mathcal{A}}_{\mathrm{v}}\left(s\right)$ and invalid action space ${\mathcal{A}}_{\mathrm{inv}}\left(s\right)$. We have:

$$\begin{array}{c}\hfill \mathcal{A}={\mathcal{A}}_{\mathrm{v}}\left(s\right)\cup {\mathcal{A}}_{\mathrm{inv}}\left(s\right),{\mathcal{A}}_{\mathrm{v}}\left(s\right)\cap {\mathcal{A}}_{\mathrm{inv}}\left(s\right)=\varnothing .\end{array}$$

(10)

According to (5), for any valid action $a_{\mathrm{v}}\in {\mathcal{A}}_{\mathrm{v}}\left(s\right)$, the action probability of the logit-level masked policy is:

$$\begin{array}{cc}\hfill {\pi }_{\theta }^m\left(a_{\mathrm{v}}\right|s)& =\frac{exp\left(m\left(l_{\theta }\left(a_{\mathrm{v}}\right|s)\right)\right)}{{\sum }_{a_i\in \mathcal{A}}exp\left(m\left(l_{\theta }\left(a_i\right|s)\right)\right)}\hfill \\ \hfill & =\frac{exp\left(m\left(l_{\theta }\left(a_{\mathrm{v}}\right|s)\right)\right)}{{\displaystyle \sum _{a_i\in {\mathcal{A}}_{\mathrm{v}}}}exp\left(m\left(l_{\theta }\left(a_i\right|s)\right)\right)+{\displaystyle \sum _{a_j\in {\mathcal{A}}_{\mathrm{inv}}}}exp\left(m\left(l_{\theta }\left(a_j\right|s)\right)\right)}\hfill \\ \hfill & =\frac{exp\left(l_{\theta }\left(a_{\mathrm{v}}\right|s)\right)}{{\sum }_{a_i\in {\mathcal{A}}_{\mathrm{v}}}exp\left(l_{\theta }\left(a_i\right|s)\right)}\hfill \end{array}$$

(11)

According to (8), for any valid action $a_{\mathrm{v}}\in {\mathcal{A}}_{\mathrm{v}}\left(s\right)$, the action probability of the action-level masking policy is:

$$\begin{array}{cc}\hfill {\pi }_{\theta }^{\varpi }\left(a_{\mathrm{v}}\right|s)& =\frac{{\pi }_{\theta }\left(a_{\mathrm{v}}\right|s)\varpi \left(a_{\mathrm{v}}\right|s)}{{\sum }_{a_i\in \mathcal{A}}{\pi }_{\theta }\left(a_i\right|s)\varpi \left(a_i\right|s)}=\frac{{\pi }_{\theta }\left(a_{\mathrm{v}}\right|s)}{{\sum }_{a_i\in {\mathcal{A}}_{\mathrm{v}}\left(s\right)}{\pi }_{\theta }\left(a_i\right|s)}\hfill \\ \hfill & =\frac{exp\left(l_{\theta }\left(a_{\mathrm{v}}\right|s)\right)}{{\displaystyle \sum _{a_j\in \mathcal{A}}}exp\left(l_{\theta }\left(a_j\right|s)\right)}/\phantom{\frac{exp\left(l_{\theta }\left(a_{\mathrm{v}}\right|s)\right)}{{\displaystyle \sum _{a_j\in \mathcal{A}}}exp\left(l_{\theta }\left(a_j\right|s)\right)}{\displaystyle \sum _{a_i\in A_{\mathrm{v}}}}\frac{exp\left(l_{\theta }\left(a_i\right|s)\right)}{{\displaystyle \sum _{a_j\in \mathcal{A}}}exp\left(l_{\theta }\left(a_j\right|s)\right)}}\sum _{a_i\in {\mathcal{A}}_{\mathrm{v}}\left(s\right)}\frac{exp\left(l_{\theta }\left(a_i\right|s)\right)}{\sum _{a_j\in \mathcal{A}}exp\left(l_{\theta }\left(a_j\right|s)\right)}\hfill \\ \hfill & =\frac{exp\left(l_{\theta }\left(a_{\mathrm{v}}\right|s)\right)}{{\sum }_{a_i\in {\mathcal{A}}_{\mathrm{v}}\left(s\right)}exp\left(l_{\theta }\left(a_i\right|s)\right)}\hfill \end{array}$$

(12)

In light of (11) and (12), one has:

$$\begin{array}{c}\hfill {\pi }_{\theta }^m\left(a_{\mathrm{v}}\right|s)={\pi }_{\theta }^{\varpi }\left(a_{\mathrm{v}}\right|s),\forall s\in \mathcal{S},\forall a_{\mathrm{v}}\in {\mathcal{A}}_{\mathrm{v}}\left(s\right).\end{array}$$

(13)

Obviously, the probabilities of invalid actions of both policies are equal to zero:

$$\begin{array}{c}\hfill {\pi }_{\theta }^m\left(a_{\mathrm{inv}}\right|s)={\pi }_{\theta }^{\varpi }\left(a_{\mathrm{inv}}\right|s)=0,\forall s\in \mathcal{S},\forall a_{\mathrm{inv}}\in {\mathcal{A}}_{\mathrm{inv}}\left(s\right).\end{array}$$

(14)

Combining (13) with (14), we obtain that:

$$\begin{array}{c}\hfill {\pi }_{\theta }^m\left(a\right|s)={\pi }_{\theta }^{\varpi }\left(a\right|s),\forall s\in \mathcal{S},\forall a\in \mathcal{A}.\end{array}$$

(15)

Thus, logit-level masking and action-level masking are equivalent. This is a significant result because it shows that the conclusions of logit-level masking [23] also apply to action-level masking. It is important to note that the substitution of negative infinity with a significantly negative value may introduce minor rounding errors. However, these errors do not impact the overall equivalence. While both action-level masking and logit-level masking are practically equivalent, action-level masking enables the explicit derivation of policy gradients, whereas logit-level masking does not. In the following, action-level masking is used to derive the policy gradient algorithm for invalid action masking.

3.3. Masking Changes Policy Gradient

This section focuses on the impact of invalid action masking on policy updates. Huang et al. [23] designed a numerical example to illustrate that the gradient generated by invalid action masking differs from the naive policy gradient. As a supplement, we further investigate this finding and reveal the difference between the naive policy gradient and the invalid action masking policy gradient.

In regular policy-based algorithms, the original policy ${\pi }_{\theta }$ is employed to collect samples and calculate the policy gradient using the policy gradient theorem [25]. To avoid invalid actions, we apply the invalid action masking to the policy ${\pi }_{\theta }$, resulting in a masked policy denoted as ${\pi }_{\theta }^{\varpi }$. The straightforward approach to policy updating is the direct utilization of the original policy gradient, called naive policy gradient:

$$\begin{array}{c}\hfill g_{\mathrm{naive}}={\mathbb{E}}_{\tau \sim {\pi }_{\theta }^{\varpi }}\left[\sum _{t=0}^H{\nabla }_{\theta }log{\pi }_{\theta }\left(a_t|s_t\right){\widehat{A}}_t\right].\end{array}$$

(16)

The naive policy gradient is intuitively improper because the policy gradient algorithm is an on-policy algorithm, but the target policy (${\pi }_{\theta }$) and behavior policy (${\pi }_{\theta }^{\varpi }$) are different. Huang et al. [23] proposed the invalid action policy gradient, which is the policy gradient for the masked policy ${\pi }_{\theta }^{\varpi }$

$$\begin{array}{c}\hfill g_{\mathrm{mask}}={\mathbb{E}}_{\tau \sim {\pi }_{\theta }^{\varpi }}\left[\sum _{t=0}^H{\nabla }_{\theta }log{\pi }_{\theta }^{\varpi }\left(a_t|s_t\right){\widehat{A}}_t\right].\end{array}$$

(17)

The distinction between the two policy gradients lies in different target policies for log probability calculation. Next, we provide a theoretical analysis to reveal the difference between the naive policy gradient (16) and invalid action policy gradient (17).

Theorem 1. 

Consider a masked policy ${\pi }_{\theta }^{\varpi }$ produced by applying invalid action masking to the original policy ${\pi }_{\theta }$. The difference between the naive policy gradient (16) and invalid action policy gradient (17) is a state-dependent gradient

$$\begin{array}{c}\hfill g_{\mathrm{mask}}-g_{\mathrm{naive}}={\mathbb{E}}_{\tau \sim {\pi }_{\theta }^{\varpi }}\left[\sum _{t=0}^H\frac{{\widehat{A}}_t}{\mathcal{I}(s_t;\theta )}\sum _{a_i\in {\mathcal{A}}_{\mathrm{inv}}\left(s_t\right)}{\nabla }_{\theta }{\pi }_{\theta }\left(a_i\right|s_t)\right]\end{array}$$

(18)

where $\mathcal{I}(s_t;\theta )={\sum }_{a_j\in {\mathcal{A}}_{\mathrm{v}}\left(s_t\right)}{\pi }_{\theta }\left(a_j\right|s_t)$.

Proof of Theorem 1. 

According to (8), the gradient item in $g_{\mathrm{mask}}$ can be written as:

$$\begin{array}{c}\hfill {\nabla }_{\theta }log{\pi }_{\theta }^{\varpi }\left(a_t\right|s_t)={\nabla }_{\theta }log\frac{{\pi }_{\theta }\left(a_t\right|s_t)\varpi \left(a_t\right|s_t)}{{\sum }_{a_i\in \mathcal{A}}{\pi }_{\theta }\left(a_i\right|s_t)\varpi \left(a_i\right|s_t)}.\end{array}$$

(19)

Since $a_t$ is sampled from the masked policy ${\pi }_{\theta }^{\varpi }$, $a_t$ must be a valid action, and $\varpi \left(a_t\right|s_t)=1$. Then, the gradient item (19) can be written as:

$$\begin{array}{cc}\hfill {\nabla }_{\theta }log{\pi }_{\theta }^{\varpi }\left(a_t\right|s_t)& ={\nabla }_{\theta }log\frac{{\pi }_{\theta }\left(a_t\right|s_t)}{{\displaystyle \sum _{a_i\in \mathcal{A}}}{\pi }_{\theta }\left(a_i\right|s_t)\varpi \left(a_i\right|s_t)}\hfill \\ \hfill & ={\nabla }_{\theta }log{\pi }_{\theta }\left(a_t\right|s_t)-{\nabla }_{\theta }log\sum _{a_i\in \mathcal{A}}{\pi }_{\theta }\left(a_i\right|s_t)\varpi \left(a_i\right|s_t)\hfill \\ \hfill & ={\nabla }_{\theta }log{\pi }_{\theta }\left(a_t\right|s_t)-\frac{{\nabla }_{\theta }{\displaystyle \sum _{a_i\in \mathcal{A}}}{\pi }_{\theta }\left(a_i\right|s_t)\varpi \left(a_i\right|s_t)}{{\displaystyle \sum _{a_j\in \mathcal{A}}}{\pi }_{\theta }\left(a_j\right|s_t)\varpi \left(a_j\right|s_t)}\hfill \end{array}$$

(20)

Since ${\nabla }_{\theta }{\sum }_{a_i\in \mathcal{A}}{\pi }_{\theta }\left(a_i\right|s_t)={\nabla }_{\theta }1=0$, (20) is written as:

$$\begin{array}{c}\hfill {\nabla }_{\theta }log{\pi }_{\theta }^{\varpi }\left(a_t\right|s_t)={\nabla }_{\theta }log{\pi }_{\theta }\left(a_t\right|s_t)+\frac{{\sum }_{a_i\in \mathcal{A}}{\nabla }_{\theta }{\pi }_{\theta }\left(a_i\right|s_t)(1-\varpi \left(a_i\right|s_t))}{{\sum }_{a_j\in \mathcal{A}}{\pi }_{\theta }\left(a_j\right|s_t)\varpi \left(a_j\right|s_t)}.\end{array}$$

(21)

According to the definition of masking function (7), we have $1-\varpi \left(a_i\right|s_t)=0,\forall a_i\in {\mathcal{A}}_{\mathrm{v}}\left(s_t\right)$ and $\varpi \left(a_i\right|s_t)=0,\forall a_i\in {\mathcal{A}}_{\mathrm{inv}}\left(s_t\right)$. Substituting the above equation into (21), we obtain that

$$\begin{array}{cc}\hfill {\nabla }_{\theta }log{\pi }_{\theta }^{\varpi }\left(a_t\right|s_t)& ={\nabla }_{\theta }log{\pi }_{\theta }\left(a_t\right|s_t)+\frac{{\displaystyle \sum _{a_i\in {\mathcal{A}}_{\mathrm{inv}}\left(s_t\right)}}{\nabla }_{\theta }{\pi }_{\theta }\left(a_i\right|s_t)}{{\displaystyle \sum _{a_j\in {\mathcal{A}}_{\mathrm{v}}\left(s_t\right)}}{\pi }_{\theta }\left(a_j\right|s_t)}\hfill \\ \hfill & ={\nabla }_{\theta }log{\pi }_{\theta }\left(a_t\right|s_t)+\frac{1}{\mathcal{I}(s_t;\theta )}\sum _{a_i\in {\mathcal{A}}_{\mathrm{inv}}\left(s_t\right)}{\nabla }_{\theta }{\pi }_{\theta }\left(a_i\right|s_t),\hfill \end{array}$$

(22)

where $\mathcal{I}(s_t;\theta )={\sum }_{a_j\in {\mathcal{A}}_{\mathrm{v}}\left(s_t\right)}{\pi }_{\theta }\left(a_j\right|s_t)$. Substituting (22) into (17), we obtain that

$$\begin{array}{cc}\hfill g_{mask}& ={\mathbb{E}}_{\tau \sim {\pi }_{\theta }^{\varpi }}\left[\sum _{t=0}^H\left({\nabla }_{\theta }log{\pi }_{\theta }\left(a_t\right|s_t)+\frac{1}{\mathcal{I}(s_t;\theta )}\sum _{a_i\in {\mathcal{A}}_{\mathrm{inv}}\left(s_t\right)}{\nabla }_{\theta }{\pi }_{\theta }\left(a_i\right|s_t)\right){\widehat{A}}_t\right]\hfill \\ \hfill & =g_{naive}+{\mathbb{E}}_{\tau \sim {\pi }_{\theta }^{\varpi }}\left[\sum _{t=0}^H\frac{{\widehat{A}}_t}{\mathcal{I}(s_t;\theta )}\sum _{a_i\in {\mathcal{A}}_{\mathrm{inv}}\left(s_t\right)}{\nabla }_{\theta }{\pi }_{\theta }\left(a_i\right|s_t)\right]\hfill \end{array}$$

This completes the Proof of Theorem 1.    □

Theorem 1 illustrates that the gradient produced by invalid action masking differs from the naive policy gradient. Specifically, the residual item is the sum of gradients corresponding to invalid actions.

The experimental results derived by Huang et al. [23] show that once the masking is removed, the performance of the original policy ${\pi }_{\theta }$ will be unstable. We infer that the residual item in the invalid action policy gradient results in the poor performance of the original policy. In the rest of this paper, we propose a novel algorithm for invalid action masking based on Theorem 1.

4. Practical Algorithms for Action Masking

Invalid action masking transforms the original policy (${\pi }_{\theta }$) into the masked policy (${\pi }_{\theta }^{\varpi }$), which is used to collect samples. The policy gradient algorithm is an on-policy algorithm where the target policy and behavior policy should be consistent. Intuitively, the naive policy gradient is improper because the behavior policy (${\pi }_{\theta }^{\varpi }$) is different from the target policy (${\pi }_{\theta }$). The design of algorithms depends on which policy we expect to optimize.

This section lists three policy gradient algorithms for invalid action masking, the On-Policy Invalid Action Masking (On-PIAM) algorithm [23], the Off-Policy Invalid Action Masking (Off-PIAM) algorithm, and the Composite Objective Invalid Action Masking (CO-IAM) algorithm.

Each algorithm utilizes the masked policy to sample but optimize different policies. The On-PIAM algorithm optimizes the performance of the masked policy (${\pi }_{\theta }^{\varpi }$), while the Off-PIAM algorithm optimizes the original policy (${\pi }_{\theta }$). The CO-IAM algorithm optimizes the masked policy (${\pi }_{\theta }^{\varpi }$) and the original policy (${\pi }_{\theta }$) in parallel. Before introducing these algorithms, we first provide a brief summary of the above algorithms in

Table 1. Summary of On-PIAM, Off-PIAM, and CO-IAM algorithms.

Algorithms	Behavior Policy	Target Policy	Objective	Policy Gradient	PPO Ratio
On-PIAM [23]	${\pi }_{\theta }^{\varpi }$	${\pi }_{\theta }^{\varpi }$	$J\left({\pi }_{\theta }^{\varpi }\right)$	Equation (23)	$r_t^{\varpi }\left(\theta \right)={\pi }_{\theta }^{\varpi }\left(a_t|s_t\right)/{\pi }_{{\theta }_{\mathrm{old}}}^{\varpi }\left(a_t|s_t\right)$
Off-PIAM	${\pi }_{\theta }^{\varpi }$	${\pi }_{\theta }$	$J\left({\pi }_{\theta }\right)$	Equation (27)	$r_t^{\mathrm{IS}}\left(\theta \right)={\pi }_{\theta }\left(a_t|s_t\right)/{\pi }_{{\theta }_{\mathrm{old}}}^{\varpi }\left(a_t|s_t\right)$
CO-IAM	${\pi }_{\theta }^{\varpi }$	${\pi }_{\theta }^{\varpi },{\pi }_{\theta }$	$J\left({\pi }_{\theta }^{\varpi }\right)-c_{\varphi }\Phi \left({\pi }_{\theta }\right)$	Equation (40)	$r_t^{\mathrm{CO}}\left(\theta \right)={\pi }_{\theta }\left(a_t|s_t\right)/{\pi }_{{\theta }_{\mathrm{old}}}\left(a_t|s_t\right)$

.

4.1. On-Policy Algorithm

In the On-Policy Invalid Action Masking (On-PIAM) algorithm [23], the target and behavior policies are both masked policies (${\pi }_{\theta }^{\varpi }$). Since ${\pi }_{\theta }^{\varpi }$ is differentiable to its parameters $\theta$, the policy gradient algorithm can be directly applied to train ${\pi }_{\theta }^{\varpi }$:

$$\begin{array}{c}\hfill {\nabla }_{\theta }J\left({\pi }_{\theta }^{\varpi }\right)={\mathbb{E}}_{\tau \sim {\pi }_{\theta }^{\varpi }}\left[\sum _{t=0}^H{\nabla }_{\theta }log{\pi }_{\theta }^{\varpi }\left(a_t|s_t\right){\widehat{A}}_t\right].\end{array}$$

(23)

To train the agent more robustly, we can take Proximal Policy Optimization (PPO) algorithm [27] as the training algorithm:

$$\begin{array}{cc}\hfill L_{\mathrm{On}-\mathrm{P}}^{CLIP}\left(\theta \right)& ={\mathbb{E}}_{\tau \sim {\pi }_{\theta }^{\varpi }}\left[min\left(r_t^{\varpi }\left(\theta \right){\widehat{A}}_t,clip\left(r_t^{\varpi }\left(\theta \right),ϵ\right){\widehat{A}}_t\right)\right],\hfill \end{array}$$

(24)

$$\begin{array}{cc}\hfill \mathrm{where}{r}_t^{\varpi }\left(\theta \right)& =\frac{{\pi }_{\theta }^{\varpi }\left(a_t|s_t\right)}{{\pi }_{{\theta }_{\mathrm{old}}}^{\varpi }\left(a_t|s_t\right)},\mathrm{and}clip(r,ϵ)=\left\{\begin{array}{cc}1+ϵ,\hfill & r>1+ϵ\hfill \\ r,\hfill & 1-ϵ⩽r⩽1+ϵ\hfill \\ 1-ϵ,\hfill & r<1-ϵ\hfill \end{array}\right..\hfill \end{array}$$

(25)

During training, the agent also maximizes the policy’s entropy to encourage exploration. We combine these optimization terms and obtain the following optimization objective

$$\begin{array}{c}\hfill L_{\mathrm{On}-\mathrm{P}}^{PPO}\left(\theta \right)=L_{\mathrm{On}-\mathrm{P}}^{CLIP}\left(\theta \right)+c_e\mathcal{H}\left({\pi }_{\theta }^{\varpi }\right).\end{array}$$

(26)

where $\mathcal{H}(·)$ is an entropy bonus and $c_e$ is a coefficient for the entropy item.

After the training, we obtain a trained policy ${\pi }_{\theta }^{\varpi }$ rather than ${\pi }_{\theta }$. Note that we cannot remove the masking when using the policy because the target policy is the masked policy ${\pi }_{\theta }^{\varpi }$.

4.2. Off-Policy Algorithm

In the Off-Policy Invalid Action Masking (Off-PIAM) algorithm, we treat ${\pi }_{\theta }^{\varpi }$ as the behavior policy, and we still optimize the original policy (${\pi }_{\theta }$). We can adapt the Importance Sampling (IS) technique to derive an unbiased estimator of a policy gradient that relies on off-policy trajectories. This is known as an off-policy policy gradient [28]. The gradient of $J\left({\pi }_{\theta }\right)$ corresponding to its parameter $\theta$ is [29]:

$$\begin{array}{c}\hfill {\nabla }_{\theta }J\left({\pi }_{\theta }\right)={\mathbb{E}}_{\tau \sim {\pi }_{{\theta }_{\mathrm{old}}}^{\varpi }}\left[\left(\prod _{t=0}^H\frac{{\pi }_{\theta }\left(a_t|s_t\right)}{{\pi }_{{\theta }_{\mathrm{old}}}^{\varpi }\left(a_t|s_t\right)}\right)\sum _{t=0}^H{\nabla }_{\theta }log{\pi }_{\theta }\left(a_t|s_t\right){\widehat{A}}_t\right].\end{array}$$

(27)

Even though the above policy gradient is unbiased, the product of the important sampling ratio results in high variance [30]. We can clip the gradient the same way as PPO [27,31]:

$$\begin{array}{cc}\hfill L_{\mathrm{Off}-\mathrm{P}}^{CLIP}\left(\theta \right)& ={\mathbb{E}}_{\tau \sim {\pi }_{{\theta }_{\mathrm{old}}}^{\varpi }}\left[min\left(r_t^{\mathrm{IS}}\left(\theta \right){\widehat{A}}_t,clip\left(r_t^{\mathrm{IS}}\left(\theta \right),ϵ\right){\widehat{A}}_t\right)\right],\hfill \end{array}$$

(28)

$$\begin{array}{cc}\hfill \mathrm{where}{r}_t^{\mathrm{IS}}\left(\theta \right)& =\frac{{\pi }_{\theta }\left(a_t|s_t\right)}{{\pi }_{{\theta }_{\mathrm{old}}}^{\varpi }\left(a_t|s_t\right)}.\hfill \end{array}$$

(29)

By adding the entropy item, the optimization objective is:

$$\begin{array}{c}\hfill L_{\mathrm{Off}-\mathrm{P}}^{PPO}\left(\theta \right)=L_{\mathrm{Off}-\mathrm{P}}^{CLIP}\left(\theta \right)+c_e\mathcal{H}\left({\pi }_{\theta }^{\varpi }\right).\end{array}$$

(30)

Different from the On-PIAM algorithm, the Off-PIAM algorithm focuses on training the original policy ${\pi }_{\theta }$. After the training, the agent can work well even if we remove the masking. That is, the agent trained by the Off-PIAM algorithm explicitly learns to avoid invalid actions.

4.3. Composite Objective Algorithm

The On-PIAM algorithm aims to improve performance of the masked policy, and the Off-PIAM algorithm focuses on optimizing the original policy. A question naturally arises, can we optimize two policies concurrently? The solution is to design a composite objective, making a compromise between the performance of the masked policy and the original policy. In the following, we present the Composite Objective Invalid Action Masking (CO-IAM) algorithm, which is actually an on-policy algorithm with an auxiliary optimization objective. Inspired by Theorem 1, we design the composite optimization objective

$$\begin{array}{c}\hfill L_{\mathrm{CO}}\left(\theta \right)=J\left({\pi }_{\theta }^{\varpi }\right)-c_{\varphi }\Phi \left({\pi }_{\theta }\right),\end{array}$$

(31)

where

$$\begin{array}{cc}\hfill J\left({\pi }_{\theta }^{\varpi }\right)& ={\mathbb{E}}_{s_0\sim \rho ,a_t\sim {\pi }_{\theta }^{\varpi },s_{t+1}\sim \mathcal{P}(·|s_t,a_t)}\left[\sum _{t=0}^H{\gamma }^t{r}_t\right],\hfill \end{array}$$

(32)

$$\begin{array}{cc}\hfill \Phi \left({\pi }_{\theta }\right)& ={\mathbb{E}}_{\tau \sim {\pi }_{\theta }^{\varpi }}\left[\sum _{t=0}^H\sum _{a_i\in {\mathcal{A}}_{\mathrm{inv}}\left(s_t\right)}{\pi }_{\theta }\left(a_i\right|s_t)\right].\hfill \end{array}$$

(33)

The first objective item $J\left({\pi }_{\theta }^{\varpi }\right)$ is the cumulative reward following the masked policy ${\pi }_{\theta }^{\varpi }$. The second objective item $\Phi \left({\pi }_{\theta }\right)$ is the cumulative probability sum of all invalid actions. $c_{\varphi }$ is the coefficient that balances two items.

The second objective item aims to reduce the probability of invalid actions when $c_{\varphi }>0$. In short, the composite objective aims to optimize the cumulative reward and adjust the probability of invalid actions concurrently. Next, we derive the gradient of the composite objective:

$$\begin{array}{c}\hfill {\nabla }_{\theta }{L}_{\mathrm{CO}}\left(\theta \right)={\nabla }_{\theta }J\left({\pi }_{\theta }^{\varpi }\right)-c_{\varphi }{\nabla }_{\theta }\Phi \left({\pi }_{\theta }\right).\end{array}$$

(34)

The gradient of the first objective item can be directly obtained according to the Policy Gradient Theorem [25]:

$$\begin{array}{c}\hfill {\nabla }_{\theta }J\left({\pi }_{\theta }^{\varpi }\right)={\mathbb{E}}_{\tau \sim {\pi }_{\theta }^{\varpi }}\left[\sum _{t=0}^H{\nabla }_{\theta }log{\pi }_{\theta }^{\varpi }\left(a_t|s_t\right){\widehat{A}}_t\right].\end{array}$$

(35)

The gradient of the second objective item is:

$$\begin{array}{c}\hfill c_{\varphi }{\nabla }_{\theta }\Phi \left({\pi }_{\theta }\right)={\mathbb{E}}_{\tau \sim {\pi }_{\theta }^{\varpi }}\left[\sum _{t=0}^H{c}_{\varphi }\sum _{a_i\in {\mathcal{A}}_{\mathrm{inv}}\left(s_t\right)}{\nabla }_{\theta }{\pi }_{\theta }\left(a_i\right|s_t)\right].\end{array}$$

(36)

Inspired by Theorem 1, we set $c_{\varphi }={\widehat{A}}_t/\mathcal{I}(s_t;\theta )$ and obtain that:

$$\begin{array}{c}\hfill c_{\varphi }{\nabla }_{\theta }\Phi \left({\pi }_{\theta }\right)={\mathbb{E}}_{\tau \sim {\pi }_{\theta }^{\varpi }}\left[\sum _{t=0}^H\frac{{\widehat{A}}_t}{\mathcal{I}(s_t;\theta )}\sum _{a_i\in {\mathcal{A}}_{\mathrm{inv}}\left(s_t\right)}{\nabla }_{\theta }{\pi }_{\theta }\left(a_i\right|s_t)\right].\end{array}$$

(37)

Substituting (35) and (37) into (34), one has:

$$\begin{array}{cc}\hfill {\nabla }_{\theta }{L}_{\mathrm{CO}}\left(\theta \right)=& {\mathbb{E}}_{\tau \sim {\pi }_{\theta }^{\varpi }}\left[\sum _{t=0}^H{\nabla }_{\theta }log{\pi }_{\theta }^{\varpi }\left(a_t|s_t\right){\widehat{A}}_t\right]\hfill \\ \hfill & -{\mathbb{E}}_{\tau \sim {\pi }_{\theta }^{\varpi }}\left[\sum _{t=0}^H\frac{{\widehat{A}}_t}{\mathcal{I}(s_t;\theta )}\sum _{a_i\in {\mathcal{A}}_{\mathrm{inv}}\left(s_t\right)}{\nabla }_{\theta }{\pi }_{\theta }\left(a_i\right|s_t)\right]\hfill \\ \hfill =& g_{\mathrm{mask}}-{\mathbb{E}}_{\tau \sim {\pi }_{\theta }^{\varpi }}\left[\sum _{t=0}^H\frac{{\widehat{A}}_t}{\mathcal{I}(s_t;\theta )}\sum _{a_i\in {\mathcal{A}}_{\mathrm{inv}}\left(s_t\right)}{\nabla }_{\theta }{\pi }_{\theta }\left(a_i\right|s_t)\right].\hfill \end{array}$$

(38)

According to Theorem 1, we have:

$$\begin{array}{c}\hfill g_{\mathrm{mask}}-g_{\mathrm{naive}}={\mathbb{E}}_{\tau \sim {\pi }_{\theta }^{\varpi }}\left[\sum _{t=0}^H\frac{{\widehat{A}}_t}{\mathcal{I}(s_t;\theta )}\sum _{a_i\in {\mathcal{A}}_{\mathrm{inv}}\left(s_t\right)}{\nabla }_{\theta }{\pi }_{\theta }\left(a_i\right|s_t)\right]\end{array}$$

(39)

In light of (16), (38) and (39), we obtain an interesting finding that the gradient of the composite objective is equal to the naive policy gradient:

$$\begin{array}{c}\hfill {\nabla }_{\theta }{L}_{\mathrm{CO}}\left(\theta \right)=g_{\mathrm{naive}}={\mathbb{E}}_{\tau \sim {\pi }_{\theta }^{\varpi }}\left[\sum _{t=0}^H{\nabla }_{\theta }log{\pi }_{\theta }\left(a_t|s_t\right){\widehat{A}}_t\right].\end{array}$$

(40)

In Section 3.3, we present that the naive policy gradient is intuitively improper because it is an on-policy algorithm, but the behavior policy is not consistent with the target policy. However, based on the above derivation, we obtain that the naive policy gradient is indeed a valid gradient because it is equivalent to the composite objective invalid action masking (CO-IAM) algorithm, which optimizes the masked policy and the original policy concurrently. Although this is a counter-intuitive conclusion, it ensures that we can directly use the naive policy gradient (16) for training without causing instability.

Based on the above discussion, we suggest directly replacing the CO-IAM algorithm with the naive policy gradient in practice. That is, we only need to use the masked policy for sampling without changing the structure of the policy gradient algorithm. Similarly, we can employ PPO as the training algorithm:

$$\begin{array}{cc}\hfill L_{\mathrm{CO}}^{CLIP}\left(\theta \right)& ={\mathbb{E}}_{\tau \sim {\pi }_{\theta }^{\varpi }}\left[min\left(r_t^{\mathrm{CO}}\left(\theta \right){\widehat{A}}_t,clip\left(r_t^{\mathrm{CO}}\left(\theta \right),ϵ\right){\widehat{A}}_t\right)\right],\hfill \end{array}$$

(41)

$$\begin{array}{cc}\hfill \mathrm{where}{r}_t^{\mathrm{CO}}\left(\theta \right)& =\frac{{\pi }_{\theta }\left(a_t|s_t\right)}{{\pi }_{{\theta }_{\mathrm{old}}}\left(a_t|s_t\right)}.\hfill \end{array}$$

(42)

By adding the entropy item, the optimization objective is:

$$\begin{array}{c}\hfill L_{\mathrm{CO}}^{PPO}\left(\theta \right)=L_{\mathrm{CO}}^{CLIP}\left(\theta \right)+c_e\mathcal{H}\left({\pi }_{\theta }^{\varpi }\right).\end{array}$$

(43)

The value functions of all algorithms are updated by the squared error loss [27], which is not listed in the optimization objectives.

5. Empirical Analysis

We evaluate how well the proposed algorithms deal with invalid actions using the Gym-$\mu$RTS environment [24]. It is a perfect testbed for our experiments because the space of invalid actions increases exponentially with the map size. We take four maps with different sizes for experiments (see Figure 3), and the environment setup is the same as Huang et al. [23].

In the Gym-$\mu$RTS environments, an agent attempts to control the unit at the top left of the map to harvest resources as fast as possible. The agent receives a reward $+1$ when the unit harvests a resource or returns a resource to the base. The action space is designed as the MultiDiscrete action space [19]. A composite action comprises eight discrete actions, Source Unit, Action Type, Move Parameter, Harvest Parameter, Return Parameter, Produce Direction Parameter, Produce Type Parameter, and Attack Target Unit. The first component is the location of the selected unit to perform an action. Since only the worker and base can be selected as Source Unit, selecting other grids on the map are usually invalid. This action space enables us to verify the scalability of invalid action masking.

We will compare On-PIAM, Off-PIAM, and CO-IAM algorithms on different maps. We further evaluate the performance of original policies with Masking Removed (i.e., On-PIAM-MR, Off-PIAM-MR, and CO-IAM-MR). To illustrate the superiority of invalid action masking, we also compare it with the invalid action penalty, which assigns a negative reward $r_{\mathrm{inv}}=\{-0.01,-0.1\}$ for choosing an invalid action (called Penalty-0.01 and Penalty-0.1). All the agents are trained for 1,000,000 time steps, and the maximal length of each episode is 200 iteration steps. All experiments were run with four random seeds.

5.1. Evaluation Metrics

Drawing from Huang et al. [23], we employ six metrics to measure our algorithms’ performance. Specifically, $r_{\mathrm{episode}}$ is the episode return. $a_{\mathrm{null}}$ is the number of actions that select an invalid Source Unit. $a_{\mathrm{busy}}$ is the number of actions that select a Source Unit that is executing other actions. $a_{\mathrm{owner}}$ is the number of actions that select a Source Unit that does not belong to the agent. $t_{\mathrm{first}}$ is the percentage of the total time steps that it takes for the agent to obtain the first positive reward. $t_{\mathrm{solve}}$ is the percentage of the total time steps that it takes for the episode return to exceed 40.

5.2. Results

The experimental results are presented in Figure 4 and Figure 5 and

Table 2. Results averaged over four random seeds.

Algorithms	Map	${\mathit{r}}_{\mathbf{episode}}$	${\mathit{a}}_{\mathbf{null}}$	${\mathit{a}}_{\mathbf{busy}}$	${\mathit{a}}_{\mathbf{owner}}$	${\mathit{t}}_{\mathbf{solve}}$	${\mathit{t}}_{\mathbf{first}}$
CO-IAM	$4\times 4$	38.50	–	–	–	4.17%	0.03%
	$10\times 10$	39.30	–	–	–	5.46%	0.04%
	$16\times 16$	39.30	–	–	–	6.21%	0.06%
	$24\times 24$	37.20	–	–	–	10.10%	0.05%
Off-PIAM	$4\times 4$	39.98	–	–	–	4.57%	0.04%
	$10\times 10$	39.67	–	–	–	9.66%	0.02%
	$16\times 16$	39.55	–	–	–	9.16%	0.04%
	$24\times 24$	35.20	–	–	–	16.15%	0.04%
On-PIAM [23]	$4\times 4$	36.55	–	–	–	3.49%	0.04%
	$10\times 10$	35.40	–	–	–	4.50%	0.02%
	$16\times 16$	33.58	–	–	–	5.40%	0.04%
	$24\times 24$	31.85	–	–	–	7.09%	0.04%
Penalty-0.01	$4\times 4$	39.33	59.62	1.70	19.38	6.01%	0.07%
	$10\times 10$	32.00	108.82	0.12	4.62	13.98%	1.34%
	$16\times 16$	8.89	30.68	0.00	0.60	–	1.10%
	$24\times 24$	11.44	163.05	0.30	1.45	–	0.93%
Penalty-0.1	$4\times 4$	29.97	18.55	0.78	5.80	34.48%	0.07%
	$10\times 10$	0.00	0.00	0.00	0.00	–	3.74%
	$16\times 16$	7.97	27.72	0.28	0.75	–	1.29%
	$24\times 24$	0.64	94.70	0.22	0.88	–	0.57%
CO-IAM-MR	$4\times 4$	21.37	0.27	0.00	152.59	–	0.03%
	$10\times 10$	33.00	131.89	0.00	0.01	–	0.04%
	$16\times 16$	32.25	133.34	0.00	0.00	–	0.06%
	$24\times 24$	25.10	149.11	0.00	0.15	–	0.05%
Off-PIAM-MR	$4\times 4$	39.88	9.53	17.81	1.31	7.22%	0.04%
	$10\times 10$	39.33	9.86	4.56	1.59	34.92%	0.02%
	$16\times 16$	38.65	10.19	1.95	1.01	25.91%	0.04%
	$24\times 24$	35.23	15.90	5.97	0.18	–	0.04%
On-PIAM-MR	$4\times 4$	7.13	49.33	0.00	133.20	–	0.04%
	$10\times 10$	0.23	188.20	0.00	10.89	–	0.02%
	$16\times 16$	0.15	195.66	0.00	3.90	–	0.04%
	$24\times 24$	0.48	197.64	0.00	1.37	–	0.04%

. Here, we provide some important observations.

Invalid action penalty is not a robust way to prevent the agent from taking invalid actions. In Figure 4, we observe that Penalty-0.01 achieves more than 30 rewards in $4\times 4$ and $10\times 10$ maps. It even surpasses On-PIAM in the $4\times 4$ map but can hardly extend to other maps. Further, the performance of Penalty-0.1 is obviously worse than Penalty-0.01. However, Table 2 shows that $a_{\mathrm{null}}$, $a_{\mathrm{busy}}$, and $a_{\mathrm{owner}}$ of Penalty-0.1 is better than Penalty-0.01. It demonstrates that the larger penalty causes lower probabilities of invalid actions, but it fails to obtain more positive rewards. Thus, we have to make more efforts to tune the value of the negative reward when we adopt the invalid action penalty.

On-PIAM solves all tasks within fewer time steps than Off-PIAM and CO-IAM, but it fails to achieve expected asymptotic performance. As shown in Table 2, On-PIAM’s $t_{\mathrm{solve}}$ is roughly 3% to 7% and very similar across different map sizes. However, we can observe that the episode return of On-PIAM achieves 34 on average, but Off-PIAM and CO-IAM achieve almost 40. The green curves in Figure 4 verify that the episode return curves of On-PIAM are not stable, and its asymptotic performance is worse than Off-PIAM and CO-IAM. Furthermore, Figure 5 shows that once the masking is removed, On-PIAM-MR only sightly works in $4\times 4$ map and cannot obtain rewards on other maps. In Table 2, we also notice that On-PIAM-MR has larger probabilities of choosing $a_{\mathrm{null}}$ and $a_{\mathrm{owner}}$. This is expected since On-PIAM aims to optimize the masked policy rather than the original policy.

CO-IAM outperforms On-PIAM and scales well on complex tasks. As shown in Figure 5 and Table 2, CO-IAM is better than On-PIAM with or without masking. When the masking is removed, the episode return of CO-IAM-MR still maintains roughly 30. It verifies our idea that although $g_{\mathrm{naive}}$ seems to be an improper policy gradient, it indeed optimizes the masked policy and the original policy concurrently. CO-IAM’s $t_{\mathrm{solve}}$ is better than Off-PIAM. In general, the performance of CO-IAM shows that CO-IAM seems to be a combination of On-PIAM and Off-PIAM. In the $24\times 24$ map, CO-IAM obtains 37.2 rewards on average and outperforms Off-PIAM and On-PIAM. It demonstrates that CO-IAM has a better exploration ability to deal with complex tasks.

Off-PIAM achieves the best performance when the masking is removed. In Figure 5, and Table 2, we observe that Off-PIAM-MR obtains the better asymptotic performance than On-PIAM-MR and CO-IAM-MR. In addition, Off-PIAM also realizes the great performance during the training. Figure 4 shows that Off-PIAM surpasses other algorithms in $10\times 10$ and $16\times 16$ maps, and it achieves similar asymptotic performance to CO-IAM in other maps. It demonstrates that the masked policy of Off-PIAM can work well even if it focuses on optimizing the original policy. Moreover, the second row of Figure 4 shows the Kullback–Leibler (KL) divergence between the target policy and the current policy. The target policy of Off-PIAM is ${\pi }_{\theta }$, and the behavior policy is ${\pi }_{\theta }^{\varpi }$. We can observe that the KL divergence of Off-PIAM is large at the beginning and gradually decreases during the training. That is, ${\pi }_{\theta }$ and ${\pi }_{\theta }^{\varpi }$ become closer during the training, which illustrates that Off-PIAM guides the agent to explicitly learn to avoid invalid actions.

6. Conclusions

In this study, we compared the performance of three invalid action masking algorithms, On-PIAM, Off-PIAM, and CO-IAM. These algorithms optimize the masked policy, the original policy, and both policies, respectively. Our main contribution lies in the successful performance of the proposed Off-PIAM and CO-IAM algorithms when removing masking after training, while CO-IAM fails to work. Through our work, we have discovered several key findings. Firstly, while On-PIAM solves all tasks within fewer time steps, it fails to achieve the expected return when masking is removed. Secondly, we found that the naive policy gradient is equivalent to the CO-IAM algorithm, which optimizes the masked policy and the original policy in parallel. Thirdly, CO-IAM demonstrates superior performance compared to On-PIAM and scales well on complex tasks. Lastly, Off-PIAM outperforms other algorithms, regardless of whether masking is present or not. Based on our empirical findings, we recommend utilizing the Off-PIAM algorithm for addressing most tasks, as it consistently delivers strong performance. Additionally, the CO-IAM algorithm should be employed for handling more complex tasks. One limitation of this study is its focus on the effects of invalid action masking on RL agents in RTS games. Therefore, the findings cannot be generalized to all solutions, such as rule-based approaches or optimization-based approaches. As part of our future work, we would like to extend our experiments to include more complex game environments, such as StarCraft II, to assess the effectiveness and adaptability of our method in those settings.