Approaches That Use Domain-Specific Expertise: Behavioral-Cloning-Based Advantage Actor-Critic in Basketball Games

Abstract

Research on the application of artificial intelligence (AI) in games has recently gained momentum. Most commercial games still use AI based on a finite state machine (FSM) due to complexity and cost considerations. However, FSM-based AI decreases user satisfaction given that it performs the same patterns of consecutive actions in the same situations. This necessitates a new AI approach that applies domain-specific expertise to existing reinforcement learning algorithms. We propose a behavioral-cloning-based advantage actor-critic (A2C) that improves learning performance by applying a behavioral cloning algorithm to an A2C algorithm in basketball games. The state normalization, reward function, and episode classification approaches are used with the behavioral-cloning-based A2C. The results of the comparative experiments with the traditional A2C algorithms validated the proposed method. Our proposed method using existing approaches solved the difficulty of learning in basketball games.

1. Introduction

Recently, research on the application of artificial intelligence (AI) in games has been on the increase. Most commercial games still use AI based on a finite state machine (FSM) due to complexity and cost considerations. However, FSM-based AI decreases user satisfaction given that it performs the same patterns of consecutive actions in the same situations. Various studies have applied reinforcement learning to AI to replace a non-player character or a player [1]. Since reinforcement learning can be applied to model-free environments, it is suitable for complicated game environments. For example, a study applied a deep Q-network (DQN) [2] to Atari games [3]. The Atari game environment is effectively used to verify reinforcement learning since it provides interfaces to experiment with various games [3]. DQN succeeded in human-level AI learning by applying deep neural networks to Q-learning [4]. In addition, DQN solved the problem of complexity through a convolutional neural network (CNN).

Three heuristic pre-processes are required to successfully apply reinforcement learning in a basketball game. The first pre-process involves normalizing the state representation for learning in a complex environment of a basketball game. Since the raw data generated in the basketball game environment increases the amount of learning, it is necessary to sort selected raw data and process them to be easily understood. For example, it is necessary to reduce the state space of reinforcement learning by representation using relative values. Learning can be stabilized by normalizing raw data in an unstable basketball game environment [5]. The second pre-process is designing the reward function that evaluates the actions of the AI. Defining the reward function to be applied to reinforcement learning in a basketball game may be difficult [6,7,8]. Accordingly, it is necessary to consider the correlation among various rewards generated in this complex environment during the design process. The third pre-process is classifying various episodes for learning. Learning various episodes at once increases learning complexity [7]. To classify episodes per situation, it is necessary to define and classify representative situations as offense, defense, and loose ball to reduce the complexity.

Additionally, reinforcement learning for basketball games requires an approach to improve its performance limitation given that reinforcement learning starts learning with a uniform policy and faces lots of trials and errors at the beginning of learning. Such trials and errors reduce learning performance [9]. Thus, a new AI approach that applies domain-specific expertise to existing reinforcement learning algorithms is needed.

We propose a behavioral-cloning-based advantage actor-critic (A2C), which improves learning performance by applying a behavioral cloning algorithm [10] to an A2C algorithm [11,12], a reinforcement learning algorithm, in a basketball game. We introduce three pre-processing methods, namely, state representation, reward function, and episode classification, that can be used to apply the behavioral-cloning-based A2C to a basketball game. The present study aims to solve the issues related to learning time and state space generated during reinforcement learning in complex basketball games.

This paper is configured as follows. Section 2 discusses studies related to the state representation, reward function, episode classification, and behavioral cloning algorithm. Section 3 explains the behavioral-cloning-based A2C proposed in this paper and pre-processes applied to improve reinforcement learning. Section 4 compares the performance of FSM-based AI applied to commercial games with the AI trained by the proposed method. Section 5 describes the experimental results achieved by the proposed method and areas of further study.

2. Background

Reinforcement learning is a circular structure that exchanges states, rewards, and actions, and the control is generally handled by the agent. First, the agent requests a state and reward from the environment. Furthermore, when the state and reward come from the environment, the agent determines the action and sends it back to the environment. The environment takes action, undergoes a transition to the environment, and then regenerates the state and reward in the next step.

The DQN algorithm contributed three methods to solve the problem that reward graphs fail to converge with unstable learning when combining deep neural networks and reinforcement learning. The DQN algorithm defined an image as a state in the Atari game and enabled effective learning with a relatively small image, even in the environment requiring a high-level sensor, by applying a CNN. Furthermore, the use of CNNs made the performance better as compared to the performance when the conventional simple linear function neural networks were used. The DQN algorithm used a replay buffer using previous experiences, which solved the sample correlation problem and stabilized the distribution of the data. The DQN algorithm used a target network to solve the problem of unstable target values.

The double deep Q network (DDQN) algorithm proves that the DQN algorithm can be overoptimistic. The DDQN algorithm also shows that the overestimation of the DQN algorithm is more common by analyzing the value estimation. The DDQN algorithm solved the existing problem and proved to be more stable. The DDQN algorithm proposed an implementation method that simply modifies the loss function without changing the network or other tasks. The DDQN algorithm obtained good results in the Atari 2600 domain.

The A2C algorithm is designed to reduce variance by adding the concept of a critic in the policy gradient method. There is an actor that updates parameters to find a policy and a critic that updates parameters to estimate a value. In the process of making a policy gradient, subtracting the constant value unrelated to the action can reduce the variance without affecting the estimation. Therefore, the advantage $\mathrm{A}\left(\mathrm{s},\mathrm{a}\right)$ can be expressed as a state-action value $\mathrm{Q}\left(\mathrm{s},\mathrm{a}\right)$ and a state value $\mathrm{V}\left(\mathrm{s}\right)$. Additionally, applying the time difference (TD) method is efficient because the number of parameters can be reduced.

The self-imitation learning (SIL) algorithm proposed a learning method that imitates past good decisions. The SIL algorithm provided a theoretical justification for their objective function. The SIL algorithm was simply implemented and could be applied to the A2C algorithm. The SIL algorithm validated the good performance in several Atari games. The SIL algorithm was applied to the proximal policy optimization (PPO) algorithm and improved performance.

The DQN algorithm and DDQN algorithm estimate the value of actions and states to determine the next actions, which is called a value-based algorithm. It is difficult to apply the proposed method given that there is no explicit policy. The SIL algorithm imitates past decisions, which may not be enough to achieve an optimal policy. Since the A2C algorithm has an explicit policy, it is suitable to imitate an expert policy. There are limitations to the A2C algorithm itself that make it insufficient for application to complex basketball game environments, but the method presented in this paper can be improved upon.

3. Related Work

This section discusses existing reinforcement learning studies, including the state representation, reward function, and episode classification in reinforcement learning. We also introduce a behavioral cloning algorithm.

3.1. Reinforcement Learning

Reinforcement learning can be applied to various industries, such as games, stocks, and automobiles. However, in the beginning of the learning stage, reinforcement learning undergoes trials and errors, so a simulation environment is essential. Without such a simulation environment, researchers would lose huge amounts of stock in stocks or suffer from car accidents in automobiles. Since the game environment is advantageous for the application of techniques such as several times of speed and synchronization that are helpful for learning, many reinforcement learning studies have been conducted on the game environment.

Schaul et al. [13] introduced a priority-based sampling method in operating replay buffers proposed in a deep Q-network (DQN). Another study modified and improved the loss function of a DQN [14]. Another study modified and applied the network to make the state value not the state-action value in DQN [15]. There is a study that uses this distribution and performs well in certain Atari games [16]. Another study performed reinforcement learning in multiple environments using an asynchronous method [12]. In the policy-based reinforcement-learning algorithm, an algorithm applied clipping using epsilon [17]. Other studies improved an A2C algorithm using a higher TD error in the replay buffer [18]. A framework study used an Atari environment for reinforcement learning [19]. Another study defined a state and reward for applying reinforcement learning to StarCraft 2 and applied reinforcement learning to a mini-game using the reinforcement learning used in the Atari game [20]. Other studies implemented the environment for applying multi-agent reinforcement learning to StarCraft 2; they defined the state, reward, and action and experimented using various multi-agent reinforcement learning algorithms [21]. Shao et al. [6] enhanced the performance of an AI using curriculum transfer learning in the StarCraft micromanagement environment. Kurach et al. [22] developed a soccer game engine for reinforcement learning, experimented with reinforcement learning algorithms, and finally, provided an experimental environment, including a soccer game benchmark and a soccer game academy. Other studies used multi-agent reinforcement learning, position learning, episode-classification learning, and curriculum learning in basketball games [7]. Liu et al. [23] introduced a deep reinforcement learning (DRL) approach for a vehicle dispatching problem by designing the corresponding simulator and a specific vehicle dispatching algorithm. The DRL approach simplifies the problem of the requirement for a large number of agents.

3.2. State Representation, Reward Function, and Episode Classification

State refers to environmental data given to an agent at a specific time. An agent determines the course of action based on the state. The performance of a reinforcement learning-based agent depends on the state definition [24]. Mnih et al. [2] defined an image as a state in the Atari game. When it is difficult to define an image as a state, the state is defined using values. Shao et al. [6] adopted a normalized relative coordinate value based on the agent to be learned. The merits of a normalized state are the reduction of risks of falling into local minima and making optimization by an optimizer faster [5].

Reward refers to the value assigned to the action of an agent at a specific time. Since the action policy depends on the methods defining the reward function, it is important to define the reward function [8]. Ng et al. [8] verified the change of the action policy of an agent depending on the reward. Shao et al. [6] demonstrated that the winning rate of an agent depended on reward in a StarCraft micromanagement environment. Even with the same algorithm, the learning performance converged faster with a denser reward.

Jia et al. [7] introduced a learning method that classified episodes per situation when it was difficult to learn for the entire episode and experimentally verified the performance improvement.

3.3. Behavioral-Cloning Algorithm

A behavioral cloning algorithm [10] is a representative imitation learning algorithm [25]. The behavioral cloning algorithm collects states and actions from game data collected by experts and uses them as learning data. Based on the collected data, it is possible to train an AI that acts similarly to the policy of the expert. However, behavioral cloning algorithms improve performance only in states included in the game data of experts. Ross et al. [26] improved a behavioral cloning algorithm by adjusting the ratio of the actions of an expert and other actions. Goecks et al. [9] introduced a method to simultaneously solve the problems of imitation learning and reinforcement learning.

3.4. Comparison of Proposed Method to Related Works

Jia et al. [7] performed multi-agent reinforcement learning, position learning, episode classification learning, and curriculum learning for a basketball game. The offense episode of a shooting guard in a basketball game has 42 actions. The multi-agent reinforcement learning assigned rewards to successful actions, including score, block, mark, steal, and pick. The episode-classification learning classified offense episodes into three categories: attack, assist, and ball clear.

The proposed method defines the offense episodes into 11 actions: movement in eight directions, shoot, breakthrough, and pass. Since the number of actions is reduced, the learning space is reduced, and learning performance can be improved in the same episode. The proposed method gives rewards in situations, including mark and avoid, in addition to completed actions, including shoot and breakthrough. In episode classification learning, the entire episode is classified into offense, defense, and loose ball. The offense episode is not segmented for correlation considerations.

Goecks et al. [9] explained the algorithm that adds imitation learning to reinforcement learning. Learning is performed with state, action, and reward data collected from an expert and exploration by reinforcement learning AI. The proposed method uses pre-defined policies without directly applying the state, action, and reward collected from an expert; it uses the state value, not the action value.

Table 1 summarizes the key differences and similarities between our work and the others. The contents in black illustrate the similarities, and items in red illustrate the differences.

Table 1. Summarizing key differences and similarities.

	Ours	Jia et al. [7]	Goecks et al. [9]
State	Normalized relative values based on basketball game data	Basketball game data	Various values
Reward	Values based on state and action	Values based on success of the action	Conventional values
Episode	Offense, defense, and loose ball	Attack, assist, defense, free ball, and ball clear	Single episode
Algorithm	Behavioral-cloning-based A2C	Multi-agent reinforcement learning	Reinforcement learning that combines imitation learning

4. Behavioral-Cloning-Based A2C and Application in Basketball Games

This section proposes a behavioral-cloning-based A2C and the method to apply it to a basketball game. The loss function for learning by behavioral-cloning-based A2C using the policies of an expert is explained. Furthermore, this section describes the structure used to apply behavioral-cloning-based A2C to a basketball game and the series of pre-processes, which include state representation, reward function, and episode classification.

4.1. Behavioral-Cloning-Based A2C

We propose a method to reduce the learning time of A2C using the behavioral-cloning algorithm [10] when the policy of an expert is provided. In general, the total loss function ${\mathcal{L}}^{\mathrm{a}2\mathrm{c}}={\mathbb{E}}_{\mathrm{s},\mathrm{a}~{\mathsf{\pi }}_{\mathsf{\theta }}}\left[{\mathcal{L}}^{\mathrm{policy}}+{\mathcal{L}}^{\mathrm{value}}\right]$ [12] is calculated using the sum of the policy loss function ${\mathcal{L}}^{\mathrm{policy}}=-{\log \mathsf{\pi }}_{\mathsf{\theta }}\left(\mathrm{a}|\mathrm{s}\right)\left(\mathrm{R}-{\mathrm{V}}_{\mathsf{\theta }}\left(\mathrm{s}\right)\right)-\mathsf{\alpha }{\mathcal{H}}^{{\mathsf{\pi }}_{\mathsf{\theta }}}$ [12] and the value loss function ${\mathcal{L}}^{\mathrm{value}}=\frac{1}{2}{\left(\mathrm{R}-{\mathrm{V}}_{\mathsf{\theta }}\left(\mathrm{s}\right)\right)}^2$ [12]. The entropy loss function is defined as ${\mathcal{H}}^{{\mathsf{\pi }}_{\mathsf{\theta }}}=-{\sum }_{\mathrm{a}}{\mathsf{\pi }}_{\mathsf{\theta }}(\mathrm{a}|\mathrm{s}){\log \mathsf{\pi }}_{\mathsf{\theta }}(\mathrm{a}|\mathrm{s})$ [12]. The entropy loss function prevents the soft-max distribution from exceeding a certain range. We propose a policy loss function ${\mathcal{L}}^{\mathrm{policy}}$ as specified in Equation (1) below. Where $-{\log \mathsf{\pi }}_{\mathsf{\theta }}\left(\mathrm{a}|\mathrm{s}\right)\left(\mathrm{R}-{\mathrm{V}}_{\mathsf{\theta }}\left(\mathrm{s}\right)\right)-\mathsf{\alpha }{\mathcal{H}}^{{\mathsf{\pi }}_{\mathsf{\theta }}}$ is the same as the policy loss function of A2C ${\mathcal{L}}^{\mathrm{policy}}$, and $-{\mathsf{\beta }\log \mathsf{\pi }}_{\mathsf{\theta }}\left({\mathrm{a}}^{\mathrm{expert}}|\mathrm{s}\right)$ is the behavioral cloning loss function. When the action of an expert is determined as ${\mathrm{a}}^{\mathrm{expert}}~{\mathsf{\pi }}^{\mathrm{expert}}({\mathrm{a}}^{\mathrm{expert}}|\mathrm{s}$), the loss of ${\mathsf{\pi }}_{\mathsf{\theta }}\left({\mathrm{a}}^{\mathrm{expert}}|\mathrm{s}\right)$ is calculated using a log function.

$${\mathcal{L}}^{\mathrm{policy}}=-{\log \mathsf{\pi }}_{\mathsf{\theta }}\left(\mathrm{a}|\mathrm{s}\right)\left(\mathrm{R}-{\mathrm{V}}_{\mathsf{\theta }}\left(\mathrm{s}\right)\right)-\mathsf{\alpha }{\mathcal{H}}^{{\mathsf{\pi }}_{\mathsf{\theta }}}-{\mathsf{\beta }\log \mathsf{\pi }}_{\mathsf{\theta }}\left({\mathrm{a}}^{\mathrm{expert}}|\mathrm{s}\right)$$

(1)

Algorithm 1 is the pseudo-code for the proposed method. Learning is repeated until the return converges. The action ${\mathrm{a}}_{\mathrm{t}}$ is calculated at ${\mathsf{\pi }}_{\mathsf{\theta }}$ using the state ${\mathrm{s}}_{\mathrm{t}}$, and the action of the expert ${\mathrm{a}}_{\mathrm{t}}^{\mathrm{expert}}$ is calculated at ${\mathsf{\pi }}^{\mathrm{expert}}$ using the state ${\mathrm{s}}_{\mathrm{t}}$. The reward ${\mathrm{r}}_{\mathrm{t}}$ is determined based on the state ${\mathrm{s}}_{\mathrm{t}}$ and action ${\mathrm{a}}_{\mathrm{t}}$. The action ${\mathrm{a}}_{\mathrm{t}}$, action of the expert ${\mathrm{a}}_{\mathrm{t}}^{\mathrm{expert}}$, and reward ${\mathrm{r}}_{\mathrm{t}}$ are written on the update buffer Ɛ. When the episodes end, the parameter $\mathsf{\theta }$ of the actor and parameter ${\mathsf{\theta }}^{\mathrm{v}}$ of the critic are updated as explained below. The return ${\mathrm{R}}_{\mathrm{t}}$ is calculated by multiplying the reward ${\mathrm{r}}_{\mathrm{k}}$ with gamma ${\mathsf{\gamma }}^{\mathrm{k}-\mathrm{t}}$. The parameter $\mathsf{\theta }$ of the actor is updated using Equation (1). The parameter ${\mathsf{\theta }}^{\mathrm{v}}$ of the critic is updated using the value loss function ${\mathcal{L}}^{\mathrm{value}}$. When the update for one episode is completed, the update buffer is initialized.

Algorithm 1 Proposed behavioral-cloning-based A2C.

1. $\mathrm{Initialize}\mathrm{parameter}\mathsf{\theta },{\mathsf{\theta }}^{\mathrm{v}}$ 2. Initialize update buffer Ɛ ←$\varnothing$ 3. When the return does not converge, do 4.     ${\mathrm{r}}_{\mathrm{t}}$←$\mathrm{Execute}\mathrm{an}\mathrm{action}{\mathrm{a}}_{\mathrm{t}}~{\mathsf{\pi }}_{\mathsf{\theta }}$$({\mathrm{a}}_{\mathrm{t}}$$|{\mathrm{s}}_{\mathrm{t}}$) 5.    $\mathrm{Get}\mathrm{expert}\mathrm{action}{\mathrm{a}}_{\mathrm{t}}^{\mathrm{expert}}~{\mathsf{\pi }}^{\mathrm{expert}}$$({\mathrm{a}}_{\mathrm{t}}^{\mathrm{expert}}$$|{\mathrm{s}}_{\mathrm{t}}$) 6.      Store transition Ɛ←Ɛ$\cup \{{\mathrm{s}}_{\mathrm{t}}$$,{\mathrm{a}}_{\mathrm{t}},{\mathrm{a}}_{\mathrm{t}}^{\mathrm{expert}},{\mathrm{r}}_{\mathrm{t}}$} 7.      If the episode is done 8.       $\mathrm{Compute}\mathrm{return}{\mathrm{R}}_{\mathrm{t}}$$={\mathsf{\Sigma }}_{\mathrm{k}=1}^{\infty }{\mathsf{\gamma }}^{\mathrm{k}-\mathrm{t}}{\mathrm{r}}_{\mathrm{k}}$ for all t in Ɛ 9.         # Perform actor update 10.       θ←$\mathsf{\theta }-{\mathsf{\pi }}_{\mathsf{\theta }}\left(\mathrm{a}|\mathrm{s}\right)\left(\mathrm{R}-{\mathrm{V}}_{{\mathsf{\theta }}^{\mathrm{v}}}\left(\mathrm{s}\right)\right)-\mathsf{\alpha }{\mathcal{H}}^{{\mathsf{\pi }}_{\mathsf{\theta }}}-{\mathsf{\beta }\log \mathsf{\pi }}_{\mathsf{\theta }}\left({\mathrm{a}}^{\mathrm{expert}}|\mathrm{s}\right)$ 11.       # Perform critic update 12.      ${\mathsf{\theta }}^{\mathrm{v}}$←${\mathsf{\theta }}^{\mathrm{v}}+\frac{1}{2}{\left(\mathrm{R}-{\mathrm{V}}_{{\mathsf{\theta }}^{\mathrm{v}}}\left(\mathrm{s}\right)\right)}^2$ 13.       Clear update buffer Ɛ ← $\varnothing$ 14.    End If 15. End While

4.2. Application of Behavioral-Cloning-Based A2C

We applied the proposed method to the Freestyle learning simulator environment for a 3 to 3 basketball game (hereinafter referred to as “basketball game”). The learning simulator had the same game rules as the basketball game shown in Figure 1 and provided the speed control, AI command interface, and learning data. Figure 1 was extracted from the basketball game, Freestyle (Joycity Corp, Seongnam-si, Republic of Korea).

Figure 1. Freestyle basketball game screen (Freestyle, Windows PC, Joycity Corp. 2004).

The basketball court was a half-court, which was 15 m wide and 11 m long in the virtual environment. Foul regulations such as outs or double dribbles were not applied. The ball could not go out of the court. When the offensive team failed at offense within 20 s, it was considered a foul, and the offense’s turn went to the opposing team. Each team had three players. At the end of the game, the team with the higher scores won.

Figure 2 describes the behavioral-cloning-based A2C structure proposed in this paper. An agent is a basketball game character determining an action after receiving the state from the environment of the learning simulator. The actions are movement in eight directions, shoot, pass, mark, steal, rebound, and breakthrough.

Figure 2. Structure of proposed method.

The normalization module in the basketball game receives only the state and normalizes it. The state ${\mathrm{s}}_{\mathrm{t}}$ includes the location, angle, distance, conditions, time, and points in a basketball game and is expressed as shown in Table 2. Team(AI, n) means the nth player in the AI team. Enemy(AI, m) means the mth player in the opposing team. Mark(AI) means the AI’s target player.

Table 2. A2C basketball game state.

State	Equation
AI location	$\frac{-\mathrm{z}\left(\mathrm{AI}\right)}{7.5},\left(\left(\frac{\mathrm{x}\left(\mathrm{AI}\right)-1}{11}\right)-0.5\right)\times 2$
Team location	$\frac{\mathrm{z}\left(\mathrm{AI}\right)-\mathrm{z}\left(\mathrm{team}\left(\mathrm{AI},\mathrm{n}\right)\right)}{15},\frac{\mathrm{x}\left(\mathrm{team}\left(\mathrm{AI},\mathrm{n}\right)\right)-\mathrm{x}\left(\mathrm{AI}\right)}{11}$
Enemy location	$\frac{\mathrm{z}\left(\mathrm{AI}\right)-\mathrm{z}\left(\mathrm{enemy}\left(\mathrm{AI},\mathrm{m}\right)\right)}{15},\frac{\mathrm{y}\left(\mathrm{enemy}\left(\mathrm{AI},\mathrm{m}\right)\right)-\mathrm{x}\left(\mathrm{AI}\right)}{11}$
Ball location	$\frac{\mathrm{z}\left(\mathrm{AI}\right)-\mathrm{z}\left(\mathrm{ball}\right)}{15},\frac{\mathrm{x}\left(\mathrm{ball}\right)-\mathrm{x}\left(\mathrm{AI}\right)}{11}$
Ball height	$\min \left(1.0,\frac{\mathrm{y}\left(\mathrm{ball}\right)}{5}\right)$
Rim location	$\frac{\mathrm{z}\left(\mathrm{AI}\right)}{7.5},\frac{10.5-\mathrm{x}\left(\mathrm{AI}\right)}{9.5}$
Team angle	$\raisebox{1ex}{$\mathrm{arccosine}\left(\frac{\mathrm{vector}\left(\mathrm{rim},\mathrm{AI}\right)·\mathrm{vector}\left(\mathrm{team}\left(\mathrm{AI},\mathrm{m}\right),\mathrm{AI}\right)}{\left|\mathrm{vector}\left(\mathrm{rim},\mathrm{AI}\right)\right|\times \left|\mathrm{vector}\left(\mathrm{team}\left(\mathrm{AI},\mathrm{m}\right),\mathrm{AI}\right)\right|}\right)$}\!\left/ \!\raisebox{-1ex}{$\mathsf{\pi }$}\right.$
Enemy angle	$\raisebox{1ex}{$\mathrm{arccosine}\left(\frac{\mathrm{vector}\left(\mathrm{rim},\mathrm{AI}\right)·\mathrm{vector}\left(\mathrm{enemy}\left(\mathrm{AI},\mathrm{m}\right),\mathrm{AI}\right)}{\left|\mathrm{vector}\left(\mathrm{rim},\mathrm{AI}\right)\right|\times \left|\mathrm{vector}\left(\mathrm{enemy}\left(\mathrm{AI},\mathrm{m}\right),\mathrm{AI}\right)\right|}\right)$}\!\left/ \!\raisebox{-1ex}{$\mathsf{\pi }$}\right.$
Mark angle	$\raisebox{1ex}{$\mathrm{arccosine}\left(\frac{\mathrm{vector}\left(\mathrm{rim},\mathrm{mark}\left(\mathrm{AI}\right)\right)·\mathrm{vector}\left(\mathrm{AI},\mathrm{mark}\left(\mathrm{AI}\right)\right)}{\left|\mathrm{vector}\left(\mathrm{rim},\mathrm{mark}\left(\mathrm{AI}\right)\right)\right|\times |\mathrm{vector}(\mathrm{AI},\mathrm{mark}\left(\mathrm{AI}\right)|}\right)$}\!\left/ \!\raisebox{-1ex}{$\mathsf{\pi }$}\right.$
Ball distance	$\min \left(1.0,\frac{\mathrm{dist}\left(\mathrm{ball},\mathrm{AI}\right)}{10}\right)$
Rim distance	$\min \left(1.0,\frac{\mathrm{dist}\left(\mathrm{rim},\mathrm{AI}\right)}{10}\right)$
Whether ball is clear or not	$\mathrm{ball}\_\mathrm{clear}\left(\mathrm{AI}\right)$
Whether pivoting or not	$\mathrm{pivot}\left(\mathrm{AI}\right)$
Game time remaining	$\frac{\mathrm{quarter}\_\mathrm{time}}{240}$
Foul time remaining	$\frac{\mathrm{violation}\_\mathrm{time}}{20}$
Team scores	$\min \left(1.0,\frac{\mathrm{score}\left(\mathrm{AI}\right)}{30}\right)$

Location means the location of a character, ball, and rim. Each character took their location as the origin and expressed other characters, the ball, and the rim as relative locations. Figure 3 illustrates the normalization process. The blue dot is the reference character, and the red dots are other objects. The AI location uses the relative value around the center of a court. The location is expressed in 2D, but in the case of a ball, height is considered. The maximum height of a ball is 5 m, with zero (0) as the basic level.

Figure 3. AI location normalization process in a basketball game.

The angle is not state-provided in the environment but is calculated and used as a state value. It is used as the reference value to identify whether the character is open. The distance is calculated by dividing the distance between the reference object and another object by the maximum allowable distance (10 m). It uses the unit meter. The conditions are whether the AI team clears the ball (ball_clear(AI)) and whether the AI pivots (pivot(AI)). The conditions have the values 0 or 1.

There are two kinds of time in seconds, the time indicating the game time remaining until the game finishes and the foul time. These are divided into a maximum of 240 s and 20 s, respectively, and normalized to values between 0 and 1. The scores display the current scores of the AI team, and the value divided by the maximum allowable scores (30 points) was used as the state.

The reward module in the basketball game established a reward system to apply A2C. The reward ${\mathrm{r}}_{\mathrm{t}}$ was defined as a shoot, pass, breakthrough, mark, avoid, etc., as shown in Table 3, and is calculated by adding all rewards generated. The importance of the frequency of the actions breakthrough, mark, avoid, and pick is considered when the corresponding rewards are calculated.

Table 3. A2C basketball game reward.

Rewards	Equation
Shoot	$\mathrm{Open}\mathrm{status}(0.0/1.0)\times$$\mathrm{Within}\mathrm{the}\mathrm{range}(0.0/1.0)\times$ Shoot status (0.0/1.0)
Pass	$\mathrm{Pivot}\mathrm{status}(0.0/1.0)\times$ Pass status (0.0/1.0)
Breakthrough	$\mathrm{Within}\mathrm{the}\mathrm{range}(0.0/1.0)\times$$\mathrm{Breakthrough}\mathrm{status}(0.0/1.0)\times$ Weighted value (0.5)
Mark	$\mathrm{Mark}\mathrm{angle}\le$$30(0.0/1.0)\times$$\mathrm{Mark}\mathrm{distance}\le$$2(0.0/1.0)\times$ Weighted value (0.002)
Avoid	$\mathrm{Open}\mathrm{status}(0.0/1.0)\times$$\mathrm{Within}\mathrm{the}\mathrm{range}(0.0/1.0)\times$ Weighted value (0.01)
Ball clear	Ball clear status (0.0/1.0)
Pick	$\mathrm{Pick}\mathrm{status}(0.0/1.0)-$$\mathrm{Movement}\mathrm{away}\mathrm{from}\mathrm{a}\mathrm{ball}(0.0/1.0)\times$ Weighted value (0.002)
Under rim	$\mathrm{Rim}\mathrm{distance}\le$$2\mathrm{m}(0.0/1.0)\times$ Weighted value (0.001)

The episode classification module classifies the state ${\mathrm{s}}_{\mathrm{t}}$, action ${\mathrm{a}}_{\mathrm{t}}$, and reward ${\mathrm{r}}_{\mathrm{t}}$ into the offense, loose ball, and defense situation and sends them to the corresponding A2C module. The episode classification criteria depend on the ball possession status. When a team has the ball, it is an offense episode. When an enemy has the ball, it is a defense episode. When no team has the ball, it is a loose ball episode.

According to the episode classification results, the state, reward, and action used in learning depend on the offense, loose ball, and defense situation. To control the characters by the situation, A2C was applied to offense and loose ball episodes, and the proposed method was applied to defense episodes. The defense episode is a situation in which the enemy has the ball, and the AI team needs to be within ±30 degrees from the marked angle and within 2 m from the mark distance, as specified in the rewards in Table 2. The behavioral-cloning-based A2C learning designs expert policies through distance and coordinates calculations and imitates expert actions. The proposed method, applied to defense, can also be applied to offense and loose ball situations.

5. Experiment

The proposed behavioral-cloning-based A2C was experimentally evaluated. This section compares the performance of the proposed algorithm with the conventional A2C algorithm and analyzes the experimental results. Moreover, this section analyzes the experimental results from the conventional FSM-based AI match experiment and the expert-designed FSM-based AI match experiment.

The architecture used for learning consists of two networks with dense layers with 128 units as the actor and critic. A rectified linear activation function was used as an activation function. The output layer applies a soft-max function for determining the action and a linear function for estimating the state value in each network. All values were initialized with the default parameters defined in TensorFlow.

The environment used for learning was a basketball game simulator. The simulator provided an accelerated learning and multi-client learning environment and provided all information about the basketball game per each frame. The agent that communicated with the environment consisted of one or more models concurrently. Many tasks covering the entire learning process, such as determining learning algorithms and defining models, were handled here. Briefly, in Figure 2, all parts except the basketball game box were actually processed inside this agent. Therefore, our proposed method was also carried out in the agent. The data received from the simulator were normalized by each state equation. We were able to reduce the complexity of basketball game states through state normalization. At this time, the reward was also calculated by each equation. The calculated state and reward went through an episode classification process that was classified according to the ball’s ownership. In the reduced and classified state, the model had repeatedly experienced similar states. Therefore, the model could experience various states even with episode-level online learning and offline learning rather than hindered learning. If an expert policy was available, the state and expert action were stored to calculate additional loss when updating the model. This agent was also used in experiments for verification after all learning had ended. The agent loaded the saved model after the learning was completed, conducted performance experiments with the opponent through the match, and stored the results as logs.

5.1. Behavioral-Cloning-Based A2C Experiment

During the experiment, the model learned the offense episode without the policy of an expert and the defense episode reflecting the policy of an expert, and the number of rewards per episode was measured. A total of 400,000 actions performed by the AI were consecutively collected from all episodes, and the number of rewards for the actions avoid and mark were recorded after every 1000 actions.

Figure 4 shows the A2C learning experiment results, applying the proposed behavioral cloning. Figure 4a shows the number of rewards for the avoid action, and Figure 4b shows the number of rewards for the mark action.

Figure 4. Experimental results of the proposed method.

Figure 5 illustrates the experimental results of the conventional A2C learning, which did not adopt behavioral cloning. Figure 5a shows the number of rewards for the avoid action, and Figure 5b shows the number of rewards for the mark action.

Figure 5. Conventional A2C experiment results.

The reward for the avoid action in the offense episode without the policy of the expert is similar in both of the above methods. For the defense episode with behavioral cloning and the policy of the expert, the average number of rewards in the final 100 data counts was 224 that of the conventional A2C experiment, and that for the proposed method was 428, showing its marked success, achieving values approximately 1.9 times higher than the conventional A2C.

The experimental results of the proposed method and the conventional A2C were compared using the linear trend curve with zero as the reference point because of the unstable number of avoid and mark actions. The red line in Figure 4b is the trend curve showing the number of rewards for the mark action in the experiment with the proposed method. The red line in Figure 5b is the trend curve showing the number of rewards for the mark action in the experiment with the conventional A2C. The slope of the proposed method is approximately 1.47, compared with 0.68 for the conventional A2C method. The slope of the proposed method is approximately 2.1 times higher than that of the conventional A2C. This implies that the proposed method selected and performed the actions, generating relatively more rewards than the conventional A2C.

5.2. Conventional FSM-Based AI Match Experimental Results and Analysis

Figure 6 presents the learning experiment results of the match with conventional FSM-based AI. Figure 6a shows the scores of the proposed method, and Figure 6b shows the scores of the conventional FSM-based AI. Figure 6c displays the accumulated number of wins, draws, and losses. These are the results of accumulating the number of wins (+1), draws (+0), and losses (−1) for the behavioral-cloning-based A2C AI from the match results. Figure 6d shows the total rewards per match acquired by AI in the proposed method of learning.

Figure 6. Experimental results for the conventional FSM-based AI match.

The proposed method shows relatively higher scores; the total average score difference is about 0.43. The average gap of scores in the matches from the first to the 1000th match with the FSM-based AI had a higher winning rate of approximately 0.4, which is higher than that of the proposed method. The proposed method achieved a higher score of approximately 0.83 in the average gap of scores in the final 1000 matches, in which the AI in the proposed method showed a higher winning rate. For accumulated wins/draws/losses, the proposed method had a relatively higher number of losses in the early stage of learning. As learning continued, the AI in the proposed method gradually reduced losses and increased the winning rate. The total rewards also converged into a constant value after gradually increasing from lower rewards at the beginning. For the final 1000 matches, the proposed method achieved a winning rate of 60%, with 512 wins, 146 draws, and 342 losses.

5.3. Expert-Designed FSM-Based AI Match Experimental Results and Analysis

The experimental match was between the AI of the proposed method and the expert-designed FSM-based AI. Figure 7 shows the match’s experimental results. A total of 103 experimental matches were played. The proposed method achieved a winning rate of 66%, with 68 wins and 35 losses.

Figure 7. Accumulated wins/losses in the matches with expert-designed FSM-based AI.

The experimental results can be analyzed using the number of successful actions by a character as well as wins/losses or scores. Table 4 illustrates the average number of successful actions per match for each AI. The AI in the proposed method achieved relatively higher numbers in offense episodes (2 points) and defense episodes (block and steal) than the expert-designed FSM-based AI. The expert-designed FSM-based AI showed a higher number of rebounds than the AI in the proposed method. The AI in the proposed method can be improved through learning in loose ball episodes, including the timing for the rebounds and preoccupying positions.

Table 4. Average number of successful actions per match with expert-designed FSM-based AI.

Team	2 Points	3 Points	Rebound	Block	Steal
FSM AI	5.3	0.06	7.2	0.68	0.38
Proposed AI	6.76	0	2.4	5.3	2.33

6. Conclusions

This paper proposed a behavioral-cloning-based A2C performing actions that reflect the policy of an expert by applying a behavioral-cloning algorithm to A2C in a basketball game. The state was normalized to apply the proposed method to a complicated basketball game, and the reward function was proposed based on the state generated from matches. Learning was performed by classifying various episodes to solve the learning time challenge of reinforcement learning. The proposed method improved performance by approximately twice the performance of the conventional A2C method.

The winning rate of the proposed method was approximately 60% in the matches against the conventional FSM-based AI and approximately 66% in the matches against the expert-designed FSM-based AI. The proposed method showed a relatively higher number of successes in 2 points, steal, and block than the expert-designed FSM-based AI. However, the proposed method demonstrated a relatively lower number of successes in 3 points and rebound actions.

In future research, we plan to investigate the optimal basketball game learning algorithm by enhancing the proposed A2C algorithm. A human-level AI will be developed by analyzing the data collected from players and using it to improve the AI. Through such efforts, future research can develop a method that makes players more interested in the game.