Improving Agent Decision Payoffs via a New Framework of Opponent Modeling

Abstract

The payoff of an agent depends on both the environment and the actions of other agents. Thus, the ability to model and predict the strategies and behaviors of other agents in an interactive decision-making scenario is one of the core functionalities in intelligent systems. State-of-the-art methods for opponent modeling mainly use an explicit model of opponents’ actions, preferences, targets, etc., that the primary agent uses to make decisions. It is more important for an agent to increase its payoff than to accurately predict opponents’ behavior. Therefore, we propose a framework synchronizing the opponent modeling and decision making of the primary agent by incorporating opponent modeling into reinforcement learning. For interactive decisions, the payoff depends not only on the behavioral characteristics of the opponent but also the current state. However, confounding the two obscures the effects of state and action, which then cannot be accurately encoded. To this end, state evaluation is separated from action evaluation in our model. The experimental results from two game environments, a simulated soccer game and a real game called quiz bowl, show that the introduction of opponent modeling can effectively improve decision payoffs. In addition, the proposed framework for opponent modeling outperforms benchmark models.

1. Introduction

Decision intelligence, i.e., the ability to make appropriate action choices and decisions in various decision environments, is one of the core functionalities in artificial intelligent, such as intelligent transportation and warehouse robots. Studies on decision intelligence [1] have attracted increasing attention since the successful implementation of the AI Go player Alphago [2].

An important issue for agents working in a strategic setting (on either collaborative or competitive tasks) is to predict the behaviors or infer the intentions of other agents [3,4,5] because all active agents can affect the world state [6]; thus, the payoff of an agent depends on both the environment and actions of other agents. Therefore, strategic reasoning [7,8] based on the characteristics of other agents is very important for the effective decision making of an agent. Some behavioral characteristics or strategy biases are often exposed when agents participate in an interaction. An agent can exploit these characteristics by modeling other agents to make better interactive decisions. For example, an agent can make use of suboptimal agents in a multiplayer game by predicting poor moves, and a financial trading agent that knows the intentions of its opponents can reach an agreement faster [9].

Challenge. This method of modeling other agents involved in the interaction is called opponent modeling [10,11,12]. The classical approach consists of establishing an opponent model based on interaction histories [13,14]. The model can be regarded as a function that takes the interaction history as input and outputs a prediction about the opponents.

Existing opponent modeling methods [12,15] focus mainly on modeling the actions, types, preferences, and beliefs of opponents. In action modeling [16,17,18], an opponent’s future actions are predicted by explicitly reconstructing the opponent’s decision-making method, such as the action frequency. In 1995, ref. [19] proposed modeling the strategies of agents as finite automata for multiagent system modeling. Opponent preference modeling [20,21,22] is used to predict and analyze an opponent’s strategy by considering the impetus for the opponent’s strategy, including the opponent’s intention, payoff function, target plan, etc. The authors of [23] proposed a regularized opponent model with a maximum entropy objective. Belief modeling simulates the reasoning process of other agents to predict their strategies. For example, ref. [24] proposed a level-k thinking scheme for adversarial reinforcement learning. Reference [25] developed a recursive rule that allows for explicit updates of the belief over the opponent’s hidden types, which led to a substantial enhancement in the quality of decision making. In opponent-type modeling [26,27], classification is used to predict the opponent’s behavioral style. For example, ref. [28] modeled the opponent type for real-time strategy games based on the use of a hierarchical structure.

In an adaptive system [29], an agent is constantly learning and making dynamic adaptive adjustments. An opponent model should be constructed to enable adaptive changes to be made based on the opponent’s real-time strategy. Methods have been proposed that realize dynamic opponent modeling to varying degrees. For example, the authors of [30] required periodic selection in a variety of fixed strategies. However, the aim of these methods is to establish an explicit opponent model, that is, a model (such as a decision tree, neural network, or Bayesian model) is constructed to directly predict the parameters of opponents, such as actions or domain-specific strategies.

Although considerable progress has been made in this research direction, opponent prediction is independent of the decision-making process in these models, which makes it difficult to coordinate the processes of modeling and planning. Moreover, it is more important for an agent to increase its payoff than for opponent characteristics to be accurately predicted. To consider dynamic opponent strategies and build an agent that can dynamically make decisions to increase payoffs based on opponent behavior, opponent prediction should be combined with the decision-making process of the agent.

Contribution. Therefore, we propose a framework synchronizing opponent modeling and agent decision making by designing two Q-network models that exhibit strong generalization capabilities in reinforcement learning. Note that for interactive decisions, the payoff depends on not only the behavioral characteristics of the opponent but also the current state. Thus, it is necessary to encode the state and action at the same time. Confounding the state and action obscures the effects of state and action, which then cannot be accurately encoded, which is a common issue found in most explicit modeling approaches. For example, the inaccuracy of explicit opponent modeling employed by [31] leads to interference in the decision-making process of the agents. To this end, in our designed neural network, state evaluation is first separated from action evaluation, and then, the state and action values are integrated, which mitigates the issue of error accumulation. As a specific opponent model is not required, the proposed framework can be generalized and adapted to various interactive decision-making scenarios. We also propose a training algorithm for such opponent networks. By introducing two separate networks, one for action selection and another for estimating the Q-values associated with those actions, we reduce the problem of overestimation in Q-value estimation.

Related literature. This study was inspired by the recent success of applying opponent modeling to deep reinforcement learning [32,33,34,35], using a neural-network-based model, called a deep reinforced opponent network (DRON). DRON is a modification of DQN [36], where one network is used to evaluate Q-values and another network is responsible for modeling the opponent. The opponent’s actions are used as inputs to capture the current opponent’s characteristics, which are then utilized for learning the opponent’s strategy. Both the model and the training algorithm that we use in this study are different from DRON, although the target is similar. In the proposed model, the analysis of the current state is the premise of predicting the opponents’ actions. Therefore, within the proposed framework, the network structure contains a separate module for evaluating the decision state. Within our approach, an appropriate learning algorithm for the opponent network is used to objectively evaluate the actions of opponents, which critically affects the quality of decisions when dynamic opponents are involved.

Experiments were conducted demonstrating that the introduction of opponent modeling can effectively improve decision payoffs, and the proposed opponent model outperformed the DRON and deep Q-network (DQN) benchmark models.

Paper structure. The rest of this article is structured as follows. The fundamentals of deep reinforcement learning without opponent modeling are introduced in Section 2. Our opponent model, including the network structure and training algorithm, is introduced in Section 3 and Section 4. The experimental results are comparatively analyzed in Section 5. In Section 6, the findings of the study are summarized, and future research directions are identified.

2. Preliminaries

2.1. Reinforcement Learning

In reinforcement learning [37], an agent observes its environment, takes actions, receives rewards, and thereby learns to act in a manner that maximizes rewards [34,38].

A reinforcement learning task is typically modeled as a Markov decision process (MDP) [38]. An MDP is a tuple $\langle S,A,P,R,\gamma \rangle$, where S is a finite set of possible states, A is a collection of actions, $P(s,a,s^{\prime })$ is the probability of the environment transitioning from state s to state $s^{\prime }$ after the agent takes action a, $R:S\times A$→$\mathbb{R}$ is the reward function indicating the reward obtained when the agent takes action a in state s, and $\gamma$ is the discount factor.

A policy is defined as the probability distribution $\pi \left(a\right|s)$ for performing an action a in state s. The action-value function $q_{\pi }(s,a)$ represents the expected return from adopting the policy $\pi$. The optimal policy ${\pi }^{*}$ is defined by the Bellman equation [39], shown as Formula (1):

$$q^{*}(s,a)=\sum _{s^{\prime }\in S}P(s,a,s^{\prime })(R(s,a,s^{\prime })+\gamma \underset{a^{\prime }\in A}{max}{q}^{*}(s^{\prime },a^{\prime }))$$

(1)

2.2. Q-Learning

The temporal difference (TD) algorithm is a model-free reinforcement learning algorithm. Q-learning is a typical TD algorithm that evaluates the results of an action performed in the current state $s_t$ and the reward $R_t$ while observing the future state $s_{t+1}$. Based on the evaluation of all behaviors in all states, the overall optimal behavior is learned by judging the long-term discounted returns. The iterative formula [40] of the action-state value function is shown as Formula (2):

$$Q(s_t,a_t)\leftarrow Q(s_t,a_t)+\alpha [U_t-Q(s_t,a_t)]$$

(2)

In Formula (2), $U_t=R_t+\gamma \underset{a\in A}{max}Q(s_{t+1},a)$, $\alpha$ is the learning rate and $\gamma$ is the discount factor.

2.3. DQN

For complex problems with continuous states, it is difficult to express the action-value function as a lookup table; thus, a continuous approximation is required. The DQN algorithm, a modern variation on Q-learning in which a deep neural network is used to approximate the Q-function, was proposed in 2015 [36].

The basic DQN algorithm is described below.

(1) A deep neural network is used to approximate the action-value function using a parameter $\omega$ given by $Q(s,a,\omega )\approx q_{\pi }(s,a).$

(2) As shown in Formula (3), the mean square error (MSE) is used as the loss function in the approximate network for the action-value function. Two unrelated Q-networks (Q_network_Local and Q_network_Target) are used to calculate the predicted value (weight $\omega$) and the target value (weight $\omega$’). After several steps, the target network is frozen, and the weight of the actual Q-network is copied to the target network.

$$L\left(\omega \right)=\mathbb{E}\left[{\left(r+\gamma \underset{a^{\prime }}{max}Q\left(s^{\prime },a^{\prime };{\omega }^{\prime }\right)-Q\left(s,a;\omega \right)\right)}^2\right]$$

(3)

In this formula, $s^{\prime }$ and $a^{\prime }$ represent the subsequent state and action, respectively.

(3) As shown in Formula (4), the gradient is calculated to update $\omega$:

$$\frac{\partial L\left(\omega \right)}{\partial \omega }=E[(r+\gamma \underset{a^{\prime }}{max}Q(s^{\prime },a^{\prime },{\omega }^{\prime })-Q(s,a,\omega )\frac{\partial Q(s,a,\omega )}{\partial \omega }]$$

(4)

Repeating the above steps, DQN continuously improves the discrepancy between its predicted Q-values and the actual target Q-values, while gradually adjusting the parameters of the neural network to approximate the optimal Q-function. In summary, DQN has made significant contributions to the advancement of reinforcement learning by combining deep learning techniques with the Q-learning framework, enabling agents to learn approximate optimal policies in complex environments [41,42,43].

3. Opponent Modeling Based on Deep Reinforcement Learning

In decision-making scenarios, an agent participating in an interaction uses a strategy to improve its payoffs, which is more important than modeling the opponent accurately. Therefore, unlike traditional decision making based on an explicit opponent model, we propose an implicit model that coordinates opponent modeling and action-planning by integrating opponent modeling into the reinforcement learning process.

3.1. Decision Process with Opponent Modeling

In a multiagent decision-making setting, the actions of all agents affect the environment. From an agent’s point of view, the result of an operation does not remain stable in a given state but depends on the operations of other agents.

We use a to represent the actions of agents we control (primary agents) and o to represent the joint actions of all other agents (auxiliary agents). Then, a joint action $(a,o)$ by all agents is composed of the actions of the primary agent and opponents. The state transition probability becomes $P\left(s^{\prime }\right|s,a,o)$= $Pr(S_{t+1}=s^{\prime }|S_t$=s, $A_t$=$(a,o))$, which means the probability of the environment state transitions from s to $s^{\prime }$ after primary agents and auxiliary agents perform a joint action $(a,o)$. The reward function becomes $R(s,a,o)$= $E\left[R_{t+1}\right|S_t$= s, $A_t$=$(a,o)]$, in which E represents the mathematical expectation. $R(s,a,o)$ outputs the reward when the environment state is s and primary agents and auxiliary agents perform a joint action $(a,o)$.

In the multiagent interaction environment, the goal of the primary agent is to learn an optimal strategy in response to the joint strategy ${\pi }^o$ of the opponents. The opponents can be regarded as part of the environment, and the optimal action-value function relative to the joint strategy of the opponents can be redefined as $q_{*|{\pi }^o}$=$\underset{\pi }{max}{q}_{\pi |{\pi }^o}(s,a)$, where $s\in S$ and $a\in A$. The recursive expression of the action-value function becomes Formula (5):

$$\begin{array}{cc}\hfill q_{\pi |{\pi }^o}(s_t,a_t)& \hfill =\sum _{o_t}{\pi }_t^o\left(o_t\right|s_t)\sum _{s_{t+1}}P(s_t,a_t,o_t,s_{t+1})[R(s_t,a_t,o_t)+\gamma E_{a_{t+1}}\left[q_{\pi |{\pi }^o}(s_{t+1},a_{t+1})\right]]\hfill \end{array}$$

(5)

In Formula (5), the first summation sums over the possible values of the opponent’s behavior $o_t$, weighted by the opponent policy ${\pi }_t^o\left(o_t\right|s_t)$ to consider the probability of opponent actions; the second summation sums over the possible values of the next state $s_{t+1}$, weighted by the transition probability $P(s_t,a_t,o_t,s_{t+1})$ to consider the probability of transitioning from the current state through the action and opponent behavior to the next state. Within the summation, there is a reward function $R(s_t,a_t,o_t)$ that evaluates the immediate reward of taking action $a_t$ in the current state and facing the opponent taking action $o_t$. Additionally, the equation includes an expectation operation $E_{a_{t+1}}$, which represents the expectation over the action $a_{t+1}$ in the next time step. This expectation is taken with respect to the state-action value function $q_{\pi |{\pi }^o}(s_{t+1},a_{t+1})$ in the next state $s_{t+1}$, based on the current policy $\pi$ and opponent policy ${\pi }^o$. Overall, this equation computes the state-action value function based on the current state and action, capturing the long-term cumulative rewards under the given policy and opponent policy.

3.2. Deep Reinforcement Network with Opponent Modeling

To coordinate opponent modeling and action-planning and thus optimize agent decision making, we designed two modules for the deep reinforcement network: an action-planning network and an opponent network. Moreover, we constructed two types of Q-networks, namely, DualFc2 (Dual and Fully Connected 2 Network) and DualMoe (Dual and Mixture of Experts Network), which are derived through distinct methods of integrating the two modules mentioned above.

Figure 1 presents the structure of the DualFc2 network, which is composed of the action-planning network and the opponent network through a full-connection layer. The state information $F_s$ is input to the input layer $I_s$ in the action-planning network, and the opponent information $F_o$ is input to the input layer $I_o$ in the opponent network. The vectors from the two input layers $I_s$ and $I_o$ are integrated in the first full-connection hidden layer $H_{so}$ and then processed by the second full-connection hidden layer $H_{va}$.

Note that in interactive decision making, the agent’s payoff depends on both the relevant characteristics of the opponent’s behavior and the current state. Therefore, it is necessary to encode the state and action simultaneously. Confounding the state and action models in the network obscures the effects of state and action, which then cannot be modeled accurately. To this end, in our novel scheme, state evaluation is first separated from action evaluation, and the state and action values are then integrated. That is, $H_{va}$ outputs the state value $V(s,o)$ and the action-advantage value $A(s,o,a)$, respectively. These two values are integrated using the VPA operation [44] defined in Formula (6):

$$Q(s,o,a)=V(s,o)+(A(s,o,a)-\frac{1}{\mid \mathcal{A}\mid }\sum _{a^{\prime }\in \mathcal{A}}A(s,o,a^{\prime }))$$

(6)

In the VPA operation, the state value is expanded to have the same dimension as the action-advantage value, and each component of the action-advantage value is subtracted from the mean value of all the components. Thus, when the action-advantage value is almost the same as the state value, the final Q value $Q(s,o,a)$ is more influenced by the state value network. The final Q value is obtained by adding the state and action-advantage values together.

Figure 2 shows the DualMoe network, which is composed of two parts. One part is the expert network (experts): opponent information is input, processed through the rectified linear unit (ReLU) activation function in the input layer $I_s$, and then outputted to a specific number of expert subnetworks (expert). Each expert captures one type of opponent strategy via a dual subnetwork, just like the state-value network and the action-advantage-value network in the DualFc2 network. The other part of the DualMoe network is the gater network. The gater network receives decision information and opponent information. This information is transmitted as a vector to the input layer of the gater $I_{so}$ and then enters the hidden layer $H_o$. The output is weight information representing the probability of each expert. The hidden layer uses the Softmax() activation function to complete the normalization process. The Softmax() function first exponentiates the input data and then normalizes them, ensuring that all data fall within the range of 0 to 1. It transforms the output of a neural network into a probability distribution, enabling gradient propagation and facilitating the algorithm’s effective update of the network parameters, thereby improving the performance of the model. After the hidden layer $H_o$, the gater outputs the weight vector W followed by the weighted sum of the V and A values given by the expert network according to the formulas $v(s,o)={\displaystyle \sum _{i=1}^k}{w}_i\times v_i(s,o)$ and $A(s,o,a)={\displaystyle \sum _{i=1}^k}{w}_i\times A_i(s,o,a)$, respectively. The final value Q is calculated by the same method used in the DualFc2 network.

4. Reinforcement Learning Algorithm for the Opponent Network

A reinforcement learning algorithm for the opponent network is used to learn the optimal strategy under interactive decision scenarios. DQN [36] is one of the most popular reinforcement learning algorithms available. However, learning in DQN does not involve opponent information, and the same action value is used for both action selection and evaluation. At each iteration, the maximum action value Q is selected in the target network according to the state s, shown as Formula (7). Although this greedy strategy can cause Q to quickly approach the possible optimization goal, overestimated values are likely to be selected, resulting in the imprecise estimation of actions.

$$y_t=r_{t+1}+\gamma \underset{a}{max}Q\left(s_{t+1},a;{\theta }_t^{-}\right)$$

(7)

To make stable decisions with dynamic opponents, we propose an algorithm (Algorithm 1) called DecoupleRlO (decoupled reinforcement learning for an opponent network to train the DualFc2 and DualMoe networks), inspired by the concept of DoubleDqn. However, unlike in DoubleDqn [45], the action of a player is combined with the actions of the opponents to produce an action profile, such that the opponent’s information is implicitly taken into account in the action evaluation. The selection and evaluation of actions are also decoupled; i.e., the max operation is carried out to select the optimal action, and the action value is then evaluated by the target network, shown as Formula (8). The actions of the agents are thereby objectively evaluated.

$$y_t=r_{t+1}+\gamma Q\left(s_{t+1},o,\underset{a}{argmax}Q\left(s_{t+1},o,a;{\theta }_t\right);{\theta }_t^{-}\right)$$

(8)

Algorithm 1: DecoupleRlO

1: initialize replay memory D to capacity N; 2: initialize Q network with random weights $\theta$; 3: initialize target network with weights ${\theta }^{-}=\theta$; 4: for i_episode in range(1,max_episode) do 5:     reset initial state s; 6:     for i_step in range(0,max_step_limit) do 7:         with probability $ϵ$ select a random action $a_t$ or select $a_t=argmax\left(Q(s_t,o_t,a;\theta )\right)$; 8:         carry out action $a_t$, observe reward $r_t$ and new state $s_{t+1}$; 9:         store experience $\langle s_t,o_t,a_t,r_t,s_{t+1}\rangle$ in replay memory D; 10:       sample random minibatch of transitions $\langle s_j,o_j,a_j,r_j,s_{j+1}\rangle$ from replay memory D; 11:       set $y_j$ = $r_j$ if episode terminates at step $j+1$; 12:       otherwise set:             $y_j$ = $r_j+\gamma Q(s_{j+1},o_{j+1},\underset{a}{argmax}Q(s_{j+1},o_{j+1},a;{\theta }_t);{\theta }_t^{-})$; 13:       perform a gradient decent step on $(y_j-Q{(s_j,o_j,a_j;\theta )}^2$ with respect to $\theta$; 14:       every C steps reset ${\theta }^{-}=\theta$; 15:    end for 16: end for

In Algorithm 1, the replay memory D is first initialized to store each $(s,a,r,s^{\prime })$ state transition. Subsequently, the random parameters are used to initialize the Q network. The Q network can use the DualFc2 network or the DualMoe network in this paper, and then initialize the target network $targe{t}_Q$ with the parameters of the Q network Q. The $targe{t}_Q$ is used to obtain the $y_t$ value during training, which can make the network fitting process relatively stable.

After the initialization process, the environment state s is reset. For each episode, there is a maximum step limit. During each step of training, an action a is selected using the $ϵ$-greedy algorithm. This algorithm chooses a random action with a probability of $ϵ$, and with a probability of $1-ϵ$, it selects the action a with the maximum estimated value output by the action-value network for the current state. Once the action is selected and executed, the agent receives a reward value r, while the environment transitions to the next state $s^{\prime }$. This process yields a transition $(s,a,r,s^{\prime })$. The transition is stored in the replay buffer D, and at regular intervals, mini-batches of samples are extracted from D to train the Q network and target network.

After sampling, the next step is to calculate the target Q value $y_t$. If the episode has terminated, the value of $y_t$ is simply the obtained reward $r_t$. If the episode has not terminated, the target Q value $y_t$ is calculated using the target network according to Formula (8), and the Q network is optimized by backpropagation using Formula (9) as the loss function. Additionally, every few steps, the parameters of the target network are updated to match the current Q network. Finally, the environment state is updated.

$$Loss=\frac{1}{2}{(y_t-Q(s_t,o,a;{\theta }_t))}^2$$

(9)

In this study, the DecoupleRlO algorithm was used to train the DualFc2 and DualMoe networks to produce the decision-making agents DecoupleDualfc2 and DecoupleDualmoe, respectively. These two agents are jointly produced by the opponent network structures (DualFc2 and DualMoe) and the reinforcement learning algorithm DecoupleRlO. For comparison, we built four additional agents. The basic DQN algorithm without opponent modeling was used to train the DualFc2 and DualMoe networks to obtain the DualFc2 and DualMoe agents, respectively, which were used to investigate the role of the opponent networks. The DecoupleRlO training algorithm was used to train two benchmark network structures, DRON-Fc2 and DRON-Moe (the networks proposed in [32]), to obtain the DecoupleFc2 and DecoupleMoe agents, respectively, to demonstrate the performance of the DecoupleRlO learning algorithm in training DRON-FC2 and DRON-Moe.

5. Experiment and Analysis

We evaluated the model performance for two tasks, a simulated soccer game and a real game quiz bowl. The experimental settings were the same as those used in He’s study [32]. Two players were used for both tasks, with different behaviors for each opponent. The main experimental parameters were as follows: the discount coefficient was $\gamma$ = 0.9, the optimizer was Adagrad, the learning rate was 0.0005, and the minibatch size was 64. A greedy exploration strategy $ϵ$ was used in the training process. The exploration rate was initially 0.3 and decayed linearly to 0.1 within 500,000 steps. All the models were trained for 50 epochs, and the cross entropy was used as a loss function for multitask learning.

5.1. Soccer Game

The first test scenario is a variant of a soccer game [46]. As shown in Figure 3, the game is played by two players, A and B, on a 27 × 18 grid. At the beginning of the game, players A and B are arranged in random squares in the left and right halves of the grid, and the ball is randomly assigned to one player (Figure 3 (1)). Players can choose five actions: move N, S, W, or E or stand still (Figure 3 (1)). Each player moves one square at a time (except when standing still). The goals have a size of 1 × 2 and are located at the center of the sidelines on the left and right sides of the grid. Players are not allowed to enter the gray areas around the goals. Any action that takes the player to a shadowed square or outside the boundary (Figure 3 (5) and (6)) is invalid. If two players move onto the same square, the player who was holding the ball before the move was made loses the ball to the other player (Figure 3 (4)), and the action is invalid. A player that moves the ball into the opponent’s goal scores a point (Figure 3 (2) and (3)), and the game ends. If neither player scores a point within one hundred moves, the game ends in a tie.

5.1.1. Experimental Parameters

Two modes are used for the agent in the experiment: offensive and defensive. In offensive mode, the player always prioritizes offense over defense; in defensive mode, the player only defends its own goals. The agent randomly selects between the two modes in each game to create a changing strategy.

The input state information for the neural network includes the following: the coordinate $(x_a,y_a)$ of player A, the coordinate of $(x_b,y_b)$ player B, the size of the football field $(x_{min},y_{min},x_{max},y_{max})$, player A’s goal position, player B’s goal position (the coordinates of each goal have a total of 6 dimensions) and ball possession (where 1 indicates that player A has possession of the ball, and 0 indicates that player B has possession of the ball).

The opponent features include the following: the frequency of the opponent’s moves, the opponent’s latest move sequence, the sequence of the opponent’s recent actions, and the frequency at which the opponent loses the ball.

5.1.2. Analysis of the Results

In addition to the six players described in Section 4, three players are selected as baselines: a basic DQN player and the opponent network players DRON-Fc2 and DRON-Moe. After each epoch, we use 5000 randomly generated games (the test set) to evaluate the strategy and calculate the average reward.

First, we compare the performance of the -Fc2 agents (i.e., DecoupleDualFc2, DecoupleFc2, and DualFc2). The games include 200,000 total rounds, with 50 epochs and 4000 times for each epoch. Figure 4 shows the trend in the average rewards for the 5000th randomly generated game (the test set) after each epoch. In Figure 4a–c, the black dotted line represents the DQN model results, the red dotted line represents the DRON-Fc2 model results, and the blue curve represents the results of DecoupleFc2, DualFc2, and DecoupleDualFc2, respectively. It can be seen that the rewards of the three -Fc2 players are higher than those of the baseline players (i.e., DRON-Fc2 and DQN).

We record the maximum reward and the average reward for the last 10 epochs for each player.

Table 1. Comparison of rewards achieved using DecoupleDualFc2 in soccer.

Player	Maximum Reward	Average Reward
DQN	0.6658	0.5479
DRON-Fc2	0.6826	0.6439
DualFc2	0.7154	0.6723
DecoupleFc2	0.7242	0.6472
DecoupleDualFc2	0.7116	0.6590

shows that the maximum and average rewards of the -Fc2 agents exceed those of DRON-Fc2 by different quantities and are significantly higher than those of DQN. Among the three -Fc2 agents, DualFc2 achieves the highest average reward, DecoupleFc2 obtains the highest maximum reward, and the reward achieved by DecoupleDualFc2 is in between those of DualFc2 and DecoupleFc2.

The -Moe agents (i.e., DecoupleDualMoe, DecoupleMoe, and DualMoe) are trained using the same parameters as those used to train the -Fc2 agents. Figure 5a–c compare the reward curves for the -Moe players against those of the DQN and DRON-Moe players, respectively. The black dotted line represents the DQN player, the red dotted line represents the DRON-Moe player, and the blue curve represents the -Moe players during training. The rewards for players with opponent modeling are significantly higher than those for the DQN player, and the -Moe players achieve clearly higher rewards than the DRON-Moe player.

The maximum reward and average reward for the last 10 epochs in the soccer game for the -Moe players are similarly recorded and are listed in

Table 2. Comparison of rewards for DecoupleDualMoe in soccer.

Player	Maximum Reward	Average Reward
DQN	0.6658	0.5479
DRON-Moe	0.6764	0.6286
DualMoe	0.7022	0.6206
DecoupleMoe	0.7056	0.6758
DecoupleDualMoe	0.722	0.6949

. The maximum rewards for the players built in this study are significantly higher than those for the baseline DRON-Moe and DQN players, and the highest reward is obtained using the DecoupleDualMoe model. In terms of the average reward for the last 10 epochs, the performance of the DualMoe and DRON-Moe players is basically the same, and the other two -Moe players significantly outperform the DRON-Moe player. The DecoupleDualMoe model exhibits the best performance in terms of both average and maximum rewards.

These results indicate that the incorporation of opponent modeling optimizes agents’ strategies and that the proposed opponent modeling framework is more effective than the existing method.

5.2. Quiz Bowl

A quiz bowl is a simple game that is usually played between two teams. Questions are read to players, who “buzz in” (usually before finishing reading the question) to provide an answer. A player that answers a question correctly is awarded 10 points. A player that answers a question incorrectly loses 5 points, and another player can buzz in to answer. If no player answers the questions correctly, no points are awarded.

5.2.1. Experimental Setting

Two models are required to train a player for the quiz bowl game: a content model that predicts the answer to a given (partial) question and a buzzing model that determines the time at which to buzz in.

In the content model, a recurrent neural network with a gated recurrent unit (GRU) is adopted for text classification. This network can read questions sequentially and output a probability distribution vector of the answers.

The buzzing model is trained based on the content model. The different reinforcement learning methods presented in the previous section are used to build agents with different buzzing strategies.

The training data are taken from [32] and contain 1045 answers, 37.7 k questions, and 3610 users. These data are divided into two disjoint subsets: one is used to train the content model, and the other is used to train the buzzing model. The players are divided into two categories according to the buzzing time: impulsive players and conservative players. Impulsive players typically answer questions early but with a low accuracy. Conservative players tend to answer later with a high accuracy.

The state information input to the neural network includes the probability distribution vector output of the content model, the number of words seen, and the current game status. The opponent characteristics include the number of questions answered by the opponent, the average buzzing time of the opponent, and the error rate of the opponent.

5.2.2. Results

Based on the content model, we trained each abovementioned player for 50 epochs on the quiz bowl game, and testing was performed using the test set after each epoch. Figure 6a–c record the changes in the rewards of the -Fc2 players. The DQN player cannot adapt to changes in the strategy of the opponents, resulting in sharp fluctuations in the rewards during the later stage of training, whereas the rewards for all the players with opponent modeling are more robust. The rewards obtained with the DualFc2, DecoupleFc2, and DecoupleDualFc2 models are significantly more stable than those obtained with the baseline model DRON-Fc2.

The highest reward and the average reward for the last 10 epochs are similarly recorded for the -Fc2 players and are shown in

Table 3. Comparison of rewards for DecoupleDualFc2 in the quiz bowl game.

Player	Maximum Reward	Average Reward
DQN	0.7709	−0.2171
DRON-Fc2	0.7918	−0.0736
DualFc2	0.8788	0.3426
DecoupleFc2	0.8331	0.2781
DecoupleDualFc2	0.7910	0.4099

. Incorporating the opponent model provides the players with clear advantages over the DQN network in terms of both the highest and average rewards. The highly effective proposed network structure and learning algorithm result in the DualFc2 player obtaining the highest reward and DecoupleDualFc2 obtaining the highest average reward.

Figure 7a–c show the performance of the -Moe players. In terms of the degree of fluctuations in the rewards, the performance of the players with opponent modeling is clearly more robust than that of the DQN model without opponent modeling. The -Moe players obtain a higher maximum reward than the DRON-Moe player.

The maximum rewards and the average rewards for the last 10 epochs are similarly recorded and calculated and are shown in

Table 4. Comparison of rewards for DecoupleDualMoe in the quiz bowl game.

Player	Maximum Reward	Average Reward
DQN	0.7709	−0.2171
DRON-Moe	0.7819	0.5292
DualMoe	0.8606	0.6174
DecoupleMoe	0.8017	0.5911
DecoupleDualMoe	0.8307	0.6346

. The maximum and average rewards of the players with opponent modeling are greater than those of the DQN player. The DecoupleDualMoe player obtains the highest average reward, and the DualMoe player achieves the highest maximum reward.

These results indicate that our model, consisting of a network structure and training algorithm for opponent modeling, is effective in helping agents make better decisions and increase payoffs in interactive scenarios.

6. Conclusions and Future Work

Opponent modeling is an important issue for agents to make appropriate decisions in various strategic settings. A novel implicit opponent modeling framework was developed in this study to improve the payoffs of players’ decisions under interactive scenarios. Two opponent networks, DualFc2 and DualMoe, were constructed by using deep neural networks. A novel reinforcement learning algorithm for the opponent networks with a coupling strategy was proposed.

The most relevant study in the literature to the present study is based on the DRON approach in which opponents are modeled implicitly. The DRON network structures and learning algorithms are different from those used in our proposed method. More importantly, our approach uses separate modules to evaluate the state and the action in the network structure and objectively evaluates the actions of the opponents via an appropriate learning algorithm for the opponent network.

Experiments were conducted demonstrating that our opponent modeling framework can effectively improve agents’ payoffs under dynamic interactive scenarios over those obtained using DQN without opponent modeling and DRON.

Future research on opponent modeling still faces some challenges. For example, the existing opponent modeling approach often assumes that agents can obtain complete information for the environment and opponents. However, environmental or agent characteristics are only partially observable in many application scenarios, such that the agent obtains incomplete or uncertain observations of the environment or other agents. This partial observation increases the difficulty of opponent modeling. Methods for opponent modeling under partially observable conditions have been proposed in the field of extensive games. Opponent modeling under partially observable conditions and wider decision intelligence scenarios warrants further exploration.