Learning Optimal Strategies in a Duel Game

Abstract

We study a duel game in which each player has incomplete knowledge of the game parameters. We present a simple, heuristically motivated and easily implemented algorithm by which, in the course of repeated plays, each player estimates the missing parameters and consequently learns his optimal strategy.

1. Introduction

We study a two-player zero-sum duel game in which certain game parameters are unknown, and we present an algorithm by which each player can estimate the opponent’s parameters and consequently learn his optimal strategy.

The duel game with which we are concerned has been presented in Polak (2007), and a similar one was examined in Prisner (2014). The game is a variation of a duel game and, more generally, games of timing, as presented in Fox and Kimeldorf (1969); Garnaev (2000); Radzik (1988).

We call the two players $P_1$ and $P_2$; for $n\in \{1,2\}$, $P_n$ has a kill probability vector ${\mathbf{p}}_n$, with the component $p_{n,d}$ (for $d\in \mathbb{N}$) giving the probability that $P_n$ kills his opponent when he shoots at him from distance d. In our analysis, we make the following assumptions:

• The kill probabilities are given by a function, f:

  $$\forall n,d:p_{n,d}=f(d,{\theta }_n),$$

  where ${\theta }_n$ is a parameter vector.
• Each $P_n$ knows the general form $f(d,\theta )$ and his own parameter vector ${\theta }_n$, but not that of his opponent.
• The duel will be repeated a large number of times.

Each $P_n$’s goal is to compute a strategy that optimizes his payoff, which (as will be seen in Section 2), depends on both ${\mathbf{p}}_1$ and ${\mathbf{p}}_2$; since $P_n$ does not know his opponents kill probability, he does not know either his own or his opponent’s payoff. In short, we have an incomplete information game.

Games of incomplete information are those in which each player has only partial information about the payoff parameters. Clearly, the players must utilize some form of learning, namely the use of information (i.e., data) collected from previous plays of a game to adjust their strategies through belief update, imitation, reinforcement etc.

The first work on learning in games was, arguably, the introduction of the fictitious play algorithm in Brown (1949); Robinson (1951). In later years, several approaches to learning were proposed in the literature.

The Bayesian game approach introduced by Harsanyi (1962) has led to intensive research on the subject of Bayesian games; for a recent review, see Zamir (2020). However, the main corpus of this literature is concerned with Bayesian reasoning for a single play of a game, rather than learning from repeated plays, and this point of view is not particularly relevant to our approach.

Another approach, which can be understood as a form of “intergenerational” learning, is that of evolutionary game theory, as presented in Alexander (2023); Tanimoto (2015); Weibull (1997), but this is not directly concerned with games of incomplete information and hence is also not very relevant to our approach. However, for some interesting remarks on the connection between learning and evolutionary game theory, see the book by Cressman (2003).

The main approach to learning in games is the one exemplified by Fudenberg and Levine (1998)’s seminal book The Theory Of Learning In Games. This has spawned a tremendous amount of research; a recent overview of the field appears in Fudenberg and Levine (2016).

Fudenberg and Levine’s approach is directly relevant to learning in the duel game, but we believe it is too general. In particular, we have not been able to find any works which address the specific case in which the functional form of payoff functions is known to the players and it is only required to estimate parameter values.

This is essentially a problem of parameter estimation, and there is another research corpus in which this problem is addressed explicitly: that of machine learning. Particularly relevant to our needs is the literature on multi-agent reinforcement learning, such as Alonso et al. (2001); Gronauer and Diepold (2022). Reinforcement learning is a basic component of machine learning and, in the machine learning community, the connection between multi-agent problems and game theory has long been recognized (Nowe et al., 2012; Rezek et al., 2008); recent and very extensive reviews can be found in Jain et al. (2024); Yang and Wang (2020).

In particular, the issue of convergence of parameter estimation has been comprehensively studied (Hussain et al., 2023; Leonardos et al., 2021; Martin & Sandholm, 2021; Mertikopoulos & Sandholm, 2016). An important conclusion of these and related works is the trade-off between exploration and exploitation. Quoting from Hussain et al. (2023): “… understanding exploration remains an important endeavour as it allows agents to avoid suboptimal, or potentially unsafe areas of their state space … In addition, it is empirically known that the choice of exploration rate impacts the expected total reward”.

In short, extensive literature on learning and incomplete information games is available but, while some ingenious and highly technical approaches have been proposed, we believe that much simpler techniques suffice for our duel learning problem. This is mainly due to the fact that each player knows the general form of the kill probabilities and only needs to estimate his opponent’s parameter vector. Hence, in the current paper, we propose a simple, heuristically motivated, and easily implemented algorithm which allows each player to estimate his optimal strategy, using the information collected from repeated plays of the duel.

The paper is organized as follows. In Section 2 we present the rules of the game. In Section 3 we solve the game under the assumption of complete information. In Section 4 we present our algorithm for solving the game when the players have incomplete information. In Section 5 we evaluate the algorithm by numerical experiments. Finally, in Section 6 we summarize our results and present our conclusions.

2. Game Description

The duel with which we are concerned is a zero-sum game played between players $P_1$ and $P_2$, under the following rules.

• It is played in discrete rounds (time steps) $t\in \left\{1,2,\dots \right\}$.
• In the first turn, the players are at distance D.
• $P_1$ (resp. $P_2$) plays on odd (resp. even) rounds.
• On his turn, each player has two choices: (i) he can shoot his opponent or (ii) he can move one step forward, reducing their distance by one.
• If $P_n$ shoots (with $n\in \{1,2\}$), he has a kill probability $p_n\left(d\right)$ of hitting (and killing) his opponent, where d is their current distance. If he misses, the opponent can walk right next to him and shoot him for a certain kill.
• Each player’s payoff is 1 if he kills the opponent and $-1$ if he is killed. Note that a player who misses his shot has payoff $-1$, since the opponent will approach him at distance one and shoot at him with kill probability one.

For $n\in \{1,2\}$, we will denote by $x_n\left(t\right)$ the position of $P_n$ at round t. The starting positions are $x_1\left(0\right)=0$ and $x_2\left(0\right)=D$, with $D=2N,N\in \mathbb{N}$. The distance between the players at time t is

$$d=|x_1\left(t\right)-x_2\left(t\right)|.$$

For $n\in \{1,2\}$, the kill probability is a decreasing function $p_n:\{1,2,\dots D\}\to [0,1]$ with $p_n\left(1\right)=1$. It is convenient to describe the kill probabilities as vectors:

$${\mathbf{p}}_n=\left(p_{n,1},\dots ,p_{n,D}\right)=\left(p_n\left(1\right),\dots ,p_n\left(D\right)\right).$$

This duel can be modeled as an extensive form game or tree game. The game tree is a directed graph $G=(V,E)$ with vertex set

$$V=\left\{1,2,\dots ,2D\right\},$$

where

• the vertex $v=d\in \left\{1,2,\dots ,D\right\}$ corresponds to a game state in which the players are at distance d, and
• the vertex $v=d+D\in \left\{D+1,D+2,\dots ,2D\right\}$ is a terminal vertex, in which the “active” player has fired at his opponent.

The edges correspond to state transitions; it is easy to see that the edge set is

$$E=\left\{\left(1,D+1\right),\left(2,1\right),\left(2,D+2\right),\left(3,2\right),\left(3,D+3\right),\dots ,\left(D,D-1\right),\left(D,2D\right)\right\}.$$

An example of the game tree, for $D=6$, appears in Figure 1. The circular (resp. square) vertices are the ones in which $P_1$ (resp. $P_2$) is active, and the rhombic vertices are the terminal ones.

To complete the description of the game, we will define the expected payoff for the terminal vertices. Note that the terminal vertex $d+D$ is the child of the nonterminal vertex d in which:

• The distance of the players is d and, assuming $P_n$ to be the active player, his probability of hitting his opponent is $p_{n,d}$.
• The active player is $P_1$ (resp. $P_2$) if d is even (resp. odd).

Keeping the above in mind, we see that the payoff (to $P_1$) of vertex $d+D$ is

$$\forall d\in \left\{1,\dots ,D\right\}:Q\left(d+D\right)=\left\{\begin{array}{cc}{p}_{1,d}·1+\left(1-p_{1,d}\right)·\left(-1\right)=2p_{1,d}-1,\hfill & \mathrm{when}d\mathrm{is}\mathrm{even};\hfill \\ p_{2,d}·\left(-1\right)+\left(1-p_{2,d}\right)·1=1-2p_{2,d}\hfill & \mathrm{when}d\mathrm{is}\mathrm{odd}.\hfill \end{array}\right.$$

Since this is a zero-sum game, the payoff to $P_2$ at vertex $d+D$ is $-Q\left(d+D\right)$. This completes the description of the duel.

3. Solution with Complete Information

It is easy to solve the above duel when each player knows D and both ${\mathbf{p}}_{\mathbf{1}}$ and ${\mathbf{p}}_{\mathbf{2}}$. We construct the game tree as described in Section 2 and we solve it by backward induction. Since the method is standard, we simply give an example of its application. Suppose that

$$\forall n,d:p_{n,d}=\left\{\begin{array}{cc}1\hfill & \mathrm{when}d=1,\hfill \\ min\left(1,{\displaystyle \frac{c_n}{d^{k_n}}}\right)\hfill & \mathrm{when}d>1.\hfill \end{array}\right.$$

We take $c_1=1$, $k_1=1$, $c_2=1$, $k_2={\displaystyle \frac{1}{2}}$. The kill probabilities are as seen in

Table 1. Kill probabilities.

d	1	2	3	4	5	6
$p_{1,d}$	1.0000	0.5000	0.3333	0.2500	0.2000	0.1667
$p_{2,d}$	1.0000	0.7071	0.5774	0.5000	0.4472	0.4082

.

The game tree with terminal payoffs is illustrated in Figure 2.

By the standard backward induction procedure, we select the optimal action at each vertex and also compute the values of the nonterminal vertices. These are indicated in Figure 3 (optimal actions correspond to thick edges). We see that the game value is $-0.1547$, attained by $P_2$ shooting when the players are at distance 3 (which happens in round 4).

We next present a proposition which characterizes each player’s optimal strategy in terms of a shooting threshold.1 In the following, we use the standard notation by which “$-n$” denotes the “other player”, i.e., $p_{-1}=p_2$ and $p_{-2}=p_1$.2

Theorem 1. 

We define for $n\in \left\{1,2\right\}$ the shooting criterion vectors ${\mathbf{K}}_n=\left(K_{n,1},\dots ,K_{n,D}\right)$ where

$$K_{n,1}=1\mathrm{and}\mathrm{for}d\ge 2:K_{n,d}=p_{n,d}+p_{-n,d-1}.$$

Then, the optimal strategy for $P_n$ is to shoot as soon as it is his turn and the distance of the players is less than or equal to the shooting threshold $d_n$, where:

$$d_1=max\{d:K_{1,d}\ge 1\},d_2=max\{d:K_{2,d}\ge 1\}.$$

Proof. 

Suppose the players are at distance d and the active player is $P_n$.

• If $P_{-n}$ will not shoot in the next round, when their distance will be $d-1$, then $P_n$ must also not shoot in the current round, because he will have a higher kill probability in his next turn, when they will be at distance $d-2$.
• If $P_{-n}$ will shoot in the next round, when their distance will be $d-1$, then $P_n$ should shoot if $p_{n,d}$ (his kill probability now) is higher than $1-p_{-n,d-1}$ ($P_{-n}$’s miss probability in the next round). In other words, $P_n$ must shoot if

  $$p_{n,d}\ge 1-p_{-n,d-1}$$

  or, equivalently, if

  $$K_{n,d}=p_{n,d}+p_{-n,d-1}\ge 1.$$

  (1)

Hence, we can reason as follows:

• At vertex 1, $P_2$ is active and his only choice is to shoot.
• At vertex 2, $P_1$ is active and he knows $P_2$ will certainly shoot in the next round. Hence, $P_1$ will shoot if he has an advantage, i.e., if

  $$Q_1\left(2\right)\ge Q_1\left(1\right)\iff 2p_{1,2}-1\ge 1-2p_{2,1}\iff p_{1,2}+p_{2,1}\ge 1.$$
  This is equivalent to $K_{1,1}\ge 1$ and will always be true.
• Hence, at vertex 3, $P_2$ is active and he knows that $P_1$ will certainly shoot in the next round (at vertex d). So, $P_2$ will shoot if

  $$Q_1\left(3\right)\le Q_1\left(2\right)\iff 1-2p_{2,3}\le 2p_{1,2}-1\iff p_{2,3}+p_{1,2}\ge 1.$$

  which is equivalent to $K_{2,3}\ge 1$. Also, if $K_{2,3}<1$, then $P_1$ will know, when the game is at vertex 4, that $P_2$ will not shoot when at 3. So,  $P_1$ will not shoot when at 4. But then, when at 5, $P_2$ knows that $P_1$ will not shoot when at 4. Continuing in this manner, we see that $K_{2,3}<1$ implies that firing will take place exactly at the vertex 2.
• On the other hand, if $K_{2,3}\ge 1$, then $P_1$ knows when at 4 that $P_2$ will shoot at the next round. So, when at 4, $P_1$ should shoot if $K_{1,4}\ge 1$. If, on the other hand, $K_{1,4}<1$, then $P_1$ will not shoot when at 4 and $P_2$ will shoot when at 3.
• We continue in this manner for increasing values of d. Since both $K_{1,d}$ and $K_{2,d}$ are decreasing with d, there will exist a maximum $d_n$ value (it could equal D) in which some $K_{n,d}$ will be greater than one and $P_n$ will be active; then, $P_n$ must shoot as soon as the game reaches or passes vertex $d_n$ and he “has the action”.

This completes the proof.    □

Returning to our previous example, we compute the vectors $K_n$ for $n\in \{1,2\}$ and list them in the following

Table 2. Shooting criterion.

d	1	2	3	4	5	6
Round	6	5	4	3	2	1
$K_{1,d}$		1.5000	1.0404	0.8274	0.7000	0.6139
$K_{2,d}$		1.7071	1.0774	0.8333	0.6972	0.6082

.

For $P_1$ the shooting criterion is last satisfied when the distance is $d=3$; this happens at round 4, in which $P_1$ is inactive, so he should shoot at round 5. However, for $P_2$ the shooting criterion is also last satisfied at distance $d=3$ and round 4, in which $P_2$ is active; so, he should shoot at round 4. This is the same result we obtained with backward induction.

4. Solution with Incomplete Information

As already mentioned, the implementation of either the backward induction or the shooting criterion requires complete information, i.e., knowledge by both players of all game parameters: D, ${\mathbf{p}}_1$, and ${\mathbf{p}}_2$.

In what follows, we will assume that the players’ kill probabilities have the functional form (where $\theta$ is a parameter vector):

$$\forall d:p_{1,d}=f\left(d;{\theta }_1\right),p_{2,d}=f\left(d;{\theta }_2\right)$$

Suppose now that both players know D and $f\left(d;\theta \right)$ but, for $n\in \{1,2\}$, $P_n$ only knows his own parameter vector $\theta ={\theta }_n$ and is ignorant of the opponent’s ${\theta }_{-n}$. Hence, each $P_n$ knows ${\mathbf{p}}_n$, but not ${\mathbf{p}}_{-n}$. Consequently, neither player knows his payoff function, which depends on both ${\mathbf{p}}_1$ and ${\mathbf{p}}_2$.

In this case, obviously, the players cannot perform the computations of either backward induction or the shooting criterion. Instead, assuming that the players will engage in multiple duels, we propose the use of an heuristic “exploration-and-exploitation” approach in which each player initially adopts a random strategy and, using information collected from played games, he gradually builds and refines an estimate of his optimal strategy. Our approach is implemented by Algorithm 1, presented below in pseudocode.

Algorithm 1 Learning the Optimal Duel Strategy.

1: Input: Duel parameters D, ${\theta }_1$, ${\theta }_2$; Learning parameters $\lambda$, ${\sigma }_0$; Number of plays R   2: ${\mathbf{p}}_1$ = CompKillProb$\left({\theta }_1\right)$   3: ${\mathbf{p}}_2$ = CompKillProb$\left({\theta }_2\right)$   4: Randomly initialize parameter estimates ${\theta }_1^0$, ${\theta }_2^0$   5: for $r\in \{1,2,\dots ,R\}$ do   6:     ${\mathbf{p}}_1^r$ = CompKillProb$\left({\theta }_1^{r-1}\right)$   7:     ${\mathbf{p}}_2^r$ = CompKillProb$\left({\theta }_2^{r-1}\right)$   8:     $d_1^r$ = CompShootDist$({\mathbf{p}}_1,{\mathbf{p}}_2^r)$   9:     $d_2^r$ = CompShootDist$({\mathbf{p}}_1^r,{\mathbf{p}}_2)$ 10:     ${\sigma }_r={\sigma }_{r-1}/\lambda$ 11:     X = PlayDuel$({\mathbf{p}}_1,{\mathbf{p}}_2,d_1^r,d_2^r,{\sigma }_r,X)$ 12:     $({\widehat{\mathbf{p}}}_1^r,{\widehat{\mathbf{p}}}_2^r)$ = EstKillProb$\left(X\right)$ 13:     ${\theta }_1^r$ = EstPars$({\widehat{\mathbf{p}}}_1^r,1)$ 14:     ${\theta }_2^r$ = EstPars$({\widehat{\mathbf{p}}}_2^r,2)$ 15: end for 16: return $d_1^R$, $d_2^R$, ${\theta }_1^R$, ${\theta }_2^R$

The following remarks explain the operation of the algorithm.

• In line 1, the algorithm takes as input: (i) the duel parameters D, ${\theta }_1$, and ${\theta }_2$; (ii) two learning parameters $\lambda ,{\sigma }_0$; and (iii) the number R of duels used in the learning process.
• Then, in lines 2–3, the true kill probability ${\mathbf{p}}_n$ (for $n\in \{1,2\}$) is computed by the function CompKillProb$\left({\theta }_n\right)$, which simply computes

  $$\forall n,d:p_n\left(d\right)=f_n\left(d;{\theta }_n\right).$$
  We emphasize that these are the true kill probabilities.
• In line 4, randomly selected parameter vector estimates ${\theta }_1^0,{\theta }_2^0$ are generated.
• Then, the algorithm enters the loop of lines 5–15 (executed for R iterations), which constitutes the main learning process.

  (a)

  In lines 6–7 we compute new estimates of the kill probabilities ${\mathbf{p}}_n^r$, by function CompKillProb, based on the estimates of parameters ${\theta }_n^{r-1}$:

  $$\forall n,d:p_n^r\left(d\right)=f_n\left(d;{\theta }_n^{r-1}\right).$$

  We emphasize that these are estimates of the kill probabilities, based on the parameter estimates ${\theta }_n^{r-1}$.

  (b)

  In lines 8–9 we compute new estimates of the shooting thresholds $d_n^r$, by function CompShootDist. For $P_n$, this is achieved by computing the shooting criterion ${\mathbf{K}}_n$ using the (known to $P_n$) ${\mathbf{p}}_n$ and the (estimated by $P_n$) ${\mathbf{p}}_{-n}^r$.

  (c)

  In line 10, ${\sigma }_r$ (which will be used as a standard deviation parameter) is divided by the factor $\lambda >1$.

  (d)

  In line 11, the result of the duel is computed by the function PlayDuel. This is achieved as follows:

  • For $n\in \left\{1,2\right\}$, $P_n$ selects a random shooting distance ${\widehat{d}}_n$ from the discrete normal distribution (Roy, 2003) with mean $d_n^r$ and standard deviation ${\sigma }_r$.
  • With both ${\widehat{d}}_1,{\widehat{d}}_2$ selected, it is clear which player will shoot first; the outcome of the shot (hit or miss) is a Bernoulli random variable with success probability $p_{n,{\widehat{d}}_n}$, where $P_n$ is the shooting player. Note that ${\mathbf{p}}_n$ is the true kill probability.

  The result is stored in a table X, which contains the data (shooting distance, shooting player, hit or miss) of every duel played up to the r-th iteration.

  (e)

  In line 12, the entire game records X are used by EstKillProb$\left(X\right)$ to obtain empirical estimates of the kill probabilities ${\widehat{\mathbf{p}}}_1^r,{\widehat{\mathbf{p}}}_2^r$. These estimates are as follows:

  $$\forall n,\forall d\in {\mathcal{D}}_n:{\widehat{p}}_{n,d}^r={\displaystyle \frac{{\sum }_{r\in {\mathcal{R}}_{n,d}}{Z}_r}{\left|{\mathcal{R}}_{n,d}\right|}}$$

  where

  $$\begin{array}{cc}\hfill {\mathcal{D}}_n& =\left\{d:d\mathrm{such}\mathrm{that}{P}_n\mathrm{may}\mathrm{shoot}\right\}\hfill \\ \hfill {\mathcal{R}}_{n,d}& =\left\{r:\mathrm{in}\mathrm{the}r\text{-}\mathrm{th}\mathrm{game}{P}_n\mathrm{actually}\mathrm{shot}\mathrm{from}\mathrm{distance}d\right\},\hfill \\ \hfill Z_r& =\left\{\begin{array}{cc}1\hfill & \mathrm{if}\mathrm{the}\mathrm{shot}\mathrm{in}\mathrm{the}r\text{-}\mathrm{th}\mathrm{game}\mathrm{hit}\mathrm{the}\mathrm{target},\hfill \\ 0\hfill & \mathrm{if}\mathrm{the}\mathrm{shot}\mathrm{in}\mathrm{the}r\text{-}\mathrm{th}\mathrm{game}\mathrm{missed}\mathrm{the}\mathrm{target}\hfill \end{array}\right.\hfill \end{array}$$

  (f)

  In lines 13–14 the function EstPars uses a least squares algorithm to find (only for the $P_n$ who currently has the action) ${\theta }_n^r$ values which minimize the squared error

  $$J\left({\theta }_n\right)=\sum _{d\in {\mathcal{D}}_n}{\left(f_n\left(d;{\theta }_n\right)-{\widehat{p}}_{n,d}^r\right)}^2.$$

  (g)

  In line 16, the algorithm returns the final estimates of optimal shooting distances $d_1^R,d_2^R$ and parameters ${\theta }_1^R,{\theta }_2^R$.

The core of the algorithm is the exploration-exploitation approach, which is implemented by the gradual reduction of ${\sigma }_r$ in line 10. Since $1/\lambda \in \left(0,1\right)$, we have ${lim}_{r\to \infty }{\sigma }_r=0$.

Exploration is predominant in the initial iterations of the algorithm, when the relatively large value of ${\sigma }_r$ implies that the players use essentially random shooting thresholds. In this phase, $P_n$’s shooting threshold and, consequently, his payoff, is suboptimal. However, since $P_{-n}$ is also using various, randomly selected shooting thresholds, $P_n$ collects information about $P_{-n}$’s kill probability at various distances. This information can be used to estimate ${\theta }_n$; the fact that the functional form of the kill probabilities is known results in a tractable parameter estimation problem.

As r increases, we gradually enter the exploitation phase. Namely, ${\sigma }_r$ tends to zero, which means that each $P_n$ uses a shooting threshold that is, with high probability, very close to the one which is optimal with respect to ${\widehat{\mathbf{p}}}_{-n}^r$, and his current estimate of ${\mathbf{p}}_{-n}$, the opponent’s kill probability. Provided a sufficient amount of information was collected in the exploration phase, ${\widehat{\mathbf{p}}}_{-n}^r$ is sufficiently close to ${\mathbf{p}}_{-n}$ to ensure that the used shooting threshold is close to the optimal one. The key issue is to choose a learning rate $\lambda$ that is “sufficiently slow” to ensure that the exploration phase is long enough to provide enough information for convergence of the parameter estimates to the true parameter values, but not “too slow”, because this will result in slow convergence. This is the exploration-exploitation dilemma, which has been extensively studied in the reinforcement learning literature. For several Q-learning algorithms it has been established that convergence to the optimal policy is guuaranteed, provided the learning rate is sufficiently slow (Ishii et al., 2002; Osband & Van Roy, 2017; Singh et al., 2000); in this manner, a balance is achieved between exploration, which collects data on long-term payoff, and exploitation, which maximizes near-term payoff. Similar conclusions have been established for multi-agent reinforcement learning (Hussain et al., 2023; Martin & Sandholm, 2021).

Since our approach is heuristic, the evaluation of an “appropriately slow” learning rate is performed by the numerical experiments in Section 5. Before proceeding, let us note that “methods with the best theoretical bounds are not necessarily those that perform best in practice” (Martin & Sandholm, 2021).

5. Experiments

Since the motivation for our proposed algorithm is heuristic, in this section, we give an empirical evaluation of its performance. In the following numerical experiments, we will investigate the influence of both the duel parameters (D, ${\mathbf{p}}_1$, ${\mathbf{p}}_2$) and the algorithm parameters ($\lambda$, ${\sigma }_0$, R).

5.1. Experiments Setup

In the following subsection, we present several experiment groups, all of which share the same structure. Each experiment group corresponds to a particular form of the kill probability functions. In each case, the kill probability parameters, along with the initial player distance D, are the game parameters. For each choice of game parameters we proceed as follows.

First, we select the learning parameters $\lambda ,{\sigma }_0$ and the number of learning steps R. These, together with the game parameters, are the experiment parameters. Then, we select a number J of estimation experiments to run for each choice of experiment parameters. For each of the J experiments, we compute the following quantities:

• The relative error of the final kill probability parameter estimates. For a given parameter ${\theta }_{n,i}$, this error is defined to be

  $$\Delta {\theta }_{n,i}=\left|{\displaystyle \frac{{\theta }_{n,i}-{\theta }_{n,i}^R}{{\theta }_{n,i}}}\right|.$$
• The relative error of the shooting threshold estimates. Letting $d_n^R$ be the estimate of the shooting threshold based on the true kill probability vector $p_n$ and kill probability vector estimate $p_{-n}^R$, this error is defined to be

  $$\Delta d_n=\left|{\displaystyle \frac{d_n-d_n^R}{d_n}}\right|.$$
• The relative error of the optimal payoff estimates. Letting $Q_n^R$ be the estimate of the optimal payoff (computed from the estimated shooting thresholds $d_1^R,d_2^R$), this error is defined to be

  $$\Delta Q_n=\left\{\begin{array}{cc}\left|{\displaystyle \frac{Q_n-Q_n^R}{Q_n}}\right|\hfill & \mathrm{iff}{Q}_n\ne 0\hfill \\ 0\hfill & \mathrm{iff}{Q}_n=0\mathrm{and}{Q}_n^R=Q_n\hfill \\ 1\hfill & \mathrm{iff}{Q}_n=0\mathrm{and}{Q}_n^R\ne Q_n\hfill \end{array}\right.$$
  Note that $\Delta Q_2=\Delta Q_1$, because the game is zero-sum.

5.2. Experiment Group A

In this group, the kill probability function has the form:

$$n\in \left\{1,2\right\}:p_{n,d}=min\left({\displaystyle \frac{c_n}{d^{k_n}}},1\right).$$

Let us look at the final results of a representative run of the learning algorithm. With $c_1=1$, $k_1=0.5$, $c_2=1$, $k_2=1$ and $D=10$, we run the learning algorithm with $R=1500$, ${\sigma }_0=6D$, and for three different values $\lambda \in \{1.001,1.01,1.05\}$. In Figure 4 we plot the logarithm (with base 10) of the relative payoff error $\Delta Q_1+ϵ$ (we have added $ϵ={10}^{-3}$ to deal with the logarithm of zero error). The three curves plotted correspond to the $\lambda$ values 1.001, 1.01, and 1.05. We see that, for all $\lambda$ values, the algorithm achieves zero relative error; in other words, it learns the optimal strategy for both players. Furthermore, convergence is achieved by the 1500th iteration of the algorithm (1500th duel played), as seen by the achieved logarithm value $-3$ (recall that we have added $ϵ={10}^{-3}$ to the error, hence the true error is zero). Convergence is fastest for the largest $\lambda$ value, i.e., $\lambda =1.05$, and slowest for the smallest value, $\lambda =1.001$.

In Figure 5, we plot the logarithmic relative errors ${log}_{10}\Delta d_n$. These also, as expected, have converged to zero by the 1500th iteration.

The fact that the estimates of the optimal shooting thresholds and strategies achieve zero error does not imply that the same is true of the kill probability parameter estimates. In Figure 6 we plot the relative errors $\Delta c_1$ and $\Delta k_1$.

It can be seen that these errors do not converge to zero; in fact, for $\lambda \in \{1.01,1.05\}$, the errors converge to fixed nonzero values, which indicates that the algorithm obtains wrong estimates. However, the error is sufficiently small to still result in zero-error estimates of the shooting thresholds. The picture is similar for the errors $\Delta c_2$ and $\Delta k_2$; hence, their plots are omitted.

In the above, we have given results for a particular run of the learning algorithm. This was a successful run, in the sense that it obtained zero-error estimates of the optimal strategies (and shooting thresholds). However, since our algorithm is stochastic, it is not guaranteed that every run will result in zero-error estimates. To better evaluate the algorithm, we ran it $J=10$ times and averaged the obtained results. In particular, in Figure 7, we plot the average of ten curves of the type plotted in Figure 4. Note that now we plot the curve for $R=5000$ plays of the duel.

Several observations can be made regarding Figure 7.

• For the smallest $\lambda$ value, namely, $\lambda =1.001$, the respective curve reaches $-3$ at $r=4521$. This corresponds to zero average error, which means that, in some algorithm runs, it took more than 4500 iterations (duel plays) to reach zero error.
• For $\lambda =1.01$, all runs of the algorithm reached zero-error after $r=656$ runs.
• Finally, for $\lambda =1.05$, the average error never reached zero; in fact, 3 out of 10 runs converged to nonzero-error estimates, i.e., to non-optimal strategies.

The above observations corroborate the remarks at the end of Section 4 regarding reinforcement learning. Namely, a small learning rate (in our case small $\lambda$) results in higher probability of converging to the true parameter values, but also in slower convergence. This can be explained as follows: a small $\lambda$ results in higher ${\sigma }_r$ values for a large proportion of duels played by the algorithm; i.e., in more extensive exploration, which however results in slower exploitation (convergence).

We concluded this group of experiments by running the learning algorithm for various combinations of game parameters; for each combination, we recorded the average error attained at the end of the algorithm (i.e., at $r=R=5000$). The results are summarized in the following

Table 3. Values of final average relative error $\Delta Q_1$ for $c_2=1$, $k_2=1$ and various values of $c_1$, $k_1$, $\lambda$. D is fixed at $D=10$.

	$\mathit{\lambda }$		1.001			1.01			1.05
	${\mathit{k}}_{\mathbf{1}}$	0.50	1.00	1.50	0.50	1.00	1.50	0.50	1.00	1.50
${\mathit{c}}_{\mathbf{1}}$
1.00		0.000	0.000	0.000	0.000	0.100	0.482	0.224	0.500	1.207
1.50		0.000	0.000	0.000	0.059	0.049	1.748	0.215	0.480	3.777
2.00		0.000	0.000	0.048	0.074	0.199	0.097	0.144	0.980	0.072

,

Table 4. Round at which $\Delta Q_1$ converged to zero for all $J=10$ sessions for $c_2=1$, $k_2=1$ and various values of $c_1$, $k_1$, $\lambda$. D is fixed at $D=10$. If $\Delta Q_1$ did not converge for all sessions, we note for how many sessions it converged.

	$\mathit{\lambda }$	1.001			1.01			1.05
	${\mathit{k}}_{\mathbf{1}}$	0.50	1.00	1.50	0.50	1.00	1.50	0.50	1.00	1.50
${\mathit{c}}_{\mathbf{1}}$
1.00		4521	1456	2983	656	9/10	8/10	7/10	5/10	5/10
1.50		3238	2939	1754	7/10	9/10	9/10	0/10	8/10	6/10
2.00		4153	423	8/10	2/10	9/10	6/10	3/10	4/10	6/10

and

Table 5. Fraction of learning sessions that converged to $\Delta Q_1-0$ for different values of D and $\lambda$.

	$\mathit{\lambda }$	1.001	1.01	1.05
$\mathit{D}$
8		163/170	144/170	89/170
10		165/170	139/170	81/170
12		161/170	123/170	68/170
14		164/170	134/170	73/170
total		653/680	540/680	311/680

.

From

Table 6. Values of final average relative error $\Delta Q_1$ for $d_{21}=1$, $d_{22}=D$ and various values of $d_{11}$, $d_{12}$, $\lambda$. D is fixed at $D=10$.

	$\mathit{\lambda }$	1.001	1.01	1.05
${\mathit{d}}_{\mathbf{11}}$	${\mathit{d}}_{\mathbf{12}}$
1	$D/2$	0.000	0.233	1.000
1	$2D/3$	0.000	0.000	0.799
1	D	$\sim 6·{10}^{-16}$	$\sim 2·{10}^{-16}$	0.800
$D/3$	$2D/3$	0.000	0.000	2.799
$D/3$	D	0.000	0.000	0.777
$D/2$	D	0.000	0.000	0.480

and

Table 7. Round at which $\Delta Q_1$ converged to zero for all $J=10$ sessions for $d_{21}=1$, $d_{22}=D$ and various values of $d_{11}$, $d_{12}$, $\lambda$. D is fixed at $D=10$. If $\Delta Q_1$ did not converge for all sessions we note for how many sessions it converged.

	$\mathit{\lambda }$	1.001	1.01	1.05
${\mathit{d}}_{\mathbf{11}}$	${\mathit{d}}_{\mathbf{12}}$
1	$D/2$	287	7/10	0/10
1	$2D/3$	812	333	9/10
1	D	7/10	9/10	9/10
$D/3$	$2D/3$	326	139	5/10
$D/3$	D	225	397	5/10
$D/2$	D	711	464	6/10

, we see that for $\lambda =1.001$, almost all learning sessions conclude in zero $\Delta Q_1$, while increasing the value of $\lambda$ results in more sessions concluding with non-zero error estimates. Furthermore, we observe that when the average $\Delta Q_1$ converges to zero for multiple values of $\lambda$, the convergence is faster for bigger $\lambda$. These results highlight the trade off between exploration and exploitation discussed above.

Finally, in

Table 8. Fraction of learning sessions that converged to $\Delta Q_1-0$ for different values of D and $\lambda$.

	$\mathit{\lambda }$	1.001	1.01	1.05
$\mathit{D}$
8		76/110	80/110	49/110
10		97/110	95/110	56/110
12		94/110	78/110	39/110
14		100/110	84/110	40/110
total		367/440	337/440	184/440

, we see how many learning sessions were run and how many converged to the zero error estimate $\Delta Q_1$ for different values of D and $\lambda$.

5.3. Experiment Group B

In this group, the kill probability function is piecewise linear:

$$p_{n,d}=\left\{\begin{array}{cc}1& \mathrm{when}d\in \left[1,d_{n1}\right]\\ -{\displaystyle \frac{1}{d_{n2}-d_{n1}}}d+{\displaystyle \frac{d_{n2}}{d_{n2}-d_{n1}}}& \mathrm{when}d\in \left[d_{n1},d_{n2}\right]\\ 0& \mathrm{when}d\in \left[d_{n2},D\right]\end{array}\right.$$

Let us look again at the final results of a representative run of the learning algorithm. With $d_{11}=D$, $d_{12}=D/3$, $d_{21}=1$, $d_{22}=D$, and $D=8$, we run the learning algorithm with $R=500$, ${\sigma }_0=6D$, and for the values $\lambda \in \{1.001,1.01,1.05\}$. In Figure 8 we plot the logarithm (with base 10) of the relative payoff error $\Delta Q_1+ϵ$. We see similar results as in Group A, for all $\lambda$ values, the algorithm achieves zero relative error. Convergence is achieved by the 300th iteration of the algorithm and it is fastest for $\lambda =1.05$, and slowest for $\lambda =1.001$. The plots of the errors ${log}_{10}\Delta d_1$, ${log}_{10}\Delta d_2$ are omitted, since they are similar to the ones given in Figure 5.

As in Group A, the fact that the estimates of the optimal shooting thresholds and strategies achieve zero error does not imply that the same is true for the kill probability parameter estimates. For example, in this particular run, the relative error $\Delta d_{11}$ converges to a nonzero value, i.e., the algorithm obtains a wrong estimate of $d_{11}$. However, the error is sufficiently small to still result in a zero-error estimate of the shooting thresholds.

As in Group A, to better evaluate the algorithm, we ran it $J=10$ times and averaged the obtained results. In particular, in Figure 9, we plot the average of ten curves of the type plotted in Figure 4. Note that we now plot the curve for $R=500$ plays of the duel.

We again ran the learning algorithm for various combinations of game parameters and recorded the average error attained at the end of the algorithm (again at $r=R=5000$) for each combination. The results are summarized in the following tables.

From Table 6 and Table 7, we observe that for most parameter combinations, all learning sessions concluded with a zero $\Delta Q_1$ for the smaller $\lambda$ values. However, for $\lambda =1.05$, the algorithm failed to converge and exhibited a high relative error. Notably, increasing $\lambda$ from 1.001 to 1.01 generally accelerated convergence, although this is not guaranteed in every case. In one instance, a higher $\lambda$ (specifically $1.01$) led to a slower average convergence of $\Delta Q_1$ to zero, indicating the importance of initial random shooting choices. We also observed that the results for the highest $\lambda$ were suboptimal, with the algorithm failing to converge in varying numbers of sessions, such as in 1 out of 10 or even 5 out of 10 cases. Notably, when convergence did occur, it happened relatively quickly, with sessions typically completing in under 1000 iterations. These findings also highlight the trade-off between exploration and exploitation, as discussed earlier.

Finally, in Table 8, we see how many learning sessions were run and how many converged to the zero-error estimate $\Delta Q_1$ for different values of D and $\lambda$.

6. Discussion

We proposed an algorithm for estimating unknown game parameters and optimal strategies for a duel game through the course of repeated plays. We tested an algorithm for two models of the kill probability function and found that it converged for the majority of tests. Furthermore, we observed the established relationship between higher learning rates and reduced convergence quality, underscoring the trade-off between learning speed and stability in convergence. Future research could investigate additional models of the accuracy probability function, including scenarios where the two players employ distinct models, and work toward establishing theoretical bounds on the algorithm’s probabilistic convergence.