Reactive Strategies: An Inch of Memory, a Mile of Equilibria

Abstract

We explore how an incremental change in complexity of strategies (“an inch of memory”) in repeated interactions influences the sets of Nash equilibrium (NE) strategy and payoff profiles. For this, we introduce the two most basic setups of repeated games, where players are allowed to use only reactive strategies for which a probability of players’ actions depends only on the opponent’s preceding move. The first game is trivial and inherits equilibria of the stage game since players have only unconditional (memory-less) Reactive Strategies (RSs); in the second one, players also have conditional stochastic RSs. This extension of the strategy sets can be understood as a result of evolution or learning that increases the complexity of strategies. For the game with conditional RSs, we characterize all possible NE profiles in stochastic RSs and find all possible symmetric games admitting these equilibria. By setting the unconditional benchmark as the least symmetric equilibrium payoff profile in memory-less RSs, we demonstrate that for most classes of symmetric stage games, infinitely many equilibria in conditional stochastic RSs (“a mile of equilibria”) Pareto dominate the benchmark. Since there is no folk theorem for RSs, Pareto improvement over the benchmark is the best one can gain with an inch of memory.

1. Introduction

In the theory of repeated games, restrictions on players’ strategies are not usually imposed (see, for example, [1]). However, an interest in bounded rationality and strategic complexity [2] has led to the study of repeated games where players’ strategies are assumed to be finite automata or to have finite memory (bounded recall). This paper falls into the strategy-restrictions framework: we make an extensive study of the Nash equilibrium in infinitely repeated $2\times 2$ games where payoffs are determined by Reactive Strategies (RSs) and players’ payoffs are evaluated according to the limit of means. A player’s reactive strategies are strategies that, for each move of the opponent in the previous round, prescribe stationary probabilities to the player of playing his/her own pure actions.

One may naturally think of two-by-two classification of RSs. The first aspect is related to information. Players may ignore the available information about the action of the opponent in the previous round; hence, we distinguish between unconditional and conditional RSs. The second aspect is related to predictability of actions. Deterministic and semi-deterministic RSs allow deterministic responses (probabilities 1 or 0) to actions in the previous rounds, while for stochastic RSs, all responses have non-degenerate probabilities.

The paper explores how an incremental change of complexity of strategies in repeated interactions influences the sets of Nash equilibrium (NE) strategy and payoff profiles. For this, we introduce, perhaps, the two most basic setups of repeated games. In the first game, which is trivial and inherits equilibria of the stage game, players have only memory-less RSs. In the second game, players in addition have conditional stochastic RSs. This extension of the strategy sets can be understood as a result of evolution or learning, resulting in increased complexity of strategies. For the game with conditional RSs, we answer the following questions:

(Q1)

What are all possible NE profiles in stochastic RSs?

(Q2)

What are all possible symmetric games admitting NE in stochastic RSs?

(Q3)

Do equilibrium profiles in conditional stochastic RSs Pareto improve over equilibrium profiles in unconditional RSs?

Surprisingly, the answers to (Q1)–(Q3), the most basic results for equilibria in reactive strategies, are still missing in the literature, while similar questions were studied for 1-memory (aka memory-one) strategies with discounted payoffs.

Before we proceed to more technical elements of the paper, let us give a simple example where reactive and 1-memory strategies meet together.

Example 1.

A textbook example of repeated games is the iterated Prisoner’s Dilemma, where the stage game has two strategies for both players, C (cooperate) and D (defect). Any round ends with a possible action profile: $CC,CD,DC,$ or $DD$. As profile i of the previous round is known, player 1 chooses C with probability $p_i$ (hence, D with probability $1-p_i$). Then a 4-tuple, say $(p_{CC},p_{CD},p_{DC},p_{DD}),$ would define a 1-memory strategy of player 1. A reactive strategy of player 1 is a 1-memory strategy that only depends on the opponent’s action, i.e., $p_{CC}=p_{CD}$ and $p_{DC}=p_{DD}$; this reduces 4-tuples to ordered pairs. By omitting the issue of opening moves, one derives 1-memory form for the strategies that were found to be the most popular in experimental study [3]. These strategies are Tit-for-Tat, $(1,0,1,0)$; Always Defect, $(0,0,0,0)$; both are reactive ones, while Grim, $(1,0,0,0)$, is not.

1.1. Related Literature

Reactive strategies were introduced by Karl Sigmund and Martin Nowak (see [4,5]) as an abstraction of bounded rationality. However, while their research has never been focused directly on the Nash equilibrium, we should note that the authors’ comprehensive study of evolutionary stable strategies is indeed relevant to (Q1), as the definition of these strategies requires a symmetric Nash equilibrium to be formed. This is why, for the infinitely repeated Prisoner’s Dilemma in [4], a full characterization of symmetric Nash equilibria in the class of reactive strategies was obtained for the limit-of-means payoff. Note that these symmetric equilibria are equilibrium points (see [6]) of adaptive dynamics on the set of all reactive strategies. The concept of adaptive dynamics was introduced in [7] to describe evolution in continuous sets of strategies. In this paper, we significantly extend the results from [4] related to the Nash equilibrium by considering arbitrary $2\times 2$ stage games and by characterizing both symmetric and non-symmetric equilibria, with non-symmetric equilibria prevailing over symmetric ones.

For the discounted Prisoner’s Dilemma, subgame perfect equilibria in the class of deterministic reactive strategies were studied in [8]. In [9], Appendix 5.5, a brief analysis can be found of equilibrium refinements in the class of deterministic reactive strategies (referred to as “memory zero” strategies) for a repeated Prisoner’s Dilemma. In [9], R. Aumann considered eight pure strategies for each player (four stationary reactions multiplied by two actions for the initial round). Note that the concepts of reactive strategies and 1-memory strategies in repeated games with a continuum of pure strategies are related to stationary “reaction functions” [10,11], “immediately reactive strategies” [12], and stationary “single-period-recall strategies” [13]. In contrast to the studies mentioned in this paragraph, we consider limit-of-means payoffs and reactive strategies mixing over (two) pure actions, that is, reactive strategies mapping the previous opponent’s moves into distributions over pure actions.

We also study the problem of the existence of a Nash equilibrium. Note that Kryazhimskiy [14] obtained conditions sufficient for the existence of a Nash equilibrium within subsets of the players’ 1-memory strategies in repeated bimatrix games with the limit-of-means payoff. The conditions require the sets of players’ strategies to have a convexity property and all strategies to be strictly randomized, so the Kakutani fixed-point theorem becomes applicable.

There is evidently much more literature available on the topic of 1-memory strategies than on the topic of reactive strategies. The recent spike of interest in 1-memory strategies was stimulated by the discovery of Zero-Determinant (ZD) strategies, a special class of 1-memory strategies enabling linear relationships between players’ payoffs, irrespective of opponent’s strategies (see [15] for the case of the limit-of-means payoff). Remarkably, ZD strategies can fix the opponent’s payoff or guarantee some surplus that outperforms the opponent’s surplus by the chosen constant factor. Hilbe et al. [16] extended the theory of ZD strategies to the case of the discounted payoff; see [16] for the most recent review of the topic.

Within the literature dealing with 1-memory strategies, one of the most relevant studies for our research was conducted by Dutta and Siconolfi [17]. In that study, the ideas, which were independently proposed in the seminal paper [18] to derive a robust folk theorem for the discounted Prisoner’s Dilemma, were extended to generic repeated $2\times 2$ games. In other words, Dutta and Siconolfi explored a Strong Mixed Equilibrium (SME) in the class of 1-memory strategies. This equilibrium is formed by completely mixed 1-memory strategies and has the salient property that both players are indifferent between their actions after every history. Specifically, in [17], conditions were obtained that are necessary and sufficient for a game to have an SME, and the set of all equilibria payoffs was described. The latter result allowed us to demonstrate that SMEs generally lead to a continuum of equilibrium payoffs but still do not generate a folk theorem. To summarize, Dutta and Siconolfi [17] studied subgame perfect equilibria formed by completely mixed (stochastic) strategies in repeated games with discounted payoffs and 1-memory strategies.

An analysis of subgame perfect equilibria (SPE) in 1-memory strategies focused on the Prisoner’s Dilemma was presented in [19]. In particular, SPEs formed by zero-determinant strategies were characterized and the corresponding upper bound for all NE and SPE payoffs was obtained (it is equal to the mutual cooperation payoff).

Moving from 1-memory to finite memory strategies, we should mention finite automata as a common abstraction of bounded rationality; see [2,20,21].

Following [22], we observe different natural levels of complexity, motivating us to distinguish between deterministic and Stochastic Reactive Strategies (SRSs). In this paper, we focus on SRSs as the simplest ones. The higher complexity of some deterministic strategies is due to the fact that the probability of the first action should be a part of the strategy to correctly define expected payoffs, while the payoffs for stochastic strategies do not depend on the first action.

To summarize, the field of finite-memory strategies (and, in particular, reactive strategies) has been studied for the last 30 years. Especially comprehensive results were obtained for 1-memory strategies but even the most basic facts for generic $2\times 2$ stage games are still missing for equilibria in RSs. The research not only delivers the missing piece of theory but also demonstrates that even an inch of memory in repeated interactions translates into a mile of new equilibria.

1.2. Results and Structure of the Article

First, we provide an expository geometrical representation to draw contrasts between unconditional and conditional SRSs in the repeated games; see Section 2.1. An example of the Prisoner’s Dilemma with equal gains from switching is considered in Section 2.2 to illustrate how the representation will be applied to study the Nash equilibrium.

Second, in Section 2.3, we derive a characterization of all Nash equilibria in the class of reactive strategies in terms of solutions to a system of equations and inequalities; see Theorem 1.

Third, in Section 2.4, Theorem 2, we present a characterization of all symmetric games admitting Nash equilibria in the class of reactive strategies; Appendix A contains all related proofs.

Fourth, we elaborate on equilibrium payoff profiles in conditional SRSs; see Section 3. Namely, we demonstrate that:

• If there exists an NE formed by a profile of conditional SRSs, then there are infinitely many NE profiles generated by conditional SRSs that, in general, have distinct payoffs, but we do not have a folk theorem.
• If there exists an NE formed by a profile of unconditional SRSs, then NE profiles in conditional SRSs either Pareto improve over it or provide the same payoff profile.

For symmetric games, the complete analysis of equilibrium payoff profiles becomes feasible in Section 3.2 and Appendix B. In our setting, an NE in unconditional RSs always exists. Among equilibrium payoff profiles in unconditional RSs, we select the symmetric one with the lowest payoffs as the benchmark. We then check if there exists an equilibrium payoff profile formed by conditional stochastic RSs that Pareto dominates this benchmark. Only in the very special games where both players (by choosing the dominating pure strategy) get the highest possible individual payoff are payoff profiles in conditional SRSs dominated by the benchmark unconditional payoff profiles. This means that it is possible to get new equilibria with lower payoffs while the strategies become more advanced; see an example in Section 3.3. In the remaining cases, if equilibrium profiles in SRSs exist and result in distinct payoff profiles, then there exists an NE profile in SRSs dominating the benchmark. In some cases, all NE profiles in conditional SRSs dominate the benchmark.

Fifth, with the extensive use of examples, we highlight some interesting properties of NE profiles in stochastic RSs; see Appendix C.

In this paper, some reactive strategies are excluded from players’ strategy sets. In Section 2.5, we discuss how this decision influences the results.

1.3. Definitions of Repeated Games

Consider an arbitrary one-shot game $\mathbf{G}$

The first player chooses rows (possible strategies are T and B); the second player chooses columns (possible strategies are L and R); $T,B,L,R$ stand for top, bottom, left, right, respectively. A mixed strategy of player 1 is a probability $s_1$ of playing action $T;$ similarly, a mixed strategy $s_2$ of player 2 is a probability of playing action $L.$ Mixed strategies defined by non-degenerate probabilities are called completely mixed strategies. Let us introduce payoff functions ${\mathbf{J}}_1,{\mathbf{J}}_2:{[0,1]}^2\to \mathbb{R}$ for the one-shot game $\mathbf{G}$ in mixed strategies by the rules: $\forall s_1,s_2\in [0,1]$

$${\mathbf{J}}_1(s_1,s_2)\stackrel{▵}{=}{a}_1{s}_1{s}_2+b_1{s}_1+c_1{s}_2+A_{B,R},{\mathbf{J}}_2(s_1,s_2)\stackrel{▵}{=}{a}_2{s}_1{s}_2+b_2{s}_2+c_2{s}_1+B_{B,R},$$

(1)

where

$$\begin{array}{c}\hfill a_1\stackrel{▵}{=}{A}_{T,L}-A_{T,R}-A_{B,L}+A_{B,R},b_1\stackrel{▵}{=}{A}_{T,R}-A_{B,R},c_1\stackrel{▵}{=}{A}_{B,L}-A_{B,R};\end{array}$$

(2)

$$\begin{array}{c}\hfill a_2\stackrel{▵}{=}{B}_{T,L}-B_{T,R}-B_{B,L}+B_{B,R},b_2\stackrel{▵}{=}{B}_{B,L}-B_{B,R},c_2\stackrel{▵}{=}{B}_{T,R}-B_{B,R}.\end{array}$$

(3)

1.3.1. Strategies

In the following, we study a game ${\mathbf{G}}^{\infty }$ that is a repeated modification of $\mathbf{G}.$ To define ${\mathbf{G}}^{\infty }$ in normal form, we first introduce reactive strategies [4,5,23].

Definition 1.

We define reactive strategies of player 1 (player 2) as arbitrary maps of $\{L,R\}$ (maps of $\{T,B\}$) into sets of all mixed strategies of player 1 (player 2); players’ mixed strategies in the current round are completely defined by the preceding opponent’s action. To simplify the notation, we understand any reactive strategy of player 1 as an ordered pair $u=(u_1,u_2)$ such that the conditional probability of action T and action B is $u_1$ and $(1-u_1),$ given that the second player’s previous move was L, and the conditional probability of T and B is $u_2$ and $(1-u_2),$ given that the second player’s previous move was $R.$ In a similar way, a reactive strategy of player 2 is a pair $v=(v_1,v_2)$ such that the conditional probability of action L and action R is $v_1$ and $(1-v_1),$ given that the first player’s previous move was T, and the conditional probability of L and R is $v_2$ and $(1-v_2),$ given that the first player’s previous move was $B.$

Reactive strategies form a subset of 1-memory strategies (see [15,16,17,18]). Figure 1 shows the classification of RSs that we follow in this paper.

To avoid ambiguity, we denote an open interval $\{x:a<x<b\}$ as $]a,b[$. For player 1 (player 2), we introduce the set of Stochastic Reactive Strategies (SRSs): $U\stackrel{▵}{=}{]0,1[}^2$ ( $V\stackrel{▵}{=}{]0,1[}^2$).

We start with the repeated setting where players use only unconditional strategies (both deterministic and stochastic), game ${\mathbf{G}}_{un}^{\infty }.$ Game ${\mathbf{G}}_{un}^{\infty }$ is basically memory-less infinite repetition of the stage game. Then we slightly increase the complexity of players’ behavior; i.e., we allow players to use also SRSs. This changes ${\mathbf{G}}_{un}^{\infty }$ into ${\mathbf{G}}^{\infty }.$ Thus, the set of all possible strategies of player 1 (of player 2) in game ${\mathbf{G}}^{\infty }$ is defined as $\tilde{U}\stackrel{▵}{=}U\cup \{(0,0),(1,1)\}$ (as $\tilde{V}\stackrel{▵}{=}U\cup \{(0,0),(1,1)\}$).

1.3.2. Payoffs

Given any initial action profile $(i_0,j_0)$ corresponding to the first round and a profile of stochastic RSs, we obtain an infinite discrete-time Markov stochastic process (hence the name for strategies) defined on four possible states: $(T,L),$$(T,R),$$(B,L),$ and $(B,R)$ (states 1, 2, 3, and 4, correspondingly). For every actual l-step trajectory of actions $(i_1,j_1)\to (i_2,j_2)\to \dots \to (i_l,j_l),$ the average payoffs are random variables. According to [14], for $l\to \infty ,$ the limits of expected l-round average payoffs are well defined for all SRSs and initial profiles. As usual, we call these limits limit-of-means payoffs.

The paper studies equilibrium profiles formed only by SRSs due to two reasons:

1.

In contrast to semi-deterministic and deterministic RSs, the payoffs for profiles of SRSs do not depend on the opening move;

2.

SRSs capture non-deterministic behavior that is the most natural for the domain of evolutionary game theory, where RSs originated to model real-life processes.

Every profile $(u,v)$ of SRSs ensures nice properties for the induced Markov process with the transition probability matrix

$$M=\left(\begin{array}{cccc}{u}_1{v}_1& u_1(1-v_1)& (1-u_1)v_1& (1-u_1)(1-v_1)\\ u_2{v}_1& u_2(1-v_1)& (1-u_2)v_1& (1-u_2)(1-v_1)\\ u_1{v}_2& u_1(1-v_2)& (1-u_1)v_2& (1-u_1)(1-v_2)\\ u_2{v}_2& u_2(1-v_2)& (1-u_2)v_2& (1-u_2)(1-v_2)\end{array}\right).$$

That is, there exists a stationary distribution $\pi$ (a row vector) such that $\pi =\pi M;$ note that $\pi$ depends on u and v but not on the initial profile. In [4,5,23], it was proven that stationary distribution $\pi :\tilde{U}\times \tilde{V}\to {\mathbb{R}}^4$ allows the following representation:

$$\pi (·)=({\mathbf{s}}_1(·){\mathbf{s}}_2(·),{\mathbf{s}}_1(·)(1-{\mathbf{s}}_2(·)),(1-{\mathbf{s}}_1(·)){\mathbf{s}}_2(·),(1-{\mathbf{s}}_1(·))(1-{\mathbf{s}}_2(·)))$$

where mappings ${\mathbf{s}}_1:\tilde{U}\times \tilde{V}\to [0,1],{\mathbf{s}}_2:\tilde{U}\times \tilde{V}\to [0,1]$ are defined by the rules

$${\mathbf{s}}_1(u,v)\stackrel{▵}{=}\frac{v_2(u_1-u_2)+u_2}{1-(u_1-u_2)(v_1-v_2)},{\mathbf{s}}_2(u,v)\stackrel{▵}{=}\frac{u_2(v_1-v_2)+v_2}{1-(u_1-u_2)(v_1-v_2)}.$$

(4)

Note that the trajectories of actions for profiles of deterministic RSs contain a deterministic cycles (hence the name of the strategies) of action profiles that may depend on the opening moves.

In what follows, we omit arguments of ${\mathbf{s}}_1,{\mathbf{s}}_2$ where it is possible. We also refer to $({\mathbf{s}}_1,{\mathbf{s}}_2)$ as a stationary distribution, since the original stationary distribution $\pi$ can be easily recovered.

For a profile of SRSs, the limit-of-means payoffs are determined by the stationary distribution that is independent from the initial actions. Hence, for a profile $(u,v)$ of SRSs, players’ payoffs in ${\mathbf{G}}^{\infty }$ are the exact same payoffs as in $\mathbf{G}$ if player 1 uses mixed strategy ${\mathbf{s}}_1(u,v)$ and player 2 uses mixed strategy ${\mathbf{s}}_2(u,v).$ This helps to specify ${\mathbf{J}}_1^{\infty }$ and ${\mathbf{J}}_2^{\infty }:$$(u,v)\in \tilde{U}\times \tilde{V}$ that denote the first and second player’s payoff function in ${\mathbf{G}}^{\infty },$ respectively:

$${\mathbf{J}}_1^{\infty }(u,v)={\mathbf{J}}_1({\mathbf{s}}_1(u,v),{\mathbf{s}}_2(u,v)),{\mathbf{J}}_2^{\infty }(u,v)={\mathbf{J}}_2({\mathbf{s}}_1(u,v),{\mathbf{s}}_2(u,v)).$$

(5)

See the summary for all introduced games in

Table 1. Summary for two-player games considered in the paper.

Game	Setting	Strategies	Payoffs	Description
$\mathbf{G}$	One-shot	Mixed strategies $([0,1],[0,1])$	${\mathbf{J}}_1,{\mathbf{J}}_2$	A one-shot $2\times 2$ game
${\mathbf{G}}_{un}^{\infty }$	Repeated	Unconditional RSs	${\mathbf{J}}_1^{\infty },{\mathbf{J}}_2^{\infty }$	the memory-less play of $\mathbf{G}$ that is infinitely repeated; ${\mathbf{G}}_{un}^{\infty }$ is ‘equivalent’ to $\mathbf{G}$ but formalised as repeated interaction.
${\mathbf{G}}^{\infty }$	Repeated	Stochastic and unconditional RSs $(\tilde{U},\tilde{V})$	${\mathbf{J}}_1^{\infty },{\mathbf{J}}_2^{\infty }$	The repeated modification of $\mathbf{G}$ where probabilities of actions can be conditioned by the preceding opponent’s action. In addition to memory-less strategies from ${\mathbf{G}}_{un}^{\infty }$, players get conditional ones.

.

1.3.3. Equilibria

To summarize, we study a game ${\mathbf{G}}^{\infty }$ with two players; the sets of strategies are $\tilde{U}$ and $\tilde{V};$ the payoff functions are ${\mathbf{J}}_1^{\infty }$ and ${\mathbf{J}}_2^{\infty }.$

Definition 2.

A profile $(\widehat{u},\widehat{v})\in U\times V$ is called a Nash equilibrium (NE) in ${\mathbf{G}}^{\infty }$ if the following conditions hold: $\forall (u,v)\in \tilde{U}\times \tilde{V}$

$${\mathbf{J}}_1^{\infty }(u,\widehat{v})\le {\mathbf{J}}_1^{\infty }(\widehat{u},\widehat{v})\mathrm{and}{\mathbf{J}}_2^{\infty }(\widehat{u},v)\le {\mathbf{J}}_2^{\infty }(\widehat{u},\widehat{v}).$$

(6)

In what follows, a pure (completely mixed) Nash equilibrium stands for a Nash equilibrium in a one-shot game $\mathbf{G}$ formed by pure (completely mixed) strategies. In ${\mathbf{G}}^{\infty },$ the stationary distribution generated by an NE is called an Equilibrium Stationary Distribution (ESD).

2. Characterization of Nash Equilibria

2.1. Geometric Intuition and Attainable Sets

Our immediate goal is to illustrate for ${\mathbf{G}}^{\infty }$ the difference between conditional and unconditional SRSs. First, let us introduce the concept of attainable sets.

Definition 3.

The set of all stationary distributions feasible for a fixed opponent’s reactive strategy is called an attainable set for a player. To be precise, $\forall u\in U,v\in V$

$${\mathcal{S}}_I(v)\stackrel{▵}{=}\{({\mathbf{s}}_1(\overline{u},v),{\mathbf{s}}_2(\overline{u},v)):\overline{u}\in U\},{\mathcal{S}}_{II}(u)\stackrel{▵}{=}\{({\mathbf{s}}_1(u,\overline{v}),{\mathbf{s}}_2(u,\overline{v})):\overline{v}\in V\}.$$

Here, ${\mathcal{S}}_I(v)$ is an attainable set for player 1 if player 2 chooses strategy $v;$ see Figure 2b. Similarly, ${\mathcal{S}}_{II}(u)$ is an attainable set for player 2 if player 1 chooses $u;$ see Figure 2a. Obviously, $\left\{({\mathbf{s}}_1(u,v),{\mathbf{s}}_2(u,v))\right\}={\mathcal{S}}_I(v)\cap {\mathcal{S}}_{II}(u);$ see Figure 2c. Using (4), we obtain

$${\mathbf{s}}_1(u,v)=u_2+{\mathbf{s}}_2(u,v)(u_1-u_2),{\mathbf{s}}_2(u,v)=v_2+{\mathbf{s}}_1(u,v)(v_1-v_2).$$

(7)

Lemma 1.

The following representations hold true $\forall u\in U\forall v\in V$

$$\begin{array}{c}\hfill \begin{array}{c}\hfill {\mathcal{S}}_I(v)=\{(s_1,s_1(v_1-v_2)+v_2):s_1\in ]0,1[\},\\ \hfill {\mathcal{S}}_{II}(u)=\{(s_2(u_1-u_2)+u_2,s_2):s_2\in ]0,1[\}.\end{array}\end{array}$$

(8)

Proof. 

We will check only the first representation, as, using similar arguments, it easy to demonstrate the second one. Fix an arbitrary $v\in V;$ let

$$\Omega \stackrel{▵}{=}\{(s_1,s_1(v_1-v_2)+v_2):s_1\in ]0,1[\}.$$

Take ${\acute{s}}_1\in ]0,1[;$ then $({\acute{s}}_1,{\acute{s}}_1(v_1-v_2)+v_2)\in \Omega .$ Let $\acute{u}$ be the reactive strategy of player 1 such that ${\acute{u}}_1={\acute{u}}_2={\acute{s}}_1.$ It follows from (7) that ${\mathbf{s}}_1(\acute{u},v)={\acute{s}}_1$ and ${\mathbf{s}}_2(\acute{u},v)={\acute{s}}_1(v_1-v_2)+v_2.$ Thus, $({\mathbf{s}}_1(\acute{u},v),{\mathbf{s}}_2(\acute{u},v))=({\acute{s}}_1,{\acute{s}}_1(v_1-v_2)+v_2).$ Then $({\acute{s}}_1,{\acute{s}}_1(v_1-v_2)+v_2)\in {\mathcal{S}}_I(v);$ therefore,

$$\Omega \subset {\mathcal{S}}_I(v).$$

(9)

Fix ù $\in U;$ then ${\mathbf{s}}_1(ù,v)\in ]0,1[.$ It follows from (7) that ${\mathbf{s}}_2(ù,v)={\mathbf{s}}_1(ù,v)(v_1-v_2)+v_2.$ Then $({\mathbf{s}}_1(ù,v),{\mathbf{s}}_2(ù,v))\in \Omega ;$ thus, ${\mathcal{S}}_I(v)\subset \Omega .$ Combining this with (9), we obtain $\Omega ={\mathcal{S}}_I(v).$ □

Remark 1.

Equation (7) implies that for player 1 to achieve any element of ${\mathcal{S}}_I(v)\forall v\in V,$ say $(s_1^{*},s_2^{*})\in {\mathcal{S}}_I(v),$ she may use the corresponding unconditional RS $(s_1^{*},s_1^{*})$; for player 2, the similar statement holds true. Figure 3a shows us the corresponding example.

We now see the key difference between unconditional and conditional SRSs. For a memory-less strategy of the first player (i.e., $u_1=u_2$), the attainable set of the second player is a horizontal line. Similarly, for a memory-less strategy of the second player (i.e., $v_1=v_2$), the attainable set of player 1 is a vertical line; this is similar for mixed strategies in one-shot games. Hence, memory-less strategies only allow intercepts to be chosen. By contrast, conditional SRSs (i.e., $u_1\ne u_2$ and $v_1\ne v_2$) also allow slopes of attainable sets to be set. Thus, conditional SRSs have more flexibility in adjusting slopes of attainable sets to gradients of payoff functions at the stationary distributions (the points where players’ attainable sets intersect). This flexibility provides significantly more opportunities for equilibria to emerge (compared to memory-less mixed strategies).

Proposition 1.

For a one-shot game $\mathbf{G}$, any NE profile $(u,v)$ in mixed strategies becomes the corresponding NE profile $((u,u),(v,v))$ in both ${\mathbf{G}}^{\infty }$ and ${\mathbf{G}}_{un}^{\infty }$.

Proof. 

Fix the memory-less RS $(u,u)$ of player 1 corresponding to an equilibrium $(u,v)$ in a one-shot game $\mathbf{G}$. We will show that $(v,v)$ is a best response to $(u,u)$ in ${\mathbf{G}}^{\infty }$. By Remark 1, for player 2, deviations in unconditional strategies are “operationally equivalent” to deviations in conditional strategies – that is, if player 2 can deviate, then he can deviate with a strategy of the form $(\tilde{v},\tilde{v})$. If player 2 picks an unconditional RS $(\tilde{v},\tilde{v}),\tilde{v}\in [0,1],$ we arrive at the stationary distribution $(u,\tilde{v}).$ Combining the last fact with (5), we obtain that

$${\mathbf{J}}_2^{\infty }((u,u),(\tilde{v},\tilde{v}))={\mathbf{J}}_2(u,\tilde{v})\forall \tilde{v}\in [0,1].$$

(10)

From the definition of NE in mixed strategies in $\mathbf{G}$, it follows that ${\mathbf{J}}_2(u,·),$ as a function defined on $[0,1],$ attains its maximum at $v.$ By (10) and the above argument on the operational equivalence,

$$\underset{\widehat{v}\in [0,1]\times [0,1]}{max}{\mathbf{J}}_2^{\infty }((u,u),\widehat{v})=\underset{\tilde{v}\in [0,1]}{max}{\mathbf{J}}_2^{\infty }((u,u),(\tilde{v},\tilde{v}))=\underset{\tilde{v}\in [0,1]}{max}{\mathbf{J}}_2(u,\tilde{v})={\mathbf{J}}_2^{\infty }((u,u),(v,v))$$

Thus, unconditional RS $(v,v),$ which emerges from the stage game NE, “remains” the best reply to $(u,u)$ in ${\mathbf{G}}^{\infty }$ (and hence in ${\mathbf{G}}_{un}^{\infty }$). Symmetrically, $(u,u)$ is a best response to $(v,v),$ and hence the profile is an NE in ${\mathbf{G}}^{\infty }$ (hence, in ${\mathbf{G}}_{un}^{\infty }$). □

Remark 2.

Note that players are free to move within their attainable sets from one point (stationary distribution) to another in a variety of ways. Moreover, for a stationary distribution within an attainable set of one player, there are infinitely many strategies leading to this stationary distribution for the given opponent’s strategy. In turn, the choice among these strategies leads to distinct attainable sets for the opponent. For example, Figure 3 shows the set of all possible strategy profiles leading to the stationary distribution $(s_1,s_2)=(1/2,2/3).$ Figure 3a depicts all possible slopes of players’ attainable sets. Note that the lines circumscribing the shaded regions have simple analytic representations. This allows us to infer all possible players’ strategies generating this stationary distribution. Namely, any pair of u from Figure 3c and v from Figure 3c leads to the same stationary distribution $(1/2,2/3)$. All $(u,v)$ such that the stationary distribution equals $(s_1,s_2)=(1/2,2/3)$ induce the attainable sets inside the shaded regions.

2.2. Prisoner’s Dilemma with Equal Gains from Switching

We consider $G^{eg}$, a Prisoner’s Dilemma with equal gains from switching (see [4]),

Equal gains from switching implies that $a_1=a_2=0;$ clearly, we have $b_1=b_2=-1,c_1=c_2=2.$ For the repeated version of $G^{eg},$ we have (see (1) and (5))

$${\mathbf{J}}_1^{\infty }(u,v)\stackrel{▵}{=}-{\mathbf{s}}_1(u,v)+2{\mathbf{s}}_2(u,v)+2,{\mathbf{J}}_2^{\infty }(u,v)\stackrel{▵}{=}-{\mathbf{s}}_2(u,v)+2{\mathbf{s}}_1(u,v)+2.$$

Lemma A2 tells us that every pair of well-defined SRSs such that $u^{*}=(u_1^{*},u_2^{*})=(u_2^{*}-\frac{b_1}{c_1},u_2^{*})$ and $v^{*}=(v_1^{*},v_2^{*})=(v_2^{*}-\frac{b_1}{c_1},v_2^{*})$ forms a Nash equilibrium. Figure 4 illustrates that for player 1, curves-of-equal-payoff coincide with attainable sets if player 2 uses the above equilibrium strategies, meaning that equilibrium strategies fix the opponent’s payoff:

$${\mathbf{J}}_1^{\infty }(u,v^{*})=2v_2^{*}+2\forall u\in U,{\mathbf{J}}_2^{\infty }(u^{*},v)=2u_2^{*}+2\forall v\in V.$$

The reactive strategy $(1,1+\frac{b_1}{c_1})$ is known as generous tit-for-tat [24]; it leads to the most cooperative symmetric equilibrium.

Note that an equilibrium can only be sustained if both players choose strategies u and v such that $u_1>u_2$ and $v_1>v_2.$ Value $v_1-v_2$ directly encourages player 1 to cooperate. Specifically, if $v_1-v_2>-\frac{b}{c}=1/2,$ then player 1 has an incentive to climb the attainable set by choosing more cooperative strategies increasing $s_1$ and $s_2$. If $v_1-v_2<1/2,$ then player 1 has an incentive to step down the attainable set; this decreases $s_1$ and $s_2$. Clearly, memory-less SRSs can not support Nash equilibria. For example, the attainable sets of player 1 become vertical lines, giving her clear incentives to defect since her payoff increases moving from top to bottom of the attainable sets. Note that, generally, curves-of-equal-payoff are non-linear; see the example in Appendix C.1.

2.3. Characterization of Nash Equilibria in $\mathbf{G}$${}^{\infty }$

Since every NE is formed by strategies that are best responses to each other, we proceed further with a standard analysis using derivatives. Let $f_x$ stand for the derivative of a function $f(·)$ w.r.t. $x.$

Lemma 2

(The necessary condition). If $\widehat{u}=(\widehat{u_1},{\widehat{u}}_2)\in U,\widehat{v}=({\widehat{v}}_1,{\widehat{v}}_2)\in V$ form a Nash equilibrium and $({\widehat{s}}_1,{\widehat{s}}_2)$ is the corresponding ESD, then the following holds true:

$$\begin{array}{ccc}\hfill a_1(2({\widehat{v}}_1-{\widehat{v}}_2){\widehat{s}}_1+{\widehat{v}}_2)+b_1+c_1({\widehat{v}}_1-{\widehat{v}}_2)=0& ,& 2a_1({\widehat{v}}_1-{\widehat{v}}_2)\le 0,\hfill \\ \hfill a_2(2({\widehat{u}}_1-{\widehat{u}}_2){\widehat{s}}_2+{\widehat{u}}_2)+b_2+c_2(u_1-{\widehat{u}}_2)=0& ,& 2a_2({\widehat{u}}_1-u_2)\le 0.\hfill \end{array}$$

Proof. 

Assume that $(\widehat{u},\widehat{v})$ is a Nash equilibrium with the corresponding ESD $({\widehat{s}}_1,{\widehat{s}}_2).$ Combining (5), (6), and (8), we have

$$\begin{array}{c}\hfill \begin{array}{c}\hfill \underset{u\in U}{max}{\mathbf{J}}_1^{\infty }(u,\widehat{v})=\underset{u\in U}{max}{\mathbf{J}}_1({\mathbf{s}}_1(u,\widehat{v}),{\mathbf{s}}_2(u,\widehat{v}))=\underset{(s_1,s_2)\in {\mathcal{S}}_I(\widehat{v})}{max}{\mathbf{J}}_1(s_1,s_2)=\\ \hfill \underset{s_1\in ]0,1[}{max}{\mathbf{J}}_1(s_1,s_1({\widehat{v}}_1-{\widehat{v}}_2)+{\widehat{v}}_2)={\mathbf{J}}_1({\widehat{s}}_1,{\widehat{s}}_2),\end{array}\end{array}$$

(11)

$$\begin{array}{c}\hfill \begin{array}{c}\hfill \underset{v\in V}{max}{\mathbf{J}}_2^{\infty }(\widehat{u},v)=\underset{v\in V}{max}{\mathbf{J}}_2({\mathbf{s}}_1(\widehat{u},v),{\mathbf{s}}_2(\widehat{u},v))=\underset{(s_1,s_2)\in {\mathcal{S}}_{II}(\widehat{v})}{max}{\mathbf{J}}_2(s_1,s_2)=\\ \hfill \underset{s_2\in ]0,1[}{max}{\mathbf{J}}_2(s_2({\widehat{u}}_1-{\widehat{u}}_2)+{\widehat{u}}_2,s_2)={\mathbf{J}}_2({\widehat{s}}_1,{\widehat{s}}_2).\end{array}\end{array}$$

(12)

It follows from (1)–(3) that

$${\mathbf{J}}_1(s_1,s_1({\widehat{v}}_1-{\widehat{v}}_2)+{\widehat{v}}_2)=a_1({\widehat{v}}_1-{\widehat{v}}_2)s_1^2+s_1(b_1+c_1({\widehat{v}}_1-{\widehat{v}}_2)+a_1{\widehat{v}}_2)+c_1{\widehat{v}}_2+A_{B,R},$$

$${\mathbf{J}}_2(s_2({\widehat{u}}_1-{\widehat{u}}_2)+{\widehat{u}}_2,s_2)=a_2({\widehat{u}}_1-{\widehat{u}}_2)s_2^2+s_2(b_2+c_2({\widehat{u}}_1-{\widehat{u}}_2)+a_2{\widehat{u}}_2)+c_2{\widehat{u}}_2+B_{B,R}.$$

Define the functions ${\mathbf{J}}_1^{*}[\widehat{v}],{\mathbf{J}}_2^{*}[\widehat{u}]:[0,1]\to \mathbb{R}$ by the rules: $\forall s_1,s_2\in [0,1]$

$${\mathbf{J}}_1^{*}[\widehat{v}](s_1)\stackrel{▵}{=}{a}_1({\widehat{v}}_1-{\widehat{v}}_2)s_1^2+s_1(b_1+c_1({\widehat{v}}_1-{\widehat{v}}_2)+a_1{\widehat{v}}_2)+c_1{\widehat{v}}_2+A_{B,R},$$

(13)

$${\mathbf{J}}_2^{*}[\widehat{u}](s_2)\stackrel{▵}{=}{a}_2({\widehat{u}}_1-{\widehat{u}}_2)s_2^2+s_2(b_2+c_2({\widehat{u}}_1-{\widehat{u}}_2)+a_2{\widehat{u}}_2)+c_2{\widehat{u}}_2+B_{B,R}.$$

We now obtain the first and second derivatives of these functions:

$${\mathbf{J}}_1^{*}{[\widehat{v}]}_{s_1}=a_1(2({\widehat{v}}_1-{\widehat{v}}_2)s_1+{\widehat{v}}_2)+b_1+c_1({\widehat{v}}_1-{\widehat{v}}_2),$$

(14)

$${\mathbf{J}}_1^{*}{[\widehat{v}]}_{s_1,s_1}=2a_1({\widehat{v}}_1-{\widehat{v}}_2),$$

(15)

$${\mathbf{J}}_2^{*}{[\widehat{u}]}_{s_2}=a_2(2({\widehat{u}}_1-{\widehat{u}}_2)s_2+{\widehat{u}}_2)+b_2+c_2({\widehat{u}}_1-{\widehat{u}}_2),$$

(16)

$${\mathbf{J}}_2^{*}{[\widehat{u}]}_{s_2,s_2}=2a_2({\widehat{u}}_1-{\widehat{u}}_2).$$

(17)

Using (11)–(17) and the second derivative test, we complete the proof. □

Theorem 1

(Characterization of all Nash equilibria). The set of all Nash equilibria coincides with the set of all $\widehat{u},\widehat{v}\in {\mathbb{R}}^2$ solving the following system:

$$\left\{\begin{array}{c}{a}_1(2({\widehat{v}}_1-{\widehat{v}}_2){\widehat{s}}_1+{\widehat{v}}_2)+b_1+c_1({\widehat{v}}_1-{\widehat{v}}_2)=0(\mathrm{i})\\ a_2(2({\widehat{u}}_1-{\widehat{u}}_2){\widehat{s}}_2+{\widehat{u}}_2)+b_2+c_2({\widehat{u}}_1-{\widehat{u}}_2)=0(\mathrm{ii})\\ a_1({\widehat{v}}_1-{\widehat{v}}_2)\le 0,a_2({\widehat{u}}_1-{\widehat{u}}_2)\le 0(\mathrm{iii})\\ {\widehat{s}}_1={\mathbf{s}}_1(\widehat{u},\widehat{v}),{\widehat{s}}_2={\mathbf{s}}_2(\widehat{u},\widehat{v}),(\mathrm{iv})\\ 0<{\widehat{u}}_1,{\widehat{u}}_2,{\widehat{v}}_1,{\widehat{v}}_2<1(\mathrm{v})\end{array}\right..$$

(18)

Proof. 

Let $\mathbf{N}$ denote the set of all Nash equilibria in ${\mathbf{G}}^{\infty };$$\mathbf{S}$ stands for the set of all $(\widehat{u},\widehat{v})$ solving (18). Let us prove that $\mathbf{S}=\mathbf{N}.$

Take $(\widehat{u},\widehat{v})\in \mathbf{S}.$ As (v) in (18) holds, $(\widehat{u},\widehat{v})$ is a pair of reactive strategies; $\mathbf{S}\subset U\times V$. Obviously, (iv) ensures that $({\widehat{s}}_1,{\widehat{s}}_2)$ is the corresponding stationary distribution; $({\widehat{s}}_1,{\widehat{s}}_2)=({\mathbf{s}}_1(\widehat{u},\widehat{v}),{\mathbf{s}}_2(\widehat{u},\widehat{v})).$ As (iii) in (18) holds true, we have three cases:

1.

$a_1({\widehat{v}}_1-{\widehat{v}}_2)<0$ and $a_2({\widehat{u}}_1-{\widehat{u}}_2)<0,$

2.

$a_1({\widehat{v}}_1-{\widehat{v}}_2)=0$ and $a_2({\widehat{u}}_1-{\widehat{u}}_2)<0$ (or, symmetrically, $a_1({\widehat{v}}_1-{\widehat{v}}_2)<0$ and $a_2({\widehat{u}}_1-{\widehat{u}}_2)=0$),

3.

$a_1({\widehat{v}}_1-{\widehat{v}}_2)=0$ and $a_2({\widehat{u}}_1-{\widehat{u}}_2)=0.$

All cases are composed from similar elements: equalities and strong inequalities. We consider only two of these elements. The analysis for the others is similar. Suppose $a_1({\widehat{v}}_1-{\widehat{v}}_2)<0.$ Then ${\mathbf{J}}_1^{*}[\widehat{v}](·)$ is a concave quadratic function (see (13)). Conditions (i) and (iii) ensure that ${\widehat{s}}_1$ is a local maximum of ${\mathbf{J}}_1^{*}[\widehat{v}](·);$ see (14), (15). As local maxima of concave functions are also global maxima, we have $\begin{array}{c}\hfill \begin{array}{c}\hfill \underset{{\overline{s}}_1\in ]0,1[}{max}{\mathbf{J}}_1({\overline{s}}_1,{\overline{s}}_1({\widehat{v}}_1-{\widehat{v}}_2)+{\widehat{v}}_2)=\underset{{\overline{s}}_1\in ]0,1[}{max}{\mathbf{J}}_1^{*}[\widehat{v}]({\overline{s}}_1)={\mathbf{J}}_1^{*}[\widehat{v}]({\widehat{s}}_1)=\\ \hfill {\mathbf{J}}_1({\widehat{s}}_1,{\widehat{s}}_1({\widehat{v}}_1-{\widehat{v}}_2)+{\widehat{v}}_2)={\mathbf{J}}_1({\widehat{s}}_1,{\widehat{s}}_2)=\underset{\overline{u}\in U}{max}{\mathbf{J}}_1^{\infty }({\mathbf{s}}_1(\overline{u},\widehat{v}),{\mathbf{s}}_2(\overline{u},\widehat{v})).\end{array}\end{array}$

Assume $a_1({\widehat{v}}_1-{\widehat{v}}_2)=0.$ Thus, $a_1=0$ and/or ${\widehat{v}}_1={\widehat{v}}_2.$ If ${\widehat{v}}_1={\widehat{v}}_2,$ then player 2 uses the repeated version of her mixed Nash equilibrium strategy. If $a_1=0,$ then it follows from (13) and (i) in (18) that $\forall s_1\in [0,1]$ ${\mathbf{J}}_1^{*}[\widehat{v}](s_1)=c_1{\widehat{v}}_2+A_{B,R}.$ We have already observed this effect in the example of Section 2.2. Thus, in all cases, player 1 has a payoff independent of her strategy.

Generalizing the last two paragraphs, from (11) and (12), we obtain that

$$\begin{array}{c}\hfill \underset{\overline{u}\in U}{max}{\mathbf{J}}_1^{\infty }({\mathbf{s}}_1(\overline{u},\widehat{v}),{\mathbf{s}}_2(\overline{u},\widehat{v}))={\mathbf{J}}_1({\widehat{s}}_1,{\widehat{s}}_2)={\mathbf{J}}_1^{\infty }({\mathbf{s}}_1(\widehat{u},\widehat{v}),{\mathbf{s}}_2(\widehat{u},\widehat{v})),\\ \hfill \underset{\overline{v}\in V}{max}{\mathbf{J}}_2^{\infty }({\mathbf{s}}_1(\widehat{u},\overline{v}),{\mathbf{s}}_2(\widehat{u},\overline{v}))={\mathbf{J}}_2({\widehat{s}}_1,{\widehat{s}}_2)={\mathbf{J}}_2^{\infty }({\mathbf{s}}_1(\widehat{u},\widehat{v}),{\mathbf{s}}_2(\widehat{u},\widehat{v})).\end{array}$$

Consequently, $\widehat{u}$ and $\widehat{v}$ are the best responses to each other, meaning that

$$\mathbf{S}\subset \mathbf{N}.$$

(19)

Take $(u,v)\in \mathbf{N}.$ Trivially, (v) holds true. Taking into account (4) and Lemma 2, we see that (i)–(iv) are fulfilled. Thus, $\mathbf{N}\subset \mathbf{S};$ from (19), we have $\mathbf{S}=\mathbf{N}.$ □

2.4. Existence of Nash Equilibria in Symmetric Games

In this section, we obtain conditions for the existence of NE when players are identical (one-shot games are symmetric). We fix a symmetric game $\mathbf{G}$

Recall that for $\mathbf{G}$ we have the following versions of (2) and (3):

$$a_1=A_1-A_2-A_3+A_4,b_1=A_2-A_4,c_1=A_3-A_4;$$

$$a_2=A_1-A_2-A_3+A_4,b_2=A_2-A_4,c_2=A_3-A_4.$$

This allows us to introduce the following convenient definitions:

$$a\stackrel{▵}{=}{a}_1=a_2,b\stackrel{▵}{=}{b}_1=b_2,c\stackrel{▵}{=}{c}_1=c_2.$$

(20)

If $a\ne 0,$ then

$$\mathbf{b}\stackrel{▵}{=}-\frac{b}{a},\mathbf{c}\stackrel{▵}{=}-\frac{c}{a}.$$

(21)

To reach the goal of the section, we must derive conditions on $a,b,$ and c that ensure the existence of a solution of (18). Taking into account inequality (iii) in (18), in Appendix A, we study three possible cases: $a=0,a<0,a>0.$ The next theorem combines Lemmas A1, A7 and A10 (see Appendix A.1, Appendix A.2 and Appendix A.3, respectively).

Theorem 2.

For repeated symmetric games, there exists a reactive Nash equilibrium iff one of the following conditions holds true:

1.

$a=b=c=0;$

2.

$a=0\&-1<-\frac{b}{c}<1;$

3.

$a\ne 0\&0<\mathbf{b}<1;$

4.

$a<0\&\mathbf{c}<2-\mathbf{b}\&\mathbf{b}\ge 1;$

5.

$a<0\&\mathbf{c}>-\mathbf{b}\&\mathbf{b}\le 0;$

6.

$a>0\&\mathbf{c}<\mathbf{b}\&\mathbf{b}\le 0;$

7.

$a>0\&\mathbf{c}>\mathbf{b}\&\mathbf{b}\ge 1.$

In this paper, we focus on symmetric stage games due to tractability of the corresponding results. Appendix C.2 shows an example of a non-symmetric stage game.

2.5. If All RSs Are Available

In this paper, players’ strategy sets do not include conditional deterministic and semi-deterministic RSs, and we focus only on equilibria generated by SRSs. Thus, in ${\mathbf{G}}^{\infty }$, the set of all possible profiles was $\tilde{U}\times \tilde{V}$ instead of ${[0,1]}^2\times {[0,1]}^2,$ the set of all profiles in RSs. Let us discuss how this decision influences the NE profiles in SRSs.

Recall that for ${\mathbf{G}}^{\infty },$ the premises of the Kakutani fixed-point theorem do not hold, as the strategy set is not compact, although payoff functions are continuous. Thus, the existence of an NE in an arbitrary game ${\mathbf{G}}^{\infty }$ is not guaranteed. By simply allowing all RSs, the strategy set becomes compact, but the payoff functions become dependent on initial action profiles, making payoff functions no longer continuous and properly defined. The last fact is due to payoff-relevance of the initial round for some strategy profiles. Thus, one may decide to complement strategies with one extra variable, an initial action, leading to more complicated analysis. There are two approaches that can be used to avoid the complication. The first approach, the one used for discounted payoffs, is to assume that nature draws initial actions. In this case, one may consider only equilibria that hold independently of initial moves (see [17,18]); actual equilibrium payoffs may depend on the initial actions. The second approach, which is possible for the-limit-of-means setting and is used in this article, is to restrict the strategy set such that players’ payoffs are independent of initial actions. Both approaches ensure that any equilibrium holds for all opening moves.

Starting with the restricted strategy sets, one may ask, what if we extend the set of strategies to include all RSs? How would it influence the set of all equilibria? One straightforward aspect is that new equilibria may form. The second (less obvious) aspect is that we increase players’ abilities to deviate; this may erase some existing equilibria.

Regarding the first aspect, the precise answer requires a thorough analysis that can not be included in this paper since, as we mentioned above, the results will depend on initial actions. More importantly, the value of such results is not obvious.

Regarding the second aspect, the characterization of all Nash equilibria generated by SRSs remains valid independently of initial moves, even if one considers the setting where all RSs are available for players. Let us show this. Consider player 1. If player 2 picks an SRS $\widehat{v}$, then the corresponding stationary distribution is well defined by (4) for all RSs of player 1 and does not depend on initial actions. For player 1, in terms of payoff-relevant deviations, the set of RSs is equivalent to $\tilde{U}.$

3. Equilibrium Payoffs in Conditional SRSs

In this section, we compare equilibrium payoffs in conditional SRSs versus unconditional RSs in the following simple way. First, we fix a stage game $\mathbf{G}$ such that Theorem 2 gives us an NE in SRSs. The repeated version ${\mathbf{G}}_{un}^{\infty }$ inherits all equilibrium payoff profiles from $\mathbf{G}$ that are generated by the corresponding unconditional RSs. Among these payoff profiles, we select the symmetric one with the lowest payoffs as the benchmark. We then compare the best symmetric NE payoff profile in ${\mathbf{G}}^{\infty }$ formed by SRSs against this benchmark. Since we have existence results only for symmetric games, the complete analysis is only feasible for this special case. Nevertheless, the existence of an NE in memory-less SRSs implies the existence of an NE in conditional SRSs that allows us to compare the corresponding payoffs even for non-symmetric games. The next subsection presents this partial result; afterwards, we complete analysis for symmetric games.

3.1. Payoffs for NE Profiles of Unconditional and Conditional SRSs

From (4) and Theorem 1, it follows that any $(u,v)\in {\mathbb{R}}^2$ admitting solutions of the system

$$\left\{\begin{array}{cc}\hfill y=v_1-v_2,a_1(s_{1}y+s_2)+b_1+c_{1}y& =0,(i)\hfill \\ \hfill x=u_1-u_2,a_2(s_1+s_{2}x)+b_2+c_{2}x& =0,(ii)\hfill \\ \hfill s_1=\frac{v_{2}x+u_2}{1-xy},s_2=\frac{u_{2}y+v_2}{1-xy},0\ge a_{2}x,0& \ge a_{1}y,\hfill \\ \hfill 0<u_1,u_2,v_1,v_2<1,-1<x,y<1\end{array}\right.$$

(22)

forms a NE. Note that (22) is a system of six equations with eight variables. We consider $(x,y)$ or $(s_1,s_2)$ as free parameters to express the remaining variables; x and y are introduced to simplify the notation. We omit the rather simple case $a_1{a}_2=0$ and start solving the system under the assumption that

$$a_1{a}_2\ne 0.$$

Suppose $-1<x,y<1;$ let us express $u,v,s_1,s_2$ from (22) in terms of x and y

$$\left\{\begin{array}{cc}\hfill u_2& =\frac{2a_{2}x(b_1+c_{1}y)-a_1(b_2+c_{2}x)(1+xy)}{a_1{a}_2(1-xy)},s_1=\frac{a_1(b_2+c_{2}x)-a_{2}x(b_1+c_{1}y)}{a_1{a}_2(-1+xy)},\hfill \\ \hfill v_2& =\frac{2a_{1}y(b_2+c_{2}x)-a_2(b_1+c_{1}y)(1+xy)}{a_1{a}_2(1-xy)},s_2=\frac{-a_1(b_2+c_{2}x)y+a_2(b_1+c_{1}y)}{a_1{a}_2(-1+xy)},\hfill \\ \hfill u_1& =u_2+x,v_1=v_2+y,0<u_1,u_2,v_1,v_2<1,\hfill \\ \hfill 0& \ge a_{2}x,0\ge a_{1}y.\hfill \end{array}\right.$$

(23)

Assume that a completely mixed NE exists in a one-shot game $\mathbf{G}$. Then $(-\frac{b_2}{a_2},-\frac{b_1}{a_1})$ is the corresponding NE profile since $a_1{a}_2\ne 0$. Furthermore, $u=(-\frac{b_2}{a_2},-\frac{b_2}{a_2})$ and $v=(-\frac{b_1}{a_1},-\frac{b_1}{a_1})$ form an NE in ${\mathbf{G}}^{\infty }$ by Proposition 1. Indeed (see (5)), $\forall \tilde{u}\in \tilde{U}\forall \tilde{v}\in \tilde{V}$

$$\begin{array}{c}\hfill {\mathbf{J}}_1^{\infty }(\tilde{u},v)=a_1{\mathbf{s}}_1(\tilde{u},v)(-\frac{b_1}{a_1})+b_1{\mathbf{s}}_1(\tilde{u},v)-\frac{c_1{b}_1}{a_1}+A_{B,R}=A_{B,R}-\frac{c_1{b}_1}{a_1}={\mathbf{J}}_1^{\infty }(u,v),\end{array}$$

(24)

$$\begin{array}{c}\hfill {\mathbf{J}}_2^{\infty }(u,\tilde{v})=a_2(-\frac{b_2}{a_2}){\mathbf{s}}_2(u,\tilde{v})+b_2{\mathbf{s}}_2(u,\tilde{v})-\frac{c_2{b}_2}{a_2}+B_{B,R}=B_{B,R}-\frac{c_2{b}_2}{a_2}={\mathbf{J}}_2^{\infty }(u,v).\end{array}$$

(25)

The next proposition shows that for both players, equilibrium payoffs in conditional SRSs are at least equilibrium payoffs in memory-less SRSs.

Proposition 2.

Suppose that $a_1{a}_2\ne 0$ and there exists an NE profile $(\widehat{u},\widehat{v})$ of memory-less SRSs in ${\mathbf{G}}_{un}^{\infty }$ (hence in ${\mathbf{G}}^{\infty }$); then, for any NE profile $(u,v)$ of conditional SRSs in ${\mathbf{G}}^{\infty }$, the following holds true

$$\begin{array}{c}\hfill {\mathbf{J}}_1^{\infty }(\widehat{u},\widehat{v})-{\mathbf{J}}_1^{\infty }(u,v)=\frac{y{(a_2(c_1+b_{1}x)-a_1(b_2+c_{2}x))}^2}{a_1{a}_2^2{(-1+xy)}^2}\le 0,\\ \hfill {\mathbf{J}}_2^{\infty }(\widehat{u},\widehat{v})-{\mathbf{J}}_2^{\infty }(u,v)=\frac{x{(a_1(c_2+b_{2}y)-a_2(b_1+c_{1}y))}^2}{a_2{a}_1^2{(-1+xy)}^2}\le 0\\ \hfill wherex=u_1-u_2,y=v_1-v_2.\end{array}$$

Let us outline the proof. From (1) and (5), we have ${\mathbf{J}}_i^{\infty }$ as a function of stationary distributions. Since we deal with ESDs, we use formulas for $s_1$ and $s_2$ from (23). Hence, ${\mathbf{J}}_i^{\infty },i=1,2,$ become functions of only x and y that are defined by the NE profile $(u,v);$ these equilibrium profiles exists by our assumption. Using (24) and (25), it remains only to perform some algebra and to recall that $a_{2}x\le 0,a_{1}y\le 0$ (see the last line in (23)).

3.2. Symmetric Games

For symmetric games, results in Section 2.4 allow us to gain more understanding on how equilibrium payoffs in conditional SRSs compare to the benchmark payoffs in memory-less RSs. Note that a generic one-shot symmetric game with distinct payoffs on the leading diagonal is strategically equivalent to

Generic one-shot symmetric games with identical payoffs on the leading diagonal are considered in Appendix B. Using these generic games, we study equilibrium payoffs for games satisfying the conditions of Theorem 2. For $G^{10}$, Figure 5 shows all pairs of $(l,g)$ such that there exists an equilibrium according to Theorem 2. For example, the first condition results in trivial stage games that $G^{10}$ cannot be. The second condition (region 2) restricted to Quadrant I, includes prisoner’s dilemmas with equal gains from switching. Quadrant I contains all prisoner’s dilemmas and the entire region 5 that embodies the ones where cycles $(C,D)\to (D,C)\to (C,D)\to \dots$ are more profitable than cycles $(C,C)\to (D,D)\to (C,C)\to \dots .$

Region 3 completely covers Quadrant II and IV where all stage games with two pure equilibria are located; here, we have completely mixed NE for stage games and, hence, an equilibrium profile in memory-less SRSs for the repeated versions. If $-l\ne 1+g,$ by Proposition 2, there exists an NE profile of conditional SRSs that Pareto dominates the benchmark of NE payoffs for memory-less SRSs; else all ESDs (hence, NE payoff profiles) for conditional and unconditional SRSs coincide.

Quadrant III contains all deadlock games, which have one dominant strategy with the equilibrium payoff being Pareto optimal. This property makes deadlock games of the least interest for theory since self-interest and mutual benefit are fully compatible. The repeated versions of games from region 4 have NE payoff profiles Pareto dominating this unconditional benchmark. Thus, in all cases considered up to this point, an equilibrium profile in conditional SRSs provides greater or equal payoffs than the benchmark equilibrium profile in memory-less RSs. The exceptions to this rule are region 2, restricted to Quadrant III, and region 6 that contain very special deadlock games: both players (by choosing dominating pure strategy) get the highest possible individual payoff. Because of this unique property, the repeated versions of these one-shot games are the only cases when all NE profile in conditional SRSs are Pareto dominated by any equilibrium profile in memory-less RSs. We provide an example of such game in Section 3.3. Finally, all one-shot games $G^{10}$ in region 6, after we swap strategies for both players, comply with condition 7 in Theorem 2; i.e., conditions 6 and 7 cover strategically equivalent deadlock games.

Appendix B provides the similar analysis for generic symmetric games with identical payoffs on the leading diagonal, where we conclude that in the cases when payoffs of the stage game are not diverse enough, and payoffs of NE profiles in SRSs coincide with the memory-less benchmark. Otherwise, there exists an NE profile in SRSs that Pareto dominates the benchmark.

Let us summarize both cases of the generic games. Only in the very special deadlock games where both players (by choosing dominating pure strategy) get the highest possible individual payoff, payoff profiles in conditional SRSs are dominated by the benchmark unconditional payoff profiles; see the next subsection for an example. In the remaining cases, if distinct ESDs do exist, then there exists an NE profile in SRSs dominating the benchmark.

3.3. A Game with Pareto-Efficient Equilibrium and Dominant Strategies

The following example highlights one counterintuitive property of NE in SRSs by showing that it is possible to establish equilibria with lower payoffs by introducing a more complex class of strategies. Consider the following symmetric game $G^{NB}$.

Once we scale all payoffs into the range $[0,1]$, the stage game falls into region 6 of Figure 5. Here $a=10,\mathbf{b}=-0.2,$ and $\mathbf{c}=-0.7.$ According to Theorem 2, a NE in SRSs exists; we depict the set of all ESDs in Figure 6. A mixed NE in the stage game does not exist since $\mathbf{b}<0$. The profile of the memory-less RSs $((1,1),(1,1))$ forms an NE and corresponds to the Pareto-efficient pure NE in the stage game. This equilibrium is excluded from our characterization as we are focusing on equilibria in SRSs. Remarkably, there is a noticeable gap between this equilibrium payoff, serving as memory-less benchmark, and payoffs corresponding to ESDs in the repeated game. For any $s\in ]\frac{1+\sqrt{161}}{40},\frac{1+\sqrt{561}}{40}[$, there exists a Nash equilibrium such that $(s,s)$ is the ESD. For example, equilibrium strategies $u=(\frac{4}{55},\frac{34}{55})$ and $v=(\frac{4}{55},\frac{34}{55})$ generate the ESD $(\frac{2}{5},\frac{2}{5})$ with payoffs $\frac{26}{5}$ (compare with 19). Thus, the example shows that it is possible to establish equilibria with lower payoffs by introducing a more complex class of strategies.

4. Conclusions

We show analytically that even an incremental change of complexity of strategies (an inch of memory) in repeated interactions has a huge influence on the sets of equilibrium payoff profiles for some classes of stage games that are circumscribed by Theorem 2. The first thing to note is that except for some trivial cases, payoff-relevant indeterminacy holds true [17]; i.e., there now exists a continuum of equilibria in “new” conditional strategies with distinct payoff profiles. Clearly, there is still no folk theorem (see the example in Appendix C.1); this still holds [17] for the next incremental improvement, 1-memory strategies. It is also now evident that one should not expect some consistency while incrementally increasing the memory of strategies. For example, in RSs, there exist equilibria with higher payoffs than the upper bound obtained independently of the discount factor in [17] (see the example in Appendix C.1) and the reverse dominance condition [17], which is important for the 1-memory case, does not influence the existence of NE in RSs.

In this paper, we focused on symmetric stage games due to tractability of the corresponding results. Appendix C.2 shows the example of a non-symmetric stage game where the set of all ESDs looks like a special combination of the corresponding sets for two symmetric stage games derived from the payoff matrix. This may be a way to study a broad class of non-symmetric stage games. An alternative approach may be to study specific classes of non-symmetric stage games, e.g., strategically zero-sum games [25].

In all examples considered so far, the region of all ESDs was a connected set. Appendix C.3 demonstrates the example with disconnected regions of equilibria. A branch of literature assumes that we may have some local dynamics on the set of all Nash equilibria, e.g., in the spirit of adaptive dynamics [4] or local cooperation patterns [26]. This example calls for new models to tackle the issue of disconnectedness that may be an obstacle on the way to higher equilibrium payoffs.

As for future work, we note that the idea of a reactive strategy was recently generalized to “reactive learning strategy” [27] allowing a player to gradually change the probability of actions depending on the past actions of the opponent. It is not known how the change from reactive strategies to reactive learning strategies would influence the equilibrium outcomes.