A Two-Player Resource-Sharing Game with Asymmetric Information

Abstract

This paper considers a two-player game where each player chooses a resource from a finite collection of options. Each resource brings a random reward. Both players have statistical information regarding the rewards of each resource. Additionally, there exists an information asymmetry where each player has knowledge of the reward realizations of different subsets of the resources. If both players choose the same resource, the reward is divided equally between them, whereas if they choose different resources, each player gains the full reward of the resource. We first implement the iterative best response algorithm to find an $ϵ$-approximate Nash equilibrium for this game. This method of finding a Nash equilibrium may not be desirable when players do not trust each other and place no assumptions on the incentives of the opponent. To handle this case, we solve the problem of maximizing the worst-case expected utility of the first player. The solution leads to counter-intuitive insights in certain special cases. To solve the general version of the problem, we develop an efficient algorithmic solution that combines online convex optimization and the drift-plus penalty technique.

1. Introduction

We consider the following game with two players, A and B. There are n resources, each denoted by an integer between 1 and n. Each player selects a resource without knowledge about the other player’s selection. The state of the game is described by the random vector $\mathit{W}={(W_1,W_2,\dots ,W_n)}^{\top }$, where $W_k$ is the reward random variable of resource k. We assume $W_k$ to be independent random variables for each $1\le k\le n$, taking non-negative real values. If both players choose the same resource k, each gets a utility of $W_k/2$. If they choose different resources $k,l$, they receive utilities of $W_k$ and $W_l$, respectively. It is assumed that the mean and the variance of $W_k$ exist and are finite for each $1\le k\le n$. Both players know the distribution of $\mathit{W}$. Our formulation allows for an information asymmetry between the players. In particular, $\{1,2,\dots ,n\}$ can be partitioned into four sets $\{\mathcal{A},\mathcal{B},\mathcal{C},\mathcal{AB}\}$ where only player A observes the realizations of $W_k$ for $k\in \mathcal{A}$, only player B observes the realizations of $W_k$ for $k\in \mathcal{B}$, no player observes the realizations of $W_k$ for $k\in \mathcal{C}$, and both players observe the realizations of $W_k$ for $k\in \mathcal{AB}$.

This game can be used to model different real-world scenarios where the agents have asymmetric information regarding the involved information structure. One classic example is the problem of Multiple-Access Control (MAC) in communication systems. Here, communication channels are accessed by multiple users, and the data rate of a channel is shared amongst the users who select it [1]. A channel can be shared using Time Division Multiple Access (TDMA) or Frequency Division Multiple Access (FDMA), where in TDMA, the channel is time-shared among the users [2,3], whereas in FDMA, the channel is frequency-shared among the users [4]. In both cases, the total data rate supported by the channel can be considered the utility of the channel. The problem of information asymmetry arises since a user might have precise information regarding the total data rate offered by some channels but not others, and the known channels can be different for different users. On the other hand, the users in such a system cannot be trusted since the system may have malicious users (for instance, jammers) who focus on reducing the data rate available to genuine users.

Modified versions of this game apply to problems in economics. For instance, consider a firm that chooses a market to enter from a pool of market options. The chosen market may also be chosen by another firm. The reward of a market is the revenue it brings. Assume a simplified model where there exists a total revenue for each market, and the total revenue is divided equally among the firms entering the market. A reward known to all firms can be considered public information, while a reward known only to one firm is private information of that firm.

The game defined above can be viewed as a stochastic version of the class of games defined in [5], which are resource-sharing games, also known as congestion games. In resource-sharing games, players compete for a finite collection of resources. In a single turn of the game, each player is allowed to select a subset of the collection of resources, where the allowed subsets make up the action space of the player. Each resource offers a reward to each player who selected the particular resource, where the reward offered depends on the number of players who selected it. The relationship between the reward offered to a player by a resource and the number of users selecting it is captured by the reward function of the resource. A player’s utility is equal to the sum of the rewards offered by the resources in the subset selected by the player. In [5], it is established that the above game has a pure-strategy (deterministic) Nash equilibrium.

Although in the classical setting, these games ignore the stochastic nature of the rewards offered by the resources, the idea of resource-sharing games has been extended to different stochastic versions [6,7]. Versions of the game with information asymmetry have been considered through the work of [8] in the context of Bayesian games, which considers the information design problem for resource-sharing with uncertainty. Similar Bayesian games have also been considered in [9,10]. It should be noted that in general resource-sharing games, no conditions are placed on the reward functions of the resources. The special case where the reward functions are non-decreasing in the number of players selecting the resource is called a cost-sharing game [11]. These games are typically treated as games where a cost is minimized rather than a utility being maximized. In fair cost-sharing games, the cost of a resource is divided equally among the players selecting the resource. We consider a fair reward allocation model, where the reward of a resource is equally shared among the players selecting the resource. It should be noted that in this model, the players have opposite incentives compared to a fair-cost sharing model.

The work on resource-sharing games assumes that the players either cooperate or have the incentive to maximize a private or a social utility. It is interesting to consider a stochastic version of the game with asymmetric information between players who do not necessarily trust each other and who place no assumptions on the incentives of the opponents. In this context, the players have no signaling or external feedback and take actions based only on their personal knowledge of the reward realizations for a subset of the resource options. In this paper, we consider the above problem and limit our attention to the two-player singleton case, where each player can choose only one resource.

In the first part of the paper, we provide an iterative best response algorithm to find an $ϵ$-approximate Nash equilibrium of the system. In the second part, we solve the problem of maximizing the worst-case expected utility of the first player. We solve the problem in two cases. The first case is when both players do not know the realizations of the reward random variables of any of the resources, in which case an explicit solution can be constructed. This case yields a counter-intuitive solution that provides insight into the problem. One such insight is that, while it is always optimal to choose from a subset of resources with the highest average rewards, within that subset, one chooses the higher-valued rewards with lower probability. For the second case, we solve the general version of the problem by developing an algorithm that leverages the online optimization technique [12,13] and the drift-plus penalty method [14]. This algorithm generates a mixture of $\mathcal{O}(1/{\epsilon }^2)$ pure strategies, which, when used in an equiprobable mixture, provides a utility within $\epsilon$ of optimality on average. Below, we summarize our major contributions.

• We consider the problem of a two-player singleton stochastic resource-sharing game with asymmetric information. We first provide an iterative best response algorithm to find an $ϵ$-approximate Nash equilibrium of the system. This equilibrium analysis uses potential game concepts.
• When the players do not trust each other and place no assumptions on the incentives of the opponent, we solve the problem of maximizing the worst-case expected utility of the first player using a novel algorithm that leverages techniques from online optimization and the drift-plus penalty methods. The algorithm developed can be used to solve the general unconstrained problem of finding the randomized decision $\alpha \in \{1,2,\dots ,n\}$, which maximizes $\mathbb{E}\left\{h\right(\mathit{x};\mathbf{\Theta }\left)\right\}$, where $\mathit{x}\in {\mathbb{R}}^n$ with $x_k=\mathbb{E}\left\{{\Gamma }_k{\mathbb{1}}_{\{\alpha =k\}}\right\}$, $\mathbf{\Theta }\in {\mathbb{R}}^m$ and $\mathbf{\Gamma }\in {\mathbb{R}}^n$ are non-negative random vectors with finite second moments, and h is a concave function such that $\tilde{h}\left(\mathit{x}\right)=\mathbb{E}\left\{h(\mathit{x};\mathbf{\Theta })\right\}$ is Lipschitz continuous, entry-wise non-decreasing and has bounded subgradients.
• We show our algorithm uses a mixture of only $\mathcal{O}(1/{\epsilon }^2)$ pure strategies using a detailed analysis of the sample path of the related virtual queues (our preliminary work on this algorithm used a mixture of $\mathcal{O}(1/{\epsilon }^3)$ pure strategies). Virtual queues are also used for constrained online convex optimization in [13], but our problem structure is different and requires a different and more involved treatment.

1.1. Background on Resource-Sharing Games

The classical resource-sharing game defined in [5] is a tuple $(\mathcal{M},\mathcal{N},\mathcal{T},\mathit{r})$, where $\mathcal{M}$ is a set of m players, $\mathcal{N}$ is a set of n resources, $\mathcal{T}={\mathcal{T}}_1\times {\mathcal{T}}_2\times \dots \times {\mathcal{T}}_m$ where ${\mathcal{T}}_j$ is the set of possible actions of player j (which is a subset of $2^{\mathcal{N}}$), and $\mathit{r}=(r_1,r_2,\dots ,r_n)$, where $r_i:{\mathbb{N}}_0\to \mathbb{R}$ is the reward function of resource i. Here, we use the notation ${\mathbb{N}}_0=\mathbb{N}\cup \left\{0\right\}$. Each player has complete knowledge about the tuple $(\mathcal{M},\mathcal{N},\mathcal{T},\mathit{r})$, but they do not have knowledge of the actions chosen by other players. For an action profile $\mathit{a}=(a_1,a_2,\dots ,a_m)\in \mathcal{T}$, the count function # is a function from $\mathcal{N}\times \mathcal{T}$ to ${\mathbb{N}}_0$ where $\#(i,\mathit{a})={\sum }_{k=1}^m{\mathbb{1}}_{\{i\in a_k\}}$. In other words, $\#(i,\mathit{a})$ is the number of players choosing resource i under action profile $\mathit{a}$. We call the quantity $r_i(\#(i,\mathit{a}))$ the per-player reward of resource i under action profile $\mathit{a}$. The utility $u_j$ of player j is a function from $\mathcal{T}$ to $\mathbb{R}$, where $u_j\left(\mathit{a}\right)={\sum }_{i=1}^n{\mathbb{1}}_{\{i\in a_k\}}{r}_i(\#(i,\mathit{a}))$. In other words, $u_j\left(\mathit{a}\right)$ is the sum of the per-player rewards of the resources chosen by player j under action profile $\mathit{a}$. Resource-sharing games fall under the general category of potential games [15]. Potential games are the class of games for which the change in reward of any player as a result of changing their strategy can be captured by the change in a global potential function.

Many game variations of the resource-sharing game have been studied [16]. Weighted resource-sharing games [17], games with player-dependent reward functions [18], and games with resources having preferences over players [19] are some of the extensions. Singleton games, where each player is allowed to choose only one resource, have also been explored explicitly in the literature [20,21]. Some of the extensions of the classical resource-sharing game possess a pure Nash equilibrium in the singleton case. Two examples would be the games with player-specific reward functions for a resource [18] and the games with priorities where the resources have preferences over the players [19].

Resource-sharing games have been extended to several stochastic versions. For instance, ref.  [6] considers the selfish routing problem with risk-averse players in a network with stochastic delays. The work of [7] considers two scenarios where, in the first scenario, each player participates in the game with a certain probability, and in the second scenario, the reward functions are stochastic. The problem of information asymmetry in resource-sharing games has been addressed through the work of [8,9,10,22]. The work of [22] considers a network congestion game where the players have different information sets regarding the edges of the network. Further, ref. [8] considers a scenario with a single random state $\theta$, which determines the reward functions. The realization of $\theta$ is known to a game manager who strategically provides recommendations (signaling) to the players to minimize the social cost. An information asymmetry arises among the players in this case due to the actions of the game manager during private signaling, where the game manager provides player-specific recommendations.

Resource-sharing games appear in a variety of applications such as service chain composition [23], congestion control [24], network design [25], load balancing networks [26,27], resource sharing in wireless networks [28], spectrum sharing [29], radio access selection [30], non-orthogonal multiple access [31,32], network selection [33,34], and migration of species [35].

Our formulation differs from the literature on resource-sharing games since we consider a scenario that is difficult to be analyzed using the standard equilibrium-based approaches. This is due to the fact that the players do not trust each other and place no assumptions on the incentives of the opponents, and they take action in the absence of a signaling mechanism or external feedback by just using their knowledge of the reward random variables. This motivates our formulation as a one-shot problem tackled using worst-case expected utility maximization.

1.2. Notation

We use calligraphic letters to denote sets. Vectors and matrices are denoted by boldface characters. For integers n and m, we denote by $[n:m]$ the inclusive set of integers between n and m. Given a vector $\mathit{w}\in {\mathbb{R}}^m$, $w_k$ is used to denote the k-th element of $\mathit{w}$; ${\mathit{w}}_{k:l}$ for $l\ge k$ represents the $l-k+1$ dimensional sub-vector ${(w_k,w_{k+1},\dots ,w_l)}^{\top }$ of $\mathit{w}$; for a subset $\mathcal{S}$ of integers from 1 to n $\{w_k;k\in \mathcal{S}\}$ represents the sub-vector of $\mathit{w}$ with index in $\mathcal{S}$. For $\mathit{z}\in {\mathbb{R}}^m$, we use ${\parallel \mathit{z}\parallel }_2$ to denote the standard Euclidean norm (L2 norm) of $\mathit{z}$. For a function $f:{\mathbb{R}}^m\to \mathbb{R}$, and $\mathit{z}\in {\mathbb{R}}^m$, we use $f^{{}^{\prime }}\left(\mathit{z}\right)=(f_1^{{}^{\prime }}\left(\mathit{z}\right),f_2^{{}^{\prime }}\left(\mathit{z}\right),\dots ,f_m^{{}^{\prime }}\left(\mathit{z}\right))$ to denote a subgradient of f at $\mathit{z}$.

2. Materials and Methods

The code used for the simulations is implemented using Python programming language in the notebook https://rb.gy/wvt33, accessed on 10 August 2023.

3. Formulation

Denote $\mathit{X}=\{W_k;k\in \mathcal{A}\}$, $\mathit{Y}=\{W_k;k\in \mathcal{B}\}$, $\mathit{Z}=\{W_k;k\in \mathcal{AB}\}$, and $\mathit{V}=\{W_k;k\in \mathcal{C}\}$. Recall that $\mathit{X}$ is known only to player A, $\mathit{Y}$ is known only to player B, $\mathit{Z}$ is known to both players, and $\mathit{V}$ is known to neither. Let us define ${\mathcal{A}}^c=[1:n]\setminus \mathcal{A}$, and ${\mathcal{B}}^c=[1:n]\setminus \mathcal{B}$. Let $\left|\mathcal{A}\right|=a$, $\left|\mathcal{B}\right|=b$, $\left|\mathcal{C}\right|=c$ and $\left|\mathcal{AB}\right|=d$. Therefore, $a+b+c+d=n$. Without loss of generality, we assume $\mathcal{A}=[1:a]$, $\mathcal{B}=[a+1:a+b]$, $\mathcal{C}=[a+b+1:a+b+c]$, and $\mathcal{AB}=[a+b+c+1:n]$.

Let $R^C(g^A,g^B)$ be the random variable representing the utility of player $C\in \{A,B\}$, given that player A uses strategy $g^A$, and player B uses strategy $g^B$. General strategies for players A and B can be represented by the Borel-measurable functions,

$$\begin{array}{cc}\hfill & g^A:[0,1)\times {\mathbb{R}}_{\ge 0}^{a+d}\to [1:n],\hfill \end{array}$$

(1)

$$\begin{array}{cc}\hfill & g^B:[0,1)\times {\mathbb{R}}_{\ge 0}^{b+d}\to [1:n],\hfill \end{array}$$

(2)

where

$$\begin{array}{cc}\hfill & {\alpha }^A=g^A(U^A,\mathit{X},\mathit{Z}),\hfill \end{array}$$

(3)

$$\begin{array}{cc}\hfill & {\alpha }^B=g^B(U^B,\mathit{Y},\mathit{Z}),\hfill \end{array}$$

(4)

are the resources chosen by players A and B, respectively. Here, $U^A$ and $U^B$ are independent randomization variables uniformly distributed in [0, 1) and independent of $\mathit{W}$. A pure strategy for player A is a function $g^A$ that does not depend on $U^A$, whereas a mixed strategy is a function $g^A$ that depends on $U^A$. Hence, we drop the randomization variable when depicting a pure strategy. Pure strategies and mixed strategies for player B are defined similarly. Let ${\mathcal{S}}^A$ and ${\mathcal{S}}^B$ denote the sets of all possible strategies for players A and B, respectively.

It turns out that our analysis is simplified when $\mathit{Z}$ is fixed. Fixing $\mathit{Z}$ does not affect the symmetry between players A and B since $\mathit{Z}$ is observed by both players A and B. Hereafter, we conduct the analysis by considering all quantities conditioned on $\mathit{Z}$.

Define

$$\begin{array}{cc}\hfill & p_k^A=\mathbb{E}\left\{{\mathbb{1}}_{\{{\alpha }^A=k\}}\right|\mathit{Z}\}\mathrm{for}1\le k\le n,\hfill \\ & q_k^A=\mathbb{E}\left\{W_k{\mathbb{1}}_{\{{\alpha }^A=k\}}\right|\mathit{Z}\}\mathrm{for}k\in \mathcal{A},\hfill \end{array}$$

(5)

and,

$$\begin{array}{cc}\hfill & p_k^B=\mathbb{E}\left\{{\mathbb{1}}_{\{{\alpha }^B=k\}}\right|\mathit{Z}\}\mathrm{for}1\le k\le n,\hfill \\ & q_k^B=\mathbb{E}\left\{W_k{\mathbb{1}}_{\{{\alpha }^B=k\}}\right|\mathit{Z}\}\mathrm{for}k\in \mathcal{B}.\hfill \end{array}$$

(6)

Note that $p_k^A$ and $p_k^B$ are the conditional probabilities of players A and B choosing k given $\mathit{Z}$. Define vectors ${\mathit{p}}^A=\{p_k^A;1\le k\le n\}$, ${\mathit{q}}^A=\{q_k^A;k\in \mathcal{A}\}$, ${\mathit{p}}^B=\{p_k^B;1\le k\le n\}$, and ${\mathit{q}}^B=\{q_k^B;k\in \mathcal{B}\}$. For $1\le k\le n$, define $E_k=\mathbb{E}\left\{W_k\right|\mathit{Z}\}$. Hence, we have

$$E_k=\left\{\begin{array}{cc}{W}_k\hfill & \mathrm{if}k\in \mathcal{AB},\\ \mathbb{E}\left\{W_k\right\}& \mathrm{otherwise},\end{array}\right.$$

(7)

which uses the independence of $W_k$ and $\mathit{Z}$ when $k\notin \mathcal{AB}$.

Note that the utility achieved by player A given the strategies $g^A$ and $g^B$ can be written as

$$\begin{array}{c}\hfill R^A(g^A,g^B)=\sum _{k=1}^n{W}_k\left({\mathbb{1}}_{\{{\alpha }^A=k\}}-\frac{1}{2}{\mathbb{1}}_{\{{\alpha }^A=k\}}{\mathbb{1}}_{\{{\alpha }^B=k\}}\right).\end{array}$$

(8)

Given the strategies $g^A$ and $g^B$, we provide an expression for the expected utility of player A given $\mathit{Z}$, where the expectation is over the random variables $\mathit{X},\mathit{Y},\mathit{V}$, and the possibly random actions ${\alpha }^A$ and ${\alpha }^B$. Taking expectations of (8) gives,

$$\begin{array}{cc}\hfill \mathbb{E}\{& R^A(g^A,g^B)\left|\mathit{Z}\right\}=\sum _{k=1}^n\mathbb{E}\left\{W_k{\mathbb{1}}_{\{{\alpha }^A=k\}}\right|\mathit{Z}\}-\frac{1}{2}\sum _{k=1}^n\mathbb{E}\left\{W_k{\mathbb{1}}_{\{{\alpha }^A=k\}}{\mathbb{1}}_{\{{\alpha }^B=k\}}\right|\mathit{Z}\}\hfill \\ & =\sum _{k\in \mathcal{A}}\mathbb{E}\left\{W_k{\mathbb{1}}_{\{{\alpha }^A=k\}}\right|\mathit{Z}\}+\sum _{k\in {\mathcal{A}}^c}\mathbb{E}\left\{W_k\right|\mathit{Z}\}\mathbb{E}\left\{{\mathbb{1}}_{\{{\alpha }^A=k\}}\right|\mathit{Z}\}-\frac{1}{2}\sum _{k=1}^n\mathbb{E}\left\{W_k{\mathbb{1}}_{\{{\alpha }^A=k\}}{\mathbb{1}}_{\{{\alpha }^B=k\}}\right|\mathit{Z}\}\hfill \\ & =\sum _{k\in \mathcal{A}}{q}_k^A+\sum _{k\in {\mathcal{A}}^c}{E}_k{p}_k^A-\frac{1}{2}\sum _{k=1}^n\mathbb{E}\left\{W_k{\mathbb{1}}_{\{{\alpha }^A=k\}}{\mathbb{1}}_{\{{\alpha }^B=k\}}\right|\mathit{Z}\}.\hfill \end{array}$$

(9)

Note that given $\mathit{Z}$, the random variables ${\alpha }^A$ and ${\alpha }^B$ are independent. Hence, we can split the last term (9) as follows,

$$\begin{array}{cc}\hfill \sum _{k=1}^n& \mathbb{E}\left\{W_k{\mathbb{1}}_{\{{\alpha }^A=k\}}{\mathbb{1}}_{\{{\alpha }^B=k\}}\right|\mathit{Z}\}=\sum _{k\in \mathcal{A}}\mathbb{E}\left\{W_k{\mathbb{1}}_{\{{\alpha }^A=k\}}{\mathbb{1}}_{\{{\alpha }^B=k\}}\right|\mathit{Z}\}+\sum _{k\in \mathcal{B}}\mathbb{E}\left\{W_k{\mathbb{1}}_{\{{\alpha }^A=k\}}{\mathbb{1}}_{\{{\alpha }^B=k\}}\right|\mathit{Z}\}\hfill \\ & +\sum _{k\in \mathcal{C}\cup \mathcal{AB}}\mathbb{E}\left\{W_k{\mathbb{1}}_{\{{\alpha }^A=k\}}{\mathbb{1}}_{\{{\alpha }^B=k\}}\right|\mathit{Z}\}=\sum _{k\in \mathcal{A}}\mathbb{E}\left\{W_k{\mathbb{1}}_{\{{\alpha }^A=k\}}\right|\mathit{Z}\}\mathbb{E}\left\{{\mathbb{1}}_{\{{\alpha }^B=k\}}\right|\mathit{Z}\}\hfill \\ & +\sum _{k\in \mathcal{B}}\mathbb{E}\left\{{\mathbb{1}}_{\{{\alpha }^A=k\}}\right|\mathit{Z}\}\mathbb{E}\left\{W_k{\mathbb{1}}_{\{{\alpha }^B=k\}}\right|\mathit{Z}\}+\sum _{k\in \mathcal{C}\cup \mathcal{AB}}{E}_k\mathbb{E}\left\{{\mathbb{1}}_{\{{\alpha }^A=k\}}\right|\mathit{Z}\}\mathbb{E}\left\{{\mathbb{1}}_{\{{\alpha }^B=k\}}\right|\mathit{Z}\}\hfill \\ & =\sum _{k\in \mathcal{A}}{q}_k^A{p}_k^B+\sum _{k\in \mathcal{B}}{p}_k^A{q}_k^B+\sum _{k\in \mathcal{C}\cup \mathcal{AB}}{E}_k{p}_k^A{p}_k^B.\hfill \end{array}$$

(10)

4. Computing the $\mathit{ϵ}$-Approximate Nash Equilibrium

This section focuses on finding an $ϵ$-approximate Nash equilibrium of the game. Fix $ϵ>0$. A strategy pair $(g^A,g^B)$ is defined as an $ϵ$-approximate Nash equilibrium if neither player can improve its expected reward by more than $ϵ$ if it changes its strategy (while holding the strategy of the other player fixed).

Combining (10) with (9), we have that

$$\begin{array}{cc}\hfill \mathbb{E}\{& R^A(g^A,g^B)\left|\mathit{Z}\right\}=\sum _{k\in \mathcal{A}}{q}_k^A+\sum _{k\in {\mathcal{A}}^c}{E}_k{p}_k^A-\frac{1}{2}\left(\sum _{k\in \mathcal{A}}{q}_k^A{p}_k^B+\sum _{k\in \mathcal{B}}{p}_k^A{q}_k^B+\sum _{k\in \mathcal{C}\cup \mathcal{AB}}{E}_k{p}_k^A{p}_k^B\right).\hfill \end{array}$$

(11)

Similarly, for player B, we have

$$\begin{array}{cc}\hfill \mathbb{E}\{& R^B(g^A,g^B)\left|\mathit{Z}\right\}=\sum _{k\in \mathcal{B}}{q}_k^B+\sum _{k\in {\mathcal{B}}^c}{E}_k{p}_k^B-\frac{1}{2}\left(\sum _{k\in \mathcal{A}}{q}_k^A{p}_k^B+\sum _{k\in \mathcal{B}}{p}_k^A{q}_k^B+\sum _{k\in \mathcal{C}\cup \mathcal{AB}}{E}_k{p}_k^A{p}_k^B\right).\hfill \end{array}$$

(12)

First, we focus on finding the best response for players A and B, given the other player’s strategy is fixed.

Lemma 1.

The best response for players A and B are given by ${\alpha }^A=arg{max}_{1\le k\le n}{A}_k$, and ${\alpha }^B=arg{max}_{1\le k\le n}{B}_k$, where $A_k$ and $B_k$ are given by,

$$\begin{array}{c}\hfill A_k=\left\{\begin{array}{cc}{W}_k\left(1-\frac{p_k^B}{2}\right)\hfill & ifk\in \mathcal{A},\hfill \\ E_k-\frac{q_k^B}{2}\hfill & ifk\in \mathcal{B},\hfill \\ E_k\left(1-\frac{p_k^B}{2}\right)\hfill & ifk\in \mathcal{C}\cup \mathcal{AB},\hfill \end{array}\right.B_k=\left\{\begin{array}{cc}{E}_k-\frac{q_k^A}{2}\hfill & ifk\in \mathcal{A},\hfill \\ W_k\left(1-\frac{p_k^A}{2}\right)\hfill & ifk\in \mathcal{B},\hfill \\ E_k\left(1-\frac{p_k^A}{2}\right)\hfill & ifk\in \mathcal{C}\cup \mathcal{AB}.\hfill \end{array}\right.\end{array}$$

(13)

Proof of Lemma 1.

We find the best response for A, and the best response for B follows similarly. Notice that we can rearrange (11) as,

$$\begin{array}{cc}\hfill & \mathbb{E}\left\{R^A(g^A,g^B)\right|\mathit{Z}\}=\sum _{k\in \mathcal{A}}{q}_k^A\left(1-\frac{p_k^B}{2}\right)+\sum _{k\in \mathcal{B}}{p}_k^A\left(E_k-\frac{q_k^B}{2}\right)+\sum _{k\in \mathcal{C}\cup \mathcal{AB}}{p}_k^A{E}_k\left(1-\frac{p_k^B}{2}\right)\hfill \\ & =\sum _{k\in \mathcal{A}}\mathbb{E}\left\{W_k{\mathbb{1}}_{\{{\alpha }^A=k\}}\right|\mathit{Z}\}\left(1-\frac{p_k^B}{2}\right)+\sum _{k\in \mathcal{B}}\mathbb{E}\left\{{\mathbb{1}}_{\{{\alpha }^A=k\}}\right|\mathit{Z}\}\left(E_k-\frac{q_k^B}{2}\right)\hfill \\ & +\sum _{k\in \mathcal{C}\cup \mathcal{AB}}{E}_k\mathbb{E}\left\{{\mathbb{1}}_{\{{\alpha }^A=k\}}\right|\mathit{Z}\}\left(1-\frac{p_k^B}{2}\right)\hfill \\ & =\mathbb{E}\left\{\sum _{k\in \mathcal{A}}{W}_k\left(1-\frac{p_k^B}{2}\right){\mathbb{1}}_{\{{\alpha }^A=k\}}+\sum _{k\in \mathcal{B}}\left(E_k-\frac{q_k^B}{2}\right){\mathbb{1}}_{\{{\alpha }^A=k\}}+\sum _{k\in \mathcal{C}\cup \mathcal{AB}}{E}_k\left(1-\frac{p_k^B}{2}\right){\mathbb{1}}_{\{{\alpha }^A=k\}}|\mathit{Z}\right\}.\hfill \end{array}$$

(14)

The above expectation is maximized when A chooses according to the given policy.    □

Next, we find a potential function for the game. A potential function is a function of the strategies of the players such that the change in the utility of a player when he changes his strategy (while the strategies of other players are held fixed) is equal to the change in the potential function [15].

Theorem 1.

The function $H(g^A,g^B)$ given by,

$$\begin{array}{cc}\hfill H(g^A,g^B)& =\sum _{k\in \mathcal{A}}(q_k^A+E_k{p}_k^B)+\sum _{k\in \mathcal{B}}(q_k^B+E_k{p}_k^A)+\sum _{k\in \mathcal{C}\cup \mathcal{AB}}{E}_k(p_k^A+p_k^B)\hfill \\ & -\frac{1}{2}\left(\sum _{k\in \mathcal{A}}{q}_k^A{p}_k^B+\sum _{k\in \mathcal{B}}{p}_k^A{q}_k^B+\sum _{k\in \mathcal{C}\cup \mathcal{AB}}{E}_k{p}_k^A{p}_k^B\right),\hfill \end{array}$$

(15)

is a potential function for the game, where $p_k^A$, $p_k^B$ for $1\le k\le n$, $q_k^A$ for $k\in \mathcal{A}$ and $q_k^B$ for $k\in \mathcal{B}$ are defined in (5) and (6). Moreover, we have that for all $g^A,g^B\in {\mathcal{S}}^A\times {\mathcal{S}}^B$, $H(g^A,g^B)\le 2{\sum }_{k=1}^n{E}_k$.

Proof of Theorem 1.

The key to the proof is separating (15) (using (11) and (12)) as,

$$\begin{array}{cc}\hfill H(g^A,g^B)& =\mathbb{E}\left\{R^A(g^A,g^B)\right\}+\sum _{k\in {\mathcal{B}}^c}{E}_k{p}_k^B+\sum _{k\in \mathcal{B}}{q}_k^B\hfill \end{array}$$

(16)

$$\begin{array}{c}\hfill =\mathbb{E}\left\{R^B(g^A,g^B)\right\}+\sum _{k\in \mathcal{A}}{q}_k^A+\sum _{k\in {\mathcal{A}}^c}{p}_k^A{E}_k.\end{array}$$

(17)

Consider updating the strategy of player A while holding the strategy of player B fixed. Notice that since ${\sum }_{k\in {\mathcal{B}}^c}{E}_k{p}_k^B+{\sum }_{k\in \mathcal{B}}{q}_k^B$ is not affected in this process, from (16), we have that the change in the expected utility of player A is equal to the change in the H function. Similarly, this holds when player B updates the strategy while holding player A’s strategy fixed. Hence, this is indeed a potential function. The proof that $H(g^A,g^B)\le 2{\sum }_{k=1}^n{E}_k$ is omitted for brevity (See technical report [36] for details).    □

Using Theorem 1 with standard potential game theory (see, for example, [37]), we have that the iterative best response algorithm with the best response found in Lemma 1 converges to an $ϵ$-approximate Nash equilibrium in at most $\left(2{\sum }_{k=1}^n{E}_k\right)/ϵ$ iterations.

5. Worst-Case Expected Utility

Finding a Nash equilibrium using the above algorithm may not be desirable when the players do not trust each other and place no assumptions on the incentives of the opponent. To mitigate this issue, we consider maximizing the worst-case expected utility of player A. Similar to the case of finding the Nash equilibrium, the analysis is simplified when $\mathit{Z}$ is fixed.

Notice that we can simplify (10) to yield,

$$\begin{array}{cc}\hfill \sum _{k=1}^n& \mathbb{E}\left\{W_k{\mathbb{1}}_{\{{\alpha }^A=k\}}{\mathbb{1}}_{\{{\alpha }^B=k\}}\right|\mathit{Z}\}\hfill \\ & =\sum _{k\in \mathcal{A}}{q}_k^A\mathbb{E}\left\{{\mathbb{1}}_{\{{\alpha }^B=k\}}\right|\mathit{Z}\}+\sum _{k\in \mathcal{B}}{p}_k^A\mathbb{E}\left\{W_k{\mathbb{1}}_{\{{\alpha }^B=k\}}\right|\mathit{Z}\}+\sum _{k\in \mathcal{C}\cup \mathcal{AB}}{E}_k{p}_k^A\mathbb{E}\left\{{\mathbb{1}}_{\{{\alpha }^B=k\}}\right|\mathit{Z}\}\hfill \end{array}$$

(18)

$$\begin{array}{cc}& =\sum _{k\in \mathcal{A}}\mathbb{E}\left\{{\Omega }_k{q}_k^A{\mathbb{1}}_{\{{\alpha }^B=k\}}\right|\mathit{Z}\}+\sum _{k\in {\mathcal{A}}^c}\mathbb{E}\left\{{\Omega }_k{p}_k^A{\mathbb{1}}_{\{{\alpha }^B=k\}}\right|\mathit{Z}\},\hfill \end{array}$$

(19)

where

$${\Omega }_k=\left\{\begin{array}{cc}1\hfill & \mathrm{if}k\in \mathcal{A},\hfill \\ W_k\hfill & \mathrm{if}k\in \mathcal{B},\hfill \\ E_k\hfill & \mathrm{if}k\in \mathcal{C}\cup \mathcal{AB}.\hfill \end{array}\right.$$

(20)

Plugging the above into (9), we find that

$$\begin{array}{cc}\hfill \mathbb{E}\left\{R^A(g^A,g^B)\right|\mathit{Z}\}& =\sum _{k\in \mathcal{A}}{q}_k^A+\sum _{k\in {\mathcal{A}}^c}{E}_k{p}_k^A-\frac{1}{2}\mathbb{E}\left\{\sum _{k\in \mathcal{A}}{\Omega }_k{q}_k^A{\mathbb{1}}_{\{{\alpha }^B=k\}}+\sum _{k\in {\mathcal{A}}^c}{\Omega }_k{p}_k^A{\mathbb{1}}_{\{{\alpha }^B=k\}}|\mathit{Z}\right\}.\hfill \end{array}$$

(21)

The difficulty in dealing with $\mathbb{E}\left\{R^A(g^A,g^B)\right|\mathit{Z}\}$ is that it depends on the strategy $g^B$ of player B, which is not known to player A. Hence, given a strategy $g^A$ of player A, we first focus on obtaining the worst-case strategy $\widehat{g^A}$ of player B. Then we focus on finding the strategy $g^A$ of player A, which maximizes $\mathbb{E}\left\{R^A(g^A,\widehat{g^A})\right|\mathit{Z}\}$. This way, we can guarantee a minimum expected utility for player A irrespective of player B’s strategy.

Lemma 2.

For given $g^A\in {\mathcal{S}}^A$, the strategy $g^B\in {\mathcal{S}}^B$ that minimizes $\mathbb{E}\left\{R^A(g^A,g^B)\right|\mathit{Z}\}$ chooses ${\alpha }^B$=$arg{max}_{1\le k\le n}{\Lambda }_k$, where

$$\begin{array}{c}\hfill {\Lambda }_k=\left\{\begin{array}{cc}{\Omega }_k{q}_k^A\hfill & ifk\in \mathcal{A},\hfill \\ {\Omega }_k{p}_k^A\hfill & ifk\in {\mathcal{A}}^c,\hfill \end{array}\right.\end{array}$$

(22)

and ${\Omega }_k$ are defined in (20).

Proof of Lemma 2.

Notice that the only term of $\mathbb{E}\left\{R^A(g^A,g^B)\right|\mathit{Z}\}$ in (21) that depends on the strategy of player B is the last expectation. This expectation is maximized when player B chooses k, for which ${\Lambda }_k$ is maximized.1    □

Hence, we have

$$\begin{array}{c}\hfill \mathbb{E}\left\{R^A(g^A,\widehat{g^A})\right|\mathit{Z}\}=\sum _{k\in \mathcal{A}}{q}_k^A+\sum _{k\in {\mathcal{A}}^c}{E}_k{p}_k^A-\frac{1}{2}\mathbb{E}\left\{\max \{{\Lambda }_k;1\le k\le n\}\right|\mathit{Z}\},\end{array}$$

(23)

where ${\Lambda }_k$ is defined in (22). We formulate a strategy for player A using the following optimization problem

$$\begin{array}{ccc}\left(\mathrm{P}1\right):\hfill & \mathrm{maximize}\hfill & f(\mathit{q},{\mathit{p}}_{a+1:n})\hfill \\ \hfill & g\in {\mathcal{S}}^A\hfill & \\ \hfill & \mathrm{subject}\mathrm{to}\hfill & \mathit{q}\in {\mathbb{R}}^a,\hfill \\ \hfill & \hfill & \mathit{p}\in {\mathbb{R}}^n,\hfill \\ \hfill & \hfill & q_k={\mathbb{E}}_{\mathit{W},U^A}\left\{W_k{\mathbb{1}}_{\left\{g(U^A,\mathit{X},\mathit{Z})=k\right\}}\right|\mathit{Z}\}\forall 1\le k\le a\hfill \\ \hfill & \hfill & p_l={\mathbb{E}}_{\mathit{W},U^A}\left\{{\mathbb{1}}_{\{g(U^A,\mathit{X},\mathit{Z})=l\}}\right|\mathit{Z}\}\forall 1\le l\le n,\hfill \end{array}$$

(24)

where $f:{\mathbb{R}}^n\to \mathbb{R}$ is defined by,

$$\begin{array}{cc}\hfill f\left(\mathit{x}\right)=& \sum _{k\in \mathcal{A}}{x}_k+\sum _{k\in {\mathcal{A}}^c}{E}_k{x}_k-\frac{1}{2}\mathbb{E}\left\{\max \{{\Omega }_j{x}_j;1\le j\le n\}\right|\mathit{Z}\}.\hfill \end{array}$$

(25)

Although not used immediately, we derive certain properties of f in the following theorem, which are useful later.

Theorem 2.

The function f

1.

is concave.

2.

is entry-wise non-decreasing.

3.

satisfies,

$$\begin{array}{c}\hfill |f\left(\mathit{x}\right)-f\left(\mathit{y}\right)|\le \frac{3}{2}\sum _{j\in \mathcal{A}}|x_j-y_j|+\frac{3}{2}\sum _{j\in {\mathcal{A}}^c}{E}_j|x_j-y_j|,\end{array}$$

(26)

for any $\mathit{x},\mathit{y}\in {\mathbb{R}}^n$.

Proof of Theorem 2.

See Appendix A.    □

It turns out that when $a=b=d=0$, an explicit solution can be obtained to (P1), which we describe in Section 5.1. In Section 5.2, we describe the solution to the general case. In the technical report [36], we provide simpler alternative solutions to the special cases $a=0$ (with no restriction on b) and $a=1$ (with the additional assumption that $W_1$ has a continuous CDF).

5.1. Explicit Solution for $a=b=d=0$

When neither player knows any of the reward realizations, we have $a=b=d=0$, and the problem reduces to the following.

$$\begin{array}{ccc}\left(\mathrm{P}2\right):\hfill & \mathrm{maximize}\hfill & {\displaystyle \sum _{k=1}^n}{p}_k{E}_k-\frac{1}{2}\max \{p_k{E}_k;1\le k\le n\}\hfill \\ \hfill & \mathrm{subject}\mathrm{to}\hfill & \mathit{p}\in \mathcal{I},\hfill \end{array}$$

(27)

where

$$\begin{array}{c}\hfill \mathcal{I}=\{\mathit{p}\in {\mathbb{R}}^n:\sum _{i=1}^n{p}_i=1,p_i\ge 0\forall i\}\end{array}$$

(28)

is the n-dimensional probability simplex. For this section, we assume without loss of generality that $E_k>0$ for all k. If at least one of the $E_k$’s is zero, we could transform (P2) into a lower dimensional problem with non-zero $E_k$’s. The following lemma constructs an explicit solution for $a=b=d=0$.2

Lemma 3.

Assume without loss of generality that $E_k\ge E_{k+1}$ for $1\le k\le n-1$. Further, let,

$$\begin{array}{c}\hfill r=arg\underset{1\le k\le n}{max}\frac{k-\frac{1}{2}}{{\sum }_{j=1}^k\frac{1}{E_j}},\end{array}$$

(29)

where the lowest index is chosen in the case of ties. The optimal solution for (P2) is given by ${\mathit{p}}^{\ast }$ where

$$\begin{array}{c}\hfill p_k^{\ast }=\left\{\begin{array}{cc}\frac{1}{E_k\left({\sum }_{j=1}^r\frac{1}{E_j}\right)}\hfill & \mathit{if}k\le r,\hfill \\ 0\hfill & \mathit{otherwise}.\hfill \end{array}\right.\end{array}$$

(30)

Proof of Lemma 3.

See Appendix B.    □

It should be noted that this solution is not unique. For instance, consider the case when $n=2$, $E_1=2$, and $E_2=1$. In this case, the lemma finds the solution $(p_1,p_2)=(1,0)$, but it should be noted that $(p_1,p_2)=(1/3,2/3)$ is also a solution. It is also interesting that the solution assigns positive probabilities to the r resources with the highest average reward, although within these r resources, higher probabilities are assigned to the resources with lower rewards.

It should also be noted that the worst-case strategy can be arbitrarily worse than the Nash equilibrium strategy. For instance, consider the simple scenario with two resources such that $E_1=E_2$, where none of the players observe any of the reward realizations. In this case, a Nash equilibrium would be player A always choosing resource 1 and player B always choosing resource 2. Another Nash equilibrium would be player B always choosing resource 1 and player A always choosing resource 2. In either case, player A’s expected utility is $E_1$. However, notice that, from Lemma 3, the maximum worst-case expected utility of player A is $3E_1{E}_2/(2E_1+2E_2)=3E_1/4$. Hence, $E_1$ can be scaled to obtain arbitrarily large deviation between the worst-case and the Nash equilibrium solutions.

5.2. Solving the General Case

In this section, we focus on solving the most general version of (P1) (with no restrictions on the sets $\mathcal{A},\mathcal{B},\mathcal{AB},\mathcal{C}$). In particular, we focus on finding a mixed strategy to optimize the worst-case expected utility for player A. It turns out that our optimal solution chooses from a mixture of pure strategies parameterized by $\mathit{Q}\in {\mathbb{R}}^n$, of the following form

$$\begin{array}{c}\hfill g_{\mathit{Q}}^A\left(\mathit{X}\right)=arg\underset{1\le j\le n}{max}\{\{Q_j{W}_j;j\in \mathcal{A}\}\cup \{Q_j;j\in {\mathcal{A}}^c\}\}.\end{array}$$

(31)

We name this special class of pure strategies as threshold strategies. We develop a novel algorithm to solve this problem. Our algorithm leverages techniques from drift-plus penalty theory [14] and online convex optimization [12,13]. It should be noted that our algorithm runs offline and is used to construct an appropriate strategy for player A that approximately solves (P1) conditioned on the observed realization of $\mathit{Z}$. We show that we can obtain values arbitrarily close to the optimal value of (P1) by using a finite equiprobable mixture of pure strategies of the above form. It should be noted that the algorithm developed in this section can be used to solve the general unconstrained problem of finding the randomized decision $\alpha \in \{1,2,\dots ,n\}$ which maximizes $\mathbb{E}\left\{h\right(\mathit{x};\mathbf{\Theta }\left)\right\}$, where $\mathit{x}\in {\mathbb{R}}^n$ with $x_k=\mathbb{E}\left\{{\Gamma }_k{\mathbb{1}}_{\{\alpha =k\}}\right\}$, $\mathbf{\Theta }\in {\mathbb{R}}^m$ and $\mathbf{\Gamma }\in {\mathbb{R}}^n$ are non-negative random vectors with finite second moments, and h is a concave function such that $\tilde{h}\left(\mathit{x}\right)=\mathbb{E}\left\{h(\mathit{x};\mathbf{\Theta })\right\}$ is Lipschitz continuous, entry-wise non-decreasing, and has bounded subgradients.

We first provide an algorithm that generates a mixture of T pure strategies, after which we establish the closeness to the optimality of the mixture. We generate a mixture of T pure strategies ${\left\{g_{\mathit{Q}\left(t\right)}^A\right\}}_{t=1}^T$ by iteratively updating vector $\mathit{Q}$ for T iterations, where $\mathit{Q}\left(t\right)$ and $g_{\mathit{Q}\left(t\right)}\left(\mathit{X}\right)$ denote the state of $\mathit{Q}$ and the pure strategy generated in the t-th iteration, respectively. In addition to $\mathit{Q}\left(t\right)$, we require another state vector $\mathit{\gamma }\left(t\right)\in {\mathbb{R}}^n$, which we also update in each iteration, and parameter V, which decides the convergence properties of the algorithm. We provide the specific details on setting V later in our analysis. We begin with $\mathit{Q}\left(1\right)=\mathit{\gamma }\left(0\right)=0$. In the t-th iteration $(t\ge 1)$, we independently sample $\mathit{X}\left(t\right)$ and $\mathbf{\Omega }\left(t\right)$ from the distributions of $\mathit{X}$ and $\mathbf{\Omega }$, respectively, where $\mathbf{\Omega }$ is defined in (20) while keeping $\mathit{Z}$ fixed to its observed value. Then, we update $\mathit{\gamma }\left(t\right)$ and $\mathit{Q}(t+1)$ as follows. First, we solve,

$$\begin{array}{}\mathrm{(32a)}& (\mathrm{P}3):\underset{\mathit{\gamma }\left(t\right)}{\mathrm{minimize}}-V{f}_t^{{}^{\prime }}{\left(\mathit{\gamma }(t-1)\right)}^{\top }\mathit{\gamma }\left(t\right)+\alpha {\parallel \mathit{\gamma }\left(t\right)-\mathit{\gamma }(t-1)\parallel }_2^2+{\displaystyle \sum _{j=1}^n}{Q}_j\left(t\right){\gamma }_j\left(t\right)\hfill \mathrm{(32b)}& \mathrm{subject}\mathrm{to} \mathit{\gamma }\left(t\right)\in \mathcal{K},\hfill \end{array}$$

to find $\mathit{\gamma }\left(t\right)$, where

$$\begin{array}{c}\hfill f_t\left(\mathit{x}\right)=\sum _{k\in \mathcal{A}}{x}_k+\sum _{k\in {\mathcal{A}}^c}{x}_k{E}_k-\frac{1}{2}\max \{x_k{\Omega }_k\left(t\right);1\le k\le n\},\end{array}$$

(33)

$\alpha >0$ and $\mathcal{K}=\left({✕}_{j\in \mathcal{A}}[0,E_j]\right)\times {[0,1]}^{n-a}$. Notice that $f_t^{{}^{\prime }}\left(\mathit{x}\right)$ is given by,

$$\begin{array}{c}\hfill f_{t,j}^{{}^{\prime }}\left(\mathit{x}\right)=\left\{\begin{array}{cc}1-\frac{1}{2}{\mathbb{1}}_{\{arg{max}_{1\le k\le n}\left\{x_k{\Omega }_k\left(t\right)\right\}=j\}}\hfill & \mathrm{if}j\in \mathcal{A},\hfill \\ E_j-\frac{1}{2}{\mathbb{1}}_{\{arg{max}_{1\le k\le n}\left\{x_k{\Omega }_k\left(t\right)\right\}=j\}}{\Omega }_j\left(t\right)\hfill & \mathrm{if}j\in {\mathcal{A}}^c,\hfill \end{array}\right.\end{array}$$

(34)

where $argmax$ returns the lowest index in the case of ties. Notice that $f_t$ is a concave function, which can be established by repeating the same argument used to establish the concavity of f in Theorem 2. Then, we choose the action for the t-th iteration ${\alpha }^A\left(t\right)=g_{\mathit{Q}\left(t\right)}^A\left(\mathit{X}\left(t\right)\right)$ (See (31)). Then, to update $\mathit{Q}(t+1)$, we use,

$$\begin{array}{cc}\hfill & Q_j(t+1)=\max \left\{Q_j\left(t\right)+{\gamma }_j\left(t\right)-X_j\left(t\right){\mathbb{1}}_{\{{\alpha }^A\left(t\right)=j\}},0\right\},\forall j\in \mathcal{A},\hfill \\ & Q_j(t+1)=\max \left\{Q_j\left(t\right)+{\gamma }_j\left(t\right)-{\mathbb{1}}_{\{{\alpha }^A\left(t\right)=j\}},0\right\},\forall j\in {\mathcal{A}}^c.\hfill \end{array}$$

(35)

The algorithm is summarized as Algorithm 1 for clarity.

Algorithm 1: Algorithm for the generation of the optimal mixture of T pure strategies.

After creating the mixture ${\left\{g_{\mathit{Q}\left(t\right)}^A\right\}}_{t=1}^T$ of pure strategies, we choose one of them randomly with probability $1/T$ to take the decision. In the following two sections, we focus on solving (P3) and evaluating the performance of Algorithm 1.

5.2.1. Solving (P3)

Notice that the objective of (P3) can be written as

$$\begin{array}{cc}\hfill & -V{f}_t^{{}^{\prime }}{\left(\mathit{\gamma }(t-1)\right)}^{\top }\mathit{\gamma }\left(t\right)+\alpha {\parallel \mathit{\gamma }\left(t\right)-\mathit{\gamma }(t-1)\parallel }_2^2+\sum _{j=1}^n{Q}_j\left(t\right){\gamma }_j\left(t\right)\hfill \\ & =\sum _{j=1}^n\left\{-V{f}_{t,j}^{{}^{\prime }}\left(\mathit{\gamma }(t-1)\right){\gamma }_j\left(t\right)+\alpha {({\gamma }_j\left(t\right)-{\gamma }_j(t-1))}^2+Q_j\left(t\right){\gamma }_j\left(t\right)\right\}.\hfill \end{array}$$

(36)

Hence (P3) seeks to minimize a separable convex function over the box constraint $\mathit{\gamma }\left(t\right)\in \mathcal{K}$. The solution vector $\mathit{\gamma }\left(t\right)$ is found by separately minimizing each component ${\gamma }_j\left(t\right)$ over $[0,u_j]$, where

$$\begin{array}{c}\hfill u_j=\left\{\begin{array}{cc}{E}_j\hfill & \mathrm{if}j\in \mathcal{A},\hfill \\ 1\hfill & \mathrm{if}j\in {\mathcal{A}}^c.\hfill \end{array}\right.\end{array}$$

(37)

The resulting solution is,

$$\begin{array}{c}\hfill {\gamma }_j\left(t\right)={\Pi }_{[0,u_j]}\left({\gamma }_j(t-1)-\frac{-V{f}_{t,j}^{{}^{\prime }}\left(\mathit{\gamma }(t-1)\right)+Q_j\left(t\right)}{2\alpha }\right),\end{array}$$

(38)

where ${\Pi }_{[0,u_j]}$ denotes the projection onto $[0,u_j]$. Notice that the above solution is obtained by projecting the global minimizer of the function to be minimized onto $[0,u_j]$.

5.2.2. How Good Is the Mixed Strategy Generated by Algorithm 1

Without loss of generality, we assume that $E_k>0$ for all $1\le k\le n$. The following theorem establishes the closeness of the expected utility generated by Algorithm 1 to the optimal value $f^{\mathrm{opt}}$ of (P1).

Theorem 3.

Assume α is set such that $\alpha \ge V^2$, and we use the mixed strategy $g^A$ generated by Algorithm 1 to make the decision. Then,

$$\begin{array}{cc}\hfill \mathbb{E}\left\{R^A(g^A,\widehat{g^A})\right|\mathit{Z}\}\ge & f^{\mathrm{opt}}-\frac{D_1}{V}-\frac{V{D}_2}{16\alpha }-\frac{\alpha D_3}{VT}-\frac{3}{2T}\sum _{k\in \mathcal{A}}\left\{\sqrt{\alpha }+E_k\left(2\sqrt{2\alpha }+1\right)\right\}\hfill \\ & -\frac{3}{2T}\sum _{k\in {\mathcal{A}}^c}\left\{E_k^2\sqrt{\alpha }+E_k\left(2\sqrt{2\alpha }+1\right)\right\},\hfill \end{array}$$

(39)

where

$$\begin{array}{cc}\hfill & D_1=n-a+\frac{1}{2}\sum _{j\in \mathcal{A}}(E_j^2+\mathbb{E}\left\{W_k^2\right\}),\hfill \\ \hfill & D_2=4a+\mathbb{E}\left\{{\parallel \mathbf{\Omega }\parallel }_2^2\right|\mathit{Z}\}+\sum _{j\in {\mathcal{A}}^c}4E_j^2,\hfill \\ \hfill & D_3=n-a+\sum _{j\in \mathcal{A}}{E}_j^2,\hfill \end{array}$$

(40)

Ω is defined in (20), and $f^{\mathrm{opt}}$ is the optimal value of (P1). Hence, by fixing $\epsilon >0$, and using $V=1/\epsilon$, $\alpha =1/{\epsilon }^2$, and $T\ge 1/{\epsilon }^2$, the average error is $\mathcal{O}\left(\epsilon \right)$.

Proof of Theorem 3.

The key to the proof is noticing that $\mathit{Q}\left(t\right)$ can be treated as n queues. Before proceeding with the proof, we define some quantities. Define the history up to time t by $\mathcal{H}\left(t\right)=\left\{\mathit{X}\right(\tau );1\le \tau <t\}\cup \{\mathbf{\Omega }(\tau );1\le \tau \le t\}$. Notice that we include $\mathbf{\Omega }\left(t\right)$ in $\mathcal{H}\left(t\right)$ since this will allow us to treat $\mathit{\gamma }\left(t\right)$ and $\mathit{Q}\left(t\right)$ as deterministic functions of $\mathcal{H}\left(t\right)$ and $\mathit{Z}$. Let us define the Lyapunov function $L\left(t\right)=\frac{1}{2}{\left|\right|\mathit{Q}\left(t\right)\left|\right|}^2=\frac{1}{2}{\sum }_{j=1}^n{Q}_j{\left(t\right)}^2$, and the drift $\Delta \left(t\right)=\mathbb{E}\left\{L\right(t+1)-L(t\left)\right|\mathcal{H}\left(t\right),\mathit{Z}\}$. Now, notice that

$$\begin{array}{c}\hfill \mathbb{E}\left\{R^A(g^A,\widehat{g^A})\right|\mathit{Z}\}=f\left(\frac{1}{T}\sum _{t=1}^T\mathbb{E}\left\{\mathit{x}\left(t\right)\right|\mathit{Z}\}\right),\end{array}$$

(41)

where

$$\begin{array}{c}\hfill x_k\left(t\right)=\left\{\begin{array}{cc}{X}_k\left(t\right){\mathbb{1}}_{\{g_{\mathit{Q}\left(t\right)}^A\left(\mathit{X}\left(t\right)\right)=k\}}\hfill & \mathrm{if}k\in \mathcal{A},\hfill \\ {\mathbb{1}}_{\{g_{\mathit{Q}\left(t\right)}^A\left(\mathit{X}\left(t\right)\right)=k\}}\hfill & \mathrm{if}k\in {\mathcal{A}}^c.\hfill \end{array}\right.\end{array}$$

(42)

□

We begin with the following two lemmas, which will be useful in the proof.

Lemma 4.

The drift is bounded above as

$$\begin{array}{c}\hfill \Delta \left(t\right)\le D_1+\sum _{j=1}^n{Q}_j\left(t\right)({\gamma }_j\left(t\right)-\mathbb{E}\left\{x_j\left(t\right)\right|\mathcal{H}\left(t\right),\mathit{Z}\}),\end{array}$$

(43)

where $D_1$ is defined in (40).

Proof of Lemma 4.

See Appendix C. □

The following is a well-known result regarding the minimization of strongly convex functions (see, for example, a more general pushback result in [38]).

Lemma 5.

For a convex function $h:{\mathbb{R}}^n\to \mathbb{R}$, a convex subset $\mathcal{C}$ of ${\mathbb{R}}^n$, $\mathit{y}\in {\mathbb{R}}^n$ and $\alpha >0$, let,

$$\begin{array}{c}\hfill {\mathit{x}}^{\ast }\in arg\underset{\mathit{x}\in \mathcal{C}}{min}\left[h\left(\mathit{x}\right)+\alpha {\parallel \mathit{x}-\mathit{y}\parallel }_2^2\right].\end{array}$$

(44)

Then,

$$\begin{array}{c}\hfill h\left({\mathit{x}}^{\ast }\right)+\alpha {\parallel {\mathit{x}}^{\ast }-\mathit{y}\parallel }_2^2\le h\left(\mathit{z}\right)+\alpha {\parallel \mathit{z}-\mathit{y}\parallel }_2^2-\alpha {\parallel \mathit{z}-{\mathit{x}}^{\ast }\parallel }_2^2,\end{array}$$

(45)

for all $\mathit{z}\in \mathcal{C}$.

Now, we move on to the main proof. Notice that the objective of (P3) can be written as

$$\begin{array}{c}\hfill g_t^{{}^{\prime }}{\left(\mathit{\gamma }(t-1)\right)}^{\top }\mathit{\gamma }\left(t\right)+\alpha {\parallel \mathit{\gamma }\left(t\right)-\mathit{\gamma }(t-1)\parallel }_2^2,\end{array}$$

(46)

where

$$\begin{array}{c}\hfill g_t\left(\mathit{x}\right)=-V{f}_t\left(\mathit{x}\right)+\sum _{j=1}^n{Q}_j\left(t\right)x_j.\end{array}$$

(47)

Let $g^{A,\ast }$ be the strategy that is optimal for (P1). Let us define ${\mathit{x}}^{\ast }\left(t\right)\in {\mathbb{R}}^n$, where

$$x_k^{\ast }\left(t\right)=\left\{\begin{array}{}\mathrm{(48a)}& X_k\left(t\right){\mathbb{1}}_{\{g^{A,\ast }(U^A\left(t\right),\mathit{X}\left(t\right),\mathit{Z})=k\}}\hfill & \mathrm{if}k\in \mathcal{A},\hfill \mathrm{(48b)}& {\mathbb{1}}_{\{g^{A,\ast }(U^A\left(t\right),\mathit{X}\left(t\right),\mathit{Z})=k\}}\hfill & \mathrm{if}k\in {\mathcal{A}}^c,\hfill \end{array}\right.$$

where $U^A\left(t\right)$ for $1\le t\le T$ is a collection of independent and identically distributed uniform $[0,1)$ random variables. Notice that ${\mathit{y}}^{\ast }=\mathbb{E}\left\{{\mathit{x}}^{\ast }\left(t\right)\right|\mathit{Z}\}$ is independent of t and belongs to $\mathcal{K}$. Hence, ${\mathit{y}}^{\ast }$ is feasible for (P3). Notice that

$$\begin{array}{cc}\hfill & -V{f}_t^{{}^{\prime }}{\left(\mathit{\gamma }(t-1)\right)}^{\top }\mathit{\gamma }\left(t\right)+\sum _{j=1}^n{Q}_j\left(t\right){\gamma }_j\left(t\right)+\alpha {\parallel \mathit{\gamma }\left(t\right)-\mathit{\gamma }(t-1)\parallel }_2^2\hfill \\ & =g_t^{{}^{\prime }}{\left(\mathit{\gamma }(t-1)\right)}^{\top }\mathit{\gamma }\left(t\right)+\alpha {\parallel \mathit{\gamma }\left(t\right)-\mathit{\gamma }(t-1)\parallel }_2^2\hfill \\ & {\le }_{\left(a\right)}{g}_t^{{}^{\prime }}{\left(\mathit{\gamma }(t-1)\right)}^{\top }{\mathit{y}}^{\ast }+\alpha {\parallel {\mathit{y}}^{\ast }-\mathit{\gamma }(t-1)\parallel }_2^2-\alpha {\parallel {\mathit{y}}^{\ast }-\mathit{\gamma }\left(t\right)\parallel }_2^2\hfill \\ & =-V{f}_t^{{}^{\prime }}{\left(\mathit{\gamma }(t-1)\right)}^{\top }{\mathit{y}}^{\ast }+\sum _{j=1}^n{Q}_j\left(t\right)y_j^{\ast }+\alpha {\parallel {\mathit{y}}^{\ast }-\mathit{\gamma }(t-1)\parallel }_2^2-\alpha {\parallel {\mathit{y}}^{\ast }-\mathit{\gamma }\left(t\right)\parallel }_2^2,\hfill \end{array}$$

(49)

where (a) follows from Lemma 5 for the convex function h given by $h\left(\mathit{x}\right)=g_t^{{}^{\prime }}{\left(\mathit{\gamma }(t-1)\right)}^{\top }\mathit{x}$, and $\mathcal{C}=\mathcal{K}$, since $\mathit{\gamma }\left(t\right)$ is the solution to (P3) and ${\mathit{y}}^{\ast }$ is feasible for (P3). Further, step 5 in each iteration of Algorithm 1 of finding the action can be represented as the maximization of

$$\begin{array}{c}\hfill \sum _{j\in \mathcal{A}}{Q}_j\left(t\right)\mathbb{E}\left\{X_j\left(t\right){\mathbb{1}}_{\{{\alpha }^A=j\}}\right|\mathcal{H}\left(t\right),\mathit{Z}\}+\sum _{j\in {\mathcal{A}}^c}{Q}_j\left(t\right)\mathbb{E}\left\{{\mathbb{1}}_{\{{\alpha }^A=j\}}\right|\mathcal{H}\left(t\right),\mathit{Z}\}\end{array}$$

(50)

over all possible actions ${\alpha }^A\in \{1,2,\dots ,n\}$ at time-slot t. Hence, comparing the scenario where $g_{\mathit{Q}\left(t\right)}^A$ is used in the t-th iteration with the scenario where $g^{A,\ast }$ is used with the randomization variable $U^A\left(t\right)$ in the t-th iteration, we have the inequality,

$$\begin{array}{cc}\hfill -\sum _{j=1}^n{Q}_j\left(t\right)\mathbb{E}\left\{x_j\left(t\right)\right|\mathcal{H}\left(t\right),\mathit{Z}\}& \le -\sum _{j=1}^n{Q}_j\left(t\right)\mathbb{E}\left\{x_j^{\ast }\left(t\right)\right|\mathcal{H}\left(t\right),\mathit{Z}\}=-\sum _{j=1}^n{Q}_j\left(t\right)y_j^{\ast },\hfill \end{array}$$

(51)

where the last equality follows since ${\mathit{x}}^{\ast }\left(t\right)$ is independent of $\mathcal{H}\left(t\right)$. Summing (49) and (51),

$$\begin{array}{cc}\hfill & -V{f}_t^{{}^{\prime }}{\left(\mathit{\gamma }(t-1)\right)}^{\top }\mathit{\gamma }\left(t\right)+\alpha {\parallel \mathit{\gamma }\left(t\right)-\mathit{\gamma }(t-1)\parallel }_2^2+\sum _{j=1}^n{Q}_j\left(t\right)({\gamma }_j\left(t\right)-\mathbb{E}\left\{x_j\left(t\right)\right|\mathcal{H}\left(t\right),\mathit{Z}\})\hfill \\ & \le -V{f}_t^{{}^{\prime }}{\left(\mathit{\gamma }(t-1)\right)}^{\top }{\mathit{y}}^{\ast }+\alpha {\parallel {\mathit{y}}^{\ast }-\mathit{\gamma }(t-1)\parallel }_2^2-\alpha {\parallel {\mathit{y}}^{\ast }-\mathit{\gamma }\left(t\right)\parallel }_2^2.\hfill \end{array}$$

(52)

Adding $D_1+V{f}_t^{{}^{\prime }}{\left(\mathit{\gamma }(t-1)\right)}^{\top }\mathit{\gamma }(t-1)$ to both sides and using Lemma 4 yields,

$$\begin{array}{cc}\hfill & \Delta \left(t\right)-V{f}_t^{{}^{\prime }}{\left(\mathit{\gamma }(t-1)\right)}^{\top }\left\{\mathit{\gamma }\left(t\right)-\mathit{\gamma }(t-1)\right\}+\alpha {\parallel \mathit{\gamma }\left(t\right)-\mathit{\gamma }(t-1)\parallel }_2^2\hfill \\ & \le D_1-V{f}_t^{{}^{\prime }}{\left(\mathit{\gamma }(t-1)\right)}^{\top }({\mathit{y}}^{\ast }-\mathit{\gamma }(t-1))+\alpha {\parallel {\mathit{y}}^{\ast }-\mathit{\gamma }(t-1)\parallel }_2^2-\alpha {\parallel {\mathit{y}}^{\ast }-\mathit{\gamma }\left(t\right)\parallel }_2^2\hfill \\ & \le D_1-V\{f_t\left({\mathit{y}}^{\ast }\right)-f_t\left(\mathit{\gamma }(t-1)\right)\}+\alpha {\parallel {\mathit{y}}^{\ast }-\mathit{\gamma }(t-1)\parallel }_2^2-\alpha {\parallel {\mathit{y}}^{\ast }-\mathit{\gamma }\left(t\right)\parallel }_2^2,\hfill \end{array}$$

(53)

where the last inequality follows from the sub-gradient inequality for the concave function $f_t$. Now, we introduce the following lemma.

Lemma 6.

We have

$$\begin{array}{cc}\hfill -V{f}_t^{{}^{\prime }}{\left(\mathit{\gamma }(t-1)\right)}^{\top }\left\{\mathit{\gamma }\left(t\right)-\mathit{\gamma }(t-1)\right\}+\alpha {\parallel \mathit{\gamma }\left(t\right)-\mathit{\gamma }(t-1)\parallel }_2^2\ge & -\frac{V^2}{4\alpha }\left(a+\sum _{j\in {\mathcal{A}}^c}{E}_j^2\right)\hfill \\ & -\frac{V^2}{16\alpha }{\parallel \Omega \left(t\right)\parallel }_2^2,\hfill \end{array}$$

(54)

Proof of Lemma 6.

See Appendix D. □

Substituting the bound from Lemma 6 in (53) we have that

$$\begin{array}{cc}\hfill & \Delta \left(t\right)-\frac{V^2}{4\alpha }\left(a+\sum _{j\in {\mathcal{A}}^c}{E}_j^2\right)-\frac{V^2}{16\alpha }{\parallel \mathbf{\Omega }\left(t\right)\parallel }_2^2\hfill \\ & \le D_1-V\{f_t\left({\mathit{y}}^{\ast }\right)-f_t\left(\mathit{\gamma }(t-1)\right)\}+\alpha {\parallel {\mathit{y}}^{\ast }-\mathit{\gamma }(t-1)\parallel }_2^2-\alpha {\parallel {\mathit{y}}^{\ast }-\mathit{\gamma }\left(t\right)\parallel }_2^2,\hfill \end{array}$$

(55)

The above holds for each $t\in \{1,2,\dots ,T\}$. Hence, we first take the expectation conditioned on $\mathit{Z}$ of both sides of the above expression, after which we sum from 1 to T, which results in,

$$\begin{array}{cc}\hfill & \mathbb{E}\left\{L(T+1)\right|\mathit{Z}\}-\mathbb{E}\left\{L\left(1\right)\right|\mathit{Z}\}-\frac{T{V}^2}{4\alpha }\left(a+\sum _{j\in {\mathcal{A}}^c}{E}_j^2\right)-\frac{T{V}^2}{16\alpha }\mathbb{E}\left\{{\parallel \mathbf{\Omega }\parallel }_2^2\right|\mathit{Z}\}\hfill \\ & \le D_{1}T-V\sum _{t=1}^T\mathbb{E}\left\{f_t\left({\mathit{y}}^{\ast }\right)\right|\mathit{Z}\}+V\sum _{t=1}^T\mathbb{E}\left\{f_t\left(\mathit{\gamma }(t-1)\right)\right|\mathit{Z}\}+\alpha \mathbb{E}\left\{{\parallel {\mathit{y}}^{\ast }-\mathit{\gamma }\left(0\right)\parallel }_2^2\right|\mathit{Z}\}\hfill \\ & -\alpha \mathbb{E}\left\{{\parallel {\mathit{y}}^{\ast }-\mathit{\gamma }\left(T\right)\parallel }_2^2\right|\mathit{Z}\}.\hfill \end{array}$$

(56)

Notice that

$$\begin{array}{c}\hfill \mathbb{E}\left\{f_t\left({\mathit{y}}^{\ast }\right)\right|\mathit{Z}\}=f\left({\mathit{y}}^{\ast }\right)=f^{\mathrm{opt}},\end{array}$$

(57)

where functions f and $f_t$ are defined in (25) and (33), respectively. Further, we have that

$$\begin{array}{cc}\hfill \mathbb{E}\{f_t(& \mathit{\gamma }(t-1)\left)\right|\mathit{Z}\}=\mathbb{E}\left\{{\mathbb{E}}_{\mathbf{\Omega }\left(t\right)}\left\{f_t\left(\mathit{\gamma }(t-1)\right)\right|\mathcal{H}(t-1),\mathit{Z}\}\right|\mathit{Z}\}{=}_{\left(a\right)}\mathbb{E}\left\{f\left(\mathit{\gamma }(t-1)\right)\right|\mathit{Z}\},\hfill \end{array}$$

(58)

where (a) follows from the definition of $f_t$ in (33), since $\mathit{\gamma }(t-1)$ is a function of $\mathcal{H}(t-1)$ and $\mathbf{\Omega }\left(t\right)$ is independent of $\mathcal{H}(t-1)$. Substituting (57) and (58) into (56), we have that

$$\begin{array}{cc}\hfill & \mathbb{E}\left\{L(T+1)\right|\mathit{Z}\}-\mathbb{E}\left\{L\left(1\right)\right|\mathit{Z}\}-\frac{T{V}^2}{4\alpha }\left(a+\sum _{j\in {\mathcal{A}}^c}{E}_j^2\right)-\frac{T{V}^2}{16\alpha }\mathbb{E}\left\{{\parallel \mathbf{\Omega }\parallel }_2^2\right|\mathit{Z}\}\hfill \\ & \le D_{1}T-VT{f}^{\mathrm{opt}}+V\sum _{t=1}^T\mathbb{E}\left\{f\left(\mathit{\gamma }(t-1)\right)\right|\mathit{Z}\}+\alpha \mathbb{E}\left\{{\parallel {\mathit{y}}^{\ast }-\mathit{\gamma }\left(0\right)\parallel }_2^2\right|\mathit{Z}\}-\alpha \mathbb{E}\left\{{\parallel {\mathit{y}}^{\ast }-\mathit{\gamma }\left(T\right)\parallel }_2^2\right|\mathit{Z}\}\hfill \\ & {\le }_{\left(a\right)}{D}_{1}T-VT{f}^{\mathrm{opt}}+V\sum _{t=1}^T\mathbb{E}\left\{f\left(\mathit{\gamma }(t-1)\right)\right|\mathit{Z}\}+\alpha \left(n-a+\sum _{k\in \mathcal{A}}{E}_k^2\right)\hfill \\ & \le D_{1}T-VT{f}^{\mathrm{opt}}+VTf\left(\frac{1}{T}\sum _{t=1}^T\mathbb{E}\left\{\mathit{\gamma }(t-1)\right|\mathit{Z}\}\right)+\alpha D_3,\hfill \end{array}$$

(59)

where (a) follows since ${\mathit{y}}^{\ast },\mathit{\gamma }\left(T\right),\mathit{\gamma }\left(0\right)\in \mathcal{K}$ and the last inequality follows from Jensen’s inequality on the concave function f. (See the definition of $D_3$ and $D_2$ in (40)). Since $\mathit{Q}\left(1\right)=0$ and $\mathbb{E}\left\{L\right(T+1\left)\right|\mathit{Z}\}\ge 0$, after some rearrangements above translates to,

$$\begin{array}{cc}\hfill f^{\mathrm{opt}}-\frac{D_1}{V}& -\frac{V{D}_2}{16\alpha }-\frac{\alpha D_3}{VT}\le f\left(\frac{1}{T}\sum _{t=0}^{T-1}\mathbb{E}\left\{\mathit{\gamma }\left(t\right)\right|\mathit{Z}\}\right),\hfill \end{array}$$

(60)

where $D_2$ is defined in (40). Now, we prove the following lemma.

Lemma 7.

We have

$$\begin{array}{cc}\hfill f\left(\frac{1}{T}\sum _{t=0}^{T-1}\mathbb{E}\left\{\mathit{\gamma }\left(t\right)\right|\mathit{Z}\}\right)\le & f\left(\frac{1}{T}\sum _{t=1}^T\mathbb{E}\left\{\mathit{x}\left(t\right)\right|\mathit{Z}\}\right)+\frac{3}{2T}\sum _{k\in \mathcal{A}}\left\{\sqrt{\alpha }+E_k\left(2\sqrt{2\alpha }+1\right)\right\}\hfill \\ & +\frac{3}{2T}\sum _{k\in {\mathcal{A}}^c}\left\{E_k^2\sqrt{\alpha }+E_k\left(2\sqrt{2\alpha }+1\right)\right\}.\hfill \end{array}$$

(61)

Proof of Lemma 7.

We first introduce the following two lemmas.

Lemma 8.

The queues $Q_j\left(t\right)$ for $1\le j\le n$ updated according to Algorithm 1 satisfy,

$$\begin{array}{c}\hfill \max \left\{\frac{1}{T}\sum _{t=1}^{T-1}\mathbb{E}\{{\gamma }_j\left(t\right)-x_j\left(t\right)|\mathit{Z}\},\mathbf{0}\right\}\le \frac{\mathbb{E}\left\{Q_j\left(T\right)\right|\mathit{Z}\}}{T}.\end{array}$$

(62)

Proof of Lemma 8.

See Appendix E. □

The following lemma is vital in constructing the $\mathcal{O}\left(\sqrt{\alpha }\right)$ bound on the queue sizes, which leads to the $\mathcal{O}(1/{\epsilon }^2)$ solution. It should be noted that an easier bound can be obtained on the queue sizes, which leads to a $\mathcal{O}(1/{\epsilon }^3)$ solution.

Lemma 9.

Given that $\alpha \ge V^2$, $\mathit{Q}\left(t\right)$ satisfy the bound

$$\begin{array}{c}\hfill Q_j\left(t\right)\le \left\{\begin{array}{cc}(1+2\sqrt{2}{E}_j)\sqrt{\alpha }+E_j\hfill & \mathit{if}j\in \mathcal{A}\hfill \\ (E_j+2\sqrt{2})\sqrt{\alpha }+1\hfill & \mathit{if}j\in {\mathcal{A}}^c,\hfill \end{array}\right.\end{array}$$

(63)

for each $t\in [1:T]$.

Proof of Lemma 9.

See Appendix F. □

Now, we move on to the main proof. Notice that

$$\begin{array}{cc}\hfill & f\left(\frac{1}{T}\sum _{t=0}^{T-1}\mathbb{E}\left\{\mathit{\gamma }\left(t\right)\right|\mathit{Z}\}\right)=f\left(\frac{\mathit{\gamma }\left(0\right)}{T}+\frac{1}{T}\sum _{t=1}^{T-1}\mathbb{E}\left\{\mathit{\gamma }\left(t\right)\right|\mathit{Z}\}\right)\hfill \\ & \le f\left(\frac{1}{T}\sum _{t=1}^T\mathbb{E}\left\{\mathit{x}\left(t\right)\right|\mathit{Z}\}+\frac{1}{T}\sum _{t=1}^{T-1}\mathbb{E}\left\{\mathit{\gamma }\left(t\right)\right|\mathit{Z}\}-\frac{1}{T}\sum _{t=1}^{T-1}\mathbb{E}\left\{\mathit{x}\left(t\right)\right|\mathit{Z}\}-\frac{\mathbb{E}\left\{\mathit{x}\right(T\left)\right|\mathit{Z}\}}{T}\right)\hfill \\ & {\le }_{\left(a\right)}f\left(\frac{1}{T}\sum _{t=1}^T\mathbb{E}\left\{\mathit{x}\left(t\right)\right|\mathit{Z}\}+\max \left\{\frac{1}{T}\sum _{t=1}^{T-1}\mathbb{E}\left\{\mathit{\gamma }\left(t\right)\right|\mathit{Z}\}-\frac{1}{T}\sum _{t=1}^{T-1}\mathbb{E}\left\{\mathit{x}\left(t\right)\right|\mathit{Z}\},\mathbf{0}\right\}\right)\hfill \\ & {\le }_{\left(b\right)}f\left(\frac{1}{T}\sum _{t=1}^T\mathbb{E}\left\{\mathit{x}\left(t\right)\right|\mathit{Z}\}\right)+\frac{3}{2}\sum _{k\in \mathcal{A}}\left(\max \left\{\frac{1}{T}\sum _{t=1}^{T-1}\mathbb{E}\{{\gamma }_k\left(t\right)-x_k\left(t\right)|\mathit{Z}\},\mathbf{0}\right\}\right)\hfill \\ & +\frac{3}{2}\sum _{k\in {\mathcal{A}}^c}{E}_k\left(\max \left\{\frac{1}{T}\sum _{t=1}^{T-1}\mathbb{E}\{{\gamma }_k\left(t\right)-x_k\left(t\right)|\mathit{Z}\},\mathbf{0}\right\}\right),\hfill \end{array}$$

(64)

where (a) follows from the entry-wise non-decreasing property of f (Theorem 2-2) and (b) follows from Theorem 2-3. Combining (64) and Lemma 8 with the bound on $\mathit{Q}\left(T\right)$ given by Lemma 9, we are finished with the proof of the lemma. □

Combining Lemma 7 with (60), we are finished with the proof of the theorem.

6. Simulations

For the simulations, we use $W_j$ as exponential random variables. Notice that since we are conditioning on $\mathit{Z}$ to solve the problem, the objective of (P1) defined in (25) has the same structure for the two scenarios $(a,b,c,d)$ and $(a,b,c+d,0)$. Hence, we use $d=0$ for all the simulations. Notice that the sets $\mathcal{A}$ and $\mathcal{B}$ denote the private information of players A and B, respectively. We consider the three scenarios given below.

• $a=0,b=0,c=3,d=0$: Both players do not have private information.
• $a=0,b=1,c=2,d=0$: Only player B has private information.
• $a=1,b=1,c=1,d=0$: Both players have private information.

Figure 1, Figure 2 and Figure 3 show pictorial representations of these cases.

We first consider scenario 1. For Figure 4 (top-left), we fix $E_2=E_3=1$ and plot the expected utilities of players A and B at the $ϵ$-approximate Nash equilibrium as functions of $E_1$, where $ϵ={10}^{-3}$ is used. For Figure 4 (top-middle and top-right), we use the same configuration and plot a solution for the probabilities of choosing different resources as a function of $E_1$ at the $ϵ$-approximate Nash equilibrium for players A and B, respectively. For scenarios 2 and 3, Figure 4 (middle) and Figure 4 (bottom), have similar descriptions to scenario 1.

We consider the same three scenarios for the simulations on maximizing the worst-case expected utility. In each scenario, for the top figure, we fix $E_2=E_3=1$ and plot the maximum expected worst-case utility of player A as a function of $E_1$. For the bottom figure, we use the same configuration and plot a solution for the probabilities of choosing different resources for player A as a function of $E_1$. Notice that the solutions may not be unique, as discussed in Section 5.1. Additionally, for Figure 5 (top-middle and top-right), we also indicate the maximum possible error of the solution calculated using the error bound derived in Theorem 3. For scenarios 2 and 3, we have obtained the solutions by averaging over ${10}^2$ independent simulations. Further, we have used $T={10}^5$, $\alpha =4\times {10}^4$, and $V=2\times {10}^2$.

Notice that it is difficult to compare the worst-case strategy and the $ϵ$-approximate Nash equilibrium strategy in general since the first can be computed without any cooperation between the players, whereas computing the second requires cooperation among players. Further, as described in Section 5.1, the worst-case strategy can be arbitrarily worse than the Nash equilibrium strategy. Nevertheless, comparing Figure 4 (left) and Figure 5 (top), it can be seen that the worst-case strategy and the strategy at $ϵ$-approximate Nash equilibrium yield comparable expected utilities for player A when $E_1\ge 2$. For instance, in scenario 1, for $E_1\ge 2$, the approximate Nash equilibrium strategy coincides with the worst-case strategy of choosing resource 1 with probability 1. However, it should be noted that our algorithm for finding the $ϵ$-approximate Nash equilibrium does not necessarily converge to a socially optimal solution. For instance, in scenario 1, when $E_1=2$, player A chooses resource 1 with probability 1 and player B chooses resource 2 with probability 1 gives a higher utility for player A without changing the utility of player B.

In Figure 5, it is interesting to notice the variation in choice probabilities of different resources with $E_1$. Notice that in scenario 1, the choice probability of resource 1 is non-decreasing for $E_1\in [0.1,0.8]$, non-increasing for $E_1\in [0.8,1.9]$, and non-decreasing for $E_1\ge 1.9$. Similar behavior can also be observed for scenario 3. This is surprising since intuition suggests that the probability of choosing a resource should increase with the increasing mean of the reward random variable. However, notice that in scenarios 1 and 3, player B does not observe the reward realization of resource 1. This might force player A, playing for the worst case, to believe that player B increases the probability of choosing resource 1 with increasing $E_1$, as a result of which player A chooses resource 1 with a lower probability. Notice that the probability of choosing resource 1 in scenario 3 does not grow as fast as the other two. This is because player A observes $W_1$ and hence can refrain from choosing it when $W_1$ takes low values.

7. Conclusions

We have implemented the iterative best response algorithm to find the $ϵ$-approximate Nash equilibrium of a two-player stochastic resource-sharing game with asymmetric information. To handle situations where the players do not trust each other and place no assumptions on the incentives of the opponent, we solved the problem of maximizing the worst-case expected utility of the first player using a novel algorithm that combines drift-plus penalty theory and online optimization techniques. An explicit solution can be constructed when both players do not observe the realizations of any of the reward random variables. This special case leads to counter-intuitive insights.

In our approach, we have assumed that the reward random variables of different resources are independent. It should be noted that this assumption can be relaxed without affecting the analysis for the special case when both players do not observe the realizations of any of the reward random variables. An interesting question would be what happens in the general case when the reward random variables are not independent. While it is still possible to implement our algorithm in this setting, it is not guaranteed that the algorithm will converge to the optimal solution. Hence, finding an algorithm for this case that exploits the correlations between the reward random variables could be potential future work.

Several other extensions can be considered as well. One would be considering a scenario with multiple players. The general multiplayer case yields a complex information structure since the set of resources has to be split into $2^m$ subsets, where m is the number of players. Additionally, the idea of conditioning on the common information is difficult to be adapted for this case. Nevertheless, various simplified schemes could be considered. One example would be a case with no common information. In this case, the set of resources is split into $m+1$ disjoint subsets where the i-th ($1\le i\le m$) subset is the subset of resources of which the i-th player observes the rewards, and the $m+1$-th subset is the subset of resources of which the rewards are observed by none of the players. Another interesting scenario is when no player observes any of the reward realizations. In both these cases, the expected utility can be calculated following a similar procedure to the two-player case, but finding the worst-case expected utility is difficult. Hence, we believe both cases could be potential future work. Another extension would be extending the algorithm to be implemented with a repeated game structure and in an online scenario.