Thompson Sampling for Stochastic Bandits with Noisy Contexts: An Information-Theoretic Regret Analysis

Abstract

We study stochastic linear contextual bandits (CB) where the agent observes a noisy version of the true context through a noise channel with unknown channel parameters. Our objective is to design an action policy that can “approximate” that of a Bayesian oracle that has access to the reward model and the noise channel parameter. We introduce a modified Thompson sampling algorithm and analyze its Bayesian cumulative regret with respect to the oracle action policy via information-theoretic tools. For Gaussian bandits with Gaussian context noise, our information-theoretic analysis shows that under certain conditions on the prior variance, the Bayesian cumulative regret scales as $\tilde{O}\left(m\sqrt{T}\right)$, where m is the dimension of the feature vector and T is the time horizon. We also consider the problem setting where the agent observes the true context with some delay after receiving the reward, and show that delayed true contexts lead to lower regret. Finally, we empirically demonstrate the performance of the proposed algorithms against baselines.

1. Introduction

Decision-making in the face of uncertainty is a widespread challenge found across various domains such as control and robotics [1], clinical trials [2], communications [3], and ecology [4]. To tackle this challenge, learning algorithms have been developed to uncover effective policies for optimal decision-making. One notable framework for addressing this is contextual bandits (CBs), which capture the essence of sequential decision-making by incorporating side information, termed context [5].

In the standard CB model, an agent interacts with the environment over numerous rounds. In each round, the environment presents a context to the agent based on which the agent chooses an action and receives a reward from the environment. The reward is stochastic, drawn from a probability distribution whose mean reward (which is a function of context-action pair) is unknown to the agent. The goal of the agent is to design a policy for action selection that can maximize the cumulative mean reward accrued over a T-length horizon.

In this paper, we focus on a CB model that assumes stochastic rewards with linear mean-reward functions, also called stochastic linear contextual bandits. Stochastic linear CB models find applications in various settings including internet advertisement selection [6], where the advertisement (i.e., action) and webpage features (i.e., context) are used to construct a linear predictor of the probability that a user clicks on a given advertisement, and article recommendation on web portals [7].

While most prior research on CBs has primarily focused on models with known exact contexts [8,9,10], in many real-world applications, the contexts are noisy, e.g., imprecise measurement of patient conditions in clinical trials, weather or stock market predictions. In such scenarios, when the exact contexts are unknown, the agent must utilize the observed noisy contexts to estimate the mean reward associated with the true context. However, this results in a biased estimate that renders the application of standard CB algorithms unsuitable. Consequently, recent efforts have been made to develop CB algorithms tailored to noisy context settings.

Related Works: ref. [11] considers a setting where there is a bounded zero-mean noise in the m-dimensional feature vector (denoted by $\varphi (a,c)$, where a is the action and c is the context) rather than in the context vector, and the agent observes only noisy features. For this setting, they develop an upper confidence bound (UCB) algorithm. Ref. [12] models the uncertainty regarding the true contexts by a context distribution that is known to the agent, while the agent never observes the true context and develops a UCB algorithm. A similar setting has also been considered in [13]. Differing from these works, ref. [14] considers the setting where the true feature vectors are sampled from an unknown feature distribution at each time, but the agent observes only a noisy feature vector. Assuming Gaussian feature noise with unknown mean and covariance, they develop an Optimism in the Face of Uncertainty (OFUL) algorithm. A variant of this setting has been studied in [15].

Motivation and Problem Setting: In this work, inspired by [14], we consider the following noisy CB setting. In each round, the environment samples a true context vector $c_t$ from a context distribution that is known to the agent. The agent, however, does not observe the true context but observes a noisy context ${\hat{c}}_t$ obtained as the output of a noise channel $P\left({\hat{c}}_t\right|c_t,{\gamma }^{*})$ parameterized by ${\gamma }^{*}$. The agent is aware of the noise present but does not know the channel parameter ${\gamma }^{*}$. Following [14], we consider Gaussian noise channels for our regret analysis.

Based on the observed noisy contexts, the agent chooses an action $a_t$ and observes a reward $r_t$ corresponding to the true context. We consider a linear bandit whose mean reward $\varphi {(a_t,c_t)}^{\top }{\theta }^{*}$ is determined by an unknown reward parameter ${\theta }^{*}$. The goal of the agent is to design an action policy that minimizes the Bayesian cumulative regret with respect to the action policy of a Bayesian oracle. The oracle has access to the reward model and the channel parameter ${\gamma }^{*}$, and uses the predictive distribution of the true context given the observed noisy context to select an action.

Our setting differs from [14] in that we assume noisy contexts rather than noisy feature vectors and that the agent knows the context distribution. The noise model, incorporating noise in the feature vector, allows [14] to transform the original problem into a different CB problem that estimates a modified reward parameter. Such a transformation, however, is not straightforward in our setting with noise in contexts rather than in feature vectors, where we wish to analyze the Bayesian regret. Additionally, we propose a de-noising approach to estimate the predictive distribution of the true context from given noisy contexts, offering potential benefits for future analyses.

The assumption of known context distribution follows from [12]. This can be motivated by considering the example of an online recommendation engine that pre-processes the user account registration information or contexts (e.g., age, gender, device, location, item preferences) to group them into different clusters [16]. The engine can then infer the ‘empirical’ distribution of users within each cluster to define a context distribution over true contextual information. A noisy contextual information scenario occurs when a guest with different preferences logs into a user’s account.

Challenges and Novelty: Different from existing works that developed UCB-based algorithms, we propose a fully Bayesian Thompson Sampling (TS) algorithm that approximates the Bayesian oracle policy. The proposed algorithm differs from the standard contextual TS [10] in the following aspects. Firstly, since the true context vectors are not accessible at each round and the channel parameter ${\gamma }^{*}$ is unknown, the agent uses its knowledge of the context distribution and the past observed noisy contexts to infer a predictive posterior distribution of the true context from the current observed noisy context. The inferred predictive distribution is then used to choose the action. This de-noising step enables our algorithm to ‘approximate’ the oracle action policy that uses knowledge of the channel parameter ${\gamma }^{*}$ to implement exact de-noising. Secondly, the reward $r_t$ received by the agent corresponds to the unobserved true context $c_t$. Hence, the agent cannot accurately evaluate the posterior distribution of ${\theta }^{*}$ and sample from it as is conducted in standard contextual TS. Instead, our algorithm proposes to use a sampling distribution that ‘approximates’ the posterior.

Different from existing works that focus on frequentist regret analysis, we derive novel information-theoretic bounds on the Bayesian cumulative regret of our algorithm. For Gaussian bandits, our information-theoretic regret bounds scale as $\tilde{O}\left(m\sqrt{T}\right)$ (the notation $\tilde{O}(•)$ suppresses logarithmic terms in •), where $m,T$ denote the dimension of the feature vector and time horizon respectively, under certain conditions on the variance of the prior on ${\theta }^{*}$. Furthermore, our Bayesian regret analysis shows that the posterior mismatch, resulting due to replacing the true posterior distribution with a sampling distribution, results in an approximation error that is captured via the Kullback–Leibler (KL) divergence between the distributions. To the best of our knowledge, quantifying the posterior mismatch via KL divergence has not been studied before and is of independent interest.

Finally, we also extend our algorithm to a setting where the agent observes the true context after the decision is made and reward is observed [12]. We call this setting CBs with delayed true contexts. Such scenarios arise in many applications where only a prediction of the context is available at the time of decision-making; however, the true context is available later. For instance, in farming-recommender systems where, at the time of making the decision regarding which crop to cultivate in a year, the true contextual information about the weather pattern is unavailable, while some ‘noisy’ weather predictions are available. In fact, the true weather pattern is observed only after the decision is made. We show that our TS algorithm for this setting with delayed true contexts results in reduced Bayesian regret.

Table 1. Comparison of the regret bounds of our proposed TS algorithm for noisy CB with state-of-the art algorithms.

Reference	Setting	Algorithm	Regret	Bound
[8]	Linear CB	LinRel	Frequentist	$\tilde{O}\left(\sqrt{mT}\right)$
[9]	Linear CB	Lin-UCB	Frequentist	$\tilde{O}\left(\sqrt{mT}\right)$
[10]	Linear CB	TS	Frequentist	$O\left(m\sqrt{T}{log}^{3/2}T\right)$
[17]	Linear CB	TS	Bayesian	$O(m\sqrt{T}logT)$
[11]	Noisy CB	SampLinUCB	Frequentist	$\tilde{O}\left(m\sqrt{T}\right)$
[12]	Noisy CB	UCB	Frequentist	$\tilde{O}\left(m\sqrt{T}\right)$
[14]	Noisy CB	OFUL	Frequentist	$\tilde{O}\left(m\sqrt{T}\right)$
Our work	Noisy CB	TS	Bayesian	$\tilde{O}\left(m\sqrt{T}\right)$

compares our regret bound with that of the state-of-the-art algorithms in the noiseless and noisy CB settings.

2. Problem Setting

In this section, we present the stochastic linear CB problem studied in this paper. Let $\mathcal{A}$ denote the action set with K actions and $\mathcal{C}$ denote the (possibly infinite) set of d-dimensional context vectors. At iteration $t\in \mathbb{N}$, the environment randomly draws a context vector $c_t\in \mathcal{C}$ according to a context distribution $P\left(c\right)$ defined over the space $\mathcal{C}$ of context vectors. The context distribution $P\left(c\right)$ is known to the agent. The agent, however, does not observe the true context $c_t$ drawn by the environment. Instead, it observes a noisy version ${\hat{c}}_t$ of the true context, obtained as the output of a noisy, stochastic channel $P\left({\hat{c}}_t\right|c_t,{\gamma }^{*})$ with the true context $c_t$ as the input. The noise channel $P\left({\hat{c}}_t\right|c_t,{\gamma }^{*})$ is parameterized by the noise channel parameter ${\gamma }^{*}$ that is unknown to the agent.

Having observed the noisy context ${\hat{c}}_t$ at iteration t, the agent chooses an action $a_t\in \mathcal{A}$ according to an action policy ${\pi }_t(·|{\hat{c}}_t)$. The action policy may be stochastic describing a probability distribution over the set $\mathcal{A}$ of actions. Corresponding to the chosen action $a_t$, the agent receives a reward from the environment given by

$$\begin{array}{c}\hfill r_t=f({\theta }^{*},a_t,c_t)+{\xi }_t,\end{array}$$

(1)

where $f({\theta }^{*},a_t,c_t)=\varphi {(a_t,c_t)}^{\top }{\theta }^{*}$ is the linear mean-reward function and ${\xi }_t$ is a zero-mean reward noise variable. The mean reward function $f({\theta }^{*},a_t,c_t)$ is defined via the feature map $\varphi :\mathcal{A}\times \mathcal{C}\to {\mathbb{R}}^m$, that maps the action and true context to an m-dimensional feature vector, and via the reward parameter ${\theta }^{*}\in {\mathbb{R}}^m$ that is unknown to the agent.

We call the noisy CB problem described above CBs with unobserved true context (see Setting 1) since the agent does not observe the true context $c_t$ and the selection of action is based solely on the observed noisy context. Accordingly, at the end of iteration t, the agent has accrued the history ${\mathcal{H}}_{t,r,a,\hat{c}}={\{r_{\tau },a_{\tau },{\hat{c}}_{\tau }\}}_{\tau =1}^t$ of observed reward-action-noisy context tuples. The action policy ${\pi }_{t+1}(·|{\hat{c}}_{t+1})$ at ${(t+1)}^{\mathrm{th}}$ iteration may depend on the history ${\mathcal{H}}_{t,r,a,\hat{c}}$.

Setting 1: CBs with unobserved true contexts

1: for  $t=1,\dots ,T$  do 2:     Environment samples $c_t\sim P\left(c\right)$. 3:     Agent observes noisy context ${\hat{c}}_t\sim P\left({\hat{c}}_t\right|c_t,{\gamma }^{*})$. 4:     Agent chooses an action $a_t\sim {\pi }_t(·|{\hat{c}}_t)$. 5:     Agent receives reward $r_t$ according to (1). 6: end for

We also consider a variant of the above problem where the agent has access to a delayed observation of the true context $c_t$ as studied in [12]. We call this setting CBs with delayed true context. In this setting, at iteration t, the agent first observes a noisy context ${\hat{c}}_t$, chooses action $a_t\sim {\pi }_t(·|{\hat{c}}_t)$, and receives reward $r_t$. Later, the true context $c_t$ is observed. It is important to note that the agent has no access to the true context at the time of decision-making. Thus, at the end of iteration t, the agent has collected the history ${\mathcal{H}}_{t,r,a,c,\hat{c}}={\{r_{\tau },a_{\tau },c_{\tau },{\hat{c}}_{\tau }\}}_{\tau =1}^t$ of observed reward-action-context-noisy context tuples.

In both of the problem settings described above, the agent’s objective is to devise an action policy that minimizes the Bayesian cumulative regret with respect to a baseline action policy. We define Bayesian cumulative regret next.

Bayesian Cumulative Regret

The cumulative regret of an action policy ${\pi }_t(·|{\hat{c}}_t)$ quantifies how different the mean reward accumulated over T iterations is from that accrued by a baseline action policy ${\pi }_t^{*}(·|{\hat{c}}_t)$. In this work, we consider as baseline the action policy of an oracle that has access to the channel noise parameter ${\gamma }^{*}$, reward parameter ${\theta }^{*}$, the context distribution $P\left(c\right)$ and the noise channel likelihood $P\left(c_t\right|{\hat{c}}_t,{\gamma }^{*}).$ Accordingly, at each iteration t, the oracle can infer the exact predictive distribution $P\left(c_t\right|{\hat{c}}_t,{\gamma }^{*})$ of the true context from the observed noisy context ${\hat{c}}_t$ via Baye’s rule as

$$\begin{array}{c}\hfill P\left(c_t\right|{\hat{c}}_t,{\gamma }^{*})={\displaystyle \frac{P(c_t,{\hat{c}}_t|{\gamma }^{*})}{P\left({\hat{c}}_t\right|{\gamma }^{*})}}.\end{array}$$

(2)

Here, $P(c_t,{\hat{c}}_t|{\gamma }^{*})=P\left(c_t\right)P\left({\hat{c}}_t\right|c_t,{\gamma }^{*})$ is the joint distribution of the true and noisy contexts given the noise channel parameter ${\gamma }^{*}$, and $P\left({\hat{c}}_t\right|{\gamma }^{*})$ is the distribution obtained by marginalizing $P(c_t,{\hat{c}}_t|{\gamma }^{*})$ over the true contexts, i.e., 

$$\begin{array}{c}\hfill P\left({\hat{c}}_t\right|{\gamma }^{*})={\mathbb{E}}_{P\left(c_t\right)}\left[P\left({\hat{c}}_t\right|c_t,{\gamma }^{*})\right],\end{array}$$

(3)

where ${\mathbb{E}}_{•}[·]$ denotes expectation with respect to ‘$•$’. The oracle action policy then adopts an action

$$\begin{array}{cc}\hfill a_t^{*}& =arg\underset{a\in \mathcal{A}}{max}{\mathbb{E}}_{P\left(c_t\right|{\hat{c}}_t,{\gamma }^{*})}\left[\varphi {(a,c_t)}^{\top }{\theta }^{*}\right]\hfill \\ \hfill & =arg\underset{a\in \mathcal{A}}{max}\psi {(a,{\hat{c}}_t|{\gamma }^{*})}^{\top }{\theta }^{*},\hfill \end{array}$$

(4)

at iteration t, where $\psi (a,\hat{c}|{\gamma }^{*}):={\mathbb{E}}_{P\left(c\right|\hat{c},{\gamma }^{*})}\left[\varphi (a,c)\right]$. Note, that as in [14,18], we do not choose the stronger oracle action policy of $arg{max}_{a\in \mathcal{A}}$$\varphi {(a,c_t)}^{\top }{\theta }^{*}$, that requires access to the true context $c_t$, as it is generally not achievable by an agent that observes only noisy context ${\hat{c}}_t$ and has no access to ${\gamma }^{*}$.

For fixed parameters ${\theta }^{*}$ and ${\gamma }^{*}$, we define the cumulative regret of the action policy ${\pi }_t(·|{\hat{c}}_t)$ as

$$\begin{array}{c}\hfill {\mathcal{R}}^T\left(\pi \right|{\theta }^{*},{\gamma }^{*})=\sum _{t=1}^T\mathbb{E}[\varphi {(a_t^{*},c_t)}^{\top }{\theta }^{*}-\varphi {(a_t,c_t)}^{\top }{\theta }^{*}|{\theta }^{*},{\gamma }^{*}],\end{array}$$

(5)

the expected difference in mean rewards of the oracle decision policy and the agent’s decision policy over T iterations. In (5), the expectation is taken with respect to the joint distribution $P\left({\mathcal{H}}_{t-1,r,\hat{c},c,a}\right|{\theta }^{*},{\gamma }^{*})P({\hat{c}}_t,c_t,a_t|{\mathcal{H}}_{t-1,r,\hat{c},c,a},{\theta }^{*},{\gamma }^{*})$, where $P({\hat{c}}_t,c_t,a_t|{\mathcal{H}}_{t-1,r,\hat{c},c,a},{\theta }^{*},{\gamma }^{*})=P({\hat{c}}_t,c_t|{\gamma }^{*}){\pi }_t\left(a_t\right|{\hat{c}}_t)=P\left({\hat{c}}_t\right|{\gamma }^{*})P\left(c_t\right|{\hat{c}}_t,{\gamma }^{*}){\pi }_t\left(a_t\right|{\hat{c}}_t)$. Using this, the cumulative regret (5) can be written as

$$\begin{array}{cc}\hfill & {\mathcal{R}}^T\left(\pi \right|{\theta }^{*},{\gamma }^{*})\hfill \\ \hfill & =\sum _{t=1}^T{\mathbb{E}}_{P\left({\mathcal{H}}_{t-1,r,\hat{c},c,a}\right|{\theta }^{*},{\gamma }^{*})}\left[{\mathbb{E}}_{P\left({\hat{c}}_t\right|{\gamma }^{*}){\pi }_t\left(a_t\right|{\hat{c}}_t)}{\mathbb{E}}_{P\left(c_t\right|{\hat{c}}_t,{\gamma }^{*})}\left[\varphi {(a_t^{*},c_t)}^{\top }{\theta }^{*}-\varphi {(a_t,c_t)}^{\top }{\theta }^{*}\right]\right]\hfill \\ \hfill & =\sum _{t=1}^T{\mathbb{E}}_{P\left({\mathcal{H}}_{t-1,r,\hat{c},c,a}\right|{\theta }^{*},{\gamma }^{*})}\left[{\mathbb{E}}_{P\left({\hat{c}}_t\right|{\gamma }^{*}){\pi }_t\left(a_t\right|{\hat{c}}_t)}\left[\psi {(a_t^{*},{\hat{c}}_t|{\gamma }^{*})}^{\top }{\theta }^{*}-\psi {(a_t,{\hat{c}}_t|{\gamma }^{*})}^{\top }{\theta }^{*}\right]\right]\hfill \\ \hfill & :=\sum _{t=1}^T\mathbb{E}\left[\psi {(a_t^{*},{\hat{c}}_t|{\gamma }^{*})}^{\top }{\theta }^{*}-\psi {(a_t,{\hat{c}}_t|{\gamma }^{*})}^{\top }{\theta }^{*}|{\theta }^{*},{\gamma }^{*}\right].\hfill \end{array}$$

(6)

Our focus in this work is on a Bayesian framework where we assume that the reward parameter ${\theta }^{*}\in \Theta$ and channel noise parameter ${\gamma }^{*}\in \mathsf{\Gamma }$ are independently sampled by the environment from prior distributions $P\left({\theta }^{*}\right)$, defined on the set $\Theta$ of reward parameters, and $P\left({\gamma }^{*}\right)$, defined on the set $\mathsf{\Gamma }$ of channel noise parameters, respectively. The agent has knowledge of the prior distributions, the reward likelihood in (1) and the noise channel likelihood $P\left({\hat{c}}_t\right|c_t,{\gamma }^{*})$, although it does not observe the sampled ${\gamma }^{*}$ and ${\theta }^{*}$. Using the above prior distributions, we define Bayesian cumulative regret of the action policy ${\pi }_t(·|{\hat{c}}_t)$ as

$$\begin{array}{c}\hfill {\mathcal{R}}^T\left(\pi \right)=\mathbb{E}\left[{\mathcal{R}}^T\left(\pi \right|{\theta }^{*},{\gamma }^{*})\right],\end{array}$$

(7)

where the expectation is taken with respect to the priors $P\left({\theta }^{*}\right)$ and $P\left({\gamma }^{*}\right)$.

In the next sections, we present our novel TS algorithms to minimize the Bayesian cumulative regret for the two problem settings considered in this paper.

3. Modified TS for CB with Unobserved True Contexts

In this section, we consider Setting 1 where the agent only observes the noisy context ${\hat{c}}_t$ at each iteration t. Our proposed modified TS Algorithm is given in Algorithm 1.

Algorithm 1: TS with unobserved true contexts (${\pi }^{\mathrm{TS}}$)

1: for  $t=1,\dots ,T$  do 2:     The environment selects a true context $c_t$. 3:     Agent observes noisy context ${\hat{c}}_t$. 4:     Agent evaluates the predictive posterior distribution $P\left(c_t\right|{\hat{c}}_t,{\mathcal{H}}_{t-1,\hat{c}})$ as in (8). 5:     Agent samples ${\theta }_t\sim \overline{P}\left({\theta }^{*}\right|{\mathcal{H}}_{t-1,r,a,\hat{c}})$. 6:     Agent chooses action $a_t$ as in (11). 7:     Agent observes reward $r_t$ as in (1). 8: end for

The proposed algorithm implements two steps in each iteration $t\in \mathbb{N}$. In the first step, called the de-noising step, the agent uses the current observed noisy context ${\hat{c}}_t$ and the history ${\mathcal{H}}_{t-1,\hat{c}}={\left\{{\hat{c}}_{\tau }\right\}}_{\tau =1}^{t-1}$ of past observed noisy contexts to obtain a predictive posterior distribution $P\left(c_t\right|{\hat{c}}_t,{\mathcal{H}}_{t-1,\hat{c}})$ of the true context $c_t$. This is a two-step process, where firstly the agent uses the history ${\mathcal{H}}_{t-1,\hat{c}}$ of past observed noisy contexts to compute the posterior distribution of ${\gamma }^{*}$ as $P\left({\gamma }^{*}\right|{\mathcal{H}}_{t-1,\hat{c}})\propto P\left({\gamma }^{*}\right){\prod }_{\tau =1}^{t-1}P\left({\hat{c}}_{\tau }\right|{\gamma }^{*}),$ where the conditional distribution $P\left({\hat{c}}_t\right|{\gamma }^{*})$ is evaluated as in (3). Note, that to evaluate the posterior, the agent uses its knowledge of the context distribution $P\left(c\right)$, the prior $P\left({\gamma }^{*}\right)$ and the noise channel likelihood $P\left({\hat{c}}_t\right|c_t,{\gamma }^{*})$. Using the derived posterior $P\left({\gamma }^{*}\right|{\mathcal{H}}_{t-1,\hat{c}})$, the predictive posterior distribution of the true context is then obtained as

$$\begin{array}{c}\hfill P\left(c_t\right|{\hat{c}}_t,{\mathcal{H}}_{t-1,\hat{c}})={\mathbb{E}}_{P\left({\gamma }^{*}\right|{\mathcal{H}}_{t-1,\hat{c}})}\left[P\left(c_t\right|{\hat{c}}_t,{\gamma }^{*})\right],\end{array}$$

(8)

where $P\left(c_t\right|{\hat{c}}_t,{\gamma }^{*})$ is defined as in (2).

The second step of the algorithm implements a modified Thompson sampling. Note, that since the agent does not have access to the true contexts, it cannot evaluate the posterior distribution with known contexts,

$$\begin{array}{c}\hfill P\left({\theta }^{*}\right|{\mathcal{H}}_{t-1,r,a,c})\propto P\left({\theta }^{*}\right)\prod _{\tau =1}^{t-1}P\left(r_{\tau }\right|a_{\tau },c_{\tau },{\theta }^{*}),\end{array}$$

(9)

as is conducted in standard contextual TS. Instead, the agent must evaluate the true posterior distribution under noisy contexts,

$$\begin{array}{cc}\hfill P_t\left({\theta }^{*}\right)& :=P\left({\theta }^{*}\right|{\mathcal{H}}_{t-1,r,a,\hat{c}})\hfill \\ & \propto P\left({\theta }^{*}\right){\mathbb{E}}_{P\left({\gamma }^{*}\right)}\left[\prod _{\tau =1}^{t-1}{\mathbb{E}}_{P\left(c_{\tau }\right)}\left[P\left({\hat{c}}_{\tau }\right|c_{\tau },{\gamma }^{*})P\left(r_{\tau }\right|a_{\tau },c_{\tau },{\theta }^{*})\right]\right].\hfill \end{array}$$

(10)

However, evaluating the marginal distribution ${\mathbb{E}}_{P\left({\gamma }^{*}\right)}\left[{\prod }_{\tau =1}^{t-1}{\mathbb{E}}_{P\left(c_{\tau }\right)}\left[P\left({\hat{c}}_{\tau }\right|c_{\tau },{\gamma }^{*})P\left(r_{\tau }\right|a_{\tau },c_{\tau },{\theta }^{*})\right]\right]$ is challenging even for Gaussian bandits as the mean $\varphi {(a_{\tau },c_{\tau })}^{\top }{\theta }^{*}$ of the reward distribution $P\left(r_{\tau }\right|a_{\tau },c_{\tau },{\theta }^{*})$ is, in general, a non-linear function of the true context $c_{\tau }$. As a result, the posterior $P_t\left({\theta }^{*}\right)$ is analytically intractable.

Consequently, at each iteration t, the agent samples ${\theta }_t\sim \overline{P}\left({\theta }^{*}\right|{\mathcal{H}}_{t-1,r,a,\hat{c}})$ from a distribution $\overline{P}\left({\theta }^{*}\right|{\mathcal{H}}_{t-1,r,a,\hat{c}})$ that ‘approximates’ the true posterior $P_t\left({\theta }^{*}\right)$. The specific choice of this sampling distribution depends on the problem setting. Ideally, one must choose a distribution that is sufficiently ‘close’ to the true posterior. In the next sub-section, we will explain the choice for Gaussian bandits.

Using the sampled ${\theta }_t$ and the predictive posterior distribution $P\left(c_t\right|{\hat{c}}_t,{\mathcal{H}}_{t-1,\hat{c}})$ obtained from the denoising step, the agent then chooses action $a_t$ at iteration t as

$$\begin{array}{cc}\hfill a_t& =arg\underset{a\in \mathcal{A}}{max}\psi {(a,{\hat{c}}_t|{\mathcal{H}}_{\hat{c}})}^{\top }{\theta }_t,\mathrm{where}\hfill \end{array}$$

(11)

$$\begin{array}{cc}\hfill \psi (a_t,{\hat{c}}_t|{\mathcal{H}}_{\hat{c}})& :={\mathbb{E}}_{P\left(c_t\right|{\hat{c}}_t,{\mathcal{H}}_{t-1,\hat{c}})}\left[\varphi (a_t,c_t)\right]\hfill \end{array}$$

(12)

is the expected feature map with respect to $P\left(c_t\right|{\hat{c}}_t,{\mathcal{H}}_{t-1,\hat{c}})$.

3.1. Linear-Gaussian Stochastic CBs

We now instantiate Algorithm 1 for Gaussian CBs. Specifically, we consider Gaussian bandits with the reward noise ${\xi }_t$ in (1) as Gaussian $\mathcal{N}(0,{\sigma }^2)$ with mean 0 and variance ${\sigma }^2>0$. We also assume a Gaussian prior $P\left({\theta }^{*}\right)=\mathcal{N}(\mathbf{0},\lambda \mathbb{I})$ on the reward parameter ${\theta }^{*}$ with mean zero and an $m\times m$ diagonal, covariance matrix with entries $\lambda >0$. Here, $\mathbb{I}$ denotes the identity matrix. The assumption of diagonal prior covariance is in line with Lemma 3 in [19].

We consider a multivariate Gaussian context distribution $P\left(c\right)=\mathcal{N}({\mu }_c,{\Sigma }_c)$ with mean ${\mu }_c\in {\mathbb{R}}^d$ and covariance matrix ${\Sigma }_c\in {\mathbb{R}}^{d\times d}$. The context noise channel $P\left(\hat{c}\right|c,{\gamma }^{*})$ is also similarly Gaussian with a mean $({\gamma }^{*}+c)$ and covariance matrix ${\Sigma }_n\in {\mathbb{R}}^{d\times d}$. We assume the prior on noise channel parameter ${\gamma }^{*}$ to be Gaussian $P\left({\gamma }^{*}\right)=\mathcal{N}(\mathbf{0},{\Sigma }_{\gamma })$ with d-dimensional zero mean vector $\mathbf{0}$ and covariance matrix ${\Sigma }_{\gamma }\in {\mathbb{R}}^{d\times d}.$ We assume that ${\Sigma }_c,{\Sigma }_{\gamma }$ and ${\Sigma }_n$ are all positive definite matrices known to the agent. The assumption of positive definite covariance matrices is to facilitate the Bayesian analysis adopted in this work. Similar assumptions were also required in the related work of [14].

For this setting, we can analytically evaluate the predictive posterior distribution $P\left(c_t\right|{\hat{c}}_t,{\mathcal{H}}_{t-1,\hat{c}})=\mathcal{N}\left(c_t\right|V_t,R_t^{-1})$ as a multi-variate Gaussian with inverse covariance matrix,

$$\begin{array}{c}\hfill R_t=M-{\Sigma }_n^{-1}{\left(H_t^{-1}\right)}^{\top }{\Sigma }_n^{-1},\end{array}$$

(13)

where $H_t=(t-1){\Sigma }_n^{-1}-(t-2){\Sigma }_n^{-1}{M}^{-1}{\Sigma }_n^{-1}+{\Sigma }_{\gamma }^{-1}$ and $M={\Sigma }_c^{-1}+{\Sigma }_n^{-1}$, and with the mean vector

$$\begin{array}{c}\hfill {\displaystyle V_t={\left(R_t^{-1}\right)}^{\top }\left({\Sigma }_c^{-1}{\mu }_c+{\Sigma }_n^{-1}{\hat{c}}_t-{\Sigma }_n^{-1}{\left(H_t^{-1}\right)}^{\top }{L}_t^{\top }\right),}\end{array}$$

(14)

where

$$\begin{array}{cc}\hfill L_t^{\top }& ={\Sigma }_n^{-1}{M}^{-1}({\Sigma }_c^{-1}{\mu }_c+{\Sigma }_n^{-1}{\hat{c}}_t)+({\Sigma }_n^{-1}-{\Sigma }_n^{-1}{M}^{-1}{\Sigma }_n^{-1})\sum _{\tau =1}^{t-1}{\hat{c}}_{\tau }\hfill \\ & -(t-1){\Sigma }_n^{-1}{M}^{-1}{\Sigma }_c^{-1}{\mu }_c.\hfill \end{array}$$

Derivations are presented in Appendix C.1.2.

For the modified-TS step, we sample ${\theta }_t$ from the approximate posterior distribution

$$\begin{array}{c}\hfill {\overline{P}}_t\left({\theta }^{*}\right):=\overline{P}\left({\theta }^{*}\right|{\mathcal{H}}_{t-1,r,a,\hat{c}})\propto P\left({\theta }^{*}\right)\prod _{\tau =1}^{t-1}\overline{P}\left(r_{\tau }\right|a_{\tau },{\hat{c}}_{\tau },{\mathcal{H}}_{\tau -1,\hat{c}},{\theta }^{*}),\end{array}$$

(15)

where

$$\begin{array}{c}\hfill \overline{P}\left(r_{\tau }\right|a_{\tau },{\hat{c}}_{\tau },{\mathcal{H}}_{\tau -1,\hat{c}},{\theta }^{*})=\mathcal{N}(\psi {(a_{\tau },{\hat{c}}_{\tau }|{\mathcal{H}}_{\hat{c}})}^{\top }{\theta }^{*},{\sigma }^2)\end{array}$$

(16)

and $\psi (a_t,{\hat{c}}_t|{\mathcal{H}}_{\hat{c}})$ is the expected feature map defined in (12). This yields the approximate posterior to be a Gaussian distribution ${\overline{P}}_t\left({\theta }^{*}\right)=\mathcal{N}({\mu }_{t-1},{\Sigma }_{t-1}^{-1})$ whose inverse covariance matrix and mean, respectively, evaluate as

$$\begin{array}{cc}\hfill {\Sigma }_{t-1}& ={\displaystyle \frac{\mathbb{I}}{\lambda }}+{\displaystyle \frac{1}{{\sigma }^2}}\sum _{\tau =1}^{t-1}\psi (a_{\tau },{\hat{c}}_{\tau }|{\mathcal{H}}_{\hat{c}})\psi {(a_{\tau },{\hat{c}}_{\tau }|{\mathcal{H}}_{\hat{c}})}^{\top }\hfill \end{array}$$

(17)

$$\begin{array}{cc}\hfill {\mu }_{t-1}& ={\displaystyle \frac{{\Sigma }_{t-1}^{-1}}{{\sigma }^2}}\left(\sum _{\tau =1}^{t-1}{r}_{\tau }\psi (a_{\tau },{\hat{c}}_{\tau }|{\mathcal{H}}_{\hat{c}})\right).\hfill \end{array}$$

(18)

The sampling distribution ${\overline{P}}_t\left({\theta }^{*}\right)$ considered above is different from the true posterior distribution (10), which is analytically intractable. However, it bears resemblance to the posterior (9) when the true contexts are known, with the reward distribution $P\left(r_{\tau }\right|a_{\tau },c_{\tau },{\theta }^{*})$ replaced by $\overline{P}\left(r_{\tau }\right|a_{\tau },{\hat{c}}_{\tau },{\mathcal{H}}_{\tau -1,\hat{c}},{\theta }^{*})$. In Section 3.2.2, we show that the above choice of sampling distribution is indeed ‘close’ to the true posterior.

3.2. Bayesian Regret Analysis

In this section, we derive information-theoretic upper bounds on the Bayesian regret (7) of the modified TS algorithm for Gaussian CBs. To this end, we first outline the key information-theoretic tools required to derive our bound.

3.2.1. Preliminaries

To start, let $P\left(x\right)$ and $Q\left(x\right)$ denote two probability distributions defined over the space $\mathcal{X}$ of random variables x. Then, the Kullback–Leibler (KL)-divergence between the distributions $P\left(x\right)$ and $Q\left(x\right)$ is defined as

$$\begin{array}{c}\hfill {\displaystyle D_{\mathrm{KL}}\left(P\left(x\right)\right|\left|Q\left(x\right)\right)={\mathbb{E}}_{P\left(x\right)}\left[log\frac{P\left(x\right)}{Q\left(x\right)}\right]},\end{array}$$

(19)

if $P\left(x\right)$ is absolutely continuous with respect to $Q\left(x\right)$, and takes value ∞ otherwise. If x and y denote two random variables described by the joint probability distribution $P(x,y)$, the mutual information $I(x;y)$ between x and y is defined as $I(x;y)=D_{\mathrm{KL}}(P(x,y)\parallel P\left(x\right)P\left(y\right))$, where $P\left(x\right)$ (and $P\left(y\right)$) is the marginal distribution of x (and y). More generally, for three random variables x, y and z with joint distribution $P(x,y,z)$, the conditional mutual information $I(x;y|z)$ between x and y given z evaluates as

$$I(x;y|z)={\mathbb{E}}_{P\left(z\right)}\left[D_{KL}(P(x,y|z)\parallel P\left(x\right|z)P\left(y\right|z))\right]$$

where $P\left(x\right|z)$ and $P\left(y\right|z)$ are conditional distributions. We will also use the following variational representation of the KL-divergence, also termed the Donskar–Varadhan (DV) inequality,

$$\begin{array}{c}\hfill {\displaystyle D_{\mathrm{KL}}(P\left(x\right)\parallel Q\left(x\right))\ge {\mathbb{E}}_{P\left(x\right)}\left[f\left(x\right)\right]-log{\mathbb{E}}_{Q\left(x\right)}\left[\exp \left(f\left(x\right)\right)\right],}\end{array}$$

(20)

which holds for any measurable function $f:\mathcal{X}\to \mathbb{R}$ satifying the inequality ${\mathbb{E}}_{Q\left(x\right)}\left[\exp \left(f\left(x\right)\right)\right]$$<\infty$.

3.2.2. Information-Theoretic Bayesian Regret Bounds

In this section, we present information-theoretic upper bounds on the Bayesian regret of the modified TS algorithm. To this end, we first state our main assumption.

Assumption 1.

The feature map $\varphi (·,·)\in {\mathbb{R}}^m$ has bounded norm, i.e., ${\parallel \varphi (·,·)\parallel }_2\le 1$.

The following theorem gives our main result.

Theorem 1.

Assume that the covariance matrices satisfy ${\Sigma }_n{\Sigma }_c^{-1}\succ 0$ and ${\Sigma }_n{\Sigma }_{\gamma }^{-1}{\Sigma }_{n}M\succ 0$ where $M={\Sigma }_n^{-1}+{\Sigma }_c^{-1}$. Under Assumption 1, if ${\displaystyle \frac{\lambda }{{\sigma }^2}}\le {\displaystyle \frac{1}{T}}\le 1$, the following upper bound on the Bayesian regret of the modified TS algorithm holds,

$$\begin{array}{cc}\hfill & {\mathcal{R}}^T\left({\pi }^{\mathrm{TS}}\right)\le U(m,{\displaystyle \frac{{\sigma }^2}{T}})+\sqrt{2Tm{\sigma }^2}+\sqrt{2T{\sigma }^2(log\left(K\right)+m)}\hfill \\ & +{\displaystyle \frac{4}{T}}\sqrt{{\displaystyle \frac{m{\sigma }^2}{2T\pi }}}+2\sqrt{4{\sigma }^{2}mlog\left(2mT\right)\left(\mathrm{Tr}\left({\left({\Sigma }_n{\Sigma }_{\gamma }^{-1}{\Sigma }_{n}M\right)}^{-1}\right)+log\left(T\right)\mathrm{Tr}\left({\Sigma }_c{\Sigma }_n^{-1}\right)\right)},\hfill \end{array}$$

where

$$\begin{array}{cc}\hfill U(m,\lambda )& {\displaystyle =\sqrt{2Tm{\sigma }^{2}min\{m,2log(1+K)\}log\left(1+\frac{T\lambda }{m{\sigma }^2}\right)}}.\hfill \end{array}$$

(21)

The theorem above shows that the proposed TS algorithm achieves $\tilde{O}\left(m\sqrt{T}\right)$ regret when the prior $P\left({\theta }^{*}\right)$ is highly informative with variance parameter satisfying the constraint $\lambda \le {\sigma }^2/T$.

Remark 1.

The assumption on covariance matrices in Theorem 1 directly holds for diagonal covariance matrices with positive eigen values.

To prove the regret bound of Theorem 1, we start by defining

$$\begin{array}{c}\hfill {\hat{a}}_t=arg\underset{a\in \mathcal{A}}{max}\psi {(a,{\hat{c}}_t|{\mathcal{H}}_{\hat{c}})}^{\top }{\theta }^{*}\end{array}$$

(22)

as the action that maximizes the mean reward $\psi {(a,{\hat{c}}_t|{\mathcal{H}}_{\hat{c}})}^{\top }{\theta }^{*}$ corresponding to reward parameter ${\theta }^{*}$. Using the above, the Bayesian cumulative regret (7) for the proposed TS algorithm ${\pi }^{\mathrm{TS}}$ can be decomposed as

$$\begin{array}{cc}\hfill & {\mathcal{R}}^T\left({\pi }^{\mathrm{TS}}\right)={\mathcal{R}}_{\mathrm{CB}}^T+{\mathcal{R}}_{\mathrm{EE}1}^T+{\mathcal{R}}_{\mathrm{EE}2}^T,\mathrm{where}\hfill \\ \hfill & {\displaystyle {\mathcal{R}}_{\mathrm{CB}}^T=\sum _{t=1}^T\mathbb{E}\left[\psi {({\hat{a}}_t,{\hat{c}}_t|{\mathcal{H}}_{\hat{c}})}^{\top }{\theta }^{*}-\psi {(a_t,{\hat{c}}_t|{\mathcal{H}}_{\hat{c}})}^{\top }{\theta }^{*}\right]},\hfill \\ & {\displaystyle {\mathcal{R}}_{\mathrm{EE}1}^T=\sum _{t=1}^T\mathbb{E}\left[\psi {(a_t^{*},{\hat{c}}_t|{\gamma }^{*})}^{\top }{\theta }^{*}-\psi {({\hat{a}}_t,{\hat{c}}_t|{\mathcal{H}}_{\hat{c}})}^{\top }{\theta }^{*}\right],}\hfill \\ & {\displaystyle {\mathcal{R}}_{\mathrm{EE}2}^T=\sum _{t=1}^T\mathbb{E}\left[\psi {(a_t,{\hat{c}}_t|{\mathcal{H}}_{\hat{c}})}^{\top }{\theta }^{*}-\psi {(a_t,{\hat{c}}_t|{\gamma }^{*})}^{\top }{\theta }^{*}\right].}\hfill \end{array}$$

(23)

In (23), the first term ${\mathcal{R}}_{\mathrm{CB}}^T$ quantifies the Bayesian regret of our action policy (11) with respect to the action policy (22) for a CB with mean reward function $\psi {(a,{\hat{c}}_t|{\mathcal{H}}_{\hat{c}})}^{\top }{\theta }^{*}$. The second term ${\mathcal{R}}_{\mathrm{EE}1}^T$ accounts for the average difference in the cumulative mean rewards of the oracle optimal action policy (4), evaluated using the exact predictive distribution $P\left(c_t\right|{\hat{c}}_t,{\gamma }^{*})$, and our action policy (11), that uses the inferred predictive posterior distribution $P\left(c_t\right|{\hat{c}}_t,{\mathcal{H}}_{t-1,\hat{c}})$. In this sense, ${\mathcal{R}}_{\mathrm{EE}1}^T$ captures the error in approximating the exact predictive distribution $P\left(c_t\right|{\hat{c}}_t,{\gamma }^{*})$ via the inferred predictive distribution $P\left(c_t\right|{\hat{c}}_t,{\mathcal{H}}_{\hat{c}})$. The third term ${\mathcal{R}}_{\mathrm{EE}2}^T$ similarly accounts for the average approximation error.

To derive an upper bound on the Bayesian regret ${\mathcal{R}}^T\left({\pi }^{\mathrm{TS}}\right)$, we separately upper bound each of the three terms in (23) as derived in the following lemmas. The lemma below presents an upper bound on ${\mathcal{R}}_{\mathrm{CB}}^T$.

Lemma 1.

Under Assumption 1, the following upper bound holds if ${\displaystyle \frac{\lambda }{{\sigma }^2}}\le {\displaystyle \frac{1}{T}}\le 1$,

$$\begin{array}{cc}\hfill {\mathcal{R}}_{\mathrm{CB}}^T& \le U(m,{\displaystyle \frac{{\sigma }^2}{T}})+\sqrt{2{\sigma }^2\sum _{t=1}^T{D}_t}+\sqrt{2{\sigma }^2\left(Tlog\left(K\right)+\sum _{t=1}^T{D}_t\right)},\hfill \end{array}$$

(24)

$$\begin{array}{cc}\hfill & \le U(m,{\displaystyle \frac{{\sigma }^2}{T}})+\sqrt{2Tm{\sigma }^2}+\sqrt{2T{\sigma }^2(log\left(K\right)+m)},\hfill \end{array}$$

(25)

where $D_t=\mathbb{E}\left[D_{\mathrm{KL}}(P_t\left({\theta }^{*}\right)\parallel {\overline{P}}_t\left({\theta }^{*}\right))\right]$ and $U(m,\lambda )$ is as defined in (21).

To derive the upper bound in (24), we leverage results from [19] that study information-theoretic Bayesian regret of standard contextual TS algorithms via lifted information-ratio. However, the results do not directly apply to our algorithm due to the posterior mismatch between the sampling distribution ${\overline{P}}_t\left({\theta }^{*}\right)$ and the true posterior distribution $P_t\left({\theta }^{*}\right)$. Consequently, our upper bound (24) consists of three terms: the first term, defined as in (21), corresponds to the upper bound on the Bayesian regret of contextual TS that assumes ${\overline{P}}_t\left({\theta }^{*}\right)$ as the true posterior. This can be obtained by applying the lifted information ratio-based analysis of Cor. 2 in [19]. The second and third terms account for the posterior mismatch via the expected KL-divergence $\mathbb{E}\left[D_{\mathrm{KL}}(P_t\left({\theta }^{*}\right)\parallel {\overline{P}}_t\left({\theta }^{*}\right))\right]$ between the true posterior $P_t\left({\theta }^{*}\right)$ and the sampling distribution ${\overline{P}}_t\left({\theta }^{*}\right)$. In particular, this expected KL divergence can be upper bounded by $2(t-1)\lambda m/{\sigma }^2$ (See Appendix C.1.3 for proof) under the prior $P\left({\theta }^{*}\right)=\mathcal{N}(\mathbf{0},\lambda \mathbb{I})$. Importantly, our result holds when this prior distribution is sufficiently concentrated with its variance satisfying the inequality $\lambda \le {\sigma }^2/T$. This ensures that the contribution of posterior mismatch to the Bayes regret scales is $O\left(\sqrt{mT}\right)$.

The following lemma gives an upper bound on the sum ${\mathcal{R}}_{\mathrm{EE}1}^T+{\mathcal{R}}_{\mathrm{EE}2}^T$.

Lemma 2.

Under Assumption 1, the following upper bound holds for $\delta \in (0,1)$,

$$\begin{array}{ll}{\mathcal{R}}_{\mathrm{EE}1}^T+{\mathcal{R}}_{\mathrm{EE}2}^T& \le 2{\mathcal{R}}_{\mathrm{EE}1}^T\\ & \le 4{\delta }^{2}T\sqrt{{\displaystyle \frac{m\lambda }{2\pi }}}+2\sqrt{4\lambda Tmlog\left({\displaystyle \frac{2m}{\delta }}\right)\sum _{t=1}^{T}I({\gamma }^{*};c_t|{\hat{c}}_t,{\mathcal{H}}_{t-1,\hat{c}})}.\hfill \end{array}$$

(26)

In addition, if the covariance matrices satisfy that ${\Sigma }_n{\Sigma }_c^{-1}\succ 0$ and ${\Sigma }_n{\Sigma }_{\gamma }^{-1}{\Sigma }_{n}M\succ 0$ where $M={\Sigma }_n^{-1}+{\Sigma }_c^{-1}$, then (26) can be further upper bounded as

$$\begin{array}{cc}\hfill {\mathcal{R}}_{\mathrm{EE}1}^T+{\mathcal{R}}_{\mathrm{EE}2}^T& \hfill \\ & \le 4{\delta }^{2}T\sqrt{{\displaystyle \frac{m\lambda }{2\pi }}}+2\sqrt{4\lambda Tmlog\left({\displaystyle \frac{2m}{\delta }}\right)\left(\mathrm{Tr}\left({\left({\Sigma }_n{\Sigma }_{\gamma }^{-1}{\Sigma }_{n}M\right)}^{-1}\right)+log\left(T\right)\mathrm{Tr}\left({\Sigma }_c{\Sigma }_n^{-1}\right)\right)}.\hfill \end{array}$$

(27)

Lemma 2 shows that the error in approximating $P\left(c_t\right|{\hat{c}}_t,{\gamma }^{*})$ with $P\left(c_t\right|{\hat{c}}_t,{\mathcal{H}}_{t-1,\hat{c}})$, on average, can be quantified via the conditional mutual information $I({\gamma }^{*};c_t|{\hat{c}}_t,{\mathcal{H}}_{t-1,\hat{c}})$ between ${\gamma }^{*}$ and true context $c_t$ given knowledge of observed noisy contexts up to and including iteration t.

Finally, combining Lemmas 1 and  2 with the choice of $\delta =1/T$ and $\lambda \le {\sigma }^2/T$ gives us the regret bound in Theorem 1.

3.3. Beyond Gaussian Bandits

In the previous sections, we studied Gaussian bandits and analyzed the Bayesian regret. We will now discuss the potential extension of results beyond Gaussian bandits. As in [14], we will focus on Gaussian context distribution and context noise distribution, which helps to derive the upper bound on the estimation errors in Lemma 2.

To extend the Bayesian regret analysis to non-Gaussian bandits, Lemma 1 requires bandit-specific modifications. Specifically, the derivation of the term $U(m,\lambda )$, that captures the standard Bayesian regret of contextual TS with ${\overline{P}}_t\left({\theta }^{*}\right)$ as the true posterior, and that of the posterior mismatch term via the expected KL divergence critically depends on the type of bandit and the choice of the sampling posterior. The Bayesian regret bound $U(m,\lambda )$ is derived using the lifted information ratio-based approach of [19]. This can indeed be extended to non-Gaussian bandits like logistic bandits (see [19]) to obtain a modified $U(m,\lambda )$ term.

However, the analysis of posterior mismatch term for non-Gaussian bandits is non-trivial and depends on the specific bandit assumed. Firstly, to characterize the posterior mismatch via the expected KL divergence, our analysis requires the chosen sampling distribution ${\overline{P}}_t\left({\theta }^{*}\right)$ to be sub-Gaussian. To choose the sampling distribution, one can follow the framework adopted in (15) and (16) and use an ‘appropriate’ reward distribution $\overline{P}\left(r_{\tau }\right|a_{\tau },{\hat{c}}_{\tau },{\mathcal{H}}_{\tau -1,\hat{c}},{\theta }^{*})$ such that $\left(a\right)$ the KL divergence $D_{\mathrm{KL}}(P\left(r_{\tau }\right|a_{\tau },c_{\tau },{\theta }^{*})\parallel \overline{P}\left(r_{\tau }\right|a_{\tau },{\hat{c}}_{\tau },{\mathcal{H}}_{\tau -1,\hat{c}},{\theta }^{*}))$ between the true reward distribution and the chosen reward distribution is small to minimize posterior mismatch, and  $\left(b\right)$ the resulting sampling distribution is easy to sample from and has sub-Gaussian tails. Thus, analyzing the posterior mismatch for non-Gaussian bandits requires a case-by-case treatment. For Gaussian bandits, we control the above KL divergence by choosing a Gaussian distribution $\overline{P}\left(r_{\tau }\right|a_{\tau },{\hat{c}}_{\tau },{\mathcal{H}}_{\tau -1,\hat{c}},{\theta }^{*})$ with mean $\psi (a_{\tau },{\hat{c}}_{\tau }|{\mathcal{H}}_{\hat{c}})$ as in (16). Finally, in Section 5, we extend Algorithm 1 to logistic bandits with the choice of sampling distribution motivated by (15) and (16) and use Langevin Monte Carlo to sample from this distribution.

4. TS for CB with Delayed True Contexts

In this section, we consider the CBs with delayed true context setting where the agent observes the true context $c_t$ after it observes the reward $r_t$ corresponding to the chosen action $a_t$. Note, that at the time of choosing action $a_t$, the agent has access only to noisy contexts. We specialize our TS algorithm to this setting, and call it Algorithm 2 (or ${\pi }_{\mathrm{delay}}^{\mathrm{TS}}$).

Algorithm 2: TS for Delayed True contexts (${\pi }_{\mathrm{delay}}^{\mathrm{TS}}$)

1: for  $t=1,\dots ,T$  do 2:     The environment selects a true context $c_t$. 3:     Agent observes noisy context ${\hat{c}}_t$. 4:     Agent evaluates the predictive posterior distribution $P\left(c_t\right|{\hat{c}}_t,{\mathcal{H}}_{t-1,c,\hat{c}})$ as in (28). 5:     Agent samples ${\theta }_t\sim P\left({\theta }^{*}\right|{\mathcal{H}}_{t-1,r,a,c})$. 6:     Agent chooses action $a_t$ as in (33). 7:     Agent observes reward $r_t$ (as in (1)) and the true context $c_t$. 8: end for

Algorithm 2 follows similar steps as in Algorithm 1. However, different from Algorithm 1, at the tth iteration, the agent knows the history ${\mathcal{H}}_{t-1,c,\hat{c}}$ of true contexts in addition to that of noisy contexts. Consequently, in the de-noising step, the agent evaluates the predictive posterior distribution as

$$\begin{array}{c}\hfill P\left(c_t\right|{\hat{c}}_t,{\mathcal{H}}_{t-1,c,\hat{c}})={\mathbb{E}}_{P\left({\gamma }^{*}\right|{\mathcal{H}}_{t-1,c,\hat{c}})}\left[P\left(c_t\right|{\hat{c}}_t,{\gamma }^{*})\right],\end{array}$$

(28)

where $P\left(c_t\right|{\hat{c}}_t,{\gamma }^{*})$ is as defined in (2) and posterior distribution $P\left({\gamma }^{*}\right|{\mathcal{H}}_{t-1,c,\hat{c}})$ is obtained via Baye’s rule as $P\left({\gamma }^{*}\right|{\mathcal{H}}_{t-1,c,\hat{c}})\propto P\left({\gamma }^{*}\right){\prod }_{\tau =1}^{t-1}P(c_{\tau },{\hat{c}}_{\tau }|{\gamma }^{*})$ using the history of true and noisy contexts.

For the Gaussian context, noise as considered in Section 3.1, the predictive posterior distribution $P\left(c_t\right|{\hat{c}}_t,{\mathcal{H}}_{t-1,c,\hat{c}})=\mathcal{N}({\tilde{V}}_t,{\tilde{R}}_t^{-1})$ is multivariate Gaussian with the inverse covariance matrix,

$$\begin{array}{cc}\hfill \widetilde{R_t}& =M-{\Sigma }_n^{-1}{\tilde{H}}_t^{-1}{\Sigma }_n^{-1},\hfill \end{array}$$

(29)

and the mean vector

$$\begin{array}{cc}\hfill \widetilde{V_t}& ={\tilde{R}}_t^{-1}\left({\Sigma }_c^{-1}{\mu }_c+{\Sigma }_n^{-1}{\hat{c}}_t+{\Sigma }_n^{-1}{\tilde{H}}_t^{-1}{\Sigma }_n^{-1}\sum _{\tau =1}^{t-1}({\hat{c}}_{\tau }-c_{\tau })-{\Sigma }_n^{-1}{\tilde{H}}_t^{-1}{\Sigma }_n^{-1}{M}^{-1}({\Sigma }_c^{-1}{\mu }_c-{\Sigma }_n^{-1}{\hat{c}}_t)\right),\hfill \end{array}$$

(30)

where $M={\Sigma }_c^{-1}+{\Sigma }_n^{-1}$ and ${\tilde{H}}_t={\Sigma }_n^{-1}{M}^{-1}{\Sigma }_n^{-1}+(t-1){\Sigma }_n^{-1}+{\Sigma }_{\gamma }^{-1}$. Derivation can be found in Appendix B.2.4.

Following the denoising step, the next step in Algorithm 2 is a conventional Thompson sampling step, thanks to access to delayed true contexts. Consequently, the agent can evaluate the posterior distribution $P\left({\theta }^{*}\right|{\mathcal{H}}_{t-1,r,a,c})$ with known contexts as in (9) and use it to sample ${\theta }_t\sim P\left({\theta }^{*}\right|{\mathcal{H}}_{t-1,r,a,c})$. For  Gaussian bandit with Gaussian prior on ${\theta }^{*}$, the posterior distribution $P\left({\theta }^{*}\right|{\mathcal{H}}_{t-1,r,a,c})=\mathcal{N}({\tilde{\mu }}_{t-1},{\tilde{\Sigma }}_{t-1}^{-1})$ is a multivariate Gaussian distribution whose inverse covariance matrix and mean, respectively, evaluate as

$$\begin{array}{cc}\hfill {\tilde{\Sigma }}_{t-1}& ={\displaystyle \frac{1}{\lambda }}\mathbb{I}+{\displaystyle \frac{1}{{\sigma }^2}}\sum _{\tau =1}^{t-1}\varphi (a_{\tau },c_{\tau })\varphi {(a_{\tau },c_{\tau })}^{\top }\hfill \end{array}$$

(31)

$$\begin{array}{cc}\hfill {\tilde{\mu }}_{t-1}& ={\displaystyle \frac{{\tilde{\Sigma }}_{t-1}^{-1}}{{\sigma }^2}}\left(\sum _{\tau =1}^{t-1}{r}_{\tau }\varphi (a_{\tau },c_{\tau })\right).\hfill \end{array}$$

(32)

Using the sampled ${\theta }_t$ and the obtained predictive posterior distribution $P\left(c_t\right|{\hat{c}}_t,{\mathcal{H}}_{t-1,c,\hat{c}})$, the agent then chooses action $a_t$ as

$$\begin{array}{cc}\hfill a_t& =arg\underset{a\in \mathcal{A}}{max}\psi {(a,{\hat{c}}_t|{\mathcal{H}}_{c,\hat{c}})}^{\top }{\theta }_t,\hfill \end{array}$$

(33)

where we use the expected feature map $\psi (a_t,{\hat{c}}_t|{\mathcal{H}}_{c,\hat{c}}):={\mathbb{E}}_{P\left(c_t\right|{\hat{c}}_t,{\mathcal{H}}_{t-1,c,\hat{c}})}\left[\varphi (a_t,c_t)\right]$.

Information-Theoretic Bayesian Regret Bounds

In this section, we derive an information-theoretic upper bound on the Bayesian regret (7) of Algorithm 2 for Gaussian CBs. The following theorem presents our main result.

Theorem 2.

Under Assumption 1 and assuming that covariance matrices satisfy ${\Sigma }_{\gamma }{\Sigma }_n^{-1}\succ 0$, the following inequality holds for $\delta \in (0,1)$ when $\lambda \le {\sigma }^2$,

$$\begin{array}{cc}\hfill & {\displaystyle {\mathcal{R}}^T\left({\pi }_{\mathrm{delay}}^{\mathrm{TS}}\right)\le U(m,\lambda )+4T{\delta }^2\sqrt{\frac{2m\lambda }{\pi }}}{\displaystyle +2\sqrt{2\lambda mTdlog\left(1+T\mathrm{Tr}\left({\Sigma }_{\gamma }{\Sigma }_n^{-1}\right)/d\right)log\left(\frac{2m}{\delta }\right)}},\hfill \end{array}$$

where $U(m,\lambda )$ is as defined in (21).

Theorem 2 shows that Algorithm 2 achieves $\tilde{O}\left(m\sqrt{T}\right)$ regret with the choice of $\delta =1/T$ if $d=O\left(m\right)$. Furthermore, due to the absence of posterior mistmatch, the upper bound above is tighter than that of Theorem 1.

We now outline the main lemmas required to prove Theorem 2. To this end, we re-use the notation

$$\begin{array}{c}\hfill {\hat{a}}_t=arg\underset{a\in \mathcal{A}}{max}\psi {(a,{\hat{c}}_t|{\mathcal{H}}_{c,\hat{c}})}^{\top }{\theta }^{*}\end{array}$$

(34)

to define the optimal action maximizing the mean reward $\psi {(a,{\hat{c}}_t|{\mathcal{H}}_{c,\hat{c}})}^{\top }{\theta }^{*}$.

To derive the regret upper bound in Theorem 2, we first decompose the Bayesian cumulative regret (7) of Algorithm 2 (${\pi }_{\mathrm{delay}}^{\mathrm{TS}}$), similar to (23), into the following three terms,

$$\begin{array}{cc}\hfill & {\displaystyle {\mathcal{R}}^T\left({\pi }_{\mathrm{delay}}^{\mathrm{TS}}\right)={\mathcal{R}}_{\mathrm{d},\mathrm{CB}}^T+{\mathcal{R}}_{\mathrm{d},\mathrm{EE}1}^T+{\mathcal{R}}_{\mathrm{d},\mathrm{EE}2}^T\mathrm{where}},\hfill \\ & {\displaystyle {\mathcal{R}}_{\mathrm{d},\mathrm{CB}}^T=\sum _{t=1}^T\mathbb{E}\left[\psi {({\hat{a}}_t,{\hat{c}}_t|{\mathcal{H}}_{c,\hat{c}})}^{\top }{\theta }^{*}-\psi {(a_t,{\hat{c}}_t|{\mathcal{H}}_{c,\hat{c}})}^{\top }{\theta }^{*}\right]},\hfill \\ & {\displaystyle {\mathcal{R}}_{\mathrm{d},\mathrm{EE}1}^T=\sum _{t=1}^T\mathbb{E}\left[\psi {(a_t^{*},{\hat{c}}_t|{\gamma }^{*})}^{\top }{\theta }^{*}-\psi {({\hat{a}}_t,{\hat{c}}_t|{\mathcal{H}}_{c,\hat{c}})}^{\top }{\theta }^{*}\right]},\hfill \\ & {\displaystyle {\mathcal{R}}_{\mathrm{d},\mathrm{EE}2}^T=\sum _{t=1}^T\mathbb{E}\left[\psi {(a_t,{\hat{c}}_t|{\mathcal{H}}_{c,\hat{c}})}^{\top }{\theta }^{*}-\psi {(a_t,{\hat{c}}_t|{\gamma }^{*})}^{\top }{\theta }^{*}\right]}.\hfill \end{array}$$

(35)

An upper bound on ${\mathcal{R}}^T\left({\pi }_{\mathrm{delay}}^{\mathrm{TS}}\right)$ can be then obtained by separately bounding each of the three terms in (35).

In (35), the first term ${\mathcal{R}}_{\mathrm{d},\mathrm{CB}}^T$ corresponds to the Bayesian cumulative regret of a standard contextual TS algorithm that uses $\psi {(a,{\hat{c}}_t|{\mathcal{H}}_{c,\hat{c}})}^{\top }{\theta }^{*}$ for $a\in \mathcal{A}$ as the mean reward function. Note, that due to availability of delayed true contexts, there is no posterior mismatch in Algorithm 2. Hence, we apply Cor. 3 in [19] to yield the following upper bound on ${\mathcal{R}}_{\mathrm{d},\mathrm{CB}}^T$.

Lemma 3.

Under Assumption 1, the following upper bound on ${\mathcal{R}}_{\mathrm{d},\mathrm{CB}}^T$ holds for ${\displaystyle \frac{\lambda }{{\sigma }^2}}\le 1$,

$$\begin{array}{cc}\hfill {\mathcal{R}}_{\mathrm{d},\mathrm{CB}}^T& \le U(m,\lambda ),\hfill \end{array}$$

(36)

where $U(m,\lambda )$ is defined as in (21).

Lemma 3 gives a tighter bound in comparison to Lemma 1 where the posterior mismatch results in additional error terms in the regret bound.

We now upper bound the second term ${\mathcal{R}}_{\mathrm{d},\mathrm{EE}1}^T$ of (35), which similar to the term ${\mathcal{R}}_{\mathrm{EE}1}^T$ in (23), captures the error in approximating the exact predictive distribution $P\left(c_t\right|{\hat{c}}_t,{\gamma }^{*})$ via the inferred predictive distribution $P\left(c_t\right|{\hat{c}}_t,{\mathcal{H}}_{c,\hat{c}})$. The following lemma shows that this approximation error over T iterations can be quantified, on average, via the mutual information $I({\gamma }^{*};{\mathcal{H}}_{T,c,\hat{c}})$ between ${\gamma }^{*}$ and the T-length history of observed true and noisy contexts. This bound also holds for the third term ${\mathcal{R}}_{\mathrm{d},\mathrm{EE}2}^T$ of (35) which similarly accounts for the average approximation error.

Lemma 4.

Under Assumption 1, for any $\delta \in (0,1)$, we have the following upper bound,

$$\begin{array}{cc}\hfill & {\displaystyle {\mathcal{R}}_{\mathrm{d},\mathrm{EE}1}^T+{\mathcal{R}}_{\mathrm{d},\mathrm{EE}2}^T\le 2{\mathcal{R}}_{\mathrm{d},\mathrm{EE}1}^T}{\displaystyle \le 4\sqrt{m\lambda Tlog\left(\frac{2m}{\delta }\right)I({\gamma }^{*};{\mathcal{H}}_{T,c,\hat{c}})}+4T{\delta }^2\sqrt{\frac{2m\lambda }{\pi }}}.\hfill \end{array}$$

Furthermore, if the covariance matrices satisfy that ${\Sigma }_{\gamma }{\Sigma }_n^{-1}\succ 0$, we obtain that

$$\begin{array}{c}\hfill I({\gamma }^{*};{\mathcal{H}}_{T,c,\hat{c}})\le {\displaystyle \frac{d}{2}}log\left(1+T\mathrm{Tr}\left({\Sigma }_{\gamma }{\Sigma }_n^{-1}\right)/d\right)).\end{array}$$

Combining Lemmas 3 and  4 then gives us the upper bound on ${\mathcal{R}}^T\left({\pi }_{\mathrm{delay}}^{\mathrm{TS}}\right)$ in Theorem 1.

5. Experiments and Final Remarks

In this section, we experimentally validate the performance of the proposed algorithms on synthetic and real-world datasets. Details of implementation can be found in Appendix D.

Synthetic Datasets: For synthetic datasets, we go beyond Gaussian bandits and evaluate our algorithms for logistic contextual bandits (see Figure 1 (Left) and (Center)). In both these settings, we consider Gaussian contexts and context noise as in Section 3.1 with parameters ${\Sigma }_c=\mathbb{I}$, ${\Sigma }_{\gamma }={\sigma }_{\gamma }^2\mathbb{I}$, ${\Sigma }_n={\sigma }_n^2\mathbb{I}$ for some ${\sigma }_{\gamma }^2,{\sigma }_n^2>0$. We further consider action $a\in \mathcal{A}$ and context $c\in \mathcal{C}$ to be $d=5$ dimensional vectors with $a_i$ and $c_i$, respectively, denoting their ith component. We use $\varphi (a,c)=[a_1^2,a_2^2,a_3^2,a_4^2,a_5^2,c_1^2,c_2^2,c_3^2,c_4^2,c_5^2,a_1{c}_1,a_2{c}_2,a_3{c}_3,a_4{c}_4,$$a_5{c}_5]$ as the $m=15$ dimensional feature vector.

Gaussian Bandits: The mean reward function is given by $f({\theta }^{*},a,c)=\varphi {(a,c)}^{\top }{\theta }^{*}$ with the feature map described above. Other parameters are fixed as ${\sigma }_{\gamma }^2={\sigma }_n^2=1.1$, ${\sigma }^2=2$ and $\lambda =0.01$. Plots are averaged over 100 independent trials.

Logistic Bandits: The reward $r_t\in \{0,1\}$ is Bernoulli with mean reward given by $\mu \left(\varphi {(a,c)}^{\top }{\theta }^{*}\right)$, where $\mu \left(z\right)=1/(1+\exp (-z\left)\right)$ is the sigmoid function. We consider a Gaussian prior $\mathcal{N}(\mathbf{0},\mathbb{I})$ over ${\theta }^{*}$. In Algorithm 1, we choose the sampling distribution

$${\overline{P}}_t\left({\theta }^{*}\right)\propto P\left({\theta }^{*}\right)\prod _{\tau =1}^{t-1}\mathrm{Ber}\left(\mu \left(\psi {(a_{\tau },{\hat{c}}_{\tau }|{\mathcal{H}}_{\hat{c}})}^{\top }{\theta }^{*}\right)\right).$$

However, the posterior ${\overline{P}}_t\left({\theta }^{*}\right)$ is analytically intractable since Bernoulli reward-Gaussian prior forms a non-conjugate distribution pair. Consequently, we use Langevin Monte Carlo (LMC) [20] to sample from ${\overline{P}}_t\left({\theta }^{*}\right)$. We run LMC for $I=50$ iterations with learning rate ${\eta }_t=0.2/t$ and inverse temperature ${\beta }^{-1}=0.001$. Plots are averaged over 10 independent trials.

MovieLens Dataset: We use the MovieLens-100K dataset [21] to evaluate the performances. To utilise this dataset, we first perform non-negative matrix factorization on the rating matrix $R=\left[r_{c,a}\right]\in {\mathbb{R}}^{943\times 1682}$ with 3 latent factors to obtain $R=WH$, where $W\in {\mathbb{R}}^{943\times 3}$ and $H\in {\mathbb{R}}^{3\times 1682}$. Each row vector $W_c$ corresponds to an user context, while each column vector $H_a$ corresponds to movie (action) features. The mean and variance of the Gaussian context distribution is estimated from the row vectors of W. We then add Gaussian noise to context as in the synthetic settings with ${\sigma }_n^2=0.1$.

We apply K-means algorithm to the column vectors of H to group the actions into $K=20$ clusters. We use $m_k\in {\mathbb{R}}^3$ to denote the centroid and $v_k$ to denote the variance of the kth cluster. We then fix the mean and variance of the Gaussian prior over ${\theta }^{*}$ as ${\mu }_{\theta }=(m_1,\dots ,m_K)$ and ${\Sigma }_{\theta }=\mathrm{diag}(v_1{\mathbb{I}}_3,\dots ,v_K{\mathbb{I}}_3)$, with ${\mathbb{I}}_3$ denoting the $3\times 3$ identity matrix, respectively. The feature vector $\varphi (a,c)$ is then fixed as a 60-dimensional vector with vector $W_c$ at the index of the cluster k to which action a belongs and zeros everywhere else. We further add mean-zero Gaussian noise to the mean reward $\varphi {(a,c)}^{\top }{\theta }^{*}$ with variance ${\sigma }^2=0.01$. The Bayesian oracle in this experiment has access to the exact context noise parameter ${\gamma }^{*}$ sampled from the Gaussian prior with variance ${\Sigma }_{\gamma }={\sigma }_{\gamma }^2\mathbb{I}$, as well as the true ${\theta }^{*}$ sampled from the Gaussian prior $P\left({\theta }^{*}\right)$.

Baselines: We compare our algorithms with two baselines: $\mathrm{TS}\_\mathrm{naive}$ and $\mathrm{TS}\_\mathrm{oracle}$. In $\mathrm{TS}\_\mathrm{naive}$, the agent observes only noisy contexts but is unaware of the presence of noise. Consequently, it naively implements conventional TS with noisy context ${\hat{c}}_t$. This sets the benchmark for the worst-case achievable regret. The second baseline $\mathrm{TS}\_\mathrm{oracle}$ assumes that the agent knows the true channel parameter ${\gamma }^{*}$, a setting studied in [18], and can thus perform exact denoising via the predictive posterior distribution $P\left(c_t\right|\hat{c},{\gamma }^{*})$. This algorithm sets the benchmark for the best achievable regret.

Figure 1 (Left) corroborates our theoretical findings for Gaussian bandits. In particular, our algorithms (Algorithms 1 and  2) demonstrate sub-linear regret and achieve robust performance comparable to the best achievable performance of $\mathrm{TS}\_\mathrm{oracle}$. We remark that while our regret analysis of Gaussian bandits is motivated due to the tractability of posterior distributions and the concentration properties of Gaussians, our empirical results for logistic bandits in Figure 1 (Center) show a promising extension of our algorithms to non-conjugate distributions. Extension of Bayesian regret analysis to such general distributions is left for future work. Further, our experiments on MovieLens data in Figure 1 (Right) validate the effectiveness of our algorithm in comparison to the benchmarks. The plot shows that our approach outperforms $\mathrm{TS}\_\mathrm{naive}$ and achieves comparable regret as that of $\mathrm{TS}\_\mathrm{oracle}$ which is the best achievable regret.

6. Conclusions

We studied a stochastic CB problem where the agent observes noisy contexts through a noise channel with an unknown channel parameter. For Gaussian bandits and Gaussian context noise, we introduced a TS algorithm that achieves $\tilde{O}\left(m\sqrt{T}\right)$ Bayesian regret. The setting of Gaussian bandits with Gaussian noise was chosen for easy tractability of posterior distributions used in the proposed TS algorithms. We believe that the algorithm and key lemmas can be extended to when the likelihood-prior form conjugate distributions. Extension to general distributions is left for future work.

Finally, we conjecture that our proposed modified TS algorithm and the information-theoretic Bayesian regret analysis could be extended to noisy contexts in multi-task bandit settings. In this regard, a good starting point would be to leverage prior works that study multi-armed hierarchical bandits [22] and contextual hierarchical bandits [23] with linear-Gaussian reward models. However, the critical challenge is to evaluate the posterior mismatch which requires a case-by-case analysis.