Thompson Sampling for Stochastic Bandits with Noisy Contexts: An Information-Theoretic Regret Analysis

Abstract

We study stochastic linear contextual bandits (CB) where the agent observes a noisy version of the true context through a noise channel with unknown channel parameters. Our objective is to design an action policy that can “approximate” that of a Bayesian oracle that has access to the reward model and the noise channel parameter. We introduce a modified Thompson sampling algorithm and analyze its Bayesian cumulative regret with respect to the oracle action policy via information-theoretic tools. For Gaussian bandits with Gaussian context noise, our information-theoretic analysis shows that under certain conditions on the prior variance, the Bayesian cumulative regret scales as $\tilde{O}\left(m\sqrt{T}\right)$, where m is the dimension of the feature vector and T is the time horizon. We also consider the problem setting where the agent observes the true context with some delay after receiving the reward, and show that delayed true contexts lead to lower regret. Finally, we empirically demonstrate the performance of the proposed algorithms against baselines.

1. Introduction

Decision-making in the face of uncertainty is a widespread challenge found across various domains such as control and robotics [1], clinical trials [2], communications [3], and ecology [4]. To tackle this challenge, learning algorithms have been developed to uncover effective policies for optimal decision-making. One notable framework for addressing this is contextual bandits (CBs), which capture the essence of sequential decision-making by incorporating side information, termed context [5].

In the standard CB model, an agent interacts with the environment over numerous rounds. In each round, the environment presents a context to the agent based on which the agent chooses an action and receives a reward from the environment. The reward is stochastic, drawn from a probability distribution whose mean reward (which is a function of context-action pair) is unknown to the agent. The goal of the agent is to design a policy for action selection that can maximize the cumulative mean reward accrued over a T-length horizon.

In this paper, we focus on a CB model that assumes stochastic rewards with linear mean-reward functions, also called stochastic linear contextual bandits. Stochastic linear CB models find applications in various settings including internet advertisement selection [6], where the advertisement (i.e., action) and webpage features (i.e., context) are used to construct a linear predictor of the probability that a user clicks on a given advertisement, and article recommendation on web portals [7].

While most prior research on CBs has primarily focused on models with known exact contexts [8,9,10], in many real-world applications, the contexts are noisy, e.g., imprecise measurement of patient conditions in clinical trials, weather or stock market predictions. In such scenarios, when the exact contexts are unknown, the agent must utilize the observed noisy contexts to estimate the mean reward associated with the true context. However, this results in a biased estimate that renders the application of standard CB algorithms unsuitable. Consequently, recent efforts have been made to develop CB algorithms tailored to noisy context settings.

Related Works: ref. [11] considers a setting where there is a bounded zero-mean noise in the m-dimensional feature vector (denoted by $\varphi (a,c)$, where a is the action and c is the context) rather than in the context vector, and the agent observes only noisy features. For this setting, they develop an upper confidence bound (UCB) algorithm. Ref. [12] models the uncertainty regarding the true contexts by a context distribution that is known to the agent, while the agent never observes the true context and develops a UCB algorithm. A similar setting has also been considered in [13]. Differing from these works, ref. [14] considers the setting where the true feature vectors are sampled from an unknown feature distribution at each time, but the agent observes only a noisy feature vector. Assuming Gaussian feature noise with unknown mean and covariance, they develop an Optimism in the Face of Uncertainty (OFUL) algorithm. A variant of this setting has been studied in [15].

Motivation and Problem Setting: In this work, inspired by [14], we consider the following noisy CB setting. In each round, the environment samples a true context vector $c_t$ from a context distribution that is known to the agent. The agent, however, does not observe the true context but observes a noisy context ${\hat{c}}_t$ obtained as the output of a noise channel $P\left({\hat{c}}_t\right|c_t,{\gamma }^{*})$ parameterized by ${\gamma }^{*}$. The agent is aware of the noise present but does not know the channel parameter ${\gamma }^{*}$. Following [14], we consider Gaussian noise channels for our regret analysis.

Based on the observed noisy contexts, the agent chooses an action $a_t$ and observes a reward $r_t$ corresponding to the true context. We consider a linear bandit whose mean reward $\varphi {(a_t,c_t)}^{\top }{\theta }^{*}$ is determined by an unknown reward parameter ${\theta }^{*}$. The goal of the agent is to design an action policy that minimizes the Bayesian cumulative regret with respect to the action policy of a Bayesian oracle. The oracle has access to the reward model and the channel parameter ${\gamma }^{*}$, and uses the predictive distribution of the true context given the observed noisy context to select an action.

Our setting differs from [14] in that we assume noisy contexts rather than noisy feature vectors and that the agent knows the context distribution. The noise model, incorporating noise in the feature vector, allows [14] to transform the original problem into a different CB problem that estimates a modified reward parameter. Such a transformation, however, is not straightforward in our setting with noise in contexts rather than in feature vectors, where we wish to analyze the Bayesian regret. Additionally, we propose a de-noising approach to estimate the predictive distribution of the true context from given noisy contexts, offering potential benefits for future analyses.

The assumption of known context distribution follows from [12]. This can be motivated by considering the example of an online recommendation engine that pre-processes the user account registration information or contexts (e.g., age, gender, device, location, item preferences) to group them into different clusters [16]. The engine can then infer the ‘empirical’ distribution of users within each cluster to define a context distribution over true contextual information. A noisy contextual information scenario occurs when a guest with different preferences logs into a user’s account.

Challenges and Novelty: Different from existing works that developed UCB-based algorithms, we propose a fully Bayesian Thompson Sampling (TS) algorithm that approximates the Bayesian oracle policy. The proposed algorithm differs from the standard contextual TS [10] in the following aspects. Firstly, since the true context vectors are not accessible at each round and the channel parameter ${\gamma }^{*}$ is unknown, the agent uses its knowledge of the context distribution and the past observed noisy contexts to infer a predictive posterior distribution of the true context from the current observed noisy context. The inferred predictive distribution is then used to choose the action. This de-noising step enables our algorithm to ‘approximate’ the oracle action policy that uses knowledge of the channel parameter ${\gamma }^{*}$ to implement exact de-noising. Secondly, the reward $r_t$ received by the agent corresponds to the unobserved true context $c_t$. Hence, the agent cannot accurately evaluate the posterior distribution of ${\theta }^{*}$ and sample from it as is conducted in standard contextual TS. Instead, our algorithm proposes to use a sampling distribution that ‘approximates’ the posterior.

Different from existing works that focus on frequentist regret analysis, we derive novel information-theoretic bounds on the Bayesian cumulative regret of our algorithm. For Gaussian bandits, our information-theoretic regret bounds scale as $\tilde{O}\left(m\sqrt{T}\right)$ (the notation $\tilde{O}(•)$ suppresses logarithmic terms in •), where $m,T$ denote the dimension of the feature vector and time horizon respectively, under certain conditions on the variance of the prior on ${\theta }^{*}$. Furthermore, our Bayesian regret analysis shows that the posterior mismatch, resulting due to replacing the true posterior distribution with a sampling distribution, results in an approximation error that is captured via the Kullback–Leibler (KL) divergence between the distributions. To the best of our knowledge, quantifying the posterior mismatch via KL divergence has not been studied before and is of independent interest.

Finally, we also extend our algorithm to a setting where the agent observes the true context after the decision is made and reward is observed [12]. We call this setting CBs with delayed true contexts. Such scenarios arise in many applications where only a prediction of the context is available at the time of decision-making; however, the true context is available later. For instance, in farming-recommender systems where, at the time of making the decision regarding which crop to cultivate in a year, the true contextual information about the weather pattern is unavailable, while some ‘noisy’ weather predictions are available. In fact, the true weather pattern is observed only after the decision is made. We show that our TS algorithm for this setting with delayed true contexts results in reduced Bayesian regret. Table 1 compares our regret bound with that of the state-of-the-art algorithms in the noiseless and noisy CB settings.

Table 1. Comparison of the regret bounds of our proposed TS algorithm for noisy CB with state-of-the art algorithms.

Reference	Setting	Algorithm	Regret	Bound
[8]	Linear CB	LinRel	Frequentist	$\tilde{O}\left(\sqrt{mT}\right)$
[9]	Linear CB	Lin-UCB	Frequentist	$\tilde{O}\left(\sqrt{mT}\right)$
[10]	Linear CB	TS	Frequentist	$O\left(m\sqrt{T}{log}^{3/2}T\right)$
[17]	Linear CB	TS	Bayesian	$O(m\sqrt{T}logT)$
[11]	Noisy CB	SampLinUCB	Frequentist	$\tilde{O}\left(m\sqrt{T}\right)$
[12]	Noisy CB	UCB	Frequentist	$\tilde{O}\left(m\sqrt{T}\right)$
[14]	Noisy CB	OFUL	Frequentist	$\tilde{O}\left(m\sqrt{T}\right)$
Our work	Noisy CB	TS	Bayesian	$\tilde{O}\left(m\sqrt{T}\right)$

2. Problem Setting

In this section, we present the stochastic linear CB problem studied in this paper. Let $\mathcal{A}$ denote the action set with K actions and $\mathcal{C}$ denote the (possibly infinite) set of d-dimensional context vectors. At iteration $t\in \mathbb{N}$, the environment randomly draws a context vector $c_t\in \mathcal{C}$ according to a context distribution $P\left(c\right)$ defined over the space $\mathcal{C}$ of context vectors. The context distribution $P\left(c\right)$ is known to the agent. The agent, however, does not observe the true context $c_t$ drawn by the environment. Instead, it observes a noisy version ${\hat{c}}_t$ of the true context, obtained as the output of a noisy, stochastic channel $P\left({\hat{c}}_t\right|c_t,{\gamma }^{*})$ with the true context $c_t$ as the input. The noise channel $P\left({\hat{c}}_t\right|c_t,{\gamma }^{*})$ is parameterized by the noise channel parameter ${\gamma }^{*}$ that is unknown to the agent.

Having observed the noisy context ${\hat{c}}_t$ at iteration t, the agent chooses an action $a_t\in \mathcal{A}$ according to an action policy ${\pi }_t(·|{\hat{c}}_t)$. The action policy may be stochastic describing a probability distribution over the set $\mathcal{A}$ of actions. Corresponding to the chosen action $a_t$, the agent receives a reward from the environment given by

$$\begin{array}{c}\hfill r_t=f({\theta }^{*},a_t,c_t)+{\xi }_t,\end{array}$$

(1)

where $f({\theta }^{*},a_t,c_t)=\varphi {(a_t,c_t)}^{\top }{\theta }^{*}$ is the linear mean-reward function and ${\xi }_t$ is a zero-mean reward noise variable. The mean reward function $f({\theta }^{*},a_t,c_t)$ is defined via the feature map $\varphi :\mathcal{A}\times \mathcal{C}\to {\mathbb{R}}^m$, that maps the action and true context to an m-dimensional feature vector, and via the reward parameter ${\theta }^{*}\in {\mathbb{R}}^m$ that is unknown to the agent.

We call the noisy CB problem described above CBs with unobserved true context (see Setting 1) since the agent does not observe the true context $c_t$ and the selection of action is based solely on the observed noisy context. Accordingly, at the end of iteration t, the agent has accrued the history ${\mathcal{H}}_{t,r,a,\hat{c}}={\{r_{\tau },a_{\tau },{\hat{c}}_{\tau }\}}_{\tau =1}^t$ of observed reward-action-noisy context tuples. The action policy ${\pi }_{t+1}(·|{\hat{c}}_{t+1})$ at ${(t+1)}^{\mathrm{th}}$ iteration may depend on the history ${\mathcal{H}}_{t,r,a,\hat{c}}$.

Setting 1: CBs with unobserved true contexts

1: for  $t=1,\dots ,T$  do 2:     Environment samples $c_t\sim P\left(c\right)$. 3:     Agent observes noisy context ${\hat{c}}_t\sim P\left({\hat{c}}_t\right|c_t,{\gamma }^{*})$. 4:     Agent chooses an action $a_t\sim {\pi }_t(·|{\hat{c}}_t)$. 5:     Agent receives reward $r_t$ according to (1). 6: end for

We also consider a variant of the above problem where the agent has access to a delayed observation of the true context $c_t$ as studied in [12]. We call this setting CBs with delayed true context. In this setting, at iteration t, the agent first observes a noisy context ${\hat{c}}_t$, chooses action $a_t\sim {\pi }_t(·|{\hat{c}}_t)$, and receives reward $r_t$. Later, the true context $c_t$ is observed. It is important to note that the agent has no access to the true context at the time of decision-making. Thus, at the end of iteration t, the agent has collected the history ${\mathcal{H}}_{t,r,a,c,\hat{c}}={\{r_{\tau },a_{\tau },c_{\tau },{\hat{c}}_{\tau }\}}_{\tau =1}^t$ of observed reward-action-context-noisy context tuples.

In both of the problem settings described above, the agent’s objective is to devise an action policy that minimizes the Bayesian cumulative regret with respect to a baseline action policy. We define Bayesian cumulative regret next.

Bayesian Cumulative Regret

The cumulative regret of an action policy ${\pi }_t(·|{\hat{c}}_t)$ quantifies how different the mean reward accumulated over T iterations is from that accrued by a baseline action policy ${\pi }_t^{*}(·|{\hat{c}}_t)$. In this work, we consider as baseline the action policy of an oracle that has access to the channel noise parameter ${\gamma }^{*}$, reward parameter ${\theta }^{*}$, the context distribution $P\left(c\right)$ and the noise channel likelihood $P\left(c_t\right|{\hat{c}}_t,{\gamma }^{*}).$ Accordingly, at each iteration t, the oracle can infer the exact predictive distribution $P\left(c_t\right|{\hat{c}}_t,{\gamma }^{*})$ of the true context from the observed noisy context ${\hat{c}}_t$ via Baye’s rule as

$$\begin{array}{c}\hfill P\left(c_t\right|{\hat{c}}_t,{\gamma }^{*})={\displaystyle \frac{P(c_t,{\hat{c}}_t|{\gamma }^{*})}{P\left({\hat{c}}_t\right|{\gamma }^{*})}}.\end{array}$$

(2)

Here, $P(c_t,{\hat{c}}_t|{\gamma }^{*})=P\left(c_t\right)P\left({\hat{c}}_t\right|c_t,{\gamma }^{*})$ is the joint distribution of the true and noisy contexts given the noise channel parameter ${\gamma }^{*}$, and $P\left({\hat{c}}_t\right|{\gamma }^{*})$ is the distribution obtained by marginalizing $P(c_t,{\hat{c}}_t|{\gamma }^{*})$ over the true contexts, i.e., 

$$\begin{array}{c}\hfill P\left({\hat{c}}_t\right|{\gamma }^{*})={\mathbb{E}}_{P\left(c_t\right)}\left[P\left({\hat{c}}_t\right|c_t,{\gamma }^{*})\right],\end{array}$$

(3)

where ${\mathbb{E}}_{•}[·]$ denotes expectation with respect to ‘$•$’. The oracle action policy then adopts an action

$$\begin{array}{cc}\hfill a_t^{*}& =arg\underset{a\in \mathcal{A}}{max}{\mathbb{E}}_{P\left(c_t\right|{\hat{c}}_t,{\gamma }^{*})}\left[\varphi {(a,c_t)}^{\top }{\theta }^{*}\right]\hfill \\ \hfill & =arg\underset{a\in \mathcal{A}}{max}\psi {(a,{\hat{c}}_t|{\gamma }^{*})}^{\top }{\theta }^{*},\hfill \end{array}$$

(4)

at iteration t, where $\psi (a,\hat{c}|{\gamma }^{*}):={\mathbb{E}}_{P\left(c\right|\hat{c},{\gamma }^{*})}\left[\varphi (a,c)\right]$. Note, that as in [14,18], we do not choose the stronger oracle action policy of $arg{max}_{a\in \mathcal{A}}$$\varphi {(a,c_t)}^{\top }{\theta }^{*}$, that requires access to the true context $c_t$, as it is generally not achievable by an agent that observes only noisy context ${\hat{c}}_t$ and has no access to ${\gamma }^{*}$.

For fixed parameters ${\theta }^{*}$ and ${\gamma }^{*}$, we define the cumulative regret of the action policy ${\pi }_t(·|{\hat{c}}_t)$ as

$$\begin{array}{c}\hfill {\mathcal{R}}^T\left(\pi \right|{\theta }^{*},{\gamma }^{*})=\sum _{t=1}^T\mathbb{E}[\varphi {(a_t^{*},c_t)}^{\top }{\theta }^{*}-\varphi {(a_t,c_t)}^{\top }{\theta }^{*}|{\theta }^{*},{\gamma }^{*}],\end{array}$$

(5)

the expected difference in mean rewards of the oracle decision policy and the agent’s decision policy over T iterations. In (5), the expectation is taken with respect to the joint distribution $P\left({\mathcal{H}}_{t-1,r,\hat{c},c,a}\right|{\theta }^{*},{\gamma }^{*})P({\hat{c}}_t,c_t,a_t|{\mathcal{H}}_{t-1,r,\hat{c},c,a},{\theta }^{*},{\gamma }^{*})$, where $P({\hat{c}}_t,c_t,a_t|{\mathcal{H}}_{t-1,r,\hat{c},c,a},{\theta }^{*},{\gamma }^{*})=P({\hat{c}}_t,c_t|{\gamma }^{*}){\pi }_t\left(a_t\right|{\hat{c}}_t)=P\left({\hat{c}}_t\right|{\gamma }^{*})P\left(c_t\right|{\hat{c}}_t,{\gamma }^{*}){\pi }_t\left(a_t\right|{\hat{c}}_t)$. Using this, the cumulative regret (5) can be written as

$$\begin{array}{cc}\hfill & {\mathcal{R}}^T\left(\pi \right|{\theta }^{*},{\gamma }^{*})\hfill \\ \hfill & =\sum _{t=1}^T{\mathbb{E}}_{P\left({\mathcal{H}}_{t-1,r,\hat{c},c,a}\right|{\theta }^{*},{\gamma }^{*})}\left[{\mathbb{E}}_{P\left({\hat{c}}_t\right|{\gamma }^{*}){\pi }_t\left(a_t\right|{\hat{c}}_t)}{\mathbb{E}}_{P\left(c_t\right|{\hat{c}}_t,{\gamma }^{*})}\left[\varphi {(a_t^{*},c_t)}^{\top }{\theta }^{*}-\varphi {(a_t,c_t)}^{\top }{\theta }^{*}\right]\right]\hfill \\ \hfill & =\sum _{t=1}^T{\mathbb{E}}_{P\left({\mathcal{H}}_{t-1,r,\hat{c},c,a}\right|{\theta }^{*},{\gamma }^{*})}\left[{\mathbb{E}}_{P\left({\hat{c}}_t\right|{\gamma }^{*}){\pi }_t\left(a_t\right|{\hat{c}}_t)}\left[\psi {(a_t^{*},{\hat{c}}_t|{\gamma }^{*})}^{\top }{\theta }^{*}-\psi {(a_t,{\hat{c}}_t|{\gamma }^{*})}^{\top }{\theta }^{*}\right]\right]\hfill \\ \hfill & :=\sum _{t=1}^T\mathbb{E}\left[\psi {(a_t^{*},{\hat{c}}_t|{\gamma }^{*})}^{\top }{\theta }^{*}-\psi {(a_t,{\hat{c}}_t|{\gamma }^{*})}^{\top }{\theta }^{*}|{\theta }^{*},{\gamma }^{*}\right].\hfill \end{array}$$

(6)

Our focus in this work is on a Bayesian framework where we assume that the reward parameter ${\theta }^{*}\in \Theta$ and channel noise parameter ${\gamma }^{*}\in \mathsf{\Gamma }$ are independently sampled by the environment from prior distributions $P\left({\theta }^{*}\right)$, defined on the set $\Theta$ of reward parameters, and $P\left({\gamma }^{*}\right)$, defined on the set $\mathsf{\Gamma }$ of channel noise parameters, respectively. The agent has knowledge of the prior distributions, the reward likelihood in (1) and the noise channel likelihood $P\left({\hat{c}}_t\right|c_t,{\gamma }^{*})$, although it does not observe the sampled ${\gamma }^{*}$ and ${\theta }^{*}$. Using the above prior distributions, we define Bayesian cumulative regret of the action policy ${\pi }_t(·|{\hat{c}}_t)$ as

$$\begin{array}{c}\hfill {\mathcal{R}}^T\left(\pi \right)=\mathbb{E}\left[{\mathcal{R}}^T\left(\pi \right|{\theta }^{*},{\gamma }^{*})\right],\end{array}$$

(7)

where the expectation is taken with respect to the priors $P\left({\theta }^{*}\right)$ and $P\left({\gamma }^{*}\right)$.

In the next sections, we present our novel TS algorithms to minimize the Bayesian cumulative regret for the two problem settings considered in this paper.

3. Modified TS for CB with Unobserved True Contexts

In this section, we consider Setting 1 where the agent only observes the noisy context ${\hat{c}}_t$ at each iteration t. Our proposed modified TS Algorithm is given in Algorithm 1.

Algorithm 1: TS with unobserved true contexts (${\pi }^{\mathrm{TS}}$)

1: for  $t=1,\dots ,T$  do 2:     The environment selects a true context $c_t$. 3:     Agent observes noisy context ${\hat{c}}_t$. 4:     Agent evaluates the predictive posterior distribution $P\left(c_t\right|{\hat{c}}_t,{\mathcal{H}}_{t-1,\hat{c}})$ as in (8). 5:     Agent samples ${\theta }_t\sim \overline{P}\left({\theta }^{*}\right|{\mathcal{H}}_{t-1,r,a,\hat{c}})$. 6:     Agent chooses action $a_t$ as in (11). 7:     Agent observes reward $r_t$ as in (1). 8: end for

The proposed algorithm implements two steps in each iteration $t\in \mathbb{N}$. In the first step, called the de-noising step, the agent uses the current observed noisy context ${\hat{c}}_t$ and the history ${\mathcal{H}}_{t-1,\hat{c}}={\left\{{\hat{c}}_{\tau }\right\}}_{\tau =1}^{t-1}$ of past observed noisy contexts to obtain a predictive posterior distribution $P\left(c_t\right|{\hat{c}}_t,{\mathcal{H}}_{t-1,\hat{c}})$ of the true context $c_t$. This is a two-step process, where firstly the agent uses the history ${\mathcal{H}}_{t-1,\hat{c}}$ of past observed noisy contexts to compute the posterior distribution of ${\gamma }^{*}$ as $P\left({\gamma }^{*}\right|{\mathcal{H}}_{t-1,\hat{c}})\propto P\left({\gamma }^{*}\right){\prod }_{\tau =1}^{t-1}P\left({\hat{c}}_{\tau }\right|{\gamma }^{*}),$ where the conditional distribution $P\left({\hat{c}}_t\right|{\gamma }^{*})$ is evaluated as in (3). Note, that to evaluate the posterior, the agent uses its knowledge of the context distribution $P\left(c\right)$, the prior $P\left({\gamma }^{*}\right)$ and the noise channel likelihood $P\left({\hat{c}}_t\right|c_t,{\gamma }^{*})$. Using the derived posterior $P\left({\gamma }^{*}\right|{\mathcal{H}}_{t-1,\hat{c}})$, the predictive posterior distribution of the true context is then obtained as

$$\begin{array}{c}\hfill P\left(c_t\right|{\hat{c}}_t,{\mathcal{H}}_{t-1,\hat{c}})={\mathbb{E}}_{P\left({\gamma }^{*}\right|{\mathcal{H}}_{t-1,\hat{c}})}\left[P\left(c_t\right|{\hat{c}}_t,{\gamma }^{*})\right],\end{array}$$

(8)

where $P\left(c_t\right|{\hat{c}}_t,{\gamma }^{*})$ is defined as in (2).

The second step of the algorithm implements a modified Thompson sampling. Note, that since the agent does not have access to the true contexts, it cannot evaluate the posterior distribution with known contexts,

$$\begin{array}{c}\hfill P\left({\theta }^{*}\right|{\mathcal{H}}_{t-1,r,a,c})\propto P\left({\theta }^{*}\right)\prod _{\tau =1}^{t-1}P\left(r_{\tau }\right|a_{\tau },c_{\tau },{\theta }^{*}),\end{array}$$

(9)

as is conducted in standard contextual TS. Instead, the agent must evaluate the true posterior distribution under noisy contexts,

$$\begin{array}{cc}\hfill P_t\left({\theta }^{*}\right)& :=P\left({\theta }^{*}\right|{\mathcal{H}}_{t-1,r,a,\hat{c}})\hfill \\ & \propto P\left({\theta }^{*}\right){\mathbb{E}}_{P\left({\gamma }^{*}\right)}\left[\prod _{\tau =1}^{t-1}{\mathbb{E}}_{P\left(c_{\tau }\right)}\left[P\left({\hat{c}}_{\tau }\right|c_{\tau },{\gamma }^{*})P\left(r_{\tau }\right|a_{\tau },c_{\tau },{\theta }^{*})\right]\right].\hfill \end{array}$$

(10)

However, evaluating the marginal distribution ${\mathbb{E}}_{P\left({\gamma }^{*}\right)}\left[{\prod }_{\tau =1}^{t-1}{\mathbb{E}}_{P\left(c_{\tau }\right)}\left[P\left({\hat{c}}_{\tau }\right|c_{\tau },{\gamma }^{*})P\left(r_{\tau }\right|a_{\tau },c_{\tau },{\theta }^{*})\right]\right]$ is challenging even for Gaussian bandits as the mean $\varphi {(a_{\tau },c_{\tau })}^{\top }{\theta }^{*}$ of the reward distribution $P\left(r_{\tau }\right|a_{\tau },c_{\tau },{\theta }^{*})$ is, in general, a non-linear function of the true context $c_{\tau }$. As a result, the posterior $P_t\left({\theta }^{*}\right)$ is analytically intractable.

Consequently, at each iteration t, the agent samples ${\theta }_t\sim \overline{P}\left({\theta }^{*}\right|{\mathcal{H}}_{t-1,r,a,\hat{c}})$ from a distribution $\overline{P}\left({\theta }^{*}\right|{\mathcal{H}}_{t-1,r,a,\hat{c}})$ that ‘approximates’ the true posterior $P_t\left({\theta }^{*}\right)$. The specific choice of this sampling distribution depends on the problem setting. Ideally, one must choose a distribution that is sufficiently ‘close’ to the true posterior. In the next sub-section, we will explain the choice for Gaussian bandits.

Using the sampled ${\theta }_t$ and the predictive posterior distribution $P\left(c_t\right|{\hat{c}}_t,{\mathcal{H}}_{t-1,\hat{c}})$ obtained from the denoising step, the agent then chooses action $a_t$ at iteration t as

$$\begin{array}{cc}\hfill a_t& =arg\underset{a\in \mathcal{A}}{max}\psi {(a,{\hat{c}}_t|{\mathcal{H}}_{\hat{c}})}^{\top }{\theta }_t,\mathrm{where}\hfill \end{array}$$

(11)

$$\begin{array}{cc}\hfill \psi (a_t,{\hat{c}}_t|{\mathcal{H}}_{\hat{c}})& :={\mathbb{E}}_{P\left(c_t\right|{\hat{c}}_t,{\mathcal{H}}_{t-1,\hat{c}})}\left[\varphi (a_t,c_t)\right]\hfill \end{array}$$

(12)

is the expected feature map with respect to $P\left(c_t\right|{\hat{c}}_t,{\mathcal{H}}_{t-1,\hat{c}})$.

3.1. Linear-Gaussian Stochastic CBs

We now instantiate Algorithm 1 for Gaussian CBs. Specifically, we consider Gaussian bandits with the reward noise ${\xi }_t$ in (1) as Gaussian $\mathcal{N}(0,{\sigma }^2)$ with mean 0 and variance ${\sigma }^2>0$. We also assume a Gaussian prior $P\left({\theta }^{*}\right)=\mathcal{N}(\mathbf{0},\lambda \mathbb{I})$ on the reward parameter ${\theta }^{*}$ with mean zero and an $m\times m$ diagonal, covariance matrix with entries $\lambda >0$. Here, $\mathbb{I}$ denotes the identity matrix. The assumption of diagonal prior covariance is in line with Lemma 3 in [19].

We consider a multivariate Gaussian context distribution $P\left(c\right)=\mathcal{N}({\mu }_c,{\Sigma }_c)$ with mean ${\mu }_c\in {\mathbb{R}}^d$ and covariance matrix ${\Sigma }_c\in {\mathbb{R}}^{d\times d}$. The context noise channel $P\left(\hat{c}\right|c,{\gamma }^{*})$ is also similarly Gaussian with a mean $({\gamma }^{*}+c)$ and covariance matrix ${\Sigma }_n\in {\mathbb{R}}^{d\times d}$. We assume the prior on noise channel parameter ${\gamma }^{*}$ to be Gaussian $P\left({\gamma }^{*}\right)=\mathcal{N}(\mathbf{0},{\Sigma }_{\gamma })$ with d-dimensional zero mean vector $\mathbf{0}$ and covariance matrix ${\Sigma }_{\gamma }\in {\mathbb{R}}^{d\times d}.$ We assume that ${\Sigma }_c,{\Sigma }_{\gamma }$ and ${\Sigma }_n$ are all positive definite matrices known to the agent. The assumption of positive definite covariance matrices is to facilitate the Bayesian analysis adopted in this work. Similar assumptions were also required in the related work of [14].

For this setting, we can analytically evaluate the predictive posterior distribution $P\left(c_t\right|{\hat{c}}_t,{\mathcal{H}}_{t-1,\hat{c}})=\mathcal{N}\left(c_t\right|V_t,R_t^{-1})$ as a multi-variate Gaussian with inverse covariance matrix,

$$\begin{array}{c}\hfill R_t=M-{\Sigma }_n^{-1}{\left(H_t^{-1}\right)}^{\top }{\Sigma }_n^{-1},\end{array}$$

(13)

where $H_t=(t-1){\Sigma }_n^{-1}-(t-2){\Sigma }_n^{-1}{M}^{-1}{\Sigma }_n^{-1}+{\Sigma }_{\gamma }^{-1}$ and $M={\Sigma }_c^{-1}+{\Sigma }_n^{-1}$, and with the mean vector

$$\begin{array}{c}\hfill {\displaystyle V_t={\left(R_t^{-1}\right)}^{\top }\left({\Sigma }_c^{-1}{\mu }_c+{\Sigma }_n^{-1}{\hat{c}}_t-{\Sigma }_n^{-1}{\left(H_t^{-1}\right)}^{\top }{L}_t^{\top }\right),}\end{array}$$

(14)

where

$$\begin{array}{cc}\hfill L_t^{\top }& ={\Sigma }_n^{-1}{M}^{-1}({\Sigma }_c^{-1}{\mu }_c+{\Sigma }_n^{-1}{\hat{c}}_t)+({\Sigma }_n^{-1}-{\Sigma }_n^{-1}{M}^{-1}{\Sigma }_n^{-1})\sum _{\tau =1}^{t-1}{\hat{c}}_{\tau }\hfill \\ & -(t-1){\Sigma }_n^{-1}{M}^{-1}{\Sigma }_c^{-1}{\mu }_c.\hfill \end{array}$$

Derivations are presented in Appendix C.1.2.

For the modified-TS step, we sample ${\theta }_t$ from the approximate posterior distribution

$$\begin{array}{c}\hfill {\overline{P}}_t\left({\theta }^{*}\right):=\overline{P}\left({\theta }^{*}\right|{\mathcal{H}}_{t-1,r,a,\hat{c}})\propto P\left({\theta }^{*}\right)\prod _{\tau =1}^{t-1}\overline{P}\left(r_{\tau }\right|a_{\tau },{\hat{c}}_{\tau },{\mathcal{H}}_{\tau -1,\hat{c}},{\theta }^{*}),\end{array}$$

(15)

where

$$\begin{array}{c}\hfill \overline{P}\left(r_{\tau }\right|a_{\tau },{\hat{c}}_{\tau },{\mathcal{H}}_{\tau -1,\hat{c}},{\theta }^{*})=\mathcal{N}(\psi {(a_{\tau },{\hat{c}}_{\tau }|{\mathcal{H}}_{\hat{c}})}^{\top }{\theta }^{*},{\sigma }^2)\end{array}$$

(16)

and $\psi (a_t,{\hat{c}}_t|{\mathcal{H}}_{\hat{c}})$ is the expected feature map defined in (12). This yields the approximate posterior to be a Gaussian distribution ${\overline{P}}_t\left({\theta }^{*}\right)=\mathcal{N}({\mu }_{t-1},{\Sigma }_{t-1}^{-1})$ whose inverse covariance matrix and mean, respectively, evaluate as

$$\begin{array}{cc}\hfill {\Sigma }_{t-1}& ={\displaystyle \frac{\mathbb{I}}{\lambda }}+{\displaystyle \frac{1}{{\sigma }^2}}\sum _{\tau =1}^{t-1}\psi (a_{\tau },{\hat{c}}_{\tau }|{\mathcal{H}}_{\hat{c}})\psi {(a_{\tau },{\hat{c}}_{\tau }|{\mathcal{H}}_{\hat{c}})}^{\top }\hfill \end{array}$$

(17)

$$\begin{array}{cc}\hfill {\mu }_{t-1}& ={\displaystyle \frac{{\Sigma }_{t-1}^{-1}}{{\sigma }^2}}\left(\sum _{\tau =1}^{t-1}{r}_{\tau }\psi (a_{\tau },{\hat{c}}_{\tau }|{\mathcal{H}}_{\hat{c}})\right).\hfill \end{array}$$

(18)

The sampling distribution ${\overline{P}}_t\left({\theta }^{*}\right)$ considered above is different from the true posterior distribution (10), which is analytically intractable. However, it bears resemblance to the posterior (9) when the true contexts are known, with the reward distribution $P\left(r_{\tau }\right|a_{\tau },c_{\tau },{\theta }^{*})$ replaced by $\overline{P}\left(r_{\tau }\right|a_{\tau },{\hat{c}}_{\tau },{\mathcal{H}}_{\tau -1,\hat{c}},{\theta }^{*})$. In Section 3.2.2, we show that the above choice of sampling distribution is indeed ‘close’ to the true posterior.

3.2. Bayesian Regret Analysis

In this section, we derive information-theoretic upper bounds on the Bayesian regret (7) of the modified TS algorithm for Gaussian CBs. To this end, we first outline the key information-theoretic tools required to derive our bound.

3.2.1. Preliminaries

To start, let $P\left(x\right)$ and $Q\left(x\right)$ denote two probability distributions defined over the space $\mathcal{X}$ of random variables x. Then, the Kullback–Leibler (KL)-divergence between the distributions $P\left(x\right)$ and $Q\left(x\right)$ is defined as

$$\begin{array}{c}\hfill {\displaystyle D_{\mathrm{KL}}\left(P\left(x\right)\right|\left|Q\left(x\right)\right)={\mathbb{E}}_{P\left(x\right)}\left[log\frac{P\left(x\right)}{Q\left(x\right)}\right]},\end{array}$$

(19)

if $P\left(x\right)$ is absolutely continuous with respect to $Q\left(x\right)$, and takes value ∞ otherwise. If x and y denote two random variables described by the joint probability distribution $P(x,y)$, the mutual information $I(x;y)$ between x and y is defined as $I(x;y)=D_{\mathrm{KL}}(P(x,y)\parallel P\left(x\right)P\left(y\right))$, where $P\left(x\right)$ (and $P\left(y\right)$) is the marginal distribution of x (and y). More generally, for three random variables x, y and z with joint distribution $P(x,y,z)$, the conditional mutual information $I(x;y|z)$ between x and y given z evaluates as

$$I(x;y|z)={\mathbb{E}}_{P\left(z\right)}\left[D_{KL}(P(x,y|z)\parallel P\left(x\right|z)P\left(y\right|z))\right]$$

where $P\left(x\right|z)$ and $P\left(y\right|z)$ are conditional distributions. We will also use the following variational representation of the KL-divergence, also termed the Donskar–Varadhan (DV) inequality,

$$\begin{array}{c}\hfill {\displaystyle D_{\mathrm{KL}}(P\left(x\right)\parallel Q\left(x\right))\ge {\mathbb{E}}_{P\left(x\right)}\left[f\left(x\right)\right]-log{\mathbb{E}}_{Q\left(x\right)}\left[\exp \left(f\left(x\right)\right)\right],}\end{array}$$

(20)

which holds for any measurable function $f:\mathcal{X}\to \mathbb{R}$ satifying the inequality ${\mathbb{E}}_{Q\left(x\right)}\left[\exp \left(f\left(x\right)\right)\right]$$<\infty$.

3.2.2. Information-Theoretic Bayesian Regret Bounds

In this section, we present information-theoretic upper bounds on the Bayesian regret of the modified TS algorithm. To this end, we first state our main assumption.

Assumption 1.

The feature map $\varphi (·,·)\in {\mathbb{R}}^m$ has bounded norm, i.e., ${\parallel \varphi (·,·)\parallel }_2\le 1$.

The following theorem gives our main result.

Theorem 1.

Assume that the covariance matrices satisfy ${\Sigma }_n{\Sigma }_c^{-1}\succ 0$ and ${\Sigma }_n{\Sigma }_{\gamma }^{-1}{\Sigma }_{n}M\succ 0$ where $M={\Sigma }_n^{-1}+{\Sigma }_c^{-1}$. Under Assumption 1, if ${\displaystyle \frac{\lambda }{{\sigma }^2}}\le {\displaystyle \frac{1}{T}}\le 1$, the following upper bound on the Bayesian regret of the modified TS algorithm holds,

$$\begin{array}{cc}\hfill & {\mathcal{R}}^T\left({\pi }^{\mathrm{TS}}\right)\le U(m,{\displaystyle \frac{{\sigma }^2}{T}})+\sqrt{2Tm{\sigma }^2}+\sqrt{2T{\sigma }^2(log\left(K\right)+m)}\hfill \\ & +{\displaystyle \frac{4}{T}}\sqrt{{\displaystyle \frac{m{\sigma }^2}{2T\pi }}}+2\sqrt{4{\sigma }^{2}mlog\left(2mT\right)\left(\mathrm{Tr}\left({\left({\Sigma }_n{\Sigma }_{\gamma }^{-1}{\Sigma }_{n}M\right)}^{-1}\right)+log\left(T\right)\mathrm{Tr}\left({\Sigma }_c{\Sigma }_n^{-1}\right)\right)},\hfill \end{array}$$

where

$$\begin{array}{cc}\hfill U(m,\lambda )& {\displaystyle =\sqrt{2Tm{\sigma }^{2}min\{m,2log(1+K)\}log\left(1+\frac{T\lambda }{m{\sigma }^2}\right)}}.\hfill \end{array}$$

(21)

The theorem above shows that the proposed TS algorithm achieves $\tilde{O}\left(m\sqrt{T}\right)$ regret when the prior $P\left({\theta }^{*}\right)$ is highly informative with variance parameter satisfying the constraint $\lambda \le {\sigma }^2/T$.

Remark 1.

The assumption on covariance matrices in Theorem 1 directly holds for diagonal covariance matrices with positive eigen values.

To prove the regret bound of Theorem 1, we start by defining

$$\begin{array}{c}\hfill {\hat{a}}_t=arg\underset{a\in \mathcal{A}}{max}\psi {(a,{\hat{c}}_t|{\mathcal{H}}_{\hat{c}})}^{\top }{\theta }^{*}\end{array}$$

(22)

as the action that maximizes the mean reward $\psi {(a,{\hat{c}}_t|{\mathcal{H}}_{\hat{c}})}^{\top }{\theta }^{*}$ corresponding to reward parameter ${\theta }^{*}$. Using the above, the Bayesian cumulative regret (7) for the proposed TS algorithm ${\pi }^{\mathrm{TS}}$ can be decomposed as

$$\begin{array}{cc}\hfill & {\mathcal{R}}^T\left({\pi }^{\mathrm{TS}}\right)={\mathcal{R}}_{\mathrm{CB}}^T+{\mathcal{R}}_{\mathrm{EE}1}^T+{\mathcal{R}}_{\mathrm{EE}2}^T,\mathrm{where}\hfill \\ \hfill & {\displaystyle {\mathcal{R}}_{\mathrm{CB}}^T=\sum _{t=1}^T\mathbb{E}\left[\psi {({\hat{a}}_t,{\hat{c}}_t|{\mathcal{H}}_{\hat{c}})}^{\top }{\theta }^{*}-\psi {(a_t,{\hat{c}}_t|{\mathcal{H}}_{\hat{c}})}^{\top }{\theta }^{*}\right]},\hfill \\ & {\displaystyle {\mathcal{R}}_{\mathrm{EE}1}^T=\sum _{t=1}^T\mathbb{E}\left[\psi {(a_t^{*},{\hat{c}}_t|{\gamma }^{*})}^{\top }{\theta }^{*}-\psi {({\hat{a}}_t,{\hat{c}}_t|{\mathcal{H}}_{\hat{c}})}^{\top }{\theta }^{*}\right],}\hfill \\ & {\displaystyle {\mathcal{R}}_{\mathrm{EE}2}^T=\sum _{t=1}^T\mathbb{E}\left[\psi {(a_t,{\hat{c}}_t|{\mathcal{H}}_{\hat{c}})}^{\top }{\theta }^{*}-\psi {(a_t,{\hat{c}}_t|{\gamma }^{*})}^{\top }{\theta }^{*}\right].}\hfill \end{array}$$

(23)

In (23), the first term ${\mathcal{R}}_{\mathrm{CB}}^T$ quantifies the Bayesian regret of our action policy (11) with respect to the action policy (22) for a CB with mean reward function $\psi {(a,{\hat{c}}_t|{\mathcal{H}}_{\hat{c}})}^{\top }{\theta }^{*}$. The second term ${\mathcal{R}}_{\mathrm{EE}1}^T$ accounts for the average difference in the cumulative mean rewards of the oracle optimal action policy (4), evaluated using the exact predictive distribution $P\left(c_t\right|{\hat{c}}_t,{\gamma }^{*})$, and our action policy (11), that uses the inferred predictive posterior distribution $P\left(c_t\right|{\hat{c}}_t,{\mathcal{H}}_{t-1,\hat{c}})$. In this sense, ${\mathcal{R}}_{\mathrm{EE}1}^T$ captures the error in approximating the exact predictive distribution $P\left(c_t\right|{\hat{c}}_t,{\gamma }^{*})$ via the inferred predictive distribution $P\left(c_t\right|{\hat{c}}_t,{\mathcal{H}}_{\hat{c}})$. The third term ${\mathcal{R}}_{\mathrm{EE}2}^T$ similarly accounts for the average approximation error.

To derive an upper bound on the Bayesian regret ${\mathcal{R}}^T\left({\pi }^{\mathrm{TS}}\right)$, we separately upper bound each of the three terms in (23) as derived in the following lemmas. The lemma below presents an upper bound on ${\mathcal{R}}_{\mathrm{CB}}^T$.

Lemma 1.

Under Assumption 1, the following upper bound holds if ${\displaystyle \frac{\lambda }{{\sigma }^2}}\le {\displaystyle \frac{1}{T}}\le 1$,

$$\begin{array}{cc}\hfill {\mathcal{R}}_{\mathrm{CB}}^T& \le U(m,{\displaystyle \frac{{\sigma }^2}{T}})+\sqrt{2{\sigma }^2\sum _{t=1}^T{D}_t}+\sqrt{2{\sigma }^2\left(Tlog\left(K\right)+\sum _{t=1}^T{D}_t\right)},\hfill \end{array}$$

(24)

$$\begin{array}{cc}\hfill & \le U(m,{\displaystyle \frac{{\sigma }^2}{T}})+\sqrt{2Tm{\sigma }^2}+\sqrt{2T{\sigma }^2(log\left(K\right)+m)},\hfill \end{array}$$

(25)

where $D_t=\mathbb{E}\left[D_{\mathrm{KL}}(P_t\left({\theta }^{*}\right)\parallel {\overline{P}}_t\left({\theta }^{*}\right))\right]$ and $U(m,\lambda )$ is as defined in (21).

To derive the upper bound in (24), we leverage results from [19] that study information-theoretic Bayesian regret of standard contextual TS algorithms via lifted information-ratio. However, the results do not directly apply to our algorithm due to the posterior mismatch between the sampling distribution ${\overline{P}}_t\left({\theta }^{*}\right)$ and the true posterior distribution $P_t\left({\theta }^{*}\right)$. Consequently, our upper bound (24) consists of three terms: the first term, defined as in (21), corresponds to the upper bound on the Bayesian regret of contextual TS that assumes ${\overline{P}}_t\left({\theta }^{*}\right)$ as the true posterior. This can be obtained by applying the lifted information ratio-based analysis of Cor. 2 in [19]. The second and third terms account for the posterior mismatch via the expected KL-divergence $\mathbb{E}\left[D_{\mathrm{KL}}(P_t\left({\theta }^{*}\right)\parallel {\overline{P}}_t\left({\theta }^{*}\right))\right]$ between the true posterior $P_t\left({\theta }^{*}\right)$ and the sampling distribution ${\overline{P}}_t\left({\theta }^{*}\right)$. In particular, this expected KL divergence can be upper bounded by $2(t-1)\lambda m/{\sigma }^2$ (See Appendix C.1.3 for proof) under the prior $P\left({\theta }^{*}\right)=\mathcal{N}(\mathbf{0},\lambda \mathbb{I})$. Importantly, our result holds when this prior distribution is sufficiently concentrated with its variance satisfying the inequality $\lambda \le {\sigma }^2/T$. This ensures that the contribution of posterior mismatch to the Bayes regret scales is $O\left(\sqrt{mT}\right)$.

The following lemma gives an upper bound on the sum ${\mathcal{R}}_{\mathrm{EE}1}^T+{\mathcal{R}}_{\mathrm{EE}2}^T$.

Lemma 2.

Under Assumption 1, the following upper bound holds for $\delta \in (0,1)$,

$$\begin{array}{ll}{\mathcal{R}}_{\mathrm{EE}1}^T+{\mathcal{R}}_{\mathrm{EE}2}^T& \le 2{\mathcal{R}}_{\mathrm{EE}1}^T\\ & \le 4{\delta }^{2}T\sqrt{{\displaystyle \frac{m\lambda }{2\pi }}}+2\sqrt{4\lambda Tmlog\left({\displaystyle \frac{2m}{\delta }}\right)\sum _{t=1}^{T}I({\gamma }^{*};c_t|{\hat{c}}_t,{\mathcal{H}}_{t-1,\hat{c}})}.\hfill \end{array}$$

(26)

In addition, if the covariance matrices satisfy that ${\Sigma }_n{\Sigma }_c^{-1}\succ 0$ and ${\Sigma }_n{\Sigma }_{\gamma }^{-1}{\Sigma }_{n}M\succ 0$ where $M={\Sigma }_n^{-1}+{\Sigma }_c^{-1}$, then (26) can be further upper bounded as

$$\begin{array}{cc}\hfill {\mathcal{R}}_{\mathrm{EE}1}^T+{\mathcal{R}}_{\mathrm{EE}2}^T& \hfill \\ & \le 4{\delta }^{2}T\sqrt{{\displaystyle \frac{m\lambda }{2\pi }}}+2\sqrt{4\lambda Tmlog\left({\displaystyle \frac{2m}{\delta }}\right)\left(\mathrm{Tr}\left({\left({\Sigma }_n{\Sigma }_{\gamma }^{-1}{\Sigma }_{n}M\right)}^{-1}\right)+log\left(T\right)\mathrm{Tr}\left({\Sigma }_c{\Sigma }_n^{-1}\right)\right)}.\hfill \end{array}$$

(27)

Lemma 2 shows that the error in approximating $P\left(c_t\right|{\hat{c}}_t,{\gamma }^{*})$ with $P\left(c_t\right|{\hat{c}}_t,{\mathcal{H}}_{t-1,\hat{c}})$, on average, can be quantified via the conditional mutual information $I({\gamma }^{*};c_t|{\hat{c}}_t,{\mathcal{H}}_{t-1,\hat{c}})$ between ${\gamma }^{*}$ and true context $c_t$ given knowledge of observed noisy contexts up to and including iteration t.

Finally, combining Lemmas 1 and  2 with the choice of $\delta =1/T$ and $\lambda \le {\sigma }^2/T$ gives us the regret bound in Theorem 1.

3.3. Beyond Gaussian Bandits

In the previous sections, we studied Gaussian bandits and analyzed the Bayesian regret. We will now discuss the potential extension of results beyond Gaussian bandits. As in [14], we will focus on Gaussian context distribution and context noise distribution, which helps to derive the upper bound on the estimation errors in Lemma 2.

To extend the Bayesian regret analysis to non-Gaussian bandits, Lemma 1 requires bandit-specific modifications. Specifically, the derivation of the term $U(m,\lambda )$, that captures the standard Bayesian regret of contextual TS with ${\overline{P}}_t\left({\theta }^{*}\right)$ as the true posterior, and that of the posterior mismatch term via the expected KL divergence critically depends on the type of bandit and the choice of the sampling posterior. The Bayesian regret bound $U(m,\lambda )$ is derived using the lifted information ratio-based approach of [19]. This can indeed be extended to non-Gaussian bandits like logistic bandits (see [19]) to obtain a modified $U(m,\lambda )$ term.

However, the analysis of posterior mismatch term for non-Gaussian bandits is non-trivial and depends on the specific bandit assumed. Firstly, to characterize the posterior mismatch via the expected KL divergence, our analysis requires the chosen sampling distribution ${\overline{P}}_t\left({\theta }^{*}\right)$ to be sub-Gaussian. To choose the sampling distribution, one can follow the framework adopted in (15) and (16) and use an ‘appropriate’ reward distribution $\overline{P}\left(r_{\tau }\right|a_{\tau },{\hat{c}}_{\tau },{\mathcal{H}}_{\tau -1,\hat{c}},{\theta }^{*})$ such that $\left(a\right)$ the KL divergence $D_{\mathrm{KL}}(P\left(r_{\tau }\right|a_{\tau },c_{\tau },{\theta }^{*})\parallel \overline{P}\left(r_{\tau }\right|a_{\tau },{\hat{c}}_{\tau },{\mathcal{H}}_{\tau -1,\hat{c}},{\theta }^{*}))$ between the true reward distribution and the chosen reward distribution is small to minimize posterior mismatch, and  $\left(b\right)$ the resulting sampling distribution is easy to sample from and has sub-Gaussian tails. Thus, analyzing the posterior mismatch for non-Gaussian bandits requires a case-by-case treatment. For Gaussian bandits, we control the above KL divergence by choosing a Gaussian distribution $\overline{P}\left(r_{\tau }\right|a_{\tau },{\hat{c}}_{\tau },{\mathcal{H}}_{\tau -1,\hat{c}},{\theta }^{*})$ with mean $\psi (a_{\tau },{\hat{c}}_{\tau }|{\mathcal{H}}_{\hat{c}})$ as in (16). Finally, in Section 5, we extend Algorithm 1 to logistic bandits with the choice of sampling distribution motivated by (15) and (16) and use Langevin Monte Carlo to sample from this distribution.

4. TS for CB with Delayed True Contexts

In this section, we consider the CBs with delayed true context setting where the agent observes the true context $c_t$ after it observes the reward $r_t$ corresponding to the chosen action $a_t$. Note, that at the time of choosing action $a_t$, the agent has access only to noisy contexts. We specialize our TS algorithm to this setting, and call it Algorithm 2 (or ${\pi }_{\mathrm{delay}}^{\mathrm{TS}}$).

Algorithm 2: TS for Delayed True contexts (${\pi }_{\mathrm{delay}}^{\mathrm{TS}}$)

1: for  $t=1,\dots ,T$  do 2:     The environment selects a true context $c_t$. 3:     Agent observes noisy context ${\hat{c}}_t$. 4:     Agent evaluates the predictive posterior distribution $P\left(c_t\right|{\hat{c}}_t,{\mathcal{H}}_{t-1,c,\hat{c}})$ as in (28). 5:     Agent samples ${\theta }_t\sim P\left({\theta }^{*}\right|{\mathcal{H}}_{t-1,r,a,c})$. 6:     Agent chooses action $a_t$ as in (33). 7:     Agent observes reward $r_t$ (as in (1)) and the true context $c_t$. 8: end for

Algorithm 2 follows similar steps as in Algorithm 1. However, different from Algorithm 1, at the tth iteration, the agent knows the history ${\mathcal{H}}_{t-1,c,\hat{c}}$ of true contexts in addition to that of noisy contexts. Consequently, in the de-noising step, the agent evaluates the predictive posterior distribution as

$$\begin{array}{c}\hfill P\left(c_t\right|{\hat{c}}_t,{\mathcal{H}}_{t-1,c,\hat{c}})={\mathbb{E}}_{P\left({\gamma }^{*}\right|{\mathcal{H}}_{t-1,c,\hat{c}})}\left[P\left(c_t\right|{\hat{c}}_t,{\gamma }^{*})\right],\end{array}$$

(28)

where $P\left(c_t\right|{\hat{c}}_t,{\gamma }^{*})$ is as defined in (2) and posterior distribution $P\left({\gamma }^{*}\right|{\mathcal{H}}_{t-1,c,\hat{c}})$ is obtained via Baye’s rule as $P\left({\gamma }^{*}\right|{\mathcal{H}}_{t-1,c,\hat{c}})\propto P\left({\gamma }^{*}\right){\prod }_{\tau =1}^{t-1}P(c_{\tau },{\hat{c}}_{\tau }|{\gamma }^{*})$ using the history of true and noisy contexts.

For the Gaussian context, noise as considered in Section 3.1, the predictive posterior distribution $P\left(c_t\right|{\hat{c}}_t,{\mathcal{H}}_{t-1,c,\hat{c}})=\mathcal{N}({\tilde{V}}_t,{\tilde{R}}_t^{-1})$ is multivariate Gaussian with the inverse covariance matrix,

$$\begin{array}{cc}\hfill \widetilde{R_t}& =M-{\Sigma }_n^{-1}{\tilde{H}}_t^{-1}{\Sigma }_n^{-1},\hfill \end{array}$$

(29)

and the mean vector

$$\begin{array}{cc}\hfill \widetilde{V_t}& ={\tilde{R}}_t^{-1}\left({\Sigma }_c^{-1}{\mu }_c+{\Sigma }_n^{-1}{\hat{c}}_t+{\Sigma }_n^{-1}{\tilde{H}}_t^{-1}{\Sigma }_n^{-1}\sum _{\tau =1}^{t-1}({\hat{c}}_{\tau }-c_{\tau })-{\Sigma }_n^{-1}{\tilde{H}}_t^{-1}{\Sigma }_n^{-1}{M}^{-1}({\Sigma }_c^{-1}{\mu }_c-{\Sigma }_n^{-1}{\hat{c}}_t)\right),\hfill \end{array}$$

(30)

where $M={\Sigma }_c^{-1}+{\Sigma }_n^{-1}$ and ${\tilde{H}}_t={\Sigma }_n^{-1}{M}^{-1}{\Sigma }_n^{-1}+(t-1){\Sigma }_n^{-1}+{\Sigma }_{\gamma }^{-1}$. Derivation can be found in Appendix B.2.4.

Following the denoising step, the next step in Algorithm 2 is a conventional Thompson sampling step, thanks to access to delayed true contexts. Consequently, the agent can evaluate the posterior distribution $P\left({\theta }^{*}\right|{\mathcal{H}}_{t-1,r,a,c})$ with known contexts as in (9) and use it to sample ${\theta }_t\sim P\left({\theta }^{*}\right|{\mathcal{H}}_{t-1,r,a,c})$. For  Gaussian bandit with Gaussian prior on ${\theta }^{*}$, the posterior distribution $P\left({\theta }^{*}\right|{\mathcal{H}}_{t-1,r,a,c})=\mathcal{N}({\tilde{\mu }}_{t-1},{\tilde{\Sigma }}_{t-1}^{-1})$ is a multivariate Gaussian distribution whose inverse covariance matrix and mean, respectively, evaluate as

$$\begin{array}{cc}\hfill {\tilde{\Sigma }}_{t-1}& ={\displaystyle \frac{1}{\lambda }}\mathbb{I}+{\displaystyle \frac{1}{{\sigma }^2}}\sum _{\tau =1}^{t-1}\varphi (a_{\tau },c_{\tau })\varphi {(a_{\tau },c_{\tau })}^{\top }\hfill \end{array}$$

(31)

$$\begin{array}{cc}\hfill {\tilde{\mu }}_{t-1}& ={\displaystyle \frac{{\tilde{\Sigma }}_{t-1}^{-1}}{{\sigma }^2}}\left(\sum _{\tau =1}^{t-1}{r}_{\tau }\varphi (a_{\tau },c_{\tau })\right).\hfill \end{array}$$

(32)

Using the sampled ${\theta }_t$ and the obtained predictive posterior distribution $P\left(c_t\right|{\hat{c}}_t,{\mathcal{H}}_{t-1,c,\hat{c}})$, the agent then chooses action $a_t$ as

$$\begin{array}{cc}\hfill a_t& =arg\underset{a\in \mathcal{A}}{max}\psi {(a,{\hat{c}}_t|{\mathcal{H}}_{c,\hat{c}})}^{\top }{\theta }_t,\hfill \end{array}$$

(33)

where we use the expected feature map $\psi (a_t,{\hat{c}}_t|{\mathcal{H}}_{c,\hat{c}}):={\mathbb{E}}_{P\left(c_t\right|{\hat{c}}_t,{\mathcal{H}}_{t-1,c,\hat{c}})}\left[\varphi (a_t,c_t)\right]$.

Information-Theoretic Bayesian Regret Bounds

In this section, we derive an information-theoretic upper bound on the Bayesian regret (7) of Algorithm 2 for Gaussian CBs. The following theorem presents our main result.

Theorem 2.

Under Assumption 1 and assuming that covariance matrices satisfy ${\Sigma }_{\gamma }{\Sigma }_n^{-1}\succ 0$, the following inequality holds for $\delta \in (0,1)$ when $\lambda \le {\sigma }^2$,

$$\begin{array}{cc}\hfill & {\displaystyle {\mathcal{R}}^T\left({\pi }_{\mathrm{delay}}^{\mathrm{TS}}\right)\le U(m,\lambda )+4T{\delta }^2\sqrt{\frac{2m\lambda }{\pi }}}{\displaystyle +2\sqrt{2\lambda mTdlog\left(1+T\mathrm{Tr}\left({\Sigma }_{\gamma }{\Sigma }_n^{-1}\right)/d\right)log\left(\frac{2m}{\delta }\right)}},\hfill \end{array}$$

where $U(m,\lambda )$ is as defined in (21).

Theorem 2 shows that Algorithm 2 achieves $\tilde{O}\left(m\sqrt{T}\right)$ regret with the choice of $\delta =1/T$ if $d=O\left(m\right)$. Furthermore, due to the absence of posterior mistmatch, the upper bound above is tighter than that of Theorem 1.

We now outline the main lemmas required to prove Theorem 2. To this end, we re-use the notation

$$\begin{array}{c}\hfill {\hat{a}}_t=arg\underset{a\in \mathcal{A}}{max}\psi {(a,{\hat{c}}_t|{\mathcal{H}}_{c,\hat{c}})}^{\top }{\theta }^{*}\end{array}$$

(34)

to define the optimal action maximizing the mean reward $\psi {(a,{\hat{c}}_t|{\mathcal{H}}_{c,\hat{c}})}^{\top }{\theta }^{*}$.

To derive the regret upper bound in Theorem 2, we first decompose the Bayesian cumulative regret (7) of Algorithm 2 (${\pi }_{\mathrm{delay}}^{\mathrm{TS}}$), similar to (23), into the following three terms,

$$\begin{array}{cc}\hfill & {\displaystyle {\mathcal{R}}^T\left({\pi }_{\mathrm{delay}}^{\mathrm{TS}}\right)={\mathcal{R}}_{\mathrm{d},\mathrm{CB}}^T+{\mathcal{R}}_{\mathrm{d},\mathrm{EE}1}^T+{\mathcal{R}}_{\mathrm{d},\mathrm{EE}2}^T\mathrm{where}},\hfill \\ & {\displaystyle {\mathcal{R}}_{\mathrm{d},\mathrm{CB}}^T=\sum _{t=1}^T\mathbb{E}\left[\psi {({\hat{a}}_t,{\hat{c}}_t|{\mathcal{H}}_{c,\hat{c}})}^{\top }{\theta }^{*}-\psi {(a_t,{\hat{c}}_t|{\mathcal{H}}_{c,\hat{c}})}^{\top }{\theta }^{*}\right]},\hfill \\ & {\displaystyle {\mathcal{R}}_{\mathrm{d},\mathrm{EE}1}^T=\sum _{t=1}^T\mathbb{E}\left[\psi {(a_t^{*},{\hat{c}}_t|{\gamma }^{*})}^{\top }{\theta }^{*}-\psi {({\hat{a}}_t,{\hat{c}}_t|{\mathcal{H}}_{c,\hat{c}})}^{\top }{\theta }^{*}\right]},\hfill \\ & {\displaystyle {\mathcal{R}}_{\mathrm{d},\mathrm{EE}2}^T=\sum _{t=1}^T\mathbb{E}\left[\psi {(a_t,{\hat{c}}_t|{\mathcal{H}}_{c,\hat{c}})}^{\top }{\theta }^{*}-\psi {(a_t,{\hat{c}}_t|{\gamma }^{*})}^{\top }{\theta }^{*}\right]}.\hfill \end{array}$$

(35)

An upper bound on ${\mathcal{R}}^T\left({\pi }_{\mathrm{delay}}^{\mathrm{TS}}\right)$ can be then obtained by separately bounding each of the three terms in (35).

In (35), the first term ${\mathcal{R}}_{\mathrm{d},\mathrm{CB}}^T$ corresponds to the Bayesian cumulative regret of a standard contextual TS algorithm that uses $\psi {(a,{\hat{c}}_t|{\mathcal{H}}_{c,\hat{c}})}^{\top }{\theta }^{*}$ for $a\in \mathcal{A}$ as the mean reward function. Note, that due to availability of delayed true contexts, there is no posterior mismatch in Algorithm 2. Hence, we apply Cor. 3 in [19] to yield the following upper bound on ${\mathcal{R}}_{\mathrm{d},\mathrm{CB}}^T$.

Lemma 3.

Under Assumption 1, the following upper bound on ${\mathcal{R}}_{\mathrm{d},\mathrm{CB}}^T$ holds for ${\displaystyle \frac{\lambda }{{\sigma }^2}}\le 1$,

$$\begin{array}{cc}\hfill {\mathcal{R}}_{\mathrm{d},\mathrm{CB}}^T& \le U(m,\lambda ),\hfill \end{array}$$

(36)

where $U(m,\lambda )$ is defined as in (21).

Lemma 3 gives a tighter bound in comparison to Lemma 1 where the posterior mismatch results in additional error terms in the regret bound.

We now upper bound the second term ${\mathcal{R}}_{\mathrm{d},\mathrm{EE}1}^T$ of (35), which similar to the term ${\mathcal{R}}_{\mathrm{EE}1}^T$ in (23), captures the error in approximating the exact predictive distribution $P\left(c_t\right|{\hat{c}}_t,{\gamma }^{*})$ via the inferred predictive distribution $P\left(c_t\right|{\hat{c}}_t,{\mathcal{H}}_{c,\hat{c}})$. The following lemma shows that this approximation error over T iterations can be quantified, on average, via the mutual information $I({\gamma }^{*};{\mathcal{H}}_{T,c,\hat{c}})$ between ${\gamma }^{*}$ and the T-length history of observed true and noisy contexts. This bound also holds for the third term ${\mathcal{R}}_{\mathrm{d},\mathrm{EE}2}^T$ of (35) which similarly accounts for the average approximation error.

Lemma 4.

Under Assumption 1, for any $\delta \in (0,1)$, we have the following upper bound,

$$\begin{array}{cc}\hfill & {\displaystyle {\mathcal{R}}_{\mathrm{d},\mathrm{EE}1}^T+{\mathcal{R}}_{\mathrm{d},\mathrm{EE}2}^T\le 2{\mathcal{R}}_{\mathrm{d},\mathrm{EE}1}^T}{\displaystyle \le 4\sqrt{m\lambda Tlog\left(\frac{2m}{\delta }\right)I({\gamma }^{*};{\mathcal{H}}_{T,c,\hat{c}})}+4T{\delta }^2\sqrt{\frac{2m\lambda }{\pi }}}.\hfill \end{array}$$

Furthermore, if the covariance matrices satisfy that ${\Sigma }_{\gamma }{\Sigma }_n^{-1}\succ 0$, we obtain that

$$\begin{array}{c}\hfill I({\gamma }^{*};{\mathcal{H}}_{T,c,\hat{c}})\le {\displaystyle \frac{d}{2}}log\left(1+T\mathrm{Tr}\left({\Sigma }_{\gamma }{\Sigma }_n^{-1}\right)/d\right)).\end{array}$$

Combining Lemmas 3 and  4 then gives us the upper bound on ${\mathcal{R}}^T\left({\pi }_{\mathrm{delay}}^{\mathrm{TS}}\right)$ in Theorem 1.

5. Experiments and Final Remarks

In this section, we experimentally validate the performance of the proposed algorithms on synthetic and real-world datasets. Details of implementation can be found in Appendix D.

Synthetic Datasets: For synthetic datasets, we go beyond Gaussian bandits and evaluate our algorithms for logistic contextual bandits (see Figure 1 (Left) and (Center)). In both these settings, we consider Gaussian contexts and context noise as in Section 3.1 with parameters ${\Sigma }_c=\mathbb{I}$, ${\Sigma }_{\gamma }={\sigma }_{\gamma }^2\mathbb{I}$, ${\Sigma }_n={\sigma }_n^2\mathbb{I}$ for some ${\sigma }_{\gamma }^2,{\sigma }_n^2>0$. We further consider action $a\in \mathcal{A}$ and context $c\in \mathcal{C}$ to be $d=5$ dimensional vectors with $a_i$ and $c_i$, respectively, denoting their ith component. We use $\varphi (a,c)=[a_1^2,a_2^2,a_3^2,a_4^2,a_5^2,c_1^2,c_2^2,c_3^2,c_4^2,c_5^2,a_1{c}_1,a_2{c}_2,a_3{c}_3,a_4{c}_4,$$a_5{c}_5]$ as the $m=15$ dimensional feature vector.

Figure 1. Comparison of Bayesian regret of proposed algorithms with baselines as a function of number of iterations. (Left): Gaussian bandits with $K=40$, ${\sigma }_n^2={\sigma }_{\gamma }^2=1.1$; (Center) Logistic bandits with $K=40$, ${\sigma }_n^2=2$, ${\sigma }_{\gamma }^2=2.5$; (Right) MovieLens dataset with added Gaussian context noise and Gaussian prior: parameters set as ${\sigma }_n^2=0.1$, ${\sigma }_{\gamma }^2=0.6$.

Gaussian Bandits: The mean reward function is given by $f({\theta }^{*},a,c)=\varphi {(a,c)}^{\top }{\theta }^{*}$ with the feature map described above. Other parameters are fixed as ${\sigma }_{\gamma }^2={\sigma }_n^2=1.1$, ${\sigma }^2=2$ and $\lambda =0.01$. Plots are averaged over 100 independent trials.

Logistic Bandits: The reward $r_t\in \{0,1\}$ is Bernoulli with mean reward given by $\mu \left(\varphi {(a,c)}^{\top }{\theta }^{*}\right)$, where $\mu \left(z\right)=1/(1+\exp (-z\left)\right)$ is the sigmoid function. We consider a Gaussian prior $\mathcal{N}(\mathbf{0},\mathbb{I})$ over ${\theta }^{*}$. In Algorithm 1, we choose the sampling distribution

$${\overline{P}}_t\left({\theta }^{*}\right)\propto P\left({\theta }^{*}\right)\prod _{\tau =1}^{t-1}\mathrm{Ber}\left(\mu \left(\psi {(a_{\tau },{\hat{c}}_{\tau }|{\mathcal{H}}_{\hat{c}})}^{\top }{\theta }^{*}\right)\right).$$

However, the posterior ${\overline{P}}_t\left({\theta }^{*}\right)$ is analytically intractable since Bernoulli reward-Gaussian prior forms a non-conjugate distribution pair. Consequently, we use Langevin Monte Carlo (LMC) [20] to sample from ${\overline{P}}_t\left({\theta }^{*}\right)$. We run LMC for $I=50$ iterations with learning rate ${\eta }_t=0.2/t$ and inverse temperature ${\beta }^{-1}=0.001$. Plots are averaged over 10 independent trials.

MovieLens Dataset: We use the MovieLens-100K dataset [21] to evaluate the performances. To utilise this dataset, we first perform non-negative matrix factorization on the rating matrix $R=\left[r_{c,a}\right]\in {\mathbb{R}}^{943\times 1682}$ with 3 latent factors to obtain $R=WH$, where $W\in {\mathbb{R}}^{943\times 3}$ and $H\in {\mathbb{R}}^{3\times 1682}$. Each row vector $W_c$ corresponds to an user context, while each column vector $H_a$ corresponds to movie (action) features. The mean and variance of the Gaussian context distribution is estimated from the row vectors of W. We then add Gaussian noise to context as in the synthetic settings with ${\sigma }_n^2=0.1$.

We apply K-means algorithm to the column vectors of H to group the actions into $K=20$ clusters. We use $m_k\in {\mathbb{R}}^3$ to denote the centroid and $v_k$ to denote the variance of the kth cluster. We then fix the mean and variance of the Gaussian prior over ${\theta }^{*}$ as ${\mu }_{\theta }=(m_1,\dots ,m_K)$ and ${\Sigma }_{\theta }=\mathrm{diag}(v_1{\mathbb{I}}_3,\dots ,v_K{\mathbb{I}}_3)$, with ${\mathbb{I}}_3$ denoting the $3\times 3$ identity matrix, respectively. The feature vector $\varphi (a,c)$ is then fixed as a 60-dimensional vector with vector $W_c$ at the index of the cluster k to which action a belongs and zeros everywhere else. We further add mean-zero Gaussian noise to the mean reward $\varphi {(a,c)}^{\top }{\theta }^{*}$ with variance ${\sigma }^2=0.01$. The Bayesian oracle in this experiment has access to the exact context noise parameter ${\gamma }^{*}$ sampled from the Gaussian prior with variance ${\Sigma }_{\gamma }={\sigma }_{\gamma }^2\mathbb{I}$, as well as the true ${\theta }^{*}$ sampled from the Gaussian prior $P\left({\theta }^{*}\right)$.

Baselines: We compare our algorithms with two baselines: $\mathrm{TS}\_\mathrm{naive}$ and $\mathrm{TS}\_\mathrm{oracle}$. In $\mathrm{TS}\_\mathrm{naive}$, the agent observes only noisy contexts but is unaware of the presence of noise. Consequently, it naively implements conventional TS with noisy context ${\hat{c}}_t$. This sets the benchmark for the worst-case achievable regret. The second baseline $\mathrm{TS}\_\mathrm{oracle}$ assumes that the agent knows the true channel parameter ${\gamma }^{*}$, a setting studied in [18], and can thus perform exact denoising via the predictive posterior distribution $P\left(c_t\right|\hat{c},{\gamma }^{*})$. This algorithm sets the benchmark for the best achievable regret.

Figure 1 (Left) corroborates our theoretical findings for Gaussian bandits. In particular, our algorithms (Algorithms 1 and  2) demonstrate sub-linear regret and achieve robust performance comparable to the best achievable performance of $\mathrm{TS}\_\mathrm{oracle}$. We remark that while our regret analysis of Gaussian bandits is motivated due to the tractability of posterior distributions and the concentration properties of Gaussians, our empirical results for logistic bandits in Figure 1 (Center) show a promising extension of our algorithms to non-conjugate distributions. Extension of Bayesian regret analysis to such general distributions is left for future work. Further, our experiments on MovieLens data in Figure 1 (Right) validate the effectiveness of our algorithm in comparison to the benchmarks. The plot shows that our approach outperforms $\mathrm{TS}\_\mathrm{naive}$ and achieves comparable regret as that of $\mathrm{TS}\_\mathrm{oracle}$ which is the best achievable regret.

6. Conclusions

We studied a stochastic CB problem where the agent observes noisy contexts through a noise channel with an unknown channel parameter. For Gaussian bandits and Gaussian context noise, we introduced a TS algorithm that achieves $\tilde{O}\left(m\sqrt{T}\right)$ Bayesian regret. The setting of Gaussian bandits with Gaussian noise was chosen for easy tractability of posterior distributions used in the proposed TS algorithms. We believe that the algorithm and key lemmas can be extended to when the likelihood-prior form conjugate distributions. Extension to general distributions is left for future work.

Finally, we conjecture that our proposed modified TS algorithm and the information-theoretic Bayesian regret analysis could be extended to noisy contexts in multi-task bandit settings. In this regard, a good starting point would be to leverage prior works that study multi-armed hierarchical bandits [22] and contextual hierarchical bandits [23] with linear-Gaussian reward models. However, the critical challenge is to evaluate the posterior mismatch which requires a case-by-case analysis.

Appendix B.1. TS Algorithm for Linear-Gaussian Bandits with Delayed True Contexts

The pseudocode for the TS algorithm for Gaussian bandits is given in Algorithm A1.

Algorithm A1: TS with Delayed Contexts for Gaussian Bandits (${\pi }_{\mathrm{delay}}^{\mathrm{TS}}$)

1: Given parameters: $({\Sigma }_n,{\sigma }^2,\lambda ,{\Sigma }_{\gamma },{\mu }_c,{\Sigma }_c)$. Initialize ${\tilde{\mu }}_0=\mathbf{0}\in {\mathbb{R}}^m$ and ${\tilde{\Sigma }}_0^{-1}=(1/\lambda )\mathbb{I}$ 2: for  $t=1,\cdots ,T$  do 3:       The environment selects a true context $c_t$. 4:       Agent observes noisy context ${\hat{c}}_t$. 5:       Agent computes ${\tilde{R}}_t$ and ${\tilde{V}}_t$ using (29) and (30) to evaluate $P\left(c_t\right|{\hat{c}}_t,{\mathcal{H}}_{t-1,c,\hat{c}})=\mathcal{N}({\tilde{V}}_t,{\tilde{R}}_t^{-1})$. 6:       Agent samples ${\theta }_t\sim \mathcal{N}({\tilde{\mu }}_{t-1},{\tilde{\Sigma }}_{t-1}^{-1})$ where ${\tilde{\mu }}_{t-1}$ and ${\tilde{\Sigma }}_{t-1}$ are defined as in (32) and (31). 7:       Agent chooses action $a_t$ as in (33) using ${\theta }_t$ and $P\left(c_t\right|{\hat{c}}_t,{\mathcal{H}}_{t-1,c,\hat{c}})$. 8:       Agent observes reward $r_t$ corresponding to $a_t$, and the true context $c_t$. 9: end for

Appendix B.2. Derivation of Posterior and Predictive Posterior Distributions

In this section, we provide detailed derivation of posterior predictive distribution for Gaussian bandits. To this end, we first derive the exact predictive distribution $P\left(c_t\right|{\hat{c}}_t,{\gamma }^{*})$.

Appendix B.2.1. Derivation of $P\left(c_t\right|{\hat{c}}_t,{\gamma }^{*})$

We begin by noting that

$$\begin{array}{cc}\hfill P\left(c_t\right|{\hat{c}}_t,{\gamma }^{*})={\displaystyle \frac{P\left(c_t\right)P\left({\hat{c}}_t\right|c_t,{\gamma }^{*})}{P\left({\hat{c}}_t\right|{\gamma }^{*})}}& \propto P\left(c_t\right)P\left({\hat{c}}_t\right|c_t,{\gamma }^{*})\hfill \\ \hfill & =\mathcal{N}({\mu }_c,{\Sigma }_c)\mathcal{N}(c_t+{\gamma }^{*},{\Sigma }_n).\hfill \end{array}$$

Subsequently,

$$\begin{array}{cc}\hfill log\left(P\left(c_t\right|{\hat{c}}_t,{\gamma }^{*})\right)& \propto log\left(P\left(c_t\right)P\left({\hat{c}}_t\right|c_t,{\gamma }^{*})\right)\hfill \\ & \propto -{\displaystyle \frac{1}{2}}\left({(c_t-{\mu }_c)}^{\top }{\Sigma }_c^{-1}(c_t-\mu )+{({\hat{c}}_t-c_t-{\gamma }^{*})}^{\top }{\Sigma }_n^{-1}({\hat{c}}_t-c_t-{\gamma }^{*})\right)\hfill \\ \hfill & =-{\displaystyle \frac{1}{2}}(c_t^{\top }({\Sigma }_c^{-1}+{\Sigma }_n^{-1})c_t-\left({\mu }_c^{\top }{\Sigma }_c^{-1}+{({\hat{c}}_t-{\gamma }^{*})}^{\top }{\Sigma }_n^{-1}\right)c_t\hfill \\ \hfill & -c_t^{\top }\left({\Sigma }_c^{-1}{\mu }_c+{\Sigma }_n^{-1}({\hat{c}}_t-{\gamma }^{*})\right)+{({\hat{c}}_t-{\gamma }^{*})}^{\top }{\Sigma }_n^{-1}({\hat{c}}_t-{\gamma }^{*}))\hfill \\ \hfill & =-{\displaystyle \frac{1}{2}}(c_t^{\top }M{c}_t-A_t^{\top }M{c}_t-c_t^{\top }MA+A_t^{\top }MA-A_t^{\top }MA\hfill \\ \hfill & +{({\hat{c}}_t-{\gamma }^{*})}^{\top }{\Sigma }_n^{-1}({\hat{c}}_t-{\gamma }^{*})),\hfill \end{array}$$

(A12)

where we have defined

$$\begin{array}{cc}\hfill M& ={\Sigma }_c^{-1}+{\Sigma }_n^{-1}\hfill \end{array}$$

(A13)

$$\begin{array}{cc}\hfill A_t& ={\left(M^{-1}\right)}^{\top }\left({\Sigma }_c^{-1}{\mu }_c+{\Sigma }_n^{-1}({\hat{c}}_t-{\gamma }^{*})\right).\hfill \end{array}$$

(A14)

From (A12), we obtain

$$\begin{array}{c}\hfill log\left(P\left(c_t\right|{\hat{c}}_t,{\gamma }^{*})\right)\propto -{\displaystyle \frac{1}{2}}\left(c_t^{\top }M{c}_t-A_t^{\top }M{c}_t-c_t^{\top }MA+A_t^{\top }MA\right).\end{array}$$

This implies that

$$\begin{array}{cc}\hfill P\left(c_t\right|{\hat{c}}_t,{\gamma }^{*})& =\mathcal{N}(A_t,M^{-1}).\hfill \end{array}$$

(A15)

Appendix B.2.2. Derivation of $P\left({\hat{c}}_t\right|{\gamma }^{*})$

We now derive the distribution $P\left({\hat{c}}_t\right|{\gamma }^{*})$ which is defined in (3) as

$$\begin{array}{c}\hfill P\left({\hat{c}}_t\right|{\gamma }^{*})={\mathbb{E}}_{P\left(c_t\right)}\left[P\left({\hat{c}}_t\right|c_t,{\gamma }^{*})\right].\end{array}$$

Hence, $P\left({\hat{c}}_t\right|{\gamma }^{*})$ can be obtained by marginalizing the joint distribution $P\left(c_t\right)P\left({\hat{c}}_t\right|c_t,{\gamma }^{*})=P\left(c_t\right|{\hat{c}}_t,{\gamma }^{*})P\left({\hat{c}}_t\right|{\gamma }^{*})$ over $c_t$. To this end, we use (A12) to obtain,

$$\begin{array}{cc}\hfill log\left(P\left(c_t\right)P\left({\hat{c}}_t\right|c_t,{\gamma }^{*})\right)& =log\left(P\left(c_t\right|{\hat{c}}_t,{\gamma }^{*})P\left({\hat{c}}_t\right|{\gamma }^{*})\right)\hfill \\ & \propto log\left(P\left(c_t\right|{\hat{c}}_t,{\gamma }^{*})\right)-{\displaystyle \frac{1}{2}}\left(-A_t^{\top }MA+{({\hat{c}}_t-{\gamma }^{*})}^{\top }{\Sigma }_n^{-1}({\hat{c}}_t-{\gamma }^{*})\right),\hfill \end{array}$$

which implies that

$$\begin{array}{cc}\hfill log\left(P\left({\hat{c}}_t\right|{\gamma }^{*})\right)& \propto -{\displaystyle \frac{1}{2}}\left(-A_t^{\top }MA+{({\hat{c}}_t-{\gamma }^{*})}^{\top }{\Sigma }_n^{-1}({\hat{c}}_t-{\gamma }^{*})\right)\hfill \\ \hfill & \propto -{\displaystyle \frac{1}{2}}({\hat{c}}_t^{\top }\left({\Sigma }_n^{-1}-{\Sigma }_n^{-1}{\left(M^{-1}\right)}^{\top }{\Sigma }_n^{-1}\right){\hat{c}}_t-{\hat{c}}_t^{\top }({\Sigma }_n^{-1}{\left(M^{-1}\right)}^{\top }(-{\Sigma }_n^{-1}{\gamma }^{*}+{\Sigma }_c^{-1}{\mu }_c)\hfill \\ & +{\Sigma }_n^{-1}{\gamma }^{*})-\left(({\mu }_c^{\top }{\Sigma }_c^{-1}-{\gamma }^{*\top }{\Sigma }_n^{-1})\left(M^{-1}\right){\Sigma }_n^{-1}+{\gamma }^{*\top }{\Sigma }_n^{-1}\right)\hat{c})\hfill \\ \hfill & =-{\displaystyle \frac{1}{2}}\left({\hat{c}}_t^{\top }G{\hat{c}}_t-F^{\top }G{\hat{c}}_t-G^{\top }F{\hat{c}}_t\right)\hfill \\ & \propto -{\displaystyle \frac{1}{2}}\left({({\hat{c}}_t-F)}^{\top }G({\hat{c}}_t-F)\right),\hfill \end{array}$$

where

$$\begin{array}{cc}\hfill G& ={\Sigma }_n^{-1}-{\Sigma }_n^{-1}{\left(M^{-1}\right)}^{\top }{\Sigma }_n^{-1}\hfill \end{array}$$

(A16)

$$\begin{array}{cc}\hfill F& ={\left(G^{-1}\right)}^{\top }\left({\Sigma }_n^{-1}{\left(M^{-1}\right)}^{\top }(-{\Sigma }_n^{-1}{\gamma }^{*}+{\Sigma }_c^{-1}{\mu }_c)+{\Sigma }_n^{-1}{\gamma }^{*}\right)={\left(G^{-1}\right)}^{\top }(G{\gamma }^{*}+{\Sigma }_n^{-1}{\left(M^{-1}\right)}^{\top }{\Sigma }_c^{-1}{\mu }_c).\hfill \end{array}$$

(A17)

Thus,

$$\begin{array}{c}\hfill P\left({\hat{c}}_t\right|{\gamma }^{*})=\mathcal{N}(F,G^{-1}).\end{array}$$

(A18)

Appendix B.2.3. Derivation of $P\left({\gamma }^{*}\right|{\mathcal{H}}_{t-1,c,\hat{c}})$

We now derive the posterior distribution $P\left({\gamma }^{*}\right|{\mathcal{H}}_{t-1,c,\hat{c}})$. To this end, we use Baye’s theorem as

$$\begin{array}{cc}\hfill P\left({\gamma }^{*}\right|{\mathcal{H}}_{t-1,c,\hat{c}})& \propto \prod _{\tau =1}^{t-1}P\left({\hat{c}}_{\tau }\right|c_{\tau },{\gamma }^{*})P\left({\gamma }^{*}\right)\hfill \\ \hfill & =\prod _{\tau =1}^{t-1}\mathcal{N}(c_{\tau }+{\gamma }^{*},{\Sigma }_n)\mathcal{N}(\mathbf{0},{\Sigma }_{\gamma }).\hfill \end{array}$$

We then have,

$$\begin{array}{cc}\hfill logP\left({\gamma }^{*}\right|{\mathcal{H}}_{t-1,c,\hat{c}})& \propto -{\displaystyle \frac{1}{2}}\left(\sum _{\tau =1}^{t-1}\left({({\hat{c}}_{\tau }-c_{\tau }-{\gamma }^{*})}^{\top }{\Sigma }_n^{-1}({\hat{c}}_{\tau }-c_{\tau }-{\gamma }^{*})\right)+{\gamma }^{*\top }{\Sigma }_{\gamma }^{-1}{\gamma }^{*}\right)\hfill \\ \hfill & =-{\displaystyle \frac{1}{2}}\left(\sum _{\tau =1}^{t-1}\left({(-{\hat{c}}_{\tau }+c_{\tau }+{\gamma }^{*})}^{\top }{\Sigma }_n^{-1}(-{\hat{c}}_{\tau }+c_{\tau }+{\gamma }^{*})\right)+{\gamma }^{*\top }{\Sigma }_{\gamma }^{-1}{\gamma }^{*}\right)\hfill \\ \hfill & \propto -{\displaystyle \frac{1}{2}}({\gamma }^{*\top }((t-1){\Sigma }_n^{-1}+{\Sigma }_{\gamma }^{-1}){\gamma }^{*}-{\gamma }^{*\top }{\Sigma }_n^{-1}\left(\sum _{\tau =1}^{t-1}({\hat{c}}_{\tau }-c_{\tau })\right)\hfill \\ & -{\left(\sum _{\tau =1}^{t-1}({\hat{c}}_{\tau }-c_{\tau })\right)}^{\top }{\Sigma }_n^{-1}{\gamma }^{*}).\hfill \end{array}$$

Consequently, we obtain that,

$$\begin{array}{c}\hfill P\left({\gamma }^{*}\right|{\mathcal{H}}_{t-1,c,\hat{c}})=\mathcal{N}\left({\gamma }^{*}\right|Y_t,W_t^{-1})\end{array}$$

(A19)

where

$$\begin{array}{cc}\hfill W_t& =(t-1){\Sigma }_n^{-1}+{\Sigma }_{\gamma }^{-1}\hfill \end{array}$$

(A20)

$$\begin{array}{cc}\hfill Y_t& ={\left(W_t^{-1}\right)}^{\top }{\Sigma }_n^{-1}\sum _{\tau =1}^{t-1}({\hat{c}}_{\tau }-c_{\tau }).\hfill \end{array}$$

(A21)

Appendix B.2.4. Derivation of Posterior Predictive Distribution $P\left(c_t\right|{\hat{c}}_t,{\mathcal{H}}_{t-1,c,\hat{c}})$

Using results from previous subsections, we are now ready to derive the posterior predictive distribution $P\left(c_t\right|{\hat{c}}_t,{\mathcal{H}}_{t-1,c,\hat{c}})$. Note, that $P\left(c_t\right|{\hat{c}}_t,{\mathcal{H}}_{t-1,c,\hat{c}})={\mathbb{E}}_{P\left({\gamma }^{*}\right|{\mathcal{H}}_{t-1,c,\hat{c}})}\left[P\left(c_t\right|{\hat{c}}_t,{\gamma }^{*})\right]$. We then have the following set of relations:

$$\begin{array}{cc}\hfill & log\left(P\left({\gamma }^{*}\right|{\mathcal{H}}_{t-1,c,\hat{c}})P\left(c_t\right|{\hat{c}}_t,{\gamma }^{*})\right)\hfill \\ & \propto -{\displaystyle \frac{1}{2}}\left({(c_t-A_t)}^{\top }M(c_t-A_t)+{({\gamma }^{*}-Y_t)}^{\top }{W}_t({\gamma }^{*}-Y_t)\right)\hfill \\ \hfill & =-{\displaystyle \frac{1}{2}}({(c_t-D-E_t+{\left(M^{-1}\right)}^{\top }{\Sigma }_n^{-1}{\gamma }^{*})}^{\top }M(c_t-D-E_t+{\left(M^{-1}\right)}^{\top }{\Sigma }_n^{-1}{\gamma }^{*})\hfill \\ & +{({\gamma }^{*}-Y_t)}^{\top }{W}_t({\gamma }^{*}-Y_t))\hfill \\ \hfill & \propto -{\displaystyle \frac{1}{2}}({\gamma }^{*\top }({\Sigma }_n^{-1}{\left(M^{-1}\right)}^{\top }{\Sigma }_n^{-1}+W_t){\gamma }^{*}-{\gamma }^{*\top }({\Sigma }_n^{-1}(D+E_t-c_t)+W_t{Y}_t)-({(D+E_t-c_t)}^{\top }{\Sigma }_n^{-1}\hfill \\ & +Y_t^{\top }{W}_t){\gamma }^{*}+{(c_t-D-E_t)}^{\top }M(c_t-D-E_t))\hfill \\ \hfill & =-{\displaystyle \frac{1}{2}}\left({\gamma }^{*\top }{\tilde{H}}_t{\gamma }^{*}-{\gamma }^{*\top }{\tilde{H}}_t^{\top }{\tilde{J}}_t-{\tilde{J}}_t^{\top }{\tilde{H}}_t{\gamma }^{*}+{\tilde{J}}_t^{\top }{\tilde{H}}_t{\tilde{J}}_t-{\tilde{J}}_t^{\top }{\tilde{H}}_t{\tilde{J}}_t+{(c_t-D-E_t)}^{\top }M(c_t-D-E_t)\right)\hfill \\ \hfill & \propto log\left(P\left({\gamma }^{*}\right|{\mathcal{H}}_{t,c,\hat{c}})\right)-{\displaystyle \frac{1}{2}}\left(-{\tilde{J}}_t^{\top }{\tilde{H}}_t{\tilde{J}}_t+{(c_t-D-E_t)}^{\top }M(c_t-D-E_t)\right)\hfill \end{array}$$

where $D={\left(M^{-1}\right)}^{\top }{\Sigma }_c^{-1}{\mu }_c$, $E_t={\left(M^{-1}\right)}^{\top }{\Sigma }_n^{-1}{\hat{c}}_t$, ${\tilde{H}}_t={\Sigma }_n^{-1}{\left(M^{-1}\right)}^{\top }{\Sigma }_n^{-1}+W_t$ and ${\tilde{J}}_t={\left({\tilde{H}}_t^{-1}\right)}^{\top }({\Sigma }_n^{-1}(D+E_t-c_t)+W_t{Y}_t)$.

Since $log\left(P\left({\gamma }^{*}\right|{\mathcal{H}}_{t-1,c,\hat{c}})P\left(c_t\right|{\hat{c}}_t,{\gamma }^{*})\right)=log\left(P\left(c_t\right|{\hat{c}}_t,{\mathcal{H}}_{t-1,c,\hat{c}})P\left({\gamma }^{*}\right|{\mathcal{H}}_{t,c,\hat{c}})\right)$, we obtain

$$\begin{array}{cc}\hfill log\left(P\left(c_t\right|{\hat{c}}_t,{\mathcal{H}}_{t-1,c,\hat{c}})\right)& \propto -{\displaystyle \frac{1}{2}}\left(-{\tilde{J}}_t^{\top }{\tilde{H}}_t{\tilde{J}}_t+{(c_t-D-E_t)}^{\top }M(c_t-D-E_t)\right)\hfill \\ \hfill & \propto -{\displaystyle \frac{1}{2}}(c_t^{\top }\left(M-{\Sigma }_n^{-1}{\left({\tilde{H}}_t^{-1}\right)}^{\top }{\Sigma }_n^{-1}\right)c_t-c_t^{\top }(M(D+E_t)\hfill \\ \hfill & -{\Sigma }_n^{-1}{\left({\tilde{H}}_t^{-1}\right)}^{\top }({\Sigma }_n^{-1}(D+E_t)+W_t{Y}_t))-(M(D+E_t)\hfill \\ & -{\Sigma }_n^{-1}{\left({\tilde{H}}_t^{-1}\right)}^{\top }({\Sigma }_n^{-1}(D+E_t)+W_t{Y}_t){)}^{\top }{c}_t).\hfill \end{array}$$

This gives

$$\begin{array}{cc}\hfill P\left(c_t\right|{\hat{c}}_t,{\mathcal{H}}_{t-1,c,\hat{c}})& =\mathcal{N}({\tilde{V}}_t,{\tilde{R}}_t^{-1})where\hfill \end{array}$$

(A22)

$$\begin{array}{cc}\hfill {\tilde{R}}_t& =M-{\Sigma }_n^{-1}{\left({\tilde{H}}_t^{-1}\right)}^{\top }{\Sigma }_n^{-1}\hfill \end{array}$$

(A23)

$$\begin{array}{cc}\hfill {\tilde{V}}_t& ={\left({\tilde{R}}_t^{-1}\right)}^{\top }\left(M(D+E_t)-{\Sigma }_n^{-1}{\left({\tilde{H}}_t^{-1}\right)}^{\top }({\Sigma }_n^{-1}(D+E_t)+W_t{Y}_t)\right).\hfill \end{array}$$

(A24)

Appendix B.3. Proof of Lemma 3

We now present the proof of Lemma 3. To this end, we first recall that ${\mathbb{E}}_t[·]=\mathbb{E}[·|{\mathcal{F}}_t]$ where ${\mathcal{F}}_t={\mathcal{H}}_{t-1,r,a,c,\hat{c}}\cup {\hat{c}}_t$, and we denote $P_t\left({\theta }^{*}\right)$ as the posterior distribution $P\left({\theta }^{*}\right|{\mathcal{H}}_{t-1,r,a,c})$ of ${\theta }^{*}$ given the history of observed reward-action-context tuples. We can then equivalently write ${\mathcal{R}}_{\mathrm{d},\mathrm{CB}}^T$ as

$$\begin{array}{cc}\hfill {\mathcal{R}}_{\mathrm{d},\mathrm{CB}}^T& =\sum _{t=1}^T\mathbb{E}\left[{\mathbb{E}}_t\left[\psi {({\hat{a}}_t,{\hat{c}}_t|{\mathcal{H}}_{c,\hat{c}})}^{\top }{\theta }^{*}-\psi {(a_t,{\hat{c}}_t|{\mathcal{H}}_{c,\hat{c}})}^{\top }{\theta }^{*}\right]\right]\hfill \\ \hfill & =\sum _{t=1}^T\mathbb{E}\left[{\mathbb{E}}_t\underset{:={\Delta }_t}{\underbrace{\left[f({\theta }^{*},{\hat{a}}_t,c_t)-f({\theta }^{*},a_t,c_t)\right]}}\right],\hfill \end{array}$$

(A25)

where $\psi (a,{\hat{c}}_t|{\mathcal{H}}_{c,\hat{c}})={\mathbb{E}}_{P\left(c_t\right|{\hat{c}}_t,{\mathcal{H}}_{t-1,c,\hat{c}})}\left[\varphi (a,c_t)\right]$ is as defined in (33) and we have used $f({\theta }^{*},a_t,c_t)=\varphi {(a_t,c_t)}^{\top }{\theta }^{*}$ to denote the mean-reward function. To obtain an upper bound on ${\mathcal{R}}_{\mathrm{d},\mathrm{CB}}^T$, we define the following lifted information ratio as in [19],

$$\begin{array}{c}\hfill {\mathsf{\Gamma }}_t={\displaystyle \frac{{\left({\mathbb{E}}_t\left[{\Delta }_t\right]\right)}^2}{{\mathsf{\Lambda }}_t}}\end{array}$$

(A26)

where

$$\begin{array}{c}\hfill {\mathsf{\Lambda }}_t={\mathbb{E}}_t\left[{\left(f({\theta }^{*},a_t,c_t)-\overline{f}(a_t,c_t)\right)}^2\right],\end{array}$$

(A27)

with $\overline{f}(a_t,c_t)={\mathbb{E}}_t\left[f({\theta }^{*},a_t,c_t)\right|a_t,c_t]$ denoting the expectation of mean reward with respect to the posterior distribution $P_t\left({\theta }^{*}\right)$. Subsequently, we obtain the following upper bound

$$\begin{array}{c}\hfill {\mathcal{R}}_{\mathrm{d},\mathrm{CB}}^T\le \sum _{t=1}^T\mathbb{E}\left[\sqrt{{\mathsf{\Gamma }}_t{\mathsf{\Lambda }}_t}\right]\le \sqrt{\mathbb{E}\left[\sum _{t=1}^T{\mathsf{\Gamma }}_t\right]\left[\sum _{t=1}^T\mathbb{E}\left[{\mathsf{\Lambda }}_t\right]\right]},\end{array}$$

(A28)

where the last inequality follows by an application of Cauchy–Schwarz inequality. An upper bound on ${\mathcal{R}}_{\mathrm{d},\mathrm{CB}}^T$ then follows by obtaining an upper bound on the lifted information ratio ${\mathsf{\Gamma }}_t$ as well as on ${\mathsf{\Lambda }}_t.$

We first evaluate the term ${\mathsf{\Lambda }}_t$. To this end, note that $\overline{f}(a_t,c_t)=\varphi {(a_t,c_t)}^{\top }{\tilde{\mu }}_{t-1}$, with ${\tilde{\mu }}_{t-1}$ defined as in (32). Using this, we obtain

$$\begin{array}{cc}\hfill {\mathsf{\Lambda }}_t& ={\mathbb{E}}_t\left[{\left(\varphi {(a_t,c_t)}^{\top }({\theta }^{*}-{\tilde{\mu }}_{t-1})\right)}^2\right]\hfill \end{array}$$

$$\begin{array}{cc}\hfill & ={\mathbb{E}}_t\left[\varphi {(a_t,c_t)}^{\top }({\theta }^{*}-{\tilde{\mu }}_{t-1}){({\theta }^{*}-{\tilde{\mu }}_{t-1})}^{\top }\varphi (a_t,c_t)\right]\hfill \end{array}$$

(A29)

$$\begin{array}{cc}\hfill & ={\mathbb{E}}_t\left[\varphi {(a_t,c_t)}^{\top }{\tilde{\Sigma }}_{t-1}^{-1}\varphi (a_t,c_t)\right]={\mathbb{E}}_t\left[\parallel \varphi (a_t,c_t){\parallel }_{{\tilde{\Sigma }}_{t-1}^{-1}}^2\right]\hfill \end{array}$$

(A30)

where ${\tilde{\Sigma }}_{t-1}$ is as in (31), and the third equality follows since conditional on ${\mathcal{F}}_t$, $(a_t,c_t)$ is independent of ${\theta }^{*}$. Subsequently, we can apply the elliptical potential lemma Lemma 19.4 in [24] using the assumption that ${\parallel \varphi (·,·)\parallel }_2\le 1$ and that ${\sigma }^2/\lambda \ge 1$. This results in

$$\begin{array}{cc}\hfill \sum _{t=1}^T{\parallel \varphi (a_t,c_t)\parallel }_{{\tilde{\Sigma }}_{t-1}^{-1}}^2& ={\sigma }^2\sum _{t=1}^T{\parallel \varphi (a_t,c_t)\parallel }_{{\left({\sigma }^2{\tilde{\Sigma }}_{t-1}\right)}^{-1}}^2\hfill \\ & \le 2{\sigma }^{2}log{\displaystyle \frac{\det \left({\tilde{\Sigma }}_T\right)}{\det ({\sigma }^2/\lambda \mathbb{I})}}=2{\sigma }^2\left(mlog\left({\sigma }^2/\lambda +T/m\right)-mlog({\sigma }^2/\lambda )\right)\hfill \\ & =2m{\sigma }^{2}log\left(1+{\displaystyle \frac{T\lambda }{m{\sigma }^2}}\right).\hfill \end{array}$$

(A31)

To upper bound the lifted information ratio term ${\mathsf{\Gamma }}_t$, we can use Lemma 7 in [19]. To demonstrate how to leverage results from [19], we start by showing that the inequality ${\mathsf{\Gamma }}_t\le m$ holds. To this end, we note that the lifted information ratio can be equivalently written as

$$\begin{array}{c}\hfill {\mathsf{\Gamma }}_t={\displaystyle \frac{{\left({\mathbb{E}}_t\left[f({\theta }_t,a_t,c_t)-\overline{f}(a_t,c_t)\right]\right)}^2}{{\mathsf{\Lambda }}_t}}\end{array}$$

(A32)

which follows since

$$\begin{array}{cc}\hfill {\mathbb{E}}_t\left[{\Delta }_t\right]& ={\mathbb{E}}_t\left[f({\theta }^{*},{\hat{a}}_t,c_t)-f({\theta }^{*},a_t,c_t)\right]\hfill \\ \hfill & ={\mathbb{E}}_t\left[f({\theta }^{*},{\hat{a}}_t,c_t)-\overline{f}(a_t,c_t)\right]\hfill \\ \hfill & =\sum _{a^{\prime }}{P}_t({\hat{a}}_t=a^{\prime }){\mathbb{E}}_t\left[f({\theta }^{*},a^{\prime },c_t)|{\hat{a}}_t=a^{\prime }\right]-\sum _{a^{\prime }}{P}_t(a_t=a^{\prime }){\mathbb{E}}_t\left[\overline{f}(a_t=a^{\prime },c_t)\right]\hfill \\ \hfill & =\sum _{a^{\prime }}{P}_t({\hat{a}}_t=a^{\prime })\left({\mathbb{E}}_t\left[f({\theta }^{*},a^{\prime },c_t)|{\hat{a}}_t=a^{\prime }\right]-{\mathbb{E}}_t\left[\overline{f}(a^{\prime },c_t)\right]\right),\hfill \end{array}$$

(A33)

where the second equality holds since conditioned on ${\mathcal{F}}_t$, $(a_t,c_t)$ and ${\theta }^{*}$ are independent. In the third equality, we denote $P_t\left({\hat{a}}_t\right)=P\left({\hat{a}}_t\right|{\mathcal{F}}_t)$ and $P_t\left(a_t\right)=P\left(a_t\right|{\mathcal{F}}_t)$. Using these, the last equality follows since $a_t\stackrel{d}{=}{\hat{a}}_t$, i.e, $P_t\left(a_t\right)=P_t\left({\hat{a}}_t\right)$. Now, let us define a $K\times K$ matrix M with entries given by

$$\begin{array}{c}\hfill M_{a,a^{\prime }}=\sqrt{P_t({\hat{a}}_t=a^{\prime })P_t(a_t=a)}\left({\mathbb{E}}_t\left[f({\theta }^{*},a,c_t)|{\hat{a}}_t=a^{\prime }\right]-{\mathbb{E}}_t\left[\overline{f}(a,c_t)\right]\right).\end{array}$$

(A34)

Using this and noting that $P_t\left({\hat{a}}_t\right)=P_t\left(a_t\right)$, we obtain that ${\mathbb{E}}_t\left[{\Delta }_t\right]=\mathrm{Tr}\left(M\right)$. We now try to bound ${\mathsf{\Lambda }}_t$ in terms of the matrix M. To see this, we can equivalently write ${\mathsf{\Lambda }}_t$ as

$$\begin{array}{cc}\hfill {\mathsf{\Lambda }}_t& =\sum _a{P}_t(a_t=a){\mathbb{E}}_t\left[{\left(f({\theta }^{*},a,c_t)-\overline{f}(a,c_t)\right)}^2\right]\hfill \\ \hfill & =\sum _{a,a^{\prime }}{P}_t(a_t=a)P_t({\hat{a}}_t=a^{\prime }){\mathbb{E}}_t\left[{\left(f({\theta }^{*},a,c_t)-\overline{f}(a,c_t)\right)}^2|{\hat{a}}_t=a^{\prime }\right]\hfill \\ \hfill & \ge \sum _{a,a^{\prime }}{P}_t(a_t=a)P_t({\hat{a}}_t=a^{\prime }){\left({\mathbb{E}}_t\left[f({\theta }^{*},a,c_t)-\overline{f}(a,c_t)|{\hat{a}}_t=a^{\prime }\right]\right)}^2\hfill \\ \hfill & =\sum _{a,a^{\prime }}{P}_t(a_t=a)P_t({\hat{a}}_t=a^{\prime }){\left({\mathbb{E}}_t\left[f({\theta }^{*},a,c_t)\right|{\hat{a}}_t=a^{\prime }]-{\mathbb{E}}_t\left[\overline{f}(a,c_t)\right]\right)}^2={\parallel M\parallel }_F^2,\hfill \end{array}$$

(A35)

where the inequality follows by the application of Jensen’s inequality. We thus obtain that

$${\mathsf{\Gamma }}_t\le {\displaystyle \frac{\mathrm{Tr}{\left(M\right)}^2}{{\parallel M\parallel }_F^2}}\le m,$$

where the last inequality follows from Prop. 5 in [25]. From Lemma 3 in [19], we also obtain ${\mathsf{\Gamma }}_t\le 2log(1+K)$. This results in the upper bound ${\mathsf{\Gamma }}_t\le min\{m,2log(1+K)\}$.

Using this and the bound of (A31) in (A28), we obtain

$$\begin{array}{c}\hfill {\mathcal{R}}_{\mathrm{d},\mathrm{CB}}^T\le \sqrt{2Tm{\sigma }^{2}min\{m,2(1+logK)\}log\left(1+{\displaystyle \frac{T\lambda }{m{\sigma }^2}}\right)}.\end{array}$$

(A36)

Appendix B.4. Proof of Lemma 4

We now prove an upper bound on the term ${\mathcal{R}}_{\mathrm{d},\mathrm{EE}1}^T$. To this end, let us define the following event:

$$\begin{array}{c}\hfill \mathcal{E}:=\left\{\parallel {\theta }^{*}{\parallel }_2\le \sqrt{2\lambda mlog\left({\displaystyle \frac{2m}{\delta }}\right)}:=U\right\}.\end{array}$$

(A37)

Note, that since ${\theta }^{*}\sim \mathcal{N}\left({\theta }^{*}\right|\mathbf{0},\lambda \mathbb{I})$, we obtain that with probability at least $1-\delta$, the following inequality holds $\parallel {\theta }^{*}{\parallel }_{\infty }\le \sqrt{2\lambda log\left({\displaystyle \frac{2m}{\delta }}\right)}$. Since $\parallel {\theta }^{*}{\parallel }_2\le \sqrt{m}{\parallel {\theta }^{*}\parallel }_{\infty }$, the above inequality in turn implies the event $\mathcal{E}$ such that $P\left(\mathcal{E}\right)\ge 1-\delta$.

$$\begin{array}{cc}\hfill {\mathcal{R}}_{\mathrm{d},\mathrm{EE}1}^T& \stackrel{\left(a\right)}{\le }\sum _{t=1}^T\mathbb{E}\left[\psi {(a_t^{*},{\hat{c}}_t|{\gamma }^{*})}^{\top }{\theta }^{*}-\psi {(a_t^{*},{\hat{c}}_t|{\mathcal{H}}_{c,\hat{c}})}^{\top }{\theta }^{*}\right]\hfill \\ \hfill & =\sum _{t=1}^T\mathbb{E}\left[\underset{:=\Delta \left(P\left(c_t\right|{\hat{c}}_t,{\gamma }^{*}),P\left(c_t\right|{\hat{c}}_t,{\mathcal{H}}_{c,\hat{c}})\right)}{\underbrace{{\mathbb{E}}_{P\left(c_t\right|{\hat{c}}_t,{\gamma }^{*})}\left[\varphi {(a_t^{*},c_t)}^{\top }{\theta }^{*}\right]-{\mathbb{E}}_{P\left(c_t\right|{\hat{c}}_t,{\mathcal{H}}_{c,\hat{c}})}\left[\varphi {(a_t^{*},c_t)}^{\top }{\theta }^{*}\right]}}\right],\hfill \end{array}$$

$$\begin{array}{cc}\hfill & =\sum _{t=1}^T\mathbb{E}[\Delta \left(P\left(c_t\right|{\hat{c}}_t,{\gamma }^{*}),P\left(c_t\right|{\hat{c}}_t,{\mathcal{H}}_{c,\hat{c}})\right)\mathbf{1}\left\{\mathcal{E}\right\}]+\sum _{t=1}^T\mathbb{E}[\Delta \left(P\left(c_t\right|{\hat{c}}_t,{\gamma }^{*}),P\left(c_t\right|{\hat{c}}_t,{\mathcal{H}}_{c,\hat{c}})\right)\mathbf{1}\left\{{\mathcal{E}}^c\right\}]\hfill \end{array}$$

(A38)

$$\begin{array}{cc}\hfill & \stackrel{\left(b\right)}{\le }\sum _{t=1}^T\mathbb{E}[\Delta \left(P\left(c_t\right|{\hat{c}}_t,{\gamma }^{*}),P\left(c_t\right|{\hat{c}}_t,{\mathcal{H}}_{c,\hat{c}})\right)\mathbf{1}\left\{\mathcal{E}\right\}]+2\delta T\mathbb{E}[\parallel {{\theta }^{*}\parallel }_2\left|{\mathcal{E}}^c\right],\hfill \end{array}$$

(A39)

where the inequality $\left(a\right)$ follows from the definition of ${\hat{a}}_t=arg{max}_{a\in \mathcal{A}}\psi {(a,\hat{c}|{\mathcal{H}}_{c,\hat{c}})}^{\top }{\theta }^{*}$, and $\mathbf{1}\{•\}$ denotes the indicator function which takes value 1 when • is true and takes value 0 otherwise. The inequality in $\left(b\right)$ follows by noting that

$$\begin{array}{cc}\hfill & \mathbb{E}[\Delta \left(P\left(c_t\right|{\hat{c}}_t,{\gamma }^{*}),P\left(c_t\right|{\hat{c}}_t,{\mathcal{H}}_{c,\hat{c}})\right)\mathbb{I}\left\{{\mathcal{E}}^c\right\}]\hfill \\ & \le \mathbb{E}\left[{\left({\mathbb{E}}_{P\left(c_t\right|{\hat{c}}_t,{\gamma }^{*})}\left[\varphi (a_t^{*},c_t)\right]-{\mathbb{E}}_{P\left(c_t\right|{\hat{c}}_t,{\mathcal{H}}_{c,\hat{c}})}\left[\varphi (a_t^{*},c_t)\right]\right)}^{\top }{\theta }^{*}\mathbf{1}\left\{{\mathcal{E}}^c\right\}\right]\hfill \\ \hfill & \le \mathbb{E}\left[\parallel {\mathbb{E}}_{P\left(c_t\right|{\hat{c}}_t,{\gamma }^{*})}\left[\varphi (a_t^{*},c_t)\right]-{\mathbb{E}}_{P\left(c_t\right|{\hat{c}}_t,{\mathcal{H}}_{c,\hat{c}})}\left[\varphi (a_t^{*},c_t)\right]{\parallel }_2{\parallel {\theta }^{*}\parallel }_2\mathbf{1}\left\{{\mathcal{E}}^c\right\}\right]\hfill \\ \hfill & \le 2\mathbb{E}\left[\parallel {\theta }^{*}{\parallel }_2\mathbf{1}\left\{{\mathcal{E}}^c\right\}\right]=2P\left({\mathcal{E}}^c\right)\mathbb{E}[\parallel {\theta }^{*}{\parallel }_2|{\mathcal{E}}^c]\le 2\delta \mathbb{E}[\parallel {\theta }^{*}{\parallel }_2\left|{\mathcal{E}}^c\right],\hfill \end{array}$$

where the last inequality is due to $P\left({\mathcal{E}}^c\right)=1-P\left(\mathcal{E}\right)\le \delta$. To obtain an upper bound on $\mathbb{E}[\parallel {\theta }^{*}{\parallel }_2|{\mathcal{E}}^c]$, we note that the following set of inequalities hold:

$$\begin{array}{cc}\hfill \mathbb{E}[\parallel {\theta }^{*}{\parallel }_2|{\mathcal{E}}^c]& \stackrel{\left(a\right)}{\le }\sqrt{m}\mathbb{E}[\parallel {\theta }^{*}{\parallel }_{\infty }|{\mathcal{E}}^c]\stackrel{\left(b\right)}{=}\sqrt{m}\mathbb{E}[\parallel {\theta }^{*}{\parallel }_{\infty }|\parallel {\theta }^{*}{\parallel }_{\infty }>u]\hfill \\ \hfill & =\sqrt{m}\sum _{i=1}^{m}P(\parallel {\theta }^{*}{\parallel }_{\infty }=|{\theta }_i^{*}\left|\right)\mathbb{E}[\parallel {\theta }^{*}{\parallel }_{\infty }\left|\parallel {\theta }^{*}{\parallel }_{\infty }=|{\theta }_i^{*}|,\parallel {\theta }^{*}{\parallel }_{\infty }>u\right]\hfill \\ \hfill & \le \sqrt{m}\sum _{i=1}^m\mathbb{E}[\left|{\theta }_i^{*}\right|\left||{\theta }_i^{*}|>u\right]\stackrel{\left(c\right)}{=}2\sqrt{m}\sum _{i=1}^m{\int }_{x>u}xg\left(x\right)dx\hfill \\ \hfill & \stackrel{\left(d\right)}{=}-2\lambda \sqrt{m}\sum _{i=1}^m{\int }_{x>u}{g}^{\prime }\left(x\right)dx=2\lambda m^{3/2}g\left(u\right)=2\lambda m^{3/2}{\displaystyle \frac{1}{\sqrt{2\pi \lambda }}}\exp (-u^2/2\lambda )\hfill \\ \hfill & =\delta \sqrt{{\displaystyle \frac{m\lambda }{2\pi }}},\hfill \end{array}$$

(A40)

where $\left(a\right)$ follows since ${\parallel \theta \parallel }_2\le \sqrt{m}{\parallel \theta \parallel }_{\infty }$, $\left(b\right)$ follows since $\parallel {\theta }^{*}{\parallel }_2>\sqrt{m}\sqrt{2\lambda log\left({\displaystyle \frac{2m}{\delta }}\right)}$ implies that $\parallel {\theta }^{*}{\parallel }_{\infty }>\sqrt{2\lambda log\left({\displaystyle \frac{2m}{\delta }}\right)}:=u$. The equality in $\left(c\right)$ follows by noting that $|{\theta }_i^{*}|$, where ${\theta }_i^{*}\sim \mathcal{N}(0,\lambda )$, follows a folded Gaussian distribution with density $2g\left({\theta }_i^{*}\right)$ where $g\left({\theta }_i^{*}\right)={\displaystyle \frac{1}{\sqrt{2\pi \lambda }}}\exp (-{\theta }_i^{*2}/\left(2\lambda \right))$ is the Gaussian density. The equality in $\left(d\right)$ follows by noting that $xg\left(x\right)=-\lambda g^{\prime }\left(x\right)$, where $g^{\prime }\left(x\right)$ is the derivative of the Gaussian density. Thus, we have the following upper bound

$$\begin{array}{cc}\hfill \sum _{t=1}^T\mathbb{E}[\Delta \left(P\left(c_t\right|{\hat{c}}_t,{\gamma }^{*}),P\left(c_t\right|{\hat{c}}_t,{\mathcal{H}}_{c,\hat{c}})\right)\mathbf{1}\left\{{\mathcal{E}}^c\right\}]& \le 2T{\delta }^2\sqrt{{\displaystyle \frac{m\lambda }{2\pi }}}.\hfill \end{array}$$

(A41)

We now obtain an upper bound on ${\sum }_{t=1}^T\mathbb{E}\left[\Delta \left(P\left(c_t\right|{\hat{c}}_t,{\gamma }^{*}),P\left(c_t\right|{\hat{c}}_t,{\mathcal{H}}_{c,\hat{c}})\right)\mathbf{1}\left\{\mathcal{E}\right\}\right]$. To this end, note that

$$\begin{array}{cc}\hfill \mathbb{E}\left[\Delta \left(P\left(c_t\right|{\hat{c}}_t,{\gamma }^{*}),P\left(c_t\right|{\hat{c}}_t,{\mathcal{H}}_{c,\hat{c}})\right)\mathbf{1}\left\{\mathcal{E}\right\}\right]& \le P\left(\mathcal{E}\right)\mathbb{E}\left[\Delta \left(P\left(c_t\right|{\hat{c}}_t,{\gamma }^{*}),P\left(c_t\right|{\hat{c}}_t,{\mathcal{H}}_{c,\hat{c}})\right)|\mathcal{E}\right]\hfill \\ \hfill & \le \mathbb{E}[\Delta \left(P\left(c_t\right|{\hat{c}}_t,{\gamma }^{*}),P\left(c_t\right|{\hat{c}}_t,{\mathcal{H}}_{c,\hat{c}})\right)\left|\mathcal{E}\right].\hfill \end{array}$$

(A42)

Note, that under the event $\mathcal{E}$, we have the following relation,

$$|\varphi {(a_t^{*},c_t)}^{\top }{\theta }^{*}\left)\right|\le \parallel \varphi (a_t^{*},c_t){\parallel }_2{\parallel {\theta }^{*}\parallel }_2\le U,$$

whereby $\varphi {(a_t^{*},c_t)}^{\top }{\theta }^{*}$ is $U^2$-sub-Gaussian.

Consequently, applying Lemma A1 gives the following upper bound

$$\begin{array}{cc}\hfill \sum _{t=1}^T\mathbb{E}[\Delta \left(P\left(c_t\right|{\hat{c}}_t,{\gamma }^{*}),P\left(c_t\right|{\hat{c}}_t,{\mathcal{H}}_{c,\hat{c}})\right)\mathbf{1}\left\{\mathcal{E}\right\}]& \le \sum _{t=1}^T\mathbb{E}\left[\sqrt{2U^2{D}_{\mathrm{KL}}(P\left(c_t\right|{\hat{c}}_t,{\gamma }^{*})\parallel P\left(c_t\right|{\hat{c}}_t,{\mathcal{H}}_{c,\hat{c}})}\right]\hfill \\ \hfill & \le \sqrt{2T{U}^2\sum _{t=1}^T\mathbb{E}\left[D_{\mathrm{KL}}(P\left(c_t\right|{\hat{c}}_t,{\gamma }^{*})\parallel P\left(c_t\right|{\hat{c}}_t,{\mathcal{H}}_{c,\hat{c}})\right]}\hfill \\ \hfill & \stackrel{\left(a\right)}{=}\sqrt{2T{U}^2\sum _{t=1}^{T}I(c_t;{\gamma }^{*}|{\hat{c}}_t,{\mathcal{H}}_{c,\hat{c}})}\hfill \end{array}$$

$$\begin{array}{cc}\hfill & \stackrel{\left(b\right)}{\le }\sqrt{2T{U}^2\sum _{t=1}^{T}I(c_t,{\hat{c}}_t;{\gamma }^{*}|{\mathcal{H}}_{c,\hat{c}})}\hfill \end{array}$$

(A43)

$$\begin{array}{cc}\hfill & \stackrel{\left(c\right)}{=}\sqrt{2T{U}^{2}I({\mathcal{H}}_{T,c,\hat{c}};{\gamma }^{*})}\hfill \end{array}$$

(A44)

where the equality in $\left(a\right)$ follows by the definition of condition mutual information

$$I(c_t;{\gamma }^{*}|{\hat{c}}_t,{\mathcal{H}}_{c,\hat{c}}):=\mathbb{E}\left[D_{\mathrm{KL}}\left(P(c_t,{\gamma }^{*}|{\hat{c}}_t,{\mathcal{H}}_{c,\hat{c}})\parallel P\left(c_t\right|{\hat{c}}_t,{\mathcal{H}}_{c,\hat{c}})P\left({\gamma }^{*}\right|{\hat{c}}_t,{\mathcal{H}}_{c,\hat{c}})\right)\right],$$

and inequality in $\left(b\right)$ follows since $I(c_t,{\hat{c}}_t;{\gamma }^{*}|{\mathcal{H}}_{c,\hat{c}})=I({\hat{c}}_t;{\gamma }^{*}|{\mathcal{H}}_{c,\hat{c}})+I(c_t;{\gamma }^{*}|{\hat{c}}_t,{\mathcal{H}}_{c,\hat{c}})\ge I(c_t;{\gamma }^{*}|{\hat{c}}_t,{\mathcal{H}}_{c,\hat{c}})$ due to the non-negativity of mutual information, and finally, the equality in $\left(c\right)$ follows from the chain rule of mutual information.

We now analyze the mutual information $I({\gamma }^{*};{\mathcal{H}}_{T,c,\hat{c}})$ which can be written as

$$\begin{array}{cc}\hfill I({\gamma }^{*};{\mathcal{H}}_{T,c,\hat{c}})& =H\left({\gamma }^{*}\right)-H\left({\gamma }^{*}\right|{\mathcal{H}}_{T,c,\hat{c}})\hfill \end{array}$$

$$\begin{array}{cc}\hfill & ={\displaystyle \frac{1}{2}}log\left({\left(2\pi e\right)}^d\det \left({\Sigma }_{\gamma }\right)\right)-{\displaystyle \frac{1}{2}}log\left({\left(2\pi e\right)}^d\det \left(W^{-1}\right)\right)\hfill \end{array}$$

(A45)

$$\begin{array}{cc}\hfill & ={\displaystyle \frac{1}{2}}log{\displaystyle \frac{\det \left({\Sigma }_{\gamma }\right)}{\det \left(W^{-1}\right)}}\hfill \end{array}$$

(A46)

where $W=(T-1){\Sigma }_n^{-1}+{\Sigma }_{\gamma }^{-1}={\Sigma }_{\gamma }^{-1}(\mathbb{I}+(T-1){\Sigma }_{\gamma }{\Sigma }_n^{-1})$. Using this, we can equivalently write

$$\begin{array}{cc}\hfill I({\gamma }^{*};{\mathcal{H}}_{T,c,\hat{c}})& ={\displaystyle \frac{1}{2}}log{\displaystyle \frac{1}{\det \left({(\mathbb{I}+(T-1){\Sigma }_{\gamma }{\Sigma }_n^{-1})}^{-1}\right)}}\hfill \end{array}$$

(A47)

$$\begin{array}{cc}\hfill & ={\displaystyle \frac{1}{2}}log\det (\mathbb{I}+(T-1){\Sigma }_{\gamma }{\Sigma }_n^{-1}).\hfill \end{array}$$

(A48)

Under the assumption that ${\Sigma }_{\gamma }{\Sigma }_n^{-1}\succ 0$, we have $\mathbb{I}+(T-1){\Sigma }_{\gamma }{\Sigma }_n^{-1}\succ 0$, whereby using the determinant-trace inequality we obtain,

$$\begin{array}{cc}\hfill I({\gamma }^{*};{\mathcal{H}}_{T,c,\hat{c}})& ={\displaystyle \frac{1}{2}}log\left(\det (\mathbb{I}+(T-1){\Sigma }_{\gamma }{\Sigma }_n^{-1})\right)\hfill \\ & \le {\displaystyle \frac{d}{2}}log\left(\mathrm{Tr}(\mathbb{I}+(T-1){\Sigma }_{\gamma }{\Sigma }_n^{-1})/d)\right)\hfill \\ \hfill & ={\displaystyle \frac{d}{2}}log\left(1+(T-1)\mathrm{Tr}\left({\Sigma }_{\gamma }{\Sigma }_n^{-1}\right)/d\right)\hfill \\ \hfill & \le {\displaystyle \frac{d}{2}}log\left(1+T\mathrm{Tr}\left({\Sigma }_{\gamma }{\Sigma }_n^{-1}\right)/d\right).\hfill \end{array}$$

Using this in (A44), we obtain that

$$\begin{array}{c}\hfill \sum _{t=1}^T\mathbb{E}[\Delta \left(P\left(c_t\right|{\hat{c}}_t,{\gamma }^{*}),P\left(c_t\right|{\hat{c}}_t,{\mathcal{H}}_{c,\hat{c}})\right)\mathbb{I}\left\{\mathcal{E}\right\}]\le \sqrt{Tdlog\left(1+T\mathrm{Tr}\left({\Sigma }_{\gamma }{\Sigma }_n^{-1}\right)/d\right)U^2}.\end{array}$$

(A49)

Finally, using this in (A39), gives the following upper bound

$$\begin{array}{c}\hfill {\mathcal{R}}_{\mathrm{d},\mathrm{EE}1}^T\le \sqrt{2\lambda mTdlog\left(1+T\mathrm{Tr}\left({\Sigma }_{\gamma }{\Sigma }_n^{-1}\right)/d\right)log\left({\displaystyle \frac{2m}{\delta }}\right)}+2T{\delta }^2\sqrt{{\displaystyle \frac{m\lambda }{2\pi }}}.\end{array}$$

(A50)

We finally note that same upper bound holds for the term ${\mathcal{R}}_{\mathrm{d},\mathrm{EE}2}^T$.

Appendix C.1. Derivation of Posterior Predictive Distribution

In this section, we derive the posterior predictive distribution $P\left(c_t\right|{\hat{c}}_t,{\mathcal{H}}_{t-1,\hat{c}})$ for Gaussian bandits with Gaussian context noise. To this end, we first derive the posterior $P\left({\gamma }^{*}\right|{\mathcal{H}}_{t-1,\hat{c}})$.

Appendix C.1.1. Derivation of Posterior $P\left({\gamma }^{*}\right|{\mathcal{H}}_{t-1,\hat{c}})$

Using Baye’s theorem, we have

$$\begin{array}{c}\hfill P\left({\gamma }^{*}\right|{\mathcal{H}}_{t-1,\hat{c}})\propto P\left({\gamma }^{*}\right)\prod _{\tau =1}^{t-1}P\left({\hat{c}}_{\tau }\right|{\gamma }^{*}),\end{array}$$

where $P\left({\hat{c}}_{\tau }\right|{\gamma }^{*})$ is derived in (A18). Subsequently, we have that

$$\begin{array}{cc}\hfill logp\left({\gamma }^{*}\right|{\mathcal{H}}_{t-1},\hat{c})& \propto -{\displaystyle \frac{1}{2}}\sum _{\tau =1}^{t-1}\left({({\hat{c}}_{\tau }-F)}^{\top }G({\hat{c}}_{\tau }-F)\right)-{\displaystyle \frac{1}{2}}\left({\gamma }^{*\top }{\Sigma }_{\gamma }^{-1}{\gamma }^{*}\right)\hfill \\ \hfill & \propto -{\displaystyle \frac{1}{2}}({\gamma }^{*\top }((t-1)G+{\Sigma }_{\gamma }^{-1}){\gamma }^{*}-{\gamma }^{*\top }(G{\hat{c}}_{1:t-1}-(t-1){\Sigma }_n^{-1}{\left(M^{-1}\right)}^{\top }{\Sigma }_c^{-1}{\mu }_c)\hfill \\ & -\left({\hat{c}}_{1:t-1}^{\top }G-(t-1){\mu }_c^{\top }{\Sigma }_c^{-1}{M}^{-1}{\Sigma }_n^{-1}\right){\gamma }^{*}),\hfill \end{array}$$

where we have denoted ${\sum }_{\tau =1}^{t-1}{\hat{c}}_{\tau }={\hat{c}}_{1:t-1}$. We then obtain

$$\begin{array}{cc}\hfill P\left({\gamma }^{*}\right|{\mathcal{H}}_{t-1,\hat{c}})& =\mathcal{N}({\tilde{M}}_t,N_t^{-1})\mathrm{where},\hfill \end{array}$$

(A51)

$$\begin{array}{cc}\hfill N_t& =(t-1)G+{\Sigma }_{\gamma }^{-1}\hfill \end{array}$$

(A52)

$$\begin{array}{cc}\hfill {\tilde{M}}_t& ={\left(N_t^{-1}\right)}^{\top }\left(G{\hat{c}}_{1:t-1}-(t-1){\Sigma }_n^{-1}{\left(M^{-1}\right)}^{\top }{\Sigma }_c^{-1}{\mu }_c\right).\hfill \end{array}$$

(A53)

Appendix C.1.2. Derivation of $P\left(c_t\right|{\hat{c}}_t,{\mathcal{H}}_{t-1,\hat{c}})$

The derivation of posterior predictive distribution $P\left(c_t\right|{\hat{c}}_t,{\mathcal{H}}_{t-1,\hat{c}})$ follows in a similar line as that in Appendix B.2.4. We start the derivation by noting that $P\left(c_t\right|{\hat{c}}_t,{\mathcal{H}}_{t-1,\hat{c}})={\mathbb{E}}_{P\left({\gamma }^{*}\right|{\mathcal{H}}_{t-1,\hat{c}})}\left[P\left(c_t\right|{\hat{c}}_t,{\gamma }^{*})\right]$.

We have

$$\begin{array}{cc}\hfill & log\left(P\left({\gamma }^{*}\right|{\mathcal{H}}_{t-1,\hat{c}})P\left(c_t\right|{\hat{c}}_t,{\gamma }^{*})\right)\hfill \\ & \propto -{\displaystyle \frac{1}{2}}\left({(c_t-A_t)}^{\top }M(c_t-A_t)+{({\gamma }^{*}-{\tilde{M}}_t)}^{\top }N({\gamma }^{*}-{\tilde{M}}_t)\right)\hfill \\ \hfill & =-{\displaystyle \frac{1}{2}}({\left(M^{-1}{)}^{\top }{\Sigma }_n^{-1}{\gamma }^{*}+c_t-D-E_t\right)}^{\top }M\left({\left(M^{-1}\right)}^{\top }{\Sigma }_n^{-1}{\gamma }^{*}+c_t-D-E_t\right)\hfill \\ & +{({\gamma }^{*}-{\tilde{M}}_t)}^{\top }{N}_t({\gamma }^{*}-{\tilde{M}}_t))\hfill \\ \hfill & =-{\displaystyle \frac{1}{2}}\left({({\gamma }^{*}-J_t)}^{\top }{H}_t({\gamma }^{*}-J_t)-J_t^{\top }{H}_t{J}_t+{(c_t-D-E_t)}^{\top }M(c_t-D-E_t)+{\tilde{M}}_t^{\top }{N}_t{\tilde{M}}_t\right),\hfill \end{array}$$

(A54)

where $A_t$ is defined in (A14), M is defined in (A13), ${\tilde{M}}_t$ in (A53), N in (A52), $D={\left(M^{-1}\right)}^{\top }{\Sigma }_c^{-1}{\mu }_c$, $E_t={\left(M^{-1}\right)}^{\top }{\Sigma }_n^{-1}{\hat{c}}_t$, $H_t={\Sigma }_n^{-1}{\left(M^{-1}\right)}^{\top }{\Sigma }_n^{-1}+N_t$ and $J_t={\left(H_t^{-1}\right)}^{\top }\left({\Sigma }_n^{-1}(-c_t+D+E_t)+N_t{\tilde{M}}_t\right)$.

Subsequently, we have

$$\begin{array}{cc}\hfill logp\left(c_t\right|{\hat{c}}_t,{\mathcal{H}}_{t-1,\hat{c}})& \propto -{\displaystyle \frac{1}{2}}\left(-J_t^{\top }{H}_t{J}_t+{(c_t-D-E_t)}^{\top }M(c_t-D-E_t)\right)\hfill \\ \hfill & \propto -{\displaystyle \frac{1}{2}}(c_t^{\top }\left(M-{\Sigma }_n^{-1}{\left(H_t^{-1}\right)}^{\top }{\Sigma }_n^{-1}\right)c_t-c_t^{\top }\left(M(D+E_t)-{\Sigma }_n^{-1}{\left(H_t^{-1}\right)}^{\top }{L}_t^{\top }\right)\hfill \end{array}$$

(A55)

$$\begin{array}{cc}& -\left({(D+E_t)}^{\top }M-L_t\left(H_t^{-1}\right){\Sigma }_n^{-1}\right),\hfill \end{array}$$

(A56)

where $L_t={(D+E_t)}^{\top }{\Sigma }_n^{-1}+{\tilde{M}}_t^{\top }{N}_t$. Thus, we have,

$$\begin{array}{cc}\hfill P\left(c_t\right|{\hat{c}}_t,{\mathcal{H}}_{t-1,\hat{c}})& =\mathcal{N}\left(c_t\right|V_t,R_t^{-1}),\mathrm{where}\hfill \end{array}$$

(A57)

$$\begin{array}{cc}\hfill & R_t=M-{\Sigma }_n^{-1}{\left(H_t^{-1}\right)}^{\top }{\Sigma }_n^{-1}\hfill \end{array}$$

(A58)

$$\begin{array}{cc}\hfill & V_t={\left(R_t^{-1}\right)}^{\top }\left(M(D+E_t)-{\Sigma }_n^{-1}{\left(H_t^{-1}\right)}^{\top }{L}_t^{\top }\right).\hfill \end{array}$$

(A59)

Appendix C.1.3. Evaluating the KL Divergence between the True Posterior $P_t\left({\theta }^{*}\right)$ and Sampling Distribution ${\overline{P}}_t\left({\theta }^{*}\right)$

In this subsection, we analyze the true posterior distribution $P_t\left({\theta }^{*}\right):=P\left({\theta }^{*}\right|{\mathcal{H}}_{t-1,r,a,\hat{c}})$ and the approximate sampling distribution ${\overline{P}}_t\left({\theta }^{*}\right):=\overline{P}\left({\theta }^{*}\right|{\mathcal{H}}_{t-1,r,a,\hat{c}})$, and derive the KL divergence $D_{\mathrm{KL}}(P_t\left({\theta }^{*}\right)\parallel {\overline{P}}_t\left({\theta }^{*}\right))$ between them. To see this, note that from Bayes’s theorem, we have the following joint probability distribution

$$\begin{array}{cc}\hfill P({\theta }^{*},{\mathcal{H}}_{t-1,r,\hat{c}}|{\mathcal{H}}_{t-1,a})& =P\left({\theta }^{*}\right)\underset{:=P\left({\mathcal{H}}_{t-1,r,\hat{c}}\right|{\mathcal{H}}_{t-1,a},{\theta }^{*})}{\underbrace{{\mathbb{E}}_{P\left({\gamma }^{*}\right)}\left[\prod _{\tau =1}^{t-1}{\mathbb{E}}_{P\left(c_{\tau }\right)}\left[P\left({\hat{c}}_{\tau }\right|c_{\tau },{\gamma }^{*})P\left(r_{\tau }\right|a_{\tau },c_{\tau },{\theta }^{*})\right]\right]}}\hfill \end{array}$$

(A60)

$$\begin{array}{cc}\hfill & =P\left({\theta }^{*}\right)P\left({\mathcal{H}}_{t-1,r,\hat{c}}\right|{\mathcal{H}}_{t-1,a},{\theta }^{*})\hfill \end{array}$$

(A61)

whereby we obtain that

$$\begin{array}{c}\hfill P_t\left({\theta }^{*}\right)\propto P\left({\theta }^{*}\right)P\left({\mathcal{H}}_{t-1,r,\hat{c}}\right|{\mathcal{H}}_{t-1,a},{\theta }^{*}).\end{array}$$

(A62)

In particular, for general feature maps $\varphi (a,c)$, the distribution $P\left({\mathcal{H}}_{t-1,r,\hat{c}}\right|{\mathcal{H}}_{t-1,a},{\theta }^{*})$ cannot be exactly evaluated, even under Gaussian assumptions, resulting in the posterior $P_t\left({\theta }^{*}\right)$ to be intractable, in general.

In contrast to this, our approximate TS-algorithm scheme assumes the following joint probability distribution,

$$\begin{array}{cc}\hfill \overline{P}({\theta }^{*},{\mathcal{H}}_{t-1,r,\hat{c}}|{\mathcal{H}}_{t-1,a})& =P\left({\theta }^{*}\right)\underset{:=\overline{P}\left({\mathcal{H}}_{t-1,r,\hat{c}}\right|{\mathcal{H}}_{t-1,a},{\theta }^{*})}{\underbrace{{\mathbb{E}}_{P\left({\gamma }^{*}\right)}\left[\prod _{\tau =1}^{t-1}{\mathbb{E}}_{P\left(c_{\tau }\right)}\left[P\left({\hat{c}}_{\tau }\right|c_{\tau },{\gamma }^{*})\overline{P}\left(r_{\tau }\right|a_{\tau },{\hat{c}}_{\tau },{\mathcal{H}}_{\tau -1,\hat{c}},{\theta }^{*})\right]\right]}}\hfill \end{array}$$

(A63)

$$\begin{array}{cc}\hfill & =P\left({\theta }^{*}\right)\overline{P}\left({\mathcal{H}}_{t-1,r,\hat{c}}\right|{\mathcal{H}}_{t-1,a},{\theta }^{*}),\hfill \end{array}$$

(A64)

where

$$\overline{P}\left(r_{\tau }\right|a_{\tau },{\hat{c}}_{\tau },{\mathcal{H}}_{\tau -1,\hat{c}},{\theta }^{*})=\mathcal{N}(\psi (a_{\tau },{\hat{c}}_{\tau }|{\mathcal{H}}_{\tau -1,\hat{c}}),{\sigma }^2).$$

Consequently, we have

$$\begin{array}{cc}\hfill {\overline{P}}_t\left({\theta }^{*}\right)& \propto P\left({\theta }^{*}\right)\overline{P}\left({\mathcal{H}}_{t-1,r,\hat{c}}\right|{\mathcal{H}}_{t-1,a},{\theta }^{*})\hfill \\ \hfill & =P\left({\theta }^{*}\right)\left(\prod _{\tau =1}^{t-1}\overline{P}\left(r_{\tau }\right|a_{\tau },{\hat{c}}_{\tau },{\mathcal{H}}_{\tau -1,\hat{c}},{\theta }^{*})\right)\left({\mathbb{E}}_{P\left({\gamma }^{*}\right)}\left[\prod _{\tau =1}^{t-1}[P\left({\hat{c}}_{\tau }\right|{\gamma }^{*})\right]\right).\hfill \end{array}$$

(A65)

As a result, we can upper bound the KL divergence $D_{\mathrm{KL}}(P_t\left({\theta }^{*}\right)\parallel {\overline{P}}_t\left({\theta }^{*}\right))$ as

$$\begin{array}{c}{D}_{\mathrm{KL}}(P_t\left({\theta }^{*}\right)\parallel {\overline{P}}_t\left({\theta }^{*}\right))\hfill \\ \le D_{\mathrm{KL}}\left(P({\theta }^{*},{\mathcal{H}}_{t-1,r,\hat{c}}|{\mathcal{H}}_{t-1,a})\parallel \overline{P}({\theta }^{*},{\mathcal{H}}_{t-1,r,\hat{c}}|{\mathcal{H}}_{t-1,a})\right)\hfill \\ =D_{\mathrm{KL}}\left(P\left({\theta }^{*}\right)P\left({\mathcal{H}}_{t-1,r,\hat{c}}\right|{\theta }^{*},{\mathcal{H}}_{t-1,a})\parallel P\left({\theta }^{*}\right)\overline{P}\left({\mathcal{H}}_{t-1,r,\hat{c}}\right|{\mathcal{H}}_{t-1,a},{\theta }^{*})\right)\hfill \\ ={\mathbb{E}}_{P\left({\theta }^{*}\right)}\left[D_{\mathrm{KL}}\left(P\left({\mathcal{H}}_{t-1,r,\hat{c}}\right|{\theta }^{*},{\mathcal{H}}_{t-1,a})\parallel \overline{P}\left({\mathcal{H}}_{t-1,r,\hat{c}}\right|{\mathcal{H}}_{t-1,a},{\theta }^{*})\right)\right]\hfill \\ \stackrel{\left(a\right)}{\le }{\mathbb{E}}_{P\left({\theta }^{*}\right)P\left({\gamma }^{*}\right)}{\mathbb{E}}_{P\left({\mathcal{H}}_{t-1,c}\right)}\left[D_{\mathrm{KL}}\left(\prod _{\tau =1}^{t-1}P\left({\hat{c}}_{\tau }\right|c_{\tau },{\gamma }^{*})P\left(r_{\tau }\right|a_{\tau },c_{\tau },{\theta }^{*})\parallel \prod _{\tau =1}^{t-1}P\left({\hat{c}}_{\tau }\right|c_{\tau },{\gamma }^{*})\overline{P}\left(r_{\tau }\right|a_{\tau },{\mathcal{H}}_{\tau ,\hat{c}},{\theta }^{*})\right)\right]\hfill \\ ={\mathbb{E}}_{P\left({\theta }^{*}\right)}{\mathbb{E}}_{P\left({\gamma }^{*}\right)}{\mathbb{E}}_{P\left({\mathcal{H}}_{t-1,c}\right)}\left[\sum _{\tau =1}^{t-1}{\mathbb{E}}_{P\left({\mathcal{H}}_{\tau -1,\hat{c}}\right|{\mathcal{H}}_{\tau -1,c},{\gamma }^{*})}\left[D_{\mathrm{KL}}\left(P\left(r_{\tau }\right|a_{\tau },c_{\tau },{\theta }^{*})\parallel \overline{P}\left(r_{\tau }\right|a_{\tau },{\hat{c}}_{\tau },{\mathcal{H}}_{\tau -1,\hat{c}},{\theta }^{*})\right)\right]\right]\hfill \end{array}$$

$$\begin{array}{cc}\hfill & \stackrel{\left(b\right)}{=}{\mathbb{E}}_{P\left({\theta }^{*}\right)}{\mathbb{E}}_{P\left({\gamma }^{*}\right)}{\mathbb{E}}_{P\left({\mathcal{H}}_{t-1,c}\right)}\left[\sum _{\tau =1}^{t-1}{\mathbb{E}}_{P\left({\mathcal{H}}_{\tau -1,\hat{c}}\right|{\mathcal{H}}_{\tau -1,c},{\gamma }^{*})}\left[{\displaystyle \frac{{(\varphi {(a_{\tau },c_{\tau })}^{\top }{\theta }^{*}-\psi {({\hat{c}}_{\tau },a_{\tau }|{\mathcal{H}}_{\tau -1,\hat{c}})}^{\top }{\theta }^{*})}^2}{2{\sigma }^2}}\right]\right],\hfill \\ \hfill & ={\mathbb{E}}_{P\left({\theta }^{*}\right)}{\mathbb{E}}_{P\left({\gamma }^{*}\right)}{\mathbb{E}}_{P\left({\mathcal{H}}_{t-1,c}\right)}\left[\sum _{\tau =1}^{t-1}{\mathbb{E}}_{P\left({\mathcal{H}}_{\tau -1,\hat{c}}\right|{\mathcal{H}}_{\tau -1,c},{\gamma }^{*})}\left[{\displaystyle \frac{|{(\varphi (a_{\tau },c_{\tau })-\psi ({\hat{c}}_{\tau },a_{\tau }|{\mathcal{H}}_{\tau -1,\hat{c}}))}^{\top }{\theta }^{*}{|}^2}{2{\sigma }^2}}\right]\right],\hfill \\ \hfill & \stackrel{\left(c\right)}{\le }{\mathbb{E}}_{P\left({\theta }^{*}\right)}{\mathbb{E}}_{P\left({\gamma }^{*}\right)}{\mathbb{E}}_{P\left({\mathcal{H}}_{t-1,c}\right)}\left[\sum _{\tau =1}^{t-1}{\mathbb{E}}_{P\left({\mathcal{H}}_{\tau -1,\hat{c}}\right|{\mathcal{H}}_{\tau -1,c},{\gamma }^{*})}\left[{\displaystyle \frac{\parallel (\varphi (a_{\tau },c_{\tau })-\psi ({\hat{c}}_{\tau },a_{\tau }|{\mathcal{H}}_{\tau -1,\hat{c}}){\parallel }_2^2\parallel {\theta }^{*}{\parallel }_2^2}{2{\sigma }^2}}\right]\right]\hfill \\ \hfill & \stackrel{\left(d\right)}{\le }{\mathbb{E}}_{P\left({\theta }^{*}\right)}{\mathbb{E}}_{P\left({\gamma }^{*}\right)}{\mathbb{E}}_{P\left({\mathcal{H}}_{t-1,c}\right)}\left[\sum _{\tau =1}^{t-1}{\mathbb{E}}_{P\left({\mathcal{H}}_{\tau -1,\hat{c}}\right|{\mathcal{H}}_{\tau -1,c},{\gamma }^{*})}\left[{\displaystyle \frac{4\parallel {\theta }^{*}{\parallel }_2^2}{2{\sigma }^2}}\right]\right]\hfill \\ \hfill & =\sum _{\tau =1}^{t-1}{\mathbb{E}}_{P\left({\theta }^{*}\right)}\left[{\displaystyle \frac{2\parallel {\theta }^{*}{\parallel }_2^2}{{\sigma }^2}}\right]\stackrel{\left(e\right)}{=}{\displaystyle \frac{2(t-1)\lambda m}{{\sigma }^2}}.\hfill \end{array}$$

(A66)

In the above series of relationships,

• equality in $\left(a\right)$ follows by noting that

  $$\overline{P}\left({\mathcal{H}}_{t-1,r,\hat{c}}\right|{\mathcal{H}}_{t-1,a},{\theta }^{*})={\mathbb{E}}_{P\left({\gamma }^{*}\right)}{\mathbb{E}}_{P\left({\mathcal{H}}_{t-1,c}\right)}\left[\prod _{\tau =1}^{t-1}P\left({\hat{c}}_{\tau }\right|c_{\tau },{\gamma }^{*})\overline{P}\left(r_{\tau }\right|a_{\tau },{\hat{c}}_{\tau },{\theta }^{*})\right]$$

  and

  $$P\left({\mathcal{H}}_{t-1,r,\hat{c}}\right|{\mathcal{H}}_{t-1,a},{\theta }^{*})={\mathbb{E}}_{P\left({\gamma }^{*}\right)}{\mathbb{E}}_{P\left({\mathcal{H}}_{t-1,c}\right)}\left[\prod _{\tau =1}^{t-1}P\left({\hat{c}}_{\tau }\right|c_{\tau },{\gamma }^{*})P\left(r_{\tau }\right|a_{\tau },c_{\tau },{\theta }^{*})\right],$$

  and applying Jensen’s inequality on the jointly convex KL divergence,
• equality in $\left(b\right)$ follows from evaluating the KL divergence between two Gaussian distributions with same variance ${\sigma }^2$ and with means $\varphi {(a_{\tau },c_{\tau })}^{\top }{\theta }^{*}$ and $\psi {({\hat{c}}_{\tau },a_{\tau }|{\mathcal{H}}_{\tau -1,\hat{c}})}^{\top }{\theta }^{*}$ respectively,
• inequality in $\left(c\right)$ follows from application of Cauchy–Schwarz inequality,
• inequality in $\left(d\right)$ follows from Assumption 1,
• inequality in $\left(e\right)$ follows from

  $${\mathbb{E}}_{P\left({\theta }^{*}\right)}[\parallel {\theta }^{*}{\parallel }_2^2]=\sum _{j=1}^m\mathbb{E}\left[{\left({\theta }_j^{*}\right)}^2\right]=\lambda m$$

  since $P\left({\theta }^{*}\right)=\mathcal{N}(\mathbf{0},\lambda \mathbb{I})$.

Appendix C.2. Proof of Lemma 1

For notational simplicity, throughout this section we use $\psi \left(a\right):=\psi (a,{\hat{c}}_t|{\mathcal{H}}_{\hat{c}})={\mathbb{E}}_{P\left(c_t\right|{\hat{c}}_t,{\mathcal{H}}_{t-1,\hat{c}})}\left[\varphi (a,c_t)\right]$ to denote the expected feature map. Furthermore, we use ${\mathcal{F}}_t={\mathcal{H}}_{t-1,r,a,\hat{c}}\cup {\hat{c}}_t$.

We start by distinguishing the true and approximated posterior distributions. Recall that $P_t\left({\theta }^{*}\right):=P\left({\theta }^{*}\right|{\mathcal{H}}_{t-1,r,a,\hat{c}})$ denotes the true posterior and ${\overline{P}}_t\left({\theta }^{*}\right):=\overline{P}\left({\theta }^{*}\right|{\mathcal{H}}_{t-1,r,a,\hat{c}})$ denotes the approximated posterior. We then denote $P_t({\hat{a}}_t,{\theta }^{*}):=P({\hat{a}}_t,{\theta }^{*}|{\mathcal{F}}_t)$ as the distribution of ${\hat{a}}_t$ and ${\theta }^{*}$ conditioned on ${\mathcal{F}}_t$, while ${\overline{P}}_t({\hat{a}}_t,{\theta }^{*}):=\overline{P}({\hat{a}}_t,{\theta }^{*}|{\mathcal{F}}_t)$ denote the distribution of ${\hat{a}}_t$ and ${\theta }^{*}$ under the sampling distribution. Furthermore, we have that ${\overline{P}}_t(a_t,{\theta }^{*})={\overline{P}}_t\left(a_t\right){\overline{P}}_t\left({\theta }^{*}\right)=P_t\left(a_t\right){\overline{P}}_t\left({\theta }^{*}\right)$. We start by decomposing ${\mathcal{R}}_{\mathrm{CB}}^T$ into the following three differences,

$$\begin{array}{cc}\hfill {\mathcal{R}}_{\mathrm{CB}}^T& =\sum _{t=1}^T\mathbb{E}\left[{\mathbb{E}}_{P_t({\hat{a}}_t,{\theta }^{*})}\left[\psi {\left({\hat{a}}_t\right)}^{\top }{\theta }^{*}\right]-{\mathbb{E}}_{P_t(a_t,{\theta }^{*})}\left[\psi {\left(a_t\right)}^{\top }{\theta }^{*}\right]\right]\hfill \\ \hfill & =\sum _{t=1}^T\mathbb{E}[\underset{:={\mathrm{Term}}_1}{\underbrace{{\mathbb{E}}_{{\overline{P}}_t({\hat{a}}_t,{\theta }^{*})}\left[\psi {\left({\hat{a}}_t\right)}^{\top }{\theta }^{*}\right]-{\mathbb{E}}_{{\overline{P}}_t(a_t,{\theta }^{*})}\left[\psi {\left(a_t\right)}^{\top }{\theta }^{*}\right]}}+\underset{:={\mathrm{Term}}_2}{\underbrace{{\mathbb{E}}_{P_t({\hat{a}}_t,{\theta }^{*})}\left[\psi {\left({\hat{a}}_t\right)}^{\top }{\theta }^{*}\right]-{\mathbb{E}}_{{\overline{P}}_t({\hat{a}}_t,{\theta }^{*})}\left[\psi {\left({\hat{a}}_t\right)}^{\top }{\theta }^{*}\right]]}}\hfill \\ & +\underset{:={\mathrm{Term}}_3}{\underbrace{{\mathbb{E}}_{{\overline{P}}_t(a_t,{\theta }^{*})}\left[\psi {\left(a_t\right)}^{\top }{\theta }^{*}\right]-{\mathbb{E}}_{P_t(a_t,{\theta }^{*})}\left[\psi {\left(a_t\right)}^{\top }{\theta }^{*}\right]]}}\hfill \end{array}$$

(A67)

We will separately upper bound each of the three terms in the above decomposition.

Appendix C.2.1. Upper Bound on ${\mathrm{Term}}_2$

To obtain an upper bound on ${\mathrm{Term}}_2$, note that the following equivalence holds ${\mathbb{E}}_{P_t({\hat{a}}_t,{\theta }^{*})}\left[\psi {\left({\hat{a}}_t\right)}^{\top }{\theta }^{*}\right]={\mathbb{E}}_{P_t\left({\theta }^{*}\right)}\left[{max}_a\psi {\left(a\right)}^{\top }{\theta }^{*}\right]$. Using this, we can rewrite ${\mathrm{Term}}_2$ as

$$\begin{array}{c}\hfill {\mathrm{Term}}_2={\mathbb{E}}_{P_t\left({\theta }^{*}\right)}\left[\underset{a}{max}\psi {\left(a\right)}^{\top }{\theta }^{*}\right]-{\mathbb{E}}_{{\overline{P}}_t\left({\theta }^{*}\right)}\left[\underset{a}{max}\psi {\left(a\right)}^{\top }{\theta }^{*}\right].\end{array}$$

(A68)

Note, here that when ${\theta }^{*}\sim {\overline{P}}_t\left({\theta }^{*}\right)$, for each $a\in \mathcal{A}$, we have that $z_a=\psi {\left(a\right)}^{\top }{\theta }^{*}$ follows Gaussian distribution $\mathcal{N}\left(z_a\right|\psi {\left(a\right)}^{\top }{\mu }_{t-1},\psi {\left(a\right)}^{\top }{\Sigma }_{t-1}^{-1}\psi \left(a\right))$ with mean $\psi {\left(a\right)}^{\top }{\mu }_{t-1}$ and variance $\psi {\left(a\right)}^{\top }{\Sigma }_{t-1}^{-1}\psi \left(a\right)$, where ${\mu }_{t-1}$ and ${\Sigma }_{t-1}$ are as defined in (18) and (17), respectively. Thus, ${\mathbb{E}}_{{\overline{P}}_t\left({\theta }^{*}\right)}\left[{max}_a{z}_a\right]$ is the average of maximum of Gaussian random variables. We can then apply Lemma A2 with $P\left(x\right)=P_t\left({\theta }^{*}\right)$, $Q\left(x\right)={\overline{P}}_t\left({\theta }^{*}\right)$, $n=\left|\mathcal{A}\right|=K$, ${\mu }_i=\psi {\left(a\right)}^{\top }{\mu }_{t-1}$ and ${\sigma }_i=\psi {\left(a\right)}^{\top }{\Sigma }_{t-1}^{-1}\psi \left(a\right)$ to obtain that

$$\begin{array}{cc}\hfill {\mathrm{Term}}_2& \le \sqrt{2(logK+D_{\mathrm{KL}}(P_t\left({\theta }^{*}\right)\parallel {\overline{P}}_t\left({\theta }^{*}\right)){max}_a\psi {\left(a\right)}^{\top }{\Sigma }_t^{-1}\psi \left(a\right)}.\hfill \end{array}$$

(A69)

Using this, we obtain that

$$\begin{array}{cc}\hfill \sum _{t=1}^T\mathbb{E}\left[{\mathrm{Term}}_2\right]& \le \mathbb{E}\left[\sum _{t=1}^T\sqrt{2(logK+D_{\mathrm{KL}}(P_t\left({\theta }^{*}\right)\parallel {\overline{P}}_t\left({\theta }^{*}\right))){max}_a\psi {\left(a\right)}^{\top }{\Sigma }_t^{-1}\psi \left(a\right)}\right]\hfill \\ \hfill & \stackrel{\left(a\right)}{\le }\sqrt{\left(\sum _{t=1}^T\mathbb{E}\left[2(logK+D_{\mathrm{KL}}(P_t\left({\theta }^{*}\right)\parallel {\overline{P}}_t\left({\theta }^{*}\right)))\right]\right)\left(\sum _{t=1}^T\mathbb{E}\left[{max}_a\psi {\left(a\right)}^{\top }{\Sigma }_t^{-1}\psi \left(a\right)\right]\right)},\hfill \\ \hfill & \stackrel{\left(b\right)}{\le }\sqrt{2\lambda T\left(\sum _{t=1}^T\mathbb{E}\left[logK+D_{\mathrm{KL}}(P_t\left({\theta }^{*}\right)\parallel {\overline{P}}_t\left({\theta }^{*}\right))\right]\right)}\hfill \\ \hfill & \stackrel{\left(c\right)}{\le }\sqrt{2\lambda T\left(TlogK+\sum _{t=1}^{T}2(t-1){\displaystyle \frac{\lambda m}{{\sigma }^2}}\right)}\hfill \\ \hfill & =\sqrt{2\lambda T^{2}logK+{\displaystyle \frac{4{\lambda }^{2}T(T^2-T)m}{2{\sigma }^2}}}\hfill \\ \hfill & \le \sqrt{2\lambda T^{2}logK+{\displaystyle \frac{2{\lambda }^2{T}^{3}m}{{\sigma }^2}}},\hfill \end{array}$$

(A70)

where the inequality in $\left(a\right)$ follows from Cauchy–Schwarz inequality, and the inequality in $\left(b\right)$ follows since

$$\begin{array}{c}\hfill \sum _{t=1}^T\mathbb{E}\left[\underset{a}{max}\psi {\left(a\right)}^{\top }{\Sigma }_t^{-1}\psi \left(a\right)\right]\le \sum _{t=1}^T\mathbb{E}\left[\underset{a}{max}\psi {\left(a\right)}^{\top }\left(\lambda \mathbb{I}\right)\psi \left(a\right)\right]\le \lambda T,\end{array}$$

which follows since ${\Sigma }_t^{-1}\le \lambda \mathbb{I}$ and $\parallel \psi \left(a\right)\parallel \le 1$. The inequality in $\left(c\right)$ follows from (A66).

If $\lambda \le {\displaystyle \frac{{\sigma }^2}{T}}$, we obtain that

$$\begin{array}{c}\hfill \sum _{t=1}^T\mathbb{E}\left[{\mathrm{Term}}_2\right]\le \sqrt{2T{\sigma }^2(log\left(K\right)+m)}.\end{array}$$

(A71)

Appendix C.2.2. Upper Bound on ${\mathrm{Term}}_3$

We can bound ${\mathrm{Term}}_3$ by observing that

$$\begin{array}{cc}\hfill {\mathrm{Term}}_3& ={\mathbb{E}}_{P_t\left(a_t\right)}{\left[\psi \left(a_t\right)\right]}^{\top }\left({\mathbb{E}}_{{\overline{P}}_t\left({\theta }^{*}\right)}\left[{\theta }^{*}\right]-{\mathbb{E}}_{P_t\left({\theta }^{*}\right)}\left[{\theta }^{*}\right]\right)\hfill \end{array}$$

(A72)

$$\begin{array}{cc}\hfill & ={\mathbb{E}}_{{\overline{P}}_t\left({\theta }^{*}\right)}\left[{\mathsf{\Psi }}_t^{\top }{\theta }^{*}\right]-{\mathbb{E}}_{P_t\left({\theta }^{*}\right)}\left[{\mathsf{\Psi }}_t^{\top }{\theta }^{*}\right]\hfill \end{array}$$

(A73)

where we used ${\mathsf{\Psi }}_t={\mathbb{E}}_{P_t\left(a_t\right)}\left[\psi \left(a_t\right)\right]$. Note, that for ${\theta }^{*}\sim {\overline{P}}_t\left({\theta }^{*}\right)$, the random variable ${\mathsf{\Psi }}_t^{\top }{\theta }^{*}$ is Gaussian with mean ${\mathsf{\Psi }}_t^{\top }{\mu }_{t-1}$ and variance ${\mathsf{\Psi }}_t^{\top }{\Sigma }_{t-1}^{-1}{\mathsf{\Psi }}_t$. Consequently, ${\mathsf{\Psi }}_t^{\top }{\theta }^{*}$ is also ${\mathsf{\Psi }}_t^{\top }{\Sigma }_{t-1}^{-1}{\mathsf{\Psi }}_t$-sub-Gaussian according to Definition A.1. By using Lemma A1, we then obtain that

$$\begin{array}{c}\hfill |{\mathrm{Term}}_3|\le \sqrt{2\left({\mathsf{\Psi }}_t^{\top }{\Sigma }_t^{-1}{\mathsf{\Psi }}_t\right)D_{\mathrm{KL}}(P_t\left({\theta }^{*}\right)\parallel {\overline{P}}_t\left({\theta }^{*}\right))}.\end{array}$$

(A74)

Using Cauchy–Schwarz inequality then yields that

$$\begin{array}{cc}\hfill \mathbb{E}\left[\sum _{t=1}^T\right|{\mathrm{Term}}_3\left|\right]& \le \sqrt{\left(\sum _{t=1}^T\mathbb{E}\left[{\mathsf{\Psi }}_t^{\top }\underset{t}{\overset{-1}{\Sigma }}{\mathsf{\Psi }}_t\right]\right)\left(\sum _{t=1}^T\mathbb{E}\left[2D_{\mathrm{KL}}(P_t\left({\theta }^{*}\right)\parallel {\overline{P}}_t\left({\theta }^{*}\right))\right]\right)}\hfill \\ \hfill & \le \sqrt{2\lambda T{\displaystyle \frac{\lambda T^{2}m}{{\sigma }^2}}}=\sqrt{2{\lambda }^2{T}^3{\displaystyle \frac{m}{{\sigma }^2}}}.\hfill \end{array}$$

where the second inequality follows from (A66). As before, if $\lambda \le {\displaystyle \frac{{\sigma }^2}{T}}$, we then obtain that

$$\begin{array}{cc}\hfill \mathbb{E}[\sum _{t=1}^T|{\mathrm{Term}}_3|]& \le \sqrt{2Tm{\sigma }^2}.\hfill \end{array}$$

(A75)

Appendix C.2.3. Upper Bound on ${\mathrm{Term}}_1$

Note, that in ${\mathrm{Term}}_1$, ${\overline{P}}_t\left({\hat{a}}_t\right)={\overline{P}}_t\left(a_t\right)=P_t\left(a_t\right)$, whereby the posterior is matched. Hence, one can apply bounds from conventional contextual Thompson Sampling here. For simplicity, we denote ${\overline{\mathbb{E}}}_t[·]={\mathbb{E}}_{\overline{P}}[·|{\mathcal{F}}_t]$ to denote the expectation with respect to ${\overline{P}}_t(a,\theta )$. To this end, as in the proof of Lemma 3, we start by defining an information ratio,

$$\begin{array}{c}\hfill {\mathsf{\Gamma }}_t={\displaystyle \frac{{{\mathrm{Term}}_1}^2}{{\overline{E}}_t\left[{\left(\psi {\left(a_t\right)}^{\top }{\theta }^{*}-\psi {\left(a_t\right)}^{\top }{\mu }_t)\right)}^2\right]:={\mathsf{\Lambda }}_t}},\end{array}$$

(A76)

using which we obtain the upper bound on ${\mathrm{Term}}_1$ as

$$\begin{array}{c}\hfill \sum _{t=1}^T\mathbb{E}\left[{\mathrm{Term}}_1\right]\le \mathbb{E}\left[\sum _{t=1}^T\sqrt{{\mathsf{\Gamma }}_t{\mathsf{\Lambda }}_t}\right]\le \sqrt{\left(\sum _{t=1}^T\mathbb{E}\left[{\mathsf{\Gamma }}_t\right]\right)\left(\sum _{t=1}^T\mathbb{E}\left[{\mathsf{\Lambda }}_t\right]\right)}\end{array}$$

(A77)

by the Cauchy–Schwarz inequality.

Furthermore, we have

$$\begin{array}{cc}\hfill {\mathsf{\Lambda }}_t={\overline{\mathbb{E}}}_t\left[{\left(\psi {\left(a_t\right)}^{\top }({\theta }^{*}-{\mu }_t)\right)}^2\right]& ={\overline{\mathbb{E}}}_t\left[\psi {\left(a_t\right)}^{\top }({\theta }^{*}-{\mu }_{t-1}){({\theta }^{*}-{\mu }_{t-1})}^{\top }\psi \left(a_t\right)\right]\hfill \\ \hfill & ={\overline{\mathbb{E}}}_t\left[\psi {\left(a_t\right)}^{\top }{\Sigma }_{t-1}^{-1}\psi \left(a_t\right)\right]={\overline{E}}_t[\parallel \psi \left(a_t\right){\parallel }_{{\Sigma }_{t-1}^{-1}}],\hfill \end{array}$$

(A78)

where ${\mu }_{t-1}$ and ${\Sigma }_{t-1}$ are defined as in (18) andd (17). Subsequently, using elliptical potential lemma, we obtain

$$\begin{array}{c}\hfill \sum _{t=1}^T{\mathsf{\Lambda }}_t\le 2m{\sigma }^{2}log\left(1+{\displaystyle \frac{\left(T\right)\lambda }{m{\sigma }^2}}\right).\end{array}$$

(A79)

To obtain an upper bound on the information ratio ${\mathsf{\Gamma }}_t$, we define $\overline{f}({\theta }^{*},a)=\psi {\left(a\right)}^{\top }{\theta }^{*}$ and $\overline{f}\left(a\right)=\psi {\left(a\right)}^{\top }{\mu }_{t-1}$ and let

$$\begin{array}{c}\hfill M_{a,a^{\prime }}=\sum _{a,a^{\prime }}\sqrt{{\overline{P}}_t(a_t=a){\overline{P}}_t({\hat{a}}_t=a^{\prime })}({\overline{\mathbb{E}}}_t\left[\overline{f}({\theta }^{*},a)\right|{\hat{a}}_t=a^{\prime }]-\overline{f}\left(a\right)).\end{array}$$

(A80)

It is easy to see that

$$\begin{array}{cc}\hfill {\mathrm{Term}}_1& ={\overline{\mathbb{E}}}_t\left[\overline{f}({\hat{a}}_t,{\theta }^{*})\right]-{\overline{\mathbb{E}}}_t\left[\overline{f}\left(a_t\right)\right]\hfill \\ \hfill & =\sum _{a^{\prime }}{\overline{P}}_t({\hat{a}}_t=a^{\prime })\left({\overline{\mathbb{E}}}_t\left[\overline{f}({\theta }^{*},a^{\prime })\right|{\hat{a}}_t=a^{\prime }]-{\overline{\mathbb{E}}}_t\left[\overline{f}\left({\hat{a}}_t\right)\right]\right)\hfill \\ \hfill & =\sum _{a^{\prime }}{\overline{P}}_t({\hat{a}}_t=a^{\prime })\left({\overline{\mathbb{E}}}_t\left[\overline{f}({\theta }^{*},a^{\prime })\right|{\hat{a}}_t=a^{\prime }]-\overline{f}\left(a^{\prime }\right)\right)=\mathrm{Tr}\left(M\right),\hfill \end{array}$$

(A81)

where the second and last equality follows since ${\overline{P}}_t\left(a_t\right)={\overline{P}}_t\left({\hat{a}}_t\right)$. Similarly, we can relate ${\mathsf{\Lambda }}_t$ with the matrix $\left(M_{a,a^{\prime }}\right)$ as

$$\begin{array}{cc}\hfill {\mathsf{\Lambda }}_t& ={\overline{\mathbb{E}}}_t\left[{(\overline{f}({\theta }^{*},a_t)-\overline{f}\left(a_t\right))}^2\right]\hfill \\ \hfill & =\sum _a{\overline{P}}_t(a_t=a){\overline{\mathbb{E}}}_t\left[{(\overline{f}({\theta }^{*},a)-\overline{f}\left(a\right))}^2\right]\hfill \\ \hfill & =\sum _{a,a^{\prime }}{\overline{P}}_t(a_t=a){\overline{P}}_t({\hat{a}}_t=a^{\prime }){\overline{\mathbb{E}}}_t\left[{(\overline{f}({\theta }^{*},a)-\overline{f}\left(a\right))}^2\right|{\hat{a}}_t=a^{\prime }]\hfill \\ \hfill & \ge \sum _{a,a^{\prime }}{\overline{P}}_t(a_t=a){\overline{P}}_t({\hat{a}}_t=a^{\prime }){\left({\overline{\mathbb{E}}}_t\left[(\overline{f}({\theta }^{*},a)-\overline{f}\left(a\right))\right|{\hat{a}}_t=a^{\prime }]\right)}^2\hfill \end{array}$$

(A82)

$$\begin{array}{cc}\hfill & =\sum _{a,a^{\prime }}{\overline{P}}_t(a_t=a){\overline{P}}_t({\hat{a}}_t=a^{\prime }){\left({\overline{\mathbb{E}}}_t[\left(\overline{f}({\theta }^{*},a)\right|{\hat{a}}_t=a^{\prime }]-\overline{f}\left(a\right)\right)}^2={\parallel M\parallel }_F^2,\hfill \end{array}$$

(A83)

whereby we obtain

$$\begin{array}{c}\hfill {\mathsf{\Gamma }}_t\le {\displaystyle \frac{\mathrm{Tr}{\left(M\right)}^2}{{\parallel M\parallel }_F^2}}\le m,\end{array}$$

(A84)

where the last inequality can be proved as in [25]. Following [19], it can be seen that ${\mathsf{\Lambda }}_t\le 2log(1+K)$ also holds. Using this, together with the upper bound (A79) gives

$$\begin{array}{c}\hfill \sum _{t=1}^T\mathbb{E}\left[{\mathrm{Term}}_1\right]\le \sqrt{2Tm{\sigma }^{2}min\{m,2log(1+K)\}log\left(1+{\displaystyle \frac{T\lambda }{m{\sigma }^2}}\right)}.\end{array}$$

(A85)

Appendix C.3. Proof of Lemma 2

We first give an upper bound on the estimation error that does not require the assumption of a linear feature map.

Appendix C.3.1. A General Upper Bound on ${\mathcal{R}}_{\mathrm{EE}1}^T$

To obtain an upper bound on ${\mathcal{R}}_{\mathrm{EE}1}^T$ that does not require the assumption that $\varphi (a,c)=G\left(a\right)c$, we leverage the same analysis as in the proof of Lemma 4. Subsequently, we obtain that

$$\begin{array}{cc}\hfill {\mathcal{R}}_{\mathrm{EE}1}^T& \le \sum _{t=1}^T\mathbb{E}\left[\Delta \left(P\left(c_t\right|{\hat{c}}_t,{\gamma }^{*}),P\left(c_t\right|{\hat{c}}_t,{\mathcal{H}}_{\hat{c}})\right)\mathbf{1}\left\{\mathcal{E}\right\}\right]+2\delta T\mathbb{E}[\parallel {\theta }^{*}{\parallel }_2\left|{\mathcal{E}}^c\right]\hfill \\ \hfill & \le \sum _{t=1}^T\mathbb{E}\left[\Delta \left(P\left(c_t\right|{\hat{c}}_t,{\gamma }^{*}),P\left(c_t\right|{\hat{c}}_t,{\mathcal{H}}_{\hat{c}})\right)\mathbf{1}\left\{\mathcal{E}\right\}\right]+2{\delta }^{2}T\sqrt{{\displaystyle \frac{m\lambda }{2\pi }}},\hfill \end{array}$$

where the event $\mathcal{E}$ is defined as in (A37). Subsequently, the first summation can be upper bounded as

$$\begin{array}{cc}\hfill \sum _{t=1}^T\mathbb{E}[\Delta \left(P\left(c_t\right|{\hat{c}}_t,{\gamma }^{*}),P\left(c_t\right|{\hat{c}}_t,{\mathcal{H}}_{\hat{c}})\right)\mathbf{1}\left\{\mathcal{E}\right\}]& \le \sqrt{2T{U}^2\sum _{t=1}^{T}I(c_t;{\gamma }^{*}|{\hat{c}}_t,{\mathcal{H}}_{\hat{c}})}\hfill \\ \hfill & =\sqrt{2T{U}^2\sum _{t=1}^T(H\left(c_t\right|{\hat{c}}_t,{\mathcal{H}}_{\hat{c}})-H\left(c_t\right|{\hat{c}}_t,{\gamma }^{*}))}\hfill \\ \hfill & =\sqrt{T{U}^2\sum _{t=1}^{T}log\left(\det \left(R_t^{-1}\right)\det \left(M_t\right)\right)}\hfill \\ \hfill & \le \sqrt{T{U}^2(\mathrm{Tr}\left({\left({\Sigma }_n{\Sigma }_{\gamma }^{-1}{\Sigma }_{n}M\right)}^{-1}\right)+log\left(T\right)\mathrm{Tr}\left({\Sigma }_c{\Sigma }_n^{-1}\right))}\hfill \end{array}$$

(A86)

where U is defined as in (A37), $M_t={\Sigma }_c^{-1}+{\Sigma }_n^{-1}$ and $R_t$ is as in (13).

To derive the last inequality, we observe the following series of relationships starting from (13):

$$\begin{array}{cc}\hfill {\Sigma }_n^{-1}{\left(H_t^{-1}\right)}^{\top }{\Sigma }_n^{-1}& ={\left({\Sigma }_n{H}_t{\Sigma }_n\right)}^{-1}={\left((t-1){\Sigma }_n-(t-2)M^{-1}+{\Sigma }_n{\Sigma }_{\gamma }^{-1}{\Sigma }_n\right)}^{-1}\hfill \\ \hfill R_t& =M-{\Sigma }_n^{-1}{\left(H_t^{-1}\right)}^{\top }{\Sigma }_n^{-1}\hfill \\ \hfill & =M-{\left((t-1){\Sigma }_n-(t-2)M^{-1}+{\Sigma }_n{\Sigma }_{\gamma }^{-1}{\Sigma }_n\right)}^{-1}\hfill \\ \hfill & =M\left[\mathbb{I}-M^{-1}{\left((t-1){\Sigma }_n-(t-2)M^{-1}+{\Sigma }_n{\Sigma }_{\gamma }^{-1}{\Sigma }_n\right)}^{-1}\right]\hfill \\ \hfill R_t^{-1}& ={\left[\mathbb{I}-M^{-1}{\left((t-1){\Sigma }_n-(t-2)M^{-1}+{\Sigma }_n{\Sigma }_{\gamma }^{-1}{\Sigma }_n\right)}^{-1}\right]}^{-1}{M}^{-1}\hfill \\ \hfill & ={\left[\mathbb{I}-{\left((t-1){\Sigma }_{n}M-(t-2)\mathbb{I}+{\Sigma }_n{\Sigma }_{\gamma }^{-1}{\Sigma }_{n}M\right)}^{-1}\right]}^{-1}{M}^{-1}\hfill \end{array}$$

$$\begin{array}{cc}\hfill R_t^{-1}M& ={\left[\mathbb{I}-{\left((t-1){\Sigma }_{n}M-(t-2)\mathbb{I}+{\Sigma }_n{\Sigma }_{\gamma }^{-1}{\Sigma }_{n}M\right)}^{-1}\right]}^{-1}\hfill \\ \hfill & ={\left[\mathbb{I}-{\left((t-1){\Sigma }_n{\Sigma }_c^{-1}+(t-1)\mathbb{I}-(t-2)\mathbb{I}+{\Sigma }_n{\Sigma }_{\gamma }^{-1}{\Sigma }_{n}M\right)}^{-1}\right]}^{-1}\hfill \\ \hfill & ={\left[\mathbb{I}-{\left(\mathbb{I}+\underset{:=P_t}{\underbrace{(t-1){\Sigma }_n{\Sigma }_c^{-1}+{\Sigma }_n{\Sigma }_{\gamma }^{-1}{\Sigma }_{n}M}}\right)}^{-1}\right]}^{-1}\hfill \\ \hfill & \stackrel{\left(a\right)}{=}{\left[\mathbb{I}-\left(\mathbb{I}-P_t{(\mathbb{I}+P_t)}^{-1}\right)\right]}^{-1}={\left(P_t{(\mathbb{I}+P_t)}^{-1}\right)}^{-1}\hfill \end{array}$$

where the equality in $\left(a\right)$ follows from Woodbury matrix identity and by the assumption that ${\Sigma }_n{\Sigma }_{\gamma }^{-1}{\Sigma }_{n}M\succ 0$, ${\Sigma }_n{\Sigma }_c^{-1}\succ 0$, we have $P_t\succ 0$ is invertible. Now,

$$\begin{array}{cc}\hfill \det \left(R_t^{-1}M\right)& ={\displaystyle \frac{1}{\det \left(P_t{(\mathbb{I}+P_t)}^{-1}\right)}}=\det \left(P_t^{-1}(\mathbb{I}+P_t)\right)\hfill \\ & =\det (P_t^{-1}+\mathbb{I})\hfill \\ \hfill & \stackrel{\left(b\right)}{\le }{\left({\displaystyle \frac{\mathrm{Tr}(P_t^{-1}+\mathbb{I})}{d}}\right)}^d={(1+\mathrm{Tr}\left(P_t^{-1}\right)/d)}^d,\hfill \end{array}$$

where the inequality in $\left(b\right)$ follows from the determinant-trace inequality. Subsequently, we have

$$\begin{array}{cc}\hfill \sum _{t=1}^{T}log\left(\det \left(R_t^{-1}M\right)\right)& \le \sum _{t=1}^{T}dlog(1+\mathrm{Tr}\left(P_t^{-1}\right)/d)\hfill \\ \hfill & \stackrel{\left(c\right)}{\le }\sum _{t=1}^T\mathrm{Tr}\left(P_t^{-1}\right)\hfill \\ \hfill & =\mathrm{Tr}\left(P_1^{-1}\right)+\sum _{t=2}^T\mathrm{Tr}\left(P_t^{-1}\right)\hfill \\ \hfill & =\mathrm{Tr}\left({\left({\Sigma }_n{\Sigma }_{\gamma }^{-1}{\Sigma }_{n}M\right)}^{-1}\right)+\sum _{t=2}^T\mathrm{Tr}\left(P_t^{-1}\right)\hfill \\ \hfill & \stackrel{\left(d\right)}{\le }\mathrm{Tr}\left({\left({\Sigma }_n{\Sigma }_{\gamma }^{-1}{\Sigma }_{n}M\right)}^{-1}\right)+log\left(T\right)\mathrm{Tr}\left({\Sigma }_c{\Sigma }_n^{-1}\right),\hfill \end{array}$$

(A87)

where the inequality in $\left(c\right)$ follows since $log(1+x)\le x$ for $x>0$ and the inequality in $\left(d\right)$ follows since by assumption $P_t⪰(t-1){\Sigma }_n{\Sigma }_c^{-1}$, whereby we have $P_t^{-1}⪯{\displaystyle \frac{1}{t-1}}{\Sigma }_c{\Sigma }_n^{-1}$ and consequently, $\mathrm{Tr}\left(P_t^{-1}\right)\le {\displaystyle \frac{1}{t-1}}\mathrm{Tr}\left({\Sigma }_c{\Sigma }_n^{-1}\right)$. Finally, we use that ${\sum }_{s=1}^{T}1/s\le log\left(T\right)$.

We thus obtain that

$$\begin{array}{c}\hfill {\mathcal{R}}_{\mathrm{EE}1}^T\le \sqrt{2\lambda mTlog(2m/\delta )\left(\mathrm{Tr}\left({\left({\Sigma }_n{\Sigma }_{\gamma }^{-1}{\Sigma }_{n}M\right)}^{-1}\right)+log\left(T\right)\mathrm{Tr}\left({\Sigma }_c{\Sigma }_n^{-1}\right)\right)}+2{\delta }^{2}T\sqrt{{\displaystyle \frac{m\lambda }{2\pi }}}\end{array}$$

(A88)

for $\delta \in (0,1)$. We note that same upper bound holds for the term ${\mathcal{R}}_{\mathrm{EE}2}^T$.

Appendix C.3.2. Upper Bound for Linear Feature Maps and Scaled Diagonal Covariance Matrices

We now obtain an upper bound on the estimation error under the assumption of a linear feature map $\varphi (a,c)=G_{a}c$ such that ${\parallel \varphi (a,c)\parallel }_2\le 1$. The following set of inequalities hold:

$$\begin{array}{cc}\hfill {\mathcal{R}}_{\mathrm{EE}1}^T& =\sum _{t=1}^T\mathbb{E}\left[\psi {(a_t^{*},{\hat{c}}_t|{\gamma }^{*})}^{\top }{\theta }^{*}-\psi {({\hat{a}}_t,{\hat{c}}_t|{\mathcal{H}}_{\hat{c}})}^{\top }{\theta }^{*}\right]\hfill \\ & \le \sum _{t=1}^T\mathbb{E}\left[\psi {(a_t^{*},{\hat{c}}_t|{\gamma }^{*})}^{\top }{\theta }^{*}-\psi {(a_t^{*},{\hat{c}}_t|{\mathcal{H}}_{\hat{c}})}^{\top }{\theta }^{*}\right]\hfill \\ & =\sum _{t=1}^T\mathbb{E}[{\mathbb{E}}_{P\left(c_t\right|{\hat{c}}_t,{\gamma }^{*})}\left[\varphi {(a_t^{*},c_t)}^{\top }{\theta }^{*}\right]-{\mathbb{E}}_{P\left(c_t\right|{\hat{c}}_t,{\mathcal{H}}_{\hat{c}})}\left[\varphi {(a_t^{*},c_t)}^{\top }{\theta }^{*}\right].\hfill \end{array}$$

(A89)

Note, that $P\left(c_t\right|{\hat{c}}_t,{\mathcal{H}}_{\hat{c}})=\mathcal{N}\left(c_t\right|V_t,R_t^{-1})$ where $R_t$ and $V_t$ are, respectively, defined in (13) and (14). Consequently, $\varphi {(a_t^{*},c_t)}^{\top }{\theta }^{*}=c_t^{\top }{G}_{a_t^{*}}^{\top }{\theta }^{*}$ is $s_t^2={\theta }^{*\top }{G}_{a_t^{*}}{R}_t^{-1}{G}_{a_t^{*}}^{\top }{\theta }^{*}$-sub-Gaussian with respect to $P\left(c_t\right|{\hat{c}}_t,{\mathcal{H}}_{\hat{c}})$. Consequently, using Lemma A1, we can upper bound the inner expectation of (A89) as

$$\begin{array}{c}\hfill |{\mathbb{E}}_{P\left(c_t\right|{\hat{c}}_t,{\gamma }^{*})}\left[\varphi {(a_t^{*},c_t)}^{\top }{\theta }^{*}\right]-{\mathbb{E}}_{P\left(c_t\right|{\hat{c}}_t,{\mathcal{H}}_{\hat{c}})}\left[\varphi {(a_t^{*},c_t)}^{\top }{\theta }^{*}\right|\le \sqrt{2s_t^2{D}_{\mathrm{KL}}(P\left(c_t\right|{\hat{c}}_t,{\gamma }^{*})\parallel P\left(c_t\right|{\hat{c}}_t,{\mathcal{H}}_{\hat{c}}))}.\end{array}$$

(A90)

Summing over t and using Cauchy–Schwarz inequality then gives

$$\begin{array}{cc}\hfill {\mathcal{R}}_{\mathrm{EE}1}^T& \le \sqrt{2\left(\sum _{t=1}^T\mathbb{E}\left[s_t^2\right]\right)\left(\sum _{t=1}^T\mathbb{E}\left[D_{\mathrm{KL}}(P\left(c_t\right|{\hat{c}}_t,{\gamma }^{*})\parallel P\left(c_t\right|{\hat{c}}_t,{\mathcal{H}}_{\hat{c}}))\right]\right)}.\hfill \end{array}$$

(A91)

We now evaluate the KL-divergence term. To this end, note that conditioned on ${\gamma }^{*}$ and ${\hat{c}}_t$, $c_t$ is independent of ${\mathcal{H}}_{\hat{c}}$, i.e., $P\left(c_t\right|{\hat{c}}_t,{\gamma }^{*},{\mathcal{H}}_{\hat{c}})=P\left(c_t\right|{\hat{c}}_t,{\gamma }^{*})$. This gives

$$\begin{array}{c}\sum _{t=1}^T\mathbb{E}\left[D_{\mathrm{KL}}(P\left(c_t\right|{\hat{c}}_t,{\gamma }^{*})\parallel P\left(c_t\right|{\hat{c}}_t,{\mathcal{H}}_{\hat{c}}))\right]\hfill \\ =\sum _{t=1}^{T}I(c_t;{\gamma }^{*}|{\hat{c}}_t,{\mathcal{H}}_{\hat{c}})\hfill \\ =\sum _{t=1}^{T}H\left(c_t\right|{\hat{c}}_t,{\mathcal{H}}_{\hat{c}})-H\left(c_t\right|{\hat{c}}_t,{\gamma }^{*})\hfill \\ ={\displaystyle \frac{1}{2}}\sum _{t=1}^T{\mathbb{E}}_{P({\hat{c}}_t,{\mathcal{H}}_{\hat{c}})}[log\det \left(R_t^{-1}\right)]-{\displaystyle \frac{1}{2}}\sum _{t=1}^T{\mathbb{E}}_{P({\hat{c}}_t,{\gamma }^{*})}[log\det \left(M^{-1}\right)]\hfill \\ =\sum _{t=1}^T{\displaystyle \frac{1}{2}}{\mathbb{E}}_{P({\hat{c}}_t,{\mathcal{H}}_{\hat{c}})}[log\left(\det \left(R_t^{-1}\right)\det \left(M\right)\right)\hfill \\ \le {\displaystyle \frac{d{\sigma }_c^2}{{\sigma }_n^2}}\left({\displaystyle \frac{{\sigma }_{\gamma }^2}{{\sigma }_n^2+{\sigma }_c^2}}+log(T-1)\right),\hfill \end{array}$$

(A92)

where $M={\Sigma }_c^{-1}+{\Sigma }_n^{-1}$ and $R_t$ is as in (13). The first equality follows by noting that

$$\begin{array}{cc}\hfill \mathbb{E}\left[D_{\mathrm{KL}}(P\left(c_t\right|{\hat{c}}_t,{\gamma }^{*})\parallel P\left(c_t\right|{\hat{c}}_t,{\mathcal{H}}_{\hat{c}}))\right]& ={\mathbb{E}}_{P({\hat{c}}_t,{\mathcal{H}}_{\hat{c}})}\left[{\mathbb{E}}_{P\left({\gamma }^{*}\right|{\hat{c}}_t,{\mathcal{H}}_{\hat{c}})}\left[D_{\mathrm{KL}}(P\left(c_t\right|{\hat{c}}_t,{\gamma }^{*},{\mathcal{H}}_{\hat{c}})\parallel P\left(c_t\right|{\hat{c}}_t,{\mathcal{H}}_{\hat{c}}))\right]\right]\hfill \\ & ={\mathbb{E}}_{P({\hat{c}}_t,{\mathcal{H}}_{\hat{c}})}\left[I(c_t;{\gamma }^{*}|{\hat{c}}_t,{\mathcal{H}}_{\hat{c}})\right]\hfill \end{array}$$

with the outer expectation taken over ${\hat{c}}_t$ and ${\mathcal{H}}_{\hat{c}}$. The last inequality is proved in Appendix C.3.3 using that ${\Sigma }_c={\sigma }_c^2\mathbb{I}$, ${\Sigma }_n={\sigma }_n^2\mathbb{I}$ and ${\Sigma }_{\gamma }={\sigma }_{\gamma }^2\mathbb{I}$.

We can now upper bound ${\sum }_t\mathbb{E}\left[s_t^2\right]$ as follows.

$$\begin{array}{cc}\hfill \mathbb{E}\left[s_t^2\right]\le \mathbb{E}\left[\underset{a}{max}{\theta }^{*\top }{G}_a{R}_t^{-1}{G}_a^{\top }{\theta }^{*}\right]\le \sum _a\mathbb{E}\left[{\theta }^{*\top }{G}_a{R}_t^{-1}{G}_a^{\top }{\theta }^{*}\right]& =\sum _a\mathrm{Tr}\left(G_a{R}_t^{-1}{G}_a^{\top }\mathbb{E}\left[{\theta }^{*}{\theta }^{*\top }\right]\right)\hfill \\ & =\lambda \sum _a\mathrm{Tr}\left(R_t^{-1}{G}_a^{\top }{G}_a\right)\hfill \\ \hfill & =\lambda b_t\mathrm{Tr}\left(\sum _a{G}_a^{\top }{G}_a\right)\hfill \end{array}$$

where the last equality uses $R_t^{-1}=b_t\mathbb{I}$ as in (A95). Using (A95), we obtain

$$\begin{array}{cc}\hfill \sum _t{b}_t& ={\displaystyle \frac{{\sigma }_c^2{\sigma }_n^2}{f}}\sum _t\left(1+{\displaystyle \frac{{\sigma }_c^2{\sigma }_{\gamma }^2}{(t-1){\sigma }_{\gamma }^2{\sigma }_n^2+f{\sigma }_n^2}}\right)\hfill \\ \hfill & ={\displaystyle \frac{{\sigma }_c^2{\sigma }_n^{2}T}{f}}+\sum _t{\displaystyle \frac{{\sigma }_c^2}{f}}{\displaystyle \frac{{\sigma }_c^2{\sigma }_{\gamma }^2}{(t-1){\sigma }_{\gamma }^2+f}}\hfill \\ \hfill & ={\displaystyle \frac{{\sigma }_c^2{\sigma }_n^{2}T}{f}}+{\displaystyle \frac{{\sigma }_c^4{\sigma }_{\gamma }^2}{f^2}}+\sum _{t>1}{\displaystyle \frac{{\sigma }_c^2}{f}}{\displaystyle \frac{{\sigma }_c^2{\sigma }_{\gamma }^2}{(t-1){\sigma }_{\gamma }^2+f}}\hfill \\ \hfill & \le {\displaystyle \frac{{\sigma }_c^2{\sigma }_n^2}{f}}T+{\displaystyle \frac{{\sigma }_c^4{\sigma }_{\gamma }^2}{f^2}}+{\displaystyle \frac{{\sigma }_c^4}{f}}log(T-1).\hfill \end{array}$$

Using the above relation, we obtain

$$\begin{array}{c}\hfill \sum _t\mathbb{E}\left[s_t^2\right]=\lambda \mathrm{Tr}\left(\sum _a{G}_a^{\top }{G}_a\right)\sum _t{b}_t\le {\displaystyle \frac{\lambda K{\sigma }_c^2}{f}}\underset{a}{max}\mathrm{Tr}\left(G_a^{\top }{G}_a\right)\left({\sigma }_n^{2}T+{\displaystyle \frac{{\sigma }_c^2{\sigma }_{\gamma }^2}{f}}+{\sigma }_c^{2}log(T-1)\right).\end{array}$$

If $\lambda \le {\displaystyle \frac{d{\sigma }^2}{T}}$, we obtain

$$\begin{array}{c}\hfill \sum _t\mathbb{E}\left[s_t^2\right]\le {\displaystyle \frac{d{\sigma }^{2}K{\sigma }_c^2}{f}}\underset{a}{max}\mathrm{Tr}\left(G_a^{\top }{G}_a\right)\left({\sigma }_n^2+{\displaystyle \frac{{\sigma }_c^2{\sigma }_{\gamma }^2}{Tf}}+{\sigma }_c^2{\displaystyle \frac{log(T-1)}{T}}\right)=L\end{array}$$

Using the above inequality together with (A92) in (A91) yields

$$\begin{array}{c}\hfill {\mathcal{R}}_{\mathrm{EE}1}^T\le \sqrt{2L{\displaystyle \frac{d{\sigma }_c^2}{{\sigma }_n^2}}\left({\displaystyle \frac{{\sigma }_{\gamma }^2}{{\sigma }_n^2+{\sigma }_c^2}}+log(T-1)\right)}.\end{array}$$

(A93)

An upper bound on ${\mathcal{R}}_{\mathrm{EE}2}^T$ similarly follows.

Appendix C.3.3. Analysis of $R_t$ for Scaled Diagonal Covariance Matrices

Assume that ${\Sigma }_n={\sigma }_n^2\mathbb{I}$, ${\Sigma }_c={\sigma }_c^2\mathbb{I}$ and ${\Sigma }_{\gamma }={\sigma }_{\gamma }^2\mathbb{I}$. Then, from (13), we obtain

$$\begin{array}{cc}\hfill H_t& =(t-1){\Sigma }_n^{-1}-(t-2){\Sigma }_n^{-1}{M}^{-1}{\Sigma }_n^{-1}+{\Sigma }_{\gamma }^{-1}\hfill \\ \hfill & =({\displaystyle \frac{(t-1)}{{\sigma }_n^2}}-{\displaystyle \frac{(t-2){\sigma }_n^2{\sigma }_c^2}{{\sigma }_n^4\underset{:=f}{\underbrace{({\sigma }_c^2+{\sigma }_n^2)}}}}+{\displaystyle \frac{1}{{\sigma }_{\gamma }^2}})\mathbb{I}\hfill \\ \hfill & ={\displaystyle \frac{(t-1){\sigma }_{\gamma }^2-(t-2){\sigma }_c^2{\sigma }_{\gamma }^2/f+{\sigma }_n^2}{{\sigma }_n^2{\sigma }_{\gamma }^2}}\mathbb{I}.\hfill \end{array}$$

(A94)

This implies

$$\begin{array}{cc}\hfill H_t^{-1}& ={\displaystyle \frac{{\sigma }_n^2{\sigma }_{\gamma }^2}{(t-1){\sigma }_{\gamma }^2-(t-2){\sigma }_c^2{\sigma }_{\gamma }^2/f+{\sigma }_n^2}}\mathbb{I}\hfill \end{array}$$

whereby we obtain

$$\begin{array}{cc}\hfill R_t& ={\displaystyle \frac{f\left((t-1){\sigma }_{\gamma }^2+{\sigma }_n^2+{\sigma }_c^2\right)}{{\sigma }_c^2((t-1){\sigma }_{\gamma }^2{\sigma }_n^2+{\sigma }_c^2{\sigma }_{\gamma }^2+f{\sigma }_n^2)}}\mathbb{I}\mathrm{and}\hfill \\ \hfill R_t^{-1}& ={\displaystyle \frac{{\sigma }_c^2((t-1){\sigma }_{\gamma }^2{\sigma }_n^2+{\sigma }_c^2{\sigma }_{\gamma }^2+f{\sigma }_n^2)}{f\left((t-1){\sigma }_{\gamma }^2+{\sigma }_n^2+{\sigma }_c^2\right)}}\mathbb{I}=b_t\mathbb{I}.\hfill \end{array}$$

(A95)

Noting that $M_t={\displaystyle \frac{f}{{\sigma }_n^2{\sigma }_c^2}}\mathbb{I}$ we then have

$$\begin{array}{cc}\hfill R_t^{-1}{M}_t& ={\displaystyle \frac{(t-1){\sigma }_{\gamma }^2{\sigma }_n^2+{\sigma }_c^2{\sigma }_{\gamma }^2+f{\sigma }_n^2}{{\sigma }_n^2\left((t-1){\sigma }_{\gamma }^2+{\sigma }_n^2+{\sigma }_c^2\right)}}\mathbb{I}={\displaystyle \frac{(t-1){\sigma }_{\gamma }^2{\sigma }_n^2+{\sigma }_c^2{\sigma }_{\gamma }^2+f{\sigma }_n^2}{(t-1){\sigma }_{\gamma }^2{\sigma }_n^2+f{\sigma }_n^2}}\mathbb{I}=(1+{\displaystyle \frac{{\sigma }_c^2{\sigma }_{\gamma }^2}{(t-1){\sigma }_{\gamma }^2{\sigma }_n^2+f{\sigma }_n^2}})\mathbb{I}.\hfill \end{array}$$

(A96)

Subsequently, we obtain that for $t>1$,

$$\begin{array}{c}\hfill log\left(\det \left(R_t^{-1}{M}_t\right)\right)=dlog\left(1+{\displaystyle \frac{{\sigma }_c^2{\sigma }_{\gamma }^2}{(t-1){\sigma }_{\gamma }^2{\sigma }_n^2+f{\sigma }_n^2}}\right)\le dlog\left(1+{\displaystyle \frac{{\sigma }_c^2/{\sigma }_n^2}{(t-1)}}\right)\le {\displaystyle \frac{d{\sigma }_c^2/{\sigma }_n^2}{t-1}},\end{array}$$

(A97)

whereby

$$\begin{array}{cc}\hfill \sum _{t=1}^{T}log\left(\det \left(R_t^{-1}{M}_t\right)\right)& \le dlog\left(1+{\displaystyle \frac{{\sigma }_c^2{\sigma }_{\gamma }^2}{f{\sigma }_n^2}}\right)+\sum _{t>1}^T{\displaystyle \frac{d{\sigma }_c^2/{\sigma }_n^2}{t-1}}\hfill \end{array}$$

(A98)

$$\begin{array}{cc}\hfill & \le {\displaystyle \frac{d{\sigma }_c^2}{{\sigma }_n^2}}\left({\displaystyle \frac{{\sigma }_{\gamma }^2}{f}}+log(T-1)\right)\hfill \end{array}$$

(A99)

where the last inequality follows since $log(1+x)\le x$ and ${\sum }_{s=1}^T{\displaystyle \frac{1}{s}}\le log\left(T\right).$

Appendix D.1. Gaussian Bandits

For Gaussian bandits, we implemented the baselines as explained below.

• TS_naive: This algorithm implements the following action policy at each iteration t,

$$\begin{array}{c}\hfill a_t=arg\underset{a\in \mathcal{A}}{max}\varphi {(a,{\hat{c}}_t)}^{\top }{\theta }_t,\end{array}$$

where ${\theta }_t$ is sampled from a Gaussian distribution $\mathcal{N}({\mu }_{t-1,\mathrm{naive}},{\Sigma }_{t-1,\mathrm{naive}}^{-1})$ with

$$\begin{array}{cc}\hfill {\Sigma }_{t-1,\mathrm{naive}}& ={\displaystyle \frac{\mathbb{I}}{\lambda }}+{\displaystyle \frac{1}{{\sigma }^2}}\sum _{\tau =1}^{t-1}\varphi (a_{\tau },{\hat{c}}_{\tau })\varphi {(a_{\tau },{\hat{c}}_{\tau })}^{\top }\hfill \\ \hfill {\mu }_{t-1,\mathrm{naive}}& ={\displaystyle \frac{{\Sigma }_{t-1,\mathrm{naive}}^{-1}}{{\sigma }^2}}\left(\sum _{\tau =1}^{t-1}{r}_{\tau }\varphi (a_{\tau },{\hat{c}}_{\tau })\right).\hfill \end{array}$$

• TS_oracle: In this baseline, the agent has knowledge of the true predictive distribution $P\left(c_t\right|{\hat{c}}_t,{\gamma }^{*})$. Consequently, at each iteration t, the algorithm chooses action

$$\begin{array}{c}\hfill a_t=arg\underset{a\in \mathcal{A}}{max}\psi {(a,{\hat{c}}_t|{\gamma }^{*})}^{\top }{\theta }_t,\end{array}$$

where ${\theta }_t$ is sampled from a Gaussian distribution $\mathcal{N}({\mu }_{t-1,\mathrm{poc}},{\Sigma }_{t-1,\mathrm{poc}}^{-1})$ with

$$\begin{array}{cc}\hfill {\Sigma }_{t-1,\mathrm{poc}}& ={\displaystyle \frac{\mathbb{I}}{\lambda }}+{\displaystyle \frac{1}{{\sigma }^2}}\sum _{\tau =1}^{t-1}\psi (a_{\tau },{\hat{c}}_{\tau }|{\gamma }^{*})\psi {(a_{\tau },{\hat{c}}_{\tau }|{\gamma }^{*})}^{\top }\hfill \\ \hfill {\mu }_{t-1,\mathrm{poc}}& ={\displaystyle \frac{{\Sigma }_{t-1,\mathrm{poc}}^{-1}}{{\sigma }^2}}\left(\sum _{\tau =1}^{t-1}{r}_{\tau }\psi (a_{\tau },{\hat{c}}_{\tau }|{\gamma }^{*})\right).\hfill \end{array}$$

For Gaussian bandits, the following figure shows additional experiment comparing the performance of our proposed Algorithm 1 for varying values of the number K of actions. All parameters are set as in Figure 1 (Left).

Figure A1. Bayesian cumulative regret of Algorithm 1 as a function of iterations over varying number K of actions.

Figure A1. Bayesian cumulative regret of Algorithm 1 as a function of iterations over varying number K of actions.

Appendix D.2. Logistic Bandits

In the case of logistic bandits, we implemented the baselines as explained below.

• TS_naive: This algorithm considers the following sampling distribution:

$$\begin{array}{c}\hfill Q\left({\theta }^{*}\right|{\mathcal{H}}_{t-1,r,a,\hat{c}})\propto P\left({\theta }^{*}\right)\prod _{\tau =1}^{t-1}\mathrm{Ber}\left(\mu \left(\varphi {(a_{\tau },{\hat{c}}_{\tau })}^{\top }{\theta }^{*}\right)\right).\end{array}$$

However, due to the non-conjugateness of Gaussian prior $P\left({\theta }^{*}\right)$ and Bernoulli reward likelihood, sampling from the above posterior distribution is not straightforward. Consequently, we adopt the Langevin Monte Carlo (LMC) sampling approach from [20]. To sample ${\theta }_t$ at iteration t, we run LMC for $I=50$ iterations with learning rate ${\eta }_t=0.2/t$ and inverse temperature parameter ${\beta }^{-1}=0.001$. Then, ${\theta }_t$ is chosen as the output of the LMC after $I=50$ iterations. Using the sampled ${\theta }_t$, the algorithm then chooses the action $a_t$ as

$$\begin{array}{c}\hfill a_t=arg\underset{a\in \mathcal{A}}{max}\varphi {(a,{\hat{c}}_t)}^{\top }{\theta }_t.\end{array}$$

• TS_oracle: This algorithm considers the following sampling distribution:

$$\begin{array}{c}\hfill Q\left({\theta }^{*}\right|{\mathcal{H}}_{t-1,r,a,\hat{c}})\propto P\left({\theta }^{*}\right)\prod _{\tau =1}^{t-1}\mathrm{Ber}\left(\mu \left(\psi {(a_{\tau },{\hat{c}}_{\tau }|{\gamma }^{*})}^{\top }{\theta }^{*}\right)\right),\end{array}$$

where $\psi (a_t,{\hat{c}}_t|{\gamma }^{*})={\mathbb{E}}_{P\left(c_t\right|{\hat{c}}_t,{\gamma }^{*})}\left[\varphi (a_t,c_t)\right]$ is the expected feature map under the posterior predictive distribution with known ${\gamma }^{*}$. As before, to sample from the above distribution, we use $I=50$ iterations of LMC with learning rate ${\eta }_t=0.2/t$ and inverse temperature parameter ${\beta }^{-1}=0.001$. Using the sampled ${\theta }_t$, the algorithm then chooses the action $a_t$ as

$$\begin{array}{c}\hfill a_t=arg\underset{a\in \mathcal{A}}{max}\psi {(a,{\hat{c}}_t|{\gamma }^{*})}^{\top }{\theta }_t.\end{array}$$