Analysis of Performance Measure in Q Learning with UCB Exploration

Abstract

Compared to model-based Reinforcement Learning (RL) approaches, model-free RL algorithms, such as Q-learning, require less space and are more expressive, since specifying value functions or policies is more flexible than specifying the model for the environment. This makes model-free algorithms more prevalent in modern deep RL. However, model-based methods can more efficiently extract the information from available data. The Upper Confidence Bound (UCB) bandit can improve the exploration bonuses, and hence increase the data efficiency in the Q-learning framework. The cumulative regret of the Q-learning algorithm with an UCB exploration policy in the episodic Markov Decision Process has recently been explored in the underlying environment of finite state-action space. In this paper, we study the regret bound of the Q-learning algorithm with UCB exploration in the scenario of compact state-action metric space. We present an algorithm that adaptively discretizes the continuous state-action space and iteratively updates Q-values. The algorithm is able to efficiently optimize rewards and minimize cumulative regret.

1. Background

1.1. Markov Decision Process

The Markov Decision Process (MDP) is a discrete-time sequential decision-making problem. In MDP, there is a finite set of states $\mathcal{S}$ and a finite set of actions $\mathcal{A}$. At time-step t, the agent is at some state $s\in \mathcal{S}$, and the agent takes some action $a\in \mathcal{A}$. By the state transition probability function $\mathcal{P}$, the agent moves to some new state $s^{\prime }\in \mathcal{S}$ at time $t+1$, where $s^{\prime }\sim \mathcal{P}(·|s,a)$. The agent will then receive some immediate reward $r(s^{\prime },a)$, the reward after transitioning to state $s^{\prime }$, due to action a. In MDP, for the certain pair $(s,a)$, the state transition process satisfies the Markov property. That means the process is conditionally independent of all of the past states and actions. For example, if there exists only one action in each state (fixed transition probability in each state) and all the rewards are zeros, then MDP reduces to the Markov Chain. Policy $\pi$ is a map from state to action ($\pi :\mathcal{S}\to \mathcal{A}$). If the environment has finite time-steps, then we call it a finite time horizon. Otherwise, the environment is an infinite time horizon. The core problem in MDP is to find an optimal policy that can maximize the cumulative function of random rewards over a finite/infinite time horizon. For example, the typical cumulative function of random rewards in the finite time horizon is a sum of rewards. For example, for the specific policy $\pi$ with a finite time horizon H, it is ${\mathbb{E}}_{s_{t+1}\sim \mathcal{P}\left(·|s_t,\pi \left(s_t\right)\right)}\left[{\sum }_{t=0}^{H}r(s_{t+1},\pi \left(s_t\right))\right]$.

MDP hence contains two important functions: the reward function r and state transition function $\mathcal{P}$. If we know both functions, that is, we can predict which reward will be received and which states will be in the next transition, then MDP can be solved by Dynamic Programming (DP). DP requires the full description of MDP, with known transition probabilities and reward functions.

1.2. Model-Based and Model-Free Reinforcement Learning

However, for most cases, reward function r and state transition function $\mathcal{P}$ are not precisely known. The purpose of reinforcement learning (RL) is to solve MDP when the reward and transition functions are unknown. One approach in RL is to learn a model of how the environment works from its observation and then find a solution using that model. For example, with a new state-action pair and the observation of the corresponding reward, we can update the estimate of r and $\mathcal{P}$. With the simulation of different actions in many states, we can have a learned transition function and learned reward function with high confidence. If we adequately model the environment, then we can use the learned model to find policy. This approach is called model-based RL. However, model-based RL requires too many trials to learn a particular task independent of the choice of policy iteration or value iteration. Hence, model-based RL methods are often inapplicable to large-scale systems. In contrast, we do not need to learn a model, but can directly find a policy. For example, the Q-learning algorithm is one approach that learns MDP and solves it to find the optimal policy at the same time. Q-learning directly estimates the optimal Q-values for each action in each state, and the policy could be derived by choosing the action with the highest Q-value in the given state. Since we do not learn a model of the environment, we call it model-free RL. The core idea of Q-learning is to balance the exploration and exploitation trade-off: trying random actions to explore the underlying MDP, while following the so far optimal policy to maximize the long-term reward.

2. Introduction

Reinforcement Learning (RL), as a learning problem to maximize the numerical reward, learns how to map situations to actions in unknown underlying system dynamics [1]. While model-based RL can approach high-quality and effective decisions, the stability issue about scaling up to the approximate setting with high-dimensional state and action spaces has been demonstrated [2,3]. Besides, due to the assumption of the model, model-based RL suffers from model bias, especially when the samples are few and informative prior knowledge is limited [4,5,6]. On the contrary, model-free RL algorithms can directly update the value and policy functions without any assumption on the model environment [1]. As the most classical model-free RL algorithm, Q-learning was introduced by Watkins in his Ph.D. dissertation. Watkins designed the action-reply MDP algorithm, which achieves convergence to the optimal Q function with a finite state and action space setting [7,8]. This approach is considered as a simpler way of learning optimality in a controlled Markov domain. From the classical Q-learning algorithm to modern Deep Q-Networks [9], Advantage Actor-Critic [10], and Trust Region Policy Optimization [11], most state-of-the-art RL are in the model-free approach. Model-free RL algorithms require less space and are more expressive since specifying value functions or policies are more flexible compared to specifying the model of the environment. The advantages of model-free methods underlie their success in deep RL applications [12].

However, model-based methods are generally more promising for efficient extraction of the information from available data than the model-free approaches, such as Q-learning [6,11]. The requirement of data efficiency in model-free RL is hence increasing. The key to achieve good data efficiency generally lies in the tradeoff between exploration and exploitation. Using the bandit to explore the uncertain environment while maximizing the reward has been imported to obtain optimal data efficiency [12,13,14]. Q-learning with a common $ϵ$-greedy exploration strategy exponentially takes many episodes to learn [12]. The Upper Confidence Bound (UCB) algorithm, as a multi-armed bandit algorithm, assigns exploration bonuses in many stochastic settings [15,16,17,18]. The UCB exploration policy is hence considered as a tool for improving the exploration bonuses in Q-learning [12]. This approach can improve data efficiency while leveraging the advantages of model-free methods. However, the Q-learning algorithm with UCB exploration is restricted to the environment of finite state and action space [7,12,19]. Due to the natural settings of RL applications, RL over underlying MDP with the environment of continuous state and action space has substantially been an intractable and challenging problem [20,21]. For example, most robotic applications of RL require continuous state spaces defined by utilizing continuous variables, such as position, velocity,  torque, and so forth. Although Q-learning is commonly applied to problems with discrete states and actions, it has been extended to continuous space under different scenarios [22,23]. This motivates us to extend the state-of-the-art Q-learning algorithm equipped with an UCB exploration policy to a continuous state-action space environment. In modern RL, the algorithm that can leverage the advantages of model-free methods and data efficiency in the continuous state-action space is demanding.

Our contributions are: (1) Development of a state-of-the-art Q-learning algorithm with an UCB exploration policy to compact state and action metric space; and (2) Achievement of a tight bound for the cumulative regret for the underlying algorithm using the covering dimension techniques. We provide a rigorous analysis of bounding the regret with the concentration of measure analysis and combinatorial approaches.

3. Notation and Settings

3.1. Q-Learning

We consider a setting of an episodic Markov decision process MDP ($\mathcal{S},\mathcal{A},H,\mathcal{P},r$), where state space is a compact metric space $(\mathcal{S},d_{\mathcal{S}})$, and action space is also a compact metric space $(\mathcal{A},d_{\mathcal{A}})$. $\mathcal{S}\times \mathcal{A}$ is a product compact metric space with metric $d_{\mathcal{S}\times \mathcal{A}}$. H is the number of steps in each episode. $\mathcal{P}$ is the state transition function where the transitional probability from the current state-action pair in $\mathcal{S}\times \mathcal{A}$ to the next state is $x^{\prime }\sim \mathcal{P}(·|x,a)$. The reward function at step $h\in \left[H\right]$, is $r_h:\mathcal{S}\times \mathcal{A}\to [0,1]$. Since we discuss the finite time horizon Q-learning here, there is no discount factor included in the value function. Let the policy function be $\pi :\mathcal{S}\to \mathcal{A}$. Denote by $V_h^{\pi }:\mathcal{S}\to \mathbb{R}$ the value function at step h under policy $\pi$. $V_h^{\pi }$ stands for the expected sum of remaining rewards collected from $x_h=x$ to the end of episode given by policy $\pi$. Mathematically,

$$V_h^{\pi }\left(x\right):=\mathbb{E}\left[\sum _{h^{\prime }=h}^H{r}_{h^{\prime }}(x_{h^{\prime }},{\pi }_{h^{\prime }}\left(x_{h^{\prime }}\right))|x_h=x\right].$$

Accordingly, we can define $Q_h^{\pi }:\mathcal{S}\times \mathcal{A}\to \mathbb{R}$ as the Q-value function at step h under policy $\pi$. This gives the expected sum of the remaining rewards, starting from $x_h=x, a_h=a$ to the end of episode. Mathematically,

$$Q_h^{\pi }(x,a):=r_h(x,a)+\mathbb{E}\left[\sum _{h^{\prime }=h+1}^H{r}_{h^{\prime }}(x_{h^{\prime }},{\pi }_{h^{\prime }}\left(x_{h^{\prime }}\right))|x_h=x,a_h=a\right].$$

For simplicity, we define the expected future value

$${\mathcal{P}}_h{V}_h(x,a):={\mathbb{E}}_{x^{\prime }\sim P(·|x,a)}{V}_{h+1}\left(x^{\prime }\right),$$

where $V_{h+1}$ is the proxy of the value function. The empirical counterpart of episode k, say $x_{h+1}^k$ the next observed state from $(x,a)$, is

$${\widehat{\mathcal{P}}}_h^k{V}_h(x,a):=V_{h+1}\left(x_{h+1}^k\right).$$

(1)

The above notations follow from [12]. Define the optimal policy ${\pi }^{*}$ by the optimal value $V_h^{*}\left(x\right)={sup}_{\pi }{V}_h^{\pi }\left(x\right)$. The Bellman equation for any (deterministic) policy $\pi$ is

$$\begin{array}{c}\hfill \left\{\begin{array}{c}{V}_h^{\pi }\left(x\right)=Q_h^{\pi }(x,\pi \left(x\right))\hfill \\ Q_h^{\pi }(x,a)=r_h(x,a)+{\mathcal{P}}_h{V}_h^{\pi }(x,a).\hfill \end{array}\right.\end{array}$$

In addition, $V^{*}$ and $Q^{*}$ are the value and Q-value functions associated with the optimal policy:

$$\begin{array}{c}\hfill \left\{\begin{array}{c}{V}_h^{*}\left(x\right)=\underset{a\in \mathcal{A}}{sup}{Q}_h^{*}(x,a)\hfill \\ Q_h^{*}(x,a)=r_h(x,a)+{\mathcal{P}}_h{V}_h^{*}(x,a).\hfill \end{array}\right.\end{array}$$

(2)

3.2. Lipschitz Assumption

In this work, we assume $V^{*}$ is Lipschitz-continuous with Lipschitz constant D. Assume the next step from the current state-action pair is Lipschitz-continuous. That is, let $x_{h+1}$ be the next step from $(x_h,a_h)$ and $y_{h+1}$ be the next step from $(y_h,b_h)$. Then $d_{\mathcal{S}}\left(x_{h+1},y_{h+1}\right) \le M·d_{\mathcal{S}\times \mathcal{A}}\left((x_h,a_h),(y_h,b_h)\right)$ with probability 1 for some constant M. This implies that $\widehat{\mathcal{P}}{V}^{*}$ is also Lipschitz-continuous:

$$\begin{array}{cc}\hfill \Vert \widehat{{\mathcal{P}}_h}{V}_h^{*}(x_h,a_h)-\widehat{{\mathcal{P}}_h}{V}_h^{*}(y_h,b_h)\Vert & = \Vert V_{h+1}^{*}\left(x_{h+1}\right)-V_{h+1}^{*}\left(y_{h+1}\right)\Vert \hfill \\ \hfill & \le D·d_{\mathcal{S}}\left(x_{h+1},y_{h+1}\right)\le D·M·d_{\mathcal{S}\times \mathcal{A}}\left((x_h,a_h),(y_h,b_h)\right)\hfill \\ \hfill & = L·d_{\mathcal{S}\times \mathcal{A}}\left((x_h,a_h),(y_h,b_h)\right)\hfill \end{array}$$

(3)

for $L=D·M$. The Lipschitz constant is L for $\widehat{\mathcal{P}}{V}^{*}$.

3.3. Performance Measure

Our main interest is to provide a practical algorithm that can bound the regret. Assume K is the total number of episodes. At the beginning of each episode k, the adversary picks a starting state $x_1^k$. The agent will choose a corresponding greedy policy ${\pi }_k$ before the start of episode k. Recall that $V_h^{{\pi }_k}$ is the value function of ${\pi }_k$ in episode k. The cumulative regret (performance measure) of K episodes is

$$\mathtt{Regret}\left(K\right):=\sum _{k=1}^K\left(V_1^{*}\left(x_1^k\right)-V_1^{{\pi }_k}\left(x_1^k\right)\right).$$

(4)

3.4. Iterative Updating Rule

Since $\mathcal{S}\times \mathcal{A}$ is a compact metric space, running the algorithm will partition the space into a discrete framework, for example, balls. In the algorithm, for $(x,a)\in \mathcal{S}\times \mathcal{A}$ at episode k, we look at all the past state-action pairs that are in the same ball of $(x,a)$, $B(x,a)$, from the episode up to $k-1$. Assume at the past episode, $k_i\in \{1,\cdots ,k-1\}$ and step h, $\left(x_h^{k_i},a_h^{k_i}\right)\in B(x,a)$ was selected. We call $\left\{\left(x_h^{k_i},a_h^{k_i}\right)\right\}$ active points. Denote by $n_h^k(x,a)$ the total number of state-action pairs $\left\{\left(x_h^{k_i},a_h^{k_i}\right)\right\}$ in $B(x,a)$ at step h up to episode k. Additionally, its observed state at step $h+1$ is $x_{h+1}^{k_i}$ for episode $k_i$. We denote $Q_h^k$, $V_h^k$, $n_h^k$ as $Q_h$, $V_h$, and $n_h$ at the beginning of episode k. We also have a bandit term $b_t$, of which more details will be given later.

Definition 1

(Iterative updating rule of $Q_h^k$).

$$Q_h^{k+1}(x,a)=\left\{\begin{array}{cc}(1-{\alpha }_t)Q_h^k(x,a)+{\alpha }_t[r_h(x,a)+V_{h+1}^k\left(x_{h+1}^k\right)+b_t]\hfill & \mathit{if}(x_h^k,a_h^k)\mathit{in}B(x,a);\hfill \\ Q_h^k(x,a)\hfill & \mathit{otherwise}.\hfill \end{array}\right.$$

(5)

Here, t is the counter for the times the algorithm has visited the $B(x,a)$ at step h, so $t=n_h^k(x,a)$. Accordingly, the iterative updating rule for $V_h^k$ is

$$V_h^k\left(x\right)\leftarrow min\left\{H,\underset{a^{\prime }\in \mathcal{A}}{sup}{Q}_h^k(x,a^{\prime })\right\},\forall x\in \mathcal{S}.$$

(6)

For the choice of learning rate ${\alpha }_t$ in the updating rule, we choose ${\alpha }_t:=\frac{1}{t}$. Then, we define the following related quantities:

$${\alpha }_t^0=\prod _{j=1}^t(1-{\alpha }_j),{\alpha }_t^i={\alpha }_i·\prod _{j=i+1}^t(1-{\alpha }_j).$$

(7)

The above definition yields the following properties that are useful in our analysis.

Lemma 1.

The following properties hold for ${\alpha }_t$ and ${\alpha }_t^i$:

1. ${\sum }_{t=1}^{\infty }{\alpha }_t=\infty$ and ${\sum }_{t=1}^{\infty }{\alpha }_t^2<\infty$ ,       2. $\frac{1}{\sqrt{t}}\le {\sum }_{i=1}^t\frac{{\alpha }_t^i}{\sqrt{i}}\le \frac{2}{\sqrt{t}}$ .

Proof. 

The first property is trivial. We will prove the second property by induction on t.

Let us check the right-hand side. Clearly the statement is true for $t=1$. Assume the result holds for some $t-1$, i.e., ${\sum }_{i=1}^{t-1}\frac{{\alpha }_{t-1}^i}{\sqrt{i}}\le \frac{2}{\sqrt{t-1}}$. Note that ${\alpha }_t^i={\alpha }_t=\frac{1}{t}$. By using the induction hypothesis we can write

$$\begin{array}{cc}\hfill \sum _{i=1}^t\frac{{\alpha }_t^i}{\sqrt{i}}& =\frac{{\alpha }_t^t}{\sqrt{t}}+(1-{\alpha }_t)\sum _{i=1}^{t-1}\frac{{\alpha }_{t-1}^i}{\sqrt{i}}\hfill \\ \hfill & =\frac{{\alpha }_t}{\sqrt{t}}+(1-{\alpha }_t)\sum _{i=1}^{t-1}\frac{{\alpha }_{t-1}^i}{\sqrt{i}}\hfill \\ \hfill & \le \frac{{\alpha }_t}{\sqrt{t}}+(1-{\alpha }_t)\frac{2}{\sqrt{t-1}}\hfill \end{array}$$

All the remaining is to prove that $\frac{{\alpha }_t}{\sqrt{t}}+(1-{\alpha }_t)\frac{2}{\sqrt{t-1}}\le \frac{2}{\sqrt{t}}$, i.e., $\frac{2(\sqrt{t}-\sqrt{t-1})}{2\sqrt{t}-\sqrt{t-1}}\le {\alpha }_t=\frac{1}{t}$.

Clearly, we have

$$\begin{array}{cc}\hfill \frac{2\sqrt{t}-\sqrt{t-1}}{2(\sqrt{t}-\sqrt{t-1})}& =\frac{1}{2}\left(\sqrt{t}+\sqrt{t-1}\right)\left(2\sqrt{t}-\sqrt{t-1}\right)\hfill \\ \hfill & =\frac{1}{2}\left(t+1+\sqrt{t(t-1)}\right)\hfill \\ \hfill & \ge \frac{1}{2}\left(t+1+t-1\right)\hfill \\ \hfill & =t.\hfill \end{array}$$

We have hence proven the right-hand side (upper bound). Followed by the same induction process, the statement on the left-hand side (lower bound) is equivalent to showing $\frac{{\alpha }_t}{\sqrt{t}}+(1-{\alpha }_t)\frac{1}{\sqrt{t-1}}\ge \frac{1}{\sqrt{t}}$. This is trivial, since ${\alpha }_t=\frac{1}{t}.$

   □

By the updated rule for Q in Definition 1, and the definition of the learning rate (7), we have

$$Q_h^k(x,a)={\alpha }_t^0·H+\sum _{i=1}^t{\alpha }_t^i[r_h(x,a)+V_{h+1}^{k_i}\left(x_{h+1}^{k_i}\right)+b_i].$$

(8)

4. Performance Measure Algorithms and Theorems

Next, we present the main algorithm and the main theorem.

Theorem 1.

For any $p\in (0,1)$, if we choose bandit $b_t=c\sqrt{\frac{H^3\iota }{t}}$ where $\iota :=log\left(\frac{K}{p}\right)$ and some absolute constant c (shown later in the analysis), then under certain conditions (shown later in this paper), the Q-learning with UCB exploration algorithm (in Algorithm 1) has the performance measure $\mathit{Regret}\left(K\right)={\sum }_{k=1}^K(V_1^{*}\left(x_1^k\right)-V_1^{{\pi }_k}\left(x_1^k\right))$ bounded above by $\tilde{\mathcal{O}}\left(K^{\frac{d+1}{d+2}}\right)$ with probability $1-p$, where d is the covering dimension of the underlying $\mathcal{S}\times \mathcal{A}$.

We can propose the corresponding main algorithm in the following:

Algorithm 1 Q-learning with UCB bandit in metric space.

1: initialize a covering $\mathcal{B}$ of $\mathcal{S}\times \mathcal{A}$ with ball of $r=1$. Initialize $Q_h(x,a)\leftarrow H$ and $n_h(x,a)\leftarrow 0$ for all $(x,a)$, and $V_h\left(x\right)\leftarrow H$ for all x. Choose $p\in (0,1)$. 2: for $k=1,2,\cdots K$do 3:     receive $x_1$ from randomly sampling. 4:     for $h=1,2,\cdots ,H$ do 5:         if $r\le \frac{c}{8L}\sqrt{\frac{H^3\iota }{1+n_h(x_h,a_h)}}$ is violated then 6:            for the violated ball $B(x_h,a_h)$, $r\leftarrow \frac{r}{2}$, while keeping the same center. 7:            while there exists some region in $\mathcal{S}\times \mathcal{A}$ not covered by $\mathcal{B}$ do 8:                $\mathcal{B}\leftarrow \mathcal{B}\cup B_r(x,a)$ 9:         update $n_h(x_h,a_h)\leftarrow n_h(x_h,a_h)+1$. 10:         take action $a_h\leftarrow {arg\; max}_{a^{\prime }\in \mathcal{A}}{Q}_h(x_h,a^{\prime })$ and observe $x_{h+1}$. 11:         iteratively update $Q_h(x,a)$ and $V_h\left(x\right)$ if $(x,a)$ in $B(x_h,a_h)$: $$Q_h(x_h,a_h)\leftarrow (1-{\alpha }_t)Q_h(x_h,a_h)+{\alpha }_t[r_h(x_h,a_h)+V_{h+1}\left(x_{h+1}\right)+b_t].$$ 12:          $$V_h\left(x_h\right)\leftarrow min\left\{H,\underset{a^{\prime }\in \mathcal{A}}{sup}{Q}_h(x_h,a^{\prime })\right\}.$$

The main theorem is a bound for the cumulative regret in (4).

In the following sections, we prove some very useful techniques to bound the performance measure. We provide the novel and rigorous analysis that can be valuable in the analysis of the Q-learning problem in general.

5. Main Analysis

5.1. Concentration of Measure

First, we use the following lemmas to bound $\left(Q^k-Q^{*}\right)$ with high concentration. Recall that the chosen bandit $b_t=c\sqrt{\frac{H^3\iota }{t}}$. Let ${\beta }_t:={\sum }_{i=1}^t{\alpha }_t^i{b}_i$. The next-step empirical observation for $(x,a)$ at step h in episode k is ${\widehat{\mathcal{P}}}_h^k{V}_h(x,a):=V_{h+1}\left(x_{h+1}^k\right)$.

Lemma 2.

For any $(x,a,h,k)\in \mathcal{S}\times \mathcal{A}\times \left[H\right]\times \left[K\right]$, $t=n_h^k(x,a)$, and suppose $B(x,a)$ was selected previously at step h of episodes ${\left\{k_i\right\}}_{i=1}^t$. We have

$$\begin{array}{cc}\hfill (Q_h^k-Q_h^{*})(x,a)& \hfill \\ \hfill & ={\alpha }_t^0\left(H-Q_h^{*}(x,a)\right)\hfill \\ \hfill & +\sum _{i=1}^t{\alpha }_t^i[\left(V_{h+1}^{k_i}-V_{h+1}^{*}\right)\hfill \\ \hfill & \left(x_{h+1}^{k_i}\right)+\left({\widehat{\mathcal{P}}}_h^{k_i}-{\mathcal{P}}_h\right)\hfill \\ \hfill & V_h^{*}(x,a)+\left({\widehat{\mathcal{P}}}_h^{k_i}{V}_h^{*}(x_h^{k_i},a_h^{k_i})-{\widehat{\mathcal{P}}}_h^{k_i}{V}_h^{*}(x,a)\right)+b_i].\hfill \end{array}$$

(9)

Proof. 

From the Bellman optimality in Equation (2), $Q_h^{*}(x,a)=(r_h+{\mathcal{P}}_h{V}_h^{*})(x,a)$. Using the fact that ${\sum }_{i=0}^t{\alpha }_t^i=1$ from the definition of learning rate, $Q_h^{*}(x,a)={\alpha }_t^0{Q}_h^{*}(x,a)+{\sum }_{i=1}^t{\alpha }_t^i·Q_h^{*}(x,a)={\alpha }_t^0{Q}_h^{*}(x,a)+{\sum }_{i=1}^t{\alpha }_t^i\left[r_h(x,a)+{\mathcal{P}}_h{V}_h^{*}(x,a)\right]$. From (8), $(Q_h^k-Q_h^{*})(x,a)={\alpha }_t^0\left(H-Q_h^{*}(x,a)\right)+{\sum }_{i=1}^t$${\alpha }_t^i\left[V_{h+1}^{k_i}\left(x_{h+1}^{k_i}\right)-{\mathcal{P}}_h{V}_h^{*}(x,a)+b_i\right]$,

$$\begin{array}{cc}\hfill (Q_h^k-Q_h^{*})(x,a)& ={\alpha }_t^0\left(H-Q_h^{*}(x,a)\right)+\sum _{i=1}^t{\alpha }_t^i\left[\left(V_{h+1}^{k_i}-V_{h+1}^{*}\right)\left(x_{h+1}^{k_i}\right)+\left({\widehat{\mathcal{P}}}_h^{k_i}-{\mathcal{P}}_h\right)V_h^{*}(x,a)+b_i\right]\hfill \\ \hfill & +\sum _{i=1}^t{\alpha }_t^i\left[V_{h+1}^{*}\left(x_{h+1}^{k_i}\right)-{\widehat{\mathcal{P}}}_h^{k_i}{V}_h^{*}(x,a)\right].\hfill \end{array}$$

(10)

Note that $x_{h+1}^{k_i}$ is the next step from $\left(x_h^{k_i},a_h^{k_i}\right)$ where $\left(x_h^{k_i},a_h^{k_i}\right)\in B(x,a)$ by the Definition 1. Additionally, $V_{h+1}^{*}\left(x_{h+1}^{k_i}\right)-{\widehat{\mathcal{P}}}_h^{k_i}{V}_h^{*}(x,a)$ satisfies

$$\begin{array}{c}{V}_{h+1}^{*}\left(x_{h+1}^{k_i}\right)-{\widehat{\mathcal{P}}}_h^{k_i}{V}_h^{*}(x,a)\hfill \\ =\left(V_{h+1}^{*}\left(x_{h+1}^{k_i}\right)-{\widehat{\mathcal{P}}}_h^{k_i}{V}_h^{*}\left(x_h^{k_i},a_h^{k_i}\right)\right)+\left({\widehat{\mathcal{P}}}_h^{k_i}{V}_h^{*}\left(x_h^{k_i},a_h^{k_i}\right)-{\widehat{\mathcal{P}}}_h^{k_i}{V}_h^{*}(x,a)\right)\hfill \\ ={\widehat{\mathcal{P}}}_h^{k_i}{V}_h^{*}\left(x_h^{k_i},a_h^{k_i}\right)-{\widehat{\mathcal{P}}}_h^{k_i}{V}_h^{*}(x,a),\hfill \end{array}$$

(11)

where $V_{h+1}^{*}\left(x_{h+1}^{k_i}\right)={\widehat{\mathcal{P}}}_h^{k_i}{V}_h^{*}\left(x_h^{k_i},a_h^{k_i}\right)$ by (1). Combining (10) and (11), we have the statement of Lemma 2.

□

To complete the concentration of $Q^k-Q^{*}$, we bound each term in (9).

Lemma 3.

$\left|{\sum }_{i=1}^t{\alpha }_t^i\left({\widehat{\mathcal{P}}}_h^{k_i}{V}_h^{*}(x,a)-{\widehat{\mathcal{P}}}_h^{k_i}{V}_h^{*}\left(x_h^{k_i},a_h^{k_i}\right)\right)\right|\le \frac{{\beta }_t}{4}$.

Proof. 

Note that ${\beta }_t={\sum }_{i=1}^t{\alpha }_t^i{b}_i={\sum }_{i=1}^t{\alpha }_t^{i}c\sqrt{\frac{H^3\iota }{i}}=c\sqrt{H^3\iota }{\sum }_{i=1}^t\frac{{\alpha }_t^i}{\sqrt{i}}\ge c\sqrt{\frac{H^3\iota }{t}}$. Since ${\left\{\left(x_h^{k_i},a_h^{k_i}\right)\right\}}_{i=1}^t\in B(x,a)$, the distance between $\left(x_h^{k_i},a_h^{k_i}\right)$ and $(x,a)$ is bounded by the radius of the ball. As Algorithm 1 goes on, it keeps refining balls, and hence, the bound will be smaller. By the Lipschitz assumption of $\widehat{\mathcal{P}}{V}^{*}$ in (3), $\Vert {\widehat{\mathcal{P}}}_h^{k_i}{V}_h^{*}(x,a)-{\widehat{\mathcal{P}}}_h^{k_i}{V}_h^{*}(x^{\prime },a^{\prime })\Vert \le L·d_{\mathcal{S}\times \mathcal{A}}\left((x,a),(x^{\prime },a^{\prime })\right)$.

Following from Algorithm 1 and Lemma 1,

$$\begin{array}{c}\hfill \left|\sum _{i=1}^t{\alpha }_t^i\left({\widehat{\mathcal{P}}}_h^{k_i}{V}_h^{*}(x,a)-{\widehat{\mathcal{P}}}_h^{k_i}{V}_h^{*}\left(x_h^{k_i},a_h^{k_i}\right)\right)\right| \le \sum _{i=1}^t{\alpha }_t^i\left(\frac{c}{8}\sqrt{\frac{H^3\iota }{1+n_h^k(x,a)}}\right)\le \frac{{\beta }_t}{4}.\end{array}$$

□

Lemma 4.

For any $p\in (0,1)$, with a probability of at least $1-p$, we have

$$0\le \left(Q_h^k-Q_h^{*}\right)(x,a)\le {\alpha }_t^{0}H+\sum _{i=1}^t{\alpha }_t^i\left(V_{h+1}^{k_i}-V_{h+1}^{*}\right)\left(x_{h+1}^{k_i}\right)+\frac{3}{2}{\beta }_t.$$

The proof of Lemma 4 is similar to Lemma 4.3 in [12].

Proof. 

Based on the definition in (7),

$${\beta }_t=\sum _{i=1}^t{\alpha }_t^i{b}_i\le \sum _{i=1}^t{\alpha }_i{b}_i=\sum _{i=1}^t\frac{1}{i}c\sqrt{\frac{H^3\iota }{i}}<4c\sqrt{\frac{H^3\iota }{t}}.$$

Recall $t=n_h^k(x,a)$. Let $k_i=min\left(\{k\in \left[K\right]|k>k_{i-1}\wedge (x_h^k,a_h^k)\in B(x,a)\}\cup \{K+1\}\right)$. That is, $k_i$ is the episode in which $B(x,a)$ was selected at step h for the $i$th time (or $k_i=K+1$ if it is selected for fewer than i times). The random variable $k_i$ is a stopping time in this case. Let ${\mathcal{F}}_i$ be the $\sigma$-field generated by all the random variables until episode $k_i$ at step h. Then ${\left({𝟙}_{k_i\le K}\left[({\widehat{\mathcal{P}}}_h^{k_i}-{\mathcal{P}}_h)V_h^{*}\right](x,a)\right)}_{i=1}^t$ is a martingale difference sequence with filtration ${\left\{{\mathcal{F}}_i\right\}}_{i\ge 0}$. By the sequence bounded difference, for any i, ${\alpha }_t^i\left[({\widehat{\mathcal{P}}}_h^{k_i}-{\mathcal{P}}_h)V_h^{*}\right](x,a)\le {\alpha }_t^i·H$. Applying the Azuma–Hoeffding inequality, for any $s>0$, we have

$$\mathcal{P}\left(\left|\sum _{i=1}^t{\alpha }_t^i\left[({\widehat{\mathcal{P}}}_h^{k_i}-{\mathcal{P}}_h)V_h^{*}\right](x,a)\right|\le s\right)\ge 1-2exp\left\{\frac{-s^2}{2H^2{\sum }_{i=1}^t{\left({\alpha }_t^i\right)}^2}\right\}.$$

(12)

There exists some absolute constant c, such that $\sqrt{2}·H\sqrt{{\sum }_{i=1}^t{\left({\alpha }_t^i\right)}^2}\sqrt{ln\left(\frac{2H}{p}\right)}\le s\le \frac{c}{4}\sqrt{\frac{H^3\iota }{t}}$. This is true since (1)

$${\beta }_t=\sum _{i=1}^t{\alpha }_t^i{b}_i=\sum _{i=1}^t{\alpha }_t^{i}c\sqrt{\frac{H^3\iota }{i}}=c\sqrt{H^3\iota }\sum _{i=1}^t\frac{{\alpha }_t^i}{\sqrt{i}}\ge c\sqrt{\frac{H^3\iota }{t}}\ge 4s,$$

(13)

and (2). since $s\ge \sqrt{2}·H\sqrt{{\sum }_{i=1}^t{\left({\alpha }_t^i\right)}^2}\sqrt{ln\left(\frac{2H}{p}\right)}$. Then

$$\begin{array}{c}\hfill 1-2exp\left\{\frac{-s^2}{2H^2{\sum }_{i=1}^t{\left({\alpha }_t^i\right)}^2}\right\}\ge 1-\frac{p}{H}>1-p.\end{array}$$

(14)

From (12) and (13), and for any fixed step h, we have

$$\mathcal{P}\left(\left|\sum _{i=1}^t{\alpha }_t^i\left[({\widehat{\mathcal{P}}}_h^{k_i}-{\mathcal{P}}_h)V_h^{*}\right](x,a)\right|\le \frac{{\beta }_t}{4}\right)\ge 1-p.$$

(15)

From (15), Lemmas 2 and 3, we have the right-hand side of Lemma 4.

Now for the left-hand side, we use the backward induction on $h=H,H-1,\cdots ,1$. Observe the initial case $\left(Q_H^k-Q_H^{*}\right)(x,a)\ge 0$. Assume $\left(Q_h^k-Q_h^{*}\right)(x,a)\ge 0$. Since $V_h^k\left(x\right)={sup}_{a^{\prime }\in \mathcal{A}}{Q}_h^k(x,a^{\prime })\ge {sup}_{a^{\prime }\in \mathcal{A}}{Q}_h^{*}(x,a^{\prime })=V_h^{*}\left(x\right)$. From (9) and (14), this implies that $\left(Q_{h-1}^k-Q_{h-1}^{*}\right)(x,a)\ge 0$ with a probability of at least $1-\frac{p}{H}$ for one step backward. Now using the union bound on $h=H,H-1,\cdots ,1$, we have the left-hand side. We hence have for all h with a probability of at least $1-p$, $0\le \left(Q_h^k-Q_h^{*}\right)(x,a)\le {\alpha }_t^{0}H+{\sum }_{i=1}^t{\alpha }_t^i\left(V_{h+1}^{k_i}-V_{h+1}^{*}\right)\left(x_{h+1}^{k_i}\right)+\frac{3}{2}{\beta }_t$.

□

5.2. Assumptions for Concentration

We need two mild and reasonable assumptions shown below. From Lemma 4,

$$\begin{array}{c}\hfill \sum _{k=1}^K\left(Q_h^k-Q_h^{*}\right)(x,a)\le \sum _{k=1}^K{\alpha }_t^{0}H+\sum _{k=1}^K\sum _{i=1}^t{\alpha }_t^i\left(V_{h+1}^{k_i}-V_{h+1}^{*}\right)\left(x_{h+1}^{k_i}\right)+\frac{3}{2}\sum _{k=1}^K{\beta }_t,\end{array}$$

(16)

for any $i=1,\cdots ,n_h^k$ where $n_h^k=t$, each $k_i$ is the actual selection. Here we make the first mild assumption: $V_{h+1}^{k_i}-V_{h+1}^{*}\le R·\sqrt{\frac{H^3\iota }{i}}$ for some large R. This assumption is very mild due to the iterative convergence. Then, $V_{h+1}^{k_i}-V_{h+1}^{*}\le \frac{R}{c}·b_i$. We hence have

$$\sum _{i=1}^t{\alpha }_t^i\left(V_{h+1}^{k_i}-V_{h+1}^{*}\right)\left(x_{h+1}^{k_i}\right)\le \frac{R}{c}\sum _{i=1}^t{\alpha }_t^i{b}_i=\frac{R}{c}·{\beta }_t.$$

(17)

In (16), we have a term ${\sum }_{k=1}^K{\alpha }_t^0·H$. Recall from (7), ${\alpha }_j=\frac{1}{j}$. Since

$${\alpha }_t^0=\prod _{j=1}^{n_k}(1-{\alpha }_j)=\left\{\begin{array}{cc}1\hfill & \mathrm{if}{n}_k=0;\hfill \\ 0\hfill & \mathrm{if}{n}_k\ge 0,\hfill \end{array}\right.$$

(18)

then ${\sum }_{k=1}^K{\alpha }_t^0·H=H{\sum }_{k=1}^K{\prod }_{j=1}^t(1-{\alpha }_j)=H{\sum }_{k=1}^K{𝟙}_{\{n_k=0\}}$. Intuitively, ${𝟙}_{\{n_k=0\}}$ stands for the partitioned balls from the algorithm with no active state-action pair contained. Here, we make the second mild assumption: ${\sum }_{k=1}^K{𝟙}_{\{n_k=0\}}\le \Theta \left(K^{\frac{d+1}{d+2}}\right)$. This assumption is again mild, since we will soon show the cumulative number of partitions ${\mathbf{P}}_K$ up to K is bounded above by $\Theta \left(K^{\frac{d}{d+2}}\right)$ in Lemma 7. Note that here, d is the covering dimension in $\mathcal{S}\times \mathcal{A}$ that will be introduced in the next section.

From the second mild assumption, (16), (17) and Lemma 4, for any $p\in (0,1)$, with a probability of at least $1-p$, we have

$$\begin{array}{cc}\hfill \sum _{k=1}^K\left(Q_h^k-Q_h^{*}\right)(x,a)& \le \sum _{k=1}^K\left({\alpha }_t^0·H+\sum _{i=1}^t{\alpha }_t^i\left(V_{h+1}^{k_i}-V_{h+1}^{*}\right)\left(x_{h+1}^{k_i}\right)+\frac{3}{2}{\beta }_t\right)\hfill \\ \hfill & \le \Theta \left(K^{\frac{d+1}{d+2}}\right)+\mu \sum _{k=1}^K{\beta }_t,\hfill \end{array}$$

(19)

where $\mu =\frac{R}{c}+\frac{3}{2}$. To bound $\left(Q_h^k-Q_h^{*}\right)$, the remaining is to bound ${\beta }_t$.

Note that we also provide some additional approaches that can be valuable in bounding Q-value function in the Appendix A. These approaches can be well applicable given some special conditions satisfied.

5.3. Upper Bound of Cumulative Regret

Now we return to our main discussion about the $\mathtt{Regret}\left(K\right)$. The following lemma shows the relationship between $\mathtt{Regret}\left(K\right)$ and $\left(Q_h^k-Q_h^{*}\right)$. Note that the following approach provides a very general trajectory analysis technique that can be easily applicable to reinforcement learning problems with the continuous state and action space framework.

Lemma 5.

For any $p\in (0,1)$, with a probability of at least $1-p$, the cumulative regret up to K has the following upper bound:

$$\begin{array}{cc}\hfill \sum _{k=1}^K{V}_1^k\left(x_1^k\right)-V_1^{{\pi }_k}\left(x_1^k\right)& \\ \hfill & \le \Theta \left(K^{\frac{d+1}{d+2}}\right)\hfill \\ \hfill & +\sum _{j=0}^{H-1}{\int }_{\mathtt{traj}\left(x_h^k,a_h^k\right)\in \mathtt{TRAJ}(h\to h+j)}\mathcal{P}(\mathtt{traj}\left(x_h^k,a_h^k\right)\hfill \\ \hfill & ·\left(\sum _{k=1}^K\left(Q_{h+j}^k-Q_{h+j}^{*}\right)(s,a^s)\right)d\mathcal{P}.\hfill \end{array}$$

Here, the sequence of actions $\left\{a^s\right\}$ is corresponding to the sequence of states $\left\{s\right\}$, which is the same as maximum selection in (21) and $\mathit{traj}\left(x_h^k,a_h^k\right)$ is the trajectory starting from $\left(x_h^k,a_h^k\right)$. $\mathit{TRAJ}(h\to h+j)$ is a trajectory space that contains all the possible trajectories from step h up to step $h+j$, where d is the covering dimension of $\mathcal{S}\times \mathcal{A}$.

Proof. 

The proof follows the idea from the Ref. [24]. Note that for some action $a_h^k$,

$$V_h^k\left(x_h^k\right)-V_h^{{\pi }_k}\left(x_h^k\right)\le \left(Q_h^k\left(x_h^k,a_h^k\right)-Q_h^{*}\left(x_h^k,a_h^k\right)\right)+\left(Q_h^{*}\left(x_h^k,a_h^k\right)-Q_h^{{\pi }_k}\left(x_h^k,a_h^k\right)\right)+\frac{k^{\frac{-1}{d+2}}}{H}.$$

(20)

The inequality holds because

$$V_h^k\left(x_h^k\right)\le \underset{a^{\prime }\in \mathcal{A}}{sup}{Q}_h^k\left(x_h^k,a^{\prime }\right)\le Q_h^k\left(x_h^k,a_h^k\right)+\frac{k^{\frac{-1}{d+2}}}{H}$$

(21)

for some $a_h^k$ from (6). We name it maximum action selection. Additionally,

$$\begin{array}{cc}\hfill Q_h^{*}\left(x_h^k,a_h^k\right)-Q_h^{{\pi }_k}\left(x_h^k,a_h^k\right)& = \left({\mathcal{P}}_h{V}_h^{*}\left(x_h^k,a_h^k\right)-{\mathcal{P}}_h{V}_h^{{\pi }_k}\left(x_h^k,a_h^k\right)\right)\hfill \\ \hfill & ={\int }_{s\in \mathcal{S}}\mathcal{P}\left(s|x_h^k,a_h^k\right)\left(V_{h+1}^{*}-V_{h+1}^{{\pi }_k}\right)\left(s\right)d\mathcal{P}\left(s\right),\hfill \end{array}$$

(22)

where we take all the states at step $h+1$ into account, $s\sim \mathcal{P}(•|x_h^k,a_h^k)$. Note that from Lemma 4, for any p, we have $Q_h^k\ge Q_h^{*}$ with a probability of at least 1 − p for all steps h. Then with a probability of at least 1 − p, $V_h^k={\sup }_{a\in \mathcal{A}}{Q}_h^k\ge {\sup }_{a\in \mathcal{A}}{Q}_h^{*}=V_h^{*}$. Combining this inequality, (20) and (22), we have

$$V_h^k\left(x_h^k\right)-V_h^{{\pi }_k}\left(x_h^k\right)\le \frac{k^{\frac{-1}{d+2}}}{H}+\left(Q_h^k\left(x_h^k,a_h^k\right)-Q_h^{*}\left(x_h^k,a_h^k\right)\right)+{\int }_{s\in \mathcal{S}}\mathcal{P}\left(s|x_h^k,a_h^k\right)\left(V_{h+1}^k-V_{h+1}^{{\pi }_k}\right)\left(s\right)d\mathcal{P}\left(s\right)$$

(23)

We observe that (23), $\left(V_h^k-V_h^{{\pi }_k}\right)\left(x_h^k\right)$ is on the left-hand side and $\left(V_{h+1}^k-V_{h+1}^{{\pi }_k}\right)\left(s\right)$ is in the integral on the right-hand side. Recall that the space that contains all the possible trajectories from step h up to step $h+j$ is $\mathtt{TRAJ}(h\to h+j)$. We have the following recursive relationship:

$$\begin{array}{cc}\hfill V_h^k\left(x_h^k\right)-V_h^{{\pi }_k}\left(x_h^k\right)& \hfill \\ \hfill & \le \frac{2k^{\frac{-1}{d+2}}}{H}+\left(Q_h^k(x_h^k,a_h^k)-Q_h^{*}(x_h^k,a_h^k)\right)\hfill \\ \hfill & +{\int }_{s\in \mathcal{S}}\mathcal{P}\left(s|x_h^k,a_h^k\right)\hfill \\ \hfill & ((Q_{h+1}^k(s,a^s)-Q_{h+1}^{*}(s,a^s))+{\int }_{s^{\prime }\in \mathcal{S}}\mathcal{P}\left(s^{\prime }|s,a\right)\hfill \\ \hfill & \left(V_{h+2}^k-V_{h+2}^{{\pi }_k}\right)\left(s^{\prime }\right)d\mathcal{P}\left(s^{\prime }\right))\hfill \\ \hfill & d\mathcal{P}\left(s\right)\cdots \hfill \\ \hfill & \le k^{\frac{-1}{d+2}}+\sum _{j=0}^{H-h}{\int }_{\mathtt{traj}\left(x_h^k,a_h^k\right)\in \mathtt{TRAJ}(h\to h+j)}(\mathcal{P}\left(\mathtt{traj}\left(x_h^k,a_h^k\right)\right)\hfill \\ \hfill & ·\left(Q_{h+j}^k-Q_{h+j}^{*}\right)\left(s,a^s\right))d\mathcal{P},\hfill \end{array}$$

(24)

where sequence $\left\{a^s\right\}$ corresponding to each $\left\{s\right\}$ is the same as our selection in (21), and $\mathtt{traj}\left(x_h^k,a_h^k\right)$ is the trajectory starting from $\left(x_h^k,a_h^k\right)$.

For simplicity, we use $\left(Q_{h+j}^k-Q_{h+j}^{*}\right)$ to stand for $\left(Q_{h+j}^k-Q_{h+j}^{*}\right)(s,a^s)$ and $\mathtt{traj}$ to stand for $\mathtt{traj}\left(x_h^k,a_h^k\right)$. Then,

$$\begin{array}{c}\hfill \sum _{k=1}^K{V}_1^k\left(x_1^k\right)-V_1^{{\pi }_k}\left(x_1^k\right)\le \sum _{k=1}^K{k}^{\frac{-1}{d+2}}+\sum _{k=1}^K\sum _{j=0}^{H-1}{\int }_{\mathtt{traj}\in \mathtt{TRAJ}(h\to h+j)}\mathcal{P}\left(\mathtt{traj}\right)·\left(Q_{1+j}^k-Q_{1+j}^{*}\right)d\mathcal{P}.\end{array}$$

(25)

If we apply Fubini’s theorem [25], then (25) is equivalent to

$$\sum _{k=1}^K{V}_1^k\left(x_1^k\right)-V_1^{{\pi }_k}\left(x_1^k\right)\le \Theta \left(K^{\frac{d+1}{d+2}}\right)+\sum _{j=0}^{H-1}{\int }_{\mathtt{traj}\in \mathtt{TRAJ}(h\to h+j)}\mathcal{P}\left(\mathtt{traj}\right)·\left(\sum _{k=1}^K\left(Q_{1+j}^k-Q_{1+j}^{*}\right)\right)d\mathcal{P}.$$

□

6. Regret Bound in Covering Dimension

We have mentioned the covering dimension in the earlier section. For bounding the cumulative regret (performance measure), a covering dimension on the regret bound has been suggested [15,16,26]. The covering dimension is a useful tool when we need to reduce the upper bound of the regret in multi-armed bandit problems. In this section, we will present the analysis of a tight upper bound in the covering dimension. Note that this proof technique is very general and can be easily extended to works where a covering dimension is applicable. We will follow the same definition as in the Refs. [15,16,26].

Definition 2.

Consider a metric space $\left(\mathcal{X},l\right)$. Denote by $N\left(r\right)$ the minimum number of balls with diameter r to cover $\mathcal{X}$. The covering dimension of $\mathcal{X}$ is defined as $COV\left(\mathcal{X}\right)=inf\{d:\exists c\forall r>0N\left(r\right)\le c·r^{-d}\}$.

Lemma 6.

Let $\mathcal{E}$ be a bounded space that can be embedded into an n-dimensional Euclidean space. Then, $\mathit{COV}\left(\mathcal{E}\right)\le n$.

Proof. 

W.L.O.G. we assume $\mathcal{E}\subseteq {[0,1]}^n$. Now, let $r>0$ be any real number, then we can find ${\left(\frac{\sqrt{n}}{r}\right)}^n$ many n-dimensional hypercubes with side-length $\frac{r}{\sqrt{n}}$ covering ${[0,1]}^n$. Now for every such hypercube, there exists a ball with diameter r such that the hypercube is contained inside the ball. Hence, we can have $c·r^{-n}$ many balls with diameter r cover $\mathcal{E}$, where the constant $c={\left(\sqrt{n}\right)}^n$ in this case. Since $\mathtt{COV}\left(\mathcal{E}\right)$ takes the infimum, then $\mathtt{COV}\left(\mathcal{E}\right)\le n$. □

Because of Lemma 6, to obtain a tighter upper bound, we use the covering dimension of $\mathcal{S}\times \mathcal{A}$ instead of the regular Euclidean dimension. In this section, we use the covering dimension to bound ${\sum }_{k=1}^K{\beta }_t$. Assume ${\mathbf{P}}_K$ stands for the number of balls (partitions) created up to episode K in the Algorithm 1.

Lemma 7.

We have the tight bound ${\mathbf{P}}_K\le \Theta \left(K^{\frac{d}{d+2}}\right)$.

Proof. 

Assume d is the covering dimension of $\mathcal{S}\times \mathcal{A}$. Then, the number of balls with diameter m to cover $\mathcal{S}\times \mathcal{A}$ will be bounded by $c·m^{-d}$ for some c. Note that Algorithm 1 produces many balls during adaptive partition. The center of the balls (past active points) of radius $\frac{m}{2}$ will always be in different balls from the covering of $\mathcal{S}\times \mathcal{A}$ because each pair of centers has a distance of at least m between them. For example, say $(s_i,a_i)$ and $(s_j,a_j)$ are two past active centers. Consider the covering of $\mathcal{S}\times \mathcal{A}$. Then, $(s_i,a_i)\in B_i$ and $(s_j,a_j)\in B_j$ for $B_i,B_j$ in the covering, and they are disjoint. Thus, the number of the balls with diameter m from Algorithm 1 is bounded by $c·m^{-d}$, the covering number.

Note that for each ball B of radius r with the center inside $\mathcal{S}\times \mathcal{A}$, we can use at most $c^{\prime }·2^n$ balls of radius $\frac{r}{2}$ to cover $B\cap (\mathcal{S}\times \mathcal{A})$, where n is the Euclidean dimension of $\mathcal{S}\times \mathcal{A}$ and $c^{\prime }$ is some constant. From the algorithm, each ball will split only if $r\le c\sqrt{\frac{H^3\iota }{1+n_k}}$ is violated. That is, for any ball of radius $\frac{r}{2}$ split from the original ball of radius r (condition violated), it must be filled by $c^2·\frac{H^3\iota }{r^2}$ past active points. Since there are K points in total, then the number of balls of radius r violated is at most $\frac{K}{c^2·\frac{H^3\iota }{r^2}}=\frac{1}{c^2{H}^3\iota }·r^{2}K$. Thus, after splitting (violation), the number of balls of radius $\frac{r}{2}$ is at most $\left(c^{\prime }·2^n\right)·\left(\frac{1}{c^2{H}^3\iota }·r^{2}K\right)=c^{″}·r^{2}K$, where $c^{″}=c^{\prime }·2^n·\frac{1}{c^2{H}^3\iota }$.

Along the algorithm, as radius r becomes $\frac{r}{2}$, the number of partitioned balls is bounded by the covering number, which is $c·r^{-d}$, and also bounded by $c^{″}·r^{2}K$ for any $r>0$ from the above analysis. Consider $r=1,\frac{1}{2},\frac{1}{4},\cdots ,2^{-i}\cdots$, where we have ${\mathbf{P}}_K={\sum }_{i=1}^{\infty }min\left\{c{\left(2^{-i}\right)}^{-d},c^{″}{\left(2^{-i}\right)}^{2}K\right\}$.

We split this term into two cases. (1). For a large radius, use $c·r^{-d}$ and (2). For a very small radius, use the bound $c^{″}·r^{2}K$. To maximize the upper bound, we find some optimal J where $c{\left(2^{-J}\right)}^{-d}=c·2^{J·d}=c^{″}·2^{-2J}·K$. Thus, $2^{J·d}=\Theta \left(K^{\frac{d}{d+2}}\right)$, where $D=\frac{c^{″}}{c}$. This implies that ${\mathbf{P}}_K\le {\sum }_{i=1}^{J}c{\left(2^{-i}\right)}^{-d}+{\sum }_{i=J+1}^{\infty }{c}^{″}{\left(2^{-i}\right)}^{2}K\le 2c·2^{J·d}+8c^{″}·2^{-2(J+1)}K=\Theta \left(K^{\frac{d}{d+2}}\right)$.

□

Lemma 8.

We have the following tight upper bound ${\sum }_{k=1}^K{\beta }_t\le \tilde{\mathcal{O}}\left(K^{\frac{d+1}{d+2}}\right)$, where d is the covering dimension of $\mathcal{S}\times \mathcal{A}$.

Proof. 

Recall that ${\beta }_t={\sum }_{i=1}^t{\alpha }_t^i{b}_i\le {\sum }_{i=1}^t{\alpha }_i{b}_i={\sum }_{i=1}^t\frac{1}{i}c\sqrt{\frac{H^3\iota }{i}}<4c\sqrt{\frac{H^3\iota }{t}}$. Up to episode K, we create ${\mathbf{P}}_K$ many balls in total. Assume there are $b_i$ past active state-action pairs in the $i^{th}$ ball. Clearly, ${\sum }_{i=1}^{{\mathbf{P}}_K}{b}_i=K$. Thus,

$$\begin{array}{cc}\hfill \sum _{k=1}^K{\beta }_t& \le \sum _{k=1}^{K}4c\sqrt{\frac{H^3\iota }{t}}\le \sum _{i=1}^{{\mathbf{P}}_K}\sum _{j=1}^{b_i}4c\sqrt{\frac{H^3\iota }{j}}\hfill \\ \hfill & \le \sum _{i=1}^{{\mathbf{P}}_K}4c\sqrt{H^3\iota }·2\sqrt{b_i}=\sum _{i=1}^{{\mathbf{P}}_K}8c\sqrt{H^3\iota }·\sqrt{b_i}\hfill \end{array}$$

(26)

$$\begin{array}{cc}\hfill & \le 8c\sqrt{H^3\iota }·{\mathbf{P}}_K\sqrt{\sum _{i=1}^{{\mathbf{P}}_K}\frac{1}{{\mathbf{P}}_K}·b_i}\hfill \end{array}$$

(27)

$$\begin{array}{cc}\hfill & =8c\sqrt{H^3\iota }·{\mathbf{P}}_K\sqrt{\frac{K}{{\mathbf{P}}_K}}\hfill \end{array}$$

(28)

$$\begin{array}{cc}\hfill & =8c\sqrt{H^3\iota }·\sqrt{K}\sqrt{{\mathbf{P}}_K},\hfill \end{array}$$

(29)

where (26) is from Jensen’s inequality on the concave function $\sqrt{x}$, and (27) is from ${\sum }_{i=1}^{{\mathbf{P}}_K}{b}_i=K$. With (28), we have ${\sum }_{k=1}^K{\beta }_t\le \tilde{\mathcal{O}}\left(K^{\frac{d+1}{d+2}}\right)$.

□

7. Conclusions of Main Theorem

7.1. Proof of Main Theorem

We can combine the above results to complete the proof of the main theorem of this paper, Theorem 1. Recall that from Lemma 5, the cumulative regret up to episode K is given by

$$\sum _{k=1}^K{V}_1^k\left(x_1^k\right)-V_1^{{\pi }_k}\left(x_1^k\right)\le \Theta \left(K^{\frac{d+1}{d+2}}\right)+\sum _{j=0}^{H-1}{\int }_{\mathtt{traj}\in \mathtt{TRAJ}(h\to h+j)}\mathcal{P}\left(\mathtt{traj}\right)·\sum _{k=1}^K\left(Q_{1+j}^k-Q_{1+j}^{*}\right)d\mathcal{P}.$$

(30)

In addition, from (19), ${\sum }_{k=1}^K\left(Q_h^k-Q_h^{*}\right)(x,a)\le \Theta \left(K^{\frac{d+1}{d+2}}\right)+\mu {\sum }_{k=1}^K{\beta }_t$. Combined with Lemma 8, we have ${\sum }_{k=1}^K(Q_h^k-Q_h^{*})\le \tilde{\mathcal{O}}\left(K^{\frac{d+1}{d+2}}\right)$, where d is the covering dimension of $\mathcal{S}\times \mathcal{A}$. Thus, ${max}_{h\le H}{\sum }_{k=1}^K\left(Q_h^k-Q_h^{*}\right)\le \tilde{\mathcal{O}}\left(K^{\frac{d+1}{d+2}}\right)$. Inequality (30) becomes

$$\begin{array}{cc}\hfill \sum _{k=1}^K{V}_1^k\left(x_1\right)-V_1^{{\pi }_k}\left(x_1\right)& \\ \hfill & \le \Theta \left(K^{\frac{d+1}{d+2}}\right)+\sum _{j=0}^{H-1}{\int }_{\mathtt{traj}\in \mathtt{TRAJ}(h\to h+j)}\mathcal{P}\left(\mathtt{traj}\right)\hfill \\ \hfill & ·\left(\underset{j}{max}\sum _{k=1}^K\left(Q_{1+j}^k-Q_{1+j}^{*}\right)\right)d\mathcal{P}\hfill \\ \hfill & \le \Theta \left(K^{\frac{d+1}{d+2}}\right)+\sum _{j=0}^{H-1}{\int }_{\mathtt{traj}\in \mathtt{TRAJ}(h\to h+j)}\mathcal{P}\left(\mathtt{traj}\right)\hfill \\ \hfill & ·\tilde{\mathcal{O}}\left(K^{\frac{d+1}{d+2}}\right)d\mathcal{P}\hfill \\ \hfill & =\tilde{\mathcal{O}}\left(K^{\frac{d+1}{d+2}}\right).\hfill \end{array}$$

This completes Theorem 1, which is the main scope of this paper.

7.2. Discussion and Conclusions

Some applications in discrete space include board games like poker, bridge, and chess. For example, DeepMind${}^{’}$s powerful computer program AlphaGo Zero has its self-learning algorithm based on a discrete space set up in Go [27]. AlphaGo demonstrated super-human capabilities in navigating through an exhaustive discrete action search space and achieved a breakthrough of AI in challenging gaming. On the other hand, most robotics and automated vehicle applications of reinforcement learning require continuous state spaces which utilize continuous variables, such as position, velocity, torque, and so forth. For example, motion estimation and depth estimation in the perception tasks of autonomous driving are in the continuous environment [28]. If reinforcement learning algorithms with a continuous state and action space environment can be efficiently implemented, then these applications, such as robotics and automated vehicles, can potentially be improved.

In this paper, to accommodate the needs of the continuous environment, we proposed a state-of-the-art Q-learning algorithm with an UCB exploration policy in the compact state-action metric space. Our algorithm adaptively partitioned the state-action space and flexibly selected the future actions to maximize Q-values with bandit exploration bonuses. By some mild assumptions for concentration, we found a tight upper bound with respect to covering dimension for the performance measure in a general state-action metric space. Our work contained rigorous techniques, including novel combinatorial and probabilistic analysis. These approaches are not limited to the analysis of performance measures in Q-learning, but are also valuable to general reinforcement learning analysis. This work suggests that the bandit strategy is an effective way to increase data efficiency for a general model-free reinforcement learning problem. Our analysis is easily applicable to reinforcement learning problems with the continuous state and action space framework. For future steps, we will apply our algorithms to various practices, including robotics and automated vehicles.