PAC-Bayes Unleashed: Generalisation Bounds with Unbounded Losses

Abstract

We present new PAC-Bayesian generalisation bounds for learning problems with unbounded loss functions. This extends the relevance and applicability of the PAC-Bayes learning framework, where most of the existing literature focuses on supervised learning problems with a bounded loss function (typically assumed to take values in the interval [0;1]). In order to relax this classical assumption, we propose to allow the range of the loss to depend on each predictor. This relaxation is captured by our new notion of HYPothesis-dependent rangE (HYPE). Based on this, we derive a novel PAC-Bayesian generalisation bound for unbounded loss functions, and we instantiate it on a linear regression problem. To make our theory usable by the largest audience possible, we include discussions on actual computation, practicality and limitations of our assumptions.

1. Introduction

Since its emergence in the late 1990s, the PAC-Bayes theory (see the seminal works of [1,2,3], the recent survey by [4] and work by [5]) has been a powerful tool to obtain generalisation bounds and to derive efficient learning algorithms. Generalisation bounds are helpful for understanding how a learning algorithm may perform on future similar batches of data. While the classical generalization bounds typically address the performance of individual predictors from a given hypothesis class, PAC-Bayes bounds typically address a randomized predictor defined by a distribution over the hypothesis class.

PAC-Bayes bounds were originally meant for binary classification problems [6,7,8], but the literature now includes many contributions involving any bounded loss function (without loss of generality, with values in $[0;1]$), not just the binary loss. Our goal is to provide new PAC-Bayes bounds that are valid for unbounded loss functions, and thus extend the usability of PAC-Bayes to a much larger class of learning problems. To do so, we reformulate the general PAC-Bayes theorem of [9] and use it as basic building block to derive our new PAC-Bayes bound.

Some ways to circumvent the bounded range assumption on the losses have been explored in the recent literature. For instance, one approach consists of assuming a tail decay rate on the loss, such as sub-gaussian or sub-exponential tails [10,11]; however, this approach requires the knowledge of additional parameters. Some other works have also looked into the analysis for heavy-tailed losses, e.g., ref. [12] proposed a polynomial moment-dependent bound with f-divergences, while [13] devised an exponential bound that assumes the second (uncentered) moment of the loss is bounded by a constant (with a truncated risk estimator, as recalled in Section 4 below). A somewhat related approach was explored by [14], who do not assume boundedness of the loss, but instead control higher-order moments of the generalization gap through the Efron-Stein variance proxy. See also [5].

We investigate a different route here. We introduce the HYPothesis-dependent rangE (HYPE) condition, which means that the loss is upper-bounded by a term that depends on the chosen predictor (but does not depend on the data). Thus, effectively, the loss may have an arbitrarily large range. The HYPE condition allows us to derive an upper bound on the exponential moment of a suitably chosen functional, which, combined with the general PAC-Bayes theorem, leads to our new PAC-Bayes bound. To illustrate it, we instantiate the new bound on a linear regression problem, which additionally serves the purpose of illustrating that our HYPE condition is easy to verify in practice, given an explicit formulation of the loss function. In particular, we shall see in the linear regression setting that a mere use of the triangle inequality is enough to check the HYPE condition. The technical assumptions on which our results are based are comparable to those of the classical PAC-Bayes bounds; we state them in full detail, with discussions, for the sake of clarity and to make our work accessible.

Our contributions are twofold. (i) We propose PAC-Bayesian bounds holding with unbounded loss functions, therefore overcoming a limitation of the mainstream PAC-Bayesian literature for which a bounded loss is usually assumed. (ii) We analyse the bound, its implications, limitations of our assumptions, and their usability by practitioners. We hope this will extend the PAC-Bayes framework into a widely usable tool for a significantly wider range of problems, such as unbounded regression or reinforcement learning problems with unbounded rewards.

Outline.Section 2 introduces our notation and definition of the HYPE condition and provides a general PAC-Bayesian bound, which is valid for any learning problem complying with a mild assumption. For the sake of completeness, we present how our approach (designed for the unbounded case) behaves in the bounded case (Section 3). This section is not the core of our work, but rather serves as a safety check and particularises our bound to more classical PAC-Bayesian assumptions. We also provide numerical experiments. Section 4 introduces the notion of softening functions and particularises Section 2’s PAC-Bayesian bound. In particular, we make explicit all terms in the right-hand side. Section 5.1 extends our results to linear regression (which has been studied from the perspective of PAC-Bayes in the literature, most recently by [15]). We also experimentally illustrate the behaviour of our bound. Finally, Section 6 presents, in detail, related works and Section 7 contains all proofs of the original claims we make in the paper.

2. Framework and Preliminary Results

The learning problem is specified by three variables $(\mathcal{H},\mathcal{Z},\ell )$ consisting of a set $\mathcal{H}$ of predictors, the data space $\mathcal{Z}$, and a loss function $\ell :\mathcal{H}\times \mathcal{Z}\to {\mathbb{R}}^{+}$.

For a given positive integer m, we consider size-m datasets. The space of all possible datasets of this fixed size is $\mathcal{S}={\mathcal{Z}}^m$; an arbitrary element of this space is $s=(z_1,\dots ,z_m)$. We denote S as a random dataset: $S=(Z_1,\dots ,Z_m)$ where the random data points $Z_i$ are independent and sampled from the same distribution $\mu$ over $\mathcal{Z}$. We call $\mu$ the data-generating distribution. The assumption that the $Z_i$’s are independent and identically distributed is typically called the i.i.d. data assumption. It means that the random sample S (of size m) has distribution ${\mu }^{\otimes m}$ which is the product of m copies of $\mu$.

For any predictor $h\in \mathcal{H}$, we define the empirical risk of h over a sample s, denoted $R_s\left(h\right)$, and the theoretical risk of h, denoted $R\left(h\right)$, as:

$$R_s\left(h\right)=\frac{1}{m}\sum _{i=1}^m\ell (h,z_i)\mathrm{and}R\left(h\right)={\mathbb{E}}_{\mu }[\ell (h,Z)]$$

respectively, where ${\mathbb{E}}_{\mu }[\ell (h,Z)]$ denotes the expectation with respect to $Z\sim \mu$. Finally, we define the risk gap ${\Delta }_s\left(h\right)=R\left(h\right)-R_s\left(h\right)$ for any $h\in \mathcal{H}$ and $s\in \mathcal{S}$. Often, ${\Delta }_s\left(h\right)$ is referred to as the generalisation gap.

Notice that for a random dataset S, the empirical risk $R_S\left(h\right)$ is random, with expected value ${\mathbb{E}}_{{\mu }^{\otimes m}}\left[R_S\left(h\right)\right]=R\left(h\right)$, where ${\mathbb{E}}_{{\mu }^{\otimes m}}$ the expectation under the distribution of the random sample S.

In general, ${\mathbb{E}}_{\mu }[·]$ denotes an expectation under the distribution $\mu$. When we want to emphasize the role of the random variable $Z\sim \mu$ we write ${\mathbb{E}}_Z[·]$ or ${\mathbb{E}}_{Z\sim \mu }[·]$ instead of ${\mathbb{E}}_{\mu }[·]$. We use a similar convention for expectations related to any other distributions and random quantities. We now introduce the key concept to our analysis.

Definition 1.

(HYPE). A loss function $\ell :\mathcal{H}\times \mathcal{Z}\to {\mathbb{R}}^{+}$ is said to satisfy the hypothesis-dependent range (HYPE) condition if there exists a function $K:\mathcal{H}\to {\mathbb{R}}^{+}\backslash \left\{0\right\}$ such that ${sup}_{z\in \mathcal{Z}}\ell (h,z)\le K\left(h\right)$ for every predictor h. We then say that ℓ is HYPE(K) compliant.

Let ${\mathcal{M}}_1^{+}\left(\mathcal{H}\right)$ be the set of probability distributions on $\mathcal{H}$. We assume that all considered probability measures on $\mathcal{H}$ are defined on a fixed $\sigma$-algebra over $\mathcal{H}$, while the notation ${\mathcal{M}}_1^{+}\left(\mathcal{H}\right)$ hides the $\sigma$-algebra, for simplicity. For $P,P^{\prime }\in {\mathcal{M}}_1^{+}\left(\mathcal{H}\right)$, the notation $P^{\prime }\ll P$ indicates that $P^{\prime }$ is absolutely continuous with respect to P (i.e., $P^{\prime }\left(A\right)=0$ if $P\left(A\right)=0$ for measurable $A\subset \mathcal{H}$). We write $P^{\prime }\sim P$ to indicate that $P^{\prime }\ll P$ and $P\ll P^{\prime }$, i.e., these two distributions are absolutely continuous with respect to each other.

We now recall a result from Germain et al. [9]. Note that while implicit in many PAC-Bayes works (including theirs), we make it explicit that both the prior P and the posterior Q must be absolutely continuous with respect to each other. We discuss this restriction below.

Theorem 1.

(Adapted from [9], Theorem 2.1.) For any $P\in {\mathcal{M}}_1^{+}\left(\mathcal{H}\right)$ with no dependency on data, for any function $F:{\mathbb{R}}^{+}\times {\mathbb{R}}^{+}\to \mathbb{R}$, define the exponential moment:

$$\chi :={\mathbb{E}}_S{\mathbb{E}}_{h\sim P}\left[e^{F(R_S\left(h\right),R\left(h\right))}\right].$$

If F is convex, then for any $\delta \in [0;1]$, with probability of at least $1-\delta$ over random samples S, simultaneously for all $Q\in {\mathcal{M}}_1^{+}\left(\mathcal{H}\right)$ such that $Q\sim P$ we have:

$$\begin{array}{c}\hfill F\left({\mathbb{E}}_{h\sim Q}\left[R_S\left(h\right)\right],{\mathbb{E}}_{h\sim Q}\left[R\left(h\right)\right]\right)\le KL\left(Q\right|\left|P\right)+log\left(\frac{\chi }{\delta }\right).\end{array}$$

The proof is deferred to Section 7.1. Note that the proof in [9] requires that $P\ll Q$, although it is not explicitly stated; we highlight this in our own proof. While $Q\ll P$ is classical and necessary for the $KL\left(Q\right|\left|P\right)$ to be meaningful, $P\ll Q$ appears to be more restrictive. In particular, we have to choose Q such that it has the exact same support as P (e.g., choosing a Gaussian and a truncated Gaussian is not possible). However, we can still apply our theorem when P and Q belong to the same parametric family of distributions, e.g., both ‘full-support’ Gaussian or Laplace distributions, but these are just two examples and there are many others.

Note that Alquier et al. [10] (Theorem 4.1) adapted a result from Catoni [8], which only requires $Q\ll P$. This comes at the expense of what Alquier et al. [10] (Definition 2.3) called a Hoeffding’s assumption, which means that the exponential moment $\chi$ is assumed to be bounded by a function depending only on the hyperparameters (such as the dataset size m or parameters given by Hoeffding’s assumption). Our analysis does not require this assumption, which might prove restrictive in practice.

Theorem 1 may be seen as a basis to recover many classical PAC-Bayesian bounds. For instance, $F(x,y)=2m{(x-y)}^2$, recovers McAllester’s bound as recalled in [4] (Theorem 1). To get a usable bound, the outstanding task is to bound the exponential moment $\chi$. Note that a previous attempt has been made in [11], as described in Section 6.1 below. Furthermore, under the assumption that the distribution P has no dependency on the data, we may swap the order of integration in the exponential moment thanks to Fubini-Tonelli’s theorem and the positiveness of the exponential:

$$\chi ={\mathbb{E}}_{h\sim P}{\mathbb{E}}_S\left[e^{F(R_S\left(h\right),R\left(h\right))}\right].$$

This is the starting point for the way that the exponential moment was handled in several works in the PAC-Bayes literature. Essentially, for a fixed h, one may upper-bound the innermost expectation (with respect to S) using standard exponential moment inequalities.

In this work, we will use Theorem 1 with $F(x,y)=m^{\alpha }D(x,y)$, where $\alpha >0$, and $D:{\mathbb{R}}^{+}\times {\mathbb{R}}^{+}\to \mathbb{R}$ is a convex function. In this case, the high-probability inequality of the theorem takes the form:

$$\begin{array}{c}D\left({\mathbb{E}}_{h\sim Q}\left[R_S\left(h\right)\right],{\mathbb{E}}_{h\sim Q}\left[R\left(h\right)\right]\right)\le \hfill \\ \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\frac{1}{m^{\alpha }}\left(KL\left(Q\right|\left|P\right)+log\left(\frac{1}{\delta }{\mathbb{E}}_{h\sim P}{\mathbb{E}}_S{e}^{m^{\alpha }D(R_S\left(h\right),R\left(h\right))}\right)\right).\hfill \end{array}$$

(1)

Our goal is to control ${\mathbb{E}}_S{e}^{m^{\alpha }D(R_S\left(h\right),R\left(h\right))}$ for a fixed h, when $D(x,y)=y-x$. This will readily give us control on the exponential moment $\chi$. To do so, we propose the following theorem:

Theorem 2.

Let $h\in \mathcal{H}$ be a fixed predictor and $\alpha \in \mathbb{R}$. If the loss function ℓ is HYPE(K) compliant, then for ${\Delta }_S\left(h\right)=R\left(h\right)-R_S\left(h\right)$ we have:

$${\mathbb{E}}_S\left[e^{m^{\alpha }{\Delta }_S\left(h\right)}\right]\le exp\left(\frac{K{\left(h\right)}^2}{2m^{1-2\alpha }}\right).$$

Proof. 

Let $h\in \mathcal{H}$. Then:

$$\begin{array}{cc}\hfill {\mathbb{E}}_S\left[e^{m^{\alpha }{\Delta }_S\left(h\right)}\right]& =\mathbb{E}\left[exp\left(m^{\alpha -1}\sum _{i=1}^m(l(h,Z_i)-R\left(h\right))\right)\right]\hfill \\ \hfill & =\mathbb{E}\left[\prod _{i=1}^{m}exp\left(m^{\alpha -1}\left(\ell (h,Z_i)-R\left(h\right)\right)\right)\right]\hfill \\ \hfill & =\prod _{i=1}^m\mathbb{E}\left[exp\left(m^{\alpha -1}\left(\ell (h,Z_i)-R\left(h\right)\right)\right)\right].\hfill \end{array}$$

We now apply Hoeffding’s lemma, for any $i\in \{1..m\}$, the random (in $Z_i$) variable $\ell (h,Z_i)-R\left(h\right)$ is centered, taking values in $[-K(h);K(h\left)\right]$, so that:

$$\mathbb{E}\left[exp\left(m^{\alpha -1}(\ell (h,Z_i)-R\left(h\right))\right)\right]\le exp\left(m^{2\alpha -2}\frac{4K{\left(h\right)}^2}{8}\right)$$

and finally:

$${\mathbb{E}}_S\left[e^{m^{\alpha }{\Delta }_S\left(h\right)}\right]\le \prod _{i=1}^{m}exp\left(m^{2\alpha -2}\frac{4K{\left(h\right)}^2}{8}\right)=exp\left(\frac{K{\left(h\right)}^2}{2m^{1-2\alpha }}\right).$$

 □

The strength of this result lies in the fact that $\frac{K{\left(h\right)}^2}{m^{1-2\alpha }}$, is a decreasing factor in m, when $\alpha \le 1/2$, and more generally, one can control how fast the exponential moment will explode when m grows by the choice of the hyperparameter $\alpha$.

For convenient cross-referencing, we state the following rewriting of Theorem 1.

Theorem 3.

Let the loss ℓ be HYPE(K) compliant. For any $P\in {\mathcal{M}}_1^{+}\left(\mathcal{H}\right)$ with no data dependency, for any $\alpha \in \mathbb{R}$ and for any $\delta \in [0;1]$, with probability of at least $1-\delta$ over size-m random samples S, simultaneously for all Q such that $Q\sim P$ we have:

$${\mathbb{E}}_{h\sim Q}\left[R\left(h\right)\right]\le {\mathbb{E}}_{h\sim Q}\left[R_S\left(h\right)\right]+\frac{1}{m^{\alpha }}\left(KL\left(Q\right|\left|P\right)+log\frac{{\mathbb{E}}_{h\sim P}\left[exp\left(\frac{K{\left(h\right)}^2}{2m^{1-2\alpha }}\right)\right]}{\delta }\right).$$

Proof. 

We first apply Theorem 1 with $F(x,y)=m^{\alpha }(y-x)$. More precisely, we use Equation (1) with $D(x,y)=y-x$. We then conclude with Theorem 2.  □

3. Safety Check: The Bounded Loss Case

3.1. Theoretical Results

At this stage, the reader might wonder whether this new approach allows for the recovery of known results in the bounded case: the answer is yes.

In this section, we study the case where ℓ is bounded by some constant $C\in {\mathbb{R}}^{+}\backslash \left\{0\right\}$. In other words, we consider the case that ${sup}_h{sup}_z\ell (h,z)\le C$. We provide a bound, valid for any choice of “priors” P and “posteriors” Q such that $P\sim Q$, which is an immediate corollary of Theorem 3.

Proposition 1.

Let ℓ be HYPE(K) compliant, with $K\left(h\right)=C$ constant, and let $\alpha \in \mathbb{R}$. Let $P\in {\mathcal{M}}_1^{+}\left(\mathcal{H}\right)$ be a distribution with no data dependency. Then, for any $\delta \in [0;1]$, with probability of at least $1-\delta$ over random m-samples S, simultaneously for all $Q\in {\mathcal{M}}_1^{+}\left(\mathcal{H}\right)$ such that $Q\sim P$ we have:

$${\mathbb{E}}_{h\sim Q}\left[R\left(h\right)\right]\le {\mathbb{E}}_{h\sim Q}\left[R_S\left(h\right)\right]+\frac{KL\left(Q\right|\left|P\right)+log(1/\delta )}{m^{\alpha }}+\frac{C^2}{2m^{1-\alpha }}.$$

Remark 1.

We provide Proposition 1 to evaluate the robustness of our approach. For instance, by comparing it with the PAC-Bayesian bound found in Germain et al. [11]. This discussion can be found in Section 6.1, where the bound from Germain et al. [11] is presented in detail.

Remark 2.

At first glance, a naive remark: in order to control the rate of convergence of all the terms of the bound in Proposition 1 (as is often the case in classical PAC-Bayesian bounds), then the only case of interest is in fact $\alpha =\frac{1}{2}$. However, one could notice that the factor $C^2$ is not optimisable, while the KL is. In this way, if it appears that $C^2$ is too big, in practice, one wants to have the ability to attenuate its influence as much as possible and this may lead us to consider $\alpha <1/2$. The following lemma answers this question.

Lemma 1.

For any given $K_1>0$, the function $f_{K_1}\left(\alpha \right):=\frac{K_1}{m^{\alpha }}+\frac{C^2}{m^{1-\alpha }}$ reaches its minimum at

$${\alpha }_0=\frac{1}{2}+\frac{1}{2log\left(m\right)}log\left(\frac{2K_1}{C^2}\right).$$

Proof. 

The explicit calculus of the $f_{K_1}^{{}^{\prime }}$ and the resolution of $f_{K_1}^{{}^{\prime }}\left(\alpha \right)=0$ provides the result.  □

Remark 3.

Lemma 1 indicates that with a fixed “prior” P and “posterior” Q, taking $K_1=KL\left(Q\right|\left|P\right)+log(1/\delta )$, gives the optimised value of the bound in Proposition 1. We numerically show in Section 3.2 (first experiment there) that optimising α leads to significantly better results.

Now the only remaining question is how to optimise the KL divergence. To do so, we may need to fix an “informed prior” to minimise the KL divergence with an interesting posterior. This idea has been studied by [16,17] and, more recently, by Mhammedi et al. [18], Rivasplata et al. [5], among others. We will adapt it to our problem in the simplest way.

We now introduce some additional notation. For a sample $s=(z_1,\dots ,z_m)$ and $k\in \{1..m\}$, we define $s_{\le k}:=\{z_1,\dots ,z_k\}$ and $s_{>k}:=\{z_{k+1},\dots ,z_m\}$. Then, similarly, for a random sample S, we have the splits $S_{\le k}$ and $S_{>k}$.

Proposition 2.

Let ℓ be HYPE(K) compliant, with constant $K\left(h\right)=C$, and ${\alpha }_1,{\alpha }_2\in \mathbb{R}$. Consider any “priors” $P_1\in {\mathcal{M}}_1^{+}\left(\mathcal{H}\right)$ (possibly dependent on $S_{>m/2}$) and $P_2\in {\mathcal{M}}_1^{+}\left(\mathcal{H}\right)$ (possibly dependent on $S_{\le m/2}$). Then, for any $\delta \in [0;1]$, with probability of at least $1-\delta$ over random size-m samples S, simultaneously for all $Q\in {\mathcal{M}}_1^{+}\left(\mathcal{H}\right)$ such that $Q\sim P_1$ and $Q\sim P_2$ we have:

$$\begin{array}{c}\hfill {\mathbb{E}}_{h\sim Q}\left[R\left(h\right)\right]\le {\mathbb{E}}_{h\sim Q}\left[R_S\left(h\right)\right]+\frac{1}{2}\left(\frac{KL\left(Q\right|\left|P_1\right)+log(2/\delta )}{{(m/2)}^{{\alpha }_1}}+\frac{C^2}{2{(m/2)}^{1-{\alpha }_1}}\right)\\ \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{0.17em}+\frac{1}{2}\left(\frac{KL\left(Q\right|\left|P_2\right)+log(2/\delta )}{{(m/2)}^{{\alpha }_2}}+\frac{C^2}{2{(m/2)}^{1-{\alpha }_2}}\right).\hfill \end{array}$$

Proof. 

Let $P_1,P_2,Q$ be as stated in Proposition 2. We first notice that by using Proposition 1 on the two halves of the sample, we obtain, with a probability of at least $1-\delta /2$:

$${\mathbb{E}}_{h\sim Q}\left[R\left(h\right)\right]\le {\mathbb{E}}_{h\sim Q}\left[\frac{1}{m/2}\sum _{i=1}^{m/2}\ell (h,Z_i)\right]+\frac{KL\left(Q\right|\left|P_1\right)+log(2/\delta )}{{(m/2)}^{{\alpha }_1}}+\frac{C^2}{2{(m/2)}^{1-{\alpha }_1}}$$

and also with probability at least $1-\delta /2$:

$${\mathbb{E}}_{h\sim Q}\left[R\left(h\right)\right]\le {\mathbb{E}}_{h\sim Q}\left[\frac{1}{m/2}\sum _{i=1}^{m/2}\ell (h,Z_{m/2+i})\right]+\frac{KL\left(Q\right|\left|P_2\right)+log(2/\delta )}{{(m/2)}^{{\alpha }_2}}+\frac{C^2}{2{(m/2)}^{1-{\alpha }_2}}.$$

Hence, with a probability of at least $1-\delta$, both inequalities hold, and the result follows by adding them and dividing by 2. □

Remark 4.

One can notice that the main difference between Proposition 2 and Proposition 1 lies in the implicit PAC-Bayesian paradigm that our priors must not depend on the data. With this last proposition, we implicitly allow $P_1$ to depend on $S_{>m/2}$ and $P_2$ on $S_{\le m/2}$, which can in practice lead to far more accurate priors. We numerically show this fact in Section 3.2’s second experiment. Note that this idea is not new and has been studied, for instance, in [19] for the specific case of SVMs.

3.2. Numerical Experiments

Our experimental framework has been inspired by the work of [18].

Settings. We generate synthetic data for classification, and we are using the 0–1 loss. The data space is $\mathcal{Z}=\mathcal{X}\times \mathcal{Y}={\mathbb{R}}^d\times \{0,1\}$ with $d\in \mathbb{N}$. The set of predictors $\mathcal{H}$ is parameterised with d-dimensional ‘weight’ vectors: $\mathcal{H}=\{h_w:\mathcal{X}\to \mathcal{Y}|w\in {\mathbb{R}}^d\}$. For simplicity, we identify $h_w$ with w and we also identify the space $\mathcal{H}$, with the weight space $\mathcal{W}={\mathbb{R}}^d$. For $z=(x,y)\in \mathcal{Z}$ and $w\in \mathcal{W}$, we define the loss as $\ell (w,z):=|𝟙\left\{\varphi \left(w^{\top }x\right)>1/2\right\}-y|$, where $\varphi \left(r\right)=\frac{1}{1+e^{-r}}$. We want to learn an optimised predictor given a dataset $S={\left(Z_i\right)}_{i=1..m}$ where $Z_i=(X_i,Y_i)$. To do so, we use regularised logistic regression and compute:

$$\begin{array}{c}\hfill \widehat{w}\left(S\right):=\arg \underset{w\in \mathcal{W}}{min}\lambda \frac{{\left|\right|w\left|\right|}^2}{2}-\frac{1}{m}\sum _{i=1}^m{y}_{i}log\left(\varphi \left(w^{\top }{x}_i\right)\right)+(1-y_i)log\left(1-\varphi \left(w^{\top }{x}_i\right)\right)\end{array}$$

(2)

where $\lambda$ is a fixed regularisation parameter.

We also restrict the probability distributions (over $\mathcal{W}={\mathbb{R}}^d$), considered for this learning problem. We consider the Gaussian distribution $\mathcal{N}(w,{\sigma }^2{I}_d)$ with centre $w\in {\mathbb{R}}^d$ and diagonal covariance ${\sigma }^2{I}_d\in {\mathbb{R}}^{d\times d}$ with ${\sigma }^2>0$.

Parameters. We set $\delta =0.05,\lambda =0.01$. We approximately solve Equation (2) by using the minimize function of the optimisation module in Python, with the Powell method. To approximate gaussian expectations, we use Monte-Carlo sampling.

Synthetic data. We generate synthetic data for $d=10$ according to the following process: for a fixed sample size m, we draw $X_1,\dots ,X_m$ under the multivariate Gaussian distribution $\mathcal{N}(0,I_d)$ and for each i we compute the label if $X_i$ as: $Y_i=𝟙\{\varphi \left(w^{\ast \top }{x}_i\right)>1/2\}$ where $w^{\ast }$ is the vector formed by the d first digits of the number $\pi$.

Normalisation trick. Given the predictors shape, we notice that for any $w\in \mathcal{W}$:

$$𝟙\{\varphi \left(w^{\top }x\right)>1/2\}=1\iff \frac{1}{1+exp(-w^{\top }x)}>\frac{1}{2}\iff w^{\top }x<0.$$

Thus, the value of the prediction is exclusively determined by the sign of the inner product, and this quantity is definitely not influenced by the norm of the vector. Then, for any sample S, we call the normalisation trick the fact of considering $\widehat{w}\left(S\right)/\left|\right|\widehat{w}\left(S\right)\left|\right|$ instead of $\widehat{w}\left(S\right)$ in our calculations. This process will not deteriorate the quality of the prediction and will considerably enhance the value of the KL divergence.

3.2.1. First Experiment

Our goal here is to highlight the point discussed in Remark 2, e.g., the influence of the parameter $\alpha$ in Proposition 1. We arbitrarily fix ${\sigma }_0^2=1/2$, and define our naive prior as $P_0=\mathcal{N}(0,{\sigma }_0^2{I}_d)$. For a fixed dataset S, we define our posterior as $P\left(S\right):=\mathcal{N}(\widehat{h}\left(S\right),{\sigma }^2{I}_d)$, with ${\sigma }^2\in \{1/2,\dots ,1/2^J\}$ (for $J={log}_2\left(m\right)$) such that it is minimising the bound among candidates. We computed two curves: first, Proposition 1 with $\alpha =1/2$ second, Proposition 1 again with $\alpha$ equals to the value proposed in Lemma 1. Notice that to compute this last bound, we first optimised our choice of posterior with $\alpha =1/2$ and then optimised $\alpha$, to be consistent with Lemma 1. Indeed, we proved this lemma by assuming that the KL divergence was already fixed, hence our optimisation process is in two steps. Note that we chose to apply the normalisation trick here, we then obtained the left curve of Figure 1.

Discussion. From this curve, we formulate several remarks. First, we remark on this specific case, our theorem provides a tight result in practice (with an error rate lesser than $10\%$ for the bound with optimised alpha). Second, we can now confirm that choosing an optimised $\alpha$ leads to a tighter bound. In further studies, it will be relevant to adjust $\alpha$ with regards to the different terms of our bound instead of looking for an identical convergence rate for all terms.

3.2.2. Second Experiment

We now study Proposition 2 to see if an informed prior effectively provides a tighter bound than a naive one. We will use the notations introduced in Proposition 2. For a dataset S, we define $w_1\left(S\right)=w\left(S_{>m/2}\right)$ as the vector resulting from the optimisation of Equation (2) on $S_{>m/2}$. Similarly, we define $w_2\left(S\right):=w\left(S_{\le m/2}\right)$. We arbitrarily fix ${\sigma }_0^2=1/2$, and define our informed priors as: $P_1=\mathcal{N}(w_1\left(S\right),{\sigma }_0^2{I}_d)$ and $P_2=\mathcal{N}(w_2\left(S\right),{\sigma }_0^2{I}_d)$. Finally, we define our posterior as $P\left(S\right):=\mathcal{N}(\widehat{w}\left(S\right),{\sigma }^2{I}_d)$, with ${\sigma }^2\in \{1/2,\dots ,1/2^J\}$ (for $J={log}_2\left(m\right)$) with ${\sigma }^2$ optimising the bound among the same candidate than the first experiment. We computed two curves: first, Proposition 1 with $\alpha$ optimised accordingly to Lemma 1 secondly, Proposition 2 with ${\alpha }_1,{\alpha }_2$ optimised as well, and informed priors as defined above. We chose to not apply the normalisation trick here, we then obtained the right curve of Figure 1.

Discussion. It is clear, that with this framework, having an informed prior is a powerful tool to enhance the quality of our bound. Notice that we voluntarily chose to not apply the normalisation trick here. The reason is that this trick appears to be too powerful in practice, and applying it leads to counterproductive results; to highlight our point: the bound without informed prior would be tighter than the one with informed prior. Furthermore, this trick is linked to the specific structure of our problem and is not valid for any classification problem. Thus, the idea of providing informed priors remains an interesting tool for most cases.

4. PAC Bayesian Bounds with Smoothed Estimator

We now move on to control the right-hand side term in Theorem 3 when K is not constant. A first step is to consider a transformed estimate of the risk, inspired by the truncated estimator from [20], also used in [21], and more recently in [13]. The following is inspired by the results of [13], which we summarise in Section 6.

The idea is to modify the estimator $R_S\left(h\right)$ for any h by introducing a threshold t and a function $\psi$ which will attenuate the influence of the empirical losses ${(\ell (h,Z_i))}_{i=1..m}$ that exceed t.

Definition 2.

$\psi$-risks. For every $t>0$, $\psi :{\mathbb{R}}^{+}\to {\mathbb{R}}^{+}$, for any $h\in \mathcal{H}$, we define the empirical ψ-risk $R_{S,\psi ,t}$ and the theoretical ψ-risk $R_{\psi ,t}$ as follows:

$$R_{S,\psi ,t}\left(h\right):=\frac{t}{m}\sum _{i=1}^m\psi \left(\frac{\ell (h,Z_i)}{t}\right)\mathrm{and}{R}_{\psi ,t}\left(h\right)={\mathbb{E}}_{\mu }\left[t\psi \left(\frac{\ell (h,Z)}{t}\right)\right]$$

where $Z\sim \mu$. Notice that ${\mathbb{E}}_S\left[R_{S,\psi ,t}\left(h\right)\right]=R_{\psi ,t}\left(h\right)$.

We now focus on what we call softening functions, i.e., functions that will temper high values of the loss function ℓ.

Definition 3.

(Softening function). We say that $\psi :{\mathbb{R}}^{+}\to {\mathbb{R}}^{+}$ is a softening function if:

• $\forall x\in [0;1],\psi \left(x\right)=x$,
• ψ is non-decreasing,
• $\forall x\ge 1,\psi \left(x\right)\le x$.

We let $\mathcal{F}$ denote the set of all softening functions.

Remark 5.

Notice that those three assumptions ensure that ψ is continuous at 1. For instance, the functions $f:x\mapsto x𝟙\{x\le 1\}+𝟙\{x>1\}$ and $g:x\mapsto x𝟙\{x\le 1\}+(2\sqrt{x}-1)𝟙\{x>1\}$ are in $\mathcal{F}$. In Section 6 we compare these softening functions and those used by Holland [13].

Using $\psi \in \mathcal{F}$, for a fixed threshold $t>0$, the softened loss function $t\psi \left(\frac{\ell (h,z)}{t}\right)$ verifies for any $h\in \mathcal{H}$, $z\in \mathcal{Z}$:

$$t\psi \left(\frac{\ell (h,z)}{t}\right)\le t\psi \left(\frac{K\left(h\right)}{t}\right)$$

because $\psi$ is non-decreasing. In this way, the exponential moment in Theorem 3 can be far more controllable. The trade-off lies in the fact that softening ℓ (instead of taking directly ℓ) will deteriorate our ability to distinguish between two bad predictions when both of them are greater than t. For instance, if we choose $\psi \in \mathcal{F}$ such as $\psi =1$ on $[1;+\infty )$ and $t>0$, if $\psi \left(\ell (h,z)/t\right)=1$ for a certain pair $(h,z)$, then we cannot tell how far $\ell (h,z)$ is from t and we only can affirm that $\ell (h,z)\ge t$.

We now move on to the following lemma, which controls the shortfall between ${\mathbb{E}}_{h\sim Q}\left[R\left(h\right)\right]$ and ${\mathbb{E}}_{h\sim Q}\left[R_{\psi ,t}\left(h\right)\right]$ for all $Q\in {\mathcal{M}}_1^{+}\left(\mathcal{H}\right)$, for a given $\psi$ and $t>0$. To do that, we assume that K admits a finite moment under any posterior distribution:

$$\begin{array}{c}\hfill \forall Q\in {\mathcal{M}}_1^{+}\left(\mathcal{H}\right),{\mathbb{E}}_{h\sim Q}\left[K\left(h\right)\right]<+\infty .\end{array}$$

(3)

For instance, in the case of $\mathcal{H}$ identified with a weight space $\mathcal{W}={\mathbb{R}}^N$, and if K is polynomial in $\left|\right|w\left|\right|$ (where $\left|\right|.\left|\right|$ denotes the Euclidean norm), then this assumption holds if we consider Gaussian priors and posteriors.

Lemma 2.

Assume that Equation (3) holds, and let $\psi \in \mathcal{F}$, $Q\in {\mathcal{M}}_1^{+}\left(\mathcal{H}\right),t>0$. We have:

$${\mathbb{E}}_{h\sim Q}\left[R\left(h\right)\right]\le {\mathbb{E}}_{h\sim Q}\left[R_{\psi ,t}\left(h\right)\right]+{\mathbb{E}}_{h\sim Q}\left[K\left(h\right)𝟙\left\{K\left(h\right)\ge t\right\}\right].$$

Proof. 

Let $\psi \in \mathcal{F}$, $Q\in {\mathcal{M}}_1^{+}\left(\mathcal{H}\right),t>0$. We have, for $h\in \mathcal{H}$:

$$\begin{array}{cc}\hfill & R\left(h\right)-R_{\psi ,t}\left(h\right)\hfill \\ \hfill & ={\mathbb{E}}_{Z\sim \mu }\left[\ell (h,Z)-t\psi \left(\frac{\ell (h,Z)}{t}\right)\right]\hfill \end{array}$$

and using that $\forall x\in [0,1],\psi \left(x\right)=x$,

$$\begin{array}{c}={\mathbb{E}}_{Z\sim \mu }\left[\left(\ell (h,Z)-t\psi \left(\frac{\ell (h,Z)}{t}\right)\right)𝟙\{\ell (h,Z)\ge t\}\right]\hfill \end{array}$$

while using that $\ell (h,z)\le K\left(h\right)$,

$$\begin{array}{c}={\mathbb{E}}_{Z\sim \mu }\left[\left(\ell (h,Z)-t\psi \left(\frac{\ell (h,Z)}{t}\right)\right)𝟙\{\ell (h,Z)\ge t\}𝟙\left\{K\left(h\right)\ge t\right\}\right]\hfill \end{array}$$

and continuing:

$$\begin{array}{cc}\le {\mathbb{E}}_{Z\sim \mu }\left[\ell (h,Z)𝟙\{\ell (h,Z)\ge t\}\right]𝟙\left\{K\left(h\right)\ge t\right\}\hfill & \hfill \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}(\psi \ge 0)\end{array}$$

$$\begin{array}{cc}\le K\left(h\right){\mathbb{P}}_{Z\sim \mu }\left\{\ell (h,Z)\ge t\right\}𝟙\left\{K\left(h\right)\ge t\right\}\hfill & \hfill \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}(\ell (h,Z)\le K(h\left)\right)\end{array}$$

Finally, by crudely bounding the probability by 1, we get:

$$R\left(h\right)\le R_{\psi ,t}\left(h\right)+K\left(h\right)𝟙\left\{K\left(h\right)\ge t\right\}.$$

Hence the result by integrating over $\mathcal{H}$ with respect to Q. □

Finally we present the following theorem, which provides a PAC-Bayesian inequality bounding the theoretical risk by the empirical $\psi$-risk for $\psi \in \mathcal{F}$.

Theorem 4.

Let ℓ be HYPE(K) compliant, and assume K satisfies Equation (3). Then for any $P\in {\mathcal{M}}_1^{+}\left(\mathcal{H}\right)$ with no data dependency, for any $\alpha \in \mathbb{R}$, for any $\psi \in \mathcal{F}$ and for any $\delta \in [0;1]$, with probability of at least $1-\delta$ over size-m random samples S, simultaneously for all Q such that $Q\sim P$ we have:

$$\begin{array}{cc}\hfill {\mathbb{E}}_{h\sim Q}\left[R\left(h\right)\right]& \le {\mathbb{E}}_{h\sim Q}\left[R_{S,\psi ,t}\left(h\right)\right]+{\mathbb{E}}_{h\sim Q}\left[K\left(h\right)𝟙\left\{K\right(h)\ge t\}\right]\hfill \\ \hfill & +\frac{KL\left(Q\right|\left|P\right)+log\left(\frac{1}{\delta }\right)}{m^{\alpha }}\hfill \\ \hfill & +\frac{1}{m^{\alpha }}log\left({\mathbb{E}}_{h\sim P}\left[exp\left(\frac{t^2}{2m^{1-2\alpha }}\psi {\left(\frac{K\left(h\right)}{t}\right)}^2\right)\right]\right).\hfill \end{array}$$

Proof. 

Let $\psi \in \mathcal{F}$, we define the $\psi$-loss:

$${\ell }_2(h,z)=t\psi \left(\frac{\ell (h,z)}{t}\right).$$

Since $\psi$ is non decreasing, we have for all $(h,z)\in \mathcal{H}\times \mathcal{Z}$:

$${\ell }_2(h,z)\le t\psi \left(\frac{K\left(h\right)}{t}\right):=K_2\left(h\right).$$

Thus, we apply Theorem 3 to the learning problem defined with ${\ell }_2$: for any $\alpha$ and $\delta \in (0,1)$, with probability at least $1-\delta$ over size-m random samples S, simultaneously for all Q such that $Q\sim P$ we have:

$$\begin{array}{cc}\hfill {\mathbb{E}}_{h\sim Q}\left[R_{\psi ,t}\left(h\right)\right]& \le {\mathbb{E}}_{h\sim Q}\left[R_{S,\psi ,t}\left(h\right)\right]+\frac{KL\left(Q\right|\left|P\right)+log\left(\frac{1}{\delta }\right)}{m^{\alpha }}\hfill \\ \hfill & +\frac{1}{m^{\alpha }}log\left({\mathbb{E}}_{h\sim P}\left[exp\left(\frac{K_2{\left(h\right)}^2}{2m^{1-2\alpha }}\right)\right]\right).\hfill \end{array}$$

We then add ${\mathbb{E}}_{h\sim Q}\left[K\left(h\right)𝟙\left\{K\left(h\right)\ge t\right\}\right]$ on both sides of the latter inequality and apply Lemma 2. □

Remark 6.

Notice that the function $\psi :x\mapsto x𝟙\{x\le 1\}+𝟙\{x>1\}$ is such that for any given prior P we have ${\mathbb{E}}_{h\sim P}\left[exp\left(\frac{t^2}{2m^{1-2\alpha }}\psi {\left(\frac{K\left(h\right)}{t}\right)}^2\right)\right]<+\infty$. So the exponential moment can be controlled with a good choice of ψ. Thus the strength of Theorem 4 is to provide a PAC-Bayesian bound valid for any set of posterior measures verifying Equation (3). The choice of ψ minimising the bound is still an open problem.

5. The Linear Regression Problem

5.1. Theoretical Result

We now focus on the celebrated linear regression problem and see how our theory translates to that particular learning problem. We assume that the data is a size-m random sample $S={\left(Z_i\right)}_{i=1..m}$ where the $Z_i$ are i.i.d. drawn from the distribution $\mu$, and $Z_i=(X_i,Y_i)$ with $X_i\in {\mathbb{R}}^N$, $Y_i\in \mathbb{R}$.

Our goal here is to find the most accurate predictor $h_w$ (with $w\in {\mathbb{R}}^N$), with respect to the loss function $\ell (h_w,z)=|\langle w,x\rangle -y|$, where $z=(x,y)$. We will make the following mild assumption: there exists $B,C\in \mathbb{R}+\backslash \left\{0\right\}$ such that for all $z=(x,y)$ drawn under $\mu$:

$$\left|\right|x\left|\right|\le B\mathrm{and}\left|y\right|\le C$$

where $\left|\right|.\left|\right|$ is the norm associated to the classical inner product of ${\mathbb{R}}^N$. Under this assumption we note that for all $z=(x,y)$ drawn according to $\mu$, we have:

$$\begin{array}{c}\hfill \ell (h_w,z)=|\langle w,x\rangle -y|\le |\langle w,x\rangle |+|y]\le \left|\right|w\left|\right|.\left|\right|x\left|\right|+\left|y\right|\le B\left|\right|w\left|\right|+C.\end{array}$$

Thus we define $K\left(h_w\right)=B\left|\right|w\left|\right|+C$ for $w\in {\mathbb{R}}^N$. If we first restrict ourselves to the framework of Section 2, we want to use Theorem 3 and doing so, our goal is to bound $\xi :={\mathbb{E}}_{w\sim P}\left[exp\left(\frac{K{\left(w\right)}^2}{2m^{1-2\alpha }}\right)\right]$. The shape of K invites us to consider a Gaussian prior. Indeed, we notice that if $P=\mathcal{N}(0,{\sigma }^2{\mathit{I}}_N)$ with $0<{\sigma }^2<\frac{m^{1-2\alpha }}{B^2}$, then $\xi <+\infty$. Notice that we cannot take just any Gaussian prior, however with a small $\alpha$, the condition $0<{\sigma }^2<\frac{m^{1-2\alpha }}{B^2}$ may become quite loose. Thus, we have the following:

Theorem 5.

Let $\alpha \in \mathbb{R}$ and $N\ge 6$. Assume that the loss ℓ is HYPE(K) compliant with $K\left(h\right)=B\left|\right|h\left|\right|+C$, with $B>0,C\ge 0$. For a prior distribution, consider any Gaussian $P=\mathcal{N}(0,{\sigma }^2{\mathit{I}}_N)$ with ${\sigma }^2=t\frac{m^{1-2\alpha }}{B^2}$, $0<t<1$. Then, for any $\delta \in [0;1]$, with probability of at least $1-\delta$ over size-m random samples S, simultaneously for all $Q\in {\mathcal{M}}_1^{+}\left(\mathcal{H}\right)$ such that $P\sim Q$ we have:

$$\begin{array}{cc}\hfill {\mathbb{E}}_{h\sim Q}\left[R\left(h\right)\right]& \le {\mathbb{E}}_{h\sim Q}\left[R_S\left(h\right)\right]+\frac{KL\left(Q\right|\left|P\right)+log\left(2/\delta \right)}{m^{\alpha }}+\frac{C^2}{2m^{1-\alpha }}\left(1+f{\left(t\right)}^{-1}\right)\hfill \\ \hfill & +\frac{N}{m^{\alpha }}\left(log\left(1+\left(\frac{C}{\sqrt{2f\left(t\right)m^{1-2\alpha }}}\right)\right)+log\left(\frac{1}{\sqrt{1-t}}\right)\right)\hfill \end{array}$$

where $f\left(t\right)=\frac{1-t}{t}$.

The proof is deferred to Section 7.2. To compare our result with those found in the literature, we can fix $\alpha =1/2$. Doing so, we lose the dependency in m for the choice of the variance of the prior (which now only depends on B), but we recover the classic decreasing factor $1/\sqrt{m}$.

Remark 7.

Notice that for now we did not use Section 4, even if we could (because K is polynomial in $\left|\right|w\left|\right|$ and we consider Gaussian priors and posteriors, so Equation (3) is satisfied). Doing so, we obtained a bound which appears to depend linearly on the dimension N. In practice, N may be too big, and in this case, introducing an adapted softening function ψ (one can think for instance of $\psi \left(x\right)=x𝟙\{x\le 1\}+𝟙\{x>1\}$) is a powerful tool to attenuate the weight of the exponential moment. This also extends the class of authorised Gaussian priors by avoidance, to stick with a variance ${\sigma }^2=t\frac{m^{1-2\alpha }}{B^2}$,  $0<t<1$.

5.2. Numerical Experiment

5.2.1. Setting

In this section we apply Theorem 5 on a concrete linear regression problem. The situation is as follows: we want to approximate the function $f\left(x\right)=\sqrt{\langle w^{\ast },x\rangle }$, where $w^{\ast }\in {\mathbb{R}}^d$. We assume that $\mathcal{W}={[-c,c]}^d$ so that $w^{\ast }$ lies in an hypercube centred at 0 of half-side $c>0$, i.e., the set $\{{\left(w_i\right)}_{i=1,\dots ,d}\mid \forall i,|w_i|\le c\}$. Doing so we have $\left|\right|w^{\ast }\left|\right|\le c\sqrt{d}$.

Furthermore, we assume that input data are drawn inside a hypercube of half-side $e>0$, i.e., $\mathcal{X}={[-e,e]}^d$. Doing so we have for any data $x,\left|\right|x\left|\right|\le e\sqrt{d}$.

For any data $x\in {\mathbb{R}}^d$, we define $y=f\left(x\right)$. As before, we identify the hypothesis set $\mathcal{H}$ with the weight space $\mathcal{W}={\mathbb{R}}^d$. As described in Section 5.1, we set $\ell (h_w,x,y)=|\langle w,x\rangle -y|$. We then remark that for any $(w,x,y)$:

$$\begin{array}{cc}\hfill \ell (h_w,x,y)& \le \left|\langle w,x\rangle \right|+\left|y\right|\le \left|\right|w\left|\right|\left|\right|x\left|\right|+|\sqrt{\langle w^{\ast },x\rangle }|\hfill \\ \hfill & \le e\sqrt{d}\left|\right|w\left|\right|+\sqrt{\left|\right|w^{\ast }\left|\right|.\left|\right|x\left|\right|}\le e\sqrt{d}\left|\right|w\left|\right|+\sqrt{c\sqrt{d}.e\sqrt{d}}\hfill \\ \hfill & \le e\sqrt{d}\left|\right|w\left|\right|+\sqrt{cde}.\hfill \end{array}$$

Then we can define $B=e\sqrt{d}$ and $C=\sqrt{cde}$ to apply Theorem 5. We restrict (as before) the class of distributions over $\mathcal{W}$ to be d-dimensional Gaussians:

$$\left\{\mathcal{N}(w,{\sigma }^2{I}_d)\mid w\in \mathcal{H},{\sigma }^2\in {\mathbb{R}}^{+}\right\},$$

which is the set of candidate distributions for this learning problem. Recall that in practice, given a fixed $\alpha \in \mathbb{R}$, we are only allowed to consider priors such that their variance ${\sigma }^2\in \left]0;\frac{m^{1-2\alpha }}{B^2}\right[$. We want to learn an optimised predictor (posterior) given a random dataset $S={\left((X_i,Y_i)\right)}_{i=1,\dots ,m}$. To do so, we consider synthetic data.

5.2.2. Synthetic Data

We draw $w^{\ast }$ under a Gaussian (with mean 0 and standard deviation equal to 5) truncated to the hypercube centered at 0 of the half-side $c>0$. We generate synthetic data according to the following process: for a fixed sample size m, we draw $X_1,\dots ,X_m$ under a Gaussian (with mean 0 and standard deviation equal to 5) truncated to the hypercube centered at 0 of the half-side $e>0$.

5.2.3. Experiment

First, we fix $c=e=10$. Our goal here is to obtain a generalisation bound on our problem. We fix arbitrarily, for a fixed $\alpha \in \mathbb{R}$, $t_0=1/2$ and ${\sigma }_0^2=t_0\frac{m^{1-2\alpha }}{B^2}$ and we define our naive prior as $P_0=\mathcal{N}(0,{\sigma }_0^2{I}_d)$. For a given dataset S, we define our posterior as $Q\left(S\right):=\mathcal{N}(\widehat{w}\left(S\right),{\sigma }^2{I}_d)$, with ${\sigma }^2\in \{{\sigma }_0^2/2,\dots ,{\sigma }_0^2/2^J\}$ ($J={log}_2\left(m\right)$), such that it is minimising the bound among candidates. Note that all the previously defined parameters are dependent on $\alpha$, which is why we choose $\alpha \in \{i/\mathtt{step}\mid 0\le i\le \mathtt{step}\}$ for step a fixed integer (in practice step = 8 or 16) and we take the value of $\alpha$ minimising the bound among the candidates as well. Figure 2 contains two figures, one with $d=10$, the other with $d=50$. On each figure are computed the right-hand side term in 5 with an optimised $\alpha$ for each step.

5.2.4. Discussion

To the the best of our knowledge, this is the first attempt to numerically compute PAC-Bayes bounds for unbounded problems, making it impossible to compare to other results. We stress, however, that obtaining numerical values for the bound without assuming a bounded loss is a significant first step. Furthermore, we consider a rather hard problem: f is not linear, so we cannot rely on a linear approximation fitting perfectly data, and the larger the dimension, the larger the error, as illustrated by Figure 2. Thus, for any posterior Q, the quantity ${\mathbb{E}}_{h\sim Q}\left[R\left(h\right)\right]$ is potentially large in practice and our bound might not be tight. Finally, notice that optimising $\alpha$ (instead of taking $\alpha =1/2$ to recover a classic convergence rate) leads to a significantly better bound. A numerical example of this assertion is presented in Section 3.2. We aim to conduct further studies to consider the convergence rate as an hyperparameter to optimise, rather than selecting the same rate for all terms in the bound.

6. Existing Work

6.1. Germain et al., 2016

In Germain et al. [11] (Section 4), a PAC-Bayesian bound has been provided for all sub-gamma losses with a variance $t^2$ and scale parameter $c>0$, under a data distribution $\mu$ and a prior P, i.e., losses such that for every $\lambda \in \left(0,\frac{1}{c}\right)$ the following is satisfied:

$$log\left(\frac{1}{\delta }{\mathbb{E}}_{h\sim P}{\mathbb{E}}_S{e}^{\lambda (R\left(h\right)-R_S\left(h\right))}\right)\le \frac{t^2}{c^2}(-log(1-c\lambda )-\lambda c)\le \frac{{\lambda }^2{t}^2}{2(1-c\lambda )}.$$

Note that a sub-gamma loss (with regards to $\mu$ and P) is potentially unbounded. Germain et al. then propose the following PAC-Bayesian bound:

Theorem 6.

Ref. [11]. If the loss ℓ is sub-gamma with a variance $t^2$ and scale parameter c, under the data distribution μ and a fixed prior $P\in \mathcal{H}$, then for any $\delta \in [0;1]$, with probability $1-\delta$ over size-m random samples, simultaneously for all $Q\ll P$ we have:

$${\mathbb{E}}_{h\sim Q}\left[R\left(h\right)\right]\le {\mathbb{E}}_{h\sim Q}\left[R_S\left(h\right)\right]+\frac{KL\left(Q\right|\left|P\right)+log(1/\delta )}{m}+\frac{t^2}{2(1-c)}.$$

Theorem 6 will be quoted several times in this paper given that it is a concrete PAC Bayesian bound provided with the will to overcome the constraint of a bounded loss. It is also one of the only one found in the literature.

Can we apply this theorem to the bounded case? The answer is yes: we remark that thanks to Hoeffding’s lemma, if ℓ is bounded by $C>0$, then for any $h\in \mathcal{H}$ it holds that $R_S\left(h\right)-R\left(h\right)\in [-C,C]$ almost surely. So, $\forall \lambda \in \mathbb{R},$ $log{\mathbb{E}}_{z\sim \mu }\left[e^{\lambda (R\left(h\right)-R_S\left(h\right)}\right]\le \frac{{\lambda }^2{C}^2}{2}$. Therefore, for any prior P, we have:

$$log{\mathbb{E}}_{h\sim P}{\mathbb{E}}_{z\sim \mu }\left[e^{\lambda (R\left(h\right)-R_S\left(h\right)}\right]\le \frac{{\lambda }^2{C}^2}{2}.$$

Thus, ℓ is sub-gamma with variance $C^2$ and scale parameter 0. Then, Theorem 6 can be applied with $t^2=C^2$, $c=0$.

Comparison with Proposition 1. We remark that by taking $K=C$ and $\alpha =1$ in Proposition 1, we are recovering Theorem 6. However, our approach allows us to say that if we can obtain a more precise form of K such that $\forall h\in \mathcal{H}$, $K\left(h\right)\le C$ and K is non-constant, 3, will ensure that:

$$\frac{1}{m^{\alpha }}log\left({\mathbb{E}}_{h\sim P}\left[exp\left(\frac{K{\left(h\right)}^2}{2m^{1-2\alpha }}\right)\right]\right)\le \frac{C^2}{2m^{1-\alpha }}.$$

Thus, having precise information on the behavior of the loss function ℓ, with regards to the predictor h, allows us to obtain a tighter control of the exponential moment, and hence a tighter bound.

Remark 8.

We can see that Theorem 6 cannot control the factor $C^2/2$. However, Ref. [11] remarked on this apparent weakness and partially corrected this issue [11] (Section 4, Equations (13) and (14)). Indeed, they proposed to balance the influence of m between the different terms of the PAC-Bayes bound by providing the same convergence rate in $1/\sqrt{m}$ to all terms.

We can then see Proposition 1 as a proper generalisation of Germain et al. [11] (Section 4, Equations (13) and (14)). Indeed, our bound exhibits properly the influence of the parameter α. Thus, we understand (and Lemma 1 proves it) that the choice of α deserves a study in itself in the way it is now a parameter of our optimisation problem. This fact has already been highlighted in Alquier et al. [10] (Theorem 4.1) (where $\lambda :=m^{\alpha }$).

6.2. Holland, 2019

In [13], Holland proposed a PAC Bayesian inequality with unbounded loss. For that, he introduced a function $\psi$ verifying a few specific conditions, different to those used in Section 4 to define our set of softening functions. Indeed, he considered a function $\psi$ such that:

• $\psi$ is bounded,
• $\psi$ is non decreasing,
• it exists $b>0$ such that for all $u\in \mathbb{R}$:

  $$\begin{array}{c}\hfill -log\left(1-u+\frac{u^2}{b}\right)\le \psi \left(u\right)\le log\left(1+u+\frac{u^2}{b}\right).\end{array}$$

  (4)

We remark that, as Holland did, we supposed that our softening functions are non-decreasing. We chose softening functions to be equal to the identity function ($x\mapsto x$) on $[0,1]$, which is quite restrictive. However, we are imposing softening functions to be lesser than the identity on $\left[1,+\infty \right)$; whereas, Holland supposed $\psi$ to be bounded and satisfy Equation (4). A concrete example of such a function $\psi$, lies in the piecewise polynomial function of Catoni and Giulini [21], defined by:

$$\psi \left(u\right)=\left\{\begin{array}{cc}-2\sqrt{2}/3\hfill & \mathrm{if}u\le -\sqrt{2}\hfill \\ u-u^3/6\hfill & \mathrm{if}u\in [-2\sqrt{2}/3,2\sqrt{2}/3]\hfill \\ 2\sqrt{2}/3\hfill & \mathrm{otherwise}.\hfill \end{array}\right.$$

As in Section 4, we are considering the $\psi$-empirical risk $R_{S,\psi ,t}$ for any $t>0$. Holland provided his theorem given the fact the following assumptions are realised:

• Bounds on lower-order moments. For all $h\in \mathcal{H}$, we have ${\mathbb{E}}_{Z\sim \mu }[\ell {(h,Z)}^2]\le M_2<+\infty$ and ${\mathbb{E}}_{Z\sim \mu }[\ell {(h,Z)}^3]\le M_3<+\infty$.
• Bounds on the risk. For all $h\in \mathcal{H}$, we suppose $R\left(h\right)\le \sqrt{m{M}_2/(4log\left({\delta }^{-1}\right)}$.
• Large enough confidence, we require $\delta \le e^{-1/9}$.

Now we can state Holland’s theorem.

Theorem 7.

Ref. [13]. Let P be a prior distribution on model $\mathcal{H}$. Let the three assumptions listed above hold. Setting $t^2=m{M}_2/(2log\left({\delta }^{-1}\right))$, then for any $\delta \in [0;1]$, with probability of at least $1-\delta$ over the random draw of the size-m sample S, simultaneously for all Q it holds that:

$$\begin{array}{cc}\hfill {\mathbb{E}}_{h\sim Q}\left[R\left(h\right)\right]& \le {\mathbb{E}}_{h\sim Q}\left[R_{S,\psi ,t}\left(h\right)\right]+\frac{1}{\sqrt{m}}\left(KL\left(Q\right|\left|P\right)+\frac{1}{2}log\left(\frac{8\pi M_2}{{\delta }^2}\right)-1\right)\hfill \\ \hfill & +\frac{1}{\sqrt{m}}{\nu }^{\ast }\left(\mathcal{H}\right)+O\left(\frac{1}{m}\right)\hfill \end{array}$$

where:

$${\nu }^{\ast }\left(\mathcal{H}\right):=\frac{{\mathbb{E}}_{h\sim P}\left[exp\left(\sqrt{m}(R\left(h\right)-R_{S,\psi ,t}\left(h\right))\right)\right]}{{\mathbb{E}}_{h\sim P}\left[exp\left(R\left(h\right)-R_{S,\psi ,t}\left(h\right)\right)\right]}.$$

7. Proofs

7.1. Proof of Theorem 1

Proof. 

Let $F:{\mathbb{R}}^{+}\times {\mathbb{R}}^{+}\mapsto \mathbb{R}$ be a convex function, P a fixed prior, and $\delta \in [0,1]$. Since ${\mathbb{E}}_{h\sim P}\left[e^{F(R_S\left(h\right),R\left(h\right))}\right]$ is a nonnegative random variable, we know that, by Markov’s inequality, for any $h\in \mathcal{H}$:

$$\mathbb{P}\left({\mathbb{E}}_{h\sim P}\left[e^{F(R_S\left(h\right),R\left(h\right))}\right]>\frac{1}{\delta }{\mathbb{E}}_S{\mathbb{E}}_{h\sim P}\left[e^{F(R_S\left(h\right),R\left(h\right))}\right]\right)\le \delta .$$

So with probability of at least $1-\delta$, we have:

$${\mathbb{E}}_{h\sim P}\left[e^{F(R_S\left(h\right),R\left(h\right))}\right]\le \frac{1}{\delta }{\mathbb{E}}_S{\mathbb{E}}_{h\sim P}\left[e^{F(R_S\left(h\right),R\left(h\right))}\right]=\frac{\chi }{\delta }.$$

Applying the log function on each side of this inequality gives us with probability of at least $1-\delta$ over samples S:

$$log\left({\mathbb{E}}_{h\sim P}\left[e^{F(R_S\left(h\right),R\left(h\right))}\right]\right)\le log\left(\frac{\chi }{\delta }\right).$$

We now rename $A:=log\left({\mathbb{E}}_{h\sim P}\left[e^{F(R_S\left(h\right),R\left(h\right))}\right]\right)$.

Furthermore, if we denote by $\frac{dQ}{dP}$ the Radon-Nikodym derivative of Q with respect to P when $Q\ll P$, we then have, for all Q such that $Q\sim P$:

$$\begin{array}{cc}\hfill A& =log\left({\mathbb{E}}_{h\sim Q}\left[\frac{dP}{dQ}{e}^{F(R_S\left(h\right),R\left(h\right))}\right]\right)\hfill \\ \hfill & =log\left({\mathbb{E}}_{h\sim Q}\left[{\left(\frac{dQ}{dP}\right)}^{-1}{e}^{F(R_S\left(h\right),R\left(h\right))}\right]\right)\hfill & \hfill \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}(\frac{dP}{dQ}={\left(\frac{dQ}{dP}\right)}^{-1})\end{array}$$

and by concavity of log and Jensen’s inequality,

$$\begin{array}{c}\ge -{\mathbb{E}}_{h\sim Q}\left[log\left(\frac{dQ}{dP}\right)\right]+{\mathbb{E}}_{h\sim Q}\left[F(R_S\left(h\right),R\left(h\right))\right]\hfill \\ =-KL\left(Q\right|\left|P\right)+{\mathbb{E}}_{h\sim Q}\left[F(R_S\left(h\right),R\left(h\right))\right]\hfill \end{array}$$

while by convexity of F with Jensen’s inequality,

$$\begin{array}{c}\ge -KL\left(Q\right|\left|P\right)+F\left({\mathbb{E}}_{h\sim Q}\left[R_S\left(h\right)\right],{\mathbb{E}}_{h\sim Q}\left[R\left(h\right)\right]\right).\hfill \end{array}$$

Hence, for Q such that $Q\sim P$,

$$F\left({\mathbb{E}}_{h\sim Q}\left[R_S\left(h\right)\right],{\mathbb{E}}_{h\sim Q}\left[R\left(h\right)\right]\right)\hspace{0.17em}\le \hspace{0.17em}KL\left(Q\right|\left|P\right)+A.$$

So with probability $1-\delta$, for Q such that $Q\sim P$,

$$\begin{array}{c}F\left({\mathbb{E}}_{h\sim Q}\left[R_S\left(h\right)\right],{\mathbb{E}}_{h\sim Q}\left[R\left(h\right)\right]\right)\hspace{0.17em}\le \hspace{0.17em}KL\left(Q\right|\left|P\right)+log\left(\frac{\chi }{\delta }\right).\hfill \end{array}$$

This completes the proof of Theorem 1. □

7.2. Proof of Theorem 5

We first provide a technical property. Recall that:

$$\xi ={\mathbb{E}}_{h\sim P}\left[exp\left(\frac{K{\left(h\right)}^2}{2m^{1-2\alpha }}\right)\right].$$

Proposition 3.

Let $\alpha \in \mathbb{R}$. Suppose the loss ℓ is HYPE(K) compliant with $K\left(h\right)=B\left|\right|h\left|\right|+C$, with $B>0$, $C\ge 0$. Then, for any Gaussian prior $P=\mathcal{N}(0,{\sigma }^2{\mathit{I}}_N)$ with ${\sigma }^2=t\frac{m^{1-2\alpha }}{B^2}$, $0<t<1$ and $N\ge 6$ we have:

$$\xi \le 2exp\left(\frac{C^2}{2m^{1-2\alpha }f\left(t\right)}\left(1+f\left(t\right)\right)\right)\frac{1}{{\left(\sqrt{1-t}\right)}^N}{\left(1+\left(\frac{C}{\sqrt{2f\left(t\right)m^{1-2\alpha }}}\right)\right)}^{N-1}$$

with $f\left(t\right)=\frac{1-t}{t}$.

Proof. 

We recall that ${\sigma }^2=t\frac{m^{1-2\alpha }}{B^2}$. By expliciting the expectation and $K\left(h\right)$ we thus obtain:

$$\begin{array}{ccc}\hfill \xi & ={\left(\frac{1}{\sqrt{2\pi {\sigma }^2}}\right)}^N{\int }_{h\in {\mathbb{R}}^N}exp\left(\frac{{\left(B\right|\left|h\right||+C)}^2}{2m^{1-2\alpha }}-\frac{{\left|\right|h\left|\right|}^2{B}^2}{2t{m}^{1-2\alpha }}\right)dh\hfill & \\ \hfill & ={\left(\frac{1}{\sqrt{2\pi {\sigma }^2}}\right)}^N{\int }_{h\in {\mathbb{R}}^N}exp\left(-\frac{1}{2m^{1-2\alpha }}\left(f\left(t\right)B^2{\left|\right|h\left|\right|}^2-2BC\left|\right|h\left|\right|-C^2\right)\right)dh\hfill & \\ \hfill & ={\left(\frac{1}{\sqrt{2\pi {\sigma }^2}}\right)}^N{\int }_{h\in {\mathbb{R}}^N}exp\left(-\frac{B^{2}f\left(t\right)}{2m^{1-2\alpha }}\left({\left|\right|h\left|\right|}^2-\frac{2C\left|\right|h\left|\right|}{Bf\left(t\right)}-\frac{C^2}{B^{2}f\left(t\right)}\right)\right)dh\hfill & \\ \hfill & =exp\left(\frac{C^2}{2m^{1-2\alpha }f\left(t\right)}\left(1+f\left(t\right)\right)\right)\frac{1}{{\left(\sqrt{2\pi {\sigma }^2}\right)}^N}{\int }_{h\in {\mathbb{R}}^N}exp\left(-\frac{B^{2}f\left(t\right)}{2m^{1-2\alpha }}{\left(\left|\right|h\left|\right|-\frac{C}{Bf\left(t\right)}\right)}^2\right)dh.\hfill \end{array}$$

We will use the spherical coordinates in N-dimensional Euclidean space given in [22]:

$$\phi :(h_1,\dots ,h_N)\to (r,{\phi }_1,\dots ,{\phi }_{N-1})$$

where especially $r=\left|\right|h\left|\right|$ and also the Jacobian of $\varphi$ is given by:

$$d^{N}V=r^{N-1}\prod _{k=1}^{N-2}{sin}^k\left({\phi }_{N-1-k}\right)=r^{N-1}{d}_{S^{N-1}}V.$$

Let us also precise that as given in Blumenson [22] (page 66), we have that the surface of the sphere of radius 1 in N-dimensional space is:

$${\int }_{{\phi }_1,\dots ,{\phi }_{N-1}}{d}_{S^{N-1}}Vd{\phi }_1\dots d{\phi }_{N-1}=\frac{2{\sqrt{\pi }}^N}{\Gamma \left(\frac{N}{2}\right)}$$

where $\Gamma$ is the Gamma function defined as:

$$\Gamma \left(x\right)={\int }_0^{+\infty }{t}^{x-1}{e}^{-t}dt\mathrm{for}x>-1.$$

Then, if we set:

$$A:={\int }_{h\in {\mathbb{R}}^N}exp\left(-\frac{B^{2}f\left(t\right)}{2m^{1-2\alpha }}{\left(\left|\right|h\left|\right|-\frac{C}{Bf\left(t\right)}\right)}^2\right)dh$$

we obtain by a change of variable:

$$\begin{array}{cc}\hfill A& ={\int }_{r,{\phi }_1,\dots ,{\phi }_{N-1}}exp\left(-\frac{B^{2}f\left(t\right)}{2m^{1-2\alpha }}{\left(r-\frac{C}{Bf\left(t\right)}\right)}^2\right)d^{N}Vdrd{\phi }_1\dots d{\phi }_{N-1}\hfill \\ \hfill & =\left(\frac{2{\sqrt{\pi }}^N}{\Gamma \left(\frac{N}{2}\right)}\right){\int }_{r=0}^{+\infty }exp\left(-\frac{B^{2}f\left(t\right)}{2m^{1-2\alpha }}{\left(r-\frac{C}{Bf\left(t\right)}\right)}^2\right)r^{N-1}dr\hfill \\ \hfill & =\left(\frac{2{\sqrt{\pi }}^N}{\Gamma \left(\frac{N}{2}\right)}\right){\int }_{r=-\frac{C}{Bf\left(t\right)}}^{+\infty }{\left(r+\frac{C}{Bf\left(t\right)}\right)}^{N-1}exp\left(-\frac{B^{2}f\left(t\right)}{2m^{1-2\alpha }}{r}^2\right)dr\hfill \\ \hfill & =\left(\frac{2{\sqrt{\pi }}^N}{\Gamma \left(\frac{N}{2}\right)}\right)\sum _{k=0}^{N-1}\left(\genfrac{}{}{0pt}{}{N-1}{k}\right){\left(\frac{C}{Bf\left(t\right)}\right)}^{N-k-1}{\int }_{r=-\frac{C}{Bf\left(t\right)}}^{+\infty }{r}^{k}exp\left(-\frac{B^{2}f\left(t\right)}{2m^{1-2\alpha }}{r}^2\right)dr.\hfill \end{array}$$

We fix a random variable X such that:

$$X\sim \mathcal{N}\left(0,\frac{m^{1-2\alpha }}{B^2(f\left(t\right)}\right).$$

We then have for any k positive integer, if k is even:

$$\begin{array}{cc}\hfill {\int }_{r=-\frac{C}{Bf\left(t\right)}}^{+\infty }{r}^{k}exp\left(-\frac{B^{2}f\left(t\right)}{2m^{1-2\alpha }}{r}^2\right)dr& \le {\int }_{r=-\infty }^{+\infty }{r}^{k}exp\left(-\frac{B^{2}f\left(t\right)}{2m^{1-2\alpha }}{r}^2\right)dr\hfill \\ \hfill & \le \sqrt{2\pi \frac{m^{1-2\alpha }}{B^{2}f\left(t\right)}}{\mathbb{E}\left[\right|X|}^k].\hfill \end{array}$$

And if k is odd:

$$\begin{array}{cc}\hfill {\int }_{r=-\frac{C}{Bf\left(t\right)}}^{+\infty }{r}^{k}exp\left(-\frac{B^{2}f\left(t\right)}{2m^{1-2\alpha }}{r}^2\right)dr& \le {\int }_{r=0}^{+\infty }{r}^{k}exp\left(-\frac{B^{2}f\left(t\right)}{2m^{1-2\alpha }}{r}^2\right)dr\hfill \\ \hfill & \le \sqrt{2\pi \frac{m^{1-2\alpha }}{B^{2}f\left(t\right)}}{\mathbb{E}\left[\right|X|}^k𝟙(X\ge 0)]\hfill \\ \hfill & \le \sqrt{2\pi \frac{m^{1-2\alpha }}{B^{2}f\left(t\right)}}{\mathbb{E}\left[\right|X|}^k].\hfill \end{array}$$

So we have:

$$\begin{array}{cc}\hfill A& \le \left(\frac{2{\sqrt{\pi }}^N}{\Gamma \left(\frac{N}{2}\right)}\right)\sum _{k=0}^{N-1}\left(\genfrac{}{}{0pt}{}{N-1}{k}\right){\left(\frac{C}{Bf\left(t\right)}\right)}^{N-k-1}\sqrt{2\pi \frac{m^{1-2\alpha }}{B^{2}f\left(t\right)}}{\mathbb{E}\left[\right|X|}^k].\hfill \end{array}$$

As precised in [23], we have for any k:

$${\mathbb{E}\left[\right|X|}^k]={\left(\sqrt{\frac{m^{1-2\alpha }}{B^{2}f\left(t\right)}}\right)}^k{2}^{k/2}\frac{\Gamma \left(\frac{k+1}{2}\right)}{\sqrt{\pi }}.$$

So finally:

$$\begin{array}{cc}\hfill A& \le 2{\sqrt{\pi }}^N\sum _{k=0}^{N-1}\left(\genfrac{}{}{0pt}{}{N-1}{k}\right){\left(\frac{C}{Bf\left(t\right)}\right)}^{N-k-1}{\left(\sqrt{\frac{2m^{1-2\alpha }}{B^{2}f\left(t\right)}}\right)}^{k+1}\frac{\Gamma \left(\frac{k+1}{2}\right)}{\Gamma \left(\frac{N}{2}\right)}.\hfill \end{array}$$

Lemma 3.

If $N\ge 6$, then:

$$\underset{k=0..N-1}{max}\frac{\Gamma \left(\frac{k+1}{2}\right)}{\Gamma \left(\frac{N}{2}\right)}=1.$$

Proof. 

As precised in the introduction of Srinivasan and Zvengrowski [24], Gauss [25] (page 147) proved that on the interval $[x_0,+\infty )$ where $x_0\in [1.46,1.47]$, $\Gamma$ is a monotonic increasing function. So, for $N-1\ge k\ge 2,\Gamma \left(\frac{k+1}{2}\right)\le \Gamma \left(\frac{N}{2}\right)$. And because $\Gamma (1/2)=\sqrt{\pi },\Gamma \left(1\right)=1$, we have:

$$\underset{k=0..N-1}{max}\frac{\Gamma \left(\frac{k+1}{2}\right)}{\Gamma \left(\frac{N}{2}\right)}=max\left(\frac{\sqrt{\pi }}{\Gamma \left(\frac{N}{2}\right)},\frac{\Gamma \left(\frac{N-1+1}{2}\right)}{\Gamma \left(\frac{N}{2}\right)}\right)=max\left(\frac{\sqrt{\pi }}{\Gamma \left(\frac{N}{2}\right)},1\right)$$

Because $N\ge 6$, and $\Gamma$ is monotone and increasing on $[3;+\infty ]$, we have $\Gamma (N/2)\ge \Gamma \left(3\right)\ge \sqrt{\pi }$. Hence the result. □

Using Lemma 3 allows us to write:

$$\begin{array}{cc}\hfill A& \le 2{\sqrt{\pi }}^N\sum _{k=0}^{N-1}\left(\genfrac{}{}{0pt}{}{N-1}{k}\right){\left(\frac{C}{Bf\left(t\right)}\right)}^{N-k-1}{\left(\sqrt{\frac{2m^{1-2\alpha }}{B^{2}f\left(t\right)}}\right)}^{k+1}.\hfill \end{array}$$

We recall that ${\sigma }^2=t\frac{m^{1-2\alpha }}{B^2}$ and $f\left(t\right)=\frac{1-t}{t}$. Then we can write:

$$A\le 2{\sqrt{\pi }}^N\sum _{k=0}^{N-1}\left(\genfrac{}{}{0pt}{}{N-1}{k}\right){\left(\frac{C}{Bf\left(t\right)}\right)}^{N-k-1}{\left(\sqrt{\frac{2{\sigma }^2}{1-t}}\right)}^{k+1}.$$

We now conclude with the final bound on $\xi$:

$$\begin{array}{cc}\xi \hfill & \le exp\left(\frac{C^2}{2m^{1-2\alpha }f\left(t\right)}\left(1+f\left(t\right)\right)\right)\frac{1}{{\left(\sqrt{2\pi {\sigma }^2}\right)}^N}A\hfill \\ \hfill & \le exp\left(\frac{C^2}{2m^{1-2\alpha }f\left(t\right)}\left(1+f\left(t\right)\right)\right)\frac{1}{{\left(\sqrt{2\pi {\sigma }^2}\right)}^N}2{\sqrt{\pi }}^N{\sum }_{k=0}^{N-1}\left(\genfrac{}{}{0pt}{}{N-1}{k}\right){\left(\frac{C}{Bf\left(t\right)}\right)}^{N-k-1}{\left(\sqrt{\frac{2{\sigma }^2}{1-t}}\right)}^{k+1}\hfill \\ \hfill & \le 2exp\left(\frac{C^2}{2m^{1-2\alpha }f\left(t\right)}\left(1+f\left(t\right)\right)\right){\sum }_{k=0}^{N-1}\left(\genfrac{}{}{0pt}{}{N-1}{k}\right){\left(\frac{C}{Bf\left(t\right)}\right)}^{N-k-1}{\left(\sqrt{\frac{1}{1-t}}\right)}^{k+1}{\left(\sqrt{\frac{B^2}{2t{m}^{1-2\alpha }}}\right)}^{N-k-1}\hfill \\ \hfill & \le 2exp\left(\frac{C^2}{2m^{1-2\alpha }f\left(t\right)}\left(1+f\left(t\right)\right)\right){\sum }_{k=0}^{N-1}\left(\genfrac{}{}{0pt}{}{N-1}{k}\right){\left(\frac{C\sqrt{t}}{(1-t)\sqrt{2m^{1-2\alpha }}}\right)}^{N-k-1}{\left(\sqrt{\frac{1}{1-t}}\right)}^{k+1}\hfill \\ \hfill & \le 2\frac{exp\left(\frac{C^2}{2m^{1-2\alpha }f\left(t\right)}\left(1+f\left(t\right)\right)\right)}{{\left(\sqrt{1-t}\right)}^N}{\sum }_{k=0}^{N-1}\left(\genfrac{}{}{0pt}{}{N-1}{k}\right){\left(\frac{C}{\sqrt{2f\left(t\right)m^{1-2\alpha }}}\right)}^{N-k-1}\hfill \\ \hfill & \le 2\frac{exp\left(\frac{C^2}{2m^{1-2\alpha }f\left(t\right)}\left(1+f\left(t\right)\right)\right)}{{\left(\sqrt{1-t}\right)}^N}{\left(1+\left(\frac{C}{\sqrt{2f\left(t\right)m^{1-2\alpha }}}\right)\right)}^{N-1}.\hfill \end{array}$$

This completes the proof of Proposition 3. □

Proof of Theorem 5.

We combine Theorem 3 with Proposition 3. We also upper-bound $N-1$ by N. □