Two New Families of Local Asymptotically Minimax Lower Bounds in Parameter Estimation

Abstract

We propose two families of asymptotically local minimax lower bounds on parameter estimation performance. The first family of bounds applies to any convex, symmetric loss function that depends solely on the difference between the estimate and the true underlying parameter value (i.e., the estimation error), whereas the second is more specifically oriented to the moments of the estimation error. The proposed bounds are relatively easy to calculate numerically (in the sense that their optimization is over relatively few auxiliary parameters), yet they turn out to be tighter (sometimes significantly so) than previously reported bounds that are associated with similar calculation efforts, across many application examples. In addition to their relative simplicity, they also have the following advantages: (i) Essentially no regularity conditions are required regarding the parametric family of distributions. (ii) The bounds are local (in a sense to be specified). (iii) The bounds provide the correct order of decay as functions of the number of observations, at least in all the examples examined. (iv) At least the first family of bounds extends straightforwardly to vector parameters.

1. Introduction

The theory of parameter estimation consists of a very large plethora of lower bounds (as well as upper bounds), which characterize the fundamental performance limits of any estimator in a given parametric model. In this context, it is common to distinguish between Bayesian bounds (see, e.g., the Bayesian Cramér–Rao bound [1], the Bayesian Bhattacharyya bound, the Bobrovsky–Zakai bound [2], the Bellini–Tartara bound [3], the Chazan–Zakai–Ziv bound [4], the Weiss–Weinstein bound [5,6], and more (see [7] for a comprehensive overview), and non-Bayesian bounds, where in the former, the parameter to be estimated is considered a random variable with a given probability law, as opposed to the latter, where it is assumed an unknown deterministic constant. The category of non-Bayesian bounds is further subdivided into two subclasses; one is associated with local bounds that hold for classes of estimators with certain limitations, such as unbiased estimators (see, e.g., the Cramér–Rao bound [8,9,10,11,12], the Bhattacharyya bound [13], the Barankin bound [14], the Chapman–Robbins bound [15], the Fraser–Guttman bound [16], the Keifer bound [17], and more), and the other is the subclass of minimax bounds (see, e.g., Ziv and Zakai [18], Hajek [19], Le Cam [20], Assouad [21], Fano [22], Lehmann (Sections 4.2–4.4 in [23]), Nazin [24], Yang and Barron [25], Guntuboyina [26,27], Kim [28], and many more).

In this paper, we focus on the minimax approach, and more concretely, on the local minimax approach. According to the minimax approach, we are given a parametric family of probability density functions (or probability mass functions, in the discrete case), $\{p(x_1,\dots ,x_n|\theta ),(x_1,\dots ,x_n)\in {\mathbb{R}}^n,\theta \in \Theta \}$, where $\theta$ is a d-dimensional parameter vector, $\Theta \subseteq {\mathbb{R}}^d$ is the parameter space, n is a positive integer designating the number of observations, and we define a loss function, $\mathcal{l}(\theta ,{\widehat{\theta }}_n)$, where ${\widehat{\theta }}_n$ is an estimator, which is a function of the observations $x_1,\dots ,x_n$ only. The minimax performance is defined as

$$R_n(\Theta )\stackrel{▵}{=}\underset{{\widehat{\theta }}_n(·)}{inf}\underset{\theta \in \Theta }{sup}{\mathbb{E}}_{\theta }\left\{\mathcal{l}(\theta ,{\widehat{\theta }}_n)\right\},$$

(1)

where ${\mathbb{E}}_{\theta }$ denotes expectation w.r.t. $p(·|\theta )$. As customary, we consider here loss functions with the property that $\mathcal{l}(\theta ,{\widehat{\theta }}_n)$ depends on $\theta$ and ${\widehat{\theta }}_n$ only via their difference, that is, $\mathcal{l}(\theta ,{\widehat{\theta }}_n)=\rho (\theta -{\widehat{\theta }}_n)$, where the function $\rho (·)$ satisfies certain assumptions (see Section 2). The local asymptotic minimax performance at the point $\theta \in \Theta$ is defined as follows (see also, e.g., [19]). Let $\{{\zeta }_n^{*},n\ge 1\}$ be a positive sequence, tending to infinity, with the property that

$$r\left(\theta \right)\stackrel{▵}{=}\underset{\delta \downarrow 0}{lim}\underset{n\to \infty }{lim\; inf}\underset{{\widehat{\theta }}_n(·)}{inf}\underset{\{{\theta }^{\prime }:\parallel {\theta }^{\prime }-\theta \parallel \le \delta \}}{sup}{\zeta }_n^{*}·{\mathbb{E}}_{{\theta }^{\prime }}\left\{\rho ({\widehat{\theta }}_n-{\theta }^{\prime })\right\}$$

(2)

is a strictly positive finite constant. Then, we say that $r\left(\theta \right)$ is the local asymptotic minimax performance with respect to (w.r.t.) $\left\{{\zeta }_n^{*}\right\}$ at the point $\theta \in \Theta$. Roughly speaking, the significance is that the performance of a good estimator, ${\widehat{\theta }}_n$, at $\theta$ is about ${\mathbb{E}}_{\theta }\left\{\rho (\theta -{\widehat{\theta }}_n)\right\}\approx r\left(\theta \right)/{\zeta }_n^{*}$. For example, in the scalar mean square error (MSE) case, where $\rho \left(\epsilon \right)={\epsilon }^2$, and where the observations are Gaussian, i.i.d., with mean $\theta$ and known variance ${\sigma }^2$, it is actually shown in Example 2.4, p. 257 in [23] that $r\left(\theta \right)={\sigma }^2$ w.r.t. ${\zeta }_n^{*}=n$, for all $\theta \in \mathbb{R}$, which is attained by the sample mean estimator, ${\widehat{\theta }}_n={\displaystyle \frac{1}{n}}{\sum }_{i=1}^n{x}_i$.

Our focus in this work is on the derivation of some new lower bounds that are as follows: (i) essentially free of regularity conditions on the smoothness of the parametric family $\left\{p\right(·\left|\theta \right),\theta \in \Theta \}$, (ii) relatively simple and easy to calculate, at least numerically, which amounts to the property that the bound contains only a small number of auxiliary parameters to be numerically optimized (typically, no more than two or three parameters), (iii) tighter than earlier reported bounds that are associated with similar calculation efforts as described in (ii), and (iv) lend themselves to extensions that yield even stronger bounds (albeit with more auxiliary parameters to be optimized), as well as extensions to vector parameters. We propose two families of lower bounds on $R_n(\Theta )$, along with their local versions, of bounding $r\left(\theta \right)$, with the four above-described properties. The first applies to any convex, symmetric loss function $\rho (·)$, whereas the second is more specifically oriented to the moments of the estimation error, $\rho \left(\epsilon \right)={\left|\epsilon \right|}^t$, where t is a positive real, not necessarily an integer, with special attention devoted to the MSE case, $t=2$. For the sake of simplicity and clarity of the exposition, in the first two main sections of the paper (Section 3 and Section 4), our focus is on the case of a scalar parameter, as most of our examples are associated with the scalar case. In Section 5, we extend some of our findings to the vector case.

To put this work in the perspective of earlier work on minimax estimation, we next briefly review some of the basic approaches in this problem area. Admittedly, considering the vast amount of literature on the subject, our review below is by no means exhaustive. For a more comprehensive review, the reader is referred to Kim [28].

First, observe the simple fact that the minimax performance is lower bounded by the Bayesian performance of the same loss function (see, e.g., [1,2,3,4,5,6,7]) for any prior on the parameter, $\theta$, and so, every lower bound on the Bayesian performance is automatically a valid lower bound also on the minimax performance (see Section 4.2 in [23]). Indeed, in Section 2.3 in [26], it is argued that the vast majority of existing minimax lower bounding techniques are based upon bounding the Bayes risk from below w.r.t. some prior. Many of these Bayesian bounds, however, are subjected to certain restrictions and regularity conditions concerning the smoothness of the prior and the family of densities, $\left\{p\right(·\left|\theta \right),\theta \in \Theta \}$.

Dating back to Ziv and Zakai’s 1969 article [18] on parameter estimation, applied mostly in the context of time-delay estimation, this prior puts all its mass equally on two values, ${\theta }_0$ and ${\theta }_1$, of the parameter $\theta$, and considering an auxiliary hypothesis testing problem of distinguishing between the two hypotheses, ${\mathcal{H}}_0:\theta ={\theta }_0$ and ${\mathcal{H}}_1:\theta ={\theta }_1$ with equal priors (see Section 2 for exact definitions). A simple argument regarding the sub-optimality of a decision rule that is based on estimating $\theta$ and deciding on the hypothesis with the closer value of $\theta$, combined with Chebychev’s inequality, yields a simple lower bound on the corresponding Bayes risk, and hence also the minimax risk, in terms of the probability of error of the optimal decision rule. Five years later, Bellini and Tartara [3], and then independently, Chazan, Zakai, and Ziv [4], improved the bound of [18] using somewhat different arguments and obtained Bayesian bounds that apply to the uniform prior. These bounds are also given in terms of the error probability pertaining to the optimal maximum a posteriori (MAP) decision rule of binary hypothesis testing with equal priors, but this time, it had an integral form. These bounds were demonstrated to be fairly tight in several application examples, but they are rather difficult to calculate in most cases. Shortly before the Bellini–Tartara and the Chazan–Zakai–Ziv articles were published, Le Cam [20] proposed a minimax lower bound, which is also given in terms of the error probability associated with binary hypothesis testing, or equivalently, the total variation between $p(·|{\theta }_0)$ and $p(·|{\theta }_1)$, under the postulate that the loss function $\mathcal{l}(·,·)$ is a metric. We will refer to Le Cam’s bound in a more detailed manner later, in the context of our first proposed bound.

A decade later, Assouad [21] extended Le Cam’s two-point testing bound to multiple points, where instead of just two test points, ${\theta }_0$ and ${\theta }_1$, as before, there are, more generally, m test points, ${\theta }_0,{\theta }_1,\dots ,{\theta }_{m-1}$, and correspondingly, the auxiliary hypothesis testing problem consists of m hypotheses, ${\mathcal{H}}_i:\theta ={\theta }_i$, $i=0,1,\dots ,m-1$, with certain priors (again, to be defined in Section 2). Based on those test points, Assouad devised the so-called hypercube method. Another related bounding technique that is based on multiple test points, and referred to as Fano’s method, amounts to further bounding from below the error probability of multiple hypotheses using Fano’s inequality (see Section 2.10 in [29]). Considering the large number of auxiliary parameters to be optimized when multiple hypotheses are present, these bounds demand heavy computational efforts. Also, Fano’s inequality is often loose, even though it is adequate enough for the purpose of proving converse-to-coding theorems in information theory [29]. In later years, Le Cam [30] and Yu [31] extended Le Cam’s original approach to apply to testing mixtures of densities. More recently, Yang and Barron [25] related the minimax problem to the metric entropy of the parametric family, $\{p(·|{\theta }_{,}\theta \in \Theta \}$, and Cai and Zhou [32] combined Le Cam’s and Assouad’s methods by considering a larger number of dimensions. Guntuboyina [26,27] pursued a different direction by deriving minimax lower bounds using f-divergences.

The outline of this article is as follows. In Section 2, we define the problem setting, provide a few formal definitions along with background, and establish the notation. In Section 3, we develop the first family of bounds, and in Section 4, we present the second family, both for the scalar parameter case. Finally, in Section 5, we extend some of our findings to the vector case.

2. Problem Setting, Background, Definitions, and Notation

As outlined in the last paragraph of the Introduction, Section 3 and Section 4 are on the scalar parameter case, whereas in Section 5, we extend some of the results to the case of a vector parameter with dimension d. In order to avoid repetition, we formalize the problem and define the notation here for the more general vector case, with the understanding that for the scalar case, all definitions remain the same, except that they are confined to the special case of $d=1$.

Consider a family of probability density functions (PDFs), $\left\{p\right(·\left|\theta \right),\theta \in \Theta \}$, where $\theta$ is a parameter vector of dimension d to be estimated, and $\Theta \subseteq {\mathbb{R}}^d$ is the parameter space. We denote by ${\mathbb{E}}_{\theta }\{·\}$ the expectation operator w.r.t. $p(·|\theta )$. Let X $=(X_1,\dots ,X_n)$ be a random vector of observations, governed by $p(·|\theta )$ for some $\theta \in \Theta$. The support of $p(·|\theta )$ is assumed ${\mathcal{X}}^n$, the nth Cartesian power of the alphabet $\mathcal{X}$ of each component $X_i$, $i=1,\dots ,n$. The alphabet $\mathcal{X}$ may be a finite set, a countable set, a finite interval, an infinite interval, or the entire real line. In the first two cases, the PDFs should be understood to be replaced by probability mass functions and integrations over the observation space should be replaced by summations. A realization of X will be denoted by $\mathit{x}=(x_1,\dots ,x_n)\in {\mathcal{X}}^n$.

An estimator, ${\widehat{\theta }}_n$, is given by any function of the observation vector, $g_n:{\mathcal{X}}^n\to \Theta$, that is, ${\widehat{\theta }}_n=g_n\left[\mathit{x}\right]$. Since X is random, so is the estimate $g_n\left[\mathit{X}\right]$, as well as the estimation error, ${\epsilon }_n\left[\mathit{X}\right]={\widehat{\theta }}_n-\theta =g_n\left[\mathit{X}\right]-\theta$. We associate with every possible vector value, $ϵ$, of ${\epsilon }_n\left[\mathit{X}\right]$ a certain loss (or “cost”, or “price”), $\rho \left(ϵ\right)$, where $\rho (·)$ is a non-negative function with the following properties: (i) monotonically non-increasing in each component, ${ϵ}_i$, of $ϵ$, wherever ${ϵ}_i<0$, $i=1,2,\dots ,d$, (ii) monotonically non-decreasing in each component, ${ϵ}_i$, of $ϵ$, wherever ${ϵ}_i\ge 0$, $i=1,2,\dots ,d$, and (iii) $\rho \left(0\right)=0$.

Referring to Section 3 and Section 4, which deals with the scalar case ($d=1$), we further adopt the following assumptions regarding the loss function $\rho$. In Section 3, we assume that $\rho (·)$ is as follows: (iv) convex, and (v) symmetric, i.e., $\rho (-ϵ)=\rho \left(ϵ\right)$ for every $ϵ$. In Section 4, we assume, more specifically, that $\rho \left(\epsilon \right)={\left|\epsilon \right|}^t$, where t is a positive constant, not necessarily an integer. This is a special case of the class of loss functions considered in Section 3, except when $t\in (0,1)$, in which case, ${\left|\epsilon \right|}^t$ is a concave (rather than a convex) function of $\epsilon$. In Section 5, for the vector extension of the results of Section 3, the above-mentioned symmetry assumption (v) is extended to become radial symmetry; namely, $\rho \left(ϵ\right)$ will be assumed to depend on $ϵ$ only via its Euclidean norm, $\parallel ϵ\parallel$.

The expected cost of an estimator $g_n$ at a point $\theta \in \Theta$, is defined as

$$R_n(\theta ,g_n)\stackrel{▵}{=}{\mathbb{E}}_{\theta }\left\{\rho (g_n\left[\mathit{X}\right]-\theta )\right\}.$$

(3)

The global minimax performance is defined as

$$R_n(\Theta )\stackrel{▵}{=}\underset{g_n}{inf}\underset{\theta \in \Theta }{sup}{R}_n(\theta ,g_n).$$

(4)

Another related notion is that of local asymptotic minimax performance, defined in Section 1, and repeated here for the sake of completeness. Let $\{{\zeta }_n^{*},n\ge 1\}$ be a positive sequence, tending to infinity, with the property that $r\left(\theta \right)$, defined as in (2), is a strictly positive finite constant. Then, we say that $r\left(\theta \right)$ is the local asymptotic minimax performance w.r.t. $\left\{{\zeta }_n^{*}\right\}$ at $\theta \in \Theta$. The sequence $\{1/{\zeta }_n^{*}\}$ is referred to as the convergence rate of the minimax estimator.

Similarly, as in earlier articles on minimax lower bounds, many of our proposed bounds are given in terms of the error probability pertaining to certain auxiliary hypothesis testing problems that are associated with two or more test points in the parameter space, and the choice of those test points is subjected to optimization. We therefore provide a few definitions, notation conventions, and background associated with elementary hypothesis testing. For further details, the reader is referred to any one of many textbooks that cover the topic, for example, Van Trees (see Sections 2.2 and 2.3 in [1]), Helstrom (see Chapter III in [33]), or Whalen (see Chapter 5 in [34]).

For given ${\theta }_0$ and ${\theta }_1$, both in $\Theta$, consider the problem of deciding between two possible hypotheses regarding the probability distribution that governs the given vector of observations, X. Under hypothesis ${\mathcal{H}}_0$, X is governed by $p(·|{\theta }_0)$, and under hypothesis ${\mathcal{H}}_1$, X is governed by $p(·|{\theta }_1)$. Suppose also that the a priori probability of ${\mathcal{H}}_0$ to be the actual underlying hypothesis is $\mathrm{Pr}\left\{{\mathcal{H}}_0\right\}=q$, where $0\le q\le 1$ is given, and so the a priori probability of ${\mathcal{H}}_1$ is the complementary probability, $\mathrm{Pr}\left\{{\mathcal{H}}_1\right\}=1-q$. When $q={\displaystyle \frac{1}{2}}$, we say that the priors are equal; otherwise, the priors are unequal. A decision rule $\Omega =({\Omega }_0,{\Omega }_1)$ is a partition of the observation space, ${\mathcal{X}}^n$, into two disjoint decision regions, ${\Omega }_0\subseteq {\mathcal{X}}^n$ and its complementary region, ${\Omega }_1={\Omega }_0^c$: Given that $\mathit{X}=\mathit{x}\in {\Omega }_0$, we decide in favor of ${\mathcal{H}}_0$; otherwise, we decide in favor of ${\mathcal{H}}_1$. The probability of error, associated with the hypotheses ${\mathcal{H}}_0$ and ${\mathcal{H}}_1$, referring to test points ${\theta }_0$ and ${\theta }_1$, respectively, with priors q and $1-q$, respectively, when using the decision rule $\Omega$, is defined as

$$P_e(q,{\theta }_0,{\theta }_1,\Omega )=q·{\int }_{{\Omega }_1}p\left(\mathit{x}\right|{\theta }_0)\mathrm{d}x+(1-q)·{\int }_{{\Omega }_0}p\left(\mathit{x}\right|{\theta }_1)\mathrm{d}x.$$

(5)

A well-known elementary result in decision theory asserts that the optimal decision rule, ${\Omega }^{*}=({\Omega }_0^{*},{\Omega }_1^{*})$, in the sense of minimizing $P_e(q,{\theta }_0,{\theta }_1,\Omega )$, is given by

$$\begin{array}{ccc}\hfill {\Omega }_0^{*}& =& \{\mathit{x}\in {\mathcal{X}}^n:q·p\left(\mathit{x}\right|{\theta }_0)\ge (1-q)·p\left(\mathit{x}\right|{\theta }_1)\}\hfill \end{array}$$

(6)

$$\begin{array}{ccc}\hfill {\Omega }_1^{*}& =& \{\mathit{x}\in {\mathcal{X}}^n:q·p\left(\mathit{x}\right|{\theta }_0)<(1-q)·p\left(\mathit{x}\right|{\theta }_1)\},\hfill \end{array}$$

(7)

where it should be pointed out that attributing the case of a tie, $q·\left(\mathit{x}\right|{\theta }_0)=(1-q)·p\left(\mathit{x}\right|{\theta }_1)$, to ${\Omega }_0^{*}$ is completely arbitrary, and could have been attributed alternatively to ${\Omega }_1^{*}$ without affecting the probability of error. Here and in the sequel, the minimum probability of error, associated with ${\Omega }^{*}$, namely, $P_e(q,{\theta }_0,{\theta }_1,{\Omega }^{*})$, will be denoted more simply by $P_e(q,{\theta }_0,{\theta }_1)$. On substituting ${\Omega }^{*}$ into the above definition of the probability of error, we obtain

$$\begin{array}{ccc}\hfill P_e(q,{\theta }_0,{\theta }_1)& =& q·{\int }_{\{\mathit{x}:q·p\left(\mathit{x}\right|{\theta }_0)<(1-q)·p\left(\mathit{x}\right|{\theta }_1)\}}p\left(\mathit{x}\right|{\theta }_0)\mathrm{d}x+\hfill \\ & & (1-q)·{\int }_{\{\mathit{x}:q·p\left(\mathit{x}\right|{\theta }_0)\ge (1-q)·p\left(\mathit{x}\right|{\theta }_1)\}}p\left(\mathit{x}\right|{\theta }_1)\mathrm{d}x\hfill \\ & =& {\int }_{\{\mathit{x}:q·p\left(\mathit{x}\right|{\theta }_0)<(1-q)·p\left(\mathit{x}\right|{\theta }_1)\}}q·p\left(\mathit{x}\right|{\theta }_0)\mathrm{d}x+\hfill \\ & & {\int }_{\{\mathit{x}:q·p\left(\mathit{x}\right|{\theta }_0)\ge (1-q)·p\left(\mathit{x}\right|{\theta }_1)\}}(1-q)·p\left(\mathit{x}\right|{\theta }_1)\mathrm{d}x\hfill \\ & =& {\int }_{\{\mathit{x}:q·p\left(\mathit{x}\right|{\theta }_0)<(1-q)·p\left(\mathit{x}\right|{\theta }_1)\}}min\{q·p\left(\mathit{x}\right|{\theta }_0),(1-q)p\left(\mathit{x}\right|{\theta }_1)\}\mathrm{d}x+\hfill \\ & & {\int }_{\{\mathit{x}:q·p\left(\mathit{x}\right|{\theta }_0)\ge (1-q)·p\left(\mathit{x}\right|{\theta }_1)\}}min\{q·p\left(\mathit{x}\right|{\theta }_0),(1-q)·p\left(\mathit{x}\right|{\theta }_1)\}\mathrm{d}x\hfill \\ & =& {\int }_{{\mathcal{X}}^n}min\{q·p\left(\mathit{x}\right|{\theta }_0),(1-q)·p\left(\mathit{x}\right|{\theta }_1)\}\mathrm{d}\mathit{x}\hfill \\ & =& 1-{\int }_{{\mathcal{X}}^n}max\{q·p\left(\mathit{x}\right|{\theta }_0),(1-q)·p\left(\mathit{x}\right|{\theta }_1)\}\mathrm{d}\mathit{x}.\hfill \end{array}$$

(8)

The expression of the second to the last line will appear in some of our lower bounds in the sequel and will be recognized and interpreted as $P_e(q,{\theta }_0,{\theta }_1)$. The expression of the last line is the one that extends to multiple hypothesis testing, as will be detailed below.

The auxiliary problem of binary hypothesis testing extends from two hypotheses to a general number, m, of hypotheses, associated with m test points, ${\theta }_0,{\theta }_1,\dots ,{\theta }_{m-1}$, in the following manner. Under hypothesis ${\mathcal{H}}_i$, the observation vector X is governed by $p(·|{\theta }_i)$ and the a priori probability of ${\mathcal{H}}_i$ being the underlying true hypothesis, $\mathrm{Pr}\left\{{\mathcal{H}}_i\right\}$, is denoted $q_i$, $i=0,1,\dots ,m-1$, where $q_0,q_1,\dots ,q_{m-1}$ are given non-negative numbers summing to unity. Here, a decision rule $\Omega =({\Omega }_0,{\Omega }_1,\dots ,{\Omega }_{m-1})$ is a partition of ${\mathcal{X}}^n$ into m disjoint regions such that if $\mathit{x}\in {\Omega }_i$, we decide in favor of ${\mathcal{H}}_i$, $i=0,1,\dots ,m-1$. The probability of error associated with $\Omega$, ${\theta }_0,{\theta }_1,\dots ,{\theta }_{m-1}$ and $\mathit{q}=(q_0,q_1,\dots ,q_{m-1})$ is defined as

$$P_e(\mathit{q},{\theta }_0,{\theta }_1,\dots ,{\theta }_{m-1},\Omega )=\sum _{i=0}^{m-1}{q}_i{\int }_{{\Omega }_i^c}p\left(\mathit{x}\right|{\theta }_i)\mathrm{d}\mathit{x},$$

(9)

where ${\Omega }_i^c$ is complementary to ${\Omega }_i$. The optimal MAP decision rule ${\Omega }^{*}$ selects the hypothesis ${\mathcal{H}}_i$ whose index i maximizes the product $q_i·p\left(\mathit{x}\right|{\theta }_i)$ among all $i\in \{0,1,\dots ,m-1\}$, for the given x, where ties are broken arbitrarily. The probability of error, associated with ${\Omega }^{*}$, denoted $P_e(\mathit{q},{\theta }_0,{\theta }_1,\dots ,{\theta }_{m-1})$, is well known to be given by

$$P_e(\mathit{q},{\theta }_0,{\theta }_1,\dots ,{\theta }_{m-1})=1-{\int }_{{\mathcal{X}}^n}max\{q_0·p\left(\mathit{x}\right|{\theta }_0,q_1·p\left(\mathit{x}\right|{\theta }_1),\dots ,q_{m-1}·p\left(\mathit{x}\right|{\theta }_{m-1})\}\mathrm{d}\mathit{x}.$$

(10)

Note that for $m>2$, this is different from the expression

$${\int }_{{\mathcal{X}}^n}min\{q_0·p\left(\mathit{x}\right|{\theta }_0,q_1·p\left(\mathit{x}\right|{\theta }_1),\dots ,q_{m-1}·p\left(\mathit{x}\right|{\theta }_{m-1})\}\mathrm{d}\mathit{x},$$

as the latter can be interpreted as the probability that the index i of the true hypothesis ${\mathcal{H}}_i$ minimizes (rather than maximizes) the product $q_{i}p\left(\mathit{x}\right|{\theta }_i)$ over $i\in \{0,1,\dots ,m-1\}$ for the given x. Imagine an observer that, upon observing a realization x of the random vector X, creates a list of indices $\left\{i\right\}$ with the k largest values of $q_{i}p\left(\mathit{x}\right|{\theta }_i)$ for some $k<m$, and an error is defined as the event where the correct i is not in that list. This is referred to as a list error, which is a term borrowed from the fields of coded communication and information theory. The last expression is the probability of list error for $k=m-1$. We will encounter this expression later in certain versions of our lower bound. This completes the background needed about hypothesis testing.

As described in Section 1, our objective in this work is to derive relatively simple and easily computable lower bounds to $r\left(\theta \right)$, which are as tight as possible. While many existing lower bounds in the literature are satisfactory in terms of yielding the correct rate of convergence, $1/{\zeta }_n^{*}$, here we wish to improve the bound on the constant factor, $r\left(\theta \right)$. Many of our examples involve numerical calculations which include optimization over auxiliary parameters and occasionally also numerical integrations. All these calculations were carried out using MATLAB R2019a (9.6.0.1072779) 64-bit (win64).

3. Lower Bounds for Convex Symmetric Loss Functions

As explained earlier, here and in Section 4, we consider a scalar parameter, namely, $d=1$.

Theorem 1.

Let the assumptions of Section 2 be satisfied for $d=1$ and let $\rho (·)$ be a symmetric convex loss function. Then,

$$R_n(\Theta )\ge \underset{{\theta }_0,{\theta }_1\in \Theta }{sup}\left\{2·\rho \left({\displaystyle \frac{{\theta }_1-{\theta }_0}{2}}\right)·\underset{0\le q\le 1}{sup}{P}_e(q,{\theta }_0,{\theta }_1)\right\},$$

(11)

where $P_e(q,{\theta }_0,{\theta }_1)$ is defined as in Equation (8).

Proof of Theorem 1.

For every ${\theta }_0,{\theta }_1\in \Theta$ and $q\in [0,1]$,

$$\begin{array}{ccc}\hfill R_n(\Theta )& \ge & q{\mathbb{E}}_{{\theta }_0}\left\{\rho (g_n\left[\mathit{X}\right]-{\theta }_0)\right\}+(1-q){\mathbb{E}}_{{\theta }_1}\left\{\rho (g_n\left[\mathit{X}\right]-{\theta }_1)\right\}\hfill \\ & \stackrel{\left(\mathrm{a}\right)}{=}& {\int }_{{\mathbb{R}}^n}[q·p\left(\mathit{x}\right|{\theta }_0)\rho (g_n\left[\mathit{x}\right]-{\theta }_0)+\hfill \\ & & (1-q)·p\left(\mathit{x}\right|{\theta }_1)\rho ({\theta }_1-g_n\left[\mathit{x}\right])]\mathrm{d}\mathit{x}\hfill \\ & \ge & 2·{\int }_{{\mathbb{R}}^n}min\{q·p\left(\mathit{x}\right|{\theta }_0),(1-q)·p\left(\mathit{x}\right|{\theta }_1)\}\times \hfill \\ & & \left[{\displaystyle \frac{1}{2}}\rho (g_n\left[\mathit{x}\right]-{\theta }_0)+{\displaystyle \frac{1}{2}}\rho ({\theta }_1-g_n\left[\mathit{x}\right])\right]\mathrm{d}\mathit{x}\hfill \\ & \stackrel{\left(\mathrm{b}\right)}{\ge }& 2·{\int }_{{\mathbb{R}}^n}min\{q·p\left(\mathit{x}\right|{\theta }_0),(1-q)·p\left(\mathit{x}\right|{\theta }_1)\}\times \hfill \\ & & \rho \left({\displaystyle \frac{g_n\left[\mathit{x}\right]-{\theta }_0}{2}}+{\displaystyle \frac{{\theta }_1-g_n\left[\mathit{x}\right]}{2}}\right)\mathrm{d}\mathit{x}\hfill \\ & =& 2·\rho \left({\displaystyle \frac{{\theta }_1-{\theta }_0}{2}}\right)·{\int }_{{\mathbb{R}}^n}min\{q·p\left(\mathit{x}\right|{\theta }_0),(1-q)·p\left(\mathit{x}\right|{\theta }_1)\}\mathrm{d}\mathit{x}\hfill \\ & =& 2·\rho \left({\displaystyle \frac{{\theta }_1-{\theta }_0}{2}}\right)·P_e(q,{\theta }_0,{\theta }_1),\hfill \end{array}$$

(12)

where (a) is due to the assumed symmetry of $\rho (·)$ and (b) is by its assumed convexity. Since the inequality,

$$R_n(\Theta )\ge 2·\rho \left({\displaystyle \frac{{\theta }_1-{\theta }_0}{2}}\right)·P_e(q,{\theta }_0,{\theta }_1)$$

(13)

applies to every ${\theta }_0,{\theta }_1\in \Theta$ and $q\in [0,1]$, it applies, in particular, also to the supremum over these auxiliary parameters. This completes the proof of Theorem 1. □

Before we proceed, two comments are in order:

• Note that $P_e(q,{\theta }_0,{\theta }_1)$ is a concave function of q for fixed $({\theta }_0,{\theta }_1)$, as it can be presented as the minimum among a family of affine functions of q, given by ${min}_{\Omega }[q·P(\Omega |{\theta }_0)+(1-q)·P\left({\Omega }^c\right|{\theta }_1)]$, where $\Omega$ runs over all possible subsets of the observation space, ${\mathcal{X}}^n$. Another way to see why this is true is by observing that $P_e(q,{\theta }_0,{\theta }_1)$ is given in the second to the last line of (8) by an integral, whose integrand, $min\{q·p\left(\mathit{x}\right|{\theta }_0),(1-q)·p\left(\mathit{x}\right|{\theta }_1)\}$, is concave in q. Clearly, $P_e(q,{\theta }_0,{\theta }_1)=0$ whenever $q=0$ or $q=1$. Thus, $P_e(q,{\theta }_0,{\theta }_1)$ is maximized by some q between 0 and 1. If $P_e(q,{\theta }_0,{\theta }_1)$ is strictly concave in q, then the maximizing q is unique.
• Note that the lower bound (11) is tighter than the lower bound of $\rho \left({\displaystyle \frac{{\theta }_1-{\theta }_0}{2}}\right)P_e({\displaystyle \frac{1}{2}},{\theta }_0,{\theta }_1)$, which was obtained in Equations (6)–(9a) in [18], both because of the factor of 2 and because of the freedom to optimize q rather than setting $q=1/2$. In a further development of [18] the factor of 2 was accomplished too, but at the price of assuming that the density of the estimation error is symmetric about the origin (see discussion after (10) therein), which limits the class of estimators to which the bound applies. The factor of 2 and the degree of freedom q are also the two ingredients that make the difference between (11) and the lower bound due to Le Cam [20] (see also [26,28]). In Chapter 2 in [26] Guntuboyina reviews standard bounding techniques, including those of Le Cam, Assouad, and Fano. In particular, in Example 2.3.2 therein, Guntiboyina presents a lower bound in terms of the error probability associated with general priors, given by ${\displaystyle \frac{\eta }{2}}·P_e(q,{\theta }_0,{\theta }_1)$, where in the case of two hypotheses, $\eta ={min}_{\vartheta }\{\rho ({\theta }_0-\vartheta )+\rho ({\theta }_1-\vartheta )\}$, in our notation. Now, if $\rho$ is symmetric and monotonically non-decreasing in the absolute error, then the minimizing $\vartheta$ is given by ${\displaystyle \frac{{\theta }_0+{\theta }_1}{2}}$, which yields ${\displaystyle \frac{\eta }{2}}=\rho \left({\displaystyle \frac{{\theta }_1-{\theta }_0}{2}}\right)$ and so, again, the resulting bound is of the same form as (11) except that it lacks the prefactor of 2.

Our first example demonstrates Theorem 1 on a somewhat technical but simple model, with an emphasis on the point that the optimal q may differ from $1/2$ and that it is therefore useful to maximize w.r.t. q in order to improve the bound relative to the choice $q=1/2$.

Example 1.

Let X be a random variable distributed exponentially according to

$$p\left(x\right|\theta )=\theta e^{-\theta x},x\ge 0,$$

(14)

and $\Theta =\{1,2\}$, so that the only possibility to select two different values of θ in the lower bound are ${\theta }_0=1$ and ${\theta }_1=2$. In terms of the hypothesis testing problem pertaining to the lower bound, the likelihood ratio test (LRT) is by comparison of $q{e}^{-x}$ to $(1-q)·2e^{-2x}$. Now, if $2(1-q)\le q$, or equivalently, $1\ge q\ge {\displaystyle \frac{2}{3}}$, the decision is always in favor of ${\mathcal{H}}_0$, and then $P_e(q,1,2)=1-q$. For $0\le q<{\displaystyle \frac{2}{3}}$, the optimal LRT compares X to $x_0\left(q\right)=ln{\displaystyle \frac{2(1-q)}{q}}$. If $X>x_0\left(q\right)$, one decides in favor of ${\mathcal{H}}_0$; otherwise, one decides in favor of ${\mathcal{H}}_1$. Thus,

$$\begin{array}{ccc}\hfill P_e(q,1,2)& =& q{\int }_0^{x_0\left(q\right)}{e}^{-x}dx+(1-q){\int }_{x_0\left(q\right)}^{\infty }2e^{-2x}dx\hfill \\ & =& q[1-e^{-x_0\left(q\right)}]+(1-q)e^{-2x_0\left(q\right)}\hfill \\ & =& q\left[1-{\displaystyle \frac{q}{2(1-q)}}\right]+(1-q){\left[{\displaystyle \frac{q}{2(1-q)}}\right]}^2\hfill \\ & =& {\displaystyle \frac{q(4-5q)}{4(1-q)}}.\hfill \end{array}$$

(15)

In summary,

$$P_e\left(q,1,2\right)=\left\{\begin{array}{cc}{\displaystyle \frac{q(4-5q)}{4(1-q)}}\hfill & 0\le q<{\displaystyle \frac{2}{3}}\hfill \\ 1-q\hfill & {\displaystyle \frac{2}{3}}\le q\le 1\hfill \end{array}\right.$$

(16)

It turns out that for $q={\displaystyle \frac{1}{2}}$, $P_e({\displaystyle \frac{1}{2}},1,2)={\displaystyle \frac{3}{8}}=0.375$, whereas the maximum is $0.382$, attained at $q=1-{\displaystyle \frac{\sqrt{5}}{5}}=0.5528$. Thus,

$$R_n(\Theta )\ge 2·0.382·\rho \left({\displaystyle \frac{2-1}{2}}\right)=0.764·\rho \left({\displaystyle \frac{1}{2}}\right).$$

(17)

This concludes Example 1.

In the above example, we considered just one observation, $n=1$. From now on, we will refer to the case where $n\gg 1$. In particular, the following simple corollary to Theorem 1 yields a local asymptotic minimax lower bound.

Corollary 1.

For a given $\theta \in \Theta$ and a constant s, let ${\left\{{\xi }_n\right\}}_{n\ge 1}$ denote a positive sequence tending to zero with the property that

$$\underset{n\to \infty }{lim}\underset{q}{max}{P}_e(q,\theta ,\theta +2s{\xi }_n)$$

(18)

exists and is given by a strictly positive constant, which will be denoted by $P_e^{\infty }(\theta ,s)$. Also, let

$$\omega \left(s\right)\stackrel{▵}{=}\underset{u\to 0}{lim}{\displaystyle \frac{\rho (s·u)}{\rho \left(u\right)}}.$$

(19)

Then, the local asymptotic minimax performance w.r.t. ${\zeta }_n=1/\rho \left({\xi }_n\right)$ is lower bounded by

$$r\left(\theta \right)\ge \underset{s\in \mathbb{R}}{sup}\left\{2\omega \left(s\right)·P_e^{\infty }(\theta ,s)\right\}.$$

(20)

Corollary 1 is readily obtained from Theorem 1 by substituting ${\theta }_0=\theta$ and ${\theta }_1=\theta +2s{\xi }_n$ in Equation (11), then multiplying both sides of the inequality by ${\zeta }_n=1/\rho \left({\xi }_n\right)$, and finally, taking the limit inferior of both sides.

Next, we study a few examples of the use of Corollary 1. As in Example 1, we emphasize again in Example 2 below the importance of having the degree of freedom to maximize over the prior q rather than to fix $q={\displaystyle \frac{1}{2}}$. Also, in all the examples that were examined, the rate of convergence, $1/{\zeta }_n=\rho \left({\xi }_n\right)$, is the same as the optimal rate of convergence, $1/{\zeta }_n^{*}$. In other words, it is tight in the sense that there exists an estimator (for example, the maximum likelihood estimator) for which $R_n(\theta ,g_n)$ tends to zero at the same rate. In some of these examples, we compare our lower bound to $r\left(\theta \right)$ to those of earlier reported results on the same models.

Example 2.

Let $X_1,\dots ,X_n$ be independently, identically distributed (i.i.d.) random variables, uniformly distributed in the range $[0,\theta ]$. In the corresponding hypothesis testing problem of Theorem 1, the hypotheses are $\theta ={\theta }_0$ and $\theta ={\theta }_1>{\theta }_0$ with priors, q and $1-q$. There are two cases: If $q/{\theta }_0^n<(1-q)/{\theta }_1^n$, or equivalently, $q<{\theta }_0^n/({\theta }_0^n+{\theta }_1^n)$, one decides always in favor of ${\mathcal{H}}_1$ and so, the probability of error is q. If, on the other hand, $q/{\theta }_0^n>(1-q)/{\theta }_1^n$, namely, $q>{\theta }_0^n/({\theta }_0^n+{\theta }_1^n)$, we decide in favor of ${\mathcal{H}}_1$ whenever ${max}_i{X}_i>{\theta }_0$ and then an error occurs only if ${\mathcal{H}}_1$ is true, yet ${max}_i{X}_i<{\theta }_0$, which happens with probability $(1-q){\left({\displaystyle \frac{{\theta }_0}{{\theta }_1}}\right)}^n$. Thus,

$$P_e(q,{\theta }_0,{\theta }_1)=\left\{\begin{array}{cc}q\hfill & q<{\displaystyle \frac{{\theta }_0^n}{{\theta }_0^n+{\theta }_1^n}}\hfill \\ (1-q){\left({\displaystyle \frac{{\theta }_0}{{\theta }_1}}\right)}^n\hfill & q\ge {\displaystyle \frac{{\theta }_0^n}{{\theta }_0^n+{\theta }_1^n}}\hfill \end{array}\right.=min\left\{q,(1-q){\left({\displaystyle \frac{{\theta }_0}{{\theta }_1}}\right)}^n\right\},$$

(21)

which is readily seen to be maximized by $q={\displaystyle \frac{{\theta }_0^n}{{\theta }_0^n+{\theta }_1^n}}$ and then

$$\underset{q}{max}{P}_e(q,{\theta }_0,{\theta }_1)={\displaystyle \frac{{\theta }_0^n}{{\theta }_0^n+{\theta }_1^n}}.$$

(22)

Now, to apply Corollary 1, we let ${\theta }_0=\theta$ and ${\theta }_1={\theta }_0(1+2\sigma /n)$, which amounts to $s={\theta }_0\sigma =\theta \sigma$ and ${\xi }_n=1/n$. Then,

$$P_e^{\infty }(\theta ,s)=\underset{n\to \infty }{lim}{\displaystyle \frac{1}{1+{[1+2s/\left(\theta n\right)]}^n}}={\displaystyle \frac{1}{1+e^{2s/\theta }}}.$$

(23)

In the case of the MSE criterion, $\rho \left(\epsilon \right)={\epsilon }^2$, we have $\omega \left(s\right)=s^2$, and so,

$$r\left(\theta \right)\ge \underset{s\ge 0}{sup}{\displaystyle \frac{2s^2}{1+e^{2s/\theta }}}={\theta }^2·\underset{u\ge 0}{sup}{\displaystyle \frac{u^2}{2(1+e^u)}}\approx 0.2414{\theta }^2$$

(24)

w.r.t. ${\zeta }_n=1/\rho \left({\xi }_n\right)=1/{(1/n)}^2=n^2$. This bound will be further improved upon in Section 4.

If instead of maximizing w.r.t. q, we select $q=1/2$, then

$$P_e\left({\displaystyle \frac{1}{2}},\theta ,\theta \left(1+{\displaystyle \frac{2\sigma }{n}}\right)\right)={\displaystyle \frac{1}{2}}·{\left[{\displaystyle \frac{\theta }{\theta (1+2\sigma /n)}}\right]}^n\to {\displaystyle \frac{1}{2}}·e^{-2\sigma }={\displaystyle \frac{1}{2}}·e^{-2s/\theta },$$

(25)

and then the resulting bound would become

$$r\left(\theta \right)\ge \underset{s\ge 0}{sup}{s}^2{e}^{-2s/\theta }={\theta }^2·\underset{u\ge 0}{sup}{\displaystyle \frac{u^2{e}^{-u}}{4}}\approx 0.1353{\theta }^2$$

(26)

w.r.t. ${\zeta }_n=n^2$. Therefore, the maximization over q plays an important role here in terms of tightening the lower bound to $r\left(\theta \right)$.

More generally, for $\rho \left(\epsilon \right)={\left|\epsilon \right|}^t$ ($t\ge 1$), $\omega \left(s\right)={\left|s\right|}^t$, and we obtain

$$r\left(\theta \right)\ge {\theta }^t·\underset{u\ge 0}{sup}{\displaystyle \frac{2u^t}{1+e^{2u}}},$$

(27)

w.r.t. ${\zeta }_n=n^t$, where the supremum, which is in fact a maximum, can always be calculated numerically for every given t. For large t, the maximizing u is approximately $t/2$, which yields

$$r\left(\theta \right)\ge {\displaystyle \frac{{\left(t\theta \right)}^t}{2^{t-1}(1+e^t)}}.$$

(28)

On the other hand, for $q=1/2$, we end up with

$$r\left(\theta \right)\ge \underset{s\ge 0}{sup}{s}^t{e}^{-2s/\theta }={\left({\displaystyle \frac{t\theta }{2e}}\right)}^t.$$

(29)

For large t, the bound of $q=1/2$ is inferior to the bound with the optimal q, by a factor of about $1/2$. This concludes Example 2.

Example 3.

Let $X_1,X_2,\dots ,X_n$ be i.i.d. random variables, uniformly distributed in the interval $[\theta ,\theta +1]$. For the hypothesis testing problem, let ${\theta }_1$ be chosen between ${\theta }_0$ and ${\theta }_0+1$. Clearly, if ${min}_i{X}_i<{\theta }_1$, the underlying hypothesis is certainly ${\mathcal{H}}_1$. Likewise, if ${max}_i{X}_i>{\theta }_0+1$, the decision is in favor of ${\mathcal{H}}_0$ with certainty. Thus, an error can occur only if all $\left\{X_i\right\}$ fall in the interval $[{\theta }_1,{\theta }_0+1]$, an event that occurs with probability ${({\theta }_0+1-{\theta }_1)}^n$. In this event, the best to do is to select the hypothesis with the larger prior with a probability of error given by $min\{q,1-q\}$. Thus,

$$P_e\left(q,{\theta }_0,{\theta }_1\right)={({\theta }_0+1-{\theta }_1)}^n·min\{q,1-q\},$$

(30)

and so,

$$\underset{q}{max}{P}_e\left(q,{\theta }_0,{\theta }_1\right)={\displaystyle \frac{1}{2}}{[1-({\theta }_1-{\theta }_0)]}^n,$$

(31)

achieved by $q=1/2$. Now, let us select ${\xi }_n=1/n$, which yields

$$\underset{n\to \infty }{lim}\underset{q}{max}{P}_e\left(q,{\theta }_0,{\theta }_1\right)=\underset{n\to \infty }{lim}{\displaystyle \frac{1}{2}}·{\left(1-{\displaystyle \frac{2s}{n}}\right)}^n={\displaystyle \frac{1}{2}}·e^{-2s}.$$

(32)

For $\rho \left(\epsilon \right)={\left|ϵ\right|}^t$, ($t\ge 1$), we have

$$\begin{array}{c}\hfill r\left(\theta \right)\ge \underset{s\ge 0}{sup}2s^t·{\displaystyle \frac{1}{2}}{e}^{-2s}=\underset{s\ge 0}{sup}{s}^t{e}^{-2s}={\left({\displaystyle \frac{t}{2e}}\right)}^t\end{array}$$

(33)

w.r.t. ${\zeta }_n=1/{|1/n|}^t=n^t$. For the case of MSE, $t=2$, $r\left(\theta \right)\ge e^{-2}\approx 0.1353$. The constant $0.1353$ should be compared with ${\displaystyle \frac{1-1/\sqrt{2}}{128}}\approx 0.0023$ (see Example 4.9 in [35]), which is two orders of magnitude smaller. This concludes Example 3.

Example 4.

Let $X_i=\theta +Z_i$, where $\left\{Z_i\right\}$ are i.i.d., Gaussian random variables with zero mean and variance ${\sigma }^2$. Here, for the corresponding binary hypothesis testing problem, the optimal value of q is always $q^{*}={\displaystyle \frac{1}{2}}$. This can be readily seen from the concavity of $P_e(q,{\theta }_0,{\theta }_1)$ in q and its symmetry around $q=1/2$, as $P_e(q,{\theta }_0,{\theta }_1)=P_e(1-q,{\theta }_0,{\theta }_1)$. Since

$$\begin{array}{ccc}\hfill P_e\left({\displaystyle \frac{1}{2}},{\theta }_0,{\theta }_1\right)& =& Pr\left\{\sum _{i=1}^n{Z}_i\ge {\displaystyle \frac{n({\theta }_1-{\theta }_0)}{2}}\right\}\hfill \\ & =& Pr\left\{{\displaystyle \frac{1}{\sigma \sqrt{n}}}\sum _{i=1}^n{Z}_i\ge {\displaystyle \frac{\sqrt{n}({\theta }_1-{\theta }_0)}{2\sigma }}\right\}\hfill \\ & =& Q\left({\displaystyle \frac{\sqrt{n}({\theta }_1-{\theta }_0)}{2\sigma }}\right),\hfill \end{array}$$

(34)

where

$$Q\left(t\right)\stackrel{▵}{=}{\int }_t^{\infty }{\displaystyle \frac{e^{-u^2/2}du}{\sqrt{2\pi }}},$$

(35)

we select ${\xi }_n={\displaystyle \frac{1}{\sqrt{n}}}$, which yields ${\theta }_1-{\theta }_0={\displaystyle \frac{2s}{\sqrt{n}}}$

$$P_e^{\infty }(\theta ,s)=Q\left({\displaystyle \frac{s}{\sigma }}\right),$$

(36)

and then for the MSE case, $\omega \left(s\right)=s^2$,

$$r\left(\theta \right)\ge \underset{s\ge 0}{sup}\left\{2s^{2}Q\left({\displaystyle \frac{s}{\sigma }}\right)\right\}={\sigma }^2·\underset{u\ge 0}{sup}\left\{2u^{2}Q\left(u\right)\right\}\approx 0.3314{\sigma }^2$$

(37)

w.r.t. ${\zeta }_n=1/{(1/\sqrt{n})}^2=n$, and so, the asymptotic lower bound to $R_n(\theta ,g_n)$ is $0.3314{\sigma }^2/n$.

We now compare this bound (which will be further improved in Section 4) with a few earlier reported results. In one of the versions of Le Cam’s bound (see Example 4.7 in [35]) for the same model, the lower bound to $r\left(\theta \right)$ turns out to be ${\displaystyle \frac{{\sigma }^2}{24}}=0.0417{\sigma }^2$, namely, an order of magnitude smaller. Also, in Example 3.1 in [28], another version of Le Cam’s method yields $r\left(\theta \right)\ge (1-\sqrt{1/2}){\sigma }^2/8=0.0366{\sigma }^2$. According to Corollary 4.3 in [36], $r\left(\theta \right)\ge {\displaystyle \frac{{\sigma }^2}{8e}}=0.046{\sigma }^2$. Yet another comparison is with Theorem 5.9 in [37], where we find an inequality, which in our notation reads as follows:

$$\underset{\theta \in \Theta }{sup}{P}_{\theta }\left\{{(g_n\left[\mathit{X}\right]-\theta )}^2\ge {\displaystyle \frac{2\alpha {\sigma }^2}{n}}\right\}\ge {\displaystyle \frac{1}{2}}-\alpha \alpha \in \left(0,{\displaystyle \frac{1}{2}}\right).$$

(38)

Combining it with Chebychev’s inequality yields

$$\underset{\theta \in \Theta }{sup}{\mathbb{E}}_{\theta }{(g_n\left[\mathit{X}\right]-\theta )}^2\ge {\displaystyle \frac{{\sigma }^2}{n}}·\underset{0\le \alpha \le 1/2}{max}\alpha (1-2\alpha )={\displaystyle \frac{0.125{\sigma }^2}{n}}.$$

(39)

In [23] (p. 257), it is shown that when $\Theta =\mathbb{R}$, the minimax estimator for this model is the sample mean, and so, in this case, the correct constant in front of ${\sigma }^2$ is actually 1. This concludes Example 4.

The next example is related to Example 4, as it is based on the use of the central limit theorem (CLT), which means that the Gaussian tail distribution is used here too.

Example 5.

Consider an exponential family,

$$p\left(x\right|\theta )=\prod _{i=1}^{n}p\left(x_i\right|\theta )=\prod _{i=1}^n{\displaystyle \frac{e^{\theta T\left(x_i\right)}}{Z\left(\theta \right)}}={\displaystyle \frac{exp\left\{\theta {\sum }_{i=1}^{n}T\left(x_i\right)\right\}}{Z^n\left(\theta \right)}}.$$

(40)

where $T(·)$ is a given function and $Z\left(\theta \right)$ is a normalization function given by

$$Z\left(\theta \right)={\int }_{\mathbb{R}}{e}^{\theta T\left(x\right)}dx,$$

(41)

assuming that the integral converges. In the auxiliary binary hypothesis problem, the test statistic is ${\sum }_{i=1}^{n}T\left(X_i\right)$. If $q=1/2$, ${\xi }_n=1/\sqrt{n}$ and ${\theta }_1-{\theta }_0=2s/\sqrt{n}$, the LRT amounts to examining whether ${\sum }_{i=1}^n[T\left(X_i\right)-{\mathbb{E}}_{\theta }\left\{T\left(X_i\right)\right\}]$ is larger than

$${\displaystyle \frac{s}{\sqrt{n}}}·n{\displaystyle \frac{d^{2}lnZ\left(\theta \right)}{d{\theta }^2}}=s\sqrt{n}·{\displaystyle \frac{d^{2}lnZ\left(\theta \right)}{d{\theta }^2}}.$$

In this case, the probability of error can be asymptotically assessed using the CLT, which after a simple algebraic manipulation, becomes:

$$\begin{array}{ccc}\hfill P_e^{\infty }(\theta ,s)& =& Q\left({\displaystyle \frac{s{d}^{2}lnZ\left(\theta \right)/d{\theta }^2}{\sqrt{d^{2}lnZ\left(\theta \right)/d{\theta }^2}}}\right)\hfill \\ & =& Q\left(s\sqrt{{\displaystyle \frac{d^{2}lnZ\left(\theta \right)}{d{\theta }^2}}}\right)\hfill \\ & =& Q\left(s\sqrt{I\left(\theta \right)}\right),\hfill \end{array}$$

(42)

where $I\left(\theta \right)$ is the Fisher information. Thus, for the MSE,

$$r\left(\theta \right)\ge \underset{s\ge 0}{sup}2s^{2}Q\left(s\sqrt{I\left(\theta \right)}\right)\approx {\displaystyle \frac{0.3314}{I\left(\theta \right)}}$$

(43)

w.r.t. ${\zeta }_n=1/{(1/\sqrt{n})}^2=n$. This concludes Example 5.

In several earlier examples, our bound was shown to outperform (sometimes significantly so) earlier reported bounds for the corresponding models. However, to be honest and fair, we should not ignore the fact that there are also situations where our bound may not be tighter than earlier bounds applied to the same model. Such a case is demonstrated in Example 6 below, where our result is compared to those of Ziv and Zakai [18] and Chazan, Zakai, and Ziv [4] in the context of estimating the delay of a known continuous-time signal corrupted by additive white Gaussian noise (for further developments and applications of the Ziv–Zakai and the Chazan–Zakai–Ziv bounds, see, e.g., [38,39,40,41,42,43,44,45] and references therein). Having said that, it should also be kept in mind that our emphasis in this work is on bounds that are relatively simple and easy to calculate (at least numerically), whereas the Chazan–Zakai–Ziv bound, although very tight in many situations, is notoriously difficult to calculate. Indeed, in this setting, the explicit behavior of the resulting complicated bound of [4] is clearly transparent only at high values of the signal-to-noise ratio (SNR)—see Equations (11), (12), and (14) in [4].

Example 6.

Let $X\left(t\right)=s(t,\theta )+N\left(t\right)$, $t\in [0,T]$, where $N\left(t\right)$ is additive white Gaussian noise (AWGN) with double-sided spectral density $N_0$ and $s(t,\theta )$ is a deterministic signal that depends on the unknown parameter, θ. It is assumed that the signal energy, $E={\int }_0^T{s}^2(t,\theta )dt$, does not depend on θ (which is the case, for example, when θ is a delay parameter of a pulse fully contained in the observation interval, or when θ is the frequency or the phase of a sinusoidal waveform). We further assume that $s(t,\theta )$ is at least twice differentiable w.r.t. θ, and that the energies of the first two derivatives are also independent of θ. Then, as shown in Appendix A, for small $|{\theta }_1-{\theta }_0|$,

$$\begin{array}{ccc}\hfill \varrho ({\theta }_0,{\theta }_1)& \stackrel{▵}{=}& {\displaystyle \frac{1}{E}}{\int }_0^{T}s(t,{\theta }_0)s(t,{\theta }_1)dt\hfill \\ & =& 1-{\displaystyle \frac{{({\theta }_1-{\theta }_0)}^2}{2E}}{\int }_0^T{\left[{\displaystyle \frac{\partial s(t,\theta )}{\partial \theta }}{|}_{\theta ={\theta }_0}\right]}^{2}dt+o(|{\theta }_1-{\theta }_0{|}^2)\hfill \\ & =& 1-{\displaystyle \frac{{({\theta }_1-{\theta }_0)}^2\dot{E}}{2E}}+o(|{\theta }_1-{\theta }_0{|}^2),\hfill \end{array}$$

(44)

where $\dot{E}$ is the energy $\dot{s}(t,\theta )=\partial s(t,\theta )/\partial \theta$.

The optimal LRT in deciding between the two hypotheses is based on the comparison between the correlations, ${\int }_0^{T}X\left(t\right)s(t,{\theta }_0)dt$ and ${\int }_0^{T}X\left(t\right)s(t,{\theta }_1)dt$. Again, the optimal value of q is $q^{*}=1/2$. Thus,

$$\begin{array}{ccc}\hfill P_e\left({\displaystyle \frac{1}{2}},{\theta }_0,{\theta }_1\right)& =& Q\left(\sqrt{{\displaystyle \frac{E}{2N_0}}\left[1-\varrho ({\theta }_0,{\theta }_1)\right]}\right)\hfill \\ & \approx & Q\left(\sqrt{{\displaystyle \frac{E\dot{E}{({\theta }_1-{\theta }_0)}^2}{4E{N}_0}}}\right)\hfill \\ & =& Q\left(\sqrt{{\displaystyle \frac{\dot{E}}{4N_0}}}·|{\theta }_1-{\theta }_0|\right)\hfill \\ & =& Q\left(\sqrt{{\displaystyle \frac{\dot{P}}{4N_0}}}·\sqrt{T}|{\theta }_1-{\theta }_0|\right),\hfill \end{array}$$

(45)

where $\dot{P}=\dot{E}/T$ is the power of $\dot{s}(t,\theta )$. Since we are dealing here with continuous time, instead of a sequence ${\xi }_n$, we use a function, $\xi \left(T\right)$, of the observation time, T, which in this case would be $\xi \left(T\right)={\displaystyle \frac{1}{\sqrt{T}}}$. Let ${\theta }_0=\theta$ and ${\theta }_1=\theta +{\displaystyle \frac{2s}{\sqrt{T}}}$. Then,

$$P_e^{\infty }(\theta ,s)=Q\left(\sqrt{{\displaystyle \frac{\dot{P}}{N_0}}}·\left|s\right|\right),$$

(46)

which, for the MSE case, yields

$$\begin{array}{ccc}\hfill r\left(\theta \right)& \ge & \underset{s\ge 0}{sup}\left\{2s^{2}Q\left(\sqrt{{\displaystyle \frac{\dot{P}}{N_0}}}·s\right)\right\}\hfill \\ & =& {\displaystyle \frac{N_0}{\dot{P}}}\underset{u\ge 0}{sup}\left\{2u^{2}Q\left(u\right)\right\}\hfill \\ & \approx & {\displaystyle \frac{0.3314N_0}{\dot{P}}}.\hfill \end{array}$$

(47)

w.r.t. $\zeta \left(T\right)=1/{(1/\sqrt{T})}^2=T$, which means that the minimax loss is lower bounded by $r\left(\theta \right)/T\ge 0.3314N_0/\dot{E}$. This has the same form as the Cramér–Rao lower bound (CRLB), except that the multiplicative factor is 0.3314 rather than 1. In Equation (20) in [18], the bound is of the same form, but with a multiplicative constant of 0.16 for a high signal-to-noise ratio (SNR). However, in [4], the constant of proportionality was improved to 1 in the high SNR limit, just like in the Cramér–Rao lower bound for the same model. The constant 0.3314 will be improved later to 0.4549 (under Example 9 in the sequel), but it will still be below 1.

The case where $s(t,\theta )$ is not everywhere differentiable w.r.t. θ can be handled in a similar manner, but some caution should be exercised. For example, consider the model,

$$X\left(t\right)=s(t-\theta )+N\left(t\right),$$

(48)

where $-\infty <t<\infty$, $\theta \in [0,T]$, $N\left(t\right)$ is AWGN as before, and $s(·)$ is a rectangular pulse with duration Δ and amplitude $\sqrt{E/\Delta }$, E being the signal energy. Here, $\varrho (\theta ,\theta +\delta )=1-\left|\delta \right|/\Delta$; namely, it also includes a linear term in $\left|\delta \right|$, not just the quadratic one. This changes the asymptotic behavior of the resulting lower bound to $r\left(\theta \right)$, which turns out to be $0.7544{(N_0\Delta /P)}^2$ w.r.t. $T^2$ (namely, a minimax lower bound of $0.7544{(N_0\Delta /E)}^2$). It is interesting to compare this bound to the Chapman–Robbins bound for the same model, which is a local bound of the same form but with a multiplicative constant of $0.1602$ instead of $0.7544$, and which is limited to unbiased estimators. The Chazan–Zakai–Ziv bound [4] for this case is difficult to calculate, but in the high SNR regime, it behaves like $3{(N_0\Delta /P)}^2$ w.r.t. $T^2$.

It is conceivable that the Chazan–Zakai–Ziv bound for estimating the delay of non-differential signals in Gaussian white noise has been improved even further ever since its publication in 1975, but the point of this particular example remains: our bound may not always be the best available bound. Still, since our bound is easy to calculate, it is worth comparing it to other bounds for every given model. This concludes Example 6.

4. Bounds Based on the Minimum Expected Loss over Some Test Points

In this section, we derive our second family of bounds for the scalar case, $d=1$.

4.1. Two Test Points

The following generic, yet conceptually very simple, lower bound assumes neither symmetry nor convexity of the loss function $\rho (·)$. For a given $(\mathit{x},q,{\theta }_0,{\theta }_1)\in {\mathbb{R}}^n\times [0,1]\times {\Theta }^2$, let us define

$$\psi (\mathit{x},q,{\theta }_0,{\theta }_1)=\underset{\vartheta }{min}\{qp\left(\mathit{x}\right|{\theta }_0)\rho (\vartheta -{\theta }_0)+(1-q)p\left(\mathit{x}\right|{\theta }_1)\rho (\vartheta -{\theta }_1)\}.$$

(49)

Then,

$$\begin{array}{ccc}\hfill R_n(\Theta )& \ge & \underset{{\theta }_0,{\theta }_1,q}{sup}q{\mathbb{E}}_{{\theta }_0}\left\{\rho (g_n\left[\mathit{X}\right]-{\theta }_0)\right\}+(1-q){\mathbb{E}}_{{\theta }_1}\left\{\rho (g_n\left[\mathit{X}\right]-{\theta }_1)\right\}\hfill \\ & =& \underset{{\theta }_0,{\theta }_1,q}{sup}{\int }_{{\mathbb{R}}^n}[qp\left(\mathit{x}\right|{\theta }_0)\rho (g_n\left[\mathit{x}\right]-{\theta }_0)+(1-q)p\left(\mathit{x}\right|{\theta }_1)\rho (g_n\left[\mathit{x}\right]-{\theta }_1)]\mathrm{d}\mathit{x}\hfill \\ & \ge & \underset{{\theta }_0,{\theta }_1,q}{sup}{\int }_{{\mathbb{R}}^n}\psi (\mathit{x},q,{\theta }_0,{\theta }_1)\mathrm{d}\mathit{x}.\hfill \end{array}$$

(50)

If we further assume the symmetry of $\rho$, then it is easy to see that the minimizer ${\vartheta }^{*}$, which achieves $\psi (\mathit{x},q,{\theta }_0,{\theta }_1)$, is always within the interval $[{\theta }_0,{\theta }_1]$. This is because the objective increases monotonically as we move away from the interval $[{\theta }_0,{\theta }_1]$ in either direction. Of course, this simple idea can easily be extended to apply to the weighted sums of more than two points, in principle, but it would become more complicated—see the next subsection for three such points.

If $\rho$ is concave, then the minimizing $\vartheta$ is either ${\theta }_0$ or ${\theta }_1$, depending on which of $qp\left(\mathit{x}\right|{\theta }_0)$ and $(1-q)p\left(\mathit{x}\right|{\theta }_1)$ is smaller, and the bound becomes

$$R_n(\Theta )\ge \underset{{\theta }_0,{\theta }_1,q}{sup}\rho ({\theta }_1-{\theta }_0)P_e(q,{\theta }_0,{\theta }_1).$$

(51)

Minimality at the edge points may also happen for some loss functions that are not concave, like the loss function $\rho \left(u\right)=1\left\{\right|u|\ge \Delta \}$.

The generic lower bound (50) is more general than our first bound in the sense that it does not require convexity or symmetry of $\rho$, but the downside is that the resulting expressions are harder to deal with directly, as will be seen shortly. For loss functions other than the MSE or general moments of the estimation error, it may not be a trivial task even to derive a closed-form expression of $\psi (\mathit{x},q,{\theta }_0,{\theta }_1)$ (i.e., to carry out the minimization associated with the definition of $\psi$).

For the case of the MSE, $\rho \left(\epsilon \right)={\epsilon }^2$, the calculation of $\psi (\mathit{x},q,{\theta }_0,{\theta }_1)$ is straightforward, and it readily yields

$$\psi (\mathit{x},q,{\theta }_0,{\theta }_1)={({\theta }_1-{\theta }_0)}^2·{\displaystyle \frac{qp\left(\mathit{x}\right|{\theta }_0)·(1-q)p\left(\mathit{x}\right|{\theta }_1)}{qp\left(\mathit{x}\right|{\theta }_0)+(1-q)p\left(\mathit{x}\right|{\theta }_1)}}.$$

(52)

However, it may not be convenient to integrate this function of x due to the summation at the denominator. One way to alleviate this difficulty is to observe that

$$\begin{array}{ccc}\hfill \psi (\mathit{x},q,{\theta }_0,{\theta }_1)& \ge & {({\theta }_1-{\theta }_0)}^2·{\displaystyle \frac{qp\left(\mathit{x}\right|{\theta }_0)·(1-q)p\left(\mathit{x}\right|{\theta }_1)}{2max\{qp\left(\mathit{x}\right|{\theta }_0),(1-q)p\left(\mathit{x}\right|{\theta }_1)\}}}\hfill \\ & =& {\displaystyle \frac{1}{2}}·{({\theta }_1-{\theta }_0)}^2·min\{qp\left(\mathit{x}\right|{\theta }_0),(1-q)p\left(\mathit{x}\right|{\theta }_1)\},\hfill \end{array}$$

(53)

which after integration yields again

$$R_n(\Theta )\ge \underset{{\theta }_0,{\theta }_1,q}{sup}\left\{{\displaystyle \frac{1}{2}}{({\theta }_1-{\theta }_0)}^2·P_e(q,{\theta }_0,{\theta }_1)\right\},$$

(54)

exactly as in Theorem 1 in the special case of the MSE. This indicates that the bound (50) is at least as tight as the bound of Theorem 1 for the MSE.

It turns out, however, that we can do better than bounding the denominator, $qp\left(\mathit{x}\right|{\theta }_0)+(1-q)p\left(\mathit{x}\right|{\theta }_1)$, by $2·max\{qp\left(\mathit{x}\right|{\theta }_0),(1-q)p\left(\mathit{x}\right|{\theta }_1)\}$ for the purpose of obtaining a more convenient integrand. To this end, we invoke the following lemma, whose proof appears in Appendix B.

Lemma 1.

Let k be a positive integer and let $a_1,\dots ,a_k$ be positive reals. Then,

$$\sum _{i=1}^k{a}_i=\underset{(r_1,\dots ,r_k)\in \mathcal{S}}{inf}max\left\{{\displaystyle \frac{a_1}{r_1}},\dots ,{\displaystyle \frac{a_k}{r_k}}\right\},$$

(55)

where $\mathcal{S}$ is the interior of the k-dimensional simplex, namely, the set of all vectors $(r_1,\dots ,r_k)$ with strictly positive components that sum to unity.

Applying Lemma 1 with the assignments $k=2$, $a_1=qp\left(\mathit{x}\right|{\theta }_0)$, and $a_2=(1-q)p\left(\mathit{x}\right|{\theta }_1)$, we have

$$qp(\mathit{x}|{\theta }_0+(1-q)p(\mathit{x}|{\theta }_1)=\underset{r\in (0,1)}{inf}max\left\{{\displaystyle \frac{qp(\mathit{x}|{\theta }_0)}{r}},{\displaystyle \frac{(1-q)p(\mathit{x}|{\theta }_1)}{1-r}}\right\}.$$

(56)

Thus,

$$\begin{array}{ccc}\hfill \psi (\mathit{x},q,{\theta }_0,{\theta }_1)& =& {\displaystyle \frac{qp\left(\mathit{x}\right|{\theta }_0)·(1-q)p\left(\mathit{x}\right|{\theta }_1)}{{inf}_{r\in (0,1)}max\left\{qp\left(\mathit{x}\right|{\theta }_0)/r,(1-q)p\left(\mathit{x}\right|{\theta }_1)/(1-r)\right\}}}\hfill \\ & =& \underset{r\in (0,1)}{sup}{\displaystyle \frac{qp\left(\mathit{x}\right|{\theta }_0)·(1-q)p\left(\mathit{x}\right|{\theta }_1)}{max\left\{qp\left(\mathit{x}\right|{\theta }_0)/r,(1-q)p\left(\mathit{x}\right|{\theta }_1)/(1-r)\right\}}}\hfill \\ & =& \underset{r\in (0,1)}{sup}\left\{\begin{array}{cc}r(1-q)p\left(\mathit{x}\right|{\theta }_1)\hfill & r\le r^{*}\hfill \\ (1-r)qp\left(\mathit{x}\right|{\theta }_0)\hfill & r\ge r^{*}\hfill \end{array}\right.\hfill \\ & =& \underset{r\in (0,1)}{sup}min\{r(1-q)p\left(\mathit{x}\right|{\theta }_1),(1-r)qp\left(\mathit{x}\right|{\theta }_0)\},\hfill \end{array}$$

(57)

where

$$r^{*}={\displaystyle \frac{qp\left(\mathit{x}\right|{\theta }_0)}{qp\left(\mathit{x}\right|{\theta }_0)+(1-q)p\left(\mathit{x}\right|{\theta }_1)}}.$$

(58)

Thus, the bound becomes

$$\begin{array}{ccc}\hfill R_n(\Theta )& \ge & \underset{\{({\theta }_0,{\theta }_1,q)\in {\Theta }^2\times (0,1)\}}{sup}{({\theta }_1-{\theta }_0)}^2\times \hfill \\ & & {\int }_{{\mathbb{R}}^n}\underset{r\in (0,1)}{sup}min\{r(1-q)p\left(\mathit{x}\right|{\theta }_1),(1-r)qp\left(\mathit{x}\right|{\theta }_0)\}\mathrm{d}\mathit{x}\hfill \\ & \ge & \underset{({\theta }_0,{\theta }_1,q)\in {\Theta }^2\times (0,1)}{sup}\underset{r\in (0,1)}{sup}{({\theta }_1-{\theta }_0)}^2\times \hfill \\ & & {\int }_{{\mathbb{R}}^n}min\{r(1-q)p\left(\mathit{x}\right|{\theta }_1),(1-r)qp\left(\mathit{x}\right|{\theta }_0)\}\mathrm{d}\mathit{x}\hfill \\ & =& \underset{({\theta }_0,{\theta }_1,q,r)\in {\Theta }^2\times {(0,1)}^2}{sup}{({\theta }_1-{\theta }_0)}^2·(q+r-2qr)\times \hfill \\ & & {\int }_{{\mathbb{R}}^n}min\left\{{\displaystyle \frac{r(1-q)}{q+r-2qr}}·p\left(\mathit{x}\right|{\theta }_1),{\displaystyle \frac{(1-r)q}{q+r-2qr}}·p\left(\mathit{x}\right|{\theta }_0)\right\}\mathrm{d}\mathit{x}\hfill \\ & =& \underset{({\theta }_0,{\theta }_1,q,r)\in {\Theta }^2\times {(0,1)}^2}{sup}{({\theta }_1-{\theta }_0)}^2·(q+r-2qr)·P_e\left({\displaystyle \frac{(1-r)q}{q+r-2qr}},{\theta }_0,{\theta }_1\right).\hfill \end{array}$$

The bound of Theorem 1 for the MSE is obtained as a special case of $r=1/2$. Therefore, after the optimization over the additional degree of freedom, r, the resulting bound cannot be worse than the MSE bound of Theorem 1. In fact, it may strictly improve as we will demonstrate shortly. The choice $r=q$ gives a prior of $1/2$ in the error probability factor, and then the maximum of the external factor, $q+r-2rq=2q(1-q)$, is maximized by $q=1/2$.

Example 7.

To demonstrate the new bound for the MSE, let us revisit Example 2 and see how it improves the multiplicative constant. In that example,

$$P_e\left({\displaystyle \frac{(1-r)q}{q+r-2qr}},{\theta }_0,{\theta }_1\right)=min\left\{{\displaystyle \frac{(1-r)q}{q+r-2qr}},{\displaystyle \frac{r(1-q)}{q+r-2qr}}·{\left({\displaystyle \frac{{\theta }_0}{{\theta }_1}}\right)}^n\right\}.$$

(59)

Let us denote $\alpha ={({\theta }_0/{\theta }_1)}^n$ and recall that $\alpha \in (0,1)$, provided that we select ${\theta }_1>{\theta }_0$. Then,

$$(q+r-2qr)·P_e\left({\displaystyle \frac{(1-r)q}{q+r-2qr}},{\theta }_0,{\theta }_1\right)=min\{(1-r)q,r(1-q)\alpha \}.$$

(60)

The maximum w.r.t. q is attained when $(1-r)q=r(1-q)\alpha$, namely, for $q=q^{*}\stackrel{▵}{=}\alpha r/[1-(1-\alpha )r]$, which yields

$$\underset{q}{max}(q+r-2qr)·P_e\left({\displaystyle \frac{(1-r)q}{q+r-2qr}},{\theta }_0,{\theta }_1\right)={\displaystyle \frac{\alpha r(1-r)}{1-(1-\alpha )r}}.$$

(61)

Let us denote $\beta \stackrel{▵}{=}1-\alpha \in (0,1)$. The maximum of $r(1-r)/(1-\beta r)$ is attained for

$$r=r^{*}\stackrel{▵}{=}{\displaystyle \frac{1-\sqrt{1-\beta }}{\beta }}={\displaystyle \frac{1-\sqrt{\alpha }}{1-\alpha }}={\displaystyle \frac{1}{1+\sqrt{\alpha }}},$$

(62)

which yields

$$\begin{array}{ccc}\hfill \underset{q,r}{max}(q+r-2qr)·P_e\left({\displaystyle \frac{(1-r)q}{q+r-2qr}},{\theta }_0,{\theta }_1\right)& =& \underset{r\in (0,1)}{sup}{\displaystyle \frac{\alpha r(1-r)}{1-(1-\alpha )r}}\hfill \\ & =& \alpha ·{\left({\displaystyle \frac{1-\sqrt{\alpha }}{1-\alpha }}\right)}^2\hfill \\ & =& {\displaystyle \frac{\alpha }{{(1+\sqrt{\alpha })}^2}}.\hfill \end{array}$$

(63)

To obtain a local bound in the spirit of Corollary 1, take ${\theta }_0=\theta$, ${\theta }_1=\theta (1+{\displaystyle \frac{s}{n\theta }})$, which yields $\alpha =e^{-s/\theta }$ in the limit of large n, and so,

$$r\left(\theta \right)\ge \underset{s\ge 0}{sup}{\displaystyle \frac{s^2{e}^{-s/\theta }}{{(1+e^{-s/\left[2\theta \right]})}^2}}={\theta }^2·\underset{u\ge 0}{sup}{u}^2·{\displaystyle \frac{e^{-u}}{{(1+e^{-u/2})}^2}}\approx 0.3102{\theta }^2,$$

(64)

w.r.t. ${\zeta }_n=n^2$, which improves on our earlier bound in Example 2, $r\left(\theta \right)\ge 0.2414{\theta }^2$. This concludes Example 7.

More generally, for general moments of the estimation error, a similar derivation yields the following:

Theorem 2.

For $\rho \left(\epsilon \right)={\left|\epsilon \right|}^t$, $t\ge 1$ (not necessarily an integer),

$$R_n(\Theta )\ge \underset{{\theta }_0,{\theta }_1,q,r}{sup}{|{\theta }_1-{\theta }_0|}^t[{(1-r)}^{t-1}q+r^{t-1}(1-q)]·P_e\left({\displaystyle \frac{{(1-r)}^{t-1}q}{{(1-r)}^{t-1}q+r^{t-1}(1-q)}},{\theta }_0,{\theta }_1\right).$$

(65)

Applying the local version of Theorem 2 to Example 7, we obtain

$$r\left(\theta \right)\ge {\theta }^t·\underset{s>0}{sup}{\displaystyle \frac{s^t{e}^{-s}}{{(1+e^{-s/t})}^t}}$$

(66)

w.r.t. ${\zeta }_n=n^t$. Changing the optimization variable from s to $\sigma =s/t$, we end up with

$$\begin{array}{ccc}\hfill r\left(\theta \right)\ge {\left(\theta t\right)}^t·{\left[\underset{\sigma >0}{sup}{\displaystyle \frac{\sigma }{e^{\sigma }+1}}\right]}^t& \approx & {\left(0.2785t\theta \right)}^t.\hfill \end{array}$$

(67)

The factor of ${\left(0.2785t\right)}^t$ should be compared with ${(1/2e)}^t=0.{1839}^t$ of Example 2, pertaining to the choice $q=r=1/2$. The gap increases exponentially with t. For the maximum likelihood estimator pertaining to Example 2, which is $g_n\left[\mathit{X}\right]={max}_i{X}_i$, it is easy to show that whenever t is an integer,

$${\mathbb{E}}_{\theta }\left\{\right|\widehat{\theta }-\theta {|}^t\}={\displaystyle \frac{n!t!{\theta }^t}{(n+t)!}}={\displaystyle \frac{t!{\theta }^t}{(n+1)(n+2)\cdots (n+t)}},$$

(68)

and so, the asymptotic gap is between $t!$ and ${\left(0.2785t\right)}^t$. Considering the Stirling approximation, the ratio between the upper bound and the lower bound is about $\sqrt{2\pi t}·{\left(1.3211\right)}^t$.

4.2. Three Test Points

The idea behind the bounds of Section 4.1 can be conceptually extended to be based on more than two test points, but the resulting expressions become cumbersome very quickly as the number of test points grows. For three points, however, this is still manageable and can provide improved bounds. Let us select the three test points to be ${\theta }_0-\Delta$, ${\theta }_0$, and ${\theta }_0+\Delta$ for some ${\theta }_0$ and $\Delta$, and let us assign weights q, r, and $w=1-q-r$. Consider the bound

$$\begin{array}{ccc}\hfill R_n(\Theta )& \ge & q{\mathbb{E}}_{{\theta }_0-\Delta }\left\{\rho (g_n\left[\mathit{X}\right]-{\theta }_0+\Delta )\right\}+r{\mathbb{E}}_{{\theta }_0}\left\{\rho (g_n\left[\mathit{X}\right]-{\theta }_0)\right\}+\hfill \\ & & w{\mathbb{E}}_{{\theta }_0+\Delta }\left\{\rho (g_n\left[\mathit{X}\right]-{\theta }_0-\Delta )\right\}\hfill \\ & =& {\int }_{{\mathbb{R}}^n}[q·p\left(\mathit{x}\right|{\theta }_0-\Delta )\rho (g_n\left[\mathit{x}\right]-{\theta }_0+\Delta )+r·p\left(\mathit{x}\right|{\theta }_0)\rho (g_n\left[\mathit{x}\right]-{\theta }_0)+\hfill \\ & & w·p\left(\mathit{x}\right|{\theta }_0+\Delta )\rho (g_n\left[\mathit{x}\right]-{\theta }_0-\Delta )]\mathrm{d}\mathit{x}\hfill \\ & \ge & {\int }_{{\mathbb{R}}^n}\Psi (\mathit{x},{\theta }_0,\Delta ,q,r)\mathrm{d}\mathit{x},\hfill \end{array}$$

(69)

where

$$\begin{array}{ccc}\hfill \Psi (\mathit{x},{\theta }_0,\Delta ,q,r)& =& \underset{\vartheta }{min}\{q·p\left(\mathit{x}\right|{\theta }_0-\Delta )\rho (\vartheta -{\theta }_0+\Delta )+r·p\left(\mathit{x}\right|{\theta }_0)\rho (\vartheta -{\theta }_0)+\hfill \\ & & w·p\left(\mathit{x}\right|{\theta }_0+\Delta )\rho (\vartheta -{\theta }_0-\Delta )\}.\hfill \end{array}$$

(70)

Considering the case of the MSE,

$$\begin{array}{ccc}\hfill \Psi (\mathit{x},{\theta }_0,\Delta ,q,r)& =& \underset{\vartheta }{min}\{q·p\left(\mathit{x}\right|{\theta }_0-\Delta ){(\vartheta -{\theta }_0+\Delta )}^2+r·p\left(\mathit{x}\right|{\theta }_0){(\vartheta -{\theta }_0)}^2+\hfill \\ & & w·p\left(\mathit{x}\right|{\theta }_0+\Delta ){(\vartheta -{\theta }_0-\Delta )}^2\}\hfill \end{array}$$

(71)

can be found in closed-form. Denoting temporarily $a=qp\left(\mathit{x}\right|{\theta }_0-\Delta )$, $b=rp\left(\mathit{x}\right|{\theta }_0)$, and $c=wp\left(\mathit{x}\right|{\theta }_0+\Delta )$, $\Psi (\mathit{x},{\theta }_0,\Delta ,q,r)$ is attained by

$$\vartheta ={\vartheta }^{*}={\displaystyle \frac{a({\theta }_0-\Delta )+b{\theta }_0+c({\theta }_0+\Delta )}{a+b+c}}={\theta }_0+{\displaystyle \frac{(c-a)\Delta }{a+b+c}}.$$

(72)

On substituting ${\vartheta }^{*}$ into the sum of squares, we end up with

$$\begin{array}{ccc}\hfill \Psi (\mathit{x},{\theta }_0,\Delta ,q,r)& =& {\displaystyle \frac{[a{(b+2c)}^2+b{(c-a)}^2+c{(2a+b)}^2]{\Delta }^2}{{(a+b+c)}^2}}\hfill \\ & =& {\displaystyle \frac{(ab+bc+4ac){\Delta }^2}{a+b+c}}\hfill \\ & =& {\displaystyle \frac{[qrp\left(\mathit{x}\right|{\theta }_0-\Delta )p\left(\mathit{x}\right|{\theta }_0)+rwp\left(\mathit{x}\right|{\theta }_0)p\left(\mathit{x}\right|{\theta }_0+\Delta )+4qwp\left(\mathit{x}\right|{\theta }_0-\Delta )p\left(\mathit{x}\right|{\theta }_0+\Delta )]{\Delta }^2}{qp\left(\mathit{x}\right|{\theta }_0-\Delta )+rp\left(\mathit{x}\right|{\theta }_0)+wp\left(\mathit{x}\right|{\theta }_0+\Delta )}}.\hfill \end{array}$$

The lower bound on the MSE is then the sum of three integrals

$$\begin{array}{ccc}\hfill I_1& =& qr{\Delta }^2·{\int }_{{\mathbb{R}}^n}{\displaystyle \frac{p\left(\mathit{x}\right|{\theta }_0-\Delta )p\left(\mathit{x}\right|{\theta }_0)\mathrm{d}\mathit{x}}{qp\left(\mathit{x}\right|{\theta }_0-\Delta )+rp\left(\mathit{x}\right|{\theta }_0)+wp\left(\mathit{x}\right|{\theta }_0+\Delta )}}\hfill \end{array}$$

(73)

$$\begin{array}{ccc}\hfill I_2& =& rw{\Delta }^2·{\int }_{{\mathbb{R}}^n}{\displaystyle \frac{p\left(\mathit{x}\right|{\theta }_0)p\left(\mathit{x}\right|{\theta }_0+\Delta )\mathrm{d}\mathit{x}}{qp\left(\mathit{x}\right|{\theta }_0-\Delta )+rp\left(\mathit{x}\right|{\theta }_0)+wp\left(\mathit{x}\right|{\theta }_0+\Delta )}}\hfill \end{array}$$

(74)

$$\begin{array}{ccc}\hfill I_3& =& 4qw{\Delta }^2·{\int }_{{\mathbb{R}}^n}{\displaystyle \frac{p\left(\mathit{x}\right|{\theta }_0-\Delta )p\left(\mathit{x}\right|{\theta }_0+\Delta )\mathrm{d}\mathit{x}}{qp\left(\mathit{x}\right|{\theta }_0-\Delta )+rp\left(\mathit{x}\right|{\theta }_0)+wp\left(\mathit{x}\right|{\theta }_0+\Delta )}}.\hfill \end{array}$$

(75)

Example 8.

Revisiting again Example 2, for a given $t>0$, let us denote $\mathcal{C}\left(t\right)={[0,t]}^n$, and then $p\left(\mathit{x}\right|{\theta }_0+i\Delta )={[{\theta }_0+i\Delta ]}^{-n}·\mathcal{I}\{\mathit{x}\in \mathcal{C}({\theta }_0+i\Delta )$, $i=-1,0,1$. Then,

$$\begin{array}{ccc}\hfill I_1& =& qr{\Delta }^2·{\int }_{\mathcal{C}({\theta }_0-\Delta )}{\displaystyle \frac{{({\theta }_0-\Delta )}^{-n}{\theta }_0^{-n}d\mathit{x}}{q{({\theta }_0-\Delta )}^{-n}+r{\theta }_0^{-n}+w{({\theta }_0+\Delta )}^{-n}}}\hfill \\ & =& qr{\Delta }^2·{\displaystyle \frac{{({\theta }_0-\Delta )}^n{({\theta }_0-\Delta )}^{-n}{\theta }_0^{-n}}{q{({\theta }_0-\Delta )}^{-n}+r{\theta }_0^{-n}+w{({\theta }_0+\Delta )}^{-n}}}\hfill \\ & =& {\displaystyle \frac{qr{\Delta }^2{\theta }_0^{-n}}{q{({\theta }_0-\Delta )}^{-n}+r{\theta }_0^{-n}+w{({\theta }_0+\Delta )}^{-n}}}\hfill \\ & =& {\displaystyle \frac{qr{\Delta }^2}{q{[{\theta }_0/({\theta }_0-\Delta )]}^n+r+w{[{\theta }_0/({\theta }_0+\Delta )]}^n}}.\hfill \end{array}$$

(76)

For $\Delta ={\theta }_{0}s/n$, we then have, for large n:

$$I_1\sim {\displaystyle \frac{{\theta }_0^2}{n^2}}·{\displaystyle \frac{qr{s}^2}{q{e}^s+r+w{e}^{-s}}}={\displaystyle \frac{{\theta }_0^2}{n^2}}·{\displaystyle \frac{qr{s}^2{e}^s}{q{e}^{2s}+r{e}^s+w}}.$$

(77)

$$\begin{array}{ccc}\hfill I_2& =& rw{\Delta }^2{\int }_{\mathcal{C}\left({\theta }_0\right)}{\displaystyle \frac{{\theta }_0^{-n}{({\theta }_0+\Delta )}^{-n}d\mathit{x}}{q{({\theta }_0-\Delta )}^{-n}\mathcal{I}\{\mathit{x}\in \mathcal{C}({\theta }_0-\Delta )\}+r{\theta }_0^{-n}+w{({\theta }_0+\Delta )}^{-n}}}\hfill \\ & =& rw{\Delta }^2·{\displaystyle \frac{{({\theta }_0-\Delta )}^n{\theta }_0^{-n}{({\theta }_0+\Delta )}^{-n}}{q{({\theta }_0-\Delta )}^{-n}+r{\theta }_0^{-n}+w{({\theta }_0+\Delta )}^{-n}}}+\hfill \\ & & rw{\Delta }^2·{\displaystyle \frac{[{\theta }_0^n-{({\theta }_0-\Delta )}^n]{\theta }_0^{-n}{({\theta }_0+\Delta )}^{-n}}{r{\theta }_0^{-n}+w{({\theta }_0+\Delta )}^{-n}}}\hfill \\ & =& rw{\Delta }^2·{\displaystyle \frac{{(1-\Delta /{\theta }_0)}^n}{q{[(1+\Delta /{\theta }_0)/(1-\Delta /{\theta }_0)]}^{-n}+r{(1+\Delta /{\theta }_0)}^n+w}}+\hfill \\ & & rw{\Delta }^2·{\displaystyle \frac{[1-{(1-\Delta /{\theta }_0)}^n]}{r{(1+\Delta /{\theta }_0)}^n+w}}\hfill \\ & \sim & {\displaystyle \frac{{\theta }_0^2}{n^2}}·rw{s}^2\left({\displaystyle \frac{e^{-s}}{q{e}^{2s}+r{e}^s+w}}+{\displaystyle \frac{1-e^{-s}}{r{e}^s+w}}\right).\hfill \end{array}$$

(78)

Similarly,

$$I_3\sim {\displaystyle \frac{{\theta }_0^2}{n^2}}·{\displaystyle \frac{4qw{s}^2}{q{e}^{2s}+r{e}^s+w}}.$$

(79)

Thus,

$$\begin{array}{ccc}\hfill r\left({\theta }_0\right)& \ge & {\theta }_0^2·\underset{s\ge 0}{sup}\underset{(q,r,w)\in {\mathbb{R}}_{+}^3:q+r+w=1}{sup}{s}^2·\left[{\displaystyle \frac{qr{e}^s+4qw+rw{e}^{-s}}{q{e}^{2s}+r{e}^s+w}}+{\displaystyle \frac{rw(1-e^{-s})}{r{e}^s+w}}\right]\hfill \\ & \approx & 0.4624{\theta }_0^2,\hfill \end{array}$$

(80)

w.r.t. ${\zeta }_n=n^2$, which is a significant improvement of nearly 50% over the previous bound, $0.3102{\theta }_0^2$, which was obtained on the basis of two test points, let alone the bound of Theorem 1, which was $0.2414{\theta }_0^2$. This concludes Example 8.

In general, the integrals $I_1$, $I_2$, and $I_3$ are not easy to calculate due to the summations at the denominators of the integrands. One way to proceed is to apply Lemma 1 to the sum of $k=3$ terms, but this would introduce two additional parameters to be optimized. Returning to the earlier shorthand notation of a, b, and c, a different approach to get rid of summations at the denominators, at the expense of some loss of tightness, is the following:

$$\begin{array}{ccc}\hfill {\displaystyle \frac{ab+bc+4ac}{a+b+c}}& =& {\displaystyle \frac{a(b+2c)}{a+b+c}}+{\displaystyle \frac{c(b+2a)}{a+b+c}}\hfill \\ & \ge & {\displaystyle \frac{a(b+2c)}{a+b+2c}}+{\displaystyle \frac{c(b+2a)}{2a+b+c}}\hfill \\ & \ge & {\displaystyle \frac{a(b+2c)}{2max\{a,b+2c\}}}+{\displaystyle \frac{c(b+2a)}{2max\{2a+b,c\}}}\hfill \\ & =& {\displaystyle \frac{1}{2}}·\left(min\{a,b+2c\}+min\{2a+b,c\}\right)\hfill \\ & \ge & {\displaystyle \frac{1}{2}}·\left(min\{a,b\}+min\{b,c\}\right)\hfill \\ & =& {\displaystyle \frac{1}{2}}·\left[min\{qp\left(\mathit{x}\right|{\theta }_0-\Delta ),rp\left(\mathit{x}\right|{\theta }_0)\}+min\{rp\left(\mathit{x}\right|{\theta }_0),wp\left(\mathit{x}\right|{\theta }_0+\Delta )\}\right],\hfill \end{array}$$

(81)

and so,

$$\begin{array}{ccc}\hfill R_n(\Theta )& \ge & {\displaystyle \frac{{\Delta }^2}{2}}[{\int }_{{\mathbb{R}}^n}min\{qp\left(\mathit{x}\right|{\theta }_0-\Delta ),rp\left(\mathit{x}\right|{\theta }_0)\}\mathrm{d}\mathit{x}+\hfill \\ & & {\int }_{{\mathbb{R}}^n}min\{rp\left(\mathit{x}\right|{\theta }_0),wp\left(\mathit{x}\right|{\theta }_0+\Delta )\}\mathrm{d}\mathit{x}]\hfill \\ & =& {\displaystyle \frac{{\Delta }^2}{2}}[(q+r)P_e\left({\displaystyle \frac{q}{q+r}},{\theta }_0-\Delta ,{\theta }_0\right)+\hfill \\ & & (r+w)P_e\left({\displaystyle \frac{r}{r+w}},{\theta }_0,{\theta }_0+\Delta \right)].\hfill \end{array}$$

(82)

Note that by setting $w=0$, we recover the bound obtained by integrating (53), and therefore, by optimizing w, the resulting bound cannot be worse. A slightly different (better, but more complicated) route in (81) is to apply Lemma 1 with $k=2$ in the following manner:

$$\begin{array}{ccc}\hfill {\displaystyle \frac{ab+bc+4ac}{a+b+c}}& =& {\displaystyle \frac{a(b+2c)}{a+b+c}}+{\displaystyle \frac{c(b+2a)}{a+b+c}}\hfill \\ & \ge & {\displaystyle \frac{a(b+2c)}{a+b+2c}}+{\displaystyle \frac{c(b+2a)}{2a+b+c}}\hfill \\ & =& {\displaystyle \frac{a(b+2c)}{{min}_{u}max\{a/u,(b+2c)/(1-u)\}}}+{\displaystyle \frac{c(b+2a)}{{min}_{v}max\{(2a+b)/(1-v),c/v\}}}\hfill \\ & =& \underset{u}{max}min\{(1-u)a,u(b+2c)\}+\underset{v}{max}min\{v(2a+b),(1-v)c\}\hfill \\ & \ge & \underset{u}{max}min\{(1-u)a,ub\}+\underset{v}{max}min\{vb,(1-v)c\}\hfill \\ & =& \underset{u}{max}min\{(1-u)qp\left(\mathit{x}\right|{\theta }_0-\Delta ),urp\left(\mathit{x}\right|{\theta }_0)\}+\hfill \\ & & \underset{v}{max}min\{vrp\left(\mathit{x}\right|{\theta }_0),(1-v)wp\left(\mathit{x}\right|{\theta }_0+\Delta )\},\hfill \end{array}$$

(83)

and so,

$$\begin{array}{ccc}\hfill R_n(\Theta )& \ge & {\Delta }^2·[\underset{u}{max}{\int }_{{\mathbb{R}}^n}min\{(1-u)qp\left(\mathit{x}\right|{\theta }_0-\Delta ),urp\left(\mathit{x}\right|{\theta }_0)\}\mathrm{d}\mathit{x}+\hfill \\ & & \underset{v}{max}{\int }_{{\mathbb{R}}^n}min\{vrp\left(\mathit{x}\right|{\theta }_0),(1-v)wp\left(\mathit{x}\right|{\theta }_0+\Delta )\}\mathrm{d}\mathit{x}]\hfill \\ & =& {\Delta }^2·\{\underset{u}{max}[(1-u)q+ur]·P_e\left({\displaystyle \frac{(1-u)q}{(1-u)q+ur}},{\theta }_0-\Delta ,{\theta }_0\right)+\hfill \\ & & \underset{v}{max}[vr+(1-v)w]·P_e\left({\displaystyle \frac{vr}{vr+(1-v)w}},{\theta }_0,{\theta }_0+\Delta \right)\}.\hfill \end{array}$$

(84)

We have just proved the following result:

Theorem 3.

For the MSE case,

$$\begin{array}{ccc}\hfill R_n(\Theta )& \ge & \underset{{\theta }_0,\Delta ,q,r,w,(u,v)\in {[0,1]}^2}{sup}{\Delta }^2·\{\underset{u}{max}[(1-u)q+ur]·P_e\left({\displaystyle \frac{(1-u)q}{(1-u)q+ur}},{\theta }_0-\Delta ,{\theta }_0\right)+\hfill \\ & & \underset{v}{max}[vr+(1-v)w]·P_e\left({\displaystyle \frac{vr}{vr+(1-v)w}},{\theta }_0,{\theta }_0+\Delta \right)\}.\hfill \end{array}$$

(85)

where the maximum over q, r, and w is under the constraints that they are all non-negative and sum to unity.

Example 9.

Revisiting Example 4, it is natural that both error probabilities would correspond to a prior of $1/2$. This dictates the relations,

$$\begin{array}{c}\hfill u={\displaystyle \frac{q}{q+r}}\end{array}$$

(86)

$$\begin{array}{c}\hfill v={\displaystyle \frac{w}{w+r}},\end{array}$$

(87)

and so, the bound becomes

$$\begin{array}{ccc}\hfill R_n(\Theta )& \ge & \underset{\Delta ,q,w,r}{max}{\Delta }^2\left({\displaystyle \frac{2qr}{q+r}}+{\displaystyle \frac{2wr}{w+r}}\right)·Q\left({\displaystyle \frac{\sqrt{n}\Delta }{2\sigma }}\right)\hfill \\ & =& \left[\underset{s\ge 0}{max}{\left({\displaystyle \frac{2s\sigma }{\sqrt{n}}}\right)}^{2}Q\left(s\right)\right]·\underset{\left\{\right(q,r):q\ge 0,r\ge 0,q+r\le 1\}}{max}\left\{{\displaystyle \frac{2qr}{q+r}}+{\displaystyle \frac{2r(1-q-r)}{1-q}}\right\}\hfill \\ & =& {\displaystyle \frac{{\sigma }^2}{n}}·\underset{s\ge 0}{max}\left\{4s^{2}Q\left(s\right)\right\}·0.6862\hfill \\ & =& {\displaystyle \frac{{\sigma }^2}{n}}·0.6629·0.6862\hfill \\ & \approx & {\displaystyle \frac{0.4549{\sigma }^2}{n}},\hfill \end{array}$$

(88)

which is an improvement on the bound of Example 4, which was $0.3314{\sigma }^2/n$. Similarly, in Example 6, for smooth signals, the multiplicative constant also improves from 0.3314 to 0.4549, and for the rectangular pulse, it improves from 0.7544 to 1.0352 (about 37% improvement in all of them). This concludes Example 9.

Example 10.

Revisiting Example 2, we now have

$$\begin{array}{ccc}\hfill R_n(\Theta )& \ge & {\Delta }^2·[\underset{u}{max}min\{(1-u)q,ur\alpha \}+\underset{v}{max}min\{vr,(1-v)w\alpha \}]\hfill \\ & =& {\Delta }^2\left({\displaystyle \frac{qr\alpha }{q+r\alpha }}+{\displaystyle \frac{wr\alpha }{r+w\alpha }}\right)\hfill \\ & =& {\Delta }^2\left({\displaystyle \frac{qr\alpha }{q+r\alpha }}+{\displaystyle \frac{r(1-q-r)\alpha }{r+(1-q-r)\alpha }}\right)\hfill \\ & =& {\displaystyle \frac{s^2}{n^2}}{e}^{-s}r\left({\displaystyle \frac{q}{q+r{e}^{-s}}}+{\displaystyle \frac{1-q-r}{r+(1-q-r)e^{-s}}}\right)\hfill \\ & =& {\displaystyle \frac{s^2}{n^2}}r\left({\displaystyle \frac{q}{q{e}^s+r}}+{\displaystyle \frac{1-q-r}{r{e}^s+(1-q-r)}}\right),\hfill \end{array}$$

(89)

whose maximum is $0.3909/n^2$, an improvement relative to the earlier bound of $0.3102/n^2$. This concludes Example 10.

5. Extensions to the Vector Case

In this section, we outline extensions of some of our findings in Section 3 and Section 4 to the vector case. Let $\theta$ be a parameter vector of dimension d and $\Theta \subseteq {\mathbb{R}}^d$. Let $\rho \left(\epsilon \right)$ be a convex loss function that depends on the d-dimensional error vector $\epsilon$ only via its norm, $\parallel \epsilon \parallel$ (that is, $\rho$ has radial symmetry).

First, observe that Theorem 1 extends verbatim to the vector case, as nothing in the proof of Theorem 1 is based on any assumption or property that holds only if $\theta$ is a scalar parameter.

Corollary 1 can also be extended by letting the two test points of the auxiliary hypothesis testing problem, ${\theta }_0$ and ${\theta }_1$, be selected such that the distance between them, $\parallel {\theta }_1-{\theta }_0\parallel$, would decay at an appropriate rate as a function of n, so as to make the probability of error, ${max}_q{P}_e(q,{\theta }_0,{\theta }_1)$, converge to a positive constant as $n\to \infty$, as was achieved in Corollary 1 for the scalar case. But in the vector case considered now, there is an additional degree of freedom, which is the direction of the displacement vector, ${\theta }_1-{\theta }_0$. This direction can now be optimized so as to yield the largest (hence the tightest) lower bound. To be specific, in the vector case, Corollary 1 remains the same except that now s should be thought of as a d-dimensional vector rather than a scalar, $\rho \left(u\right)$ in the denominator of (19) should be replaced by $\rho (v·u)$, where $v\in {\mathbb{R}}^d$ is an arbitrary fixed non-zero vector (for example, any unit norm vector in the case of MSE), and the supremum in (20) should be taken over ${\mathbb{R}}^d$. To demonstrate this point, we now revisit Example 5, but this time, for the vector case.

Example 11.

Consider the case where $p\left(\mathit{x}\right|\theta )={\prod }_{i=1}^{n}p\left(x_i\right|\theta )$, with each factor in this product PDF being given by a d-dimensional exponential family,

$$p\left(x\right|\theta )={\displaystyle \frac{exp\left\{{\theta }^{\prime }T\left(x\right)\right\}}{Z\left(\theta \right)}}$$

(90)

where ${\theta }^{\prime }T\left(x\right)$ is the inner product of the d-dimensional parameter vector θ and a d-dimensional vector of statistics, $T\left(x\right)=(T_1\left(x\right),T_2\left(x\right),\dots ,T_d\left(x\right))$, and

$$Z\left(\theta \right)={\int }_{\mathbb{R}}exp\left\{{\theta }^{\prime }T\left(x\right)\right\}dx,$$

(91)

provided that the integral converges. In the above notation, both θ and $T\left(x\right)$ are understood to be column vectors and ${\theta }^{\prime }$ denotes transposition of θ to a row vector. Similarly as in Corollary 1, to obtain a local bound at a given θ, we let ${\theta }_0=\theta$ and ${\theta }_1=\theta +{\displaystyle \frac{2s}{\sqrt{n}}}$, where $s\in {\mathbb{R}}^d$. A simple extension of the derivation in Example 5 (using the CLT) yields

$$P_e^{\infty }(\theta ,s)=Q\left(\sqrt{s^{\prime }I\left(\theta \right)s}\right),$$

(92)

where s is considered a column vector, the superscript prime denotes vector transposition as before, and $I\left(\theta \right)={\nabla }^{2}lnZ\left(\theta \right)$ is the $d\times d$ Fisher information matrix of the exponential family. Considering the case of the MSE, $\rho \left(ϵ\right)={\parallel ϵ\parallel }^2$ with v being any unit norm vector, we have

$$w\left(s\right)=\underset{u\to 0}{lim}{\displaystyle \frac{{\parallel s·u\parallel }^2}{{\parallel v·u\parallel }^2}}={\parallel s\parallel }^2,$$

(93)

and then the following lower bound is obtained w.r.t. $1/\parallel v/\sqrt{n}{\parallel }^2=n$:

$$\begin{array}{ccc}\hfill r\left(\theta \right)& \ge & \underset{s\in {\mathbb{R}}^d}{sup}\left\{{2\parallel s\parallel }^{2}Q\left(\sqrt{s^{\prime }I\left(\theta \right)s}\right)\right\}\hfill \\ & =& \underset{t\ge 0}{sup}\left\{2t^2·\underset{\{s:\parallel s{\parallel }^2=t^2\}}{max}Q\left(\sqrt{s^{\prime }I\left(\theta \right)s}\right)\right\}\hfill \\ & =& \underset{t\ge 0}{sup}\left\{2t^2·Q\left(t\sqrt{{\lambda }_{min}\left(\theta \right)}\right)\right\}\hfill \\ & =& {\displaystyle \frac{1}{{\lambda }_{min}\left(\theta \right)}}\underset{\tau \ge 0}{sup}\left\{2{\tau }^2·Q\left(\tau \right)\right\}\hfill \\ & \approx & {\displaystyle \frac{0.3314}{{\lambda }_{min}\left(\theta \right)}},\hfill \end{array}$$

(94)

where ${\lambda }_{min}\left(\theta \right)$ is the smallest eigenvalue of $I\left(\theta \right)$. In this example, it is apparent that the chosen direction of the vector s is that of the eigenvector corresponding to the smallest eigenvalue of the Fisher information matrix. This completes Example 11.

In the vector case considered in this section, it is also instructive to extend the scope from two test points, ${\theta }_0$ and ${\theta }_1$, to multiple test points, ${\theta }_0,{\theta }_1,\dots ,{\theta }_{m-1}$, along with the corresponding weights (or priors), $q_0,q_1,\dots ,q_{m-1}$ (see the exposition at the end of Section 2). Interestingly, this will also lead to new bounds even for the case of $m=2$ test points.

To this end, we also consider a set of m unitary transformation matrices, $T_0,T_1,T_2,\dots ,$$T_{m-1}$, with the properties: (i) $\theta \in \Theta$ if and only if $T_i\theta \in \Theta$ for all $i=0,1,\dots ,m-1$, and (ii) $T_0+T_1+\cdots +T_{m-1}=0$. For example, if $d=2$, take $T_i$ to be matrices of rotation by $2\pi i/m$, $i=0,1,\dots ,m-1$.

Theorem 4.

Let ${\theta }_0,{\theta }_1,\dots ,{\theta }_{m-1}\in \Theta \subseteq {\mathbb{R}}^d$, $d\ge 1$ and $q_0,q_1,\dots ,q_{m-1}$ be given, and let $T_0,T_1,\dots T_{m-1}$ be unitary transformations that sum to zero, as described in the above paragraph. Then, for a convex loss function $\rho \left(\epsilon \right)$ that depends on ε only via $\parallel \epsilon \parallel$, we have:

$$R_n(\Theta )\ge m·\rho \left({\displaystyle \frac{1}{m}}\sum _{i=0}^{m-1}{T}_i{\theta }_i\right)·{\int }_{{\mathbb{R}}^n}min\{q_0·p\left(\mathit{x}\right|{\theta }_0),\dots ,q_{m-1}p\left(\mathit{x}\right|{\theta }_{m-1})\}d\mathit{x}.$$

(95)

As explained in Section 2, the integral on the right-hand side can be interpreted as the probability of list error with a list size of $m-1$.

Proof of Theorem 4.

The proof is a direct extension of the proof of Theorem 1:

$$\begin{array}{ccc}\hfill \underset{\theta \in \Theta }{sup}{\mathbb{E}}_{\theta }\left\{\rho (g_n\left[\mathit{X}\right]-\theta )\right\}& \ge & \sum _{i=0}^{m-1}{q}_i{\mathbb{E}}_{{\theta }_i}\left\{\rho (g_n\left[\mathit{X}\right]-{\theta }_i)\right\}\hfill \\ & =& {\int }_{{\mathbb{R}}^n}\left[\sum _{i=0}^{m-1}{q}_i·p\left(\mathit{x}\right|{\theta }_i)\rho (g_n\left[\mathit{x}\right]-{\theta }_i)\right]\mathrm{d}\mathit{x}\hfill \\ & \ge & m·{\int }_{{\mathbb{R}}^n}min\{q_0·p\left(\mathit{x}\right|{\theta }_0),\dots ,q_{m-1}p\left(\mathit{x}\right|{\theta }_{m-1})\}\times \hfill \\ & & \left[{\displaystyle \frac{1}{m}}\sum _{i=0}^{m-1}\rho ({\theta }_i-g_n\left[\mathit{X}\right])\right]\mathrm{d}\mathit{x}\hfill \\ & \stackrel{\left(\mathrm{a}\right)}{=}& m·{\int }_{{\mathbb{R}}^n}min\{q_0·p\left(\mathit{x}\right|{\theta }_0),\dots ,q_{m-1}p\left(\mathit{x}\right|{\theta }_{m-1})\}\times \hfill \\ & & \left[{\displaystyle \frac{1}{m}}\sum _{i=0}^{m-1}\rho \left(T_i({\theta }_i-g_n\left[\mathit{X}\right])\right)\right]\mathrm{d}\mathit{x}\hfill \\ & \stackrel{\left(\mathrm{b}\right)}{\ge }& m·{\int }_{{\mathbb{R}}^n}min\{q_0·p\left(\mathit{x}\right|{\theta }_0),\dots ,q_{m-1}p\left(\mathit{x}\right|{\theta }_{m-1})\}\times \hfill \\ & & \rho \left({\displaystyle \frac{1}{m}}\sum _{i=0}^{m-1}{T}_i({\theta }_i-g_n\left[\mathit{X}\right])\right)\mathrm{d}\mathit{x}\hfill \\ & \stackrel{\left(\mathrm{c}\right)}{=}& m·{\int }_{{\mathbb{R}}^n}min\{q_0·p\left(\mathit{x}\right|{\theta }_0),\dots ,q_{m-1}p\left(\mathit{x}\right|{\theta }_{m-1})\}\times \hfill \\ & & \rho \left({\displaystyle \frac{1}{m}}\sum _{i=0}^{m-1}{T}_i{\theta }_i\right)\mathrm{d}\mathit{x}\hfill \\ & =& m\rho \left({\displaystyle \frac{1}{m}}\sum _{i=0}^{m-1}{T}_i{\theta }_i\right)·{\int }_{{\mathbb{R}}^n}min\{q_0·p\left(\mathit{x}\right|{\theta }_0),\dots ,q_{m-1}p\left(\mathit{x}\right|{\theta }_{m-1})\}\mathrm{d}\mathit{x},\hfill \end{array}$$

where in (a), we have used the unitarity (and hence the norm-preserving property) of $\left\{T_i\right\}$, as $\rho (·)$ is assumed to depend only on the norm of the error vector; in (b), we used the convexity of $\rho$, and in (c), we used the fact that ${\sum }_{i=0}^{m-1}{T}_i=0$, which implies that ${\sum }_{i=0}^{m-1}{T}_i{g}_n\left[\mathit{X}\right]=0$, thus making the bound independent of the estimator, $g_n$. This completes the proof of Theorem 4. □

Theorem 1 is a special case where $m=2$, $T_0=I$, and $T_1=-I$, where I is the $d\times d$ identity matrix. The integral associated with the lower bound of Theorem 4 might not be trivial to evaluate in general for $m\ge 3$. However, there are some choices of the auxiliary parameters that may facilitate calculations. One such choice is as follows. For some positive integer $k<m$, take ${\theta }_0={\theta }_1=\dots ={\theta }_{k-1}\stackrel{▵}{=}{\vartheta }_0$, for some ${\vartheta }_0\in \Theta$, $q_0=q_1=\dots =q_{k-1}\stackrel{▵}{=}Q/k$, for some $Q\in (0,1)$, ${\theta }_k={\theta }_{k+1}=\dots ={\theta }_{m-1}\stackrel{▵}{=}{\vartheta }_1$, for some ${\vartheta }_1\in \Theta$, and finally, $q_k=q_{k+1}=\dots =q_{m-1}=(1-Q)/(m-k)$. The integrand then becomes the minimum between two functions only, as in Section 3. Denoting $\alpha =k/m$, the bound then becomes

$$\begin{array}{ccc}\hfill R_n(\Theta )& \ge & m\rho \left({\displaystyle \frac{1}{m}}\sum _{i=0}^{k-1}{T}_i({\vartheta }_0-{\vartheta }_1)\right)·{\int }_{{\mathbb{R}}^n}min\left\{{\displaystyle \frac{Q}{k}}·p\left(\mathit{x}\right|{\vartheta }_0),{\displaystyle \frac{1-Q}{m-k}}·p\left(\mathit{x}\right|{\vartheta }_1)\right\}\mathrm{d}\mathit{x}\hfill \\ & =& \rho \left({\displaystyle \frac{1}{m}}\sum _{i=0}^{k-1}{T}_i({\vartheta }_0-{\vartheta }_1)\right)·{\int }_{{\mathbb{R}}^n}min\left\{{\displaystyle \frac{Q}{\alpha }}·p\left(\mathit{x}\right|{\vartheta }_0),{\displaystyle \frac{1-Q}{1-\alpha }}·p\left(\mathit{x}\right|{\vartheta }_1)\right\}\mathrm{d}\mathit{x}\hfill \\ & =& \rho \left({\displaystyle \frac{1}{m}}\sum _{i=0}^{k-1}{T}_i({\vartheta }_0-{\vartheta }_1)\right)·\left({\displaystyle \frac{Q}{\alpha }}+{\displaystyle \frac{1-Q}{1-\alpha }}\right)·P_e\left({\displaystyle \frac{(1-\alpha )Q}{(1-\alpha )Q+\alpha (1-Q)}},{\vartheta }_0,{\vartheta }_1\right).\hfill \end{array}$$

(96)

Redefining

$$q={\displaystyle \frac{(1-\alpha )Q}{(1-\alpha )Q+\alpha (1-Q)}},$$

(97)

we have

$${\displaystyle \frac{Q}{\alpha }}+{\displaystyle \frac{1-Q}{1-\alpha }}={\displaystyle \frac{1}{1-\alpha -q+2\alpha q}},$$

(98)

and the following corollary to Theorem 2 is obtained.

Corollary 2.

Let the conditions of Theorem 4 be satisfied. Then,

$$\begin{array}{ccc}\hfill R_n(\Theta )& \ge & \underset{{\vartheta }_0,{\vartheta }_1,\alpha ,q}{sup}\rho \left({\displaystyle \frac{1}{m}}\sum _{i=0}^{k-1}{T}_i({\vartheta }_0-{\vartheta }_1)\right)·{\displaystyle \frac{P_e(q,{\vartheta }_0,{\vartheta }_1)}{1-\alpha -q+2\alpha q}}\hfill \\ & =& \underset{{\vartheta }_0,{\vartheta }_1,q}{sup}\rho \left({\displaystyle \frac{1}{m}}\sum _{i=0}^{k-1}{T}_i({\vartheta }_0-{\vartheta }_1)\right)·{\displaystyle \frac{P_e(q,{\vartheta }_0,{\vartheta }_1)}{min\{q,1-q\}}}.\hfill \end{array}$$

(99)

Note that if m is even, $\alpha ={\displaystyle \frac{1}{2}}$, and $T_0=I$; then, we are actually back to the bound of $m=2$, and so, the optimal bound for even $m>2$ cannot be worse than our bound of $m=2$. We do not have, however, precisely the same argument for an odd m, but for a large m, it becomes immaterial if m is even or odd. In its general form, the bound of Theorem 4 is a heavy optimization problem, as we have the freedom to optimize ${\theta }_0,\dots ,{\theta }_{m-1}$, $T_0,\dots ,T_{m-1}$ (under the constraints that they are all unitary and sum to zero), and $q_0,\dots ,q_{m-1}$ (under the constraints that they are all non-negative and sum to unity).

Another relatively convenient choice is to take ${\theta }_i=T_i^{-1}{\theta }_0$, $i=1,\dots ,m-1$ to obtain another corollary to Theorem 4:

Corollary 3.

Let the conditions of Theorem 2 be satisfied. Then,

$$R_n(\Theta )\ge \underset{{\theta }_0,T_0,\dots ,T_{m-1},q_0,\dots ,q_{m-1}}{sup}m\rho \left({\theta }_0\right)·{\int }_{{\mathbb{R}}^n}min\{q_{0}p\left(\mathit{x}\right|{\theta }_0),q_{1}p\left(\mathit{x}\right|T_1^{-1}{\theta }_0),\dots ,q_{m-1}p\left(\mathit{x}\right|T_{m-1}^{-1}{\theta }_0)\}d\mathit{x}.$$

(100)

Example 12.

To demonstrate a calculation of the extended lower bound for $m=3$, consider the following model. We are observing a noisy signal,

$$Z_i=\vartheta {\varphi }_i+(\vartheta +\zeta ){\psi }_i+N_i,i=1,2,\dots ,n,$$

(101)

where ϑ is the desired parameter to be estimated, ζ is a nuisance parameter, taking values within an interval $[-\delta ,\delta ]$ for some $\delta >0$, $\left\{N_i\right\}$ are i.i.d. Gaussian random variables with zero mean and variance ${\sigma }^2$, and ${\varphi }_i$ and ${\psi }_i$ are two given orthogonal waveforms with ${\sum }_{i=1}^n{\varphi }_i^2={\sum }_{i=1}^n{\psi }_i^2=n$. Suppose we are interested in estimating ϑ based on the sufficient statistics $X={\displaystyle \frac{1}{n}}{\sum }_{i=1}^n{Z}_i{\varphi }_i$ and $Y={\displaystyle \frac{1}{n}}{\sum }_{i=1}^n{Z}_i{\psi }_i$, which are jointly Gaussian random variables with the mean vector $(\vartheta ,\vartheta +\zeta )$ and the covariance matrix ${\displaystyle \frac{{\sigma }^2}{n}}·I$, I being the $2\times 2$ identity matrix. We denote realizations of $(X,Y)$ by $(x,y)$. Let us also denote $\theta =(\vartheta ,\vartheta +\zeta )$. Since we are interested only in estimating ϑ, our loss function will depend only on the estimation error of the first component of θ, which is ϑ. Consider the choice $m=3$ and let $T_i$ be counter-clockwise rotation transformations by $2\pi i/3$, $i=0,1,2$. For a given $\Delta \in (0,\delta ]$, let us select ${\theta }_0=(-\Delta ,0)$, ${\theta }_1=T_1^{-1}{\theta }_0=(\Delta /2,\Delta \sqrt{3}/2)$, and ${\theta }_2=T_2^{-1}{\theta }_0=(\Delta /2,-\Delta \sqrt{3}/2)$. Finally, let $q_0=q_1=q_2={\displaystyle \frac{1}{3}}$. In order to calculate the integral

$$I={\int }_{{\mathbb{R}}^2}min\left\{{\displaystyle \frac{1}{3}}p(x,y|{\theta }_0),{\displaystyle \frac{1}{3}}p(x,y|{\theta }_1),{\displaystyle \frac{1}{3}}p(x,y|{\theta }_2)\right\}dxdy,$$

the plane ${\mathbb{R}}^2$ can be partitioned into three slices over which the integrals contributed are equal. In each such region, the smallest $p(x,y|{\theta }_i)$ is integrated. In other words, every $p(x,y|{\theta }_i)$ in its turn is integrated over the region whose Euclidean distance to ${\theta }_i$ is larger than the distances to the other two values of θ. For ${\theta }_0=(-\Delta ,0)$, this is the region $\{(x,y):x\ge 0,|y|\le x\sqrt{3}\}$. The factor of ${\displaystyle \frac{1}{3}}$ cancels with the three identical contributions from ${\theta }_0$, ${\theta }_1$, and ${\theta }_2$ due to the symmetry. Therefore,

$$\begin{array}{ccc}\hfill I& =& {\int }_0^{\infty }dx{\int }_{-x\sqrt{3}}^{x\sqrt{3}}dy·p(x,y|{\theta }_0)\hfill \\ & =& {\int }_0^{\infty }dx{\int }_{-x\sqrt{3}}^{x\sqrt{3}}·{\displaystyle \frac{n}{2\pi {\sigma }^2}}exp\left\{-{\displaystyle \frac{n{(x+\Delta )}^2+n{y}^2}{2{\sigma }^2}}\right\}dy\hfill \\ & =& \sqrt{{\displaystyle \frac{n}{2\pi {\sigma }^2}}}·{\int }_0^{\infty }exp\left\{-{\displaystyle \frac{n{(x+\Delta )}^2}{2{\sigma }^2}}\right\}\left[1-2Q\left({\displaystyle \frac{x\sqrt{3n}}{\sigma }}\right)\right]dx.\hfill \end{array}$$

(102)

Our next mathematical manipulations in this example are in the spirit of the passage from Theorem 1 to Corollary 1, that is, selecting the test points increasingly close to each other as functions of n, so that the probability of list error would tend to a positive constant. To this end, we change the integration variable x to $u=x\sqrt{n}/\sigma$ and select $\Delta =s\sigma /\sqrt{n}$ for some $s\ge 0$ to be optimized later. Then,

$$\begin{array}{ccc}\hfill I& =& \sqrt{{\displaystyle \frac{n}{2\pi {\sigma }^2}}}·{\int }_0^{\infty }exp\left\{-{\displaystyle \frac{n{(u\sigma /\sqrt{n}+s\sigma /\sqrt{n})}^2}{2{\sigma }^2}}\right\}\left[1-2Q\left(u\sqrt{3}\right)\right]d\left({\displaystyle \frac{u\sigma }{\sqrt{n}}}\right)\hfill \\ & =& {\displaystyle \frac{1}{\sqrt{2\pi }}}·{\int }_0^{\infty }{e}^{-{(u+s)}^2/2}·[1-2Q\left(u\sqrt{3}\right)]du.\hfill \end{array}$$

(103)

The MSE bound then becomes

$$\begin{array}{ccc}\hfill R_n(\theta ,g_n)& \ge & \underset{s\ge 0}{sup}\left\{3·{\left({\displaystyle \frac{s\sigma }{\sqrt{n}}}\right)}^2·{\displaystyle \frac{1}{\sqrt{2\pi }}}·{\int }_0^{\infty }{e}^{-{(u+s)}^2/2}·[1-2Q\left(u\sqrt{3}\right)]du\right\}\hfill \\ & =& {\displaystyle \frac{{\sigma }^2}{n}}·\underset{s\ge 0}{sup}\left\{{\displaystyle \frac{3s^2}{\sqrt{2\pi }}}·{\int }_0^{\infty }{e}^{-{(u+s)}^2/2}·[1-2Q\left(u\sqrt{3}\right)]du\right\}\hfill \\ & \approx & {\displaystyle \frac{0.2514{\sigma }^2}{n}}.\hfill \end{array}$$

(104)

This bound is not as tight as the corresponding bound of $m=2$, which results in $0.3314{\sigma }^2/n$, but it should be kept in mind that here, we have not attempted to optimize the choices of ${\theta }_0$, ${\theta }_1$, ${\theta }_2$, $T_0$, $T_1$, $T_2$, $q_0$, and $q_1$. Instead, we have chosen these parameter values from considerations of computational convenience, just to demonstrate the calculation. This concludes Example 12.

Finally, a comment is in order regarding the possible extensions of Section 4 to the vector case. Such extensions are conceptually straightforward whenever the loss function of the error vector is given by the sum of losses associated with the different components of the error vector. Most notably, when the loss function is the MSE, $\rho \left(ϵ\right)={\parallel ϵ\parallel }^2$, each component of the estimation error can be handled separately by the methods of Section 4. Of course, the two or three test points should be chosen such that they differ in all components of the parameter vector. In the case of three test points, it makes sense to select them equally spaced along a certain straight line of a general direction in $\Theta$. We will not pursue any further this extension in the framework of this work.