Finite-Sample Bounds on the Accuracy of Plug-In Estimators of Fisher Information

Abstract

Finite-sample bounds on the accuracy of Bhattacharya’s plug-in estimator for Fisher information are derived. These bounds are further improved by introducing a clipping step that allows for better control over the score function. This leads to superior upper bounds on the rates of convergence, albeit under slightly different regularity conditions. The performance bounds on both estimators are evaluated for the practically relevant case of a random variable contaminated by Gaussian noise. Moreover, using Brown’s identity, two corresponding estimators of the minimum mean-square error are proposed.

1. Introduction

This work considers the problem of estimating the Fisher information for the location of a univariate probability density function (PDF) f based on n random samples $Y_1,\dots ,Y_n$ independently drawn from f. To clarify, the Fisher information of a differentiable density function f is given by

$$I(f)={\int }_{\{f(t)>0\}}\frac{{(f^{\prime }(t))}^2}{f(t)}\mathrm{d}t,$$

(1)

where $f^{\prime }$ is the derivative of f. For the remainder of the paper, it is assumed that $\{f(t)>0\}=\mathbb{R}$, but an extension to the general case is not difficult. The paper considers plug-in estimators based on kernel density estimates of f. That is, the Fisher information is estimated by plugging a kernel density estimate of f into the right-hand side of (1).

Estimation of the Fisher information in (1) via a plug-in estimator based on kernel density estimates was first considered by Bhattacharya in [1]. Bhattacharya showed that, under mild conditions on f, the plug-in estimator is consistent for a large class of kernels, and he provided bounds on its accuracy in the large (asymptotic) sample regime. These bounds were later revised and improved by Dmitriev and Tarasenko in [2]. However, to the best of our knowledge, no finite-sample regime bounds on the accuracy of Bhattacharya’s estimator can be found in the literature. The paper aims at closing this gap.

Bounds on the accuracy of plug-in estimators rely on bounds on the accuracy of the underlying density estimators. For kernel-based density estimators, such bounds have received considerable attention in the literature. For example, Schuster [3] showed that, under mild regularity conditions, the estimation error for higher-order derivatives can be controlled by the estimation error for the corresponding cumulative distribution function (CDF). The interested reader is referred to [4,5,6,7,8] and the references therein. In this paper, as a preliminary result for the analysis of Bhattacharya’s estimator, the bounds in [3] are further tightened by replacing some suboptimal constants with the optimal ones.

A problem that arises in the performance analysis of plug-in estimators for Fisher information is that the score function of the estimated density, that is, the ratio of the derivative of the PDF and the PDF itself, is hard to bound, especially in the tails. Bhattacharya worked around this problem by truncating the integration range in (1), thus avoiding evaluation of the estimated score function on these critical regions. However, in order for the estimator to stay consistent, this truncation has to be done rather aggressively so that the error introduced by ignoring the tails can outweigh the approximation error introduced by the density estimate. In this paper, we propose a simple remedy that allows for a much less aggressive truncation of the integration range and, in turn, for significantly tighter bounds on the approximation error. Namely, we propose the clipping of the score function whenever it exceeds a suitably chosen upper bound. In the vast majority of cases, the corresponding clipped estimates of Fisher estimation are identical to their non-clipped counterparts, meaning that the clipping has a negligible influence on the estimation accuracy. However, the knowledge that extreme values of the score would have been clipped, had they occurred, allows for much-improved performance guarantees.

It should be explicitly stated that this paper does not address the question of how best to estimate Fisher information. Although this question is highly interesting and relevant, it is far beyond the scope of this work. In addition, it is not the aim of the paper to compare the plug-in estimator to alternative estimators for the Fisher information or to claim that it provides superior result. A variety of well-motivated parametric and nonparametric Fisher information estimators have been proposed in the literature; see, for example, [9,10,11] and the references therein. However, comparing and contrasting these estimators in a fair manner is not straightforward and arguably constitutes a research question in its own right. Finally, the problem of obtaining estimator-independent bounds on the sample complexity of Fisher information falls under the umbrella of estimation of nonlinear functionals; see, for example, [12]. Most of the commonly used information measures, such as entropy, relative entropy, and mutual information, are nonlinear functionals, and their estimation has recently received considerable attention; the interested reader is referred to [13,14,15,16,17] and the references therein.

Despite its limited scope, we are convinced that the work presented in this paper is useful in a wider context. First, from a theoretical point of view, it strengthens some classic results in nonparametric estimation and, as explained above, provides bounds for the finite-sample regime, thus filling a gap in the literature. Second, from a practical perspective, the Fisher information typically provides useful bounds or limits on the estimation error (e.g., the well-known Cramér–Rao lower bound), but is not in itself the quantity of interest—an exception is the case of estimating a random signal in additive Gaussian noise, where the minimum mean square error (MMSE) and other relevant quantities can be expressed in term of the Fisher information. The problem of estimating Fisher information also arises in image processing, model selection, experimental design, and many more areas. Applications of our results include, for example, to provide the Cramér–Rao bound and, for the case of a random variable in additive Gaussian noise, to address the power allocation problem [18]. These connections will be discussed in more detail in Section 4. Most often, however, Fisher information plays the role of side information, and its estimation does not warrant investing large computational resources. This prevents the use of sophisticated estimators, which require solving non-trivial optimization problems. In contrast, kernel density estimates are relatively easy to compute and have been widely used in nonparametric statistics so that efficient implementations in software or even hardware [19] are readily available. Hence, for the foreseeable future, plug-in estimators are bound to remain a common and often the only viable option for estimating Fisher information in practice.

The paper is organized as follows: Section 2 revisits Bhattacharya’s estimator. In particular, Theorem 1 provides explicit and tighter non-asymptotic bounds on its convergence rate, improving the results in [1,2]. Furthermore, Theorem 2 provides an alternative bound under the additional assumption that the density function is upper bounded within any given interval. The explicit non-asymptotic results enable us to see that the sample complexity of Bhattacharya’s estimator is considerable and that the potentially unbounded score function is a critical bottleneck for tighter bounds. Section 3 proposes a “harmless” modification of Bhattacharya’s estimator, namely, a clipping of the estimated score function, which is shown to be sufficient to remedy its large sample complexity. In particular, Theorem 3 shows that the clipped estimator has significantly better bounds on rates of convergence, albeit with slightly different assumptions on the PDF. Section 4 evaluates the convergence rates of the two estimators for the practically relevant case of a random variable contaminated by additive Gaussian noise. Moreover, using Brown’s identity, which relates the Fisher information and the MMSE, consistent estimators for the MMSE are proposed and their rates of convergence are evaluated in Proposition 1. Section 5 concludes the paper.

Notation

The expected value and variance of a random variable X are denoted by $\mathbb{E}[X]$ and $Var(X)$, respectively. The gamma function is denoted by $\mathsf{\Gamma }(·)$. Estimators of a PDF f based on n samples are denoted by $f_n$. No notational distinction is made between an estimator, which is a random variable, and its realizations (estimates), which are deterministic. However, the difference will be clear from the context or will be highlighted explicitly otherwise. The nth derivative of a function $F:\mathbb{R}\to \mathbb{R}$ is denoted by $F^{(n)}$; the first-order derivative is also denoted by $F^{\prime }$ to improve readability.

2. Bhattacharya’s Estimator

In this section, we revisit the asymptotically consistent estimator proposed by Bhattacharya in [1] and produce explicit and non-asymptotic bounds on its accuracy.

Bhattacharya’s estimator is given by

$$I_n={\int }_{-k_n}^{k_n}\frac{{\left(f_n^{\prime }(t)\right)}^2}{f_n(t)}\mathsf{d}t,$$

(2)

where $k_n\ge 0$ determines the integration interval as a function of the sample size n and the unknown functions f and $f^{\prime }$ are replaced by their kernel estimates, that is,

$$\begin{array}{cc}\hfill f_n(t)& =\frac{1}{n}\sum _{i=1}^n\frac{1}{a_0}K\left(\frac{t-Y_i}{a_0}\right),\hfill \end{array}$$

(3)

$$\begin{array}{cc}\hfill f_n^{\prime }(t)& =\frac{1}{n}\sum _{i=1}^n\frac{1}{a_1}{K}^{\prime }\left(\frac{t-Y_i}{a_1}\right).\hfill \end{array}$$

(4)

Here, $a_0,a_1>0$ are bandwidth parameters, and $K:\mathbb{R}\to \mathbb{R}$ denotes the kernel, which is assumed to satisfy certain regularity conditions that will be discussed later in this section.

2.1. Estimating a Density and Its Derivatives

In order to analyze plug-in estimators, it is necessary to obtain rates of convergence for $f_n$ and $f_n^{\prime }$, that is, the kernel estimators of the density and its derivative. The following result, which is largely based on the proof by Schuster in [3], provides such rates. The proof in [3] makes use of the Dvoretzky–Kiefer–Wolfowitz (DKW) inequality for the empirical CDF. The next lemma refines the result in [3] by using the best possible constant for the DKW inequality shown in [20].

Lemma 1.

Let $r\in \{0,1\}$ and

$$\begin{array}{cc}\hfill v_r& ={\int }_{-\infty }^{\infty }\left|K^{(r+1)}(t)\right|\mathsf{d}t,\hfill \end{array}$$

(5)

$$\begin{array}{cc}\hfill {\delta }_{r,a_r}& =\underset{t\in \mathbb{R}}{sup}\left|\mathbb{E}\left[f_n^{(r)}(t)\right]-f^{(r)}(t)\right|.\hfill \end{array}$$

(6)

Then, for any $ϵ>{\delta }_{r,a_r}$ and any $n\ge 1$, the following bound holds:

$$\mathbb{P}\left[\underset{t\in \mathbb{R}}{sup}\left|f_n^{(r)}(t)-f^{(r)}(t)\right|>ϵ\right]\le 2{\mathrm{e}}^{-2n\frac{a_r^{2r+2}{(ϵ-{\delta }_{r,a_r})}^2}{v_r^2}}.$$

(7)

Proof. 

See Appendix A. □

2.2. Analysis of Bhattacharya’s Estimator

The following theorem is a non-asymptotic refinement of the result obtained by Bhattacharya in Theorem 3 of [1] and Dmitriev and Tarasenko in Theorem 1 of [2].

Theorem 1.

Assume that there exists a function $\varphi :\mathbb{R}\to \mathbb{R}$ such that

$$\underset{\left|t\right|\le x}{sup}\frac{1}{f(t)}\le \varphi (x),\forall x\in \mathbb{R}.$$

(8)

Then, provided that

$$\underset{\left|t\right|\le k_n}{sup}\left|f_n^{(r)}(t)-f^{(r)}(t)\right|\le {ϵ}_r,r\in \{0,1\},$$

(9)

and

$${ϵ}_0\varphi (k_n)<1,$$

(10)

the following bound holds:

$$\left|I(f)-I_n\right| \le \frac{4{ϵ}_1{k}_n{\rho }_{max}(k_n)+2{ϵ}_1^2{k}_n\varphi (k_n)+{ϵ}_0\varphi (k_n)I(f)}{1-{ϵ}_0\varphi (k_n)}+c(k_n),$$

(11)

where

$$\begin{array}{cc}\hfill {\rho }_{max}(k_n)& =\underset{\left|t\right|\le k_n}{sup}\left|\frac{f^{\prime }(t)}{f(t)}\right|,\hfill \end{array}$$

(12)

$$\begin{array}{cc}\hfill c(k_n)& =I(f)-{\int }_{-k_n}^{k_n}\frac{{(f^{\prime }(t))}^2}{f(t)}\mathsf{d}t.\hfill \end{array}$$

(13)

Proof. 

See Appendix B. □

The bound in (11) is an improvement of the original bound in [1,2], which contains terms of the form ${ϵ}_0{\varphi }^4(k_n)$.

Note that $\varphi (k_n)$ in (8) can be rapidly increasing with $k_n$. For example, as will be shown later, $\varphi (k_n)$ increases super-exponentially with $k_n$ for a random variable contaminated by Gaussian noise. This implies that, while Bhattacharya’s estimator converges, the rate of convergence guaranteed by the bound in (11) is extremely slow. A modified bound is proposed in the subsequent theorem.

Theorem 2.

Assume that $f(t)$ is bounded on the interval $t\in [-k_n,k_n]$, i.e.,

$$\underset{\left|t\right|\le k_n}{sup}f(t)\le f_0,$$

(14)

for some $f_0\in \mathbb{R}$. If the assumptions in (8), (9), and (10) hold, then

$$|I(f)-I_n|\le \left[{ϵ}_1\left(4+d_f(k_n)+d_{f_n}(k_n)\right)+{ϵ}_0\left(2+d_{f_n}(k_n)\right){\rho }_{max}(k_n)\right]\psi ({ϵ}_0,k_n)+c(k_n),$$

(15)

where ${\rho }_{max}$ and c are given by (12) and (13), respectively,

$$\psi ({ϵ}_0,k_n)=max\left(log(f_0+{ϵ}_0),log\left(\frac{\varphi (k_n)}{1-{ϵ}_0\varphi (k_n)}\right)\right),$$

(16)

and $d_g(k_n)$ denotes the number of zeros of the derivative of the function g on the interval $[-k_n,k_n]$, i.e.,

$$d_g(k_n)=\left|\left\{t\in [-k_n,k_n]:g^{\prime }(t)=0\right\}\right|.$$

(17)

Proof. 

See Appendix C. □

Remark 1.

Note that ψ in (15) is on the order of $log(\varphi (k_n))$, which typically increases much more slowly with $k_n$ than ϕ in (11). As a result, the bound in Theorem 2 can lead to a better bound on the convergence rate than that in Theorem 1, given appropriate upper bounds on $d_f$ and $d_{f_n}$. Since Gaussian blurring of a univariate density function never creates new maxima, we have that $d_{f_Y}\le d_{f_X}$, which is a constant. However, to the best of our knowledge, the only known upper bound on $d_{f_n}$ is given by $d_{f_n}\le n$ [21] (Theorem 2), which is not useful in practice. Despite this drawback, we include Theorem 2 for the sake of completeness and in the hope that tighter bounds on $d_{f_n}$ might be established in the future.

The main problem in the convergence analysis of the estimator in (2) is that $1/f_n(t)$ is only bounded if $f(t)>{ϵ}_0$. For distributions with sub-Gaussian tails, this implies that the interval $[-k_n,k_n]$, on which this is guaranteed to be the case, grows sub-logarithmically (compare Theorem 4), causing the required number of samples to grow super-exponentially. In next section, we propose an estimator that has better guaranteed rates of convergence.

3. The Clipped Bhattacharya Estimator

In order to remedy the slow guaranteed convergence rates of Bhattacharya’s estimator, we dispense with the tail assumption in (8), but introduce the new assumption that the unknown true score function $\rho (t)=f^{\prime }(t)/f(t)$ is bounded (in absolute value) by a known function $\overline{\rho }$. This allows us to clip $f_n^{\prime }(t)/f_n(t)$ and, in turn, $1/f_n(t)$ without affecting the consistency of the estimator.

Theorem 3.

Assume that there exists a function $\overline{\rho }:\mathbb{R}\to \mathbb{R}$ such that

$$\left|\rho (t)\right|\le \left|\overline{\rho }(t)\right|,\forall t\in \mathbb{R}$$

(18)

and let

$$I_n^{\mathrm{c}}={\int }_{-k_n}^{k_n}min\left\{\left|{\rho }_n(t)\right|,\left|\overline{\rho }(t)\right|\right\}\left|f_n^{\prime }(t)\right|\mathsf{d}t,$$

(19)

where

$${\rho }_n(t)=\frac{f_n^{\prime }(t)}{f_n(t)}.$$

(20)

Under the assumptions in (9), it holds that

$$\begin{array}{}\hfill (21)& \hfill |I(f)-I_n^{\mathrm{c}}|& \le max\left\{4{ϵ}_1{\mathsf{\Phi }}^1(k_n)+2{ϵ}_0{\mathsf{\Phi }}^2(k_n)+c(k_n),3{ϵ}_1{\mathsf{\Phi }}_{\max }^1(k_n)+{ϵ}_0{\mathsf{\Phi }}_{\max }^2(k_n)\right\}\hfill \hfill (22)& & \le 4{ϵ}_1{\mathsf{\Phi }}_{\max }^1(k_n)+2{ϵ}_0{\mathsf{\Phi }}_{\max }^2(k_n)+c(k_n),\hfill \end{array}$$

where $c(k_n)$ is defined in (13) and

$$\begin{array}{cc}\hfill {\mathsf{\Phi }}^m(x)& ={\int }_{-x}^x\left|{\rho }^m(t)\right|\mathsf{d}t,\hfill \end{array}$$

(23)

$$\begin{array}{cc}\hfill {\mathsf{\Phi }}_{\max }^m(x)& ={\int }_{-x}^x\left|{\overline{\rho }}^m(t)\right|\mathsf{d}t.\hfill \end{array}$$

(24)

In addition, if $f(t)$ is bounded as in (14), then

$${\mathsf{\Phi }}^m(k_n)\le min\left\{(2+d_f){\overline{\rho }}^{m-1}(k_n)\psi (0,k_n),{\mathsf{\Phi }}_{\max }^m(k_n)\right\},$$

(25)

where ψ and $d_f$ are defined in (16) and (17), respectively.

Proof. 

See Appendix D. □

For the upper-bound function $\overline{\rho }(t)$ in assumption (18), in practice, we can set $\overline{\rho }(k_n)={\rho }_{max}(k_n)$ if the latter is available. Although ${\rho }_{max}(k_n)$ also increases with $k_n$, it usually increases much more slowly than $\varphi (k_n)$. For example, as shown later, ${\rho }_{max}(k_n)$ is linear in $k_n$ in the Gaussian noise case. As a result, better bounds on the convergence rate can be shown for the clipped estimator.

4. Estimation of the Fisher Information of a Random Variable in Gaussian Noise

This section evaluates the results of Section 2 and Section 3 for the important special case of a random variable contaminated by additive Gaussian noise. To this end, we let $f_Y$ denote the PDF of a random variable

$$Y=\sqrt{\mathsf{snr}}X+Z,$$

(26)

where $\mathsf{snr}>0$ is a signal-to-noise-ratio parameter, X is an arbitrary random variable, Z is a standard Gaussian random variable, and X and Z are independent. We are interested in estimating the Fisher information of $f_Y$. We only make the very mild assumption that X has a finite second moment, but otherwise, it is allowed to be an arbitrary random variable. We further assume that $\mathsf{snr}$ is known and that Gaussian kernels are used in the density estimators, i.e.,

$$K(t)=\frac{1}{\sqrt{2\pi }}{\mathrm{e}}^{-\frac{t^2}{2}}.$$

(27)

The following lemma provides explicit expressions for the quantities appearing in Section 2 and Section 3 that are needed to evaluate the error bounds for the Bhattacharya and the clipped estimator.

Lemma 2.

Let K be as in (27). Then,

$$\begin{array}{cc}\hfill {\delta }_{r,a_r}& \le a_r·\left\{\begin{array}{cc}\frac{1}{\pi \sqrt{e}},\hfill & r=0\hfill \\ \frac{\frac{2}{\mathrm{e}}+1}{\pi },\hfill & r=1,\hfill \end{array}\right.\hfill \end{array}$$

(28)

$$\begin{array}{cc}\hfill v_r& =\left\{\begin{array}{cc}\sqrt{\frac{2}{\pi }},\hfill & r=0\hfill \\ \sqrt{\frac{2}{\mathrm{e}\pi }},\hfill & r=1,\hfill \end{array}\right.\hfill \end{array}$$

(29)

$$\begin{array}{cc}\hfill {\rho }_{max}(k_n)& \le \sqrt{3\mathsf{snr}Var(X)}+3k_n,\hfill \end{array}$$

(30)

$$\begin{array}{cc}\hfill I(f_Y)& \le 1,\hfill \end{array}$$

(31)

$$\begin{array}{cc}\hfill \varphi (t)& \le \sqrt{2\pi }{\mathrm{e}}^{t^2+\mathsf{snr}\mathbb{E}[X^2]}.\hfill \end{array}$$

(32)

Proof. 

See Appendix F. □

We now bound $c(k_n)$. To this end, we need the notion of sub-Gaussian random variables: A random variable X is said to be $\alpha$-sub-Gaussian if

$$\mathbb{E}[{\mathrm{e}}^{tX}]\le {\mathrm{e}}^{\frac{{\alpha }^2{t}^2}{2}}\forall t\in \mathbb{R}.$$

(33)

Lemma 3.

Suppose that $\mathbb{E}[X^2]<\infty$. Then,

$$c(k_n)\le \underset{v>0}{inf}\frac{2{\mathsf{\Gamma }}^{\frac{1}{(1+v)}}\left(v+\frac{1}{2}\right)}{{\pi }^{\frac{1}{2(1+v)}}}{\left(\frac{\mathsf{snr}\mathbb{E}[X^2]+1}{k_n^2}\right)}^{\frac{v}{1+v}}.$$

(34)

In addition, if $\left|X\right|$ is α-sub-Gaussian, then

$$c(k_n)\le \underset{v>0}{inf}\frac{2{\mathsf{\Gamma }}^{\frac{1}{(1+v)}}\left(v+\frac{1}{2}\right)}{{\pi }^{\frac{1}{2(1+v)}}}{\left(2{\mathrm{e}}^{\frac{{\alpha }^2\mathsf{snr}-k_n^2}{2}}\right)}^{\frac{v}{1+v}}.$$

(35)

Proof. 

See Appendix G. □

4.1. Convergence of Bhattacharya’s Estimator

By combining the results in Lemma 1, Theorem 1, Lemma 2, and Lemma 3, we have the following theorem.

Theorem 4.

Let K be as in (27). Choose the parameters of Bhattacharya’s estimator as follows: $a_0=n^{-w_0}$, where $w_0\in \left(0,\frac{1}{4}\right)$, $a_1=n^{-w_1}$, where $w_1\in \left(0,\frac{1}{6}\right)$, and $k_n=\sqrt{ulog(n)}$, where $u\in \left(0,min(w_0,w_1)\right)$. Then, for $n^{w_0-u}>c_5$,

$$\mathbb{P}\left[\left|I_n-I(f_Y)\right|\ge {\epsilon }_n\right]\le 2{\mathrm{e}}^{-c_1{n}^{1-4w_0}}+2{\mathrm{e}}^{-c_2{n}^{1-6w_1}},$$

(36)

where

$${\epsilon }_n\le \frac{n^{-w_1}\sqrt{ulog(n)}\left(4c_3+12\sqrt{ulog(n)}+2c_5{n}^{u-w_1}\right)+c_5{n}^{u-w_0}}{1-c_5{n}^{u-w_0}}+\frac{c_4}{\sqrt{ulog(n)}},$$

(37)

and where the constants are given by

$$\begin{array}{cc}\hfill c_1& =\pi {\left(1-\frac{1}{\sqrt{2\pi \mathrm{e}}}\right)}^2,\hfill \end{array}$$

(38)

$$\begin{array}{cc}\hfill c_2& =\mathrm{e}\pi {\left(1-\frac{\frac{2}{\mathrm{e}}+1}{\sqrt{2\pi }}\right)}^2,\hfill \end{array}$$

(39)

$$\begin{array}{cc}\hfill c_3& =\sqrt{3\mathsf{snr}Var(X)},\hfill \end{array}$$

(40)

$$\begin{array}{cc}\hfill c_4& =\frac{2{\mathsf{\Gamma }}^{\frac{1}{2}}\left(\frac{3}{2}\right)\sqrt{\mathsf{snr}\mathbb{E}[X^2]+1}}{{\pi }^{\frac{1}{4}}},\hfill \end{array}$$

(41)

$$\begin{array}{cc}\hfill c_5& =\sqrt{2\pi }{\mathrm{e}}^{\mathsf{snr}\mathbb{E}\left[X^2\right]}.\hfill \end{array}$$

(42)

In addition, if $\left|X\right|$ is α-sub-Gaussian, then

$${\epsilon }_n\le \frac{n^{-w_1}\sqrt{ulog(n)}\left(4c_3+12\sqrt{ulog(n)}+2c_5{n}^{u-w_1}\right)+c_5{n}^{u-w_0}}{1-c_5{n}^{u-w_0}}+c_6{n}^{-\frac{u}{4}},$$

(43)

where

$$c_6=\frac{2^{\frac{3}{2}}{\mathsf{\Gamma }}^{\frac{1}{2}}\left(\frac{3}{2}\right){\mathrm{e}}^{\frac{{\alpha }^2\mathsf{snr}}{4}}}{{\pi }^{\frac{1}{4}}}.$$

(44)

Proof. 

See Appendix H. □

Note that the parameters $k_n$, $a_0$, and $a_1$ are chosen so as to guarantee the convergence of $I_n(f_n)$ to $I(f_Y)$ with probability 1. For the details, please refer to the proof in Appendix H.

The parameters u and w in the above theorem are auxiliary variables that couple the bandwidth of the kernel density estimators in (3) and (4) with the integration range of the Fisher information estimator in (2). Choosing them according to Theorem 4 results in a trade-off between precision, ${\epsilon }_n$, and confidence, i.e., the probability of the estimation error exceeding ${\epsilon }_n$. On the one hand, small values of u and large values of w result in better precision (i.e., small ${\epsilon }_n$) at the cost of a lower confidence (i.e., large probability of exceeding ${\epsilon }_n$). On the other hand, large values of u and small values of w improve the confidence but deteriorate the precision. In turn, this also affects the convergence rates, meaning that faster convergence of the precision can be achieved at the expense of a slower convergence of the confidence and vice versa.

4.2. Convergence of the Clipped Estimator

From the evaluation of Bhattacharya’s estimator in Theorem 4, it is apparent that the bottleneck term is the truncation parameter $k_n=\sqrt{ulog(n)}$, which results in slow precision decay of the order ${\epsilon }_n=O\left(\frac{1}{\sqrt{ulog(n)}}\right)$. Next, it is shown that the clipped estimator results in improved precision over Bhattacharya’s estimator. Specifically, the precision will be shown to decay polynomially in n instead of logarithmically. Another benefit of the clipped estimator is that its error analysis holds for every $n\ge 1$.

By utilizing the results in Lemma 1, Lemma 2, and Lemma 3, we specialize the result in Theorem 2 to the Gaussian noise case.

Theorem 5.

Let K be as in (27). Choose the parameters of the clipped estimator as follows: $a_0=n^{-w_0}$, where $w_0\in \left(0,\frac{1}{4}\right)$, $a_1=n^{-w_1}$, where $w_1\in \left(0,\frac{1}{6}\right)$, and $k_n=n^u$, where $u\in \left(0,min\left(\frac{w_0}{3},\frac{w_1}{2}\right)\right)$. Then, for $n\ge 1$

$$\mathbb{P}\left[\left|I_n^{\mathrm{c}}-I(f_Y)\right|\ge {\epsilon }_n\right]\le 2{\mathrm{e}}^{-c_1{n}^{1-4w_0}}+2{\mathrm{e}}^{-c_2{n}^{1-6w_1}},$$

(45)

where

$${\epsilon }_n\le 4n^{3u-w_0}\left(c_3^2{n}^{-2u}+3\right)+4n^{2u-w_1}\left(2c_3{n}^{-u}+3\right)+c_4{n}^{-u},$$

(46)

and the constants $c_1$ to $c_4$ are as in Theorem 4. In addition, if $\left|X\right|$ is α-sub-Gaussian, then

$${\epsilon }_n\le 4n^{3u-w_0}\left(c_3^2{n}^{-2u}+3\right)+4n^{2u-w_1}\left(2c_3{n}^{-u}+3\right)+c_6{\mathrm{e}}^{-\frac{n^{2u}}{4}},$$

(47)

where $c_6$ is given by (44).

Proof. 

See Appendix I. □

Again, the parameters $k_n$, $a_0$, and $a_1$ are chosen to guarantee the consistency of the estimator. For further details, please refer to Appendix I.

4.3. Applications to the Estimations of the MMSE

As discussed in the introduction, the Fisher information is often merely a proxy for the actual quantity of interest. One accuracy measure that is typically of interest is the MMSE, which is defined as

$$\mathrm{mmse}(X|Y)=\mathbb{E}\left[{(X-\mathbb{E}[X|Y])}^2\right].$$

(48)

In additive Gaussian noise, the MMSE can not only be bounded by the Fisher information, but both are related via Brown’s identity:

$$I(f_Y)=1-\mathsf{snr}\mathrm{mmse}(X|Y).$$

(49)

Based on this relation, we propose the following estimators for the MMSE:

$${\mathrm{mmse}}_n(X,\mathsf{snr})=\frac{1-I_n}{\mathsf{snr}}$$

(50)

and

$${\mathrm{mmse}}_n^{\mathrm{c}}(X,\mathsf{snr})=\frac{1-I_n^{\mathrm{c}}}{\mathsf{snr}}.$$

(51)

The results for the estimators of Fisher information in Theorem 4 and Theorem 5 can be immediately extended to the MMSE estimators as follows.

Proposition 1.

Let K be as in (27), and let $w_0,w_1$, and n be such that they satisfy the conditions in Theorem 4. It then holds that

$$\mathbb{P}\left[\left|{\mathrm{mmse}}_n(X,\mathsf{snr})-\mathrm{mmse}(X,\mathsf{snr})\right|\ge \mathsf{snr}{\epsilon }_n\right]\le 2{\mathrm{e}}^{-c_1{n}^{1-4w_0}}+2{\mathrm{e}}^{-c_2{n}^{1-6w_1}},$$

(52)

where ${\epsilon }_n$, $c_1$, and $c_2$ are given in Theorem 4.

Proposition 2.

Let K be as in (27), and let $w_0$ and $w_1$ be such that they satisfy the conditions in Theorem 5. It then holds that

$$\mathbb{P}\left[\left|{\mathrm{mmse}}_n^{\mathrm{c}}(X|Y)-\mathrm{mmse}(X|Y)\right|\ge \mathsf{snr}{\epsilon }_n\right]\le 2{\mathrm{e}}^{-c_1{n}^{1-4w_0}}+2{\mathrm{e}}^{-c_2{n}^{1-6w_1}},$$

(53)

where ${\epsilon }_n$, $c_1$, and $c_2$ are given in Theorem 5.

4.4. Sample Complexity

Finally, we demonstrate the difference in the bounds on the convergence rates between Bhattacharya’s estimator and its clipped version by comparing the sample complexity of the two estimators, that is, the required number of samples to guarantee a given accuracy with a given confidence. MATLAB implementations of both estimators, as well as the code used to generate the figures below, can be found in [22].

To this end, we consider the simple example of estimating the density of a Gaussian random variable in additive Gaussian noise. More precisely, we assume that X and Z in (26) are independent and identically distributed according to the standard normal distribution $\mathcal{N}(0,1)$, and that $\mathsf{snr}=1$. This trivially implies that X is $\alpha$-sub-Gaussian with $\alpha =1$. In order to make the comparison as fair as possible, the parameters of the kernel estimators, $a_0$, $a_1$, and $k_n$, are not chosen according to Theorem 4 or Theorem 5, but are calculated by numerically minimizing the required number of samples; see [22] for details.

Let $P_{\mathrm{err}}=\mathbb{P}\left[\left|I_n-I(f_Y)\right|\ge {\epsilon }_n\right]$. The left-hand plot in Figure 1 shows the corresponding bounds on the sample complexities of the two estimators with $P_{\mathrm{err}}=0.2$ and ${\epsilon }_n$ varying from $0.1$ to $0.9$. Note that the results with larger ${\epsilon }_n$ are not shown because $I(f_Y)\le 1$, as shown in Lemma 2. Moreover, the right-hand plot in Figure 1 shows the sample complexities for ${\epsilon }_n=0.5$ with $P_{\mathrm{err}}$ varying from $0.1$ to $0.9$. By inspection, the clipped estimator reduces the sample complexity by several orders of magnitude; note that the y-axes scale logarithmically. As discussed before, this does not imply that the clipped estimator is more accurate in general. However, it does imply that the clipped estimator provides significantly better worst-case performance, i.e., it requires significantly fewer samples to guarantee a certain precision or confidence. Finally, note that this improvement comes at a low cost in terms of complexity and regularity assumptions. The complexity of both algorithms is almost identical, with the clipped estimator only requiring an additional evaluation of $\overline{\rho }$. The regularity conditions are identical for bounded density functions, and slightly stronger for the clipped estimator for unbounded density functions.

5. Conclusions

This work focused on the estimation of the Fisher information for the location of a univariate random variable using plug-in estimators based on estimators of the PDF and its derivative. Two estimators of the Fisher information were considered. The first estimator is the estimator due to Bhattacharya, for which new, sharper convergence results were shown. The paper also proposed a second estimator, termed a clipped estimator, which provides better bounds on the convergence rates. The accuracy bounds on both estimators were specialized to the practically relevant case of a random variable contaminated by additive Gaussian noise. Moreover, using special proprieties of the Gaussian noise case, two estimators for the MMSE were proposed, and their convergence rates were analyzed. This was done by using Brown’s identity, which connects the Fisher information and the MMSE. Finally, using a numerical example, it was demonstrated that the proposed clipped estimator can achieve a significantly lower sample complexity at little or no additional cost.