On the Admissibility of Simultaneous Bootstrap Confidence Intervals

Abstract

Simultaneous confidence intervals are commonly used in joint inference of multiple parameters. When the underlying joint distribution of the estimates is unknown, nonparametric methods can be applied to provide distribution-free simultaneous confidence intervals. In this note, we propose new one-sided and two-sided nonparametric simultaneous confidence intervals based on the percentile bootstrap approach. The admissibility of the proposed intervals is established. The numerical results demonstrate that the proposed confidence intervals maintain the correct coverage probability for both normal and non-normal distributions. For smoothed bootstrap estimates, we extend Efron’s (2014) nonparametric delta method to construct nonparametric simultaneous confidence intervals. The methods are applied to construct simultaneous confidence intervals for LASSO regression estimates.

1. Introduction

In statistical sciences, the computation of simultaneous confidence intervals (SCIs) for parameters of interest plays a major role [1]. For instance, multiple testing problems, regression analysis, and multivariate inference are well known application areas. When the finite sampling distribution of the estimator (here, a vector) under the alternative is known, exact SCIs can be computed. However, that is often not the case, and when the underlying distribution of the estimator is difficult to obtain analytically, bootstrap methods (using drawing with replacement from the data) can be used to approximate its sampling distribution (see, e.g., [2,3], among others). The authors of [4] provide an algorithm to construct SCIs for multivariate estimates, which calculates the number of bootstrap samples that fall outside a confidence region. To achieve the pre-specified level of $(1-\alpha )\times 100\%$, one can trial different values of the lower and upper bounds of the intervals until attaining the target level. In [5], a more efficient way of computing $(1-\alpha )\times 100\%$ upper SCIs was developed by sorting the bootstrap realizations of the multivariate estimates by the maximum rank across all the coordinates. Then, the $(1-\alpha )\times 100\%$ percentile of the maximum rank is determined. The limit of the confidence interval for each coordinate is the value of the estimate sharing the same rank as the $(1-\alpha )$ percentile of the maximum rank. By symmetry, the $(1-\alpha )\times 100\%$ lower SCIs can also be constructed.

In this note, we improve their method by sharpening the limits of the SCIs for each coordinate. It is observed that the value of the estimate sharing the same rank as the $(1-\alpha )\times 100\%$ percentile of the maximum rank is indeed higher than the estimates in the selected $(1-\alpha )\times 100\%$ samples, but the threshold for each coordinate is too conservative. We can tighten them rather than using the same maximum rank cutoff values (see details below). Furthermore, we show that the new proposed SCIs are admissible in the sense that there exist no other bootstrap SCIs which have the same $(1-\alpha )\times 100\%$ coverage probability while being uniformly smaller than or equal to the proposed confidence intervals in all coordinates and strictly smaller in at least one coordinate. We apply the proposed nonparametric SCIs on LASSO penalized regression estimates to perform post model selection inference. We observe that the nonparametric method is able to preserve the asymmetry of the sampling distribution of the LASSO penalized regression estimates. This is advantageous compared to parametric-based SCIs, which are often constructed symmetrically around the point estimate ignoring the asymmetric nature of the penalized estimates.

2. Methods

In the following, let Z denote a dataset and let $\theta$ = ${({\theta }_1,\dots ,{\theta }_p)}^{\prime }$ denote a vector of parameters of interest. Then, the SCI $C_j=[a_j(Z),b_j(Z)],j=1,\dots ,p$ has a simultaneous coverage level of $1-\alpha$, if

$$P\left(\bigcap _{j=1}^p\left\{a_j(Z)\le {\theta }_j\le b_j(Z)\right\}\right)\ge 1-\alpha .$$

(1)

Bootstrap-based algorithms for the computations of the thresholds $a_j(Z)$ and $b_j(Z)$ will now be discussed. First, we will introduce the Mandel–Betensky intervals (MB) and our proposed modification (MMB), which are obtained by sharpening their individual thresholds. Later, we show that they are admissible at level $(1-\alpha ).$

2.1. Mandel–Betensky Intervals and Modified (MMB) Upper Intervals

We first estimate the upper bounds (using resampling) and will focus later on two-sided SCIs. Since the computation of the upper MBB bounds uses the same initial steps as those of the MB intervals, we provide their computational details simultaneously:

• Generate B bootstrap samples $Z_b,$$b=1,\dots ,B,$ by sampling with replacement and obtain the bootstrap estimates ${\tilde{\theta }}_b=\{{\tilde{\theta }}_{b1},\dots ,{\tilde{\theta }}_{bp}\}.$
• For each coordinate j, $j=1,\dots ,p$, order the bootstrap estimates as ${\tilde{\theta }}_{(1j)}\le {\tilde{\theta }}_{(2j)}\le \cdots \le {\tilde{\theta }}_{(Bj)}.$ In the case of tied observation, we use random ranks. The rank of ${\tilde{\theta }}_{bj}$ is denoted as $r(b,j).$ For each sample, find the maximum rank $r(b)={\max }_{j}r(b,j)$, which is the largest rank associated with the bth bootstrap sample.
• Calculate the $(1-\alpha )$—percentile $r_{1-\alpha }$ of $r(b),$$b=1,2,\dots ,B.$
• Denote the collection of bootstrap samples with the maximum rank below $r_{1-\alpha }$ as $\mathsf{\Phi }=\{b:r(b)\le r_{1-\alpha }\}.$
• Construct the upper limits for each coordinate: For coordinate j, calculate $t_j={\max }_{b\in \mathsf{\Phi }}r(b,j).$ The upper limits of the SCIs are given by ${\tilde{\theta }}_{(t_{1}1)},{\tilde{\theta }}_{(t_{2}2)}\dots ,{\tilde{\theta }}_{(t_{p}p)}.$

In comparison, the MB method uses $t_j=r_{1-\alpha }$ for all j. It can be shown that $t_j={\max }_{b\in \mathsf{\Phi }}r(b,j)\le r_{1-\alpha }.$ This is because for any $b\in \mathsf{\Phi },$$r(b,j)\le r(b)\le r_{1-\alpha }.$ The proposed method using individual rank thresholds instead of the same maximum rank threshold for all coordinates. By this means, the new method sharpens the upper bound and leads to universally shorter or equal SCIs. We summarize the results in the following theorem.

Theorem 1.

Assume there are no ties for the $(1-\alpha )$—percentile of $r(b).$ The proposed modified MB upper intervals have $(1-\alpha )$ coverage and they are admissible at level $(1-\alpha )$ in the sense that there are no other simultaneous confidence intervals which have $(1-\alpha )$ coverage but their lengths (of all the coordinates) are universally shorter than the modified MB intervals.

2.2. Modified MB Two-Sided Simultaneous Confidence Intervals

• Follow the same steps 1 and 2 as in the algorithm for the computation of the MMB upper intervals.
• Calculate the $(1-\alpha /2)$ percentile $r_{1-\alpha /2}$ of $r(b).$
• Denote the collection of bootstrap samples with the maximum sample rank equal or below the $r_{1-\alpha /2}$ as $\mathsf{\Phi }$, i.e., $b\in \mathsf{\Phi }$ if $r(b)\le r_{1-\alpha /2}$.
• For each coordinate j, order the bootstrap estimates in set $\mathsf{\Phi }$. The new rank of ${\tilde{\theta }}_{bj}$ in set $\mathsf{\Phi }$ is denoted as $r^{\prime }(b,j).$ For each sample, find the minimum sample rank $r^{\prime }(b)={\min }_j{r}^{\prime }(b,j)$ which is the smallest rank associated with the bth bootstrap sample in set $\mathsf{\Phi }$.
• Calculate the $\alpha /(2-\alpha )$ percentile the $r_{\alpha /(2-\alpha )}^{\prime }$ of $r^{\prime }(b).$
• Denote the collection of bootstrap samples within $\mathsf{\Phi }$ with the minimum sample rank above $r_{\alpha /(2-\alpha )}^{\prime }$ as $\mathsf{\Psi }=\{b\in \mathsf{\Phi }\mid r^{\prime }(b)\ge r_{\alpha /(2-\alpha )}^{\prime }\}.$
• To construct the upper and lower limits for each coordinate, calculate $t_j={\max }_{b\in \mathsf{\Psi }}r(b,j),$ and $w_j={\min }_{b\in \mathsf{\Psi }}r(b,j).$ The simultaneous confidence interval upper limits are ${\tilde{\theta }}_{(t_{j}j)}$ and the lower limits are ${\tilde{\theta }}_{(w_{j}j)}$, $j=1,\dots ,p$, respectively.

Theorem 2.

Assume there are no ties for the $(1-\alpha /2)$—percentile of $r(b),$ and $\alpha /(2-\alpha )$ percentile of $r^{\prime }(b).$ The proposed modified MB two-sided intervals have $(1-\alpha )$ coverage and they are admissible at level $(1-\alpha )$ in the sense that there are no other two-sided simultaneous confidence intervals which have $(1-\alpha )$ coverage but their interval lengths (for all of the coordinates) are universally smaller than the MBB interval.

Proofs of Theorem 1 and 2 are provided in Appendix A. The theoretical results above assume that there are no ties for the $(1-\alpha /2)$ percentile of $r(b),$ and the $(\alpha /(2-\alpha ))$ percentile of $r^{\prime }(b).$ In practice, however, ties can often occur. Increasing the number of bootstrap samples can decrease the chance of having ties. In the presence of ties in ranking $r(b)$, there could be multiple samples denoted as set $S_1$ with the same maximum rank $r_{1-\alpha /2}.$ We can arbitrarily split them to set $\mathsf{\Phi }$ and set ${\mathsf{\Phi }}^c$ while maintaining that there are $B(1-\alpha /2)$ samples in $\mathsf{\Phi }$. A similar strategy can be applied when there are multiple samples denoted as set $S_2$ with tied minimum sample rank for the $(\alpha /(2-\alpha ))$—percentile of $r^{\prime }(b).$ The resulting SCIs are not unique, but the admissibility result still holds true if the competing set of intervals have the same coverage rate and the covered set of bootstrap samples $\mathsf{\Omega }$ satisfies the condition that $\mathsf{\Omega }\cap {\mathsf{\Psi }}^c⊈(S_1\cup S_2).$ The condition implies $\mathsf{\Omega }$ contains at least a sample $b^{\prime }$ such that $b^{\prime }$ is not in $\mathsf{\Psi }$ and also not in $S_1\cup S_2$. That means either maximum rank $r(b^{\prime })>r_{1-\alpha /2},$ or the minimum rank $r^{\prime }(b^{\prime })<r_{\alpha /2-\alpha }^{\prime }.$ Then, the competing set cannot have intervals universally shorter than the proposed MMB intervals.

We conduct simulation studies to verify the validity of the proposed method. We generate a regression model $y_i=x_i^T\beta +{ϵ}_i,$ where $\beta ={({\beta }_1,\dots ,{\beta }_p)}^T,$ and ${ϵ}_i$ either follows a normal $N(0,1),$ or $lognormal(0,1)$ distribution, $i=1,\dots ,n$. We construct the nonparametric $100(1-\alpha )\%$ SCIs for the unknown regression parameters with varying sample sizes and regression parameters. We compared the performance of the percentile-based methods with the simultaneous confidence intervals based on the multivariate normal (MVN) critical values. We also include the Bonferroni method which uses the critical value of the univariate normal distribution but the significance level is adjusted to $\alpha /p.$ The covariance structure of the multivariate normal is estimated by the sample covariance of the bootstrap estimates. For each simulation setting, we simulate 1000 datasets. For each dataset, we generate 10,000 bootstrap samples. The significance level $\alpha$ is set to $0.05$. It is shown in

Table 1. Simultaneous coverage probability and size of the confidence interval.

			MMB		MB		MVN		Bonferroni
p	n	Error	Cov-Prob	Size	Cov-Prob	Size	Cov-Prob	Size	Cov-Prob	Size
3	200	$N(0,1)$	0.9490	0.5893	0.9490	0.5907	0.9500	0.5880	0.9500	0.5900
3	200	$logN(0,1)$	0.9250	1.5566	0.9260	1.5635	0.9470	1.5513	0.9500	1.5625
15	200	$N(0,1)$	0.9410	1.6827	0.9440	1.6915	0.9410	1.6756	0.9430	1.6827
15	200	$logN(0,1)$	0.9340	4.4606	0.9380	4.5004	0.9660	4.4374	0.9700	4.4821
30	200	$N(0,1)$	0.9440	2.7455	0.9450	2.7757	0.9380	2.7245	0.9420	2.7359
30	200	$logN(0,1)$	0.9570	7.2960	0.9600	7.4004	0.9820	7.2451	0.9820	7.3127

Results are obtained from 1000 simulated datasets, the number of bootstrap samples is 10,000; “cov-prob” denotes the coverage probability; the size of intervals is calculated as $\left|\right|{\widehat{\theta }}_U-{\widehat{\theta }}_L{\left|\right|}_2^2.$ The simultaneous confidence level is 0.95.

that the proposed MMB intervals maintain satisfactory simultaneous coverage rates close to the nominal level for both normal and lognormal errors and different values of p. The MMB method’s coverage probability is very close to that of the MB method, while the interval lengths are uniformly shorter across all the simulation settings. The Bonferroni method is always more conservative than the MVN method. When p increases, for lognormal errors, the MVN method and the Bonferroni method are conservative with familywise type I error rate equal to $0.018$ which is much less than the norminal level of $0.05.$

3. Simultaneous Confidence Interval for Penalized Regression Estimates

In this application, we construct simultaneous confidence intervals using penalized estimates outputted from LASSO generalized linear regression [6]. In order to make post model selection confidence intervals, two strategies were proposed by [7]: bootstrap confidence intervals and smoothed bootstrap confidence interval.

Hereby, different bootstrap samples may lead to different identified submodels $s.$ This implies that for some of the bootstrap samples, ${\theta }_j$ could be thresholded to be zero, whereas in other bootstrap samples, the resulting ${\theta }_j$ could be nonzero. The author of [7] proposed using the bootstrap of the smoothed estimate $\widehat{\theta }={\sum }_{b=1}^B{\tilde{\theta }}_b/B$ ([8,9,10]) to correct the erratic jumpiness of the estimates obtained from different bootstrap samples. He also proposed a method for estimating the asymptotic standard error and to construct a confidence interval for a single parameter using the nonparametric delta method. It is, however, desirable to construct SCIs using the vector of smoothed bootstrap estimates. First, we compute the (asymptotic) covariance matrix of two smoothed bootstrap estimates. This result is established for the ideal bootstrap, where B is equal to all the $n^n$ possible choices of the bootstrap samples. Given the original dataset $Z=(z_1,\dots ,z_n)$, let the bth bootstrap sample be $(z_{b1}^{*},\dots ,z_{bn}^{*})$ and the two estimates be ${\tilde{\theta }}_{bl},$ and ${\tilde{\theta }}_{bm}.$ Let $Z_{bi}^{*}=\#\{k\in \{1,\dots ,n\}\mid z_{bk}^{*}=z_i\},$ which is the number of data points in the bootstrap sample equal to $z_i.$ Then, the nonparametric delta-method-based estimator of the asymptotic covariance matrix for the ideal smoothed bootstrap estimates ${\widehat{\theta }}_l={\sum }_{b=1}^B{\tilde{\theta }}_{bl}/B,$ and ${\widehat{\theta }}_m={\sum }_{b=1}^B{\tilde{\theta }}_{bm}/B,$ is

$$\begin{array}{c}\hfill {\tilde{\sigma }}_{lm}=\sum _{i=1}^n{\mathrm{Cov}}_{*}(Z_{bi}^{*},{\tilde{\theta }}_{bl}){\mathrm{Cov}}_{*}(Z_{bi}^{*},{\tilde{\theta }}_{bm}),\end{array}$$

(2)

where ${\mathrm{Cov}}_{*}(Z_{bi}^{*},{\tilde{\theta }}_{bl})$ and ${\mathrm{Cov}}_{*}(Z_{bi}^{*},{\tilde{\theta }}_{bm})$ denote the bootstrap covariance between $Z_{bi}^{*}$ and ${\tilde{\theta }}_{bl}$, and the bootstrap covariance between $Z_{bi}^{*}$ and ${\tilde{\theta }}_{bm}$, respectively.

The ideal smoothed bootstrap estimates are a functional on the data $(z_1,\dots ,z_n),$ and symmetric in their argument. They can be considered as the functionals of the empirical distribution $\widehat{F}$, which assigns $1/n$ probability to each of the observed data points. It has been shown by [11], that the asymptotic covariance of two functionals of the empirical distribution, can be obtained as $\frac{1}{n^2}{\sum }_{i=1}^n{\widehat{D}}_i^l{\widehat{D}}_i^m$, where ${\widehat{D}}_i^l={\lim }_{ϵ\to 0}\frac{{\widehat{\theta }}_l(p_0+ϵ({\delta }_i-p_0))-{\widehat{\theta }}_l(p_0)}{ϵ},$ ${\widehat{D}}_i^m={\lim }_{ϵ\to 0}\frac{{\widehat{\theta }}_m(p_0+ϵ({\delta }_i-p_0))-{\widehat{\theta }}_m(p_0)}{ϵ},$ ${\delta }_i$ denotes a vector of all zeros except a one for the ith coordinate and $p_0=(1/n,\dots ,1/n).$ From Efron (2014), we obtain ${\widehat{D}}_i^l=n{\sum }_{b=1}^B(Z_{bi}^{*}-1){\tilde{\theta }}_{bl}/B=n{\mathrm{Cov}}_{*}(Z_{bi}^{*},{\tilde{\theta }}_{bl}),$ and ${\widehat{D}}_i^m=n{\sum }_{b=1}^B(Z_{bi}^{*}-1){\tilde{\theta }}_{bm}/B=n{\mathrm{Cov}}_{*}(Z_{bi}^{*},{\tilde{\theta }}_{bm}).$ In practice, the estimator of the covariance of the non-ideal bootstrap estimates can take the form ${\widehat{\sigma }}_{lm}={\sum }_{i=1}^n{\widehat{\mathrm{Cov}}}_{il}{\widehat{\mathrm{Cov}}}_{im},$ and ${\widehat{\mathrm{Cov}}}_{il}=\frac{1}{B}{\sum }_{b=1}^B(Z_{bi}^{*}-Z_{.i}^{*})({\tilde{\theta }}_{bl}-{\widehat{\theta }}_l),$ ${\widehat{\mathrm{Cov}}}_{im}=\frac{1}{B}{\sum }_{b=1}^B(Z_{bi}^{*}-Z_{.i}^{*})({\tilde{\theta }}_{bm}-{\widehat{\theta }}_m),$ respectively. Then, the joint limiting distribution of $({\widehat{\theta }}_1,\dots ,{\widehat{\theta }}_p)$ is multivariate normal with covariance matrix that can be estimated as $\widehat{\mathsf{\Sigma }}=({\widehat{\sigma }}_{lm})$. The numerical evaluation of the multivariate normal distribution is provided with the R package mvtnorm, see [12]. Let $c_{1-\alpha }$ denote the critical value of the multivariate distribution of $\mathsf{\Sigma }$, then the SCIs for all the parameters are given by

$$C{I}_l=\left({\widehat{\theta }}_l-c_{1-\alpha }\sqrt{{\widehat{\sigma }}_{ll}},{\widehat{\theta }}_l+c_{1-\alpha }\sqrt{{\widehat{\sigma }}_{ll}}\right),l=1,\dots ,p.$$

We perform the generalized linear regression with the LASSO penalty on the heart data [13]. The dataset contains the coronary heart disease status of 462 patients and the covariates include sbp (systolic blood pressure), tobacco (cumulative tobacco), ldl (low density lipoprotein cholesterol), adiposity, famhist (family history of heart disease), typea (type-A behavior), obesity, alcohol (current alcohol consumption), and age (age at onset). The optimum penalty size is determined by the Bayesian information criterion. Then, we perform 1000 bootstrap samples and construct the SCIs for the unknown regression parameters. We compare the results of MMB and MB methods with the smoothed bootstrap confidence intervals using the critical values from the multivariate normal distributions and the Bonferroni critical values. The nonparametric delta method is used to derive the limiting covariance structure. It is shown in

Table 2. Comparison of four methods for the construction of simultaneous confidence intervals of LASSO regression estimates for heart data.

	MMB		MB		MVN NP Delta		Bonferroni NP Delta
	Upper	Lower	Upper	Lower	Upper	Lower	Upper	Lower
intercept	0.8804	−6.9971	0.8999	−7.4115	−0.3777	−6.0964	−0.3514	−6.1227
sbp	0.0156	−0.0243	0.0156	−0.0292	0.0092	−0.0097	0.0093	−0.0098
tobacco	0.2374	0.0000	0.2374	0.0000	0.1921	−0.0480	0.1932	−0.0491
ldl	0.4703	0.0000	0.4703	0.0000	0.3693	−0.1772	0.3718	−0.1797
adiposity	0.1543	0.0000	0.1846	−0.0408	0.0595	−0.0430	0.0599	−0.0434
famhist	1.3744	0.0000	1.3994	0.0000	1.0511	−0.5562	1.0585	−0.5636
typea	0.0929	0.0000	0.0929	−0.0111	0.0453	−0.0278	0.0456	−0.0281
obesity	0.0000	−0.2857	0.0000	−0.3246	0.0873	−0.1324	0.0884	−0.1334
alcohol	0.0272	−0.0073	0.0272	−0.0172	0.0154	−0.0103	0.0155	−0.0104
age	0.0920	0.0000	0.0920	0.0000	0.0853	−0.0092	0.0857	−0.0096

The bootstrap number is 1000. Upper and lower stands for the upper and lower limits of the SCI. The simultaneous confidence level is 0.95. Both MVN and Bonferroni methods are implemented using nonparametric Delta method to obtain the covariance estimates for the smoothed bootstrap estimates.

that all four methods provide similar lower and upper bounds. The MVN and Bonferroni methods generate confidence intervals all containing zeros. In comparison, most of the confidence intervals generated by MMB and MB methods have upper bounds or lower bounds at zero value. Figure 1 depicts all the confidence intervals generated by all four methods. Each confidence interval is scaled by the standard deviation of the point estimate so that all intervals can be presented on the same scale. The MMB and MB methods have mostly asymmetric confidence intervals which are either non-negative or non-positive. This asymmetric structure not only reflects the sparse nature of the penalized estimates, but also provides evidence on the direction of the parameter. Compared to the MB method, the proposed MMB method produces shorter intervals.

4. Discussion

We propose nonparametric simultaneous confidence intervals to provide distribution-free joint inferences for multiple parameters. Compared to existing MB intervals, the proposed modified MB intervals have uniformly sharper upper and lower bounds. When used in post model selection inference, the MMB intervals are advantageous as they preserve the asymmetry of the sampling distributions of the penalized estimates.

When the number of parameters increases with the sample size, it is difficult to develop simultaneous confidence intervals with desired coverage probability for all the parameters. There are a number of challenges for parametric or nonparametric methods: (1) To obtain the empirical multivariate distribution of ${\widehat{\theta }}_1,\dots ,{\widehat{\theta }}_P$, it requires much larger bootstrap samples. (2) The correction for the adjustment of multiplicities is more severe. (3) The quantile for the high-dimensional multivariate distribution is harder to compute. To overcome this issue, we may adopt the approach of false-discovery-rate adjusted multiple confidence intervals [14]. This warrants future investigations to develop nonparametric simultaneous intervals for high dimensional parameters.