From p-Values to Posterior Probabilities of Null Hypotheses

Abstract

Minimum Bayes factors are commonly used to transform two-sided p-values to lower bounds on the posterior probability of the null hypothesis, in particular the bound $-e·p·log\left(p\right)$. This bound is easy to compute and explain; however, it does not behave as a Bayes factor. For example, it does not change with the sample size. This is a very serious defect, particularly for moderate to large sample sizes, which is precisely the situation in which p-values are the most problematic. In this article, we propose adjusting this minimum Bayes factor with the information to approximate an exact Bayes factor, not only when p is a p-value but also when p is a pseudo-p-value. Additionally, we develop a version of the adjustment for linear models using the recent refinement of the Prior-Based BIC.

1. Introduction

By now, it is well known by practitioners that p-values are not posterior probabilities of a null hypothesis, which is what science would need to declare a scientific finding. So p-values, and particularly the threshold of $0.05$, need to be recalibrated. Two widespread practical attempts are (i) the so-called Robust Lower Bound on Bayes factors $BF\ge -e·p·log\left(p\right)$ [1] and (ii) the replacement of the ubiquitous $\alpha =0.05$ by ${\alpha }^{*}=0.005$ [2]. These suggestions, which are an improvement of usual practice, fall short of being a real solution, mainly because the dependence of the evidence on the sample size is not considered. Still, the Robust Lower Bound is useful since it is valid from small sample sizes and onward and only depends on the p-value. It is known that the evidence of a p-value against a point null hypothesis depends on the sample size. In [3], they consider p-values in linear models and propose new monotonic minimum Bayes factors that depend on the sample size and converge to $-e·p·log\left(p\right)$ as the sample size approaches infinity, which implies it is not consistent, as Bayes factors are. It turns out that the maximum evidence for an exact two-tailed p-value increases with decreasing sample size. There are several proposals in the literature, and most do not depend on the sample size, while those that do continue to be Robust Lower Bounds; however, neither behaves like a real Bayes factor. In this article, we propose to adjust the Robust Lower Bound $-e·p·log\left(p\right)$ so that it behaves in a similar or approximate way to actual Bayes factors for any sample size. A further complication arises, however, when the null hypotheses are not simple, that is, when they depend on unknown nuisance parameters. In this situation, what is usually called p-values are only pseudo-p-values [4] (p. 397). So, we first need to extend the validity of the Robust Lower Bound to pseudo-p-values. The effect of adjusting this minimum Bayes factor with the sample size is shown in a simulation in Section 5.1.

The outline of the article is as follows: In Section 2 we define pseudo-p-values using the p-value definition of [4] (p. 397) and extend for them the validity of the Robust Lower Bound. In Section 3, we present the adaptive significance levels that will be used for incorporating the sample size in the lower bound: the general adaptive significance level presented in [5] and the refined version for linear models developed in [6]; in both cases, we use versions calibrated using the Prior-Based BIC (PBIC) [7]. In Section 4, we derive adaptive approximate Bayes factors and apply them to pseudo-p-values in Section 5. We close in Section 6 with some final comments.

2. Valid p-Values and Robust Lower Bound

Under the null hypotheses, p-values are well known to have Uniform(0, 1); in [4] (p. 397), a more general definition is given.

Definition 1. 

A p-value $p\left(\mathit{X}\right)$ is a statistic satisfying $0\le p\left(\mathit{x}\right)\le 1$ for every sample point x. Small values of $p\left(\mathit{X}\right)$ give evidence that $H_1:\theta \in {\Theta }_0^c$ is true, where ${\Theta }_0$ is some subset of the parameter space and ${\Theta }_0^c$ is its complement. A p-value is valid if, for every $\theta \in {\Theta }_0$ and every $0\le \alpha \le 1$,

$$P_{\theta }(p\left(\mathit{X}\right)\le \alpha )\le \alpha .$$

Based on this definition, we can say that there are valid p-values that are Uniformly Distributed in (0, 1), that is,

$$P_{\theta }(p\left(\mathbf{X}\right)\le \alpha )=\alpha \mathrm{for}\mathrm{every}\theta \in {\Theta }_0\mathrm{and}\mathrm{every}0\le \alpha \le 1,$$

(1)

and others that are not, that is, when there is at least one $\alpha$, such that

$$P_{\theta }(p\left(\mathbf{X}\right)\le \alpha )<\alpha \mathrm{for}\mathrm{every}\theta \in {\Theta }_0.$$

(2)

Remark 1. 

We consider any valid p-value complying with (2) a pseudo-p-value.

The “Robust Lower Bound” ($RLB$), as we call it here and proposed by [1], is

$$B_L\left(p\right)=\left\{\begin{array}{cc}-e·p·log\left(p\right)\hfill & p<e^{-1}\hfill \\ 1\hfill & \mathrm{otherwise}\hfill \end{array}\right.$$

The authors consider that under the null hypothesis, the distribution of the p-value, $p\left(\mathbf{X}\right)$, is Uniform(0, 1). Alternatives are typically developed by considering alternative models for $\mathbf{X}$, but the results then end up being quite problem-specific. An attractive approach is instead to directly consider alternative distributions for p itself. In effect, they consider that, under $H_1$, the density of p is $f\left(p\right|\xi )$, where $\xi$ is an unknown parameter. So, consider testing

$$H_0:p\sim \mathrm{Uniform}(0,1)\mathrm{versus}{H}_1:p\sim f\left(p\right|\xi )$$

If the test statistic (T) has been appropriately chosen so that large values of $T\left(\mathbf{X}\right)$ would be evidence in favor of $H_1$, then the density of p under $H_1$ should be decreasing in p. A class of decreasing densities for p that is very easy to work with is the class of $\mathrm{Beta}(\xi ,1)$ densities, for $0<\xi \le 1$, given by $f\left(p\right|\xi )=\xi p^{\xi -1}$. The uniform distribution (i.e., $H_0$) arises from the choice $\xi =1$ [1]. The expression $B_L\left(p\right)={inf}_{all\pi }{B}_{\pi }\left(p\right)$, where $B_{\pi }\left(p\right)$ is the Bayes factor of $H_0$ to $H_1$ for a given prior density $\pi \left(\xi \right)$ on this alternative.

Note that this calibration has already been proposed in [8]. Another class of decreasing densities is $\mathrm{Beta}(1,\xi )$ with $\xi >1$. This leads to the “$-e·q·log\left(q\right)$” calibration, where $q=1-p$ see [9].

In contrast with Remark 1, if we consider $p\left(\mathbf{X}\right)$ a pseudo-p-value under $H_0$, that is,

$$p\sim \mathrm{Beta}({\xi }_0,1)\mathrm{with}{\xi }_0>1,\mathrm{fixed}\mathrm{but}\mathrm{arbitrary},$$

under the test

$$H_0:p\sim \mathrm{Beta}({\xi }_0,1)\mathrm{vs}.H_1:p\sim f\left(p\right|\xi )$$

with $f\left(p\right|\xi )\sim \mathrm{Beta}(\xi ,1)$ for $0<\xi \le {\xi }_0$, then a generalized Robust Lower Bound $RL{B}_{{\xi }_0}$ can be defined as

$$B_L(p,{\xi }_0)=\left\{\begin{array}{cc}-e·{\xi }_0·p^{{\xi }_0}log\left(p\right)\hfill & p<e^{-\frac{1}{{\xi }_0}}\hfill \\ 1\hfill & \mathrm{otherwise}\hfill \end{array}\right.$$

(3)

where ${\xi }_0$ has to be estimated or calculated theoretically (see [10] for a proposal when extending for multiple testing). Any value ${\xi }_0\ne 1$ corresponds to a pseudo-p-value.

On the other hand, since $f\left(p\right|\xi )=\xi p^{\xi -1}$ has its maximum in $\xi =-\frac{1}{log\left(p\right)}<1$ with $p<e^{-1}$, then $f\left(p\right|\xi )$ is decreasing for $\xi >-\frac{1}{log\left(p\right)}$, thus for any Bayes factor $B_{01}$

$$B_{01}\ge B_L\left(p\right)>B_L(p,{\xi }_0)\mathrm{with}{\xi }_0>1$$

(4)

See Figure 1.

In the following, we calibrate $RL{B}_{{\xi }_0}$ such that $RL{B}_{{\xi }_0}\approx B_{01}$.

Lemma 1. 

$B_L(p_{val},\xi )=-e·\xi ·p_{val}^{\xi }·log\left(p_{val}\right)\ge e·\xi ·p_{val}^{\xi }>p_{val}^{\xi },$   for, $0<p_{val}<e^{-1}$ and $\xi \ge 1$. Note that $B_L(p_{val},1)=B_L\left(p_{val}\right)$

Proof. 

Appendix A.    □

Theorem 1. 

The ${\mathit{RLB}}_{\xi }$ is a valid p-value for $\xi \ge 1$, that is,

$$P(B_L(p,\xi )\le \alpha |p\sim f\left(p\right|\xi ))\le \alpha ,\mathit{for}\mathit{each}0\le \alpha \le 1.$$

Proof. 

Appendix A.    □

3. Adaptive $\mathit{\alpha }$ with PBIC Strategy

The Bayesian literature has been criticizing for several decades the implementation of hypothesis testing with fixed significance levels and, in particular, the use of the scale p-value < 0.05. An adaptive $\alpha$ allows us to adjust the statistical significance with the amount of information; see [5,11,12]. The adaptive values we work with in this section were calculated so that they allow to arrive to results equivalent to those obtained with a Bayes factor. In [5], the authors present an adaptive $\alpha$ based on BIC as

$${\alpha }_n\left(q\right)=\frac{{[{\chi }_{\alpha }^2\left(q\right)+qlog\left(n\right)]}^{\frac{q}{2}-1}}{2^{\frac{q}{2}-1}{n}^{\frac{q}{2}}\Gamma \left(\frac{q}{2}\right)}\times C_{\alpha },$$

(5)

where $C_{\alpha }$ is a calibration constant, and strategies for calculating it are presented in [5]. It yields a consistent procedure; it alleviates the problem of the divergence between practical and statistical significance; and it makes it possible to perform Bayesian testing by computing intervals with the calibrated $\alpha$-levels.

An adaptive $\alpha$ is also presented in [6], but this time it is a version refined to nested linear models with calibration based on the Bayesian information criterion based on Prior PBIC [7],

$${\alpha }_{(b,n)}\left(q\right)={\displaystyle \frac{{[g_{n,\alpha }\left(q\right)+log\left(b\right)+C]}^{\frac{q}{2}-1}}{b^{\frac{n-j}{2(n-1)}}·{\left(\frac{2(n-1)}{n-j}\right)}^{q/2-1}\Gamma \left(\frac{q}{2}\right)}}\times exp\left\{-\frac{n-j}{2(n-1)}\left(g_{n,\alpha }\left(q\right)+C\right)\right\}.$$

(6)

Here, $b=\frac{|{\mathbf{X}}_j^t{\mathbf{X}}_j|}{|{\mathbf{X}}_i^t{\mathbf{X}}_i|}$ and ${\mathbf{X}}_i,{\mathbf{X}}_j$ are design matrices and

$$C=2\sum _{m_i=1}^{q_i}log\frac{(1-e^{-v_{m_i}})}{\sqrt{2}{v}_{m_i}}-2\sum _{m_j=1}^{q_j}log\frac{(1-e^{-v_{m_j}})}{\sqrt{2}{v}_{m_j}},$$

$v_{m_l}=\frac{{\widehat{\xi }}_{m_l}}{\left[d_{m_l}(1+n_{m_l}^e)\right]}$ with $l=i,j$ corresponding to each model. Here, $n_{m_l}^e$, with $l=i,j$, refers to The Effective Sample Size (called TESS) corresponding to that parameter; see [7].

The adaptive $\alpha$ in (5) can also be presented using the PBIC strategy (this strategy was not considered in [5]), and the following expression is obtained

$${\alpha }_n\left(q\right)={\displaystyle \frac{{[{\chi }_{\alpha }^2\left(q\right)+qlog\left(n\right)+C]}^{\frac{q}{2}-1}}{n^{\frac{q}{2}}{2}^{\frac{q}{2}-1}\Gamma \left(\frac{q}{2}\right)}}\times exp\left\{-\frac{1}{2}\left({\chi }_{\alpha }^2\left(q\right)+C\right)\right\}.$$

(7)

Note that this adaptive $\alpha$ is still of BIC structure, since the expression ${\chi }_{\alpha }^2\left(q\right)+qlog\left(n\right)$ remains.

Example: Binomial Models

Consider comparing two binomial models $S_1\sim \mathrm{binomial}(n_1,p_1)$ and $S_2\sim$ $\mathrm{binomial}(n_2,p_2)$ via the test

$$H_0:p_1=p_2\mathrm{vs}.H_1:p_1\ne p_2.$$

Defining $n=n_1+n_2$ and $\widehat{p}$, the MLE from $p_1-p_2$, then (7) gives

$${\alpha }_n={\left[{\displaystyle \frac{2}{n\pi ({\chi }_{\alpha }^2\left(1\right)+log\left(n\right)+C)}}\right]}^{1/2}\times exp\left\{-\frac{1}{2}\left({\chi }_{\alpha }^2\left(1\right)+C\right)\right\},$$

(8)

here, ${\chi }_{\alpha }^2\left(1\right)$ is the quantile $\alpha$ from chi-square with $df=1$, $C=-2log{\displaystyle \frac{(1-e^{-v})}{\sqrt{2}v}}$, $v={\widehat{p}}^2/\left[d(1+n^e)\right]$, $d=\left(\frac{{\sigma }_1^2}{n_1}+\frac{{\sigma }_2^2}{n_2}\right),n^e=max\left\{\frac{n_1^2}{{\sigma }_1^2},\frac{n_2^2}{{\sigma }_2^2}\right\}d$.

Table 1. Adaptive $\alpha$ via PBIC in (8) for testing equality of two proportions for different sample sizes when $\alpha =0.05$.

		Adaptive $\mathit{\alpha }$ via PBIC (${\mathit{\alpha }}_{\mathit{n}}$)
${\mathit{n}}_{\mathbf{1}}$	${\mathit{n}}_{\mathbf{2}}$	$\mathit{n}={\mathit{n}}_{\mathbf{1}}+{\mathit{n}}_{\mathbf{2}}$
10	10	0.0068
25	25	0.0040
50	50	0.0027
100	50	0.0021
50	100	0.0021
100	100	0.0018

shows the behavior of this adaptive ${\alpha }_n$ for $\alpha =0.05$ and different values of $n_1$ and $n_2$.

4. Adjusting ${\mathit{RLB}}_{\mathit{\xi }}$ Using Adaptive $\mathit{\alpha }$

In this section, we combine (3) with the formulas for adaptive $\alpha$ in (6) and (7) for adjusting $RL{B}_{\xi }$ and obtaining an approximation to an objective Bayes factor. Indeed, we adjust the $RL{B}_{\xi }$ through the expression $B\left(\alpha \right)=B_L(\alpha ,{\xi }_0)·g(·)$, where g is determined in such a way that when $B\left(\alpha \right)$ is evaluated in (6) or (7), it converges to a constant (this allows us to obtain equivalent results from the Frequentist and Bayesian point of view, that is, the decision does not change).

Substituting p in (3) by the adaptive $\alpha$ value in (7) results in the following expression.

$$B(\alpha ,q,n,{\xi }_0)=-{\alpha }^{{\xi }_0}log\left(\alpha \right)\Gamma {(q/2)}^{{\xi }_0}{n}^{\frac{{\xi }_{0}q}{2}}{\left[{\displaystyle \frac{2}{{\chi }_{\alpha }^2\left(q\right)+q·log\left(n\right)+C}}\right]}^{\frac{{\xi }_{0}q}{2}-({\xi }_0-1)}.$$

(9)

For a $\mathrm{Uniform}(0,1)$ p-value with ${\xi }_0=1$, this expression simplifies to

$$B(\alpha ,q,n)=-\alpha log\left(\alpha \right)\Gamma (q/2)n^{\frac{q}{2}}{\left[{\displaystyle \frac{2}{{\chi }_{\alpha }^2\left(q\right)+q·log\left(n\right)+C}}\right]}^{\frac{q}{2}}.$$

(10)

The refined version of this calibration for linear models is obtained when (3) is evaluated in (6)

$$B(\alpha ,q,n,b)=-\alpha log\left(\alpha \right)\Gamma (q/2)b^{\frac{n-j}{2(n-1)}}{\left[{\displaystyle \frac{2(n-1)}{(g_{n,\alpha }\left(q\right)+log\left(b\right)+C)(n-j)}}\right]}^{\frac{q}{2}}$$

(11)

in this case, we only consider ${\xi }_0=1$.

Balanced One-Way Anova

Suppose we have k groups with r observations each, for a total sample size of $kr$, and let $H_0:{\mu }_1=\cdots ={\mu }_k=\mu \mathrm{vs}.H_1:\mathrm{At}\mathrm{least}\mathrm{one} {\mu }_i \mathrm{different}$. Then, the design matrices for both models are

$${\mathbf{X}}_1=\left(\begin{array}{c}1\\ 1\\ ⋮\\ 1\end{array}\right),{\mathbf{X}}_k=\left(\begin{array}{cccc}1& 0& \dots & 0\\ 1& 0& \dots & 0\\ ⋮& ⋮& \dots & ⋮\\ 1& 0& \dots & 0\\ 0& 1& \dots & 0\\ 0& 1& \dots & 0\\ ⋮& ⋮& \dots & ⋮\\ 0& 1& \dots & 0\\ ⋮& ⋮& \dots & ⋮\\ 0& 0& \dots & 1\\ 0& 0& \dots & 1\\ ⋮& ⋮& \dots & ⋮\\ 0& 0& \dots & 1\end{array}\right),b=\frac{|{\mathbf{X}}_k^t{\mathbf{X}}_k|}{|{\mathbf{X}}_1^t{\mathbf{X}}_1|}=k^{-1}{r}^{k-1},$$

and the adaptive $\alpha$ for the linear model in accordance with what was presented in [6] is

$$\alpha (k,r)={\displaystyle \frac{{[g_{r,\alpha }(k-1)-log\left(k\right)+(k-1)log\left(r\right)+C]}^{\frac{k-3}{2}}}{{\left(k^{-1}{r}^{k-1}\right)}^{\frac{r-1}{2(r-1/k)}}{\left(\frac{2(r-1/k)}{r-1}\right)}^{\frac{k-3}{2}}\Gamma \left(\frac{k-1}{2}\right)}}\times exp\left\{-\frac{r-1}{2(r-1/k)}\left(g_{r,\alpha }(k-1)+C\right)\right\}.$$

Here, the number of replicas r is The Effective Sample Size (TESS). Therefore, the approximate Bayes factor for this test calculated with (8) is

$$B(\alpha ,k,r)=-\alpha log\left(\alpha \right)\Gamma ((k-1)/2){\left(k^{-1}{r}^{k-1}\right)}^{\frac{r-1}{2(r-1/k)}}{\left[{\displaystyle \frac{2(r-1/k)}{(g_{r,\alpha }(k-1)-log\left(k\right)+(k-1)log\left(r\right)+C)(r-1)}}\right]}^{\frac{k-1}{2}}$$

A very important case arises when $k=2$. For this situation, the last formula simplifies to

$$B(\alpha ,r)=-\alpha log\left(\alpha \right){\left({\displaystyle \frac{r}{2}}\right)}^{\frac{r-1}{2r-1}}{\left[{\displaystyle \frac{2(r-1)\pi }{(g_{r,\alpha }\left(1\right)-log\left({\displaystyle \frac{r}{2}}\right)+C)(r-1)}}\right]}^{\frac{1}{2}}$$

(12)

5. Obtaining Bounds for $\mathit{P}\left({\mathit{H}}_{\mathbf{0}}\right|\mathit{Data})$

In this section, we use (9) and (11) to produce bounds for the posterior probability of the null hypothesis $H_0$.

Since for any Bayes factor $B_{01}$

$$B_{01}\ge B_L(p,{\xi }_0)\mathrm{with}{\xi }_0\ge 1,\mathrm{fixed}\mathrm{but}\mathrm{arbitrary},$$

a lower bound for the posterior probability of the null hypothesis can be obtained as

$$minP\left(H_0\right|Data)={\left[1+{\displaystyle \frac{1}{B_L(p,{\xi }_0)}}\right]}^{-1}.$$

(13)

Figure 2 shows these posterior probabilities (called $P_{RL{B}_{{\xi }_0}}$) for different values of ${\xi }_0$. To simplify the use of these Bayes factors, we call $BF{G}_{{\xi }_0}$ the Bayes factor of Equation (9), $BFG$ the Bayes factor of Equation (10), and $BFL$ the Bayes factor of Equation (11).

5.1. Testing Equality of Two Means

Consider comparing two normal means via the test

$$H_0:{\mu }_1={\mu }_2\mathrm{versus}{H}_1:{\mu }_1\ne {\mu }_2,$$

where the associated known variances, ${\sigma }_1^2$ and ${\sigma }_2^2$, are not equal.

$$\mathbf{Y}=\mathbf{X}\mu +ϵ=\left(\begin{array}{cc}1& 0\\ ⋮& ⋮\\ 1& 0\\ 0& 1\\ ⋮& ⋮\\ 0& 1\end{array}\right)\left(\begin{array}{c}{\mu }_1\\ {\mu }_2\end{array}\right)+\left(\begin{array}{c}{ϵ}_{11}\\ ⋮\\ {ϵ}_{2n_2}\end{array}\right),$$

$$\times ϵ\sim N(\mathbf{0},\mathrm{diag}\{\underset{n_1}{\underbrace{{\sigma }_1^2,\cdots ,{\sigma }_1^2}},\underset{n_2}{\underbrace{{\sigma }_2^2,\cdots ,{\sigma }_2^2}}\})$$

Defining $\nu =({\mu }_1+{\mu }_2)/2$ and $\zeta =({\mu }_1-{\mu }_2)/2$ places this in the linear model comparison framework,

$$\mathbf{Y}=\mathbf{B}\left(\genfrac{}{}{0pt}{}{\nu }{\zeta }\right)+ϵ$$

with

$$\mathbf{B}=\left(\begin{array}{cc}1& 1\\ ⋮& ⋮\\ 1& 1\\ 1& -1\\ ⋮& ⋮\\ 1& -1\end{array}\right)$$

where we are comparing $M_0:\zeta =0$ versus $M_1:\zeta \ne 0$.

So, for $BFG$ and $BFL$,

$$C=-2log\frac{(1-e^{-v})}{\sqrt{2}v}$$

$$v=\frac{{\widehat{\zeta }}^2}{d(1+n^e)},d=\left(\frac{{\sigma }_1^2}{n_1}+\frac{{\sigma }_2^2}{n_2}\right),n^e=max\left\{\frac{n_1^2}{{\sigma }_1^2},\frac{n_2^2}{{\sigma }_2^2}\right\}\left(\frac{{\sigma }_1^2}{n_1}+\frac{{\sigma }_2^2}{n_2}\right).$$

A special case is the standard test of equality of means when ${\sigma }_1^2={\sigma }_2^2={\sigma }^2$. Then,

$$n^e=min\left\{n_1\left(1+\frac{n_1}{n_2}\right),n_2\left(1+\frac{n_2}{n_1}\right)\right\}.$$

On the other hand, considering $\mu ={\mu }_1-{\mu }_2$ with ${\sigma }_1^2={\sigma }_2^2={\sigma }^2$:

• $H_0:{\mu }_1={\mu }_2⟷\mu =0$;
• $H_1:{\mu }_1\ne {\mu }_2⟷\mu \ne 0$.

Assuming priors:

• $\mu |{\sigma }^2,H_1\sim Normal(0,{\sigma }^2/{\tau }_0),{\tau }_0\in (0,\infty )$;
• $\pi \left({\sigma }^2\right)\propto 1/{\sigma }^2$ for both $H_0$ and $H_1$.

The Bayes factor is

$$B{F}_{01}={\left({\displaystyle \frac{n+{\tau }_0}{{\tau }_0}}\right)}^{1/2}{\left({\displaystyle \frac{t^2\frac{{\tau }_0}{n+{\tau }_0}+l}{t^2+l}}\right)}^{\frac{l+1}{2}}$$

(14)

where

$$t=\frac{|\overline{\mathbf{Y}}|}{s/\sqrt{n}}$$

a t-statistic with degrees of freedom $l=n-1$ and $n=n_1+n_2$; see [13].

Figure 3 shows the posterior probability for the null hypothesis $H_0$ when $n=50$ and $n=100$ for the Robust Lower Bound with ${\xi }_0=1$ (called $P_{RLB}$), the Bayes factor $BFL$ (called $P_{BFL}$), the Bayes factor $BFG$ (called $P_{BFG}$), and the Bayes factor $B{F}_{01}$ (called $P_{B{F}_{01}}$). Note that the posterior probability with $B{F}_{01}$ when ${\tau }_0=6$ looks very similar to the result obtained using the Bayes factors $BFL$ and $BFG$.

We now present a simulation that shows how our adjustment, or calibration, to $RL{B}_{\xi }$ works quite similarly to an exact Bayes factor. We perform the following experiment: We simulate r data points from each of the two normal distributions, $N({\mu }_1,\sigma )$ and $N({\mu }_2,\sigma )$. We reproduce this K times. For all K simulations, ${\mu }_1-{\mu }_2=0$. For all K replicates, we test the hypotheses $H_0:{\mu }_1={\mu }_2$ vs. $H_1:{\mu }_1\ne {\mu }_2$, and then we count how many of the p-values lie between $0.05-\epsilon$ and $0.05$. Note that all of these p-values would be considered sufficient to reject $H_0$ if $\alpha =0.05$ is selected. Finally, we determine the proportion of these “significant” p-values obtained from samples where $H_0$ is true.

Table 2. Mean percentage of p-values less than $0.05$ (considered significant) coming from data generated under the null hypothesis for 100 experiments, where $K=8000$ testing problems are generated under $H_0:{\mu }_1={\mu }_2$. This experiment is performed for different groups with sample sizes r. Corrected and uncorrected Bayes factors are considered, as well as an exact Bayes factor.

		% Of Samples with $\mathit{P}\left({\mathit{H}}_0\right|\mathit{x})\ge \mathbf{0.5}$
r	% Of Samples with $\mathit{p}<\mathbf{0.05}$	${\mathit{RLB}}_{\mathbf{\xi }}$	$\mathit{BFG}$	$\mathit{BFL}$	${\mathit{BF}}_{\mathbf{01}}$
10	$5\%$	$0\%$	$58\%$	$66\%$	$75\%$
50	$5\%$	$0\%$	$81\%$	$86\%$	$87\%$
100	$5\%$	$0\%$	$86\%$	$89\%$	$91\%$
500	$5\%$	$0\%$	$94\%$	$96\%$	$96\%$
1000	$5\%$	$0\%$	$95\%$	$96\%$	$97\%$

presents the mean percentage of these significant p-values coming from samples, where $H_0$ is true for 100 iterations of the simulation scheme with $K=8000$, $\sigma =1$, and $\epsilon =0.05$ for $r=10,50,100,500$, and 1000. As expected, the distribution of the p-values behaved $\mathrm{Uniform}(0,1)$ under $H_0$, since $H_0$ was assumed true in the K replicates. Table 2 also presents the proportion of posterior probability of $H_0$ greater than or equal to $0.5$ (50%) when using the $RL{B}_{\xi }$, when corrected according to the method suggested in this document (Equations (10) and (11)), and when an exact Bayes factor (Equation (14)) is used. It is clear that the method suggested here behaves very similarly to an exact Bayes factor.

5.2. Fisher’s Exact Test

This is an example where the p-value is a pseudo-p-value (see the example 8.3.30 in [4]). Let $S_1$ and $S_2$ be independent observations with $S_1\sim \mathrm{binomial}(n_1,p_1)$ and $S_2\sim \mathrm{binomial}(n_2,p_2)$. Consider testing $H_0:p_1=p_2$ vs. $H_1:p_1\ne p_2$.

Under $H_0$, if we let p be the common value of $p_1=p_2$, the joint pmf of $(S_1,S_2)$ is

$$f(s_1,s_2|p)=\left(\genfrac{}{}{0pt}{}{n_1}{s_1}\right)\left(\genfrac{}{}{0pt}{}{n_2}{s_2}\right)p^{s_1+s_2}{(1-p)}^{n_1+n_2-(s_1+s_2)}$$

and the conditional pseudo-p-value is

$$p(s_1,s_2)=\sum _{j=s_1}^{min\{n_1,s\}}f\left(j\right|s),$$

(15)

the sum of hypergeometric probabilities, with $s=s_1+s_2$.

Remark 2. 

It does not seem to be simple to estimate the appropriate ${\xi }_0$ that best fits the pseudo-p-value in (15), in Figure 4 some arbitrary possibilities are given.

It is important to note that in Bayesian tests with a point null hypothesis, it is not possible to use continuous prior densities, because these distributions (as well as posterior distributions) will grant zero probability to $p=(p_1=p_2)$. A reasonable approximation will be to give $p=(p_1=p_2)$, a positive probability ${\pi }_0$, and to $p\ne (p_1=p_2)$ the prior distribution ${\pi }_1{g}_1\left(p\right)$, where ${\pi }_1=1-{\pi }_0$ and $g_1$ proper. One can think of ${\pi }_0$ as the mass that would be assigned to the real null hypothesis, $H_0:p\in ((p_1=p_2)-b,(p_1=p_2)+b)$ if it had not been preferred to approximate by the null point hypothesis. Therefore, if

$$\pi \left(p\right)=\left\{\begin{array}{cc}{\pi }_0\hfill & p=(p_1=p_2)\hfill \\ {\pi }_1{g}_1\left(p\right)\hfill & p\ne (p_1=p_2)\hfill \end{array}\right.$$

then

$$\begin{array}{ccc}\hfill m\left(s\right)& =& {\int }_{\Theta }f\left(s\right|p)\pi \left(p\right)dp\hfill \\ & =& f\left(s\right|(p_1=p_2)){\pi }_0+{\pi }_1{\int }_{p\ne (p_1=p_2)}f\left(s\right|p)g_1\left(p\right)dp\hfill \\ & =& f\left(s\right|(p_1=p_2)){\pi }_0+(1-{\pi }_0)m_1\left(s\right)\hfill \end{array}$$

where ${\displaystyle m_1\left(s\right)={\int }_{p\ne (p_1=p_2)}f\left(s\right|p)g_1\left(p\right)dp}$ is the marginal density of $(S=S_1+S_2)$ with respect to $g_1$.

So,

$$\pi \left((p_1=p_2)\right|s)=\frac{{\pi }_{0}f\left(s\right|(p_1=p_2))}{m\left(s\right)}$$

thus

$$\begin{array}{ccc}\hfill \mathrm{posterior}\mathrm{odds}& =& {\displaystyle \frac{\pi \left((p_1=p_2)\right|s)}{1-\pi \left((p_1=p_2)\right|s)}}\hfill \\ & =& {\displaystyle \frac{f\left(s\right|(p_1=p_2)){\pi }_0}{m\left(s\right)(1-{\displaystyle \frac{f\left(s\right|(p_1=p_2)){\pi }_0}{m\left(s\right)}})}}\hfill \\ & =& {\displaystyle \frac{f\left(s\right|(p_1=p_2)){\pi }_0}{m\left(s\right)-f\left(s\right|(p_1=p_2)){\pi }_0}}\hfill \\ & =& {\displaystyle \frac{f\left(s\right|(p_1=p_2)){\pi }_0}{(1-{\pi }_0)m_1\left(s\right)}}\hfill \\ & =& {\displaystyle \frac{{\pi }_{0}f\left(s\right|(p_1=p_2))}{{\pi }_1{m}_1\left(s\right)}}\hfill \\ & =& \mathrm{prior}\mathrm{odds}·{\displaystyle \frac{f\left(s\right|(p_1=p_2))}{m_1\left(s\right)}}\hfill \end{array}$$

and the Bayes factor is

$$B_{01}={\displaystyle \frac{f\left(s\right|(p_1=p_2))}{m_1\left(s\right)}}.$$

Now, if we take $g_1\left(p\right)=\mathrm{Beta}(a,b)$ such that $E\left(p\right)={\displaystyle \frac{a}{a+b}}=(p_1=p_2)$, then

$$B{F}_{Test}={\displaystyle \frac{B(a,b)}{B(s+a,n_1+n_2-s+b)}}{p}^s{(1-p)}^{n_1+n_2-s}.$$

Figure 4 shows the posterior probability for the null hypothesis $H_0$ when $n=n_1+n_2=50$ and 100, for the Robust Lower Bound, the Bayes factor $BF{G}_{{\xi }_0}$ (called $P_{BF{G}_{{\xi }_0}}$), the Bayes factor $BFG$ (called $P_{BFG}$), and the Bayes factor $B{F}_{Test}$ (called $P_{B{F}_{Test}}$). We can note that all the $P_{BF{G}_{{\xi }_0}}$ are comparable, even though in the case ${\xi }_0=1$ ($P_{BFG}$) it is a p-value and not a pseudo-p-value.

5.3. Linear Regression Models

Consider comparing two nested linear models $M_3:y_l={\lambda }_1+{\lambda }_2{x}_{l2}+{\lambda }_3{x}_{l3}+{ϵ}_l$ with $M_2:y_l={\lambda }_1+{\lambda }_2{x}_{l2}+{ϵ}_l$ via the test

$$H_0:M_2\mathrm{versus}{H}_1:M_3,$$

with $1\le l\le n$, and the errors ${ϵ}_l$ are assumed to be independent and normally distributed with unknown residual variance ${\sigma }^2$. According to the Equation (3) in [6,7]

$$b=(n-1)s_3^2(1-{\rho }_{23}^2),$$

where $s_3^2$ is the variance $x_{v3}$, ${\rho }_{23}$ is the correlation between $x_{v2}$ and $x_{v3}$, and

$$C=2log\frac{(1-e^{-v_2})}{\sqrt{2}{v}_2}-2log\frac{(1-e^{-v_3})}{\sqrt{2}{v}_3},$$

where $v_2={\widehat{\lambda }}_2^2/\left[d_2(1+n_2^e)\right]$, $d_2={\sigma }^2/s_{x_{l2}}^2$, $n_2^e=s_{x_{l2}}^2/{max}_i\left\{{(x_{i2}-{\overline{x}}_2)}^2\right\}$ and $v_3={\widehat{\lambda }}_3^2/$$\left[d_3(1+n_3^e)\right]$, $d_3={\sigma }^2{\left({\stackrel{\sim }{X}}^t\stackrel{\sim }{X}\right)}^{-1}$, $n_3^e={\stackrel{\sim }{X}}^t\stackrel{\sim }{X}/{max}_i\left\{\right|{\stackrel{\sim }{X}}_i{|}^2\}$ with $\stackrel{\sim }{X}=({\mathbf{I}}_n-X^{*}{\left(X^{*t}{X}^{*}\right)}^{-1}{X}^{*})x_{l3}$ and $X^{*}=\left({\mathbf{1}}_n\right|x_{l2})$.

As an example, we analyze a data set taken from [14], which can be accessed at http://academic.uprm.edu/eacuna/datos.html (accessed on 13 January 2022). We want to predict the average mileage per gallon (denoted by mpg) of a set of $n=82$ vehicles using four possible predictor variables: cabin capacity in cubic feet (vol), engine power (hp), maximum speed in miles per hour (sp), and vehicle weight in hundreds of pounds (wt).

Through the Bayes factors $BFG$ and $BFL$, we want to choose the best model to predict the average mileage per gallon by calculating the posterior probability of the null hypothesis of the following test   

$$H_0:M_2:\mathrm{mpg}={\lambda }_1+{\lambda }_2{\mathrm{wt}}_l+{ϵ}_l\mathrm{vs}.H_1:M_3:\mathrm{mpg}={\lambda }_1+{\lambda }_2{\mathrm{wt}}_l+{\lambda }_3{\mathrm{sp}}_l+{ϵ}_l$$

with $\alpha =0.05$, $q=1$, $j=3$, the posterior probabilities for the null hypothesis $H_0$ are

$$P_{BFL}=0.9253192,P_{BFG}=0.7209449.$$

The use of this posterior probability in both cases will change the inference, since the p-value of the F test is $p=0.0325$, which is smaller than $0.05$.

Findley’s Counterexample

Consider the following simple linear model [15]

$$Y_i=\frac{1}{\sqrt{i}}·\theta +{ϵ}_i,\mathrm{where}{ϵ}_i\sim N(0,1),i=1,2,3,\dots ,n$$

and we are comparing the models $H_0:\theta =0$ and $H_1:\theta \ne 0$. This is a classical and challenging counterexample against BIC and the Principle of Parsimony. In [7], the inconsistency of BIC is shown, but the consistency of PBIC is shown in this problem.

Here, we show through the posterior probabilities of the null hypothesis that the Bayes factor $BFG$ ( based on BIC) is inconsistent, while the Bayes factor $BFL$ ( based on PBIC) is consistent if it is. We perform the analysis in two contexts: First, when n grows and $\alpha =0.05$ or $\alpha =0.01$ are fixed. Second, when n is fixed and $0<\alpha <0.05$. For calculations

$$C=-2log\frac{(1-e^{-v})}{\sqrt{2}v},v=\frac{{\widehat{\theta }}^2}{d(1+n^e)},d={\left(\sum _{i=1}^n\frac{1}{i}\right)}^{-1},n^e=\sum _{i=1}^n\frac{1}{i}.$$

Figure 5 and Figure 6 show, through the posterior probability of the null hypothesis $H_0$, the consistency of the Bayes factor based in PBIC ($P_{BFL}$), as well as the inconsistency of the Bayes factor based in BIC ($P_{BFG}$).

6. Discussion and Final Comments

1.

Lower bounds have been an important development to give practitioners alternatives to classical testing with fixed $\alpha$ levels. A deep-seated problem with the useful bound $-e·p·log\left(p\right)$ is that it depends on the p-value, which it should, but it is static, not a function of the sample size n. This limitation makes the bound of little use for moderate to large sample sizes, where it is arguably the correction to p-values more needed.

2.

The approximation develops here as a function of p-values, and sample size has a distinct advantage over other approximations, such as BIC, in that it is a valid approximation for any sample size.

3.

The (approximate) Bayes factors (9) and (11) are simple to use and provide results equivalent to the sensitive p-value Bayes factors of hypothesis tests. In this article, we extended the validity of the approximation for “pseudo-p-values,” which are ubiquitous in statistical practice. We hope that this development will give tools to the practice of statistics to make the posterior probability of hypotheses closer to everyday statistical practice, on which p-values (or pseudo-p-values) are calculated routinely. This allows an immediate and useful comparison between raw-p-values and (approximate) posterior odds.