Logical Contradictions in the One-Way ANOVA and Tukey–Kramer Multiple Comparisons Tests with More Than Two Groups of Observations

Abstract

We show that the one-way ANOVA and Tukey–Kramer (TK) tests agree on any sample with two groups. This result is based on a simple identity connecting the Fisher–Snedecor and studentized probabilistic distributions and is proven without any additional assumptions; in particular, the standard ANOVA assumptions (independence, normality, and homoscedasticity (INAH)) are not needed. In contrast, it is known that for a sample with $k>2$ groups of observations, even under the INAH assumptions, with the same significance level $\alpha$, the above two tests may give opposite results: (i) ANOVA rejects its null hypothesis $H_0^A:{\mu }_1=\dots ={\mu }_k$, while the TK one, $H_0^{\mathit{TK}}(i,j):{\mu }_i={\mu }_j$, is not rejected for any pair $i,j\in \{1,\dots ,k\}$; (ii) the TK test rejects $H_0^{\mathit{TK}}(i,j)$ for a pair $(i,j)$ (with $i\ne j$), while ANOVA does not reject $H_0^A$. We construct two large infinite pseudo-random families of samples of both types satisfying INAH: in case (i) for any $k\ge 3$ and in case (ii) for some larger k. Furthermore, case (ii) ANOVA, being restricted to the pair of groups $(i,j)$, may reject equality ${\mu }_i={\mu }_j$ with the same $\alpha$. This is an obvious contradiction, since ${\mu }_1=\dots ={\mu }_k$ implies ${\mu }_i={\mu }_j$ for all $i,j\in \{1,\dots ,k\}.$ Such contradictions appear already in the symmetric case for $k=3$, or in other words, for three groups of $d,d$, and c observations with sample means $+1,-1$, and 0, respectively. We outline conditions necessary and sufficient for this phenomenon. Similar contradictory examples are constructed for the multivariable linear regression (MLR). However, for these constructions, it seems difficult to verify the Gauss–Markov assumptions, which are standardly required for MLR. Mathematics Subject Classification: 62 Statistics.

1. One-Way ANOVA and Tukey–Kramer Multiple Comparisons Tests

We use standard statistical definitions and notation; the reader can find more details in [1] or [2].

1.1. One-Way ANOVA

Consider an arbitrary sample that consists of k groups of randomly chosen real values. A group $j\in \{1,\dots ,k\}$ contains $n_j$ values $x_{\ell j}$ with $\ell =1,\dots ,n_j$. Then, $n=n_1+\dots +n_k$ is the total number of values in the sample.

Standardly, ${\overline{x}}_j$ and ${\mu }_j$ denote the sample and population means for $j=1,\dots ,k$.

We test

$$\begin{array}{cc}\hfill H_0^A:& {\mu }_1=\dots ={\mu }_k\hfill \\ \hfill H_1^A:& \mathit{not}\mathit{all}{\mu }_{\mathit{i}}\mathit{are}\mathit{the}\mathit{same},i=1,\dots ,k.\hfill \end{array}$$

The one-way ANOVA test rejects the null hypothesis $H_0^A$ with significance $\alpha$, that is, with confidence $100(1-\alpha )\%$, if and only if

$$F_{stat}>F_{crit}(\alpha ,k-1,n-k),$$

or equivalently, if the p-value corresponding to $F_{stat}$ is less than $\alpha$.

Here, $F_{crit}(\alpha ,k-1,n-k)$ is the critical value of the Fisher–Snedecor distribution corresponding to the significance level $\alpha$, with degrees of freedom of the numerator $d{f}_1=k-1$ and of the denominator $d{f}_2=n-k$.

The value $F_{stat}$ is given by the ratio

$$F_{stat}=\frac{MS\left(Tr\right)}{MSE},$$

where

$$MS\left(Tr\right)=\frac{SS\left(Tr\right)}{k-1},\hspace{1em}SS\left(Tr\right)=\sum _{j=1}^k{n}_j{({\overline{x}}_j-\overline{\overline{x}})}^2=\frac{1}{n}\sum _{j=1}^k\sum _{i=j+1}^k{n}_j{n}_i{({\overline{x}}_j-{\overline{x}}_i)}^2,$$

$$\overline{\overline{x}}=\frac{1}{n}\sum _{j=1}^k\sum _{\ell =1}^{n_j}{x}_{\ell j}=\frac{1}{n}\sum _{j=1}^k{n}_j{\overline{x}}_j.$$

(1)

$$MSE=\frac{SSE}{n-k},\hspace{1em}SSE=\sum _{j=1}^k\sum _{i=1}^{n_j}{(x_{ij}-{\overline{x}}_j)}^2,$$

Thus, ANOVA rejects $H_0^A$ if and only if

$${\displaystyle MSE<{\left(n(k-1)F_{crit}(\alpha ,k-1,n-k)\right)}^{-1}\sum _{j=1}^k\sum _{i=j+1}^k{n}_j{n}_i{({\overline{x}}_j-{\overline{x}}_i)}^2.}$$

(2)

1.2. Tukey–Kramer’s Test

For each pair $i,j\in \{1,\dots k\}$ we test the null hypothesis:

$$\begin{gathered} H_0^{\mathit{TK}}(i,j):{\mu }_i={\mu }_j \\ {H}_1^{\mathit{TK}}(i,j):{\mu }_i\ne {\mu }_j \end{gathered}$$

for all $i\ne j$.

Tukey [3] proposed a procedure for testing these hypotheses in case of equal group sizes $n_1=\dots =n_k$. Then, it was extended in [4,5] to arbitrary group sizes. This test, called the Tukey–Kramer (TK) test, uses the studentized range statistic

$$Q=\frac{{\overline{y}}_{max}-{\overline{y}}_{min}}{\sqrt{\frac{MSE}{n}}},$$

where ${\overline{y}}_{max}$ and ${\overline{y}}_{min}$ are the largest and the smallest sample means, out of a collection of k sample means.

TK rejects $H_0^{\mathit{TK}}$ if and only if

$$|{\overline{x}}_i-{\overline{x}}_j|>CR(\alpha ,k,n,i,j),$$

(3)

where the critical range (CR) is defined by the formula

$$CR(\alpha ,k,n;i,j)=Q(\alpha ,k,n-k)\sqrt{\frac{MSE}{2}\left(\frac{1}{n_i}+\frac{1}{n_j}\right)},$$

(4)

and $Q(\alpha ;k,n-k)$ is the critical value of the studentized range Q corresponding to the significance level $\alpha$, with degrees of freedom of numerator $d{f}_1=k$ and of denominator $d{f}_2=n-k$.

Equivalently, (3) can be stated as

$$MSE<2Q^{-2}(\alpha ,k,n-k){({\overline{x}}_i-{\overline{x}}_j)}^2\frac{n_i{n}_j}{n_i+n_j}.$$

(5)

1.3. Comparing ANOVA and Tukey–Kramer Tests

Both ANOVA and TK tests are based on the following standard assumptions: independence, normality, and homoscedasticity (INAH). The last one means that all groups have equal standard deviations, ${\sigma }_1=\dots ={\sigma }_k$. Both rejection criteria (2) and (5) are based on these assumptions; see [1,2,6] for more details.

In this note, we concentrate on the agreement between the above two tests rather than on their validity. Both inequalities (2) and (5) have the same left-hand side, $MSE$, which can be any number and is irrelevant for the sake of comparison of two tests.

By definition, $H_0^A$ holds if and only if $H_0^{\mathit{TK}}$ holds for all pairs $(i,j)$ with $i\ne j$. When $k=2$, there is only one such pair, and hence, the ANOVA and TK tests should agree, and indeed they are. In Section 2, we prove it for an arbitrary sample. In particular, even if the INAH assumptions are not met, still both tests either reject their null hypothesis or both do not, for any fixed significance level $\alpha$.

However, when $k>2$, even under INAH assumptions, the ANOVA and TK tests may disagree and both cases (i) and (ii) defined in the Abstract may occur. Case (i) is not a paradox. Indeed, if $H_0^{TK}(i,j):{\mu }_i={\mu }_j$ holds with significance slightly larger than $\alpha$, then it is not rejected by the TK test. This may hold for all pairs $i,j\in \{1,\dots ,k\}$ with $i\ne j$. Yet, the number of these pairs $\frac{k(k-1)}{2}$ is more than 1 when $k>2$. Thus, ANOVA may reject $H_0^A:{\mu }_1=\dots ={\mu }_k$ with significance $\alpha$.

Somewhat surprisingly, the inverse also happens: $H_0^A$ may hold with a fixed $\alpha$, while $H_0^{\mathit{TK}}$ may be rejected for some pair $(i,j)$ with the same $\alpha$. Such examples are known. Hsu [7], on page 177, remarks: “An unfortunate common practice is to pursue multiple comparisons only when the null hypothesis of homogeneity is rejected.”

We construct two large families of samples of both types considered above. In Section 3.1, we provide two randomly generated samples with three groups in each, $k=3$, and in Section 3.2, two infinite families of pseudo-random samples with $k\ge 3$ for type (i) and with some larger k for type (ii). It is important to note that these constructions are realized under INAH assumptions.

When $k>2$, Formula (4) appears to be somewhat strange: the critical range is defined for a given pair $(i,j)$ via the value of $MSE$ that depends on all observations, in all k groups. These observations are independent random variables; hence, their values in a group ℓ cannot affect the equality $H_0^{TK}(i,j):{\mu }_i={\mu }_j$ whenever $i,j,\ell$ are pairwise distinct. Moreover, group ℓ may not relate to groups i and j at all.

1.4. Modified Tukey–Kramer Test

Given significance level $\alpha$, a sample with $k>2$ groups, and a pair $(i,j)$ with $i\ne j$, let us modify the TK test for $H_0^{TK}(i,j):{\mu }_i={\mu }_j$ by eliminating all groups but i and j from the sample. Thus, we obtain a new sample with $k^{\prime }=2, n^{\prime }=n_i+n_j$, and

$$MS{E}^{\prime }=\frac{SS{E}^{\prime }}{n-2},\hspace{1em}SS{E}^{\prime }=\sum _{i^{\prime }=1}^{n_i}{(x_{i^{\prime }i}-\overline{x_i})}^2+\sum _{j^{\prime }=1}^{n_j}{(x_{j^{\prime }j}-\overline{x_j})}^2.$$

(6)

Then, we define

$$C{R}^{\prime }(\alpha ,k,n;i,j)=Q(\alpha ,2,n^{\prime }-2)\sqrt{\frac{MS{E}^{\prime }}{2}\left(\frac{1}{n_i}+\frac{1}{n_j}\right)}.$$

(7)

In Section 3.3, assuming homoscedasticity (${\sigma }_1=\dots ={\sigma }_k$) and also that $n_1=\dots =n_k$ and $\frac{n}{k}$ is large enough, we will show that $C{R}^{\prime }\le CR$. Hence, the modified TK test rejects $H_0^{TK}(i,j):{\mu }_i={\mu }_j$ whenever the standard TK test does.

Remark 1.

Note that in general, inequality $C{R}^{\prime }\le CR$ may fail since $MSE$ may be much smaller than $MS{E}^{\prime }$. Indeed, if $s_{\ell }=0$ (resp., small) for all $\ell \in \{1,\dots ,k\}\setminus \{i,j\}$, while $s_i>0$ and $s_j>0$ (resp., large), then $SS{E}^{\prime }=SSE>0$ (resp., $SSE-SS{E}^{\prime }$ may be an arbitrarily small non-negative number). Notice however that the homoscedasticity assumption might not hold when $s_i$ and $s_j$ differ significantly. Furthermore, $n-k$ may be much larger than $n^{\prime }-2$ resulting in $Q(k,n-k)\sqrt{MSE}<Q(2,n^{\prime }-2)\sqrt{MS{E}^{\prime }}$.

1.5. Counterintuitive Examples with Symmetric Samples of Two and Three Groups

Another infinite set of pseudo-random examples will be constructed in Section 4. Given two groups of observations with $n_1=n_2=d$, ${\overline{x}}_1=-1,{\overline{x}}_2=1$, ${\sigma }_1={\sigma }_2=\sigma$, and a significance level $\alpha$, we find d, and $\alpha$ such that ANOVA rejects $H_0:{\mu }_1={\mu }_2$, for some $\sigma$, with confidence $1-\alpha$. Then, we add a third group of observations with $n_3=c, {\overline{x}}_3=0, {\sigma }_3=\sigma$ and show that $H_0^A:{\mu }_1={\mu }_2={\mu }_3$, for some $\sigma$, is not rejected by ANOVA with the same confidence when $0<c<d$.

As we already mentioned, this is a logical contradiction. Let us add that condition $c<d$ seems counterintuitive. Indeed, ${\overline{x}}_3=0$, hence group 3, contains values that are typically between ${\overline{x}}_1=-1$ and ${\overline{x}}_2=1$, which could be viewed as “an argument” in support of ${\mu }_1={\mu }_2={\mu }_3$. Furthermore, the larger c is, the stronger is this argument.

1.6. ANOVA Is Not Inclusion Monotone on the Subsets of Its k Groups of Observations $\{1,\dots ,k\}$

Given a significance level $\alpha$, a sample with $k>2$ groups, and a pair $i,j\in \{1,\dots ,k\}$ with $i\ne j$, recall case (ii) (the TK test rejects $H_0^{\mathit{TK}}(i,j):{\mu }_i={\mu }_j$, while ANOVA does not reject $H_0^A:{\mu }_1=\dots ={\mu }_k$).

Reduce the sample to only two groups i and j, eliminating $k-2$ remaining groups, and apply the ANOVA and modified TK tests to the obtained sample. According to the previous subsection, the latter still rejects the equality ${\mu }_i={\mu }_j$ and, based on Theorem 1, ANOVA also rejects it, while ${\mu }_1=\dots ={\mu }_k$ is not rejected. This is a contradiction.

1.7. Logical Contradictions in F- and t-Tests of Multivariable Linear Regression (MLR)

The general multivariable linear regression model with k predictors $X^1,\dots ,X^k$ and response Y can be written as

$$Y={\beta }_0+{\beta }_1{X}^1+\dots +{\beta }_k{X}^k+ϵ.$$

The properties of the estimators of the coefficients ${\beta }_i$ are derived under the Gauss–Markov assumptions (GMA); see, for example, [8].

Commonly used tests in regressions analysis are the F-test

$$\begin{array}{cc}\hfill H_0^F:& {\beta }_1=\dots ={\beta }_k=0\hfill \\ \hfill H_1^F:& \mathrm{at} \mathrm{least} \mathrm{one} {\beta }_i \mathrm{is} \mathrm{not} 0, \mathrm{for} i=1,\dots ,k,\end{array}$$

and the t-test for individual coefficients ${\beta }_i$ for $i\in \{1,\dots ,k\}$ as follows:

$$\begin{array}{c}\hfill H_0^{t_i}:{\beta }_i=0\\ \hfill H_1^{t_i}:{\beta }_i\ne 0.\end{array}$$

It is well-known that the F- and t-tests are equivalent in the case of simple linear regression (SLR), that is, when $k=1$. In this case, the p-values of the tests are equal due to identity $F(1,\nu )=t^2\left(\nu \right)$ for all natural $\nu$, where $F(1,\nu )$ is a Fisher–Snedecor random variable with $d{f}_1=1$ and $d{f}_2=\nu$, and $t\left(\nu \right)$ is a random variable having Student’s distribution with the degrees of freedom $\nu$.

Yet, for MLR, $k>1$, logical contradictions similar to ones outlined in Section 1.3, Section 1.4 and Section 1.5 appear. With the same $\alpha$, the F- and t-tests for MLR may give opposite results as follows:

(j)

F-test rejects $H_0^F:{\beta }_1=\dots ={\beta }_k=0$, while $H_0^{t_i}:{\beta }_i=0$ is rejected for no $i\in \{1,\dots ,k\}$;

(jj)

F-test does not reject $H_0^F$, while t-test rejects $H_0^{t_i}$ for some (or even for all) $i\in \{1,\dots ,k\}$.

Similarly to case (i) of ANOVA, case (j) is not a paradox: $H_0^{t_i}$ cannot be rejected with significance $\alpha$ for each particular i, but it can be rejected with this significance for at least one i. In contrast, case ($jj$) is an obvious contradiction, since $H_0^F:{\beta }_1=\dots ={\beta }_k=0$ implies $H_0^{t_i}:{\beta }_i=0$ for every $i\in \{1,\dots ,k\}$.

The corresponding examples are shown in Section 5 for $k=2$ with the following inequalities for the p-values:

• $p_{12}>p_1$ and $p_{12}>p_2$ in case (j) in Section 5;
• $p_{12}<p_1$ and $p_{12}<p_2$ in case ($jj$) in Section 5,

where $p_{12}$ is the p-value of the F-test, while $p_1$ and $p_2$ are the p-values of the t-tests for ${\beta }_1$ and ${\beta }_2$, respectively.

Similarly to Section 1.5 for ANOVA, we will show that MLR may not be a monotonic inclusion on the set $\{1,\dots ,k\}$ of its predictors. More precisely, consider MLR F-test with k predictors and eliminate $k-1$ of them, all but i, obtaining k SLR problems, one for each predictor $X^i$ and the same response Y, for $i\in \{1,\dots ,k\}$. Denote $p_i^{\prime }$ the p-value of the SLR test i. (Recall that the F- and t-tests for SLR are equivalent and have equal p-values.)

The example also has the following property: after removing predictor $X^1$, we obtain $p_{12}>0.05>p_2^{\prime }$. Hence, for significance level $\alpha =0.05$, the null hypothesis $H_0^F:{\beta }_1={\beta }_2=0$ is not rejected, while $p_2$ and $p_2^{\prime }$ are both less than $0.05$. Thus, Case ($jj$) holds and, furthermore, both predictors $X^1$ and $X^2$ are not significant, while $X^2$ alone is significant.

Let us remark finally that our constructions of Section 1.3, Section 1.4 and Section 1.5 satisfy INAH assumptions for ANOVA; however, it seems difficult to verify the GMA, which are standardly required for MLR.

2. Two Groups, $\mathit{k}=\mathbf{2}$

In this case, a unique pair $(i,j)=(1,2)$ of means and multiple comparisons turn into a single one. ANOVA and TK tests’ null hypotheses $H_0$ coincide, stating that ${\mu }_1={\mu }_2$.

Theorem 1.

In the case of two groups, k = 2, ANOVA and TK tests are equivalent.

Proof. 

It is enough to show that inequalities (2) and (5) are equivalent when $k=2$. In this case, Formulas (2) and (5) can be rewritten as follows:

$$MSE<\frac{n_1{n}_2}{n}{F}_{crit}^{-1}(\alpha ,1,n-2){({\overline{x}}_1-{\overline{x}}_2)}^2,$$

and

$$MSE<\frac{2n_1{n}_2}{n}{Q}^{-2}(\alpha ,2,n-2){({\overline{x}}_1-{\overline{x}}_2)}^2,$$

where

$${\displaystyle MSE=\frac{1}{n-2}\sum _{j=1}^k\sum _{i=1}^{n_j}{(x_{ij}-{\overline{x}}_j)}^2.}$$

Thus, it is sufficient to prove the identity

$$Q^2(\alpha ,2,n-2)=2F_{crit}(\alpha ,1,n-2),$$

which is implied by the following lemma.

Let $F(1,\nu )$ be a Fisher–Snedecor random variable with $d{f}_1=1$ and $d{f}_2=\nu$, and $Q(2,\nu )$ be a random variable having studentized range distribution with the number of groups $k=2$ and the degrees of freedom $\nu$. □

Lemma 1.

Equation $2F(1,\nu )=Q^2(2,\nu )$ holds.

Proof. 

The probability density function of studentized range Q in case $k=2$ is given by

$$\begin{array}{cc}\hfill f_Q(q;2,\nu )& {\displaystyle =\frac{4\sqrt{2\pi }{\left(\frac{\nu }{2}\right)}^{\frac{\nu }{2}}}{\mathsf{\Gamma }\left(\frac{\nu }{2}\right)}{\int }_0^{\infty }{s}^{\nu }\varphi \left(\sqrt{\nu }s\right)\left[{\displaystyle {\int }_{\infty }^{\infty }\varphi (z+qs)\varphi \left(z\right)dz}\right]ds}\hfill \\ & {\displaystyle =\frac{4\sqrt{2\pi }{\left(\frac{\nu }{2}\right)}^{\frac{\nu }{2}}}{\mathsf{\Gamma }\left(\frac{\nu }{2}\right)}{\int }_0^{\infty }{s}^{\nu }\frac{1}{\sqrt{2\pi }}{e}^{-\frac{\nu s^2}{2}}\left[{\displaystyle {\int }_{\infty }^{\infty }\frac{1}{2\pi }{e}^{-\frac{{(z+qs)}^2+z^2}{2}}dz}\right]ds;}\hfill \end{array}$$

see [3]. We transform this formula as follows: substitute $u=\sqrt{2}z$ to obtain

$$\begin{array}{cc}\hfill f_Q(q;2,\nu )& {\displaystyle =\frac{2\sqrt{2}{\left(\frac{\nu }{2}\right)}^{\frac{\nu }{2}}}{\mathsf{\Gamma }\left(\frac{\nu }{2}\right)}{\int }_0^{\infty }{s}^{\nu }\frac{1}{\sqrt{2\pi }}{e}^{-\frac{\nu s^2}{2}}\left[{\displaystyle {\int }_{\infty }^{\infty }\frac{1}{\sqrt{2\pi }}{e}^{-\frac{{\left(u+\frac{\sqrt{2}qs}{2}\right)}^2+\frac{q^2{s}^2}{2}}{2}}du}\right]ds}\hfill \\ & {\displaystyle =\frac{2\sqrt{2}{\left(\frac{\nu }{2}\right)}^{\frac{\nu }{2}}}{\sqrt{\pi }\mathsf{\Gamma }\left(\frac{\nu }{2}\right)}{\int }_0^{\infty }{s}^{\nu }{e}^{-\frac{\left(\nu +\frac{q^2}{2}\right)s^2}{2}}ds.}\hfill \end{array}$$

Then, by substituting $t=s^2$,

$$\begin{array}{cc}\hfill f_Q(q;2,\nu )& {\displaystyle =\frac{{\left(\frac{\nu }{2}\right)}^{\frac{\nu }{2}}}{\sqrt{\pi }\mathsf{\Gamma }\left(\frac{\nu }{2}\right)}{\int }_0^{\infty }{t}^{\frac{\nu -1}{2}}{e}^{-\frac{\left(\nu +\frac{q^2}{2}\right)t}{2}}dt,}\hfill \end{array}$$

and by substituting $y=\frac{\left(\nu +\frac{q^2}{2}\right)t}{2}$,

$$\begin{array}{cc}\hfill f_Q(q;2,\nu )& {\displaystyle =\frac{\sqrt{2}{\nu }^{\frac{\nu }{2}}}{\sqrt{\pi }\mathsf{\Gamma }\left(\frac{\nu }{2}\right){\left(\nu +\frac{q^2}{2}\right)}^{\frac{\nu +1}{2}}}{\int }_0^{\infty }{y}^{\frac{\nu +1}{2}-1}{e}^{-y}dy}\hfill \\ & =\frac{\sqrt{2}{\nu }^{\frac{\nu }{2}}}{\sqrt{\pi }\mathsf{\Gamma }\left(\frac{\nu }{2}\right){\left(\nu +\frac{q^2}{2}\right)}^{\frac{\nu +1}{2}}}\mathsf{\Gamma }\left(\frac{\nu +1}{2}\right).\hfill \end{array}$$

(8)

For $X=\frac{Q^2}{2}$, we obtain

$$P\left[X\le x\right]=P\left[\frac{Q^2}{2}\le x\right]=P\left[Q\le \sqrt{2x}\right],$$

and then

$$f_X\left(x\right)=\frac{d}{dx}P\left[Q\le \sqrt{2x}\right]=\frac{1}{\sqrt{2x}}{f}_Q(\sqrt{2x};2,\nu ),$$

which, based on (8), implies that

$$\begin{array}{cc}\hfill f_X\left(x\right)& =\frac{1}{\sqrt{2x}}\frac{\sqrt{2}{\nu }^{\frac{\nu }{2}}}{\sqrt{\pi }\mathsf{\Gamma }\left(\frac{\nu }{2}\right){\left(\nu +x\right)}^{\frac{\nu +1}{2}}}\mathsf{\Gamma }\left(\frac{\nu +1}{2}\right)\hfill \\ & =\frac{1}{B(\frac{1}{2},\frac{\nu }{2})}{\left(\nu x\right)}^{-\frac{1}{2}}{\left(1+\frac{x}{\nu }\right)}^{-\frac{\nu +1}{2}},\hfill \end{array}$$

(9)

where $B(a,b)$ is the beta function.

It is well known (see, for example, [9]) that (9) defines the probability density function of the Fisher–Snedecor distribution with degrees of freedom of the numerator $d{f}_1=1$ and of the denominator $d{f}_2=\nu$. □

This proves the Theorem.

Note that Theorem 1 holds for an arbitrary sample. In particular, the p-values for ANOVA and TK tests are equal regardless of the validity of assumptions INAH.

3. Some Cases When ANOVA and TK Tests Disagree

In this section, we provide several examples where the considered two tests disagree: (i) $H_0^A$ is rejected, while $H_0^{\mathit{TK}}$ is not or (ii) vice versa. In Section 3.1, we provide two randomly generated samples illustrating (i) and (ii) with three groups in each, $k=3$; and in Section 3.2, we construct two infinite families of pseudo-random samples with $k\ge 3$ for (i) and with some larger k for (ii).

3.1. Two Examples with Three Groups

Using R, we generated two random samples with $k=3$ groups, $n_1=n_2=n_3=10$, from normal distributions with parameters ${\mu }_1=10$, ${\mu }_2=25$, ${\mu }_3=40$, and ${\sigma }_1={\sigma }_2={\sigma }_3=25$. (see

Table 1. Generated random sample for Case 1.

Group 1	Group 2	Group 3
33.73617429	41.34327861	1.949654854
6.532109599	−2.29596015	64.73534452
−15.87068125	37.80911436	17.47791461
24.41853292	−38.5488504	43.91077426
32.52469512	28.81447508	15.70006485
−36.67775074	91.99464773	54.51355702
4.946144821	54.96895462	31.54941908
−11.48789077	77.16877	0.819316511
32.04750431	95.1318948	73.84330239
−24.02530782	6.722405014	45.92357859

)

3.1.1. Case 1: ANOVA Rejects $H_0^A$, While TK Does Not Reject $H_0^{TK}$

Let us fix the significance level $\alpha =0.05$; then, $TK$ does not distinguish any pair ${\mu }_i$ and ${\mu }_j$ (see

Table 2. The results of TK test for the example in Case 1.

Group	Diff	lwr	upr	p adj
Group2-Group1	34.696519918	−1.294542536	70.68758237	0.06045
Group3-Group1	30.427939620	−5.563122834	66.41900207	0.10952
Group3-Group2	−4.268580298	−40.259642752	31.72248216	0.95353

), while

Table 3. ANOVA table for the example in Case 1.

	Df	Sum Sq	Mean Sq	F Value	Pr(>F)
group	2	7159.763	3579.881	3.39789	0.04828
Residuals	27	28,446.164	1053.562

shows that the ANOVA test rejects the null hypothesis $H_0^A$.

3.1.2. Case 2: ANOVA Does Not Reject $H_0^A$, While TK Rejects $H_0^{TK}$

Let us fix again the significance level $\alpha =0.05$. The generated random sample for Case 2 is shown in

Table 4. Generated random sample for Case 2.

Group 1	Group 2	Group 3
19.65656273	30.47282693	97.66594506
31.63471018	2.359493274	37.29706457
5.474716521	25.94822801	37.28238885
7.325738946	−6.706730014	−9.215515132
47.16633	56.00337827	44.75306142
−28.99487682	22.37945513	72.60365833
14.99564807	73.81358543	21.39501942
48.12035772	5.44699726	71.5651277
25.54178184	−3.745973145	63.33149261
−16.61305101	48.61987107	26.01262136

. Then,

Table 5. The results of TK test for the example in Case 2.

Group	Diff	lwr	upr	p adj
Group2-Group1	−10.0283214	−40.8231285393	20.76648573	0.70172
Group3-Group1	30.8382946	0.0434874678	61.63310174	0.04962
Group3-Group2	20.8099732	−9.9848339369	51.60478033	0.23279

shows that $\mathit{TK}$ distinguishes ${\mu }_1$ and ${\mu }_3$ at significance level $\alpha =0.05$. In contrast,

Table 6. ANOVA table for the example in Case 2.

	Df	Sum Sq	Mean Sq	F Value	Pr(>F)
group	2	4948.742	2474.371	3.20804	0.05624
Residuals	27	20,825.208	771.304

shows that the ANOVA test does not reject $H_0^A$ for the same $\alpha$. In this case, we can apply the approach suggested in Section 1.4 and Section 1.5. Let us reduce the sample by eliminating group 2 and apply the ANOVA and (modified) TK test.

According to Theorem 1, these two tests are equivalent: p-value is $0.02432$ (see

Table 7. ANOVA table for groups 1 and 3 in Case 2.

	Df	Sum Sq	Mean Sq	F Value	Pr(>F)
group	1	4755.002	4755.002	6.044	0.02432
Residuals	18	14,161.164	786.731

and

Table 8. The results of modified TK test for groups 1 and 3 in Case 2.

Group	Diff	lwr	upr	p adj
Group3-Group1	30.8382946	4.484804399	57.1917848	0.02431

) for the equality ${\mu }_1={\mu }_2$ in both cases. Yet, for the original sample of three groups, the p-value was $0.05624$ for the equality ${\mu }_1={\mu }_2={\mu }_3$. This is an obvious contradiction: ANOVA rejects ${\mu }_1={\mu }_2$ with confidence $97.5\%$ but cannot reject the stronger statement ${\mu }_1={\mu }_2={\mu }_3$ (which is easier to do) even with confidence $95\%$.

Recall that this example was generated by R under INAH assumptions. This did not take too many trials: with given parameters $k=3, n_1=n_2=n_3=10, {\mu }_1=10,{\mu }_2=25,{\mu }_3=40$, and ${\sigma }_1={\sigma }_2={\sigma }_3=25$, about each 20 trials provide an example with such properties.

3.2. Two Large Families of Examples with K Groups

In this subsection, we consider samples with k groups such that n is divisible by k, and

$$n_1=\dots =n_k=\frac{n}{k},$$

(10)

$$s_1=\dots =s_k=s,$$

(11)

where $s_i$ is the standard deviation of the ith group. In this case, we have

$$SSE=k\left(\frac{n}{k}-1\right)s^2, MSE=\frac{SSE}{n-k}=s^2.$$

(12)

3.2.1. Case 1: ANOVA Rejects $H_0^A$, While TK Does Not Reject $H_0^{\mathit{TK}}$

Based on (2) and (5), this happens if and only if

$$\begin{array}{cc}& {\displaystyle {\left(n(k-1)F_{crit}(\alpha ,k-1,n-k)\right)}^{-1}\sum _{j=1}^k\sum _{i=j+1}^k{n}_j{n}_i{({\overline{x}}_j-{\overline{x}}_i)}^2}\hfill \\ & >MSE>2Q^{-2}(\alpha ,k,n-k){({\overline{x}}_i-{\overline{x}}_j)}^2\frac{n_i{n}_j}{n_i+n_j},\hfill \end{array}$$

(13)

which implies

$$\begin{array}{cc}& {\displaystyle Q^2(\alpha ,k,n-k)\sum _{j=1}^k\sum _{i=j+1}^k{n}_j{n}_i{({\overline{x}}_j-{\overline{x}}_i)}^2}\hfill \\ & >2n(k-1)F_{crit}(\alpha ,k-1,n-k){({\overline{x}}_i-{\overline{x}}_j)}^2\frac{n_i{n}_j}{n_i+n_j}.\hfill \end{array}$$

(14)

Consider any sample with k groups, k is even, satisfyng (10) and (11), and

$${\overline{x}}_1=\dots ={\overline{x}}_{\frac{k}{2}}=1,{\overline{x}}_{\frac{k}{2}+1}=\dots ={\overline{x}}_k=0.$$

(15)

Based on (12), $MSE=s^2$, and (14) can be rewritten as

$${\left(\frac{n}{k}\right)}^2\frac{k^2}{4}{Q}^2(\alpha ,k,n-k)>2\frac{n^3}{k^2\left(2\frac{n}{k}\right)}{F}_{crit}(\alpha ,k-1,n-k),$$

which can be simplified to

$$G(\alpha ,k,n-k)=\frac{Q^2(\alpha ,k,n-k)}{F_{crit}(\alpha ,k-1,n-k)}-4\left(1-\frac{1}{k}\right)>0.$$

(16)

Function $G(\alpha ,k,n-k)$ has the following properties:

• $G(\alpha ,k,n-k)\equiv 0$ if $k=2$;
• $G(\alpha ,k,n-k)$ is monotonically increasing with respect to $n-k$ and converging as $n\to \infty$ for each k;
• $G(\alpha ,k,n-k)>0$ for $k\ge 3$ and all $n-k\ge 0$ for $\alpha =$$0.005$, $0.01$, $0.025$, $0.05$, $0.1$, $0.25$, and $0.5$.

It is not our goal to study function $G(\alpha ,k,n-k)$ in detail; we are primarily interested only in its positivity, required by condition (16). The required inequality (16) holds for any $k\ge 3$.

Given an even k and n divisible by k, we generate a desired pseudo-random sample as follows: It satisfies (10), (11), (15), and in addition, whenever (16) holds, we still can choose $s^2=\mathit{MSE}$ satisfying (13).

3.2.2. Case 2: ANOVA Does Not Reject $H_0^A$, While TK Rejects $H_0^{TK}$

Based on (2) and (5), this happens if and only if

$$\begin{array}{cc}& {\displaystyle {\left(n(k-1)F_{crit}(\alpha ,k-1,n-k)\right)}^{-1}\sum _{j=1}^k\sum _{i=j+1}^k{n}_j{n}_i{({\overline{x}}_j-{\overline{x}}_i)}^2}\hfill \\ & <MSE<2Q^{-2}(\alpha ,k,n-k){({\overline{x}}_i-{\overline{x}}_j)}^2\frac{n_i{n}_j}{n_i+n_j},\hfill \end{array}$$

(17)

which implies

$$\begin{array}{c}{\displaystyle Q^2(\alpha ,k,n-k)\sum _{j=1}^k\sum _{i=j+1}^k{n}_j{n}_i{({\overline{x}}_j-{\overline{x}}_i)}^2}\hfill \\ <2n(k-1)F_{crit}(\alpha ,k-1,n-k){({\overline{x}}_i-{\overline{x}}_j)}^2\frac{n_i{n}_j}{n_i+n_j}.\hfill \end{array}$$

(18)

Note that if $k=2$, we obtain (8).

Consider any sample with k groups satisfying (10) and (11), and

$${\overline{x}}_1=1,{\overline{x}}_2=\dots ={\overline{x}}_k=0.$$

(19)

Then, (18) turns into

$$\begin{array}{c}\hfill {\left(\frac{n}{k}\right)}^2(k-1)Q^2(\alpha ,k,n-k)<\frac{2n(k-1){\left(\frac{n}{k}\right)}^2}{2\frac{n}{k}}{F}_{crit}(\alpha ,k-1,n-k),\end{array}$$

which simplifies to

$$\begin{array}{c}\hfill H(\alpha ,k,n-k)=\frac{Q^2(\alpha ,k,n-k)}{F_{crit}(\alpha ,k-1,n-k)}-k<0.\end{array}$$

(20)

Since s can be chosen arbitrarily, we can always find $MSE$ satisfying (17) whenever (18) holds.

Function $H(\alpha ,k,n-k)$ shares properties (1) and (2) of $G(\alpha ,k,n-k)$; furthermore, $H(\alpha ,k,n-k)>0$ for sufficiently small k, and $H(\alpha ,k,n-k)<0$ for sufficiently large k. Again, it is not our goal to study $H(\alpha ,k,n-k)$ in detail since we are primarily interested only in its negativity required by condition (20).

The signs of $H(\alpha ,k,n-k)$ depending on k are shown in

Table 9. The signs of $H(\alpha ,k,n-k)$ for selected $\alpha$ depending on k.

$\mathit{\alpha }$	$\mathit{H}(\mathit{\alpha },\mathit{k},\mathit{n}-\mathit{k})>0$	$\mathit{H}(\mathit{\alpha },\mathit{k},\mathit{n}-\mathit{k})<0$
0.005	$3\le k\le 10$	$k\ge 14$
0.01	$3\le k\le 10$	$k\ge 14$
0.025	$3\le k\le 10$	$k\ge 13$
0.05	$3\le k\le 10$	$k\ge 12$
0.1	$3\le k\le 10$	$k\ge 12$
0.25	$3\le k\le 10$	$k\ge 11$
0.5	$3\le k\le 9$	$k\ge 10$

for some values of $\alpha$. The second (resp., third) column contains the values of k such that $H(\alpha ,k,n-k)>0$ (resp., $H(\alpha ,k,n-k)<0$) for all n. Missing values of k correspond to the cases when the sign of $H(\alpha ,k,n-k)$ depends on n.

One can see that the required inequality (20) holds when the number of groups k is large enough.

Given k and n divisible by k, we generate the desired pseudo-random sample as follows: It satisfies (10), (11), (19), and in addition, whenever (20) holds, we still can choose $s^2=\mathit{MSE}$ satisfying (17).

Remark 2.

We variate the choice of sample means in (15) and (19) to increase the feasible area for (16) and (20), respectively. Obviously, $k\ge 4\left(1-\frac{1}{k}\right)$ and equality hold if and only if $k=2$.

Remark 3.

We can extend considerably the family of the constructed examples by relaxing equalities (11), (15), and (19), and replacing them with approximate equalities.

3.3. Critical Range in Modified TK Test

In Section 1.4, we modified the standard TK multiple comparisons test, replacing it with the pairwise comparison version as follows: Given significance level $\alpha$, a sample with $k>2$ groups, and a pair $(i,j)\in \{1,\dots ,k\}$ with $i\ne j$, consider the null hypothesis for the corresponding two groups, $H_0^{\mathit{TK}}(i,j):{\mu }_i={\mu }_j$ and eliminate all groups but i and j from the sample, obtaining a new one with $k^{\prime }=2, n^{\prime }=n_i+n_j$. For the standard and modified TK tests, the critical ranges $CR=CR(\alpha ,k,n-k;i,j)$ and $C{R}^{\prime }=C{R}^{\prime }(\alpha ,k,n-k;i,j)$ and the corresponding values of $MSE$ and $MS{E}^{\prime }$ are given by Formulas (1), (4), (6) and (7).

We are looking for conditions implying the inequality $C{R}^{\prime }\le CR$, in which case the modified TK test rejects $H_0^{TK}(i,j)$ whenever the standard one does. In general, this inequality may fail; see Remark 1.

Let us assume INAH, and in addition (10) and (11). As we know, in this case, $MSE=MS{E}^{\prime }=s^2$ and formulas for $CR$ and $C{R}^{\prime }$ are simplified as follows:

$$\begin{array}{cc}\hfill CR& =Q(\alpha ,k,n-k)s\sqrt{\frac{k}{n}},\hfill \\ \hfill C{R}^{\prime }& =Q(\alpha ,2,n^{\prime }-k^{\prime })s\sqrt{\frac{k^{\prime }}{n^{\prime }}}\hfill \\ & =Q\left(\alpha ,2,2\frac{n}{k}-2\right)s\sqrt{\frac{2}{2n/k}}=Q\left(\alpha ,\frac{k}{k/2},\frac{n-k}{k/2}\right)s\sqrt{\frac{k}{n}}.\hfill \end{array}$$

Thus, in the considered case,

$$\frac{C{R}^{\prime }}{CR}=\frac{Q(\alpha ,2,\nu )}{Q(\alpha ,2\ell ,\nu \ell )},$$

where $\ell =\frac{k}{2}\ge 1$ and $\nu =n^{\prime }-k^{\prime }=2\left(\frac{n}{k}-1\right)$.

The critical value of the studentized range $Q(\alpha ,2\ell ,\nu \ell )$ monotonically increases with ℓ when $\nu =2\left(\frac{n}{k}-1\right)$ is large enough; see

Table 10. Conditions for monotonic increasing of the studentized range $Q(\alpha ,2\ell ,\nu \ell )$.

$\mathit{\alpha }$	0.005	0.01	0.025	0.05	0.1	0.25	0.5
	$\nu \ge 7$	$\nu \ge 5$	$\nu \ge 4$	$\nu \ge 3$	$\nu \ge 2$	$\nu \ge 1$	$\nu \ge 1$

.

In these cases, $C{R}^{\prime }\le CR$, and hence, conclusions of Section 1.5 are applicable. Recall the construction of Section 3.2 Case 2, in which ANOVA does not reject $H_0^A:{\mu }_1=\dots ={\mu }_k$, while the standard TK test rejects $H_0^{\mathit{TK}}(i,j):{\mu }_i={\mu }_j$. This pseudo-random construction satisfies INAH. Let us apply ANOVA and TK tests to the reduced sample that consists of only two groups i and j, with the remaining $k-2$ groups eliminated. Based on the above arguments, the modified TK test still rejects its hypothesis $H_0^{\mathit{TK}}(i,j):{\mu }_i={\mu }_j$ and, based on Theorem 1, ANOVA rejects it as well. However, ANOVA does not reject a stronger hypothesis $H_0^A:{\mu }_1=\dots ={\mu }_k$, with the same significance level $\alpha$, which is an obvious contradiction.

4. Symmetric Samples with Two and Three Groups

4.1. Two Groups

Consider two groups 1 and 2 with d observations in each, that is, $k=2, n_1=n_2=d, n=n_1+n_2=2d$, with means ${\overline{x}}_1=-1, {\overline{x}}_2=-1$, and standard deviations ${\sigma }_1={\sigma }_2=\sigma$. We can assume that $INAH$ holds.

Obviously, $SS\left(Tr\right)=MSR=2d$; furthermore, according to (1),

$$SSE={\sigma }^2, MSE=SSE/(n-k)={\sigma }^2/(2d-2),$$

$$F_{stat}=MSR/MSE=4d(d-1){\sigma }^{-2}.$$

According to (2), ANOVA rejects its null hypothesis $H_0^A:{\mu }_1={\mu }_2$ if and only if

$$4d(d-1){\sigma }^{-2}>F_{crit}(\alpha ,k-1,n-k)=F_{crit}(\alpha ,1,2d-2)$$

(21)

As for the $TK$ (in this case just Tukey test), we have $|{\overline{x}}_1-{\overline{x}}_2|=2$, and based on (4), the critical range is given by formula

$$CR(1,2)=Q(\alpha ,2,2d-2)\sqrt{MSE/d}=Q(\alpha ,2,2d-2)\sigma /\sqrt{2d(d-1)}.$$

Thus, according to (3), the Tukey test rejects ${\mu }_1={\mu }_2$ if and only if

$$8d(d-1){\sigma }^{-2}>Q^2(\alpha ,2,2d-2).$$

(22)

Criteria (21) and (22) are equivalent, based on Lemma 1.

4.2. Three Groups

Let us add to groups 1 and 2 one more group 3 of c observations, obtaining $k=3$ and $n=n_1+n_2+n_3=2d+c$. Furthermore, set ${\overline{x}}_3=0$ and ${\sigma }_3=\sigma$ and assume that $INAH$ holds. Then, based on (1) and (2),

$$SS\left(Tr\right)=2d, MSR=SSR/(k-1)=d;$$

$$SSE={\sigma }^2, MSE=SSE/(n-k)={\sigma }^2/(2d+c-3),$$

$$F_{stat}=MSR/MSE=d(2d+c-3){\sigma }^{-2}.$$

Thus, ANOVA rejects its null hypothesis $H_0^A:{\mu }_1={\mu }_2={\mu }_3$ if and only if

$$d(2d+c-3){\sigma }^{-2}>F_{crit}(\alpha ,k-1,n-k)=F_{crit}(\alpha ,2,2d+c-3).$$

(23)

As for the $TK$ test, we have $|{\overline{x}}_1-{\overline{x}}_2|=2$, while based on (4), the critical range

$$CR(1,2)=Q(\alpha ,k,n-k)\sqrt{MSE/d}=\sigma Q(\alpha ,3,2d+c-3)/\sqrt{d(2d+c-3)}.$$

Thus, based on (3), the $TK$ test rejects ${\mu }_1={\mu }_2$ if and only if

$$4d(2d+c-3){\sigma }^{-2}>Q^2(\alpha ,3,2d+c-3).$$

(24)

4.3. ANOVA for $k=2$ and $k=3$

ANOVA rejects ${\mu }_1={\mu }_2$ and does not reject ${\mu }_1={\mu }_2={\mu }_3$ if and only if (21) holds while (23) fails. It happens, for some $\sigma$, if and only if

$$\frac{F_{crit}(\alpha ,1,2a)}{F_{crit}(\alpha ,2,2a+b)}<2\frac{2a}{2a+b},$$

(25)

where $a=d-1, b=c-1$.

Remark 4.

Obviously, the set of feasible σ is an interval. In fact, this holds for all our examples showing logical contradictions. Such examples are relatively rare because the interval for σ is small.

Consider $\alpha =0.05$. Inequality (25) holds whenever $b<a$, or equivalently, $c<d$. It seems that (25) can be solved, with respect to a and b, explicitly. Consider the following sequence of positive integers $S=(6,A^2,B^4,A,B^5,{(A,B^6)}^{\infty })$, where $A=(8,9,8,8,9)$, $B=(8,9,8,8,9,8,8,9)$; a power denotes the number of repetitions. Thus, S is a quasi-periodic sequence with the period $(A,B^6)$ of length $5+8·6=53$. To each a we assign a nonnegative integer $a\left(s\right)$ uniquely defined by the inequalities

$$\sum _{i=1}^{a\left(s\right)}{s}_i\le a<\sum _{i=1}^{a\left(s\right)+1}{s}_i.$$

Then, (25) holds if and only if $b<a+a\left(s\right)$. This criterion is confirmed by computations for $a\le 500$. We conjecture that it holds for all a and that similar criteria hold for all $\alpha$.

4.4. TK Test for $k=2$ and $k=3$

$TK$ rejects equality ${\mu }_1={\mu }_2$ for $k=2$ and does not reject it for $k=3$ if and only if (22) holds while (24) fails. It happens, for some $\sigma$, if and only if

$$\frac{Q^2(\alpha ,3,2a+b)}{Q^2(\alpha ,2,2a)}<\frac{2a+b}{2a},$$

(26)

Consider $\alpha =0.05$. Inequality (26) holds whenever $b\ge a$, or equivalently, $c\ge d$. Again, it seems that (26) can be solved, with respect to a and b, explicitly. Consider sequence of positive integers $S=(3,7,{(8,7^7,8,7^6)}^{\infty })$. It is quasi-periodic with the period $(8,7^7,8,7^6)$ of length 15.

Then, (26) holds if and only if $b\ge a-a\left(s\right)$. This criterion is confirmed by computations for $a\le 500$. We conjecture that it holds for all a and that similar criteria hold for all $\alpha$.

5. Logical Contradictions in Multivariable Linear Regression

Here, we provide examples presented in Section 1.7, see

Table 11. Generated random sample for Case (j).

X1	X2	Y
1.713673333	0.891652019	1.718488057
0.932830925	0.353231823	1.311861467
−0.053673724	1.132586717	1.903344806
1.055482137	0.248411619	1.582305067
−0.248355435	−0.174256727	2.607296494
−0.004449867	0.115550588	2.352411276
0.086988258	−0.833496007	2.558602277
0.687284914	−0.417171685	1.721811264
−0.253474712	0.045371123	1.982673543
0.135747949	−0.145817805	2.309533234

,

Table 12. Regression output for the sample in Table 11.

Coefficients	Estimate	Std. Error	t Value	$\mathit{Pr}(>|\mathit{t}\left|\right)$
(Intercept)	2.193398826	0.12112005	18.10929596	3.87188 × 10${}^{-7}$
X1	−0.397578417	0.173782994	−2.28778667	0.05599
X2	−0.225853392	0.196597153	−1.148813135	0.28837
Residual standard error: 0.321371159 on 7 degrees of freedom
Multiple R-squared: 0.578426902, Adjusted R-squared: 0.457977446
F-statistic: 4.802237548 on 2 and 7 DF, p-value: 0.04865

,

Table 13. Generated random sample for Case ($jj$).

X1	X2	Y
1.173699045	1.507797593	1.693611518
1.527866588	1.204880159	1.719565524
−0.237756887	0.321525784	2.313343543
0.424876707	0.372472796	2.215619921
0.155008273	−0.382097849	1.752313506
0.078297635	0.202406996	1.985018225
−0.739378749	−1.77490523	1.280511608
−0.325947264	−0.170739193	1.751709441
0.057639294	0.025498039	2.285726127
0.317517151	0.439566564	1.809984615

,

Table 14. Regression output for the sample in Table 13.

Coefficients	Estimate	Std. Error	t Value	$\mathit{Pr}(>|\mathit{t}\left|\right)$
(Intercept)	1.927174047	0.094242869	20.44901714	1.67739 × 10${}^{-7}$
X1	−0.534309107	0.266285323	−2.006528569	0.08480
X2	0.478129512	0.20172263	2.37023239	0.04959
Residual standard error: 0.271926659 on 7 degrees of freedom
Multiple R-squared: 0.445574672, Adjusted R-squared: 0.287167435560639
F-statistic: 2.812842909 on 2 and 7 DF, p-value: 0.12690

,

Table 15. Another generated random sample for Case ($jj$).

X1	X2	Y
1.568562319	0.927834903	1.612462698
1.48286001	1.09773946	2.466033052
−0.573115658	0.981537183	2.518881417
−0.050008016	−0.49329821	1.301806858
0.165268254	−0.397500853	1.436310825
−0.306203404	−0.193130393	2.072432714
−0.399941489	−0.096035236	1.573998771
0.21069356	0.603984432	1.827021014
0.431810105	−0.383312909	1.933014077
0.080628207	0.231611299	2.002964703

,

Table 16. Regression output for the sample in Table 15.

Coefficients	Estimate	Std. Error	t Value	$\mathit{Pr}(>|\mathit{t}\left|\right)$
(Intercept)	1.8040579	0.1149434	15.69519	1.0317 × 10${}^{-6}$
X1	−0.1814855	0.1720305	−1.05496	0.32649
X2	0.5168505	0.2013693	2.56668	0.03719
Residual standard error: 0.3324189 on 7 degrees of freedom
Multiple R-squared: 0.486189, Adjusted R-squared: 0.3393858
F-statistic: 3.311843 on 2 and 7 DF, p-value: 0.09723

,

Table 17. Regression output for the generated random sample for Case ($jj$) with independent variable $X1$ only.

Coefficients	Estimate	Std. Error	t Value	$\mathit{Pr}(>|\mathit{t}\left|\right)$
(Intercept)	1.86701480	0.14635023	12.75717	1.3432 × 10${}^{-6}$
X1	0.02864453	0.19718444	0.14527	0.88809
Residual standard error: 0.4332275 on 8 degrees of freedom
Multiple R-squared: 0.002631, Adjusted R-squared: −0.122040
F-statistic: 0.0211027 on 1 and 8 DF, p-value: 0.8880929

and

Table 18. Regression output for the sample in Table 15 with independent variable $X2$ only.

Coefficients	Estimate	Std. Error	t Value	$\mathit{Pr}(>|\mathit{t}\left|\right)$
(Intercept)	1.7797247	0.1133974	15.69458	2.7117 × 10${}^{-7}$
X2	0.4157529	0.1783505	2.33110	0.04808
Residual standard error: 0.3347572 on 8 degrees of freedom
Multiple R-squared: 0.404497, Adjusted R-squared: 0.330059
F-statistic: 5.434026 on 1 and 8 DF, p-value: 0.04808

.

5.1. Construction for Case (j): F-Test Rejects $H_0^F:{\beta }_1={\beta }_2=0$, While $H_0^{t_1}:{\beta }_1=0$, $H_0^{t_2}:{\beta }_2=0$ Are Not Rejected by t-Tests with the Same Significance $\alpha =0.05$

Note that $p_1=0.05598>0.05,p_2=0.28837>0.05,p_{12}=0.04865<0.05.$ Hence, Case (j) holds.

5.2. Constructions for Case (jj): F-Test Does Not Reject $H_0^F$, While t-Tests Reject $H_0^{t_1}$ or $H_0^{t_2}$ with the Same Significance

Here $p_1=0.08480$, $p_2=0.04959$, and $p_{12}=0.12690$. Hence, F-test does not reject $H_0^F$, while t-tests reject $H_0^{t_1}$ and $H_0^{t_2}$ with significance $\alpha =0.1$.

The next example also illustrates case ($jj$) and, in addition, shows that F-test can be as not monotone inclusion on the set of predictors.

Note that $p_2=0.03719<0.05<0.09723=p_{12}$. Hence, Case ($jj$) holds. Furthermore, eliminating predictor $X1$ yields the following SLR table:

We observe again that $p_2^{\prime }=0.04808<0.05<0.09723=p_{12}$. Thus, F test states that both predictors $X1$ and $X2$ are insignificant, while $X2$ alone is significant at $\alpha =0.05$.

6. Concluding Remarks

Both ANOVA and $TK$ multiple comparisons tests with k groups may result in logical contradictions when $k>2$, even if $INAH$ assumptions hold. Therefore, the good old approach of using pairwise comparisons instead of multiple ones is a bit slower but more reliable. Furthermore, all contradictions disappear if we replace the ANOVA and $TK$ tests by their pairwise versions, applying them for any pair of groups $i,j\in \{1,\cdots ,k\}$ with $i\ne j$. Then, according to Theorem 1, these two tests become equivalent.

Similar contradictions appear for the linear regression with the number of predictors $k>1$ (MLR). Already for $k=2$, with the same level of significance $\alpha$, it may happen that t-test rejects $H_0^{t_1}:{\beta }_1=0$, while F-test fails to reject the stronger null hypothesis $H_0^F:{\beta }_1={\beta }_2=0$.

In general, estimating the quality of a prediction made by ANOVA or MLR seems much more doubtful than the prediction itself.