Testing and Ranking of Asset Pricing Models Using the GRS Statistic

Abstract

We clear up an ambiguity in the statement of the GRS statistic by providing the correct formula of the GRS statistic and the first proof of its F-distribution in the general multiple-factor case. Casual generalization of the Sharpe-ratio-based interpretation of the single-factor GRS statistic to the multiple-portfolio case makes experts in asset pricing studies susceptible to an incorrect formula. We illustrate the consequences of using the incorrect formulas that the ambiguity in GRS leads to—over-rejecting and misranking asset pricing models. In addition, we suggest a new approach to ranking models using the GRS statistic p-value.

1. Introduction

In an influential paper, Gibbons et al. (1989) developed and analyzed a test of the ex ante mean-variance efficiency of portfolios. This test statistic is now widely used to evaluate asset pricing models and has also been exploited to rank competing models. For the single factor case, Gibbons et al. (1989) carefully developed the statistic in a linear regression model (hereafter referred to as the GRS statistic or test), derived its small-sample F distribution, investigated its power properties, and highlighted its significance in asset pricing theory by purveying an alternative interpretation involving the Sharpe ratio (Sharpe 1966)—the excess return to a portfolio per unit of risk (or volatility, measured by standard deviation)—which is a key measure of portfolio efficiency. For the multiple factor case, however, Gibbons et al. (1989, sec. 7), were ambiguous on how the statistic should be constructed.

The solution to the portfolio optimization problem that yields the Sharpe ratio has us estimate a variance–covariance matrix of the portfolio excess returns, but the equivalence of the GRS statistic and the F test statistic relies on this matrix arising in the projection of the test asset returns on the column space of the asset pricing factors, not as a variance–covariance matrix. Unfortunately, Gibbons et al. (1989) used equivocal language to describe this matrix, referring to it as a “variance-covariance matrix”, and this has apparently caused confusion about the function of the GRS statistic, which is further exacerbated by the fact that the small-sample F distribution wrongfully conjures up a degrees-of-freedom (d.f. hereafter) adjustment that is improper in this case. This has led to the application of a very common incorrect formula that, paradoxically, is more likely to be used by financial economists, the experts in the field, than by someone who focuses only on the statistical aspects of the problem.1 We find that using the incorrect formula, which we will refer to for conciseness as $\widehat{W}$ below, leads to (i) a test statistic that does not follow the F distribution as prescribed and over-rejects the null hypothesis of portfolio efficiency; and (ii) smaller models often being favored over larger ones when the statistic is used to rank asset pricing models. This error comes from mixing terms that fall out of portfolio optimization with a statistical object that comes from the small-sample F test derivation.

The main contribution of our paper is to clear up the ambiguity in the calculation of the GRS statistic and highlight (both theoretically and empirically) issues that arise from the use of $\widehat{W}$ and two related and popularly used statistics, one which folds in a second degree of freedom error (we will refer to this as $\breve{W}$), and the asymptotic ${\chi }^2$ version often used to replace the GRS test.2 The asymptotic ${\chi }^2$ and $\breve{W}$ implementations result in much higher model rejection rates than the correct GRS statistic, most notably for the asymptotic ${\chi }^2$ test, even with 50 years of monthly data. The use of an incorrect implementation of the GRS statistic also results in inconsistent model rankings across $\widehat{W}$, $\breve{W}$ and the correct calculation of the GRS statistic, with 40 or even 50 years of data. Finally, we propose a new methodology for the ranking of competing asset pricing models, making use of test p-values rather than the raw GRS statistic values, meant to properly internalize the model sizes. While a determination of statistically significantly different model performance is valuable, often researchers are simply attempting to rank models. Our approach is a computationally straightforward approach to answering this question.

We will adopt the notation in Gibbons et al. (1989, sec. 7), whenever possible. The proofs of the theoretical results and the details of the empirical results are in the Appendix A.

2. The GRS Test for Multiple Factors

The proofs of all the claims in this section can be found in Appendix A. The problem is to test the mean-variance efficiency of L portfolios utilizing another type of N assets (known as test assets).

We start with a linear regression model:

$${\tilde{r}}_{it}={\delta }_{i0}+{\delta }_i^{\prime }{\tilde{r}}_{pt}+{\tilde{\eta }}_{it},\forall i=1,\dots ,N,\mathrm{and}t=1,\dots ,T,$$

(1)

where ${\tilde{r}}_{it}$ denotes the excess return on test asset i in period t, the L-vector of portfolio excess returns ${\tilde{r}}_{pt}$ serves as factors, and ${\tilde{\eta }}_{it}$ denotes the disturbance. Mean-variance efficiency of the L portfolios implies (Sharpe 1964)

$$H_0:{\delta }_{i0}=0,\forall i=1,\dots ,N.$$

(2)

Lemma 1

(Joint F test). Let ${\tilde{r}}_p\equiv {\left[{\tilde{r}}_{p1},\dots ,{\tilde{r}}_{pT}\right]}^{\prime }$, ${\overline{r}}_p\equiv T^{-1}{\displaystyle {\sum }_{t=1}^T}{\tilde{r}}_{pt}$, and let ${\widehat{\delta }}_0$ be the ordinary least squares (OLS) estimator of ${\delta }_0\equiv {\left({\delta }_{10},\dots ,{\delta }_{N0}\right)}^{\prime }$; also let ${\widehat{\eta }}_t\equiv {\left({\widehat{\eta }}_{1t},\dots ,{\widehat{\eta }}_{Nt}\right)}^{\prime }$ be the OLS residuals of model (1). We follow Gibbons et al. (1989) to assume that the disturbance ${\tilde{\eta }}_t\equiv {({\tilde{\eta }}_{1t},\dots ,{\tilde{\eta }}_{Nt})}^{\prime }$ is independent from the factors ${\tilde{r}}_{pt}$ and has a joint normal distribution3 with mean zero and nonsingular variance–covariance matrix Σ and is iid over t. Define

$$\begin{array}{cc}\hfill {\tilde{\Omega }}_{\ast }& \equiv \frac{1}{T}\sum _{t=1}^T{\tilde{r}}_{pt}{\tilde{r}}_{pt}^{\prime },\hfill \end{array}$$

(3)

$$\begin{array}{cc}\hfill \widehat{\Sigma }& \equiv \frac{1}{T-L-1}\sum _{t=1}^T{\widehat{\eta }}_t{\widehat{\eta }}_t^{\prime }.\hfill \end{array}$$

(4)

Then, the F statistic

$${\tilde{W}}_{\ast }\equiv \frac{T(T-N-L)}{N(T-L-1)}\left(1-{\overline{r}}_p^{\prime }{\tilde{\Omega }}_{\ast }^{-1}{\overline{r}}_p\right){\widehat{\delta }}_0^{\prime }{\widehat{\Sigma }}^{-1}{\widehat{\delta }}_0$$

(5)

follows the $F_{N,T-N-L}$ distribution under $H_0$.

From a purely statistical perspective, Lemma 1 is all we need for testing the implication (2) of mean-variance efficiency, which is just the usual joint F test of zero intercepts in a linear regression system.4

The economic interpretation of the GRS test, however, is better understood via another implication of mean-variance efficiency—${\theta }_{N+L}^{\ast }$, the Sharpe ratio of the optimal portfolio consisting of the L portfolios and the N test assets, equals ${\theta }_p^{\ast }$, the Sharpe ratio of the L portfolios alone (Gibbons et al. 1989).

We consider a general portfolio optimization problem that yields the Sharpe ratio. Let $\tilde{r}$ denote a vector of excess returns of K assets ($K\ge 1$), and let ${\mu }_{\tilde{r}}$ and ${\Omega }_{\tilde{r}}$ be their ex ante mean vector and variance–covariance matrix, respectively. Let m be the target mean excess return and $\omega$ be a vector of K asset weights. The optimal portfolio weights ${\omega }^{\ast }$ solve

$$\underset{\omega }{min}{\omega }^{\prime }{\Omega }_{\tilde{r}}\omega ,\mathrm{subject}\mathrm{to}{\omega }^{\prime }{\mu }_{\tilde{r}}=m.$$

The square of the Sharpe ratio of the optimal portfolio composed of these K assets, therefore, is

$${\theta }^{\ast 2}\equiv {\left(\frac{m}{\sqrt{{\omega }^{\ast \prime }{\Omega }_{\tilde{r}}{\omega }^{\ast }}}\right)}^2={\mu }_{\tilde{r}}^{\prime }{\Omega }_{\tilde{r}}^{-1}{\mu }_{\tilde{r}},$$

in which the variance–covariance matrix ${\Omega }_{\tilde{r}}$ of the K assets plays a central role. Applying this general result twice, we obtain that

$$W\equiv {\left(\frac{\sqrt{1+{\theta }_{N+L}^{\ast 2}}}{\sqrt{1+{\theta }_p^{\ast 2}}}\right)}^2-1={\left(1+{\mu }_{{\tilde{r}}_p}^{\prime }{\Omega }^{-1}{\mu }_{{\tilde{r}}_p}\right)}^{-1}{\delta }_0^{\prime }{\Sigma }^{-1}{\delta }_0,$$

(6)

where $\Omega$ is the variance–covariance matrix of L portfolio excess returns ${\tilde{r}}_{pt}$ and $\Sigma$ is that of the disturbances ${\tilde{\eta }}_t$. So, $W=0$ if the L portfolios are efficient, and this is the basis of the GRS test in asset pricing theory.

Theorem 1

(Generalized GRS statistic). Define

$$\tilde{\Omega }\equiv \frac{1}{T}\sum _{t=1}^T\left({\tilde{r}}_{pt}-{\overline{r}}_p\right){\left({\tilde{r}}_{pt}-{\overline{r}}_p\right)}^{\prime }=\frac{1}{T}\sum _{t=1}^T{\tilde{r}}_{pt}{\tilde{r}}_{pt}^{\prime }-{\overline{r}}_p{\overline{r}}_p^{\prime }$$

(7)

and the generalized GRS statistic

$$\tilde{W}\equiv \frac{T(T-N-L)}{N(T-L-1)}{\left(1+{\overline{r}}_p^{\prime }{\tilde{\Omega }}^{-1}{\overline{r}}_p\right)}^{-1}{\widehat{\delta }}_0^{\prime }{\widehat{\Sigma }}^{-1}{\widehat{\delta }}_0.$$

(8)

Then, $\tilde{W}={\tilde{W}}_{\ast }$, and therefore under the conditions of Lemma 1, $\tilde{W}$ follows the $F_{N,T-N-L}$ distribution under the $H_0$.

Theorem 1 connects the statistical perspective to the economic interpretation of the GRS test, because $\tilde{W}$ equals the F statistic ${\tilde{W}}_{\ast }$ and can be regarded as a sample analog of W—replace $\Omega$ in Equation (6) with its maximum likelihood estimator (MLE) $\tilde{\Omega }$, $\Sigma$ with its unbiased estimator $\widehat{\Sigma }$, ${\mu }_{{\tilde{r}}_p}$ with ${\overline{r}}_p$, ${\delta }_0$ with ${\widehat{\delta }}_0$, and pre-multiply the ratio $\frac{T(T-N-L)}{N(T-L-1)}$, then one obtains $\tilde{W}$ in Equation (8).

Common Mistakes and Consequences

$\tilde{W}$ equals the original GRS statistic when $L=1$. For the $L>1$ case, however, Gibbons et al. (1989, p. 1146) gave a statistic $\widehat{W}$, almost identical to $\tilde{W}$, but instead of $\tilde{\Omega }$, they prescribed “sample variance-covariance matrix” $\widehat{\Omega }$ without giving its explicit formula. Since the sample variance–covariance matrix customarily entails a d.f. adjustment, i.e.,

$$\widehat{\Omega }\equiv \frac{1}{T-1}\sum _{t=1}^T\left({\tilde{r}}_{pt}-{\overline{r}}_p\right){\left({\tilde{r}}_{pt}-{\overline{r}}_p\right)}^{\prime }=\frac{T}{T-1}\tilde{\Omega },$$

(9)

this would cause $\widehat{W}$ to differ from $\tilde{W}$, and therefore Theorem 1 implies that $\widehat{W}$ does not follow the $F_{N,T-N-L}$ distribution as prescribed.

This incorrect GRS statistic $\widehat{W}$ inflicts two consequences on empirical asset pricing studies. First, it over-rejects mean-variance efficiency when gauged against the $F_{N,T-N-L}$ distribution, because the ratio between $\widehat{W}$ and $\tilde{W}$ is always larger than 1. Second, it misranks competing asset pricing models, because the ratio between $\widehat{W}$ and $\tilde{W}$ tends to be disproportionally larger for models with more factors.

The significance of the GRS statistic in recent financial studies, as Fama and French (2015) advocate, resides in the ranking of competing asset pricing models, rather than testing them. Using even the correct GRS statistics to rank models, albeit its portfolio optimization interpretation, is subject to a familiar critique akin to the use of $R^2$ for linear regression model comparison. Instead, the p-values associated with $\tilde{W}$ in respective $F_{N,T-N-L}$ distributions are a statistically sound metric for this purpose, as they internalize the difference in the second d.f.

The implementations of the GRS test found in popular user-defined software packages, such as GRS.test in R and grstest and grstest2 in Stata, not only use $\widehat{\Omega }$ when computing the GRS statistic, but also fold in additional errors. Results for the R formula are labeled with $\breve{W}$ in the rest of this paper.

The asymptotic ${\chi }^2$ test is frequently recommended as an alternative of the F test. One might think that when T is large, the d.f. issue we point out here can be circumvented by using the asymptotic ${\chi }^2$ test. Unfortunately, we find that the commonly used ${\chi }^2$ test statistics also over-reject for any sample size, especially if the number of test assets N or the number of factors L is large. In addition, they erroneously favor smaller models to an extent worse than $\widehat{W}$.

3. Empirical Results

We use portfolios of test assets borrowed from Fama and French (2015, 2016) to show that over-rejection and misranking of $\widehat{W}$, $\breve{W}$ and ${\chi }^2$ relative to $\tilde{W}$ is empirically significant, even remarkable in many cases. To summarize our empirical findings (which are detailed in Appendix B and below), the $\breve{W}$ misapplication of the GRS statistic and the asymptotic ${\chi }^2$ both result in much higher model rejection rates than the correct GRS statistic, most notably for the asymptotic ${\chi }^2$ test, even with 50 years of monthly data, and also result in scrambled model rankings, with 40 or even 50 years of data, most notably for the $\breve{W}$ version of the GRS statistic. While the F test is asymptotically equivalent to ${\chi }^2$, typical sample sizes available in financial markets research are not large enough to make this approximation innocuous. The exact F test construction is also the most conservative test, resulting in less over-rejection of the null hypothesis when the null is correct, even with highly non-normal return data, by measure of the bootstrap resampling experiments we perform.

In

Table 1. Number and proportion of cases with more rejections relative to the GRS test for five-year windows sampled during 1963–2019, across 19 sets of test assets.

Test Assets	Asymptotic			$\stackrel{\mathbf{˘}}{\mathit{W}}$			$\widehat{\mathit{W}}$
Nb of Subsamples = 53	${\mathit{\chi }}^{\mathbf{2}}$
	1%	5%	10%	1%	5%	10%	1%	5%	10%
2 × 4 × 4 MExMEBExINV	289	275	247	4	17	21	0	1	0
2 × 4 × 4 MExMEBExOP	262	246	220	4	14	18	0	0	0
2 × 4 × 4 MExOPxINV	282	238	194	10	27	28	0	3	1
5 × 5 AccrualsxME	222	218	207	7	17	31	0	0	1
5 × 5 BExME	212	191	166	8	21	20	1	0	1
5 × 5 BetaxME	229	221	204	11	16	25	0	1	0
5 × 5 MExOP	238	227	188	9	27	42	0	0	0
5 × 5 MomentumxME	210	166	124	14	29	27	0	0	0
5 × 5 NetIssuexME	216	176	147	9	29	27	0	2	2
5 × 5 RVariancexME	147	94	65	18	28	12	0	2	0
5 × 5 VariancexME	137	81	45	26	27	12	0	3	1
5 × 5 BExInv	248	259	245	4	18	16	0	2	1
5 × 5 MExInv	214	189	171	14	17	29	0	0	0
Industry	126	153	152	12	9	22	0	1	0
Book-to-Market Deciles	48	70	67	5	22	21	0	1	1
Investment Deciles	22	30	56	4	4	6	0	0	0
Momentum Deciles	56	60	66	18	14	20	1	1	1
Size Deciles	49	90	68	6	23	27	0	0	2
Operating Profitability	48	70	65	4	24	15	0	1	0
Deciles
Average	171.3	160.7	142	9.8	20.2	22.1	0.11	0.95	0.58
Proportion (%)	53.9	50.5	44.6	3.1	6.3	6.9	0.0	0.3	0.2

Notes: (1) The figures are the number of all decisions at the stated significance level for which the test statistic rejects the model when the correct GRS statistic $\tilde{W}$ does not reject, out of a total of 318 possible cases, with the exception of the last row in which the proportion is given. The sample periods of five years are sampled from July 1963 to December 2019, and the number of models tested is 6 for each window, which we sum over to obtain the total number of over-rejections. These windows overlap, adjusted in a rolling window so that all but 1 year of data overlaps with the next sample window. This means that there are 53 samples for the 5-year window. (2) For a detailed description of the factor and test asset construction see Fama and French (2015, 2016).

, we highlight the over-rejection issue, with five-year windows. This span of data shows serious over-rejection of asset pricing models from the application of the alternative formulations of the GRS test. Results for the largest number of test assets we considered, 32, are displayed in the first three rows of the table, the results for cases with 25 test assets follow in rows four through thirteen, the 17 test assets of the industry portfolios follow in row fourteen, and the remaining six rows present results for sets of 10 test assets. In Table 1, we use a total of 53 five-year overlapping windows starting from 1963 and consider six different asset pricing models, meaning that we have 318 cases for which an asset pricing model might be rejected, for each of the 19 sets of test assets.

The asymptotic ${\chi }^2$ statistic fares the worst relative to the correct GRS statistic among the alternatives; it over-rejects dramatically relative to the correct GRS statistic, 50% of the time on average, faring worse when we consider more test assets. $\breve{W}$ and $\widehat{W}$ over-reject relative to the correct GRS statistic about 10% and 1% of the time on average. $\widehat{W}$ is unstable across test assets, however, with a few cases displaying close to 4% over-rejection, and some with no over-rejection.

In

Table 2. Number and proportion of cases with different ranking outcomes from the GRS statistic for five-year windows sampled during 1963–2019, across 19 sets of test assets.

	Any Model Mis-Ranked		Top Model Mis-Ranked
Test Assets	$\stackrel{\mathbf{˘}}{\mathbf{W}}$	$\widehat{\mathbf{W}}$	$\stackrel{\mathbf{˘}}{\mathbf{W}}$	$\widehat{\mathbf{W}}$
Nb of Subsamples = 53
2 × 4 × 4 MExMEBExINV	26	2	5	0
2 × 4 × 4 MExMEBExOP	37	1	6	0
2 × 4 × 4 MExOPxINV	31	1	12	0
5 × 5 AccrualsxME	36	1	12	0
5 × 5 BExME	26	1	7	0
5 × 5 BetaxME	31	1	10	0
5 × 5 MExOP	33	0	10	0
5 × 5 MomentumxME	33	0	14	0
5 × 5 NetIssuexME	27	4	9	2
5 × 5 RVariancexME	32	0	14	0
5 × 5 VariancexME	34	1	10	0
5 × 5 BExInv	35	3	6	0
5 × 5 MExInv	30	3	9	0
Industry	29	2	4	0
Book-to-Market Deciles	28	0	4	0
Investment Deciles	24	2	4	0
Momentum Deciles	30	2	5	0
Size Deciles	27	3	5	0
Operating Profitability	30	0	6	0
Deciles
Average	30.47	1.42	8.00	0.11
Proportion (%)	57.5	2.7	15.1	0.2

Notes: (1) The figures are the number of all misrankings by the test statistic value across models relative to the correct GRS statistic ranking, out of a total of 53 possible cases, with the exception of the last row in which the proportion is given. The sample periods of five years are sampled over July 1963 to December 2019, and the number of models ranked is 6 for each window. These windows overlap, adjusted in a rolling window, so that all but 1 year of data overlaps with the next sample window. This means that there are 53 samples for the 5-year window. (2) For a detailed description of the factor and test asset construction see Fama and French (2015, 2016).

, we consider the model misranking among six models for each of the 53 five-year windows of Table 1. Misranking of models shows similar problems for the ${\chi }^2$ statistic as we saw for test rejections, with over 40% of the cases displaying misranking of at least one asset pricing model. The $\breve{W}$ statistic displays much worse performance than test rejections, with close to 60% of the cases displaying misranking, and we see worse performance even for the $\widehat{W}$, with close to 3% of the cases misranked.

While the typical Fama and French paper uses 40 or 50 years of data, it is also true that much empirical work uses far less data. Gibbons et al. (1989) noted that issues of stationarity can reasonably constrain the length of a time series used, so that “it is not uncommon to see published work where T is around 60”, Affleck-Graves and McDonald (1989) limited their analysis and simulations to 60 month periods, Ferson and Foerster (1994) studied 60, 120, and 720 monthly observations in their simulation, Rouwenhorst (1999) used five years in subsample analysis, and among recent works that exploited as little as four or five years of data are Belimam et al. (2018) and Qin (2019). Leite et al. (2018) used as few as 98 months of data, Lewellen et al. (2010) used 168 observations of quarterly data, Choi et al. (2020) performed subsample stability tests using eight years of monthly data, and many studies of emerging economy markets have used 10 to 15 years of monthly data. See, for instance, Alhomaidi et al. (2019), Alshammari and Goto (2022), Merdad et al. (2015), and Sha and Gao (2019).

One takeaway from these papers is that many situations involving specialized data (like (Sha and Gao 2019) and their exploration of mutual fund returns in China) or subsample robustness checks (like Baek and Bilson 2015) are necessarily constrained to shorter samples than fifty or even twenty years, so that the bias from an incorrectly calculated GRS statistic becomes large.

4. Concluding Remarks

The GRS statistic of Gibbons et al. (1989), developed to provide a test of the ex ante mean-variance efficiency of portfolios and more recently exploited to rank competing models, can be easily implemented incorrectly due to an ambiguity in the presentation of the multivariate form of the test in Gibbons et al. (1989). This presentation suggests a degree-of-freedom-adjusted unbiased variance–covariance matrix estimator $\widehat{\Omega }$ of the portfolio excess returns used in the small-sample GRS F test. Indeed, the portfolio optimization problem naturally has us estimate the variance–covariance matrix $\Omega$ of the portfolio excess returns, but the equivalence of the GRS statistic to the F test relies on $\tilde{\Omega }$, which arises in the projection matrix of the test asset returns on the column space of the asset pricing factors, not as a variance–covariance matrix. Paradoxically, this error is clearly visible when turning a blind eye to the economic interpretation of the GRS statistic and taking a purely statistical approach. Although an unbiased estimator $\widehat{\Omega }$ appears intuitive in the context of portfolio optimization, it does not yield a correct small-sample exact F test.

Further complicating this ambiguity, Cochrane (2005) presented the GRS statistic omitting a degree-of-freedom adjustment in the calculation of the variance–covariance matrix of the regression residuals, $\Sigma$.5 Perhaps an outcome of Cochrane (2005), there is an implementation of the GRS statistic in an R package, which we label $\breve{W}$, that omits the d.f. adjustment when estimating $\Sigma$ but fails to pre-multiply the correct ratio.6 It has also become common in the field to ignore the F distribution completely and employ an asymptotic ${\chi }^2$ approximation in place of the F test.

The main results for both the asymptotic asymptotic ${\chi }^2$ and $\breve{W}$ implementations is much higher model rejection rates than the correct GRS statistic, most notably for the asymptotic ${\chi }^2$ test, and they also result in scrambled model rankings. Further, the F distribution is inherently pertinent to small-sample exact tests, where one should make a point of computing the d.f. correctly. For this reason, we recommend the exact F test construction with its attendant F distribution, for both testing and ranking of asset pricing models. The exact F test construction is also the most conservative test, resulting in less over-rejection of the null hypothesis when the null is correct, even with highly non-normal return data.

Another result of this research inquiry is that we provided the first proof of the F-distribution of this test for the general multi-factor case and we recommended a new ranking method, making use of the p-value rather than the raw GRS statistic value. Although ranking by the values of the GRS statistic has a desirable economic intuition attached to it, the applied researcher taking advantage of this must recognize that this ranking is statistically as unsound as favoring a regression model with the highest $R^2$.