A Simple Out-of-Sample Test of Predictability against the Random Walk Benchmark

Abstract

We show that a straightforward modification of a trading-based test for predictability displays interesting advantages over the Excess Profitability (EP) test proposed by Anatolyev and Gerco when testing the Driftless Random Walk Hypothesis. Our statistic is called the Straightforward Excess Profitability (SEP) test, and it avoids the calculation of a term that under the null of no predictability should be zero but in practice may be sizable. In addition, our test does not require the strong assumption of independence used to derive the EP test. We claim that dependence is the rule and not the exception. We show via Monte Carlo simulations that the SEP test outperforms the EP test in terms of size and power. Finally, we illustrate the use of our test in an empirical application within the context of the commodity-currencies literature.

1. Introduction

The objective of this paper is to provide a new out-of-sample test of the Random Walk Hypothesis (RWH) for the particular case in which the predictand is a zero mean variable. Leading examples of a setup such as ours include the prediction of high frequency asset returns, for which the economic literature has normally used either a random walk or a driftless random walk as a benchmark to test for predictability. In the context of asset prices, for instance, the driftless random walk posits that today’s price is the best forecast of tomorrow’s price. In other words, this model assumes that future expected returns, given information available today, are zero (strictly speaking, the implication of a zero forecast for future expected returns emerges when a driftless random walk model is assumed for the natural logarithm of the corresponding asset price).

Although extremely simple, this driftless random walk model that generates the trivial zero forecast is of tremendous importance in the financial and economic literature. For instance, in the context of exchange rates, the review by Rossi (2013) [1] identifies the driftless random walk as the most difficult benchmark to beat in forecast competitions. Similarly, Clark and West (2006) [2] derived an out-of-sample test of predictive ability for cases in which a given model is compared to the driftless random walk. Finally, Pincheira and Hardy (2021) [3] explain that part of the problem of beating the driftless random walk relies on its minimum magnitude: The zero forecast is so small that it can outperform a non-zero forecast in terms of Mean Squared Prediction Error despite having zero covariance with the predictand. In addition, driftless random walk is used as a common benchmark in the literature to compare more sophisticated models (see for instance, Meese and Rogoff (1983) [4], Chen, Rogoff and Rossi (2010) [5], Pincheira et al. (2021) [6], Pincheira and Hardy (2018, 2019, 2021) [7,8,9], ref. [2] and many others).

We propose the construction of the Straightforward Excess Profitability (SEP) test. This test is built as a simple modification of the Excess Profitability (EP) test originally developed by Anatolyev and Gerco (2005) [10]. SEP and EP test statistics differ because the former avoids the calculation of a term that under the null of no predictability should be zero but in practice is sizable. In addition, our test does not require the strong assumption of independence used to derive the EP test. We claim that dependence is the rule and not the exception. We also show via Monte Carlo simulations that both size and power of the SEP test are in general better than those of the EP test. Despite these distinctions, the SEP and EP tests share the same rich economic interpretation: The finding of predictability indicates that a given trading strategy provides positive returns on average. Therefore, these two tests depart from the vast literature focused on statistical measures of predictability as Mean Squared Prediction Errors (MSPE), Least Absolute Deviations and Linex loss functions and Direction of Change, among others (see McCracken and West (2002) [11], Chinn and Meese (1995) [12], Cheung, Chinn and Garcia (2005) [13], Patton and Timmermann (2007) [14] and Giacomini and White (2006) [15]).

While the simple RWH is generally rejected when the econometrician engages in conventional in-sample analysis, it is indeed a difficult benchmark to beat when an out-of-sample approach is followed. The seminal paper of [4] is a classic example of this problem in the context of the exchange rate literature. This is sometimes interpreted as indication that in-sample analyses are affected by overfitting or data mining problems and, therefore, should be disregarded. While the conflicting results from in-sample and out-of-sample approaches are not entirely clear, Inoue and Kilian (2005) [16] emphasize that in-sample strategies display higher power relative to out-of-sample analyses. According to this argument, out-of-sample tests of the RWH would fail to reject the null of no predictability mainly due to the low power of these tests. Therefore, it is essential to derive out-of-sample tests of the RWH displaying power improvements with respect to their competitors.

A key point to consider when testing RWH is that traditional statistical methods of comparing predictive accuracy, as those presented by West (1996) [17] and Diebold and Mariano (1995) [18], are not adequate. The reason for this is that the models under analysis are nested (see [17]). McCracken (2007) [19] and Clark and McCracken (2005) [20] show that when comparing nested models, direct application of traditional methods may result in tests of inadequate size. In particular, [2], Clark and West (2007) [21] and Pincheira and West (2016) [22] show that traditional comparisons of MSPE render tests with low power and questionable size when comparing forecasts from nested models.

Two interesting alternatives to construct proper statistical methods to compare predictive accuracy when models are nested are presented by [2,19,21]. In the first paper, an asymptotic distribution of a t-type statistic, comparable to that suggested by [17,18], is derived. While the distribution is nonstandard, tables of asymptotically correct critical values are provided. In the other two papers, refs. [2,21] make an important observation about the null hypothesis of equal MSPE when models are nested. They show that under the null, the sample distribution of the difference in MSPE is not zero. Instead, the sample MSPE from the null model is expected to be smaller than that of the alternative model. They propose an adjusted procedure that considers this finding and demonstrate via simulations that this procedure is well-sized and relatively powerful in small and moderate samples. A third alternative to properly test for model adequacy when models are nested is provided by Chao, Corradi and Swanson (2000) [23]. They provide a test of Granger causality that displays good size and power properties in small samples when applied to the RWH.

Despite these interesting alternatives to properly test RWH, this field of research is still not closed. From a theoretical point of view, we have already mentioned that it is desirable to derive more powerful tests in small and moderate samples and, more generally, to derive tests with better small sample properties. We would also like to have a test with a direct economic interpretation that allows us, for example, to obtain positive returns when using a given trading rule. This is exactly what we perform in this paper. We introduce the SEP test, which has a direct economic interpretation and displays better size and power than the EP test in our simulations.

The rest of the paper is organized as follows: Section 2 introduces the econometric environment; Section 3 shows that the original EP test may be biased under the null hypothesis of a driftless random walk model; Section 4 introduces the new SEP test. Section 5 describes the experimental design and delivers simulation results; Section 6 presents an empirical application within the commodity-currencies literature, and Section 7 provides the conclusion.

2. Econometric Environment

Let us consider the setup introduced by [2] in which two simple models for a scalar stationary and ergodic time series $y_{t+1}$ are defined as follows:

$$\mathrm{Model} 1 \left(\mathrm{null}\right): y_{t+1}=e_{t+1}$$

(1)

$$\mathrm{Model} 2 \left(\mathrm{alternative}\right): y_{t+1}=X_{t+1}^{\prime }\beta +e_{t+1}$$

(2)

where $X_{t+1}$ is a vector of ergodic and stationary exogenous random variables, and $e_{t+1}$ is a zero mean martingale difference meaning that $\mathbb{E}\left(e_{t+1}|{\Im }_t\right)=0$, where $\left\{{\mathfrak{F}}_t\right\}$ represents a filtration such that ${\mathfrak{F}}_t$ is the sigma-field generated by current and past $X^{\prime }s$ and $e^{\prime }s$.

$${\Im }_t=\sigma \left\{X_{t+1}, e_{t }, X_t, e_{t-1}, X_{t-1 }, e_{t-2},\dots \right\}$$

If $y_{t+1}$ represents log returns of an asset, then Model 1 contains the special case of a driftless random walk model for the natural logarithm of asset prices.

Notice that we are using the index t+1 to denote exogenous variables known at time t. Thus, $X_{t+1}$ is a vector containing known variables at time t. Notice also that we will use notations $\mathbb{E}\left(e_{t+1}|{\mathfrak{F}}_t\right)$ or ${\mathbb{E}}_t\left(e_{t+1}\right)$. The alternative model posits that the conditional expectation of $y_{t+1}$ with respect to the filtration ${\mathfrak{F}}_t$ only depends linearly on the vector $X_{t+1}$ and an unknown vector of parameters $\beta$.

$$\mathbb{E}\left(y_{t+1}|{\mathfrak{F}}_t\right)=X_{t+1}^{\prime }\beta$$

(3)

According to this condition, the null hypothesis may be written in terms of a restriction over the vector of parameters $\beta =0$ against the alternative $\beta \ne 0$.

Ref. [10] provide a very attractive out-of-sample test in a slightly different econometric context but that in principle might be used to test $\beta =0$ against $\beta \ne 0$. They introduce the Excess Profitability (EP) test, which is based on the idea that $y_{t+1}$ corresponds to an asset return. Given a forecast $y_{t,t+1}^f$ built on information available up until time t for the return $y_{t+1},$ they consider the following trading strategy:

• To buy shares of the asset if the forecast $y_{t,t+1}^f$ predicts nonnegative returns in the following period; that is to say that $y_{t,t+1}^f\ge 0$;
• To sell shares of the asset if the forecast $y_{t,t+1}^f$ predicts negative returns in the following period; that is to say that $y_{t,t+1}^f<0$;
• The one-period-return of this trading strategy is given by $r_{t+1}=sign\left(y_{t,t+1}^f\right)y_{t+1}$, where the following is the case.

  $$sign\left(z\right)=\left\{\begin{array}{c}1 if z\ge 0\\ -1 if z<0\end{array}\right.$$

Based upon this one-period-return, [10] construct an asymptotically normal statistic under the null hypothesis of conditional mean independence:

$$A–G \mathrm{null} \mathrm{hypothesis}: \mathbb{E}\left(y_{t+1}|{\Im }_t\right)=\mathbb{E}\left(y_{t+1}\right)=\mu$$

(4)

where ${\Im }_t$ represents the information set available to the forecaster at time t. This asymptotically normal statistic is given by the following:

$$\mathrm{EP}\equiv \frac{A_T-B_T}{\sqrt{\widehat{V}\left(A_T-B_T\right)}}$$

(5)

where the following is the case:

$$A_T=\frac{1}{T}{{\displaystyle \sum }}_t{r}_{t+1}$$

(6)

$$B_T=\left(\frac{1}{T}{{\displaystyle \sum }}_{t}sign\left(y_{t,t+1}^f\right)\right)\left(\frac{1}{T}{{\displaystyle \sum }}_t{y}_{t+1}\right)$$

(7)

and $\widehat{V}\left(A_T-B_T\right)$ represents a consistent estimator of the Variance of $A_T-B_T$ that is computed according to the following expressions.

$${\widehat{V}}_{EP}\equiv \widehat{V}\left(A_T-B_T\right)=\frac{4}{T^2}\widehat{p_{y^f}}\left(1-\widehat{p_{y^f}}\right){{\displaystyle \sum }}_t{\left(y_{t+1}-\overline{y}\right)}^2$$

(8)

$$\widehat{p_{y^f}}=\frac{1}{2}\left[1+\left(\frac{1}{T}{{\displaystyle \sum }}_{t}sign\left(y_{t,t+1}^f\right)\right)\right]$$

(9)

The rationale for using these two expressions above comes from the fact that under the null hypothesis (4) and some strong assumptions of independence (see [10] for details), we have the following.

$$V\left(A_T-B_T\right)=\frac{4\left(T-1\right)}{T^2}{p}_{y^f}\left(1-p_{y^f}\right)var\left(y_{t+1}\right)$$

$$p_{y^f}=\mathrm{Pr}\left(sign\left(y_{t,t+1}^f\right)=1\right)$$

The intuition behind the definition of the EP test relies on the fact that under the null of no predictability, both $A_T$ and $B_T$ converge in probability to the same value; thus, they are asymptotically equal.

It is tempting to export the EP test to our econometric framework. In fact, our null hypothesis coincides with (4) when we consider the same information sets ${\mathfrak{F}}_t={\Im }_t$ and $\mu =0$. Nevertheless, when $\mu =0$ and no additional assumptions of independence are imposed we will show that the core EP statistic may be biased under the null hypothesis when some usual out-of-sample methods are implemented. We observe this next.

3. Bias in the Core EP Statistic

Traditional out-of-sample analyses must split the available sample into two windows: an estimation window and a predictive window. For the sake of simplicity, we will focus on one-step-ahead forecasts, and we will use the notation in [2,17]. If we have a total of T + 1 observations, then the last P will be used for predictions and $R=T+1-P$ will be used for the initial estimation of the parameters. $\widehat{{\beta }_t}$ denotes a generic estimate of $\beta$ with information available until time t. Usually, the updating scheme may be either fixed, rolling or recursive. The fixed scheme is one in which $\widehat{{\beta }_t}$ is estimated only once using the first R observations. Thus, no updating occurs here. The rolling scheme updates the estimate of $\widehat{{\beta }_t}$ using always the last R available observations.

The recursive scheme also updates the estimate of $\widehat{{\beta }_t}$, but this time using all available information until time t. That is to say that, in the recursive scheme, the estimation sample increases with t. We will work with both the rolling and recursive scheme because they are the most used in empirical applications. It is important to mention that the asymptotic theory in [2] is only compatible with the rolling scheme, yet via simulations the same authors show that their test works well with the recursive scheme, which is also commonly used in applied research.

Let us analyze the behavior of the core EP statistic under the null hypothesis. Under our notation, we will now have a total of P forecasts for evaluation. The EP test should be written according to this notation as follows:

$$\mathrm{EP}\equiv \frac{A_P-B_P}{\sqrt{\widehat{V}\left(A_P-B_P\right)}}$$

(10)

where the following is the case.

$$A_P=\frac{1}{P}{{\displaystyle \sum }}_{t=R+1}^{T+1}{r}_t$$

(11)

$$B_P=\left(\frac{1}{P}{{\displaystyle \sum }}_{t=R+1}^{T+1}sign(y_{t-1,t}^f)\right)\left(\frac{1}{P}{{\displaystyle \sum }}_{t=R+1}^{T+1}{y}_t\right)$$

(12)

The next proposition shows the behavior of both terms, $A_P$ and $B_P$, of the core EP statistic under the null hypothesis.

Proposition 1.

In the econometric framework characterized by models (1) and (2), we have the following under the null hypothesis.

$$\begin{gathered} \mathbb{E}\left(A_P\right)=0 \\ \mathbb{E}\left(B_P\right)=\frac{1}{P^2}\left\{{\displaystyle \sum }_{t>j}^{T+1}\mathbb{E}(sign\left(y_{t-1,t}^f\right)y_j)\right\} \end{gathered}$$

This last term may or may not be zero.

Proof. 

See Proposition 1 in Appendix A. □

Proposition 1 tells us that, under the null hypothesis, the core statistic from the EP test might not be zero when there is dependence between future values of the forecasts and past values of the target variable. A simple example in which $\mathbb{E}\left(B_P\right)\ne 0$ is given next.

Let us consider the simple case in which there is only one predictor under the alternative: a constant different from zero. Let us also assume that $e_{t+1}$ is not only a zero mean martingale difference sequence but also an independent sequence. Accordingly, here, we assume that the alternative model looks as follows.

$$\mathrm{Model} 2 \left(\mathrm{alternative}\right): y_{t+1}=c+e_{t+1}$$

(13)

Under this assumption, the best forecast is given by $y_{t-1,t}^f=c\ne 0$. If this constant c is unknown; it needs to be estimated. Let us assume that we use a rolling scheme to estimate c by OLS. Therefore, we have it that our feasible forecast corresponds to the following.

$${\widehat{y}}_{t-1,t}^f=\widehat{c_t}=\frac{1}{R}{\displaystyle \sum }_{i=t-R}^{t-1}{y}_i;t=R+1,\dots , T+1$$

In the very particular case when R = 1, we have the following.

$${\widehat{y}}_{t-1,t}^f=\widehat{c_t}=y_{t-1} ; t=2, \dots , T+1$$

Therefore, the following is the case.

$$\begin{gathered} \mathbb{E}\left(B_P\right)=\frac{1}{P^2}\left\{{\displaystyle \sum }_{t>j}^{T+1}\mathbb{E}(sign\left({\widehat{y}}_{t-1,t}^f\right)y_j\right\}=\frac{1}{P^2}\left\{{\displaystyle \sum }_{t>j}^{T+1}\mathbb{E}(sign\left(y_{t-1}\right)y_j\right\} \\ \stackrel{H_0}{=}\frac{1}{P^2}\left\{{\displaystyle \sum }_{t>j}^{T+1}\mathbb{E}(sign\left(e_{t-1}\right)e_j\right\} \\ =\frac{1}{P^2}{\displaystyle \sum }_{j=2}^{T+1}\mathbb{E}\left(e_{j}sign\left(e_j\right)\right)=\frac{\mathbb{E}\left|e_j\right|}{P}>0 \end{gathered}$$

In this case, the term under consideration is strictly positive, but it proceeds to zero as the number of forecasts increases.

This is just an example to show that this $\mathbb{E}\left(B_P\right)$ term might not be zero when the null hypothesis is true. We have used an extreme simplification in the context of a rolling estimation scheme to prove our point. Nevertheless, we should also mention that the term $\mathbb{E}\left(B_P\right)$ will be zero in some circumstances. For instance, this happens when using the fixed scheme and our alternative model is given by (13). Under these conditions, the forecast $y_{t-1,t}^f$ is constructed using information contained in ${\mathfrak{F}}_R$, and it is never updated again. In this case, we have the following expression of forecasts:

$${\widehat{y}}_{t-1,t}^f=\widehat{c_t}=\widehat{c_R} \mathrm{for} \mathrm{all} t=R+1, \dots , T+1$$

where $\widehat{c_R}$ is a function of the data $y_1, y_2, \dots , y_R$. A natural candidate for $\widehat{c_R}$ is the simple average.

$$\overline{c_R}=\frac{1}{R}{\displaystyle \sum }_{j=1}^R{y}_j$$

Therefore, the following is the case:

$$\begin{gathered} \mathbb{E}\left(B_P\right)=\frac{1}{P^2}\left\{{\displaystyle \sum }_{\begin{array}{c}t>j\\ j=R+1\end{array}}^{T+1}\mathbb{E}(sign\left({\widehat{y}}_{t-1,t}^f\right)y_j\right\}=\frac{1}{P^2}\left\{{\displaystyle \sum }_{\begin{array}{c}t>j\\ j=R+1\end{array}}^{T+1}\mathbb{E}\left(sign\left(\widehat{c_R}\right)y_j\right)\right\} \\ =\frac{1}{P^2}\left\{{\displaystyle \sum }_{\begin{array}{c}t>j\\ j=R+1\end{array}}^{T+1}\mathbb{E}\left\{\mathbb{E}(sign\left(\widehat{c_R}\right)y_j|{\mathfrak{F}}_R\right\}\right\}=\frac{1}{P^2}\left\{{\displaystyle \sum }_{\begin{array}{c}t>j\\ j=R+1\end{array}}^{T+1}\mathbb{E}\left(sign\left(\widehat{c_R}\right)\mathbb{E}\left\{y_j|{\mathfrak{F}}_R\right\}\right)\right\} \end{gathered}$$

but under the null hypothesis, we have the following:

$$\begin{gathered} \mathbb{E}\left(B_P\right)\stackrel{H_0}{=}\frac{1}{P^2}\left\{{\displaystyle \sum }_{\begin{array}{c}t>j\\ j=R+1\end{array}}^{T+1}\mathbb{E}(sign\left(\widehat{c_R}\right)\mathbb{E}\left\{y_j|{\mathfrak{F}}_R\right\})\right\}=\frac{1}{P^2}\left\{{\displaystyle \sum }_{\begin{array}{c}t>j\\ j=R+1\end{array}}^{T+1}\mathbb{E}\left(sign\left(\widehat{c_R}\right)\mathbb{E}\left\{e_j|{\mathfrak{F}}_R\right\}\right)\right\} \\ =\frac{1}{P^2}\left\{{\displaystyle \sum }_{\begin{array}{c}t>j\\ j=R+1\end{array}}^{T+1}\mathbb{E}\left(sign\left(\widehat{c_R}\right)0\right)\right\}=0 \end{gathered}$$

and the core test EP statistic is unbiased under the null.

In the next sections, we will observe that the bias of the test statistic may be important in more general examples than the extremely simple one we have described so far. Before that, we show next that for the null hypothesis in (1), we could construct an adjusted EP test, which is unbiased under the null. The idea is simply to remove the $B_P$ term from the construction of the test. The new test is called the Straightforward Excess Profitability (SEP) test.

4. The Straightforward Excess Profitability Test (SEP)

Here, we propose a simple modification of the EP test by removing the $B_P$ term from the EP test. In other words, we propose the construction of a test for the random walk hypothesis based on the following statistic.

$$A_P=\frac{1}{P}{{\displaystyle \sum }}_{t=R+1}^{T+1}{r}_t$$

(14)

$$r_{t+1}=sign\left(y_{t,t+1}^f\right)y_{t+1}$$

(15)

Assuming stationarity and ergodicity for $r_t$ we have the following under the null.

$$\begin{gathered} H_0:\mathbb{E}\left[r_{t+1}\right]=\mathbb{E}\left[\mathbb{E}\left[r_{t+1}|{\mathfrak{F}}_t\right]\right]=\mathbb{E}\left[\mathbb{E}\left[sign\left(y_{t,t+1}^f\right)e_{t+1}|{\mathfrak{F}}_t\right]\right] \\ =\mathbb{E}\left[sign\left(y_{t,t+1}^f\right)\mathbb{E}\left[e_{t+1}|{\mathfrak{F}}_t\right]\right]=0 \end{gathered}$$

This is because the following is the case.

$$sign\left(y_{t,t+1}^f\right) \in {\mathfrak{F}}_t$$

Under the alternative, we have the following.

$$\begin{gathered} H_A:\mathbb{E}\left[r_{t+1}\right]=\mathbb{E}\left[\mathbb{E}\left[r_{t+1}|{\mathfrak{F}}_t\right]\right]=\mathbb{E}\left[\mathbb{E}\left[sign\left(y_{t,t+1}^f\right)y_{t+1}|{\mathfrak{F}}_t\right]\right] \\ \mathbb{E}\left[\mathbb{E}\left[sign\left(y_{t,t+1}^f\right)\left[X_{t+1}^{\prime }\beta +e_{t+1}\right]|{\mathfrak{F}}_t\right]\right]= \\ \mathbb{E}\left[\mathbb{E}\left[sign\left(y_{t,t+1}^f\right)\left[X_{t+1}^{\prime }\beta \right]|{\mathfrak{F}}_t\right]\right]+\mathbb{E}\left[sign\left(y_{t,t+1}^f\right)\mathbb{E}\left[e_{t+1}|{\mathfrak{F}}_t\right]\right] \\ \mathbb{E}\left[sign\left(y_{t,t+1}^f\right)\left[X_{t+1}^{\prime }\beta \right]\right]=\mathbb{E}\left[sign\left(X_{t+1}^{\prime }\widehat{{\beta }_t}\right)\left[X_{t+1}^{\prime }\beta \right]\right] \end{gathered}$$

As long as $\widehat{{\beta }_t}$ is an adequate estimate of $\beta$, then we obtain the following.

$$H_A: \mathbb{E}\left[sign\left(X_{t+1}^{\prime }\widehat{{\beta }_t}\right)\left[X_{t+1}^{\prime }\beta \right]\right]\approx \mathbb{E}\left[\left|X_{t+1}^{\prime }\beta \right|\right]>0$$

(16)

Under the null, $r_t$ is a martingale difference sequence; thus, under mild assumptions we could use the central limit theorem for martingale difference sequences (see Hamilton, 1994 [24]) to obtain the following:

$$\sqrt{P}{A}_P=\frac{\sqrt{P}}{P}{\displaystyle \sum }_{t=R+1}^{T+1}{r}_t⇝N\left(0,V\right)$$

where the following is the case.

$$V=\underset{P\to \infty }{\lim }\frac{1}{P}{\displaystyle \sum }_{t=R+1}^{T+1}\mathbb{E}{[r_t]}^2$$

We could then construct a consistent estimate of V, which we denote $\widehat{V}$, to build the following statistic.

$$SEP\equiv \frac{\frac{\sqrt{P}}{P} {{\displaystyle \sum }}_{t=R+1}^{T+1}{r}_t}{\sqrt{\widehat{V}}}⇝N\left(0,1\right)$$

Such a consistent estimator $\widehat{V }$ could be given by the following.

$$V=\frac{1}{P}{\displaystyle \sum }_{t=R+1}^{T+1}{r}_t^2$$

Considering (16), we will construct a one side test, as under the alternative we expect a positive value for our statistic.

In the next section, we compare size and power properties of EP and SEP tests via simulations. We also include in the comparison the widely cited test developed by [2] in order to have an external and simple benchmark.

5. Monte Carlo Simulations

We use Monte Carlo simulations for three multivariate data-generating processes (DGPs) to evaluate the finite-sample properties of the SEP test. The idea is to compare properties of this test with those of some other tests already available in the literature. We consider two benchmark tests: the traditional EP test and the MSPE-Adjusted test introduced by [2]. This last test is derived from typical comparisons of MSPE between a linear model and the null of a martingale difference sequence. Clark and West claim that the MSPE-Adjusted test has a better size than traditional tests of MSPE comparisons when the models under evaluation are nested. Intuitively, this test shows good size because it does not consider a term that introduces noise into its forecasts by estimating a parameter vector that under the null should be zero. To observe this, we notice that the sample analog of the difference in MSPE between the two models considered in (1) and (2) is given by the following.

$$\begin{gathered} \Delta \widehat{MSPE}={\widehat{MSPE}}_1-{\widehat{MSPE}}_2 \\ =\frac{1}{P}{\displaystyle \sum }_{t=R}^T{\left(y_{t+1}\right)}^2-\frac{1}{P}{\displaystyle \sum }_{t=R}^T{\left(y_{t+1}-X_{t+1}^{\prime }\widehat{{\beta }_t}\right)}^2 \\ =\frac{2}{P}{\displaystyle \sum }_{t=R}^T{y}_{t+1}{X}_{t+1}^{\prime }\widehat{{\beta }_t}-\frac{1}{P}{\displaystyle \sum }_{t=R}^T{\left(X_{t+1}^{\prime }\widehat{{\beta }_t}\right)}^2 \end{gathered}$$

Clark and West notice that the second term in the right-hand side introduces a bias that does not vanish as P goes to $\infty$. They propose to build a test based on the first term in the right-hand side. Their test is given by the following.

$$MSPE-Adjusted: P^{1/2}\frac{\frac{2}{P}{{\displaystyle \sum }}_{t=R}^T{y}_{t+1}{X}_{t+1}^{\prime }\widehat{{\beta }_t}}{\sqrt{4\widehat{V}\left(y_{t+1}{X}_{t+1}^{\prime }\widehat{{\beta }_t}\right)}}$$

They also show that, under the null hypothesis and mild assumptions, their test is asymptotically normal; thus, the three tests we consider here are asymptotically normal.

We use DGPs based on [2] and [10]. These DGPs are calibrated to match features of exchange rate and asset returns for which the driftless random walk is a plausible null model. Specifically, the first DGP (DGP1) is given by the following process:

DGP1:

$$y_{t+1}=\beta x_t+e_{t+1}$$

$$x_t=0.95x_{t-1}+u_t;$$

$$e_{t+1}=N\left(0,{\sigma }_e^2\right); u_{t+1}=N\left(0,{\sigma }_u^2\right)$$

with $\mathbb{E}\left(e_{t+1}|{\mathfrak{F}}_t\right)=0, \mathbb{E}\left(u_{t+1}|{\mathfrak{F}}_t\right)=0$ and ${\sigma }_e^2=1$.

DGP1 is calibrated to match exchange rate features based on interest parity; thus, we will have $var\left(u_t\right)={\sigma }_u^2$ (with ${\sigma }_u=0.025$) and $corr\left(e_t,u_t\right)=0$. We set $\beta =-2$ in experiments evaluating power and $\beta =0$ in experiments evaluating size. We assume data to be generated from homoscedastic draws of their respective distributions. We are aware that β = 1 is the theoretical implication. Empirical estimations for a number of industrial countries, however, provide estimates around −2 (see [2] for further details).

The next two DGPs are simple processes calibrated to match S&P 500 returns and are identical to those used by [10].

DGP2:

This is a simple first order autoregressive model for S&P returns.

$$y_t=\rho y_{t-1}+{\epsilon }_t$$

$${\epsilon }_t\sim N\left(0,0.000249\right)$$

We set $\rho =0.1256$ for power results and $\rho =0$ for size results.

DGP3:

This DGP is similar to DGP2, but now this process includes a threshold parameter to simulate possible breaks in the data.

$$y_t=\left\{\begin{array}{c}{c}_1+{\rho }_1{y}_{t-1}+{\epsilon }_t\to \left|y_{t-1}\right|\le 0.01848\\ c_2+{\rho }_2{y}_{t-1}+{\epsilon }_t\to \left|y_{t-1}\right|>0.01848\end{array}\right\}$$

$${\epsilon }_t\sim N\left(0,0,000245\right)$$

We set $c_1={\rho }_1=c_2={\rho }_2=0$ when analyzing the size of the tests. For power and following [10], we set the following.

$$c_1=0.000844; {\rho }_1=0.2453; c_2=0.002679; {\rho }_2=0.0664$$

For all simulations, we focus only on one-step-ahead forecasts. Estimation always includes a constant term in each regression. We explore the performance of rolling and recursive schemes for a number of sample sizes (T + 1) and decompositions of the sample in the estimation window (size R) and the prediction window (size P, (T + 1 = R + P)). We run simulations for the following sample sizes: R = 25, 50 and 100; P = 25, 50, 100, 300 and 750. The total number of simulations is 5000. We use OLS to obtain estimates of the parameters. The main features arising from the simulations with DGP1, DGP2 and DGP3 are analyzed in the following subsection. Notice that results for DGP 3 are mostly used as a robustness check. We perform this because, for this DGP, we estimate an AR(1) model with drift; thus, the threshold parameter is omitted in the estimation and a misspecification error is present.

5.1. Simulation Results

Table 1. Empirical Size DGP1; Nominal size is 10%.

	Rolling							Recursive
	P = 25	P = 50	P = 100	P = 300	P = 750			P = 25	P = 50	P = 100	P = 300	P = 750
	R = 25							R = 25
CW	0.074	0.082	0.084	0.078	0.090		CW	0.075	0.068	0.070	0.056	0.063
SEP	0.101	0.103	0.101	0.102	0.101		SEP	0.102	0.100	0.105	0.104	0.096
EP	0.045	0.052	0.059	0.075	0.086		EP	0.035	0.037	0.046	0.043	0.035
	R = 50							R = 50
CW	0.071	0.069	0.069	0.076	0.089		CW	0.075	0.067	0.061	0.060	0.065
SEP	0.101	0.094	0.097	0.098	0.108		SEP	0.100	0.095	0.106	0.093	0.100
EP	0.036	0.042	0.049	0.064	0.083		EP	0.029	0.039	0.048	0.036	0.036
	R = 100							R = 100
CW	0.084	0.069	0.067	0.068	0.076		CW	0.080	0.071	0.067	0.060	0.060
SEP	0.115	0.105	0.100	0.096	0.096		SEP	0.110	0.106	0.102	0.098	0.101
EP	0.035	0.040	0.051	0.054	0.063		EP	0.031	0.037	0.044	0.046	0.044

Notes: CW stands for the [2] test. EP stands for Excess Profitability test by [10]. SEP is our Straightforward Excess Profitability test. R stands for the length of the initial estimation window, and P is the number of forecasts. Each entry reports the rate of rejections with a nominal size of 10% and 5000 Monte Carlo simulations.

and

Table 2. Empirical Size DGP2, Nominal size is 10%.

	Rolling							Recursive
	P = 25	P = 50	P = 100	P = 300	P = 750			P = 25	P = 50	P = 100	P = 300	P = 750
	R = 25							R = 25
CW	0.079	0.075	0.089	0.083	0.093		CW	0.081	0.070	0.065	0.061	0.071
SEP	0.107	0.102	0.108	0.099	0.100		SEP	0.107	0.105	0.106	0.104	0.102
EP	0.051	0.052	0.056	0.070	0.082		EP	0.049	0.049	0.043	0.041	0.040
	R = 50							R = 50
CW	0.089	0.072	0.083	0.079	0.079		CW	0.087	0.074	0.079	0.063	0.062
SEP	0.108	0.099	0.106	0.105	0.101		SEP	0.104	0.101	0.104	0.106	0.099
EP	0.056	0.047	0.048	0.064	0.073		EP	0.057	0.048	0.044	0.046	0.038
	R = 100							R = 100
CW	0.087	0.083	0.072	0.069	0.084		CW	0.089	0.085	0.071	0.060	0.065
SEP	0.101	0.103	0.106	0.094	0.101		SEP	0.102	0.109	0.107	0.103	0.102
EP	0.057	0.054	0.054	0.048	0.062		EP	0.055	0.052	0.049	0.045	0.045

Notes: See notes of Table 1.

display results of empirical size for the three tests under consideration for DGP1 and DGP2, respectively.

Table 3. Raw Power DGP1 (Nominal Size of 10%).

	Rolling							Recursive
	P = 25	P = 50	P = 100	P = 300	P = 750			P = 25	P = 50	P = 100	P = 300	P = 750
	R = 25							R = 25
CW	0.141	0.188	0.244	0.413	0.658		CW	0.152	0.207	0.318	0.670	0.954
SEP	0.183	0.219	0.264	0.406	0.598		SEP	0.183	0.246	0.337	0.646	0.941
EP	0.067	0.104	0.154	0.327	0.541		EP	0.066	0.109	0.194	0.532	0.901
	R = 50							R = 50
CW	0.175	0.228	0.303	0.525	0.813		CW	0.185	0.257	0.361	0.703	0.967
SEP	0.211	0.262	0.317	0.483	0.726		SEP	0.209	0.274	0.371	0.664	0.949
EP	0.068	0.113	0.178	0.381	0.668		EP	0.063	0.118	0.222	0.549	0.915
	R = 100							R = 100
CW	0.206	0.272	0.384	0.641	0.894		CW	0.218	0.287	0.418	0.754	0.975
SEP	0.224	0.285	0.381	0.594	0.821		SEP	0.232	0.292	0.401	0.694	0.952
EP	0.075	0.129	0.231	0.474	0.761		EP	0.075	0.138	0.260	0.601	0.923

Notes: See notes of Table 1.

,

Table 4. Raw Power DGP2 (Nominal Size of 10%).

	Rolling							Recursive
	P = 25	P = 50	P = 100	P = 300	P = 750			P = 25	P = 50	P = 100	P = 300	P = 750
	R = 25							R = 25
CW	0.126	0.154	0.206	0.312	0.506		CW	0.138	0.174	0.242	0.507	0.845
SEP	0.155	0.165	0.199	0.271	0.405		SEP	0.161	0.186	0.245	0.479	0.817
EP	0.065	0.082	0.120	0.214	0.360		EP	0.063	0.095	0.153	0.394	0.764
	R = 50							R = 50
CW	0.159	0.181	0.237	0.404	0.647		CW	0.160	0.190	0.283	0.545	0.876
SEP	0.169	0.188	0.237	0.338	0.497		SEP	0.166	0.198	0.270	0.505	0.828
EP	0.083	0.101	0.144	0.261	0.435		EP	0.080	0.114	0.181	0.423	0.784
	R = 100							R = 100
CW	0.174	0.220	0.285	0.499	0.769		CW	0.181	0.235	0.323	0.611	0.907
SEP	0.176	0.217	0.279	0.428	0.638		SEP	0.181	0.226	0.298	0.537	0.852
EP	0.094	0.137	0.183	0.332	0.571		EP	0.097	0.143	0.217	0.466	0.810

Notes: See notes of Table 1.

and

Table 5. Raw Power DGP3 (Nominal Size of 10%).

	Rolling							Recursive
	P = 25	P = 50	P = 100	P = 300	P = 750			P = 25	P = 50	P = 100	P = 300	P = 750
	R = 25							R = 25
CW	0.158	0.195	0.263	0.440	0.689		CW	0.167	0.223	0.325	0.660	0.949
SEP	0.186	0.230	0.274	0.417	0.625		SEP	0.191	0.253	0.346	0.648	0.936
EP	0.070	0.095	0.129	0.223	0.373		EP	0.066	0.109	0.158	0.347	0.598
	R = 50							R = 50
CW	0.194	0.236	0.313	0.542	0.825		CW	0.203	0.259	0.379	0.699	0.966
SEP	0.216	0.262	0.334	0.503	0.750		SEP	0.222	0.279	0.386	0.672	0.941
EP	0.079	0.107	0.150	0.270	0.456		EP	0.078	0.124	0.184	0.367	0.614
	R = 100							R = 100
CW	0.229	0.281	0.378	0.665	0.907		CW	0.236	0.299	0.426	0.766	0.974
SEP	0.238	0.289	0.395	0.615	0.850		SEP	0.242	0.300	0.408	0.713	0.944
EP	0.091	0.129	0.187	0.317	0.553		EP	0.094	0.139	0.213	0.406	0.625

Notes: See notes of Table 1.

exhibit results of raw power for DGP1 through DGP3, respectively. Notice that we do not report size results for DGP3 since, under the null hypothesis, this DGP is almost identical to DGP2 and size results are, consequently, almost identical as well. DGP3 is estimated as a simple first-order autoregressive process, and under the null hypothesis, this DGP is white noise, but under the alternative, it is a SETAR model.

Each table exhibits our results for both the rolling and the recursive scheme. While Table 1, Table 2, Table 3, Table 4 and Table 5 consider a nominal size of 10%,

Table A1. Empirical Size DGP1 (Nominal Size of 5%).

	Rolling							Recursive
	P = 25	P = 50	P = 100	P = 300	P = 750			P = 25	P = 50	P = 100	P = 300	P = 750
	R = 25							R = 25
CW	0.039	0.043	0.040	0.039	0.043		CW	0.038	0.037	0.036	0.026	0.029
SEP	0.056	0.051	0.050	0.048	0.051		SEP	0.053	0.050	0.056	0.048	0.045
EP	0.022	0.023	0.028	0.033	0.041		EP	0.016	0.019	0.021	0.019	0.015
	R = 50							R = 50
CW	0.037	0.033	0.034	0.037	0.050		CW	0.036	0.034	0.030	0.029	0.030
SEP	0.053	0.050	0.050	0.051	0.056		SEP	0.052	0.051	0.053	0.048	0.051
EP	0.018	0.020	0.024	0.030	0.039		EP	0.014	0.018	0.024	0.020	0.017
	R = 100							R = 100
CW	0.044	0.036	0.033	0.032	0.038		CW	0.043	0.036	0.032	0.025	0.027
SEP	0.059	0.050	0.051	0.046	0.047		SEP	0.057	0.051	0.052	0.046	0.049
EP	0.016	0.019	0.023	0.023	0.030		EP	0.016	0.017	0.022	0.020	0.020

Notes: CW stands for the [2] test. EP stands for the Excess Profitability test by [10]. SEP is our Straightforward Excess Profitability test. R stands for the length of the initial estimation window, and P is the number of forecasts. Each entry reports the rate of rejections with a nominal size of 5% and 5000 Monte Carlo simulations.

,

Table A2. Empirical Size DGP2 (Nominal Size of 5%).

Rolling							Recursive
P = 25	P = 50	P = 100	P = 300	P = 750			P = 25	P = 50	P = 100	P = 300	P = 750
R = 25							R = 25
0.040	0.036	0.040	0.042	0.044		CW	0.041	0.035	0.035	0.032	0.033
0.054	0.053	0.052	0.052	0.049		SEP	0.052	0.051	0.052	0.050	0.052
0.023	0.023	0.025	0.033	0.038		EP	0.021	0.025	0.020	0.018	0.020
R = 50							R = 50
0.044	0.033	0.042	0.036	0.041		CW	0.041	0.036	0.041	0.029	0.031
0.059	0.050	0.055	0.046	0.051		SEP	0.055	0.050	0.057	0.052	0.048
0.028	0.022	0.024	0.025	0.037		EP	0.024	0.021	0.021	0.022	0.016
R = 100							R = 100
0.045	0.042	0.035	0.034	0.038		CW	0.044	0.045	0.038	0.032	0.033
0.051	0.051	0.054	0.046	0.051		SEP	0.056	0.053	0.055	0.050	0.052
0.025	0.025	0.026	0.020	0.028		EP	0.027	0.023	0.025	0.023	0.020

Notes: See notes of Table A1.

,

Table A3. Raw Power DGP1 (Nominal Size of 5%).

	Rolling							Recursive
	P = 25	P = 50	P = 100	P = 300	P = 750			P = 25	P = 50	P = 100	P = 300	P = 750
	R = 25							R = 25
CW	0.086	0.114	0.160	0.299	0.532		CW	0.095	0.137	0.221	0.553	0.923
SEP	0.109	0.138	0.170	0.278	0.466		SEP	0.116	0.153	0.235	0.516	0.891
EP	0.033	0.055	0.092	0.211	0.409		EP	0.034	0.058	0.123	0.410	0.842
	R = 50							R = 50
CW	0.105	0.147	0.206	0.406	0.716		CW	0.112	0.170	0.258	0.603	0.944
SEP	0.129	0.172	0.216	0.366	0.611		SEP	0.128	0.173	0.257	0.542	0.905
EP	0.035	0.063	0.110	0.275	0.546		EP	0.033	0.064	0.146	0.429	0.862
	R = 100							R = 100
CW	0.130	0.174	0.275	0.534	0.837		CW	0.131	0.186	0.305	0.657	0.956
SEP	0.142	0.187	0.266	0.471	0.729		SEP	0.147	0.193	0.288	0.581	0.910
EP	0.042	0.076	0.151	0.366	0.665		EP	0.041	0.086	0.167	0.486	0.875

Notes: See notes of Table A1.

,

Table A4. Raw Power DGP2 (Nominal Size of 5%).

	Rolling							Recursive
	P = 25	P = 50	P = 100	P = 300	P = 750			P = 25	P = 50	P = 100	P = 300	P = 750
	R = 25							R = 25
CW	0.072	0.090	0.129	0.214	0.376		CW	0.080	0.100	0.157	0.389	0.776
SEP	0.090	0.093	0.122	0.170	0.277		SEP	0.094	0.111	0.150	0.352	0.719
EP	0.034	0.038	0.065	0.128	0.242		EP	0.033	0.054	0.087	0.277	0.660
	R = 50							R = 50
CW	0.094	0.106	0.151	0.286	0.523		CW	0.094	0.119	0.182	0.423	0.810
SEP	0.101	0.110	0.156	0.229	0.364		SEP	0.100	0.115	0.173	0.364	0.733
EP	0.042	0.053	0.087	0.172	0.317		EP	0.042	0.058	0.110	0.299	0.683
	R = 100							R = 100
CW	0.101	0.129	0.182	0.378	0.663		CW	0.103	0.145	0.210	0.483	0.845
SEP	0.108	0.136	0.168	0.303	0.517		SEP	0.107	0.139	0.196	0.401	0.763
EP	0.045	0.075	0.108	0.230	0.447		EP	0.046	0.081	0.141	0.352	0.722

Notes: See notes of Table A1.

and

Table A5. Raw Power DGP3 (Nominal Size of 5%).

	Rolling							Recursive
	P = 25	P = 50	P = 100	P = 300	P = 750			P = 25	P = 50	P = 100	P = 300	P = 750
	R = 25							R = 25
CW	0.095	0.119	0.172	0.323	0.571		CW	0.106	0.143	0.222	0.556	0.920
SEP	0.120	0.142	0.183	0.303	0.502		SEP	0.119	0.164	0.237	0.528	0.887
EP	0.033	0.048	0.074	0.139	0.257		EP	0.029	0.056	0.096	0.255	0.508
	R = 50							R = 50
CW	0.119	0.151	0.221	0.430	0.740		CW	0.127	0.171	0.264	0.596	0.941
SEP	0.130	0.176	0.237	0.389	0.642		SEP	0.138	0.181	0.268	0.552	0.891
EP	0.037	0.059	0.085	0.175	0.331		EP	0.040	0.066	0.113	0.269	0.527
	R = 100							R = 100
CW	0.144	0.179	0.274	0.546	0.851		CW	0.146	0.197	0.308	0.660	0.953
SEP	0.153	0.192	0.281	0.491	0.775		SEP	0.153	0.199	0.293	0.586	0.899
EP	0.044	0.069	0.120	0.218	0.430		EP	0.047	0.080	0.134	0.307	0.544

Notes: See notes of Table A1.

in the Appendix A section consider a nominal size of 5% with similar conclusions. Each panel in Table 1, Table 2, Table 3, Table 4 and Table 5 considers a different length for the initial estimation window R (either R = 25, 50 or 100).

Table 1 and Table 2 show similar results. In general, the original EP seems to be remarkably undersized, while the SEP test has better size properties than both the EP and MSPE-Adjusted tests. For instance, in Table 1, EP exhibits an average empirical size of 3.9% for the recursive scheme, ranging from 2.9% to 4.8% (i.e., the maximum empirical size is less than half the nominal size). In contrast, the SEP test has an average empirical size of 10.1%, ranging from 9.3 to 11.0%. Across the two DGPs, considering the rolling (recursive) scheme: the average empirical size for the EP test is 5.7% (4.3%), while for the SEP and CW tests they are 10.2% (10.3%) and 7.9% (7.0%), respectively.

In terms of power, Table 3, Table 4 and Table 5 suggest that for small P, the SEP test has much more power than the EP test and a little more power than the MSPE-Adjusted test. As the number of forecasts becomes larger, we still observe a better performance of the SEP test over the EP test. Nevertheless, the MSPE-Adjusted test dominates the rest of the tests. The size of the estimation window does not seem to have a significant impact on these conclusions. Nevertheless, it does have an impact on absolute magnitudes: we can observe that with a larger estimation window, all three tests display higher power.

All in all, Table 1, Table 2, Table 3, Table 4 and Table 5 display a similar message: First, the SEP test dominates the EP test both in terms of size and power. Second, the MSPE-Adjusted test has no better empirical size compared to the SEP test. Third, the MSPE-Adjusted test is the best in terms of power when the number of forecasts is moderate or large, and fourth, with a larger estimation window, all three tests display higher power.

5.2. Robustness

Thus far, we have focused on the null hypothesis of a driftless random walk. Yet, in practice, the returns of many assets may have expected values that are close to zero but are not exactly zero.

Table A6. Summary statistics of several monthly returns.

	ALUMINUM	AUSTRALIA	CANADA	CHILE	COPPER	LEAD	LMEX	NEWZEALAND	NICKEL	SOUTHAFRICA	TIN
Mean	0.0008	−0.0004	−0.0005	0.0015	0.0055	0.0053	0.0036	−0.0009	0.0032	0.0040	0.0049
Median	−0.0016	−0.0022	−0.0006	0.0002	0.0072	0.0084	0.0051	−0.0025	0.0023	0.0006	0.0037
Std. Dev.	0.0576	0.0360	0.0257	0.0340	0.0737	0.0853	0.0605	0.0373	0.1011	0.0477	0.0686
Variance	0.0033	0.0013	0.0007	0.0012	0.0054	0.0073	0.0037	0.0014	0.0102	0.0023	0.0047
t-stat	0.1890	−0.1540	−0.2860	0.7110	0.9690	0.9830	0.7510	−0.3650	0.4560	1.3440	0.9230
Obs	251	251	251	251	251	251	251	251	251	251	251

in the Appendix A shows descriptive statistics of several monthly returns of commodities and exchange rates. The null hypothesis of zero returns cannot be rejected in any asset at usual significance levels, yet numerically, mean and median returns are slightly different from zero. With this in mind, we explore next the behavior of our proposed SEP test in situations in which we face small deviations from zero mean returns. To that end we consider the following DGP 4 inspired in [9]:

$$\begin{gathered} Y_{t+1}=c+\beta X_t+{\epsilon }_{t+1} \\ {X}_{t+1}=\alpha +\rho X_t+v_{t+1} \end{gathered}$$

where ${\epsilon }_{t+1}$ and $v_{t+1}$ are white noise processes:

$$\begin{gathered} Y_{t+1}=\Delta log\left(Aluminu{m}_{t+1}\right) \\ {X}_{t+1}=\Delta log\left(Australi{a}_{t+1}\right) \end{gathered}$$

and $Australi{a}_{t+1}$ denotes the Australian dollar at time t + 1.

By using monthly data from 31 September 1999 to August 2020, we calibrate our parameters as follows.

$$c=0.0008, {\sigma }_{\epsilon }=0.058, {\sigma }_v=0.036, \rho =0.0375, \alpha =-0.0004, Corr\left({\epsilon }_t;v_t\right)=-0.49$$

Under the null, we set $\beta =0$. Under the alternative, we set it to $\beta =-0.27$. Notice also that under the null, the expected value of $Y_{t+1}$ is set to $c=0.0008\ne 0$.

Table 6. Empirical Size DGP4, Nominal size is 10%.

	Rolling							Recursive
	P = 25	P = 50	P = 100	P = 300	P = 750			P = 25	P = 50	P = 100	P = 300	P = 750
	R = 25							R = 25
CW	0.085	0.075	0.074	0.090	0.094		CW	0.083	0.075	0.067	0.073	0.073
SEP	0.106	0.099	0.098	0.109	0.101		SEP	0.100	0.105	0.106	0.116	0.112
EP	0.054	0.045	0.050	0.077	0.079		EP	0.048	0.044	0.046	0.037	0.033
	R = 50							R = 50
CW	0.081	0.074	0.072	0.081	0.094		CW	0.081	0.073	0.068	0.070	0.069
SEP	0.103	0.102	0.102	0.108	0.110		SEP	0.103	0.102	0.100	0.105	0.114
EP	0.053	0.053	0.044	0.063	0.076		EP	0.050	0.052	0.049	0.040	0.040
	R = 100							R = 100
CW	0.102	0.081	0.073	0.075	0.086		CW	0.098	0.081	0.074	0.073	0.073
SEP	0.114	0.111	0.107	0.109	0.108		SEP	0.111	0.110	0.106	0.111	0.124
EP	0.064	0.057	0.048	0.050	0.064		EP	0.061	0.053	0.050	0.045	0.042

Notes: See notes of Table 1.

and

Table 7. Raw Power DGP4 (Nominal Size of 10%).

	Rolling							Recursive
	P = 25	P = 50	P = 100	P = 300	P = 750			P = 25	P = 50	P = 100	P = 300	P = 750
	R = 25							R = 25
CW	0.179	0.211	0.289	0.512	0.780		CW	0.183	0.254	0.376	0.755	0.985
SEP	0.179	0.205	0.264	0.417	0.628		SEP	0.188	0.238	0.355	0.700	0.968
EP	0.111	0.127	0.183	0.355	0.582		EP	0.123	0.171	0.280	0.631	0.950
	R = 50							R = 50
CW	0.200	0.254	0.370	0.637	0.896		CW	0.211	0.293	0.445	0.797	0.987
SEP	0.201	0.245	0.339	0.515	0.768		SEP	0.200	0.269	0.392	0.736	0.976
EP	0.140	0.168	0.244	0.442	0.722		EP	0.140	0.205	0.326	0.676	0.966
	R = 100							R = 100
CW	0.251	0.323	0.445	0.743	0.957		CW	0.254	0.337	0.490	0.844	0.993
SEP	0.229	0.292	0.400	0.645	0.891		SEP	0.231	0.300	0.422	0.762	0.977
EP	0.166	0.225	0.326	0.566	0.860		EP	0.178	0.243	0.375	0.721	0.969

Notes: See notes of Table 1.

next show results of size and power for SEP, EP and CW tests in 5000 replications based on the previous DGP. As usual, we report results for different values of R and P and for recursive and rolling windows.

Interestingly, the general picture looks the same as before, and our proposed SEP test is better sized and more powerful than the original EP test. The main difference now occurs for large R and P in which our proposed SEP test is slightly oversized. This is expected, as with a large number of observations our SEP test is subtle enough to detect deviations from zero mean returns.

We also carried out simulations for returns displaying means far from zero. In these conditions, our proposed test is importantly oversized, and from that point of view it would be better to use the original EP test. Other than that, for zero mean returns or close to zero mean returns, our proposed SEP test is preferable.

5.3. Bias Results

In previous sections we proved that the core EP statistic might have an expected value different from zero under the null hypothesis.

Table 8. Median of the Tests Statistics under the Null hypothesis: DGP1.

	Rolling							Recursive
	P = 25	P = 50	P = 100	P = 300	P = 750			P = 25	P = 50	P = 100	P = 300	P = 750
	R = 25							R = 25
Median (CW)	−0.259	−0.209	−0.134	−0.087	−0.074		Median (CW)	−0.306	−0.283	−0.242	−0.267	−0.264
Median (SEP)	−0.029	−0.006	0.019	0.000	0.006		Median (SEP)	−0.018	−0.033	0.020	0.009	−0.035
Median (EP)	−0.475	−0.377	−0.293	−0.148	−0.101		Median (EP)	−0.551	−0.532	−0.523	−0.521	−0.591
	R = 50							R = 50
Median (CW)	−0.259	−0.247	−0.187	−0.171	−0.087		Median (CW)	−0.317	−0.299	−0.283	−0.307	−0.278
Median (SEP)	−0.023	0.005	−0.002	−0.051	0.010		Median (SEP)	−0.027	−0.001	0.012	−0.010	0.008
Median (EP)	−0.507	−0.464	−0.394	−0.290	−0.131		Median (EP)	−0.518	−0.508	−0.485	−0.556	−0.543
	R = 100							R = 100
Median (CW)	−0.200	−0.276	−0.254	−0.223	−0.144		Median (CW)	−0.224	−0.290	−0.292	−0.306	−0.289
Median (SEP)	0.029	−0.028	−0.009	−0.035	−0.020		Median (SEP)	0.015	−0.019	−0.014	−0.002	−0.015
Median (EP)	−0.464	−0.420	−0.462	−0.331	−0.241		Median (EP)	−0.514	−0.470	−0.489	−0.508	−0.544

and

Table 9. Median of the Tests Statistics under the Null hypothesis: DGP2.

	Rolling							Recursive
	P = 25	P = 50	P = 100	P = 300	P = 750			P = 25	P = 50	P = 100	P = 300	P = 750
	R = 25							R = 25
Median (CW)	−0.232	−0.200	−0.142	−0.100	−0.060		Median (CW)	−0.289	−0.297	−0.303	−0.232	−0.236
Median (SEP)	−0.027	−0.030	0.003	−0.008	0.005		Median (SEP)	−0.044	−0.010	−0.034	0.004	0.016
Median (EP)	−0.501	−0.409	−0.307	−0.179	−0.119		Median (EP)	−0.465	−0.485	−0.542	−0.548	−0.594
	R = 50							R = 50
Median (CW)	−0.211	−0.219	−0.194	−0.099	−0.108		Median (CW)	−0.250	−0.252	−0.264	−0.296	−0.289
Median (SEP)	0.005	0.005	0.023	0.007	0.002		Median (SEP)	−0.005	0.001	0.047	−0.041	−0.003
Median (EP)	−0.327	−0.412	−0.402	−0.236	−0.161		Median (EP)	−0.364	−0.421	−0.467	−0.546	−0.585
	R = 100							R = 100
Median (CW)	−0.198	−0.209	−0.224	−0.196	−0.129		Median (CW)	−0.193	−0.243	−0.252	−0.313	−0.275
Median (SEP)	−0.007	0.001	−0.012	−0.040	0.022		Median (SEP)	0.008	0.005	0.007	−0.015	0.019
Median (EP)	−0.271	−0.356	−0.436	−0.372	−0.209		Median (EP)	−0.286	−0.342	−0.405	−0.508	−0.556

show the median for SEP, EP and MSPE-Adjusted statistics under the null. The idea is to explore whether this bias is present in our simulations. From Table 8 and Table 9, we observe that it is indeed present. In fact, we detect a negative bias for the EP test in all our simulations.

Interestingly, at least under the rolling scheme, this bias decreases (in absolute terms) as the number of forecasts grows larger. The MSPE-Adjusted statistic follows a similar pattern. As mentioned by [2] and Pincheira (2013) [25], this adjusted statistic also has a negative bias under the null hypothesis that decreases as the number of forecasts grows larger, at least under the rolling scheme. Finally, we observe that our proposed SEP test presents a negligible bias with no clear signs, which suggests that the modification we propose here is unbiased under the null hypothesis.

Figure 1 considers the EP and SEP statistics kernel densities using DGP1 with R = 200, P = 750 and a rolling scheme. Notice that this figure clearly illustrates the negative bias of the EP statistic compared to the SEP statistic.

In summary, our simulations show that both the EP test and MSPE-Adjusted test are undersized, while the SEP test presents an empirical size closer to nominal size. In addition, the SEP test presents lower bias under the null and more power than the EP test, although sometimes it is less powerful than the MSPE-Adjusted test.

6. Empirical Illustration

In this section, based on the commodity-currencies literature, we provide an empirical illustration in which we compare the SEP and the EP tests. Following [5], Chen, Rossi and Rogoff (2011) [26]; [7,8,9]; Pincheira, Hardy and Muñoz (2021) [27]; and [6], we evaluate the performance of five exchange rates from major commodity exporters (Australia, Canada, Chile, New Zealand and Norway) when predicting commodity prices. We consider five commodity series: West Texas Intermediate (WTI), heating oil, two-month WTI futures, copper and the London metal exchange index (LMEX). Similarly to [6], we have different data sources.

For the case of fuels, we download our series from the Energy Information Administration (EIA) database. The exchange rates, copper and LMEX are obtained from Thomson Reuters Datastream. In each case, we downloaded daily prices, which we converted to monthly frequencies by sampling from the last observation of each month. The sample period is from October 1999 to December 2019. Notice that the starting point of our database coincides with the time when monetary authorities in Chile decided to pursue a pure flotation regime for the exchange rate.

Our econometric model for one-step-ahead forecasts closely follows specification (1) in Table 1 of [6], page 5:

$$\begin{array}{ll}\Delta \ln \left(C{P}_t\right)=c+& {\beta }_1\left[\Delta \ln \left(E{R}_{t-1}\right)+\Delta \ln \left(E{R}_{t-2}\right)+\Delta \ln \left(E{R}_{t-3}\right)\right]\\ & +{\beta }_2\left[\Delta \ln \left(E{R}_{t-4}\right)+\Delta \ln \left(E{R}_{t-5}\right)+\Delta \ln \left(E{R}_{t-6}\right)\right]+\rho \Delta \ln \left(C{P}_{t-1}\right)+{\epsilon }_t\end{array}$$

where $C{P}_t$ stands for a generic commodity price, $E{R}_t$ stands for a generic exchange rate, $\Delta \ln ()$ is the logarithmic return and ${\epsilon }_t$ is an error term. Notice that this specification considers up to six lags of exchange rates to predict commodity returns. These lags are aggregated into two quarters, where each one is constructed by adding three subsequent monthly returns. In other words, we consider two different predictors, where each one of them is constructed by adding three monthly returns. Additionally, we consider a constant and an AR(1) component.

We evaluate out-of-sample the null hypothesis of a driftless random walk for the natural logarithm of commodity prices. In terms of our model, the null is $H_0:c={\beta }_1={\beta }_2=\rho =0$. The idea is to evaluate if the information added by a nonzero constant, the exchange rates and the AR(1) component is useful for improving the forecast of a model that simply predicts zero returns (the driftless random walk).

Table 10. Empirical Illustration Forecasting Commodity Prices with Commodity Currencies.

SEP
	WTI	Heating Oil	WTIF2	LMEX	Copper
CLP	3.499 ***	3.139 ***	3.340 ***	2.438 ***	1.901 **
AUD	0.826	1.124	1.024	0.125	1.369 *
CAD	0.700	1.646 **	−0.074	0.041	1.416 *
NZD	1.179	1.684 **	1.515 *	−0.050	1.392 *
NOK	1.690 **	2.056 **	1.634 *	1.580 *	1.253
EP
	WTI	Heating Oil	WTIF2	LMEX	Copper
CLP	3.520 ***	3.083 ***	3.334 ***	2.355 ***	1.722 **
AUD	0.705	0.994	0.896	−0.047	1.143
CAD	0.557	1.531 *	−0.325	−0.158	1.167
NZD	1.087	1.565 *	1.453 *	−0.311	1.161
NOK	1.625 *	1.972 **	1.551 *	1.446 *	0.996

Notes: CLP, AUD, CAD, NZD and NOK stand for Chilean peso, Australian Dollar, Canadian dollar, New Zealand dollar and Norwegian krone, respectively. * p < 10%, ** p < 5%, *** p < 1%.

next reports the SEP and EP statistics using recursive regressions estimated by OLS, with an initial estimation window of R = 45 observations. Three features of Table 10 are worth mentioning. First, consistent with the negative bias reported in our simulations, the SEP statistic is greater than the EP in 98% of our exercises. Second, SEP rejects the null hypothesis in three more entries than the EP. In particular, the EP rejects the null hypothesis in 48% of our exercises, while the SEP rejects the null in 56% of the entries at usual significance levels. Finally, SEP not only rejects the null more frequently but also at tighter significance levels. Considering a 5% significance level, EP rejects the null in 24% of our exercises, while SEP rejects the null in 36% of our exercises, providing more evidence in favor of the predictability of commodity returns with commodity-currencies.

7. Conclusions

In this paper, we propose a new out-of-sample test of predictability that is constructed as a simple modification of the Excess Profitability (EP) test developed by [10]. Our proposed test is called Straightforward Excess Profitability (SEP) test and avoids the calculation of a term in the EP test that under the null of no predictability should be zero but in practice may be sizable. In addition, our test does not require the strong assumption of independence used to derive the EP test. We claim that dependence is the rule and not the exception.

We use Monte Carlo simulations to explore the small and moderate sample behavior of the SEP test, comparing it with the EP test and the MSPE-Adjusted test derived by [2]. Our simulations make use of three different data generating processes designed to match features of exchange rates and S&P 500 monthly returns, as shown by [2] and [10]. Simulation results show that the SEP test displays better size and more power than the EP test. We link this better behavior to an important bias that is present in the EP test under the null of no predictability. This bias generates an undersized test, which may also explain the lower power of the test. By removing the term that generates the bias, the SEP test displays a superior performance.

Our SEP test also fares well when compared to the MSPE-Adjusted test in terms of empirical size. Nevertheless, the MSPE-Adjusted test sometimes show more power than the SEP test, especially when the number of forecasts is large.

We illustrate the use of the SEP test in an empirical application within the commodity-currencies literature. Using monthly exchange rates of commodity-exporting economies, we find that the SEP statistic tends to be greater than the EP statistic, rejecting the null hypothesis more frequently and at tighter significance levels. These results are consistent with our simulations: compared to the EP, the SEP test has better size properties, more power and less bias.

As a final remark let us notice that the SEP test could also be considered as a “shrinkage-based-test” according to the terminology used in [25]. In this regard, the estimation of an optimal shrinkage factor to improve forecast accuracy is an open question that might be addressed in future research.