2.2.1. Value

The most commonly used value measures are (i) measures related to likelihood,

$$\mathcal{L} = \prod\_{n} p\left(y\_n|\beta\right);\tag{5}$$

(ii) those based on the proportion of variation in *y* explained by the model. The canonical measure is

$$R^2 = \frac{RSS}{TSS}.\tag{6}$$

These value measures may be interpreted in terms of explanation and prediction. If the objective of the model is taken to be the explanation of observed outcomes in terms of a given model, the best explanation may be seen as the choice of *β* that maximises the likelihood of the observed *y*. If the objective of the model is taken to be prediction of *y* given *X*, *R*<sup>2</sup> is the value function implied by the minimisation of a quadratic loss function.

2.2.2. Cost

The simplest cost functions *C* are monotonic functions of *k*, the number of non-zero parameters in the model. More generally, we may attach a cost *c* to the relaxation of a constraint *Rβ* = 0 suggested by theory.

Any cost function gives rise to a partition of the set of models into classes M*κ* where *κ* = 1 ... *K*, where the equivalence relation is defined by the property that if *κ* (*m*) = *κ* (*m*), *C* (*m*) = *C* (*m*). Standard

 choices include

$$\begin{array}{rcl} \mathbb{C} \begin{pmatrix} m \\ \end{pmatrix} &=& k, \\ \mathbb{C} \begin{pmatrix} m \\ \end{pmatrix} &=& \frac{k-1}{N-k-1}, \\ \mathbb{C} \begin{pmatrix} m \\ \end{pmatrix} &=& K-J. \end{array} \tag{7}$$

2.2.3. Profit

> In a general setting, profit may be defined as

$$
\Pi = py - w\mathbf{x},
\tag{8}
$$

where *y* is an output measure, *p* is the output price, *x* is an input measure, and *w* is the wage or factor price. Treating output as numeraire, we may simplify to

$$
\Pi = y - w\mathbf{x}.\tag{9}
$$

Given the interpretation above, profit maximisation is represented by the choice of the best-fitting model, subject to a cost associated with rejecting restrictions suggested by theory or by concern about Type 1 error.

As in a market setting, profit maximisation will be consistent with a broader notion of optimality if *y* and *x* are measured correctly and if the price *w* represents social cost.

### **3. Model Selection and Hypothesis Testing**

To examine problems of hypothesis testing and model selection, we introduce an adjacency relationship between models. Informally, two models are adjacent if they differ by a single step such as the inclusion of an additional variable or the imposition of a linear restriction. As these examples indicate, it will typically be the case that adjacent models *m*, *m* are ranked by parsimony so that, if *m* is derived by imposing a restriction in *<sup>m</sup>*, then *m κ* (*m*) < *κ* (*m*), and there exists no *m* such that *κ* (*m*) < *κ* (*m*) < *κ* (*m*) . The adjacency relation is directed and will be denoted by *m* → *m* (for the case *κ* (*m*) < *κ* (*m*)).

### *3.1. Global Model Selection Criteria*

Many widely used model selection criteria may be interpreted as profit functions, with a reversal of sign to make the problem one of maximisation rather than minimisation

Examples include the Akaike information criterion (AIC),

$$-AIC = 2\log\left(\mathcal{L}\right) - 2k,\tag{10}$$

the corrected AIC,

$$-AICc = AIC - \frac{2k^2 + 2k}{N - k - 1},\tag{11}$$

and the Bayesian information criterion (BIC)

$$-BIC = 2\log\left(\mathcal{L}\right) - \log\left(N\right)k.\tag{12}$$

It is also possible to include a cost function in measures based on *R*2. For example, we may write

$$V = R^2 - \frac{k-1}{N-k-1}.\tag{13}$$

Closely related is the *R*¯ 2 criterion, which satisfies

$$1 - R^2 = (1 - R^2) \frac{N - 1}{N - k - 1} \tag{14}$$

or

$$
\vec{R}^2 = \frac{ESS/(k-1)}{RSS/(N-k-1)}.\tag{15}
$$

### *3.2. Local Model Selection Criteria*

**Definition 1.** *Let m* → *m be adjacent models. Then m is a stepwise profit improvement (reduction) over m if* Π (*m* ; *X*, *y*) > (<) Π (*m* ; *X*, *y*).

This definition immediately suggests consideration of the stepwise regression algorithm proposed by Efroymson (1960). Let the profit function Π be *R*¯ 2. From an initial *m*, choose the adjacent *m* that maximises Π (*m* ; *X*, *y*) terminating when, for all adjacent *<sup>m</sup>*, Π (*m* ; *X*, *y*) ≥ Π (*m*; *X*, *y*). Forward selection adds the requirement *m* → *m* so that selection proceeds by adding variables. Conversely, backward selection requires *m* → *m*.

### *3.3. Hypothesis Testing*

The problems of model selection and hypothesis testing are commonly treated separately. From the viewpoint offered here, they may be regarded as global and local versions of the same problem, that of profit maximisation. This point may be illustrated with respect to the large class of classical hypothesis tests encompassed by the Wald, likelihood ratio, and Lagrange multiplier statistics Engle (1984), and also with respect to the *t* and *F* tests used in hypothesis testing in the OLS framework.

Model selection may be considered locally in terms of a pairwise choice between models, so that *m* is preferred to *m* if and only if Π (*m*; *X*, *y*) > Π (*m* ; *X*, *y*), which may be written as

$$V^\*\left(m; X, y\right) - \mathbb{C}\left(m\right) > V^\*\left(m'; X, y\right) - \mathbb{C}\left(m'\right) \tag{16}$$

or

$$V^\*\left(m; \mathcal{X}, y\right) - V^\*\left(m'; \mathcal{X}, y\right) > \mathcal{C}\left(m\right) - \mathcal{C}\left(m'\right). \tag{17}$$

If *V* (*m*; *X*, *y*) = log (L), the left-hand side of (16) and (17) is a log likelihood ratio. If *m* is obtained from *m* by the imposition of a vector of *J* constraints *Rβ* = *υ*, then under the standard assumptions of the general linear model, the Likelihood Ratio (LR) statistic is distributed as *χ*2 with *J* degrees of freedom in the limit. Hence, if we set *C* (*m* ) = 0 and *C* (*m* ) equal to a suitably chosen critical value for *χ*2 (*J*), we obtain the usual LR test.

Compare this approach to the various information criteria discussed in the previous section. The domains of the two approaches coincide in the case of nested models where the restrictions imposed in *m* consist of setting some coefficients equal to zero. The canonical criteria set out above have in common that more parsimonious models are always preferred, at least weakly. In formal terms, the classical hypothesis-testing framework shares this characteristic. The only difference is in the choice of cost function.

The same point may be made with respect to value functions based on explained variation. The natural starting point is the *F*-test,

$$F = \frac{\left(RSS\left(m\right) - RSS\left(m'\right)\right) / \left(\right)}{RSS\left(m\right) / \left(N - k\right)}.\tag{18}$$

Alternatively, we may consider the framework of constrained optimisation. The objective for a linear model *m* may be written as

$$V\left(m; \mathbf{X}, \mathbf{y}\right) - \lambda\left(R\boldsymbol{\beta} - \boldsymbol{\upsilon}\right),\tag{19}$$

where *λ* is a vector of Lagrange multipliers, such that *λ* (*Rβ* − *υ*) = 0. In the context of constrained optimisation, as observed by Breusch and Pagan (1980), the Lagrange multiplier represents the shadow price of relaxing a constraint.

The most usual restriction is that of setting the coefficient on some variable equal to zero. In this case, the *F* statistic is typically replaced by its square root, and the test is a *t*-statistic.

Such tests of significance give rise to an estimation strategy in which variables are included sequentially in the model. The most usual criterion is the *t*-statistic associated with the added variable, typically with a stopping criterion associated with 5 per cent significance.

As Dhrymes (1970) observes, this strategy is closely related to the criterion of maximising *R*¯ 2. However, *R*¯ 2 is (locally) optimised by including variables whenever their *t*-statistic is greater than one and, therefore, implies a lower price for including variables. A similar point may be made with respect to the relationship between information criteria such as the AIC and the likelihood ratio test.

### **4. Market Failure**

When the model selection problem is interpreted as one of profit maximisation, it is natural to interpret problems within econometric practice as market failures. In particular, the profit function being maximised by researchers differs from that which would be suggested by an objective of social welfare maximisation.

In relation to the replication crisis, the core of the problem, as it is generally perceived, is that the price of including a variable of interest as statistically or economically significant in the reported model is too low.

On the one hand, the bias against publication of negative results means that the benefit to researchers of reporting models with no significant variables of interest is limited and, in many fields, close to zero.

On the other hand, the availability of the set of techniques pejoratively referred to as "p-hacking" means that the test statistics reported in published studies are more likely to arise through chance than would be suggested by the classical hypothesis-testing framework.

Taken together, these observations sugges<sup>t</sup> that the price to researchers of including a variable of interest in a published model is lower than would be socially optimal. As a result, too many positive results are reported.

But what is the socially optimal price? The classical hypothesis-testing framework is of limited value here. Informal reasoning about Type 1 and Type 2 errors implies that there must exist a trade-off that can be expressed in terms of relative prices. But the relationship between that trade-off and the notion of statistical significance is opaque, to say the least. Holding the size (likelihood of Type 1 error) constant means that the power (likelihood of Type 2 error) depends on a combination of sample size and the variance of the error term in a way that bears no obvious relationship to the relative desirability of avoiding the two types of error.

Bayesian approaches based on loss functions are much closer to the spirit of the approach being suggested here. The main difference between the profit maximisation approach proposed here and the loss function approach (apart from a trivial change of sign) is the inclusion of an explicit cost associated with the estimation of a more general model.

The central point of the market failure analogy is that excessive publication of fragile results implies that the price of a positive result is too low. As with other market failures, a solution may be sought either in regulation or in pricing.

Regulatory approaches include measures such as the preregistration of estimation strategies. To develop an appropriate model of socially optimal prices, we must consider how, as a society, we respond to research publications.

This is, in some sense, an equilibrium outcome. If we accept at face value either the classical interpretation of a finding of statistical significance or its natural subjective misinterpretation (the probability that the variable in question is causally related to the variable of interest is near one), then the fact that many findings cannot be replicated is a major social problem. The initial result would lead us to act on a belief that cannot be substantiated.

Such an outcome cannot be sustained, however. Most people now understand that a reported result, whatever the supposed statistical significance, is not conclusive in the absence of replication, and certainly not at the stated level of significance. The question, therefore, is whether the publication of the result is, on balance, socially beneficial.

In this view, the most important requirement is that the price should be high enough to offset the inevitable effects of publication bias. Publication of a result showing a statistically significant result should be treated as an indication that further research is warranted, rather than conclusive or even highly probable evidence that the reported relationship is real.

### **5. Concluding Comments**

Statistical research is a social and economic enterprise aimed at discovering, testing, and ultimately acting on relationships between variables of interest. The economic concepts of profit maximisation and market failure provide a way of thinking about statistical research that reflects its social role.

**Funding:** This research received no external funding.

**Conflicts of Interest:** The author declares no conflicts of interest.
