Higher-Order Expansions for Estimators in the Presence of Nuisance Parameters

Abstract

Higher-order asymptotic methods for nonlinear models with nuisance parameters are developed. We allow for both one-step estimators, in which the nuisance and parameters of interest are jointly estimated; and also two-step (or iterated) estimators, in which the nuisance parameters are first estimated. The properties of the former, although in principle simpler to conceptualize, are more difficult to establish explicitly. The iterated estimators allow for a variety of scenarios. The results indicate when second-order considerations should be taken into account when conducting inferences with two-step estimators. The results in the paper accomplish three objectives: (i) provide simpler methods for deriving higher-order moments when nuisance parameters are present; (ii) indicate more explicitly the sources of deviations of estimators’ sampling distributions from that given by standard first-order asymptotic theory; and, in turn, (iii) indicate in which situations the corrections (either analytically or by a resampling method such as bootstrap or jackknife) should be made when making inferences. We illustrate using several popular examples in econometrics. We also provide a numerical example which highlights how a simple analytical bias correction can improve inferences.

1. Introduction

Standard higher-order asymptotics are usually applicable in situations which focus on method-of-moments estimators of k parameters, which solve a set of k estimating equations (some specific counter-examples are Rilstone and Ullah’s [1] derivation of the second-order bias of Heckman’s estimator and Newey and Smith’s [2] derivation of the second-order properties of the Generalized method-of-moments (GMM) and Generalized Empirical Likelihood (GEL) estimators). It is often the case that estimators are not expressed this way. Examples are generalized least squares estimators, in which variance estimates are obtained using residuals from a preliminary regression. GMM estimators typically depend on an estimated weighting matrix, which can itself be seen as an estimate of its probability limit. Two-stage least squares (2SLS) estimators use a preliminary linear regression to estimate instruments. In these cases, an estimated parameter appears in the estimating equation defining the parameter of interest. Often, one’s interest is simply in a subset of all the parameters in a model. In this paper, we develop a few generic tools that help in these situations. In each of these cases, we distinguish between the “parameter of interest”, ${\beta }_0$, about which we want to make inferences, and the “nuisance parameter”, ${\eta }_0$, which needs to be estimated in order to make inferences about ${\beta }_0$, but in which we have no intrinsic interest. Thus, in this paper the author develops methods for deriving stochastic expansions and approximate moments for the estimators of the parameters of interest when there are additional nuisance parameters.

In this paper, we distinguish between two types of estimation situations. In the first case, the more general, the nuisance parameter cannot be estimated without also estimating the parameter of interest. In the second case, the nuisance parameter can be estimated consistently by itself. Here, we equate this with situations in which the subset of moment conditions germane to the nuisance parameter are not a function of the parameter of interest. This corresponds to the case of two-step estimation. We refer to the more general case as one-step estimation. We spend time on both situations here, developing some tools to assist in higher-order asymptotics.

There is an extensive literature on higher-order expansions of nonlinear estimators, and specifically their second-order or approximate moments. We highlight some related examples. Nagar [3] examined the second-order bias and mean squared error (MSE) of k-class estimators. Firth [4] examined the bias of maximum likelihood estimators. Rilstone et al. [5] derived the second-order bias and MSE of method of moment estimators. Newey and Smith [2] examined the bias and MSE of GMM and GEL estimators. Hahn and Newey [6] examined bias reduction for nonlinear panel models. Bao and Ullah [7] examined the second-order skewness and kurtosis of estimators. Iglesias [8] derived the approximate bias of the smoothed maximum score estimator. Chen and Giles [9] derived the approximate bias and MSE of the binary logit model. Khundi and Rilstone [10] provided simplified methods for deriving higher-order moments. Rilstone [11] derived higher-order expansions and moments for nonlinear models with heterogeneous observations.

We proceed in a few steps. In Section 2, we review the standard setup for estimation with nuisance parameters and first-order asymptotic results. In Section 3 and Section 4, we develop techniques which streamline the derivation of higher-order stochastic expansions and approximate moments for the estimator of the parameter of interest. In Section 5, these results are then illustrated with three popular models used in econometrics: Ordinary least squares (OLS) when the focus is on a subset of parameters; estimation of the variance in a regression model; and 2SLS/instrumental variables (IV) estimation. In Section 6, we provide a numerical example based on the latter, which highlights how a simple analytical bias correction can improve inferences. The paper has three objectives: (i) provide simpler methods for deriving higher-order moments when nuisance parameters are present; (ii) indicate more explicitly the sources of deviations of estimators’ sampling distributions compared to that given by standard first-order asymptotic theory; and, in turn, (iii) indicate in which situations higher-order corrections (either analytically or by a resampling method such as bootstrap or jackknife) should be made when making inferences. Section 7 concludes and Section 8 provides detailed derivations of the results.

2. Joint Estimation

2.1. Estimating Equations and Their Derivatives

The approach we use is based on a set of $m=l+k$ estimating equations (this is identical to the setup in Rilstone et al. ([5], Equation 2.2), except here we decompose the parameter vector and the set of estimating equations into two components for the nuisance parameter and the parameter of interest):

$$\psi \left(\theta \right)={\displaystyle \frac{1}{N}}\sum q_i\left(\theta \right),q_i\left(\theta \right)=\left(\begin{array}{c}{q}_{\eta ,i}\left(\theta \right)\\ q_{\beta ,i}\left(\theta \right)\end{array}\right),\theta =\left(\begin{array}{c}\eta \\ \beta \end{array}\right)$$

(1)

and

$$\widehat{\theta }=\left(\begin{array}{c}\widehat{\eta }\\ \widehat{\beta }\end{array}\right)$$

(2)

is the solution to $\psi \left(\widehat{\theta }\right)=0$, where $\eta$ is $l\times 1$ and $\beta$ is $k\times 1$. $q_{\eta ,i}\left(\theta \right)$ and $q_{\beta ,i}\left(\theta \right)$ denote the first l and last k elements of $q_i\left(\theta \right)$. For example, in a regression framework $y_i={\eta }^{\top }{X}_{1i}+{\beta }^{\top }{X}_{2i}+{ϵ}_i$, where we have two sets of regressors, we would have $q_{\eta i}\left(\theta \right)={ϵ}_i{X}_{1i}$ and $q_{\beta i}\left(\theta \right)={ϵ}_i{X}_{2i}$. True values are indicated by ${\theta }_0={\left(\begin{array}{cc}{\eta }_0^{\top }& {\beta }_0^{\top }\end{array}\right)}^{\top }$. ${ϵ}_i$ is a regression residual, orthogonal to the regressors, such as $\mathbb{E}\left[{ϵ}_i{X}_i\right]=0$, with $X_i^{\top }=(X_{1i}^{\top },X_{2i}^{\top })$.

With two-step estimation (and, for our purposes, what distinguishes it), $q_{\eta ,i}\left(\theta \right)=q_{\eta ,i}\left(\eta \right)$ (with a slight abuse of notation) is only a function of $\eta$ and the estimate of ${\eta }_0$ is obtained by solving ${\displaystyle \frac{1}{N}}\sum q_{\eta ,i}\left(\widehat{\eta }\right)=0$. There are many advantages to such models and we focus much of our attention in this paper on them.

We suppose in this paper that the regularity conditions used in Rilstone et al. [5] to obtain the stochastic expansions and approximate moments for the estimator of the full parameter vector, $\widehat{\theta }$, are satisfied. It is convenient to introduce some notation when we have partitioned the parameter space in the way that we have. We define the selection matrices,

$$S_{\eta }=\left(\begin{array}{cc}{I}_l& 0_{l\times k}\end{array}\right),S_{\beta }=\left(\begin{array}{cc}{0}_{k\times l}& I_k\end{array}\right),$$

(3)

so that $\eta =S_{\eta }\theta$ and $\beta =S_{\beta }\theta$. These selection matrices will prove useful in a number of ways. Throughout this paper, we denote the component of a vector or matrix germane to $\beta$ (or $\eta$) by affixing an additional subscript $\beta$ (or $\eta$). This will be clear in the context. Formally, we have, e.g., $q_{\beta ,i}=S_{\beta }{q}_i$.

We use notation which indicates differentiation and cross-differentiation with respect to $\eta$ and $\beta$. In this paper, we are interested in first- through third-order derivatives, indicating these by using parameters in parentheses as superscripts. Temporarily, let ${\theta }_1$, ${\theta }_2$, and ${\theta }_3$ denote the $m_1\times 1$, $m_2\times 1$, and $m_3\times 1$ sub-vectors of $\theta$, $1\le m_j\le m$, $j=1,2,3$. In this paper, each ${\theta }_j$ is either $\theta$, $\eta$, or $\beta$. Then, we define

$$q_i^{\left({\theta }_1\right)}\left({\theta }_1\right)={\displaystyle \frac{\partial q_i\left(\theta \right)}{\partial {\theta }_1^{\top }}},q_i^{\left({\theta }_1{\theta }_2\right)}\left(\theta \right)={\displaystyle \frac{\partial q_i^{\left({\theta }_1\right)}\left(\theta \right)}{\partial {\theta }_2^{\top }}},q_i^{\left({\theta }_1{\theta }_2{\theta }_3\right)}\left(\theta \right)={\displaystyle \frac{\partial q_i^{\left({\theta }_1{\theta }_2\right)}\left(\theta \right)}{\partial {\theta }_3^{\top }}}$$

(4)

as the row-wise $m\times m_1$, $m\times m_1{m}_2$, and $m\times m_1{m}_2{m}_3$ matrices of derivatives of $q_i\left(\theta \right)$ with respect to the given arguments. As examples we have

$$q_i^{\left(\theta \eta \right)}\left(\theta \right)={\displaystyle \frac{\partial q_i^{\left(\theta \right)}\left(\theta \right)}{\partial {\eta }^{\top }}},q_i^{\left(\theta \eta \beta \right)}\left(\theta \right)={\displaystyle \frac{\partial q_i^{\left(\theta \eta \right)}\left(\theta \right)}{\partial {\beta }^{\top }}},$$

(5)

indicating the row-wise $m\times ml$ and $m\times mlk$ matrices of derivatives of $q_i^{\left(\theta \right)}\left(\theta \right)$ with respect to $\eta$, and  $q_i^{\left(\theta \eta \right)}$ with respect to $\beta$. We could establish more generality, but that is not required here and would obscure some of the results.

2.2. First-Order Asymptotics

With the assumptions of Rilstone et al. [5] satisfied, we can define the Jacobian of the estimating equations as

$$q_i^{\left(\theta \right)}\left(\theta \right)=\left(\begin{array}{cc}{q}_{\eta ,i}^{\left(\eta \right)}\left(\theta \right)& q_{\eta ,i}^{\left(\beta \right)}\left(\theta \right)\\ q_{\beta ,i}^{\left(\eta \right)}\left(\theta \right)& q_{\beta ,i}^{\left(\beta \right)}\left(\theta \right)\end{array}\right).$$

(6)

We denote $Q={\left(\mathbb{E}\left[q_1^{\left(\theta \right)}\left({\theta }_0\right)\right]\right)}^{-1}$ and partition this:

$$Q\equiv \left(\begin{array}{cc}{Q}_{\eta \eta }& Q_{\eta \beta }\\ Q_{\beta \eta }& Q_{\beta \beta }\end{array}\right)$$

(7)

where

$$\begin{array}{cc}\hfill Q_{\eta \eta }& ={\overline{q}}_{\eta ,1}^{\left(\eta \right)-1}+{\overline{q}}_{\eta ,1}^{\left(\eta \right)-1}{\overline{q}}_{\eta ,1}^{\left(\beta \right)}{Q}_{\beta \beta }{\overline{q}}_{\beta ,1}^{\left(\eta \right)}{\overline{q}}_{\eta ,1}^{\left(\eta \right)-1}\hfill \\ \hfill Q_{\eta \beta }& =-{\overline{q}}_{\eta ,1}^{\left(\eta \right)-1}{\overline{q}}_{\eta ,1}^{\left(\beta \right)}{Q}_{\beta \beta }\hfill \\ \hfill Q_{\beta \eta }& =-Q_{\beta \beta }{\overline{q}}_{\beta ,1}^{\left(\eta \right)}{\overline{q}}_{\eta ,1}^{\left(\eta \right)-1}\hfill \\ \hfill Q_{\beta \beta }& ={({\overline{q}}_{\beta ,1}^{\left(\beta \right)}-{\overline{q}}_{\beta ,1}^{\left(\eta \right)}{\overline{q}}_{\eta ,1}^{\left(\eta \right)-1}{\overline{q}}_{\eta ,1}^{\left(\beta \right)})}^{-1}.\hfill \end{array}$$

(8)

The formulae we use for the components of Q are standard (the elements of Q correspond to the elements of the partitioned inverse given in Greene ([12], Equation A-74)). Note that $Q_{\eta \eta }$, $Q_{\eta \beta }$, $Q_{\beta \eta }$, and $Q_{\beta \beta }$ are $l\times l$, $l\times k$, $k\times l$, and $k\times k$, respectively. (When a function’s argument is suppressed, that argument is being evaluated at its true value. For emphasis we sometimes include it.)

We have the ongoing assumption that $\mathbb{E}\left[q_1\right]=0$, and we let

$$\mathrm{\Omega }\equiv \mathbb{E}\left[q_1{q}_1^{\top }\right]=\left(\begin{array}{cc}\mathbb{E}\left[q_{\eta ,1}{q}_{\eta ,1}^{\top }\right]& \mathbb{E}\left[q_{\eta ,1}{q}_{\beta ,1}^{\top }\right]\\ \mathbb{E}\left[q_{\beta ,1}{q}_{\eta ,1}^{\top }\right]& \mathbb{E}\left[q_{\beta 1}{q}_{\beta 1}^{\top }\right]\end{array}\right)\equiv \left(\begin{array}{cc}{\mathrm{\Omega }}_{\eta \eta }& {\mathrm{\Omega }}_{\eta \beta }\\ {\mathrm{\Omega }}_{\beta \eta }& {\mathrm{\Omega }}_{\beta \beta }\end{array}\right).$$

(9)

We first state some results from standard first-order asymptotics. The usual (joint) influence function for $\widehat{\theta }$ is written

$$d_i=-Q{q}_i$$

(10)

with

$$\sqrt{N}(\widehat{\theta }-{\theta }_0)={\displaystyle \frac{1}{\sqrt{N}}}\sum d_i+o_P\left(1\right)\stackrel{d}{\to }N(0,V_1)$$

(11)

where

$$V_1=\mathbb{E}\left[d_1{d}_1^{\top }\right]=Q\mathrm{\Omega }{Q}^{\top }.$$

(12)

Our focus is on $\widehat{\beta }$. $\widehat{\beta }-{\beta }_0$ is of course asymptotically normal with the first-order variance matrix given by the lower right $k\times k$ sub-matrix of $V_1$:

$$\begin{array}{cc}\hfill V_{\beta ,1}& =S_{\beta }{V}_1{S}_{\beta }^{\top }\hfill \\ \hfill & =\left(\begin{array}{cc}{Q}_{\beta \eta }& Q_{\beta \beta }\end{array}\right)\left(\begin{array}{cc}{\mathrm{\Omega }}_{\eta \eta }& {\mathrm{\Omega }}_{\eta \beta }\\ {\mathrm{\Omega }}_{\beta \eta }& {\mathrm{\Omega }}_{\beta \beta }\end{array}\right)\left(\begin{array}{c}{Q}_{\beta \eta }^{\top }\\ Q_{\beta \beta }^{\top }\end{array}\right)\hfill \\ \hfill & =\left(\begin{array}{cc}{Q}_{\beta \eta }{\mathrm{\Omega }}_{\eta \eta }+Q_{\beta \beta }{\mathrm{\Omega }}_{\beta \eta }& Q_{\beta \eta }{\mathrm{\Omega }}_{\eta \beta }+Q_{\beta \beta }{\mathrm{\Omega }}_{\beta \beta }\end{array}\right)\left(\begin{array}{c}{Q}_{\beta \eta }^{\top }\\ Q_{\beta \beta }^{\top }\end{array}\right)\hfill \\ \hfill & =Q_{\beta \eta }{\mathrm{\Omega }}_{\eta \eta }{Q}_{\beta \eta }^{\top }+Q_{\beta \beta }{\mathrm{\Omega }}_{\beta \eta }{Q}_{\beta \eta }^{\top }+Q_{\beta \eta }{\mathrm{\Omega }}_{\eta \beta }{Q}_{\beta \beta }^{\top }+Q_{\beta \beta }{\mathrm{\Omega }}_{\beta \beta }{Q}_{\beta \beta }^{\top }\hfill \end{array}$$

(13)

Since each of the $Q$’s contain a number of terms, $V_{\beta ,1}$ can be rather complicated, at least analytically. The complexity carries over to higher-order asymptotics. Fortunately, additional information on the structure of models often simplifies matters. By inspection of Equations (7) and (9) it is useful to see under which conditions $V_{\beta ,1}$ simplifies.

Note that if ${\mathrm{\Omega }}_{\eta \beta }={\mathrm{\Omega }}_{\beta \eta }^{\top }$ is zero, the two middle terms in $V_{\beta ,1}$ vanish. In general, this is not the case. However, it does hold for some very important situations. This will occur with a linear regression model with orthogonal regressors. It occurs when estimating the mean and variance of a random variable when ${\mathrm{\Omega }}_{\eta \beta }$ is proportional to the third moment of the random variable and is zero if the third moment is zero.

In the case of two-step estimators, we take as given that $\beta$ does not appear in $q_{\eta ,i}\left(\theta \right)$, so that $q_{\eta ,i}^{\left(\beta \right)}\left(\theta \right)=0$. (We are largely interested in the resulting impact on the first-order variance. In this case, the important condition can be weakened: $\mathbb{E}\left[q_{\eta ,1}^{\left(\beta \right)}\left({\theta }_0\right)\right]=0$.) The form of Q is critical to the properties of $\widehat{\theta }$ and, by implication, to the properties of $\widehat{\beta }$. Using the formula for inverting a partitioned matrix we see that Q is upper diagonal in the two-step case and that the other elements of Q simplify as well, with $Q_{\beta \beta }={\left({\overline{q}}_{\beta ,1}^{\left(\beta \right)}\right)}^{-1}$. Typically, ${\overline{q}}_{\eta ,1}^{\left(\eta \right)}$ and ${\overline{q}}_{\beta ,1}^{\left(\beta \right)}$ are symmetric Hessian matrices, so that

$$Q=\left(\begin{array}{cc}{Q}_{\eta \eta }& 0\\ Q_{\beta \eta }& Q_{\beta \beta }\end{array}\right)\equiv \left(\begin{array}{cc}{\overline{q}}_{\eta ,1}^{\left(\eta \right)-1}& 0\\ -{\overline{q}}_{\beta ,1}^{\left(\beta \right)-1}{\overline{q}}_{\beta ,1}^{\left(\eta \right)}{\overline{q}}_{\eta ,1}^{\left(\eta \right)-1}& {\overline{q}}_{\beta ,1}^{\left(\beta \right)-1}\end{array}\right)$$

(14)

and the components of $V_{\beta ,1}$ simplify:

$$\begin{array}{cc}\hfill Q_{\beta \eta }{\mathrm{\Omega }}_{\eta \eta }{Q}_{\beta \eta }^{\top }& ={\overline{q}}_{\beta ,1}^{\left(\beta \right)-1}{\overline{q}}_{\beta ,1}^{\left(\eta \right)}{\overline{q}}_{\eta ,1}^{\left(\eta \right)-1}{\mathrm{\Omega }}_{\eta \eta }{\overline{q}}_{\eta ,1}^{\left(\eta \right)-1}{\overline{q}}_{\beta ,1}^{\left(\eta \right)\top }{\overline{q}}_{\beta ,1}^{\left(\beta \right)-1}\hfill \\ \hfill Q_{\beta \beta }{\mathrm{\Omega }}_{\beta \eta }{Q}_{\beta \eta }^{\top }& =-{\overline{q}}_{\beta ,1}^{\left(\beta \right)-1}{\mathrm{\Omega }}_{\beta \eta }{\overline{q}}_{\eta ,1}^{\left(\eta \right)-1\top }{\overline{q}}_{\beta ,1}^{\left(\eta \right)\top }{\overline{q}}_{\beta ,1}^{\left(\beta \right)-1}\hfill \\ \hfill Q_{\beta \eta }{\mathrm{\Omega }}_{\eta \beta }{Q}_{\beta \beta }^{\top }& =-{\overline{q}}_{\beta ,1}^{\left(\beta \right)-1}{\overline{q}}_{\beta ,1}^{\left(\eta \right)}{\overline{q}}_{\eta ,1}^{\left(\eta \right)-1}{\mathrm{\Omega }}_{\eta \beta }{\overline{q}}_{\beta ,1}^{\left(\beta \right)-1}\hfill \\ \hfill Q_{\beta \eta }{\mathrm{\Omega }}_{\beta \beta }{Q}_{\beta \beta }^{\top }& ={\overline{q}}_{\beta ,1}^{\left(\beta \right)-1}{\mathrm{\Omega }}_{\beta \beta }{\overline{q}}_{\beta ,1}^{\left(\beta \right)-1}.\hfill \end{array}$$

(15)

Suppose ${\eta }_0$ is known. Under regularity conditions, standard results would inform us that the corresponding estimating equations

$${\psi }^{*}\left(\beta \right)={\displaystyle \frac{1}{N}}\sum q_{\beta ,i}({\eta }_0,\beta )$$

(16)

would result in an estimator, ${\widehat{\beta }}^{*}$, which would be approximated as

$$\sqrt{N}({\widehat{\beta }}^{*}-{\beta }_0)={\displaystyle \frac{1}{\sqrt{N}}}\sum d_i^{*}+o_P\left(1\right)\stackrel{d}{\to }N(0,V_{\beta ,1}^{*})$$

(17)

where $d_i^{*}=-Q^{*}{q}_{\beta ,i}$; $Q^{*}={\left(\mathbb{E}\left[q_{\beta ,1}^{\left(\beta \right)}\right]\right)}^{-1}$; and $V_{\beta ,1}^{*}=Q^{*}{\mathrm{\Omega }}_{\beta \beta }{Q}^{*}$. In the case that the estimating equations $q_{\eta ,i}$ and $q_{\beta ,i}$ are uncorrelated, the two middle terms of $V_{1,\beta }$ drop out. In this case, $V_{1,\beta }-V_{1,\beta }^{*}=Q_{\beta \eta }{\mathrm{\Omega }}_{\eta \eta }{Q}_{\beta \eta }^{\top }$ is positive definite. There are cases (when ${\mathrm{\Omega }}_{\eta \beta }\ne 0$) when $V_{1,\beta }-V_{1,\beta }^{*}$ is negative definite, e.g., with a fully and correctly specified maximum likelihood estimator. See Wooldridge [13] for a discussion.

3. Stochastic Expansions with Nuisance Parameters

Whether the estimation method is one or two steps (or more), we can adapt the methodology of Rilstone et al. [5] in a common manner. We suppose that the standard first- through third-order stochastic expansions and approximate moments for the $m\times 1$ estimator $\widehat{\theta }$ are valid.

We recall the standard form of the influence functions. In order to systematically derive the moments of the various estimators, we write the stochastic expansions for $\widehat{\theta }$, as in Rilstone et al. [5], as

$$\widehat{\theta }-{\theta }_0=\left\{\begin{array}{cc}{a}_{-1/2}+o_P\left(N^{-1/2}\right),\hfill & \\ a_{-1/2}+a_{-1}+o_P\left(N^{-1}\right),\hfill & \\ a_{-1/2}+a_{-1}+a_{-3/2}+o_P\left(N^{-3/2}\right)\hfill \end{array}\right.$$

(18)

and

$$a_{-1/2}={\displaystyle \frac{1}{N}}\sum _1{d}_i,a_{-1}={\displaystyle \frac{1}{N^2}}\sum _{i_1{i}_2}{d}_{i_1{i}_2},a_{3/2}={\displaystyle \frac{1}{N^3}}\sum _{i_1{i}_2{i}_3}{d}_{i_1{i}_2{i}_3}$$

(19)

where

$$\begin{array}{cc}\hfill d_i& =-Q{q}_i\hfill \\ \hfill d_{i_1{i}_2}& =\sum _{s=1}^2{A}_{i_1{i}_2}\left(s\right)\hfill \\ \hfill A_{i_1{i}_2}\left(1\right)& ={\tilde{d}}_{i_1}^{\left(\theta \right)}{d}_{i_2},\hfill \\ \hfill A_{i_1{i}_2}\left(2\right)& ={\displaystyle \frac{1}{2}}{\overline{d}}_1^{\left(\theta \theta \right)}(d_{i_1}\otimes d_{i_2}),\hfill \\ \hfill d_{i_1{i}_2{i}_3}& =\sum _{s=1}^6{E}_{i_1{i}_2{i}_3}\left(s\right),\hfill \\ \hfill E_{i_1{i}_2{i}_3}\left(1\right)& ={\overline{d}}_1^{\left(\theta \theta \right)}(d_{i_1}\otimes {\tilde{d}}_{i_2}^{\left(\theta \right)}{d}_{i_3}),\hfill \\ \hfill E_{i_1{i}_2{i}_3}\left(2\right)& ={\displaystyle \frac{1}{2}}{\overline{d}}_1^{\left(\theta \theta \right)}(d_{i_1}\otimes {\overline{d}}_1^{\left(\theta \theta \right)}(d_{i_2}\otimes d_{i_3})),\hfill \\ \hfill E_{i_1{i}_2{i}_3}\left(3\right)& ={\displaystyle \frac{1}{6}}{\overline{d}}_1^{\left(\theta \theta \theta \right)}(d_{i_1}\otimes d_{i_2}\otimes d_{i_3}),\hfill \\ \hfill E_{i_1{i}_2{i}_3}\left(4\right)& ={\tilde{d}}_{i_1}^{\left(\theta \right)}{\tilde{d}}_{i_2}^{\left(\theta \right)}{d}_{i_3},\hfill \\ \hfill E_{i_1{i}_2{i}_3}\left(5\right)& ={\displaystyle \frac{1}{2}}{\tilde{d}}_{i_1}^{\left(\theta \right)}{\overline{d}}_1^{\left(\theta \theta \right)}(d_{i_2}\otimes d_{i_3}),\hfill \\ \hfill E_{i_1{i}_2{i}_3}\left(6\right)& ={\displaystyle \frac{1}{2}}{\tilde{d}}_{i_1}^{\left(\theta \theta \right)}(d_{i_2}\otimes d_{i_3})\hfill \end{array}$$

(20)

and ⊗ denotes the usual Kroenecker product where, for the matrices $A={\left[a_{ij}\right]}_{m\times n}$ and $B_{p\times q}$, we have $A\otimes B={\left[a_{ij}B\right]}_{mp\times nq}$.

Note here that each of the terms is an $m\times 1$ vector. We single out the k rows of these pertinent to $\widehat{\beta }$ using the following notation:

$$\begin{array}{cc}\hfill d_{\beta ,i_1}& =S_{\beta }{d}_{i_1}\hfill \\ \hfill d_{\beta ,i_1{i}_2}& =S_{\beta }{d}_{i_1{i}_2},\hfill \\ \hfill d_{\beta ,i_1{i}_2{i}_3}& =S_{\beta }{d}_{i_1{i}_2{i}_3},\hfill \\ \hfill A_{\beta ,i_1{i}_2}\left(s\right)& =S_{\beta }{A}_{i_1{i}_2}\left(s\right),s=1,2,\hfill \\ \hfill E_{\beta ,i_1{i}_2{i}_3}\left(s\right)& =S_{\beta }{E}_{i_1{i}_2{i}_3}\left(s\right),s=1,2,\dots ,6.\hfill \end{array}$$

(21)

Note that each $A_{\beta ,i_1{i}_2}\left(s\right)$ and $E_{\beta ,i_1{i}_2{i}_3}\left(s\right)$ is constructed from the k bottom rows of each $A_{i_1{i}_2}\left(l\right)$ and $E_{i_1{i}_2{i}_3}\left(s\right)$, which depend on the entire vectors of $d_i$ and its derivatives with respect to $\theta$.

Defining

$$\begin{array}{cc}\hfill a_{\beta ,-1/2}& ={\displaystyle \frac{1}{N}}\sum _{i_1}{d}_{\beta ,i_1}\hfill \\ \hfill a_{\beta ,-1}& ={\displaystyle \frac{1}{N^2}}\sum _{i_1{i}_2}{d}_{\beta ,i_1{i}_2}\hfill \\ \hfill a_{\beta ,-3/2}& ={\displaystyle \frac{1}{N^3}}\sum _{i_1{i}_2{i}_3}{d}_{\beta ,i_1{i}_2{i}_3},\hfill \end{array}$$

(22)

we see that the stochastic expansion for $\widehat{\beta }$ can thus be written as

$$\widehat{\beta }-{\beta }_0=\left\{\begin{array}{cc}{a}_{\beta ,-1/2}+o_P\left(N^{-1/2}\right),\hfill & \\ a_{\beta ,-1/2}+a_{\beta ,-1}+o_P\left(N^{-1}\right),\hfill & \\ a_{\beta ,-1/2}+a_{\beta ,-1}+a_{\beta ,-3/2}+o_P\left(N^{-3/2}\right).\hfill \end{array}\right.$$

(23)

These expressions are then used in order to obtain the approximate moments of $\widehat{\beta }$.

Recall also the manner in which derivatives are defined recursively using this notation, so we can also decompose matrices of mixed derivatives in a similar manner. In many cases, models will be such that certain subsets of derivatives are zero. In these cases, and particularly in the cases of the GMM and GEL estimators we consider in future work, it will be convenient to decompose the derivatives into sub-blocks. With respect to the first-order derivatives, we see immediately that we can write the Jacobian of $q_i\left(\theta \right)$ as

$$q_i^{\left(\theta \right)}\left(\theta \right)=q_i^{\left(\eta \right)}\left(\theta \right)S_{\eta }+q_i^{\left(\beta \right)}\left(\theta \right)S_{\beta }.$$

(24)

With respect to the second-order derivatives, we have the following:

$$\begin{array}{cc}\hfill q_i^{\left(\theta \theta \right)}\left(\theta \right)& =q_i^{\left(\eta \eta \right)}\left(\theta \right)(S_{\eta }\otimes S_{\eta })+q_i^{\left(\eta \beta \right)}\left(\theta \right)(S_{\eta }\otimes S_{\beta })+q_i^{\left(\beta \eta \right)}\left(\theta \right)(S_{\beta }\otimes S_{\eta })\hfill \\ \hfill & +q_i^{\left(\beta \beta \right)}\left(\theta \right)(S_{\beta }\otimes S_{\beta })\hfill \\ \hfill & =q_i^{\left(\eta \eta \right)}\left(\theta \right)S_{\eta }^{\otimes 2}+q_i^{\left(\eta \beta \right)}\left(\theta \right)(S_{\eta }\otimes S_{\beta })+q_i^{\left(\beta \eta \right)}\left(\theta \right)(S_{\beta }\otimes S_{\eta })+q_i^{\left(\beta \beta \right)}\left(\theta \right)S_{\beta }^{\otimes 2}.\hfill \end{array}$$

(25)

Similarly, with respect to the third-order derivatives we have the following:

$$\begin{array}{cc}\hfill q_i^{\left(\theta \theta \theta \right)}\left(\theta \right)& =q_i^{\left(\eta \eta \eta \right)}\left(\theta \right)S_{\eta }^{\otimes 3}+q_i^{\left(\eta \eta \beta \right)}\left(\theta \right)(S_{\eta }^{\otimes 2}\otimes S_{\beta })+q_i^{\left(\eta \beta \eta \right)}\left(\theta \right)(S_{\eta }\otimes S_{\beta }\otimes S_{\eta })\hfill \\ \hfill & +q_i^{\left(\eta \beta \beta \right)}\left(\theta \right)(S_{\eta }\otimes S_{\beta }^{\otimes 2})+q_i^{\left(\beta \eta \eta \right)}\left(\theta \right)(S_{\beta }\otimes S_{\eta }^{\otimes 2})\hfill \\ \hfill & +q_i^{\left(\beta \eta \beta \right)}\left(\theta \right)(S_{\beta }\otimes S_{\eta }\otimes S_{\beta })+q_i^{\left(\beta \beta \eta \right)}\left(\theta \right)(S_{\beta }^{\otimes 2}\otimes S_{\eta })+q_i^{\left(\beta \beta \beta \right)}\left(\theta \right)S_{\beta }^{\otimes 3}.\hfill \end{array}$$

(26)

These expressions may seem overly long. However, in many situations, blocks of the cross-derivatives are equal to zero (this may hold, e.g., at the true and/or expected values and/or in deviations from expectations). Also, these matrices are often post-multiplied by a block diagonal matrix, which will automatically nullify the resulting product. A formal proof that these matrices can be decomposed in this way can be obtained by application of the product rule of calculus.

As a practical matter, it is often simpler to compute derivatives by decomposing them into their, say, $\eta$ and $\beta$ (by inspection, many of these are often zero) components. Also note that it is often convenient to decompose the derivatives row-wise so that

$$q_i^{\left(\theta \right)}\left(\theta \right)=\left(\begin{array}{c}{q}_{\eta i}^{\left(\theta \right)}\left(\theta \right)\\ q_{\beta i}^{\left(\theta \right)}\left(\theta \right)\end{array}\right),q_i^{\left(\theta \theta \right)}\left(\theta \right)=\left(\begin{array}{c}{q}_{\eta i}^{\left(\theta \theta \right)}\left(\theta \right)\\ q_{\beta i}^{\left(\theta \theta \right)}\left(\theta \right)\end{array}\right),q_i^{\left(\theta \theta \theta \right)}\left(\theta \right)=\left(\begin{array}{c}{q}_{\eta i}^{\left(\theta \theta \theta \right)}\left(\theta \right)\\ q_{\beta i}^{\left(\theta \theta \theta \right)}\left(\theta \right)\end{array}\right).$$

(27)

4. Approximate Moments with Nuisance Parameters

Once we have obtained the stochastic expansions for the estimators of interest, we define the approximate moments for them in the same manner as we do the moments for the entire parameter, typically affixing a subscript $\beta$. We can obtain the approximate moments in a couple of ways. One is by constructing these from the appropriate moments of the “marginal” influence functions $d_{\beta ,i}$, $d_{\beta ,i_1,i_2}$, and $d_{\beta ,i_1,i_2,i_3}$.

Alternatively, if we have obtained the approximate moment for $\widehat{\theta }$, it may be possible to obtain that for $\widehat{\beta }$ directly. Note that if we have obtained, say, the lth approximate (Kronecker) moment of $\widehat{\theta }$ using $\mathbb{E}\left[{\widehat{\theta }}_{-j/2}^{\otimes l}\right]$, then we can obtain the lth approximate (Kronecker) moment of $\widehat{\beta }$ using

$$\mathbb{E}\left[{\widehat{\beta }}_{-j/2}^{\otimes l}\right]=\mathbb{E}\left[{\left(S_{\beta }{\widehat{\theta }}_{-j/2}\right)}^{\otimes l}\right]=S_{\beta }^{\otimes l}\mathbb{E}\left[{\widehat{\theta }}_{-j/2}^{\otimes l}\right].$$

(28)

In each of these cases, there will often be some simplification possible. Note that the approximate bias for $\widehat{\beta }$ is obtained as

$${\mathrm{Bias}}_2\left[\widehat{\beta }\right]=S_{\beta }{\mathrm{Bias}}_2\left[\widehat{\theta }\right]=S_{\beta }\mathbb{E}\left[{\widehat{\theta }}_{-1}\right].$$

(29)

The approximate second, third, and fourth (Kronecker) moments of $\widehat{\beta }$ can be obtained from Equation (28) by setting, respectively, $j=3,l=2$; $j=2,l=3$; and $j=3,l=4$. In matrix form, the approximate MSE of $\widehat{\beta }$ is obtained by evaluating

$$\mathbb{E}\left[{\beta }_{-3/2}{\beta }_{-3/2}^{\top }\right]=\mathbb{E}\left[S_{\beta }{\theta }_{-3/2}{\left(S_{\beta }{\theta }_{-3/2}\right)}^{\top }\right]=S_{\beta }\mathbb{E}\left[{\theta }_{-3/2}{\theta }_{-3/2}^{\top }\right]S_{\beta }^{\top },$$

(30)

so that

$${\mathrm{MSE}}_2\left[\widehat{\beta }\right]=S_{\beta }{\mathrm{MSE}}_2\left[\widehat{\theta }\right]S_{\beta }^{\top }=S_{\beta }{\mathrm{MSE}}_2\left[{\widehat{\theta }}_{-3/2}{\widehat{\theta }}_{-3/2}^{\top }\right]S_{\beta }^{\top },$$

(31)

retaining terms up to $O\left(N^{-2}\right)$. In the illustrations that follow, we use a variety of these techniques.

5. Illustrations

In this section, we examine in detail three examples. First, the case of OLS when we are only interested in a subset of the regression parameters. This is typically a case of one-step estimation: neither parameter can generally be estimated consistently without also estimating the other. The other two examples in this section are two-step estimators. In each case, we derive the first through fourth approximate moments. It is useful for researchers to examine these higher moments, including skewness and kurtosis, when conducting inferences as they can indicate when deviations from standard first-order results can impact on the conclusions of their inferences. For example, Hansen et al. [14] conducted simulations using estimators which fall into our framework (various GMM estimators). They documented extensive deviations from normality in the sampling distributions, notably in higher moments, of these estimators.

5.1. Ordinary Least Squares with Nuisance Parameters

We reconsider the OLS estimator of the linear regression model with the focus now on a subset of the parameters. We can obtain the relevant stochastic expansions and approximate moments for these in a variety of ways. Here, we achieve this in a manner to highlight the approach in this paper. To put this in the current context we modify the notation slightly. Again, we have a model and estimating equations that can be written as

$$\begin{array}{cc}\hfill y_i& =X_i{\theta }_0+{ϵ}_i,q_i\left(\theta \right)=X_i(y_i-X_i^{\top }\theta ).\hfill \end{array}$$

(32)

Note that we have labeled the entire regression parameter $\theta$, which is $m\times 1$, and partitions such that $\theta ={({\eta }^{\top },{\beta }^{\top })}^{\top }$, where $\eta$ is the $l\times 1$ nuisance parameter and $\beta$ is the $k\times 1$ parameter of interest. Denote the OLS estimators as $\widehat{\theta }={({\widehat{\eta }}^{\top },{\widehat{\beta }}^{\top })}^{\top }$. Denote the influence functions for $\widehat{\theta }$ as $d_i$, $d_{i_1{i}_2}$, and $d_{i_1{i}_2{i}_3}$. The influence functions for $\widehat{\beta }$ are obtained simply by pre-multiplying the influence functions for $\widehat{\theta }$ by $S_{\beta }$:

$$\begin{array}{cc}\hfill d_{\beta ,i}=S_{\beta }{d}_i& =S_{\beta }{V}_X^{-1}{X}_i{ϵ}_i,\hfill \\ \hfill d_{\beta ,i_1{i}_2}=S_{\beta }{d}_{i_1{i}_2}& =-S_{\beta }{V}_X^{-1}{W}_{i_1}{V}_X^{-1}{X}_{i_2}{ϵ}_{i_2},\hfill \\ \hfill d_{\beta ,i_1{i}_2{i}_3}=S_{\beta }{d}_{i_1{i}_2{i}_3}& =S_{\beta }{V}_X^{-1}{W}_{i_1}{V}_X^{-1}{W}_{i_2}{V}_X^{-1}{X}_{i_3}{ϵ}_{i_3}.\hfill \end{array}$$

(33)

where $V_X=\mathbb{E}\left[X_1{X}_1^{\top }\right]$, $W_i=X_i{X}_i^{\top }-V_X$. We note the following, some of which are very familiar as they arise extensively with the linear regression model. We also see here that

$$\mathrm{\Omega }={\sigma }^2\left(\begin{array}{cc}\mathbb{E}\left[X_{\eta ,1}{X}_{\eta ,1}^{\top }\right]& \mathbb{E}\left[X_{\eta ,1}{X}_{\beta ,1}^{\top }\right]\\ \mathbb{E}\left[X_{\beta ,1}{X}_{\eta ,1}^{\top }\right]& \mathbb{E}\left[X_{\beta ,1}{X}_{\beta ,1}^{\top }\right]\end{array}\right)={\sigma }^2{V}_X.$$

(34)

The approximate bias (and exact bias) for $\widehat{\beta }$ is of course zero, noting that $\mathbb{E}\left[d_{\beta ,11}\right]=S_{\beta }\mathbb{E}\left[d_{11}\right]=0$.

The decomposition of $V_X^{-1}$ will be familiar to most readers. Using the partitioned inverse formula we have

$$V_X^{-1}=\left(\begin{array}{cc}{V}^{\eta \eta }& V^{\eta \beta }\\ V^{\beta \eta }& V^{\beta \beta }\end{array}\right)$$

(35)

$$\begin{array}{cc}\hfill V^{\eta \eta }& =\mathbb{E}{\left[X_{\eta ,1}{X}_{\eta ,1}^{\top }\right]}^{-1}+\mathbb{E}{\left[X_{\eta ,1}{X}_{\eta ,1}^{\top }\right]}^{-1}\mathbb{E}\left[X_{\eta ,1}{X}_{\beta ,1}^{\top }\right]V^{\beta \beta }\mathbb{E}\left[X_{\beta ,1}{X}_{\eta ,1}^{\top }\right]\mathbb{E}{\left[X_{\eta ,1}{X}_{\eta ,1}^{\top }\right]}^{-1}\hfill \\ \hfill V^{\eta \beta }& =-\mathbb{E}{\left[X_{\eta ,1}{X}_{\eta ,1}^{\top }\right]}^{-1}\mathbb{E}\left[X_{\eta ,1}{X}_{\beta ,1}^{\top }\right]V^{\beta \beta }\hfill \\ \hfill V^{\beta \eta }& =-V^{\beta \beta }\mathbb{E}\left[X_{\beta ,1}{X}_{\eta ,1}^{\top }\right]\mathbb{E}{\left[X_{\eta ,1}{X}_{\eta ,1}^{\top }\right]}^{-1}\hfill \\ \hfill V^{\beta \beta }& ={(\mathbb{E}\left[X_{\beta ,1}{X}_{\beta ,1}^{\top }\right]-\mathbb{E}\left[X_{\beta ,1}{X}_{\eta ,1}^{\top }\right]\mathbb{E}\left[X_{\eta ,1}{X}_{\eta ,1}^{\top }\right]\mathbb{E}\left[X_{\eta ,1}{X}_{\beta ,1}^{\top }\right])}^{-1}.\hfill \end{array}$$

(36)

The usual first-order asymptotic variance for $\widehat{\theta }$ is given by

$$V_1=\mathbb{E}\left[d_1{d}_1^{\top }\right]=V_X^{-1}\mathrm{\Omega }{V}_X^{-1}={\sigma }^2{V}_X^{-1}.$$

(37)

and that for $\widehat{\beta }$ is

$$V_{\beta ,1}=S_{\beta }{V}_1{S}_{\beta }^{\top }={\sigma }^2{V}^{\beta \beta }$$

(38)

Let $X_i^{*}=V_X^{-1/2}{X}_i$. With respect to the approximate MSE, we now obtain the approximate MSE of $\widehat{\beta }$ by pre-multiplying the standard formula by $S_{\beta }$ and post-multiplying by $S_{\beta }^{\top }$. In this way, we see that the approximate MSE of $\widehat{\beta }$ can be written as

$${\mathrm{MSE}}_{\beta ,2}\left[\widehat{\beta }\right]={\displaystyle \frac{1}{N}}{S}_{\beta }{V}_1{S}_{\beta }^{\top }+{\displaystyle \frac{1}{N^2}}{S}_{\beta }\left(V_1^{1/2}\mathbb{E}\left[{\left(X_1^{*}{X}_1^{*\top }\right)}^2\right]V_1^{1/2}-V_1\right)S_{\beta }^{\top }$$

(39)

As a special case, consider when $X_i\sim N(0,V_X)$. Then, $X_1^{*}\sim N(0,I_m)$. The second and fourth moments of the pth element of $X_1^{*}$, say $X_{1p}^{*}$, are 1 and 3. The $pq$th element of $S={\left(X_1^{*}{X}_1^{*\top }\right)}^2=X_1^{*}\left(X_1^{*\top }{X}_1^{*}\right)X_1^{*\top }$ can be written as

$$S_{pq}=X_{1p}^{*}{X}_{1q}^{*}\sum _{r=1}^k{X}_{lr}^{*2}$$

For $p\ne q$, $\mathbb{E}\left[S_{pq}\right]=0$. For $p=q$ we have

$$\mathbb{E}\left[S_{pp}\right]=\mathbb{E}\left[X_{pp}^{*2}(X_{pp}^2+\sum _{r\ne p}^m{X}_{1r}^{*2})\right]=3+m-1=m+2$$

so that $\mathbb{E}\left[{\left(X_1^{*}{X}_1^{*\top }\right)}^2\right]=(m+2)I_m$ and

$${\mathrm{MSE}}_{\beta ,2}\left[\widehat{\beta }\right]={\displaystyle \frac{1}{N}}\left(1+{\displaystyle \frac{m+1}{N}}\right)S_{\beta }{V}_1{S}_{\beta }^{\top }={\displaystyle \frac{1}{N}}\left(1+{\displaystyle \frac{m+1}{N}}\right)V_{\beta ,1}.$$

(40)

Note here the presence of m in the second-order term.

With regards to the approximate skewness of $\widehat{\beta }$, it follows from Equation (28) that

$${\mathrm{Skew}}_2\left[\widehat{\beta }\right]=S_{\beta }^{\otimes 3}{\mathrm{Skew}}_2\left[\widehat{\theta }\right]={\displaystyle \frac{1}{N^2}}{\left(S_{\beta }{V}_X^{-1}\right)}^{\otimes 3}\mathbb{E}\left[X_1^{\otimes 3}\right]\mathbb{E}\left[{ϵ}_1^3\right].$$

(41)

Note the presence of $S_{\beta }{V}_X^{-1}$. Typically, this does not simplify. The notable case in which it does is when $X_{\beta ,1}$ and $X_{\eta ,1}$ are orthogonal and $\mathbb{E}\left[X_{\beta ,1}{X}_{\eta ,1}^{\top }\right]=0$. Then, $S_{\beta }{V}_X^{-1}=\left(\begin{array}{cc}0& \mathbb{E}{\left[X_{\beta ,1}{X}_{\beta ,1}^{\top }\right]}^{-1}\end{array}\right)$ and

$${\mathrm{Skew}}_2\left[\widehat{\beta }\right]={\displaystyle \frac{1}{N^2}}{\left(\mathbb{E}{\left[X_{\beta ,1}{X}_{\beta ,1}^{\top }\right]}^{-1}\right)}^{\otimes 3}\mathbb{E}\left[X_{\beta ,1}^{\otimes 3}\right]\mathbb{E}\left[{ϵ}_1^3\right].$$

(42)

To obtain the approximate kurtosis of $\widehat{\beta }$, it follows from Equation (28) that

$$\begin{array}{cc}\hfill {\mathrm{Kurt}}_2\left[\widehat{\beta }\right]& =S_{\beta }^{\otimes 4}{\mathrm{Kurt}}_2\left[\widehat{\theta }\right]\hfill \\ \hfill & ={\displaystyle \frac{1}{N^3}}{S}_{\beta }^{\otimes 4}\left(V_X^{-1\otimes 4}\mathbb{E}\left[X_1^{\otimes 4}\right]\mathbb{E}\left[{ϵ}_1^4\right]-3{\overrightarrow{V}}_1^{\otimes 2}+6\overrightarrow{[}\mathbb{E}\left[V_1^{1/2}{W}_1^{*2}{V}_1^{1/2}\right]\otimes {\overrightarrow{V}}_1\right)\hfill \\ \hfill & ={\displaystyle \frac{1}{N^3}}({\left(S_{\beta }{V}_X^{-1}\right)}^{\otimes 4}\mathbb{E}\left[X_1^{\otimes 4}\right]\mathbb{E}\left[{ϵ}_1^4\right]-3{\overrightarrow{V}}_{\beta ,1}^{\otimes 2}\hfill \\ \hfill & +6S_{\beta }^{\otimes 2}\overrightarrow{[}\mathbb{E}\left[V_1^{1/2}{W}_1^{*2}{V}_1^{1/2}\right]\otimes {\overrightarrow{V}}_{\beta ,1})\hfill \end{array}$$

(43)

As we saw with the approximate MSE and skewness, we can verify that this corresponds to the results with the full OLS estimator. This is simply achieved here and above by setting $m=k$ and $S_{\beta }=I_m$.

We now consider examples of estimation of the parameter of interest with two-step estimators.

5.2. Regression-Variance Problem

A simple illustration with two-step estimators is the estimation of the variance of a linear regression model. The first step estimates the regression parameters and these are then used in the second step to estimate the variance. This example is useful pedagogically and it also serves as an important segue into other very common and more complex models which use regression coefficient estimates from a first stage. These include various two-stage least squares estimators which we examine, also weighted and generalized least squares estimators, where OLS estimates are used in a first stage and then used to estimate weights in a second stage. Generated regressor models and treatment effects parameters also fall into this category. Before going on to those extensions, however, the focus here is on the second moment of the random variable, ${ϵ}_i$:

$${\beta }_0=\mathbb{E}\left[{ϵ}_i^2\right]$$

(44)

where ${ϵ}_i$ is the residual from the regression model

$$y_i=X_i^{\top }{\eta }_0+{ϵ}_i\mathbb{E}\left[X_i{ϵ}_i\right]=0$$

(45)

and $X_i$ is $l\times 1$. Framing this as a two-step estimator, we treat $\eta$ as the first step or “nuisance” parameter and $\beta$ as the second step or parameter “of interest”. The joint estimating equations are given by

$$q_i\left(\theta \right)=\left(\begin{array}{c}{q}_{\eta ,i}\\ q_{\beta ,i}\end{array}\right)=\left(\begin{array}{c}{X}_i(y_i-X_i^{\top }\eta )\\ {(y_i-X_i^{\top }\eta )}^2-\beta \end{array}\right),q_i=\left(\begin{array}{c}{ϵ}_i\\ {\mu }_i\end{array}\right)$$

(46)

where ${\mu }_i={(y_i-X_i^{\top }{\eta }_0)}^2-{\beta }_0$. Put $V_X=\mathbb{E}\left[X_1{X}_1^{\top }\right]$, $W_i=X_i{X}_i^{\top }-V_X$.

$$\begin{array}{cc}\hfill q_i^{\left(\theta \right)}& =\left(\begin{array}{cc}-X_i{X}_i^{\top }& 0_{l\times 1}\\ -2(y_i-\eta )X_i^{\top }& -1\end{array}\right),{\overline{q}}_1^{\left(\theta \right)}=-\left(\begin{array}{cc}{V}_X& 0_{l\times 1}\\ 0_{1\times l}& 1\end{array}\right),\hfill \end{array}$$

(47)

$$\begin{array}{cc}\hfill {\tilde{q}}_i^{\left(\theta \right)}& =-\left(\begin{array}{cc}{W}_i& 0_{l\times 1}\\ 2{ϵ}_i{X}_i^{\top }& 0_{1\times l}\end{array}\right)\hfill \\ \hfill q_i^{\left(\theta \theta \right)}& =\left(\begin{array}{cccc}{0}_{l\times l^2}& 0& 0& 0\\ 2X_i^{\top \otimes 2}& 0& 0& 0\end{array}\right),{\overline{q}}_1^{\left(\theta \theta \right)}=\left(\begin{array}{cccc}{0}_{l\times l^2}& 0& 0& 0\\ 2{\overrightarrow{V}}_X^{\top }& 0& 0& 0\end{array}\right)\hfill \end{array}$$

(48)

Noting that the derivative matrices of $q_i$, ${\tilde{q}}_i$, and ${\overline{q}}_1$ are quite sparse, it is useful to use the partitioned matrix notation to write

$$q_i^{\left(\theta \right)}=q_i^{\left(\eta \right)}{S}_{\eta }+q_i^{\left(\eta \right)}{S}_{\beta },$$

(49)

$$q_i^{\left(\eta \right)}=\left(\begin{array}{c}-X_i{X}_i^{\top }\\ -2(y_i-X_i^{\top }\eta )X_i^{\top }\end{array}\right),q_i^{\left(\beta \right)}=\left(\begin{array}{c}{0}_{l\times 1}\\ -1\end{array}\right)$$

(50)

This partitioning becomes particularly useful with higher-order derivatives, here and with more complex models, whence we see immediately that

$$\begin{array}{cc}\hfill q_i^{\left(\theta \theta \right)}& =q_i^{\left(\eta \eta \right)}{S}_{\eta }^{\otimes 2},{\tilde{q}}_i^{\left(\theta \theta \right)}={\tilde{q}}_i^{\left(\eta \eta \right)}{S}_{\eta }^{\otimes 2},{\overline{q}}_1^{\left(\theta \theta \right)}={\overline{q}}_1^{\left(\eta \eta \right)}{S}_{\eta }^{\otimes 2}\hfill \\ \hfill q_i^{\left(\eta \eta \right)}& =\left(\begin{array}{c}{0}_{l\times l^2}\\ 2X_i^{\top \otimes 2}\end{array}\right),{\tilde{q}}_i^{\left(\eta \eta \right)}=\left(\begin{array}{c}{0}_{l\times l^2}\\ 2{\overrightarrow{W}}_i^{\top }\end{array}\right),{\overline{q}}_1^{\left(\eta \eta \right)}=\left(\begin{array}{c}{0}_{l\times l^2}\\ 2{\overrightarrow{V}}_X^{\top }\end{array}\right)\hfill \end{array}$$

(51)

At the end of the paper we show that the influence functions for $\widehat{\beta }$ are given by

$$\begin{array}{cc}\hfill d_{\beta ,i}& ={\mu }_i\hfill \\ \hfill d_{\beta ,i_1{i}_2}& =-X_{i_1}^{\top }{V}_X^{-1}{X}_{i_2}{ϵ}_{i_1}{ϵ}_{i_2}\hfill \\ \hfill d_{\beta ,i_1{i}_2{i}_3}& =X_{i_2}^{\top }{V}_X^{-1}{W}_{i_1}{V}_X^{-1}{X}_{i_3}{ϵ}_{i_1}{ϵ}_{i_2}.\hfill \end{array}$$

(52)

These are then used to obtain the approximate moments of $\widehat{\beta }$.

The approximate moments are derived at the end of the paper. There, we see that the second-order bias is

$${\mathrm{Bias}}_2\left[\widehat{\beta }\right]=-{\displaystyle \frac{1}{N}}l{\beta }_0.$$

(53)

We note that the second-order bias is the usual $-l{\beta }_0/N$, which is an alternative way of obtaining the result that the method-of-moments estimator of the variance in a linear regression model has the expected value ${\beta }_0(N-l)/N$.

The components of the approximate MSE are as follows.

$$\begin{array}{cc}\hfill V_{\beta ,1}& =\mathbb{E}\left[d_{\beta ,1}^2\right]=\mathbb{E}\left[{\mu }_1^2\right]\hfill \\ \hfill V_{\beta ,21}& =-l\mathbb{E}\left[{\mu }_1^2\right]\hfill \\ \hfill V_{\beta ,22}& =l^2{\beta }_0^2+2l{\beta }_0^2\hfill \\ \hfill V_{\beta ,23}& =0\hfill \end{array}$$

(54)

and the approximate MSE is given by

$$\begin{array}{cc}\hfill {\mathrm{MSE}}_2\left[\widehat{\beta }\right]& ={\displaystyle \frac{1}{N}}{V}_{\beta ,1}+{\displaystyle \frac{1}{N^2}}\left(2V_{\beta ,21}+V_{\beta ,22}+2V_{\beta ,23}\right)\hfill \\ \hfill & ={\displaystyle \frac{1}{N}}\mathbb{E}\left[{\mu }_1^2\right]+{\displaystyle \frac{1}{N^2}}\left(-2l\mathbb{E}\left[{\mu }_1^2\right]+l^2{\beta }_0^2+2l{\beta }_0^2\right)\hfill \end{array}$$

(55)

The approximate skewness is given by

$${\mathrm{Skew}}_2\left[\widehat{\beta }\right]=S_{\beta ,1}+3S_{\beta ,2}$$

(56)

where

$$S_{\beta ,1}=\mathbb{E}\left[{\mu }_1^3\right],S_{\beta ,2}=-l{V}_{\beta ,1}{\beta }_0-2\mathbb{E}{\left[{ϵ}_1^3\right]}^2\mathbb{E}{\left[X_1\right]}^{\top }{V}_X^{-1}\mathbb{E}\left[X_1\right]$$

(57)

The approximate kurtosis is

$${\mathrm{Kurt}}_2\left[\widehat{\beta }\right]={\displaystyle \frac{1}{N^3}}\left(K_{\beta ,1}+4K_{\beta ,2}+4K_{\beta ,3}+6K_{\beta ,4}\right)$$

(58)

with

$$\begin{array}{cc}\hfill K_{\beta ,1}& =\mathbb{E}\left[{\mu }_1^4\right]-3V_{\beta ,1}^2\hfill \\ \hfill K_{\beta ,2}& =-3l\mathbb{E}\left[{ϵ}_1^2{\mu }_1\right]\mathbb{E}\left[{\mu }_1^2\right]-l{\beta }_0\mathbb{E}\left[{\mu }_1^3\right]-6\mathbb{E}\left[{ϵ}_1{\mu }_1^2\right]\mathbb{E}\left[{ϵ}_1^3\right]\mathbb{E}{\left[X_1\right]}^{\top }{V}_X^{-1}\mathbb{E}\left[X_1\right]\hfill \\ \hfill K_{\beta ,3}& =0\hfill \\ \hfill K_{\beta ,4}& =(l^2+2l)\mathbb{E}\left[{\mu }_1^2\right]{\beta }_0^2+(4l+8){\beta }_0\mathbb{E}{\left[{\mu }_1{ϵ}_1\right]}^2\mathbb{E}\left[X_1^{\top }\right]V_X^{-1}\mathbb{E}\left[X_1\right]\hfill \end{array}$$

(59)

Note that, under symmetry, some terms simplify: $\mathbb{E}\left[{\mu }_1{ϵ}_1\right]=0=\mathbb{E}\left[{ϵ}_2{\mu }_2^2\right]$. Also, when $X_i=1$, the approximate moments correspond to those of the sample variance estimator.

5.3. Two-Stage Least Squares

The 2SLS estimator of a linear model with endogenous regressors fits nicely within this framework. For simplicity, we consider the case where there is only one (endogenous) variable:

$$y_i=Y_i{\beta }_0+{ϵ}_i$$

(60)

where ${ϵ}_i$ is possibly correlated with $Y_i$. There is an $l\times 1$ “instrument” $X_i$ available, with

$$Y_i={\pi }_0^{\top }{X}_i+{\eta }_i$$

(61)

such that $\mathbb{E}\left[X_i(y_i-Y_i^{\top }\beta )\right]=0$ if $\beta ={\beta }_0$, the “true” value of the parameter. We use $\pi$ to denote the nuisance parameter here, with

$$\theta =\left(\begin{array}{c}\pi \\ \beta \end{array}\right).$$

(62)

Let $\theta ={({\widehat{\pi }}^{\top },{\widehat{\beta }}^{\top })}^{\top }$ solve the sample moments:

$${\displaystyle \frac{1}{N}}\sum q_i\left(\widehat{\theta }\right)=0$$

(63)

where

$$q_i\left(\theta \right)=\left(\begin{array}{c}{q}_{\pi ,i}\left(\theta \right)\\ q_{\beta ,i}\left(\theta \right)\end{array}\right)=\left(\begin{array}{c}{X}_i(Y_i-{\pi }^{\top }{X}_i)\\ {\pi }^{\top }{X}_i(y_i-Y_i\beta )\end{array}\right),q_i=\left(\begin{array}{c}{X}_i{\eta }_i\\ g_i{ϵ}_i\end{array}\right),$$

(64)

and

$$g_i={\pi }_0^{\top }{X}_i.$$

(65)

$\widehat{\beta }$ is the usual 2SLS estimator (in this case also an instrumental variables, or IV, estimator). The properties of $\widehat{\pi }$ are those of the usual OLS estimator from the regression of $Y_i$ on $X_i$. Note that l here denotes the dimension of $\pi$, the number of instruments. l factors into the approximate moments of the 2SLS estimator as it did with the moments of the variance estimator associated with a regression model. Let

$$\begin{array}{cc}\hfill {\sigma }_{ϵ}^2& =\mathbb{E}\left[{ϵ}_1^2\right],{\sigma }_{\eta }^2=\mathbb{E}\left[{\eta }_1^2\right],\varrho =\mathbb{E}\left[{ϵ}_1{\eta }_1\right],\hfill \\ \hfill V_X& =\mathbb{E}\left[X_1{X}_1^{\top }\right],V_g=\mathbb{E}\left[g_1^2\right]={\pi }_0^{\top }{V}_X{\pi }_0.\hfill \end{array}$$

(66)

For simplicity, we suppose the $X_i$s are independent of ${ϵ}_i,{\eta }_i$. (This assumption can be relaxed by making moments of the disturbances conditional on the $X_i$s, in which case the results would then be modified with terms such as ${\sigma }_{ϵ}^2{V}_X$ changed to $\mathbb{E}\left[\mathbb{E}\left[{ϵ}_i^2\right|X_i]X_i{X}_i^{\top }\right]$, and so on. This makes the derivations substantially messier without providing much more insight.) We see that

$$\mathrm{\Omega }=\mathbb{E}\left[q_1{q}_1^{\top }\right]=\left(\begin{array}{cc}{\sigma }_{\eta }^2\mathbb{E}\left[X_1{X}_1^{\top }\right]& \varrho \mathbb{E}\left[X_1{g}_1\right]\\ \varrho \mathbb{E}\left[g_1{X}_1^{\top }\right]& {\sigma }_{ϵ}^2\mathbb{E}\left[g_1^2\right]\end{array}\right)=\left(\begin{array}{cc}{\sigma }_{\eta }^2{V}_X& \varrho V_X{\pi }_0\\ \varrho {\pi }_0^{\top }{V}_X& {\sigma }_{ϵ}^2{\pi }_0^{\top }{V}_X{\pi }_0\end{array}\right).$$

(67)

We place many of the derivations at the end of the paper. Note that the influence functions are based on the first three derivatives of $q_i\left(\theta \right)$ (evaluated at ${\theta }_0$):

$$\begin{array}{cc}\hfill q_i^{\left(\pi \right)}\left(\theta \right)& =\left(\begin{array}{c}-X_i{X}_i^{\top }\\ X_i^{\top }(y_i-Y_i\beta )\end{array}\right),q_i^{\left(\beta \right)}\left(\theta \right)=\left(\begin{array}{c}{0}_{l\times 1}\\ -{\pi }^{\top }{X}_i{Y}_i\end{array}\right)\hfill \end{array}$$

(68)

so that $q_i^{\left(\theta \right)}=q_i^{\left(\pi \right)}{S}_{\pi }+q_i^{\left(\beta \right)}{S}_{\beta }$, i.e.,

$$q_i^{\left(\theta \right)}=\left(\begin{array}{cc}-X_i{X}_i^{\top }& 0\\ X_i^{\top }{ϵ}_i& -g_i{Y}_i\end{array}\right),{\overline{q}}_1^{\left(\theta \right)}=-\left(\begin{array}{cc}{V}_X& 0\\ 0& V_g\end{array}\right),Q=-\left(\begin{array}{cc}{V}_X^{-1}& 0\\ 0& V_g^{-1}\end{array}\right)$$

(69)

We have $q_i^{\left(\pi \pi \right)}=0$ and $q_i^{\left(\beta \beta \right)}=0$:

$$\begin{array}{cc}\hfill q_i^{\left(\pi \beta \right)}=q_i^{\left(\beta \pi \right)}& =\left(\begin{array}{c}{0}_{l\times l}\\ -X_i^{\top }{Y}_i\end{array}\right),\hfill \end{array}$$

(70)

so that

$$q_i^{\left(\theta \theta \right)}=\left(\begin{array}{c}{0}_{l\times l}\\ -X_i^{\top }{Y}_i\end{array}\right)(S_{\pi }\otimes S_{\beta })+(S_{\beta }\otimes S_{\pi })$$

(71)

and $q_i^{\left(\theta \theta \theta \right)}\left(\theta \right)=0$.

The influence function for $\widehat{\beta }$ is

$$d_{\beta ,i}=-S_{\beta }Q{q}_i=V_g^{-1}{g}_i{ϵ}_i$$

(72)

Note that the first-order asymptotic variance for $\widehat{\beta }$ is

$$\begin{array}{c}\hfill V_{\beta ,1}=\mathbb{E}\left[d_{\beta ,1}^2\right]={\sigma }_{ϵ}^2{V}_g^{-1}.\end{array}$$

(73)

We derive the higher-order influence functions and approximate moments at the end of the paper. Some additional notation facilitates the presentation and also permits some additional intuition along the way. For a random variable $e_i$, put ${\dot{e}}_i=e_i/\sqrt{V_g}$. Also, denote the scaled second moments: ${\dot{\sigma }}_{ϵ}^2=\mathbb{E}\left[{\dot{ϵ}}_1^2\right]$, ${\dot{\sigma }}_{\eta }^2=\mathbb{E}\left[{\dot{\eta }}_1^2\right]$ and $\dot{\varrho }=\mathbb{E}\left[{\dot{ϵ}}_1{\dot{\eta }}_1\right]$. We note in advance that the approximate moments of $\widehat{\beta }$ can all be written in terms of these (and third and fourth) scaled moments. Scaling streamlines and simplifies the notation.

Scaling the moments also underlines one of the fundamental problems in 2SLS estimation: weak instruments. Since each of these moments is being scaled by multiples of $1/\sqrt{V_g}$ and $V_g={\pi }^{\top }{V}_X\pi$, if $\pi \approx 0$, as is the case with weak instruments, each of the approximate moments (not just the variance) will be large in absolute value. This will make traditional corrections based directly on approximate moments, including bias and variance corrections and Edgeworth and similar analytical approximations, unreliable. Also, to the extent that resampling-based methods such as bootstrap and jackknife are seen as mimicking the properties of analytical corrections, these too will be unreliable. This is to say that the methods explored here, while providing insight into the weak instruments problem, may not be the best approach for inference in that context.

For the 2SLS estimator, the terms entering into the approximate bias are given by

$$B_{\beta ,21}=(l-1)\dot{\varrho },B_{\beta ,22}=-\dot{\varrho }$$

(74)

so that using the standard formula,

$${\mathrm{Bias}}_2\left[\widehat{\beta }\right]={\displaystyle \frac{1}{N}}(B_{\beta ,21}+B_{\beta ,22})=(l-2){\displaystyle \frac{1}{N}}{\displaystyle \frac{\varrho }{V_g}}.$$

(75)

This result has been known since at least Nagar’s [3] paper. We note its dependence on (a), the degree of endogeneity as measured by the correlation $\varrho$ between the two equations in the model; (b) the number of instruments, l; (c) the strength of the instruments as measured by $V_g={\pi }_0^{\top }{V}_X{\pi }_0$; and (d) the sample size. We will see that each of these factors impacts on the other approximate moments. The components of the approximate bias are trivially estimated from the corresponding sample moments: the cross-moment of the residuals from the structural equation and reduced-form equation for the endogenous variable.

The approximate MSE is given by

$${\mathrm{MSE}}_2\left[\widehat{\beta }\right]={\displaystyle \frac{1}{N}}{V}_{\beta ,1}+{\displaystyle \frac{1}{N^2}}(2V_{\beta ,21}+V_{\beta ,23}+2V_{\beta ,22})$$

(76)

with

$$V_{\beta ,1}={\dot{\sigma }}_{ϵ}^2\equiv {\displaystyle \frac{{\sigma }_{ϵ}^2}{V_g}},$$

(77)

being the usual first-order variance for this model, and the higher-order terms are given by

$$\begin{array}{cc}\hfill V_{\beta ,21}& =\mathbb{E}\left[{\dot{ϵ}}_1^2{\dot{\eta }}_1\right](\mathbb{E}\left[{\dot{g}}_1{X}_1^{\top }{V}_X^{-1}{X}_1\right]-2\mathbb{E}\left[{\dot{g}}_1^3\right])-V_{\beta ,1}\mathrm{Var}\left[{\dot{g}}_1^2\right],\hfill \\ \hfill V_{\beta ,22}& =-(l-2){\dot{\sigma }}_{ϵ}^2{\dot{\sigma }}_{\eta }^2-2(l-2){\dot{\varrho }}^2+V_{\beta ,1}\mathrm{Var}\left[{\widehat{g}}_1^2\right]\hfill \\ \hfill V_{\beta ,23}& =B_{\beta ,2}^2+l({\dot{\sigma }}_{ϵ}^2{\dot{\sigma }}_{\eta }^2+{\dot{\varrho }}^2)+V_{\beta ,1}\mathrm{Var}\left[{\dot{g}}_1^2\right].\hfill \end{array}$$

(78)

We note a few things here. Under the cross-symmetry (we use the term “cross-symmetry” to underscore that there is not an assumption that, say, $\mathbb{E}\left[{ϵ}_1^3\right]=0$ or $\mathbb{E}\left[{\eta }_1^3\right]=0$) condition that $\mathbb{E}\left[{\dot{ϵ}}_1^2{\dot{\eta }}_1\right]=0$, the first expression in $V_{\beta ,21}$ drops out. Second, the $\mathrm{Var}\left[{\widehat{g}}_1^2\right]$ terms in $V_{\beta ,21}$ and $V_{\beta ,22}$ cancel, leaving one $V_{\beta ,1}\mathrm{Var}\left[{\dot{g}}_1^2\right]$ term in the approximate MSE. This may be attributed to the random $X_i$s. If these are assumed “fixed”, “fixed” regressors have no variance and this term drops out (the same holds if we derive results conditional on the $X_i$s). Rothenberg [15] derived an expression for the approximate MSE under normality and fixed regressors. The expression here for MSE conforms with his under-symmetry and fixed regressors. It is useful to note therefore that the second-order variance result in Rothenberg has robustness against departures from normality, so long as the departures satisfy the cross-symmetry condition.

The approximate skewness is given by

$$\begin{array}{cc}\hfill {\mathrm{Skew}}_2\left[\widehat{\beta }\right]& =S_{\beta ,1}+3S_{\beta ,2}\hfill \end{array}$$

(79)

with

$$S_{\beta ,1}=\mathbb{E}\left[{\dot{g}}_1^3\right]\mathbb{E}\left[{\dot{ϵ}}_1^3\right],S_{\beta ,2}=V_{\beta ,1}{B}_{\beta ,2}-2{\dot{\sigma }}_{ϵ}^2\dot{\varrho }.$$

(80)

Note that the first term is simply the third moment of the influence function, which can be simply estimated from sample moments. The other terms are comprised of terms which already occur in the first-order variance and second-order bias. Hence, estimating skewness is a trivial step when conducting standing inferences.

The approximate kurtosis is

$${\mathrm{Kurt}}_2\left[\widehat{\beta }\right]={\displaystyle \frac{1}{N^3}}\left(K_{\beta ,1}+4K_{\beta ,2}+4K_{\beta ,3}+6K_{\beta ,4}\right)$$

(81)

with

$$\begin{array}{cc}\hfill K_{\beta ,1}& =\mathrm{Var}\left[{\dot{g}}_1^2\right]\mathbb{E}\left[{\dot{ϵ}}_1^4\right]+(\mathbb{E}\left[{\dot{ϵ}}_1^4\right]-3{\left({\dot{\sigma }}_{ϵ}^2\right)}^2)\hfill \\ \hfill K_{\beta ,2}& =S_{\beta ,1}{B}_{\beta ,2}+3({\dot{\sigma }}_{ϵ}^2{V}_{\beta ,21}-\mathbb{E}\left[{\dot{g}}_1^3\right]\mathbb{E}\left[{\dot{ϵ}}_1^2{\dot{\eta }}_1\right]{\dot{\sigma }}_{ϵ}^2-S_{\beta ,1}\dot{\varrho }-\mathrm{Var}\left[{\dot{g}}_1^2\right]{\dot{\sigma }}_{ϵ}^2\dot{\varrho })\hfill \\ \hfill K_{\beta ,3}& =3V_{\beta ,1}(V_{\beta ,22}+2{\dot{\rho }}^2)\hfill \\ \hfill K_{\beta ,4}& =V_{\beta ,1}(V_{\beta ,23}-4\dot{\varrho }{B}_{\beta ,2}+2{\dot{\sigma }}_{ϵ}^2{\dot{\sigma }}_{\eta }^2+6{\dot{\varrho }}^2)\hfill \end{array}$$

(82)

Note that all but one of the terms in the approximate kurtosis appear in the lower-order approximate moments. The only other term, $\mathbb{E}\left[{\dot{ϵ}}_1^4\right]$, can be simply estimated by its empirical analogue of the residuals.

6. An Alternative Approach

In this section, we outline an alternative approach which can be used in the case of two-step estimation. This alternative method is useful if we wish to remain agnostic with respect to the manner in which the nuisance parameter is estimated. This may be seen as an advantage, although it begs the question as to the provenance of the first-step estimator.

The setup in this case is as follows. The parameter of interest is the solution to the subset of moment conditions applying to the estimation of the parameters of interest. Let

$${\psi }_{\beta }(\eta ,\beta )={\displaystyle \frac{1}{N}}\sum q_{\beta ,i}\left(\theta \right)$$

(83)

$$q_{\beta ,i}\left(\theta \right)=q(Z_i;\theta ),\theta =(\eta ,\beta )$$

(84)

where $\mathbb{E}\left[q_{\beta ,i}\left(\theta \right)\right]=0$ only at $\theta ={\theta }_0=({\eta }_0,{\beta }_0)$. The estimator, $\widehat{\beta }$, is the solution to the $k\times 1$ set of equations ${\psi }_{\beta }(\widehat{\eta },\widehat{\beta })=0$. This may appear to be the same setup as previously, and to all intents and purposes it is, except we simply make additional assumptions on $\widehat{\eta }$ and on $q_{\beta ,i}\left(\theta \right)$ in some neighborhood of ${\theta }_0$. If we were simply interested in consistency, we might, e.g., assume that $\widehat{\eta }\stackrel{p}{\to }{\eta }_0$. To obtain first-order asymptotic results we may additionally assume that

$$\widehat{\eta }-{\eta }_0=c_{-1/2}+o_P\left(N^{-1/2}\right),c_{-1/2}={\displaystyle \frac{1}{N}}\sum W_i$$

(85)

where $W_i$ is mean-zero independent and identically distributed vectors or matrices and that we are able to perform a linearization of ${\psi }_{\beta }(\widehat{\eta },\beta )$ uniformly in some neighborhood of ${\theta }_0$. For first-order asymptotics, this may be expedient. For example, if we have established the consistency of $\widehat{\eta }$ and $\widehat{\beta }$, we can often write

$$\begin{array}{cc}\hfill 0& ={\psi }_{\beta }(\widehat{\eta },\widehat{\beta })\hfill \\ \hfill & ={\psi }_{\beta }({\eta }_0,\widehat{\beta })+F_0{c}_{-1/2}+o_P\left(N^{-1/2}\right)\hfill \\ \hfill & ={\psi }_{\beta }({\eta }_0,{\beta }_0)+{\psi }_{\beta }^{\left(\beta \right)}({\eta }_0,{\beta }_0)(\widehat{\beta }-{\beta }_0)+F_0{c}_{-1/2}+o_P\left(N^{-1/2}\right)\hfill \end{array}$$

(86)

where $F_0$ is a constant $k\times l$ matrix which leads to the first-order approximation

$$\widehat{\beta }-{\beta }_0={\displaystyle \frac{1}{N}}\sum d_i+o_P\left(N^{-1/2}\right)$$

(87)

where

$$d_i=-Q(q_i+F_0{W}_i),Q={\left({\overline{q}}_{\beta ,1}^{\left(\beta \right)}({\eta }_0,{\beta }_0)\right)}^{-1},q_i=q_i({\eta }_0,{\beta }_0)$$

(88)

Under standard regularity conditions $\sqrt{N}(\widehat{\beta }-{\beta }_0)\stackrel{d}{\to }N(0,V)$, where

$$\begin{array}{cc}\hfill V& =\mathbb{E}\left[d_1{d}_1^{\top }\right]\hfill \\ \hfill & =Q(\mathbb{E}\left[q_i{q}_i^{\top }\right]+\mathbb{E}\left[q_i{W}_i^{\top }\right]F_0^{\top }+F_0\mathbb{E}\left[W_i{q}_i^{\top }\right]+F_0\mathbb{E}\left[W_i{W}_i^{\top }\right]F_0)Q^{\top }.\hfill \end{array}$$

The exact form of V depends on the variance of $W_i$, the correlation of $W_i$ with $q_i$, and the form of $F_0$. To say anything meaningful, one needs to be more specific about $W_i$, which typically requires a more complete statement regarding how $\widehat{\eta }$ was obtained. As Wooldridge [13] points out, this generally results in a statement corresponding to a set of moment conditions (first-order conditions) such as we started out with in the second section. Wooldridge has a good summary of the issues involved. Typically, $F_0$ is defined as the limiting value of the derivative: ${\psi }_{\beta }^{\left(\eta \right)}(\eta ,\beta )$ evaluated at ${\eta }_0,{\beta }_0$: $F_0={\overline{q}}_{\beta ,1}^{\left(\eta \right)}$. Also, since $\widehat{\eta }$ can typically be framed as the solution to a set of l estimating equations: ${\displaystyle \frac{1}{N}}{r}_i\left(\widehat{\eta }\right)=0$, then $W_i$ can be thought of as the influence function for $\widehat{\eta }$: $W_i=-{\left(\mathbb{E}\left[r_i^{\left(\eta \right)}\left(\eta \right)\right]\right)}^{-1}{r}_i$ if we are willing to assume standard regularity conditions hold. We thus have

$$\begin{array}{cc}\hfill F_0& ={\overline{q}}_{\beta ,1}^{\left(\eta \right)}({\eta }_0,{\beta }_0)\hfill \\ \hfill W_i& =-{\left(\mathbb{E}\left[r_i^{\left(\eta \right)}\left({\eta }_0\right)\right]\right)}^{-1}{r}_i\hfill \end{array}$$

(89)

Now, when we substitute these expressions into V, we see that, with  $r_i=q_{\eta ,i}$, we obtain the same expression for the variance as in Equation (13). Thus, from a practical perspective, nothing is gained in the models we examine by using a generic approach, at least from the perspective of first-order asymptotics.

In terms of conducting higher-order asymptotics, it is possible to obtain “generic” higher-order stochastic expansions in the following way. First, assume that we have expansions $\widehat{\eta }-{\eta }_0={\widehat{\eta }}_j+O_p\left(N^{-(j+1)/2}\right)$:

$${\widehat{\eta }}_j=c_{-1/2}+\cdots c_{-j/2}$$

(90)

where each $c_{-r/2}$ is an rth-order V-statistic, $c_{-1/2}$ is as above, and

$$c_{-1}={\displaystyle \frac{1}{N^2}}\sum _{i_1}\sum _{i_2}{W}_{i_1{i}_2},c_{-3/2}={\displaystyle \frac{1}{N^3}}\sum _{i_1}\sum _{i_2}\sum _{i_3}{W}_{i_1{i}_2{i}_3}$$

(91)

Second, write down a higher-order Taylor series expansion of $\psi (\widehat{\eta },\widehat{\beta })$ as a function of $\widehat{\eta }$ and $\widehat{\beta }$, iteratively solving for $\widehat{\beta }-{\beta }_0$.

Third, replace those terms in the Taylor series containing $\widehat{\eta }-{\eta }_0$ with the approximations in Equation (90). This results in a generic stochastic expansion. Some explicit results can be tweaked out of this once one is more specific about the nature of the Ws. Newey and Smith [2] use this approach to obtain the bias of GMM estimators. We find that it becomes more difficult to examine higher-order moments in this manner and it is simpler to specify the first-step as a method-of-moments estimator. The method here for two-step estimation is naturally extended to three- or more step estimation, which permits examination of iterated estimators, as are common with GMM estimation.

7. Numerical Illustration

We provide a brief numerical illustration based on the 2SLS framework in Section 5.3. The 2SLS estimator in the case examined here is also an IV estimator. Variations of the bias result in Equation (75) were obtained by Nagar [3] for fixed $X_i$s and Rilstone et al. [5] for random $X_i$s. Nagar’s result has the sample analog, with ${\displaystyle \frac{1}{N}}\sum X_i^2$ rather than $\mathbb{E}\left[X_i^2\right]$, say, appearing in the denominator. The importance of this simple result and others related to it cannot be over-emphasized. The sampling properties of 2SLS/IV estimators, and the difference of these from those predicted by first-order asymptotic theory in different contexts, have been examined extensively in econometrics. Note that the approximate bias in the model is a function of (i) the degree of endogeneity, measured by $\rho$; and (ii) the explanatory power of the instruments, as measured by $V_g={\pi }_0^{\top }{V}_X\pi$ (low values of ${\pi }_0^{\top }{V}_X\pi$ correspond to “weak” instruments) as well as the number of regressors.

With respect to the first problem, the methods explored here can perform well in correcting bias and other moments. Numerous studies have performed detailed simulations to examine the performance of bias approximations. To illustrate approximate bias, we conducted simulations following the basic 2SLS/IV framework with one instrument, $X_i$. (Note that this is for illustrative purposes only. Moments such as the actual bias may not exist and moments approximating to, say, $\widehat{\beta }$, are based on those of the random variable (the stochastic expansion which is close in the probabilistic sense to $\widehat{\beta }$).) We drew $N=50$ observations with the $X_i$s from the fixed design $X_i=1+i/N$, the  ${\eta }_i$ from $N(0,{\sigma }_{\eta })$, and ${ϵ}_i=\rho {\eta }_i+u_i$, with $u_i\sim N(0,{\sigma }_u)$. ${\sigma }_{ϵ}=1$, ${\sigma }_{\eta }=1$, ${\beta }_0=0$, and ${\pi }_0=1$. We varied the measure of endogeneity, $\rho$, as per the table. The table reports the average over $M=10,000$ replications of the usual 2SLS/IV estimator; the corrected version, using the analytic formula; and a bias-corrected version, where actual moments have been replaced with sample moments. We see that both bias corrections can yield substantial improvements in the point estimates. We also provide plots of the (standardized (estimates are divided by the asymptotic standard error)) distribution of these estimates (these densities are actually kernel estimates using standard normal kernel and window width $\gamma =M^{-1/5}{\sigma }_{\beta }$, ${\sigma }_{\beta }^2=\mathrm{Var}\left[\widehat{\beta }\right]$) for the case with $\rho =1$ (plots of the corresponding densities of the 2SLS/IV estimators with values of $\rho =0.5,0.75$ are similar) against N(0, 1).

On the other hand, when the underlying problem is that of weak instruments, the issue is more fundamental (one way to see this is that when $\pi \approx 0$, (1) estimates based on $B_2$ will be unreliable and (2) we will see that the same issue crops up in all approximate moments of $\widehat{\beta }$, making them all unreliable and in fact undermining standard analytical asymptotic expansions and resampling methods which are justified by standard analytical results) and the analytical methods in this paper are not necessarily that helpful. The problem of weak instruments was emphasized in the influential articles by Nelson and Startz [16] and Bound et al. [17]. In the case when ${\pi }_0=0$, ${\beta }_0$ is in fact unidentified, and the sorts of corrections here are not appropriate (note that even the usual asymptotic variance of the 2SLS estimator explodes as ${\pi }_0^{\top }{V}_X\pi \to 0$). Much subsequent literature followed, including Staiger and Stock [18], who considered situations of near non-identification (${\pi }_0\approx 0$). In the Staiger and Stock [18] approach, the concentration parameter ${\mu }^2$ is defined (in our notation) as ${\mu }_0^2={\pi }_0^{\top }{V}_X\pi N/{\sigma }_v^2$. Their expansions are then performed as $\mu$ increases, setting $\mu =C{N}^{-1/2}$, with C a positive constant, similar to expansions for local power. When the primary concern to a researcher is weak instruments, these alternatives and various extensions are more likely to provide improved inferences rather than simple moment corrections (although traditional analytical techniques (and resampling methods as well) may perform poorly in the weak instrument case when applied to standard inference procedures, Kleibergen [19] has shown that this is not the case when these corrections are applied to tests which are not dependent on strong instruments) (see

Table 1. Average of OLS, IV, and two bias-corrected IV estimators.

$\mathit{\rho }$	OLS	$\widehat{\mathit{\beta }}$	${\widehat{\mathit{\beta }}}_{\mathbf{BC}}^{*}$	${\widehat{\mathit{\beta }}}_{\mathbf{BC}}$
0.50	0.3733	−0.0342	−0.0050	0.0234
0.75	0.5591	−0.0531	−0.0095	0.0341
1.00	0.7449	−0.0721	−0.0139	0.0447

and Figure 1).

8. Conclusions

This paper has developed methods for deriving higher-order stochastic expansions and approximate moments for the estimators of the parameters of interest for models in which there are additional nuisance parameters. These methods can be useful in a number of ways. They can be useful for computing the approximate moments. An inspection of the expansions and approximate moments may indicate when these should be taken into account. For example, in the case where the nuisance parameter estimate may not affect the first-order distribution of the estimate of the parameter of interest, but does affect its higher-order moments. In any case, the results may well indicate when any first-order approximation may be deficient. The Edgeworth and related expansions for the estimators of the parameters of interest can be derived immediately from the approximate moments.

This paper also sets the stage for examination of some more difficult estimators which may not appear to fit into the framework of estimating m parameters with some estimating equations. Two-stage least squares and certain other forms of GMM estimators based on an initial consistent estimate can be fit into the two-step paradigm. However, other estimators such as three-stage least squares and other iterated GMM estimators require more stages. This can be extended to GMM estimators by extending the two-step approach by stacking equations and creating additional nuisance parameters. The notation developed here is helpful in that regard. Somewhat different are certain “harder” problems such as GEL, where there may be equal numbers of parameters and estimating equations, but the relevant information regarding the parameters of interest is not easily retrieved. The techniques developed in this paper set the stage for that.

9. Derivations

This section provides the details for the two-step illustrations in the paper. Throughout, when summations are taken over a set denoted by $\sigma \left(\right)$, this is taken to refer to the set of unique permutations of the given arguments. Specifically, the following sets of permutations of subscripts are used below.

$$\begin{array}{cc}\hfill \sigma \left(1122\right)& =\left\{\right(1122),(1221),(1212\left)\right\},\hfill \\ \hfill \sigma \left(11122\right)& =\left\{\right(11122),(11212),(11221),(12112),(12121),(12211\left)\right\},\hfill \\ \hfill \sigma \left(11222\right)& =\left\{\right(11222),(12122),(12212),(12221\left)\right\},\hfill \\ \hfill \sigma \left(112233\right)& =\left\{\right(112233),(112323),(112332),(121233),(121323),\hfill \\ \hfill & \left(121332\right),\left(122133\right),\left(123123\right),\left(123132\right),\left(123213\right),\hfill \\ \hfill & \left(122313\right),\left(123312\right),\left(123231\right),\left(123321\right),\left(122331\right)\}.\hfill \end{array}$$

(92)

A more detailed discussion can be found in Rilstone [11].

• Regression-Variance Problem

We first derive the influence functions for the parameter of interest, $\beta$, recalling the definition ${\mu }_i={ϵ}_i^2-{\beta }_0$. Note here that $S_{\beta }Q=\left(\begin{array}{cc}0& -1\end{array}\right)$ and $S_{\eta }Q=\left(\begin{array}{cc}-V_X^{-1}& 0\end{array}\right)$. The influence function for $\widehat{\theta }$ is

$$\begin{array}{cc}\hfill d_i& =-Q{q}_i\hfill \\ \hfill & =-\left(\begin{array}{cc}-V_X^{-1}& 0\\ 0& -1\end{array}\right)\left(\begin{array}{c}{X}_i{ϵ}_i\\ {\mu }_i\end{array}\right)\hfill \\ \hfill & =\left(\begin{array}{c}{V}_X^{-1}{X}_i{ϵ}_i\\ {\mu }_i\end{array}\right),\hfill \\ \hfill d_i^{\left(\theta \right)}& =-Q{q}_i^{\left(\theta \right)}\hfill \\ \hfill & =-\left(\begin{array}{cc}-V_X^{-1}& 0\\ 0& -1\end{array}\right)\left(\begin{array}{cc}-X_i{X}_i^{\top }& 0\\ -2{ϵ}_i{X}_i^{\top }& -1\end{array}\right)\hfill \\ \hfill & =-\left(\begin{array}{cc}{V}_X^{-1}{X}_i{X}_i^{\top }& 0\\ 2{ϵ}_i{X}_i^{\top }& 1\end{array}\right)\hfill \\ \hfill {\tilde{d}}_i^{\left(\theta \right)}& =-\left(\begin{array}{cc}{V}_X^{-1}{W}_i& 0\\ 2{ϵ}_i{X}_i^{\top }& 0\end{array}\right)\hfill \end{array}$$

(93)

 $W_i=X_i{X}_i^{\top }-V_X$.

$$\begin{array}{cc}\hfill d_i^{\left(\theta \theta \right)}& =-Q{q}_i^{\left(\theta \theta \right)}\hfill \\ \hfill & =-\left(\begin{array}{cc}-V_X^{-1}& 0\\ 0& -1\end{array}\right)\left(\begin{array}{cccc}0& 0& 0& 0\\ 2X_i^{\top \otimes 2}& 0& 0& 0\end{array}\right)\hfill \\ \hfill & =\left(\begin{array}{cccc}0& 0& 0& 0\\ 2X_i^{\top \otimes 2}& 0& 0& 0\end{array}\right)\hfill \\ \hfill {\overline{d}}_1^{\left(\theta \theta \right)}& =\left(\begin{array}{cccc}0& 0& 0& 0\\ 2{\overrightarrow{V}}_X^{\top }& 0& 0& 0\end{array}\right),\hfill \\ \hfill {\tilde{d}}_1^{\left(\theta \theta \right)}& =\left(\begin{array}{cccc}0& 0& 0& 0\\ 2{\overrightarrow{W}}_i^{\top }& 0& 0& 0\end{array}\right)\hfill \\ \hfill d_i^{\left(\theta \theta \theta \right)}& =0.\hfill \end{array}$$

(94)

So,

$$\begin{array}{cc}\hfill d_{\beta ,i}& =S_{\beta }{d}_i\hfill \\ \hfill & ={\mu }_i\hfill \end{array}$$

(95)

$$\begin{array}{cc}\hfill d_{\beta ,i_1{i}_2}& =S_{\beta }{d}_{\beta i_1{i}_2}\hfill \\ \hfill & =\sum _{s=1}^2{A}_{\beta ,i_1{i}_2}\left(s\right)\hfill \\ \hfill A_{\beta ,i_1{i}_2}\left(1\right)& =S_{\beta }{\tilde{d}}_{i_1}^{\left(\theta \right)}{d}_{i_2}\hfill \\ & =S_{\beta }\left(\begin{array}{cc}-V_X^{-1}{W}_{i_1}^{\top }& 0\\ -2{ϵ}_{i_1}{X}_{i_1}^{\top }& 0\end{array}\right)\left(\begin{array}{c}{V}_X^{-1}{X}_{i_2}{ϵ}_{i_2}\\ {\mu }_{i_2}\end{array}\right)\hfill \\ \hfill & =S_{\beta }\left(\begin{array}{c}-V_X^{-1}{W}_{i_1}^{\top }{V}_X^{-1}{X}_{i_2}{ϵ}_{i_2}\\ -2{ϵ}_{i_1}{X}_{i_1}^{\top }{V}_X^{-1}{X}_{i_2}{ϵ}_{i_2}\end{array}\right)\hfill \\ \hfill & =-2X_{i_1}^{\top }{V}_X^{-1}{X}_{i_2}{ϵ}_{i_1}{ϵ}_{i_2}\hfill \\ \hfill A_{\beta ,i_1{i}_2}\left(2\right)& =S_{\beta }{A}_{i_1{i}_2}\left(2\right)\hfill \\ \hfill & =S_{\beta }{\displaystyle \frac{1}{2}}{\overline{d}}_1^{\left(\theta \theta \right)}(d_{i_1}\otimes d_{i_2})\hfill \\ \hfill & ={\displaystyle \frac{1}{2}}{S}_{\beta }\left(\begin{array}{c}0\\ 2{\overrightarrow{V}}_X^{\top }\end{array}\right)S_{\eta }^{\top \otimes 2}\left(\left(\begin{array}{c}{V}_X^{-1}{X}_{i_1}{ϵ}_{i_1}\\ {\mu }_{i_1}\end{array}\right)\otimes \left(\begin{array}{c}{V}_X^{-1}{X}_{i_2}{ϵ}_{i_2}\\ {\mu }_{i_2}\end{array}\right)\right)\hfill \\ \hfill & ={\overrightarrow{V}}_X^{\top }\left(V_X^{-1}{X}_{i_1}{ϵ}_{i_1}\otimes V_X^{-1}{X}_{i_2}{ϵ}_{i_2}\right)\hfill \\ \hfill & =\overrightarrow{[}{X}_{i_1}^{\top }{V}_X^{-1}{V}_X^{\top }{V}_X^{-1}{X}_{i_2}]{ϵ}_{i_1}{ϵ}_{i_2}\hfill \\ \hfill & =X_{i_1}^{\top }{V}_X^{-1}{X}_{i_2}{ϵ}_{i_1}{ϵ}_{i_2}.\hfill \end{array}$$

(96)

Therefore,

$$\begin{array}{cc}\hfill d_{\beta ,i_1{i}_2}& =-X_{i_1}^{\top }{V}_X^{-1}{X}_{i_2}{ϵ}_{i_1}{ϵ}_{i_2}\hfill \end{array}$$

(97)

and

$$\begin{array}{cc}\hfill d_{\beta ,i_1{i}_2{i}_3}& =\sum _{s=1}^6{E}_{\beta ,i_1{i}_2{i}_3}\left(s\right),\hfill \end{array}$$

$$\begin{array}{cc}\hfill E_{\beta ,i_1{i}_2{i}_3}\left(1\right)& =S_{\beta }{\overline{d}}_1^{\left(\theta \theta \right)}(d_{i_1}\otimes {\tilde{d}}_{i_2}^{\left(\theta \right)}{d}_{i_3}),\hfill \\ \hfill & =S_{\beta }\left(\begin{array}{c}0\\ 2{\overrightarrow{V}}_X^{\top }\end{array}\right)S_{\eta }^{\top \otimes 2}\left(\left(\begin{array}{c}{V}_X^{-1}{X}_{i_1}{ϵ}_{i_1}\\ {\mu }_{i_1}\end{array}\right)\otimes \left(\begin{array}{c}-V_X^{-1}{W}_{i_2}^{\top }{V}_X^{-1}{X}_{i_3}{ϵ}_{i_3}\\ -2{ϵ}_{i_2}{X}_{i_2}^{\top }{V}_X^{-1}{X}_{i_3}{ϵ}_{i_3}\end{array}\right)\right)\hfill \\ \hfill & =2{\overrightarrow{V}}_X^{\top }\left(V_X^{-1}{X}_{i_1}{ϵ}_{i_1}\otimes (-V_X^{-1}{W}_{i_2}^{\top }{V}_X^{-1}{X}_{i_3}{ϵ}_{i_3})\right)\hfill \\ \hfill & =-2{\overrightarrow{V}}_X^{\top }\left(V_X^{-1}{X}_{i_1}\otimes V_X^{-1}{W}_{i_2}^{\top }{V}_X^{-1}{X}_{i_3}\right){ϵ}_{i_1}{ϵ}_{i_3}\hfill \\ \hfill & =-2\overrightarrow{[}{X}_{i_1}^{\top }{V}_X^{-1}{V}_X{V}_X^{-1}{W}_{i_2}^{\top }{V}_X^{-1}{X}_{i_3}]{ϵ}_{i_1}{ϵ}_{i_3}\hfill \\ \hfill & =-2X_{i_1}^{\top }{V}_X^{-1}{W}_{i_2}^{\top }{V}_X^{-1}{X}_{i_3}{ϵ}_{i_1}{ϵ}_{i_3}\hfill \end{array}$$

(98)

$$\begin{array}{cc}\hfill E_{\beta ,i_1{i}_2{i}_3}\left(2\right)& ={\displaystyle \frac{1}{2}}{S}_{\beta }{\overline{d}}_1^{\left(\theta \theta \right)}(d_{i_1}\otimes {\overline{d}}_1^{\left(\theta \theta \right)}(d_{i_2}\otimes d_{i_3}))\hfill \\ \hfill & ={\displaystyle \frac{1}{2}}2{\overrightarrow{V}}_X^{\top }{S}_{\eta }^{\otimes 2}\left(d_{i_1}\otimes \left(\begin{array}{c}0\\ {\overrightarrow{V}}_X^{\top }\end{array}\right)S_{\eta }^{\otimes 2}\left(d_{i_2}\otimes d_{i_3}\right)\right)\hfill \\ & =0\hfill \end{array}$$

(99)

$$E_{\beta ,i_1{i}_2{i}_3}\left(3\right)=S_{\beta }{\displaystyle \frac{1}{6}}{\overline{d}}_1^{\left(\theta \theta \theta \right)}(d_{i_1}\otimes d_{i_2}\otimes d_{i_3})=0.$$

(100)

$$\begin{array}{cc}\hfill E_{\beta ,i_1{i}_2{i}_3}\left(4\right)& =S_{\beta }{\tilde{d}}_{i_1}^{\left(\theta \right)}{\tilde{d}}_{i_2}^{\left(\theta \right)}{d}_{i_3}\hfill \\ \hfill & =S_{\beta }\left(\begin{array}{cc}{V}_X^{-1}{W}_{i_1}^{\top }& 0\\ 2{ϵ}_{i_1}{X}_{i_1}^{\top }& 0\end{array}\right)\left(\begin{array}{cc}{V}_X^{-1}{W}_{i_2}^{\top }& 0\\ 2{ϵ}_{i_2}{X}_{i_2}^{\top }& 0\end{array}\right)\left(\begin{array}{c}{V}_X^{-1}{X}_{i_3}{ϵ}_{i_3}\\ {\mu }_{i_3}\end{array}\right)\hfill \\ \hfill & =\left(\begin{array}{cc}2{ϵ}_{i_1}{X}_{i_1}^{\top }& 0\end{array}\right)\left(\begin{array}{c}{V}_X^{-1}{W}_{i_2}^{\top }{V}_X^{-1}{X}_{i_3}{ϵ}_{i_3}\\ 2{ϵ}_{i_2}{X}_{i_2}^{\top }{V}_X^{-1}{X}_{i_3}{ϵ}_{i_3}\end{array}\right)\hfill \\ \hfill & =2{ϵ}_{i_1}{X}_{i_1}^{\top }{V}_X^{-1}{W}_{i_2}^{\top }{V}_X^{-1}{X}_{i_3}{ϵ}_{i_3}\hfill \\ \hfill & =-2E_{\beta ,i_1{i}_2{i}_3}\left(1\right).\hfill \end{array}$$

(101)

$$\begin{array}{cc}\hfill E_{\beta ,i_1{i}_2{i}_3}\left(5\right)& =S_{\beta }{\displaystyle \frac{1}{2}}{\tilde{d}}_{i_1}^{\left(\theta \right)}{\overline{d}}_1^{\left(\theta \theta \right)}(d_{i_2}\otimes d_{i_3})\hfill \\ \hfill & ={\displaystyle \frac{1}{2}}\left(\begin{array}{cc}-2{ϵ}_{i_1}{X}_{i_1}^{\top }& 0\end{array}\right)\left(\begin{array}{c}0\\ {\overrightarrow{V}}_X^{\top }\end{array}\right)S_{\eta }^{\otimes 2}(d_{i_2}\otimes d_{i_3})\hfill \\ \hfill & =0.\hfill \end{array}$$

(102)

$$\begin{array}{cc}\hfill E_{\beta ,i_1{i}_2{i}_3}\left(6\right)& =S_{\beta }{\displaystyle \frac{1}{2}}{\tilde{d}}_{i_1}^{\left(\theta \theta \right)}(d_{i_2}\otimes d_{i_3})\hfill \\ \hfill & =S_{\beta }{\displaystyle \frac{1}{2}}\left(\begin{array}{c}0\\ 2{\overrightarrow{W}}_{i_1}^{\top }\end{array}\right)S_{\eta }^{\otimes 2}\left(\left(\begin{array}{c}{V}_X^{-1}{X}_{i_2}{ϵ}_{i_2}\\ {\mu }_{i_2}\end{array}\right)\otimes \left(\begin{array}{c}{V}_X^{-1}{X}_{i_3}{ϵ}_{i_3}\\ {\mu }_{i_3}\end{array}\right)\right).\hfill \\ \hfill & ={\overrightarrow{W}}_{i_1}^{\top }\left(V_X^{-1}{X}_{i_2}{ϵ}_{i_2}\otimes V_X^{-1}{X}_{i_3}{ϵ}_{i_3}\right).\hfill \\ \hfill & =X_{i_2}^{\top }{V}_X^{-1}{W}_{i_1}{V}_X^{-1}{X}_{i_3}{ϵ}_{i_2}{ϵ}_{i_3}.\hfill \end{array}$$

(103)

We thus have

$$d_{\beta ,i_1{i}_2{i}_3}=\sum _{s=1}^6{E}_{\beta ,i_1{i}_2{i}_3}\left(s\right)=X_{i_2}^{\top }{V}_X^{-1}{W}_{i_1}{V}_X^{-1}{X}_{i_3}{ϵ}_{i_2}{ϵ}_{i_3}.$$

(104)

We examine now the approximate moments of $\widehat{\beta }$. We obtain the approximate bias as ($N^1$ times) the expected value of the second-order influence function.

$$\begin{array}{cc}\hfill \mathbb{E}\left[d_{\beta ,11}\right]& =-\mathbb{E}\left[X_1^{\top }{V}_X^{-1}{X}_1\right]\mathbb{E}\left[{ϵ}_1^2\right]=-l{\sigma }^2.\hfill \end{array}$$

(105)

We obtain the components of the approximate MSE as follows.

$$\begin{array}{cc}\hfill V_{\beta ,1}& =\mathbb{E}\left[d_{\beta ,1}^2\right]=\mathbb{E}\left[{\mu }_1^2\right].\hfill \end{array}$$

(106)

$$\begin{array}{cc}\hfill V_{\beta ,21}& =\mathbb{E}\left[d_{\beta ,1}{d}_{\beta ,11}\right]\hfill \\ \hfill & =\mathbb{E}\left[{\mu }_1(-X_1^{\top }{V}_X^{-1}{X}_1{ϵ}_1{ϵ}_1)\right]\hfill \\ \hfill & =-\mathbb{E}\left[{\mu }_1{ϵ}_1^2\right]\mathbb{E}\left[X_1^{\top }{V}_X^{-1}{X}_1\right]\hfill \\ \hfill & =-l\mathbb{E}\left[{\mu }_1{ϵ}_1^2\right]\hfill \\ \hfill & =-l\mathbb{E}\left[{\mu }_1^2\right].\hfill \end{array}$$

(107)

$$\begin{array}{cc}\hfill V_{\beta ,22}& =\mathbb{E}\left[d_{\beta ,11}\right]\mathbb{E}\left[d_{\beta ,11}\right]+\mathbb{E}\left[d_{\beta ,12}{d}_{\beta ,12}\right]+\mathbb{E}\left[d_{\beta ,12}{d}_{\beta ,21}\right]\hfill \\ & =\mathbb{E}\left[X_1^{\top }{V}_X^{-1}{X}_1{ϵ}_1{ϵ}_1\right]\mathbb{E}\left[X_1^{\top }{V}_X^{-1}{X}_1{ϵ}_1{ϵ}_1\right]+\mathbb{E}\left[X_1^{\top }{V}_X^{-1}{X}_2{ϵ}_1{ϵ}_2{X}_1^{\top }{V}_X^{-1}{X}_2{ϵ}_1{ϵ}_2\right]\hfill \\ & +\mathbb{E}\left[X_1^{\top }{V}_X^{-1}{X}_2{ϵ}_1{ϵ}_2{X}_2^{\top }{V}_X^{-1}{X}_1{ϵ}_2{ϵ}_1\right]\hfill \\ & =l^2{\beta }_0^2+\mathbb{E}\left[X_2^{\top }{V}_X^{-1}{X}_1{X}_1^{\top }{V}_X^{-1}{X}_2\right]{\beta }_0^2+\mathbb{E}\left[X_1^{\top }{V}_X^{-1}{X}_1\right]{\beta }_0^2\hfill \\ \hfill & =l^2{\beta }_0^2+2l{\beta }_0^2.\hfill \end{array}$$

(108)

$$\begin{array}{cc}\hfill V_{\beta ,23}& =\mathbb{E}\left[d_{\beta ,1}{d}_{\beta ,122}\right]+\mathbb{E}\left[d_{\beta ,1}{d}_{\beta ,212}\right]+\mathbb{E}\left[d_{\beta ,1}{d}_{\beta ,221}\right]\hfill \\ \hfill & =\mathbb{E}\left[{\mu }_1{X}_2^{\top }{V}_X^{-1}{W}_1{V}_X^{-1}{X}_2{ϵ}_2{ϵ}_2\right]+\mathbb{E}\left[{\mu }_1{X}_1^{\top }{V}_X^{-1}{W}_2{V}_X^{-1}{X}_2{ϵ}_1{ϵ}_2\right]\hfill \\ & +\mathbb{E}\left[{\mu }_1{X}_2^{\top }{V}_X^{-1}{W}_2{V}_X^{-1}{X}_1{ϵ}_2{ϵ}_1\right]\hfill \\ & =0.\hfill \end{array}$$

(109)

so that the approximate variance is given by

$$\begin{array}{cc}\hfill {\mathrm{MSE}}_2\left[\widehat{\beta }\right]& ={\displaystyle \frac{1}{N}}{V}_{\beta ,1}+{\displaystyle \frac{1}{N^2}}\left(2V_{\beta ,21}+V_{\beta ,22}+2V_{\beta ,23}\right)\hfill \\ \hfill & ={\displaystyle \frac{1}{N}}\mathbb{E}\left[{\mu }_1^2\right]+{\displaystyle \frac{1}{N^2}}\left(-2l\mathbb{E}\left[{\mu }_1^2\right]+l^2{\beta }_0^2+2l{\beta }_0^2\right).\hfill \end{array}$$

(110)

We obtain the components of the skewness as follows.

$$S_{\beta ,1}=\mathbb{E}\left[d_{\beta ,1}^3\right]=\mathbb{E}\left[{\mu }_1^3\right].$$

(111)

To obtain $S_{\beta ,2}$, we have

$$\begin{array}{cc}\hfill \mathbb{E}\left[d_{\beta ,1}^2{d}_{\beta ,22}\right]& =-V_{\beta ,1}\mathbb{E}\left[X_2^{\top }{V}_X^{-1}{X}_2{ϵ}_2{ϵ}_2\right]\hfill \\ \hfill & =-l{V}_{\beta ,1}{\beta }_0\hfill \\ \hfill \mathbb{E}\left[d_{\beta ,1}{d}_{\beta ,2}{d}_{\beta ,12}\right]& =\mathbb{E}\left[d_{\beta ,1}{d}_{\beta ,2}{d}_{\beta ,21}\right]\hfill \\ \hfill & =-\mathbb{E}\left[{\mu }_1{ϵ}_1\right]\mathbb{E}\left[{\mu }_2{ϵ}_2\right]\mathbb{E}\left[X_1^{\top }{V}_X^{-1}{X}_2\right]\hfill \\ \hfill & =-\mathbb{E}{\left[{\mu }_1{ϵ}_1\right]}^2\mathbb{E}{\left[X_1\right]}^{\top }{V}_X^{-1}\mathbb{E}\left[X_1\right]\hfill \\ \hfill & =-\mathbb{E}{\left[{ϵ}_1^3\right]}^2\mathbb{E}{\left[X_1\right]}^{\top }{V}_X^{-1}\mathbb{E}\left[X_1\right]\hfill \end{array}$$

(112)

so that

$$\begin{array}{cc}\hfill S_{\beta ,2}& =\sum _{\sigma \left(1122\right)}\mathbb{E}\left[d_{\beta ,i_1}{d}_{\beta ,i_2}{d}_{\beta ,i_3{i}_4}\right]\hfill \\ \hfill & =-l{V}_{\beta ,1}{\beta }_0-2\mathbb{E}{\left[{ϵ}_1^3\right]}^2\mathbb{E}{\left[X_1\right]}^{\top }{V}_X^{-1}\mathbb{E}\left[X_1\right].\hfill \end{array}$$

(113)

We obtain the components of the approximate kurtosis as follows.

$$\begin{array}{cc}\hfill K_{\beta ,1}& =\mathbb{E}\left[d_{\beta ,1}^4\right]-3\sum _{\sigma \left(1122\right)}\mathbb{E}\left[d_{\beta ,i_1}{d}_{\beta ,i_2}{d}_{\beta ,i_3}{d}_{\beta ,i_4}\right]\hfill \\ \hfill & =\mathbb{E}\left[{\mu }_1^4\right]-3\mathbb{E}{\left[{\mu }_1^2\right]}^2\hfill \\ \hfill & =\mathbb{E}\left[{\mu }_1^4\right]-3V_{\beta ,1}^2.\hfill \end{array}$$

(114)

$$\begin{array}{cc}\hfill K_{\beta ,2}& =\sum _{\begin{array}{c}\sigma \left(11122\right)\\ \cup \sigma \left(11222\right)\end{array}}\mathbb{E}\left[d_{\beta ,i_1{i}_2}{d}_{\beta ,i_3}{d}_{\beta ,i_4}{d}_{\beta ,i_5}\right]\hfill \\ \hfill & =-\sum _{\begin{array}{c}\sigma \left(11122\right)\\ \cup \sigma \left(11222\right)\end{array}}\mathbb{E}\left[X_{i_1}^{\top }{V}_X^{-1}{X}_{i_2}{ϵ}_{i_1}{ϵ}_{i_2}{\mu }_{i_3}{\mu }_{i_4}{\mu }_{i_5}\right].\hfill \end{array}$$

(115)

We see by symmetry that many terms are the same. We have

$$\begin{array}{cc}\hfill \mathbb{E}\left[d_{\beta ,11}{d}_{\beta ,1}{d}_{\beta ,2}{d}_{\beta ,2}\right]& =\mathbb{E}\left[d_{\beta ,11}{d}_{\beta ,2}{d}_{\beta ,1}{d}_{\beta ,2}\right]=\mathbb{E}\left[d_{\beta ,11}{d}_{\beta ,2}{d}_{\beta ,2}{d}_{\beta ,1}\right]\hfill \\ \hfill & =-\mathbb{E}\left[X_1^{\top }{V}_X^{-1}{X}_1{ϵ}_1{ϵ}_1{\mu }_1{\mu }_2{\mu }_2\right]\hfill \\ \hfill & =-l\mathbb{E}\left[{ϵ}_1^2{\mu }_1\right]\mathbb{E}\left[{\mu }_1^2\right],\hfill \\ \hfill \mathbb{E}\left[d_{\beta ,12}{d}_{\beta ,1}{d}_{\beta ,1}{d}_{\beta ,2}\right]& =\mathbb{E}\left[d_{\beta ,12}{d}_{\beta ,1}{d}_{\beta ,2}{d}_{\beta ,1}\right]=\mathbb{E}\left[d_{\beta ,12}{d}_{\beta ,2}{d}_{\beta ,1}{d}_{\beta ,1}\right]\hfill \\ \hfill & =-\mathbb{E}\left[X_1^{\top }{V}_X^{-1}{X}_2{ϵ}_1{ϵ}_2{\mu }_1{\mu }_2{\mu }_1\right]\hfill \\ \hfill & =-\mathbb{E}{\left[X_1\right]}^{\top }{V}_X^{-1}\mathbb{E}\left[X_2\right]\mathbb{E}\left[{ϵ}_1{\mu }_1^2\right]\mathbb{E}\left[{ϵ}_2{\mu }_2\right]\hfill \\ \hfill & =-\mathbb{E}{\left[X_1\right]}^{\top }{V}_X^{-1}\mathbb{E}\left[X_1\right]\mathbb{E}\left[{ϵ}_1{\mu }_1^2\right]\mathbb{E}\left[{ϵ}_1^3\right],\hfill \\ \hfill \mathbb{E}\left[d_{\beta ,12}{d}_{\beta ,1}{d}_{\beta ,2}{d}_{\beta ,2}\right]& =\mathbb{E}\left[d_{\beta ,12}{d}_{\beta ,2}{d}_{\beta ,1}{d}_{\beta ,2}\right]=\mathbb{E}\left[d_{\beta ,12}{d}_{\beta ,2}{d}_{\beta ,2}{d}_{\beta ,1}\right]\hfill \\ \hfill & =-\mathbb{E}\left[X_1^{\top }{V}_X^{-1}{X}_2{ϵ}_1{ϵ}_2{\mu }_1{\mu }_2{\mu }_2\right]\hfill \\ \hfill & =-\mathbb{E}{\left[X_1\right]}^{\top }{V}_X^{-1}\mathbb{E}\left[X_2\right]\mathbb{E}\left[{ϵ}_1{\mu }_1^2\right]\mathbb{E}\left[{ϵ}_2{\mu }_2\right]\hfill \\ \hfill & =-\mathbb{E}{\left[X_1\right]}^{\top }{V}_X^{-1}\mathbb{E}\left[X_1\right]\mathbb{E}\left[{ϵ}_1{\mu }_1^2\right]\mathbb{E}\left[{ϵ}_1^3\right],\hfill \\ \hfill \mathbb{E}\left[d_{\beta ,11}{d}_{\beta ,2}{d}_{\beta ,2}{d}_{\beta ,2}\right]& =\mathbb{E}\left[X_1^{\top }{V}_X^{-1}{X}_1{ϵ}_1{ϵ}_1{\mu }_2{\mu }_2{\mu }_2\right]\hfill \\ \hfill & =-l{\beta }_0\mathbb{E}\left[{\mu }_1^3\right],\hfill \end{array}$$

(116)

so

$$K_{\beta ,2}=-3l\mathbb{E}\left[{ϵ}_1^2{\mu }_1\right]\mathbb{E}\left[{\mu }_1^2\right]-l{\beta }_0\mathbb{E}\left[{\mu }_1^3\right]-6\mathbb{E}\left[{ϵ}_1{\mu }_1^2\right]\mathbb{E}\left[{ϵ}_1^3\right]\mathbb{E}{\left[X_1\right]}^{\top }{V}_X^{-1}\mathbb{E}\left[X_1\right].$$

(117)

$$\begin{array}{cc}\hfill K_{\beta ,3}& =\sum _{\sigma \left(112233\right)}\mathbb{E}\left[d_{i_1}{d}_{i_2}{d}_{i_3}{d}_{i_4{i}_5{i}_6}\right]\hfill \\ \hfill & =\sum _{\sigma \left(112233\right)}\mathbb{E}\left[{\mu }_{i_1}{\mu }_{i_2}{\mu }_{i_3}{X}_{i_5}^{\top }{V}_X^{-1}{W}_{i_4}{V}_X^{-1}{X}_{i_6}{ϵ}_{i_5}{ϵ}_{i_6}\right]\hfill \\ \hfill & =0.\hfill \end{array}$$

(118)

which we have obtained by noting that in any of the summands in $K_{\beta ,3}$, there is a term containing either $\mathbb{E}\left[{ϵ}_1{W}_1\right]$ or $\mathbb{E}\left[{\mu }_1{W}_1\right]$, which are both zero due to the independence of the residuals and the $X_i$s. Note that this also holds if independence is replaced with a conditional mean-zero assumption. Note that the corresponding term in the sample variance example is also zero.

$$\begin{array}{cc}\hfill K_{\beta ,4}& =\sum _{\sigma \left(112233\right)}\mathbb{E}\left[d_{i_1}{d}_{i_2}{d}_{i_3{i}_4}{d}_{i_5{i}_6}\right]\hfill \\ \hfill & =\sum _{\sigma \left(112233\right)}\mathbb{E}\left[{\mu }_{i_1}{\mu }_{i_2}{X}_{i_3}^{\top }{V}_X^{-1}{X}_{i_4}{ϵ}_{i_3}{ϵ}_{i_4}{X}_{i_5}^{\top }{V}_X^{-1}{X}_{i_6}{ϵ}_{i_5}{ϵ}_{i_6}\right]\hfill \end{array}$$

(119)

$$\begin{array}{cc}\hfill \mathbb{E}\left[d_1{d}_1{d}_{22}{d}_{33}\right]& =\mathbb{E}\left[{\mu }_1{\mu }_1{X}_2^{\top }{V}_X^{-1}{X}_2{ϵ}_2{ϵ}_2{X}_3^{\top }{V}_X^{-1}{X}_3{ϵ}_3{ϵ}_3\right]\hfill \\ \hfill & =l^2\mathbb{E}\left[{\mu }_1^2\right]{\beta }_0^2,\hfill \\ \hfill \mathbb{E}\left[d_1{d}_1{d}_{23}{d}_{23}\right]& =\mathbb{E}\left[d_1{d}_1{d}_{23}{d}_{32}\right]\hfill \\ \hfill & =\mathbb{E}\left[{\mu }_1{\mu }_1{X}_2^{\top }{V}_X^{-1}{X}_3{ϵ}_2{ϵ}_3{X}_2^{\top }{V}_X^{-1}{X}_3{ϵ}_2{ϵ}_3\right]\hfill \\ \hfill & =l\mathbb{E}\left[{\mu }_1^2\right]{\beta }_0^2,\hfill \\ \hfill \mathbb{E}\left[d_1{d}_2{d}_{12}{d}_{33}\right]& =\mathbb{E}\left[d_1{d}_2{d}_{21}{d}_{33}\right]=\mathbb{E}\left[d_1{d}_2{d}_{33}{d}_{21}\right]=\mathbb{E}\left[d_1{d}_2{d}_{33}{d}_{12}\right]\hfill \\ \hfill & =\mathbb{E}\left[{\mu }_1{\mu }_2{X}_1^{\top }{V}_X^{-1}{X}_2{ϵ}_1{ϵ}_2{X}_3^{\top }{V}_X^{-1}{X}_3{ϵ}_3{ϵ}_3\right]\hfill \\ \hfill & =l\mathbb{E}{\left[{\mu }_1{ϵ}_1\right]}^2\mathbb{E}\left[X_1^{\top }\right]V_X^{-1}\mathbb{E}\left[X_1\right]{\beta }_0.\hfill \end{array}$$

(120)

$$\begin{array}{cc}\hfill \mathbb{E}\left[d_1{d}_2{d}_{13}{d}_{23}\right]& =\mathbb{E}\left[d_1{d}_2{d}_{13}{d}_{32}\right]=\mathbb{E}\left[d_1{d}_2{d}_{31}{d}_{23}\right]=\mathbb{E}\left[d_1{d}_2{d}_{31}{d}_{32}\right]\hfill \\ & =\mathbb{E}\left[d_1{d}_2{d}_{32}{d}_{13}\right]=\mathbb{E}\left[d_1{d}_2{d}_{23}{d}_{13}\right]=\mathbb{E}\left[d_1{d}_2{d}_{32}{d}_{31}\right]\hfill \\ \hfill & =\mathbb{E}\left[d_1{d}_2{d}_{23}{d}_{31}\right]\hfill \\ \hfill & =\mathbb{E}\left[{\mu }_1{\mu }_2{X}_1^{\top }{V}_X^{-1}{X}_3{ϵ}_1{ϵ}_3{X}_2^{\top }{V}_X^{-1}{X}_3{ϵ}_2{ϵ}_3\right]\hfill \\ \hfill & =\mathbb{E}{\left[{\mu }_1{ϵ}_1\right]}^2\mathbb{E}\left[X_1^{\top }\right]V_X^{-1}\mathbb{E}\left[X_1\right]{\beta }_0,\hfill \end{array}$$

(121)

so

$$\begin{array}{cc}\hfill K_{\beta ,4}& =(l^2+2l)\mathbb{E}\left[{\mu }_1^2\right]{\beta }_0^2+(4l+8){\beta }_0\mathbb{E}{\left[{\mu }_1{ϵ}_1\right]}^2\mathbb{E}\left[X_1^{\top }\right]V_X^{-1}\mathbb{E}\left[X_1\right]\hfill \end{array}$$

(122)

• Derivations for Two-Stage Least Squares

We derive the influence functions and moments for the 2SLS estimator. Define $W_i=X_i{X}_i^{\top }-V_X$ and $Z_i={\dot{g}}_i{\dot{Y}}_i-\mathbb{E}\left[{\dot{g}}_i{\dot{Y}}_i\right]={\dot{g}}_i{\dot{Y}}_i-1={\dot{g}}_i^2\dot{+}{\dot{g}}_i{\dot{\eta }}_i-1$. Some intermediate expressions are useful. We easily see that

$$\begin{array}{cc}\hfill d_i& =-Q{q}_i=\left(\begin{array}{cc}{V}_X^{-1}& 0\\ 0& V_g^{-1}\end{array}\right)\left(\begin{array}{c}{X}_i{\eta }_i\\ g_i{ϵ}_i\end{array}\right)=\left(\begin{array}{c}{V}_X^{-1}{X}_i{\eta }_i\\ {\dot{g}}_i{\dot{ϵ}}_i\end{array}\right),\hfill \\ \hfill d_i^{\left(\theta \right)}& =-Q{q}_i^{\left(\theta \right)}=\left(\begin{array}{cc}-V_X^{-1}{X}_i{X}_i^{\top }& 0\\ {\dot{X}}_i^{\top }{\dot{ϵ}}_i& -{\pi }_0{\dot{X}}_i{\dot{Y}}_i\end{array}\right),\hfill \\ \hfill {\tilde{d}}_i^{\left(\theta \right)}& =-Q{q}_i^{\left(\theta \right)}=\left(\begin{array}{cc}-V_X^{-1}{W}_i& 0\\ {\dot{X}}_i^{\top }{\dot{ϵ}}_i& -Z_i\end{array}\right),\hfill \\ \hfill d_i^{\left(\theta \theta \right)}& =-Q{q}_i^{\left(\theta \theta \right)}\hfill \\ \hfill & =-\left(\begin{array}{cc}{V}_X^{-1}& 0\\ 0& V_g^{-1}\end{array}\right)\left(\begin{array}{c}{0}_{l\times l}\\ X_i^{\top }{Y}_i\end{array}\right)\left((S_{\pi }\otimes S_{\beta })+(S_{\beta }\otimes S_{\pi })\right)\hfill \\ \hfill & =-\left(\begin{array}{c}{0}_{l\times l}\\ {\dot{X}}_i^{\top }{\dot{Y}}_i\end{array}\right)\left((S_{\pi }\otimes S_{\beta })+(S_{\beta }\otimes S_{\pi })\right),\hfill \\ \hfill {\overline{d}}_1^{\left(\theta \theta \right)}& =-\left(\begin{array}{c}{0}_{l\times l}\\ \mathbb{E}\left[{\dot{X}}_1^{\top }{\dot{Y}}_1\right]\end{array}\right)\left((S_{\pi }\otimes S_{\beta })+(S_{\beta }\otimes S_{\pi })\right)\hfill \\ \hfill {\tilde{d}}_i^{\left(\theta \theta \right)}& =-\left(\begin{array}{c}{0}_{l\times l}\\ {\dot{X}}_i^{\top }{\dot{Y}}_i-\mathbb{E}\left[{\dot{X}}_i^{\top }{\dot{Y}}_i\right]\end{array}\right)\left((S_{\pi }\otimes S_{\beta })+(S_{\beta }\otimes S_{\pi })\right).\hfill \end{array}$$

(123)

Note that the first-order asymptotic variance for $\widehat{\theta }$ is

$$\begin{array}{cc}\hfill V_1=\mathbb{E}\left[d_1{d}_1^{\top }\right]=& \left(\begin{array}{cc}{\sigma }_{\eta }^2{V}_X^{-1}& \varrho V_X^{-1}\mathbb{E}\left[{\dot{X}}_1{\dot{g}}_1\right]\\ \varrho \mathbb{E}\left[{\dot{g}}_1{\dot{X}}_1^{\top }\right]V_X^{-1}& {\sigma }_{ϵ}^2{V}_g^{-1}\end{array}\right)\hfill \\ \hfill & =\left(\begin{array}{cc}{\sigma }_{\eta }^2{V}_X^{-1}& \varrho {\pi }_0{V}_g^{-1}\\ \varrho {\pi }_0^{\top }{V}_g^{-1}& {\sigma }_{ϵ}^2{V}_g^{-1}\end{array}\right)\hfill \end{array}$$

(124)

and the influence function for $\widehat{\beta }$ is

$$d_{\beta ,i}=S_{\beta }{d}_i={\dot{g}}_i{\dot{ϵ}}_i$$

(125)

which are standard representations. The two terms making up the second-order influence function for $\widehat{\beta }$ can be determined as follows. First,

$$\begin{array}{cc}\hfill A_{\beta ,i_1{i}_2}\left(1\right)& =S_{\beta }{\tilde{d}}_{i_1}^{\left(\theta \right)}{d}_{i_2}\hfill \\ \hfill & =S_{\beta }\left(\begin{array}{cc}-V_X^{-1}{W}_{i_1}& 0\\ {\dot{X}}_{i_1}^{\top }{\dot{ϵ}}_{i_1}& -Z_{i_1}\end{array}\right)\left(\begin{array}{c}{V}_X^{-1}{X}_{i_2}{\eta }_{i_2}\\ {\dot{g}}_{i_2}{\dot{ϵ}}_{i_2}\end{array}\right)\hfill \\ \hfill & =\left(\begin{array}{cc}{\dot{X}}_{i_1}^{\top }{\dot{ϵ}}_{i_1}& -Z_{i_1}\end{array}\right)\left(\begin{array}{c}{V}_X^{-1}{X}_i{\eta }_i\\ {\dot{g}}_{i_2}{\dot{ϵ}}_{i_2}\end{array}\right)\hfill \\ \hfill & =X_{i_1}^{\top }{\dot{ϵ}}_{i_1}{V}_X^{-1}{X}_{i_2}{\dot{\eta }}_{i_2}-Z_{i_1}{\dot{g}}_{i_2}{\dot{ϵ}}_{i_2}.\hfill \end{array}$$

(126)

Second, noting that we can write

$$S_{\beta }{\overline{d}}_1^{\left(\theta \theta \right)}=-V_g^{-1}{\pi }_0^{\top }{V}_X\{(S_{\pi }\otimes S_{\beta })+(S_{\beta }\otimes S_{\pi })\},$$

(127)

we have

$$\begin{array}{cc}\hfill 2A_{\beta ,i_1{i}_2}\left(2\right)& =S_{\beta }{\overline{d}}_1^{\left(\theta \theta \right)}(d_{i_1}\otimes d_{i_2})\hfill \\ \hfill & =-V_g^{-1}{\pi }_0^{\top }{V}_X\{(S_{\pi }{d}_{i_1}\otimes S_{\beta }{d}_{i_2})+(S_{\beta }{d}_{i_1}\otimes S_{\pi }{d}_{i_2})\}\hfill \\ \hfill & =-V_g^{-1}{\pi }_0^{\top }{V}_X\{(V_X^{-1}{X}_{i_1}{\eta }_{i_1}\otimes {\dot{g}}_{i_2}{\dot{ϵ}}_{i_2})+({\dot{g}}_{i_1}{\dot{ϵ}}_{i_1}\otimes V_X^{-1}{X}_{i_2}{\eta }_{i_2}\}\hfill \\ \hfill & =-V_g^{-1}{\pi }_0^{\top }\{\left(X_{i_1}{\dot{g}}_{i_2}{\eta }_{i_1}{\dot{ϵ}}_{i_2}\right)+X_{i_2}{\dot{g}}_{i_1}{\eta }_{i_2}{\dot{ϵ}}_{i_1}\}\hfill \\ \hfill & =-\{{\dot{g}}_{i_1}{\dot{g}}_{i_2}{\dot{\eta }}_{i_1}{\dot{ϵ}}_{i_2}+{\dot{g}}_{i_2}{\dot{g}}_{i_1}{\dot{\eta }}_{i_2}{\dot{ϵ}}_{i_1}\}.\hfill \end{array}$$

(128)

The terms making up the third-order influence function are derived as follows:

$$\begin{array}{cc}\hfill E_{\beta ,i_1{i}_2{i}_3}\left(1\right)& =S_{\beta }{\overline{d}}_1^{\left(\theta \theta \right)}(d_{i_1}\otimes {\tilde{d}}_{i_2}^{\left(\theta \right)}{d}_{i_3})\hfill \\ \hfill & =-V_g^{-1}{\pi }_0^{\top }{V}_X\left\{(S_{\pi }\otimes S_{\beta })+(S_{\beta }\otimes S_{\pi })\right\}(d_{i_1}\otimes {\tilde{d}}_{i_2}^{\left(\theta \right)}{d}_{i_3})\hfill \\ \hfill & =-V_g^{-1}{\pi }_0^{\top }{V}_X\left\{(S_{\pi }{d}_{i_1}\otimes S_{\beta }{\tilde{d}}_{i_2}^{\left(\theta \right)}{d}_{i_3})+(S_{\beta }{d}_{i_1}\otimes S_{\pi }{\tilde{d}}_{i_2}^{\left(\theta \right)}{d}_{i_3})\right\}\hfill \\ & =-V_g^{-1}{\pi }_0^{\top }{V}_X\{V_X^{-1}{X}_{i_1}{\eta }_{i_1}\otimes \left(\begin{array}{cc}{\dot{X}}_{i_2}^{\top }{\dot{ϵ}}_{i_2}& -Z_{i_2}\end{array}\right)\left(\begin{array}{c}{V}_X^{-1}{X}_{i_3}{\eta }_{i_3}\\ {\dot{g}}_{i_3}{\dot{ϵ}}_{i_3}\end{array}\right)\hfill \\ \hfill & +{\dot{g}}_{i_1}{\dot{ϵ}}_{i_1}\otimes \left(\begin{array}{cc}-V_X^{-1}{W}_{i_2}& 0\end{array}\right)\left(\begin{array}{c}{V}_X^{-1}{X}_{i_3}{\eta }_{i_3}\\ {\dot{g}}_{i_3}{\dot{ϵ}}_{i_3}\end{array}\right)\}\hfill \\ \hfill & =-V_g^{-1}{\pi }_0^{\top }\{X_{i_1}{\eta }_{i_1}\left(\begin{array}{cc}{\dot{X}}_{i_2}^{\top }{\dot{ϵ}}_{i_2}{V}_X^{-1}{X}_{i_3}{\eta }_{i_3}& -Z_{i_2}{\dot{g}}_{i_3}{\dot{ϵ}}_{i_3}\end{array}\right)\hfill \\ \hfill & -{\dot{g}}_{i_1}{\dot{ϵ}}_{i_1}\left(W_{i_2}{V}_X^{-1}{X}_{i_3}{\eta }_{i_3}\right)\}\hfill \\ \hfill & =-\{{\dot{g}}_{i_1}{\dot{\eta }}_{i_1}\left(\begin{array}{cc}{X}_{i_2}^{\top }{\dot{ϵ}}_{i_2}{V}_X^{-1}{X}_{i_3}{\dot{\eta }}_{i_3}& -Z_{i_2}{\dot{g}}_{i_3}{\dot{ϵ}}_{i_3}\end{array}\right)\hfill \\ \hfill & -{\dot{g}}_{i_1}{\dot{ϵ}}_{i_1}\left({\pi }_0^{\top }{W}_{i_2}{V}_X^{-1}{\dot{X}}_{i_3}{\dot{\eta }}_{i_3}\right)\}\hfill \\ \hfill & =-{\dot{g}}_{i_1}\left\{{\dot{\eta }}_{i_1}\left(\begin{array}{cc}{X}_{i_2}^{\top }{\dot{ϵ}}_{i_2}{V}_X^{-1}{X}_{i_3}{\dot{\eta }}_{i_3}& -Z_{i_2}{\dot{g}}_{i_3}{\dot{ϵ}}_{i_3}\end{array}\right)-{\dot{ϵ}}_{i_1}\left({\pi }_0^{\top }{W}_{i_2}{V}_X^{-1}{\dot{X}}_{i_3}{\dot{\eta }}_{i_3}\right)\right\}\hfill \\ \hfill & =-{\dot{g}}_{i_1}{\dot{\eta }}_{i_1}{X}_{i_2}^{\top }{\dot{ϵ}}_{i_2}{V}_X^{-1}{X}_{i_3}{\dot{\eta }}_{i_3}+{\dot{g}}_{i_1}{\dot{\eta }}_{i_1}{Z}_{i_2}{\dot{g}}_{i_3}{\dot{ϵ}}_{i_3}\hfill \\ & +{\dot{g}}_{i_1}{\dot{ϵ}}_{i_1}{\dot{g}}_{i_2}{X}_{i_2}{V}_X^{-1}{\dot{X}}_{i_3}{\dot{\eta }}_{i_3}-{\dot{g}}_{i_1}{\dot{ϵ}}_{i_1}{\dot{g}}_{i_3}{\dot{\eta }}_{i_3},\hfill \end{array}$$

(129)

$$\begin{array}{cc}\hfill 2E_{\beta ,i_1{i}_2{i}_3}\left(2\right)& =S_{\beta }{\overline{d}}_1^{\left(\theta \theta \right)}(d_{i_1}\otimes {\overline{d}}_1^{\left(\theta \theta \right)}(d_{i_2}\otimes d_{i_3}))\hfill \end{array}$$

(130)

Note first that

$$\begin{array}{cc}\hfill {\overline{d}}_1^{\left(\theta \theta \right)}(d_{i_2}\otimes d_{i_3})& =-\left(\begin{array}{c}{0}_{l\times l}\\ V_g^{-1}{\pi }_0^{\top }{V}_X\end{array}\right)\left((S_{\pi }\otimes S_{\beta })+(S_{\beta }\otimes S_{\pi })\right)\hfill \\ & \times \left\{d_{i_2}\otimes d_{i_3}\right\}\hfill \\ \hfill & =-\left(\begin{array}{c}{0}_{l\times l}\\ V_g^{-1}{\pi }_0^{\top }{V}_X\end{array}\right)\{(V_X^{-1}{X}_{i_2}{\eta }_{i_2}\otimes {\dot{g}}_{i_3}{\dot{ϵ}}_{i_3})\hfill \\ & +({\dot{g}}_{i_2}{\dot{ϵ}}_{i_2}\otimes V_X^{-1}{X}_{i_3}{\eta }_{i_3})\}\hfill \\ \hfill & =-\left(\begin{array}{c}{0}_{l\times 1}\\ {\dot{g}}_{i_2}{\dot{\eta }}_{i_2}{\dot{g}}_{i_3}{\dot{ϵ}}_{i_3}\end{array}\right)-\left(\begin{array}{c}{0}_{l\times l}\\ {\dot{g}}_{i_2}{\dot{ϵ}}_{i_2}{\dot{g}}_{i_3}{\dot{\eta }}_{i_3}\end{array}\right)\hfill \\ \hfill & =-\left(\begin{array}{c}{0}_{l\times 1}\\ {\dot{g}}_{i_2}{\dot{g}}_{i_3}({\dot{\eta }}_{i_2}{\dot{ϵ}}_{i_3}+{\dot{ϵ}}_{i_2}{\dot{\eta }}_{i_3})\end{array}\right)\hfill \end{array}$$

(131)

so that

$$\begin{array}{cc}\hfill 2E_{\beta ,i_1{i}_2{i}_3}\left(2\right)& =-S_{\beta }{\overline{d}}_1^{\left(\theta \theta \right)}\left(d_{i_1}\otimes \left(\begin{array}{c}{0}_{l\times 1}\\ {\dot{g}}_{i_2}{\dot{g}}_{i_3}({\dot{\eta }}_{i_2}{\dot{ϵ}}_{i_3}+{\dot{ϵ}}_{i_2}{\dot{\eta }}_{i_3})\end{array}\right)\right)\hfill \\ \hfill & =V_g^{-1}{\pi }_0^{\top }{V}_X\left\{(S_{\pi }\otimes S_{\beta })+(S_{\beta }\otimes S_{\pi })\right\}\hfill \\ & \times \left(\left(\begin{array}{c}{V}_X^{-1}{X}_{i_1}{\eta }_{i_1}\\ {\dot{g}}_{i_1}{\dot{ϵ}}_{i_1}\end{array}\right)\otimes \left(\begin{array}{c}{0}_{l\times 1}\\ {\dot{g}}_{i_2}{\dot{g}}_{i_3}({\dot{\eta }}_{i_2}{\dot{ϵ}}_{i_3}+{\dot{ϵ}}_{i_2}{\dot{\eta }}_{i_3})\end{array}\right)\right)\hfill \\ \hfill & =V_g^{-1}{\pi }_0^{\top }{V}_X\times V_X^{-1}{X}_{i_1}{\eta }_{i_1}\otimes {\dot{g}}_{i_2}{\dot{g}}_{i_3}({\dot{\eta }}_{i_2}{\dot{ϵ}}_{i_3}+{\dot{ϵ}}_{i_2}{\dot{\eta }}_{i_3})\hfill \\ \hfill & ={\dot{g}}_{i_1}{\dot{\eta }}_{i_1}{\dot{g}}_{i_2}{\dot{g}}_{i_3}({\dot{\eta }}_{i_2}{\dot{ϵ}}_{i_3}+{\dot{ϵ}}_{i_2}{\dot{\eta }}_{i_3}).\hfill \end{array}$$

(132)

Since third derivatives of $q_i\left(\theta \right)$ are zero,

$$E_{\beta ,i_1{i}_2{i}_3}\left(3\right)=0.$$

(133)

Next,

$$\begin{array}{cc}\hfill E_{\beta ,i_1{i}_2{i}_3}\left(4\right)& =S_{\beta }{\tilde{d}}_{i_1}^{\left(\theta \right)}{\tilde{d}}_{i_2}^{\left(\theta \right)}{d}_{i_3}\hfill \\ & =S_{\beta }\left(\begin{array}{cc}-V_X^{-1}{W}_{i_1}& 0\\ {\dot{X}}_{i_1}^{\top }{\dot{ϵ}}_{i_1}& -Z_{i_1}\end{array}\right)\left(\begin{array}{cc}-V_X^{-1}{W}_{i_2}& 0\\ {\dot{X}}_{i_2}^{\top }{\dot{ϵ}}_{i_2}& -Z_{i_2}\end{array}\right)\left(\begin{array}{c}{V}_X^{-1}{X}_{i_3}{\eta }_{i_3}\\ {\dot{g}}_{i_3}{\dot{ϵ}}_{i_3}\end{array}\right)\hfill \\ & =\left(\begin{array}{cc}{\dot{X}}_{i_1}^{\top }{\dot{ϵ}}_{i_1}& -Z_{i_1}\end{array}\right)\left(\begin{array}{c}-V_X^{-1}{W}_{i_2}{V}_X^{-1}{X}_{i_3}{\eta }_{i_3}\\ X_{i_2}^{\top }{\dot{ϵ}}_{i_2}{V}_X^{-1}{X}_{i_3}{\dot{\eta }}_{i_3}-Z_{i_2}{\dot{g}}_{i_3}{\dot{ϵ}}_{i_3}\end{array}\right)\hfill \\ & =-X_{i_1}^{\top }{\dot{ϵ}}_{i_1}{V}_X^{-1}{W}_{i_2}{V}_X^{-1}{X}_{i_3}{\dot{\eta }}_{i_3}\hfill \\ & -Z_{i_1}\left(X_{i_2}^{\top }{\dot{ϵ}}_{i_2}{V}_X^{-1}{X}_{i_3}{\dot{\eta }}_{i_3}-Z_{i_2}{\dot{g}}_{i_3}{\dot{ϵ}}_{i_3}\right),\hfill \end{array}$$

(134)

$$\begin{array}{cc}\hfill 2E_{\beta ,i_1{i}_2{i}_3}\left(5\right)& =S_{\beta }{\tilde{d}}_{i_1}^{\left(\theta \right)}{\overline{d}}_1^{\left(\theta \theta \right)}(d_{i_2}\otimes d_{i_3})\hfill \\ \hfill & =-\left(\begin{array}{cc}{\dot{X}}_{i_1}^{\top }{\dot{ϵ}}_{i_1}& -Z_{i_1}\end{array}\right)\left(\begin{array}{c}{0}_{l\times 1}\\ {\dot{g}}_{i_2}{\dot{g}}_{i_3}({\dot{\eta }}_{i_2}{\dot{ϵ}}_{i_3}+{\dot{ϵ}}_{i_2}{\dot{\eta }}_{i_3})\end{array}\right)\hfill \\ \hfill & =Z_{i_1}{\dot{g}}_{i_2}{\dot{g}}_{i_3}({\dot{\eta }}_{i_2}{\dot{ϵ}}_{i_3}+{\dot{ϵ}}_{i_2}{\dot{\eta }}_{i_3}),\hfill \end{array}$$

(135)

$$\begin{array}{cc}\hfill -2E_{\beta ,i_1{i}_2{i}_3}\left(6\right)& =-S_{\beta }{\tilde{d}}_{i_1}^{\left(\theta \theta \right)}(d_{i_2}\otimes d_{i_3})\hfill \\ \hfill & =({\dot{X}}_{i_1}^{\top }{\dot{Y}}_{i_1}-\mathbb{E}\left[{\dot{X}}_i^{\top }{\dot{Y}}_i\right])\{\left(V_X^{-1}{X}_{i_2}{\eta }_{i_2}{\dot{g}}_{i_3}{\dot{ϵ}}_{i_3}\right)+\left({\dot{g}}_{i_2}{\dot{ϵ}}_{i_2}{V}_X^{-1}{X}_{i_3}{\eta }_{i_3}\right\}\hfill \\ \hfill & =V_g^{-1}({\pi }_0{X}_{i_1}{X}_{i_1}^{\top }+X_{i_1}^{\top }{\eta }_{i_1}-{\pi }_0\mathbb{E}\left[X_i{X}_i^{\top }\right])V_X^{-1}\{X_{i_2}{\eta }_{i_2}{\dot{g}}_{i_3}{\dot{ϵ}}_{i_3}+{\dot{g}}_{i_2}{\dot{ϵ}}_{i_2}{X}_{i_3}{\eta }_{i_3}\}\hfill \\ \hfill & =V_g^{-1}\left({\pi }_0{X}_{i_1}{X}_{i_1}^{\top }\right)V_X^{-1}\{X_{i_2}{\eta }_{i_2}{\dot{g}}_{i_3}{\dot{ϵ}}_{i_3}+{\dot{g}}_{i_2}{\dot{ϵ}}_{i_2}{X}_{i_3}{\eta }_{i_3}\}\hfill \\ \hfill & +V_g^{-1}\left(X_{i_1}^{\top }{\eta }_{i_1}\right)V_X^{-1}\{X_{i_2}{\eta }_{i_2}{\dot{g}}_{i_3}{\dot{ϵ}}_{i_3}+{\dot{g}}_{i_2}{\dot{ϵ}}_{i_2}{X}_{i_3}{\eta }_{i_3}\}\hfill \\ \hfill & -V_g^{-1}\left({\pi }_0\mathbb{E}\left[X_i{X}_i^{\top }\right]\right)V_X^{-1}\{X_{i_2}{\eta }_{i_2}{\dot{g}}_{i_3}{\dot{ϵ}}_{i_3}+{\dot{g}}_{i_2}{\dot{ϵ}}_{i_2}{X}_{i_3}{\eta }_{i_3}\}\hfill \\ \hfill & ={\dot{g}}_{i_1}{X}_{i_1}^{\top }{V}_X^{-1}{X}_{i_2}{\dot{\eta }}_{i_2}{\dot{g}}_{i_3}{\dot{ϵ}}_{i_3}+{\dot{g}}_{i_1}{X}_{i_1}^{\top }{V}_X^{-1}{\dot{g}}_{i_2}{\dot{ϵ}}_{i_2}{X}_{i_3}{\dot{\eta }}_{i_3}\hfill \\ \hfill & +X_{i_1}^{\top }{\eta }_{i_1}{V}_X^{-1}{X}_{i_2}{\dot{\eta }}_{i_2}{\dot{g}}_{i_3}{\dot{ϵ}}_{i_3}+X_{i_1}^{\top }{\eta }_{i_1}{V}_X^{-1}{\dot{g}}_{i_2}{\dot{ϵ}}_{i_2}{X}_{i_3}{\dot{\eta }}_{i_3}\hfill \\ \hfill & -({\dot{g}}_{i_2}{\eta }_{i_2}{\dot{g}}_{i_3}{\dot{ϵ}}_{i_3}+{\dot{g}}_{i_2}{\dot{ϵ}}_{i_2}{\dot{g}}_{i_3}{\dot{\eta }}_{i_3})\hfill \end{array}$$

(136)

The approximate bias (times N) is given by the expected value of $A_{\beta ,11}\left(1\right)+A_{\beta ,11}\left(2\right)$. We have

$$\begin{array}{cc}\hfill B_{\beta ,21}={\overline{A}}_{\beta ,11}\left(1\right)& =\mathbb{E}[X_1^{\top }{\dot{ϵ}}_1{V}_X^{-1}{X}_1{\dot{\eta }}_1+(-{\dot{g}}_1{\dot{Y}}_1+1){\dot{g}}_1{\dot{ϵ}}_1]\hfill \\ \hfill & =\mathbb{E}\left[X_1^{\top }{V}_X^{-1}{X}_1\right]\mathbb{E}\left[{\dot{\eta }}_1{\dot{ϵ}}_1\right]-\mathbb{E}\left[{\dot{g}}_1{\dot{g}}_1\right]\mathbb{E}\left[{\dot{\eta }}_1{\dot{ϵ}}_1\right]\hfill \\ \hfill & =l\dot{\varrho }-\dot{\varrho },\hfill \end{array}$$

(137)

$${\overline{A}}_{\beta ,11}\left(1\right)=(l-1)\varrho V_g^{-1},$$

(138)

$$\begin{array}{cc}\hfill 2B_{\beta ,22}=2{\overline{A}}_{\beta ,11}\left(2\right)& =-\mathbb{E}\{{\dot{g}}_1{\dot{g}}_1{\dot{\eta }}_1{\dot{ϵ}}_1+{\dot{g}}_1{\dot{g}}_1{\dot{\eta }}_1{\dot{ϵ}}_1\}\hfill \\ \hfill & =-2\mathbb{E}\left[{\dot{g}}_1^2\right]\mathbb{E}\left[{\dot{\eta }}_1{\dot{ϵ}}_1\right]\hfill \\ \hfill & =-2\dot{\varrho }\hfill \end{array}$$

(139)

so

$${\overline{A}}_{11}\left(1\right)+{\overline{A}}_{11}\left(2\right)=(l-2)\varrho V_g^{-1}.$$

(140)

To find the third approximate moment of $\widehat{\beta }$, we need to evaluate the two terms from the standard formula, which we denote here as $S_{\beta ,1}$ and $S_{\beta ,2}$. First,

$$\begin{array}{cc}\hfill S_{\beta ,1}& =\mathbb{E}\left[d_{\beta ,1}^3\right]\hfill \\ & =\mathbb{E}\left[{\dot{g}}_1^3\right]\mathbb{E}\left[{\dot{ϵ}}_1^3\right].\hfill \end{array}$$

(141)

Second, by symmetry,

$$\begin{array}{cc}\hfill S_{\beta ,2}& =\sum _{\sigma \left(1122\right)}\mathbb{E}\left[d_{\beta ,i_1}{d}_{\beta ,i_2}{d}_{\beta ,i_3{i}_4}\right]\hfill \\ \hfill & =\mathbb{E}\left[d_{\beta ,1}{d}_{\beta ,1}{d}_{\beta ,22}\right]+\mathbb{E}\left[d_{\beta ,1}{d}_{\beta ,2}{d}_{\beta ,12}\right]+\mathbb{E}\left[d_{\beta ,1}{d}_{\beta ,2}{d}_{\beta ,21}\right]\hfill \\ \hfill & =\mathbb{E}\left[d_{\beta ,1}^2\right]\mathbb{E}\left[d_{\beta ,22}\right]+2\mathbb{E}\left[d_{\beta ,1}{d}_{\beta ,2}{d}_{\beta ,12}\right].\hfill \end{array}$$

(142)

We see immediately that

$$\mathbb{E}\left[d_{\beta ,1}^2\right]\mathbb{E}\left[d_{\beta ,22}\right]=V_{\beta ,1}{B}_{\beta ,2}$$

(143)

and almost as immediately that

$$\begin{array}{cc}\hfill \mathbb{E}\left[d_{\beta ,1}{d}_{\beta ,2}{d}_{\beta ,12}\right]& =\mathbb{E}\left[{\dot{g}}_1{\dot{ϵ}}_1{\dot{g}}_2{\dot{ϵ}}_2(X_1^{\top }{\dot{ϵ}}_1{V}_X^{-1}{X}_2{\dot{\eta }}_2-Z_1{\dot{g}}_2{\dot{ϵ}}_2-({\dot{g}}_1{\dot{g}}_2{\dot{\eta }}_1{\dot{ϵ}}_2+{\dot{g}}_2{\dot{g}}_1{\dot{\eta }}_2{\dot{ϵ}}_1)/2)\right]\hfill \\ \hfill & =\mathbb{E}\left[{\dot{ϵ}}_1^2\right]\mathbb{E}\left[{\dot{ϵ}}_2{\dot{\eta }}_2\right]-\mathbb{E}\left[{\dot{ϵ}}_1^2\right]\mathbb{E}\left[{\dot{ϵ}}_2{\dot{\eta }}_2\right]-(\mathbb{E}\left[{\dot{ϵ}}_1^2\right]\mathbb{E}\left[{\dot{ϵ}}_2{\dot{\eta }}_2\right]+\mathbb{E}\left[{\dot{ϵ}}_1^2\right]\mathbb{E}\left[{\dot{ϵ}}_2{\dot{\eta }}_2\right])/2\hfill \\ \hfill & =-{\dot{\sigma }}_{ϵ}^2\dot{\varrho }\hfill \end{array}$$

noting that $g_1{X}_1^{\top }={\pi }_0^{\top }{X}_1{X}_1^{\top }$ so $\mathbb{E}\left[g_1{X}_1^{\top }\right]={\pi }_0^{\top }{V}_X$. We thus have

$$S_{\beta ,2}=V_{\beta ,1}{B}_{\beta ,2}-{\dot{\sigma }}_{ϵ}^2\dot{\varrho }.$$

(144)

The components of the higher-order terms in the approximate MSE of $\widehat{\beta }$ can be written as

$$V_{\beta ,21}=\mathbb{E}\left[d_{\beta ,1}{d}_{\beta ,11}^{\top }\right],V_{\beta ,22}=\sum _{\sigma \left(1122\right)}\mathbb{E}\left[d_{\beta ,i_1}{d}_{\beta ,i_2{i}_3{i}_4}^{\top }\right],V_{\beta ,23}=\sum _{\sigma \left(1122\right)}\mathbb{E}\left[d_{\beta ,i_1{i}_2}{d}_{i_3{i}_4}\right].$$

(145)

Evaluating these in turn we have

$$\begin{array}{cc}\hfill V_{\beta ,21}& =\mathbb{E}\left[d_{\beta ,1}{d}_{\beta ,11}^{\top }\right]\hfill \\ \hfill & =\mathbb{E}\left[{\dot{g}}_1{\dot{ϵ}}_1(X_1^{\top }{\dot{ϵ}}_1{V}_X^{-1}{X}_1{\dot{\eta }}_1-Z_1{\dot{g}}_1{\dot{ϵ}}_1-({\dot{g}}_1{\dot{g}}_1{\dot{\eta }}_1{\dot{ϵ}}_1+{\dot{g}}_1{\dot{g}}_1{\dot{\eta }}_1{\dot{ϵ}}_1)/2)\right]\hfill \\ \hfill & =\mathbb{E}\left[{\dot{g}}_1{X}_1^{\top }{V}_X^{-1}{X}_1\right]\mathbb{E}\left[{\dot{ϵ}}_1^2{\dot{\eta }}_1\right]-\mathbb{E}\left[{\dot{g}}_1{\dot{ϵ}}_1(V_g^{-1}{g}_1(g_1+{\eta }_1)-1){\dot{g}}_1{\dot{ϵ}}_1\right]-\mathbb{E}\left[{\dot{g}}_1^3{\dot{ϵ}}_1^2{\dot{\eta }}_1\right]\hfill \\ \hfill & =\mathbb{E}\left[{\dot{g}}_1{X}_1^{\top }{V}_X^{-1}{X}_1\right]\mathbb{E}\left[{\dot{ϵ}}_1^2{\dot{\eta }}_1\right]-\mathbb{E}\left[{\dot{g}}_1^4{\dot{ϵ}}_1^2\right]-\mathbb{E}\left[{\dot{g}}_1^3{\dot{ϵ}}_1^2{\dot{\eta }}_1\right]+\mathbb{E}\left[{\dot{g}}_1^2{\dot{ϵ}}_1^2\right]-\mathbb{E}\left[{\dot{g}}_1^3\right]\mathbb{E}\left[{\dot{ϵ}}_1^2{\dot{\eta }}_1\right]\hfill \\ \hfill & =\mathbb{E}\left[{\dot{g}}_1{X}_1^{\top }{V}_X^{-1}{X}_1\right]\mathbb{E}\left[{\dot{ϵ}}_1^2{\dot{\eta }}_1\right]-\mathbb{E}\left[{\dot{g}}_1^4\right]\mathbb{E}\left[{\dot{ϵ}}_1^2\right]-2\mathbb{E}\left[{\dot{g}}_1^3\right]\mathbb{E}\left[{\dot{ϵ}}_1^2{\dot{\eta }}_1\right]+\mathbb{E}\left[{\dot{ϵ}}_1^2\right]\hfill \\ \hfill & =\mathbb{E}\left[{\dot{ϵ}}_1^2{\dot{\eta }}_1\right](\mathbb{E}\left[{\dot{g}}_1{X}_1^{\top }{V}_X^{-1}{X}_1\right]-2\mathbb{E}\left[{\dot{g}}_1^3\right])-V_{\beta ,1}\mathrm{Var}\left[{\dot{g}}_1^2\right].\hfill \end{array}$$

(146)

To evaluate $V_{\beta ,23}$ we break it up into its components:

$$\begin{array}{cc}\hfill V_{\beta ,23}& =\sum _{\sigma \left(1122\right)}\mathbb{E}\left[d_{\beta ,i_1{i}_2}{d}_{\beta ,i_3{i}_4}\right]\hfill \\ \hfill & =\mathbb{E}\left[d_{\beta ,11}\right]\mathbb{E}\left[d_{\beta ,22}\right]+\mathbb{E}\left[d_{\beta ,12}^2\right]+\mathbb{E}\left[d_{\beta ,12}{d}_{\beta ,21}\right].\hfill \end{array}$$

(147)

First, we see that $\mathbb{E}\left[d_{\beta ,11}\right]\mathbb{E}\left[d_{\beta ,22}\right]=B_{\beta ,2}^2$. With

$$\mathbb{E}\left[d_{\beta ,i_1{i}_2}{d}_{\beta ,i_3{i}_4}\right]=\mathbb{E}\left[(A_{\beta ,i_1{i}_2}\left(1\right)+A_{\beta ,i_1{i}_2}\left(2\right))(A_{\beta ,i_3{i}_4}\left(1\right)+A_{\beta ,i_3{i}_4}\left(2\right))\right],$$

(148)

we evaluate

$$\mathbb{E}\left[d_{\beta ,12}^2\right]=\mathbb{E}\left[A_{\beta ,12}{\left(1\right)}^2\right]+2\mathbb{E}\left[A_{\beta ,12}\left(1\right)A_{\beta ,12}\left(2\right)\right]+\mathbb{E}\left[A_{\beta ,12}{\left(2\right)}^2\right]$$

(149)

term by term. The first term in $\mathbb{E}\left[d_{\beta ,12}^2\right]$ is

$$\begin{array}{cc}\hfill \mathbb{E}\left[A_{\beta ,12}{\left(1\right)}^2\right]& =\mathbb{E}\left[{(X_1^{\top }{\dot{ϵ}}_1{V}_X^{-1}{X}_2{\dot{\eta }}_2-Z_1{\dot{g}}_2{\dot{ϵ}}_2)}^2\right]\hfill \\ \hfill & =\mathbb{E}\left[{\left(X_1^{\top }{\dot{ϵ}}_1{V}_X^{-1}{X}_2{\dot{\eta }}_2\right)}^2\right]-2\mathbb{E}\left[X_1^{\top }{\dot{ϵ}}_1{V}_X^{-1}{X}_2{\dot{\eta }}_2{Z}_1{\dot{g}}_2{\dot{ϵ}}_2\right]+\mathbb{E}\left[{\left(Z_1{\dot{g}}_2{\dot{ϵ}}_2\right)}^2\right]\hfill \\ \hfill & =l{\dot{\sigma }}_{ϵ}^2{\dot{\sigma }}_{\eta }^2-2{\dot{\varrho }}^2+(\mathrm{Var}\left[{\dot{g}}_1^2\right]+{\dot{\sigma }}_{\eta }^2){\dot{\sigma }}_{ϵ}^2\hfill \\ \hfill & =(l+1){\dot{\sigma }}_{ϵ}^2{\dot{\sigma }}_{\eta }^2-2{\dot{\varrho }}^2+{\dot{\sigma }}_{ϵ}^2\mathrm{Var}\left[{\dot{g}}_1^2\right],\hfill \end{array}$$

(150)

where we have used

$$\begin{array}{cc}\hfill \mathbb{E}{\left(X_1^{\top }{V}_X^{-1}{X}_2\right)}^2]& =\mathbb{E}\left[X_1^{\top }{V}_X^{-1}{X}_2{X}_2^{\top }{V}_X^{-1}{X}_1\right]=l,\hfill \\ \hfill \mathbb{E}\left[X_1^{\top }{\dot{V}}_X^{-1}{X}_2{\dot{g}}_1{\dot{g}}_2\right]& =\mathbb{E}{\dot{X}}_1^{\top }{\dot{V}}_X^{-1}{X}_2{X}_2^{\top }{\pi }_0{\dot{g}}_1]=\mathbb{E}\left[{\dot{g}}_1^{\top :}{g}_1\right]=1,\hfill \\ \hfill \mathbb{E}\left[Z_1^2\right]& =\mathbb{E}\left[{({\dot{g}}_1({\dot{g}}_1+{\dot{\eta }}_1)-1)}^2\right]=\mathbb{E}[{\dot{g}}_1^2{({\dot{g}}_1+{\dot{\eta }}_1)}^2-2{\dot{g}}_1({\dot{g}}_1+{\dot{\eta }}_1)]+1\hfill \\ \hfill & =\mathbb{E}\left[{\dot{g}}_1^4\right]+\mathbb{E}\left[{\dot{g}}_1^2\right]\mathbb{E}\left[{\dot{\eta }}_1^2\right]-1=\mathrm{Var}\left[{\dot{g}}_1^2\right]+\mathbb{E}\left[{\dot{\eta }}_1^2\right].\hfill \end{array}$$

(151)

Next, the middle term in $\mathbb{E}\left[d_{\beta ,12}^2\right]$ is

$$\begin{array}{cc}\hfill 2\mathbb{E}\left[A_{\beta ,12}\left(1\right)A_{\beta ,12}\left(2\right)\right]& =-\mathbb{E}\left[(X_1^{\top }{\dot{ϵ}}_1{V}_X^{-1}{X}_2{\dot{\eta }}_2-Z_1{\dot{g}}_2{\dot{ϵ}}_2)({\dot{g}}_1{\dot{g}}_2{\dot{\eta }}_1{\dot{ϵ}}_2+{\dot{g}}_2{\dot{g}}_1{\dot{\eta }}_2{\dot{ϵ}}_1)\right]\hfill \\ & =-\mathbb{E}\left[(X_1^{\top }{\dot{ϵ}}_1{V}_X^{-1}{X}_2{\dot{\eta }}_2-{\dot{g}}_1{\dot{\eta }}_1{\dot{g}}_2{\dot{ϵ}}_2)({\dot{g}}_1{\dot{g}}_2{\dot{\eta }}_1{\dot{ϵ}}_2+{\dot{g}}_2{\dot{g}}_1{\dot{\eta }}_2{\dot{ϵ}}_1)\right]\hfill \\ \hfill & =-\mathbb{E}\left[{\dot{g}}_1{\dot{g}}_2\left(X_1^{\top }{\dot{ϵ}}_1{V}_X^{-1}{X}_2{\dot{\eta }}_2\right)({\dot{\eta }}_1{\dot{ϵ}}_2+{\dot{\eta }}_2{\dot{ϵ}}_1)\right]\hfill \\ \hfill & +\mathbb{E}\left[{\dot{g}}_1{\dot{g}}_2{\dot{g}}_1{\dot{\eta }}_1{\dot{g}}_2{\dot{ϵ}}_2({\dot{\eta }}_1{\dot{ϵ}}_2+{\dot{\eta }}_2{\dot{ϵ}}_1)\right]\hfill \\ \hfill & =-{\dot{\varrho }}^2-\mathbb{E}\left[{\dot{ϵ}}_1^2\right]\mathbb{E}\left[{\dot{\eta }}_1^2\right])+\mathbb{E}\left[{\dot{\eta }}_2^2\right]\mathbb{E}\left[{\dot{ϵ}}_2^2\right]+{\dot{\varrho }}^2\hfill \\ \hfill & =0.\hfill \end{array}$$

(152)

The third component of $\mathbb{E}\left[d_{\beta ,12}^2\right]$, $\mathbb{E}\left[A_{\beta ,12}{\left(2\right)}^2\right]$, can be written

$$\begin{array}{cc}\hfill 4\mathbb{E}\left[A_{\beta ,12}{\left(2\right)}^2\right]& =\mathbb{E}\left[\{{\dot{g}}_1{\dot{g}}_2{\dot{\eta }}_1{\dot{ϵ}}_2+{\dot{g}}_1{\dot{g}}_2{\dot{\eta }}_2{\dot{ϵ}}_1\}\{{\dot{g}}_1{\dot{g}}_2{\dot{\eta }}_1{\dot{ϵ}}_2+{\dot{g}}_1{\dot{g}}_2{\dot{\eta }}_2{\dot{ϵ}}_1\}\right]\hfill \\ \hfill & =\mathbb{E}\left[{\left({\dot{g}}_1{\dot{g}}_2{\dot{\eta }}_1{\dot{ϵ}}_2\right)}^2\right]+2\mathbb{E}\left[{\dot{g}}_1{\dot{g}}_2{\dot{\eta }}_1{\dot{ϵ}}_2{\dot{g}}_1{\dot{g}}_2{\dot{\eta }}_2{\dot{ϵ}}_1\right]+\mathbb{E}\left[{\left({\dot{g}}_1{\dot{g}}_2{\dot{\eta }}_2{\dot{ϵ}}_1\right)}^2\right]\hfill \\ & ={\dot{\sigma }}_{\eta }^2{\dot{\sigma }}_{ϵ}^2+2{\dot{\varrho }}^2+{\dot{\sigma }}_{\eta }^2{\dot{\sigma }}_{ϵ}^2\hfill \\ \hfill & =2({\dot{\sigma }}_{\eta }^2{\dot{\sigma }}_{ϵ}^2+{\dot{\varrho }}^2).\hfill \end{array}$$

(153)

Putting these three results together we have

$$\begin{array}{cc}\hfill \mathbb{E}\left[d_{\beta ,12}^2\right]& =(l+1){\dot{\sigma }}_{ϵ}^2{\dot{\sigma }}_{\eta }^2-2{\dot{\varrho }}^2+{\dot{\sigma }}_{ϵ}^2\mathrm{Var}\left[{\dot{g}}_1^2\right]+{\displaystyle \frac{1}{2}}({\dot{\sigma }}_{\eta }^2{\dot{\sigma }}_{ϵ}^2+{\dot{\varrho }}^2)\hfill \end{array}$$

(154)

Similarly, we evaluate the components of

$$\begin{array}{cc}\hfill \mathbb{E}\left[d_{\beta ,12}{d}_{\beta ,21}\right]& =\mathbb{E}\left[A_{\beta ,12}\left(1\right)A_{\beta ,21}\left(1\right)\right]+\mathbb{E}\left[A_{\beta ,12}\left(1\right)A_{\beta ,21}\left(2\right)\right]\hfill \\ \hfill & +\mathbb{E}\left[A_{\beta ,12}\left(2\right)A_{\beta ,21}\left(1\right)\right]+\mathbb{E}\left[A_{\beta ,12}\left(2\right)A_{\beta ,21}\left(2\right)\right]\hfill \end{array}$$

(155)

as follows. The first term in $\mathbb{E}\left[d_{\beta ,12}{d}_{\beta ,21}\right]$ is

$$\begin{array}{cc}\hfill \mathbb{E}\left[A_{\beta ,12}\left(1\right)A_{\beta ,21}\left(1\right)\right]& =\mathbb{E}\left[(X_1^{\top }{\dot{ϵ}}_1{V}_X^{-1}{X}_2{\dot{\eta }}_2-Z_1{\dot{g}}_2{\dot{ϵ}}_2)(X_2^{\top }{\dot{ϵ}}_2{V}_X^{-1}{X}_1{\dot{\eta }}_1-Z_2{\dot{g}}_1{\dot{ϵ}}_1)\right]\hfill \\ \hfill & =\mathbb{E}\left[X_1^{\top }{\dot{ϵ}}_1{V}_X^{-1}{X}_2{\dot{\eta }}_2{X}_2^{\top }{\dot{ϵ}}_2{V}_X^{-1}{X}_1{\dot{\eta }}_1\right]\hfill \\ \hfill & -2\mathbb{E}\left[X_1^{\top }{\dot{ϵ}}_1{V}_X^{-1}{X}_2{\dot{\eta }}_2{Z}_2{\dot{g}}_1{\dot{ϵ}}_1\right]+\mathbb{E}\left[Z_1{\dot{g}}_2{\dot{ϵ}}_2{Z}_2{\dot{g}}_1{\dot{ϵ}}_1\right]\hfill \\ \hfill & =l{\dot{\varrho }}^2-2\mathbb{E}\left[{\dot{g}}_2^2{\dot{\sigma }}_{\eta }^2{\dot{\sigma }}_{ϵ}^2\right]+\mathbb{E}\left[{\dot{g}}_1{\dot{\eta }}_1{\dot{g}}_2{\dot{ϵ}}_2{\dot{g}}_2{\dot{\eta }}_2{\dot{g}}_1{\dot{ϵ}}_1\right]\hfill \\ \hfill & =(l+1){\dot{\varrho }}^2-2{\dot{\sigma }}_{\eta }^2{\dot{\sigma }}_{ϵ}^2\hfill \end{array}$$

(156)

By symmetry, the second and third terms in $\mathbb{E}\left[d_{\beta ,12}{d}_{\beta ,21}\right]$ are

$$\begin{array}{cc}\hfill 2\mathbb{E}\left[A_{\beta ,12}\left(1\right)A_{\beta ,21}\left(2\right)\right]& =-\mathbb{E}\left[(X_1^{\top }{\dot{ϵ}}_1{V}_X^{-1}{X}_2{\dot{\eta }}_2-Z_1{\dot{g}}_2{\dot{ϵ}}_2)({\dot{g}}_1{\dot{g}}_2{\dot{\eta }}_1{\dot{ϵ}}_2+{\dot{g}}_2{\dot{g}}_1{\dot{\eta }}_2{\dot{ϵ}}_1)\right]\hfill \\ & =-\mathbb{E}\left[(X_1^{\top }{\dot{ϵ}}_1{V}_X^{-1}{X}_2{\dot{\eta }}_2-{\dot{g}}_1{\dot{\eta }}_1{\dot{g}}_2{\dot{ϵ}}_2)({\dot{g}}_1{\dot{g}}_2{\dot{\eta }}_1{\dot{ϵ}}_2+{\dot{g}}_2{\dot{g}}_1{\dot{\eta }}_2{\dot{ϵ}}_1)\right]\hfill \\ \hfill & =\mathbb{E}\left[A_{\beta ,12}\left(2\right)A_{\beta ,12}\left(1\right)\right]\hfill \\ \hfill & =0.\hfill \end{array}$$

(157)

Also by symmetry, the fourth term in $\mathbb{E}\left[d_{\beta ,12}{d}_{\beta ,21}\right]$ is

$$\begin{array}{cc}\hfill 4\mathbb{E}\left[{\left(A_{\beta ,12}\left(2\right)A_{\beta ,21}\left(2\right)\right)}^2\right]& =4\mathbb{E}\left[{\left(A_{\beta ,12}\left(2\right)\right)}^2\right]\hfill \\ \hfill & =2({\dot{\sigma }}_{\eta }^2{\dot{\sigma }}_{ϵ}^2+{\dot{\varrho }}^2),\hfill \end{array}$$

(158)

so that

$$\begin{array}{cc}\hfill \mathbb{E}\left[d_{\beta ,12}{d}_{\beta ,21}\right]& =(l+1){\dot{\varrho }}^2-2{\dot{\sigma }}_{\eta }^2{\dot{\sigma }}_{ϵ}^2+0+0+{\displaystyle \frac{1}{2}}({\dot{\sigma }}_{\eta }^2{\dot{\sigma }}_{ϵ}^2+{\dot{\varrho }}^2)\hfill \end{array}$$

(159)

and

$$\begin{array}{cc}\hfill V_{\beta ,23}& =B_{\beta ,2}^2+(l+1){\dot{\sigma }}_{ϵ}^2{\dot{\sigma }}_{\eta }^2-2{\dot{\varrho }}^2+{\dot{\sigma }}_{ϵ}^2\mathrm{Var}\left[{\dot{g}}_1^2\right]+{\displaystyle \frac{1}{2}}({\dot{\sigma }}_{\eta }^2{\dot{\sigma }}_{ϵ}^2+{\dot{\varrho }}^2)\hfill \\ \hfill & +(l+1){\dot{\varrho }}^2-2{\dot{\sigma }}_{\eta }^2{\dot{\sigma }}_{ϵ}^2+{\displaystyle \frac{1}{2}}({\dot{\sigma }}_{\eta }^2{\dot{\sigma }}_{ϵ}^2+{\dot{\varrho }}^2)]\hfill \\ \hfill & =B_{\beta ,2}^2+l({\dot{\sigma }}_{ϵ}^2{\dot{\sigma }}_{\eta }^2+{\dot{\varrho }}^2)+{\dot{\sigma }}_{ϵ}^2\mathrm{Var}\left[{\dot{g}}_1^2\right].\hfill \end{array}$$

(160)

To evaluate $V_{\beta ,22}$ we break it up into its components:

$$\begin{array}{cc}\hfill V_{\beta ,22}& =\sum _{\sigma \left(1122\right)}\mathbb{E}\left[d_{\beta ,i_1}{d}_{\beta ,i_2{i}_3{i}_4}\right]\hfill \\ \hfill & =\sum _{\sigma \left(1122\right)}\sum _{l=1}^6\mathbb{E}\left[d_{\beta ,i_1}{E}_{\beta ,i_2{i}_3{i}_4}\right]\hfill \end{array}$$

(161)

For $l=1$,

$$\begin{array}{cc}\hfill \mathbb{E}\left[d_{\beta ,1}{E}_{\beta ,122}\left(1\right)\right]& =\mathbb{E}[{\dot{g}}_1{\dot{ϵ}}_1(-{\dot{g}}_1{\dot{\eta }}_1{X}_2^{\top }{\dot{ϵ}}_2{V}_X^{-1}{X}_2{\dot{\eta }}_2+{\dot{g}}_1{\dot{\eta }}_1{Z}_2{\dot{g}}_2{\dot{ϵ}}_2\hfill \\ & +{\dot{g}}_1{\dot{ϵ}}_1{\dot{g}}_2{X}_2{V}_X^{-1}{\dot{X}}_2{\dot{\eta }}_2-{\dot{g}}_1{\dot{ϵ}}_1{\dot{g}}_2{\dot{\eta }}_2\left)\right]\hfill \\ \hfill & =-l{\dot{\varrho }}^2+{\dot{\varrho }}^2+0-0\hfill \\ & =-(l-1){\dot{\varrho }}^2\hfill \\ \hfill \mathbb{E}\left[d_{\beta ,1}{E}_{\beta ,212}\left(1\right)\right]& =\mathbb{E}[{\dot{g}}_1{\dot{ϵ}}_1(-{\dot{g}}_2{\dot{\eta }}_2{X}_1^{\top }{\dot{ϵ}}_1{V}_X^{-1}{X}_2{\dot{\eta }}_2+{\dot{g}}_2{\dot{\eta }}_2{Z}_1{\dot{g}}_2{\dot{ϵ}}_2\hfill \\ & +{\dot{g}}_2{\dot{ϵ}}_2{\dot{g}}_1{X}_1{V}_X^{-1}{\dot{X}}_2{\dot{\eta }}_2-{\dot{g}}_2{\dot{ϵ}}_2{\dot{g}}_2{\dot{\eta }}_2\left)\right]\hfill \\ \hfill & =-\mathbb{E}\left[{\dot{g}}_1{\dot{ϵ}}_1{\dot{g}}_2{\dot{\eta }}_2{X}_1^{\top }{\dot{ϵ}}_1{V}_X^{-1}{X}_2{\dot{\eta }}_2\right]+\mathbb{E}\left[{\dot{g}}_1{\dot{ϵ}}_1{\dot{g}}_2{\dot{\eta }}_2{Z}_1{\dot{g}}_2{\dot{ϵ}}_2\right]\hfill \\ \hfill & +\mathbb{E}\left[{\dot{g}}_1{\dot{ϵ}}_1{\dot{g}}_2{\dot{ϵ}}_2{\dot{g}}_1{X}_1{V}_X^{-1}{\dot{X}}_2{\dot{\eta }}_2\right]-\mathbb{E}\left[{\dot{g}}_1{\dot{ϵ}}_1{\dot{g}}_2{\dot{ϵ}}_2{\dot{g}}_2{\dot{\eta }}_2\right]\hfill \\ \hfill & =-{\dot{\sigma }}_{ϵ}^2{\dot{\sigma }}_{\eta }^2+{\dot{\varrho }}^2+0-0\hfill \\ \hfill & =-{\dot{\sigma }}_{ϵ}^2{\dot{\sigma }}_{\eta }^2+{\dot{\varrho }}^2\hfill \\ \hfill \mathbb{E}\left[d_{\beta ,1}{E}_{\beta ,221}\left(1\right)\right]& =\mathbb{E}[{\dot{g}}_1{\dot{ϵ}}_1(-{\dot{g}}_2{\dot{\eta }}_2{X}_2^{\top }{\dot{ϵ}}_2{V}_X^{-1}{X}_1{\dot{\eta }}_1+{\dot{g}}_2{\dot{\eta }}_2{Z}_2{\dot{g}}_1{\dot{ϵ}}_1\hfill \\ & +{\dot{g}}_2{\dot{ϵ}}_2{\dot{g}}_2{X}_2{V}_X^{-1}{\dot{X}}_1{\dot{\eta }}_1-{\dot{g}}_2{\dot{ϵ}}_2{\dot{g}}_1{\dot{\eta }}_1\left)\right]\hfill \\ \hfill & =-\mathbb{E}\left[{\dot{g}}_1{\dot{ϵ}}_1{\dot{g}}_2{\dot{\eta }}_2{X}_2^{\top }{\dot{ϵ}}_2{V}_X^{-1}{X}_1{\dot{\eta }}_1\right]+\mathbb{E}\left[{\dot{g}}_1{\dot{ϵ}}_1{\dot{g}}_2{\dot{\eta }}_2{Z}_2{\dot{g}}_1{\dot{ϵ}}_1\right]\hfill \\ \hfill & +\mathbb{E}\left[{\dot{g}}_1{\dot{ϵ}}_1{\dot{g}}_2{\dot{ϵ}}_2{\dot{g}}_2{X}_2{V}_X^{-1}{\dot{X}}_1{\dot{\eta }}_1\right]-\mathbb{E}\left[{\dot{g}}_1{\dot{ϵ}}_1{\dot{g}}_2{\dot{ϵ}}_2{\dot{g}}_1{\dot{\eta }}_1\right]\hfill \\ \hfill & =-{\dot{\varrho }}^2+{\dot{\sigma }}_{ϵ}^2{\dot{\sigma }}^2+0-0\hfill \\ \hfill & =-{\dot{\varrho }}^2+{\dot{\sigma }}_{ϵ}^2{\dot{\sigma }}^2\hfill \end{array}$$

(162)

so that

$$\sum _{\sigma \left(1122\right)}\mathbb{E}\left[d_{\beta ,i_1}{E}_{\beta ,i_2{i}_3{i}_4}\left(1\right)\right]=-(l-1){\dot{\varrho }}^2$$

(163)

For $l=2$,

$$\begin{array}{cc}\hfill 2\mathbb{E}\left[d_{\beta ,1}{E}_{\beta ,122}\left(2\right)\right]& =\mathbb{E}[{\dot{g}}_1{\dot{ϵ}}_1{\dot{g}}_1{\dot{\eta }}_1{\dot{g}}_2{\dot{g}}_2({\dot{\eta }}_2{\dot{ϵ}}_2+{\dot{ϵ}}_2{\dot{\eta }}_2)\hfill \\ \hfill & =2{\dot{\varrho }}^2\hfill \\ \hfill 2\mathbb{E}\left[d_{\beta ,1}{E}_{\beta ,212}\left(2\right)\right]& =\mathbb{E}[{\dot{g}}_1{\dot{ϵ}}_1{\dot{g}}_2{\dot{\eta }}_2{\dot{g}}_1{\dot{g}}_2({\dot{\eta }}_1{\dot{ϵ}}_2+{\dot{ϵ}}_1{\dot{\eta }}_2)\hfill \\ \hfill & ={\dot{\varrho }}^2+{\dot{\sigma }}_{ϵ}^2{\dot{\sigma }}_{\eta }^2\hfill \\ \hfill 2\mathbb{E}\left[d_{\beta ,1}{E}_{\beta ,221}\left(2\right)\right]& =\mathbb{E}[{\dot{g}}_1{\dot{ϵ}}_1{\dot{g}}_2{\dot{\eta }}_2{\dot{g}}_2{\dot{g}}_1({\dot{\eta }}_2{\dot{ϵ}}_1+{\dot{ϵ}}_2{\dot{\eta }}_1)\hfill \\ \hfill & ={\dot{\sigma }}_{ϵ}^2{\dot{\sigma }}_{\eta }^2+{\dot{\varrho }}^2\hfill \end{array}$$

(164)

so that

$$\sum _{\sigma \left(1122\right)}\mathbb{E}\left[d_{\beta ,i_1}{E}_{\beta ,i_2{i}_3{i}_4}\left(2\right)\right]={\dot{\sigma }}_{ϵ}^2{\dot{\sigma }}_{\eta }^2+2{\dot{\varrho }}^2.$$

(165)

For $l=3$, $\mathbb{E}\left[d_{\beta ,.}{E}_{\beta ,\cdots }\left(3\right)\right]=0.$

For $l=4$, we have

$$\begin{array}{cc}\hfill \mathbb{E}\left[d_{\beta ,1}{E}_{\beta ,122}\left(4\right)\right]& =\mathbb{E}[{\dot{g}}_1{\dot{ϵ}}_1\{-X_1^{\top }{\dot{ϵ}}_1{V}_X^{-1}{W}_2{V}_X^{-1}{X}_2{\dot{\eta }}_2\hfill \\ & -Z_1(X_2^{\top }{\dot{ϵ}}_2{V}_X^{-1}{X}_2{\dot{\eta }}_2-Z_2{\dot{g}}_2{\dot{ϵ}}_2)\left\}\right]\hfill \\ \hfill & =0-l{\dot{\varrho }}^2+{\dot{\varrho }}^2\hfill \\ \hfill & =-(l-1){\dot{\varrho }}^2,\hfill \\ \hfill \mathbb{E}\left[d_{\beta ,1}{E}_{\beta ,212}\left(4\right)\right]& =\mathbb{E}[{\dot{g}}_1{\dot{ϵ}}_1\{-X_2^{\top }{\dot{ϵ}}_2{V}_X^{-1}{W}_1{V}_X^{-1}{X}_2{\dot{\eta }}_2\hfill \\ & -Z_2(X_1^{\top }{\dot{ϵ}}_1{V}_X^{-1}{X}_2{\dot{\eta }}_2-Z_1{\dot{g}}_2{\dot{ϵ}}_2)\left\}\right]\hfill \\ \hfill & =0-{\dot{\sigma }}_{ϵ}^2{\dot{\sigma }}_{\eta }^2+{\dot{\varrho }}^2\hfill \\ \hfill & ={\dot{\varrho }}^2-{\dot{\sigma }}_{ϵ}^2{\dot{\sigma }}_{\eta }^2,\hfill \\ \hfill \mathbb{E}\left[d_{\beta ,1}{E}_{\beta ,221}\left(4\right)\right]& =\mathbb{E}[{\dot{g}}_1{\dot{ϵ}}_1\{-X_2^{\top }{\dot{ϵ}}_2{V}_X^{-1}{W}_2{V}_X^{-1}{X}_1{\dot{\eta }}_1\hfill \\ & -Z_2(X_2^{\top }{\dot{ϵ}}_2{V}_X^{-1}{X}_1{\dot{\eta }}_1-Z_2{\dot{g}}_1{\dot{ϵ}}_1)\left\}\right]\hfill \\ \hfill & =0-{\dot{\varrho }}^2+{\dot{\sigma }}_{ϵ}^2{\dot{\sigma }}_{\eta }^2+{\dot{\sigma }}_{ϵ}^2\mathrm{Var}\left[{\widehat{g}}_1^2\right]\hfill \\ \hfill & ={\dot{\sigma }}_{ϵ}^2{\dot{\sigma }}_{\eta }^2-{\dot{\varrho }}^2+{\dot{\sigma }}_{ϵ}^2\mathrm{Var}\left[{\widehat{g}}_1^2\right],\hfill \end{array}$$

(166)

using the expression $\mathbb{E}\left[Z_1^2\right]={\dot{\sigma }}_{\eta }^2+\mathrm{Var}\left[{\widehat{g}}_1^2\right]$, so that

$$\sum _{\sigma \left(1122\right)}\mathbb{E}\left[d_{\beta ,i_1}{E}_{\beta ,i_2{i}_3{i}_4}\left(4\right)\right]=-(l-1){\dot{\varrho }}^2+{\dot{\sigma }}_{ϵ}^2\mathrm{Var}\left[{\widehat{g}}_1^2\right].$$

(167)

For $l=5$, we have

$$\begin{array}{cc}\hfill 2\mathbb{E}\left[d_{\beta ,1}{E}_{\beta ,122}\left(5\right)\right]& =\mathbb{E}\left[{\dot{g}}_1{\dot{ϵ}}_1{Z}_1{\dot{g}}_2{\dot{g}}_2({\dot{\eta }}_2{\dot{ϵ}}_2+{\dot{ϵ}}_2{\dot{\eta }}_2)\right]\hfill \\ & =2{\dot{\varrho }}^2,\hfill \\ \hfill 2\mathbb{E}\left[d_{\beta ,1}{E}_{\beta ,212}\left(5\right)\right]& =\mathbb{E}\left[{\dot{g}}_1{\dot{ϵ}}_1{Z}_2{\dot{g}}_1{\dot{g}}_2({\dot{\eta }}_1{\dot{ϵ}}_2+{\dot{ϵ}}_1{\dot{\eta }}_2)\right]\hfill \\ & ={\dot{\varrho }}^2+{\dot{\sigma }}_{ϵ}^2{\dot{\sigma }}_{\eta }^2,\hfill \\ \hfill 2\mathbb{E}\left[d_{\beta ,1}{E}_{\beta ,221}\left(5\right)\right]& =\mathbb{E}\left[{\dot{g}}_1{\dot{ϵ}}_1{Z}_2{\dot{g}}_2{\dot{g}}_1({\dot{\eta }}_2{\dot{ϵ}}_1+{\dot{ϵ}}_2{\dot{\eta }}_1)\right]\hfill \\ & ={\dot{\sigma }}_{ϵ}^2{\dot{\sigma }}_{\eta }^2+{\dot{\varrho }}^2\hfill \end{array}$$

(168)

so that

$$\sum _{\sigma \left(1122\right)}\mathbb{E}\left[d_{\beta ,i_1}{E}_{\beta ,i_2{i}_3{i}_4}\left(5\right)\right]={\dot{\sigma }}_{ϵ}^2{\dot{\sigma }}_{\eta }^2+2{\dot{\varrho }}^2.$$

(169)

For $l=6$, we have

$$\begin{array}{cc}\hfill -2\mathbb{E}\left[d_{\beta ,1}{E}_{\beta ,122}\left(6\right)\right]& =\mathbb{E}[{\dot{g}}_1{\dot{ϵ}}_1\{{\dot{g}}_1{X}_1^{\top }{V}_X^{-1}{X}_2{\dot{\eta }}_2{\dot{g}}_2{\dot{ϵ}}_2+{\dot{g}}_1{X}_1^{\top }{V}_X^{-1}{\dot{g}}_2{\dot{ϵ}}_2{X}_2{\dot{\eta }}_2\hfill \\ \hfill & +X_1^{\top }{\eta }_1{V}_X^{-1}{X}_2{\dot{\eta }}_2{\dot{g}}_2{\dot{ϵ}}_2+X_1^{\top }{\eta }_1{V}_X^{-1}{\dot{g}}_2{\dot{ϵ}}_2{X}_2{\dot{\eta }}_2\hfill \\ \hfill & -({\dot{g}}_2{\eta }_2{\dot{g}}_2{\dot{ϵ}}_2+{\dot{g}}_2{\dot{ϵ}}_2{\dot{g}}_2{\dot{\eta }}_2)\left\}\right]\hfill \\ & =0+0+{\dot{\varrho }}^2+{\dot{\varrho }}^2+0+0\hfill \\ & =2{\dot{\varrho }}^2,\hfill \\ \hfill -2\mathbb{E}\left[d_{\beta ,1}{E}_{\beta ,212}\left(6\right)\right]& =\mathbb{E}[{\dot{g}}_1{\dot{ϵ}}_1\{{\dot{g}}_2{X}_2^{\top }{V}_X^{-1}{X}_1{\dot{\eta }}_1{\dot{g}}_2{\dot{ϵ}}_2+{\dot{g}}_2{X}_2^{\top }{V}_X^{-1}{\dot{g}}_1{\dot{ϵ}}_1{X}_2{\dot{\eta }}_2\hfill \\ \hfill & +X_2^{\top }{\eta }_2{V}_X^{-1}{X}_1{\dot{\eta }}_1{\dot{g}}_2{\dot{ϵ}}_2+X_2^{\top }{\eta }_2{V}_X^{-1}{\dot{g}}_1{\dot{ϵ}}_1{X}_2{\dot{\eta }}_2\hfill \\ \hfill & -({\dot{g}}_1{\eta }_1{\dot{g}}_2{\dot{ϵ}}_2+{\dot{g}}_1{\dot{ϵ}}_1{\dot{g}}_2{\dot{\eta }}_2)\left\}\right]\hfill \\ & =0+0+{\dot{\varrho }}^2+l{\dot{\sigma }}_{ϵ}^2{\dot{\sigma }}_{\eta }^2-0-0\hfill \\ & ={\dot{\varrho }}^2+l{\dot{\sigma }}_{ϵ}^2{\dot{\sigma }}_{\eta }^2,\hfill \\ \hfill -2\mathbb{E}\left[d_{\beta ,1}{E}_{\beta ,221}\left(6\right)\right]& =\mathbb{E}[{\dot{g}}_1{\dot{ϵ}}_1\{{\dot{g}}_2{X}_2^{\top }{V}_X^{-1}{X}_2{\dot{\eta }}_2{\dot{g}}_1{\dot{ϵ}}_1+{\dot{g}}_2{X}_2^{\top }{V}_X^{-1}{\dot{g}}_2{\dot{ϵ}}_2{X}_1{\dot{\eta }}_1\hfill \\ \hfill & +X_2^{\top }{\eta }_2{V}_X^{-1}{X}_2{\dot{\eta }}_2{\dot{g}}_1{\dot{ϵ}}_1+X_2^{\top }{\eta }_2{V}_X^{-1}{\dot{g}}_2{\dot{ϵ}}_2{X}_1{\dot{\eta }}_1\hfill \\ \hfill & -({\dot{g}}_2{\eta }_2{\dot{g}}_1{\dot{ϵ}}_1+{\dot{g}}_2{\dot{ϵ}}_2{\dot{g}}_1{\dot{\eta }}_1)\left\}\right]\hfill \\ & =0+0+l{\dot{\sigma }}_{ϵ}^2{\dot{\sigma }}_{\eta }^2+{\dot{\varrho }}^2-0-0\hfill \\ & =l{\dot{\sigma }}_{ϵ}^2{\dot{\sigma }}_{\eta }^2+{\dot{\varrho }}^2,\hfill \end{array}$$

(170)

so that

$$\sum _{\sigma \left(1122\right)}\mathbb{E}\left[d_{\beta ,i_1}{E}_{\beta ,i_2{i}_3{i}_4}\left(6\right)\right]=-l{\dot{\sigma }}_{ϵ}^2{\dot{\sigma }}_{\eta }^2-2{\dot{\varrho }}^2.$$

(171)

Putting together these results, we have

$$\begin{array}{cc}\hfill V_{\beta ,22}& =\sum _{l=1}^6\sum _{\sigma \left(1122\right)}\mathbb{E}\left[d_{\beta ,i_1}{E}_{\beta ,i_2{i}_3{i}_4}\left(l\right)\right]\hfill \\ \hfill & =-(l-2){\dot{\sigma }}_{ϵ}^2{\dot{\sigma }}_{\eta }^2-2(l-2){\dot{\varrho }}^2+{\dot{\sigma }}_{ϵ}^2\mathrm{Var}\left[{\widehat{g}}_1^2\right]\hfill \end{array}$$

(172)

We find the components of the approximate kurtosis for the 2SLS estimator as follows:

$$\begin{array}{cc}\hfill K_{\beta ,1}& =\mathbb{E}\left[d_{\beta ,1}^4\right]-3V_{\beta ,1}^2\hfill \\ & =\mathbb{E}\left[{\left({\dot{g}}_1{\dot{ϵ}}_1\right)}^4\right]-3\mathbb{E}{\left[{\left({\dot{g}}_1{\dot{ϵ}}_1\right)}^2\right]}^2\hfill \\ & =\mathbb{E}\left[{\dot{g}}_1^4\right]\mathbb{E}\left[{\dot{ϵ}}_1^4\right]-3{\left({\dot{\sigma }}_{ϵ}^2\right)}^2\hfill \\ & =\mathrm{Var}\left[{\dot{g}}_1^2\right]\mathbb{E}\left[{\dot{ϵ}}_1^4\right]+(\mathbb{E}\left[{\dot{ϵ}}_1^4\right]-3{\left({\dot{\sigma }}_{ϵ}^2\right)}^2).\hfill \end{array}$$

(173)

$$\begin{array}{cc}\hfill K_{\beta ,2}& =\sum _{\begin{array}{c}\sigma \left(11122\right)\\ \cup \sigma \left(11222\right)\end{array}}\mathbb{E}\left[d_{\beta ,i_1}{d}_{\beta ,i_2}{d}_{\beta ,i_3}{d}_{\beta ,i_4{i}_5}\right].\hfill \end{array}$$

(174)

We note from symmetry that some of the summands in $K_{\beta ,2}$ are equal for $i,j$. Noting the symmetry we evaluate the summands in $K_{\beta ,2}$ as follows. Recall that

$$d_{\beta ,i_1,i_2}=A_{\beta ,i_1{i}_2}\left(1\right)+A_{\beta ,i_1{i}_2}\left(2\right).$$

(175)

$$\begin{array}{cc}\hfill A_{\beta ,i_1{i}_2}\left(1\right)& =X_{i_1}^{\top }{\dot{ϵ}}_{i_1}{V}_X^{-1}{X}_{i_2}{\dot{\eta }}_{i_2}-Z_{i_1}{\dot{g}}_{i_2}{\dot{ϵ}}_{i_2},\hfill \\ \hfill 2A_{\beta ,i_1{i}_2}\left(2\right)& =-\{{\dot{g}}_{i_1}{\dot{g}}_{i_2}{\dot{\eta }}_{i_1}{\dot{ϵ}}_{i_2}+{\dot{g}}_{i_2}{\dot{g}}_{i_1}{\dot{\eta }}_{i_2}{\dot{ϵ}}_{i_1}\}.\hfill \end{array}$$

(176)

$$\mathbb{E}\left[d_{\beta ,1}{d}_{\beta ,1}{d}_{\beta ,1}{d}_{\beta ,22}\right]=\mathbb{E}\left[d_{\beta ,1}^3\right]\mathbb{E}\left[d_{\beta ,22}\right]=S_{\beta ,1}{B}_2.$$

(177)

$$\begin{array}{cc}\hfill \mathbb{E}\left[d_{\beta ,1}{d}_{\beta ,1}{d}_{\beta ,2}{d}_{\beta ,12}\right]& =\mathbb{E}\left[d_{\beta ,1}{d}_{\beta ,2}{d}_{\beta ,1}{d}_{\beta ,12}\right]=\mathbb{E}\left[d_{\beta ,1}{d}_{\beta ,2}{d}_{\beta ,2}{d}_{\beta ,21}\right]\hfill \\ \hfill & =\mathbb{E}\left[{\left({\dot{g}}_1{\dot{ϵ}}_1\right)}^2{\dot{g}}_2{\dot{ϵ}}_2(X_1^{\top }{\dot{ϵ}}_1{V}_X^{-1}{X}_2{\dot{\eta }}_2-Z_1{\dot{g}}_2{\dot{ϵ}}_2)\right]\hfill \\ \hfill & -\mathbb{E}\left[{\left({\dot{g}}_1{\dot{ϵ}}_1\right)}^2{\dot{g}}_2{\dot{ϵ}}_2({\dot{g}}_1{\dot{g}}_2{\dot{\eta }}_1{\dot{ϵ}}_2+{\dot{g}}_2{\dot{g}}_1{\dot{\eta }}_2{\dot{ϵ}}_1)\right]/2\hfill \\ \hfill & =\mathbb{E}\left[{\left({\dot{g}}_1{\dot{ϵ}}_1\right)}^2{\dot{g}}_2{\dot{ϵ}}_2(X_1^{\top }{\dot{ϵ}}_1{V}_X^{-1}{X}_2{\dot{\eta }}_2-({\dot{g}}_1^2+{\dot{g}}_1{\dot{\eta }}_1-1){\dot{g}}_2{\dot{ϵ}}_2)\right]\hfill \\ \hfill & -\mathbb{E}\left[{\left({\dot{g}}_1{\dot{ϵ}}_1\right)}^2{\dot{g}}_2{\dot{ϵ}}_2({\dot{g}}_1{\dot{g}}_2{\dot{\eta }}_1{\dot{ϵ}}_2+{\dot{g}}_2{\dot{g}}_1{\dot{\eta }}_2{\dot{ϵ}}_1)\right]/2\hfill \\ \hfill & =S_{\beta ,1}\dot{\varrho }-\mathbb{E}\left[{\dot{g}}_1^3\right]\mathbb{E}\left[{\dot{ϵ}}_1^2{\dot{\eta }}_1\right]{\dot{\sigma }}_{ϵ}^2-\mathbb{E}\left[{\dot{g}}_1^4\right]{\dot{\sigma }}_{ϵ}^2\dot{\varrho }+{\dot{\sigma }}_{ϵ}^2\dot{\varrho }\hfill \\ \hfill & -\mathbb{E}\left[{\dot{g}}_1^3\right]\mathbb{E}\left[{\dot{ϵ}}_1^2{\dot{\eta }}_1\right]{\dot{\sigma }}_{ϵ}^2/2-S_{\beta ,1}\dot{\varrho }/2\hfill \\ \hfill & =S_{\beta ,1}\dot{\varrho }/2-3\mathbb{E}\left[{\dot{g}}_1^3\right]\mathbb{E}\left[{\dot{ϵ}}_1^2{\dot{\eta }}_1\right]{\dot{\sigma }}_{ϵ}^2/2-\mathrm{Var}\left[{\dot{g}}_1^2\right]{\dot{\sigma }}_{ϵ}^2\dot{\varrho }.\hfill \end{array}$$

(178)

$$\begin{array}{cc}\hfill \mathbb{E}\left[d_{\beta ,1}{d}_{\beta ,1}{d}_{\beta ,2}{d}_{\beta ,21}\right]& =\mathbb{E}\left[d_{\beta ,1}{d}_{\beta ,2}{d}_{\beta ,1}{d}_{\beta ,21}\right]=\mathbb{E}\left[d_{\beta ,1}{d}_{\beta ,2}{d}_{\beta ,2}{d}_{\beta ,12}\right]\hfill \\ \hfill & =\mathbb{E}\left[{\left({\dot{g}}_1{\dot{ϵ}}_1\right)}^2{\dot{g}}_2{\dot{ϵ}}_2(X_2^{\top }{\dot{ϵ}}_2{V}_X^{-1}{X}_1{\dot{\eta }}_1-Z_2{\dot{g}}_1{\dot{ϵ}}_1)\right]\hfill \\ \hfill & -\mathbb{E}\left[{\left({\dot{g}}_1{\dot{ϵ}}_1\right)}^2{\dot{g}}_2{\dot{ϵ}}_2({\dot{g}}_2{\dot{g}}_1{\dot{\eta }}_2{\dot{ϵ}}_1+{\dot{g}}_1{\dot{g}}_2{\dot{\eta }}_1{\dot{ϵ}}_2)\right]/2\hfill \\ \hfill & =\mathbb{E}\left[{\left({\dot{g}}_1{\dot{ϵ}}_1\right)}^2{\dot{g}}_2{\dot{ϵ}}_2(X_2^{\top }{\dot{ϵ}}_2{V}_X^{-1}{X}_1{\dot{\eta }}_1-({\dot{g}}_2^2+{\dot{g}}_2{\dot{\eta }}_2-1){\dot{g}}_1{\dot{ϵ}}_1)\right]\hfill \\ \hfill & -\mathbb{E}\left[{\left({\dot{g}}_1{\dot{ϵ}}_1\right)}^2{\dot{g}}_2{\dot{ϵ}}_2({\dot{g}}_2{\dot{g}}_1{\dot{\eta }}_2{\dot{ϵ}}_1+{\dot{g}}_1{\dot{g}}_2{\dot{\eta }}_1{\dot{ϵ}}_2)\right]/2\hfill \\ \hfill & =\mathbb{E}\left[{\dot{g}}_1^3\right]\mathbb{E}\left[{\dot{ϵ}}_1^2{\dot{\eta }}_1\right]{\dot{\sigma }}_{ϵ}^2-0-\mathbb{E}\left[{\dot{g}}_1^3\right]\mathbb{E}\left[{\dot{ϵ}}_1^3\right]\dot{\varrho }+0\hfill \\ \hfill & -\mathbb{E}\left[{\dot{g}}_1^3\right]\mathbb{E}\left[{\dot{ϵ}}_1^3\right]\dot{\varrho }/2-\mathbb{E}\left[{\dot{g}}_1^3\right]\mathbb{E}\left[{\dot{ϵ}}_1^2{\dot{\eta }}_1\right]{\dot{\sigma }}_{ϵ}^2\hfill \\ \hfill & =\mathbb{E}\left[{\dot{g}}_1^3\right]\mathbb{E}\left[{\dot{ϵ}}_1^2{\dot{\eta }}_1\right]{\dot{\sigma }}_{ϵ}^2/2-3S_{\beta ,1}\dot{\varrho }/2.\hfill \end{array}$$

(179)

$$\begin{array}{cc}\hfill \mathbb{E}\left[d_{\beta ,1}{d}_{\beta ,2}{d}_{\beta ,2}{d}_{\beta ,11}\right]& =\mathbb{E}\left[d_{\beta ,1}{d}_{\beta ,1}{d}_{\beta ,2}{d}_{\beta ,22}\right]=\mathbb{E}\left[d_{\beta ,1}{d}_{\beta ,2}{d}_{\beta ,1}{d}_{\beta ,22}\right]\hfill \\ \hfill & =\mathbb{E}\left[d_{\beta ,2}^2\right]\mathbb{E}\left[d_{\beta ,1}{d}_{\beta ,1}\right]\hfill \\ \hfill & =V_{\beta ,1}{V}_{\beta ,21}.\hfill \end{array}$$

(180)

Putting these results together, we have

$$\begin{array}{cc}\hfill K_{\beta ,2}& =S_{\beta ,1}{B}_{\beta ,2}+3(s_1\dot{\varrho }/2-3\mathbb{E}\left[{\dot{g}}_1^3\right]\mathbb{E}\left[{\dot{ϵ}}_1^2{\dot{\eta }}_1\right]{\dot{\sigma }}_{ϵ}^2/2)\hfill \\ \hfill & +3(\mathbb{E}\left[{\dot{g}}_1^3\right]\mathbb{E}\left[{\dot{ϵ}}_1^2{\dot{\eta }}_1\right]{\dot{\sigma }}_{ϵ}^2/2-3s_1\dot{\varrho }/2)+3{\dot{\sigma }}_{ϵ}^2{V}_{\beta ,21}\hfill \\ \hfill & =S_{\beta ,1}{B}_{\beta ,2}+3({\dot{\sigma }}_{ϵ}^2{V}_{\beta ,21}-\mathbb{E}\left[{\dot{g}}_1^3\right]\mathbb{E}\left[{\dot{ϵ}}_1^2{\dot{\eta }}_1\right]{\dot{\sigma }}_{ϵ}^2-S_{\beta ,1}\dot{\varrho }-\mathrm{Var}\left[{\dot{g}}_1^2\right]{\dot{\sigma }}_{ϵ}^2\dot{\varrho }).\hfill \end{array}$$

(181)

$$K_{\beta ,3}=\sum _{\sigma \left(112233\right)}\mathbb{E}\left[d_{i_1}{d}_{i_2}{d}_{i_3}{d}_{i_4{i}_5{i}_6}\right].$$

(182)

We note from symmetry that many of the summands in $K_{\beta ,3}$ are equal for different permutations of the indices. These are

$$\begin{array}{cc}\hfill {\sigma }_a& =\left\{\right(112233),(121233),(122133\left)\right\},\hfill \\ \hfill {\sigma }_b& =\left\{\right(112323),(121323),(122313\left)\right\}.\hfill \\ \hfill {\sigma }_c& =\left\{\right(112332),(121332),(122331\left)\right\},\hfill \\ \hfill {\sigma }_d& =\left\{\right(123123),(123132),(123213),(123312),(123231),(123321\left)\right\}.\hfill \end{array}$$

(183)

First, we see that

$$\begin{array}{cc}\hfill \mathbb{E}\left[d_{\beta ,1}{d}_{\beta ,1}{d}_{\beta ,2}{d}_{\beta ,233}\right]& =\mathbb{E}\left[d_{\beta ,1}{d}_{\beta ,2}{d}_{\beta ,1}{d}_{\beta ,233}\right]=\mathbb{E}\left[d_{\beta ,1}{d}_{\beta ,2}{d}_{\beta ,2}{d}_{\beta ,133}\right]\hfill \\ & =V_{\beta ,1}\mathbb{E}\left[d_2{d}_{233}\right],\hfill \\ \hfill \mathbb{E}\left[d_{\beta ,1}{d}_{\beta ,1}{d}_{\beta ,2}{d}_{\beta ,323}\right]& =\mathbb{E}\left[d_{\beta ,1}{d}_{\beta ,2}{d}_{\beta ,1}{d}_{\beta ,323}\right]=\mathbb{E}\left[d_{\beta ,1}{d}_{\beta ,2}{d}_{\beta ,2}{d}_{\beta ,313}\right]\hfill \\ & =V_{\beta ,1}\mathbb{E}\left[d_{\beta ,2}{d}_{\beta ,323}\right],\hfill \\ \hfill \mathbb{E}\left[d_{\beta ,1}{d}_{\beta ,1}{d}_{\beta ,2}{d}_{\beta ,332}\right]& =\mathbb{E}\left[d_{\beta ,1}{d}_{\beta ,2}{d}_{\beta ,1}{d}_{\beta ,332}\right]=\mathbb{E}\left[d_{\beta ,1}{d}_{\beta ,2}{d}_{\beta ,2}{d}_{\beta ,331}\right]\hfill \\ & =V_{\beta ,1}\mathbb{E}\left[d_{\beta ,2}{d}_{\beta ,332}\right]\hfill \end{array}$$

(184)

By inspection and comparison with (161) we see that

$$\begin{array}{cc}\hfill V_{\beta ,1}{V}_{\beta ,22}& =\mathbb{E}\left[d_{\beta ,1}{d}_{\beta ,1}{d}_{\beta ,2}{d}_{\beta ,233}\right]+\mathbb{E}\left[d_{\beta ,1}{d}_{\beta ,1}{d}_{\beta ,2}{d}_{\beta ,323}\right]\hfill \\ \hfill & +\mathbb{E}\left[d_{\beta ,1}{d}_{\beta ,1}{d}_{\beta ,2}{d}_{\beta ,332}\right]\hfill \end{array}$$

(185)

so that

$$\sum _{{\sigma }_a\cup {\sigma }_b\cup {\sigma }_c}\mathbb{E}\left[d_{\beta ,i_1}{d}_{\beta ,i_2}{d}_{\beta ,i_3}{d}_{\beta ,i_4{i}_5{i}_6}\right]=3V_{\beta ,1}{V}_{\beta ,22}.$$

(186)

With respect to the six terms in ${\sigma }_d$ we have

$$\begin{array}{cc}\hfill \mathbb{E}\left[d_{\beta ,1}{d}_{\beta ,2}{d}_{\beta ,3}{d}_{\beta ,123}\right]& =\mathbb{E}\left[d_{\beta ,1}{d}_{\beta ,2}{d}_{\beta ,3}{d}_{\beta ,132}\right]=\mathbb{E}\left[d_{\beta ,1}{d}_{\beta ,2}{d}_{\beta ,3}{d}_{\beta ,213}\right]\hfill \\ \hfill & =\mathbb{E}\left[d_{\beta ,1}{d}_{\beta ,2}{d}_{\beta ,3}{d}_{\beta ,312}\right]=\mathbb{E}\left[d_{\beta ,1}{d}_{\beta ,2}{d}_{\beta ,3}{d}_{\beta ,231}\right]\hfill \\ \hfill & =\mathbb{E}\left[d_{\beta ,1}{d}_{\beta ,2}{d}_{\beta ,3}{d}_{\beta ,321}\right]\hfill \\ \hfill & =\sum _{l=1}^6\mathbb{E}\left[d_{\beta ,1}{d}_{\beta ,2}{d}_{\beta ,3}{E}_{\beta ,123}\left(l\right)\right].\hfill \end{array}$$

(187)

We evaluate the summands for $l=1,2,\dots ,6$. Recall that $Z_i={\dot{g}}_i^2+{\dot{g}}_i{\dot{\eta }}_i-1$.

$$\begin{array}{cc}\hfill E_{\beta ,i_1{i}_2{i}_3}\left(1\right)& =-{\dot{g}}_{i_1}{\dot{\eta }}_{i_1}{X}_{i_2}^{\top }{\dot{ϵ}}_{i_2}{V}_X^{-1}{X}_{i_3}{\dot{\eta }}_{i_3}+{\dot{g}}_{i_1}{\dot{\eta }}_{i_1}{Z}_{i_2}{\dot{g}}_{i_3}{\dot{ϵ}}_{i_3}\hfill \\ & +{\dot{g}}_{i_1}{\dot{ϵ}}_{i_1}{\dot{g}}_{i_2}{X}_{i_2}{V}_X^{-1}{\dot{X}}_{i_3}{\dot{\eta }}_{i_3}-{\dot{g}}_{i_1}{\dot{ϵ}}_{i_1}{\dot{g}}_{i_3}{\dot{\eta }}_{i_3}.\hfill \end{array}$$

$$\begin{array}{cc}\hfill \mathbb{E}\left[d_{\beta ,1}{d}_{\beta ,2}{d}_{\beta ,3}{E}_{\beta ,123}\left(1\right)\right]& =-\mathbb{E}\left[{\dot{g}}_1{\dot{ϵ}}_1{\dot{g}}_2{\dot{ϵ}}_2{\dot{g}}_3{\dot{ϵ}}_3{\dot{g}}_1{\dot{\eta }}_1{X}_2^{\top }{\dot{ϵ}}_2{V}_X^{-1}{X}_3{\dot{\eta }}_3\right]+\mathbb{E}\left[{\dot{g}}_1{\dot{ϵ}}_1{\dot{g}}_2{\dot{ϵ}}_2{\dot{g}}_3{\dot{ϵ}}_3{\dot{g}}_1{\dot{\eta }}_1{Z}_2{\dot{g}}_3{\dot{ϵ}}_3\right]\hfill \\ & +\mathbb{E}\left[{\dot{g}}_1{\dot{ϵ}}_1{\dot{g}}_2{\dot{ϵ}}_2{\dot{g}}_3{\dot{ϵ}}_3{\dot{g}}_1{\dot{ϵ}}_1{\dot{g}}_2{X}_2{V}_X^{-1}{\dot{X}}_3{\dot{\eta }}_3\right]-\mathbb{E}\left[{\dot{g}}_1{\dot{ϵ}}_1{\dot{g}}_2{\dot{ϵ}}_2{\dot{g}}_3{\dot{ϵ}}_3{\dot{g}}_1{\dot{ϵ}}_1{\dot{g}}_3{\dot{\eta }}_3\right]\hfill \\ \hfill & =-{\dot{\rho }}^2{\dot{\sigma }}_{ϵ}^2+{\dot{\rho }}^2{\dot{\sigma }}_{ϵ}^2+0-0\hfill \\ \hfill & =0.\hfill \end{array}$$

(188)

$$\begin{array}{cc}\hfill 2\mathbb{E}\left[d_1{d}_2{d}_3{E}_{\beta ,123}\left(2\right)\right]& =\mathbb{E}\left[{\dot{g}}_1{\dot{ϵ}}_1{\dot{g}}_2{\dot{ϵ}}_2{\dot{g}}_3{\dot{ϵ}}_3{\dot{g}}_1{\dot{\eta }}_1{\dot{g}}_2{\dot{g}}_3({\dot{\eta }}_2{\dot{ϵ}}_3+{\dot{ϵ}}_2{\dot{\eta }}_3)\right]\hfill \\ & =2{\dot{\rho }}^2{\dot{\sigma }}_{ϵ}^2.\hfill \end{array}$$

(189)

$$\mathbb{E}\left[d_1{d}_2{d}_3{E}_{\beta ,123}\left(3\right)\right]=0.$$

(190)

$$\begin{array}{cc}\hfill \mathbb{E}\left[d_1{d}_2{d}_3{E}_{\beta ,123}\left(4\right)\right]& =-\mathbb{E}\left[{\dot{g}}_1{\dot{ϵ}}_1{\dot{g}}_2{\dot{ϵ}}_2{\dot{g}}_3{\dot{ϵ}}_3{X}_1^{\top }{\dot{ϵ}}_1{V}_X^{-1}{W}_2{V}_X^{-1}{X}_3{\dot{\eta }}_3\right]\hfill \\ & -\mathbb{E}\left[{\dot{g}}_1{\dot{ϵ}}_1{\dot{g}}_2{\dot{ϵ}}_2{\dot{g}}_3{\dot{ϵ}}_3{Z}_1(X_2^{\top }{\dot{ϵ}}_2{V}_X^{-1}{X}_3{\dot{\eta }}_3-Z_2{\dot{g}}_3{\dot{ϵ}}_3)\right]\hfill \\ \hfill & =0-{\dot{\rho }}^2{\dot{\sigma }}_{ϵ}^2+{\dot{\rho }}^2{\dot{\sigma }}_{ϵ}^2\hfill \\ \hfill & =0.\hfill \end{array}$$

(191)

$$\begin{array}{cc}\hfill 2\mathbb{E}\left[d_1{d}_2{d}_3{E}_{\beta ,123}\left(5\right)\right]& =\mathbb{E}\left[{\dot{g}}_1{\dot{ϵ}}_1{\dot{g}}_2{\dot{ϵ}}_2{\dot{g}}_3{\dot{ϵ}}_3{Z}_1{\dot{g}}_2{\dot{g}}_3({\dot{\eta }}_2{\dot{ϵ}}_3+{\dot{ϵ}}_2{\dot{\eta }}_3)\right]\hfill \\ \hfill & =2{\dot{\rho }}^2{\dot{\sigma }}_{ϵ}^2.\hfill \end{array}$$

(192)

$$\begin{array}{cc}\hfill -2\mathbb{E}\left[d_1{d}_2{d}_3{E}_{\beta ,123}\left(6\right)\right]& =\mathbb{E}[{\dot{g}}_1{\dot{ϵ}}_1{\dot{g}}_2{\dot{ϵ}}_2{\dot{g}}_3{\dot{ϵ}}_3{\dot{g}}_1{X}_1^{\top }{V}_X^{-1}{X}_2{\dot{\eta }}_2{\dot{g}}_3{\dot{ϵ}}_3\hfill \\ \hfill & +\mathbb{E}[{\dot{g}}_1{\dot{ϵ}}_1{\dot{g}}_2{\dot{ϵ}}_2{\dot{g}}_3{\dot{ϵ}}_3{\dot{g}}_1{X}_1^{\top }{V}_X^{-1}{\dot{g}}_2{\dot{ϵ}}_2{X}_3{\dot{\eta }}_3\hfill \\ \hfill & +\mathbb{E}[{\dot{g}}_1{\dot{ϵ}}_1{\dot{g}}_2{\dot{ϵ}}_2{\dot{g}}_3{\dot{ϵ}}_3{X}_1^{\top }{\eta }_1{V}_X^{-1}{X}_2{\dot{\eta }}_2{\dot{g}}_3{\dot{ϵ}}_3\hfill \\ \hfill & +\mathbb{E}[{\dot{g}}_1{\dot{ϵ}}_1{\dot{g}}_2{\dot{ϵ}}_2{\dot{g}}_3{\dot{ϵ}}_3{X}_1^{\top }{\eta }_1{V}_X^{-1}{\dot{g}}_2{\dot{ϵ}}_2{X}_3{\dot{\eta }}_3\hfill \\ \hfill & -\mathbb{E}[{\dot{g}}_1{\dot{ϵ}}_1{\dot{g}}_2{\dot{ϵ}}_2{\dot{g}}_3{\dot{ϵ}}_3({\dot{g}}_2{\eta }_2{\dot{g}}_3{\dot{ϵ}}_3+{\dot{g}}_2{\dot{ϵ}}_2{\dot{g}}_3{\dot{\eta }}_3)\hfill \\ & =0+0+{\dot{\rho }}^2{\dot{\sigma }}_{ϵ}^2+{\dot{\rho }}^2{\dot{\sigma }}_{ϵ}^2+0\hfill \\ & =2{\dot{\rho }}^2{\dot{\sigma }}_{ϵ}^2.\hfill \end{array}$$

(193)

Thus,

$$\begin{array}{cc}\hfill \mathbb{E}\left[d_{\beta ,1}{d}_{\beta ,2}{d}_{\beta ,3}{d}_{\beta ,123}\right]& =\sum _{l=1}^6\mathbb{E}\left[d_{\beta ,1}{d}_{\beta ,2}{d}_{\beta ,3}{E}_{\beta ,123}\left(l\right)\right]\hfill \\ \hfill & ={\dot{\rho }}^2{\dot{\sigma }}_{ϵ}^2\hfill \end{array}$$

(194)

and so

$$\begin{array}{cc}\hfill \sum _{{\sigma }_d}\mathbb{E}\left[d_{i_1}{d}_{i_2}{d}_{i_3}{d}_{i_4{i}_5{i}_6}\right]& =6\sum _{l=1}^6{\dot{g}}_1{\dot{ϵ}}_1{\dot{g}}_2{\dot{ϵ}}_2{\dot{g}}_3{\dot{ϵ}}_3{E}_{\beta ,123}\left(l\right)\hfill \\ \hfill & =6{\dot{\rho }}^2{\dot{\sigma }}_{ϵ}^2\hfill \end{array}$$

(195)

so that

$$\begin{array}{cc}\hfill K_{\beta ,3}& =3V_{\beta ,1}{V}_{\beta ,22}+6{\dot{\rho }}^2{\dot{\sigma }}_{ϵ}^2\hfill \\ \hfill & =3V_{\beta ,1}(V_{\beta ,22}+2{\dot{\rho }}^2).\hfill \end{array}$$

(196)

Next, we evaluate

$$\begin{array}{c}\hfill K_{\beta ,4}=\sum _{\sigma \left(112233\right)}\mathbb{E}\left[d_{\beta ,i_1}{d}_{\beta ,i_2}{d}_{\beta ,i_3{i}_4}{d}_{\beta ,i_5{i}_6}\right].\end{array}$$

(197)

Examining the terms in $K_{\beta ,4}$, we see that the first three terms (as we go through the set of indices in $\sigma \left(112233\right)$) are

$$\begin{array}{cc}\hfill \mathbb{E}\left[d_{\beta ,1}^2(d_{\beta ,22}{d}_{\beta ,33}+d_{\beta ,23}{d}_{\beta ,23}+d_{\beta ,23}{d}_{\beta ,32})\right]& =\mathbb{E}\left[d_{\beta ,1}^2\right]\times \mathbb{E}[d_{\beta ,22}{d}_{\beta ,33}\hfill \\ \hfill & +d_{\beta ,23}{d}_{\beta ,23}+d_{\beta ,23}{d}_{\beta ,32}]\hfill \\ \hfill & =V_{\beta ,1}{V}_{\beta ,23}.\hfill \end{array}$$

(198)

Next, we see that

$$\begin{array}{cc}\hfill \mathbb{E}\left[d_1{d}_2{d}_{12}{d}_{33}\right]& =\mathbb{E}\left[d_1{d}_2{d}_{21}{d}_{33}\right]=\mathbb{E}\left[d_1{d}_2{d}_{33}{d}_{12}\right]=\mathbb{E}\left[d_1{d}_2{d}_{33}{d}_{21}\right]\hfill \\ \hfill & =\mathbb{E}[{\dot{g}}_1{\dot{ϵ}}_1{\dot{g}}_2{\dot{ϵ}}_2(X_1^{\top }{\dot{ϵ}}_1{V}_X^{-1}{X}_2{\dot{\eta }}_2-Z_1{\dot{g}}_2{\dot{ϵ}}_2\hfill \\ & -\{{\dot{g}}_1{\dot{g}}_2{\dot{\eta }}_1{\dot{ϵ}}_2+{\dot{g}}_2{\dot{g}}_1{\dot{\eta }}_2{\dot{ϵ}}_1\}/2\left)\right]\mathbb{E}\left[d_{33}\right]\hfill \\ \hfill & =({\dot{\sigma }}_{ϵ}^2\dot{\varrho }-{\dot{\sigma }}_{ϵ}^2\dot{\varrho }-\{{\dot{\sigma }}_{ϵ}^2\dot{\varrho }+{\dot{\sigma }}_{ϵ}^2\dot{\varrho }\}/2)B_{\beta ,2}\hfill \end{array}$$

(199)

$$\begin{array}{cc}\hfill & =-{\dot{\sigma }}_{ϵ}^2\dot{\varrho }{B}_{\beta ,2}.\hfill \end{array}$$

(200)

There are eight terms remaining, which can be treated two-by-two and then summed together neatly as follows.

$$\begin{array}{cc}\hfill \mathbb{E}\left[d_1{d}_2{d}_{13}{d}_{23}\right]& =\mathbb{E}\left[d_1{d}_2{d}_{23}{d}_{13}\right]\hfill \\ \hfill & =\mathbb{E}[{\dot{g}}_1{\dot{ϵ}}_1{\dot{g}}_2{\dot{ϵ}}_2(X_1^{\top }{\dot{ϵ}}_1{V}_X^{-1}{X}_3{\dot{\eta }}_3-Z_1{\dot{g}}_3{\dot{ϵ}}_3-\{{\dot{g}}_1{\dot{g}}_3{\dot{\eta }}_1{\dot{ϵ}}_3+{\dot{g}}_3{\dot{g}}_1{\dot{\eta }}_3{\dot{ϵ}}_1\}/2)\hfill \\ & (X_2^{\top }{\dot{ϵ}}_2{V}_X^{-1}{X}_3{\dot{\eta }}_3-Z_2{\dot{g}}_3{\dot{ϵ}}_3-\{{\dot{g}}_2{\dot{g}}_3{\dot{\eta }}_2{\dot{ϵ}}_3+{\dot{g}}_3{\dot{g}}_2{\dot{\eta }}_3{\dot{ϵ}}_2\}/2)]\hfill \\ \hfill & =\mathbb{E}[{\dot{ϵ}}_1{\dot{ϵ}}_2({\dot{ϵ}}_1{\dot{g}}_3{\dot{\eta }}_3-{\dot{\eta }}_1{\dot{g}}_3{\dot{ϵ}}_3-\{{\dot{g}}_3{\dot{\eta }}_1{\dot{ϵ}}_3+{\dot{g}}_3{\dot{\eta }}_3{\dot{ϵ}}_1\}/2)\hfill \\ & ({\dot{ϵ}}_2{\dot{g}}_3{\dot{\eta }}_3-{\dot{\eta }}_2{\dot{g}}_3{\dot{ϵ}}_3-\{{\dot{g}}_3{\dot{\eta }}_2{\dot{ϵ}}_3+{\dot{g}}_3{\dot{\eta }}_3{\dot{ϵ}}_2\}/2)]\hfill \\ \hfill & =\mathbb{E}[{\dot{ϵ}}_1{\dot{ϵ}}_2({\dot{ϵ}}_1{\dot{\eta }}_3-{\dot{\eta }}_1{\dot{ϵ}}_3-\{{\dot{\eta }}_1{\dot{ϵ}}_3+{\dot{\eta }}_3{\dot{ϵ}}_1\}/2)\hfill \\ \hfill & ({\dot{ϵ}}_2{\dot{\eta }}_3-{\dot{\eta }}_2{\dot{ϵ}}_3-\{{\dot{\eta }}_2{\dot{ϵ}}_3+{\dot{\eta }}_3{\dot{ϵ}}_2\}/2)]\hfill \\ \hfill & =\mathbb{E}\left[({\dot{\sigma }}_{ϵ}^2{\dot{\eta }}_3-\dot{\varrho }{\dot{ϵ}}_3-\{\dot{\varrho }{\dot{ϵ}}_3+{\dot{\eta }}_3{\dot{\sigma }}_{ϵ}^2\}/2)({\dot{\sigma }}_{ϵ}^2{\dot{\eta }}_3-\dot{\varrho }{\dot{ϵ}}_3-\{\dot{\varrho }{\dot{ϵ}}_3+{\dot{\eta }}_3{\dot{\sigma }}_{ϵ}^2\}/2)\right].\hfill \end{array}$$

(201)

$$\begin{array}{cc}\hfill \mathbb{E}\left[d_1{d}_2{d}_{13}{d}_{32}\right]& =\mathbb{E}\left[d_1{d}_2{d}_{32}{d}_{13}\right]\hfill \\ \hfill & =\mathbb{E}[{\dot{g}}_1{\dot{ϵ}}_1{\dot{g}}_2{\dot{ϵ}}_2(X_1^{\top }{\dot{ϵ}}_1{V}_X^{-1}{X}_3{\dot{\eta }}_3-Z_1{\dot{g}}_3{\dot{ϵ}}_3-\{{\dot{g}}_1{\dot{g}}_3{\dot{\eta }}_1{\dot{ϵ}}_3+{\dot{g}}_3{\dot{g}}_1{\dot{\eta }}_3{\dot{ϵ}}_1\}/2)\hfill \\ & (X_3^{\top }{\dot{ϵ}}_3{V}_X^{-1}{X}_2{\dot{\eta }}_2-Z_3{\dot{g}}_2{\dot{ϵ}}_2-\{{\dot{g}}_3{\dot{g}}_2{\dot{\eta }}_3{\dot{ϵ}}_2+{\dot{g}}_2{\dot{g}}_3{\dot{\eta }}_2{\dot{ϵ}}_3\}/2)]\hfill \\ \hfill & =\mathbb{E}[{\dot{ϵ}}_1{\dot{ϵ}}_2({\dot{ϵ}}_1{\dot{g}}_3{\dot{\eta }}_3-{\dot{\eta }}_1{\dot{g}}_3{\dot{ϵ}}_3-\{{\dot{g}}_3{\dot{\eta }}_1{\dot{ϵ}}_3+{\dot{g}}_3{\dot{\eta }}_3{\dot{ϵ}}_1\}/2)\hfill \\ & ({\dot{ϵ}}_3{\dot{g}}_3{\dot{\eta }}_2-{\dot{\eta }}_3{\dot{g}}_3{\dot{ϵ}}_2-\{{\dot{g}}_3{\dot{\eta }}_3{\dot{ϵ}}_2+{\dot{g}}_3{\dot{\eta }}_2{\dot{ϵ}}_3\}/2)]\hfill \\ \hfill & =\mathbb{E}[{\dot{ϵ}}_1{\dot{ϵ}}_2({\dot{ϵ}}_1{\dot{\eta }}_3-{\dot{\eta }}_1{\dot{ϵ}}_3-\{{\dot{\eta }}_1{\dot{ϵ}}_3+{\dot{\eta }}_3{\dot{ϵ}}_1\}/2)\hfill \\ & ({\dot{ϵ}}_3{\dot{\eta }}_2-{\dot{\eta }}_3{\dot{ϵ}}_2-\{{\dot{\eta }}_3{\dot{ϵ}}_2+{\dot{\eta }}_2{\dot{ϵ}}_3\}/2)]\hfill \\ \hfill & =\mathbb{E}\left[({\dot{\sigma }}_{ϵ}^2{\dot{\eta }}_3-\dot{\varrho }{\dot{ϵ}}_3-\{\dot{\varrho }{\dot{ϵ}}_3+{\dot{\eta }}_3{\dot{\sigma }}_{ϵ}^2\}/2)(\dot{\varrho }{\dot{ϵ}}_3-{\dot{\sigma }}_{ϵ}^2{\dot{\eta }}_3-\{{\dot{\sigma }}_{ϵ}^2{\dot{ϵ}}_3+{\dot{\eta }}_3\dot{\varrho }\}/2)\right].\hfill \end{array}$$

(202)

$$\begin{array}{cc}\hfill \mathbb{E}\left[d_1{d}_2{d}_{31}{d}_{23}\right]& =\mathbb{E}\left[d_1{d}_2{d}_{23}{d}_{31}\right]\hfill \\ \hfill & =\mathbb{E}[{\dot{g}}_1{\dot{ϵ}}_1{\dot{g}}_2{\dot{ϵ}}_2(X_3^{\top }{\dot{ϵ}}_3{V}_X^{-1}{X}_1{\dot{\eta }}_1-Z_3{\dot{g}}_1{\dot{ϵ}}_1-\{{\dot{g}}_3{\dot{g}}_1{\dot{\eta }}_3{\dot{ϵ}}_1+{\dot{g}}_1{\dot{g}}_3{\dot{\eta }}_1{\dot{ϵ}}_3\}/2)\hfill \\ & (X_2^{\top }{\dot{ϵ}}_2{V}_X^{-1}{X}_3{\dot{\eta }}_3-Z_2{\dot{g}}_3{\dot{ϵ}}_3-\{{\dot{g}}_2{\dot{g}}_3{\dot{\eta }}_2{\dot{ϵ}}_3+{\dot{g}}_3{\dot{g}}_2{\dot{\eta }}_3{\dot{ϵ}}_2\}/2)]\hfill \\ \hfill & =\mathbb{E}[{\dot{ϵ}}_1{\dot{ϵ}}_2({\dot{ϵ}}_3{\dot{g}}_3{\dot{\eta }}_1-{\dot{\eta }}_3{\dot{g}}_3{\dot{ϵ}}_1-\{{\dot{g}}_3{\dot{\eta }}_3{\dot{ϵ}}_1+{\dot{g}}_3{\dot{\eta }}_1{\dot{ϵ}}_3\}/2)\hfill \\ & ({\dot{ϵ}}_2{\dot{g}}_3{\dot{\eta }}_3-{\dot{\eta }}_2{\dot{g}}_3{\dot{ϵ}}_3-\{{\dot{g}}_3{\dot{\eta }}_2{\dot{ϵ}}_3+{\dot{g}}_3{\dot{\eta }}_3{\dot{ϵ}}_2\}/2)]\hfill \\ \hfill & =\mathbb{E}[{\dot{ϵ}}_1{\dot{ϵ}}_2({\dot{ϵ}}_3{\dot{\eta }}_1-{\dot{\eta }}_3{\dot{ϵ}}_1-\{{\dot{\eta }}_3{\dot{ϵ}}_1+{\dot{\eta }}_1{\dot{ϵ}}_3\}/2)\hfill \\ & ({\dot{ϵ}}_2{\dot{\eta }}_3-{\dot{\eta }}_2{\dot{ϵ}}_3-\{{\dot{\eta }}_2{\dot{ϵ}}_3+{\dot{\eta }}_3{\dot{ϵ}}_2\}/2)]\hfill \\ \hfill & =\mathbb{E}\left[(\dot{\varrho }{\dot{ϵ}}_3-{\dot{\sigma }}_{ϵ}^2{\dot{\eta }}_3-\{{\dot{\sigma }}_{ϵ}^2{\dot{\eta }}_3+{\dot{ϵ}}_3\dot{\varrho }\}/2)({\dot{\sigma }}_{ϵ}^2{\dot{\eta }}_3-\dot{\varrho }{\dot{ϵ}}_3-\{\dot{\varrho }{\dot{ϵ}}_3+{\dot{\eta }}_3{\dot{\sigma }}_{ϵ}^2\}/2)\right].\hfill \end{array}$$

(203)

$$\begin{array}{cc}\hfill \mathbb{E}\left[d_1{d}_2{d}_{31}{d}_{32}\right]& =\mathbb{E}\left[d_1{d}_2{d}_{32}{d}_{31}\right]\hfill \\ \hfill & =\mathbb{E}[{\dot{g}}_1{\dot{ϵ}}_1{\dot{g}}_2{\dot{ϵ}}_2(X_3^{\top }{\dot{ϵ}}_3{V}_X^{-1}{X}_1{\dot{\eta }}_1-Z_3{\dot{g}}_1{\dot{ϵ}}_1-\{{\dot{g}}_3{\dot{g}}_1{\dot{\eta }}_3{\dot{ϵ}}_1+{\dot{g}}_1{\dot{g}}_3{\dot{\eta }}_1{\dot{ϵ}}_3\}/2)\hfill \\ & (X_3^{\top }{\dot{ϵ}}_3{V}_X^{-1}{X}_2{\dot{\eta }}_2-Z_3{\dot{g}}_2{\dot{ϵ}}_2-\{{\dot{g}}_3{\dot{g}}_2{\dot{\eta }}_3{\dot{ϵ}}_2+{\dot{g}}_2{\dot{g}}_3{\dot{\eta }}_2{\dot{ϵ}}_3\}/2)]\hfill \\ \hfill & =\mathbb{E}[{\dot{ϵ}}_1{\dot{ϵ}}_2({\dot{ϵ}}_3{\dot{g}}_3{\dot{\eta }}_1-{\dot{\eta }}_3{\dot{g}}_3{\dot{ϵ}}_1-\{{\dot{g}}_3{\dot{\eta }}_3{\dot{ϵ}}_1+{\dot{g}}_3{\dot{\eta }}_1{\dot{ϵ}}_3\}/2)\hfill \\ & ({\dot{ϵ}}_3{\dot{g}}_2{\dot{\eta }}_2-{\dot{\eta }}_3{\dot{g}}_2{\dot{ϵ}}_2-\{{\dot{g}}_2{\dot{\eta }}_3{\dot{ϵ}}_2+{\dot{g}}_2{\dot{\eta }}_2{\dot{ϵ}}_3\}/2)]\hfill \\ \hfill & =\mathbb{E}[{\dot{ϵ}}_1{\dot{ϵ}}_2({\dot{ϵ}}_3{\dot{\eta }}_1-{\dot{\eta }}_3{\dot{ϵ}}_1-\{{\dot{\eta }}_3{\dot{ϵ}}_1+{\dot{\eta }}_1{\dot{ϵ}}_3\}/2)\hfill \\ & ({\dot{ϵ}}_3{\dot{\eta }}_2-{\dot{\eta }}_3{\dot{ϵ}}_2-\{{\dot{\eta }}_3{\dot{ϵ}}_2+{\dot{\eta }}_2{\dot{ϵ}}_3\}/2)]\hfill \\ \hfill & =\mathbb{E}\left[(\dot{\varrho }{\dot{ϵ}}_3-{\dot{\sigma }}_{ϵ}^2{\dot{\eta }}_3-\{{\dot{\sigma }}_{ϵ}^2{\dot{\eta }}_3+{\dot{ϵ}}_3\dot{\varrho }\}/2)(\dot{\varrho }{\dot{ϵ}}_3-{\dot{\sigma }}_{ϵ}^2{\eta }_3-\{{\dot{\sigma }}_{ϵ}^2{\eta }_3+{\dot{ϵ}}_3\dot{\varrho }\}/2)\right].\hfill \end{array}$$

(204)

Temporarily, put

$$\begin{array}{cc}\hfill C_1& =\mathbb{E}\left[d_1{d}_2{d}_{13}{d}_{32}\right]+\mathbb{E}\left[d_1{d}_2{d}_{13}{d}_{32}\right],\hfill \\ \hfill C_2& =\mathbb{E}\left[d_1{d}_2{d}_{31}{d}_{23}\right]+\mathbb{E}\left[d_1{d}_2{d}_{31}{d}_{32}\right],\hfill \\ \hfill C_3& =\mathbb{E}\left[d_1{d}_2{d}_{23}{d}_{13}\right]+\mathbb{E}\left[d_1{d}_2{d}_{32}{d}_{13}\right],\hfill \\ \hfill C_4& =\mathbb{E}\left[d_1{d}_2{d}_{23}{d}_{31}\right]+\mathbb{E}\left[d_1{d}_2{d}_{32}{d}_{31}\right].\hfill \end{array}$$

(205)

We see that

$$\begin{array}{cc}\hfill C_1& =-\mathbb{E}\left[({\dot{\sigma }}_{ϵ}^2{\dot{\eta }}_3-\dot{\varrho }{\dot{ϵ}}_3-\{\dot{\varrho }{\dot{ϵ}}_3+{\dot{\eta }}_3{\dot{\sigma }}_{ϵ}^2\}/2)({\dot{\sigma }}_{ϵ}^2{\dot{ϵ}}_3+{\dot{\eta }}_3\dot{\varrho })\right]\hfill \\ \hfill C_2& =-\mathbb{E}\left[(\dot{\varrho }{\dot{ϵ}}_3-{\dot{\sigma }}_{ϵ}^2{\dot{\eta }}_3-\{{\dot{\sigma }}_{ϵ}^2{\dot{\eta }}_3+{\dot{ϵ}}_3\dot{\varrho }\}/2)({\dot{\sigma }}_{ϵ}^2{\eta }_3+{\dot{ϵ}}_3\dot{\varrho })\right],\hfill \end{array}$$

so that

$$\begin{array}{cc}\hfill C_1+C_2& =\mathbb{E}\left[({\dot{\sigma }}_{ϵ}^2{\dot{\eta }}_3+{\dot{ϵ}}_3\dot{\varrho })({\dot{\sigma }}_{ϵ}^2{\eta }_3+{\dot{ϵ}}_3\dot{\varrho })\right]\hfill \\ \hfill & =\mathbb{E}\left[{({\dot{\sigma }}_{ϵ}^2{\dot{\eta }}_3+{\dot{ϵ}}_3\dot{\varrho })}^2\right]\hfill \\ \hfill & ={\left({\dot{\sigma }}_{ϵ}^2\right)}^2{\dot{\sigma }}_{\eta }^2+2{\dot{\sigma }}_{ϵ}^2{\dot{\varrho }}^2+{\dot{\sigma }}_{ϵ}^2{\dot{\varrho }}^2\hfill \\ \hfill & ={\dot{\sigma }}_{ϵ}^2\left({\dot{\sigma }}_{ϵ}^2\right){\dot{\sigma }}_{\eta }^2+3{\dot{\varrho }}^2).\hfill \end{array}$$

(206)

Noting the equalities between $C_1$ and $C_3$ and $C_2$ and $C_4$, the last eight terms in $K_{\beta ,4}$ can be written as

$$C_1+C_2+C_3+C_4=2{\dot{\sigma }}_{ϵ}^2({\dot{\sigma }}_{ϵ}^2{\dot{\sigma }}_{\eta }^2+3{\dot{\varrho }}^2)$$

(207)

and we have

$$\begin{array}{cc}\hfill K_{\beta ,4}& =V_{\beta ,1}{V}_{\beta ,23}-4{\dot{\sigma }}_{ϵ}\dot{\varrho }{B}_{\beta ,2}+2{\dot{\sigma }}_{ϵ}^2({\dot{\sigma }}_{ϵ}^2{\dot{\sigma }}_{\eta }^2+3{\dot{\varrho }}^2)\hfill \\ \hfill & =V_{\beta ,1}(V_{\beta ,23}-4\dot{\varrho }{B}_{\beta ,2}+2{\dot{\sigma }}_{ϵ}^2{\dot{\sigma }}_{\eta }^2+6{\dot{\varrho }}^2).\hfill \end{array}$$

(208)