On a Robust MaxEnt Process Regression Model with Sample-Selection

Abstract

In a regression analysis, a sample-selection bias arises when a dependent variable is partially observed as a result of the sample selection. This study introduces a Maximum Entropy (MaxEnt) process regression model that assumes a MaxEnt prior distribution for its nonparametric regression function and finds that the MaxEnt process regression model includes the well-known Gaussian process regression (GPR) model as a special case. Then, this special MaxEnt process regression model, i.e., the GPR model, is generalized to obtain a robust sample-selection Gaussian process regression (RSGPR) model that deals with non-normal data in the sample selection. Various properties of the RSGPR model are established, including the stochastic representation, distributional hierarchy, and magnitude of the sample-selection bias. These properties are used in the paper to develop a hierarchical Bayesian methodology to estimate the model. This involves a simple and computationally feasible Markov chain Monte Carlo algorithm that avoids analytical or numerical derivatives of the log-likelihood function of the model. The performance of the RSGPR model in terms of the sample-selection bias correction, robustness to non-normality, and prediction, is demonstrated through results in simulations that attest to its good finite-sample performance.

1. Introduction

The Bayesian nonparametric method is a powerful approach for regression problems when the shape of the underlying regression function is unknown, the function may be difficult to evaluate analytically, or other requirements such as design costs may complicate the process of information acquisition process. Bayesian orthogonal basis expansion regression, spline smoothing regression, wavelet regression, and Gaussian process regression (GPR) are powerful nonparametric Bayesian approaches that address these regression problems. These regression techniques have been extensively used in fields such as psychology, data science, engineering, neuroscience, and fishery [1,2,3,4,5].

Sample selection (or incidental truncation) in regression analysis is known to often arise in a wide variety of practical problems and standard analysis of data with sample selection leads to biased results because the selected sample represents only a subset of a full population; see [6,7,8]. Regression analysis also has problems regarding sensitivity to outliers and departures from the normality of the dependent variable (see [9,10]). Thus, when one implements nonparametric Bayesian regression with non-normal data with sample selection, the selection mechanism and non-normality of the data must be jointly modeled with the Bayesian nonparametric regression model to correct the sample-selection bias and to implement a robust statistical inference. In this regard, several estimation procedures have been considered in the literature to produce robust linear regression models that are subject to sample selection including for instance [6,7], for frequentist methods, and [8,9,11] for Bayesian methods. See [9,12] to obtain robust Bayesian sample-selection models other than the regression model. In addition, no studies have generalized a nonparametric regression model to deal with non-normal data with the sample selection.

The objective of this paper is to introduce the Maximum Entropy (MaxEnt) process regression model as a new Bayesian nonparametric regression model and to then generalize this model to propose a robust sample-selection Bayesian nonparametric regression model along with its inferential methodology. The MaxEnt process regression model is obtained by assuming a MaxEnt prior distribution for its nonparametric regression function, and it includes the GPR model as a special case. This provides a relationship between the MaxEnt nonparametric regression approach and the rationale to conduct a Gaussian regression analysis. This study focuses on the GPR model as a special MaxEnt process regression model and a powerful analysis model towards nonparametric regression problems. Then, the GPR model is generalized to obtain a robust sample-selection Gaussian process regression (RSGPR) model. This RSGPR model extends the GPR model to account for the sample selection scheme, and it is robust when the data are heavy-tailed or contain outliers.

The RSGPR model consists of two components. The first is a robust GPR model that determines the level of the dependent variable of interest and the second is an equation that describes the selection mechanism that determines whether we have observed the dependent variable or not. The sample-selection bias arises when these two components are correlated and must be modeled jointly. A Bayesian hierarchical methodology is developed here to estimate the RSGPR model. This methodology relies on a stochastic representation technique (see, e.g., [13]) to set up the Bayesian hierarchy of the RSGPR model, and it has three attractive features. First, given the likelihood function of the model, the posterior of its parameters does not belong to any well-known parametric family, but the methodology uses a simple Markov chain Monte Carlo (MCMC) algorithm that does not resort to generating random draws from the complex posterior. Second, the output of the algorithm not only provides a Bayesian analogue of confidence intervals for the regression function, but it also readily gives an indication of the presence (or absence) of the sample-selection bias. Third, if there is prior information, such as restrictions on the regression function, such information can be incorporated easily through a prior distribution.

The remainder of this study is organized as follows. Section 2 introduces the MaxEnt process regression model that strictly includes the GPR model. Then, this section formulates the RSGPR model that is obtained by incorporating the GPR model with a class of scale mixtures of normal errors as well as a selection model comprising a class of scale mixtures of the probit sample selection equations. Properties of this RSGPR model are studied including the exact distribution of a selected observation, a stochastic representation, a distributional hierarchy, and the magnitude of the sample-selection bias. In Section 3, we construct a Bayesian hierarchical model for inference in the RSGPR model by exploiting the stochastic representation and distributional hierarchy. Then we develop a Bayesian estimation methodology based on the hierarchical model to provide a simple estimation procedure for the RSGPR model. We further construct a computationally feasible MCMC algorithm through a Bayesian hierarchical approach. Section 4 examines the finite-sample performance of the method through a limited but informative simulation. This numerical illustration shows the usefulness of the RSGPR model for the Gaussian process regression analysis of non-normal data with the sample selection. The study then concludes with a discussion in Section 5. Proofs and additional details are provided in the Appendix A.

2. Robust Sample-Selection GPR Model

2.1. MaxEnt Process Regression Model

Consider the following nonparametric regression model,

$${\mathit{y}}_n={\mathit{\eta }}_n\left(\mathbf{x}\right)+{\mathit{ϵ}}_n,$$

(1)

where ${\mathit{y}}_n={(y_1,\dots ,y_n)}^{\top }$ is $n\times 1$ vector of responses, $y_i=\eta \left({\mathit{x}}_i\right)+{ϵ}_i,$ ${\mathit{\eta }}_n\left(\mathbf{x}\right)={(\eta \left({\mathit{x}}_1\right),\dots ,\eta \left({\mathit{x}}_n\right))}^{\top }$ is $n\times 1$ vector of regression function values satisfying $\eta \left({\mathit{x}}_i\right)=E\left[y_i\right|{\mathit{x}}_i],$ $i=1,\dots ,n$, $\mathbf{x}={({\mathit{x}}_1,\dots ,{\mathit{x}}_n)}^{\top }$ is the $n\times p$ design matrix, and ${\mathit{ϵ}}_n={({ϵ}_1,\dots ,{ϵ}_n)}^{\top }$ is a $n\times 1$ vector of i.i.d. random noises with zero mean vector. In the basic model structure of (1), the parametric form of the regression function ${\mathit{\eta }}_n\left(\mathbf{x}\right)$ is not assumed, but ${\mathit{\eta }}_n\left(\mathbf{x}\right)$ is assumed to have specific types of functional structure. For example, ${\mathit{\eta }}_n\left(\mathbf{x}\right)$ can be represented with a Fourier series [14], splines [15], kernels [16] and others.

In the Bayesian nonparametric regression, we assume that the regression function (or signal term) ${\mathit{\eta }}_n\left(\mathbf{x}\right)$ is a random function that follows a particular distribution. This distribution is subjective in the sense that the distribution reflects our uncertain prior information regarding the function. Sometimes, we have a situation in which partial prior information on ${\mathit{\eta }}_n\left(\mathbf{x}\right)$ is available, outside of which it is desirable to use a prior that is as non-informative as possible. In this situation, Boltzmann’s maximum entropy theorem (see, e.g., [17]) yields a maximum entropy prior ${\pi }_{max}\left({\mathit{\eta }}_n\left(\mathbf{x}\right)\right)$ that is an exponential form and maximizes the entropy,

$$\mathrm{H}\left(\pi \right)=-{\int }_{{\mathrm{R}}^n}\pi \left({\mathit{\eta }}_n\left(\mathbf{x}\right)\right)log\pi \left({\mathit{\eta }}_n\left(\mathbf{x}\right)\right)\mathrm{d}{\mathit{\eta }}_n\left(\mathbf{x}\right),$$

in the presence of partial information for various moment functions of ${\mathit{\eta }}_n\left(\mathbf{x}\right).$ In a special case where we only have partial prior information about the mean vector and covariance matrix functions of ${\mathit{\eta }}_n\left(\mathbf{x}\right)$ of the Bayesian nonparametric regression model (1), Boltzmann’s maximum entropy theorem yields the following prior distribution of ${\mathit{\eta }}_n\left(\mathbf{x}\right)$.

Lemma 1.

Let $n\times 1$ regression function vector ${\mathit{\eta }}_n\left(\mathit{x}\right)$ have a prior distribution on ${\mathrm{R}}^n$ whose partial information on the mean and covariance functions are $m\left(\mathit{x}\right)={(m\left({\mathit{x}}_1\right),\dots ,m\left({\mathit{x}}_n\right))}^{\top }$ and $K\left(\mathit{x}\right)=\left\{\kappa ({\mathit{x}}_i,{\mathit{x}}_j)\right\},$ respectively. Then the maximum entropy prior of ${\mathit{\eta }}_n\left(\mathit{x}\right)$ is

$${\pi }_{max}\left({\mathit{\eta }}_n\left(\mathit{x}\right)\right)={\left(2\pi \right)}^{-n/2}{\left|K\left(\mathit{x}\right)\right|}^{-1/2}exp\{-\frac{1}{2}{({\mathit{\eta }}_n\left(\mathit{x}\right)-m\left(\mathit{x}\right))}^{\top }K{\left(\mathit{x}\right)}^{-1}({\mathit{\eta }}_n\left(\mathit{x}\right)-m\left(\mathit{x}\right))\}$$

(2)

for ${\mathit{\eta }}_n\left(\mathit{x}\right)\in {\mathrm{R}}^n.$ This is a density of $\mathcal{G}\mathcal{P}\left(m\right(\mathit{x}),K(\mathit{x}\left)\right),$ a Gaussian process defined by the mean function $m\left(\mathit{x}\right)$ and the covariance function $K\left(\mathit{x}\right)$.

Note that the Gaussian process $\mathcal{G}\mathcal{P}\left(m\left(\mathbf{x}\right),K\left(\mathbf{x}\right)\right)$ defines a collection of random functions wherein any finite subset of the process has multivariate normal (Gaussian) distribution. From now on, we will write the Gaussian process as ${\mathit{\eta }}_n\left(\mathbf{x}\right)\sim \mathcal{G}\mathcal{P}\left(m\left(\mathbf{x}\right),K\left(\mathbf{x}\right)\right).$ The only restriction on the Gaussian process is that the covariance function $K\left(\mathbf{x}\right)$ must be an $n\times n$ positive definite symmetric (pds) matrix. If $K\left(\mathbf{x}\right)$ is not a pds matrix, then the corresponding value of $\mathrm{H}\left({\pi }_{max}\right)=p(1+log\left(2\pi \right))/2+log\left(\right|K\left(\mathbf{x}\right)\left|\right)/2$ will not be defined (see, e.g., [18]). As a result, this paper introduces yet another Bayesian nonparametric regression model by combining the regression model (1) and the MaxEnt prior in Lemma 1 and introducing a normal regression error distribution. The model is named as a “MaxEnt process regression model” and defined by

$$\begin{array}{ccc}\hfill {\mathit{y}}_n& =& {\mathit{\eta }}_n\left(\mathbf{x}\right)+{\mathit{ϵ}}_n,{\mathit{ϵ}}_n\sim {\mathcal{N}}_n(\mathbf{0},{\sigma }^2{\mathrm{I}}_n),\hfill \\ \hfill {\sigma }^2& \sim & \mathcal{I}\mathcal{G}({\nu }_1,{\nu }_2),\hfill \\ \hfill {\mathit{\eta }}_n\left(\mathbf{x}\right)& \sim & \mathcal{G}\mathcal{P}\left(m\left(\mathbf{x}\right),K\left(\mathbf{x}\right)\right),\hfill \end{array}$$

(3)

where ${\mathcal{N}}_n(\mathbf{0},{\sigma }^2{\mathrm{I}}_n)$ is an n-variate normal distribution with mean vector $\mathbf{0}$ and covariance matrix ${\sigma }^2{\mathrm{I}}_n,$ $\mathcal{I}\mathcal{G}({\nu }_1,{\nu }_2)$ denotes an inverse gamma distribution with shape parameter ${\nu }_1$ and scale parameter ${\nu }_2,$ ${\mathit{\eta }}_n\left(\mathbf{x}\right)$ is independent of ${\sigma }^2$ and ${\mathit{ϵ}}_n,$ the mean function $m\left({\mathit{x}}_i\right)$ reflects the expected function value at input ${\mathit{x}}_i$, i.e., $m\left({\mathit{x}}_i\right)=\mathrm{E}\left[\eta \left({\mathit{x}}_i\right)\right],$ and the covariance function $\kappa ({\mathit{x}}_i,{\mathit{x}}_j)$ models the dependence between the function values at different input points ${\mathit{x}}_i$ and ${\mathit{x}}_j,$ i.e.,$\kappa ({\mathit{x}}_i,{\mathit{x}}_j)=\mathrm{E}\left[(\eta \left({\mathit{x}}_i\right)-m\left({\mathit{x}}_i\right))(\eta \left({\mathit{x}}_j\right)-m\left({\mathit{x}}_j\right))\right],$ $i,j=1,\dots ,n.$ See [19] for choice of an appropriate covariance function based on assumptions such as smoothness and likely patterns to be expected in the data. A commonly used isotropic covariance function in practice is the squared exponential covariance function given by

$$\kappa ({\mathit{x}}_i,{\mathit{x}}_j)=Cov(\eta \left({\mathit{x}}_i\right),\eta \left({\mathit{x}}_j\right))=u_{0}exp\left\{-\frac{w_0}{2}{({\mathit{x}}_i-{\mathit{x}}_j)}^{\top }({\mathit{x}}_i-{\mathit{x}}_j)\right\},$$

(4)

where $u_0$ and $w_0$ are hyperparameters and which are relevant for the shape of MaxEnt process regression. Here $u_0$ stands for global scale of the covariance matrix $K\left(\mathbf{x}\right)$ and $w_0$ stands for smoothing parameter, respectively.

We can easily see that the MaxEnt process regression model (5) is the same as the GPR model considered by [19,20]. This proves the following corollary and can hence be used as an information theoretic justification for using the GPR model as a Bayesian approach for a nonparametric regression analysis.

Corollary 1.

Suppose $\mathrm{E}\left[{\mathit{\eta }}_n\left(\mathit{x}\right)\right]=m\left(\mathit{x}\right)$ and $\mathrm{Cov}\left({\mathit{\eta }}_n\left(\mathit{x}\right)\right)=K\left(\mathit{x}\right)$ are all the prior information on ${\mathit{\eta }}_n\left(\mathit{x}\right)$ for the Bayesian nonparametric regression model (1). Then MaxEnt prior distribution of ${\mathit{\eta }}_n\left(\mathit{x}\right)$ is $\mathcal{G}\mathcal{P}\left(m\left(\mathit{x}\right),K\left(\mathit{x}\right)\right)$ which defines the GPR model.

According to Corollary 1, we shall denote the MaxEnt process regression model by the GPR model. When there is no functional constraint in the GPR model, then the prior specification in the model (3) can be used, and posterior inference can be performed without difficulty. It is seen, from [20], that the conditional posterior distribution of ${\mathit{\eta }}_n\left(\mathbf{x}\right)$ is normal with the mean and covariance given by

$$\begin{array}{ccc}\hfill \mathrm{E}[{\mathit{\eta }}_n(\mathbf{x})|({\mathit{y}}_n,\mathbf{x}),{\sigma }^2]& =& K(\mathbf{x}){(K(\mathbf{x})+{\sigma }^2{\mathrm{I}}_n)}^{-1}{\mathit{y}}_n+{\sigma }^2{\mathrm{I}}_n{(K(\mathbf{x})+{\sigma }^2{\mathrm{I}}_n)}^{-1}m(\mathbf{x}),\hfill \\ \hfill \mathrm{Var}[{\mathit{\eta }}_n\left(\mathbf{x}\right)|({\mathit{y}}_n,\mathbf{x}),{\sigma }^2]& =& {(K{\left(\mathbf{x}\right)}^{-1}+{\sigma }^{-2}{\mathrm{I}}_n)}^{-1}={\sigma }^{2}K\left(\mathbf{x}\right){(K\left(\mathbf{x}\right)+{\sigma }^2{\mathrm{I}}_n)}^{-1}.\hfill \end{array}$$

(5)

However, in the GPR analysis, a sample-selection scheme often applies to the response variable that results in missing not at random (MNAR) observations on the variable. In this case, the regression analysis using only the selected cases will lead to biased results (see, e.g., [6,7,8]). This study provides a bias correction procedure for the GPR analysis with MNAR data generated from the sample-selection scheme. For the analysis, we propose a robust sample-selection GPR (RSGPR) model based on a family of scale mixtures of normal (SMN) distributions (see [21,22] for details). This approach reflects the MNAR mechanism as well as its robustness against departures from the normality assumption (see, e.g., [11,12]), and proposes a robust GPR model to analyze the partially observed sample-selection data.

2.2. Proposed Model

We propose the RSGPR model through the following steps. First, we modify the GPR model (3) by incorporating the SMN error distribution for a robust GPR analysis. Then, we connect the robust GPR model directly to a sample-selection model by introducing some latent variables to explain the partially observed sample-selection data. To model the sample-selection mechanism, we need to introduce some notation for the partially observed data.

Let $s_i$ be a binary variable that takes on value 1 if $y_i$ of subject i is observed using the sample-selection scheme, and 0 if that of the subject is not observed using the same scheme. Then, we introduce the following RSGPR model to represent the regression equation of the observable variable $y_i:$

$$\begin{array}{ccc}\hfill y_i& =& \left\{\begin{array}{cc}\eta \left({\mathit{x}}_i\right)+{ϵ}_i\hfill & \mathrm{for}{s}_i=\mathrm{I}(z_i\ge 0),\hfill \\ \mathrm{missing}\hfill & \mathrm{for}{s}_i=\mathrm{I}(z_i<0),\hfill \end{array}\right.\hfill \\ \hfill z_i& =& {\mathit{v}}_i^{\top }\mathit{\gamma }+{\epsilon }_i,i=1,\dots ,n,\left(\begin{array}{c}{ϵ}_i\\ {\epsilon }_i\end{array}\right)\stackrel{iid}{\sim }{\mathcal{S}\mathcal{M}\mathcal{N}}_2(\mathbf{0},\Sigma ,\delta ,G),\hfill \\ \hfill {\mathit{\eta }}_n\left(\mathbf{x}\right)& \sim & \mathcal{G}\mathcal{P}\left(m\left(\mathbf{x}\right),K\left(\mathbf{x}\right)\right),\hfill \end{array}$$

(6)

where $\mathrm{I}(·)$ is an indicator function, ${\mathcal{SMN}}_2(\mathbf{0},\Sigma ,\delta ,G),$ a scale mixture of bivariate normal distributions with mixture function $\delta \left(\omega \right)$ and mixing variable $\omega \sim G.$ Here $\mathit{\gamma }={({\gamma }_1,\dots ,{\gamma }_q)}^{\top }$ and $\Sigma =\left(\begin{array}{cc}{\sigma }^2& \rho \sigma \\ \rho \sigma & 1\end{array}\right)$ are parameters to be elicited by using the priors $p_0\left(\mathit{\gamma }\right)$ and $p_0(\Sigma )$.

Without loss of generality, we assume that the single sample selection scheme $s_i=I(z_i\ge 0)$ is applied to a random sample of n observations ($y_i$’s) associated with the model (1) and gives only the first $n_1$ observed values of $y_i$’s out of the n$(n>n_1)$ observations according to the sample-selection scheme. Thus, the overall available data information of the RSGPR model consists of the set of $s_i$ binary values and the $n_1$-tuples of observations $(y_i,{\mathit{v}}_i)$ corresponding to individuals with $s_i=1,$ while ${\mathit{v}}_i$ for those with $s_i=0.$ The purpose of this study is to estimate the regression function ${\mathit{\eta }}_n\left(\mathbf{x}\right)$ based on partially observed data (i.e., sample-selection data) with size $n_1.$

For fixed $\eta \left({\mathit{x}}_i\right)$, the density of the RSGPR model (6) is composed of a continuous component $h\left(y_i\right|s_i=1)$ and a discrete component $p\left(s_i\right).$ The discrete component is

$$\begin{array}{ccc}\hfill p\left(s_i\right)& =& {\left[\overline{F}(C_i;0,1)\right]}^{s_i}{\left[1-\overline{F}(C_i;0,1)\right]}^{1-s_i},\hfill \end{array}$$

(7)

where $\overline{F}(C_i;d,\tau )={\int }_0^{\infty }\overline{\Phi }(C_i;d,\delta \left(\omega \right)\tau )dG\left(\omega \right)$ with a selection interval $C_i=({\alpha }_i,\infty ),$ ${\alpha }_i=-{\mathit{v}}_i^{\top }\gamma ,$ and $\overline{\Phi }(C_i;d,\kappa \left(\omega \right)\tau )={\int }_{C_i}\varphi (x;d,\delta \left(\omega \right)\tau )dx$ denotes the probability of the interval $C_i$ under the $\mathcal{N}(d,\delta (\omega \left)\tau \right)$ distribution with the density $\varphi (x;d,\delta (\omega \left)\tau \right).$ The continuous component is a density of $\left[y_i\right|\eta \left({\mathit{x}}_i\right),s_i=1]\stackrel{d}{=}\left[y_i\right|\eta \left({\mathit{x}}_i\right),{\epsilon }_i\in C_i]$ for $i=1,\dots ,n_1.$ This density is given by

$$\begin{array}{ccc}\hfill h\left(y_i\right|s_i=1)& =& \frac{{\int }_0^{\infty }\varphi (y_i;\eta ({\mathit{x}}_i),\delta (\omega ){\sigma }^2)\overline{\Phi }\left(C_i;{\theta }_{{\epsilon }_i|y_i},\delta (\omega ){\sigma }_{{\epsilon }_i|y_i}^2\right)dG(\omega )}{\overline{F}(C_i;0,1)},y_i\in R,\hfill \end{array}$$

(8)

where ${\theta }_{{\epsilon }_i|y_i}=\rho [y_i-\eta \left({\mathit{x}}_i\right)]/\sigma ,$ and ${\sigma }_{{\epsilon }_i|y_i}^2=1-{\rho }^2.$ This distribution is essentially a member of the class of skew-scale mixtures of normal (skew-SMN) distributions discussed by [13,23,24]. We will denote the distribution law of $\left[y_i\right|\eta \left({\mathit{x}}_i\right),s_i=1]$ with density (8) by skew-$\mathcal{SMN}(C_i;{\mathit{\theta }}_i,\Sigma ,\delta ,G),$ where ${\mathit{\theta }}_i={(\eta \left({\mathit{x}}_i\right),0)}^{\top }.$ The following lemma is useful to generate the partially observed $y_i$’s and indicate the difference between the RSGPR model (6) and the GPR model (3).

Lemma 2.

For a given value of $\eta \left({\mathit{x}}_i\right),$ the selected observation $\left[y_i\right|\eta \left({\mathit{x}}_i\right),s_i=1]$ for the RSGPR model can be represented by the following two-stages of distributional hierarchy:

$$\begin{array}{ccc}\hfill [y_i|\omega ,\eta ({\mathit{x}}_i),s_i=1]& \stackrel{d}{=}& \eta ({\mathit{x}}_i)+\rho \sigma Z_{C_i}+\sigma {(1-{\rho }^2)}^{1/2}{U}_i,i=1,\dots ,n_1,\hfill \\ \hfill \omega & \sim & G(\omega ),\hfill \end{array}$$

(9)

where $U_i\stackrel{iid}{\sim }\mathcal{N}(0,\delta \left(\omega \right))$ and $Z_{C_i}\stackrel{ind}{\sim }{\mathcal{TN}}_{C_i}(0,\delta \left(\omega \right))$ are independent conditionally on $\omega ,$ and ${\mathcal{TN}}_{C_i}(0,\delta \left(\omega \right))$ denotes a $\mathcal{N}(0,\delta (\omega \left)\right)$ distribution truncated to the interval $C_i=({\alpha }_i,\infty ).$

Lemma 2 shows that the RSGPR model applies to relax the classic assumption of the underlying normality as well as to reflect the sample-selection scheme. This lemma also indicates that the partially observed data $y_i$’s does not represent a random sample from the GPR model generating $y_i$’s, even after controlling for the regression function $\eta \left({\mathit{x}}_i\right).$ If we want to apply a GPR analysis to the partially observed sample-selection data, a fitted model should be the RSGPR model. The RSGPR model changes depending on the choice of the distribution of $\omega$ and its function $\delta \left(\omega \right).$ In the special case wherein the distribution of $\omega$ degenerates at $\delta \left(\omega \right)=1,$ the RSGPR model produces a sample-selection Gaussian process normal error regression (SGPR${}_N$) model. When we choose $\omega \sim \mathcal{G}(\nu /2,\nu /2),$ a gamma distribution with mean 1 and $\delta \left({\omega }_i\right)=1/{\omega }_i,$ the model becomes a sample-selection Gaussian process $t_{\nu }$ error regression (SGPR${}_{t_{\nu }}$) model, allowing to regulate the tail distribution of the model by means of the degrees of freedom. We also see that the RSGPR model strictly includes the GPR model because the latter is obtained by setting $\rho =0.$ For the remainder of this study, we use the symbols in the preceding sections with the same definitions.

2.3. The Sample-Selection Bias

As indicated by the density (8) and Lemma 2 the selected observations $\left[y_i\right|s_i=1]$’s do not represent a random sample from the GPR model generating $y_i$’s, but they are missing not at random (MNAR) [25] inducing a sample-selection bias. The following results on the sample-selection bias are noted in the Bayesian estimation of the GPR model with the partially observed data.

Lemma 3.

Given the RSGPR model (6), a stochastic representation of conditional posterior distribution of the regression function ${\mathit{\eta }}_{n_1}={(\eta ({\mathit{x}}_1),\dots ,\eta ({\mathit{x}}_{n_1}))}^{\top }$ is

$$\begin{array}{ccc}\hfill [{\mathit{\eta }}_{n_1}|\mathit{y},\omega ,\Psi ]& \stackrel{d}{=}& {\mathit{\theta }}_1+\Gamma {\Omega }_2^{-1}{\mathit{W}}_1^{{\mathit{C}}_{\beta }}+{\mathit{W}}_2,\hfill \end{array}$$

(10)

where $\Psi =\{{\sigma }^2,\rho ,\gamma \},$ ${\mathit{W}}_1\sim {\mathcal{N}}_{n_1}(\mathit{0},{\Omega }_2)$ and ${\mathit{W}}_2\sim {\mathcal{N}}_{n_1}(\mathit{0},{\Omega }_1-\Gamma {\Omega }_2^{-1}{\Gamma }^{\top })$ are independent random vectors, ${\mathit{W}}_1^{{\mathit{C}}_{\beta }}\stackrel{d}{=}[{\mathit{W}}_1|{\mathit{W}}_1\in {\mathit{C}}_{\beta }],$ ${\mathit{C}}_{\beta }={\cap }_{i=1}^{n_1}\{w_{1i};{\beta }_i\le w_{1i}\le \infty \},$ ${\mathit{W}}_1={(w_{11},\dots ,w_{1n_1})}^{\top },$ ${\beta }_i=({\alpha }_i-{\theta }_{2i})/\sqrt{\delta (\omega )},$ ${\mathit{\theta }}_1=K_{11}{(\mathit{x})}^{-1}{H}^{-1}{\mathit{y}}_{n_1}+\delta (\omega ){\sigma }^2{H}^{-1}{m}_1(\mathit{x}),$ ${\mathit{\theta }}_2={({\theta }_{21},\dots ,{\theta }_{2n_1})}^{\top }=\rho ({\mathit{y}}_{n_1}-m_1(\mathit{x}))/\sigma ,$ $\Gamma =-\rho {\Omega }_1/\sigma ,$ $H=\left(K_{11}(\mathit{x})+\delta (\omega ){\sigma }^2{I}_{n_1}\right),$ ${\Omega }_1=\delta (\omega ){\sigma }^2{K}_{11}(\mathit{x})H^{-1},$ ${\Omega }_2=(1-{\rho }^2)I_{n_1}+{\rho }^2{\Omega }_1/{\sigma }^2,$ ${\mathit{y}}_{n_1}={(y_1,\dots ,y_{n_1})}^{\top }$ be an $n_1\times 1$ observed vector, $m_1(\mathit{x})={(m({\mathit{x}}_1),\dots ,m({\mathit{x}}_{n_1}))}^{\top },$ and $K_{11}(\mathit{x})$ is the first $n_1\times n_1$ diagonal sub-matrix of $K(\mathit{x}).$

As shown in Lemma 3, if we use the partially observed $y_i$’s to estimate the GPR model, the existence of the truncated normal distribution term (i.e., ${\mathbf{W}}_1^{{\mathbf{C}}_{\beta }}$) in Equation (10) induces a biased estimation of the regression function. Note that the distribution becomes normal (i.e., ${\mathbf{W}}_1\sim {\mathcal{N}}_{n_1}(\mathbf{0},{\Omega }_2)$) in the GPR model for the case where $y_i$’s are fully observed. Therefore, the usual estimation of the regression function based on the GPR model will produce inconsistent results when $\rho \ne 0.$ This clearly reveals that sample-selection bias occurs in Bayes estimation of the regression function ${\mathit{\eta }}_{n_1}.$ The magnitude of this bias is as follows.

Corollary 2.

Instead of the SGPR${}_N,$ if the GPR model is used for estimating ${\mathit{\eta }}_{n_1}$ based on observed data ${\mathit{y}}_{n_1}$ then a sample-selection bias occurs in its conditional posterior mean. This bias is

$$\mathrm{E}\left[{\mathit{\eta }}_{n_1}\right|{\mathit{y}}_{n_1},\Psi ]-\mathrm{E}\left[{\mathit{\eta }}_{n_1}\right|{\mathit{y}}_{n_1},{\sigma }^2]=-{\left(\frac{\rho }{\sigma }{I}_{n_1}+\frac{\sigma (1-{\rho }^2)}{\rho }{\Omega }_1^{*-1}\right)}^{-1}\mathit{\xi },$$

where $\mathit{\xi }=\mathrm{E}\left[{\mathit{W}}_1^{{\mathit{C}}_{\beta }}\right]={({\xi }_1,\dots ,{\xi }_{n_1})}^{\top },$ ${\xi }_i={\omega }_{ii}^{*}\varphi ({\beta }_i;0,{\omega }_{ii}^{*})/[1-\Phi ({\beta }_i/\sqrt{{\omega }_{ii}^{*}})],$ ${\omega }_{ii}^{*}$ denotes the i-th diagonal element of ${\Omega }_2^{*},$ ${\Omega }_1^{*}={\Omega }_1{|}_{\delta \left(\omega \right)=1},$ and ${\Omega }_2^{*}={\Omega }_2{|}_{\delta \left(\omega \right)=1}.$

The sample-selection bias in calculating the marginal effect (or propensity) of a predictor can be also expected.

Corollary 3.

Suppose that $v_{ki}=x_{ki},$ where $v_{ki}$ and $x_{ki}$ are k-th element of ${\mathit{v}}_i$ and ${\mathit{x}}_i,$ then difference in the marginal effect of the predictor $x_{ki}$ on the selected observation $y_i$ between the RSGPR model and the GPR model is

$$\begin{array}{c}\hfill {\gamma }_k\rho \sigma E_{\omega }\left[\frac{1}{\delta \left(\omega \right)}\left({\delta }_2({\mathit{v}}_i^{\top }\mathit{\gamma },\omega )-{\delta }_1{({\mathit{v}}_i^{\top }\mathit{\gamma },\omega )}^2\right)\right],\end{array}$$

(11)

where ${\gamma }_k$ is the k-th element of $\mathit{\gamma },$

$$\begin{array}{ccc}\hfill b_1({\mathit{v}}_i^{\top }\mathit{\gamma },\omega )& =& \delta \left(\omega \right)\varphi ({\alpha }_i;0,\delta \left(\omega \right))/\left[1-\Phi \left({\alpha }_i/\sqrt{\delta \left(\omega \right)}\right)\right],\hfill \\ \hfill b_2({\mathit{v}}_i^{\top }\mathit{\gamma },\omega )& =& {\alpha }_i\delta \left(\omega \right)\varphi ({\alpha }_i;0,\delta \left(\omega \right))/\left[1-\Phi \left({\alpha }_i/\sqrt{\delta \left(\omega \right)}\right)\right],\hfill \end{array}$$

${\alpha }_i=-{\mathit{v}}_i^{\top }\mathit{\gamma },$ and $E_{\omega }$ denotes the expectation is taken with respect to the distribution of $\omega \sim G\left(\omega \right).$

To compare the SGPR${}_N$ model with the $GPR$ model, various values of the sample-selection bias associated with the posterior mean (see, Corollary 2) and the difference in the marginal effect of the k-th predictor (see, Corollary 3) were calculated and are depicted in Figure 1. For the calculation, we set $\sigma =1,$ ${\gamma }_k=1,$ and $K_{11}\left(\mathbf{x}\right)=0.5I_{n_1}+0.5{\mathbf{1}}_{n_1}{\mathbf{1}}_{n_1}^{\top }/n_1,$ an intra-class covariance matrix, where ${\mathbf{1}}_{n_1}$ is an $n_1\times 1$ summing vector whose elements are all one. The left panel in Figure 1 is a graph of the sample-selection bias for different values of ${\beta }_i$ and $\rho .$ This graph shows the values of the first element of the bias vector given in Corollary 2. From the graph, we see that the sample-selection bias occurs in the GPR analysis with sample selection, and its magnitude becomes larger as the values of $\left|\rho \right|$ or ${\beta }_i$ become larger. The sign of the bias is opposite to that of $\rho .$ The right panel shows a graph of the difference in the marginal effect (defined by Equation (11)) as a function of ${\alpha }_i$ and $\rho .$ This graph shows that the absolute value of the difference increases rapidly as ${\alpha }_i$ tends to have a large value, and this difference tends to be larger as the absolute value of $\rho$ becomes larger. Furthermore, the signs of the difference and $\rho$ are different, which is expected for the case where ${\gamma }_k>0.$ These panels imply that an inconsistent nonparametric regression analysis is unavoidable, provided that the GPR model is fitted to the partially observed sample-selection data. Instead, the proposed RSGPR model should be used to correct the sample-selection bias and to estimate the true marginal effect of each predictor in the regression analysis.

3. Bayesian Hierarchical Methodology

3.1. Hierarchical Representation of the RSGPR Model

Let us revisit the RSGPR model (6) in Section 2.2. From Equations (7) and (8), we see that the log-likelihood function of the RSGPR model based on the partially observed n-tuples of observations $(y_i,{\mathit{x}}_i,{\mathit{v}}_i,s_i)$ is

$$\begin{array}{ccc}\hfill l({\mathit{\eta }}_{n_1},\mathit{\gamma },\rho ,{\sigma }^2)& =& \sum _{i=1}^n\left[s_i\left\{ln\overline{F}\left(C_i;0,1\right)+lnh\left(y_i\right|s_i=1)\right\}+(1-s_i)ln\left\{1-\overline{F}(C_i;0,1)\right\}\right].\hfill \end{array}$$

(12)

This is a complex function for the Bayesian estimation of the parameters (${\mathit{\eta }}_{n_1}$ and $\Psi$) of the RSGPR model. Instead, the following hierarchical representation of the RSGPR model is useful for a simple estimation of the parameters.

First, the likelihood function in Equation (12) can be represented by the following distributional hierarchy.

Theorem 1.

For the n-pairs of independent observations, $(y_i,s_i),$ generated from the RSGPR model defined by Equation (6), their distribution can be written by the following Bayesian hierarchical model:

$$\begin{array}{ccc}\hfill \left[y_i|{\omega }_i,z_{C_i},s_i=1\right]& \sim & \mathcal{N}(\eta \left({\mathit{x}}_i\right)+\zeta z_{C_i},\delta \left({\omega }_i\right){\tau }^2),\hfill \\ \hfill p\left(s_i\right|z_i,{\omega }_i)& =& I(z_i\ge 0)I(s_i=1)+I(z_i<0)I(s_i=0),\hfill \\ \hfill \left[z_i|{\omega }_i\right]& \sim & \mathcal{N}({\mathit{v}}_i^{\top }\mathit{\gamma },\delta \left({\omega }_i\right)),\hfill \\ \hfill {\omega }_i& \sim & G\left({\omega }_i\right),i=1,\dots ,n,\hfill \\ \hfill {\mathit{\eta }}_{n_1}& \sim & {\mathcal{N}}_{n_1}(m_1\left(\mathit{x}\right),K_{11}\left(\mathit{x}\right)),\hfill \\ \hfill \left[\zeta |{\tau }^2\right]& \sim & \mathcal{N}({\theta }_0,{\sigma }_0{\tau }^2),\hfill \\ \hfill {\tau }^2& \sim & \mathcal{IG}(c,d),\hfill \\ \hfill \mathit{\gamma }& \sim & {\mathcal{N}}_q({\gamma }_0,{\Omega }_0),\hfill \end{array}$$

where $z_{C_i}=z_i-{\mathit{v}}_i^{\top }\mathit{\gamma },$ $\zeta =\rho \sigma ,$ ${\tau }^2={\sigma }^2(1-{\rho }^2),$ $\mathcal{IG}(c,d)$ denotes an inverse gamma distribution with the p.d.f. $IG({\tau }^2;c,d)=d^c{\tau }^{-2(c+1)}{e}^{-d/{\tau }^2}/\Gamma \left(c\right),$ and $G(·)$ is a distribution function of the scale mixing variable $\omega .$

When the prior information on $\xi ,$ ${\tau }^2,$ and $\mathit{\gamma }$ is not available, a convenient strategy of avoiding improper posterior distribution is to use proper priors with their hyperparameters fixed as appropriate quantity to reflect the diffuseness of the priors (i.e., limiting non-informative priors). For this convenience, the prior distributions in Theorem are used to elicit the prior distributions of $\xi ,$ ${\tau }^2,$ and $\mathit{\gamma }.$ All hyperparameters that appeared in the prior distributions of the Bayesian hierarchical model are assumed to be given from the prior information of previous studies or other sources.

3.2. Full Conditional Posteriors

Let ${\mathbf{y}}_{n_1}={(y_1,\dots ,y_{n_1})}^{\top }$, $\mathbf{V}=({\mathit{v}}_1,\dots ,{\mathit{v}}_n),$ and $\mathbf{s}={(s_1,\dots ,s_n)}^{\top }$ be observed. Further suppose that $\mathbf{z}={(z_1,\dots ,z_n)}^{\top }$ and $\mathit{\omega }={({\omega }_1,\dots ,{\omega }_n)}^{\top }$ are the latent observation vector and the scale mixing vector, respectively. Then, based on the RSGPR model, we obtained joint posterior distribution of $\Theta =\{{\mathit{\eta }}_{n_1},{\tau }^2,\zeta ,\mathit{\gamma },\mathbf{z},\mathit{\omega }\}$ given the observed data set ${\mathcal{D}}_n=\{{\mathbf{y}}_{n_1},\mathbf{V},\mathbf{s}\}:$

$$\begin{array}{ccc}\hfill p(\Theta |{\mathcal{D}}_n)& \propto & \prod _{i=1}^{n_1}\varphi (y_i;\eta \left({\mathit{x}}_i\right)+\zeta z_{C_i},\delta \left({\omega }_i\right){\tau }^2)\hfill \\ & \times & \prod _{i=1}^n\left[p\left(s_i\right|z_i,{\omega }_i)\varphi (z_i;{\mathit{v}}_i^{\top }\mathit{\gamma },\delta \left({\omega }_i\right))g\left({\omega }_i\right)\right]\hfill \\ & \times & IG({\tau }^2;c,d)\varphi (\zeta ;{\theta }_0,{\sigma }_0{\tau }^2){\varphi }_{n_1}({\mathit{\eta }}_{n_1};{\mathrm{m}}_1\left(\mathbf{x}\right),K_{11}\left(\mathbf{x}\right)){\varphi }_q(\mathit{\gamma };{\mathit{\gamma }}_0,{\Omega }_0),\hfill \end{array}$$

(13)

where $g(·)$ is the p.d.f. of the scale mixing variable $\omega .$ Note that the joint posterior in (13) is not simplified in an analytic form of the known density and is thus intractable for posterior inference. Instead, we derive conditional posterior distribution of each parameter in $\Theta$ in an explicit form, which will be useful for posterior inference by using a Markov chain Monte Carlo (MCMC) method.

Given the joint posterior distribution (13), we can obtain the following posterior distributions whose derivations are provided in Appendix A:

(1)

The full conditional posterior distribution of ${\mathit{\eta }}_{n_1}$ is given by

$$\left[{\mathit{\eta }}_{n_1}|{\Theta }_{\setminus {\mathit{\eta }}_{n_1}},{\mathcal{D}}_n\right]\sim {\mathcal{N}}_{n_1}({\mathit{\theta }}_{{\mathit{\eta }}_{n_1}},{\Sigma }_{{\mathit{\eta }}_{n_1}}),$$

(14)

where ${\mathit{\theta }}_{{\mathit{\eta }}_{n_1}}={\Sigma }_{{\mathit{\eta }}_{n_1}}\left(K_{11}{\left(\mathbf{x}\right)}^{-1}{m}_1\left(\mathbf{x}\right)+D_1{\left(\delta \left(\mathit{w}\right)\right)}^{-1}({\mathbf{y}}_{n_1}-\zeta {\mathbf{z}}_C)/{\tau }^2\right),{\Sigma }_{{\mathit{\eta }}_{n_1}}={\left(K_{11}{\left(\mathbf{x}\right)}^{-1}+D_1{\left(\delta \left(\mathit{w}\right)\right)}^{-1}/{\tau }^2\right)}^{-1},$ $D_1\left(\delta \left(\mathit{\omega }\right)\right)=\mathrm{diag}\{\delta \left({\omega }_1\right),\dots ,\delta \left({\omega }_{n_1}\right)\},$ ${\mathbf{z}}_C={(z_{C_1},\dots ,z_{C_{n_1}})}^{\top },$ and $z_{C_i}=z_i-{\mathit{v}}_i^{\top }\mathit{\gamma }.$

(2)

The full conditional posterior distribution of ${\tau }^2$ is an inverse Gamma distribution:

$$\left[{\tau }^2|{\Theta }_{\setminus {\tau }^2},{\mathcal{D}}_n\right]\sim \mathcal{IG}(c+\frac{n_1+1}{2},d+\frac{1}{2}\sum _{i=1}^{n_1}\frac{{(y_i-\eta \left({\mathit{x}}_i\right)-\zeta z_{C_i})}^2}{\delta \left({\omega }_i\right)}+\frac{{(\zeta -{\theta }_0)}^2}{2{\sigma }_0}).$$

(15)

(3)

The full conditional posterior distribution of $\zeta$ is a normal distribution:

$$\begin{array}{c}\hfill \left[\zeta |{\Theta }_{\setminus \zeta },{\mathcal{D}}_n\right]\sim \mathcal{N}({\theta }_{\zeta },{\sigma }_{\zeta }^2),\end{array}$$

(16)

where

$${\theta }_{\zeta }=\frac{{\theta }_0/{\sigma }_0+{\sum }_{i=1}^{n_1}(y_i-\eta \left({\mathit{x}}_i\right))z_{C_i}/\delta \left({\omega }_i\right)}{1/{\sigma }_0+{\sum }_{i=1}^{n_1}{z}_{C_i}^2/\delta \left({\omega }_i\right)}\mathrm{and}{\sigma }_{\zeta }^2=(\frac{1}{{\sigma }_0{\tau }^2}+\frac{{\sum }_{i=1}^{n_1}{z}_{C_i}^2}{\delta ({\omega }_i){\tau }^2}{)}^{-1}.$$

(4)

The full conditional posterior distributions of $z_i$’s are independent and their distributions are given by

$$\begin{array}{c}\hfill \left[z_i|{\Theta }_{\setminus z_i},{\mathcal{D}}_n\right]\stackrel{ind}{\sim }\left\{\begin{array}{cc}{\mathcal{TN}}_{(-\infty ,0)}({\mathit{v}}_i^{\top }\mathit{\gamma },\delta \left({\omega }_i\right))\hfill & \mathrm{if}{s}_i=0,\hfill \\ {\mathcal{TN}}_{(0,\infty )}({\theta }_{z_i},{\sigma }_{z_i}^2)\hfill & \mathrm{if}{s}_i=1\hfill \end{array}\right.\end{array}$$

(17)

for $i=1,\dots ,n,$ where

$${\theta }_{z_i}={\mathit{v}}_i^{\top }\mathit{\gamma }+\frac{\zeta (y_i-\eta \left({\mathit{x}}_i\right))}{{\zeta }^2+{\tau }^2}\mathrm{and}{\sigma }_{z_i}^2=\frac{\delta \left({\omega }_i\right){\tau }^2}{{\zeta }^2+{\tau }^2}.$$

(5)

The full conditional posterior density of $\mathit{\gamma }$ is:

$$\left[\mathit{\gamma }|{\Theta }_{\setminus \mathit{\gamma }},{\mathcal{D}}_n\right]\propto {\mathcal{N}}_q({\mathit{\theta }}_{\mathit{\gamma }},{\Sigma }_{\mathit{\gamma }}),$$

(18)

where ${\mathit{\theta }}_{\mathit{\gamma }}={\Sigma }_{\mathit{\gamma }}\left({\sum }_{i=1}^n\frac{1}{\delta \left({\omega }_i\right)}{z}_i{\mathit{v}}_i+{\sum }_{i=1}^{n_1}\frac{\zeta }{{\tau }^2\delta \left({\omega }_i\right)}(\zeta z_i+\eta \left({\mathit{x}}_i\right)-y_i){\mathit{v}}_i+{\Omega }_0^{-1}{\mathit{\gamma }}_0\right)$ and ${\Sigma }_{\mathit{\gamma }}=({\Omega }_0^{-1}+{\sum }_{i=1}^n\frac{1}{\delta \left({\omega }_i\right)}{\mathit{v}}_i{\mathit{v}}_i^{\top }+{\sum }_{i=1}^{n_1}\frac{{\zeta }^2}{{\tau }^2\delta ({\omega }_i)}{\mathit{v}}_i{\mathit{v}}_i^{\top }{)}^{-1}.$

(6)

The full conditional posterior densities of ${\omega }_i$’s are independent and they are given by

$$\begin{array}{ccc}\hfill p\left({\omega }_i\right|{\Theta }_{\setminus {\omega }_i},{\mathcal{D}}_n)& \propto & \varphi (y_i;\eta \left({\mathit{x}}_i\right)+\zeta z_{C_i},\delta \left({\omega }_i\right){\tau }^2)\varphi (z_i;{\mathit{v}}_i^{\top }\mathit{\gamma },\delta \left({\omega }_i\right))g\left({\omega }_i\right)\mathrm{I}(i\le n_1)\hfill \\ & +& \varphi (z_i;{\mathit{v}}_i^{\top }\mathit{\gamma },\delta \left({\omega }_i\right))g\left({\omega }_i\right)\mathrm{I}(i>n_1).\hfill \end{array}$$

(19)

3.3. Markov Chain Monte Carlo Method

The MCMC scheme, working with the full conditional distributions of the parameters in $\Theta ,$ is not complicated to implement. A routine Gibbs sampler can be used to generate posterior samples of ${\mathit{\eta }}_{n_1},$ ${\tau }^2,$ $\zeta ,$ $z_i,$ and $\mathit{\gamma }$ based on each of their full conditional posterior distributions obtained in Section 3.2. In posterior sampling of ${\omega }_i$’s, Metropolis–Hastings (M–H) within the Gibbs algorithm can be applied because their conditional posterior densities may not have explicit form of known distribution as in Equation (19). For Gibbs sampling, one should note the following points:

(1)

Given the initial values of ${\Theta }^{\left(0\right)},$ the implementation of the Gibbs sampler involves R iterative sampling from each of the full conditional posterior distributions obtained in Equation (14) through Equation (19).

(2)

Gibbs samples of $\rho$ and ${\sigma }^2$ can be obtained by using those of $\zeta =\rho \sigma$ and ${\tau }^2={\sigma }^2(1-{\rho }^2)$.

(3)

If ${\omega }_i$ degenerates at $\delta \left({\omega }_i\right)=1,$ the RSGPR model can be reduced to the SGPR${}_N$ model. In this case, the MCMC procedure excludes the Gibbs sampling of ${\omega }_i$’s by using the posterior distribution (19).

(4)

For various distributions of mixing variable ${\omega }_i$ and mixing functions $\delta \left({\omega }_i\right)$ of the SMN distributions such as $t_{\nu }$, $logit$, $stable,$ $slash,$ and $exponential$$power$ models (see, e.g., [21,22]).

(5)

When ${\omega }_i\stackrel{iid}{\sim }\mathcal{G}(\nu /2,\nu /2)$ and $\delta \left({\omega }_i\right)=1/{\omega }_i,$ the RSGPR model becomes the SGPR${}_{t_{\nu }}$ model. For generating ${\omega }_i$’s, we may use the following posteriors

$$\begin{array}{ccc}\hfill {\omega }_i& \stackrel{ind}{\sim }& \left\{\begin{array}{cc}\mathcal{G}\left(\frac{\nu +2}{2},\frac{\nu +z_{C_i}^2}{2}+\frac{{(y_i-\eta \left({\mathbf{x}}_i\right)-\zeta z_{C_i})}^2}{2{\xi }^2}\right)\hfill & \mathrm{for}i\le n_1,\hfill \\ \mathcal{G}\left(\frac{\nu +1}{2},\frac{\nu +z_{C_i}^2}{2}\right)\hfill & \mathrm{for}i>n_1,\hfill \end{array}\right.\hfill \end{array}$$

(20)

where $z_{C_i}=z_i-{\mathit{v}}_i^{\top }\mathit{\gamma }.$ Except for the SGPR${}_N$ and SGPR${}_{t_{\nu }},$ we need to adopt the Metropolis–Hastings algorithm within the Gibbs sampler because the conditional posterior density of ${\omega }_i$ does not have explicit form of known distribution. See [26,27] for the algorithm for sampling ${\omega }_i$ from the posterior density.

(6)

When the squared exponential covariance function $K\left(\mathbf{x}\right)$ in Equation (4) is chosen with unknown hyperparameters $u_0$ and $w_0,$ we need to elicit the priors of $u_0$ and $w_0$ for the full Bayes methods based on the MCMC method. The priors considered by [28] can be used for this assessment as follows. The prior distributions are a conjugate $u_0\sim \mathcal{IG}(a,b)$ and $w_0\sim \mathcal{HC}(c,d).$ Here $\mathcal{HC}(c,d)$ denotes the half-Cauchy distribution with the p.d.f. $HC(w_0;c,d),$ location parameter $c,$ and scale parameter $d.$ See [28], for compatibility with $w_0\sim \mathcal{HC}(c,d)$ to elicit the prior information on $w_0.$

(7)

Full conditional posterior distributions of $u_0$ and $w_0$ are

$$\left[u_0\right|\Theta ,w_0,\mathcal{D}]\sim \mathcal{IG}(a^{*},b^{*})\mathrm{and}p\left(w_0\right|\Theta ,u_0,\mathcal{D})\propto {\varphi }_{n_1}({\mathit{\eta }}_{n_1};m_1\left(\mathbf{x}\right),K_{11}\left(\mathbf{x}\right))HC(w_0;c,d),$$

where $a^{*}=a+n_1/2$ and $b^{*}=b+u_0{({\mathit{\eta }}_{n_1}-m_1\left(\mathbf{x}\right))}^{\top }{K}_{11}{\left(\mathbf{x}\right)}^{-1}({\mathit{\eta }}_{n_1}-m_1\left(\mathbf{x}\right)).$ Note that the conditional posterior density of $w_0$ does not have explicit form of known distribution. This implies the use of the Metropolis–Hastings algorithm within the Gibbs sampler to generate $w_0$ from the posterior density.

(8)

After obtaining the Gibbs samples of $\Theta ,$ we can use them for Monte Carlo estimation of regression function ${\mathit{\eta }}_{n_2}$ and missing observations ${\mathbf{y}}_{n_2}.$ They can be also used for predicting regression functions and $y_i^{\prime }$s evaluated at new predictors (see, e.g., [26]).

3.4. Prediction with Bias Corrected Regression Function

According to the Gaussian (i.e., MaxEnt) process prior, the joint distribution of the training outputs (${\mathit{\eta }}_{n_1}$) and test outputs $\left({\mathit{\eta }}_{n_2}\right)$ is

$$\left(\begin{array}{c}{\mathit{\eta }}_{n_1}\\ {\mathit{\eta }}_{n_2}\end{array}|\mathbf{x}\right)\sim {\mathcal{N}}_n\left(m\left(\mathbf{x}\right)=\left(\begin{array}{c}{m}_1\left(\mathbf{x}\right)\\ m_2\left(\mathbf{x}\right)\end{array}\right),K\left(\mathbf{x}\right)=\left[\begin{array}{cc}{K}_{11}\left(\mathbf{x}\right)& K_{12}\left(\mathbf{x}\right)\\ K_{21}\left(\mathbf{x}\right)& K_{22}\left(\mathbf{x}\right)\end{array}\right]\right),$$

where ${\mathit{\eta }}_n={({\mathit{\eta }}_{n_1}^{\top },{\mathit{\eta }}_{n_2}^{\top })}^{\top },$ $K_{12}\left(\mathbf{x}\right)$ denotes the $n_1\times n_2$ matrix of the covariances evaluated at all pairs of training points $\left\{{\mathit{x}}_i\right|i=1,\dots ,n_1\}$ and test points $\left\{{\mathit{x}}_j\right|j=n_1+1,\dots ,n\},$ and similarly for the other entities $K_{11}\left(\mathbf{x}\right),$ $K_{21}\left(\mathbf{x}\right),$ $K_{22}\left(\mathbf{x}\right).$ The RSGPR framework provides a straightforward way of predicting test outputs based on the relevant test points and the training outputs. Conditioning the joint Gaussian prior distribution on the training observations, we arrive at the predictive distribution for the future (or missing) regression function given by

$$\left[{\mathit{\eta }}_{n_2}\right|{\mathit{\eta }}_{n_1},\mathbf{x}]\sim {\mathcal{N}}_{n_2}\left(m_2\left(\mathbf{x}\right)+K_{21}\left(\mathbf{x}\right)K_{11}{\left(\mathbf{x}\right)}^{-1}({\mathit{\eta }}_{n_1}-m_1\left(\mathbf{x}\right)),K_{22}\left(\mathbf{x}\right)-K_{21}\left(\mathbf{x}\right)K_{11}{\left(\mathbf{x}\right)}^{-1}{K}_{12}\left(\mathbf{x}\right)\right).$$

(21)

The regression function (${\mathit{\eta }}_{n_2}$) value can be sampled from the predictive distribution (21) by evaluating the mean and covariance matrix of the distribution. Thus, it can be generated within the preceding MCMC algorithm for estimating the RSGPR model: We can generate ${\mathit{\eta }}_{n_2}$ and unobserved observation vector ${\mathbf{y}}_{n_2}={(y_{N_1+1},\dots ,y_n)}^{\top }$ in the r-th iteration of the algorithm whose Markov chain is augmented by the following conditional distributions.

$$\begin{array}{ccc}\hfill \left[{\mathit{\eta }}_{n_2}^{\left(r\right)}\mid {\mathit{\eta }}_{n_1}^{\left(r\right)},\mathbf{x}\right]& \sim & {\mathcal{N}}_{n_2}\left(m_2\left(\mathbf{x}\right)+K_{21}\left(\mathbf{x}\right)K_{11}{\left(\mathbf{x}\right)}^{-1}({\mathit{\eta }}_{n_1}^{\left(r\right)}-m_1\left(\mathbf{x}\right)),K_{22}\left(\mathbf{x}\right)-K_{21}\left(\mathbf{x}\right)K_{11}{\left(\mathbf{x}\right)}^{-1}{K}_{12}\left(\mathbf{x}\right)\right),\hfill \\ \hfill \left[{\mathbf{y}}_{n_2}^{\left(r\right)}\mid {\Theta }^{\left(r\right)},{\mathbf{y}}_{n_1},\mathbf{x}\right]& \sim & \mathcal{N}({\mathit{\eta }}_{n_2}^{\left(r\right)},{\sigma }^{2,\left(r\right)}{D}_2{\left(\delta \left(\mathit{\omega }\right)\right)}^{\left(r\right)}),\hfill \end{array}$$

where ${\mathit{\eta }}_{n_2}^{\left(r\right)}={(\eta {({\mathit{x}}_{n_1+1})}^{\left(r\right)},\dots ,\eta {\left({\mathit{x}}_n\right)}^{\left(r\right)})}^{\top }$ and $D_2{\left(\delta \left(\mathit{\omega }\right)\right)}^{\left(r\right)}=\mathrm{diag}\{\delta {\left({\omega }_{n_1+1}\right)}^{\left(r\right)},\dots ,\delta {\left({\omega }_n\right)}^{\left(r\right)}\}.$ Let ${\mathit{\eta }}_{n_2}^{\left(1\right)},\dots ,{\mathit{\eta }}_{n_2}^{\left(R\right)}$ and ${\mathbf{y}}_{n_2}^{\left(1\right)},\dots ,{\mathbf{y}}_{n_2}^{\left(R\right)}$ are respective samples generated from R iterations, then bias corrected expected value of ${\mathit{\eta }}_{n_2}$ and that of posterior predictive distribution of ${\mathbf{y}}_{n_2}$ can be approximated via Monte Carlo by

$${\widehat{\mathit{\eta }}}_{n_2}=\mathrm{E}\left[{\mathit{\eta }}_{n_2}\right|\mathbf{x}]\approx \frac{1}{R}\sum _{r=1}^R{\mathit{\eta }}_{n_2}^{\left(r\right)}\mathrm{and}\mathrm{E}\left[{\mathbf{y}}_{n_2}\right|{\mathbf{y}}_{n_1},\mathbf{x}]\approx \frac{1}{R}\sum _{r=1}^R{\mathbf{y}}_{n_2}^{\left(r\right)}.$$

Note that $\mathrm{Cov}\left({\mathit{\eta }}_{n_2}\right|\mathbf{x})=K_{22}\left(\mathbf{x}\right)-K_{21}\left(\mathbf{x}\right)K_{11}{\left(\mathbf{x}\right)}^{-1}{K}_{12}\left(\mathbf{x}\right).$

4. Numerical Illustrations

This section presents empirical results of the Bayesian hierarchical RSGPR analysis of non-normal data with the sample selection. We provide results obtained from simulated data applications comparing the performance of the RSGPR model with that of the GPR model. We developed our program written in R (see, e.g., [29]), which is available from the authors upon request.

4.1. Simulation Scheme

In this simulation, we evaluated the finite-sample performance of the RSGPR model by using sample-selection data generated for different sizes. The performance was assessed in terms of sample-selection bias correction and robustness to non-normal model errors. These could be measured by comparing the posterior estimation and prediction results of the RSGPR model with those based on the GPR model. Specifically, we compared the results obtained from the SGPR${}_N$ (or SGPR${}_{t_{10}}$) analysis with the results of the GPR (or GPR${}_{t_{10}}$) analysis based on a partially observed sample-selection data. This study also demonstrated that the SGPR${}_{t_{\nu }}$ model is more robust against outliers compared to the SGPR${}_N$ model. To evaluate the performance, we generated $M=300$ sets of partially observed sample-selection data with size $n=300$ with $n_1=150$ (i.e., the missing rate is 0.5) from each of the three models (see details below). The general form of the three models is as follows:

$$\begin{array}{ccc}\hfill y_i& =& \left\{\begin{array}{cc}50x_i+5\sin \left(10x_i\right)+{ϵ}_i\hfill & \mathrm{for}{s}_i=1,x_i\in (0,1),\hfill \\ \mathrm{missing}\hfill & \mathrm{for}{s}_i=0,i=1,\dots ,n,\hfill \end{array}\right.\hfill \\ \hfill z_i& =& \gamma +{\epsilon }_i,\left(\begin{array}{c}{ϵ}_i\\ {\epsilon }_i\end{array}\right)\stackrel{iid}{\sim }{\mathcal{SMN}}_2(\mathbf{0},\Sigma ,\delta ,G),\hfill \end{array}$$

(22)

where $s_i=\mathrm{I}(z_i\ge 0),$ $\gamma =0,$ $\rho =0.5,$ and $\sigma =3.$

Model 1 was defined by assuming that the distribution G degenerates at $\omega =1.$ Model 2 was obtained from the model (22) by setting $\delta \left(\omega \right)=1/\omega$ and $G\sim \mathcal{G}(10/2,10/2).$ Model 3 assumed a mixture of bivariate normal errors instead of the ${\mathcal{SMN}}_2(\mathbf{0},\Sigma ,\delta ,G)$ distribution. Throughout our simulation, the hyper-parameters for the Bayesian hierarchical model in Theorem 1 were chosen to reflect the diffuseness of the priors. To obtain the limiting non-informative priors of $\zeta ,$ ${\tau }^2$ and $\gamma ,$ their hyper-parameters were assessed as ${\theta }_0=0,$ ${\sigma }_0=10,$ $c=0.001,$ $d=0.001,$ ${\gamma }_0=0,$ and ${\Omega }_0=10.$ Note that when our observational data were augmented through proper prior information, as in this simulation study, the issue to identify the parameters in the RSGPR model disappeared.

In the simulation, we proceeded as follows to estimate the parameters. Using each of $M=300$ datasets generated from the models (Model 1, Model 2, and Model 3), we fitted the RSGPR and GPR models and applied the proposed Bayesian hierarchical methodology to estimate the parameters of the fitted models by assuming the above prior distributions. To implement the methodology by using each generated dataset, we obtained 15,000 posterior samples from the developed MCMC algorithm (in Section 3) with 5 thinning periods after a burn-in period of 5000 samples. This sampling plan guaranteed a convergence of the chain of the MCMC algorithm. The MCMC method (applied to each of $M=300$ datasets) gave estimates (or predictions) of the nonparametric regression function ($\eta \left(x\right)$) as well as the other parameters of the RSGPR model.

The variability in the regression function estimates (${\widehat{\mathit{\eta }}}_{n_1}$) and predictions (${\widehat{\mathit{\eta }}}_{n_2}$) obtained by using a dataset were then visualized as shown in Figure 2 and Figure 3. These figures compare the estimates (or predictions) of the nonparametric regression function obtained from two models (the RSGPR and GPR models). The black line of each graph in the figures shows the true regression function of the model (22). The red dashed line denotes the posterior mean (or predicted value) of the regression function of the model (22) obtained by using a Bayesian hierarchical RSGPR analysis with the sample-selection data of size $n_1=150,$ while the blue dashed line depicts that obtained by using a GPR analysis of the sample-selection data. The 97.5th quantile and 2.5th quantile of 3000 posterior samples (predictions) of each regression function ($\eta \left(x_i\right)$) in the RSGPR model were also calculated. In each figure, these quantiles were used to draw 95% posterior (or prediction) intervals of $\eta \left(x_i\right)$’s by using the gray band. The accuracy of parameter estimates was calculated by using the mean absolute bias (MAB) and the root mean square error (RMSE):

$$\mathrm{MAB}=\frac{1}{M}\sum _{k=1}^M\sum _{\ell =1}^p|{\widehat{\theta }}_{k\ell }-{\theta }_{\ell }|\mathrm{and}\mathrm{RMSE}={\left\{\frac{1}{M}\sum _{k=1}^M\sum _{\ell =1}^p{({\widehat{\theta }}_{k\ell }-{\theta }_{\ell })}^2\right\}}^{1/2},$$

where $M=300$ and ${\widehat{\theta }}_{k\ell }$ is the posterior estimate of ℓ-th element of $p\times 1$ parameter vector $\mathit{\theta }$ in the k-th replication.

4.2. Performance of the RSGPR Model

4.2.1. Sample-Selection Data Generated from Model 1

If the distribution G is degenerated at $\omega =1,$ we then obtain the SGPR${}_N$ model from the RSGPR model (6). Using each of $M=300$ datasets generated from Model 1, the proposed Bayesian hierarchical methodology was applied to estimate the parameters of the model. If we set $\rho =0,$ the methodology yielded posterior estimates of the GPR model. The estimation results for the parameter ${\mathit{\eta }}_{n_1}$ of our primary interest, based on the SGPR${}_N$ and GPR models, are shown in the left panel of Figure 2. The left panel provides the following results. First, the posterior estimates of ${\mathit{\eta }}_{n_1}$ based on the proposed SGPR${}_N$ model (red dashed line) are close to their true values (black line), while those based on the GPR model (blue dashed line) tend to have severe sample-selection bias. Second, when the SGPR${}_N$ model was used to fit the generated sample-selection dataset, the posterior estimates of regression function based on the model concentrated true values of the $\eta \left(x_i\right)$’s as shown in their 95% posterior intervals (gray band). Third, the difference between the true regression function (black line) and the estimated regression function obtained by using the GPR model (the blue dashed line) confirms Lemma 4, which shows the existence of the sample-selection bias in the GPR regression for data with the sample selection. In summary, the left panel of Figure 2 illustrates the existence of the sample-selection bias in the GPR analysis with the sample-selection data, as discussed in Section 2.3. It also demonstrates the performance of the proposed methodology based on the SGPR${}_N$ model to eliminate the sample-selection bias (or inconsistency in estimating the regression function), which could not be achieved by using the GPR model.

The mean of $M=300$ estimation results for the other parameters were listed in

Table 1. Posterior Summary.

True Value	Mean	s.d.	SGPR${}_{\mathit{N}}$ Model		MC Error	Mean	s.d.	GPR Model		MC Error
			RMSE	MAB				RMSE	MAB
$\sigma =3$	2.831	0.308	0.351	0.426	0.018	2.094	0.104	0.912	0.800	0.002
$\rho =0.5$	0.380	0.376	0.563	0.287	0.064	NA	NA	NA	NA	NA
			SGPR ${}_{{\mathbf{t}}_{\mathbf{10}}}$ Model					GPR ${}_{{\mathbf{t}}_{\mathbf{10}}}$ Model
$\sigma =3$	2.880	0.974	0.509	0.515	0.050	2.130	0.109	0.876	0.800	0.003
$\rho =0.5$	0.435	0.275	0.627	0.422	0.032	NA	NA	NA	NA	NA

s.d.: standard deviation; SGPR${}_N$: sample-selection Gaussian process normal error regression; RMSE: root mean square error; MAB: mean absolute bias; MC: Monte Carlo; GPR: Gaussian process regression.

. As shown in Table 1, the MCMC parameter estimates were close to their true values for the SGPR${}_N$ model, while those based on the GPR model were severely biased. In addition, the MAB and RMSE values in the table ensure that the performance of the SGPR${}_N$ model is far better than the GPR model when the sample selection data was used for Bayesian nonparametric regression analysis. For both models, the small values of Monte Carlo (MC) error (compared to the RMSE) of each parameter suggests that approximate convergence was reached and the sequence generated from the MCMC samples was well mixed.

4.2.2. Data Generated from Model 2

The proposed Bayesian hierarchical methodology for the SGPR${}_{t_{10}}$ model was applied to each of $M=300$ datasets generated from Model 2. The SGPR${}_{t_{10}}$ model can be obtained from the SGPR model (6) by setting $\delta \left(\omega \right)=1/\omega$ and $\omega \sim \mathcal{G}(10/2,10/2).$ If we set $\rho =0,$ the methodology could also be used to obtain posterior samples to estimate the GPR${}_{t_{10}}$ model (GPR model with $t_{10}$ errors). The results of the simulation appear in the right panel of Figure 2 and Table 1. Graphs in the right panel of Figure 2 depict the prediction results of ${\mathit{\eta }}_{n_2}$ based on the SGPR${}_{t_{10}}$ and GPR${}_{t_{10}}$ models. The graphs clearly show that the sample-selection bias in predicting $\eta \left({\mathit{x}}_i\right)$’s based on the GPR${}_{t_{10}}$ model is too large to allow for a prediction of the true regression function ${\mathit{\eta }}_{n_2}$ (or future regression function). However, the proposed methodology using the SGPR${}_{t_{10}}$ model correctly predicted the true regression function; see 95% prediction interval and ${\widehat{\mathit{\eta }}}_{n_2},$ i.e., red dashed line. The prediction of ${\mathit{\eta }}_{n_2}$ based on the SGPR${}_{t_{10}}$ model is far better than that based on the GPR${}_{t_{10}}$ model. Compared to the GPR${}_{t_{10}}$ model, the methodology based on the SGPR${}_{t_{10}}$ model yields smaller MAB and smaller RMSE of the parameter estimates; see Table 1. Table 1 shows that the parameter estimates of the SGPR${}_{t_{10}}$ model with heavy-tailed errors tend to produce larger estimation errors (MAB and RMSE) than those of the SGPR${}_N$ model. The results of the above simulation demonstrate the superior performance of the SGPR${}_{t_{10}}$ model and the usefulness of the proposed Bayesian hierarchical methodology to remedy the sample-selection bias in the prediction that occurred in the GPR${}_{t_{10}}$ analysis of the sample-selection data.

4.2.3. Data Generated from Model 3 with Normal Mixture Errors

We generated datasets from Model 3 with size $n=300.$ Model 3 was defined by the model (22) with independent bivariate normal mixture errors: viz.

$$0.4{\mathcal{N}}_2(\mathbf{0},{\Sigma }_{\left(1\right)})+0.2{\mathcal{N}}_2(\mathbf{0},{\Sigma }_{\left(2\right)})+0.2{\mathcal{N}}_2(\mathbf{0},{\Sigma }_{\left(4\right)})+0.1{\mathcal{N}}_2(\mathbf{0},{\Sigma }_{\left(8\right)})+0.1{\mathcal{N}}_2(\mathbf{0},{\Sigma }_{\left(16\right)}),$$

where 50% of the outcomes were missing in each dataset and ${\Sigma }_{\left(k\right)}$ was equal to $\Sigma$ whose value of ${\sigma }^2$ was $9k.$ The generated dataset was fitted to the SGPR${}_N$, SGPR${}_{t_5}$, and SGPR${}_{t_{10}}$ models in turn. Based on posterior samples, we calculated the Bayes estimates of the three models’ parameters together with their deviance information criterion (DIC) values introduced by [30]. The average DIC values obtained from the dataset were found to be 2727.06, 1477.43, 1396.24 for the SGPR${}_N$, SGPR${}_{t_{10}}$, and SGPR${}_{t_5}$ models, respectively. This suggested that the SGPR${}_{t_5}$ model is the best fitting model among the three models, while the SGPR${}_N$ model is the worst.

The graphs in the left panel of Figure 3 show the true regression function (black line) and estimated regression functions (${\widehat{\mathit{\eta }}}_{n_1}$) under the best fitting SGPR${}_{t_5}$ model (red line) and the SGPR${}_N$ model (blue line). The graphs in the right panel of Figure 3 depict predicted regression function (${\widehat{\eta }}_{n_2}$) based on the best fitted model (in red), the SGPR${}_N$ model (in blue), and the true regression function (in black). Even though the best fitted model based on bivariate $t_5$ error distributions was misspecified, 95% posterior intervals (or prediction intervals) of $\eta \left(x_i\right)$ obtained from the SGPR${}_{t_5}$ model did include the true regression function values (see gray bands in Figure 3). The prediction result of the SGPR${}_{t_5}$ and SGPR${}_N$ models are very wild due to outliers generated by the normal mixture errors, while the graphs of the SGPR${}_{t_5}$ are more robust for the model misspecification.

5. Conclusions

This study considered a MaxEnt approach to develop a Bayesian nonparametric regression analysis of non-normal data with the sample selection. For this purpose, by using Boltzmann’s maximum entropy theorem, we introduced a MaxEnt process regression model that reflects partial prior information for an uncertain regression function. We found that a special case of the MaxEnt regression model reduced to the well-known GPR model. Second, we generalized the GPR model to propose the RSGPR model and explored its theoretical properties. These properties showed that the new model was well-designed to correct the sample-selection bias and implement a robust GPR analysis. Third, we developed a hierarchical RSGPR model based on a stochastic representation of the RSGPR model and proposed a Bayesian hierarchical methodology for the RSGPR analysis of a non-normal data with sample selection. A simulation study showed that the finite sample performance of the proposed methodology eliminated the sample selection bias and estimated the population model parameters with robustness and high accuracy. In a comparative numerical study on the analysis of nonparametric regression models with sample selection data, we found that the estimation results using the RSGPR model outperformed those using the GPR model for both in-sample estimation and out-of-sample forecasts.

The theoretical results of the RSGPR model and the methodology for the RSGPR analysis proposed in this study have several interesting issues that are worth considering further. First, the RSGPR framework using the MaxEnt process prior can be generalized to the so called stochastically constrained RSGPR regression that uses the constrained MaxEnt process as the prior distribution of the regression function with uncertain constraints. Second, an empirical study with real data as well as an asymptotic evaluation, such as consistency, would be particularly noteworthy to explore. For example, estimating monotone regression function with or without uncertainty and testing the monotonicity of the regression function can be considered in the context of a constrained RSGPR analysis with sample-selection data. Finally, the Bayesian hierarchical methodology can be broadened in various regression models with the general class of skew-$\mathcal{SMN}$ error distributions considered by [11]. For example, this methodology can be applied to a von Bertalanffy growth curve analysis of heavy-tailed fishery data with sample selection (see, e.g., [28]). We hope to address all of these in the future.