Adaptive Bayesian Nonparametric Regression via Stationary Smoothness Priors

Abstract

A procedure for Bayesian nonparametric regression is described that automatically adjusts the degree of smoothing as the curvature of the underlying function changes. Relative to previous work adopting a similar approach that either employs a single global smoothing parameter or assumes that the smoothing process follows a random walk, the model considered here permits adaptive smoothing and imposes stationarity in the autoregressive smoothing process. An efficient Markov Chain Monte Carlo (MCMC) scheme for model estimation is fully described for this stationary case, and the performance of the method is illustrated in several generated data experiments. An application is also provided, analyzing the relationship between behavioral problems in students and academic achievement. Point estimates from the nonparametric methods suggest (a) expected achievement declines monotonically with a behavioral problems index (BPI) score and (b) the rate of decline is relatively flat at the left tail of the BPI distribution and then becomes sharply more negative.

1. Introduction

The literature on nonparametric Bayesian estimation of regression relationships is substantial, as a variety of approaches have been advanced to successfully recover shapes and properties of unknown functions in the presence of observational data. One approach to Bayesian nonparametric regression, for example, employs basis functions or splines to flexibly model regression relationships (e.g., Smith and Kohn [1], Kohn et al. [2], and Chib et al. [3]). These approaches typically require the researcher to choose a functional form for the basis and the potential locations of a series of knots, and as estimation proceeds, the data are used to determine if terms in the series representation are retained or discarded. Other studies employ Gaussian processes (GPs) for this purpose (e.g., Rasmussen [4] and Dixon et al. [5] (in particular, Chapter 3 of Dixon et al. [5] provides considerable treatment of the GP approach to Bayesian nonparametric regression)). GP models, which require the researcher to choose a mean function and covariance kernel, place a prior distribution over the function space, and plausible functional realizations from the associated stochastic process are identified given the data that are observed. Still another set of studies estimate relationships nonparametrically by treating each point on the regression surface as a parameter to estimate, and then by representing the nonparametric relationship in traditional regression form. Carefully constructed priors are employed in this setting that impose similarity among neighboring points on the regression surface, thereby smoothing out what might otherwise be an erratic collection of regression coefficients. This paper follows in the path of this “smoothness prior” approach and extends it by allowing for local adaptability in the degree of smoothing, by imposing stationarity in the smoothing parameter process, and by allowing the data to update some features of how the smoothing parameters evolve.

To generally describe the approach taken, the case where our covariate, say x, is discrete-valued serves as a useful explanatory starting point. When seeking to estimate a regression function $f\left(x\right)$ with such data, one would likely begin by creating a series of dummy/indicator variables which exhaust all possible x outcomes. The resulting collection of estimated dummy variable coefficients when proceeding this way, however, may produce a rather erratic estimate of the regression function, particularly when there are few observations within some of the discrete cells and/or the error variance is large. To help remedy this problem, one could adopt an informative prior expressing that adjacent dummy variable coefficients should be similar in value, essentially borrowing informational strength across the discrete cells. Expressions of such local “similarity” can be achieved, for example, by constructing a prior that centers differences of the dummy variable coefficients over zero. The extent of shrinkage/smoothing that takes place is then determined by how tightly those differences are centered over zero.

Studies such as Koop and Poirier [6], Koop and Tobias [7], Chib and Jeliazkov [8], and Chan and Tobias [9], for example, employ this smoothness prior approach. Their specifications contain a single variance parameter in the construction of the smoothness prior, which is akin to a constant global bandwidth parameter in frequentist kernel-based nonparametric regression. While these methods generally perform quite well, they are nonetheless somewhat limited by the unique smoothing parameter: the same degree of smoothing is employed over regions of the support where the function is flat or linear, and over regions where the function may be rapidly changing. Tobias and Bond [10] extend this single-parameter model, allow for adaptability in the degree of smoothing, and structure their prior so that the smoothing parameters follow a random walk. Although the adaptability afforded by such an algorithm may offer real benefit in empirical work, the assumed random walk process is nonetheless somewhat limiting as it imposes, for example, that prior variances grow as one moves across the support of x.

In this paper, we describe an alternative to the random walk model which may offer improved performance in practice. We allow the data to partially inform the way smoothness evolves by assuming the smoothing parameters follow an AR(1) process rather than a random walk. A truncated prior for the autoregressive coefficient is employed to impose stationarity of the smoothing parameter process. This elaborated structure introduces several new parameters to the model (e.g., a stationary mean and an autoregressive coefficient), and an efficient Markov Chain Monte Carlo (MCMC) scheme is derived and employed for estimating this generalized model. We show in several generated experiments that the corresponding AR(1) smoothing process can adapt more quickly than the random walk to functional curvature changes when they occur. Moreover, our experience suggests that the AR(1) structure tends to be computationally more stable and perhaps less sensitive to changes in prior hyperparameters. Offsetting this, the random walk model contains fewer parameters than the AR(1), as it imposes that the autoregressive coefficient equals unity, which both simplifies the resulting estimation algorithm and could potentially lead to increased precision. In some of the conducted experiments, support for the random walk specification is revealed, and in those cases the mass of the posterior distribution of the autoregressive coefficient is seen to concentrate near the unit root. In other cases, we find evidence against the random walk, and in those cases, the posterior distribution of the autoregressive coefficient concentrates in the interior of the support and away from the unit root.

The outline of this paper is as follows. The next section outlines our model and describes the prior specifications used to conduct nonparametric estimation and smoothing. Section 3 fully describes an MCMC scheme for model estimation, while generated data experiments are conducted in Section 4. Section 5 provides an application investigating the impact of behavioral problems in adolescents on achievement scores in mathematics and two measures of reading acumen. In this application, we find some evidence of nonlinearities, which are then further supported by simpler parametric models whose specifications are guided by our initial exploratory nonparametric findings. Specifically, additional behavioral problems are found to have a relatively small impact on achievement when the extent of behavioral issues is currently small, while increasing the number of behavioral problems has clearly negative impacts on achievement when the number of behavioral issues is already moderate or large. Finally, this paper concludes with a summary in Section 6.

2. The Model

Before describing our model, we establish a few notational conventions. Univariate random variables (i.e., x) are denoted with lowercase letters or symbols, while vector-valued quantities (i.e., $\mathit{x}$) are presented in bold lowercase font. Matrices (i.e., $\mathit{X}$) are denoted with bold capital letters or Greek symbols, as appropriate.

In this paper, we will introduce a new framework for estimating the conditional mean function in a univariate nonparametric regression model. Before describing our approach, we note that many extensions of this base model are easily handled. For example, the posterior simulator can be easily adapted to include additional controls in a partially linear framework: $y_i=f\left(x_i\right)+{\mathit{z}}_i\mathit{\beta }+u_i$. Non-normality of the disturbance vector can also be accommodated through scale or finite mixtures (see, e.g., Chan et al. [11], chapter 15, and associated references). The strategy for nonparametric estimation described below can also be included as a component of a larger treatment response model when seeking to identify the causal effect of a covariate or set of covariates on an outcome of interest (e.g., Kline and Tobias [12]). We fix ideas here, however, on the basic univariate nonparametric regression model in order to focus attention on the contributions of our estimation method:

$$y_i=f\left(x_i\right)+{ϵ}_i,i=1,2,\dots ,n,\mathit{ϵ}|\mathbf{X}\sim \mathcal{N}(\mathbf{0},{\sigma }^2{\mathbf{I}}_n)$$

where $\mathcal{N}(\mathit{\mu },\mathbf{\Sigma })$ denotes a normal/Gaussian distribution with mean $\mathit{\mu }$ and covariance matrix $\mathbf{\Sigma }$. The covariate x takes on $k_x\le n$ distinct values. We let $x_1^{*},x_2^{*},\dots ,x_{k_x}^{*}$ denote the realized distinct ordered values of the random variable x so that $x_j^{*}>x_{j-1}^{*}\forall j=2,3,\dots k_x$. Furthermore, we let ${\Delta }_j$ denote the distance between consecutive $x^{*}$ values, i.e., ${\Delta }_j\equiv x_j^{*}-x_{j-1}^{*}$.

Our approach treats each $f\left(x_j^{*}\right)$ as a parameter to estimate. Smoothness will be achieved by specifying a prior expressing local similarity among neighboring $f^{\prime }\left(x^{*}\right)$ values, and the function is recovered by piecing together the collection of smoothed coefficients. We begin this process by constructing a ($1\times k_x$) label vector ${\mathbf{d}}_i,$ for each observation i, as follows:

$${\mathbf{d}}_i=[\mathbb{1}(x_i=x_1^{*})\mathbb{1}(x_i=x_2^{*})\cdots \mathbb{1}(x_i=x_{k_x}^{*})],$$

where $\mathbb{1}(·)$ denotes a standard indicator function. The vector ${\mathbf{d}}_i$ contains zeros everywhere except for a one in the position that corresponds to the observed value of $x_i$. We let $\mathbf{D}$ denote the stacked $n\times k_x$ label matrix for the full sample and $\mathit{\pi }$ denote the associated $(k_x\times 1)$ vector of (sorted) dummy variable coefficients:

$$\mathbf{D}=\left[\begin{array}{c}{\mathbf{d}}_1\\ {\mathbf{d}}_2\\ ⋮\\ {\mathbf{d}}_n\end{array}\right],\mathit{\pi }=\left[\begin{array}{c}{\pi }_1\\ {\pi }_2\\ ⋮\\ {\pi }_{k_x}\end{array}\right].$$

With this construction, we can write our nonparametric regression problem in standard and familiar regression form as

$$\mathit{y}=\mathbf{D}\mathit{\pi }+\mathit{ϵ}.$$

In the following section, we describe how this method is operationalized through the construction of smoothness priors on $\mathit{\pi }$. The priors employed will also be adaptive, automatically adjusting the degree of smoothing as the curvature of the function f changes.

Priors and Regression Function Smoothing

We smooth the collection of $\mathit{\pi }$ coefficients via a prior which imposes that neighboring pointwise slopes of the regression function should be similar in value. Such priors have been employed previously by, for example, Koop and Poirier [6], Koop and Tobias [7], Chib and Jeliazkov [8], Chib et al. [13], and Bond and Tobias [10], among others. The prior employed here will be different, however, as it does not employ a single global smoothing parameter. Instead, it associates every cell with its own smoothing parameter to permit local adaptability, while imposing stationarity in the smoothing parameter distribution.

We begin by defining the $(k_x\times k_x)$ second-order differencing matrix:

$${\mathbf{H}}_{\pi }=\left[\begin{array}{cccccccc}1& 0& 0& 0& \cdots & 0& 0& 0\\ 0& 1& 0& 0& \cdots & 0& 0& 0\\ {\Delta }_2^{-1}& -({\Delta }_3^{-1}+{\Delta }_2^{-1})& {\Delta }_3^{-1}& 0& \cdots & 0& 0& 0\\ 0& {\Delta }_3^{-1}& -({\Delta }_4^{-1}+{\Delta }_3^{-1})& {\Delta }_4^{-1}& \cdots & 0& 0& 0\\ ⋮& ⋮& ⋮& ⋮& \ddots & ⋮& ⋮& ⋮\\ 0& 0& 0& 0& \cdots & {\Delta }_{k_x-1}^{-1}& -({\Delta }_{k_x}^{-1}+{\Delta }_{k_x-1}^{-1})& {\Delta }_{k_x}^{-1}\end{array}\right].$$

The matrix ${\mathbf{H}}_{\pi }$ is constructed so that the quantity ${\mathbf{H}}_{\pi }\mathit{\pi }$ will select off the first two elements of the dummy variable coefficient vector (${\pi }_1$ and ${\pi }_2$) and then select the first differences of adjacent pointwise slopes of our regression function. These slope-first differences will take the form ${\Delta }_j^{-1}({\pi }_j-{\pi }_{j-1})-{\Delta }_{j-1}^{-1}({\pi }_{j-1}-{\pi }_{j-2}),j=3,4,\dots ,k_x$.

We achieve smoothing by imposing a belief that adjacent slopes should be similar in value. We accomplish this through the prior

$${\mathbf{H}}_{\pi }\mathit{\pi }|{\mathit{\pi }}_0,\mathit{\eta }\sim \mathcal{N}\left({\mathbf{X}}_{\pi }{\mathit{\pi }}_0,{\mathbf{V}}_{\mathit{\eta }}\right),$$

(1)

where

$${\mathbf{X}}_{\pi }=\left[\begin{array}{c}{\mathbf{I}}_2\\ {\mathbf{0}}_{(k_x-2)\times 2}\end{array}\right],{\mathit{\pi }}_0=\left[\begin{array}{c}{\pi }_{01}\\ {\pi }_{02}\end{array}\right],\mathrm{and}{\mathbf{V}}_{\mathit{\eta }}=\left[\begin{array}{ccccc}exp\left({\eta }_1\right)& 0& 0& \cdots & 0\\ 0& exp\left({\eta }_2\right)& 0& \cdots & 0\\ ⋮& ⋮& ⋮& \ddots & ⋮\\ 0& 0& 0& \cdots & exp\left({\eta }_{k_x}\right)\end{array}\right].$$

(2)

The $\eta$ parameters are log variances/log volatilities, which govern the degree of smoothing of the regression function and summarize how that smoothing might change across the support of x. To explain the role and interpretation of the $\eta$ parameters in greater detail, first note that the prior in (1) centers first differences of slopes of our regression function over zero. As $exp\left({\eta }_i\right)\to 0$, $j>2$, the prior imposes that adjacent slope differences are essentially equal to zero, thus constraining the regression function to be linear. More generally, however, the $exp\left({\eta }_j\right)$ can (and typically will) depart from zero. In such a case, the prior expresses preference for local similarity in neighboring function values (thereby achieving “smooth” results) yet does not strictly impose linearity. In regions where the true function behaves according to a linear or near-linear relationship, we would hope to associate those $f\left(x^{*}\right)$ with small $\eta$ values. In contrast, when the function is changing rapidly, a larger $\eta$ would base results on relatively small local neighborhoods, thus allowing for improved detection of local changes in functional curvature. The model as presented offers the potential for adaptability and the automatic determination of the appropriate neighborhood size/degree of smoothing, as $\eta$ can change with the local properties of the underlying function. We provide some examples of the algorithm’s success in adapting in this way in the experiments of Section 4. It is worth noting that many studies pursuing nonparametric estimation through priors like these employ a constant smoothing parameter, specifying ${\mathbf{V}}_{\eta }$ to be diagonal with a common ${\eta }^2$ across all diagonal elements. Our model nests this as a special case and associates each distinct $x_j^{*}$ with its own smoothing parameter. To operationalize the approach, some additional structure is required which describes how $exp\left(\mathit{\eta }\right)$ evolves. We will return to complete this description shortly.

An additional hierarchical prior is placed over ${\mathit{\pi }}_0$ of the form

$${\mathit{\pi }}_0\sim \mathcal{N}\left({\underline{\mathit{\mu }}}_{{\pi }_0},{\underline{\mathbf{V}}}_{{\pi }_0}\right),$$

(3)

where the underline notation is used to denote terminal hyperparameters selected by the researcher. In specifying our model this way, the prior covariance matrix ${\underline{\mathbf{V}}}_{{\pi }_0}$ can be selected with large diagonal elements to let the data speak regarding the first two points on our regression curve. Specifically, if small values of $\eta$ are generally desired in order to smooth the curve, while the elements of $\mathit{\eta }$ are also correlated in their evolution, the resulting prior for the initial conditions may be undesirably concentrated around the prior mean ${\underline{\mathit{\mu }}}_{{\pi }_0}$. The hierarchical specification in (1) and (3) combined with diffuse choices for ${\underline{\mathbf{V}}}_{{\pi }_0}$ can help mitigate this problem and allow the data to better inform ${\pi }_1$ and ${\pi }_2$.

As discussed earlier in this section, the variances $\{exp\left({\eta }_j\right)\}$ control the degree of smoothing of our regression function. It seems reasonable to assume that the ${\eta }_j$ parameters, like those of the regression function itself, should be locally similar. While Tobias and Bond [10] express this similarity by assuming $\mathit{\eta }$ smoothly evolves as a random walk, this (potentially undesirably) implies that the prior variance of $\mathit{\eta }$ must increase as one moves into the right tail of the x distribution (Tobias and Bond [10] mention that a stationary version of the process produced similar results, but did not present an algorithm for such estimation. We formally describe such an algorithm in Section 3). When adopting such a specification, the imposed unit root may also imply that $\mathit{\eta }$ adapts to curvature changes in $f\left(x\right)$ rather slowly, while alternate specifications may permit more rapid adaptability. We provide examples that both favor and oppose the random walk structure in the generated data experiments of the following section.

In what follows, we assume that the elements of $\mathit{\eta }$ follow an autoregressive or AR(1) process:

$$\begin{array}{ccc}\hfill {\eta }_2& =& \mu +\rho ({\eta }_1-\mu )+u_2\hfill \\ \hfill {\eta }_3& =& \mu +\rho ({\eta }_2-\mu )+u_3\hfill \\ \hfill & ⋮& \hfill \\ \hfill {\eta }_{k_x}& =& \mu +\rho ({\eta }_{(k_x-1)}-\mu )+u_{k_x}\hfill \end{array}$$

where $u_j\stackrel{iid}{\sim }\mathcal{N}(0,{\tau }^2),j=2,3,\dots ,k_x.$ The parameter $\rho$ measures the degree of persistence in the series, and the process is stationary provided $\left|\rho \right|<1$. We will employ priors that guarantee stationarity by truncating the support of $\rho$ accordingly, although results from the nonstationary random walk model may be approximately reproduced when the posterior for $\rho$ concentrates on values near, but strictly less than, unity. The smoothing process is initialized by assuming that ${\eta }_1$ is drawn from the stationary distribution of the process, i.e., ${\eta }_1\sim \mathcal{N}(\mu ,{\tau }^2/(1-{\rho }^2)).$ We refer to this model in what follows as the “stationary” case or the “estimated-$\rho$” case. This is in contrast to the random walk model, which imposes that $\rho =1$ (and thus eliminates $\mu$) and adds a prior for the initial condition of the form ${\eta }_1\sim \mathcal{N}(\underline{\mu },\underline{V}).$ Other higher-order autoregressive processes for the smoothing parameters could also be entertained, although we restrict attention to the baseline AR(1) model here (in the experiments and application of the following sections, we show that, while learning takes place regarding the persistence parameter $\rho$, it is not strongly identified by the data. This suggests that further generalizations of the smoothing parameter process may prove quite difficult to implement in practice).

To further describe the implied joint prior for $\mathit{\eta }$, we first define a few objects. We define the $(k_x\times k_x)$ matrix ${\mathit{G}}_{\rho }$, $(k_x\times 1)$ vector ${\mathit{g}}_{\rho }$, and diagonal $(k_x\times k_x)$ covariance matrix $\mathbf{\Omega }=\mathbf{\Omega }({\tau }^2,\rho )$ as follows:

$${\mathit{G}}_{\rho }=\left[\begin{array}{cccccc}1& 0& 0& \cdots & 0& 0\\ -\rho & 1& 0& \cdots & 0& 0\\ 0& -\rho & 1& \cdots & 0& 0\\ ⋮& ⋮& ⋮& \ddots & ⋮& ⋮\\ 0& 0& 0& \cdots & -\rho & 1\end{array}\right],{\mathit{g}}_{\rho }=\left[\begin{array}{c}(1-{\rho }^2)\\ (1-\rho )\\ (1-\rho )\\ ⋮\\ (1-\rho )\end{array}\right],\mathbf{\Omega }=\left[\begin{array}{ccccc}{\displaystyle \frac{{\tau }^2}{(1-{\rho }^2)}}& 0& 0& \cdots & 0\\ 0& {\tau }^2& 0& \cdots & 0\\ 0& 0& {\tau }^2& \cdots & 0\\ \cdots & \cdots & \cdots & \ddots & \cdots \\ 0& 0& 0& \cdots & {\tau }^2\end{array}\right].$$

With these definitions in hand, we can express the joint prior for $\mathit{\eta }$ as

$${\mathit{G}}_{\rho }(\mathit{\eta }-{\mathit{\iota }}_{k_x}\mu )|\rho ,{\tau }^2\sim \mathcal{N}\left(\mathbf{0},\mathbf{\Omega }\right),$$

(4)

where ${\mathit{\iota }}_z$ denotes a $z\times 1$ vector of ones. The model specification is complete upon choosing priors for $\mu$, $\rho$, ${\tau }^2$, and ${\sigma }^2$. In what follows, we make the following choices:

$$\mu \sim \mathcal{N}(\underline{\mu },{\underline{V}}_{\mu }),{\tau }^2\sim IG({\underline{a}}_{\tau },{\underline{b}}_{\tau }),\mathrm{and}{\sigma }^2\sim IG({\underline{a}}_{\sigma },{\underline{b}}_{\sigma }),$$

with $IG(·,·)$ denoting an inverse gamma distribution (see, e.g., Chan et al. [11], page 442). In addition, we consider two different prior specifications for $\rho$. The first is a discrete prior with J points of support (denoted ${\left\{{\underline{\rho }}_j\right\}}_{j=1}^J$, $|{\underline{\rho }}_j|<1\forall j$):

$$\mathsf{P}\mathsf{r}(\rho ={\underline{\rho }}_j)={\underline{p}}_j,\rho \in \{{\underline{\rho }}_1,{\underline{\rho }}_2,\dots ,{\underline{\rho }}_J\},\sum _{j=1}^J{\underline{p}}_j=1,{\underline{p}}_j\in (0,1)\forall j.$$

(5)

In addition, we also consider a continuous prior of the form

$$\rho \sim T{N}_{(-1,1)}({\underline{\mu }}_{\rho },{\underline{V}}_{\rho }),$$

(6)

where $T{N}_{(a,b)}(\mu ,{\sigma }^2)$ denotes a normal distribution with mean $\mu$ and variance ${\sigma }^2$ that is truncated to the interval $(a,b)$ (see, e.g., Chan et al. [11] pages 132–134). We observe that the truncation in the prior for $\rho$ imposes that the smoothing process is stationary.

3. Posterior Simulation

In this section, we seek to derive a Markov Chain Monte Carlo (MCMC) scheme to estimate our nonparametric regression model. To this end, we begin by noting that the joint posterior distribution of our model parameters is given up to proportionality as

$$p(\mathit{\pi },{\mathit{\pi }}_0,\mathit{\eta },\rho ,{\sigma }^2,{\tau }^2,\mu |\mathit{y})\propto \varphi \left(\mathit{y};\mathbf{D}\mathit{\pi },{\sigma }^2{\mathit{I}}_n\right)p\left(\rho \right)p\left({\sigma }^2\right)p\left({\tau }^2\right)p\left(\mu \right)p\left({\mathit{\pi }}_0\right)p\left(\mathit{\eta }\right|\mu ,\rho ,{\tau }^2)p\left(\mathit{\pi }\right|{\mathit{\pi }}_0,\mathit{\eta }),$$

(7)

with $p(·)$ serving as generic notation for a density or mass function. In (7), $\varphi (\mathit{x};\mathit{\mu },\mathbf{\Sigma })$ denotes a multivariate normal density function for the random vector $\mathit{x}$ with mean vector $\mathit{\mu }$ and covariance matrix $\mathbf{\Sigma }$. The specific forms of the priors appearing in (7) were given in the previous section, and $p\left(\rho \right)$ may represent either the continuous or discrete variant of the prior for the autoregressive coefficient $\rho$.

Below we fully describe all steps of our Markov Chain Monte Carlo sampling scheme, which involves iterative simulation from the conditional posterior distributions of the model parameters. With just one exception (the sampling of $\rho$ in the continuous case), each step only requires simulation from standard distributions. When required, a well-mixing Metropolis–Hastings (MH) algorithm is suggested for the sampling of $\rho$, and thus the method should be generally accessible by practitioners.

The approach that follows can be interpreted as an approximate (though highly accurate) MCMC procedure. The approximate nature of our results owes to a Gaussian mixture approximation that will be used to facilitate drawing the smoothing parameters/log volatilities $\mathit{\eta }$. Specifically, we use a 10-component Gaussian mixture approximation of the log of a ${\chi }^2\left(1\right)$ random variable, as in Omori et al. [14], to aid in the sampling of $\mathit{\eta }$. This will require the introduction of a vector of component indicators, denoted $\mathbf{s}$, and the sampling of these indicators will conclude one sweep of the sampler. As discussed in Del Negro and Primiceri [15] in the context of stochastic volatility models, the ordering of the sampling steps is important in this context.

To describe the ordering issue formally, we will conceptually group a subset of parameters together as follows: $\mathit{\gamma }\equiv \{\mathit{\pi },{\mathit{\pi }}_0,{\sigma }^2,\mu ,\rho ,{\tau }^2\}.$ We will also make use of the notation ${\mathit{\gamma }}_{-x}$ to denote all elements of $\mathit{\gamma }$ other than x. Each element of $\mathit{\gamma }$ will be sampled from its exact conditional posterior distribution, as derived from (7). Our sampling scheme will draw, in order, from $\left[\mathit{\eta }\right|\mathbf{s},\mathit{\gamma },\mathit{y}]$, $\left[\mathit{\gamma }\right|\mathit{\eta },\mathit{y}]$, and finally $\left[\mathbf{s}\right|\mathit{\eta },\mathit{\gamma },\mathit{y}]$. Sampling from the last two of these amounts to drawing from the joint conditional posterior distribution $[\mathbf{s},\mathit{\gamma }|\mathit{\eta },\mathit{y}]$ given that elements of $\mathit{\gamma }$ will be drawn from their exact conditional posteriors, which do not involve any approximations (and thus $\mathbf{s}$). Importantly, the sampling of $\mathit{\gamma }$ in this scheme needs to take place immediately before $\mathbf{s}$. Before describing each of the required eight steps, we note the structural similarity of the nonparametric regression problem considered here with traditional stochastic volatility models. The reader is invited to see, for example, Chan [16] for related discussions. As described in Kim, Shephard, and Chib [17], one can make adjustments for the mixture approximation error via, for example, importance sampling reweighting of the produced simulations. This is seldom performed in practice, however, as the approximation is highly accurate and such adjustments tend to produce no meaningful differences in results.

• Step 1: $\mathit{\eta }|\mathit{s},\mathit{\gamma },\mathbf{y}$:

In equation form, the prior in (1) can be expressed as

$$\mathit{\psi }={\mathbf{X}}_{\pi }{\mathit{\pi }}_0+\mathit{\nu },$$

where $\mathit{\psi }\equiv {\mathbf{H}}_{\pi }\mathit{\pi }$ and $\mathit{\nu }$ is multivariate Gaussian with mean zero and diagonal covariance matrix ${\mathbf{V}}_{\mathit{\eta }}$ containing ${\{exp\left({\eta }_i\right)\}}_{i=1}^{k_x}$ stacked across the diagonal. Rearranging this equation by squaring both sides and taking the natural logarithm, we can express this prior as

$$log\left[{(\mathit{\psi }-{\mathbf{X}}_{\pi }{\mathit{\pi }}_0)}^{\odot 2}\right]=\mathit{\eta }+\mathit{\zeta },$$

where each element of the $(k_x\times 1)$ vector $\mathit{\zeta }$ are $iid$ draws from a $log{\chi }^2\left(1\right)$ distribution and ⊙ denotes the Hadamard or element-wise product. This representation is beneficial, as $\mathit{\eta }$ enters the right-hand side as essentially a vector of intercept parameters, thus opening the door to standard sampling possibilities. However, $\mathit{\zeta }$ is not Gaussian, which introduces an unfortunate computational complication.

To address this complication, we replace the log ${\chi }^2\left(1\right)$ with a highly accurate Gaussian mixture approximation. Specifically, we use a ten-component mixture approximation, as described in Omori et al. [14]:

$${\zeta }_i\approx \sum _{s=1}^{10}{p}_s\mathcal{N}(m_s,{\omega }_s^2).$$

The component probabilities $p_s$, means $m_s$, and variances ${\omega }_s^2$ are fixed (rather than estimated) and appear in Table 1 of Omori et al. [14]. A mixture approximation-based analysis thus proceeds by introducing a set of component indicators $\mathbf{s}={\left\{s_i\right\}}_{i=1}^{k_x}$ such that

$${\tilde{\zeta }}_i|s_i\sim \mathcal{N}(m_{s_i},{\omega }_{s_i}^2),\mathrm{Pr}(s_i=j)=p_j,j=1,2,\dots ,10.$$

(8)

Conditioned on the component indicators $\mathbf{s}$, we have

$$log\left({[\mathit{\psi }-{\mathbf{X}}_{\pi }{\mathit{\pi }}_0]}^{\odot 2}\right)={\mathbf{m}}_s+\mathit{\eta }+\tilde{\mathit{\zeta }},\tilde{\mathit{\zeta }}|\mathbf{s}\sim \mathcal{N}(\mathbf{0},{\mathbf{\Gamma }}_s),$$

(9)

where ${\mathbf{\Gamma }}_s\equiv \mathrm{diag}\left\{{\omega }_{s_i}^2\right\}.$

The prior for $\mathit{\eta }$ in (4) can be expressed as

$$\mathit{\eta }\sim \mathcal{N}\left({\mathit{\iota }}_{k_x}\mu ,{\mathit{G}}_{\rho }^{-1}\mathbf{\Omega }{\left({\mathit{G}}_{\rho }^{-1}\right)}^{\prime }\right).$$

(10)

Combining (9) and (10), one can show

$$\mathit{\eta }|\mathit{\gamma },\mathbf{s},\mathbf{y}\sim \mathcal{N}\left({\mathbf{D}}_{\eta }{\mathbf{d}}_{\eta },{\mathbf{D}}_{\eta }\right),$$

(11)

where

$${\mathbf{D}}_{\eta }={\left({\mathbf{\Gamma }}_s^{-1}+{\mathit{G}}_{\rho }^{\prime }{\mathbf{\Omega }}^{-1}{\mathit{G}}_{\rho }\right)}^{-1},{\mathbf{d}}_{\eta }={\mathbf{\Gamma }}_s^{-1}\left[log\left({\left[\mathit{\psi }-{\mathbf{X}}_{\pi }{\mathit{\pi }}_0\right]}^{\odot 2}\right)-{\mathbf{m}}_s\right]+{\mathit{G}}_{\rho }^{\prime }{\mathbf{\Omega }}^{-1}{\mathit{G}}_{\rho }{\iota }_{kx}\mu .$$

• Step 2: $\mathit{\pi }|{\mathit{\gamma }}_{-\pi },\mathit{\eta },\mathbf{y}$

When conducting this step, we choose to sample $\mathit{\psi }={\mathbf{H}}_{\pi }\mathit{\pi }$. Once $\mathit{\psi }$ is sampled, the desired smoothed regression coefficients can be calculated at each iteration by simply inverting the mapping, i.e., calculating $\mathit{\pi }={\mathbf{H}}_{\pi }^{-1}\mathit{\psi }.$

Our regression equation can be reparameterized in terms of $\mathit{\psi }$ by noting

$$\mathit{y}=\mathbf{D}\mathit{\theta }+\mathit{ϵ}=\mathbf{D}{\mathbf{H}}_{\pi }^{-1}{\mathbf{H}}_{\pi }\mathit{\pi }+\mathit{ϵ}=\tilde{\mathbf{D}}\mathit{\psi }+\mathit{ϵ},$$

where $\tilde{\mathbf{D}}\equiv \mathbf{D}{\mathbf{H}}_{\pi }^{-1}$.

It follows that the conditional posterior distribution of $\mathit{\psi }$ is Gaussian:

$$\mathit{\psi }|{\mathit{\gamma }}_{-\psi },\mathit{\eta },\mathbf{y}\sim \mathcal{N}({\mathbf{D}}_{\psi }{\mathbf{d}}_{\psi },{\mathbf{D}}_{\psi }),$$

(12)

where

$${\mathbf{D}}_{\psi }\equiv {\left({\tilde{\mathbf{D}}}^{\prime }\tilde{\mathbf{D}}/{\sigma }^2+{\mathbf{V}}_{\mathit{\eta }}^{-1}\right)}^{-1},{\mathbf{d}}_{\psi }\equiv {\tilde{\mathbf{D}}}^{\prime }\mathbf{y}/{\sigma }^2+{\mathbf{V}}_{\mathit{\eta }}^{-1}{\mathbf{X}}_{\pi }{\mathit{\pi }}_0.$$

• Step 3: ${\mathit{\pi }}_0|{\mathit{\gamma }}_{-{\pi }_0},\mathit{\eta },\mathbf{y}$

From (7), the conditional posterior distribution of the initial conditions ${\mathit{\pi }}_0$ is proportional to the product of the prior ${\mathbf{H}}_{\pi }\mathit{\pi }\sim \mathcal{N}({\mathbf{X}}_{\pi }{\mathit{\pi }}_0,{\mathbf{V}}_{\eta })$ and the prior for ${\mathit{\pi }}_0$. This implies

$${\mathit{\pi }}_0|{\mathit{\gamma }}_{-{\pi }_0},\mathit{\eta },\mathbf{y}\sim \mathcal{N}\left({\mathbf{D}}_{{\pi }_0}{\mathbf{d}}_{{\pi }_0},{\mathbf{D}}_{{\pi }_0}\right),$$

(13)

where

$${\mathbf{D}}_{{\pi }_0}={\left({\mathbf{X}}_{\pi }^{\prime }{\mathbf{V}}_{\mathit{\eta }}^{-1}{\mathbf{X}}_{\pi }+{\underline{\mathbf{V}}}_{{\pi }_0}^{-1}\right)}^{-1},{\mathbf{d}}_{{\pi }_0}={\mathbf{X}}_{\pi }^{\prime }{\mathbf{V}}_{\mathit{\eta }}^{-1}\mathit{\psi }+{\underline{\mathbf{V}}}_{{\pi }_0}^{-1}{\underline{\mathit{\mu }}}_{{\pi }_0}.$$

• Step 4: ${\sigma }^2|{\mathit{\gamma }}_{-{\sigma }^2},\mathit{\eta },\mathbf{y}$

From (7), it is evident that the conditional posterior distribution of the error variance ${\sigma }^2$ follows an inverse gamma distribution:

$${\sigma }^2|{\gamma }_{-{\sigma }^2},\mathit{\eta },\mathit{y}\sim IG\left({\underline{a}}_{\sigma }+{\displaystyle \frac{n}{2}},{\left[{\underline{b}}_{\sigma }^{-1}+{\displaystyle \frac{1}{2}}{(\mathit{y}-\mathbf{D}\mathit{\pi })}^{\prime }(\mathit{y}-\mathbf{D}\mathit{\pi })\right]}^{-1}\right).$$

• Step 5: $\mu |{\mathit{\gamma }}_{-\mu },\mathit{\eta },\mathbf{y}$

We begin by writing the prior for $\mathit{\eta }$ in (4) as

$${\mathit{G}}_{\rho }\mathit{\eta }=\mu \left[\begin{array}{c}1\\ (1-\rho )\\ ⋮\\ (1-\rho )\end{array}\right]+\left[\begin{array}{c}{u}_1\\ u_2\\ ⋮\\ u_{k_x}\end{array}\right],$$

where $\mathrm{V}\mathrm{a}\mathrm{r}\left(u_1\right)={\tau }^2/(1-{\rho }^2)$ and $\mathrm{V}\mathrm{a}\mathrm{r}\left(u_j\right)={\tau }^2\forall j\ge 2$. The right-hand-side vector ${[1(1-\rho )(1-\rho )\dots ]}^{\prime }$ acts as a covariate vector (with known values given $\rho$), and $\mu$ is its corresponding parameter. One can then show

$$\mu |{\gamma }_{-\mu },\mathit{\eta },\mathit{y}\sim \mathcal{N}\left(D_{\mu }{d}_{\mu },D_{\mu }\right)$$

where

$$D_{\mu }={\left[{\tau }^{-2}\left((1-{\rho }^2)+(k_x-1){(1-\rho )}^2\right)+{\underline{V}}_{\mu }\right]}^{-1},d_{\mu }={\mathit{g}}_{\rho }^{\prime }{\mathit{G}}_{\rho }\mathit{\eta }/{\tau }^2+{\underline{V}}_{\mu }^{-1}{\underline{\mu }}_0.$$

• Step 6: ${\tau }^2|{\mathit{\gamma }}_{-{\tau }^2},\mathit{\eta },\mathit{y}$

Defining

$${\tilde{\mathit{G}}}_{\rho }=\left[\begin{array}{ccccc}\sqrt{1-{\rho }^2}& 0& 0& \cdots & 0\\ 0& 1& 0& \cdots & 0\\ 0& 0& 1& \cdots & 0\\ ⋮& ⋮& ⋮& \ddots & ⋮\\ 0& 0& 0& \cdots & 1\end{array}\right]{\mathit{G}}_{\rho },$$

the prior in (4) implies

$${\tilde{\mathit{G}}}_{\rho }(\mathit{\eta }-{\mathit{\iota }}_{k_x}\mu )\sim \mathcal{N}\left(\mathbf{0},{\tau }^2{\mathit{I}}_{k_x}\right).$$

It follows that

$${\tau }^2|{\mathit{\gamma }}_{-{\tau }^2},\mathit{\eta },\mathit{y}\sim IG\left({\underline{a}}_{\tau }+{\displaystyle \frac{k_x}{2}},{\left[{\underline{b}}_{\tau }^{-1}+{\displaystyle \frac{1}{2}}{(\mathit{\eta }-{\mathit{\iota }}_{k_x}\mu )}^{\prime }{\tilde{\mathit{G}}}_{\rho }^{\prime }{\tilde{\mathit{G}}}_{\rho }(\mathit{\eta }-{\mathit{\iota }}_{k_x}\mu )\right]}^{-1}\right).$$

• Step 7: $\rho |{\mathit{\gamma }}_{-\rho },\mathit{\eta },\mathbf{y}$

For $\rho$, the conditional posterior distribution is proportional to

$$\rho |{\mathit{\gamma }}_{-\rho },\mathit{\eta },\mathit{y}\propto p\left(\rho \right)h\left(\rho \right)\prod _{i=2}^{k_x}\varphi \left({\eta }_i-\mu ;\rho ({\eta }_{i-1}-\mu ),{\tau }^2\right),$$

where the function $h\left(\rho \right)$ is defined as

$$h\left(\rho \right)={(1-{\rho }^2)}^{1/2}exp\left(-{\displaystyle \frac{1}{2{\tau }^2}}{({\eta }_1-\mu )}^2(1-{\rho }^2)\right).$$

• Case 1: Continuous Prior

In the continuous case, $p\left(\rho \right)$ is given in (6). The function $h\left(\rho \right)$ introduces a non-conjugacy, as the resulting conditional posterior does not take the form of a standard distribution. We will thus use a Metropolis–Hastings (M-H) step to conduct the sampling.

If we were to ignore the contribution of the initial condition through the term $h\left(\rho \right)$, the conditional posterior distribution would be proportional to

$$\rho |{\mathit{\gamma }}_{-\rho },\mathit{\eta },\mathbf{y}\sim T{N}_{(-1,1)}(D_{\rho }{d}_{\rho },D_{\rho })$$

(14)

where

$$D_{\rho }={\left(\sum _{i=2}^{k_x}{({\eta }_{i-1}-\mu )}^2/{\tau }^2+{\underline{V}}_{\rho }^{-1}\right)}^{-1},d_{\rho }=\sum _{i=2}^{k_x}({\eta }_i-\mu )({\eta }_{i-1}-\mu )/{\tau }^2+{\underline{V}}_{\rho }^{-1}{\underline{\mu }}_{\rho }.$$

Using (14) as a proposal distribution in an M-H step, we sample a candidate ${\rho }_c$ from (14) given that our chain is currently at $\rho ={\rho }_t$. The M-H acceptance probability thus reduces to $h\left({\rho }_c\right)/h\left({\rho }_t\right)$. Specifically, we draw $u^{*}\sim U(0,1)$ and accept the candidate ${\rho }_c$ provided

$$u^{*}<{\left({\displaystyle \frac{1-{\rho }_c^2}{1-{\rho }_t^2}}\right)}^{1/2}exp\left({\displaystyle \frac{1}{2{\tau }^2}}{({\eta }_1-\mu )}^2({\rho }_c^2-{\rho }_t^2)\right).$$

Otherwise, we set the new value of the chain, ${\rho }_{t+1}$, equal to ${\rho }_t$.

• Case 2: Discrete Prior

If a discrete prior for $\rho$ is specified, as in (5), the posterior is also discrete with support points ${\left\{{\underline{\rho }}_j\right\}}_{j=1}^J$. Furthermore, the posterior probabilities are given up to proportionality as

$$\mathrm{P}\mathrm{r}(\rho ={\underline{\rho }}_j|{\mathit{\gamma }}_{-\rho },\mathit{\eta },\mathit{y})\propto {\underline{p}}_{j}h\left({\underline{\rho }}_j\right)\prod _{i=2}^{k_x}\varphi \left({\eta }_i-\mu ;{\underline{\rho }}_j({\eta }_{i-1}-\mu ),{\tau }^2\right).$$

The expression on the right-hand side of the equation above can be computed for all $j=1,2,\dots ,J$ and then normalized by dividing by the sum of all such values. A draw from the corresponding discrete distribution can then be obtained via the method of inversion.

• Step 8: $\mathbf{s}|\mathit{\gamma },\mathit{\eta },\mathbf{y}$

The conditional posterior distribution of each component indicator is proportional to

$$\mathrm{P}\mathrm{r}(s_i=j|\mathit{\gamma },\mathit{\eta },\mathbf{y})\propto p_j{\omega }_j^{-1}\varphi \left({\displaystyle \frac{log\left({[{\psi }_i-{\mathbf{x}}_{i\pi }{\mathit{\pi }}_{0c}]}^2\right)-m_j-{\eta }_i}{{\omega }_j}}\right),j=1,2,\dots ,10,\mathrm{and}\forall i=1,2,\dots ,k_x,$$

(15)

with $\varphi (·)$ denoting the standard normal density function. For a given i, the above can be calculated for all $j=1,2,\dots ,10$ and then normalized to obtain a valid discrete distribution. The process is then repeated for all i.

4. Generated Data Experiments

In this section, we perform brief generated data experiments to investigate the performance of the stationary/estimated-$\rho$ adaptive smoothing procedure outlined in Section 2 and Section 3. We seek to illustrate the method’s ability to recover the shape of an unknown regression function and to contrast the behavior of smoothing parameters under this method with those obtained under other assumed processes, including the random walk.

To this end, we begin by generating 1000 observations from a regression model with interesting changes in curvature. Specifically, we first sample $x_i\stackrel{iid}{\sim }U(-2,1.5)$ with $U(a,b)$ denoting a uniform distribution on the interval $(a,b)$. To induce some degree of discreteness, consistent with our application of the following section, we round each $x_i$ value to the second decimal place. We then sample ${ϵ}_i\stackrel{iid}{\sim }\mathcal{N}(0,1)$ and generate $\mathit{y}$ as follows:

$$y_i=-1+0.5x_i+2exp\left(-100{(x_i+1)}^2\right)+\left({\displaystyle \frac{2}{\sqrt{(3/5)}{\pi }^{1/4}}}\right)exp\left(-{\displaystyle \frac{25x_i^2}{2}}\right)(1-25x_i^2)+\sigma {ϵ}_i,$$

with the error standard deviation $\sigma =0.25$. For our priors, we set ${\underline{\mathit{\mu }}}_{{\pi }_0}={[-2-2]}^{\prime }$, ${\underline{\mathit{V}}}_{{\pi }_0}=1.0\times {10}^5{\mathit{I}}_2$, $a_{\sigma }=3$, $b_{\sigma }=5/2$, $\underline{\mu }=-10$, ${\underline{V}}_{\mu }=16$, $a_{\tau }=3$, and $b_{\tau }=0.5$. For the prior for $\rho$, we consider the discrete prior variant, with $\rho \in \{0.5,0.51,0.52,\dots ,0.99\}.$ This support is selected to express preference for at least a moderate degree of positive persistence in the smoothing parameter process. Cases with negative $\rho$ may also be considered, although positive persistence is seemingly a natural position to take when modeling regression functions that are believed to be smooth. In experiments not reported here, cases with negative $\rho$ tended to produce immediate and large changes in $\eta$ values and difficult-to-interpret oscillatory smoothing patterns as one departs from the stationary mean $\mu$.

To construct the prior probabilities ${\underline{p}}_j$ for each element of this discrete vector, we first calculate the ordinate of the standard normal density at ${[({\underline{\rho }}_j-\mu )/\sigma ]}^2$ where $\mu =0.8$ and $\sigma =0.1$. We perform this for each of the discrete ${\underline{\rho }}_j$ cells. The resulting collection of values is then normalized by dividing by the sum of all such ordinates to produce a proper prior over this discrete vector of points. Of course, a variety of other methods for assigning/constructing the prior probabilities can also be employed.

Figure 1 provides a summary of the results from this generated data experiment. The top panel of Figure 1 provides a scatterplot of the raw data together with the stationary nonparametric estimates under the employed discrete prior for $\rho$. As shown in the figure, the method performs very well and accurately recovers the shape of and changes in the true regression function. The bottom panel of Figure 1 plots (in red) posterior means of the smoothing parameters $\mathit{\eta }$ across each $x^{*}$ value. To gain a sense of how these results might differ from, or be similar to, smoothing results from other approaches (like the random walk), we estimate the model again under two different assumptions regarding the evolution of $\mathit{\eta }$. In the first model, we simply fix $\rho =0.3$ throughout. In the second model, we consider the random walk specification. Parameters common across approaches are assigned the same priors, with hyperparameters of those priors given previously. Posterior means of the smoothing parameters/log volatilities $\mathit{\eta }$ are presented for each of these two additional specifications in the lower panel of Figure 1.

The lower panel reveals a few interesting patterns and highlights important differences across approaches. First, results under all three priors show evidence of adaptability: values of $\mathit{\eta }$ are relatively large where the function exhibits the most rapid changes in curvature. Conversely, when $f\left(x\right)$ locally behaves as a linear function, $\mathit{\eta }$ values are relatively small, shrinking results toward linearity over the corresponding region as a consequence. Second, we see differences in the degree of adjustment across approaches with different treatments of the AR parameter $\rho$. The smoothing parameters under the random walk model evolve slowly, exhibiting small local changes in $\mathit{\eta }$ values, while results with estimated-$\rho$ and those fixing $\rho =0.3$ adjust quickly by comparison. Near the functional spike at $x=-1$, for example, all approaches admit the largest value of $\mathit{\eta }$, while the $\rho =0.3$ results then decline sharply to either side, returning rather quickly to values near the stationary mean $\mu$. The approach with estimated-$\rho$ lies somewhere in between the random walk and $\rho =0.3$ cases: it resembles overall patterns produced by the random walk model, but adjusts the values of $\mathit{\eta }$ more quickly as one moves away from the peaks in the regression function where curvature changes are most pronounced.

Figure 2 documents how learning takes place in the stationary model regarding the AR coefficient $\rho$ under two different discrete priors. The left panel plots the prior (top left) for the analysis used to generate Figure 1 and the corresponding marginal posterior (bottom left). The prior employed is rather informative regarding the degree of persistence, but learning clearly occurs, as the posterior becomes left-skewed and shifts toward values of $\rho$ near one. In the right panels of Figure 2, we repeat the analysis, this time increasing the generating $\sigma$ value from 0.1 to 1, thus creating a nearly uniform prior for $\rho$ across the discrete cells from 0.5 to 0.99 (top right). The posterior (bottom right) again differs considerably from the prior, and places even more of its mass near the unit root, suggesting preference for the random walk specification (or a close approximation to it) for this experiment. Mean absolute error (MAE), for example (letting $f\left(x\right)$ denote the true value of the function in the experiment and $\widehat{f}\left(x\right)$ the estimated value (i.e., the posterior mean), we calculate the MAE as $k_x^{-1}{\sum }_{i=1}^{k_x}|\widehat{f}\left(x_i^{*}\right)-f\left(x_i^{*}\right)|$), was found to be 0.029 and 0.030 for the random walk and estimated-$\rho$ specifications, respectively, and 0.037 when fixing $\rho =0.3$. The movement of the posterior of $\rho$ toward the unit root is consistent with this overall preference for the random walk model in this experiment.

We consider an additional set of experiments, this time generating data from a linear model with $f\left(x\right)=-1+0.5x$. With the exception of $\rho$, we continue to employ the same priors as those just used in the nonlinear experiment. For $\rho$, we again employ a discrete prior, this time over the support points $\rho \in \{0.3,0.31,0.32,\dots ,0.99\}$, and with discrete prior probabilities ${\underline{p}}_j$ generated in a similar manner, with $\mu =0.8$ and $\sigma =0.2$. Results from this experiment are provided in Figure 3.

In the top left panel of Figure 3, we again see that the method performs admirably in recovering the simple linear regression function. In the bottom left panel, we compare posterior means of the smoothing parameters $\mathit{\eta }$ for both the random walk and estimated-$\rho$ cases. Here, we see that the random walk specification again slowly evolves, gradually moving toward smaller values of $\mathit{\eta }$, which further shrink the vector $\mathit{\pi }$ toward linearity. Values of $\mathit{\eta }$ are largest for small x in the random walk specification and then decline, owing to the influence of the $\mathcal{N}(-10,16)$ prior for the initial smoothing parameter ${\eta }_1$. This prior is chosen to match the prior employed for its counterpart, the stationary mean $\mu$, in the estimated-$\rho$ case. The relatively slowly adapting random walk specification takes some time to move away from the influence of this initial condition, whereas the estimated-$\rho$ model immediately jumps to the stationary mean $\mu$ and remains there across the support of x.

Posterior means of $\mathit{\eta }$ under the estimated-$\rho$ model are uniformly smaller than those of the random walk model and thus more strongly impose linearity—an appropriate and desirable outcome for this example. The right panels of Figure 3 provide additional results, as they plot prior (top) and posterior (bottom) probabilities of $\rho$ over its discrete support. Here, we again see that learning has taken place, as the marginal prior and posterior distributions are clearly different. Specifically, the posterior is observed to move toward interior $\rho$ values and to place little mass around $\rho \approx 1$, unlike our previous nonlinear example of Figure 1 where the posterior moved toward the unit root. Finally, we note that the MAE for the estimated-$\rho$ model was 0.005, while that of the random walk specification was 0.012. Taken together, we see that the linear and nonlinear experiments considered here provide examples where learning about $\rho$ clearly occurs. This learning leads to posterior shifts in opposite directions in our two experiments, although both movements are in directions leading to improved fits.

5. Application

We illustrate our methods using data from the National Longitudinal Survey of Youth (NLSY) 1979. The NLSY is a nationally representative sample of young men and women, all born between 1957 and 1964, containing a wealth of demographic, academic, and labor market data on its participants. We use these data to better understand how behavioral problems among adolescents affect their academic achievement, the latter measured through a series of test scores within the NLSY. Specifically, the nonparametric methods derived in Section 2 and Section 3 are employed to flexibly explore the relationship between behavioral problem severity and academic achievement. Before discussing the results of this investigation, we first describe the variables analyzed in more detail.

The NLSY contains scores from the Peabody Individual Achievement Test (PIAT) in both mathematics and reading. The specific variables we extract and separately analyze are the total standard score in mathematics (MATHZ2000) (these parenthetical designations give the variable names in the NLSY), total standard score in reading recognition (RECOGZ2000), and total standard score in reading comprehension (COMPZ2000). Test scores range from 65 to 135 for each of the three tests in our sample, with sample means (and standard deviations) equal to 106.8 (13.5), 109.0 (14.2), and 105.0 (13.5) for math, reading recognition, and reading comprehension, respectively.

The NLSY also contains a supplemental mother’s file, in which mothers of the NLSY participants are asked to respond to a series of questions regarding their children. Included among responses in the mother’s file are a set of 28 questions surrounding specific behaviors that their children may or may not have demonstrated over the preceding three months. These questions touch on characteristics such as antisocial behavior, anxiousness/depression, headstrongness, hyperactivity, immature dependency, and peer conflict/social withdrawal (see https://nlsinfo.org/content/cohorts/nlsy79-children/topical-guide/assessments/behavior-problems-index-bpi (accessed on February 2025) for additional details). In answering these questions, mothers categorize the stated behaviors as occurring “often”, “sometimes”, or “not” (true). An overall index (BPIZX2000) is then constructed in the NLSY (which we denote as the BPI) by aggregating assigned scores across questions, with higher values of the index indicative of higher degrees of behavioral issues. In our final sample, the BPI ranges from a low of 74 to a high of 145, creating $k_x=72$ distinct values when constructing $\mathbf{D}$ as in Section 2. Of these 72 distinct values, nine cells are not represented in the sample, three contain exactly one observation, and no cells contain more than 44 observations. When cells are not empty, the average number of observations associated with each distinct BPI value is approximately 16.3. A seeming advantage of our approach is that our smoothness prior allows us to handle cells that are either infrequently represented or not represented at all; the prior borrows information from neighboring outcomes in forming posterior predictions at those values. Our final sample analyzed consists of $n=1076$ observations, and each test score outcome is considered in a separate analysis. For our priors, we employ the continuous prior for $\rho$, setting ${\underline{\mu }}_{\rho }=0.8$ and ${\underline{V}}_{\rho }=0.{15}^2$. As discussed in Section 3, posterior simulation thus employs a Metropolis–Hastings step for sampling $\rho$, and associated acceptance rates were very high, equaling 0.91, 0.91, and 0.89 for mathematics, reading recognition, and reading comprehension, respectively. For the remaining hyperparameters, we set $a_{\sigma }=3$, $b_{\sigma }=0.005$, ${\underline{\mu }}_0=-3$, ${\underline{V}}_{{\mu }_0}=4$, $a_{\tau }=3$, and $b_{\tau }=0.5$. Each posterior simulator is run for 82,500 iterations, with the first 2500 discarded as the burn-in period.

The results of our analysis are summarized in Figure 4. The upper left panel of the figure provides plots of the nonparametric conditional mean estimates $\widehat{E\left(\mathrm{Score}\right|\mathrm{BPI})}=E\left(\mathit{\pi }\right|\mathit{y})$ for all three tests (solid) along with estimates from simple linear models (dashed). The remaining three panels of the figure provide more detailed results for each test score outcome. These panels plot nonparametric estimates (black, solid), linear estimates (black, dashed), posterior regions associated with the nonparametric estimates (light blue, shaded) (these are calculated as the posterior mean of each element of $\mathit{\pi }$ ± 1.96 times the corresponding posterior standard deviation), and dummy variable coefficient estimates (red, dashed). The latter are obtained from OLS regressions of the given test score on a series of indicator variables for all BPI scores observed in the sample.

The upper left panel of the figure reveals several interesting results. First, the nonparametric estimates are suggestive of a pattern of nonlinearity that is consistent across all tests. Specifically, each conditional mean function is somewhat flat, though declining, for low values of the BPI, and then they turn more sharply negative once the BPI exceeds a value of approximately 90. Despite this near initial flatness, posterior means are found to be monotonically decreasing in the BPI across the full BPI support for all of our tests. This clearly suggests that behavioral issues have a globally negative effect on student achievement. When comparing the nonparametric results against linear estimates, another consistent pattern emerges: linear point estimates tend to overstate expected achievement at both tails of the BPI distribution, and this is particularly true at high values of the BPI for the reading recognition test score. Though not reported in the figure, we also note that the marginal posterior distributions of the autoregressive coefficient $\rho$ are similar across tests, with posterior means (and standard deviations) equal to 0.78 (0.132), 0.78 (0.134), and 0.79 (0.139) for mathematics, reading recognition, and reading comprehension, respectively.

The remaining three panels of the figure reveal the value of smoothing for this application. Dummy variable coefficient estimates—which may be interpreted as the starting point for the smoothed analysis—are highly variable, making it difficult to clearly identify overall patterns in the data. The nonparametric estimates clearly cut through the middle of this erratic collection and produce results that are easily interpretable by comparison.

Figure 4 also reveals that estimates from the linear specifications do fall within our nonparametric posterior regions for all tests, as there is considerable estimation uncertainty at the right tail of the BPI distribution. We therefore consider an additional model, guided by the exploratory nonparametric analysis, to further investigate the potential existence of nonlinearities in the test score/BPI relationships. Specifically, we estimate a linear spline with a single unknown changepoint. Such an analysis will allow us to determine the most likely location for a change in curvature (slope) if one is present, along with values of the slopes themselves both before and after the change. The model we take to the data is of the form

$$y_i={\beta }_0+{\beta }_{1}BP{I}_i+{\beta }_2{(BP{I}_i-\lambda )}_{+}+{ϵ}_i,$$

(16)

for each of our three test score outcomes. In Equation (16), $\lambda$ denotes our single changepoint, the notation $x_{+}\equiv max\{x,0\}$ and ${ϵ}_i\stackrel{iid}{\sim }\mathcal{N}(0,{\sigma }^2)$. It is clear that the spline function permits a change in slope at the value $\lambda$: the slope of our line is ${\beta }_1$ when the BPI $\le \lambda$ and ${\beta }_1+{\beta }_2$ when the BPI $>\lambda$. We also place a uniform prior across integer-valued $\lambda$ in the set $\{79,80,81,\dots ,130\}$, thus allowing at least a modest number of observations in each of the two slope regimes. For our priors, we choose

$$\left[\begin{array}{c}{\beta }_0\\ {\beta }_1\\ {\beta }_2\end{array}\right]\sim \mathcal{N}\left(\left[\begin{array}{c}100\\ 0\\ 0\end{array}\right],\left[\begin{array}{ccc}100& 0& 0\\ 0& 100& 0\\ 0& 0& 100\end{array}\right]\right)\mathrm{a}\mathrm{n}\mathrm{d}{\sigma }^2\sim IG(3,0.005).$$

The Gibbs sampler is employed to fit the model under these priors, running each posterior simulator for 110,000 iterations and discarding the first 10,000 as the burn-in period. Chan et al [11], Exercise 12.10, provide detailed steps for posterior simulation in a model of this form, and we follow those steps here.

If our earlier interpretations of the initial “flatness” of the regression functions followed by a more precipitous decline are correct, we should see those patterns echoed in the changepoint specification. Specifically, we expect to find (a) a revision of our uniform prior for the changepoint $\lambda$ to a posterior for $\lambda$ with a mode near BPI = 90 and (b) evidence that the slope of our regression function is more negative after the changepoint then before, suggesting that increments to the BPI have more substantial negative impacts on student achievement as one moves into the right tail of the BPI distribution.

Our estimation results provide confirmation of these patterns. Figure 5 plots posterior frequencies of the changepoint $\lambda$ for each of our three test score outcomes. Interestingly, the posterior mode of the changepoint in each case is exactly 91, consistent with Figure 4, which shows a relatively flat function until the BPI ≈ 90, followed by a negative decline.

Table 1 provides posterior means and standard deviations for the regression coefficients $\mathit{\beta }$ and variance parameter ${\sigma }^2$ in the changepoint model. As shown in the table, ${\beta }_1$—which is interpreted as the pre-changepoint slope—has a posterior mean near zero and a correspondingly large posterior standard deviation. This, again, is consistent with an initial period of relative flatness in the relationship between the BPI and test scores. The regression coefficient ${\beta }_2$, however, which is interpreted as the change in slope after $\lambda$, has consistently negative posterior means across test scores and places most of its mass over negative values: the posterior probability that ${\beta }_2<0$ is 0.93, 0.98, and 0.92 across math, reading recognition, and reading comprehension scores, respectively. These results provide additional confirmation that further movements into the right tail of the behavioral problems index distribution produce the largest negative impacts on student achievement.

We conclude with a comparison of our Bayesian nonparametric results and those obtained via frequentist kernel regression. Specifically, we additionally employ the Nadaraya–Watson (Nadaraya [18] and Watson [19]) or constant kernel estimator to estimate our regression relationships nonparametrically:

$$\widehat{E\left(Score\right|BPI=BP{I}_0)}={\displaystyle \frac{{\sum }_{i=1}^{n}Scor{e}_{i}K\left(\frac{BP{I}_i-BP{I}_0}{h}\right)}{{\sum }_{i=1}^{n}K\left(\frac{BP{I}_i-BP{I}_0}{h}\right)}},$$

where a Gaussian kernel K is selected and the bandwidth/global smoothing parameter h is set equal to five. The results are presented in Figure 6, with analysis again conducted separately for each test score outcome.

Table 1. Coefficient posterior means (and standard deviations below) from changepoint model.

	Math	Reading Rec.	Reading Comp.
${\beta }_0$	108.08	108.62	106.16
	(8.72)	(9.20)	(9.02)
${\beta }_1$	0.008	0.029	0.014
	(0.106)	(0.113)	(0.110)
${\beta }_2$	−0.217	−0.323	−0.204
	(0.182)	(0.183)	(0.197)
${\sigma }^2$	178.27	195.84	176.92
	(7.88)	(8.65)	(7.82)

Table 1. Coefficient posterior means (and standard deviations below) from changepoint model.

	Math	Reading Rec.	Reading Comp.
${\beta }_0$	108.08	108.62	106.16
	(8.72)	(9.20)	(9.02)
${\beta }_1$	0.008	0.029	0.014
	(0.106)	(0.113)	(0.110)
${\beta }_2$	−0.217	−0.323	−0.204
	(0.182)	(0.183)	(0.197)
${\sigma }^2$	178.27	195.84	176.92
	(7.88)	(8.65)	(7.82)

As shown in the figure, point estimates from the Bayesian and Nadaraya–Watson approaches are similar, but depart at the right tail of the BPI distribution, particularly for the mathematics and reading comprehension test score outcomes. These departures both illustrate the potential value of adaptive smoothing and reveal limitations when using a constant global smoothing parameter. In our sample, only 1.46% ($n=15$) of observations contain a BPI in excess of 130, and thus considerable uncertainty exists when estimating the function over this region. A constant bandwidth $h=5$ is chosen within our kernel regression to suitably smooth the functions over small/moderate BPI values, while this same level of smoothing also implies that just a few noisy data points are used to estimate the functions at the largest BPI values. This often implies a rather erratic (and seemingly implausible) kernel-based point estimate when the BPI > 130. In contrast, the Bayesian adaptive procedure imposes a higher degree of smoothing over this region and leads to a continuation of the overall downward trend present in the figures for smaller values of the BPI.

6. Conclusions

A new scheme for nonparametric Bayesian estimations of regression functions was presented. An efficient Gibbs sampling algorithm was developed that involves sampling from standard distributions, permits local adaptability in the degree of smoothing, and imposes stationarity in the smoothing parameter process. The algorithm was shown to perform well in generated data experiments, and its value was illustrated in an application relating a behavioral problems index (BPI) to student achievement. Analysis of those data showed that the test score/BPI relationship is relatively flat/slightly negative when behavioral problems are low, while marginal increments to the BPI have a clearly negative effect on student achievement in math and reading as one moves into the right tail of the BPI distribution.