Model-Based and Model-Free Point Prediction Algorithms for Locally Stationary Random Fields

Abstract

The Model-Free Prediction Principle has been successfully applied to general regression problems, as well as problems involving stationary and locally stationary time series. In this paper, we demonstrate how Model-Free Prediction can be applied to handle random fields that are only locally stationary such as pixel values over an image or satellite data observed on an ocean surface, i.e., they can be assumed to be stationary only across a limited part over their entire region of definition. We construct novel one-step-ahead Model-Based and Model-Free point predictors and compare their performance using synthetic data as well as images from the CIFAR-10 dataset. In the latter case, we demonstrate that our best Model-Free point prediction results outperform those obtained using Model-Based prediction.

1. Introduction

Consider a real-valued random field dataset $\{Y_{\underline{t}},\underline{t}\in Z^2\}$ defined over a 2D index-set D, e.g., pixel values over an image or satellite data observed on an ocean surface. It may be unrealistic to assume that the stochastic structure of such a random field $Y_{\underline{t}}$ has stayed invariant over the entire region of definition D; hence, we cannot assume that $\left\{Y_{\underline{t}}\right\}$ is stationary. Therefore it is more realistic to assume a slowly-changing stochastic structure, i.e.,  a locally stationary model. The theory of locally stationary time series and parametric and nonparametric methods for their estimation have been covered extensively in the literature including references [1,2,3,4,5,6,7,8]. In [9] we propose Model-Based and Model-Free algorithms for point prediction and prediction intervals of locally stationary time series and demonstrate their applications for both synthetic and real-life datasets. Our work in this paper extends this framework to point prediction over locally stationary random fields with applications involving synthetic and real-life image data.

In the context of random fields two principal modeling approaches are usually followed in order to perform estimation. In the first case for fields of study such as econometrics and ecology where the sampling points can be irregular the random field data {$Y_{\underline{s}},\underline{s}\in S$} is defined over a continuous subset S of $R^d$. Modeling strategies for such non-uniformly spaced spatial data have been discussed in [10,11]. Kernel estimation for locally stationary random fields defined over such irregularly spaced locations has been proposed in [12,13] while autoregressive estimation of similarly defined locally stationary random fields has been proposed in [14]. In the other style of modeling the random field {$Y_{\underline{t}},\underline{t}\in S$} is defined over a regularly spaced grid $S\subset Z^d$. Examples of such applications arise in fields of study such as image processing and radiography. Two dimensional (2D) autoregressive models for such random fields covering various regions of support have been proposed in [15]. Autoregressive estimation of such data regularly spaced on a lattice has also been discussed in [16]. Applications of 2D autoregressive models for analysis and synthesis of textural images are shown in [17]. Local Linear based nonparametric estimators of such random fields and their theoretical properties have been discussed in [18,19]. In this paper, we assume a locally stationary model for random fields $Y_{\underline{t}}\in R$ defined over $\underline{t}\in S$ where $S\subset Z^d$, $d=2$. Given data $Y_{{\underline{t}}_1},Y_{{\underline{t}}_2},\dots ,Y_{{\underline{t}}_n}$, our objective is to perform point prediction for a future unobserved data point $Y_{{\underline{t}}_{n+1}}$. Here, ${\underline{t}}_1,{\underline{t}}_2,\dots ,{\underline{t}}_n,{\underline{t}}_{n+1}\in Z^2$ denote the coordinates of the random field over the 2D index set D and the notion of a future datapoint over a coordinate of a random field for purposes of predictive inference over $\underline{t}\in Z^2$ is defined in Section 2.

The usual approach for dealing with locally stationary series is to assume that the data can be decomposed as the sum of three components:

$$\mu \left(\underline{t}\right)+S_{\underline{t}}+W_{\underline{t}}$$

where $\mu \left(\underline{t}\right)$ is a deterministic trend function, $S_{\underline{t}}$ is a seasonal (periodic) series, and $\left\{W_{\underline{t}}\right\}$ is (strictly) stationary with mean zero. This type of decomposition has been proposed for time series [20] and can also be used for the decomposition of locally stationary random field data. The seasonal (periodic) component, be it random or deterministic, can be easily estimated and removed and having completed that, the ‘classical’ decomposition simplifies to the following model with the additive trend, i.e.,

$$Y_{\underline{t}}=\mu \left(\underline{t}\right)+W_{\underline{t}}$$

(1)

which can be generalized to accommodate a coordinate-changing variance as well, i.e.,

$$Y_{\underline{t}}=\mu \left(\underline{t}\right)+\sigma \left(\underline{t}\right)W_{\underline{t}}.$$

(2)

In both above models, the series $\left\{W_{\underline{t}}\right\}$ is assumed to be (strictly) stationary, weakly dependent, e.g., strong mixing, and satisfying $E\left(W_{\underline{t}}\right)=0$; in model (2), it is also assumed that $var\left(W_{\underline{t}}\right)=1$. The deterministic functions $\mu (·)$ and $\sigma (·)$ are unknown and can be assumed to belong to a class of functions that is either finite-dimensional (parametric) or not (nonparametric). In this paper, we focus on the nonparametric case and assume that $\mu (·)$ and $\sigma (·)$ have some degree of smoothness i.e., change smoothly (and slowly) with $\underline{t}$.

Models (1) and (2) can be used to capture the first two moments of the locally stationary random field; however it may be the case that the skewness and/or kurtosis of $Y_{\underline{t}}$ changes with $\underline{t}$. In addition it may also be the case that the correlation Corr$(Y_{{\underline{t}}_j},Y_{{\underline{t}}_{j+1}})$ changes smoothly (and slowly) with ${\underline{t}}_j\in Z^2$. To address this more general case we propose a methodology for point prediction of locally stationary random fields that does not rely on simple additive models such as (1) and (2). This is accomplished by using the Model-Free Prediction Principle of [21,22]. The key to Model-Free inference is to be able to construct an invertible transformation $H_n:{\underline{Y}}_{{\underline{t}}_n}\mapsto {\underline{ϵ}}_n$ where ${\underline{Y}}_{{\underline{t}}_n}=(Y_{{\underline{t}}_1},Y_{{\underline{t}}_2},\dots ,Y_{{\underline{t}}_n})$ denotes the random field data under consideration and ${\underline{ϵ}}_n={({ϵ}_1,\dots ,{ϵ}_n)}^{\prime }$ is a random vector with i.i.d. components.

The rest of the paper is arranged as follows. In Section 2, we set up the framework for defining the causality of random fields in order to enable us to perform point prediction. In Section 3, we visit the problem of Model-Based inference and develop a point prediction methodology for locally stationary random fields. In Section 4, we construct the framework for point prediction of locally stationary random fields using Model-Free inference. In Section 5, we describe how cross-validation can be used to determine the optimal bandwidths for both Model-Based and Model-Free inference. Finally, in Section 6, using finite sample experiments we compare the two novel approaches namely Model-Based of Section 3 and Model-Free of Section 4 using synthetic and real-life data.

2. Causality of Random Fields

Given the random field observations $Y_{{\underline{t}}_1},\dots ,Y_{{\underline{t}}_n}$ our goal is to perform predictive inference for the “next” unknown datapoint $Y_{{\underline{t}}_{n+1}}$. In this context, a definition of causality is necessary to specify the random field coordinate ${\underline{t}}_{n+1}$ where predictive inference will be performed. For this purpose, we adopt the framework proposed in [15] and consider random fields discussed in this paper to be defined over a subset of the nonsymmetric half-plane (NSHP) denoted as $H_{\infty }$. Figure 1 shows an NSHP centered at (0, 0). The NSHP can also be centered at any other point $\underline{t}$ as follows:

$$NSHP\left(\underline{t}\right)=\underline{t}+\underline{s}\forall \underline{s}\in NSHP(0,0)$$

(3)

Such nonsymmetric half-planes have been used previously for specifying causal 2D AR models [15]. In such cases, a causal 2D AR model with $H_p\subset H_{\infty }$ can be defined as below in Equation (4) where the set $H_p$ is termed as the region of support (ROS) of the 2D AR model. Here $H_p=\left\{(j,k)\right|j=1,2,\dots ,p$ and $k=0,\pm 1,\dots ,\pm p\}\cup \{(0,k)|k=1,2,\dots ,p\}$ and $v_{t_1,t_2}$ is a 2D white noise process with mean 0 and variance ${\sigma }^2>0$.

$$Y_{t_1,t_2}=\sum _{(j,k)\in H_p}{\beta }_{j,k}{Y}_{t_1-j,t_2-k}+v_{t_1,t_2}$$

(4)

Based on [23], a 2D AR process with ROS S is causal if there exists a subset C of $Z^2$ satisfying the following conditions:

• The set C consists of 2 rays emanating from the origin and the points lie between the rays;
• The angle between the 2 rays is strictly less than 180 degrees;
• $S\subset C$

In this case, since $H_p\subset H_{\infty }$ satisfies these conditions the 2D AR process denoted by (4) is causal. We use this framework to describe a causal random field defined over the NSHP and perform predictive inference on the same. Given this, our setup for point prediction of random fields is described as below.

Consider random field data $\{Y_{\underline{t}},\underline{t}\in E\}$ where E can be any finite subset of $Z^2$ for, e.g., $E_{\underline{n}}=\{\underline{t}\in Z^2$ with $\underline{n}=(n_1,n_2)\}$. Our goal is predictive inference at $\underline{t}=(t_1,t_2)$ where $0<t_1<n_1\&0<t_2<n_2$. This “future” value $Y_{t_1,t_2}$ is determined using data defined over the region as shown in Figure 2:

$$E_{\underline{t},\underline{n}}=NSHP\left(\underline{t}\right)\cap E_{\underline{n}}$$

Both Model-Based and Model-Free causal inference for $Y_{t_1,t_2}$ are performed using the data specified over this region $E_{\underline{t},\underline{n}}$. We consider predictive inference at $Y_{\underline{t}}=Y_{t_1,t_2}$ given the data ($Y_{\underline{s}}\mid \underline{s}\prec \underline{t}\&\underline{s}\in E_{\underline{t},\underline{n}}$) where the symbol ≺ denotes lexicographical ordering on the region of support of the random field, i.e., $(a_k,b_k)\prec (a_{k+1},b_{k+1})$ if and only if either $a_k<a_{k+1}$ or $(a_k=a_{k+1}$ and $b_k<b_{k+1})$ [15]. In the subsequent discussion the lexicographically ordered “past” data $Y_{\underline{s}}$ will be denoted as $Y_{{\underline{t}}_1},Y_{{\underline{t}}_2},\dots ,Y_{{\underline{t}}_n}$ and point prediction will be performed at $Y_{\underline{t}}=Y_{{\underline{t}}_{n+1}}$.

3. Model-Based Point Prediction

We adopt the time-changing mean and variance model as given by Equation (2). The $L_2$–optimal predictor of $Y_{{\underline{t}}_{n+1}}$ given the data $Y_{\underline{s}}=$${\underline{Y}}_{{\underline{t}}_n}={(Y_{{\underline{t}}_1},\dots ,Y_{{\underline{t}}_n})}^{\prime }$ is the conditional expectation $E\left(Y_{{\underline{t}}_{n+1}}\right|{\underline{Y}}_{{\underline{t}}_n})$. Using model (2) and assuming that ${\underline{W}}_{{\underline{t}}_n}$ is weakly dependent it can be shown that [9]:

$$E\left(Y_{{\underline{t}}_{n+1}}\right|{\underline{Y}}_{{\underline{t}}_n})=\mu \left({\underline{t}}_{n+1}\right)+\sigma \left({\underline{t}}_{n+1}\right)E\left(W_{{\underline{t}}_{n+1}}\right|{\underline{W}}_{{\underline{t}}_n}).$$

(5)

From the above equation, we can see that for Model-Based point prediction we need to estimate the conditional expectation $E\left(W_{{\underline{t}}_{n+1}}\right|{\underline{W}}_{{\underline{t}}_n})$ as well as the coordinate changing trend and variance, i.e., $\mu \left({\underline{t}}_{n+1}\right)$ and $\sigma \left({\underline{t}}_{n+1}\right)$.

Estimating the conditional expectation: This is performed by fitting a (causal) AR($p,q$) model to the data ${\underline{W}}_{{\underline{t}}_n}=(W_{{\underline{t}}_1},\dots ,W_{{\underline{t}}_n}$) with $p,q$ chosen by minimizing AIC, BIC or a related criterion as described in [15]. Using this framework involves estimating the coefficients of the following 2D AR model defined over a ROS $H_p$ as described in Section 2:

$$W_{r,s}=\sum _{(j,k)\in H_p}{\beta }_{j,k}{W}_{r-j,s-k}+v_{r,s}$$

(6)

Here, $v_{r,s}$ is a 2D white noise process, i.e., an uncorrelated sequence, with mean 0 and variance ${\sigma }^2>0$. For our point prediction problem setting this implies that

$$\overline{E}\left(W_{{\underline{t}}_{n+1}}\right|{\underline{W}}_{{\underline{t}}_n})=\sum _{(j,k)\in H_p}{\beta }_{j,k}{W}_{t_{n_1}-j,t_{n_2}-k}$$

(7)

Estimating the trend and variance: This can be performed by using kernel smoothing [24,25,26] using 2D kernels, i.e., Nadaraya–Watson (NW) estimation. In addition, since predicting $Y_{{\underline{t}}_{n+1}}$ is essentially a boundary problem it is also possible to use local linear fitting, which has been reported to have smaller bias than kernel smoothing for such estimation problems [26,27,28]. For time series problems, {$Y_t,t\in Z$} local linear nonparametric estimation can approximate the trend locally by a straight line whereas for the case of random fields $\{Y_{\underline{t}},\underline{t}\in Z^2\}$ discussed in this paper local linear estimation can be used to approximate the trend locally with a plane.

In order to estimate $E\left(W_{{\underline{t}}_{n+1}}\right|{\underline{W}}_{{\underline{t}}_n})$, the stationary data $W_{{\underline{t}}_1},\dots ,W_{{\underline{t}}_n}$ needs to be estimated. In this case, $W_{\underline{t}}$ has to be calculated in a one-sided manner for all points including those at the center of the dataset else the obtained values ${\widehat{W}}_{{\underline{t}}_1},\dots ,{\widehat{W}}_{{\underline{t}}_n}$ will not be stationary, which leads to incorrect estimation of the conditional expectation of $W_{{\underline{t}}_{n+1}}$. The one-sided kernel smoothed and local linear estimators used for calculating ${\widehat{W}}_{{\underline{t}}_1},\dots ,{\widehat{W}}_{{\underline{t}}_n}$ can be defined in two ways as shown in the equations below on NW–Regular, NW–Predictive, LL–Regular and LL–Predictive fitting. Here, the bandwidth parameter b is assumed to satisfy

$$b\to \infty asn\to \infty \mathrm{but}b/n\to 0$$

(8)

We will assume throughout that $K(·)$ is a nonnegative, symmetric 2D Gaussian kernel function for which the diagonal values are set to the bandwidth b and the off-diagonal terms are set to 0. Random field data is denoted as $Y_{{\underline{t}}_1},\dots ,Y_{{\underline{t}}_k},\dots Y_{{\underline{t}}_n}$.

• NW–Regular fitting: Let ${\underline{t}}_k\in [{\underline{t}}_1,{\underline{t}}_n]$, and define

  $$\begin{array}{c}\hfill \widehat{\mu }\left({\underline{t}}_k\right)=\sum _{i=1}^k{Y}_{{\underline{t}}_i}\widehat{K}\left(\frac{{\underline{t}}_k-{\underline{t}}_i}{b}\right)\mathrm{and}\\ \hfill \widehat{M}\left({\underline{t}}_k\right)=\sum _{i=1}^k{Y}_{{\underline{t}}_i}^2\widehat{K}\left(\frac{{\underline{t}}_k-{\underline{t}}_i}{b}\right)\end{array}$$

  (9)

  where

  $$\begin{array}{c}\hfill \widehat{\sigma }\left({\underline{t}}_k\right)=\sqrt{{\widehat{M}}_{{\underline{t}}_k}-\widehat{\mu }{\left({\underline{t}}_k\right)}^2}\mathrm{and}\\ \hfill \widehat{K}\left(\frac{{\underline{t}}_k-{\underline{t}}_i}{b}\right)=\frac{K\left(\frac{{\underline{t}}_k-{\underline{t}}_i}{b}\right)}{{\sum }_{j=1}^{k}K\left(\frac{{\underline{t}}_k-{\underline{t}}_j}{b}\right)}.\end{array}$$

  (10)
  Using $\widehat{\mu }\left({\underline{t}}_k\right)$ and $\widehat{\sigma }\left({\underline{t}}_k\right)$ we can now define the fitted residuals by

  $${\widehat{W}}_{{\underline{t}}_k}=\frac{Y_{{\underline{t}}_k}-\widehat{\mu }\left({\underline{t}}_k\right)}{\widehat{\sigma }\left({\underline{t}}_k\right)}\mathrm{for}{\underline{t}}_k={\underline{t}}_1,\dots ,{\underline{t}}_n.$$

  (11)
• NW–Predictive fitting:

  $$\begin{array}{c}\hfill \tilde{\mu }\left({\underline{t}}_k\right)=\sum _{i=1}^{k-1}{Y}_{{\underline{t}}_i}\tilde{K}\left(\frac{{\underline{t}}_k-{\underline{t}}_i}{b}\right)\mathrm{and}\\ \hfill \tilde{M}\left({\underline{t}}_k\right)=\sum _{i=1}^{k-1}{Y}_{{\underline{t}}_i}^2\tilde{K}\left(\frac{{\underline{t}}_k-{\underline{t}}_i}{b}\right)\end{array}$$

  (12)

  where

  $$\begin{array}{c}\hfill \tilde{\sigma }\left({\underline{t}}_k\right)=\sqrt{{\tilde{M}}_{{\underline{t}}_k}-\tilde{\mu }{\left({\underline{t}}_k\right)}^2}\mathrm{and}\\ \hfill \tilde{K}\left(\frac{{\underline{t}}_k-{\underline{t}}_i}{b}\right)=\frac{K\left(\frac{{\underline{t}}_k-{\underline{t}}_i}{b}\right)}{{\sum }_{j=1}^{k-1}K\left(\frac{{\underline{t}}_k-{\underline{t}}_j}{b}\right)}.\end{array}$$

  (13)
  Using $\tilde{\mu }\left({\underline{t}}_k\right)$ and $\tilde{\sigma }\left({\underline{t}}_k\right)$ we can now define the predictive residuals by

  $${\tilde{W}}_{{\underline{t}}_k}=\frac{Y_{{\underline{t}}_k}-\tilde{\mu }\left({\underline{t}}_k\right)}{\tilde{\sigma }\left({\underline{t}}_k\right)}\mathrm{for}{\underline{t}}_k={\underline{t}}_1,\dots ,{\underline{t}}_n.$$

  (14)

Similarly, the one-sided local linear (LL) fitting estimators of $\mu \left({\underline{t}}_k\right)$ and $\sigma \left({\underline{t}}_k\right)$ can be defined in two ways.

• LL–Regular fitting: Let ${\underline{t}}_k\in [{\underline{t}}_1,{\underline{t}}_n]$, and define

  $$\begin{array}{c}\hfill \widehat{\mu }\left({\underline{t}}_k\right)=\frac{{\sum }_{j=1}^k{w}_j{Y}_{{\underline{t}}_j}}{{\sum }_{j=1}^k{w}_j+n^{-2}}\mathrm{and}\\ \hfill \widehat{M}\left({\underline{t}}_k\right)=\frac{{\sum }_{j=1}^k{w}_j{Y}_{{\underline{t}}_j}^2}{{\sum }_{j=1}^k{w}_j+n^{-2}}\end{array}$$

  (15)
  Denoting

  $$\underline{a}=(a_1,a_2)=({\underline{t}}_j-{\underline{t}}_k)$$

  (16)

  $$s_{t1,1}=\sum _{j=1}^{k}K\left(\frac{{\underline{t}}_j-{\underline{t}}_k}{b}\right)a_1$$

  (17)

  $$s_{t2,1}=\sum _{j=1}^{k}K\left(\frac{{\underline{t}}_j-{\underline{t}}_k}{b}\right)a_2$$

  (18)

  $$s_{t1,2}=\sum _{j=1}^{k}K\left(\frac{{\underline{t}}_j-{\underline{t}}_k}{b}\right)a_1^2$$

  (19)

  $$s_{t2,2}=\sum _{j=1}^{k}K\left(\frac{{\underline{t}}_j-{\underline{t}}_k}{b}\right)a_2^2$$

  (20)

  $$s_{t1,t2}=\sum _{j=1}^{k}K\left(\frac{{\underline{t}}_j-{\underline{t}}_k}{b}\right)a_1{a}_2$$

  (21)

  $$\begin{array}{c}\hfill w_j=K\left(\frac{{\underline{t}}_j-{\underline{t}}_k}{b}\right)\\ \hfill \{s_{t1,2}{s}_{t2,2}-s_{t1,t2}^2\\ \hfill -a_1(s_{t1,1}{s}_{t2,2}-s_{t2,1}{s}_{t1,t2})\\ \hfill +a_2(s_{t1,1}{s}_{t1,t2}-s_{t1,2}{s}_{t2,1})\}\end{array}$$

  (22)
  The term $n^{-2}$ in Equation (15) is just to ensure the denominator is not zero; see [29]. Equation (10) then yields $\widehat{\sigma }\left({\underline{t}}_k\right)$, and Equation (11) yields ${\widehat{W}}_{{\underline{t}}_k}$.
• LL–Predictive fitting:

  $$\begin{array}{c}\hfill \tilde{\mu }\left({\underline{t}}_k\right)=\frac{{\sum }_{j=1}^{k-1}{w}_j{Y}_{{\underline{t}}_j}}{{\sum }_{j=1}^{k-1}{w}_j+n^{-2}}\mathrm{and}\\ \hfill \tilde{M}\left({\underline{t}}_k\right)=\frac{{\sum }_{j=1}^{k-1}{w}_j{Y}_{{\underline{t}}_j}^2}{{\sum }_{j=1}^{k-1}{w}_j+n^{-2}}\end{array}$$

  (23)

  where

  $$\underline{a}=(a_1,a_2)=({\underline{t}}_j-{\underline{t}}_k)$$

  (24)

  $$s_{t1,1}=\sum _{j=1}^{k-1}K\left(\frac{{\underline{t}}_j-{\underline{t}}_k}{b}\right)a_1$$

  (25)

  $$s_{t2,1}=\sum _{j=1}^{k-1}K\left(\frac{{\underline{t}}_j-{\underline{t}}_k}{b}\right)a_2$$

  (26)

  $$s_{t1,2}=\sum _{j=1}^{k-1}K\left(\frac{{\underline{t}}_j-{\underline{t}}_k}{b}\right)a_1^2$$

  (27)

  $$s_{t2,2}=\sum _{j=1}^{k-1}K\left(\frac{{\underline{t}}_j-{\underline{t}}_k}{b}\right)a_2^2$$

  (28)

  $$s_{t1,t2}=\sum _{j=1}^{k-1}K\left(\frac{{\underline{t}}_j-{\underline{t}}_k}{b}\right)a_1{a}_2$$

  (29)

  $$\begin{array}{c}\hfill w_j=K\left(\frac{{\underline{t}}_j-{\underline{t}}_k}{b}\right)\\ \hfill \{s_{t1,2}{s}_{t2,2}-s_{t1,t2}^2\\ \hfill -a_1(s_{t1,1}{s}_{t2,2}-s_{t2,1}{s}_{t1,t2})\\ \hfill +a_2(s_{t1,1}{s}_{t1,t2}-s_{t1,2}{s}_{t2,1})\}\end{array}$$

  (30)
  Equation (13) then yields $\tilde{\sigma }\left({\underline{t}}_k\right)$, and Equation (14) yields ${\tilde{W}}_{{\underline{t}}_k}$.

Using one of the above four methods (NW vs. LL, regular vs. predictive) gives estimates of the quantities needed to compute the $L_2$–optimal predictor (5). The bandwidth b in all four algorithms can be determined by cross-validation as described in Section 5.

4. Model-Free Point Prediction

For the Model-Based case, Equation (2) accounts for spatially-changing mean and variance of $Y_{\underline{t}}$. More generally, however, it may happen that the random field $\{Y_{\underline{t}}$ for $\underline{t}\in Z^2\}$ has a nonstationarity in its third (or higher moment), and/or in some other feature of its mth marginal distribution. This is addressed by the Model-Free Prediction Principle of Politis (2013, 2015).

The key towards Model-Free inference is to be able to construct an invertible transformation $H_n:{\underline{Y}}_{{\underline{t}}_n}\mapsto {\underline{ϵ}}_n$ where ${\underline{Y}}_{{\underline{t}}_n}=(Y_{{\underline{t}}_1},Y_{{\underline{t}}_2},\dots ,Y_{{\underline{t}}_n})$ denotes the random field data under consideration and ${\underline{ϵ}}_n={({ϵ}_1,\dots ,{ϵ}_n)}^{\prime }$ is a random vector with i.i.d. components. In order to do this in our context, let some $m\ge 1$, and denote by $\mathcal{L}(Y_{{\underline{t}}_k},Y_{{\underline{t}}_{k-1}},\dots ,Y_{{\underline{t}}_{k-m+1}})$ the mth marginal of $Y_{{\underline{t}}_k}$ i.e., the joint probability law of the vector ${(Y_{{\underline{t}}_k},Y_{{\underline{t}}_{k-1}},\dots ,Y_{{\underline{t}}_{k-m+1}})}^{\prime }$. We assume that $\mathcal{L}(Y_{{\underline{t}}_k},Y_{{\underline{t}}_{k-1}},\dots ,Y_{{\underline{t}}_{k-m+1}})$ changes smoothly (and slowly) with ${\underline{t}}_k$ in order to use nonparametric smoothing for estimation. In this case, $\{Y_{{\underline{t}}_k},{\underline{t}}_k\in Z^2\}$ can be defined over a 2D index-set D and the set $(Y_{{\underline{t}}_k},Y_{{\underline{t}}_{k-1}},\dots ,Y_{{\underline{t}}_{k-m+1}})$ can be considered to be lexicographically ordered as discussed previously in Section 2.

Similar to the framework proposed in [9] in order to ensure both the smoothness and data-based consistent estimation of $\mathcal{L}(Y_{{\underline{t}}_k},Y_{{\underline{t}}_{k-1}},\dots ,Y_{{\underline{t}}_{k-m+1}})$ we assume that, for all ${\underline{t}}_k,$

$$Y_{{\underline{t}}_k}={\mathbf{f}}_{{\underline{t}}_k}(W_{{\underline{t}}_k},W_{{\underline{t}}_{k-1}},\dots ,W_{{\underline{t}}_{k-m+1}})$$

(31)

for some function ${\mathbf{f}}_{{\underline{t}}_k}$(w) that is smooth in both arguments ${\underline{t}}_k$ and w, and some strictly stationary and weakly dependent, univariate series $W_{{\underline{t}}_k}$ where without loss of generality, it is assumed that $W_{{\underline{t}}_k}$ is a Gaussian series. Here model (2) is a special case of Equation (31) with $m=1$, and  the function ${\mathbf{f}}_{{\underline{t}}_k}$(w) being affine/linear in w. Therefore, for  comparison with the Model-Based case of Equation (2), in this section, we focus on the case $m=1$. For reference Model-Free estimators for point prediction and prediction intervals in the case of locally stationary time series for $m=1$ have been discussed in [9]. Below, we describe the steps necessary to construct the invertible transformation $H_n$ required to perform Model-Free point prediction for locally stationary random fields for the case $m=1$.

Step 1: 

Transformation to uniform samples

With $m=1$ let $D_{\underline{t}}\left(y\right)=P\{Y_{\underline{t}}\le y\}$ denote the first marginal distribution of the random field $\left\{Y_{\underline{t}}\right\}$. Applying the probability integral transform we have

$$U_{\underline{t}}=D_{\underline{t}}\left(Y_{\underline{t}}\right)\mathrm{for}\underline{t}={\underline{t}}_1,\dots ,{\underline{t}}_n$$

(32)

Here, $U_{{\underline{t}}_1},\dots ,U_{{\underline{t}}_n}$ are random variables having distribution Uniform $(0,1)$. In this case, it is assumed that $D_{\underline{t}}\left(y\right)$ is (absolutely) continuous in y for all $\underline{t}$. Therefore, we can use either local constant or local linear fitting to estimate it.

Using local constant fitting a smooth estimator can be defined as:

$${\overline{D}}_{{\underline{t}}_k}\left(y\right)=\sum _{i=1}^T\mathsf{\Lambda }\left(\frac{y-Y_{{\underline{t}}_i}}{h_0}\right)\tilde{K}\left(\frac{{\underline{t}}_k-{\underline{t}}_i}{b}\right)$$

(33)

where $\tilde{K}\left(\frac{{\underline{t}}_k-{\underline{t}}_i}{b}\right)=K\left(\frac{{\underline{t}}_k-{\underline{t}}_i}{b}\right)/{\sum }_{j=1}^{T}K\left(\frac{{\underline{t}}_k-{\underline{t}}_j}{b}\right)$, $\mathsf{\Lambda }\left(y\right)$ is a smooth distribution function, which is strictly increasing with density $\lambda \left(y\right)>0$ i.e., $\mathsf{\Lambda }\left(y\right)={\int }_{-\infty }^y\lambda \left(s\right)ds$ and $h_0$ is a secondary bandwidth. Furthermore, as in Section 3, we can let $T=k$ or $T=k-1$ leading to a fitted vs. predictive way to estimate $D_{{\underline{t}}_k}\left(y\right)$ using ${\overline{D}}_{{\underline{t}}_k}\left(y\right)$. Similar to the Model-Based case, we will assume throughout that $K(·)$ is a nonnegative, symmetric 2D Gaussian kernel function for which the diagonal values are set to the bandwidth b and the off-diagonal terms are set to 0. Note that the kernel estimator (33) is one-sided for the same reasons discussed before in Section 3. Cross-validation is used to determine the bandwidths $h_0$ and b; details are described in Section 5.

Since point prediction is performed on the boundary of the random field one can also consider local linear estimation as an alternative to the local constant-based smoothing approach. ${\overline{D}}_{{\underline{t}}_k}\left(y\right)$ as defined in Equation (33) is the Nadaraya–Watson smoother of the variables $v_1,\dots ,v_n$ where $v_i=\mathsf{\Lambda }\left(\frac{y-Y_{{\underline{t}}_i}}{h_0}\right)$. It is possible to define ${\overline{D}}_{{\underline{t}}_k}^{LL}\left(y\right)$ which is the local linear estimator of $D_{{\underline{t}}_k}\left(y\right)$ based on the smoothed variables $\mathsf{\Lambda }\left(\frac{y-Y_{{\underline{t}}_i}}{h_0}\right)$. This estimator is expected to have a smaller bias than ${\overline{D}}_{{\underline{t}}_k}\left(y\right)$. However, there is no guarantee that this will be a proper distribution function as a function of y, i.e., being nondecreasing in y with a left limit of 0 and a right limit of 1 as discussed in [26]. A proposed solution put forward by Hansen [30] involves a straightforward adjustment to the local linear estimator of a conditional distribution function that maintains its favorable asymptotic properties. The local linear version of ${\overline{D}}_{{\underline{t}}_k}\left(y\right)$ adjusted via Hansen’s (2004) proposal is given as follows:

$${\overline{D}}_{{\underline{t}}_k}^{LLH}\left(y\right)=\frac{{\sum }_{i=1}^T{w}_i^{\diamond }\mathsf{\Lambda }\left(\frac{y-Y_{{\underline{t}}_i}}{h_0}\right)}{{\sum }_{i=1}^T{w}_i^{\diamond }}.$$

(34)

The weights $w_i^{\diamond }$ are derived from weights $w_i$ described in Equations (22) and (30) for the fitted and predictive cases where

$$\begin{array}{c}\hfill w_i^{\diamond }=\left\{\begin{array}{cc}0\hfill & \mathrm{when}{w}_i<0\hfill \\ w_i\hfill & \mathrm{when}{w}_i\ge 0\hfill \end{array}\right.\end{array}$$

(35)

As with Equation (33), we can let $T=k$ or $T=k-1$ in the above, leading to a fitted vs. predictive local linear estimators of $D_{{\underline{t}}_k}\left(y\right)$ using ${\overline{D}}_{{\underline{t}}_k}^{LLH}\left(y\right)$.

One problem with the local linear estimator described above is that it replaces negative weights with zeros, and then renormalizes the nonzero weights. However, if estimation is performed on the boundary (as in the case with one-step ahead prediction of random fields), negative weights are crucially needed in order to ensure the extrapolation takes place with minimal bias. To address this problem, we modify the original, possibly nonmonotonic local linear distribution estimator ${\overline{D}}_{{\underline{t}}_k}^{LL}\left(y\right)$ to construct a monotonic version denoted by ${\overline{D}}_{{\underline{t}}_k}^{LLM}\left(y\right)$. The Monotone Local Linear Distribution Estimator ${\overline{D}}_{{\underline{t}}_k}^{LLM}\left(y\right)$ can be constructed by Algorithm 1 as given below [31].   

Algorithm 1: Monotone Local Linear Distribution Estimation

Recall that the derivative of ${\overline{D}}_{{\underline{t}}_k}^{LL}\left(y\right)$ with respect to y is given by $${\overline{d}}_{{\underline{t}}_k}^{LL}\left(y\right)=\frac{\frac{1}{h_0}{\sum }_{j=1}^T{w}_j\lambda \left(\frac{y-Y_{{\underline{t}}_j}}{h_0}\right)}{{\sum }_{j=1}^n{w}_j}$$ where $\lambda \left(y\right)$ is the derivative of $\mathsf{\Lambda }\left(y\right)$ and the weights $w_j$ can be derived based on Equations (22) and (30) for the fitted and predictive cases. Define a nonnegative version of ${\overline{d}}_{{\underline{t}}_k}^{LL}\left(y\right)$ as ${\overline{d}}_{{\underline{t}}_k}^{LL+}\left(y\right)=max({\overline{d}}_{{\underline{t}}_k}^{LL}\left(y\right),0)$. To make the above a proper density function, renormalize it to area one, i.e., let $${\overline{d}}_{{\underline{t}}_k}^{LLM}\left(y\right)=\frac{{\overline{d}}_{{\underline{t}}_k}^{LL+}\left(y\right)}{{\int }_{-\infty }^{\infty }{\overline{d}}_{{\underline{t}}_k}^{LL+}\left(s\right)ds}.$$ (36) Finally, define ${\overline{D}}_{{\underline{t}}_k}^{LLM}\left(y\right)={\int }_{-\infty }^y{\overline{d}}_{{\underline{t}}_k}^{LLM}\left(s\right)ds.$

The above modification of the local linear estimator allows one to maintain monotonicity while retaining the negative weights that are helpful in problems that involve estimation at the boundary. As with Equation (33), we can let $T=k$ or $T=k-1$ in the above, leading to a fitted vs. predictive local linear estimators of $D_{{\underline{t}}_k}\left(y\right)$ that are monotone.

Step 2: 

Transformation to iid normal samples

Starting from the original random field data $Y_{{\underline{t}}_1},\dots ,Y_{{\underline{t}}_n}$ by using either the local constant, the local linear or the monotone local linear distribution estimator in Step 1 it is possible to obtain samples $U_{{\underline{t}}_1},\dots ,U_{{\underline{t}}_n}$ having distribution Uniform $(0,1)$. However, these samples are dependent, and therefore, additional steps are necessary to convert them to i.i.d. samples as required for Model-Free inference. This is performed as described below.

Let $\mathsf{\Phi }$ denote the cumulative distribution function (cdf) of the standard normal distribution. Therefore we have:

$$Z_{\underline{t}}={\mathsf{\Phi }}^{-1}\left(U_{\underline{t}}\right)\mathrm{for}\underline{t}={\underline{t}}_1,\dots ,{\underline{t}}_n;$$

(37)

Here $Z_{{\underline{t}}_1},\dots ,Z_{{\underline{t}}_n}$ are correlated standard normal random variables. Now let ${\mathsf{\Gamma }}_n$ denote the $n\times n$ covariance matrix of the random vector ${\underline{Z}}_{{\underline{t}}_n}={(Z_{{\underline{t}}_1},\dots ,Z_{{\underline{t}}_n})}^{\prime }$. Consider the Cholesky decomposition ${\mathsf{\Gamma }}_n=C_n{C}_n^{\prime }$ where $C_n$ is (lower) triangular, and  construct the whitening transformation:

$${\underline{ϵ}}_n=C_n^{-1}{\underline{Z}}_{{\underline{t}}_n}.$$

(38)

It then follows that the entries of ${\underline{ϵ}}_n={({ϵ}_1,\dots ,{ϵ}_n)}^{\prime }$ are uncorrelated standard normal. Assuming that the random variables $Z_{{\underline{t}}_1},\dots ,Z_{{\underline{t}}_n}$ are jointly normal it can then be inferred that ${ϵ}_1,\dots ,{ϵ}_n$ are i.i.d. $N(0,1)$. Joint normality can be established by assuming a generative model of the random field as given by Equation (31); for a more detailed discussion refer to [9].

To implement the whitening transformation (38), it is necessary to estimate ${\mathsf{\Gamma }}_n$, i.e., the $n\times n$ covariance matrix of the random vector ${\underline{Z}}_{{\underline{t}}_n}={(Z_{{\underline{t}}_1},\dots ,Z_{{\underline{t}}_n})}^{\prime }$ where the $Z_{\underline{t}}$ are the normal random variables defined in Equation (37). The problem involves positive definite estimation of ${\mathsf{\Gamma }}_n$ based on the sample $Z_{{\underline{t}}_1},\dots ,Z_{{\underline{t}}_n}$. This estimate is based on the sample autocovariance, which is defined for a 2D second-order stationary random field {$y_{r,s}|r=1,2,\dots ,R,s=1,2,\dots ,S$} as follows [15]:

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill \breve{\gamma }(j,k)& =\breve{\gamma }(-j,-k)\hfill \\ & =\frac{1}{(R-j)(S-k)}\sum _{s=1}^{R-j}\sum _{t=1}^{S-k}\{y_{r+j,s+k}-\overline{y}\}\{y_{r,s}-\overline{y}\}\hfill \end{array}\end{array}$$

(39)

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill \breve{\gamma }(j,-k)& =\breve{\gamma }(-j,k)\hfill \\ & =\frac{1}{(R-j)(S-k)}\sum _{s=1}^{R-j}\sum _{t=k+1}^S\{y_{r+j,s-k}-\overline{y}\}\{y_{r,s}-\overline{y}\}\hfill \end{array}\end{array}$$

(40)

where ($j,k=0,1,2,\dots$).

Now, let ${\widehat{\mathsf{\Gamma }}}_n^{AR}$ be the $n\times n$ covariance matrix associated with the fitted AR(p,q) model to the data $Z_{{\underline{t}}_1},\dots ,Z_{{\underline{t}}_n}$ with $p,q$ by minimizing AIC, BIC or a related criterion as described in [15]. Let ${\widehat{\gamma }}_{|i-j|}^{AR}$ denote the $i,j$ element of the Toeplitz matrix ${\widehat{\mathsf{\Gamma }}}_n^{AR}$. Using the 2D Yule–Walker equations to fit the AR model implies that ${\widehat{\gamma }}_{k,l}^{AR}={\breve{\gamma }}_{k,l}$ for $k=0,1,\dots ,p$ and $l=0,1,\dots ,q$. For the cases where $k>p$ or $l>q,{\widehat{\gamma }}_{k,l}^{AR}$ can be fitted by iterating the difference equation that characterizes the fitted 2D AR model. In the R software, this procedure is automated for time series using the ARMAacf() function, and here we extend the same approach for stationary data over random fields.

Estimating the ‘uniformizing’ transformation $D_{\underline{t}}(·)$ and the whitening transformation based on ${\mathsf{\Gamma }}_n$ allows us to construct the transformation $H_n:{\underline{Y}}_{{\underline{t}}_n}\mapsto {\underline{ϵ}}_n$. Here, ${\underline{ϵ}}_n$ is a random vector with i.i.d. components as required by the Model-Free prediction principle. Since all the steps in the transformation, i.e.,  Equations (32), (37) and (38), are invertible; therefore, the composite transformation $H_n:{\underline{Y}}_{{\underline{t}}_n}\mapsto {\underline{ϵ}}_n$ is also invertible. However, in order to put the Model-Free Prediction Principle to work, we also need to estimate the transformation $H_{n+1}$ (and its inverse). To do so, we need a positive definite estimator for the matrix ${\mathsf{\Gamma }}_{n+1}$; this can be accomplished by extending the covariance matrix associated with the fitted 2D AR(p,q) model to $(n+1)$ by $(n+1)$ i.e., calculate ${\widehat{\mathsf{\Gamma }}}_{n+1}^{AR}$.

Consider the following vectors which include the additional values $Y_{{\underline{t}}_{n+1}},Z_{{\underline{t}}_{n+1}}$ and ${ϵ}_{n+1}$ that have not yet been estimated:

• ${\underline{Y}}_{{\underline{t}}_{n+1}}={(Y_{{\underline{t}}_1},\dots ,Y_{{\underline{t}}_n},Y_{{\underline{t}}_{n+1}})}^{\prime }$;
• ${\underline{Z}}_{{\underline{t}}_{n+1}}={(Z_{{\underline{t}}_1},\dots ,Z_{{\underline{t}}_n},Z_{{\underline{t}}_{n+1}})}^{\prime }$;
• ${\underline{ϵ}}_{n+1}={({ϵ}_1,\dots ,{ϵ}_n,{ϵ}_{n+1})}^{\prime }$.

We now show how to obtain the inverse transformation $H_{n+1}^{-1}:{\underline{ϵ}}_{n+1}\mapsto {\underline{Y}}_{{\underline{t}}_{n+1}}$. Since ${\underline{ϵ}}_n$ and ${\underline{Y}}_{{\underline{t}}_n}$ are related in a one-to-one way via transformation $H_n$; therefore, the values $Y_{{\underline{t}}_1},\dots ,Y_{{\underline{t}}_n}$ are obtainable by ${\underline{Y}}_{{\underline{t}}_n}=H_n^{-1}\left({ϵ}_n\right).$ Similar to the framework proposed for locally stationary time series in [9] below we show how to create the unobserved $Y_{{\underline{t}}_{n+1}}$ from ${\underline{ϵ}}_{n+1}$ using the following three steps (Algorithm 2).   

Algorithm 2: Generation of Unobserved Datapoint from Future Innovations

Let $${\underline{Z}}_{{\underline{t}}_{n+1}}=C_{n+1}{\underline{ϵ}}_{n+1}$$ (41) where $C_{n+1}$ is the (lower) triangular Cholesky factor of (our positive definite estimate of) ${\mathsf{\Gamma }}_{n+1}$. From the above, it follows that $$Z_{{\underline{t}}_{n+1}}={\underline{c}}_{n+1}{\underline{ϵ}}_{n+1}$$ (42) where ${\underline{c}}_{n+1}=(c_1,\dots ,c_n,c_{n+1})$ is a row vector consisting of the last row of matrix $C_{n+1}$. Create the uniform random variable $$U_{{\underline{t}}_{n+1}}=\mathsf{\Phi }\left(Z_{{\underline{t}}_{n+1}}\right).$$ (43) Finally, define $$Y_{{\underline{t}}_{n+1}}=D_{{\underline{t}}_{n+1}}^{-1}\left(U_{{\underline{t}}_{n+1}}\right);$$ (44) where in practice, the above will be based on an estimate of $D_{{\underline{t}}_{n+1}}^{-1}(·)$.

Since ${\underline{Y}}_{{\underline{t}}_n}$ has already been created using (the first n coordinates of) ${\underline{ϵ}}_{n+1}$, the above completes the construction of ${\underline{Y}}_{{\underline{t}}_{n+1}}$ based on ${\underline{ϵ}}_{n+1}$, i.e., the mapping $H_{n+1}^{-1}:{\underline{ϵ}}_{n+1}\mapsto {\underline{Y}}_{{\underline{t}}_{n+1}}$. By combining Equations (42)–(44) we can write the formula:

$$Y_{{\underline{t}}_{n+1}}=D_{{\underline{t}}_{n+1}}^{-1}\left(\mathsf{\Phi }\left({\underline{c}}_{n+1}{\underline{ϵ}}_{n+1}\right)\right).$$

The term ${\underline{c}}_{n+1}{\underline{ϵ}}_{n+1}$ can be written as ${\sum }_{i=1}^n{c}_i{ϵ}_i+c_{n+1}{ϵ}_{n+1}$; hence, the above can be compactly denoted as

$$\begin{array}{c}\hfill Y_{{\underline{t}}_{n+1}}=g_{n+1}\left({ϵ}_{n+1}\right)\mathrm{where}\\ \hfill g_{n+1}\left(x\right)=D_{{\underline{t}}_{n+1}}^{-1}\left(\mathsf{\Phi }\left(\sum _{i=1}^n{c}_i{ϵ}_i+c_{n+1}x\right)\right).\end{array}$$

(45)

Equation (45) is the predictive equation required in the Model-free Prediction Principle where $Y_{{\underline{t}}_{n+1}}$ is estimated conditionally on ${\underline{Y}}_{{\underline{t}}_n}=(Y_{{\underline{t}}_1},Y_{{\underline{t}}_2},\dots ,Y_{{\underline{t}}_n})$. The complete algorithm for constructing the Model-Free point predictors is as described below (Algorithm 3):   

Algorithm 3: Model-free (MF) point predictors for $Y_{{\underline{t}}_{n+1}}$

Construct $U_{{\underline{t}}_1},\dots ,U_{{\underline{t}}_n}$ by Equation (32) with $D_{{\underline{t}}_k}(·)$ estimated by either ${\overline{D}}_{{\underline{t}}_k}(·)$, ${\overline{D}}_{{\underline{t}}_k}^{LLH}(·)$ or ${\overline{D}}_{{\underline{t}}_k}^{LLM}(·)$ where ${\underline{t}}_k\in [{\underline{t}}_1,{\underline{t}}_n]$ Construct $Z_{{\underline{t}}_1},\dots ,Z_{{\underline{t}}_n}$ by Equation (37) Construct ${ϵ}_1,\dots ,{ϵ}_n$ by Equation (38), and let ${\widehat{F}}_n$ denote their empirical distribution. The Model-free $L_2$–optimal point predictor of $Y_{{\underline{t}}_{n+1}}$ is then $${\widehat{Y}}_{{\underline{t}}_{n+1}}=\int g_{n+1}\left(x\right)d{F}_n\left(x\right)=\frac{1}{n}\sum _{i=1}^n{g}_{n+1}\left({ϵ}_i\right)$$ where the function $g_{n+1}$ is defined in the predictive Equation (45) with $D_{{\underline{t}}_{n+1}}(·)$ being again estimated by either ${\overline{D}}_{{\underline{t}}_{n+1}}(·)$, ${\overline{D}}_{{\underline{t}}_{n+1}}^{LLH}(·)$ or ${\overline{D}}_{{\underline{t}}_{n+1}}^{LLM}(·)$ The Model-free $L_1$–optimal point predictor of $Y_{{\underline{t}}_{n+1}}$ is given by the median of the set $\{g_{n+1}\left({ϵ}_i\right)$ for $i=1,\dots ,n\}$.

5. Random Fields Cross-Validation

To choose the bandwidth b for either Model-Based or Model-Free point prediction, we perform one-step-ahead prediction at several coordinates of the given random field data. To elaborate, consider a random field $Y_{{\underline{t}}_1},Y_{{\underline{t}}_2},\dots ,Y_{{\underline{t}}_n}$ and suppose only subseries $Y_{{\underline{t}}_1},Y_{{\underline{t}}_2},\dots ,Y_{{\underline{t}}_k}$ has been observed where $k<n$. Let ${\widehat{Y}}_{{\underline{t}}_{k+1}}$ denote the predicted value based on the data $Y_{{\underline{t}}_1},\dots ,Y_{{\underline{t}}_k}$; this can be estimated by using either the Model-Based or Model-Free approaches as described in Section 3 and Section 4 for some choice of b. However, since $Y_{{\underline{t}}_{k+1}}$ is known, the quality of the predictor can be assessed. Therefore, for each value of b over a reasonable range, we calculate the sum of squared errors:

$$\begin{array}{c}\hfill SSE\left(b\right)=\sum _{k=k_o}^{n-1}{({\widehat{Y}}_{{\underline{t}}_{k+1}}-Y_{{\underline{t}}_{k+1}})}^2\end{array}$$

(46)

here $k_o$ should be big enough so that the estimation is accurate, e.g., $k_o$ can be of the order of $\sqrt{n}$. The cross-validated bandwidth choice would then be the b that minimizes $SSE\left(b\right)$. For the problem of selecting $h_0$ in the case of Model-Free point predictors, as in [21], our final choice is $h_0=h^2$ where $h=b/n$. Note that an initial choice of $h_0$ (needed to perform cross-validation to determine the optimal bandwidth b) can be set by any plug-in rule as the effect of choosing an initial value of $h_0$ is minimal.

6. Model-Free vs. Model-Based Inference: Empirical Comparisons

The point prediction performance of the Model-Free and Model-Based predictors described above are empirically compared using simulated as well as real-life data. The Model-Based local constant and local linear methods are denoted as MB-LC and MB-LL, respectively. Model-Based predictors MB-LC and MB-LL are described in Section 3. The Model-Free methods using local constant, local linear (Hansen) and local linear (Monotone) are denoted as MF-LC, MF-LLH, and MF-LLM. Model-Free predictors are described in Section 4. Both fitted and predictive residuals as described in Section 3 and Section 4 are used for point prediction and their performance as indicated by Mean Squared Error (MSE) is used to compare the various estimators.

Baseline comparisons: Besides the above MB and MF estimators we also provide three baselines estimators for comparison as described below:

• Model-Based estimation of $Y_{{\underline{t}}_{n+1}}$ involves nonparametric mean and variance estimation followed by estimating the conditional expectation $E\left(W_{{\underline{t}}_{n+1}}\right|{\underline{W}}_{{\underline{t}}_n})$ which involves calculating the coefficients of the 2D AR model as given by Equation (6). In this case as a baseline we have included results for both the synthetic and real-life datasets using only local linear estimation of the mean i.e., in this case the $L_2$–optimal predictor of $Y_{{\underline{t}}_{n+1}}$ is given by:

  $${\widehat{Y}}_{{\underline{t}}_{n+1}}=\mu \left({\underline{t}}_{n+1}\right)$$

  (47)
  Here, $\mu \left({\underline{t}}_{n+1}\right)$ is calculated using local linear fitting based on Equations (23)–(30) (in this case regular and predictive fitting are the same as stated in Remark 2.2 in [9]). In Table 1 and Table 2 this estimator is shown as LL.
• Model-Free estimation of $Y_{{\underline{t}}_{n+1}}$ involves nonparametric estimation of the first marginal distribution followed by estimating the autocovariance matrix ${\widehat{\mathsf{\Gamma }}}_n^{AR}$. In this case, as a baseline we have included results using only local linear estimation of the uniformizing transformation as given by Equations (34) and (35) (Local Linear Hansen) and Algorithm 1 (Monotone Local Linear Distribution Estimation). For the same reasons as stated above, regular and predictive fitting are also the same in this case. The $L_2$–optimal predictor of $Y_{{\underline{t}}_{n+1}}$ in this case is given by:

  $${\widehat{Y}}_{{\underline{t}}_{n+1}}=\frac{1}{M}\sum _{i=1}^M{D}_{{\underline{t}}_{n+1}}^{-1}\left(u_i\right)$$

  (48)
  Here, $u_1,\dots ,u_M\sim U[0,1]$ where M is some large integer, U is the uniform distribution and $D_{{\underline{t}}_{n+1}}$ is estimated by using ${\overline{D}}_{{\underline{t}}_{n+1}}^{LLH}(·)$ or ${\overline{D}}_{{\underline{t}}_{n+1}}^{LLM}(·)$. In Table 1 and Table 2 these estimators are shown as LLH, LLM.

The code for all algorithms used for the synthetic and real-life datasets as discussed in this paper can be found under https://github.com/srinjoyd/randomfields_pp (accessed on 24 July 2023).

6.1. Simulation: Additive Model with Stationary 2D AR Errors

Let a random field be generated using the 2D AR process as below:

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill W_{\underline{t}}=W_{t_1,t_2}& =0.25W_{t_1-1,t_2-1}+0.2W_{t_1-1,t_2+1}\hfill \\ & -0.05W_{t_1-2,t_2}+v_{t_1,t_2}\hfill \end{array}\end{array}$$

(49)

Let this field be generated over the region defined by $0\le t_1\le n_1\&0\le t_2\le n_2$ where $n_1=101,n_2=101$. The NSHP limits are set from $(101,101)$ to $(50,50)$, this defines the region $E_{\underline{t},\underline{n}}$ as shown in Figure 2. The data $Y_{\underline{t}}$ is generated using the additive model in Equation (1) with trend specified as $\mu \left(\underline{t}\right)=\mu (t_1,t_2)=sin\left(4\pi \frac{t_2-1}{n_2-1}\right)$ where $0\le t_1\le n_1\&0\le t_2\le n_2$. Here $v_{t_1,t_2}$ are i.i.d. $N(0,{\tau }^2)$ where $\tau =0.1$. Let $t_1=50,t_2=50$ where point prediction is performed. Bandwidths for all Model-Based, Model-Free, and baseline predictors are calculated using cross-validation as described in Section 5.

Results for point prediction using mean square error (MSE) over all MB and MF methods are shown in Table 1. A total of 100 realizations of the dataset were used for measuring point prediction performance. From this table it can be seen that MB-LL is the best point predictor. This is expected since the data was generated by a 2D AR model which is the same used in MB-LL prediction. In addition, the estimation is performed at the boundary of the random field with a strong linear trend as shown in Figure 3 where LL regression is expected to perform the best. In addition, it can be observed that MF-LLM performs the best among all MF point predictors and approaches the performance of MB-LL. This shows that monotonicity correction in the LLM distribution estimator has minimal effect on the center of the distribution that is used for point prediction.

Table 1. Point Prediction performance for 2D AR dataset.

Prediction Method	Residual Type	MSE
MB-LC	P	$1.488\times {10}^{-2}$
	F	$1.520\times {10}^{-2}$
MB-LL	P	$1.393\times {10}^{-2}$
	F	$1.400\times {10}^{-2}$
MF-LC	P	$1.530\times {10}^{-2}$
	F	$1.549\times {10}^{-2}$
MF-LLH	P	$1.471\times {10}^{-2}$
	F	$1.515\times {10}^{-2}$
MF-LLM	P	$1.414\times {10}^{-2}$
	F	$1.456\times {10}^{-2}$
LL	Not Applicable	$1.488\times {10}^{-2}$
LLH	Not Applicable	$1.651\times {10}^{-2}$
LLM	Not Applicable	$1.455\times {10}^{-2}$

Table 1. Point Prediction performance for 2D AR dataset.

Prediction Method	Residual Type	MSE
MB-LC	P	$1.488\times {10}^{-2}$
	F	$1.520\times {10}^{-2}$
MB-LL	P	$1.393\times {10}^{-2}$
	F	$1.400\times {10}^{-2}$
MF-LC	P	$1.530\times {10}^{-2}$
	F	$1.549\times {10}^{-2}$
MF-LLH	P	$1.471\times {10}^{-2}$
	F	$1.515\times {10}^{-2}$
MF-LLM	P	$1.414\times {10}^{-2}$
	F	$1.456\times {10}^{-2}$
LL	Not Applicable	$1.488\times {10}^{-2}$
LLH	Not Applicable	$1.651\times {10}^{-2}$
LLM	Not Applicable	$1.455\times {10}^{-2}$

In addition, comparing the performance of MB-LL versus its corresponding baseline LL and that of MF-LLH and MF-LLM versus their corresponding baselines LLH and LLM, respectively, show that the baseline estimators underperform as they do not take into account the spatial dependence present in the data either by estimating the coefficients of the 2D AR model (Equation (6) as in the Model-Based case) or by estimating the autocovariance matrix ${\widehat{\mathsf{\Gamma }}}_n^{AR}$ (Model-Free case).

6.2. Real-Life Example: CIFAR Images

The CIFAR-10 dataset [32] is used as a real-life example to compare the Model-Based and Model-Free prediction algorithms discussed before. The original CIFAR-10 dataset consists of 60,000 32 by 32 color images in 10 classes, with 6000 images per class. We pick 100 images from the class “dog” where the original images have 3 RGB (red, green, blue) channels with discrete pixel values. We pick the R (red) channel of each image, and standardize these to generate a new real-valued dataset. Our final transformed dataset has 100 32 by 32 random fields. The NSHP limits are set from $(32,32)$ to $(16,16)$, this defines the region $E_{\underline{t},\underline{n}}$ as shown in Figure 2. The rest of the image is considered occluded and their pixel values are not available for prediction. Sample images used for prediction are shown in Figure 4. Let $t_1=16,t_2=16$ where point prediction is performed. Bandwidths for all Model-Based, Model-Free, and baseline predictors are calculated using cross-validation as described in Section 5.

The results for point prediction using mean square error (MSE) over all MB and MF methods are shown in Table 2. From this table, it can be seen that MF-LLH and MF-LLM are the best point predictors. The superior performance of the Model-Free estimators as compared to their Model-Free counterparts can be attributed to the fact that the CIFAR-10 image data is not compatible with the additive model as given by Equation (1). It can also be seen that unlike the synthetic 2D AR dataset the two best predictors MF-LLH and MF-LLM are closer in performance which is owing to the lack of a linear trend at the point where prediction is performed.

In addition, comparing the performance of MB-LL versus its corresponding baseline LL and that of MF-LLH and MF-LLM versus their corresponding baselines LLH and LLM, respectively, show that the baseline estimators underperform as they do not take into account the spatial dependence present in the data either by estimating the coefficients of the 2D AR model (Equation (6) as in the Model-Based case) or by estimating the autocovariance matrix ${\widehat{\mathsf{\Gamma }}}_n^{AR}$ (Model-Free case).

Table 2. Point Prediction performance for CIFAR-10 dataset.

Prediction Method	Residual Type	MSE
MB-LC	P	$1.98\times {10}^{-1}$
	F	$2.20\times {10}^{-1}$
MB-LL	P	$1.79\times {10}^{-1}$
	F	$1.95\times {10}^{-1}$
MF-LC	P	$1.79\times {10}^{-1}$
	F	$2.12\times {10}^{-1}$
MF-LLH	P	$1.60\times {10}^{-1}$
	F	$1.89\times {10}^{-1}$
MF-LLM	P	$1.64\times {10}^{-1}$
	F	$1.70\times {10}^{-1}$
LL	Not Applicable	$2.12\times {10}^{-1}$
LLH	Not Applicable	$2.38\times {10}^{-1}$
LLM	Not Applicable	$2.14\times {10}^{-1}$

Table 2. Point Prediction performance for CIFAR-10 dataset.

Prediction Method	Residual Type	MSE
MB-LC	P	$1.98\times {10}^{-1}$
	F	$2.20\times {10}^{-1}$
MB-LL	P	$1.79\times {10}^{-1}$
	F	$1.95\times {10}^{-1}$
MF-LC	P	$1.79\times {10}^{-1}$
	F	$2.12\times {10}^{-1}$
MF-LLH	P	$1.60\times {10}^{-1}$
	F	$1.89\times {10}^{-1}$
MF-LLM	P	$1.64\times {10}^{-1}$
	F	$1.70\times {10}^{-1}$
LL	Not Applicable	$2.12\times {10}^{-1}$
LLH	Not Applicable	$2.38\times {10}^{-1}$
LLM	Not Applicable	$2.14\times {10}^{-1}$

7. Conclusions and Future Work

In this paper, we investigate the problem of one-sided prediction over random fields that are stationary only across a limited part of their entire region of definition. For such locally stationary random fields, we develop frameworks for point prediction using both a Model-Based approach, which includes a coordinate changing trend and/or variance and also by using the Model-Free principle proposed by [21,22]. We apply our algorithms to both synthetic data as well as a real-life dataset consisting of images from the CIFAR-10 dataset. In the latter case, we obtain the best performance by using the Model-Free approach and thereby demonstrate the superiority of this technique versus the Model-Based case where an additive model is assumed arbitrarily for purposes of prediction. In future work, we plan to investigate both Model-Based and Model-Free prediction using random fields with nonuniform spacing of data as well as consider extending our algorithms for estimating prediction intervals.