Data-Driven Model Reduction for Stochastic Burgers Equations

Abstract

We present a class of efficient parametric closure models for 1D stochastic Burgers equations. Casting it as statistical learning of the flow map, we derive the parametric form by representing the unresolved high wavenumber Fourier modes as functionals of the resolved variable’s trajectory. The reduced models are nonlinear autoregression (NAR) time series models, with coefficients estimated from data by least squares. The NAR models can accurately reproduce the energy spectrum, the invariant densities, and the autocorrelations. Taking advantage of the simplicity of the NAR models, we investigate maximal space-time reduction. Reduction in space dimension is unlimited, and NAR models with two Fourier modes can perform well. The NAR model’s stability limits time reduction, with a maximal time step smaller than that of the K-mode Galerkin system. We report a potential criterion for optimal space-time reduction: the NAR models achieve minimal relative error in the energy spectrum at the time step, where the K-mode Galerkin system’s mean Courant–Friedrichs–Lewy (CFL) number agrees with that of the full model.

1. Introduction

Closure modeling aims for computationally efficiently reduced models for tasks requiring repeated simulations such as Bayesian uncertainty quantification [1,2] and data assimilation [3,4]. Consisting of low-dimensional resolved variables, the closure model must take into account the non-negligible effects of unresolved variables so as to capture both the short-time dynamics and large-time statistics. As suggested by the Mori–Zwanzig formalism [5,6,7], trajectory-wise approximation is no longer appropriate, and the approximation is in a statistical sense. That is, the reduced model aims to generate a process that approximates the target process in distribution, or at least, reproduce the key statistics and dynamics for the quantities of interest. For general nonlinear systems, such a reduced closure model is out of the reach of direct derivations from first principles.

Data-driven approaches, which are based on statistical learning methods, provide useful and practical tools for model reduction. The past decades witness revolutionary developments of data-driven strategies, ranging from parametric models (see, e.g., [8,9,10,11,12,13,14] and the references therein) to nonparametric and machine learning methods (see, e.g., [15,16,17,18]). These developments demand a systematic understanding of model reduction from the perspectives of dynamical systems (see, e.g., [7,19,20]), numerical approximation [21,22], and statistical learning [17,23].

With 1D stochastic Burgers equation as a prototype model, we aim to further the understanding of model reduction from an interpretable statistical inference perspective. More specifically, we consider a stochastic Burgers equation with a periodic solution on $[0,2\pi ]$:

$$\begin{array}{c}\hfill \begin{array}{cc}& u_t=\nu u_{xx}-u{u}_x+f(x,t),0<x<2\pi ,t>0\hfill \\ & u(0,t)=u(2\pi ,t)=0,u_x(0,t)=u_x(2\pi ,t),\hfill \end{array}\end{array}$$

(1)

from an initial condition $u(·,0)$. We consider a stochastic force $f(x,t)$ that is smooth in space, residing on $K_0$ low wavenumber Fourier modes, and white in time, given by

$$f(x,t)=\sigma \sum _{m=1}^{K_0}sin\left(mx\right){\dot{W}}_m\left(t\right)+cos\left(mx\right){\dot{W^{\prime }}}_m\left(t\right),$$

(2)

where $\{W_m,W_m^{\prime }\}$ are independent Brown motions. Here $\nu >0$ is the viscosity constant and $\sigma >0$ represents the strength of the stochastic force.

Our goal is to find a discrete-time closure model for the first K Fourier modes, so as to efficiently reproduce the energy spectrum and other statistics of these modes.

We present a class of efficient parametric reduced closure models for 1D stochastic Burgers equations. The key idea is to approximate the discrete-in-time flow map statistically, in particular, to represent the unresolved high wavenumber Fourier modes as functionals of the resolved variable’s trajectory. The reduced models are nonlinear autoregression (NAR) time series models, with coefficients estimated from data simply by least squares. We test the NAR models in four settings: reduction of deterministic responses ($K>K_0$) vs. reduction involving unresolved stochastic force ($K<K_0$), and small vs. large scales of stochastic force (with $\sigma =0.2$ and $\sigma =1$), where $K_0$ is the number of Fourier modes of the white-in-time stochastic force and $\sigma$ is the scale of the force. In all these settings, the NAR models can accurately reproduce the energy spectrum, invariant densities, and autocorrelation functions (ACF). We also discuss model selection, consistency of estimators, and memory length of the reduced models.

Taking advantage of our NAR models’ simplicity, we further investigate a critical issue in model reduction of (stochastic) partial differential equations: maximal space-time reduction. The space dimension can be reduced arbitrarily in our parametric inference approach: NAR models with two Fourier modes perform well. The time reduction is another story. The maximal time step is limited by the NAR model’s stability and is smaller than those of the K-mode Galerkin system. Numerical tests indicate that the NAR models achieve the minimal relative error at the time step where the K-mode Galerkin system’s mean CFL (Courant–Friedrichs–Lewy) number agrees with the full model’s, suggesting a potential criterion for optimal space-time reduction.

One can readily extend our parametric closure modeling strategy to general nonlinear dissipative systems beyond quadratic nonlinearities. Along with [14], we may view it as a parametric inference extension of the nonlinear Galerkin methods [24,25,26,27]. However, it does not require the existence of an inertial manifold (and the stochastic Burgers equation does not satisfy the spectral gap condition that is sufficient for the existence of an inertial manifold [28]), and it applies to resolved variables of any dimension (e.g., lower than the dimension of the inertial manifold if it exists [14]). Notably, one may use NAR models that are linear in parameters and estimate them by least squares. Therefore, the algorithm is computationally efficient and is scalable for large systems.

The limitation of the parametric modeling approach is its reliance on the derivation of a parametric form using the Picard iteration, which depends on the nonlinearity of the unresolved variables (see Section 3.1). When the nonlinearity is complicated, a linear-in-parameter ansatz may be out of reach. One can overcome this limitation by nonparametric techniques [23,29] and machine learning methods (see, e.g., [16,17,30]).

The stochastic Burgers equation is a prototype model for developing closure modeling techniques for turbulence (see e.g., [31,32,33,34,35,36,37]). In particular, Dolaptchiev et al. [37] propose a closure model for stochastic Burgers equation in a similar setting, based on local averages of finite-difference discretization, reproducing accurate energy spectrum similar to this study. We directly construct a simple yet effective NAR model for the Fourier modes, providing the ground of a statistical inference examination of model reduction.

We note that the closure reduced models based on parametric inference are different from the widely studied proper orthogonal decomposition (POD)-based reduced order models (ROM) for parametric full models [38,39]. These POD-ROMs seek new effective bases to capture the effective dynamics by a linear system for the whole family of parametric full models. The inference-based closure models focus on nonlinear dynamics in a given basis and aim to capture both short-time dynamics and large-time statistics. In a probabilistic perspective, both approaches approximate the target stochastic process: the POD-ROMs are based on Karhunen-Loéve expansion, while the inference-based closure models aim to learn the nonlinear flow-map. One may potentially combine the two and find nonlinear closure models for the nonlinear dynamics in the POD basis.

The exposition of our study proceeds as follows. We first summarize the notations in Table 1. Following a brief review of the basic properties of the stochastic Burgers equation and its numerical integration, we introduce in Section 2 the inference approach to closure modeling and compare it with the nonlinear Galerkin methods. Section 3 presents the inference of NAR models: derivation of the parametric form, parameter estimation, and model selection. Examining NAR models’ performance in four settings in Section 4, we investigate the space-time reduction. Section 5 concludes our main findings and possible future research.

Table 1. Notations: the variables in the full and reduced models.

Model	Notation	Description
Full model	$u(x,t)={\sum }_{\left|k\right|\ge 1}{\widehat{u}}_k\left(t\right)e^{i{q}_{k}x}$	solution of (1) in its Fourier series
	$f(x,t)={\sum }_{\left|k\right|\ge 1}^{K_0}{\widehat{f}}_k\left(t\right)e^{i{q}_{k}x}$	stochastic force in (2) in its Fourier series
	$v(x,t)={\sum }_{\left|k\right|\le K}{\widehat{u}}_k\left(t\right)e^{i{q}_{k}x}$	the resolved variable, the target process for closure modeling
	$w(x,t)={\sum }_{\left|k\right|>K}{\widehat{u}}_k\left(t\right)e^{i{q}_{k}x}$	the unresolved variable; $u=v+w$ in (12)
	$\nu$, $\sigma$	the viscosity in (1) and the strength of the stochastic force
	N,$dt$	number of modes and time step-size in numerical solution
Reduced models	K	number of modes in reduced (NAR) models in (17)
	${\left(u_k^n\right)}_{\left|k\right|\le K}$	state variable in reduced model, corresponding to ${\widehat{u}}_k\left(t_n\right)$
	$\delta =dt\times \mathrm{Gap}$	observation time interval
	$R_k^{\delta }$, ${\Phi }^n$, $g^n$	parametric terms in the NAR model in (10) and (17)

2. Space-Time Reduction for Stochastic Burgers Equationations

In this section, we first review basic properties of the stochastic Burgers equation and its numerical integration. Then, we introduce inference-based model reduction and compare it with the nonlinear Galerkin methods.

2.1. The Stochastic Burgers Equationation

A Fourier transform of Equation $\left(1\right)$ leads to

$$\begin{array}{ccc}\hfill \frac{d}{dt}{\widehat{u}}_k& =& -\nu q_k^2{\widehat{u}}_k-\frac{i{q}_k}{2}\sum _{l=-\infty }^{\infty }{\widehat{u}}_l{\widehat{u}}_{k-l}+{\widehat{f}}_k\left(t\right)\hfill \end{array}$$

(3)

with $q_k=k,k\in \mathbb{Z}$, where ${\widehat{u}}_k$ are Fourier modes:

$${\widehat{u}}_k\left(t\right)=\mathcal{F}{\left[u\right]}_k=\frac{1}{2\pi }{\int }_0^{2\pi }u(x,t)e^{-i{q}_{k}x}dx,u(x,t)={\mathcal{F}}^{-1}\left[\widehat{u}\right]=\sum _k{\widehat{u}}_k\left(t\right)e^{i{q}_{k}x},$$

The system has the following properties. First, it is Galilean invariant: if $u(x,t)$ is a solution, then $u(x-ct,t)+c$, with c an arbitrary constant speed, is a solution. To see this, let $v(x,t)=u(x-ct,t)+c$. Then, $v_t=-c{u}_x+u_t$, $v_x=u_x$, and

$$v_t=c{v}_x+u_{xx}+u{u}_x+f=c{v}_x+v_{xx}+(v-c)v_x+f=v_{xx}+v{v}_x+f.$$

Without loss of generality, we set ${\int }_0^{2\pi }u(x,0)dx=0$. This implies that ${\widehat{u}}_0\left(0\right)=0$. In this study, we only consider forces with mean zero, i.e., ${\int }_0^{2\pi }f(x,t)dx=0$, therefore from Equation (3), we see that ${\widehat{u}}_0\left(t\right)\equiv 0$, or equivalently, ${\int }_0^{2\pi }u(x,t)dx\equiv 0$. Second, the system has an invariant measure [31,40,41], due to a balance between the diffusion term, which dissipates energy, and the stochastic force, which injects energy. In particular, the initial condition does not affect the large time statistical properties of the solution. Third, since u is real, the Fourier modes satisfies ${\widehat{u}}_{-k}={\widehat{u}}_k^{*}$, where ${\widehat{u}}_k^{*}$ is the complex conjugate of ${\widehat{u}}_k$.

2.2. Galerkin Spectral Method

We consider the Galerkin spectral method for numerical solutions of the Burgers equation. The system is approximated as follows: the function $u(x,t)$ is represented at grid points $x_i=i\Delta x$ with $i=0,\dots ,2N-1$ and $\Delta x=\frac{2\pi }{2N}$. The Fourier transform $\mathcal{F}$ is replaced by discrete Fourier transform

$${\widehat{u}}_k\left(t\right)={\mathcal{F}}_{2N}{\left[u\right]}_k=\sum _{i=0}^{2N-1}u(x_i,t)e^{-i{q}_k{x}_i},u(x_i,t)={\mathcal{F}}_{2N}^{-1}{\left[\widehat{u}\right]}_i=\frac{1}{2N}\sum _{k=-N+1}^N{\widehat{u}}_k{e}^{i{q}_k{x}_i}.$$

For simplicity of notation, we abuse the notation $u(x_i,t)$ so that it denotes either the true solution or its high-resolution $2N$-mode approximation. Since u is real, we have ${\widehat{u}}_{-k}={\widehat{u}}_k^{*}$. Noticing further that ${\widehat{u}}_0=0$ due to Galilean invariance, and setting ${\widehat{u}}_N=0$, we obtain a truncated system

$$\begin{array}{c}\hfill \frac{d}{dt}{\widehat{u}}_k=-\nu q_k^2{\widehat{u}}_k-\frac{ik}{2}\sum _{\begin{array}{c}|k-l|\le N,\left|l\right|\le N\end{array}}{\widehat{u}}_l{\widehat{u}}_{k-l}+{\widehat{f}}_k,\mathrm{with}\left|k\right|=1,\dots ,N.\end{array}$$

(4)

We solve Equation (4) using the exponential time differencing fourth order Rouge–Kutta method (ETDRK4) (see [42,43]) with standard $3/2$ zero-padding for dealiasing (see e.g., [44]), with the force term ${\widehat{f}}_k$ treated as a constant in each time step. Such a mixture scheme is of strong order 1, but it has an advantage of preserving both the numerical stability of ETDRK4 and the simplicity of Euler–Maruyama.

We will consider a relatively small viscosity $\nu =0.02$, so that random shocks are about to emerge in the solution. In general, a smaller viscosity constant demands a higher resolution in space-time to resolve the solution, particularly the emerging shocks as $\nu$ vanishes. To sufficiently resolve the solution, we set $N=128$ and $dt=0.001$. The solution is accurately resolved, with mean Courant–Friedrichs–Lewy (CFL) numbers being 0.139 and 0.045 for $\sigma =1$ and $\sigma =0.2$, respectively. Here the mean CFL number is computed as the average along a trajectory with $N_t={10}^5$ steps

$$\mathrm{Mean}\mathrm{CFL}\mathrm{number}=\frac{1}{N_t}\sum _{n=1}^{N_t}\underset{x}{sup}\left|u(x,t_n)\right|\frac{\Delta t}{\Delta x},$$

where $\Delta t$ and $\Delta x$ are the time step and space step, respectively. Furthermore, numerical tests show that the marginal densities converge as trajectory length increases.

2.3. Nonlinear Galerkin and Inferential Model Reduction

For simplicity of notation, we write the Burgers equation in an operator form as

$${\partial }_{t}u+Au=B\left(u\right)+f,u\left(0\right)=u_0$$

(5)

with a linear operator $A:H_0^1(0,2\pi )\to L^2(0,2\pi )$ and a nonlinear operator $B:H_0^1(0,2\pi )\to L^2(0,2\pi )$

$$A=-\nu {\partial }_{xx},B\left(u\right)=-{\left(u^2\right)}_x/2.$$

We first decompose the Fourier modes of u into resolved and unresolved variables. Recall that our goal of model reduction is to derive a closed system that can faithfully describe the dynamics of the coefficients ${\left\{{\widehat{u}}_k\left(t\right)\right\}}_{\left|k\right|=1}^K$, or equivalently, the low dimensional process $v(x,t)={\sum }_{\left|k\right|=1}^K{\widehat{u}}_k\left(t\right)e^{i{q}_{k}x}$.

Denote by P the projection operator from $H_0^1(0,2\pi )$ to $\mathrm{span}{\left\{e^{i{q}_{k}x}\right\}}_{\left|k\right|=1}^K$, and let $Q:=I-P$ (and for simplicity of notation, we will also denote them as projections on the corresponding vector spaces of Fourier modes). With $u=Pu+Qu=v+w$, we can write the system (5) as

$$\begin{array}{}\mathrm{(6)}& \frac{dv}{dt}& =-PAv+PB\left(v\right)+Pf+\left[PB\right(v+w)-PB(v\left)\right],\mathrm{(7)}& \frac{dw}{dt}& =-QAw+QB(v+w)+Qf.\end{array}$$

To find a closed system for v, we quantify the truncation error $PB(v+w)-PB\left(v\right)$ in (6), which represents the nonlinear interaction between the low and high wavenumber modes, by either a function of v or a functional of the trajectory of v. In particular, in the nonlinear Galerkin method based on inertial manifold theory, (see e.g., [24,25,26,27]), one aims to represent the high modes w as a function of the low modes v (and hence obtaining an approximate inertial manifold). In the simplest implementation, one neglects the time derivative in Equationation (7) and solves $w=\psi \left(v\right)$ from

$$w\approx -{\left(QA\right)}^{-1}[QB(v+w)+Qf]$$

by fixed point iterations: ${\psi }_0=0,{\psi }_{i+1}=-{\left(QA\right)}^{-1}[QB(u+{\psi }_i)+Qf]$. This leads to an approximation of w as a function of v, which exists if K is large enough and if the system satisfies a gap condition (so that an inertial manifold exists). However, among many dissipative systems with global attractor, only a few have been proven to satisfy the gap condition (see [28] for a recent review). More importantly, we can not always expect K to be larger than the dimension of an inertial manifold, which is unknown in general. Therefore, such a nonlinear Galerkin approach works for neither a system without an inertial manifold nor for a K smaller than the dimension of the inertial manifold.

We take a different perspective on the reduction. Unlike the nonlinear Galerkin which aims for a trajectory-wise approximation, we aim for a probabilistic approximation of the distribution of the stochastic process $\left(v\right(·,t),t\ge 0)$. The randomness of the process v can come from random initial conditions and/or from the stochastic force. We emphasize that a key is to represent the dependence of the model error $PB(v+w)-PB\left(v\right)$ on the process v, not simply constructing a stochastic process with the same distribution as $PB(v+w)-PB\left(v\right)$, which may be independent of the process of v.

In a data-driven approach, such a probabilistic approximation leads naturally to the statistical inference of the underlying process, aiming to represent the model error $\left[PB\right(v+w)-PB(v\left)\right]\left(t\right)$ as a functional of the past trajectory $\left(v\right(·,s),s\le t)$. This inferential reduction approach works flexibly for general settings: there is no need of an inertial manifold and the dimension K can be arbitrary (e.g., less than the dimension of the inertial manifold, as shown in [14]).

Space-time reduction. To achieve a space-time reduction for practical computation, the reduced model should be a time series model with a time step $\delta >dt$ for time reduction, instead of a differential system. It approximates the flow map (with $t_n=n\delta$)

$${\widehat{u}}_k\left(t_{n+1}\right)=F{({\widehat{u}}_{·}\left(t_n\right),{\widehat{f}}_{·}\left([t_n:t_{n+1}]\right))}_k,\left|k\right|\le K,$$

(8)

where ${\widehat{u}}_{·}\left(t_n\right)=({\widehat{u}}_k\left(t_n\right),\left|k\right|\ge 0)$ is the vector of all Fourier modes, and thus the above map is not a closed system for the low modes. Recall that for $\left|k\right|\le K$,

$$\begin{array}{c}\hfill \underset{K-\mathrm{mode}\mathrm{truncation}}{\underbrace{\frac{d}{dt}{\widehat{u}}_k=-\nu q_k^2{\widehat{u}}_k-\frac{ik}{2}\sum _{\begin{array}{c}|k-l|\le K,\\ \left|l\right|\le K\end{array}}{\widehat{u}}_l{\widehat{u}}_{k-l}}}\underset{\mathrm{truncation}\mathrm{error}}{\underbrace{-\frac{i{q}_k}{2}\sum _{\begin{array}{c}|k-l|>K\\ \mathrm{or}\left|l\right|>K\end{array}}{\widehat{u}}_l{\widehat{u}}_{k-l}}}+{\widehat{f}}_k\left(t\right)\end{array}$$

(9)

Clearly, the K-mode truncated Galerkin system can provide an immediate approximation to F in (8). Making use of it, we propose a time series model for ${\left\{{\widehat{u}}_k\left(t_n\right)\right\}}_{\left|k\right|=1}^K$ in the form of

$$u_k^{n+1}=u_k^n+\delta [R_k^{\delta }\left(u^n\right)+f_k^n+{\Phi }_k^n]+g_k^{n+1},\left|k\right|\le K,$$

(10)

where $R_{·}^{\delta }\left(u^n\right)$ is from a one-step forward integrator with time step-size $\delta$ of the deterministic K-mode Galerkin, and $f_k^n={\widehat{f}}_k\left(t_n\right)$ is white noise in the kth Fourier mode of the stochastic force at time $t_n$. Here, the term ${\Phi }^n$ and the noise $g^{n+1}$ aim to represent the truncation error, as well as the discretization error. Together with the other terms in (10), they provide a statistical approximation to the flow map F in (8). In particular, the term ${\Phi }^n$ approximates F based on information up to time n (e.g., the conditional expectation), and the noise $g^{n+1}$ aims to statistically represent the residual of the approximation. Since the truncation error depends on the past history of the low wavenumber modes, and as suggested by the Mori–Zwanzig formalism [6,7], we make ${\Phi }^n$ depend on the trajectory $u^{1:n}$ of the state process, as well as the trajectories $f^{1:n}$ and $g^{1:n}$:

$$\begin{array}{c}\hfill {\Phi }^n:=\Phi (u^{1:n},f^{1:n},g^{1:n}).\end{array}$$

(11)

For simplicity, we assume the noise $\left\{g^n\right\}$ to be iid Gaussian, and the resulted time series model in (10) is a nonlinear autoregression moving average model (NARMA) [13,45,46].

The right hand side of Equation (10), together with ${\Phi }^n$ defined in Equation (11), aims for a statistical approximation of the discrete-time map (8). However, the general form in Equation (11) leads to a high dimensional function to be learned from data, which is intractable by regression methods using either global or local polynomial basis, due to the well-known curse of dimensionality. Fortunately, the physical model provides informative structures to reduce the dimension, and we can obtain effective approximations based on only a few basis functions with finite memory. In the next section, we derive from the physical model a parametric form for the reduced model, whose coefficients can be efficiently estimated from data.

To avoid confusions between notations, we summarize the correspondence of the variables between the full and reduced models in Table 2.

Table 2. Correspondence of the variables between the full and reduced models.

	Full Model in (4)	Reduced Model in (10) or (17)
State variables	${\widehat{u}}_k\left(t_n\right)$ or $\widehat{u}\left(t_n\right)$ in (4) and (9)	$u_k^n$ or $u^n$ in (10)
Resolved variable	$v(x,t_n)$ or v, in (6) and (12)	the vector $(u_{-K}^n,\dots ,u_K^n)$ in (17)
Unresolved variable	$w(x,t)$ or w in (7) and (12)	NA
Stochastic force	white noise ${\widehat{f}}_k\left(t_n\right)$ in (9)	white noise $f_k^n$ in (10)
Noise introduced in inference	NA	$g^n$ in (10)
Flow map of resolved variable	F in Equation (8)	Equation (10)

3. Inference of Reduced Models

We present here the parametric inference of NAR models: derivation of parametric forms, estimation of the parameters, and model selection.

3.1. Derivation of Parametric Reduced Models

We derive parametric reduced models by extracting basis functions from numerical integration of Equation (6). The combination of these basis functions will give us $\Phi (u^{1:n},f^{1:n},g^{1:n})$ in (11), which approximates the flow maps $\{F{({\widehat{u}}_{·}\left(t_n\right),{\widehat{f}}_{·}\left([t_n:t_{n+1}]\right))}_k,|k|\le K\}$ in (8) in a statistical sense.

We first write a closed integro-differential system for the low-mode process $\left(v\right(·,t),t\ge 0)$. In view of Equation (6), this can be simply done by integrating the equation of the high modes w in Equation (7):

$$\left\{\begin{array}{cc}\frac{dv}{dt}\hfill & =-PAv+PB\left(v\right)+Pf+\left[PB\right(v+w)-PB(v\left)\right],\hfill \\ w\left(t\right)\hfill & =e^{-QA\tau }w(t-\tau )+{\int }_{t-\tau }^t{e}^{-QA(t-s)}[QB(v\left(s\right)+w\left(s\right))+Qf\left(s\right)]ds,\hfill \end{array}\right.$$

(12)

where $\tau \in [0,t]$. Note that in addition to the trajectories $\left(v\right(·,s),s\in [t-\tau ,t\left]\right)$ and $\left(Qf\right(s),s\in [t-\tau ,t\left]\right)$, which we can assume to be known, the state $w(·,t)$ also depends on the initial condition $w(·,t-\tau )$. Therefore, this equation is not strictly closed. However, as $\tau$ increases, the effect of the initial condition decays exponentially, allowing for possible finite time approximate closure. Given $w(·,t-\tau )$ and $\left(Qf\right(s),s\in [t-\tau ,t\left]\right)$, the Picard iteration can provide us an approximation of w as a functional of the trajectory of v. That is, the sequence of functions $\left\{w^{\left(l\right)}\right\}$, defined by

$$w^{(l+1)}\left(t\right)=e^{-QA\tau }{w}^{\left(l\right)}(t-\tau )+{\int }_{t-\tau }^t{e}^{-QA(t-s)}[QB(v\left(s\right)+w^{\left(l\right)}\left(s\right))+Qf\left(s\right)]ds,$$

with $w^{\left(0\right)}\left(s\right)=0$ for $s\in [t-\tau ,t]$, will converge to w as $l\to \infty$. In particular, the first Picard iteration

$$w^{\left(1\right)}\left(t\right)={\int }_{t-\tau }^t{e}^{-QA(t-s)}[QB\left(v\left(s\right)\right)+Qf\left(s\right)]ds$$

(13)

provides us a closed representation: from its numerical integrator, we can derive parametric terms for the reduced model. We emphasize that the goal is to derive parametric terms for statistical inference, but not to have a trajectory-wise approximation. Thus, high-order numerical integrators or high-order Picard iterations are helpful but may complicate the parametrization. For simplicity, we shall consider only the first Picard iteration and Riemann sum approximation of this integral.

We can now propose parametric numerical reduced models from the above integro-differential equation. In a simple form, we parametrize both the Riemann sum approximation of the first Picard iteration and a numerical scheme of the differential equation to obtain

$$\begin{array}{cc}\hfill v\left(t_n\right)& \approx v\left(t_{n-1}\right)+a_1\delta R^{\delta }\left(v\left(t_{n-1}\right)\right)+a_2\delta Pf\left(t_{n-1}\right)+\delta [PB(v+w)-PB\left(v\right)]\left(t_{n-1}\right),\hfill \\ \hfill w\left(t_{n-1}\right)& \approx \sum _{j=0}^p{c}_j{e}^{-QAj\delta }[QB\left(v\left(t_{n-j}\right)\right)+Qf\left(t_{n-j}\right)].\hfill \end{array}$$

Here $\delta =t_n-t_{n-1}$ denotes the time step-size, the nonlinear function $R^{\delta }(·)$ comes from a numerical integration of the deterministic truncated Galerkin equation $\frac{dv}{dt}\approx -PAv+PB\left(v\right)$ at time $t_{n-1}$ and with time step-size $\delta$, and the coefficients $(a_1,a_2,c_j)$ are to be estimated by fitting to data in a statistical sense. To distinguish the approximate process in the reduced model from the original process, we denote it by $v^n$, and write the reduced model as

$$\begin{array}{}\mathrm{(14a)}& \hfill v^n& =v^{n-1}+a_1\delta R^{\delta }\left(v^{n-1}\right)+a_2\delta Pf\left(t_{n-1}\right)+\delta [PB(v^{n-1}+w^{n-1})-PB\left(v^{n-1}\right)]+g^n,\hfill \mathrm{(14b)}& \hfill w^{n-1}& =\sum _{j=1}^p{c}_j{e}^{-QAj\delta }[QB\left(v^{n-j}\right)+Qf\left(t_{n-j}\right)],\hfill \end{array}$$

where $\left\{g^n\right\}$ is a process representing the residual, can be assumed to be stochastic force for simplicity, but can also be assumed to be a moving average part to better capture the time correlation as in [13,46]. The second Equation (14b) does not have a residual term, as its goal is to provide a set of basis functions for the approximation of the forward map $v\left(t_n\right)=F(v\left(t_{n-1}\right),w\left(t_{n-1}\right),f)$ as in Equation (8), not to model the high modes.

Note that the time step-size $\delta$ can be relatively large, as long as the truncated Galerkin equation $\frac{dv}{dt}\approx -PAv+PB\left(v\right)$ of the slow variable v can be reasonably resolved. In general, such a step-size can be much larger than the time step-size needed to resolve the fast process w, because the effect of the unresolved fast process is “averaged” statistically when fitting the coefficients $(a_1,a_2,c_j)$ to data. Furthermore, the numerical error in the discretization is taken into account statistically.

Theoretically, the right-hand side of Equation (14a) is an approximation of the conditional expectation $\mathbb{E}\left[v\left(t_n\right)|v\left(t_{n-p:n-1}\right),Pf\left(t_{n-p:n-1}\right)\right]$, which is the optimal $L^2$ estimator of the forward map conditional on the information up time $t_{n-1}$. Here, the $L^2$ is with respect to the joint measure of the vector $(v\left(t_{·-p:·-1}\right),Pf\left(t_{·-p:·-1}\right))$, which is approximated by their joint empirical measure when fitting to data.

To avoid nonlinear optimization, the parametric form may be further simplified to be linearly dependent on the coefficients by dropping the terms that are nonlinear in the parameter, which is quadratic. In fact, recall that in the Burgers equation $B\left(u\right)=u{u}_x$ and $PB(v+w)-PB\left(v\right)=v_{x}w+v{w}_x+w{w}_x$. By dropping the interaction between the high modes $w{w}_x$ and approximating

$$PB(v^{n-1}+w^{n-1})-PB\left(v^{n-1}\right)\approx v_x^{n-1}{w}^{n-1}+v^{n-1}{w}_x^{n-1}$$

in (14a), we obtain a reduced model that depends linearly on the coefficients $\{a_j,c_j\}$.

3.2. The Numerical Reduced Model in Fourier Modes

We now write the reduced model in terms of the Fourier modes as in Equation (10).

As discussed in the above section, the major task is to parametrize the truncation error $PB{(v+w)}_k-PB{\left(v\right)}_k$. Recall that the operator P projects u to modes with wavenumber $1\le \left|k\right|\le K$ and that the bilinear function $PB{\left(v\right)}_k={\sum }_l{\widehat{u}}_l{\widehat{u}}_{k-l}$ (hereafter, to simplify notation, we also denote P and Q on the corresponding vector spaces of Fourier modes).

$$PB{(v+w)}_k-PB{\left(v\right)}_k=-\frac{ik}{2}\sum _{\left|l\right|>K\mathrm{or}|k-l|>K}{\widehat{u}}_l{\widehat{u}}_{k-l}.$$

(15)

Since the quadratic term $B\left(v\right)$ can only propagate energy from $({\widehat{u}}_k,1\le |k|\le K)$ to modes with wave numbers less than $2K+1$, we get only the high modes with wave numbers $K<\left|k\right|\le 2K$ when we compute w by a single iteration of $QB\left(v\right)$. (We use a single iteration for simplicity, but one can reach higher wave numbers by multiple iterations at the price of more complicated parametric forms.) Therefore, in a single iteration approximation, the truncated error will involve the first $2K$ Fourier modes:

$$PB{(v+w)}_k-PB{\left(v\right)}_k\approx -\frac{ik}{2}\sum _{\begin{array}{c}K<|k-l|\le 2K\\ \mathrm{or}K<\left|l\right|\le 2K\end{array}}{\widehat{u}}_l{\widehat{u}}_{k-l}.$$

Dropping the interaction between the high-modes to avoid nonlinear optimization in parameter estimation, we have

$$PB{(v+w)}_k-PB{\left(v\right)}_k\approx -\frac{ik}{2}\sum _{\begin{array}{c}|k-l|\le K,K<\left|l\right|\le 2K\\ \mathrm{or}\left|l\right|\le K,K<|k-l|\le 2K\end{array}}{\widehat{u}}_l{\widehat{u}}_{k-l}.$$

We approximate the high modes $({\widehat{u}}_k,K<|k|\le 2K)$ by a functional of low modes as in (14b),

$${\widehat{u}}_k\left(t_{n-1}\right)\approx \sum _{j=1}^p{c}_{k,j}{e}^{-QAj\delta }[{\tilde{u}}_k\left(t_{n-j}\right)+{\widehat{f}}_k\left(t_{n-1}\right)],K<\left|k\right|\le 2K$$

where ${\tilde{u}}_k$ is the high modes of the nonlinear function $B\left(v\right)$:

$${\tilde{u}}_k=QB{\left(v\right)}_k=-\frac{ik}{2}\sum _{\left|l\right|\le K,|k-l|\le K}{\widehat{u}}_l{\widehat{u}}_{k-l},\mathrm{for}K<\left|k\right|\le 2K.$$

Here $QB\left(v\right)$ only represents the modes up to wavenumber $2K$, due to the fact that quadratic nonlinearity only involves interaction between double wave-numbers. One can reach higher wave numbers by iterations of the quadratic interaction.

The truncation error term can now be linearly parametrized as

$$\begin{array}{cc}\hfill {[PB(v+w)-PB\left(v\right)]}_k\left(t_n\right))& \approx -\frac{i{q}_k}{2}\sum _{j=0}^p{c}_{k,j}{e}^{-QAj\delta }\sum _{\begin{array}{c}|k-l|\le K,K<\left|l\right|\le 2K\\ \mathrm{or}\left|l\right|\le K,K<|k-l|\le 2K\end{array}}{\tilde{u}}_l\left(t_n\right){\tilde{u}}_{k-l}\left(t_{n-j}\right),\hfill \end{array}$$

(16)

where we also denote ${\tilde{u}}_k={\widehat{u}}_k$ for $\left|k\right|\le K$ for simplicity of notation.

We have now reached a parametric numerical reduced model for the Fourier modes. Denote $u^n=(u_k^n,\left|k\right|\le K)\in {\mathbb{C}}^K$ the low-modes in the reduced model that approximates the original low modes $({\widehat{u}}_k\left(t_n\right),|k|\le K)$. The reduced model is

$$\begin{array}{}\mathrm{(17a)}& \hfill u_k^n& =u_k^{n-1}+\delta [R^{\delta }\left(u_{·}^{n-1}\right)+f_k^n+{\Phi }_k^n]+g_k^n,1\le k\le K,\hfill \mathrm{(17b)}& \hfill {\Phi }_k^n& =\sum _{j=1}^p\left[c_{k,j}^v{u}_k^{n-j}+c_{k,j}^R{R}^{\delta }\left(u_{·}^{n-j}\right)+c_{k,j}^f{f}_k^{n-j}+c_{k,j}^w\sum _{\begin{array}{c}|k-l|\le K,K<\left|l\right|\le 2K\\ \mathrm{or}\left|l\right|\le K,K<|k-l|\le 2K\end{array}}{\tilde{u}}_l^{n-1}{\tilde{u}}_{k-l}^{n-j}\right]\hfill \end{array}$$

with the convention that $u_{-k}^n={\left(u_k^n\right)}^{*}$ (with the sup-script ${}^{*}$ denoting complex conjugate), and where the notion ${\tilde{u}}_l^{n-j}$ represents the high modes and is defined by

$${\tilde{u}}_k^{n-j}=\left\{\begin{array}{cc}{u}_k^{n-j},\hfill & 1\le k\le K;\hfill \\ \frac{i{q}_k}{2}{e}^{-\nu q_k^{2}j\delta }\sum _{\left|l\right|\le K,|k-l|\le K}{u}_{k-l}^{n-j}{u}_l^{n-j},\hfill & K<k\le 2K.\hfill \end{array}\right.$$

(18)

The reduced model is in the form of a nonlinear auto-regression moving average (NARMA) model:

• The map $R^{\delta }(·):{\mathbb{C}}^K\to {\mathbb{C}}^K$ is the 1-step forward of the deterministic K-mode Galerkin truncation equation $\frac{dv}{dt}=-PAv+PB\left(v\right)$ using a numerical integration scheme with a time step-size $\delta$, i.e., $v^{n+1}=v^n+\delta R^{\delta }\left(v^n\right)$. We use the ETDRK4 scheme.
• The term $f_k^n$ denotes the increment of the k-th Fourier modes of the stochastic force in the time interval $[t_{n-1},t_n]$, scaled by $1/\delta$, and it is separated from $R^{\delta }$ so that the reduced model can linearly quantify the response of the low-modes to the stochastic force.
• The function ${\Phi }_k^n:={\Phi }_k^n(u^{n-p:n-1},f^{n-p:n-1})$ is a function ${\mathbb{C}}^{Kp+Kp}\to {\mathbb{C}}^K$ with parameters $\theta =(c^v,c^R,c^f,c^w)\in {\mathbb{R}}^{4Kp}$ to be estimated from data. In particular, the coefficients $c_{k,1}^v$ and $c_{k,1}^R$ act as a correction to the integration of the truncated equation.
• The new noise terms $\{g^n\in {\mathbb{C}}^K\}$ are assumed for simplicity to be a white noise independent of the original stochastic force $\left(f^n\right)$. That is, we assume that $\left\{g^n\right\}$ is a sequence of independent identically distributed (iid) Gaussian random vectors, with independent real and imaginary parts, distributed as $\mathcal{N}(0,\mathrm{Diag}\left({\sigma }_k^g\right))$ with ${\sigma }_k^g$ to be estimated from data. Under such a white noise assumption, the parameters can be estimated simply by least squares (see next section). In general, one can also assume other distributions for $g^n$, or other structures such as moving average $\{g^n:={\xi }_n+{\sum }_{j=1}^q{c}_j^g{\xi }_{n-j}\}$ with $\left\{{\xi }_n\right\}$ being a white noise sequence [13,46].

3.3. Data Generation and Parameter Estimation

We estimate the parameters of the NAR model by maximizing the likelihood of the data.

Data for the NAR model. To infer a reduced model in form of Equation (17), we generate relevant data from a numerical scheme that sufficiently resolve the system in space and time, as introduced in Section 2.2. The relevant data are trajectories of the low-modes of the state and the stochastic force, i.e., $\{{\widehat{u}}_k\left(t_n\right),{\widehat{f}}_k\left(t_n\right)\}$ for $\left|k\right|\le K$ and $n\ge 0$, which are taken as $\{u_k^n,f_k^n\}$ in the reduced model. Here, the time instants are $t_n=n\delta$, where $\delta$ can be much larger than the time step-size $dt$ needed to resolve the system. Furthermore, the data do not include the high modes. In short, the data are generated by a downsampling, in both space and time, of the high-resolution solutions of the system.

The data can be either a long trajectory or many independent short trajectories. We denote the data consisting of M independent trajectories by

$$\mathrm{Data}:{\{u_k^{1:N_t,m},f_k^{1:N_t,m}\}}_{m,k=1}^{M,K}\mathrm{with}{u}_k^{1:N_t,m}={\widehat{u}}_k{\left(t_{1:N_t}\right)}^{\left(m\right)},f_k^{1:N_t,m}={\widehat{f}}_k{\left(t_{1:N_t}\right)}^{\left(m\right)},$$

(19)

where m indexes the trajectories, $t_n=n\delta$ with $\delta$ being the time interval between two observations, and $N_t$ denotes the number of steps for each trajectory,

Parameter estimation. The parameters in the discrete-time reduced model Equation (17) is estimated by maximum likelihood methods. Our discrete-time reduced model has a few attractive features: (i) the likelihood function can be computed exactly, avoiding possible approximation error that could lead to biases in estimators; (ii) the maximum likelihood estimator (MLE) may be computed by least squares under the assumption that the process $\left\{g^n\right\}$ is white noise, avoiding time-consuming nonlinear optimizations.

Under the assumption that $\left\{g^n\right\}$ is white noise, the parameters can be estimated simply by least squares, because the reduced model in Equation (17) depends linearly on the parameters. More precisely, the log-likelihood of the data ${\{u^{1:N_t,m},f^{1:N_t,m}\}}_{m=1}^M$ in (19) can be written as

$$l(\theta ,{\sigma }^g)=-\sum _{\left|k\right|\le K}\left[log{\sigma }_k^g+\sum _{n,m=1}^{T,M}\frac{|u_k^{n,m}-u_k^{n-1,m}+\delta R^{\delta }\left(u_k^{n-1,m}\right)+\delta f_k^{n,m}+\delta {\Phi }_k^{n,m}\left(\theta \right){|}^2}{2MT{\sigma }_k^g}\right],$$

(20)

where $|·|$ denotes the absolute value of a complex number, $\theta =(c^v,c^R,c^f,c^w)\in {\mathbb{R}}^{4Kp}$ and ${\sigma }^g=({\sigma }_1^g,\dots ,{\sigma }_K^g)\in {\mathbb{R}}^K$. To compute the maximum likelihood estimator (MLE) of the parameter $(\theta ,{\sigma }^g)$, we note that ${\Phi }_k^n\left(\theta \right)$ in (17b) depends linearly on the parameter $\theta$. Therefore, the estimators of $\theta$ and ${\sigma }^g$ can be analytically computed by finding a zero of the gradient of the likelihood function. More precisely, denoting

$${\Phi }_k^n\left(\theta \right)=\sum _{j=1}^{4p}{\theta }_j{\Phi }_{k,j}^n$$

with ${\Phi }_{k,j}^n$ denoting the parameterized terms in (17b), we compute the MLE as

$$\begin{array}{cc}\hfill {\widehat{\theta }}_k& ={\left({\mathbf{A}}_k\right)}^{-1}{\mathbf{b}}_k,1\le k\le K,\hfill \\ \hfill {\widehat{\sigma }}_k^g& =\frac{1}{MT}\sum _{n,m=1}^{T,M}{\parallel u_k^{n,m}-u_k^{n-1,m}+\delta R^{\delta }(u_k^{n-1,m})+\delta f_k^{n,m}+\delta {\Phi }_k^{n,m}(\widehat{\theta })\parallel }^2\hfill \end{array}$$

(21)

where the normal matrix ${\mathbf{A}}_k$ and vector ${\mathbf{b}}_k$ are defined by

$$\begin{array}{cc}\hfill {\mathbf{A}}_k(j^{\prime },j)& =\frac{\delta }{MT}\sum _{n,m=1}^{T,M}⟨{\Phi }_{k,j^{\prime }}^{n,m},{\Phi }_{k,j}^{n,m}⟩,1\le j^{\prime },j\le 4p,\hfill \\ \hfill {\mathbf{b}}_k\left(j\right)& =\frac{1}{MT}\sum _{n,m=1}^{T,M}⟨u_k^{n,m}-u_k^{n-1,m}+\delta R^{\delta }(u_k^{n-1,m})+\delta f_k^{n,m},{\Phi }_{k,j}^{n,m}⟩.\hfill \end{array}$$

(22)

In practice, $A_k$ may be singular and it can be dealt with by pseudo inverse or regularization. We assume for simplicity that the stochastic force g has independent components, so that the coefficients can be estimated by simple least square regression. One may further improve the NAR model by considering spatial correlation between the components of g or by using moving average models [13,46,47].

3.4. Model Selection

The parametric form in Equation (17b) leaves a family of reduced models with many freedoms underdetermined, such as the time lag p and possible redundant terms. To avoid overfitting and redundancy, we proposed to select the reduced model by the following criterion.

• Cross validation: the reduced model should be stable and can reproduce the distribution of the resolved process, particularly the main dynamical-statistical properties. We will consider the energy spectrum, the marginal invariant densities, and temporal correlations:

  $$\begin{array}{cc}\hfill \mathrm{Energy}\mathrm{spectrum}:\mathbb{E}|{\widehat{u}}_k{|}^2& =\underset{N_{t}M\to \infty }{lim}\frac{1}{N_{t}M}\sum _{m,n=1}^{M,N_t}{\left|{\widehat{u}}_k{\left(t_n\right)}^{\left(m\right)}\right|}^2;\hfill \\ \hfill \mathrm{Invariant}\mathrm{density}\mathrm{of}\mathrm{Re}\left({\widehat{u}}_k\right):p_k\left(z\right)dz& =\underset{N_{t}M\to \infty }{lim}\frac{1}{N_{t}M}\sum _{m,n=1}^{M,N_t}{\mathbf{1}}_{(z,z+dz)}(\mathrm{Re}({\widehat{u}}_k{\left(t_n\right)}^{\left(m\right)});\hfill \\ \hfill \mathrm{Auto}-\mathrm{correlation}\mathrm{function}:{\mathrm{ACF}}_k\left(\tau \right)& =\mathbb{E}\left[\mathrm{Re}{\widehat{u}}_k(t+\tau )\mathrm{Re}{\widehat{u}}_k\left(t\right)\right]\hfill \\ \hfill & \approx \frac{1}{N_{t}M}\sum _{m,n=1}^{M,N_t}\mathrm{Re}({\widehat{u}}_k{(t_n+\tau )}^{\left(m\right)})\mathrm{Re}({\widehat{u}}_k{\left(t_n\right)}^{\left(m\right)});\hfill \end{array}$$

  (23)

  for $k=1,\dots ,K$.
• Consistency of the estimators. If the model is perfect and the data are either independent trajectories or a long trajectory from an ergodic measure, the estimators should converge as the data size increases (see e.g., [45,48]). While our parametric model may not be perfect, the estimators should also become less oscillatory as the data size increases, so that the algorithm is robust and can yield similar reduced models from different data sets.
• Simplicity and sparsity. When there are multiple reduced models performing similarly, we prefer the simplest model. We remove the redundant terms and enforce sparsity by LASSO (least absolute shrinkage and selection operator) regression [49]. Particularly, a singular normal matrix (22) indicates the redundancy of the terms and the need to remove strongly correlated terms.

These criteria are by no means exhaustive. Other methods include Bayesian information criterion (BIC, see, e.g., [50]), and the error reduction ratio [51] may be applied, but in our experience, they provide limited help for the selection of reduced models [7,14,46].

In view of statistical learning of the high-dimensional nonlinear flow map in (8), each linear-in-parameter reduced model provides an optimal approximation to the flow map in the hypothesis space spanned by the proposed terms. A possible future direction is to select adaptive-to-data hypothesis spaces in a nonparametric fashion [23] and analyze the distance between the flow map and the hypothesis space [52,53].

4. Numerical Study on Space-Time Reduction

We examine the inference and performance of NAR models for the stochastic Burgers equation in (1) and (2). We will consider two settings of the full model: the stochastic force has a scale of either $\sigma =1$ or $\sigma =0.2$, representing that the stochastic force either dominates or subordinates to the dynamics, respectively. We will also consider two settings for reduction: the number of the Fourier modes of the reduced model is either $K>K_0$ or $K<K_0$, representing a reduction of the deterministic responses and a reduction involving stochastic force, respectively.

4.1. Settings

As reviewed in Section 2.2, we integrate the Equation (4) of $2N$ Fourier modes by ETD-RK4 with a time-stepping $dt$ that the solution is resolved accurately. We call this discretized system the full model and its configuration is specified in Table 3. We will consider two different scales for the stochastic force, with standard deviations $\sigma =1$, leading to a dynamics dominated by the stochastic force, and $\sigma =0.2$, representing dynamics dominated by the deterministic drift.

Table 3. Settings of the full and reduced models.

Full model	$\nu =0.02$, $L=1$	viscosity, interval length of the equation
	$N=128$, $dt=0.001$	number of modes, time step-size
	$K_0=4$	number of modes in the stochastic force
	$\sigma =1\mathrm{or}0.2$	standard deviation of the stochastic force
Reduced models	$K=8\mathrm{or}2$	number of modes in the reduced model
	$\delta =dt\times \mathrm{Gap}$	observation time interval
	$\mathrm{Gap}\in \{5,10,20,30,40,50,80,160\}$	gap of time steps

We generate data in (19) from the full model as described in Section 3.3. We generate an ensemble of initial conditions by first integrating the system for ${10}^4$ time units from an initial condition $u_0\left(x\right)=sin\left(x\right)+2cos\left(x\right)$ and draw ${10}^3$ samples uniformly from this long trajectory. Then, we generate either a long trajectory or an ensemble of trajectories starting from randomly picked initial conditions, and we save data with the time-stepping $\delta$. Numerical tests show that the invariant densities and the correlation functions vary little when the data are generated from different initial conditions.

We then infer NAR models for the first K Fourier modes with a time step $\delta$. We will consider two values for K (recall that $K_0$ is the number of Fourier modes in the stochastic force)

• $K=8>K_0=4$. In this case, $Qf=0$, i.e., the stochastic force does not act on the unresolved Fourier modes w in (7), so w is a deterministic functional of the history of the resolved Fourier modes. In view of (14b), the reduced model mainly quantifies this deterministic map. We call this case “reduction of the deterministic response” and present the results in Section 4.3.
• $K=2<K_0$. In this case, $Qf\ne 0$, and w in (7) depend on the unobserved Fourier modes of the stochastic force. Thus, the reduced model has to quantify the effects of the unresolved Fourier modes of both the solution and the stochastic force. We call this case “reduction involving unresolved stochastic force” and present the results in Section 4.4.

In either case, we explore the maximal time step that NAR models can reach by testing time steps $\delta =dt\times \{5,10,20,30,40,50,80,160\}$.

We summarize the configurations and notations in Table 3.

4.2. Model Selection and Memory Length

We demonstrate model selection and the effect of memory length for reduced models with time step $\delta =5dt$. We aim to select a universal parametric form of the NAR model for different setting of $(K,\sigma )$, where $K\in \{8,2\}$ is the number of Fourier modes in the NAR model and $\sigma \in \{1,0.2\}$ is the standard deviation of the full model’s stochastic force. Such a parametric form will be used later for the exploration of maximal time reduction by NAR models in the next sections.

We select the model according to Section 3.4: for each pair $(K,\sigma )$, we test a pool of NAR models and select the simplest model that best reproduces the statistics and has consistent estimators. The statistics are computed along a long trajectory of $T=2000$ time units. We say that an NAR is numerically unstable if it blows up (e.g., $|u^n|$ exceeding ${10}^5$) before reaching $T=2000$ time units.

We estimate the coefficients in (17b) for a few time lag ps. Numerical tests show that the normal matrix in regression is almost singular, either when the stochastic force $f_k^{n-j}$ presents or when the lag for $u_k^{n-j}$ or $R^{\delta }\left(u^{n-j}\right)$ is bigger than two. Thus, for simplicity, we remove them by setting:

$$c_{k,j}^f=0\mathrm{for}\mathrm{all}1\le j\le p,\mathrm{and}{c}_{k,j}^v=c_{k,j}^R=0\mathrm{for}\mathrm{all}1<j\le p,$$

(24)

and estimate only $c_{k,1}^v,c_{k,1}^R,c_{k,j}^w$ for $1\le j\le p$.

That is, in (17b), the terms $u_k^{n-j}$ and $R^{\delta }\left(u^{n-j}\right)$ have a time lag 1, the stochastic force term $f_k^{n-j}$ is removed, and only the high-order (the fourth) term has a time lag p. The memory length is $p\delta$.

Memory length. To select a memory length, we test NAR models with time lags $p\in \{1,5,10,20\}$ and consider their reproduction of the energy spectrum in (23). Figure 1 shows the relative error in energy spectrum of these NAR models. It shows that as p increases: (1) when the scale of the stochastic force is large ($\sigma =1$), the error oscillates without a clear pattern; (2) when $\sigma =0.2$, the error first decreases and then increases. Thus, a longer memory does not necessarily lead to a better reduced model when the stochastic force dominates the dynamics; but when deterministic flow dominates the dynamics, a proper memory can be helpful.

Figure 1. Relative error in energy spectrum reproduced by the NAR models with different memory lengths p, in four settings of $(K,\sigma )$. As the time lag p increases, the relative error tends to first decrease and then increase, particularly in (b,d) with $\sigma =0.2$.

In all four settings, the simplest NAR models with $p=1$ can consistently reproduce the energy spectrum with relative errors within 5%. Remarkably, the accuracy remains when the true energy spectrum is at the scale of ${10}^{-2}$ for the modes with $k=7,8$ in Figure 2a,b and $k=2$ in Figure 2d. Figure 2 also shows that the truncated K-mode Galerkin systems cannot reproduce the true energy spectrum in any of these settings, with upward tails, due to the lack of fast energy dissipation from the high modes. Thus, the NAR model has introduced additional energy dissipation through ${\Phi }^n$.

Figure 2. Energy spectrum of NAR models with $p=1$ and the K-mode Galerkin systems in four settings of $(K,\sigma )$. The time step is $\delta =5dt$ for the NAR models and is $dt$ for the Galerkin models. The NAR models accurately reproduce the true energy spectrum in all settings.

Consistency of estimators. The estimator of the NAR models tends to converge as data size increases. Figure 3 shows that the estimated coefficients of NAR with $p=1$ from data consisting of M trajectories, each with length T, where $M\in \{2,8,32,128,512\}$ and $T\in \{40,80,160,320,640,1280\}$. As $T\times M$ increases, all the estimators tend to converge (note that the coefficients $c_{k,1}^w$ are at the scale of ${10}^{-4}$ or ${10}^{-3}$). In particular, they converge faster when $\sigma =1$ than when $\sigma =0.2$: the estimators in (a)–(c) oscillate little after $T\times M>{10}^3$, indicating that different trajectories lead to similar estimators, while the estimators (take $c_{K,1}^R$ for example) in (b)–(d) oscillate until $T\times M>{10}^5$. This agrees with the fact that a larger stochastic force makes the system mix faster, so each trajectory provides more effective samples driving the estimator to converge faster.

Figure 3. Estimated coefficients $(c_{k,1}^v,c_{k,1}^R,c_{k,j}^w)$ in NAR models with $p=1$ and $\delta =5dt$ in four settings of $(K,\sigma )$. The estimators tend to converge fast as the trajectory length T and number M increase: note that the coefficients $c_{k,1}^w$ are at the scale of ${10}^{-4}$ or ${10}^{-3}$.

Numerical tests also show that an NAR model can be numerically unstable, while its coefficient estimator was consistent (i.e., tending to converge as above). Thus, consistency is not sufficient for the selection of an NAR model.

In our tests, sparse regression algorithms such as LASSO (see e.g., [49]) or sequential thresholding (see e.g., [54,55]) have difficulty in proper thresholding, because the coefficient $c^w$ of the high order terms are small and can vary in scales in different settings, but these high order terms are important for the NAR model.

Since the NAR models with $p=1$ perform well in all the four settings, and since they are the simplest, we use them in the next sections to explore the maximal time reduction.

4.3. Reduction of the Deterministic Response

We explore in this and the next section the maximal time step $\delta$ that the NAR models can reach. We consider only the simplest models with time lag $p=1$.

We consider first the models with $K=8$ Fourier modes. Since the stochastic force acts directly only on the first $K_0=4$ Fourier modes, the unresolved variable w in (12) is a deterministic functional of the path of the K modes, so is the truncation error $PB(v+w)-PB\left(v\right)$ in (14b). Thus, the NAR model mainly reduces the deterministic response of the resolved variables to the unresolved variables. In particular, the term ${\Phi }^n$ in the NAR model (17a) optimally approximates this deterministic response on the function space linearly spanned by the terms in (17b).

We consider time steps $\delta =dt\times \mathrm{Gap}$ with $\mathrm{Gap}\in \{5,10,20,30,40,50\}$. For each $\delta$, we first estimate the coefficients $(c_{k,1}^v,c_{k,1}^R,c_{k,1}^w)$ of the NAR model from the data with the same time step. We then validate the estimated NAR model by its statistics.

Numerical tests show that the NAR models with $\mathrm{Gap}\ge 20$ are numerically unstable for the setting $(K=8,\sigma =1)$, and the number is $\mathrm{Gap}=50$ for the setting $(K=8,\sigma =0.2)$. Figure 4a,b shows the relative error in energy spectrum reproduced by NAR models with those stable time steps. The relative errors increase as the $\mathrm{Gap}$ increases. Note that the relative errors for modes $k=1,2$ change little, but those with $k\in \{3,4,5,6\}$ increase significantly. In particular, note that in (b), the relative errors at $k=8$ are about 8% for $\mathrm{Gap}\in \{20,30,40\}$, but the relative errors at $k\in \{3,4,5,6\}$ increase sharply to form a peak at $k=6$ when $\mathrm{Gap}=40$. We will discuss connections with CFL numbers in Section 4.5.

Figure 4. Relative error in energy spectrum reproduced by the NAR models with time steps $\delta =dt\times \mathrm{Gap}$ for $\mathrm{Gap}\in \{5,10,20,30,40,50\}$ in four settings of $(K,\sigma )$. All NAR models are with time lag $p=1$. The missing $\mathrm{Gap}$s in (a,b) lead to numerically unstable NAR models. Thus, the maximal $\delta$s that an NAR model can reach are $\delta \in [0.01,0.02)$ and $\delta \in [0.04,0.05)$ for (a,b) respectively, and $\delta \ge 0.16$ for (c,d).

These NAR models reproduce the PDFs and ACFs relatively accurately. Figure 5 shows the marginal PDFs of the real parts of the modes. The top row shows the marginal PDFs for the NAR models with $\mathrm{Gap}=5$, in comparison with those of the full model and the Galerkin truncated system (solved with time step $dt$). For the modes with wave numbers $k\in \{1,2,3,4\}$, the NAR model captures the shape and spread of the PDFs almost perfectly, improving those of the Galerkin truncated system. For the modes with $k\in \{5,6,7,8\}$, the NAR model still performs well, significantly improving those of the truncated Galerkin system. The discrepancy between the PDFs becomes larger as the wavenumber increases, because these modes are affected more by the unresolved modes. The bottom row shows that the Kolmogorov–Smirnov statistics (the maximal difference between the cumulative distribution functions) increase slightly as the $\mathrm{Gap}$ increases. Figure 6 shows the ACFs. The top row shows that both the NAR model (with $\mathrm{Gap}=5$) and the Galerkin system can reproduce the ACFs accurately. The bottom row shows that the relative error of the ACF, in $L^2\left([0,3]\right)$-norm, increases as $\mathrm{Gap}$ increases (particularly in the case $\sigma =0.2$). Recall that the truncated Galerkin system produces PDFs with support much wider than the truth for the high modes (see Figure 5), and that $R^{\delta }$ becomes less accurate as $\delta$ increases. Thus, the terms u and $R^{\delta }\left(u\right)$ in the NAR model (17) preserve the temporal correlation, and the high order term helps to dissipate energy and preserve the invariant measure.

Figure 5. Marginal PDFs and K-S statistics (Kolmogorov--Smirnov statistics, which is the maximum difference between the cumulative distribution functions). In each of (a–d), the top panels are plots of the empirical marginal PDFs of the real parts of the Fourier modes, from data (True), the K-mode Galerkin system (Galerklin) and the NAR models with $p=1$ and $\delta =\mathrm{Gap}dt$ with Gap = 5; the bottom panels are the K-S statistics of NAR models with different time steps $\delta =dt\times \mathrm{Gap}$, up to the largest Gap such that the NAR model is numerically stable.

Figure 6. ACF (auto correlation functions. In each of (a–d), the top panels are the ACFs of the real parts of the Fourier modes when Gap = 5; the bottom panels are the relative errors (in $L^2\left(\right[0,3\left]\right)$-norm) of the NAR models with different time steps $\delta =dt\times \mathrm{Gap}$, up to the largest Gap such that the NAR model is numerically stable.

In summary, when $K=8$, the maximal time steps are $\delta \in dt\times [10,20)=[0.01,0.02)$ and $\delta =dt\times [40,50)=[0.04,0.05)$ when $\sigma =1$ and $\sigma =0.2$, respectively, for NAR models with $p=1$. All these NAR models can accurately reproduce the energy spectrum, the invariant measure and the temporal autocorrelation.

4.4. Reduction Involving Unresolved Stochastic Force

We consider next NAR models with $K=2$. In this case, the unresolved variable w in (12) is a functional of both the path of the K modes and the unresolved stochastic force. Thus, in view of (14b), (16) and (17), the NAR model quantifies the response of the K-modes to both the unresolved Fourier modes and the unresolved stochastic force.

Note first that $K=2$ is too small for the K-mode Galerkin system to meaningfully reproduce any of the statistical or dynamical properties; see Figure 2c,d for the energy spectrum, Figure 5c,d for the marginal PDFs and Figure 6c,d for the ACFs. On the contrary, the NAR models with $\delta =5dt$, whose term $R^{\delta }$ comes from the K-mode Galerkin, reproduce these statistics accurately. Remarkably, the NAR models remain accurate even when the time step is as large as $\delta =80dt$, with the K-S statistics being less than $0.025$ in Figure 5c,d, and with the relative error in ACFs less than 6% in Figure 6c,d.

To explore the maximal time step that NAR models can reach, we consider time steps $\delta =dt\times \mathrm{Gap}$ with $\mathrm{Gap}\in \{5,10,20,40,80,160\}$. Numerical tests show that the NAR models are numerically stable for all of them in both settings of $\sigma =1$ and $\sigma =0.2$. Figure 4c,d shows the relative error in energy spectrum reproduced by NAR models with these time steps. The relative error first decreases and then increases as $\mathrm{Gap}$ increases, reaching the lowest when $\mathrm{Gap}=10$ and $\mathrm{Gap}=20$ for the settings $\sigma =1$ and $\sigma =0.2$, respectively. In particular, all of these relative errors remain less than 9%, except when $\mathrm{Gap}=160$ in the setting $\sigma =1$.

In summary, when $K=2$, NAR models can tolerate large time steps. The maximal time steps are at least $\delta =dt\times 80=0.08$ and $\delta dt\times 160=0.16$ when $\sigma =1$ and $\sigma =0.2$, respectively, for the NAR models to reproduce the energy spectrum with relative error less than 9%.

4.5. Discussion on Space-Time Reduction

Since model reduction aims for space-time reduction, it is natural to consider the maximal reduction in space-time; in other words, the minimum “spatial” dimension K and maximum time step $\delta =dt\times \mathrm{Gap}$. We have the following observations from the previous sections:

• Space dimension reduction, memory length of the reduced model and the stochastic force are closely related. As suggested by the discrete Mori–Zwanzig formalism for random dynamics (see e.g., [7]), space dimension reduction would lead to non-Markovian closure models. Figure 1 suggests that a proper medium length of the memory leads to best NAR model. It also suggests that the scale of the white in time stochastic force can affect the memory length, and a larger scale of stochastic force leads to shorter memory. We leave it as future work to investigate the relations between memory length (colored or white in time), stochastic force, and energy dissipation.
• Maximal time step depends on the space dimension and the scale of the stochastic force, mainly limited by the stability of the nonlinear reduced model. Figure 4 shows that the maximum time step when $K=2$ is at least $\delta =dt\times \mathrm{Gap}$ with $\mathrm{Gap}=160$, much larger than those of the case of $K=8$. It also shows that as the scale of stochastic force increases from $\sigma =0.2$ to $\sigma =1$, the NAR models’ maximal time step decreases (because the NAR models either become unstable or have larger errors in energy spectrum). It is noteworthy to mention that these maximal time steps of NAR models are smaller than those that the K-mode Galerkin system can tolerate. Figure 7 shows that the K-mode Galerkin system can be stable for time steps much larger than those of the NAR models: the maximal time step for the K-mode Galerkin system is when the mean CFL number (which increases linearly) reaches 1, but the maximal time step for the NAR models to be stable is smaller. For example, in the setting $(K=8,\sigma =0.2)$, the maximal time gap for the Galerkin system is $\mathrm{Gap}=80$ (the end of the red diamond line), but the maximal time gap for the NAR model is about $\mathrm{Gap}=10$. The increased numerical instability of the NAR model is likely due to the nonlinear terms ${\Phi }^n$, which are important for the NAR model to preserve energy dissipation and the energy spectrum (see Figure 2 and the coefficients in Figure 3).

Figure 7. The mean CFL numbers of the full models and the K-mode Galerkin systems. The mean CFL number is computed along a trajectory with ${10}^5$ steps. The time step is $dt=0.001$ for the full model, and is $\delta =dt\times \mathrm{Gap}$ for the K-mode Galerkin system. When $(\sigma =1,K=8)$, the K-mode Galerkin system blows up after $\mathrm{Gap}>80$, so its CFL number is missing afterwards. The stars are the maximal $\mathrm{Gap}$, such that the NAR model is stable. The red and blue squares are where the full model’s mean CFL numbers agree with those of the K-mode Galerkin systems. The stars (✩) are the largest time Gap that our NAR model is numerically stable. The relative errors in energy spectrum in Figure 4c,d are the smallest when the $\mathrm{Gap}$’s are the closest to these squares.

Beyond maximal reduction, an intriguing question arises: when does the reduced model perform the best (i.e., the least relative error in energy spectrum)? We call it optimality of space-time reduction. It is more interesting and relevant to model reduction than maximal reduction in space-time, because one may achieve a large time step or a small space dimension at the price of a large error in the NAR model, as we have seen in Figure 4. We note that the relative errors in energy spectrum in Figure 4c,d are the smallest when the $\mathrm{Gap}$s are the closest to the squares in Figure 7, where the full model’s mean CFL numbers agree with those of the K-mode Galerkin system. We conjecture that optimal space-time reduction can be achieved by an NAR model when the K-mode Galerkin system preserves the CFL number of the full model.

5. Conclusions

We consider a data-driven model reduction for stochastic Burgers equations, casting it as a statistical learning problem on approximating the flow map of low-wavenumber Fourier modes. We derive a class of efficient parametric reduced closure models, based on representing the high modes as functionals of the resolved variables’ trajectory. The reduced models are nonlinear autoregression (NAR) time series models, with coefficients estimated from data by least squares. In various settings, the NAR models can accurately reproduce the statistics such as the energy spectrum, the invariant densities, and the autocorrelations.

Using the simplest NAR model, we investigate the maximal space-time reduction in four settings: reduction of deterministic responses ($K>K_0$) vs. reduction involving unresolved stochastic force ($K<K_0$), and small vs. large scales of stochastic force (with $\sigma =0.2$ and $\sigma =1$), where $K_0$ is the number of Fourier modes of the white-in-time stochastic force, and $\sigma$ is the scale of the force. Reduction in space dimension is unlimited, and NAR models with $K=2$ Fourier modes can reproduce the energy spectrum with relative errors less than 5%. The time reduction is another story. Maximal time reduction depends on both the dimension reduction and the stochastic force’s scale, as they affect the stability of the NAR model. The NAR model’s stability limits the maximal time step to be smaller than those of the K-mode Galerkin system. Numerical tests indicate that the NAR models achieve the minimal relative error at the time step where the K-mode Galerkin system’s mean CFL number agrees with the full model’s. This is a potential criterion for optimal space-time reduction.

The simplicity of our NAR model structure opens various fronts for a further understanding of data-driven model reduction. Future directions include: (1) studying the connection between optimal space-time reduction, the CFL number, and quantification of the accuracy of reduced models; (2) investigating the relation between memory length, dimension reduction, the stochastic force, and the energy dissipation of the system; (3) developing post-processing techniques to efficiently recover information of the high Fourier modes, so as to predict the shocks using the reduced models.