Efficient Space–Time Reduced Order Model for Linear Dynamical Systems in Python Using Less than 120 Lines of Code

Abstract

A classical reduced order model (ROM) for dynamical problems typically involves only the spatial reduction of a given problem. Recently, a novel space–time ROM for linear dynamical problems has been developed [Choi et al., Space–tume reduced order model for large-scale linear dynamical systems with application to Boltzmann transport problems, Journal of Computational Physics, 2020], which further reduces the problem size by introducing a temporal reduction in addition to a spatial reduction without much loss in accuracy. The authors show an order of a thousand speed-up with a relative error of less than ${10}^{-5}$ for a large-scale Boltzmann transport problem. In this work, we present for the first time the derivation of the space–time least-squares Petrov–Galerkin (LSPG) projection for linear dynamical systems and its corresponding block structures. Utilizing these block structures, we demonstrate the ease of construction of the space–time ROM method with two model problems: 2D diffusion and 2D convection diffusion, with and without a linear source term. For each problem, we demonstrate the entire process of generating the full order model (FOM) data, constructing the space–time ROM, and predicting the reduced-order solutions, all in less than 120 lines of Python code. We compare our LSPG method with the traditional Galerkin method and show that the space–time ROMs can achieve $\mathcal{O}\left({10}^{-3}\right)$ to $\mathcal{O}\left({10}^{-4}\right)$ relative errors for these problems. Depending on parameter–separability, online speed-ups may or may not be achieved. For the FOMs with parameter–separability, the space–time ROMs can achieve $\mathcal{O}\left(10\right)$ online speed-ups. Finally, we present an error analysis for the space–time LSPG projection and derive an error bound, which shows an improvement compared to traditional spatial Galerkin ROM methods.

1. Introduction

Many computational models for physical simulations are formulated as linear dynamical systems. Examples of linear dynamical systems include, but are not limited to, the Schrödinger equation that arises in quantum mechanics, the computational model for the signal propagation and interference in electric circuits, storm surge prediction models before an advancing hurricane, vibration analysis in large structures, thermal analysis in various media, neuro-transmission models in the nervous system, various computational models for micro-electro-mechanical systems, and various particle transport simulations. These linear dynamical systems can quickly become large scale and computationally expensive, which prevents fast generation of solutions. Thus, areas in design optimization, uncertainty quantification, and control where large parameter sweeps need to be done can become intractable, and this motivates the need for developing a Reduced Order Model (ROM) that can accelerate the solution process without loss in accuracy.

Many ROM approaches for linear dynamical systems have been developed, and they can be broadly categorized as data-driven or non data-driven approaches. We give a brief background of some of the methods here. For the non data-driven approaches, there are several methods, including: balanced truncation methods [1,2,3,4,5,6,7,8,9], moment-matching methods [10,11,12,13,14], and Proper Generalized Decomposition (PGD) [15] and its extensions [16,17,18,19,20,21,22,23,24,25]. The balanced truncation method is by far the most popular method, but it requires the solution of two Lyapunov equations to construct bases, which is a computationally expensive task in large-scale problems using dense solvers. However, the fast and memory-efficient low rank solvers presented in [26,27] can be applied to the large-scale Lyapunov equations. On the other hand, this expensive task can be bypassed by various data-driven empirical and non-intrusive approaches [3,28,29,30,31]. Moment matching methods were originally developed as non data-driven, although later studies developed ideas to benefit from the data. They provide a computationally efficient framework using Krylov subspace techniques in an iterative fashion where only matrix–vector multiplications are required. The optimal $H_2$ tangential interpolation for nonparametric systems [11] is also available. Proper Generalized Decomposition was first developed as a numerical method for solving boundary value problems. It utilizes techniques to separate space and time for an efficient solution procedure and is considered a model reduction technique. For the detailed description of PGD, we refer to a short review paper [32]. Many data driven ROM approaches have been developed as well. When datasets are available either from experiments or high-fidelity simulations, these datasets can contain rich information about the system of interest and utilizing this in the construction of a ROM can produce an optimal basis. Although there are some data-driven moment matching works available [33,34], two popular methods are Dynamic Mode Decomposition (DMD) and Proper Orthogonal decomposition (POD). DMD generates reduced modes that embed an intrinsic temporal behavior and was first developed by Peter Schmid [35]. The method has been actively developed and extended to many applications [36,37,38,39,40,41,42,43]. For a more detailed description about DMD, we refer to this preprint [44] and book [45]. POD utilizes the method of snapshots to obtain an optimal basis of a system and typically applies only to spatial projections [46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61].

Although the data-driven ROMs have brought many benefits, such as speed-ups and accuracy, they have been mainly the reduction of the complexity on spatial domains. However, reducing the temporal domain complexity brings even more speed-ups for time dependent problems. Therefore, we focus on building a space–time ROM, where both spatial and temporal projections are applied to achieve an optimal reduction. Several space–time ROMs have been developed, including, but not limited to, the ones derived from the corresponding space–time full order models [62,63,64,65], the spectral POD and resolvent modes-based ones [66,67,68], and the ones that take advantage of the right singular vectors [69,70,71]. In particular, a space–time ROM for large-scale linear dynamical systems has been recently introduced [72]. The authors show a speed-up of greater than 8000 with good accuracy for a large-scale transport problem. However, the space–time ROM in [72] was limited to Galerkin projection, which can cause stability issues [22,73,74] for nonlinear problems. It is well-studied that the LSPG ROM formulation [75] brings various benefits, such as better stability, over Galerkin projection with some known issues, such as structure preservation [76]. Furthermore, it is not clear if the space–time least-squares Petrov–Galerkin (LSPG) brings any advantage over the space–time Galerkin approach for a linear dynamical system. The current manuscript addresses this aspect.

In this work, we present several contributions on the space–time ROM development for linear dynamical systems:

• We derive the block structures of least-squares Petrov–Galerkin (LSPG) space–time ROM operators for linear dynamical systems for the first time and compare them with the Galerkin space–time ROM operators.
• We present an error analysis for LSPG space–time ROMs for the first time and demonstrate the growth rate of the stability constant with the actual space–time operators used in our numerical results.
• We compare the performance between space–time Galerkin and space–time LSPG reduced order models on several linear dynamical systems.
• For each numerical problem, we cover the entire space–time ROM process in less than 120 lines of Python code, which includes sweeping a wide parameter space and generating data from the full order model, constructing the space–time ROM, and generating the ROM prediction in the online phase.

We note that the first two bullet points above are minor but useful contributions by making the block structure and error analysis readily available. Additionally, we hope that, by providing full access to the Python source codes, researchers can easily apply space–time ROMs to their linear dynamical problem of interest. Furthermore, we have curated the source codes to be simple and short so that it may be easily extended in various multi-query problem settings, such as design optimization [77,78,79,80,81,82,83], uncertainty quantification [84,85,86], and optimal control problems [87,88,89].

The paper is organized in the following way: Section 2 describes a parametric linear dynamical systems and space–time formulation. Section 3 introduces linear subspace solution representation in Section 3.1 and space–time ROM formulation using Galerkin projection in Section 3.2 and LSPG projection in Section 3.3. Then, both space–time ROMs are compared in Section 3.4. Section 4 describes how to generate space–time basis. We investigate block structures of space–time ROM basis in Section 5.1. We introduce the block structures of Galerkin space–time ROM operators derived in [72] in Section 5.2. In Section 5.3, we derive LSPG space–time ROM operators in terms of the blocks. Then, we compared Galerkin and LSPG block structures in Section 5.4. We show computational complexity of forming the space–time ROM operators in Section 5.5. The error analysis is presented in Section 6. We demonstrate the performance of both Galerkin and LSPG space–time ROMs in two numerical experiments in Section 7. Finally, the paper is concluded with summary and future works in Section 8. Appendix A presents six Python codes with less than 120 lines that are used to generate our numerical results.

2. Linear Dynamical Systems

We consider the parameterized linear dynamical system shown in Equation (1):

$$\begin{array}{cc}\hfill \frac{\partial \mathit{u}(t;\mathit{\mu })}{\partial t}& =\mathit{A}\left(\mathit{\mu }\right)\mathit{u}(t;\mathit{\mu })+\mathit{B}\left(\mathit{\mu }\right)\mathit{f}(t;\mathit{\mu }),\mathit{u}(0;\mathit{\mu })={\mathit{u}}^0\left(\mathit{\mu }\right),\hfill \end{array}$$

(1)

where $\mathit{\mu }\in {\Omega }_{\mu }\subset {\mathbb{R}}^{N_{\mu }}$ denotes a parameter vector, $\mathit{u}:[0,T]\times {\mathbb{R}}^{N_{\mu }}\to {\mathbb{R}}^{N_s}$ denotes a time dependent state variable function, ${\mathit{u}}^0:{\mathbb{R}}^{N_{\mu }}\to {\mathbb{R}}^{N_s}$ denotes an initial state, and $\mathit{f}:[0,T]\times {\mathbb{R}}^{N_{\mu }}\to {\mathbb{R}}^{N_i}$ denotes a time dependent input variable function. The operators $\mathit{A}:{\mathbb{R}}^{N_{\mu }}\to {\mathbb{R}}^{N_s\times N_s}$ and $\mathit{B}:{\mathbb{R}}^{N_{\mu }}\to {\mathbb{R}}^{N_s\times N_i}$ are real valued matrices that are independent of state variables.

Although any time integrator can be used, for the demonstration purpose, we choose to apply a backward Euler time integration scheme shown in Equation (2):

$$\left({\mathit{I}}_{N_s}-\Delta t^{\left(k\right)}\mathit{A}\left(\mathit{\mu }\right)\right){\mathit{u}}^{\left(k\right)}={\mathit{u}}^{(k-1)}+\Delta t^{\left(k\right)}\mathit{B}\left(\mathit{\mu }\right){\mathit{f}}^{\left(k\right)}\left(\mathit{\mu }\right),$$

(2)

where ${\mathit{I}}_{N_s}\in {\mathbb{R}}^{N_s\times N_s}$ is the identity matrix, $\Delta t^{\left(k\right)}$ is the kth time step size with $T={\sum }_{k=1}^{N_t}\Delta t^{\left(k\right)}$ and $t_{\left(k\right)}={\sum }_{j=1}^k\Delta t^{\left(j\right)}$, and ${\mathit{u}}^{\left(k\right)}\left(\mathit{\mu }\right):=\mathit{u}(t_{\left(k\right)};\mathit{\mu })$ and ${\mathit{f}}^{\left(k\right)}\left(\mathit{\mu }\right):=\mathit{f}(t_{\left(k\right)};\mathit{\mu })$ are the state and input vectors at the kth time step where $k\in \mathbb{N}\left(N_t\right)$. The Full Order Model (FOM) solves Equation (2) for every time step, where its spatial dimension is $N_s$ and the temporal dimension is $N_t$. Each time step of the FOM can be written out and put in another matrix system shown in Equation (3). This is known as the space–time formulation.

$${\mathit{A}}^{\mathrm{st}}\left(\mathit{\mu }\right){\mathit{u}}^{\mathrm{st}}\left(\mathit{\mu }\right)={\mathit{f}}^{\mathrm{st}}\left(\mathit{\mu }\right)+{\mathit{u}}_0^{\mathrm{st}}\left(\mathit{\mu }\right),$$

(3)

where

$$\begin{array}{c}{\mathit{A}}^{\mathrm{st}}\left(\mathit{\mu }\right)=\left[\begin{array}{cccc}{\mathit{I}}_{N_s}-\Delta t^{\left(1\right)}\mathit{A}\left(\mathit{\mu }\right)& & & \\ -{\mathit{I}}_{N_s}& {\mathit{I}}_{N_s}-\Delta t^{\left(2\right)}\mathit{A}\left(\mathit{\mu }\right)& & \\ & \ddots & \ddots & \\ & & -{\mathit{I}}_{N_s}& {\mathit{I}}_{N_s}-\Delta t^{\left(N_t\right)}\mathit{A}\left(\mathit{\mu }\right)\end{array}\right],\end{array}$$

(4)

$$\begin{array}{c}{\mathit{u}}^{\mathrm{st}}\left(\mathit{\mu }\right)=\left[\begin{array}{c}{\mathit{u}}^{\left(1\right)}\left(\mathit{\mu }\right)\\ {\mathit{u}}^{\left(2\right)}\left(\mathit{\mu }\right)\\ ⋮\\ {\mathit{u}}^{\left(N_t\right)}\left(\mathit{\mu }\right)\end{array}\right],\end{array}$$

(5)

$$\begin{array}{c}{\mathit{f}}^{\mathrm{st}}\left(\mathit{\mu }\right)=\left[\begin{array}{c}\Delta t^{\left(1\right)}\mathit{B}\left(\mathit{\mu }\right){\mathit{f}}^{\left(1\right)}\left(\mathit{\mu }\right)\\ \Delta t^{\left(2\right)}\mathit{B}\left(\mathit{\mu }\right){\mathit{f}}^{\left(2\right)}\left(\mathit{\mu }\right)\\ ⋮\\ \Delta t^{\left(N_t\right)}\mathit{B}\left(\mathit{\mu }\right){\mathit{f}}^{\left(N_t\right)}\left(\mathit{\mu }\right)\end{array}\right],\end{array}$$

(6)

$$\begin{array}{c}{\mathit{u}}_0^{\mathrm{st}}\left(\mathit{\mu }\right)=\left[\begin{array}{c}{\mathit{u}}^0\left(\mathit{\mu }\right)\\ \mathbf{0}\\ ⋮\\ \mathbf{0}\end{array}\right].\end{array}$$

(7)

The space–time system matrix ${\mathit{A}}^{\mathrm{st}}$ has dimensions ${\mathbb{R}}^{N_{\mu }}\to {\mathbb{R}}^{N_s{N}_t\times N_s{N}_t}$, the space–time state vector ${\mathit{u}}^{\mathrm{st}}$ has dimensions ${\mathbb{R}}^{N_{\mu }}\to {\mathbb{R}}^{N_s{N}_t}$, the space–time input vector ${\mathit{f}}^{\mathrm{st}}$ has dimensions ${\mathbb{R}}^{N_{\mu }}\to {\mathbb{R}}^{N_s{N}_t}$, and the space–time initial vector ${\mathit{u}}_0^{\mathrm{st}}$ has dimensions ${\mathbb{R}}^{N_{\mu }}\to {\mathbb{R}}^{N_s{N}_t}$. Although it seems that the solution can be found in a single solve, in practice, there is no computational saving gained from doing so since the block structure of the space–time system will solve the system in a time-marching fashion anyways. However, we formulate the problem in this way since our reduced order model (ROM) formulation can reduce and solve the space–time system efficiently. In the following sections, we describe the parametric Galerkin and LSPG ROM formulations.

3. Space–Time Reduced Order Models

We investigate two projection-based space–time ROM formulations: the Galerkin and LSPG formulations.

3.1. Linear Subspace Solution Representation

Both the Galerkin and LSPG methods reduce the number of space–time degrees of freedom by approximating the space–time state variables as a smaller linear combination of space–time basis vectors:

$${\mathit{u}}^{\mathrm{st}}\left(\mathit{\mu }\right)\approx {\tilde{\mathit{u}}}^{\mathrm{st}}\left(\mathit{\mu }\right)\equiv {\mathsf{\Phi }}_{\mathrm{st}}{\widehat{\mathit{u}}}^{\mathrm{st}}\left(\mathit{\mu }\right),$$

(8)

where ${\widehat{\mathit{u}}}^{\mathrm{st}}\left(\mathit{\mu }\right):{\mathbb{R}}^{N_{\mu }}\to {\mathbb{R}}^{n_s{n}_t}$ with $n_s\ll N_s$ and $n_t\ll N_t$. The space–time basis, ${\mathsf{\Phi }}_{\mathrm{st}}\in {\mathbb{R}}^{N_s{N}_t\times n_s{n}_t}$ is defined as

$${\mathsf{\Phi }}_{\mathrm{st}}\equiv \left[\begin{array}{cccc}{\mathit{\varphi }}_1^{\mathrm{st}}& \cdots & {\mathit{\varphi }}_{i+n_s(j-1)}^{\mathrm{st}}& \cdots {\mathit{\varphi }}_{n_s{n}_t}^{\mathrm{st}}\end{array}\right],$$

(9)

where $i\in \mathbb{N}\left(n_s\right)$, $j\in \mathbb{N}\left(n_t\right)$. Substituting Equation (8) into the space–time formulation in Equation (3) gives an over-determined system of equations:

$${\mathit{A}}^{\mathrm{st}}\left(\mathit{\mu }\right){\mathsf{\Phi }}_{\mathrm{st}}{\widehat{\mathit{u}}}^{\mathrm{st}}\left(\mathit{\mu }\right)={\mathit{f}}^{\mathrm{st}}\left(\mathit{\mu }\right)+{\mathit{u}}_0^{\mathrm{st}}\left(\mathit{\mu }\right)$$

(10)

This over-determined system of equations can be closed by either the Galerkin or LSPG projections.

3.2. Galerkin Projection

In the Galerkin formulation, Equation (10) is closed by the Galerkin projection, where both sides of the equation of multiplied by ${\mathsf{\Phi }}_{\mathrm{st}}^T$. Thus, we solve following reduced system for the unknown generalized coordinates, ${\widehat{\mathit{u}}}^{\mathrm{st}}$:

$${\mathsf{\Phi }}_{\mathrm{st}}^T{\mathit{A}}^{st}\left(\mathit{\mu }\right){\mathsf{\Phi }}_{\mathrm{st}}{\widehat{\mathit{u}}}^{\mathrm{st}}\left(\mathit{\mu }\right)={\mathsf{\Phi }}_{\mathrm{st}}^T{\mathit{f}}^{\mathrm{st}}\left(\mathit{\mu }\right)+{\mathsf{\Phi }}_{\mathrm{st}}^T{\mathit{u}}_0^{\mathrm{st}}\left(\mathit{\mu }\right).$$

(11)

For notational simplicity, let us define the reduced space–time system matrix as ${\widehat{\mathit{A}}}^{st,g}\left(\mathit{\mu }\right):={\mathsf{\Phi }}_{\mathrm{st}}^T{\mathit{A}}^{st}\left(\mathit{\mu }\right){\mathsf{\Phi }}_{\mathrm{st}}$, reduced space–time input vector as ${\widehat{\mathit{f}}}^{st,g}\left(\mathit{\mu }\right):={\mathsf{\Phi }}_{\mathrm{st}}^T{\mathit{f}}^{\mathrm{st}}\left(\mathit{\mu }\right)$, and reduced space–time initial state vector as ${\widehat{\mathit{u}}}_0^{st,g}\left(\mathit{\mu }\right):={\mathsf{\Phi }}_{\mathrm{st}}^T{\mathit{u}}_0^{\mathrm{st}}\left(\mathit{\mu }\right)$.

3.3. Least-Squares Petrov–Galerkin (LSPG) Projection

In the LSPG formulation, we first define the space–time residual as

$${\tilde{\mathit{r}}}^{\mathrm{st}}({\widehat{\mathit{u}}}^{\mathrm{st}};\mathit{\mu }):={\mathit{f}}^{\mathrm{st}}\left(\mathit{\mu }\right)+{\mathit{u}}_0^{\mathrm{st}}\left(\mathit{\mu }\right)-{\mathit{A}}^{\mathrm{st}}\left(\mathit{\mu }\right){\mathsf{\Phi }}_{\mathrm{st}}{\widehat{\mathit{u}}}^{\mathrm{st}}$$

(12)

where ${\tilde{\mathit{r}}}^{\mathrm{st}}:{\mathbb{R}}^{n_s{n}_t}\times {\mathbb{R}}^{N_{\mu }}\to {\mathbb{R}}^{N_s{N}_t}$. Note Equation (12) is an over-determined system. To close the system and solve for the unknown generalized coordinates, ${\widehat{\mathit{u}}}^{\mathrm{st}}$, the LSPG method takes the squared norm of the residual vector function and minimize it:

$${\widehat{\mathit{u}}}^{\mathrm{st}}=\underset{\widehat{\mathit{v}}\in {\mathbb{R}}^{n_s{n}_t}}{\mathrm{argmin}}\frac{1}{2}{∥{\tilde{\mathit{r}}}^{\mathrm{st}}(\widehat{\mathit{v}};\mathit{\mu })∥}_2^2.$$

(13)

The solution to Equation (13) satisfies

$${\left({\mathit{A}}^{st}\left(\mathit{\mu }\right){\mathsf{\Phi }}_{\mathrm{st}}\right)}^T{\tilde{\mathit{r}}}^{\mathrm{st}}({\widehat{\mathit{u}}}^{\mathrm{st}};\mathit{\mu })=\mathbf{0}$$

(14)

leading to

$${\mathsf{\Phi }}_{\mathrm{st}}^T{\mathit{A}}^{st}{\left(\mathit{\mu }\right)}^T{\mathit{A}}^{st}\left(\mathit{\mu }\right){\mathsf{\Phi }}_{\mathrm{st}}{\widehat{\mathit{u}}}^{\mathrm{st}}\left(\mathit{\mu }\right)={\mathsf{\Phi }}_{\mathrm{st}}^T{\mathit{A}}^{st}{\left(\mathit{\mu }\right)}^T{\mathit{f}}^{\mathrm{st}}\left(\mathit{\mu }\right)+{\mathsf{\Phi }}_{\mathrm{st}}^T{\mathit{A}}^{st}{\left(\mathit{\mu }\right)}^T{\mathit{u}}_0^{\mathrm{st}}\left(\mathit{\mu }\right).$$

(15)

For notational simplicity, let us define the reduced space–time system matrix as

$${\widehat{\mathit{A}}}^{st,pg}\left(\mathit{\mu }\right):={\mathsf{\Phi }}_{\mathrm{st}}^T{\mathit{A}}^{st}{\left(\mathit{\mu }\right)}^T{\mathit{A}}^{st}\left(\mathit{\mu }\right){\mathsf{\Phi }}_{\mathrm{st}},$$

reduced space–time input vector as ${\widehat{\mathit{f}}}^{st,pg}\left(\mathit{\mu }\right):={\mathsf{\Phi }}_{\mathrm{st}}^T{\mathit{A}}^{st}{\left(\mathit{\mu }\right)}^T{\mathit{f}}^{\mathrm{st}}\left(\mathit{\mu }\right)$, and reduced space–time initial state vector as ${\widehat{\mathit{u}}}_0^{st,pg}\left(\mathit{\mu }\right):={\mathsf{\Phi }}_{\mathrm{st}}^T{\mathit{A}}^{st}{\left(\mathit{\mu }\right)}^T{\mathit{u}}_0^{\mathrm{st}}\left(\mathit{\mu }\right)$.

3.4. Comparison of Galerkin and LSPG Projections

The reduced space–time system matrices, reduced space–time input vectors, and reduced space–time initial state vectors for Galerkin and LSPG projections are presented in

Table 1. Comparison of Galerkin and LSPG projections.

Galerkin	LSPG
${\widehat{\mathit{A}}}^{st,g}\left(\mathit{\mu }\right)={\mathsf{\Phi }}_{\mathrm{st}}^T{\mathit{A}}^{st}\left(\mathit{\mu }\right){\mathsf{\Phi }}_{\mathrm{st}}$	${\widehat{\mathit{A}}}^{st,pg}\left(\mathit{\mu }\right)={\mathsf{\Phi }}_{\mathrm{st}}^T{\mathit{A}}^{st}{\left(\mathit{\mu }\right)}^T{\mathit{A}}^{st}\left(\mathit{\mu }\right){\mathsf{\Phi }}_{\mathrm{st}}$
${\widehat{\mathit{f}}}^{st,g}\left(\mathit{\mu }\right)={\mathsf{\Phi }}_{\mathrm{st}}^T{\mathit{f}}^{\mathrm{st}}\left(\mathit{\mu }\right)$	${\widehat{\mathit{f}}}^{st,pg}\left(\mathit{\mu }\right)={\mathsf{\Phi }}_{\mathrm{st}}^T{\mathit{A}}^{st}{\left(\mathit{\mu }\right)}^T{\mathit{f}}^{\mathrm{st}}\left(\mathit{\mu }\right)$
${\widehat{\mathit{u}}}_0^{st,g}\left(\mathit{\mu }\right)={\mathsf{\Phi }}_{\mathrm{st}}^T{\mathit{u}}_0^{\mathrm{st}}\left(\mathit{\mu }\right)$	${\widehat{\mathit{u}}}_0^{st,pg}\left(\mathit{\mu }\right)={\mathsf{\Phi }}_{\mathrm{st}}^T{\mathit{A}}^{st}{\left(\mathit{\mu }\right)}^T{\mathit{u}}_0^{\mathrm{st}}\left(\mathit{\mu }\right)$

.

4. Space-Time Basis Generation

In this section, we repeat Section 4.1 in [72] to be self-contained.

A data set for space–time basis generation is chosen by the method of snapshots introduced by Sirovich [90]. This data set is called a snapshot matrix. By solving full order model problems at each of parameters in a parameter set $\{{\mathit{\mu }}_1,\dots ,{\mathit{\mu }}_{n_{\mu }}\}$, we obtain a full order model solution matrix ${\mathit{U}}_p$ at a certain parameter ${\mathit{\mu }}_p,p\in \mathbb{N}\left(n_{\mu }\right)$, i.e.,

$${\mathit{U}}_p\equiv \left[\begin{array}{cccc}{\mathit{u}}^{\left(1\right)}\left({\mathit{\mu }}_p\right)& \cdots & {\mathit{u}}^{\left(k\right)}\left({\mathit{\mu }}_p\right)& \cdots {\mathit{u}}^{\left(N_t\right)}\left({\mathit{\mu }}_p\right)\end{array}\right],$$

(16)

where ${\mathit{u}}^{\left(k\right)}(·)\in {\mathbb{R}}^{N_s}$ is a solution at a time step $k\in \mathbb{N}\left(N_t\right)$. Concatenating all the solution matrices for the every parameter yields the snapshot matrix, which is given by

$$\mathit{U}\equiv \left[\begin{array}{ccc}{\mathit{U}}_1& \cdots & {\mathit{U}}_{n_{\mu }}\end{array}\right]\in {\mathbb{R}}^{N_s\times n_{\mu }{N}_t}.$$

(17)

To generate the spatial basis ${\mathsf{\Phi }}_{\mathrm{s}}$, Proper Orthogonal Decomposition (POD) is used [49]. The POD procedure seeks the reduced dimensional subspace that optimally represents the solution snapshot, $\mathit{U}$. The spatial POD basis ${\mathsf{\Phi }}_{\mathrm{s}}$ is obtained by solving the following minimization problem:

$${\mathsf{\Phi }}_{\mathrm{s}}=\underset{\mathsf{\Phi }\in {\mathbb{R}}^{N_s\times n_s}}{\mathrm{argmin}}{∥\mathit{U}-\mathsf{\Phi }{\mathsf{\Phi }}^T\mathit{U}∥}_F^{2}s.t.{\mathsf{\Phi }}^T\mathsf{\Phi }={\mathit{I}}_{n_s},$$

(18)

where ${\parallel ·\parallel }_F$ denotes the Frobenius norm and ${\mathit{I}}_{n_s}\in {\mathbb{R}}^{n_s\times n_s}$ is the identity matrix with $n_s<n_{\mu }{N}_t$. Then, the solution to the minimization problem is obtained by choosing the leading $n_s$ columns of the left singular matrix of Singular Value Decomposition (SVD) of the snapshot matrix $\mathit{U}$:

$$\mathit{U}=\mathit{W}\mathsf{\Sigma }{\mathit{V}}^T$$

(19)

where $\mathit{W}\in {\mathbb{R}}^{N_s\times \ell }$ and $\mathit{V}\in {\mathbb{R}}^{n_{\mu }{N}_t\times \ell }$ are orthogonal matrices and $\mathsf{\Sigma }\in {\mathbb{R}}^{\ell \times \ell }$ is a diagonal matrix with singular values on its diagonal with $\ell \equiv min(N_s,n_{\mu }{N}_t)$.

Equation (19) can be written in the summation form

$$\mathit{U}=\sum _{i=1}^{\ell }{\sigma }_i{\mathit{w}}_i{\mathit{v}}_i^T,$$

(20)

where ${\sigma }_i\in {\mathbb{R}}^{}$ is the ith singular value, ${\mathit{w}}_i$ and ${\mathit{v}}_i$ are the ith left and right singular vectors, respectively. Here, $n_{\mu }$ different temporal behavior of ${\mathit{w}}_i$ are represented by ${\mathit{v}}_i$, e.g., ${\mathit{v}}_i^1$, ${\mathit{v}}_i^2$, and ${\mathit{v}}_i^3$ describe three different temporal behavior of a certain basis vector ${\mathit{w}}_i$ in Figure 1. Now, let us set ${\mathbf{Y}}_i=\left[\begin{array}{ccc}{\mathit{v}}_i^1& \cdots & {\mathit{v}}_i^{n_{\mu }}\end{array}\right]$ to be a temporal snapshot matrix associated with the ith spatial basis vector ${\mathit{w}}_i$, where ${\mathit{v}}_i^k\equiv {[{\mathit{v}}_i(1+(k-1)N_t),{\mathit{v}}_i(2+(k-1)N_t),\cdots ,{\mathit{v}}_i\left(k{N}_t\right)]}^T\in {\mathbb{R}}^{N_t}$ for $k\in \mathbb{N}\left(n_{\mu }\right)$, where ${\mathit{v}}_i\left(j\right),j\in \mathbb{N}\left(n_{\mu }{N}_t\right)$ is the jth component of the vector ${\mathit{v}}_i$. Then, applying SVD on ${\mathbf{Y}}_i$ gives us

$${\mathbf{Y}}_i={\mathsf{\Lambda }}_i{\mathsf{\Sigma }}_i{\mathsf{\Psi }}_i^T.$$

(21)

Leading $n_t$ vectors of the left singular matrix ${\mathsf{\Lambda }}_i$ become the temporal basis ${\mathsf{\Phi }}_{\mathrm{t}}^i$ which describes a temporal behavior of the ith column of spatial basis ${\mathsf{\Phi }}_{\mathrm{s}}$. Finally, we construct a space–time basis vector ${\mathit{\varphi }}_{i+n_s(j-1)}^{\mathrm{st}}\in {\mathbb{R}}^{N_s{N}_t}$ given in Equation (9) as

$${\mathit{\varphi }}_{i+n_s(j-1)}^{\mathrm{st}}={\mathit{\varphi }}_{ij}^{\mathrm{t}}\otimes {\mathit{\varphi }}_i^{\mathrm{s}},$$

(22)

where ⊗ denotes Kronecker product, ${\mathit{\varphi }}_i^{\mathrm{s}}\in {\mathbb{R}}^{N_s}$ is the ith column of the spatial basis ${\mathsf{\Phi }}_{\mathrm{s}}$ and ${\mathit{\varphi }}_{ij}^{\mathrm{t}}\in {\mathbb{R}}^{N_t}$ is the jth column of the temporal basis ${\mathsf{\Phi }}_{\mathrm{t}}^i$.

5. Space-Time Reduced Order Models in Block Structure

We avoid building the space–time basis vector defined in Equation (22) because it requires a lot of memory for storage. Instead, we can exploit the block structure of the matrices to save computational cost and storage of the matrices in memory. Section 5.2 introduces such block structures for the space–time Galerkin projection, while Section 5.3 shows block structures for the space–time LSPG projection. First of all, we introduce common block structures that appear both in the Galerkin and LSPG projections.

5.1. Block Structures of Space–Time Basis

Following [72]’s notation, we define the block structure of the space–time basis to be:

$${\mathsf{\Phi }}_{st}=\left(\begin{array}{ccccc}{\mathsf{\Phi }}_s{\mathit{D}}_1^1& \cdots & \cdots & \cdots & {\mathsf{\Phi }}_s{\mathit{D}}_1^{n_t}\\ ⋮& \ddots & ⋮& ⋰& ⋮\\ ⋮& \cdots & {\mathsf{\Phi }}_s{\mathit{D}}_k^j& \cdots & ⋮\\ ⋮& ⋰& ⋮& \ddots & ⋮\\ {\mathsf{\Phi }}_s{\mathit{D}}_{N_t}^1& \cdots & \cdots & \cdots & {\mathsf{\Phi }}_s{\mathit{D}}_{N_t}^{n_t}\end{array}\right)\in {\mathbb{R}}^{N_s{N}_t\times n_s{n}_t}$$

(23)

where the kth time step of the temporal basis matrix is a diagonal matrix defined as

$${\mathit{D}}_k^j=\left(\begin{array}{ccccc}{\varphi }_{1j,k}^t& 0& \cdots & \cdots & 0\\ 0& \ddots & 0& \cdots & ⋮\\ ⋮& 0& {\varphi }_{ij,k}^t& \cdots & ⋮\\ ⋮& ⋮& \ddots & \ddots & 0\\ 0& 0& \cdots & 0& {\varphi }_{n_{s}j,k}^t\end{array}\right)\in {\mathbb{R}}^{n_s\times n_s}$$

(24)

where ${\varphi }_{ij,k}^t\in \mathbb{R}$ is the kth element of ${\mathit{\varphi }}_{ij}^t\in {\mathbb{R}}^{N_t}$.

5.2. Block Structures of Galerkin Projection

As shown in Table 1, the reduced space–time Galerkin system matrix, ${\widehat{\mathit{A}}}^{st,g}\left(\mathit{\mu }\right)$ is:

$$\begin{array}{cc}\hfill {\widehat{\mathit{A}}}^{st,g}\left(\mathit{\mu }\right)=& {\mathsf{\Phi }}_{\mathrm{st}}^T{\mathit{A}}^{st}\left(\mathit{\mu }\right){\mathsf{\Phi }}_{\mathrm{st}}\hfill \end{array}$$

(25)

Now, we define the block structure of this matrix as:

$$\begin{array}{c}\hfill {\widehat{\mathit{A}}}^{st,g}\left(\mathit{\mu }\right)=\left(\begin{array}{ccccc}{\widehat{\mathit{A}}}_{(1,1)}^{st,g}\left(\mathit{\mu }\right)& \cdots & \cdots & \cdots & {\widehat{\mathit{A}}}_{(1,n_t)}^{st,g}\left(\mathit{\mu }\right)\\ ⋮& \ddots & ⋮& ⋰& ⋮\\ ⋮& \cdots & {\widehat{\mathit{A}}}_{(j^{\prime },j)}^{st,g}\left(\mathit{\mu }\right)& \cdots & ⋮\\ ⋮& ⋰& ⋮& \ddots & ⋮\\ {\widehat{\mathit{A}}}_{(n_t,1)}^{st,g}\left(\mathit{\mu }\right)& \cdots & \cdots & \cdots & {\widehat{\mathit{A}}}_{(n_t,n_t)}^{st,g}\left(\mathit{\mu }\right)\end{array}\right)\end{array}$$

(26)

so that we can exploit the block structure of these matrices such that we do not need to form the entire matrix. By inserting Equations (4) and (23) to Equation (25), we derive that the $(j^{\prime },j)$th block matrix ${\widehat{\mathit{A}}}_{(j^{\prime },j)}^{st,g}\left(\mathit{\mu }\right)\in {\mathbb{R}}^{n_s\times n_s}$ is:

$${\widehat{\mathit{A}}}_{(j^{\prime },j)}^{st,g}\left(\mathit{\mu }\right)=\sum _{k=1}^{N_t}\left({\mathit{D}}_k^{j^{\prime }}{\mathit{D}}_k^j-\Delta t^{\left(k\right)}{\mathit{D}}_k^{j^{\prime }}{\mathsf{\Phi }}_s^T\mathit{A}\left(\mathit{\mu }\right){\mathsf{\Phi }}_s{\mathit{D}}_k^j\right)-\sum _{k=1}^{N_t-1}{\mathit{D}}_{k+1}^{j^{\prime }}{\mathit{D}}_k^j.$$

(27)

The reduced space–time Galerkin input vector ${\widehat{\mathit{f}}}^{st,g}\left(\mathit{\mu }\right)\in {\mathbb{R}}^{n_s{n}_t}$ is

$${\widehat{\mathit{f}}}^{st,g}\left(\mathit{\mu }\right)={\mathsf{\Phi }}_{\mathrm{st}}^T{\mathit{f}}^{\mathrm{st}}\left(\mathit{\mu }\right).$$

(28)

Again, utilizing the block structure of matrices, we compute the jth block vector ${\widehat{\mathit{f}}}^{st,g}{\left(\mathit{\mu }\right)}_{\left(j\right)}\in {\mathbb{R}}^{n_t}$ to be:

$${\widehat{\mathit{f}}}^{st,g}{\left(\mathit{\mu }\right)}_{\left(j\right)}=\sum _{k=1}^{N_t}\Delta t^{\left(k\right)}{\mathit{D}}_k^j{\mathsf{\Phi }}_s^T\mathit{B}\left(\mathit{\mu }\right){\mathit{f}}^{\left(k\right)}\left(\mathit{\mu }\right).$$

(29)

Finally, the space–time Galerkin initial vector, ${\widehat{\mathit{u}}}_0^{st,g}\left(\mathit{\mu }\right)\in {\mathbb{R}}^{n_s{n}_t}$, can be computed as:

$${\widehat{\mathit{u}}}_0^{st,g}\left(\mathit{\mu }\right)={\mathsf{\Phi }}_{\mathrm{st}}^T{\mathit{u}}_0^{\mathrm{st}}\left(\mathit{\mu }\right)$$

(30)

where the jth block vector, ${\widehat{\mathit{u}}}_0^{st,g}{\left(\mathit{\mu }\right)}_{\left(j\right)}\in {\mathbb{R}}^{n_s}$, is:

$${\widehat{\mathit{u}}}_0^{st,g}{\left(\mathit{\mu }\right)}_{\left(j\right)}={\mathit{D}}_1^j{\mathsf{\Phi }}_s^T{\mathit{u}}_0\left(\mathit{\mu }\right).$$

(31)

5.3. Block Structures of LSPG Projection

In Section 3.3, the reduced space–time LSPG system matrix, ${\widehat{\mathit{A}}}^{st,pg}\left(\mathit{\mu }\right)$ is defined as

$$\begin{array}{cc}\hfill {\widehat{\mathit{A}}}^{st,pg}\left(\mathit{\mu }\right)=& {\mathsf{\Phi }}_{st}^T{\mathit{A}}^{st}{\left(\mathit{\mu }\right)}^T{\mathit{A}}^{st}\left(\mathit{\mu }\right){\mathsf{\Phi }}_{st}.\hfill \end{array}$$

(32)

Then, the block structure of this matrix can be defined as

$$\begin{array}{c}\hfill {\widehat{\mathit{A}}}^{st,pg}\left(\mathit{\mu }\right)=\left(\begin{array}{ccccc}{\widehat{\mathit{A}}}_{(1,1)}^{st,pg}\left(\mathit{\mu }\right)& \cdots & \cdots & \cdots & {\widehat{\mathit{A}}}_{(1,n_t)}^{st,pg}\left(\mathit{\mu }\right)\\ ⋮& \ddots & ⋮& ⋰& ⋮\\ ⋮& \cdots & {\widehat{\mathit{A}}}_{(j^{\prime },j)}^{st,pg}\left(\mathit{\mu }\right)& \cdots & ⋮\\ ⋮& ⋰& ⋮& \ddots & ⋮\\ {\widehat{\mathit{A}}}_{(n_t,1)}^{st,pg}\left(\mathit{\mu }\right)& \cdots & \cdots & \cdots & {\widehat{\mathit{A}}}_{(n_t,n_t)}^{st,pg}\left(\mathit{\mu }\right)\end{array}\right),\end{array}$$

(33)

where the size of each block matrix is $n_s\times n_s$. Instead of forming the ${\widehat{\mathit{A}}}^{st,pg}\left(\mathit{\mu }\right)$ directly from Equation (32), each block matrix ${\widehat{\mathit{A}}}_{(j^{\prime },j)}^{st,pg}\left(\mathit{\mu }\right)\in {\mathbb{R}}^{n_s\times n_s}$ is computed and put into its location. Inserting Equations (4) and (23) to Equation (32) gives us ${\widehat{\mathit{A}}}^{st,pg}\left(\mathit{\mu }\right)$ and the $(j^{\prime },j)$th block of this matrix is derived as

$$\begin{array}{cc}\hfill {\widehat{\mathit{A}}}_{(j^{\prime },j)}^{st,pg}\left(\mathit{\mu }\right)=& \sum _{k=1}^{N_t}\left[{\mathit{D}}_k^{j^{\prime }}{\mathsf{\Phi }}_s^T\left({\mathit{I}}_{N_s}-\Delta t^{\left(k\right)}\mathit{A}{\left(\mathit{\mu }\right)}^T\right)\left({\mathit{I}}_{N_s}-\Delta t^{\left(k\right)}\mathit{A}\left(\mathit{\mu }\right)\right){\mathsf{\Phi }}_s{\mathit{D}}_k^j\right]\hfill \\ \hfill & +\sum _{k=1}^{N_t-1}[{\mathit{D}}_k^{j^{\prime }}{\mathit{D}}_k^j-{\mathit{D}}_k^{j^{\prime }}{\mathsf{\Phi }}_s^T\left({\mathit{I}}_{N_s}-\Delta t^{(k+1)}\mathit{A}\left(\mathit{\mu }\right)\right){\mathsf{\Phi }}_s{\mathit{D}}_{k+1}^j\hfill \\ \hfill & -{\mathit{D}}_{k+1}^{j^{\prime }}{\mathsf{\Phi }}_s^T\left({\mathit{I}}_{N_s}-\Delta t^{(k+1)}\mathit{A}{\left(\mathit{\mu }\right)}^T\right){\mathsf{\Phi }}_s{\mathit{D}}_k^j].\hfill \end{array}$$

(34)

The reduced space–time LSPG input vector ${\widehat{\mathit{f}}}^{st,pg}\left(\mathit{\mu }\right)\in {\mathbb{R}}^{n_s{n}_t}$ is given by

$${\widehat{\mathit{f}}}^{st,pg}\left(\mathit{\mu }\right)={\mathsf{\Phi }}_{st}^T{\mathit{A}}^{st}{\left(\mathit{\mu }\right)}^T{\mathit{f}}^{\mathrm{st}}\left(\mathit{\mu }\right),$$

(35)

where the jth block vector ${\widehat{\mathit{f}}}^{st,pg}{\left(\mathit{\mu }\right)}_{\left(j\right)}\in {\mathbb{R}}^{n_t}$ is computed as

$$\begin{array}{cc}\hfill {\widehat{\mathit{f}}}^{st,pg}{\left(\mathit{\mu }\right)}_{\left(j\right)}=& \sum _{k=1}^{N_t}\left[{\mathit{D}}_k^j{\mathsf{\Phi }}_s^T\left({\mathit{I}}_{N_s}-\Delta t^{\left(k\right)}\mathit{A}{\left(\mathit{\mu }\right)}^T\right)\Delta t^{\left(k\right)}\mathit{B}\left(\mathit{\mu }\right){\mathit{f}}^{\left(k\right)}\left(\mathit{\mu }\right)\right]\hfill \\ \hfill & -\sum _{k=1}^{N_t-1}\left[{\mathit{D}}_k^j{\mathsf{\Phi }}_s^T\Delta t^{(k+1)}\mathit{B}\left(\mathit{\mu }\right){\mathit{f}}^{(k+1)}\left(\mathit{\mu }\right)\right].\hfill \end{array}$$

(36)

Lastly, the space–time LSPG initial vector ${\widehat{\mathit{u}}}_0^{st,pg}\left(\mathit{\mu }\right)\in {\mathbb{R}}^{n_s{n}_t}$ is

$${\widehat{\mathit{u}}}_0^{st,pg}\left(\mathit{\mu }\right)={\mathsf{\Phi }}_{st}^T{\mathit{A}}^{st}{\left(\mathit{\mu }\right)}^T{\mathit{u}}_0^{\mathrm{st}}\left(\mathit{\mu }\right).$$

(37)

By exploiting the block structure of the vector, we compute the jth block vector ${\widehat{\mathit{u}}}_0^{st,pg}{\left(\mathit{\mu }\right)}_{\left(j\right)}\in {\mathbb{R}}^{n_s}$ to be:

$${\widehat{\mathit{u}}}_0^{st,pg}{\left(\mathit{\mu }\right)}_{\left(j\right)}={\mathit{D}}_1^j{\mathsf{\Phi }}_s^T\left({\mathit{I}}_{N_s}-\Delta t^{\left(1\right)}\mathit{A}{\left(\mathit{\mu }\right)}^T\right){\mathit{u}}_0\left(\mathit{\mu }\right).$$

(38)

5.4. Comparison of Galerkin and LSPG Block Structures

The block structures of space–time reduced order model operators are summarized in

Table 2. Comparison of Galerkin and LSPG block structures.

Galerkin	LSPG
${\widehat{\mathit{A}}}^{st,g}{\left(\mathit{\mu }\right)}_{(j^{\prime },j)}=$	${\widehat{\mathit{A}}}^{st,pg}{\left(\mathit{\mu }\right)}_{(j^{\prime },j)}=$
${\displaystyle \sum _{k=1}^{N_t}}\left({\mathit{D}}_k^{j^{\prime }}{\mathit{D}}_k^j-\Delta t^{\left(k\right)}{\mathit{D}}_k^{j^{\prime }}{\mathsf{\Phi }}_s^T\mathit{A}\left(\mathit{\mu }\right){\mathsf{\Phi }}_s{\mathit{D}}_k^j\right)$	${\displaystyle \sum _{k=1}^{N_t}}\left[{\mathit{D}}_k^{j^{\prime }}{\mathsf{\Phi }}_s^T\left({\mathit{I}}_{N_s}-\Delta t^{\left(k\right)}\mathit{A}{\left(\mathit{\mu }\right)}^T\right)\left({\mathit{I}}_{N_s}-\Delta t^{\left(k\right)}\mathit{A}\left(\mathit{\mu }\right)\right){\mathsf{\Phi }}_s{\mathit{D}}_k^j\right]$
$-{\displaystyle \sum _{k=1}^{N_t-1}}{\mathit{D}}_{k+1}^{j^{\prime }}{\mathit{D}}_k^j$	$+{\displaystyle \sum _{k=1}^{N_t-1}}[{\mathit{D}}_k^{j^{\prime }}{\mathit{D}}_k^j-{\mathit{D}}_k^{j^{\prime }}{\mathsf{\Phi }}_s^T\left({\mathit{I}}_{N_s}-\Delta t^{(k+1)}\mathit{A}\left(\mathit{\mu }\right)\right){\mathsf{\Phi }}_s{\mathit{D}}_{k+1}^j$
	$-{\mathit{D}}_{k+1}^{j^{\prime }}{\mathsf{\Phi }}_s^T\left({\mathit{I}}_{N_s}-\Delta t^{(k+1)}\mathit{A}{\left(\mathit{\mu }\right)}^T\right){\mathsf{\Phi }}_s{\mathit{D}}_k^j]$
${\widehat{\mathit{f}}}^{st,g}{\left(\mathit{\mu }\right)}_{\left(j\right)}=$	${\widehat{\mathit{f}}}^{st,pg}{\left(\mathit{\mu }\right)}_{\left(j\right)}={\displaystyle \sum _{k=1}^{N_t}}\left[{\mathit{D}}_k^j{\mathsf{\Phi }}_s^T\left({\mathit{I}}_{N_s}-\Delta t^{\left(k\right)}\mathit{A}{\left(\mathit{\mu }\right)}^T\right)\Delta t^{\left(k\right)}\mathit{B}\left(\mathit{\mu }\right){\mathit{f}}^{\left(k\right)}\left(\mathit{\mu }\right)\right]$
${\displaystyle \sum _{k=1}^{N_t}}{\mathit{D}}_k^j\Delta t^{\left(k\right)}{\mathsf{\Phi }}_s^T\mathit{B}\left(\mathit{\mu }\right){\mathit{f}}^{\left(k\right)}\left(\mathit{\mu }\right)$	$-{\displaystyle \sum _{k=1}^{N_t-1}}\left[{\mathit{D}}_k^j{\mathsf{\Phi }}_s^T\Delta t^{(k+1)}\mathit{B}\left(\mathit{\mu }\right){\mathit{f}}^{(k+1)}\left(\mathit{\mu }\right)\right]$
${\widehat{\mathit{u}}}_0^{st,g}{\left(\mathit{\mu }\right)}_{\left(j\right)}={\mathit{D}}_1^j{\mathsf{\Phi }}_s^T{\mathit{u}}_0\left(\mathit{\mu }\right)$	${\widehat{\mathit{u}}}_0^{st,pg}{\left(\mathit{\mu }\right)}_{\left(j\right)}={\mathit{D}}_1^j{\mathsf{\Phi }}_s^T\left({\mathit{I}}_{N_s}-\Delta t^{\left(1\right)}\mathit{A}{\left(\mathit{\mu }\right)}^T\right){\mathit{u}}_0\left(\mathit{\mu }\right)$

.

5.5. Computational Complexity of Forming Space–Time ROM Operators

To show computational complexity of forming the reduced space–time system matrices ${\widehat{\mathit{A}}}^{st,g}\left(\mathit{\mu }\right)$ and ${\widehat{\mathit{A}}}^{st,pg}\left(\mathit{\mu }\right)$, input vectors ${\widehat{\mathit{f}}}^{st,g}\left(\mathit{\mu }\right)$ and ${\widehat{\mathit{f}}}^{st,pg}\left(\mathit{\mu }\right)$, and initial state vectors ${\widehat{\mathit{u}}}_0^{st,g}\left(\mathit{\mu }\right)$ and ${\widehat{\mathit{u}}}_0^{st,pg}\left(\mathit{\mu }\right)$ for Galerkin and LSPG projections, we assume that $\mathit{A}\left(\mathit{\mu }\right)\in {\mathbb{R}}^{N_s\times N_s}$ is a band matrix with the bandwidth, b and $\mathit{B}\left(\mathit{\mu }\right)$ is an identity matrix of size $N_s$ in Equation (1). The band structure of $\mathit{A}\left(\mathit{\mu }\right)$ is often seen in mathematical models because of local approximations to derivative terms. The structure of ${\mathit{A}}^{st}\left(\mathit{\mu }\right)\in {\mathbb{R}}^{N_s{N}_t\times N_s{N}_t}$ that is formed with backward Euler scheme is lower triangular matrix with bandwidth $N_s$. We also assume that the spatial basis vectors ${\mathit{\varphi }}_i^{\mathrm{s}}\in {\mathbb{R}}^{N_s},i\in \mathbb{N}\left(n_s\right)$ and temporal basis vectors ${\mathit{\varphi }}_{ij}^{\mathrm{t}}\in {\mathbb{R}}^{N_t},i\in \mathbb{N}\left(n_s\right)\mathrm{and}j\in \mathbb{N}\left(n_t\right)$ are given.

Let us start to show the computational complexity without use of block structures as presented in Table 1. Space–time basis ${\mathsf{\Phi }}_{\mathrm{st}}$ is constructed by Equation (22), and it involves $n_s\times n_t$ times Kronecker product of vector of size $N_s$ and vector of size $N_t$. Thus, it costs $\mathcal{O}\left(N_s{N}_t{n}_s{n}_t\right)$. For Galerkin projection, the reduced space–time system matrix ${\widehat{\mathit{A}}}^{st,g}\left(\mathit{\mu }\right)$ involves matrix multiplication twice. The first operation is matrix multiplication of $n_s{n}_t\times N_s{N}_t$ matrix and $N_s{N}_t\times N_s{N}_t$ lower triangular matrix with bandwidth $N_s$. Then, the second one is matrix multiplication of $n_s{n}_t\times N_s{N}_t$ matrix and $N_s{N}_t\times n_s{n}_t$ matrix. Thus, the cost of computing the reduce space–time system matrix, ${\widehat{\mathit{A}}}^{st,g}\left(\mathit{\mu }\right)$ is $\mathcal{O}\left(N_s^2{N}_t{n}_s{n}_t\right)+\mathcal{O}\left(N_s{N}_t{\left(n_s{n}_t\right)}^2\right)$. The input vector ${\widehat{\mathit{f}}}^{st,g}\left(\mathit{\mu }\right)$ requires $n_s{n}_t\times N_s{N}_t$ matrix and $N_s{N}_t\times 1$ vector multiplication, resulting in $\mathcal{O}\left(N_s{N}_t{n}_s{n}_t\right)$. The initial state vector ${\widehat{\mathit{u}}}_0^{st,g}\left(\mathit{\mu }\right)$ requires matrix ${\mathsf{\Phi }}_{\mathrm{st}}^T$ and vector ${\mathit{u}}_0^{\mathrm{st}}$ multiplication. Considering that the ${\mathit{u}}_0^{\mathrm{st}}$ contains zero values except the first $N_s$ components, the computational cost is $\mathcal{O}\left(N_s{n}_s{n}_t\right)$. Keeping the dominant terms leads to $\mathcal{O}\left(N_s^2{N}_t{n}_s{n}_t\right)+\mathcal{O}\left(N_s{N}_t{\left(n_s{n}_t\right)}^2\right)$. For LSPG projections, we first compute ${\mathsf{\Phi }}_{\mathrm{st}}^T{\mathit{A}}^{st}\left(\mathit{\mu }\right)$ which is going to be re-used, resulting in $\mathcal{O}\left(N_s^2{N}_t{n}_s{n}_t\right)$. Then, computing the reduced space–time system matrix ${\widehat{\mathit{A}}}^{st,pg}\left(\mathit{\mu }\right)$ requires multiplication of $n_s{n}_t\times N_s{N}_t$ precomputed matrix and its transpose matrix, resulting in $\mathcal{O}\left(N_s{N}_t{\left(n_s{n}_t\right)}^2\right)$. The input vector ${\widehat{\mathit{f}}}^{st,pg}\left(\mathit{\mu }\right)$ is computed by multiplication between $n_s{n}_t\times N_s{N}_t$ precomputed matrix and $N_s{N}_t\times 1$ vector with $\mathcal{O}\left(N_s{N}_t{n}_s{n}_t\right)$ operation counts. Constructing initial state vector ${\widehat{\mathit{u}}}_0^{st,pg}\left(\mathit{\mu }\right)$ costs $\mathcal{O}\left(N_s{n}_s{n}_t\right)$ because of $n_s{n}_t\times N_s{N}_t$ precomputed matrix and $N_s{N}_t\times 1$ vector whose components except first $N_s$ components are zero. Summing operation counts and keeping the dominant terms gives us $\mathcal{O}\left(N_s^2{N}_t{n}_s{n}_t\right)+\mathcal{O}\left(N_s{N}_t{\left(n_s{n}_t\right)}^2\right)$. With the assumption of $n_s{n}_t\ll N_s$, the computational cost without use of block structures for both Galerkin and LSPG projections is $\mathcal{O}\left(N_s^2{N}_t{n}_s{n}_t\right)$.

Now, let us show the computational complexity with the use of block structures. For Galerkin projection, we first compute ${\mathsf{\Phi }}_s^T\mathit{A}\left(\mathit{\mu }\right){\mathsf{\Phi }}_s$ which will be reused, resulting in $\mathcal{O}\left(2b{N}_s{n}_s\right)+\mathcal{O}\left(N_s{n}_s^2\right)$. Then, we compute $n_t^2$ blocks for the reduced space–time system matrix. Each block requires $2N_t$ times multiplication between diagonal matrices of size $n_s$ and $N_t$ times multiplication by diagonal matrices of size $n_s$ on the left and right of the precomputed $n_s\times n_s$ matrix, which takes $\mathcal{O}\left(N_t(2n_s^2+2n_s)\right)$. Thus, it costs $\mathcal{O}\left(N_t{n}_t^2(2n_s^2+2n_s)\right)+\mathcal{O}\left(2b{N}_s{n}_s\right)+\mathcal{O}\left(N_s{n}_s^2\right)$ to compute the reduced space–time system matrix ${\widehat{\mathit{A}}}^{st,g}\left(\mathit{\mu }\right)$. For the reduced space–time input vector ${\widehat{\mathit{f}}}^{st,g}\left(\mathit{\mu }\right)$, $n_t$ blocks are needed. Each block is computed by $n_s\times N_s$ matrix and $N_s\times 1$ vector multiplication followed by the diagonal matrix of size $n_s$ and $n_s\times 1$ vector multiplication, resulting in $\mathcal{O}\left(N_t(N_s{n}_s+n_s)\right)$. Thus, the cost of computing the reduced space–time input vector ${\widehat{\mathit{f}}}^{st,g}\left(\mathit{\mu }\right)$ is $\mathcal{O}\left(N_t{n}_t(N_s{n}_s+n_s)\right)$. Computing the reduced space–time initial vector ${\widehat{\mathit{u}}}_0^{st,g}\left(\mathit{\mu }\right)$ requires $n_s\times N_s$ matrix and $N_s\times 1$ vector multiplication followed by diagonal matrix of size $n_s$ and $n_s\times 1$ vector multiplication. Its cost is $\mathcal{O}(N_s{n}_s+n_s)$. Summing each computational cost for Galerkin projection gives us $\mathcal{O}\left(N_t{n}_t^2(2n_s^2+2n_s)\right)+\mathcal{O}\left(2b{N}_s{n}_s\right)+\mathcal{O}\left(N_s{n}_s^2\right)+\mathcal{O}\left(N_t{n}_t(N_s{n}_s+n_s)\right)+\mathcal{O}(N_s{n}_s+n_s)$. Keeping the dominant terms and taking off coefficient 2 lead to $\mathcal{O}\left(b{N}_s{n}_s\right)+\mathcal{O}\left(N_s{n}_s^2\right)+\mathcal{O}\left(N_t{\left(n_s{n}_t\right)}^2\right)+\mathcal{O}\left(N_s{N}_t{n}_s{n}_t\right)$. For LSPG projection, we compute $({\mathit{I}}_{N_s}-\Delta t\mathit{A}\left(\mathit{\mu }\right)){\mathsf{\Phi }}_s$ which will be re-used, resulting in $\mathcal{O}\left(2b{N}_s{n}_s\right)$. Then, we compute ${\mathsf{\Phi }}_s^T({\mathit{I}}_{N_s}-\Delta t\mathit{A}{\left(\mathit{\mu }\right)}^T)({\mathit{I}}_{N_s}-\Delta t\mathit{A}\left(\mathit{\mu }\right)){\mathsf{\Phi }}_s$, using precomputed $n_s\times N_s$ matrix and its transpose matrix. Its computational cost is $\mathcal{O}\left(N_s{n}_s^2\right)$. Similarly, when we compute ${\mathsf{\Phi }}_s^T({\mathit{I}}_{N_s}-\Delta t\mathit{A}\left(\mathit{\mu }\right)){\mathsf{\Phi }}_s$, the precomputed $n_s\times N_s$ matrix is multiplied by $N_s\times n_s$ matrix, resulting in computational complexity of $\mathcal{O}\left(N_s{n}_s^2\right)$. Now, we compute $n_t^2$ blocks for the reduced space–time system matrix ${\widehat{\mathit{A}}}^{st,pg}\left(\mathit{\mu }\right)$. Each block involves $3N_t$ times multiplication by diagonal matrices of size $n_s$ on the left and right side of $n_s\times n_s$ precomputed matrices and multiplication between diagonal matrices of size $n_s$, which gives us computational cost of $\mathcal{O}\left(N_t(6n_s^2+n_s)\right)$. Thus, it costs $\mathcal{O}\left(N_t{n}_t^2(6n_s^2+n_s)\right)$ to compute the reduced space–time system matrix ${\widehat{\mathit{A}}}^{st,pg}\left(\mathit{\mu }\right)$. For the reduced space–time input ${\widehat{\mathit{f}}}^{st,pg}\left(\mathit{\mu }\right)$ vector, $n_t$ blocks are needed and each block requires $2N_t$ times $n_s\times N_s$ matrix and $N_s\times 1$ vector multiplication followed by diagonal matrix of size $n_s$ and $n_s\times 1$ vector multiplication. The computational cost of computing each block is $\mathcal{O}\left(2N_t(N_s{n}_s+n_s)\right)$. Thus, it takes $\mathcal{O}\left(2N_t{n}_t(N_s{n}_s+n_s)\right)$ to compute the reduced space–time input ${\widehat{\mathit{f}}}^{st,pg}\left(\mathit{\mu }\right)$. Computing the reduced space–time initial vector ${\widehat{\mathit{u}}}_0^{st,pg}\left(\mathit{\mu }\right)$ costs $\mathcal{O}(N_s{n}_s+n_s)$ because we multiply the precomptued $n_s\times N_s$ matrix by a vector of size $N_s$; then, a diagonal matrix of size $n_s$ is multiplied on the left. Summing computational cost of ${\widehat{\mathit{A}}}^{st,pg}\left(\mathit{\mu }\right)$, ${\widehat{\mathit{f}}}^{st,pg}\left(\mathit{\mu }\right)$, and ${\widehat{\mathit{u}}}_0^{st,pg}\left(\mathit{\mu }\right)$ yields $\mathcal{O}\left(2b{N}_s{n}_s\right)+\mathcal{O}\left(2N_s{n}_s^2\right)+\mathcal{O}\left(N_t{n}_t^2(6n_s^2+n_s)\right)+\mathcal{O}\left(2N_t{n}_t(N_s{n}_s+n_s)\right)+\mathcal{O}(N_s{n}_s+n_s)$. Keeping the dominant terms and taking off coefficients 2 and 6 lead to $\mathcal{O}\left(b{N}_s{n}_s\right)+\mathcal{O}\left(N_s{n}_s^2\right)+\mathcal{O}\left(N_t{\left(n_s{n}_t\right)}^2\right)+\mathcal{O}\left(N_s{N}_t{n}_s{n}_t\right)$. With the assumptions of $b\ll N_t{n}_t$, $n_s\ll N_t{n}_t$, and $n_s{n}_t\ll N_s$, we have a computational cost of $\mathcal{O}\left(N_s{N}_t{n}_s{n}_t\right)$ for both Galerkin and LSPG projections with use of block structures.

In summary, the computational complexities of forming space–time ROM operators in the training phase for Galerkin and LSPG projections are presented in

Table 3. Comparison of Galerkin and LSPG computational complexities.

	Galerkin	LSPG
Not using block structures	$\mathcal{O}\left(N_s^2{N}_t{n}_s{n}_t\right)$	$\mathcal{O}\left(N_s^2{N}_t{n}_s{n}_t\right)$
Using block structures	$\mathcal{O}\left(N_s{N}_t{n}_s{n}_t\right)$	$\mathcal{O}\left(N_s{N}_t{n}_s{n}_t\right)$

. We observe that a lot of computational costs are reduced by making use of block structures for forming space–time reduced order models.

6. Error Analysis

We present error analysis of the space–time ROM method. The error analysis is based on [72]. An a posteriori error bound is derived in this section. Here, we drop the parameter dependence for notational simplicity.

Theorem 1.

We define the error at the kth time step as ${\mathit{e}}^{\left(k\right)}\equiv {\mathit{u}}^{\left(k\right)}-{\tilde{\mathit{u}}}^{\left(k\right)}\in {\mathbb{R}}^{N_s}$ where ${\mathit{u}}^{\left(k\right)}\in {\mathbb{R}}^{N_s}$ denotes FOM solution, ${\tilde{\mathit{u}}}^{\left(k\right)}\in {\mathbb{R}}^{N_s}$ denotes approximate solution, and $k\in \mathbb{N}\left(N_t\right)$. Let ${\mathit{A}}^{\mathrm{st}}\in {\mathbb{R}}^{N_s{N}_t\times N_s{N}_t}$ be the space–time system matrix, ${\mathit{r}}^{\left(k\right)}\in {\mathbb{R}}^{N_s}$ be the residual computed using FOM solution at the kth time step, and ${\tilde{\mathit{r}}}^{\left(k\right)}\in {\mathbb{R}}^{N_s}$ be the residual computed using approximate solution at the kth time step. For example, ${\mathit{r}}^{\left(k\right)}$ and ${\tilde{\mathit{r}}}^{\left(k\right)}$ after applying the backward Euler scheme with the uniform time step become

$$\begin{array}{cc}\hfill {\mathit{r}}^{\left(k\right)}({\mathit{u}}^{\left(k\right)},{\mathit{u}}^{(k-1)})& =\Delta t{\mathit{f}}^{\left(k\right)}+{\mathit{u}}^{(k-1)}-(\mathit{I}-\Delta t\mathit{A}){\mathit{u}}^{\left(k\right)}=\mathbf{0}\hfill \end{array}$$

(39)

$$\begin{array}{cc}\hfill {\tilde{\mathit{r}}}^{\left(k\right)}({\tilde{\mathit{u}}}^{\left(k\right)},{\tilde{\mathit{u}}}^{(k-1)})& =\Delta t{\mathit{f}}^{\left(k\right)}+{\tilde{\mathit{u}}}^{(k-1)}-(\mathit{I}-\Delta t\mathit{A}){\tilde{\mathit{u}}}^{\left(k\right)}\hfill \end{array}$$

(40)

with ${\tilde{\mathit{u}}}^{\left(0\right)}={\mathit{u}}^0$. Then, the error bound is given by

$$\underset{k\in \mathbb{N}\left(N_t\right)}{max}\parallel {\mathit{e}}^{\left(k\right)}{\parallel }_2\le \eta \underset{k\in \mathbb{N}\left(N_t\right)}{max}{\parallel {\tilde{\mathit{r}}}^{\left(k\right)}\parallel }_2$$

(41)

where $\eta \equiv \sqrt{N_t}{\parallel {\left({\mathit{A}}^{\mathrm{st}}\right)}^{-1}\parallel }_2$ denotes the stability constant.

Proof. 

Let us define the space–time residual as

$${\mathit{r}}^{\mathrm{st}}:\mathit{v}\mapsto {\mathit{f}}^{\mathrm{st}}+{\mathit{u}}_0^{\mathrm{st}}-{\mathit{A}}^{\mathrm{st}}\mathit{v}$$

(42)

with ${\mathit{r}}^{\mathrm{st}}:{\mathbb{R}}^{N_s{N}_t}\to {\mathbb{R}}^{N_s{N}_t}$. Then, we have

$$\begin{array}{cc}\hfill {\mathit{r}}^{\mathrm{st}}\left({\mathit{u}}^{\mathrm{st}}\right)& ={\mathit{f}}^{\mathrm{st}}+{\mathit{u}}_0^{\mathrm{st}}-{\mathit{A}}^{\mathrm{st}}{\mathit{u}}^{\mathrm{st}}=\mathbf{0}\hfill \end{array}$$

(43)

$$\begin{array}{cc}\hfill {\mathit{r}}^{\mathrm{st}}\left({\tilde{\mathit{u}}}^{\mathrm{st}}\right)& ={\mathit{f}}^{\mathrm{st}}+{\mathit{u}}_0^{\mathrm{st}}-{\mathit{A}}^{\mathrm{st}}{\tilde{\mathit{u}}}^{\mathrm{st}}\hfill \end{array}$$

(44)

where ${\mathit{u}}^{\mathrm{st}}\in {\mathbb{R}}^{N_s{N}_t}$ is the space–time FOM solution, and ${\tilde{\mathit{u}}}^{\mathrm{st}}\in {\mathbb{R}}^{N_s{N}_t}$ is the approximate space–time solution. Subtracting Equation (44) from Equation (43) gives

$${\mathit{r}}^{\mathrm{st}}\left({\tilde{\mathit{u}}}^{\mathrm{st}}\right)={\mathit{A}}^{\mathrm{st}}{\mathit{e}}^{st}$$

(45)

where ${\mathit{e}}^{st}\equiv {\mathit{u}}^{\mathrm{st}}-{\tilde{\mathit{u}}}^{\mathrm{st}}\in {\mathbb{R}}^{N_s{N}_t}$. Inverting ${\mathit{A}}^{\mathrm{st}}$ yields

$${\mathit{e}}^{st}={\left({\mathit{A}}^{\mathrm{st}}\right)}^{-1}{\mathit{r}}^{\mathrm{st}}\left({\tilde{\mathit{u}}}^{\mathrm{st}}\right).$$

(46)

Taking ${\ell }_2$ norm and Hölders’ inequality gives

$$\parallel {\mathit{e}}^{st}{\parallel }_2\le \parallel {\left({\mathit{A}}^{\mathrm{st}}\right)}^{-1}{\parallel }_2{\parallel {\mathit{r}}^{\mathrm{st}}\left({\tilde{\mathit{u}}}^{\mathrm{st}}\right)\parallel }_2.$$

(47)

We can rewrite this in the following form:

$$\sqrt{\sum _{k=1}^{N_t}{\parallel {\mathit{e}}^{\left(k\right)}\parallel }_2^2}\le {\parallel {\left({\mathit{A}}^{\mathrm{st}}\right)}^{-1}\parallel }_2\sqrt{\sum _{k=1}^{N_t}{\parallel {\tilde{\mathit{r}}}^{\left(k\right)}\parallel }_2^2}.$$

(48)

Using the relations

$$\underset{k\in \mathbb{N}\left(N_t\right)}{max}\parallel {\mathit{e}}^{\left(k\right)}{\parallel }_2^2\le \sum _{k=1}^{N_t}{\parallel {\mathit{e}}^{\left(k\right)}\parallel }_2^2$$

(49)

and

$$\sum _{k=1}^{N_t}\parallel {\tilde{\mathit{r}}}^{\left(k\right)}{\parallel }_2^2\le N_t\underset{k\in \mathbb{N}\left(N_t\right)}{max}{\parallel {\tilde{\mathit{r}}}^{\left(k\right)}\parallel }_2^2,$$

(50)

we have

$$\underset{k\in \mathbb{N}\left(N_t\right)}{max}\parallel {\mathit{e}}^{\left(k\right)}{\parallel }_2\le \sqrt{N_t}\parallel {\left({\mathit{A}}^{\mathrm{st}}\right)}^{-1}{\parallel }_2\underset{k\in \mathbb{N}\left(N_t\right)}{max}{\parallel {\tilde{\mathit{r}}}^{\left(k\right)}\parallel }_2,$$

(51)

which is equivalent to the error bound in (41). □

A numerical demonstration with space–time system matrices, ${\mathit{A}}^{\mathrm{st}}$, that have the same structure as the ones used in Section 7.1 and Section 7.2.1 shows the magnitude of $\parallel {\left({\mathit{A}}^{\mathrm{st}}\right)}^{-1}{\parallel }_2$ increases linearly for small $N_t$, while it becomes eventually flattened for large $N_t$ as shown in Figure 2a for the backward Euler time integrator with uniform time step size. Combined with $\sqrt{N_t}$, the stability constant $\eta$ growth rate is shown in Figure 2b. This error bound shows much improvement against the ones for the spatial Galerkin and LSPG ROMs, which grows exponentially in time [72].

7. Numerical Results

In this section, we apply both the space–time Galerkin and LSPG ROMs to two model problems: (i) a 2D linear diffusion equation in Section 7.1 and (ii) a 2D linear convection-diffusion equation in Section 7.2. We demonstrate their accuracy and speed-up. The space–time ROMs are trained with solution snapshots associated with train parameters in a chosen domain. Then, the space–time ROMs are used to predict the solution of a parameter that is not included in the trained parameter domain. We refer to this as the predictive case. The accuracy of space–time ROM solution ${\tilde{\mathit{u}}}^{\mathrm{st}}\left(\mathit{\mu }\right)$ is assessed from its relative error by:

$$\mathrm{relative}\mathrm{error}=\frac{{∥{\tilde{\mathit{u}}}^{\mathrm{st}}\left(\mathit{\mu }\right)-{\mathit{u}}^{\mathrm{st}}\left(\mathit{\mu }\right)∥}_2}{{∥{\mathit{u}}^{\mathrm{st}}\left(\mathit{\mu }\right)∥}_2}$$

(52)

and the ${\ell }_2$ norm of space–time residual:

$${∥{\mathit{r}}^{\mathrm{st}}\left({\tilde{\mathit{u}}}^{\mathrm{st}}\left(\mathit{\mu }\right)\right)∥}_2.$$

(53)

The computational cost is measured in terms of wall-clock time. The online phase includes all computations required to solve a problem with a new parameter. The online phase of FOM includes assembly of the FOM operators and time integration (i.e., FOM solving), and the online phase of the ROM consists of assembly of the ROM operators and time integration (i.e., ROM solving). The offline phase includes all computations that can be computed once and re-used for a new parameter. The offline phase of the FOM consists of computations that are independent of parameters. The offline phase of the ROM includes FOM snapshot generation, ROM basis generation, and parameter-independent ROM operator generation. Note that the parameter-independent ROM operators can be pre-computed during the offline phase only for parameter-separable cases. Then, the online speed-up is evaluated by dividing the wall-clock time of the FOM runtime by the online phase of the ROM. For the multi-query problems, total speed-up is evaluated by dividing the sum of runtime of all FOMs by the sum of elapsed time of the online and offline phase of all ROMs. All calculations are performed on an Intel(R) Core(TM) i9-10900T CPU @ 1.90 GHz and DDR4 Memory @ 2933 MHz.

In the following subsections, the backward Euler with uniform time step size is employed as a time integrator. For spatial differentiation, the second order central difference scheme for the diffusion terms and the first order backward difference scheme for the convective terms are implemented with uniform mesh size.

We conduct two kinds of tests for each model problem. In the first case, we compare relative errors, space–time residuals, and speed-ups of Galerkin and LSPG projections when the number of spatial basis vectors and temporal basis vectors are varied with the target parameter being fixed. We show the singular value decay of solution snapshot and temporal snapshot matrix for each problem.

We also show the accuracy (i.e., relative error) of the reduced order models versus the number of reduced basis. This clearly shows that the number of reduced basis affects the accuracy of the reduced order models, depending on the singular value decay. We observe that the relative errors of Galerkin projection are slightly smaller, but the space–time residual is slightly larger than LSPG projection. This is because LSPG space–time ROM solution minimizes the space–time residual as formulated in Section 3.3. However, the Galerkin and LSPG methods give very similar performance both in terms of accuracy and speed-up.

Secondly, we investigate the generalization capability of both Galerkin and LSPG ROMs by solving several predictive cases. We see that, as the parameter points go beyond the train parameter domain, the accuracy of the Galerkin and LSPG ROMs start to deteriorate gradually. This implies that the Galerkin and LSPG ROMs have a trust region. Its trust region should be determined by the application space. We also see that the Galerkin and LSPG space–time ROMs achieve total speed-up in Section 7.2 by taking advantage of parameter–separability but both space–time ROMs are not able to achieve total speed-up in Section 7.1 because we cannot take advantage of parameter–separability, which plays an important role in efficient parametric reduced order models.

7.1. 2D Linear Diffusion Equation

We consider a parameterized 2D linear diffusion equation with a source term

$$\begin{array}{cc}\hfill \frac{\partial u}{\partial t}=\left[\frac{{\partial }^{2}u}{\partial x^2}+\frac{{\partial }^{2}u}{\partial y^2}\right]& -\left[\frac{1}{\sqrt{{(x-{\mu }_1)}^2+{(y-{\mu }_2)}^2}}\right]u\hfill \\ \hfill & +\frac{sin2\pi t}{\sqrt{{(x-{\mu }_1)}^2+{(y-{\mu }_2)}^2}}\hfill \end{array}$$

(54)

where $(x,y)\in [0,1]\times [0,1]$, $t\in [0,2]$ and $({\mu }_1,{\mu }_2)\in [-1.7,-0.2]\times [-1.7,-0.2]$. The boundary condition is

$$\begin{array}{cc}\hfill u(x=0,y,t)& =0\hfill \\ \hfill u(x=1,y,t)& =0\hfill \\ \hfill u(x,y=0,t)& =0\hfill \\ \hfill u(x,y=1,t)& =0\hfill \end{array}$$

(55)

and the initial condition is

$$u(x,y,t=0)=0.$$

(56)

In Equation (54), the parameters are nonlinearly used in terms of operators, i.e., fractional, so we cannot take advantage of parameter–separability for efficient formation of the reduced order model.

The uniform time step size is given by $\frac{2}{N_t}$ where we set $N_t=50$. The space domain is discretized into $N_x=70$ and $N_y=70$ uniform meshes in x and y directions, respectively. Excluding boundary grid points, we have $N_s=(N_x-1)\times (N_y-1)=$ 4761 spatial degrees of freedom. Multiplying $N_t$ and $N_s$ gives us $\mathrm{238,050}$ free degrees of freedom in space–time.

For training phase, we collect solution snapshots associated with the following parameters:

$$({\mu }_1,{\mu }_2)\in \{(-0.9,-0.9),(-0.9,-0.5),(-0.5,-0.9),(-0.5,-0.5)\},$$

(57)

at which the FOM is solved. The singular value decays of the solution snapshot and the temporal snapshot are shown in Figure 3.

The Galerkin and LSPG space–time ROMs solve the Equation (54) with the target parameter $({\mu }_1,{\mu }_2)=(-0.7,-0.7)$. Figure 4, Figure 5 and Figure 6 show the relative errors, the space–time residuals, and the online speed-ups as a function of the reduced dimension $n_s$ and $n_t$. We observe that both Galerkin and LSPG ROMs with $n_s=5$ and $n_t=3$ achieve a good accuracy (i.e., relative errors of $0.012\%$ and $0.026\%$, respectively). However, they are not able to achieve an online speed-up (i.e., $0.082$ and $0.073$, respectively) because parameter–separability cannot be made use of. For such problems, local ROM operators can be constructed; then, we can use an interpolation to obtain ROM operator at a certain parameter point. For a detailed description of the local ROM interpolation method, please see [81]. By doing so, online speed-up can be achieved. Excluding assembly time of the FOM and ROM operators from the online phase yields $\mathcal{O}\left(10\right)$ solving speed-ups as shown in Figure 7. The Galerkin space–time ROM gives slightly lower relative error than the LSPG space–time ROM (see Figure 4), while the LSPG method gives a slightly lower space–time residual norm than the Galerkin method (see Figure 5), as we repeatedly stated in Section 7.

The final time snapshots of FOM, Galerkin space–time ROM, and LSPG space–time ROM are seen in Figure 8. Both ROMs have a basis size of $n_s=5$ and $n_t=3$, resulting in a reduction factor of $\left(N_s{N}_t\right)/\left(n_s{n}_t\right)=\mathrm{15,870}$. For the Galerkin method, the FOM and space–time ROM simulation with $n_s=5$ and $n_t=3$ take an average time of $3.9434\times {10}^{-2}$ and $5.0507\times {10}^{-1}$ s, respectively, resulting in online speed-up of $0.082$. For the LSPG method, the FOM and space–time ROM simulation with $n_s=5$ and $n_t=3$ take an average time of $3.9434\times {10}^{-2}$ and $5.0507\times {10}^{-1}$ s, respectively, resulting in speed-up of $0.078$. For accuracy, the Galerkin method results in $1.210\times {10}^{-2}$% relative error and $1.249\times {10}^{-2}$ space–time residual norm, while the LSPG results in $2.626\times {10}^{-2}$% relative error and $1.029\times {10}^{-2}$ space–time residual norm.

Next, we construct space–time ROMs with a basis of $n_s=5$ and $n_t=3$ using the same train parameter set given in Equation (57) for solving the predictive cases. The test parameter set is $({\mu }_1,{\mu }_2)\in \left\{{\mu }_1\right|{\mu }_1=-1.7+1.5/14i,i=0,1,\cdots ,14\}\times \left\{{\mu }_2\right|{\mu }_2=-1.7+1.5/14j,j=0,1,\cdots ,14\}$. Figure 9 shows the relative errors over the test parameter set. The relative errors of both projection methods are the lowest within the rectangular domain defined by the train parameter points, i.e., $[-0.9,-0.5]\times [-0.9,-0.5]$. For Galerkin ROM, online speed-up is about $0.088$ in average and total time for ROM and FOM are $105.89$ and $9.31$ s, respectively, resulting in total speed-up of $0.088$. The total time for the ROM consists of $0.13\%$ of FOM snapshot generation, $0.36\%$ of ROM basis generation, $99.19\%$ of ROM operator assembly, and $0.32\%$ of time integration. For LSPG ROM, online speed-up is about $0.080$ in average and total time for ROM and FOM are $114.87$ and $9.17$ s, respectively, resulting in total speed-up of $0.080$. The breakdown of the total time for the ROM is as follows: $0.12\%$ of FOM snapshot generation, $0.33\%$ of ROM basis generation, $99.27\%$ of ROM operator assembly, and $0.29\%$ of time integration.

7.2. 2D Linear Convection Diffusion Equation

7.2.1. Without Source Term

We consider a parameterized 2D linear convection diffusion equation

$$\frac{\partial u}{\partial t}=-{\mu }_1\left[\frac{\partial u}{\partial x}+\frac{\partial u}{\partial y}\right]+{\mu }_2\left[\frac{{\partial }^{2}u}{\partial x^2}+\frac{{\partial }^{2}u}{\partial y^2}\right]$$

(58)

where $(x,y)\in [0,1]\times [0,1]$, $t\in [0,1]$ and $({\mu }_1,{\mu }_2)\in [0.01,0.07]\times [0.31,0.37]$. The boundary condition is given by

$$\begin{array}{cc}\hfill u(x=0,y,t)& =0\hfill \\ \hfill u(x=1,y,t)& =0\hfill \\ \hfill u(x,y=0,t)& =0\hfill \\ \hfill u(x,y=1,t)& =0.\hfill \end{array}$$

(59)

The initial condition is given by

$$\begin{array}{c}\hfill u(x,y,t=0)=\left\{\begin{array}{cc}100sin{\left(2\pi x\right)}^3·sin{\left(2\pi y\right)}^3\hfill & \mathrm{if}(x,y)\in [0,0.5]\times [0,0.5]\hfill \\ 0\hfill & \mathrm{otherwise}\hfill \end{array}\right.\end{array}$$

(60)

and shown in Figure 10.

In this example, the parameters are used linearly in terms of operators, thus we can take advantage of parameter–separability for efficient projection of the full order model.

The time domain is discretized with the the uniform time step size $\frac{1}{N_t}$ with $N_t=50$. Discretizing the space domain into $N_x=70$ and $N_y=70$ uniform meshes in x and y directions, respectively and excluding boundary grid points give us $N_s=(N_x-1)\times (N_y-1)=$ 4761 grid points. As a result, there are 238,050 free degrees of freedom in space–time.

For the training phase, we collect solution snapshots associated with the following parameters:

$$({\mu }_1,{\mu }_2)\in \{(0.03,0.33),(0.03,0.35),(0.05,0.33),(0.05,0.35)\},$$

(61)

at which the FOM is solved. Figure 11 shows the singular value decay of the solution snapshot and the temporal snapshot.

The Galerkin and LSPG space–time ROMs solve the Equation (58) with the target parameter $({\mu }_1,{\mu }_2)=(0.04,0.34)$. Figure 12, Figure 13 and Figure 14 show the relative errors, the space–time residuals, and the online speed-ups as a function of the reduced dimension $n_s$ and $n_t$. We observe that both Galerkin and LSPG ROMs with $n_s=5$ and $n_t=3$ achieve a good accuracy (i.e., relative errors of $0.049\%$ and $0.059\%$, respectively) and speed-up (i.e., $26.52$ and $28.59$, respectively). Although the relative errors and space–time residuals are similar, the relative error of Galerkin space–time ROM is slightly lower than the LSPG space–time ROM as shown in Figure 12. On the other hand, Figure 13 shows that the space–time residual of the LSPG method is lower than the Galerkin method.

The final time snapshots of FOM, Galerkin space–time ROM, and LSPG space–time ROM are seen in Figure 15. Both ROMs have a basis size of $n_s=5$ and $n_t=3$, resulting in a reduction factor of $\left(N_s{N}_t\right)/\left(n_s{n}_t\right)=$ 15,870. For the Galerkin method, the FOM and space–time ROM simulation with $n_s=5$ and $n_t=3$ takes an average time of $3.4619\times {10}^{-2}$ and $1.3052\times {10}^{-3}$ s, respectively, resulting in speed-up of $26.52$. For the LSPG method, the FOM and space–time ROM simulation with $n_s=5$ and $n_t=3$ takes an average time of $3.4447\times {10}^{-2}$ and $1.2049\times {10}^{-3}$ s, respectively, resulting in speed-up of $28.59$. For accuracy, the Galerkin method results in $4.898\times {10}^{-2}$% relative error and $1.503$ space–time residual norm while the LSPG results in $5.878\times {10}^{-2}$% relative error and $1.459$ space–time residual norm.

Now, the space–time reduced order models with $n_s=5$ and $n_t=3$ are generated using the same train parameter set given in Equation (61) to solve the predictive cases with the test parameter set, $({\mu }_1,{\mu }_2)\in \left\{{\mu }_1\right|{\mu }_1=0.01+0.06/11i,i=0,1,\cdots ,11\}\times \left\{{\mu }_2\right|{\mu }_2=0.31+0.06/11j,j=0,1,\cdots ,11\}$. Figure 16 shows the relative errors over the test parameter set. The Galerkin and LSPG ROMs are the most accurate within the range of the train parameter points, i.e., $[0.03,0.33]\times [0.05,0.35]$. For Galerkin ROM, online speed-up is about $24.98$ in average and total time for ROM and FOM are $1.23$ and $5.62$ s, respectively, resulting in total speed-up of $4.59$. The total time for the ROM is composed of $10.97\%$ of FOM snapshot generation, $32.12\%$ of ROM basis generation, $38.38\%$ of parameter-independent ROM operator generation, $0.35\%$ of ROM operator assembly, and $18.18\%$ of time integration. For LSPG ROM, online speed-up is about $24.79$ in average and total time for ROM and FOM are $1.69$ and $5.57$ s, respectively, resulting in total speed-up of $3.30$. The total time for the ROM breaks down into $7.84\%$ of FOM snapshot generation, $22.99\%$ of ROM basis generation, $55.81\%$ of parameter-independent ROM operator generation, $0.38\%$ of ROM operator assembly, and $12.97\%$ time integration.

7.2.2. With Source Term

We consider a parameterized 2D linear convection diffusion equation

$$\frac{\partial u}{\partial t}=-{\mu }_1\left[0.1\frac{\partial u}{\partial x}+\frac{\partial u}{\partial y}\right]+{\mu }_2\left[\frac{{\partial }^{2}u}{\partial x^2}+\frac{{\partial }^{2}u}{\partial y^2}\right]+f(x,y,t)$$

(62)

with the source term $f(x,y,t)$ which is given by

$$f(x,y,t)={10}^{5}exp(-({(\frac{x-0.5+0.2sin(2\pi t)}{0.1})}^2+{(\frac{y}{0.05})}^2))$$

(63)

where $(x,y)\in [0,1]\times [0,1]$, $t\in [0,2]$ and $({\mu }_1,{\mu }_2)\in [0.195,0.205]\times [0.018,0.022]$. The boundary condition is given by

$$\begin{array}{cc}\hfill u(x=0,y,t)& =0\hfill \\ \hfill u(x=1,y,t)& =0\hfill \\ \hfill u(x,y=0,t)& =0\hfill \\ \hfill u(x,y=1,t)& =0\hfill \end{array}$$

(64)

and the initial condition is given by

$$u(x,y,t=0)=0.$$

(65)

Note that the parameters can be factored out when forming reduced order model operators in this example. Thus, we can avoid a lot of re-computation for a parametric case.

For time domain, we set the uniform time step size $\frac{2}{N_t}$ with $N_t=50$. For spatial domain, the number of uniform meshes in x and y directions are set $N_x=70$ and $N_y=70$, respectively. Considering free spatial degrees of freedom (i.e., excluding boundary grid points), we have $N_s=(N_x-1)\times (N_y-1)=4761$ grid points. Then, the number of space–time free degrees of freedom becomes 238,050.

For training phase, we collect solution snapshots associated with the following parameters:

$$({\mu }_1,{\mu }_2)\in \{(0.195,0.018),(0.195,0.022),(0.205,0.018),(0.205,0.022)\},$$

(66)

at which the FOM is solved. In Figure 17, we can see how the singular values of the solution snapshot and the temporal snapshot decay.

The Galerkin and LSPG space–time ROMs solve the Equation (62) with the target parameter $({\mu }_1,{\mu }_2)=(0.2,0.02)$. Figure 18, Figure 19 and Figure 20 show the relative errors, the space–time residuals, and the online speed-ups as a function of the reduced dimension $n_s$ and $n_t$. We observe that both Galerkin and LSPG ROMs with $n_s=19$ and $n_t=3$ achieve a good accuracy (i.e., relative errors of $0.217\%$ and $0.265\%$, respectively) and speed-up (i.e., $9.18$ and $8.54$, respectively). From Figure 18 and Figure 19, we find that there is the same observation that the LSPG method gives lower space–time residuals but higher relative errors as in Section 7.1 and Section 7.2.1.

The final time snapshots of FOM, Galerkin space–time ROM, and LSPG space–time ROM are seen in Figure 21. Both ROMs have a basis size of $n_s=19$ and $n_t=3$, resulting in a reduction factor of $\left(N_s{N}_t\right)/\left(n_s{n}_t\right)=$ 4176. For the Galerkin method, the FOM and space–time ROM simulation with $n_s=19$ and $n_t=3$ takes an average time of $3.9159\times {10}^{-2}$ and $4.2661\times {10}^{-3}$ s, respectively, resulting in speed-up of $9.18$. For the LSPG method, the FOM and space–time ROM simulation with $n_s=19$ and $n_t=3$ takes an average time of $3.6594\times {10}^{-2}$ and $4.2827\times {10}^{-3}$ s, respectively, resulting in speed-up of $8.54$. For accuracy, the Galerkin method results in $2.174\times {10}^{-1}$% relative error and $1.564\times {10}^3$ space–time residual norm while the LSPG results in $2.652\times {10}^{-1}$% relative error and $1.550\times {10}^3$ space–time residual norm.

Next, we set the test parameter set to be $({\mu }_1,{\mu }_2)\in \left\{{\mu }_1\right|{\mu }_1=0.160+0.08/11i,i=0,1,\cdots ,11\}\times \left\{{\mu }_2\right|{\mu }_2=0.016+0.008/11j,j=0,1,\cdots ,11\}$. Then, predictive cases with the test parameter set are solved using the space–time ROMs which are generated using the train parameter set given by Equation (66) with $n_s=19$ and $n_t=3$. The relative errors over the test parameter set are shown in Figure 22. We see that the Galerkin and LSPG ROMs are the most accurate inside region of the rectangle defined by the train parameter points, i.e., $[0.195,0.205]\times [0.018,0.022]$. For Galerkin ROM, online speed-up is about $8.94$ in average and total time for ROM and FOM are $1.82$ and $5.99$ s, respectively, resulting in total speed-up of $3.29$. The total time for the ROM breaks down into $7.82\%$ of FOM snapshot generation, $27.33\%$ of ROM basis generation, $28.04\%$ of parameter-independent ROM operator generation, $1.06\%$ of ROM operator assembly, and $35.75\%$ of time integration. For LSPG ROM, online speed-up is about $8.94$ in average and total times for ROM and FOM are $2.35$ and $6.11$ s, respectively, resulting in total speed-up of $2.60$. The total time of the ROM consists of $6.59\%$ of FOM snapshot generation, $22.70\%$ of ROM basis generation, $41.52\%$ of parameter-independent ROM operator generation, $1.09\%$ of ROM operator assembly, and $28.10\%$ of time integration.

8. Conclusions

In this work, we have formulated Galerkin and LSPG space–time ROMs for linear dynamical systems using block structures which enable us to implement the space–time ROM operators efficiently. We also presented an a posteriori error bound for both Galerkin and LSPG space–time ROMs. We demonstrated that both Galerkin and LSPG space–time ROMs solve 2D linear dynamical systems with parameter–separability accurately and efficiently. Both space–time reduced order models were able to achieve $\mathcal{O}\left({10}^{-3}\right)$ to $\mathcal{O}\left({10}^{-4}\right)$ relative errors with $\mathcal{O}\left(10\right)$ online speed-ups, and their differences were negligible. We also presented our Python codes used for the numerical examples in Appendix A so that readers can easily reproduce our numerical results. Furthermore, each Python code is less than 120 lines, demonstrating the ease of implementing our space–time ROMs.

We used a linear subspace based ROM which is suitable for accelerating physical simulations whose solution space has a small Kolmogorov n-width. However, the linear subspace based ROM is not able to represent advection-dominated or sharp gradient solutions with a small number of bases. To address this challenge, a nonlinear manifold based ROM can be used, and, recently, a nonlinear manifold based ROM has been developed for spatial ROMs [91,92,93]. In future work, we aim to develop a nonlinear manifold based space–time ROM. Another interesting future direction is to build a component-wise space–time ROM as an extension to the work in [94], which is attractive for extreme-scale problems, whose simulation data are too big to store in a given memory.