Reduction in Degrees of Freedom for Large-Scale Nonlinear Systems in Computer-Assisted Proofs

Abstract

In many physical systems, it is important to know the exact trajectory of a solution. Relevant applications include celestial mechanics, fluid mechanics, robotics, etc. For cases where analytical methods cannot be applied, one can use computer-assisted proofs or rigorous computations. One can obtain a guaranteed bound for the solution trajectory in the phase space. The application of rigorous computations poses few problems for low-dimensional systems of ordinary differential equations (ODEs) but is a challenging problem for large-scale systems, for example, systems of ODEs obtained from the discretization of the PDEs. A large-scale system size for rigorous computations can be as small as about a hundred ODE equations because computational complexity for rigorous algorithms is much larger than that for simple computations. We are interested in the application of rigorous computations to the problem of proving the existence of a periodic orbit in the Kolmogorov problem for the Navier–Stokes equations. One of the key issues, among others, is the computation complexity, which increases rapidly with the growth of the problem dimension. In previous papers, we showed that 79 degrees of freedom are needed in order to achieve convergence of the rigorous algorithm only for the system of ordinary differential equations. Here, we wish to demonstrate the application of the proper orthogonal decomposition (POD) in order to approximate the attracting set of the system and reduce the dimension of the active degrees of freedom.

1. Introduction

The question of rigorous computations arises in many theoretical and applied problems. These problems include particle physics and astronomy problems [1,2], global optimization [3], data analysis [4], electrical engineering [5], chemical processes [6], ray tracing [7], uncertainty bounds in Kalman filters [8], economics [9], etc. Theoretical problems are related to rigorous computations and verified proofs in graph theory, combinatorics [10], mathematical logic, hydrodynamics [11], dynamical systems [12] and other fields related with constructive algorithms. Interval arithmetic [13] is used in all these results in order to obtain guaranteed bounds of the computed values. It is interesting to note that some problems, like the 14th Smale’s problem, were probed only by using rigorous computations.

Let us consider a 2D Navier–Stokes system, subject to spatial periodic boundary conditions. The existence of stationary solutions and symmetry breaking bifurcations is proved in [14,15] using computer-assisted proofs. We wish to prove the existence of a periodic orbit (cycle) in this system using computer-assisted proofs, e.g., we wish to prove an existence of a closed curve in the infinite dimensional phase space of the system which is invariant under the action of the nonlinear operator, i.e., invariant curve. The general approach to solve this problem is the following:

• The infinite ODE system of equations is obtained from the Navier–Stokes system with spatial periodic boundary conditions and some restrictions, related with the reduction in the $SO\left(2\right)$ symmetric group by using the Galerkin method with trigonometric polynomials.
• The system is divided into three parts: (A), (B) and (C).
• The finite dimensional system (A) is numerically simulated until a stable periodic orbit is found numerically.
• The first part (A) is simulated using rigorous computations to obtain a rigorous bound on the periodic solution, thus proving that system (A) with the influence from (B) has an invariant curve.
• The second part of the system is passive and is not simulated, but its influence is taken into account during the simulation of the (A) system (via a nonlinear term). To validate the consistency of system (B), one needs to demonstrate that the phase flow of the infinite-dimensional system for the harmonics from (B) is directed towards the invariant curve that we wish to prove exists. This methodology is used for the 1D Burgers equation; see [16].
• The third part is the analysis of the infinite system (C), which is the remaining part of the system without the finite dimensional parts (A) and (B). It is known [17] (Equation 1.13) that starting from a certain number of the component, all components with larger numbers decay exponentially fast. This fact is exploited in order to include the bound on (C) to (B) to obtain a final bound on the contraction of the phase space by the infinite dimensional operator, thus proving that the invariant curve exists in the phase space of the original system.

By combining Steps 4–6, one can prove the existence of the invariant curve for the underlying problem.

Here, we are focusing on Step 4. In order to solve this step, the authors apply rigorous computation to prove the existence of periodic trajectories in systems of ordinary differential equations (ODEs); for example, see [18,19]. Several methods exist for this approach; however, all of them require the usage of interval arithmetic [13] and obtaining interval bounds of the ODE solutions; the bounds are derived from the Picard iterations used in the ODE systems that are extended on the interval arithmetic. Such approach (known as the Moore’s method [13]) results in the interval estimate exponentially increasing when moving along the trajectory: the so-called packing effect [20]. To overcome this drawback, a new object, the interval Taylor model, or simply the Taylor model (TM), was proposed in [2,21,22] by Martin Berz and co-authors and implemented in COSY software [23]. The Taylor Models are the polynomial generalization of a simple interval approach for rigorous computations in differential equations. The method of TM calculations requires $\mathcal{O}\left(m^{2n}\right)$ computational operations, where m is the number of operations required to calculate the right-hand side (RHS) of the ODE system and n is the degree of polynomials that are used in a TM. Such computational difficulty can become substantial for large m and n. To overcome these shortcomings, a different variant of Taylor Models was introduced by the authors in [19,24], where arithmetic operations, algorithms, estimates and rigorous theory is constructed both for new TMs and computer-assisted proofs of periodic trajectories. Such updated TMs now have $\mathcal{O}\left(m^n\right)$ computational complexity and are extended to parallel implementation of all required operations; see [25], where the authors presented generalization of these TMs on the parallel multiple graphic processing unit (GPU) computational architecture. This can, potentially, allow one performance of computer-assisted proofs of periodic orbits for ODE systems derived from PDE discretization (e.g., by Galerkin method), in other words, for a relatively large n. One way to prove the existence of periodic orbits in autonomous ODE systems is based on Poincare sections. It requires constructing estimates for the solution on a sufficiently large time interval, namely on an interval equal to the cycle period. In this case, one wishes to obtain as few remainders in TMs as possible on a relatively large time integration span. Since the degree of polynomials resides in the exponent in computational complexity, there are two ways to obtain reasonable computational times: reduce n and reduce m. In order to achieve the first reduction in n, we use shrink wrapping for TMs; see [26]. This allows us a decrease in n from four to two for a simple Van der Pol oscillator example to prove a periodic orbit using Poincare section approach. The second reduction requires the analysis of the underlying dynamical system and possible reduction in its dimension. The latter question is discussed in this paper.

We are interested in the application of proper orthogonal decomposition (POD) approach to the 2D Navier–Stokes system and the estimation of how well such reduction in dimensions can be retrofitted into the framework of interval TM analysis for further optimization. This 2D problem, though interesting by itself, is still a toy example that we use while attempting to prove the existence of periodic orbits in 3D Navier–Stokes equations.

The paper is laid out as follows. First, we formulate the problem and introducing the POD method into the presented system of equations and discuss methods of solving the underlying system. Next, we present the results on one of possible periodic orbits for the whole system dimension and reduced POD systems by the analysis of the distance between periodic trajectories, POD modes energies and spectral properties of monodromy matrices. The paper is finalized by a discussion. All calculations are carried out on a personal multiple GPU cluster.

2. Problem Formulation and Solution Methods

As discussed in the introduction, we are interested in the dimension reduction for the finite dimensional system (A) obtained from the Navier–Stokes system, which is derived below.

Governing Equations

The 2D Kolmogorov flow problem [27] of a viscous incompressible fluid on a 2D stretched periodic domain ${\mathbb{T}}^2\left(\alpha \right):=[2\pi /\alpha \times 2\pi ]$ is considered, where $\alpha <1$ is a stretching factor in the first direction. In the current case, $\alpha$ is selected such that $1/\alpha =2$. The problem is described by the Navier–Stokes equations. Vorticity formulation of these equations in the two-dimensional spatial case is

$$w_t+\left(\mathbf{u}·\nabla \right)w=R^{-1}△w+\beta cos\left(\beta y\right)$$

(1)

with

$$\begin{array}{c}w=\nabla \times \mathbf{u},u_1={(-{△}^{-1}w)}_y,u_2={\left({△}^{-1}w\right)}_x,\end{array}$$

where ${(·)}_j$ is a partial derivative in coordinate j, $\mathbf{u}={(u_1,u_2)}^T$, R is the Reynolds number and $\beta \ge 1$ is an integer. The nonlinear dynamics of this problem with different aspect ratios and values of $\beta$ was considered in [28,29], where limit cycles of different period and invariant tori in phase space are revealed using high-order Galerkin spline and spectral numerical methods. Currently, the system of ODEs is considered, which is derived from the Galerkin approximation of (1) in the space of trigonometric polynomials:

$$w(t,x,y)=\sum _{\{j,k\}\in {\mathbb{Z}}^2}{\widehat{w}}_{j,k}\left(t\right)exp\left(\mathrm{i}(\alpha jx+ky)\right).$$

(2)

This way, the periodic boundary conditions are automatically satisfied.

The finite dimensional system is formulated as the vector of time-dependent expansion coefficients $\widehat{w}\left(t\right)$ as

$${\left({\widehat{w}}_{j,k}\right)}_t=-R^{-1}({\alpha }^2{j}^2+k^2){\widehat{w}}_{j,k}+\beta /2{\delta }_{\beta }^{\left|k\right|}+\mathcal{B}(\widehat{w},\widehat{w})$$

(3)

for all $\{j,k\}\in S(\alpha ,M)$, with M being a finite positive integer, $S(\alpha ,M)$ being the space of indexes, $S^{*}(\alpha ,M)$ the space of conjugate harmonics indexes and

$${\displaystyle \mathcal{B}(\widehat{w},\widehat{w}):=\sum _{\begin{array}{c}-M/\alpha \le l\le M/\alpha ,\\ -M\le m\le M,\\ \{l,m\}\in S\cup S^{*}\end{array}}\left[\frac{\alpha m(j-l)}{{\alpha }^2{l}^2+m^2}{\widehat{w}}_{l,m}{\widehat{w}}_{(j-l),(k-m)}-\frac{\alpha l(m-k)}{{\alpha }^2{l}^2+m^2}{\widehat{w}}_{l,m}{\widehat{w}}_{(j-l),(k-m)}\right]}$$

(4)

the convolution term.

Additional reduction in degrees of freedom (DOF) is obtained by considering only real parts in (2) because of the following.

Proposition 1.

System (3) with Nonlinear term (4) maps the space of pure real coefficients into itself.

Proof. 

The linear part of System (3) (without the $\mathcal{B}\left(w\right)$ term) maps real vector to real vector, which can be easily checked by substitution. For the nonlinear part, we can observe that each term in the double summation can be written as $a{\widehat{w}}_1{\widehat{w}}_2$, where $a,{\widehat{w}}_1,{\widehat{w}}_2\in \mathbb{R}$. Hence, each term maps real numbers to real numbers and, by extension, the whole System (3) with Nonlinear part (4) preserves real valued vector of coefficients. □

In addition, we assume zero mean flow through the periodic domain, hence the zero coefficient ${\widehat{w}}_{0,0}$ is zero and also excluded from the consideration. Finally, this kind of reduction automatically removes the continuous group of translations of the system in the first direction, hence removing all traveling wave solutions in the form $w(t,x,y)=w_0(x+ct,y)$.

The space of indexes is defined by using the ellipsoid:

$$S(\alpha ,M):\{(j,k)\in {\mathbb{Z}}^2:{\left(\alpha j\right)}^2+k^2\le M^2,(j>0)\vee ((j=0)\wedge (k>0))\},$$

(5)

and the number of degrees of freedom N is defied by the number of points in S, i.e., $N=\left|S\right|$. The chosen ellipsoid space takes reality condition on the vector $\widehat{w}$ and Kolmogorov problem anisotropy [30] into account. Initial conditions of the system are taken such that the supposed system has an attracting periodic orbit. Initial conditions are provided in the Supplementary Material for all active DOFs.

If rigorous computations are used, then each vector $\widehat{w}$ becomes a TM with a polynomial of degree n; this was demonstrated in [25]. Hence, the total number of computational operations is $\mathcal{O}\left(N^{2+n}\right)$, where $\mathcal{O}\left(N^2\right)$ is the cost of Convolution term (4).

Now, we formulate the POD method [31] in the form that it is used for this problem. For a general overview of the POD method in fluid dynamics as well as different approaches and other methods, we address the reader to an excellent review [32].

Usually, one formulates the snapshot matrix by the method of snapshots [33] in the following way. We suppose that the discrete set $\left\{\tau \right\}$ of timesteps is available with $\mathit{dim}\left(\right\{\tau \left\}\right)=Q$. For each element $t_k\in \left\{\tau \right\}$, we have a vector $\widehat{w}\left(t_k\right)\in {\mathbb{R}}^N$ that is obtained from the numerical simulation of the problem. The matrix $X\in {\mathbb{R}}^{N\times Q}$ can be formed by stacking the vectors as columns of the matrix X and removing the average value of the vector. It is assumed that the most significant flow details are represented in the chosen timesteps. In the original and followed ideas, $Q<<N$, but it represents some significant details of the flow that can describe the underlying dynamics of the problem. In our case, we use $Q>>N$, since N is not large (but large for rigorous computations, since the total number of computational operations is $\mathcal{O}\left(N^4\right)$ for TM polynomials of degree 2 and $\mathcal{O}\left(N^6\right)$ for TM polynomials of degree 4). The set $\left\{\tau \right\}$ is formulated around the calculated assumed periodic orbit and is usually some partition of the closed loop such that whole orbit is well represented. Each row of matrix X has a zero mean value.

Next, the Singular Value Decomposition (SVD) is executed on the real matrix X, deriving orthogonal matrix $U\in {\mathbb{R}}^{N\times N}$, rectangular matrix $\mathsf{\Sigma }\in {\mathbb{R}}^{N\times Q}$ of singular values located on the diagonal and orthogonal matrix $V\in {\mathbb{R}}^{Q\times Q}$, s.t.,

$$U\mathsf{\Sigma }{V}^T=X.$$

From this decomposition, we obtain the POD matrix U that has the property

$$U\widehat{p}=\widehat{w},\widehat{p}=U^T\widehat{w},$$

where $\widehat{p}$ are the POD vector of unknowns. If singular values in $\mathsf{\Sigma }$ are sorted in descending order, then the first elements of $\widehat{p}$ contain modes with the most energy.

A system in POD coordinates can be written as

$${\widehat{p}}_t=-R^{-1}{U}^{T}LU\widehat{p}+\beta /2U^{T}d+U^T\mathcal{B}(U\widehat{p},U\widehat{p}),$$

(6)

with

$$L:=\mathrm{diag}({\alpha }^2{j}^2+k^2),\forall (j,k)\in S(\alpha ,M),$$

$$d:=\left\{\begin{array}{c}1,\beta =\left|k\right|\wedge j=0\\ 0,\mathrm{otherwise}\end{array}\right.\forall (j,k)\in S(\alpha ,M),$$

and

$$\begin{array}{c}\hfill {\displaystyle \mathcal{B}(U\widehat{p},U\widehat{p}):=\sum _{\begin{array}{c}-M/\alpha \le l\le M/\alpha ,\\ -M\le m\le M,\\ \{l,m\}\in S\cup S^{*}\end{array}}\left[\frac{\alpha m(j-l)}{{\alpha }^2{l}^2+m^2}{\left(U\widehat{p}\right)}_{l,m}{\left(U\widehat{p}\right)}_{(j-l),(k-m)}\right.-}\\ \hfill \left.\frac{\alpha l(m-k)}{{\alpha }^2{l}^2+m^2}{\left(U\widehat{p}\right)}_{l,m}{\left(U\widehat{p}\right)}_{(j-l),(k-m)}\right].\end{array}$$

In order to obtain a reduced system of ODEs, we first leave $N_r\le N$ POD modes from (6) using straightforward restriction ($\mathcal{R}$) and prolongation ($\mathcal{P}$) operators:

$$r_t=\mathcal{R}(-R^{-1}{U}^{T}LU\mathcal{P}\left(r\right)+\beta /2U^{T}d+U^T\mathcal{B}(U\mathcal{P}\left(r\right),U\mathcal{P}\left(r\right))),$$

(7)

where $r\in {\mathbb{R}}^{N_r}$ is reduced dimension vector, $\mathcal{R}:{\mathbb{R}}^N\to {\mathbb{R}}^{N_r}$ throws away all vector components except for the first $N_r$ ones and $\mathcal{P}:{\mathbb{R}}^{N_r}\to {\mathbb{R}}^N$ adds to $N_r$ components of reduced vector $N-N_r$ components with constant values taken as average over snapshots for corresponding components.

For all tests, we use the following hardware:

• Intel Xeon E5-2697 v2 @ 3.00 GHz with 12 cores and 64 GB of RAM,
• $1\times$ NVIDIA GeForce GTX TITAN X with 12 GB,
• $5\times$ NVIDIA GeForce GTX TITAN Black with 6 GB.

All our calculations are executed on a single Graphics Processing Unit (GPU), which is selected depending on the required device memory. CPU has negligible influence on the calculations because all data are stored on a GPU; see [25] for more detail.

3. Validation and Results

3.1. Validation

The error estimation, proof of convergence of the numerical method and analysis of the method are provided in [29], where the analysis is performed for different basis functions. In addition, we present a very sensitive test of the right-hand side of Equation (3) by using our bifurcation analysis software [34,35]. We seek a symmetry breaking bifurcation that was proven in [15]: for $\alpha =0.35$, the secondary symmetry-breaking bifurcation occurs for $R_0\in [4.80719543296090,4.807195432960905]$; see Theorem 5.6 in [15]. This test is relatively easy to implement since our bifurcation analysis code only requires the right-hand side and the Jacobi operator in the form of a matrix–vector application. The Stokes preconditioning operator is used to accelerate the bifurcation analysis; see [36] for more information.

For this test, we use $M=12$, $\beta =1$ and $\alpha =0.35$ to construct a bifurcation diagram of the Kolmogorov flow problem for $0<R\le 10$. The results are provided in Figure 1.

The bisection method used in the bifurcation analysis software [34] is able to recover a bifurcation point at $R_0=4.8071954$ which is close to the rigorous result. This result indicates that the discrete set of nonlinear algebraic Equation (3) approximates the Kolmogorov problem well and the numerical scheme is correct. The visual representation of solutions in the form of $w(x,y)$ distributions that are found for $R=6$ are presented in Supplementary Material.

3.2. Results

The following parameters are used for the simulation: $R=19$, $\alpha =1/2$, $\beta =2$, $M=5$, and by (5) we have $N=79$ DOF. A full simulation with TMs of order two requires $\sim 3.8\times {10}^7$, with TMs of order three it requires $\sim 3.08\times {10}^9$ and for TMs of order four it requires $2.44\times {10}^{11}$ computational operations per each time step. As one can see, even for TMs of the third degree, reduction in DOFs is desirable. In order to solve the system (3) numerically to find a periodic orbit, a RK46SSP method is used.

The solution method for a stable periodic orbit consists of the following steps. First, a periodic orbit is sought by using an adaptive time step strategy. It is achieved by detecting a return map on an all-state vector $\widehat{w}$ in a given initial epsilon tube. Then, a time step is modified (decreased) in such a way that a whole number of time steps (not larger than the predefined value) is fitted with a constant time step value. This ensures a unified convergence of the time stepping method towards a stable periodic orbit and minimizes possible numerical issues related with the rapidly changing time step values. In addition, it allows us the use of these time steps in Picard iterations in proving algorithms; see [26]. Such process is executed for both full and reduced systems. It is performed until the convergence is achieved in the return map of the state vector norm; in our case, up to $1\times {10}^{-10}$.

Projection of the periodic orbit to the 3D subspace is presented in Figure 2. The formulation of the set $\left\{\tau \right\}$ is achieved by splitting the converged periodic orbit in such a manner that the time for very 10th step is stored in $\left\{\tau \right\}$. The points in Figure 2 on the right are the times in $\left\{\tau \right\}$ at which snapshots are stored in columns of X.

The resulting magnitudes of the snapshots in X are presented in Figure 3, on the left. We can observe a relatively well-resolved periodic orbit with modes with large number constituting negligible magnitude. A representation of the magnitude of POD modes $U^{T}X$ is presented in Figure 3, on the right. POD modes obtain good fitting by the selected snapshots. In addition, we visualize the solution in Figure 4, on the left, and present energies stored in POD modes in Figure 4, on the right. It is shown that after 22 POD modes, the energies are negligible, but even for the first five POD modes, the level of energies around $5.0\times {10}^{-6}$ is obtained, which in many studies is considered to be a sufficient criterion for the number of POD mode selection.

Next, we study the influence of the POD approximation space dimension on the convergence of the periodic orbit. The first convergence criterion is the value of period T. It is presented in Figure 5, on the left, for different dimensions of POD mode approximations. We can observe a relatively fast convergence towards the values of the full system. Another measure of convergence is the distance (in a metric space) between POD-approximated periodic orbit (re-projected back to the full system) and the periodic orbit obtained by the simulation of the full system. The distance $\rho$ between two periodic orbits is defined as

$$\rho (p_A,p_B):=max\{\underset{a\in p_A}{max}\underset{b\in p_B}{min}\rho (a,b),\underset{b\in p_B}{max}\underset{a\in p_A}{min}\rho (a,b)\},$$

where $p_A$ and $p_B$ is a set that contains all calculated points in a periodic orbit A or B, $\rho (a,b)$ is a usual Euclidean distance between points in ${\mathbb{R}}^N$ and internal sup and inf values can be substituted for max and min values due to the finite set of points in $p_A$ and $p_B$. The results are provided in Figure 5, on the right. The graphical representation of convergence is presented in the projections in Figure 6.

Finally, we measure the convergence in terms of the spectral space of monodromy matrices (MMs). In order to demonstrate the distance, we find eigenvalues of the full and POD approximated MMs. The results are presented in Figure 7. This figure also includes Gershgorin disks (GDs) that are constructed from the data of POD MMs. GDs can serve as a rough estimate of whether POD eigenvalues encompass any of the original eigenvalues. A more advanced method of POD approximation is presented in Figure 8 that is discussed below.

4. Discussion

As it was presented in the paper, we applied the POD method to the problem of approximating the Navier–Stokes system on a stable periodic orbit solution. We showed that the approximation from the POD point of view is successfully obtained; even for as few as 22 POD modes, we obtained machine–epsilon remainders in the rest of POD modes; see Figure 3 and Figure 4. Since the periodic orbit under consideration was stable, the convergence towards the trajectory was achieved relatively fast. This is indicated by Figure 5 and visualized in Figure 6.

However, the spectral convergence of the POD method is not satisfactory, since eigenvalues of the monodromy matrix are not well resolved. We performed the analysis of the spectral properties of POD MMs (Figure 7), which demonstrated that the POD reduced system may not work well in the interval arithmetic with TMs. For example, the pair of complex conjugate eigenvalues related to the converging rotation of the phase flow towards the periodic orbit is not well approximated and is only roughly covered by one of Gershgorin disks, only for 50 POD modes. If this behavior is not approximated well, then the interval estimates may diverge, which results in the full divergence of the rigorous computations. Another approach that will be checked is a Balanced POD method [37] that constructs bi-orthogonal reduced basis.

Meanwhile, the following remedy can be proposed. First, we calculated the MMs of the original system in the set $\left\{\tau \right\}$ (all POD points, e.g., presented in Figure 2, on the right). In each point, we calculated the desired number of MM eigenvectors related to the $e<N$, the largest magnitude eigenvalues. These eigenvectors (and their negative conjugate pairs) were stacked into the columns of the matrix $X_{MM}\in {\mathbb{R}}^{2eQ\times N}$. Then, the SVD was applied to the matrix $X_{MM}$, resulting in a POD matrix $U_{MM}\in {\mathbb{R}}^{N\times N}$ that is related to the POD approximation of the tangent space of the periodic orbit. Then, a set of leading POD modes from the original system and a set of leading POD modes from $U_{MM}$ were mixed together and orthogonalized using Householder transformation. This new basis is called a POD-mixed approximation, and it was applied to System (7). The results of this approach are presented in Figure 8 in terms of POD-mixed MM eigenvalues compared to the MM eigenvalues of the original system. On the left, only five leading eigenvectors were used to form the $X_{MM}$ matrix. Then, 10 original POD modes and 10 MM POD modes were used to form the system that had 20 POD-mixed modes in total. On the right, 10 leading eigenvectors were used and 20 original and MM POD modes were taken to form the reduced system. We could observe that not only were we able to approximate the periodic orbit well, but we were also able to represent the tangent space to the periodic orbit relatively well, at least for the chosen set of leading eigenvalues. In order to verify this fact, we performed rigorous computations of the reduced systems using TMs of the second order for the original POD approximation and POD-mixed approximation using the same system size of 20; see Figure 9 and [25,26] for more information. All calculations were carried out on GPUs. We can observe that the simple POD approximation is divergent (left figure), while the POD-mixed approximation is convergent. These calculations are rigorous and show that a periodic orbit exists in the reduced POD-mixed system. In order to complete the proof of this fact, we need to introduce a Poincare section using TMs, which will be published elsewhere.

Our next step is to test this POD-mixed reduced scheme with the extended system part, as outlined in the introduction.