Modeling Error and Nonuniqueness of the Continuous-Time Models Learned via Runge–Kutta Methods

Abstract

In the present study, we consider continuous-time modeling of dynamics using observed data and formulate the modeling error caused by the discretization method used in the process. In the formulation, a class of linearized dynamics called Dahlquist’s test equations is used as representative of the target dynamics, and the characteristics of each discretization method for various dynamics are taken into account. The family of explicit Runge–Kutta methods is analyzed as a specific discretization method using the proposed framework. As a result, equations for predicting the modeling error are derived, and it is found that there can be multiple possible models obtained when using these methods. Several learning experiments using a simple neural network exhibited consistent results with theoretical predictions, including the nonuniqueness of the resulting model.

1. Introduction

In the present study, we propose an analytical framework for modeling errors in modeling dynamics using discretization methods based on a linearization of the target dynamics. We also analyze the explicit Runge–Kutta methods in particular as the discretization method and show that the modeling results may not be uniquely determined when using these methods. (This paper is a revision of the preprint titled “An Error Analysis Framework for Neural Network Modeling of Dynamical Systems”, first available on 8 December 2021 at arXiv).

Assuming that the dynamics to be modeled are represented by a system of unknown ordinary differential equations (ODEs) of the form

$$\begin{array}{c}\hfill \dot{x}\left(t\right)=f\left(x\right),\end{array}$$

(1)

we consider the modeling errors that arise in the modeling. In fact, dynamics in various fields are represented in this form, such as equations of motion in mechanics, state variable models in control, and models of chemical reactions of materials [1,2]. An important problem in modeling such dynamics is to obtain a continuous-time model:

$$\begin{array}{c}\hfill \dot{x}\left(t\right)=\widehat{f}\left(x\right),\end{array}$$

(2)

where $\widehat{f}$ is constructed by some function approximation method such as neural network, Gaussian process, etc., from a sequence of observed data for a variable vector x of the system [3,4]. (The class to which $\widehat{f}$ belongs varies depending on each specific target dynamics and approximation method, but here, we assume that the method has the universal approximation property [5] for the class of $\widehat{f}$). If the time derivative $\dot{x}$ of the variables of the system at each time in addition to x can be observed, then the determination of $\widehat{f}$ can be attributed to a regression problem [6], but there are situations in which such data are not available.

Therefore, as a method to construct the continuous model shown in Equation (2) without using the derivative values of the system variable, the numerical solution of the continuous model $\dot{x}\left(t\right)=\widehat{f}$ is computed, and then the model is indirectly optimized so that the numerical solution is as close as possible to the observed data [3,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24]. The modeling procedure is briefly shown in Figure 1.

Such a modeling approach can be said to use the discretization method in the opposite direction from the way it is normally used. Usually, when using and evaluating discretization methods, we consider the problem of finding an unknown discrete solution from a given ODE (we call this forward computation). On the other hand, when the discretization method is used in the modeling approach where the numerical solutions are compared with the observed data, the relationship between the ODE and the solution is reversed because the solution of the target system is given as observed data, and the ODE that defines the original unknown system is then approximated. Such an inverse relationship has not been considered in the usual numerical analysis [25,26,27]. In the present study, we propose a new evaluation framework of discretization methods for such inverse applications.

One of the early studies that proposed this type of continuous-time modeling approach was that of Rico-Martinez et al. in 1993 [12]. The possibility of combining the learning model with known continuous models is mentioned in some studies [16,20], which is difficult when using discrete-time models. An application for PDE is also studied [10]. In recent years, modeling methods focusing on the Hamiltonian structure of dynamics, which is characterized in continuous time, have been proposed [3,9,21]. (The loss function of Hamiltonian Neural Networks using finite difference can be considered the same as using the explicit Euler method.) In the existing studies on dynamics modeling, the key contribution to the performance of dynamics modeling is sought in the structure of the neural network [3,22] or the optimization method [15,16,19], and little attention is paid to the effect of the discretization method [9].

As shown in Figure 2, one of the advantages of the continuous models is that one can use a different discretization method and time step for the forward calculation to make predictions, and even couple the resulting model to other models modeled with different modeling methods, regardless of the methods used in the modeling process [12,15,19]. It is therefore desirable that the discretization process not be considered to be part of the resulting model and should affect the results as little as possible. In practice, however, there will be a modeling error that arises from the discretization process, i.e., the error between the original unknown dynamics f and $\widehat{f}$ in the continuous model shown in Equation (2).

Several studies focus on the different discretization methods used for modeling. It is known that certain types of methods, such as implicit or linearly implicit methods, are suitable for the forward integration of stiff systems [26]. Rico-Martinez and Kevrekidis [12] and Oussar and Dreyfus [23] focused on the effectiveness of implicit methods for the integration of a stiff system [26], and they proposed to use implicit methods for the modeling of stiff target systems. Raissi et al. [17] and Chen et al. [24] compared the effectiveness of multiple discretization methods in the modeling task. However, these studies evaluate the difference in discretization methods mainly in terms of the prediction error (i.e., how close the model’s trajectory is to the target dynamics) or noise robustness.

An important study that directly addresses the error between the target and the model is presented by Zhu et al. [9] In the study, the error between the two is distinguished as the “target error” from the other errors introduced in the optimization, generalization, and approximation processes. The concept is extended to more general target dynamics and discretization methods in later studies [7,8], allowing an order evaluation of the target error using the formally defined equation named inverse modified differential equations (IMDE), which is identified with the modeled dynamics.

The IMDE-based analysis method by Zhu et al. is general and rigid. However, if one can directly know how the nature of the specific target dynamics (i.e., oscillatory, dissipative, divergent) affects the error, it becomes easier to appropriately choose or design the discretization method to be used for the modeling of the target dynamics. A successful example of analyzing discretization methods with a focus on the nature of dynamics is the stability analysis of discretization methods [26,27,28]. In this study, we propose an error analysis framework of discretization methods used in the modeling purpose following the concept of the stability analysis.

A more detailed description of the problem setup and the definition of the modeling error in the present study is given below. To represent the nature of the target dynamics, we formulate the modeling error using Dahlquist’s test equation, referring to the stability analysis of the discretization method [26,27,28]. Dahlquist’s test equation is an approximation of the target dynamics for estimating the result of the modeling process, in the sense that it is obtained by linearizing the equation of dynamics and considering the Jordan canonical form. If a rough distribution of modes of the target dynamics is known, e.g., using DMD [29], the results for the test equations corresponding to such modes can be used to estimate the modeling error for the target dynamics. In addition, Dahlquist’s test equation is characterized by a single complex parameter, allowing the properties of the discretization method to be examined graphically in the complex plane, as is carried out in the stability analysis.

As a main result, we derived an equation that is satisfied by the learned model when the model perfectly reproduces the sampled solution of the test equation using an explicit Runge–Kutta method. The solution of the equation predicts the resulting model, and we used it to define the modeling error.

One important result is that the above equations for more than two-stage explicit Runge–Kutta methods have multiple solutions. (A similar result is addressed in an analysis example for a composite method of the explicit Euler method in the work by Zhu et al. [7].) This means that the modeling results may not be uniquely determined. This is not only a potential problem in practical modeling, but it also makes the design of explicit Runge–Kutta methods for modeling purposes difficult.

In order to confirm the validity of the theoretical considerations, learning experiments are conducted using the explicit midpoint method, a two-stage Runge–Kutta method, for test equations with various parameters. A multilayer perceptron, a simple neural network, is used as the learning model. The results confirm that any of the expected results are reproduced and that the nonuniqueness described above occurs in some cases.

Remark 1.

The family of Runge–Kutta methods, especially the four-stage, fourth-order explicit Runge–Kutta method (the classic Runge–Kutta method; RK4), is a very widely used discretization method in forward computation and is also implemented in various ODE solvers. With this background, the Runge–Kutta method is considered a natural candidate for modeling. In fact, a learning model composed of neural networks is discretized and trained using the classic Runge–Kutta method in many studies [12,15,18,19], and related research explicitly uses Runge–Kutta in the name of modeling architectures, e.g., Runge–Kutta Neural Networks (RKNNs) [13,15,22]. However, when a Runge–Kutta method is used for modeling as described above, it is shown in the present study that there are multiple possible models that reproduce the same observed data. Among such models, some may approximate the dynamics of the target well, while others may not. This may not be a problem when the model is used as a discretized model that assumes the same time step as the observed data, but this can lead to undesirable results when used as a continuous model.

2. Modeling Error Analysis

2.1. The Test Equation

Dynamics expressed in the form of Equation (2) range so widely that it is not easy to define the modeling error for all of them. Therefore, as a representative of these various dynamics, we use one-dimensional dynamics called Dahlquist’s test equation [26,28]:

$$\begin{array}{c}\hfill \dot{x}\left(t\right)=\lambda x(\lambda \in \mathbb{C}).\end{array}$$

(3)

The test equation can be thought of as a linearized and diagonalized form of general dynamics.

The test equation is characterized by a complex parameter $\lambda$. The dynamics $x\left(t\right)={\mathrm{e}}^{\lambda t}{x}_0$ expressed by the test equation are more damped the more $\mathrm{Re}\left(\lambda \right)$ is negative, and the oscillation becomes more intense for larger values of $\mathrm{Im}\left(\lambda \right)$.

Instead of examining modeling errors for various f, we examine modeling errors for the test equations at various $\lambda$. That is, we define the modeling error parameterized by the complex number $\lambda$. By examining the modeling errors for various $\lambda$, we can see what dynamics the method is suitable for modeling. Conversely, by studying the modeling errors for various $\lambda$ for various discretization methods in advance, it is possible to determine which discretization method is suitable for modeling based on prior knowledge of the rough mode distribution of the dynamics to be modeled. Moreover, it can be applied not only to evaluate existing discretization methods but also to design discretization methods that minimize modeling error for $\lambda$ within a specified region [30].

The test equation has been used to analyze the stability of discretization methods. Figure 3 shows the results of the stability analysis for some methods. It shows that the dynamics by $\lambda$s in the region inside the plotted line can be integrated stably by the method and the time step h. The idea of the present study is analogous to the stability analysis in that the test equation is used as a sort of benchmark for the analysis, and the properties of the discretization method are represented on the complex plane.

2.2. Continuous Modeling of Dynamics with Observed Data

In the present study, we investigate modeling errors when modeling the target dynamics with ordinary differential equation models. Modeling here refers to the assuming that the dynamics to be modeled is an ordinary differential equation:

$$\begin{array}{c}\hfill \dot{x}\left(t\right)=f\left(x\right),\end{array}$$

(4)

and approximating the function f on the right-hand side using observed data of the dynamics. That is, we consider the problem of optimizing the model $\widehat{f}$ so that the solution of the ordinary differential equation model:

$$\begin{array}{c}\hfill \dot{x}\left(t\right)=\widehat{f}\left(x\right)\end{array}$$

(5)

reproduces the observed data considering the data to be a sampling of trajectories of Equation (4).

Since it is generally difficult to directly observe the derivative of a variable in a system, we assume that $\dot{x}$ is not included in the observed data. Under such an assumption, we cannot directly optimize $\widehat{f}$ as a regression problem in the space of maps mapping x to $\dot{x}$. Therefore, we discretize $\widehat{f}$ using a discretization method (in particular, an explicit Runge–Kutta method) and introduce a map $F(x;\widehat{f})$ from the space X of variables x to X that depends on $\widehat{f}$, thereby optimizing $\widehat{f}$ using only observed data on X.

Let the set of observed data consisting of m trajectories be denoted as follows:

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill D:& ={\left\{{\tau }_i\right\}}_{i=0}^m\hfill \\ \hfill {\tau }_i:& =(x(nh;x^{\left(i\right)}),x((n-1)h;x^{\left(i\right)}),\dots ,x(0;x^{\left(i\right)}))\hfill \\ \hfill & ={\left(x(jh;x^{\left(i\right)})\right)}_{j=0}^n,\hfill \end{array}\end{array}$$

(6)

where $x(t;x^{\left(i\right)})$ is the exact solution of (4) for the initial value $x^{\left(i\right)}$ and each of those trajectories is denoted by ${\tau }_i$. The observed data are assumed to be sampled at equally spaced intervals with step size h, and the effect of noise is ignored. Here, n is the number of sample points in a trajectory, which is equal to the number of steps in the discretization method used for modeling.

Remark 2.

If the observed data consist of trajectories with more steps than the discretization method, they can be split into smaller subtrajectories to apply this modeling method. Since this study considers the ideal case where n-step trajectories from any initial value can be completely modeled, considering n-step trajectories for n-step methods by patching n-step results taking the points in the middle of the trajectory as initial values is sufficient.

Using such observed data, we optimize the model shown in Equation (5). Optimization is performed so that the observed data in each trajectory match the numerical solution of the model. Let

$$\begin{array}{c}\hfill F((x_n,\dots ,x_0);g(·),h)=0\end{array}$$

(7)

denote an equation that defines a discretization method (n-step method) for an ODE $\dot{x}=g\left(x\right)$. g in the definition of the method is a vector field, which is given and integrated into the traditional forward application of the method. In the modeling process, substitute g with $\widehat{f}$, the model to be optimized, and consider the following optimization problem:

$$\begin{array}{c}\hfill \underset{\widehat{f}}{min}\sum _{\tau \in D}\parallel F(\tau ;\widehat{f},h)\parallel .\end{array}$$

(8)

Note that $\parallel ·\parallel$ is an appropriate norm on X. We call

$$\begin{array}{c}\hfill L(·,D):=\sum _{\tau \in D}\parallel F(\tau ;·,h)\parallel \end{array}$$

(9)

the objective function or loss function.

Example 1.

Since F for the explicit midpoint method (see Section 3.1)

$$\begin{array}{c}{x}_1=x_0+hg(x_0+\frac{h}{2}g\left(x_0\right))\end{array}$$

(10)

can be written as

$$\begin{array}{c}F((x_1,x_0);g,h)=x_1-(x_0+hg(x_0+\frac{h}{2}g\left(x_0\right))),\end{array}$$

(11)

the optimization problem (8) becomes

$$\begin{array}{c}\underset{\widehat{f}}{min}\sum _{\left(x\right(h),x(0\left)\right)\in D}\parallel F((x\left(h\right),x\left(0\right));\widehat{f},h)\parallel ,\end{array}$$

(12)

$$\begin{array}{c}F((x(h),x(0));\widehat{f},h)=x(h)-(x(0)+h\widehat{f}(x(0)+\frac{h}{2}\widehat{f}(x(0))))\end{array}$$

(13)

Example 2.

For the implicit Euler method, F is represented as

$$\begin{array}{c}F((x_1,x_0);g,h)=x_1-(x_0+hg\left(x_1\right)).\end{array}$$

(14)

2.3. Modeling Error

Suppose that the modeling process described in the previous section yields $\widehat{f}$ that perfectly reproduces the observed data. Even in such an ideal case, f and $\widehat{f}$ do not necessarily coincide due to the effect of discretization, and the error depends on both the discretization method and the nature of the target dynamics f. In this section, we formulate such an error as a modeling error and propose a new evaluation criterion for discretization methods.

In defining the modeling error, as is performed in the stability analysis of numerical methods, we consider the results of modeling when using the test Equation (3) as a representative of target dynamics f. We also make two idealized assumptions to investigate certain limits of the optimization problem (8). First, assume that a sufficient amount of data are available and that $\widehat{f}$ can be learned such that the objective function (9) is zero for any $\tau$, a sampling of the exact solution with a step size h. That is, for any initial value of $x^{\left(i\right)}$, the following holds:

$$\begin{array}{c}\hfill F({\tau }_i;\widehat{f},h)=0.\end{array}$$

(15)

Furthermore, we assume the universal approximation property [5] for the construction of $\widehat{f}$. Thus, there exists an $\widehat{f}$ that reduces the objective function to as close to 0 as desired. Among such $\widehat{f}$s, when there exists a linear $\widehat{f}$ in particular, the modeling error is defined as follows:

$$\begin{array}{c}\hfill \widehat{f}\left(x\right)=\widehat{\lambda }x(\widehat{\lambda }\in \mathbb{C}).\end{array}$$

(16)

Although there may be other models that reduce the loss function to 0, we assume in particular a result of the form (16) because this is an ideal case in the sense that it belongs to the same class as the test equations, i.e., the modeling target.

Remark 3.

As discussed in Theorem 1, if using explicit Runge–Kutta methods, there exists a linear $\widehat{f}$ such that the objective function becomes 0.

Definition 1

(Modeling Error). For $\widehat{f}\left(x\right)=\widehat{\lambda }x$ such that the objective function (9) reduces to 0, define the learnability coefficient ℓ as follows:

$$\begin{array}{c}\hfill \ell :=\left|\frac{\widehat{\lambda }-\lambda }{\lambda }\right|.\end{array}$$

(17)

Errors for the real and imaginary parts are also defined in the same way as the modeling error.

Definition 2.

For $\widehat{f}\left(x\right)=\widehat{\lambda }x$ such that the objective function (9) reduces to 0, define the modeling error ${\ell }^{\mathcal{R}}$ with respect to the real part and the modeling error ${\ell }^{\mathcal{I}}$ with respect to the imaginary part as follows:

$$\begin{array}{c}\hfill {\ell }^{\mathcal{R}}:=\left|\frac{\mathrm{Re}\widehat{\lambda }-\mathrm{Re}\lambda }{\mathrm{Re}\lambda }\right|,{\ell }^{\mathcal{I}}:=\left|\frac{\mathrm{Im}\widehat{\lambda }-\mathrm{Im}\lambda }{\mathrm{Im}\lambda }\right|.\end{array}$$

(18)

Remark 4.

These two types of modeling errors (ℓ and, ${\ell }^{\mathcal{R}}$ or ${\ell }^{\mathcal{I}}$) are intended for different uses. The overall error ℓ is useful for designing discretization methods, whereas the component errors are useful for analyzing the effectiveness of existing discretization methods for given dynamics.

The characteristics of various discretization methods can be compared by illustrating on the complex plane the dynamics that result in a small modeling error for each given discretization method.

Definition 3.

If the modeling error ℓ can be written as a function of $z=h\lambda$, define the learnability region ${\Omega }_{\ell ,ϵ}$ for a given discretization method with an acceptable error $ϵ>0$ by the following:

$$\begin{array}{c}\hfill {\Omega }_{\ell ,ϵ}:=\{z\in \mathbb{C}:\ell \left(z\right)\le ϵ\}.\end{array}$$

(19)

In addition, the learnable region for the modeling error of real and imaginary parts is defined in the same way.

Remark 5.

Note that although the constant 1 that appears in the inequality in the definition of the stability region in the stability analysis has meaning as a limit with respect to divergence, the tolerance $ϵ=1$ in the learnable region has no special meaning. Basically, a smaller modeling error is better, and it is important to investigate the learnability regions by changing ϵ to 0.1, 0.01 (10%, 1% error allowed), and so on, depending on the error allowed for each application.

3. Analysis of Explicit Runge–Kutta Methods

3.1. Explicit Runge–Kutta Methods

The explicit s-stage Runge–Kutta methods for an initial value problem

$$\begin{array}{c}\hfill \dot{x}=g\left(x\right),x\left(0\right)=x_0\end{array}$$

(20)

are given by

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill k_1& =g\left(x_0\right)\hfill \\ \hfill k_2& =g(x_0+h{a}_{21}{k}_1)\hfill \\ \hfill k_3& =g\left(x_0+h(a_{31}{k}_1+a_{32}{k}_2)\right)\hfill \\ \hfill & ⋮\hfill \\ \hfill k_s& =g\left(x_0+h(a_{s1}{k}_1+\cdots +a_{s,s-1}{k}_{s-1})\right)\hfill \\ \hfill x_1& =x_0+h(b_1{k}_1+\cdots +b_s{k}_s)\hfill \end{array}\end{array}$$

(21)

where $a_{ij}$ and $b_i$ are real coefficients that determine the method.

Lemma 1.

The explicit Runge–Kutta methods for the test equation can be expressed as

$$\begin{array}{cc}\hfill k& ={(I-h\lambda A)}^{-1}\mathbb{1}\lambda x_0\hfill \end{array}$$

(22)

$$\begin{array}{cc}\hfill x_1& =x_0+hbk\hfill \end{array}$$

(23)

$$\begin{array}{cc}\hfill & =(1+h\lambda b{(I-h\lambda A)}^{-1}\mathbb{1})x_0\hfill \end{array}$$

(24)

where

$$\begin{array}{c}A=\left(\begin{array}{ccccc}0& \cdots & & & \\ a_{21}& 0& \cdots & & \\ a_{31}& a_{32}& 0& \cdots & \\ ⋮& ⋮& \ddots & \ddots & \\ a_{s1}& a_{s2}& \cdots & a_{s,s-1}& 0\end{array}\right),b=\left(\begin{array}{c}{b}_1\cdots b_s\end{array}\right),k=\left(\begin{array}{c}{k}_1\\ ⋮\\ k_s\end{array}\right),\mathbb{1}=\left(\begin{array}{c}1\\ ⋮\\ 1\end{array}\right).\end{array}$$

(25)

Proof. 

By definition, the following holds for k:

$$\begin{array}{c}\hfill k=\mathbb{1}\lambda x_0+h\lambda Ak.\end{array}$$

This can be transformed as $(I-h\lambda A)k=\mathbb{1}\lambda x_0$. Since the Runge–Kutta method is explicit, the matrix $I-h\lambda A$ becomes a lower triangular matrix with nonzero diagonal elements. Therefore, ${(I-h\lambda A)}^{-1}$ exists for any h and $\lambda$, and k and $x_1$ can be expressed as

$$\begin{array}{cc}k\hfill & ={(I-h\lambda A)}^{-1}\mathbb{1}\lambda x_0\hfill \\ x_1\hfill & =(1+h\lambda b{(I-h\lambda A)}^{-1}\mathbb{1})x_0.\hfill \end{array}$$

□

Example 3.

The explicit midpoint method, Kutta’s third-order method, and the classic fourth-order method are expressed as in Figure 4 using the table-like notation of A, b, and c (the Butcher Tableau [25,31]):

$$\begin{array}{c}\hfill \begin{array}{cc}c& A\\ & b\end{array}.\end{array}$$

(26)

Since we assume the target dynamics to be time-independent, the c vector in the standard Butcher tableau can be ignored.

3.2. Idealized Prediction Equation

For a given explicit Runge–Kutta method (21), an equation satisfied by the linear model $\widehat{\lambda }$ that makes the objective function 0 is derived, leading to the existence of such a model.

Theorem 1.

Let a discretization method for Dahlquist’s test equation $\dot{x}=\lambda x$ ($\lambda \in \mathbb{C})$ be an explicit s-stage Runge–Kutta method $(A,b,c)$. Let D a set of 1-step trajectories of the test equation starting from initial points $x^{\left(i\right)}(i=0,\dots ,m)$ as:

$$\begin{array}{c}\hfill D:={\left\{{\tau }_i\right\}}_{i=0}^m,{\tau }_i:=({\mathrm{e}}^{h\lambda }{x}^{\left(i\right)},x^{\left(i\right)})\end{array}$$

(27)

where m is a positive integer. Denote the learning model as

$$\begin{array}{c}\hfill \dot{x}=\widehat{f}\left(x\right).\end{array}$$

(28)

When learning the test equation by considering an optimization problem

$$\begin{array}{c}\hfill \underset{\widehat{f}}{min}L(\widehat{f},D),\end{array}$$

(29)

where the loss is

$$\begin{array}{c}\hfill L(\widehat{f},D):=\sum _{{\tau }_i\in D}\parallel F({\tau }_i;\widehat{f},h)\parallel ,\end{array}$$

(30)

and

$$\begin{array}{c}\hfill F((x_1,x_0);\widehat{f},h)=x_1-(x_0+h(b_1{k}_1+\cdots +b_s{k}_s))\end{array}$$

(31)

is the equation of the Runge–Kutta method for the model, there exists a model

$$\begin{array}{c}\hfill \widehat{f}\left(x\right)=\widehat{\lambda }x(\widehat{\lambda }\in \mathbb{C})\end{array}$$

such that $L(\widehat{\lambda }x,D)=0$ for any trajectories D, satisfying the following:

$$\begin{array}{c}\hfill 1+h\widehat{\lambda }b{(I-h\widehat{\lambda }A)}^{-1}\mathbb{1}-{\mathrm{e}}^{h\lambda }=0.\end{array}$$

(32)

Proof. 

Assume that we have a model $\widehat{f}\left(x\right)=\widehat{\lambda }x(\widehat{\lambda }\in \mathbb{C})$. With Lemma 1, the equation of the method is as follows:

$$\begin{array}{cc}\hfill F((x_1,x_0);\widehat{\lambda }x,h)& =x_1-(1+h\widehat{\lambda }b{(I-h\widehat{\lambda }A)}^{-1}\mathbb{1})x_0=0.\hfill \end{array}$$

(33)

Therefore, the optimization problem becomes

$$\begin{array}{c}\hfill \underset{\widehat{\lambda }}{min}\sum _{{\tau }_i\in D}\parallel {\mathrm{e}}^{h\lambda }{x}^{(i)}-(1+h\widehat{\lambda }b{(I-h\widehat{\lambda }A)}^{-1}\mathbb{1})x^{(i)}\parallel .\end{array}$$

(34)

Factoring out $x^{\left(i\right)}$, we obtain

$$\begin{array}{c}\hfill \underset{\widehat{\lambda }}{min}\sum _{{\tau }_i\in D}|1+h\widehat{\lambda }b{(I-h\widehat{\lambda }A)}^{-1}\mathbb{1}-{\mathrm{e}}^{h\lambda }|\parallel x^{\left(i\right)}\parallel .\end{array}$$

(35)

If there exists $\widehat{\lambda }$ satisfying (32), then Equation (35) is zero for any sampled trajectories D of the test equation with a step size h. Since (32) is a polynomial of $\widehat{\lambda }$, such a $\widehat{\lambda }$ exists by the fundamental theorem of algebra.    □

Definition 4.

We call Equation (32) the learnability equation of the Runge–Kutta method $(A,b,c)$ and the left-hand side of the equation is referred to as the learnability polynomial.

The learnability equation describes the ideal result $\widehat{\lambda }$ of modeling for the target $\lambda$ and the Runge–Kutta method $A,B$ used, in the sense of the optimization problem (8).

Remark 6.

The parameter λ, which represents the target dynamics, appears as the term ${\mathrm{e}}^{h\lambda }$ in the learnability equation. Note that the solution of the learnability equation is periodic with respect to λ in the direction of the imaginary axis by $2\pi$.

Corollary 1.

In general, the learnability polynomial of an s-stage Runge–Kutta method is an s-order polynomial of $\widehat{\lambda }$, and thus the learnability equation has s solutions. Therefore, the resulting model may not be uniquely determined by the optimization problem (8). See Section 3.3 for details.

Example 4.

The learnability equation for the explicit midpoint method is

$$\begin{array}{c}\hfill 1+h\widehat{\lambda }+\frac{1}{2}{(h\widehat{\lambda })}^2-{\mathrm{e}}^{h\widehat{\lambda }}=0,\end{array}$$

and it has two solutions

$$\begin{array}{c}\hfill {\widehat{\lambda }}_{+}:=\frac{-1+\sqrt{{\mathrm{e}}^{h\lambda }-1}}{h},{\widehat{\lambda }}_{-}:=\frac{-1-\sqrt{{\mathrm{e}}^{h\lambda }-1}}{h}.\end{array}$$

(36)

For the modeling error ℓ of the explicit Runge–Kutta methods, the scaling law with respect to step size h holds, as in the stability analysis. This is an important property for the modeling error to serve as an evaluation metric for discretization methods.

Theorem 2.

The modeling error of the explicit Runge–Kutta methods

$$\begin{array}{c}\hfill {\ell }_{\mathrm{RK}}=\left|\frac{\widehat{\lambda }-\lambda }{\lambda }\right|=\left|\frac{\widehat{\lambda }}{\lambda }-1\right|\end{array}$$

is a function of $z=h\lambda$.

Proof. 

From the learnability equation, we have

$$\begin{array}{cc}\hfill & 1+\frac{\widehat{\lambda }}{\lambda }h\lambda b{(I-\frac{\widehat{\lambda }}{\lambda }h\lambda A)}^{-1}\mathbb{1}-{\mathrm{e}}^{h\lambda }=0\hfill \end{array}$$

(37)

and substituting $z=h\lambda$ yields

$$\begin{array}{cc}\hfill & 1+\frac{\widehat{\lambda }}{\lambda }zb{(I-\frac{\widehat{\lambda }}{\lambda }zA)}^{-1}\mathbb{1}-{\mathrm{e}}^z=0.\hfill \end{array}$$

(38)

This shows that the solution $\widehat{\lambda }/\lambda$, and also $\widehat{\lambda }/\lambda -1$, is a function of z.    □

The same holds for the component-wise modeling error.

Theorem 3.

The component-wise modeling errors for the explicit Runge–Kutta methods ${\ell }_{\mathrm{RK}}^{\mathcal{R}},{\ell }_{\mathrm{RK}}^{\mathcal{I}}$ are functions of $z=h\lambda$ and $\overline{z}=h\overline{\lambda }$.

Proof. 

We prove this for ${\ell }_{\mathrm{RK}}^{\mathcal{R}}$. The same goes for ${\ell }_{\mathrm{RK}}^{\mathcal{I}}$.

If $\widehat{\lambda }$ is a solution of the learnability equation for $\lambda$, then $\overline{\widehat{\lambda }}$ is also a solution of the equation for $\overline{\lambda }$. Thus, as in the proof of Theorem 2, $\overline{\widehat{\lambda }}/\overline{\lambda }$ is a function of $\overline{z}=h\overline{\lambda }$. Since $\mathrm{Re}\widehat{\lambda }/\mathrm{Re}\lambda$ in the definition of ${\ell }_{\mathrm{RK}}^{\mathcal{R}}$ can be transformed as

$$\begin{array}{c}\hfill \frac{\mathrm{Re}\widehat{\lambda }}{\mathrm{Re}\lambda }=\frac{\widehat{\lambda }+\overline{\widehat{\lambda }}}{\lambda +\overline{\lambda }}=\frac{\frac{\widehat{\lambda }}{\lambda }+\frac{\overline{z}}{z}\frac{\overline{\widehat{\lambda }}}{\overline{\lambda }}}{1+\frac{\overline{z}}{z}},\end{array}$$

(39)

it follows that ${\ell }_{\mathrm{RK}}^{\mathcal{R}}=|\mathrm{Re}\widehat{\lambda }/\mathrm{Re}\lambda -1|$ is a function of z and $\overline{z}$.    □

3.3. Nonuniqueness of Modeling Results

As mentioned in Theorem 1, the learnability polynomial for the explicit s-stage Runge–Kutta methods is a polynomial in $\widehat{\lambda }$ and thus has s roots. Each of these roots is a parameter that characterizes the linear model obtained as a result of modeling, meaning that there can be more than one candidate among the linear models. In modeling in the sense of the optimization problem (8), even if a model can be learned such that the loss is zero, the result is not expected to be uniquely determined. On the other hand, this is not considered to be a critical issue in learning a discrete model, since the same result can be obtained as a one-step flow map regardless of which solution is chosen.

This nonuniqueness is caused by the fact that in modeling with the Runge–Kutta method, multiple compositions of the unknown function $\widehat{f}$ are optimized with only one step of the discrete solution. The point is that, whereas the performance of forward calculation is improved by applying a known function multiple times, in modeling, the composition of an unknown function $\widehat{f}$ increases a certain degree of freedom, making it impossible to determine $\widehat{f}$ with only one step of observation data.

Remark 7.

In actual learning, some selectivity may occur in the learning results depending on the structure of the specific learning model and the optimization method used in the learning process. The selectivity here refers to the difference in learnability among multiple λ for a target λ. For example, the following are possible criteria for selectivity, but it is not clear that selectivity necessarily arises based on such easily interpretable criteria:

• Degree of approximation (some distance to the target λ).
• Closeness to the origin.

It is necessary to examine and analyze the models and algorithms used for each individual case.

3.4. Minimum Modeling Error

In this section, we consider the most desirable result to be that which is closest to $\lambda$ among the candidate $\widehat{\lambda }$s. Specifically, we examine the modeling error for the solution $\widehat{\lambda }$ of the learnability equation that has the smallest value of $|\widehat{\lambda }-\lambda |$. This is the $\widehat{\lambda }$ with the smallest modeling error, and, in this sense, it is the most desirable result. At the same time, it also provides a lower bound on the modeling error as a limitation of optimization in the sense of (8).

The results for several Runge–Kutta methods are shown in Figure 5, Figure 6, Figure 7 and Figure 8. The $\widehat{\lambda }$ and modeling error based on the above criterion were calculated for $\lambda$ on a grid with 0.2 ticks in a rectangular region $-10\le \mathrm{Re}\lambda \le 10,-10\le \mathrm{Im}\lambda \le 10$ on the complex plane and are plotted in contour plots. For visibility, only portions with a modeling error of 1 or less are colored. In order to find such $\widehat{\lambda }$s, all solutions of the learnability equation were calculated numerically for each $\lambda$, and then the smallest of $|\widehat{\lambda }-\lambda |$ was chosen.

These results show that the higher the order of the Runge–Kutta method, the wider the region of small modeling error near the origin. On the other hand, looking locally, there are regions where the relationship between the order of the Runge–Kutta method and the modeling error is inverted. For example, comparing the explicit Euler method with the explicit midpoint method, for purely decaying dynamics in the negative region on the real axis, even if a $\widehat{\lambda }$ is obtained that minimizes the modeling error, the explicit Euler method could yield a model with less error than the explicit midpoint method. Although, intuitively, it would seem that a higher-order method would be more appropriate for modeling, it turns out that there are cases in which a lower-order method should be used, depending on the dynamics of the modeling target.

4. Learning Experiment

4.1. Targets

In Section 2 and Section 3, we proposed a modeling error analysis for discretization methods and predicted the results when using the explicit Runge–Kutta methods. In this section, we verify whether any of the predicted results can actually be obtained, and whether nonuniqueness appears, by conducting learning experiments using simple neural networks. Specifically, the explicit midpoint method, which is a two-stage explicit Runge–Kutta method, is used as one of the simplest of the Runge–Kutta methods that could yield multiple modeling results, and the test equations by $\lambda$ in a specific region of the complex plane are learned. The resulting model is then approximated as linear and is compared with the original $\lambda$. The dynamics to be trained was $\lambda$ on $81\times 81$ grid points satisfying $\mathrm{Re}\lambda =-5,-4.9,\dots ,2.9,3,\mathrm{Im}\lambda =-5,-4.9,\dots ,4.9,5$. The time step size h was set to 1. For each of these $z=h\lambda$, $\widehat{f}$ was learned independently.

4.2. Model and Method

In this learning experiment, a neural-network-based learning model, $\widehat{f}$, the multilayer perceptron (MLP), was used. This is a simple and basic neural network, and there is no elaboration on learning the test equations.

The learning model consists of three layers: an input layer, a hidden layer, and an output layer. In order to learn a mapping from complex to complex, the input and output layers both have two real elements corresponding to the real and imaginary parts, respectively. The hidden layer is a fully connected layer with a width of 200 nodes. ReLU is used as the activation function inserted between each layer. The overall structure can be summarized as follows: a linear layer with 200 outputs from two inputs, ReLU, a linear layer with 200 outputs from 200 inputs, ReLU, and a linear layer with two outputs from 200 inputs.

The training data consist of pairs of initial points $x_0$ and one-step solution $exp\left(h\lambda \right)x_0$. The real and imaginary parts of the initial point $x_0$ are independently produced by uniform random numbers with values in $-10\le \mathrm{Re}{x}_0\le 10,-10\le \mathrm{Im}{x}_0\le 10$, and $x_0$ is regarded as sampled from a rectangular region in the complex plane. The training data set D is a set of 5000 pairs of $x_0$ and $exp\left(h\lambda \right)x_0$ with $h=1$. The test data are generated in the same way, using 5000 pairs of data.

The seed of the random numbers is increased by 1 each time the $\lambda$ is changed. Random numbers are used to initialize the parameters of the learning model. Therefore, when a common phenomenon is obtained for each $\lambda$ as a result of the experiment by fixing the seed, it is difficult to distinguish whether the results are similar because of the same initial values or whether the results are really independent of $\lambda$ (such as selectivity of $\widehat{\lambda }$ caused by the structure of the neural network).

For an optimizer, we used the Adam method with a learning rate of 0.001. We trained a total of 10,000 epochs, with one epoch being a batch training using all of the training data at once. The loss is saved every 100 epochs.

4.3. Estimation of $\widehat{\lambda }$

The definition of the modeling error assumed that a linear model $\widehat{f}\left(x\right)=\widehat{\lambda }x$ is obtained as a result of learning. However, general learning models, including the MLP described in the previous section, are nonlinear, and the quantity corresponding to $\widehat{\lambda }$ may not be obtained directly.

Therefore, in this experiment, the input–output ratio of the training model is averaged over the range of the initial point $x_0$ used for training, and the subsequent analysis is performed by considering it as $\widehat{\lambda }$. That is, the average $\tilde{\lambda }$, defined as

$$\begin{array}{c}\hfill \tilde{\lambda }:=\frac{1}{M}\sum _{x_0\in E}\frac{\widehat{f}\left(x_0\right)}{x_0},\end{array}$$

(40)

is used as an estimation of $\widehat{\lambda }$, where E is the set of $x_0$ for evaluating the input–output ratio, and M is the number of its elements (evaluation points). As E, we used $x_0$ on a grid of 20 points each in the real- and imaginary-axis directions from the range $-10\le \mathrm{Re}{x}_0\le 10,$$-10\le \mathrm{Im}{x}_0\le 10$, for a total of $M=400$ points.

The experimental results confirm that the learned model was close to linear functions for most of the $\lambda$. For each $\lambda$, the standard deviation of the input-output ratio $\widehat{f}\left(x_0\right)/x_0$ with respect to $x_0\in E$ was calculated, and its common logarithm plotted in the z plane is shown in Figure 9 (the first case in Section 4.4.1, where the seed is incremented from 0). The smaller the standard deviation, the closer the input–output ratio is to a constant independent of $x_0$, indicating that the model is almost linear. However, the standard deviation is relatively large for $\lambda$ on the semicircle in the third quadrant. This is located at the boundary of the nonuniqueness region (see Section 4.4.2) and is considered to be a difficult region to learn for some reason.

4.4. Results

4.4.1. Comparison with Theoretical Prediction

Figure 10 shows the modeling errors ${\ell }_{\mathrm{learn}}:=|\tilde{\lambda }/\lambda -1|$ computed from the learning results $\tilde{\lambda }$ for the two cases with different seeds. The modeling error ${\ell }_{\widehat{\lambda }}$ calculated with $\tilde{\lambda }$ estimated by the learned $\widehat{f}$ for each $\lambda$ is plotted on the z-plane. In both cases, the seeds are incremented by 1 for each different $\lambda$. In the first case, the seed is set to 0 for $\lambda =-5-5\mathrm{i}$ and ${81}^2-1$ for $\lambda =5+5\mathrm{i}$, and, in the second case, the seed is set from 1 to ${81}^2$.

This plot shows that the learning results near the origin are in good agreement with the minimum modeling error expected in Figure 6. A discontinuous surface appears around $\mathrm{Im}z=\pm 3$, which is considered to be a result of the periodicity of $2\pi$ due to the learnability equation having z in the form ${\mathrm{e}}^z$. Note that there are regions in which different $\tilde{\lambda }$ are learned when the seed of the random number is changed. This nonuniqueness is considered to correspond to the existence of two solutions to the learnability equation for the explicit midpoint method (see Section 3.3).

Next, we verified that the results were those predicted by either of the two solutions of the learnability equation. For this purpose, the minimum absolute value of the difference between the modeling error of the training result ${\ell }_{\mathrm{learn}}$ and the modeling error expected from the two solutions ${\ell }_{+,-}:=|{\widehat{\lambda }}_{+,-}/\lambda -1|$ (see Equation (36)) was calculated for the training results of both cases as

$$\begin{array}{c}\hfill {\epsilon }_{\tilde{\lambda }}:=min\left(|{\ell }_{\mathrm{learn}}-{\ell }_{+}|,|{\ell }_{\mathrm{learn}}-{\ell }_{-}|,\right).\end{array}$$

(41)

Figure 11 shows the error ${\epsilon }_{\tilde{\lambda }}$, which was confirmed to be small.

4.4.2. Variation in Nonunique Results

In order to further examine the variation of the results, we conducted the same experiment with 28 different sets of seeds following the previous two cases (30 in total) as 2 to ${81}^2+1$, 3 to ${81}^2+2$, …, and 29 to ${81}^2+28$. Then, for each case, we examined which of the two solutions of the learnability equation was learned. The distance between the expected results ${\widehat{\lambda }}_{+}$ and ${\widehat{\lambda }}_{-}$ and the actual result $\tilde{\lambda }$ was calculated for each $\lambda$ in each case and was labeled $+1$ if $\tilde{\lambda }$ was close to ${\widehat{\lambda }}_{+}$ and $-1$ if $\tilde{\lambda }$ was close to ${\widehat{\lambda }}_{-}$. Figure 12 shows the results of calculating the standard deviation of the 30 labels for each $\lambda$. If the standard deviation is 0, then the result is unique, and if the standard deviation is large, then different results are trained for different seeds.

This analysis reveals that the learning results are nonunique with a large standard deviation, especially in the boundary region of the $2\pi$ periodicity and in the region in which the real part of z is negative to some extent. This shows that the actual learning outcome may not be uniquely determined, as theoretically expected. The standard deviation was close to 0.5, indicating that ${\widehat{\lambda }}_{+}$ and ${\widehat{\lambda }}_{-}$ were learned randomly without bias. The cause of this phenomenon could be the structure of the model used for training or the optimization method used, but further discussion is left for future study.

5. Conclusions

In the present study, a framework for investigating modeling errors caused by the discretization method used for the continuous-time modeling of dynamics is proposed. As a specific result of the analysis using the proposed framework, an equation that predicts the results when an explicit Runge–Kutta method is used for modeling is obtained. It was shown that even if a model that perfectly reproduces the observed data is obtained, then there is a nonzero lower bound on the modeling error and that more than one model can be obtained. Furthermore, training experiments using a multilayer perceptron confirmed that nonuniqueness of results does in fact occur. Future study will include clarification of the cause of the nonuniqueness observed in the learning experiments and analysis of the modeling error for a system with inputs.