Optimal Data-Driven Estimation of Generalized Markov State Models for Non-Equilibrium Dynamics

Abstract

There are multiple ways in which a stochastic system can be out of statistical equilibrium. It might be subject to time-varying forcing; or be in a transient phase on its way towards equilibrium; it might even be in equilibrium without us noticing it, due to insufficient observations; and it even might be a system failing to admit an equilibrium distribution at all. We review some of the approaches that model the effective statistical behavior of equilibrium and non-equilibrium dynamical systems, and show that both cases can be considered under the unified framework of optimal low-rank approximation of so-called transfer operators. Particular attention is given to the connection between these methods, Markov state models, and the concept of metastability, further to the estimation of such reduced order models from finite simulation data. All these topics bear an important role in, e.g., molecular dynamics, where Markov state models are often and successfully utilized, and which is the main motivating application in this paper. We illustrate our considerations by numerical examples.

1. Introduction

Metastable molecular systems under non-equilibrium conditions have recently attracted increasing interest. Examples include systems that evolve under an external force, such as a pulling force generated by an optical tweezer or an atomic force microscope, an electrostatic force across a biomembrane that drives ion through a channel protein, or the periodically changing force generated by a spectroscopic electromagnetic field. Such non-equilibrium conditions can be built into molecular dynamics (MD) simulations in order to probe their effects on the molecule. Despite the relevance of non-equilibrium effects, reliable tools for the quantitative description of non-equilibrium phenomena like the conformational dynamics of a molecular system under external forcing are still lacking.

In this paper, we say that a process is in “equilibrium”, if it is statistically reversible with respect to its equilibrium distribution (see Table 1). For MD simulations under equilibrium conditions, reliable analysis tools have been developed. For example, Markov state models (MSMs) allow for an accurate description of the transitions between the main conformations of the molecular system under investigation. MSMs for equilibrium MD have been well developed and established over the past decade in theory [1,2], applications (see the recent book [3] for an overview), and software implementations [4,5]. The principal idea of equilibrium MSMs is to approximate the long-timescale and stationary properties of the MD system by a reduced Markovian dynamics over a finite number of (macro-)states, i.e., in discrete state space. These states represent or at least separate the dominant metastable sets of the system, i.e., sets in which typical MD trajectories stay substantially longer than the system needs for a transition to another such set [1,6].

Table 1. Nomenclature used here for stochastic processes.

A Stochastic (Markov) Process Is Called
time-homogeneous	if the transition probabilities from time s to time t depend only on $t-s$ (in analogy to the evolution of an autonomous ordinary differential equation).
stationary	if the distribution of the process does not change in time (such a distribution is also called invariant, cf. (2)).
reversible	if it is stationary and the detailed balance condition (5) holds (reversibility means that time series are statistically indistinguishable in forward and backward time).

Equilibrium Markovian processes are associated with real-valued eigenvalues and eigenfunctions in their propagators—a property that methods for the approximation of equilibrium dynamics are built upon. For example, the approximation error of MSMs and their slowest relaxation timescales can be expressed in terms of how well the state space discretization approximates the dominant eigenfunctions [7,8]. This has been formulated in the variational approach for conformation dynamics (VAC), or Rayleigh–Ritz method, which provides an optimization method to systematically search for best MSMs or other models of the equilibrium dynamics [9,10]. Perron-Cluster Cluster Analysis (PCCA) [11,12] identify the metastable states of a molecule by conducting a spectral clustering in the space spanned by the dominant eigenfunctions. Additionally, equilibrium MSMs are the foundation of analyzing multi-ensemble simulations that help to sample the rare events [13,14].

In the non-equilibrium setting the above tools break down or are not defined. The purpose of the present paper is to summarize and reconcile recently developed methods for the description of non-equilibrium processes, and to merge them with their equilibrium counterparts into a unified framework. Note that there are different possibilities to deviate from the “equilibrium” situation, and this makes the term “non-equilibrium” ambiguous. To avoid confusion, we consider one of the following cases when referring to the non-equilibrium setting (again, see Table 1 on terminology).

(i)

Time-inhomogeneous dynamics, e.g., the system feels a time-dependent external force, for instance due to an electromagnetic field or force probing.

(ii)

Time-homogeneous non-reversible dynamics, i.e., where the governing laws of the system do not change in time, but the system does not obey detailed balance, and, additionally we might want to consider the system in a non-stationary regime.

(iii)

Reversible dynamics but non-stationary data, i.e., the system possesses a stationary distribution with respect to which it is in detailed balance, but the empirical distribution of the available data did not converge to this stationary distribution.

Even though we consider genuinely stochastic systems here, the algorithm of Section 5 can be used for deterministic systems as well—and indeed it is, see Remark 2 and references therein.

Note that with regard to the considered dynamics (i)–(iii) represent cases with decreasing generality. For (i), time-dependent external fields act on the system, such that the energy landscape depends on time, i.e., the main wells of the energy landscape can move in time. That is, there may no longer be time-independent metastable sets in which the dynamics stays for long periods of time before exiting. Instead, the potentially metastable sets will move in state space. Generally, moving “metastable” sets cannot be considered metastable anymore. However, the so-called coherent sets, which have been studied for non-autonomous flow fields in fluid dynamics [15,16], permit to give a meaning to the concept of metastability [17]. For (iii), the full theory of equilibrium Markov state modeling is at one’s disposal, but one needs to estimate certain required quantities from non-equilibrium data [18]. Case (ii) seems the most elusive, due to the fact that on the one hand it could be handled by the time-inhomogeneous approach, but on the other hand it is a time-homogeneous system and some structural properties could be carried over from the reversible equilibrium case that are out of reach for a time-inhomogeneous analysis. For instance, if the dynamics shows cyclic behavior, it admits structures that are well captured by tools from the analysis of time-homogeneous dynamical systems (e.g., Floquet theory and Poincaré sections [19,20]), and a more general view as in (i) might miss them; however, cyclic behavior is not present in reversible systems, such that the tools from (iii) are doomed to failure in this respect. In order to avoid confusion, however, it should be emphasized that the three cases distinguished above do not suffice to clarify the discussion about the definition of equilibrium or non-equilibrium, e.g., see the literature on non-equilibrium steady state systems [21,22].

Apart from MSMs the literature on kinetic lumping schemes offers several other techniques for finding a coarse-grained descriptions of systems [23,24,25]. These techniques are, however, not built on the intuition of metastable behavior in state space. What we consider here can be seen in connection to optimal prediction in the sense of the Mori–Zwanzig formalism [26,27,28,29], but we will try to choose the observables of the system such that projecting the dynamics on these keeps certain properties intact without including memory terms.

In what follows, we will review and unify some of the theoretical and also data-driven algorithmic approaches that attempt to model the effective statistical behavior of non-equilibrium systems. To this end, a MSM—or, more precisely, a generalized MSM—is sought, i.e., a possibly small matrix T that carries the properties of the actual system that are of physical relevance. In the equilibrium case, for example, this includes the slowest timescales on which the system relaxes towards equilibrium (Section 2). The difference of generalized to standard MSMs is that we do not strictly require the former to be interpretable in terms of transition probabilities between some regions of the state space (Section 3), however usually there is a strong connection between the matrix entries and metastable sets. While heavily building upon recent results for MSMs of nonstationary MD [17] and a nonequilibrium generalization of the variational principle [30], we will focus on a slightly different characteristic of the approximate model, namely its “propagation error”. It turns out that this notion permits a straightforward generalization from equilibrium (reversible) to all our non-equilibrium cases (Section 4), and even retain the physical intuition behind true MSMs through the concept of coherent sets. We will show in Section 5 how these considerations can be carried over to the case when only a finite amount of simulation data is available. The above non-equilibrium cases (ii)–(iii) can be then given as specific instances of the construction (Section 6). The theory is illustrated with examples throughout the text. Bringing the formerly known equilibrium and time-inhomogeneous concepts into a unified framework (Section 3 and Section 4) can be seen as the main contribution of this paper, together with the novel application of this framework to time-homogeneous non-equilibrium systems relying on non-stationary data (Section 6).

We note in advance that in course of the (generalized) Markov state modeling we will consider different instances of approximations to a certain linear operator $\mathcal{T}:\mathbb{S}\to \mathbb{S}$ mapping some space to itself (and sometimes to a different one). On the one hand, there will be a projected operator ${\mathcal{T}}_k:\mathbb{S}\to \mathbb{S}$, where ${\mathcal{T}}_k=\mathcal{Q}\mathcal{T}\mathcal{Q}$ with a projection $\mathcal{Q}:\mathbb{S}\to \mathbb{V}$ having a k-dimensional range $\mathbb{V}\subset \mathbb{S}$. On the other hand, we will consider the restriction of the projected operator ${\mathcal{T}}_k$ to this k-dimensional subspace, i.e., ${\mathcal{T}}_k:\mathbb{V}\to \mathbb{V}$, also called $\mathbb{V}$-restriction of ${\mathcal{T}}_k$, which has a $k\times k$ matrix representation (with respect to some chosen basis of $\mathbb{V}$) that we will denote by $T_k$.

2. Studying Dynamics with Functions

2.1. Transfer Operators

In what follows, $\mathsf{P}[·\mid \mathfrak{E}]$ and $\mathsf{E}[·\mid \mathfrak{E}]$ denote probability and expectation conditioned on the event $\mathfrak{E}$. Furthermore, ${\{x_t\}}_{t\ge 0}$ is a stochastic process defined on a state space $\mathbb{X}\subset {\mathbb{R}}^d$. For instance, we can think of $x_t$ being the solution of the stochastic differential equation

$$\mathrm{d}{x}_t=-\nabla W(x_t)\mathrm{d}t+\sqrt{2{\beta }^{-1}}\mathrm{d}{w}_t,$$

(1)

describing diffusion in the potential energy landscape given by W. Here, $\beta$ is the non-dimensionalized inverse temperature, and $w_t$ is a standard Wiener process (Brownian motion). The transition density function $p^t:\mathbb{X}\times \mathbb{X}\to {\mathbb{R}}_{\ge 0}$ of a time-homogeneous stochastic process ${\{x_t\}}_{t\ge 0}$ is defined by

$$\mathsf{P}[x_t\in \mathbb{A}\mid x_0=x]={\int }_{\mathbb{A}}{p}^t(x,y)\mathrm{d}y,\mathbb{A}\subseteq \mathbb{X}.$$

That is, $p^t(x,y)$ is the conditional probability density of $x_t=y$ given that $x_0=x$. We also write $x_t\sim p^t(x_0,·)$ to indicate that $x_t$ has density $p^t(x_0,·)$.

With the aid of the transition density function, we will now define transfer operators, i.e., the action of the process on functions of the state. Note, however, that the transition density is in general not known explicitly, and thus we will need data-based approximations to estimate it. We assume that there is a unique stationary density $\mu$, such that ${\{x_t\}}_{t\ge 0}$ is stationary with respect to $\mu$; that is, it satisfies $x_0\sim \mu$ and

$$\mu (x)={\int }_{\mathbb{X}}\mu (y)p^t(y,x)\mathrm{d}y\mathrm{for}\mathrm{all}t\ge 0.$$

(2)

Let now f be a probability density over $\mathbb{X}$, $u=f/\mu$ a probability density with respect to $\mu$ (meaning that $\mu$ is to be interpreted as a physical density), and g a scalar function of the state (an “observable”). We define the following transfer operators, for a given lag time $\tau$:

(a)

The Perron–Frobenius operator (also called propagator),

$${\mathcal{P}}^{\tau }f(x)={\int }_{\mathbb{X}}f(y)p^{\tau }(y,x)\mathrm{d}y$$

evolves probability distributions.

(b)

The Perron–Frobenius operator with respect to the equilibrium density (also called transfer operator, simply),

$${\mathcal{T}}^{\tau }u(x)=\frac{1}{\mu (x)}{\int }_{\mathbb{X}}u(y)\mu (y)p^{\tau }(y,x)\mathrm{d}y.$$

evolves densities with respect to $\mu$.

(c)

The Koopman operator

$${\mathcal{K}}^{\tau }g(x)={\int }_{\mathbb{X}}{p}^{\tau }(x,y)g(y)\mathrm{d}y=\mathsf{E}[g(x_{\tau })\mid x_0=x]$$

(3)

evolves observables.

We denote by $L^q=L^q(\mathbb{X})$ the space (equivalence class) of q-integrable functions with respect to the Lebesgue measure. $L_{\nu }^q$ denotes the same space of function, now integrable with respect to the weight function $\nu$. All our transfer operators are well-defined non-expanding operators on the following Hilbert spaces: ${\mathcal{P}}^{\tau }:L_{1/\mu }^2\to L_{1/\mu }^2$, ${\mathcal{T}}^{\tau }:L_{\mu }^2\to L_{\mu }^2$, and ${\mathcal{K}}^{\tau }:L_{\mu }^2\to L_{\mu }^2$ [31,32,33]. The equilibrium density $\mu$ satisfies ${\mathcal{P}}^{\tau }\mu =\mu$, that is, $\mu$ is an eigenfunction of ${\mathcal{P}}^{\tau }$ with associated eigenvalue ${\lambda }_0=1$. The definition of ${\mathcal{T}}^{\tau }$ relies on $\mu$, we have

$$\mu {\mathcal{T}}^{\tau }u={\mathcal{P}}^{\tau }(u\mu ),$$

(4)

thus ${\mathcal{P}}^{\tau }\mu =\mu$ translates into ${\mathcal{T}}^{\tau }11=11$, where $11=11_{\mathbb{X}}$ is the constant one function on $\mathbb{X}$.

2.2. Reversible Equilibrium Dynamics and Spectral Decomposition

An important structural property of many systems used to model molecular dynamics is reversibility. Reversibility means that the process is statistically indistinguishable from its time-reversed counterpart, and it can be described by the detailed balance condition

$$\mu (x)p^t(x,y)=\mu (y)p^t(y,x)\forall x,y\in \mathbb{X},t\ge 0.$$

(5)

The process generated by (1) is reversible and ergodic, i.e., it has a unique positive equilibrium density, given by $\mu (x)\propto exp(-\beta W(x))$, under mild conditions on the potential W [34,35]. The subsequent considerations hold for all reversible and ergodic (with respect to a unique positive invariant density) stochastic processes, and are not limited to the class of systems given by (1). Ref. [1] discusses a variety of stochastic dynamical systems that have been considered in this context. Furthermore, if $p^t(·,·)$ is a continuous function in both its arguments for $t>0$, then all the transfer operators above are compact, which we also assume from now on. This implies that they have a discrete eigen- and singular spectrum (the latter meaning it has a discrete set of singular values). For instance, the process generated by (1) has continuous transition density function under mild growth and regularity assumptions on the potential W.

As a result of the detailed balance condition, the Koopman operator ${\mathcal{K}}^{\tau }$ and the Perron–Frobenius operator with respect to the equilibrium density ${\mathcal{T}}^{\tau }$ become identical and we obtain

$${⟨{\mathcal{P}}^{\tau }f,g⟩}_{1/\mu }={⟨f,{\mathcal{P}}^{\tau }g⟩}_{1/\mu }\mathrm{and}{⟨{\mathcal{T}}^{\tau }f,g⟩}_{\mu }={⟨f,{\mathcal{T}}^{\tau }g⟩}_{\mu },$$

(6)

i.e., all the transfer operators become self-adjoint on the respective Hilbert spaces from above. Here, ${⟨·,·⟩}_{\nu }$ denotes the natural scalar products on the weighted space $L_{\nu }^2$, i.e., ${⟨f,g⟩}_{\nu }={\int }_{\mathbb{X}}f(x)g(x)\nu (x)\mathrm{d}x$; the associated norm is denoted by ${\parallel ·\parallel }_{\nu }$. Due to the self-adjointness, the eigenvalues ${\lambda }_i^{\tau }$ of ${\mathcal{P}}^{\tau }$ and ${\mathcal{T}}^{\tau }$ are real-valued and the eigenfunctions form an orthogonal basis with respect to ${⟨·,·⟩}_{1/\mu }$ and ${⟨·,·⟩}_{\mu }$, respectively.

Ergodicity implies that the dominant eigenvalue ${\lambda }_1$ is the only eigenvalue with absolute value 1 and we can thus order the eigenvalues so that

$$1={\lambda }_1^{\tau }>{\lambda }_2^t\ge {\lambda }_3^t\ge \cdots .$$

The eigenfunction of ${\mathcal{T}}^{\tau }$ corresponding to ${\lambda }_1=1$ is the constant function ${\varphi }_1={\mathbb{1}}_{\mathbb{X}}$. Let ${\varphi }_i$ be the normalized eigenfunctions of ${\mathcal{T}}^{\tau }$, i.e., ${⟨{\varphi }_i,{\varphi }_j⟩}_{\mu }={\delta }_{ij}$, where ${\delta }_{ij}$ denotes the Kronecker-delta. Then any function $f\in L_{\mu }^2$ can be written in terms of the eigenfunctions as $f={\sum }_{i=1}^{\infty }{⟨f,{\varphi }_i⟩}_{\mu }{\varphi }_i$. Applying ${\mathcal{T}}^{\tau }$ thus results in

$${\mathcal{T}}^{\tau }f=\sum _{i=1}^{\infty }{\lambda }_i^{\tau }{⟨f,{\varphi }_i⟩}_{\mu }{\varphi }_i.$$

For more details, we refer to [33] and references therein.

For some $k\in \mathbb{N}$, we call the k dominant eigenvalues ${\lambda }_1^{\tau },\cdots ,{\lambda }_k^{\tau }$ of ${\mathcal{T}}^{\tau }$ the dominant spectrum of ${\mathcal{T}}^{\tau }$, i.e.,

$${\lambda }_{\mathrm{dom}}({\mathcal{T}}^{\tau })=\{{\lambda }_1^{\tau },\cdots ,{\lambda }_k^{\tau }\}.$$

Usually, k is chosen in such a way that there is a spectral gap after ${\lambda }_k^{\tau }$, i.e., $1-{\lambda }_k^{\tau }\ll {\lambda }_k^{\tau }-{\lambda }_{k+1}^{\tau }$. The (implied) time scales on which the associated dominant eigenfunctions decay are given by

$$t_i=-\tau /log({\lambda }_i^{\tau }).$$

(7)

If ${\{{\mathcal{T}}^t\}}_{t\ge 0}$ is a semigroup of operators (which is the case for every time-homogeneous process, as, e.g., the transfer operator associated with (1)), then there are ${\kappa }_i\le 0$ with ${\lambda }_i^{\tau }=exp({\kappa }_i\tau )$ such that $t_i=-{\kappa }_i^{-1}$ holds. Assuming there is a spectral gap, the dominant time scales satisfy $\infty =t_1>\dots \ge t_k\gg t_{k+1}$. These are the time scales of the slow dynamical processes, also called rare events, which are of primary interest in applications. The other, fast processes are regarded as fluctuations around the relative equilibria (or metastable states) between which the relevant slow processes travel.

3. Markov State Models for Reversible Systems in Equilibrium

In the following, we will fix a lag time $\tau >0$, and drop the superscript $\tau$ from the transfer operators for clarity of notation.

3.1. Preliminaries on Equilibrium Markov State Models

Generally, in the equilibrium case, a generalized MSM (GMSM) is any matrix $T_k\in {\mathbb{R}}^{n_k\times n_k}$, $n_k\ge k$, that approximates the k dominant time scales of $\mathcal{T}$, i.e., its dominant eigenvalues;

$${\lambda }_{\mathrm{dom}}(T_k)\approx {\lambda }_{\mathrm{dom}}(\mathcal{T}).$$

(8)

It is natural to ask for some structural properties of $\mathcal{T}$ to be reproduced by $T_k$, such as:

• $\mathcal{T}$ is a positive operator ⟷ all entries of $T_k$ are non-negative;
• $\mathcal{T}$ is probability-preserving ⟷ each column sum of $T_k$ is 1.

These two properties together make $T_k$ to a stochastic matrix, and in this case $T_k$ is usually called a MSM. We shall use the term Generalized MSM for a matrix $T_k$ that violates these requirements but still approximates the dominant spectral components of the underlying operator. Another structural property that one would usually ask for is to have apart from the time scales/eigenvalues also some approximation of the associated eigenvectors of $\mathcal{T}$, as these are the dynamic observables related to the slow dynamics. This is incorporated in the general approach, which we discuss next.

The question is now how to obtain a GMSM $T_k$ for a given $\mathcal{T}$. To connect these objects, a natural and popular approach is to obtain the reduced model $T_k$ via projection. To this end, let $\mathcal{Q}:L_{\mu }^2\to \mathbb{V}\subset L_{\mu }^2$ be a projection onto a $n_k$-dimensional subspace $\mathbb{V}$. The GMSM is then defined by the projected transfer operator

$${\mathcal{T}}_k=\mathcal{Q}\mathcal{T}\mathcal{Q};$$

(9)

and $T_k$ can now be taken as the matrix representation of the $\mathbb{V}$-restriction of the projected operator ${\mathcal{T}}_k:\mathbb{V}\to \mathbb{V}$ with respect to a chosen basis of $\mathbb{V}$.

Is there a “best” choice for the projection? If we also ask for perfect approximation of the time scales, i.e., ${\lambda }_{\mathrm{dom}}(T_k)={\lambda }_{\mathrm{dom}}(\mathcal{T})$, the requirement of parsimony—such that the model size is minimal, i.e., $n_k=k$—leaves us with a unique choice for $\mathbb{V}$, namely the space spanned by the dominant (normalized) eigenfunctions ${\varphi }_i$ of $\mathcal{T}$, $i=1,\dots ,k$. This follows from the so-called variational principle (or Rayleigh–Ritz method) [9,10]. In fact, it makes a stronger claim: every projection to a k-dimensional space ${\mathbb{V}}^{\prime }$ yields a GMSM ${\mathcal{T}}_k^{\prime }:{\mathbb{V}}^{\prime }\to {\mathbb{V}}^{\prime }$ which underestimates the dominant time scales, i.e., ${\lambda }_i({\mathcal{T}}_k^{\prime })\le {\lambda }_i(\mathcal{T})$, $i=1,\dots ,k$; and equality holds only for the projections on the eigenspaces.

Note that the discussion about the time scales (equivalently, the eigenvalues) involves only the range of the projection, the space $\mathbb{V}$. However, there are multiple ways to project on the space $\mathbb{V}$. It turns out, that the $\mu$-orthogonal projection given by

$$\mathcal{Q}f=\sum _{i=1}^k{⟨{\varphi }_i,f⟩}_{\mu }{\varphi }_i$$

(10)

is superior to all of them, if we consider a stronger condition than simply reproducing the dominant time scales. This condition is the requirement of minimal propagation error, and it will be central to our generalization of GMSMs for non-equilibrium, or even time-inhomogeneous systems. Let us define the best k-dimensional approximation ${\mathcal{T}}_k$ to $\mathcal{T}$, i.e., the best projection $\mathcal{Q}$, as the rank-k operator satisfying

$$\parallel \mathcal{T}-{\mathcal{T}}_k\parallel \le \parallel \mathcal{T}-{\mathcal{T}}_k^{\prime }\parallel ,$$

(11)

where $\parallel ·\parallel$ denotes the induced operator norm for operators mapping $L_{\mu }^2$ to itself. The induced norm of an operator $\mathcal{A}:\mathbb{X}\to \mathbb{Y}$ is defined by $\parallel \mathcal{A}\parallel ={max}_{{\parallel f\parallel }_{\mathbb{X}}=1}{\parallel \mathcal{A}f\parallel }_{\mathbb{Y}}$, where ${\parallel ·\parallel }_{\mathbb{X}}$ and ${\parallel ·\parallel }_{\mathbb{Y}}$ are the norms on the spaces $\mathbb{X}$ and $\mathbb{Y}$, respectively.

Equivalently, (11) can be viewed as a result stating that ${\mathcal{T}}_k$ is the k-dimensional approximation of $\mathcal{T}$ yielding the smallest (worst-case) error in density propagation:

$${\mathcal{T}}_k=\underset{\begin{array}{c}{\mathcal{T}}_k^{\prime }={\mathcal{Q}}^{\prime }\mathcal{T}{\mathcal{Q}}^{\prime }\\ \mathrm{rank}{\mathcal{Q}}^{\prime }=k\end{array}}{\arg \min }\underset{{\parallel f\parallel }_{\mu }=1}{\max }{\parallel \mathcal{T}f-{\mathcal{T}}_k^{\prime }f\parallel }_{\mu },$$

(12)

where $x_{*}={\arg \min }_{x}h(x)$ means that $x_{*}$ minimizes the function h, possibly subject to constraints that are listed under arg min.

To summarize, the best GMSM (9) in terms of (11) (or, equivalently, (12)) is given by the projection (10). This follows from the self-adjointness of $\mathcal{T}$ and the Eckard–Young theorem; details can be found in [30] and in Appendix A. Caution is needed however, when interpreting ${\mathcal{T}}_{k}f$ as the propagation of a given probability density f. The projection to the dominant eigenspace in general does not respect positivity (i.e., $f\ge 0\nRightarrow {\mathcal{T}}_{k}f\ge 0$), thus ${\mathcal{T}}_{k}f$ loses its probabilistic meaning. This is the price to pay for the perfectly reproduced dominant time scales. We can retain a physical interpretation of a MSM if we accept that the dominant time scales will be slightly off, as we discuss in the next section.

3.2. Metastable Sets

There is theoretical evidence [1] that the more pronounced the metastable behavior of system is (in the sense that the size of the time scale gap $t_1\ge \dots \ge t_k\gg t_{k+1}$ is large), the more constant the dominant eigenfunctions ${\varphi }_i$ are on the metastable sets ${\mathbb{M}}_1,\dots ,{\mathbb{M}}_d$, given the lag time with respect to which the transfer operator $\mathcal{T}={\mathcal{T}}^{\tau }$ is taken satisfies $\tau \gg t_{k+1}$. Assuming such a situation, the eigenfunctions of $\mathcal{T}$ can approximately be combined from the characteristic functions over the metastable sets, i.e., with the abbreviation $11_i:=11_{{\mathbb{M}}_i}$ it holds that

$${\varphi }_i\approx \sum _{j=1}^k{c}_{ij}11_j=:{\widehat{\varphi }}_i,$$

(13)

where the $c_{ij}$ are components of the linear combination, such that the ${\widehat{\varphi }}_i$ are orthonormal. Using the “approximate eigenfunctions” ${\widehat{\varphi }}_i$ defined in (13), the modified projection

$$\widehat{\mathcal{Q}}f=\sum _{i=1}^k{⟨{\widehat{\varphi }}_i,f⟩}_{\mu }{\widehat{\varphi }}_i$$

(14)

defines a new MSM ${\widehat{\mathcal{T}}}_k:=\widehat{\mathcal{Q}}\mathcal{T}\widehat{\mathcal{Q}}$. Since $\mathbb{V}=\mathrm{span}\{{\varphi }_i\}\approx \mathrm{span}\{{\widehat{\varphi }}_i\}=\widehat{\mathbb{V}}$, also $\widehat{\mathcal{Q}}\approx \mathcal{Q}$, and thus we have ${\widehat{\mathcal{T}}}_k\approx {\mathcal{T}}_k$. This implies [36], Lemma 3.5 that also their dominant eigenvalues, hence time scales are close. Further, we have that in the basis ${\{11_i/{⟨11_i,11_i⟩}_{\mu }\}}_{i=1}^k$ the matrix representation ${\widehat{T}}_k$ of the $\widehat{\mathbb{V}}$-restriction of the operator ${\widehat{\mathcal{T}}}_k$ has the entries

$$\begin{array}{cc}\hfill {\widehat{T}}_{k,ij}& =\frac{{⟨11_i,\mathcal{T}11_j⟩}_{\mu }}{{⟨11_j,11_j⟩}_{\mu }}\hfill \\ & ={\int }_{{\mathbb{M}}_i}\mathcal{T}\left(\frac{11_j}{{⟨11_j,11_j⟩}_{\mu }}\right)\mu (x)dx\hfill \\ & =\frac{1}{{\mathsf{P}}_{\mu }[x_0\in {\mathbb{M}}_j]}{\int }_{{\mathbb{M}}_i}{\int }_{{\mathbb{M}}_j}\mu (x)p^t(x,y)dxdy\hfill \\ & ={\mathsf{P}}_{\mu }[x_t\in {\mathbb{M}}_i|x_0\in {\mathbb{M}}_j],\hfill \end{array}$$

(15)

where ${\mathsf{P}}_{\mu }[·|x\in \mathbb{M}]$ denotes the probability measure that arises if $x\in \mathbb{M}$ has distribution $\mu$ (restricted to $\mathbb{M}$). That is, ${\widehat{T}}_k$ has the transition probabilities between the metastable sets as entries, giving a direct physical interpretation of the MSM. Note, however, that for this approximation to reproduce the dominant time scales well, i.e., to have $t_i\approx {\widehat{t}}_i$, $i=1,\dots ,k$, we need a strong separation of time scales in the sense that $t_k\gg t_{k+1}$ has to hold, and the lag time $\tau$ needs to be chosen sufficiently large [7].

3.3. Example: Stationary Diffusion in Double-Well Potential

Let us consider the diffusion (1) in the potential landscape $W(x)={(x^2-1)}^2$ with $\beta =5$; cf. Figure 1 (left). With the lag time $\tau =10$ we approximate the Perron–Frobenius operator $\mathcal{P}={\mathcal{P}}^t$ and compute its eigenvector $\mu$ at the eigenvalue ${\lambda }_1=1$. Then, we compute the transfer operator $\mathcal{T}={\mathcal{T}}^{\tau }$ with respect to the stationary distribution $\mu$, and its dominant eigenvalues ${\lambda }_2,{\lambda }_3,\dots$ and corresponding eigenvectors ${\varphi }_2,{\varphi }_3,\dots$ (Figure 1, right). While ${\lambda }_2=0.888$, we have $|{\lambda }_3|<{10}^{-12}$, hence we have a clear time scale separation, $t_2=84.1\gg 0.35=t_3$, cf. (7).

Figure 1. Left: double-well potential. Right: invariant distribution $\mu$ (gray dashed) and second eigenfunction ${\varphi }_2$ (solid black) of the associated transfer operator $\mathcal{T}$.

Thus, we expect a rank-2 MSM to recover the dominant time scales very well. Indeed, choosing ${\mathbb{M}}_1=(-\infty ,0]$ and ${\mathbb{M}}_2=[0,\infty )$ gives ${\varphi }_2\approx -11_{{\mathbb{M}}_1}+11_{{\mathbb{M}}_2}$, and we obtain by (15) that

$${\widehat{T}}_2=\left(\begin{array}{cc}0.943& 0.057\\ 0.057& 0.943\end{array}\right).$$

This is a stochastic matrix with eigenvalues ${\widehat{\lambda }}_1=1$ and ${\widehat{\lambda }}_2=0.886$, i.e., yielding an approximate time scale ${\widehat{t}}_2=82.4$.

4. Markov State Models for Time-Inhomogeneous Systems

As all our non-equilibrium cases will be special instances of the most general, time-inhomogeneous case, we consider this next.

4.1. Minimal Propagation Error by Projections

4.1.1. Conceptual Changes

The above approach to Markov state modeling is relying on the existence of an stationary distribution and reversibility. In the case of a time-inhomogeneous system there will not be, in general, any stationary distribution $\mu$. Additionally, we are lacking physical meaning, since it is unclear with respect to which ensemble the dynamical fluctuations should be described. From a mathematical perspective there is a problem as well, since the construction relies on the reversibility of the underlying system, which gives the self-adjointness of the operator $\mathcal{T}$ with respect to the weighted scalar product ${⟨·,·⟩}_{\mu }$. Time-inhomogeneous systems are not reversible in general.

Additionally to these structural properties, we might need to depart from some conceptional ones as well. As time-inhomogeneity usually stems from an external forcing that might not be present or known for all times, we need a description of the system on a finite time interval. This disrupts the concept of dominant time scales as they are considered in equilibrium systems, because there it relies on self-similarity of observing an eigenmode over and over for arbitrary large times. It also forces us to re-visit the concept of metastability for two reasons. First, many definitions of metastability rely on statistics under the assumption that we observe the system for infinitely long times. Second, as an external forcing may theoretically arbitrarily distort the energy landscape, it is a priori unclear what could be a metastable set.

As a remedy, we aim at another property when trying to reproduce the effective behavior of the full system by a reduced model; this will be minimizing the propagation error, as in (12). Remarkably, this will also allow for a physical interpretation through so-called coherent sets; analogous to metastable sets in the equilibrium case.

A prototypical time-inhomogeneous system can be given by

$$\mathrm{d}{x}_t=-\nabla W(t,x_t)\mathrm{d}t+\sqrt{2{\beta }^{-1}}\mathrm{d}{w}_t,$$

(16)

where the potential W now depends explicitly on time t. In this case, a lag time $\tau$ is not sufficient to parametrize the statistical evolution of the system, because we need to know when we start the evolution. Thus, transition density functions need two time parameters, e.g., $p^{s,t}(x,·)$ denotes the distribution of $x_t$ conditional to $x_s=x$. Similarly, the transfer operators $\mathcal{P},\mathcal{T},\mathcal{K}$ are parametrized by two times as well, e.g., ${\mathcal{P}}^{s,t}$ propagates probability densities from initial time s to final time t (alternatively, from initial time s for lag time $\tau =t-s$). To simplify notation, we will settle for some initial and final times, and drop these two time parameters, as they stay fixed.

4.1.2. Adapted Transfer Operators

Let us observe the system from initial time $t_0$ to final time $t_1$, such that its distribution at initial time is given by ${\mu }_0$. Then, if $\mathcal{P}$ denotes the propagator of the system from $t_0$ to $t_1$, then we can express the final distribution at time $t_1$ by ${\mu }_1=\mathcal{P}{\mu }_0$. As the transfer operator in equilibrium case was naturally mapping $L_{\mu }^2$ to itself (because $\mu$ was invariant), here it is natural to consider the transfer operator mapping densities (functions) with respect to ${\mu }_0$ to densities with respect to ${\mu }_1$. Thus, we define the transfer operator $\mathcal{T}:L_{{\mu }_0}^2\to L_{{\mu }_1}^2$ by

$$\mathcal{T}u:=\frac{1}{{\mu }_1}\mathcal{P}\left(u{\mu }_0\right),$$

(17)

which is the non-equilibrium analogue to (4). This operator naturally retains some properties of the equilibrium transfer operator [37]:

• $\mathcal{T}11=11$, encoding the property that ${\mu }_0$ is mapped to ${\mu }_1$ by the propagator $\mathcal{P}$.
• $\mathcal{T}$ is positive and integral-preserving, thus ${\sigma }_{max}(\mathcal{T})=1$.
• Its adjoint is the Koopman operator $\mathcal{K}:L_{{\mu }_1}^2\to L_{{\mu }_0}^2$, $\mathcal{K}g(x)=\mathsf{E}\left[g(x_t)\right|x_0=x]$.

4.1.3. An Optimal Non-Stationary GMSM

As already mentioned above, it is not straightforward how to address the problem of Markov state modeling in this time-inhomogeneous case via descriptions involving time scales or metastability. Instead, our strategy will be to search for a rank-k projection ${\mathcal{T}}_k$ of the transfer operator $\mathcal{T}$ with minimal propagation error, to be described below.

The main point is now that due to the non-stationarity the domain $L_{{\mu }_0}^2$ (where $\mathcal{T}$ maps from) and range $L_{{\mu }_1}^2$ (where $\mathcal{T}$ maps to) of the transfer operator $\mathcal{T}$ are different spaces, hence it is natural to choose different rank-k subspaces as domain and range of ${\mathcal{T}}_k$ too. In fact, it is necessary to choose domain and range differently, since $f\in L_{{\mu }_0}^2$ has a different meaning than $f\in L_{{\mu }_1}^2$. Thus, we will search for projectors ${\mathcal{Q}}_0:L_{{\mu }_0}^2\to {\mathbb{V}}_0\subset L_{{\mu }_0}^2$ and ${\mathcal{Q}}_1:L_{{\mu }_1}^2\to {\mathbb{V}}_1\subset L_{{\mu }_1}^2$ on different k-dimensional subspaces ${\mathbb{V}}_0$ and ${\mathbb{V}}_1$, respectively, such that the reduced operator

$${\mathcal{T}}_k:={\mathcal{Q}}_1\mathcal{T}{\mathcal{Q}}_0$$

(18)

has essentially optimal propagation error. In quantitative terms, we seek to solve the optimization problem

$${\mathcal{T}}_k=\underset{\begin{array}{c}{\mathcal{T}}_k^{\prime }={\mathcal{Q}}_1^{\prime }\mathcal{T}{\mathcal{Q}}_0^{\prime }\\ \mathrm{rank}{\mathcal{Q}}_0^{\prime }=k\\ \mathrm{rank}{\mathcal{Q}}_1^{\prime }=k\end{array}}{\arg \min }\underset{{\parallel f\parallel }_{{\mu }_0}=1}{max}\parallel \mathcal{T}f-{\mathcal{T}}_k^{\prime }{f\parallel }_{{\mu }_1}\mathrm{or},\mathrm{equivalently}{\mathcal{T}}_k=\underset{\begin{array}{c}{\mathcal{T}}_k^{\prime }={\mathcal{Q}}_1^{\prime }\mathcal{T}{\mathcal{Q}}_0^{\prime }\\ \mathrm{rank}{\mathcal{Q}}_0^{\prime }=k\\ \mathrm{rank}{\mathcal{Q}}_1^{\prime }=k\end{array}}{\arg \min }\parallel \mathcal{T}-{\mathcal{T}}_k^{\prime }\parallel ,$$

(19)

where $\parallel ·\parallel$ denotes the induced operator norm of operators mapping $L_{{\mu }_0}^2$ to $L_{{\mu }_1}^2$.

As an implication of the Eckart–Young theorem [38] Theorem 4.4.7, the solution of (19) can explicitly be given through singular value decomposition of $\mathcal{T}$; yielding the variational approach for Markov processes (VAMP) [30]. More precisely, the k largest singular values ${\sigma }_1\ge \dots \ge {\sigma }_k$ of $\mathcal{T}$ have right and left singular vectors ${\varphi }_i,{\psi }_i$ satisfying ${⟨{\varphi }_i,{\varphi }_j⟩}_{{\mu }_0}={\delta }_{ij},{⟨{\psi }_i,{\psi }_j⟩}_{{\mu }_1}={\delta }_{ij}$, respectively, i.e., $\mathcal{T}{\varphi }_i={\sigma }_i{\psi }_i$. Choosing

$${\mathcal{Q}}_{0}f=\sum _{i=1}^k{⟨{\varphi }_i,f⟩}_{{\mu }_0}{\varphi }_i\mathrm{and}{\mathcal{Q}}_{1}g=\sum _{i=1}^k{⟨{\psi }_i,g⟩}_{{\mu }_1}{\psi }_i$$

(20)

solves (19), see Appendix A.

4.2. Coherent Sets

Similarly to the reversible equilibrium case with pronounced metastability in Section 3.2, it is also possible in the time-inhomogeneous case to give our GMSM (18) from Section 4.1 a physical interpretation—under some circumstances.

In the reversible equilibrium situation, recall from (13) that in the case of sufficient time scale separation the eigenfunctions are almost constant on metastable sets. In the time-inhomogeneous situation, considered now, we have just shown that the role played before by the eigenfunctions is taken by left- and right singular functions. Thus, let us assume for now that there are two collections of sets, ${\mathbb{M}}_{0,1},\dots ,{\mathbb{M}}_{0,k}$ at initial time, and ${\mathbb{M}}_{1,1},\dots ,{\mathbb{M}}_{1,k}$ at final time, such that

$${\varphi }_i\approx \sum _{j=1}^k{c}_{ij}11_{0,j}=:{\widehat{\varphi }}_i\mathrm{and}{\psi }_i\approx \sum _{j=1}^k{d}_{ij}11_{1,j}=:{\widehat{\psi }}_i$$

(21)

holds with appropriate scalars $c_{ij}$ and $d_{ij}$, where we used the abbreviation $11_{0,i}=11_{{\mathbb{M}}_{0,i}}$ and $11_{1,i}=11_{{\mathbb{M}}_{1,i}}$. That means, dominant right singular functions ${\varphi }_i$ are almost constant on the sets ${\mathbb{M}}_{0,j}$, and dominant left singular functions ${\psi }_i$ are almost constant on the sets ${\mathbb{M}}_{1,j}$. In analogy to (14), we modify the projections ${\mathcal{Q}}_0,{\mathcal{Q}}_1$ from (20) to ${\widehat{\mathcal{Q}}}_0:L_{{\mu }_0}^2\to {\widehat{\mathbb{V}}}_0,{\widehat{\mathcal{Q}}}_1:L_{{\mu }_1}^2\to {\widehat{\mathbb{V}}}_1$ by using ${\widehat{\varphi }}_i$ and ${\widehat{\psi }}_i$ instead of ${\varphi }_i$ and ${\psi }_i$, respectively, and define the modified GMSM by ${\widehat{\mathcal{T}}}_k={\widehat{\mathcal{Q}}}_1\mathcal{T}{\widehat{\mathcal{Q}}}_0$. An analogous computation to (15) yields for the matrix representation ${\widehat{T}}_k$ of the restriction ${\widehat{\mathcal{T}}}_k:{\widehat{\mathbb{V}}}_0\to {\widehat{\mathbb{V}}}_1$ with respect to the bases ${\{11_{0,i}/{⟨11_{0,i},11_{0,i}⟩}_{{\mu }_0}\}}_{i=1}^k$ and ${\{11_{1,i}/{⟨11_{1,i},11_{1,i}⟩}_{{\mu }_1}\}}_{i=1}^k$ that

$${\widehat{T}}_{k,ij}={\mathsf{P}}_{{\mu }_0}\left[x_{t_1}\in {\mathbb{M}}_{1,i}|x_{t_0}\in {\mathbb{M}}_{0,j}\right].$$

(22)

In other words, the entries of ${\widehat{T}}_k$ contain the transition probabilities from the sets ${\mathbb{M}}_{0,i}$ (at initial time) into the sets ${\mathbb{M}}_{1,j}$ (at final time). Thus, ${\widehat{T}}_k$ has the physical interpretation of a MSM, with the only difference to the reversible stationary situation being that the “metastable” sets at initial and final time are different. This can be seen as a natural reaction to the fact that in the time-inhomogeneous case the dynamical environment (e.g., the potential energy landscape governing the dynamics of a molecule) can change in time.

It remains to discuss when does (21) actually hold true. It is comprehensively discussed in [17] that a sufficient condition for (21) is if

$${\mathsf{P}}_{{\mu }_0}\left[x_{t_1}\in {\mathbb{M}}_{1,i}|x_{t_0}\in {\mathbb{M}}_{0,i}\right]\approx 1\mathrm{and}{\mathsf{P}}_{{\mu }_1}\left[x_{t_0}\in {\mathbb{M}}_{0,i}|x_{t_1}\in {\mathbb{M}}_{1,i}\right]\approx 1$$

(23)

holds for $i=1,\dots ,k$. Equation (23) says that if the process starts in ${\mathbb{M}}_{0,i}$, it ends up at final time with high probability in ${\mathbb{M}}_{1,i}$, and that if the process ended up in ${\mathbb{M}}_{1,i}$ at final time, in started with high probability in ${\mathbb{M}}_{0,i}$; see Figure 2. This can be seen as a generalization of the metastability condition from Section 3.2 that allows for an efficient low-rank Markov modeling in the time-homogeneous case. The pairs of sets ${\mathbb{M}}_{0,i},{\mathbb{M}}_{1,i}$ are called coherent (set) pair, and they have been shown to be very effective tools identifying time-dependent regions in non-autonomous flow fields that do not mix with their surrounding (this is, effectively, what (23) says), e.g., moving vortices in atmospheric and oceanographic applications [15,16,39,40]. More details on the generalization of the concept of metastability by coherent sets, and on subsequent Markov state modeling can be found in [17].

Figure 2. Cartoon representation of (23) with a coherent pair ${\mathbb{M}}_{0,i},{\mathbb{M}}_{1,i}$, which are represented by the thick horizontal lines left and right, respectively. Condition (23) can be translated into $\mathcal{T}11_{0,i}\approx 11_{1,i}$, or equivalently $\mathcal{P}(11_{0,i}{\mu }_0)\approx 11_{1,i}{\mu }_1$. In other words, the part of the ensemble ${\mu }_0$ supported on the set ${\mathbb{M}}_{0,i}$ (dark gray region on the left) is mapped by the propagator to an ensemble (dark gray region on the right) that is almost equal to the part of the ensemble ${\mu }_1$ supported on ${\mathbb{M}}_{1,i}$. Note that little of $\mathcal{P}(11_{0,i}{\mu }_0)$ is supported outside of ${\mathbb{M}}_{1,i}$, and little of $11_{1,i}{\mu }_1$ came from outside ${\mathbb{M}}_{0,i}$.

4.3. Example: Diffusion in Shifting Triple-Well Potential

Let us consider the diffusion (1) in the time-dependent potential landscape

$$W(t,x)=7{\left((x-t/10)(x-1-t/10)(x+1-t/10)\right)}^2$$

with $\beta =5$ and on the time interval $[t_0,t_1]=[0,10]$; cf. Figure 3 (left). Taking the initial distribution ${\mu }_0\propto exp(-\beta W(0,·))$, we build the transfer operator (17), and consider its dominant singular values:

$${\sigma }_1=1.000,{\sigma }_2=0.734,{\sigma }_3=0.536,{\sigma }_4\approx 0.$$

Figure 3. Left: shifting triple-well potential. All three wells are coherent sets, as the plateaus of the singular vectors indicate. Middle: second right (initial) and left (final) singular vectors of the transfer operator (solid black and gray dashed lines, respectively). Right: third right (initial) and left (final) singular vectors of the transfer operator (solid black and gray dashed lines, respectively). The singular vectors are for reasons of numerical stability only computed in regions where ${\mu }_0$ and ${\mu }_1$ are, respectively, larger than machine precision.

This indicates that a rank-3 GMSM is sufficient to approximate the system, and that we have three coherent sets. We observe the characteristic almost constant behavior (21) of the left and right singular vectors over the respective coherent sets; Figure 3 (middle and right). Recall that right singular vectors show coherent sets at initial time, and left singular vectors the associated coherent sets at final time.

We can identify the three wells as three coherent sets. Figure 4 shows that they are coherent indeed: the respective parts of the initial ensemble ${\mu }_0$ is to a large extent mapped onto the corresponding part of the final ensemble ${\mu }_1$, cf. Figure 2 and (23).

Figure 4. Top: initial ensemble ${\mu }_0$ (black solid) and its respective parts in the three coherent sets (gray shading). Bottom: final ensemble ${\mu }_1$ (black solid) and the image of the corresponding gray ensembles from the top row (gray shading).

Computing the MSM from the transition probabilities between the coherent sets as in (22) gives the stochastic matrix

$${\widehat{T}}_3=\left(\begin{array}{ccc}0.794& 0.150& 0.026\\ 0.196& 0.767& 0.274\\ 0.010& 0.083& 0.701\end{array}\right).$$

The initial distribution ${\widehat{\mu }}_0$ of this MSM is given by the probability that ${\mu }_0$ assigns to the respective coherent sets at initial time. Analogously, collecting the probabilities from ${\mu }_1$ in the coherent sets at final time gives the final distribution ${\widehat{\mu }}_1$ of the MSM. We have

$${\widehat{\mu }}_0=\left(\begin{array}{c}0.250\\ 0.500\\ 0.250\end{array}\right)\mathrm{and}{\widehat{\mu }}_1=\left(\begin{array}{c}0.280\\ 0.500\\ 0.219\end{array}\right).$$

The singular values of ${\widehat{T}}_k$ as mapping from the ${\widehat{\mu }}_0$-weighted ${\mathbb{R}}^3$ to the ${\widehat{\mu }}_1$-weighted ${\mathbb{R}}^3$ are

$${\widehat{\sigma }}_1=1.000,{\widehat{\sigma }}_2=0.733,{\widehat{\sigma }}_3=0.534;$$

they are in good agreement with the true singular values of $\mathcal{T}$.

We repeat the computation with a different initial distribution ${\mu }_0$, where only the left and right well are initially populated, as shown in Figure 5.

Figure 5. The same as Figure 4, for a different initial distribution ${\mu }_0$.

The largest singular values of $\mathcal{T}$,

$${\sigma }_1=1.000,{\sigma }_2=0.643,{\sigma }_3=0.030,{\sigma }_4\approx 0,$$

already show that there are only two coherent sets, as the third singular value is significantly smaller than the second one. The left well forms one coherent set, and the union of the middle and right ones form the second coherent set.

5. Data-Based Approximation

5.1. Setting and Auxiliary Objects

We would like to estimate the GMSM (18) from trajectory data. In the time-inhomogeneous setting, let us assume that we have m data points $x_1,\dots ,x_m$ at time $t_0$, and their (random) images $y_1,\dots ,y_m$ at time $t_1$, meaning that $y_i$ is a random sample of the underlying process at time $t_1$, given it started in $x_i$ at time $t_0$. We can think of the empirical distribution of the $x_i$ and $y_i$ being estimates of ${\mu }_0$ and ${\mu }_1$, respectively.

Let us further define two sets of basis functions ${\chi }_{0,1},\dots ,{\chi }_{0,n}$ and ${\chi }_{1,1},\dots ,{\chi }_{1,n}$, which we would like to use to approximate the GMSM. If we would like to estimate the first k dominant modes, the least requirement is $n\ge k$; in general we have $n\gg k$. The vector-valued functions

$${\chi }_0=\left(\begin{array}{c}{\chi }_{0,1}\\ ⋮\\ {\chi }_{0,n}\end{array}\right),{\chi }_1=\left(\begin{array}{c}{\chi }_{1,1}\\ ⋮\\ {\chi }_{1,n}\end{array}\right)$$

are basis functions at initial and final times, respectively. One can take ${\chi }_0$ and ${\chi }_1$ to have different lengths too, we just chose them to have the same lengths for convenience. Now we can define the data matrices

$${\mathbf{\chi }}_0=\left(\begin{array}{cc}|& |\\ {\chi }_0(x_1)& \dots & {\chi }_0(x_m)\\ |& |\end{array}\right),{\mathbf{\chi }}_1=\left(\begin{array}{cc}|& |\\ {\chi }_1(y_1)& \dots & {\chi }_1(y_m)\\ |& |\end{array}\right).$$

The following $n\times n$ correlation matrices $C_{00},C_{01},C_{11}$ will be needed later:

$$C_{00,ij}={⟨{\chi }_{0,i},{\chi }_{0,j}⟩}_{{\mu }_0},C_{01,ij}={⟨{\chi }_{1,i},\mathcal{T}{\chi }_{0,j}⟩}_{{\mu }_1},C_{11,ij}={⟨{\chi }_{1,i},{\chi }_{1,j}⟩}_{{\mu }_1}.$$

Their Monte Carlo estimates from the trajectory data are given by products of the data-matrices, as

$$C_{00}\approx \frac{1}{m}{\mathbf{\chi }}_0{\mathbf{\chi }}_0^T,C_{01}\approx \frac{1}{m}{\mathbf{\chi }}_1{\mathbf{\chi }}_0^T,C_{11}\approx \frac{1}{m}{\mathbf{\chi }}_1{\mathbf{\chi }}_1^T.$$

(24)

Note that the approximations in (24) become exact if we take ${\mu }_0,{\mu }_1$ to be the empirical distributions ${\mu }_0=\frac{1}{m}{\sum }_i\delta (·-x_i)$ and ${\mu }_1=\frac{1}{m}{\sum }_i\delta (·-y_i)$, where $\delta (·)$ denotes the Dirac delta. We assume that $C_{00},C_{11}$, just as their data-based approximations in (24) are invertible. If they are not, all the occurrences of their inverses below need to be replaced by their Moore–Penrose pseudoinverses. Alternatively, one can also discard basis functions that yield redundant information, until $C_{00},C_{11}$ are invertible. Further strategies to deal with the situation where the correlation matrices are singular or ill-conditioned can be found in [18].

5.2. Projection on the Basis Functions

To find the best GMSM representable with the bases ${\chi }_0$ and ${\chi }_1$, we would like to solve (19) under the constraint that the ranges of ${\mathcal{Q}}_0$ and ${\mathcal{Q}}_1$ are in ${\mathbb{W}}_0:=\mathrm{span}({\chi }_0)$ and ${\mathbb{W}}_1:=\mathrm{span}({\chi }_1)$, respectively. To the knowledge of the authors it is unknown whether this problem has an explicitly computable solution, because it involves a non-trivial interaction of ${\mathbb{W}}_0,{\mathbb{W}}_1$ and $\mathcal{T}$.

Instead, we will proceed in two steps. First, we compute the projected transfer operator ${\mathcal{T}}_n={\mathsf{\Pi }}_1\mathcal{T}{\mathsf{\Pi }}_0$, where ${\mathsf{\Pi }}_0$ and ${\mathsf{\Pi }}_1$ are the ${\mu }_0$- and ${\mu }_1$-orthogonal projections on ${\mathbb{W}}_0$ and ${\mathbb{W}}_1$, respectively. Second, we reduce ${\mathcal{T}}_n$ to its best rank-k approximation ${\mathcal{T}}_k$ (best in the sense of density propagation).

Thus, the restriction ${\mathcal{T}}_n$ to ${\mathbb{W}}_0\to {\mathbb{W}}_1$ is simply the ${\mu }_1$-orthogonal projection of $\mathcal{T}$ on ${\mathbb{W}}_1$, giving the characterization

$${⟨{\chi }_{1,j},\mathcal{T}{\chi }_{0,i}-{\mathcal{T}}_n{\chi }_{0,i}⟩}_{{\mu }_1}=0,\forall i,j.$$

(25)

It is straightforward to compute that with respect to the bases ${\chi }_0$ and ${\chi }_1$ the matrix representation $T_n$ of ${\mathcal{T}}_n:{\mathbb{W}}_0\to {\mathbb{W}}_1$ is given by

$$T_n=C_{11}^{-1}{C}_{01},$$

(26)

see [30].

5.3. Best Low-Rank Approximation

To find the best rank-k projection of ${\mathcal{T}}_n$, let us now switch to the bases ${\tilde{\chi }}_0=C_{00}^{-1/2}{\chi }_0$ and ${\tilde{\chi }}_1=C_{11}^{-1/2}{\chi }_1$. We can switch between representations with respect the these bases by

$$f=\sum _{k=1}^n{c}_k{\chi }_{0,k}⟺f=\sum _{k=1}^n{\tilde{c}}_k{\tilde{\chi }}_{0,k},\mathrm{where}\tilde{c}=C_{00}^{1/2}c,$$

and similarly for ${\chi }_1$ and ${\tilde{\chi }}_1$. Again, a direct calculation shows that ${\tilde{\chi }}_0$ and ${\tilde{\chi }}_1$ build orthonormal bases, i.e., ${⟨{\tilde{\chi }}_{0,i},{\tilde{\chi }}_{0,j}⟩}_{{\mu }_0}={\delta }_{ij}$ and ${⟨{\tilde{\chi }}_{1,i},{\tilde{\chi }}_{1,j}⟩}_{{\mu }_1}={\delta }_{ij}$. This has the advantage, that for any operator ${\mathcal{S}}_n:{\mathbb{W}}_0\to {\mathbb{W}}_1$ having matrix representation $S_n$ with respect to the bases ${\tilde{\chi }}_0$ and ${\tilde{\chi }}_1$ we have

$$\parallel {\mathcal{S}}_n\parallel =\parallel S_n{\parallel }_2,$$

(27)

where ${\parallel ·\parallel }_2$ denotes the spectral norm of a matrix (i.e., the matrix norm induced by the Euclidean vector norm). The matrix representation of ${\mathcal{T}}_n$ in the new bases is

$${\tilde{T}}_n=C_{11}^{-1/2}{C}_{01}{C}_{00}^{-1/2}.$$

(28)

However, finding now the best rank-k approximation ${\mathcal{T}}_k$ of ${\mathcal{T}}_n$ amounts, written in these new bases, to

$$\parallel {\mathcal{T}}_n-{\mathcal{T}}_k\parallel =\parallel {\tilde{T}}_n-{\tilde{T}}_k{\parallel }_2\to \underset{\mathrm{rank}({\tilde{T}}_k)=k}{min}.$$

Again, by the Eckart–Young theorem [38] Theorem 4.4.7, the solution to this problem is given by

$${\tilde{T}}_k=\tilde{V}\tilde{\mathsf{\Sigma }}{\tilde{U}}^T,$$

(29)

where $\tilde{U},\tilde{V}\in {\mathbb{R}}^{n\times k}$ are the matrices with columns being the right and left singular vectors of ${\tilde{T}}_n$ to the largest k singular values ${\sigma }_1\ge \dots \ge {\sigma }_k$, and $\tilde{\mathsf{\Sigma }}$ is the diagonal matrix with these singular values on its diagonal. Thus, the best GMSM in terms of propagation error is given with respect to the bases ${\chi }_0$ and ${\chi }_1$ by

$$T_k=C_{11}^{-1/2}\tilde{V}\tilde{\mathsf{\Sigma }}{\tilde{U}}^T{C}_{00}^{1/2}.$$

(30)

The resulting algorithm to estimate the optimal GMSM is now identical to the time-lagged canonical correlation algorithm (TCCA) that results from VAMP and is described in [30].

Algorithm 1 TCCA algorithm to estimate a rank-k GMSM.

Choose bases ${\chi }_0$ and ${\chi }_1$. Estimate the correlation matrices $C_{00},C_{01},C_{11}$ from data, as in (24). Build the projection ${\tilde{T}}_n$ of the transfer operator with respect to the modified bases ${\tilde{\chi }}_0=C_{00}^{-1/2}{\chi }_0$ and ${\tilde{\chi }}_1=C_{11}^{-1/2}{\chi }_1$, i.e., ${\tilde{T}}_n=C_{11}^{-1/2}{C}_{01}{C}_{00}^{-1/2}$, cf. (28). Compute the k largest singular values and corresponding right and left singular vectors of ${\tilde{T}}_n$, collected into the matrices $\tilde{\mathsf{\Sigma }}$ and $\tilde{U},\tilde{V}$, respectively. The optimal rank-k GMSM has with respect to the original bases ${\chi }_0$ and ${\chi }_1$ the matrix representation $C_{11}^{-1/2}\tilde{V}\tilde{\mathsf{\Sigma }}{\tilde{U}}^T{C}_{00}^{1/2}$; cf. (30).

Remark 1 (Reversible system with equilibrium data).

If the system in consideration is reversible, the data samples its equilibrium distribution, i.e., ${\mu }_0={\mu }_1=\mu$, and also ${\chi }_0={\chi }_1$, then $C_{00}=C_{11}$, and by the self-adjointness of $\mathcal{T}$ from (6) we have $C_{01}=C_{01}^T$. Thus, ${\tilde{T}}_n$ in (28) is a symmetric matrix, and as such, its singular value and eigenvalue decompositions coincide. Hence, the construction for the best GMSM in this section (disregarding the projection on the basis functions) coincides with the one from Section 3. This is not surprising, as both give the best model in terms of propagation error.

Remark 2 (Other data-based methods).

The approximation (26) of the transfer operator has natural connections to other data-based approximation methods. It can be seen as a problem-adapted generalization of the so-called Extended Dynamic Mode Decomposition (EDMD) [41,42]. Strictly speaking, however, EDMD uses an orthogonal projection with respect to the distribution ${\mu }_0$ of the initial data $\{x_i\}$, and so approximation (32) below is equivalent to it [43]. EDMD has been shown in [33] to be strongly connected to other established analytic tools for (molecular) dynamical data, such as time-lagged independent component analysis (TICA) [44,45], blind source separation [46], and the variational approach to conformation analysis [9].

Remark 3 (Sampling the correlation matrices).

Our point of view in (24) and in the discussion following it is that without further knowledge of the system the empirical distributions are the best estimates of the actual system distributions ${\mu }_0,{\mu }_1$; and they are considered to be the same for a given finite set of data. The question how to approximate the correlation matrices efficiently if we are allowed to generate arbitrary amount of simulation data is an important and hard one. The reason for this is that in high dimensions these quantities can only be accessed via (Markov chain) Monte Carlo methods, the convergence speed of which, however, suffers immensely from the rareness of transitions between metastable regions. There are different approaches to circumvent this problem, like importance sampling, where driving the system out of equilibrium accelerates sampling, and fluctuation theorems are used to determine the original quantity of interest [47,48,49,50,51,52,53].

6. Time-Homogeneous Systems and Non-Stationary Data

In this final section, we illustrate how the above methods can be used to construct a GMSM for and assess properties of a stationary system, even if the simulation data at our disposal does not sample the stationary distribution of the system. In the first example we reconstruct the equilibrium distribution of a reversible system—hence we are able to build an equilibrium GMSM. In the second example we approximate a non-reversible stationary system (i.e., detailed balance does not hold) by a (G)MSM, again from non-stationary data.

Of course, all the examples presented so far can also be computed by the data-based algorithm of Section 5.

6.1. Equilibrium MSM from Non-Equilibrium Data

When working with simulation data, we need to take into account that this data might not be in equilibrium. Then, obviously, the empirical distribution does not reflect the stationary distribution of the system. In general, any empirical statistical analysis (e.g., counting transitions between a priori known metastable states) will be biased in such a case.

Let us consider a reversible system with equilibrium distribution $\mu$, and let the available trajectory data be ${\mu }_{\mathrm{ref}}$-distributed. Then, it is natural to describe the system by its transfer operator ${\mathcal{T}}_{\mathrm{ref}}:L_{{\mu }_{\mathrm{ref}}}^2\to L_{{\mu }_{\mathrm{ref}}}^2$ with respect to the reference distribution [18,42]; given explicitly by

$${\mathcal{T}}_{\mathrm{ref}}u(x)=\frac{1}{{\mu }_{\mathrm{ref}}(x)}{\int }_{\mathbb{X}}u(y){\mu }_{\mathrm{ref}}(y)p^t(y,x)\mathrm{d}y.$$

(31)

Note that ${\mu }_{\mathrm{corr}}:=\mu /{\mu }_{\mathrm{ref}}$ is the stationary distribution of this transfer operator, hence we can retrieve the equilibrium distribution of the system by correcting the reference distribution, $\mu ={\mu }_{\mathrm{corr}}{\mu }_{\mathrm{ref}}$.

In the data-based context, we choose the same basis ${\chi }_0={\chi }_1$ for initial and final times, since the system is time-homogeneous. In complete analogy to (26) above, the ${\mu }_{\mathrm{ref}}$-orthogonal projection of ${\mathcal{T}}_{\mathrm{ref}}:L_{{\mu }_{\mathrm{ref}}}^2\to L_{{\mu }_{\mathrm{ref}}}^2$ to ${\mathbb{V}}_0$ is given by the matrix

$$T_{\mathrm{ref},n}=C_{00}^{-1}{C}_{01}.$$

(32)

We will now apply this procedure to the double-well system from Section 3.3 with initial points $x_1,\dots ,x_m$ distributed as shown in Figure 6 (gray histogram). We chose the number of points to be $m={10}^5$, the basis functions ${\chi }_{0,i}$ to be indicator functions of subintervals of an equipartition of the interval $[-2,2]$ into $n=100$ subintervals, and the lag time $\tau =10$. In a preprocessing step we discard all basis functions that do not have any of the points $x_i$ in their support, thus obtaining a non-singular $C_{00}$, and use the remaining 77 to compute $T_{n,\mathrm{ref}}\in {\mathbb{R}}^{77\times 77}$.

Figure 6. The empirical initial distribution of the simulation data, i.e., the reference distribution ${\mu }_{\mathrm{ref}}$ (gray histogram), and the corrected equilibrium distribution computed from this data (solid black line). Left: sample size $m={10}^5$, right: sample size $m={10}^4$.

We obtain ${\lambda }_2=0.894$ giving a time scale $t_2=89.6$, and the corrected equilibrium distribution—$\mu ={\mu }_{\mathrm{corr}}{\mu }_{\mathrm{ref}}$, where ${\mu }_{\mathrm{corr}}$ is the right eigenvector of $T_{\mathrm{ref},n}$ at eigenvalue 1—is shown in Figure 6 (left) by the black curve. The right-hand side of this figure shows the results of the same computations, but for a sample size $m={10}^4$. Then, we obtain an eigenvalue $0.890$ and corresponding time scale $85.9$.

It is now simple to reconstruct the approximation ${\mathcal{T}}_n$ of $\mathcal{T}$, the transfer operator with respect to the equilibrium density. Let $D_{\mathrm{corr}}$ denote the diagonal matrix with the elements of ${\mu }_{\mathrm{corr}}$ as diagonal entries. Then, $T_n=D_{\mathrm{corr}}^{-1}{T}_{n,\mathrm{ref}}{D}_{\mathrm{corr}}$ approximates the matrix representation of ${\mathcal{T}}_n$ with respect to our basis of step functions.

Remark 4 (Koopman reweighting).

One can make use of the knowledge that the system that one estimates is reversible, even though due to the finite sample size m this is not necessarily valid for $T_{\mathrm{ref},n}$. In [18], the authors add for each sample pair $(x_i,y_i)$ also the pair $(x_{i+m}=y_i,y_{i+m}=x_i)$ to the sample set, thus numerically forcing the estimate to be reversible. In practice, one defines the diagonal matrix $\mathit{W}$ with diagonal ${\mathit{\chi }}^T{\mu }_{\mathit{corr}}$, builds the reweighted correlation matrices ${\overline{C}}_{00}=\frac{1}{2}({\mathit{\chi }}_0\mathit{W}{\mathit{\chi }}_0^T+{\mathit{\chi }}_1\mathit{W}{\mathit{\chi }}_1^T)$ and ${\overline{C}}_{01}=\frac{1}{2}({\mathit{\chi }}_1\mathit{W}{\mathit{\chi }}_0^T+{\mathit{\chi }}_0\mathit{W}{\mathit{\chi }}_1^T)$, and uses them instead of $C_{00},C_{01}$.

6.2. A Non-Reversible System with Non-Stationary Data

Reversible dynamics gives rise to self-adjoint transfer operators, and their theory of Markov state modeling is well developed. However, transfer operators of non-reversible systems are not self-adjoint, hence their spectrum is in general not purely real-valued. Thus, the definition of time scales, and in general the approximation by GMSMs is not fully evolved. Complex eigenvalues indicate cyclic behavior of the process. As this topic is beyond the scope of this paper, we refer the reader to [20,54,55] and to [25,56] for Markov state modeling with cycles.

We will consider a non-reversible system here, and show that restricting its behavior to the dominant singular modes of its transfer operator is able to reproduce its dominant long-time behavior, and even allows for a good, few-state MSM. Note that the best rank-k GMSM (18) maps to the k-dimensional subspace ${\mathbb{V}}_1$ of left singular vectors, thus its eigenvectors also fall into this subspace.

The system in consideration consists of two driving “forces”, one is a reversible part $F_r(x)=-\nabla W(x)$ coming from the potential

$$W(x)=cos(7\phi )+10{(r-1)}^2,\mathrm{where}x=\left(\begin{array}{c}rcos(\phi )\\ rsin(\phi )\end{array}\right),$$

and the other is a circular driving given by

$$F_c(x)={\mathrm{e}}^{-\beta W(x)}\left(\begin{array}{cc}0& 1\\ -1& 0\end{array}\right)x,$$

where $\beta =2$ is the inverse temperature, as in (1). The dynamics now is governed by the SDE $\mathrm{d}{x}_t=(F_r+F_c)(x_t)\mathrm{d}t+\sqrt{2{\beta }^{-1}}\mathrm{d}{w}_t$. It is a diffusion in a 7-well potential (the wells are positioned uniformly on the unit circle) with an additional clockwise driving that is strongest along the unit circle and decreases exponentially in the radial distance from this circle.

For our data-based analysis we simulate a trajectory of this system of length 500 and sample it every 0.01 time instances to obtain an initial set of $5\times {10}^4$ points. Every point herein is taken as initial condition of 100 independent simulations of the SDE for lag time $\tau =1$, thus obtaining $5\times {10}^6$ point pairs $(x_i,y_i)$.

We observe in Figure 7 (left) that the empirical distribution of the $x_i$ did not yet converge to the invariant distribution of the system, which would populate every well evenly.

Figure 7. Left: empirical distribution of the $x_i$ (histogram with $40\times 40$ bins). Middle: corrected invariant distribution. Right: clustering of the populated partition elements based on the 7 dominant eigenfunctions of the low-rank GMSM.

To approximate the transfer operator we use $\chi ={\chi }_0={\chi }_1$ consisting of the characteristic functions of a uniform $40\times 40$ partition of $[-2,2]\times [-2,2]$, and restrict this basis set to those 683 partition elements that contain at least one $x_i$ and $y_j$. The associated projected transfer operator, $T_n$ from (26) is then used to compute ${\tilde{T}}_n$ from (28), and its singular values

$${\sigma }_1=1.000,{\sigma }_2=0.872,{\sigma }_3=0.588,\dots ,{\sigma }_7=0.383,{\sigma }_8=0.052,$$

indicating a gap after seven singular values. Thus, we assemble a rank-7 GMSM $T_k$ via (30). This GMSM maps $L_{{\mu }_0}^2$ to $L_{{\mu }_1}^2$, thus to make sense of its eigenmodes, we need to transform its range to densities with respect to ${\mu }_0$ instead of ${\mu }_1$. As a density u with respect to ${\mu }_1$ is made by $\frac{{\mu }_{1}u}{{\mu }_0}$ to a density with respect to ${\mu }_0$,

$$T_k^{\prime }=C_{00}^{-1}{C}_{11}{T}_n$$

rescales the GMSM to map $L_{{\mu }_0}^2$ to itself. Note here that since the basis functions are characteristic functions with disjoint support, the correlation matrices $C_{00},C_{11}$ are diagonal, having exactly the empirical distributions as diagonal entries—i.e., the number of data point falling into the associated partition element.

We are also interested in the system’s invariant distribution. As in Section 6.1, we can correct the reference distribution ${\mu }_{\mathrm{ref}}={\mu }_0$ by the first eigenfunction ${\mu }_{\mathrm{corr}}$ of $T_k^{\prime }$ to yield the invariant distribution $\mu ={\mu }_{\mathrm{corr}}{\mu }_{\mathrm{ref}}$, cf. Figure 7 (middle). The dominant eigenvalues of $T_k^{\prime }$ are

$$\begin{array}{cc}\hfill {\lambda }_{k,1}^{\prime }=0.998+0.000i,& {\lambda }_{k,2/3}^{\prime }=0.803\pm 0.261i,\hfill \\ \hfill {\lambda }_{k,4/5}^{\prime }=0.511\pm 0.230i,& {\lambda }_{k,6/7}^{\prime }=0.378\pm 0.077i,\hfill \end{array}$$

Note that ${\lambda }_{k,1}^{\prime }<1$. This is due to our restriction of the computation to certain partition elements, as specified above. This set of partition elements is not closed under the process dynamics, and this “leakage of probability mass” (about $0.2\%$) is reflected by the dominant eigenvalue. All eigenvalues of $T_k^{\prime }$ are within $0.5\%$ error from the dominant eigenvalues of the transfer operator $T_n^{\prime }$ with respect to the stationary distribution (projected on the same basis set, and computed with higher accuracy), which is a surprisingly good agreement.

The 8-th eigenvalue of $T_n^{\prime }$ is smaller in magnitude than $0.03$. As indicated by this spectral gap, we may obtain a few-state MSM ${\widehat{T}}_k$ here as well. To this end we need to find “metastable sets” (although in the case of this cyclically driven system the term metastability is ambiguous) on which we can project the system’s behavior. Let $v_i={(v_{i,1},\dots ,v_{i,n})}^T$ denote the i-th eigenvector of $T_k^{\prime }$. As in the reversible case, where eigenvectors are close to constant on metastable sets, we will seek also here for regions that are characterized by almost constant behavior of the eigenvectors. More precisely, if the p-th and q-th partition elements belong to the same metastable set, then we expect $v_{i,p}\approx v_{i,q}$ for $i=1,\dots ,7$. Thus, we embed the p-th partition element into ${\mathbb{C}}^7\equiv {\mathbb{R}}^{14}$ (i.e., a complex number is represented by two coordinates: its real and imaginary parts) by $p\mapsto {(v_{1,p},\dots ,v_{7,p})}^T$, and cluster the hence arising point cloud into 7 clusters by the k-means clustering algorithm. The result is shown in Figure 7 (right). Taking these sets we can assemble the MSM ${\widehat{T}}_k\in {\mathbb{R}}^{7\times 7}$ via (15). We obtain a MSM that maps a Markov state (i.e., a cluster) with probability $0.62$ to itself, with probability $0.29$ to the clockwise next cluster, and with probability $0.06$ to the second next cluster in clockwise direction. The probability to jump one cluster in the counterclockwise direction is below $0.001$. The eigenvalues of ${\widehat{T}}_k$,

$$\begin{array}{cc}\hfill {\widehat{\lambda }}_1=0.998+0.000i,& {\widehat{\lambda }}_{2/3}=0.800\pm 0.260i,\hfill \\ \hfill {\widehat{\lambda }}_{4/5}=0.507\pm 0.227i,& {\widehat{\lambda }}_{6/7}=0.374+0.076i,\hfill \end{array}$$

are also close to those of $T_n^{\prime }$ (below $1\%$ error), justifying this MSM.

Remark 5.

The k-means algorithm provides a hard clustering; i.e., every point belongs entirely to exactly one of the clusters. An automated way to find fuzzy metastable sets from a set of eigenvectors is given by the PCCA+ algorithm [12]. A fuzzy clustering assigns to each point a set of non-negative numbers adding up to 1, indicating the affiliations of that point to each cluster.