TGN: A Temporal Graph Network for Physics Prediction

Abstract

Long-term prediction of physical systems on irregular unstructured meshes is extremely challenging due to the spatial complexityof meshes and the dynamic changes over time; namely, spatial dependence and temporal dependence. Recently, graph-based next-step prediction models have achieved great success in the task of modeling complex high-dimensional physical systems. However, due to these models ignoring the temporal dependence, they inevitably suffer from the effects of error accumulation. To capture the spatial and temporal dependence simultaneously, we propose a temporal graph network (TGN) to predict the long-term dynamics of complex physical systems. Specifically, we introduce an Encode-Process-Decode architecture to capture spatial dependence and create low-dimensional vector representations of system states. Additionally, a temporal model is introduced to learn the dynamic changes in the low-dimensional vector representations to capture temporal dependence. Our model can capture spatiotemporal correlations within physical systems. On some complex long-term prediction tasks in fluid dynamics, such as airfoil flow and cylinder flow, the prediction error of our method is significantly lower than the competitive GNN baseline. We show accurate phase predictions even for very long prediction sequences.

1. Introduction

Mesh-based simulations are central to the modeling of complex physical systems. Since the computational cost of such simulations is very expensive, more and more research has started to use mesh-based machine learning models to accelerate the numerical simulation of physical systems such as fluid dynamics [1,2,3], structural mechanics [4,5], and quantum mechanics [6,7]. AI-based prediction of physical systems is the process of analyzing past system states and training a simulator to predict future system states. However, the prediction of physical systems has been a challenging task due to its complex temporal and spatial dependence. Temporal dependence refers to the changes in system states over time, which are characterized by periodicity and trend. Spatial dependence refers to the changes in system states that are affected by the mesh topology, which is manifested as the interaction of physical quantities between neighboring mesh nodes. This spatial dependence is related to the discretization methods (Finite Difference, Finite Volume, or Finite Element) in CFD. In the mesh shown in Figure 1, we use the explicit finite difference method to solve the one-dimensional heat transfer equation, where the state of node i at moment $n+1$ is governed by the state of node i and its neighboring nodes at moment n.

Graph neural networks are widely used for the prediction of physical systems on unstructured meshes. These methods encode simulated states into graphs and adaptively assign computations to spatial regions with strong gradients or where high accuracy is required [3,8,9,10]. However, the majority of GNN-based prediction models for physical systems are designed as next-step prediction models. This means that they forecast the next state of a physical system based on its current state. When facing long-term time series forecasting (LTSF) tasks, next-step models inevitably suffer from severe error accumulation, even accompanied by drifting phenomena, and such drifting is difficult to mitigate.

On the other hand, some current autoregressive prediction models have achieved great success in LTSF tasks, such as Recurrent Neural Networks (RNNs) [11], Transformer [12,13,14,15,16,17], or simple linear models [18]. They can capture trending and periodic information from historical time series. For example, cylinder flows show periodic changes during the vortex shedding phase; their system states also trend over time, and the next system state is influenced by the system state at the previous moment or even longer. By utilizing this temporal information, they can greatly mitigate error accumulation and better maintain phase and conserved quantities [19,20]. However, these methods only consider temporal dependence but ignore the topology of the mesh, making the changes of physical systems unconstrained by the physical space and thus failing to accurately predict system states.

In this paper, based on GNNs and linear models, we design a graph space encoder (GSE) and a time encoder (AutoLinear). The GSE leverages message passing to aggregate information but introduces several enhancements, such as employing multiple basis functions to map the messages, combining multiple headers, and utilizing multiple aggregators. The AutoLinear embeds the decomposition blocks used in Autoformer as internal operators and employs lineal models to extract periodic and trending information from time series. This design allows AutoLinear to progressively decompose intermediate results throughout the forecasting process, capturing future time information. We propose a hybrid method, named TGN, by combining GSE and AutoLinear. TGN is utilized for spatiotemporal modeling of complex physical systems on unstructured meshes. First, GSE performs multiple rounds of message passing on the input system state sequence to aggregate the local information into node representations, thus obtaining a graph sequence with spatial characteristics. The node representations of each graph in the sequence form a latent vector that represents the spatial encoding of the corresponding system state in the low-dimensional space. We then transform the graph sequence into a latent vector sequence. Subsequently, AutoLinear captures periodicity and trend in this latent vector sequence, modeling temporal dependence and predicting the next system state. Experiments on two fluid dynamics datasets confirm that TGN achieves state-of-the-art performance. The contributions are summarized as follows:

(1) We propose a new GNN architecture, graph space encoder (GSE), and demonstrate that it can capture the mesh topology to model spatial dependence.

(2) For long-term prediction, we use AutoLinear as a decomposition architecture, which breaks the preprocessing convention of series decomposition and embeds it as an internal block in a linear model. AutoLinear captures periodicity and trends in the time series to mitigate error accumulation.

(3) We model the spatiotemporal dependence of physical systems by combining GSE and AutoLinear.

(4) We evaluate our method on two fluid dynamics datasets. The results show that the accuracy of our method outperforms the state-of-the-art MeshGraphNets [3] baseline, and unlike the baseline, no strong error accumulation or drift is observed in our method.

2. Related Work

The development and execution of simulations for complex physical systems can be highly time consuming, and modeling complex physical problems using machine learning models has become an important area of research. Learnable models are useful for accelerating simulations of aerodynamics [1] or turbulence [21,22] and have achieved superior performance in tasks such as weather prediction [23] and graphical visualization [24]. Several studies have introduced physical expertise by incorporating loss terms [25] or feature normalization based on physical information [2] to enhance prediction accuracy. Most of these approaches utilize convolutional neural networks on regular meshes, which are widely employed for learning complex physical systems. However, there has been a recent surge of interest in leveraging GNNs for learnable simulators. GNNs offer the capability to simulate on meshes with irregular or adaptive resolutions. For instance, Belbute-Peres et al. [26] embedded an aerodynamic solver within a graph convolution architecture [27] to achieve super-resolution predictions. Alet et al. [28] employed graph element networks to make predictions on 2D grid domains, whereas MeshGraphNets [3] extended GNN-based predictions to complex 3D systems with thousands of nodes.

These methods can be categorized as either steady-state or next-step prediction models. However, next-step models often suffer from drift and error accumulation in the face of LTSF tasks due to the lack of information about historical time series. In contrast, sequence models can prevent error accumulation by modeling historical time series. Recently, Transformer-based solutions for LTSF tasks have proliferated and have been successfully applied to predict simple small physical systems [19]. However, when predicting complex large physical systems, graph coarsening [28,29,30] is required due to its high memory overhead, which can lead to higher errors in the first few steps of prediction [20]. Zeng et al. [18] introduced a set of simple single-layer linear models (LTSF-Linear), which significantly outperforms the Transformer-based complex LTSF models in experimental results under direct multi-step (DMS) prediction [31]. However, linear models cannot directly deal with unstructured data such as scale-varying meshes, and only take into account temporal features and ignore spatial dependence, making the predicted system dynamics unconstrained by physical space, so applying linear models to predict complex physical systems remains a challenge.

In the field of traffic prediction, much research [32,33,34,35,36,37] makes full use of spatiotemporal dependence to solve traffic prediction problems. Since these methods are based on CNNs to model spatial dependence, they are only applicable to Euclidean spaces, such as images or regular meshes, and cannot be extended to traffic networks with complex topology. In recent years, with the development of GNNs, a series of studies have extended traffic prediction to graph-structured data. Li et al. [38] proposed the DCRNN model, which introduces GNNs to model temporal dependence. Zhu et al. [39] proposed the A3T-GCN model to capture global time dynamics and spatial features. The model utilizes gated recurrent units (GRUs) [40] and graph convolution (GCN) to model the spatiotemporal dependence of traffic flow, and introduces an attention mechanism to focus on the global time information to improve prediction accuracy. Although these methods take into account spatiotemporal dependence and have achieved great success in traffic prediction tasks, they are inherently applicable to small 2D systems based on urban road networks and have limitations for complex high-dimensional physical systems.

Based on this background, this paper develops a new neural network method for the spatiotemporal dependence of physical systems that can extend spatiotemporal modeling to complex physical systems with thousands of nodes.

3. Methodology

3.1. Problem Definition and Overview

Our aim is to model and predict the dynamics of complex physical systems using deep learning algorithms. These physical systems are usually governed by a set of nonlinear partial differential equations (PDEs), and the spatiotemporal solution of the physical system can be obtained by given initial and boundary conditions. Currently, mesh-based finite element simulation is the basis of popular methods in PDE systems. It discretizes the physical space into an unstructured mesh $M=(\mathcal{V},\mathcal{E})$, where each node $i\in \mathcal{V}$ is connected by mesh edges E and $\left|\mathcal{V}\right|=N$, i.e., the unstructured mesh contains N mesh points. A mesh consists of nodes and topological relationships of nodes, which can be naturally understood as a graph. Thus, we denote the mesh M of the current physical system as a graph $G=(\mathcal{V},\mathcal{E})$, the mesh nodes become graph nodes $\mathcal{V}$, the mesh edges become bi-directional edges $\mathcal{E}$ in the graph, and utilize the graph G to describe the topology of the mesh. We fix the graph G in all time steps. At moment t, each node $i\in \mathcal{V}$ uses the physical quantity ${\mathit{q}}_{i,t}$ (velocity, density, or pressure) as its attribute. The attributes of these nodes can then be used to define the state field ${\mathit{S}}_t=\{{\mathit{q}}_{i,t}:i\in \mathcal{V}\}$ at moment t. Solving such discrete systems using traditional methods involves complex numerical integration of time and space and is usually costly. Therefore, our objective is to develop a learned simulator f that, given the mesh topology G and the state field sequence at the initial t moments, can quickly predict the state field sequence at the next T moments, as shown in Equation (1):

$$({\mathit{S}}_{t+1},\dots ,{\mathit{S}}_{t+T})=f(G;({\mathit{S}}_1,\dots ,{\mathit{S}}_t)).$$

(1)

These physical systems involve complex spatiotemporal dependence, whereas GSE excels at aggregating local information to learn the topology of the graph, and AutoLinear is able to capture periodic and trending information well from time series (Section 4.3). We fuse these two models and propose a new method, TGN, to predict the dynamics of physical systems. For each time step, we use the GSE to aggregate local information to obtain spatially characterized node representations, and summarize these node representations to create a low-dimensional vector representation for each time step (Section 3.2). These low-dimensional vectors constitute the historical time series, and we use a simple moving average kernel to decompose the spatially characterized historical time series into a trend component and a seasonal component, and focus on the seasonal pattern, alternating between linear transformations and decompositions of the intermediate results of the forecasting process, progressively refining the seasonal component and continuously accumulating the trend component. Finally, two linear models are applied to each of these two components and the results are added together to update the system state in the next step (Section 3.3). The combination of GSE and AutoLinear can effectively utilize spatiotemporal correlations to make stable and accurate predictions (Section 3.4). Figure 2 shows the visualization scheme of the TGN architecture.

3.2. Spatial Encoding

After representing the mesh as a graph G, GSE combines the topology information in G and the attributes of the nodes in ${\mathit{S}}_t$ to capture the spatial features at each time step t and represent them with a latent vector ${\mathit{z}}_t$. In the discussion on encoding spatial information, we ignore the time variable t to simplify the subsequent expression.

GSE is a graph neural network model with an Encode-Process-Decode architecture [9,41]. It initially extracts node features and edge features from the graph G using Multi-Layer Perceptrons (MLPs).

$${\mathit{v}}_i^0={\mathrm{mlp}}_v(\mathit{S}\left[i\right],{\mathit{n}}_i),{\mathit{e}}_{ij}^0={\mathrm{mlp}}_e({\mathit{p}}_{ij},\left|{\mathit{p}}_{ij}\right|),$$

(2)

where $\mathit{S}\left[i\right]$ denotes the physical quantity of node i and ${\mathit{n}}_i$ denotes the one-hot encoding of the type of node i. ${\mathit{p}}_i$ is the spatial coordinate of node i. We encode the relative position vector ${\mathit{p}}_{ij}$ and its norm $|{\mathit{p}}_{ij}|$ as the edge feature.

After that, GSE uses L GraphNet blocks to capture spatial features through the message-passing mechanism to further represent the nodes. Each GraphNet block contains an independent set of parameters, and in Figure 3, we briefly demonstrate the updating process of the nodes of the l-th GraphNet block.

For the l-th GraphNet block we use B basis functions ${\mathrm{mlp}}_{l,b}^e$ implemented by MLPs to capture different edge features

$${\mathit{e}}_{ij}^{l,b}={\mathrm{mlp}}_{l,b}^e({\mathit{e}}_{ij}^{l-1},{\mathit{v}}_i^{l-1},{\mathit{v}}_j^{l-1}),b=1,\dots ,B,$$

(3)

and then we use the concatenation operator ‖ to concatenate the results of the basis functions for updating the edge features:

$${\mathit{e}}_{ij}^l=\underset{b=1}{\overset{B}{\Vert }}{\mathit{e}}_{ij}^{l,b}.$$

(4)

We update the node features using the output of the previous block and the aggregation result of the edge features ${\mathit{e}}_{ij}^{l,b}$, where the aggregation of the edge features can be divided into three steps. First, we aggregate each set of edge features from different basis functions ${\mathrm{mlp}}_{l,b}^e$ separately. Second, based on the features of each node $i\in \mathcal{V}$ we compute the combined weight coefficient ${\mathit{W}}^{l,i}=\mathbf{\Phi }{\mathit{v}}_i^{l-1}+\mathit{b}\in {\mathbb{R}}^B$, where $\mathbf{\Phi }$ and $\mathit{b}$ denote the learnable weights and bias parameters. Third, we weight the aggregation results of each set of edge features:

$${\widehat{\mathit{v}}}_i^l=\sum _{b=1}^B{W}_b^{l,i}\sum _{j\in \mathcal{N}\left(i\right)}{\mathit{e}}_{ij}^{l,b}.$$

(5)

Here $\mathcal{N}\left(i\right)$ denotes all neighbors of node i. Inspired by the work of Transformers and Corso et al. [42], we add H heads and combine multiple aggregators $\oplus \in \mathcal{A}$ ($\mathcal{A}$ denotes a set of different aggregators such as mean, sum, min, or max). We apply different weighting coefficients to each header and aggregator and extend Equation (5) as

$${\widehat{\mathit{v}}}_i^l=\underset{h=1}{\overset{H}{\Vert }}\sum _{\oplus \in \mathcal{A}}\sum _{b=1}^B{w}_{h,\oplus ,b}^{l,i}\underset{j\in \mathcal{N}\left(i\right)}{\oplus }{\mathbf{e}}_{ij}^{l,b}.$$

(6)

We then combine the aggregation result ${\widehat{\mathit{v}}}_i^l$ of the edge features with the node features and encode them via MLPs to update the node features:

$${\mathit{v}}_i^l={\mathrm{mlp}}_l^v({\mathit{v}}_i^{l-1},{\widehat{\mathit{v}}}_i^l).$$

(7)

After L GraphNet blocks updating the graph G, we obtain a new set of node representations $V=(v_i^L:i\in \mathcal{V})$. Finally, from the new node representations V, GSE uses an MLP to decode each node’s spatial features

$${\mathit{h}}_i={\mathrm{mlp}}_s\left(v_i^L\right),i\in \mathcal{V}.$$

(8)

We combine these vectors to form a latent vector $\mathit{z}=\mathrm{concat}({\mathit{h}}_i:i\in \mathcal{V})$ as a low-dimensional representation of the spatial features of the whole graph. We collectively refer to the above steps for extracting the spatial features of the graph as $\mathit{z}=\mathrm{GSE}(G,\mathit{S})$. The GSE encodes each time step t independently, and it models the spatial dependence of graphs to obtain a sequence of latent vectors $({\mathit{z}}_1,{\mathit{z}}_2,\dots ,{\mathit{z}}_t)$, which are used as inputs to the temporal model.

3.3. Temporal Encoding and State Updating

Prediction of physical system dynamics is the process of predicting the most probable future sequence of length O given a past historical time series of length I. LTSF is the prediction of a larger O, which is a long-standing task and currently a major challenge in the field of physical system prediction. To address this problem, we introduce the decomposition schemes used in Autoformer and FEDformer, and with reference to the architecture of Autoformer, we propose Autolinear, a time encoder based on linear models. Briefly, Autolinear is the combination of linear layers with the decomposition blocks used in Autoformer. The decomposition blocks smooth out cyclical fluctuations and highlight long-term trends by adjusting the moving average kernel, which decomposes the input historical time series ${\mathcal{X}}_{en}=({\mathit{z}}_1,{\mathit{z}}_2,\dots ,{\mathit{z}}_t)$ into a seasonal component and a trend component

$${\mathcal{C}}^0,{\mathcal{T}}^0=\mathrm{SeriesDecomp}\left({\mathcal{X}}_{en}\right).$$

(9)

These two components, respectively, reflect the periodicity and the long-term trend of the input series.

However, for long-term forecasting, decomposing the input sequence alone is not sufficient as it does not provide information about future time steps. To address this problem, we fuse decomposition blocks and linear models to construct a series of Linear-Decomp blocks. Linear-Decomp blocks capture the periodicity and trend of the intermediate hidden series during the forecasting period, which allows it to extract the stable trend component and remove the disturbing information from the seasonal component. Autolinear uses L Linear-Decomp blocks to alternatively linearly transform and decompose the seasonal components. The equation for the l-th Linear-Decomp block is:

$${\mathcal{C}}^l,{\mathcal{T}}^l=\mathrm{SeriesDecomp}(\mathrm{Linear}\left({\mathcal{C}}^{l-1}\right)+{\mathcal{C}}^{l-1}),{\mathcal{T}}_f^l={\mathcal{T}}_f^{l-1}+{\mathcal{T}}^l,$$

(10)

where ${\mathcal{T}}_f^0={\mathcal{T}}^0$ is for accumulation. $\mathrm{Linear}(·)$ is divided into three steps. First $\mathrm{Linear}$ encodes the features of the bracketed sequences into high dimensions with an MLP; then it linearly combines the sequences in the high dimensional space; and finally reduces to low dimensions with an MLP.

We use two single-level linear models ${\mathit{W}}_{\mathcal{C}}$, ${\mathit{W}}_{\mathcal{T}}\in {\mathbb{R}}^t$ on the time axis to perform a weighted sum operation on the seasonal component ${\mathcal{C}}^L$ and the trend component ${\mathcal{T}}_f^L$ of the last Linear-Decomp block, respectively. Finally, we use an MLP to transform the sum of the two results into output features

$${\mathit{P}}_t={\mathrm{mlp}}_{de}({\mathit{W}}_{\mathcal{C}}{\mathcal{C}}^L+{\mathit{W}}_{\mathcal{T}}{\mathcal{T}}_f^L).$$

(11)

We collectively refer to the process of extracting output features from the input sequence as ${\mathit{P}}_t=\mathrm{AutoLinear}({\mathcal{X}}_{en}$).

For velocity and density, we interpret the output feature ${\mathit{P}}_t$ as the derivative of the state field ${\mathit{S}}_t$ and integrate it using $\Delta t=1$ to compute the next state field ${\tilde{\mathit{S}}}_{t+1}={\mathit{S}}_t+{\mathit{P}}_t$. The feature ${\mathit{P}}_t$ are also used to directly predict the pressure. More information about the feature ${\mathit{P}}_t$ in different systems can be found in Appendix A.1.

3.4. Temporal Graph Network

In order to capture both spatial and temporal dependences of physical systems, we propose a temporal graph network model (TGN) by fusing GSE and AutoLinear. The TGN model predicts the state field sequence after time step t in an autoregressive manner:

$${\tilde{\mathit{S}}}_{t+1}=\mathrm{TGN}(G,({\mathit{S}}_1,\dots ,{\mathit{S}}_t)),\dots ,{\tilde{\mathit{S}}}_{t+T}=TGN(G,({\mathit{S}}_1,\dots ,{\mathit{S}}_t,{\tilde{\mathit{S}}}_{t+1},\dots ,{\tilde{\mathit{S}}}_{t+T-1})).$$

(12)

During training, for each predicted time step k, we use the error between the predicted state ${\tilde{\mathit{S}}}_k$ and the corresponding true state ${\mathit{S}}_k$ to supervise the TGN model in generating the predicted state ${\tilde{\mathit{S}}}_k$ for time step k. The loss function of the T-GCN model is shown in Equation (13).

$${\mathcal{L}}_k={\Vert {\mathit{S}}_k-{\tilde{\mathit{S}}}_k\Vert }_2^2.$$

(13)

Compared with next-step models such as MeshGraphNets, our model not only utilizes graph neural networks to learn the topology of the mesh to capture spatial dependence, but also captures trending and periodicity information from historical time series. Modeling from both spatial and temporal perspectives, our model greatly mitigates the error accumulation and yields stable and accurate predictions.

4. Experiments

4.1. Experimental Settings

Dataset. We evaluate our method on two datasets from fluid dynamics systems, including the CylinderFlow dataset and the AirFoil dataset. The CylinderFlow dataset simulates water flow around cylinders with different Reynolds numbers for incompressible fluid experiments. The AirFoil dataset simulates aerodynamics around airfoil cross-sections for compressible fluid experiments. See Appendix A.1 for more information on these two datasets.

Evaluation indicator. To quantitatively evaluate the performance of the model, we calculate the relative mean square error (RMSE), defined as $\mathrm{RMSE}(\tilde{\mathit{S}},\mathit{S})=\frac{\sum {({\tilde{\mathit{S}}}_i-{\mathit{S}}_i)}^2}{\sum {\left({\tilde{\mathit{S}}}_i\right)}^2}$, where ${\mathit{S}}_i$ and ${\tilde{\mathit{S}}}_i$, respectively, denote the true and predicted flow field states at step i.

Methods. We compare our proposed method with the state-of-the-art baseline MeshGraphNets, which is the next step in predictive modeling and has achieved superior performance in mesh-based prediction tasks. In addition, we replace the time encoder (AutoLinear) in our method with a single-layer linear model (Linear) and Transformer for studying variants of the TGN model. These variants use the same spatial encoder (GSE) and number of history time steps as our method. All methods use the same training noise. Since Transformer cannot be directly applied to meshes of varying sizes, we made minor adjustments to the data structure of the historical time series. See Appendix A.2 for details of the adjustments. See Appendix A.3 for modeling details.

4.2. Experimental Results

Qualitative results. We tested our model in two fluid systems, cylinder flow and airfoil flow. In Figure 4, we apply the proposed method to predict the velocity and pressure on the trajectories of the test set for the two systems and compare them with the ground truth (CFD). For cylindrical flow, our model can capture periodic and trending information from historical time series at different Reynolds numbers to accurately predict laminar flow and Karman vortex street behavior in trajectories. In the high-speed airfoil flow, our model can successfully learn the changing flow field at different Mach numbers and angles of attack, and accurately predict its flow separation phenomena as well as vortex-shedding behavior. In both systems, cylinder flow and airfoil flow, our model accurately captures the frequency and phase of vortex shedding and visually presents predictions that are largely consistent with the CFD reference. This long-term stable prediction demonstrates that our model can be applied to complex physical systems.

Model performance comparison. To quantitatively evaluate the performance of our model, model variants, and baseline, we calculate the relative mean square error (RMSE) on single-step rollout (1-step), 50-step rollouts, and the rollouts of the entire trajectory (500-step). In

Table 1. The average relative mean square error (RMSE) of two systems, with unit of ${10}^{-4}$. The smaller the RMSE the better. The cylinder flow has history time steps of 12, whereas the airfoil flow has history time steps of 3. The best results are highlighted in bold. Our model significantly outperforms MeshGraphNets and model variants in terms of long-term prediction for both datasets.

Rollout	Model	Cylinder Flow			Airfoil
		u	v	p	u	v	$\mathit{\rho }$	p
1-step	MeshGraphNets	0.1	0.77	923.15	0.41	15.53	0.01	0.07
	Linear	0.06	1.52	66.55	0.35	13.14	0.01	0.04
	Transformer	0.02	0.37	33.07	0.37	11.09	0.01	0.17
	TGN	0.01	0.24	20.63	0.33	10.49	0.009	0.05
50-step	MeshGraphNets	0.21	0.97	186.28	1.12	47.12	0.03	0.08
	Linear	0.99	10.47	101.85	1.15	37.01	0.04	0.08
	Transformer	0.31	0.98	10.76	1.22	44.44	0.06	0.11
	TGN	0.02	0.3	4.44	0.81	24.01	0.02	0.06
500-step	MeshGraphNets	18.36	182.01	102.8	1.63	34.86	0.03	0.05
	Linear	69.73	434.28	215.2	3.53	19.45	0.56	1.14
	Transformer	5.97	59.78	65.56	3.26	28.25	0.07	0.12
	TGN	1.26	19.95	19.08	1.38	7.99	0.02	0.03

, we compare the average prediction errors on the two systems. In most cases, our model based on spatiotemporal encoding outperforms the baseline and model variants. For both short-term forecasts (1-step, 50-step) and long-term forecasts (500-step), our model shows superior performance. For airfoil flow, due to its larger mesh size, it requires larger memory consumption, so we reduce its history time steps to three. Although this will lose part of the model performance, our model still performs the best under long-term prediction. The cylinder flow with a larger history time steps (12 time steps) validates the effectiveness of our method and the RMSE of our method is significantly lower than that of the next-step model and the model variants. Surprisingly, our model is able to maintain a stable rollout even beyond the training range, as discussed in Section 4.3.

We also found an interesting experimental result. Although the simple Linear model shows poor results on the long history time steps of the cylinder flow, it achieves competitive prediction accuracy on the airfoil flow. In particular, the Linear model outperforms the complex transformer-based method on the airfoil flow. This is mainly due to the short history time steps of airfoil flow, resulting in its time information being well captured by simple linear models. The use of complex attention mechanisms in the transformer-based method may lead to overfitting, making its prediction accuracy lower than that of the simple Linear model instead.

Figure 5 shows how error accumulation occurs in the different models. The error accumulation of our model is very slow in both systems. Especially in the cylinder flow based on long history time steps, the long-term rollout of our model has significantly lower error accumulation than the baseline and model variants. We attribute this slow error accumulation behavior to the ability of the sequential model to capture periodic and trending information in long time series, as discussed further later in this section. We also note that the Linear and Transformer models have opposite performance results on the two systems. This is due to the fact that the simple Linear model cannot handle the complex temporal information in long time steps (cylinder flow), and the complex Transformer model leads to overfitting when handling short time steps (airfoil flow). On the contrary, our model maintains superior performance for history time steps of different lengths.

4.3. Model Analysis

Role of spatial encoding. Since GSE aggregates neighborhood information on meshes through multiple rounds of message passing, we believe that GSE can capture the topology of irregular unstructured meshes. To investigate the effect of GSE, we remove GSE from TGN but retain the time encoder (AutoLinear). AutoLinear directly processes the raw data from both datasets and performs flow field predictions. In Figure 6, we compare the prediction precision of the model variants AutoLinear and TGN on the cylinder flow and the airfoil flow. We can clearly observe that AutoLinear has a large prediction error and that TGN significantly outperforms AutoLinear, indicating that GSE is necessary for modeling spatiotemporal correlations of complex physical systems.

In cylinder flow, we also visualized the nodes in the low-dimensional vector ${\mathit{z}}_t$ and showed that GSE can capture the relative positions of the mesh nodes. We first randomly selected three nodes on the mesh of the cylinder flow dataset, as shown in Figure 7 (left). To visualize the nodes after GSE encoding, we use principal component analysis (PCA) to reduce the dimensionality of the nodes in ${\mathit{z}}_t$ to two and plot the three selected nodes in Figure 7 (middle). The relative positions of the three nodes are consistent with their actual positions in the mesh, indicating that the GSE learns the topology of the mesh and that each node contains information about its relative position with respect to other nodes. We also apply PCA directly to the nodes in the state field (${\mathit{S}}_t$) and plot these nodes in Figure 7 (right). These three nodes from the state field are essentially on a straight line, and their relative positions are significantly different from the true relative positions. This poses a challenge for AutoLinear to recognize the topology of the mesh and explains why its error is much higher than that of TGN.

Role of Temporal encoding. In the process of autoregressive forecasting, noise is inevitably present in the historical time series, which can significantly affect the accuracy of forecasts and is the main reason for error accumulation. We believe that the accuracy of long-term forecasting depends on whether the model is able to capture the periodicity and trend well from the noise-filled historical time series.

To test the ability of our time encoder (AutoLinear) for long-term prediction, we use single-step prediction and add Gaussian noise with different distributions to the historical time series. For the cylinder flow, we add noise obeying the Gaussian distribution $N\in (0,{\sigma }^2)(\sigma \in (0.002,0.004,0.006,0.008,0.01,0.02,0.03))$ to the velocity. In Figure 8, (left), we show how the average prediction error of all models for the whole trajectory varies with the noise. It can be seen that our model outperforms the next-step model and model variants in terms of error and stability, regardless of the noise distribution. This indicates that our model is able to resist the interference of noise and extract the inherent periodicity and trend information from the time series. On the other hand, MeshGraphNets can maintain a stable error when the noise is small. However, its error increases dramatically as the noise increases, which explains why MeshGraphNets suffers from severe error accumulation in long-term forecasting. Since our sequence models are all based on linear transformation or attention mechanisms for prediction, the weights of the linear layer and attention can directly reveal how the sequence models work. We min-max normalize the weights of the historical time series and plot them in Figure 8 (right). As can be seen from the weight distributions of the three models, the Linear model focuses only on the last history time step, meaning that it basically relies only on the information in the most recent time step to make predictions, and is therefore sensitive to noise. The Transformer model assigns a greater weight to history time steps that are closer to the output, and is able to utilize the information in a larger number of time steps compared to the Linear model for prediction, which helps to capture periodicity and trends in the series and improves the model’s resistance to noise. TGN, in addition to focusing on the most recent time steps, also assigns a larger weight to some of the more distant time steps. We hypothesize that TGN can identify periodic phenomena in the flow process and directly query the information of the previous cycle in the prediction stage to make stable and accurate predictions.

For airfoil flow, we add Gaussian noise to both velocity and density. Even though its historical time series are short, the long-term prediction error of our model is still lower than that of the comparison model. This demonstrates the strong ability of our model to encode temporal information. A complete analysis of this can be found in Appendix A.4.

Longer rollouts. We designed an experiment to test whether our model maintains superior performance beyond the training range. We trained the model on cylinder flow trajectories of 500 time steps and evaluated its performance on trajectories of 900 steps. Figure 9 illustrates the difference in the long-term prediction ability of all models. For cylinder flow, TGN and Transformer are able to preserve velocity magnitude and phase information. On the other hand, Linear and MeshGraphNets show drift, and MeshGraphNets especially have significantly different velocity magnitudes from the real situation.

Table 2. The average relative mean square error in units of ${10}^{-4}$ for cylinder flow. We trained the model on training trajectories of 500 time steps and tested the model on test trajectories of 900 steps.

Variable	MeshGraphNet	Linear	Transformer	TGN
u	197.1	173.14	32.38	8.68
v	1069.3	2862.5	494.1	154.31
p	287.4	610.85	139.5	56.71

quantitatively compares the performance of the models when rolling out longer time steps. We find that TGN remains stable and outperforms all comparison models for rollouts outside the training range.

Key hyperparameters. We tested the model under different hyperparameters and found that our method is not very sensitive to most of the hyperparameters, such as the dimensionality of the feature vectors, the number of layers of the MLPs, and the number and type of aggregators. However, we still found a key hyperparameter that affects the performance of the model (Figure 10). The number of history time steps usually greatly affects the prediction accuracy as it determines how much we can learn from the historical data. In general, models with strong temporal information extraction capabilities should be able to obtain better results with more history time steps. We found that the model performs best given as many history time steps as possible. This is in contrast to MeshGraphNets, which leads to overfitting given additional history time steps.

5. Conclusions

In this paper, we propose a method based on spatiotemporal encoding, called TGN, which can accurately perform long-term prediction of complex physical systems. On the one hand, we utilize GSE to capture the topology of the graph to obtain the relative position information among nodes and the low-dimensional vector representation of the whole graph; on the other hand, we utilize AutoLinear to capture the periodicity and trend information of the historical time series, and thus accurately predict the dynamics of the physical system. By modeling the spatiotemporal dependence, our model can be successfully applied to the task of long-term prediction of complex physical systems. Compared with the next-step model, our model can greatly mitigate error accumulation and achieve higher prediction accuracy. Our work is not limited to the prediction of physical systems, but can also be applied to other spatiotemporal tasks, such as traffic prediction. However, our method has to use training noise to augment the training data to maintain stable rollout. The training noise is difficult to adjust and may limit the accuracy of the model. In the future, we will explore predictive models that are more accurate and do not require training noise. We sincerely hope that our research will be useful for future spatiotemporal modeling of complex systems in engineering.