Dynamic Spatial–Temporal Self-Attention Network for Traffic Flow Prediction

Abstract

Traffic flow prediction is considered to be one of the fundamental technologies in intelligent transportation systems (ITSs) with a tremendous application prospect. Unlike traditional time series analysis tasks, the key challenge in traffic flow prediction lies in effectively modelling the highly complex and dynamic spatiotemporal dependencies within the traffic data. In recent years, researchers have proposed various methods to enhance the accuracy of traffic flow prediction, but certain issues still persist. For instance, some methods rely on specific static assumptions, failing to adequately simulate the dynamic changes in the data, thus limiting their modelling capacity. On the other hand, some approaches inadequately capture the spatiotemporal dependencies, resulting in the omission of crucial information and leading to unsatisfactory prediction outcomes. To address these challenges, this paper proposes a model called the Dynamic Spatial–Temporal Self-Attention Network (DSTSAN). Firstly, this research enhances the interaction between different dimension features in the traffic data through a feature augmentation module, thereby improving the model’s representational capacity. Subsequently, the current investigation introduces two masking matrices: one captures local spatial dependencies and the other captures global spatial dependencies, based on the spatial self-attention module. Finally, the methodology employs a temporal self-attention module to capture and integrate the dynamic temporal dependencies of traffic data. We designed experiments using historical data from the previous hour to predict traffic flow conditions in the hour ahead, and the experiments were extensively compared to the DSTSAN model, with 11 baseline methods using four real-world datasets. The results demonstrate the effectiveness and superiority of the proposed approach.

1. Introduction

With the rapid development and breakthroughs in technologies such as the Internet of Things (IoT), digital communication, artificial intelligence, and big data mining, the process of urban smartification continues to advance. Intelligent transportation systems (ITSs) [1,2,3,4,5], as an integral part of smart city development, play a crucial role in managing, analysing, optimising, and improving urban traffic conditions. Among the core technologies of ITSs, traffic flow prediction [6,7,8,9] aims to forecast the future traffic conditions for a certain period of time based on the historical traffic data collected from sensors in the road traffic network [10,11,12,13,14,15]. This prediction capability is essential for proactive traffic management, real-time route planning, congestion mitigation, and the overall improvement of urban mobility.

Different from traditional time series analysis tasks, traffic flow prediction not only requires considering temporal and spatial dependencies at multiple scales [16], but it is also susceptible to external factors such as weather conditions, unexpected events, and people’s travel habits, exhibiting unstable conditions. Moreover, traffic conditions exhibit significant periodic patterns, such as weekdays and weekends, as well as differences in traffic conditions between peak and off-peak hours. Therefore, how to predict the traffic flow situation in real time and accurately is of great significance to reduce traffic congestion, improve the urban traffic environment, and enhance the quality of life of residents.

As an early solution for traffic flow prediction, initial work employed queuing theory to simulate traffic behaviours [17,18]. However, such methods lacked sufficient consideration of the highly complex correlations present in the traffic data. Subsequently, some studies abstracted the traffic flow prediction problem as a simple time series forecasting task and attempted to combine various statistical methods in search of effective solutions [19,20]. However, most of these methods required traffic data to satisfy specific assumptions, which rendered them ineffective in addressing complex issues such as nonlinearities and dynamic spatiotemporal dependencies in the data. As a result, the prediction outcomes often fell short of expectations. Later on, machine learning-based approaches [21,22] emerged, which were capable of capturing the complex nonlinear features in the traffic data. However, these methods overly relied on manual feature engineering during the processing stage, making it challenging to effectively incorporate the dynamic spatiotemporal correlations within the data. Consequently, the modelling capabilities of these methods were limited, and the prediction results remained unsatisfactory.

In recent years, with the development of deep neural network models, some works [4,23,24,25] have proposed the use of Recurrent Neural Networks (RNNs) and their variants, such as Long Short-Term Memory (LSTM) [26] and Gated Recurrent Units (GRUs) [20,27], to capture the dynamic spatiotemporal dependencies in the traffic data, or combined the attention mechanisms [28,29] with RNN models to improve the fusion efficiency of dynamic spatiotemporal dependencies in the data. Additionally, some works [2,30,31,32] have introduced Convolutional Neural Networks (CNNs) [33] to model the spatial correlations in the traffic data. Subsequently, Graph Neural Network (GNN) models [34,35,36] have emerged as popular solutions for traffic flow prediction tasks due to their natural suitability for handling non-Euclidean data with high-dimensional spatial structures and their ability to capture complex spatiotemporal features and correlations in the traffic data, and have garnered widespread attention and research interest [2,4,37,38,39,40,41]. However, despite the fruitful achievements and the significant advancement they have brought to the field of traffic prediction, these methods still inevitably face the following issues:

(1)

Insufficient modelling of spatial dependencies in traffic flow data.

The traffic flow reflects the daily travel patterns of residents, providing insights into their commuting habits. Traffic conditions are influenced by the diverse functional areas within a city, including residential areas, industrial parks, and commercial districts. It is observed that adjacent regions with different functions, such as industrial parks and residential areas, exhibit noticeable disparities in traffic conditions. Conversely, geographically distant areas with similar functions, such as commercial districts, tend to have comparable traffic conditions. However, most existing methods primarily focus on the local spatial correlations based on the topological structure of the road network, while neglecting the inherent global spatial dependencies in traffic flow data;

(2)

Insufficient modelling of the dynamic spatial–temporal correlations in the traffic data.

Unlike the stable propagation of information in information networks, traffic flow states are highly susceptible to the influence of weather conditions and unexpected events, resulting in highly dynamic variations. Most existing research heavily relies on specific static assumptions, such as predefined static adjacency matrices, which severely restrict the modelling capacity and lead to the loss of many pieces of crucial dynamic spatiotemporal correlation information in the traffic data. Consequently, accurately capturing the true dynamic characteristics of spatiotemporal dependencies in the traffic data becomes challenging.

To solve the above problems, this paper proposes a new model called the Dynamic Spatial–Temporal Self-Attention Network (DSTSAN). The model utilises a feature augmentation module to amplify the information interaction among different dimensions of features, thereby enhancing the representation capacity of the model. Subsequently, the spatial self-attention module and two masking matrices are employed to adequately model the local and global spatial dependencies in the traffic data. Finally, a temporal self-attention module is used to capture the dynamic patterns in the data and effectively fuse them with the spatial dependency features.

In summary, the main contributions can be summarised as follows:

(1)

This research proposed a new Dynamic Spatial–Temporal Self-Attention Network to solve the problem of insufficient dynamic spatial–temporal correlation modelling in the traffic flow prediction process;

(2)

Building upon the spatial self-attention mechanism, this methodology introduces two masking matrices to model the local and global spatial correlations in the data, respectively. Furthermore, a temporal self-attention module is employed to integrate dynamic temporal dependencies, thereby achieving effective modelling of the dynamic spatiotemporal dependencies in the traffic data;

(3)

The current investigation conducted extensive comparative experiments with 11 baseline methods on four real datasets. The experimental results demonstrate the validity and superiority of this model.

This paper is organised as follows: Section 2 covers related works. Section 3 mainly explains and formally defines the relevant concepts and problems. In Section 4, this research provides a detailed description of the proposed DSTSAN model. And Section 5 conducts extensive comparative experiments between the DSTSAN model and 11 baseline methods on four publicly available real-world datasets, and the experimental results are thoroughly analysed. Additionally, ablation studies and parameter sensitivity analysis are conducted in this section. Section 6 summarises and concludes the research conducted in this paper.

2. Related Work

2.1. Graph Convolution Network

Graph Convolutional Neural Networks (GCNs) [42,43] are an extension of Convolutional Neural Networks (CNNs) applied to non-Euclidean spatial data represented by graph structures. They can be categorised into spectral domain graph convolution networks and spatial domain graph convolution networks [14,44,45,46,47,48]. The spectral domain approach extends the Convolutional Neural Network algorithm to process graph-structured data by utilising the graph Laplacian matrix, which resembles the Fourier transform in signal processing. However, computing all the eigenvalues of the graph Laplacian matrix introduces high computational complexity. To address this, ChebNet [43] replaces the convolutional kernel in spectral domain graph convolution with Chebyshev polynomials, resulting in significant efficiency improvements. Building upon ChebNet, the classical GCN [42] further simplifies the model by employing a first-order Chebyshev polynomial as an approximation of the ChebNet filter. Consequently, the classical GCN is regarded as a first-order approximation model of ChebNet.

Spatial domain graph convolution networks are defined based on the mechanism of information propagation [48]. For example, Message Passing Neural Networks (MPNNs) [49] utilise message passing functions to unify different variants in the spatial domain. In GraphSAGE [50] and Diffusion Graph Convolutional Networks (DGCs) [51], the graph convolution operation is modelled as a process of diffusion or direct information propagation from a node to its adjacent nodes with a certain probability. Graph Attention Networks (GATs) [52] employ attention mechanisms to learn the relative weights between adjacent nodes, and incorporate the attention mechanism into the information propagation process. Moreover, GAT utilises multi-head attention mechanisms to ensure stable learning.

2.2. Attention Mechanism

The attention mechanism is a method inspired by the human visual and cognitive systems, enabling models to learn automatically and selectively focus on the most crucial information in the input data for the current task, thereby enhancing the model’s performance and generalisation ability. In multitask processing, attention mechanisms exhibit excellent capability in capturing long-range dependencies and have found widespread applications in natural language processing (NLP), speech recognition (ASR), machine translation (MT), computer vision (CV), and other domains. Among recent notable attention models, GMAN [29] employs a multi-level attention mechanism to dynamically adjust the correlations between different geographical sensors in temporal prediction tasks. However, this approach requires setting up a separate model at each time step, resulting in high computational complexity. GAT [52] utilises a self-attention mechanism to compute attention weights between the central node and its adjacent nodes, capturing local spatial dependencies of the central node. Transformer [53] employs a self-attention mechanism to facilitate the propagation of temporal information, addressing the challenge of capturing long-range dependencies in multitask processing that traditional Recurrent Neural Networks (RNNs) and Convolutional Neural Networks (CNNs) struggle with. However, Transformer is computationally expensive and is not suitable for large-scale long sequence prediction tasks. Subsequently, Informer [54], based on the Transformer architecture, replaces the dot-product attention mechanism with the ProbSparse self-attention mechanism, significantly alleviating the computational cost issue in long sequence prediction tasks.

2.3. Traffic Flow Prediction

Traffic flow prediction, originally originated from the task of univariate time series forecasting, has evolved into a typical case of multivariate time series forecasting [45,55]. Early research on traffic flow prediction was mainly based on statistical models, with classical methods including Vector Autoregression (VAR) [56], Kalman filtering [19], Autoregressive Integrated Moving Average (ARIMA) [57], and so on. These methods were primarily linear and based on specific static assumptions, making them unable to handle complex and highly nonlinear traffic data. Subsequently, classical machine learning methods such as Support Vector Regression (SVR) [21] and XGBoost [22] were proposed. Although these methods could effectively capture the nonlinear features in the traffic data, they heavily relied on manually designed feature engineering, limiting the models’ feature processing capabilities. In recent years, deep neural networks have dominated the field of traffic flow prediction due to their efficient modelling capabilities for complex nonlinear features in spatial–temporal data [45]. The most common methods are based on Recurrent Neural Networks (RNNs) and Convolutional Neural Networks (CNNs). DCRNN [4] and GRUED [27] both adopt an encoder–decoder architecture. The former utilises Diffusion Graph Convolutional Networks (GCNs) [4] and random walk to describe the spatial dependencies in the traffic data, while incorporating pre-sampling techniques to capture dynamic temporal features. The latter deeply integrates the encoder–decoder architecture with the GRU model and performs traffic flow prediction through temporal modelling. DSANet [58] utilises two CNN modules to model local and global temporal dependencies, and then integrates spatial correlations using self-attention mechanisms. However, CNN-based methods often combine one-dimensional spatial–temporal convolutions with graph convolutions [2], which results in a complex model due to the need for stacking multiple layers to expand the receptive field. Graph Convolutional Neural Networks (GCNs), on the other hand, have inherent advantages in modelling high-dimensional spatiotemporal data, allowing for a better fit to the complex spatiotemporal dependencies inherent in the traffic data. STGCN [2] and ASTGCN [29] both employ graph convolution (GCN) modules and Convolutional Neural Networks (CNNs) to model spatial and temporal dependencies, respectively. Graph WaveNet [37] was the first to use two learnable embedding matrices to automatically construct adaptive graphs, enabling the efficient modelling of spatiotemporal correlations in the traffic data. FOGS [10] addresses the issue of insufficient fitting in traffic prediction by utilising first-order gradient training models. D²STGNN [11] designs a spatiotemporal decoupling framework to separate diffusion signals and intrinsic signals hidden in the traffic data, aiming for the accurate modelling of traffic data. PDFormer [13] introduces two graph mask matrices to model local and global spatial correlations, and then combines them with temporal dynamic dependencies using spatial self-attention mechanisms.

In addition to the aforementioned methods, other approaches such as differential equations [59,60], spatial–temporal graph neural point processes [61], and Dynamic Time Warping (DTW) [62] have also been used to improve traffic flow prediction and have achieved a series of remarkable results. These methods provide new insights into modelling the dynamic spatiotemporal correlations in the traffic data and have contributed to the vibrant development of the field of traffic prediction.

3. Preliminaries

In this subsection, a brief description of the important concepts is introduced and the problems are addressed, as well as a formal definition of them:

Traffic sensors are sensors deployed at various intersections on the traffic network to record and collect information such as traffic flow and speed.

The road traffic network is the traffic network formed by intersecting urban roads. Intersections are regarded as nodes in the traffic network, and urban roads connecting two nodes are regarded as edges in the traffic network.

Definition 1.

(Road Traffic Network): This model defines the road traffic network as a traffic network or a traffic graph, denoted as $G=\left(V,E,A\right)$, where $V=\{v_1,v_2,\dots ,v_N\}$ is a finite set of nodes in the graph $G$, with $\left|V\right|=N$, where $N$ is the total number of nodes in the graph $G$, corresponding to $N$ traffic sensors deployed at various locations in the traffic network. $E$ represents the set of edges in the traffic graph $G$. $\mathit{A}={\left\{a_{ij}\right\}}_{N\times N}\in \left\{0,1\right\}$ is the adjacency matrix of the traffic graph $G$. For any node $v_i,v_j\in V$ in the traffic graph $G$, if there is an edge connecting node $v_i$ and node $v_j$, then $a_{ij}=1$, otherwise $a_{ij}=0$..

Traffic signals refer to the actual traffic flow measurements recorded by all traffic sensors in a traffic network at a given time step.

Definition 2.

(Traffic Signal): The traffic signal $X_t=\{x_1^t,x_2^t,\dots ,x_N^t\}\in R^{N\times D}$, represents the measurement values of all the sensors in the traffic graph $G$ at time $t$. $D$ denotes the feature dimension of the traffic information such as vehicle speed, traffic volume, etc. $N$ represents the total number of nodes in the traffic graph $G$.

Traffic flow prediction refers to the process of predicting the traffic conditions in a road network for a future period based on the traffic data collected by sensors deployed in the network over a certain historical time interval.

Problem Definition

(Traffic Flow Prediction): Given a traffic graph $G$ and traffic signals $X^T=\{X_{\left(t-T+1\right)},···,X_t\}$ within $T$ consecutive historical time steps, t’s objective is to learn a mapping function $\mathcal{F}(·)$ that accurately predicts the traffic flow conditions of graph $G$ for the next $T^{\prime }$ consecutive time steps:

$$\left[X_{\left(t-T+1\right)},···,X_t;G\right]\stackrel{\mathcal{F}(·)}{\to }\left[X_{\left(t+1\right)},···,X_{\left(t+T^{\prime }\right)}\right]$$

(1)

4. Methodology

This research has developed a novel model called the Dynamic Spatial–Temporal Self-Attention Network (DSTSAN), which is depicted in Figure 1. The DSTSAN model comprises three main layers: the feature embedding layer, the spatial–temporal self-attention layer, and the output layer. The spatial–temporal self-attention layer is constructed by connecting $l$ self-attention modules in parallel. In the subsequent sections, this paper will provide a comprehensive description of each component in the DSTSAN model.

4.1. Embedding Layer

The embedding layer is primarily composed of a fully connected network, which converts the input data into a shared high-dimensional representation space, thereby facilitating further processing and analysis.

Firstly, the traffic signal $X^T$ is converted and the traffic network adjacency matrix $\mathit{A}$ is converted into vector representations, resulting in converted signal vectors ${\mathcal{X}}_{\mathit{s}\mathit{i}\mathit{g}\mathit{n}\mathit{a}\mathit{l}}^{\mathit{T}}\in R^{T\times N\times d}$ and ${\mathcal{X}}_{\mathit{r}\mathit{o}\mathit{a}\mathit{d}}\in R^{N\times d}$. Here, $T$ represents the historical time steps, $N$ denotes the total number of nodes in the traffic network, and $d$ represents the embedding dimension. To effectively capture the weekly periodicity (distinguishing weekdays from weekends), two embedding matrices are introduced for time periods: ${\mathit{T}}^{\mathit{D}}\in R^{N_D\times d}$ and ${\mathit{T}}^{\mathit{W}}\in R^{N_W\times d}$. $N_D$ represents the number of time slots in a day, determined by the data collection frequency of traffic sensors, and $N_W$ represents the number of days in a week ($N_W=7$). This implies that the same time slot within a day and the corresponding time slot on the same day of the week share the same time period embedding. By concatenating the periodic embedding vectors of $T$ time steps, the daily periodicity embedding vector ${\mathcal{X}}_{\mathit{D}}\in R^{N\times d}$ is obtained, and the weekly periodicity embedding vector ${\mathcal{X}}_{\mathit{W}}\in R^{N\times d}$. Finally, this method concatenates the aforementioned embedding vectors to obtain the output of the embedding layer:

$${\mathcal{X}}_{\mathit{e}\mathit{m}\mathit{b}}=\oplus \left({\mathcal{X}}_{\mathit{s}\mathit{i}\mathit{g}\mathit{n}\mathit{a}\mathit{l}}^{\mathit{T}},{\mathcal{X}}_{\mathit{r}\mathit{o}\mathit{a}\mathit{d}},{\mathcal{X}}_{\mathit{D}},{\mathcal{X}}_{\mathit{W}}\right)$$

(2)

where $\oplus (·)$ is the concatenate operation.

4.2. Spatial–Temporal Self-Attention Layer

In order to capture the intricate dynamic spatiotemporal dependencies in the traffic data, this research has devised a spatial–temporal self-attention layer, as illustrated in Figure 2. This layer employs residual connections and comprises $l$ self-attention modules that are connected in parallel. Each self-attention module consists of key components, including the feature augmentation module, spatial self-attention module, and temporal self-attention module.

4.2.1. Feature Augmentation Module

Traffic data are highly complex, and different features have varying influences on the traffic flow. In order to comprehensively model the intrinsic correlations among the features in the traffic data, inspired by the channel attention mechanism [63], this research utilises a feature augmentation module to enhance the interactions between different feature dimensions of the traffic data. This enables the model to learn the importance of different feature dimensions automatically.

Figure 3 depicts the schematic diagram of the feature augmentation module. The input data are compressed using a multilayer perceptron (MLP) with a single hidden layer. Then, a single activation process is applied to determine the importance and correlation of each feature dimension with respect to the crucial information. Subsequently, the temporal features of the traffic data are enhanced through the concatenation of two one-dimensional convolutions. This process strengthens the interaction of information in the traffic data across both spatial and temporal dimensions.

Given the embedding vector ${\mathcal{X}}_{\mathit{e}\mathit{m}\mathit{b}}\in R^{T\times N\times d}$ of the input data, the feature augmentation process is formally represented as follows:

$${\mathcal{X}}_{\mathit{M}\mathit{L}\mathit{P}}=\sigma \left(\delta \left({\mathit{W}}_1{\mathcal{X}}_{\mathit{e}\mathit{m}\mathit{b}}+{\mathit{b}}_1\right)\right)\odot {\mathcal{X}}_{\mathit{e}\mathit{m}\mathit{b}}$$

(3)

$${\mathcal{X}}_{\mathit{A}\mathit{u}\mathit{g}}=\sigma \left({\mathcal{K}}_{\mathit{k}}\star \rho \left({\mathcal{K}}_{\mathit{k}}\star {\mathcal{X}}_{\mathit{M}\mathit{L}\mathit{P}}\left(t\right)\right)\right)\odot {\mathcal{X}}_{\mathit{M}\mathit{L}\mathit{P}}\left(t\right)$$

(4)

where ⊙ denotes the Hadamard product, $\sigma \left(·\right)$ is the $sigmoid$ activation function, and $\rho \left(·\right)$ is the $ReLU$ activation function, ${\mathit{W}}_1$, ${\mathit{b}}_1$ are the learnable weight parameter matrix and bias matrix, respectively, ${\mathcal{K}}_{\mathit{k}}$ is the convolution kernel, and $\star$ denotes the convolution operation. $t$ represents the time step and $\mathit{k}$ denotes the dimensions of ${\mathcal{K}}_{\mathit{k}}$.

4.2.2. Spatial Self-Attention Module

Traffic data exhibit complex spatial correlations, encompassing both local spatial dependencies, such as the traffic conditions at a single node in the traffic network, which often affect the traffic conditions at adjacent road nodes, as well as global spatial dependencies, for example, when there is traffic congestion near one commercial district, residents may travel to another district, thereby impacting the traffic conditions near that district. In order to adequately model the intricate spatial dependencies in the traffic data, a spatial self-attention module is designed as illustrated in Figure 4. Assuming at time step $t$ that the output of the feature augmentation module is represented as ${\mathcal{X}}_{\mathit{A}\mathit{u}\mathit{g}\_\mathit{t}}\in R^{N\times d}$, the spatial self-attention module can be defined as follows:

$${Atten}_{spa}\left({\mathit{Q}}_{\mathit{S}},{\mathit{K}}_{\mathit{S}},{\mathit{V}}_{\mathit{S}}\right)=\phi \left(\mathcal{S}({\mathit{Q}}_{\mathit{S}},{\mathit{K}}_{\mathit{S}})\right)·{\mathit{V}}_{\mathit{S}}$$

(5)

where ${\mathit{Q}}_{\mathit{S}}={\mathcal{X}}_{\mathit{A}\mathit{u}\mathit{g}\_\mathit{t}}{\mathit{W}}_{\mathit{S}\_\mathit{Q}}, {\mathit{K}}_{\mathit{S}}={\mathcal{X}}_{\mathit{A}\mathit{u}\mathit{g}\_\mathit{t}}{\mathit{W}}_{\mathit{S}\_\mathit{K}}, {\mathit{V}}_{\mathit{S}}={\mathcal{X}}_{\mathit{A}\mathit{u}\mathit{g}\_\mathit{t}}{\mathit{W}}_{\mathit{S}\_\mathit{V}}$ are the Query, Key, and Value matrices in the spatial self-attention mechanism. $\mathcal{S}\left({\mathit{Q}}_{\mathit{S}},{\mathit{K}}_{\mathit{S}}\right)=\frac{{\mathit{Q}}_{\mathit{S}}·{\left({\mathit{K}}_{\mathit{S}}\right)}^{\mathrm{⏉}}}{\sqrt{d^{\prime }}}$ denotes the attention score. ${\mathit{W}}_{\mathit{S}\_\mathit{Q}},{\mathit{W}}_{\mathit{S}\_\mathit{K}},{\mathit{W}}_{\mathit{S}\_\mathit{V}}\in R^{d\times d^{\prime }}$ are the learnable parameter matrices and $d^{\prime }$ is the dimension of the Query, Key, and Value matrices. $\phi \left(·\right)$ is the $softmax$ activation function.

In self-attention mechanisms, each node’s receptive field includes all other nodes in the network. However, this situation does not align with reality. In traffic networks, the influence is often effective only between several adjacent nodes. Therefore, this research introduces a masking matrix ${\mathcal{M}}_{\mathit{h}\mathit{o}\mathit{p}}$, based on the concept of k-hop.

For the masking matrix ${\mathcal{M}}_{\mathit{h}\mathit{o}\mathit{p}}$, the weight between two nodes is set to 1 when the number of hops between them is less than a threshold $\epsilon$, and 0 otherwise. By using the masking matrix ${\mathcal{M}}_{\mathit{h}\mathit{o}\mathit{p}}$, distant nodes with weaker influence are masked, ensuring that the model focuses on the local spatial dependencies of traffic flow. Subsequently, after introducing the masking matrix ${\mathcal{M}}_{\mathit{h}\mathit{o}\mathit{p}}$, the computation of local spatial self-attention is as follows:

$$\widetilde{{Atten}_{Spa}}\left({\mathit{Q}}_{\mathit{S}},{\mathit{K}}_{\mathit{S}},{\mathit{V}}_{\mathit{S}}\right)=\phi \left(\mathcal{S}({\mathit{Q}}_{\mathit{S}},{\mathit{K}}_{\mathit{S}})\odot {\mathcal{M}}_{\mathit{h}\mathit{o}\mathit{p}}\right)·{\mathit{V}}_{\mathit{S}}$$

(6)

where ⊙ denotes the Hadamard product and $\phi \left(·\right)$ is the $softmax$ activation function.

To capture the global spatial dependencies in the traffic data, this method introduces a semantic masking matrix ${\mathcal{M}}_{\mathit{s}\mathit{e}\mathit{m}}$, based on the spatial self-attention module. Inspired by previous studies [10,13,64], this research utilises the DDTW algorithm [64] to calculate the historical traffic flow similarity for each node. Then, for each node, the top $\kappa$ semantic neighbours are selected with the highest similarity and assign a weight of 1 to the connections between the node and its semantic neighbours, and 0 otherwise. This process allows us to construct the semantic masking matrix ${\mathcal{M}}_{\mathit{s}\mathit{e}\mathit{m}}$.

The selection of the DDTW algorithm for computing node similarity in this context is motivated by the fact that the first-order derivative of time series data reflects the trends in data development over time. Compared to the irregular distribution of traffic data, the trends in data development are relatively concentrated, providing more supervisory information for traffic flow prediction [10]. Therefore, the computation of global spatial self-attention is as follows:

$$\widetilde{{Atten}_{Sem}}\left({\mathit{Q}}_{\mathit{S}},{\mathit{K}}_{\mathit{S}},{\mathit{V}}_{\mathit{S}}\right)=\phi \left(\mathcal{S}({\mathit{Q}}_{\mathit{S}},{\mathit{K}}_{\mathit{S}})\odot {\mathcal{M}}_{\mathit{s}\mathit{e}\mathit{m}}\right)·{\mathit{V}}_{\mathit{S}}$$

(7)

Further calculations from Equations (6) and (7) show that for the input data ${\mathcal{X}}_{\mathit{A}\mathit{u}\mathit{g}}$, the local spatial correlation is calculated as follows:

$${\widehat{\mathcal{Z}}}_{\mathit{s}\mathit{p}\mathit{a}}=\sum _{t-T+1}^t\phi \left(\mathcal{S}({\mathit{Q}}_{\mathit{S}},{\mathit{K}}_{\mathit{S}})\odot {\mathcal{M}}_{\mathit{h}\mathit{o}\mathit{p}}\right)·{\mathit{V}}_{\mathit{S}}$$

(8)

The global spatial correlation is calculated as follows:

$${\widehat{\mathcal{Z}}}_{\mathit{s}\mathit{e}\mathit{m}}=\sum _{t-T+1}^t\phi \left(\mathcal{S}({\mathit{Q}}_{\mathit{S}},{\mathit{K}}_{\mathit{S}})\odot {\mathcal{M}}_{\mathit{s}\mathit{e}\mathit{m}}\right)·{\mathit{V}}_{\mathit{S}}$$

(9)

To obtain the output of the spatial self-attention module, this research simply concatenates Equations (8) and (9):

$${\mathcal{Z}}_{spa}=\oplus \left({\widehat{\mathcal{Z}}}_{\mathit{s}\mathit{p}\mathit{a}},{\widehat{\mathcal{Z}}}_{\mathit{s}\mathit{e}\mathit{m}}\right)$$

(10)

Due to the adoption of the multi-head attention mechanism in the spatial self-attention module, the final output of the spatial self-attention module is obtained by concatenating the outputs of the individual attention heads:

$${\mathcal{X}}_{\mathit{s}\mathit{p}\mathit{a}}=\oplus \left({\mathcal{Z}}_{spa}^{h_1},{\mathcal{Z}}_{spa}^{h_2},\dots ,{\mathcal{Z}}_{spa}^{h_i}\right){\mathit{W}}^{\mathit{s}\mathit{p}\mathit{a}}$$

(11)

where ${\mathcal{Z}}_{spa}^{h_i}=\oplus \left({\widehat{\mathcal{Z}}}_{\mathit{s}\mathit{p}\mathit{a}}^{h_i},{\widehat{\mathcal{Z}}}_{\mathit{s}\mathit{e}\mathit{m}}^{h_i}\right)$ is the output of each spatial self-attentive head, $\oplus \left(·\right)$ is the concatenation operation, ${\mathit{W}}^{\mathit{s}\mathit{p}\mathit{a}}\in R^{{h_{spa}d}^{\prime }\times d}$ is the learnable parameter matrix, and $h_i\in [1,h_{spa}]$ is the number of heads of the spatial self-attention module. $h_{spa}$ is the upper bound of the number of heads for the spatial self-attention module, which is determined by humans.

4.2.3. Temporal Self-Attention Module

As shown in Figure 5, a temporal self-attention module is employed to capture the dynamic temporal correlations in the traffic data. The model first takes the output of the spatial self-attention module as input and integrates it with the historical temporal position encoding.

$${\mathcal{X}}_{\mathit{i}\mathit{n}\mathit{p}\mathit{u}\mathit{t}}=\oplus \left({\widehat{\mathcal{Z}}}_{\mathit{s}\mathit{p}\mathit{a}},{\widehat{\mathcal{Z}}}_{\mathit{s}\mathit{e}\mathit{m}},{\mathcal{X}}_{\mathit{p}\mathit{o}\mathit{s}\mathit{i}\mathit{t}\mathit{i}\mathit{o}\mathit{n}}\right){\mathit{W}}^{\mathit{i}\mathit{n}}$$

(12)

Here, ${\mathit{W}}^{\mathit{i}\mathit{n}}\in R^{d\times d}$ is a learnable parameter matrix, and ${\mathcal{X}}_{\mathit{p}\mathit{o}\mathit{s}\mathit{i}\mathit{t}\mathit{i}\mathit{o}\mathit{n}}$ represents the positional encoding embedding vector of the current time step in the historical time steps. The positional encoding in this case adopts the same encoding method as the Transformer model [53].

Assuming the embedding matrix of an arbitrary node $v\in V$ in the traffic network is denoted as ${\overline{\mathcal{X}}}_v\in R^{T\times d}$, the calculation of the temporal self-attention is as follows:

$${Atten}_{tem}\left({\mathit{Q}}_{\mathit{t}},{\mathit{K}}_{\mathit{t}},{\mathit{V}}_{\mathit{t}}\right)=\phi \left(\mathcal{S}({\mathit{Q}}_t,{\mathit{K}}_t)\right)·{\mathit{V}}_{\mathit{t}}$$

(13)

where ${\mathit{Q}}_{\mathit{t}}={\overline{\mathcal{X}}}_v{\mathit{W}}_{\mathit{t}\_\mathit{Q}},{\mathit{K}}_{\mathit{t}}={\overline{\mathcal{X}}}_v{\mathit{W}}_{\mathit{t}\_\mathit{K}},{\mathit{V}}_{\mathit{t}}={\overline{\mathcal{X}}}_v{\mathit{W}}_{\mathit{t}\_\mathit{V}}$ denote the Query, Key, and Value matrices, respectively, within the temporal self-attention mechanism. $\mathcal{S}\left({\mathit{Q}}_t,{\mathit{K}}_t\right)=\frac{{\mathit{Q}}_{\mathit{t}}·{\left({\mathit{K}}_{\mathit{t}}\right)}^{\mathrm{⏉}}}{\sqrt{d^{\prime }}}$ denotes the attention score. And ${\mathit{W}}_{\mathit{t}\_\mathit{Q}},{\mathit{W}}_{\mathit{t}\_\mathit{K}},{\mathit{W}}_{\mathit{t}\_\mathit{V}}\in R^{d\times d^{″}}$ are parameter matrices that can be learned, where $d^{″}$ represents the dimension of the Query, Key, and Value matrices within the temporal self-attention module. $\phi \left(·\right)$ is the $softmax$ activation function.

Similarly to the computation process of spatial self-attention, this method can further obtain the final output of the temporal self-attention module:

$${\mathcal{Z}}_{\mathit{t}\mathit{e}\mathit{m}}=\oplus \left({\mathcal{Z}}_{tem}^{h_1},{\mathcal{Z}}_{tem}^{h_2},\dots ,{\mathcal{Z}}_{tem}^{h_{\tau }}\right){\mathit{W}}^{\mathit{t}\mathit{e}\mathit{m}}$$

(14)

where $h_{\tau }\in [1,h_{tem}]$ represents the number of heads in the temporal self-attention module, and $h_{tem}$ is the upper bound of the number of heads for the temporal self-attention module. ${\mathit{W}}^{\mathit{t}\mathit{e}\mathit{m}}\in R^{{h_{tem}d}^{″}\times d}$ is a learnable parameter matrix.

In summary, the final output of the spatial–temporal attention layer is calculated as follows:

$${\mathcal{X}}_{\mathit{o}\mathit{u}\mathit{t}\mathit{p}\mathit{u}\mathit{t}}={\mathcal{X}}_{\mathit{e}\mathit{m}\mathit{b}}+Norm\left(\oplus \left({\mathcal{Z}}_{\mathit{t}\mathit{e}\mathit{m}}^1,{\mathcal{Z}}_{\mathit{t}\mathit{e}\mathit{m}}^2,\dots ,{\mathcal{Z}}_{\mathit{t}\mathit{e}\mathit{m}}^{\mathit{l}}\right){\mathit{W}}^{\mathit{o}\mathit{u}\mathit{t}}\right)$$

(15)

where $Norm(·)$ is the layer normalisation operation, $\oplus (·)$ represents the concatenation operation, and ${\mathit{W}}^{\mathit{o}\mathit{u}\mathit{t}}\in R^{T\times N\times d}$ is the learnable parameter matrix.

4.3. Output Layer

The output layer of the model comprises two feedforward neural networks, which are augmented with residual connections and layer normalisation operations (as illustrated in Figure 6). This design enables us to project the fused hidden representations, incorporating spatial–temporal dynamic dependencies, onto the final prediction space via the output layer:

$${\widehat{\mathcal{X}}}_{\mathit{P}}=FFN\left(Norm\left({\mathcal{X}}_{\mathit{o}\mathit{u}\mathit{t}\mathit{p}\mathit{u}\mathit{t}}+FFN\left({\mathcal{X}}_{\mathit{o}\mathit{u}\mathit{t}\mathit{p}\mathit{u}\mathit{t}}\right)\right)\right)$$

(16)

where $FFN\left(\mathcal{X}\right)=\rho \left(\mathcal{X}{\mathit{W}}_{\mathit{F}\mathit{F}\mathit{N}\_1}+{\mathit{b}}_{\mathit{F}\mathit{F}\mathit{N}\_1}\right){\mathit{W}}_{\mathit{F}\mathit{F}\mathit{N}\_2}+{\mathit{b}}_{\mathit{F}\mathit{F}\mathit{N}\_2}$ is the formal definition of the feedforward neural network. $Norm(·)$ is the layer normalization operation and $\rho \left(·\right)$ is the $ReLU$ activation function.

4.4. Model Training Strategy

This research utilises Huber loss [65] as the objective function. In contrast to the MAE and MSE loss functions, the Huber loss function combines the strengths of both while being less sensitive to outliers, thus exhibiting higher robustness. This makes it more suitable for predicting spatial–temporal sequence data. The Huber loss function is defined as follows:

$${\mathcal{L}}_{\delta }\left(\mathcal{Y},{\widehat{\mathcal{Y}}}_i\right)\left\{\begin{array}{c}\frac{1}{2}{\left(\mathcal{Y}-{\widehat{\mathcal{Y}}}_i\right)}^2, if \left|\mathcal{Y}-{\widehat{\mathcal{Y}}}_i\right|\le \delta \\ \delta \left|\mathcal{Y}-{\widehat{\mathcal{Y}}}_i\right|-\frac{1}{2}{\delta }^2, if \left|\mathcal{Y}-{\widehat{\mathcal{Y}}}_i\right|>\delta \end{array}\right.$$

(17)

where $\mathcal{Y}$ denotes the ground truth, ${\widehat{\mathcal{Y}}}_i$ denotes the predicted value for the $i$th sample, and $\delta$ is a threshold used to control the smoothing range.

5. Experiments

5.1. Datasets

To validate the DSTSAN model, four real traffic datasets are selected and widely utilised in the PeMS system [66], namely PEMS03, PEMS04, PEMS07, and PEMS08. These datasets were collected by the Caltrans Performance Measurement System (PeMS) using sensors that span across major urban areas in the state of California, USA. The sensors collected data at a frequency of once every 30 s, and the collected data were aggregated every 5 min to form a data point. Each sensor accumulated 288 data points per day. The dataset includes information such as traffic speed, traffic flow, and the adjacency matrix of the traffic network.

Table 1. Statistics of datasets.

Datasets	Nodes	Edges	Samples	Time Span
PEMS03	358	547	26,208	3 months
PEMS04	307	340	16,992	2 months
PEMS07	883	866	28,224	3 months
PEMS08	170	295	17,856	2 months

presents a summary of the detailed parameters for the four datasets:

5.2. Baselines

To thoroughly validate the advantages of the DSTSAN model, this method performed comparative experiments with 11 representative baseline methods. These methods were selected based on their availability of the open-source code and are considered to be representative of the field. The selected baselines cover a range of approaches, including traditional time series prediction models, neural network-based models, Graph Neural Network-based models, and some of the more recent advanced methods.

• Vector Autoregression Model (VAR) [56]: The model first assumes a smooth historical time series state and then accomplishes the forecasting task by estimating the relationship between the time series and its lagged values;
• Autoregressive Integrated Moving Average Model (ARIMA) [19]: The model extracts the time series patterns hidden in the data by means of data autocorrelation and differencing, and then completes the prediction task through these patterns;
• Fully Connected Long Short-Term Memory Model (FC-LSTM) [67]: A Long Short-Term Memory network model with fully connected hidden units that can effectively capture dependencies in time series data;
• Diffusion Convolutional Recurrent Neural Network (DCRNN) [4]: The model integrates the features of bidirectional diffusion graph convolution and Recurrent Neural Networks to predict traffic flow accurately;
• Graph WaveNet [37]: The model adopts a hierarchical composition of Gate TCN (Temporal Convolutional Network) modules and GCN (Graph Convolutional Network) modules to model the spatial–temporal correlations in the traffic data effectively;
• Spatial–Temporal Graph Convolutional Networks (STGCNs) [2]: The model integrates Chebyshev graph convolution with 1D temporal convolution to effectively capture comprehensive spatial–temporal dependency features in the traffic data;
• Spatial–Temporal Fusion Graph Neural Networks (STFGNNs) [39]: The model leverages the fusion of spatial graphs and temporal graphs to capture hidden spatiotemporal dependencies in the traffic data effectively;
• Dynamic Spatial–Temporal Aware Graph Neural Networks (DSTAGNNs) [44]: The model incorporates dynamic spatial–temporal perception graph and improved gate convolutional modules to capture the inherent spatiotemporal dynamic features of the traffic data effectively;
• Graph Multi-Attention Network (GMAN) [29]: The multilevel attention model captures the dynamic spatiotemporal dependencies in the traffic data by stacking various attention modules, including spatial attention, temporal attention, and self-attention;
• Spatial–Temporal Graph Neural Controlled Differential Equations (STG-NCDEs) [59]: The temporal dynamics of the traffic data are modelled using the neural control differential equations, while leveraging the Graph Convolutional Networks to capture the spatial dependencies among different data points;
• Propagation Delay-aware Dynamic Long-range Transformer (PDFormer) [13]: The model utilises the self-attention mechanism to capture the dependencies in the traffic data. It introduces different graph mask matrices to model local spatial dependencies and global spatial dependencies separately. Additionally, the model incorporates a traffic delay perception feature module to address the issue of time delay in the spatial information propagation process.

5.3. Experimental Setups

The experiments were performed on a server that was equipped with two NVIDIA Tesla P100 GPUs, each having 16 GB of memory, and had a storage capacity of 256 GB. The server was running on the Ubuntu 20.04 operating system. The DSTSAN model was implemented using Pytorch 1.13.0 and Python 3.10.6.

To ensure the validity of the comparative experiments, this research adhered to the experimental settings commonly used in most baseline methods. The four datasets were divided into training, validation, and testing sets in a proportion of 6:2:2, respectively. For the prediction task, we utilised 12 historical time steps to forecast the subsequent 12 time steps of traffic flow conditions, denoted as $T=T^{\prime }=12$. As this research only considered traffic data from the datasets, the traffic signal dimension, D, was set to 1. The model was trained using the AdamW optimiser [13], with a learning rate of 0.001. The batch size for the data was set to 32, and the training process was conducted for 200 epochs. Additionally, the early stopping strategy was applied to the validation set with a patience value of 15.

The performance of model has been assessed using three metrics: Mean Absolute Error (MAE), Mean Absolute Percentage Error (MAPE), and Root Mean Squared Error (RMSE). The MAE measures the absolute difference between the predicted values and the actual values, providing an indication of the accuracy of the predictions. The RMSE quantifies the difference between the predicted value and the ground truth, primarily serving as a measure of how well the model fits the data, as well as being sensitive to outliers. The MAPE represents the percentage error between the predicted results and the actual observed values, providing insights into the relative degree of error between the predictions and the ground truth. The formulas for these metrics are as follows:

$$\left\{\begin{array}{c}\begin{array}{c}MAE\left(x,\widehat{x}\right)=\frac{1}{\mathrm{\Omega }}{\displaystyle \sum _{i\in \mathrm{\Omega }}}\left|x_i-{\widehat{x}}_i\right|\\ RMSE\left(x,\widehat{x}\right)=\sqrt{\frac{1}{\mathrm{\Omega }}{\displaystyle \sum _{i\in \mathrm{\Omega }}}{\left(x_i-{\widehat{x}}_i\right)}^2}\\ MAPE\left(x,\widehat{x}\right)=\frac{1}{\mathrm{\Omega }}{\displaystyle \sum _{i\in \mathrm{\Omega }}}\frac{\left|x_i-{\widehat{x}}_i\right|}{x_i}\end{array}\end{array}\right.$$

(18)

where $x_i$ denotes the $i$th ground truth, ${\widehat{x}}_i$ represents the $i$th predicted value, and $\mathrm{\Omega }$ represents the number of samples, which in this experiment is set to 12. Moreover, this research performed data pre-processing by removing instances with missing values and filtering out data samples with a traffic volume below 10 to mitigate the impact of noise. To ensure robustness, all experiments are repeated 10 times and calculated the average values as the final results.

5.4. Experiment Results

The experimental results comparing DSTSAN with other baseline methods on four real datasets are presented in

Table 2. Performance comparison of DSTSAN and others baseline models.

Model	PEMS03			PEMS04			PEMS07			PEMS08
	MAE	RMSE	MAPE	MAE	RMSE	MAPE	MAE	RMSE	MAPE	MAE	RMSE	MAPE
VAR	24.68	37.91	21.62%	25.35	39.13	19.67%	52.50	76.28	34.73%	24.31	35.64	16.32%
ARIMA	34.47	46.53	29.81%	32.65	43.50	25.41%	38.57	56.26	26.49%	32.30	39.77	18.74%
FC-LSTM	23.49	37.03	20.37%	24.33	39.06	19.51%	29.98	44.27	14.73%	22.79	32.30	14.80%
DCRNN	22.85	36.52	19.46%	23.61	37.57	18.86%	25.41	38.84	13.74%	18.56	28.67	12.24%
STGCN	21.40	34.83	18.91%	22.98	36.41	18.20%	25.67	38.92	13.27%	18.18	27.83	12.06%
Graph WaveNet	19.49	32.30	18.14%	21.25	33.40	16.76%	22.41	35.51	11.44%	16.03	25.26	10.54%
STFGNN	19.83	33.19	18.35%	21.42	33.67	16.20%	22.06	34.58	10.74%	16.97	26.31	11.28%
GMAN	17.77	29.07	16.46%	20.70	32.07	15.80%	21.83	34.32	10.05%	15.50	24.95	10.83%
DSTAGNN	17.81	29.23	16.33%	20.42	31.90	15.34%	21.67	34.30	9.83%	15.86	24.61	10.34%
PDFormer	16.57	27.66	15.90%	19.64	30.65	14.31%	20.38	33.56	9.27%	14.83	23.05	9.84%
STG-NCDE	17.75	28.92	16.60%	20.21	31.74	15.48%	21.42	34.04	9.71%	15.79	24.83	10.29%
DSTSAN	16.04	27.11	15.23%	19.06	29.83	14.04%	20.01	32.78	9.01%	14.36	22.44	9.29%

. The best results are highlighted in bold. Based on the analysis of Table 2, the following conclusions can be drawn:

(1)

Methods incorporating neural networks, Graph Neural Networks, and attention mechanisms generally outperform traditional time series forecasting models such as VAR and ARIMA. This is attributed to the fact that traditional time series forecasting models often rely on static assumptions, which are insufficient for capturing the highly nonlinear spatial dependencies present in the traffic data. Consequently, these traditional models tend to overlook crucial information, resulting in an unsatisfactory prediction performance;

(2)

Graph Neural Network models, such as STGNN and Graph WaveNet, exhibit substantial advantages over neural network models like FC-LSTM and DCRNN in terms of prediction performance. This is attributed to the ability of Graph Neural Network models to effectively model both local and global spatial dependencies present in the traffic data, enabling them to capture a richer set of essential information compared to the aforementioned models. Furthermore, attention-based methods outperform Graph Neural Network-based methods. This can be attributed to the superior capability of attention mechanisms in integrating dynamic spatiotemporal dependencies present in the traffic data;

(3)

It is not universally true that methods based on Graph Neural Networks or attention mechanisms are always superior to other approaches. For instance, the STG-NCDE model, which combines neural controlled differential equations with Graph Convolutional Network models, exhibits comparable performance and competitiveness. This suggests that there is diversity in the solutions for the same problem, and different approaches can yield competitive results;

(4)

This model demonstrates a superior performance compared to other baseline methods for several reasons. Firstly, a comprehensive consideration of both local and global spatial dependencies is incorporated in the spatial dimension. Additionally, in the temporal dimension, this method accounts for the multi-periodicity of traffic flow. This holistic approach to spatial and temporal factors contributes to the improved performance of this model. Secondly, a feature augmentation layer is introduced that automatically adjusts the information interaction among different features. This feature augmentation facilitates the enhancement of representation capabilities for each feature dimension, resulting in an improved overall performance. Lastly, this model utilises two spatiotemporal attention modules to effectively capture and integrate the dynamic spatiotemporal dependencies present in the traffic data. This enables the model to accurately capture the complex patterns and relationships within the data, leading to more precise predictions.

5.5. Ablation Study

To further investigate the effectiveness of each core module in the DSTSAN model, modifications were made to the core modules, and comparative experiments were conducted between the modified variants and the original DSTSAN model. Specifically, this research performed the following variant experiments:

(1)

Variant FA: removal of the feature enhancement layer;

(2)

Variant Mh: spatial self-attention module using only the ${\mathcal{M}}_{\mathit{h}\mathit{o}\mathit{p}}$ mask matrix, considering only local spatial dependencies;

(3)

Variant Ms: spatial self-attention module using only the ${\mathcal{M}}_{\mathit{s}\mathit{e}\mathit{m}}$ mask matrix, considering only global spatial dependencies;

(4)

Variant TSA: removal of the temporal self-attention module.

By comparing these variant models with the original DSTSAN model, this method can assess the performance differences and evaluate the contributions of each core module to the model’s performance.

This experiment conducted comparative experiments between DSTSAN and its variants on the PEMS04 and PEMS07 datasets, as shown in Figure 7:

(1)

The absence of any core module in the DSTSAN leads to a decrease in the model’s predictive performance, indicating that all core modules in the model capture key information to some extent;

(2)

The model exhibits the most significant decrease in predictive performance when only considering global spatial dependencies, indicating that local spatial dependencies have the greatest impact on the model’s predictive performance. Furthermore, this suggests that the traffic conditions at a road node are most influenced by the traffic conditions at its adjacent road nodes, aligning with the real-life observations;

(3)

In addition to local spatial correlations, global spatial correlations and dynamic temporal correlations are also crucial for influencing traffic flow.

5.6. Parameter Sensitivity Studies

To investigate the impact of certain parameter settings in the model on traffic flow prediction results, parameter sensitivity tests were conducted on the data embedding dimension $d$ and the number of layers for spatiotemporal self-attention $l$ on the PEMS04 dataset. For the data embedding dimension $d$, this research tested values of $\{\mathrm{4,8},\mathrm{16,32,64}\}$, and the results are shown in Figure 8a in terms of MAE and MAPE. For the number of layers for spatiotemporal self-attention $l$, this method tested values of $\{\mathrm{1,3},\mathrm{5,7},9\}$, and the results are shown in Figure 8b in terms of MAE and MAPE.

Figure 8a demonstrates that the predictive performance of the model follows a trend of initially increasing and then decreasing as the embedding dimension $d$ increases, reaching its optimum at $d=8$. This is because as the embedding dimension increases from small to large, the embedded information increases, enhancing the expressive power of the model and enabling it to capture more crucial information. However, as the embedding dimension continues to increase, the number of parameters also increases, which may lead to model overfitting and consequently result in a decline in performance. This indicates that in selecting the embedding dimension, it is necessary to balance the model’s expressive power and the risk of overfitting, thereby choosing an appropriate embedding dimension to achieve optimal predictive performance.

From Figure 8b, it can be observed that the predictive performance of the model exhibits an increasing and then decreasing trend as the number of self-attention layers increases. The model achieves its optimal predictive performance when the number of layers is $l=7$, and subsequent increases in the number of layers lead to a decline in the model’s predictive capability. This is because in the initial stage, as the number of layers $l$ increases, the model captures an increasing amount of key information. However, when the number of layers exceeds a certain threshold, the model starts to overfit, leading to a decrease in the accuracy of the predictions.

In summary, the model DSTSAN is sensitive to the data embedding dimension $d$ and the number of self-attention layers $l$.

6. Conclusions

This paper proposes a new Dynamic Spatial–Temporal Self-Attention Network model (DSTSAN) to solve the traffic flow prediction problem. Compared to current approaches that focus only on local dependencies in traffic flow data based on the topology of traffic networks, this approach, based on the spatial self-attention module, achieves effective modelling of local spatial correlations and global spatial correlations in traffic flow data through the introduction of spatial masking matrices and semantic masking matrices. As for some current approaches that focus only on spatial dependencies, this approach not only models the features such as daily periodicity and weekly periodicity that traffic flow exhibits in real life, but it also fuses the dynamic temporal dependencies in traffic flow data through the temporal self-attention module. The effectiveness of the proposed method is verified through comparison tests with 11 representative baseline methods on four real traffic flow datasets.

However, there are some limitations in this approach: (1) There are many conditions that affect the traffic flow conditions, such as weather, temporary activities, etc., for which the approach has not been able to model such features effectively. (2) The computational cost in this model is high due to the computation of multiple multi-head self-attention mechanisms, and the model is inefficiently executed when the amount of data is large. As a result, subsequent work will also focus on addressing the issues of improving model performance and execution efficiency, as well as multimodal feature fusion for traffic flow.