AddAG-AE: Anomaly Detection in Dynamic Attributed Graph Based on Graph Attention Network and LSTM Autoencoder

Abstract

Recently, anomaly detection in dynamic networks has received increased attention due to massive network-structured data arising in many fields, such as network security, intelligent transportation systems, and computational biology. However, many existing methods in this area fail to fully leverage all available information from dynamic networks. Additionally, most of these methods are supervised or semi-supervised algorithms that require labeled data, which may not always be feasible in real-world scenarios. In this paper, we propose AddAG-AE, a general dynamic graph anomaly-detection framework that can fuse node attributes and spatiotemporal information to detect anomalies in an unsupervised manner. The framework consists of two main components. The first component is a feature extractor composed of a dual autoencoder, which captures a joint representation of both the network structure and node attributes in a latent space. The second component is an anomaly detector that combines a Long Short-Term Memory AutoEncoder (LSTM-AE) and a predictor, effectively identifying abnormal snapshots among most normal graph snapshots. Compared with baselines, experimental results show that the method proposed has broad applicability and higher robustness on three datasets with different sparsity.

1. Introduction

With the advancement of science and technology, numerous domains, such as network security, intelligent transportation systems, social media, and computational biology [1,2,3,4] are producing large amounts of network-structured data composed of many interdependent objects and time-varying components. However, in these data, there are often anomalies (some unusual patterns or behaviors that significantly deviate from most of the data) that are typically associated with network attacks in network security and network fraud in social networks. Therefore, effectively detecting anomalies in network-structured data is of great significance for mitigating potential risks, monitoring system status, and ensuring system security.

Due to the high complexity of these data, integrating information fully and reasonably from various dimensions and effectively identifying anomalies has become a significant challenge for anomaly detection. To obtain as much information as possible from the network, attributed graphs are usually adopted in contemporary research [5]. However, with the graphs’ structural and attributed features contained, graph anomaly detection (GAD) in attributed networks raises a more complex problem in non-Euclidean space. The authors in [6,7,8] detected anomalous nodes in static attributed graphs and achieved a better performance than their baselines. However, those methods, based on static attributed graphs, usually ignore the dynamic evolution of graph structures and node attributes. There are also some papers focusing on investigating anomaly detection in dynamic graphs. Some literature [9,10] has explored traditional outlier detection algorithms such as Robust Random Cut Forest (RRCF) [11], IsolationForest [12], etc., while others explore anomaly detection in dynamic attributed graphs using deep-learning approaches [13]. Graph Convolutional Network (GCN) [14], Graph Attention Network (GAT) [15], GraphSage [16] and AutoEncoder (AE) [17] are representative methods in dealing with graph data. However, the above methods neglect the impact of the temporal information of graph data. Long Short-Term Memory (LSTM) [18], Gate Recurrent Unit (GRU) [19] and a forecasting-based model are well-designed for dealing with temporal information. However, it neglects both the attribute information of the nodes and the structural information of the graph. To leverage all available information in the network effectively, attempts have been made [20,21,22] to combine both types of techniques in anomaly detection for specific application domains. These efforts have demonstrated the feasibility and efficiency of detecting anomalies in graph-structured data.

Among the abovementioned methods, it is evident that each of them offers a novel algorithmic solution with high performance. However, some of these methods have not fully integrated all the features of network data, including structural, attribute, and temporal factors in dynamic networks. Additionally, certain methods rely on labeled data, which is sometimes impractical in real-world scenarios. Another significant limitation of existing works is that these methods apply to specific types of network datasets only. For instance, one method may demonstrate better performance on networks with fewer nodes and dense connections between them but experience a severe decline when applied to sparse networks. Correspondingly, a method that excels in sparse networks may exhibit worse performance when dealing with dense networks.

To alleviate the aforementioned problems, we proposed a novel framework AddAG-AE for anomaly detection in dynamic attributed networks. AddAG-AE can combine network structural, node attribute, and temporal information reasonably. The framework consists of two main components, namely the feature extractor and anomaly detector. The feature extractor includes a structure autoencoder and an attribute autoencoder, which reconstruct the original node attributes and network adjacency matrix to obtain a joint representation vector in the latent space. The anomaly detector, composed of an LSTM-AE and a forecasting-based model, detects anomalies in data with node attribute vector, adjacency matrix, and the represented vector as input. The main contributions of this work are summarized as follows:

• We propose an anomaly-detection framework, named AddAG-AE, that effectively integrates spatiotemporal, structural, and attribute information to achieve higher accuracy in anomaly detection in a self-supervised manner. It addresses the issue of integrating different types of information and improves applicability to different graph structures (sparsity levels).
• In the graph embedding phase, we design and implement a new encoding–decoding mechanism based on GAT, which makes full use of the horizontal and vertical dimensions of the latent matrix, mining the potential associations between different types of information and enhancing the effectiveness of node representations.
• In the anomaly-detection phase, a joint optimization objective is introduced to effectively integrate the reconstruction model and the prediction model. The reconstruction loss in this joint optimization objective takes into account latent vector reconstruction, graph structure reconstruction, and attribute reconstruction, effectively improving the model’s robustness and performance in anomaly detection.

We conduct extensive experiments on three different kinds of datasets, including dense-connected graphs, sparse-connected graphs, and graphs with frequent changes in connections or weights. The result shows the broader applicability and superior performance of AddAG-AE for anomaly detection in different kinds of graphs.

2. Related Work

In this section, we first review the existing methods for anomaly detection, including some typical frameworks and deep-learning-based ones, and then summarize some approaches relevant to our work.

2.1. Anomaly Detection in Dynamic Networks

In anomaly detection for dynamic networks, in order to represent the real-world dynamic network with evolving relationships between real objects and their attributes, we usually need to model them as dynamic graphs. The benefits and limitations of the existing work are shown in

Table 1. Benefits and limitations of related work.

Methods	Benefits	Limitations
Steamspot [10]	Broad adaptability to graph types, fast processing speed.	Easily affected by noise, disregard the contextual relationships
Spotlight [9]	Simple, fast and scalable	Only suitable for specific type graphs
Snapsketch [23]	Strong feature representation, real-time anomaly detection	Only suitable for specific type datasets, susceptible to experience
GmapAD [24]	Full structural and attribute information	Vulnerable to data size, high computational complexity
AddGraph [22]	A few labeled dataset, spatiotemporal information	Highly sensitive to noise
DeepSphere [25]	No manual labeling required	Not suitable for sparse graphs
Factorization [26,27,28]	High processing efficiency, capable for large-scale graph	Sensitive to network topology structures
Random Walk [29,30]	Suitable for a few samples and missing data	Not suitable for large graphs or highly heterogeneous graphs
Deep Learning [31]	Robustness on sparse networks	High computational complexity, easily affected by hyperparameters

. Typical methods, such as Steamspot [10], Spotlight [9], and Snapsketch [23] detect anomalies by mapping the graphs modeled by real network data to graph vectors in a sketching space through a special sketching method, and then classifying them using typical one-class classification algorithms. These works take full advantage of network structure features but cannot maintain and process time-varying components.

Recently, an increasing number of deep-learning-based methods have been proposed to tackle this issue. The authors in [24] proposed a novel anomaly-detection framework, named GmapAD, which fully explores the structural and attribute information within and between graphs. The framework utilizes representative nodes in the graph to map it to a new feature space and employs traditional machine-learning classifiers for anomaly detection. Although the aforementioned method extensively exploits structural and attribute information at the graph level, it overlooks the long and short-term patterns of nodes. In order to reasonably combine all possible information of dynamic graph data, the authors in [22] proposed a semi-supervised model for anomalous edge detection, named AddGraph, which employs an attention-based GRU to capture hidden information for long-term patterns and a temporal GCN to deal with graph structural information for short-term patterns of the nodes as input of GRU at current timestamp. However, this method is deployed by assuming that the training data are ideal data without any anomalous edges contained at the initial timestamps. The authors in [25] proposed an unsupervised model DeepSphere, which first incorporates hypersphere learning into LSTM-AE, and can overcome the problem that may seriously degrade the quality of neural network during training by learning the boundary between normal and abnormal data. In practice, this method performs well for relatively stable dense graphs but loses the ability to detect anomalies when applied to unstable sparse graphs. Additionally, it does not consider graph structure.

2.2. Graph Embedding Techniques and Deep Autoencoder

Graph Embedding aims at obtaining the embedding vector of a node or graph in latent space, which can sufficiently preserve valuable information about graph structural data. Corresponding techniques [32] have been widely exploited. Most can be grouped into three kinds, namely factorization-based methods, such as (i) Locally Linear Embedding (LLE) [26], Laplacian Eigenmaps [27] and Graph Factorization (GF) [28]; (ii) random walk-based methods, such as DeepWalk [29] and Node2vec [30]; and (iii) deep-learning-based methods, such as Structural Deep Network Embedding (SDNE) [31], GCN, GAT and others.

Factorization-based methods and random walk-based methods focus on the preservation of structural similarity only. Deep-learning-based methods, especially Graph Neural Network (GNN)-related techniques with their strong capability to fuse structural and content features, have attracted extensive interest recently. GCN generalizes the idea of a convolution model to non-Euclidean space so that it can process graph structural features and node content features in general graphs but not be limited to regular ones by aggregating the representations of node-selves and their one-step neighbors. Building on this idea, GAT improves the aggregating manner using a self-attention layer to learn related weights for different neighbors, using neighbor information more reasonably than GCN. Additionally, it can further enhance the capability of capturing information by applying multi-head attention.

General GNN methods are well-designed for combining all possible features of graph data, but, in dynamic graphs, temporal features reflecting the evolution process of graph structure have not been considered. Naturally, an architecture that can capture temporal information needs to be added to dynamic graph anomaly detection. In time-series anomaly detection, Recurrent Neural Networks (RNNs) [33] designed for capturing temporal information have been widely used. In order to overcome the problem of gradient vanishing and the exploding of standard RNN, the authors in [18] proposed LSTM, which successfully solved the long-term dependency problem. Combing the idea of deep autoencoding, LSTM-AE has become a reasonable choice to capture temporal information in dynamic graphs and achieve anomalous detection by reconstructing original input data. Additionally, inspired by the work of the authors in [34], we introduce a forecasting-based model detecting anomalies by predicting errors of the next timestamp as a complementary method to improve the overall performance of our model under the joint optimization strategy.

3. Problem Definition

In this section, we introduce notations and definitions for our framework. The notations used in this article are listed in

Table 2. Notations.

Notation	Description
$\mathbb{G}$	Dynamic attributed graph stream
${\nu }^t$	A set of nodes(t denotes timestamp t)
${\epsilon }^t$	A set of edges
${\chi }^t$	A set of nodes’ attribution
$G\left(t\right)$	A snapshot
N	The number of nodes
M	The number of edges
d	The dimension of the nodes’ attribution
$V^t$	A set of nodes
$A^t\in {\mathbb{R}}^{N\times N}$	An adjacency matrix
${\mathit{X}}^t\in {\mathbb{R}}^{N\times d}$	An attribute matrix (${\mathit{X}}_i^t$ is the attribute of the i-th node)
${\mathit{Z}}_i^t\in {\mathbb{R}}^{1\times d^{\prime }}$	Latent representation of node i after GAT layer
${\widehat{A}}^t\in {\mathbb{R}}^{N\times N}$	Reconstructed adjacency matrix
${\widehat{\mathit{X}}}^t\in {\mathbb{R}}^{N\times d}$	Reconstructed attribute matrix after GAT decoding
${\mathit{h}}_{en}^t\in {\mathbb{R}}^{1\times d^{″}}$	Latent representation of node i after LSTM layer
$\mathcal{L}$	Loss function
$S\left(V_i\right)$	Anomalous score of node i

.

3.1. Definition 1: Dynamic Attributed Graph

A dynamic attributed graph stream can be defined as $\mathbb{G}={\{{\nu }^t,{\epsilon }^t,{\chi }^t\}}_{t=1}^T$. Next, we set $G\left(t\right)=(V^t,A^t,{\mathit{X}}^t)$ as a snapshot in $\mathbb{G}$, where t is the timestamp, $V^t={\left\{V_i^t\right\}}_{i=1}^N$ is the node set, $V_i^t$ is the i-th node, and each row of the attribute matrix ${\mathit{X}}^t$ represents a node’s attribute vector. $A^t$ is an adjacency matrix. $A_{i,j}^t=\omega$, where $\omega$ is the weight of edge. We adopt unweighted graphs in this work, so $A_{i,j}^t\in \{0,1\}$. $A_{i,j}^t=1$, when $i=j$, because GAT-AE considers node-self’s connection in Equations (1)–(4).

3.2. Definition 2: Anomaly Detection

Given a graph steam $\left\{G\right(t\left)\right\}$, our goal is to detect anomalous snapshots (i.e., anomalous graphs) where the state of the whole system at the current timestamp differs from most other timestamps. For each node i at any timestamp t, we have learned a function $f\left(V_i\right)$ that reflects the node’s anomaly probability. Additionally, this function can serve as a score for node anomaly. If the average anomaly score of all nodes in the graph corresponding to a specific timestamp t is greater than a certain threshold, the graph is defined as an anomalous graph.

4. Proposed Method

In this section, we elaborate on the proposed AddAG-AE framework, the overview of which is illustrated in Figure 1a. AddAG-AE contains two parts, namely graph embedding and anomaly detection. The former can be viewed as a feature extractor aiming to learn node-embedding vectors fusing graph structural and node attribute information. Meanwhile, the latter is an anomaly detector aiming to detect anomalies with temporal information considered. We make corresponding improvements in two phases based on the original algorithm.

4.1. Graph Embedding Based on GAT-AE

The details of GAT-AE are shown in Figure 1b. In a dynamic attributed graph stream, each snapshot contains graph structural and node attribute feature. Therefore, we aggregate the information of a node with its neighbors in the current graph at timestamp t to obtain its latent representation ${\mathit{Z}}_i^t$ as follows,

$${\mathit{Z}}_i^t=\mathrm{Re}LU\left({\beta }_{i,i}{\mathit{X}}_i^t\mathit{W}+{\sum }_{j\in {\mathcal{N}}^t\left(i\right)}{\beta }_{i,j}{\mathit{X}}_i^t\mathit{W}\right)$$

(1)

where ${\mathit{X}}_i^t\in {\mathbb{R}}^{1\times d}$ is the node i’s attribute feature, $\mathit{W}\in {\mathbb{R}}^{d\times d^{\prime }}$ is the weight matrix ($d^{\prime }$ denotes the embedding dimension of GAT layer), whose parameters are learned by the encoder and shared among all nodes at all timestamps, ${\mathcal{N}}^t\left(i\right)=\left\{j|A_{ij}^t>0\right\}$ is the set of the node i’s neighbors, and the attention score ${\beta }_{i,j}$ can be computed as

$$e_{ij}=Leaky\mathrm{Re}\mathrm{LU}\left({\mathit{o}}^{Tr}\left({\mathit{X}}_i^t\mathit{W}\oplus {\mathit{X}}_j^t\mathit{W}\right)\right)$$

(2)

$${\beta }_{i,j}=\frac{exp\left(e_{i,j}\right)}{{\sum }_{k\in {\mathcal{N}}^t\left(i\right)\cup \left(i\right)}exp\left(e_{i,k}\right)}$$

(3)

where $\mathit{o}\in {\mathbb{R}}^{2d^{\prime }}$ is a vector of learned parameters for the attention mechanism, ⊕ denotes represents the concatenation of two vectors and $Tr$ denotes the transpose of a matrix. The attention score is computed by the $LeakyReLU$ function and normalized by the SoftMax function. After obtaining representation $\left\{{\mathit{Z}}_i^t\right\}$ of all nodes, we decode them to achieve the reconstruction of the original graph $G\left(t\right)$ by structure and attribute decoder. For graph structural decoding, we take latent representation as the structural decoder’s input and calculate the inner product between them to get the reconstructed adjacency matrix

$${\widehat{A}}^t=Sigmoid\left({\mathit{Z}}^t{\left({\mathit{Z}}^t\right)}^{Tr}\right)$$

(4)

where ${\mathit{Z}}^t\in {\mathbb{R}}^{N\times d^{\prime }}$ is a matrix that consists of all nodes of representation $\left\{{\mathit{Z}}_1^t,{\mathit{Z}}_2^t,\dots ,{\mathit{Z}}_N^t\right\}$ at time step t. ${\left({\mathit{Z}}^t\right)}^{Tr}$ denotes the transpose of ${\mathit{Z}}^t$. ${\widehat{A}}^t\in {\mathbb{R}}^{N\times N}$ denotes the reconstructed adjacency matrix. For node attribute decoding, we improve the idiomatic decoding manner [35] by employing two fully connected layers in two different directions, which take full advantage of more information from the node’s latent representation, reconstruct the attribute vector as follows

$${{\mathit{Z}}^{\prime }}^t=\mathrm{Re}LU\left({\left({\mathit{Z}}^t\right)}^{Tr}{\mathit{W}}^1+{\mathit{B}}^1\right)$$

(5)

$${\widehat{\mathit{X}}}^t={\left({{\mathit{Z}}^{\prime }}^t\right)}^{Tr}{\mathit{W}}^2+{\mathit{B}}^2$$

(6)

where ${\mathit{W}}^1\in {\mathbb{R}}^{N\times N}$ and ${\mathit{B}}^1\in {\mathbb{R}}^{d^{\prime }\times N}$ denote weights and bias in one dimension, and ${\mathit{W}}^2\in {\mathbb{R}}^{d^{\prime }\times d}$ and ${\mathit{B}}^2\in {\mathbb{R}}^{N\times d}$ denote the weights and bias in the other dimension. These parameters are learned by the decoder and shared across all timestamps. ${\widehat{\mathit{X}}}^t$ is the reconstructed attribute matrix at timestamp t. As shown above, reconstruction models, which include two parts (i.e., graph structural and node attribute reconstruction), are introduced in our model. Correspondingly, the loss of two parts ought to be taken into consideration. Therefore, the loss function of reconstruction errors can be formulated as follows:

$${\mathcal{L}}_1=\alpha {\mathcal{L}}_S+\left(1-\alpha \right){\mathcal{L}}_X$$

(7)

where $\alpha$ is a tradeoff parameter that balances the graph structure and node attribute reconstruction errors. ${\mathcal{L}}_S$ is a reconstruction loss derived from a kind of maximum likelihood estimation, which can be written as

$${\mathcal{L}}_S=\frac{1}{N}{\sum }_{i=1}^N{\sum }_{j=1}^N-\left[A_{ij}^{t}log\left({{\widehat{A}}^t}_{ij}\right)+\left(1-A_{ij}^t\right)log\left(1-{{\widehat{A}}^t}_{ij}\right)\right]$$

(8)

where N is the number of nodes, $A_{ij}^t$ is the real element value of adjacency matrix and ${\widehat{A}}_{ij}^t$ is predictive value from Equation (4). For the node attribute reconstruction, we take the mean square error between the original node attributes and reconstructed vector as the loss function as follows:

$${\mathcal{L}}_X^t=\frac{1}{N}{\sum }_{i=1}^N{∥{\mathit{X}}_i^t-{\widehat{\mathit{X}}}_i^t∥}^2$$

(9)

where ${\mathit{X}}_i^t$ and ${\widehat{\mathit{X}}}_i^t$ are node i’s attribute vector and corresponding reconstructed vector respectively. After training using the aforementioned method, we only retain the GAT encoding part (i.e., Equations (1)–(3) corresponding to the encoding layer). Next, we describe the graph embedding based on GAT using an example. For a given graph at a specific time step, we have the node attribute matrix $\mathit{X}\in {\mathbb{R}}^{N\times d}$ and the adjacency matrix $A\in {\mathbb{R}}^{N\times N}$ (the adjacency matrix is represented in the form of ${\mathcal{N}}^t$ as in Equation (1)). We input these matrices into the GAT layer, and after applying Equation (1), we obtain the embedding matrix $\mathit{Z}\in {\mathbb{R}}^{N\times d^{\prime }}$.

4.2. Anomaly Detection

After the above process of the GAT-AE, we obtain embedding vectors of all nodes in all graphs, which represent the structure and attribute features in latent space. However, until now, the time-dependency of the same node at different time steps has not been taken into account for anomaly detection. Furthermore, a node with a high anomalous score in Section 4.1 only represents the degree of deviation from other normal nodes in the same graph (the same timestamp), but not the degree of deviation from the historical feature of nodes, i.e., we are more likely to score nodes by comparing the changes in the state of node i at current timestamp t with this node historical state at timestamp $t-1,t-2,\dots$.

Thus, considering temporal information in anomaly detection, the detector includes two parts, one of which is a reconstruction-based model aiming at capturing the data distribution of the entire graph stream and the other of which is a forecasting-based model aimed at predicting the value at the next timestamp. During the process, the parameters from both models are updated simultaneously. Finally, the loss function can be defined as follows:

$${\mathcal{L}}_2={\mathcal{L}}_{rec}+{\mathcal{L}}_{for}$$

(10)

where ${\mathcal{L}}_{rec}$ and ${\mathcal{L}}_{for}$ denote the reconstruction loss and forecasting loss, respectively.

4.2.1. Reconstruction-Based Model

We adopt an LSTM-AE as a reconstruction model to reconstruct the embedding vector of the GAT-AE to find anomalous nodes in the data. It is noted that we reshape an output tensor of the GAT-AE so that the embedding vector of the same node i at different timestamp t can form a time sequence $\left[{\mathit{Z}}_i^1,{\mathit{Z}}_i^2,\dots ,{\mathit{Z}}_i^T\right]$ (named ${\mathcal{X}}_i$) into the LSTM-AE. In the process of encoding, the encoder transforms ${\mathcal{X}}_i$ into a hidden representation with a regular LSTM model, as follows:

$${\mathit{h}}_{en}^t=LST{M}_{en}({\mathit{Z}}_i^t,{\mathit{h}}_{en}^{t-1})$$

(11)

where ${\mathit{h}}_{en}^t\in {\mathbb{R}}^{1\times d^{″}}$ denotes a hidden vector of node i after processing of the LSTM encoder ($d^{″}$ denotes the embedding dimension of LSTM layer). In the process of decoding, we take the last hidden vector ${\mathit{h}}_{en}^T$ as the decoder’s initial parameter to reconstruct sequence ${\mathcal{X}}_i$,

$${\widehat{\mathit{Z}}}_i^{t-1}=LST{M}_{de}\left({\widehat{\mathit{Z}}}_i^t,{\mathit{h}}_{en}^{t-1}\right)$$

(12)

where ${\widehat{\mathit{Z}}}_i^{t-1}$ is the reconstruction vector at timestamp $t-1$. From Equation (12), we use the output of timestamp t as the input of timestamp $t-1$ in the decoding phase. Finally, the training loss function of our model is computed as follows:

$${\mathcal{L}}_{rec}=\frac{1}{T}{\sum }_{i=1}^N{{\sum }_{t=1}^T∥{\mathit{Z}}_i^t-{\widehat{\mathit{Z}}}_i^t∥}^2$$

(13)

where ${\mathit{Z}}_i^t$ and ${\widehat{\mathit{Z}}}_i^t$ are feature vector and reconstructed vector, respectively. N is the number of nodes. The number of layers in the LSTM can be set to different values. We maintain the same number of layers for both the encoding LSTM and decoding LSTM. Based on the comprehensive performance considerations, the number of layers in the LSTM encoder–decoder is set as two in this paper (relevant experiments refer to Section 5.4.3).

4.2.2. Forecasting-Based Model

First, we stack fully connected layers with hidden vector ${\mathit{h}}_{en}^t$ of LSTM-AE as input and use $ReLU$ as the nonlinear activation function, to extract features from hidden vectors that contain all spatio–temporal information for the node

$${\tilde{\mathit{Z}}}^t=f_{\mathrm{Re}LU}\left({\mathit{h}}^{t-1}\right)$$

(14)

Then, we transform $\left\{{\tilde{\mathit{Z}}}_1^t,{\tilde{\mathit{Z}}}_2^t,\dots ,{\tilde{\mathit{Z}}}_N^t\right\}$ at time step t into matrix ${\tilde{\mathit{Z}}}^t$ and then use Equations (4)–(6) to decode ${\tilde{\mathit{Z}}}^t$ to predict the next timestamp’s original attribute vector and adjacency matrix. The loss function of the predictor can be formulated as follows:

$${\mathcal{L}}_{for}=\frac{1}{T}{\sum }_{t=1}^T\left[{\sum }_{i=1}^N\left({∥{\mathit{X}}_i^t-{\widehat{\mathit{X}}}_i^t∥}^2+{\sum }_{j\in {\mathcal{N}}^t\left(i\right)\cup \left\{i\right\}}{∥A_{i,j}^t-{\widehat{A}}_{i,j}^t∥}^2\right)\right]$$

(15)

where ${\mathit{X}}_i^t$ and ${\widehat{\mathit{X}}}_i^t$ are the original attribute and predicting vector of node i using our predictor, respectively. $A_{i,j}^t$ and ${\widehat{A}}_{i,j}^t$ denote the real existing edge and prediction probability value of the edge, respectively. N is the number of nodes. t ranges from 1 to T, denoting the timestamp. Finally, we adopt a node’s reconstruct error of the LSTM and forecasting error as its anomalous score, which can be formulated as follows:

$$S^t\left(V_i\right)={∥{\mathit{Z}}_i^t-{\widehat{\mathit{Z}}}_i^t∥}^2+{∥{\mathit{X}}_i^{t+1}-{\widehat{\mathit{X}}}_i^{t+1}∥}^2+{\sum }_{j\in {\mathcal{N}}^{t+1}\left(i\right)}{∥A_{i,j}^{t+1}-{\widehat{A}}_{i,j}^{t+1}∥}^2$$

(16)

where $S^t\left(V_i\right)$ is the anomalous score of the node i at timestamp t. It demonstrates that AddAG-AE can detect anomalous nodes in dynamic graphs. However, in practical applications, when working with a graph stream, the focus often shifts to assessing whether the whole system state at a specific timestamp (i.e., the statistical properties of a graph) is anomalous or not. To address this, we can evaluate the mean value of the anomalous scores, denoted as $S\left(t\right)={\sum }_i^N{S}^t\left(V_i\right)/N$, or consider the number of nodes whose anomalous score exceeds a particular threshold, denoted as $S\left(t\right)=\left|\left\{i|S^t\left(V_i\right)>\lambda \right\}\right|$, as the evaluation metric for determining whether the current network is anomalous. We adopt the first one in this article.

5. Experiment

5.1. Datasets

In this paper, we evaluate AddAG-AE on three commonly used real-world datasets, including Enron Mail (graph whose connections frequently changing,

Table 3. Sample data of Enron mail.

Datatime	From	To
1-Jan-2000	[email protected]	[email protected]
1-Jan-2000	[email protected]	[email protected]
…	…	…
31-Dec-2002	[email protected]	[email protected]

), NYC Taxi Trips (dense graph,

Table 4. Sample data of NYC Taxi Trips.

Pickup_Datatime	Dropoff_Datatime	Pickup_Longitude	Pickup_Latitude	Dropoff_Longitude	Dropoff_Latitude
1-Oct-2015 17:24	1-Oct-2015 17:32	−73.98215485	40.76793671	−73.96463013	40.76560211
1-Oct-2015 0:43	1-Oct-2015 0:54	−73.98041534	40.73856354	−73.9994812	40.73115158
…	…	…	…	…	…
31-Jan-2016 18:14	31-Jan-2016 18:20	−74.00785065	40.72357178	−74.00765228	40.74083328

) and IDS 2017 (sparse graph,

Table 5. Sample of IDS 2017.

Datatime	Sip	Dip
5-Jul-2017 8:42	192.168.10.50	192.168.10.3
5-Jul-2017 8:43	192.168.10.19	129.6.15.29
…	…	…
5-Jul-2017 14:43	192.168.10.14	192.168.10.19

). After processing and calculation, the statistics of nodes (#v) and edges (#$ϵ$) of different datasets are shown in

Table 6. Statistics of the used real-world datasets.

Dataset	#v	#$\mathit{ϵ}$
NYC Taxi Trips	56	32,675,244
Enron Message	147	62,479
IDS 2017	1323	620,652

, where the symbol “#” represents abbreviation of “number”.

• A. NYC Taxi Trips. This is a dataset of NYC taxi trips from 2009 to 2018, including details such as time and coordinates. In the experiment, we extracted data from October 2015 to January 2016 and converted it into a chart broken down by day. NYC taxi trips are divided into 56 zones using K-means clustering, which represents the nodes of the graph, with the trips between regions representing the edges. For each node, the trips from that region are aggregated and normalized into a single attribute vector. Special days with unusual traffic patterns are flagged as anomalies for analysis.
• B. IDS 2017. The dataset was intercepted on 5 July 2017, including source IP, target IP, attack type, timestamp, and other fields, resulting in more than 640,000 edges and five attack types. After processing, the graph flow was modeled into 5-minute intervals and marked as anomalies if the graph contained at least 200 attack edges.
• C. Enron Mail dataset. The dataset consists of 352,550 emails exchanged between Enron employees from 1979 to 2002. We focused on the data since 1999 and select the top 147 individuals with the highest number of senders or recipients as nodes. The dataset was segmented into graph streams in days. Since there are no direct anomaly data, the charts were marked as anomalies if they related to a major scandal event.

5.2. Evaluation Metrics

To verify the performance of the model more comprehensively, this paper adopts the AUC score, precision, recall, and loss as evaluation metrics. Among them, the AUC score is the main evaluation metric, which has been widely used in many abnormal detection methods in the past. Specifically, the AUC score is the area under the ROC (receiver operating characteristic) curve, which implies the sorting quality of the sample forecasted by the model. ROC is the spot of the true positive rate against the false positive rate. We treat the anomalous graphs we labeled as a positive sample in statistics and sort all samples by anomalous score provided by the model. Therefore, a higher AUC score reflects the better anomaly-detection performance of the model. In addition, in order to confirm the effectiveness of the model, we added two supplementary metrics, namely recall and precision.

5.3. Experimental Section Setup

In the experiment, we implement AddAG-AE using the Adam optimizer with a learning rate of 0.001 and weight decay of 0.001 on three commonly used datasets. The dimensions of GAT-AE embedding are $2d^{\prime }$, $d^{\prime }$, and $1.5d^{\prime }$ for NYC Taxi Trips, IDS 2017, and Enron Mail dataset, respectively, where $d^{\prime }$ is its own node-embedding vector’s dimension. $\alpha$ is set to 0.1, 0.5, and 0.3 for NYC Taxi Trips, IDS 2017, and Enron Mail respectively. The number of both GAT layers and LSTM layers is set to two for all datasets.

5.4. Experimental Section Results

We compare AddAG-AE with the following six anomaly-detection methods.

5.4.1. Performance Evaluation

The anomaly-detection performance (AUC scores, precision, and recall in the testing phase) of all baseline methods and AddAG-AE on three datasets are reported in

Table 7. AddAG-AE evaluation results.

Index	Dataset	NYC Taxi Trips			IDS 2017			Enron Mail
Method		Precision	Recall	AUC	Precision	Recall	AUC	Precision	Recall	AUC
Spotlight		0.6986	0.7007	0.6757	0.7954	0.6862	0.7353	0.3427	0.4885	0.3633
LSTM-AE		0.5802	0.5734	0.5743	0.3203	0.4322	0.3036	0.4003	0.4403	0.4927
DeepSphere		0.9233	0.7424	0.9008	0.4722	0.5383	0.4443	0.6074	0.5560	0.6228
AnomalyDAE		0.8763	0.7565	0.8876	0.7032	0.5716	0.6827	0.5058	0.5845	0.5610
Dominant		0.6934	0.6577	0.6956	0.6286	0.6373	0.6367	0.6292	0.6409	$\mathbf{0.6544}$
GmapAD		0.9312	0.7532	0.9201	0.7136	0.6704	0.6803	0.6213	0.6081	0.6451
AddAG-AE		$\mathbf{0.9807}$	0.7143	$\mathbf{0.9459}$	$\mathbf{0.8160}$	$\mathbf{0.7125}$	$\mathbf{0.7521}$	$\mathbf{0.6385}$	$\mathbf{0.6551}$	0.6521

, where the numbers in bold represent the optimal results for their corresponding indexes. The related analyses and evaluations are as follows:

• Spotlight [9]: This identifies an anomalous graph by focusing on a sudden change in the localized graph structures, which is effectively for IDS 2017, because its graph structure changes drastically when a network attack occurs in a stable network. While NYC Taxi Trips has dense connections between nodes, the Spotlight’s manner of acquiring features can result in changes to the embedding vector of certain special anomalous graph structures that are not obvious. For Enron Mail, many normal graphs have similar graph vectors with anomaly graphs owing to its special sketching method, which led to its poor performance.
• LSTM-AE [36]: This displays a set of worse precision, recall, and AUC scores on all three datasets and even be worse than random guesses on IDS 2017. The main reason for this is that the method completely disregards the graph structure and the evolution of structure over time. For IDS 2017, with many nodes and sparse connections between nodes, the simple encode–decode strategy cannot generate effective node representation.
• DeepSphere [25]: This combines an LSTM-AE and hypersphere learning, and learns a compact boundary for distinct, normal, and abnormal data. For NYC Taxi Trips and Enron Mail, whose nodes number fewer, it can effectively capture the structural differences between normal and abnormal graphs. DeepSphere’s embedding manner is expanding the adjacency matrix of a graph into a high-dimensional vector simply by concatenating every line of the adjacency matrix. For IDS 2017, whose number of nodes is more than 1000, the manner will generate a more than one million-dimensional sparse vector for a graph, which may be the main reason for the poor performance.
• AnomalyDAE [7]: This has a better performance on NYC Taxi Trips and IDS 2017 while it exhibits ordinary performance on Enron Mail. It is worth noting that it achieves the best recall and the better precision on NYC Taxi Trips data. The method acquires node representation using a dual autoencoder, which is sensitive to node attribution and graph structure. Thus, it has generally better performance on different datasets, especially on dense graphs.
• Dominant [37]: This is stable on the three datasets, which ignores the dynamic evolution of the same node over time. It adopts the single graph encoder while neglecting complex interactions between the graph structure and node attribute. It is maybe suitable for Enron Mail, and achieves the best AUC score.
• GmapAD [24]: GmapAD exhibits a stable and better precision, recall, and AUC score on the three datasets. It combines cleverly multiple algorithms such as graph neural networks, differential evolution, and graph mapping and leading to a significant recognition effect on anomalous graphs. It performs better on dense graphs (NYC Taxi Trips data) and slightly worse on sparse graphs.
• AddAG-AE: This shows the best AUC score on NYC Taxi Trips and IDS 2017, and has a better performance on Enron Mail. In addition, it outperforms all baselines in the precision of all three datasets, and has the best recall on IDS 2017 and Enron Mail. We improved the GAT decoding process and enabled the model to capture a richer set of structure and attribute information. As a result, it produces more effective node representations, which can potentially increase precision and AUC scores. Then, based on the traditional LSTM-AE, we design and incorporate a novel prediction module to aid in anomaly detection, which ensures the stability of the model. On the NYC Taxi Trips dataset, in terms of recall, the best method is not AddAG-AE but AnomalyDAE. The reason for this may be that in the GAT decoding stage, AnomalyDAE jointly considers hidden vectors generated by GAT and the embedded vectors of attribute vectors to reconstruct the attribute vectors, while AddAG-AE ignores this point. For Enron Mail, whose graph structure is from email exchange statistics, more normal graphs are similar to the anomaly graph, which leads to AUC scores declining slightly.

The experimental results are illustrated in Figure 2, with the loss graph during the training phase for three datasets depicted in Figure 2a and the AUC score comparison of different methods across the three datasets showcased in Figure 2b. It can be observed that AddAG-AE increases by 2.8% and 2.3% in AUC score, respectively, on NYC Taxi Trips and IDS 2017 compared to the corresponding second-best model, and its performance is approximately flat with the best model on Enron Mail. In addition, Table 7 shows that AddAG-AE achieves gains of 5.3%, 2.6%, and 1.5% in precision on NYC Taxi Trips, IDS 2017 and Enron Mail, respectively, and increases the recall by 3.8% and 2.2% on IDS 2017 and Enron Mail. In summary, compared with all baselines, AddAG-AE has stable and better performance in three metrics on three datasets.

5.4.2. Ablation Experiment

In this section, we carry out a comparative study by contrasting the AddAG-AE method with the unenhanced method (AddAG-AE without 2D-mlp) and the method lacking SA-decoding improvement (AddAG-AE without SA-decoding). Evaluating this control group through the AUC score can further verify the necessity and influence of the improvement of AddAG-AE. The experimental results are shown in

Table 8. AUC scores of ablation experiment on three datasets.

Methods	NYC Taxi Trips	IDS 2017	Enron Mail
AddAG-AE without 2D-mlp	0.9355	0.7368	0.6256
AddAG-AE without SA-decoding	0.9369	0.7413	0.6325
AddAG-AE	0.9459	0.7521	0.6521

.

In graph embedding, we improve the decoding manner by considering two-dimensional information (2D-MLP), compared with idiomatic manner [35]. The experiment results showed that it improves by 1.11%, 2.08%, and 4.24% in the AUC scores on the three real-world datasets, respectively. In the anomaly-detection phase, we improved the forecasting model by introducing structure and attribute re-decoding to the latent tensor (SA-decoder), compared with [38]. The experiment results showed that it improves by 0.96%, 1.46%, and 3.10% in AUC scores on three real-world datasets, respectively. It is proved that the two improvements are beneficial to the performance.

5.4.3. Parameter Sensitivity

In this section, we mainly investigate the influence of hyperparameters on model performance, including the embedding vector’s dimension $d^{\prime }$ for GAT-AE, tradeoff parameter $\alpha$, and the number of GAT and LSTM layers.

Figure 3 shows the AUC scores with different hyperparameters on the datasets. For $d^{\prime }$, whose suitable values are related to node attribute dimension and usually vary greatly in different datasets, we use the ratio of the embedding vector’s dimension to its node attributes instead of $d^{\prime }$ as the x-axis (we take $\frac{1}{2}$, 1, $\frac{3}{2}$, …). As shown in Figure 3a, this is the point at which the highest performance is different on different kinds of datasets. It is larger for dense graphs, while it is smaller for sparse ones because of the difference in the information density of the graph. For tradeoff parameter $\alpha$, it shows a similar trend, where the AUC score increases first, then stays flat, and finally decreases as the $\alpha$ value increases from 0 to 1.0, which is especially obvious on IDS 2017 and Enron Mail. Although the highest points on the three datasets differ from one another, this implies the importance of jointly considering the contribution of network structure and node attributes for attributed graph anomaly detection. In terms of the number of network layers, including two parameters—the number of GAT encode layers and that of LSTM encode layers—and taking NYC Taxi Trips datasets as an example, we can see that the two-layer model has the best performance.

6. Discussion

In many practical application scenarios, anomalies in network-structured data from many domains often mean that the system may have potential risks. Accurately and effectively detecting anomalies in these data is of great significance for preventing potential threats, monitoring system status, etc. In this paper, AddAG-AE is proposed to solve the above problems, which considers graph structure, node attribute, and temporal information to achieve anomalous graph (or node) detection by combining the GAT dual autoencoder, LSTM-AE, and forecasting-based model effectively. First, it solves the problem that different information is difficult to fuse in dynamical graph anomaly detection. Second, we improve the GAT dual autoencoder by considering two-dimensional information in the graph embedding phase. Additionally, the dual self-decoding mechanism of GAT is introduced, which improves the performance of the model in the anomaly-detection stage. The whole framework detects anomalies in a self-supervised manner, not depending on labeled data, which can be more applicable to real-world situations. Experiments on three commonly used real-world datasets show that AddAG-AE has superior performance in AUC score, precision, and recall. It has a broad application in different kinds of datasets. Compared with the second-best baseline, the AUC score has increased by 2.8% and 2.3%, respectively, on NYC Taxi Trips and IDS 2017, and its performance is approximately flat with Dominant on Enron Mail; Precision improved by 5.3%, 2.6%, and 1.5% on NYC Taxi Trips, IDS 2017, and Enron Mail, respectively, and the recall increased by 3.8% and 2.2% on IDS 2017 and Enron mail, respectively.

However, research into graph attributes is exploratory and not comprehensive. For example, this paper only uses the weight attribute of nodes, but does not use the type of attribute (such as type of attack in IDS) and edge property attribute of nodes, which is important in some special networks. In future work, we plan to explore more node and edge features in networks and will attempt to improve the model as an end-to-end algorithm without losing the model’s current advantages.