Anomaly Detection over Streaming Graphs with Finger-Based Higher-Order Graph Sketch

Abstract

A streaming graph is a constantly growing sequence of edges, which forms a dynamic graph that changes with every edge in the stream. An anomalous behavior in a streaming graph can be modeled as an edge or a subgraph that is unusual compared to the rest of the graph. Identifying anomalous behaviors in real time is essential to the early warning of abnormal or notable events. Due to the complexity of the problem, little work has been reported so far to solve the problem. In this paper, we propose Finger-based Higher-order Graph Sketch (FHGS for short), which is an approximate data structure for streaming graphs with linear memory usage, high update speed, and high accuracy and supports both edge and subgraph anomaly detection. FHGS first maps each edge into a matrix based on hash functions, and then counts its frequency in a time window with unique fingerprints for detecting anomalies. Extensive experiments confirm that our approach generate high-quality results compared to baseline methods.

1. Introduction

Streaming graphs are drawing increasing attention in both the academic and industrial communities as many graphs in real applications evolve over time. A streaming graph G is an unbounded sequence of items that arrive at a high speed, and each item indicates an edge between two nodes. Typical examples of streaming graphs include social media streams [1], computer network traffic data [2], financial trade network [3], and author-paper graph [4]. Streaming graph analysis is gaining importance in various fields due to the natural dynamicity in many real graph applications.

Anomaly detection has wide applications in different fields, such as railway safety inspection in the traffic field [5], health monitoring in the medicine field [6], spam identification in the communication field [7], and so on. In the graph mining task, anomaly detection can be used to find edges or subgraphs that are unusual compared to the rest of the graph. These anomalous edges or anomalous subgraphs often indicate the happening of abnormal or notable events. We next use an example of monitoring the happening of anomalous behavior to illustrate its basic idea.

Example 1: Consider a computer network, where nodes correspond to machines, and each edge represents a timestamped connection from one machine to another. In such a streaming graph, anomalous behavior can be described as an individual or a group of attackers making a large number of connections to some set of targeted machines to restrict accessibility or look for potential vulnerabilities [8]. These anomalous behaviors can be characterized as edges (i.e., malicious connections) and subgraphs (i.e., malicious patterns) that are unusual compared to the rest of the graph. Detecting both these edge and subgraph anomalies together can provide valuable insights into the structure and behavior of the network.

Challenges: In practice, the large scale and high dynamicity of streaming graph make it both memory- and time-consuming to detect anomalous edges and subgraphs accurately. Conventional machine learning algorithms and deep learning techniques are not suitable for real-time anomaly detection over streaming graphs. For example, Bayesian models [6,9] and hybrid models [10] detect anomalies based on statistical probability, nearest neighbour-based approaches [11,12], and support vector machine (SVM) [13] typically represent real-world objects as feature vectors to detect anomalies. All of these approaches discard the relational information in real-world data, so they cannot be used to detect anomalies over graphs. Generative adversarial networks (GANs) [5], graph neural networks (GNNs) [7], and other deep learning models [14,15,16] rely heavily on training processes and exhibit high time overhead for large-scale data, so they cannot achieve real-time anomaly detection over streaming graphs. So, it is necessary to design a novel data structure that cannot only store streaming graph data but also support anomaly detection algorithms. Existing graph data structures have limitations on supporting anomaly detection over streaming graphs. The adjacency matrix and adjacency list are two traditional graph data structures. However, the adjacency matrix has high memory costs and is not applicable to store large-scale streaming data. On the other hand, the adjacency list has high time costs and is not suitable for storing frequently updated streaming data. Graph summarization techniques can map the graph to the rows and columns of a matrix and preserve the topological information of the graph, which effectively improves the real-time processing ability of streaming graphs. A lot of data structures with graph summarization techniques have been proposed for streaming graphs, such as Count-Min (CM) sketch [17], gSketch [18], TCM [19], gMatrix [20], H-CMS [21], and GSS [22]. However, these prior structures either support limited graph algorithms or have poor accuracy. Specifically, CM sketch and gSketch cannot query the topology of the graph, and TCM and gMatrix support the topology query but have poor accuracy. Though H-CMS extends the CM sketch from a two-dimensional structure to a high-order structure, it has poor storage and query accuracy for a large number of hash collisions. GSS proposes fingerprint technology to improve query accuracy, but it cannot directly be used to solve the anomaly detection problem.

Our solution: In this paper, we propose a novel graph data structure, Finger-based Higher-order Graph Sketch (FHGS for short), which can compress the graph node into the rows and columns of a matrix by using hash functions and can uniquely identify the edges in the graph to improve the storage accuracy by using fingerprint technology. Most importantly, it can preserve the topological information of the graph to support graph algorithms. Then, we propose two edge anomaly detection methods, FHGS-EdgeG and FHGS-EdgeL, and two subgraph anomaly detection methods, FHGS-Graph and FHGS-GraphK, based on our proposed FHGS. These four approaches can detect anomalous edges and subgraphs over streaming graphs, respectively, in near real time.

Contributions: The main contributions of our paper are as follows: (1) We propose FHGS, a novel data structure for streaming graphs. It can store streaming graph data under a constant time and space and achieve high accuracy. Most importantly, it preserves the topological information of the graph and can support subsequent graph algorithms. (2) We propose four approaches based on FHGS to detect anomalous edges and subgraphs over streaming graphs in near real time within constantly updated time and memory. And (3) a large number of experiments demonstrate that our methods outperform state-of-the-art anomaly detection methods on three real-world datasets.

The rest of this paper is organized as follows: Section 2 presents the related work. Section 3 describes the research problem. Section 4 presents the data structure. Section 5 and Section 6 describe the anomaly detection algorithms about edges and subgraphs, respectively. Section 7 analyzes the experimental results on three datasets. Section 8 provides the summary of this work.

2. Related Work

Many reviews have been conducted on the graph-based anomaly detection problem in the last few years [23,24,25,26,27]. In this section, we briefly provide an introduction to prior works.

2.1. Graph Data Structure

Traditional methods for graph storage can be divided into two kinds according to underlying implementation principles. The first kind is native graph storage, which uses a graph prototype to organize and manage data, mainly including adjacency matrix, adjacency list, Neo4j [28], and gStore [29].This kind has a strong traversal capability and high query efficiency, but it is not suitable for frequently updating scenarios. The second kind of graph storage is based on the relationship model, which uses relational database to store the triplet data of the graph, such as 3store [30], SW-Store [31], RDF-3X [32], and Hexastore [33]. This kind can update the data in real time but has high query costs.

2.2. Edge Anomaly Detection

The method for anomaly detection on the edge stream was initially proposed by Sharma [34], which established a data-driven structured model to define the edges that deviate from the expected structural pattern as anomalies. However, this approach has a poor generalization ability due to the constraints of empirical indicators. Deep learning techniques were employed to detect edge anomaly in Yu et al. [35]. Moreover, Aggarwal et al. [36] proposed a network division method to detect anomaly. These two methods have high time costs and cannot achieve real-time edge anomaly detection. Eswaran et al. [2] proposed a principle random algorithm to score the arriving edges according to an anomaly scoring function, while this method is only applicable to specific scenarios and is not scalable.

2.3. Subgraph Anomaly Detection

Akoglu et al. [37] proposed a self-centered algorithm that focused on the derived subgraph formed by neighbors around a single node and found subgraph anomalies through a large number of numerical features extracted from the subgraph. Ji et al. [38] divided the dynamic graph into a sequence of snapshots to detect anomalies between any two snapshots. These two methods are mainly for static graphs or snapshots. Community-based [39] and clustering-based [40] anomaly detection methods are both used to detect subgraph anomalies, but they are only suitable for specific scenarios. Eswaran et al. [41] defined anomalies as large dense subgraphs that suddenly appear or disappear and proposed the spotlight algorithm for anomaly detection. However, this approach is time-intensive when processing large-scale graphs.

3. Problem

A streaming graph G consists of an unbounded sequence of items, denoted by $\{e_1,e_2,\cdots ,e_n\}$. Each arriving item is a directed edge represented as $e=(u,v,w,t)$, where u represents the source node, v represents the destination node, w is the edge weight, and t is the time edge arrived. Thus, the edge streaming sequence forms a directed graph $G=(V,E)$ that changes with the arrival of every item $e_i$, where V and E denote the set of nodes and the set of edges in the graph, respectively.

Definition 1

(Graph Stream Summarization). Given a streaming graph G, the graph stream summarization problem concerns designing a compact data structure $DS$ to represent the streaming graph, where the following conditions hold: (1) The space cost of $DS$ is $O\left(E\right)$; (2) $DS$ changes with each new arriving data item in the streaming graph, and the time complexity of updating $DS$ should be as small as possible; and (3) $DS$ supports various queries over the streaming graph G with small and controllable errors.

In order to reduce the storage scale of the streaming graph, we use summarization to compress the data. The summarization of graph $G=(V,E)$ is a smaller graph $G_h=(V_h,E_h)$ satisfying $|V_h|\le |V|$ and $|E_h|\le |E|$, where a hash function $H(·)$ is used to map each node v in G into a node $H\left(v\right)$ in $G_h$. Nodes with the same hash value are compressed into one node in $G_h$, and the edges between them are also compressed together.

Figure 1 shows an example of graph G and its summarization $G_h$. In this example, we set the value range of the hash function $H(·)$ as [0,32). Nodes c and e are mapped to the same node with a hash value of 5 in $G_h$. The weight of edge $<2,5>$ is 3 in $G_h$, which is the sum of the weight of edge $<a,c>$ and edge $<a,e>$ in G.

Thus, given graph $G=(V,E)$, the graph stream summarization problem is to design a data structure to store the summarization $G_h$, and we call such a data structure as sketch.

In reality, a lot of activities are associated with graph anomalies. For example, the intrusion attacks in computer networks [2], the spread of false information in social networks [42], and the sudden collaboration between scholars of different fields in academic networks [43] can be modeled as edge anomalies. Moreover, behaviors like group fraud in financial networks [3] and coordinated attacks in computer systems [21] can be represented as subgraph anomalies. Both edge anomaly and subgraph anomaly will lead to a large number of connections between nodes, which makes the graph suddenly denser.

As the definition of anomaly is scenario-related, in this paper, we focus on detecting edges and subgraphs that make the graph dense. Further, the graph sketch preserves the topological information of the graph, so a dense graph can be translated into a dense matrix in the sketch structure. So, we define the edge anomaly and subgraph anomaly as follows.

Definition 2

(Edge anomaly). Given an edge, we store it into the sketch structure; if the edge makes a submatrix dense, then it is an anomalous edge.

Definition 3

(Subgraph anomaly). Given a subgraph, we store it into the sketch structure; if the subgraph makes a submatrix dense, then it is an anomalous subgraph.

We later use submartix density and likehood to measure the degree of a matrix to be dense (see Section 5). Based on the above, we can transform the anomaly detection problem within a network into a dense submatrix search problem in the sketch structure. The desired aims of our paper are as follows:

• Storing streaming graph data: Given a streaming graph, design a data structure with graph summarization techniques to store graph data quickly and satisfy accuracy requirement under constant space and time.
• Detecting anomalous edges: Given an edge, store it into the designed data structure and determine whether the edge is a part of a dense submatrix within constant memory and processing time.
• Detecting anomalous subgraphs: Given a subgraph (consisting of edges arrived over a period of time), store it into the designed data structure and determine the presence of a dense submatrix within constant memory and processing time.

Frequently used notations in our paper are outlined in

Table 1. Notations.

Notation	Description
G/$G_t$	streaming graph/graph summarization
$H(·)$	hash function
M	range of hash function
k	number of hash functions/layers
X/m	matrix/matrix order
$X\left[i\right]\left[j\right]$	element at row i and column j
$X\left[i\right]{\left[j\right]}_{-w}$	weight of element at row i and column j
F	range of fingerprint
$h(·)$/$f(·)$	node address/node fingerprint
r	number of addresses
$h_i(·)$/$i_s$	address sequence/address index
$\alpha$	attenuation factor
S	set of all row indices
$S_{cur}$	set of current submatrix row indices
$S_{rem}$	set of remaining row indices
T	set of all column indices
$T_{cur}$	set of current submatrix column indices
$T_{rem}$	set of remaining column indices
$f_{src}(S_x,T_x)$	set of source node fingerprints in submatrix $(S_x,T_x)$
$f_{dst}(S_x,T_x)$	set of destination node fingerprints in submatrix $(S_x,T_x)$
$D(X,S_x,T_x)$	density of submatrix $(S_x,T_x)$
$d_{max}$	maximum of submatrix density
$R(X,u,T_x)$	row sum of elements in submatrix $(u,T_x)$
$C(X,S_x,v)$	column sum of elements in submatrix $(S_x,v)$
$L(X,u,v,S_x,T_x)$	likelihood of index $(u,v)$ relative to submatrix $(S_x,T_x)$

.

4. Finger-Based Higher-Order Graph Sketch

In this section, inspired by Gou et al. [22], we propose FHGS, a novel data structure, which can store streaming graph data under constant time and space and satisfy the accuracy requirement. Most importantly, FHGS can support subsequent graph algorithms. Both theoretical analysis and case illustration show the effectiveness of our approach.

4.1. FHGS: Basic Version

The basic version of FHGS uses a size of $m\times m$ in matrix X to store the edges in $G_h$ and can distinguish these edges by fingerprints. For each node $H\left(v\right)(0\le H(v)<M)$ in $G_h$, we define the node address $h\left(v\right)(0\le h(v)<m)$ as follows:

$$h\left(v\right)=\lfloor {\displaystyle \frac{H\left(v\right)}{F}}\rfloor$$

(1)

Define the node fingerprint $f\left(v\right)(0\le f(v)<F)$ as follows:

$$f\left(v\right)=H\left(v\right)\%F$$

(2)

where M is the value range of the hash function, F is the value range of the fingerprint, and m is the order of the matrix X.

$$M=m\times F$$

(3)

Each edge $<H\left(u\right),H\left(v\right)>$ in $G_h$ is mapped to a cell in row $h\left(u\right)$ and column $h\left(v\right)$ of the matrix X. If the cell is empty, we store the fingerprint pair and weight information $[<f(u),f(v)>,w]$ into the position. If it is not empty, we compare the fingerprint pair information of them. If they are the same, we add the weight to the existing one; otherwise, it means that the position is already occupied by other edges, and a storage collision occurs.

Figure 2 shows the basic version of FHGS for the $G_h$ in Figure 1. Here, we set $m=4$ and $F=8$. The nodes and their corresponding hash values, addresses, and fingerprints are shown in the table. In this example, the edge $<5,15>$ is not stored normally because of the collision with edge $<2,15>$.

4.2. FHGS: Optimization Strategy

In order to solve the storage collision problem shown above, we introduce a position-resetting strategy to optimize the FHGS structure.

For each node $H\left(v\right)$ in $G_h$, we generate a sequence of addresses $\left\{h_i\left(v\right)\right|1\le i\le r\}$ for it. Thus, a node has r addresses, and an edge has $r\times r$ storable positions. If a position is occupied by other edges, it can look for the next position according to the address sequence.

An example of the position-resetting strategy is shown in Figure 3. A node has two addresses, and an edge has four storable positions. In this example, the cells $<h_1\left(s\right),h_1\left(d\right)>$ and $<h_2\left(s\right),h_2\left(d\right)>$ have been occupied, so the edge can be stored in two remaining empty cells. If we set row-first order when selecting the position, then the edge will be stored in $<h_2\left(s\right),h_1\left(d\right)>$.

We use the linear congruence method [44] to generate a linear congruence random sequence $\left\{q_i\left(v\right)\right|1\le i\le r\}$ by choosing an appropriate parameter a, small prime b, and a module p, then

$$\left\{\begin{array}{c}{q}_1\left(v\right)=(a\times f\left(v\right)+b)\%p\hfill \\ q_i\left(v\right)=(a\times q_{i-1}\left(v\right)+b)\%p,(2\le i\le r)\hfill \end{array}\right.$$

(4)

We generate the address sequence as follows:

$$\{h_i\left(v\right)|h_i\left(v\right)=(h\left(v\right)+q_i\left(v\right))\%m,1\le i\le r\}$$

(5)

Khan et al. [20] confirms that the address sequence generated using the linear congruence method is random and independent without duplicate elements.

Apart from fingerprint pair and weight information, we also store an index pair $(i_u,i_v)$ in the cell, which indicates the order of this cell in the address sequence. Figure 4 shows an example of optimized FHGS based on Figure 2. Here, we set $r=2,a=5,b=3$ and $p=8$. The edge $<5,15>$ are stored properly due to the position-resetting strategy.

The position-resetting strategy solves the collision by providing r addresses for a node. Thus, each edge can find the storage position in matrix X by setting the appropriate r value.

4.3. FHGS: Higher-Order Extension

In order to further improve the accuracy, we extend the sketch to multiple layers. We extend the two-dimensional basic version of FHGS to a higher-order structure by using different hash functions.

Figure 5 shows an example of a k-layer graph summarization with k different hash functions. For each edge in G, different hash functions compress it to corresponding graph summarizations, and we report the minimum weight value over all layers as the accurate result.

The complete structure of FHGS is shown in Figure 6, which consists of k $m\times m$ matrices, and each matrix stores the corresponding graph summarization in Figure 5.

When a new edge $(s,d,w,t)$ of G arriving, FHGS stores the edge as follows:

• Map it to $<H\left(s\right),H\left(d\right)>$ in the graph summarization by using hash function $H(·)$ and record the weight w;
• Calculate the fingerprints of the source node $f\left(s\right)$, destination node $f\left(d\right)$, and the address sequences $\left\{h_i\left(s\right)\right\}$,$\left\{h_i\left(d\right)\right\}$;
• Store the edge information $[<f\left(s\right),f\left(d\right)>,<i_s,i_d>,w]$ to a cell in matrix X. Check the $r^2$ positions with addresses $\left\{(h_i\left(s\right),h_j\left(d\right))\right|1\le i\le r,1\le j\le r\}$ one by one. For a position $(h_{i_s}\left(s\right),h_{i_d}\left(d\right))$, if the cell is empty, store the information into it and end up traversal. If the cell is not empty, compare the fingerprint pair and index pair stored in the cell with the ones of the edge; if both of them are equal, add w to the weight in the cell, and end up traversal. Otherwise, continue to check next position.
• When all $r^2$ positions are occupied, randomly select one of the $r^2$ positions as the storage position of the edge, add the weight w to the existing weight of the cell, and assign the updated weight to the edge. This occurs in memory-constrained conditions, and the error is acceptable.

Proposition 1.

FHGS has $O\left(\right|E\left|\right)$ memory cost and constantly updating speed. The storage of graph summarization $G_h$ in the data structure is accurate, and the hash collision rate is controllable.

Proof. 

FHGS stores, at most, $\left|E\right|$ edges in the graph, so the memory complexity is $O\left(\right|E\left|\right)$. When storing an edge, it mostly traverses $r^2$ positions. In a k-layer structure, it costs a total of $k\times r^2$, which is a constant, so the updating time complexity is $O\left(1\right)$. For two edges $e_1=<H\left(s_1\right),H\left(d_1\right)>$, $e_2=<H\left(s_2\right),H\left(d_2\right)>$ in $G_h$, the weights of them are added up if and only if $H\left(s_1\right)=H\left(s_2\right)$ and $H\left(d_1\right)=H\left(d_2\right)$, which means $e_1$ and $e_2$ are the same edge in $G_h$. So, the storage of $G_h$ in FHGS is accurate. The collision errors only exist in the hash procedure. We use $\overline{P}$ to represent the probability of edge collision, which means given an edge e, there is at least one $e^{\prime }\ne e$ satisfying $H\left(e\right)=H\left(e^{\prime }\right)$. So, the probability of no collisions is $P=1-\overline{P}$. We assume there are $\left|E\right|$ edges in G; for an edge $e=<s,d>$, there are D edges connecting with e. The value range of hash function is M. For an edge that is not connected to e, if the edge collides with e, it means both nodes of the edge have collision. Thus, the probability of this is $P_1={\displaystyle \frac{1}{M^2}}$. The total probability of $\left|E\right|-D-1$ edges have no collisions with e is $P_1^{\prime }={(1-P_1)}^{\left|E\right|-D-1}$. For an edge that is connected to e, if the edge collides with e, it means one of the two nodes has collision. The probability of this is $P_2={\displaystyle \frac{1}{M}}$. The total probability of D edges have no collisions with e is $P_2^{\prime }={(1-P_2)}^D$. So, the probability of no edges have collisions with e is $P=P_1^{\prime }\times P_2^{\prime }=e^{-{\displaystyle \frac{\left|E\right|+(M-1)\times D-1}{M^2}}}$. For a graph with $\left|E\right|={10}^5$, we set $D=100,F=256,m=1000$, so $M=256000$. According to the above formula, the accurate rate is $e^{-0.00039}=0.99961$. For the structures which use the adjacent matrix to store $G_h$, there is $M=m$, so the accurate rate is $e^{-0.1999}=0.81881$. So, the hash collision rate is controllable, and the accuracy of FHGS is much higher than existing structures.    □

5. Edge Anomaly Detection

In this section, we propose two edge anomaly detection algorithms based on the FHGS data structures, FHGS-EdgeG and FHGS-EdgeL, which search the dense submatrix from global and local views, respectively.

5.1. FHGS-EdgeG

The FHGS-EdgeG (described in Algorithm 1) searches a global dense submatrix on FHGS and outputs the submatrix density as the anomaly score of the edge.

Firstly, we initialize the FHGS. The streaming data are time-sensitive which means the data received recently contain more valuable information. In order to reflect the timeliness of the data, we set an attenuation factor $\alpha$ and multiply the edge weight in FHGS with $\alpha$ at each timestep to reduce the value of outdated information.

Then, we select the cell $(h_{i_s}\left(s\right),h_{i_d}\left(d\right))$, where the edge is stored, as the initial submatrix $(S_{cur},T_{cur})$, where $S_{cur}=\left\{h_{i_s}\left(s\right)\right\}$ and $T_{cur}=\left\{h_{i_d}\left(d\right)\right\}$, and calculate the initial submatrix density. In line with Saha’s work [45], we define the submatrix density as follows:

Definition 4

(Submatrix Density). Given a matrix X, the density of a submatrix $(S_x,T_x)$, where $S_x\subseteq S$ and $T_x\subseteq T$, is expressed as follows:

$$D(X,S_x,T_x)={\displaystyle \frac{{\sum }_{s\in S_x}{\sum }_{t\in T_x}X\left[s\right]{\left[t\right]}_{-w}}{\sqrt{\left|f_{src}(S_x,T_x)\right|\left|f_{dst}(S_x,T_x)\right|}}}$$

(6)

We iteratively extend the submatrix by selecting row $u_p$ from $S_{rem}$ (or column $v_p$ from $T_{rem}$), which has a maximum row (or column) sum in the remaining matrix, adding the row (or column) to $S_{cur}$ (or $T_{cur}$) and removing it from $S_{rem}$ (or $T_{rem}$) until both $S_{rem}$ and $T_{rem}$ are empty. We calculate the density of current submatrix in each iteration and report the maximum one over all densities.

We take the density as the anomaly score of the edge. A higher score means the edge is more likely in a dense submatrix and thus more likely to be abnormal. In the k-layer FHGS, the final score is the minimum value returned by all layers.

Algorithm 1: FHGS-EdgeG($\mathcal{G}$)

Proposition 2.

The time complexity of FHGS-EdgeG is $O\left(\right|E|\times k\times (m^2+r^2))$.

Proof. 

Submatrix-Density procedure consists of three operations: (a) selecting a row with the maximum sum; (b) selecting a column with the maximum sum; and (c) calculating density. (a) and (b) require at most m steps, so the time complexity is $O\left(m\right)$. Calculating the node fingerprint information of a selected row or column requires the most m steps, so (c) has a a time complexity of $O\left(m\right)$. The total number of iterations is $m+m-2$, so the total time complexity of Submatrix-Density is $O((m+m-2)\times (m+m+m))=O\left(m^2\right)$. Initializing and decaying FHGS needs $O(k\times m^2)$. When receiving an edge, storing it to FHGS needs the most $O(k\times r^2)$. Then, Submatrix-Density costs $O(k\times m^2)$. So, the total time complexity of FHGS-EdgeG is $O(k\times m^2+\left|E\right|\times (k\times r^2+k\times m^2))=O(\left|E\right|\times k\times (m^2+r^2))$.    □

Proposition 3.

The memory complexity of FHGS-EdgeG is $O(k\times m^2)$.

Proof. 

In Submatrix-Density procedure, we use an array of m sizes to remark rows (or columns) and record the sum, and use a list of $m^2$ size to record fingerprints. So, the memory cost of Submatrix-Density is $O(4\times m+2\times m^2)=O\left(m^2\right)$. The FHGS structure costs $O(k\times m^2)$, and Submatrix-Density costs $O(k\times m^2)$ for k layers. Thus, the total memory complexity of FHGS-EdgeG is $O(k\times m^2+k\times m^2)=O(k\times m^2)$.    □

5.2. FHGS-EdgeL

The FHGS-EdgeG has high time complexity because of global searching. Thus, we propose FHGS-EdgeL (described in Algorithm 2), which maintains a local dense submatrix on FHGS and outputs the likelihood value of the edge relative to the dense submatrix as the anomaly score of the edge.

Different from FHGS-EdgeG, FHGS-EdgeL randomly selects an element from matrix X as the initial submatrix $(S_{cur},T_{cur})$ and updates $(S_{cur},T_{cur})$ with a greedy strategy, which aims to keep the submatrix as dense as possible in the subsequent processes.

When an edge arrives, we store it to the position $(h_{i_s}\left(s\right),h_{i_d}\left(d\right))$ and determine whether to expand the initial submatrix. If the submatrix density increases with the addition of a row (or column) where the edge is located, we add row $h_{i_s}\left(s\right)$ (or column $h_{i_d}\left(d\right))$ to $S_{cur}$ (or $T_{cur}$). Then, we iteratively select the row (or column) which has the minimum row (or column) sum in the submatrix and remove it from $S_{cur}$ (or $T_{cur}$) until the submatrix density no longer increases.

Like density, we define the likelihood of the matrix cell $\left(h\right(u),h(v\left)\right)$ relative to the submatrix $(S_x,T_x)$, which implies the closeness of the cell to the submatrix.

Definition 5

(Likelihood). Given a matrix X, the likelihood of the index $\left(h\right(u),h(v\left)\right)$ with respect to a submatrix $(S_x,T_x)$ where $S_x\subseteq S$ and $T_x\subseteq T$, is expressed as follows:

$$L(X,u,v,S_x,T_x)={\displaystyle \frac{{\sum }_{(s,t)\in S_x\times \left\{h\left(v\right)\right\}\cup \left\{h\left(u\right)\right\}\times T_x}X\left[s\right]{\left[t\right]}_{-w}}{|S_x\times \left\{h\left(v\right)\right\}\cup \left\{h\left(u\right)\right\}\times T_x|}}$$

(7)

We take the likelihood as the anomaly score of the edge. A higher score means the edge is more likely to be abnormal. In the k-layer FHGS, the final score is the minimum value returned by all layers.

Algorithm 2: FHGS-EdgeL($\mathcal{G}$)

Proposition 4.

The time complexity of FHGS-EdgeL is $O(k\times m^2+|E|\times k\times (m+r^2))$.

Proof. 

Initializing and decaying FHGS needs $O(k\times m^2)$. When receiving an edge, storing it to FHGS needs the most $O(k\times r^2)$. Expanding and condensing the submatrix both needs the most $O(k\times m)$; thus, the time complexity of Expand and Condense is $O(k\times m+k\times m)$. Calculating the likelihood costs $O(k\times m)$. So, the total time complexity of FHGS-EdgeL is $O(k\times m^2+\left|E\right|\times (k\times r^2+k\times m+k\times m+k\times m))=O(k\times m^2+\left|E\right|\times k\times (m+r^2))$.    □

Proposition 5.

The memory complexity of FHGS-EdgeL is $O(k\times m^2)$.

Proof. 

The FHGS structure costs $O(k\times m^2)$. We use an array of m sizes to remark the submatrix information which takes $O(k\times m)$. So, the total memory complexity of FHGS-EdgeL is $O(k\times m^2+k\times m)=O(k\times m^2)$.    □

6. Subgraph Anomaly Detection

In this section, using the FHGS data structure, we propose using FHGS-Graph and FHGS-GraphK to detect subgraph anomaly. FHGS-Graph searches the dense submatrix from the global view. FHGS-GraphK strategically picks up K elements to search the dense submatrix around them.

6.1. FHGS-Graph

The FHGS-Graph (described in Algorithm 3) searches the dense submatrix on FHGS and outputs the submatrix density as the anomaly score of the subgraph.

We divide the streaming graph into subgraphs according to the edge arrival time and defined time window. Then, we reinitialize FHGS when a new subgraph arrives at each time window.

We first take matrix X as the initial submatrix $(S_{cur},T_{cur})$ and calculate the initial submatrix density. We then iteratively select a row (or column) with the minimum row (or column) sum in the submatrix and remove the row (or column) from $S_{cur}$ (or $T_{cur}$) until the submatrix is empty. In each iteration, we calculate the density of the current submatrix according to Equation (6) and return the maximum density of all submatrices.

Algorithm 3: FHGS-Graph($\mathcal{G}$)

Proposition 6.

The time complexity of FHGS-Graph is $O\left(\right|G|\times k\times m^2+|E|\times k\times r^2)$.

Proof. 

In the Density procedure, we iteratively delete the rows and columns at the highest $m+m-2$ times. Selecting the rows (or columns) with the minimum sum and updating sum both take $O\left(m\right)$. Calculating density costs $O\left(m\right)$. So, the total time complexity of Density is $O((m+m-2)\times m)=O\left(m^2\right)$. Initializing FHGS needs $O(k\times m^2)$. When receiving a subgraph, resetting the matrix needs $O(k\times m^2)$, updating the edges costs $O(k\times r^2)$, and Density takes $O(k\times m^2)$. So, the total time complexity of FHGS-Graph is $O(k\times m^2+\left|G\right|\times k\times m^2+\left|E\right|\times k\times r^2+\left|G\right|\times k\times m^2\left)\right)=O\left(\right|G|\times k\times m^2+\left|E\right|\times k\times r^2)$, where $\left|G\right|$ is the number of subgraphs, and $\left|E\right|$ is the number of edges in G.    □

Proposition 7.

The memory complexity of FHGS-Graph is $O(k\times m^2)$.

Proof. 

In the Density procedure, we use an array of m sizes to remark the rows (or columns) and record the sum, and use a list of $m^2$ sizes to record fingerprints. So, the memory cost of Density is $O(4\times m+2\times m^2)=O\left(m^2\right)$. The FHGS structure costs $O(k\times m^2)$, and Density costs $O(k\times m^2)$ for k layers. Thus, the total memory complexity of FHGS-Graph is $O(k\times m^2+k\times m^2)=O(k\times m^2)$.    □

6.2. FHGS-GraphK

Inspired by the idea that the element with a higher weight is more likely to be a part of the dense submatrix, we propose FHGS-GraphK (described in Algorithm 4), which searches dense submatrix around top-K elements in the matrix and returns the submatrix density as the anomaly score of the subgraph.

We iteratively select the largest element in matrix X as the initial submatrix $(S_{cur},T_{cur})$ and use the $Submatrix$-$Density$ process in Algorithm 1 to calculate the submatrix density. Then, we remove this element from the matrix after each iteration and continue to select the next largest element until K elements are selected.

Algorithm 4: FHGS-GraphK($\mathcal{G},K$)

Proposition 8.

The time complexity of FHGS-GraphK is $O\left(\right|G|\times k\times K\times m^2+|E|\times k\times r^2)$.

Proof. 

The Density-K procedure is the same as Submatrix-Density, so the time complexity of Density-K is $O(K\times m^2)$. Other procedures are similar to FHGS-Graph; thus, the total time complexity of FHGS-GraphK is $O(k\times m^2+\left|G\right|\times k\times m^2+\left|E\right|\times k\times r^2+\left|G\right|\times k\times K\times m^2\left)\right)=O\left(\right|G|\times k\times K\times m^2+\left|E\right|\times k\times r^2)$, where $\left|G\right|$ is the number of subgraphs, and $\left|E\right|$ is the number of edges in G. □

Proposition 9.

The memory complexity of FHGS-GraphK is $O(k\times m^2)$.

Proof. 

The memory complexity of Density-K is $O(K\times m^2)$. The FHGS structure costs $O(K\times m^2)$. Thus, the total memory complexity of FHGS-GraphK is $O(k\times m^2+k\times m^2)=O(k\times m^2)$. □

7. Experiments

In this section, we evaluate the effectiveness and scalability of the experimental results. All the algorithms were implemented in C++ and Python, run on a PC with an Intel i5 1.6GHz CPU and 8GB memory. We take the Area under the ROC (AUC) and the running time as evaluation metrics. Every quantitative test was repeated for 5 times, and the average is reported.

Datasets: We use three real-life datasets:

• DARPA [8]: This dataset includes four attack types: DoS (Denial of Service), U2R (User to Root), R2L (Remote to Local), and Probe. It records the IP-IP connection and provides the ground truth. It contains 25K nodes, 4M edges, and 46K timestamps.
• ISCX-IDS2012 [46]: This dataset simulates intrusion scenarios based on real network traffic and records network activity over a seven-day period. It contains 30k nodes, 1M edges, and 165K timestamps.
• UNSW-NB15 [47]: This dataset contains a large amount of network traffic data and simulated attack data, covering nine attack types: Fuzzers, Analysis, Backdoors, DoS, Exploits, Generic, Reconnaissance, Shellcode, and Worms. It contains 45 nodes, 1M edges, and 43K timestamps.

Baselines: We implement and compare our algorithms with the below baselines:

• AnoEdge-G and AnoEdge-L [21]: Two baselines for detecting edge anomalies using the H-CMS data structure.
• MIDAS-R [48]: The baseline for detecting edge anomalies based on microcluster.
• F-FADE [49]: The baseline for detecting edge anomalies using frequency factorization.
• FHGS-EdgeG and FHGS-EdgeL: Our methods for detecting edge anomalies.
• AnoGraph and AnoGraph-K [21]: Two baselines for detecting subgraph anomalies using the H-CMS data structure.
• SPOTLIGHT [41]: The baseline for detecting subgraph anomalies using a randomized approach.
• ANOMRANK [50]: The baseline for detecting subgraph anomalies using PageRank score vectors.
• FHGS-Graph and FHGS-GraphK: Our methods for detecting subgraph anomalies.

Parameter settings: For our methods, we fix FHGS layers $k=1$, hash function range M = 320,000, parameters $a=5,b=3,p=8$, and matrix order $m=32$. We set the number of addresses on DARPA, ISCX-IDS2012, and UNSW-NB15 as 1, 1, and 4, respectively. In edge anomaly detection experiments, we set the attenuation factor on DARPA, ISCX-IDS2012, and UNSW-NB15 as 0.9, 0.96, and 0.98, respectively. In subgraph anomaly detection experiments, on DARPA, ISCX-IDS2012, and UNSW-NB15, we set time window as 15, 30, and 15, fix the edge threshold as 100, and fix $K=5$. For baselines, we set the layers of H-CMS to 1 in AnoEdge and AnoGraph for fairness and follow other parameter settings, as suggested in previous papers. We conduct a large number of experiments under different parameter settings and choose the most representative parameter values here.

7.1. Experiments on Edge Anomaly Detection

Effectiveness: 

Table 2. AUC and running time on edge anomaly detection.

Datasets	AnoEdge-G	AnoEdge-L	MIDAS-R	F-FADE	FHGS-EdgeG	FHGS-EdgeL
DARPA	0.956	0.949	0.938	0.905	0.973	0.965
	221.193 s	16.327 s	7.297 s	649.622 s	269.902 s	43.903 s
ISCX-IDS2012	0.939	0.933	0.804	0.522	0.955	0.960
	50.836 s	9.371 s	52.753 s	346.427 s	70.265 s	18.210 s
UNSW-NB15	0.356	0.510	0.427	0.282	0.795	0.649
	46.027 s	4.900 s	0.714 s	177.417 s	67.425 s	12.704 s

and Figure 7 show the AUC and running time of edge anomaly detection approaches on three datasets. The results show that our approaches outperform on detection accuracy and can achieve the near real-time requirement. Compared to the best prior results, FHGS-EdgeG and FHGS-EdgeL improve the AUC by 1.78% and 0.94% on DARPA, 1.7% and 2.24% on ISCX-IDS2012, and 55.88% and 27.25% on UNSW-NB15, respectively. F-FADE shows a large computation time because it needs to extract patterns for every update. Our approaches have additional time expenses due to the use of position-resetting strategy on the FHGS structure which focuses on improving storage accuracy, even though our approaches can also detect edge anomalies in a small time span. Moreover, since FHGS-EdgeL maintains a local dense submatrix, it is faster than FHGS-EdgeG.

Scalability: Figure 8a–d plot the running time with an increasing number of edges, FHGS layers, matrix size ($m^2$), and address sequences, respectively. All of these results imply the scalability of FHGS-EdgeG and FHGS-EdgeL.

7.2. Experiments on Subgraph Anomaly Detection

Effectiveness: 

Table 3. AUC and running time on subgraph anomaly detection.

Datasets	AnoGraph	AnoGraph-K	SPOTLIGHT	ANOMRANK	FHGS-Graph	FHGS-GraphK
DARPA	0.821	0.824	0.715	0.741	0.869	0.874
	2.006 s	2.110 s	72.320 s	25.314 s	29.809 s	29.580 s
ISCX-IDS2012	0.936	0.935	0.859	0.691	0.988	0.989
	2.476 s	3.298 s	121.832 s	30.025 s	14.328 s	14.324 s
UNSW-NB15	0.959	0.959	0.837	0.786	0.990	0.991
	2.600 s	3.843 s	135.948 s	20.271 s	10.059 s	10.506 s

and Figure 9 show the AUC and running time of subgraph anomaly detection approaches on three datasets. We can see that our approaches demonstrate superior performance in terms of detection accuracy and can meet near real-time demand. Compared to the best prior results, FHGS-Graph and FHGS-GraphK improve the AUC by 5.46% and 6.07% on DARPA, 5.55% and 5.66% on ISCX-IDS2012, and 3.23% and 3.34% on UNSW-NB15, respectively. ANOMRANK has a low AUC because it is not meant for processing streaming data. SPOTLIGHT has a large computation time due to the randomized searching approach. Although our algorithms take more time, they can also detect subgraph anomalies in near real time.

Scalability: Figure 10a–d plot the running time with an increasing number of edges, FHGS layers, matrix size, and K value, respectively. All of these results imply the scalability of FHGS-Graph and FHGS-GraphK.

8. Conclusions

Network anomaly detection is widely used in various real-world scenarios. However, existing methods cannot detect anomalies over streaming graphs quickly and accurately. In this paper, we first propose a higher-order graph data structure, FHGS, which can store streaming graph data under constant time and space with high accuracy and preserve the topological structure of the graph to support graph algorithms. Based on the FHGS structure, we then propose two edge anomaly detection algorithms, FHGS-EdgeG and FHGS-EdgeL, and two subgraph anomaly detection algorithms, FHGS-Graph and FHGS-GraphK. These four algorithms detect anomalies by searching a dense submatrix on the data structure. For edge anomaly detection, FHGS-EdgeG searches the dense submatrix from a global view and has high accuracy, while FHGS-EdgeL maintains the dense submatrix on a local scale and performs better in time complexity. For subgraph anomaly detection, FHGS-Graph searches the dense submatrix on global scale, while FHGS-GraphK strategically picks up K elements to search the dense submatrix around them. The experimental results on the three datasets demonstrate that our methods can detect both edge and subgraph anomalies quickly and accurately compared to state-of-the-art baselines.