A Regularized Graph Neural Network Based on Approximate Fractional Order Gradients

Abstract

Graph representation learning is a significant challenge in graph signal processing (GSP). The flourishing development of graph neural networks (GNNs) provides effective representations for GSP. To effectively learn from graph signals, we propose a regularized graph neural network based on approximate fractional order gradients (FGNN). The regularized graph neural network propagates the information between neighboring nodes. The approximation strategy for calculating fractional order derivatives avoids falling into fractional order extrema and overcomes the high computational complexity of fractional order derivatives. We further prove that such an approximation is feasible and FGNN is unbiased towards the global optimization solution. Extensive experiments on citation and community networks show that the proposed FGNN has improved recognition accuracy and convergence speed than vanilla FGNN. The five datasets of different sizes and domains confirm the great scalability of our proposed method.

1. Introduction

Network data are increasingly available in the digital world where communication networks, social networks, biological networks, and vehicular networks generate a large amount of data daily [1,2]. Graphs provide effective mathematical descriptions of such data, which are often irregular and complex but with significant potential to reveal intrinsic properties of the networks, such as structures, link information, and object information [3,4]. Thus, learning from such data has attracted significant research interest in both academia and industry. Particularly, graph neural networks (GNNs)—built upon graph convolutions to design the learning process for graph structure properties—have shown the great potential to process graph signals, extract the multi-scale spatial information, and map it into higher dimensions for node classification, edge prediction, and graph clustering [3,5,6,7,8,9,10,11,12,13].

GNNs build upon the classic neural networks incorporating gradient descent methods [14,15,16]. Such networks converge by following the steepest gradient searching direction on the first order derivative, which is usually slower than higher order methods (with the order greater than 1) [17]. Furthermore, there exists a potential risk of getting stuck in the first order local optimum, even though the random sampling methods have significantly reduced this risk [15,16].

To overcome such limitations, the application of fractional calculus in neural networks has been studied to exploit the long-term memory and non-locality characteristics of fractional order derivatives [18]. For example, Pu et al. adopted fractional order derivatives to replace integer order derivatives in the gradient descent method directly [19]. Wang et al. and Bao et al. proposed the novel fractional order backpropagation deep networks based on fractional order derivatives [20,21]. Khan et al. proposed the fractional order gradient radial basis function network [22]. Compared with the first order gradient models, fractional order gradient methods have two noticeable properties. First, for many functions, the local optimum points with different fractional orders are usually different (see Figure 1). Second, the computational complexity of fractional order derivatives is higher than the first order derivative. These two properties bring challenges to design a graph neural network based on fractional order gradients.

• Although the fractional order gradient descent algorithm can effectively avoid falling into the first order local extreme point by exploiting the inconsistency of different order extreme points, it will converge to the local extreme point of its order, which differs from the first order extremum. Therefore, the fractional order algorithm is usually not optimal because the first order derivative is usually not zero when its fractional order derivative reaches zero (see Figure 1).
• The fractional derivative of weight parameters in the neural networks requires a complex fractional order chain rule of composite functions, leading to high computational complexity in the iterative optimization of neural networks.

This paper proposes a regularized graph neural network based on approximate fractional order gradients (FGNN) to address the problems by integrating fractional order derivatives into graph neural networks and employing an unbiased simplified approximation strategy to reduce the computation of the fractional order derivatives. The main contributions are summarized as follows.

• We propose a regularized graph neural network based on approximate fractional order gradients for graph representation learning. The proposed approximation strategy avoids falling into fractional order local extreme points, improves the overall performance, and promises a faster convergent speed. Particularly, FGNN achieves better recognition accuracy and faster convergent speed than vanilla GNN on semi-supervised node classification tasks.
• We prove that the approximation for calculating fractional derivatives is biased for its order optimum, while unbiased for the first order optimum. Therefore, the approximation strategy is beneficial to the optimization problem and guarantees the determined convergency towards the first order extremum, where the first order derivative is zero while the fractional order derivative is not zero.
• We analyze the convergence behavior of fractional order derivatives theoretically and identify the convergence condition of the proposed FGNN.

This article is arranged as follows. Section 2 reviews the related work. Section 3 proposes the regularized graph neural network based on approximate fractional order gradients. Section 4 proves the feasibility and convergence of the proposed FGNN. Section 5 evaluates the proposed method using experiments on citation and community network datasets. Section 6 discusses the limitations of our work and future research work. Section 7 concludes this paper.

The key symbols used in this paper are listed in

Table 1. Summary of the key symbols.

Notations	Description
V	vertex set of a graph
E	edge set of a graph
${\mathit{H}}^m$	features of all nodes at the m-th layer
$\mathit{A}$	adjacent matrix of a graph
$\mathit{D}$	degree matrix of a graph
$\sigma (·)$	activation function
${\mathit{W}}^m$	weight parameters of the m-th layer
$\tilde{\mathit{A}}$	normalized adjacent matrix
$\widehat{\mathit{Y}}$	predicted values
$\mathit{Y}$	ground-truth values
$\mathcal{L}$	loss function
$\eta$	learning rate
${\mathit{W}}^{m(+)}$	updated weights of the m-th layer
$\mathcal{D}$	fractional order gradient operator
$\nu$	arbitrary real order
a	initial point of fractional order derivative
⊙	Hadamard product
t	index for the current iteration epoch
T	iterative upper bound
$\mathsf{\lambda }$	L2 regularization coefficient
${\mathcal{L}}_{L2}$	loss function with L2 regularization

.

2. Related Work

This section reviews the related research on GNN and introduces fractional calculus theory to set the background for the proposed method.

2.1. Graph Neural Networks

Typical neural networks (NNs) usually operate in the Euclidean space: either sequences or grid structured data (audios, images, videos, etc.). To study graph-structured data, GNNs extend NNs into the graph domain through three main types of propagation methods: convolution aggregators, attention aggregators, and gate aggregators.

The convolution operation of classic convolutional neural networks is generalized to the graph domain in convolution aggregators, which are composed of spectral and non-spectral methods. Spectral methods focus on the graph spectrum, which calculates the Laplacian eigenbasis of the specific graph structure. Bruna et al. [10] proposed a spectral network, which extends convolution to the Laplacian spectrum. To solve the high computational complexity, Defferrard et al. [12] defined a K localized convolution instead of computing the Laplacian eigenvectors; Kipf and Welling [6] limited the graph convolution operation to $K=1$ to moderate the redundant information aggregation in the graph spectrum. Non-spectral methods directly work on the graph, defining the spatial convolution operation on the adjacent neighbors. However, it is rather difficult to specify uniform convolution operations on neighbors of different sizes. Hamilton et al. [3] proposed the GraphSAGE, which generates embedding by sampling and aggregating information from nodes and their neighbors.

Attention mechanism, e.g., self- and intra-attention, has proven to be an effective way to represent sequence-based data [23]. Velickovic et al. proposed the graph attention networks (GAT) [24] to integrate the attention mechanism into the graph propagation process. GAT aggregates the information of nodes and their neighbors by computing the self-attention and intra-attention coefficients of the concatenated features of the vertex pair. This method also utilizes K independent attention heads to calculate the intermediate state and concatenates or averages their features.

Gate aggregators are proposed to address the limitations of existing GNN methods and enhance the long-term information broadcast while processing graph data. Similar to gated recurrent units (GRU) and long short-term memory (LSTM), gated graph neural networks are proposed to utilize the GRUs to aggregate information from other vertices and from the previous stage [25,26]; Zhang et al. proposed the Sentence LSTM (S-LSTM) [27] to enhance text coding by transforming texts into graphs.

2.2. Fractional Calculus

Different from integer calculus, fractional order derivative does not have a unified definition. Common definitions of fractional order derivative include Grünwald–Letnikov (GL), Riemann–Liouville (RL), and Caputo derivatives [28,29]. The GL fractional order derivative is given below,

$${}_a^{GL}{\mathcal{D}}_x^{\nu }f\left(x\right)=\underset{h\to 0}{lim}{h}^{-\nu }\sum _{k=0}^{\left[\right(x-a)/h]}\left(\begin{array}{c}-\nu \\ k\end{array}\right)f(x-kh),$$

(1)

where

$$\left(\begin{array}{c}-\nu \\ k\end{array}\right)=\frac{(-\nu )(-\nu +1)\cdots (-\nu +k-1)}{k!}.$$

(2)

Furthermore, ${}_a^{GL}{\mathcal{D}}_x^{\nu }$ denotes the fractional order gradient operator according to GL definition, $f\left(x\right)$ is a differentiable and integrable function, $\nu$ is the fractional order, which can take any real value. a denotes the start of the duration $[a,x]$, and $[·]$ is the rounding function.

The RL fractional order derivative is given below,

$${}_a^{RL}{\mathcal{D}}_x^{\nu }f\left(x\right)=\frac{1}{\Gamma (n-\nu )}\frac{d^n}{d{x}^n}{\int }_a^x\frac{f\left(y\right)}{{(x-y)}^{\nu -n+1}}dy,$$

(3)

where ${}_a^{RL}{\mathcal{D}}_x^{\nu }$ denotes the fractional order gradient operator based on RL definition. $n=[\nu +1]$ means that n is the minimum integer greater than $\nu$, and $\Gamma (·)$ is the gamma function. Furthermore, GL fractional order derivative can be inferred from the rule of RL principle.

The Caputo fractional order derivative is given below,

$${}_a^C{\mathcal{D}}_x^{\nu }f\left(x\right)=\frac{1}{\Gamma (n-\nu )}{\int }_a^x{(x-y)}^{n-\nu -1}{f}^{\left(n\right)}\left(y\right)dy,$$

(4)

where ${}_a^C{\mathcal{D}}_x^{\nu }$ denotes the fractional order gradient operator based on Caputo principle. In this paper, we mainly adopt the GL definition following fractional order derivative and use the equivalent notations ${\mathcal{D}}_x^{\nu }{=}_0^{GL}{\mathcal{D}}_x^{\nu }$ .

3. Fractional Order Graph Neural Network

In this section, we propose a regularized graph neural network based on approximate fractional order gradients to address the problem that the vanilla fractional order neural network may become stuck in its local optima. We first introduce the fractional order GNN, then extend fractional order GNN to semi-supervised tasks by employing transductive learning. Finally, the approximation strategy is proposed for calculating fractional order derivatives.

3.1. Fractional Order GNN

We firstly introduce the backpropagation for GNN [6]. Graph neural networks were proposed to extend neural networks into non-Euclidean domain. In a graph $G=(V,E)$, each node $v^i\in V$ has d-dimension features ${\mathit{h}}^i\in {\mathbb{R}}^{d\times 1}$ and each edge $e_{ij}\in E$ denotes the connection relationship between node i and node j. $e_{ij}=1$ if node i connects to node j else $e_{ij}=0$.

For an undirected graph G, the adjacent matrix $\mathit{A}$ represents the set of all edges.

$${\mathit{A}}_{ij}=\left\{\begin{array}{cc}1,\hfill & e_{ij}=1;\hfill \\ 0,\hfill & e_{ij}=0,\hfill \end{array}\right.$$

(5)

where $\mathit{A}$ is symmetric, with all diagonal entries as zeros ${\mathit{A}}_{ii}=0$. Then, the degree matrix $\mathit{D}$ can be obtained, which counts the number of neighbors of each node,

$${\mathit{D}}_{ij}=\left\{\begin{array}{cc}\sum _{k=1}^{\parallel V\parallel }{\mathit{A}}_{ik},\hfill & i=j;\hfill \\ 0,\hfill & i\ne j,\hfill \end{array}\right.$$

(6)

where the diagonal entries of $\mathit{D}$ are equivalent to their corresponding column or row summation of the adjacent matrix $\mathit{A}$. Furthermore, non-diagonal entries of $\mathit{D}$ are zeros.

All node features and adjacent matrix are the input of GNN. Moreover, the input features of next layer $m+1$ come from the output of the previous layer m. Both ${\mathit{H}}^0$ and $\mathit{A}$ are the input of the first layer,

$${\mathit{H}}^0=\left[{\mathit{h}}^1,{\mathit{h}}^2,\cdots ,{\mathit{h}}^{\parallel V\parallel }\right].$$

(7)

Based on the node classification task, the iterative scheme for computing the state ${\mathit{H}}^{m+1}$ is given below,

$$\begin{array}{cc}\hfill {\mathit{Z}}^{m+1}=& {\mathit{W}}^m{\mathit{H}}^m\tilde{\mathit{A}};\hfill \\ \hfill {\mathit{H}}^{m+1}=& \sigma \left({\mathit{Z}}^{m+1}\right);\hfill \\ \hfill \widehat{\mathit{Y}}=& g\left({\mathit{H}}^M\right),\hfill \end{array}$$

(8)

where $\sigma (·)$ denotes the activation function, ${\mathit{W}}^m$ means weight parameters matrix at step m, $g(·)$ is the output classification function (such as sigmoid layer or softmax layer) and $\widehat{\mathit{Y}}$ is the prediction result. Here $\tilde{\mathit{A}}$ is the normalized adjacent matrix,

$$\tilde{\mathit{A}}={\mathit{D}}^{-\frac{1}{2}}(\mathit{I}+\mathit{A}){\mathit{D}}^{-\frac{1}{2}}.$$

(9)

The mean square error is adopted as the loss function,

$$\mathcal{L}=\frac{1}{2}{(\widehat{\mathit{Y}}-\mathit{Y})}^2,$$

(10)

where $\mathcal{L}$ function evaluates the similarity between the predicted $\widehat{\mathit{Y}}$ and the ground-truth value $\mathit{Y}$. Firstly, we define a factor ${\delta }^m$ to simplify the representation of the gradient,

$${\delta }^m=\frac{\partial \mathcal{L}}{\partial {\mathit{Z}}^m}.$$

(11)

According to Equation (10), we can obtain,

$${\delta }^M=(\widehat{\mathit{Y}}-\mathit{Y})\odot g^{\prime }\left({\mathit{H}}^M\right)\odot {\sigma }^{\prime }\left({\mathit{Z}}^M\right),$$

(12)

where ⊙ denotes the Hadamard product (entrywise product). ${\sigma }^{\prime }(·)$ denotes the first order derivative of the activate function $\sigma (·)$. Then the relationship between ${\delta }^m$ and ${\delta }^{m+1}$ can be given by,

$${\delta }^m=\frac{\partial \mathcal{L}}{\partial {\mathit{Z}}^{m+1}}\frac{\partial {\mathit{Z}}^{m+1}}{\partial {\mathit{Z}}^m}={\left({\mathit{W}}^m\right)}^T{\delta }^{m+1}\tilde{\mathit{A}}\odot {\sigma }^{\prime }\left({\mathit{Z}}^m\right).$$

(13)

Calculate the derivative of $\mathcal{L}$ with respect to ${\mathit{W}}^m$,

$$\frac{\partial \mathcal{L}}{\partial {\mathit{W}}^m}={\delta }^{m+1}\frac{\partial {\mathit{Z}}^{m+1}}{\partial {\mathit{W}}^m}={\delta }^{m+1}\tilde{\mathit{A}}{\left({\mathit{H}}^m\right)}^T.$$

(14)

The weight parameters are thus updated iteratively,

$${\mathit{W}}^{m(+)}={\mathit{W}}^m-\eta \frac{\partial \mathcal{L}}{\partial {\mathit{W}}^m},\{m=0,1,\cdots ,M-1\},$$

(15)

where ${\mathit{W}}^{m(+)}$ is the updated weight parameters and $\eta >0$ is the learning rate. Similarly, the updated weight parameters of the fractional order GNN is given below,

$${\mathit{W}}^{m(+)}={\mathit{W}}^m-\eta {\mathcal{D}}_{{\mathit{W}}^m}^{\nu }\mathcal{L},\{m=0,1,\cdots ,M-1\}.$$

(16)

Neural networks always overfit easily for a small-scale dataset. Utilizing L2 regularization can efficiently avoid overfitting without modifying the structure of networks [30]. Therefore, by introducing the L2 regularization term into the total loss, the modified loss function can be presented as,

$${\mathcal{L}}_{L2}=\mathcal{L}+\frac{\mathsf{\lambda }}{2}\sum _{m=0}^{M-1}{\parallel {\mathit{W}}^m\parallel }^2,$$

(17)

where $\mathsf{\lambda }$ denotes the L2 regularization coefficient. According to (17), we obtain

$${\mathit{W}}^{m(+)}={\mathit{W}}^m-\eta {\mathcal{D}}_{{\mathit{W}}^m}^{\nu }{\mathcal{L}}_{L2},\{m=0,1,\cdots ,M-1\}.$$

(18)

The regularized fractional order GNN updates the weight parameters of graph neural networks by means of the fractional order gradient descent method.

3.2. Semi-Supervised Training Mechanism

Although the neural network structure has been developed, the training mechanism should be different for different tasks. In semi-supervised node classification tasks, transductive learning should be adopted [6,31], where all samples are visible for the training process and only the labeled samples are involved in the backward propagation.

In the forward propagation, all node features and adjacent matrix are input to the network, as follows,

$${\mathit{H}}^0=\mathrm{concat}\left(\left[{\mathit{X}}_L,{\mathit{X}}_U\right]\right).$$

(19)

where $\mathit{X}$ denotes the node features, the subscripts U and L denote the labeled and unlabeled samples for training and testing, respectively. Function $\mathrm{concat}(·)$ denotes the concatenation of matrices. Therefore, the input feature ${\mathit{H}}^0$ of the network contains ${\mathit{X}}_L$ and ${\mathit{X}}_U$.

In the backward propagation, only the loss between the predicted results and the labels of training nodes is calculated,

$${\mathcal{L}}_{L2}=\frac{1}{2}{({\widehat{\mathit{Y}}}_L,{\mathit{Y}}_L)}^2+\frac{\mathsf{\lambda }}{2}\sum _{m=0}^{M-1}{\parallel {\mathit{W}}^m\parallel }^2.$$

(20)

According to the given training loss (20), the fractional order gradients of the weight parameters are calculated and the weight parameters are updated by (18).

3.3. Approximation Strategy

The computational complexity of vanilla fractional order derivatives is usually high, given the Di Bruno’s formula [32,33]. This subsection introduces approximate computation for calculating fractional order derivatives. We will prove such approximation is unbiased and feasible for the optimization in Section 4.1.

According to the Di Bruno’s formula, the Caputo fractional order derivative of a composite function expression can be derived as follows,

$$\begin{array}{cc}\hfill {\mathcal{D}}_x^{\nu }& f\left[u\left(x\right)\right]=\sum _{m=0}^{\infty }\frac{f^m\left[u\left(x\right)\right]}{m!}\sum _{k=m}^{\infty }\frac{sin\left(\pi \right(\nu -k\left)\right)}{\pi (\nu -k)}\left(\frac{\Gamma (\nu +1)}{\Gamma (k+1)}\right)\hfill \\ \hfill \times & x^{k-\nu }\left(\sum _{j=0}^m{(-1)}^j\left(\begin{array}{c}m\\ j\end{array}\right)u{\left(x\right)}^j\frac{d^k}{d{x}^k}u{\left(x\right)}^{m-j}\right).\hfill \end{array}$$

(21)

We can thus obtain the approximate fractional order chain rule [20,21],

$${\mathcal{D}}_x^{\nu }\left(f\left[u\left(x\right)\right]\right)\approx \frac{df}{du}{\mathcal{D}}_x^{\nu }u\left(x\right).$$

(22)

Obviously, when $\nu =1$, (22) is completed. The approximate chain rule based fractional order derivative of $\mathcal{L}$ and ${\mathcal{L}}_{L2}$ with respect to ${\mathit{W}}^l$ are given below,

$${\mathcal{D}}_{{\mathit{W}}^m}^{\nu }\mathcal{L}\approx \frac{\partial \mathcal{L}}{\partial {\mathit{W}}^m}\frac{{\left({\mathit{W}}^m\right)}^{1-\nu }}{\Gamma (2-\nu )}={\delta }^{m+1}\tilde{\mathit{A}}{\left({\mathit{H}}^m\right)}^T\odot \frac{{\left({\mathit{W}}^m\right)}^{1-\nu }}{\Gamma (2-\nu )},$$

(23)

$$\begin{array}{cc}\hfill {\mathcal{D}}_{{\mathit{W}}^m}^{\nu }{\mathcal{L}}_{L2}& \approx \frac{\partial \mathcal{L}}{\partial {\mathit{W}}^m}\frac{{\left({\mathit{W}}^m\right)}^{1-\nu }}{\Gamma (2-\nu )}+\mathsf{\lambda }\frac{{\left({\mathit{W}}^m\right)}^{2-\nu }}{\Gamma (3-\nu )}\hfill \\ \hfill & ={\delta }^{m+1}\tilde{\mathit{A}}{\left({\mathit{H}}^m\right)}^T\odot \frac{{\left({\mathit{W}}^m\right)}^{1-\nu }}{\Gamma (2-\nu )}+\mathsf{\lambda }\frac{{\left({\mathit{W}}^m\right)}^{2-\nu }}{\Gamma (3-\nu )}.\hfill \end{array}$$

(24)

Based on the fractional order gradient descent method, semi-supervised training mechanism, and approximate fractional order derivatives, the detailed pipeline and algorithm of FGNN are shown in Figure 2 and Algorithm 1.

Algorithm 1: Approximate fractional order GNN

4. Theoretical Analysis

This section proves the feasibility proof of the approximate fractional order GNN, then presents the convergence proof of the proposed FGNN.

4.1. Approximation and Feasibility Proof

It has been revealed that the fractional order extreme point is not coincident with the first order extrema point [19]. Therefore, it is potential for the fractional order gradient to escape from the local optimal point. Nevertheless, such a gradient method cannot promise convergence to the real extreme point. To solve this problem, the truncate and approximate principle [20,34,35] is proposed to modify the fractional order gradient method which could reach the real extreme point just like the integer order method, but with faster convergence than the first order.

Here we prove the approximation and feasibility. For the truncation and approximate principle, it can be divided into two types. The first type is the chain rule approximation [20,21] (22), where the fractional order chain rule of the composite function $f\left[u\right(x\left)\right]$ can be approximately divided into the product of the first order derivative $\frac{df}{du}$ and fractional order derivative $D_x^{\nu }u\left(x\right)$. If $u\left(x\right)=x$, the fractional order derivative of the composite function reduces as,

$$D_x^{\nu }f\left[u\left(x\right)\right]\approx \frac{df}{dx}\frac{x^{1-\nu }}{\Gamma (2-\nu )}.$$

(25)

The second type is the approximate expansion [34,35],

$${}_{x_{k-1}}{D}_{x_k}^{\nu }f\left(x\right)=\sum _{i=1}^{\infty }\frac{f^{\left(i\right)}\left(x_{k-1}\right)}{\Gamma (i+1-\nu )}{(x_k-x_{k-1})}^{i-\nu }\approx \frac{f^{\left(1\right)}\left(x_{k-1}\right)}{\Gamma (2-\nu )}{(x_k-x_{k-1})}^{1-\nu },$$

(26)

where the fractional order of the function $f\left(x\right)$ can be decomposed into the Taylor expansion. The first item significantly dominates the searching direction. Therefore, researchers mainly focus on the first order truncation.

These two types of fractional order derivatives can be divided into the product between the first order derivative $f^{\left(1\right)}\left(x\right)$ and the power function of the variable x. Therefore, their extreme points contain the first order one. Furthermore, if $\parallel x\parallel <1$, (25) has a greater step than the first order gradient. If $\parallel x_k-x_{k-1}\parallel <1$, (26) also has a greater step than the first order gradient.

Figure 3 shows the minimizing process of the function $f\left(x\right)=x^2-x$ with different gradient descent methods, which include the first order gradient, the fractional order gradient based on the approximate chain rule, the fractional order gradient based on the approximate expansion, and the accurate fractional order gradient. The first row shows the searching process on the function $f\left(x\right)=x^2-x$ from the starting point (red square) to the endpoint (black circle). The second row shows the minimizing function during 20 iterations. The first order and fractional order extreme points are different for the quadratic system, $x_1^{\star }=0.5$ and $x_{\nu }^{\star }=\frac{\Gamma (3-\nu )}{2\Gamma (2-\nu )}$. Therefore $x_1^{\star }\ne x_{\nu }^{\star }$, if $\nu \ne 1$. For the mentioned approximate fractional order derivatives, iterative values have been calculated and their results confirm the consistency between the first order and approximate fractional order extreme points. Therefore, the approximate fractional order extrema are unbiased towards the first order extrema but biased towards the accurate fractional order extrema. Furthermore, these two approximate strategies converge faster than the first order one, whereas the approximate expansion strategy needs the determined initial value $x_0$, which is sensitive to optimization performance. Therefore, we choose the approximate chain rule strategy (25).

Since the example is explicit, the analytic expression for the first order and fractional order derivatives are available. However, for complex implicit functions, we need to use the chain rule or difference method to calculate the first order and fractional order derivatives of the functions. For an implicit function $f\left[u\right(x\left)\right]$, for the sake of easy calculation, we assume that the function value of each point is known, and the first order derivative is,

$$\frac{df\left[u\right(x\left)\right]}{dx}=\frac{f\left[u\right(x\left)\right]-f\left[u\right(x-h\left)\right]}{h}.$$

(27)

Then the approximate fractional order derivative is,

$${\mathcal{D}}_x^{\nu }f\left[u\right(x\left)\right]\approx \frac{df\left[u\right(x\left)\right]}{dx}\frac{x^{1-\nu }}{\Gamma (2-\nu )}=\frac{f\left[u\right(x\left)\right]-f\left[u\right(x-h\left)\right]}{h}\frac{x^{1-\nu }}{\Gamma (2-\nu )}.$$

(28)

Furthermore, the accurate fractional order derivative is,

$${\mathcal{D}}_x^{\nu }f\left[u\right(x\left)\right]=\sum _{i=1}^{\infty }\frac{f^{\left(i\right)}\left[u\left(x\right)\right]}{\Gamma (i+1-\nu )}{h}^{i-\nu }\approx \sum _{i=1}^N\frac{f^{\left(i\right)}\left[u\left(x\right)\right]}{\Gamma (i+1-\nu )}{h}^{i-\nu },$$

(29)

where the higher order (greater than N) items are considered as zeros if $f^{N+1}\left[u\left(x\right)\right]=0$ or $h^{N-\nu }\approx 0$.

From these three derivatives of the complex implicit function $f\left[u\right(x\left)\right]$, we know that the first order derivative needs an addition and a multiplication and the approximate fractional order derivative needs an addition and three multiplications. For the accurate fractional order derivative, the ith item ($i\in \{1,2,\cdots ,N\}$) is given as follows,

$$\frac{f^{\left(i\right)}\left[u\left(x\right)\right]}{\Gamma (i+1-\nu )}{h}^{i-\nu }=\frac{f^{(i-1)}\left[u\left(x\right)\right]-f^{(i-1)}\left[u(x-h)\right]}{\Gamma (i+1-\nu )}{h}^{i-\nu -1},$$

(30)

which needs $2^i-1$ additions and two multiplications. Therefore, the accurate fractional order derivative totally needs $2^{N+1}-3$ additions and $2N$ multiplications. The addition and multiplication complexity of these above three derivatives are concluded in

Table 2. The addition and multiplication complexity of the first order, the approximate fractional order, and the accurate fractional order derivatives.

Derivative	Addition	Multiplication
First order	1	1
Approximate fractional order	1	3
Accurate fractional order	$2^{N+1}-3$	$2N$

. The approximate fractional order derivative has a computational complexity close to the first order one, while the computational complexity of the accurate fractional order derivative is much higher than the first order one.

4.2. Convergence Proof of the Fractional Gradient Descent Method

Although the fractional order gradient descent method has promising performance in dealing with local optimum problems theoretically, the convergence of such a mechanism remains to be proved.

For the sake of simple expression, we utilize the following notations,

$$\Delta \mathit{W}={\mathit{W}}^{+}-\mathit{W}=-\eta {\mathcal{D}}_{\mathit{W}}^{\nu }\mathcal{L}.$$

(31)

Without loss of generality, we analyze the weight parameters ${\mathit{W}}^m$ of the m-th layer, simply denoted as $\mathit{W}$. The superscript k means the k-th iteration.

Lemma 1.

The MSE set $\left\{\mathcal{L}\left({\mathit{W}}^k\right)\right\},(k=1,2,\cdots ,T$) is monotonically decreasing, i.e.,

$$\mathcal{L}\left({\mathit{W}}^{k+1}\right)\le \mathcal{L}\left({\mathit{W}}^k\right).$$

Proof. 

By using the Taylor mean value theorem with Lagrange remainder,

$$\Delta =\mathcal{L}\left({\mathit{W}}^{k+1}\right)-\mathcal{L}\left({\mathit{W}}^k\right)={\mathcal{L}}^{\prime }\Delta {\mathit{W}}^k+\frac{1}{2}{\mathcal{L}}^{″}\Delta {\parallel {\mathit{W}}^k\parallel }^2,$$

(32)

where ${\mathcal{L}}^{\prime }$ and ${\mathcal{L}}^{″}$ denote the first order and the second order derivatives of $\mathcal{L}$, and $\Delta \parallel {\mathit{W}}^k{\parallel }^2$ denote the high-order item of $\parallel {\mathit{W}}^k\parallel$ .

Add (23) and (31) into (32),

$$\begin{array}{cc}\hfill \Delta & =\left({\mathcal{L}}^{\prime }\frac{1}{-\eta {\mathcal{D}}_{{\mathit{W}}^k}^{\nu }\mathcal{L}}+\frac{1}{2}{\mathcal{L}}^{″}\right){\parallel \Delta {\mathit{W}}^k\parallel }^2\hfill \\ \hfill & =\left({\mathcal{L}}^{\prime }\frac{\Gamma (2-\nu )}{-\eta {\mathcal{L}}^{\prime }{\left({\mathit{W}}^k\right)}^{1-\nu }}+\frac{1}{2}{\mathcal{L}}^{″}\right){\parallel \Delta {\mathit{W}}^k\parallel }^2\hfill \\ \hfill & \le \left(-\frac{1}{\eta }\Gamma (2-\nu ){\left({\mathit{W}}^k\right)}^{\nu -1}+C_1\right){\parallel \Delta {\mathit{W}}^k\parallel }^2,\hfill \end{array}$$

(33)

where $C_1\ge \frac{1}{2}{\mathcal{L}}^{″}$. Generally, in the computation of the fractional order derivative, we select $\nu \in [0,2)$, and let $\parallel {\mathit{W}}^k\parallel$ replace ${\mathit{W}}^k$, which guarantees $\Gamma (2-\nu )>0$. The upper bound of the learning rate is $\eta \le \frac{\Gamma (2-\nu )\parallel {\mathit{W}}^k{\parallel }^{\nu -1}}{C_1}$ .

The proof of Lemma 1 is completed. □

Lemma 2.

The MSE function set $\left\{\mathcal{L}\left({\mathit{W}}^k\right)\right\},(k=1,2,\cdots ,T$) is convergent, i.e.,

$$\underset{k\to \infty }{lim}\mathcal{L}\left({\mathit{W}}^k\right)={\mathcal{L}}^{\star }.$$

Proof. 

$\mathcal{L}\left({\mathit{W}}^k\right)$ is the square error, thus $\mathcal{L}\left({\mathit{W}}^k\right)\ge 0$. Applying Lemma 1, $\mathcal{L}\left({\mathit{W}}^{k+1}\right)\le \mathcal{L}\left({\mathit{W}}^k\right)$. Hence there exits ${\mathcal{L}}^{\star }\ge 0$ satisfying,

$$\underset{k\to \infty }{lim}\mathcal{L}\left({\mathit{W}}^k\right)={\mathcal{L}}^{\star }.$$

(34)

The proof of Lemma 2 is completed. □

Lemma 3.

The MSE function set $\left\{\mathcal{L}\left({\mathit{W}}^k\right)\right\},(k=1,2,\cdots ,T$) converges to zero, i.e.,

$$\underset{k\to \infty }{lim}\parallel D_{\mathit{W}}^{\nu }\mathcal{L}\left({\mathit{W}}^k\right)\parallel =0.$$

Proof. 

According to (33), let $\beta =\frac{1}{\eta }\Gamma (2-\nu ){\parallel {\mathit{W}}^k\parallel }^{\nu -1}-C_1\ge 0$,

$$\begin{array}{cc}\hfill \mathcal{L}\left({\mathit{W}}^{k+1}\right)& \le \mathcal{L}\left({\mathit{W}}^k\right)-\beta {\parallel \Delta {\mathit{W}}^k\parallel }^2\hfill \\ \hfill & \le \left(\mathcal{L}\left({\mathit{W}}^{k-1}\right)-\beta {\parallel \Delta {\mathit{W}}^{k-1}\parallel }^2\right)-\beta {\parallel \Delta {\mathit{W}}^k\parallel }^2\hfill \\ \hfill & \cdots \hfill \\ \hfill & \le \mathcal{L}\left({\mathit{W}}^0\right)-\beta \sum _{i=0}^k{\parallel \Delta {\mathit{W}}^i\parallel }^2.\hfill \end{array}$$

(35)

Because $\mathcal{L}\left({\mathit{W}}^{k+1}\right)\ge 0$, therefore,

$$\beta \sum _{i=0}^k{\parallel \Delta {\mathit{W}}^i\parallel }^2\le \mathcal{L}\left({\mathit{W}}^0\right).$$

(36)

Let $k\to \infty$, it holds that,

$$\sum _{i=0}^{\infty }{\parallel \Delta {\mathit{W}}^i\parallel }^2<\infty .$$

(37)

According to the principle of series convergence, the general term is derived as follows,

$$\underset{k\to \infty }{lim}\parallel \Delta {\mathit{W}}^k\parallel =0.$$

(38)

Meanwhile,

$$-\eta {\mathcal{D}}_{\mathit{W}}^{\nu }\mathcal{L}\left({\mathit{W}}^k\right)=\Delta {\mathit{W}}^k;$$

(39)

$$\underset{k\to \infty }{lim}{\mathcal{D}}_{\mathit{W}}^{\nu }\left({\mathit{W}}^k\right)=0.$$

(40)

The proof of Lemma 3 is completed. □

Based on these three lemmas, we conclude that the fractional order gradient method converges when $\eta \le \frac{\Gamma (2-\nu )\parallel {\mathit{W}}^k{\parallel }^{\nu -1}}{C_1}$ .

5. Experiments

In this section, we evaluate the proposed FGNN on citation and community networks to demonstrate its effectiveness. First, we introduce three citation network datasets and two community network datasets. Second, we give implementation details and evaluation metrics. Third, seven comparative methods are introduced as the baselines. Finally, we discuss the effectiveness of the proposed FGNN by quantitatively and qualitatively comparing our proposed FGNN against the baseline models.

5.1. Citation and Community Network Datasets

Following the experimental setup in [36], we introduce citation network datasets including Cora, Citeseer [37], and Pubmed [38] to establish the benchmark, where articles are regarded as nodes and citation links between articles are regarded as undirected edges. These edges contribute to the binary and symmetric adjacent matrix. The datasets contain sparse bag-of-words feature or Term Frequency-Inverse Document Frequency (TF-IDF) vectors for each article and a list of citation links between different articles. The detailed number of nodes, edges, features, and classes is shown in

Table 3. Information of citation network datasets Cora , Citeseer and Pubmed.

Property	Cora	Citeseer	Pubmed
Nodes	2708	3327	19,717
Edges	5429	4732	44,338
Features	1433	3703	500
Classes	7	6	3
Training nodes	140	120	60
Validation nodes	500	500	500
Test nodes	1000	1000	1000

. 20 labeled nodes of each class are adopted for training. 500 and 1000 labeled nodes are adopted for validation and testing, respectively.

In addition, we further introduce two community network datasets Karate club and Football [39,40] (see Figure 4). In Karate club, each node denotes a club member. The nodes have clubs that indicate to which clubs they belong, either ‘Mr. Hi’ or ‘Officer’. In Football, each node denotes an American university. The nodes have values that indicate to which conferences they belong. In both two datasets, 10% of samples are adopted for training, and the other 90% of samples are adopted for testing. Different from the citation network datasets, the community network datasets have no node features. Therefore, two identity matrices of different sizes are the input node features of the networks in Karate club and Football. The detailed number of nodes, edges, features, and classes is shown in

Table 4. Information of community network datasets Karate club and Football.

Property	Karate Club	Football
Nodes	34	115
Edges	78	613
Features	Null	Null
Classes	2	12
Training nodes	4	12
Testing nodes	30	103

.

5.2. Implementation Details and Evaluation Metrics

We implemented our work with Numpy. The first order and fractional order backward propagations are manually derived rather than the auto-gradient computation, such as PyTorch and TensorFlow. FGNN and GNN are trained on these citation networks for 5 thousand epochs with a learning rate of $\eta =0.006$. Both FGNN and GNN contain two graph convolution layers and 128 neurons. We will save the model parameters if the model achieves better recognition accuracy in the validation set. The performance of FGNN with different fractional orders ($\nu \in \{0.5,0.6,\cdots ,1.7\}$) is verified.

For quantitative assessment, recognition accuracy is used as the main evaluation metric for all baseline models and our proposed model,

$$\mathrm{Accuracy}=\frac{\mathrm{TP}+\mathrm{TN}}{\mathrm{TP}+\mathrm{TN}+\mathrm{FP}+\mathrm{FN}},$$

(41)

where TP, FP, TN, and FN are the number of true positives, false positives, true negatives, and false negatives, respectively.

To describe the performance difference between GNN and FGNN in more detail, we adopt the area under curve (AUC), which is defined as the area of the receiver operating characteristic curve (ROC), as the evaluation metric.

5.3. Comparative Methods

Seven methods are selected as the baseline models, i.e., label propagation (LP) [41], semi-supervised embedding (SemiEmb) [42], manifold regularization (ManiReg) [43], skip-gram based graph embeddings (DeepWalk) [44], iterative classification algorithm (ICA), Planetoid [36], and vanilla graph neural network (GNN) [6]. The recognition performance of the first six methods is cited from [6], while vanilla GNN and our proposed FGNN are manually calculated utilizing no optimizers, only employing the first order and fractional order gradient descent algorithms.

5.4. Accuracy Comparison on Citation Network

A group of experiments verify our proposed FGNN on the citation network datasets. To investigate the impact of fractional orders on FGNN, we also carried out experiments on FGNN with different orders. Finally, we discuss the influence of L2 regularization on model performance.

Table 5. Recognition accuracy (%) of the baseline models and FGNN on the citation network dataset Cora, Citeseer, and Pubmed.

Method	ManiReg	SemiEmb	LP	DeepWalk	ICA	Planetoid	GNN	FGNN (Ours)
Cora	59.5	59.0	68.0	67.2	75.1	75.7	79.9	80.9
Citeseer	60.1	59.6	45.3	43.2	69.1	64.7	69.4	70.0
Pubmed	70.7	71.1	63.0	65.3	73.9	77.2	79.2	79.4

presents the recognition accuracy of the baseline models and FGNN with order $\nu =1.3$ on Cora, Citeseer, and Pubmed. FGNN obtains the best performance against existing models, achieving 80.9%, 70.0%, and 79.4%, respectively. Note that Cora and Citeseer have seven and six categories with the higher feature dimension and the lower graph size, while Pubmed has three categories with the lower feature dimension and the higher graph size. A total of 140 and 120 nodes are trained for the first two datasets, while 60 nodes are trained for Pubmed. Our proposed FGNN is better at handling the higher feature dimension and more training samples and has 1.0–21.4% and 0.6–9.9% improvements, respectively. For Pubmed, FGNN has a slight performance gain of 0.2–8.7% with limited training samples and feature dimensions.

Figure 5 shows the recognition accuracy of FGNN with different orders $\nu \in [0.5,\cdots ,1.7]$ on Cora, Citeseer, and Pubmed. With the order increasing from 0.5 to 1.0, the recognition accuracy of FGNN increases from 53.9%, 25.4%, 75.5% to 79.9%, 69.3%, 79.2%. With the order increasing from 1.3 to 1.7, the recognition accuracy of FGNN decreases from 80.9%, 70.0%, 79.4% to 77.3%, 38.5%, 36.5%. For Cora, FGNN has the best performance of 80.9% with $\nu =1.3$ and achieves a gain of 1.0% than GNN; for Citeseer, FGNN has the best performance of 70.4% with $\nu =1.2$ and achieves a gain of 1.1% than GNN; for Pubmed, FGNN has the best performance of 79.4% with $\nu =1.2,1.3$ and achieves a gain of 0.2% than GNN.

To illustrate the effectiveness of L2 regularization, the recognition accuracy of GNN and FGNN with or without L2 regularization has been shown in Figure 6. On these three citation network datasets, L2 regularization plays a significant role in improving the recognition performance for both GNN and FGNN. For GNN, it improves 9.6%, 0.3%, and 6.2% on Cora, Citeseer, and Pubmed. For FGNN, it improves 3.3%, 0.7%, and 6.0% on Cora, Citeseer, and Pubmed. Furthermore, whether L2 regularization is utilized or not, the recognition accuracy of the proposed FGNN outperforms vanilla GNN.

It is worth mentioning that we did not tune the network structure and parameters against specific datasets, and thus further fine-tuning may lead to even greater performance gains. However, this is beyond the scope of this paper.

5.5. Convergence Analysis on Citation Network

A group of experiments verify the MSE loss and recognition accuracy curves of vanilla GNN and our proposed FGNN, then describe ROC curves and AUC values, and further visualize node features of FGNN.

Table 6. Convergence time of GNN and FGNN on citation network datasets.

Method	Cora	Citeseer	Pubmed
GNN	16.08 s	34.62 s	25.13 s
FGNN	11.83 s	14.25 s	15.27 s

shows the convergence time of GNN and FGNN on citation network datasets. FGNN converges after 11.83 s, 14.25 s, and 15.27 s on Cora, Citeseer, and Pubmed, respectively. Meanwhile, GNN converges after 16.08s, 34.62 s, and 25.13 s on Cora, Citeseer, and Pubmed, respectively. These experimental results show that FGNN has a faster convergence time than vanilla GNN. From a theoretical perspective, the approximate fractional order derivative has a similar computational complexity to the first order one. Meanwhile, the approximate fractional order derivative has a greater iterative step than the first order one under the same learning rate. Therefore, it has been proven experimentally and theoretically that FGNN converges faster than vanilla GNN.

Figure 7 shows the MSE loss and recognition accuracy curves of GNN and FGNN on Cora and Citeseer. The horizontal axis denotes training epochs. The left vertical axis denotes the MSE loss. The right vertical axis denotes the recognition accuracy. With the training epoch increasing, the MSE loss of GNN and FGNN have decreased, while the recognition accuracy has increased until they converge to the extreme point. Furthermore, FGNN has a faster convergent speed and achieves higher performance than vanilla GNN.

Figure 8 shows the ROC curves of GNN and FGNN on Cora and Citeseer. The horizontal axis denotes the false positive rate. The vertical axis denotes the true positive rate. The curve of FGNN is closer to the “upper left” than GNN on Cora and Citeseer, so the proposed FGNN has greater AUC value and recognition performance. Particularly, FGNN achieves the AUC value of 0.97 and 0.93, while GNN achieves the AUC value of 0.96 and 0.91.

Figure 9 demonstrates dimensionality reduction visualization results of node features of the input layer, hidden layer, and output layer of FGNN on Cora, Citeseer, and Pubmed. Nodes with different colors represent different categories, and the distance of each point is the 2D visualization of node features with the t-SNE (t-distributed Stochastic Neighbor Embedding) method. Visualization results show that various types of nodes in the input layer are mingled in the space and cannot be directly classified. As nodes are embedded in the graph, nodes from the same category gradually converge, and the chaos of various nodes in the hidden layer is relatively reduced. In the output layer, all kinds of nodes are clustered near the center of the category, but there are still some nodes in the cross-region of different clusters, the embedding and information aggregation of which will be the key to further improving the recognition rate.

5.6. Accuracy Comparison on Community Network

To further illustrate the scalability of the proposed model, we evaluate GNN and FGNN on the community network datasets.

Table 7. Recognition accuracy (%) of GNN and FGNN on community network datasets.

Method	Karate Club	Football
GNN	93.33	51.46
FGNN	96.67	61.17

shows the recognition accuracy of GNN and FGNN on the community network datasets. Compared with Karate club, the node classification task on Football is more difficult because it has twelve categories, which is larger than Karate club’s three. FGNN achieves the recognition accuracy of 96.67% and 61.17% and has the performance gains of 3.34% and 9.71% over the vanilla GNN on Karate club and Football, respectively. Therefore, the proposed FGNN achieves great scalability on five datasets from different domains—citation network and community network.

6. Discussion

Although the proposed FGNN achieves promising performance, there are some potential limitations. First, the comparative results are calculated by the first order and fractional order gradient descent methods utilizing no optimizers. The advanced optimizer could be beneficial to improving model performance. Second, the proposed model only contains two graph convolution layers because of the limitation of the graph over smoothing. The deep graph model should be exploited.

7. Conclusions

In this paper, we generalize the fractional order gradient descent method to the regularized graph neural network. In order to address the local optimality and high complexity problem of fractional order GNNs, we propose an approximate fractional order mechanism to underpin GNN. Then we further prove the feasibility and unbiased property of such approximation towards the first order optimization. Theoretical analysis can promise the convergence of the proposed FGNN. Experiments on node classification tasks demonstrate the performance of the proposed FGNN—it achieves a greater advantage over the baseline models and faster convergent speed than vanilla GNN.