Interpreting Deep Graph Convolutional Networks with Spectrum Perspective

Abstract

Graph convolutional network (GCN) architecture is the basis of many neural networks and has been widely used in processing graph-structured data. When dealing with large and sparse data, deeper GCN models are often required. However, the models suffer from performance degradation as the number of layers increases. The mainstream attribution of the current research is over-smoothing, and there are also gradient vanishing, training difficulties, etc., so a consensus cannot be reached. In this paper, we theoretically analyze the degradation problem by adopting spectral graph theory to globally consider the propagation and transformation components of the GCN architecture, and conclude that the over-smoothing problem caused by the propagation matrices is not the key factor for performance degradation. Afterwards, in addition to using conventional experimental methods, we proposed an experimental analysis strategy under the guidance of random matrix theory to analyze the singular value distribution of the model weight matrix. We concluded that the key factor leading to the degradation of model performance is the transformation component. In the context of a lack of consensus on the problem of model performance degradation, the paper proposes a systematic analysis strategy, as well as theoretical and empirical evidence.

1. Introduction

In recent years, graph neural networks (GNNs) [1] have exhibited outstanding performance in processing graph-structured data, and graph convolutional networks (GCNs) [2] have emerged as the most popular and widely used due to their efficiency, scalability, and ease of application in various domains such as social networks [3], chemical molecules [4], etc. Furthermore, GCN serves as the fundamental building block of numerous complex models [5,6,7]. Although GCN has been highly successful, the increasing need to process large, sparse, and non-linearly separable graphs, as well as higher demands from various applications, has created the requirement for models that can aggregate multi-hop neighbors at a farther distance and improve representation power with a larger number of parameters. Specifically, when processing large and sparse graph-structured data, the limitations of shallow GCN models become more apparent, primarily manifested as an inability to access and aggregate information from more distant neighbors and insufficient parameters, which restricts their feature extraction ability. Deep GCN models have potential advantages in perceiving more graph-structured information and obtaining higher expressive power, and are one of the current important research topics. Many real-world applications involve large-scale graph data, such as social networks, recommendation systems [8], bioinformatics, and transportation networks. In these applications, deep GCN models can better capture the complex relationships between nodes, thereby improving the effectiveness of graph data analysis. In addition, deep GCN models can also be applied to some fields that require the processing of time-series data, such as traffic prediction and financial risk prediction. Therefore, the motivation for studying deep GCN models is to improve the efficiency and accuracy of large-scale graph data analysis and achieve more extensive applications. It is worth noting that deep residual networks for image classification tasks can have up to 152 layers [9], while the Transformer model for natural language processing can have up to 1000 layers [10]. The studies have provided evidence that deeper architectures can enhance the representation power of models. Consequently, it motivates us to investigate deeper GCN models to address the requirements.

However, current research indicates that deep GCN models often suffer from performance degradation as depth increases [2]. In practical applications, shallow GCN models with only 2 to 4 layers typically achieve the best generalization performance. By comparing the node classification accuracy of the two most representative GCN architectures, vanilla GCN and its variant ResGCN [11], the degradation phenomenon of the deep model can be observed in detail. As shown by the solid curves in Figure 1, on different datasets, as the depth of the GCN models increases, their testing set accuracy continues to decline from the two-layer model, and ResGCNs always maintain a stable and very slow decline. The results indicate that although ResGCNs exhibits better generalization performance compared to vanilla GCNs, ResGCNs cannot consistently improve the generalization performance as the model depth increases. While recent studies have proposed more powerful and deeper models that can overcome the degradation problem, achieving effectiveness often requires strong assumptions or tuning many parameters specifically for a given dataset [6,7]. Therefore, identifying the key factors that cause model performance degradation is crucial in guiding the development of models that can enhance generalization performance, which remains a significant challenge.

The degradation of the model’s performance as it goes deeper is widely attributed to “over-smoothing” [12,13,14,15,16,17], which is reflected on the model output as the indistinguishability of node representations. Current theoretical studies on the over-smoothing problem focus on the low-pass filter effect of the graph, which refers to the phenomenon that after multiple graph convolution operations (i.e., propagation operations), the augmented normalized Laplacian matrix of the model converges mathematically in the spectral space [5,18,19]. Specifically, the node representations approach an invariant subspace determined by node degree information alone. This is consistent with the experimental observation that the outputs of the model have indistinguishable properties. Based on the understanding, they have proposed strategies to mitigate or address the problem of over-smoothing. Meanwhile, some studies attribute the degradation problem to aspects such as gradient vanishing and training difficulty [11,20,21], and have proposed corresponding optimization strategies. Despite the effectiveness of these strategies in enhancing model representation power, there is still no consensus on the understanding of the degradation problem. Furthermore, current experimental studies on the degradation problem of deep GCN models tend to focus on shallow layers with less than 10 layers, which is insufficient for studying the problem comprehensively. As a result, in the absence of consensus, we conducted both theoretical and experimental investigations into the degradation of model performance with increasing numbers of layers.

In this paper, we specifically conduct research from the following aspects: 1. The GCN architecture includes not only the graph convolution aggregation operation (i.e., propagation operation) affected by the augmented normalized Laplacian matrix, but also the transformation operation affected by the weight matrix. However, the weight matrix is often ignored in investigations of the over-smoothing problem. We conduct a theoretical analysis of both operations of GCN from a global perspective. 2. As the key factors leading to model performance degradation still have no consensus, we continue to investigate this problem by conducting comparative experiments between deep GCN models, designed reasonably. Motivated by the aforementioned thoughts, the contributions of this paper are as follows.

Firstly, we integrate the aforementioned studies on graph signals and extend the findings on the over-smoothing problem under reasonable assumptions. Rather than adopting the theoretical analysis perspective of inertial decoupling, we analyze the changing trend of graph signals in the spectral space from a unified global perspective, combined with transformation operations. Our analysis shows that the GCN architecture naturally avoids the over-smoothing problem and does not undergo the process of converging to the invariant subspace. Additionally, we show that the random noise in the graph signals have a decreasing impact on the model after passing through the low-pass filter as the number of layers increases.

Secondly, in addition to using conventional experimental methods such as comparing node classification accuracy, we propose an experimental analysis strategy to analyze the singular value distribution of the model weight matrix under the guidance of random matrix theory (RMT). The experimental results explicitly show the process of multiple transformation operations leading to the gradual capture of less information by deep models. This enables us to better understand how the representational power of the model degrades as the number of layers increases. The results indicate that the transformation component in the GCN architecture is a key factor leading to the degradation in model performance. This lays the foundation for our subsequent research.

Overall, our analytical theory is well-suited to explain the degradation of deep GCNs in multi-layer architectures. To further support our theoretical analysis, we employ more angles of experimental analysis strategies to confirm the theoretical analysis conclusions in deeper models.

2. Related Work

Research suggests that a deeper model with increasing complexity can improve accuracy in computer vision or natural language processing tasks [22,23,24,25]. This has been demonstrated through the Weisfeiler–Lehman (WL) graph isomorphism test, which shows that deep models have a better capacity to distinguish subgraphs compared to shallow models [21]. Additionally, ref. [26] conducts an image classification experiment and finds that the model with the largest number of parameters, reaching 550 M, achieved the highest top-1 classification accuracy. Research has shown that increasing the depth and complexity of models can improve the accuracy of computer vision or natural language processing tasks. However, in our research, we have extended the problem to the graph domain, which involves operating on structured data represented as graphs and learning messages through the iterative structure of the graph. However, we have encountered several difficulties in this process. Firstly, graphs have two types of information: topological information and node information, which differ from the information handled in computer vision or natural language processing. Secondly, the performance degradation phenomenon exhibited by deep graph neural networks is unique. There is no consensus on the attribution of the degradation phenomenon in deep models, and it remains an area for exploration. We will provide specific examples below. Finally, many studies have addressed deep problems in a framework that corresponds to shallow models [27,28]. However, analyzing the degradation of model performance in shallow models cannot solve the problems in deep models.

It is known that the single-layer GCN is decomposed into three operations (components): propagation operation (i.e., graph convolution aggregation operation) with augmented normalized Laplacian matrix, transformation operation with weight matrix, and non-linear operation with ReLU activation function. We extend the setting to a multi-layer GCN for a semi-supervised node classification task [2], which requires the model to learn a hypothesis to extract node features and topology information from the graph and to predict the labels of nodes.

Firstly, the lack of consensus in research on the degradation phenomenon is mainly manifested in the controversial and even contradictory attribution of the cause of degradation. For example, ref. [13] proposed a GPR-GNN model with Generalized PageRank techniques to trade-off node features and topological features of the graph, thus preventing the over-smoothing issue of node representations. Ref. [29] shows that anti-over-smoothing processes can occur via GCN models during transformation operations, and they suggest that overfitting is a major contributor to model deterioration. Ref. [30] further refutes the insights such as overfitting and gradient vanishing, and asserts that it is the weight matrix multiplication. Secondly, there is a diversity in optimization strategies, and their effectiveness is limited. If the problem is attributed to a specific type of propagation operation, transformation operation, or non-linear operation, it would be reasonable to propose a decoupling structure to solve the problem, such as increasing propagation operations and reducing transformation operations. However, these optimization strategies are often controlled by various hyperparameters (such as learning rate, number of training iterations, etc.), output at each layer, gradient distribution, and other settings. However, the analysis remains insufficiently thorough and governed by the setting of various hyper-parameters (i.e., the learning rate, iterations of training, etc.), the distribution of each layer output, gradient distribution, and other settings. For instance, ref. [19] shows that the representation capacity of the model cannot be improved with the increasing number of layers and nonlinear operations. Ref. [5] proposed a SGC model, which consists of a fixed low-pass filter followed by a linear classifier, to remove the excessive complexity caused by the nonlinear activation function and excessive weight matrix multiplication. At the same time, the experimental evaluation shows that this variant will not hurt the classification accuracy. Actually, these research ideas violate the fundamental needs of the continuous deepening of neural networks. The architectures have strict requirements on data sets, which are required to be linearly separable, and also have strict restrictions on the assumptions of the model itself. Processing real sparse big data generally does not meet these conditions. The research on the phenomena that a modeląŕs performance degrades as its depth increases is still very controversial, as the above summary makes clear. If there is consensus on the underlying factors of this issue, that would be helpful. We need newer perspectives, more detailed experiments, and a more systematic evaluation inductive inspection of this problem.

3. Preliminary

3.1. Notation

We consider an undirected graph $\mathcal{G}=(\mathcal{V},\mathcal{E},\mathbf{X}),$ where $\mathcal{V}$ is the node set with $\left|\mathcal{V}\right|=N$. Each node $v_i\in \mathcal{V}$ has a d-dimensional feature vector ${\mathbf{x}}_{\mathbf{i}}\in {\mathbb{R}}^d$, and $\mathbf{X}=\{{\mathbf{x}}_{\mathbf{1}},{\mathbf{x}}_{\mathbf{2}},\cdots ,{\mathbf{x}}_{\mathbf{n}}\}$. $\mathcal{E}$ is the edge set with $\left|\mathcal{E}\right|=M$, and $(v_i,v_j)\in \mathcal{E}$. Let $\mathbf{A}\in {\mathbb{R}}^{N\times N}$ denote the adjacency matrix of $\mathcal{G}$, where $A_{ij}=1$ if $(v_i,v_j)\in \mathcal{E}$ and 0 otherwise, and $\mathbf{D}\in {\mathbb{R}}^{N\times N}$ denote the degree matrix of $\mathcal{G}$, where $D_{ii}={\sum }_{j\in \mathcal{V}}{A}_{ij}$ and 0 otherwise.

3.2. Single-Layer Graph Convolutional Network

We first demonstrate a number of settings and inherent properties of a single-layer GCN architecture, and then extend it to multi-layer GCNs architecture. As originally developed by [2,31], the vanilla GCN is a Chebyshev polynomial-based one-order approximation of spectral GCN. The Laplacian matrix of graph $\mathcal{G}$ is defined as $\mathbf{L}=\mathbf{D}-\mathbf{A}\in {\mathbb{R}}^{N\times N}$, and the normalized Laplacian matrix is ${\mathbf{L}}_{\mathbf{sym}}:={\mathbf{I}}_N-{\mathbf{D}}^{-\frac{1}{2}}\mathbf{A}{\mathbf{D}}^{-\frac{1}{2}}$. We set the augmented propagation matrix as $\mathbf{P}={\mathbf{I}}_{\mathbf{N}}+{\mathbf{D}}^{-\frac{1}{2}}\mathbf{A}{\mathbf{D}}^{-\frac{1}{2}}={\tilde{\mathbf{D}}}^{-\frac{1}{2}}\tilde{\mathbf{A}}{\tilde{\mathbf{D}}}^{-\frac{1}{2}}$, where the adjacency matrix augmented with self-loops as $\tilde{\mathbf{A}}=\mathbf{A}+{\mathbf{I}}_{\mathbf{N}}$, $\tilde{\mathbf{D}}=\mathbf{D}+{\mathbf{I}}_{\mathbf{N}}$. We define the augmented normalized Laplacian ${\tilde{\mathbf{L}}}_{\mathbf{sym}}={\mathbf{I}}_{\mathbf{N}}-{\tilde{\mathbf{D}}}^{-\frac{1}{2}}\tilde{\mathbf{A}}{\tilde{\mathbf{D}}}^{-\frac{1}{2}}$, thus

$$\mathbf{P}={\mathbf{I}}_{\mathbf{N}}-{\tilde{\mathbf{L}}}_{\mathbf{sym}}.$$

(1)

The single-layer GCN architecture is formulated as the following:

$$f\left(\mathbf{X}\right)=\sigma \left(\mathbf{P}\mathbf{X}\mathbf{W}\right),$$

(2)

where $\sigma (·)$ is ReLU activation function, i.e., $ReLU(·)=max(0,·)$ and $\mathbf{W}$ is the weight matrix of transformation operation, which can be thought of as the standard multi-layer perceptron (MLP)’s learnable parameters.

3.3. Multi-Layer Graph Convolutional Networks

Next, we examine an architecture with multiple GCN hidden layers. Let $\left[\mathbf{K}\right]:=\{1,2,\cdots ,K\}$. The multi-layer GCNs architecture is defined as

$$\mathbf{H}=\mathrm{softmax}\left(\sigma (\mathbf{P}\cdots \sigma \left(\mathbf{P}\sigma \left(\mathbf{P}\mathbf{X}{\mathbf{W}}^{\left(1\right)}\right){\mathbf{W}}^{\left(2\right)}\right)\cdots {\mathbf{W}}^{\left(K\right)})\right),$$

(3)

where ${\mathbf{W}}^{\left(k\right)}\in {\mathbb{R}}^{d_k\times d_{k-1}}$ is the k-th layer weight matrix, and $k\in \left[\mathbf{K}\right]$. Figure 2 is a schematic diagram of the Multi-layer Graph Convolutional Networks architecture.

We make the following assumptions about the properties of Graph $\mathcal{G}$.

Assumption 1.

Graph $\mathcal{G}$ is connected, that is $\mathcal{G}$ has 1 connected component. $\mathbf{U}$ is a space of graph Fourier Transform with non-negative eigenvectors, which has orthonormal bases ${\left(e_m\right)}_{m\in \left[M\right]}$.

Assumption 2.

The observed graph signals ${\left({\mathbf{x}}_{\mathbf{i}}\right)}_{i\in \mathcal{V}}$ are composed with true signals and noise signals. The random noise follows a Gaussian distribution $\mathcal{N}(0,{\sigma }^2)$.

Intuitively, the stability of the multi-layer GCNs architecture depends on the largest absolute eigenvalue of its propagation matrix. The following theorem holds under Assumption 1.

Theorem 1

(Laplacian Spectral Property [5,19]). Let ${\mu }_t\le {\mu }_{t-1}\le \cdots \le {\mu }_1$ be the eigenvalue of $\mathbf{P}$, sorted in ascending order. Suppose the multiplicity of the largest eigenvalue ${\mu }_N$ is M, then we have $-1<{\mu }_t,{\mu }_{M+1}<1$, and ${\mu }_M=\cdots ={\mu }_1=1$. Further, $e_m:={\tilde{\mathbf{D}}}^{\frac{1}{2}}{u}_m$ is the eigenvector associated with eigenvalue 1. Under Assumption 1, we have $M=1$; $-1<{\mu }_t,{\mu }_2<1$; ${\mu }_N=1$; $e_1={\tilde{\mathbf{D}}}^{\frac{1}{2}}{u}_1$.

Under Theorem 1, there is a fact that the Laplacian of the graph can generalize the eigenpair (i.e., eigenvalue and eigenvector) through ${\mathbf{L}}_{\mathbf{sym}}u=\lambda \tilde{\mathbf{D}}u$, and the corresponding eigenpair of the augmented normalized Laplacian ${\tilde{\mathbf{L}}}_{\mathbf{sym}}$ is $(\lambda ,{\tilde{\mathbf{D}}}^{\frac{1}{2}}u)$. Further, ${\tilde{\mathbf{L}}}_{\mathbf{sym}}$ has the eigenvalue 0 with eigenvectors ${\tilde{\mathbf{D}}}^{\frac{1}{2}}\mathbf{1}$. Ref. [18] shows that the convolution operation by iteratively multiplying the sugmented propagation matrix corresponds to a low-pass filter. Combining Equation (1), we realize that the graph filter $\mu =h\left(\lambda \right)=1-\lambda$ is the first-order Taylor Approximation of a Laplacian regularized least squares [32]. So, the graph filter with K-th propagation operations is $\mu =h\left(\lambda \right)={(1-\lambda )}^K$.

Let $\mathrm{\Lambda }=diag({\mu }_1,{\mu }_2,\cdots ,{\mu }_t)$. According to spectral graph theory [29], the Laplace Spectral Decomposition of the propagation matrix can be indicated as

$$\mathbf{P}=\mathbf{U}\mathrm{\Lambda }{\mathbf{U}}^{\mathsf{T}}=\sum _{i=1}^t{u}_i{\mu }_i{u}_i^{\mathsf{T}},x=\sum _{i=1}^t{\alpha }_i{u}_i,$$

(4)

so the output after multiple propagation operations is

$$\begin{array}{c}y={\mathbf{P}}^{K}x={\left(\sum _{i=1}^t{u}_i{\mu }_i{u}_i^{\mathsf{T}}\right)}^K\left(\sum _{i=1}^t{\alpha }_i{u}_i\right)=\sum _{i=1}^t{\alpha }_i{\mu }_i^K{u}_i,\end{array}$$

(5)

$$\begin{array}{c}\underset{K\to +\infty }{lim}{\mathbf{P}}^{K}x\propto {\alpha }_1{\mu }_1^K{u}_1={\tilde{\mathbf{D}}}^{\frac{1}{2}}\mathbf{1}.\end{array}$$

(6)

With the foregoing deduction, multiple propagation operations will cause the input graph structure data to converge to invariant subspaces. This has the effect of making node representations more similar and difficult to distinguish, which causes an over-smoothing issue. However, does the over-smoothing really lead to a gradual degradation in model performance as the number of layers increases? We provide our own views through theoretical and empirical investigation.

3.4. Supplementary Baseline Architecture

ResGCN is a variant model of GCN [11], which adds residual connections to solve the gradient vanishing problem. It can guarantee that training can still be completed when the depth of the model is more than 32 layers without taking the model’s performance into account. It is defined as

$${\mathbf{H}}^{(k+1)}=\sigma (\mathbf{P}{\mathbf{H}}^{\left(k\right)}{\mathbf{W}}^{\left(k\right)})+{\mathbf{H}}^{\left(k\right)}.$$

(7)

The ResGCN architecture is specifically described as combining the output of the current layer with the output of the previous layer to create the input of the next layer.

4. Analysis

4.1. Empirical Analysis of Over-Smoothing

In the previous section, we provided theoretical evidence that multiple convolution operations can cause over-smoothing. However, it is currently unclear whether the performance on real graph data is consistent with the theoretical analysis. To address this question, we compare and analyze the relative norms of the output of each layer in shallow and deep GCN models, as well as ResGCN models, to explicitly observe smoothness on the Cora dataset.

In the following, we explicitly show the indistinguishability of the model output (i.e., over-smoothing) and indicate problems. Specifically, the empirical results show that the outputs of the model at each layer have a low-rank structure, indicating that deep GCN models exhibit over-smoothing in practice. However, by observing the effectiveness of deep ResGCN in addressing the over-smoothing problem, we can conclude that convergence is not only related to propagation operations, but also to transformation operations. In previous analyses, transformation operations were often not considered.

We evaluate the model’s degree of convergence during forward pass using

$$con\left(\mathbf{X}\right)=\mathbf{X}-\mathbf{1}{\mathbf{x}}^{\mathsf{T}},where\mathbf{x}=argmi{n}_{\mathbf{x}}\parallel \mathbf{X}-\mathbf{1}{\mathbf{x}}^{\mathsf{T}}\parallel$$

(8)

and $\mathbf{1}{\mathbf{x}}^{\mathsf{T}}$ is a rank-1 matrix [33]. Then, the closer the output of each layer is to it, the greater the degree of convergence $con\left(\mathbf{X}\right)$. As shown in Figure 3 and Figure 4, curves represent the change process of the relative norm for each layer output in different models (shallow and deep). The curves depict the variation of the relative norm for the convergence degree of the output of each layer in different models, with the horizontal axis representing the layer index of the models. The vertical axis represents ${\parallel con{\left(\mathbf{X}\right)}_k\parallel }_{1,\infty }/{\parallel {\mathbf{X}}_k\parallel }_{1,\infty }$, where ${\parallel \mathbf{X}\parallel }_{1,\infty }=\sqrt{{\parallel \mathbf{X}\parallel }_1{\parallel \mathbf{X}\parallel }_{\infty }}$.

Firstly, Figure 3a shows that the shallow GCN model performs normally, with curves staying at a high level and achieving good node classification results. Figure 3a and Figure 4a show that the curve changes from a stable high level to a stable low level, indicating that the deep GCN model does suffer from over-smoothing. This seems to confirm the conclusions of the theoretical derivation in Section 3. Figure 4b demonstrates the superior effectiveness of the deep ResGCN model, which uses residual connection technology, with the relative norm curve changing from a low level to a stable high level, indicating its superior effectiveness in addressing the over-smoothing problem in the forward pass process. As a supplement, Figure 3a,b primarily show that the performance of the shallow ResGCN models, which use residual connection technology, is more stable. Based on the experimental observations above, we propose the following idea: inputting the original graph information into each layer of the model instead of just the first layer, which is considered to be a practical and traditional solution for addressing the convergence problem [34,35]. However, as shown in Equation (7), the ResGCN architecture combines the output of the current layer with the output of the previous layer to create the input for the following layer. This approach is actually different from the traditional view and, by acting on the weight matrix of each layer, has the effect of reducing convergence, which was originally proposed as a strategy for alleviating gradient vanishing.

This makes us comprehend that merely considering the property of propagation operation converging to the invariant subspace is not enough to address the over-smoothing problem. A holistic perspective that takes into account both the propagation and transformation operations, with particular attention on the weight matrix’s existence, is required.

4.2. Graph Noise Signal Analysis

As the propagation operation acts as a low-pass filter obtained through Graph Fourier Transform, it is necessary to investigate the impact of random noise signals on the model before proceeding with further derivations to ensure that the subsequent results are not affected by noise. The conclusion is presented in Theorem 2.

Theorem 2

(Informal Noise Signal Bound). Let $\delta \in (0,1)$, and define Q as the random noise of the signal. Then, with the probability at least $1-\delta$ over the model depth K increasing, we have

$$\parallel {\mathbf{P}}^{K}Q\parallel \le O\left(1/de{g}^{K/2}(\sqrt{2\log (1/\delta )}+1)\right)\mathbb{E}\left[{\parallel Q\parallel }_2\right].$$

(9)

It is known that the observed signal is composed of real signal features and noise signals. The following holds under Assumption 2. Theorem 2 guarantees that the observed signals are probably approximately correct (PAC) [36,37] estimates of the true features excluding the noise signals.

Figure 3. Reflection of how each layer of the shallow GCN (8-layers) and ResGCN models’ output converged on the Cora dataset. The vertical axis represents the relative norm of the degree of conversion. (a) The curve of shallow GCNs remains stable at high points. (b) The curve of shallow ResGCNs remains stable at high points for comparison.

Figure 3. Reflection of how each layer of the shallow GCN (8-layers) and ResGCN models’ output converged on the Cora dataset. The vertical axis represents the relative norm of the degree of conversion. (a) The curve of shallow GCNs remains stable at high points. (b) The curve of shallow ResGCNs remains stable at high points for comparison.

Figure 4. Reflection of how each layer of the deep GCN and ResGCN models’ output converged on the Cora dataset. The vertical axis represents the relative norm of the degree of conversion. (a) The curve of deep GCNs remains stable at low points. (b) The curve of deep ResGCNs remains stable at high points for comparison.

Figure 4. Reflection of how each layer of the deep GCN and ResGCN models’ output converged on the Cora dataset. The vertical axis represents the relative norm of the degree of conversion. (a) The curve of deep GCNs remains stable at low points. (b) The curve of deep ResGCNs remains stable at high points for comparison.

Armed with Lemma 5 in [18], it mainly demonstrates that the filtered noise with a probability at least $1-\delta$ over the choice of $\lambda$ is small enough through an exponential inequality for chi-square distributions [38]. Thus, we theoretically guarantee the probability of the distribution of signals that contain graph topology information. It provides the premise for the subsequent perceptron analysis and constitutes a rigorous analysis process.

The inspiration for the derivation comes from [18], but it should be noted that we only exclude the interference of random noise because we argue that there is node feature information in deterministic noise. Specifically, we argue that the node expression after the filter more closely represents the topology features of the graph. If the distance between the true features and the topology features is directly regarded as noise, then the information in the noise will also contain node features, resulting in an excessive amount of noise being observed. In reality, however, the random noise continuously decreases as the model depth increases (by Theorem 2). As a result, the model will incorrectly believe that it has been overfit to the noise, and will attempt to avoid overfitting in the following operations.

Proof of Theorem 2.

We assume the noise signals to be i.i.d. Gaussian variables with zero-mean and the same diagonal variable ${\sigma }^2$, i.e., $Q\sim \mathcal{N}(0,{\sigma }^2)$. Then, the filter noise

$${\parallel {\mathbf{P}}^{k}Q\parallel }^2=\sum _{\lambda }{(1-\lambda )}^{2k}{\parallel \mathbf{q}\left(\lambda \right)\parallel }_2^2.$$

(10)

According to Lemma 1 in [38], we adopt the exponential inequality of chi-square distributions for any positive c. Through the logarithm of the Laplace transform of ${\parallel \mathbf{q}\left(\lambda \right)\parallel }_2^2/{\sigma }^2-1$, we set

$$\mathbb{P}\left\{\sum _{\lambda }{(1-\lambda )}^{2k}({\parallel \mathbf{q}\left(\lambda \right)\parallel }_2^2/{\sigma }^2-1)\ge 2\sqrt{c\sum _{\lambda }{(1-\lambda )}^{4k}}+2c\right\}\le e^{-c}.$$

(11)

Then, for any $\delta >0$, by substituting $c=\log (1/\delta )$, we have

$$\mathbb{P}\left\{\sum _{\lambda }{(1-\lambda )}^{2k}{\parallel \mathbf{q}\left(\lambda \right)\parallel }_2^2\le \left[\sum _{\lambda }{(1-\lambda )}^{2k}(\sqrt{2\log (1/\delta )}+1)+2\log (1/\delta )\right]d{\sigma }^2\right\}\ge 1-\delta .$$

(12)

To be concrete, we note that $\mathbb{E}[{\parallel Q\parallel }_2^2]={\sigma }^{2}dn$. The well-known Dirichlet energy [39] property for normalized Laplacian ${\mathbf{L}}_{\mathbf{sym}}$ is calculated through

$$\frac{1}{2}\sum _{i,j}^N{A}_{ij}{∥\frac{x_i}{\sqrt{d_i+1}}-\frac{x_j}{\sqrt{d_j+1}}∥}^2=\frac{1}{2}\mathbf{tr}\left(x^{\mathsf{T}}{\mathbf{L}}_{\mathbf{sym}}x\right).$$

(13)

We have $\mathbf{P}[i,j]=\frac{1}{\sqrt{\widetilde{deg\left(i\right)}}\sqrt{\widetilde{deg\left(j\right)}}}$, and ${\sum }_{\lambda }{(1-\lambda )}^{2k}=\mathrm{tr}({\mathbf{P}}^{2k})$ decreases in $\mathcal{O}(1/{\mathrm{deg}}^k)$,

$$\parallel {\mathbf{P}}^{k}Q\parallel \le \mathcal{O}\left(1/de{g}^{k/2}(\sqrt{2\log (1/\delta )}+1)\right)\mathbb{E}\left[{\parallel Q\parallel }_2\right].$$

(14)

□

4.3. Spectrum Analysis and Rethinking Over-Smoothing

In the previous subsections, we suggested analyzing the GCN architecture from a holistic perspective. In this subsection, we rethink the over-smoothing problem by examining the trend of the distance between the node embedding and the invariant subspace after propagation and transformation operations. If the distance does not decrease, it means that the over-smoothing caused by propagation operations has not actually occurred. To address this issue, we introduce Theorem 3.

Remember that $\mu ={\sup }_{i=1,\cdots ,t}\left|{\mu }_i\right|$ is the eigenvalue of $\mathbf{P}$, and $s={\sup }_{j=1,\cdots ,K}\left|s_j\right|$ is the maximum singular values of weight matrices. The distance mentioned in Section 3 differs from the distance used here in that it just takes the propagation operation into account and does not take the transformation operation into account.

Theorem 3.

The distance between node representations of K-th layer ${\mathbf{H}}^{\left(K\right)}$ and invariant subspace ${\mathbf{P}}^k\mathbf{X}$ has an exponential variation that is dependent on $({(s\mu )}^k-1)$ as the number of layers increasing, that is, ${\parallel {\mathbf{H}}^{\left(K\right)}-{\mathbf{P}}^k\mathbf{X}\parallel }_{\mathrm{F}}-{\parallel \mathbf{X}-{\mathbf{P}}^k\mathbf{X}\parallel }_{\mathrm{F}}\le ({(s\mu )}^k-1){\parallel \mathbf{X}\parallel }_{\mathrm{F}}$. Here, ${\parallel \mathbf{X}-{\mathbf{P}}^k\mathbf{X}\parallel }_{\mathrm{F}}$ denotes the distance between the original inputs of the model and the invariant subspace.

Proof of Theorem 3.

We denote the distance between $\mathbf{PX}$ and ${\mathbf{P}}^k\mathbf{X}$ by ${\parallel \mathbf{PX}-{\mathbf{P}}^k\mathbf{X}\parallel }_{\mathrm{F}}$, and the distance between ${\mathbf{H}}^{\left(K\right)}$ and ${\mathbf{P}}^k\mathbf{X}$ by ${\parallel {\mathbf{H}}^{\left(K\right)}-{\mathbf{P}}^k\mathbf{X}\parallel }_{\mathrm{F}}$. By Lemma 1 of [19], we have

$${\parallel \mathbf{P}\mathbf{X}-{\mathbf{P}}^k\mathbf{X}\parallel }_{\mathrm{F}}\le \mu {\parallel \mathbf{X}-{\mathbf{P}}^k\mathbf{X}\parallel }_{\mathrm{F}},$$

(15)

where ${\parallel ·\parallel }_{\mathrm{F}}$ denotes the Frobenius norm, and $\mathbf{PX}$ refers to the node observed features after one propagation operation.

Note that $\sigma$ is a ReLU activation function and acts as a contraction mapping. We have the output of the K-th layer ${\mathbf{H}}^{\left(K\right)}$, which satisfies

$$\begin{array}{cc}\hfill {\parallel {\mathbf{H}}^{\left(K\right)}\parallel }_{\mathrm{F}}=& {\parallel \sigma (\cdots \sigma \left(\mathbf{P}\sigma \left(\mathbf{P}\mathbf{X}{\mathbf{W}}^{\left(1\right)}\right){\mathbf{W}}^{\left(2\right)}\right)\cdots {\mathbf{W}}^{\left(k\right)})\parallel }_{\mathrm{F}}\hfill \\ \hfill \le & \parallel \sigma {(\cdots \sigma \left(\sigma \left({\mathbf{P}}^k\mathbf{X}{\mathbf{W}}^{\left(1\right)}\right){\mathbf{W}}^{\left(2\right)}\right)\cdots {\mathbf{W}}^{\left(k\right)}\parallel }_{\mathrm{F}}.\hfill \end{array}$$

(16)

Combining Equation (16), we have

$$\begin{array}{cc}\hfill {\parallel {\mathbf{H}}^{\left(K\right)}-{\mathbf{P}}^k\mathbf{X}\parallel }_{\mathrm{F}}=& {\parallel \sigma (\cdots \sigma \left(\mathbf{P}\sigma \left(\mathbf{P}\mathbf{X}{\mathbf{W}}^{\left(1\right)}\right){\mathbf{W}}^{\left(2\right)}\right)\cdots {\mathbf{W}}^{\left(k\right)})-{\mathbf{P}}^k\mathbf{X}\parallel }_{\mathrm{F}}\hfill \\ \hfill \le & \parallel \sigma {(\cdots \sigma \left(\sigma \left({\mathbf{P}}^k\mathbf{X}{\mathbf{W}}^{\left(1\right)}\right){\mathbf{W}}^{\left(2\right)}\right)\cdots {\mathbf{W}}^{\left(k\right)}-{\mathbf{P}}^k\mathbf{X}\parallel }_{\mathrm{F}}\hfill \\ \hfill \le & \parallel s_k\sigma {(\cdots \sigma \left(\sigma \left({\mathbf{P}}^k\mathbf{X}{\mathbf{W}}^{\left(1\right)}\right){\mathbf{W}}^{\left(2\right)}\right)\cdots {\mathbf{W}}^{(k-1)}-{\mathbf{P}}^k\mathbf{X}\parallel }_{\mathrm{F}}\hfill \\ \hfill & \cdots \hfill \\ \hfill \le & {∥\left(\prod _{i=1}^K{s}_i\right){\mathbf{P}}^k\mathbf{X}-{\mathbf{P}}^k\mathbf{X}∥}_{\mathrm{F}}\hfill \\ \hfill \le & {\parallel s^k{\mathbf{P}}^k\mathbf{X}-{\mathbf{P}}^k\mathbf{X}\parallel }_{\mathrm{F}}.\hfill \end{array}$$

(17)

Combining Equations (15)–(17), we have

$$\begin{array}{cc}\hfill {\parallel {\mathbf{H}}^{\left(K\right)}-{\mathbf{P}}^k\mathbf{X}\parallel }_{\mathrm{F}}-{\parallel \mathbf{X}-{\mathbf{P}}^k\mathbf{X}\parallel }_{\mathrm{F}}\le & {\parallel s^k{\mathbf{P}}^k\mathbf{X}-{\mathbf{P}}^k\mathbf{X}\parallel }_{\mathrm{F}}-{\parallel \mathbf{X}-{\mathbf{P}}^k\mathbf{X}\parallel }_{\mathrm{F}}\hfill \\ \hfill \le & {\parallel s^k{\mu }^k\mathbf{X}-\mathbf{X}\parallel }_{\mathrm{F}}\hfill \\ \hfill \le & ({\left(s\mu \right)}^k-1){\parallel \mathbf{X}\parallel }_{\mathrm{F}}.\hfill \end{array}$$

(18)

□

We know that the maximum frequency of the propagation matrix $\mu$ is always below and close to 1 [21], and according to Gordonąŕs theorem for Gaussian matrices in [40], the maximum singular value of the weight matrix s is usually greater than 1. Therefore, in general, $s\mu >1$. Some studies impose a strong assumption on the model to make $s\mu <1$ [19].

By Equation (18), we can know that if $s\mu <1$, the distance between the model’s output ${\mathbf{H}}^{\left(K\right)}$ and the invariant subspace ${\mathbf{P}}^k\mathbf{X}$ is less than the distance between the original input $\mathbf{X}$ and the invariant subspace ${\mathbf{P}}^k\mathbf{X}$, and their relative size decreases exponentially with respect to the number of layers. If $s\mu >1$, the upper bound of the relative distance will increase exponentially with respect to the number of layers. As the number of layers increases, the constraints are continuously relaxed, which contradicts the absolute convergence derived in Section 3.3.

Theorem 3 indicates that GCNs working with real-world data do not suffering the over-smoothing problem brought on by multiple propagation operations, as this problem can be resolved by combining multiple transformation operations, while the degradation of model performance is due to multiple transformation operations in high possibility. In fact, we investigate and determine the statistical distribution of s using the Random Matrix Theory in the following section, and then empirically analysis the transformation operation.

5. Experiments

In this section, we aim at researching the relationship between the problem of model performance degradation and multiple transformation operations. The problem of multiple transformation operations arises from the multiplication of weight matrices of each GCN layer, which poses a challenge to the GCN architecture, and is strongly related to the problem of model performance degradation.

5.1. Experiment Setup

We conducted experiments on three widely used real-world benchmark citation network datasets [41], namely Cora, Citeseer, and PubMed, to verify the findings and ideas proposed in our research.

Table 1. Real-world benchmark datasets for node classification task.

Datasets	Type	Nodes	Edges	Classes	Features
CORA	Citation network 1	2708	10,556	7	1433
CITESEER	Citation network	3327	9104	6	3703
PUBMED	Citation network	19,717	88,648	3	500

¹ A citation network is a collection of citing and cited relationships between documents.

provides a statistical summary of the datasets.

We conduct semi-supervised node classification experiments on different datasets and analyze the classification accuracy of models with different depths. The goal is to classify each document into one class. For each dataset, we randomly sample 20 instances for each class as the training set. For example, in the Cora dataset, the size of the training set is 140 (20 × classes), and the size of the testing set is 1000 instances. Each layer of the model’s hidden dimension is set to 64, the dropout rate to 0.1, and we use the Adam optimizer [42] with a learning rate of 0.01 or 0.001.

As the ResGCN architecture guarantees to some extent the gradient updating of weight matrices, it is more conducive for analyzing the impact of transformation operations on the model. Therefore, the ResGCN model is the main experimental object. As a comparative model, we incorporate LayerNorm (LN) technology into the ResGCN model to modify the distribution of the output from each layer. LN [43] has been shown to enhance model performance by optimizing the landscape of gradient updates.

5.2. Analyzing the Singular Value Distribution of the Weight Matrices

We propose an experimental analysis strategy to understand the impact of multiple transformation operations on model performance degradation by analyzing the changes in the singular value distribution of weight matrices as the model becomes deeper.

According to random matrix theory, we analyze the singular value distribution of the weight matrices from a statistical perspective. Theorem 1 in [44] provides theoretical support for the fact that singular value distributions are heavy-tailed. The goal of optimization is to improve this heavy-tailed distribution as much as possible.

As shown in Figure 5 and Figure 6, we observe heavy-tailed distributions in different datasets as the model depth increased. The four subgraphs in Figure 5, for instance, represent the statistical distribution of singular values for the weight matrices of ResGCN models with increasing depths. As can be observed, the singular value distribution gradually becomes more heavy-tailed as the model becomes deeper and multiplies more matrices. This implies that the maximum singular value tends to become much larger than the others and gradually becomes dominant, and the condition number of the model gradually increases, where the condition number $\kappa$ is the ratio of the maximum singular value to the minimum singular value. We are all aware that these factors, such as the singular value distribution and the condition number, are indicative of a model’s complexity. A relatively flat singular value distribution and a smaller condition number are desirable as they promote stronger representation capabilities and deeper layers.

In general, we observe heavy-tailed singular value distributions as the model becomes deeper across different datasets. This often indicates that the model output is only sensitive to a few directions of the input, which is not conducive to enhancing the model’s representation power. Moreover, the condition number tends to increase after multiple transformation operations, which is not conducive to improving performance. This actually explains the degradation problem of ResGCN.

As shown in Figure 7 and Figure 8, we have observed that, on different datasets, the heavy-tailed phenomenon of the singular value distribution in the LN-optimized ResGCN models at different depths is alleviated (i.e., the orange part), and the condition numbers decrease accordingly, indicating an improvement in the model’s representation power. As shown in

Table 2. Statistics on the accuracy of node classification for models with different depths.

Datasets	Models	2 Layers(epoch) 1	4 Layers	8 Layers	16 Layers	32 Layers	64 Layers
Cora	GCN	0.8010(600)	0.8011(600) 3	0.7471(600)	FTT 2	–
	ResGCN	0.8106(600)	0.7993(600)	0.7735(600)	0.7885(600)	0.7793(600)	FTT
	ResGCN+LN	0.7928(600)	0.8010(600)	0.7968(600)	0.7956(600)	0.7959(600)	0.5908(600)
Citeseer	GCN	0.6298(600)	0.6010(600)	0.5745(600)	FTT	–	–
	ResGCN	0.6927(600)	0.6801(600)	0.6994(600)	0.6889(600)	0.6353(600)	FTT
	ResGCN+LN	0.6947(600)	0.6837(600)	0.6792(600)	0.6826(600)	0.6854(600)	0.4421(600)
PubMed	GCN	0.7301(600)	0.7292(600)	0.6748(600)	FTT	–	–
	ResGCN	0.7642(600)	0.7677(600)	0.7581(600)	0.7585(600)	0.7564(600)	FTT
	ResGCN+LN	0.7730(600)	0.7949(600)	0.7709(600)	0.7624(600)	0.7760(600)	0.5824(600)

¹ epoch: Number of iterations required for training. ² FTT: Failure to Train, that is, the model cannot complete training within acceptable iterations. ³ The bold numbers in the table represent the optimal values for each row.

, the longitudinal data comparison reveals that the LN-optimized ResGCN model exhibits the optimal performance (bold value), while the horizontal data comparison reveals that the degradation phenomenon of the LN-optimized ResGCN model, although not completely solved, is alleviated as the depth increases. Figure 9 provides a more detailed view of the improved representation power exhibited by the ResGCN model with LN optimization, particularly on larger datasets (i.e., the right subfigure, PubMed dataset), compared to the ResGCN model without LN optimization.

In general, the ResGCN model with LayerNorm (LN) optimization improves the learning process of weight matrices, reduces the model’s condition number, further enhances the model performance and alleviates the degradation problem. This demonstrates that the transformation operation is indeed a key factor for the degradation problem. We propose to shift attention from the propagation operation in the GCN architecture to the transformation operation when studying the degradation problem.

6. Conclusions

This study mainly analyzes the issue that the generalization performance of Deep GCN models continues to decrease as the number of layers increases.

Firstly, we integrate the existing studies on graph signals under our appropriate assumptions. This is manifested as the smooth behavior of the input data converging to the invariant subspace after multiple propagation operations. These serve as the theoretical basis for studying the degradation issue later.

Afterwards, by comparing the node classification accuracy of the GCN and ResGCN models of different layers, as well as the relative norms reflecting the convergence of the outputs at each layer, we observe that the ResGCN model with residual connections does not suffer from the convergence to the invariant subspace (i.e., over-smoothing) problem. Based on the above research, we conclude from a global perspective in the spectral space by considering the transformation and propagation operations of the GCN architecture that GCN does not suffer from over-smoothing due to multiple propagation operations, as it can be resolved through multiple transformation operations. The degradation of the model is more likely to be affected by the transformation operation.

Finally, the ResGCN model only alleviates the degradation phenomenon, and it fails to realize the ideal situation where the model performance improves with increasing depth. We proposed an experimental analysis strategy to investigate the change in the distribution of singular values of the weight matrices of deep GCN models with increasing depth, with the goal of directly observing the impact of multiple transformation operations on model performance. The experimental results confirmed our theoretical speculations that multiple transformation operations are indeed a key factor leading to the degradation phenomenon of the model performance. Therefore, we suggest shifting attention from the propagation operation to the transformation operation in the GCN architecture when studying the degradation problem.