Modeling Tree-like Heterophily on Symmetric Matrix Manifolds

Abstract

Tree-like structures, characterized by hierarchical relationships and power-law distributions, are prevalent in a multitude of real-world networks, ranging from social networks to citation networks and protein–protein interaction networks. Recently, there has been significant interest in utilizing hyperbolic space to model these structures, owing to its capability to represent them with diminished distortions compared to flat Euclidean space. However, real-world networks often display a blend of flat, tree-like, and circular substructures, resulting in heterophily. To address this diversity of substructures, this study aims to investigate the reconstruction of graph neural networks on the symmetric manifold, which offers a comprehensive geometric space for more effective modeling of tree-like heterophily. To achieve this objective, we propose a graph convolutional neural network operating on the symmetric positive-definite matrix manifold, leveraging Riemannian metrics to facilitate the scheme of information propagation. Extensive experiments conducted on semi-supervised node classification tasks validate the superiority of the proposed approach, demonstrating that it outperforms comparative models based on Euclidean and hyperbolic geometries.

1. Introduction

The prevalence of hierarchical tree-like structures, characterized by power-law distributions, is a ubiquitous phenomenon observed across various real-world applications, encompassing domains from social networks [1,2] to data mining [3] and recommendation systems [4]. This pervasive structural pattern has garnered significant attention within the realm of computer science and network analysis due to its profound implications for comprehending network dynamics, functionality, and evolution [5,6,7].

In recent years, there has been a burgeoning interest among researchers in employing hyperbolic space modeling to elucidate tree structures. In contrast to the conventional Euclidean spaces characterized by zero curvature, hyperbolic spaces, endowed with negative curvature, offer a more nuanced measure of inter-nodal distances within a tree. Moreover, the intrinsic property of hyperbolic space to manifest exponential expansion aligns seamlessly with the exponential proliferation inherent in tree growth dynamics.

The complexities inherent in real-world networks often entail a broad spectrum of structural motifs, encompassing flat, tree-like, and circular substructures, thereby giving rise to heterophily within the network. Heterophily contrasts with homophily, where nodes sharing similar attributes tend to cluster together. As depicted in Figure 1, within the overarching tree-like structure, the diverse properties of local substructures yield a variety of graphs. The left graph shows cluster-forming sub-trees, reflecting homophily, while the right graph exhibits hierarchical sub-trees, indicative of heterophily. Hyperbolic spaces offer a nuanced depiction of hierarchical structures and exponential growth dynamics, whereas Euclidean spaces are valued for their simplicity and intuitive geometric properties. Regardless of whether one opts to model such networks within the framework of hyperbolic or Euclidean spaces, both approaches inevitably encounter challenges related to local distortion, resulting in the inaccurate modeling of distances between nodes.

To mitigate the limitations above, this study seeks to explore a more expressive space that could tolerate structural heterophily. The aim is to encode the information inherent in the graph topology into a continuous embedding space with less distortion, thus enhancing the performance of the downstream node classification task. From a geometric perspective, the quality of the embedding in geometric learning depends on the compatibility between the intrinsic graph structure and the embedding space. In light of this principle, we employ the Riemannian manifold of symmetric positive-definite matrices to embed node representations. As shown in Figure 2, symmetric spaces have a rich structure of totally geodesic subspaces, including flat (Euclidean) subspaces and tree-like (hyperbolic) subspaces, facilitating the representations of various substructures within a continuous space.

In Riemannian geometry, a Riemannian metric is a fundamental concept used to define distances, angles, and other geometric properties on smooth manifolds. Various Riemannian metrics have been proposed to guarantee the geometric properties of a symmetric positive-definite manifold (SPD), including the affine-invariant metric (AIM) [8], log-Euclidean metric (LEM) [9,10], and log-Cholesky metric (LCM) [11]. Equipped with these metrics, many Euclidean methods can be generalized into the domain of the Riemannian manifold.

In this study, we introduce a novel approach termed Riemannian graph convolutional neural network (RGCN) aimed at effectively capturing tree-like heterophily within graphs. RGCN operates on the Riemannian symmetric positive-definite matrix manifold and utilizes pullback techniques to generalize Riemannian metrics, such as LEM and LCM, to reconstruct key components of graph convolutional neural networks. In particular, the pullback technique first maps the embedding from the SPD manifold onto the tangent space, proceeds with the operations of information propagation, and ultimately pulls the resulting embeddings back to the SPD manifold. These information propagation components encompass feature transformation, neighborhood aggregation, and non-linear activation, as detailed in prior work [12]. Specifically, the integration of feature transformation and non-linear activation enriches the expressive capacity of the SPD neural network. Concurrently, the iterative process of neighborhood aggregation updates the node embeddings by transporting neighboring features across the graph topology. Our experimental results on semi-supervised node classification tasks substantiate the superiority of our proposed methodology, consistently surpassing comparative models grounded in Euclidean and hyperbolic geometries. The principal contributions of this research can be outlined as follows:

• Introduction of a graph convolutional neural network framework operating on the Riemannian symmetric positive-definite matrix manifold, facilitating graph embedding with reduced distortion and enhanced expressiveness.
• Development of a comprehensive scheme of information propagation on the symmetric positive-definite matrix manifold through the utilization of pullback techniques for the generalization of various Riemannian metrics.
• Extensive experimental evaluations showing the significant performance enhancements achieved by our proposed RGCN model compared to existing Euclidean and hyperbolic baselines in the context of semi-supervised node classification tasks.

The rest of this paper is organized as follows. In Section 2, we briefly survey the related works about GNNs and Riemannian manifolds of symmetric positive-definite matrices. Section 3 introduces some preliminaries. Section 4 presents the details of our proposed model. In Section 5, experimental results on eight benchmark datasets are shown and analyzed to highlight the benefits of our approach. Finally, we conclude the paper in Section 6.

2. Related Work

2.1. Graph Neural Networks

Contemporary graph neural network (GNN) models commonly embrace the message-passing paradigm [13] to encode node representations, demonstrating significant achievements across tasks such as node classification [12], link prediction [14], and graph classification [15]. Advancements in this domain are typically categorized into two primary branches: spectral approaches [16,17] and spatial approaches [12,18]. Spectral approaches leverage graph spectral theory to define graph convolutional operations. Taking inspiration from [19], which suggests approximating spectral filters via truncated Chebyshev polynomial expansions of the graph Laplacian, ChebNet [17] introduces K-localized convolutions, laying the groundwork for convolutional neural networks on graphs. Expanding upon this, graph convolutional network (GCN) [12] restricts the K-localized convolution to $K=1$, employing multiple layers to implement rich convolutional filter functions. To address both local and global consistency, deep graph convolutional neural network (DGCNN) [20] extends GCN by integrating a convolutional operation with a positive pointwise mutual information matrix. Conversely, spatial approaches directly aggregate neighborhood information around the central node. For example, GraphSAGE [12] introduces a versatile inductive framework that samples fixed-size local neighborhoods and aggregates their features using mean, long short-term memory (LSTM) or pooling mechanisms. Graph attention network (GAN) [21] enhances this aggregation process with attention mechanisms, assigning varying weights to aggregated neighborhoods through self-attention mechanisms. Despite the robust theoretical foundation of spectral-based GCNs, spatial-based GCNs demonstrate superior efficiency, generality, and adaptability. For deeper insights into graph neural networks, numerous comprehensive surveys are available [22,23].

Researchers have observed that numerous graphs, including social networks and biological networks, often manifest a pronounced hierarchical structure [24]. Krioukov et al. [25] emphasized that the strong clustering and power-law degree distribution properties in such graphs can be ascribed to a latent hierarchy. Recent investigations have underscored the remarkable representational efficacy of hyperbolic spaces in modeling underlying hierarchies across diverse domains, such as taxonomies [26,27], knowledge graphs [28,29], images [30], semantic classes [31], and actions [32], yielding promising outcomes. Liu et al. [33] and Chami et al. [34] have proposed hyperbolic graph convolutional networks (HGCNs), extending GCNs to hyperbolic spaces for capturing hierarchical structures in graphs. Recently, a series of GNNs have emerged in these spaces, executing graph convolution on various Riemannian manifolds to accommodate diverse graph structures, such as hyperbolic space on tree-like graphs [25], spherical space on spherical graphs [35], and their Cartesian products [36,37].

2.2. Riemannian Manifold of Symmetric Positive-Definite Matrices

The utilization of symmetric positive-definite (SPD) matrices for data representation has been a topic of extensive investigation, primarily leveraging covariance matrices to capture the statistical dependencies among Euclidean features [38,39]. Recent research endeavors have shifted towards the development of foundational components of neural networks within the covariance matrix space. This includes techniques for feature transformation, such as mapping Euclidean features to covariance matrices using geodesic Gaussian kernels [40], non-linear operations applied to the eigenvalues of covariance matrices [41], convolutional operations employing SPD filters [42], and the Frechét mean [43]. Furthermore, proposals for Riemannian recurrent networks [44] and Riemannian batch normalization [45] have been put forth. In comparison to these prior approaches, our proposal introduces an adaptive framework utilizing the pullback paradigm to construct the information propagation component with both LEM and LCM.

3. Preliminaries and Problem Definition

In this section, we initially introduce the preliminaries and notation essential for constructing an SPD embedding space. Subsequently, we define the problem of semi-supervised node classification on the SPD manifold.

3.1. Riemannian Manifold

A smooth manifold $\mathcal{M}$ extends the concept of a surface to higher dimensions. At each point $\mathbf{x}\in \mathcal{M}$, there is an associated tangent space ${\mathcal{T}}_{\mathbf{x}}\mathcal{M}$, representing the first-order approximation of $\mathcal{M}$ around $\mathbf{x}$, which is locally Euclidean. The Riemannian metric $g_{\mathbf{x}}(·,·):{\mathcal{T}}_{\mathbf{x}}\mathcal{M}\times {\mathcal{T}}_{\mathbf{x}}\mathcal{M}\to \mathbb{R}$ defined on the tangent space ${\mathcal{T}}_{\mathbf{x}}\mathcal{M}$ induces an inner product, enabling the derivation of geometric concepts. The pair $(\mathcal{M},g)$ constitutes a Riemannian manifold. The transition between the tangent space and the manifold is facilitated by the exponential and logarithmic maps, denoted as ${\exp }_{\mathbf{x}}(\mathbf{v}):{\mathcal{T}}_{\mathbf{x}}\mathcal{M}\to \mathcal{M}$ and ${\log }_{\mathbf{x}}(\mathbf{y}):\mathcal{M}\to {\mathcal{T}}_{\mathbf{x}}\mathcal{M}$, respectively. Here, ${\exp }_{\mathbf{x}}(\mathbf{v})$ projects the vector $\mathbf{v}\in {\mathcal{T}}_{\mathbf{x}}\mathcal{M}$ onto the manifold $\mathcal{M}$ at point $\mathbf{x}$, while ${\log }_{\mathbf{x}}(\mathbf{y})$ projects the vector $\mathbf{y}\in \mathcal{M}$ back to the tangent space ${\mathcal{T}}_{\mathbf{x}}\mathcal{M}$. For further elucidation, please consult the mathematical references [46].

3.2. Geometry of SPD Manifold

SPD matrices constitute a subset of the Euclidean space ${\mathbb{R}}^{n(n+1)/2}$, and various well-established Riemannian metrics exist on the SPD manifold. Here, we briefly provide an overview of two such metrics, namely, LEM [9] and LCM [11]. The matrix logarithms $\log :{\mathit{S}}_{++}^n\to {\mathit{S}}^n$ and ${\log }^{lcm}:{\mathit{S}}_{++}^n\to {\mathsf{𝓛}}^n$ are defined as follows:

$${\log }^{lem}(\mathbf{S})=\mathbf{U}\ln (\mathrm{\Lambda }){\mathbf{U}}^{\top },$$

(1)

$${\log }^{lcm}(\mathbf{S})=\varphi (\mathcal{L}(\mathbf{S})),$$

(2)

where $\mathbf{S}=\mathbf{U}\mathrm{\Lambda }{\mathbf{U}}^{\top }$ denotes the eigenvalue decomposition, $\mathbf{L}=\mathcal{L}(\mathbf{S})$ represents the Cholesky decomposition, $\varphi (\mathbf{L})=\lfloor \mathbf{L}\rfloor +\ln (\mathbb{D}(\mathbf{L}))$ signifies a coordinate transformation from the ${\mathsf{𝓛}}_{+}^n$ manifold onto the Euclidean space ${\mathsf{𝓛}}^n$, $\lfloor \mathbf{L}\rfloor$ denotes the strictly lower triangular part of $\mathbf{L}$, and $\mathbf{D}(\mathbf{L})$ represents the diagonal elements. It is noteworthy that, topologically, ${\mathsf{𝓛}}^n\simeq {\mathit{S}}^n\simeq {\mathbb{R}}^{n(n+1)/2}$, as their metric topology stems from the Euclidean metric tensor. Leveraging the matrix logarithm, Arsigny et al. [9] propose LEM via Lie group translation, while Lin et al. [11] introduce LCM based on the Cholesky logarithm, establishing an isometry between ${\mathit{S}}_{++}^n$ and ${\mathsf{𝓛}}_{+}^n$. In this investigation, we posit that LEM and LCM are fundamentally analogous, reflecting a high-level mathematical abstraction.

The Riemannian metric and corresponding geodesic distance under the LEM are expressed as follows:

$$g_{\mathbf{S}}^{\mathrm{lem}}({\mathbf{V}}_1,{\mathbf{V}}_2)=g^E({\log }_{\ast ,\mathbf{S}}^{\mathrm{lem}}({\mathbf{V}}_1),{\log }_{\ast ,\mathbf{S}}^{\mathrm{lem}}({\mathbf{V}}_2)),$$

(3)

$$d^{\mathrm{lem}}({\mathbf{S}}_1,{\mathbf{S}}_2)={\parallel {\log }^{\mathrm{lem}}({\mathbf{S}}_1),{\log }^{\mathrm{lem}}({\mathbf{S}}_2)\parallel }_F,$$

(4)

where $\mathbf{S}\in {\mathit{S}}_{++}^n$, ${\mathbf{V}}_1,{\mathbf{V}}_2\in {\mathcal{T}}_{\mathbf{S}}{\mathit{S}}_{++}^n$ are tangent vectors, ${\log }_{\ast ,\mathbf{S}}^{\mathrm{lem}}(·)$ denotes the differential map of the matrix logarithm at $\mathbf{S}$, $g^E$ represents the standard Euclidean metric tensor, and ${\parallel ·\parallel }_F$ stands for the Frobenius norm.

Similarly, the Riemannian metric and geodesic distance under LCM are defined as

$$g_{\mathbf{S}}^{\mathrm{lcm}}({\mathbf{V}}_1,{\mathbf{V}}_2)={\tilde{g}}_{\mathbf{L}}(\mathbf{L}{({\mathbf{L}}^{-1}{\mathbf{V}}_1{\mathbf{L}}^{-\top })}_{\frac{1}{2}},\mathbf{L}{({\mathbf{L}}^{-1}{\mathbf{V}}_2{\mathbf{L}}^{-\top })}_{\frac{1}{2}}),$$

(5)

$$d^{\mathrm{lcm}}({\mathbf{S}}_1,{\mathbf{S}}_2)=\{\parallel \lfloor {\mathbf{L}}_1\rfloor -\lfloor {\mathbf{L}}_2\rfloor {\parallel }_F^2+\parallel \ln (\mathbb{D}({\mathbf{L}}_1))-\ln (\mathbb{D}({\mathbf{L}}_2)){{\parallel }_F^2\}}^{\frac{1}{2}},$$

(6)

where $\mathbf{S}\in {\mathit{S}}_{++}^n$, ${\mathbf{V}}_1,{\mathbf{V}}_2\in {\mathcal{T}}_{\mathbf{S}}{\mathit{S}}_{++}^n$, ${\mathbf{X}}_{\frac{1}{2}}=\lfloor \mathbf{X}\rfloor +\mathbb{D}(\mathbf{X})/2$, and ${\tilde{g}}_{\mathbf{L}}(·,·)$ denotes the Riemannian metric on ${\mathsf{𝓛}}_{+}^n$, defined as

$${\tilde{g}}_{\mathbf{L}}({\mathbf{X}}_1,{\mathbf{X}}_2)=g^E(\lfloor {\mathbf{X}}_1\rfloor ,\lfloor {\mathbf{X}}_2\rfloor )+g^E(\mathbb{D}{(\mathbf{L})}^{-1}(\mathbb{D}{\mathbf{X}}_1),\mathbb{D}{(\mathbf{L})}^{-1}\mathbb{D}({\mathbf{X}}_2)).$$

(7)

3.3. Problem Definition

In this study, we delve into semi-supervised graph representation learning within the SPD space. For clarity and without loss of generality, we define a graph $\mathcal{G}=(\mathcal{V},\mathcal{E},\mathbf{X})$, where $\mathcal{V}=\{v_1,\cdots ,v_n\}$ represents the node set and $\mathcal{E}=\left\{(v_i,v_j)\right|v_i,v_j\in \mathcal{V}\}$ denotes the edge set. The edges are encapsulated in the adjacency matrix $\mathbf{A}$, where ${\mathbf{A}}_{ij}=1$ if $(v_i,v_j)\in \mathcal{E}$ and 0 otherwise. Each node $v_i$ is characterized by a feature vector $x_i\in {\mathbb{R}}^d$, and matrix $\mathbf{X}\in {\mathbb{R}}^{\left|\mathcal{V}\right|\times d}$ represents the collective features of all nodes. We now formalize the problem at hand.

Definition 1

(Semi-supervised graph representation learning in the SPD space). Given a graph $\mathcal{G}=(\mathcal{V},\mathcal{E},\mathbf{X})$, the objective of semi-supervised graph representation learning in the SPD space is to ascertain an encoding function $\mathrm{\Phi }:\mathcal{V}\to \mathbf{Z}$ that maps each node v to a vector $\mathbf{z}$ within an SPD space. This encoding should encapsulate the intrinsic complexity of the graph structure, leveraging information from a subset of labeled nodes to enable accurate label predictions for unlabeled nodes.

4. SPD Graph Convolutional Networks

Our approach, RGCN, introduces an innovative graph neural network framework constructed on the SPD manifold. Drawing upon the foundation established by HGCN, we conduct graph convolution operations within the substituted Euclidean space and subsequently pull the embeddings back to the SPD manifold. Following the paradigm of GCN and HGNN architectures, RGCN comprises three essential components: feature transformation, neighborhood aggregation, and non-linear activation.

4.1. Mapping from Euclidean to SPD Spaces

RGCN initially projects input features onto the SPD manifold using the exp map. Let ${\mathit{x}}^E\in \mathbb{R}$ represent input Euclidean features, which may be generated by pre-trained Euclidean neural networks. The objective is to devise a transformation that maps these Euclidean features to a point within the SPD space. To achieve this, we learn a linear map that converts the input Euclidean features into a vector of dimension $n(n+1)/2$, which is reshaped to form the lower triangle of an initially zero matrix $\mathit{A}\in {\mathbb{R}}^{n\times n}$. Subsequently, we apply the exponential map to transition the coordinates from the substituted Euclidean space to the original manifold ${\mathit{S}}_{++}^n$. For instance, in the case of LEM, we define a symmetric matrix $\mathit{U}\in {\mathit{S}}^n$ such that $\mathit{U}=\mathit{A}+{\mathit{A}}^{\top }$, followed by the exp map as the inverse of Equation (1):

$${\mathit{Z}}^0={\exp }^{lem}(\mathbf{U});$$

(8)

whereas for LCM, we directly employ the exp map as the inverse map of Equation (2):

$${\mathit{Z}}^0={\exp }^{lcm}(\mathit{A})=\mathcal{S}(\mathrm{\Phi }(\mathit{A})),$$

(9)

where $\mathcal{S}(·)$ represents the inverse of the Cholesky decomposition, $\mathrm{\Phi }(\mathbf{L})=\lfloor \mathbf{L}\rfloor +\exp (\mathbb{D}(\mathbf{L}))$ signifies a coordinate transformation from the Euclidean space ${\mathsf{𝓛}}^n$ onto the ${\mathsf{𝓛}}_{+}^n$ manifold. This one-time mapping process enables input features to operate within the SPD manifold seamlessly.

4.2. Feature Transformation

The feature transformation employed in the standard GCN is utilized to map the embedding space of one layer to the embedding space of the next layer, aiming to capture large neighborhood structures. In our approach, we aim to learn transformations of points on the SPD manifold. However, SPD space lacks the notion of a vector space structure. To address this, we extend the framework provided by HGCN and derive transformations within this space. The core concept is to leverage the matrix exponential (exp) and logarithm (log) maps, enabling us to perform Euclidean transformations using substituted Euclidean subspaces ${\mathit{S}}^n$ or ${\mathsf{𝓛}}^n$. Assuming W is an $n^{\prime }\times n$ weight matrix, we define the SPD linear transformation as follows:

$$\mathit{W}\otimes \mathit{Z}:=\exp (\mathit{W}\log (\mathit{Z}){\mathit{W}}^{\top }),$$

(10)

where both the exp and log maps can be formulated using techniques such as the log-Euclidean metric (LEM) or log-Cholesky metric (LCM).

4.3. Neighborhood Aggregation

Neighborhood aggregation stands as a pivotal operation within GCNs, enabling the capture of intricate neighborhood structures and features. Let us consider that ${\mathit{x}}_i$ aggregates information from its neighbors $({\mathit{x}}_j)j\in \mathcal{N}(i)$ with associated weights $(w_j)j\in \mathcal{N}(i)$. While mean aggregation in Euclidean GCNs computes the weighted average ${\sum }_{j\in \mathcal{N}(i)}{w}_j{x}_j$, an analogous operation in hyperbolic space, known as the Fréchet mean, lacks a closed-form solution. To address this, we propose aggregation within substituted Euclidean subspaces ${\mathit{S}}^n$ or ${\mathsf{𝓛}}^n$ employing an attention mechanism.

In GCNs, attention learns the significance of neighbors and aggregates their information based on their relevance to the central node. Yet, attention on Euclidean embeddings often overlooks the tree-like structure prevalent in many real-world graphs. Thus, we further propose an SPD attention-based aggregation operation. Given SPD embeddings $({\mathit{Z}}_i,{\mathit{Z}}_j)$, we initially map ${\mathit{Z}}_i$ and ${\mathit{Z}}_j$ to substituted Euclidean subspaces ${\mathit{S}}^n$ or ${\mathsf{𝓛}}^n$ to compute attention weights $w_{ij}$ using concatenation and a Euclidean multi-layer perceptron (MLP). Subsequently, we propose SPD aggregation to update node embeddings as follows:

$$w_{ij}={\mathrm{SOFTMAX}}_{j\in \mathcal{N}(i)}(\mathrm{MLP}(log({\mathit{Z}}_i)\parallel log({\mathit{Z}}_j))),$$

(11)

$$\mathrm{AGG}({\mathit{Z}}_i)=\exp (\sum _{j\in \mathcal{N}(i)}{w}_{ij}\log ({\mathit{Z}}_j)).$$

(12)

Similar to Euclidean aggregation, RGCN employs a non-linear activation function, ${\sigma }^{\mathit{S}}(·)$, to learn non-linear transformations. Specifically, RGCN applies the Euclidean non-linear activation in substituted Euclidean subspaces ${\mathit{S}}^n$ or ${\mathsf{𝓛}}^n$, and then, maps back to the SPD manifold ${\mathit{S}}_{++}^n$:

$${\sigma }^{\mathit{S}}(\mathit{Z})=\exp ({\sigma }^E(\log (\mathit{Z}))).$$

(13)

It is worth noting that the exponential and logarithm maps are instantiated by both the log-Euclidean metric (LEM) and log-Cholesky metric (LCM).

4.4. RGCN Architecture

Having introduced all the building blocks of RGCN, we now summarize the model architecture, as illustrated in Figure 3. Given a graph $\mathcal{G}=(\mathcal{V},\mathcal{E})$ and input Euclidean features ${({\mathit{x}}^E)}_{i\in \mathcal{V}}$, the first layer of RGCN maps from Euclidean to SPD space. RGCN then stacks multiple SPD graph convolution layers. At each layer HGCN transforms and aggregates neighbor’s embeddings in the substituted Euclidean subspaces. Hence, the information propagation in an RGCN layer is:

$$\begin{array}{ccc}\hfill & {\mathit{H}}_i^{\ell }=\mathit{W}\otimes {\mathit{Z}}_i^{\ell -1}\hfill & \hfill (\mathrm{feature}\mathrm{transformation})\\ \hfill & {\mathit{Y}}_i^{\ell }=\mathrm{AGG}({\mathit{H}}_i^{\ell })\hfill & \hfill (\mathrm{neighborhood}\mathrm{aggregation})\\ \hfill & {\mathit{Z}}_i^{\ell }={\sigma }^{\mathit{S}}({\mathit{Y}}_i^{\ell })\hfill & \hfill (\mathrm{non-linear}\mathrm{activation})\end{array}$$

(14)

SPD embeddings ${(\mathit{Z})}_{i\in \mathcal{V}}$ of the last RGCN layer can then be used to predict node labels. For the node classification task, we directly classify the nodes on the SPD manifold using the SPD multinomial logistic loss.

5. Experiments

In this section, we present our experimental evaluation to validate the effectiveness of the proposed method and analyze the results.

5.1. Experimental Setup

Datasets. Our evaluation employs several real-world graph datasets, encompassing two tree-like graphs labeled as Disease and Airport, five tree-like heterophily graphs (hyperlink networks of universities including Texas, Wisconsin, and Cornell, as well as webpage graphs discussing related topics such as Squirrel and Chameleon), and two benchmark homophily graphs (Cora, and PubMed). Gromov’s $\delta$-hyperbolicity [34], an index derived from group theory, quantifies the tree-like structure of a graph. A lower $\delta$ value indicates a stronger tendency towards a tree-like structure, i.e., a hierarchical arrangement. Specifically, $\delta =0$ represents a fully tree-like structure. The data’s statistics and hyperbolicity metrics are summarized in

Table 1. Data statistics.

Dataset	#Node	#Feature	#Class	#Edge	$\mathit{\delta }$-Hyperbolicity
Disease	1044	1000	2	1043	0
Texas	183	1703	5	295	1
Wisconsin	251	1703	5	466	1
Cornell	183	1703	5	280	1
Squirrel	5201	-	5	198,493	1.5
Chameleon	2277	-	5	31,421	1.5
PubMed	19,717	500	3	88,651	3.5
Cora	2708	1433	7	5429	11

.

Baselines. We benchmark our proposed model against various baselines: (1) Shallow Euclidean models, specifically MLP; (2) Euclidean GNN models, comprising GCN [12], SGC [47], GAT [21], SAGE [48], GeomGCN [18], GCNII [49], and H₂GCN [50]; and (3) hyperbolic GNN models, namely, HGCN [34], HAT [51], LGCN [52], and HyboNet [53].

Table 2. Comparison of model capabilities regarding tree-like structure modeling, tree-like heterophily modeling, and neighbor interaction. A checkmark (✓) indicates the presence of the capability, while a cross (×) denotes its absence.

Model Type	Model	Tree	Heterophily	Neighbor
Shallow model	MLP	×	×	×
Euclidean GNNs	GCN [12]	×	×	✓
	SGC [47]	×	×	✓
	GAT [21]	×	×	✓
	SAGE [48]	×	×	✓
	GCNII [49]	×	×	✓
	GeomGCN [18]	×	✓	✓
	H2GCN [50]	×	✓	✓
Hyperbolic GNNs	HGCN [34]	✓	×	✓
	HAT [51]	✓	×	✓
	LGCN [52]	✓	×	✓
	HyboNet [53]	✓	×	✓
SPD model	RGCN	✓	✓	✓

presents a comparative analysis of these models, delineating their respective capabilities in terms of global tree-likeness modeling, local heterophily perception, and interactional proficiency with neighboring information.

Experimental Details. We adhere to a consistent data splitting strategy employed in previous studies [18,34]. Specifically, nodes within the Disease category are partitioned into training (30%), validation (10%), and test (60%) sets. For other categories such as Texas, Wisconsin, Cornell, Squirrel, and Chameleon, the node distribution is set at 70%, 15%, and 15%, respectively. However, for Cora and PubMed, we utilize 20 labeled training examples per class. Our methodology closely mirrors the parameter configurations and optimization techniques outlined in the original works.

The implementation of the proposed RGCN is realized using PyTorch and PyTorch Geometric (PyG), a specialized deep learning library tailored for graph-structured data and built upon PyTorch. To ensure equitable model comparisons across datasets, we employ identical data splitting and 10-fold cross-validation procedures, reporting average F1 scores and standard deviation. Specifically, for RGCN, we report optimal results within LEM and LCM, adjusting the following hyperparameters: (1) hidden layer dimension ($dim\in 3,5,7,10,15$), (2) number of propagation layers ($layer\in 2,3,4,5,6$), (3) dropout rate ($dropout\in 0,0.1,0.5,0.7,0.9$), (4) learning rate ($lr\in 0,0.005,0.01,0.05,0.1$), and (5) weight decay ($\in 0,1\times {10}^{-4},1\times {10}^{-3},1\times {10}^{-2},0.1$). RGCN employs early stopping with 100 epochs based on validation set performance. The experiments are conducted on an Intel(R) Xeon(R) Gold 5220 CPU @ 2.20 GHz, Quadro @ RTX 6000 hardware configuration.

5.2. Experimental Results

The proposed RGCN model is initially assessed in the context of node classification to gauge its discriminative capacity across tree-like and grid-like structures.

Table 3. Node classification performance on various $\delta$-hyperbolicity tree-like graphs (F1 score ± std).

Space	Model	Disease	Airport	PubMed	Cora
Euclidean	GCN [12]	69.7 ± 0.4	81.4 ± 0.6	78.1 ± 0.2	81.3 ± 0.3
	GAT [21]	70.4 ± 0.4	81.5 ± 0.3	79.0 ± 0.3	83.0 ± 0.7
	SAGE [48]	69.1 ± 0.6	82.1 ± 0.5	77.4 ± 2.2	77.9 ± 2.4
	SGC [47]	69.5 ± 0.2	80.6 ± 0.1	78.9 ± 0.0	81.0 ± 0.1
Hyperbolic	HGCN [34]	82.8 ± 0.8	90.6 ± 0.2	78.4 ± 0.4	81.3 ± 0.6
	HAT [51]	83.6 ± 0.9	–	78.6 ± 0.5	83.1 ± 0.6
	LGCN [52]	84.4 ± 0.8	90.9 ± 1.7	78.6 ± 0.7	83.3 ± 0.7
	HyboNet [53]	96.0 ± 1.0	90.9 ± 1.4	78.0 ± 1.0	80.2 ± 1.3
SPD	RGCN	96.9 ± 0.9	92.6 ± 1.8	79.4 ± 1.2	80.5 ± 1.5

presents performance comparisons between different models, encompassing those operating within Euclidean and hyperbolic spaces. Notably, models leveraging hyperbolic geometry exhibit substantial performance gains over several comparative models, particularly evident in datasets resembling tree structures, such as the Disease dataset, exhibiting complete tree-like structures when $\delta =0$. This underscores the efficacy of hyperbolic geometry in adeptly capturing hierarchical structures within graphs. As illustrated in the table, the proposed RGCN achieves peak performance on three out of four datasets, displaying slightly lower performance only on Cora, which tends towards Euclidean geometry. This underscores the effectiveness of symmetric positive-definite (SPD) geometry as an adaptive mixed space, encompassing both Euclidean and hyperbolic subspaces, for modeling intricate graphs comprising hierarchical and grid-like structures. Particularly noteworthy is the relative performance enhancements of 13.7% and 2.2% achieved by RGCN compared to methods solely based on Euclidean or hyperbolic geometry, respectively, on the real network Airport with $\delta =1$. In summary, the utilization of SPD geometry by RGCN surpasses individual models grounded in hyperbolic and Euclidean geometries in modeling complex networks, with experimental outcomes validating the effective exploitation of SPD geometric properties in crafting neural network modules, thereby enhancing experimental performance.

Moreover,

Table 4. Node classification performance on heterophily graphs (F1 score ± std).

Dataset	Texas	Wisconsin	Cornell	Squirrel	Chameleon
MLP	80.8 ± 4.7	85.2 ± 3.3	81.9 ± 6.4	63.6 ± 2.1	72.8 ± 1.5
GCN [12]	55.1 ± 5.1	51.7 ± 3.0	60.5 ± 5.3	38.2 ± 1.6	60.4 ± 2.1
SAGE [48]	82.4 ± 6.1	81.1 ± 5.5	75.9 ± 5.0	41.6 ± 0.7	58.7 ± 1.6
GeomGCN [18]	66.7 ± 2.7	64.5 ± 3.6	60.5 ± 3.6	38.1 ± 0.9	60.0 ± 2.8
GCNII [49]	77.5 ± 3.8	80.3 ± 3.4	77.8 ± 3.7	56.6 ± 2.1	67.3 ± 2.4
H2GCN [50]	84.8 ± 7.2	86.6 ± 4.6	82.7 ± 5.2	51.0 ± 4.2	69.5 ± 1.8
HGCN [34]	70.1 ± 3.3	83.2 ± 4.5	79.4 ± 4.4	62.3 ± 1.5	74.9 ± 1.5
HyboNet [53]	72.2 ± 4.9	86.5 ± 4.5	77.2 ± 4.7	69.1 ± 1.6	78.7 ± 0.9
RGCN	89.9 ± 6.6	88.7 ± 3.8	85.9 ± 5.1	75.3 ± 1.4	80.3 ± 0.6

presents the outcomes of graph neural network models predicated on Euclidean, hyperbolic, and symmetric positive-definite (SPD) geometries for node classification tasks on heterophily graphs. Notably, on intricate heterophily graphs, models grounded in hyperbolic geometry (e.g., HGCN and HyboNet) do not consistently surpass MLP. Specifically, hyperbolic models outperform the conventional homophily model GCN across nearly all five graph datasets; nevertheless, in comparison to heterophily models, they demonstrate superior performance solely on the Squirrel and Chameleon graphs, potentially attributable to disparities in specific graph structures.

Given that RGCN comprehensively harnesses the attributes of the SPD manifold, compatible with both Euclidean and hyperbolic geometries, it achieves the most remarkable classification results across all five datasets when compared to all comparative methodologies. In the outcomes pertaining to the initial three datasets, RGCN outshines the Euclidean heterophily graph model H₂GCN, while in the results on the latter two datasets, RGCN’s performance also eclipses that of the hyperbolic model HyboNet. This unequivocally validates the geometric versatility of the SPD manifold and underscores the superiority of the proposed RGCN in modeling representation capabilities.

5.3. Analysis and Discussion

In this subsection, we analyze the sensitivity to hyperparameters regarding the embedding dimension and propagation layer.

For the hidden layer dimension, Figure 4 illustrates that on graphs biased towards hierarchical structures, such as Disease and Airport, optimal performance is attained at larger feature dimensions. Conversely, on graphs biased towards grid-like structures, such as Cora and PubMed, optimal performance is achieved at smaller dimensions, specifically at five dimensions. This variance in representation space dimensions due to geometric structural disparities aligns with the expectations of this study, indicating that the symmetric positive-definite (SPD) space encompasses both Euclidean and hyperbolic subspaces, enabling adaptive encoding of distinct spatial structures.

Regarding the number of propagation layers, as depicted in Figure 5, the challenge of over-smoothing has long impeded graph neural networks from effectively capturing long-distance dependency relationships. Consequently, optimal performance of graph neural network models is typically achieved with fewer layers. The analysis of the number of propagation layers validates this observation. Although the optimal layer settings may vary due to different graph properties, optimal performance is generally attained within four layers, with a risk of over-smoothing when surpassing this threshold.

6. Conclusions

In this study, we systematically reconstructed the components of information propagation in classical Euclidean graph convectional networks, such as linear feature transformations, information aggregation, and non-linear activation functions, to symmetric manifold spaces, specifically symmetric positive-definite matrix spaces. By integrating Riemannian geometry with the log-Euclidean metric (LEM) and log-Cholesky metric (LCM) in pullback techniques, we develop a comprehensive scheme of information propagation on the symmetric positive-definite matrix manifold. Experimental results show that the proposed model outperforms its Euclidean and hyperbolic geometry counterparts on complex network data exhibiting implicit hierarchy. The efficacy of this approach further validates the applicability of deep learning to symmetric manifolds, offering a novel avenue for processing data with intricate structures. Although this study demonstrates the superiority of SPD manifolds over Euclidean and hyperbolic geometries for graph embedding, the neural network operations defined on SPD manifolds are computationally expensive. To enhance the scalability of SPD geometry on large-scale graph data, we will focus on the efficiency optimization of SPD neural networks in the future.