NAGNE: Node-to-Attribute Generation Network Embedding for Heterogeneous Network

Abstract

Heterogeneous network embedding aims to project multiple types of nodes into a low-dimensional space, and has become increasingly ubiquitous. However, several challenges have not been addressed so far. First, existing heterogeneous network embedding techniques typically rely on meta-paths to deal with the complex heterogeneous network. Using these meta-paths requires prior knowledge from domain experts for optimal meta-path selection. Second, few existing models can effectively consider both heterogeneous structural information and heterogeneous node attribute information. Third, existing models preserve the structure information by considering the first- and/or the second-order proximities, which cannot capture long-range structural information. To address these limitations, we propose a novel attributed heterogeneous network embedding model referred to as Node-to-Attribute Generation Network Embedding (NAGNE). NAGNE comprises two major components, the attributed random walk which samples node sequences in attributed heterogeneous network, and the node-to-attribute generation which learns the mapping that translates each node sequence itself from the node sequence to the node attribute sequence. Extensive experiments on three heterogeneous network datasets demonstrate that NAGNE outperforms state-of-the-art baselines in various data mining tasks.

1. Introduction

Network embedding, or network representation learning, has attracted a great deal of attention in recent years [1]. Most of the existing network embedding methods focus on homogeneous networks [2,3,4]. However, real-world networks typically consist of multiple types of nodes and edges, commonly known as heterogeneous information networks (HIN) [5]. Some prevalent instances of such networks include the Facebook entity network, academic networks and movie networks. Using the movie network IMDB as an illustration, it contains three types of nodes, movie, actor and director, with different types of relationships between them.

Over the past decade, a significant amount of research has been conducted to explore mining in heterogeneous networks. Most of these studies’ concepts are based on the idea of meta-paths. A meta-path describes a composite relationship between the involved node types, representing an ordered sequence of node types and edge types defined in the network schema. For example, the relationship between two movies can be revealed by the meta-paths Movie–Actor–Movie (MAM), describing the co-actor relationship, and Movie–Director–Movie (MDM), which signifies that they share the same director. Therefore, a meta-path can be perceived as high-order proximity between two nodes, with different meta-paths conveying semantic meanings. To integrate meta-paths in network embedding, existing techniques usually perform meta-path-guided random walks to sample node pairs for learning node embeddings. For instance, ESim [6] accepts meta-paths as guidance to learn node embeddings for similarity searches. Metapath2Vec [7] proposes meta-path-guided random walks to handle the heterogeneity in HIN. HAN [8] generates node embeddings by aggregating features from meta-path-based neighbors in a hierarchical manner.

Although existing heterogeneous network embeddings have achieved state-of-the-art results in many tasks, such as node classification and link prediction, they still suffer from at least one of the following limitations: (1) It is quite challenging to find an optimal combination of meta-paths. (2) The models do not leverage node attribute information; as the attribute and the topological structure are highly non-linear, capturing this non-linearity makes it difficult to discover the underlying patterns. (3) The existing methods preserve the structured information such as the first- and second-order proximities, but they cannot capture long-range structures.

In light of these limitations, we propose a novel attributed heterogeneous network embedding model that translates node sequences to node attributes, named the Node-to-Attribute Generation Network Embedding (NAGNE) model. To address the limitations of meta-paths mentioned above, we introduce an attributed random walk method based on the node attributes. More precisely, we first apply a type-specific mapping function for each type of node by mapping node attributes into attribute types. Then, we conduct random walks on the network based on the attribute type. When performing random walks, the next node is generated by probabilistically considering the following two steps: (1) selecting a target attribute type based on the current node and (2) sampling the target node from the selected attribute type. To capture long-range structure and non-linearity between node attributes and node topological structures, we cast the network embedding problem as a machine translation problem and determine the mapping from node identity sequences to node attribute type sequences. Specifically, given a mass of sampled node sequences using the attributed random walk, we employ attention mechanisms to encode each node identity sequence into a compressed vector as mentioned in Transformer [9], and then we decode this vector to generate the corresponding node attribute type sequence. By mapping the node sequences to node attribute type sequences, the node attribute information and structural information are seamlessly merged into the latent vectors of hidden layers, which can effectively serve as the embedding of nodes. Since the node structure is preserved in the sequence context vector learned via attention mechanisms, it avoids the rigid assumption of the scope of neighborhood and can flexibly determine the appropriate scope of neighborhood information.

The contributions of this paper can be summarized as follows:

• We propose an attributed random walk method on a heterogeneous network, which can integrate node attributes into the generated node sequence and does not require a predefined meta-path.
• We propose a heterogeneous network neural network model, i.e., NAGNE, for network embedding on a heterogeneous network. NAGNE is able to capture both structural and attribute heterogeneity, and can flexibly determine the appropriate scope of neighborhood information.
• We conduct extensive experiments on several public datasets, and the results show that our model significantly outperforms state-of-the-art network embedding models in various tasks.

2. Related Work

2.1. Graph Neural Network

The goal of graph neural networks (GNNs) is to extend deep neural networks to accommodate arbitrary graph-structured data, which has recently gained substantial attention. GraphSAGE [4] uses a neural network-based aggregator over a fixed-size node neighborhood. GAT [3] incorporates a self-attention mechanism into the aggregation function to measure the impacts of different neighbors and combine them to obtain node embeddings. GGNN [10], on the other hand, treats the aggregated neighborhood information as input to the gated recurrent unit (GRU) of the current timestep, enhancing the aggregator function with a GRU. To address spatiotemporal graphs, GaAN [11] merges the GRU with a gated multi-head attention mechanism. However, all the aforementioned GNNs apply only to homogeneous graphs and cannot be naturally adapted to heterogeneous networks.

2.2. Heterogeneous Network Embedding

Heterogeneous network embedding aims to map nodes in a heterogeneous network into a low-dimensional vector space. This approach has recently attracted increasing research interest due to its focus on preserving meta-path-based structural information. Metapath2vec [7] generates meta-path-based random walks and applies the Skip-Gram [12] model to address heterogeneity in HIN. ASPEM [13] captures multiple aspects of semantic information in HINs. HERec [14] introduces a type constraint strategy to transform a heterogeneous network into a homogeneous graph, then uses DeepWalk [1] to learn node embeddings. HAN [8] employs node-level and semantic-level attention to discern the importance of nodes and meta-paths, capturing the complex structures and rich semantics within a heterogeneous network. HetGNN [15] proposes a model that preserves both first- and second-order proximities. HGT [16] designs node-type- and edge-type-dependent parameters to characterize the heterogeneous attention for each edge. NSHE [17] proposes a novel network schema sampling method to address the bias among different types of nodes.

RHINE [18] introduces two criteria for consistently distinguishing heterogeneous relations into affiliation and interaction relationships. HGNN-AC [19] leverages topological relationships to tackle missing attributes in heterogeneous graphs. HGSL [20] addresses structure learning and GNN parameter learning for classification tasks by considering feature similarity and complex heterogeneous interactions jointly. PT-HGNN [21] employs node- and schema-level pretraining tasks to preserve heterogeneous semantic and structural properties contrastively, suggesting the use of relation-based personalized PageRank for efficient pretraining of large-scale heterogeneous graphs. HeCo [22] employs a cross-view contrastive mechanism to concurrently capture local and high-order structures, extracting positive and negative embeddings from two views. HGCN [23] devises a hierarchical aggregation architecture that evaluates meta-paths within a set limit, identifying and utilizing the most beneficial ones for each target object, thus reducing computational costs. Megnn [24] utilizes heterogeneous convolution to integrate various bipartite subgraphs associated with edge types into a unified, trainable graph structure. SeHGNN [25] precomputes the neighbor aggregation using a light-weight mean aggregator, which reduces complexity by removing overused neighbor attention and avoiding repeated neighbor aggregation in every training epoch. HGMAE [26] is a novel heterogeneous graph masked autoencoder model that develops meta-path masking and adaptive attribute masking to enable effective and stable learning on heterogeneous graphs.

In order to enhance the comparison of NAGNE with previous methods, we select several typical heterogeneous network embedding methods. The comparison is conducted from three aspects: whether they can handle structural heterogeneity, whether they can handle attribute heterogeneity, and whether they require predefined meta-paths. The results are shown in

Table 1. The comparison of different methods.

Method	Heterogeneity		Predefined Meta-Paths
	Structure	Attribute
Metapath2Vec [7]	Yes	No	Yes
HAN [8]	Yes	No	Yes
HetGNN [15]	Yes	Yes	Yes
HGT [16]	Yes	Yes	Yes
HGCN [23]	Yes	Yes	Yes
HGMAE [26]	Yes	Yes	Yes
NAGNE	Yes	Yes	No

.

3. Preliminary

In this section, we give formal definitions of some important terminologies frequently used in this paper.

Definition 1

(Attributed Heterogeneous Network [27,28]). An attributed heterogeneous network is denoted as $\mathcal{G}=(\mathcal{V},\mathcal{E},\mathcal{C})$, and is associated with a node type mapping function $\varphi :\mathcal{V}\to \mathcal{A}$ and an edge type mapping function $\psi :\mathcal{E}\to \mathcal{R}$. $\mathcal{A}$ and $\mathcal{R}$ denote the sets of predefined node types and edge types, with $\left|\mathcal{A}\right|+\left|\mathcal{R}\right|>2$, where $\mathcal{V}=\{v_1,v_2,\cdots ,v_n\}$ is the node set; $\mathcal{E}=\left\{e_{ij}\right\}$ is the edge set, where $e_{ij}$ indicates an edge exists between nodes $v_i$ and $v_j$; $\mathcal{C}=\{{\mathit{X}}_1,{\mathit{X}}_2,\cdots ,{\mathit{X}}_{\left|\mathcal{C}\right|}\}$ is the node attribute set; ${\mathit{X}}_k\in {\mathbb{R}}^{|{\mathcal{V}}_k|\times d_k}$ is the node attribute matrices of node type $A_k\in \mathcal{A}$; ${\mathcal{V}}_k$ is the node set of $A_k$; and $d_k$ is the node attribute dimension for node type $A_k$. The weighted adjacency matrix of $\mathcal{G}$ is denoted by $\mathit{D}\in {\mathbb{R}}^{\left|\mathcal{V}\right|\times \left|\mathcal{V}\right|}$. If $e_{ij}\in \mathcal{E}$, ${\mathit{D}}_{ij}>0$; otherwise, ${\mathit{D}}_{ij}=0$. We use ${\mathit{D}}_i$ to denote the neighborhood of node $v_i$.

Definition 2

(Attributed Heterogeneous Network Embedding). Given an attributed heterogeneous network $\mathcal{G}=(\mathcal{V},\mathcal{E},\mathcal{C})$, attributed heterogeneous network embedding aims to design a model to learn the d-dimensional embeddings $\mathcal{H}\in {\mathbb{R}}^{\left|\mathcal{V}\right|\times d}(d\ll |\mathcal{V}\left|\right)$ that are able to capture the rich structural and node attribute information involved in $\mathcal{G}$.

Definition 3

(Node Attribute Mapping). Given an attributed heterogeneous network $\mathcal{G}=(\mathcal{V},\mathcal{E},\mathcal{C})$, node attribute mapping aims to map each node attribute to a set $T=\{t_1,t_2,\cdots ,t_{\left|T\right|}\}$ of $\left|T\right|$ attribute types, where $\left|T\right|\ll \left|\mathcal{V}\right|$.

Definition 4

(Node-to-Attribute Generation). Let $\mathcal{L}=\{v_1,v_2,\cdots ,v_{\left|\mathcal{L}\right|}\}$ be a sequence of nodes sampled from a network, ${\mathcal{L}}^d=\{{\mathit{D}}_1,{\mathit{D}}_2,\cdots ,{\mathit{D}}_{\left|\mathcal{L}\right|}\}$ and ${\mathcal{L}}^t=\{t_{v_1},t_{v_2},\cdots ,t_{v_{\left|\mathcal{L}\right|}}\}$ be the corresponding node neighborhood sequence and node attribute type sequence respectively. Given a set of node neighborhood sequences ${\mathcal{L}}^D=\{{\mathcal{L}}_j^d,j=1,2,\cdots ,J\}$ and a set of the corresponding attribute type sequences ${\mathcal{L}}^T=\{{\mathcal{L}}_j^t,j=1,2,\cdots ,J\}$, node-to-attribute generation learns a mapping function $f:{\mathcal{L}}_j^d\to {\mathcal{L}}_j^t$ for each ${\mathcal{L}}_j^d\in {\mathcal{L}}^D$ and ${\mathcal{L}}_j^t\in {\mathcal{L}}^T$.

4. Methodology

In this section, we propose a novel network embedding model for attributed heterogeneous networks. Figure 1 illustrates the framework of the proposed NAGNE. The model is constructed by three major components: node attribute mapping, attributed random walks and node-to-attribute generation.

4.1. Node Attribute Mapping

Due to the fact that different types of nodes may lie in different attribute spaces, we apply a type-specific mapping function for each type of node by mapping node attributes into attribute types. Given the node attribute matrices ${\mathit{X}}_k\in {\mathbb{R}}^{|{\mathcal{V}}_k|\times d_k}$ for node type $A_k\in \mathcal{A}$, for one node attribute vector $\mathit{x}\in {\mathit{X}}_k$, we have

$${\mathrm{\Phi }}_k:\mathit{x}\to t$$

(1)

Thus, for each node type $A_k\in \mathcal{A}$, we have a function ${\mathrm{\Phi }}_k$ that maps the $N_k$ nodes of type $A_k$ to a set $T_k=\{t_{k1},\cdots ,t_{k{M}_k}\}$ of $M_k$ attribute types based on the attribute matrix ${\mathit{X}}_k$, where $M_k\ll N_k$. The node attribute mapping aims to map the nodes with similar attributes into the same attribute types. Note that the function $\mathrm{\Phi }$ is a modeling choice, which could be learned automatically or defined manually by the user. In this paper, we map the nodes to their attribute types by using the Non-negative Matrix Factorization (NMF) [29] algorithm. Given the node attribute matrices ${\mathit{X}}_k\ge 0$ for node type $A_k\in \mathcal{A}$, the attribute similarity matrix ${\mathit{P}}^k\in {\mathbb{R}}^{|{\mathcal{V}}_k|\times |{\mathcal{V}}_k|}$ is calculated based on the node attributes. Each element ${\mathit{P}}_{ij}^k$ is the attribute cosine similarity of nodes $v_i$ and $v_j$. NMF aims to find non-negative matrix factors ${\mathit{M}}^k\ge 0$ and ${\mathit{H}}^k\ge 0$ such that

$${\mathit{P}}^k\approx {\mathit{M}}^k{\mathit{H}}^k$$

(2)

where ${\mathit{M}}^k\in {\mathbb{R}}^{|{\mathcal{V}}_k|\times r_k}$ and ${\mathit{H}}^k\in {\mathbb{R}}^{r_k\times d_k}$; $r_k$ is the number of attribute types of $A_k$, which is usually chosen to be smaller than $|{\mathcal{V}}_k|$; and ${\mathit{M}}^k$ is the attribute type indicator matrix, where each element ${\mathit{M}}_{ij}^k$ presents the tendency that the i-th node belongs to the j-th attribute type. Then, we partition ${\mathit{M}}^k$ into $r_k$ disjoint sets based on the values of ${\mathit{M}}^k$.

4.2. Attributed Random Walks

Random walks [1] have received much attention in homogeneous networks. However, as nodes in heterogeneous networks are associated with a dominant number of paths, random walks in heterogeneous networks are biased to highly visible types of nodes. Subsequently, the learned node embeddings are also biased to these types in the sense that they mostly preserve the node proximity in these types. In addition, random walks ignore the attributes of nodes and cannot integrate attributes into the node embeddings. Thus, we propose an attributed random walk method in a heterogeneous network. In our attributed random walks, there are two steps to select the next node in a random walk. Consider an attributed random walk that just resides at node $v_i$, and the next node $v_{i+1}$ is selected via the following steps:

• Selecting a target attribute type t. Sampling one attribute type t from those that have nodes connected to $v_i$ via heterogeneous edges. The candidate set of attribute types for selecting is denoted as $\mathcal{T}\left(v_i\right)=\left\{t\right|\exists v_m,e_{mi}\in \mathcal{E},t_{v_m}=t\}$. According to the node type of $v_i$, $\mathcal{T}\left(v_i\right)$ can be divided into two parts, ${\mathcal{T}}_p\left(v_i\right)$ and ${\mathcal{T}}_n\left(v_i\right)$, which are denoted as

  $${\mathcal{T}}_p\left(v_i\right)=\left\{t\right|t\in \mathcal{T}\left(v_i\right),\forall t_{v_j}=t,A_{v_j}=A_{v_i}\}$$

  (3)

  $${\mathcal{T}}_n\left(v_i\right)=\left\{t\right|t\in \mathcal{T}\left(v_i\right),\forall t_{v_j}=t,A_{v_j}\ne A_{v_i}\}$$

  (4)

  where $A_{v_i}$ is the node type of $v_i$. ${\mathcal{T}}_p\left(v_i\right)$ contains the attribute types of the node which has the same node type as $v_i$, and ${\mathcal{T}}_n\left(v_i\right)$ otherwise. For example, in Figure 2, $\mathcal{T}\left(v_i\right)=\{t_1,t_2,t_3,t_4\}$, ${\mathcal{T}}_p\left(v_i\right)=\{t_3,t_4\}$ and ${\mathcal{T}}_n\left(v_i\right)=\{t_1,t_2\}$.
  To overcome the problem that the node distribution from the random walk sequences are skewed towards these highly visible node types, we first sample one set ${\mathcal{T}}_t\left(v_i\right)$ from $\{{\mathcal{T}}_p\left(v_i\right),{\mathcal{T}}_n\left(v_i\right)\}$ with the following probability:

  $$P({\mathcal{T}}_t\left(v_i\right)={\mathcal{T}}_p\left(v_i\right))=\left\{\begin{array}{cc}1,\hfill & \mathrm{if}{\mathcal{T}}_n\left(v_i\right)=\varnothing ,\hfill \\ {\alpha }^s,\hfill & \mathrm{otherwise}.\hfill \end{array}\right.$$

  (5)

  where $\alpha \in [0,1]$ is an initial probability, which is determined in advance. s is the number of nodes consecutively visited in the same node type of the current node $v_i$. By sampling the attribute type t from ${\mathcal{T}}_p\left(v_i\right)$ with a probability ${\alpha }^s$, and ${\mathcal{T}}_n\left(v_i\right)$ otherwise, we adopt an exponential decay function ${\alpha }^s$ to penalize the case that the walk stays in one node type for too long. After that, we uniformly sample one attribute type t from ${\mathcal{T}}_t\left(v_i\right)$.
• Sampling the target node $v_{i+1}$. Sampling one node from the nodes with the selected attribute type t. The corresponding candidate set of nodes is denoted as $V\left(v_i\right)=\left\{v\right|t_v=t\}$. After selecting a target attribute type t, we can uniformly sample one node as $v_{i+1}$ from $V\left(v_i\right)$ with the following probability:

  $$P\left(v_{i+1}\right|t,v_i)=\left\{\begin{array}{cc}\frac{{\pi }_{v_i{v}_{i+1}}}{Z},\hfill & \mathrm{if}{v}_{i+1}\in V\left(v_i\right),\hfill \\ 0,\hfill & \mathrm{otherwise}.\hfill \end{array}\right.$$

  (6)

  where ${\pi }_{v_i{v}_{i+1}}$ is the un-normalized transition probability between nodes $v_i$ and $v_{i+1}$; in unweighted networks, ${\pi }_{v_i{v}_{i+1}}=1$, Z is the normalizing constant. However, intuitively, the nodes in $V\left(v_i\right)$ that have edges with $v_i$ have more similarities with it, which should make them easier to sample. Figure 2 shows an example of the sampling. In Figure 2, if $t_1$ is the selected attribute type, the two nodes in the red dotted circle should be easier to sample than the others in $t_1$ as they have edges with $v_i$. We use $V_p\left(v_i\right)=\left\{v_m\right|v_m\in V\left(v_i\right),e_{mi}\in \mathcal{E}\}$ to denote the node sets that have edges with $v_i$, and $V_n\left(v_i\right)=\left\{v_m\right|v_m\in V\left(v_i\right),e_{mi}\notin \mathcal{E}\}$ otherwise. Then, we define the un-normalized transition probability in networks as the following:

  $${\pi }_{v_i{v}_{i+1}}=\left\{\begin{array}{cc}\frac{1}{q}·\frac{1}{|V_p\left(v_i\right)|},\hfill & \mathrm{if}{v}_{i+1}\in V_p\left(v_i\right),\hfill \\ \frac{1}{|V_n\left(v_i\right)|},\hfill & \mathrm{if}{v}_{i+1}\in V_n\left(v_i\right).\hfill \end{array}\right.$$

  (7)
  Parameter q controls the search procedure in the walk. As we hope the nodes in $V_p\left(v_i\right)$ are more likely to be sampled, we set $0<q<1$. Note that by generating the next node $v_{i+1}$ based on the attribute type, the attributed random walk samples nodes not only from those connected to $v_i$, but also from the nodes have similar attributes with $v_i$, which allows the random walk to explore both the structural and attribute information of the network. The pseudocode is shown in Algorithm 1.

Algorithm 1 Attributed Random Walks

Require: graph $\mathcal{G}=(\mathcal{V},\mathcal{E},\mathcal{C})$, initial stay probability controlling parameters $\alpha$ and s, random walks parameter q, number of random walks per node r, maximum walk length $L_{\mathit{max}}$ 1: Initialize an empty set of walks $\mathcal{S}=\varnothing$ 2: for iter = 1 to r do 3:    for each node $v_i\in \mathcal{V}$ do 4:      Initialize a random walk $g=\left\{v_i\right\}$ by adding $v_i$ 5:      while $\left|g\right|<L_{\mathit{max}}$ do 6:         Pick one candidate set ${\mathcal{T}}_t\left(v_i\right)$ of attribute types according to Equation (5) 7:         Uniformly sample one attribute type t from the selected attribute types set ${\mathcal{T}}_t\left(v_i\right)$ 8:         Sample a node according to Equation (7) 9:         Update g 10:      end while 11:      Add g to $\mathcal{S}$ 12:    end for 13: end for 14: return The set of random walks $\mathcal{S}$

4.3. Node-to-Attribute Generation

4.3.1. Node Sequence Encoder

Let $\mathcal{L}=\{v_1,v_2,\cdots ,v_{\left|\mathcal{L}\right|}\}$ be a sequence of nodes sampled from a network using attributed random walks; the corresponding nodes neighborhood sequence and node attribute type sequence are ${\mathcal{L}}^d=\{{\mathit{D}}_1,{\mathit{D}}_2,\cdots ,{\mathit{D}}_{\left|\mathcal{L}\right|}\}$ and ${\mathcal{L}}^t=\{t_{v_1},t_{v_2},\cdots ,t_{v_{\left|\mathcal{L}\right|}}\}$, respectively. For each type of node, we apply a type-specific linear transformation to project the neighborhood of different types of nodes into the same latent factor space. For a node $v_m\in A_i$ in node sequence $\mathcal{L}$, we have

$$h_{vm}^{\prime }={\mathit{W}}_{A_i}{\mathit{D}}_m$$

(8)

where $h_{vi}^{\prime }\in {\mathbb{R}}^{d^{\prime }}$ is the projected latent vector of node $v_i$, and ${\mathit{W}}_{A_i}\in {\mathbb{R}}^{d^{\prime }\times n}$ is the parametric weight matrix for node type $A_i$. Then, the sequence of node neighborhood ${\mathcal{L}}^d=\{{\mathit{D}}_1,{\mathit{D}}_2,\cdots ,{\mathit{D}}_{\left|\mathcal{L}\right|}\}$ is projected as

$${\mathcal{L}}^{\prime }=[h_{v_1}^{\prime },h_{v_2}^{\prime },\cdots ,h_{v_{\left|\mathcal{L}\right|}}^{\prime }]$$

(9)

where ${\mathcal{L}}^{\prime }$ is an $\left(\right|\mathcal{L}|,d^{\prime })$ matrix. To capture the global topology information over sequences, NAGNE uses an encoder to encode the sequence of the node neighborhood vector $[h_{v_1}^{\prime },h_{v_2}^{\prime },\cdots ,h_{v_{\left|\mathcal{L}\right|}}^{\prime }]$, as mentioned in Transformer. The encoder is composed of a stack of L identical layers. Each layer has four sub-layers. The output of each layer is obtained from the output of the previous via four sub-layers as follows:

$${\mathit{Z}}^l=RL\circ F\circ RL\circ MH({\mathit{Z}}^{l-1},{\mathit{Z}}^{l-1})$$

(10)

where ${\mathit{Z}}^l\in {\mathbb{R}}^{\left|\mathcal{L}\right|\times d}$ denotes the hidden units of the $l_{th}$ layer and, particularly, ${\mathit{Z}}^0$ is the node sequence neighborhood projecting ${\mathcal{L}}^{\prime }$; $RL$, F and $MH$, respectively, denote the sub-layers of residual connection plus layer normalization, feed-forward network and multi-head attention; and ∘ denotes the composition operator between two functions. $RL$ and F map a vector to another vector and we refer to reader to Transformer for detailed definitions. Multi-head attention allows the model to jointly attend to information from different representation sub-spaces, which is defined as follows:

$$MH(\mathit{Z},\mathit{Z})=\phi ({\mathit{head}}_1,\cdots ,{\mathit{head}}_k)$$

(11)

where $\phi$ is a linear projection function, and

$${\mathit{head}}_i=\mathit{Attention}({\mathit{ZW}}_i^1,{\mathit{ZW}}_i^2,{\mathit{ZW}}_i^3)$$

(12)

where ${\mathit{ZW}}_i^1$, ${\mathit{ZW}}_i^2$, ${\mathit{ZW}}_i^3$ are transformation matrices for head i, and

$${\displaystyle \mathit{Attention}({\mathit{ZW}}_i^1,{\mathit{ZW}}_i^2,{\mathit{ZW}}_i^3)=\mathit{softmax}\left(\frac{{\mathit{ZW}}_i^1{\left({\mathit{ZW}}_i^2\right)}^{⊺}}{\sqrt{d_k}}\right){\mathit{ZW}}_i^3}$$

(13)

The softmax function applied to the scaled dot-product attention scores normalizes them into a probability distribution over the input tokens. It is mathematically defined as

$$\mathit{softmax}{\left(\mathit{x}\right)}_j=\frac{exp\left(x_j\right)}{{\sum }_{k=1}^{K}exp\left(x_k\right)}$$

(14)

Here, $\mathit{x}$ is a vector containing attention scores before normalization, $x_j$ is the j-th element of $\mathit{x}$, and K is the total number of elements in the vector. By exponentiating and normalizing each score, the softmax function ensures that the output is a valid probability distribution, with all elements between 0 and 1 and summing up to 1.

After applying the attention mechanism at each layer, the node’s neighborhood sequence ${\mathcal{L}}^d$ is effectively compressed into the representation ${\mathit{Z}}^L$ through a stack of L identical layers, which each potentially leverage the aforementioned attention computation.

4.3.2. Node Attribute Generation

The representation ${\mathit{Z}}^L$ seamlessly fuses the attribute information and the structural information. Similar to the encoder, the decoder is also composed of a stack of L identical layers. The output of each layer is obtained from the output of the previous layer as follows:

$${\mathit{C}}^l=RL\circ F\circ RL\circ MH(RL\circ MH({\mathit{C}}^{l-1},{\mathit{C}}^{l-1}),{\mathit{Z}}^L)$$

(15)

where ${\mathit{C}}^l$ is the hidden units of the $l_{th}$ layer in the decoding phase and, particularly, ${\mathit{C}}^0$ denotes the embedding of the corresponding node attribute type sequence ${\mathcal{L}}^t$. Note that the encoding phase differs from the decoding phase in an additional multi-head attention mechanism over the hidden unit sequence ${\mathit{Z}}^L$ derived from the encoding phase. After obtaining ${\mathit{C}}^L$, we need to map each vector ${\mathit{c}}_y$ in ${\mathit{C}}^L$ to a $\left|T\right|$-dimension vector ${\mathit{h}}_y$ through a fully connected layer, and then a softmax layer transforms ${\mathit{h}}_y$ into the probabilities ${\mathit{p}}_y$:

$${\mathit{p}}_y=\mathit{softmax}\left({\mathit{h}}_y\right)=\mathit{softmax}(\sigma \left({\mathit{W}}_f{\mathit{c}}_y\right)+{\mathit{b}}_f)$$

(16)

Then, we use a cross-entropy loss to measure the correctness of the translation:

$$\mathit{Loss}=-{\sum }_{j=1}^J{\sum }_{t_{v_i}\in {\mathcal{L}}_j^t}{\sum }_k^{\left|T\right|}\delta (t_{v_i},k){\mathit{p}}_y\left[k\right]$$

(17)

where $\delta (t_{v_i},k)$ is a binary function that outputs 1 if $t_{v_i}$ equals k; otherwise, it outputs 0. ${\mathit{p}}_y\left[k\right]$ denotes the k-$th$ element of ${\mathit{p}}_y$. Then, we use Momentum to optimize the parameters in the model.

5. Experiment

5.1. Experimental Setup

5.1.1. Datasets

• DBLP [18]: A frequently used bibliographic network in heterogeneous network studies. We extract a subset of DBLP containing nodes from three domains, including 2000 authors (A), 9556 papers (P) and 20 conferences (C).
• IMDB [30]: A movie rating website with nodes also in three domains. We composed a heterogeneous network with 4353 actors (A), 3676 movies (M) and 1678 directors (D). Movies were categorized into action, comedy and drama classes asabels based on their genres.
• ACM [30]: This network contains papers from prestigious conferencesike KDD, SIGMOD, SIGCOMM, MobiCOMM and VLDB. Our subset of the ACM includes 3025 papers, 5835 authors and 56 subjects. Paper features are represented through a bag-of-words model of keywords.
• Aminer [31]: Bibliographic graphs. The papers in Aminer areabeled with 17 research fields, e.g., Artificial Intelligence, which are used for node classification. Our subset of the Aminer includes 302402 papers and 520456 authors.

5.1.2. Baselines

• DeepWalk [1]: A homogeneous network embedding model that integrates random walks and the skip-gram to learn network embeddings.
• Metapath2Vec [7]: A heterogeneous network embedding method adopting meta-path-based random walks and a heterogeneous skip-gram model.
• HIN2Vec [32]: A neural network-based approach that learns embeddings through multiple predictive training tasks.
• MAGNN [33]: This model uses node content transformation along with intra- and inter-metapath aggregations to generate node embeddings.
• HAN [8]: This approach utilizes neural networks for heterogeneous networks, generating embeddings by aggregating features from meta-path-based neighbors hierarchically.
• GTN [34]: GTN transforms heterogeneous networks into new graphs using meta-paths and applies convolution to these meta-path-derived graphs to learn node embeddings.
• HGNN-AC [19]: This model transforms a heterogeneous graph into multiple meta-path-defined graphs, learning node representations through convolution and completing attributes for nodes lacking them via weighted aggregation.
• HGSL [20]: HGSL concurrently learns the structure of a heterogeneous graph and the parameters of GNNs for classification tasks.
• Megnn [24]: Megnn merges different bipartite sub-graphs tied to edge types into a novel trainable graph structure using heterogeneous convolution.

5.2. Multi-Label Classification

In node classification, we predict the labels of unlabeled nodes, as performed in previous studies. We designate 80% of labeled nodes as training data to train an SVM classifier, and the remainder for testing. The performances in Micro-F1 and Macro-F1 are displayed in

Table 2. Performance evaluation of multi-label classification.

	DBLP		IMDB		ACM		Aminer
	Micro-F1	Macro-F1	Micro-F1	Macro-F1	Micro-F1	Macro-F1	Micro-F1	Macro-F1
DeepWalk	90.21	89.52	56.53	55.34	82.22	81.77	83.23	82.91
Metapath2Vec	92.50	91.87	52.38	51.15	83.56	82.31	84.43	83.65
HIN2Vec	84.77	83.45	49.57	48.71	55.37	49.18	78.76	77.45
MAGNN	92.92	92.50	59.44	58.63	83.42	82.80	84.65	83.92
HAN	93.31	92.22	58.35	54.22	82.86	81.67	84.27	83.81
GTN	94.16	93.63	61.24	59.45	81.23	80.53	85.31	84.87
HGNN-AC	93.53	92.57	60.96	59.27	80.33	79.66	84.54	83.48
HGSL	93.07	92.56	60.05	59.25	80.22	79.68	84.08	83.45
Megnn	93.35	92.64	60.85	58.91	82.30	81.92	84.37	83.52
NAGNE	95.02	94.32	62.42	60.38	83.86	82.61	85.72	85.13

. Our model, NAGNE, sans prior knowledge, consistently outperforms other baselines across various datasets, affirming its proficiency in multi-label classification tasks.

5.3. Node Clustering

By applying the K-Means algorithm to the derived node embeddings, we cluster nodes into classes, evaluating the clustering outcomes via normalized mutual information (NMI) [35], with results in

Table 3. Normalized mutual information (NMI) on the node clustering task.

	DBLP	IMDB	ACM	Aminer
DeepWalk	66.25	0.41	42.82	42.86
Metapath2Vec	68.74	0.09	42.79	44.98
HIN2Vec	65.84	0.04	42.21	44.86
MAGNN	69.81	0.49	42.36	44.97
HAN	69.12	0.67	41.56	44.41
GTN	69.54	0.70	42.69	45.26
HGNN-AC	68.31	0.64	40.59	43.16
HGSL	67.24	0.59	41.81	44.43
Megnn	66.34	0.47	40.95	43.59
NAGNE	69.76	0.73	43.35	46.63

. NAGNE again surpasses others in most cases, corroborating its effectiveness.

5.4. Visualization

We use the t-SNE [36] algorithm to project the learned node embeddings of author nodes in DBLP into a 2-dimensional space. We select the visualization results of DeepWalk and Metapath2Vec as the representative baselines for homogeneous and heterogeneous embedding methods. The result is shown in Figure 3. As we can see, our model clearly clusters the author nodes into different research areas, which demonstrates that our model learns superior node embeddings by distinguishing the heterogeneous relations in heterogeneous networks.

6. Conclusions

This paper has delved into attributed heterogeneous network embedding. We introduced an innovative attributed random walk strategy that addresses the limitations of meta-path reliance. Subsequently, we formulated the embedding task as akin to machine translation and presented NAGNE to capture both structural and attribute heterogeneity. Through a node-to-attribute generation model, NAGNE flexibly ascertains the neighborhood information’s appropriate breadth. Experimental validations reinforce the model’s superiority over state-of-the-art alternatives.