Multi-Source Information Graph Embedding with Ensemble Learning for Link Prediction

Abstract

Link prediction is a key technique for connecting entities and relationships in a graph reasoning field. It leverages known information about the graph structure data to predict missing factual information. Previous studies have either focused on the semantic representation of a single triplet or on the graph structure data built on triples. The former ignores the association between different triples, and the latter ignores the true meaning of the node itself. Furthermore, common graph-structured datasets inherently face challenges, such as missing information and incompleteness. In light of this challenge, we present a novel model called Multi-source Information Graph Embedding with Ensemble Learning for Link Prediction (EMGE), which can effectively improve the reasoning of link prediction. Ensemble learning is systematically applied throughout the model training process. At the data level, this approach enhances entity embeddings by integrating structured graph information and unstructured textual data as multi-source information inputs. The fusion of these inputs is effectively addressed by introducing an attention mechanism. During the training phase, the principle of ensemble learning is employed to extract semantic features from multiple neural network models, facilitating the interaction of enriched information. To ensure effective model learning, a novel loss function based on contrastive learning is devised, effectively minimizing the discrepancy between predicted values and the ground truth. Moreover, to enhance the semantic representation of graph nodes in link prediction, two rules are introduced during the aggregation of graph structure information. These rules incorporate the concept of spreading activation, enabling a more comprehensive understanding of the relationships between nodes and edges in the graph. During the testing phase, the EMGE model is validated on three datasets, including WN18RR, FB15k-237, and a private Chinese financial dataset. The experimental results demonstrate a reduction in the mean rank (MR) by 0.2 times, an improvement in the mean reciprocal rank (MRR) by 5.9%, and an increase in the Hit@1 by 12.9% compared to the baseline model.

1. Introduction

Link prediction in graph reasoning is a key technique for connecting entities and relationships. It has been widely used in various fields, such as information retrieval [1], information recommendation [2], intelligent chat systems [3], and intelligent detection [4,5,6]. Many open-source knowledge graph datasets have been built, such as DBpedia [7], Freebase [8], and ConceptGraph [9]. Those knowledge graph datasets store knowledge of the graph structure in the form of fact triples (head(h), relation(r), tail(t)). Nodes h and t represent entities. Edges $\left(r\right)$ represent relationships. The left part of Figure 1 illustrates the extraction of structured knowledge from unstructured text and the formation of a knowledge graph. For example, (Chinese Badminton Association, located _ in, China) means “Chinese Badminton Association located in China”. Link prediction is to predict the entity (head or tail) and ensure that each fact triplet structure is complete. To represent these symbolic structural fact triplets as computable digitized information, graph embedding encodes entities and relationships into computable embedding vectors in a unified space and designs a model to predict unseen entities. The right half of Figure 1 demonstrates that in the link prediction task, graph representation learning primarily focuses on studying the structural information of the graph. However, it often overlooks the semantic information embedded within the graph, such as the textual node descriptions.

Link prediction has witnessed numerous models being proposed, which have achieved remarkable success in this field. Those models transform graph data into a computable vector space to represent semantic features of the entities and relations. We categorize these models into three types: decomposition-based methods, including DistMult [10] SimplE [11], and ComplEx [12]; Translational Vector Methods, such as TransE [13], TransH [14], HAKE [15], and RoCS [16]; and Deep Learning Methods, which consist of ConvE [17], CompGCN [18], KG-BERT [19], SAttLE [20], RUGA [21], CKCB [22], and Caps-OWKG [23]. The following shortcomings are found in the mainstream methods according to the related papers:

• Common graph-structured datasets inherently face challenges, such as missing information and incompleteness. Previous link prediction studies have tended to favor a single triplet representation, ignoring contextual information about entities-associated nodes in graph-structured data, as demonstrated in models such as KG-BERT and SAttLE. Conversely, models like CompGCN and SE-GNN [24] have considered the graph structure information while overlooking the textual characteristics of nodes. Specific approaches, such as kNN-KGE [25], have also attempted to leverage text data and graph structure information. Still, their integration remains rudimentary, lacking a more sophisticated fusion of these two information sources. Therefore, the restricted utilization of multi-source information in existing models diminishes their capacity to express semantic relationships effectively. The right half of Figure 1 highlights that in the link prediction task, graph representation learning predominantly emphasizes the study of the graph’s structural information. Nevertheless, it tends to neglect the valuable semantic information present in the graph, such as the textual descriptions of the nodes. Furthermore, the process of mapping graph node information compresses the semantic features of the nodes, exacerbating the scarcity of semantic information for graph nodes.
• Previous models have typically utilized a single network model for training and learning semantic features. However, due to the distinct characteristics of different neural networks, the learned features can vary significantly. Ensemble learning addresses this issue by integrating multiple models and leveraging the strengths of other models to improve the final results. However, current ensemble learning approaches often focus solely on information fusion at the decision level, neglecting the crucial information fusion at the data and feature layers. This limitation leads to inadequate semantic mining and exploration by the link prediction model.

Based on these considerations, we propose a novel framework called Multi-source Information Graph Embedding with Ensemble Learning for Link Prediction (EMGE) to tackle the challenges in link prediction. EMGE enhances entity embeddings at the data level by incorporating both structured graph information and unstructured textual data as inputs from multiple sources. An attention mechanism is introduced to effectively fuse these inputs. This integration enables a more comprehensive understanding of nodes and their relationships within the graph. During the training process, EMGE leverages the principles of ensemble learning to extract semantic features from various types of neural network models. This approach promotes the interaction and integration of information across different models, resulting in a more robust data representation. In addition, a novel loss function based on contrastive learning is designed to optimize the learning process. This innovative loss function aims to minimize the discrepancy between predicted and actual values and enhance the reliability of predicted values. EMGE also defines two node information fusion rules during the aggregation of graph structure information, inspired by the concept of spreading activation. These rules aggregate node information from different perspectives, further enhancing the semantic representation of nodes in the graph. By integrating the multi-source information of nodes, the EMGE model captures a more comprehensive understanding of the nodes and their relationships, leading to improved predictions. Additionally, the adoption of contrastive learning and ensemble learning enables EMGE to learn from both positive and negative examples, leveraging the strengths of multiple models to make more accurate predictions. Overall, the combination of these innovative features in EMGE can significantly enhance link prediction performance.

In summary, the main contributions of this paper are as follows:

• Our method introduces revolutionary solutions for the link prediction task. The EMGE model enhances the semantic expressiveness of graph representation learning and improves the prediction accuracy for unknown data by integrating diverse information sources and mapping them into a unified semantic vector space.
• We introduce text description information to enhance the semantic richness of nodes in the graph structure and employ attention mechanisms to integrate textual and diagrammatic structure information. Subsequently, ensemble learning is applied to facilitate information interaction among different networks and enhance the overall predictive capability of the model.
• Based on experimental evaluations, it demonstrates that EMGE achieves competitive performance on two public datasets and a private dataset.

2. Related Work

2.1. Link Prediction in Graph Reasoning

Link prediction has been identified as a hot area of research in previous studies, where most studies have focused on graph embeddings that represent the elements in triples. In this paper, we divide the existential research into three types. The first type is decomposition-based, including DistMult, SimplE, and ComplEx. These methods map entities and relationships to a unified and continuous vector space, which can capture more semantic features and then formulate a scoring function to verify the model’s validity. However, these early methods only consider each element and ignore the connections between the elements, which makes these methods unable to capture deep semantic features. The second is Translational Vector Methods, such as TransE, TransH, and HAKE. These methods are inspired by the translation invariance of word2vec and define a scoring function to measure the plausibility of triples. Upon examining the results of these models, we observed that despite the introduction of numerous new methods, the overall experimental outcomes have not shown significant improvement. The third is Deep Learning Methods, such as ConvE, R-GCN [26], cacGCN-W [27], and DTAE [28]. These models use the characteristics of different kinds of neural networks to extract the semantic features of data from the spatiotemporal and structural information of data.

2.2. Ensemble Learning in Link Prediction

Ensemble learning is essential in machine learning research because it combines different models to solve the problem of a single model’s poor prediction and improve the model’s accuracy. Several studies have utilized ensemble learning techniques to achieve improved results in link prediction. For instance, Ref. [29] employs multiple ensemble learning methods to tackle link prediction and uses a voting mechanism to aggregate the final predictive outcomes. Similarly, Ref. [30] utilizes random forest-based recursive feature elimination and constructs a two-level ensemble model designed explicitly for link prediction tasks. Furthermore, Ref. [31] applies ensemble learning techniques to perform link prediction on large-scale biomedical knowledge graphs. It is worth noting that existing ensemble learning methods, such as Refs. [21,22,23], primarily integrate various base models at the decision-making layer, overlooking the fusion of data and feature layers. In contrast, EMGE places a stronger emphasis on multi-level information fusion. EMGE enhances the model’s overall performance by considering information integration across multiple layers and improves link prediction accuracy.

2.3. Contrastive Learning in Graph Embeddings

By comparing data with both positive and negative samples, contrastive learning is a potent technique for extracting representations and learning meaningful feature representations. There are related applications in natural language processing as well. For instance, Ref. [32] introduces contrastive learning into sentence embeddings, achieving impressive results in unsupervised and supervised semantic similarity tasks. By leveraging contrastive learning, Ref. [33] develops a schema-augmented multi-level approach for knowledge graph link prediction. Ref. [34] applies contrastive learning to extract rich information from multiple perspectives and constructs a line graph for conducting link prediction. Similarly, Ref. [35] employs contrastive learning to enrich node information through a shared encoder to facilitate link prediction. The utilization of contrastive learning in these studies demonstrates its effectiveness in enhancing representation learning and improving link prediction tasks in various domains.

The EMGE model focuses more on integrating the multi-source information and improving the semantic expression ability of graph nodes. In contrast to previous work, we propose a novel framework that leverages the power of graph neural networks, ensemble learning, and contrastive learning to improve the semantic representation of graph representation learning.

3. Method

Firstly, this section establishes a clear definition of the link prediction task to establish a consensus on the problem. Secondly, we will delve into the frameworks of the EMGE model, which encompass the methods, techniques, and algorithms intended to effectively address the link prediction challenge. By introducing these ideas and modeling frameworks, this paper aims to present a comprehensive and robust strategy for efficiently and effectively solving the link prediction problem.

3.1. Preliminary

This paper adopts a widely accepted definition, where a graph is defined as $Graph=\{V,E,T\}$. Here, V represents a set of entities (nodes), E represents a set of relations (edges), and T represents the set of triple facts. Triple facts can be defined as $s=\left(head\right(h),relation(r),tail(t\left)\right)$, where $head\left(h\right)$ and $tail\left(t\right)$ are entities belonging to V, and $relation\left(r\right)$ is a relation from E.

The link prediction task is focused on predicting the tail entity t given $(h,r,?)$ or indicating the head entity h given $(?,r,t)$, where “?” denotes missing information. For example, given h = “USA” and r = “capital of”, the link prediction task entails predicting the missing tail entity t. In this example, the possible candidates for t could be “Washington” or “New York”. The underlying objective is to build models using existing fact triples to make predictions and inferences about missing information, thereby reducing the fragmentation and incompleteness within the graph.

The EMGE model adopts a question-and-answer-based approach for the link prediction task. It considers the head entity and the relation as questions and the tail entity as answers and, conversely, predicts the head entity given the relation and the tail entity. By treating the task as a question–answer matching problem, the EMGE model aims to improve the semantic representation of the node information in the graph network from multiple perspectives. Finally, we validate the effectiveness of the model through experiments.

3.2. Model Framework

This part provides a comprehensive overview of the EMGE model for link prediction, which encompasses three stages: the input stage, the encoding stage, and the decoding stage. Figure 2 illustrates the structural framework and core ideas of the EMGE model. It enhances the graph representation learning by applying ensemble learning and the node text description information of graphs. The input stage outlines the composition of the data, identifying the entities and relationships that make up the knowledge graph. This stage ensures that the data are properly prepared for subsequent processing. The encoding stage focuses on embedding the representation of the entity and relation. Specific techniques are used to effectively capture the semantic features of the entities and relationships within the knowledge graph. This embedding process lays the foundation for the subsequent analysis and prediction. The decoding stage includes capturing relevant features and training the score function, which is essential for predicting actual entities in the link prediction task. The EMGE model uses concepts from contrastive learning and comparative learning to improve the semantic representability of the model. These methodologies enhance the holistic comprehension of the entity relationships, consequently augmenting the precision and efficacy of the link prediction. By combining these three stages, our model provides a robust framework for link prediction. It embeds entity and relationship representations effectively, captures relevant features, and uses advanced learning techniques to make accurate predictions. So, we can define the EMGE model as:

$$EMGE=\{A,B,C\}$$

(1)

where A represents the input stage, B represents the encoding stage, and C represents the decoding stage.

3.2.1. Input Stage

The initial phase of this framework is primarily concerned with the composition of the dataset. This critical stage requires two different types of data as multi-source information inputs. One type is essential for constructing the intricate graph, while the other is used to provide textual descriptions for the nodes within the graph data. To carefully create the graph structures, it meticulously organizes the triples and uses sophisticated coding techniques to transform them into a coherent and comprehensive graph structure. It is worth noting that previous research has predominantly focused on exploiting only the node information of the graph, lacking the semantic properties necessary to elucidate the intricate relationships between nodes effectively. Consequently, the representation only captures the topological aspects within the graph.

In the case of textual description data, it is imperative that we meticulously filter the existing data and focus only on the relevant information that meets the objectives. Figure 3 highlights that triple information is a condensed form of unstructured information, resulting in a loss of certain semantic features. The mapping process of graph-structured information further aggravates the scarcity of semantic information. By introducing textual description information, it enhances the semantic and contextual information of the nodes. Furthermore, it discovers that longer text descriptions are not necessarily advantageous. Long text descriptions not only consume significant computational resources but also undergo significant compression during vector transformations, which ultimately results in a reduction in the accurate semantic information encapsulated within the vector. Through careful observation, it is found that the opening sentence within the textual description tends to be the most informative and revealing. Consequently, this paper has implemented a careful filtering mechanism to extract and retain only the most salient textual information. The input stage can be formulated as:

$$A=f(T,G,I)$$

(2)

where f represents the data preprocessing function, T represents the set of triples $(h,r,t)$, G represents the graph, and I represents the text description.

3.2.2. Encoding Stage

The encoding stage primarily clarifies the methodology used to embed the representation of entities and relationships. At this crucial stage, this paper adopts the concept of ensemble learning, where it constructs two different methods for representing entities and relationships. These methods provide a multifaceted and semantic representation of entities and relationships, capturing different perspectives and enhancing the overall understanding of the data. The encoding stage can be formulated as:

$$B=g\left(A\right)$$

(3)

where g represents the encoding part.

A. Graph Embedding

Graph embedding is a powerful technique for preserving network topology and node content information within a graph by representing vertices as compact, low-dimensional vectors. The EMGE model exploits the concept of spreading activation to enhance the semantic expression of node information. Rooted in cognitive psychology, spreading activation theory explains how the brain traverses associative thought networks to retrieve specific information. In this theory, a spreading activation network can be visualized as the shortest straight line connecting two nodes in a network graph, indicating that these two nodes are more closely related and tend to establish a connection with the initial node quickly.

Inspired by spreading activation, the EMGE model establishes two rules to effectively convey the semantic essence of the nodes within the graph. Figure 4 illustrates two methods of information aggregation in graph-structured data, based on the concept of activation diffusion. These methods represent the graph nodes by incorporating neighboring nodes and edge features according to distinct rules.

• Rule 1: The information that defines the edges and nodes directly connected to node t should be integrated into node t and updated. The figure on the left in Figure 4 shows the information integration process for Rule 1. As is well known, nodes that have only a one-hop relationship with node t are the most closely related to node t, so the information of these nodes needs to be added to node t.
• Rule 2: It is defined that under the same conditions as the head node h and relationship r, the information of these two is added to node $t_i$; these nodes $t_1,t_2,t_3,\dots ,t_i,\dots ,t_n$ are reintegrated; and the integrated information is added to the original node $t_i$ again. The figure on the right in Figure 4 shows the information integration process for Rule 2.

We can obtain the preliminary graph node embedding through the above two rules. Before that, it needs to understand the information propagation mechanism of graph neural networks, which can be formulated as:

$$m_{h\to r}^l=M^l(E_h^{l-1},E_r^{l-1})$$

(4)

$$m_t^l=\sum _{h,r\in N}{m}_{h\to r}^l$$

(5)

$$E_t^l=U^l(E_t^{l-1},m_t^l)$$

(6)

where $E_h$ and $E_r$ represent the node embedding and edge embedding. N represents a collection of nodes and edges. ∑ represents the information aggregation. U represents the information update.

Equations (4)–(6) show the graph neural network messaging framework. Equation (4) represents the messaging function, which means passing the node information to an edge that has a relationship with it. Equation (5) represents the reduce function, which means updating the new self-node embedding based on messages from the neighbors. Equation (6) represents the update function, triggering the messaging and reducing functions.

Rule 1 describes the representation of node t as a result of the joint action with node h, which has a one-hop relationship, and the relationship r between two nodes. According to the information propagation mechanism of the graph neural network, it forms a calculation formula based on Rule 1, which can be formulated as:

$${\alpha }_{h\to r}^l=\frac{exp(E_h^{l-1}·E_r^{l-1})}{{\sum }_{h,r\in N}exp(E_h·E_r)}$$

(7)

$$m_t^l=\sum _{h,r\in N}{\alpha }_{h\to r}\ast E_r^{l-1}$$

(8)

$$E_t^l=U^l\left(m_t^{l}W\right)$$

(9)

$$E_{rule1}=\sigma \left(E_t^l\right)$$

(10)

where $\sigma$ is the non-linear activation function. W is a matrix of the weight parameters.

Rule 2 describes how information transfer and aggregation are accomplished under certain conditions. According to the information propagation mechanism of the graph neural network, we define the calculation formulas based on Rule 2, which can be formulated as:

$${\alpha }_{h\to r\to t}^l=\frac{exp(\delta (E_h^{l-1}·E_r^{l-1})·E_t^{l-1})}{{\sum }_{h,r\in N}exp(\delta (E_h·E_r)·E_t)}$$

(11)

$$m_t^l=\sum _{h,r,t\in N}{\alpha }_{h\to r\to t}\ast \delta \left({E_r^{l-1}·E}_r^{l-1}\right)$$

(12)

$$E_t^l=U^l\left(m_t^{l}W\right)$$

(13)

$$E_{rule2}=\sigma \left(E_t^l\right)$$

(14)

We take Rule 1, Rule 2, and the aggregation of random initial nodes as the original embedding representation of the nodes, which can be formulated as:

$$E^{\prime }=E+E_{rule1}+E_{relu2}$$

(15)

where E is the initial node embedding.

B. Text Embedding

Text embedding, with its ability to enrich the information encapsulated within the nodes of the graph and elevate the semantic expression of these nodes, assumes immense significance. Section 3.2.1 expounds upon the requisite data format for our model. The present study has adopted the illustrious GloVe English word embedding [36] as our preferred means to encode sentences. Textual information enables the model to concentrate on the most influential information components affecting model performance significantly, thereby improving overall performance.

$$E^{″}=Embedding\left(Text\right)$$

(16)

where $Embedding$ represents a vectorization function. $Text$ represents the textual information.

In this paper, we concatenate the vectorized graph structure nodes with the text information to form a new vector. Next, we employ an attention mechanism to extract features from the new vector. The extracted features are then added to the graph structure node vectors to create updated node vectors. Simultaneously, the extracted features are concatenated with the text vectors to generate new text vectors. The attention mechanism effectively focuses on and balances the importance of the node data and text information by constructing a weight parameter feature matrix, achieving the efficient fusion of data. This process can be defined as follows: leveraging attention weights to linearly combine node and textual information, allowing the model to focus on the most informative aspects that impact model performance maximally, thereby enhancing overall performance. The formula is presented as:

$$E_{Inter}=Atten(E^{\prime },E^{″},W)$$

(17)

$$E^{″}=E^{″}\oplus E_{Inter}$$

(18)

$$E^{\prime }=E^{\prime }+E_{Inter}$$

(19)

where W represents a hyperparameter. $Atten$ represents an attention function. ⊕ represents the concatenation of two vectors. + represents the addition of two vectors.

Recognizing the inherent variability in the sentence length of individual entity descriptions, we successfully overcame this challenge by using a carefully chosen batch size to ensure uniformity during the training process. Consequently, the spliced word vector is then skillfully introduced to a BiLSTM architecture, enabling the extraction of pertinent features. In concise terms, the process can be succinctly defined as follows:

$$\overrightarrow{h_t}=LSTM(E^{″},len\left(E^{″}\right))$$

(20)

$$\overleftarrow{h_t}=LSTM(E^{″},len\left(E^{″}\right))$$

(21)

$$h_t=Dropout(\overrightarrow{h_t},\overleftarrow{h_t})$$

(22)

where $Dropout$ is a function that prevents model overfitting.

3.2.3. Decoding Stage

The decoding stage elucidates the methodology in capturing essential features and training the score function, facilitating accurate query responses. This stage encompasses three integral components: Att-Maxpooling, ConvE, and the loss function. The decoding stage can be formulated as:

$$C=p\left(B\right)$$

(23)

where p represents the data decoding function.

A. Att-Maxpooling

Att-Maxpooling is an important part of the decoding stage, which can be defined as:

$$at{t}_{score}=softmax(W_{1}tanh\left(h_t{W}_2\right))h_t$$

(24)

$$F^{\prime }=maxpooling\left(h_t\right)$$

(25)

$$F_{Att-Maxpooling}=maxpooling(F^{\prime }+at{t}_{score})$$

(26)

$${logit}_{Maxp}=sigmoid(F_{Att-Maxpooling}·N_e)$$

(27)

where $softmax$ is a kind of non-linear function. $maxpooling$ is a feature extraction function. $sigmoid$ is a kind of non-linear function. $N_e$ is a vector representation of all the nodes.

B. ConvE

ConvE is a convolutional neural network that primarily maps the entity and relationship features in fact triples into a uniform vector space.

$$F_{ConvE}=ConvE(F^{\prime }\oplus E^{\prime }\oplus F_{Att-Maxpooling},R)$$

(28)

$${logit}_{ConvE}=sigmoid(F_{ConvE}·N_e)$$

(29)

where R represents the relationship between two nodes.

C. Loss Function

The loss function plays a crucial role in model learning. To this end, we formulate a composite loss function, primarily comprising a contrastive loss function and a binary cross-entropy loss, to evaluate the alignment between $(h,r)$ and the prospective answer t. This paper proposes the construction of a loss function based on contrastive learning, which involves comparing an example with a semantically similar example (positive example) or with one that is semantically different (negative example). The aim is to devise a model architecture and contrastive loss function in a manner that brings the representations of semantically similar instances closer in the feature space while pushing apart the representations of semantically dissimilar instances, thereby achieving a clustering effect. Contrastive learning is often used for unsupervised learning, which can be defined as:

$$L_q=-log\frac{exp(q·k_{+}/\tau )}{{\sum }_{i=0}^{K}exp(q·k_i/\tau )}$$

(30)

where q is a feature vector of the sample data, and $k_{+}$ is a positive sample. $\tau$ is a temperature hyperparameter. K represents negative samples.

Inspired by unsupervised contrastive learning, this study presents a supervised contrastive learning loss function. The formulation of the supervised contrastive learning loss entails four key steps: feature normalization, predicted value generation, actual value generation, and loss function construction. Normalizing and building predicted values are introduced. It should also be noted that the predicted values constructed here are based on the batch size, which can be described as:

$$Z_{ConvE}=Norm\left(F_{ConvE}\right)$$

(31)

$$Z_{Att-Maxpooling}=Norm\left(F_{Att-Maxpooling}\right)$$

(32)

$${\theta }_{ConvE}=(Z_{ConvE}·Z_{ConvE}^T)/\tau$$

(33)

$${\theta }_{Maxp}=(Z_{Att-Maxpooling}·Z_{Att-Maxpooling}^T)/\tau$$

(34)

where $\tau$ is a temperature hyperparameter. $Norm$ is a normalized function.

Then, we introduce building actual values, which can be defined as:

$$p_{ConvE}^{true}=Equ({\theta }_{ConvE},{\theta }_{ConvE}^T)/sum\left(Equ({\theta }_{ConvE},{\theta }_{ConvE}^T)\right)$$

(35)

$$p_{Maxp}^{true}=Equ({\theta }_{Maxp},{\theta }_{Maxp}^T)/sum\left(Equ({\theta }_{Maxp},{\theta }_{Maxp}^T)\right)$$

(36)

where $Equ$ is to determine whether two vectors are equal.

Finally, the contrastive learning loss function can be defined as:

$${loss}_{ConvE}^{CL}=-wlog\frac{exp({\theta }_{ConvE},p_{ConvE}^{true})}{{\sum }_{c=1}^{C}exp\left({\theta }_{ConvE,c}\right)}$$

(37)

$${loss}_{Maxp}^{CL}=-wlog\frac{exp({\theta }_{Maxp},p_{Maxp}^{true})}{{\sum }_{c=1}^{C}exp\left({\theta }_{Maxp,c}\right)}$$

(38)

where w is the weight parameter, and C is the number of classes. $CL$ is a contrastive learning loss.

Next, the joint loss function combined with contrastive learning loss is introduced, which can be defined as:

$$\begin{array}{c}\hfill {loss}_{ConvE}^{BCE}=-w[y_{label}log\left({logit}_{ConvE}\right)+(1-y_{label})log(1-{logit}_{ConvE})]\end{array}$$

(39)

$$\begin{array}{c}\hfill {loss}_{Maxp}^{BCE}=-w[y_{label}log\left({logit}_{Maxp}\right)+(1-y_{label})log(1-{logit}_{Maxp})]\end{array}$$

(40)

$$\begin{array}{c}\hfill loss=\alpha \ast \left(\frac{\beta \ast {loss}_{Maxp}^{BCE}\ast {loss}_{Maxp}^{CL}}{{loss}_{Maxp}^{BCE}{+loss}_{Maxp}^{CL}}\right)+(1-\alpha )\ast \left(\frac{\lambda \ast {loss}_{ConvE}^{BCE}\ast {loss}_{ConvE}^{CL}}{{loss}_{ConvE}^{BCE}+{loss}_{ConvE}^{CL}}\right)\end{array}$$

(41)

where $BCE$ is the binary cross-entropy loss, and y is the label. $\alpha$, $\beta$, and $\lambda$ are coefficient factors that can be adjusted based on the experimental results.

To better understand the model framework, the training process in the form of pseudocode is shown. Algorithm 1 shows the model framework. Algorithm 2 shows the contrastive learning loss.

Algorithm 1: EMGE Framework

Input: E, R, Text Output: Score 01: for i in epoch: 02:       $E_{rule1}$←$Rule1(E,R)$ 03:       $E_{rule2}$$\leftarrow Rule2(E,R)$ 04:       $E^{\prime }$$\leftarrow E+E_{rule1}+E_{rule2}$ 05:       $E^{″}$$\leftarrow Embedding\left(Text\right)$ 06:       $E_{Inter}$$\leftarrow Atten(E^{\prime },E^{″},W)$ 07:       $E^{″}$$\leftarrow E^{″}\oplus E_{Inter}$ 08:       $E^{\prime }$$\leftarrow E^{\prime }+E_{Inter}$ 09:       $h\leftarrow BiLSTM(E^{″},len\left(E^{″}\right))$ 10:       $h\leftarrow Dropout\left(h\right)$ 11:       $at{t}_{score}\leftarrow Attention\left(h\right)$ 12:       $F^{\prime }$$\leftarrow Maxpooling\left(h\right)$ 13:       $F_{Att-Maxpooling}$$\leftarrow Maxpooling(F^{\prime }+h)$ 14:       $logitMaxp\leftarrow sigmoid(F_{Att-Maxpooling}·Ne)$ 15:       $F_{ConvE}\leftarrow ConvE(F^{\prime }\oplus E^{\prime }\oplus F_{Att-Maxpooling},R)$ 16:       $logi{t}_{ConvE}\leftarrow sigmoid(F_{ConvE}·Ne)$ 17:       $loss\leftarrow loss(F_{Att-Maxpooling},logi{t}_{Maxp}$,                         $F_{ConvE},logi{t}_{ConvE},y_{label})$

Algorithm 2: Contrastive Learning Loss

Input: ($F_{att-maxpooling}$, $y_{label}$) or ($F_{ConvE}$, $y_{label}$) Output: ${loss}_{ConvE}^{CL}$ or ${loss}_{Maxp}^{CL}$ 01: Z ← Norm(F) # normalizing 02: $\theta$ ← (Z ·$Z^T$)/$\tau$ # build predicted values 03: # build actual values 04: temp ← $y_{label}$.type_as($\theta$) 05: x ← temp. unsqueeze (−1) 06: y ← $tem{p}^T$. unsqueeze (0) 07: m ← initialize a matrix 08: for i = 0; i < len(temp); i++ do 09:       for j = 0; j < len(temp); j++ do 10:             if x == y: 11:                   $m_{ij}$ ← True 12:             else: 13:                   $m_{ij}$ ← False 14:             endif 15: m ← m.all(1) 16: m ← m.type_as($\theta$) 17: $p^{true}$ ← m/sum(m) 18: # contrastive learning loss 19: $los{s}^{CL}$ ← $-wlog\frac{exp({\theta }_{Maxp},p_{Maxp}^{true})}{{\sum }_{c=1}^{C}exp\left({\theta }_{Maxp,c}\right)}$

4. Experiments

This section provides a detailed analysis of the experimental findings. It begins by outlining the benchmark dataset, evaluation protocol, and experimental configuration utilized. Following this, a comparative analysis is conducted between the performance of the EMGE model and the baseline model. An ablation study is subsequently performed to assess the influence of various components on the EMGE’s performance. Finally, the effectiveness of the EMGE model is reinforced through a case study analysis.

4.1. Datasets

Our experiment utilizes two publicly available benchmark datasets: FB15K-237 and WN18RR. FB15K-237 is a refined subset of FB15K, which improves and simplifies the relationships between entities in the dataset. Similarly, WN18RR is a subset of WN18 that removes duplicate and inverse relations. Additionally, this paper introduces a proprietary Chinese financial-domain dataset (CFD), which focuses on the knowledge graph pertaining to relationships between enterprises. In the CFD dataset, the entities are the names of financial enterprises, and the relationships represent the interactions between these enterprises (such as ownership or investments), mirroring the data structure of two other publicly available datasets. It is noteworthy that the FB15K-237, WN18RR, and CFD datasets are meticulously crafted to ensure that each triple $(h,r,t)$ within the valid and test sets does not possess any direct connections $(h,r^{\prime },t)$ within the training set, thereby rendering the experiment more arduous and challenging. A comprehensive overview of the statistical aspects associated with these three datasets is presented in

Table 1. Statistics of three datasets.

Dataset	Entity	Relation	Train	Valid	Test
CFD	5934	8	14,869	1316	929
FB15k-237	14,541	237	272,115	17,535	20,446
WN18RR	40,943	11	86,835	3034	3134

. The model learning process exclusively utilizes the training set, while the remaining two sets are employed for model verification and evaluation. Notably, a well-trained model exhibits the capability to accurately predict the tail entity, especially for previously unseen data. This setup significantly enhances the model’s predictive ability on novel data instances.

4.2. Evaluation Protocols

This paper uses evaluation metrics widely adopted in the link prediction task to demonstrate the effectiveness of the EMGE model. These metrics include the mean reciprocal rank of correct (MRR), mean rank of correct (MR), and the proportion of valid triples within the top-N ranks (Hits@N, where N = 1, 3, and 10). A more favorable performance of our model is indicated by lower values of the MR and higher values of the MRR and Hits@N. These metrics collectively provide valuable insights into the model’s ability to accurately predict links and rank correct triples. PROCRUSTES [37], SE-GNN, NePTuNe [38], SAttLE, HIE [39], SEA [40], SHGNet [41], CoDLR [42], and DTAE act as baselines for public datasets. Additionally, we have included a table to illustrate the differences between the baseline models and EMGE, as shown in Figure 5.

4.3. Settings

To better represent the entity’s text, this paper uses GloVe English word embedding. The number of the epoch is 200. The number of hidden units for the BiLSTM is 200. And the layer of the BiLSTM is 1. The temperature $\tau$ is 0.1. The Adam optimizer is adopted. This paper adopts the method of dynamically adjusting the learning rate to speed up the model training. To improve the computational efficiency, we transform the non-directionality of the graphical data into bi-directionality in data processing. It defines that $(h,r,?)\to t$ and $(?,r,t)\to h$ transform them as $(h,r,?)\to t$ and $(t,r,?)\to h$. In addition, during the training process, this study randomly removes a certain proportion of edges in the graph to enhance the robustness of the model. It not only can prevent unknown information from being exposed in advance, but the absence of edges is closer to the real scene. This model is implemented using PyTorch (1.9) on an NVIDIA RTX 3090 Ti GPU.

4.4. Results

We present and analyze the experimental results for the CFD dataset, as well as the publicly available WN18RR and FB15K-237 datasets in this part. The recent research models are selected as baseline models to compare and contrast the performance differences between the models. The specific experimental results are illustrated in Figure 6 and Figure 7.

By observing the results of the experiment, the following can be found:

• The EMGE model demonstrates superior performance compared to the baseline model in seven out of ten evaluation metrics for the WN18RR and FB15K-237 datasets. Notably, the MR index shows a significant reduction in the WN18RR dataset, reaching approximately 0.27 times the value observed in the baseline model. Furthermore, the EMGE model exhibits commendable experimental results in the H@10 metric. The experimental findings validate the model’s adeptness in handling diverse data types. The challenge of the WN18RR dataset lies in its abundance of entities but limited relationship types, which are exacerbated by the scarcity of training data. Similarly, the FB15k-237 dataset poses difficulties due to its relatively fewer entities, yet intricate relationships and imbalanced data, thus intensifying the training complexity. In addition, compared to the FB15k-237 dataset, the WN18RR training data have a smaller volume and a larger number of entities, as evident from Table 1. This has also impacted the performance of the EMGE model. However, this situation is common across other models as well, representing a common issue. However, the EMGE model tackles these two dataset variations by leveraging multi-source information to enhance the semantic understanding of the nodes within the graph structure data, thereby bolstering the predictive prowess of the model. The experimental results further substantiate the efficacy of the EMGE model. Compared to SE-GNN, which also utilizes node information aggregation, it shows significant performance differences compared to the EMGE model. This suggests that our information aggregation method is superior. In addition, the CoDLR method also incorporates textual information for description but shows significant performance differences compared to the EMGE model. This suggests that the method may not be as strong in feature extraction as our approach.
• As the CFD dataset focuses on inter-firm linkages and is privately curated, it does not have a comparable baseline model like the publicly available datasets for comparison. Therefore, this paper selects ConvE, SE-GNN, and CompGCN as the benchmark models. The experimental results indicate that our model ranks second in terms of the MR metric, closely following the top-ranked model, and outperforms the other models in four additional metrics. Given the small scale of the CFD dataset, with relatively fewer entities than the other two datasets, effectively handling small-sample data becomes a test of the model’s generalizability. The experimental results demonstrate that our model excels in handling Chinese datasets, showcasing outstanding performance even in scenarios with limited samples and exhibiting stable experimental results. This further validates the effectiveness of leveraging multi-source information and ensemble learning in enhancing the predictive capabilities of the EMGE model.

4.5. Ablation Study

To assess the impact of each critical section on the experiment, this paper performs ablation experiments to remove one or more sections in sequence. Here, we conducted seven sets of experiments. Each set involves removing different components to demonstrate the impact of different modules on the model itself. These seven sets of experiments involve removing different components as follows: (1) Remove the text description information. (2) Remove the text description information and the contrastive loss function. (3) Remove node aggregation Rule 1 and the text description information. (4) Remove node aggregation Rule 1, the text description information, and the contrastive loss function. (5) Remove node aggregation Rule 2 and the text description information. (6) Remove node aggregation Rule 2, the text description information, and the contrastive loss function. (7) Remove node aggregation Rules 1 and 2, the text description information, and the contrastive loss function.

Figure 7 and Figure 8 show the results of CFD, WN18RR, and FB15k-237. The analysis reveals a consistent decrease in the overall performance of the EMGM model following the removal of each key component, underscoring the crucial role that each element plays in enhancing the model’s effectiveness. In addition, the model’s overall performance is worse when critical parts are removed. It is further explained that each part has a limited effect, but the combined model performs better. Removing each part of the model is relatively stable for WN18RR and FB15-237 but has a more significant impact on CFD. Removing the text information resulted in a significant decrease in overall model performance, highlighting the importance of text description information. The absence of text information also implies that the model cannot utilize ensemble learning for data fusion, feature fusion, and decision information fusion, leading to an overall reduction in model performance. Additionally, the ablation experiments demonstrate that the model’s effectiveness does not solely rely on the performance of any single module but rather on the collective contribution of all modules working together to achieve results.

4.6. Case Study

We employ a detailed case study to highlight the effectiveness of the EMGE model, demonstrating its capability to predict unseen data. This paper carefully selects specific cases from the FB15k-237 test set and presents the ranking results in Figure 9, including models such as EMGE, EMGE (w/o Rule 1 and text), SE-GNN, and ConvE. Through these representative data, it observes how different models perform in predicting unknown instances.

For example, consider the tail entity prediction (Silkwood, award, ?) → BAFTA Award for Best Actress in a Leading Role. It is worth noting that this particular triplet only appears in the test set, further substantiating the predictive power of the EMGE model for previously unseen data. Additionally, when it comes to head entity prediction (?, film performance, Screen Actors Guild Award for Outstanding), it discovers five candidate entities that can potentially serve as the head entities for this set of triples. To further establish the predictive ability of our model for unseen data, the average ranking method is used to calculate the prediction performance of different models. As shown in Figure 8, the EMGE model consistently exhibits a more prominent average ranking than the other models, corroborating its superior predictive capabilities.

Furthermore, to corroborate the experimental outcomes, a random selection of 100 data points from the FB15k-237 dataset is conducted. The data analysis results are depicted in Figure 8, and the analysis of the experimental results is performed.

• In Figure 9, this paper compares the rankings produced by the four models and divides the ranking values into four intervals ($[1,3),[3,5),[5,10],(10,10+)$). By comparing the experimental results, it is found that the EMGE algorithm performs better. In particular, it performs better than other algorithms in the $[5,10]$ interval, consistent with the experimental results in Figure 7. At the same time, in the $(10,10+)$ interval, the EMGE algorithm obtains the minor data and performs better, indicating that the stability, robustness, and generalization ability of the algorithm itself are better.
• In the second half of Figure 9, based on the results of 100 random data experiments, four pie charts are drawn to show the overall effect of different models. The analysis shows that the EMGE model accounts for the most minor proportion in the interval $(10,10+)$. This further proves that the EMGE model can effectively improve the accuracy of link prediction and the overall data ranking.

By setting different experimental conditions, the experimental results demonstrate the application of the concept of ensemble learning to discover entities and relationships by combining textual and graph structural data. We employ various neural networks to capture diverse features and leverage a range of scoring functions to comprehensively evaluate prediction accuracy. Additionally, we introduce contrastive learning to construct loss functions, thus amplifying the differences between various samples.

5. Conclusions

In this paper, we introduce a novel link prediction model, EMGE, to address the challenges of knowledge fragmentation and hidden knowledge extraction in graph-structured data. It uses both structured graph information and unstructured textual data as multi-source information inputs, makes use of the attention mechanism to fusion the multi-source information, and adopts the idea of ensemble learning to extract semantic features from different types of neural network models and facilitate information interaction. In addition, we employ the concept of contrastive learning to create a new loss function that minimizes the gap between predicted values and the ground truth. This method achieves competitive results on the link prediction task and demonstrates good adaptability and robustness to datasets from different domains. Moreover, the EMGE model is validated on three datasets, including WN18RR, FB15k-237, and a Chinese financial dataset. The experimental results demonstrate a reduction in the mean rank (MR) by 0.2 times, an improvement in the mean reciprocal rank (MRR) by 5.9%, and an increase in the Hit@1 by 12.9% compared to the baseline model.

In the future, we plan to explore more information aggregation patterns and develop a novel framework based on a GNN. We believe that our work will contribute to the advancement of link prediction and facilitate the development of intelligent systems that can reason and learn from complex knowledge graphs.