Enhanced Temporal Knowledge Graph Completion via Learning High-Order Connectivity and Attribute Information

Abstract

Temporal knowledge graph completion (TKGC) refers to the prediction and filling in of missing facts on time series, which is essential for many downstream applications. However, many existing TKGC methods suffer from two limitations: (1) they only consider direct relations between entities and fail to express high-order structural dependencies between entities; and (2) they only leverage relation quadruples of temporal knowledge graphs, ignoring attribute information that contains rich semantic information. This makes them vulnerable to sparsity and incompleteness problems. In response, we propose HCAE, a temporal knowledge graph completion model that includes high-order connectivity and attribute information. This consists mainly of a recursive embedding propagation layer and a multi-head attention aggregation layer. The former leverages a recursive mechanism to update entity embeddings and can learn high-order connectivity information between entities in linear complexity time. The latter leverages an attention mechanism to understand the importance of different attributes for entity representation automatically. Combining high-order connectivity and attribute information can lead to more diverse entity representations and help enhance the model’s ability to infer unknown entities. Comparative experiments on three real-world datasets show that the model’s inference accuracy significantly outperforms other benchmark methods, especially regarding knowledge graphs with many unknown entities or relations.

1. Introduction

A knowledge graph (KG) is a structured representation of entities and their attribute relationships in the real world. It has been widely applied in various fields, such as natural language processing [1], recommendation systems [2], and information retrieval [3]. However, traditional KGs are static, snapshot-like knowledge representations that typically form facts in the form of triplets $\left(s,r,o\right)$, where S represents the subject entity, O represents the object entity, and $r$ represents the relationship between entities. Static knowledge graphs (SKGs) fail to capture the evolution of entities and relationships over time, limiting their modeling and predicting capabilities in terms of temporal aspects. This has led to the development and application of temporal knowledge graphs (TKGs). TKGs introduce timestamps $t$ and extend the representation to quadruplets $\left(s,r,o,t\right)$. TKGs are a collection of snapshot graphs with different timestamps, where facts about entities and relationships coexist within the same snapshot [4].

TKGs provide a new perspective for downstream applications such as resource allocation [5], question answering [6], and stock prediction [7]. However, many TKGs need help with the problem of incompleteness, which affects their reliability and accuracy, severely limiting their usage. Therefore, it is of great significance to design an efficient and adaptive method with which to enhance the completeness of traditional TKGs. The TKGC task aims to predict missing facts based on the existing information in TKGs. It can be divided into two subtasks: predicting missing entities for queries like $\left(?,r,o,t\right)$ or $\left(s,r,?,t\right)$ and predicting missing relationships for queries like $\left(s,?,o,t\right)$. The symbol $?$ represents the missing entity or relation. The main challenge lies in effectively integrating timestamps into the model to accurately capture and leverage the temporal dynamics of entities, relationships, and the underlying graph, thereby enhancing the ability to predict missing facts [8]. In this regard, researchers have explored various timestamp integration strategies, such as tensor decomposition based on timestamps [9], timestamp transformation-based methods [10], dynamic embeddings [11], and predictions based on historical contexts [12]. The goal is to incorporate timestamps into the fact-scoring function. However, these methods mainly model the structural features and temporal dependencies of TKGs at a single timescale, and they do not consider the potential temporal variability of TKGs, which leads to the limited performance of TKGC solutions. Building upon this, the MtGCN [13] model utilizes a gated recurrent unit (GRU) to learn the temporal dependencies of TKGs across multiple timescales. Additionally, it combines organically with graph convolutional networks (GCNs) to achieve end-to-end temporal link prediction and synergistic enhancement of temporal modeling and graph structure learning. However, existing research still has some limitations: firstly, it only considers direct relationships between entities while overlooking the high-order connectivity among entities. Secondly, it models only the relationship quadruples, neglecting the abundant attribute information in the KG. These limitations make these methods susceptible to the sparsity and incompleteness of the KG. Additionally, the structural information cannot express the specific meanings between entities and relations, severely inhibiting model prediction ability enhancement.

To overcome these limitations, this paper proposes HCAE, a temporal knowledge graph completion model that fuses high-order connectivity and attribute information, which continues the idea of MtGCN multi-timescale modeling. There are two novelties: (1) Explicitly capturing the high-order connectivity between entities. High-order structured relationships contain rich semantic information that can reflect the complex connections between entities, which is conducive to multi-step prediction and to improving the matching ability of KG representation learning. As shown in Figure 1, the relationship between $Donald Trump$ and $Jared Kushner$ can be mined through the facts of $Donald Trump\stackrel{Father of}{\to }Ivanka Trump\stackrel{Spouse}{\to }Jared Kushner$. Existing path construction methods attempt to capture the high-order connectivity of KGs using multi-step relationship chains [14]. However, the vast paths pose a significant computational complexity challenge for such methods. These methods can only perform path sampling or select approximate treatments for efficient path modeling. Our analysis reveals that such approximations can significantly impact the final model performance. (2) Explicitly modeling attribute information in KGs. There is a large amount of attribute information in real KGs, as shown in a statistic of Sun et al. [15] on the English version of the DBpedia dataset (2016-04), in which attribute triples are three times as many as relation triples. Attribute triples (denoted as $\left(s,r,v\right)$, with $v$ indicating attribute values) form a rich network of associations between entities that can characterize specific meanings of entities and relationships. As shown in Figure 1, attributes such as $Education$ and $Profile$ associated with the entity $Donald Trump$ can provide insights into his educational background and personal profile, facilitating the extraction of rich semantic information and enhancing the model’s prediction and completion capabilities. Such information cannot be obtained from relationship quadruplets alone. Existing works have attempted to incorporate attribute information into TKGC models [16,17]. However, they suffer from two limitations. On the one hand, some methods only utilize partial attribute information, such as AATE_E [17], that solely uses entity description information. On the other hand, both relational and attribute information are optimized in a unified framework, but the loss function must be balanced for both tasks. For instance, JAPE [15] adjusts hyperparameters to achieve the optimal balance between the two loss functions.

The limitations of existing methods severely restrict the performance of the models. Given this, designing a model that can learn attribute information and high-order connectivity uniformly and efficiently is essential. Modeling high-order connectivity allows for the capturing of dependencies and semantic associations among entities, thus enabling a more comprehensive understanding of entity characteristics from multiple perspectives. Attribute information explicitly describes entity features and complements structural information, allowing the model to generalize better to unseen entities. The joint modeling of structure and attributes enables the learning of representations that fuse connectivity patterns and content features, leading to a more comprehensive understanding of entity characteristics. The combination of both approaches provides complementary benefits.

Unlike existing approaches, HCAE computes entity-specific representations by weighting attribute information and performing embedding propagation based on high-order structural relationships. Specifically, HCAE elaborates on two improvement strategies. Firstly, recursive embedding propagation of relational quaternions using a GCN-based architecture aims at the real-time updating of entity embedding representations. Using a recursive embedding mechanism enables HCAE to learn high-order connectivity between entities with linear time complexity. Secondly, the automatic learning of the importance of different attributes to entity representation through the multi-attention mechanism, which adequately fuses structural and attribute information to enhance the ability to model entities from multiple perspectives.

The main contributions of this work are as follows:

• We emphasize the importance of attribute information and high-order connectivity in temporal knowledge graph completion, highlighting the complementary benefits of their combination.
• We propose the HCAE model based on the limitations of existing work, aiming to utilize attribute information and higher-order structural information to improve the model’s prediction of unknown facts.
• The experimental results on three real-world datasets demonstrate that HCAE improves the predicting ability for missing facts, indicating that learning high-order connectivity and attribute information enhances the model’s TKGC capability.

2. Related Works

2.1. Static Knowledge Graph Completion

Static knowledge graph completion (SKGC) methods are based mainly on knowledge representation learning, which captures semantic associations and patterns between entities and relations by mapping them to low-dimensional vector representations [13]. The standard approaches are broadly categorized into three types: (1) Translation-based models: both TransE [18] and RotateE [19] follow the idea that a relationship causes head and tail entities to be translated in the embedding space. The difference lies in the technique of using translation and rotation, respectively. The higher the proximity of the transformed vectors of head and tail entities, the higher the matching degree. ComTR [20] uses double-quadratic number-based multiplication to model timestamps and relations as translation and rotation operations. MuRMP [21] embeds multi-relationship KGs into the hybrid curvature space of the SKGC task. (2) Bilinear model: also known as tensor decomposition. For example, RESCAL [22] decomposes the relationship matrix into the product of three matrices using singular value decomposition. The result represents the association scores between entities and relationships. However, it suffers from high computational complexity and the risk of overfitting. DistMult [23] is a simplified version of RESCAL that represents relationships as dot products between entity vectors, using a symmetric bilinear function to compute scores between entity pairs, offering advantages in terms of computational efficiency but with limitations in multi-relational modeling. ComplEx [24] is an extension of DistMult that represents relationships as complex products of entity vectors, which is more advantageous in handling multiple relationships. GIE [25] learns interaction structures in various single spaces, such as Euclidean, hyperbolic, or spherical spaces, to capture more complex topological structures in KGs, including chain-like, hierarchical, and cyclical structures. (3) Neural network models: rGCN [26] is a multi-layer relation aware GCN that performs representation learning directly on the KG structure. It incorporates relationship-type information to differentiate relationships and employs de-featurization and batch normalization to improve training stability. ConvE [27] maps entities and relations to a continuous vector space and calculates similarity between entities and relationships using element-wise multiplication and pooling operations. FTL-LM [28] is a language model that combines topological structure information and logical rules. MKGformer [29] utilizes a hybrid transformer and multi-level fusion architecture, achieving multi-level fusion of visual and textual multimodal knowledge representations through prefix-guided coarse-grained interaction and correlation-aware fine-grained fusion modules. KRAT [30] leverages graph context to project neighboring triplets into different latent spaces and jointly aggregates messages with attention mechanisms.

2.2. Temporal Knowledge Graph Completion

Many TKGC models extend existing SKGC models by introducing time embeddings or modeling the temporal dimension, to capture the evolution of entities and relationships at different time points. For example, TTransE [31] is an extension of the TransE model that encodes timestamps as translation operations to perform additional translation on the representations of head entities. TBDRI [32] is a block term decomposition (similar to a variant of Tucker decomposition) model based on binary tensor representation of KG quadruples. It enhances the binding of forward and backward relations and introduces a core tensor as temporal information to capture temporal relationships between entities better. ChronoR [33] uses the RotatE model as a foundation and combines the timestamp with the relation representation to form a unified rotation vector. This enables the representation of temporal relationship pairs with rotation and scaling in the embedding space. TComplEx [34] and TNTComplEx [34] are extensions of ComplEx. The former applies factorization methods to model a fourth-order tensor, while the latter decomposes the tensor into temporal and non-temporal components during the factorization process. TASTER [35] represents different TKGs using sparse transformation matrices to reduce the complexity of entity relation evolution.

Fusion neural networks are another vital research direction for TKGC models. TA-DistMult [36] learns time-aware relationship representations using recursive neural networks (RNN) based on DistMult and TransE models, injecting the capability of temporal modeling into these two SKGC models. TeMP [37] utilizes message-passing graph neural networks (GNN) to learn structure-based entity representations at each timestamp. It aggregates the representations of all timestamps using TeMP-GRU and TeMP-SA encoders, capturing both time dependency and structure dependency. FTMF [38] encodes entities using self-attention mechanisms, aggregates parameter sets using RNNs, and employs a fault-tolerant mechanism to handle incorrect information. It effectively handles few-shot TKGC tasks. SANe [39] maps facts corresponding to different timestamps to different latent spaces using a convolution neural network (CNN). It explores the overlap in the latent space through a time-aware parameter generator to capture contextual information about timestamps. Many existing studies rely on handcrafted GNNs, which cannot fully explore the diverse topology and temporal information of TKGs. To address this, SPA [40] adopts neural architecture search (NAS) techniques to design a general framework. Through single-path sampling training, it efficiently searches for the optimal message-passing structure on various datasets. RoAN [41] is a time-oriented attention network that embeds time features by applying attention mechanisms to multi-chain relationships in TKGs. ExKGR [42] employs GNNs to achieve interpretable multi-hop completion. TARGCN [43] is a simple but powerful graphical encoder with valid parameters. The contextual information of each entity can be extensively mined.

The two types of methods described above require predefined temporal granularity and modeling based on this granularity. Nevertheless, temporal information in TKG exhibits multi-granularity, and the time scales for the evolution of different facts or relationships are other. Using a fixed single time scale makes capturing historical information at additional time granularities challenging and will lead to inadequate modeling of specific temporal patterns, thus reducing the expressive power of the model. Therefore, designing a model that can support multi-timescale modeling in the TKGC task is a direction worth exploring in depth. Based on multi-timescale modeling, HCAE enhances the model’s ability to complement TKG by learning attribute information and high-order connectivity.

3. Proposed Model

3.1. Problem Formulation

This study involves both relationship quadruples and static attribute triples. However, considering the dimensional conflicts between them and the difficulty of accurately predicting the properties of unknown entities using fixed attribute values, this paper introduces additional timestamps to static attribute triples, expanding them into dynamic attribute quadruples.

Definition 1.

Temporal Knowledge Graph: A TKG consists of a relation quadruple $\left(s,r,o,t\right)\in T_R$ and an attribute quadruple $\left(s,r,v,t\right)\in T_A$. Therefore, a TKG can be denoted as $\mathcal{G}=\left\{\mathcal{E},\mathcal{R},\mathcal{A},\mathcal{T},T_R,T_A\right\}$, while $\mathcal{E}=\left\{s,o|\left(s,r,o,t\right)\in T_R\cup \left(s,r,v,t\right)\in T_A\right\}$, $\mathcal{R}=\left\{r|\left(s,r,o,t\right)\in T_R\cup \left(s,r,v,t\right)\in T_A\right\}$, $\mathcal{A}=\left\{v|\left(s,r,v,t\right)\in T_A\right\}$ and $\mathcal{T}=\left\{t|\left(s,r,o,t\right)\in T_R\cup \left(s,r,v,t\right)\in T_A\right\}$ denote the set of entities, relations, attributes and the timestamps, respectively.

The research objective of this study is to generate continuous vector representations for elements in each relationship quadruple and attribute quadruple by mining the high-order connectivity and attribute information of the TKG. This includes generating complete vector representations for $\left(\mathit{s},\mathit{r},\mathit{o},t\right)$ and $\left(\mathit{s},\mathit{r},\mathit{v},t\right)$ facts, where the bolded elements represent vectors. Here, the subject entity is $\mathit{s}\in {ℝ}^d$ ($d$ is the dimension of the embedding representation), the relation is $\mathit{r}\in {ℝ}^d$, the object entity is $\mathit{o}\in {ℝ}^d$, the attribute is $\mathit{v}\in {ℝ}^d$, and the timestamp t is a scalar.

In this work, the inputs are all facts (including relations and attributes) in a given TKG $\mathcal{G}$ before the timestamp t, and the goal is to maximize the performance metrics (using MRR and Hits@n) for the following two tasks:

• Entity prediction: given $\left(?,r,o,t\right)$ or $\left(s,r,?,t\right)$, predict the missing subject or object entity at t moment.
• Relation prediction: given $\left(s,?,o,t\right)$, predict the missing relationship at the t moment.

3.2. Overall Structure

The HCAE model takes as input all relation quadruples and attribute quadruples in the TKG. It consists of four parts: (1) Attribute embedding layer: using long short-term memory (LSTM), it maps semantically related attributes to nearby positions in the vector space to capture attribute correlations. (2) Embedding propagation layer: entity and attribute vectors are used as inputs, and graph attention embedding is employed to learn high-order connectivity between entities. (3) Temporal encoding layer: multiple layers of GRU are set up to learn historical information at different time granularities. $k$ represents the time window size used to control the length of historical information. Finally, multi-granularity temporal features are fused. (4) Decoder: the entity vectors and relationship vectors generated by the structure mentioned above are used as inputs to compute entity-relation scores for all quadruples. The model’s overall structure is shown in Figure 2, and the notation used in the paper is summarized in

Table 1. Notations used in the paper.

Symbol	Description
$\mathcal{G}$	The temporal knowledge graph
$\mathit{s}\in {ℝ}^d,\mathit{r}\in {ℝ}^d,\mathit{o}\in {ℝ}^d,\mathit{v}\in {ℝ}^d$	The embedding vector of subject entity S, relation r, object entity O, attribute v
d	The dimension of the representation
$T_R,T_A$	The set of relation quadruple, attribute quadruple
$\mathcal{E}=\left\{s,o|\left(s,r,o,t\right)\in T_R\cup \left(s,r,v,t\right)\in T_A\right\}$	The set of entities
$\mathcal{R}=\left\{r|\left(s,r,o,t\right)\in T_R\cup \left(s,r,v,t\right)\in T_A\right\}$	The set of relations
$\mathcal{A}=\left\{v|\left(s,r,v,t\right)\in T_A\right\}$	The set of attribute values
$\mathcal{T}=\left\{t|\left(s,r,o,t\right)\in T_R\cup \left(s,r,v,t\right)\in T_A\right\}$	The set of timestamps
t	Timestamp scalar
k	The length of the time window
${\mathit{w}}_i\in {ℝ}^d$	The word embedding of the word $w_i$ at position i
${\mathcal{N}}_{i,t}=\left\{o,v|\left(s_i,r,o,t\right)\in T_R\cup \left(s_i,r,v,t\right)\in T_A\right\}$	The neighborhood of entity $s_i$ at timestamp t
${\mathit{h}}_{i,t}\in {ℝ}^d,{\overrightarrow{\mathit{h}}}_{i,t}\in {ℝ}^d$	The embedding of entity $s_i$ and updated embedding
${\overrightarrow{\mathit{h}}}_{i,t}^{\left(l\right)}$	The embedding of the entity $s_i$ in l-th layers
$\mathit{W}\in {ℝ}^{d\times d}$	The learn-able linear transformation
${\mathit{W}}^j$	The linear transformation of the j-th attention head
${\mathit{H}}_{i,t}^s\in {ℝ}^{\left|\mathcal{E}\right|\times d},{\mathit{H}}_{i,t}^m\in {ℝ}^{\left|\mathcal{E}\right|\times d},{\mathit{H}}_{i,t}^l\in {ℝ}^{\left|\mathcal{E}\right|\times d}$	The temporal representation of short-term, mid-term and long-term history
${\hat{\mathit{H}}}_{i,t}\in {ℝ}^{\left|\mathcal{E}\right|\times d}$	The output of the temporal encoder
$\left|\epsilon \right|$	The number of entities
$N_1,N_2,N_3$	The length of short-term, mid-term and long-term history
${\lambda }_1,{\lambda }_2,{\lambda }_3$	The weight coefficients for three timescale histories
ω	The convolutional kernel matrix

.

3.3. Attribute Embedding Layer

The performance of machine learning algorithms relies heavily on the consistency of interpreting input vectors. Poor data interpretation can lead to suboptimal results. Therefore, feature extraction is crucial in machine learning algorithms [44]. Attribute values are often in the form of variable-length text, and to obtain dense feature representations, an encoder is required to encode the text into fixed-length vector representations. The bag-of-words (BOW) encoder is a commonly used method for feature representation in natural language processing (NLP) due to its simplicity and ability to capture word frequency information. However, it disregards word order and semantic information, making it unable to handle contextual relationships and word sense ambiguity. Hence, this paper adopts LSTM as the encoder. Deep learning techniques, on the other hand, facilitate faster information extraction, enabling scientists to extract discriminative features without relying solely on limited domain knowledge and manual effort [44]. The attribute value is denoted as $v=w_1,w_2,\dots w_n$, where $w_i$ denotes the position of i word, and the output of the last hidden state of LSTM is taken as the attribute vector $\mathit{v}$. Figure 3 shows the detailed process of the attribute embedding layer.

$$\mathit{v}=f_{lstm}\left({\mathit{w}}_1,{\mathit{w}}_2,\dots ,{\mathit{w}}_n\right),$$

(1)

where ${\mathit{w}}_i\in {ℝ}^d$ denotes the word embedding vector and $f_{lstm}$ denotes the LSTM network.

3.4. Embedding Propagation Layer

This layer is primarily based on the GCN architecture for recursive embedding propagation, including attention embedding propagation and embedding aggregation. Specifically, the graph attention mechanism is employed to learn the attention weights of neighboring entities for each entity, enabling selective cascading embedding propagation. The entity embeddings are then recursively propagated and aggregated onto the central entity in the KG, ultimately achieving entity representation learning. The detailed structure of the embedding propagation layer is illustrated in Figure 4.

Attention Embedding Propagation. Its inputs are entities, relations, and attribute embeddings. The embedding representation of entity $s_i$ as ${\mathit{h}}_{i,t}\in {ℝ}^d$ at timestamp t and the set of its neighbors as ${\mathcal{N}}_{i,t}=\left\{o,v|\left(s_i,r,o,t\right)\in T_R\cup \left(s_i,r,v,t\right)\in T_A\right\}$, aims to obtain an updated embedded representation ${\overrightarrow{\mathit{h}}}_{i,t}$ of entity $s_i$ by encoding neighbor entity information and attribute information. In this paper, we take the quadruple $\left(s_i,r,o,t\right)$ as an example:

$${\overrightarrow{\mathit{h}}}_{i,t}={\displaystyle \sum _{o\in {\mathcal{N}}_{i,t}}\pi \left(s_i,r,o,t\right)\mathit{W}}\left(\mathit{r}+\mathit{o}\right),$$

(2)

where $W\in {ℝ}^{d\times d}$ is a learnable linear transformation that transforms the input embedding into a high-order feature space. $\pi \left(s_i,r,o,t\right)$ is an attention coefficient representing the importance of entity O to entity $s_i$.

The attention coefficient is used to control the amount of information propagation between the entities, and is computed by the softmax function for all quadruple connected to entity $s_i$:

$$\pi \left(s_i,r,o,t\right)=\frac{exp\left(\pi \left(s_i,r,o,t\right)\right)}{{\displaystyle \sum _{o^{\prime }\in {\mathcal{N}}_{i,t}}exp\left(\pi \left(s_i,r^{\prime },o^{\prime },t\right)\right)}},$$

(3)

In this paper, the attention coefficients are realized by a simple single-layer feedforward neural network:

$$\pi \left(s_i,r,o,t\right)=LeakyRelu\left({\left(\mathit{W}\mathit{r}\right)}^T\mathit{W}\left(\mathit{r}+\mathit{o}\right)\right),$$

(4)

where $LeakyRelu\left(\cdot \right)$ denotes the activation function.

Embedding Aggregation. To learn more stable entity representations and obtain richer semantic information, the last layer of HCAE uses a multi-head attention mechanism. The role of embedding aggregation is to integrate the embedding representations of each head in order to obtain the final embedding representation of the center entity. In this paper, the average aggregation method is used to extend Equation (2):

$${\overrightarrow{\mathit{h}}}_{i,t}=LeakyRelu\left(\frac{1}{m}{\displaystyle \sum _{j=1}^m{\displaystyle \sum _{o\in {\mathcal{N}}_{i,t}}\pi {\left(s_i,r,o,t\right)}^j{\mathit{W}}^j\left(\mathit{r}+\mathit{o}\right)}}\right),$$

(5)

where m denotes the number of attention heads, $\pi {\left(s_i,r,o,t\right)}^j$ indicates the j normalized attention coefficient, and ${\mathit{W}}^j$ is a linear transformation.

Next, multiple layers of embedding propagation are stacked to aggregate features from multi-hop neighboring entities, encoding deeper-level connectivity information to capture semantic information within the high-order connectivity structures in the TKG. The embedding representation of the l layer of entity $s_i$ is:

$${\overrightarrow{\mathit{h}}}_{i,t}{}^{\left(l\right)}={\displaystyle \sum _{o\in {\mathcal{N}}_{i,t}}\pi \left(s_i,r,o,t\right)\mathit{W}\left({\mathit{r}}^{\left(l-1\right)}+{\mathit{o}}^{\left(l-1\right)}\right)},$$

(6)

3.5. Temporal Encoding Layer

The temporal encoding layer consists of multiple layers of GRU. Its input is the entity embedding sequences at different time granularities, which incorporate attribute and high-order structural information. Controlled by the time window, it models the short-term, medium-term, and long-term historical information of entities separately. Finally, the outputs of GRUs at different granularities are concatenated and passed to the next layer. The multi-granularity temporal features provide abundant temporal information for TKGC tasks and help improve the model’s prediction accuracy:

$${\mathit{H}}_{i,t}^s=GRU\left({\mathit{h}}_{i,t-N_1},{\mathit{h}}_{i,t-N_1+1},\dots ,{\mathit{h}}_{i,t-1}\right),$$

(7)

$${\mathit{H}}_{i,t}^m=GRU\left({\mathit{h}}_{i,t-N_2},{\mathit{h}}_{i,t-N_2+1},\dots ,{\mathit{h}}_{i,t-1}\right),$$

(8)

$${\mathit{H}}_{i,t}^l=GRU\left({\mathit{h}}_{i,t-N_3},{\mathit{h}}_{i,t-N_3+1},\dots ,{\mathit{h}}_{i,t-1}\right),$$

(9)

$${\hat{\mathit{H}}}_{i,t}={\lambda }_1{\mathit{H}}_{i,t}^s+{\lambda }_2{\mathit{H}}_{i,t}^m+{\lambda }_3{\mathit{H}}_{i,t}^l,$$

(10)

where ${\mathit{H}}_{i,t}^s\in {ℝ}^{\left|\mathcal{E}\right|\times d}$, ${\mathit{H}}_{i,t}^m\in {ℝ}^{\left|\mathcal{E}\right|\times d}$ and ${\mathit{H}}_{i,t}^l\in {ℝ}^{\left|\mathcal{E}\right|\times d}$ denote the temporal aggregation generated from short-term, medium-term and long-term historical information, respectively. $\left|\epsilon \right|$ denote the number of entities and $N_1<N_2<N_3$ represents the time window size, which is used to regulate the timescale of the input. ${\hat{\mathit{H}}}_{i,t}\in {ℝ}^{\left|\mathcal{E}\right|\times d}$ is a weighted sum of ${\mathit{H}}_{i,t}^s$, ${\mathit{H}}_{i,t}^m$ and ${\mathit{H}}_{i,t}^l$, controlled by predefined weighting coefficients ${\lambda }_1$, ${\lambda }_2$ and ${\lambda }_3$, respectively, and represents the final output of the temporal coding layer.

3.6. Decoder

The HCAE model employs ConvTransE [45] as the decoder. Its input consists of entity and relationship representations learned from the embedding propagation layer and the temporal encoding layer. The decoder outputs scores for quadruples, which are used to link the features learned by the encoder and compute the conditional probability of quadruples, thereby completing the TKGC task of the entire model. The scoring function can be represented as follows:

$$p\left(o|s_i,r,t\right)=\sigma \left(vec\left(\left[s_i:r\right]\ast \omega \right)W\right)o,$$

(11)

where $\left[s_i:r\right]$ represents the spliced subject entities and relations, $\omega$ is the convolution kernel matrix for performing convolution operations on the spliced vectors, ∗ denotes the convolution operation, W represents the linear transformation matrix of the fully connected layer, and $vec\left(\cdot \right)$ is a nonlinear function containing a convolution step.

3.7. Optimization

According to the above, the loss function of the HCAE model is defined as:

$$L={\displaystyle \sum _{t=0}^T{\displaystyle \sum _{\left(s_i,r,o,t\right)\in G_{t+1}}{\displaystyle \sum _{i=1}^N{y}_{i,t+1}}}}\log p_i\left(o|s_i,r,t\right),$$

(12)

where T is the number of timestamps controlling the number of sampled training quadruple, N denotes the number of entities, and $y_{i,t+1}$ is a labelled value that takes the value 0/1. A value of 1 denotes the existence of a quadruple of entity i in TKG at timestamp $t+1$. Otherwise, it is 0. In addition, this paper uses Adam as the optimization algorithm.

3.8. Applications

HCAE is a general framework that can be applied to different scenarios such as transportation, healthcare, social networks, economy, and finance. Take transportation scenarios as an example: (1) Construct TKG, using vehicles, routes, and events, etc. as entities: A specific vehicle travelling on a particular route during a certain period, and a special event occurring on a specific route, etc., are represented as relationships. Vehicle type, color, the route starting and ending points, and event type, etc. are used as entity attributes. (2) Encoding attribute information: This can reflect behavioral patterns under different conditions, such as route selection varying based on weekdays or weekends and other start and endpoints. It helps in learning the correlations between entities, such as certain events that significantly impact specific types of vehicles. It also provides decision-making foundations, such as planning public transportation routes based on road segment attributes. (3) Mining high-order structural information: Learning high-order relationships between vehicles and roads can reflect the way in which a vehicle drives certain road sections for an extended period, while high-order connections between roads can demonstrate the way in which congestion on a road section often affects neighboring road sections, and high-order relationships between events and related entities, such as a road construction events, affect related vehicles and road sections. (4) Multi-scale historical modeling: Long-term history can capture vehicle driving habits and patterns. A medium-term account can learn changes in road congestion over the past month or two. A short-term history can predict traffic peaks during daily traffic hours. Therefore, the HCAE model can predict future traffic conditions, traffic flow, and optimal route planning. Compared with the use of KG or historical data alone, HCAE effectively combines attribute, structural, and multi-scale historical information, improving traffic-related prediction and planning tasks.

4. Experimentation

4.1. Datasets

To evaluate the performance of the HCAE model, we will conduct experiments on the following three publicly available datasets: (1) Subset of the Global Database of Events, Language, and Tone (GDELT) known as GDELT [46]. (2) Two subsets of the Integrated Crisis Early Warning System (ICEWS), namely ICEWS14 and ICEWS05-15 [36]. Among these, GDELT records daily timestamped facts from 1 April 2015, to 31 March 2016, and contains the most common 500 entities and 20 common relations. ICEWS14 focuses on political and military events on a global scale and records facts between entities that occurred frequently in 2014. Due to limitations in its design, ICEWS14 is more limited in its temporal patterns, and geopolitical events between different entities are often not correlated in time. ICEWS05-15 records frequent entity-to-entity facts with a longer period from 2005 to 2015. To help select the best model parameters and hyperparameters, this paper divides the dataset into a training set (80%), a validation set (10%), and a test set (10%). The time interval indicates the time granularity between neighboring facts. More statistical information is shown in

Table 2. The statistics of datasets.

Dataset	Entities	Relations	Training	Validation	Test	Interval
GDELT	7691	240	1,734,399	238,765	305,241	15 min
ICEWS14	6869	230	74,845	8514	7371	24 h
ICEWS05-15	10,094	251	368,868	46,302	46,159	24 h

.

4.2. Baselines

To verify the validity of the HCAE model, comparative tests with the four SKGC methods and the existing state-of-the-art TKGC methods will be conducted in this paper. The introduction of the benchmark methods is described below:

• TransE [18] learns low-dimensional embeddings by translating entity and relationship vectors, achieving KGC tasks.
• DistMult [23] learns entity and relationship representations by using the inner product of entity and relationship triplets.
• ComplEx [24] represents entities and relationships as complex-valued embeddings and utilizes complex-valued inner products to capture symmetric/antisymmetric relationships, thereby improving KG link prediction performance.
• R-GCN [26] effectively captures complex relationships and contextual information between nodes.
• TTransE [31] innovatively embeds spatiotemporal information into KG, which provides richer contextual information for entity and relationship representation.
• TA-DistMult [36] is the first approach to introduce a time-sensitive attention mechanism in the TKGC task, which can enhance the presentation of relations across time.
• ChronoR [33], in which the representations of the relationship and timestamp are concatenated and applied as a rotation vector to the entity, enabling the temporal representation of the relationship to possess time dependency.
• TNTComplEx [34] decomposes the KG fourth-order tensor into material and non-temporal components and then specializes in learning the earthly manifestation of relations to achieve dynamic completion of KG.
• TeMP [37], in which, using message passing GNN, the structured representation of each entity at each time step is learned. The dynamic entity representation is obtained by aggregating all of the representations through an encoder.
• MtGCN [13] uses GCN to mine latent semantic information in TKG. GRU explicitly models historical information at different scales and uses the generated valid entity and relationship representations to complete TKGC.
• BoxTE [47] introduces dedicated time embedding representations, enabling the relationship representation at each time point to be unique and highly expressive.
• RoAN [41] directly encodes relationships and captures their temporal characteristics through a multi-chain structure and relationship attention mechanism.
• TASTER [35] learns global and local TKG embeddings using a sparse transition matrix, considering global statistics and local temporal evolution information.
• RotateQVS [48] represents entities and relationships in a quaternion vector space, utilizing rotation to represent the temporal evolution of relationships while also considering patterns such as symmetry and antisymmetry in relationships.

4.3. Evaluation Metrics

This paper adopts two commonly used metrics in the TKGC task, mean reciprocal rank (MRR) and Hits@{1, 3, 10}, which usually appear together. MRR denotes the average of the reciprocal rank of the correct answers in the prediction results, which is used to measure the model’s ranking of the correct answers in the prediction results, and higher values indicate higher model predicting ability, which is defined as follows:

$$MRR=\frac{1}{2\ast \left|G_{test}\right|}{\displaystyle \sum _{\left(s,r,o,t\right)\in G_{test}}\frac{1}{rank\left(s|r,o,t\right)}+\frac{1}{rank\left(o|s,r,t\right)}},$$

(13)

Hits@1, Hits@3, and Hits@10 denote the percentage of correct answers ranked in the top 1/3/10, respectively, reflecting the model’s accuracy in incorporating the correct answers into the top n predictions. The higher the value of Hits@{1, 3, 10} the more effective is the model, which is defined as follows:

$$Hits@n=\frac{1}{2\ast \left|G_{test}\right|}{\displaystyle \sum _{\left(s,r,o,t\right)\in G_{test}}\left(I\left(rank\left(s|o,r,t\right)\le n\right)+I\left(rank\left(o|s,r,t\right)\le n\right)\right)},$$

(14)

where $rank\left(s|r,o,t\right)$ and $rank\left(o|s,r,t\right)$ denote the ranking of the two-query subject and object entities, respectively. $I\left(\cdot \right)$ represents the indicator function.

4.4. Implementation and Hyperparameters

The HCAE model was implemented using PyCharm 2021.3.3 (Professional Edition) as the IDE platform. The operating system used was Windows 10 Home Chinese Edition, and the programming language utilized was Python 3.8. The CPU employed was an i7-7700HQ, and the PyTorch version utilized was 1.4. After the grid search, the hyperparameters of the HCAE model were set as follows: in this paper, the training set was first batch-processed, and the batch size set to 128. The number of training round epochs was set to 400. The learning rate lr was set to 0.01. The embedding dimensionality d of the entities and relationships was set to 128. The lengths of the time windows used to control the long-term, medium-term, and short-term historical information N₁, N₂, and N₃ were set to 2, 4, and 10, respectively. The weighting coefficients ${\lambda }_1$, ${\lambda }_2$, and ${\lambda }_3$ were 0.4, 0.4, and 0.2, respectively.

4.5. Experimental Results

To verify the validity of the HCAE model, this paper conducts a comparison experiment with the benchmark method in Section 4.2. It should be noted that the results of the comparison models are derived from the best results reported in the respective papers. For visualization, the MRR values are expanded by a factor of 100 in this paper, while the Hits@n values indicate percentages. The experimental results on the three datasets are shown in

Table 3. Comparison of prediction results on the GDELT, ICEWS14, and ICEWS05-15 datasets.

Models	GDELT				ICEWS14				ICEWS05-15
	MRR	H@1	H@3	H@10	MRR	H@1	H@3	H@10	MRR	H@1	H@3	H@10
TransE	11.3	0.0	15.8	31.2	28.0	9.4	-	63.7	29.4	9.0	-	66.3
DistMult	19.6	11.7	20.8	34.8	43.9	32.3	-	67.2	45.6	33. 7	-	69.7
ComplEx	22.8	15.8	24.0	36.3	24.5	16.1	27.5	41.1	-	-	-	-
R-GCN	23.3	17.2	25.0	34.4	26.3	18.2	30.4	45.3	-	-	-	-
TTransE	11.5	0.0	16.0	31.8	25.5	7.4	-	60.1	27.1	8.4	-	61.6
TA-DistMult	20.6	12.4	21.9	36.5	47.7	36.3	-	68.6	47.4	34.6	-	72.8
ChronoR(a)	-	-	-	-	59.4	49.6	65.4	77.3	68.4	61.1	73.0	82.1
ChronoR(b)	-	-	-	-	62.5	54.7	66.9	77.3	67.5	59.6	72.3	82.0
TNTComplEx	-	-	-	-	62.0	52.0	66.0	76.0	67.0	59.0	71.0	81.0
TeMP-SA	23.2	15.2	24.5	37.7	60.7	48.4	68.4	84.0	68.0	55.3	76.9	91.3
TeMP-GRU	27.5	19.1	29.7	43.7	60.1	47.8	68.1	82.5	69.1	56.6	78.2	91.7
MtGCN	23.4	14.7	25.9	40.7	35.0	25.8	38.9	53.0	-	-	-	-
BoxTE	35.2	26.9	37.7	51.1	61.3	52.8	66.4	76.3	66.7	58.2	71.9	82.0
RoAN-DES	29.0	18.7	31.5	49.6	58.8	47.6	66.1	78.8	59.9	47.9	67.9	82.3
RotateQVS	27.0	17.5	29.3	45.8	59.1	50.7	64.2	75.4	63.3	52.9	70.9	81.3
TASTER	-	-	-	-	61.1	52.7	-	76.7	65.4	56.2	-	81.8
HCAE	36.1	27.3	39.6	54.0	62.1	54.9	68.7	83.7	71.2	62.5	78.4	91.3

Note that—indicates that the original paper did not report results. Bold indicates the best results.

.

4.5.1. Comparison Experiment

The experimental results of the three datasets indicate that temporal knowledge graph completion methods generally outperform static methods. This is because TKG contains temporal information between facts, which provides crucial contextual information for time-sensitive link prediction tasks. Additionally, TKG can capture the patterns of facts changing over time, such as adding or deleting facts, which helps predict missing entities or relationships. HCAE demonstrates significant performance improvements compared with the four static methods. For example, compared with the best-performing static method R-GCN, HCAE achieves improvements of 10.1, 14.6, and 19.6 in Hits@1/3/10 on the GDELT dataset. This highlights the effectiveness of modeling temporal dependencies between facts.

The HCAE model was compared with several TKGC methods on the GDELT dataset, and from Table 3, it can be observed that HCAE outperformed TTransE by significant margins in all four metrics. Specifically, HCAE achieved improvements of 24.6, 27.3, 23.6, and 22.2 percentage points in the respective metrics. Because TTransE only models the translation operations of entities and relations, it cannot effectively model complex event evolution. Compared with TA-DistMult, HCAE achieved 15.5, 14.9, 17.7, and 17.5 improvements in the respective metrics. This is primarily because TA-DistMult relies on matrix factorization, which cannot handle large-scale datasets. Additionally, TeMP-SA and TeMP-GRU struggle to model complex long-term dependencies due to their reliance on temporal ordering information. Furthermore, MtGCN only considers direct relationships between entities, disregarding the rich attribute information present in TKG. These factors contribute to the significant differences in results when compared with HCAE. BoxTE achieved relatively similar results, with slight decreases of 0.9, 0.4, 1.9, and 2.9 in the four metrics. BoxTE models temporal order constraints and KG structural constraints while effectively modeling uneven time spans using the Box embedding model. It can handle the temporal characteristics of TKG well. However, BoxTE’s model tends to be overly smooth, leading to the loss of fine-grained information related to sudden events. This makes it challenging for BoxTE to identify and link such events. RoAN and RotateQVS exhibit similar results but differ significantly from HCAE in performance. This is because RoAN’s relation attention mechanism is too simplistic to learn complex temporal patterns thoroughly. At the same time, RotateQVS’s quaternion space assumption is overly idealized and unsuitable for capturing complex relationships in real-world TKGs. Additionally, both methods assume fixed time intervals, whereas time gaps often exhibit irregular patterns. HCAE achieves the best results because the GDELT dataset has a relatively small number of entities and relationships but a highly complex topology. Furthermore, some facts persist across multiple consecutive timestamps, while others are instantaneous and sparse. These benchmark methods struggle to extract and model structural information in such scenarios effectively. HCAE, on the other hand, has improved attribute embedding and embedding propagation layers, allowing it to adapt effectively to complex graph structures. It demonstrates strong capabilities in learning complex topology and temporal induction.

Table 3 presents the comparison results between HCAE and several TKGC methods on ICEWS14 and ICEWS05-15 datasets. On the ICEWS14 dataset, ChronoR(b) outperforms HCAE by 0.4 in terms of MRR metric, while TeMP-SA surpasses HCAE by 0.3 in the Hits@10 metric. On the ICEWS05-15 dataset, TeMP-GRU outperforms HCAE by 0.4 in Hits@10 metric. This is because the time information provided by ICEWS14, and ICEWS05-15 is limited. The relationships in these two datasets could be sparser as they involve fewer events such as negotiations, visits, and conflicts. Additionally, different entities are primarily unrelated in time and exhibit limited patterns, resulting in reduced richness and variability of temporal information in the dataset, relying more on entity-based predictions. On the other hand, HCAE models multi-scale historical details, to some extent depending on temporal information, leading to slightly lower performance than the baseline methods in specific cases. However, it is essential to note that HCAE significantly outperforms other baseline methods in most cases. For instance, it exceeds ChronoR(b) by 0.2, 1.8, and 6.4 percentage points in Hits@1/3/10 metrics. Additionally, the performance of TeMP-SA and TeMP-GRU is notably inferior to HCAE in other metrics. Specifically, HCAE is substantially improved when compared with recent TKGC methods such as RoAN, RotateQVS, and TASTER. Due to the inclusion of many unprecedented new events in event-based KG datasets, HCAE can exploit these new events’ historical and non-historical dependencies to uncover latent patterns that have not been observed. Ultimately, it is proven that HCAE, which integrates high-order connectivity and attribute information, is an effective TKGC method.

4.5.2. Model Robustness Analysis

To validate the robustness of the HCAE model, we conducted TKGC experiments on an irregular timestamp dataset called ICEWS14-irregular. The goal was to evaluate the performance of HCAE in handling unevenly distributed timestamp intervals. The construction of the dataset follows [43], where a snapshot graph is randomly sampled from ICEWS14, ensuring that the time interval between two consecutive snapshots does not exceed 4. The experimental results are presented in Figure 5.

From the experimental results, it can be observed that HCAE performs the best in all four metrics. Notably, it achieves significant improvement in Hits@1. For instance, it outperforms TeMP-SA by 11.5. This is because HCAE adopts the idea of multi-scale historical modeling, which allows it to explore temporal dependency patterns at different granularities without overly relying on a specific temporal regularity. On the other hand, TComplEx’s temporal modeling approach is relatively simple, representing temporal influence through a single time weight parameter with limited expressive power. TNTComplEx does not delve deep into modeling temporal regularities and primarily relies on linear transformations, which fail to capture the semantic information related to entities and time effectively. TeMP-SA only utilizes graph information within a fixed short time range for embedding learning, disregarding the rich graph information in other time intervals. T-GAP [12] employs time-shifted embeddings to encode different time differences. However, HCAE’s modeling of attribute information reduces the reliance on structural information and mitigates the impact of irregular timestamps. Moreover, mining high-order structural information enables the capture of more complex entity relationships, allowing HCAE to effectively withstand fluctuations in timestamp intervals and better adapt to irregular time gaps.

4.5.3. Parameter Sensitivity Analysis

Hyperparameters ${\lambda }_1$, ${\lambda }_2$ and ${\lambda }_3$ are used to control the importance of short-term, medium-term, and long-term historical information, respectively, for the task in the HCAE model. To investigate their impact on the performance of HCAE in knowledge graph completion, this paper conducted parameter sensitivity experiments on the ICEWS14 dataset. The values of these hyperparameters were gradually increased from 0.1 to 0.9 with a step size of 0.2. When exploring one hyperparameter, the values of the other two hyperparameters were fixed at 0.3. Additionally, other parameter settings remained unchanged.

A larger attention coefficient indicates a higher importance of the corresponding period’s historical information for the completion task. From Figure 6, it can be observed that the HCAE model performs the best when the value of ${\lambda }_1$ is 0.3. Subsequently, the performance gradually decreases as the weight of short-term historical information increases. This is because an excessive reliance on short-term history can lead to neglecting long-term patterns, resulting in the model overfitting the current situation. Short-term situations are more susceptible to various random factors and exhibit greater dynamism. Figure 7 shows that the HCAE performs best when the value of ${\lambda }_2$ is 0.5. The ICEWS14 dataset has one year, and the distribution of event times is relatively uniform. The medium-term history captures the development patterns of international relations over a specific period, and paying appropriate attention to medium-term history helps understand the global context. However, excessive reliance on medium-term history leads to neglect of current changes, making it challenging to explain sudden events and predict future trends. Figure 8 shows that the model achieves the best performance when the value of ${\lambda }_3$ is 0.7. Long-term history has a larger time window and can capture temporal dependencies overlooked by short-term and medium-term history. However, excessive reliance on long-term history fails to perceive the short-term patterns of new events, so an increase in ${\lambda }_3$ beyond a certain point leads to a decline in performance.

In conclusion, historical information from different periods plays unique roles, and neglecting the history from any period can result in a partial understanding of the evolution of events and difficulty in capturing global patterns. Therefore, when dealing with datasets with different patterns, it is advisable to adjust the hyperparameters ${\lambda }_1$, ${\lambda }_2$ and ${\lambda }_3$ appropriately to strike a balance between the influences of historical information from different periods, thus achieving optimal performance.

5. Conclusions

This paper proposes a temporal knowledge graph completion (TKGC) model that combines high-order connectivity between entities and attribute information. The model utilizes LSTM networks to extract rich semantic information from attribute values, providing additional supervision and addressing the problem of indistinguishable structural information for entities and relations with different meanings. Additionally, it employs graph attention networks to model high-order connectivity between entities, enhancing the model’s ability to learn complex relationships between entities, implicit dependencies between events, and constraints. Experimental results on the GDELT and ICEWS demonstrate that the proposed HCAE model performs well in terms of mean reciprocal rank (MRR) and Hits@n metrics, outperforming other baseline models. It is particularly suitable for addressing complex temporal evolution tasks involving unknown entities or relations. The model also holds significant value for practical engineering tasks. However, compared with other models, HCAE has a larger number of parameters and computational efficiency limitations. Future work will focus on designing a more lightweight yet powerful model and improving its applicability in real-world engineering tasks.