A Brief Survey on Deep Learning-Based Temporal Knowledge Graph Completion

Abstract

Temporal knowledge graph completion (TKGC) is the task of inferring missing facts based on existing ones in a temporal knowledge graph. In recent years, various TKGC methods have emerged, among which deep learning-based methods have achieved state-of-the-art performance. In order to understand the current research status of TKGC methods based on deep learning and promote further development in this field, in this paper, for the first time, we summarize the deep learning-based methods in TKGC research. First, we detail the background of TKGC, including task definition, benchmark datasets, and evaluation protocol. Then, we divide the existing deep learning-based TKGC methods into eight fine-grained categories according to their core technology and summarize them. Finally, we conclude the paper and present three future research directions for TKGC.

1. Introduction

Knowledge graphs (KGs) represent real-world information as structured facts in the form of triples, i.e., (subject entity, relation, object entity), abbreviated as $(s,r,o)$, where s and o represent entities, and r represents a relation that holds between the two entities. For example, given a triple (Barack Obama, PresidentOf, USA), it indicates that there is a relation “PresidentOf” between the entities “Barack Obama” and “USA”. The rich information carried by KGs makes them popular in various artificial intelligence and data mining applications, such as question answering [1], recommender [2], and dialogue systems [3], etc. Although current KGs consist of billions of facts, they are still far from complete. For instance, in Freebase [4], more than 75% of person entities have no known nationality [5], which hampers their usefulness in the aforementioned applications.

Knowledge graph completion (KGC) tries to solve this incompleteness problem by inferring new triples based on the existing ones. It operates under the assumption that KGs are static, i.e., the ground truth of triples is independent of time. However, this assumption is not always realistic, e.g., as shown in Figure 1, the triple (Barack Obama, PresidentOf, USA) is only valid during the period [January 2009, January 2017]. Indeed, many real-world KGs such as Wikidata [6], YAGO3 [7], and ICEWS include temporal information for their facts. Specifically, temporal facts are facts of the form of quadruple $(s,r,o,t)$, where s and o are entities, r is a relation, and t is a timestamp. It indicates that, at a time specified by t, there exists a relation r between entities s and o. Consequently, temporal knowledge graph completion (TKGC) has become an important research topic, which aims to automatically predict missing facts by making inferences from the existing temporal facts.

As TKGC has attracted increased attention from the research community, various TKGC methods have been proposed, among which, deep learning (DL)-based methods have achieved state-of-the-art performance. Therefore, in order to promote further progress in this field, it is necessary to comprehensively summarize and compare the existing DL-based TKGC methods. We note that: (1) Ref. [8] provides a review of temporal knowledge graphs, which includes a section on TKGC. However, this section only covers five early methods related to TKGC. (2) Ref. [9] provides a review of temporal knowledge graphs, focusing on representation learning-based TKGC methods. (3) There are three review papers [10,11,12] on TKGC, but they either mainly focus on the interpolation-based methods or, perhaps due to space limitations, introduce TKGC methods from a macro perspective. Most importantly, the above reviews do not include research work from the last one to two years.

Figure 1. A knowledge graph example that contains temporal information [10].

Figure 1. A knowledge graph example that contains temporal information [10].

This paper reviews TKGC methods from the perspective of deep learning for the first time, and the proportions of the latest research after 2023 and the literature not included in the previous reviews reach 50%, respectively.

Our main contributions are summarized as follows:

• For the first time, we propose a survey that analyzes the existing DL-based TKGC methods to date and further subdivide them into eight categories according to their core techniques.
• We summarize eight common benchmark datasets and the general evaluation protocol.
• We point out three future directions for TKGC, providing new ideas for the work of relevant researchers.

The remainder of this paper is organized as follows. Section 2 introduces the background of TKGC, including task definition, benchmark datasets, and evaluation protocol. Section 3 summarizes the DL-based TKGC methods and Section 4 summarizes the experimental results of these methods. Section 5 concludes this paper and highlights the future directions of TKGC.

2. Background

In this section, we first provide the definitions of TKG and TKGC and then introduce available benchmark datasets and the general evaluation protocol.

2.1. Task Definition

A temporal knowledge graph (TKG) is a multi-relational directed graph with labeled timestamped edges between entities. It is usually expressed as $\mathcal{G}=\left(\mathcal{F}\right|\mathcal{E},\mathcal{R},\mathcal{T})$, where $\mathcal{E}$, $\mathcal{R}$, and $\mathcal{T}$ are the sets of entities, relations, and timestamps, respectively, and $\mathcal{F}\subset \mathcal{E}\times \mathcal{R}\times \mathcal{E}\times \mathcal{T}$ is the set of facts that represent real events in the world. A fact f is denoted as $(s,r,o,t)$, where $s\in \mathcal{E}$, $r\in \mathcal{R}$, $o\in \mathcal{E}$, and $t\in \mathcal{T}$ are the subject (head) entity, relation, object (tail) entity, and timestamp, respectively.

It is worth mentioning that, in TKGs, the timestamp is attached to each fact indicating when it is valid. According to the form of the timestamp, facts in TKGs can be classified into two categories [13]: (a) discrete fact, in which the valid time of the fact is a time point, denoted as $(s,r,o,t_p)$, and (b) continuous fact, in which the valid time of the fact is a time period, denoted as $(s,r,o,[t_b,t_e])$, with $t_b$ and $t_e$ representing the beginning and the end of the valid time.

Temporal knowledge graph completion (TKGC) is the task of inferring missing facts based on the set of observed facts, which usually includes three subtasks: subject entity prediction, object entity prediction, and relation prediction.

(i)

Subject entity prediction: for an incomplete temporal fact $(?,r,o,t)$, predict the subject entity s from the entity set $\mathcal{E}$ at timestamp $t\in \mathcal{T}$.

(ii)

Object entity prediction: for an incomplete temporal fact $(s,r,?,t)$, predict the object entity o from the entity set $\mathcal{E}$ at timestamp $t\in \mathcal{T}$.

(iii)

Relation prediction: for an incomplete temporal fact $(s,?,o,t)$, predict the relation r from the relation set $\mathcal{R}$ at timestamp $t\in \mathcal{T}$.

In addition to the above three subtasks, several studies [14,15] have attempted to predict temporal information, namely, time prediction, which aims to predict the timestamp for which the given triple is consider valid, i.e., $(s,r,o,?)$. Depending on the timestamp range, TKGC has two settings: interpolation and extrapolation. Specifically, for a TKG from timestamp $t_0$ to timestamp $t_T$, the interpolation setting aims to infer the facts with timestamp $t_0<t<t_T$, and the extrapolation setting aims to predict future facts that occur at timestamp t with $t>t_T$. This paper focuses on the DL-based TKGC methods that adopt the interpolation setting.

2.2. Benchmark Datasets

There are eight commonly used datasets for TKGC, and the main statistics for these datasets are summarized in

Table 1. Statistics for TKGC benchmark datasets. #Entities, #Relations, #Timestamps, #Train, #Validation, and #Test denote the number of entities, relations, timestamps, training set, validation set, and test set, respectively.

Dataset	#Entities	#Relations	#Timestamps	#Train	#Validation	#Test	Timestamps
ICEWS14	6869	230	365	72,826	8941	8963	time point
ICEWS05-15	10,094	251	4017	368,962	46,275	46,092	time point
ICEWS18	23,033	256	304	373,018	45,995	49,545	time point
GDELT	500	20	366	2,735,685	341,961	341,961	time point
YAGO11k	10,623	10	70	16,406	2050	2051	time interval
YAGO15k	15,403	34	198	29,381	3635	3685	time interval
Wikidata12k	12,554	24	81	32,497	4062	4062	time interval
WIKI	12,554	24	232	539,286	67,538	63,110	time interval

.

ICEWS14, ICEWS05-15, and ICEWS18. These three datasets are subsets of Integrated Crisis Early Warning Systems (ICEWS). ICEWS processes vast amounts of data from digitized news, social media, and other sources to predict, track, and respond to global events with time points, primarily for early warning purposes. These three datasets contain events in 2014, 2005–2015, and 2018, respectively.

GDELT. This dataset is a subset of the larger Global Database of Events, Language, and Tone (GDELT) [16]. GDELT is a comprehensive global database that includes broadcast, print, and web news from every country starting in 1979. The GDELT subset contains facts with daily timestamps in the form of time points between 1 April 2015 and 31 March 2016. It should be emphasized that GDELT exhibits a complex geometry characterized by a small number of nodes and a substantial volume of training data.

YAGO11k, YAGO15k, WIKI, and Wikidata12k. These datasets are subsets of YAGO3 [7] and Wikidata [6]. YAGO3 and Wikidata are two large-scale TKGs where a portion of the facts involves temporal information. Specifically, YAGO3 is a linked database developed by the Max Planck Institute in Germany, which integrates data from Wikipedia, WordNet, GeoNames, and other data sources [7]. Wikidata is a Wikimedia hosted and maintained project launched in 2012 that aims to create a free knowledge graph about the world that can be read and edited by both humans and machines. Notably, different from ICEWS dataset that only involves time points, temporal information in YAGO3 and Wikidata is expressed in different forms, including time points such as 1 January 2000, time intervals containing begin or end time with shapes such as ‘occursSince 2003’ or ‘occursUntil 2005’, and time intervals containing both begin and end time, such as [2000, 2010]. In YAGO11k and Wikidata12k, all facts have temporal information in the form of time intervals with begin and end times. Among these two subsets, the frequency distribution of timestamps in YAGO11k exhibits the long-tail property, which makes it difficult to predict. In YAGO15k, temporal information is represented in the form of time intervals containing begin or end time, and some facts do not even include timestamps. Therefore, this subset is temporally incomplete and more challenging.

2.3. Evaluation Protocol

The evaluation protocol describes how to measure the performance of the TKGC method. Typically, for a test quadruple $(s,p,o,t)$, it generates corresponding corrupted quadruples for the subtasks entity predict, relation predict, and time prediction, respectively. For entity predict, the corrupted quadruples are generated by replacing either the subject entity (for subject entity predict) or the object entity (for object entity predict) with all possible entities such that the resulting corrupted quadruples do not belong to the TKG, i.e., $C_s=\{(s^{\prime },r,o,t):s^{\prime }\in E,(s^{\prime },r,o,t)\notin G\}$ or $C_o=\{(s,r,o^{\prime },t):o^{\prime }\in E,(s,r,o^{\prime },t)\notin G\}$. The corrupted quadruples of relation prediction and time prediction are generated in a similar way: $C_r=\{(s,r^{\prime },o,t):r^{\prime }\in R,(s,r^{\prime },o,t)\notin G\}$ and $C_t=\{(s,r,o,t^{\prime }):t^{\prime }\in T,(s,r,o,t^{\prime })\notin G\}$, respectively. Then, the corrupted quadruples and test quadruple on each subtask are sorted according to the scoring function to obtain the rank of the test quadruple.

Commonly used evaluation metrics include Mean Rank (MR), Mean Reciprocal Ranking (MRR), and Hits@N. MR is the average rank for all test quadruples, and MRR is the inverse of MR. Hits@N is the percentage of test quadruplets with ranks lower than or equal to N.

3. Deep Learning-Based TKGC Methods

The papers included in this survey were identified by searching for the phrase “temporal knowledge graph completion” in Google Scholar. We observed that topics tended to drift away from our focus after 100 search results, so we stopped the investigation there. Given our focus on the latest research trends in this field, we restricted candidate papers to those published after 1 January 2018.

Since the candidate papers use different kinds of deep learning algorithms for processing temporal information in TKGs, the deep learning-based TKGC methods can be further subdivided according to different algorithms. From another perspective, the candidate papers can also be subdivided according to different representations of temporal information, such as independent representation of temporal information, representation that integrates temporal information and entities, representation that integrates temporal information and relations, and so on. In order to highlight the technical details and underlying logic of the TKGC methods, this survey uses the former to subdivide them, and the specific subdivision is shown in Table 2. It is worth mentioning that, considering that some TKGC methods are extensions of static KGC methods, we also list the corresponding static KGC methods in the table and give a general introduction to them in the next paragraph.

TransE [17] is a classic model of static KGC, which maps entities and relations into a continuous embedding (i.e., vector) space, and which regards relation embedding as a translation operator from subject entity embedding to object entity embedding. TransR [18] is a variant of TransE that uses relation-specific spaces to handle different relations, mining more semantic information. DISTMULT [19] learns embeddings using a bilinear diagonal model, which is a special case of the bilinear objective used in TransE. CapsE [20] applies capsule network (CapsNet) [21] to model the entries at the same dimension in entity and relation embeddings. SimplE [22] is a tensor factorization approach based on the canonical polyadic for static KGC. ConvE [23] first applies convolutional neural network (CNN) [24] to the static KGC, which uses 2D convolution to model the interactions between entity and relation embeddings. ConvKB [25] improves ConvE by exploring the global relationships among same dimensional entries of entity and relation embeddings. ConvTransE [26] can be seen as a combination of ConvE and TransE, which uses a convolutional network to model relations as translation operations and preserves the translation characteristic between entities and relations.

Table 2. Fine-grained categorization of deep learning-based temporal knowledge graph completion methods.

Categories	TKGC Methods	Extended from Static KGC Methods	Encoder–Decoder
LSTM-based methods	TA-TransE [27]	TransE [17]	✘
	TA-DISTMULT [27]	DISTMULT [19]	✘
	TA-TransR  [28]	TransR [18]	✘
CNN-based methods	CATKGC [29]	✘	✘
	ConvTKG [30]	✘	✔
	SANe [31]	✘	✘
	SANe+ [32]	✘	✘
CapsNet-based methods	TempCaps [33]	CapsE [20]	✘
	BiQCap [34]	✘	✘
Attention-based methods	RoAN [35]	SimplE [22]	✘
	HSAE [36]	✘	✔
GNN-based methods	TGAP [37]	✘	✔
	STRGNN [38]	✘	✔
	HyGNet [39]	✘	✘
GCN-based methods	TeMP [40]	TransE [17]	✔
	MtGCN [41]	ConvTransE [26]	✔
	TAGCN [42]	ConvE [23]	✔
	TAL-TKGC [43]	✘	✘
	THOR [44]	✘	✘
GAN-based methods	IAGAT [45]	✘	✘
	DEGAT [46]	ConvKB [25]	✔
	T-GAE [47]	ConvKB [25]	✔
Language model-based methods	SToKE [48]	✘	✘
	Llama-2-7b-CoH [49]	✘	✘
	Vicuna-7b-CoH [49]	✘	✘

Table 2. Fine-grained categorization of deep learning-based temporal knowledge graph completion methods.

Categories	TKGC Methods	Extended from Static KGC Methods	Encoder–Decoder
LSTM-based methods	TA-TransE [27]	TransE [17]	✘
	TA-DISTMULT [27]	DISTMULT [19]	✘
	TA-TransR  [28]	TransR [18]	✘
CNN-based methods	CATKGC [29]	✘	✘
	ConvTKG [30]	✘	✔
	SANe [31]	✘	✘
	SANe+ [32]	✘	✘
CapsNet-based methods	TempCaps [33]	CapsE [20]	✘
	BiQCap [34]	✘	✘
Attention-based methods	RoAN [35]	SimplE [22]	✘
	HSAE [36]	✘	✔
GNN-based methods	TGAP [37]	✘	✔
	STRGNN [38]	✘	✔
	HyGNet [39]	✘	✘
GCN-based methods	TeMP [40]	TransE [17]	✔
	MtGCN [41]	ConvTransE [26]	✔
	TAGCN [42]	ConvE [23]	✔
	TAL-TKGC [43]	✘	✘
	THOR [44]	✘	✘
GAN-based methods	IAGAT [45]	✘	✘
	DEGAT [46]	ConvKB [25]	✔
	T-GAE [47]	ConvKB [25]	✔
Language model-based methods	SToKE [48]	✘	✘
	Llama-2-7b-CoH [49]	✘	✘
	Vicuna-7b-CoH [49]	✘	✘

3.1. LSTM-Based Methods

Long short-term memory (LSTM) [50] is a special type of recurrent neural network that can learn long-term dependencies in time series data, such as machine translation [51] and speech recognition [52]. Given the input data at time t, i.e., $x_t$, LSTM performs the following computation:

$$\begin{array}{cc}\hfill & i_t=sigmoid(W_{ii}{x}_t+b_{ii}+W_{hi}{h}_{t-1}+b_{hi})\hfill \\ \hfill & f_t=sigmoid(W_{if}{x}_t+b_{if}+W_{hf}{h}_{t-1}+b_{hf})\hfill \\ \hfill & o_t=sigmoid(W_{io}{x}_t+b_{io}+W_{ho}{h}_{t-1}+b_{ho})\hfill \\ \hfill & \widehat{c_t}=tanh(W_{ic}{x}_t+b_{ic}+W_{hc}{h}_{t-1}+b_{hc})\hfill \\ \hfill & c_t=f_t\ast c_{t-1}+i_t\ast \widehat{c_t}\hfill \\ \hfill & h_t=o_t\ast tanh\left(c_t\right),\hfill \end{array}$$

(1)

where $i_t,f_t$, and $o_t$ are the input gate, forget gate and output gate, respectively; $c_t$, $\widehat{c_t}$ are the cell state and candidate cell state, respectively; and $h_t$ is the hidden state. The various W and b terms are the weight matrices and bias vectors.

Garcia-DuranDN and Niepert [27] convert quadruples into triples by merging timestamps into relations, i.e., $(s,r,o,t)$ to $(s,r+t,o)$, and apply LSTM to encode the merged relations. Then, they extend two classic static KGC methods, TransE [17] and DISTMULT [19], and propose TA-TransE and TA-DISTMULT, respectively. TA-TransE models merge relations as translation operations into the embeddings of subject and object entities and use a distance-based scoring function $f(s,r+t,o)=\left|\right|e_s+e_{r+t}-e_o{\left|\right|}_2$ to measure the plausibility of a triple fact, where $e_s$, $e_o$, and $e_{r+t}$ are the embeddings of subject entity, object entity, and merged relation, respectively. The essential idea of TA-DISTMULT is the same as that of TA-TransE; the only difference is the score function, which is $f(s,r+t,o)=(e_s\otimes e_o)e_{r+t}^{\top }$, where ⊗ is the element-wise product, and ⊤ represents the matrix transpose.

Since LSTM can handle sequences of varying lengths, a natural advantage is that the merged relation is adaptive to timestamps of different relations, such as points or intervals. Inspired by [27], Chen et al. [28] present TA-TransR, which expands the static KGC method TransR [18] to TKGC, and the scoring function is $f(s,r+t,o)=\left|\right|(e_s-e_s{M}_r)+e_{r+t}-(e_o-e_o{M}_r){\left|\right|}_2^2$, where $M_r$ is the projection matrix corresponding to relation r. Furthermore, they impose three kinds of temporal consistency constraints (i.e., temporal disjointness, temporal ordering, and temporal spans) on TA-TransR, and formulate TKGC task as an integer linear programming optimization problem.

Methods in this category combine LSTM with existing classical static KGC methods to learn time-aware relations or entities for TKGC. They have clear motivations, simple implementations, and are efficient, but they are unable to model the graph structure inherent in TKGs.

3.2. CNN-Based Methods

Convolutional neural networks (CNNs) [24] have achieved remarkable performance in the field of computer vision, such as object detection [53] and image segmentation [54]. The core operation of CNN is to extract different feature maps of input data through convolution kernels. Formally, the feature value at location $(i,j)$ in the k-th feature map is calculated by:

$$z_{i,j,k}=W_k{x}_{i,j}+b_k,$$

(2)

where $W_k$ and $b_k$ are the weight matrix and bias vector of the k-th convolution kernel, respectively, and $x_{i,j}$ denotes the overlapped input data centered at location $(i,j)$.

Liu et al. [29] propose a context-aware TKGC method (CATKGC), which consists of two modules: one for context selection and the other for modeling temporal interactions. Specifically, the context selection module decomposes a fact into two parts, i.e., relation and entities, and measures the usefulness of the context for each part, respectively. Then, a CNN is used to encode context and extract features of temporal interactions, based on which a fully connected layer is adopted to learn the temporal consistency score. Finally, the consistency score is used together with a plausibility score to predict the correctness of a fact.

He et al. [30] propose a query-aware (a query is an incomplete quadruple) embedding model based on a convolutional neural network, named ConvTKG, which has an encoder–decoder architecture. In the encoder, it converts the timestamp into a sequence of atomic-level temporal tokens. Then, these tokens are fed into a gate recurrent unit (GRU) layer and an attention layer to obtain query-aware representations of timestamps. In the decoder, multiple one-dimensional kernels are employed to extract global features among elements in the query, yielding various feature maps that are then reduced to feature vectors through multi-layer perception (MLP). Finally, the feature vectors of a query are computed with the candidate entity embeddings to produce the plausibility scores of candidate entities.

SANe [31] extends CNN to map the facts with different timestamps into different latent spaces. Essentially, it turns TKGC into static KGC by handling different temporal snapshots in separate spaces. In particular, a time-aware parameter generator (TaPG) is designed to constrain the overlap of latent spaces according to the distance of the timestamps, which endows the CNN with specific parameters in terms of the context of timestamps. Lee et al. [32] present an expanded version of SANe by introducing an adaptive parameter generator (AdaPG), which overcomes the limitations of TaPG in handling unbalanced TKGs through allocating parameters based on the number of facts instead of timestamps.

Methods in this category use CNNs to model temporal information and its interactions, which have more powerful feature learning capabilities than the above LSTM-based methods but which are still unable to model the graph structure of TKGs.

3.3. CapsNet-Based Methods

Capsule network (CapsNet) [21] can be seen as an improved version of CNN, which can encode the intrinsic spatial relationships between a part and the whole in images. It consists of two key components: capsules and a dynamic routing mechanism. A capsule is a group of neurons that processes different information about an object, and the length of its output vector represents the probability that the object represented by the capsule is present in the current input. To guarantee the length of the vector stays between 0 and 1, a squash function is used:

$${\mathbf{v}}_j={\displaystyle \frac{\left|\right|{\mathbf{s}}_j{\left|\right|}^2}{1+\left|\right|{\mathbf{s}}_j{\left|\right|}^2}}{\displaystyle \frac{{\mathbf{s}}_j}{\left|\right|{\mathbf{s}}_j\left|\right|}},$$

(3)

where ${\mathbf{s}}_j$ is the input of a capsule, and ${\mathbf{v}}_j$ is its squashed output.

The dynamic routing mechanism specifies how capsules communicate with each other between layers. Specifically, all output vectors ${\mathbf{u}}_i$ of capsules in the lower layer are first multiplied by a weight matrix ${\mathbf{W}}_{ij}$. Then, the weighted sum of newly obtained vectors are input into a capsule ${\mathbf{s}}_j$ in the next layer:

$${\widehat{\mathbf{u}}}_{j|i}={\mathbf{W}}_{ij}{\mathbf{u}}_i,{\mathbf{s}}_j=\sum _i{c}_{ij}{\widehat{\mathbf{u}}}_{j|i},$$

(4)

where $c_{ij}$ is the coupling coefficient between capsule i and capsule j.

TempCaps [33] extends the static KGC method CapsE [20] to TKGC, which consists of a neighbor selector, a dynamic routing aggregator, and a multi-layer perceptron (MLP) decoder. Specifically, it first selects two types of neighboring entities for each entity, namely local entities and global relation entities, through the neighbor selector. Then, it uses the dynamic routing aggregator to learn the embedding of the entity based on the retrieved neighbors. Finally, the score for each candidate entity is generated by feeding the embeddings of the entities in the candidate set into the MLP decoder.

BiQCap [34] combines biquaternions and capsule networks for TKGC. Specifically, it first embeds entities, relations, and timestamps into the biquaternions of k-dimension, i.e., ${\mathbf{Q}}_s$, ${\mathbf{Q}}_r$, ${\mathbf{Q}}_o$, ${\mathbf{Q}}_t\in {\mathbb{R}}^k$, where timestamps are represented as translations of the subject entities, and relations are represented as rotations in the biquaternion space after translation. On this basis, a distance-based score function is designed as $f(s,r,o,t)=\left|\right|(({\mathbf{Q}}_s+{\mathbf{Q}}_t)\odot {\mathbf{Q}}_r)\otimes \mathbf{b}-{\mathbf{Q}}_o\left|\right|$, where $\mathbf{b}\in {\mathbb{R}}^k$ is a bias vector, and ⊙ and ⊗ denote the Hamilton product and element-wise multiplication, respectively. Then, the trained embeddings of entities, relations, and timestamps from biquaternions are used as the input to the capsule network, and a novel routing algorithm called inverted dot-product attention is proposed, which determines the routing probability directly through the agreement between parent and child capsules.

Methods in this category have similar characteristics to the CNN-based methods. Since CapsNet is an improved version of CNN, it inherits the characteristics of CNN and can model local and global spatial relationships. Therefore, methods in this category can theoretically better characterize the spatial characteristics of entities in TKGs.

3.4. Attention-Based Methods

The attention mechanism [55] is popular in sequence modeling tasks such as machine translation [56] and speech recognition [57]. Formally, the relevant score of the i-th data in the input sequence and the j-th data in the output sequence is computed as:

$$e_{ji}=a(S_i^{in},S_j^{out}),$$

(5)

where $a\left(\right)$ represents the alignment function, which can be implemented using a neural network, $S_i^{in}$ is the i-th data in the input sequence, and $S_j^{out}$ is the j-th data in the output sequence. In addition, the softmax function is often used to compress the relevance scores into positive values between 0 and 1.

Bai et al. [35] restructure TKG into a form of relational multi-chains and propose a relation-oriented attention mechanism to capture the features of relations while considering temporal information. Then, they employ DE-SimplE [58] to obtain the features of entities. Finally, these features are fed into the scoring function of the static KGC model SimplE [22] to calculate the plausibility scores of the given facts.

Inspired by the self-attention mechanism [59] and the diachronic embedding technique [58], Ren et al. [36] propose HSAE, which has an encoder–decoder architecture. The encoder module is composed of two layers—an entity self-attention embedding layer and a relation self-attention embedding layer—which are used to obtain embeddings of entities and relations containing time series information and graph structure information, respectively. The decoder module feeds these embeddings into a scoring function to compute their plausibility scores.

Methods in this category use the attention mechanism to model the importance of entities and/or relations in TKGs. They have the characteristics of theoretical simplicity but usually need to be combined with other methods to complete the TKGs.

3.5. GNN-Based Methods

A graph neural network (GNN) [60] is an effective model that operates directly on graph-structured data, such as social networks [61] and road networks [62]. It follows a recursive neighborhood aggregation (or message passing) scheme, where the feature vector of a node (i.e., entity) is computed by recursively aggregating and transforming embeddings of its neighboring nodes. Formally, the computation of the feature vector of each node can be described as:

$$\begin{array}{cc}\hfill & m_v^{\left(k\right)}=AGGREGAT{E}^{\left(k\right)}\left(\{h_u^{(k-1)}:u\in N\left(v\right)\}\right),\hfill \\ \hfill & h_v^{\left(k\right)}=UPDAT{E}^{\left(k\right)}(h_v^{(k-1)},m_v^{\left(k\right)}),\hfill \end{array}$$

(6)

where $m_v^{\left(k\right)}$ and $h_v^{\left(k\right)}$ stand for the message vector and the feature vector of node v at the k-th layer, respectively. $AGGREGATE\left(\right)$ and $UPDATE\left(\right)$ are the aggregation function (e.g., mean, average, and max pooling) and the update function (e.g., linear-layer combination) [63], respectively.

TGAP [37] views timestamps as properties of relations between entities and has an encoder–decoder architecture. In the encoder, a temporal GNN is introduced to aggregate query-related (a query is a incomplete quadruple) information from the local neighborhood of each entity, focusing on encoding the temporal displacement between the timestamp of the input query and each relation being encoded. In addition, a decoder based on attention propagation is proposed. Different from the score function decoder, the attention propagation decoder propagates attention through existing edges (i.e., relations) in the graph and determines the plausibility of facts based on the attention values. It naturally allows multi-hop reasoning, and the inferred attention distribution provides better interpretability.

Wang et al. [38] propose a semantic-driven, time-aware relational GNN model, named STRGNN, for the TKGC task, which consists of a semantic-enhanced encoder and a convolution-based decoder. In the encoder, it first abstracts a subgraph for the given subject entity s (given an incomplete quadruple $(s,r,?,t)$ as an example) according to both its temporal neighborhoods and semantically similar neighborhoods. Then, it generates time-aware representation vectors for the temporal neighbors and performs aggregation. In the decoder, after updating the time-aware representation of the subject entity s at timestamp t, a convolution-based decoder is introduced to calculate the score of each candidate object entity.

HyGNet [39] exploits hyperbolic geometry to model temporal knowledge graphs, which comprises two key components: the Hyperbolic Gated Graph Neural Network (HGGNN) and the Hyperbolic Convolutional Neural Network (HCNN). HGGNN aggregates neighborhood information in hyperbolic space, aiming to capture the contextual information and dependencies between entities. HCNN interacts with embeddings in hyperbolic space to model the complex interactions among entities, relations, and timestamps. Finally, a distance-based score function is introduced to measure the plausibility of facts.

Methods in this category use GNN to model the graph structure in TKGs, assuming that similar entities have similar graph structures. However, relying solely on structural information limits the completion performance. Therefore, methods in this category usually combine with other mechanisms to model information other than structural information in TKGs, such as relations, entity descriptions, and attributes, etc., for TKGC.

3.6. GCN-Based Methods

Graph convolutional network (GCN) [64,65] is a variant of GNN that uses a convolution-like operation to aggregate the features of each node’s corresponding neighbor nodes and then performs a non-linear transformation to update the feature vector of each node. Formally, at the k-th layer, the feature vector of each node is computed as (the aggregate and update function are integrated):

$$h_v^{\left(k\right)}=\sigma (\sum _{u\in N\left(v\right)\cup \left\{v\right\}}W{h}_u^{(k-1)}),$$

(7)

where W is a weight matrix, $\sigma$ is the activate function ReLU, and $N\left(v\right)$ is the set of neighbor nodes of v.

TeMP [40] is a GCN-based method that has an encoder–decoder structure. The encoder maps each entity to a time-dependent low-dimensional embedding at each time step, while the decoder uses these entity embeddings to score the likelihood of the facts. Specifically, it discretizes a TKG into multiple static KG snapshots and generates dynamic entity features along two dimensions: structural neighborhoods and temporal dynamics. Structural encoder uses GNN to extract features from local node neighborhoods in each snapshot through message passing and aggregation, while temporal encoder captures feature evolution over multiple time steps by sequential models. To the decoder, the score function of TransE [17] is applied to calculate the plausibility of the facts.

MtGCN [41] also has an encoder–decoder structure, which consists of three components: structural encoder, temporal encoder, and decoder. The structural encoder uses relation-aware GCN to capture the topology of each time step in the knowledge graph and generate entity representations at each timestamp. Then, the temporal encoder uses a sequence-based model GRU to integrate the output of the structure encoder and learn historical information at different time scales to generate entity representations that contain structural and temporal features. The decoder is a scoring function, specifically the scoring function of the static KGC method ConvTransE [26], which predicts missing entities or relations in the facts in a conditional probability manner.

TAGCN [42] can simultaneously extract structural information and temporal dependency and is also in the architecture of encoder–decoder. It decomposes TKGC into two phases, namely, neighboring temporal message aggregation and entity temporal focus attachment. Accordingly, on the encoder side, a temporal self-attention layer is defined to capture the temporal dependency of the neighboring fact, and a temporal message aggregation module is defined to extract neighborhood structure in TKG and increase the expressiveness of entities. The decoder side chooses the score function of ConvE [23] to estimate the plausibility of the facts.

Nie et al. [43] propose TAL-TKGC, which consists of three key components: temporal attention module, TKG embedding module, and completion module. The temporal attention module is designed to capture the deep connection between timestamps, entities, and relations at the semantic level. The TKG embedding module embeds entities using an importance-weighted GCN, which considers both the temporal semantic importance and the structural importance of entities. The completion module uses TTransE [14], which treats the plausibility of facts as the distance between the subject and object entities in the vector space that integrates relations and temporal information.

Lee et al. [44] propose a self-supervised approach named THOR (Three-tower grapH cOnvolution networks), which extracts latent knowledge from TKG by jointly exploiting temporal and non-temporal dependencies between entities and structural dependencies between relations. Specifically, they design three-tower GCNs with different roles: Temporal GCN (TempGCN for short), aTemporal GCN (aTempGCN for short), and Relational GCN (RelGCN for short). Given a fact $(s,r,o,t)$, THOR obtains the temporal embeddings of s and o relative to t (via TempGCN), the atemporal embeddings of s and o (via aTempGCN), and the embedding of r (via RelGCN). Finally, THOR learns embeddings of entities and relations with the following two objectives: (1) maximize the likelihood of facts in TKG; (2) make the temporal embedding of each entity more accurate. For (1), they adopt the negative log-likelihood loss, and for (2), they propose a time-aware self-supervised loss that can learn auxiliary supervision signals for each entity.

Methods in this category have similar characteristics as the GNN-based methods. Since GCN is a special version of GNN, which uses the convolution-like operation to aggregate information, to some extent, methods in this category can be regarded as a subset of the GNN-based methods.

3.7. GAN-Based Methods

A graph attention network (GAN) [66] is an improved model of GCN, which applies an attention mechanism to impose different weights on nodes in the neighborhood in the aggregation step. Formally, the feature vector of each node in the k-th layer is computed as:

$$h_v^{\left(k\right)}=\sigma (\sum _{u\in N\left(v\right)\cup \left\{v\right\}}{\alpha }_{vu}W{h}_u^{(k-1)}),$$

(8)

in which

$${\alpha }_{vu}=softmax\left(e_{vu}\right)={\displaystyle \frac{exp\left(e_{vu}\right)}{{\sum }_{j\in N\left(v\right)}exp\left(e_{vj}\right)}},e_{vu}=a(W{h}_v^{(k-1)},W{h}_u^{(k-1)}),$$

(9)

where W is a weight matrix; ${\alpha }_{vu}$ represents the weight value of node u for node v; $\sigma$ indicates the activate function, e.g., ReLU; $e_{vu}$ indicates the importance of node u’s feature to node v; and $a\left(\right)$ represents the attention mechanism and is a single-layer feed-forward neural network.

Chen et al. [45] divide the attribute information of entities into inherent attributes and temporal attributes and propose IAGAT for TKGC. IAGAT utilizes GAN to learn the inherent embeddings of entities and relations under the inherent attributes. In addition, a rotation in complex space is used to obtain temporal-evolving embeddings of entities and relations, which are then combined with intrinsic embeddings. Finally, a distance-based scoring function is used to calculate the plausibility of the facts.

DEGAT [46] is an end-to-end encoder–decoder model. It uses GAN as an encoder to obtain entity features by aggregating features of neighboring entities, and it obtains relation features by capturing multi-hop information in the neighborhood of any given entity. The encoder does not take into account temporal information. In the decoder, a diachronic embedding function is introduced to perceive the dynamic features of entities and learn the changing features of entities over time. Furthermore, a novel model DE-ConvKB is proposed, which combines DE [58] and ConvKB [25] to calculate the plausibility of facts.

T-GAE [47] simultaneously applies GAN and LSTM to the TKGC task. In essence, the model is an encoder–decoder architecture, where the encoder consists of LSTM and GAN. Specifically, LSTM encodes both timestamps and relations into token sequences to produce relation embeddings with temporal information. GAN considers neighborhood aggregators to learn the features of entities. As for the decoder, the scoring function of ConvKB, a KGC method, is selected to calculate the plausibility of the facts. T-GAE is trained in a step-by-step training paradigm, which means that the encoder (LSTM and GAT) and the decoder (ConvKB) are trained independently.

Methods in this category have similar characteristics to the GNN-based and GCN-based methods. Since GAN introduces an attention mechanism based on GCN, which enables it to characterize the different importance of different neighbor entities in the aggregation process, in theory, the GAN-based methods can better model the heterogeneity of TKGs.

3.8. Language Model-Based Methods

Pre-trained language models [67,68] and large language models [69] are making new waves in the field of natural language processing and artificial intelligence due to their emergent ability and generalizability.

SToKE [48] employs the pre-trained language model BERT [70] to learn joint structural and temporal contextualized knowledge embeddings. Specifically, it first builds an event evolution tree (EET) for each query. TKG can be regarded as a structured event sequence that records the query-related structure and temporal contexts. Then, a temporal embedding and structure matrix are proposed to learn the temporal information and structural dependencies of facts in EET. Finally, TKGC is formulated as a mask prediction problem, and the pre-trained language model BERT is fine-tuned by masking the missing entities of the query.

Luo et al. [49] propose Llama-2-7b-CoH and Vicuna-7b-CoH, which conceptualize TKGC as a dual process of large language model fine-tuning and generation along a historical chain. During the fine-tuning, the Low-Rank Adaptation [71] method is used, and the data in TKG are divided into an input part and a supervised labeling part, guiding the large language model to adapt to the mapping relationship between the textual information of the specific TKG and the inherent complex logic in temporal events. Additionally, the local information across multiple single-step graphs is used for historical data augmentation to explore the ability of a large language model to perceive graph-modality information.

Methods in this category use pre-trained language models or large language models for TKGC, which can provide additional text information outside the TKGs to assist the TKGC task, but which have high requirements on the hardware platforms and require a large amount of training data.

4. Performance Comparison of Deep Learning-Based TKGC Methods

In this section, we summarize the experimental results of existing deep learning-based TKGC methods on the public dataset ICEWS05-15, which contains the most temporal information, and analyze their performance, aiming to help researchers understand the current status of deep learning-based TKGC methods in more detail and provide references for subsequent research.

It should be pointed out that the methods in

Table 3. Experimental results on ICEWS05-15.

Category	TKGC Method	MRR	MR	Hits@10	Hits@1
LSTM-based methods	TA-TransE [27] ⋆	0.299	79	66.8	9.6
	TA-DISTMULT [27] ⋆	0.474	98	72.8	34.6
	TA-TransR  [28]	-	-	-	-
CNN-based methods	CATKGC [29]	-	-	-	-
	ConvTKG [30] ⋄	0.740	-	86.2	67.4
	SANe [31] *	0.683	-	82.3	60.5
	SANe+ [32] *	0.686	-	82.6	60.8
CapsNet-based methods	TempCaps [33] *	0.521	-	70.5	42.3
	BiQCap [34] †	0.691	-	83.7	62.1
Attention-based methods	RoAN [35] ⋆	0.599	-	82.3	47.9
	HSAE [36] †	0.864	-	94.1	81.7
GNN-based methods	TGAP [37] †	0.670	-	84.5	56.8
	STRGNN [38] ⋆	0.781	-	85.1	66.7
	HyGNet [39] *	0.693	-	83.7	61.2
GCN-based methods	TeMP [40] ⋆	0.691	-	91.7	56.6
	MtGCN [41]	-	-	-	-
	TAGCN [42] ⋆	0.641	-	81.2	55.3
	TAL-TKGC [43] ⋆	0.354	-	69.2	22.6
	THOR [44] †	0.799	-	88.2	75.0
GAN-based methods	IAGAT [45] *	0.615	-	80.8	50.3
	DEGAT [46] †	0.777	241	87.1	72.4
	T-GAE [47] *	0.696	-	86.2	-
Language model-based methods	SToKE [48] *	0.712	-	88.5	60.5
	Llama-2-7b-CoH [49] *	-	-	69.9	38.6
	Vicuna-7b-CoH [49] *	-	-	70.7	39.2

We use different symbols to represent the setting information, where ^⋆ represents the static filtered setting, ^⋄ represents the raw setting, * represents the time-wise filtered setting, and ^† represents that the setting information is not given.

were implemented under different evaluation settings, i.e., raw setting, static filtered setting, and time-wise filtered setting. The same method may produce different experimental results under different settings. Therefore, it is necessary to explain the settings first. The raw setting and the static filtered setting are a pair of opposing settings, originally designed for static KGC [17]. The time-wise filtered setting [58] is a modified version of the static filter setting for TKGC. Specifically, in the raw setting, for each test triple, its corrupted triples are constructed by randomly replacing its subject entity or object entity with every entity in the entity set, while the static filtered setting constructs corrupted triples by removing all corrupted triples that are repeated with triples in the train, validation, and test sets to reduce the impact of these repeated corrupted triples. Since directly applying the static filtered setting to TKGC may wrongly remove corrupted quadruplets, Goel et al. [58] introduced temporal information into the static filtered setting and proposed the time-wise filtered setting. More specifically, for each test quadruple $(s,r,o,t)\in test$, it constructs the corrupted quadruples as $(s^{\prime },r,o,t),(s,r,o^{\prime },t)\notin \mathcal{G}$, where $s^{\prime },o^{\prime }\in \mathcal{E}$.

From Table 3, we can observe the following:

(1) Among the methods under the static filtered setting, the GNN-based method STRGNN achieves the highest Hits@1 value of 66.7%, which shows the effectiveness of GNN in TKGC. Similarly, among the methods under the time-wise filtered setting, the GNN-based method HyGNet achieves the highest Hits@1 value of 61.2%, which also shows the effectiveness of GNN in TKGC. Both of these indicate that GNN is worthy of further exploration for TKGC.

(2) By comparing GNN-based, GCN-based, and GAN-based methods under the static and time-wise filtered settings, we find that GNN-based method achieved the best experimental results, which demonstrates that using methods with stronger feature characterization capabilities or more parameters does not necessarily improve experimental performance. Therefore, it is necessary to consider the characteristics of the TKGC task itself and design a model that is as simple as possible.

(3) The highest Hits@1 value of the methods under the time-wise filtered setting does not exceed 61.2% (HyGNet), indicating that there is a lot of room for improvement in all categories of deep learning-based TKGC methods.

(4) The method ConvTKG under the raw setting has a higher Hits@1 value than the methods under the static and time-wise filtered settings. This reflects the effectiveness of CNN in TKGC to a certain extent; however, in order to make a fairer comparison, further verification is needed under the static and time-wise filtered settings. Similarly, methods that do not provide setting information also need further verification.

(5) TA-TransE directly merges temporal information into relations, thereby converting TKGC into static KGC, and introduces LSTM to encode the merge information. This method has a simple structure, but the number of parameters increases and the characteristics of TKGC are not fully exploited, resulting in a Hits@1 value much lower than other methods.

(6) The application of large language models, especially Llama and Vicuna, in TKGC is still in its initial exploratory stage, and its experimental results are lower than most other categories of deep learning-based methods. Large language models have huge parameters and require a lot of training data or clever fine-tuning strategies. It may even be necessary to reconsider the way in which the TKGC task is modeled on large language models.

5. Conclusions and Future Directions

In this paper, we present a review of the deep learning-based TKGC methods and subdivide them into eight categories according to their core technology. Despite the significant progress made in TKGC research, there still remain several important future research directions.

• Few-shot TKGC. The relations in TKGs obey the long-tail distribution; in other words, a large portion of relations have only a few quadruples in TKGs. Therefore, few-shot TKGC (one-shot and zero-shot TKGC can be seen as special kinds of few-shot TKGC) is a research direction worthy of great attention. Although several approaches [72,73] have been proposed to try to solve the few-shot TKGC task, this direction is still in its infancy. In addition, we can draw on the ideas of the existing static few-shot KGC methods [74], especially those based on large language models and rule-based methods, to propose methods with high representation ability and interpretability.
• Various KGC. At present, there are many different types of KGs, including static knowledge graphs, temporal knowledge graphs, few-shot knowledge graphs, few-shot temporal knowledge graphs, and spatio-temporal knowledge graphs [75], etc. How to solve the problem of completing these knowledge graphs with a unified method is one of the key points of future research.
• Interpretability. Temporal knowledge graphs transform complex real-world information into structured data that can provide interpretability analysis and decision support for its downstream applications. Although the existing temporal knowledge graph completion methods based on deep learning have achieved state-of-the-art performance, the inherent characteristics of the structured data of temporal knowledge graphs are seriously affected due to the inexplicability of the deep learning methods itself. Therefore, improving the interpretability of temporal knowledge graph completion by researching the interpretability of deep learning methods [76,77] is a promising direction in the future.