Unleashing the Power of Decoders: Temporal Knowledge Graph Extrapolation with Householder Transformation

Abstract

In the realm of artificial intelligence, knowledge graphs (KGs) serve as an essential structured framework, capturing intricate relationships between diverse entities and supporting a broad spectrum of AI applications. Despite their utility, the static characteristic of KGs poses challenges in dynamically evolving information landscapes. This has catalyzed the development of temporal knowledge graphs (TKGs), which introduce a temporal layer to KGs, facilitating the representation of knowledge progression through time. This study zeroes in on the critical task of TKG extrapolation, which is vital for forecasting future occurrences and offering foresight into emerging situations across a variety of fields. Most contemporary approaches to TKG extrapolation are predicated on the symmetrical encoder–decoder paradigm, wherein the processes of representation learning and reasoning are harmoniously intertwined. While the encoder often garners the most attention due to its role in capturing and encoding information, the pivotal role of the decoder, which is often overlooked, is essential for direct inference and the accurate projection of temporal dynamics. To this end, we present the Householder-transformation-based temporal knowledge graph extrapolation (HTKGE) method: a groundbreaking encoder–decoder framework that reimagines the decoder’s contribution to TKG extrapolation. Our approach spotlights an adaptive decoder propelled by Householder transformations, which engage dynamically with the temporal encoding from the encoder. This interaction fosters a nuanced comprehension of the TKG’s temporal trajectory. Our empirical evaluations across four benchmark TKG datasets substantiate HTKGE’s consistent efficacy in TKG extrapolation tasks.

1. Introduction

Knowledge graphs (KGs) have become an indispensable asset in the realm of artificial intelligence, providing a structured and comprehensive framework for representing complex interconnections between disparate entities [1,2,3]. They are the bedrock of modern AI systems, underpinning applications ranging from personalized recommendation engines to advanced question-answering services [4,5]. The ability of KGs to encode semantic relationships has made them a critical component in the development of large-scale AI applications, from virtual assistants to autonomous vehicles. Despite the transformative impact of KGs, their static representation of knowledge poses a significant limitation in a world where information is inherently dynamic. This has catalyzed the evolution of temporal knowledge graphs (TKGs), which introduce a temporal layer to traditional KGs. TKGs not only capture the intricate web of relationships between entities but can also track the evolution of these relationships over time. By embedding timestamps within the graph, TKGs offer a temporal perspective that is crucial for understanding the progression of events, the trajectory of entity interactions, and the shift in relationships—a feature that is particularly valuable for historical analysis, trend prediction and real-time decision-making in various sectors, including finance, healthcare, and social sciences [6,7]. The integration of time as a first-class citizen in knowledge representation marks a significant advancement in the field of AI, enabling more nuanced and context-aware applications that can adapt to the ever-changing landscape of information.

Temporal knowledge graphs (TKGs) are instrumental in modeling the evolution of knowledge over time and encompass facts from an initial time point $t_0$ up to a terminal time point $t_T$. The reasoning within TKGs is essential for addressing their inherent incompleteness, which can impede their utility in various applications. TKG reasoning is typically categorized into two fundamental tasks: interpolation [8,9,10] and extrapolation [11,12,13]. Interpolation is the process of deducing missing entities or relationships that are expected to exist within the known time frame, i.e., between $t_0$ and $t_T$. This task is grounded in the existing data structure and is, to a certain extent, informed by the historical context provided by the TKG. On the other hand, extrapolation is the more complex endeavor of forecasting facts that are anticipated to occur after $t_T$. Unlike interpolation, extrapolation must project beyond the given data points to make predictions that are not directly supported by the existing TKG. This task requires an understanding of underlying trends and patterns and the ability to anticipate future developments, which is significantly more challenging and is the primary focus of this paper [14]. Given these considerations, our research endeavors to concentrate on the extrapolation aspect of TKGs. The ability to perform effective extrapolation is crucial for applications that aim to predict future events or to provide insights into forthcoming scenarios, such as financial forecasting, disaster preparedness, and strategic planning.

The prevalent approach to TKG reasoning leverages efficient encoder–decoder [15,16,17] frameworks, which have been widely adopted for their ability to distill and utilize the complex structural and semantic information present in TKGs [11,18]. The encoder component is tasked with learning meaningful representations that capture the essence of the TKG’s topology and the temporal dynamics of its facts. These representations are then passed to the decoder, which is tailored to perform reasoning for specific downstream tasks, such as link prediction or fact retrieval. Among the various methods, Recurrent Event Graph Convolutional Network (REGCN) [14] employs a recurrent structure to sequentially process the temporal information, treating the TKG as an evolving graph wherein nodes and edges are updated over time. This allows the REGCN to capture the temporal dependencies and structural changes within the TKG. On the other hand, the Time-Guided Recurrent Graph Network (TiRGN) [19] introduces a mechanism to integrate both local and global historical patterns and uses a time-guided approach to enhance the reasoning process. The TiRGN model excels at discerning recurrent and periodic behaviors: a capability indispensable for precise forecasting, particularly in sectors where the ubiquity of such patterns is a defining characteristic. While symmetrical encoder–decoder architectures have illuminated their promise in propelling the frontiers of TKG reasoning, a critical examination reveals a potential shortcoming from a technical perspective. The prevailing inclination has been to concentrate enhancements on the encoder component, seeking to distill efficient and information-rich representations from the input data [11,14,19,20,21]. This focus, while beneficial, has inadvertently resulted in relative neglect of the decoder’s equally crucial function within the architecture. The decoder’s role is not merely to reconstruct or generate output from the encoded representation; it is the linchpin that translates compressed information back into a useful form for reasoning tasks. The oversight could lead to suboptimal reasoning capabilities, as the decoder may not be sufficiently adept at interpreting the encoder’s output, especially when dealing with the complexities of temporal dynamics and evolving relationships in TKGs.

Analogous to the findings in the Convolutional Neural Networks for Multi-Relational Learning (ConvR) [22], where the interaction between the convolutional filters and the input features is paramount for capturing the nuanced relationships in multi-relational data, the decoder in TKG reasoning holds a similarly critical position. The decoder’s interaction with the encoded representations is what ultimately translates the learned features into actionable insights and predictions. It is the decoder’s responsibility to interpret the entity and relation embeddings in a manner that is aligned with the task at hand, whether it be predicting future interactions or inferring missing links within the TKG. This underscores the necessity to develop decoders that are not only capable of understanding the complex patterns encoded within the representations but are also adept at leveraging this understanding to make informed predictions. The decoder’s design must facilitate a nuanced exploration of the multifaceted interactions between entities and relations across different temporal contexts. It is this very interaction that enables the model to transcend a simplistic mapping of inputs to outputs to allow for a deeper and more contextualized reasoning process.

In this work, we introduce the Householder-transformation-based temporal knowledge graph extrapolation (HTKGE) method: a novel encoder–decoder architecture specifically tailored to address the intricacies of predicting future facts within temporal knowledge graphs (TKGs). This method stands out due to its innovative approach to the decoder’s role in the reasoning process, which we deem indispensable to the overall performance of TKG extrapolation. While the encoder in our method is acknowledged for its essential function of capturing the temporal dynamics and relational structures of TKGs through a sophisticated recurrent mechanism, it is the decoder that represents our unique contribution. The proposed Householder-transformation-based adaptive decoder is meticulously designed to capitalize on the rich temporal encoding provided by the encoder, enhancing the model’s predictive capabilities through a series of transformative operations. The decoder distinguishes itself by leveraging Householder transformations: a mathematical technique that allows for the creation of dynamic filters. These filters interact intimately with the encoded representations, adeptly capturing the nuanced and evolving interactions between entities and relationships over time. This adaptive feature is inspired by the adaptive filters [22], which enable a deep and contextual understanding of the TKG’s temporal evolution, which is indispensable for making accurate predictions. HTKGE offers a robust and versatile framework for not only predicting future states within TKGs but also for uncovering the latent temporal dependencies that are often obscured in complex temporal data structures.

We summarize our main contributions as follows:

• We introduce a novel Householder-transformation-based temporal knowledge graph extrapolation (HTKGE) method: an innovative encoder–decoder architecture that addresses the challenge of predicting future facts in temporal knowledge graphs (TKGs).
• We propose a new Householder-transformation-based adaptive decoder, which is meticulously designed to capitalize on the rich temporal encoding provided by the encoder, enhancing the model’s predictive capabilities through a series of transformative operations.
• Experiments on four public TKG datasets demonstrate that HTKGE is consistently effective for the TKG extrapolation task.

2. Related Work

2.1. Static KG Reasoning

Within the domain of static KG reasoning, a variety of approaches have been introduced and broadly fall into three categories. The first category includes translation-based models that represent relationships between entities through vector arithmetic, with TransE [23] and its extensions [24,25,26] being particularly noteworthy. The second category, semantic matching models, assesses the plausibility of entity relationships using triangular norms, with DistMult [27] and ComPlEx [28] recognized for employing bilinear and complex number asymmetry, respectively. The third category is query-centric methods, where models as cited in [29,30] aim to capture the distributed nature of KG entities and relations. Enhancements to graph convolutional networks, such as R-GCN [31] and CompGCN [32], have also been proposed to improve relation-awareness in KGs. However, these models mainly focus on static reasoning and often do not account for the temporal dynamics inherent in KGs.

2.2. Interpolation TKG Reasoning

TKG interpolation methods build upon static KG models by incorporating temporal dimensions. TTransE [33], extending TransE, integrates temporal constraints to refine the modeling of time-sensitive relationships. HyTE [34] incorporates timestamps with hyperplanes, blending time directly into the entity-relation context. TA-DistMult [35] extends this approach by intertwining relations and timestamps through an LSTM, addressing the diversity in temporal expressions. TComplEx [36], advancing from ComPlEx, employs tensor decomposition to create timestamp embeddings, thus enhancing temporal adaptability. ATiSE [37] introduces additive time series decomposition to embed temporal information, aligning TKG representations with multidimensional Gaussian distributions to account for uncertainty. TeRo [8] proposes dual embeddings for relations in order to handle the temporal interval’s inception and conclusion. TIMEPLEX-base [38] harmonizes entity, relation, and temporal embeddings, leveraging recurrent facts and temporal interactions for improved predictive tasks. RotateQVS [10] and T-GAP [39] prioritize model interpretability, with RotateQVS utilizing quaternion space for temporal evolution and T-GAP focusing on temporal displacement for inference. TeLM [9] investigates the role of time granularity in TKGs through tensor decomposition, deepening the understanding of temporal dynamics within knowledge graphs.

2.3. Extrapolation TKG Reasoning

Extrapolation reasoning in TKGs, which aim to predict future relationships, has become increasingly popular. The RE-GCN model [14] approaches TKG modeling by factoring in structural and temporal features along with static attributes in a unified manner without separate query encoding. Other models like RE-NET [11] and CyGNeT [40] employ subgraph aggregators and sequential copy networks to represent sequences of facts. TANGO-TuckER [41] and TANGO-DistMult [41] utilize neural ordinary differential equations (ODEs) within GCNs to encode temporal dynamics for better predictive performance. TLogic [42] provides an interpretable framework based on logical rules derived from random walks, while xERTE [43] incorporates a temporal association attention mechanism to preserve causal relationships. TITer [12] stands out due to its use of reinforcement learning and has a time coding function and a reward system based on the Dirichlet distribution. CEN [13] tackles evolutionary patterns with a CNN that is sensitive to sequence lengths and a curriculum learning approach. TiRGN [19] delves into historical dependencies using recurrent graph encoders at both the local and global levels, enhancing the model’s ability to leverage historical contexts for extrapolation. At present, the creators of the majority of these cutting-edge encoder–decoder-based methods for TKG extrapolation have predominantly focused their research efforts on the encoder component, with the aim of learning efficient knowledge representations. This focus has often overshadowed the equally crucial role of the decoder in the reasoning process. This imbalance has emerged as a significant gap in the literature and serves as the primary motivation for our research.

3. Preliminaries

3.1. TKG Extrapolation

TKG extrapolation involves predicting future temporal facts based on a collection of known temporal facts. A TKG is defined as $\mathcal{G}=\langle \mathcal{E},\mathcal{R},\mathcal{T},\mathcal{F}\rangle$ and encompasses entities $\mathcal{E}$, relations $\mathcal{R}$, timestamps $\mathcal{T}$, and facts $\mathcal{F}$. Each fact is articulated as a quadruple $(e_s,r,e_o,t)$, where the relation $r\in \mathcal{R}$ is predicated between a subject $e_s\in \mathcal{E}$ and an object $e_o\in \mathcal{E}$ at the specified timestamp $t\in \mathcal{T}$. To enhance the TKG with temporal dynamics, inverse relations are introduced, augmenting the set of facts with the inverse quadruple $(e_o,r^{-1},e_s,t)$ for each existing fact. The observable facts, denoted by $𝒪$, are confined to a time interval $[t_0,T]$ and are represented as $𝒪=\left\{(e_s,r,e_o,t)\right|t\in [t_0,T]\}$. Given a query $Q=(e_q,r_q,\ast ,t_q)$, where ∗ denotes the entity to be determined, the goal of TKG extrapolation is to infer the unknown object entity, presupposing that $t_q$ exceeds the latest timestamp T in the observable facts, thus signifying extrapolation into the future beyond the known time frame.

3.2. Householder Transformation

The Householder transformation, a pivotal technique in numerical linear algebra, is adept at performing elementary reflections within a vector space. This method is particularly useful for transforming a given vector into a canonical form wherein all but one of its components are zero.

Given a non-zero vector $\mathit{v}\in {\mathbb{R}}^n$, the Householder transformation projects $\mathit{v}$ onto a hyperplane defined by a unit vector $\omega$ that is normal to the desired reflection direction. The reflection operation results in a new vector ${\mathit{v}}^{\prime }$, which is a modified version of $\mathit{v}$ with specific components annihilated. This is graphically illustrated in Figure 1, where the vector $\mathit{v}$ is decomposed into $\mathit{x}$ and $\mathit{y}$, with ${\mathit{v}}^{\prime }$ representing $\mathit{x}-\mathit{y}$.

Mathematically, the Householder matrix $\mathit{H}\left(\mathit{\omega }\right)$ is constructed from the unit vector $\omega$ and is defined as:

$$\mathit{H}\left(\mathit{\omega }\right)=\mathit{I}-2\mathit{\omega }{\mathit{\omega }}^T$$

where $\mathit{I}$ is the identity matrix, and ${\mathit{\omega }}^T$ denotes the transpose of $\mathit{\omega }$. This matrix $\mathit{H}\left(\mathit{\omega }\right)$ is symmetric and orthogonal and has the property that $\mathit{H}\left(\mathit{\omega }\right)$$={\left[\mathit{H}\left(\mathit{\omega }\right)\right]}^T$$={\left[\mathit{H}\left(\mathit{\omega }\right)\right]}^{-1}$. When applied to a vector $\mathit{v}$, the transformation $\mathit{H}\mathit{v}$ yields ${\mathit{v}}^{\prime }$, maintaining the vector’s two-norm.

The Householder transformation is not only instrumental in QR decomposition but also advantageous in terms of numerical stability and efficiency, as it requires less computational memory and is more precise compared to other methods like the classical Gram–Schmidt process. The QR decomposition using Householder transformations is especially effective for constructing an orthogonal matrix $\mathit{Q}$ from a given matrix $\mathit{A}$, which can simplify subsequent matrix operations.

4. Proposed Method

In this section, we introduce a novel Householder-transformation-based temporal knowledge graph extrapolation (HTKGE) method, as illustrated in Figure 2. Our HTKGE is an encoder–decoder architecture designed to capture the multi-relational and temporal dynamics of knowledge graphs by implementing a novel application of Householder transformations in the decoder.

4.1. Encoder Architecture

The encoder module in our HTKGE framework is designed to encapsulate the temporal evolution and multi-relational aspects of knowledge graphs, with a focus on capturing the sequential dependencies and historical patterns inherent in temporal knowledge graphs.

4.1.1. RecurrentEvolutionary Embedding

At the core of our encoder is the recurrent evolutionary embedding, which is responsible for updating the representations of entities and relations over successive timestamps. This process is modeled as a recurrent graph convolutional operation that leverages the structural information within the knowledge graph at each timestamp t.

For an entity e and a relation r at timestamp t, the entity’s embedding ${\mathbf{h}}_{e_q}^t$ is updated by aggregating information from its neighboring entities through a graph convolutional layer, which can be expressed as:

$${\mathbf{h}}_{e_q}^t={\mathrm{GCN}}_{\theta }\left(\{{\mathbf{h}}_{e^{\prime }}^{t-1}\mid e^{\prime }\in \mathcal{N}\left(e\right)\},{\mathbf{W}}^r\right),$$

(1)

where $\mathcal{N}\left(e\right)$ denotes the neighbors of entity e, ${\mathbf{W}}^r$ represents the weight matrix corresponding to relation r, and ${\mathrm{GCN}}_{\theta }$ is the graph convolutional operation parameterized by $\theta$.

Our recurrent evolutionary embedding module is designed to progressively refine entity and relation representations as new information becomes available over time. This component is pivotal as it allows our model to adapt to the ever-changing landscape of TKGs, ensuring that the encoded representations are as current and relevant as possible. The recurrent nature of this embedding enables it to capture the temporal flow and evolving relationships within the graph.

4.1.2. Temporal Dependency Modeling

To capture the temporal dependencies, our encoder employs a recurrent mechanism that attends to the sequence of graph states across timestamps. This is achieved by a gated unit that updates the entity embeddings while retaining the informative historical structure. The updated entity embedding ${\overline{\mathbf{h}}}_e^t$ at time t is computed as:

$${\overline{\mathbf{h}}}_e^t=\mathrm{GRU}\left({\mathbf{h}}_{e_q}^t,{\overline{\mathbf{h}}}_e^{t-1};\mathbf{U},\mathbf{W},\mathbf{b}\right),$$

(2)

where GRU is the gated recurrent unit with the update gate parameter $\mathbf{U}$, the hidden state parameter $\mathbf{W}$, and the bias term $\mathbf{b}$. And ${\overline{\mathbf{h}}}_e^{t-1}$ is the entity embedding from the previous timestamp $t-1$.

Temporal dependency modeling is a cornerstone of our encoder and focuses on the temporal sequences that characterize TKGs. By employing a recurrent mechanism with gating units, we ensure that our model can maintain historical memory while continuously updating its understanding of the graph’s state. This component is essential for capturing the causal relationships and temporal progressions that are vital for accurate TKG extrapolation.

4.1.3. Global Historical Pattern Integration

In addition to the local temporal dependencies, our encoder also incorporates global historical patterns to enhance the reasoning capabilities. This is accomplished by a global history encoder network that accumulates repeated historical facts and identifies periodic behaviors over time. The global embedding ${\mathbf{g}}_e$ for an entity e is obtained by:

$${\mathbf{g}}_e=\mathrm{AGG}\left(\{{\mathbf{h}}_{e_q}^t\mid t=1,\dots ,T\}\right),$$

(3)

where AGG is an aggregation function, such as mean or sum, over the entity’s embeddings across all timestamps T.

Recognizing the importance of historical context in TKGs, our global historical pattern integration component provides a macroscopic view of the graph’s evolution. By identifying and aggregating repeated historical facts and periodic behaviors, this component enriches the encoder’s understanding with a global perspective. This integration of global patterns is crucial for uncovering long-term trends and cyclical behaviors that may influence future states of the TKG.

4.1.4. Local–Global Interaction

The final step in our encoder architecture is to integrate the local sequential patterns with the global historical patterns. This is achieved by a fusion mechanism that balances the contributions from both the local and global contexts. The fused embedding ${\mathbf{h}}_{e_q}$ for each entity e is given by:

$${\mathbf{h}}_{e_q}=\alpha ·{\overline{\mathbf{h}}}_e^T+(1-\alpha )·{\mathbf{g}}_e,$$

(4)

where $\alpha$ is a weighting factor that determines the influence of the local pattern ${\overline{\mathbf{h}}}_e^T$ (at the latest timestamp T) versus the global pattern ${\mathbf{g}}_e$. And we use d to denote the dimension for all the embeddings above.

The resulting fused embeddings ${\mathbf{h}}_{e_q}$ serve as the input to the decoder module, which will generate the final predictions. This encoder framework provides a comprehensive representation of the temporal knowledge graph by jointly modeling the local sequential dependencies and the global historical patterns.

The fusion of local sequential patterns with global historical patterns is a key innovation in our encoder architecture. This interaction allows our model to balance the fine-grained, immediate dependencies with broader, long-term trends. The resulting fused embeddings are thus a nuanced representation that encapsulates both the micro and macro dynamics of the TKG, providing a solid foundation for the decoder’s predictive tasks.

4.2. Decoder Architecture with Adaptive Convolution

The decoder in our HTKGE framework is designed to translate the comprehensive entity representations from the encoder into accurate predictive outcomes. This decoder employs adaptive convolution techniques integrated with Householder transformations for embedding combination that align with the numerical linear algebra practices established in the Householder transformation subsection. The framework of our decoder is depicted in Figure 3.

4.2.1. Householder Transformation for Embedding Integration

For a given query $q=(e_q,r_q,\ast ,t_q)$, we integrate relation embeddings and time embeddings within the query using the Householder transformation, which is a method known for its utility in numerical linear algebra, particularly for performing elementary reflections within a vector space. Given a relation embedding ${\mathbf{r}}_q$ and a time embedding ${\mathbf{t}}_q$, we construct a Householder matrix $\mathit{H}\left(\mathit{\omega }\right)$ parameterized by a unit vector $\mathit{\omega }$, which is derived from the embeddings ${\mathbf{r}}_q$ and ${\mathbf{t}}_q$ through a predefined function $\varphi$ that captures the desired reflection direction:

$$\mathit{\omega }=\varphi ({\mathbf{r}}_q,{\mathbf{t}}_q).$$

(5)

The Householder matrix $\mathit{H}\left(\mathit{\omega }\right)$ is then defined as:

$$\mathit{H}\left(\mathit{\omega }\right)=\mathit{I}-2\mathit{\omega }{\mathit{\omega }}^T,$$

(6)

where I is the identity matrix. This matrix is applied to the relation embedding $\mathbf{r}$ to produce a transformed embedding ${\mathbf{r}}_q^{\prime }$ that is reflective of the temporal context provided by ${\mathbf{t}}_q$ and the relation-specific characteristics of ${\mathbf{r}}_q$:

$${\mathbf{r}}_q^{\prime }=\mathit{H}\left(\mathit{\omega }\right){\mathbf{r}}_q.$$

(7)

4.2.2. Dynamic Filter Generation

The transformed relation embeddings are then reshaped to create dynamic convolutional filters (the channel number is denoted as n) following the adaptive convolutional network strategy outlined in [22]. Each filter ${\mathbf{F}}_{r_q}$, which is tailored for the relation $r_q$, is generated by reshaping the Householder-transformed relation embedding ${\mathbf{r}}_q^{\prime }$ into a 2D filter format conducive for convolution. And considering the channels, we actually have c various filters ${\mathbf{F}}_{r_q}^{\left(i\right)}$, $i=1,2,···n$, for each relation:

$${\mathbf{F}}_{r_q}^{\left(1\right)},{\mathbf{F}}_{r_q}^{\left(2\right)},···{\mathbf{F}}_{r_q}^{\left(n\right)}=\mathrm{split}\left(\mathrm{RESHAPE}\left({\mathbf{r}}^{\prime }\right)\right),$$

(8)

where the operation RESHAPE is used to convert a d-dimensional embedding into a three-dimensional matrix suitable for convolutional processing, i.e., $d=h\times w\times n$; here, h and w represent the dimensions of the filter. After a split operation, we have n different filters, where each filter ${\mathbf{F}}_{r_q}^{\left(i\right)}\in {\mathbb{R}}^{h\times w}$.

4.2.3. Adaptive Convolution Operation

The decoder performs an adaptive convolution operation with the dynamic filters ${\mathbf{F}}_{r_q}$ across the entity embeddings ${\mathbf{h}}_{e_q}$ from the encoder. This operation is detailed as follows:

Filter application: Each filter ${\mathbf{F}}_{r_q}^{\left(i\right)}$ is convolved with the query entity embedding ${\mathbf{h}}_{e_q}$ (obtained from Equation (4)) to produce a feature map ${\mathbf{c}}^{\left(i\right)}$. The convolution operation at a specific position is defined as:

$$c^{\left(i\right)(m,n)}=\sum _{j=0}^{h-1}\sum _{k=0}^{w-1}{F}_{r_q}^{\left(i\right)(j,k)}·h_e^{(m+j,n+k)},$$

(9)

where $c^{\left(i\right)(m,n)}$ is the value at position $(m,n)$ in the feature map for the i-th filter, and $h_e^{(m+j,n+k)}$ is the element of the entity embedding ${\mathbf{h}}_{e_q}$ at the corresponding position after padding, if necessary.

Feature map generation: The feature map $\mathbf{c}$ is constructed by aggregating the results of the convolution operation across multiple convolution filters:

$$\mathbf{c}=\mathrm{AGG}\left({\mathbf{c}}^{\left(i\right)}\right|\mathrm{for}i=1,2,···n).$$

(10)

This adaptive convolution operation is central to the decoder’s capability to capture the intricate interplay between entities and relations within the temporal knowledge graph, enabling HTKGE to forecast future facts with enhanced accuracy. The dynamic nature of the filters allows for relation-specific feature extraction that is sensitive to the unique characteristics of each relation, thereby improving the predictive power of the model.

Scoring for prediction: The final prediction is articulated by a scoring function:

$$\mathrm{Score}\left(q,e_a\right)={\mathbf{W}}^{\top }\mathbf{c}·{\mathbf{h}}_{e_o}^0,$$

(11)

where $\mathbf{W}$ is a learnable weight matrix, and ${\mathbf{h}}_{e_a}^0$ is the initial embedding for a candidate answer entity. In our approach, we refine the prediction process by meticulously selecting the candidate entities that achieve the highest scores as the definitive answers.

4.3. Training Objective

Our HTKGE is refined through the optimization of a cross-entropy loss function. This function quantifies the discrepancy between the anticipated and actual results for a specified query. The loss function L is designed for entity prediction and is formalized below:

$$L=\sum _{\left(q,e_a\right)\in {\mathcal{T}}_{\mathrm{train}}}{\mathbf{Y}}_{e_a}logscore\left(e_t\mid e_h,r_{h,t},t\right),$$

(12)

where ${\mathcal{T}}_{\mathrm{train}}$ is the set of training queries, and $score\left(q,e_a\right)$ is the probabilistic score for the facts calculated by Equation (11). ${\mathbf{Y}}_{e_a}$ are the label vectors, where the element is 1 if the fact occurs and 0 otherwise.

5. Experiments

5.1. Experimental Setup

5.1.1. Benchmark Datasets

For assessing our HTKGE method, we utilize four prominent datasets from the TKG extrapolation benchmarks: ICEWS14* [43], ICEWS05-15 [35], WIKI [33], and GDELT [11]. A detailed depiction of these datasets is provided in

Table 1. Statistics of the datasets.

Dataset	GDELT	ICEWS14*	WIKI	ICEWS05-15
Entities	7691	7128	12,554	10,094
Relations	240	230	24	251
Train	1,734,399	63,685	539,286	368,868
Validation	238,765	13,823	67,538	46,302
Test	305,241	13,222	63,110	46,159
Time granularity	15 min	24 h	1 year	24 h
Timestamps	2975	365	232	4017

. The ICEWS14* and ICEWS05-15 datasets are derived from subsets of the Integrated Crisis Early Warning System (ICEWS) [44], which is an expansive event-based database. The former encompasses events from the year 2014, while the latter includes a decade-long span from 2005 to 2015. For the WIKI dataset, we streamlined the data by excluding month and day information to achieve year-level granularity consistent with the methodology in [11]. The GDELT dataset, extracted from a comprehensive global event repository [45], boasts a detailed 15 min time granularity.

For extrapolation tasks, we organize the factual data from these four datasets chronologically and segment each into training, validation, and test subsets in accordance with the temporal stratification approach of Jin et al. [11], Han et al. [43] in order to guarantee (timestamps for training) < (timestamps for validation) < (timestamps for testing). This chronological division ensures a logical sequence from past events (training set) to future events (test set), enabling the utilization of historical data for predictive analysis.

5.1.2. Evaluation Protocol

The assessment of our proposed method’s capacity for extrapolation reasoning within TKGs is facilitated through a link prediction task. This task is specifically designed to deduce missing entities within temporal facts, which are symbolically represented as either $(e_s,r,\ast ,t)$ or $(\ast ,r,e_o,t)$, where $e_s$ and $e_o$ are entities, r symbolizes the relationship, and t indicates the timestamp. For a given test sample $(e_s,r,e_o,t)$, the process commences with the generation of a candidate quadruple set C, defined as $C=\{(e_s,r,\overline{e_o},t):\overline{e_o}\in \mathcal{E}\}\cup \{(\overline{e_s},r,e_o,t):\overline{e_s}\in \mathcal{E}\}$, by substituting $e_s$ or $e_o$ with every entity from the entity set $\mathcal{E}$. Following this, the quadruples are ranked according to their scores, as determined by Equation (11). For reporting our experimental results, we adhere to the time-wise filtered setting as referenced in [37,46]. This approach is selected due to its alignment with TKG reasoning and ensures that facts that are not observable at a specific timestamp t are still accounted for in the assessment of the test quadruple, as elucidated in [8]. This method stands in contrast to time-unwise filtering settings, which, as indicated in [14,23], may lead to potentially incorrect higher-ranking scores.

Performance metrics are reported using the standard benchmarks: the proportion of correct triples ranked within the top 1, 3, and 10 positions (H@1, H@3, and H@10, respectively) and the mean reciprocal rank (MRR). These metrics are designed such that higher values are indicative of better performance. For the sake of consistency, we present the mean results from five experimental runs, electing to omit the variance due to its general minimality.

5.1.3. Baselines

In our comprehensive evaluation, the HTKGE method is rigorously compared against a suite of eleven state-of-the-art models that are recognized for their proficiency in TKG extrapolation. Our baselines include the neural ordinary-equation-based TANGO [41], the probabilistic model CyGNet [40], the reinforcement-learning-based TITer [12], the subgraph-focused xERTE [43], and the rule-based TLogic [42]. Particularly, we place a strong emphasis on comparing with methods that are founded on the encoder–decoder framework, such as RE-NET [11], RE-GCN [14], TiRGN [19], HiSMatch [18], DaeMon [20], and RPC [21].

5.2. Performance Comparison

We present a comparative analysis of our proposed HTKGE method against eleven state-of-the-art models on four experimented datasets: GDELT, ICEWS14*, WIKI, and ICEWS05-15. The performance is evaluated using standard link prediction metrics, including the mean reciprocal rank (MRR) and the proportion of correct triples ranked in the top 1 (H@1), top 3 (H@3), and top 10 (H@10) positions. The results, as presented in

Table 2. Results on link prediction task for four experimented datasets, where the results of RE-NET are from HiSMatch. The best score is in bold.

Dataset	GDELT				ICEWS14*
Metric	MRR	H@1	H@3	H@10	MRR	H@1	H@3	H@10
RE-NET [11]	19.6	12.4	21.0	34.0	38.3	28.7	41.3	54.5
TANGO [41]	19.2	12.2	20.4	32.8	26.3	17.3	29.1	44.2
CyGNet [40]	18.5	11.5	19.6	32.0	32.7	23.7	36.3	50.7
TITer [12]	20.2	14.1	22.2	31.2	41.7	32.7	46.5	58.4
RE-GCN [14]	19.8	12.5	21.0	34.0	41.8	31.6	46.7	61.5
xERTE [43]	18.9	12.3	20.1	30.3	40.8	32.7	45.7	57.3
TiRGN [19]	21.7	13.6	23.3	37.6	45.1	34.4	51.3	65.0
HiSMatch [18]	22.0	14.5	23.8	36.6	45.8	35.8	50.8	65.1
TLogic [42]	19.8	12.2	21.7	35.6	43.0	33.6	48.3	61.2
DaeMon [20]	20.7	13.7	22.5	34.2	-	-	-	-
RPC [21]	22.4	14.4	24.4	38.3	44.6	34.9	49.8	65.1
HTKGE (ours)	23.9	16.2	26.1	38.7	46.1	36.2	51.5	66.4
	$\pm 0.022$	$\pm 0.016$	$\pm 0.029$	$\pm 0.025$	$\pm 0.033$	$\pm 0.028$	$\pm 0.035$	$\pm 0.042$
Dataset	WIKI				ICEWS05-15
Metric	MRR	H@1	H@3	H@10	MRR	H@1	H@3	H@10
RE-NET [11]	49.7	46.9	51.2	43.5	43.3	33.4	47.8	63.1
TANGO [41]	50.4	48.5	51.5	53.6	42.9	32.7	48.1	62.3
CyGNet [40]	58.8	47.9	66.4	78.7	36.8	26.6	41.6	56.2
TITer [12]	73.9	71.7	75.4	77.0	47.7	38.0	52.9	65.8
RE-GCN [14]	78.5	74.5	81.6	84.7	48.0	37.3	53.9	68.3
xERTE [43]	71.1	68.1	76.1	79.0	46.6	37.8	52.3	63.9
TiRGN [19]	81.7	77.8	85.1	87.1	50.0	39.3	56.1	70.7
HiSMatch [18]	78.1	73.9	81.32	84.7	52.9	42.0	59.1	73.3
TLogic [42]	79.7	75.4	81.9	85.2	47.0	36.2	53.1	67.4
DaeMon [20]	82.4	78.3	86.0	88.0	-	-	-	-
RPC [21]	81.2	76.3	85.4	88.7	51.4	39.9	57.0	71.8
HTKGE (ours)	82.7	78.5	86.8	89.1	52.4	41.3	58.4	72.9
	$\pm 0.049$	$\pm 0.036$	$\pm 0.043$	$\pm 0.046$	$\pm 0.035$	$\pm 0.027$	$\pm 0.030$	$\pm 0.034$

, evidence that HTKGE achieves state-of-the-art (SOTA) performance across the board, with particularly notable superiority on the GDELT, ICEWS14*, and WIKI datasets, where it outperforms existing SOTA methods across all evaluated metrics. These results are indicative of HTKGE’s robustness at capturing the nuances of temporal dependencies within TKGs and its effectiveness at predicting future facts. Specifically, our HTKGE demonstrates a marked improvement over the state-of-the-art (SOTA) method RPC, which also employs an encoder–decoder architecture, across four benchmark datasets. When considering the comprehensive metric MRR, HTKGE achieves a notable enhancement in performance: it outperforms RPC by 6.7% on the GDELT dataset, by 3.4% on the ICEWS14* dataset, by 1.8% on the WIKI dataset, and by 1.9% on the ICEWS05-15 dataset. HTKGE’s consistent outperforming across various datasets underscores its generalizability and reliability in the context of TKG reasoning. The adaptive Householder transformations within our decoder are particularly instrumental to these results and allow for nuanced understanding of and dynamic interaction with the temporal evolution of TKGs. This innovation sets HTKGE apart from other methods and positions it as a formidable candidate for future research and application in the field of TKG extrapolation.

Regarding the fact that our HTKGE does not surpass HiSMatch [18] on ICEWS05-15, we believe this could be due to several factors. Firstly, the ICEWS05-15 dataset may possess unique attributes that are better-suited to the approach taken by HiSMatch. Secondly, despite HTKGE’s consistent performance across most datasets, HiSMatch’s dominance on ICEWS05-15 could be a result of its model architecture, training process, or other undisclosed proprietary enhancements that contribute to its performance.

5.3. Ablation Study

Ablation studies are conducted on two datasets—namely, ICEWS14* and ICEWS05-15—to investigate the impact of individual components on the performance of our HTKGE. The H@1 performances are shown in Figure 4, where five sub-models are compared, including (1) the original HTKGE, (2) a variant of HTKGE that lacks temporal dependency modeling, denoted as “−TP”, (3) another variant without the incorporation of global historical patterns, denoted as “−GP”, (4) a modified version wherein the Householder transformation is replaced by direct addition of the relational and temporal representations, denoted as “+Add”, and (5) a version wherein the Householder transformation is substituted with an element-wise product of the relational and temporal representations, denoted as “+Prod”. Based on the experimental outcomes, the following conclusions can be drawn: (a) Both the “−TP” and “−GP” models exhibit a decline in performance compared to the original HTKGE. This suggests that within the encoder’s design, both temporal dependency modeling and the inclusion of global historical patterns significantly contribute to the overall performance. (b) The “+Add” and “+Prod” models also demonstrate a decrease in performance relative to the original HTKGE. This indicates that the Householder transformation utilized within the decoder’s architecture plays a crucial role in enhancing the model’s predictive capabilities. The observed performance drop underscores the importance of this specific transformation in the decoding process.

5.4. Parameter Analysis on the Decoder

Our exploration into the adaptability and efficiency of the proposed Householder-transformation-based adaptive decoder within the HTKGE framework extends to a thorough parameter analysis. This analysis is pivotal for optimizing the decoder’s configuration to achieve a balance between computational resources and performance metrics. We specifically focus on two critical hyperparameters: the channel number n and the filter size $h\times w$, which are known to substantially influence the decoder’s functional capacity. In our analysis, we meticulously adjust the channel number n across a spectrum of values $\{20,40,60,80,100\}$ and concurrently vary the filter size $h\times w$ through a set of dimensions $\{2\times 2,3\times 3,4\times 4,5\times 5\}$. This methodical tuning process is designed to elucidate the relationship between the parameter count and the decoder’s performance, as measured by its ability to accurately predict future facts within TKGs.

The findings of our parameter analysis are encapsulated in Figure 5. As the table illustrates, the parameter count of the decoder increases monotonically with the growth in the channel number n and the dimensions of the filter size $h\times w$. However, the performance metrics, particularly the H@10 score, exhibit a far less pronounced variation. This observation is particularly noteworthy as it suggests that our decoder is capable of achieving a commendable level of performance with a relatively modest parameter count. The relatively flat performance curve across different parameter configurations underscores the robustness of our decoder’s design. It suggests that the adaptive Householder transformations are adept at capturing the nuances of the temporal encoding, even when the complexity of the model is scaled back. This finding is a testament to the effectiveness of our approach and paves the way for further optimizations that could lead to even more efficient models.

6. Conclusions

In this paper, we introduce a novel Householder-based temporal knowledge graph extrapolation (HTKGE) method that makes significant strides in the domain of TKG reasoning by introducing an innovative encoder–decoder architecture. The unique adaptive decoder utilizes Householder transformations and is proven to be instrumental in enhancing predictive capabilities, allowing for a nuanced understanding of temporal dynamics within TKGs. Empirical results on four benchmark datasets confirm HTKGE’s effectiveness, highlighting its potential as a robust framework for future state prediction and the revelation of latent temporal dependencies in TKGs. This work paves the way for further advancements in the field and emphasizes the pivotal role of the decoder in TKG extrapolation tasks.

Limitations and future work: Our model’s performance is contingent upon the quality of input data: high-quality data are essential for precise extrapolations within sparse TKGs. Given the deep learning framework, the quality of parameter training is paramount to our model’s performance. We acknowledge the possibility that our model may have been inadvertently tuned to the specific data distribution and temporal properties of the datasets. In our future work, we will address scalability by refining our approach to manage large TKGs efficiently. We will explore data augmentation to counter data sparsity and seek algorithms that maintain the precision of Householder transformations while enhancing computational efficiency. Additionally, we will focus on improving the decoder’s interpretability by examining its decision-making process with the aim of optimizing its parameters for superior performance. We also intend to explore the causal relationships between events across different datasets, recognizing the complexity and potential impact of such interconnections on our model’s predictive capabilities.