Unleashing the Power of Decoders: Temporal Knowledge Graph Extrapolation with Householder Transformation
Abstract
:1. Introduction
- We introduce a novel Householder-transformation-based temporal knowledge graph extrapolation (HTKGE) method: an innovative encoder–decoder architecture that addresses the challenge of predicting future facts in temporal knowledge graphs (TKGs).
- We propose a new Householder-transformation-based adaptive decoder, which is meticulously designed to capitalize on the rich temporal encoding provided by the encoder, enhancing the model’s predictive capabilities through a series of transformative operations.
- Experiments on four public TKG datasets demonstrate that HTKGE is consistently effective for the TKG extrapolation task.
2. Related Work
2.1. Static KG Reasoning
2.2. Interpolation TKG Reasoning
2.3. Extrapolation TKG Reasoning
3. Preliminaries
3.1. TKG Extrapolation
3.2. Householder Transformation
4. Proposed Method
4.1. Encoder Architecture
4.1.1. RecurrentEvolutionary Embedding
4.1.2. Temporal Dependency Modeling
4.1.3. Global Historical Pattern Integration
4.1.4. Local–Global Interaction
4.2. Decoder Architecture with Adaptive Convolution
4.2.1. Householder Transformation for Embedding Integration
4.2.2. Dynamic Filter Generation
4.2.3. Adaptive Convolution Operation
4.3. Training Objective
5. Experiments
5.1. Experimental Setup
5.1.1. Benchmark Datasets
5.1.2. Evaluation Protocol
5.1.3. Baselines
5.2. Performance Comparison
5.3. Ablation Study
5.4. Parameter Analysis on the Decoder
6. Conclusions
Author Contributions
Funding
Data Availability Statement
Conflicts of Interest
References
- Bollacker, K.D.; Evans, C.; Paritosh, P.; Sturge, T.; Taylor, J. Freebase: A collaboratively created graph database for structuring human knowledge. In Proceedings of the SIGMOD 2008, Vancouver, BC, Canada, 9–12 June 2008. [Google Scholar]
- Miller, G.A. WordNet: A lexical database for English. Commun. ACM 1995, 38, 39–41. [Google Scholar] [CrossRef]
- Suchanek, F.M.; Kasneci, G.; Weikum, G. Yago: A Core of Semantic Knowledge. In Proceedings of the 16th International Conference on World Wide Web, WWW ’07, New York, NY, USA, 8–12 May 2007; pp. 697–706. [Google Scholar]
- Jiang, L.; Yan, G.; Luo, H.; Chang, W. Improved Collaborative Recommendation Model: Integrating Knowledge Embedding and Graph Contrastive Learning. Electronics 2023, 12, 4238. [Google Scholar] [CrossRef]
- Liu, K.; Wu, J.; Sun, Q.; Yang, H.; Wan, R. Harnessing Test-Oriented Knowledge Graphs for Enhanced Test Function Recommendation. Electronics 2024, 13, 1547. [Google Scholar] [CrossRef]
- Gao, H.; Wang, Y.; Shao, J.; Shen, H.; Cheng, X. User Identity Linkage across Social Networks with the Enhancement of Knowledge Graph and Time Decay Function. Entropy 2022, 24, 1603. [Google Scholar] [CrossRef] [PubMed]
- Du, Z.; Qu, L.; Liang, Z.; Huang, K.; Cui, L.; Gao, Z. IMF: Interpretable multi-hop forecasting on temporal knowledge graphs. Entropy 2023, 25, 666. [Google Scholar] [CrossRef] [PubMed]
- Xu, C.; Nayyeri, M.; Alkhoury, F.; Shariat Yazdi, H.; Lehmann, J. TeRo: A Time-aware Knowledge Graph Embedding via Temporal Rotation. In Proceedings of the COLING 2020, Barcelona, Spain, 8–13 December 2020. [Google Scholar]
- Xu, C.; Chen, Y.; Nayyeri, M.; Lehmann, J. Temporal Knowledge Graph Completion using a Linear Temporal Regularizer and Multivector Embeddings. In Proceedings of the NAACL-HLT 2021, Online, 6–11 June 2021. [Google Scholar] [CrossRef]
- Chen, K.; Wang, Y.; Li, Y.; Li, A. RotateQVS: Representing Temporal Information as Rotations in Quaternion Vector Space for Temporal Knowledge Graph Completion. In Proceedings of the ACL 2022, Dublin, Ireland, 22–27 May 2022. [Google Scholar]
- Jin, W.; Qu, M.; Jin, X.; Ren, X. Recurrent Event Network: Autoregressive Structure Inference over Temporal Knowledge Graphs. In Proceedings of the EMNLP 2020, Online, 16–20 November 2020; pp. 6669–6683. [Google Scholar]
- Sun, H.; Zhong, J.; Ma, Y.; Han, Z.; He, K. TimeTraveler: Reinforcement Learning for Temporal Knowledge Graph Forecasting. In Proceedings of the EMNLP 2021, Punta Cana, Dominican Republic, 16–20 November 2021; pp. 8306–8319. [Google Scholar] [CrossRef]
- Li, Z.; Guan, S.; Jin, X.; Peng, W.; Lyu, Y.; Zhu, Y.; Bai, L.; Li, W.; Guo, J.; Cheng, X. Complex Evolutional Pattern Learning for Temporal Knowledge Graph Reasoning. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), ACL 2022, Dublin, Ireland, 22–27 May 2022; Muresan, S., Nakov, P., Villavicencio, A., Eds.; Association for Computational Linguistics: Stroudsburg, PA, USA, 2022; pp. 290–296. [Google Scholar] [CrossRef]
- Li, Z.; Jin, X.; Li, W.; Guan, S.; Guo, J.; Shen, H.; Wang, Y.; Cheng, X. Temporal Knowledge Graph Reasoning Based on Evolutional Representation Learning. In Proceedings of the SIGIR 2021, Virtual Event, 11–15 July 2021. [Google Scholar] [CrossRef]
- Mak, H.W.L.; Han, R.; Yin, H.H.F. Application of Variational AutoEncoder (VAE) Model and Image Processing Approaches in Game Design. Sensors 2023, 23, 3457. [Google Scholar] [CrossRef] [PubMed]
- Zhong, G.; Gao, W.; Liu, Y.; Yang, Y.; Wang, D.H.; Huang, K. Generative adversarial networks with decoder–encoder output noises. Neural Netw. 2020, 127, 19–28. [Google Scholar] [CrossRef] [PubMed]
- Saha, U.; Saha, S.; Fattah, S.A.; Saquib, M. Npix2Cpix: A GAN-Based Image-to-Image Translation Network With Retrieval- Classification Integration for Watermark Retrieval From Historical Document Images. IEEE Access 2024, 12, 95857–95870. [Google Scholar] [CrossRef]
- Li, Z.; Hou, Z.; Guan, S.; Jin, X.; Peng, W.; Bai, L.; Lyu, Y.; Li, W.; Guo, J.; Cheng, X. HiSMatch: Historical Structure Matching based Temporal Knowledge Graph Reasoning. In Proceedings of the EMNLP 2022, Abu Dhabi, United Arab Emirates, 7–11 December 2022; pp. 7328–7338. [Google Scholar] [CrossRef]
- Li, Y.; Sun, S.; Zhao, J. TiRGN: Time-Guided Recurrent Graph Network with Local-Global Historical Patterns for Temporal Knowledge Graph Reasoning. In Proceedings of the IJCAI 2022, Vienna, Austria, 23–29 July 2022. [Google Scholar] [CrossRef]
- Dong, H.; Ning, Z.; Wang, P.; Qiao, Z.; Wang, P.; Zhou, Y.; Fu, Y. Adaptive Path-Memory Network for Temporal Knowledge Graph Reasoning. In Proceedings of the IJCAI 2023, Macao, China, 19–25 August 2023. [Google Scholar] [CrossRef]
- Liang, K.; Meng, L.; Liu, M.; Liu, Y.; Tu, W.; Wang, S.; Zhou, S.; Liu, X. Learn from Relational Correlations and Periodic Events for Temporal Knowledge Graph Reasoning. In Proceedings of the SIGIR 2023, Taipei, Taiwan, 23–27 July 2023. [Google Scholar] [CrossRef]
- Jiang, X.; Wang, Q.; Wang, B. Adaptive convolution for multi-relational learning. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), Minneapolis, MN, USA, 2–7 June 2019; pp. 978–987. [Google Scholar]
- Bordes, A.; Usunier, N.; Garcia-Duran, A.; Weston, J.; Yakhnenko, O. Translating embeddings for modeling multi-relational data. In Proceedings of the Advances in Neural Information Processing Systems, Lake Tahoe, NV, USA, 5–8 December 2013; pp. 2787–2795. [Google Scholar]
- Wang, Z.; Zhang, J.; Feng, J.; Chen, Z. Knowledge Graph Embedding by Translating on Hyperplanes. In Proceedings of the AAAI 2014, Québec, QC, Canada, 27–31 July 2014. [Google Scholar]
- Lin, Y.; Liu, Z.; Sun, M.; Liu, Y.; Zhu, X. Learning Entity and Relation Embeddings for Knowledge Graph Completion. In Proceedings of the AAAI 2015, Austin, TX, USA, 25–30 January 2015. [Google Scholar]
- Sun, Z.; Deng, Z.; Nie, J.; Tang, J. RotatE: Knowledge Graph Embedding by Relational Rotation in Complex Space. In Proceedings of the 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, 6–9 May 2019. [Google Scholar]
- Yang, B.; Yih, W.; He, X.; Gao, J.; Deng, L. Embedding Entities and Relations for Learning and Inference in Knowledge Bases. In Proceedings of the ICLR 2015, San Diego, CA, USA, 7–9 May 2015. [Google Scholar]
- Trouillon, T.; Welbl, J.; Riedel, S.; Gaussier, É.; Bouchard, G. Complex Embeddings for Simple Link Prediction. In Proceedings of the 33nd International Conference on Machine Learning, ICML 2016, New York, NY, USA, 19–24 June 2016; Volume 48, pp. 2071–2080. [Google Scholar]
- Dettmers, T.; Minervini, P.; Stenetorp, P.; Riedel, S. Convolutional 2D Knowledge Graph Embeddings. In Proceedings of the AAAI 2018, New Orleans, LA, USA, 2–7 February 2018. [Google Scholar]
- Shang, C.; Tang, Y.; Huang, J.; Bi, J.; He, X.; Zhou, B. End-to-End Structure-Aware Convolutional Networks for Knowledge Base Completion. In Proceedings of the AAAI 2019, Honolulu, HI, USA, 27 January–1 February 2019; pp. 3060–3067. [Google Scholar]
- Schlichtkrull, M.S.; Kipf, T.N.; Bloem, P.; van den Berg, R.; Titov, I.; Welling, M. Modeling Relational Data with Graph Convolutional Networks. In Proceedings of the ESWC 2018, Heraklion, Crete, Greece, 3–7 June 2018. [Google Scholar]
- Vashishth, S.; Sanyal, S.; Nitin, V.; Talukdar, P.P. Composition-based Multi-Relational Graph Convolutional Networks. In Proceedings of the 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, 26–30 April 2020. [Google Scholar]
- Leblay, J.; Chekol, M.W. Deriving validity time in knowledge graph. In Proceedings of the Web Conference 2018, Lyon, France, 23–27 April 2018; pp. 1771–1776. [Google Scholar]
- Dasgupta, S.S.; Ray, S.N.; Talukdar, P. Hyte: Hyperplane-based temporally aware knowledge graph embedding. In Proceedings of the EMNLP 2018, Brussels, Belgium, 31 October–2 November 2018; pp. 2001–2011. [Google Scholar]
- García-Durán, A.; Dumancic, S.; Niepert, M. Learning Sequence Encoders for Temporal Knowledge Graph Completion. In Proceedings of the EMNLP 2018, Brussels, Belgium, 31 October–2 November 2018. [Google Scholar]
- Lacroix, T.; Obozinski, G.; Usunier, N. Tensor Decompositions for Temporal Knowledge Base Completion. In Proceedings of the ICLR 2020, Addis Ababa, Ethiopia, 26–30 April 2020. [Google Scholar]
- Xu, C.; Nayyeri, M.; Alkhoury, F.; Lehmann, J.; Yazdi, H.S. Temporal Knowledge Graph Embedding Model based on Additive Time Series Decomposition. arXiv 2019, arXiv:1911.07893. [Google Scholar]
- Jain, P.; Rathi, S.; Mausam; Chakrabarti, S. Temporal Knowledge Base Completion: New Algorithms and Evaluation Protocols. In Proceedings of the EMNLP 2020, Online, 16–20 November 2020; 20 November 2020. [Google Scholar] [CrossRef]
- Jung, J.; Jung, J.; Kang, U. Learning to Walk across Time for Interpretable Temporal Knowledge Graph Completion. In Proceedings of the KDD ’21: The 27th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, Virtual Event, Singapore, 14–18 August 2021; pp. 786–795. [Google Scholar] [CrossRef]
- Zhu, C.; Chen, M.; Fan, C.; Cheng, G.; Zhang, Y. Learning from History: Modeling Temporal Knowledge Graphs with Sequential Copy-Generation Networks. In Proceedings of the AAAI 2021, Virtual, 2–9 February 2021. [Google Scholar]
- Han, Z.; Ding, Z.; Ma, Y.; Gu, Y.; Tresp, V. Learning Neural Ordinary Equations for Forecasting Future Links on Temporal Knowledge Graphs. In Proceedings of the EMNLP 2021, Punta Cana, Dominican Republic, 16–20 November 2021. [Google Scholar] [CrossRef]
- Liu, Y.; Ma, Y.; Hildebrandt, M.; Joblin, M.; Tresp, V. TLogic: Temporal Logical Rules for Explainable Link Forecasting on Temporal Knowledge Graphs. In Proceedings of the AAAI 2022, Virtual, 22 February–1 March 2022. [Google Scholar]
- Han, Z.; Chen, P.; Ma, Y.; Tresp, V. Explainable Subgraph Reasoning for Forecasting on Temporal Knowledge Graphs. In Proceedings of the ICLR 2021, Vienna, Austria, 4 May 2021. [Google Scholar]
- Lautenschlager, J.; Shellman, S.; Ward, M. ICEWS Events and Aggregations. Harv. Dataverse 2015, 3. [Google Scholar]
- Leetaru, K.; Schrodt, P.A. Gdelt: Global data on events, location, and tone, 1979–2012. In Proceedings of the ISA Annual Convention, Citeseer, San Francisco, CA, USA, 2–4 April 2013; pp. 1–49. [Google Scholar]
- Goel, R.; Kazemi, S.M.; Brubaker, M.A.; Poupart, P. Diachronic Embedding for Temporal Knowledge Graph Completion. In Proceedings of the AAAI 2020, New York, NY, USA, 2–7 February 2020. [Google Scholar] [CrossRef]
Dataset | GDELT | ICEWS14* | WIKI | ICEWS05-15 |
---|---|---|---|---|
Entities | 7691 | 7128 | 12,554 | 10,094 |
Relations | 240 | 230 | 24 | 251 |
Train | 1,734,399 | 63,685 | 539,286 | 368,868 |
Validation | 238,765 | 13,823 | 67,538 | 46,302 |
Test | 305,241 | 13,222 | 63,110 | 46,159 |
Time granularity | 15 min | 24 h | 1 year | 24 h |
Timestamps | 2975 | 365 | 232 | 4017 |
Dataset | GDELT | ICEWS14* | ||||||
---|---|---|---|---|---|---|---|---|
Metric | MRR | H@1 | H@3 | H@10 | MRR | H@1 | H@3 | H@10 |
RE-NET [11] | 19.6 | 12.4 | 21.0 | 34.0 | 38.3 | 28.7 | 41.3 | 54.5 |
TANGO [41] | 19.2 | 12.2 | 20.4 | 32.8 | 26.3 | 17.3 | 29.1 | 44.2 |
CyGNet [40] | 18.5 | 11.5 | 19.6 | 32.0 | 32.7 | 23.7 | 36.3 | 50.7 |
TITer [12] | 20.2 | 14.1 | 22.2 | 31.2 | 41.7 | 32.7 | 46.5 | 58.4 |
RE-GCN [14] | 19.8 | 12.5 | 21.0 | 34.0 | 41.8 | 31.6 | 46.7 | 61.5 |
xERTE [43] | 18.9 | 12.3 | 20.1 | 30.3 | 40.8 | 32.7 | 45.7 | 57.3 |
TiRGN [19] | 21.7 | 13.6 | 23.3 | 37.6 | 45.1 | 34.4 | 51.3 | 65.0 |
HiSMatch [18] | 22.0 | 14.5 | 23.8 | 36.6 | 45.8 | 35.8 | 50.8 | 65.1 |
TLogic [42] | 19.8 | 12.2 | 21.7 | 35.6 | 43.0 | 33.6 | 48.3 | 61.2 |
DaeMon [20] | 20.7 | 13.7 | 22.5 | 34.2 | - | - | - | - |
RPC [21] | 22.4 | 14.4 | 24.4 | 38.3 | 44.6 | 34.9 | 49.8 | 65.1 |
HTKGE (ours) | 23.9 | 16.2 | 26.1 | 38.7 | 46.1 | 36.2 | 51.5 | 66.4 |
Dataset | WIKI | ICEWS05-15 | ||||||
Metric | MRR | H@1 | H@3 | H@10 | MRR | H@1 | H@3 | H@10 |
RE-NET [11] | 49.7 | 46.9 | 51.2 | 43.5 | 43.3 | 33.4 | 47.8 | 63.1 |
TANGO [41] | 50.4 | 48.5 | 51.5 | 53.6 | 42.9 | 32.7 | 48.1 | 62.3 |
CyGNet [40] | 58.8 | 47.9 | 66.4 | 78.7 | 36.8 | 26.6 | 41.6 | 56.2 |
TITer [12] | 73.9 | 71.7 | 75.4 | 77.0 | 47.7 | 38.0 | 52.9 | 65.8 |
RE-GCN [14] | 78.5 | 74.5 | 81.6 | 84.7 | 48.0 | 37.3 | 53.9 | 68.3 |
xERTE [43] | 71.1 | 68.1 | 76.1 | 79.0 | 46.6 | 37.8 | 52.3 | 63.9 |
TiRGN [19] | 81.7 | 77.8 | 85.1 | 87.1 | 50.0 | 39.3 | 56.1 | 70.7 |
HiSMatch [18] | 78.1 | 73.9 | 81.32 | 84.7 | 52.9 | 42.0 | 59.1 | 73.3 |
TLogic [42] | 79.7 | 75.4 | 81.9 | 85.2 | 47.0 | 36.2 | 53.1 | 67.4 |
DaeMon [20] | 82.4 | 78.3 | 86.0 | 88.0 | - | - | - | - |
RPC [21] | 81.2 | 76.3 | 85.4 | 88.7 | 51.4 | 39.9 | 57.0 | 71.8 |
HTKGE (ours) | 82.7 | 78.5 | 86.8 | 89.1 | 52.4 | 41.3 | 58.4 | 72.9 |
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. |
© 2024 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Share and Cite
Yang, F.; Zhang, Y.; Zhao, X.; Pang, S. Unleashing the Power of Decoders: Temporal Knowledge Graph Extrapolation with Householder Transformation. Symmetry 2024, 16, 1166. https://doi.org/10.3390/sym16091166
Yang F, Zhang Y, Zhao X, Pang S. Unleashing the Power of Decoders: Temporal Knowledge Graph Extrapolation with Householder Transformation. Symmetry. 2024; 16(9):1166. https://doi.org/10.3390/sym16091166
Chicago/Turabian StyleYang, Fuqiang, Yue Zhang, Xuechen Zhao, and Shengnan Pang. 2024. "Unleashing the Power of Decoders: Temporal Knowledge Graph Extrapolation with Householder Transformation" Symmetry 16, no. 9: 1166. https://doi.org/10.3390/sym16091166