TransE-MTP: A New Representation Learning Method for Knowledge Graph Embedding with Multi-Translation Principles and TransE
Abstract
:1. Introduction
- For complex optimization tasks, we first propose multi-translation principles (MTPs). Unlike some existing methods, such as TransE, MTPs can focus on different relationship types and establish corresponding translation principles.
- To produce accurate reasoning and prediction, multiple score functions are constructed based on MTPs for model training. Subsequently, the loss function is optimized with the gradient descent algorithm to minimize the score function.
- Representation learning of a knowledge graph with MTPs aims to optimize multiple objectives by applying translation principles to the four types of relationships: one-to-one, one-to-many, many-to-one, and many-to-many. Furthermore, we introduce TransE-MTP (combining MTPs with TransE). Extensive experiments on two widely used knowledge graph datasets, Freebase and WordNet, demonstrate the efficiency and competitiveness of our proposed model.
2. Related Work
2.1. Knowledge Graphs
2.2. Reasoning and Complement for Knowledge Graphs
2.3. Knowledge Representation Learning
3. The Proposed Method
- For one-to-one and many-to-one relations, the translation principle is defined as h + r = t;
- For one-to-many relations, the translation principle is defined as t − r = h;
- For many-to-many relations, the translation principle is defined as t − h = r.
4. Experiment and Analysis
4.1. Dataset and Experimental Setup
4.2. Link Prediction
4.3. Triplet Classification
5. Conclusions and Future Work
Author Contributions
Funding
Data Availability Statement
Conflicts of Interest
References
- Miller, G.A. WordNet: A lexical database for English. Commun. ACM 1995, 38, 39–41. [Google Scholar] [CrossRef]
- Bollacker, K.; Evans, C.; Paritosh, P.; Sturge, T.; Taylor, J. Freebase: A collaboratively created graph database for structuring human knowledge. In Proceedings of the 2008 ACM SIGMOD International Conference on Management of Data, Vancouver, BC, Canada, 9–12 June 2008; pp. 1247–1250. [Google Scholar] [CrossRef]
- Goyal, Y.; Khot, T.; Summers-Stay, D.; Batra, D.; Parikh, D. Making the V in VQA matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA, 21–26 July 2017; pp. 6904–6913. [Google Scholar] [CrossRef]
- Xiao, H.; Huang, M.; Zhu, X. From one point to a manifold: Knowledge graph embedding for precise link prediction. In Proceedings of the Twenty-Fifth International Joint Conference on Artificial Intelligence, New York, NY, USA, 9–15 July 2016; pp. 1315–1321. [Google Scholar] [CrossRef]
- Green, B.F., Jr.; Wolf, A.K.; Chomsky, C.; Laughery, K. Baseball: An automatic question-answerer. In Proceedings of the Western Joint IRE-AIEE-ACM Computer Conference, San Francisco, CA, USA, 3–5 May 1960; pp. 219–224. [Google Scholar] [CrossRef]
- Hamilton, W.; Bajaj, P.; Zitnik, M.; Jurafsky, D.; Leskovec, J. Embedding logical queries on knowledge graphs. In Advances in Neural Information Processing Systems; MIT Press: Cambridge, MA, USA, 2018; pp. 2030–2041. [Google Scholar] [CrossRef]
- Bengio, Y. Learning deep architectures for AI. Found. Trends® Mach. Learn. 2009, 2, 1–127. [Google Scholar] [CrossRef]
- Bengio, Y.; Courville, A.; Vincent, P. Representation learning: A review and new perspectives. IEEE Trans. Pattern Anal. Mach. Intell. 2013, 35, 1798–1828. [Google Scholar] [CrossRef] [PubMed]
- Turian, J.; Ratinov, L.; Bengio, Y. Word representations: A simple and general method for semi-supervised learning. In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, Uppsala, Sweden, 11–16 July 2010; pp. 384–394. [Google Scholar]
- Gilmer, J.; Schoenholz, S.S.; Riley, P.F.; Vinyals, O.; Dahl, G.E. Neural message passing for quantum chemistry. In Proceedings of the International Conference on Machine Learning, Sydney, Australia, 6–11 August 2017; pp. 1263–1272. [Google Scholar]
- Freitas, A.; Curry, E.; Oliveira, J.G.; O’Riain, S. Querying heterogeneous datasets on the linked data web: Challenges, approaches, and trends. IEEE Internet Comput. 2011, 16, 24–33. [Google Scholar] [CrossRef]
- Ebisu, T.; Ichise, R. Graph pattern entity ranking model for knowledge graph completion. arXiv 2019. [CrossRef]
- Pan, J.Z.; Zhang, M.; Singh, K.; Harmelen, F.V.; Gu, J.; Zhang, Z. Entity Enabled Relation Linking. In Semantic Web—ISWC 2019: 18th International Semantic Web Conference, Auckland, New Zealand, 26–30 October 2019; Springer: Berlin/Heidelberg, Germany, 2019; pp. 523–538. [Google Scholar] [CrossRef]
- Mulang, I.O.; Singh, K.; Orlandi, F. Matching natural language relations to knowledge graph properties for question answering. In Proceedings of the 13th International Conference on Semantic Systems, Amsterdam, The Netherlands, 11–14 September 2017; pp. 89–96. [Google Scholar] [CrossRef]
- Gandomi, A.; Haider, M. Beyond the hype: Big data concepts, methods, and analytics. Int. J. Inf. Manag. 2015, 35, 137–144. [Google Scholar] [CrossRef]
- Catherine, R.; Mazaitis, K.; Eskenazi, M.; Cohen, W. Explainable entity-based recommendations with knowledge graphs. In Proceedings of the Poster Track of the 11th ACM Conference on Recommender Systems, Como, Italy, 27–31 August 2017; pp. 1210–1221. [Google Scholar] [CrossRef]
- Miller, E.J. An introduction to the resource description framework. J. Libr. Adm. 2001, 34, 245–255. [Google Scholar] [CrossRef]
- Kaminski, M.; Kostylev, E.V.; Grau, B.C. Query nesting, assignment, and aggregation in SPARQL 1.1. In ACM Transactions on Database Systems (TODS); Association for Computing Machinery: New York, NY, USA, 2017; Volume 42, pp. 1–46. [Google Scholar] [CrossRef]
- Dubey, M.; Banerjee, D.; Chaudhuri, D.; Lehmann, J. EARL: Joint Entity and Relation Linking for Question Answering over Knowledge Graphs. In The Semantic Web—ISWC 2018: 17th International Semantic Web Conference, Monterey, CA, USA, 8–12 October 2018; Springer: Berlin/Heidelberg, Germany, 2018; Volume 11136, pp. 108–126. [Google Scholar] [CrossRef]
- Fernández, J.D.; Martínez-Prieto, M.A.; Gutiérrez, C.; Polleres, A.; Arias, M. Binary RDF representation for publication and exchange (HDT). J. Web Semant. 2013, 19, 22–41. [Google Scholar] [CrossRef]
- Bao, J.; Duan, N.; Yan, Z.; Zhou, M.; Zhao, T. Constraint-based question answering with knowledge graph. In Proceedings of the 26th International Conference on Computational Linguistics, Osaka, Japan, 11–16 December 2016; pp. 2503–2514. [Google Scholar]
- Bojanowski, P.; Grave, E.; Joulin, A.; Mikolov, T. Enriching word vectors with subword information. Trans. Assoc. Comput. Linguist. 2017, 5, 135–146. [Google Scholar] [CrossRef]
- Battaglia, P.W.; Hamrick, J.B.; Bapst, V.; Sanchez-Gonzalez, A.; Zambaldi, V.; Malinowski, M.; Pascanu, R. Relational inductive biases, deep learning, and graph networks. arXiv 2018. [Google Scholar] [CrossRef]
- Bonatti, P.A.; Decker, S.; Polleres, A.; Presutti, V. Knowledge graphs: New directions for knowledge representation on the semantic web (dagstuhl seminar 18371). Schloss Dagstuhl-Leibniz-Zent. Fuer Inform. 2019, 8, 29–111. [Google Scholar] [CrossRef]
- Perrault, C.R.; Grosz, B.J. Natural-language interfaces. In Exploring Artificial Intelligence; Morgan Kaufmann: Burlington, MA, USA, 1988; pp. 133–172. [Google Scholar] [CrossRef]
- Nakashole, N.; Weikum, G.; Suchanek, F. PATTY: A taxonomy of relational patterns with semantic types. In Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, Jeju Island, Republic of Korea, 12–14 July 2012; pp. 1135–1145. [Google Scholar]
- Bordes, A.; Weston, J.; Collobert, R.; Bengio, Y. Learning structured embeddings of knowledge bases. In Proceedings of the AAAI Conference on Artificial Intelligence, San Francisco, CA, USA, 7–11 August 2011; Volume 25, pp. 301–306. [Google Scholar] [CrossRef]
- Socher, R.; Chen, D.; Manning, C.D.; Ng, A. Reasoning with neural tensor networks for knowledge base completion. Adv. Neural Inf. Process. Syst. 2013, 26, 926–934. [Google Scholar]
- Bordes, A.; Glorot, X.; Weston, J.; Bengio, Y. A semantic matching energy function for learning with multi-relational data: Application to word-sense disambiguation. Mach. Learn. 2014, 94, 233–259. [Google Scholar] [CrossRef]
- Jenatton, R.; Roux, N.; Bordes, A.; Obozinski, G.R. Joint learning of words and meaning representations for open-text semantic parsing. In Proceedings of the Artificial Intelligence and Statistics, PMLR, La Palma, Canary Islands, Spain, 21–23 April 2012; pp. 127–135. [Google Scholar]
- Socher, R.; Chen, D.; Manning, C.D.; Ng, A.; Jenatton, R.; Roux, N.; Bordes, A.; Obozinski, G.R. A latent factor model for highly multi-relational data. In Proceedings of the 26th Annual Conference on Neural Information Processing Systems: Advances in Neural Information Processing Systems, Lake Tahoe, NV, USA, 3–6 December 2012; Volume 25. [Google Scholar]
- Sutskever, I.; Tenenbaum, J.; Salakhutdinov, R.R. Modelling relational data using bayesian clustered tensor factorization. In Proceedings of the 22nd International Conference on Neural Information Processing Systems: Advances in Neural Information Processing Systems, Red Hook, NY, USA, 7–10 December 2009; Volume 22. [Google Scholar]
- Al-Moslmi, T.; Ocaña, M.G.; Opdahl, A.L.; Veres, C. Named entity extraction for knowledge graphs: A literature overview. IEEE Access 2020, 8, 32862–32881. [Google Scholar] [CrossRef]
- Nickel, M.; Tresp, V.; Kriegel, H.P. A three-way model for collective learning on multi-relational data. In Proceedings of the 28th International Conference on International Conference on Machine Learning, Bellevue, WA, USA, 28 June–2 July 2011; Volume 11, pp. 809–816. [Google Scholar]
- Nickel, M.; Tresp, V.; Kriegel, H.P. Factorizing Yago: Scalable machine learning for linked data. In Proceedings of the 21st International Conference on World Wide Web, Lyon, France, 16–20 April 2012; pp. 271–280. [Google Scholar] [CrossRef]
- Mikolov, T.; Chen, K.; Corrado, G.; Dean, J. Efficient estimation of word representations in vector space. arXiv 2013. [Google Scholar] [CrossRef]
- Bordes, A.; Usunier, N.; Garcia-Duran, A.; Weston, J.; Yakhnenko, O. Translating embeddings for modeling multi-relational data. Adv. Neural Inf. Process. Syst. 2013, 26, 2787–2795. [Google Scholar]
- Wang, Z.; Zhang, J.; Feng, J.; Chen, Z. Knowledge graph embedding by translating on hyperplanes. In Proceedings of the AAAI Conference on Artificial Intelligence, Québec City, QC, Canada, 27–31 July 2014; Volume 28, pp. 1112–1119. [Google Scholar] [CrossRef]
- Lin, Y.; Liu, Z.; Sun, M.; Liu, Y.; Zhu, X. Learning entity and relation embeddings for knowledge graph completion. In Proceedings of the AAAI Conference on Artificial Intelligence, Austin, TX, USA, 25–30 January 2015; Volume 29, pp. 2181–2187. [Google Scholar] [CrossRef]
- Ji, G.; He, S.; Xu, L.; Liu, K.; Zhao, J. Knowledge graph embedding via dynamic mapping matrix. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing, Beijing, China, 26–31 July 2015; Volume 1, Long Papers. pp. 687–696. [Google Scholar] [CrossRef]
- Ji, G.; Liu, K.; He, S.; Zhao, J. Knowledge graph completion with adaptive sparse transfer matrix. In Proceedings of the AAAI Conference on Artificial Intelligence, Phoenix, AZ, USA, 12–17 February 2016; Volume 30. [Google Scholar] [CrossRef]
- Xiao, H.; Huang, M.; Hao, Y.; Zhu, X. TransA: An Adaptive Approach for Knowledge Graph Embedding. arXiv 2015. [Google Scholar] [CrossRef]
- Chen, M.; Tian, Y.; Chang, K.W.; Skiena, S.; Zaniolo, C. Co-training embeddings of knowledge graphs and entity descriptions for cross-lingual entity alignment. In Proceedings of the 27th International Joint Conference on Artificial Intelligence, Stockholm, Sweden, 13–19 July 2018; pp. 3998–4004. [Google Scholar] [CrossRef]
- Jamour, F.T.; Abdelaziz, I.; Kalnis, P. A Demonstration of MAGiQ: Matrix Algebra Approach for Solving RDF Graph Queries; The VLDB Endowment: Los Angeles, CA, USA, 2018; pp. 1978–1981. [Google Scholar] [CrossRef]
- Kaufmann, E.; Bernstein, A. How useful are natural language interfaces to the semantic web for casual end-users? In Proceedings of the International Semantic Web Conference, Busan, Republic of Korea, 11–15 November 2007; pp. 281–294. [Google Scholar]
- Kepner, J.; Aaltonen, P.; Bader, D.; Buluç, A.; Franchetti, F.; Gilbert, J.; Moreira, J. Mathematical foundations of the GraphBLAS. In Proceedings of the 2016 IEEE High Performance Extreme Computing Conference (HPEC), Waltham, MA USA, 13–15 September 2016; pp. 1–9. [Google Scholar] [CrossRef]
- Wang, S.; Zhang, Y.; Che, W.; Liu, T. Joint extraction of entities and relations based on a novel graph scheme. In Proceedings of the International Joint Conference on Artificial Intelligence, Stockholm, Sweden, 13–19 July 2018; pp. 4461–4467. [Google Scholar] [CrossRef]
- Gao, T.; Yao, X.; Chen, D. Simcse: Simple contrastive learning of sentence embeddings. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, Punta Cana, Dominican Republic, 7–11 November 2021; pp. 6894–6910. [Google Scholar] [CrossRef]
- Liu, Z.; Sun, M.; Lin, Y.; Xie, R. Knowledge Representation Learning: A Review. J. Comput. Res. Dev. 2016, 53, 247–261. [Google Scholar] [CrossRef]
Dataset | #Rel | #Ent | #Train | #Valid | #Test |
---|---|---|---|---|---|
WN11 | 11 | 38,696 | 112,581 | 2609 | 10,544 |
WN18 | 18 | 40,493 | 141,442 | 5000 | 5000 |
FB13 | 13 | 75,043 | 316,232 | 5908 | 23,733 |
FB15K | 1345 | 14,951 | 483,142 | 50,000 | 59,071 |
Model | #Parameters | #Operations (Time Complexity) |
---|---|---|
SLM | ||
NTN | O (((m2 + m) + 2mk+ k)Nt) | |
TransE | ||
TransH | ||
TransE-MTP |
Dataset | Model | Λ | γ | n, m | B | D.S |
---|---|---|---|---|---|---|
WN18 | TransE-MTP | 0.01 | 3.5 | 50 | 1440 | L1 |
FB15K | TransE-MTP | 0.001 | 1 | 200 | 4800 | L1 |
Dataset | WN18 | FB15K | |||||||
---|---|---|---|---|---|---|---|---|---|
Model | |||||||||
Metric | Mean Rank | Hits@10 | Mean Rank | Hits@10 | |||||
Raw | Filt | Raw | Filt | Raw | Filt | Raw | Filt | ||
RESCAL | 1180 | 1163 | 37.2 | 52.8 | 828 | 683 | 28.4 | 44.1 | |
SE | 1011 | 985 | 68.5 | 80.5 | 273 | 162 | 28.8 | 39.8 | |
SME | 542/526 | 533/509 | 65.1/54.7 | 74.1/61.3 | 274/284 | 154/158 | 30.7/31.3 | 40.8/41.3 | |
LFM | 469 | 456 | 71.4 | 81.6 | 283 | 164 | 26.0 | 33.1 | |
TransE | 263 | 251 | 75.4 | 89.2 | 243 | 125 | 34.9 | 47.1 | |
TransH (unif/bern) | 318/401 | 303/388 | 75.4/73.0 | 86.7/82.3 | 211/212 | 84/87 | 42.5/45.7 | 58.5/64.4 | |
TransE-MTP (unif/bern) | 261/238 | 249/226 | 73.2/76.6 | 85.2/87.3 | 237/241 | 83/160 | 50.8/52.5 | 78.6/75.2 |
Dataset | Model | α | λ | n, m | B | D.S |
---|---|---|---|---|---|---|
WN11 | TransE-MTP | 0.01 | 4 | 20 | 120 | L1 |
FB13 | TransE-MTP | 0.01 | 2 | 100 | 480 | L1 |
FB15K | TransE-MTP | 0.001 | 1 | 200 | 4800 | L1 |
Dataset | WN11 | FB13 | FB15K | |
---|---|---|---|---|
Model | ||||
SE | 53.0 | 75.2 | - | |
SME (bilinear) | 70.0 | 63.7 | - | |
SLM | 69.9 | 85.3 | - | |
LFM | 73.8 | 84.3 | - | |
NTN | 70.4 | 87.1 | 68.2 | |
TransE (unif/bern) | 75.9/75.9 | 70.9/81.5 | 77.3/79.8 | |
TransH (unif/bern) | 77.7/78.8 | 76.5/83.3 | 74.2/79.9 | |
TransE-MTP (unif/bern) | 81.7/81.3 | 79.8/84.86 | 84.2/83.6 |
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. |
© 2024 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Share and Cite
Li, Y.; Zhu, C. TransE-MTP: A New Representation Learning Method for Knowledge Graph Embedding with Multi-Translation Principles and TransE. Electronics 2024, 13, 3171. https://doi.org/10.3390/electronics13163171
Li Y, Zhu C. TransE-MTP: A New Representation Learning Method for Knowledge Graph Embedding with Multi-Translation Principles and TransE. Electronics. 2024; 13(16):3171. https://doi.org/10.3390/electronics13163171
Chicago/Turabian StyleLi, Yongfang, and Chunhua Zhu. 2024. "TransE-MTP: A New Representation Learning Method for Knowledge Graph Embedding with Multi-Translation Principles and TransE" Electronics 13, no. 16: 3171. https://doi.org/10.3390/electronics13163171