Learning Translation-Based Knowledge Graph Embeddings by N-Pair Translation Loss
Abstract
:1. Introduction
2. Related Work
3. Translation-Based Knowledge Graph Embeddings
4. Considering Multiple Negative Triples though N-Pair Translation Loss
Algorithm 1: Learning Translation-based Knowledge Graph Embeddings by Considering Multiple Negative Triples |
|
5. Experiments
5.1. Experimental Settings
5.1.1. Dataset
5.1.2. Evaluation Task and Protocol
5.1.3. Implementation
5.2. Experimental Results
6. Conclusions
Author Contributions
Funding
Acknowledgments
Conflicts of Interest
References
- Bordes, A.; Weston, J.; Collobert, R.; Bengio, Y. Learning Structured Embeddings of Knowledge Bases. In Proceedings of the Twenty-Fifth AAAI Conference on Artificial Intelligence, San Francisco, CA, USA, 7–11 August 2011; pp. 301–306. [Google Scholar]
- Nickel, M.; Murphy, K.; Tresp, V.; Gabrilovich, E. A Review of Relational Machine Learning for Knowledge Graphs. Proc. IEEE 2016, 104, 11–33. [Google Scholar] [CrossRef]
- Wang, Q.; Mao, Z.; Wang, B.; Guo, L. Knowledge Graph Embedding: A Survey of Approaches and Applications. IEEE Trans. Knowl. Data Eng. 2017, 29, 2724–2743. [Google Scholar] [CrossRef]
- Huang, H.; Heck, L.; Ji, H. Leveraging deep neural networks and knowledge graphs for entity disambiguation. arXiv 2015, arXiv:1504.07678. [Google Scholar]
- Riedel, S.; Yao, L.; McCallum, A.; Marlin, B.M. Relation Extraction with Matrix Factorization and Universal Schemas. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Atlanta, GA, USA, 9–15 June 2013; pp. 74–84. [Google Scholar]
- Weston, J.; Bordes, A.; Yakhnenko, O.; Usunier, N. Connecting Language and Knowledge Bases with Embedding Models for Relation Extraction. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, Washington, DC, USA, 18–21 October 2013; pp. 1366–1371. [Google Scholar]
- Bordes, A.; Chopra, S.; Weston, J. Question Answering with Subgraph Embeddings. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, Doha, Qatar, 25–29 October 2014; pp. 615–620. [Google Scholar]
- Cui, W.; Xiao, Y.; Wang, H.; Song, Y.; Hwang, S.W.; Wang, W. KBQA: Learning Question Answering over QA Corpora and Knowledge Bases. Proc. VLDB Endow. 2017, 10, 565–576. [Google Scholar] [CrossRef] [Green Version]
- Bordes, A.; Usunier, N.; Garcia-Duran, A.; Weston, J.; Yakhnenko, O. Translating Embeddings for Modeling Multi-relational Data. In Proceedings of the Advances in Neural Information Processing Systems 26, Lake Tahoe, NV, USA, 5–8 December 2013; pp. 2787–2795. [Google Scholar]
- Ji, G.; He, S.; Xu, L.; Liu, K.; Zhao, J. Knowledge Graph Embedding via Dynamic Mapping Matrix. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing, Beijing, China, 26–31 July 2015; pp. 687–696. [Google Scholar]
- Jia, Y.; Wang, Y.; Lin, H.; Jin, X.; Cheng, X. Locally Adaptive Translation for Knowledge Graph Embedding. In Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, Phoenix, AZ, USA, 12–17 February 2016; pp. 992–998. [Google Scholar]
- Lin, Y.; Liu, Z.; Sun, M.; Liu, Y.; Zhu, X. Learning Entity and Relation Embeddings for Knowledge Graph Completion. In Proceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence, Austin, TX, USA, 25–30 January 2015; pp. 2181–2187. [Google Scholar]
- Wang, Z.; Zhang, J.; Feng, J.; Chen, Z. Knowledge Graph Embedding by Translating on Hyperplanes. In Proceedings of the Twenty-Eighth AAAI Conference on Artificial Intelligence, Quebec City, QC, Canada, 27–31 July 2014; pp. 1112–1119. [Google Scholar]
- Xiao, H.; Huang, M.; Zhu, X. TransG: A Generative Model for Knowledge Graph Embedding. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, Berlin, Germany, 7–12 August 2016; pp. 2316–2325. [Google Scholar]
- Cai, L.; Wang, W.Y. KBGAN: Adversarial Learning for Knowledge Graph Embeddings. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, New Orleans, LA, USA, 1–6 June 2018; pp. 1470–1480. [Google Scholar]
- Wang, P.; Li, S.; Pan, R. Incorporating GAN for Negative Sampling in Knowledge Representation Learning. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, New Orleans, LA, USA, 2–7 February 2018; pp. 2005–2012. [Google Scholar]
- Zhang, Y.; Yao, Q.; Shao, Y.; Chen, L. NSCaching: Simple and Efficient Negative Sampling for Knowledge Graph Embedding. In Proceedings of the 2019 IEEE 35th International Conference on Data Engineering, Macau, China, 8–12 April 2019; pp. 614–625. [Google Scholar]
- Dash, S.; Gliozzo, A. Distributional Negative Sampling for Knowledge Base Completion. arXiv 2019, arXiv:1908.06178. [Google Scholar]
- Goodfellow, I.J.; Pouget-Abadie, J.; Mirza, M.; Xu, B.; Warde-Farley, D.; Ozair, S.; Courville, A.; Bengio, Y. Generative Adversarial Nets. In Proceedings of the Advances in Neural Information Processing Systems, Montreal, QC, Canada, 8–13 December 2014; pp. 2672–2680. [Google Scholar]
- Sohn, K. Improved Deep Metric Learning with Multi-class N-pair Loss Objective. In Proceedings of the Advances in Neural Information Processing Systems, Barcelona, Spain, 5–10 December 2016. [Google Scholar]
- Bollacker, K.; Evans, C.; Paritosh, P.; Sturge, T.; Taylor, J. Freebase: A Collaboratively Created Graph Database for Structuring Human Knowledge. In Proceedings of the 2008 ACM SIGMOD International Conference on Management of Data, Wuhan, China, 14–16 September 2008; pp. 1247–1250. [Google Scholar]
- Dong, X.; Gabrilovich, E.; Heitz, G.; Horn, W.; Lao, N.; Murphy, K.; Strohmann, T.; Sun, S.; Zhang, W. Knowledge vault: A web-scale approach to probabilistic knowledge fusion. In Proceedings of the 20th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, New York, NY, USA, 24–27 August 2014; pp. 601–610. [Google Scholar]
- Fader, A.; Soderland, S.; Etzioni, O. Identifying Relations for Open Information Extraction. In Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing, Scotland, UK, 27–31 July 2011; pp. 1535–1545. [Google Scholar]
- Mitchell, T.; Cohen, W.; Hruschka, E.R., Jr.; Talukdar, P.P.; Betteridge, J.; Carlson, A.; Mishra, B.D.; Gardner, M.; Kisiel, B.; Krishnamurthy, J.; et al. Never Ending Learning. In Proceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence, Austin, TX, USA, 25–30 January 2015; pp. 2302–2310. [Google Scholar]
- Nguyen, D.Q.; Sirts, K.; Qu, L.; Johnson, M. STransE: A novel embedding model of entities and relationships in knowledge bases. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, San Diego, CA, USA, 12–17 June 2016; pp. 460–466. [Google Scholar]
- Zhou, X.; Zhu, Q.; Liu, P.; Guo, L. Learning Knowledge Embeddings by Combining Limit-Based Scoring Loss. In Proceedings of the 2017 ACM on Conference on Information and Knowledge Management, Singapore, 6–10 November 2017; pp. 1009–1018. [Google Scholar]
- Qian, W.; Fu, C.; Zhu, Y.; Cai, D.; He, X. Translating Embeddings for Knowledge Graph Completion with Relation Attention Mechanism. In Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence, Stockholm, Sweden, 13–19 July 2018; pp. 4286–4292. [Google Scholar]
- Ebisu, T.; Ichise, R. Generalized Translation-Based Embedding of Knowledge Graph. IEEE Trans. Knowl. Data Eng. 2020, 32, 941–951. [Google Scholar] [CrossRef]
- Zhu, Q.; Zhou, X.; Zhang, P.; Shi, Y. A neural translating general hyperplane for knowledge graph embedding. J. Comput. Sci. 2019, 30, 108–117. [Google Scholar] [CrossRef]
- Lei, J.; Ouyang, D.; Liu, Y. Adversarial Knowledge Representation Learning Without External Model. IEEE Access 2019, 7, 3512–3524. [Google Scholar] [CrossRef]
- Han, X.; Cao, S.; Lv, X.; Lin, Y.; Liu, Z.; Sun, M.; Li, J. OpenKE: An Open Toolkit for Knowledge Embedding. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, Brussels, Belgium, 31 October–4 November 2018; pp. 139–144. [Google Scholar]
- Zhang, D.; Li, M.; Jia, Y.; Wang, Y.; Cheng, X. Efficient Parallel Translating Embedding for Knowledge Graphs. In Proceedings of the International Conference on Web Intelligence, Leipzig, Germany, 23–26 August 2017; pp. 460–468. [Google Scholar]
- Miller, G.A. WordNet: A lexical database for English. Commun. ACM 1995, 38, 39–41. [Google Scholar] [CrossRef]
- Toutanova, K.; Chen, D. Observed versus latent features for knowledge base and text inference. In Proceedings of the 3rd Workshop on Continuous Vector Space Models and Their Compositionality, Beijing, China, 31 July 2015; pp. 57–66. [Google Scholar]
- Kingma, D.P.; Ba, J. Adam: A Method for Stochastic Optimization. In Proceedings of the International Conference on Learning Representations, San Diego, CA, USA, 7–9 May 2015. [Google Scholar]
- Glorot, X.; Bengio, Y. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, Sardinia, Italy, 13–15 May 2010; pp. 249–256. [Google Scholar]
- Trouillon, T.; Welbl, J.; Riedel, S.; Gaussier, É.; Bouchard, G. Complex embeddings for simple link prediction. In Proceedings of the 33th International Conference on Machine Learning, New York, NY, USA, 19–24 June 2016; pp. 2071–2080. [Google Scholar]
Dataset | WN18RR | FB15K-237 |
---|---|---|
# Entities | 93,003 | 14,541 |
# Relations | 11 | 237 |
# Training Triples | 86,835 | 272,115 |
# Validation Triples | 3034 | 17,535 |
# Test Triples | 3134 | 20,466 |
Embeddings | Dataset | WN18RR | FB15K-237 | ||
---|---|---|---|---|---|
Metric | MRR | Hits10 | MRR | Hits10 | |
TransE | Margin | 14.5 | 37.6 | 27.0 | 43.6 |
N-pair | 23.7 | 53.0 | 32.6 | 50.5 | |
KBGAN [15] | 21.0 | 47.9 | 27.8 | 45.3 | |
NSCaching [17] | 20.5 | 47.4 | 30.0 | 47.4 | |
TransD | Margin | 15.0 | 38.7 | 25.4 | 43.3 |
N-pair | 22.6 | 49.4 | 31.8 | 50.3 | |
KBGAN [15] | 27.7 | 45.8 | 21.5 | 46.9 | |
NSCaching [17] | 20.1 | 48.4 | 28.8 | 48.3 |
© 2020 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).
Share and Cite
Song, H.-J.; Kim, A.-Y.; Park, S.-B. Learning Translation-Based Knowledge Graph Embeddings by N-Pair Translation Loss. Appl. Sci. 2020, 10, 3964. https://doi.org/10.3390/app10113964
Song H-J, Kim A-Y, Park S-B. Learning Translation-Based Knowledge Graph Embeddings by N-Pair Translation Loss. Applied Sciences. 2020; 10(11):3964. https://doi.org/10.3390/app10113964
Chicago/Turabian StyleSong, Hyun-Je, A-Yeong Kim, and Seong-Bae Park. 2020. "Learning Translation-Based Knowledge Graph Embeddings by N-Pair Translation Loss" Applied Sciences 10, no. 11: 3964. https://doi.org/10.3390/app10113964