Knowledge Graph Construction: Extraction, Learning, and Evaluation

Abstract

A Knowledge Graph (KG), which structurally represents entities (nodes) and relationships (edges), offers a powerful and flexible approach to knowledge representation in the field of Artificial Intelligence (AI). KGs have been increasingly applied in various domains—such as natural language processing (NLP), recommendation systems, knowledge search, and medical diagnostics—spurring continuous research on effective methods for their construction and maintenance. Recently, efforts to combine large language models (LLMs), particularly those aimed at managing hallucination symptoms, with KGs have gained attention. Consequently, new approaches have emerged in each phase of KG development, including Extraction, Learning Paradigm, and Evaluation Methodology. In this paper, we focus on major publications released after 2022 to systematically examine the process of KG construction along three core dimensions: Extraction, Learning Paradigm, and Evaluation Methodology. Specifically, we investigate (1) large-scale data preprocessing and multimodal extraction techniques in the KG Extraction domain, (2) the refinement of traditional embedding methods and the application of cutting-edge techniques—such as Graph Neural Networks, Transformers, and LLMs—in the KG Learning domain, and (3) both intrinsic and extrinsic metrics in the KG Evaluation domain, as well as various approaches to ensure interpretability and reliability.

1. Introduction

1.1. Background

Knowledge Graphs (KGs) were initially popularized by Google as a means to enhance search capabilities. Subsequently, the rise of Large Language Models (LLMs) has positioned KGs as a robust method for organizing large-scale heterogeneous data, and they have been applied in semantic search, recommendation systems, and question answering [1,2,3]. By representing entities (nodes) and their interlinked relationships (edges), KGs improve data retrieval efficiency and the interpretability of trained models, and even mitigate the hallucination effects often associated with LLMs [4]. However, constructing high-quality KGs remains complex due to diverse data sources, nuanced relationship learning, and the need for rigorous enhancement and evaluation.

This survey provides an overview of research from 2022 to 2024 on KG construction—covering extraction, learning, and evaluation—and examines recent work addressing hallucinations and knowledge gaps in various LLM-driven tasks. Through this exploration, we identify key challenges and unresolved issues in the current knowledge graph landscape, offering insights for future innovation.

1.2. Survey Taxnomy

Figure 1 provides a high-level overview of the proposed research framework for knowledge graph construction, visually representing the three core phases—Extraction, Learning, and Evaluation—along with their key sub-components. This figure is crucial not only for understanding the logical progression of each phase but also for situating the various methodologies discussed in this paper. The diverse methodologies presented in the main text are sometimes reiterated to help the readers better comprehend the content. By aligning the surveyed taxonomy with the structure shown in Figure 1, readers can easily understand how foundational extraction techniques, advanced machine learning approaches, and rigorous evaluation protocols collectively form a cohesive pipeline. In particular, the sub-stages within each phase (e.g., Pre-Construction Preparation for Extraction, Graph Neural Network-Based Learning, Qualitative Evaluation methods, etc.) reflect the positioning of the various research threads synthesized in this paper, thereby clarifying how each method and reference discussed in the following sections fits into the broader knowledge graph construction roadmap.

Extraction. Section 2 discusses the various processes involved in collecting and transforming raw data into structured data to construct knowledge graphs. This includes a range of extraction techniques, from rule-based and machine learning-based methods for foundational entity and relationship extraction to approaches that enhance extraction accuracy using LLMs (e.g., transformer-based models). By analyzing diverse methods, from traditional rule-based approaches to deep learning architectures, this section aims to provide an understanding of both conventional techniques and the latest advancements.

Learning. Section 3 focuses on machine learning techniques that infer and refine relational patterns within knowledge graphs. The discussion begins by exploring traditional embedding models and relation learning algorithms—such as TransE and DistMult—that provide a foundational framework for representing entities and their interconnections in a low-dimensional space. Building on these established methods, this section then delves into recent developments that utilize Graph Neural Networks, attention-based architectures, and contrastive learning techniques to capture complex interactions and higher-order structures. Additionally, it highlights advanced strategies, including transfer learning and various few-shot, one-shot, and zero-shot approaches that address data sparsity and enhance the generalization and robustness of the knowledge graph models.

Evaluation. Section 4 examines the various metrics and evaluation frameworks used to assess the performance of Knowledge Graphs (KGs). The evaluation process consists of intrinsic and extrinsic evaluations. Intrinsic evaluation measures the internal quality of a knowledge graph by focusing on three key aspects: First, accuracy evaluates whether the entities and relationships are correctly identified. Second, coverage indicates the extent to which a KG represents a relevant domain. Third, consistency ensures that the information in the KG is maintained without contradictions. In contrast, extrinsic evaluation measures how effectively a KG can be utilized in practical downstream tasks such as information retrieval and question answering. Furthermore, this section provides a comparative analysis of various evaluation benchmarks, including widely used datasets and challenge tasks, to comprehensively highlight the strengths and limitations of current approaches.

In summary, this survey aims to integrate the recent research landscape surrounding knowledge graph construction into a coherent roadmap, providing a reference for researchers and practitioners seeking to transcend the current limitations of the field. By comprehensively exploring extraction, learning, enhancement, and evaluation, we aim to lay a foundation for future innovations. The following sections will delve into each aspect in detail, thoroughly reviewing the methodologies, challenges, and emerging trends in this domain.

1.3. Semi-Automatic Paper Selection Process

In this paper, a semi-automatic survey process was implemented to rapidly and systematically synthesize research trends on Knowledge Graphs spanning 2022 to 2024. The overall procedure comprises four key steps.

Collection Articles. The Web of Science (WOS) database was utilized to automatically collect approximately 4000 articles related to Knowledge Graphs. This approach enabled the systematic acquisition of a vast body of literature, providing a robust foundation for subsequent evaluation and analysis.

Relevance Assessment. Each collected article was evaluated for alignment with the survey objectives using a predefined evaluation prompt. Specifically, GPT-4o mini and GPT-4o were employed to automatically generate relevance scores based on the content and research direction of the articles. This objective assessment facilitated the selection of articles most pertinent to the survey, ensuring an unbiased and systematic review process.

As a result of this scoring process, the number of papers in Knowledge Graph (KG) research was determined across three major categories: Extraction, Learning, and Evaluation. Figure 2 presents the distribution of these papers, with 74, 87, and 40 papers in each category, respectively. These values reflect the assessed relevance and suitability of each paper for this study, providing a structured overview of the research landscape.

Category Generation. Guided by the relevance scores, the articles were initially clustered into three primary themes—Extraction, Learning, and Evaluation. Subsequent analysis refined these broad categories, yielding three distinct sub-categories within each theme. While automated methods—such as keyword matching and topic modeling—helped identify candidate labels, researchers manually finalized and validated these sub-themes, ensuring their conceptual alignment with the broader survey goals.

By incorporating these semi-automated and human-centered measures, this study efficiently handled a large volume of literature while maintaining methodological rigor and transparency. Automated scoring (via GPT-4o mini and GPT-4o) proved valuable for initial filtering; however, the writing phase prioritized manual composition and expert review. Consequently, the final survey manuscript combines the speed and breadth of automation with the depth, reliability, and nuanced perspectives of human oversight.

2. KGs Extraction

This illustration details the annual distribution of research papers across various subfields of KG extraction from 2022 to 2024. This highlights that while some areas—such as relation extraction and multimodal/domain-specific extraction—maintain steady, high levels of scholarly activity, other subsections, like graph-based methods and intelligent reasoning, experience significant surges in certain years.

Steady or Growing Areas. Multimodal and Domain-Specific Extraction (Section 2.2.2) and Relation Extraction (Section 2.1.2) appear to maintain consistently high interest. This sustained attention could be driven by the increasing need to handle diverse data formats—such as images, videos, and specialized domain texts—and the growing importance of accurate entity-to-entity linking in complex applications.

Peaks and Surges in Specific Years. Topics like Graph-based Methods (Section 2.2.1) and Intelligent Reasoning (Section 2.3.1) show marked surges in certain years. These spikes may be attributed to breakthroughs in neural architectures (e.g., attention-based GNNs) or new reasoning paradigms that have temporarily shifted the research focus. Additionally, improvements in computational hardware and libraries can accelerate development in these areas, causing sudden increases in publication volume.

Fluctuations and Declines. Some subsections experienced sporadic peaks followed by rapid declines, suggesting that research efforts in those areas either reached a saturation point or were eclipsed by more innovative methods. Shifting priorities—such as the rise of end-to-end extraction pipelines or more advanced reasoning techniques—can lead to a decrease in traditional rule-based or less flexible approaches.

Emerging Use Cases and Practical Constraints. Real-world applications requiring scalable, domain-specific extraction solutions often motivate the research community to refine existing methods and explore novel frameworks. For instance, heightened interest in industry adoption might sustain certain topics for longer, whereas academic research might pivot quickly if a method shows diminishing returns or new challenges arise.

Overall, the trends in KG extraction highlight how some techniques remain stable focal points, whereas others gain or lose prominence in response to new methodological breakthroughs, changing computational capabilities, and domain-driven requirements.

2.1. Pre-Construction Preparation

2.1.1. Named Entity Recognition (NER)

In the initial stage of KG construction, it is essential to identify key terms (entities) from unstructured text and consistently map them to predefined ontologies or schemas. Approaches that leverage pretrained language models, such as BERT, have been shown to effectively improve NER precision when processing large corpora [5]. By combining statistical distributions with language graphs, these methods can extend not only commonsense knowledge but also domain-specific information.

In the relation extraction stage, techniques such as dependency parsing, semantic feature extraction, multi-head stacked graph convolution networks (GCN), and attention mechanisms are widely used [6]. For example, the entities identified by NER are embedded using a Bi-LSTM model, and methods such as Simplified Graph Convolution Networks (SGC) are then applied to optimize relational information, thereby enhancing the accuracy of relation extraction. Research has also been conducted on combining BERT-based NER with rule-learning methods (e.g., the Open Path or OPRL algorithms) to generate extraction queries and complete the KG [7].

Another study applied a supervised information extraction model based on a BERT transformer to perform tokenization, contextual embedding, and entity classification, and then fine-tuned it on a domain-specific dataset to improve its accuracy [8]. BERT’s core strength lies in its exceptional contextual understanding at the token level, which surpasses traditional approaches.

The entities identified by NER are then mapped to a predefined ontology or set of labels, which is a critical process for ensuring the consistency of KG creation [9]. This mapping is further reinforced by combining the statistical distributions with the language graph information. In addition, some studies have optimized KG performance using multi-relational graph attention networks or embedding-based models that reflect the importance or attention weights of adjacent nodes [10].

2.1.2. Relation Extraction

Relation extraction identifies the semantic connections between entities to generate triple structures (subject–relation–object), which form the basic structure of the knowledge graph. Techniques such as bidirectional relation-guided attention have been used to create a complementary effect between entity recognition and relation extraction [11]. In the cybersecurity domain, a method combining BERT-based NER with a GCN to extract attack behaviors significantly improved the accuracy of threat behavior recognition [12]. In various domains (e.g., healthcare, security, software, and education), BERT-based NER has become a core technology for KG construction.

Recent approaches have incorporated Retrieval-Augmented Generation (RAG) into KG applications to integrate engineering knowledge from texts. This method employs token classification and sequence-to-sequence (Seq2Seq) models to extract explicit engineering facts from the KG [13]. In contrast, the AutoAlign technique leverages large language models to automate the alignment process of knowledge graphs, significantly reducing the need for manual intervention [14].

Domain-specific studies include methods for extracting the relationships between ketamine and the gut microbiota from the literature and pathway data [15], approaches that capture diverse features using multi-channel convolutions in triple embeddings [16], methods for constructing a processed knowledge graph [17], approaches based on hierarchical transformers and dual quaternions [18], and triple extraction via sentence transformers utilizing dependency parsing [19].

Generative approaches extend and refine the KG by directly generating triple structures from learned data distributions, while sampling techniques effectively extract sub-structures from large graphs to balance accuracy and computational cost. For example, a multi-information preprocessing event extraction approach using BiLSTM-CRF with attention extracts events from academic texts to construct a KG [20]. Document-level dynamic graph attention integrates distributed information from an entire document using a two-stage graph strategy to improve the relation accuracy [21]. Seq2EG employs a sequence-to-sequence transformer to generate event graphs and incorporate complex event structures into the KG [22]. In addition, methods such as biomedical relation comparison, cognitively inspired multi-task frameworks [23], and KG linking through self-supervised learning have been proposed. SelfKG [24] leverages a novel self-supervised contrastive learning strategy that does not require labeled entity pairs by focusing on pushing away many negative samples. By employing a relative similarity metric along with self-negative sampling and multiple negative queues (inspired by momentum contrast techniques), SelfKG demonstrates that accurate entity alignment across different KGs can be achieved without any manual supervision. This approach not only reduces the labeling cost, but also achieves performance comparable to that of state-of-the-art supervised methods. along with the ERGM method—a multi-stage joint extraction technique that includes global entity matching [25]—and KG-based methods for automating complex scenarios, such as emergency planning or water resource management [26].

While Large Language Models (LLMs) have demonstrated state-of-the-art performance in extracting entity-relation tuples, they can occasionally generate “hallucinated” relationships that are not grounded in the source text. For example, a model may infer a causal link between two entities that are merely co-mentioned but lack any actual causal or logical connection. To mitigate such issues, recent studies have explored automated validation steps—such as cross-referencing extracted relationships against an external knowledge base or domain-specific rules—to filter out spurious links. Furthermore, incorporating fine-grained attention mechanisms and contrastive learning approaches can help ensure that relation extraction remains faithful to the available evidence, thereby reducing the risk of erroneously generated triples.

2.2. Advanced Entity and Relation Extraction

In some domain-specific corpora, LLM-driven extraction pipelines have generated “hallucinated” triples, such as attributing non-existent functions to certain genes or synthesizing medical relations (e.g., drug–condition links) that are not supported by clinical evidence. These errors typically arise when the training data are sparse, ambiguous or contradictory. To address this, researchers have proposed post-processing heuristics—using either knowledge graph consistency checks or multimodal cross-validation (e.g., text plus imaging data)—to detect and remove unsubstantiated relationships before finalizing the KG.

2.2.1. Graph-Based Methods (GNN and Attention)

Graph-based extraction methods organize the analysis results (entities and relations) from text into a graph structure and use Graph Neural Networks (GNN) and attention mechanisms to infer and validate information. For example, one approach achieved high precision by combining NER with a deep learning transformer to verify the entities and relations extracted at the sentence level [27].

In the biological domain, studies have combined BERT-based NER with topological clustering to learn the gene interaction structures within protein networks. In addition, research has been conducted on extracting drug–drug interactions by integrating heterogeneous knowledge graphs [28] and methods that extract relationships by merging web tables [29]. Other approaches include graph-based methods using multimodal learning [30], research on expanding KGs from large heterogeneous data sources [31], and comprehensive information extraction pipelines, where BERT-based NER plays a core role [32].

Studies have reported the use of multi-task graph convolution networks (MGCN) to extract entities and relations. For example, by integrating dependency trees into a GCN, structure information can be captured while simultaneously learning node and edge features, effectively handling overlapping or complex relation labels [33]. Furthermore, various methods have been employed, such as combining BERT-based NER with translation-based frameworks like TransE for accurate extraction from domain datasets [34] and constructing KGs using multimodal educational data [35].

Other approaches include using BiLSTM-CRF models to extract named entities and adding entity type information through unsupervised learning to reduce false positives [36] and employing TrellisNet-CRF to complement the accuracy of BERT-based NER [37]. Strategies that integrate graph structure learning, attention-based BERT NER, and automated feature generation (combining deep learning with graph learning) further enhance KG extraction performance [38]. Finally, a strategy was developed that uses BERT-based NER to automatically extract triple structures from geological reports and link them to a predefined ontology [39].

2.2.2. Multimodal and Domain-Specific Extraction

Multimodal and domain-specific approaches are designed to handle various data types—not only text but also images, audio, and sensor data—thus enabling the high-precision extraction of entities and relations in specific industries. For instance, the GridOnto system integrates fault events from power grids into a KG, demonstrating its effective application in the energy sector [40]. In the security domain, LLM-TIKG utilizes large language models to construct a threat intelligence KG that enhances the extraction of security-related events [41]. Additionally, causal relation extraction methods focus on deriving deep causal relationships from long texts, thereby enriching the KG with explicit engineering and domain-specific facts [42]. Knowledge Guided Attention focuses on chemical–disease relationships by combining attention mechanisms with GCN to reduce noise [43]. Methods that use distant supervision to incorporate the global context and suppress errors have also been proposed [44]. The Experiencer-Driven Graph approach captures deep semantic information in emotion-cause relationships [45], and dual-fusion models that combine text with equipment sensor data have been developed to simultaneously analyze logs and documents [46]. An automated clinical knowledge graph framework that integrates clinical records and literature to support evidence-based medicine (EBM) is also included [47].

Additional domain-specific improvement techniques include progressive entity type assignment methods to effectively learn domain-specific ontologies [48] and the CG-JKNN model, which integrates image and graph data to form medical relations for tuberculosis diagnosis [49]. In the industrial and healthcare sectors, systems such as the Novel Rational Medicine Use System [50], semantic-spatial aware data integration for place KGs [51], relation labeling in product KGs for e-commerce [52], pipelines for extracting relations using transportation authority data [53], and task-centric KG construction based on multimodal representation learning for industrial maintenance automation [54] have been proposed. For geological data, approaches include deep learning-based methods that combine text and images to extract entities [55], methods that fuse entity descriptions with type information to enhance the completeness of a medical KG [56], joint extraction approaches for geological reports [57], BDCore, which uses bidirectional decoding and joint graph representations [58], prompt learning for biomedical relation extraction using GCN [59], and NER for equipment fault diagnosis that combines RoBERTa-wwm-ext with deep learning [60].

2.3. Advanced Knowledge Graph Construction

2.3.1. Intelligent Reasoning Methods

KG construction is increasingly leveraging advanced techniques to automatically extend complex domain knowledge. In particular, methods such as multi-hop reasoning, causal reasoning, and quantum embeddings are being integrated to enhance knowledge representation. For example, PharmKE is a knowledge extraction platform that applies transfer learning to pharmaceutical texts to automatically extract various medical entities and relations (e.g., drug names, drug–drug interactions (DDI), adverse drug reactions (ADR), and indications) within an integrated environment [61]. By fine-tuning a BERT-like model on pharmaceutical corpora, it accurately recognizes specialized terms, such as drug codes and clinical names, and integrates them into the KG using subsequent pipelines. Similarly, an event ontology-based knowledge-enhanced event relation method actively leverages domain ontologies [62].

In addition, the Learning Relation Prototype maps long-tail relations with few labels in an unsupervised manner using prototype embeddings [63]. In software project KGs, the CAJP technique has been utilized [64], and Query Path Generation—performing bidirectional inference to handle complex question-answering tasks—has been applied [65]. The MHlinker model, which integrates the extraction of fault entity relations in mine hoists, has also been reported [66], as has a novel joint extraction approach that combines interactive encoding with visual attention to extract entities and relations simultaneously [67].

2.3.2. Rule-Based Approaches

Rule-based approaches achieve high precision by utilizing predefined rules and patterns established by domain experts, although they may be somewhat vulnerable to new variations. For example, the Heterogeneous Affinity Graph Inference method proposes a way to reduce noise when constructing document-level entity relations [68]. In addition, Image2Triplets combines BERT-based NER with computer vision techniques to extract relations simultaneously from images and text, applying specific rules (e.g., focus targets) to improve accuracy [69].

In the medical field, joint models that combine sequence labeling, GCN, and transformers have been proposed to extract adverse drug event (ADE) relationships from complex pharmaceutical texts [70]. QLogicE integrates BERT-based NER with quantum embeddings and translation-based models to enhance the efficiency of entity and relationship representations [71]. A generative model called KGGen manages the quality of the generated triple structures through adversarial learning, which combines negative sampling and quality control rules [72]. Furthermore, an improved attention network that combines deep learning with rule-based techniques enhances the accuracy of equipment used for KG construction [73]. AtenSy-SNER utilizes syntactic features and semantic augmentation for entity extraction in the software domain and reduces false positives using rule-based validation [74].

2.3.3. Entity and Relationship Alignment

Recent studies addressing the problem of entity and relation alignment have proposed various approaches aimed at effectively modeling complex contextual information and mitigating noise. One notable model is the Dual Attention Graph Convolutional Network (DAGCN) [75], which combines local attention and global attention to capture both local structural information around entities (e.g., neighboring nodes, adjacent words, attributes, etc.) and long-range dependencies arising from global contextual and syntactic relationships within sentences. By modeling both local and global contexts, DAGCN achieves high accuracy for complex sentences, as shown by benchmarks like TACRED and SemEval.

The Contextual Dependency-Aware GCN [76] integrates dependency-based graph structures with contextual information by segmenting sentences at the token level to build and process a dependency graph with a GCN. Unlike earlier methods that only consider simple dependency relations, it incorporates contextual embeddings to resolve polysemy and ambiguity, thereby enhancing entity and relation extraction, even in complex sentences.

Another notable line of research focuses on extracting and aligning relationships between objects and their corresponding actions and states in knowledge graphs. A proposed approach [77] automatically identifies and connects object-action and object-state relationships in knowledge graphs—such as linking “cup” with “drink” or “cup” with “fragile”. This method applies a sophisticated matching algorithm that enables the effective association of actions or states with objects, even in noisy or structurally complex contexts. By enriching the knowledge graph information with object-centric features and their associated attributes and actions, this approach enhances the graph’s semantic richness and utility.

Furthermore, an Interactive Optimization approach [78] has been introduced to incrementally improve model performance by incorporating continuous user or expert feedback during the relation extraction and alignment process. In this framework, misclassified cases or additional information identified during human inspection are fed back into the model, which adjusts its knowledge graph representation learning based on the feedback. Through repeated cycles of interaction and adjustment, the model gradually refines its alignment capabilities, enhancing its accuracy and reliability, even in complex domains where fully automated systems might struggle.

In summary, approaches based on attention-based modeling, dependency graph integration, object-action/state extraction, and interactive feedback have significantly improved the accuracy and robustness of entity alignment and relationship alignment. Specifically, the DAGCN [75] and Contextual Dependency-Aware GCN [76] effectively integrate diverse contextual information, achieving superior performance even in complex sentence structures. The object-action/state extraction technique [77] robustly captures intricate relationships in noisy environments, while the Interactive Optimization approach [78] leverages continuous feedback to iteratively refine model performance, ensuring higher reliability in challenging domains.

2.4. Final Summary

In the Pre-Construction Preparation phase, the focus is on laying a robust foundation for KG construction through two key steps. First, Named Entity Recognition employs advanced techniques such as BERT-based models, Bi-LSTM, CRF, and graph-based methods to accurately identify key entities and map them to predefined ontologies [5,6,7,8,9,10]. Next, Relation Extraction leverages methods like dependency parsing, semantic feature extraction, multi-head graph convolution networks, and attention mechanisms to extract subject–relation–object triples from unstructured text [11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26]

Building on these initial tasks, the Advanced Extraction stage further refines the process. In this stage, Graph-based Methods—using graph neural networks (GNNs) and attention mechanisms—organize entities and relations into structured graphs [27,28,29,30,31,32,33,34,35,36,37,38,39]. Additionally, Multimodal and Domain-Specific Extraction integrates diverse data sources (including text, images, and sensor inputs) to enhance precision in specialized fields such as healthcare, cybersecurity, and industrial applications [40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60].

Finally, during the Construction phase, the process is extended and automated by incorporating Intelligent Reasoning Methods—such as multi-hop reasoning, causal inference, and quantum embeddings—to support complex inference tasks [61,62,63,64,65]. This is complemented by Rule-based Approaches that utilize predefined rules, knowledge-guided attention, and logic-based models to further refine extraction accuracy in specialized domains [66,67,68,69,70,71,72]. Moreover, advanced techniques for Entity and Relationship Alignment ensure consistency and seamless integration across heterogeneous knowledge graphs [73,74,75,76,77,78].

3. KGs Learning

This section provides a structured guide to various methodologies in Knowledge Graph (KG) learning, covering foundational techniques, advanced models, and other learning methods. The emphasis lies on how these approaches contribute to tasks such as link prediction, relational inference, and structured data analysis. In the subsections that follow, key models and enhancements are described with both theoretical insights and practical examples drawn from diverse application domains.

Figure 3 illustrates the annual distribution of research papers on KG learning from 2022 to 2024, offering a closer look at how interest in specific subsections fluctuated over time. Overall, the learning domain exhibits patterns in which certain areas undergo substantial surges in 2023 or 2024, while others remain steady or even decline. Several potential reasons and driving factors are outlined below:

Notable surges in 2022–2024. Supervised plus Reinforcement Learning Hybrid Approach (Section 3.5.3): A dramatic increase in 2024 could stem from breakthroughs in reinforcement learning frameworks and the growing availability of large-scale data. Researchers have been motivated by the promise of achieving more robust and adaptive link prediction or reasoning tasks by combining supervised signals with reinforcement-based explorations.

Text Integration (Section 3.4.1) and Fusion of Topology with Logical Rules (Section 3.4.2): These areas display either steady growth or a marked resurgence. This uptake may be attributed to the release of more powerful natural language processing models (e.g., large language models) and an expanding set of well-structured corpora. As a result, integrating textual information with topological or logical features has become more feasible, thereby driving further research.

Consistent or Gradual Growth. Certain subfields (e.g., Heterogeneous Graphs or Temporal Knowledge Graphs) may show incremental increases in research output. This trend is often fueled by improvements in computational resources and an industry-wide shift toward real-time, dynamic KG applications—leading to a steady demand for incremental or temporal modeling approaches.

Fluctuations and Declines. Some subsections (e.g., Section 3.2.3 and Section 3.5.2) exhibit a concentration of research in a single year, followed by a sharp decrease. Such sporadic peaks may reflect shifting priorities—perhaps due to the saturation of certain methods, a pivot toward more promising alternatives, or newly emerging challenges that redirect attention. Additionally, competition from advanced approaches (like reinforcement learning hybrids or deep generative models) can overshadow previously popular techniques.

Overall, the observed trends highlight how KG learning research has evolved in response to novel techniques, growing computational capabilities, and increasing availability of domain-specific datasets. As new methodologies prove their value (e.g., reinforcement learning hybrids or improved NLP integration), they spur a surge of interest, whereas areas that face fewer unresolved challenges may experience relative decline.

3.1. TransE-Based Approaches

TransE [79] learns embeddings for triples (h, r, t) by representing entities (h, t) and relations (r) as low-dimensional vectors that ideally satisfy using a margin-based ranking loss that pulls real triples closer while pushing negative (corrupted) triples apart. Its simplicity, rapid training on large datasets, and scalability have made it a popular foundational method.

h + r ≈ t

(1)

However, a single-vector operation often struggles to capture complex relational patterns. In particular, one-to-many or many-to-one relationships pose a significant challenge. For example, by constructing a KG from patent and technical report data, basic embeddings are obtained with TransE and then enhanced by training additional diffusion and absorption vectors to predict “how ideas flow between technological domains” in Technological Knowledge Flow models [80]. To further improve accuracy, the Multi-Filter Soft Shrinkage technique combines convolutional layers with TransE embeddings as initialization, suppressing noise while modeling higher-order interactions [81].

Because TransE embeddings can degrade on sparse or incomplete graphs, iterative rule-guided reasoning methods employing reinforcement learning (RL) have been proposed to dynamically supplement “missing paths” and update embeddings [82]. The SSKGE framework integrates structural reinforcement and semantic guidance to reduce the training time and improve link prediction performance [83]. HolmE extends TransE by capturing compositional patterns in Riemannian space, providing a generalized embedding framework that encompasses models like TransE and RotatE [84].

Another extension employs TransE as the embedding backbone with a two-level RL mechanism for multi-hop reasoning. In this hierarchical RL approach, a high-level agent decides which relation to explore while a low-level agent selects the appropriate node for that relation, effectively compressing the action space and updating the model end-to-end [85]. To handle continuously emerging entities and relations, embedding generators are trained to create vectors dynamically from side information (e.g., text or attributes), preserving the overall vector space structure for real-time or online KG completion [86].

Finally, for temporal knowledge graphs (TKGs), techniques such as the Temporal-Structural Adaptation Network (TSA-Net) incorporate specialized embedding modules or recurrent mechanisms to capture complex temporal dependencies among temporal, relational, and entity representations [87].

Collectively, these studies demonstrate that while TransE [79] provides a simple and scalable foundation for knowledge graph embedding, its inherent limitations—such as difficulty in capturing complex relational patterns and sensitivity to data sparsity—have spurred various enhancements. For example, hierarchical and convolutional extensions [80,81] improve the modeling of knowledge flow and noise suppression, whereas iterative rule-guided reasoning frameworks using reinforcement learning [82,85] effectively supplement the missing paths in sparse graphs. Additionally, approaches for open-world and temporal graph completion [86,87] have been developed to dynamically adapt to emerging entities and time-dependent data.

3.2. Graph Neural Network (GNN)-Based Learning

GNNs learn node embeddings by iteratively aggregating and transforming information from nodes, edges, and their neighbors using message passing. When applied to KGs, GNNs can flexibly handle heterogeneous graphs with diverse nodes and relationship types.

3.2.1. Key Features and Application Highlights How GNNs

Graph Neural Networks (GNNs) learn embeddings for each node by iteratively aggregating and transforming the structure of the nodes, edges, and their neighbors using message-passing techniques. When applied to knowledge graphs, GNNs exhibit the following characteristics: First, heterogeneous graphs with diverse types of nodes and relationships can be flexibly handled using techniques like meta-paths or attention mechanisms. Second, while iterative structural learning enables information from neighboring nodes to propagate into deeper layers, allowing higher-order structure learning, caution is required to avoid over-smoothing. Multi-Grained Semantics-Aware Graph Neural Networks [88] proposed a GNN that effectively learns the rich semantic information of knowledge graphs by considering multi-grained semantics. Furthermore, the Graph Relearn Network [89] introduced methods to mitigate performance fluctuations in GNNs and improve prediction accuracy. Recent work has further expanded the application scope of GNNs. For instance, ref. [90] leveraged spatiotemporal scene graph embeddings—where GNNs are combined with LSTM layers—to predict autonomous vehicle collisions by capturing dynamic inter-object relationships in complex traffic scenes. Similarly, ref. [91] demonstrated that integrating Graph Convolutional Neural Networks with Label Propagation can significantly enhance node classification performance by adaptively learning edge weights based on node label information.

The GNN-based Imbalanced Node Classification Model (GNN-INCM) [92] is designed to address class imbalance issues by applying loss balancing along with node sampling strategies to enhance performance in minority classes. Specifically, it selects nodes based on importance and utilizes the structural characteristics of graph networks to effectively learn from minority class data. This leads to a robust node classification performance, even in datasets with severe class imbalances. INS-GNN [93], which integrates self-supervised learning into GNNs, ensures stable node classification performance on imbalanced datasets. By autonomously learning the characteristics of graph data, this model alleviates the problem of under-learning in minority class nodes, achieving practical performance improvements in various real-world networks. GNNs have emerged as a powerful paradigm for learning structured graph data by leveraging both node attributes and underlying graph topology to generate informative embeddings [94]. By aggregating information from neighbors, GNNs capture complex dependencies within graphs, making them highly effective for tasks like node classification and link prediction. The Graph Convolutional Network (GCN), an early GNN framework, introduced convolution-like operations tailored for graph-structured data [95].

The Graph Attention Network (GAT) architecture refines this process by incorporating an attention mechanism to assign adaptive weights to neighbor information [94]. The same GAT architecture leverages a specialized relational aggregator that captures nuanced interactions associated with various entity and relationship types, making it well-suited for heterogeneous graphs [94,96]. The Dynamic Representation of Relations and global information (DRR-GAT) [97] extends the existing GAT by applying dynamic relation-specific weights, effectively modeling complex graph structures. Iterative Graph Self-Distillation (IGSD) [98] employs an iterative self-distillation technique to enhance the prediction stability of GNN models by refining the latent representation of nodes. At each iteration, it generates new embeddings to improve the model prediction accuracy, particularly in large graph datasets, reducing the model variance and delivering more precise results.

Beyond general graph analytics, GNNs serve as the cornerstone of knowledge graph (KG) learning, supporting various tasks such as entity classification, relationship prediction, and KG completion [99,100]. In this context, GNNs leverage node embeddings to exploit relational information, uncovering complex semantic patterns encoded in KGs. The integration of contrastive learning techniques within GNN frameworks is especially noteworthy, as it promotes the alignment of embeddings with meaningful semantic subspaces. This alignment aids in distinguishing relational patterns, ultimately enhancing the performance of KG completion and other downstream tasks [101]. Additionally, the flexibility of GNNs has spurred the emergence of advanced architectures, such as higher-order models that capture multi-hop dependencies and path-based models that incorporate longer relational contexts to further refine these relationships [102]. These diverse approaches underscore the adaptability of GNNs, highlighting their capability to tackle various learning scenarios and enhance the utility of knowledge graphs in real-world applications.

Compared to TransE-based methods, GNN-based approaches leverage iterative message passing to capture both local and higher-order relational structures. This makes them particularly effective for heterogeneous graphs, where diverse node types and complex interactions are prevalent [88,89,90,91,92,93,94,95,96,97,98,99,100,101,102]. In particular, architectures such as the Graph Relearn Network [89] and Graph Attention Network [94,96] have been shown to mitigate issues like over-smoothing and performance fluctuations, resulting in more robust performance in tasks such as node classification and link prediction.

3.2.2. Link Prediction and Node Classification

In Graph Neural Network (GNN)-based learning, link prediction and node classification are two key tasks that leverage the inherent structure of graphs. Link prediction infers missing or potential future connections by analyzing the relational patterns among nodes, while node classification assigns labels to nodes by considering both their individual features and the influence of their neighboring nodes. Together, these tasks empower various applications, such as recommendation systems, network analysis, and anomaly detection, by effectively capturing the underlying connectivity within the graph.

By representing objects such as vehicles, pedestrians, and road facilities as a graph and combining it with a temporal LSTM, SG2VEC [88] can predict future collision events. Node embeddings are directly optimized for the downstream task of “collision likelihood” and are lightweight for fast inference on actual AV (autonomous vehicle) edge hardware. MuCoMiD [103] addresses biological knowledge graphs (genes, miRNAs, diseases, proteins, etc.) using GNN-based multi-task learning. It addresses challenges such as data scarcity and noise by enabling automatic feature extraction and integrating more than five different biological data sources. The GNN layers learn the interactions between nodes (biomolecules), demonstrating robust predictions even with limited labeled data.

The heterogeneous graph-based knowledge tracing method with spatiotemporal evolution (TSKT) [104] is designed to learn from graph data containing heterogeneous nodes and relationships. This model captures complex interactions among various node and relationship types through meta-paths and meta-graphs, achieving excellent performance in modeling learners’ learning patterns. The Graph Structure Self-Contrasting (GSSC) [105] approach enhances graph-based learning performance by integrating graph structures into a multilayer perceptron (MLP). By incorporating graph structures into the input layer of the MLP, this model effectively learns the nonlinear interactions between nodes, significantly improving the node classification performance.

The Graph-Aware Tensor Factorization Convolutional Network (GATFCN) [106] combines tensor factorization with graph convolution (GCN). By integrating graph structural information into the tensor factorization process, this model learns more precise interactions between nodes and relationships, achieving outstanding performance in tasks like link prediction. Ref. [107] combined a GNN with self-supervised learning to detect adverse effects in drug−drug interactions. This model learns complex relationships from large-scale drug datasets and captures patterns of drug−drug similarities, interactions, and side effects. It predicts potential side effects based on real clinical data, aiding in drug safety evaluation and clinical trial design. GraphX-Net [108], a GNN model designed for cancer survival prediction, represents patient data as nodes and similarities as edges to construct the graph structure. This model evaluates the contribution of each sample (patient) within the graph and accurately predicts the probability of survival in patients with cancer. By leveraging geometric features and graph structural information, advanced cancer survival predictions are supported, significantly contributing to personalized treatment design and survival analysis.

3.2.3. Heterogeneous, Biomedical, and Large-Scale Graphs

In heterogeneous information networks (HINs) containing different types of nodes and relationships (e.g., person-organization, paper-author, drug-disease), GNNs leverage meta-paths and meta-graphs to learn diverse proximities [94]. For example, HIGCN (Heterogeneous graph convolutional network with local influence) calculates the local importance of each node to extract multi-faceted semantics within heterogeneous graphs. GNNs are actively utilized to predict the survival rates of patients with cancer, drug−drug interaction (DDI) predictions, and hospital knowledge graph expansions [95]. Advanced variants like Graph Neural Topic Models and Bayesian GNNs, have also emerged to address challenges in medical data, such as uncertainty, heterogeneity, and scalability. For instance, the Geometric GNN [109], used for cancer survival prediction based on omics data, calculates node embeddings by considering the geometric features of the data (e.g., curvature and Ricci curvature). A comprehensive survey of graph embedding techniques tailored for biomedical data [110] provides an overview of graph-specific embedding methods for biomedical applications.

For large-scale knowledge graphs (KGs) containing hundreds of millions to billions of nodes, the cost of message passing becomes a significant bottleneck, leading to research on distributed implementations or model parallelization [111]. For example, the distributed non-negative RESCAL algorithm integrates GNN-like operations to maximize the efficiency. Moreover, automatic model selection in large-scale environments is a critical issue. The distributed non-negative RESCAL with automatic model selection for exascale data [111] introduces a distributed RESCAL embedding method suitable for exascale datasets, enabling the efficient learning of large-scale knowledge graphs.

3.3. Dynamic and Task-Specific Learning

3.3.1. Temporal Knowledge Graphs (TKG)

Research is being conducted on GNNs that learn temporal knowledge graphs (TKGs) that change over time, enabling the inference of future facts from past data. When tracking structural changes over time, the GNN layers update the node representations by integrating the current timestamp graph with past (or neighboring) timestamps. For example, SFTe [112] predicts future interactions by embedding the structure, factuality, and temporality individually.

To adapt quickly, even with few-shot training samples for new relations or entities, research is exploring GNNs with meta-learning to dynamically adjust parameters [113]. Models like MTRN (task-related network based on meta-learning) employ specialized components, such as neighbor-aware encoders and self-attention encoders, to sensitively respond to task transitions. Knowledgebra [114] proposed an embedding method that optimizes the model performance by leveraging the algebraic structure of KGs to enhance global consistency. Dynamic relation learning for link prediction in knowledge hypergraphs [115] improves the link prediction performance in hypergraphs through dynamic relation learning.

T-GAE [116] utilizes temporal information in graphs to learn the interactions between nodes. It employs a graph attention mechanism to integrate spatiotemporal dependencies, which allows the model to capture dynamic changes in temporal graphs, predict future interactions from past data and handle complex temporal patterns efficiently. The Bayesian hierarchical graph neural network (BHGNN) [117] incorporates uncertainty in graph data into the learning process to produce highly reliable results. It is particularly effective in tasks where reliability is crucial, such as industrial fault diagnosis, in which uncertainty feedback is used to simultaneously improve prediction accuracy and model confidence. This approach contributes to error detection and refinement in several scenarios.

CDRGN-SDE [118] employs a cross-dimension network to model interactions across multiple dimensions when learning temporal data. The model captures changing graph structures along the temporal axis and learns complex dependencies in temporal graphs to make accurate predictions. This enables effective learning, even in large-scale temporal graphs. Ref. [119] proposed a model that simultaneously learns multiple graph structures along the temporal axis and infers relationships and patterns in TKGs. This model efficiently captures time-dependent patterns and reflects interactions across the past, present, and future.

Dynamic learning approaches in temporal knowledge graphs not only adapt to evolving data but also help mitigate issues such as hallucinated relationships. For instance, models like T-GAE adjust node representations over time to reduce the occurrence of spurious or unsupported links. For example, models like T-GAE [116] incorporate temporal-structural adaptations to adjust node representations over time, reducing the occurrence of spurious or unsupported relations. Additionally, frameworks that integrate reinforcement learning [117] provide mechanisms to continuously refine the graph structure, ensuring that errors due to hallucination are minimized in real-world applications.

3.3.2. Extensions: Recommendations, Reinforcement Learning, and Beyond

When user-item, item-attribute, or item-item interactions are represented as a graph and GNNs are used to learn node embeddings, the recommendation accuracy improves significantly [120]. EMKR employs multi-task learning (recommendation + KGE) to enhance the performance of both tasks. Reinforcement Negative Sampling (KGRec-RNS) uses reinforcement learning (RL) to identify “meaningful” negative samples [121], which provide far more informative negatives than random sampling, improving training efficiency and interpretability.

For fully inductive learning in evolving KGs, one method has been proposed to classify or predict links using GNNs on sampled subgraphs rather than the entire graph [96]. Techniques such as personalized PageRank (PPR) are employed to extract neighboring nodes, which helps mitigate over-smoothing and enables flexible inductive reasoning. Ref. [122] proposed a sequential recommendation system based on knowledge graphs and transformers, and ref. [123] introduced an enhanced multi-task learning framework that integrates recommendation and KG embedding tasks. Additionally, some approaches combine first-order neuro-symbolic and logic rule embeddings to improve the explainability of recommendation tasks [124]. For example, logical rules such as “If X is related to Y, and Y is similar to Z, then X is related to Z” can be used in conjunction with neural embeddings. Moreover, BiG-Fed [125] further enhances the recommendation performance by combining hierarchical optimization with federated learning.

Ref. [126] introduced a method for extracting local subgraphs from large-scale graphs to reduce the training time and improve prediction accuracy. This approach performs link prediction based on subgraphs, achieving high efficiency and performance without requiring the entire graph to be trained, making it particularly useful for large-scale datasets like recommendation systems. Ref. [127] proposed a model for knowledge graph-based question generation using subgraphs. By focusing on specific nodes and relationships, the model generates questions that reflect the structural information from the graph, enabling more precise responses in question-answering systems.

Global-Local Anchor Representation (GLAR) [128] is an inductive link prediction model centered on subgraphs, capable of effectively predicting links in dynamic environments where new nodes and relationships are continuously added. This model leverages the properties of local subgraphs to enable scalability and achieves high performance with an efficient computation.

In summary, dynamic and task-specific learning models offer significant advantages over static embedding methods by adapting to new data using meta-learning and dynamic relation learning. This adaptability not only improves the overall accuracy but also enhances the robustness of the knowledge graphs by reducing errors, such as hallucinated relationships.

3.4. Transformer-Based Learning

Transformers have demonstrated exceptional performance in natural language processing, leading to a significant increase in research combining Transformers with knowledge graphs (KGs). For KGs that extensively utilize textual information from documents, the self-attention mechanism allows modeling of long-range dependencies and subtle contextual relationships between entities. This capability has positioned Transformers as an increasingly influential approach to KG learning [129].

3.4.1. Text Integration and Finetuning

Research such as MHAVGAE [130] combines weakly supervised learning with a Multi-head Attention Variational Graph Auto-Encoder, integrating Transformer-style multi-head attention into a VAE to automatically learn prerequisite relationships in large-scale conceptual graphs. Metrics like Resource-Prerequisite Reference Distance (RPRD) are used to augment incomplete labels, significantly reducing the need for manual labeling.

Pretrained text-based models (e.g., BERT) have inspired dual attention structures in graph layers, assigning different attention channels based on the importance of the relations [131]. For instance, the Dual-AEN model independently updates entities and relations while facilitating their interactions.

Methods like SATrans [132] inject structural information from tables and graphs directly into the attention mask of a Transformer encoder, mitigating structural loss caused by “flattening” data into plain text. The lower layers of self-attention focus on local information, while the upper layers emphasize global information, achieving high accuracy in tasks that reflect local-global structures, such as validation and question answering. Fine-tuning these models on domain-specific KG data enables the capture of nuanced semantic features that facilitate tasks like relation extraction and entity linking [133].

Ref. [134] proposed a technique to expand knowledge graphs (KGs) using pretrained language models (e.g., BERT, GPT). This approach extracts entities and relationships from text data and integrates them into the KG, thereby enriching its representation. By leveraging the strong language understanding capabilities of pretrained models, this method adds new information that existing KGs may lack and enhances the semantics of the existing relationships. The synergy between NLP and KG in this approach improves performance across various applications, including question answering and recommendation systems.

Transformer architectures (e.g., BERT) have become the dominant paradigm in knowledge graph construction, particularly due to their ability to capture long-range dependencies. However, traditional recurrent neural networks (RNNs), such as LSTM or GRU models, can still offer advantages in terms of computational efficiency for shorter sequences.

Table 1. Comparison of RNN-based methods vs. transformer-based methods.

Method	Core Technique	Pros	Cons	Typical Use Cases
RNN (e.g., LSTM/GRU)	Sequential processing of tokens over time	Efficient, Simple	Limited, Forgetful	Small-scale, Streaming
Transformer (e.g., BERT)	Self-attention mechanism (non-sequential)	Powerful, Accurate	Expensive, Memory intensive	Large-scale, Complex

summarizes the key differences between these methods in terms of accuracy, efficiency, and practical applicability.

3.4.2. Fusion of Topology and Logical Rules

The Fuse Topology contexts and Logical rules in Language Model (FTL-LM) [135] enhances knowledge graph (KG) learning by combining graph topology information (structural relationships between nodes) with logical rules. This model imposes constraints on relationships within the graph through logical rules while learning the structural interactions of nodes and edges via topology information. For instance, it improves the reliability of relation prediction using rules like “If X is related to Y, and Y is related to Z, then X is also related to Z”. This approach excels in domains where logical consistency is critical, such as healthcare and the law.

The Medical Bidirectional Encoder Representations from Transformers (MCBERT) [136] system learns the relationships among diseases, drugs, and symptoms using domain-specific KGs. By modeling complex relationships in medical data using KGs, MCBERT facilitates efficient analysis and prediction. It is particularly useful for tasks such as drug-disease interactions, prescription suitability evaluations, and patient profiling, supporting medical professionals in decision-making.

The Adaptive Hierarchical Transformer with Memory (AHTM) [137] combines a hierarchical Transformer architecture with model compression techniques to effectively generate KG embeddings for large-scale medical datasets. This method accounts for the high-dimensional nature of medical data and learns key patterns and relationships while reducing memory usage and computational costs. For example, it processes extensive medical datasets, including patient records, pharmaceutical databases, and genomic information to produce embeddings that maintain predictive accuracy while enhancing computational efficiency.

Ref. [138] explored the potential of integrating large language models (LLMs) with graph data to expand and utilize KGs. This survey proposes methods for combining LLMs’ language understanding and generation capabilities with graph-based learning to effectively learn new entities and relationships. By merging the structural information of graph data with the rich textual representations of LLMs, this approach achieves strong performance in various tasks, such as KG-based question answering, summarization, and relation prediction.

The Unsupervised Semantics and Syntax into heterogeneous Graphs (USS-Graph) [139] adopts unsupervised learning techniques to efficiently learn from graph data. This model simultaneously captures the structural features and syntactic semantics of a graph and models node and edge interactions with precision. By automatically learning significant patterns and relationships in the graph without explicit labels, it demonstrates high scalability, even with large datasets. This approach is applicable to graph-based text classification, relation extraction, and other similar tasks.

3.4.3. Hyperbolic Space, IRL, and Other Extensions

Traditional Transformers often assume Euclidean space; however, many studies suggest that hyperbolic space (curvature < 0) is better suited for hierarchical KGs (e.g., tree-structured or layered graphs). For instance, HyGGE [140] performs graph attention in a hyperbolic space to capture complex neighbor structures and relationship patterns more effectively, while DeER [141] adopts a trainable curvature mechanism that automatically adjusts geometric properties to fit the data. Similarly, RpHGNN [142] refines embeddings in heterogeneous graphs via an iterative “propagation-update” cycle, thus reducing information loss and preserving hidden relationships across multi-type nodes. Each of these methods demonstrates how hyperbolic embeddings can offer more nuanced representations than standard Euclidean approaches, particularly for KGs with strongly hierarchical features.

Beyond the choice of embedding space, AInvR [143] integrates IRL (Inverse Reinforcement Learning) with a Transformer-based inference module to automatically learn reward functions—thereby tackling the sparse reward issues frequently seen in multi-hop reasoning tasks. This IRL-based approach can also be combined with meta-learning strategies (discussed in Section 3.5.4) to further address data sparsity or dynamically adapt to new reasoning paths. Meanwhile, ref. [122] models user-item interactions for sequential recommendation tasks, highlighting how Transformers handle temporal or sequential dependency in KGs, and ref. [129] processes object-region relationships within video data to enhance interpretability in Video QA. Furthermore, HG-SCM [144] adopts a structural causal model perspective to identify events and their causes in a human-like manner, emphasizing causal inference from graph data. Lastly, ref. [145] proposed a graph-augmented MLP-based parallel learning model that employs quantization and parallelized processes—allowing real-time training and inference for massive graph datasets.

3.5. Additional Learning Methods

3.5.1. Self-Supervised and Contrastive Learning

Techniques for self-supervised learning of embeddings by constructing pairs of nodes or subgraphs within graphs are emerging [98]. Implicit GCL (iGCL) constructs augmentations in the latent space, avoiding excessive random edge deletion/addition to balance structural preservation and training stability. Fine-Grained GCL (FSGCL) [146] enhances representation learning by contrasting meanings across graph motifs and meta-paths. Dynamic attention networks adaptively focus on different parts of the graph based on task requirements, helping to better distinguish between entities and relationships [147]. This approach emphasizes context-specific features, enabling models to better understand the significance of entities and their relationships.

TCKGE [148] combines contrastive learning with Transformers to enhance embedding performance, while Clustering Enhanced Multiplex Graph Contrastive Representation Learning [149] improves clustering performance across multiple graphs using contrastive learning. Graph Contrastive Learning with Implicit Augmentations [150] proposes contrastive learning leveraging implicit augmentations, and [95] improves knowledge graph completion performance by separating relational GNNs through contrastive learning. Graph Contrastive Learning with Personalized Augmentation [133] introduced contrastive learning with personalized augmentations, and LLM-TIKG [41] proposed the generation of threat intelligence knowledge graphs based on large language models using contrastive learning. In addition, one method [150] shows that self-attention mechanisms can generate low-dimensional knowledge graph embeddings by dynamically adjusting inter-node interactions, thereby enhancing the link prediction performance. Ref. [151] introduced a multi-level graph knowledge contrastive learning framework that effectively integrates both local and global graph information for superior representation learning. Ref. [152] proposed a method that leverages self-attention mechanisms to generate low-dimensional embeddings, thereby dynamically adjusting inter-node interactions and significantly enhancing link prediction performance.

3.5.2. Generative Adversarial Network (GAN)-Based Learning

In KGE models, generating negative triples by random replacement is often too easy, which allows the model to quickly distinguish them. To address this, GANs are used to generate plausible negative triples, while a discriminator simultaneously classifies real/fake triples and learns embeddings [153]. After training with Generative Adversarial Graph Embedding, the generator effectively restores the distribution of the training data, and the discriminator acquires expressive capabilities for triple classification or link prediction [121]. Moreover, one study [154] proposed a GAN-based framework that effectively models complex relational patterns—such as one-to-many and many-to-one—thus overcoming the limitations of traditional single-vector representations.

3.5.3. Integrating Supervised and Reinforcement Learning

Traditional supervised learning excels at learning from labeled triples and node annotations, ensuring stable and reliable embeddings. However, supervised methods alone may miss latent or less frequent relationships, especially in sparse regions. Reinforcement learning (RL) addresses this limitation by dynamically exploring the graph to discover additional link candidates and multi-hop paths. By integrating both approaches—as exemplified in RD-MPNN [122]—the model benefits from the strengths of each method. Specifically, supervised learning provides a solid foundation by reliably learning known relationships, while RL complements this by continuously exploring and validating new connections in underrepresented areas. This hybrid strategy leads to higher-quality embeddings that are both robust and comprehensive.

In recommendation systems, for example, KGRec-RNS [155] leverages RL to model user activity as a Markov Decision Process. This approach identifies “meaningful” negative samples that are more informative than random negatives, thereby enhancing the learning efficiency and interpretability. Additionally, in multi-hop reasoning tasks, using Inverse Reinforcement Learning (IRL) [122] to learn reward parameters from real data allows the model to better prioritize correct paths and mitigate over-rewarding of incorrect routes. The combination of supervised learning and RL thus achieves a balanced trade-off between stability and dynamic exploration, resulting in improved overall performance.

3.5.4. Hyperbolic and Geometric Learning, and Meta-Learning

Building on the hyperbolic concepts introduced in Section 3.4.3, more advanced hyperbolic and non-Euclidean geometric learning methods further exploit the properties of curved spaces to model hierarchical and complex relationships in KGs. For example, one approach [156] leverages Poincaré geometry to enhance link prediction by capturing intricate interactions between entities and relations, while another [157] proposes a deep hyperbolic convolutional model that expands the expressive power of KG embeddings. CoPEs (Composition-based Poincaré embeddings) [158] similarly use hyperbolic geometry to preserve distance relationships in large-scale or deeply hierarchical graphs, improving generalization and interpretability [159]—a critical advantage when dealing with real-world KGs that often contain multi-level hierarchies.

Meanwhile, meta-learning techniques address the challenges of data sparsity and frequent domain shifts by focusing on both the global graph structure and local few-shot scenarios [160]. Approaches such as MAML [161] and Prototypical Networks [159] enable learning from minimal examples [162], offering robust KG completion in environments with limited or intermittently available data. Building on these fundamentals, ref. [163] introduced the selective transfer of KG embeddings for few-shot relation prediction, while ref. [164] applied multi-granularity meta-learning to tackle long-tail classification—a scenario in which many entities or relations appear infrequently. Furthermore, ref. [165] integrated dynamic prompt learning with meta-learning for multi-label disease diagnosis, swiftly adapting to newly introduced diseases and label structures by modeling disease-symptom relationships within a KG. This approach ensures high diagnostic accuracy in data-sparse conditions and is extendable to personalized diagnostic systems, underscoring the potential synergy between hyperbolic embedding methods and meta-learning for real-world medical applications.

3.6. Final Summary

KG learning encompasses a broad range of methodologies that enable effective link prediction, relational inference, and structured data analysis. Foundational approaches, such as TransE-based embeddings [79,80,81,82,83,84,85,86,87], efficiently represent entities and relations, while GNN-based methods [88,89,90,91,92,93,94,95,96,97,98,99,100,101,102,103,104,105,106,107,108,109,110,111] capture complex, heterogeneous interactions—supporting tasks like node classification and link prediction. Dynamic and task-specific strategies leverage temporal modeling and meta-learning to adapt to evolving graph structures and further enhance recommendation systems and reinforcement learning [96,112,113,114,115,116,117,118,119,120,121,122,123,124,125,126,127,128]. Transformer-based approaches—notably those incorporating pretrained text models [122]—integrate textual and structural information via self-attention [129,130,131,132,133,134] and improve relational consistency by fusing topology with logical rules [135,136,137,138,139]. Finally, additional learning methods—including self-supervised/contrastive learning, GAN-based negative sampling, and supervised plus reinforcement learning hybrids—address challenges such as few-shot and long-tail learning [41,95,98,121,122,133,146,147,148,149,150,151,152,153,154,155,156,157,158,159,160,161,162,163,164,165]. Together, these diverse strategies form a comprehensive framework that drives significant advancements in KG learning in various domains.

4. KGs Evaluation

Intrinsic Evaluation (Section 4.1), Extrinsic Evaluation (Section 4.2), Qualitative Evaluation (Section 4.3), and Dataset and Domain-Specific Evaluation (Section 4.4). Overall, the number of publications peaks in 2023 before declining in 2024, but each sub-section follows its own trajectory.

Steady or Growing Areas Intrinsic Evaluation (Section 4.1) gradually increases and stabilizes over time. This steady rise may be attributed to the ongoing need for robust internal metrics—such as accuracy, consistency, and completeness—that allow researchers to benchmark KG models in controlled settings.

Peaks and Surges in Specific Years Extrinsic Evaluation (Section 4.2) and Dataset and Domain-Specific Evaluation (Section 4.4) show a marked surge in 2023, possibly reflecting a heightened focus on real-world applications and specialized datasets during that period. As industries and academic communities sought more tangible results and domain-aligned benchmarks, publications in these areas spiked to address the practical constraints.

Fluctuations and Declines Qualitative Evaluation (Section 4.3) reaches its highest point in 2023, then experiences a decline in 2024. This decline may indicate a shift in research priorities toward more standardized or quantifiable metrics or the saturation of existing qualitative methods. Over time, as certain qualitative approaches mature, interest may wane in favor of new, data-intensive evaluation techniques.

Overall, these trends underscore how the KG Evaluation landscape evolves in tandem with methodological advances and the practical demands of specific domains. While some approaches—like intrinsic metrics—remain consistently relevant, others experience spikes or dips as researchers adapt to changing expectations for rigorous, real-world performance and domain-driven requirements.

4.1. Intrinsic Evaluation

4.1.1. What Is an Intrinsic Evaluation?

Intrinsic evaluation offers a suite of quantitative metrics that assess the quality and performance of the outputs generated by a knowledge graph model, and each metric is particularly well-suited to different tasks and application scenarios. For example, classification metrics such as Precision, Recall, and F1-Score are advantageous when evaluating discrete prediction tasks—like determining whether a predicted RDF triple correctly represents the relationship between entities. In contexts where false positives are especially detrimental, a high Precision is critical, while Recall becomes paramount when it is crucial to capture all relevant relations. The F1-Score then serves as a balanced indicator when both types of errors are equally consequential.

In contrast, rank-based metrics, such as Mean Reciprocal Rank (MRR) and Hits@K, are more appropriate for tasks involving link prediction or information retrieval, where the model generates a ranked list of candidate relations. MRR is beneficial because it captures the average rank of the first correct answer, thereby favoring models that bring correct candidates to the top, while Hits@K focuses on the proportion of cases in which the correct answer appears within the top K predictions—making these metrics particularly useful in systems that can operate effectively on a shortlist of candidates.

For continuous numerical prediction tasks, regression metrics like Root Mean Squared Error (RMSE), Mean Absolute Error (MAE), and Mean Absolute Percentage Error (MAPE) are used. RMSE is especially sensitive to large errors, which is advantageous in applications where significant deviations can have a profound impact, whereas MAE provides a straightforward average error measure that is easier to interpret. MAPE, which expresses errors as percentages relative to true values, is particularly useful in scenarios in which the relative scale of error is more informative than the absolute difference.

Additionally, when assessing the quality of the learned embeddings, methods such as the Pearson correlation coefficient can quantify the linear relationship between embedding similarities and ground-truth semantic relationships. In some cases, an F1-Score is also computed for clustering or classification tasks performed on embedding spaces, providing further insight into the expressive power of the representations.

Finally, evaluating algorithm optimization and efficiency through measures like average effectiveness, standard deviation, execution time, and memory usage is essential when scaling models to large datasets or deploying them in real-time applications. These metrics not only reflect the model’s internal optimization and stability across various runs but also determine its practical viability in resource-constrained environments.

By carefully selecting and interpreting these intrinsic evaluation metrics in accordance with the specific requirements and constraints of the knowledge graph application, researchers can gain a comprehensive understanding of the model performance and make informed decisions regarding further optimization and deployment strategies. Through

Table 2. Comprehensive overview of intrinsic evaluation metrics and their mathematical definitions.

Name	Equation	Definition
Precision	$\mathrm{P}\mathrm{r}\mathrm{e}\mathrm{c}\mathrm{i}\mathrm{s}\mathrm{i}\mathrm{o}\mathrm{n}={\displaystyle \frac{\mathrm{T}\mathrm{P}}{\mathrm{T}\mathrm{P}+\mathrm{F}\mathrm{P}}}$	TP: True Positives FP: False Positives
Recall	$\mathrm{R}\mathrm{e}\mathrm{c}\mathrm{a}\mathrm{l}\mathrm{l}={\displaystyle \frac{\mathrm{T}\mathrm{P}}{\mathrm{T}\mathrm{P}+\mathrm{F}\mathrm{N}}}$	TP: True Positives FN: False Negatives
F1 score	$\mathrm{F}1=2\times {\displaystyle \frac{\mathrm{P}\mathrm{r}\mathrm{e}\mathrm{c}\mathrm{i}\mathrm{s}\mathrm{i}\mathrm{o}\mathrm{n}\times \mathrm{R}\mathrm{e}\mathrm{c}\mathrm{a}\mathrm{l}\mathrm{l}}{\mathrm{P}\mathrm{r}\mathrm{e}\mathrm{c}\mathrm{i}\mathrm{s}\mathrm{i}\mathrm{o}\mathrm{n}+\mathrm{R}\mathrm{e}\mathrm{c}\mathrm{a}\mathrm{l}\mathrm{l}}}$	Precision, Recall
MRR (Mean Reciprocal Rank)	$\mathrm{M}\mathrm{R}\mathrm{R}={\displaystyle \frac{1}{\mathrm{N}}}{\displaystyle \sum _{\mathrm{i}=1}^{\mathrm{N}}}{\displaystyle \frac{1}{{\mathrm{r}\mathrm{a}\mathrm{n}\mathrm{k}}_{\mathrm{i}}}}$	N: Total number of queries ranki: Rank position of the correct answer for the ith query
Hits@K (Hits at K)	$\mathrm{H}\mathrm{i}\mathrm{t}\mathrm{s}@\mathrm{K}={\displaystyle \frac{1}{\mathrm{N}}}{\displaystyle \sum _{\mathrm{i}=1}^{\mathrm{N}}}1\{{\mathrm{r}\mathrm{a}\mathrm{n}\mathrm{k}}_{\mathrm{i}}\le \mathrm{K}\}$	N: Total number of queries K: Rank threshold ranki: Rank of the correct answer for the ith query
RMSE (Root Mean Squared Error)	$\mathrm{R}\mathrm{M}\mathrm{S}\mathrm{E}=\sqrt{{\displaystyle \frac{1}{\mathrm{N}}}{\displaystyle \sum _{\mathrm{i}=1}^{\mathrm{N}}}{({\widehat{\mathrm{y}}}_{\mathrm{i}}-{\mathrm{y}}_{\mathrm{i}})}^2}$	N: Number of samples ŷi: Predicted value for sample i yi: Actual value for sample i
MAE (Mean Absolute Error)	$\mathrm{M}\mathrm{A}\mathrm{E}={\displaystyle \frac{1}{\mathrm{N}}}{\displaystyle \sum _{\mathrm{i}=1}^{\mathrm{N}}}\left|{\widehat{\mathrm{y}}}_{\mathrm{i}}-{\mathrm{y}}_{\mathrm{i}}\right|$	Same variables as RMSE
MAPE (Mean Absolute Percentage Error)	$\mathrm{M}\mathrm{A}\mathrm{P}\mathrm{E}={\displaystyle \frac{100\mathrm{\%}}{\mathrm{N}}}{\displaystyle \sum _{\mathrm{i}=1}^{\mathrm{N}}}\left|{\displaystyle \frac{{\widehat{\mathrm{y}}}_{\mathrm{i}}-{\mathrm{y}}_{\mathrm{i}}}{{\mathrm{y}}_{\mathrm{i}}}}\right|$	N: Number of samples ŷi: Predicted value yi: Actual value
PCC (Pearson Correlation Coefficient)	$\mathrm{r}={\displaystyle \frac{{\sum }_{\mathrm{i}=1}^{\mathrm{N}}({\mathrm{x}}_{\mathrm{i}}-\overline{\mathrm{x}})({\mathrm{y}}_{\mathrm{i}}-\overline{\mathrm{y}})}{\sqrt{{\sum }_{\mathrm{i}=1}^{\mathrm{N}}{({\mathrm{x}}_{\mathrm{i}}-\overline{\mathrm{x}})}^2}\sqrt{{\sum }_{\mathrm{i}=1}^{\mathrm{N}}{({\mathrm{y}}_{\mathrm{i}}-\overline{\mathrm{y}})}^2}}}$	xi, yi: Paired values from two datasets $\overline{\mathrm{x}}$, $\overline{\mathrm{y}}$: Mean of x and y values, respectively
Average Effectiveness	$Average Effectiveness={\displaystyle \frac{1}{N}}{\displaystyle \sum _{i=1}^N}{E}_i$	N: Total number of runs Ei: Effectiveness measure for the ith run
SD (Standard Deviation)	$\sigma =\sqrt{{\displaystyle \frac{1}{N}}{\displaystyle \sum _{i=1}^N}{(x_i-\overline{x})}^2}$	N: Total number of measurements $\overline{\mathrm{x}}$: Mean of the measurements xi: Individual measurement value
Mean Execution Time	$Mean Execution Time={\displaystyle \frac{1}{N}}{\displaystyle \sum _{i=1}^N}{t}_i$	N: Total number of runs ti: Execution time for the ith run
Execution Time Standard Deviation	${\sigma }_t=\sqrt{{\displaystyle \frac{1}{N}}{\displaystyle \sum _{i=1}^N}{(t_i-\overline{t})}^2}$	ti: Individual execution time $\overline{\mathrm{t}}$: Mean execution time
Mean Memory Usage	$Mean Memory Usage=\sqrt{{\displaystyle \frac{1}{N}}{\displaystyle \sum _{i=1}^N}{m}_i}$	N: Total number of runs mi: Memory usage for the ith run
Memory Usage Standard Deviation	${\sigma }_m=\sqrt{{\displaystyle \frac{1}{N}}{\displaystyle \sum _{i=1}^N}{(m_i-\overline{m})}^2}$	mi: Individual memory usage measurement $\overline{\mathrm{m}}$: Mean memory usage

, we can see how the evaluation metrics are assessed.

4.1.2. Intrinsic Evaluation: Metrics, Methods, and Case Studies

In a study based on the public standard for public transportation data, researchers used the GTFS Madrid benchmark to precisely measure how well a CSV-based RML mapping tool produced results that matched public transportation data and to evaluate the impact of data types (e.g., differences in handling xsd:long versus xsd:integer) on mapping errors and attribute matching success rates [166]. Additionally, a study was conducted that rigorously evaluated the performance differences among link prediction techniques using various knowledge graph datasets under the same preprocessing conditions and demonstrated the statistical significance between methods using the Wilcoxon test and the Kolmogorov-Smirnov test [167]. In the Chinese domain, intrinsic performance was systematically verified by quantitatively evaluating the embedding representation and the effect of negative samples in an entity linking model using the FB15K237, WN18RR, and SCWS datasets, measured using the Pearson correlation coefficient and F1-Score [168].

Furthermore, a study evaluating the effect of mutation operators introduced several metrics—including average effectiveness, average n-step effectiveness, standard deviation of effectiveness, and execution time—to quantify the algorithm optimization performance [169]. Another research compared the performance of entity-relation extraction using the New York Times, WebNLG, and ADE datasets with Micro-F1 and Macro-F1 metrics [66]. In an air quality prediction task, the KnowAir and UrbanAir (North) datasets were used to quantitatively evaluate prediction errors with RMSE, MAE, and MAPE [170], and a study on patient similarity search based on medical data reported excellent performance in terms of Accuracy and F1-Score using the MIMIC-III and MIMIC-IV datasets [171]. Additionally, research utilizing datasets that include spatiotemporal information employed Mean Reciprocal Rank, Hit@1, and Hit@10 to demonstrate performance improvements exceeding three times that of TransE [172], and another study analyzed the effect of text embeddings and graph neural network layer tuning in multilingual entity alignment using the DBP15K dataset [173]. Studies evaluating the effect of temporal logical rules using the ICEWS and GDELT datasets [174,175] have provided valuable insights; furthermore, an iterative approach for applying temporal logical rules to improve reasoning performance was proposed in [176]. Finally, research that quantitatively evaluated performance in fields such as legal judgment prediction, link prediction, KG generation, and railway defect detection in the domains of law, biomedicine, agriculture, and railway fault detection further exemplifies intrinsic evaluation [177,178,179,180].

4.2. Extrinsic Evaluation

4.2.1. What Is an Extrinsic Evaluation?

Extrinsic evaluation goes beyond assessing a model’s internal consistency or predictive accuracy by quantifying its practical value in real-world application environments. This evaluation framework examines how effectively a knowledge graph (KG) model performs in operational settings, considering factors such as processing speed, usability of the outputs, and performance improvements over existing systems when integrated into tasks like question answering, recommendation, ranking, legal judgment prediction, medical diagnosis, code mapping, and more.

For example, in the public transportation domain, extrinsic evaluation may involve comparing a CSV-based mapping tool with systems like Morph-KGC and SPARQL Anything using GTFS data, to verify whether they produce identical results. In question-answering tasks, the integration of large-scale language models with KGs has been shown to enhance metrics such as Mean Average Precision and Mean Reciprocal Rank, demonstrating the model’s ability to rank and retrieve correct answers effectively.

In medical applications, studies leveraging datasets such as MIMIC-III and MIMIC-IV have shown that patient similarity search models can achieve high accuracy and F1-Scores, while in multilingual entity alignment, improvements are quantified using datasets like DBP15K. Similarly, in the legal domain, judgment prediction models based on datasets like ECHR and CAIL2018 have attained superior Accuracy and AUC-ROC compared to traditional legal AI systems. Other studies in biomedical, agricultural, and railway defect detection applications have demonstrated tangible performance benefits—such as improved link prediction accuracy and reduced query response times—when KG models are applied in real-world scenarios.

Overall, extrinsic evaluation provides an objective measure of how well a KG model performs in practical settings, highlighting its benefits in operational environments and its advantages over conventional methods through measurable improvements in various application-specific metrics.

Knowledge graphs constructed or refined using LLM-based systems can exhibit hallucinated relationships that degrade the overall performance. For instance, a clinical decision-support application may rely on KG-based inferences only to discover that certain predictive links between symptoms and diagnoses are fabricated by the generative model. Extrinsic evaluation thus benefits from dedicated error-detection strategies—such as outlier analysis or correlation checks with trusted medical databases—to ensure that the discovered relationships are both accurate and clinically valid. By incorporating such validation layers, practitioners can reduce erroneous recommendations and maintain higher reliability in domains where trust is essential.

4.2.2. Extrinsic Evaluation Across Diverse Knowledge Graph Applications

In the field of public transportation, a quantitative analysis was conducted comparing a CSV-based mapping tool with Morph-KGC and SPARQL Anything, using GTFS public transportation data to determine whether the same results were generated [166]. Another study constructed an application pipeline that evaluated various link prediction techniques under the same conditions to analyze their impact on a real-world knowledge graph evaluation workflow [167]. In the question-answering task, the integration of large-scale language models with knowledge graphs was shown to improve Mean Average Precision and Mean Reciprocal Rank using both Chinese and English QA datasets (NLPCC2017, cMedQA, TREC-QA, and WikiQA) [181].

In the medical application field, the use of the MIMIC-III and MIMIC-IV datasets demonstrated that a patient similarity search model achieved high accuracy and F1-score [171], while in multilingual entity alignment, performance improvements across languages were quantitatively verified using the DBP15K dataset [173]. In the legal domain, a judgment prediction model based on the ECHR and CAIL2018 datasets achieved superior Accuracy and Area Under the Curve for Receiver Operating Characteristic (AUC-ROC) compared to conventional legal AI models [177]. In the biomedical field, studies using the KEGG50k, Hetionet, SuppKG, and ADInt datasets have demonstrated improved link prediction performance [178]. In agricultural applications, research based on Bangladesh agricultural data (BBS) reported that after KG generation, OLAP queries and business intelligence analyses experienced significantly reduced query response times [179], and in railway defect detection, evaluations using the NEU RSDDS-AUG dataset quantitatively verified the benefits of external applications [180]. Additionally, evaluations through RDF triple store benchmarks [182] and open-world KG completion assessment via extended pretrained language models [134], as well as multilingual KG alignment performance evaluation [183] and studies that quantitatively analyzed the effects of relation-based graph attention and mutual information maximization techniques [184,185,186] are presented as extensive applications of extrinsic evaluation.

Other applications, such as software bug triaging, clinical decision support, mobile app recommendation, literature-based research hypothesis generation, cybersecurity, federated learning in healthcare, link prediction under few-shot conditions, and storage space savings through normalization techniques, have also been evaluated.

4.3. Qualitative Evaluation

Qualitative evaluation is a method that deeply analyzes the internal decision-making process and inference mechanisms of a knowledge graph model through case studies, capturing aspects such as interpretability and transparency, which are difficult to assess through numerical evaluation. Through such evaluations, one can understand how the model processes information and makes decisions and identify the causes of errors. This, in turn, provides important insights into model improvement and optimization strategies.

For example, in a study that evaluated internal optimization performance, the validity and completeness of the mutation operator were observed to affect the model performance. This study qualitatively analyzed the efficiency trade-offs to explain how each component of the algorithm contributes to the overall performance [169]. In another study, by incorporating a visual attention mechanism, the evaluation focused on which parts of the input sentence the model concentrated on during the entity-relation extraction process, thereby greatly enhancing the intuitiveness of the result interpretation [66]. A study that qualitatively interpreted the phenomenon in which a Sub-Entity Embedding model consistently outperformed existing methods contributed to a better understanding of the subtle performance differences between models [172]. Additionally, research that strengthened the transparency of the model’s decision-making process through a balance between text embeddings and internal information provided a qualitative analysis of how the model reached its decisions [173].

A study that deeply analyzed the use of temporal logical rules and reliability through cases using the ICEWS and GDELT datasets offered qualitative insights into the inference process of the model and the impact of errors [175]. Another study, through a multi-stage inference technique, qualitatively evaluated the interpretability of legal judgment prediction, clearly revealing the decision-making process in practical applications [177]. In the domain of biomedical relationship extraction, research that analyzed the effects of integrating large-scale language model-based text embeddings with domain knowledge on a case-by-case basis provided intuitive insights into the model’s internal workings [178]. Furthermore, a study evaluating the readability of community structures and node layouts in graph exploration and visualization through user studies and case analyses qualitatively demonstrated the model’s interpretability in applied settings [181].

In the e-commerce field, the role of a language model that exhibits high relationship labeling performance, even with limited data, was qualitatively analyzed [182]. In a question-answering task, a study qualitatively discussed the strengths and limitations of the system based on the inclusion of SPARQL modifiers [183]. Additionally, research evaluating the results of EHR data integration among medical institutions alongside domain experts [184] and a study analyzing the interpretability of information extraction after generating a large-scale scientific KG [27] have contributed to an in-depth understanding of the model’s practical usability. Moreover, studies that qualitatively interpreted the bias of commonsense knowledge models [186], discussed the challenges of classifying sensitive data with domain expert evaluations [187], and assessed the stability of dynamic KG refinement through case studies [188] are important examples of qualitative evaluation. Finally, diverse qualitative analyses have been conducted in various studies on topics such as clinical data integration, semantic relevance evaluation, enhanced path inference for medical diagnosis, AMR parsing, complex entity resolution, fall risk detection, hydraulic engineering, community exploration, cybersecurity, reverse-engineering question generation, and IoT applications.

4.4. Dataset and Domain-Specific Evaluation

In studies evaluating industry and multilingual scenarios, one research work utilized the GTFS Madrid benchmark to assess the consistency and coherence of mapping tools based on the public standard for transit data [166]. Another study compared various link prediction techniques across different knowledge graph datasets under identical preprocessing conditions [167]. Some studies have evaluated entity linking performance in the Chinese domain using the FB15K237, WN18RR, and SCWS datasets [168], as well as a study that verified model performance in multilingual knowledge graph alignment tasks—considering the language-specific characteristics—based on the DBP15K series datasets [173]. Evaluations using large-scale healthcare and medical datasets, such as Yago (ST), DBpedia (ST), and Wikidata (ST), have also been conducted [174], along with research that quantitatively assessed storage space and complex query performance in industrial settings using various RDF triple store benchmarks [189]. Lastly, a study evaluating the interpretability and alignment performance of multilingual KGs also falls under this category [190].

In the domains of medicine, law, biomedicine, agriculture, and air quality prediction, one study used the KnowAir and UrbanAir (North) datasets for an air quality prediction task to evaluate the prediction accuracy and computational efficiency [170]. In the context of patient similarity search based on medical data, research employing the MIMIC-III and MIMIC-IV datasets evaluated model performance according to domain-specific characteristics [171]. Moreover, studies evaluating link prediction and completion performance in spatiotemporal knowledge graphs using the Yago (ST), DBpedia (ST), and Wikidata (ST) datasets [172], as well as research evaluating the application performance of judgment prediction models in the legal domain using the ECHR and CAIL2018 datasets [177] have been conducted. In biomedicine, link prediction evaluations using datasets such as KEGG50k, Hetionet, SuppKG, and ADInt have been carried out [178]. Additionally, research based on Bangladesh agricultural data (BBS) evaluated the completeness, currency, and OLAP compatibility of fisheries, forestry, and agriculture data after KG generation [179], and a study using the NEU RSDDS-AUG dataset analyzed the extrinsic application effects in the railway defect detection field [180]. Finally, a study on the open-world KG completion task utilizing various KG benchmark datasets [134] also falls into this category.

In the fields of science, commonsense, clinical, natural language processing, entity resolution, human activity monitoring, hydraulic engineering, community exploration, cybersecurity, reverse-engineering question generation, and IoT applications, research using the ICEWS and GDELT datasets evaluated the effects of temporal events and logical rules on event data [175]. Additionally, studies on graph exploration and visualization using the Les Misérables and Graph of Science datasets [181], as well as research evaluating relationship-labeling performance in product knowledge graphs using the Electronics and Instacart datasets in the e-commerce domain [182] have been conducted. There is research quantitatively evaluating the effect of SPARQL modifier handling in question-answering tasks using the MQALD dataset [183] and studies assessing data integration and code mapping performance between medical institutions using real EHR data and medical codes (RxNorm, VA codes) [184].

Other examples include a study that evaluated scientific KG generation and information extraction performance using a large-scale computer science paper dataset (6.7 million papers, 10 million entities, and 41 million triples) [185], and research that assessed the bias in commonsense knowledge models using the ConceptNet and Wikidata-CS datasets [186]. There has also been work on data privacy evaluation using DPV-based sensitive personal data [187]. In addition, studies using datasets such as FB15K, NELL, WN18, and YAGO3-10 evaluated the performance of dynamic KG refinement and continuous updates [188], and research based on cerebral aneurysm and COVID-19 datasets evaluated clinical data integration and evidence-based medicine support performance [191]. Evaluations of semantic relevance have been carried out using DBpedia and 14 benchmark datasets [192], and research utilizing the MIMIC-III, MIMIC-IV, and eICU datasets evaluated the interpretability of medical diagnosis and disease prediction through path inference [193]. There are also evaluations in natural language processing using the AMR2.0 and AMR3.0 datasets [194], studies on complex entity resolution tasks using seven real-world industrial and domain-specific datasets [195], and evaluations of fall risk detection and human activity monitoring using everyday activity and event data [196]. Research based on a dedicated hydraulic engineering dataset to assess KG embedding and link prediction performance in the field of hydraulic engineering has been conducted [197]. Other examples include studies on edge attribute-based community exploration using standard graph datasets [198], evaluations of KG construction and quality in cybersecurity using datasets such as FB15K [199], assessments of reverse-engineering question generation, question quality, and filtering performance using a Wikidata-based KG [200], and research on collision detection and context recognition in the IoT field using trigger-action programming-related datasets [201]. In the question-answering field, there is research evaluating extrinsic QA performance using the NLPCC2017, cMedQA, TREC-QA, and WikiQA datasets [202], as well as studies that quantitatively evaluated the effects of relation-based graph attention and mutual information maximization techniques using the FB15K-237 dataset [203,204,205].

Finally, in Section 4.4. Dataset and Domain-Specific Evaluation, the Evaluation Metrics/Considerations are summarized in

Table 3. Datasets used in knowledge graphs across various domains.

Dataset Name	Domain/Field	Main Usage	Key Feature
GTS Madrid benchmark	Public Transportation	Comparing mapping tools, data consistency evaluation	Based on the GTFS standard for public transit data
FB15K, FB15K-237	Knowledge Graph Embedding (KGE)	Link prediction, embedding evaluation	FB15K-237 refines FB15K by removing redundant/inverse relations
WN18, WN18RR	Natural Language Embedding	Link prediction, embedding evaluation	Based on WordNet; WN18RR removes duplicate relations
SCWS (Stanford Contextual Word Similarities)	Natural Language Processing	Word/context similarity evaluation	Used for assessing context-based word embeddings
New York Times, WebNLG, ADE	Text Analysis Relation Extraction	Entity-relation extraction	Data from news articles, web content, and medical reports
KnowAir, UrbanAir (North)	Environmental (Air Quality)	Air quality prediction	Evaluated using metrics such as RMSE, MAE, and MAPE
MIMIC-III, MIMIC-IV, eICU	Healthcare (ICU Records)	Patient similarity search, disease prediction, clinical support	Large-scale clinical datasets based on electronic health records (EHR)
ICEWS, GDELT	Time-series Events	Event prediction, temporal inference, logical rule evaluation	Contains spatiotemporal data on political/social events
DBP15K	Multilingual Knowledge Graph	Entity alignment across languages	Derived from DBpedia for cross-lingual entity mapping
ECHR	Legal	Legal judgment prediction, legal AI evaluation	ECHR from the European Court of Human Right;
CAIL2018	Legal	Legal judgment prediction, legal AI evaluation	CAIL2018 is a Chinese legal dataset
KEGG50k, Hetionet, SuppKG, ADInt	Biomedical	Drug-disease link prediction, relation extraction	Large-scale biomedical data for analyzing biological pathways
Bangladesh Agricultural Data (BBS)	Agriculture	KG generation, OLAP queries, business intelligence analysis	Includes fisheries, forestry, and agriculture data from Bangladesh
NEU RSDDS-AUG	Railway Defect Detection	Railway defect detection and analysis	Uses railway images/signals for defect classification
Yago (ST), DBpedia (ST), Wikidata (ST)	Knowledge Graph (Spatiotemporal)	Spatial/temporal KG analysis, link prediction	Extended versions of Yago, DBpedia, and Wikidata with spatiotemporal information
RDF triple store benchmarks	Semantic Web Infrastructure	Evaluating RDF store performance (storage, querying, etc.)	Focuses on SPARQL query performance and scalability
NLPCC2017, cMedQA, TREC-QA, WikiQA	Question Answering	QA system evaluation in Chinese and English	Uses metrics like Mean Average Precision (MAP) and Mean Reciprocal Rank (MRR)
RxNorm, VA codes	Healthcare (Standard Codes)	Mapping drug/diagnosis codes, data integration between hospitals	Standard coding systems used in the U.S. healthcare domain
ConceptNet, Wikidata-CS	Commonsense Knowledge Graph	Bias analysis, commonsense reasoning	Rich set of entities and relations for everyday knowledge
DPV-based sensitive personal data	Data Privacy	Identifying sensitive information, data classification	Based on Data Privacy Vocabulary (DPV)
NELL, YAGO3-10	Knowledge Graph (General)	Link prediction, continuous learning	NELL is an online learning project; YAGO3-10 is a large-scale knowledge graph
Les Misérables, Graph of Science	Graph Exploration Visualization	Community detection, graph layout analysis	Based on a literary work (Les Misérables) and a citation network (Graph of Science)
Electronics, Instacart	E-commerce	Product relationship extraction, recommendation systems	Includes product categories, purchase histories, etc
MQALD	SPARQL-based QA	Evaluating SPARQL query processing and modifier effects	Designed for testing natural language to SPARQL conversion
AMR2.0, AMR3.0	Natural Language Processing	AMR parsing, structured semantic analysis	Represents sentence meaning using graph-based structures

and

Table 4. Summary of evaluation metrics and considerations for various datasets in knowledge graph applications.

Dataset Name	Evaluation Metrics/Considerations
GTS Madrid benchmark	Evaluate mapping accuracy, data consistency, processing speed, and scalability based on GTFS-standard transit data.
FB15K, FB15K-237	Use ranking metrics (MRR, Hits@K) and classification metrics (Precision, Recall, F1-Score). FB15K-237 refines FB15K by removing redundant/inverse relations.
WN18, WN18RR	Similar ranking and classification metrics as FB15K, with WN18RR removing duplicate relations for more reliable link prediction.
SCWS (Stanford Contextual Word Similarities)	Apply Pearson correlation coefficient and similarity-based F1-Score to evaluate context-based word embeddings.
New York Times, WebNLG, ADE	Use Precision, Recall, and F1-Score to assess relation extraction performance from diverse text sources.
KnowAir, UrbanAir (North)	Utilize regression metrics (RMSE, MAE, MAPE) and efficiency measures (e.g., execution time) for air quality prediction.
MIMIC-III, MIMIC-IV, eICU	Evaluate patient similarity search and disease prediction using Accuracy, F1-Score, and AUC-ROC, emphasizing clinical reliability.
ICEWS, GDELT	Combine regression metrics (RMSE, MAE) with ranking metrics (MRR, Hits@K) to assess spatiotemporal event prediction and temporal inference.
DBP15K	Evaluate cross-lingual entity alignment using Precision, Recall, and F1-Score to capture multilingual performance nuances.
ECHR, CAIL2018	Use Accuracy, AUC-ROC, and F1-Score to assess legal judgment prediction, ensuring reliable performance in legal applications.
KEGG50k, Hetionet, SuppKG, ADInt	Apply MRR, Hits@K, Precision, Recall, and F1-Score for biomedical link prediction and relation extraction in large-scale biological datasets.
Bangladesh Agricultural Data (BBS)	Measure query response time, throughput, and data completeness to evaluate OLAP queries and business intelligence analysis in agricultural contexts.
NEU RSDDS-AUG	Use Accuracy, Precision, Recall, F1-Score, and computational efficiency metrics (execution time, memory usage) to evaluate railway defect detection performance.
Yago (ST), DBpedia (ST), Wikidata (ST)	Employ regression metrics (RMSE, MAE) and ranking metrics (MRR, Hits@K) to assess spatiotemporal KG analysis and link prediction accuracy.
RDF triple store benchmarks	Focus on SPARQL query response time, throughput, and scalability to evaluate RDF store performance.
NLPCC2017, cMedQA, TREC-QA, WikiQA	Use MAP, MRR, and Accuracy to evaluate QA system performance in both Chinese and English.
RxNorm, VA codes	Assess mapping accuracy and use Precision and Recall to evaluate drug/diagnosis code mapping and hospital data integration.
ConceptNet, Wikidata-CS	Evaluate commonsense reasoning and bias using F1-Score and ranking-based metrics to assess the quality of everyday knowledge representations.
DPV-based sensitive personal data	Use Precision, Recall, and F1-Score to measure the performance of sensitive information classification based on the Data Privacy Vocabulary (DPV).
NELL, YAGO3-10	Apply MRR, Hits@K, Precision, Recall, and F1-Score to evaluate continuous learning and link prediction in general-purpose knowledge graphs.
Les Misérables, Graph of Science	Use clustering quality metrics and qualitative assessments to evaluate graph exploration and visualization performance.
Electronics, Instacart	Assess product relationship extraction and recommender systems using Precision, Recall, F1-Score, and recommendation ranking metrics.
MQALD	Evaluate natural language to SPARQL conversion by measuring query accuracy and response time.
AMR2.0, AMR3.0	Use parsing accuracy and F1-score to evaluate AMR parsing quality and structured semantic analysis.

.

4.5. Final Summary

Intrinsic evaluation, as addressed in studies [66,166,167,168,169,170,171,172,173,174,175,176,177,178,179,180], shows that it is a method of evaluating the internal outputs generated by a knowledge graph model—such as data processing, entity and relation inference, and spatiotemporal prediction—using quantitative metrics (e.g., precision, recall, F1-score, mean reciprocal rank, root mean square error) to assess consistency, accuracy, embedding representation, algorithm optimization, and efficiency.

Extrinsic evaluation, as demonstrated in studies [134,166,167,171,173,177,178,179,180,181,182,183,184,185,186], quantitatively proves the practical value provided by the model in real industrial data and application tasks (e.g., question-answering, recommendation, judgment prediction, medical diagnosis, and code mapping).

Qualitative evaluation, as carried out in studies [27,66,169,172,173,175,177,178,181,182,183,184,186,187,188], assesses the interpretability and transparency of the model by deeply analyzing its internal decision-making process, inference mechanisms, and usability in application environments through case studies.

Finally, Dataset and Domain-Specific Evaluations have been demonstrated through studies in the first paragraph using [166,167,168,173,174,189,190], in the second paragraph using [134,170,171,172,177,178,179,180], and in the third paragraph using [175,181,182,183,184,185,186,187,188,191,192,193,194,195,196,197,198,199,200,201,202,203,204,205]. These studies objectively evaluate the model’s performance and potential for application across various fields, including industry, multilingual contexts, healthcare, law, biomedicine, agriculture, air quality prediction, spatiotemporal analysis, science, commonsense reasoning, data privacy, dynamic knowledge graph refinement, clinical integration, natural language processing, entity resolution, human activity monitoring, hydraulic engineering, community exploration, cybersecurity, reverse-engineering question generation, and IoT applications.

5. Conclusions

This study comprehensively examines trends in knowledge graph (KG) research from 2022 to 2024, focusing on three main areas: extraction (2.x), learning (3.x), and evaluation (4.x). The findings indicate that KG research is progressing rapidly, driven by advancements in deep learning and natural language processing and the growing importance of hybrid approaches and domain-specific techniques.

First, KG extraction (2.x) showed notable fluctuations in research interest over time. In 2022, foundational studies focused on improving basic models for extracting entities and relations from textual and unstructured data, as well as accelerating large-scale data preprocessing. In 2023, attention shifted toward multimodal and domain-specific extraction (Section 2.2.2), particularly in the context of large-scale patent and technical documents. By 2024, extraction research had taken another leap, focusing on automated pipelines capable of handling complex contexts and multiple domains, resulting in a marked increase in more sophisticated methods.

In the KG Learning (3.x) domain, the focus also varied by year. In 2022, enhancements to classical embedding methods such as TransE took center stage. Moving into 2023, Graph Neural Networks (GNNs) and Transformer-based models gained traction for tasks like node classification and link prediction. Then in 2024, there was a surge in novel methods that integrated large language models (LLMs) and applied hybrid approaches combining supervised and reinforcement learning (Section 3.5.3). These approaches enabled more accurate modeling of complex relational structures, significantly improving performance in specialized fields dealing with large-scale data, such as patents and technical reports.

Lastly, KG Evaluation (4.x) peaked in 2023 before showing some decline in 2024. In 2022, intrinsic evaluation (Section 4.1) played a central role in establishing accuracy and precision metrics for early-stage models. In 2023, a spike was observed in extrinsic evaluation (Section 4.2) and Dataset/Domain-Specific Evaluation (Section 4.4), reflecting an increasing need to validate KG models for real-world applications in industry, healthcare, law, the Internet of Things (IoT), and more. While 2024 saw growing interest in interpretability and qualitative evaluation (Section 4.3), the overall research volume slightly decreased, likely due to concurrent efforts emphasizing quantitative metrics.

Reflecting these year-to-year shifts in research priorities, this study provides a holistic review of the latest methods and trends in KG extraction, learning, and evaluation. In particular, 2022 featured substantial work on improving classical embeddings and intrinsic metrics, while 2023 saw an expansion into multimodal/domain-specific extraction, GNN/Transformer-based embeddings, and extrinsic evaluation. Subsequently, 2024 brought LLM integration, supervised + reinforcement learning hybrids, renewed attention to advanced extraction methods, and greater emphasis on interpretability and qualitative metrics. These developments highlight the simultaneous pursuit of broader applications and higher performance, implying that adaptive and integrated approaches are vital as data scales and complexities increase. This trajectory suggests that knowledge graphs will continue to serve as a foundational infrastructure in diverse areas—from industry to healthcare and law—while also demanding solutions for challenges in scalability, complexity, and reliability.

6. Future Work

Despite continuous and rapid advancements in knowledge graph (KG) construction methodologies, a universally recognized standard encompassing the entire KG development pipeline remains elusive. This methodological gap arises from the complex interactions among diverse factors, including domain-specific requirements, data heterogeneity, and algorithmic complexity. Each stage—from initial data preprocessing and entity-relationship extraction to multimodal integration of diverse data types, such as text, images, and videos—necessitates tailored strategies and sophisticated technical choices, presenting researchers with significant challenges.

Future KG research should explicitly focus on several promising areas. First, developing advanced techniques for robust multimodal data integration, including the efficient extraction and accurate alignment of entities and relations across heterogeneous sources (e.g., text, images, and videos), remains critically important. Establishing standardized, scalable, and reproducible methodologies for multimodal data integration will significantly enhance the usability and applicability of KG across domains.

Second, addressing the current limitations of KG Evaluation methodologies through comprehensive and robust evaluation frameworks is essential. Future evaluations should expand beyond traditional intrinsic metrics (accuracy, precision, recall) and integrate more sophisticated metrics and benchmarks inspired by recent advancements in language model evaluation, such as F-score-based measures and metrics emphasizing interpretability, consistency, and domain-specific applicability. Such multidimensional evaluation frameworks will provide more reliable assessments of the quality and practical effectiveness of KG.

Third, establishing real-time dynamic knowledge updates and self-maintenance mechanisms represents a significant research opportunity. Many existing KGs rely primarily on static datasets, limiting their applicability in dynamic and evolving domains. Designing automated pipelines capable of continuously integrating and updating information from rapidly changing external sources (e.g., news streams, social media, and scientific publications) will ensure that KGs remain current, accurate, and highly relevant for downstream applications.

Finally, leveraging KG structures to enhance model explainability and interpretability remains a critical, yet relatively underexplored, area. Utilizing structured relational information within KGs to produce transparent, interpretable explanations of AI predictions—independent of specific model architectures—constitutes an essential step toward more trustworthy and transparent AI systems. Further research in this direction will significantly contribute to the broader acceptance and deployment of AI in sensitive or high-stakes contexts.