Advancements in Complex Knowledge Graph Question Answering: A Survey

Abstract

Complex Question Answering over Knowledge Graph (C-KGQA) seeks to solve complex questions using knowledge graphs. Currently, KGQA systems achieve great success in answering simple questions, while complex questions still present challenging issues. As a result, an increasing number of novel methods have been proposed to remedy this challenge. In this survey, we proposed two mainstream categories of methods for C-KGQA, which are divided according to their use for knowledge graph representation and construction, namely, graph metric (GM)-Based Methods and graph neural network (GNN)-based methods. Additionally, we also acknowledge the influence of ChatGPT, which has prompted further research into utilizing knowledge graphs as a knowledge source to assist in answering complex questions. We also introduced methods based on pre-trained models and knowledge graph joint reasoning. Furthermore, we have compiled research achievements from the past three years to make it easier for researchers with similar interests to obtain state-of-the-art research. Finally, we discussed the resources and evaluation methods for tackling C-KGQA tasks and summarized several research prospects in this field.

1. Introduction

A knowledge graph (KG) is a structured database for knowledge management. It describes the concepts and entities and their relationships in the real world and is used to extract knowledge in specific fields. Users retrieve questions based on the triple format (subject, relation, object) [1] or text form of knowledge in KGs. Knowledge graph question answering (KGQA) can be utilized to enhance the presentation of search engine results and improve the search experience. By incorporating knowledge graphs, search engines can offer more precise and relevant search outcomes, thereby assisting users in swiftly locating the desired information. Furthermore, employing a KGQA system for semantic search enables the provision of more accurate and personalized search results and recommendations. Various industrial sectors also employ domain-specific expert knowledge graphs [2,3,4,5] to manage data and guide decision-making processes through knowledge graph question answering. Large-scale KGs include Freebase [6], DBpedia [7], Wikidata [8], and YAGO [9], and the tasks of KGQA can be divided into simple questions and complex questions according to the difficulty of the question. A simple question has a direct answer that only requires the knowledge of a single triple, and the answer requires a few facts in KG, such as a subject–relation–object triple [10]. Complex questions need more knowledge than simple questions, and it is necessary to utilize more sophisticated query operations, such as using indirect relationships between entities, multi-hop relationships, and qualitative and quantitative constraints, to collect facts in the knowledge graph. There are still significant difficulties in responding to complicated queries, despite the gradual maturation of the study on simple knowledge graph question answering.

The challenges in the field of complex question answering over KGs (C-KGQA) can be roughly divided into two categories of complexity: questions with constraints and questions involving multiple hops of relations. Researchers also try to solve constraints and multi-hop properties over KGs synchronously (see [11]). A multi-hop question means that an answer is not simply related to a single piece of knowledge but may involve knowledge between multiple triples to reason out the answer. The question “What was the first novel of the 2022 Nobel Prize winner in Literature?” The KG contains the relevant triples, but a single triple cannot infer the answer. According to the topic entity “Nobel prize” in the question find the entity “Anne Herno”; then, another triple related to Anne Herno’s novel is found, and the final answer “Les Armoires vides” can be obtained. The corresponding process can be reflected in Figure 1. The model must deal with multiple entities and relationships from the KG, and the entities detected in these questions need to be linked to the KG triples. The reasoning process of the answer undergoes an information hop between two triples. Multi-hop questions are those that need to hop between two or more triples to submit messages. Constrained questions refer to questions that have certain constraints, which can be divided into time type (“... years ago”), sequential type (“1st... ”), quantitative type (“more than five times...”), and others. In Figure 1, we can lock the next triple with the time constraint “2022” and the quantifier constraint “first part”. These constraints modify the subject of a natural language question and, thus, change the final answer [12].

KGs, as a structured representation of relationships between entities and concepts in the real world, provide vital support for KGQA tasks [13]. We believe that solving the question of C-KGQA can be primarily divided into two main tasks. One is to better represent natural language questions (NLQs) and extract the true intent and key information relevant to reasoning [14]. The other is to effectively represent the graph, learn the knowledge represented in the graph, and accurately answer the questions. Regarding the related surveys, we observe Wu et al. [15] initially divided KGQA methods into semantic parsing (SP)-based methods and information retrieval (IR)-based methods, and this classification framework is still prevalent. They also provided an overview of the paradigms and advancements in these two approaches. However, due to the limitation of the publication year, current methods have further evolved. It is worth noting that they highlighted the challenge of the semantic gap in this field, which remains a prominent research trend today. Jr et al. [12] employed a systematic mapping methodology to provide an overview of C-KGQA. They offered a methodical mapping of the literature to identify the most popular venues, domains, and KGs utilized in the literature. The literature they compiled is available on a GitHub repository, providing a valuable resource for researchers seeking to explore relevant methodological studies. Furthermore, they presented a comprehensive overview of the methodologies, datasets, and metrics employed in the C-KGQA field while also addressing the challenges associated with the existing datasets. Lan et al. [10] conducted a comprehensive review of the recent advances in C-KGQA. They extensively discussed two mainstream categories of methods, provided a deep comparison of core modules, and presented a unified paradigm of neural–symbolic reasoning. Moreover, they employed a fine-grained classification approach to offer a nuanced description of the challenges associated with these two methods and solutions in recent work. Furthermore, they explored the significant impact of cutting-edge pre-training language models (PLMs) on C-KGQA. Zhang et al. [16] examined how the ability to answer complicated factual questions has developed across different data sources. They identify the features that these strategies have in common and put them all under the analysis–extend–reason framework. Specifically, they represented C-KGQA as answering complex factual questions in structured data sources and divided them into semantic parsing (SP)-based methods and graph-based methods, which diverges from our classification approach. They offered valuable insights into the types of complex questions that each method is best suited for. Subsequently, they presented the framework of the two methods individually. However, their analysis of the engagement of pre-trained language models (PLMs) in C-KGQA was concise, with a focus solely on their role as generators.

Following a brief discussion, our work is distinct in that we divide the methods into three categories based on various subtasks and give an overview of a module for each category. Additionally, we present the research ideas behind the latest methods to facilitate researchers with similar interests in accessing the most recent research achievements. Finally, we review the resources and evaluation methods for tackling C-KGQA tasks and summarize the research prospects in the field.

2. Preliminary

In this section, we will introduce examples of graph representations and provide explanations of certain characters that may be encountered, thus facilitating the comprehension of subsequent sections for researchers.

2.1. Knowledge Graph

KG aims to describe the concepts and entities, as well as their relationships, in the real world [17]. As an application of the semantic web, it is divided into three components: concepts, entities, and their relationship triples. Furthermore, use the “Resource Description Framework (RDF)” to describe. The basic data model of RDF includes three object types: (1) Resource: objects that can be represented by RDF are called resources, including entities, events, and concepts on the Internet. (2) Relation: it mainly describes the characteristics of resources and the relationship between resources. (3) Statements: a statement contains three parts, which are represented as $\langle subject,relation,object\rangle$, and usually called SPO triples [18]. In Figure 2, we show a KG instance in which each entity has a unique ID, and the relationship type of the person entity “Jackie Chan” is Person.marriage.spouses, which are associated with the named entity “FungGiu Lam”, which can be stored as $\langle JackieChan,people.marriage.spouse,FungGiuLam\rangle$ in KG as an SPO triple.

Large-scale open KGs (such as Freebase [6] and DBPedia [7]) are published by the rules of the Resource Description Framework(RDF), which support structured graph query languages to retrieve and manipulate triples [10]. To facilitate query large-scale knowledge graphs, graph query language SPARQL [19] is often used to retrieve and manipulate data stored in the knowledge graph. Similar graph query languages, such as Cypher [20] and Lambda-DCS [21], are also designed to query related knowledge graphs. So, how to unify the query language to improve the reusability of the KGQA model is also an obvious challenge. Specific KGs are designed for specific research. For example, the language knowledge graph WordNet [22], ConceptNet [23], and the open version OpenCyc, based on the largest general knowledge base Cyc, are the main knowledge graphs. HowNet [24] is a typical language cognitive knowledge graph. In addition, there are some knowledge graphs constructed for specific research, such as military or medical knowledge graphs.

2.2. Task Formulation

The knowledge is stored in the graph in the form of quadruples $(s,r,o)$, and we denote a graph as $G=\{\langle s,r,o\rangle |s,o\in \mathcal{E},r\in \mathcal{R}\}$, and for a natural language question, the token representation of the processed natural language question is represented as $q=\{w_1,w_2,...,w_m\}$, and the predicted answer is represented as $\widehat{A_q}$, usually assuming that the correct answer $A_q$ to the question will be found in G. The knowledge graph question-answering dataset $\left\{(q,A_q)\right\}$ usually contains some question–answer pairs to train the model.

For each triplet used in the next sections, we define the notation and problem definition in

Table 1. Notation and problem definition for each triplet used in the following sections.

Notation	Description
$e_s,e_r,e_o$	Embedding vector of subject entity, relation, and object entity
$\mathcal{E},\mathcal{R}$	Entity set and relation set
$(s,r,o)$	(Subject entity, relation, and object entity)
N	Number of entities
d	Dimensionality of embeddings
$M_r$	The matrix of relation
$\widehat{e_s},\widehat{e_r}$	Two-dimensional vectors of subject entity and relation
w	Kernel

.

3. Materials and Methods

We summarize the research on C-KGQA in recent years. The methods can be roughly divided into graph metric model (GMM)-based methods, graph neural network (GNN)-based methods, and a combination paradigm that combines PLMs with knowledge graphs.

3.1. Graph-Metric-Based Methods

The KG triples knowledge is embedded using the graph-metric-based approach into a continuous, low-dimensional embedding representation space, preserving the original entities and relations in the vectors [13]. For example, each entity e in a KG can be represented by a point in vector space, and each relation r can be modeled as an operation-like projection, translation, etc. [25]. Graph-metric-based methods are latent factorization methods that learn a scoring function $f(·)$ to measure the plausibility of triplets $(subject,predicate,object)$. Given $s,o\in \mathcal{E}$ and $r\in \mathcal{R}$, the complex metric embeddings $e_s,e_r,e_o\in {\mathcal{C}}^d$ are generated; then, by defining a scoring function $f(e_s,e_r,e_o)$, the objective of GM-based methods is to minimize the loss function or maximize optimization objectives. In short, typical GM methods generally consist of three steps: (a) embedding entities and relations; (b) defining a scoring function; and (c) learning entity and relation representations [25].

In Figure 3, we show the overall pipeline architecture for the GM-based methods for C-KGQA. Given a question and an available KG, GM-based methods first learn the question embedding and then identify topic entities and constraint embedding. Furthermore, the KG representation in the embedding space is learned to obtain entity embedding and relation embedding to reasoning. The reasoning chain module jointly leverages the embeddings we learned before to provide a set of candidate answer paths and score all candidate entities in KG to obtain the final answer.

The representative GM-based approach utilizes the distance-based scoring function as the distance between the subject entity and object entities. For instance, TransE [26] or RESCAL [27] generate high-dimensional embeddings of entities in real space by learning a scoring function to address questions. However, this approach only considers individual questions and overlooks the intrinsic relationships, thus applying only to solving one-hop problems. Another method is CompLEx [28], a tensor decompositional-based approach, proposing complex-valued embeddings to enhance the modeling of asymmetric relations between entities that are linear in both space and time complexity owing to the benefits of the dot product. This motivated researchers to use this method in their recent work. Saxena et al. [29] proposed a method called KGT5, which tackles link prediction and incomplete KGQA systems as sequence-to-sequence tasks using a single encoder–decoder Transformer model. This approach significantly reduces the model size and maintains scalability compared to traditional KGE models while achieving excellent performance on KGQA tasks over incomplete KGs. RotatE [30] embeds entities and relations into a complex vector space by representing relations as complex rotations. This model excels in handling relations with multiple meanings and performs remarkably well in C-KGQA tasks [31]. Chen et al. [32] proposed a logical query embedding framework called FuzzQE, which is based on fuzzy logic and used for answering complex first-order logical queries on large-scale incomplete graphs. FuzzQE utilizes fuzzy logic to define logical operators and improves performance by learning entity and relation embeddings. Jin et al. [13] applied an additional inference process called the relational chain reasoning module to prune the candidate entities ranked by the answer filtering module. They utilized long short-term memory (LSTM) networks [33] and the RoBERTa model to learn the semantic similarity between the relationship chains in the question description and the KG relationship chains. They also leveraged external supervision signals from the relational chains in the training samples to assist the inference. The effectiveness of this method in answering complex multi-hop questions was demonstrated. The success of knowledge graph embedding in diverse real-world NLP tasks inspired us to delve into its potential benefits within the KGQA task.

In

Table 2. Different KGE models used in C-KGQA.

Categories	KGE Model	Score Function	Memory Complexity
Distanced-based	TransE [26]	${\parallel s+o-r\parallel }_{l_1/l_2}$	$O(N_{e}d+N_{r}d)$
	RotatE [30]	$-\parallel s\odot r-o\parallel$	$O(2N_{e}d+2N_{r}d)$
Tensor-decompositional-based	ComplEx [28]	$Re\left(\langle s,r,\widehat{o}\rangle \right)$	$O(2N_{e}d+2N_{r}d)$
	RESCAL [27]	$s^{\top }{M}_{r}o$	$O(N_{e}d+N_r{d}^2)$
	DistMult [34]	$s^{\top }diag\left(M_r\right)o$	$O(N_{e}d+N_{r}d)$
Convolutional-based	ConvE [35]	$\begin{array}{c}f(vec(f([\widehat{e_s};\widehat{e_r}]\\ w))M)e_o\end{array}$	$\begin{array}{c}O(N_{e}d+N_{r}d+T{m}_w{n}_w\\ +Td(2d_m-m_w+1)(d_n-n_w+1))\end{array}$

, we have summarized the research in recent years that utilizes GM-based knowledge graph embedding models to enhance C-KGQA. We have analyzed their features, and we describe the memory complexity of the state-of-the-art KGE model [25], as well as the studies that utilize them, to facilitate researchers with similar interests in accessing this information.

3.2. Graph Neural Network (GNN)-Based Methods

Methods based on GNN are utilized to retrieve topic-specific graphs from the entire knowledge graph. Generally, entities one hop away from the topic entity and their connections form subgraphs that can solve simple questions. The NLQ and candidate answer in the subgraph can be represented as low-dimensional dense vectors. Different ranking functions [36,37] are proposed to calculate the similarity score of these candidate answers, and the entity with the highest score is considered the final predicted answer. Then, a memory network [38] is employed to generate the final answer. Recent works [39,40] have introduced attention mechanisms and multi-column modules to improve the ranking accuracy in this framework. We summarize the procedure of GNN-based methods into four modules, as illustrated in Figure 4.

Question Parsing: In this module, the question is encoded into a vector representation firstly to extract the topic entity and relationship, and then extract a subgraph related to the topic entity. This process [41] emphasizes the learning of the implicit mention of the question topic entity in the context and decomposes the question into multiple sub-questions. However, it is worth noting that they only consider single-hop relations, so the performance may be limited on problems involving multi-hop relations. Chen et al. [32] proposed a method for generating questions from knowledge graph subgraphs, aiming to generate natural language questions based on the subgraph and target answers. The approach utilizes a bidirectional Graph2Seq model to encode the KG subgraph, capturing both the context and structure information. Additionally, a node-level copying mechanism is employed in the RNN decoder to directly copy node attributes from the subgraph into the generated questions.

Subgraph generation: Starting from the topic entity, a subgraph related to a specific question is selected from the knowledge graph based on entity linking. This subgraph uses all entities and relationships related to the question as nodes and edges, respectively. Ideally, it should contain all relevant information for answering the question. Some graph filtering algorithms, such as Personalized PageRank (PPR) [42], can be used to trim the subgraph. Das et al. [37] proposed a semiparametric model that answers queries requiring subgraph reasoning patterns by leveraging the structural similarity between different subgraphs. The approach consists of a nonparametric component that dynamically retrieves similar queries and subgraphs, and a parametric component that identifies reasoning patterns from the subgraphs of the nearest neighbor queries and applies them to the subgraph of the target query. Additionally, the authors introduced an adaptive subgraph collection strategy to select a query-specific compact subgraph, enabling scalability on large-scale knowledge bases.

Subgraph retriever: A graph-based reasoning module converts the questions and candidate answers from the subgraph into low-dimensional dense vectors and performs semantic matching through vector-based calculation. During this process, the module propagates and aggregates information along adjacent entities, obtains path information, and, finally, updates according to the reasoning instructions. Retrieving subgraphs can simplify the reasoning process, and Zhang et al. [36] proposed a trainable subgraph retriever (SR) implemented via a dual encoder for KGQA. SR is decoupled from the subsequent reasoning process, allowing it to be combined with any subgraph-oriented KBQA model, creating a plug-and-play framework. Furthermore, when combined with NSM (a subgraph-oriented reasoner), SR achieved state-of-the-art performance in embedding-based KBQA methods through weakly supervised pre-training and end-to-end fine-tuning.

Answer generation: The answer generation module generates answers based on the final state of reasoning.

The method based on GNN can answer complex questions by constructing subgraphs and extracting relationships within them. However, this method often faces problems, such as excessively large or small subgraphs and poor information extraction in practical applications. To solve these problems, some improvement measures can be adopted, such as setting appropriate limiting conditions in advance to restrict the types of entities and relationships, narrowing down the scope of candidate subgraphs, and reducing redundant information. The efficiency of subgraph retrievers can be improved through policy adjustment; self-supervised methods can also be applied to improve the accuracy of answer sorting. We have compiled recent major improvements in this area and organized them in

Table 3. Different strategies of subgraph retrievers in recent works.

Model	Year	Strategy
HQGC [43]	2020	Based the director–actor–critic framework on hierarchical reinforcement learning with intrinsic motivation
CBR-SUBG [37]	2022	Dynamically retrieves similar queries and subgraphs; adaptive subgraph collection strategy
SR [36]	2022	Trainable subgraph retriever implemented via a dual encoder
Graph2seq [44]	2023	Utilized a bidirectional Graph2Seq model to encode the KG subgraph

. With these improvements, we can more accurately answer complex knowledge graph question-answering tasks.

3.3. The Joint Reasoning of PLM+KG

Training language models on a large-scale text corpus through unsupervised pre-training and then fine-tuning the pre-trained language models on downstream tasks have become a popular paradigm in natural language processing [45]. This approach can also be used to improve the capabilities of knowledge graph question-answering systems. Pre-trained language models based on a Transformer architecture have achieved the greatest success thus far, such as BERT [46] and its variants [47] and a series model of GPT [48]. By pre-training on large-scale text data, these models can learn powerful language representation abilities that can be applied to a variety of natural language processing tasks. Similarly, we summarize the procedure of the PLM+KG-based method into three modules, as illustrated in Figure 5. The challenge of the method of PLM+KG is how to bridge the semantic gap between natural language problems and structured data; thus, we add the solution to the pipeline from the current research.

LM encoding: Due to the strong language representation ability of pre-trained models, some researchers adapt PLMs to the task of NLP. The main technique used by pre-trained language models is the Transformer structure, which uses a self-attention mechanism to extract important information from the question sequence and encode it into vector representations. At the same time, the Transformer has the characteristic of improving the representational ability by stacking multiple layers. Finally, the pre-trained language model encodes the entire natural language question into a vector sequence; then, in the joint reasoning module, the problem representation is further updated by combining knowledge graph information. In the KGQA task, the pre-trained model is fine-tuned to obtain the optimal problem representation. Phuc Do et al. [49] proposed an enhanced question-answering system that utilizes the BERT model and a knowledge graph. The system incorporates two models based on BERT: a triple classification model and a text classification model. The triple classification model is built using BERT and classifies triples generated from all meta paths. The text classification model, also based on BERT, handles the content of triples by converting them into text and addressing text classification problems.

Knowledge Representation Learning: To represent entities and relationships in knowledge graphs as vector forms, some graph embedding algorithms are widely used in knowledge graph representation learning. These algorithms map entities and relationships in knowledge graphs to low-dimensional vector spaces by maximizing the similarity between neighboring nodes and minimizing the similarity between non-neighbor nodes so that more efficient computation and reasoning can be performed. For example, Sun et al. [50] used the GCN representation learning method to learn node embedding by utilizing local neighborhood structure information, generating node representation by iteratively aggregating each entity and its surrounding relationships. At the same time, as a graph neural network based on attention mechanism, graph attention networks (GATs) [51] can aggregate neighboring nodes to different degrees to better capture the relationships between nodes, thereby generating node representation.

Joint Reasoning: Combining pre-trained language models and knowledge graph representation learning, a joint reasoning framework is constructed to jointly represent natural language questions and entities/relationships in the knowledge graph. In this framework, the pre-trained language model generates a question representation vector by inputting a natural language question, while knowledge graph representation learning generates a knowledge graph representation vector by inputting entities and relationships. These two vectors can be fused using some fusion strategies to obtain the final question–answer pairs. Wang et al. [52] designed an interactive scheme after each layer of the LM and GNN, allowing bidirectional information exchange between the two modalities through specially initialized interaction representations (interaction token for the LM and interaction token for the GNN).

Answer Prediction: After the joint reasoning module updates the representation of the question and knowledge graph, their combined representation is obtained. In this module, the score of the prediction result, which indicates the probability of the answer being correct, is computed based on the combined representation. In addition to the basic framework, researchers are continuously exploring new technologies and methods, such as attention mechanisms, multi-task learning, transfer learning, and semi-supervised learning.

The main challenge of these classes of methods is how to better integrate information between the question and the knowledge graph for joint reasoning. We have summarized the strategies of the latest methods and organized them in

Table 4. Different strategies for integrating two modalities.

Model	Year	Strategy
Qagnn [57]	2020	Connect question and KG to form a joint graph.
JointLK [50]	2022	A dense bidirectional attention module.
GreaseLM [58]	2022	Designed a special interaction token and passed through N LM-based unimodal encoding layers.
DHLK [52]	2023	Proposed a dynamic heterogeneous graph with LMs and KG; relation mask self-attention.

. In addition to the basic framework, researchers are continuously exploring new technologies and methods, such as attention mechanisms, multi-task learning, transfer learning, and semi-supervised learning [53], to improve the efficacy and generalization ability of the models. Furthermore, some language models, such as T5 [54] and GShard [55], have been proposed to integrate multiple tasks and languages during pre-training, which can further enhance the generalization ability and adaptability of models [56].

4. Resource and Evaluation

In this section, we provide a detailed overview of the resources and evaluation metrics required to address complex problem-solving tasks. We discuss the knowledge graphs and datasets used in different subtasks and introduce evaluation metrics from various perspectives. In addition to traditional evaluation metrics, we acknowledge the emergence of new perspectives for assessing model performance with the advent of pre-trained models. We will delve into these perspectives in the upcoming chapters, offering a thorough exploration of their significance.

4.1. Resource

In this section, we will introduce the knowledge graphs and datasets that serve the tasks of C-KGQA. We provide an overview of commonly used general-purpose knowledge graphs and datasets. Additionally, we discuss the knowledge graphs and datasets [59] specific to certain subtasks or competitions. We acknowledge the importance of the availability of these resources and their impact on research and development.

4.1.1. KGs

Large-scale and general-purpose knowledge graphs, such as Freebase [6], YAGO [9], DBpedia [7], and Wikidata [8], are usually used to retrieve answers to the tasks of question and answer over knowledge graph. These systems are primarily designed to tackle open-domain problems. For commonsense questions, ConceptNet [23] is typically used as the knowledge source for retrieval.

Freebase [6] is an open and collaborative database, in which nodes are defined using $/type/object$ and links using $/type/link$. It is based on the RDF triple model [18] and stored in a graph database. Freebase enables quick traversal of arbitrary connections between topics and allows the easy addition of new schemas without changing the structure of the data. It should be noted that Freebase has been deprecated, but most current research works still use it.

DBpedia [7] is built by automatically extracting structured knowledge from Wikipedia pages and is one of the largest multi-domain knowledge bases in the world. CN-DBpedia is a Chinese knowledge graph that integrates general and domain-specific data.

YAGO [9] is a high-quality triple store that covers a broad range of concepts. It integrates data from Wikipedia, WordNet, and GeoNames, and links the highly accurate Wikipedia and WordNet sources together.

Wikidata [8] originated from Wikipedia and supports free collaborative editing in multilingual formats. It has been used as the knowledge base for Wikipedia and uses pages as the basic organizational unit, where entities refer to the top-level objects.

4.1.2. Datasets

Researchers have constructed diverse QA datasets to address different tasks. We categorize datasets based on the sub-tasks of complex knowledge graph question answering, and compile a list of datasets that contain constraint questions, as shown in Table 5, along with their source KGs, and an indication of providing corresponding query statements.

We have compiled a list of typical datasets in the C-KGQA field based on different sub-tasks, as shown in Table 5. The datasets for different sub-tasks are arranged according to the year of their creation. GrailQA [60] is a large-scale and high-quality knowledge graph question-answering dataset on Freebase, consisting of 64,331 questions labeled with answers and corresponding logic forms in different grammars such as SPARQL and S-expressions. It can be used to test three levels of generalization, which are i.i.d., compositional, and zero-shot in the task of KGQA; KQAPro [61] is a large-scale dataset containing nearly 120k natural language questions with challenging multi-hop reasoning, attribute comparison, set operation, etc. This dataset offers the KoPL and SPARQL queries for each question, making it valuable for evaluating KGQA system performance, as well as comparing the effectiveness of different algorithms. Moreover, the release of this dataset provides new resources for further research and development in KGQA technology.

The QALD challenge has become a well-established competition in terms of KGQA on DBpedia facts. Based on the current feasibility, we have chosen two datasets to list in the table. The test dataset of LC-QuAD contains 1000 question–query pairs [62], and the training dataset contains 4000 question–query pairs. Compared to many other knowledge graph question-answering datasets, LC-QuAD is a relatively difficult dataset because its questions involve complex cross-link reasoning. Specifically, each sample in LC-QuAD consists of two parts: a question and a query graph. The question is a natural language question, while the query graph is a graphical representation of the corresponding SPARQL query. This enables LC-QuAD to be used to evaluate the performance of knowledge graph question-answering models based on graph neural networks. LC-QuAD2.0 [63] is an upgraded version of the LC-QuAD, which contains over 30,000 questions, 57 knowledge graphs, and approximately 185,000 logical expressions. Similar to LC-QuAD, this dataset is more difficult and extends the diversity and complexity of the questions. In contrast to LC-QuAD, the questions in LC-QuAD2.0 not only involve cross-link reasoning but also include more complex question types such as entity relation identification, property value calculation, and entity retrieval. Additionally, the dataset contains more phrases, named entities, properties, and categories, as well as more KG architectures and languages.

Table 5. Introduction of datasets: “LB” means Leaderboard; “MC” means metric.

Datasets	KG	Year	Size			LB Score 1		MC 2
ComplexQuestions [64]	Freebase [6]	2016	2100			43.3 20		Acc, P, R, F1
						42.9 22
						42.8 20
ComplexWebQuestions [19]	Freebase [6]	2018	34,689			70.4 21		P, F1
						53.9 21
						50 20
GrailQA [60]	Freebase [6]	2021	64,331		73.42 23		81.87 23	EM, F1
					75.38 22		81.70 22
					76.31 22		81.52 22
KQApro [61]	Wikidata [8]	2022	117,970			95.32 23		Acc, F1
						93.85 22
						92.45 21
LC-QuAD [62]	DBpedia [7]	2018	5000	-		-	91 22	P, R, F1
				88 23		56 23	68 23
				88.11 22		83.04 22	83.08 22
LC-QuAD2.0 [63]	DBpedia [7] and Wikidata [8]	2019	30,000			92 22 91 22 86 22		F1

¹ We list the score in the order of the metric listed in the MC. The superscript on the scores indicates the year of the update. To simplify the table, we have only listed the scores of the top three, with the highest score highlighted in bold. ² The metrics in the table: “Acc”—accuracy; “P”—precision; “R”—recall; “F1”—F1 score; “EM”—Exact Match.

Table 5. Introduction of datasets: “LB” means Leaderboard; “MC” means metric.

Datasets	KG	Year	Size			LB Score 1		MC 2
ComplexQuestions [64]	Freebase [6]	2016	2100			43.3 20		Acc, P, R, F1
						42.9 22
						42.8 20
ComplexWebQuestions [19]	Freebase [6]	2018	34,689			70.4 21		P, F1
						53.9 21
						50 20
GrailQA [60]	Freebase [6]	2021	64,331		73.42 23		81.87 23	EM, F1
					75.38 22		81.70 22
					76.31 22		81.52 22
KQApro [61]	Wikidata [8]	2022	117,970			95.32 23		Acc, F1
						93.85 22
						92.45 21
LC-QuAD [62]	DBpedia [7]	2018	5000	-		-	91 22	P, R, F1
				88 23		56 23	68 23
				88.11 22		83.04 22	83.08 22
LC-QuAD2.0 [63]	DBpedia [7] and Wikidata [8]	2019	30,000			92 22 91 22 86 22		F1

¹ We list the score in the order of the metric listed in the MC. The superscript on the scores indicates the year of the update. To simplify the table, we have only listed the scores of the top three, with the highest score highlighted in bold. ² The metrics in the table: “Acc”—accuracy; “P”—precision; “R”—recall; “F1”—F1 score; “EM”—Exact Match.

In this section, we have introduced several commonly used datasets in C-KGQA tasks. We provided information on both traditional and newly released firing datasets, as well as some datasets used in competitions. Due to the limitations of the table, we have provided an official leaderboard and detailed information links for the aforementioned datasets to facilitate researchers with similar interests accessing them. We have noticed that some scores on the leaderboards lack clear sources, so we have excluded those from the scores listed in the table.

4.2. Metrics

To comprehensively evaluate KGQA systems, effective evaluation needs to be conducted from multiple perspectives. The main evaluation methods can be divided into two aspects: evaluating the reliability and robustness of the model.

4.2.1. Reliability

In KGQA tasks, for each question, there is an answer set containing its answer. The answer with the highest confidence score is usually selected as the answer to the question. Let Q be the set of questions, q be a single question in Q, A be the answer set, and $\widehat{A}$ represent the predicted answer.

Precision measures the proportion of correct predictions for a single question among all predicted answers. Its formula is as follows:

$$Precision=\frac{|A_q\cap \widehat{A_q}|}{|\widehat{A_q}|}$$

(1)

Accuracy represents the proportion of correctly answered questions by the system among all questions. Its formula is as follows:

$$Accuracy=\frac{|A_q\cap \widehat{A_q}|}{\left|Q\right|}$$

(2)

Hist@K is a metric defined based on accuracy. If the answer list is sorted in a certain order, and at least one correct answer exists among the first K answers, it is called a hit (1 hit); otherwise, if there is no correct answer among the first K answers, it is regarded as a miss (0 hit). Finally, the performance of the system will be evaluated by calculating the average number of hits for the entire set of questions. The value of K is defined manually based on the specific requirements of the task.

Recall indicates the proportion of correct answers predicted by the system out of all correct answers for a single question. Its formula is expressed as follows:

$$Recall=\frac{|A_q\cap \widehat{A_q}|}{|A_q|}$$

(3)

F1 denotes the weighted average of precision and recall, reflecting the comprehensive performance of the system. Its calculation formula is expressed as follows:

$$F1=2\times \frac{precision\ast recall}{precision+recall}$$

(4)

For the above metrics, there are corresponding precision@K, recall@K, and F1@K metrics, the calculation method of which is the same as Hits@K. The scope of examination K is manually specified, and the first K answers returned by the system are used as the scope of examination for calculation.

4.2.2. Robustness

The actual KGQA models need to be extended to practical daily applications. However, most of the current datasets are based on template collection, lacking diversity, which cannot train the generalizability of models. Moreover, training datasets are limited by expensive manual costs, and the current training data cannot cover all types of user queries. To improve the robustness of KGQA models, Gu et al. [60] proposed three levels of generalization as follows:

i.i.d. generation: i.i.d. (independent and identically distributed) generation, which is an important concept in probability and statistics. In the KGQA task, the concept of i.i.d. is mainly reflected in the construction and evaluation of datasets. To ensure that a KGQA model has good generalization performance, independent and identically distributed data should be used for training and testing.

Compositional generation: Compositional generalization refers to the ability of a model to utilize the previously learned knowledge structure and rules to generate new entities and relations by combining known entities and relations to adapt to different problem domains when processing new queries. Compositional generalization can be considered a structured and recursive way of modeling that is very effective in handling various complex KGQA tasks. Compositional generalization methods need to be adjusted and optimized according to specific KG and task characteristics. Generally, it is needed to combine techniques such as language representation learning and graph neural networks to achieve efficient and accurate KGQA systems.

Zero-shot: Zero-shot refers to the ability of a model to answer related questions correctly without encountering entities and relations that have never been seen during the model training. Although these entities and relations have not appeared in the training set, the model can utilize prior knowledge and infer the correct answer via reasoning, composition, and other methods. Zero-shot generalization enables the model to have a stronger generalization performance, which is suitable for the question-answering scenarios of new entities and relations in practical applications.

The level of generalization represents the expectation for a KGQA model: a model should be able to handle questions related to its training objects at a minimum. Furthermore, it should be extended to new combinations of trained structures. Ideally, a model should also be able to handle new patterns, even entire domains not covered by limited training data, including combinations of new structures. It is believed that models with improved generalization capabilities can be created by explicitly identifying several generalization levels. A practical KGQA model should also have strong generalization abilities to language variations and different reformulations for the same logical form. Some works [65,66,67] utilize generative models to address the issue of coverage and have achieved good performance on GrailQA.

5. Trends and Conclusions

This article provides a comprehensive introduction to complex knowledge graph question answering, including the task definition, main challenges and solutions, current mainstream research methods, and popular datasets. It also indicates the bottleneck challenge in this field, which is the generalization ability of a model. In addition, there are still some challenges in model performance:

(a) Multilingual question answering: Currently, most knowledge graph question-answering systems are designed for English questions. However, efficient QA systems are needed for users in different languages worldwide. Therefore, how to extend knowledge graph question-answering technology to multiple languages is a future problem to be solved.

(b) Incremental learning: When the knowledge graph is updated, using incremental learning methods to update existing node representations can maintain the accuracy and real-time performance of the knowledge graph.

(c) Real-time application optimization: In real-time question-answering applications, fast response time is essential, so optimizing system response speed and efficiency for real-time applications is required. Tian et al. [68] proposed a novel knowledge distillation (KD) method for incremental learning. The designed loss function can transfer the feature knowledge from PLMs to the incremental learning model, providing a new perspective for knowledge extraction from PLMs.

(d) Multi-modal question answering: Future question-answering systems need to support multiple forms of input, such as text, images, and voice [69], to meet the diverse needs of users. For example, Ning et al. [70] proposed a novel method called Differentiable Image–Language Fusion (DILF) for multi-view image and language fusion. This approach utilizes large-scale language models (LLMs) to generate text prompts with rich 3D semantics, enabling efficient fusion between image and text information.