Explainability of Automated Fact Verification Systems: A Comprehensive Review

Abstract

The rapid growth in Artificial Intelligence (AI) has led to considerable progress in Automated Fact Verification (AFV). This process involves collecting evidence for a statement, assessing its relevance, and predicting its accuracy. Recently, research has begun to explore automatic explanations as an integral part of the accuracy analysis process. However, the explainability within AFV is lagging compared to the wider field of explainable AI (XAI), which aims at making AI decisions more transparent. This study looks at the notion of explainability as a topic in the field of XAI, with a focus on how it applies to the specific task of Automated Fact Verification. It examines the explainability of AFV, taking into account architectural, methodological, and dataset-related elements, with the aim of making AI more comprehensible and acceptable to general society. Although there is a general consensus on the need for AI systems to be explainable, there a dearth of systems and processes to achieve it. This research investigates the concept of explainable AI in general and demonstrates its various aspects through the particular task of Automated Fact Verification. This study explores the topic of faithfulness in the context of local and global explainability. This paper concludes by highlighting the gaps and limitations in current data science practices and possible recommendations for modifications to architectural and data curation processes, contributing to the broader goals of explainability in Automated Fact Verification.

1. Introduction

Advances in Artificial Intelligence (AI), particularly the transformer architecture [1] and the sustained success it has brought in transferring learning approaches in natural language processing, have led to advances in Automated Fact Verification (AFV). AFV systems are increasingly used in AI applications, making it imperative that AI-assisted decisions are also accompanied by reasoning, especially in sensitive sectors like medicine and finance [2]. In addition, researchers have also increasingly recognized the significance of AFV in the modern media landscape, where the rapid dissemination of information and misinformation has become a pressing concern [3]. Consequently, AFV systems have become pivotal in addressing the challenges posed by the spread of online misinformation, particularly in verifying claims and assessing their accuracy based on evidence from textual sources [4]. An AFV pipeline involves the sub tasks of collecting evidence related to a claim, sorting the most relevant evidence sentences, and predicting the veracity of the claim. Some systems such as [5] follow an additional step in the preliminary stage to detect whether a claim is check-worthy or not before commencing on the other sub tasks in the pipeline. Besides these sub tasks, recent studies like [6] have started exploring how to generate automatic explanations as the reason for veracity prediction. However, not as much effort has been put into the explanation functionality of AFV compared to the strong progress made over the past few years both in fact checking technology and datasets [7]. The lack of focus on explanation is behind the growing interest in explainable AI research [7]. Explainable AI is also known as interpretable AI or explainable machine learning. Although used interchangeably, there is a subtle difference between explainability and interpretability, where the latter is not necessarily easily understood by those with little experience in the field, unlike the former. Explainable AI aims to provide the reasoning behind the decision (prediction) made, in contrast to the ‘black box’ impression (https://en.wikipedia.org/wiki/Explainable_artificial_intelligence, accessed on 15 September 2023) of machine learning, where even the AI practitioners fail to explain the reason behind a particular decision made by an AI system they designed. Similarly, the goal of explainable AFV systems is to go beyond simple fact verification by generating interpretations that are grounded in facts and that are communicated in a way that is easily understood and accepted by humans. Although there is broad agreement in the research community on the importance of the explainability of AI systems [8,9,10], there is much less agreement on the current state of explainable AFV. The latest studies on the verification of facts [11,12,13] do not cohere around an aligned view on the subject, while researchers like [12] state that “Modern fact verification systems have distanced themselves from the black-box paradigm”. The authors of [13] contradict this by stating that modern AFV systems estimate the truthfulness “using numerical scores which are not human-interpretable”. The same impression, as articulated in the latter statement, can be drawn from the literature review of state-of-the-art AFV systems. Another of the most recent arguments supporting this view is [11]. They assert that, despite being a “nontrivial task”, the explainability of AFV is “mostly unexplored” and “needs to evolve” compared to the developments in explainable NLP.

This issue is further exacerbated by the fact that providing justifications for claim verdicts has always been a crucial aspect of human fact checking [3,7]. Therefore, it becomes evident that the transition from manual to automated fact checking falls short of achieving its intended human aspect to the functionality for ‘Automated Fact Verification’ unless there is a clear incorporation of explainability.

In this paper, exploration of the concepts of explainability in XAI is initiated in Section 2, followed by a specific focus on its implementation in AFV in Section 3. By defining explainability within the context of AFV and introducing the architectural, methodological, and dataset-based aspects for discussing interpretations, the objective is to support and inspire research and implementations that can initiate the process of bridging the current explainability gap in AFV. The emphasis lies on the importance of datasets in achieving global explainability in AFV systems, suggesting that they should be a major focus of future research. Building upon these considerations, this paper has the following key objectives, which collectively aim to examine the challenges and prospects of explainability in AFV:

• Provide a comprehensive overview of the current state of explainability in AFV.
• Identify the challenges in the current landscape of AFV explainability, including the concepts of local and global explainability.
• Highlight the importance of creating explanation-learning-friendly (ELF) datasets to advance research in AFV explainability.
• Propose future research directions, including a balanced approach to explainability.
• Contribute to bridging the gap between AFV and XAI principles and ultimately enhancing the transparency and accountability of AFV systems.

Shortcomings of the Previous Reviews

There are a few prominent studies [3,7,14] encompassing the relevant research conducted in the areas of explainability in AFV and explainable NLP. In [14], explainable NLP datasets and collection methodologies for textual explanations in the context of explainable NLP, rather than AFV are primarily focused on. In [3], automated fact-checking is explored and the existing datasets and models are discussed, but their applications and challenges are primarily focused on, rather than explainability. In [7], an in-depth survey of the explanation functionality within fact-checking systems is provided, yet the emphasis remains on generating fact-checking explanations rather than reviewing the broader landscape of AFV. These studies, while valuable in their respective domains, exhibit limitations in offering a comprehensive overview of explainability within AFV systems. As this review progresses, the objective is to address these gaps by conducting a comprehensive analysis of the literature concerning explainability in AFV systems, highlighting both the strengths and limitations of existing methodologies and datasets. In this process, this paper contributes significantly to a more profound comprehension of vital aspects within the realm of explainable AFV, particularly in the context of XAI.

2. Explainable Artificial Intelligence

The field of interpretability in Artificial Intelligence is experiencing rapid growth, with numerous studies [2,10,15] exploring different facets of interpretation. These investigations are often conducted under the umbrella of explainable AI (XAI), which encompasses various approaches and methodologies aimed at providing explanations for the decision-making processes of black-box AI models, providing information on how they generate their outcomes. This section reviews the pertinent literature on XAI, with three main objectives: What is explainability? Why is it needed? How can it be implemented?

The primary objective of XAI is to build models that humans can interpret effectively, especially in sensitive sectors like military, banking, and healthcare. These domains rely on the expertise of specialists to solve problems more efficiently, while also seeking meaningful outputs to understand and trust the solutions provided [16]. Additionally, it benefits both domain specialists and developers when appropriate outputs are available, as it encourages investigation into the system when discrepancies occur. However, many AI systems that support decision making are developed as opaque structures, often referred to as ‘black-box’ models, which conceal their internal logic from the user and raise practical and ethical concerns [9,17]. These black-box models, while typically offering higher accuracy, do so at the cost of transparency, creating an inherent tension with the need for explainability in machine learning systems [15]. In contrast, white-box models are deliberately designed to be interpretable, which makes their outputs easier to understand, but the drawback of these models is the compromise on accuracy. Gray-box models attempt to strike a balance between interpretability and accuracy, offering a favorable trade-off [2]. Figure 1 demonstrates a comparison of these models and briefly conveys the idea of explainability in terms of AI systems. However, it is worth acknowledging that in certain scenarios, such as those involving structured data with naturally meaningful features, simpler classifiers, such as logistic regression or decision lists, may produce competitive results after appropriate preprocessing, as emphasized in the work by [18].

Upon investigating the reasons for the increase in popularity of this research field, it is evident that XAI has received increasing attention from both academia and industry [2,7,8,10], exhibiting an inflection point in the middle of the last decade [15]. This paragraph presents a brief analysis of the factors contributing to this surge in research interest in an attempt to resolve why explainability is important and continues to be a pressing requirement in AFV or AI in general. According to studies [2,19], as AI becomes more widely implemented, concerns about AI’s black-box working paradigm have also become prevalent among governments and the general public. This has led to the need for regulatory authorities to push for some form of explainability. An initial step towards this AI regulation was taken by the European Parliament in 2016, when it adopted the General Data Protection Regulation (GDPR) (Regulation (EU) 2016/679 of the European Parliament and of the Council, of 27 April 2016, uri=CELEX:32016R0679 ). With the GDPR policy requiring citizens to receive explanations for algorithmic decisions, explainability became a significant aspect of algorithm design thereafter [2,15,20]. Another authoritative supervision on XAI practices was by the R&D Agency of the United States Department of Defense (https://www.darpa.mil/program/explainable-artificial-intelligence, accessed on 20 September 2023), the Defense Advanced Research Projects Agency (DARPA) [21]. The DARPA regulated an XAI research program and funded 11 research groups from the USA; they worked in the direction of a common conception: AI systems need to be more explainable to be better understood, trusted, and controlled. The main contribution of this DARPA XAI initiative is the creation of an XAI Toolkit, consolidating the diverse artifacts of the program (such as code, papers, reports, etc.) and the lessons learned from the 4-year program into a centralized publicly accessible repository (https://xaitk.org/, accessed on 20 September 2023). This DARPA XAI program and the GDPR policy of the EU Parliament, along with the introduction of the EU AI Act (proposed the European law on Artificial Intelligence) (https://artificialintelligenceact.eu/, accessed on 20 September 2023), contributed majorly to the explainable AI movement observed today [2]. The drive for greater transparency and accountability in AI is not limited to the global stage; it is also reflected in national initiatives, including those spearheaded by the New Zealand government. For example, various initiatives and intergovernmental standards have been adopted by the New Zealand government to address the transparency and accountability of AI algorithms. The G20 AI Principles, endorsed by leaders in June 2019 and based on the OECD AI Policy Observatory, serve as a framework to promote responsible AI use. Similarly, the ‘Algorithm Charter for Aotearoa New Zealand’ aligns with the OECD principles and aims to improve transparency and accountability in AI algorithm usage. The country also actively contributes to global efforts through the OECD portal (https://oecd.ai/en/dashboards/overview, accessed on 20 September 2023), which showcases AI policy initiatives worldwide and emphasizes the importance of AI transparency. The AI Forum NZ has released its own set of principles, including a focus on AI transparency, while other measures such as the New Zealand Pilot Project from the World Economic Forum (https://www3.weforum.org/docs/WEF_Reimagining_Regulation_Age_AI_2020.pdf, accessed on 20 September 2023) further support the objective of improving AI transparency and explainability.

After defining explainability and its significance, the subsequent text in this section explores the various objectives and approaches for XAI implementation, as outlined in the current literature. Following this, it delves into the taxonomy of XAI.

2.1. XAI Objectives

The current body of work on XAI [2,9,16] identifies several essential conditions necessary for the implementation of explainable AI models. These are categorized into interpretability, accuracy, and fidelity.

Interpretability refers to the degree to which a model and its predictions can be understood by humans. It is often gauged by the complexity of the model, with simpler models generally being more interpretable. Accuracy denotes the model’s ability to correctly predict outcomes, particularly for new, unseen instances. It is quantified using metrics such as the accuracy score and the F1-score. Fidelity assesses how well an interpretable model replicates the behavior of a corresponding complex black-box system. It is crucial for ensuring that the simplified interpretations of the model’s decisions are faithful to the original model’s logic. Fidelity is evaluated by comparing the interpretable model’s predictions to those of the black-box model, often using the same metrics as for accuracy.

The next section reviews the standard approaches suggested in the literature to achieve these XAI objectives.

2.2. XAI Approaches

Based on a review of the relevant literature, including studies by various researchers [2,9,19,22], the practical approaches to achieving the aforementioned XAI objectives are generally categorized into model explainability and post hoc explainability; the latter is further appropriated into interpretations at the prediction level and at the dataset level [22]. It is important to note that the model explainability is interchangeably referred to as interpretability in certain works, such as [23].

2.2.1. Model-Based Explainability

Model-based explainability methods involve the creation of simple and transparent AI models that can be easily understood and interpreted. Such methods are particularly useful when the underlying data relationships are not highly complex, allowing simple models to effectively capture data patterns [17]. Models like decision trees and linear regression (unlike deep neural networks), inherently possess model explainability [15]. Models that are inherently interpretable provide their own faithful explanations, accurately representing the computations within the model [18]. However, when dealing with data that exhibit higher degrees of complexity or non-linearity, more intricate black-box models are designed and implemented. In such cases, post hoc explainability techniques are used to extract information about the relationships learned by the model [2,17,24].

2.2.2. Post Hoc Explainability

A post hoc explainability method operates on a trained and/or tested AI model, generating approximations of the model’s internal workings and decision logic [17,22]. Post hoc methods aim to reveal relationships between feature values and the model’s predictions, without requiring access to its internal mechanisms. This enables users to identify the most crucial features in a machine learning task, quantify their importance, reproduce decisions made by the black-box model, and uncover potential biases in the model or the underlying data. Prediction-level interpretation methods revolve around explaining the rationale behind the individual predictions made by the models. These methods delve into identifying the specific features and interactions that contribute to a particular prediction. For example, local interpretability might clarify why a specific loan application was rejected by highlighting the contributing factors [16]. In the literature, this concept is referred to by various terms such as local interpretations [16], local explanations [23], or local fidelity [25]. On the other hand, the approaches at the dataset level focus on comprehending the broader associations and patterns that the model has learned, with the aim of discerning the patterns related to the predicted responses on a global scale [22], such as understanding key features governing galaxy formation in astrophysics [16].

2.3. XAI Taxonomy

Taxonomy in XAI provides a structured framework to categorize and organize methods, techniques, and approaches for explaining AI and machine learning models, facilitating systematic discussions of model explainability and interpretability. Figure 2 provides a hierarchical visualization of XAI by offering a structured view of the XAI approaches mentioned in Section 2.2.

On local explainability, it is worth noting that while an explanation may not achieve complete faithfulness unless it provides a full model description, it is imperative for it to at least achieve local faithfulness. This means that the explanation must accurately correspond to the model’s behavior in (at least) the proximity of the instance being predicted, ensuring its meaningfulness. However, it is crucial to highlight that while global fidelity (globally faithful explanations) encompasses local fidelity, local fidelity does not imply global fidelity. Features that have global importance may not necessarily be significant in the local context, and hence the search for globally faithful but interpretable explanations remains a challenging endeavor, especially when dealing with complex models [25].

Section 2 presented an overview of explainability concepts, their significance, and the associated terminology. The following section will delve into their application within the context of AFV models.

3. Explainable AFV

Despite notable progress in the development of explainable AI techniques, achieving comprehensive global explainability in AFV models remains a challenging task. However, this issue encompasses multiple aspects that pose significant obstacles to research in the field of explainable AFV. First, only a relatively small number of automated fact-checking systems include explainability components. Second, explainable AFV systems currently do not possess the capability of global explainability. Finally, the existing datasets for AFV suffer from a lack of explanations. This study addresses these factors through three main perspectives: architectural, methodological, and data-based. The examination is conducted in alignment with the objectives and approaches of explainability discussed in the previous section. The emphasis is placed on the data perspective (Section 3.3) due to its crucial role in achieving global explainability within the context of AFV. The significance of this perspective is derived from the training dataset’s influence on AI model behavior and the critical necessity to achieve a high level of data explainability in AFV.

3.1. Architectural Perspective

The majority of AFV systems broadly adopt a three-stage pipeline architecture similar to the Fact Extraction and VERification (FEVER) shared task [26], as identified and commented on by many researchers [26,27,28,29,30,31,32]. These three stages (also called sub-tasks) are document retrieval (evidence retrieval), sentence selection (evidence selection), and recognizing textual entailment or RTE (label/veracity prediction). The document retrieval component is responsible for gathering relevant documents from a knowledge base, such as Wikipedia, based on a given query. The sentence-retrieval component then selects the most pertinent evidence sentences from the retrieved documents. Lastly, the RTE component predicts the entailment relationship between the query and the retrieved evidence. Although the above framework is generally followed in AFV, alternative approaches incorporate additional distinct components to identify credible claims and provide justifications for label predictions, as shown in Figure 3. The inclusion of a justification component in such alternative approaches contributes to the system’s capacity for explainability within the AFV paradigm.

The majority of AFV systems are highly dependent on deep neural networks (DNNs) for the label prediction task [7]. Furthermore, in recent years, deep-learning-based approaches have demonstrated exceptional performance in detecting fake news [33]. As mentioned in Section 2, there is, however, an inherent conflict between the performance of AI models and their ability to explain how they make decisions. However, although existing AFV systems lack inherent explainability [7], it would be foolish to overlook the potential to use these less interpretable deep models for AFV, as these models possess the ability to achieve state-of-the-art results with a remarkable level of prediction accuracy. However, this also indicates that model-based interpretation approaches may not be a suitable solution for AFV systems; the reason being that these methods require the involvement of simple and transparent AI models that can be easily understood and interpreted.

Therefore, considering the architectural characteristics of state-of-the-art AFV systems, a potential trade-off solution to achieve explainability may involve incorporating post hoc measures of explainability, either at the prediction level or dataset level, while still leveraging the capabilities of less interpretable deep transformer models. The subsequent subsections delve into the attempts made in the literature to incorporate post hoc explainability in terms of methods and input within the context of AFV.

3.2. Methodological Perspective

The methodological aspect looks at the different approaches utilized in the existing literature to develop explainable AFV systems.

3.2.1. Summarization Approach

In AFV, extractive and abstractive explanations serve as two types of summarization methodologies, providing a summary along with the predicted label as a form of justification or explanation. Extractive explanations involve directly extracting relevant information or components from the input data that contribute to the prediction or fact-checking outcome. These explanations typically rely on the emphasis of specific words, phrases, or evidence within the input. On the other hand, abstractive explanations involve generating novel explanations that may not be explicitly present in the input data. These explanations focus on capturing the essence or key points of the prediction or fact-checking decision by generating new text that conveys the rationale or reasoning behind the outcome. It is important to note that terminology can vary across fields. For instance, in the explainable natural language processing (explainable NLP) literature, ref. [14] refers to extractive explanations as ‘Highlights’ and abstractive explanations as ‘Free-text explanations’.

The approaches to explainability employed by existing explainable AFV systems are primarily extractive. For example, the work of [23] presents the first investigation of the generation of explanations automatically based on the available claim context, utilizing the transformer model architecture for extraction summarization purposes. Two models are trained with the intention of addressing this issue. One model focuses on generating post hoc explanations, where the predictive and explanation models are trained independently, while the other model is trained jointly to handle both tasks simultaneously. The model that trains the explainer separately tends to slightly outperform the model trained jointly. In [34], the task of explanation generation as a form of summarization is also approached. However, their methodology differs from that of [23]. Specifically, the explanation models of [34] are fine-tuned for extractive and abstractive summarization, with the aim of generating novel explanations that go beyond mere extractive summaries. By training the models on a combination of extractive and abstractive summarization tasks, they enabled the models to generate more comprehensive and insightful explanations by both leveraging existing information in the input and generating new text to convey the reasoning behind the fact-checking outcomes.

A potential concern is that these models (both extractive and abstractive) may generate explanations that, while plausible in relation to the decision, do not accurately reflect the actual veracity prediction process. This issue is particularly problematic in the case of abstractive models, as they can generate misleading justifications due to the possibility of hallucinations [3].

3.2.2. Logic-Based Approach

In logic-based explainability, the focus is on capturing the logical relationships and dependencies between various pieces of information involved in fact verification. This includes representing knowledge in the form of logical axioms, rules, and constraints to provide justifications for the verification results. For example, refs. [6,32] are recent studies that focus on the explainability of fact verification using logic-based approaches. In [6], a logic-regularized reasoning framework, LOREN, is proposed for fact verification. By incorporating logical rules and constraints, LOREN ensures that the reasoning process adheres to logical principles, improving the transparency and interpretability of the fact verification system. The experimental results demonstrate the effectiveness of LOREN in achieving an explainable fact verification. Similarly, ref. [32] highlights the potential of natural logic theorem proving as a promising approach for explainable fact verification systems. The system, named ProoFVer, applies logical inference rules to derive conclusions based on given premises, providing transparent explanations for the verification process. The experimental evaluation shows the efficacy of ProoFVer in accurately verifying factual claims while also offering interpretable justifications through the logical reasoning steps.

It is important to acknowledge certain limitations and drawbacks associated with this logic-based approach. First, the complexity and computational cost of logic-based reasoning can limit its scalability and practical applicability in real-world fact verification scenarios. Furthermore, while logic provides a structured and interpretable framework for reasoning, it may not capture all the nuances and complexities of natural language and real-world information. This means that the effectiveness of these approaches heavily relies on the adequacy and comprehensiveness of the predefined logical rules, which may not cover all possible scenarios and domains. Lastly, the interpretability of the generated explanations may still be challenging for non-expert users. They may involve complex logical steps that require expertise to fully understand and interpret.

3.2.3. Attention-Based Approach

Different from the summarization and the logic-based techniques, explainable AFV systems such as [24,35] use visualizations to illustrate important features or evidence utilized by AFV models for predictions. This provides users with a means to understand the relationships that influence the decision-making process. For example, the AFV model proposed in [35] introduces an attention mechanism that directs the focus towards the salient words in an article in relation to a claim. This enables the generation of the most significant words in the article as evidence (words with more weights are highlighted in darker shades in the verdict) and [35] claims that this strategy enhances the transparency and interpretability of the model. The explanation module of the fact checking framework in [24] also utilizes the attention mechanism to generate explanations for the model’s predictions, highlighting the important features and evidence used for classification.

However, ref. [3] illustrated several critical concerns associated with the reliability of attention as an explanatory method, citing pertinent studies [36,37,38] to reinforce the argument. The authors point out that the removal of tokens assigned high attention scores does not invariably affect the model’s predictions, illustrating that some tokens, despite their high scores, may not be pivotal. On the contrary, certain tokens with lower scores have been found to be crucial for accurate model predictions. These observations collectively indicate a possible ‘fidelity’ issue in the explanations yielded by attention mechanisms, questioning the reliability and interpretability of attention mechanisms in models. Furthermore, ref. [3] argue that the complexity of these attention-based explanations can pose substantial challenges for people lacking an in-depth understanding of the model architecture, compromising readability and overall comprehension. This scrutiny of the limitations inherent to attention-based explainability methods highlights the pressing need to reevaluate their applicability and reliability within the realm of AFV.

3.2.4. Counterfactual Approach

Counterfactual explanations, also known as inverse classifications, describe minimal changes to input variables that would lead to an opposite prediction, offering the potential for recourse in decision-making processes [18]. These explanations allow users to understand what modifications are needed to reverse a prediction made by a model. In the context of AFV, counterfactual explanations have been explored. The study in [39], for example, explicitly focuses on the interpretability aspect of counterfactual explanations in order to help users understand why a specific piece of news was identified as fake. The comprehensive method introduced in that work involves question answering and entailment reasoning to generate counterfactual explanations, which could enhance users’ understanding of model predictions in AFV. In a recent study [40] exploring debiasing for fact verification, researchers propose a method called CLEVER that operates from a counterfactual perspective to mitigate biases in predicting the veracity. CLEVER stands out by training separate models for claim–evidence fusion and claim-only prediction, allowing the unbiased aspects of predictions to be highlighted. This method could be explored further in the context of explainability in AFV, as it allows users to discern the factors that lead to specific predictions, even if the main emphasis of the cited work was on bias mitigation.

Nevertheless, counterfactual explanations in AFV, while providing valuable insights into why a model makes specific predictions, may also face challenges in their practical application. One notable limitation lies in the potential complexity and difficulty of interpreting minimal changes in input variables, especially in cases involving complex facts and evidence. This complexity could pose challenges to users in grasping the precise factors that influence the predictions of the models, which is a key aspect in achieving a broader interpretability in AI systems, as discussed in Section 2.

Table 1. Comparative analysis of diverse methodologies employed in enhancing explainability in Automated Fact Verification systems.

Methodological Aspect	Examples	Drawbacks
Summarization Approach	Ref. [23] utilizes the transformer model for extractive summarization. Two models are trained separately and jointly. Ref. [34] Fine-tuned for both extractive and abstractive summarization.	May generate misleading or inaccurate explanations. Particularly problematic for abstractive models.
Logic-based Approach	Ref. [6] LOREN framework uses logical rules for transparency. Ref. [32] ProoFVer uses natural logic theorem proving.	Complexity and computational cost limit scalability. May not capture all nuances of natural language.
Attention-based Approach	Ref. [35] uses the attention mechanism to focus on salient words. Ref. [24] utilizes attention for feature and evidence highlighting.	Relies on human experts for visualizations, diverging from the principles of XAI.
Counterfactual Approach	Ref. [39] focuses on interpretability by generating counterfactual explanations in AFV through question answering and entailment reasoning. Ref. [40] proposes the CLEVER method, which operates from a counterfactual perspective to mitigate biases in veracity prediction within AFV.	May face complexity in interpreting minimal input changes, particularly in intricate factual claims and evidence scenarios, potentially hindering the broader interpretability.

categorizes the existing approaches to develop explainable AFV systems into the four methodological aspects discussed: summarization, logic-based, attention-based, and counterfactual. Each category is illustrated with examples of studies that employ these methods, highlighting their unique contributions as well as potential drawbacks. The table serves as a comprehensive overview, aiding in understanding the various techniques used to enhance the explainability and interpretability in state-of-the-art AFV systems.

Additionally, it is important to note the inherent complexity in typical DNN-based AFV systems. When considered alongside the objectives of XAI outlined in Section 2.1, which emphasize that the interpretability of a predictive model is often assessed through its complexity (commonly measured by its size), this factor adds another layer of complexity to the already challenging task of achieving model explainability in state-of-the-art AFVs. However, the situation is equally challenging when it comes to post hoc explainability, especially in terms of achieving global explainability. None of the explainable AFV systems discussed provide global explainability; they mainly focus on prediction-level or local explainability by explaining the model’s decision-making process for specific instances or cases. On the other hand, global interpretability at the dataset level aims to uncover more general relationships learned by the model and provides a greater understanding of how the model learns and generalizes across different examples [22]. The following section explores the extent of dataset-level explainability in AFV, leading to an examination of its current state.

3.3. Data Perspective

The potential of data explainability lies in its ability to provide deep insights that enhance the explainability of AI systems (which rely heavily on data for knowledge acquisition) [2,9]. Data explainability methods encompass a collection of techniques aimed at better comprehending the datasets used in the training and design of AI models [2]. The importance of a training dataset in shaping the behavior of AI models highlights the need to achieve a high level of data explainability. Therefore, it is crucial to note that constructing a high-performing and explainable model requires a high-quality training dataset. In AFV, the nature of this dataset, also known as the source of evidence, has evolved over time. Initially, the evidence was primarily based on claims, where information directly related to the claim was used for verification. Subsequently, knowledge-base-based approaches were introduced, utilizing structured knowledge sources to support the verification process. Further advances led to the adoption of text-based evidence, where relevant textual sources were used for verification. In recent developments, there has been a shift towards dynamically retrieved sentences, where the system dynamically retrieves and selects sentences that are most relevant to the claim for verification purposes. The subsequent text examines the implications of these changes through the lens of explainability.

Systems such as [41] that process the claim itself, using no other source of information as evidence, can be termed as ‘knowledge-free’ or ‘retrieval-free’ systems. In these systems, the linguistic characteristics of the claim are considered as the deciding factor. For example, claims that contain a misleading phrase are labeled ‘Mostly False’. In [42], a similar approach is also employed, focusing on linguistic patterns, but a hybrid methodology is incorporated by including claim-related metadata with the input text to the deep learning model. These additional data include information such as the claim reporter’s profile and the media source where the claim is published. These knowledge-free systems face limitations in their performance, as they depend only on the information inherent in the claim and do not consider the current state of affairs [43]. The absence of contextual understanding and the inability to incorporate external information make dataset-level explainability infeasible in these systems.

In knowledge-base-based fact verification systems [44,45,46], a claim is verified against the RDF triples present in a knowledge graph. The veracity of the claim is calculated by assessing the error between the claim and the triples based on different approaches such as rule-based, subgraph-based, or embedding-based methods. The drawback of such systems is the likelihood of a claim being verified as false, based on the assumption that the supporting facts of a true claim are already present in the graph, which is not always feasible. This limited scalability and the inability to capture nuanced information hinder the achievement of explainability in these types of fact verification models.

Unlike the latter two approaches, in the evidence retrieval approach, supporting pieces of evidence for the claim verdict have to be fetched from a relevant source using an information retrieval method. Although the benefits of such systems outweigh the limitations of static approaches mentioned earlier, there are certain significant constraints that can also affect the explainability of these models. While the quality of the source (biased or unreliable), availability of the source (geographical or language restrictions), and resources for the retrieval process (time-consuming and expensive human and computational resources) can have a significant impact on the evidence retrieval and limit the scope of evidence, a deep understanding of the claim’s context is critical to avoid misinterpreted and incomplete evidence which leads to erroneous verdicts. Nevertheless, these limitations suggest that the evidence retrieval approach might not be entirely consistent with key XAI principles such as ‘Accuracy’ and ‘Fidelity’. This, in turn, casts doubt on the effectiveness of any post hoc explainability measures attempted within this data aspect.

An alternative approach is using text from verified sources of information as evidence; encyclopedia articles, journals, Wikipedia, and fact-checked databases are some examples. Since Wikipedia is an open-source web-based encyclopedia and contains articles on a wide range of topics, it is consistently considered an important source of information for many applications, including economic development [47], education [48], data mining [49], and AFV. For example, the FEVER task [26], an application in AFV, relies on the retrieval of evidence from Wikipedia pages. In the FEVER dataset, each SUPPORTED/REFUTED claim is annotated with evidence from Wikipedia. This evidence could be a single sentence, multiple sentences, or a composition of evidence from multiple sentences sourced from the same page or multiple pages of Wikipedia. This approach aligns well with the XAI principle of ‘Interpretability’, as Wikipedia is a widely accessible and easily understandable source of information. However, it is crucial to note that Wikipedia also comes with limitations that could impact the ‘Accuracy’ and ‘Fidelity’ principles of XAI, which can potentially impact the interpretability of models relying on Wikipedia as a primary data source. Firstly, like any other source, Wikipedia pages can contain biased and inaccurate content, and these can remain undetected for a longer period (the same goes for outdated information); this compromises the ‘Accuracy’ of any AFV model trained on these data. Secondly, despite covering a wide range of topics, Wikipedia suffers deficiencies in comprehensiveness (https://en.wikipedia.org/wiki/Reliability_of_Wikipedia#Coverage, accessed on 15 September 2023), limiting a model’s ability to understand contextual information fully, thereby affecting ‘Interpretability’. Lastly, models trained predominantly on Wikipedia’s textual content can develop biases and limitations inherent to the nature and scope of Wikipedia’s content, impacting both ‘Fidelity’ and ‘Interpretability’ when applied to diverse real-world scenarios and varied types of unstructured data.

Given these considerations and their misalignment with the XAI objectives of ‘Interpretability’, ‘Accuracy’, and ’Fidelity’, it becomes evident that relying solely on Wikipedia as a training dataset may not be the most effective pathway toward explainable AFV.

Alternatively, Wikipedia can be used as an elementary corpus to train the AI model to achieve a general understanding of various knowledge domains for AFV, and this background or prior knowledge can then be harnessed further with additional domain data to gain a deeper context (which helps the model to attain information on global relationships and thus increase explainability). As the largest Wikipedia-based benchmark dataset for fact verification [28,50], the FEVER dataset can unarguably be considered as this elementary corpus for AFV tasks, and transformers and transfer learning is the most pragmatic technology choice for AFV according to state-of-the-art systems [31,32,51].

The quality of the dataset used or created for an application is a major factor in determining the explainability of a transformer-based AFV model and its ability to comprehend the underlying context. For example, ref. [52] developed the SCIFACT dataset in order to expand the ideas of FEVER to COVID-19 applications. SCIFACT comprises 1.4 K expert-written scientific claims along with 5K+ abstracts (from different scientific articles) that either support or refute each claim and are annotated with rationales, which consist of a minimal collection of sentences from the abstract that imply the claim. This study demonstrated the obvious advantages of using such a domain-specific dataset (it can also be called a subdomain here as scientific claim verification is a sub task of claim verification) as opposed to just using a Wikipedia-based evidence dataset. In [52], it is argued that the inclusion of rationales in the training dataset “facilitates the development of interpretable models” that not only label predictions but also identify the specific sentences necessary to support the decisions. However, the limited scale of the dataset, consisting of only 1.4 K claims, necessitates caution in interpreting assessments of system performance and underscores the need for more expansive datasets to propel advancements in explainable fact checking research.

Building on this perspective of improving the quality and diversity of the dataset, ref. [53] critically evaluated the FEVER corpus, emphasizing its reliance on synthetic claims from Wikipedia and advocating for a corpus that incorporates natural claims from a variety of web sources. In response to this identified need, they introduced a new, mixed-domain corpus, which includes domains like blogs, news, and social media—the mediums often responsible for the spread of unreliable information. This corpus, which encompasses 6422 validated claims and over 14,000 documents annotated with evidence, addresses the prevalent limitations in existing corpora, including restricted sizes, lack of detailed annotations, and domain confinement. However, through a meticulous error analysis, ref. [53] discovered inherent challenges and biases in claim classification attributed to the heterogeneous nature of the data and the incorporation of fine-grained evidence (FGE) from unreliable sources. These findings illustrate substantial barriers to realizing the fundamental goals of XAI, particularly accuracy and fidelity. Moreover, ref. [53]’s focus on diligently modeling meta-information related to evidence and claims could be understood as their implicit recognition of the crucial role of explainability in the realm of automated fact checking. By suggesting the integration of diverse forms of contextual information and reliability assessments of sources, they highlight the necessity of developing models that are not only more accurate but also capable of providing reasoned and understandable decisions, a pivotal step towards fostering explainability in automated fact checking systems.

Table 2. Comparative analysis of dataset types and their impact on explainability in AFV systems.

Fact_Verification Dataset Type	Example_Studies	Knowledge_Type		Text_Type		Domain_Type		Remarks
		Knowledge-Free	External Knowledge	Structured Data	Free Text	Single Domain	Multi Domain
Knowledge-free Systems	[41,42]	√	×	−	−	−	−	Lack of contextual understanding; dataset-level explainability infeasible
Knowledge-Base-Based	[44,45]	×	×	√	×	√	√	Limited scalability; inability to capture nuanced information
Wikipedia-Based	[54]	×	√	×	√	×	√	Biased and inaccurate content; limited comprehensiveness
Domain (Single)-Specific Corpus	[52]	×	√	×	√	√	×	Limited size; potential for biased evaluation
Mixed-domain Corpus (non-Wikipedia-based)	[53]	×	√	×	√	×	√	Challenges in classification due to heterogeneous data (impacts accuracy); evidence from unreliable sources (impacts fidelity)

Note: In this table (√) indicates ‘Yes’ or ‘Applicable’, (×) indicates ‘No’ or ‘Not Applicable’, and (−) signifies the absence of a particular attribute or feature.

offers a comprehensive categorization of the datasets used in fact verification systems, highlighting a variety of dataset types, each with distinctive attributes and challenges. The datasets are categorized meticulously based on their inherent nature and source, such as ‘Knowledge-free Systems’, ‘Knowledge-Base-Based’, ‘Wikipedia-Based’, ‘Domain (Single)-Specific Corpus’, and ‘Mixed-domain Corpus (non-Wikipedia-based)’. Each type is represented with illustrative studies and remarks to provide insight into the inherent limitations or challenges in relation to enhancing explainability in AFV systems. The categorization is enriched with sub classifications under ‘Knowledge Type’, ‘Text Type’, and ‘Domain Type’. ‘Knowledge-free systems’ are denoted with dashes (-) under ‘Text Type’ and ‘Domain Type’, indicating the inherent absence of these attributes. This underscores the retrieval-free nature of such systems, which predominantly rely on the intrinsic linguistic features of the claims, thus lacking contextual understanding. The ‘Knowledge-Base-Based’ type can be either single domain or multi domain, represented by check marks in both sub-categories under ‘Domain Type’. This illustrates the versatility of knowledge-based systems in utilizing structured information from a specialized domain or amalgamating insights from multiple domains. The ability to cater to varied domains accentuates the expansive applicability of such systems, though it also brings forth challenges related to scalability and capturing nuanced information. ‘Wikipedia-Based’ datasets, inherently multi-domain, are highlighted separately to focus on the specific challenges of using Wikipedia as the main information source, such as dealing with potential biases and inaccuracies. The ‘Domain (Single)-Specific Corpus is distinguished as it focuses on a specialized or singular domain, providing depth and specificity. While this focus allows for a detailed exploration of a particular domain, it also poses limitations due to the restricted scope and potential biases inherent to the selected domain, thereby affecting the overall evaluation and applicability of the system. Additionally, the ‘Mixed-domain Corpus’ type emphasizes the inclusion of diverse domains, especially those not solely reliant on Wikipedia, addressing the challenges arising from data heterogeneity and reliability.

The categorization in Table 2, coupled with associated remarks, is intended to act as a resource, providing information on the various challenges and possibilities to improve explainability within AFV systems. This categorization can guide researchers and practitioners in making informed decisions regarding dataset selection and utilization, providing a clearer understanding of the implications and limitations of different dataset types in the context of Automated Fact Verification.

This study acknowledges the extensive investigations conducted by [14] in explainable NLP and by [7] in explainable AFV, which provide meticulous lists and insightful analyses of prevalent datasets in their respective fields. It is crucial to clarify that the aim of this section (Section 3.3) is not to perform an exhaustive review of datasets, a task diligently undertaken by the aforementioned studies. Instead, it is uniquely positioned to illuminate the distinctive attributes and inherent diversity within various dataset types in AFV. This attempt to examine the impact of different data types on explainability serves as a thoughtful addition to ongoing discussions and reflections on the subject, offering a new perspective on the multifaceted interactions between data diversity and explainability in AFV.

4. Discussion

While fact checking datasets commonly support the standard three-stage pipeline of fact verification, there is currently a lack of datasets that specifically facilitate explanation learning aligned with the government and intergovernmental standards on XAI. This is of paramount importance in explainable AFV; if an AI system is expected to produce explanations, it should have the ability or opportunity to consume explanations. To achieve this, it is necessary to train a model network using an explanation-learning-friendly (ELF) dataset. However, prominent large-scale datasets like FEVER [54] and MultiFC [55] lack this aspect of the fact verification task. Furthermore, currently there are no alternative resources available to address this limitation, as commented on in [56]. To create an ELF dataset, it is essential to analyze the dataset practices of fact verification systems with a focus on explainability. This paper has undertaken this crucial initial step and found that the absence of an explanation-based fact verification corpus presents a significant obstacle to advancing research in the field of explainable fact checking.

In addition to the lack of suitable ELF datasets in AFV, another significant challenge to the growth of the explainable AFV field is the ambiguity and discrepancies surrounding the concepts of local and global explainability. Global interpretability refers to the ability to comprehend the overall logic and reasoning of a model, including all possible outcomes. It involves understanding the complete decision-making process and the underlying principles of the model. On the other hand, local interpretability refers to the ability to understand the specific reasons or factors that contribute to a particular decision or prediction. It focuses on the interpretability of individual predictions or decisions rather than the entire model. These terms are not consistently understood and implemented in different research communities, leading to confusion and slowing progress in the field. Although XAI researchers [9,16,17,22] generally agree on the definitions of local and global explainability, explainable AFV researchers have different interpretations and perspectives, contributing to the ambiguity surrounding explainable AFV. For example, ref. [34] focuses on local coherence and global coherence, evaluating sentence cohesion and the appropriateness of explanations in relation to the claim and associated label, both at the prediction level. On the other hand, ref. [23] discusses the explainability as providing local explanations for individual data points, without specifically addressing local or global aspects. As a result, the definitions of local coherence, global coherence, and explainability in AFV studies predominantly refer to prediction-level explainability, leaving the concept of global explainability in AFV insufficiently defined.

The lack of recognition of the importance of global explainability is evident in implementations as well. Existing systems primarily focus on local explainability, which hampers an adequate understanding of a system’s decision-making process on the dataset level. In an extensive survey conducted by [7] on explainable AFV systems, it was found that all the examined systems focused primarily on providing explanations for individual predictions rather than offering explanations about the underlying fact-checking model itself. This indicates a prevalent trend in the field of explainable AFV, where the emphasis is on local explainability. However, this local explainability is not sufficient for AFV systems because it only provides insights into individual predictions without offering a holistic view of the system’s overall behavior. Furthermore, global explainability is crucial for AFV systems, as it provides a comprehensive understanding of how the system arrives at its predictions and decisions. This global approach also allows AFV systems to align with advances in XAI research and comply with the XAI principles, enabling transparency and accountability.

In addition to the ambiguity surrounding local and global concepts, the field of explainable AFV is further complicated due to variations in how explainability concepts are categorized, suggesting a lack of consensus on taxonomy. For example, while [24] distinguishes between intrinsic explainability and post hoc explainability, other researchers in explainable AFV, such as [23], propose a categorization that broadly divides XAI into interpretability and explainability. In [24], intrinsic explainability is described as the process of creating self-explanatory models that inherently incorporate explainability. This suggests that their definition of ‘intrinsic explainability’ closely aligns with the general notion of ‘interpretability’ related to model-level reasoning, as discussed in Section 2.3. However, the choice of the term ‘intrinsic’ in [24] adds a distinct nuance to this categorization. On the other hand, their view on post hoc explainability is in line with standard XAI. In contrast, while ref. [23] aligns ‘interpretability’ with the mainstream XAI taxonomy, they adopt a narrower view for ‘explainability’, reserving it for local explanations of individual instances, which is a subset of post hoc explainability. This deviates from the wider view, where ‘explainability’ usually refers to model-level justifications.

These disparities in taxonomy demonstrate that the ambiguity extends beyond the local and global dimensions, contributing to the overall ambiguity within the field of explainable AFV. This disagreement and discrepancy among the relatively few existing explainable AFV systems pose significant challenges for the growth and advancement of research in this field and highlight the need for a more standardized approach to explainability in AFV systems.

Limitations

This study concentrates on exploring the explainability of DNN-based AFV models, consequently not addressing other explainability approaches such as rule discovery [57,58]. This research gap provides an opportunity for future studies to investigate the model explainability of DNNs, particularly transformer models, in the context of AFV.

Similarly, to limit the scope of this paper, this study did not address the absence of a clear and established link between the various interpretation methods proposed in the literature and the evaluation criteria for measuring explainability; the lack of clarity regarding how to measure explainability is a significant challenge in this field of the research. This aspect warrants further investigation in future research to enhance the assessment of explainable AFV systems.

5. Future Research Directions

In addition to the future plans outlined in the limitations of this study, the following directions for exploration and research are proposed.

• Direction 1: Exploring a Balanced Approach to Explainability in AFV: Researchers should explore the development of techniques and methodologies aimed at achieving a balanced approach to explainability, integrating both global and local perspectives into AFV systems. This involves understanding the broader relationships and patterns that underlie AFV model decisions across diverse factual claims and evidence (global explainability), while also addressing the specific concerns related to individual instances (local explainability). For instance, a nuanced exploration of gray explainability could involve refining gray-box models to optimize the trade-off between interpretability and accuracy, ensuring that the explanations provided are as meaningful and understandable as possible without incurring a substantial loss in predictive accuracy. Although examples such as dispute resolution and individual patient treatment decisions illustrate the broader applicability and importance of this approach beyond the realm of fact verification, they underscore the universal need for tailored and comprehensible explanations in individual cases. In fact verification systems, a balanced approach is particularly crucial for gaining both a localized understanding of individual claims and a broader insight that can inform strategies for handling diverse types of information and evidence. By investigating methods that provide insights into AFV model behavior and reasoning patterns on both the macro and micro levels, researchers can work towards achieving a holistic understanding of explainability in AFV systems. Following the principles of XAI, a potential starting point could be to explain multiple representative individual predictions (locally) as a means to gain insights towards a more comprehensive understanding, as suggested in [25]. This nuanced exploration, which aligns with the overarching goal of achieving explainability in AFV systems, ensures that the insights gained are as widely applicable as they are individually relevant, potentially leading to more informed and equitable decision-making processes across different domains.
• Direction 2: Comprehensive Investigation and Comparative Analysis of AFV Datasets: Future research endeavors could benefit from undertaking a meticulous and comprehensive review of the applicable datasets for AFV, informed by the insights provided in Table 1 and Table 2. Table 1 outlines a comparative analysis of various methodologies used to improve explainability in AFV systems, while Table 2 delves into the distinctive attributes and inherent diversity within various types of datasets in AFV. A focused study in this direction could reveal deeper insights into the suitability and compatibility of various datasets with different AFV models and explainability techniques, providing a more nuanced understanding of the interactions between dataset characteristics and explainability. Such an investigation would not only enrich the understanding of the influence of diverse datasets on the explainability of AFV models but also reveal untapped potential in utilizing underexplored types of dataset to enhance model transparency and interpretability. By synergizing the diverse techniques for explainability and the variety of dataset types highlighted in the tables, this research direction has substantial potential to reduce the gap in the field of explainable AFV.
• Direction 3: Development of an Explainability-Focused, Explanation-Learning-Friendly (ELF) Dataset: As a logical progression from Direction 2, researchers should prioritize developing an ELF dataset to address the lack of explanations in existing AFV datasets, enabling more nuanced studies in explainability in AFV. This customized dataset would serve as a benchmark to assess the effectiveness of various AFV models in generating meaningful explanations, thereby fostering advancements in creating explainable AFV systems. Such a focused endeavor would be pivotal in bridging existing gaps and furthering research in explainable AFV, allowing for an exploration of the interplay between dataset attributes, model structures, and explainability methodologies.

6. Conclusions

This study addresses the challenge of explainability within AFV systems, shedding light on key insights that both researchers and practitioners in the field can leverage.

Drawing from a comprehensive analysis of AFV models and the principles of XAI, this paper outlines a clear road map for future research. This study firmly advocates for a shift in focus from local explainability, which currently dominates AFV systems, to a broader embrace of global explainability in line with XAI objectives. To catalyze this transition, the necessity of developing specialized training datasets tailored explicitly for the pursuit of global explainability is highlighted.

The examination of data practices within current AFV frameworks revealed critical gaps and limitations. Moreover, it exposed inconsistencies and discrepancies among AFV systems regarding the concepts and perspectives of explainability.

While this study serves as a foundation for future research, it is imperative to recognize that the path from Manual to Automated Fact Verification remains incomplete. The incorporation of explainability as an essential functionality in modern AI systems must be prioritized, as highlighted in the problem statement in the Introduction.

In conclusion, this study makes a valuable contribution to the expanding field of AFV and XAI by offering a determinate, interconnected approach to address the pressing challenge of explainability in AFV systems. It is anticipated that the analysis and insights gained from this paper will catalyze further research on explainability by both researchers and practitioners in the field, ultimately leading to more transparent and accountable AFV systems.