1. Introduction
Explainable artificial intelligence (XAI) is a set of methods used to tackle the interpretability problem [
1] by providing users explanations on how a model came to its conclusion. By providing these additional insights into its reasoning or internal workings, the model’s transparency is increased, resulting in higher trust from the user.
In the context of deep learning, the model learns from the data, and the internal learning of the model is, generally, a black box. There is a constant call to make these black-box models more interpretable. It is especially crucial to understand how the model made certain decisions for critical systems. It is also essential for systems that are not critical, but where their black-box nature induces biases or other ethical dilemmas.
One example of the usage of XAI in daily life is reasons why a mortgage is approved or denied by a bank. This benefits two parties: the applicant for the mortgage and the bank. The applicant benefits because accepting a possible denial may be easier if a reason is provided. At the same time, the bank benefits from increased insight into how their model behaves and can therefore avoid biases or other ethical issues induced by the model [
2].
Another example concerns healthcare [
3], where a heavy debate is ongoing on AI implementation as “explainability is not a purely technological issue, instead it invokes a host of medical, legal, ethical, and societal questions that require thorough exploration” [
3]. Another example is the bias in diagnosis based on X-ray data. The diagnosis is biased toward under-served-populations (see Seyyed-Kalantari et al. [
4]).
Embracing the need for XAI, in April 2021 the European Commission proposed a regulation to ensure that complex “high-risk AI systems shall be designed and developed in such a way to ensure that their operation is sufficiently transparent to enable users to interpret the system’s output and use it appropriately” [
5] [Article 13]. Half a year later, in September 2021, Brazil followed by approving a similar proposal.
Although the terms “explainable AI” and “interpretable AI” are often confused as synonyms, they have subtle differences. Interpretable AI refers to the characteristic of an AI system that allows humans to understand the processes it uses to make decisions or predictions. In other words, “we will consider interpretability as the possibility of understanding the mechanics of a Machine Learning model, but not necessarily knowing why [
6]”. On the other hand, XAI focuses on the overall understanding of the data, including the unobserved cases. This also includes certain feature values not present in the data or some data points that have not occurred. In this paper, we do not use both terms in their strict meaning; instead, we focus on the larger class of XAI while keeping in mind that the cited paper may have used these terms interchangeably.
Numerous XAI methods have been developed over the past 5 to 10 years [
7], ranging from model-agnostic methods (those that can be applied to all existing and future models) to model-specific intrinsic methods (those that can only be applied to a single or subset of methods). New methods and techniques are actively researched and the number of scientific papers describing these techniques is rapidly increasing.
The current rapid publication rate calls for higher-tier research that summarizes and synthesizes the current state of the art and highlights existing gaps. The most commonly used method is conducting systematic literature reviews that summarize and synthesize the current state of research on a topic [
8].
Systematic literature reviews (SLRs) aim to summarize multiple individual research articles, whereas meta-reviews are used to summarize both individual research articles as well as existing SLRs. The top-tier evidence synthesis method is called a
tertiary review, which is a systematic review of systematic literature reviews [
9].
In this article, we will conduct a tertiary review of existing XAI methods and their characteristics, such as input data type or model type. Furthermore, we aim to categorize these methods into well-defined boxes to create a clear overview of the existing literature. Example categories could be heat maps, graphs, or decision trees.
Therefore, the goal of this paper is to provide a mapping of XAI categories with their characteristics and fill this mapping with existing XAI methods. This mapping will result in 2D matrices, with each cell indicating whether an XAI method in that category has or has not been researched with that characteristic.
We want to emphasize that this paper is aimed at meta-level reviews where the focus is on summarizing existing literature reviews and finding open research directions. There are multiple literature reviews (and most of them are cited in the coming section) written discussing an exciting research topic within the realm of explainable artificial intelligence (as it is detailed in this paper). However, to the best of our knowledge, this paper is the first attempt to combine knowledge from existing literature reviews into a single article. During the final editing of this paper, we came across the recent paper work of Saeed and Omlin [
10] where they provided a scoping review of the field of XAI. Their review presents challenges and research directions in XAI as discussed in the research papers. Their approach is different from ours. Instead of focusing on the discussion points mentioned in primary review papers, we catalog which research directions have and have not been explored and extract new research gaps from this. Furthermore, this paper specifically focuses on XAI techniques, whereas Saeed and Omlin [
10] discuss challenges in XAI in a broader sense.
To clarify the terminology used in this research, the terms used throughout this paper are defined in
Section 2. In
Section 3, the research method used for this tertiary review is explained. In
Section 4, we synthesize the findings of the review. We then describe the limitations of our research guidelines for future work in
Section 5. Finally, we conclude our findings in
Section 6.
3. Review Methodology
This section describes the search strategy, search string, and inclusion and exclusion criteria. A complete overview of their application, from research question formation to collected studies, is provided to ensure a rigorous and reproduceable approach for this SLR.
3.1. Research Questions and High-Level Synthesis
Over the last five years, numerous systematic literature reviews on XAI have been conducted (as we shall see in the coming section).
We aim to present a high-level synthesis based on these literature reviews focused on answering the following questions:
RQ 1: What are distinguishing characteristics of XAI methods?
RQ 2: What are the distinct categories that can be established to classify various XAI methods based on their shared characteristics and methodologies?
RQ 3: Which combinations of XAI categories and characteristics have been researched?
The tertiary review (see [
9]) method is considered above a systematic literature review in the hierarchy of evidence synthesis methods, as seen in
Figure 1. Since there are no well-defined guidelines for conducting tertiary reviews, we adapted the guidelines of Kitchenham et al. [
8], where only existing systematic literature reviews and meta-analyses are taken into account, instead of individual articles.
3.2. Literature Retrieval and Selection
To retrieve literature, we need to craft a query. To craft the search query, we adapted a fine-tuning-based approach, where we started with a query and fine-tuned it by looking at the results returned by the database Scopus [
11]. Scopus was chosen since it contains articles from a broader area of science and engineering and due to the ease of retrieving the returned results.
Based on the research question, we started with the combination of “Explainable AI” and “Systematic Literature Review”, together with their respective synonyms, as the initial search query. After a couple of iterations, it resulted in a more comprehensive search query, as given in Listing 1.
Listing 1. Search query. |
![Make 06 00098 i001]() |
As can be seen in Listing 1, the synonyms used for “explainable AI” are “explainable artificial intelligence”, “xai”, “interpretable AI”, and “explainable machine learning”. The synonyms used for “systematic literature review” are “systematic review” (which makes the search term “systematic literature review” redundant), and “meta-analysis review”. We search the broader area of science and engineering instead of confining ourselves to computer science. This choice is made because relevant articles could be focused on a specific domain and would then generally be published in sources associated with the application domain.
To reduce the bias of database choice, this query was run in multiple databases resulting in varying results. Executing this query string on Scopus resulted in 92 articles. After applying the same query to IEEE Xplorer (12 results), ACM (four results), ScienceDirect (80 results), and Springer (41 results), 229 articles were retrieved. The search query is adapted to match the specific needs of each database.
Furthermore, one article was added manually, as this article is not yet published but does fit the search query, resulting in 230 articles, as seen in
Figure 2.
3.3. Literature Selection
The proceeding subsection describes the steps followed based on the guidelines outlined in [
8] to select only the literature relevant to our research questions.
3.3.1. Step 1: Duplicate Removal
Some of the resulting articles appear in multiple databases. Hence, the duplicates were removed. In total, 33 duplicates were removed, resulting in 197 unique articles.
3.3.2. Step 2: Inclusion Criteria
To find the most relevant articles, the following inclusion and exclusion criteria are defined:
IC1: include only articles performing a systematic literature review or meta-analysis on XAI methods.
Motivation: this is the aim of our tertiary review.
IC2: include only articles fully available through the queried databases or manually added, using the University of Twente academic access.
Motivation: fully available papers are needed to summarize the papers properly.
IC3: include only articles written in English.
Motivation: as the research is performed in English, only English-written articles are taken into account.
IC4: include only scientific peer-reviewed papers.
Motivation: Scientific papers that went through a peer-review process ensure a level of credibility and quality.
The initial selection step included abstract and title screening. The first author read the remaining 197 articles after duplicate removal, and the fourth and fifth authors critically evaluated this process. After the title and abstract screening, 51 articles were selected. IC1 provided the foundation for the exclusion of 146 articles. One major discovery of the title and abstract screening is the significant number of articles that contain the relevant keywords. However, upon closer examination, these articles did not substantively address the concepts in question or contribute novel methodologies to the field.
3.3.3. Step 3: Quality Assessment
To further strengthen the screening process, we defined quality criteria. It is possible that an article may inadvertently be included after the initial title and abstract screening, even if it is not completely relevant to our research focus. This could occur when the article employs certain terminology from our search query in a manner that is primarily attention-grabbing, thereby giving an impression of relevance that may not be entirely accurate. Additionally, there are cases wherein an article discusses the topic of explainable artificial intelligence (XAI) but does not engage in a detailed exploration of specific XAI methodologies. We adopted these two considerations as key aspects of our quality assessment criteria. From the 51 articles, 11 were excluded via the quality assessment. An overview of the excluded papers with their exclusion reasoning can be found in
Appendix A.
3.3.4. Step 4: Backwards Snowballing
To mitigate the risk of missing important articles, we used the backwards snowballing technique (see [
12]). This technique allowed us to add articles found in the reference list of included articles when they matched our inclusion criteria. In total, we identified 27 additional articles by performing this technique. Due to time constraints, these articles were not explicitly read in full but instead were added in
Appendix B. However, these papers are already included at a meta level since they are referenced by the papers that were already included in this review. Therefore, we have not used these papers explicitly in our results.
5. Discussion
This section will discuss possible threats to the validity of this tertiary review. Furthermore, we provide some recommendations for future tertiary reviews based on limitations that we encountered.
5.1. Future Research Directions
This review aims to provide a comprehensive overview of the current state of research in explainable artificial intelligence (XAI), identifying key areas ripe for further investigation. Note that we have followed the matrix-based methodological approach where one can assume that “If an XAI category on the vertical axis has not been matched with a set of variables on the horizontal axis, it indicates a potential research gap to explore—unless it is technically infeasible”, which may lead to the limited conclusion. Here, we would like to highlight that the proposed combinations are not exhaustive, but they are a starting point for an in-depth exploration direction.
The gaps outlined in the tables of
Section 4.4 guide our proposed research directions. From the research grid presented in
Section 4.4, one can easily identify 14 open research directions and an equal number of combinations that are unfeasible for research.
We have used 12 XAI-type categorizations that span the current XAI research landscape. Future research should explore whether the given XAI type categorization is complete. Since XAI is rapidly evolving, one can expect that the list of XAI type categories will enlarge in the future. One noticeable opportunity for progress is developing visualization methods that are accessible to non-experts. This need is echoed in the works of [
46,
47], emphasizing the potential for making complex AI models more understandable to a broader audience.
We observe a notable research gap in the utilization of various data types, particularly time-series data, point cloud data, and other dynamic data forms. Time-series data, critical in fields like finance, healthcare, and environmental studies, presents unique challenges and opportunities for XAI. This is complemented by the emerging relevance of point cloud data in sectors like autonomous vehicles and railway digital twins [
48,
49]. Time-series data are currently actively researched (see, e.g., [
43,
50]), while point cloud data still need more attention, though there are some attempt in this direction [
51,
52]. Expanding XAI research to include these diverse data types, alongside unsupervised and semi-supervised learning techniques, will broaden the scope and applicability of XAI methods.
An important aspect, as highlighted by [
53,
54], is the prevalent use of model-agnostic methods that create local surrogate models. These methods need to be refined to more accurately reflect the intricacies of the original “black box” models they are interpreting. Improving the accuracy and reliability of these model-agnostic methods, especially in their treatment of locality, is essential for the development of more transparent AI systems.
A pivotal direction for future research is enhancing the capabilities of existing XAI methods. For instance, addressing the computational limitations of widely used techniques like SHAP [
55] is crucial. Research should focus on making these methods more computationally efficient and applicable for real-world industry scenarios. This enhancement will ensure that existing techniques remain relevant and useful in rapidly evolving AI landscapes.
Despite theoretical advancements, a gap exists in the practical application of XAI methods in industry. Future research must bridge this gap by refining XAI techniques to suit diverse industrial needs, considering aspects such as computational efficiency, usability, and scalability.
While our review successfully navigates around the issues of double counting in tertiary reviews, it primarily offers a binary overview of covered and uncovered topics in XAI. An extension of this work through a quantitative analysis of the volume of research in each identified gap could offer a more comprehensive understanding of the distribution and depth of current research efforts in XAI.
5.2. Threats to Validity and Limitations
To perform a tertiary review on a field that has existed for roughly six years, various assumptions are required, and certain limitations apply.
To conduct a comprehensive tertiary review, the choice of search query terms played a crucial role. Initially, “meta-analysis review” was employed as the primary term for our queries. However, upon reflection, the term “meta-review” may have been more inclusive, potentially capturing a broader spectrum of existing literature. While this alternative could have also increased the number of false positives, such instances could have been effectively filtered out during the title and abstract screening stage. Adopting this approach may have resulted in the inclusion of more relevant articles, thereby enriching our review.
A significant limitation we encountered was restricted access to some of the existing literature. To counter this, we utilized the institutional access provided by the University of Twente, which facilitated the collection of literature beyond what is openly accessible. Despite these efforts, certain papers remained inaccessible in their entirety, leading to their exclusion from our review. This restriction may have consequently narrowed the scope of our analysis, impacting the comprehensiveness of our findings.
Conducting a tertiary review inherently involves reliance on systematic literature reviews of primary research, which adds a layer of abstraction to our conclusions. This method means that our insights are indirectly shaped by the depth and interpretations presented in these secondary sources. Such a reliance could introduce variations or potential misinterpretations in our analysis, stemming from the methodologies, interpretations, and selection bias used in both the primary studies and the systematic literature reviews.
5.3. Limitations
The presented methods and techniques are extracted from the published systematic literature review; therefore, current state-of-the-art techniques could be missed from our analysis. For example, the recent proposal that combines LIME and evolutionary algorithms [
56] is not included in our analysis because it has not yet been reported in any systematic literature review.
Another limitation of our work is an absence of certain combinations in the matrices. Due to the rapidly evolving nature of XAI, certain combinations may have been missed for reasons such as their absence in the selected SLRs or because they were recently reported in the literature and are therefore not part of the selected SLRs. Some examples are already reported in
Section 4.4, albeit not exhaustively. Therefore, possible research gaps should be taken with caution.
We would like to emphasize an inherent limitation of tertiary reports, namely selection bias (see, e.g., [
57]), as they tend to include only studies that meet specific criteria or report positive outcomes. Since a tertiary review is a meta study that extracts knowledge from systematic reviews, the selection bias is potentially enhanced. This often results in missing details from primary studies and an over-representation of successful or widely accepted methods while excluding those that were less effective or industry-oriented.
In this paper, we consciously refrained from describing XAI methods in detail since that would add more volume to the paper and there are other sources that describe them much better. For a brief description of the most well-known techniques, we refer to the chapter [
58] and a detailed description can be found in the wonderful book by Molnar [
27].
6. Conclusions
This comprehensive tertiary review systematically synthesized XAI methodologies, distilling key characteristics and categorizing them into a grid framework. Our analysis of 40 systematic literature reviews from 1992 to 2023 has revealed significant gaps in the field, particularly in under-researched areas of XAI characteristics and categories. We identified 14 open research directions and a similar number for research directions that are unfeasible to research. These findings underscore the necessity for targeted research to bridge these gaps, enriching the body of knowledge in XAI. Also, it emphasizes the need to further refine the existing methods and develop new techniques for the other underdeveloped areas in the XAI landscape. Furthermore, this study highlights the diverse nature of XAI methods, ranging from intrinsic to post hoc explainability. The implications of our findings are far-reaching, offering a road map for future research and development in XAI, which is crucial for the advancement of transparent, accountable, and ethical AI systems. While our study provides a foundational understanding of the current state of XAI research, it also acknowledges its limitations, including potential selection biases and the scope of the literature reviewed. This work serves as a call to action for the research community to delve deeper into the unexplored territories of XAI, fostering innovation and progress in this vital field.
In conclusion, the future of XAI research lies in its expansion to unexplored domains, diversification of data types and methodologies, and the bridging of the gap between theoretical research and practical, industry-oriented applications. This directional shift will not only enrich the field of XAI but also ensure its relevance and applicability in solving real-world problems.