Recent Applications of Explainable AI (XAI): A Systematic Literature Review

Abstract

This systematic literature review employs the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) methodology to investigate recent applications of explainable AI (XAI) over the past three years. From an initial pool of 664 articles identified through the Web of Science database, 512 peer-reviewed journal articles met the inclusion criteria—namely, being recent, high-quality XAI application articles published in English—and were analyzed in detail. Both qualitative and quantitative statistical techniques were used to analyze the identified articles: qualitatively by summarizing the characteristics of the included studies based on predefined codes, and quantitatively through statistical analysis of the data. These articles were categorized according to their application domains, techniques, and evaluation methods. Health-related applications were particularly prevalent, with a strong focus on cancer diagnosis, COVID-19 management, and medical imaging. Other significant areas of application included environmental and agricultural management, industrial optimization, cybersecurity, finance, transportation, and entertainment. Additionally, emerging applications in law, education, and social care highlight XAI’s expanding impact. The review reveals a predominant use of local explanation methods, particularly SHAP and LIME, with SHAP being favored for its stability and mathematical guarantees. However, a critical gap in the evaluation of XAI results is identified, as most studies rely on anecdotal evidence or expert opinion rather than robust quantitative metrics. This underscores the urgent need for standardized evaluation frameworks to ensure the reliability and effectiveness of XAI applications. Future research should focus on developing comprehensive evaluation standards and improving the interpretability and stability of explanations. These advancements are essential for addressing the diverse demands of various application domains while ensuring trust and transparency in AI systems.

1. Introduction

In recent decades, there has been a rapid surge in the development and widespread utilization of artificial intelligence (AI) and Machine Learning (ML). The complexity and scale of these models have expanded in pursuit of improved predictive capabilities. However, there is growing scrutiny directed towards the sole emphasis on model performance. This approach often results in the creation of opaque, large-scale models, making it challenging for users to assess, comprehend, and potentially rectify the system’s decisions. Consequently, there is a pressing need for interpretable and explainable AI (XAI), which aims to enhance the comprehensibility of AI systems and their outputs for humans. The advent of deep learning over the past decade has intensified efforts to devise methodologies for elucidating and interpreting these opaque systems [1,2,3].

The literature on XAI is highly diverse, spanning multiple (sub-)disciplines [4], and has been growing at an exponential rate [5]. While numerous reviews have been published on XAI in general [1,2,5], there is a noticeable gap when it comes to in-depth analyses focused specifically on XAI applications. Existing reviews predominantly explore foundational concepts and theoretical advancements, but only a few concentrate on how XAI is being applied across different domains. Although a few reviews on XAI applications do exist [6,7,8], they have limitations in terms of the coverage period and the number of articles reviewed. For instance, Hu et al. [6] published their review in 2021, thus excluding any articles published thereafter. Additionally, they do not specify the total number of articles reviewed, and their reference list includes only 70 articles. Similarly, Islam et al. [7] and Saranya and Subhashini [8] reviewed 137 and 91 articles, respectively, but also focused on earlier periods, leaving a gap in the literature regarding the latest XAI applications.

In contrast, our review fills this gap by providing a more comprehensive and up-to-date synthesis of XAI applications, analyzing a significantly larger set of 512 recent articles. Each article was thoroughly reviewed and categorized according to predefined codes, enabling a systematic and detailed examination of current trends and developments in XAI applications. This broader scope not only captures the latest advancements but also offers a more thorough and nuanced overview than previous reviews, making it a valuable resource for understanding the current landscape of XAI applications.

Given the rapid advancements and diverse applications of XAI, our research focuses on addressing the following key questions:

• Domains: what are the most common domains of recent XAI applications, and what are emerging XAI domains?
• Techniques: Which XAI techniques are utilized? How do these techniques vary based on the type of data used, and in what forms are the explanations presented?
• Evaluation: How is explainability measured? Are specific metrics or evaluation methods employed?

The remainder of this review is structured as follows: In Section 2, we provide a brief overview of XAI taxonomies. Section 3 details the process used to identify relevant recent XAI application articles, along with our coding and review procedures. Section 4 presents the findings, highlighting the most common and emerging XAI application domains, the techniques employed based on data type, and a summary of how the different XAI explanations were evaluated. Finally, in Section 5, we discuss our findings in the context of our research questions and suggest directions for future research.

2. Background: XAI Taxonomies

The primary focus of this review is on the recent applications of XAI across various domains. However, to fully appreciate how XAI has been implemented in these areas, it is essential to provide a brief overview of the key taxonomies of XAI methods. While an exhaustive discussion of these taxonomies, along with the advantages and disadvantages of each method, lies beyond the scope of this article, a concise summary is necessary to ensure that the content and findings of this review are accessible to a broad audience. For those seeking a more comprehensive exploration of XAI taxonomies and detailed discussions on the pros and cons of various XAI methods, we recommend consulting recent reviews [5,9,10,11] and comprehensive books on the subject [12,13].

Generally, XAI methods can be categorized based on their explanation mechanisms, which may rely on examples [14,15,16], counterfactuals [17], hidden semantics [18], rules [19,20,21], or features/attributions/saliency [22,23,24,25]. Among these, feature importances are the most common explanation for classification models [26]. Feature importances leverage scoring and ranking of features to quantify and enhance the interpretability of a model, thereby explaining its behavior [27]. In cases where the model is trained on images, leading to features representing super pixels, methods such as saliency maps or pixel attribution are employed. Evaluating the saliency of features aids in ranking their explanatory power, applicable for both feature selection and post-hoc explainability [5,28,29].

Other approaches to categorizing XAI methods are related to the techniques applied, such as (i) ante-hoc versus post-hoc, (ii) global versus local, and (iii) model-specific versus model-agnostic (see Figure 1). Ante-hoc/intrinsic XAI methods encompass techniques that are inherently transparent, often due to their simplistic structures, such as linear regression models. Conversely, post-hoc methods elucidate a model’s reasoning retrospectively, following its training phase [5,26,30]. Moreover, distinctions are made between local and global explanations: while modular global explanations provide an overarching interpretation of the entire model, addressing it comprehensively, local explanations elucidate specific observations, such as individual images [31,32]. Furthermore, explanation techniques may be categorized as model-specific, relying on aspects of the particular model, or model-agnostic, applicable across diverse models [5,33]. Model-agnostic techniques can be further categorized into perturbation- or occlusion-based versus gradient-based. Techniques like occlusion- or perturbation-based methods manipulate sections of input features or images to generate explanations, while gradient-based methods compute the gradient of prediction (or classification score) concerning input features [34].

As with machine learning models themselves, there is no universally best XAI approach; the optimal technique depends on factors such as the nature of the data, the specific application, and the characteristics of the underlying AI model. For instance, local explanations are particularly useful when seeking insights into specific instances, such as identifying the reasons behind false positives in a model’s predictions [35]. In cases where the AI model is inherently complex, post-hoc techniques may be necessary to provide explanations, with some methods, like those relying on gradients, being applicable only to specific models, such as neural networks with differentiable layers [34,36]. While a variety of XAI methods are available, evaluating their effectiveness remains a less-explored area [4,11]. As illustrated in Figure 1, XAI evaluation approaches can be categorized into consultations with human experts, anecdotal evidence, and quantitative metrics.

As explained above, our review extends existing work on XAI methods and taxonomies [5,9,10,11] by shifting the focus towards the practical applications of XAI across various domains. In the next section, we will describe how we used the categorizations in Figure 1 to classify the recent XAI application papers in our review.

3. Research Methodology

Based on the research questions posed in Section 1 and the different taxonomies of XAI described in Section 2, we initiated our systematic review on recent applications of XAI. To collect the relevant publications for this review, we followed the analytical protocol of the Preferred Reporting Items for Systematic Reviews and Meta-Analysis (PRISMA) guidelines [37]. A systematic review “is a review of a clearly formulated question that uses systematic and explicit methods to identify, select, and critically appraise relevant research, and to collect and analyze data from the studies that are included in the review” [12]. According to the PRISMA guidelines, our evaluation consisted of several stages: defining eligibility criteria, defining information sources, presenting the search strategy, specifying the selection process, data collection process, data item selection, studying the risk of bias assessment, specifying effect measures, describing the synthesis methods, reporting bias, and certainty assessment [37].

Information sources and search strategy: The search was conducted in February 2024 on Web of Science (WoS) by using the following Boolean search string on the paper topic (note that searches for topic terms in WoS search the following fields within a record: Title, Abstract, Author Keywords, Keywords Plus): TS = ((“explainable artificial intelligence” OR XAI) AND (application* OR process*)). The asterisk (*) at the end of a keyword ensures the inclusion of the term in both singular and plural forms and its derivatives. The search was limited to English-language non-review articles published between 1 January 2021 and 20 February 2024 (the search results can be found here: https://www.webofscience.com/wos/woscc/summary/495b659d-8f9e-4b77-8671-2fac26682231-cda1ce8b/relevance/1, accessed on 24 September 2024). We exclusively used WoS due to its authoritative status and comprehensive coverage. Birkle et al. (2020) [38] highlight WoS as the world’s oldest and most widely used research database, ensuring reliable and high-quality data. Its extensive discipline coverage and advanced citation indexing make it ideal for identifying influential works and mapping research trends [38].

Eligibility criteria and selection process: The literature selection process flow chart is summarized in Figure 2. The database search produced 664 papers. After removing non-English articles (n = 4), 660 were eligible for the full-text review and screening. During the full-text screening, we implemented the inclusion and exclusion criteria (

Table 1. Inclusion and exclusion criteria for the review of recent applications of XAI.

Criterion	Included	Excluded
Language	English	Other languages, such as German, Chinese, and Spanish.
Publication type	Peer-reviewed journal articles	Book chapters, conference papers, magazine articles, reports, theses, and other gray literature.
Recentness	Recent papers published in 2021 or after	Papers published before 2021.
Study content	Application of XAI methods	Papers that generally described XAI or reviewed other works without describing any XAI applications.
Quality	Papers of sufficient quality	Papers that were exceptionally short (less than six pages) or those that did not fulfill the basic requirements for a publication channel (e.g., be peer-reviewed, have an international board [35]).

) established through iterative discussions among the two authors. The reviewers assessed each article under the inclusion and exclusion criteria, with 512 research articles meeting the inclusion criteria and being incorporated into the evaluation procedure.

As reported in Figure 2, five articles were not retrievable from our universities’ networks, and 143 were excluded because they did not meet our inclusion criteria (primarily because they introduced general XAI taxonomies or new methods without describing specific XAI applications). Consequently, 512 articles remained for data extraction and synthesis. For reasons of reproducibility, the entire list of included articles is attached in

Table A1. Included articles in our corpus of recent applications of XAI articles, their application, and the reasons why the authors argue that explainability is important in their application.

Authors & Year	XAI Application	Why Explainability Is Important
Li et al. (2023) [94]	Face mask detection	To verify the models predictions.
Zhang et al. (2022) [65]	Diagnosis and surgery	The key to AI deployment in the clinical environment is not the model’s accuracy but the explainability of the AI model. Medical AI applications should be explained before being accepted and integrated into the medical practice.
Hilal et al. (2022) [121]	Detect and explain Autism Spectral Disorder	For explaining the logic behind decisions, describe the strengths and weaknesses of decision-making and offer insights about the upcoming behaviors.
Manoharan et al. (2023) [185]	Decision-making in banking and finance sector applications	To assure transparency in banking and finance and to assure that in case of deception occurrence, the individual will be identified clearly in the sector.
Rjoub et al. (2023) [175]	Cybersecurity	To better understand the behavior of cyber threats and to design more effective defenses.
Astolfi et al. (2023) [244]	Wind turbine maintenance	Explainability increases transparency and trustworthiness.
Berger (2023) [186]	Asset pricing	Economic data is noisy, and there are many correlations. Explainability increases understanding of economically relevant variables and correlations.
Alqaralleh et al. (2022) [170]	Intrusion detection	Explainability increases transparency for the user and gives more insight into the decisions/recommendations made by the intrusion detection system.
Neghawi et al. (2023) [272]	Evaluating performance of SSML	Machine learning models are becoming more and more difficult, and explainability is needed to be able to evaluate questionable outcomes of the models.
Meskauskas et al. (2022) [302]	Risk assessment	Traceability of the decision the model makes increases the credibility of the model and can be achieved by implementing explainability techniques on the model.
Fouladgar et al. (2022) [242]	Sensitivity of XAI models on time series data	ML models that process time series data are often quite complex (due to the nature of time series data), and so explainability would increase usability of time series data in ML.
Jean-Quartier et al. (2023) [245]	Tracking emissions of ML algorithms	ML models consume a lot of energy, and XAI implementations can reduce the amount of energy needed to get the wanted outcomes of utilizing a ML model.
Almohimeed et al. (2023) [280]	Cancer prediction	Explainability is added to increase efficiency and reliability.
Leem et al. (2023) [303]	Box office analysis	They say explainability here is essential for stakeholders in the film industry to gain insights into the model’s decision-making process, assess its reliability, and make informed decisions about film production, marketing, and distribution strategies.
Ayoub et al. (2023) [304]	Lightpath quality of transmission estimation	Lack of explainability is hindering the deployment of ML systems because the results cannot be interpreted by domain experts. With a better understanding of the model’s decision-making process, domain experts can evaluate decisions and make better choices when designing a network.
Bhambra et al. (2022) [249]	Image processing in astronomy	Explainability would give information on what parts of a picture of a galaxy are important for classification for the CNN used for the task.
Arrotta et al. (2022) [243]	Sensor-based activity recognition	They mentioned that while heat maps generated by the model may be informative for data scientists, they are poorly understandable by non-expert users. Therefore, the inclusion of a module to transform heat maps into sentences in natural language was deemed necessary to enhance interpretability for a wider audience. Additionally, providing explanations in natural language targeted towards non-expert users was highlighted as a key aspect of their work to ensure that the rationale behind the classification decisions made by the model could be easily understood and trusted by individuals without a deep technical background.
Jena et al. (2023) [130]	Earthquake spatial probability assessment (predicting where the earthquake hits)	Getting an explanation of why the ML model predicts an earthquake enables interpretation and evaluation based on expertise and knowledge of the area, and therefore one can judge if the model performs and/or if there is actually a risk of an earthquake.
Alshehri et al. (2023) [131]	Groundwater quality monitoring	Explainability techniques provide valuable insights on ML-made decisions, which is valuable for decision-making in water quality management. When important features are known, decisions can be made to focus on getting them better first.
Kim et al. (2022) [197]	Safety of self-driving cars	Mistakes made by self-driving cars can lead to dangerous accidents. Explainability gives insight into why models make mistakes and, therefore, leads to better development and safer cars.
Raval et al. (2023) [187]	Predicting credit card fraud	Explanations on model predictions help users understand significant features that the LSTM model predicts credit card fraud with.
Lim et al. (2022) [179]	Detecting deepfake voice	The authors highlighted the explainability of deepfake voice detection to ensure the system’s reliability and trustworthiness by allowing users to understand and trust its decisions. They aimed to deliver interpretations at a human perception level, making the results comprehensible for non-experts. This approach differentiates human and deepfake voices, improving the system’s effectiveness.
Vieira et al. (2023) [122]	Detection of epileptic seizures	Applying explainability methods in ML models used in the healthcare field is important because otherwise practitioners cannot understand the reasons behind decisions made by ML models.
Jena et al. (2023) [129]	Predicting an earthquake	Earthquakes can lead to significant financial losses and casualties, and that is why ML models for earthquake prediction are developed. Because ML models get more complex, explainability is needed to interpret the results and to design better models.
Youness et al. (2023) [153]	Prognostic lifetime estimation of turbofan engines	In system prognostics and health management, explainability is needed to increase the reliability of decisions made by remaining useful lifetime prediction models and also to gain knowledge on what parts caused the engine to fail. Increasing the reliability of remaining useful lifetime models is important because too early maintenance is a useless cost, and too late maintenance results in unexpected downtime, which is also a useless cost.
Ornek et al. (2021) [67]	Detecting health status of neonates	In healthcare, doctors need explanations of the ML model’s decisions so they can make the right decisions in patient care.
Sarp et al. (2021) [69]	Chronic wound classification	The authors highlight that explainability is essential because it builds trust and transparency in healthcare and supports doctors’ decision-making through visual cues like heatmaps. With the help of this method, AI decisions are more understandable for non-experts and can provide unexpected insights, improving wound management and treatment outcomes.
Hanchate et al. (2023) [158]	Average surface roughness prediction in smart grinding process	Grinding is a part of the process of manufacturing devices and machines and their parts in many (critical) fields. Post-process quality control can be long and costly, and so quality control is shifting towards in-line processes. In-line quality control is often achieved with ML methods, and explainability gives important insight into key features.
Aguilar et al. (2023) [305]	Interpretable ML model for (general) anomality detection	In detecting anomalies in sensitive fields (like healthcare and cyber security), it is important that decisions made by ML models are interpretable, because actions based on those decisions can cause serious harm if they are wrong.
del Castillo Torres et al. (2023) [306]	Facial recognition	Modern ML models are quite good at facial recognition but give no insight in their decision-making process. In facial recognition, explainability is needed to gain confidence in ML methods and their solutions.
Wang et al. (2023) [70]	Age-related macular degeneration detection	The authors mentioned that explainability is essential for improving the robustness, performance, and clinical adoption of AI-based models in medical applications, especially when it comes to tasks like AMD detection.
Dewi et al. (2023) [307]	Image captioning	The authors highlight that explainability supports technical validation and model improvement. As well as ensuring the AI system can be trusted and effectively utilized, particularly in assistive technologies for visually impaired individuals.
Ghnemat et al. (2023) [52]	Detecting COVID-19 with medical imaging	Medical devices use actual rather than synthetic Legal regulations can hinder the use of ML models in medical imaging because the models are not interpretable. Interpretability increases available use cases for ML in medical imaging because of this issue. Explainability also increases fairness in diagnostic work because practitioners are able to evaluate models’s decisions. Explainability with a good ML model can also give new information about illnesses (COVID in this case).
Martinez et al. (2023) [188]	Predicting decision to approve or reject a loan	Financial institutes use an increasing amount of AI in bank loan decision-making processes, and these decisions can affect the loan applicants significantly. Explainability is needed to evaluate AI-based decisions and to improve the models.
Younisse et al. (2022) [171]	Intrusion detection	Explainability adds reliance and trust towards ML systems. Explainability can also shift the focus on decision-making from humans to AI. Trust and reliability are as important in intrusion detection as efficiency.
Chelgani et al. (2023) [155]	Modeling hydrocyclone performance	Modeling and AI are crucial to determining hypercyclone operational variables and their impact on particle sizes. Explainability techniques applied to AI methods can help to gain insight on the sensitivities of industrial modeling (hypercyclone processes in this case).
Rietberg et al. (2023) [66]	Identifying reasons for taking MRI scan from MS (multiple sclerosis) patient	It is constant demand in the healthcare field to make processes less costly while ensuring the quality of patient care doesn’t drop, and AI can help reduce costs by increasing efficiency. Explainability can make AI more trustworthy, and it is also crucial that medical professionals know the reasons behind AI-made decisions (patient health is on the line).
Martins et al. (2024) [189]	Credit related problems, fraud detection, risk assessment, investment decisions, algorithmic trading and other financial decision-making processes	The authors did not directly state why explainability is important for their specific application. But as a summary, they highlighted that explainability is critical for ensuring transparency, trust, and informed decision-making in the financial domain.
Diaz et al. (2022) [206]	Churn prediction	Explainability with predictive AI methods can give more insight on important factors that lead to churn. When it is known why valuable customers churn, decisions can be made to avoid that.
Lohaj et al. (2023) [53]	Predicting the need of ICU on COVID-19 patients	COVID-19 is a quickly evolving disease, and all the features influencing the course of the illness are not understood. With the help of AI and XAI methods, more understanding of COVID-19 illness can be gained.
Geetha et al. (2022) [163]	Identification of concrete cracks	The authors mentioned that the explainability can address the “black box” nature of the deep learning models that they are using in their domain. They are aiming to generate high-quality, interpretable explanations of the decisions for concrete crack detection and classification.
Clare et al. (2022) [132]	Predicting ocean circulation regimes	Explainability would enable better decision-making based on AI-based knowledge on climate change because good decisions cannot be made based on uncertain knowledge, and wrong decisions can have wide-ranging impacts. For example, decisions made on the coasts where sea level rise can cause great damage need to be based on interpretable knowledge to ensure safety.
Zhang et al. (2023) [89]	Medical text processing	The authors highlighted that for practical usage in the healthcare context, AI models must be able to explain things to people. And also, understanding how those models work is essential for adopting and using medical AI applications. And also, they mentioned explainability is necessary to ensure the acceptability of AI in medicine and to use in clinical applications.
Ramon et al. (2021) [212]	Classifying psychological traits from digital footprints	Professionals using the AI-made decisions need explainability to trust the models. EU and GDPR also require certain levels of interpretability from ML models (especially on applications in critical areas). Interpretability would increase understanding on the issue that ML is used for and also reveal relations that wouldn’t have been found otherwise.
Alkhalaf et al. (2023) [308]	Cancer diagnosis	They mentioned that this helps doctors and patients understand the reasons behind the automated diagnosis made by the ML models. And also, they say, experts can provide better medical interpretations of the diagnosis and give suitable treatment options using this explainability. They also mentioned this can build trust between patients, medical staff, and AI in the medical field.
Noh et al. (2023) [164]	Intelligent system fault diagnosis of the robotic strain wave gear reducer	It is hard to convince stakeholders and engineers of ML models usability if they are not interpretable.
Chen et al. (2023) [281]	Interpretation of ML results from image data	Use cases of AI are increasing, also in fields that use a lot of image-based AI (medical fields, for example). LIME is usually used for text data or numerical data. Methods for applying LIME to image data would increase the use cases of explainability methods for image processing AI.
Nunez et al. (2023) [133]	Water resources management (snowmelt-driven streamflow prediction)	The authors mentioned that the explainability contributes to a greater understanding of hydrological processes and ensures the trust and transparency of the models and decision-making processes used in this context.
Chowdhury et al. (2023) [154]	Fault prediction on 3D printer	Models can sometimes make decisions based on wrong or irrelevant information. Interpretability increases trust because the user can evaluate if the ML-made decision makes sense.
Shah et al. (2023) [223]	Detecting AI-generated text	AI is nowadays very good at producing text that seems human-made, and detection techniques are needed to ensure safety and prevent identity theft. Interpretation techniques give more insight and help evaluate detection models’ decisions, which increases the benefits of detection model usage.
Kolevatova et al. (2021) [134]	Analysis of the impact of land cover changes on climate	The authors mentioned that explainability helps to understand the complex relationships between land cover changes and temperature changes.
Mehta et al. (2022) [296]	Social media analysis	The authors mentioned that explainability can be used for users to understand the results and to trust the decisions of the algorithms. And also, they highlighted that explainability is essential for gaining trust from AI regulators and business partners, enabling commercially beneficial and ethically viable decision-making.
Ferretti et al. (2022) [174]	Detecting vulnerabilities in source code	Neural network models are becoming more accurate but also more complex and harder to understand. In the domain of cybersecurity, it is important to know the reasons behind AI-made decisions when it comes to source code vulnerability detection, because wrong decisions can lead to disaster.
Cha et al. (2024) [217]	Sentence embedding	In the field of natural language processing, sentence embedding models do not tend to perform very well. It is not sure how to make sentence embedding models perform better. In this study, explanations are added in the middle of the model to enhance model performance.
Veitch et al. (2021) [198]	Marine autonomous surface vehicles engineering	They mentioned that explainability is essential to enhance usability, trust, and safety in the context of decision-making, AI functionality, sensory perception, and behavior. The aim was to build trust among ASV users by providing transparent and understandable representations.
Kulasooriya et al. (2023) [162]	Predicting strength characteristics of concrete	Interpretability of ML models in the structural engineering domain is important so (1) engineers can identify reasons behind model-based decisions, (2) users and domain experts can trust more in ML-made decisions, and (3) proposed methods can be explained clearly for the non-technical community (especially without knowledge of AI).
Elkhawaga et al. (2022) [156]	Predictive process monitoring	In predictive process monitoring, stakeholders need ML-based decisions to be interpretable so they can evaluate them properly and make good business decisions with the help of AI.
Nascita et al. (2021) [309]	Mobile traffic classification	The authors mentioned that it is important to have an explanation due to the lack of interpretability of the classification models used in this context. Further, they highlighted that lack of explainability can cause untrustworthy behaviors, lack of transparency, legal and ethical issues, and especially in cybersecurity applications.
Larriva-Novo et al. (2023) [172]	Intrusion detection	It is a security threat when a technician operates an ML-based intrusion detection system without interpretability in the model or knowledge of AI. This can also lead to a lack of trust in AI and ML tools.
Andreu-Perez et al. (2021) [120]	Cognitive neuroscience development	The authors mentioned that infant fNIRS data are still quite limited, and by using XAI learning and inference mechanisms, they can overcome that limitation. And also, they mentioned that XAI provides explanations for classification in their context.
El-khawaga et al. (2022) [157]	Predictive process monitoring	The goal of predictive process monitoring is to inform stakeholders about how business processes are operating now and in the near future. When business processes are described by black-box models (which is often the case), stakeholders don’t get good explanations on ML-made decisions, which reduces trust. Interpretability is needed to increase trust and to help stakeholders make data-driven decisions.
Silva-Aravena et al. (2023) [310]	Cancer prediction	Interpretability can increase ML model usage in the field of cancer prediction, especially in more complex use cases. Explainability increases the amount of knowledge gained from ML-made decisions, which leads to better decision-making in patient care. Explainability of ML models benefits both practitioners and management.
Bjorklund et al. (2023) [311]	Explainable AI methods	Highly performing black-box AI models can be insufficient if they make predictions based on the wrong features. Interpretability gives information on a model’s decision-making process and so can lead to the development of better AI models. Interpretability is crucial in safety-critical fields (like medical) and when finding new information (like physics research).
Dobrovolskis et al. (2023) [312]	Agent development	The authors mentioned that the use of explainability can improve user experience and trust by providing clear and understandable explanations of the system’s behavior. And also, they mentioned it can lead to great acceptance and adoption by users of the systems. They highlighted that the explainability in the smart home domain is essential due to the sensitive and high-risk nature of some AI applications that are closely related to human lives, wellness, and safety.
Kamal et al. (2022) [76]	Glaucoma prediction	The authors mentioned that explainability increases the user’s confidence in the decision-making process with existing ML models that are limited to glaucoma prediction. And also, they mentioned that explainability provides convincing and coherent decisions for clinicians/medical experts and patients.
Kumar et al. (2021) [114]	Brain tumor diagnosis	Explainability increases trust towards AI systems and safety for use in the medical field. Explainability should also be measurable so the explanations can be trusted.
Pandey et al. (2023) [149]	Predicting product gas composition and total amount of gas yield	Gas production systems are complex, and black-box methods are used for product gas prediction for that reason. Production systems can fail without anyone knowing why. Explainability would increase the use of AI and increase safety and efficiency due to increased knowledge of the system.
Amoroso et al. (2023) [105]	Predicting Alzheimer’s disease	It is difficult for clinical practitioners to adopt highly developed AI systems due to their lack of interpretability. There are lots of great tools for brain disease prediction that could help diagnose illness in its early stages. Explainability would allow people with little to no knowledge of AI to use these diagnostic tools.
Tao et al. (2023) [228]	Cheating detection and player churn prediction in online games	Lack of interpretability in black-box models hinders the development of the models and use of AI in online gaming. Explainability is needed to make sure models are learning the right relations, allow practitioners to adjust the model in problematic cases, make sure that models perform the same way in an online setting, and enable easy debugging.
Stassin et al. (2024) [246]	Vision transformers	The authors mentioned that explainability in the context of Vision Transformers is essential for ensuring transparency, mitigating biases, enhancing safety, and promoting trust in AI systems.
Bobek et al. (2022) [167]	Hot-rolling process (steel industry)	In manufacturing industries, data is collected straight from the machines and processed with AI to provide information about the machines to help making decisions. Decision makers usually aren’t experts of AI, so they cannot rely fully on AI-made decisions. Explainability would make relevant decision-making easier and adopting AI in decision-making processes more worthwhile.
Mollaei et al. (2022) [85]	Functional work ability prediction	As ML models become more widely used, explainability is needed so the right decisions can be made based on AI-made decisions. There are several methods to analyze an ML model’s performance, but those don’t give insight on the decision-making process.
Lin et al. (2021) [178]	Face verification	Explainability is needed in complex ML systems that do face verification so that ML-made decisions can be trusted. False-positive results on face recognition used in security applications are a big threat to security and privacy. Interpretability increases users’ trust and helps develop better and more accurate models.
Petrauskas et al. (2021) [86]	Decision support system for the nutrition-related geriatric syndromes	They mentioned that explainability is needed for medical professionals to understand the reasoning behind the decisions made by the clinical decision support system (CDSS). And also, they mentioned this approach enables physicians to comprehend the system’s assessment errors and identify areas for improvement. And also, they mentioned CDSS’s explainability allows less experienced physicians to pay attention to nutrition-related geriatric syndromes and perform detailed examinations of nutrition-related disorders.
Sharma et al. (2023) [161]	Thermal management in manufacturing process	The authors mentioned that the explainability of their context is important to develop a highly precise model. Further, they recognized the significance of transparency and interpretability in their model, particularly in the context of predicting the thermophysical properties of nanofluids.
Torky et al. (2023) [194]	Prediction and recognition of financial crisis roots	The authors mentioned that explainability enables the interpretation of complex data patterns, allowing humans to understand and interpret the logic behind classifying patterns efficiently. This is essential for financial crisis prediction as it helps in providing evidence for financial decisions to regulators and customers, especially where the results of the AI model may be inaccurate. And also, they mentioned that this will help with financial institutions’ work.
Perl et al. (2024) [313]	Fault location in power systems	They mentioned that explainability in this domain is essential because the lack of transparency in ML models for fault location in power systems poses a significant challenge. And also, they mentioned the black box ML models make it difficult for power system experts to understand the connections between input bus measurements and the output fault classification. This can cause less trust in the model’s recommendations and makes it challenging to improve PMU placement for better fault classification.
Luo et al. (2021) [182]	Aircraft detection from synthetic aperture radar (SAR) imagery	The authors mentioned that explainability helps to address the trustworthiness of SAR image analytics. And also they mentioned that explainability helps to provide a better understanding of the DNN feature extraction effectiveness, select the optimal backbone DNN for aircraft detection, and map the detection performance.
Andresini et al. (2023) [139]	Recognition of bark beetle infested forest areas	Because bark beetles can affect forest health quickly and in large areas, AI methods are needed to aid the recognition of bark beetle infestations. Explainability in these AI methods is important to gain trust in forest managers and other non-AI-experts and remote stakeholders.
van Stein et al. (2022) [140]	Plant breeding	The authors mentioned that the explainability of this context is important because it helps to gain a deep understanding of the role of each feature (SNP) in the model’s predictions. And also, they have mentioned that providing transparency and interpretability through sensitivity analysis can enhance the reliability and applicability of genomic prediction models in real-world scenarios.
Moscato et al. (2021) [190]	Credit risk assessment	In peer-to-peer lending, lenders use P2P platforms to aid their decision-making. There platforms use complex models that are hard to interpret (especially without knowledge of AI). Explainability of AI-made decisions on P2P platforms is important to help lenders make accurate loan decisions.
Nwafor et al. (2023) [314]	Non-technical losses in electricity supply chain in sub-Saharan Africa	The goal of this study is to find if and to what extent staff-related issues impact non-technical loss of electricity in sub-Saharan Africa. To answer this research question, feature importance is necessary.
Panagoulias et al. (2023) [315]	Intelligent decision support for energy management	The authors mentioned that explainability is essential for their context because it builds user trust and ensures faster adoption rates, especially in the energy sector, where AI can provide a more sustainable future. And also, they mentioned that it is essential for providing justification for recommended actions and ensuring transparency and interpretability of the analytics results.
Rodriguez Oconitrillo et al. (2021) [214]	Detecting reasons behind judge’s decision-making process	Studying judges’ decision-making process is very sensitive because of their freedom and juridical independence. Studying judges’ behavior and decision-making is still important to help other judges when they are reviewing previous cases to help their decision-making. XAI is an important tool here because XAI techniques give insight into the reasons behind decisions.
Kim et al. (2023) [316]	Designing an XAI interface for BCI experts	The authors mentioned that the explainability is important for BCI researchers to understand the decisions made by AI models in classifying neural signals or analyzing signals based on their domain expertise.
Qaffas et al. (2023) [202]	Inventory management	The authors mentioned that providing explanations for the assignment of items to classes A, B, and C allows for a better analysis of the items, easy detection of misclassifications, improved understanding of inventory classes, and flexibility in inventory management decisions. And also they mentioned explainability helps make decisions more transparent and enhances interpretability.
Wang et al. (2023) [317]	Improving performance of XAI techniques for image classification	Use of black-box models in critical fields is increasing, and explainability is much needed to help users evaluate models’ decisions and increase trust from users. The efficiency and accuracy of modern XAI visual explanation methods (CAM, LIME) can be improved, which is the goal of this study.
Mahbooba et al. (2021) [173]	Trust management in intrusion detection systems	The authors mentioned that human experts need to understand the underlying data evidence and causal reasoning behind the decisions made by AI in their domain. Further, the network administrators can enforce security policies more effectively for identified attacks, leading to improved trust in the systems by providing explanations.
Puechmorel (2023) [318]	Creating better XAI techniques with manifolds and geometry	Data manifolds are not much researched, even when computing on manifolds can result in higher performance and the ability to compute on very high-dimensional data.
Rozanec et al. (2021) [166]	Prediction of pentane content during liquefied petroleum gas debutanization process	Explainability is important so users can understand the model’s limitations in operational use.
Heuillet et al. (2022) [288]	Explaining reinforcement learning systems	As ML models get progressively more complex, transparency is needed so the use of a black-box model can be justified. Explainability also increases trust toward AI systems from the user end. Reinforcement learning is an ML technique that is increasingly used in critical fields where the interpretability of the ML model is necessary for the end users.
Gramespacher et al. (2021) [191]	Predicting loan defaults	ML models used for credit score/loan risk prediction in finance are becoming increasingly complex, which is contrary to increasing demand for transparency from authorities. As loan default is more costly to businesses than the unaccepted possible clients, the most beneficial model might not be the same as the most optimal model; interpretability is needed to aid in developing these models.
Mohamed et al. (2022) [275]	Small-object detection	Explainability increases reliability and therefore accelerates the ML model’s approval for real-life applications.
Xue et al. (2022) [135]	Predicting spatiotemporal distributions of the lake surface temperature in the Great Lakes	There is a growing need to understand the relationships between features and predictions in black-box models. The Great Lakes area is so big (84% of North America’s surface water) that it impacts the environment very differently than “regular” lakes. Explainable AI methods are needed to understand the climate of the Great Lakes area better.
Muna et al. (2023) [181]	Attack detection on IoT infrastructures of smart cities	Complex ML models are widely used in the areas of IoT and smart cities. Lack of interpretability is a security issue because ML models are hard to develop to be more safe if it is not known how they make decisions.
Yigit et al. (2022) [63]	Detect paratuberculosis from histopathological images	To help pathologists in the diagnosis of paratuberculosis
Machlev et al. (2022) [319]	Power quality disturbances classification	XAI is important because power experts may find it hard to trust the results of such algorithms if they do not fully understand the reasons for a certain algorithm’s output.
Monteiro et al. (2023) [320]	Machine learning model surrogates	The authors mentioned that the explainability of their context is important because there is a need for AI models that balance the tradeoff between interpretability and accuracy and explain the feature relevance in complex algorithms.
Chen et al. (2022) [95]	Studying sustainability of smart technology applications in healthcare	Existing methods based on AI are not easy to understand or communicate, so explainability is needed to enhance the usability of AI systems within users. Smart technology can also be deemed unsustainable because it is not easy to implement and repair; explainability makes implementation easier due to higher trust and understanding; and repairing is easier because problems can be located more easily.
Shi et al. (2022) [321]	Finding bugs in software	Explainability can help fuzzy models in bug detection by giving explanations of which parts of code need to be searched. Explainability also gives information about false positives/negatives, which helps develop a better model.
Chen et al. (2024) [168]	Job cycle time prediction	Fuzzification of complex DNN is a difficult task. Explainability simplifies the decisions made by the DNN model, which can help in the fuzzification. It is common in the domain of manufacturing and management that complex black-box models are used without understanding of AI, and explainability would lead to more meaningful model usage and more efficient processes.
Li et al. (2023) [124]	Predicting risk of brain metastase on patients with lung cancer	More complex and accurate models are needed to predict brain metastasis because there are lots of patients at risk of developing brain tumors. More complex models are usually not interpretable, so explanations are needed so interpretability is not compromised.
Igarashi et al. (2024) [322]	The effects of secondary cavitation bubbles on the velocity of a laser-induced microjet	The authors mentioned that the explainability is important in their domain because it supports understanding the physical phenomena related to the influence of secondary cavitation bubbles on jet velocity.
Yilmazer et al. (2021) [203]	On-shelf availability monitoring	The authors mentioned that explainability is useful in their context because it provides users with an explanation about individual decisions, enabling them to manage, understand, and trust the on-self availability model. And also, they mentioned explainability allows non-experts and engineers in grocery stores to understand, trust, and manage AI applications to increase OSA. Further, they mentioned it provides transparency and understandability.
Zhang et al. (2023) [180]	Attacking ML classifier of EEG signal-based human emotion assessment system with data poisoning attack	EEG data is unstable and complex, which makes interpreting models that use this data very difficult to understand. Explainability is needed to make sure models are doing what they are supposed to do and also help develop better models. In attack detection systems, explainability is needed to identify, analyze, and explain DP attacks.
Kim et al. (2023) [146]	Urban growth modeling and prediction	The authors mentioned that the explainability in their domain strengthens the interpretive aspect of ML algorithms. And also, they mentioned XAI models are likely to increase use in urban and environmental planning fields because they effectively supplement the black box features of AI.
Ilman et al. (2022) [261]	Predicting transient and residual vibration levels	Mathematical and statistical models may not be able to find all relations between parameters and to predict accurately in complex settings. When predicting response to vibration, explainability is needed to find unknown relationships between parameters and therefore build more efficient and stable systems.
Deperlioglu et al. (2022) [77]	Glaucoma diagnosis	The authors mentioned that explainability in their specific application is important because it provides transparency and improves trust and confidence in the automated deep learning solution among medical professionals.
Bermudez et al. (2023) [195]	Risk management in insurance savings products	The authors mentioned that the explainability of their specific application is important for understanding, trust, and management of ML methods that are not directly interpretable. And also, they mentioned XAI techniques are useful for risk managers to identify patterns, gain insights, and understand the limitations and potential biases of the models, finally leading to more informed and accurate decisions for their organizations and stakeholders.
Sarp et al. (2023) [54]	COVID-19 diagnosis using chest X-ray images	The authors mentioned that the explainability of their application is important for providing insights and understanding the inner workings of the black box AI model, especially in COVID-19 diagnosis. And also they mentioned that the explainability helps non-expert end-users understand the AI model by providing explainability and transparency, which is essential for feedback and providing more information to assist doctors in decision-making.
Soto et al. (2023) [323]	Improving counterfactual explanation models	Lack of explainability can lead to using less developed AI systems, which is not optimal in any field and can lead to losses and other serious consequences. Counterfactuals is becoming an increasingly important area of research for XAI because it provides very human-like explanations (very interpretable, hard to misunderstand).
Ganguly et al. (2023) [80]	Diabetes prediction	The authors mentioned that to make ML models crystal clear and authentically explainable, they have to use explainability. And also, they mentioned that lack of explanation and transparency in AI systems in healthcare can lead to less trust from patients and healthcare providers.
Messner (2023) [259]	Improving regression model explanation techniques	Using AI systems in research in social and behavioral sciences is increasing because of the high performance of black-box models. Researchers in these areas are not usually experts of AI, and therefore using complex models without explainability can lead to mistrust towards models, misuse, or wrong conclusions. Explainability is needed to make sure quality, data-driven research can be made in all fields. Also, more research is needed to improve the interpretability of regression models because XAI research has been focused on explaining classification models.
Rudin et al. (2023) [193]	Predicting credit-risk	Complex AI models without explainability are used to accept or turn down loan applications, and lack of explainability leads to lack of fairness and transparency. When the explanation technique approximates a ML model, explanations may not always be accurate or global-consistent, which again affects the fairness.
Han et al. (2022) [324]	Competitor analysis	The authors mentioned that understanding the competitive factors and points of differentiation from the customer’s perspective is essential for product developers. And also, they mentioned that their method effectively reflects customers opinions, which is essential for understanding customer preferences and improving product competitiveness. Therefore, they mentioned the explainability of their specific application.
Jo et al. (2023) [177]	Malware detection	The fact that deep-learning-based malware detection models don’t (usually) provide explanations for their classification decisions is a cybersecurity threat. Explainability of malware detection models would increase user trust and make integrating these models into cybersecurity systems more easy and accessible (because of regulations). Both malware detection and explanation models need to constantly improve to keep up with the improvement of malware.
Quach et al. (2023) [141]	Disease detection in agriculture	The authors mentioned that the explainability in their context is important because it is essential for making in-depth assessments and ensuring reliability in practice.
Hasan et al. (2024) [325]	Productivity prediction	Productivity in production is a complex situation and is hard to model with linear ML models. More complex models would perform better, but with the cost of interpretability.
Perez-Landa et al. (2021) [183]	Social media monitoring for xenophobic content detection	The authors mentioned that the explainability is essential here because it gives the ability to understand why a tweet has been classified as xenophobic. They mentioned that tweets can affect people’s behavior, and the development of an XAI model is essential to providing a set of explainable patterns describing xenophobic posts. This method can enable experts in the application area to analyze and predict xenophobic trends effectively.
Lorente et al. (2021) [201]	Development of advanced driver-assistance systems (ADAS)	The authors mentioned that explainability is essential in this context because the level of automation is constantly increasing according to the development of AI. And also, they mentioned ADAS assists drivers in driving functions, and it is essential to know the reasons for the decisions taken. And also, they said trusted AI is the cornerstone of the confidence needed in this research area.
Raza et al. (2022) [83]	Classifying different arrythmias	When using AI-based systems for decision-making in healthcare, it is important for patient health that the model is interpretable and that practitioners can justify the model’s decisions. Both clinical healthcare practitioners and patients need to trust the AI system when AI is used for diagnostic decision-making.
Gim et al. (2024) [165]	Optimization for injection molding	The authors mentioned that the IMC features are difficult to interpret and control independently without affecting other features, and therefore the quality differences cannot be regarded as the sole response due to the change of a specific feature. To address this issue, the explainability requires and interprets the relationship between the features in the IMC and each part quality.
Varghese et al. (2023) [104]	Alzheimer’s disease classification	The authors mentioned that explainability provides insights into the features or characteristics of the model used to make predictions in the context of AD classification. And also, they mentioned it is more important for improving trust in the system and its results.
Sajjad et al. (2022) [326]	Heat rransfer optimization for nanoporous coated surfaces	The authors mentioned that explainability uncovers the most influencing surface features for the nanoporous coatings.
Aquino et al. (2023) [98]	Human activity recognition	The authors mentioned that it is essential to understand how the model makes decisions and to ensure the model’s predictions are not based on biased features.
Lee et al. (2023) [169]	Yield prediction	The authors mentioned that explainability in their specific application is important to increase the transparency of the model to improve usability. And also, they mentioned it is important to improve the decision-making process and to understand the factors influencing the semiconductor manufacturing field.
Althoff et al. (2021) [257]	Hydrological modeling and prediction	The authors mentioned that explainability is important in their context because it extends the interpretability of ML models and makes the results more understandable to humans. Also, they mentioned it is important to uncover how runoff routing is being resolved and to run black box models into glass box models.
Posada-Moreno et al. (2024) [298]	Explaining in both global and local way with same method	Global explanations are used to explain the model as a whole, but those explanations don’t provide exact information of where important features are. More research is needed to create explanation methods that give both global and local explanations, because both of those methods separately have severe limitations.
Ravi et al. (2023) [327]	Predicting hardness in alloy based on composition and condition	Lack of explainability leads to models not being used in unexplored use cases and use cases with low amounts of data. In material science/engineering, there are so many different use cases for AI that the models need to be explainable so they are trustworthy. Explainability can also help find new features about physical phenomena under experiments.
Tasci (2023) [116]	Brain tumor classification	The authors mentioned that explainability enables humans to interpret and understand the results of artificial intelligence, which is crucial in the medical field for ensuring the safety and reliability of the diagnostic solutions offered by deep learning techniques.
Sauter et al. (2022) [328]	Computational histopathology	Deep learning models can learn unwanted or wrong correlations that are not causally related to the classification task. Explainability helps detect possible biases and ensure the model performs correctly.
Laios et al. (2022) [51]	Surgical decision-making in advanced ovarian cancer	The authors mentioned that explainability is supporting explaining feature effects and interactions associated with specific threshold surgical effort. And also they mentioned surgical decision-making at cytoreductive surgery for epithelial ovarian cancer (EOC) is a complex matter, and an accurate prediction of surgical effort is required to ensure the good health and care of patients. AI applications are encountered with several challenges derived from their “black box” nature, which limits their adoption by clinicians, and that is why explainability is important.
Kalyakulina et al. (2022) [111]	Disease classification (Parkinson’s and Schizophrenia)	The authors mentioned that explainability is giving the ability to interpret and verify the decisions made by ML models that is essential for medical experts. And also, they mentioned that it helps to improve the system and understand the internal mechanics of the model, which can lead to enhancements and refinements.
Bhatia et al. (2023) [96]	Tracing food behaviors	Explainability enhances comprehension and trust, and in this use case, it can make ML-based software more comfortable to use.
Rozanec et al. (2021) [196]	Time series forecasting and anomaly detection	They mentioned that explainability in their context is important because the increasing adoption of AI demands understanding the logic beneath the forecasts to determine whether such forecasts can be trusted. And also, they mentioned understanding when and why global time series forecasting models work is essential for users to detect anomalous forecasts and comprehend the features that influence the forecast.
Huang et al. (2023) [136]	Soil moisture prediction	AI methods are becoming increasingly important for soil drought prediction due to climate change. Explainability is needed so models can be interpreted and their decisions evaluated by an end user that has knowledge of physics (and other things related).
Bandstra et al. (2023) [138]	Detection and quantification of isotopes using gamma-ray spectroscopy	AI models to detect gamma rays are used in high-stakes security situations, where explainability is necessary and crucial to avoid model-induced damage. Explainability increases trust and helps understand and evaluate model performance.
Konradi et al. (2022) [97]	Aspiration detection in flexible endoscopic evaluation of swallowing (FEES)	The authors mentioned that the lack of transparency in automated processing conflicts with the European General Data Protection Regulation (GDPR), which prohibits decisions based solely on automated processing.
Mishra et al. (2022) [226]	Solution development for analyzing and optimizing the performance of agents in a classic arcade game “Fuzzy Asteroids”	The authors mentioned that the results provided by AI models would be more acceptable by end users if there were explanations in layman’s terms associated with them.
Lysov et al. (2023) [142]	Diagnosis of plant stress	The authors mentioned that the explainability of the AI models used in the early diagnostics of plant stress using the HSI process is essential for understanding the decision-making process and the features that contribute to the diagnostic outcomes.
Yagin et al. (2023) [44]	Breast cancer prediction	The authors mentioned that explainability has the potential to advance a more comprehensive understanding of breast cancer metastasis and the identification of genomic biomarkers, and it is opening new paths for transformative advances in breast cancer research and patient care.
Dworak et al. (2022) [199]	Autonomous vehicles for object detection using LiDAR	The authors mentioned that in the domain of autonomous driving, where decisions based on ML models could impact human lives, it is essential to understand how neural networks process data and make decisions. And also, they mentioned explainability is essential for ensuring the safety and reliability of autonomous driving systems.
Thi-Minh-Trang Huynh et al. (2022) [137]	Predicting heavy metals in groundwater	When predicting heavy metals and groundwater quality, future data might be much different than historical data. Explainability is needed to gain trust from domain experts and ensure usability when utilizing models that have been trained with historical data. Relationships between heavy metals and other chemical contents are highly non-linear, so white-box models give poor results. Explainability would increase both implementation and improvement of ML models used in this use case.
Bhandari et al. (2023) [110]	Parkinson’s disease diagnosis	The authors mentioned that interpretability and explainability of predictions are essential in critical areas like healthcare, medicine, and therapeutic applications, and while ML models are effective in predicting outcomes, trust issues and transparency can be addressed through explainability.
Akyol et al. (2023) [160]	Modeling refrigeration system performance	Interpretable white-box models don’t perform well when predicting refrigeration system performance due to non-linearity in data, so black-box models are in wide use. Explainability would increase understanding of how input values, which are system components, affect the goal values.
Vijayvargiya et al. (2022) [99]	Human lower limb activity recognition	The authors mentioned that explainability is essential due to the difficulty in understanding how the classifiers predicted the actions.
Renda et al. (2022) [200]	automated vehicle networking (In the context of 6G Systems)	The authors mentioned that understanding the decisions made by AI models is essential for ensuring the safety and reliability of the automated driving systems. As well as explainability permits improving the user experience of the offered communication services by helping end users trust (by design) that in-network AI functionality issues appropriate action recommendations. And also, they mentioned explainability needs at the designing stage to perform model debugging and knowledge discovery.
Akilandeswari et al. (2022) [329]	Factory/plant location selection	The authors mentioned that the explainability in their specific application is important because it is providing insights into why the model makes certain decisions. And also, they mentioned that this transparency helps stakeholders understand the reasoning behind the model’s predictions, enabling them to make informed decisions and potentially improve the model further.
Zlahtic et al. (2023) [90]	ML model development in medicine	The authors mentioned that many prevailing ML algorithms used in medicine are often considered black box models, and this lack of transparency hinders medical experts from effectively leveraging these models in high-stakes decision-making scenarios. Therefore, explainability is needed. And also, they mentioned that by empowering white box algorithms like Data Canyons, they hope to allow medical experts to contribute their knowledge to the decision-making process and obtain clear and transparent output.
Aghaeipoor et al. (2023) [330]	Explaining DNN with fuzzy methods	Explanations given by modern XAI methods aren’t always intuitive for non-expert users to comprehend, especially when it comes to rule-based explanations. Fuzzy linguistic representations in rule explanations would increase comprehension and therefore usability of XAI methods.
Lee et al. (2021) [331]	Generating global explanations for model using unstructured data	There are little to no good methods for creating global explanations for unstructured data; for structured data, these methods exist and are valid. Visualizing globally high-level features on predictions on unstructured data in an easily interpretable way is important to gain knowledge on the model’s inner processes.
Gouverneur et al. (2023) [91]	Pain recognition	The authors mentioned that understanding the decisions made by the classifiers is essential for gaining insights into the mechanisms of pain in detail.
Hung et al. (2021) [332]	Improving image data quality at preprocessing stage	Explainability methods can be used in image classification tasks to give insight on relationships between inputs and outputs. Explainability is also needed to analyze and demonstrate the importance of the proposed method for image quality improvement.
Kamal et al. (2021) [106]	Detecting Alzheimer’s disease from MRI images and gene expression data	Black-box models are hard to interpret. Explainability is necessary to gain knowledge about the model’s decision-making process and to find possibly new features that predict and affect the appearance of Alzheimer’s disease.
Qaffas et al. (2023) [285]	Inventory management	The authors mentioned that the explainability of their specific domain is important to enhance the decision-making process by providing transparent justifications for item assignments and interpretations of obtained clusters.
Dindorf et al. (2021) [68]	Pathology-independent classifier	The authors mentioned that explainability is important in their specific application to understand why subjects were classified, including instances of misclassification, and to reduce the black box nature of the machine learning model.
Javed et al. (2023) [72]	Cognitive health assessment	The authors mentioned that explainability is providing insights into the decision-making process of ML models, particularly in the context of healthcare and cognitive health assessment. This transparency and interpretability are essential for understanding how the models identify and classify activities, especially in scenarios involving individuals with dementia or cognitive impairments.
Gramegna et al. (2021) [192]	Credit risk estimation	The authors mentioned that explainability in credit risk assessment is important to address the trade-off between predictive power and interpretability. They mentioned that new algorithms offer high accuracy but lack intelligibility with limited understanding of their inner workings. Therefore, the use of explainability provides transparency and insights into why certain outputs are generated by these models.
Wani et al. (2024) [49]	Lung cancer detection	The authors mentioned that explainability, along with interpretability and transparency, is an essential aspect of AI in healthcare.
Nguyen et al. (2023) [148]	Optimization of membraneless microfluidic fuel cells (MMFCs) for energy production	The authors mentioned that explainability is important in their specific application because the black box nature of AI optimization models reduces their credibility and hinders additional understanding of the importance and contributions of each feature in the decision-making process of these models.
Kuppa et al. (2021) [176]	XAI methods in cybersecurity	The authors mentioned that the explainability of their specific application is important because it enhances trust, gives understanding of model decisions, and addresses security concerns in the cybersecurity domain.
Iatrou et al. (2022) [143]	Prediction of nitrogen requirement in rice	The authors mentioned that the explainability of their specific application is important to provide rice growers with sound nitrogen fertilizer recommendations in precision agriculture.
Sevastjanova et al. (2021) [218]	Question classification	The authors mentioned that explainability is important to provide insights into linguistic structures and patterns. Also, they mentioned that traditional ML models are black boxes, making it challenging to extract meaningful linguistic insights.
Real et al. (2023) [92]	Drug response prediction	The authors mentioned that explainability is essential for providing insights into the underlying mechanisms of drug actions, which is critical for effective clinical decision-making and patient care.
Aghaeipoor et al. (2022) [262]	Big data preprocessing	The authors mentioned that the explainability of their specific application needs understanding and explaining the internal logic of their model. Also, they mentioned that explainability helps human users to trust sincerely, manage effectively, avoid biases, evaluate decisions, and provide more robust machine learning models.
Galli et al. (2022) [147]	Building energy performance benchmarking	The authors mentioned that understanding why a certain prediction is provided by a black-box model is essential in modern contexts where the decisions of an AI system are required to be transparent and fair, such as for certification purposes. And also they mentioned that the proposed method is providing insight about the behavior of classification models used to benchmark the energy performance of buildings and to understand the motivations behind correct and wrong classifications. This information is helpful for certification entities, technical figures, and other stakeholders involved in the decision-making process.
Kaplun et al. (2021) [45]	Cancer cell profiling	The authors mentioned that understanding the reasons behind the test results is essential to relay analyzing, retraining, or modifying the model.
Moreno-Sanchez (2023) [74]	Cardiovascular medicine	The authors mentioned that explainability is important in their specific application of heart failure survival prediction to facilitate healthcare professionals’ understanding and interpretation of the model’s outcomes.
Wongburi et al. (2022) [150]	Wastewater treatment	The authors mentioned that while complex models like RNNs offer high accuracy, they can be challenging to interpret. Therefore, explainability was crucial to understand why the algorithm made certain predictions in the context of predicting the Sludge Volume Index in a Wastewater Treatment Plant.
Obayya et al. (2022) [79]	Diabetic retinopathy grading and classification	The authors mentioned that explainability is essential in healthcare settings, especially in diagnosing diseases like diabetic retinopathy, to provide transparent and interpretable insights into the decision-making process of the AI model.
Heistrene et al. (2023) [333]	Electricity price forecasting	To identify whether or not the model prediction at a given instance is trustworthy.
Azam et al. (2023) [125]	Automating the skull stripping from brain magnetic resonance (MR) images	They need visualizations to detect/segment the brain from non-brain tissue.
Ribeiro et al. (2022) [334]	Detection of abnormal screw tightening processes	For an interactive visualization tool that provides explainable artificial intelligence (XAI) knowledge for the human operators, helping them to better identify the angle–torque regions associated with screw tightening failures.
Zinonos et al. (2022) [144]	Grape leaf diseases Identification	To visualize the decisions of CNN’s output layer.
Neupane et al. (2022) [184]	Intrusion detection/cybersecurity	For example, to enable users to locate malicious instructions.
Aslam et al. (2022) [151]	Prediction of undesirable events in oil wells	To enable surveillance engineers to interpret black box models to understand the causes of abnormalities.
Pisoni et al. (2021) [225]	Explanations for art	To provide improved accessibility to museums and cultural heritage sites.
Blomerus et al. (2022) [335]	Synthetic aperture radar target detection	To furnish the user with additional information about classification decisions.
Estivill-Castro et al. (2022) [336]	Human-in-the-loop machine learning	Humans’ (domain experts or not) tend to not trust ML systems if they do not provide explanations. Sufficient explanations are important because they show correlations between features and therefore help understand the model.
Mardian et al. (2023) [152]	Predicting drought in Canadian prairies	Droughts in prairies are becoming increasingly worse due to climate change, and droughts cause losses in agriculture. Explainable AI can give insight on what factors predict or induce drought, and with this knowledge losses can be minimized.
Park et al. (2023) [93]	Predicting drug response	ML models do not give information about feature importance, but it is important to know genomic features that affect drug response prediction. XAI is not much researched in drug response prediction, and because of these reasons, explainability is necessary and needs to be researched more in this use case.
Danilevicz et al. (2023) [145]	Plant genomics	The authors mentioned that by extracting and ranking the most relevant genomic features employed by the best performing models, they can provide insights into the interpretability of the models and the identification of important motifs for IncRNA classification. And also, they mentioned that explainability is essential for understanding the underlying mechanisms driving the classification of IncRNAs and for gaining insights into the regulatory motifs present in plant genomes.
Alfeo et al. (2022) [159]	Predictive maintenance in manufacturing systems	The authors mentioned that explainable ML provides human-understandable insights about the mechanism used by the model to produce a result, such as the contribution of each input in the prediction, and this is essential in the context of predictive maintenance.
Sargiani et al. (2022) [55]	Predicting COVID-19	Lack of explanations makes ML systems incomprehensible for medical experts. Explanations are needed to make sure the doctor can be the one that makes the final decision. Explainability can also help detect biases in models, because biases are not uncommon in COVID prediction models due to unbalanced data.
Angelotti et al. (2023) [337]	Cooperative multi-agent systems	The authors mention the importance of explainability in their specific application to shed light on why one agent is more important than another in a cooperative game setting. Also, the authors mentioned that they can provide insights into the factors that influence the achievement of a common goal within a multi-agent system by assessing the contributions of individual agents’ policies and attributes.
Jeong et al. (2022) [103]	Predicting Alzheimer’s disease dementia	Explainability is needed to make sure that ML models’ decision-making processes line up with current knowledge on Alzheimer’s disease and dementia. When a model is proven to work accurately, it can be implemented in concrete use cases.
Pereira et al. (2021) [208]	Early prediction of student performance	The authors mentioned that the explainability is important in their specific application to facilitate human AI collaboration towards perspective analytics. And also, they mentioned that by providing explanations for the predictive model decisions, they can support students, instructors, and other stakeholders in understanding why certain predictions were made.
Wickramasinghe et al. (2021) [286]	Cyber-physical systems	The authors mentioned that it is essential to understand the reasoning behind the model’s decisions in CPSs because the outcomes of machine learning models can have significant impacts on safety, security, and operations. And also they mentioned that explainable unsupervised machine learning methods are needed to enhance transparency, trust, and decision-making in CPS applications.
Bello et al. (2024) [241]	Object detection and image analysis	The authors mentioned that the explainability is essential to providing insights into the decision-making process of complex deep learning architectures.
Song et al. (2022) [253]	Predicting minimum energy pathways in chemical reaction	White-box models often fail to express complex chemical systems (like enzyme catalysis). Explainability is needed to enhance understanding of ML models and assist in responsible decision-making.
Tang et al. (2022) [338]	Safety of XAI, preventing adversarial attack on XAI system	It has been shown that explanation techniques are vulnerable to adversarial attacks; one can change explanations without changing the model outcome. Stability of explanations is important and needs to be studied to achieve safer ML/XAI systems.
Al-Sakkari et al. (2023) [339]	Carbon dioxide capture and storage (CCS)	The authors mentioned that explainability is important in their specific application to gain a better understanding of the effects of process and material properties on the variables of interest. Further, they mentioned that by adding explainability to accurate models, it can provide insights into the impact of different variables on the measured variables, enhancing the overall understanding of the system.
Iliadou et al. (2022) [100]	Predicting factors that influence hearing aid use	Automated decision-making systems that utilize AI are not easily accepted in healthcare because usually they aren’t interpretable and therefore not trustworthy. In healthcare and medical decision-making, accountability in case of a wrong decision is a serious ethical question when AI is used to make decisions. Explainability is needed to increase trust and transparency.
Kwong et al. (2022) [46]	Prostate cancer management	The authors mentioned that explainability is important not only to provide predictions but also to highlight the variables driving the predictions. Further, they mentioned that this transparency helps build trust in the model by ensuring that the predictions and explanations align with clinical intuition.
Ge et al. (2023) [297]	Cyber threat intelligence (CTI) analysis and classification	The authors mentioned that explainability is important in their application to enhance the interpretability, reliability, and effectiveness of cyber threat behavior identification through the clear delineation of key evidence and decision-making processes.
Alcauter et al. (2023) [209]	Predicting dropout in faculty of engineering	Explanations can give comprehensive insight on prediction factors. This increases trust and also enables precise actions on decreasing dropout rates in engineering studies, where dropout rates are high.
Apostolopoulos et al. (2022) [340]	Skin cancer detection and classification	The authors mentioned that explainability in their specific application is important to help in understanding and interpreting the decisions made by the model, especially in the context of diagnosing skin lesions. Further, they mentioned that by visualizing the important regions, the model’s predictions can be better understood and trusted, leading to more transparent and interpretable results in the classification of skin lesions.
Brdar et al. (2023) [254]	Pollen identification	Classifying pollen is a complicated ML task because of complex data (chemical structure of pollen, etc.), and intrinsically interpretable models often fail in performance. Explainability of black-box models’s solutions is needed to create trust towards AI systems.
Apostolopoulos et al. (2022) [340]	COVID-19 detection	The authors mentioned that explainability in their specific application is important to provide insights into the decision-making process of the deep learning model. Also, they mentioned that transparency is also essential for building trust in the model’s decisions, ensuring accountability, and enabling the actual users of the model to understand and interpret the image findings.
Henzel et al. (2021) [61]	COVID-19 data classification	The authors mentioned that explainability in their specific application is important to explain the relationship between symptoms and the predicted outcomes. And also to enhance the interpretability of the models and provide a transparent understanding of how the classifiers make decisions.
Deramgozin et al. (2023) [341]	Facial action unit detection	Explanations in systems that detect facial expressions give interpretability and deeper understanding about how the model works.
Maouche et al. (2023) [43]	Breast cancer metastasis prediction	The authors mentioned that explainability in their specific application is important to address the issue of model interpretability. Further, they mentioned that the increased complexity of the models is associated with decreased interpretability, which causes clinicians to distrust the prognosis.
Zaman et al. (2021) [263]	Control chart patterns recognition	Researchers and other users need explanations of ML-made decisions to evaluate the model and to make correct final decisions. It is important to find explainable and efficient ML systems that do not require too many resources.
Dassanayake et al. (2022) [342]	Autonomous vehicles	The authors mentioned that explainability in their specific application is important to enhance the confidence of DNN-based solutions. They mentioned that for autonomous systems operating in unpredictable environmental conditions, the rationale behind the decisions made by DNNs is essential for accountability, reliability, and transparency, specifically in safety-critical edge systems like autonomous transportation.
McFall et al. (2023) [112]	Early detection of dementia in Parkinson’s disease	The authors mentioned that explainability in their specific application is important to selectively identify and interpret early dementia risk factors in Parkinson’s disease patients.
Zhang et al. (2023) [56]	Diagnosis of COVID-19	The authors mentioned that the existing deep learning classifiers lack transparency in interpreting findings, which can limit their applications in clinical practice. Further, they mentioned that providing explainable results (like the proposed CXR-Net model to assist radiologists in screening patients with suspected COVID-19) is reducing the waiting time for clinical decisions.
Qayyum et al. (2023) [343]	Material property prediction	The authors mentioned that explainability in their specific application is important to gain insights, interpret model predictions, identify key factors influencing the outcomes, and advance material discovery in the field of PZT ceramics.
Lellep et al. (2022) [344]	Relaminarization events in wall-bounded shear flows	The authors mentioned that explainability in their specific application is important to provide a physical interpretation of the machine learning model’s output. And also, they mentioned that the interpretability is crucial for understanding the underlying physical processes driving relaminarization events and gaining insights into the dynamics of turbulent flows close to the onset of turbulence.
Bilc et al. (2022) [345]	Retinal nerve fiber layer segmentation	Standard software suffers from noisy data and unclear decision-making processes. Explainability would enable controlling the model’s learning process and also validating the results.
Sakai et al. (2022) [346]	Congenital heart disease detection	Medical professionals tend not to trust black-box models and therefore not use them. Explainability would increase use of AI systems by medical professionals and therefore enhance their performance.
Terzi et al. (2021) [347]	Credit card fraud detection	Explainability is needed to ensure fairness and ethics of ML model-made decisions and also to improve ML model performance. Credit card frauds are constantly evolving, and explainability techniques give insight on ML models and therefore can help detect new kinds of attacks.
Allen (2023) [260]	Obesity prevalence prediction	Explainability of ML models that predict obesity can help detect features that effect obesity rates the most and therefore lead to better decision-making in obesity prevention.
Kothadiya et al. (2023) [348]	Sign language recognition	The authors mentioned that explainability in their specific application is important for sign language recognition to address variability in sign gestures, facilitate communication for physically impaired individuals, and enhance user trust and understanding of the recognition model.
Slijepcevic et al. (2023) [349]	Human gait analysis in children with Cerebral Palsy	The authors mentioned that explainability in their specific application is important to promote trust and understanding of machine learning models in clinical practice, especially in the medical field where decisions impact patient care and outcomes. And also, they mentioned that by examining whether the features learned by the models are clinically relevant, explainability ensures that the decisions made by the models align with the expertise and expectations of healthcare professionals.
Hwang et al. (2021) [350]	Sensor fault detection	The authors mentioned that explainability in their specific application is important to enhance the reliability of AI models, facilitate direct response to threats, and provide comprehensive explanations for security experts to ensure the safety of Industrial Control Systems.
Rivera et al. (2023) [351]	predict the arrivals at an emergency department	Classical techniques for explaining regression models are often biased and model-specific. It is important to search for more generalizable and global explaining techniques when AI is used in increasing amounts in critical fields.
Park et al. (2023) [352]	Prediction of nitrogen oxides in diesel engines	The authors mentioned that explainability in their specific application is important to understand the influence of input features on NOx prediction. Further, they mentioned that by explaining the model’s decisions and the relationships between input and output variables, the model becomes more transparent and trustworthy in applications where prediction accuracy and feature importance are essential, such as in the automotive industry for developing low-carbon vehicles.
Abdollahi et al. (2021) [353]	Urban vegetarian mapping	The authors mentioned that explainability in their specific application is important to comprehend model decisions, to grasp complicated inherent non-linear relations, and to determine the model’s suitability for monitoring and evaluation purposes.
Xie et al. (2021) [354]	Air-traffic management	The authors mentioned that explainability in their specific application is important for transparency in decision-support systems to ensure that the AI/ML algorithms used in predicting risks in uncontrolled airspace can be understood and trusted by human operators.
Al-Hawawreh et al. (2024) [355]	Cyber-physical attacks (use case: gas pipeline system)	The authors mentioned that explainability in their specific application is important to enhance the trustworthiness of the AI models and to contribute to performance improvements, safety, audit capabilities, learning, and compliance with regulations.
Laios et al. (2023) [50]	Cancer prediction	The authors mentioned that explainability in their specific application is important for understanding the features that drive a model prediction, which can potentially aid in decision-making in complex healthcare scenarios. They also mentioned that as natural language processing moves towards deep learning, transparency becomes increasingly challenging, making explainability essential for ensuring trust and understanding in the model’s predictions.
Ramirez-Mena et al. (2023) [47]	Prostate cancer prediction	Predicting and identifying prostate cancer is difficult because of complex indicators of disease. Medical professionals would benefit from using clinical decision support systems for diagnosing prostate cancer, but they often do not use them because they are not interpretable and trustworthy. XAI is needed to increase trust and therefore allow the use of diagnostic tools for cancer prediction.
Srisuchinnawong et al. (2021) [356]	Robotics	State-of-the-art explanation techniques often fail to visualize the whole structure of a neural network (neural ingredients) and also do not support robot interface.
Dai et al. (2023) [357]	Using classical statistical analysis methods for explaining NN model	Explainability increases trust and understanding of AI systems and therefore enables AI system use in clinical settings.
Feng et al. (2022) [300]	Remote sensing image scene classification	Remote sensing image scene classification is a computationally demanding task, and deep learning methods and neural networks provide the computational accuracy and efficiency needed. Lack of explainability in those black-box methods leads to distrust towards models.
Li (2022) [358]	Ride-hailing service demand modeling	The authors mentioned that the explainability of their specific application is important to understand the processes underlying the observed data rather than solely performing predictive tasks in the context of spatial data modeling. And also, they mentioned that explainability is essential for extracting spatial relationships, visualizing them on maps, and enabling analysts to understand and interpret the spatial effects captured by the machine learning models.
Palatnik de Sousa et al. (2021) [57]	Predicting COVID-19 disease from chest X-ray image and CT-scan	Explainability is important in AI systems that are used in healthcare to ensure accuracy of models’ decisions and trust towards models. Explanations can also help detect different kinds of biases in AI systems.
Delgado-Gallegos et al. (2023) [62]	Assessment of perceived stress in healthcare professionals attending COVID-19	The authors mentioned that the explainability of their specific application is important to locate the combination of factors necessary to correctly classify healthcare professionals based on their perceived stress levels. Further, they mentioned that the decision tree model served as a graphical tool that allowed for a clearer interpretation of the factors influencing stress levels in healthcare professionals.
Gonzalez-Gonzalez et al. (2022) [359]	Industrial carbon footprint estimation	The authors mentioned that the explainability of their specific application is important to provide a human operator with an in-depth understanding of the classification process and to validate the relevant explanation terms.
Elayan et al. (2023) [360]	Power consumption prediction	The authors mentioned that the explainability of their specific application is important to ensure that users understand and trust the system’s predictions and decisions regarding power consumption. Further, they mentioned that users can gain insights into the model’s behavior, biases, and outcomes, and explainability increased transparency and user confidence in the system.
Duc Q Nguyen et al. (2022) [58]	COVID-19 forecasting	The authors mentioned that the explainability of their specific application is important and essential for building trust in the AI model, facilitating collaboration between AI systems and human experts, and ultimately improving the effectiveness of decision-making processes in managing the COVID-19 pandemic.
Cheng et al. (2022) [361]	Deep learning models used in forestry	Researchers and users are having a difficult time trying to understand black-box models that are widely used due to their high performance and efficiency. Explanations can give otherwise not found hints about how the ML model can be made better.
Gomez-Cravioto et al. (2022) [210]	Alumni income prediction	Studying alumni income and socioeconomic status can help educational institutions in improving efficiency and planning the studies, which helps future graduates. Explainability can give necessary insight on important factors that influence the success of the alumni.
Qiu et al. (2023) [362]	Biological age prediction	Linear models are used in biological age prediction because of their interpretability, but they offer low accuracy. Explainability of black-box models is needed so more efficient and accurate models can be used. Local explanations are needed so ML models can be used on individuals.
Abba et al. (2023) [363]	Water quality assessment	The authors mentioned that the explainability of their specific application is important to demonstrate the impact of individual features on the model’s predictions and supports stakeholders and decision-makers in making informed choices regarding groundwater resource management.
Martinez-Seras et al. (2023) [364]	Image classification	The authors mentioned that the explainability of their specific application is important to build the trustworthiness of machine learning models, especially with Spiking Neural Networks, and it is essential for ensuring the acceptance and adoption of these models in real-world settings.
Krupp et al. (2023) [365]	Tool life prediction	The authors mentioned that the explainability of their specific application is important to enable domain experts from the field of machining to develop, validate, and optimize remaining tool life models without extensive machine learning knowledge.
Nayebi et al. (2023) [366]	Clinical time series analysis	The authors mentioned that explainability in their specific application is important for understanding the features that drive a model prediction, which can potentially aid in decision-making in complex healthcare scenarios. They also mentioned that as natural language processing moves towards deep learning, transparency becomes increasingly challenging, making explainability essential for ensuring trust and understanding in the model’s predictions.
Lee et al. (2022) [367]	A de-identification of medical data	A de-identification system without explainability is unusable in the medical domain because of critical data. Explainability is needed to gain transparency and also assist in developing better de-identification models and modifying processes connected to de-identification.
Nahiduzzaman et al. (2023) [368]	Mulberry leaf disease classification	Mulberry is a culturally important plant in the Himalayan area, and little to no studies have been made to improve mulberry leaf disease detection. Explainability enables the use of AI systems for disease detection by mulberry farmers and enhances model development. An explainable model could also be used to detect disease from other plants’ leaves.
Khan et al. (2022) [369]	Vision-based industrial applications	The authors mentioned that the model’s decisions and predictions can be understood and interpreted by ensuring that the explainable correct annotations by the proxy model. And also, they mentioned that transparency in the decision-making process is essential for building trust in the model’s outputs in industrial applications.
Beucher et al. (2022) [370]	Detecting acid sulfate in wetland areas	Explanations would validate and clarify otherwise uninterpretable ML model decisions. Explainability is needed so that the results of research can be communicated to both expert and non-expert audiences. Explanations also help build better ML models.
Kui et al. (2022) [371]	Disease severity prediction	The authors mentioned that the explainability in their specific application is important to help physicians understand the decision-making process of the ML model.
Szandala (2023) [372]	Explaining ML model with saliency maps	AI methods cannot be implemented in critical fields without a good understanding of how models work. State-of-the-art visual XAI techniques don’t explain why important areas are important. Saliency maps with information about the selection of important areas give more comprehensive insight about the ML model’s decision-making.
Rengasamy et al. (2021) [373]	Safety critical systems	The authors mentioned that in the context of safety-critical systems, explainability is essential for ensuring transparency, accountability, and trust in the decision-making process facilitated by ML models.
Jahin et al. (2023) [374]	Supply chain management	The authors mentioned that practical implications of their specific application include improved inventory control, reduced backorders, and enhanced operational efficiency. Thus, by using explainability, it empowers the decision-making and efficient resource allocation in supply chain management systems. Further, they mentioned that this transparency and interpretability are essential for stakeholders to understand the model’s predictions and trust its recommendations.
Nielsen et al. (2023) [375]	Evaluating and explaining XAI methods	Explaining and evaluating the explanations given by XAI methods is important to ensure model robustness, faithfulness, and safety.
Hashem et al. (2023) [376]	Brain computer interface system to Analyze EEG signals	The authors mentioned that by using explainability, they can get greater transparency and understanding of the relationship between the EEG features and the model’s predictions. Further, they mentioned that this transparency is essential for enhancing the interpretability of the BCI systems in the context of controlling diverse limb motor tasks to assist individuals with limb impairments and improve their quality of life.
Lin et al. (2022) [377]	Classifying lncRNA and protein-coding transcripts	RNA data is complicated, and neural network models have shown the best performance in classification tasks, but the neural network models lack interpretability. Explainability increases understanding of the ML model’s decision-making process in the RNA classification task.
Chen et al. (2023) [378]	Land cover mapping and monitoring	The authors mentioned that the explainability of their specific application is important to enable researchers and practitioners to understand how ML models work in order to strategically improve model performance for land cover mapping with Google Earth Engine, to support fine-tuning and optimizing models, to help gauge trust in the models, and to address the lack of explainability in some parts of the scientific process.
Oveis et al. (2023) [379]	Automatic target recognition	In automatic target recognition, it is very important to know that the ML model learns to look at the right things (target) in image data because when a new kind of situation (new kind of truck/car/tank) appears, the model has to be able to recognize the target correctly.
Llorca-Schenk et al. (2023) [380]	Designing porthole aluminium extrusion dies	The authors mentioned that explainability is important in their specific application to help when deciding the best way in which to adjust an initial design to the predictive model.
Diaz et al. (2023) [381]	HR decision-making	The authors mentioned that the explainability is essential in their application of predicting employee attrition as it helps in designing effective retention and recruitment strategies as well as enhances trust, transparency, and informed decision-making in human resources management.
Pelaez-Rodriguez et al. (2023) [382]	Extreme low-visibility events prediction	To enable the consideration of interpretability, which is an extremely important additional design driver, especially in some areas where the physics of the problem plays a major role, such as geoscience and Earth observation problems.
An et al. (2023) [383]	NA	To understand how deep learning models make predictions.
Anjara et al. (2023) [48]	Oncolocy (lung cancer relapse prediction)	To improve trust and adoption of AI models.
Glick et al. (2022) [384]	Dental radiography	To assist/help novice dental clinicians (dental students) in decision-making.
Qureshi et al. (2023) [385]	Mosquito trajectory analysis	To give insights into the mechanisms that may limit mosquito breeding and disease transmission.
Kim et al. (2023) [78]	Cardiology	To reduce a high rate of false alarms in cardiac arrest prediction models and to make their results clinically (more) interpretable.
Wen et al. (2023) [386]	Alzheimer disease detection (from patient transcriptions)	To discover the underlying relationships between PoS features and AD.
Alvey et al. (2023) [387]	Aerial images analysis	To understand and explain the behavior of deep learning models.
Maaroof et al. (2023) [82]	Diabetes prediction	To gain insight into how the model makes its predictions and build trust in its decision-making process.
Hou et al. (2022) [388]	Image classification	To produce improved filters for preventing advanced backdoor attacks.
Nakagawa et al. (2021) [389]	Mortality prediction of COVID-19 patients (from healthcare data)	To allow data scientists and developers to have a holistic view, a better understanding of the explainable machine learning process, and to build trust.
Yang et al. (2022) [390]	Process execution time prediction	To explain how ML models predicting the time until the next activity in the manufacturing process works.
O’Shea et al. (2023) [391]	Lung tumor detection	Interpretability is essential for reliable convolutional neural network (CNN) image classifiers in radiological applications.
Tasnim et al. (2023) [392]	Cardiology	To examine the contribution of features to the decision-making process and to foster public confidence and trust in ML model predictions.
Marques-Silva et al. (2023) [393]	NA	NA
Lin et al. (2023) [274]	Visual reasoning	To disclose/explain the decision-making process from the numerous parameters and complex non-linear functions.
Pedraza et al. (2023) [394]	Sensor measurements	To better understand the AI model.
Kwon et al. (2023) [395]	NA	To derive a mechanism of quantifying the importance of words from the explainability score of each word in the text.
Rosenberg et al. (2023) [396]	Integer linear programming and quadratic unconstrained binary optimization	Explainability is needed so that ML models can be trusted. Explainability can also help detect biases and help improve the ML model. Expressive boolean formulas for explanations can increase flexibility and interpretability.
O’Sullivan et al. (2023) [397]	Water quality modeling	Neural network models provide good performance in simulating water quality, but because of bad explainability, the model’s decisions are hard to use to make management decisions. Explainability would increase trust and usability of ANN models in water quality modeling.
Richter et al. (2023) [398]	Radar-based target classification	The authors mentioned that the explainability is important in their specific application to obtain insights about the decision-making processes of the model and ensure the reliability and effectiveness of their system.
Khan et al. (2024) [237]	Traffic sign recognition	Traffic sign recognition systems need to be accurate and reliable, and that is why neural network models are used. They lack interpretability, which makes detecting bias and evaluating model performance difficult. Transparent and safe systems in this kind of critical application of ML models are very necessary.
Heimerl et al. (2022) [235]	Emotional facial expression recognition	The authors mentioned that the explainability is important in their specific application to fully understand the underlying process in the classification because the classification results could lead to harmful events for individuals. Further, they have mentioned that the transparency and interpretability of AI models are really important in applications involving sensitive information and safety-critical scenarios.
Dong et al. (2021) [399]	Medical image noise reduction by feature extraction	Portable ultrasound devices are cheap and very convenient, but they can give noisy images. Explainability and identifying important features are crucial for successful noise reduction with feature extraction in the medical domain.
Murala et al. (2023) [400]	Healthcare metaverse online model	Explainability is needed in online healthcare (medical metaverse) so doctors’ can have more information about patients’ status and therefore make better medical decisions. Explainability of AI-made medical decisions enhances transparency, reliability, predictability, and therefore safety, which benefits both the doctor and the patient.
Brakefield et al. (2022) [401]	Health surveillance and decision support	The authors mentioned that explainability is important in their specific application to improve decision-making capability for physicians, researchers, and health officials at both patient and community levels. Further, they mentioned that there are many existing digital health solutions that lack the ability to explain their decisions and actions to human users, which can hinder informed decision-making in public health.
Lee et al. (2021) [204]	Predicting online purchase based on information about online behaviour	End users can find it hard to trust a black-box model and therefore end up not using the model. Explainability increases trust towards models and enables reliable use of AI, which can be very beneficial in the context of online marketing.
Ortega et al. (2021) [402]	Applying inductive logic programming to explain black-box model	In some applications of AI, explanations are required by law or needed to ensure ethnicity of decision-making. Inductive logic programming systems are interpretable by design, and applying this method to classical ML models enhances their interpretability and performance.
An et al. (2022) [403]	Producing clear visual explanations to black-box models	Lack of explainability leads to users not trusting the ML model in critical applications (like healthcare, finance, and security).
De Bosscher et al. (2023) [293]	Airport terminal operations	The authors mentioned that the explainability is important in their specific application to interpret and understand these emergent properties in a more efficient way because existing airport terminal operation models have heavy computational requirements. And also, they mentioned airport terminals are involving complex systems, and explainability helps to understand the dynamics of these complex sociotechnical systems. Further, they mentioned that using explainability, they can identify opportunities for optimization and improvement in processes such as passenger flow, security checkpoints, and overall terminal efficiency.
Huang et al. (2022) [404]	Remote sensing scene classification	Explainability in computer vision tasks is important so the ML model can be trusted and used safely. State-of-the-art heatmap explanation methods (CAM-methods) can give a good explanation to a black-box model, but they are not always accurate (failing to detect multiple planes, for example).
Senocak et al. (2023) [405]	Precipitation forecast	The authors mentioned that the explainability is important in their specific application to make the machine learning models more transparent, interpretable, and aligned with domain expertise. And also to enhance the reliability and utility of the predictions.
Kalutharage et al. (2023) [406]	Anomaly detection	Explainability is important in AI applications in the field of cybersecurity because one mistake made by the model can lead to a large amount of damage. Explanations make the ML model trustworthy and also enable better model development. Detecting important features can also increase the performance of the intrusion detection system.
Sorayaie Azar et al. (2023) [407]	Monkeypox detection	The authors mentioned that the explainability is important in their specific application to provide a clear understanding of the decision-making process of the AI models. Further, they mentioned that by providing explainability, the clinicians can gain deeper insights into how the AI models arrive at their predictions, which is crucial for fostering trust and confidence in the reliability of AI systems in real-world clinical applications.
Di Stefano et al. (2023) [408]	Early diagnosis of ATTRv Amyloidosis	The authors mentioned that explaining the prediction is essential in medical domains because the patterns a model discovers may be more important than its performance.
Huong et al. (2022) [409]	Industrial Control Systems (ICS)	The authors mentioned that the explainability is important in their specific application of anomaly detection in Industrial Control Systems (ICS) because explaining the detection outcomes and providing explanations for anomaly detection results is essential for ensuring that experts can understand and trust the decisions made by the model.The authors mentioned that the explainability is important in their specific application of anomaly detection in Industrial Control Systems (ICS) because explaining the detection outcomes and providing explanations for anomaly detection results is essential for ensuring that experts can understand and trust the decisions made by the model.
Diefenbach et al. (2022) [410]	Smart living room	To understand how the technology works, what its limits are, and what consequences regarding autonomy and privacy emerge.
Gkalelis et al. (2022) [265]	Video event and action recognition	To derive explanations along the spatial and temporal dimensions for the event recognition outcome.
Patel et al. (2022) [411]	Water quality prediction	To provide the transparency of the model to evaluate the results of the model.
Mandler et al. (2023) [292]	Data-driven turbulence models	To make the prediction process of neural network-based turbulence models more transparent.
Kim et al. (2023) [412]	Cognitive load prediction	To detect important features.
Huelsmann et al. (2023) [270]	Energy system design	To better understand the influence of all design parameters on the computed energy system design.
Schroeder et al. (2023) [413]	Predictive maintenance	For creating a wide acceptance of AI models in real-world applications and aiding the identification of artifacts.
Singh et al. (2022) [414]	Bleeding detection (from streaming gastrointestinal images)	To reverse engineer the test results for the impact of features on a given test dataset.
Pianpanit et al. (2021) [113]	Parkinson’s disease (PD) recognition (from SPECT images)	For easier model interpretation in a clinical environment.
Khanna et al. (2022) [289]	Assessing AI agents	To be able to assess an AI agent.
Kumara et al. (2023) [415]	Performance prediction for deployment configurable cloud applications	To provide explanations for the prediction outcomes of valid deployment variants in terms of the deployment options.
Konforti et al. (2023) [416]	Image recognition	To explain neural network decisions and internal mechanisms.
Ullah et al. (2022) [255]	Credit card fraud detection; customer churn prediction	To improve trust and credibility in ML models.
Gaur et al. (2022) [115]	Prediction of brain tumors (from MRI images)	To realize disparities in predictive performance, to help in developing trust, and in integration into clinical practice.
Al-Hussaini et al. (2023) [123]	Seizure detection (from EEG)	To foster trust and accountability among healthcare professionals.
Oblak et al. (2023) [417]	Fingermark quality assessment	To make the models more transparent.
Sovrano et al. (2022) [219]	Questions answering (as explaining)	To make the models more transparent.
Zytek et al. (2022) [213]	Child welfare screening (risk score perdiction)	To overcome ML usability challenges, such as lack of user trust in the model, inability to reconcile human-ML disagreement, and ethical concerns about oversimplification of complex problems to a single algorithm output.
Quach et al. (2024) [247]	Tomato detection and classification	To assess model reliability.
Guarrasi et al. (2023) [59]	Prediction of the progression of COVID-19 (from images and health record)	To enable physicians to explore and understand data-driven DL-based system.
Le et al. (2021) [418]	NA	Not mentioned.
Capuozzo et al. (2022) [419]	Glioblastoma multiforme identification (from brain MRI images)	To assess the interpretability of the solution showing the best performance and thus to take a little step further toward the clinical usability of a DL-based approach for MGMT promoter detection in brain MRI.
Vo et al. (2023) [420]	Dragon-fruit ripeness (from images)	To explain the outcomes of the image classification model and thereby enhance its performance, optimization, and reliability.
Artelt et al. (2022) [421]	NA	There is no specific application.
Abeyrathna et al. (2021) [422]	NA	There is no specific application.
Krenn et al. (2021) [276]	Experimental quantum optics	The interpretable representation and enormous speed-up allow one to produce solutions that a human scientist can interpret and gain new scientific concepts from outright.
Pandiyan et al. (2023) [423]	Laser powder bed fusion process	To highlight the most relevant parts of the input data for making a prediction.
Huang et al. (2023) [222]	Assessment of familiarity ratings for domain concepts	To be able to evaluate familiarity ratings of domain concepts more in-depth and to underline the importance of focusing on domain concepts’ familiarity ratings to pinpoint helpful linguistic predictors for assessing students’ cognitive engagement during language learning or online discussions.
Jeon et al. (2023) [424]	Land use (from satellite images)	To enhance the reliability of the image analysis.
Fernandez et al. (2022) [264]	NA	There is no specific application; however, the explainability is important because of the increasing number of applications where it is advisable and even compulsory to provide an explanation.
Jia et al. (2022) [425]	WiFi fingerprint-based localization	To improve the trust of the proposed method.
Munkhdalai et al. (2022) [426]	NA	There is no specific application.
Schrills et al. (2023) [295]	Subjective information processing awareness (in automated insulin delivery)	To help users cooperate with AI systems by addressing the challenge of opacity, subjective information processing awareness (SIPA) is strongly correlated with trust and satisfaction with explanations; therefore, explanations and higher levels of transparency may improve cooperation between humans and intelligent systems.
Gouabou et al. (2022) [427]	Melanoma detection	To overcome the dermatologist’s fear of being misled by a false negative and the assimilation of CNNs to a “black box”, making their decision process difficult to understand by a non-expert.
Okazaki et al. (2022) [205]	Customer journey mapping automation (through model-level data fusion)	Trustworthiness and fairness have to be established (by using XAI) in order for the black-box AI to be used in the social systems it is meant to support.
Mridha et al. (2023) [41]	Skin cancer classification	Using provided explanations, the clinician may notice the color irregularity in the dermatoscopic picture, which is not evident on the lesion, and figure out why the classifier predicted incorrectly.
Abeyrathna et al. (2021) [428]	NA	There is no specific application.
Nagaoka et al. (2022) [429]	COVID-19 prediction (from lung CT slice images)	The Grad-CAM method has been used for the authors to be sure that their method has used only the pixels from certain locations (where lungs are) of the image used for classification; the explanation itself is not important here.
Joshi et al. (2023) [234]	Misinformation detection (specifically COVID-19 misinformation; from texts)	Knowing the reasoning behind the outcomes is essential to making the detector trustworthy.
Ali et al. (2022) [430]	COVID-19 prediction (from X-ray images)	To maintain the transparency, interpretability, and explainability of the model.
Elbagoury et al. (2023) [431]	Stroke prediction (based on EMG signals)	To support (personalized) decision-making.
Yuan et al. (2022) [432]	Human identification and activity recognition	Explainability is necessary so relationships between model inputs and outputs can be identified. This is necessary so the behavior of the proposed method (fusion model) can be inferred.
Someetheram et al. (2022) [433]	Explaining and improving discrete hopfield neural network	Election algorithms are good at reducing the complexity of HNN models, but it is not known how they do that. It is known how it must reduce the complexity, so explainability is needed to ensure the models work the right way.
Sudars et al. (2022) [434]	Traffic sign classification	The authors mentioned that understanding the decision-making process of the CNNs is essential in applications where human lives are at stake, such as autonomous driving. Further, they mentioned that using explainability can provide insights into the inner workings of the CNN model for improved transparency and trust in the classification results.
Altini et al. (2023) [269]	Kidney tumor segmentation	The authors mentioned that the explainability is important in their specific application of anomaly detection in Industrial Control Systems (ICS) because explaining the detection outcomes and providing explanations for anomaly detection results is essential for ensuring that experts can understand and trust the decisions made by the model.
Serradilla et al. (2021) [273]	Predictive maintenance	Explanations about the model’s decision in the anomaly detection task enable the operator to evaluate the model’s accuracy and act based on their own expertise.
Aslam et al. (2023) [435]	Malware detection	Black-box models are widely used in studies and real-life applications of malware detection, but black-box models lack interpretability that could be used to validate models’ decisions. Explainability in the context of malware detection on Android devices has not been studied much, and new information about attacks on Android devices could be gained by applying explainability to malware detection models. Explainability also would help users trust malware detection models.
Shin et al. (2023) [436]	Network traffic classification	The authors mentioned that explainability in their specific application is important to increase reliability. And also, they mentioned that as the performance of both ML and DL models improves, the derivation process of the results becomes more opaque, highlighting the need for research on transparent design and post-hoc explanation for artificial intelligence.
Samir et al. (2023) [437]	Bug assignment and developer allocation	The authors mentioned that the explainability of their specific application is important to increase user trust and satisfaction with the system.
Guidotti et al. (2021) [438]	Distinguish time series representing heart rate between normal heartbeat and myocardial infarction.	To reveal how the AI system is reasoning and agree with it or not in an easier way. Also, developers can unveil misclassification reasons and vulnerabilities and act to align the AI reasoning with human beliefs.
Ekanayake et al. (2023) [439]	Predict adhesive strength	For identifying the importance of features and elucidating the ML model’s inner workings.
Hendawi et al. (2023) [81]	Diabetes prediction	The authors mentioned that providing easily interpretable explanations for complex machine learning models and their outcomes is essential for healthcare professionals to get a clear understanding of AI-generated predictions and recommendations for diabetes care.
Kobayashi et al. (2024) [440]	Predict remaining useful life within intelligent digital twin frameworks	For AI decisions to be audited, accounted for, and easy to understand.
Misitano et al. (2022) [271]	Multiobjective optimization	Explanations support the decision maker (user) to make the changes needed in the multiobjective optimization task. The opaqueness of black-box methods is problematic when these methods are applied in critical domains (healthcare, security, etc.).
Leite et al. (2023) [282]	Predict the geographic location of a vehicle	To assure stable and understandable rule-based modeling.
Varam et al. (2023) [248]	Endoscopy image classification	The authors mentioned that the explainability is essential in their specific application to enhance the reliability, trust, and interpretability of deep learning models for Wireless Capsule Endoscopy image classification, which is benefiting clinical research and decision-making processes in the medical domain.
Bitar et al. (2023) [441]	Explaining spike neural network	There is very little research done about explaining spike neural networks. Explainability of these models is needed so they can be understood better and therefore improved more efficiently. Also, developing model-specific explanation tools for SNN models is beneficial because model-specific tools are often less computationally exhaustive than model-agnostic XAI tools.
Kim et al. (2023) [442]	Cerebral cortices processing	The authors mentioned that the explainability is essential in their specific application to understand the neural representations of various human behaviors and cognitions, such as semantic representation according to words, neural representation of visual objects, or kinetics of movement. Further, they have mentioned that the explainability allows for a deeper understanding of the cortical contributions to decoding kinematic parameters, which is essential for advancing the study of neural representations in different cognitive processes and behaviors.
Khondker et al. (2022) [443]	Pediatric urology	The authors mentioned that the explainability is essential in their specific application for transparency in the medical field in pediatric urology, as it allows clinicians to comprehend the factors influencing the model’s decisions and enhances confidence in the model’s predictions.
Lucieri et al. (2023) [444]	Biomedical image analysis	The authors mentioned that the explainability is essential in their specific application to address the privacy risks posed by concept-based explanations. And also they mentioned the need to investigate the privacy risk posed by different human-centric explanation methods such as Concept Localization Maps (CLMs) and TCAV scores to properly reflect practical application scenarios.
Suhail et al. (2023) [445]	Cyber-physical systems	The authors mentioned that the explainability is essential in their specific application to provide justifiable decisions by reasoning what, why, and how specific cybersecurity defense decisions are made in a gaming context. Further, they mentioned the transparency and interpretability given by the explainability are helping in building trust, confidence, and understanding among stakeholders and finally leading to more informed and effective cybersecurity measures in the context of Digital Twins (DTs) and Cyber-Physical Systems (CPS).
George et al. (2023) [87]	Predictive modeling for emergency department admissions among cancer patients	The authors mentioned that the explainability is essential in their specific application to enable clinicians to intervene prior to unplanned emergency department admissions. Further, they mentioned that clinicians can better understand the factors influencing the risk of ED visits using explainability, and it leads to more informed decision-making and potentially improved patient outcomes.
Bacco et al. (2021) [446]	Sentiment analysis	AI systems for sentiment analysis have a great effect on the real world because sentiment analysis is usually used to analyze customer behavior or public opinion. Explainability is needed to ensure the models make ethical and rightful decisions.
Szczepanski et al. (2021) [229]	Fake news detection	Several kinds of biases are prevalent and hard to detect when detecting fake news with AI due to the fact that fake news spreads on social media. Explainability is needed to gain understanding of the model’s decision process and therefore prevent biases. Explainability increases trust and therefore enables wider use of AI.
Dong et al. (2021) [256]	Classifying functional connectivity for brain-computer interface system	Explainability can lead to new knowledge about aging and using brain-computer interface systems on elderly people.
El-Sappagh et al. (2021) [108]	Alzheimer’s disease prediction	The authors mentioned that the explainability is essential in their specific application to ensure that the AI model’s decisions are transparent, understandable, and actionable for clinical practice.
Prakash et al. (2023) [447]	Electrocardiogram beat classification	Lack of explainability makes AI methods challenging to implement in real-life use cases. Explainability increases trust, user performance, and user satisfaction.
Alani et al. (2022) [448]	Malware detection	The authors mentioned that explainability builds trust in the AI model in their context as well as ensures that the high accuracy originates from explainable conditions rather than from a black-box operation.
Sasahara et al. (2021) [126]	Metabolic stability and CYP inhibition prediction	Explainability could give new information about the importance of different physicochemical parameters. This knowledge can be used to design better drugs and to understand underlying structures better.
Maloca et al. (2021) [449]	Classify medical (retinal OCT) images	Understanding an AI model’s decision process will provide confidence and acceptance of the machine.
Tiensuu et al. (2021) [258]	Stainless steel manufacturing	The authors mentioned that the explainability in their specific application is essential for facilitating human decision-making, early detection of quality risks, and conducting root cause analysis to improve product quality and production efficiency.
Valladares-Rodriguez et al. (2022) [73]	Cognitive impairment detection	The authors mentioned that explainability may become a fundamental requirement in their domain and tasks, such as detecting MCI, to improve transparency and interpretability of AI-based decisions.
Ahn et al. (2021) [450]	Hospital management and patient care	The authors mentioned that the explainability is important in their specific application to provide persuasive discharge information, such as the expected individual discharge date and risk factors related to cardiovascular diseases. Further, they have mentioned that this explainability can assist in precise bed management and help the medical team and patients understand the conditions in detail for better treatment preparation.
Hammer et al. (2022) [451]	Brain Computer Interfacing (BCI)	The authors mentioned that the explainability is important in their specific application to uncover and understand how functional specialization emerges in artificial deep convolutional neural networks during a brain-computer interfacing (BCI) task.
Ikushima et al. (2023) [452]	Age prediction based on bronchial image	Explainability increases trust toward ML model due to its ability to justify model-made decisions. Explainability can also reveal new information and connections that have not been discovered otherwise.
Kalir et al. (2023) [453]	Semiconductor manufacturing	The authors mentioned that explainability in their specific application to provide insights into the decision-making process of the machine learning models to domain experts. Further, they mentioned that this transparency in model predictions helps in building trust in the AI systems and aids in decision-making processes related to capacity, productivity, and cost improvements in semiconductor manufacturing processes.
Shin et al. (2022) [454]	Cardiovascular age assessment	Explainability can give new information about features that predict cardiovascular aging. Explainability also helps evaluate model performance and improve the model.
Chandra et al. (2023) [455]	Soil fertility prediction	The authors mentioned that explainability in their specific application is important to build trust and transparency, to enhance the decision-making process, and to provide user-friendly interpretation.
Blix et al. (2022) [456]	Water quality monitoring	The authors mentioned that explainability in their specific application is important to understand the relevance of spectral features for optical water types. Further, they mentioned that explainability provides insights into which variables were affecting each derived water type. And also, they mentioned that this understanding is essential for improving the estimation of chlorophyll-a content through the application of preferred in-water algorithms and improving the accuracy and interpretability of water quality monitoring processes.
Resendiz et al. (2023) [238]	Cancer diagnosis	The authors mentioned that explainability in their specific application is important to address the black box problem associated with deep learning methods in medical diagnosis. And also, they mentioned that the lack of semantic associations between input data and predicted classes in deep learning models can hide the interpretability, which can lead to potential risks when applying these systems to different databases or integrating them into routine clinical practice.
Topp et al. (2023) [457]	Predicting water temperature change	ML models can behave unpredictably when new data is used. Explainability is needed to be able to evaluate and justify the model’s performance and decisions. Explanations are used to evaluate the fidelity and generalizability of the ML model.
Till et al. (2023) [458]	Wrist fracture detection	Explainability in ML models used in healthcare is needed to ensure trust toward the model because the IT knowledge of healthcare professionals is often limited. Trading predictive performance to explainability (black-box–white-box models) is problematic, which makes explaining black-box models important.
Aswad et al. (2022) [459]	Flood prediction	Variables used in flood prediction can have some complexity, and explainability is needed to evaluate model performance and understand the decisions.
Kalyakulina et al. (2023) [71]	Predicting immunological age	Explainability enhances understanding of a model’s decision-making process, but it can also be used to improve model performance by selecting only important features in computing. This reduces the cost of using the ML model and therefore makes it more usable. Local explanations are necessary to personalize treatments when needed.
Ghosh et al. (2024) [460]	Predicting energy generation patterns	Explanations give practical insight on previously theoretically studied energy generation patterns. Explanations give important information about relationships and depencies between different features that effect energy production, especially when clean energy is being focused on.
Katsushika et al. (2023) [75]	Predicting reduced left ventricular ejection fraction (LVEF)	Medical practitioners using ML models in medical decision-making need explainability to be able to evaluate and validate model-made decisions. Without explainability, medical practitioners cannot utilize ML models to help their work.
Hernandez et al. (2022) [107]	Predicting Alzheimer’s disease and mild cognitive impairment	Explainability enables getting information about important features and also about the model (preprocessing, feature selection, methods). Explainability is important for validating model-made decisions with domain knowledge from the user.
Mohanrajan et al. (2022) [461]	Predicting the land use/land cover changes	That the predicted results will be more informative and trustworthy to the urban planners and forest department to take appropriate measures in the protection of the environment.
Zhang et al. (2023) [462]	Strain prediction	Explainability is needed to understand relationships and qualitative/quantitative impacts of input parameters in modeling mechanical properties. Explainability can also reveal new information about the effects of stress and strain on each other.
Wang et al. (2023) [463]	ML-based decision support in medical field	Explainability is important so that medical practitioners can communicate their and ML model-made decisions to patients, therefore ensuring patient autonomy and informed consent. Explanations also help develop better ML models by revealing the model’s decision-making process. Here explainability is used to evaluate the effect of feature selection on model performance.
Pierrard et al. (2021) [464]	Medical imaging	The authors mentioned the need for transparency and human understandability in the reasoning of the model in critical scenarios where decisions based on image classification and annotation can have significant consequences.
Praetorius et al. (2023) [465]	Detecting intramuscular fat	Explainability is needed to ensure generalizability of the ML model used in intramuscular fat detection.
Escobar-Linero et al. (2023) [215]	Predicting withdrawal from legal process in case of violence towards woman in intimate relationship	Explainability is important so reasons behind (predicted) withdrawal from legal process can be recognized. Explainability can give new knowledge about features that affect participation and therefore help taking care of intimate relationship violence victims. The data from legal cases of intimate relationship violence is quite complex, and explainability is needed to interpret the black-box models needed to use in this prediction task.
Pan et al. (2022) [466]	Biometric presentation attack detection	The authors mentioned that the explainability in their specific application is important to enhance the usability, security, and performance of their Facial Biometric Presentation Attack Detection system.
Wang et al. (2023) [467]	Drug discovery applications	The authors mentioned that understanding the decisions made by models is crucial for building trust and credibility in their predictions in the context of drug discovery, where interpretability is essential for inferring target properties of compounds from their molecular structures. Further, they mentioned that explainability in drug design is a way to leverage medicinal chemistry knowledge, address model limitations, and facilitate collaboration between different experts in the field.
Jin et al. (2023) [252]	Medical image analysis	The authors mentioned that explaining model decisions from medical image inputs is essential for deploying ML models as clinical decision assistants. Further, they mentioned that providing explanations helps clinicians understand the reasoning behind the model’s predictions. And also, they mentioned that explainability is essential in their specific application to enhance the transparency, trustworthiness, and utility of ML models in the context of multi-modal medical image analysis.
Naser (2022) [468]	Evaluating fire resistance of concrete-filled steel tubular columns	Black-box models cannot be used reliably and effectively by engineers in practice because they don’t provide explanations about their decision-making process. Explainability is also needed to ensure liability and fairness in the fire engineering domain because human lives and legal aspects are involved.
Karamanou et al. (2022) [469]	Anomaly detection from open governmental data	Explainability is needed to ensure accountability, transparency, and interpretability of black-box ML models used in unsupervised learning tasks.
Kim et al. (2021) [470]	Predicting the wave transmission coefficient of low-crested structures	Explainability is needed to evaluate the performance and decision-making process of the ML model.
Saarela et al. (2021) [211]	Student agency analytics	The authors mentioned that explainability is important in their specific application to ensure transparency, accountability, and actionable insights for both students and educators. Further, they mentioned the General Data Protection Regulation (GDPR) includes a right for explanation, and for that, automatic profiling must be used in a Learning Analytics (LA) tool. And also, they mentioned explainability can help teachers increase their awareness of the effects of their pedagogical planning and interventions.
Gong et al. (2022) [471]	COVID-19 detection	White-box models are widely used in the medical field because of their high interpretability despite their low performance. Explainability of back-box models is needed to enable the use of more effective ML models in the medical domain.
Burzynski (2022) [472]	Battery health diagnosis	The authors mentioned that explainability is important in their specific application to provide insights into the model’s behavior and facilitate the interpretation of the relationships between input parameters and predictions. Further, they mentioned that this way enables a better understanding of the model’s decision-making process and enhances the trustworthiness of the predictions, which is essential for optimizing battery management systems and extending battery life.
Kim et al. (2022) [473]	Forecasting particle matter of aerodynamic diameter less than 2.5 $\mathsf{\mu }$m	Ambient particle matter forecast experts tend to question the reliability of ML models and validity of their predictions. Explainability is needed to increase trust towards black-box models.
Galiger et al. (2023) [474]	Histopathology tissue type detection	The authors mentioned that explainability is important in their specific application to align the decision-making process with that of human radiologists and to provide clear, human-readable justifications for model decision-making. Further, they mentioned explainability is essential for gaining trust in the model’s decisions and ensuring its reliability in the medical imaging domain.
Naeem et al. (2023) [475]	Malware detection	The proposed ensemble method for malware prediction is quite complex, and explainability is needed to enable interpretation and validation of model-made decisions.
Burzynski (2022) [472]	Battery health monitoring	The authors mentioned that explainability is important in their specific application to understand and interpret the results of machine learning and deep learning models applied to lithium-ion battery datasets. Further, they mentioned that using XAI researchers can gain insights into the outcomes produced by the algorithms, describe the model’s accuracy, fairness, transparency, and results in decision-making, and investigate any biases in predicted results.
Uddin et al. (2021) [476]	Human activity recognition	People tend to not accept ML systems that might be accurate and efficient if they lack interpretability. Explainability is needed to gain trust toward ML models and allow use of more efficient models.
Sinha et al. (2023) [477]	Fault diagnosis of low-cost sensors	The authors mentioned that explainability is important in their specific application to increase the trust and reliability of the AI model used for fault diagnosis of low-cost sensors.
Jacinto et al. (2023) [478]	Mapping karstified zones	Explainability is needed to validate ML-made decisions and detect biases. Explainability allows the use of more complex models when interpretability is necessary. Explainability gives information about relationships between model inputs and outputs.
Jakubowski et al. (2022) [479]	Anomaly detection in asset degradation process	The authors mentioned that by providing explanations in their context, experts can understand the reasoning behind the model’s decisions and ensure the reliability of the predictive maintenance actions taken based on those decisions.
Guo et al. (2024) [480]	Intelligent fault diagnosis in rotating machinery	The authors mentioned that providing explanations for the model’s predictions is essential to improving the trust and understanding of the diagnostic process. And also, they mentioned explainability helps in validating the diagnostic results and improving the generalization ability of the model in unseen domains.
Shi et al. (2021) [481]	Age-related macular degeneration diagnosis	The authors mentioned that explainability in their specific application is important because it helps in understanding the decision-making process of the model and the rationale behind its classifications. And also, they mentioned explainability helps to maximize its clinical applicability for the specific task of geographic atrophy detection, and clinicians trust the model’s predictions and integrate them into their decision-making process.
Wang et al. (2023) [127]	Drug repurposing	Explanations are not always interpretable or reliable; do not provide information that is well related to the application domain. Explanations need to connect well to the problem they explain so that reliable interpretations and decisions can be made.
Klar et al. (2024) [290]	Factory layout design	Explainability enables evaluating training processes and model decisions when using AI in factory layout planning. Explainability also enhances trust towards decisions. Explainability reveals relationships and importance of features and therefore can give valuable information that can be used later in the factory layout design process.
Panos et al. (2023) [482]	Predicting solar flares	Explainability is important so model decisions can be evaluated and justified. Explainability can also help improve model performance and possibly give new information about solar flares and features that predict them because of the high diagnostic capabilities of spectral data.
Fang et al. (2023) [483]	Predicting landslide	Explainability helps to make decisions on evacuations and interventions in landslide areas in an effective and ethical way. Explanations can also help identify the need for a specific intervention (slope stabilization, for example).
Karami et al. (2021) [484]	Predicting response to COVID-19 virus	Explainability is needed to allow interpretation of model-made decisions and to find information about connections between features.
Baek et al. (2023) [485]	Semiconductor equipment production	The authors mentioned that explainability is important in their specific application to understand how deep learning algorithms make decisions due to their complexity and to explain the outputs.
Antoniou et al. (2022) [283]	Attention deficit hyperactivity disorder (ADHD) diagnosis	Because clinicians are only willing to adopt a technological solution if they understand the basis of the provided recommendation.
Nguyen et al. (2022) [486]	Decision-making agents	The authors mentioned that explainability is important in their specific application to enhance trust, ensure legal compliance, improve user understanding, and increase user satisfaction in their specific application.
Solorio-Ramirez et al. (2021) [119]	Predicting brain hemorrhage	In ML tasks in the healthcare domain, it is usual that the model has to make predictions on data that it hasn’t seen before. Model-made decisions in this kind of use case have to be explainable so the decision can be evaluated and justified. Explainability increases transparency and therefore understanding of the ML model’s decision-making process.
de Velasco et al. (2023) [221]	Identifying emotions from speech	Identifying emotions from speech data is a complex task, and complex models are needed to achieve appropriate results. Explainability is needed to increase understanding of computational methods and models’ decision-making processes in this use case.
Shahriar et al. (2022) [487]	Predicting state of battery charge in electric vehicle	Electric vehicles and their batteries are constantly evolving and can be very different, which makes developing a globally applicable model for state of charge estimation difficult. Explainability is needed for evaluating and improving model performance.
Kim et al. (2023) [488]	Maritime engineering	The authors mentioned that explainability is important in their specific application to provide transparency on how the ML model produces its predictions. Further, they mentioned that using XAI, they can get a clear understanding of how different predictors influence the outcome of the prediction regarding vessel shaft power, which is essential for decision-making processes in the shipping industry.
Lemanska-Perek et al. (2022) [489]	Sepsis management	The authors mentioned that explainability is important in their specific application to support medical decision-making for individual patients, such as to better understand the model predictions, identify important features for each patient, and show how changes in variables affect predictions.
Minutti-Martinez et al. (2023) [490]	Classifying chest X-ray images	Healthcare professionals tend not to trust easily black-box models, which could be fixed by utilizing explainability methods. Explainability is also legally required in the healthcare domain.
Wang et al. (2023) [101]	Predicting chronic obstructive pulmonary disease (COPD)	Lack of explainability makes well-performing ML models useless in the healthcare domain. Explainability is needed to ensure interpretability and transparency, which leads to a wider application of ML in healthcare. Explainability also helps detect biases and improve model performance.
Kim et al. (2023) [491]	Medical imaging for fracture detection	The authors mentioned that the use of AI with explainability for fracture diagnosis has the potential to serve as a basis for specialist diagnosis. And also, they mentioned that AI could assist specialists by offering reliable opinions, preventing misinterpretations, and also speeding up the decision-making process for diagnosis.
Ivanovic et al. (2023) [88]	Medical data management; cancer patient case	The authors mentioned that explainability is important in their specific application to ensure that the AI models are not only accurate but also transparent, trustworthy, and interpretable for the end users in the medical and healthcare domains.
Sullivan et al. (2023) [227]	Deep Q-learning experience replay	The authors mentioned that explainability is important in their specific application, Deep Reinforcement Learning (DRL), because the lack of transparency in DRL models leads to challenges in debugging and interpreting the decision-making process.
Humer et al. (2022) [492]	Drug discovery	The authors mentioned that explainability helps in identifying chemical regions of interest and gaining insights into the ML model’s reasoning.
Zhang et al. (2023) [493]	Power systems dispatch and operation	The authors mentioned that the explainability of their specific application is important to provide a more intuitive and comprehensive explanation of decision-making for power systems with complex topology. Further, they mentioned that this is essential for operators to obtain noteworthy power grid areas as the basis of auxiliary decision-making to realize efficient and accurate control.
Yang et al. (2023) [494]	Machinery health prediction	Explainability is needed in industry machinery health assessment systems to increase reliability, allow evaluation of the model’s decision-making process, and help the end user understand and trust the model.
Altini et al. (2023) [269]	Nuclei classification from breast cancer images	Explainability is legally required in ML models used in the healthcare domain, and because complex models are needed because of their high performance, explainability techniques need to be used. Explainability also reveals important features and therefore increases interpretability and usability.
Papandrianos et al. (2022) [64]	Predicting coronary artery disease from myocardial perfusion images	Explainability is needed so medical professionals can verify the model’s decisions.
Liang et al. (2021) [230]	Identifying deceptive online content	Explainability is needed to evaluate the model in cases of wrong decisions and to help develop model that are more reliable against targeted attacks when using ML to detect deceptive text/content.
Alabdulhafith et al. (2023) [60]	Remote prognosis of the state of intensive care unit patients	Explainability is needed to ensure reliability of the model in addition to performance metrics. Medical professionals need explanations to evaluate models’ decisions and their medical relevance to be able to use ML models in practice.
Zolanvari et al. (2023) [299]	Intrusion detection	Lack of explainability leads to lack of trust, and trusting ML models without explanations leads to a lack of applicability and legitimacy. Explainability is needed to ensure transparency and applicability.
Carta et al. (2021) [279]	Stock market forecasting	The authors mentioned that the explainability of their specific application is important to provide transparency and understanding of the prediction process. Further, they mentioned that explainability allows for a deep understanding of the obtained set of features and provides insights into the factors influencing the stock market forecasting results.
Esmaeili et al. (2021) [117]	Brain tumor localization	The authors mentioned that the explainability of their specific application is important to improve the interpretability, transparency, and reliability of deep learning models in the context of tumor localization in brain imaging.
Cheng et al. (2022) [495]	Healthcare predictive modeling	The authors mentioned that explainability is important in their specific application because models in the healthcare domain require being transparent and interpretable. And also, they mentioned clinicians may not have technical expertise in machine learning, and therefore explanations need to be provided in a way that aligns with their domain knowledge rather than technical details.
Wenninger et al. (2022) [294]	Building energy performance prediction	The authors mentioned that explainability is important in their specific application because to understand the mechanics behind the methods applied for increasing trust and accountability in the context of retrofit implementation where uncertainty is a major barrier. Further, they mentioned that explainability provides insights for experts on the influence of various building characteristics on the final energy performance predictions.
Laqua et al. (2023) [496]	E-bikes	The authors mentioned that explainability is important in their specific application to enhance the understanding of the user experience of e-bike riding.
Espinoza et al. (2021) [268]	Antibiotic discovery	The authors mentioned that explainability is important in their specific application for model interpretability, validation, feature selection optimization, and advancing scientific discovery in the context of predicting antimicrobial mechanisms of action using AI models.
Sanderson et al. (2023) [497]	Flood inundation mapping	The authors mentioned that deep learning models are often considered “black boxes”, which can have challenges regarding transparency and potential ethical biases. Using XAI to flood inundation mapping, they aimed to enhance insight into the behavior of their proposed deep learning model and how it is impacted by varying input data types.
Abe et al. (2023) [498]	Estimating pathogenicity of genetic variants	The authors mentioned that by incorporating explainability into their AI model, they can provide understandable explanations to physicians and make informed decisions based on the AI’s estimation results and genomic medical knowledge. And also, they mentioned this approach eliminates the bottlenecks in genomic medicine by combining high accuracy with explainability and supporting the identification of disease-causing variants in patients.
Kerz et al. (2023) [499]	Mental health detection	The authors mentioned that there is a growing need for explainable AI approaches in psychiatric diagnosis and prediction to ensure transparency in the decision-making process.
Kim et al. (2022) [500]	Satellite image analysis for environment monitoring and analysis	The authors mentioned that explainability in their specific application is important to improve the reliability of AI-based systems by providing visual explanations of predictions made by black-box deep learning models. And also, they mentioned explainability helps in preventing critical errors, especially false negative errors in image selection, and by providing visual explanations, the system can be refined based on supervisor feedback, which can reduce the risk of misinterpretation or incorrect predictions.
Thrun et al. (2021) [501]	Water quality prediction	Explainability is necessary to enable the use of complex and high-performing models in predicting water quality, because domain experts usually are not familiar with AI. They need interpretable and clear explanations to evaluate and trust the model’s decisions.
Gowrisankar et al. (2024) [231]	Detecting deepfake images	Explainability is necessary to ensure users’ trust toward the ML model and to help users understand the ML model better. Different XAI methods perform differently (especially saliency map techniques), and therefore efficient XAI evaluation techniques are needed to help find the most accurate and interpretable XAI technique.
Beni et al. (2023) [502]	Predicting weathering on rock slopes	The ML model used in weather prediction does not give information about the contributions of different features. Explainability is needed to gain insight on model performance and therefore evaluate the model’s decisions.
Singh et al. (2022) [84]	Arrhythmia classification	A ML model often has to deal with unseen data in arrythmia classification task, and explainability is needed to evaluate model performance and decisions in these cases. Healthcare professionals tend not to trust AI-based diagnostic tools, and explainability would increase trust towards ML models and therefore enable use of AI diagnostic tools. In the healthcare domain, explainability is also necessary in an ethical and legal sense.
Zhou et al. (2023) [503]	Predict dissolved oxygen concentrations in karst springflow	Explainability is needed for accessing information about physical processes and mechanisms learned by the ML model. Data from karstic areas is often complex, and explainability is therefore even more necessary for evaluating model performance.
Maqsood et al. (2022) [118]	Brain tumor detection	Brain image data is complex, and models that make predictions based on those images need to be complex. Explaining the model provides more information about the model.
Cui et al. (2022) [287]	Machine reading comprehension	Explainability allows users to understand how the model answers questions, which can be very helpful for educational purposes.
Barros et al. (2023) [504]	Cement industry dispatch workflow	The authors mentioned that explainability in their specific application is important for obtaining information about potential blockages of transportation vehicles, enabling monitoring and inspection to prevent delays or process restarts in advance. They also mentioned that explainability helps avoid security issues such as violations of federal regulations on vehicle weight. Also, in the context of finances, they mentioned explainability assists in preventing orders from being sent in quantities greater than requested, and it helps to avoid monetary losses.
Kayadibi et al. (2023) [505]	Recognizing and classifying retinal disorders	Explainability helps medical professionals understand ML-made diagnoses and use them as diagnostic tools. Explainability enables more accurate, efficient, and reliable diagnosis because of the necessary human evaluation step and the complex nature of retinal data.
Qamar et al. (2023) [506]	Fruit classification	The authors mentioned that explainability is important in fruit classification because it can enhance processes such as sorting, grading, and packaging, reducing waste and increasing profitability. Further, they mentioned that by using explainability, they can enhance the transparency and interpretability of the models used in automated fruit classification systems, and it improves trust, identifies biases, meets regulatory requirements, and increases users’ confidence in the system.
Crespi et al. (2023) [507]	Multi-agent systems for military operations	The authors mentioned that explainability is important in their specific application because it can provide insights into the inner workings of the learned strategies, facilitate human understanding of agent behaviors, and enhance transparency and trust in the decision-making processes of the multi-agent system.
Sabrina et al. (2022) [508]	Optimizing crop yield	The authors mentioned that explainability is important in their specific application to ensure that the system is trusted and easily adopted by farmers. Further, they mentioned that this explainability is essential for making the system understandable, trustworthy, and user-friendly for farmers.
Wu et al. (2023) [509]	Flood prediction	The authors mentioned that explainability is important in their specific application to improve model credibility and provide insights into the factors influencing runoff predictions. Further, they mentioned that explainability is essential for understanding the complex relationships between meteorological variables and runoff dynamics.
Nakamura et al. (2023) [510]	Disease prevention	The authors mentioned that explainability is important in their specific application to identify concrete disease prevention methods at the individual level. They also mentioned explainability is essential for setting intervention goals for future disease development prevention and improving outcomes through targeted health condition improvements.
Damian et al. (2022) [232]	Detecting fake news	Explainability is needed to gain insight into the ML model’s reasoning and decision-making process. Explainability can also help develop better and more effective models by revealing the most important features, which is important with text data, where there are thousands of features (aka different words).
Oh et al. (2021) [511]	Glaucoma Ddagnosis	The authors mentioned that explainability is important in their specific application to provide a basis for ophthalmologists to determine whether to trust the predicted results.
Borujeni et al. (2023) [512]	Air pollution forecasting	The authors mentioned that explainability is important in their specific application to get a better understanding of how the model reaches its decisions. And also they referred to the phenomenon of “Clever Hans” predictors, where models might perform well on training and test datasets but fail in practical scenarios. Thus they mentioned that by understanding how the model makes decisions, it is possible to identify instances where the model may be relying on incorrect criteria for predictions. And also, they mentioned explainability is essential for efficient feature selection and model optimization.
Alharbi et al. (2023) [513]	Unmanned aerial vehicle (UAVs) operation	The authors mentioned that explainability is important in their specific application to ensure the safe, efficient, and equitable allocation of airspace system resources in UTM operations.
Sheu et al. (2023) [514]	Pneumonia prediction	In the pneumonia classification application of ML, explainability is needed to gain insight about important features that affect the classification of pneumonia. Explainability is also needed to convince medical professionals about the model’s reliability and therefore to gain acceptance from the medical domain. Users need to be able to interpret and trust the ML model in order to use it efficiently in practice as a diagnostic tool.
Solis-Martin et al. (2023) [291]	Predictive maintenance	Time series data in predictive maintenance is complex and hard to interpret. Explainability is needed to understand the ML model and relationships between inputs and outputs better.
Castiglione et al. (2023) [128]	Drug repurposing	Explainability increases the reliability of ML models. In drug repurposing tasks, explainability is also mandatory to ensure transparency and accountability.
Aslam et al. (2022) [515]	Antepartum fetal monitoring and risk prediction of IUGR	The authors mentioned the explainability in their specific application is important to enhance the interpretability of ML models, generate confidence in the predictions, add to comprehensibility, and assist doctors in their decision-making process regarding antepartum fetal monitoring to predict the risk of IUGR.
Peng et al. (2022) [516]	Fault detection and diagnosis	Explainability is needed to gain insight on reasons behind predicted faults and to ensure model performance with complex data and possible online/offline use.
Na Pattalung et al. (2021) [517]	Critical care medicine for ICU patients	The authors mentioned that the explainability in their specific application is important to provide a causal explanation in the ML models, and making predictions visible from a black box model is essential to understanding the severity of illness and to enable early interventions for patients in ICU.
Oliveira et al. (2023) [518]	Decision support system	Lack of explainability is concerning when ML is used in high-stake cases. Explainability is also needed to ensure the legitimacy of AI use.
Burgueno et al. (2023) [519]	Land cover classification	Using explainability techniques leads to transparency, justifiability, and informativeness of the ML model, which is necessary in applications where there are critical aspects involved.
Horst et al. (2023) [520]	Human gait recognition	The authors mentioned that explainability is important in their specific application to identify the most relevant characteristics used for classification in clinical gait analysis.
Napoles et al. (2023) [521]	Predictive analytics; case study in diabetes	The authors mentioned that the explainability is important in their specific application to understand how an algorithm works and how it can help analysts with the understanding of key questions and needs of their organization.
Ni et al. (2023) [522]	Hydrometeorology	The authors mentioned that providing physical explanations for data-driven models is essential. Further, they mentioned that it is important to understand the inner workings of the deep learning model and provide insights into what the network has learned.
Amiri-Zarandi et al. (2023) [523]	Threat detection in IoT	The authors mentioned that the explainability is important to help cybersecurity experts understand the reasons behind detected threats, improve security monitoring practices, and communicate with users about the reasons for their investigation.
Huang et al. (2023) [250]	Soil moisture prediction	Explainability enables extracting information about relationships between features in data and/or in the model. Explainability increases trust in ML models among users and decision-makers.
Niu et al. (2022) [524]	Diabetic retinopathy detection	The authors mentioned that the explainability in their specific application is important to understand how DL models make predictions, to improve trust, and to encourage collaboration within the medical community.
Kliangkhlao et al. (2022) [525]	Predicting demand and supply behavior	Causality explanations help decision-makers understand the reasons behind models’ decisions.
Singha et al. (2023) [266]	Cancer treatment; drug response prediction	The authors mentioned that traditional AI models operate as black boxes, and in critical domains like cancer therapy, where trust, accountability, and regulatory compliance are essential, the lack of explainability in AI models is a significant drawback. Further, they mentioned using explainability can provide clear, interpretable, and human-understandable explanations for the model’s actions and decisions, and it improves trustworthiness and usability and facilitates further research on potential drug targets for cancer therapy.
Thrun (2022) [278]	Stock market analysis	The authors mentioned that commonly known explanations for stock-picking processes are often too vague to be applied in concrete cases. They also mentioned that explainability is important to provide specific criteria for stock picking that are explainable and can lead to above-average returns in the stock market.
Dissanayake et al. (2021) [526]	Heart anomaly detection	The authors mentioned that the explainability in their specific application is important to trust the predictions made by the models in the medical domain. And also, they mentioned that even if a model performs with excellent accuracy, understanding its behavior and predictions is important for medical experts and patients to trust the validity of the system.
Dastile et al. (2021) [527]	Credit scoring	The authors mentioned that the explainability in their specific application of credit scoring is important to regulatory requirements like the Basel Accord, which mandates that lending institutions must be able to explain to loan applicants why their applications were denied. Also, they mentioned explainability is important to gain trust in model predictions, ensure no discrimination occurs during the credit assessment process, and meet the “right to explanation” requirement under regulations like the European Union General Data Protection Regulation (GDPR).
Khan et al. (2022) [528]	COVID-19 classification	The authors mentioned that the explainability in their specific application is important to provide significant proof that explainable AI is essential in the context of healthcare applications like COVID-19 diagnosis. Also, they mentioned that using visualization techniques like Grad-CAM helps highlight the crucial regions in the input images that influenced the deep learning model’s predictions and enhances understanding and trust in the classification results for COVID-19 detection.
Moon et al. (2021) [529]	Alzheimer’s disease	The authors mentioned that explainability is important in their specific application to provide insights into the complex models used for classification.
Carrieri et al. (2021) [42]	Skin microbiome composition	The authors mentioned that explainability is important in their specific application to enhance the utility and reliability of ML models in microbiome research and to facilitate the translation of research findings into actionable insights.
Beker et al. (2023) [236]	Volcanic deformation detection	The authors mentioned that the explainability is important in their specific application to understand model behavior, improve performance, validate predictions, and determine the sensitivity of the model in detecting subtle volcanic deformations in the InSAR data.
Kiefer et al. (2022) [530]	Document classification	The authors mentioned that explainability is important in their specific application to align machine learning systems with human goals, contexts, concerns, and ways of working.
Sokhansanj et al. (2022) [216]	Inter Partes Review (IPR) predictions	The authors mentioned that explainability is important in their specific application to align machine learning systems with human goals, contexts, concerns, and ways of working.
Matuszelanski et al. (2022) [207]	Customer churn prediction	The authors mentioned that the explainability in their specific application is important to understand the limitations of the model and address issues without sacrificing the performance gain from black-box models.
Franco et al. (2021) [531]	Face recognition	The authors mentioned that the explainability in their specific application of face recognition is important because of the widespread and controversial use of facial recognition technology in various contexts. Further, they mentioned that making face recognition algorithms more trustworthy through explainability, fairness, and privacy can improve public opinion and general acceptance of these technologies.
Montiel-Vazquez et al. (2022) [532]	Empathy detection in textual communication	The authors mentioned that the explainability in their specific application is important to improve transparency and a better understanding of how the model makes decisions, which is essential for building trust in the system and for potential applications in various fields where empathy detection is valuable.
Mollas et al. (2023) [533]	User-oriented/interpretable XAI	Explaining the explanations with good metrics is important when complex models are approximated with understandable explanations. Understandability of these explainability metrics is important so the end user can evaluate the outcomes.
Wei et al. (2022) [251]	Detecting disease from fruit leaves	Lack of explainability hinders widespread use of black-box models that could be effective and beneficial in agriculture. Black-box models and their interpretability are important in agriculture because of complex data and the variety of plant species that are of interest.
Samih et al. (2021) [224]	Movie recommendations	Explainability in recommendation systems increases efficiency, transparency, and user satisfaction.
Juang et al. (2021) [534]	Hand palm tracking	The authors mentioned that the explainability is important because the linguistic relationship between the input and output variables of each fuzzy rule is explainable, and providing human explainable fuzzy features and inference models can improve the interpretability of the tracking method. Further, they mentioned that visualization of fuzzy features can give a clear understanding of the decision-making process.
Cicek et al. (2023) [535]	Diagnosing nephrotoxicity	In the field of healthcare, explainability and interpretability are needed to enable the use of black-box ML models for diagnosing diseases.
Jung et al. (2023) [536]	Medicinal plants classification	The authors mentioned that explainability helps in understanding how the ML model makes predictions and enabling the assessment of whether the model’s learning intentions are consistent in the context of classifying similar medicinal plant species like Cynanchum wilfordii and Cynanchum auriculatum.
De Magistris et al. (2022) [233]	Detecting fake news	Explanations are needed to convince people about the classification of fake news.
Rawal et al. (2023) [537]	Identification of variables associated with the risk of developing neutralizing antidrug antibodies to factor VIII in hemophilia A patients	The authors mentioned that the explainability is important in their specific application for identifying and ranking variables associated with the risk of developing neutralizing antidrug antibodies to Factor VIII in hemophilia A patients.
Kumar et al. (2021) [220]	Sarcasm detection in dialogues	The authors mentioned that the explainability in their specific application is important to understand which words or features influence the model’s decision-making process and how the model identifies sarcasm in conversational threads.
Yeung et al. (2022) [538]	Photonic device design	The authors mentioned the explainability in their specific application is important to understand the relationship between the device structure and performance in photonic inverse design. Further, they mentioned that explainability is important to reveal the structure-performance relationships of each device, highlight the features contributing to the figure-of-merit (FOM), and potentially optimize the devices further by overcoming local minima in the adjoint optimization process.
Naeem et al. (2022) [539]	Malware detection in IoT devices	The authors mentioned that explainability is important in their specific application to enhance model transparency, facilitate security analysis, evaluate model performance, and support continuous model improvement.
Mey et al. (2022) [540]	Machine fault diagnosis	The authors mentioned that the black-box nature of deep learning models hides the understanding of the decision-making process and makes it challenging for humans to interpret the classifications. They mentioned that using explainability can make the classification process transparent and provide insights into why certain decisions were made by the model.
Martinez et al. (2023) [541]	Genomics and gene regulation	The authors mentioned explainability can provide transparency to ML models and allow for a better understanding of how the predictions were made. And this transparency was essential in delivering a high-scale annotation of archaeal promoter sequences and ensuring the reliability of the curated promoter sequences generated by the model.
Nkengue et al. (2024) [542]	COVID-19 detection	The authors mentioned explainability in their specific application is important to provide a cross-validation tool for practitioners. Further, they mentioned that highlighting the different patterns of the ECG signal that are related to a COVID-19/non-COVID-19 classification helps practitioners to understand which features of the signal are responsible for the classification, and it helps in decision-making and validation of results.
Behrens et al. (2022) [543]	Climate modeling	The authors mentioned explainability in their specific application is important to enhance the interpretability of convective processes in climate models.
Fatahi et al. (2022) [544]	Cement production	The authors mentioned explainability in their specific application is important to understand the correlations between operational variables and energy consumption factors in an industrial vertical roller mill circuit.
De Groote et al. (2022) [545]	Mechatronic systems modeling	The authors mentioned that by incorporating physics-based relations within the Neural Network Augmented Physics (NNAP) model, they aimed to provide interpretable explanations that align with physical laws. Further, they mentioned that in this way, the understanding of the system dynamics with partially unknown interactions leads to more reliable and insightful predictions.
Takalo-Mattila et al. (2022) [546]	Steel quality prediction	The authors mentioned explainability is important in their specific application to enhance transparency and allow users to understand why a particular decision was reached, to build trust in the model’s predictions, to audit the decisions made by the model, and to ensure compliance with regulations and standards.
Drobnic et al. (2023) [102]	Assessment of developmental status in children	The authors mentioned explainability in their specific application is important to enhance the interpretability of the model’s predictions and to provide insights into the features that influence the motor efficiency index (MEI) assessment of children and adolescents.
Saarela et al. (2022) [32]	Skin cancer classification	The authors mentioned that explainability in their specific application is important for building trust and confidence in the model’s decisions and to know why the system has made a particular decision. Further, they mentioned that using explanations for the model’s decisions increases trust among users and also potentially teaches humans to make better decisions in skin lesion classification.
Jang et al. (2023) [547]	Energy management	The authors mentioned that in the field of energy management, understanding complex AI models can be challenging because of the black box nature. And they mentioned that using explainability can explain the impact of input variables on the model’s output. Further, they mentioned that explainability is essential for EMS managers to comprehend why specific predictions are made, enabling informed decision-making in energy management processes.
Aishwarya et al. (2022) [548]	Diagnostic of common lung pathologies	The authors mentioned that the explainability of their specific application is important to improve the interpretability of the deep learning model’s outputs, which is essential for medical professionals to trust and understand the diagnostic results, and it is leading to faster diagnosis and early treatment.
Kaczmarek-Majer et al. (2022) [549]	Mental health; bipolar disorder	The authors mentioned that explainability is important in their specific application to build trust, enhance understanding, improve decision-making processes, and manage uncertainty in the context of psychiatric care and mental health diagnosis.
Bae (2024) [550]	Malware classification	The authors mentioned that the explainability in their specific application is important to address the challenges of interpreting heterogeneous data and to provide reliable explanations for the models used in cybersecurity applications, especially in malware detection.
Mahim et al. (2024) [109]	Alzheimer’s disease detection and classification	The authors mentioned that in medical image classification, XAI is essential for helping medical professionals understand and interpret the decisions made by AI systems, and this leads to more informed decisions about patient care and treatment plans. Also, the authors mentioned that XAI is essential for regulatory and ethical reasons, as transparency in the decision-making process of AI systems in medical applications is required to ensure consistency with medical standards and regulations.
Gerussi et al. (2022) [551]	Primary Biliary Cholangitis (PBC) risk prediction	The authors mentioned that the explainability in their specific application is important to provide insights into the decision-making process of the ML model and facilitate its application in precision medicine and risk stratification for PBC.
Li et al. (2022) [552]	MRI imaging	The authors mentioned that the explainability in their specific application is important to increase the transparency and interpretability of the super-resolution process for clinical MRI scans. Further, they mentioned that explainability is essential for understanding the decision-making process of the deep learning model and ensuring that the generated high-resolution images are clinically relevant and accurate.
Shang et al. (2021) [267]	Clinical practice	The authors mentioned that the explainability in their specific application is important to help clinicians better understand and utilize important clinical information buried in electronic health record (EHR) data. Also, they mentioned that explainable illustrations of important clinical findings are necessary to provide comprehensive and convincing details for better understanding and acceptance by clinicians beyond their specialties.

, along with the XAI application and the reason(s) why the authors say that explainability is essential in their domain.

Data collection process, data items, study risk of bias assessment, effect measures, synthesis methods, and reporting bias assessment: To categorize and summarize the included articles in this review, the first author developed a Google Survey that was filled out for each selected article. The survey included both categorical (multiple-choice) and open-ended questions designed to systematically categorize the key aspects of the research. This approach ensured a consistent and comprehensive analysis across all articles. The survey provided an Excel file with all responses, simplifying the analysis process.

Each reviewer assessed their allocated articles using the predefined codes and survey questions created by the first author. In cases of uncertainty regarding the classification of an article, reviewers noted the ambiguity, and these articles, along with their tentative classifications, were discussed collectively among both authors to reach a consensus. This discussion was conducted in an unbiased manner to ensure accurate classifications. While no automated tools were used for the review process, Python libraries were employed for quantitative assessment.

Some of the developed codes (survey questions) were as follows:

• What was the main application domain, and what was the specific application?
• In what form (such as rules, feature importance, counterfactual) was the explanation created?
• Did the authors use intrinsically explainable models or post-hoc explainability, and did they focus on global or local explanations?
• How was the quality of the explanation(s) evaluated?
• What did the authors say about why the explainability of their specific application is important? (Open-ended question.)

After completing the coding process and filling out the survey for each included article, we synthesized the data using both qualitative and quantitative techniques to address our research questions [39]. Qualitatively, we summarized the characteristics of the included studies based on the predefined codes. Quantitatively, we performed statistical analysis of the data, utilizing Python 3.11.5 to extract statistics from the annotated Excel table. This combination of qualitative and quantitative approaches, along with collaborative efforts, ensured the reliability and accuracy of our review process.

To assess the risk of reporting bias, we examined the completeness and transparency of the data reported in each article, focusing on the availability of results related to our predefined research questions. Articles that lacked essential data or failed to report key outcomes were flagged for potential bias, and this was considered during the certainty assessment.

Certainty assessment: Regarding the quality of the articles, potential bias, and the certainty of their evidence, we followed the general recommendations [40] and included only articles for which at least seven out of the ten quality questions proposed by Kitchenham and Charters (2007) [39] could be answered affirmatively. Additionally, we ensured quality by selecting only articles published in prestigious journals that adhere to established academic standards, such as being peer-reviewed and having an international editorial board [35].

Table 2. Number of publications for the ten journals with the highest publication counts in our sample of articles on recent XAI applications.

Journal	# of Publications
IEEE Access	45
Applied Sciences-Basel	37
Sensors	28
Scientific Reports	15
Electronics	14
Remote Sensing	8
Diagnostics	7
Information	7
Machine Learning And Knowledge Extraction	7
Sustainability	7

reports the number of publications per journal for the ten journals with the highest publication counts in our sample. As shown in the table, IEEE Access has the highest number of publications, totaling 45, which represents 8.79% of our sample of articles on recent XAI applications. It is followed by this journal (Applied Sciences-Basel) with 37 publications (7.23%) and Sensors with 28 publications (5.47%).

4. Results

In this section, we present the results of the 512 recent XAI application articles that met our inclusion and quality criteria. As detailed in Section 3, we included only those articles that satisfied our rigorous standards and were not flagged for bias. Once the articles passed our inclusion criteria and were coded and analyzed, we did not conduct further assessments of potential bias within the study results themselves. Our analysis relied on quantitative summary statistics and qualitative summaries derived from these high-quality articles. The complete list of these articles is provided in Table A1, along with their specific XAI applications and the authors’ justifications for the importance of explainability in their respective domains. Next, we provide an overview of recent XAI applications by summarizing the findings from these 512 included articles.

4.1. Application Domains

As shown in Figure 3, the absolute majority of recent XAI applications are from the health domain. For instance, several works have focused on different kinds of cancer prediction and diagnosis, such as skin cancer detection and classification [32,41,42], breast cancer prediction [43,44,45], prostate cancer management and prediction [46,47], lung cancer (relapse) prediction [48,49], and ovarian cancer classification and surgery decision-making [50,51]. In response to the COVID-19 pandemic, significant research has been directed toward using medical imaging for detecting COVID-19 [52], predicting the need for ICU admission for COVID-19 patients [53], diagnosing COVID-19 using chest X-ray images [54], predicting COVID-19 [55,56,57,58,59,60], COVID-19 data classification [61], assessment of perceived stress in healthcare professionals attending COVID-19 [62], and COVID-19 forecasting [58].

Medical imaging and diagnostic applications are also prominent, including detecting paratuberculosis from histopathological images [63], predicting coronary artery disease from myocardial perfusion images [64], diagnosis and surgery [65], identifying reasons for MRI scans in multiple sclerosis patients [66], detecting the health status of neonates [67], spinal postures [68], and chronic wound classification [69]. Additionally, studies have focused on age-related macular degeneration detection [70], predicting immunological age [71], cognitive health assessment [72,73], cardiovascular medicine [74,75], glaucoma prediction and diagnosis [76,77,78], as well as predicting diabetes [79,80,81,82] and classifying arrhythmia [83,84].

General management applications in healthcare include predicting patient outcomes in ICU [60], functional work ability prediction [85], a decision support system for nutrition-related geriatric syndromes [86], predicting hospital admissions for cancer patients [87], medical data management [88], medical text processing [89], ML model development in medicine [90], pain recognition [91], drug response prediction [92,93], face mask detection [94], and studying the sustainability of smart technology applications in healthcare [95]. Lastly, studies about tracing food behaviors [96], aspiration detection in flexible endoscopic evaluation of swallowing [97], human activity recognition [98], human lower limb activity recognition [99], factors influencing hearing aid use [100], predicting chronic obstructive pulmonary disease [101], and assessing developmental status in children [102] underline the diverse use of XAI in the health domain.

It is also noteworthy that brain and neuroscience studies have frequently been the main application (Figure 3), often related to health. For example, Alzheimer’s disease classification and prediction have been major areas of focus [103,104,105,106,107,108,109], and Parkinson’s disease diagnosis has been extensively studied [110,111,112,113]. There is also significant research on brain tumor diagnosis and localization [114,115,116,117,118], predicting brain hemorrhage [119], cognitive neuroscience development [120], and detecting and explaining autism spectrum disorder [121]. Other notable brain studies include the detection of epileptic seizures [122,123], predicting the risk of brain metastases in patients with lung cancer [124], and automating skull stripping from brain magnetic resonance images [125]. Similarly, three pharmacy studies are related to health, including metabolic stability and CYP inhibition prediction [126] and drug repurposing [127,128].

In the field of environmental and agricultural applications, various studies have utilized XAI techniques for a wide range of purposes. For instance, earthquake-related studies have focused on predicting an earthquake [129] and assessing the spatial probability of earthquake impacts [130]. In the area of water resources and climate analysis, research has been conducted on groundwater quality monitoring [131], predicting ocean circulation regimes [132], water resources management through snowmelt-driven streamflow prediction [133], and analyzing the impact of land cover changes on climate [134]. Additionally, studies have addressed predicting spatiotemporal distributions of lake surface temperature in the Great Lakes [135] and soil moisture prediction [136]. Environmental monitoring and resource management applications also include predicting heavy metals in groundwater [137], detection and quantification of isotopes using gamma-ray spectroscopy [138], and recognizing bark beetle-infested forest areas [139]. Agricultural applications have similarly leveraged XAI techniques for plant breeding [140], disease detection in agriculture [141], diagnosis of plant stress [142], prediction of nitrogen requirements in rice [143], grape leaf disease identification [144], and plant genomics [145].

Urban and industrial applications are also prominent, with studies on urban growth modeling and prediction [146], building energy performance benchmarking [147], and optimization of membraneless microfluidic fuel cells for energy production [148]. Furthermore, predicting product gas composition and total gas yield [149], wastewater treatment [150], and the prediction of undesirable events in oil wells [151] have been significant areas of research. Lastly, environmental studies have also focused on predicting drought conditions in the Canadian prairies [152].

In the manufacturing sector, XAI techniques have been employed for a variety of predictive and diagnostic tasks. For instance, research has focused on prognostic lifetime estimation of turbofan engines [153], fault prediction in 3D printers [154], and modeling hydrocyclone performance [155]. Moreover, the prediction and monitoring of various manufacturing processes have seen substantial research efforts. These include predictive process monitoring [156,157], average surface roughness prediction in smart grinding processes [158], and predictive maintenance in manufacturing systems [159]. Additionally, modeling refrigeration system performance [160] and thermal management in manufacturing processes [161] have been explored. Concrete-related studies include predicting the strength characteristics of concrete [162] and the identification of concrete cracks [163]. In the realm of industrial optimization and fault diagnosis, research has addressed the intelligent system fault diagnosis of the robotic strain wave gear reducer [164] and the optimization of injection molding processes [165]. The prediction of pentane content [166] and the hot rolling process in the steel industry [167] have also been areas of focus. Studies have further examined job cycle time [168] and yield prediction [169].

In the realm of security and defense, XAI techniques have been widely applied to enhance cybersecurity measures. Several studies have focused on intrusion detection systems [170,171,172], as well as trust management within these systems [173]. Research has also explored detecting vulnerabilities in source code [174]. Cybersecurity applications include general cybersecurity measures [175], the use of XAI methods in cybersecurity [176], and specific studies on malware detection [177]. In the context of facial and voice recognition and verification, XAI techniques have been employed for face verification [178] and deepfake voice detection [179]. Additionally, research has addressed attacking ML classifiers in EEG signal-based human emotion assessment systems using data poisoning attacks [180]. Emerging security concerns in smart cities have led to studies on attack detection in IoT infrastructures [181]. Furthermore, aircraft detection from synthetic aperture radar (SAR) imagery has been a significant area of research [182]. Social media monitoring for xenophobic content detection [183] and the broader applications of intrusion detection and cybersecurity [184] highlight the diverse use of XAI in this domain.

In the finance sector, XAI techniques have been employed to enhance various decision-making processes. Research has focused on decision-making in banking and finance sector applications [185], asset pricing [186], and predicting credit card fraud [187]. Studies have also aimed at predicting decisions to approve or reject loans [188] and addressing a range of credit-related problems, including fraud detection, risk assessment, investment decisions, algorithmic trading, and other financial decision-making processes [189]. Credit risk assessment has been a significant area of research, with studies on credit risk assessment [190], predicting loan defaults [191], and credit risk estimation [192,193]. The prediction and recognition of financial crisis roots have been explored [194], alongside risk management in insurance savings products [195]. Furthermore, time series forecasting and anomaly detection have been important areas of study [196].

XAI has also been used for transportation and self-driving car applications, such as the safety of self-driving cars [197], marine autonomous surface vehicle engineering [198], autonomous vehicles for object detection and networking [199,200], and the development of advanced driver-assistance systems [201]. Similarly, XAI offered support in retail and sales, such as inventory management [202], on-shelf availability monitoring [203], predicting online purchases based on information about online behavior [204], customer journey mapping automation [205], and churn prediction [206,207].

In the field of education, XAI has been applied to various areas such as the early prediction of student performance [208], predicting dropout rates in engineering faculties [209], forecasting alumni income [210], and analyzing student agency [211]. In psychology, XAI was used for classifying psychological traits from digital footprints [212]; in social care, for child welfare screening [213]; and in the laws, for detecting reasons behind a judge’s decision-making process [214], predicting withdrawal from the legal process in cases of violence towards women in intimate relationships [215], and inter partes institution outcomes predictions [216]. In natural language processing, XAI was used for explaining sentence embedding [217], question classification [218], questions answering [219], sarcasm detection in dialogues [220], identifying emotions from speech [221], assessment of familiarity ratings for domain concepts [222], and detecting AI-generated text [223].

In entertainment, XAI was used, for example, for movie recommendations [224], explaining art [225], and different gaming applications, including analyzing and optimizing the performance of agents in a game [226], deep Q-learning experience replay [227], and cheating detection and player churn prediction [228]. Furthermore, several studies concentrated on (social) media deceptive online content (such as fake news and deepfake images) detection [229,230,231,232,233,234]. In summary, the recent applications of XAI span a diverse array of domains, reflecting its evolving scope; Figure 4 illustrates eight notable application areas.

4.2. XAI Methods

As shown in Figure 5, the majority of recent XAI papers used local explanations (53%), or a combination of global and local explanations (29%).

SHapley Additive exPlanations (SHAP) and Local Interpretable Model-agnostic Explanations (LIME) are the most commonly used local XAI methods (Figure 6). While LIME is fully model-agnostic, meaning it is independent of the prediction model and can be used on top of any linear or non-linear model, the SHAP toolbox includes both model-agnostic XAI tools (such as the SHAP Kernel Explainer) and model-specific XAI tools (such as the TreeExplainer, which has been optimized for tree-based models [239]). However, LIME has faced criticism for its instability, meaning the same inputs do not always result in the same outputs [32], and its local approximation lacks a stable connection to the global level of the model. In contrast, SHAP boasts four desirable properties: efficiency, symmetry, dummy, and additivity [240], providing mathematical guarantees to address the local-to-global limitation. These guarantees may explain SHAP’s higher popularity in recent XAI application papers (Figure 6). Another local model-agnostic method used in recent XAI application papers is Anchors, which belongs to the same XAI group as SHAP and LIME but is much less popular in recent XAI application papers (e.g., [167,188,190,229,241]).

While perturbation-based techniques, such as LIME (e.g., [65,175,187,242,243]) and SHAP (e.g., [65,175,186,244,245]), are often the choices in recent XAI studies for tabular data, studies involving images or other more complex data frequently use gradient-based techniques such as Grad-CAM (e.g., [89,94,164,178,243]), Grad-CAM++ (e.g., [41,94,246,247,248]), SmoothGrad (e.g., [246,249,250,251,252]), Integrated Gradients (e.g, [50,179,182,253,254]), or Layer-Wise Relevance Propagation (LRP), such as those in [175,179,241,255,256]. Figure 4 shows eight examples of saliency maps from image data of diverse recent XAI applications from various domains.

The most commonly used global model-agnostic techniques are Partial Dependence Plots (PDP), such as those in [65,74,102,257,258], Accumulated Local Effects (ALE), as seen in [136,157,258,259,260], and Permutation Importance (e.g., [74,136,137,156,180]). Conversely, the most commonly used global intrinsically explainable methods are decision trees (e.g., [88,91,183,191,261]) and logistic regression (e.g., [50,53,61,191,211]). It should be noted that the latter two are used in countless other papers, but, given their inherent interpretability, they are often not explicitly listed as XAI methods [31].

4.3. ML Models and Tasks

Figure 7 represents the mostly used ML models in recent XAI papers (please note that more than one ML model can be used in the same paper). Various neural network models (predominantly deep NN) are mostly used ML models (used in 59% of papers), followed by the tree-based modes (e.g., decision tree, random forest, gradient boosting, used in 37% of papers), support vector machine (11%), linear or logistic regression (9%), K nearest neighbor (4%), Bayesian-based models (3%), and Gaussian models (e.g., Gaussian process regression and Gaussian mixture model, used in 2% of papers). The distribution of the ML models used in the reviewed articles is comparable to what is generally used.

Besides the most common ML models, there are some others that are less used and could therefore provide interesting alternative views on XAI. These include methods based on fuzzy logic (e.g., fuzzy rule-based classification [262], rule-based fuzzy inference [226,263], fuzzy decision tree [264], fuzzy nonlinear programming [95]), graph-based models (e.g., graph-deep NN [265,266], knowledge graph [267]), or some sort of optimization with computational intelligence (e.g., particle swarm optimization [148,160], clairvoyance optimization [268]).

The ML models have been used mainly for classification purposes (70%), followed by regression (21%), clustering (4%) and reinforcement learning (1%), as can be seen in Figure 8. Other tasks, which occurred in only one or at most two articles, include segmentation [97,269], optimization [270,271], semi-supervised [272,273] or self-supervised tasks [274], object detection [275], and novelty search [276].

There is no substantial difference between the major ML models with regard to the ML task of their target application. The distributions of ML tasks for specific ML models (NN, DT, LR, kNN, etc.) are all very similar to the overall one represented in Figure 8. Among all major ML models, SVM stands out the most, which is used for classification somewhat more often than the others (in 80% of cases).

With regard to the application domain, health, environment, industry, and security and defense are among the top five domains for all the major ML models, with the only exception being linear or logistic regression. In the case when linear or logistic regression was used as an ML model, finance is among the top three application domains, which is never the case for other major ML models. As finance is the second most used with the tree-based ML models, which, similarly to linear and logistic regression, can be characterized as the most transparent and inherently interpretable models, it suggests that the users in the financial domain are especially keen on getting insights and explanations on how the ML models operate on their data.

4.4. Intrinsically Explainable Models

As shown in Figure 9, the majority of recent XAI papers used post-hoc explainability approaches on ML models, which are not naturally easily interpretable (79%), as opposed to the intrinsically explainable models (12%); other papers (9%) reported a combination of both. Figure 10 presents the distribution of intrinsically explainable ML models. From all the reviewed XAI papers that reported their used method as intrinsically explainable, the majority were tree-based (41%), followed by deep NN (19%), linear or logistic regression (5%), and some Bayesian models (3%). The predominance of tree-based ML models could have been expected, as well as a relatively high number of linear and logistic regression models, which both are considered naturally transparent and simpler to understand, given their inherent interpretability. On the other hand, the relatively high number of deep neural networks that have been represented as intrinsically explainable is somewhat surprising.

There are significant differences between different ML models represented as intrinsically explainable with regard to the form of explanation they use. While the intrinsically explainable tree-based ML models use a variety of forms of explanation, including feature importance (in 50% of all cases), rules (38%), and visualization (31%), the deep NN models being reported as intrinsically explainable rely mainly on visualization (in more than 67% of all cases). The intrinsically explainable linear and logistic regression ML models, however, use predominantly feature importance as their form of explanation (in 75% of all cases).

In the most frequent XAI application domain, namely health, the use of tree-based ML models is predominant, as the tree-based models are used in 28% of all health applications, followed by (deep) neural networks (22%), and interestingly fuzzy logic (11%), while all other models were used only once in health. Given the known history of the development of ML methods in the field of medicine and healthcare, where the ability to validate predictions is as important as the prediction itself, and consequently the key role of decision trees [277], this result does not even surprise us.

With regard to other application domains, we can see that intrinsically explainable ML models, like tree-based models and linear or logistic regression models, are used for finance and education applications much more often than other ML models. While the financial domain represents only 1% of (deep) neural network applications, it represents 6% of all tree-based ML model applications (used for credit risk estimation [192,193], risk management in insurance [195], financial crisis prediction [194], investment decisions and algorithmic trading [189], and asset pricing [186]) and even 9% of linear or logistic regression applications (used primarily for credit risk assessment [190] and prediction [193], as well as financial decision-making processes [189]). While post-hoc explainability methods, primarily SHAP and LIME, are the most favored in the financial sector [189], intrinsically explainable modes are gaining popularity for revealing the insights and are being used for stock market analysis [278] and forecasting [279], profit optimization, and predicting loan defaults [191]. Education represents 2% of all applications of tree-based ML models (including early prediction of student performance [208], predicting student dropout [209], and advanced learning analytics [211]) and 4% of linear or logistic regression models (such as pedagogical decision-making [211] and prediction of post-graduate success and alumni income [210]), while (deep) neural networks are used for comparison with other methods in only two of all the reviewed XAI papers concerning education [209,211].

4.5. Evaluating XAI

The use of well-defined and validated metrics for evaluating the quality of XAI results (i.e., explanations) is of great importance for widespread adoption and further development of XAI. However, a significant number of authors still use XAI methods as a sort of add-on to their ML models and results without properly addressing the quality aspects of provided explanations, and only a few articles in our corpus use metrics to quantitatively measure the quality of their XAI results (Figure 11). More than 58% of the reviewed articles applied XAI but did not provide any evaluation of their XAI results (e.g., [65,121,170,175,186]). Among those that evaluated their XAI results, most relied on anecdotal evidence (20% of the reviewed articles, e.g., [185,245,249,272,280]). In approximately 8% of papers, the authors evaluated their XAI results by asking domain experts to evaluate the explanations (e.g., [66,70,89,114,167]). In approximately 19% of papers, however, some sort of quantitative metrics are used to provide the quality assessment (e.g., [94,179,187,242,244,281]).

These numbers are in line with a recent review article about XAI evaluation methods that also highlighted the lack of reporting metrics to measure explanation quality, according to Nauta et al. [4], only one in three studies that developed XAI algorithms evaluates explanations with anecdotal evidence, and only one in five studies evaluates explanations with users. Also, Leite et al. state that “evaluation measures for the interpretability of a computational model is an open issue” [282]. To address this issue, they introduced an interpretability index to quantify how a granular rule-based model is interpretable during online operation. In fact, the gap of “no agreed approach on evaluating produced explanations” [283] is often mentioned as future work. Having such a metric would solve several XAI issues, such as decreasing the risk of confirmation bias [283,284].

For this purpose, we further analyzed the articles that used metrics to measure explanation quality, primarily to see what the authors reported about the explainability of their results. Since different ML tasks and/or ML models may focus on different aspects, we divided the analysis according to the main task of the ML model.

In the case when metrics have been used for evaluating the quality of clustering, segmentation, and other unsupervised ML methods’ explanations, the findings highlight that the evaluated XAI approaches provided accurate, transparent, and robust explanations, aiding in the interpretation of the ML models and results (e.g., [285,286]). Human and quantitative evaluations confirmed the methods’ superiority in generating reliable, interpretable, and meaningful explanations [287], despite occasional contradictory insights that proved useful for identifying anomalies [269].

For the reinforcement learning applications, the findings of papers assessing their XAI results by evaluation metrics demonstrate that the proposed methods effectively explained complex models and highlighted the potential of Shapley values for explainable reinforcement learning [288]. Additionally, participants using the AAR/AI approach identified more bugs with greater precision [289], and while explanations improved factory layout efficiency, their interpretability remains an area for improvement [290].

The findings of the papers using regression as their main task, which used some metric to evaluate the explanations, underscore the critical role of explainability techniques like Shapley and Grad-CAM in enhancing model interpretability and accuracy (e.g., [157,291]) across various domains, from wind turbine anomaly detection [244] to credit card fraud prediction [187]. While global scores aid in feature selection, semi-local analyses offer more meaningful insights [292]. XAI methods revealed system-level insights and emergent properties [293], though challenges like inconsistency, instability, and complexity persist [157,294]. User studies and model retraining confirmed the practical benefits of improved explanations [213,295]. However, the authors mentioned that the explainability of their results was limited by the lack of suitable metrics for evaluating the explainability of algorithms [294].

Finally, for the most frequent ML task of classification, the analysis of the papers, which used some metrics to evaluate their explainability results, emphasizes the importance of explainability in enhancing model transparency, robustness, and decision-making accuracy across various applications, from object detection from SAR images [182] and hate speech detection [296] to classification of skin cancer [32] and cyber threats [297]. Techniques like SHAP, LIME, and Grad-CAM provided insights into feature importance and model behavior (e.g., [124,298,299]). In some situations, the adopted XAI methods showed improved performance and more meaningful explanations, aiding in tasks like malware detection [177], diabetes prediction [82], extracting concepts [298], and remote sensing [300]. Evaluations confirmed that aligning explanations with human expectations and ensuring local and global consistency are key to improving the effectiveness and trustworthiness of AI systems [235]. The authors concluded that while explanation techniques showed promise, there is still a long way to go before automatic systems can be reliably used in practice [32], and widely adopted XAI metrics can help here a lot.

In summary, the results reveal distinct preferences and practices in using XAI. Tree-based models, commonly used in health applications, employ various explanation forms like feature importance, rules, and visualization, while deep neural networks primarily utilize visualization. Linear and logistic regression models favor feature importance. In finance and education, tree-based and regression models are more prevalent than deep neural networks. However, despite the widespread application of XAI methods, evaluation practices remain underdeveloped. Over half of the studies did not assess the quality of their explanations, with only a minority using quantitative metrics. There is a need for standardized evaluation metrics to improve the reliability and effectiveness of XAI systems.

5. Discussion and Conclusions

This systematic literature review explored recent applications of Explainable AI (XAI) over the last three years, identifying 664 relevant articles from the Web of Science (WoS). After applying exclusion criteria, 512 articles were categorized based on their application domains, utilized techniques, and evaluation methods. The findings indicate a dominant trend in health-related applications, particularly in cancer prediction and diagnosis, COVID-19 management, and various other medical imaging and diagnostic uses. Other significant domains include environmental and agricultural applications, urban and industrial optimization, manufacturing, security and defense, finance, transportation, education, psychology, social care, law, natural language processing, and entertainment.

In health, XAI has been extensively applied to areas such as cancer detection, brain and neuroscience studies, and general healthcare management. Environmental applications span earthquake prediction, water resources management, and climate analysis. Urban and industrial applications focus on energy performance, waste treatment, and manufacturing processes. In security, XAI techniques enhance cybersecurity and intrusion detection. Financial applications improve decision-making processes in banking and asset management. Transportation studies leverage XAI for autonomous vehicles and marine navigation. The review also highlights emerging XAI applications in education for predicting student performance and in social care for child welfare screening.

In categorizing recent XAI applications, we aimed to identify and highlight the most significant overarching themes within the literature. While some categories, such as “health”, are clearly defined and widely recognized within the research community, others, like “industry” and “technology”, are broader and less distinct. The latter categories encompass a diverse range of applications, reflecting the varied contexts in which XAI methods are employed across different sectors. This categorization approach, though occasionally less precise, captures the most critical global trends in XAI research. It acknowledges the interdisciplinary nature of the field, where specific categories may overlap or lack the specificity found in others. Despite these challenges, our goal was to provide a comprehensive overview that highlights the most prominent domains where XAI is being applied while recognizing that some categories, by their nature, are more general and encompass a wider array of subfields.

By far the most frequent ML task among the reviewed XAI papers is classification, followed by regression and clustering. Among the used ML models, deep neural networks are predominant, especially convolutional neural networks. The second most used group of ML models are tree-based models (decision and regression trees, random forest, and other types of tree ensembles). Interestingly, there is no substantial difference between the major ML models with regard to the ML task of their target application.

Feature importance, referring to techniques that assign a score to input features based on how useful they are at predicting a target variable [26], is the most common form of explanation among the reviewed XAI papers. Some sort of visualization, trying to visually represent the (hidden) knowledge of a ML model [301], is used very often as well. Other commonly used forms of explanation include the use of saliency maps, rules, and counterfactuals.

Regarding methods, local explanations are predominant, with SHAP and LIME being the most commonly used techniques. SHAP is preferred for its stability and mathematical guarantees [240], while LIME is noted for its model-agnostic nature but criticized for its instability [32]. Gradient-based techniques such as Grad-CAM, Grad-CAM++, SmoothGrad, LRP, and Integrated Gradients are frequently used for image and complex data [179,182]. In general, post-hoc explainability is much more frequent than the use of some intrinsically explainable ML model. However, only a few studies quantitatively measure the quality of XAI results, with most relying on anecdotal evidence or expert evaluation [4].

In conclusion, the recent surge in XAI applications across diverse domains underscores its growing importance in providing transparency and interpretability to AI models [4,5]. Health-related applications, particularly in oncology and medical diagnostics, dominate the landscape, reflecting the critical need for explainable and trustworthy AI in sensitive and high-stakes areas. The review also reveals significant research efforts in environmental management, industrial optimization, cybersecurity, and finance, demonstrating the versatile utility of XAI techniques.

Despite the widespread adoption of XAI, there is a notable gap in the evaluation of explanation quality. The analysis of how the authors evaluate the quality of their XAI approaches and results revealed that in the majority of studies, the authors still do not evaluate the quality of their explanations or simply rely on subjective or anecdotal methods, with only a few employing rigorous quantitative metrics [284]. Cooperation with domain experts and including users can greatly contribute to the practical usefulness of the results, but above all, more attention needs to be paid to the development and use of well-defined and generally adopted metrics for evaluating the quality of explanations. It turns out that in such a case, we can expect reliable, interpretable, and meaningful explanations with a significantly higher degree of confidence. There is an urgent need for standardized evaluation frameworks to ensure the reliability and effectiveness of XAI methods, as well as to improve the interpretability and stability of explanations. The development of such metrics could mitigate risks like confirmation bias and enhance the overall robustness of XAI applications.

Limitations and Future Work

This systematic literature review has several limitations that should be acknowledged. Firstly, the review relied exclusively on the WoS database to identify and retrieve relevant studies. While WoS is recognized as one of the most prestigious and widely utilized research databases globally, known for its rigorous indexing standards and the high quality of its data sources [38], the reliance on a single database may introduce a potential bias by omitting relevant literature indexed in other databases such as Scopus, IEEE Xplore, or Google Scholar. However, it is important to note that the comprehensive nature of WoS mitigates this limitation to some extent. WoS encompasses a vast array of high-impact journals across various disciplines, ensuring that the most significant and influential works in the field of XAI are likely to be included. Moreover, the substantial volume of results yielded from WoS alone necessitated a practical constraint on the scope of the review. Including additional databases would have exponentially increased the literature volume, rendering the review process unmanageable within the given resources and timeframe.

Secondly, the exclusion criteria applied in this review present additional limitations. Only studies published in English were included, which could potentially skew the findings by overlooking valuable contributions from non-English-speaking researchers and regions. Furthermore, the review was limited to studies published after 2021 to ensure the “recentness” of the applications of XAI. While this criterion was essential to focus on the latest advancements and trends, it may have excluded foundational studies that, although older, remain highly relevant to the current state of the field. Additionally, the review was restricted to journal articles, excluding conference papers that often publish seminal work, particularly in the fast-evolving domain of XAI. Given the considerable volume of literature, including conference papers would have extended the scope beyond what was feasible within the current study.

Moreover, the review process involved manually reading and categorizing each paper to develop detailed codes, allowing for a nuanced analysis of the literature. While more automated approaches to systematic reviews could have incorporated a broader range of sources, such methods may lack the precision and depth achieved through manual categorization. Future research could explore the use of automated methods to include key conference papers and older foundational studies, providing a more comprehensive understanding of the field’s development over time. However, for this review, our focus on recent journal publications, combined with an in-depth manual analysis, was necessary to provide a manageable and focused examination of the most current trends in XAI.

In summary, while these limitations—namely, the reliance on a single database, language restrictions, the specific timeframe, and the focus on journal articles excluding conference papers—are noteworthy, they were necessary to manage the scope and ensure a focused and feasible review process. Future research could address these limitations by incorporating multiple databases, including non-English studies, expanding the temporal range to include older foundational work, and considering a broader set of sources, such as conference papers. This approach would provide a more comprehensive overview of the literature on XAI and its development over time. Finally, it is important to highlight that the field of XAI is rapidly evolving. During the course of conducting and writing this review, numerous additional relevant articles emerged that could not be incorporated due to time constraints. This underscores the dynamic and ongoing nature of research in this area.