XAI-Based Clinical Decision Support Systems: A Systematic Review

Abstract

With increasing electronic medical data and the development of artificial intelligence, clinical decision support systems (CDSSs) assist clinicians in diagnosis and prescription. Traditional knowledge-based CDSSs follow an accumulated medical knowledgebase and a predefined rule system, which clarifies the decision-making process; however, maintenance cost issues exist in the medical data quality control and standardization processes. Non-knowledge-based CDSSs utilize vast amounts of data and algorithms to effectively make decisions; however, the deep learning black-box problem causes unreliable results. EXplainable Artificial Intelligence (XAI)-based CDSSs provide valid rationales and explainable results. These systems ensure trustworthiness and transparency by showing the recommendation and prediction result process using explainable techniques. However, existing systems have limitations, such as the scope of data utilization and the lack of explanatory power of AI models. This study proposes a new XAI-based CDSS framework to address these issues; introduces resources, datasets, and models that can be utilized; and provides a foundation model to support decision-making in various disease domains. Finally, we propose future directions for CDSS technology and highlight societal issues that need to be addressed to emphasize the potential of CDSSs in the future.

1. Introduction

A clinical decision support system (CDSS) [1] supports decision-making for clinicians in diagnosing and treating diseases based on the patient’s clinical information. With the advancing big data analysis and artificial intelligence (AI) lately, the research on the use of these techniques in CDSSs has gained considerable attention. Notably, AI-based CDSSs are highly valuable because of their effectiveness in supporting clinical diagnosis, prescription, prognosis, and treatment using AI models. Originally developed for image processing tasks, CNNs have been applied to a variety of medical imaging applications, including lesion detection, organ segmentation, and disease classification. By leveraging hierarchical feature extraction and spatial hierarchy, CNN-based CDSSs can leverage advanced computational techniques to analyze medical images in depth, which is a significant performance improvement over traditional machine learning CDSSs.

Traditional CDSSs are categorized into knowledge-based CDSSs, where rules are defined in advance, and non-knowledge-based CDSSs, where the rules are not predefined [2]. The principle of knowledge-based CDSSs is to make decisions using correlations and if–then rules for accumulated data. Knowledge-based CDSSs are beneficial as the decision-making process is transparent because it follows predefined rules. However, it has the limitation that knowledge and rules must be defined in advance for all cases.

Conversely, non-knowledge-based CDSSs provide decision-making by learning patterns found in past clinical information using machine learning or AI without rules. Non-knowledge-based CDSSs have been extensively studied in various medical areas [3,4], dealing with issues such as hypertension, heart failure, and lung disease. These CDSSs are expected to be a breakthrough methodology that can reduce the cost of knowledge construction and provide personalized treatment. However, the black-box problem [5], in which explaining the process behind the results derived by AI models is not possible, makes it difficult to apply them in healthcare, where transparency is essential. Therefore, to utilize non-knowledge-based CDSSs, it is necessary to introduce robust clinical validation and evaluations or provide convincing evidence to support the results. In other words, rules in knowledge-based CDSSs are generated by structuring literature-based, practice-based, or patient-driven evidence, which is too costly for experts, and non-knowledge-based CDSSs suffer from the inability to explain model results, making them unusable in practice in the healthcare domain.

EXplainable AI (XAI) [6] was proposed to explain the process of the results of AI-based systems. XAI is a set of processes and methods that allows human users to comprehend and trust the results and output created by machine learning algorithms. The main features of XAI techniques involve creating a suite of ML techniques that (1) produce more explainable models while maintaining a high level of learning performance (e.g., prediction accuracy) and (2) enable humans to understand, appropriately trust, and effectively manage the emerging generation of artificially intelligent partners. The differences in knowledge-, non-knowledge-, and XAI-based CDSSs are shown in Figure 1 [7]. XAI can explain the process and rationale behind an AI model’s decision in a manner that can be interpreted by the user. XAI technologies can be broadly categorized into feature-, model-, complexity-, and methodology-based technologies. XAI-based CDSSs provide high performance of AI models in the healthcare domain, where transparency must be ensured, along with explainability and interpretability, enabling the development of real-world healthcare services. In particular, XAI-based CDSSs can explain the rationale and reasoning behind AI-based model results in real-world clinical processes. For example, in the field of disease detection, when an XAI-based CDSS detects a tumor in a patient’s MRI image, it explains how the image detected the tumor and what evidence exists for the result. In this study, we propose an XAI-based CDSS framework and introduce its application. Our main contributions are summarized as follows:

• We perform a systematic review of explainable AI techniques that can ensure trust-worthiness and transparency in the medical domain and present a forward-looking roadmap for XAI-based CDSSs.
• We categorized various studies ranging from traditional CDSSs to state-of-the-art XAI-based CDSSs using appropriate criteria and summarize the features and limitations of each CDSS to propose a new XAI-based CDSS framework.
• We propose a novel CDSS framework using the latest XAI technology and show its potential value by introducing areas that can be utilized most effectively.

This paper is organized as follows. In Section 2, we describe the research trends related to knowledge-based, non-knowledge-based, and XAI-based CDSSs and the limitations of existing technologies. In Section 3, we present the XAI-based CDSS framework and introduce the datasets and models required for its construction. Applications of XAI-based CDSSs are presented in Section 4, and conclusion are presented in Section 5.

2. Related Work

This section categorizes CDSSs into knowledge-based, non-knowledge-based, and XAI-based CDSSs, and each section describes the main research, technologies, and methodologies, with an overview shown in Figure 2.

2.1. Knowledge-Based CDSS

To manage and utilize big data, an architecture called a knowledge base has emerged, and a methodology has been proposed to incorporate a knowledge base built by correlation of the accumulated data based on the experience of clinicians in a CDSS [8]. This is categorized as a knowledge-based CDSS that supports decision-making by inferring results from a rule-based knowledge base with an inference engine. Accordingly, it is important to design a knowledge base structure and appropriate rule system for each field. The features, functions and applied domain of knowledge-based CDSSs are organized in Table 1. Table 1 divides the categories based on the technologies used in knowledge-based CDSSs and details the characteristics of each technology, relevant research references, the main function (diagnosis and treatment as well as alert and monitoring) of the relevant CDSS studies, and the applied medical domain. Table 2 and Table 3 are presented in the same format.

Knowledge-based CDSSs define rules based on the literature, practice, or patient-oriented evidence [9] and are therefore often used in clinical practice based on clinical guidelines or in evidence-based medicine (EBM). Rule-based inference methodology using the evidential reasoning (RIMER) is based on a belief rule base (BRB) system [10,11,12]. BRB systems set belief degrees to represent different types of uncertain knowledge and extend the if–then rules to represent knowledge. Most BRB-based CDSS frameworks comprise an interfacial layer, an application processing layer, and a data management layer [13,14,15]. These frameworks have proven their performance in various fields, such as COVID-19 [16] and heart failure [17], psychogenic pain [18], tuberculosis [19], acute coronary syndrome (ACS) [20], and lymph node cancer metastasis [21].

Figure 2. Overview of related work.

Figure 2. Overview of related work.

However, an effective knowledge representation for CDSSs could be decision tree, Bayesian network, or nearest neighbors [22]. A study leveraging decision trees proposed a knowledge modeling method in which a clinical model extracted from glaucoma clinical practice guidelines was represented as mind maps and converted into iterative decision trees by clinicians and engineers [23]. In a similar study, mind maps representing the clinical treatment process for thyroid nodules obtained from clinicians is converted into an iterative decision tree model to extract rules [24]. This is followed by the process of representing tacit knowledge to explicit knowledge and finally converting it to executable knowledge. In another study on a decision-tree-based CDSS, a pediatric allergy knowledge base, ONNX inference engine, and tree algorithm were used to provide knowledge of diagnosis and treatment to clinicians [25].

Additionally, Bayesian network-based CDSSs, which are used in various medical areas, such as liver disease [26], breast cancer [27], infectious diseases [28], diabetes [29], angina [30], respiratory diseases [31], and lymph node cancer metastasis [32], use a what–if analysis mechanism. In the field of dental hygiene, Bayesian network framework based on the expectation-maximization (EM) algorithm have recently been used to detect abnormal oral images [33]. Additionally, a hepatitis C virus (HCV) diagnosis system has been proposed using a fuzzy Bayesian network with a fuzzy ontology to resolve ambiguity and uncertainty in outbreaks [34].

Research using the K-nearest neighbor (KNN) algorithm structured medical information by classifying similar clinical cases using ontology extraction methods for case-based reasoning (CBR) [35]. Moreover, a computer-aided diagnosis (CAD) system proposed for melanoma diagnosis provides a related ontology based on asymmetry, border, color, and differential (ABCD) structure rules and classifies similar melanoma cases using the KNN algorithm [36]. Another study that uses similarity of knowledge for decision-making, as in the aforementioned studies, provides an appropriate diagnosis for patients semantically classified by time series similarity based on the patient’s medical history [37].

Knowledge-based CDSSs support decision-making based on a pre-built knowledge base, effective data modeling, and ongoing knowledge-based updating for each domain. Recently, the most influential research area in knowledge-based CDSSs has been genomics, where clinical genomic data models have been proposed to analyze and attribute clinical genomic workflows using genomic data for clinical applications of genomic information [38]. Additionally, various methodologies have also emerged to facilitate knowledge base updates by analyzing newly acquired textual knowledge through natural language processing to generate rules [39].

Knowledge-based CDSSs exhibit potential in that the decision-making process is clear and traceable. However, these systems are limited by maintenance and construction costs because they relies on medical specialists and knowledge engineers for standardization and error correction, as data quality control is essential [40].

Table 1. Knowledge-based CDSSs.

Category	Features	Paper	Functions		Applied Domain
			Diagnosis and Treatment	Alert and Monitoring
BRB	An extension of if–then rules utilizing confidence rules that include confidence levels to represent knowledge	[18]	✓		Tuberculosis
		[16]	✓		Heart Failure
		[19]	✓		Acute Coronary
		[14]	✓		Measles
		[20]	✓		Lymph Node Cancer
		[15]		✓	COVID-19
Decision Tree	Represent knowledge based on a tree structure with a hierarchy of knowledge relationships	[17]		✓	Psychogenic Pain
		[22]	✓		Glaucoma
		[23]	✓		Thyroid
		[24]	✓		Allergies
Bayesian Network	Utilizes probabilities based on Naïve Bayes to classify data	[38]	✓		Glaucoma
		[25]	✓		Liver Disease
		[26]	✓		Breast Cancer
		[27]	✓		Infectious Diseases
		[29]	✓		Angina Pectoris
		[30]	✓		Respiratory Diseases
		[31]	✓		Lymph Node Cancer
		[32]	✓		Dental Hygiene
		[33]	✓		Hepatitis
		[28]		✓	Diabetes
Nearest neighbors	Determines the class of a new instance using the attributes of the nearest neighbor	[35]	✓		Melanoma Diagnosis
		[36]	✓		Diagnostics

Table 1. Knowledge-based CDSSs.

Category	Features	Paper	Functions		Applied Domain
			Diagnosis and Treatment	Alert and Monitoring
BRB	An extension of if–then rules utilizing confidence rules that include confidence levels to represent knowledge	[18]	✓		Tuberculosis
		[16]	✓		Heart Failure
		[19]	✓		Acute Coronary
		[14]	✓		Measles
		[20]	✓		Lymph Node Cancer
		[15]		✓	COVID-19
Decision Tree	Represent knowledge based on a tree structure with a hierarchy of knowledge relationships	[17]		✓	Psychogenic Pain
		[22]	✓		Glaucoma
		[23]	✓		Thyroid
		[24]	✓		Allergies
Bayesian Network	Utilizes probabilities based on Naïve Bayes to classify data	[38]	✓		Glaucoma
		[25]	✓		Liver Disease
		[26]	✓		Breast Cancer
		[27]	✓		Infectious Diseases
		[29]	✓		Angina Pectoris
		[30]	✓		Respiratory Diseases
		[31]	✓		Lymph Node Cancer
		[32]	✓		Dental Hygiene
		[33]	✓		Hepatitis
		[28]		✓	Diabetes
Nearest neighbors	Determines the class of a new instance using the attributes of the nearest neighbor	[35]	✓		Melanoma Diagnosis
		[36]	✓		Diagnostics

2.2. Non-Knowledge-Based CDSS

With the explosion of data and specialized knowledge, the amount of information that must be processed to make clinical decisions is growing astronomically. By learning on its own like a human using massive amounts of data, deep learning and AI, which are based on artificial neural networks, can be used to support clinical decision-making. These methods analyze patterns in patient data to draw associations between symptoms and diagnoses. Moreover, deep learning and AI can be used to analyze various data, including text, images, videos, audio, and signals, enabling the development of non-knowledge-based CDSSs that can understand the overall clinical situation and context. The first step toward a non-knowledge-based CDSS begins with analyzing images and using them to make clinical decisions. A convolutional neural network (CNN) [41] that trains image patterns by mimicking the structure of the human optic nerve has been used to diagnose obstructive sleep apnea by learning high-order correlation features between polysomnography images and their labels [42,43], and an automated system has been proposed to optimize patient satisfaction by analyzing patients’ experiences with ambulance transport services with a combined model of CNN and word embeddings [44]. Similarly, a technique for diagnosing melanoma using a single CNN trained on a dataset of clinical images has been introduced [45].

There are also a number of cases of recurrent neural networks (RNNs) that can handle time series data. Electronic health records (EHRs) data, which is a digital version of a patient’s paper chart, are good candidates for using RNNs [46] because these data provide clinical records with temporal information. A previous study [47] applied RNNs to the EHR data of heart failure patients to predict heart failure, and the method outperformed machine learning methods such as SVMs [48], MLPs [49], logistic regression [50], and KNNs [51]. Because ECG data also contain temporal information, ECG signals can be analyzed using RNN-based models to detect sleep apnea [52].

When dealing with clinical data, owing to its long-term properties, the problem of forgetting previous data and ignoring past information may arise. Therefore, studies using LSTMs [53] to predict future data by considering past data have been proposed. An LSTM was used to learn multiple diagnostic labels to classify diagnoses [54] and oral–nasal thermal airflow, nasal pressure, and abdominal breathing-guided plethysmography data from polysomnography were analyzed with a bidirectional LSTM model to diagnose sleep apnea [55]. Deep learning is frequently applied in medical image analysis. Chest radiographs can be analyzed using deep learning to diagnose chest diseases such as lung nodules [56], lung cancer [57], and pulmonary tuberculosis [58].

Unlike traditional supervised and unsupervised learning, reinforcement learning [59] generated its own training data by observing the current state and selecting future actions. Because existing CDSSs are trained based on evaluations made by different clinicians with different criteria, interrelated symptoms are not considered in some cases. This problem can be solved by applying reinforcement learning, which is used to learn complex environments. A CDSS based on a deep reinforcement learning algorithm has been introduced to determine the initial dose for ICU patients, where an accurate medication prescription is critical and prevents mis-dosing and complications. [60] Reinforcement learning of secure computations enables the implementation of patient-centered clinical decision-making systems while protecting sensitive clinical data. A privacy-preserving reinforcement learning framework with iterative secure computation was proposed to provide dynamic treatment decisions without leaking sensitive information to unauthorized users [3]. A reinforcement learning-based conversational software for radiotherapy was also studied, where the framework used graph neural networks and reinforcement learning to improve clinical decision-making performance in radiology with many variables, uncertain treatment responses, and inter-patient heterogeneity [61].

BERT [62], a large language model based on a transformer [63], was used to develop a CDSS with natural language understanding capabilities. To reduce diagnostic errors, a framework for multi-classification of diagnosis codes in EHRs using BERT [64] has been developed to help clinicians predict the most likely diagnosis. However, specialists have raised concerns about the reliability and responsibility of these deep learning and AI models because of their inability to explain their decisions. Therefore, they are often unwilling to use them in diagnosis. To this end, it is necessary to adopt AI, which provides evidence for prediction and an understandable explanation.

2.3. XAI-Based CDSS

EXplainable AI (XAI) [5], has emerged to overcome the black-box [6] problem of deep learning models, which means that deep learning models have the highest performance compared to other rule-based or machine learning models but have the limitation of lacking interpretability. This can be described as the “performance–interpretability trade-off” and is shown in Figure 3 [65,66]. Performance is highest for deep learning, followed by machine learning models (decision tree, nearest neighbors, and Bayesian network), and rule-based models; however, transparency (interpretability) is inversely proportional. In other words, applying XAI to deep learning models is useful for explaining the reason and logic behind the results predicted by the model to ensure transparency and reliability of the results with high-performance deep learning. With attempts to apply XAI in various fields [67], XAI has gained attention as a solution to the uncertainty problem in CDSSs where accuracy and reliability are important [66].

Table 2. Non-knowledge-based CDSSs.

Category	Features	Paper	Functions		Applied Domain
			Diagnosis and Treatment	Alert and Monitoring
CNN	Features extracted were analyzed by connecting them to convolutional layers	[42]	✓		Sleep Apnea
		[43]	✓		Sleep Apnea
		[45]	✓		Melanoma
		[44]		✓	Ambulance assignment
RNN	Process time series data to find sequential patterns in the data	[47]	✓		Heart Failure
		[52]	✓		Sleep Apnea
LSTM	Useful when dealing with long-term data and utilizing historical data to predict what to expect in the future.	[54]	✓		Diagnostics
		[55]	✓		Sleep Apnea
		[56]	✓		Lung nodules
		[57]	✓		Lung cancer
Reinforcement Learning	Trains software to make decisions to achieve the most optimal results	[58]	✓		Pulmonary tuberculosis
		[60]	✓		Medications
		[3]	✓		Protecting patient information
Transformer	Specializes in processing text data with large language models using transformers in an encoder–decoder structure	[61]	✓		Radiology
		[64]	✓		Diagnosis code categorization

Table 2. Non-knowledge-based CDSSs.

Category	Features	Paper	Functions		Applied Domain
			Diagnosis and Treatment	Alert and Monitoring
CNN	Features extracted were analyzed by connecting them to convolutional layers	[42]	✓		Sleep Apnea
		[43]	✓		Sleep Apnea
		[45]	✓		Melanoma
		[44]		✓	Ambulance assignment
RNN	Process time series data to find sequential patterns in the data	[47]	✓		Heart Failure
		[52]	✓		Sleep Apnea
LSTM	Useful when dealing with long-term data and utilizing historical data to predict what to expect in the future.	[54]	✓		Diagnostics
		[55]	✓		Sleep Apnea
		[56]	✓		Lung nodules
		[57]	✓		Lung cancer
Reinforcement Learning	Trains software to make decisions to achieve the most optimal results	[58]	✓		Pulmonary tuberculosis
		[60]	✓		Medications
		[3]	✓		Protecting patient information
Transformer	Specializes in processing text data with large language models using transformers in an encoder–decoder structure	[61]	✓		Radiology
		[64]	✓		Diagnosis code categorization

The description techniques used in XAI can be broadly categorized into scoop-, model-, complexity-, and methodology-based techniques [67]. The most popular XAI methods in recent research include Shapley additive explanations (SHAP) [68], local interpretable model–agnostic explanations (LIME) [69], post hoc interpretability [70], and gradient-weighted class activation mapping (GradCAM) [70].

Scoop-based techniques determine the contribution of data based on the importance features to train the AI model. A prominent example is the local explainers method (LIME) [71], which is a method specific to a particular instance or outcome. LIME is a technique for generating local approximations to model predictions. For example, given the task of predicting emotions with a deep neural network, LIME focuses on surrounding local words to highlight words that are important for a particular prediction. LIME directly explain how the model’s input data change the outcome; after training the model, it can make guesses about samples that have not appeared before [72]. For COVID-19, LIME and traditional machine learning models were combined to identify the features that had the greatest impact on medical history, time of onset, and patients’ primary symptoms [68]. Similarly, an LSTM model was used in a study on depressive symptom detection, in conjunction with a LIME approach to identify text suggestive of depressive symptoms [73]. Other applications include the diagnosis of Parkinson disease [74], hip disease [75], Alzheimer’s disease, and mild cognitive impairment [76].

By contrast, SHAP uses the explanations on Shapley values that measure the contribution of each feature to the model and is a global explainer method that provides a theoretical interpretation of any dataset using cooperative game theory concepts [77] to calculate the contributions of biomarkers or clinical features (players) for a specific disease outcome (reward) [72]. To predict postoperative malnutrition in children with congenital heart disease, the XGBoost and SHAP algorithms were used to calculate the average of five risk factors (weight one month after surgery, weight at discharge, etc.) for all patients [78]. PHysiologicAl Signal Embeddings (PHASE), a method for transforming time series signals into input features, was first applied to embed body signals with an LSTM model to features extracted using SHAP from EMR/EHR data [79]. In addition, a multi-layer XAI framework utilizing multimodal data, such as MRI and PET images, cognitive scores, and medical histories, have been proposed [80].

SHAP is applied to all layers of the framework, where the first layer performs multiple classification for the early diagnosis of AD. In the second layer, the binary classification score is used to determine the transition from cognitive impairment to AD [80]. Similarly, SHAP has been widely used in various diseases and clinical domains such as predicting readmission [81,82], COVID-19 [83,84,85,86], liver cancer [87], influenza [88], and malignant cerebral edema [89].

More recently, researchers have utilized LIME and SHAP simultaneously to ensure a convincing description of the system. A hybrid approach combining vision transformer (ViT) and a gated recurrent unit was used to generate LIME heat maps using the top three features from the brain MRI images, and SHAP was used to visualize the model’s predictions to demonstrate the validity of data patterns [90]. In addition, the Department of Chronic Kidney Disease also used LIME and SHAP algorithms simultaneously to represent the importance of features in the best model trained by five machine learning methods (random forest, decision tree, naïve Bayes, XGBoost, and logistic regression) [91].

Model-based techniques can be classified into model-specific and model-agnostic methods. Model-specific methods utilize the unique features of a model to make decisions, indicating that they can only be applied to the internal operations of a specific model. An example is Score-CAM [92], which is based on CNNs and compares output for the given input features, thereby indicating their importance. A previous study proposed a system for classifying images from a clock drawing test as a tool for diagnosing dementia that was trained on API-Net [93] and visualized using Score-CAM to provide explainability and transparency [94]. However, model-agnostic methods are model independent and can be applied to any model or algorithm. As a CDSS tool that reduces the model dependency, a COVID-19 symptom severity classifier that utilizes different machine learning models to identify high-risk patients for COVID-19 has been proposed [95].

Complexity-based techniques make machine learning or deep learning models fully interpretable. Interpretability can be categorized into intrinsic interpretability [96] and post hoc interpretability [72] depending on the viewpoint. In general, intrinsic interpretability indicates that a model with a simple architecture can be explained by the trained model itself, whereas post hoc interpretability means that the trained model has a complex architecture and must be retrained to explain this phenomenon. In a study on brain tumor detection based on MRI images, three pre-trained CNNs, DarkNet53 [92], EfficientNet-B0 [97], and DenseNet201 [98], were used to extract features using a hybrid methodology to explain post-interpretability [99].

Another framework for brain tumor diagnosis, NeuroNet19, combines a 19-layer VGG19 that detects complex hierarchical features in images with an inverted pyramid pooling module (iPPM) model, which refines these features, leveraging post-interpretability [100]. Methodology-based techniques are categorized into backpropagation-based and perturbation-based methods [67], among which backpropagation-based GradCAM was proposed to describe CNN models with good performance [101]. GradCAM was applied to the convolutional layer at the end of the CNN and uses the gradient information of the layer to find the features that are highly involved in a particular decision [72,102]. To further improve classification performance, several studies have been proposed to predict oral cancer from oral images using guided attention inference network (GAIN) along with the aforementioned CNN-based VGG19 model GradCAM [103], and these methods are also used to diagnose glaucoma from fundus images using GradCAM’s heatmap and ResNet-50 model [104]. Because CNN models are widely applied in image classification and processing, GradCAM technology is used in several studies utilizing image data [105,106,107,108,109,110,111].

Table 3. XAI-based CDSSs.

XAI	Techniques	Features	Paper	Functions		Applied Domain
				Diagnosis and Treatment	Alert and Monitoring
Scoop Based	Local	Considers the model as a black box and focus on the local variables that contribute to the decision	[73]	✓		Depression
			[74]	✓		Parkinson
			[75]	✓		Gait classification
			[76]	✓		Alzheimer’s disease
			[90]	✓		Alzheimer’s disease
			[91]	✓		Chronic kidney disease
			[68]		✓	COVID-19
	Global	Explains the contribution that relates to the output by getting an understanding of the interaction mechanism of the model variables	[78]	✓		Malnutrition and heart disease
			[83]	✓		Adverse
			[84]	✓		COVID-19
			[86]	✓		COVID-19
			[90]	✓		Alzheimer’s disease
			[91]	✓		Chronic kidney disease
			[79]		✓	Surgical event
Scoop Based	Global	Explains the contribution that relates to the output by getting an understanding of the interaction mechanism of the model variables	[81]		✓	Hospital readmission risk
			[82]		✓	Reattendant risk
			[85]		✓	Hospital mortality
			[87]		✓	Lung cancer
			[88]		✓	Mortality
			[89]		✓	Malignant
Model Based	Model Specific	Applied to a certain scope of application	[95]	✓		COVID-19
	Model-Agnostic	Has no special requirement for the model	[94]	✓		Visuospatial deficits
Complexity Based	Intrinsic	Model is structured to be understandable	[112]	✓		Breast cancer
	Post Hoc	Interpretable information obtained using external methods	[100]	✓		Heart failure
Methodology Based	Backpropagation Based	Backpropagates a significant signal from the output to the input	[103]	✓		Oral cancer
			[104]	✓		Glaucoma
			[105]	✓		Breast cancer
			[106]	✓		COVID-19
			[107]	✓		Glaucoma
			[108]	✓		Fungal keratitis
			[109]	✓		COVID-19
			[111]	✓		COVID-19
			[110]		✓	Hepatocellular carcinoma
	Perturbation Based	Uses techniques to changes the feature set of a given input and investigate the impact of these changes on the network output	[113]	✓		Glioma

Table 3. XAI-based CDSSs.

XAI	Techniques	Features	Paper	Functions		Applied Domain
				Diagnosis and Treatment	Alert and Monitoring
Scoop Based	Local	Considers the model as a black box and focus on the local variables that contribute to the decision	[73]	✓		Depression
			[74]	✓		Parkinson
			[75]	✓		Gait classification
			[76]	✓		Alzheimer’s disease
			[90]	✓		Alzheimer’s disease
			[91]	✓		Chronic kidney disease
			[68]		✓	COVID-19
	Global	Explains the contribution that relates to the output by getting an understanding of the interaction mechanism of the model variables	[78]	✓		Malnutrition and heart disease
			[83]	✓		Adverse
			[84]	✓		COVID-19
			[86]	✓		COVID-19
			[90]	✓		Alzheimer’s disease
			[91]	✓		Chronic kidney disease
			[79]		✓	Surgical event
Scoop Based	Global	Explains the contribution that relates to the output by getting an understanding of the interaction mechanism of the model variables	[81]		✓	Hospital readmission risk
			[82]		✓	Reattendant risk
			[85]		✓	Hospital mortality
			[87]		✓	Lung cancer
			[88]		✓	Mortality
			[89]		✓	Malignant
Model Based	Model Specific	Applied to a certain scope of application	[95]	✓		COVID-19
	Model-Agnostic	Has no special requirement for the model	[94]	✓		Visuospatial deficits
Complexity Based	Intrinsic	Model is structured to be understandable	[112]	✓		Breast cancer
	Post Hoc	Interpretable information obtained using external methods	[100]	✓		Heart failure
Methodology Based	Backpropagation Based	Backpropagates a significant signal from the output to the input	[103]	✓		Oral cancer
			[104]	✓		Glaucoma
			[105]	✓		Breast cancer
			[106]	✓		COVID-19
			[107]	✓		Glaucoma
			[108]	✓		Fungal keratitis
			[109]	✓		COVID-19
			[111]	✓		COVID-19
			[110]		✓	Hepatocellular carcinoma
	Perturbation Based	Uses techniques to changes the feature set of a given input and investigate the impact of these changes on the network output	[113]	✓		Glioma

3. Methods

3.1. Proposed Architecture

This study proposes an XAI-based CDSS framework that can handle both high performance and interpretation of the decision-making process. This is achieved by applying explainable AI technology to artificial intelligence models. Traditional CDSSs have evolved from rule-based to machine learning CDSSs to AI-based CDSSs. However, although AI-based CDSSs guarantees high performance, these systems have a black-box problem that cannot explain the process of model results. In this paper, we propose an XAI framework with explainable and interpretable techniques. As mentioned in Section 2.3, XAI technologies can be broadly categorized into scoop-based, model-based, complexity-based, and methodology-based technologies, and the proposed framework presents a flexible structure that can be applied to the four XAI technologies according to the desired purpose. In other words, the XAI CDSS framework, which ensures high performance of deep learning while ensuring interpretability and explainability, is a necessary technology for the medical domain that requires transparency of AI models. Details of this framework are shown in Figure 4.

The proposed framework handles multimodal data from various medical domains, including text, audio, images, and genomes. It applies the explainable AI methodology to representative deep learning models. Finally, we demonstrate the potential value of the proposed framework through the presentation of application plans, illustrating the circumstances under which they can be utilized effectively.

Most of the existing AI-based CDSSs are limited to text data. Even if the data usage range is extended, only one additional type of biometric signal or image is used. However, a multimodal data utilization plan is necessary to make clinical decisions considering factors that are difficult to record in formal form, such as the patient’s condition, facial expressions, and behavior that change in real time. Consequently, the proposed framework must be capable of expanding its knowledge through the continuous learning and analysis of information derived from multimodal data. Because multimodal CDSSs can handle different types of data in combination, they can analyze patient-related information (e.g., vital signs, test results images, and consultation records) in a multifaceted way and improve decision-making capabilities, especially in the context of patient condition monitoring at the bedside or tumor detection during surgery. As illustrated in Figure 2, a knowledge graph was constructed by extracting the relationship between features based on multimodal features obtained using models such as large-scale language models, VATT, and audio analysis models derived from multimodal data such as text, images, and signals. Furthermore, through reinforcement learning or continuous learning, knowledge graphs can expand automatically, enabling them to respond flexibly to new knowledge, while retaining previous information.

A multimodal clinical XAI learning model for an explanatory clinical decision-making system requires three elements: a deep explanatory label, an interpretable model, and model inference. First, the features that can explain the prediction results must be identified and labeled, and a model must be created in connection with a decision tree with high explanatory power. The explainable models were inferred from the black boxes, which are the largest problem of existing AI methodologies. Explanations are generated with predictions. All of the aforementioned processes [114] are conducted through interactions for patients, medical professionals, and clinical systems in real time. The predictions and explanations generated by analyzing the data obtained from each participant are provided to the participants once more to facilitate overall clinical decision-making, including monitoring, diagnosis, prescription, warning, and document management.

3.2. Dataset

3.2.1. Clinical Dataset

The first large-scale multimodal clinical dataset is UK Biobank [115]. Data have been collected since 2006, and the dataset includes several hierarchical data types such as lifestyle habits, body information, biological samples, electrocardiogram data, and EHR data from more than 500,000 participants. In addition to basic biometric data, the dataset provides genomic analysis, exon mutation testing, full-length genome sequencing, brain MRI, cardiac MRI, abdominal MRI, and carotid artery ultrasonography and radiographic findings. Similar datasets were also obtained from the China Kadoori Biobank [116] and Biobank Japan [117]. The MMIC dataset [118], published by the Massachusetts Institute of Technology, has now been published up to the fourth version. This is an open-source dataset comprising EHR data, including demographic information, diagnostic codes, and medications obtained from ICU patients at the Beth Israel Deaconess Medical Center. MMIC-IV is one of the most representative datasets of clinical AI models that aim to predict clinical events or readmissions. It contains textual data, such as reports, medical notes, and imaging data, including laboratory and physiological data and chest radiographs. Furthermore, it is possible to reconstruct the multimodal dataset using a combination of data from a single modality. This may include, for example, an Alzheimer’s patient’s brain image dataset, the Alzheimer’s Disease Neuroimaging Initiative (ADNI) [119], and exercise activity data from patients with schizophrenia and depression [120].

Table 4. Datasets.

Category	Modality	Dataset	Features	# of Samples
Single Modality	Image	MURA [120]	Musculoskeletal radiology images	40,561 radiograph images
		MRNet [121]	MRI images	1370 knee MRI images
		RSNA [122]	Chest X-ray images	29,687 X-ray images
		Demner, F., 2016 [123]	Chest X-ray images	3996 radiology reports and 8121 associated images
		OASIS [124]	MRI images	140,000 MRI images
		ADNI [119]	MRI images	2733 MRI images
		X. Wang, 2017 [125]	Chest X-ray images	108,948 frontal-view X-ray images of 32,717 patients
		Armato III 2011 [126]	CT images	7371 lesions images
Single Modality	Image	TCIA [127]	Cancer tumor images	50 million MRI Images
	EHR	MIMIC-III [128]	Demographics, clinical history, diagnoses, prescriptions, physical information, etc.	112,000 Clinical reports records
		eICU [129]	Management activities, clinical assessments, treatment plans, vital sign measurements, etc.	200,000 critical care patients
	Text	2010 i2b2/VA [130]	Discharge statements, progress reports, radiology reports, pathology reports	72,846 entities from 826 notes
		2012 i2b2 [131]	Discharge statements	30,062 entities from 310 notes
		2014 i2b2/UTHealth [132]	Longitudinal medical record	1304 medical records of 297 patients
		2018 n2c2 [133]	Discharge statements	202 records of patient with document-level annotations
Multiple Modalities	Genome, Image	TCGA [134]	Genome, medical images	11,000 human tumors across 33 different cancer types
	Genome, Image, EHR	UK Biobank [115]	Clinical information, genomic analysis, exon variant testing, whole genome sequencing testing, MRI, etc.	500,000 individuals clinical records
	Image, Text	ImageCLEFmed [135]	Diagnostic images, visible light images, signals, waves, etc.	4000 radiology images with 4000 associated Question-Answer pairs
		Openi [136]	Medical articles, images	3.7 million images from about 1.2 million PubMed articles
	Multiple signals	PhysiNet [137]	Biomedical signals	Deidentified health data associated with over 200,000 admissions to ICUs
	Video	MedVidCL [138]	Medical instruction images	6617 videos annotation
		UNBC-McMaster [139]	Patient with shoulder pain	Recorded 200 sequences for 25 patients
		MedVidOA [140]	Medical instruction images	3010 manually created health-related questions and timestamps as visual answers

lists the summary of the monomodal and multimodal clinical datasets.

3.2.2. Knowledge Graph

The use of knowledge graphs enables the formal expression of medical expertise and the semantic structuring of unstructured multimodal data. During this period, while learning existing HKGs, the knowledge graph can be expanded using new data. In particular, the emergence of large language models has facilitated the construction of more comprehensive and accurate HKGs. HKGs for CDSSs exist in fields such as medicine (prescription) [141,142], genetics [142,143], and disease [144,145], and based on this, an extended knowledge graph can be generated.

To effectively respond to new clinical cases, information collected in real time should be used in conjunction with existing knowledge. As mentioned previously, an automatic scalability of knowledge graphs is required, and reinforcement learning can be applied to ensures persuasive power. Numerous knowledge graphs exist in the clinical fields other than CDSSs, which are summarized in

Table 5. Clinical knowledge graphs.

	CDSS	Bio-Informatics	Medicine	Pharmaceutical Chemistry
HKG	DrugBank [141]	Gene Ontology [146]	GEFA [147]	HetioNet [148]
	POBOKOP [142]		Reaction [149]
	KnowLife [143]	KEGG [150]	ASICS [151]	DrKG [152]
	Disease Ontology [144]		Hetionet [152]
	iBKH [145]	Cell Ontology [153]	GP-KG [154]	PrimeKG [155]
	PharmKG [156]		DRKF [157]

based on application range.

3.3. XAI Model

Prior to the analysis of the multimodal data, the process of fusing multimodal data is required. In particular, it is important to select meaningful features to obtain the desired information from the vast amounts of data. To achieve this, deep learning, which trains a neural network composed of multiple layers, can be used to extract features and representations from complex data. Deep learning models [158] are well-suited to the integration and extraction of meaningful information owing to their capacity to learn complex patterns and generate knowledge for decision-making through the processing of vast amounts of data. Deep learning models [159], such as those pre-trained on large datasets such as ImageNet [160] or natural language corpora, can be employed to obtain correlations through the generation of new samples from multiple modalities with generative models from GAN [161] or VAE [162].

The advent of the transformer [62] has made multimodal data more accessible as it enables immediate inference based on an attention network rather than a model using convolutional structure, as previously described. Using the transformer structure, it is possible to learn multiple modalities together because encoding is possible in a consistent structure for all data modalities. Since the proposal of the vision transformer (VT) [163] that encodes images in a manner similar to natural language, there have been numerous attempts to apply the transformer to other modalities in video and voice.

Recently, VATT [164], a framework for learning multimodal representations from unlabeled data, has been developed for the extraction of multimodal representations from unlabeled signals. It is possible to perform a range of tasks, including behavioral recognition, voice event classification, image classification, and text-video search using video and audio, and text features extracted from Internet images. Based on the VATT model, the patient’s daily and counseling videos can be analyzed to identify biometric signals, changes over time, and nonverbal expressions for use in clinical decision-making.

XAI techniques can be largely divided based on explanation method, interpretation method, model specificity, and explanation range [165], as shown in Figure 5. The backpropagation-based XAI measures the degree to which each feature affects the result as a gradient value. Class activity map-based XAI visualizes features with a significant impact using the feature map of the uppermost layer, which aggregates the necessary information. Finally, input interference-based XAI provides explanatory power through the process of repeatedly investigating the model while making various changes to the input value of the model.

4. Applications

4.1. Function of CDSSs

The objective of CDSSs is to facilitate optimal decision-making for patient safety. Consequently, the role of CDSSs is to support the decision-making processes involved in diagnosis, treatment, and prescription, which are directly related to patient safety.

Table 6. XAI Models.

Category	Model	Features	XAI Methods
			Post Hoc	Global	Local	Model Specific	Model Agnostic
CAM Based	GradCAM [166]	Feature classification by treating the slope of the prediction over the activation map as a weight	✓		✓	✓
Backpropagation Based	Gradient [167]	Reflect the rate of output change as input changes	✓		✓	✓
	Guided BackProp [168]	Backpropagate non-negative input and output gradients	✓		✓	✓
	GuidedGradCAM [101]	Compute the binwise product between the guided backpropagation and the CAM signal	✓		✓	✓
	DeepLift [169]	Differentiate backpropagation rules from reference operations to account for prediction differences from extraneous informational	✓		✓	✓
	Integrated Gradients [170]	Approximate a gradient from a neutral baseline input to a target input	✓		✓	✓
Backpropagation Based	GradientShap [77]	Approximate Shapley values by computing the expectation of a gradient	✓			✓
	Deconvolution [171]	Modify gradient calculation rules in ReLU functions instead of backpropagating non-negative input gradients	✓	✓			✓
	SmoothGrad [172]	Smoothed noisy gradient signals by averaging heatmaps over input and imputed neighbor samples	✓		✓		✓
Input Inference Based	Occlusion [171]	Occlude a portion of the image with a sliding window and average the difference in output as a feature attribute	✓		✓
	Shapley Value Sampling [173]	Compute Shapley values over a subset of all possible feature combinations	✓	✓		✓
	Kernel Shap [77]	Use non-modality-specific heatmaps	✓		✓		✓
	Feature Permutation [174]	Replace image features by shuffling feature values within a batch and compute the resulting prediction difference	✓		✓		✓
	LIME [68]	Sampling neighboring data around the input to learn an interpretable model	✓		✓		✓

illustrates the functions of CDSS.

The first function of CDSSs is diagnostic support. CDSSs provide diagnostic information on suspected diseases collecting and monitoring figures showing the patient’s conditions, such as biosignals. This function provides efficient management of beds by supporting useful decision-making to inexperienced clinicians or nursing personnel.

The second function is treatment support. This function supports the determination of the optimal treatment by analyzing all applicable treatment methods in consideration of the patient’s current condition. Through treatment method analysis, information analysis of factors such as inconsistencies, errors, omissions, and side effects between the treatment methods is possible. This function is the most commonly studied area and provides interaction information between prescribed drugs or checks for side effects when multiple drugs are prescribed together. The third function of CDSSs is medical image analysis. As the performance of deep learning models using image data develops, it is the most commonly used CDSS function related to AI in the medical field. By analyzing medical image data such as X-rays, MRIs, and CT scans, which are most commonly used to identify patient diseases through deep learning, more accurate decision support can be provided to clinicians.

Finally, the system serves as a risk notification function. This function is typically activated in patients admitted to the hospital and immediately alerts medical personnel when abnormal symptoms or dangerous levels are identified while collecting and monitoring patient biometric signals, such as pulse rate, blood pressure, and temperature.

4.2. The Potential of XAI-Based CDSSs

The most significant features of the proposed XAI-based CDSS framework are generalizations owing to the use of multimodal data, scalability to apply various deep learning SOTA models, and trustworthiness through explainable AI technologies. The fusion and learning of various types of multimodal data generated in the medical domain can result in the discovery of features and patterns that were not found in models that only address single modalities. This can be achieved by developing a generalized foundation model using deep learning models that achieved SOTA performance in each field. Finally, the decision-making process can be explained and interpreted using various explainable AI technologies, enabling the development of a transparent and reliable clinical decision-making system. Figure 6 illustrates the areas where the XAI-based CDSS. framework proposed in this study are the most efficiently utilized. Disease detection represents the first field in which it can be used. Research related to deep learning-based disease detection has already demonstrated high performance; however, there is a challenge in utilizing it in the actual medical and nursing domain because of issues such as the black-box problem of deep learning. To address this challenge, explainable AI technology is employed to elucidate the rationale behind the predictions generated by the deep learning model, thereby facilitating its deployment in the medical and nursing domain.

The second area of application is nurses’ clinical decision support for inpatients. In clinical situations, there is a significant challenge in that the number of professional nurses provide the surveillance for patients is insufficient. Furthermore, in the case of an inexperienced nurses, it is challenging to identify and diagnose appropriate symptoms necessary for patients. The XAI-based CDSS framework is capable of simultaneously monitoring multiple patients and efficiently supporting the decision-making of professional nurses by providing the cause and basis when abnormal symptoms are detected.

The third area of application is treatment and prescription. By analyzing the patient’s medical history or medical record, customized treatment and prescription optimized for the current patient are possible. Specifically, when prescribing a drug, it is possible to derive more optimal prescription results for patient safety by analyzing potential side effects that may occur between the components of the drug prescribed together in advance and providing information to professional medical personnel.

The final area in which XAI-based CDSSs can be utilized is clinical practice. Unlike the aforementioned applications in hospitals and medical institutions, it is employed in the education of professional personnel. In particular, in the case of professional nurses, clinical practice is conducted in educational institutions on cases that may occur in hospitals. The use of XAI-based CDSSs explains the decisions that should be made about the situations that may occur in the ward and the reasons for the decisions.

5. Discussion and Conclusions

The main contribution of this paper is to categorize various studies ranging from traditional CDSSs to state-of-the-art XAI-based CDSSs according to appropriate criteria and to systematically summarize the features and limitations of each CDSS. Furthermore, we propose a new CDSS framework utilizing the latest XAI technology, systematically review explainable AI techniques that can ensure trustworthiness and transparency in the healthcare domain by introducing the areas where they can be most effectively utilized, and provide a future-proof roadmap for XAI-based CDSSs. By organizing the CDSS services introduced above by field and feature, it is possible to grasp the strengths and weaknesses of each CDSS service, and the matters required for CDSS services in the future. However, in the case of existing systems, there are limitations. These include a limited data utilization range, lack of explanatory power for AI models, and the opacity of the decision-making process. Consequently, in the medical field, which demands reliability and transparency, the decision-making process must be clearly represented for users to interpret it. This paper addressed the black-box problem of unknown-based CDSSs as a solution to the problem and proposed an XAI-based CDSS framework that provides valid evidence and reasons for the results. Furthermore, it introduced the available datasets, models, and resources.

The proposed framework is designed to construct an automated extended knowledge graph with multimodal features derived from multi-format data. It comprises three key elements, including a deep explanatory label, an interpretable model, and model inference, which collectively facilitate explainable AI. The framework has the potential to automate the entire process of medical clinical services, from personalized treatment to real-time patient condition reflection. It distinguishes itself from existing systems in terms of multimodal data management, utilization plan, explainable AI application plan, and CDSS application range. Furthermore, the medical knowledge graph and HKG, which are structured using available medical multimodal data, are summarized to enhance expertise in decision-making. The proposed XAI-based CDSS framework serves as a foundational model that can be flexibly applied to multiple disease domains. This approach enables the development of a medical system with minimal temporal and spatial constraints.

Furthermore, advancements in the medical field are facilitated by the ease with which computerized data can be used for research purposes. However, measures to enhance social awareness are necessary because the current medical data are not readily accessible owing to concerns regarding the protection of sensitive personal information. Additionally, there is a prevailing attitude that humans should be trusted even when explanations of the data sources, transparency regarding decision-making processes, and the results are provided. As non-face-to-face medical systems become increasingly prevalent, it is imperative that relevant legal deregulation be enacted. If social awareness and institutional improvements are guaranteed, it is expected that this will facilitate the development of compelling medical solutions through CDSS research and development that integrate richer medical multimodal data, medical knowledge graphs, and XAI technology. For future research, we have officially obtained data of post-abdominal surgery patients from the hospital and will apply an XAI-based CDSS to abdominal surgery patients to verify its clinical feasibility in practice.