Explainable AI in Manufacturing and Industrial Cyber–Physical Systems: A Survey

Abstract

This survey explores applications of explainable artificial intelligence in manufacturing and industrial cyber–physical systems. As technological advancements continue to integrate artificial intelligence into critical infrastructure and industrial processes, the necessity for clear and understandable intelligent models becomes crucial. Explainable artificial intelligence techniques play a pivotal role in enhancing the trustworthiness and reliability of intelligent systems applied to industrial systems, ensuring human operators can comprehend and validate the decisions made by these intelligent systems. This review paper begins by highlighting the imperative need for explainable artificial intelligence, and, subsequently, classifies explainable artificial intelligence techniques systematically. The paper then investigates diverse explainable artificial-intelligence-related works within a wide range of industrial applications, such as predictive maintenance, cyber-security, fault detection and diagnosis, process control, product development, inventory management, and product quality. The study contributes to a comprehensive understanding of the diverse strategies and methodologies employed in integrating explainable artificial intelligence within industrial contexts.

1. Introduction

Artificial intelligence (AI) imitates natural intelligence in machines by mimicking human thinking and problem-solving capabilities. Industrial and manufacturing systems, and, in particular, industrial cyber–physical systems (ICPS) can greatly benefit from AI as they are continually looking for ways to reduce the operational and maintenance costs, improve the process efficiency, and enhance their safe and secure operation over a long period of time. In the era of digitalized manufacturing equipment and the growing use of the internet of things (IoT) and ICPSs, a high volume of data can be gathered from physical environments and smart devices that can be deployed to make the process smarter and more efficient. With the help of AI, productivity and efficiency can be enhanced by increasing production speed, lowering product defects, reducing labor costs, and minimizing unplanned downtime. Despite the vast potential of AI-based systems, it is still risky for the human users to blindly trust their recommendations, insights, or predictions. As we have a poor understanding of how artificial intelligence makes decisions, we cannot fully take advantage of what it offers. This has resulted in the unveiling of so-called explainable AI (XAI). In the following, we address three basic questions regarding XAI: What?—the main idea behind XAI, Why?—the reasons for exploiting XAI, and How?—the methods and techniques developed for explainability.

1.1. What?

Thanks to the recent development of many machine learning (ML) techniques, solving complex problems has become more feasible; however, they cannot be easily examined after implementation to understand their logic. A deep neural network (DNN) has always been considered a “black-box”, without providing any explanation or reason regarding the decisions made in a human-friendly manner. This black-box model utilizes different ML techniques that take several input features and predict one or more outputs. Therefore, a serious concern arises when evaluating the results in most applications, because the prediction pattern formed during the learning phase is not easily described [1]. Certainly, relying on a black-box model to make decisions without knowing the logic and proof underneath is not of much interest in critical applications [2]. If the application dictates the user’s awareness and trustworthiness of the results, then XAI will be imperative [3]. In other words, XAI is essential for unlocking AI and gaining deep insights into the deep learning process, as mandated by regulations imposed by the European Parliament in May 2018 regarding the use of XAI-based systems: “a right of explanation for all individuals to obtain meaningful explanations of the logic involved” [4].

Recently, XAI has gained considerable attention and become a prominent field of study. According to Google Trends, the global interest in XAI surged in 2021, as shown in Figure 1. The graph depicts the search interest for the term “Explainable AI”.

1.2. Why?

AI systems have consistently demonstrated their effectiveness in providing highly accurate results in a variety of application areas. However, not all ML systems demand interpretability. In such systems, either unacceptable results generally entail no consequences, or, despite potential imperfections, its decisions are deemed trustworthy due to rigorous study and validation in practical scenarios. According to [6], in the presence of an incomplete problem formalization, the need for interpretability arises. In numerous real-world applications, it is necessary to have extremely accurate and understandable AI models. In the literature, multiple incentives are discussed for XAI, depending on the users and applications targeted by the AI system. According to [7], they could be summarized as follows:

• Trustworthiness: In order to use the system predictions in real world applications, the user needs to be trust the applied model. Offering an explanation for a prediction is an important aspect for ensuring human trust and the effective use of ML, if the explanations are faithful and intelligible [8].
• Causality: Causality reveals the cause-and-effect relationship between feature space and possible outputs. Assigning a set of causes for an effect demands wide knowledge, as ML models try to find correlations between the features.
• Transferability: In general, a model is usually trained and tested based on limited data. One reason for pursuing model explainability is to properly utilize a trained model in another domain with similar characteristics. Hence, it can be referred to as reusability.
• Informativeness: Informativeness is the information that an XAI model provides about how it works in order to avoid misconceptions. The model explains the relations inside the box and increases the knowledge of the user regarding the internal process.
• Confidence: A measure that provides an expectation that a decision will prove to be correct or incorrect. The measure should always be assessed within a system where reliability is a concern. Therefore, an explainable model designed for this purpose must offer a confidence level for its predictions.
• Fairness: ML algorithms are products of their data and any bias in the input data will influence the attained results and avoid fair conclusions. An XAI system could reveal imbalances within the data and ensure fairness in ML models.
• Accessibility: Explainability can be considered as a tool for improving internal process of ML models. It gives the users and non-professionals the ability to tune performance based on their requirements.
• Interactivity: This goal is considered for models that require interaction with end-users. The model should describe the decision made and the choices considered, then present the explanation in straightforward, natural language to resolve ambiguity.
• Privacy Awareness: The ability to explain the logic in a model could provide a tool for assessing the privacy. An opaque model may capture sensitive data and cause a privacy breach.

Although providing interpretability and explainability in the system increases the trust of the users and helps in rectifying deficiencies, it adversely affects the performance of the models [7]. The more interpretable the systems are, the lower the performance will be. Consequently, achieving a balance between interpretability and performance is always a challenge [9]. Performance is considered as the accuracy in an out-of-sample prediction of the model with no concern regarding the reason for such an outcome. Conversely, interpretability refers to the ability of the model to convey information in a comprehensible manner to humans [6].

1.3. How?

Experts across various fields have created a variety of XAI tools to help uncover the internal mechanisms of AI models that operate as black boxes [10,11,12]. In general, they can be categorized into three groups based on the sources of explanation, the scopes of explanation, and the dependency on the model [13]. The taxonomy is shown in Figure 2. The source of an explanation could inherently stem from the model structure itself, as seen in simple models like small decision trees (DT) and linear models, or it could be derived by applying post hoc methods to a trained model. In [7], they are referred to as transparent and post hoc explainable models. Then, three levels of transparency are defined: algorithmic transparency, decomposability, and simulatability. The post hoc explainability is also divided into text, visual, local, example-based, simplification-based, and feature relevance explanation techniques.

Based on the taxonomy, the scope of explanations could be global or local. Global interpretability is referred to as the ability to describe the whole logic and any possible outcome of the model. Conversely, local interpretability focuses on explaining the model’s interpretation of specific situations or decisions [2]. When considering dependence on the predictor model, model-specific interpretation tools depend on a single model or a group of models, while model-agnostic techniques can be applied to any AI model. Model-specific techniques have the advantage of accessing the internal structure and weights of a particular model, but they are not easily transferable to other models [26]. Model-agnostic techniques only analyze models’ input–output pairs and generate explanations after the training session. Therefore, they are applicable to any ML model to offer post hoc explanations.

In another review by F. Bodria et al. [27], the explainers are categorized based on different data types: tabular, image, and text. For each type of data, they have distinguished a different type of explanation, as illustrated in Figure 3. For tabular data, feature importance is introduced as the most commonly used type of explanation. The explainer assigns an importance value to each feature, reflecting its contribution to the prediction under analysis. The sign and magnitude of each importance value signify whether the feature positively or negatively influences the outcome. Importance-based methods are effective for domain experts who understand the relevance of the features. Other types of explanations for tabular data include rules, prototypes, and counterfactuals.

Nonetheless, it may be too complicated for a common end-user to understand the feature impact. In these cases, rule-based explanations, prototypes and counterfactuals are more suitable for end-users because of their coherence and use of example-based explanations. For image data, the explanations are provided by the following methods: saliency maps, concept attribution, prototypes, and counterfactuals. The saliency maps change the brightness of image pixels to differentiate visual features in the image. A colored saliency map shows pixels with a positive contribution in red and negative ones in blue. The problem with saliency maps is the confirmation bias. Again, the popular saliency map is suitable for domain experts, as the explanations are at pixel-level, not in the form of straightforward interpretation. Concept attribution generates explanations in terms of higher features called concepts. They compute a score that assesses the probability that the concept selected by a human team affected the prediction. Therefore, this method could provide human-like explanations. The other approaches are based on providing examples to demonstrate the explanation. In contrast to tabular and image data, text data are not structured, making classification more difficult. Text classification refers to the task of tagging or categorizing text based on its content. XAI techniques help to understand which words affect the specific tag assignment. For text data, the explanations are usually provided by the saliency highlighting, attention-based methods among others.

Figure 4 presents a timeline chart for the significant XAI methods proposed since 2011. Model-agnostic methods, such as Shapley values for explaining reinforcement learning (SVERL) [42], diverse counterfactual explanations (DICE) [43], Anchors [44], and testing with concept activation vectors (TCAV) [45], are displayed above the time axis, while model-specific methods, like class activation map (CAM) [46], deep learning important features (DeepLIFT) [47], and layer-wise relevance propagation (LRP) [48], are shown below. During the period from 2017 to 2020, significant growth in both categories is evident. Based on the number of citations, local interpretable model-agnostic explanations (LIME), Shapley additive explanations (SHAP), and Anchors are the most cited methods in the model-agnostic category. In the model-specific category, deconvolutional network (DeconvNet) [49], CAM, and GradCAM [50] are the most frequently cited. The explanation types include feature importance [14], rules [44], counterfactuals [51], saliency maps [52], prototypes [53], and concepts [45]. Moreover, it can be seen that several variations of gradient-based explanatory methods have been proposed, such as saliency gradient (SG) [54], integrated gradient (IntGrad) [55], smoothing gradient (SmoothGrad) [52], and gradient-weighted CAM (grad-CAM) [50]. These methods access the internal signals of the model to estimate the model’s decision process. The gradients are used to realize how the output responds to variations in the input. Perturbation-based explanation algorithms can also be identified, such as individual conditional expectation (ICE) [56], SHAP, LIME, and Anchors. They differ from gradient-based methods because they do not require access to the model’s internal parameters. Instead, they rely only on input–output pairs to analyze the model’s decision-making process. This characteristic makes them generally model-agnostic.

In SHAP, the feature importance is quantified using Shapley values, a concept derived from coalitional game theory. The primary approach involves approximating the original complex model, $f\left(x\right)$, with a simpler and interpretable function, $g\left(x\right)$. This simpler model captures the contribution of each feature to the prediction. All the explanation models provided by SHAP are known as additive feature attribution methods and adhere to the following property [14]:

$$g\left(x^{\prime }\right)={\phi }_0+\sum _{j=1}^M{\phi }_j{x}_j^{\prime }$$

(1)

where $x^{\prime }$ represents simplified input features, ${\phi }_j$ is the contribution of the j-th feature, and M is the number of simplified input features. For a given input $x_j$, the simplified variable $x_j^{\prime }$ is defined with a mapping function, where $x_j=h_x\left(x_j^{\prime }\right)$. The Shapley values, $\phi$, are computed using the following equation:

$${\phi }_j(f,x)=\sum _{k^{\prime }\subseteq x^{\prime }}{\displaystyle \frac{|k^{\prime }|!(M-|k^{\prime }|-1)!}{M!}}[f\left(h_x\left(k^{\prime }\right)\right)-f\left(h_x(k^{\prime }\setminus j)\right)]$$

(2)

where $|k^{\prime }|$ denotes the number of non-zero elements in $k^{\prime }$, and $k^{\prime }\setminus j$ indicates setting the j-th element of $k^{\prime }$ to zero. If Shapley values are used for individual samples, they provide a local explanation. Conversely, when aggregated across samples, they offer a global explanation. The computation of Shapley values can require significant computational resources, especially for models with many features, as it involves evaluating every possible combination of features.

Equation (1) also applies to the LIME method, classifying it as an additive attribution method that provides local explanations in the form of feature importance vectors. For the target sample, LIME aims to approximate a linear model by generating and analyzing synthetic data points (perturbed data) in the neighborhood of the sample being explained. Then, the local feature importance vector is derived from the weights of the linear model. The objective function for this method is to minimize the weighted sum of squared errors between the original model’s predictions and the linear model’s predictions [8]:

$$\xi =\underset{g\in G}{argmin}L(f,g,{\pi }_x)+\mathsf{\Omega }\left(g\right)$$

(3)

where G represents the set of explanation models, $L(f,g,{\pi }_x)$ denotes the squared loss of explanation model g, and $\mathsf{\Omega }\left(g\right)$ is a complexity penalty. ${\pi }_x$ is the proximity measure that quantifies the distance between the sample x and synthetic data points.

Several algorithms have been proposed for generating synthetic neighborhoods, such as DLIME [75], ALIME [76], QLIME [77], leading to different implementations of this method and thus different outcomes. The primary drawback of LIME is its instability, particularly in high-dimensional data where defining a local neighborhood is challenging. Even minor changes in the neighborhood can result in significantly different explanations, making the results unreliable.

Although various explanation methods have been investigated, it remains essential to evaluate their quality quantitatively [78]. The evaluation helps to determine the extent to which the provided explainability aligns with the defined objectives [79]. Additionally, comparing the available explanation methods allows us to identify the most suitable explanation for a specific task. However, a significant challenge arises from the lack of a definitive ground truth when evaluating post hoc explanations. This is primarily because the internal mechanisms of the model remain undisclosed [80].

2. XAI in Manufacturing and Industrial Systems

Manufacturers are consistently seeking innovative strategies to maximize profits, minimize risks, and enhance production efficiency. This is vital for their survival and ensure a prosperous and sustainable future. Through Industry 4.0, data-rich ICPSs, and IoT devices, AI-based and ML-powered techniques are unlocking new opportunities to leverage data for the aforementioned business objectives [81,82]. In Figure 5, major application areas for AI in ICPSs and manufacturing industries have been identified.

Despite the growing demand for AI technologies in the industry, the specialists now need more explanations about how decisions are made and what instructions are given. Hence, for a smooth deployment and acceptance, decisions need to be clear and comprehensible [83]. Although the need for XAI is very clear, achieving this goal can be challenging. The root cause is the complexity of AI systems and the challenge of summarizing it to suit human intuition. In the rest of this paper, we will review the articles that employ XAI methods in ICPSs, IoT, and manufacturing devices and classify them according to use cases mentioned in Figure 5. The selection criteria were designed to identify studies that specifically addressed recent advancements in XAI, showcased practical applications in industrial settings, and were subject to peer review.

2.1. Product Development

AI methods are revolutionizing product development (PD) across industries, as they can reduce the design costs, optimize the product design, enable the early detection of potential issues, and accelerate the design verification through virtual prototyping and simulation tools. XAI aids in gaining the trust of developers and stakeholders by revealing the laws and strategies of black-box models in the learning process.

The history shows a large number of studies regarding the benefits gained from XAI in chemistry and material science in terms of assessing the characteristics of new material systems and structures, such as predicting the material properties of high-strength metal alloys. S. Park et al. [84] proposed employing a Keras-based DNN algorithm to recommend new chemical compositions and fabrication processes, leading to the enhanced mechanical properties of 7xxx aluminium alloys. The LIME algorithm then reveals the significance of certain chemical compositions and processing parameters in this improvement. Yan et al. [85] analyzed the fatigue strength of the steel material by combining extreme gradient boosting (XGB) and light gradient-boosting machine (LGBM) methods coupled with SHAP feature importance graph. The oxidation resistance of a FeCrAl alloy was predicted by neural networks in [86], and the subsequent SHAP method unveiled the contribution of each material. For the first time in [87], SHAP, which was built upon the Gaussian process regression (GPR) model, was employed as an XAI tool to analyze how feature values impact the hardness variation of FeCrAl. Xiong et al. [88] and Yang et al. [89] used SHAP to identify crucial parameters for enhancing hardness of the high-entropy alloys. SHAP values were also used in the modeling of the dielectric constant of crystals [23]. To accurately predict the creep life, ref. [19] compared various AI models, which led to the proposal of a new alloy composition with a predicted creep life exceeding 100,000 h under specified conditions. This work utilized SHAP to deliver clear insights into the impact of individual variables on creep-life prediction for high-temperature components.

XAI has also demonstrated benefits in the field of drug design [90,91]. Authors believe that XAI has the potential to enhance human intuition and expertise in the creation of novel bioactive compounds with specific properties [90].

2.2. Process Control and Automation

Process control (PC) in manufacturing refers to the application of technology to manage and regulate manufacturing processes to ensure a consistent and economic production level. This concept introduces various forms of automation to the field aimed at minimizing the labor needed and the possibility of human error. The automation entails the utilization of sensors, robots, programmable controllers, actuators, interfaces, and software. An AI-driven automated process can significantly enhance efficiency, workplace safety, and reduce human error. Furthermore, it provides valuable insights that contribute to informed decision-making. These insights comprise data analysis, trend identification, predictive modeling, or other methods where the AI system extracts valuable knowledge from the processed data. XAI ensures that those insights are transparent and interpretable to users, which can be crucial for the decision-making processes.

Ref. [92] explored the requirements and implications of implementing XAI in manufacturing environments, emphasizing the demand for transparency and interpretability in AI-driven systems. The study investigated a big car manufacturing plant as a use case. The work focused on formulating and integrating business-oriented XAI requirements into processes. The author believed that the early integration of XAI was essential for realizing the complete capabilities of AI systems. In a similar use case, ref. [93] employed LIME to provide an explanation for the deviation found from the expected production. The explanation specified what elements influenced the production level to infer the root cause of the deviation. In an optimization process, ref. [94] tested the use of XGB and LGBM methods to detect product defects in an injection molding machine. Then, the SHAP method extracted the key variables influencing product defects. In the manufacturing of a semiconductor, SHAP values contributed to a deeper understanding of yield-related factors [95]. Likewise, Shapley values quantified the significance of each process step on the performance of the semiconductor device [96]. A recent study by Zhai et al. [97] introduced a domain-specific, explainable automated machine learning technique (xAutoML). The method was capable of learning the optimal models for yield prediction. The method provided explainability by assessing all elements, such as features and hyper-parameters, locally and globally.

2.3. Inventory Management

Efficient inventory management (InvM) is vital for cutting costs in the supply chain. It aims to optimize inventory levels efficiently to prevent both overstocking and stockouts. ML and data-driven methods could help to categorize commodities, identify demand patterns, optimize inventory levels, automate reordering, and pinpoint bottlenecks in the supply chain [98]. Ref. [99] proposed an explainable k-means approach for multi-criteria ABC item classification. ABC classification is a method used in the inventory management that classifies inventory items based on their importance to the business. Authors believe that the automatic classification process made by the black-box model is difficult to understand and is not sufficient in real life business applications. Consequently, using the SHAP method, the contribution of each criterion is visualized during the construction of inventory classes. In the work by [100], the backorder prediction problem was addressed. A backorder happens when customers can place an order for a product despite that product being temporarily out of stock at the time of the order placement. Several ML models, such as random forest (RF), XGB, and LGBM, were compared to solve this binary classification problem. The SHAP algorithm was utilized to discover the critical features that impact material backorders. For the backorder prediction task, ref. [101] proposed a convolutional neural network-based (CNN) predictive model coupled with SHAP as a global explainer. The LIME algorithm could also analyze individual decisions for stakeholders in order to enhance the interpretability and trust in the model.

2.4. Fault Detection and Diagnosis

The fault detection and diagnosis (FDD) of machinery and ICPSs is one of the main applications of AI in the industry, since the health of components and subsystems, machines, and equipment could significantly impact productivity and efficiency [102,103,104,105]. In the literature, numerous research studies have been conducted to address explainable fault detection, diagnosis, and prognosis in industrial applications. The task of fault detection in system components and subsystems, machines, and equipment mainly resembles the task of anomaly or outlier detection, in the realm of computational intelligence and data analytics. As a definition, anomaly or outlier detection is the process of identifying samples, observations or events that deviate significantly from other instances in data. Early anomaly detection enables quick and informed decision-making, allowing for necessary actions to be taken before critical failures occur. This proactive approach prevents collateral damage to other system components and, in extreme scenarios, ensures personnel safety. Upon detecting an anomaly and notifying the operator, it is essential to provide additional explanations. This ensures that the information can effectively assist the operator in making informed decisions.

In [106], authors devised a practical anomaly detection system for large-scale rotating machinery using the power spectrum of machine vibrations. They employed a CNN visualization technique to obtain an explanation for each prediction. The resulting visualizations effectively guided experts’ focus toward specific machine regions, streamlining fault analysis. In a related study, Grezmak et al. [107] employed CNN and LRP for diagnosing motor faults. They evaluated the CNN model trained on time-frequency spectra images derived from vibration signals of an induction motor. Although many XAI techniques are designed for image, tabular, or textual data, their application to typical industrial data, especially multivariate time-series data, may not adequately enhance human interpretability [108]. Hence, to improve XAI applicability in industrial contexts, more advanced approaches tailored to multivariate time-series data are necessary. M. Dix et al. [21] addressed this limitation in the industrial process automation domain. Three architectures were considered: dense autoencoders, long short-term memory (LSTM) autoencoders, and LSTMs. An autoencoder is an unsupervised model composed of two artificial neural networks (ANN): an encoder, which compresses data, and a decoder, which reconstructs the compressed data. Dense layers are used to learn the nonlinear relationships in data, enabling efficient reduction to a lower-dimensional latent space. This model is used for anomaly detection. The model is initially constructed based on the normal plant data. When a new data sample emerges, the model can predict whether or not the sample represents an anomaly. This can be achieved by using the reconstruction error, which is calculated as the mean squared error (MSE) between the model’s input and output. A threshold is set during the training session by averaging the errors of all normal training samples and adding a standard deviation to this average. If the error of a new sample exceeds this threshold, an anomaly is identified and reported. LSTM is a type of recurrent neural network that can capture long-term dependencies within data layers. It learns temporal patterns in time-series data for future point predictions. By using backpropagation through time, LSTM tracks errors between predicted and actual outputs. It can be used for anomaly detection by reproducing input time windows and measuring MSE to assess prediction accuracy and identify anomalies in data samples. In [21], authors evaluated the detection of 20 simulated failure cases involving various valve failures in separator process time-series data. The study assesses the model’s ability to explain the root causes of these anomalies. The proposed approach involves breaking down the reconstruction error into various signals to guide the operator to the machine likely responsible for a specific anomaly in the plant process. As a variant of LSTM, a Bayesian LSTM integrates Bayesian inference by employing dropout during training to handle uncertainty. In [109], Bayesian LSTM was employed for gas turbine anomaly detection and prognosis, with outputs explained by SHAP. The model included two output layers to assess data and parameter uncertainty, reflecting confidence levels. Performance was evaluated using root-mean-square error (RMSE) and early prediction score, while SHAP explanations were assessed for local accuracy and consistency.

Brito et al. [110] diagnosed faults in rotating machinery using different unsupervised approaches, including minimum covariance determinant (MCD) and isolation forest (IF), etc. Feature importance was then determined using the model-agnostic SHAP and the model-specific local depth-based isolation forest feature importance (DIFFI) method. Finally, root cause analysis identified the most critical features. Similarly, [22] diagnosed a CNC machine faults using IF in a supervised manner, utilizing a large history of sensory data for learning. The SHAP library then generated explanations through custom charts.

In rotating machinery, bearing fault detection is vital for maintaining the performance and longevity of the machine, In [29], an additive Shapley explanation combined with the k-nearest neighbors (KNN) classifier was used to diagnose bearing faults. The algorithm demonstrated significant accuracy on experimental data. This approach was believed to be adaptable to various datasets with different configurations, making it a versatile and effective model. LRP is an explanation method specifically designed for deep convolutional neural networks (DCNNs). It quantifies the contribution of individual input features to the DCNN’s output. Grezmak et al. [111] employed the LRP to clarify the classification decisions of a DCNN developed for diagnosing gearbox faults. LRP helped to identify the time-frequency points in spectra images that most significantly contribute to determining fault type and severity.

A linear motion guide, also known as a linear guide or linear bearing, is a mechanical component designed to provide accurate linear motion with low friction. Kim et al. [34] identified faults in linear guides by training a 1-D CNN using time-domain data and visualizing classification criteria with frequency-domain-based grad-CAM (FG-CAM). Grad-CAM interprets the model through reverse learning process. The proposed method was anticipated to be applicable across various complex physical models. Similar approaches were presented in [35,106,112].

Srinivasan et al. [113] proposed an XAI-based approach for detecting and diagnosing faults in chillers. In this work, the XGB model, an ensemble of classification and regression trees, was employed. The LIME technique was introduced to aid in detecting initial faults, potentially with human assistance. Furthermore, this information enhanced both accuracy and transparency while shortening fault detection time. Likewise, LIME validated the ANN and support vector machine (SVM) models in the study conducted by [114] on air handling unit (AHU) fault detection. The primary goal of this validation was to enhance the model’s trustworthiness for the users.

Remaining useful life (RUL) refers to the estimated duration remaining before a machine or device requires repair or replacement. It provides valuable information for maintenance planning and decision-making. In the study by Hong et al. [115], three algorithms, including CNN, LSTM, and Bi-LSTM, were utilized to accurately predict the RUL of turbofan engines. The impact of each input variable was confirmed through SHAP. Conventional CNN filters lack transparency and may contain noisy or undesirable spectral patterns. This concern was resolved in [116] through the application of SincNet. SincNet encourages the initial convolutional layer to generate interpretable filter kernels. The approach known as Deep-SincNet demonstrated superior performance, greater explainability, broader generalization, noise immunity, and lower implementation costs. In a similar approach, T. Li [25] integrated a wavelet convolutional layer (CWConv) into the initial layer of the CNN, resulting in a wavelet-driven DNN named wavelet kernel net (WKN) that utilizes more meaningful kernels. It was proven that WKN achieves better accuracy, and the CWConv layer also enhances interpretability. The classification results of a CNN model can be interpreted using CAM, which incorporates a final layer known as the global average pooling (GAP) layer. The CAM layer identifies regions of the input image that contribute to a specific prediction. K. Sun et al. [117] applied this approach to diagnose faults in two separate datasets comprising water pumps and cantilever beams.

Table 1. Predictive models used for fault detection and diagnosis.

Models	XAI Methods	References
Bayesian Network	Intrinsic	[118]
Fuzzy Logic System	Intrinsic	[16]
Isolation Forest	SHAP, Local-DIFFI	[22,110,119]
RF	SHAP, ELI5, LIME, CEM	[120,121,122,123,124]
CNN	CAM, grad-CAM, FAM, SHAP, LRP	[34,35,106,107,111,112,115,117,125,126,127,128,129,130,131,132]
RNN	LRP	[133]
LSTM	SHAP, MAE	[115,134]
Bi-LSTM	SHAP	[115]
Bayesian DL	SHAP	[109]
KNN	SHAP	[29,135]
ANN	LIME	[114,122]
DNN	SHAP, LIME, LRP, KBDBN, CEM	[124,136,137,138,139]
SVM	SHAP, LIME, DIFFI, CEM	[114,122,124,135,140,141]
XGB	LIME, SHAP	[113,125,142,143,144,145]

provides an overview of the models employed in the context of FDD, along with their corresponding XAI techniques.

2.5. Predictive Maintenance

Predictive maintenance (PdM), also referred to as “condition-based maintenance" or “risk-based maintenance", involves monitoring system and equipment performance during regular operations to reduce the risk of breakdowns. The ability to predict when maintenance is needed optimizes system lifetime and minimizes downtime [146]. PdM, along with preventive maintenance, aim to schedule maintenance activities in order to prevent system failures. However, unlike traditional preventive maintenance, PdM relies on data collected from sensors and their analysis [147]. Figure 6 compares the costs associated with different maintenance strategies. As a machine experiences degradation over time, repair costs increase. In contrast, the cost of preventive measures decreases. This extended operational lifespan enhances profitability. However, equipment replacement can lead to significant losses, highlighting the importance of pinpointing the optimal time for maintenance, which is facilitated by PdM. In PdM, three kinds of approaches are available: (1) data-driven, (2) model-based, and (3) hybrid. Generally, data-driven methods are preferred because of the abundance of sensory measurements available, and to avoid the complexities associated with building intricate physical models for systems [148]. A primary concern with data-driven methods is the lack of model interpretability. In fact, the interpretability of models poses significant challenges to the adoption of data-driven methods in industrial settings. There are various papers addressing PdM in the industry, but few consider the explainability of predictions to the service engineers.

A recent work by L. Cummins et al. [149] reviewed explainable predictive maintenance (XPM) methods. This systematic review identified deficiencies in the field, particularly the under-utilization of explanatory metrics in PdM. Additionally, the survey introduced a variety of potential metrics from the existing literature that could be adopted in this domain.

Figure 6. Cost relation between the different maintenance strategies [150].

Figure 6. Cost relation between the different maintenance strategies [150].

Matzka [151] utilized DTs for explainability on a synthetic PdM dataset that mirrors real-world industry data. This work evaluates two methods for explaining the classification results of a complex ensemble. While both methods provided overall benefits to the user without incurring significant additional costs, DTs offered explanations of higher quality, albeit occasionally absent. On the other hand, normalized feature deviations consistently provided explanations, albeit of slightly lower quality.

For preventive maintenance in industry, two intrinsic XAI methods were introduced by [16,148]. Upasane et al. [16] employed an interpretable rule-based type-2 fuzzy logic system (FLS) to monitor water pump health, demonstrating superior explainability compared to other transparent models. Langone et al. [148] proposed an interpretable anomaly prediction (IAP) method that benefited from regularized logistic regression as the core model to explain detected anomalies.This methodology breaks down anomaly detection into interpretable steps, providing clarity at each stage and offering a probabilistic anomaly score for future abnormal data events. The method was validated using a time-series dataset collected from a high-pressure pump in a chemical plant.

The prediction of bearing health was conducted in the work by Haiyue et al. [152], where they employed an LSTM-RNN model. Additionally, The LRP technique was used to understand how the model learned from input data and visualize the contribution and relevance distribution across the neural network’s input space. [153] introduced the use of learning fuzzy cognitive maps (LFCMs) to enhance the explainability of LSTMs in PdM for industrial bearings. The authors demonstrated how LFCMs could provide insights into which input features contributed to predictions and how adjusting specific values could potentially prevent faults, offering advantages over the existing explainability methods. In another work by the same authors [154], a novel model-agnostic explainability method, known as the Gumbel–Sigmoid explanator (GSX), was developed to illustrate the contribution of features to predictions.

Ref. [155] evaluated the quantitative association rule mining algorithms (QARMAs) on real-life datasets from two production lines to enable a PdM paradigm. When sufficient data was available, QARMA achieved excellent results in terms of the RUL prediction, outperforming other popular models. However, QARMA was less effective in scenarios with limited sensory data, emphasizing that quality control (QC) measurements alone cannot predict the maintenance needs. Furthermore, QARMA’s rule-based outputs made its models straightforward to interpret, providing a higher level of explainability compared to other deep learning approaches.

2.6. Product Quality

In manufacturing plants, the quality of products may not meet the desired specification due to defects in several factors such as raw materials, production technology, labor skills, storage, and transport facility. Quality control and quality assurance (QA) are vital operations in manufacturing that ensure the company process is working as planned and the final product fulfills the quality requirements. AI solutions have made a major breakthrough in automated inspection and quality control. They outperform humans in inspection accuracy and rate. One of the most powerful types of AI techniques used in the product inspection is computer vision, i.e., the methods used to provide image-based automatic inspection. The use of deep learning and machine vision (MV) provides opportunities for building smart systems that perform thorough quality checks down to the smallest details. A detailed review of MV systems for industrial quality control inspection was presented in [156]. Although there are numerous research studies have been accomplished in this field, a few works have been found that implement explainability in quality management domain.

XAI techniques can clarify how process parameters relate to the quality of the product [22]. Goldman et al. [157] employed XAI techniques, including CAM and contrastive gradient-based saliency maps, to interpret a black-box classifier in the context of assessing weld quality in ultrasonically welded battery tabs. They produced heatmaps to gain insights into distinguishing true positive predictions from false positives. Lee et al. [158] applied various XAI methods to explain defect classification in thin film-transistor liquid-crystal display panels. Methods such as CAM, LRP, IntGrad, guided backpropagation, and SmoothGrad were employed and visualized using the VGG-16 model. Among these, LRP and guided backpropagation were chosen for their well-distributed heatmaps. Furthermore, the authors enhanced explainability by converting model predictions into human-readable texts within a DT framework, achieving maximum interpretability for evaluation by domain experts. Senoner et al. [159] introduced a data-driven decision model aimed at selecting improvement actions for a transistor chip product. Initially, the method prioritizes processes for quality enhancement and subsequently identifies suitable improvement measures. A novel estimation approach for process importance was introduced using SHAP values, quantifying the extent to which production parameters of a specific process influence variations in the overall process quality. Enhancement efforts can be allocated in an efficient way with this approach. The detection of surface defects of steel plates is studied in [160]. Kharal et al. compared nine different classifiers that ultimately found the best figures. Due to imbalanced multi-class data, several methodologies were applied and compared for balancing data, such as oversampling, undersampling and synthetic minority oversampling method (SMOTE). For explainability, the rules were extracted through RF and association rule mining (ARM). In addition, a local model agnostic XAI tool, called the ceteris peribus (CP) technique, was used for feature importance visualization. For a global explanation, the authors found the dependence of the average prediction upon different variables using partial dependence plots (PDPs). The automated classification of fibre layup defects was studied in [161]. A combination of CNN classifiers with smoothed IntGrad, guided grad-CAM and DeepSHAP was found to be suitable in the proposed context. The research findings offer valuable insights to engineers developing camera-based monitoring systems in the composites sector. These insights pertain to designing and implementing sophisticated yet reliable ML solutions. The 3D laser scanners are widely employed as quality control tools in industries such as automotive, aeronautics, and energy sectors. E. Lavasa et al. [162] have developed an AI-based decision support system called Modified PointNet architecture to forecast the point-wise accuracy of the laser scanner throughout the surface of the analyzed part. Additionally, they utilized SHAP to offer a in-depth understanding of the most critical parameters influencing the predictions made by the model.

Table 2. XAI applications in manufacturing industry and ICPSs.

Reference	Year	Use Case	Algorithm	Data Type	XAI Approach	XAI Output
[85]	2020	PD	XGB and LGBM	Tabular	SHAP	Feature Importance
[84]	2022	PD	Keras-based DNN	Chemical composition	LIME	Feature Importance
[86]	2023	PD	NN	Tabular	SHAP	Feature Importance
[87]	2023	PD	GPR	Tabular	SHAP	Feature Importance
[96]	2020	PC	RF, XGB, Catboost	Tabular	SHAP	Feature Importance
[93]	2023	PC	RF	Tabular	LIME	Feature Importance
[94]	2023	PC	XGB and LGBM	Tabular	SHAP	Feature Importance
[95]	2023	PC	RF	Tabular	SHAP	Feature Importance
[100]	2021	InvM	RF, KNN, LGBM, XGB, BB, NN, LR, SVM	Tabular	SHAP	Feature Importance
[101]	2022	InvM	CNN	Tabular	SHAP, LIME	Feature Importance
[99]	2023	InvM	SS-E-k-means	Tabular	SHAP	Feature Importance
[106]	2019	FDD	CNN	Vibration	grad-CAM	Visual Explanation
[114]	2019	FDD	SVM and DNN	Temperature and Speed	LIME	Feature Importance
[111]	2019	FDD	DCNN	Vibration	LRP	Visual Explanation
[35]	2020	FDD	CNN	Vibration	grad-CAM	Visual Explanation
[107]	2020	FDD	CNN	Vibration	LRP	Visual Explanation
[115]	2020	FDD	CNN, LSTM, Bi-LSTM	Temperature, Pressure and Speed	SHAP	Feature Importance
[116]	2020	FDD	deep-SincNet	Current	Intrinsic	Temporal and Spectral Presentation
[117]	2020	FDD	CNN	Vibration	CAM	Visual Explanation
[110]	2021	FDD	KNN, IF, etc.	Vibration	SHAP and Local-DIFFI	Feature Importance
[29]	2021	FDD	KNN	Vibration	SHAP	Feature Importance
[34]	2021	FDD	CNN	Vibration	grad-CAM	Visual Explanation
[112]	2021	FDD	CNN	Vibration	FAM	Visual Explanation
[109]	2021	FDD	DNN	Temperature, Pressure, Speed and Position	SHAP	Feature Importance
[113]	2021	FDD	XGB	Temperature, Flow and Power	LIME	Feature Importance
[25]	2021	FDD	WKN	Vibration	Intrinsic	Feature Map
[142]	2022	FDD	XGB	Tabular	SHAP	Feature Importance
[125]	2023	FDD	CNN-XGB	Vibration	SHAP	Feature Importance
[137]	2018	CybSec	DNN	Tabular	LRP	Feature Importance
[163]	2021	CybSec	Bi-LSTM	Tabular	SHAP	Feature Importance
[20]	2021	CybSec	Conv-LSTM	Tabular	LIME	Feature Importance
[164]	2022	CybSec	CNN	Image	grad-CAM, LIME	Visual Explanation
[165]	2023	CybSec	LSTM	Tabular	SHAP	Feature Importance
[166]	2023	CybSec	BiLSTM	Tabular	SHAP, LIME	Feature Importance
[146]	2020	PdM	LGBM	Tabular	Intrinsic	Feature Importance
[148]	2020	PdM	Regularized Logistic Regression	Temperature, Pressure, and Speed	Intrinsic	Feature Importance
[16]	2021	PdM	FLS	Pressure and Current	Intrinsic	Rules
[24]	2022	PdM	PCA + DCAE	Tabular	Intrinsic	Feature Importance
[167]	2023	PdM	XGB, RF, LR, FFNN	Tabular	SHAP, LIME, Anchor	Rules, Feature Importance
[17]	2024	PdM	FLS	Pressure, vibration, ultrasonic, and Current	Intrinsic	Rules
[28]	2024	PdM	SVM, RF, DT, KNN	Vibration, current, and temperature	LIME, SHAP, PDP, ICE	Feature Importance
[157]	2021	QA	CNN	Tabular	CAM	Visual Explanation
[158]	2021	QA	CNN	Image	LRP and DT	Visualization and Rules
[159]	2021	QA	Non-linear Meta Model	Tabular	SHAP	Feature Importance
[160]	2020	QA	RF	Tabular	ARM	Rules
[161]	2021	QA	CNN	Image	SHAP, Smooth IG, grad-CAM	Visual Explanation
[168]	2022	QA	CNN	Image	grad-CAM	Visual Explanation
[169]	2023	QA	CNN	Image	LIME	Feature Importance

provides a summary of the latest AI and XAI approaches applied to various use cases in the manufacturing industry and ICPS. It also highlights the types of data processed (e.g., images, tabular data) and the forms of explanations provided to users (e.g., feature importance).

3. XAI in Industrial Cyber–Physical Systems

Historically, the evolution from traditional manufacturing to ICPSs marks a significant shift towards smarter, more interconnected production environments and industrial systems. This progression reflects the industry’s growing focus on integrating advanced technologies to enhance precision, adaptability, and security in modern manufacturing processes [170]. Building on our discussion of XAI in the manufacturing industry, it is essential to consider how these technologies are integrated within ICPSs.

The rapid development of information technology demands sophisticated cyber–physical systems (CPSs) that integrate computing, communication, and control (3C technologies), crucial for Industry 4.0 and various applications. CPSs integrate computing, networks, and physical environments to achieve real-time systems that monitor and control physical entities through a computing core [171]. Based on the OSI model [172], CPS architectures include seven layers: a physical layer, a data-link layer, a network layer, a transport layer, a session layer, a presentation layer, and an application layer.

Cybersecurity is crucial for CPSs, which are vulnerable to cyberattacks threatening data confidentiality and infrastructure [173]. Cyber–physical attacks exploit network vulnerabilities to disrupt physical systems using methods like malware injection and man-in-the-middle attacks. Stealthy attacks, such as replay and zero-dynamics attacks, manipulate control signals undetected by conventional systems. Therefore, cyber–physical security must address both the physical and the network aspects [174], unlike traditional cybersecurity. It includes passive measures, like encryption, and active measures, such as recovery strategies, to ensure CPS resiliency [175]. Cyber-resilience ensures CPS function despite attacks, essential for sectors like healthcare and manufacturing.

Implementing AI in ICPSs without explainability introduces significant risks, including security vulnerabilities, undetected biases, misuse, unfair outcomes, and lack of accountability. XAI mitigates these risks by ensuring transparency, fostering trust, and enabling proper oversight. XAI enhances cybersecurity by allowing users to monitor and analyze system behavior, thereby improving intrusion detection and response. Additionally, XAI helps reduce costs, prevent accidents, and ensure legal compliance.

XAI can help in enhancing the transparency and trust of AI models used to support CPS operations. Combining XAI and CPS fosters safe, secure, accountable, and resilient systems driving significant societal and environmental benefits [176]. The current research on XAI for CPS is limited, focusing on specific applications like medical and industrial CPS. Challenges include biases, the lack of standardized methods, and the inadequate handling of time-series data.

Cyber-Security

The advent of connected and digital manufacturing devices and systems promises to significantly increase the efficiency and speed of manufacturing. This optimizes supply chain processes and allows for effective automation. In ICPS automation, multiple layers of critical connections are established, comprising cyber–physical components, systems, networks, and controls. This configuration increases both the complexity and vulnerability of ICPS to cyber-security (CybSec) threats [177].

Ensuring efficient manufacturing automation is vital for the success of industries. Hence, by using AI techniques, we can detect abnormal behaviors in ICPSs and respond quickly to prevent security incidents. Although AI models guarantee high detection accuracy, no explanation is given for the decision made, specifically in ICPSs, where information is often abstract. Here, we review the works that bring explainability into their solutions for cyber threat detection.

Amarasinghe et al. [137] proposed a framework to identify DoS attacks in ICPSs. A DNN was used to detect anomalies. Then, a post hoc explanation was generated using the LRP technique to assess the relevance of input features in explaining predictions made by the trained DNN model. The explainer also provided the user with the confidence of the prediction and a textual description of detected anomalies. The evaluation was performed using a subset of the NSL-KDD dataset. In a related work, Hwang et al. [163] employed multiple Bi-LSTM networks to detect security threats in an ICPS ecosystem. The approach aimed to reduce false detections within each model, while also detecting a broader range of anomalies across all models collectively. Then, the contribution score of each feature was provided using SHAP. A novel Conv-LSTM-based autoencoder framework for explainable attack detection in time-series data collected from the industrial internet of things (IIoT) has been proposed by I.A.Khan et al. [20]. They demonstrated that the method effectively detects both known and unknown attacks in IIoT environments. With the integration of the LIME method into the framework, the most relevant features were recognized as the basis of interpretation.

Le et al. [177] developed a visual analytic framework for monitoring and assessing complex automation networks in real time to highlight possible errors, warnings, and malicious threats. In the most recent research study, a federated learning-based (FL) explainable anomaly detection (FedeX) has been proposed to detect and analyze anomalies in ICPSs. FL has demonstrated its potential as an effective approach for edge computing in distributed environments. The authors demonstrated that the FedeX is considerably accurate, fast, and lightweight compared to compelling methods. After the detection process, the XAI-SHAP model interprets and validates the model, identifying the elements causing anomalies in ICPSs.

In [178,179], SHAP has been used to provide local explanations over the results of intrusion detection systems trained on NSL-KDD. In [180], an XAI-based scheme, based on SHAP, LIME and RuleFit, has been developed for detecting intrusions. In [181], a SHAP-based intrusion detection system has been developed for DNS over HTTPS attacks. A framework has been proposed in [182], which makes use of SHAP to provide local and global explanations for a deep learning-based intrusion detection system. In [183], LIME and SHAP are used to explain the outcomes of a trained multi-layer perceptron model for detecting intrusions in IoT devices. An intrusion detection system, called X-CANIDS, has been proposed in [184] for detecting intrusion, including zero-day attacks in a controller area network. An end-to-end framework has been developed in [185] to assess XAI techniques for network intrusion detection tasks. In [186], SHAP and LIME are used to interpret the predictions of deep learning-based intrusion detection systems. In [187], an explainable deep learning-based intrusion detection system has been developed for Industry 5.0. This method integrates a bidirectional-gated recurrent unit, a bidirectional LSTM network, fully connected layers, and a softmax model along with the SHAP method, to determine the most impactful features on the predictions. Refs. [188,189,190] review recent intrusion detection techniques and provide insight on how XAI can be used to improve the state of the art in the field.

Clarity and trust are vital for the adoption of XAI in ICPSs. XAI builds trust, accountability, resilience, and legal compliance, assisting in safety and security through interpretations. Future CPS should feature context-aware interpretations and self-explainability in order to adapt to non-stationary environments.

4. Discussion and Future Directions

As AI becomes increasingly integrated into manufacturing and industrial processes, ensuring the transparency and interpretability of AI models becomes crucial. In this study, we found that XAI techniques serve more as a means to enhance the trustworthiness and reliability of AI systems in CPS and smart manufacturing, enabling human operators to comprehend and validate the decisions made by these intelligent systems. T.C. Chen [191] believes that XAI development in manufacturing can enhance the practicality and effectiveness of existing AI technology by explaining the reasoning process and integrating easy to interpret visual features. While XAI techniques are commonly used in domains like medicine, service, and, education, their application in manufacturing remains limited, highlighting an imperative and an opportunity for its integration in this sector. From a systematic literature review by [192], it was shown that industrial and manufacturing applications accounted for about 10 percent of the domains where different XAI methods were applied. Following the review of the implemented explainable models, a number of comments and potential research directions are suggested below:

• Focus areas of XAI in manufacturing: In the literature, predictive maintenance, fault diagnosis and prognosis emerge as the most extensively discussed fields where XAI is applied within manufacturing. Conversely, other use cases such as product development, process control, and inventory management are rarely addressed or explored in this context.
• Need for tailored explanations: It was noted that the majority of XAI explanations are primarily conveyed through feature importance. However, given the diverse range of users in the manufacturing industry, including machine operators, engineers, scientists, and managers, it is essential to tailor explanations based on the specific expertise and comprehension levels of the intended users, thus ensuring optimal understanding and usability.
• Limitations of existing XAI methods: It was observed that most of the explanations were generated using post hoc methods like SHAP and LIME, covering both global and local perspectives. Nevertheless, these methods exhibited limitations in delivering real-time insights and actionable recommendations for immediate decision-making.
• Advancing XAI techniques: There could be further advancements in XAI techniques tailored specifically for manufacturing applications. This could include the development of hybrid approaches that combine traditional AI methods with XAI techniques to achieve optimal transparency and interpretability while maintaining high performance. Besides, addressing challenges related to scalability, robustness, and compliance with industry regulations will be crucial for the widespread adoption of XAI in industrial settings.
• Real-time XAI: There could be a focus on real-time XAI capabilities to enable dynamic decision-making and adaptation in rapidly changing industrial environments.
• XAI and emerging technologies: The integration of XAI into emerging technologies such as the IoT and edge computing could open up new possibilities for enhancing transparency and interpretability in this domain.
• Regulatory compliance: In the context of ICPSs and industrial settings, adherence to industry standards and regulations is particularly critical due to the potential impact on safety, security, reliability, and performance. For example, safety-critical domains such as healthcare, automotive, and aerospace have stringent regulatory requirements governing the development and deployment of AI systems to ensure patient safety, vehicle reliability, and aviation security. Therefore, any future trends in XAI within ICPSs and industrial settings must take into account the unique regulatory landscape of each industry. This includes conducting thorough assessments of regulatory requirements, integrating compliance measures into the design and development process of AI systems, and establishing transparent mechanisms for documenting and auditing XAI implementations. Furthermore, collaboration between industry stakeholders, regulatory bodies, and AI researchers is essential to address regulatory challenges effectively and ensure that XAI techniques align with industry standards and regulations. By prioritizing compliance and transparency, organizations can build trust in AI systems and facilitate their widespread adoption across diverse industrial sectors.

5. Conclusions

This study offers valuable insights into the integration of XAI in various manufacturing applications and industrial cyber–physical systems. In the introduction, we discussed the importance of XAI, its implementation, and introduced a taxonomy and categorization of XAI methods. We also highlighted advancements in XAI techniques. Throughout the study, we explored various use cases in manufacturing and outlined different XAI approaches documented in the literature. Post hoc methods, particularly the model-agnostic techniques LIME and SHAP, are the most popular approaches used. We observed the limited utilization of XAI in specific use cases, such as product development and process control. Consequently, there are substantial research opportunities in manufacturing, particularly in sensitive and safety-related domains. Promising areas include the development of self-explaining approaches and exploration into hybrid models that combine traditional AI methods with XAI techniques. These approaches have the potential to enhance transparency and interpretability while maintaining high performance. Moreover, to ensure optimal understanding and usability, it is crucial to tailor XAI explanations to the diverse range of users in the manufacturing industry. Future research could also focus on real-time XAI techniques, integrating emerging technologies with XAI, and developing regulatory-aware XAI solutions. In conclusion, the application of XAI in manufacturing industries and ICPSs could enhance the decision-making procedure, boost operational efficiency, and ensure the reliability of AI-powered systems. By providing clarity, XAI also significantly contributes to improving safety and security within ICPSs.