Improving the Consistency of the Failure Mode Effect Analysis (FMEA) Documents in Semiconductor Manufacturing

Abstract

Digitalization of causal domain knowledge is crucial. Especially since the inclusion of causal domain knowledge in the data analysis processes helps to avoid biased results. To extract such knowledge, the Failure Mode Effect Analysis (FMEA) documents represent a valuable data source. Originally, FMEA documents were designed to be exclusively produced and interpreted by human domain experts. As a consequence, these documents often suffer from data consistency issues. This paper argues that due to the transitive perception of the causal relations, discordant and merged information cases are likely to occur. Thus, we propose to improve the consistency of FMEA documents as a step towards more efficient use of causal domain knowledge. In contrast to other work, this paper focuses on the consistency of causal relations expressed in the FMEA documents. To this end, based on an explicit scheme of types of inconsistencies derived from the causal perspective, novel methods to enhance the data quality in FMEA documents are presented. Data quality improvement will significantly improve downstream tasks, such as root cause analysis and automatic process control.

1. Introduction

Two centuries have passed since the industrial revolution transformed manufacturing processes. Now in the era of industry digitalization and increased use of data, causal domain knowledge is, more than ever before, crucial. For example, in automated data analytics processes, causal domain knowledge inclusion helps to avoid biased results [1] and increases data analysis algorithms’ robustness (e.g., increases machine learning algorithms’ robustness [2]).

Risk assessment and root cause analysis are two practices performed in the industry that capture and document causal domain knowledge. Almost a century has passed since the Failure Mode Effect Analysis (FMEA) process was first proposed as a tool for risk assessment [3]. In the FMEA process, a multidisciplinary team often uses the brainstorming method to identify lists of the failure modes and their possible root causes and potential effects. In addition to other information such as corresponding authors’ information and the scope of the FMEA process, these lists are saved in a document (the FMEA document) typically using a standard tabular format. This tabular format’s columns contain the failure mode, potential effects, and possible root causes, in addition to other columns such as risk priority number, detection, etc. The FMEA process successfulness as a risk assessment tool depends on (i) the complete and accurate identification of potential failure modes contained in a system and (ii) the rigorous evaluation of the risk level of these failure modes [4].

Now, the FMEA process is widely used in many industries. For example, the FMEA process is used in the design and validation of a hydraulic torque converter [5]. Additionally, the diagnosis of the bearing condition using the FMEA process is presented in [6]. Additionally, in [7], Bayesian networks are built from the FMEA process used by Markov decision processes to model uncertainties within unmanned aerial vehicles. Thus, in the context of extracting causal domain knowledge in the industry, FMEA documents represent a valuable data source.

As highlighted by previous research, the FMEA process is error-prone. As a result, FMEA documents suffer from data consistency issues. For example, Bluvband et al., in [8] argued that due to the insufficient comprehensibility of the brainstorming session, cases of missing information might occur (i.e., by omitting failure modes). Consequently, many approaches are proposed to differently aggregate sources of information to improve the brainstorming sessions’ comprehensibility. Thus, solutions based on ontologies are presented in [9,10,11] to satisfy the requirement to share, reuse, and maintain FMEA knowledge. Such solutions are labor-intensive and time-consuming. As such, text mining algorithms are proposed to extract a list of frequent failure modes and build the standard failure mode vocabulary (e.g., the method proposed in [12]).

Additionally, FMEA process performance is limited in capturing rich information about implicit causal relations expressed between its columns. In order to enrich these causal relations with more information, a more sophisticated methodology, namely Failure Modes, Mechanisms, and Effects Analysis (FMMEA), is proposed in [13].

Since our research aims to extract causal knowledge from FMEA documents, which is, by design, limited in capturing rich information about the causal relations, we are faced with consistency impairments in these documents. Moreover, these consistency impairments, to our best knowledge, were not previously addressed. Especially when the intention is to extract causal domain knowledge, these consistency impairments limit the effectiveness of the extracted knowledge for downstream tasks. Therefore, in this paper, we argue that consistency impairments concerning the implicit causal relation between the FMEA documents’ columns (i.e., root cause and failure mode, and failure mode and effect) are also significantly important.

Ideally, in a given FMEA cell, a short, descriptive text represents a single concept. In this case, a concept is a separable (identifiable) phenomenon that acts either as (i) a root cause, (ii) an effect of the failure mode observed in the product characteristics, or (iii) a failure mode as an intermediate state in the causal chain. However, based on the analysis of actual FMEA documents from a semiconductor manufacturing company, the majority of data consistency impairments are attributed to one of two main categories: (a) FMEA documents’ cell consistency; (b) FMEA relations consistency (i.e., consistency impairments concerning the implicit causal relation between the FMEA documents’ columns). Figure 1 depicts our observations concerning consistencies found in actual FMEA documents. In the category of FMEA documents’ cell consistency impairments, we noticed cases of merged cells wherein the short text of a single cell comprises multiple concepts or even relations between multiple concepts, e.g., a causal chain of multiple causes and effects. In the category of FMEA relations consistency, we noticed the following:

• Cases of missing information in the causal chain, where the documented relation actually describes a subsequent effect of the cause but skips its direct effect;
• Cases of conflict in the direction of the relations, i.e., the direction of the causal effect relation is reversed.

These consistency impairments are mainly attributed to the manual creation of FMEA documents by human domain experts, the broad definition of FMEAs’ columns, and the complex nature of causal relations (e.g., many-to-many relations with transitive perception [14]).

Hence, any disagreement among experts on the exact semantics of these columns will increase consistency impairments within these documents. Formally, these documents will still conform to the structure (i.e., tabular); their content no longer aligns with the intended semantics. Consequently, in the era of industry digitalization, these consistency impairments seriously limit the effectiveness of including causal domain knowledge documented by the FMEA process.

In this paper, we first propose methods to improve the consistency of FMEA documents by defining a classification scheme to provide an improved understanding of the consistency impairments with respect to the causal relations expressed in the FMEA documents. The classification scheme is derived from domain experts’ perception of the short text in the FMEA cell. The classification scheme can be applied in the form of metadata annotations to the tabular data to assess the data consistency for a given FMEA. Still, the dataset size limits the feasibility of a manual annotation process, i.e., in practice, it is not possible to manually label all available datasets in the production environment. As such, we leverage advances in artificial intelligence and natural language processing for an automated or at least semiautomated classification method. Consequently, the reversed direction of causal relations, which indicates swapped cells in the FMEA, is effectively addressed via this classification method combined with experts’ logic. Moreover, we are able to distinguish different types of causal relations based on the classes of the concept that it connects. We hypothesize that these classes and these different types of relationships would be extremely beneficial for downstream tasks, such as information retrieval and knowledge discovery, effectively addressing the challenges related to missing information issues. Next, this paper proposes a pattern-based method for merged cell identification in FMEA documents. This method leverages predefined patterns for causal cues to identify merged FMEA cells. Additionally, a patterns extraction method calculates the Mutual Information based on the co-occurrence of the terms on FMEA cells to identify merged cells is also presented.

In summary, our contributions are as follows:

• We highlight the importance of including causal domain knowledge to improve the data analytics methods robustness;
• We propose to use FMEA documents as a source of causal domain knowledge;
• We highlight the main challenges with respect to manually written FMEA documents and their consistency impairments for extracting causal domain knowledge;
• We propose a framework to address these challenges first based on explicitly defining these consistency impairments types;
• Based on manually labeled examples from real-world FMEA documents, we derive methods for the classification and identification of consistency impairments;
• The improved FMEA documents can then be used for many downstream tasks.

2. Materials and Methods

Based on the analysis of actual FMEA documents from a semiconductor manufacturing company, most data consistency impairments are attributed to one of two main categories: (a) FMEA documents’ cell consistency or (b) FMEA relations consistency, where our proposed methods address both types of inconsistencies.

Our primary approach relies on causal data science for checking the consistency of FMEA documents. Causal data science is concerned with the underlying data generation process. Thus, causal data science aims to adjust for spurious correlations present in the data. The spurious correlations could be a result but not limited to confounding bias or collider bias. Biases are translated in the FMEA documents to cases of merged cells. One example for confounder bias could be: “layer thickness A” affect “Functional parameter A and Functional parameter B”. Collider bias may look like: “layer thickness A is out of spec” due to “process A deviation or process B deviation”. In the cases of merged cells due to biases (i.e., confounding and conditioning on colliders), they result from how the FMEA documents are crafted, i.e., first identify the failure modes, then place the potential effects and the root causes. Thus, the failure mode could be a collider for the root causes or a confounder for the effects.

The FMEA documents do not seem to support the cases of interaction between multiple concepts, based on our study of FMEA documents. Thus, cases of merged cells that describe the interaction between concepts are typical to occur. An example could be: “layer A stress in combination with layer B thickness limit violation” causes “functional parameter C to deviate”. The “layer A stress” alone is insufficient to cause “the functional parameter C to deviate”. Also “the functional parameter C deviation” is not caused by “layer B thickness limit violation” alone. In such cases, the merged cell might contain additional information about the relationships of the individual concepts. However, in some cases, not all the individual concepts are stated. To handle such cases, we follow Vanderweele and Robins steps as proposed in [15]. Vanderweele and Robins studied the interaction between concepts introducing sufficient causal directed acyclic graphs. The sufficient cause is added to the original causal directed acyclic graphs as an artificial node to describe the interaction between the causes. In addition, the individual concepts are also represented in the graph and connected to the artificial node. The interaction of such concepts happens only when these concepts fulfill the role of causes in causal relations. Thus, in the FMEA documents, such merged cells of cases are only occurring as failure mode or root cause.

We also observed cases of merged cells that contain an entire causal chain of multiple causes and effects (e.g., “layer thickness A violation causes layer thickness B violation”). We assume that this is mainly attributed to the manual creation of FMEA documents by human domain experts, in combination with a broad definition of FMEA’s columns. For example, domain experts may try to document causal chains, which stretch across multiple cause and effect pairs.

To summarize, in the category of FMEA documents’ cell consistency impairments, we noticed cases of merged cells, for which we further categorized into four categories, based on our observation of real-world FMEA documents. These four categories of merged cell types are depicted in Figure 2 and can be described as:

(a).

Merged cells containing a causal chain;

(b).

Merged cells as a result of confounding bias;

(c).

Merged cells as a result of bias caused by conditioning on a collider;

(d).

Merged cells caused by describing the interaction between concepts.

As a result of merged cells, the usability of the FMEA document for further processing is seriously limited. In many cases, these limitations can be attributed to the absence of standard terminology for the possible concepts. This is especially the case in many complex and rapidly developing industries, such as semiconductor manufacturing. Here, domain experts typically describe the concepts using short text, which comprises domain-specific language and many abbreviations.

In comparison to other proposed approaches concerned with FME cells’ consistency, most approaches are only concerned with the consistency of failure modes to achieve a standardized description. Our research is concerned with the effect, root causes, and failure modes. However, before addressing the standardization of all FMEA cells, we noticed cases of merged cells, which we address in part of this paper. Similar to [12], we opted for a data-driven approach that requires limited labeling effort.

In the case of identifying merged cells, multiple indicators could be devised. For example, in the case of individual cells containing entire causal chains, causal relation extraction from texts could be of value. In [16], the authors survey causal relation extraction from text and describe the most relevant approaches. They distinguish between explicit and implicit causal relations. For the case of FMEA documents, the causal relations are most likely to occur explicitly, i.e., the causal relation is articulated explicitly in the short text of the FMEA cell. Thus, we propose to leverage causal patterns identified as in [17] to identify merged cells containing causal cues.

The first indicator (i.e., causal cues) and the proposed methods to detect mainly assumes consistent use of cues through the data (which is typically not the case) and does not cover cases (b), (c), (d) of merged cells. Thus, it yields high precision, but limited recall. As a response, we build upon the work presented in [18], leveraging the Mutual Information (MI) to detect merged cells. The proposed approach to extract the intra-sentential patterns, which are meaningful combinations of terms (e.g., “voltage threshold”, “leakage current “, etc.), from FMEA cells. To construct patterns, It calculates MI between the terms based on their occurrence in the text of FMEA cells. The proposed approach is based on six steps, explained as follow:

Step 1 FMEA cells’ short text n_grams: In this step, the algorithm splits the FMEA short text into a list of terms based on spaces. The algorithm collects all the possible n_grams (i.e., sequence of terms of a length n) that occurred in the FMEA cells’ short text.

Step 2 n_grams occurrence extraction: This step counts the FMEA cells where an n-gram occurred in its short texts.

Step 3 n_grams Mutual Information calculation: This step calculates the collected n-grams Mutual Information based on the occurrence of n_gram terms in FMEA cells (i.e., based on the occurrences in FMEA cells). Hence, an n_gram with higher Mutual Information is more likely to represent a pattern. Whereas high Mutual Information indicates a large reduction in uncertainty, low Mutual Information indicates a small reduction, and zero Mutual Information between a set of terms means the terms are independent. The Mutual Information is calculated based on the occurrence of the terms in the FMEA cell using the equation below:

$$MI\left(t_1,t_2,...,t_n\right)=P\left(t_1,t_2,...,t_n\right)\log \frac{P\left(t_1,t_2,...,t_n\right)}{{{\displaystyle \prod }}_{i=1}^{i=n}P\left(t_i\right)}$$

Step 4 Patterns extraction (n_gram filtering): The collected n_grams are filtered based on two criteria (their number of occurrences in FMEA cells and their Mutual Information). Thus, this approach is performed using two hyperparameters: the Mutual Information threshold (MIth) and the occurrences threshold (OCth). These two hyperparameters increase the approach precision in detecting merged cells for a range of values where MIth > 0 and OCth > 1. However, it decreases the recall. The filtered n_grams are considered as patterns.

Step 5 Top Patterns and subpatterns extraction (pattern clustering): A Top Pattern is a pattern (n_gram) that is not contained by any other pattern. In this step, all the Top Patterns are extracted from the filtered n_grams resulting from Step 4. Furthermore, all the patterns from the same filtered n_grams contained in a Top Pattern are collected and considered subpatterns. The results of this step are a number of clusters represented by the Top Patterns and the member of this cluster, which are the subpatterns.

Step 6 Merged FMEA cell detection: A rule-based approach leverages the patterns extracted from Step 4 and Step 5 to predict the merged FMEA cell. This step matches all the patterns from Step 4 contained in the FMEA cell then checks if there is a Top Pattern (cluster) that contains all the matched patterns. If this criterion is not met, the algorithm predicts the cell is merged.

The proposed approach to detect merged cells based on the intra-sentential pattern is highly dependent on the dataset where the Mutual Information is calculated. Thus, the MIth and OCth need to be adapted.

As for the causal relation documented during the FMEA process, it depends on the scope of the FMEA and the internal agreement between the multidisciplinary team creating the FMEA. This agreement typically is not documented. Thus, for FMEA documents conducted on the same scope, significant differences in the semantics of the concepts described in the columns are found. This is worsened by the broad definition of the FMEA columns. For example, while Bluvband et al. argue in [8] that the definition of failure is too narrow and that this might lead to the omission of failure modes, we believe that the definitions of the root causes and effects are too broad. This claim is also supported by observing common practices in datasets for knowledge graphs that include causal relations such as Atomic [19] and Glucose [20]. In such datasets, the authors define different types of causal relations and attempt to describe the semantics of causal relations.

Consequently, the descriptions of the effects and the root causes are less consistent. Thus, because the transitive perception of causal relation might lead to the description of a distant cause or effect skipping the direct one. For example, given the causal chain [A]-[causes]->[B]-[causes]->[C] (A, B, C are three concepts in the FMEA documents), due to the transitive perception of causal relation [A]-[causes]->[C] is also valid and might be documented in the FMEA documents. However, the information about the mediation of the causal effect between A and C through B is missing, especially if the original causal chain [A]-[causes]->[B]-[causes]->[C] is not documented in FMEA documents. This missing information is critical, especially in industries with a long manufacturing process, where the manufacturing typically extends for hundreds of processing steps.

Additionally, in some cases, the direction of the causal relation is reversed. For example, [C]-[causes]->[B] is documented instead of [B]-[causes]->[C]. This impairment might be attributed to human error or ambiguity of the temporal aspect while creating the FMEA document.

In general, this might not pose a large problem for domain experts to cope with this missing information and automatically correct for the reversed relation due to their profound knowledge of the domain. However, in the case of automated data analytics, missing information and contradictions in the documented causal relation direction severely affects the usefulness of the data analytics method, especially in the considered causal model.

As a response, we propose to define a classification scheme to provide an improved understanding of the consistency impairments with respect to the causal relations expressed in the FMEA documents. As such, it is critical to firstly identify the concept classes that logically should be represented by the data (i.e., matching content interpretation by domain experts). In many cases, the concept classes are domain-specific and typically cannot be inferred from the data alone; they need to be established with the help of domain knowledge. To this end, we propose a set of rules to govern the identification of the classes:

• Concepts classes need to be completely separable;
• Concepts classes need to allow for the assessment of the causal relations consistency between concepts;
• Concepts classes need to be aligned with domain experts’ perception of the cells’ short text content.

For our case study on FMEA documents, the identification of individual concept classes is rooted in actual production lines and based upon the knowledge of experienced domain experts. They were interviewed in order to establish our concept classification scheme, which forms the base for the FMEA documents consistency improvement methods. While our work is focused on causal relations found in FMEA documents of the semiconductor manufacturing industry, other domains also require domain-specific classification schemes, also taking expert knowledge and its formalization into consideration.

First, on a high level, in the semiconductor manufacturing company under study, FMEA documents can be split into multiple types depending on the respective scope of the FMEA. One can distinguish between Process FMEA documents (P_FMEA) and Unit Process FMEA documents (UP_FMEA). An individual P_FMEA contains multiple causal chains, each of which might be associated with numerous processes used in a product production line. UP_FMEAs include causal chains only belonging to single processes, which in turn could be related to multiple products’ production lines.

The expected concepts described in the P_FMEA documents are different from those expected to be described in UP_FMEA documents. In P_FMEA documents, one can expect concepts physical characteristics’ deviation of the products’ structures, which causes concepts describing functional characteristics’ deviation of the product. Many reasons could cause these physical characteristics’ deviation. However, due to the high degree of manufacturing automation and the highly controlled environment (i.e., the typical manufacturing conditions in the semiconductor manufacturing industry), the most relevant causes stem from the so-called unit process deviation.

In the UP_FMEA documents, it is expected to find concepts describing the so-called unit process key parameters deviation that causes the unit process deviation (which is also described in P_FMEA documents). Additionally, in the UP_FMEA documents, the root causes of the unit process key parameters deviation follow the 5M (Man, Machine, Material, Method, Measurement) risk-management model similar to the Ishikawa causal diagram. To connect the two types of FMEA documents, experts aim to link unit process cause in the P_FMEA document to the respective unit process effect of UP_FMEA. The linking concept (i.e., the unit process deviation) is abbreviated as UP-C/E. Domain experts further explained that the product functional characteristics’ deviations are caused by deviations in physical characteristics of the product structures, while the reversed direction is not valid.

To summarize, leveraging the information acquired from the discussions and multiple rounds of interviews with domain experts, we are finally able to distinguish five distinctive concepts classes: Functional, Physical, UP-C/E, Parametric, and 5M. The concept classes and their causal chains are depicted in Figure 3. To readers who are familiar with measurement data typically found in a modern semiconductor manufacturing FAB, these measurements types are also in line with the proposed concept classes. Whereas in [21], Qin et al. summarized the measurement data types available in semiconductor manufacturing as follows:

• Real-time trace data at the tool level reflects equipment health conditions;
• Integrated metrology or inline metrology provides geometric dimensions;
• The sample and final electrical test provide data about the products’ electrical properties.

Thus, domain experts could map defined concept classes to the corresponding measurement data types. Namely, concepts belonging to the Functional, Physical, and Parametric concept classes could be mapped to the final electrical test, integrated metrology, real-time trace data, respectively. This alignment intends to enhance the reuse of FMEA knowledge concerning root cause analysis. However, the benefits of this alignment will be addressed in future research.

Consequently, the defined concept classes can be applied in the form of metadata annotations to the existing FMEA documents. This metadata allows for causal relation consistency assessments of a given FMEA document. Thereby, for an annotated causal relation found in the data, one can check for consistency via compliance with the consistency rules. Examples of such rules are:

(i)

A causal relation from a concept that belongs to the Functional class to a concept that belongs to the Physical class is considered to have a consistency status of reversed;

(ii)

A causal relation from a concept that belongs to the UP-CE class to a concept that belongs to the Functional class is considered to have a consistency status Missing Information due to the missing concepts belonging to the physical class.

Here, causal relations are found in the FMEA document that does not adhere to the consistency schema; one can distinguish two cases:

Case 1 Missing information: In this case, one or more intermediate concepts are missing from the causal chain. In an automated setting, this missing information would need to be completed using other FMEA documents or other data sources. These concepts might have other (root) causes, which also need to be considered and completed. In general, this case is not trivial and mostly needs to be dealt with on a case-by-case basis.

Case 2 Reversed direction: In this case, the direction of the causal relation is reversed according to the consistency schema. Such cases are attributed to the manual creation of the FMEA documents. In such cases, the data might be complete while inconsistencies concerning the classification scheme and defined causal chain can be observed. In this case, the problem can be automatically detected and automatically rectified.

In an actual scenario, the number of data is typically too large to allow for a complete manual classification of all the available documents according to the classification scheme, i.e., to conduct a large-scale manual annotation process by (highly paid) experts. (This also could be the case in transferring the FMEA to FMMEA). Yet from the perspective of dataset sizes required to train contemporary machine learning models, it is an open question if the available data fulfills these requirements, primarily since FMEA documents we consider comprise: (i) just short text snippets that (ii) do not follow grammatical rules (i.e., no full sentences), and (iii) represent domain-specific language (i.e., terms not found in generic dictionaries).

Historically, the task of building classifiers for textual input data has been approached following a rule-based approach [22,23]. Such methods are highly dependent on the experts’ knowledge, which is consecutively modeled as a set of guiding rules. As a welcome consequence, these systems do not require ground truth data for training. Thus, these systems evolve naturally by extending and optimizing the rules set. While these methods give good results for well-constrained and well-understood environments, the drawbacks of such methods concern the scalability and the difficulty of identifying a complete and consistent rule set (e.g., no loops or contradicting rules). As an alternative, machine learning methods are devised to learn directly from the data. The initial challenge associated with text classification is transforming the text into a representation suitable for machine learning models, a process comprising the tasks of feature extraction and feature engineering. Rule-based feature extraction faces similar challenges as rule-based classification methods. At the same time, general text classification and representation learning benefit highly from deep learning approaches [24,25]. Thus, for this research, we opted for an end-to-end deep learning classification model. Our text classification model consists of a preprocessing pipeline, an embedding layer, a recurrent network layer, and a fully connected layer as an output layer. Namely, we test four recurrent neural networks layers: LSTM [26], GRU [27], and their bidirectional variation [28]. The classifier is trained on a multilabel, multiclass classification using weighted binary cross-entropy loss to account for the classes’ imbalance in the training dataset. The loss function is calculated using the equation:

$$dL=-\left[w_p\times y_{true}\times \log \left(y_{pred}\right)+w_n\times \left(1-y_{true}\right)\times \log \left(1-y_{pred}\right)\right]$$

where $w_p$ and $w_n$ are positive and negative class weights, respectively, for each concept class.

Our classification model is achieved as follows: First, the FMEA dataset is split into three datasets: training, testing, and development. The training and development datasets are used in the training phase. During the training phase, preprocessing is learned on the training dataset. The preprocessing pipeline is responsible for text data cleaning and transformation into a numerical representation using techniques of abbreviation substitution, text cleaning, and text tokenization. The resulting text representation is sparse. Next, the embedding layer and the recurrent neural network layer try to learn the mapping function of the sparse text representation to the target labels based on the loss function. Here, to control the training of the model, the model performance on the development dataset is used for an early stopping approach. Finally, the testing dataset is used to evaluate and report the model performance. In our case, we noticed that the model performance is affected by the dataset splits. Thus, we opted for k-folds cross-validation to adjust for biases induced by dataset splits. Hence, the average model performance is reported.

3. Results

3.1. FMEA Documents Dataset Description and FMEA Relation Consistency Assessment

For this research, we opted for the strategy to initially generate a seed dataset, upon which consecutively a classifier is trained, which finally can automatically annotate the complete available documents. Thus, we collected a sample of FMEA documents that domain experts annotated for this research. The annotation process is conducted using an annotation tool. This annotation tool shows the complete cells’ text and allows for multilabeled FMEA cells’ text annotation. For a FMEA cell text that does not belong to the defined classes or is ambiguous, domain experts are given the option to annotate the text as out of the scheme. The experts annotated four P_FMEA and nine UP_FME. In P_FMEA, the number of distinctive cells is five times higher than the number of distinctive cells in UP_FMEA. Based on the annotation, 10% and 18% of the P_FMEA and UP_FMEA cells are annotated as out of scheme, respectively. These cells are excluded from further analysis. The comparison of concepts class distribution over P_FME vs. UP FMEA cells is depicted in Figure 4.

Based on the annotation acquired from the experts, we can annotate the relations between the cells present in the FMEA documents. Here, we distinguish five types of annotated relations. The first type is considered to be consistent relation if the head cell and tail cell concept classes conform to the sequence in the classification scheme. The second type is considered to be a relation with missing information (Missing Information relation) if the head cell and tail cell concept classes do not conform to the classification scheme sequence, skipping concepts belonging to one or more of the concept classes (e.g., 5M to Functional). The third type is considered to be a relation with reversed direction (Reversed relation) if the head cell and tail cell concept classes do not conform to the sequence in the classification scheme and link concepts in the reversed direction (e.g., Physical to Functional). The fourth type is considered to be a relation within the same concept (Same Concept relation) if the head cell and tail cell concept belong to the same class. In this case, we cannot determine any information about the relation consistency. Moreover, a relation is considered out of scheme (Out of scheme relation) if it connects to a head cell or a tail cell that has annotation out of scheme. It should be noted that for the comparison of the head cell concept classes and the tail node concept classes, we compare the highest (closest to the functional concept class) concept classes from the tail cell to the lowest (closest to 5M concept class) concept class for the head cell.

Based on the analysis of the relations present in P_FMEA and UP_FMEA, the percentages of relations that are out of scheme in P_FMEA and UP_FMEA are 37% and 22%, respectively. Additionally, the number of relations in P_FMEAs is six times higher than UP_FMEAs. Moreover, we noticed that both types of documents have consistency impairments concerning the causal relation. Additionally, the percentages of the relations between concepts within the same concept class and those that indicate reversed direction are comparable between the UP_FMEA and P_FME documents. However, in UP_FMEA, the relations with missing information are more present. Thus, we think that the definition of the scope of UP_FMEA is insufficient. Consequently, an increasing number of relations with missing information are observed. The comparison of relation type distribution present in P_FMEA and UP_FMEA is depicted in Figure 5.

3.2. Text Classification for FMEA’s Relations Consistency

For the purpose of this research, we opted for an end-to-end deep learning classification model. The objective of the model is to predict the concept classes of FMEA cells’ text. Based on this prediction, we can apply the same analysis presented in Section 3.1 for assessing the consistency of FMEA’s relations consistency presented before.

First, we established a data preprocessing pipeline that consists of abbreviation substitution, text cleaning, and text tokenization. Together with domain experts, we collected a set of abbreviations commonly used in the FMEA documents for abbreviation substitution. Additionally, we applied the basic cleaning function to remove punctuation and numbers for the text cleaning. Finally, tokenization based on spaces is applied to FMEA cells’ text.

Second, we trained a recurrent neural network-based classification model. The model consists of an embedding layer with an input size of the number of distinct words from the training set with a masking function to handle the different lengths, followed by a recurrent neural network layer with 100 hidden units. We test LSTM, Bidirectional LSTM (BiLSTM), GRU, and Bidirectional GRU (BiGRU). Finally, a fully connected layer of five neurons followed by sigmoid functions is established as an output layer.

We propose two approaches to evaluate our classification model and method (automatically assessing the relations consistency in FMEA documents). First, we propose conducting a cell-based test training split on the full dataset to evaluate the classifier on unseen FMEA cells. Here, we collect all the FMEA cells from all the FMEA documents, apply the cleaning pipeline and remove duplicated cells. Second, we propose to do document-based splitting. Here, we divided the FMEA documents into training and testing in the training phase. The criteria are made under the assumption that if we have annotated FMEA documents of a specific distribution, we evaluate the proposed method’s performance in assessing a new FMEA document from the same distribution. Thus, we use one P_FMEA document and one UP_FMEA for testing. We use four-fold cross-validation to test the classification performance. This approach is used to test the effectiveness of the proposed method for assessing the relation consistency. The first evaluation strategy targets the classification performance removing any biases caused by a potential target leak. Similarly, we use four-fold cross-validation to test the classification performance.

Based on the first evaluation approach, we noticed that all the classifiers’ architecture performances drop with respect to minority classes (i.e., the lower three classes that are the least represented in the collected dataset). However, with respect to the Functional and the Physical concept classes, LSTMS and Bidirectional LSTMS surpass the other architecture by a small margin. The results of the comparison are depicted in Figure 6 and summarized in

Table 1. The classification results of FMEA cells using cell-based splitting. The best performing architecture and its performance with respect to the concept class is summarized.

Class Name	Architecture	Average F1 Score
Functional	BiLSTM	92%
Physical	LSTM	88%
UP-C/E	LSTM	61%
Parametric	GRU	56%
5M	GRU	63%

. Additionally,

Table A1. Classification results based on the architecture and the data training and testing splits; we compare four different classification methods based on different recurrent neural network layers.

Class	Classifier	Train/Test Split	Average MCC	Average F1
Functional	LSTM	Documents based	0.89 (SD 0.02)	91% (SD 0.02)
		Cell based	0.89 (SD 0.04)	91% (SD 0.03)
	BiLSTM	Documents based	0.90 (SD 0.05)	92% (SD 0.04)
		Cell based	0.91 (SD 0.06)	92% (SD 0.05)
	GRU	Documents based	0.88 (SD 0.05)	90% (SD 0.04)
		Cell based	0.88 (SD 0.06)	91% (SD 0.05)
	BiGRU	Documents based	0.91 (SD 0.01)	93% (SD 0.01)
		Cell based	0.89 (SD 0.11)	90% (SD 0.1)
Physical	LSTM	Documents based	0.77 (SD 0.01)	89% (SD 0.01)
		Cell based	0.77 (SD 0.04)	88% (SD 0.01)
	BiLSTM	Documents based	0.76 (SD 0.01)	88% (SD 0.01)
		Cell based	0.75 (SD 0.04)	87% (SD 0.02)
	GRU	Documents based	0.77 (SD 0.03)	89% (SD 0.02)
		Cell based	0.75 (SD 0.03)	88% (SD 0.02)
	BiGRU	Documents based	0.77 (SD 0.03)	89% (SD 0.02)
		Cell based	0.76 (SD 0.02)	88% (SD 0.01)
UP-C/E	LSTM	Documents based	0.56 (SD 0.09)	64% (SD 0.07)
		Cell based	0.53 (SD 0.09)	61% (SD 0.07)
	BiLSTM	Documents based	0.57 (SD 0.10)	65% (SD 0.08)
		Cell based	0.52 (SD 0.06)	59% (SD 0.07)
	GRU	Documents based	0.53 (SD 0.12)	62% (SD 0.09)
		Cell based	0.51 (SD 0.08)	58% (SD 0.08)
	BiGRU	Documents based	0.56 (SD 0.09)	64% (SD 0.07)
		Cell based	0.53 (SD 0.11)	60% (SD 0.09)
Parametric	LSTM	Documents based	0.62 (SD 0.02)	64% (SD 0.02)
		Cell based	0.4 7(SD 0.15)	51% (SD 0.15)
	BiLSTM	Documents based	0.58 (SD 0.08)	59% (SD 0.09)
		Cell based	0.45 (SD 0.11)	49% (SD 0.10)
	GRU	Documents based	0.57 (SD 0.04)	59% (SD 0.06)
		Cell based	0.52 (SD 0.12)	56% (SD 0.11)
	BiGRU	Documents based	0.65 (SD 0.03)	67% (SD 0.02)
		Cell based	0.49 (SD 0.04)	53% (SD 0.05)
5M	LSTM	Documents based	0.63 (SD 0.07)	65% (SD 0.07)
		Cell based	0.59 (SD 0.11)	62% (SD 0.10)
	BiLSTM	Documents based	0.63 (SD 0.12)	65% (SD 0.11)
		Cell based	0.54 (SD 0.05)	58% (SD 0.05)
	GRU	Documents based	0.58 (SD 0.07)	61% (SD 0.06)
		Cell based	0.60 (SD 0.09)	63% (SD 0.07)
	BiGRU	Documents based	0.65 (SD 0.12)	67% (SD 0.10)
		Cell based	0.59 (SD 0.06)	63% (SD 0.05)

(a full table of results) is added to Appendix A.

Moreover, we used the second approach of splitting (documents-based splitting) to test the effectiveness of the proposed method for assessing the relation consistency. The results of each FMEA relation type detection are summarized in

Table 2. Results of FMEA consistency using document-based splitting.

Class Name	Architecture	Average F1 Score
Missing Information	BiGRU	64%
Reversed	LSTM	41%
Same Concept	BiGRU	72%
Consistent	BiGRU	76%
Out of Schema	BiLSTM	71%

.

3.3. Pattern-Based Approach for FMEA Cell Consistency

Based on the analysis of actual FMEA documents, we noticed cases of merged cells. In these cases, the short text of a single cell comprises multiple concepts or even relations between multiple concepts, e.g., a causal chain of multiple causes and effects. Thus, together with domain experts, we collected and labeled a subset of FMEA cells to evaluate the severity of such cases. The labeling criterion is based on more than one concept that belongs to our classification scheme. Based on the collected dataset, 34% of the annotated dataset is considered as merged. As a response, we collected lexical causal cues (such as ”causes”, “caused by”, ”due to”, etc.) in addition to some special abbreviation used by domain experts to indicate the causal relation (e.g., ”->“). In the first merged cell detection approach, we employed a search for the collected causal cues within the FMEA cells’ text. As a result, this approach yielded high precision but limited recall. To this end, we proposed an approach-based intra-sentential pattern. For this method, we need to set two hyperparameters (MIth, OCth) in order to optimize the proposed approach performance. Thus, we split the labeled subset of FMEA cells into validation and testing sets. We used the validation set to conduct a grid search to optimize MIth and OCth. Based on the grid search, we decided for MIth = 0.0015 and OCth = 3.

Furthermore, we tested the union ensemble for both of the approaches assuming that the patterns-based method could increase the recall of the causal cues-based approach. Here, we report the results of the three methods (intra-sentential pattern, causal cues, and the ensemble of both methods). The results are summarized in

Table 3. The detection of merged FMEA cells.

Approach	F1 Score	Precision	Recall
Causal cues	60%	92%	45%
Intra-sentential pattern	70%	61%	81%
Union ensemble	72%	62%	86%

.

4. Discussion

The proposed consistency improvement methods address two main aspects of data consistency: cell consistency and relation consistency.

For the relation consistency, the results show the usefulness and the importance of domain knowledge being represented in the proposed classification scheme, which was derived from extensive interviews with multiple domain experts. This classification scheme allows for the assessment of relation consistency presented in the FMEA documents. Interestingly, we noticed that although the distribution of the concept classes in P_FMEA is different from the one in UP_FMEA, the expected concepts class distributions based on the discussion with the experts is not present. For example, despite the fact that in UP_FMEA, concepts belonging to the lower two classes are more represented than in P_FMEA, we noticed that the majority concept class in both P_FMEA and UP_FMEA cells is the Physical concept class. This does not conform to the initial expectation and the definition of the FMEAs. In addition to the presence of concepts belonging to concepts’ classes out of the FMEA’s type scope, these observations are the first indication of inconsistency in the causal relations. Leveraging the annotated dataset, we discovered that UP_FMEA documents are less consistent than P_FMEA, where more Missing Information relations occur. In other words, as a result of this research, we can quantitatively assess the relation consistency presented in the FMEA documents. The results of this research can be integrated into many downstream tasks. An example is to support domain experts in the process of creating new, more consistent FMEA documents.

Additionally, for the automated relation consistency checking, we found this to be highly dependent on the classification method. Thus, different classification architectures are required due to the highly imbalanced distribution of the classes. Namely, for minority classes (e.g., 5M and Parametric), models with fewer parameters, such as the model with GRU-based architecture, perform better than models with a higher number of parameters, such as the model with LSTM-based architecture. Additionally, each type of relation has a different distribution regarding FMEA cell concept classes. For example, Same Concept relations mainly occur between FMEA cells labeled with concept class Physical. In contrast, Reversed relations mainly occur between FMEA cells labeled with concept classes Physical and UP-C/E. Moreover, Missing Information relations mainly occur between FMEA cells labeled with concept class “‘Parametric’, ‘Physical’, ‘5M’, ‘Physical’” and “‘UP-C/E’, ‘Functional’”. Finally, consistent relation FMEA cells are labeled with concept class “Physical”, “Functional” and “UP-C/E”, “Physical”. Consequently, the FMEA relation consistency performance depends on the classifier performance. The combination of the classes mainly occurs for each relation type.

For the cell consistency, in FMEA documents, merged cells occur frequently. Over a third of the FMEA cells are merged based on our labeled subset. Thus, we have devised two approaches to detect merged cells. The first approach is based on causal cues. This approach achieved a 60% F1 score. The high F1 score is attributed to the high precision, which is over 90%. However, as expected, the causal cues-based approach has limited recall because it does not cover cases (b), (c), and (d) of merged cells (please check Figure 2). As such, we proposed the intra-sentential-based approach as an alternative solution.

The intra-sentential pattern-based approach showed good results even with no labeled data for the cell consistency improvement. The intra-sentential-based approach achieved a 70% F1 score. This score was achieved with two hyperparameters that were optimized based on grid search. The method has high recall but limited precision. To improve the precision of the intra-sentential-based approach, we devised a union ensemble combining its results with the causal cues-based approach. As a result, the union ensemble achieved the best outcome with a 72% F1 score. To increase the paper readability we added acronyms glossary (

Table A2. Acronyms glossary.

Acronyms	Clarification
FMEA	Failure Mode Effect Analysis
FMMEA	Failure Modes, Mechanisms, and Effects Analysis
P_FMEA	Process FMEA documents
UP_FMEA	Unit Process FMEA documents
Functional	Concept class that includes concepts describing functional characteristics’ deviation of the product
Physical	Concept class that includes concepts describing physical characteristics’ deviation of the product structures
UP-C/E	Concept class that includes concepts describing unit process deviation
Parametric	Concept class that include concepts describing the so-called unit process key parameters deviation
5M	Concept class that include concepts describing Man, Machine, Material, Method, and Measurement from the risk-management model
Intra-sentential pattern	A meaningful combination of terms
MI	Mutual Information
N_gram	A sequence of terms of a length n
MIth	Mutual Information threshold
OCth	Occurrences threshold
Top Patterns	A pattern (filtered n_gram) that is not contained by any other pattern
Subpatterns	Patterns (filtered n_gram) contained in a Top Pattern

) to the Appendix A.

5. Conclusions

We presented work on FMEA documents, a form of tabular documents comprising short snippets of domain-specific text, where causal relationships are being captured. These documents were originally intended for interpretation by human experts. Consequently, these documents tend to contain many inconsistencies. As a response, we systematically defined: (i) a domain-specific model, consisting of a concept scheme (i.e., types of cause/effects) and a relationship consistency scheme (i.e., valid relations between the concepts), and (ii) a model of inconsistencies, consisting of mixed-up cells (including reverse direction), missing information, and merged cells. We supported domain experts by developing an annotation tool to collect a specific dataset for the semiconductor manufacturing industry. As a result of this research, we can quantitatively assess the relation consistency presented in the FMEA documents.

We trained and compared different models directly on the FMEA documents to obtain a consistency checking method, which achieves between 56% and 96% F1 on identifying the concept classes. The challenge of merged cells requires separate processing. Here, we adapted an existing method on intra-sentential pattern mining, which was adapted to suit our needs. The intra-sentential pattern extraction finally achieved an F1-score of 70% and 72% when combined with the causal cues-based method.

The result of the developed models can also be added to the FMEA documents as additional information. The additional information added to existing FMEA (i.e., the classes of the concepts in an FMEA cell, also the cell status if it is merged) is beneficial for the reuse of the FMEA knowledge. A simple example may include retrieving only the direct effect or root cause.

To summarize, data inconsistency detection in complex environments, similar to highly automated production lines, is likely to fail if conducted exclusively in a data-driven manner. Domain-specific knowledge needs to be considered in multiple stages of the processing pipeline in order to achieve sufficiently good results. In our case, having a causal model and the constraints implied by this model helped to detect many cases of data inconsistency. Our method can be adapted in the future by considering more external data sources for more meaningful representation learning of the text contained in the FMEAs and providing means of interactions and feedback of human experts to address the detected consistency impairments. Finally, as future work, we aim to integrate the results of this research (i.e., FMEA and the additional information provided by the proposed method) in a knowledge discovery approach to predict missing links in the FMEA.