A Graphical Model to Diagnose Product Defects with Partially Shuffled Equipment Data

Abstract

The diagnosis of product defects is an important task in manufacturing, and machine learning-based approaches have attracted interest from both the industry and academia. A high-quality dataset is necessary to develop a machine learning model, but the manufacturing industry faces several data-collection issues including partially shuffled data, which arises when a product ID is not perfectly inferred and yields an unstable machine learning model. This paper introduces latent variables to formulate a supervised learning model that addresses the problem of partially shuffled data. The experimental results show that our graphical model deals with the shuffling of product order and can detect a defective product far more effectively than a model that ignores shuffling.

1. Introduction

The diagnosis of product defects is essential for improving productivity in manufacturing. The task involves deciding whether a product is defective, given product-related information (e.g., image of completed products, and equipment data). Thanks to large volumes of data collected from factories and the advent of machine learning algorithms, data-driven defect diagnosis is receiving increasing attention from both the industry and academia. Data-driven defect diagnosis involves the development of a classifier $f$ by using a training dataset that consists of $\left({\mathit{x}}^{\left(i\right)},y^{\left(i\right)}\right)$ for $i=1,2,\cdots ,n$, where ${\mathit{x}}^{\left(i\right)}$ is a feature vector and $y^{\left(i\right)}$ is the label of product $i$, as well as using the classifier to diagnose product defects.

Data-driven defect diagnosis uses either a feature vector (i.e., input data), such as completed product data (e.g., the image and specification of products) or equipment (e.g., command and sensor) data. Equipment data, including the manufacturing process for each product, can be used to diagnose defects before a product is produced. For example, Manco et al. [1] applied outlier detection to predict door failures on metro trains, and Khalastchi et al. [2] proposed a hybrid approach to detect faults in robotic systems based on sensor readings, commands, and feedbacks from actuators. Song and Shi [3] proposed a quality-supervised double-layer method to detect quality-related faults in which the first layer uses principal component analysis (PCA) to detect the fault and the second layer finds the key variable orthogonal weight of the fault from the PCA. Fakhfakh et al. [4] developed a mathematical model for on-line fault diagnosis, resulting in the schedule delays of a flexible manufacturing system to minimize the total delays caused by faults. They also developed a constraint programming technique to solve this mathematical problem.

Multi-source data fusion that combines or merges data from multiple sources (e.g., multiple sensors and two or more machines) is necessary for realistic defect diagnosis with equipment data [5]. It can be categorized as signal-level, feature-level, or decision-level fusion [6]. Among them, feature-level fusion extracts feature vectors from each data point and merges them based on product ID. It is used primarily to combine a dataset to train a defect diagnosis model [7,8].

However, multi-source data fusion cannot always be effectively conducted due to imperfections, correlations, inconsistency, disparateness, etc. [9]. In many cases (e.g., a specific subprocess such as milling in the manufacturing process), a product ID cannot be attached until the production process is finished. In such cases, inferring a product ID should be conducted first. However, it cannot be perfectly inferred when there are one or more workstations such that (1) two or more independent parallel machines reside in a workstation, (2) the processing time of each machine is variable, and (3) workstations cannot be observed from the outside due to safety issues. When equipment data are partially shuffled, there can be no guarantee that a record in data merged by feature-level fusion will indicate a product. We propose a graphical model to diagnose product defects with partially shuffled equipment data. The proposed graphical model calculates the probabilities that products are shuffled and the likelihood that a product is defective by considering the shuffling probability.

To the authors’ best knowledge, this is the first study to address the real and common manufacturing situation in which each product cannot be perfectly traced because merged equipment data based on product IDs may be partially shuffled. This study proposes a simulation method to calculate the probability that product order is shuffled at each workstation based on production-line analysis. The method also includes the calculation of the maximum shuffling range $R$ at workstation $j$ and probability, $Q_{j,r}$, of two products are shuffled with shuffling range $r\le R$. It also proposes a graphical model to diagnose product defects with partially shuffled equipment data with a consideration of product shuffling based on $Q_{j,r}$.

Section 2 of this paper describes the problem in detail, focusing on the situation in which partially shuffled equipment data are merged from multiple data sources. Section 3 develops a graphical model to diagnose defects with partially shuffled equipment data through three phases: (1) production-line analysis with simulation, (2) graphical model design, and (3) graphical model inference. Section 4 provides an example of a numerical application of the proposed graphical model, and Section 5 suggests future research directions.

2. Problem Description

Suppose there are serial workstations and that a product must visit each machine at every workstation to be completed. More formally, let product $i$ be processed by machine $k$ at workstation $j$ from time $t_1$ to $t_2$. The feature for product $i$ at workstation $j$, ${\mathit{x}}_j^{\left(i\right)},$ is $\left(h_{j,k}^{(t_1)},h_{j,k}^{(t_1+1)},\cdots ,h_{j,k}^{(t_2)}\right)$, where $h_{j,k}^{\left(t\right)}$ is the feature value collected from machine $k$ at workstation $j$ at time $t$ and $t_1+1$ means time a unit time elapse (e.g., second and millisecond) after $t_1$. By expanding the problem to every product $i=1,2,\cdots ,n$ and every workstation $j=1,2,\cdots ,m$, a defect diagnosis dataset, $\left\{\left({\mathit{x}}^{\left(1\right)},y^{\left(1\right)}\right),\cdots ,\left({\mathit{x}}^{\left(n\right)},y^{\left(n\right)}\right)\right\}$ can be generated in which ${\mathit{x}}^{\left(i\right)}=\left({\mathit{x}}_1^{\left(i\right)},{\mathit{x}}_2^{\left(i\right)},\cdots ,{\mathit{x}}_m^{\left(i\right)}\right)$ and $y^{\left(i\right)}$ are feature vector and a label of product $i$, respectively.

The problem is to train a classifier $f(·)$ with training records ${\left\{\left({\mathit{x}}^{\left(i\right)},y^{\left(i\right)}\right)\right\}}_{i=1}^n$ and diagnose whether a given product is defective by using the trained classifier. It is a general approach to diagnose defects by using machine learning. However, this approach cannot be employed when some product-processing orders are partially shuffled and the shuffled orders are not known.

Consider a production line that consists of three workstations (${\mathrm{W}}_1$, ${\mathrm{W}}_2$, ${\mathrm{W}}_3$) where product orders maybe shuffled and shuffled orders are not perfectly inferred in ${\mathrm{W}}_2$ because the workstation includes two parallel machines whose processing times are not constant and cannot be observed from outside, as illustrated in Figure 1. The circle indicates a machine, and the dotted rectangle indicates a workstation. Defect diagnosis for each product is conducted after the product is processed by ${\mathrm{W}}_3$.

Suppose there are two products $i-1,$ with $i$ following the sequence ${\mathrm{M}}_{1,1}\to {\mathrm{M}}_{2,1}\to {\mathrm{M}}_{3,1}$ and ${\mathrm{M}}_{1,1}\to {\mathrm{M}}_{2,2}\to {\mathrm{M}}_{3,1}$. Suppose also that product $i-1$ enters ${\mathrm{M}}_{1,1}$ ahead of product $i$, but product $i-1$ enters ${\mathrm{M}}_{3,1}$ after product $i$. This situation can be illustrated in a Gantt chart (Figure 2), in which the arrow, rectangle, and colored rectangle denote the time to extract data, observed data, and shuffled observed data, respectively.

As seen in Figure 2, equipment data for product $i-1$ are a concatenation of ${\mathrm{M}}_{1,1}$’s data during [1,3], ${\mathrm{M}}_{2,1}$’s data during [6,8], and ${\mathrm{M}}_{3,1}$’s data during [10], and equipment data for product $i$ is a concatenation of ${\mathrm{M}}_{1,1}$’s data during [4,5], ${\mathrm{M}}_{2,2}$’s data during [6,7], and ${\mathrm{M}}_{3,1}$’s data during [8,9]. One cannot know which product is processed by machines ${\mathrm{M}}_{2,1}$ and ${\mathrm{M}}_{2,2}$ in ${\mathrm{W}}_2$, because this workstation is not observed; one can only know if each machine is busy or idle. It is therefore impossible to identify which product is released at time 9 from ${\mathrm{W}}_3$. In this situation, the feature vector for product $i$ is not ${\mathit{x}}^{\left(i\right)}=\left({\mathit{x}}_1^{\left(i\right)},{\mathit{x}}_2^{\left(i\right)},{\mathit{x}}_3^{\left(i\right)}\right)$, but ${\widehat{\mathit{x}}}^{\left(i\right)}=\left({\mathit{x}}_1^{\left(i\right)},{\mathit{x}}_2^{\left(i-1\right)},{\mathit{x}}_3^{\left(i-1\right)}\right)$, which affects the training of the classifier $f(·)$. In this paper, we call ${\widehat{\mathit{x}}}^{\left(i\right)}$ the observed feature vector for product $i$, and we called the equipment data collected for each product partially shuffled data.

3. Proposed Graphical Model

In this section, we construct a graphical model to diagnose defects with partially shuffled equipment data through three phases: (1) production-line analysis with simulation, (2) graphical model design, and (3) graphical model inference. In the first phase, we characterize the partially shuffled data by means of the shuffling probabilities saved in matrix $\mathrm{Q}$. In the second phase, latent variables are introduced to design a graphical model structure. In the third phase, probabilistic supervised models for defect detection are trained, and an expectation of probability that a product is defective given the observed feature vector is calculated using the graphical model.

3.1. Production-Line Analysis with Simulation (Characteristics of Shuffled Data)

Let each ${\tilde{a}}_j^{\left(i\right)}$ and ${\tilde{c}}_j^{\left(i\right)}$ indicate the arrival order and completion order of product $i$ at workstation $j$. In the absence of shuffling, ${\tilde{a}}_j^{\left(i\right)}={\tilde{c}}_j^{\left(i\right)}$. If shuffling exists, ${\tilde{a}}_j^{\left(i\right)}\ne {\tilde{c}}_j^{\left(i\right)}$ for some $i$. Let us define the range of shuffling as $r\equiv |{\tilde{a}}_j^{\left(i\right)}-{\tilde{c}}_j^{\left(i\right)}|$ for product $i$. We can expect that the actual $i^{th}$ completed product is either the ${\left(i-r\right)}^{th}$ or ${\left(i+r\right)}^{th}$ arriving product. Here, $r$ is regarded as a random variable.

Let $Q_{j,r}$ denote the probability of a shuffle with a range of $r$ at workstation $j$, which is formally expressed as:

$$Q_{j,r}=\mathrm{Pr}\left({\tilde{c}}_j^{\left(i_2\right)}-{\tilde{c}}_j^{\left(i_1\right)}\in \left\{\gamma +r,\gamma -r\right\}|{\tilde{a}}_j^{\left(i_1\right)}-{\tilde{a}}_j^{\left(i_2\right)}=\gamma \right)$$

(1)

where ${\tilde{a}}_j^{\left(i_1\right)}-{\tilde{a}}_j^{\left(i_2\right)}=\gamma$ means that $\gamma$ products arrive between the $i_1^{th}$ and $i_2^{th}$ arrival times at workstation $j$, and ${\tilde{c}}_j^{\left(i_2\right)}-{\tilde{c}}_j^{\left(i_1\right)}\in \left\{\gamma +r,\gamma -r\right\}$ implies that either $\mathsf{\gamma }+r$ or $\mathsf{\gamma }-r$ products are completed between the $i_1^{th}$ and $i_2^{th}$ completion at workstation $j$. That is, the $i^{th}$ arrived product may not be the $i^{th}$ completed product at the workstation because some products are completed faster or slower than expected. Maximum shuffling range $R$ at workstation $j$ is calculated as $R=\max \left\{r|Q_{j,r}>0\right\}$, and $Q_{j,r}$ is defined for $r\le R$.

Even though $Q_{j,r}$ plays an important role in developing the graphical model, it is not possible to analytically calculate it when the range of $r$ is wide because there are many machines whose processing times are variables. However, we adopted a simulation method that could estimate $Q_{j,r}$. The first step is to generate simulated data that consist of simulated production routes for $\widehat{n}$ products. Let $V^{\left(p\right)}$ be the processing route of simulated product $p$, which is a tuple of the machine, start time of processing, and the end time of processing at each workstation:

$$V^{\left(p\right)}=\left(\left(M_1^{\left(p\right)},a_1^{\left(p\right)},c_1^{\left(p\right)}\right),\left(M_2^{\left(p\right)},a_2^{\left(p\right)},c_2^{\left(p\right)}\right),\cdots ,\left(M_m^{\left(p\right)},a_m^{\left(p\right)},c_m^{\left(p\right)}\right)\right)$$

(2)

where $M_j^{\left(p\right)}$, $a_j^{\left(p\right)}$, and $c_j^{\left(p\right)}$ indicate the machine that processes product $p$, its start time, and its end time (release time) at workstation ${\mathrm{W}}_j$, respectively.

A processing route generation algorithm is presented as Algorithm 1. It repeats (1) randomly selecting one of the idle machines to process a product at a workstation, (2) sampling processing time from the distribution of the selected machine’s processing time, (3) calculating the start and end time of the selected machine for the product, and (4) updating the busy/idle status of the selected machine for every product and workstation.

Algorithm 1 Product Route Generation Algorithm
Input	$\widehat{n}$, ${\mathrm{M}}_{j,k}$, ${\mathrm{F}}_{j,k}$ for all $j$, $k$
Procedure	01	Initialize $I_{j,k}^{\left(t\right)}$ as 1 for all $t,j,k$
	02	Initialize $t$ as 1
	03	For $\left(p=1;p\le \widehat{n},j++\right)$ do {
	04	For $\left(j=1;j\le m,j++\right)$ do {
	05	While $\left({{\displaystyle \sum }}_{k=1}^{m_j}{I}_{j,k}^{\left(t\right)}=0\right)$ do {Increase $t$ by 1}
	06	$\widehat{k}\stackrel{Random}{\leftarrow }\left\{k|I_{j,k}^{\left(t\right)}=1\right\}$
	07	$a_j^{\left(p\right)}=t$
	08	$b_j^{\left(p\right)}\stackrel{sampling}{\leftarrow }{\mathrm{F}}_{j,k}(\cdot )$
	09	$c_j^{\left(p\right)}=t+b_j^{\left(p\right)}$
	10	$I_{j,\widehat{k}}^{\left(t+\tau \right)}=0$ for $\tau =1,2,\cdots ,b_j^{\left(p\right)}$}}
Output	$V^{\left(p\right)}$ for $p=1,2,\cdots ,\widehat{n}$

A specific explanation of Algorithm 1 follows. Every machine’s state for all $t$ is initialized as idle (Line 01), and the time is also initialized (Lines 01–02). Product $p$ is simulated (Lines 04~10): If there is no idle machine in workstation $j$ at $t$, then $t$ is increased by 1 until one or more machines become idle (Line 05); the product waits for processing when there is no idle machine. This product then enters a randomly selected idle machine $\widehat{k}$ in workstation $j$ (Line 06), and the start time for processing the product $p$ is set to $t$ (Line 07), and the processing time is sampled from its probability distribution (Line 8). The processing time determines the end time of processing product $p$. (Line 09), and the selected machine’s state becomes busy (Line 10).

Using $V^{\left(p\right)}$ for $p=1,2,\dots ,\widehat{n}$, obtained from Algorithm 1, we can calculate $Q_{j,r}$ in Equation (1). First, we calculate ${\tilde{a}}_j^{\left(p\right)}$ and ${\tilde{c}}_j^{\left(p\right)}$. based on $\left\{a_j^{\left(1\right)},a_j^{\left(2\right)},\cdots ,a_j^{\left(\widehat{n}\right)}\right\}$ and $\left\{c_j^{\left(1\right)},c_j^{\left(2\right)},\cdots ,c_j^{\left(\widehat{n}\right)}\right\}$ for all $p$, respectively. Then, we obtain the estimate of $Q_{j,r}$ from the ratio of pairs $\left(i_1,i_2\right)$ that satisfies ${\tilde{c}}_j^{\left(i_2\right)}-{\tilde{c}}_j^{\left(i_1\right)}\in \left\{\gamma +r,\gamma -r\right\}$ among those ${\tilde{a}}_j^{\left(i_1\right)}-{\tilde{a}}_j^{\left(i_2\right)}=\gamma$.

3.2. Graphical Model Design (Graphical Model for Shuffled Data)

We can introduce a latent variable ${\mathit{z}}^{\left(i\right)}=\left(z_1^{\left(i\right)},z_2^{\left(i\right)},\cdots ,z_m^{\left(i\right)}\right)$ to indicate the $i\mathrm{th}$ completed product at each workstation. For example, if $z_j^{\left(7\right)}=9$, then the 7th completed product at workstation $j$ is the one that has entered into that workstation as the 9th. Actually, $z_j^{\left(i\right)}$ is unknown if workstation $j$ is unobservable, but there can be only one $i^{\prime }$, such that $z_j^{\left(i^{\prime }\right)}=i$. By introducing these latent variables, we can develop a graphical model that represents the relationships between all ${\mathit{x}}^{\left(i\right)}$ and ${\widehat{\mathit{x}}}^{\left(i\right)}$ values, and an inference to defect diagnosis is possible.

An exemplary structure of graphical model can be given with the help of latent variable ${\mathit{z}}^{\left(i\right)}$, as presented in Figure 3 in the case $R=2$.

In this figure, shaded circles indicate observable variables, while the others are unobservable. In general, node ${\mathit{z}}^{\left(i\right)}$ is connected to ${\mathit{z}}^{\left(i-R\right)},{\mathit{z}}^{\left(i-R+1\right)},\cdots ,{\mathit{z}}^{\left(i+R\right)}$ because the variable ${\mathit{z}}^{\left(i\right)}$ mathematically affects ${\mathit{z}}^{\left(i-R\right)}$, ${\mathit{z}}^{\left(i-R+1\right)}$, …, ${\mathit{z}}^{\left(i+R\right)}$, and ${\widehat{\mathit{x}}}^{\left(i\right)}$.

With the help of the graphical structure, ${\widehat{\mathit{x}}}^{\left(i\right)}$ given ${\mathit{z}}^{\left(i\right)}$ can be expressed as:

$${\widehat{\mathit{x}}}^{\left(i\right)}|{\mathit{z}}^{\left(i\right)}=\left({\mathit{x}}_1^{\left(z_1^{\left(i\right)}\right)},{\mathit{x}}_2^{\left(z_2^{\left(i\right)}\right)},\cdots ,{\mathit{x}}_m^{\left(z_m^{\left(i\right)}\right)}\right)$$

(3)

${\mathit{x}}^{\left(i\right)}$ can be then inferred from ${\widehat{\mathit{x}}}^{\left(i^{\prime }\right)}$ and ${\mathit{z}}^{\left(i^{\prime }\right)}$ for $i^{\prime }=i-R,i-R+1,\cdots ,i+R$ as:

$${\mathit{x}}^{\left(i\right)}=\left({{\displaystyle \sum }}_{i^{\prime }=i-R}^{i+R}I\left(z_j^{\left(i^{\prime }\right)}=i\right)\times {\widehat{\mathit{x}}}_j{}^{\left(i^{\prime }\right)}\right){}_{j=1}^m$$

(4)

where $I\left(condition\right)$ is 1 if the condition is satisfied, and 0, otherwise.

3.3. Inference (Defect Diagnosis)

The objective of this paper was to calculate the probability, $\mathrm{Pr}\left(y^{\left(i\right)}|{\mathit{x}}^{\left(i\right)}\right),$ that product $i$ is defective given its feature vector, which may be calculated by a probabilistic supervised model such as logistic regression. However, it cannot be directly obtained because ${\mathit{x}}^{\left(i\right)}$ is unobservable, unlike the situation to which a supervised learning model is applied. Thus, we expanded $\mathrm{Pr}\left(y^{\left(i\right)}|{\mathit{x}}^{\left(i\right)}\right)$ by means of the designed graphical model structure to consider all possible situations in which ${\mathit{x}}^{\left(i\right)}$ is distributed in ${\widehat{\mathit{x}}}^{\left(i^{\prime }\right)}$ for $i^{\prime }=i-R,i-R+1,\cdots ,i+R$ as:

$$\mathrm{Pr}\left(y^{\left(i\right)}|{\mathit{x}}^{\left(i\right)}\right)={{\displaystyle \sum }}_{{\tilde{\mathit{z}}}^{\left(i\right)}}\mathrm{Pr}\left(y^{\left(i\right)}|\left({{\displaystyle \sum }}_{i^{\prime }=i-R}^{i+R}I\left({\tilde{z}}_j^{\left(i^{\prime }\right)}=i\right)\times {\widehat{\mathit{x}}}_j{}^{\left(i^{\prime }\right)}\right){}_{j=1}^m\right)\times \mathrm{Pr}\left({\mathit{z}}^{\left(i\right)}={\tilde{\mathit{z}}}^{\left(i\right)}\right)$$

(5)

Now, each part of the right-hand side of Equation (5) can be calculated.

(i)  Calculation of $\mathrm{Pr}\left({\mathit{z}}^{\left(i\right)}={\tilde{\mathit{z}}}^{\left(i\right)}\right)$

$\mathrm{Pr}\left({\mathit{z}}^{\left(i\right)}={\tilde{\mathit{z}}}^{\left(i\right)}\right)$ in Equation (5) can be calculated using $Q_{j,r}$ from the production-line analysis in Section 3.1. When $R=0$ (when there is no partial shuffling), then $\mathrm{Pr}\left({\mathit{z}}^{\left(i\right)}=i\times \mathbf{1}\right)=1$ for all $i$. In this case, $\left({{\displaystyle \sum }}_{i^{\prime }=i-R}^{i+R}I\left(z_j^{\left(i^{\prime }\right)}=i\right)\times {\widehat{\mathit{x}}}_j{}^{\left(i^{\prime }\right)}\right){}_{j=1}^m$ equals ${\mathit{x}}^{\left(i\right)}$, implying that $\mathrm{Pr}\left(y^{\left(i\right)}|{\mathit{x}}^{\left(i\right)}\right)$ is directly calculated from a probabilistic supervised model because ${\mathit{x}}^{\left(i\right)}={\widehat{\mathit{x}}}^{\left(i\right)}$. When $R\ge 1$, a conditional distribution of $z_j^{\left(i\right)}$ given $z_{j-1}^{\left(i\right)}$ can be obtained.

$$\mathrm{Pr}\left(z_j^{\left(i\right)}=z_{j-1}^{\left(i\right)}|z_{j-1}^{\left(i\right)}\right)=Q_{j,0}$$

(6)

$$\mathrm{Pr}\left(z_j^{\left(i\right)}=z_{j-1}^{\left(i\right)}+r|z_{j-1}^{\left(i\right)}\right)=\mathrm{Pr}\left(z_j^{\left(i\right)}=z_{j-1}^{\left(i\right)}-r|z_{j-1}^{\left(i\right)}\right)=\frac{Q_{j,r}}{2},\mathrm{for}r=1,\cdots ,R.$$

(7)

${\mathit{z}}^{\left(i\right)}$’s distribution can be obtained recursively using Equations (6) and (7).

(ii)  Calculation of $\mathrm{Pr}\left(y^{\left(i\right)}|\left({{\displaystyle \sum }}_{i^{\prime }=i-R}^{i+R}I\left({\tilde{z}}_j^{\left(i^{\prime }\right)}=i\right)\times {\widehat{\mathit{x}}}_j{}^{\left(i^{\prime }\right)}\right){}_{j=1}^m\right)$

This probability is obtained by a supervised machine learning model with a training dataset composed of $\left({\widehat{\mathit{x}}}^{\left(i\right)}, y^{\left(i\right)}\right)=\left({\widehat{\mathit{x}}}_1^{\left(i\right)},{\widehat{\mathit{x}}}_2^{\left(i\right)},\cdots ,{\widehat{\mathit{x}}}_m^{\left(i\right)}, y^{\left(i\right)}\right)$, i.e., the observed feature vector for product $i$ and its label. Specifically, ${\widehat{\mathit{x}}}_j^{\left(i\right)}, j=1,2,\cdots ,m$ is the $i\mathrm{th}$ feature collected at workstation $j$; $y^{\left(i\right)}$ is 1 if the $i\mathrm{th}$ product is defective, and $y^{\left(i\right)}$ is 0 otherwise. Note that we do not know the exact value of ${\mathit{x}}^{\left(i\right)}$, and it cannot be guaranteed that ${\mathit{x}}^{\left(i\right)}={\widehat{\mathit{x}}}^{\left(i\right)}$ because of possible shuffling. Instead, we utilized the value ${\tilde{\mathit{z}}}^{\left(i\right)}$ of the latent variable ${\mathit{z}}^{\left(i\right)}$ to compose a dataset to train the model.

All terms of $I\left({\tilde{z}}_j^{\left(i^{\prime }\right)}=i\right)\times {\widehat{\mathit{x}}}_j{}^{\left(i^{\prime }\right)}$ $\left(i-R\le i^{\prime }\le i+R\right)$ in the condition part of the probability vanish, except the one that satisfies ${\tilde{z}}_j^{\left(i^{\prime }\right)}=i$. Therefore, only those observed feature vectors ${\widehat{\mathit{x}}}_j{}^{\left(i^{\prime }\right)}$ where ${\tilde{z}}_j^{\left(i^{\prime }\right)}=i$ are required. This implies the feature vectors used for learning each model are determined by the distribution of ${\mathit{z}}^{\left(i\right)}$. As an example, suppose that ${\mathit{z}}^{\left(i\right)}$ can take either $\left(i,i+1,i\right)$, $\left(i,i-1,i\right)$ or $\left(i,i,i\right)$ when the shuffle range is $R=1$. When ${\mathit{z}}^{\left(i\right)}=\left(i,i+1,i\right)$ or ${\mathit{z}}^{\left(i\right)}=\left(i,i-1,i\right)$, $\left\{{\widehat{\mathit{x}}}_1,{\widehat{\mathit{x}}}_3\right\}$ is used as a feature set, but when ${\mathit{z}}^{\left(i\right)}=\left(i,i,i\right)$, then the feature set is $\left\{{\widehat{\mathit{x}}}_1,{\widehat{\mathit{x}}}_2,{\widehat{\mathit{x}}}_3\right\}$.

4. Numerical Example

This section provides a numerical example to apply the proposed model. In addition, the classification performance of the defect diagnosis is compared between our model, which takes the possibility of shuffling of product order into consideration, and the model that ignores it and assumes ${\mathit{x}}^{\left(i\right)}={\widehat{\mathit{x}}}^{\left(i\right)}$ for all values of $i$. An F1 score, harmonic mean of the precision and recall, was employed as the classification performance measure to consider both cost of false positives (FPs) and false negatives (FNs). FN cost occurs when an actual defective product is not detected, and this cost covers selling defective products such as customer claim and safety accidents (note that the addressed product is a brake disc, which is extremely safety-critical). FP cost occurs when a normal product is detected as defective, and this cost covers the costs of re-check, re-work, or disposal. Even though FN cost is much greater than FP cost, considering only one cost makes a classifier biased, so we employed an F1 score of 4.1. Line data

We generated an illustrative dataset by referring to a disk-brake processing line of a large-sized auto parts company in Korea for this numerical example’s purpose. The generated data are very similar to the real data except for machine specifications and related parameters. Even though the data are artificial, we believe that the research result can be effectively applied to the real production field when a possible shuffle of production data exists because the main idea and methodology of the research focus on data-shuffling and its effect on the detection of defectives.

The processing line consists of seven workstations (${\mathrm{S}}_1$, ${\mathrm{S}}_2$, $\cdots$, ${\mathrm{S}}_7$) including rough grinding, drilling, and tapping. Figure 4 illustrates the processing line, where a circle, a rectangle, and a colored rectangle denote a machine, a workstation, and a workstation that is not observable, respectively.

As seen in this figure, ${\mathrm{S}}_2$, ${\mathrm{S}}_3$, and ${\mathrm{S}}_5$ are composed of parallel machines, and the others include a single machine. We assumed that when a machine is abnormal the products processed by it are all defective.

Table 1. Machine specifications of the example.

Machine	Processing Time	Probability That the Machine Is Abnormal	Sensor Values
			At Normal State	At Abnormal State
${\mathrm{M}}_{1,1}$	$\mathrm{Exp}\left(0.1\right)$	0.001	$\mathcal{N}\left(10,3^2\right)$	$\mathcal{N}\left(15,4^2\right)$
${\mathrm{M}}_{2,1} {\mathrm{M}}_{2,2}$	$\mathrm{Exp}\left(0.05\right)$	0.005	$\mathcal{N}\left(8,3^2\right)$	$\mathcal{N}\left(12,3^2\right)$
${\mathrm{M}}_{3,1} {\mathrm{M}}_{3,2}{\mathrm{M}}_{3,3}$	$\mathrm{Exp}\left(0.1\right)$	0.003	$\mathcal{N}\left(9,3^2\right)$	$\mathcal{N}\left(7,2^2\right)$
${\mathrm{M}}_{4,1}$	$\mathrm{Exp}\left(0.2\right)$	0.010	$\mathcal{N}\left(10,3^2\right)$	$\mathcal{N}\left(12,3^2\right)$
${\mathrm{M}}_{5,1} {\mathrm{M}}_{5,2}$	$\mathrm{Exp}\left(0.15\right)$	0.001	$\mathcal{N}\left(10,3^2\right)$	$\mathcal{N}\left(13,3^2\right)$
${\mathrm{M}}_{6,1}$	$\mathrm{Exp}\left(0.1\right)$	0.001	$\mathcal{N}\left(12,3^2\right)$	$\mathcal{N}\left(15,4^2\right)$
${\mathrm{M}}_{7,1}$	$\mathrm{Exp}\left(0.1\right)$	0.001	$\mathcal{N}\left(12,3^2\right)$	$\mathcal{N}\left(15,4^2\right)$

shows machine specifications, including probability density functions, of processing times and sensor values attached to each machine in normal and abnormal states. $\mathcal{N}\left(\mu ,{\sigma }^2\right)$ denotes a normal distribution with mean $\mu$ and standard deviation $\sigma$, and $\mathrm{Exp}\left(\lambda \right)$ denotes an exponential distribution with mean $1/\lambda$. The numbers were all created artificially for the purpose of illustration.

4.1. Production-Line Analysis

One million simulated products were generated using Algorithm 1 to estimate partially shuffling probability $Q_{j,r}$. The result is summarized in

Table 2. Partially shuffling probability matrix of the numerical example.

		$\mathbf{Shuffle}\mathbf{Range}\mathit{r}$
		0	1	2	3	4	5	6	7	8
Work-Station $j$	1	1.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000
	2	0.6212	0.2272	0.0901	0.0360	0.0151	0.0063	0.0025	0.0010	0.0004
	3	0.7254	0.1808	0.0618	0.0207	0.0072	0.0026	0.0010	0.0003	0.0001
	4	1.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000
	5	0.8504	0.1204	0.0229	0.0050	0.0010	0.0003	0.0000	0.0000	0.0000
	6	1.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000
	7	1.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000

.

With Table 2 and Equations (6) and (7), we can obtain a probability distribution of ${\mathit{z}}^{\left(i\right)}$. For example, if we calculate the probability that the $i=78\mathrm{th}$ product arrives at Workstation (1), and overtakes the 77th product at Workstation (2), and is overtaken by two products at Workstation (3) and keeps its order afterward, then it is given by as:

overtaken by two products at Workstation (3) and keeps its order afterward. The probability is then given by:

$$\begin{array}{l}\mathrm{Pr}\left({\mathit{z}}^{\left(i\right)}=\left(i,i-1,i+1,i+1,i+1,i+1,i+1\right)\right)=Q_{1,0}\times \frac{{\mathrm{Q}}_{2,1}}{2}\times \frac{{\mathrm{Q}}_{3,2}}{2}\times {\mathrm{Q}}_{4,0}\times {\mathrm{Q}}_{5,0}\times {\mathrm{Q}}_{6,0}\times \\ {\mathrm{Q}}_{7,0} =1.000\times \frac{0.2272}{2}\times \frac{0.0901}{2}\times 0.7254\times 1.000\times 0.8504\times 1.000\times 1.000=0.00315699526\end{array}$$

(8)

4.2. Feature Data

A total of $100,000$ values of ${\mathit{x}}^{\left(i\right)},y^{\left(i\right)}$ were generated for all $j$ (i.e., $\widehat{n}=100,000$), and they were artificially shuffled to get the observed feature values, ${\widehat{\mathit{x}}}_j^{\left(i\right)}$. Product $i$ could then be expressed as a concatenation of the machine sequence, start and end times, machine state, feature vector, and label as follows:

$$\left(M_1^{\left(i\right)},\cdots ,M_7^{\left(i\right)},a_1^{\left(i\right)},\cdots ,a_7^{\left(i\right)},c_1^{\left(i\right)},c_2^{\left(i\right)},\cdots c_7^{\left(i\right)},s_1^{\left(i\right)},s_2^{\left(i\right)},\cdots ,s_7^{\left(i\right)},{\mathit{x}}_1^{\left(i\right)},{\mathit{x}}_2^{\left(i\right)},\cdots ,{\mathit{x}}_7^{\left(i\right)},y^{\left(i\right)}\right)$$

(9)

where ${\mathit{x}}_j^{\left(i\right)}$ represents the feature of statistical values, including variance, skewness, mean, kurtosis, and median extracted from sensor values of the machine processing the product at workstation $j$ [10], and $s_j^{\left(i\right)}$ indicates machine state (i.e., 0: normal or 1: abnormal). $y^{\left(i\right)}$ is the label (i.e., whether product $i$ is defective or not), calculated as ${\max }_j\left(s_j^{\left(i\right)}\right)$. The number of defective products was 2,136; that is, the defective ratio during that period was 2.136%.

Among the 100,000 values of ${\mathit{x}}^{\left(i\right)},y^{\left(i\right)}$, $70,000$ randomly chosen values were used to train the proposed graphical model, as well as the one that ignores shuffling of product order. The remaining 30,000 values were reserved for the test of the two models.

4.3. Defect Diagnosis Result

A logistic regression model was employed as a classifier because it is one of the simplest probabilistic classifiers. Other probabilistic classification models may yield similar results. We compared the classification performance between our model and the model that classified the product without consideration of possible shuffling. The proposed model considered the probability that the feature values were shuffled among the products when classifying the product, but the other, “no shuffling,” model assumed the order of collection of feature values was the same as that of the production. The preparation procedure for the dataset to be used for comparison is depicted in Figure 5.

Figure 6 shows the F1 scores of the proposed model and no shuffling model according to various cutoff values of threshold $\mathsf{\delta }$. The model decided whether a product was defective if $\mathrm{Pr}\left(y_i=1|{\widehat{\mathit{x}}}^{\left(i\right)}\right)\ge \mathsf{\delta }$.

From Figure 6, we can see that the proposed model produced a much higher F1 score over all thresholds. This implies that our graphical model dealt with the shuffling of product order and could detect a defective product far more effectively than a model that ignores shuffling. As a result, one can expect an improved classification result by considering the shuffling when some workstations are unobservable from the outside and the shuffling of product order is expected. When the threshold was small, i.e., under $\mathsf{\delta }=0.1$, then the F1 scores of the two models were almost identical, which was expected because both models tried to judge almost all products as defective. Another implication is that the best performance of the “no shuffling” model appeared when the cutoff threshold was near $\mathsf{\delta }=0.08$, while the proposed model’s performance was the best when $\mathsf{\delta }=0.17$. Therefore, the proposed model was less likely to produce a false alarm (classifying the non-defective as falsely defective) than the “no shuffling” model.

Figure 7 shows values of the F1 score, recall, precision and specificity of the proposed model for the range of $0.1\le \mathsf{\delta }\le 0.5$. As seen in this figure, the specificity was close to 1 irrespective of $\mathsf{\delta }$, implying that most non-defective products were diagnosed as non-defective (i.e., negative), which was natural because the number of negative records was much greater than the number of positive records. Precision increased as $\mathsf{\delta }$ got higher, while recall decreased. The F1 score was more affected by recall than precision because of very small values of recall. From this analysis, the following implications are drawn: First, specificity may not be a proper measure for defective diagnosis when classes are imbalanced. Second, precision and recall show the opposite trend as $\mathsf{\delta }$ increases, implying that using only one of them may lead to wrong decision. Specifically, a classifier that determines most records as positive will get higher recall, and the precision of a classifier that regards most records as negative tends to be bigger. Since the F1 score of a highly imbalanced class dataset tends to show a similar trend to recall, a classifier which is not biased to negative class such as a naïve Bayes classifier with uniform prior can be suggested to reduce both misclassification costs.

5. Conclusions

This paper addresses a realistic data quality problem of manufacturing equipment data when developing a product-defect diagnosis model—the partially shuffled data problem. Each record should correspond to a product in order to develop a proper defect diagnosis model, but most records in partially shuffled data do not correspond with a product. This problem results from one or more unobservable workstations that consist of parallel machines. One alternative may be to omit those features that may be partially shuffled if those workstations that produce partially shuffled data are near the end of the manufacturing process. However, if one or more of these workstations is near the front of the process, such as in the numerical example of this paper, omitting features obviously yields a very poor classifier with insufficient feature vector. This is because all the features collected from the workstations followed by that shuffling workstation may be possibly shuffled, and, therefore, we have to remove all features from then on. Thus, omitting features from the workstations cannot recover the shuffle of other features.

Therefore, this paper presents a probabilistic graphical model to diagnose product defects by considering the probabilities that product orders are partially shuffled. The graphical model was designed by means of a production-line analysis, and it was trained by partially shuffled data. From the numerical example, we verified that the graphical model exhibited a similar performance to a classifier trained by data that were not shuffled.

For future research, we will expand the graphical model to diagnose defects in products where the manufacturing sequence of some products varies due to machine breakdown, reworking, and other factors. In addition, we can apply this model to other fields such as traffic management and bio-informatics. Additionally, in order to enhance the model’s diagnosis accuracy as well as to improve its computational efficiency, it will be necessary to develop a proper method for extracting critical features with a shuffling strategy.