Stain Defect Classification by Gabor Filter and Dual-Stream Convolutional Neural Network

Abstract

A stain defect is difficult to detect with the human eye because of its characteristic of having a very minimal difference in brightness with the local area of the surface. Recently, with the development of Deep learning, the Convolutional Neural Network (CNN) based stain defect classification method has been proposed. This paper proposes a Dual-stream CNN for stain defect classification using a Gabor filter image. Using Dual-stream structure CNN, Gabor filter images and Gray image (Original) preserve their respective features. The experiment based on the Magnetic Tile (MT) stain data set and the Compact Camera Module (CCM) stain dataset confirms that the proposed method has an improved performance based on the precision, recall, and F1-score in comparison to the Single-stream extraction-based method. Gabor filter images have an advantage in image texture analysis and can be used as an input to the CNNs. The Dual-stream structure better extracts the features needed for classification.

1. Introduction

Detecting a stain defect image is challenging because the ambiguous boundary of a stain defect with an image processing–based method is complex. A stain defect is caused by an error in the manufacturing process, such as external contaminants, uneven paint coating, and other errors. It is slightly bright or has a dark brightness in contrast to a local area, and this contrast is difficult to detect with the naked eye.

Previous stain defect detection studies have detected stain defects using image processing techniques [1,2,3,4,5,6,7]. Kong et al. [1] proposed a method for calculating the variance of the inspection area image and judging it as a stain defect when it exceeds a certain threshold. Zhang et al. [3] detected the stain defect by calculating the Just Noticeable Difference (JND) value after applying the watershed method in the forward and reverse directions of the image to detect the boundary of the stain defect area. Han and Shi [6] extracted the wavelet image of an input image using wavelet transform and detected a stain defect by removing the background component with the difference image between the wavelet image and the input image. The above methods are methods for modeling and removing the background component and have a disadvantage in that accurate modeling of the background component is required.

A study using the Gabor filter to detect stain defects has been proposed to solve this shortcoming [8,9,10,11]. The Gabor filter was presented as a human visual system model and as a filter that extracts texture features using a sine wave with a specific frequency and direction. The Gabor filter detected and modeled the texture feature of the background component having the same continuous pattern and was used to detect the stain defect by removing the background component of the input image. Bi et al. [8] extracted the Gabor filter image using the Gabor filter’s actual part and created a binarized image based on a specific threshold to detect the stain defect on the liquid crystal display (LCD). Mao et al. [10] selected Gabor filter parameters with high stain defect classification performance by applying them to the training image. The stain defect was then detected using the selected Gabor filter.

Recently, with the development of deep learning, several studies to detect defects using a CNN have been proposed [12,13,14,15,16,17,18,19,20,21,22,23,24]. Xiao et al. [12] proposed a hierarchical feature–based CNN (H-CNN) structure that generates Regions Of Interest (ROI) using region-based CNN (R-CNN) and detects the defect using a fully CNN (F-CNN). Singh et al. [14] compared the classification performance for stain defects on metal surfaces using a CNN such as ResNet [25] and YOLO [26]. Yang et al. [15] created a defect detection model by learning stain defects using the transfer learning technique of a deep CNN extractor trained in advance on the ImageNet Large Scale Visual Recognition Challenge 2012 (ILSVRC2012) dataset. Hou et al. [24] proposed a directional kernel to detect weak scratches.

The segmentation method can accurately explore the shape of the defect, but it takes a long time to compute. In addition, several studies to detect defects using segmentation have been proposed [27,28,29]. Tabernik [27] proposed After applying segmentation, the defect was detected using the method using the detection. In addition, Huang [28] introduced The MT set first, and defects were detected using the McuePushU network.

The previous deep learning–based stain defect classification method used a single inspection image as input [14,15]. The Gabor filter image has an advantage in image texture analysis, so performance improvement is expected when used as an input for CNN. However, it is necessary to design a CNN to extract features advantageous for stain defect classification using different domains, such as inspection and Gabor filter images.

This paper proposes a CNN for stain defect classification using inspection images and Gabor filter images intending to improve the performance of stain defect classification. First, a Gabor filter image is created using a Gabor filter on the inspection image obtained from the camera. A GLCM is constructed for the Gabor filter image, and an IDM value, a local homogeneity feature, is extracted from GLCM. The Gabor filter image is selected based on the IDM, and the inspection image and the selected Gabor filter image are used as input to the CNN.

The contribution of this paper is as follows.

1. Gabor images with Gabor filters applied were used to improve the accuracy of stain defect detection. A method of adding features for spot detection using Gabor images was used rather than a method without Gabor filter images. It was confirmed through experiments that the method using the Gabor image improved the accuracy.

2. The stain defect classification performance was improved through a Dual-stream extraction based on inspection image and Gabor filter images, compared to the existing Single-stream extraction. The most suitable image for stain defect classification from the extracted Gabor filter image was selected and used as an input to a Dual-stream extraction that connects two streams in parallel. Through this, the features of each input image were extracted independently and merged in a fully connected layer to improve the stain defect classification accuracy.

This paper is structured as follows. Section 2 describes the structure of the stain defect classification system proposed in this paper. Section 3 describes the Gabor filter image creation and selection process. Section 4 describes the structure of a Single-stream extraction and a Dual-stream extraction. Section 5 demonstrates the performance of the proposed method, the Dual-stream extraction, by conducting experiments on the MT and CCM datasets. Lastly, Section 6 describes the conclusion of this paper and future research.

2. Stain Defect Classification System

The structure of the stain defect classification system proposed in this paper is shown in Figure 1. The proposed method comprises gray-scale transform, Gabor image generation, Gabor image selection, and CNN. The general process is as follows. The input inspection image $I_{RGB}$ is converted to a gray-scale inspection image $I_{Gray}$ through gray-scale transform. After this, the Gabor image was generated and selected from the input image. Finally, the classification result O was obtained using a CNN with the selected Gabor images and gray-scale inspection image.

Let $R_{mn}$, $G_{mn}$, and $B_{mn}$ are red channel, green channel, and blue channel of the image, respectively, while M and N are the height and width of the image, respectively.

$$\begin{array}{c}\hfill {\mathrm{I}}_{\mathrm{RGB}}=\left[\begin{array}{cccc}{i}_{11}& i_{12}& \cdots & i_{1N}\\ i_{21}& i_{22}& \cdots & i_{2N}\\ ⋮& ⋮& \ddots & ⋮\\ i_{M1}& i_{M2}& \cdots & i_{MN}\end{array}\right]\\ \hfill where,i_{mn}\left[\begin{array}{ccc}{R}_{mn}& G_{mn}& B_{mn}\end{array}\right]\\ \hfill i_{mn}\in {\mathbb{R}}^{1X3}(m=1,\cdots ,M,n=1,\cdots ,N)\end{array}$$

(1)

$I_{Gray}$ can be obtained as follows.

$$\begin{array}{c}\hfill {\mathrm{I}}_{\mathrm{Gray}}=\left[\begin{array}{cccc}{j}_{11}& j_{12}& \cdots & j_{1N}\\ j_{21}& j_{22}& \cdots & j_{2N}\\ ⋮& ⋮& \ddots & ⋮\\ j_{M1}& j_{M2}& \cdots & j_{MN}\end{array}\right]\\ \hfill where,j_{mn}=\frac{0.229R_{mn}+0.587G_{mn}+0.114B_{mn}}{3}\\ \hfill j_{mn}\in {\mathbb{R}}^{1X1}(m=1,\cdots ,M,n=1,\cdots ,N)\end{array}$$

(2)

The Gabor image generation step is a step to obtain the Gabor filter image $G\left(k\right)$ for each phase. The Gabor filter transform uses a complex kernel with real and imaginary parts.

$$G=\left\{G\left(k\right)|k=1,2,\cdots ,K\right\}$$

(3)

In the Gabor filter image selection step, a Gabor filter image with features useful for stain defects is selected among the generated Gabor filter images. By inputting the gray-scale inspection image $I_{Gray}$ and the selected Gabor filter image $\acute{G}$ into the CNN $E\left(\ast \right)$, the classification result O for the inspection image is the output.

$$O=E(I_{Gray},\acute{G})$$

(4)

3. Gabor Image from Stain Defect Image

Gabor filter is used for image processing for reasons like how the human visual system responds. Gabor filter is a Gaussian filter modulated by a sine function. The Gabor filter image is the result of a convolution operation between the Gabor filter and the image. Because the Gabor filter can acquire the result according to the direction and frequency, it is possible to check what features exist in the input image in a specific direction and frequency band.

Let x and y be the pixel coordinates of the Gabor filter. Gabor filter can be defined as the following equation. $\sigma$ is a variable that determines the width of the Gabor filter kernel. $\lambda$ is a variable that determines the addition/subtraction period of the Gabor filter kernel. $\theta$ is a variable that determines the direction of the Gabor filter kernel, and $\psi$ is a mobility variable of the filter. $\gamma$ is a variable for adjusting the horizontal and vertical ratio of the filter. Lastly, $x^{\prime }$ and $y^{\prime }$ are the x and y coordinates of the Gabor filter rotated by $\theta$, respectively.

$$\begin{array}{c}\hfill g(x,y;\lambda ,\theta ,\psi ,\sigma ,\gamma )=exp(-\frac{{x\prime }^2+{\gamma }^2{y\prime }^2}{2{\sigma }^2})cos(2\pi \frac{x^{\prime }}{\lambda }+\psi )\\ \hfill where,x^{\prime }=xcos\theta +ysin\theta y^{\prime }=-xsin\theta +ycos\theta \end{array}$$

(5)

Gabor filter can obtain various result values according to internal parameters. Among these results, to select an image to improve the stain defect classification performance, the GLCM and the Haralick texture feature based on it were used. GLCM transformation is an algorithm that transforms two neighboring pixel values in an image into spatial coordinates of an M × N matrix. Let the Δm and Δn are offsets in the width and height directions, respectively. the GLCM result matrix $C_{\Delta m,\Delta n}$ can be defined as follows.

$$\begin{array}{c}\hfill C_{\Delta m,\Delta n}(i,j)=\sum _{m=1}^M\sum _{n=1}^N=\left\{\begin{array}{c}1if{I}_{Gray}(m,n)=iand{I}_{Gray}(\acute{m},\acute{n})=j\hfill \\ 0otherwise\hfill \end{array}\right.\\ \hfill where,\acute{m}=m+\Delta m\acute{n}=n+\Delta n\end{array}$$

(6)

The GLCM matrices for $0^{\circ }$, ${45}^{\circ }$, ${90}^{\circ }$, and ${135}^{\circ }$ directions can be obtained, respectively, according to the Δm and Δn. The Δm and Δn values for each angle are respectively (0,1), (1,0), (−1,1), (−1,−1). As the stain defect has an unspecified texture, features for all four directions should be used. Therefore, the GLCM matrix for four directions was obtained, and then the average value was used. The average matrix P for the GLCM can be calculated as follows.

$$P=\frac{C_{0,1}+C_{1,0}+C_{-1,1}+C_{-1,-1}}{4}$$

(7)

This paper used the Inverse Difference Moment (IDM) feature, one of the Haralick texture features, as a selection criterion for the generated Gabor filter image. IDM is an index indicating the local homogeneity of an image and it can be expressed as the following equation. Let W and H be the width and height of GLCM, respectively, and W and H have the same value (W = H).

$$IDM=\sum _{i=0}^W\sum _{j=0}^H\frac{P(i,j)}{1+{(i-j)}^2}$$

(8)

The IDM value was used to select the Gabor filter images for stain defect classification. The denominator of the IDM equation becomes 1 in the diagonal component of the GLCM, having the same brightness value as the neighboring pixels, and the IDM value becomes the maximum in the main diagonal section. Meanwhile, as the distance from the diagonal component of the GLCM increases, the denominator’s value rises, and the IDM value decreases. In other words, a large IDM value suggests the presence of multiple pixels having similar brightness, and a small IDM value means that there are many pixels with a difference between neighboring pixels. Therefore, the smaller the IDM value, the lower the local homogeneity in the image, and it can be said that the image has a relatively clear boundary of the stain defect.

3.1. Gabor Image Generation

To extract the stain defect features for various frequency bands and directional, a total of 64 Gabor filters are generated according to the 8 amplitudes ${\lambda }_{\alpha }$ and eight phases ${\theta }_{\beta }$ as shown in the following equation, respectively. The size of each Gabor filter is 3 × 3.

$$\begin{array}{c}\hfill g(x,y;{\lambda }_{\alpha },{\theta }_{\beta },\psi ,\sigma ,\gamma )=exp(-\frac{x^{\prime 2}+{\gamma }^2{y}^{\prime 2}}{2{\sigma }^2})cos(2\pi \frac{x^{\prime }}{{\lambda }_{\alpha }}+\psi )\\ \hfill where,x^{\prime }=xcos{\theta }_{\beta }+ysin{\theta }_{\beta },y^{\prime }=-xsin{\theta }_{\beta }+ycos{\theta }_{\beta }\end{array}$$

(9)

$${\lambda }_{\alpha }=2^{-\frac{4\alpha +2}{2}},{\theta }_{\beta }=\frac{\beta }{8}\pi where,\alpha =0,1,\cdots ,7,and\beta =0,1,\cdots ,7$$

(10)

Gabor filter image $I_{Gabor}$ are created through convolution operation between $I_{Gray}$ and Gabor filter created.

$$I_{Gabor}(m,n;\alpha ,\beta )=\sum _{x=-1}^{x=1}\sum _{y=-1}^{y=1}g(x,y;{\lambda }_{\alpha },{\theta }_{\beta },\psi ,\sigma ,\gamma )·I_{Gray}(m+x,n+y)$$

(11)

3.2. Gabor Image Selection

Among the 8 Gabor images created in Section 3.1, N images with the smallest IDM value are selected. An image with a small IDM is an image with a large change in values between pixels. Efficient feature extraction for stain detection is possible using the selected Gabor image. The Gabor image selection steps consist of follows.

Step (1) For the Gabor image created in Section 3.1, create an average image ${\acute{I}}_{Gabor}$ for each phase.

$${\acute{I}}_{Gabor}(m,n;\beta )=\frac{1}{7}\sum _{\alpha =0}^7{I}_{Gabor}(m,n;\alpha ,\beta )$$

(12)

Step (2) Normalization is performed on the Gabor filter image ${\acute{I}}_{Gabor}$.

$${\acute{I}}_{Gabor}(m,n;\beta )=\frac{{\acute{I}}_{Gabor}(m,n;\beta )-min({\acute{I}}_{Gabor}(m,n;\beta ))}{max({\acute{I}}_{Gabor}(m,n;\beta ))-min({\acute{I}}_{Gabor}(m,n;\beta ))}$$

(13)

Step (3) According to Equations (6) and (7), the Gabor filter image $I_{Gabor}$ (m,n;$\beta$) is transformed into a GLCM, and the average matrix $P_{\beta }$ is obtained.

Step (4) IDM feature values are extracted from each average matrix $P_{\beta }$.

$$F=\left\{F_{\beta }|F_{\beta }=\sum _{i=0}^W\sum _{j=0}^H\frac{P_{\beta }(i,j)}{1+{(i-j)}^2}\right\}where,\beta =0,1,\cdots ,7$$

(14)

Step (5) The top N Gabor filter images with the highest IDM feature values are selected. The Gabor filter image selection Algorithm 1 proceeds as follows.

Figure 2 and Figure 3 show the Gabor filter images for each phase concerning the stain defect image generated on the MT dataset and CCM dataset. Figure 2a and Figure 3a are a gray-scale image for the inspection image, and it also is a ground truth image showing the location of the stain defect. Figure 2b–i and Figure 3b–i are Gabor filter images according to phase, selected for CNN model creation according to their IDM values. The clarity of the boundary difference between the stain defect and the non-defect one depends on the Gabor filter’s phase. In the MT dataset of Figure 2, the IDM values increase in the order of (b), (c), (h), (d), (f), (i), (e), and (g), and in the CCM dataset of Figure 3, the IDM values increase in the order of (b), (c), (d), (i), (f), (h), (e), (g) increase.

Algorithm 1 Select N Gabor filter image

Require: Gabor filter image ${\acute{I}}_{Gabor}(m,n;\beta )$, IDM value F Ensure: Selected Gabor filter image set S  $S=\left\{0\right\}$  for $Size\left(S\right)<N$do      $\gamma =\underset{\beta }{\mathrm{argmin}}\left(F\right)$      $S\Leftarrow {\acute{I}}_{Gabor}(m,n;\gamma )$  end for

4. Feature Vector Extraction Based CNN

According to the previous process, the input of the CNN is a $I_{Gray}$ and a set of Gabor filter images $S=\left\{s_1,s_2,\cdots ,s_N\right\}$. In general, CNN has one stream and uses a method of inputting a single image with N+1 channels by combining a gray-scale inspection image and a Gabor filter image. However, a Gabor filter image is generated through a Gabor filter and has a different domain from a gray-scale inspection image. When features are extracted from images of other parts with a CNN composed of a Single-stream, the process of extracting unique features that can be extracted from each part may not be performed smoothly. In this case, constructing a CNN with multiple streams to analyze the features of each part and then combining them can increase the detection accuracy. This section describes the structures of a Single-stream extraction and a Dual-stream extraction, respectively.

4.1. Single-Stream Extraction

Single-stream extraction is the most common type of CNN. Single-stream extraction is a structure that has been used in many CNNs, such as GoogLeNet. Single-stream extraction has a funnel-shaped structure in which layers are stacked and descended.

The CNN used in this paper was designed based on Res_Block in Figure 4 consisting of three convolution layers, is a method devised in Res Net [25] that uses the residual learning method. Res_Block simplifies the loss function with a shortcut structure between the input and output and is designed to alleviate the gradient vanishing problem that occurs when designing a deep CNN. The CNN used in this paper was designed based on Res_Block in Figure 4. Res_Block was applied to both Single-stream and Dual-stream extraction structures.

Figure 4 shows the structure of the Single-stream extraction used in this paper. The $I_{Merge}$ was constructed by stacking Gabor filter image N channels and gray-scale inspection images in parallel, and this $I_{Merge}$ is used as an input to CNN. A feature map with 64 channels was extracted through a convolution layer using a 7 × 7 kernel and sampled at a size of 112 × 112 using max pooling. After that, the feature map was extracted through 4 Res_Blocks, and max pooling was applied between each Res_Block. The numbers in parentheses of Res_Block mean $C1$, $C2$, $C3$ of Res_Block, and the number of repetitions of Res_Block, respectively. The number at the bottom of Res_Block means the width and height of the feature map, which is the intermediate result of the layer. 2048 × 1 size feature vector V Merge is output from the last Res_Block, and when this vector passes through the fully connected layer, it is converted to O as a classification result. The output result O consists of a confidence score for OK and stain defects.

4.2. Dual-Stream Extraction

A stream is an independent layer structure before the feature maps merge in a CNN. In a Single-stream extraction, there is only one stream because there is no merging process. The feature map of the Gabor filter image and the gray-scale inspection image pass through a Single-stream, so the features of the Gabor filter image and the gray-scale inspection image are mixed in the training process. This process may cause a decrease in classification accuracy.

Alternatively, a Dual-stream extraction is composed of two streams. In this study, separate streams for the gray-scale inspection image and the Gabor filter image were designed to solve this decrease in classification accuracy. A CNN independently extracted features for each gray-scale inspection image and the Gabor filter image was constructed through the separate streams.

Figure 5 shows the dual-stream extraction of the network proposed in this paper. The Dual-stream extraction has two streams that extract features from the input gray-scale inspection image $I_{Gray}$ and Gabor filter image $S=\left\{s_1,s_2,\cdots ,s_N\right\}$, and each stream for extracting features can be configured. The upper stream extracts a 2048 × 1 size feature vector $V_{Gray}$ for the gray-scale inspection image through the convolution layer, pooling layer, and Res_Block. At the same time, the stream at the bottom, which has the same structure as the upper stream but has a different weight, extracts the 2048 × 1 size feature vector $V_{Gabor}$ for the Gabor filter image through the convolution layer, pooling layer, and Res_Block. The feature vector $V_{Concat}$ of 4096 × 1 size, connecting the two feature vectors $V_{Gray}$ and $V_{Gabor}$, is converted to O as a classification result of the fully connected layer. The two are separate streams that do not share weights, and their features are preserved independently without mixing.

Table 1. Dual-stream network configuration.

Layer	Output Size	${\mathit{I}}_{\mathbf{Gray}}\mathit{stream}$	${\mathit{s}}_1,{\mathit{s}}_2,\cdots ,{\mathit{s}}_{\mathit{N}}\mathit{stream}$
Input	224 × 224	224 × 224 @ 1	224 × 224 @ N
Conv	112 × 112	7 × 7, 64, stride 2	7 × 7, 64, stride 2
pooling	56 × 56	3 × 3, stride 2	3 × 3, stride 2
Res_Block1	56 × 56	$\left[\begin{array}{c}1\times 1,64\\ 3\times 3,64\\ 1\times 1,256\end{array}\right]$ × 3	$\left[\begin{array}{c}1\times 1,64\\ 3\times 3,64\\ 1\times 1,256\end{array}\right]$ × 3
Res_Block2	28 × 28	$\left[\begin{array}{c}1\times 1,128\\ 3\times 3,128\\ 1\times 1,512\end{array}\right]$ × 4	$\left[\begin{array}{c}1\times 1,128\\ 3\times 3,128\\ 1\times 1,512\end{array}\right]$ × 4
Res_Block3	14 × 14	$\left[\begin{array}{c}1\times 1,256\\ 3\times 3,256\\ 1\times 1,1024\end{array}\right]$ × 6	$\left[\begin{array}{c}1\times 1,256\\ 3\times 3,256\\ 1\times 1,1024\end{array}\right]$ × 6
Res_Block4	7 × 7	$\left[\begin{array}{c}1\times 1,512\\ 3\times 3,512\\ 1\times 1,2048\end{array}\right]$ × 3	$\left[\begin{array}{c}1\times 1,512\\ 3\times 3,512\\ 1\times 1,2048\end{array}\right]$ × 3
Concat	4096 × 1	Average pool
Output	2 × 1	Fully Connected

shows the Dual-stream extraction structure. $I_{Gray}$ stream is the first flow in Figure 5, and gray-scale images are input. For the structure, the resnet50 structure was used. The $S=\left\{s_1,s_2,\cdots ,s_N\right\}$ stream is the second flow in Figure 5 and inputs the selected Gabor images. The structure uses resnet50 in the same way. $V_{Gray}$ and $V_{Gabor}$ in Figure 5 output to each flow are merged with $V_{Concat}$, and the classification result is output as a fully connected layer. In this paper, ResNet is used as an example, but other networks for feature extraction may be used. We propose improving classification performance depending on the input and Gabor images, where the network is unimportant.

5. Experimental Result

5.1. Setup

For the experiments of this paper, a workstation equipped with an i7 CPU and NVIDIA RTX 2080 Ti was used. The PyTorch library was used in the Python environment for training, validation, and testing CNNs. An experimental program was created based on MFC in Visual Studio 2017. In the Gabor filter, $\sigma$ and $\psi$ were designated as $\sigma$ = 2.5 and $\psi$ = 45°, respectively.

Precision, Recall, and F1-score were used as evaluation indicators for the network performance. F1-score is a harmonic average of Precision and Recall. It evaluates the overall classification model performance and is defined as follows.

$$F1-score=\frac{2\times Recall\times Precision}{Recall+Precision}$$

(15)

The datasets used in the experiment were the MT dataset and the CCM dataset. Each dataset was divided into train, validation, and test groups, and these groups were respectively used for CNN training, CNN validation, and evaluation indicators calculation. The Single-stream extraction was trained and verified by only the gray-scale inspection image. In contrast, the Dual-stream extraction was trained and verified using the gray-scale inspection image and all Gabor filter images generated. All experiments were repeated five times, and the average value was compared. The learning epoch is 100, the learning rate is 10 × 10⁻⁵, the batch size is the maximum size of the GPU, and the optimizer uses Adam. The image input to the network is 224 × 224, the network size of the dual stream is doubled that of the single stream, and the size of the network (the number of weights) does not change significantly even though the input Gabor image is increased.

5.2. Magnetic Tile Dataset

Figure 6 shows the MT dataset [28]. In the MT dataset, stain defects caused by external contamination exist on the metal tile surface. This paper constructed a dataset by extracting a 224 × 224 image patch from an original image. If the image extract based on the ground truth image contained a stain defect region, it was labeled as a stain defect image, and if it did not contain a stain defect, it was labeled as an “OK” image.

Table 2. Magnetic tile dataset.

Type	Train	Validation	Test
OK	193	58	100
Stain	188	60	100
Total	381	118	200

shows the number of training, validation, and evaluation data of the MT dataset used in the experiment.

Table 3. Performance of proposed method on magnetic tile dataset.

Network Structure	Number of Gray Image	Number of Gabor Images	Precision	Recall	F1-Score
Single-stream	1	0	93.2	97.6	95.3
Single-stream	0	8	96.7	99.0	97.8
Single-stream	1	8	96.5	99.0	97.7
Dual-stream	1	8	95.8	99.4	97.6

shows the classification performance according to the CNN structure and gray and Gabor filter images select method. As shown in Table 3, the F1-score shows the highest performance with 97.8 when using eight Gabor filter images. It improved by about 2.5 percent compared to the network trained by inputting only Gray Image. Through this, it was confirmed that the Gabor filter was effective in detecting stains. In addition, Single-stream or dual-stream inputting Gray Image and Gabor image together can confirm that the accuracy is slightly reduced, but both methods can confirm that the accuracy is increased compared to the network using only Gray Image.

Table 4. Performance of defect detection method on magnetic tile dataset.

Method	Precision	Recall	F1-Score
SEGDEC	75.1	92.2	82.0
MCuePush	98.6	96.6	97.6
CS-FLOW	99.6	90.4	94.7
Proposed	96.7	99.0	97.8

shows the comparison results with the existing defect detection method using the MT data set. SEGDEC [27] and MCuePush [28] are defect detection methods based on a segmentation image, and CS-Flow [30] is a method of detecting defects by learning only a normal image. SEGDEC Among the comparative experiments, the proposed method had the highest F1-score. In the case of SEGDEC and MCuePush, defects were output from OK images as the method for detecting defects using the segmentation network. In addition, the F1-score was higher than CS-Flow, in which only OK images were input.

5.3. Compact Camera Module Dataset

The CCM dataset [31] is where a stain defect exists in the cover of the camera module. Figure 7 shows an example of a CCM dataset.

Table 5. Compact camera module dataset.

Type	Train	Validation	Test
OK	353	96	100
Stain	346	82	100
Total	699	178	200

shows the number of training, validation, and evaluation data of the CCM dataset used in the experiment. In this paper, the 2048 × 2048 CCM image is divided into 224 × 224 image patches, and then the image patch is trained with CNN to generate a stain defect classification model. The MT stain defect dataset is labeled as a stain defect if a stain defect is included in the image patch region in the ground truth image. Meanwhile, it is labeled as “OK” if it is excluded. The CNN receives an image patch and outputs the classification result of each image patch. In the classification result of each image patch, the image was classified as a stain defect if even one image patch was a stain defect. Otherwise, it was classified as OK if none of the image patches was a stain defect.

Table 6. Performance of proposed method on compact camera module dataset.

Network Structure	Number of Gray Image	Number of Gabor Images	Precision	Recall	F1-Score
Single-stream	1	0	79.6	93.2	85.7
Single-stream	0	8	96.6	97.2	96.8
Single-stream	1	8	92.7	92.2	92.3
Dual-stream	1	8	89.5	93.0	91.1

shows the classification performance according to the CNN structure and gray and Gabor filter images select method. As shown in Table 6, the F1-score shows the highest performance with 97.6 when using eight Gabor filter images. It improved by about 11.1 percent compared to the network trained by inputting only Gray Image. Through this, it was confirmed that the Gabor filter was effective in detecting stains. In addition, Single-stream or dual-stream inputting Gray Image and Gabor image together can confirm that the accuracy is slightly reduced, but both methods can confirm that the accuracy is increased compared to the network using only Gray Image.

Table 7. Performance of defect detection method on compact camera module dataset.

Method	Precision	Recall	F1-Score
SEGDEC	56.2	100	71.9
MCuePush	85.4	91.4	88.3
CS-FLOW	50.0	100	66.7
Proposed	96.6	97.2	96.8

shows the comparison results with the existing defect detection method using the CCM data set. Among the comparative experiments, the proposed method had the highest F1-score. In the case of SEGDEC and MCuePush, defects were output from OK images as the method for detecting defects using the segmentation network. In addition, the CS-Flow method, in which only OK images were input, classified all images as stain images.

5.4. Defect Detection Performance According to the Number of Gabor Images

The comparison was made according to the number of selected Gabor images. In Section 5.2 and Section 5.3, it was confirmed that the network using the Gabor image performed better than the network using only the gray image. In addition, we compared the accuracy according to the number of inputs of Gabor images. All gray images are input, and only the number of Gabor images is adjusted to 2, 4, 6, and 8, and the learning epoch and learning rate are the same as the settings of Section 5.1.

Table 8. Performance according to the number of gabor images on magnetic tile dataset.

Network Structure	Number of Gray Image	Number of Gabor Images	Precision	Recall	F1-Score
Single-stream	1	2	95.4	98.4	96.9
	1	4	98.6	98.2	98.4
	1	6	96.9	99.2	98.0
	1	8	96.5	99.0	97.7
Dual-stream	1	2	96.1	99.0	97.6
	1	4	95.1	96.6	95.8
	1	6	95.4	98.6	97.0
	1	8	95.8	99.4	97.6

shows the results of the experiments on the use of the MT dataset with different numbers of Gabor filter images selected. The single-stream structure had the highest accuracy when the Gabor filter image was 4 or 6, and the dual-stream structure had the highest accuracy when the Gabor filter image was 2 or 8. As a result of comparing the accuracy of single-stream and dual-stream in the same MT dataset, the accuracy of single-stream was higher. The blob of the MT dataset showed that the single-stream structure had higher accuracy. The above experimental results confirmed that many Gabor filters did not always have high accuracy and that the accuracy was increased when an image with good features was selected and used based on the IDM value. Unnecessary Gabor filter images acted as noise components in stain defect classification.

Table 9. Performance according to the number of gabor images on compact camera module dataset.

Network Structure	Number of Gray Image	Number of Gabor Images	Precision	Recall	F1-Score
Single-stream	1	2	99.6	98.4	99.0
	1	4	96.6	96.8	96.7
	1	6	96.6	96.2	96.3
	1	8	92.7	92.2	92.3
Dual-stream	1	2	99.6	99.0	99.3
	1	4	97.8	97.6	97.7
	1	6	94.5	94.2	94.3
	1	8	89.5	93.0	91.1

shows the results of the experiments on the use of the CCM dataset with different numbers of Gabor filter images selected. As a result of the experiment, both the Single-stream and Dual-stream showed the highest performance when two Gabor filter image was added. it was confirmed that using images with well-extracted IDM reference features is more efficient, rather than increasing accuracy by having many Gabor filter images. It was confirmed that the dual-stream structure has higher accuracy than the single-stream structure, albeit in a minute way.

5.5. Class Activation Map According to the Number of Gabor Images

The Grad CAM (Class Activation Map) method [32] was applied to visualize the detection area for CCM images. $V_{Merge}$ of the Single-stream network in Figure 4, and $V_{Gray}$ and $V_{Gabor}$ of the Dual-stream network in Figure 5 were investigated and displayed.

Figure 8 shows a single-stream result for the stain image of the CCM dataset, (a) is a stain image with stains on the left and right sides, (b) and (g) are the output vector according to the input image as a CAM image This is the output image. (b) is a single-stream output CAM image inputting only one gray-scale image, and it was classified as OK without detecting stains. It can be seen that the red points of the CAM image are distributed in the center. (c) is the output CAM image from inputting eight Gabor filter images into a single stream. It is classified as a stain, and it can be confirmed that the red point of the CAM image is visible around the stain. (d) is an output CAM image by inputting a gray-scale image and two selected Gabor filter images. It was classified as a stain, and red points were distributed around the stain. (e) is a single-stream CAM image input of a gray-scale image and four selected Gabor filter images. Stains present on the left side were detected. (f) is a CAM image input of a gray-scale image and six selected Gabor filter images. It was classified as a stain, and the red distribution of CAM confirmed a wide area so that the stain could be well detected. (g) is the CAM result when all eight gray-scale images and Gabor filter images are input as single-stream. By checking the stain on the left, you can confirm that it is classified as a stain.

Figure 9 shows the dual-stream result of the stain image of the CCM dataset, (a) is the stain image with stains on the left and right, (b) to (i) are the output vector according to the input image to the CAM image This is the output image. (b) and (c) are dual-stream output CAM images of one gray-scale image, and two selected Gabor filters, (b) is a gray-scale image stream, and (c) is a CAM image resulting from a Gabor filter image stream. (b) and (c) were classified as stains. The red part in (b) does not exist around the stain. Still, the red part in (c) exists around the stain, so the Gabor filter stream detects the stain and can check that (d) and (e) are dual-stream output CAM images of one gray-scale image, and four selected Gabor filters, (d) is a gray-scale image stream, and (e) is a CAM image resulting from a Gabor filter image stream. (d) and (e) were classified as stains, and it was confirmed that both the red part of (d) and the red part of (e) were marked toward the stain. (f) and (g) are dual-stream output CAM images of one gray-scale image and six selected Gabor filters, (f) is a gray-scale image stream, and (g) is a CAM image resulting from a Gabor filter image stream. (f) and (g) were classified as stains, and it can be confirmed that (f) detected the right stain and (g) detected the left stain. (h) and (i) are dual-stream output CAM images of one gray-scale image and eight Gabor filters, (h) is a gray-scale image stream, and (i) is a CAM image resulting from a Gabor filter image stream. (h) and (i) were classified as stains, and (i) detected the left stain compared to (h).

Figure 10 shows the single-stream result for the OK image of the CCM dataset, and the defect located in the middle of (a) is a shadow and not a stain. (b) to (g) are images in which the output vector, according to the input image, is output as a CAM image. (b) is an output CAM image of a single-stream with only one gray-scale image input, and it can be confirmed that the red point was misclassified as a stain by looking at the shadow. (c) is the CAM image of the output vector obtained by inputting 8 Gabor filter images into a single stream. Although the red point in the CAM image was out of the shadow area, it was misclassified as a stain. (d) is an output CAM image by inputting a gray-scale image and two selected Gabor filter images. It was misclassified as a stain, and red points were distributed around the shadow. (e) is a single-stream CAM image input of a gray-scale image and four selected Gabor filter images. The shadow on the left side was misclassified as a stain. (f) is a CAM image input of a gray-scale image and six selected Gabor filter images. A slight noise on the right side of the image was misclassified as a stain. (g) is the CAM result when all eight gray-scale images and Gabor filter images are input as single-stream. The shadow on the left was misclassified as a stain.

Figure 11 shows the dual-stream result for the OK image of the CCM dataset, and the defect located in the middle of (a) is a shadow and not a stain. (b) to (i) are images in which the output vector, according to the input image, is output as a CAM image. (b) and (c) are dual-stream output CAM images of one gray-scale image, and two selected Gabor filters, (b) is a gray-scale image stream, and (c) is a CAM image resulting from a Gabor filter image stream. (b) and (c) were classified as OK, and it can be seen that the red part of (b) and (c) does not exist in the shadow. (d) and (e) are a gray-scale image and a selected Gabor. 4 filters are output CAM images of Dual-stream, (d) is a gray-scale image stream, and (e) is a CAM image resulting from a Gabor filter image stream. (d) and (e) were classified as OK, and it was confirmed that both the red part of (d) and the red part of (e) were not displayed toward the shadow side. (f) and (g) are dual-stream output CAM images of one gray-scale image and six selected Gabor filters, (f) is a gray-scale image stream, and (g) is a CAM image resulting from a Gabor filter image stream. (f) and (g) are classified as OK, and (g) has red color in the left shadow, but other places are similarly distributed, confirming that only the shadow part is not checked. (h) and (i) are dual-stream output CAM images of one gray-scale image and eight Gabor filters, (h) is a gray-scale image stream, and (i) is a CAM image resulting from a Gabor filter image stream. (h) and (i) were classified as OK, and it can be confirmed that both (h) and (i) do not see the shadow part (the red part does not exist in the shadow).

6. Conclusions

This paper proposed a Dual-stream network using a Gabor filter image and gray-scale inspection image as input to detect stain defects. The Gabor filter image was selected according to the IDM score based on the GLCM and used as the input of the Dual-stream network for stain defect classification.

As a result of the experiment, the Gabor filter method was more accurate than the method without using it, and the single-stream structure in the MT dataset and the dual-stream structure in the CCM dataset had higher accuracy. Through this, the performance of single and dual-stream structures can be different depending on the dataset’s characteristics (background, degree of stain). In addition, it was confirmed that selecting N Gabor filter images based on the IDM value is more accurate than using all eight images without selecting them. Through this, it was confirmed that among the Gabor filter images, there are images with important information and other images with lower accuracy due to noise. In addition, it was confirmed that the stream in which the Gabor image was input by outputting the CAM image finds stain.

The disadvantage of the proposed method is that the method for selecting N Gabor filter images depends on the dataset. Although it was confirmed that the method using Gabor filter images is more accurate than the method without, the value of selecting N Gabor filter images according to the dataset must be directly compared with learning and evaluation. It would be nice always to find a good method, but you have to find N chapters while comparing yourself.

Developing a method to calculate optimized Gabor filter parameters for defect classification of datasets through deep learning without using static Gabor filter parameters is planned as a future study. In addition, a study on the structure of a CNN that effectively detects stain defects will be conducted.