Fabric Defect Detection Algorithm Based on Image Saliency Region and Similarity Location

Abstract

In order to solve the problem of defect detection and to contour accurate segmentation of periodic texture fabric images, a fabric defect detection method based on saliency region and similarity location is proposed. Firstly, the image to be detected was processed by color space conversion, Gaussian filtering, and contrast enhancement, and a frequency-tuned (FT) salient region detection algorithm was used to estimate a saliency map of the enhanced image. The fabric image was divided into image blocks of the same size with overlapping areas through a sliding window, and then the statistical parameters of each image block were calculated. The outliers in the statistical parameters were filtered out using inter-quartile range (IQR). Through the positioning and processing of image defects, abnormal elimination was carried out, and the defect outline was finally obtained. The experimental results show that the method proposed in this paper has better performance in terms of qualitative characterization of Acc, Precision, Recall, and F₁ score.

1. Introduction

In order to comply with the trend of global economic integration and because fabric defects seriously affect product quality and cause huge economic losses, the market requirements for the quality of textiles are also correspondingly increasing [1].

Product quality control plays an important role in the textile production process. The scientific and efficient detection of fabric defects is a key component of product quality control [2]. In the process of fabric production, defects caused by yarn quality and mechanical failure seriously affect fabric quality [3]. The traditional manual method of fabric defect detection has serious shortcomings, such as low efficiency, low accuracy, and lack of reliability [4,5]. In recent years with the rapid development of computer science and technology, people are paying more attention to the field of machine vision. The advantages of no contact, high efficiency, convenience, and objectivity make machine vision more widely used in defect detection. Many machine-learning and neural network techniques have been used to automate the visual inspection of industrial production of fabrics [6,7], cracks [8], ceramics [9], microsurfaces [10], printing [11], welding [12], wood [13], and more.

The existing fabric defect detection methods based on digital image processing can be roughly divided into the following four categories: mathematical statistical detection methods, visual saliency detection methods, transform domain detection methods, and deep-learning detection methods. Mathematical statistical detection methods guide fabric defect detection through the statistical analysis of image gray values, texture, and other features. The detection effects of these methods are greatly affected by the selection of statistical parameters and statistical methods, and there are problems of large amounts of calculation and low calculation efficiency [14]. The method proposed by Cao Junjie et al. used a principal component analysis method and added noise to the model, resulting in low computational efficiency [15]. Visual saliency detection methods use random defects to block significant changes in the periodic characteristics of fabric texture to realize defect detection. These methods have high detection efficiency and a good detection effect for obvious defects but a poor detection effect for subtle defects [16,17]. E. Essa, et al. presented a defection detection system based on cognitive support using the minimum ratio between neighborhood regions. However, this method was not tested on many kinds of fabric defects and could not give qualitative conclusions [18]. Transform domain detection methods convert an image from a spatial domain to a frequency domain for detection, which has a good detection effect on periodic, simple-texture fabric images, but the detection effect on complex-texture fabric images is general, which is not suitable for multiscale and multiresolution cloth detection [19,20]. Deep-learning detection methods proposed in recent years have high detection accuracy, but the determination of a detection model and parameters requires the provision of a large number of training samples. The hardware cost is high, and the implementation process is complex, which cannot meet the application requirements of embedded platforms [21,22].

In order to further improve the accuracy of fabric defect detection and realize the accurate segmentation of defect contours, this paper combines mathematical statistics with visual saliency detection and proposes a fabric defect detection method based on saliency region and similarity location. Firstly, the image to be detected is processed by color space conversion, Gaussian filtering, and contrast enhancement, and a frequency-tuned (FT) salient region detection algorithm is used to estimate a saliency map of the enhanced image. The fabric image is divided into image blocks of the same size with overlapping areas through a sliding window, and then the statistical parameters of each image block are calculated. The outliers in the statistical parameters are filtered out using inter-quartile range (IQR). At this time, the initial positioning of defects in the saliency map is realized. Finally, binarization is performed on the initially located image block, and morphological operations are used to eliminate outliers to obtain complete defect contours. The defect detection method for fabric images proposed in this paper is mainly divided into three steps, and a flow chart is shown in Figure 1.

1. Image preprocessing mainly includes image enhancement and FT significance estimation;

2. Combination of statistical parameters, such as mean, variance, and IQR algorithm, to detect an image sub-block where a defect is located;

3. Performing image postprocessing, such as binarization and morphological operation on the image sub-block detected in step 2, and counting the number and area of defects.

2. Materials and Methods

2.1. Image Preprocessing

Considering the problems of noise and blur in the original fabric image, we filtered and enhanced the defect before locating to reduce the noise interference and improve the contrast between the defect and the background. The preprocessing in this method included four parts: color space conversion, Gaussian filtering, saliency map generation, and contrast stretching. Figure 2 is the original fabric image.

• Firstly, we convert the input color image from an RGB model to a lab space model according to Equations (1)–(5). The variable $I_i$ is used to represent the input color fabric image. R, g, and b are the red, green, and blue components of the input image $I_i$, and their range is $\left[0,225\right]$. R, G, and B represent the normalized values of r, g, and b, and their range is $\left[0,1\right]$. The variable $I_l$ represents the image of the converted lab color space model. L, a, and b represent the three channels of $I_l$.

  $$\{\begin{array}{l}R=gamma\left(\frac{r}{255}\right)\\ G=gamma\left(\frac{g}{255}\right)\\ B=gamma\left(\frac{b}{255}\right)\end{array}$$

  (1)

  $$gamma\left(x\right)=\{\begin{array}{ll}{\left(\frac{x+0.055}{1.055}\right)}^{2.4}& if x>0.04045\\ \frac{x}{12.92}& otherwise\end{array}$$

  (2)

  $$\left[\begin{array}{l}X\\ Y\\ Z\end{array}\right]=M\times \left[\begin{array}{l}R\\ G\\ B\end{array}\right],M=\left[\begin{array}{ccc}0.4124& 0.3576& 0.1805\\ 0.2126& 0.7152& 0.0722\\ 0.0193& 0.1192& 0.9502\end{array}\right]$$

  (3)

  $$\{\begin{array}{c}L=116\times f\left(\frac{Y}{Y_n}\right)-16\\ a=500\times \left[f\left(\frac{X}{X_n}\right)-f\left(\frac{Y}{Y_n}\right)\right]\\ b=200\times \left[f\left(\frac{Y}{Y_n}\right)-f\left(\frac{Z}{Z_n}\right)\right]\end{array}$$

  (4)

  $$f\left(t\right)=\{\begin{array}{ll}{t}^{\frac{1}{3}}& ift>{\left(\frac{29}{6}\right)}^3\\ \frac{1}{3}\times {\left(\frac{29}{6}\right)}^2\times t+\frac{4}{29}& otherwise\end{array}$$

  (5)
• Equation (2) is used to perform nonlinear tone editing on the image in order to improve the image contrast. The default values of variables $X_n,Y_n$, and $Z_n$ are 95.047, 100.0, and 108.883, respectively. Figure 3 shows the results of color conversion space.
• The method proposed in this paper selects a Gaussian filter operator $G\left(x,y\right)$ to filter $I_l$. The size of the filter operator is 3 × 3, $\mathsf{\sigma }=1$. $I_g$ represents the filtered image, and $\star$ in Equation (6) represents the convolution operation, and the value range of $x$ and $y$ in Equation (7) is $\left[-1,0,1\right]$. Figure 4 shows the filtering results.
• In order to weaken the influence of periodic texture on defect detection, this paper uses an FT saliency detection algorithm [23] to estimate a saliency map corresponding to $I_l$. We use $I_s$ to represent this saliency map. $I_s$ can significantly improve the gray value of the defect and further improve the contrast between the defect and the background. Its calculation method is shown in Equation (8), where ${\overline{I}}_{li}$ represents the mean value of the ith component in $I_l$, and $I_{gi}$ represents the ith component in $I_g$.
• Considering that the pixel gray value of $I_s$ exceeds the value range of $\left[0,255\right]$, we need to compress the gray value of $I_s$ to obtain $I_n,n=1,2,3$. The calculation process is as follows: First, calculate the maximum gray value of all pixels in $I_s$ as $\mathrm{maxValue}$ and the minimum gray value as $\mathrm{minValue}$, and then stretch the contrast according to Equation (9). The final saliency map is shown in Figure 5.

  $$I_g=I_l\star G,$$

  (6)

  $$G\left(x,y\right)=\frac{1}{2\pi {\sigma }^2}{e}^{-\frac{x^2+y^2}{2{\sigma }^2}},$$

  (7)

  $$I_s\left(i,j\right)={\displaystyle \sum }_{n=1}^3{\left({\overline{I}}_{ln}-I_{gn}\left(i,j\right)\right)}^2,$$

  (8)

  $$I_n\left(i,j\right)=\frac{255}{\mathrm{maxValue}-\mathrm{minValue}}\left(I_s\left(i,j\right)-\mathrm{minvalue}\right),$$

  (9)

2.2. Defect Location Method in Fabric Image

In order to reduce the defect detection range, the method proposed in this paper preliminarily located the image sub-block where the defect was located through (1)−(2) of the following steps and realized the final location of the defect through step (3) and step (4).

(1)

The variance and mean values of all image sub-blocks are detected using the sliding window method. In this paper, the size of the sliding window is 24, and the moving step size is 12. We determine the size of the sliding window based on the experience of many experiments. In practice, it can be determined experimentally according to the resolution of the fabric image collected.

(2)

Screen out all outliers in the mean and variance values in step (1). Outliers specifically mean that the data values in the sample are abnormally large or small, and their distributions deviate significantly from the rest of the observed values. In this paper, the IQR (inter-quartile range) method of statistics is used to analyze outliers. Interquartile range (IQR) is a method in mathematical statistics to determine the difference between the third quartile and the first quartile. Quartile difference is a kind of robust statistics. Figure 6 is the schematic diagram of the IQR method, and Equations (10) and (11) detail the calculation process of the IQR method.

$$\mathrm{IQR}=\mathrm{Q}3-\mathrm{Q}1,$$

(10)

$$\mathrm{outliers}=\mathrm{value}<(\mathrm{Q}1-2.5\times \mathrm{IQR})\mathrm{or}\mathrm{value}>(\mathrm{Q}3+2.5\times \mathrm{IQR}),$$

(11)

After sorting all samples from small to large, Q3 is the point at 75%, Q1 is the point at 25%, IQR is the difference between 75% and 25%, and outliers are the abnormal points. The threshold of Equation (11) is defined according to the principle of normal distribution. According to normal distribution, about 99.85% of the data are within the four standard deviations of the average (<4$\sigma$), and the remaining 0.15% of the data are outside the four standard deviations of the average (>4$\sigma$). The distance from Q1 to the average value is −0.675$\sigma$, and the distance from Q2 to the average value is $0.675\sigma$. Substituting it into Equation (11) can obtain the following: Q1 − 2.5 × IQR = −4.05σ. and Q3 + 2.5 × IQR = $4.05\sigma$. Through many experiments, it is found that, when the threshold in Equation (11) is set to [Q1 − 2.5 × IQR, Q1 + 2.5 × IQR], the application scenario in the manuscript has a better experimental effect.

We use a fixed-size window to slide over the whole image according to a fixed step size and calculate the mean and variance of each window sub-block at the same time. We sort the mean and variance of all sub-blocks and then determine the sub-blocks whose mean and variance values are outliers using the IQR method. At this time, we think that this is the sub-block containing defects that we are looking for, as shown in Figure 7. We use $\mathrm{maskImg}$ to represent the mask image of the corresponding sub-block, as shown in Figure 8. This step is used for rough defect location.

(3)

The adaptive threshold binarization operation is performed on the sub-block where the defect is located. The results obtained are represented by $\mathrm{binaryImg}$, as shown in Figure 9.

(4)

By searching the outline in $\mathrm{binaryImg}$ and combining $\mathrm{maskImg}$, we finally locate the final position of the defect, as shown in Figure 10. In this paper, some small noise points are filtered out according to the contour area of the target.

Table 1. Method 1: location of defects.

1.	The enhancement result obtained in Section 2.1 is represented with enhanceImg. Its size is M × N. M is the height, N is the width, and the size of the sliding window is 24, i.e., size = 24. The result of the preliminary location of defects is represented with locationImg. The result of binarization is represented with binaryImg. Mask results are represented with maskImg.
2. 3. 4. 5. 6. 7.   8. 9. 10. 11. 12. 13. 14. 15. 16. 17. 18. 19. 20. 21. 22. 23. 24. 25. 26. 27. 28. 29. 30. 31. 32. 33. 34. 35. 36. 37. 38. 39. 40. 41.	For i = 0 : M For j = 0 : N subImg = enhanceImg(j, i, min(N − j , size), min(M − i, size)) Calculate the mean value of subImg, represented by m    Calculate the variance of subImg, represented by sd    Use a structure variable named subInfo to save the i, j, m, and sd, and then save it    with a list named infoList    infoList.push_back(subInfo)    Save m with a list named meanList, and save sd with a list named sdList    meanList.push_back(m)    sdList.push_back(sd)    j = j + size × 0.5 End i = i + size × 0.5 End Next, the outliers are obtained using the IQR method. Sort meanList and sdList from small to large length = meanList.size meanQ1 = meanList [0.25 × length] meanQ3 = meanList [0.75 × length] meanIQR = meanQ3 − meanQ1 meanUpper = meanQ1 − 2.5 × meanIQR meanDown = meanQ3 + 2.5 × meanIQR sdQ1 = sdList [0.25 × length] sdQ3 = sdList [0.75 × length] sdIQR = sdQ3 − sdQ1 sdUpper = sdQ1 − 2.5 × sdIQR sdDown = sdQ3 + 2.5 × sdIQR The defects are initially located according to the outliers of mean and variance For t = 0 : length mTmp = infoList[t].mean mSd = infoList[t].sd If mTmp > meanDown and mSd > sdDown Obtain the position of the current sub-image and save it to Rect r r.x = infoList[t].i,  r.y = infoList[t].j    r.width = min(N, width − r.x),    r.height = min(M, height − r.y)    Obtain the mask of the sub-image where the defect is located, maskImg(r) = 255    Copy this sub-image into locationImg, enhanceImg(r).copyto(locationImg(r))    Adaptive threshold binarization for this sub-image    threshValue = min(40,infoList[t].mean × 10)    Binarize the sub image with threshValue as the threshold and assign it to binaryImg (r)  End
42.	End

is the Pseudo code of the algorithm to locate defects.

3. Results and Discussion

In order to verify the effectiveness of the proposed method, it was compared with the following three methods for direct visual and parametric performance. (1) A method (AC) for determining saliency maps based on selecting the mean value in different size neighborhoods [24]. This algorithm used a similar image pyramid algorithm to evaluate difference and accumulation from the standard image in different dimensions, and then used normalization to obtain the final saliency by adding the saliency of multiscale fuzzy images. (2) A method (HC) of constructing saliency maps based on reducing the number of colors in the lab color space [25]. This method proposed a scheme to reduce the number of colors. By mapping each component of RGB into 12 equal parts, the implicit image could only have 12 × 12 × 12 colors at most, so a smaller histogram could be constructed to accelerate. (3) A method (LC) for constructing saliency maps by calculating the global contrast of pixels in the entire image [26]. This method computed spatial saliency maps using the color statistics of images. The algorithm was designed with a linear computational complexity with respect to the number of image pixels. A saliency map of the image was built upon the color contrast between image pixels. We implemented a comparison of the methods mentioned in

Table 2. Comparison of parameters of fabric image detection results using different methods.

Detection Method	${\mathit{T}}_{\mathit{P}}$	${\mathit{T}}_{\mathit{N}}$	${\mathit{F}}_{\mathit{P}}$	${\mathit{F}}_{\mathit{N}}$	Acc	Precision	Recall	F1	T (s)
AC	77	64	23	36	0.705	0.77	0.681	0.723	0.078
HC	65	68	35	32	0.665	0.65	0.670	0.659	0.061
LC	69	71	31	29	0.7	0.69	0.704	0.696	0.053
Our method	97	98	3	2	0.975	0.97	0.979	0.974	0.067

. In the experiment, 200 fabric images with different periodic textures were selected as the test set, of which 100 were fabric images without defects and 100 were fabric images with defects, including kinks, different fibers, yarn skipping, holes, oil stains, wool balls, and other defect types. The image size was 250 × 250 pixels. The fabric image data in this manuscript were from the public data set of the Alibaba Cloud Tianchi Dataset. The experimental environment was a computer with a 2.11 Ghz Core i5-10210u processor, and the programming software was Visual Studio 2015 and OpenCV. In the experiment, the fabric images were divided into two categories according to whether there were defects. In the experiment, the accuracy (Acc), precision, recall, and ${\mathrm{T}}_1$ shown in Equations (12)–(15), as well as the average detection time, were selected for comparison. The closer the Acc value to 1 and the smaller the $\mathrm{T}$ value, the better the detection performance of the method.

$$\mathrm{Acc}=\frac{T_P+T_N}{T_P+T_N+F_P+F_N},$$

(12)

$$\mathrm{Precision}=\frac{T_P}{T_P+F_P},$$

(13)

$$\mathrm{Recall}=\frac{T_P}{T_P+F_N},$$

(14)

$$\mathrm{F}1=2\times \frac{\mathrm{Precision}\times \mathrm{Recall}}{\mathrm{Precision}+\mathrm{Recall}},$$

(15)

where $T_P$ is the number of samples with defects correctly detected, $T_N$ is the number of flawless samples correctly detected, $F_P$ is the number of samples with defects detected incorrectly, and $F_N$ is the number of flawless samples for error detection. Each method tested the fabric images in the test set. The experimental results are shown in Figure 11, Figure 12, Figure 13, Figure 14, Figure 15, Figure 16, Figure 17, Figure 18 and Figure 19 and Table 2.

It can be seen from Table 2 that the method proposed in this paper was superior to the other three comparison methods, as seen in the Acc, precision, recall, and F₁ score values characterizing the detection effect. The three comparison methods were prone to the false detection of fabric images without defects. Due to the high false detection rates of the three comparison methods, their detection accuracy values were reduced. However, the method proposed in this paper had high detection sensitivity, so the detection accuracy was significantly improved compared with the three comparison methods. The obvious improvement in the detection effect of the method proposed in this paper benefitted from the combination of mathematical statistics and significance region estimation, which not only improved the detection rate of various defects but also significantly improved the influence of texture on defect detection and reduced the occurrence of false detections. The detection speeds of the HC method and LC method were faster, but the detection effect was lower. The detection effect of the method proposed in this paper was the best, but the detection speed was not dominant, only ranking third. The method proposed in this paper calculated the statistical parameters of each overlapping image block through a sliding window when the defect was initially located. This comparison strategy not only effectively improved the detection effect but also significantly increased the amount of computation, resulting in a long average detection time. Compared with other research, the algorithm proposed in this paper was more sensitive to relatively small defects, and the contours of detected defects were also more accurate. The algorithm proposed in this paper had a better performance for small defects in fabric images.

Based on the above experimental results, it can be seen that the AC, HC, and LC methods could be used to detect a variety of defect types in the test set, and the reasons for their low accuracy mainly included the following two aspects.

(1)

From the direct visual comparison in Figure 11, it can be seen that the three comparison methods had good detection effects on the relatively obvious defect areas in the fabric image, but the detection effects on the complex-texture fabric image or the fabric image with no obvious defects were general, so its sensitivity was slightly lower than that of the method proposed in this paper.

(2)

The test set selected in this paper contained 100 fabric images with defects and 100 fabric images without defects. It can be seen from the parameter performance comparison in Table 2 that the three comparison methods easily misjudged the fabric images without defects as fabric images with defects, so the detection accuracy was reduced.

The method proposed in this paper combined mathematical statistics with saliency region estimation, calculated the statistical parameters of each overlapping image block through a sliding window, and combined statistical IQR to realize the preliminary positioning of defects in the saliency maps of different periodic textured fabric images, thus significantly improving the adverse impact of fabric texture on defect detection. Therefore, its accuracy was significantly improved compared with the three comparison methods. To sum up, the method proposed in this paper could detect defects in more types of periodic textured fabric images and could accurately extract the complete contours of defects. Compared with the other three comparison methods, the detection accuracy was higher, and the extracted defect contours were more complete.

4. Conclusions

This paper combined mathematical statistics with visual saliency detection and proposed a fabric defect detection method based on saliency region detection and similarity location. Firstly, a saliency map was calculated, and then the statistical parameters, such as the mean and variance, of each image block with the same size and overlapping each other in the saliency map were used to locate a block where a defect was located. This method effectively improved the adverse effect of background texture on detection effect and could effectively locate the defects in simple-texture fabric images and complex-texture fabric images. In addition, adaptive threshold binarization was used to process the coarse positioning image blocks, and a more precise extraction of defect contours was realized. There are many kinds of fabric defects, with about 92 kinds in total. Most algorithms are good for obvious defects, but the algorithm proposed in this paper was more sensitive to relatively small defects, and the contours of detected defects were also more accurate. The algorithm proposed in this paper had a better performance for small defects in fabric images. The defect detection effect obtained using this method was compared with defect detection effect maps generated by other detection methods. Through the detection results, it was concluded that the defect detection effect map generated by this method was the best. The experimental results showed the effectiveness of the proposed algorithm. We think the method proposed in this paper could be extended to other visual inspection fields, such as stone defects and metal defects. The experiments showed that the method proposed in this paper was effective for images with periodic textures. In the future, we plan to combine deep-learning models and traditional methods to improve the accuracy and robustness of fabric defect detection.