Identification and Classification of Defects in PE Gas Pipelines Based on VGG16

Abstract

For the problem of classification and identification of defects in polyethylene (PE) gas pipelines, this paper firstly performs preliminary screening of the acquired images and acquisition efficiency of defective image acquisition was improved. Images of defective PE gas pipelines were pre-processed. Then, edge detection of the defective images was performed using the improved Sobel algorithm and an adaptive threshold segmentation method was applied to segment the defects in the pipeline images. Finally, the defect images were morphologically processed to obtain binary images. The obtained binary images were applied with VGG16 to complete the training of the defect classifier. The experimental findings show that in the TensorFlow API environment, the test set’s highest accuracy reached 97%, which can achieve the identification of defect types of underground PE gas transmission pipelines.

1. Introduction

In the past decades, polyethylene (PE) pipelines have been widely used in natural gas networks around the world because of their flexibility and corrosion resistance [1]. Therefore, the long-term performance of PE pipes and their materials is of great concern to date [2]. According to international natural gas pipeline accident statistics, natural gas pipeline defects are frequently caused by localized corrosion [3], operator mistakes, defective materials, and construction flaws [4]. As the use of pipelines for transporting hazardous substances becomes more popular worldwide, the possibility of major accidents caused by pipeline failures is gradually increasing [5]. For example, the explosion caused by a gas pipeline leak in a residential building in Slovakia in 2019, which killed at least seven people, reminds us that gas pipelines must be checked regularly [6]. There are many nondestructive inspection methods for gas pipelines in practice (e.g., ultrasonic-based sensors, laser-based systems, etc.), and compared to other inspection techniques commonly used for PE pipelines, due to their distinctive benefits of intuitiveness, accuracy and convenience, visual inspection techniques have been used extensively in a variety of fields [7]. Pipeline defect detection robots equipped with intra-visual inspection of image processing technology can directly collect, transmit and process images, reducing labor costs [8].

Traditionally, automatic classification of images is carried out using extracted image features, which are used to represent unclear information in the original pixel values. Convolutional neural networks (CNN) have taken the place of that approach in recent years [9]. Image pre-processing, which includes image de-noising [10], image enhancement [11], image segmentation [12], morphological operations [13], etc., is generally performed before image classification. Based on image pre-processing, Zhou et al. [14] investigated an improved spline local mean decomposition (ISLMD), proposed to be CNN-based and enabling noise reduction of images to locate pipe leakage locations. Ma et al. [15] proposed a sewer multi-defect detection system based on CNN-style GAN-SDM image pre-processing, and the proposed model’s average accuracy and macro F1 score were 95.64% and 0.955, respectively. Hosseinzadeh et al. [16] presented a small and simple probe design that was used to check small-bore pipes for defects. Hua et al. [17] developed a visual recognition-based pipeline fault detection algorithm. It is capable of both autonomous localization and pipeline fault detection.

Several frameworks based on the original CNN structure have been proposed to enhance target detection performance as a result of the advancement of CNN technology and classification, such as R-CNN [18], Fast R-CNN [19], SSD [20], YOLO series [21], etc. In this paper, a CNN-based classification framework for PE pipe defect detection is proposed, which can automatically extract the abstract features of defects for accurate classification of three defects, including cracks, fractures, and holes. In this paper, three different algorithms are applied to the existing framework and the confusion matrix is used to determine which model framework has the highest accuracy. The experimental results indicate that the highest accuracy of the test set in this paper reached 97% in the environment of TensorFlow API.

2. Image Pre-Processing

The pre-processing of pipe images does more than remove noise; it also enhances the contrast between the pipeline’s background and any pipeline defects, making it simpler to locate and categorize pipeline defects [22]. Figure 1 depicts the image pre-processing process used in this paper.

2.1. Greyscale Processing for Pipeline Images

Grayscale is an important feature to characterize the brightness and darkness of an image. In recent years, on the basis of image grayscale differences and discontinuous changes, it has been used in target recognition, image segmentation, and machine vision technique [23]. Grayscale occupies less memory and enables faster computer operations compared to color images. The mean value method, maximum value method, and weighted average method are the three commonly used techniques for transforming color photos into grayscale.

Where the maximum value method is to directly take the value of the component with the largest value among the three components of a R,G,B Equation (1). Red (R), green (G) and blue (B) are the three color channels of color images.

$$R=G=B=\max \left(R,G,B\right)$$

(1)

The mean method is to take the mean of the values in the three components of R,G,B Equation (2).

$$R=G=B=\left(R+G+B\right)/3$$

(2)

The weighted average method is based on the sensitivity of the human eye for the R,G,B’s three colors, according to a certain weighted average, obtained in Equation (3).

$$I\left(u,v\right)=0.3\times I_R\left(u,v\right)+0.59\times I_G\left(u,v\right)+0.11\times I_B\left(u,v\right)$$

(3)

where: I(u,v) denotes the gray value at coordinate, I_R(u,v), I_B(u,v) and I_G(u,v) denote the luminance value of the pixel’s three color components, respectively.

The maximum value method (Figure 2B1–B3), the average value method (Figure 2C1–C3), and the weighted average method (Figure 2D1–D3) were used to grayscale process the three original defect images (Figure 2A1–A3) of cracks, fractures and holes, respectively. From Figure 2, it can be seen that the weighted average method produces the best results for the grayscale image, and the grayscale image has moderate brightness and does not cover the characteristics of the pipe defects. As a result, the image grayscale uses the weighted averaging method.

2.2. Defect Image Acquisition

We compared and examined the continuously captured PE pipeline images and discovered that: 1. the percentage of pipeline defects in the entire PE gas pipeline network system is small; 2. there is a significant grayscale discrepancy between the defective and normal parts of the grayscale processed defective images; 3. after grayscale processing of any two adjacent PE pipeline images, there is a significant grayscale discrepancy between the normal PE pipeline images and the defective PE pipeline images in the same position, while the grayscale discrepancy between the two normal PE pipeline images is very small. Thus, in order to determine whether there are defects in the pipeline images, we propose a screening method for pipeline defects to improve the detection effectiveness [24].

First, let x and y be consecutive images of any adjacent PE pipes in our pipe database, and Δ be the greyscale discrepancy between x_k and y_k, which is the correspondent pixel locations of the two images x and y, as shown in Equation (4). However, there will be some information loss during the compression and transmission of the image data [25], which will make the corresponding pixel grayscale values of the two adjacent images differ greatly, even if they are both normal, resulting in incorrect judgments of the system. To decrease this error, we improve the grayscale discrepancy Δ of two corresponding pixels to the grayscale discrepancy of the corresponding region $\overline{\Delta }$, which is the mean of the grayscale discrepancy of all pixels in the designated size region, as shown in Equation (5).

$$\Delta =\left|x_k-y_k\right|$$

(4)

$$\overline{\Delta }=\frac{1}{S_n}{\displaystyle \sum }_{k=1}^n\left|x_k-y_k\right|$$

(5)

where $S_n$ denotes number of pixels in designated size region.

Then, for the minimum mean ${\overline{\Delta }}_1$ of the pixel grayscale discrepancy between the defective pipeline images and the normal pipeline images computed in the designated size region, a statistical method can be used; the number of pixels with the smallest defect area is calculated statistically as M, let the length of the defect area image be M_l pixels, the width be M_w pixels, as shown in Equation (6).

$$\mathrm{M}=M_w\times M_l$$

(6)

Finally, to further improve the accuracy of defect detection, we set the discrepancy ${\overline{\Delta }}_{min}$ to be somewhat less than ${\overline{\Delta }}_1$. When $\overline{\Delta }\ge {\Delta }_{min}$, it is judged as abnormal and the number of pixels in the abnormal area is counted as B, when $\overline{\Delta }<{\overline{\Delta }}_{min}$, it is judged as normal and the number of pixels in the normal area is counted as B1; calculate B1 and B according to Equation (6). If B ≥ M, then there are defects in x and y, otherwise x and y are considered as normal images.

More than 9000 consecutive images of pipes were screened using the above defect screening method (Figure 3). A total of 160 out of 163 defective images were picked out, and the screening accuracy rate was up to 98.15%. Figure 3 can better help us understand the PE pipeline defect detection algorithm.

2.3. Images Enhancement for Pipe Defects

Image enhancement is the process of enhancing an image’s display by highlighting its edges and significant texture details and suppressing the display of unimportant areas. This somewhat enhances the image’s visual impact [26] or highlights some “useful” and compresses other “useless” information in the image. In this paper, global histogram equalization (Figure 4B1–B3), adaptive histogram equalization (Figure 4C1–C3), and gamma transform (Figure 4D1–D3) was applied to enhance the image of the grayscale (Figure 4A1–A3).

Figure 4 illustrates that after the gamma transform, the defect image is not distorted; additionally, the defect’s edges become more noticeable and stand out against the background with a greater difference. In order to increase the contrast between the background of the pipe and the pipe defects, gamma transform was applied.

2.4. Pipe Defect Images Filtering and Denoising

During image transmission, the final image is often received with a lot of noise due to the interference of the environment or the limitations of the equipment. Image denoising is a classical image recovery task aiming to predict clean images from noisy observations [27]. Bilateral filtering (Figure 5B1–B3), Gaussian filtering (Figure 5C1–C3), mean filtering (Figure 5D1–D3), and adaptive median filtering (Figure 5E1–E3) were applied in this paper to denoise the images obtained above (Figure 5A1–A3).

Step-by-step processing is a very common tactic for resolving complicated noisy images [28]. One of the weighted averages used for bilateral filtering is based on Gaussian distribution, which removes Gaussian noise from the image. However, the removal of Gaussian noise will ignore salt-and-pepper noise, and adaptive median filtering is the best algorithm to remove salt-and-pepper noise; however, it has poor results when removing Gaussian noise [24], Therefore, it was proposed to use dual filtering (Figure 5F1–F3) to remove the Gaussian noise by bilateral filtering after removing the salt-and-pepper noise by median filtering. Compared to other filtering algorithms, the dual filtering preserves the details of the edges and eliminates the noise points to achieve the effect of keeping the edges (as shown in Figure 5). Therefore, dual filtering was used for noise reduction in this paper.

3. Image Edge Detection and Segmentation for Pipe Defects

Image edges are the most basic feature of an image and using this feature the image can be segmented. In many imaging applications, it is sufficient to detect the periphery of an unknown object [29]. Image segmentation is usually performed before image feature quantization [30]. Threshold segmentation is a pixel-division technique used in region-based image segmentation, according to gray levels, into regions that have consistent properties, while neighboring regions do not have such consistent properties.

3.1. Improved Edge Detection with Sobel Operator

The conventional Sobel operator first performs a weighted average process for each pixel using a convolutional template (shown in Figure 6) and then acquires the gradient values in the X and Y directions by performing a difference process. It is challenging for the algorithm to achieve the desired detection results and the localization accuracy is not satisfactory, because the prevalent Sobel algorithm is just sensitive to the X directions and Y directions and can only assess the edges in both X directions and Y directions [24]. Various shapes and depths of the PE gas pipeline defects lead to negligible regional variations in the grayscale of the deficiency images; the collected images of PE gas pipeline defects contain a lot of interference data because of compression and real-time transmission. It is ineffective and very likely to result in missing edges, relying solely on two directional templates to identify the edges of pipe defects. In order to detect edge pixels in images more accurately, the Sobel algorithm was improved to eight directions [31] (as in Figure 7), which not only detects image edges more effectively but also increases edge detection accuracy and lowers the likelihood of incorrect edges.

For the pipe defect filtered image obtained above (Figure 8A1–A3), the improved Sobel algorithm (Figure 8B1–B3), the Sobel edge detection algorithm (Figure 8C1–C3) and the Prewitt algorithm (Figure 8D1–D3) are used for edge detection of the image, respectively. Figure 8 displays the outcomes of the three defects’ edge detection. According to the comparison study, the improved Sobel algorithm extracts defect edges with more continuity and integrity and can completely display the defect shape characteristics. Therefore, the improved Sobel algorithm was used for edge detection in this paper.

3.2. Adaptive Threshold Segmentation

In the case of inhomogeneous illumination or uneven distribution of gray values, the segmentation results obtained if global threshold is used are often unsatisfactory, and adaptive threshold (also called local segmentation) can produce good results [32]. Adaptive threshold segmentation does not use one threshold for the whole matrix as a global threshold, but has a corresponding threshold for each value at each position of the input matrix.

The images (Figure 9A1–A3) present a comparison after the above edge detection was processed by adaptive threshold (Figure 9B1–B3), global threshold (Figure 9C1–C3) and Otsu threshold segmentation (Figure 9D1–D3), respectively. According to visual observation, an adaptive threshold can be used to distinguish pipeline defects from the pipe background, with the best segmentation of PE gas pipeline defects, with complete edge segmentation and less disturbing information. Therefore, the adaptive threshold algorithm was used to segment the image in this paper.

3.3. Morphological Operation

The shape of features in an image is typically the focus of morphological operation. It has been applied to eliminate defects in a variety of shapes with the purpose of smoothing the contours and preserving the object’s size and shape [33]. Erosion and dilation are the two main operations.

Let f(m,n) be the input image and g(m,n) be a structure element, If the set of real integers is denoted by Z, while assuming that (m,n) is an integer f(m,n) from Z*Z, g(m,n) is a function of a pixel’s gray value with the given coordinates (m,n) and the gray value is also an integer. Namely, g(m,n) to f(m,n) for grayscale dilation can be defined as $f\oplus g$, which is shown as Equation (7).

$$\left(f\oplus g\right)\left(s,t\right)=max\left\{f\left(s-m,t-n\right)+g\left(m,n\right)|\left(s-m\right),\left(t-n\right)\in D_f;\left(m,n\right)\in D_g\right\}$$

(7)

Equation (7) D_f, D_g is the definition domain of f(m,n) and g(m,n), respectively, g(m,n) is the structural element of the morphological treatment is also a function, the displacement parameters (s − m), (t − n) must be in the definition domain of the function f(m,n).

The dilation operation is to find the local maximum value, and the anchor point is assigned the maximum of pixels in the nucleus coverage area. The erosion operation is to find the local minimum value. The minimum value of the pixel in the kernel coverage area is assigned to the anchor point. Erosion of grayscale image is defined as Equation (8).

$$\left(f⊝g\right)\left(s,t\right)=min\left\{f\left(s+m,t+n\right)-g\left(m,n\right)|\left(s+m\right),\left(t+n\right)\in D_f;\left(m,n\right)\in D_g\right\}$$

(8)

Equation (8) D_f, D_g is the definition domain of f(m,n) and g(m,n), respectively, g(m,n) is the structural element of the morphological treatment is also a function, the displacement parameters (s + m), (t + n) must be in the definition domain of the function f(m,n).

The expressions for the opening operation and closing operation of the grayscale image have the same form as the erosion and dilation, and the structural element g(m,n) performing the opening operation on the image f(m,n) can be defined $f\circ g$, in Equation (9).

$$f\circ g=\left(f⊝g\right)\oplus g$$

(9)

The opening operation is an erosion operation of g(m,n) on f(m,n) followed by an dilation operation on the result of the erosion. A similar closing operation of g(m,n) on f(m,n) can be defined $f\cdot g$ in Equation (10).

$$f\cdot g=\left(f\oplus g\right)\circ g$$

(10)

In addition to the above operations, there is the morphological gradient operation; let the gradient be denoted by h in Equation (11).

$$h=\left(f\oplus g\right)-\left(f⊝g\right)$$

(11)

The basic morphological operations: closed operation (Figure 10B1–B3), dilation (Figure 10C1–C3), erosion (Figure 10D1–D3), morphological gradient (Figure 10E1–E3), and opening operation (as in Figure 10F1–F3) are applied to the threshold segmentation image obtained above (Figure 10A1–A3) to compare the results. As shown in Figure 10, The opening operation’s result is to remove the image area that is slightly relative to the structure element and keeping the image area that is larger than the structure element [34]. The open operation more fully preserves the overall impact, and in this paper, we use the opening operation to fill the contours of defects in the images.

4. CNN-Based Image Defect Classification

The CNN’s primary characteristic is that its front-end input obtains image information by using multiple layers of locally interconnected neurons. The CNN can extract view invariant features [35] and takes full account of the translation, rotation, and scaling in space. Deep learning and convolutional networks have greatly improved the capability of target detection and classification using images. In this paper, three classification models, VGG16, Resnet50 and the original CNN model, are used for comparison, and the corresponding network structures are shown in Figure 11.

4.1. Classification Model

VGG16 has 16 layers, including 3 fully-connected layers, 5 pooling layers, and 13 convolutional layers. Resnet50 has 50 layers, including 1 fully-connected layer, 2 pooling layers, and 49 convolutional layers. The original CNN has 6 layers, including 2 fully-connected layers, 2 pooling layers, and 4 convolutional layers. The pooling layer is not counted as a pooling layer when determining the number of layers in the network because it has no parameters.

4.2. Convolutional Layer

The convolutional layer’s function is to extract the data from the source image, also referred to as image features. In the CNN architecture, the convolutional layer is typically the first layer, and the weight (w_ij) controls it, where i and j are the number of input and output feature mappings, respectively [36]. Thus, the entire convolution process can be defined, as in Equation (12).

$$a_j:=\mathsf{ƒ} ({\displaystyle \sum }_{i\in M_j}^{}{a}_j\ast w_{ij}+b_j)$$

(12)

In Equation (12), $\ast$ denotes the 2D convolution operation, a_j is the set of input feature maps, b_j denotes the jth underlying feature map, and $\mathsf{ƒ}$ denotes the activation function. In the CNN, each individual convolution layer performs a linear transformation from its input to the output representation through a multichannel multidimensional convolution operation. The convolution can be represented as a matrix–vector product, where the linear transform matrix is derived from the convolution filter and the vector represents the reshaped input of the layer [37].

4.3. Pooling Layer

The pooling layer’s primary function is to use specific factors to downsample the input feature mapping size, the average pool averages the features in the neighborhood to produce a blurring effect [38]. Spatial invariance can be attained by the pooling layer by lowering the feature map’s resolution. Each feature map that has been combined corresponds to a feature map from the layer before. Their cells combine inputs from a small block of n × n cells. The size of the pooling window is flexible, and the windows may overlap [39]. In this paper, maximum pooling is used.

4.4. Fully Connected Layer

Each feature matrix from the pooling layer is converted by the fully connected layer into a single column feature macrovector of dimension 1 × m [40]. The most common layer in a neural network is the fully connected layer, and each of its nodes is connected to each node in the previous and next layer. The dimensionality may increase, decrease, or remain constant during the transformation process [41]. In a convolutional neural network structure for classifying images, the fully connected layer is typically placed at the very end, and for better fitting the nonlinear problem, three fully connected layers are used in this paper. The VGG16 framework is shown in Figure 12.

4.5. Classification Results

This project was developed using Python and the TensorFlow API, and the graphics card used for training, verification, and testing was an NVIDIA 2080TI, 32GB, DDR4. Each training run is 100 epochs (as shown Figure 13). In this paper, to evaluate the classification effectiveness of the model, the precision and recall of our confusion matrix were applied. Equations (13) and (14) give their definitions, where, respectively, TP, FP, and FN stand for true positives, false positives, and false negatives.

$$Precision=\frac{TP}{TP+FP}$$

(13)

$$Recall=\frac{TP}{TP+FN}$$

(14)

Three types of pipe defect images, 100 of each were randomly selected for each defect as a test set from

Table 1. VGG16 defect classification results.

Defect Type	Picture	True Positive	False Positive	False Negative	Precision	Recall
Cracks	100	99	1	5	99%	95.19%
Holes	100	94	6	2	94%	97.91%
Fractures	100	98	2	2	98%	98%

,

Table 2. Resnet 50 defect classification results.

Defect Type	Picture	True Positive	False Positive	False Negative	Precision	Recall
Crack	100	96	4	7	96%	93.2%
Hole	100	92	8	6	92%	93.87%
Fractures	100	95	5	4	95%	95.96%

and

Table 3. CNN defect classification results.

Defect Type	Picture	True Positive	False Positive	False Negative	Precision	Recall
Crack	100	95	5	13	95%	87.96%
Hole	100	87	13	10	87%	89.69%
Fractures	100	89	11	6	89%	93.68%

, made evident that the VGG16 model’s recognition accuracy has improved significantly, particularly for cracks and fractures in the pipe body, where the recognition accuracy can reach an average of 98.5%, while the accuracy rate of holes is relatively low, but still can reach about 94%. From the confusion matrix (as shown in Figure 14), as can be seen, the classification algorithm in this paper has successfully identified three different types of defects with an average recognition rate of 97%. Compared with 94.96% percent of Xie et al. [36], the accuracy rate of this article has been improved by 2.04%. Compared with the 96.3% percent of Cong et al. [24], the accuracy rate of this article has been improved by 0.7%, which can be applied to the engineering practice of pipe defect detection and classification.

5. Conclusions

This paper proposes a framework for classifying and identifying defects of underground PE gas transmission pipes. Fast and accurate detection of defects in PE pipelines is made possible through experimental verification. The following are the main results of this experiment.

• Based on the similarity of images of continuous pipelines, the images were first grayed out using the weighted average method and then preliminary screening of the acquired images was conducted to identify images with suspected defects, which improved the acquisition efficiency of defect image. For the defective PE gas pipeline images, contrast was enhanced using gamma transform, and finally noise was removed using the dual filtering method. The results show that the pre-processing technique can speed up image processing while retaining much of the detail of the originals.
• In order to detect the edges of image defects, an improved Sobel algorithm was presented. An adaptive thresholding segmentation method was used to segment the defects in the image. Then, after morphologically processing the image to obtain a binary image, the feature parameters of the defects were extracted.
• The obtained binary images were trained using VGG16, and the classifier was completed by selecting appropriate parameters for classification of various defects in images of the PE gas pipes. The experimental test results reveal that the highest accuracy of the classification approach adopted in this paper is 97% and it can be used not only for the identification of defective types of underground PE gas transmission pipes, but also for steel pipes, cast iron pipes, etc.