Noncontact Surface Roughness Estimation Using 2D Complex Wavelet Enhanced ResNet for Intelligent Evaluation of Milled Metal Surface Quality

Abstract

Machined surfaces are rough from a microscopic perspective no matter how finely they are finished. Surface roughness is an important factor to consider during production quality control. Using modern techniques, surface roughness measurements are beneficial for improving machining quality. With optical imaging of machined surfaces as input, a convolutional neural network (CNN) can be utilized as an effective way to characterize hierarchical features without prior knowledge. In this paper, a novel method based on CNN is proposed for making intelligent surface roughness identifications. The technical scheme incorporates there elements: texture skew correction, image filtering, and intelligent neural network learning. Firstly, a texture skew correction algorithm, based on an improved Sobel operator and Hough transform, is applied such that surface texture directions can be adjusted. Secondly, two-dimensional (2D) dual tree complex wavelet transform (DTCWT) is employed to retrieve surface topology information, which is more effective for feature classifications. In addition, residual network (ResNet) is utilized to ensure automatic recognition of the filtered texture features. The proposed method has verified its feasibility as well as its effectiveness in actual surface roughness estimation experiments using the material of spheroidal graphite cast iron 500-7 in an agricultural machinery manufacturing company. Testing results demonstrate the proposed method has achieved high-precision surface roughness estimation.

1. Introduction

Quality, including all attributes of manufactured products, is an essential objective of industrial production. Encompassing the measurable quantities and characteristics of the product, quality is regarded as a summary of the product determining its capability to meet needs, fulfill functions, and be effective [1]. In the development and production of industrial parts, both the macroscopic shape and the microstructure of the surface on an μm-scale strongly influence the properties and performance of machined parts, including friction, wear, precision, fitting, anti-fatigue and anti-corrosion characteristics [2,3]. Controlling of surface roughness is mandatory to define the process and validate the quality of the machined part [4]. Due to the friction between the insert and the workpiece surface during material removing process, some inevitable phenomena, such as plastic deformation during chip separation and high-frequency vibrations of the processing system, emerge and are harmful to machining quality. It is well known that surface roughness is regarded as a microscale geometry error. Therefore, it can also serve as a useful vehicle that represents the operational state of the processing system.

In the past few decades, with the advent of tactile profilometers, the stylus tracing (ST) method has become the most popular way of evaluating the surface quality of components. The principle of ST is based on measuring the surface texture and calculating the roughness parameter of Ra [5]. A schematic structure of a profilometer is shown in Figure 1. As illustrated, a kinematic mechanism drives the motion of a stylus, in which a diamond is fixed at the end. This kinematic unit records the vertical deflection of the stylus that moves over the straight paths on the investigated surface. In the process of measurement, the stylus is initially loaded on the measured surface and the surface height variation is obtained as it moves across the surface at a constant velocity [6]. However, there are some disadvantages to using this type of instrument. Owing to mechanical contact with the measured surface there is a potential risk of soft material damage. That is, the measuring process, as described above, inevitably results in scratches by the diamond stylus. On the other hand, the problem of inefficiency should be seriously considered as it cannot work online [7].

Besides the traditional contact stylus tracing technique, the availability of modern measuring technologies emerging from digital computer enhanced machine vision techniques have enabled us to evaluate surface quality through non-contact means [1,6]. There are several promising methods to measure the roughness of machined surfaces. These methods include laser reflectivity, non-contact laser stylus metrology, scanning electron microscopy, compressed air measuring, focus variation, fringe projection technique, and a confocal laser scanning microscope [8,9]. In particular, the development of a method of measuring surface roughness via machine vision-based techniques has attracted attention. Huaian proposed a method based on the difference of RGB color space to develop a correlation model between the sharpness and the surface roughness [3]. Krehel reported a novel device design for non-contact measuring of the cutting tool as well as the roughness of the machined surfaces using an optical sensor [1]. Zheng presented a new method that extracts the characteristic parameters of average texture cycle for surface texture with a straight shape to establish the relationship between the average texture cycle and the actual surface roughness [10]. Koçer investigated the relationship between surface roughness and image grey [11]. Lee reported a workpiece profile acquisition method using image processing and fast Fourier transform [12,13]. All the above mentioned methods require prior knowledge or equipment transformation for the roughness measurement.

Correcting of the texture skew is also very important for the surface roughness estimation. According to the instructions of ISO 4288, the workpiece shall be properly positioned so that the direction of the section corresponds to the maximum value of the height of the roughness parameters (arithmetic height parameter Ra, 10-point roughness Rz). This direction will be normal to the lay of the surface being measured. Generally, this direction can be assessed by visual examination. Therefore, some measure errors occur. Figure 2 illustrates the influence of the surface roughness measuring direction, where the red plane indicates the idle measure plane of the measured surface and the yellow and green planes indicate the measure planes when the incline is 8° and 4°, respectively. The measuring profiles (Figure 2a–c) show that measuring direction errors lead to frequency changes compared with the idle direction. Figure 2d,e also show the measured profile errors. These errors may result in roughness measuring errors.

There are many different roughness parameters in use, but Ra is by far the most common, though this is often for historical reasons and not for particular merit. National standard (GB/T 1031-2009) also recommends Ra as the preferred roughness parameter. Meanwhile, in the given surface roughness specification of the agricultural machinery manufacturing company, Ra is the only required surface roughness constraint. Therefore, Ra is chosen as the representative of surface roughness in this research.

Hough transform, projection characteristics, Fourier transform and shearlet transform are algorithms commonly used for skew correction [14,15,16,17]. However, these algorithms are mainly proposed for detecting deviations of the document orientation angles from the horizontal direction or the vertical direction. In this research, the investigated subjects are textures of the machined surface. Prior researchers have shown that acquired surface images contain strong noise in a broadband frequency range [18]. Therefore, in order to ensure more meaningful surface roughness estimation results, some proper preprocessing of skew correction is indispensable.

Recently, the neural network has aroused heated discussion in the scientific and industrial communities because it achieves good performance for pattern recognition [19,20,21,22,23,24]. The concept of pattern recognition can be defined as identifying or classifying complex signal samples or objects [25]. Surface roughness evaluation can also be considered a special case of pattern recognition. Although it is finding more and more applications in pattern recognition problems, the neural network of the deeper layers also involves more sophisticated training process. Aiming at handling this problem, residual network (ResNet) has recently been proposed for optimizing the deep network and has achieved huge success [26]. Gong investigated the change of classification accuracy when the acquired images are preprocessed by transformation-based methods [27]. Results indicate that the classification accuracy of the convolutional neural network is substantially affected by global translation, rotation, and scaling. As such, variations in measurement direction influence not only the roughness measurement results but also the classification accuracy.

Inspired by the idea of ResNet, we present a surface roughness evaluation method using complex wavelet-enhanced ResNet. This method is performed on the adjusted images processed by the improved texture skew correction algorithm. In this method, 2D-DTCWT with fixed decomposition depth is performed to retrieve the main signal component within the adjusted images. ResNet is utilized to ensure the automatic classification of the surface roughness. Compared with the traditional stylus tracing method, this novel method can be implemented with improved efficiency for surface roughness evaluation purpose. Moreover, the proposed method is also promising for finding artificial intelligence applications to meet the surface roughness evaluation demands in industrial fields.

The major contributions of this research article are summarized below.

(1)

A texture skew correction method, based on the combination of an improved Sobel operator and Hough transform, is proposed for the surface texture direction adjustment. The results show that the proposed method is able to correct texture skew.

(2)

The paper proposes an intelligent surface roughness evaluation method based on machine-vision-enhanced artificial intelligence technology for 2D images. After 2D-DTCWT filtering of adjusted images, ResNet is employed for the surface texture pattern recognition. Owing to the engagement of ResNet in artificial feature learning, the model does not rely on prior knowledge.

(3)

Surface roughness estimation using the proposed method was performed on the images of milled metal surfaces made from the material spheroidal graphite cast iron 500-7. The high estimation accuracy of surface roughness revealed by the experimental results shows that the proposed method has good generalization ability.

The rest of the paper is organized as follows. The image processing techniques, including skew correction and 2D-DTCWT, are briefly described in Section 2. The intelligent learning method based on ResNet is presented in Section 3. The framework of the proposed method is explained in detail in Section 4. The model training process and the validation experiment with results are shown in Section 5. Functions of the elements within the proposed method are discussed in Section 6. Some concluding remarks are given in Section 7.

2. Image Pre-Processing

Regarding engineering applications, proper image pre-processing is indispensable for achieving high precision in pattern recognition problems when estimating surface roughness. In this section, an improved skew correction algorithm and the fundamentals of 2D-DTCWT are introduced.

2.1. Improved Texture Skew Correction

In this sub-section, on the basis of an improved Sobel operator and Hough transform, a texture skew correction algorithm for input surface images is put forward.

2.1.1. Improved Sobel Operator

Edge detection techniques have attracted much attention in recent decades. Among these algorithms, the Sobel operator is well known for its high computation efficiency. As a special realization of discrete differentiation, the Sobel operator computes the approximation of the gradient of the image intensity function. The Sobel operator consists of two kernels of size 3 × 3. These kernels are convolved with input images to derive approximations of partial derivatives on two orthogonal directions, which are shown as

$$G_x=\left[\begin{array}{ccc}+1& 0& -1\\ +2& 0& -2\\ +1& 0& -1\end{array}\right]\ast I\mathrm{and}{G}_y=\left[\begin{array}{ccc}+1& +2& +1\\ 0& 0& 0\\ -1& -2& -1\end{array}\right]\ast I,$$

(1)

where the operator $\ast$ denotes the 2-dimensional convolution and the variable I is defined as the source image [28]. Because the orthogonality between the two directions, the above partial gradient approximations can be combined to derive the gradient magnitude at a specified pixel, which is shown as

$$G={\left(G_x^2+G_y^2\right)}^{1/2}.$$

(2)

After selecting the sensitivity threshold for the classical Sobel operator, edge features can be detected automatically. However, as can be inferred from the above manipulations, the classical Sobel operator is only sensitive to the horizontal direction and the vertical direction, leading to an incomprehensive detection effect.

By employing additional templates with directional sensitivity at angles of 45°, 135°, 180°, 255°, 270°, and 315°, an improved Sobel operator was developed by Shi [29]. As shown in Figure 3, the additional templates can guarantee more comprehensive and accurate edge information.

During the numerical implementation, the matrix {f(m, n)} represents the image to be detected; v{f(m, n)} represents the mask window of size 3 × 3 (as presented in Figure 3) centering at the pixel (m, n); ω(m, n) represents the output image after edge detection. The flow chart of the improved Sobel operator is described below.

Step 1. Conduct the convolution operation between the templates u{t} (t = 1, 2,…, 8) and v{f(m, n)}. Then store the absolute value of the calculation results in the corresponding array v{t} (t = 1, 2,…, 8).

Step 2. Find the maximal value of all data in the corresponding array v{t}.

(1)

Initialization such that Max = v{0}, t = 1.

(2)

If v{t} > Max, set Max = v{t}

(3)

If t < 8, then t = t + 1 and go to (2). If not, go to Step 3.

Step 3. Assign Max to ω(m, n) as the output image.

As a result, edge curves of the original image are extracted.

2.1.2. Hough Transform

Hough transform is another popular feature extraction technique used in image analysis to derive image inclination angles of specified pixels. As known, surfaces of mechanical rubbing are rough from the microscopic viewpoint. In a zoomed-in area with sufficient resolution, cures of the milled mechanical components texture can be approximated as straight lines. Generally, the mathematical model of a straight line can be parametrized as

$$y=kx+b.$$

(3)

The parameters k, b in Equation (3) are the slope and intercept, respectively.

For straight lines to go through a specified pixel in the original plane, all possible slope intercept pairs associated with this point correspond to a straight line in the parameter space. Considering a group of collinear pixels in the original plane, we can infer that their equivalents in the parameter space intersect with each other at the point (k, b). However, in such a form, the value of the slope of a vertical line would give rise to unbounded values. Alternatively, Duda and Hart [30] proposed an equivalent Hesse normal form:

$$r=x\cos \theta +y\sin \theta ,$$

(4)

where r is the distance from the origin to the closest point on the straight line; $\theta$ is the angle between the x axis and the line connecting the origin with that closest point. In Hesse normal form, a straight line in the x–y plane corresponds to a sinusoidal curve in the r–θ plane. Therefore, the problem of detecting collinear points can be converted to the problem of finding concurrent curves.

After the pre-processing combining edge detection and Hough transform, the main incline angle can be computed. Thus, the image is rotated by the main inclination angle.

2.2. Two-Dimensional Dual Tree Complex Wavelet Transform (2D-DTCWT)

In the literature, DTCWT is reported to enjoy a higher degree of designing freedom because it is an overcomplete frame expansion method [31]. DTCWT is usually utilized for enhancing multi-scale decompositions for the raw data acquired.

2.2.1. Framework of DTCWT

Wavelet transform has been exploited with great success across many applications in both the scientific and engineering fields. In the theory of wavelet transform, a continuous signal $x(t)$ of finite energy can be decomposed in terms of wavelets and scaling functions via

$$x(t)={\displaystyle \sum _{n=-\infty }^{\infty }c(n)\varphi (t-n)}+{\displaystyle \sum _{j=0}^{\infty }{\displaystyle \sum _{n=-\infty }^{\infty }d(j,n)2^{j/2}}\psi (2^{j}t-n)},$$

(5)

where $\varphi (t)$ is the scaling function and $\psi (t)$ is the wavelet function. The scaling coefficient series $c(n)$ and wavelet coefficient series $d_j(n)$ are computed via the inner products

$$c(n)={\displaystyle {\int }_{-\infty }^{\infty }x(t)\varphi (t-n)dt}$$

(6)

$$d_j(n)=2^{j/2}{\displaystyle {\int }_{-\infty }^{\infty }x(t)\psi (2^{j}t-n)dt}.$$

(7)

Discrete wavelet transform (DWT) is the fast implementation of the continuous wavelet transform. Although DWT has many advantages, there are still some fundamental problems such as oscillations, shift variance, aliasing, and lack of directionality. Inspirited by Fourier transform, complex wavelet transform (CWT) is proposed with a complex-valued scaling function and complex-valued wavelet function, shown as

$${\psi }_{ℂ}(t)={\psi }^{\Re e}(t)+j\cdot {\psi }^{\Im m}(t).$$

(8)

The filter-bank topology of DTCWT is shown in Figure 4, where the wavelet functions in ‘Tree $\Re e$’ and in ‘Tree $\Im m$’ form an approximate Hilbert transform pair:

$${\psi }^{\Re e}(t)\approx Hilbert\left[{\psi }^{\Im m}(t)\right],$$

(9)

where $Hilbert[\cdot ]$ denotes the Hilbert transform operator.

In the time domain, there is an equivalent expression,

$$h_1^{\Im m}(n)=h_1^{\Re e}(n-0.5),$$

(10)

where $h_1^{\Re e}(n)$ and are real-valued finite impulse response (FIR) filters corresponding to ${\psi }^{\Re e}(t)$ and ${\psi }^{\Im m}(t)$. In each filtering tree, the scaling functions of ${\psi }^{(\cdot )}(t)$ and ${\phi }^{(\cdot )}(t)$ satisfy the following two-scale relationship

$${\phi }^{(\cdot )}(t)=\sqrt{2}{\displaystyle {\sum }_{n\in \mathrm{Z}}{h}_0^{(\cdot )}}(n){\phi }^{(\cdot )}(2t-n)$$

(11)

$${\psi }^{(\cdot )}(t)=\sqrt{2}{\displaystyle {\sum }_{n\in \mathrm{Z}}{h}_1^{(\cdot )}}(n){\phi }^{(\cdot )}(2t-n),$$

(12)

where the superscript $(\cdot )$ can be either $\Re e$ or $\Im m$. The complex-valued wavelet coefficient series $d_l^{ℂ}(k)$ are calculated via inner product computation between the input signal and the wavelet systems of $\{{\Xi }_{j,k}[{\psi }^{\Re e}]\}$ and $\{{\Xi }_{j,k}[{\psi }^{\Im m}]\}$. These complex-valued series are computed using the following expression:

$$\begin{array}{l}{d}_l^{ℂ}\left(k\right)=\langle x,{\mathit{\Xi }}_{j,k}({\psi }^{\Re e})\rangle +\mathrm{j}\cdot \langle x,{\mathit{\Xi }}_{j,k}({\psi }^{\Im m})\rangle \\ =d_l^{\Re \mathrm{e}}(k)+\mathrm{j}\cdot d_l^{\Im m}(k)\end{array},$$

(13)

where the notation ${\mathit{\Xi }}_{j,k}[\cdot ]$ denotes the translation and dilation operations simultaneously on a function belonging to $L^2(ℝ)$. The mathematical definition of ${\mathit{\Xi }}_{j,k}[\cdot ]$ is

$${\Xi }_{j,k}[\psi ]={\psi }_{j,k}(t)=2^{\mathrm{j}/2}{\psi }_{j,k}(2^{\mathrm{j}}t-k),$$

(14)

where the binary operator $\langle \cdot ,\cdot \rangle$ represents the inner product transform. While, in the reconstruction phase, $d_l(t)$ and $a_i(t)$ can be computed via

$$d_l(t)=2^{\frac{l-1}{2}}\left[{\displaystyle \sum _n{d}_l^{\Re e}(k){\psi }_h(2^{l}t-n)}+{\displaystyle \sum _m{d}_l^{\Im m}(k){\psi }_g(2^{l}t-m)}\right]$$

(15)

$$a_J(t)=2^{\frac{J-1}{2}}\left[{\displaystyle \sum _n{c}_J^{\Re e}(k){\phi }_h(2^{J}t-n)}+{\displaystyle \sum _m{c}_J^{\Im m}(k){\phi }_g(2^{J}t-m)}\right].$$

(16)

Let $J$ be the decomposition stage depth of dual tree wavelet decomposition in Figure 3; $J+1$ wavelet subbands, including $\{d_1(t),\dots ,d_J(t)\}$ as the detail coefficient series and $c_1(t)$ as the approximation series, will be produced.

2.2.2. Nearly Analytic Complex Wavelet Basis

In this paper, the basis for implementing DTCWT is constructed via the method in [32]. The time-frequency atoms of the wavelet basis are shown in Figure 5. This quarter-shift basis is adopted due to its smooth envelope. On the other hand, this quarter-shift basis also has the merit of annihilating energy leakage.

2.2.3. Directional Filtering Atoms of 2D-DTCWT

Associated with the row-column implementation of the wavelet transform, wavelets of 2D-DTCWT can be presented as

$$\psi (x,y)=\psi (x)\psi (y),$$

where $\psi (x)$ is a complex wavelet given by

$$\psi (x)={\psi }_h(x)+j{\psi }_g(x).$$

Therefore, the expression of $\psi (x,y)=\psi (x)\psi (y)$ is

$$\begin{array}{l}\psi (x,y)=\left[{\psi }_h(x)+j{\psi }_g(x)\right]\left[{\psi }_h(y)+j{\psi }_g(y)\right]\\ ={\psi }_h(x){\psi }_h(y)-{\psi }_g(x){\psi }_g(y)+j\left[{\psi }_g(x){\psi }_h(y)-{\psi }_h(x){\psi }_g(y)\right]\end{array}.$$

(17)

The extension of the complex wavelets in Figure 5 into 2D decomposition is shown in Figure 6. 2D-DTCWT produces six strong directional atoms at angles of $\pm {15}^{\circ }$, $\pm {45}^{\circ }$, and $\pm {75}^{\circ }$. As comparison, only three directional atoms can be produced by a classical 2D wavelet transform. Therefore, features in high spectral bands can be preserved by 2D-DTCWT more effectively at the price of redundancy filtering.

3. Intelligent Learning Method Based on ResNet

3.1. Convolutional Layer

A convolutional neural network (CNN) was originally invented to deal with the variability of 2D shapes. A basic CNN is composed of a convolutional layer and a pooling layer [33]. Each level consists of a certain number of feature maps, meaning that CNN possesses good hierarchical feature representation ability from the lower level to higher level [34]. Through the propagation of CNN, the feature maps’ size will decrease layer by layer and the extracted features are more global.

Let a time-domain signal to be processed be represented as $x(t)$. After the multi-scale decomposition, the resulted signals can be represented as ${\mathbf{x}}_S=[{\mathbf{x}}_S^1,{\mathbf{x}}_S^2,\dots ,{\mathbf{x}}_S^L]$, where S is the number of the training samples and L is the decomposition level. The corresponding network output can be written as $y=[y_1,y_2,\dots ,y_S]$. Each $y_j$ means a specific output class from the finite set of classes. Defining $w_{ji}^l$ as the filters with a sliding filter bank and $b_j^l$ as the bias, the convolutional layer output feature maps can be expressed as

$$g_j^l=relu\left({\displaystyle \sum _{i=1}^m{x}_i^{l-1}}\ast w_{ji}^l+b_j^l\right),$$

(18)

where i means the i-th input feature map; j means the j-th output feature map; l means the l layer ; and $relu(\cdot )$ means the activation function in the network is rectified linear units (ReLU).

An example of the convolutional layer is shown in Figure 7. The input of the layer is filtered images using 2D-DTCWT. After the sliding filtering, several feature maps are acquired according to the filter setting.

3.2. Polling Layer

Pooling operations significantly reduce the computational complexity. Max-pooling and average-pooling are the two most common pooling methods across various tasks [35]. In this research, max-pooling can be expressed as

$$X_l^j=down(X_{l-1}^i),$$

(19)

where $down(\cdot )$ is the sub-sampling function to computer the max value of each m × n (m is the vertical downscale; n is the horizontal downscale) region in the $X_{l-1}^i$ map [36].

3.3. Output Layer

The output layer determines the relation label of input signal, and it consists of a full-connected layer and a softmax layer [37,38]. The full connected layer can be expressed as

$$a_j^l=sig\left({\displaystyle \sum _{i=1}^n{x}_i^{l-1}}\times w_{ji}^l+b_j^l\right),$$

(20)

where $sig(\cdot )$ means the activation function in the network is sigmoid.

The final layer is composed of softmax units. Accordingly, the conditional probability is computed as

$$p(y_s=j|a_s;\theta )=\frac{e^{{\theta }_j^T{a}_s}}{{\displaystyle {\sum }_{j=1}^K{e}^{{\theta }_j^T{a}_s}}},$$

(21)

where $y_s$ is the actual output of the network; K is the number of the class; $a_s$ is the feature vector derived by the full connected layer; $\theta$ is the parameter set to be learned via an algorithm for first-order gradient-based optimization of stochastic objective functions Adam.

3.4. Residual Block

The estimation accuracy of deeper networks may become saturated due to degradation problems. To address this problem, He Kaiming proposed a residual network (ResNet) and verified that ResNet achieves considerable accuracy improvements with the increase in the network depth [26]. A fundamental structure of ResNet is shown in Figure 8. The residual mapping to be learned, $F(x)$, can be represented as

$$F=W_2\sigma (W_{2}x),$$

(22)

where x is the input vector and $\sigma$ is the rectified linear unit. Inserting a shortcut connection, the output vector, y, can be represented as

$$y=F(x,\{W_i\})+x.$$

(23)

The form of the residual function F is flexible. It can be inferred from Equation (23) that a function F with only a single layer is similar to a linear layer. Therefore, more layers are necessary for better effect.

3.5. Network Architectures

The network employed in this research is shown in Figure 9a. There are 64 kernels in convolutional Layer 1 and the size of each kernel is set as 7 × 7. The convolution layer is accompanied by an activation ReLU layer and a maxpolling layer. The rest of the layers are the identity block. When the input and output dimensions do not match up, it is necessary to add a convolutional layer in the shortcut path. The convolutional identity block and identity block are given in Figure 9b,c, respectively. As can be seen, in the identity block and the convolutional identity block there are four layers. The four layers are 1 × 1, 3 × 3, 1 × 1, and 1 × 1 convolutions, where the 1 × 1 layers are responsible for reducing and then increasing the dimensions.

4. The Proposed Intelligent Surface Roughness Estimation Method

As a type of feed-forward artificial neural network, CNN possesses good hierarchical feature representation ability from lower levels to higher levels [39]. In this paper, a novel intelligent surface roughness estimation method based on 2D-DTCWT and CNN is proposed for surface roughness estimation problems. The flow chart of the proposed method is presented in Figure 10. Relevant details are also described below.

Step 1. Place the camera on a machined workpiece surfaces to capture the surface images. The surface roughness is estimated by the measuring instrument.

Step 2. Correct the texture skew of the original image with the proposed skew correction algorithm in Section 2.

Step 3. Decompose the adjusted images using 2D-DTCWT with a depth of n to retrieve the useful surface textures. The 2D-DTCWT decomposition depth is set to 3.

Step 4. Divide the processed images into two groups, namely the training dataset and test dataset. The processed images are classified into 10 classes according to the surface roughness measured by the measuring instrument. In this model, each example is a combination of an input object (an image in this research) and a desired output value (designated roughness class). The model is trained with such iterations. The test dataset is utilized to validate the trained ResNet. In this paper, 14 layers are employed for the intelligent surface roughness estimation.

5. Surface Roughness Estimation

5.1. Experiment and Data Acquisition

In order to verify the effectiveness of the proposed method, a series of spheroidal graphite cast iron 500-7 milling cutting experiments were carried out. The milling cutting experiments are conducted on three machining centers (VMC650E/850E produced by Shenyang Machine Tool Company (Shenyang, China), XK63100 produced by Kaichuang Equipment Technology Company (Luoyang, China), and NBH800 produced by Huller Hille (Mosbach, Germany)). The dimension of the workpiece is 200 × 140 × 100 mm. The experimental setup for the milling test is shown in Figure 11. Machining parameters for the milling tests are shown in

Table 1. Information of the milling cutting experiment.

Property	VMC650E/850E	XK63100	NBH800
Milling mode	Down milling	Up milling	Up milling
Milling tool	Carbide disc milling tool	Carbide disc milling tool	Carbide disc milling tool
Feeding speed	200 mm/min	500 mm/min	700 mm/min
Spindle speed	960 rpm	450 rpm	2228 rpm
Cutting depth	0.5 mm	4 mm	0.5 mm

. A mobile roughness measuring device (type: Marsurf PS1 (produced by Mahr (Goettingen, Germany)), Figure 11d) is engaged to measure the machined surface roughness.

Images used to train the proposed algorithm in this paper are collected by a self-made image acquisition system. The schematic sketch of the structure of the image acquisition system is shown in Figure 12.

As can be seen in Figure 12, this image acquisition system is constructed by an industrial camera, a coaxial light source (blue light), and other auxiliary mechanisms. Coaxial lights generated by an LED array are reflected by a semi-reflective mirror to ensure their reflected light directions are identical to the axis of the camera lens, such that the reflected coaxial lights are sent to the workpiece without unevenness. After machining, the image acquisition device was placed on workpiece surfaces and captured digital images of the surface. Details about the parameters for capturing the images are listed in

Table 2. Capture parameters.

Property	Information
Exposure time	150 ms 10 μs
Gamma	0
Goal	49
Gain	1.375×
Saturation	100

.

5.2. Surface Roughness Classification

There are 2040 acquired samples in the experiment, where the surface roughness Ra is estimated by a mobile roughness measuring instrument. Values of the surface roughness Ra are randomly distributed between 0.244 and 7.376. The mean surface roughness Ra is 3.727276 and the standard deviation is 2.217645. As a state-of-art approach in pattern recognition problems [40], CNN can make proper classifications on the labeled dataset.

In practical applications, balanced datasets have been empirically shown to outperform imbalanced datasets [39]. In this research, oversampling which duplicating instances of under-represented classes until a balanced dataset is created is used to lessen the impact of imbalanced datasets. Ten roughness conditions are considered: [−Inf, 0.957), [0.957, 1.67), [1.67, 2.38), [2.38, 3.1), [3.1, 3.81), [3.81, 4.52), [4.52, 5.24), [5.24, 5.95), [5.95, 6.66) and [6.66, $\infty$). The description of the surface roughness condition is listed in

Table 3. Description of the surface roughness condition.

Roughness	Original Sample Number	Adjusted Sample Number	Label
[−Inf, 0.957)	308	370	C1
[0.957, 1.67)	152	370	C2
[1.67, 2.38)	100	370	C3
[2.38, 3.1)	135	370	C4
[3.1, 3.81)	120	370	C5
[3.81, 4.52)	200	370	C6
[4.52, 5.24)	365	370	C7
[5.24, 5.95)	370	370	C8
[5.95, 6.66)	190	370	C9
[6.66, $\infty$)	100	370	C10

. After adjustment, the total sample numbers is 3700.

5.3. Texture Skew Correction and DTCWT Filtering

One of the captured images is shown in Figure 13a. It can be observed that the phenomenon of texture skew obviously occur in the original image. Texture skew influences not only the roughness measurement but also the classification accuracy. As presented in Section 2, an improved Sobel operator enhanced by Hough transform is proposed for the surface texture direction adjustment. The detected edge information is shown in Figure 13b. The adjusted image is shown in Figure 13c. The skewed image results show that the proposed method has good skew correction ability.

With 2D-DTCWT performed on the adjusted image, the decomposition images representing wavelet sub-spaces are shown in Figure 14. Compared with interfering noises, the low frequency periodic component (Figure 14d) contains more useful information of the surface topology. The first two sub-bands located in higher frequency range contain much more irregular contents (Figure 14a,b). w₃(t) sub-band (Figure 14c) can be seen as generated random noises of the boundary noise. Therefore, low frequency periodic component (Figure 14d) will be used as the surface topology in the following research.

A step of normalization is applied to the filtered image such that the image gray value ranges from 0 to 255 in the 8-bit grayscale. In this paper, an additional step is also employed to scaling all values into the range of [0,1]. The feature scaling formula is defined as

$$X^{\prime }=\frac{X-X_{\min }}{X_{\max }-X_{\min }}=\frac{X}{255},$$

(24)

where X is the original signal, X’ is the new signal after normalization.

5.4. Network Training

Surface images were collected from the milled spheroidal graphite cast iron 500-7 workpiece. In the experiment, as mentioned in the previous part, 370 images acquired for each condition. Therefore, the dataset contains 3700 records of images in total. Among these 3700 samplers, 370 images are randomly selected from the testing dataset and the rest are used as the training dataset. Some typical samples of input images are displayed in Figure 15.

As can be seen in Figure 15, throughout the whole image sensing process, noise is added from various sources, which may include fixed pattern noise, dark current noise, shot noise, amplifier noise and quantization noise. These additional irregular noises would strongly influence the generated image gray-scale value [41]. Therefore, the actual camera settings are unstable fluctuated in every single capture process.

In this research, the employed ResNet is shown in Figure 9. The input shape of the network for each image is a normalized patch. After the convolutional layer (the kernel number is 64) there is a batch normalization layer. The size of the kernels in the first layer was chosen to be 7 × 7. Following the first convolutional layer, there is a ReLU. After that, there are one convolutional identity block and two identity blocks. In the final output layer, softmax activation is chosen for the classification to represent the categorical distribution. Adam is a first-order gradient-based algorithm, designed for the optimization of stochastic objective functions with adaptive weight updates based on lower-order moments. In this study, Adam optimizer is used to minimize the categorical entropy.

All of the experiments in this research were performed under a desktop (operating system: Microsoft Window 7, CPU: Intel Xeon E5-2609 v4 @ 1.7 GHz, GPU: NVIDIA Quadro M2000). Because of the hardware limit, the batch size is set to 5 and 0.003 in learning rate. The performance curves representing the training of the established model are shown in Figure 16. The blue solid ascending curve shows the accuracy rate change during the training process where accuracy is the most intuitive performance measure and it is simply a ratio of correctly predicted observation to the total observations. The red solid descending curve corresponds to the loss function values (cross entropy in each epoch). The results show that the loss function value reaches a stable value after 25 epochs, and the accuracy rate achieves stability after epochs 25 with almost 92%.

5.5. Experiment Results

Confusion matrix is an effective visualization tool to estimate the performance of classification algorithm. Each column of the confusion matrix represents the instances in a predicted class (output class), while each row represents the instances in an actual class (target class). Figure 17 illustrates the confusion matrix using ResNet model for the 10-pattern recognition problem, where $\mathrm{C}i$ means the surface roughness condition in Table 3. As can be seen in Figure 17, the trained model presents a good generalization result, with only 18 misclassifications in the entire 370 testing records. Therefore, the accuracy rate of the proposed method is calculated at 95.14%.

The metrics used for evaluating the final classification results also include precision, recall and F1 score. Precision is the ratio of correctly predicted positive observations to the total predicted positive observations. Recall is the ratio of correctly predicted positive observations to the all observations in actual class. F1 score is the weighted average of Precision and Recall. Denoted TP as true positives (the value of actual class is yes and the value of predicted class is also yes), FP as false positives (actual class is no and predicted class is yes), FN as false negatives (actual class is yes but predicted class in no), the three evaluation parameters can be defined as [42]:

• Precision = TP/TP + FP
• Recall = TP/TP + FN
• F1 Score = 2 × (Recall × Precision)/(Recall + Precision)

The results evaluation effects of different surface roughness condition are shown in

Table 4. Precision, recall and F1 score.

Parameters	C1	C2	C3	C4	C5	C6	C7	C8	C9	C10	Mean
Precision	1	0.95	0.94	0.86	1	1	0.97	0.95	0.88	1	0.95
Recall	1	0.95	0.84	1	0.95	0.97	1	1	0.95	0.86	0.95
F1 score	1	0.95	0.89	0.93	0.97	0.99	0.99	0.97	0.91	0.93	0.95

. The acquired mean precision, recall and F1 score were 0.9543, 0.9514 and 0.9512 respectively. The result validates the effectiveness of proposed classification method in surface roughness estimation.

Through visual inspection on the various layers in Figure 18, filter characteristics can be obtained. Filters on the first layer (row 1) show that their main functions lie in the utilization of different brightness contrast ratio to eliminate disturbances of ambient light. Filters on the middle layer (row 2) are used to distinguish background information and main surface roughness information. However, the functions of the filters on the last layer (row 3) are difficult to be explained. It should be noticed that the network function is similar to the surface roughness comparison specimens, which used to tactile and visual comparison of the workpiece surface. Generally, the effect of surface roughness comparison specimens is strongly influenced by the operator prior knowledge and experience. Therefore, the proposed surface roughness estimation method can be a superior alternative for the surface roughness estimation.

6. Discussion and Comparison

(1) The accuracy of surface roughness estimation can be significantly enhanced by the skew correction preprocess. As presented in introduction part, measuring direction errors could lead to frequency change compare with idle direction. Related research also reports that direction inclination can directly affect the classification accuracy [27].

As the comparison, the authors have tested a similar method without texture skew correction preprocess. The results show an inferior accuracy of 74.62%. Moreover, a CNN model of over 30 layers without skew correction preprocess is also explored and the accuracy was found to be 69.73%. The above comparison results are listed in

Table 5. Comparisons the results.

Method	Accuracy
The proposed method (Texture skew correction + 2D-DTCWT + ResNet)	95.14%
2D-DTCWT + ResNet	74.62%
CNN with over 30 layers	69.73%
Texture skew correction + ResNet	79.88%
Texture skew correction + ResNet with 18 layers	91.14%

. In contrast, with the texture skew correction process, the classification accuracy reveals a remarkable improvement in accuracy.

The sources of these differences may be caused by the convolutional kernel weights. Once the convolutional kernel weights are trained, the kernel weights are fixed and not able to respond to directional changes. Therefore, proper skew correction is definitely indispensable for the surface roughness estimation via CNN model.

(2) Proper image filtering is beneficial to reduce the training model complexity. As described in relevant sections, the shallow layers mainly focus on the brightness contrast ratio adjustment and feature enhancement. Therefore, instead of feeding the network with raw data, properly processed image (based on the prior knowledge) is helpful to reduce the model complexity.

As the comparison, the authors also explored a contrasting method without image filtering preprocess and acquired an accuracy rate of only 79.88%. While the classification accuracy rises to 91.14% when the layer number is set to be 18. The above comparison results are listed in Table 5.

7. Conclusions

In this paper, we proposed an intelligent surface roughness estimation method. This novel technique consists steps of texture skew correction, image filtering, and image feature classification. An improved Sobel operator combined with Hough transform is used to correct image skew phenomena. 2D-DTCWT is employed to preserve features more effective for pattern recognition. The resulted images are used to train a ResNet based intelligent model. Major findings of this work can be summarized below:

(1)

ResNet has proven to be an effective method for the surface roughness evaluation. Compared with traditional surface roughness measuring methods, the proposed method is a non-contact one without additional surface damages on the workpiece. Because of the engagement of ResNet in feature learning, this model does not rely on prior knowledge.

(2)

Surface milling experiments show that the proposed texture skew correction method is a feasible way to adjust image variabilities.

(3)

Effectiveness of the proposed novel method is verified by the surface roughness estimation on milled components. Results indicate that this method can distinguish different surface roughness classes with high precision.

(4)

Analysis of filters has demonstrated the function networks can be regarded as an automatic and intelligent realization of comparison specimen based manual surface roughness estimation. Despite similarities in principle the proposed method is also not sensitive to ambient light and does not rely on staffs’ experience. Therefore, it can be served as a superior alternative scheme for this task.

Limited by the measurement mean and the concrete manufacturing condition, only arithmetic mean roughness is investigated in this research. However, authors believe that the proposed method is also capable for more roughness parameters and worth to do further research. In the future, it is worthwhile to investigate the surface roughness recognition for machining tasks of higher machining accuracy requirements based on acquisitions of huge datasets. Besides, investigations of surface roughness estimation applications on other concrete engineering products are also worthy further explorations.