Deep Residual Network Combined with Transfer Learning Based Fault Diagnosis for Rolling Bearing

Abstract

Fault diagnosis of rolling bearings is significant for mechanical equipment operation and maintenance. Presently, the deep convolutional neural network (CNN) is increasingly used for fault diagnosis of rolling bearings, but CNN has challenges with incomplete training and lengthy training times. This paper proposes a residual network combined with the transfer learning (ResNet-TL) based diagnosis method for rolling bearing, which can preprocess the one-dimensional data of vibration signals into image data. Then, the transfer learning theory in parameter transfer is applied to the training of the network model, and the ResNet34 network is pre-trained and re-trained; the image data are selected to be the inputs of the fault diagnosis model. The experimental validation of the rolling bearing fault dataset collected from the practical bench and Case Western Reserve University shows the superiority of the ResNet34-TL model compared with other classification models.

1. Introduction

Rolling bearing is one of the most frequently used parts in mechanical equipment; it will affect the production efficiency of large machinery, thus affecting the benefits of companies and the safety of employees. It is significant to find out the degree and type of faults in rolling bearings timely and effectively.

Traditional bearing fault diagnosis methods based on machine learning mainly include feature extraction and fault classification. Feature extraction is an essential step in fault diagnosis. Xue et al. [1] proposed a novel feature extraction method based on hierarchical analysis. Guo et al. [2] proposed a noise reduction method based on ensemble empirical mode decomposition (EEMD). Hou et al. [3] obtained the optimal features of EEMD by using some feature selection algorithms. Yang et al. [4] based on EEMD designed an effective frequency cut-off criteria and extracted the unknown fault features. Traditional bearing fault diagnosis technology based on machine learning has many requirements for technicians, and there is the risk of information loss [5].

Among the various types of research for rolling bearing fault diagnosis, the most practical and mature method is mechanical signal analysis because it can identify bearing faults by focusing on the intrinsic frequencies with tiny vibrations. Numerous scholars have concluded from their analyses that it is feasible to use current motor frequencies to detect bearing faults [6,7]. However, there are two difficulties in using motor current signals for fault diagnosis: One is that there is no clear signal band to observe bearing faults effectively, and the other is that signal strengths’ temporal characteristics as the deterioration progresses are unclear. Kanemaru et al. [8] diagnosed bearing degradation by monitoring both the bearing’s natural frequency and its rotational frequency and demonstrated the feasibility of using motor current signals for fault diagnosis through relevant motor experiments. Hoang et al. [9] presented a current signal-based bearing fault diagnosis using a Deep Learning (DL) algorithm CNN and a novel decision-level IF technique. The proposed bearing fault diagnosis method has suitable practical applications with cost-effective and relatively high diagnostic accuracy. Singh et al. [10] evaluated the detection of load machine bearing outer ring faults by efficiently extracting features of faulty components from the stator current using continuous wavelet transform. Schoen et al. [11] reported a relationship between vibration and current frequencies caused by faults in the bearings while using vibration monitoring and motor current signature analysis (MCSA) techniques. The above papers found by summarizing that the small amplitude (i.e., low energy) of the fault frequencies extracted from the faulty bearing stator currents can easily be mistaken as noise coming from the system, resulting in the poor diagnosis of bearing faults based on current signals.

Regarding the research on rolling bearing fault diagnosis, more and more scholars have started to experiment with other signals in recent years, such as stray magnetic flux signals, thermal imaging images, acoustic emission signals, etc. Harlişca et al. [12] used statistical analysis of stray flux signals around the motor to conduct fault diagnosis of bearings. Zamudio-Ramirez et al. [13] used an advanced algorithm based on a combination of the MUSIC method and artificial neural networks (ANN), which enables the automatic calculation of fault severity indicators when using stray flux signals. Azeez et al. [14] used a thermal imager to acquire data on rolling bearings and used the bearing temperature to distinguish the type of failure. Although more economical than the traditional signal acquisition method, experiments showed that the approach was more applicable to the outer race of the bearing. Choudhary et al. [15] proposed an emerging two-dimensional discrete wavelet transform (2D-DWT) based infrared thermography method to diagnose different bearing faults in induction motors. Pham et al. [16] used acoustic emission (AE) signals containing bearing health information. They converted AE into 2D spectrograms by Constant Q-Transform (CQT) and then used CNN to infer the bearing status. Chen et al. [17] used Fast Fourier Transforms (FFT) to preprocess AE signals and constructed a low-speed rolling bearing fault diagnosis model based on AE signals and subspace embedded feature distribution alignment. They concluded experimentally that AE signals outperformed vibration signals when rolling bearings were operating at low speeds.

The diagnosis of rolling bearing defects is mainly studied using vibration signal features. Nowadays, the application of deep learning based on 2D-image data to diagnosis methods has gained more attention; it solves the uncertainty problem of artificial feature extraction and promotes the automation of fault diagnosis. Oh et al. [18] produced research based on the Deep Belief Networks (DBN) and using vibration images to diagnose faults of the rotor system. Hoang et al. [19] converted the vibration signal into a 2D form and analyzed the identified flaws of the bearing through Convolutional Neural Networks (CNN) based on vibration images. Ma et al. [20] combined transfer learning with CNN to use 2D time-frequency image sets for bearing fault diagnosis and confirmed RGB images contained more information than grayscale images. Wen et al. [21] proposed a 2-level hierarchical diagnosis network to classify rotating machinery faults with the time-frequency images. Verstraete et al. [22] generated Short-time Fourier, Wavelet Transform, and Hilbert-Huang transform maps and proved the Wavelet Time-frequency image has the highest accuracy. The results of previous research show that the Wavelet Time-frequency transform is more suitable for the time-frequency transform of RGB images. Besides, the Wavelet Time-frequency image can reflect the details of the signal, so it is ideal for fault classification [23].

When using deep learning for fault diagnosis of rolling bearings, CNN models have limitations because they require amounts of labeled data for training. Nevertheless, fault samples are sometimes challenging to meet this condition [24]. In this case, the classifier model is at risk of overfitting and poor generalization, which transfer learning (TL) can solve. Chen et al. [25] used TL to deal with the problems of data imbalance and different distributions. Zhang et al. [26] proposed a method that combines CNN and TL for fault diagnosis. Wu et al. [27] suggested a fault detection and diagnosis method based on TL to overcome the fault data rareness and no label issues. Li et al. [28] proposed a novel fault diagnosis method for deep convolutional domain adversarial TL. Many studies have shown that TL can effectively improve the situation of the model being limited by data and solve the problems of low accuracy of classifier caused by insufficient samples and changes in data distribution [29,30].

The contribution of this paper is to propose a fault diagnosis framework with a high diagnostic rate. The framework uses transfer learning to improve the accuracy of the diagnostic network and converts 1D vibration data to 2D RGB image data before inputting it into the classification model to enhance the stability of the features. After pre-training the ResNet34 network, the time-frequency image set is fed into the ResNet34 network to learn the specific features of the fault data. Comparative experiments are conducted on the CWRU dataset and a laboratory dataset to compare ResNet34-TL with four other models (ResNet34, MobileNet-TL, MobileNet, and CNN) by the loss function value of the training set, the accuracy of the test set, and the accuracy of the validation set to diagnose the ResNet34-TL framework for detecting rolling bearing faults. The performance is analyzed in a detailed comparison.

This paper is organized as follows: the basic theory is given in Section 2. Section 3 presents the rolling bearings fault diagnosis method of ResNet based on transfer learning. Section 4 verifies the experiments and results of the proposed fault diagnosis method in the rolling bearing simulation test bench. Conclusions and future work are provided in Section 5.

2. The Basic Theory of the Model

2.1. Continuous Wavelet Transform Time-Frequency Image Conversion

The continuous wavelet transform (CWT) time-frequency image is a time-frequency representation of the energy density of a signal obtained using the continuous wavelet transform, which decomposes the original signal into time scales, represented by scaling and conversion operations [31]. The Morlet wavelet is used as the mother wavelet because the shape of the Morlet wavelet resembles the features of an impulse occurring in a machine fault.

Assume the input signal is $x(t)$, CWT was obtained by scaling $x(t)$ with Morlet wavelet as shown in Equation (1).

$$W_{\psi }(a,b)={\displaystyle {\int }_{\infty }^{-\infty }x}(t)\cdot {\psi }_{a,b}(t)dt$$

(1)

where a is the wavelet translation, and b is the wavelet scale. To extract more time-frequency information in a limited space, the mother wavelet was used to generate a child wavelet, which is defined by Equation (2).

$${\psi }_{a,b}(t)=\frac{1}{\sqrt{a}}\psi \left(\frac{t-b}{a}\right)$$

(2)

This paper uses the sliding window method to truncate the 1D vibration signal and convert it to a 2D image signal. Before converting the vibration signal to the image signal, the vibration signal is truncated in units of 256 data points by the sliding window function and then the market wavelet basis function is used to generate a time-frequency image. However, the RGB image contains more information than the gray image, and the continuous wavelet transform generates RGB time-frequency images in this paper. The schematic diagram of the sliding window to intercept the signal and convert it into a time-frequency image is shown in Figure 1.

2.2. Convolutional Neural Network

As a standard deep learning algorithm, CNN has a deep feature extraction structure and can dig deeper features of the input features. Its network structure usually includes convolution, activation, pooling, and fully connected layers [32,33]. The convolution calculation equation is

$$y^{l\left(i,j\right)}=K_i^l\ast x^{l\left(r^j\right)}={\displaystyle \sum _{j\prime =0}^{c-1}{K}_i^{l\left(j\prime \right)}{x}^{l\left(j+j\prime \right)}}$$

(3)

where $K_i^{l\left(j\prime \right)}$ is the j’-th weight in the i-th convolution kernel of the l-th layer, with $x^{l\left(j+j\prime \right)}$ as the j’-th perceived weight position in the j-th perceived local region of the l-th layer, and c is the width of the convolution kernel.

The activation layer is to make the neural network better solve the indivisible linear problem. The pooling layer reduces the parameters of the network by maximum pooling, which is defined by Equation (4).

$$p^{l(i,\mathrm{j})}=\underset{(j-1)\mathrm{w}+1\le t\le jw}{\max }\left\{a^{l(i,t)}\right\}$$

(4)

where $a^{l(i,t)}$ is the i-th feature map in the t-th activation value of the l-th layer, and $w$ is the pooling width. The fully connected layer uses softmax as the output function, which is defined by Equation (5).

$$Z_j^l=f\left({\displaystyle \sum _{k=i}^n{w}_i^l{x}_i^l+b_j^l}\right)$$

(5)

Where $w_i^l$ is the weight of the l-th layer, with $b_j^l$ as the offset term of the l-th layer, $Z_j^l$ is the j-th neuron of the l-th layer, and $f\left(•\right)$ is the softmax function when l is the output layer, with $f\left(•\right)$ is the Sigmoid function when l is the hidden layer.

The objective function is to explain the consistency between the output value and the target value, which is defined by Equation (6).

$$E=-\frac{1}{m}{\displaystyle \sum _{k=1}^m{\displaystyle \sum _j{p}_k^j\log \left(q_k^j\right)}}$$

(6)

where m is the number of samples in small batches, $p_k^j$ is the theoretical output of networks, $q_k^j$ is the actual output of the network, and j is the dimension of the data.

2.3. Deep Residual Network

The deep residual network (ResNet) model is a generalized convolutional neural network feature extractor that cleverly addresses the problems of deep learning models in deep learning that may suffer from gradient disappearance or bursting [34]. Besides, good results can be obtained even if the network structure of the deep residual network has a profound number of layers.

In contrast to ordinary deep networks, ResNet networks add the short-circuiting mechanism of residual units to ensure the complexity of the network layers, which is residual learning. The article [35] illustrates that the ResNet network can improve the accuracy by increasing the depth, while ResNet50 and ResNet101 deepen the network depth; however, the training error in the experiments of this paper is not much improved compared to ResNet34, and its training time is also longer. Figure 2 and

Table 1. Results of Resnet network with different depths.

Network Model	Test Accuracy of the Test Data	Time of Training (s)
ResNet34	0.8787	1016.93
ResNet50	0.8476	1115.12
ResNet101	0.8095	1291.68

concluded the results obtained using the Qianpeng dataset from Section 4.1 for different depths of the ResNet network. The results show that ResNet34 has a lower training loss function, a higher correct diagnosis rate for fault types in the validation set, and faster and more accurate test results in the test set.

Therefore, the 34-layer ResNet is chosen in this paper, and its specific residual network parameters are shown in

Table 2. Parameter table of each layer of deep residual network.

Layer Name	Output Size	34-Layer
conv1	112 × 112	7 × 7, 64, stride2
conv2_x	56 × 56	3 × 3, max pool, stride2
		$\left[\begin{array}{c}3\times 3,64\\ 3\times 3,64\end{array}\right]\times 3$
conv3_x	28 × 28	$\left[\begin{array}{c}3\times 3,128\\ 3\times 3,128\end{array}\right]\times 4$
conv4_x	14 × 14	$\left[\begin{array}{c}3\times 3,256\\ 3\times 3,256\end{array}\right]\times 6$
conv5_x	7 × 7	$\left[\begin{array}{c}3\times 3,512\\ 3\times 3,512\end{array}\right]\times 3$
1 × 1		average pool, 1000-d fc, softmax
FLOPs		$3.6\times {10}^9$

.

2.4. Transfer Learning Network

Parameter transfer is a kind of transfer learning. In this paper, transfer learning is used to transfer the parameters of the classification network to the convolutional layer, and the classification network can extract general features. Then, secondary training is performed in the last layers of the network to learn specific features [36]. Transfer learning is generally defined as:

Given a source domain $D_s$ and a target domain $D_t$, the learning tasks are $T_s$ and $T_t$, and the purpose of transfer learning is to use the knowledge in $D_s$ and $T_s$ to improve the target prediction function $f_t(•)$ in $D_t$, where $D_s\ne D_t,T_s\ne T_t$. The source domain $D_s$ and target domain $D_t$ are defined as:

$$\begin{array}{l}{D}_s=\left\{X,P(X)\right\}\\ D_t=\left\{Y,f(•)\right\}\end{array}$$

where X is the feature space, and P(X) is the marginal probability distribution. Therefore, $D_s=\left\{(x_{s_1},y_{s_1}),(x_{s_2},y_{s_2}),\cdots ,(x_{s_n},y_{s_n})\right\}$ can be obtained, and where data samples satisfy $x_{s_i}\in X_s$, the corresponding class label satisfies $y_{s_i}\in Y_s$. Similarly, $D_t$ can be represented by data samples $X_t$ and label space $Y_t$. Network migration based on this theory can effectively improve computational efficiency [37,38].

3. Diagnostic Model

Aiming at the incomplete training problem of CNN, this paper proposes to convert vibration data signals into 2D time-frequency image signals and applies TL to augment the data. The intelligent diagnosis method of deep residual network based on migration learning is shown in Figure 3, and the main steps are described as follows:

• Manually process the faults of three rolling bearings of different sizes, and use acceleration sensors to collect the vibration signals of rolling bearings at different rotational speeds;
• The vibration signals are segmented with equal spacing of signal overlap, and the original vibration signals are reconstructed into time-frequency signals using CWT to obtain the 2D time-frequency images of the reconstructed signals.;
• Import the pre-training weights into the classification model. Reasonably divide the training set, validation set, and test set, and use the ResNet-TL model based on these images for training and testing to realize the classification of different health states of rolling bearings under minor sample conditions.
• Calculate the fault diagnosis results and visualize the test set accuracy of the ResNet-TL model.

4. Experimental Study

In this section, experiments are conducted on two data sets: one is the QianPeng experimental dataset, and the other is the public dataset from Case Western Reserve University. The two sets of experiments use the initial ResNet34, MobileNet, MobileNet-TL, and CNN as comparison methods in the trained cross-entropy loss function values, training accuracy, validation accuracy, confusion matrix of the test set, and training time to verify the superiority of ResNet34-TL. Among them, the initial learning rate of each network is 0.0005, the Batch Size is 32, and the number of iterations is 20, where the optimization method of a deep convolutional neural network is the Adam optimization algorithm. The hardware and software environments in which the experiments were conducted are shown in

Table 3. Experimental environment description.

	Part	Configured Version
hardware	CPU	Intel Core i9-10900
	GPU	NVIDIA Quadro RTX 4000
	Memory	32G
software	Operating System	windows10 64 bit
	Python	3.7.11
	Pytorch	1.7.1
	CUDA	10.2.89

.

4.1. Classification of Qianpeng Datasets

(1) Dataset: The experimental data set in this section is obtained from the vibration signal data set collected by the QPZZ-II rotating machinery fault simulation experiment bench. The experimental platform and the faulty bearing are shown in Figure 4. The experiment uses a piezoelectric acceleration sensor model CA-YD-187 to pick up the bearing vibration signal. The acquisition system is based on the NI USB-4431 multifunctional high-precision data acquisition module from National Instruments, USA. The acceleration signal is first converted to an electrical signal by the acceleration sensor, then converted to a voltage signal by the charge amplifier, and converted to a digital signal by the data acquisition card. Finally, the code written by LabView software is linked to the data acquisition card for data acquisition and storage.

The experimental bearing speed is 1188 r/min, the sampling frequency is 12 kHz, and the loading of the tested machine is zero. The bearing fault states in the experiment are divided into seven kinds, which are normal bearings, rolling element faults of different fault sizes, inner ring faults of different fault sizes, and outer ring faults of different fault sizes, as shown in

Table 4. Introduction to image data sets.

Fault Type	Fault Size (mm)	Total Sample Qty	Samples in the Training Set	Samples in the Validation Set	Samples in the Test Set	Label
ball wear	0.05	300	210	60	30	B00050
	0.45	300	210	60	30	B00450
inner race wear	0.05	300	210	60	30	IR00050
	1.50	300	210	60	30	IR01500
outer race wear	0.05	300	210	60	30	OR00050
	1.50	300	210	60	30	OR01500
normal bearing	0	300	210	60	30	NORMAL

. Examples of transformed images are shown in Figure 5, which shows that the light and dark distributions of the converted images differ for different fault levels on the same fault type.

(2) Testing Accuracy: The image dataset is input to the trained network, the loss function of the model and the accuracy of the validation set are obtained after 20 iterations of training. The results are shown in Figure 6. The loss error of the network training shows that the ResNet34-TL model based on migration learning pre-training can drop rapidly at the beginning of the iteration and eventually stabilize. However, the loss function of the CNN network has been kept at a shallow value, and the accuracy from the validation set of the ResNet34-TL model shows higher and more stable accuracy. The already trained ResNet34-TL model performs well on training loss error and accuracy tests.

The trained models were used for intelligent classification of the test set. All models were trained ten times to prevent random uncertainty in models, and the average of the ten tests was used as the final diagnostic result. The results are shown in

Table 5. Average results for each model.

Model	Test Set Accuracy	Time of Training (s)	False Alarm Rate	Missing Alarm Rate
ResNet34-TL	96.29%	1044.99	6.67% (2/30)	0.00% (0/180)
MobileNet-TL	93.05%	762.99	6.67% (2/30)	0.55% (1/180)
ResNet34	87.87%	1016.93	13.33% (4/30)	0.55% (1/180)
MobileNet	73.62%	783.89	26.67% (8/30)	0.55% (1/180)
CNN	90.23%	1022.46	13.33% (4/30)	0.00% (0/180)

; the ResNet34-TL model has the highest test accuracy of 96.29%. In this section, the ResNet34-TL model is sufficient to solve the accurate diagnosis of rolling bearing faults with fewer samples during network training and with only 20 iterations, which has application value for sizeable rotating machinery production work. It is also evident from the training time that the proposed network structure can substantially improve the recognition rate of fault types without affecting the speed of the actual network operation. In addition, the ResNet34-TL model has the lowest false alarm rate and missed alarm rate, which is 6.67% and 0.00%, respectively, and the ResNet34-TL model certainly has advantages.

For more clarity in showing the superiority of the ResNet34-TL model, visualization of the test set data using the confusion matrix shows the accurate recognition rate of the ResNet34-TL model with the best training results for each type of fault, as shown in Figure 7. The horizontal direction represents the true labels, and the vertical direction represents the predicted labels. The 7 × 7 matrix is the number of samples of each type, and the percentage includes each type’s prediction accuracy rate, false alarm rate and missing alarm rate. The confusion matrix for each method shows that ResNet34-TL has the highest correct fault diagnosis rate for each type.

4.2. Classification of CWRU Datasets

(1) Dataset: In this section, experiments are conducted using a publicly available dataset from the Bearing Data Center of Case Western Reserve University [39] to verify the robustness of the ResNet34-TL model in terms of diagnostic performance. Its sampling frequency is 12 kHz, the speed is 1797 r/min, no load is applied in the radial direction, and the data of standard bearings and bearings with different types of faults are collected. The fault depth of each kind of faulty bearing is 0.280 mm, and the fault width is 0.178 mm and 0.533 mm. This section selects two fault sizes of the rolling element, two fault sizes of the inner ring, two fault sizes of the outer ring, and the normal condition. The specific dataset description is shown in

Table 6. Introduction to image data sets.

Fault Type	Fault Size (mm)	Total Sample Qty	Samples in the Training Set	Samples in the Validation Set	Samples in the Test Set	Label
ball wear	0.178	300	210	60	30	B007
	0.533	300	210	60	30	B021
inner race wear	0.178	300	210	60	30	IR007
	0.533	300	210	60	30	IR021
outer race wear	0.178	300	210	60	30	OR007
	0.533	300	210	60	30	OR021
normal bearing	0	300	210	60	30	NORMAL

. An example of converting the vibration signal collected by the CWRU rolling bearing test bench into a wavelet time-frequency image by overlapping sampling interception is shown in Figure 8.

(2) Testing Accuracy: Importing the CWRU image dataset converted using CWT into each classification model, the results are shown in Figure 9. It can be seen from the loss error of the training set of the network that the ResNet34-TL model can drop quickly at the beginning of the iteration and eventually stabilize. Moreover, the loss function of the ResNet34-TL model has decreased more quickly. On the other hand, the ResNet34-TL model has achieved more than 90% accuracy since the first generation, and the accuracy has reached 99% in the fourth generation. The accuracy of the diagnostic networks that have been improved by transfer learning is higher than the original diagnostic networks, indicating that the application of transfer learning has led to optimizing the networks.

The trained models are used for intelligent classification on the test set, and the average of the 10 test accuracies and pre-training times is taken as the final diagnosis result. The results are shown in

Table 7. Average results for each model.

Model	Test Set Accuracy	Time of Training (s)	False Alarm Rate	Missing Alarm Rate
ResNet34-TL	99.86%	1054.88	0.00% (0/30)	0.00% (0/180)
MobileNet-TL	99.52%	752.15	0.00% (0/30)	0.00% (0/180)
ResNet34	97.48%	999.05	0.00% (0/30)	0.00% (0/180)
MobileNet	93.00%	756.35	0.00% (0/30)	0.00% (0/180)
CNN	97.86%	1037.30	0.00% (0/30)	0.00% (0/180)

, the ResNet34-TL model achieved 99.86% accuracy on the test set, but the required training time was the longest. Although the MobileNet-TL model performed well in terms of time and accuracy, the ResNet34-TL model performed more consistently and robustly in the experiments with different datasets, proving that the ResNet34-TL model is more stable and robust.

In addition, Figure 10 shows a confusion matrix plot of the best test results for all models. The confusion matrix results show that through 20 generations of training, each deep learning model has been able to classify each type of fault on the test set with a relatively high rate of correct classification. In contrast, the ResNet34-TL model can achieve 100% accuracy in each class.

5. Conclusions

This paper studies a ResNet-TL fault diagnosis of rolling bearing, the ResNet34-TL model is proved to be superior in the classification problem by the experimental, and the following conclusions can be obtained:

(1) Before fault diagnosis, CWT converts the vibration signal into the image signal, which is used to solve the instability problem of manual feature extraction;

(2) Applying the theory of parameter transfer in transfer learning to the ResNet model allows the network to improve fault diagnosis accuracy without significantly increasing the required training time;

(3) The proposed ResNet-TL model solves the problem of incomplete model training. Experiments prove that the final fault identification rate of the deep residual network rolling bearing fault diagnosis model based on migration learning is the best.

This study provides a solution for intelligent fault diagnosis of rotating machinery and demonstrates the potential of combining migration learning with other deep learning models in fault diagnosis. In future research, we will target the direction of the classification model after fusing the transfer learning models so that the network can significantly reduce the time used while achieving high diagnostic rates.